Total questions: 20 Total points: 100 (20 questions x 5 points each) Duration: 60-90 minutes
“Thanks for making time for this. We’re going to walk through how your team builds software today, with a focus on how AI tools fit into your workflow. There are no wrong answers – we’re trying to understand where you are so we can figure out the most useful next steps. I’ll ask questions across six areas: your AI tooling, development workflow, CI/CD, metrics, governance, and org structure. Feel free to jump in with context at any point.”
Weight: 15 points (3 questions) Time: ~10 minutes
Ask: “Walk me through how your engineers use AI tools today – from IDE to deployment. What tools are in play, and how consistently are they used?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No AI tools in use, or only 1-2 engineers experimenting on their own | | 1 | A few engineers use AI tools (Copilot, ChatGPT) but it is entirely ad hoc and self-directed | | 2 | Multiple AI tools adopted across the team, but no standardization or shared configuration | | 3 | Standardized primary tool (e.g., company-wide Copilot license), some shared config, but gaps in coverage across the lifecycle | | 4 | Standardized toolset covering multiple lifecycle phases, shared configuration, approved tool list | | 5 | Fully standardized and managed AI toolchain across the lifecycle, with shared config, version-controlled settings, and usage tracking |
Ask: “How do you decide which AI tools to adopt? Is there a process, or does it happen organically?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No process; engineers install whatever they want with no oversight | | 1 | Informal process; someone on the team evaluates tools but there is no formal framework | | 2 | Some evaluation criteria exist (security, cost) but the process is inconsistent | | 3 | Defined evaluation process with security review and cost analysis, but slow or bureaucratic | | 4 | Streamlined evaluation process with clear criteria, fast turnaround, and budget owner identified | | 5 | Formal but fast governance: evaluation framework, security review, pilot period, rollout plan, and ongoing effectiveness measurement |
Ask: “What percentage of your engineers use AI tools weekly? How do you know that number?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | “I don’t know” or clearly guessing with no basis | | 1 | Rough guess based on anecdotal observation (“most people I think”) | | 2 | Knows the license count but not actual usage (“we have 50 Copilot seats”) | | 3 | Has some usage data (e.g., Copilot admin dashboard) but does not actively monitor or act on it | | 4 | Actively tracks usage with breakdowns by team or role, reviews periodically | | 5 | Real-time or weekly dashboards showing usage depth (not just logins), with trends and team-level breakdowns, used to drive adoption decisions |
Weight: 20 points (4 questions) Time: ~15 minutes
Ask: “When a new feature comes in, what does the journey from idea to first PR look like? Walk me through a recent example.”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No consistent process; engineers go from ticket to code with no intermediate steps | | 1 | Loose process exists (maybe a ticket template) but no spec phase, AI used only during coding | | 2 | Some features get specs, but it is inconsistent; AI used primarily for code generation | | 3 | Defined workflow with spec phase for major features; AI used in coding and some testing | | 4 | Consistent spec-first workflow; AI participates in spec drafting, coding, and test generation | | 5 | Fully spec-driven workflow where AI is involved at every phase: spec generation, design review, implementation planning, coding, testing, and PR description |
Ask: “Do engineers write specs or design docs before coding? How structured are they?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No specs or design docs; engineers code from tickets directly | | 1 | Occasional design docs for large features, but no standard format | | 2 | Specs exist for most features but quality varies widely; no template | | 3 | Spec template exists and is used for most features; includes basic structure (goal, approach, risks) | | 4 | Structured specs with template enforcement, reviewed before coding, includes acceptance criteria and edge cases | | 5 | Rigorous spec process: structured templates with acceptance criteria, constraints, and test scenarios; reviewed and approved before coding; version-controlled in the repo; AI-consumable format |
Ask: “How does AI participate in the design phase vs. just the coding phase? Is AI involved before the first line of code is written?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | AI is used only for inline code completion; no involvement before coding | | 1 | AI used for code completion and occasional ChatGPT queries during design thinking | | 2 | Some engineers use AI to draft specs or brainstorm designs, but it is not systematic | | 3 | AI is regularly used to help draft specs and plan implementation, with human review | | 4 | AI is integrated into the design phase: spec drafting, design review, implementation planning, with structured prompts and context | | 5 | AI participates across the full design lifecycle: generates spec drafts from requirements, reviews specs for gaps, produces implementation plans, and the output feeds directly into coding agents with full context |
Ask: “Show me your last 3 merged PRs. Can you tell me which parts were AI-assisted?”
This is a “show me” question – ask them to share their screen.
Listening for:
Co-authored-by, ai-tool:, custom trailers)Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Cannot tell which code is AI-assisted; no metadata, no awareness | | 1 | Can guess based on memory (“I think I used Copilot for this function”) but no systematic tracking | | 2 | Some PRs mention AI in the description, but it is inconsistent and manual | | 3 | Convention exists for noting AI assistance (e.g., PR template checkbox, commit message convention) but not enforced | | 4 | Consistent AI attribution via commit trailers or PR metadata, enforced by convention or CI check | | 5 | Automated AI attribution: tooling tags AI contributions with trailers, PR descriptions auto-populated, searchable and auditable, feeds into quality metrics |
Weight: 20 points (4 questions) Time: ~15 minutes
Ask: “Walk me through your CI/CD pipeline. Where does AI-generated code get validated differently from human-written code, if at all?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Standard CI only; no AI-specific steps; AI code is treated identically to human code | | 1 | Awareness that AI code should be validated differently, but no action taken | | 2 | Extra review attention for known AI-generated PRs, but no automated gates | | 3 | Some automated checks that apply more scrutiny to AI code (e.g., stricter linting, mandatory test coverage thresholds) | | 4 | Dedicated AI validation steps in CI: eval gates, AI-specific security scanning, automated quality benchmarks | | 5 | Comprehensive AI validation pipeline: eval gates with Bedrock Evaluations or equivalent, AI-specific security scanning, quality benchmarks, automated rollback triggers, and feedback loops to improve AI prompts |
Ask: “Have you ever had an AI-generated bug reach production? What happened, and what did you learn?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | “Probably, but we don’t track AI origin for bugs” or “AI doesn’t cause bugs” (denial) | | 1 | Aware of at least one AI-related bug, but no systematic tracking or follow-up | | 2 | Can describe specific AI-related incidents, but response was ad hoc | | 3 | AI-related bugs are discussed in retros; some process changes made but tracking is informal | | 4 | AI-origin bugs are tagged in the issue tracker; post-mortems explicitly address AI failure mode; process improvements documented | | 5 | Systematic AI bug tracking with defect attribution, post-mortems that feed back into prompt engineering and eval gates, and quantified AI defect rate trends |
Ask: “How do you measure the quality of AI-generated code vs. human-written code? Is there a difference?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Do not measure code quality in any systematic way, let alone by AI origin | | 1 | General code quality metrics exist (test coverage, linting) but no AI dimension | | 2 | Anecdotal awareness of AI code quality differences (“AI code tends to be more verbose”) but no measurement | | 3 | Some quality metrics tracked with AI awareness (e.g., know their Copilot acceptance rate) | | 4 | Quality metrics explicitly compare AI vs. human code: defect rates, review feedback, complexity scores | | 5 | Comprehensive quality measurement: AI vs. human defect rates, review cycle times, acceptance rates, complexity scores, with dashboards and trend analysis that feeds back into tooling decisions |
Ask: “What’s your deployment frequency and lead time? How has AI affected these numbers?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Do not track deployment metrics; cannot state deployment frequency | | 1 | Rough awareness (“we deploy a few times a week”) but no formal tracking | | 2 | Track deployment frequency and maybe lead time, but have not analyzed AI’s impact | | 3 | Track DORA metrics; have anecdotal sense that AI has helped but no rigorous attribution | | 4 | Track DORA metrics with trend analysis; can show before/after AI adoption data; some attribution methodology | | 5 | Full DORA tracking with AI-attributed impact analysis: can show how AI adoption changed each metric, with controlled comparison and confidence in the attribution |
Weight: 15 points (3 questions) Time: ~10 minutes
Ask: “If your CTO asked you right now, ‘What is AI doing for our engineering velocity?’ – what would you show them?”
This is a “show me” question – if they say they have a dashboard, ask to see it.
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Nothing; would rely on anecdotes (“engineers say they feel faster”) | | 1 | Could point to license costs and adoption numbers, but no velocity or quality data | | 2 | Could assemble a slide deck with some data, but it would take effort and be mostly manual | | 3 | Has a periodic report or dashboard with AI-related metrics, updated monthly or quarterly | | 4 | Real-time or weekly dashboard showing AI contribution metrics: acceptance rates, AI-attributed commits, velocity trends | | 5 | Executive-ready dashboard with real-time AI contribution metrics, ROI calculations, quality comparisons, trend analysis, and automated reporting cadence |
Ask: “What engineering metrics do you currently track? Which ones include an AI dimension?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Minimal or no engineering metrics tracked | | 1 | Basic metrics (tickets closed, deploy count) with no AI dimension | | 2 | Standard engineering metrics (cycle time, throughput) but no AI dimension | | 3 | Good engineering metrics plus 1-2 AI-specific metrics (e.g., Copilot acceptance rate) | | 4 | Comprehensive engineering metrics with AI dimensions: DORA + AI attribution, cycle time with AI breakdown, quality with AI comparison | | 5 | Enhanced DORA with full AI dimensions, AI-specific metrics (acceptance rate, AI commit ratio, AI defect rate, prompt effectiveness), used actively to drive engineering decisions |
Ask: “How do you report AI ROI to leadership? What’s the cadence and what does it include?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No AI ROI reporting to leadership | | 1 | Occasional informal updates (“AI is helping, we should keep paying for it”) | | 2 | Periodic updates with some data (cost of tools, anecdotal time savings) but no rigorous ROI | | 3 | Quarterly reporting with quantified metrics: time savings estimates, tool costs, adoption rates | | 4 | Regular reporting with quantified ROI: measured time savings, quality improvements, cost vs. benefit, delivered to specific exec audience | | 5 | Structured executive readouts with full ROI model: quantified velocity gains, quality impact, cost analysis, hiring/retention impact, with trend lines and forecasts, at regular cadence (monthly or quarterly) |
Weight: 15 points (3 questions) Time: ~10 minutes
Ask: “What guardrails do you have around AI-generated code and AI agents? How do you limit what AI can do autonomously?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No guardrails; AI tools operate with whatever access the developer has | | 1 | Informal guidance (“review AI code carefully”) but no enforced guardrails | | 2 | Some guardrails: AI PRs require human review, but no formal policy or autonomy tiers | | 3 | Documented guardrails: AI review requirements, restricted areas, basic autonomy rules | | 4 | Formal guardrail framework: autonomy tiers (what AI can do alone vs. with review), enforced by tooling, documented and communicated | | 5 | Comprehensive guardrail system: autonomy tiers enforced by CI/CD and tooling, agent sandboxing, restricted codebase zones, regular guardrail review and updates, with audit trail |
Ask: “How do you handle AI access to sensitive data, credentials, or production systems? Does AI get the same access as the developer using it?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | AI tools have the same access as the developer; no distinction, no audit trail | | 1 | Awareness that AI access is a concern, but no action taken | | 2 | Some basic controls (AI tools cannot access production directly) but not comprehensive | | 3 | AI-specific access controls: scoped permissions for AI tools, credential isolation, basic audit logging | | 4 | Comprehensive AI access management: scoped IAM for AI agents, credential exclusion, audit trails, explicit trust boundaries documented | | 5 | Full AI access governance: least-privilege IAM for all AI tools, audit trails with AI action attribution, trust boundaries enforced by infrastructure, regular access reviews, secrets scanning for AI output |
Ask: “Do you have an AI-specific incident response process? If an AI agent causes a production issue, what happens?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | No AI-specific incident response; all incidents handled the same way | | 1 | Awareness that AI could cause unique incidents, but no specific process | | 2 | Some ad hoc handling of AI incidents, but no documented process | | 3 | AI incident response considerations added to existing runbooks; post-mortems consider AI factors | | 4 | Dedicated AI incident response process: AI-specific runbooks, escalation paths, post-mortem template that addresses AI failure modes | | 5 | Comprehensive AI incident response: dedicated runbooks with eval checkpoints, automated detection of AI-related anomalies, AI-specific escalation, post-mortems that feed back into guardrails and eval gates, regular AI incident drills |
Weight: 15 points (3 questions) Time: ~10 minutes
Ask: “Who owns AI engineering transformation in your org? Is there a dedicated person, team, or budget?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | Nobody owns it; purely grassroots, no budget, no executive awareness | | 1 | Grassroots with informal champion (an enthusiastic engineer) but no authority or budget | | 2 | Engineering leadership is supportive and involved, but no dedicated owner or budget line | | 3 | Named owner (e.g., platform lead, DevEx lead) with partial responsibility and some budget | | 4 | Dedicated owner with clear mandate, budget, and executive backing; AI transformation is a stated priority | | 5 | Named owner with dedicated team, explicit budget, executive sponsorship at C-level, AI transformation on the company roadmap with OKRs |
Ask: “How do new engineers get onboarded to your AI toolchain? What does their first week look like with respect to AI tools?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | AI tools are not part of onboarding; new engineers discover them on their own | | 1 | Mentioned informally during onboarding (“by the way, we use Copilot”) but no structured setup | | 2 | AI tools are set up during onboarding, but no guidance on effective use | | 3 | Structured AI onboarding: tools installed, basic usage guide, team conventions documented | | 4 | Comprehensive AI onboarding: tools configured, usage guides with codebase-specific tips, mentoring from experienced AI users, first-week AI tasks | | 5 | Full AI onboarding program: structured setup, codebase-specific prompt libraries, mentoring, effectiveness benchmarks, ongoing training program, feedback loop from new hires to improve the process |
Ask: “What’s blocking you from getting more value from AI in engineering? If you could fix one thing tomorrow, what would it be?”
Listening for:
Follow-up probes:
Scoring: | Score | Evidence | |——-|———-| | 0 | “Nothing, we’re fine” (lack of self-awareness) or “AI isn’t useful for us” | | 1 | Vague blockers (“culture”, “time”) with no specifics | | 2 | Can name specific blockers but has not taken action to address them | | 3 | Specific, actionable blockers identified; some efforts underway to address them | | 4 | Clear understanding of gaps with prioritized action plan; some progress already made | | 5 | Deep self-awareness with specific blockers, root cause analysis, prioritized roadmap, and evidence of iterating on solutions; answer is consistent with the rest of the interview |
“That covers everything I wanted to ask. A few wrap-up items:
- Is there anything about your AI engineering practices that I didn’t ask about but you think is important?
- What’s the most impactful change you’ve made in the last quarter related to AI in engineering?
- We’ll compile this into a report and share recommendations within a week.
Thanks for your time – this was very helpful.”
Note any additional context from the closing in the free-form notes section of the scoring sheet. These answers do not affect scoring but can provide useful color for the final report.