Vivek Katial & Belen Garcia Fava

Trust is engineered, not granted: why we focus on verifying before background coding agents

Heidi is an AI scribe used by 130K clinicians a week. Our 150 engineers ship 100+ PRs daily into prod, and AI made writing code so cheap that review became the bottleneck: our P75 review wait was 14 hours, almost all of it queue time. A 14-hour queue is a reliability problem — it batches changes, delays fixes, and pushes people toward the "just approve it" path. So we built Hubert, a suite of review agents that runs on every PR and can approve the safe ones straight to production.

The hard part wasn't the agent. It was earning the authority to approve. As such we built the eval set out of our own incident history: every PR implicated in a past postmortem, replayed against the reviewer, scored on whether it would have blocked the change. That corpus is the thing that made approval authority defensible to a skeptical engineering org, not hype-d benchmarks on "SWEBench", but "here is the change that took us down in April, and here is the gate that stops it now." Every new incident adds a case; the agent is only ever as good as the failures we've taught it.

I'll cover how we built and scored the replay corpus, the shadow-mode rollout that ran Hubert alongside human reviewers until the disagreement rate was boring, the per-gate telemetry that tells us which reviewers pull their weight and which are theatre, and where the blast radius stops when it's wrong. Since May: 4,000+ auto-approved PRs, about a quarter of everything we merge, zero production incidents traced back to an auto-approval.