Yasith Fernando

Define Good, Then Ship: Blind Evals for Agentic Features

In May I rebuilt our brand-strategy pipeline as a fully agentic system, replacing a chain of LLM calls with a managed agent. I compared old and new across 44 customers, and the new system won every time. But I was the judge, and it was my build.

So I reran it blind, with labels randomised per customer and three judges per pair who didn’t know a contest existed. The anonymous labels split 25 to 19. Mapped back to the systems, the result was 44 to 0. The preference held up. It still didn’t ship.

My head of product wasn’t convinced, and it took us both a while to say why. Beating the old system didn’t mean the new one was good enough. He sent me strategies he thought were good and bad, with his reasons for each. We turned those into a golden set of 26 real customer fields, a list of failure modes, four deterministic checks and one LLM judge. I tuned the agent against it, and it went live the next day.

I’ll walk through the mistakes and corrections behind this process, showing how each evaluation changed a shipping decision. That includes what to blind, when to use multiple judges, how to turn product judgment into useful checks, and when variation between runs is bigger than the improvement you’re trying to measure. I’ll finish with a short checklist you can use to set up an evaluation for your own agentic features.