Jiggy Kakkad & Tinus Willemse

From Prompt Rules to Structural Guarantees: The Harness Behind a Production Analytics Agent

Checkout AI answers open-ended questions about retail sales data in natural language. It plans, calls analytics tools over MCP, executes Python in a sandbox, and returns a written analysis with charts. In a system like this, failure is rarely a crash: the chart renders, the prose is fluent, and an incorrect figure reaches a decision-maker unchallenged.

Prompts define the agent’s behaviour but cannot enforce it. Enforcement is engineered into the layers around the model, at three points in the lifecycle: a bad answer is caught before it reaches the user, a bad change before it reaches the codebase, and a bad build before it reaches production.

Before the user. The agent does not author its own charts. It calls typed tools whose outputs are validated against data contracts shared with the renderer, so a malformed visualisation cannot be constructed, let alone displayed. Deterministic validation sits between analysis and synthesis, and every figure in the narrative is checked against the data the user can actually see.

Before the merge. Fourteen evaluation metrics gate development: golden-case metrics run on every prompt and plan change, while metrics that need no expected answer score live production traffic. Production failures are replayed in a local development harness that reproduces the full agent stack, and are captured as golden cases before a fix is written.

Before production. Immutable, commit-tagged builds, a single source of deployment truth, and supply-chain scanning ahead of every merge.

None of these layers required a better model. We close with the case for treating the harness, not the model, as the primary engineering surface of a production LLM agent.