Donna Zhou

From vibes to a systematic eval flywheel: evals for high stakes AI agents

If you are sceptical about how evals drive real results, or looking for more depth than introductory tutorial videos, this talk is for you.

We'll show you how Lorikeet, an AI customer service startup, built an eval system to rapidly raise the quality of a complex AI agent. Coach is our AI assistant that helps customers configure their AI concierges that automate millions of tickets every month, with a huge surface area of use cases. We'll share our journey from receiving vague vibes-based feedback about low quality answers, to how we systematically built a flywheel to rapidly improve the quality.