Programme topic
Evals & evidence
Define good, preserve the evidence and decide whether a change is ready to ship. Explore blind comparisons, human-labelled cases, trace-based investigation, regression testing and evaluation loops built from real production failures.
18 published talks
Talks about evals & evidence
Define Good, Then Ship: Blind Evals for Agentic Features
Yasith FernandoPrincipal Engineer, Cuttable
In May I rebuilt our brand-strategy pipeline as a fully agentic system, replacing a chain of LLM calls with a managed agent. I compared old and new…
Prompts Rot. Keep the Source.
Drew BreunigCEO and Co-founder, Cmpnd
AI agents learn what we want from our feedback, then compress that intent into prompts, skills and memories—and discard the evidence. The result is a…
Safely releasing AI applications while continuously patching vulnerabilities
Jay YuanStaff Software Engineer, Commonwealth Bank of Australia
AI applications still need regular security patches and dependency updates, but those software changes can also affect the data processing, model…
The Machine That Changed the World, vAI
Andre KolodochkaSolutions Engineer, Sonar
AI agents can write code faster than any team can review it. Code is abundant. Confidence is scarce. Join Andre to learn how leading enterprises are…
Trust is engineered, not granted: why we focus on verifying before background coding agents
Vivek KatialEngineering Lead, Applied AI, Heidi Health
Belen Garcia FavaHead of Engineering, Heidi Health
Heidi is an AI scribe used by 130K clinicians a week. Our 150 engineers ship 100+ PRs daily into prod, and AI made writing code so cheap that review…
The Benchmark Ends. The World Doesn’t: Building Persistent Engineering Environments for Continual Learning
Theodoros GalanosFDE @ APAC, Nomic
Large language models can complete sequences of tasks—but continual learning is not just doing more tasks. Most agent environments reset after every…
Extracting Tacit Knowledge for Production Agents
Khali Kalpa-YoungFounder, Alchymie
An eval is a definition of "good" written down well enough for a machine to grade against. In a real business, "good" isn't written down anywhere. It…
Building AI for students who can't tell you when it's wrong
Ez HerrmannTechnical Solution Architect, Sitback Solutions
Teachers in Australian specialist schools spend two hours or more preparing a single lesson, then adapt it again for every ability group in the room.…
From Prompt Rules to Structural Guarantees: The Harness Behind a Production Analytics Agent
Jiggy KakkadStaff AI Engineer, Quantium
Tinus WillemseExecutive Manager, AI & Data Science, Quantium
Checkout AI answers open-ended questions about retail sales data in natural language. It plans, calls analytics tools over MCP, executes Python in a…
Building a Security Agent: Model choice, Harnesses and Evals
Simon HarloffCTO, Dam Secure
In this presentation, I’ll show how we set out to give developers useful security feedback on every pull request in under three minutes. The benchmark…
Classifiers are dead. Long live classifiers!
Charli PosnerBuilder, Stile Education
Training a classifier used to mean collecting labelled data, choosing an architecture, training a model, evaluating it, and deploying it. Today, for a…
Tools Before Autonomy: What 200,000 Tool Calls Taught Us About Agentic Evals
Dean SosteSenior Machine Learning Engineer, Canva
Production AI problems are rarely observable through deterministic CI checks. They surface as user feedback, bad LLM-as-a-Judge scores, alerts, or…
Give Every Agent a Flight Recorder
Rahul TrikhaPrincipal AI Engineer, Zendesk
Agent teams should not need to file a ticket with a central evaluation team just to learn whether a new prompt, model, or tool made their agent…
How to Change an LLM System Without Guessing
Yulia KuchinaStaff AI Engineer, Software at Scale
We were running a production LLM pipeline that classified legal documents, and every change was a guess. Swap a prompt, change a model — better or…
Don't Fight Hallucinations. Make Them Impossible
Nadia MakarevichPrincipal Engineer, Heatseeker
The Heatseeker AI chat answers data questions for marketers who make decisions with million-dollar budgets. Wrong answers or hallucinated numbers are…
The agents went rogue at 2%
James PeterCo-Founder, JustEvery
Giving an agent a skill sounds straightforward: it loads a Markdown file, follows the instructions and completes the task. Ours launched a…
Where Should the Dice Roll? Placing Non-Determinism Deliberately in Enterprise AI
Vighnesh DeshpandeAI Engineer, Vivanti Consulting
Most production AI guidance assumes you want consistency: pin the prompt, lower the temperature, eval for drift. But a whole class of enterprise use…
From vibes to a systematic eval flywheel: evals for high stakes AI agents
Donna ZhouSoftware engineer, Lorikeet
Rok StularSoftware Engineer, Lorikeet
If you are sceptical about how evals drive real results, or looking for more depth than introductory tutorial videos, this talk is for you. We'll show…