Build and Evaluate a Security Agent: Models, Harnesses and Evals

Can an AI security reviewer find vulnerabilities it has never seen—or is it simply repeating familiar benchmarks? In this three-hour hands-on workshop, you’ll build a security-review agent and the evaluations needed to tell whether it actually works.

Starting with a supplied TypeScript agent, you’ll examine how its context, tools, prompts and structured outputs shape the result. You’ll run a baseline evaluation, inspect the complete execution trace and investigate benchmark contamination: whether apparently strong performance comes from genuine reasoning or familiarity with public vulnerability data.

You’ll then create replayable test fixtures derived from real CVEs across several vulnerability classes, compare model and harness configurations using recall, runtime and cost, diagnose failed or incomplete results and improve the system against evidence.

You’ll leave with a working security-review agent, CVE-backed evaluation fixtures, a method for separating model failures from harness failures, example vulnerability cases and workshop notes you can adapt to your own language and security work.

Who it’s for

Intermediate AI engineers, application-security engineers and software engineers. You should be comfortable reading TypeScript and working in a terminal. Familiarity with application-security concepts or an agent framework such as the Vercel AI SDK will help, but is not required.

What you’ll need

Bring a laptop with Docker configured, GitHub access and a coding-agent subscription such as Claude Code, ChatGPT or Cursor. Setup and repository instructions will be provided before the workshop.

About the presenter

Simon Harloff

Simon has spent the last decade building platform infrastructure and developer tooling at some of Australia's leading software companies.