Jon Shen

AI Practice Executive - Strategy and Governance

Suncorp

Oops I Hacked It Again: Six Months of Red Teaming Our AI Agents

In March I onboarded our first team member dedicated to AI Red Teaming. That was a month before the announcement of Claude Mythos Preview, and the open letters that followed from APRA in April and ASIC in May (Australia’s prudential and conduct regulators) warning organisations to act by shoring up their AI risk management and security practices. We moved before these events because we could see the impending and growing risks, and that specialised skills are needed to keep our customers safe.

Six months and eight deployments later we have broken in repeatedly. We have great data scientists and AI engineers, but the reality is that the skills and mindset needed to build robust AI systems are different from what's necessary to defend them. The surprising thing was the multitude of failure modes we observed, across prompts and architecture, demonstrating the many gaps developers must intentionally think about.

This talk covers what we tested and how, including the process failures we didn't expect, and what we've built to prepare for the future. Leaders will leave with practical steps to start building their own AI red team and defending their AI systems.

Jon Shen

Jon is a data science actuary and AI Practice Executive at Suncorp, building AI applications to make insurance easier for customers in their time of need. That means better, faster claims decisions so they can get on with their lives. In his role, Jon ensures deployed AI systems can be trusted by staff and customers alike, by establishing best practices around AI risks, security, ethics and safety. This requires a pragmatic balance between commercial value, risk management, and change adoption.

Jon recently published the dialogue paper "Building Tomorrow: Preparing Australia for the Age of AI" to spark conversations about Australia's approach to AI. He also serves on the Board of the Actuaries Institute.