Anthony Ivan

Running AI Coding Agents at Scale: Adopting New Models, Controlling Cost, and Governing What Agents Do

Running coding agents across a company comes down to three questions: which models to offer, how to control cost, and what agents should be allowed to do. This talk gives four recommendations on those questions, drawn from running coding agents for 12,000 employees at Databricks.

Offer new models early, and let data choose the default: every model reaches everyone on release day, and within a week, test results, user feedback, and cost decide whether it becomes the default. Use separate limits for accidents and for planned work: a daily limit engineers can extend themselves, and monthly limits a manager approves. Look at how agents use tokens, not just the price: one day of logs found tool bugs costing an estimated $1.2M a year. Control what agents do, not only what they can access, because actions that are allowed on their own can be harmful together.

You'll leave with a process for adopting new models, a budget design you can copy, a way to find wasted spend in your own logs, and a measure for each recommendation that shows whether it's working.