Not Everything Needs an LLM

I got frustrated. My support tickets kept getting routed to the wrong team. Every misroute added a day to resolution. I decided to fix it.

The obvious approach in 2024 was to throw the problem at an LLM. I knew that wasn’t going to work. The overlap in team responsibilities made it impossible to write a concise prompt. Tokens would burn fast, accuracy would be unpredictable, and the whole thing would be fragile.

So I built Gata instead. It’s an open source ticket router for Zendesk, and the core routing uses a fine-tuned BERT model trained on your organisation’s own ticket history. LLMs do appear in the stack, but for the tasks they’re actually good at: priority classification and ticket summarisation. The routing itself doesn’t need one.

This talk walks through that decision-making process. How do you evaluate whether a problem actually needs an LLM? What signals tell you a fine-tuned classifier will outperform a general-purpose model? And how do you build something that gets more accurate over time rather than drifting with every prompt change?

You’ll leave with a practical framework for matching automation problems to the right tool, and a concrete case study of what that looks like in production.