What Killed My Chat-as-a-Service?

In 2024, I built Fermi AI—a multi-tenant “Agent-as-a-Service” platform for the SMB market. The stack was robust: a custom RAG pipeline with automated web-scraping/chunking and a React-based deployment widget. Technically, it was an engineering win; commercially, it was a “well-engineered failure.” This talk is a technical autopsy of an AI startup that survived the “demo phase” but died in the production trenches. We will move past model benchmarks to discuss the four architectural “killers” that no one warns you about: • The Latency-Value Paradox: SMBs operate on “Zero-CTO” logic. They don’t value a $O(\log n)$ vector search; they value a lead in their CRM. I’ll share production stories of optimizing for “Time to Value” over “Time to First Token”. • Design as an Engineering Guardrail: In a non-deterministic system, UX is a technical constraint. I’ll explain why we moved from “Open Chat” to Constrained State Machines (buttons, cards, and structured flows) to prevent “prompt drift” and manage graceful failure when the LLM reached the limits of its context window. • The “Dirty Data” Integration Trap: The hardest part wasn’t the prompt engineering; it was the idiosyncratic plumbing. I’ll discuss the reality of building RAG for messy, unoptimized Squarespace/Shopify sites and the technical debt incurred by syncing with legacy, rate-limited SMB CRMs.