Tanya Dixit

The Elephant and the Goldfish: Architecture Patterns for Cutting 70% of Agent Token Costs in Production

LLM providers sell you a 2-million-token context window like an elephant that never forgets. If you actually build production agents that way, your latency explodes, your retrieval drifts, and your CFO will shut you down in 90 days. In production, the best agents think like elephants, but operate like goldfish.

Details: While frontier models offer million-token context windows, treating autonomous agents like an "elephant" that carries entire conversational histories and raw tool outputs into every turn leads to compounding inference costs, bloated time-to-first-token (TTFT), and severe retrieval distraction. This talk introduces the Elephant and Goldfish architecture—an open engineering pattern that decouples persistent system state (external session stores and vector indexes) from ephemeral inference context, strictly bounding working memory to lean, task-scoped execution frames. Grounded in reproducible benchmarks across multi-turn agent tasks, we explore four public, cost-cutting optimization layers: enforcing Subagent Boundary Isolation using strictly-typed JSON schemas (<500 tokens) to kill quadratic transcript inheritance; applying upstream semantic context pruning (e.g., cross-encoder sentence masking and extractive chunk filtering) to drop 40%–70% of retrieved RAG bulk prior to tokenization; implementing multimodal triage that prioritizes structured accessibility trees and keyframe diffs over brute-force video/pixel streaming; and shifting evaluations and judging from monolithic frontier models to decomposed, pointwise rubrics on lightweight flash models. Attendees will walk away with an experimental framework and concrete, open-source-friendly code patterns to audit token leakage, benchmark context compression, and slash agent operating costs without degrading task success rates.