Jack Bear

Your website is a terrible API: serving agents a different page at the edge

Every enterprise site we work on was built for a human with a browser. When an AI agent fetches the same URL it receives navigation, cookie banners, client-rendered components and marketing prose, then has to guess at the facts underneath. Retrieval quality suffers and the citation goes to whoever structured their data better.

Our first attempt put the structured data on a separate directory subdomain. It worked, and then it did not. Crawl budget fragmented, authority did not transfer, and the subdomain was treated as a secondary source rather than the primary one. This talk is the rebuild. We now map structured data onto the client's existing sitemap and use a Cloudflare worker to detect verified agent requests and serve an agent-optimised representation of the same page in place of the human HTML.

I will walk the architecture end to end: verifying agent identity rather than trusting user agent strings, deciding what belongs in HTML versus JSON-LD versus Markdown versus an MCP tool call, keeping both representations semantically identical, and the canonicalisation and cloaking questions every reviewer raises within thirty seconds of seeing this.

Includes a live side-by-side demonstration of a production enterprise page as a human sees it and as an agent sees it, plus before and after crawler and citation data from the real world. I will also cover what still does not work and where I think this approach may be wrong.