Web Directions Year round learning for product, design and engineering professionals

Recently we sat down to book flights for speakers coming in for the conference. Two or three dozen of them. The task itself is genuinely simple: pick the dates, find the right flights, enter the passenger details, pay. Easy when it’s one. Painful when it’s many. Each booking is multiple steps, each step several clicks and rounds of data entry, even though we’re largely reproducing the same information over and over again. The work has been spread, on and off, across several days now, and it has been the source of considerable frustration.

The reality is that this should all already be easily doable by an agent. The structural information is all there — flights, prices, seats, fare classes, passenger details. It just isn’t exposed in a way an agent can consume cheaply. So instead, the work falls to us, and the airlines get to keep us on their websites for hours at a time doing structured data entry we have no business doing.

This is, I think, the shape of an enormous amount of remaining work on the web. Tasks that look simple when you do one of them, and become unconscionable when you do many. The cost is not the task. The cost is the assumption — buried in the page design, the navigation, the form layout — that the consumer is a human doing the task once.

Now, if I believe that an important user of pretty much any website is already an agent, then what are we doing about it? Well, for AI Engineer Melbourne, we’ve actually gone and implemented a way agents can use the conference programme directly.

The programme — speakers, sessions, schedule — is published in two parallel forms. There’s the human-facing version at webdirections.org/ai-engineer/, with the schedule grid and the speaker pages and all the things you’d expect a conference site to have. And there’s a parallel agent-facing version at data.webdirections.org/ai-engineer/ which exposes the same underlying data through a small set of interfaces designed for agents to consume — JSON endpoints, an llms.txt, an MCP server, semantic search.

This is not, in the scheme of things, a particularly hard piece of engineering. The schedule is small. The data is mostly text. The endpoints are mostly views over a single source of truth. And yet I’ve spent as much time thinking about it than just about any other part of the conference site this year, because I think the pattern matters more than this particular instance of it.

Why we did it

A reasonable first question is whether agents needed any of this. Couldn’t an agent just visit webdirections.org/ai-engineer/schedule.php, read the HTML, and figure it out? Yes. Sort of. It’s a brittle yes. Web pages are designed for people, with all the layout and navigation and visual hierarchy that human reading needs and agent reading struggles with. An agent screen-scraping a schedule page might get the right answer most of the time. It will also break in subtle ways the next time we redesign the page, and it will burn many more tokens than it needs to in figuring out structure that we already know.

The deeper reason — and this is the thesis I keep coming back to — is that I think we’ve already passed the point where the most significant use of many websites is a human reading them. For a lot of sites, including ours, agents are now a meaningful share of the audience. For some sites — airline schedules, government forms, product catalogues, anything where the underlying job is structured data retrieval — agents are arguably already the more important audience. The human interface is, for these sites, the leftover from a previous era when the only way to reach the data was through a page designed for eyes.

If that’s right, then the question of how we treat the agent audience stops being a curiosity and starts being a question about what your site is for. If you keep building only for humans, and you make agents work harder than they need to, Or, as in many cases I’ve discovered in my research, impossible, you are essentially shipping a worse experience to the part of your audience that’s growing fastest-Or excluding that audience entirely. And worse still, you’re shipping an experience that you don’t control. Whatever the agent infers from the page is its problem. Whatever you said clearly in structured form is yours.

This is the same argument that has been made for a while now under the heading of the dual-interface pattern: systems should serve their human and agent audiences through deliberate, separate interfaces, rather than expecting one to figure out the other. I wrote about this earlier in the year. The conference data layer is what it looks like when you actually do it.

What’s there

The agent-facing surface has a few pieces, each doing slightly different work.

llms.txt is the entry point. It’s a tiny plain-text document that says, in human-and-agent-readable prose, what’s at this URL space — what the conference is, what data is available, where to find things. The convention was proposed by Jeremy Howard last year (who just happens to be one of our keynote speakers, by the way) and is becoming the de facto way to make a site discoverable to agents. The point of llms.txt is to be the first thing an agent looks at, so that everything else can be smaller and more specialised. Ours is at data.webdirections.org/ai-engineer/llms.txt.

llms-full.txt is the same idea taken further. It’s the entire conference — every session, every speaker, every abstract — dumped as plain text in a structure that fits in any modern context window. If an agent needs to “know about the conference,” it can pull this single file and have everything. No tool calls, no follow-ups. Cheap and complete.

Three JSON endpoints — sessions, speakers, schedule — give the same data in a structured, queryable form. These are for code rather than agents per se: someone building an app, or a script, or any kind of integration. The URLs are stable and the schemas don’t change capriciously. CORS is open. There is no auth. It’s just data.

An iCal feed that drops straight into any calendar app. This one isn’t agent-facing in the LLM sense — it’s just a long-overdue thing for any conference to have, and once we had the data layer it took about ten minutes to add.

A search endpoint at /api/search that does semantic search over session content using Cloudflare Workers AI embeddings (Cloudflare are sponsors of the conference, but we’ve long been using them for things like this). Pass it a query like “production agent observability” and it returns the sessions that are actually about that, even if those exact words don’t appear. This is the piece that opens up the more interesting agent use cases. Token-efficient, low-latency, works against the whole programme.

An MCP server at /mcp that exposes the whole thing as Model Context Protocol tools. An agent connecting to it gets nine named tools — get_conference_info, list_sessions, get_session, list_speakers, get_speaker, get_schedule, search_sessions, search_speakers, and whats_happening_now. Add the URL to Claude Desktop or Cursor or any MCP-aware client and the agent has structured, live access to the conference programme.

The whole thing is auto-published from our internal source-of-truth within seconds of edits. CORS-open, no auth, no rate limits, no permission needed. The data is yours. We made some decisions there deliberately. I’ll come back to them.

Why each layer is there

People sometimes ask why we have so many different formats. Couldn’t we just pick one? The answer is that they’re for different consumers with different constraints, and the cost of supporting each one is small once you have the underlying data well-modelled.

llms.txt is the cheapest possible discovery surface. An agent that knows nothing about us can fetch one short text file and understand the shape of what’s available.

llms-full.txt is the cheapest possible context-window load. If you want an agent to be able to reason about the conference without making tool calls, this file is what you paste into the context.

The JSON endpoints are for systems with strong typing requirements — code that wants to validate, transform, integrate. They’re also the source of truth: every other surface is computed from them.

Semantic search is for agents (or humans) doing fuzzy, intent-led queries. “What’s the most relevant talk to my work in production observability?” doesn’t translate into a structured filter; it translates into a vector similarity computation, and we’ve already done the embedding work.

The MCP server is for agents that want live tool access during a conversation — where the user is sitting with Claude or Cursor open and asking questions, and the agent should be able to look things up in real time without making the user paste anything.

Each of these is roughly the right interface for a different kind of need. None of them is hard to build once you’ve decided the underlying data model. The expensive part is deciding to do the work at all.

What you can do with it

Well, I have a few suggestions, but most importantly, what we hope is people will come up with solutions we haven’t imagined yet.

Build a personal schedule app. The data is open and stable. If you want a custom scheduler that knows your interests, calculates your walking time between rooms, or syncs with your team’s picks, all the inputs are there.

Point Claude or Cursor at the MCP server. Add it to your config and have your agent help you plan your two days. “What sessions are about agent observability and which speakers have I not heard before?” “Which talks on day two clash with the Stile cluster, and which should I prioritise given I work in a regulated industry?” “Tell me which speakers are talking about spec-driven development and pull up the Stile and Beaugeard sessions side by side.” All of this works, today, with the MCP endpoint.

Build a recommender. Take the embeddings, take a description of your role and interests, and rank the 88 (and counting!) sessions by relevance. This is a weekend project at most. It’s also the kind of thing someone in your organisation will probably want, because most engineering teams would benefit from a structured way to decide who attends what.

Build a Slack or Teams bot. “What’s happening now?” “Who’s speaking next in the Leadership track?” “Find me the sessions about evals.” The whats_happening_now tool is in the MCP server specifically for this.

Generate personalised attendee briefings. Feed the speaker JSON and an attendee’s LinkedIn or interests into an agent and have it produce a one-pager: who they should meet, which sessions are most relevant, what to ask in the corridor. This is a tens-of-lines-of-code job, not a project.

Build dual-interface tooling for your own conferences and events. This one isn’t strictly about ours, but the patterns transfer. If you organise something — a meetup, a team summit, a multi-day workshop — you can take this approach and ship it in a weekend. We’re happy to share the underlying patterns.

Run experiments on the data. Use it as a small, well-curated dataset for your own learning. Practice MCP integration against it. Test how different models handle semantic search. Compare the cost of llms-full.txt context-window load versus repeated MCP tool calls. The data is small enough to fit in one head, real enough that working with it teaches you something, and stable enough to be a good substrate.

Build something we haven’t thought of. This is the bit I most want to see. The whole reason for shipping the data layer is that we don’t know what people will do with it, and we want to find out.

The decisions worth naming

A few decisions in the data layer are worth talking about explicitly, because they’re the kind of thing engineering teams routinely get wrong.

No auth, no rate limits, no permission required. It is genuinely open data. We could have charged for API access, gated it behind a developer signup, throttled it to discourage casual use. We didn’t, because the entire point is to make it easy for people to do interesting things with the programme. Friction kills experiments. We chose less friction.

CORS is open everywhere. Anyone can hit the endpoints from their browser. If we’d locked CORS to webdirections.org, we’d have made it impossible to build anything client-side against the data. We chose less friction.

The source of truth lives somewhere stable. All the endpoints derive from a single internal source — our speakers.webdirections.org system — and the public layer auto-republishes within seconds of any edit. This means the agent-facing data doesn’t drift from the human-facing data, ever. Edit-to-live is fast enough that for practical purposes the two views are the same view.

Stable URLs. The URLs at data.webdirections.org won’t change. If you build something against them today, it will keep working. That’s a small commitment in writing and a large one in practice — it constrains how we evolve the schema — but it’s the commitment that makes the data layer actually useful for building on.

The data layer is its own thing, not a side door into the website. This matters because it means we can iterate on the website, redesign it, restructure the IA, without breaking the data. The human and agent surfaces are decoupled. Each can change at its own pace. This is the deepest version of the dual-interface pattern: not just two interfaces, but two systems that happen to share a backend.

What we’re going to learn from it

I don’t fully know yet what people will build with this, and that’s the most interesting part. We’ve made the pattern available. We’ve made the data open. Whatever happens next is up to the people in the community who decide it’s worth exploring.

What I do know is that I expect this to be the default for events and information services, eventually. Not because it’s mandated, but because the dual-interface pattern is what good engineering looks like in a world where half the consumers of your data aren’t human. People are slow to absorb that. The conference, in some small way, is here to argue that they shouldn’t be.

If you build something against the data, tell me what you built and how it went. If you run an event of any kind and want help applying the pattern, get in touch. And if you’re coming to AI Engineer Melbourne in June, the data layer is a small worked example of what the conference is otherwise about: the practical, deliberate, well-engineered version of the future, rather than the demo-driven version.

---

What’s there, in one place

• data.webdirections.org/ai-engineer/ — the index page with all endpoints listed.
• llms.txt — agent discovery doc.
• llms-full.txt — full conference dump as plain text.
• sessions.json, speakers.json, schedule.json — structured data.
• calendar.ics — iCal feed.
• api/search — semantic search.
• MCP server at https://data.webdirections.org/ai-engineer/mcp — full tool access for MCP-aware clients.

Companion pieces

• How long is your loop? — on the spectrum of AI-assisted development.
• What the maturity ladders miss — on the existing taxonomies and what they don’t capture.
• Three conversations, one conference — the framing piece for the AI Engineer Melbourne programme.

The conference

AI Engineer Melbourne runs June 3rd and 4th 2026 at Federation Square. Friends-of-Web-Directions pricing is available until May 15. Register here.