James Dumay
Daniel Winter-Wijntjes
James Dumay & Daniel Winter-Wijntjes
Founder & Co-CEO and Founder
Stealth AI Startup
Skippy: The Little Inference Engine That CUDA
Tuesday, 8 December · 3 PM · Research / Deep Tech track
Skippy: The Little Inference Engine That CUDA
What happens under the hood when you run a large language model? This talk explores Skippy, a custom inference engine built for Mesh on top of llama.cpp, and the engineering behind local and distributed AI inference.
We'll start with a practical primer on how inference works, then look at splitting large models across multiple computers, reusing work through KV caching, and accelerating generation with speculative decoding. Drawing on the experience of building Skippy as a standalone engine, we'll unpack the nuts and bolts, trade-offs and lessons involved in making large models run on the hardware you have.
James Dumay
James Dumay is the founder of a stealth AI startup, building tools for local and distributed AI inference. He works on Mesh and Skippy, a custom inference engine based on llama.cpp, with a focus on running large models across multiple computers and making inference faster through KV caching and speculative decoding.
Daniel Winter-Wijntjes
Ex-Atlassian Head of Engineering, now co-heading up a stealth startup in the distributed AI inference space. Active contributor to mesh-llm, with a specific interest in splitting model inference across multiple systems.