James Dumay
Founder
Stealth AI Startup
Skippy: The Little Inference Engine That CUDA
Skippy: The Little Inference Engine That CUDA
What happens under the hood when you run a large language model? This talk explores Skippy, a custom inference engine built for Mesh on top of llama.cpp, and the engineering behind local and distributed AI inference.
We'll start with a practical primer on how inference works, then look at splitting large models across multiple computers, reusing work through KV caching, and accelerating generation with speculative decoding. Drawing on the experience of building Skippy as a standalone engine, we'll unpack the nuts and bolts, trade-offs and lessons involved in making large models run on the hardware you have.
James Dumay
James Dumay is the founder of a stealth AI startup, building tools for local and distributed AI inference. He works on Mesh and Skippy, a custom inference engine based on llama.cpp, with a focus on running large models across multiple computers and making inference faster through KV caching and speculative decoding.