Web Directions Year round learning for product, design and engineering professionals

Edge AI with Direct Device Control: Moving Intelligence Off the Cloud

We're in what might be called the timeshare mainframe moment of AI. Even the devices in our pockets — phones with powerful processors, cameras, microphones — still route most of their AI inference through the cloud. Your voice assistant sends audio to a data centre. Your smart camera uploads frames for analysis. The intelligence lives somewhere else, and your device is mostly a relay.

This is fine for many use cases. But it's a fundamental limitation for others. Anything that needs real-time responsiveness — robotics, accessibility tools, industrial control, live production systems — can't afford the round trip. Anything that handles sensitive data — medical devices, security systems, personal assistants — shouldn't be sending everything to the cloud. And anything that needs to work reliably in places without constant connectivity is simply out of luck.

The edge is where this changes. Not as a compromise or a cost-saving measure, but as a genuinely different architecture where local intelligence meets local data and local action. The AI doesn't just process information — it controls devices directly.

Jeremy Kelaher has been building exactly these systems, using platforms like ESP32 microcontrollers and Axera edge AI system-on-chips. The approach is concrete and practical: run compact speech-to-text and video models directly on low-power hardware, then translate the AI outputs into device control signals through USB and Bluetooth HID interfaces — keyboard inputs, mouse movements, and other human interface device signals that interact with the physical world.

This isn't theoretical. It's happening on hardware you can buy today for a few dollars. Platforms like M5Stack provide ready-made modules that dramatically accelerate edge development. Tools like PlatformIO provide the development environment. The barrier to building AI systems that operate independently of the cloud is lower than most people realise.

The implications are significant. Latency drops from hundreds of milliseconds to single digits. Privacy is preserved by default because data never leaves the device. Reliability improves because there's no network dependency. And entirely new categories of application become possible — systems that need to perceive and act in the same physical space, in real time.

For anyone building AI systems that interact with the physical world — whether that's media production, accessibility, industrial automation, or the growing ecosystem of smart devices — the edge isn't a niche concern. It's where the most interesting engineering challenges are, and where the next wave of practical AI applications will be built.

Jeremy Kelaher is presenting on edge AI with direct device control at AI Engineer Melbourne 2026 on June 3-4, drawing on work at SBS and a lifetime of hands-on electronics and AI experimentation.