Edge AI: Running Models Where the Data Lives

Every phone now ships with a neural processing unit. Cars run vision models on dozens of cameras. Factory gateways score sensor streams continuously. Edge AI — running inference on the device or near it, rather than sending data to a data centre — has gone from research topic to default architecture for anything that needs to be fast, private or cheap at scale. The interesting questions are no longer whether but which models, on what hardware, with what trade-offs.

Why not just use the cloud?

Cloud inference is simpler to operate and gives you the biggest models. Edge inference wins when one of four constraints bites:

• Latency. A driver-assistance system, a robotic picker or a real-time translation earpiece cannot afford a 50–200 ms round trip plus network jitter.
• Bandwidth and cost. Streaming video from a thousand cameras to the cloud costs more than the analysis is worth. Running detection locally and sending only events is orders of magnitude cheaper.
• Privacy. Health data, images of people's homes, voice in a living room — processing on-device means the raw data never leaves. This is a product feature, not just compliance.
• Availability. Mines, ships, farms and vehicles lose connectivity. Local inference keeps working.

The trade is model capability. Edge devices run models with millions to a few billion parameters, not hundreds of billions. The engineering discipline of edge AI is mostly about closing that gap.

The hardware landscape

Inference at the edge runs on a spectrum of silicon:

• Mobile NPUs in phone and laptop chips (Apple Neural Engine, Qualcomm Hexagon, Intel and AMD NPUs) deliver tens of TOPS at a few watts, enough for on-device speech recognition, photo processing and small language models.
• Embedded GPU modules such as NVIDIA's Jetson line power robots, drones and industrial vision at 10–100 W.
• Dedicated accelerators — Google's Edge TPU, Hailo, and a long list of startups — target cameras and gateways at very low power per inference.
• Microcontrollers with a few hundred kilobytes of RAM run keyword spotting and anomaly detection at milliwatts — the TinyML end of the spectrum.

Choosing well means matching the model's compute and memory footprint to the power budget and the required frame rate — and checking that the hardware's software stack actually supports the operators your model uses, which is where many projects stall.

Making models small enough

A model trained in the cloud almost never runs as-is at the edge. The standard toolkit:

• Quantisation: storing weights and activations in 8-bit or even 4-bit integers instead of 32-bit floats. Cuts memory and speeds up inference 2–4× on hardware with integer units, usually with under 1% accuracy loss when done with calibration or quantisation-aware training.
• Pruning: removing weights or whole channels that contribute little. Structured pruning yields real speed-ups; unstructured pruning mostly saves storage.
• Knowledge distillation: training a small "student" model to mimic a large "teacher". This is how most compact language models and vision models are produced today.
• Efficient architectures designed for the edge from the start — MobileNet and EfficientNet lineages for vision, small transformer variants for language, and the growing family of sub-4-billion-parameter LLMs (Phi, Gemma, Llama and Qwen small variants) that run on a phone.

Runtimes such as TensorFlow Lite (now LiteRT), ONNX Runtime, Core ML, OpenVINO and llama.cpp handle the last mile of compiling a model to a specific chip.

Small language models at the edge

The newest wave is generative. Phones now run multi-billion-parameter language models locally for summarisation, rewriting, on-device assistants and photo understanding, and the major mobile operating systems have made on-device models a platform feature. The pattern that's emerging is hybrid: a small local model handles quick, private or offline tasks and escalates to a large cloud model when the task needs it. Getting the routing right — knowing when the small model is out of its depth — is an active engineering problem.

Operating a fleet of models

Deploying one model to one device is a demo. Deploying dozens of model versions to a hundred thousand devices with different hardware, updating them safely, monitoring their accuracy without seeing the raw data, and rolling back when a version misbehaves is the real work. It borrows from MLOps and from mobile release engineering: staged rollouts, on-device metrics, and — increasingly — federated learning to improve models from device data that never leaves the device.