Edge AI: when to run inference on-device, when to send it to the cloud

According to IDC, global AI infrastructure spending more than doubled from 2024 to 2025 and is projected to keep growing sharply in the years ahead. An important shift comes with it: per multiple 2026 industry analyses, inference — not training — now accounts for the majority of hyperscalers' AI compute spend, reflecting a move from experimentation to production-scale AI deployment. For IoT systems, this shift raises a concrete architectural question: once a model is ready, should it run on the device (edge) or should data be sent to the cloud for inference? The answer isn't a general rule — it depends on four concrete factors specific to the problem at hand.

Four deciding factors

• **Latency** — does the system need to respond within tens of milliseconds (emergency stop, robot control, automatic braking), or can it tolerate seconds to minutes (reporting, trend analysis)?
• **Bandwidth** — what is the input data and how much of it? High-resolution video or vibration data sampled at thousands of Hz generates far more traffic than a few sensor readings per second
• **Cost** — the cost of continuously streaming raw data to the cloud versus a one-time investment in edge hardware — this calculation depends on device count and operating duration
• **Sensitive data** — is the data bound by regulation or contract to never leave the site (surveillance footage, confidential manufacturing data, personal information)?

When edge wins

Edge AI fits best when at least one of the following applies: real-time response under 100ms is required and a cloud round-trip can't meet it; network connectivity at the deployment site is unstable or absent; the data volume is too large to stream continuously (video, audio, high-frequency vibration), making bandwidth cost far exceed edge hardware cost; or the data is bound to never leave the site. In the predictive maintenance system for 4 steel plants we deployed, the failure-prediction model built on vibration data and thermal cameras runs right at the edge — not because the cloud isn't powerful enough, but because round-trip latency can't meet the requirement of catching a failure before it happens, and because streaming continuous high-frequency vibration data from 3,200 sensors to the cloud is an impractical bandwidth problem.

When cloud still wins

Cloud remains the right choice when: the model needs higher accuracy than a version quantized down to fit edge hardware can deliver — as covered in why VRAM and HBM matter in AI infrastructure, a 70B model quantized to fit a small edge device trades away meaningful accuracy compared to running full-size on a cloud GPU; inference volume is low or intermittent, making the cost of idle edge hardware most of the time higher than paying per API call; or the model needs frequent updates and synchronizing a new version across thousands of edge devices is far more complex than updating a single central endpoint.

Hybrid architecture: edge pre-processes, cloud handles depth

In practice, most mature IoT systems don't pick one extreme — they use the edge for a lightweight filtering/pre-processing layer (simple anomaly detection, data compression, deciding when something needs to go to the cloud), reserving deeper, less latency-sensitive analysis for the cloud. Choosing the right size and quantization level for a model running at the edge — a topic we cover in more depth in deploying Qwen3-VL for computer vision — directly determines whether a model fits hardware like a Jetson, a Coral TPU, or a microcontroller running TFLite Micro/ONNX Runtime. The same hot-path/cold-path principle we apply when designing a database for IoT time-series data — recent data processed fast in one layer, historical data analyzed deeply in another — applies to splitting inference between edge and cloud too.

An important caveat: a quantized edge model isn't automatically "good enough"

A model quantized to fit edge hardware doesn't necessarily retain the same capability across every kind of input — exactly the jagged frontier principle we've covered before: a model performing well on a test set doesn't automatically guarantee it performs well on every real-world input, and quantization can degrade accuracy unevenly across different input types. Before rolling out a quantized model across a fleet of edge devices, test it specifically against your actual data — don't extrapolate from the original model's general benchmark scores.

Conclusion

The edge-versus-cloud decision for AI inference shouldn't start from "which is cheaper" — it should start from four concrete factors: required latency, input data bandwidth, cost structure at scale, and sensitive-data constraints. Most mature IoT systems don't pick one extreme — they design a hybrid architecture, exactly how we approach the IoT & Sensors layer in every Pilot Build. Get in touch if your system needs a fresh look at where AI should actually run.