On-device AI explained: a guide for developers and product leaders

TL;DR: On-device AI runs the model on the device itself instead of a server. You get near-instant responses, data that never leaves the phone, and features that keep working with no signal. The hard part is not making it run. It is the device matrix, the fallback path and the model update pipeline.

On-device AI means the model runs on the device itself. The phone, the watch, the sensor, the laptop. Not a server somewhere. The data stays put, the answer comes back in milliseconds, and there is no network call in the middle.

Three things follow straight away.

• Speed. No server round trip, so real-time features like live captions and camera effects feel instant.
• Privacy. Sensitive data never leaves the device, which cuts your exposure under the Australian Privacy Act 1988.
• It works offline. That matters a lot once your users drive out of mobile coverage.

People often say "edge AI" and "on-device AI" like they mean the same thing. They do not, quite. Edge AI is the wider bucket: any inference outside a central data centre, including gateways and servers sitting at the network edge. On-device AI is narrower. It runs on the end-user device.

Here is the thing most guides skip. Getting a model to run on a phone is easy. Getting it to run properly across thousands of different phones, for two years, with updates, is the actual job. That is what this guide covers.

How does on-device AI work?

The hardware

Every on-device model runs on one of three processing layers. Which layer handles your workload decides whether the thing is fast, slow, or broken.

Apple's Neural Engine, Qualcomm's Hexagon NPU and Google's Tensor chip are all dedicated accelerators. They beat general-purpose cores badly on neural work. If your target device has no NPU, the runtime drops back to GPU or CPU. Your latency and battery numbers then need recalculating.

The software

Above the hardware sits a stack of layers. Drivers and firmware at the bottom. Then system-level AI services, like Android's Neural Networks API and AICore, or Apple's Core ML. Those hide the hardware differences and route each operation to the best available chip. Your model runtime sits above that. Your app code calls the runtime.

The model lifecycle

Training almost always happens in the cloud. You train a full-precision model on a GPU cluster, then squeeze it down for the device. That usually means:

1. Quantisation. Drop weight precision from 32-bit float to INT8 or INT4. Cuts model size several times over, with some accuracy loss.
2. Pruning. Strip out weights near zero. You get a sparse model that runs faster on hardware that supports it.
3. Distillation. Train a smaller student model to copy a bigger teacher. You keep most of the accuracy at a fraction of the parameters.
4. Ahead-of-time compilation. Compile the model for a specific chip before shipping, rather than at runtime. Apple's Core AI guidance recommends this for good performance on Apple silicon.

Then the model gets packaged, signed and pushed to devices over the air. On Android, Gemini Nano and the AICore service handle model management at system level, so your app can call a shared model instead of bundling its own.

Quantise to INT8 first and measure what accuracy you lost before you try INT4. INT8 alone is usually enough for mobile. INT4 can degrade in ways you cannot predict without profiling on real devices.

On-device or cloud? The honest comparison

In production it is rarely one or the other. Most mature products run both. Knowing the trade-offs tells you where to draw the line. If you are still choosing the model itself, we covered that separately in our guide to AI model selection criteria for apps.

• Latency. On-device gives you well under a tenth of a second for typical mobile models. Cloud adds a network round trip, which can stretch to seconds on a congested regional connection.
• Model size. Cloud can run models with billions of parameters. On-device tops out far lower, depending on the chip.
• Privacy. On-device keeps the raw data local. Cloud means shipping it out, which creates exposure in transit and on the server.
• Cost. On-device inference rides on hardware the user already paid for. Cloud costs you per call, and that bill grows with every user.
• Resilience. On-device keeps working with no signal. Cloud features fail when the connection drops.

The hybrid patterns that actually ship

The pattern that works is tiered. The device takes the fast, small, private jobs. The cloud takes the heavy ones.

• Local first, cloud backstop. Run on-device. Fall back to a cloud call when the local model's confidence drops below your threshold, or the device lacks the hardware.
• Split inference. Run the early layers on-device and the later layers in the cloud. Less data leaves the phone, less compute sits on it.
• Federated learning. Devices train locally on private data and send back gradients only, never raw data.
• Retrieval for freshness. A small local model does the reasoning. A cloud retrieval system supplies current knowledge. The local model stays small, the knowledge stays fresh.

Fallback is not optional. If a device has no NPU or a broken driver, your app has to spot that at runtime and either load a lighter model or call the cloud. Silently breaking is not an option. The same discipline applies to any hybrid setup, which we go into in the LLM integration guide.

Why it matters commercially

Four benefits, each tied to something you can measure.

Real-time UX. Live transcription, object detection, voice assistants. These need inference under about 200 milliseconds to feel responsive. Over a network you cannot promise that. Locally you can.

Less regulatory exposure. Data that never leaves the device never crosses a network and never sits on someone else's server. For Australian businesses handling health, financial or biometric data, that cuts your obligations under the Privacy Act and the Australian Privacy Principles. There is a broader point here about owning your data, not just your model. We also wrote a full breakdown of AI software compliance.

Offline capability. Australia has real coverage gaps outside the capitals. Field workers, remote health staff and regional utilities cannot count on 4G. On-device inference keeps the app useful regardless.

Lower cost at volume. Every inference call you move off the cloud is a call you stop paying for. At millions of sessions, that adds up.

Where you actually see it

Mobile

The obvious ones are the ones people use daily. Live captions, real-time translation in messaging, smart compose, background noise removal on calls, portrait and night mode on the camera. All local, because latency and privacy make cloud impractical. This overlaps heavily with how apps personalise themselves, which we covered in types of AI personalisation in mobile apps.

Wearables, IoT and medical

Smartwatches run local models for heart rate anomalies, sleep staging and fall detection. The data never leaves, which matters for anything medical. Industrial sensors use local machine learning to spot vibration signatures that mean a bearing is about to fail. In both cases the device has to act alone, often with no connection at all.

Browser and desktop

Chrome runs small generative models directly in the browser for summarising, writing help and translation, with nothing sent to Google's servers. Users can turn them off in settings. This pattern is spreading as NPUs become standard in laptops.

Australian scenarios

Data sovereignty, patchy regional coverage and sector privacy rules all point the same way here. A field inspection app for a regional utility. A clinical tool for a rural GP. A site compliance app for a builder. All three need to work with no signal, and none of them should be shipping operational data to a cloud server.

Which frameworks should you use?

The tooling has matured. The short list:

• Apple Core ML and Core AI. Apple's framework for iOS, macOS, watchOS and visionOS. Handles ahead-of-time compilation targeting the Neural Engine, GPU or CPU, and routes to hardware automatically.
• PyTorch ExecuTorch.** PyTorch's edge deployment stack, and the current recommended path for getting PyTorch models onto mobile hardware. Supports quantisation and accelerated runtimes including Core ML.
• Gemini Nano via AICore. Android's system-level generative model service. Your app calls it without bundling a model, and the OS handles updates and acceleration.

When a system service like AICore or Core ML covers your case, use it. You get hardware routing, OS-managed updates, and no extra weight in your binary. Bundle your own runtime only when the system service cannot run the model you need, or you have to support older OS versions.

For cross-platform work, the Google AI Edge toolchain gives you one conversion and deployment pipeline across Android, iOS and web.

What you have to get right in production

This is where projects live or die.

1. Profile before you optimise. Measure P50 and P95 latency, peak memory, energy per inference, and what happens thermally under sustained load. Do it on real hardware. A model that flies on a flagship can crawl on a three-year-old mid-range handset.
2. Define your device matrix early. Minimum supported hardware. Is an NPU required or optional? Which devices fall back to GPU or CPU, and what does that cost you in speed and accuracy?
3. Degrade gracefully. Detect the hardware at runtime. No accelerator, load a lighter model or call the cloud. A missing NPU should never produce a crash or a blank screen.
4. Sign and version your models. Treat model files as security-critical. Sign them, check the signature on load, and version them separately from your app binary so you can update without a full release.
5. Plan the update path. Bundled with the binary, or downloaded after install? Post-install keeps your app store download small, but you then need a download manager, retry logic, and a way back to the previous model when something fails.
6. Instrument without logging raw data. Capture latency, error rates and how often you fall back. Aggregate on the device where you can. Send summaries, not inputs.

Test on the oldest device you support, under thermal stress, with the camera or GPS running. Throttling on an older mid-range Android will cut your NPU throughput noticeably, and you will never see that in a clean benchmark. Getting this repeatable across a whole fleet is its own discipline, and the same rules show up when you wire AI systems into an existing stack.

The limits, honestly

On-device solves real problems. It also brings its own set.

• Hardware fragmentation. Android spans thousands of configurations, with different NPUs, drivers and memory limits. A model that works on a Pixel can behave differently, or fail, on a budget handset.
• Accuracy versus size. Quantisation and pruning cost you accuracy. For most consumer features that is fine. For medical or financial work it may not be. Validate on your task, not a generic benchmark.
• Power and heat. Sustained inference makes the device hot. It throttles to protect itself, and your latency degrades in ways that are hard to reproduce in a lab.
• Update complexity. Managing compatibility, signed updates and fallbacks across a fragmented fleet is the real engineering hurdle. A bad model update is hard to roll back when the model is baked into the binary.
• Security. Model files on a device can be extracted and reverse-engineered. Adversarial inputs can produce strange behaviour. If the device storage is compromised, any local data the model uses goes with it.

Where this is heading

• TinyML and cheaper NPUs. Models under a million parameters are putting inference into microcontrollers and low-power sensors. Dedicated NPU silicon is arriving in mid-range phones, not just flagships.
• Federated learning at scale. Continuous model improvement without centralising anyone's raw data. Directly useful for Australian health and finance work.
• Better compilers. Toolchains are getting smarter at targeting mixed hardware automatically, so less manual tuning to hit accelerated performance.
• System-level model services. The AICore pattern, where the OS runs one shared model any app can call, is likely to become the default on both platforms.
• On-device safety filters. Content classifiers that run locally before output reaches the user, with no cloud moderation call.

The ACM survey on on-device AI models puts hardware-aware engineering as the deciding factor in whether these deployments work. That matches what we see.

A checklist to start with

1. Define the feature. What must the model do? What is your acceptable P95 latency? What accuracy do you need? Does the data have to stay local for legal reasons?
2. Map the device matrix. Minimum and recommended hardware. Which have NPUs, which do not, what happens below the line.
3. Start small and quantised. Pick the smallest model that could plausibly hit your accuracy target. Quantise to INT8. Measure on real hardware before adding anything.
4. Build and test the fallback. Simulate a missing NPU and a failed model load. Confirm you degrade to something, not nothing.
5. Set up signing and delivery. Bundled or post-install. Signature checks on load. A rollback path.
6. Instrument carefully. Latency percentiles, error rates, fallback frequency. No raw inputs.
7. Security review the storage and inputs. Protected directories for model files. Validate inputs. Sanitise outputs.

The projects that go wrong are almost never the ones where the model was too small or the chip too slow. They are the ones where nobody defined the device matrix, nobody built a fallback, and nobody thought about updates until it was already live.

Why this deserves more rigour than it gets

The framing that annoys me most is treating on-device AI as a privacy feature. Privacy is a consequence of running locally. It is not the reason to do it.

The reason is that your product works better, in more places, for more people. Australian developers building for regional markets get this instinctively. A field app that dies when the user drives out of coverage is not a privacy problem. It is a broken product. Local inference fixes the broken product. The privacy win is real and worth money, but it comes second.

The other thing teams underrate is updates. Plenty of effort goes into picking and shrinking the model, then updates get treated as an afterthought. In practice the update pipeline is where these deployments succeed or fail. Signed delivery, compatibility testing across the matrix, a rollback that works. A model you cannot safely update is a liability, not an asset.

The teams who get this right treat the model as a proper software artefact. Its own versioning, its own tests, its own release process. That is the difference between a demo and a platform.

That view comes from a decade of shipping apps, and watching which ones survive contact with real users.

Talk to Devwiz

If you are weighing up on-device AI, the frameworks are the easy part. The device matrix, the update pipeline, the fallback design and the compliance mapping are where projects stall.

Devwiz builds AI apps and mobile apps for Australian founders, CTOs and business owners. We have shipped 200+ apps, including work for the NSW Government, Briometrix, Vivid and Huskee.

Turning an existing offer into a product is its own job, and we cover that in our AI programs work. If you want to see how a lean MVP becomes a multi-tenant platform, read the white-label AI SaaS case study.

Send us what you are building and the devices it has to run on. We will tell you straight whether on-device is the right call.

Frequently asked questions

What does on-device AI do?

On-device AI runs model inference locally on the device's own hardware. That powers things like transcription, translation, image recognition and summarising without sending data to a server. You get lower latency, offline capability and less data exposure.

Can you give me an example of on-device AI?

Apple's Neural Engine running Face ID, Android's Gemini Nano generating text suggestions, and Chrome summarising a page with a local model are all examples. Each runs inference on the device with no cloud call.

Is on-device AI better than cloud AI?

It depends on the job. On-device wins for fast, private or offline features. Cloud wins when you need a large model, current knowledge, or more compute than the device has. Most production apps run both.

How does on-device machine learning handle model updates?

Models ship over the air, either bundled in an app release or downloaded after install. Production setups need cryptographic signing, version management, compatibility testing across the device matrix, and a rollback path when a model breaks on specific hardware.

Does on-device AI help with Australian privacy rules?

It can. Data that never leaves the device never crosses a network and never sits on a third-party server, which reduces obligations under the Privacy Act 1988 and the Australian Privacy Principles. Map your data flows before you pick the architecture.