4 min read - On-Device Language Models: When Local Inference Makes Sense

On-device inference can keep a model close to the user and available without a round trip to a hosted API. It can also create a fleet of model runtimes that must be secured, updated and evaluated across different hardware. The architecture is useful in the right setting, not inherently faster, cheaper or more private.

Earlier versions of this article included device speeds, API prices, electricity estimates and break-even periods. Those figures depended on unspecified benchmarks and became obsolete. They have been removed.

Begin with the constraint

Local inference is worth testing when at least one constraint is real:

• the workflow must continue during poor or absent connectivity;
• sending the input to a remote inference service is not acceptable;
• interactive latency must remain predictable on known hardware;
• the task is narrow enough for a model that fits the device;
• central inference capacity is unavailable or undesirable.

If none applies, a managed service may be simpler to update, monitor and scale.

Model size is not the decision

Parameter count does not determine task quality, memory use or speed by itself. Architecture, active parameters, context length, cache, numerical format, runtime and hardware all matter. A model that loads successfully can still be too slow or inaccurate for the workflow.

Quantisation reduces the precision used to store model weights. The llama.cpp documentation explains that this can shrink a model and may speed inference, while also introducing accuracy loss. The correct format is therefore an empirical choice for a particular task.

Pruning and distillation are separate techniques with their own training and evaluation requirements. Do not promise a generic reduction or quality loss without identifying the model, method, dataset and measurement.

Privacy depends on the whole path

A local model can avoid transmitting prompts to an external model API. Data may still leave the device through logging, crash reporting, model downloads, analytics, connected tools or application sync. The runtime may also expose a local server without authentication.

Map the complete data path and verify it with network observation. Encrypt sensitive local storage, minimise logs, authenticate any service exposed beyond the process boundary and define how model files are obtained and checked.

Updates become a product responsibility

Hosted APIs centralise model changes. On-device systems distribute them. The team needs a way to:

• identify the model and runtime on each supported device;
• verify downloaded artefacts;
• roll out and roll back versions;
• rerun the task evaluation after a change;
• handle devices that cannot load the new model;
• retire vulnerable or unsupported runtimes.

An offline feature also needs an offline failure design. Tell the user when the model is unavailable or below a confidence threshold, and avoid silently substituting a materially different model.

Benchmark a deployment candidate

Use the exact target device, runtime, model file and application prompt. Record time to first output, sustained generation, peak memory, battery or power impact where relevant, thermal behaviour during repeated tasks and quality on a reviewed evaluation set.

Test long inputs and concurrent application activity. A short terminal prompt on a cool device does not represent a production session.

Compare full operating cost

On-device inference can remove a per-request API charge, but not cost. Include hardware already required by the product, engineering, packaging, support, security review, update distribution and the capacity lost when a device runs the model. Compare those costs with current hosted-service invoices for the same measured workload.

There is no universal break-even period. The decision should be recalculated when the model, hardware, volume or vendor price changes.

Use a hybrid boundary when evidence supports it

A practical design may keep classification, redaction or a small offline feature on the device while routing harder tasks to a hosted model after consent and policy checks. The boundary should follow evaluation results, not an assumption that all local models are weak or all remote models are better.

Local inference earns its place when it meets a defined task threshold and the team can operate the resulting device fleet. That is a narrower claim than “the future is on-device,” and a more useful one.

Primary source

• llama.cpp quantisation documentation, reviewed 27 August 2026