LLM Quantization

LLM Quantization for Mobile Deployment: What Changes Size, Speed, and Quality

A practical guide to quantizing LLMs for mobile deployment. Learn when INT4 beats INT8, what quality regressions to watch for, and how to ship smaller models without breaking the rollout.

LLM Quantization for Mobile Deployment: What Changes Size, Speed, and Quality

Quantization is the step that turns “this model runs on my workstation” into “this model fits on a phone people actually own.”

For mobile teams, the question is not whether to quantize. It is how far to quantize, which runtime will carry the artifact, and what quality loss you can tolerate before the product feels worse than a smaller baseline model.

This guide covers the practical decisions: INT8 versus INT4, where quality usually degrades first, how quantization changes device coverage, and what to validate before you roll a new model across a fleet.


The short version

Use INT8 when:

  • you want the safest quality retention
  • the model already fits your target devices
  • you are optimizing for predictable behavior over maximum compression

Use INT4 when:

  • the unquantized or INT8 model is too large for mainstream phones
  • download size matters as much as runtime speed
  • you need broader device coverage and can benchmark quality carefully

The usual production path is simple:

  1. start from the smallest model that can do the job
  2. quantize to INT8 and benchmark
  3. test INT4 on the same prompts and device cohorts
  4. ship the smallest version that still clears your quality bar

Most mobile teams get more value from a smaller model plus disciplined rollout than from chasing the largest model that barely fits. If you are still choosing the packaging layer, read our guide on CoreML vs ONNX for mobile AI.

What quantization actually does

Quantization stores model weights with fewer bits. A float16 model uses 16 bits per weight. INT8 uses 8. INT4 uses 4.

That change affects three things immediately:

  • artifact size
  • RAM pressure during inference
  • throughput on constrained hardware

It can also affect a fourth thing that matters most to product teams:

  • response quality

The first three are easy to measure. The fourth is where teams get sloppy.

A quantized model can still look fine on a short demo prompt while quietly getting worse at instruction following, long-tail classification, multilingual output, or tool-call formatting. That is why quantization is not a one-number optimization problem.

Why mobile deployment pushes teams toward INT4

Phones and tablets are unforgiving environments.

You are balancing:

  • app binary size
  • download time on weak networks
  • disk pressure on user devices
  • runtime memory headroom
  • thermal limits
  • older devices that are still inside your support policy

A 3B parameter model that feels reasonable on a Mac can become unusable on a mid-range Android phone once the app, tokenizer, and runtime overhead are included. INT4 is often the difference between “flagship-only experiment” and “real fleet rollout.”

That is why many teams land on:

Quantization Typical role
FP16 / BF16 Lab benchmark, desktop, or high-end tablet testing
INT8 Safer first production build
INT4 Broad mobile deployment

There is no prize for shipping a larger model to fewer users.

INT8 vs INT4 in practice

INT8 is the conservative option. It usually preserves behavior well enough that regressions are modest, especially on classification, extraction, and short-answer generation tasks.

INT4 is the practical mobile option. It compresses more aggressively and often keeps latency and memory within bounds on devices that would reject an INT8 artifact. The tradeoff is that quality loss becomes more noticeable on harder tasks.

Where INT4 problems usually show up first:

  • brittle JSON or tool-call formatting
  • multilingual accuracy
  • long responses that drift or repeat
  • edge-case reasoning
  • small ranking differences in retrieval or reranking tasks

Where INT4 often works well enough:

  • short-form chat
  • summarization with narrow prompts
  • classification
  • extraction
  • fixed workflow copilots with tight output constraints

If the product depends on reliable structured output, evaluate that directly. Do not assume a chat benchmark tells you enough.

Quantization does not replace model selection

Teams waste time trying to save the wrong model.

If a 7B model needs heroic quantization to limp across your mobile fleet, the better answer may be a 1B or 3B model with a narrower prompt template and better routing. We see this constantly in on-device deployments: the biggest improvement comes from picking the right model class, not from one more compression pass.

Start with the product requirement:

  • what latency is acceptable
  • what devices must be supported
  • whether requests stay local or can fall back to cloud
  • whether output must be deterministic or merely useful

Then choose the smallest model that clears that bar. Quantization is a deployment multiplier, not a rescue mission for an oversized architecture.

Format and runtime still matter

Quantization is only part of the artifact story. The same model can move through different formats and runtimes:

  • GGUF for llama.cpp-style local runtimes
  • ONNX for cross-platform packaging and runtime flexibility
  • CoreML for Apple-native deployment

Those choices change how quantized weights are represented, which kernels execute them, and how much work the team carries later. If you need one portable graph across mobile and desktop targets, the safer default is usually ONNX Runtime deployment. If you are Apple-only, CoreML may still be cleaner.

The mistake is treating quantization format as a pure research concern. It is an operational concern because it shapes:

  • conversion work
  • reproducibility
  • runtime selection
  • rollback mechanics
  • validation scope across device cohorts

What to benchmark before release

A quantized mobile model should clear four checks before rollout:

1. Quality on your real tasks

Use prompts and documents from the product, not generic benchmark samples. Measure the failure modes users will notice.

2. Latency by device tier

Flagship iPhones and high-end Android phones hide problems. Include the oldest or weakest devices that are still in scope.

3. Memory pressure and load success

A model that passes one prompt after a cold start can still fail under background memory pressure, after app resume, or on devices with less free storage than your test lab.

4. Rollback safety

You need a known-good prior artifact and a way to target cohorts back to it without waiting for an app-store release. That is part of the same operational discipline in our local AI rollout checklist.

A practical rollout pattern

The safest mobile rollout usually looks like this:

  1. Export one canonical model artifact.
  2. Produce an INT8 build and an INT4 build.
  3. Benchmark both on a representative prompt set.
  4. Start a canary on newer devices with the smaller build.
  5. Watch latency, error rate, and fallback rate by hardware tier.
  6. Expand only if the smaller build stays inside the quality bar.

This matters because quantization is rarely a global yes-or-no decision. The same model may be right at INT4 for newer flagships, INT8 for tablets, and cloud fallback for devices below a memory threshold.

Where Octomil fits

Octomil sits above the quantization pass.

Teams use it to:

  • package and ship quantized models across runtimes
  • roll out by device cohort instead of all at once
  • monitor latency and failures across the fleet
  • route harder requests to cloud when the local path is not enough
  • roll back a bad model version without rebuilding the delivery workflow

That separation matters. Quantization changes the model artifact. Production success depends on how you deliver, monitor, and control it afterward.

If you want a blunt rule that works more often than not:

  • start with the smallest model that solves the task
  • prefer INT8 when quality risk is high and the model already fits
  • prefer INT4 when device coverage and download size are the binding constraints
  • use cohort-based rollout instead of assuming one quantization level fits the whole fleet

That rule keeps the decision grounded in product reality instead of benchmark vanity.

FAQ

Does INT4 always make an LLM faster on mobile?

Often, but not automatically. It usually reduces memory pressure and artifact size, which helps mobile deployment. Actual speed depends on the runtime, kernels, and device hardware.

How much quality loss should I expect from INT4?

It depends on the model and task. Many workflows keep acceptable quality at INT4, especially short-form tasks. Structured output, multilingual use, and longer reasoning often show regressions first.

Should I quantize before or after choosing a mobile runtime?

Treat the two decisions together. Quantization changes the artifact, and the runtime determines how well that artifact actually runs. A clean export path is as important as the bit width.

Is INT8 safer than INT4 for production?

Usually yes. INT8 is the lower-risk starting point. INT4 is often the better mobile deployment choice once you confirm the quality tradeoff is acceptable.

What is the biggest quantization mistake teams make?

Trying to force the wrong model onto the device. Start with a model small enough for the fleet, then quantize. Do not use quantization as a last-minute fix for a model that was oversized from the start.