Mobile AI
Mobile LLM Memory Budget Guide: How Much RAM Your On-Device Model Actually Needs
A practical memory-budget guide for shipping LLMs to phones and tablets. Covers model size, KV cache growth, iOS vs Android headroom, and when to route requests off device.
Mobile LLM Memory Budget Guide: How Much RAM Your On-Device Model Actually Needs
The first failed mobile LLM rollout usually looks the same. The prototype works on one flagship phone. The team extrapolates. Then older devices start getting killed by memory pressure, generation stalls under long context, and crash reports say nothing useful beyond “process exited.”
Model size is only one line item in the memory budget. Runtime allocations, KV cache growth, tokenizer buffers, concurrent requests, app state, and OS headroom all matter. Ignore any of them and the rollout gets fragile fast.
This guide shows how to build a memory budget for on-device LLMs before you ship.
The fast rule
Do not size a mobile deployment from the quantized model file alone.
For a production rollout, budget for:
- model weights in memory
- KV cache at your expected context length
- runtime workspace and graph buffers
- tokenizer and prompt staging
- your app’s normal foreground memory use
- OS headroom for thermal and background behavior
If the total looks close to the device’s advertised RAM, the device is already out of budget.
A practical memory formula
Start with this rough planning model:
| Component | What to estimate |
|---|---|
| Quantized weights | The model artifact after quantization |
| KV cache | Grows with context length and layer count |
| Runtime overhead | Execution-provider buffers, scratch memory, allocator fragmentation |
| App overhead | Your UI, network state, media, local DB, and anything else loaded in-process |
| Safety margin | 20-30% headroom for OS pressure and device variance |
For small mobile models, the KV cache is often the term teams miss. A model that feels comfortable at 512 tokens can become unstable at 4k context because the cache, not the weights, blows through the budget.
Example budgets by device tier
These are planning ranges, not guarantees. They are still more useful than the usual “it fit once on my phone.”
| Device tier | Usable RAM for model + runtime | Recommended LLM class |
|---|---|---|
| 4 GB phone | ~1.0-1.4 GB | Tiny task models, aggressive quantization only |
| 6 GB phone | ~1.8-2.4 GB | ~1B models, short context |
| 8 GB phone | ~2.8-3.8 GB | 1B to 3B models, moderate context |
| 12 GB tablet / flagship | ~5.0-7.0 GB | 3B models comfortably, selective 7B experiments |
The gap between physical RAM and usable inference budget is the part people undercount. iOS and Android both reserve memory aggressively when the app is foregrounded, background tasks wake up, or the user is switching between apps.
Where the budget really goes
1. Model weights
This is the obvious part. A 1B model at INT4 might land around 500-800 MB. A 3B model can sit closer to 1.5-2.2 GB depending on format and quantization strategy.
That number is not the final in-memory footprint. Some runtimes expand parts of the graph, remap buffers, or keep additional copies during load.
2. KV cache
The cache grows with sequence length. That growth is why a model that looks fine on short prompts can fall apart when users paste a long thread, PDF summary, or support transcript.
Teams often ask whether they should deploy the bigger model or the smaller one. The more important question is often whether they should cap context lower on mobile.
3. Runtime workspace
ONNX Runtime, llama.cpp, Core ML, and other execution paths each carry their own buffer behavior. Some are tighter on CPU. Some need more workspace to get the throughput you want. That tradeoff belongs in deployment planning, not in postmortem notes.
4. Everything else in your app
If your app already loads camera frames, local embeddings, web views, or media caches, your LLM does not get the whole foreground budget. It gets what is left after the rest of the product is alive.
iOS and Android fail differently
iOS
iOS tends to fail hard when you cross the line. You often get jetsam behavior instead of a graceful slowdown. That makes headroom more important than average-case throughput.
For iPhone deployments, plan around:
- less tolerance for memory spikes
- less visibility into failure modes after the fact
- stronger benefit from tighter model targeting by device generation
Android
Android gives you a wider hardware spread and more runtime variance. A model that works well on one 8 GB flagship can still fail on another 8 GB device with heavier OEM services, weaker storage, or more constrained thermal behavior.
For Android fleets, segmentation is mandatory:
- split by RAM tier
- split by SoC family when possible
- watch for OEM-specific memory pressure
- expect more variance in long-context behavior
Context length is a product decision, not a default
The easiest way to wreck a mobile memory budget is to copy a desktop context window onto a phone.
If the use case is:
- short chat replies
- classification
- extraction
- summarizing one note at a time
then you usually do not need a giant local context window. Cap it lower, keep the model stable, and route outlier requests elsewhere.
That routing line is where a control plane matters. Octomil lets teams keep the fast local path for normal requests, then shift long prompts or weak devices away from the on-device model before the app falls over.
A rollout pattern that works
This pattern is boring. That is the point.
Step 1: define per-tier budgets
Pick target classes first. Example:
- 6 GB Android: 1B model, short context only
- 8 GB Android / recent iPhone: 1B or 3B depending on latency target
- 12 GB tablets: larger model, longer context allowed
Step 2: measure peak memory, not just steady state
Record:
- cold load peak
- first-generation peak
- long-context peak
- multi-turn peak
- background-to-foreground recovery peak
Average memory numbers hide the failure cases.
Step 3: route by device capability
Do not publish one policy for every device. Use capability gates.
Step 4: watch regressions on every new model version
A small tokenizer change, graph optimization, or longer default prompt template can move memory enough to break borderline devices. Treat memory like a release gate, not a nice-to-have metric.
When to stop forcing the local path
Run the request somewhere else when:
- the prompt is too long for the target tier
- the device is already under memory pressure
- generation latency drops below your usable threshold
- the product needs a larger model than the device can hold safely
That is not failure. That is a sane routing policy.
If you still need a local baseline to compare against, start with the lower-friction setup in Run LLMs Locally, then move to Pricing, Benchmarks, and the inference cost calculator when you are deciding what stays on device versus what goes to cloud.
Memory-budget mistakes that keep repeating
Sizing from disk footprint
The quantized file looked small enough, so the team shipped it. Runtime memory said otherwise.
Testing only on flagships
Flagships prove the ceiling. They do not prove the fleet.
Ignoring context growth
The first 200 tokens were fine. The fourth long conversation was not.
Treating RAM labels as usable budget
An 8 GB phone is not an 8 GB inference target.
Shipping one policy for the whole fleet
Mixed hardware needs mixed rollout rules.
A simple budget example
Say you want to ship a 3B model to an 8 GB phone class:
- weights: ~1.9 GB
- KV cache at your target context: ~700 MB
- runtime workspace: ~500 MB
- app foreground memory: ~900 MB
- safety margin: ~900 MB
That already puts you near 4.9 GB before the OS decides anything else should stay warm. This can work on some devices. It will still be risky on a wide mobile fleet unless you tighten context, reduce concurrency, or segment harder.
The same app with a 1B model may fit comfortably enough to survive real users instead of demo conditions.
Next steps
- Download Octomil to test local inference on real hardware
- Benchmarks to compare throughput across platforms
- Calculator to compare device execution against cloud spend
- Early access if you need device targeting, staged rollouts, and routing controls
FAQ
Is 8 GB RAM enough for a mobile LLM?
Usually yes for a 1B model, often yes for a 3B model, and still not enough to skip budgeting. The usable headroom depends on context length, runtime choice, and what the rest of your app loads in memory.
Why does a model crash only on long prompts?
Because the KV cache grows with context length. The short prompt proved the weights fit. The long prompt proved the full runtime budget did not.
Should I pick the smaller model or shorten context?
Both are valid levers. Shortening context often preserves the better model on stronger devices, while a smaller model broadens device coverage. The right choice depends on whether quality or coverage is the tighter constraint.
Can I rely on swap or virtual memory on mobile?
Not in any way that makes a bad deployment safe. Once mobile inference starts leaning on heavy memory pressure behavior, latency gets worse and app stability follows.
What is a good safety margin?
Plan to keep at least 20-30% headroom beyond your measured app-plus-model footprint. Borderline deployments look fine in staging and break under real multitasking.