Speech To Text
On-Device Speech-to-Text for Mobile Apps: Latency, Memory, and Rollout Plan
How to ship speech-to-text on Android and iOS without turning every voice feature into a server dependency. Covers model size, latency budgets, fallback policy, and staged rollout.
On-Device Speech-to-Text for Mobile Apps: Latency, Memory, and Rollout Plan
Speech-to-text is one of the fastest ways to make an app feel useful. Tap once, talk naturally, get text back. It is also one of the fastest ways to create an operational mess if every transcript depends on a server round-trip.
That tradeoff shows up immediately in production. Cloud transcription is easy to prototype, but the failure modes are obvious once usage grows: higher latency on weak connections, more backend spend, harder privacy reviews, and voice features that stop working the moment the network gets unreliable.
On-device speech-to-text changes the shape of the problem. The microphone stream stays local, first-token latency drops, and the app can still function in elevators, hospitals, warehouses, airplanes, and other places where connectivity is inconsistent. You give up some convenience in exchange for better control.
This guide is about the practical side of that decision: which workloads fit on-device transcription, what hardware constraints matter most, and how to roll out speech models without bundling every future update into an app release.
When on-device speech-to-text is the right call
Local transcription is strongest when the product needs one or more of these properties:
- low perceived latency
- offline or intermittent-network support
- privacy-sensitive audio handling
- predictable unit economics at higher volume
- device-side wake-word, dictation, or note-taking flows
Good examples include field-service apps, clinician dictation tools, voice notes, accessibility features, in-car assistants, and hands-free mobile workflows. In these cases, the user expects the feature to respond immediately. Waiting on a network request makes the whole interaction feel fragile.
If the product mostly transcribes long recordings after the fact, cloud processing can still make sense. Batch transcription is a different problem from interactive transcription. Treat them separately.
The latency budget that users actually feel
Most teams evaluate speech models on word error rate first. That matters, but it is not the first thing users notice. They notice delay.
For interactive mobile transcription, the rough targets are:
| Metric | Feels good | Starts to feel slow |
|---|---|---|
| First partial text | under 300 ms | over 700 ms |
| Stable segment update | under 1 s | over 2 s |
| Finalized short utterance | under 2 s | over 4 s |
Those are product thresholds, not research benchmarks. A model that is marginally more accurate but regularly misses the latency budget will often lose in real use because people stop trusting the feature.
This is the same pattern we see in local LLM deployment. The best model on paper is not always the one you should ship. If you have not read the broader deployment tradeoffs yet, start with the small language models guide and the mobile memory budget guide. The same hardware discipline applies here.
What constrains mobile speech models
Three things usually decide whether an on-device speech stack is viable.
1. Model footprint
Speech models compete for the same storage and memory budget as the rest of your app. If the transcription model is several hundred megabytes, you need a clear reason to carry that cost. Install size still affects conversion. Memory spikes still trigger crashes on lower-end devices.
2. Streaming behavior
A speech model that only works well on full recorded files is a poor fit for live dictation. Streaming support matters because users expect partial text while they are talking, not five seconds after they stop.
3. Hardware spread
The median device is not the demo device. A recent iPhone Pro and a mid-range Android phone will not deliver the same throughput. If you ship one speech model for every device tier, you are forcing your slowest hardware to define the product.
That is why model tiering matters. High-end devices can carry a larger acoustic model or language model head. Lower-end devices usually need a smaller default with tighter streaming windows and simpler post-processing.
A deployment pattern that holds up
The pattern that usually works is simple:
- Bundle a baseline speech model that is good enough for first launch.
- Let the deployment layer fetch better variants in the background.
- Route by device tier, language pack, and product surface.
- Keep rollback instant.
This avoids the two common mistakes.
The first mistake is bundling every speech model the team might ever need. The app gets bloated, update cycles get slower, and most users carry assets they never use.
The second mistake is making transcription depend on a fresh network fetch before the feature works. That defeats the point of local speech.
Octomil is useful in the middle layer. The runtime stays local, while model selection, staged rollout, and health monitoring move out of the app bundle. That is the same split we cover in the MediaPipe deployment guide: keep inference local, move deployment control into the platform.
Choosing a fallback policy
Not every request should stay local forever. The right policy is not “local only” or “cloud only.” It is explicit routing.
Useful local-first policies look like this:
- short-form dictation stays local
- wake-word and command recognition stay local
- offline mode stays local by definition
- long recordings can upload later with user consent
- low-confidence transcripts can request confirmation before any cloud fallback
The important part is that fallback is a product decision, not an accident. If the app silently uploads audio every time the local model hesitates, you still built a cloud transcription feature. You just hid it.
What to measure in production
Do not stop at word error rate in offline evaluation. Production speech needs operational metrics too.
Track:
- time to first partial transcript
- time to final transcript
- device-tier-specific latency
- failure rate by OS version and hardware class
- battery impact during longer sessions
- user correction rate
- percentage of sessions that trigger fallback
User correction rate matters more than most teams expect. If people routinely fix a certain word type, such as names, product terms, or medical vocabulary, you have found the real quality problem. It is often a vocabulary and post-processing issue, not a reason to replace the whole model.
Rolling out speech models safely
Speech updates should be rolled out like any other high-impact model change.
Start with a small canary, ideally on one device group and one language pack. Watch latency, crash rate, correction rate, and session completion. Expand only if those metrics hold.
This sounds obvious, but many teams still treat speech models like static assets. They ship a new build, wait for app-store adoption, and hope the transcript quality is fine across the entire fleet. That is too slow when the model is the feature.
The deployment loop should look more like this:
| Stage | Audience | Goal |
|---|---|---|
| Canary | 5% of target device tier | catch regressions fast |
| Expansion | 25-50% | confirm latency and quality hold |
| Full rollout | all eligible devices | make the model default |
| Rollback | any failing cohort | recover without app release |
If your voice feature is business-critical, keep one known-good previous version cached on device. Recovery time matters more than elegant packaging.
Where teams usually get this wrong
The recurring mistakes are boring, which is why they keep happening.
Shipping for the best phone in the office
Your internal demo device hides memory pressure and thermal throttling. Production will not.
Ignoring background-download failure
If the improved speech model never finishes downloading on metered or unstable networks, it is not really deployed.
Treating cloud fallback as free
It is not free operationally, financially, or from a privacy-review standpoint.
Measuring only accuracy
Speech can be technically accurate and still feel bad because it arrives too slowly.
A practical build order
If you are adding local transcription to an existing app, the lowest-risk order is:
- Start with one voice surface, usually short dictation.
- Set hard latency and memory budgets before choosing the model.
- Bundle the smallest acceptable baseline model.
- Add staged model delivery and device-tier routing.
- Instrument correction rate and fallback rate from day one.
- Expand to longer-form speech only after the short-form path is stable.
This keeps the first release narrow enough to learn from.
The bottom line
On-device speech-to-text is worth it when the product depends on fast, private, reliable voice input. It is not a universal answer, but it is often the right default for mobile interaction loops.
The technical question is not whether local speech is possible. It is. The real question is whether you can keep the model small enough, the latency low enough, and the rollout disciplined enough that the feature keeps working across the devices your users actually own.
If you want the local runtime without hard-coding every model update into the app, that is the layer Octomil is built to handle. For adjacent deployment problems, see the Core ML iOS deployment guide, the on-device RAG guide, and the MediaPipe deployment guide.