Ollama on your server still costs you
Pointing LiteLLM at a self-hosted model removes the per-token vendor bill and keeps data in your network, which is genuinely valuable. It does not remove the GPU. Capacity still has to be provisioned against peak concurrent users.
On-device changes the slope, not the rate
When the model runs on the user's silicon, adding users adds compute. That inverts the usual relationship between growth and inference cost rather than discounting it.
The hard part is shipping, not calling
Calling a local model is a few lines. Getting a quantized model onto a fleet of mixed devices, holding a rollout when a cohort regresses, and falling back cleanly when the hardware cannot cope is where the work is.