Self-hosted speech-to-text that scales to zero

Context

Mavio runs its live transcription against a managed speech provider, and that is the right default. Two things a managed provider does not solve: unit cost once volume is real, and workloads that are not permitted to leave your own infrastructure at all.

So we built the alternative and ran both. This case study is about the infrastructure design, because that is where the difficulty actually was.

What was hard

GPUs are expensive when idle and slow when cold. A transcription service has bursty demand — nothing for hours, then a wave when a working day starts in a timezone. Keeping warm GPUs available costs money continuously to serve demand that is intermittent. Scaling to zero saves that money and introduces a cold start of thirty to sixty seconds, which is unacceptable at the front of a live meeting.

Streaming and batch are genuinely different services. Real-time captioning wants the lowest latency the model allows and can tolerate slightly weaker alignment. File transcription wants word-level timestamps, diarisation and alignment, and does not care about latency at all. Building one service that does both badly is the obvious mistake.

Unauthenticated traffic on a GPU endpoint costs real money. A public inference endpoint where authentication happens inside the container means an unauthorised request has already started a GPU before it is rejected.

What we built

Two endpoints, two engines, two platforms.

Streaming runs WhisperLiveKit against a large-v3 model over a WebSocket on a T4 GPU, for live meetings and real-time captions. Batch runs WhisperX large-v2 over HTTP with a job-polling interface, producing word-level timestamps, alignment and speaker labels across a hundred-plus languages.

Both are containerised and run identically on a laptop with Docker and NVIDIA support as they do in the cloud, which is what makes the thing debuggable at all.

Technical decisions and why

Authentication at the gateway, before the container. Requests are checked by the platform's proxy authentication and rejected before any container spins up. An unauthorised request therefore costs nothing. Doing this inside the application would have been a few lines simpler and would have meant every scanner on the internet could start a GPU.

Scale to zero for batch, warm capacity for streaming. Batch workers run with a minimum of zero — a thirty to sixty second cold start is invisible on a job that was always going to be asynchronous. Live streaming cannot absorb that, so it is provisioned differently. Splitting the two services is what made this possible; a combined service would have had to take the worse tradeoff on both sides.

Two model sizes, deliberately. large-v3 for streaming and large-v2 for batch alignment is not an oversight. The alignment tooling is more mature against v2, and batch is where alignment quality matters.

Run it beside the managed provider, not instead of it. Because the speech provider in Mavio sits behind our own interface, we could evaluate this against live traffic without a migration. A self-hosted stack that cannot be compared to the incumbent under real load is a bet, not a decision.

Outcome

The service runs, it costs nothing when idle, and it gives us a credible answer for the workloads that cannot use a third-party API. It has not replaced the managed provider for live meetings, and we would not claim otherwise — the decision is per workload rather than architectural.

Cost and latency figures are deliberately absent. Ours are a function of our traffic shape and our GPU pricing, and reproducing them on your workload is not something we can promise from here.

Stack

• Streaming — WhisperLiveKit, large-v3, WebSocket, NVIDIA T4
• Batch — WhisperX, large-v2, HTTP with job polling; word-level timestamps, alignment, diarisation
• Platforms — Modal and RunPod serverless GPU, flex workers with a minimum of zero
• Auth — platform proxy authentication at the gateway, ahead of container start
• Local — Docker Compose with NVIDIA runtime; identical images to production