Live captions, translated subtitles, content moderation, and voice-agent hosts have moved from nice-to-have to expected features in live streaming. All of them start with the same primitive: turning the host’s speech into text while the stream is still live.
A general speech-to-text API is not automatically a good fit for this job. Batch transcription, where you upload a finished file and wait for a transcript, solves a different problem than streaming transcription, where audio goes in continuously and partial text comes back within hundreds of milliseconds. Most vendor comparisons benchmark model latency and word error rate in isolation. Almost none explain how the transcript gets back into the media pipeline and stays in sync with what viewers see and hear.
This guide covers what actually makes a speech-to-text API real-time, the dimensions that matter for live streaming, how the leading streaming APIs compare, and how to wire transcription into a live stream without adding a visible lag between the speaker and the caption.
What makes a speech-to-text API “real-time”
A real-time (streaming) speech-to-text API accepts a continuous audio stream over a persistent connection, usually WebSocket or gRPC, and returns two kinds of results:
- Partial (interim) transcripts — tentative text for speech that is still in progress, used to render captions as the person talks.
- Final transcripts — committed text for a completed utterance, used for archiving, moderation decisions, and translation.
The distinction matters because vendors report latency differently. Some quote time to first partial token, which can be under 150 ms. Others quote time from end-of-speech to a finalized sentence, which includes voice-activity detection and segmentation and is naturally several hundred milliseconds higher. A vendor that streams partials at 150 ms and one that finalizes at 600 ms may both advertise “real-time” while describing different points in the same pipeline.
For live streaming, the number that matters is end-to-end: the time from a word being spoken to its caption appearing on a viewer’s screen. That budget includes audio capture and encoding, upstream transport, ASR processing, downstream delivery to the player, and caption rendering — not just the model.
How to evaluate real-time STT for live streaming
1. End-to-end latency, not model latency
A practical target for live captions is to keep the spoken-word-to-on-screen-caption delay under roughly 300 ms so text tracks the speaker without feeling like a dub. Every additional network hop between your media server and the ASR service eats into that budget. Two architectures are common:
- Client relay. The publisher’s client forks audio to the STT vendor over a separate WebSocket. Simple to prototype, but it doubles upstream bandwidth, depends on the broadcaster’s network, and leaves sync between the transcript and the published stream entirely to you.
- Server-side stream pull. The ASR service subscribes to the audio already circulating inside your RTC or streaming network. No extra client upload, and transcript results can be timestamped against the same stream the player receives, which makes lip-sync-accurate caption injection feasible.
2. Accuracy under real stream conditions
Word error rate measured on clean, read-aloud English tells you little about a live room. Streams contain background music, gift sound effects, game audio, far-field voices, 8 kHz voices on mobile, and overlapping speakers. Independent testing of commercial STT services consistently shows large gaps between clean benchmark audio and real-world recordings, as well as materially higher error rates for accented and non-native speech. Evaluate on your own stream recordings, score streaming endpoints separately from batch endpoints, and check hallucination and punctuation behavior during music and silence.
3. Language coverage and code-switching
Count the languages you need today, but also check how the model behaves when a speaker switches languages mid-sentence. Leading streaming models differ substantially in native code-switching support, dialect coverage, and whether translation is bundled or billed separately.
4. Concurrency and scale
An interactive room with six speakers and a one-to-many broadcast with 100,000 viewers are different workloads. The transcription cost scales with speaking streams, not viewers, so multi-host panels and voice-chat rooms multiply per-minute charges quickly. Confirm concurrent-stream limits, per-connection quotas, and how per-second or per-minute rounding works before launch.
5. Caption injection and sync
This is the dimension most comparisons skip. The STT API returns text; your product still has to deliver it to every viewer, order it correctly against packet loss, and render it in time with the audio. Ask whether the platform provides timestamped, sequenced transcript messages, a ready caption-rendering component, and a path for translated subtitles alongside the original.
Leading real-time speech-to-text APIs compared
The APIs below are the streaming STT providers most frequently cited for live captions and voice AI. They are all hosted model APIs: each gives you a WebSocket to send audio and receive transcripts, and none ships the audio transport or the viewer-facing media layer. Pricing and language counts change frequently, so treat the table as an architecture-level comparison and verify current numbers on each vendor’s pricing page.
| Provider | Streaming interface | Latency positioning | Language breadth | Deployment | Media pipeline included? |
|---|---|---|---|---|---|
| Deepgram (Nova) | WebSocket streaming, interim + final results | Advertised around 300 ms end-to-end | Broad multilingual support with code-switching | SaaS plus self-hosted option | No — you bring capture, transport, and caption rendering |
| AssemblyAI (Realtime) | WebSocket streaming with latency modes | Latency modes trade speed against accuracy; interim results stream continuously | Multilingual with code-switching; widest coverage on fallback models | SaaS only | No — transcripts return to your backend, integration is yours |
| ElevenLabs (Scribe) | Realtime WebSocket STT | Advertised sub-150 ms live transcription latency | 90+ languages | SaaS only | No — strongest fit when paired with ElevenLabs TTS in an agent stack |
| Gladia | WebSocket streaming API | Competes on low partial-result latency | 100+ languages claimed, including long-tail coverage | SaaS only | No — model API with bundled add-ons, no media layer |
| Google Cloud Speech-to-Text (Chirp) | gRPC streaming | Typically higher latency than the latency-focused specialists | Among the broadest catalogs (100+ languages/dialects) | Google Cloud regions, data residency and CMEK | No — fits Google Cloud media pipelines you build yourself |
The pattern is consistent: these are excellent transcription engines, but for a streaming product they are one component. Your team still owns audio capture from the broadcaster, resilient transport to the API, transcript fan-out to viewers, synchronization, the caption UI, and the fallback behavior when the STT connection drops mid-broadcast.
How real-time captions get into a live stream
At an architecture level, wiring STT into live video involves four stages:
- Capture. The host’s client publishes an audio track into your RTC or streaming network, ideally after echo cancellation, gain control, and noise suppression — all of which materially improve downstream ASR accuracy.
- Stream to ASR. Audio frames are sent to the transcription service. In a server-side pull design, the ASR service subscribes to the existing room stream, so the publisher opens no extra connection.
- Receive partial and final transcripts. Interim results drive the live caption; final results are committed for recording, search, moderation, and translation. Messages need sequence numbers, because real networks deliver them out of order, and timestamps so late packets can be discarded rather than displayed late.
- Inject captions in sync. The transcript is delivered to viewers through the same real-time channel that carries the media and rendered by a caption component tied to the stream’s timeline. Keeping text on the same transport path as audio is what keeps the caption under the 300 ms budget; routing it through an unrelated HTTP fan-out typically does not.
If captions must survive in replays and VOD, the finalized transcript should also be passed to your recording pipeline so subtitles can be muxed or stored sidecar with the recorded stream, rather than reconstructed afterward.
The integrated path: STT inside the RTC layer
If your streaming product already runs on a real-time communication platform, an alternative to bolting a standalone STT API onto the side is transcription that operates inside the media network. ZEGOCLOUD Cloud Real-Time ASR is built this way for voice calls, live streaming, and online meetings.
The flow is server-driven. After a client publishes audio into a ZEGOCLOUD RTC room, your business server calls StartRealtimeASRTask for a whole room or selected streams. The ASR service pulls the audio directly from the RTC room — no second upload from the broadcaster — and returns ASRResult callbacks to your server. With SubtitleType set to 1, 2, or 3, recognition results, translation results, or both are also delivered into the RTC room as sequenced signaling messages, which the client subtitle component renders typewriter-style, ordered by SeqId with forward error correction. StopRealtimeASRTask ends the task, and streams can be added or removed while it runs.
Other documented details worth knowing before you design around it:
- Latency is documented at around 600 ms from the user finishing speaking to the ASR result. Note this is an utterance-finalization figure that includes speech segmentation (the silence-segmentation interval defaults to 500 ms and is configurable from 200–2,000 ms), so it is not directly comparable to a competitor’s first-partial-token number — benchmark both partials and finals on your own streams.
- Multiple ASR engines are selectable (including Tencent, Alibaba Bailian Paraformer/Gummy, and Microsoft), covering 20+ languages with notably deep Chinese dialect coverage, plus optional translation through dedicated translation models.
- Audio is pre-conditioned for recognition: the service applies noise reduction and AI echo cancellation tuned for music, gift effects, BGM, and cross-talk in live rooms.
- It requires activation through ZEGOCLOUD support before use, and a task auto-stops after a configurable 120 seconds with no real user in the room.
- Finalized transcripts pair naturally with cloud recording for subtitled replays, and the same real-time pipeline supports voice agents built on ZEGOCLOUD’s AI Agent stack; for the output side, see the notes on streaming text-to-speech.
The trade-off is the standard one for integrated versus best-of-breed: you give up free choice among every specialist model in exchange for one vendor owning the audio path, transcription pull, transcript delivery, and caption rendering, which removes the hops and the sync engineering that usually consume the latency budget.
Key takeaways
- “Real-time STT” means streamed audio with partial transcripts; compare first-partial latency and finalization latency separately.
- Raw model latency is only part of the caption delay — capture, transport, fan-out, and rendering decide the viewer experience.
- Benchmark on your own streams: music, accents, 8 kHz mobile audio, and code-switching matter far more than clean-audio WER claims.
- Standalone STT leaders (Deepgram, AssemblyAI, ElevenLabs, Gladia, Google) give you a transcript endpoint; integrating it into a live stream at scale remains your engineering.
- An RTC-native ASR service that pulls streams server-side and delivers sequenced captions over the media channel removes the extra hops and the sync work, at the cost of model choice.
Frequently asked questions
What makes a speech-to-text API “real-time”?
It streams audio in and returns partial transcripts within a few hundred milliseconds, rather than processing a finished file. This is required for live captions, subtitles, and voice agents where waiting for a full recording is not acceptable.
What latency should I target for live streaming captions?
Aim for under 300 ms from spoken word to on-screen caption so text tracks the speaker. That budget must also cover network transport and caption rendering, not just the ASR model, and server-side stream pull helps by avoiding extra upload and fan-out hops.
Which real-time STT APIs are most cited in 2026?
AssemblyAI and Deepgram are the most frequently referenced, with ElevenLabs, Gladia, and Google Cloud as common alternatives. The best choice depends on your language coverage, concurrency, pricing tolerance for add-ons, and — decisively for streaming — how the transcript integrates with your media pipeline.
How do captions get added to a live video stream?
Audio is streamed to the STT API, partial transcripts come back over a persistent connection, and the text is injected back toward viewers in sync with the media. Running transcription inside the same RTC platform that carries the audio avoids extra hops and gives you sequenced, timestamped messages that a caption component can render reliably.
Conclusion
The best real-time speech-to-text API for live streaming is the one whose results actually reach your viewers on time. Specialist STT providers lead on model-level latency and language breadth, but every one of them leaves the media integration — capture, transport, sync, caption rendering, and replay — in your hands. If you are building on an RTC platform, a transcription service that pulls the room’s existing streams and delivers sequenced captions over the same media channel is often the faster path to a caption experience that stays under the viewer’s latency threshold.
Ready to add captions to your live stream? Explore the Cloud Real-Time ASR documentation, or talk to our team to activate real-time speech-to-text for your RTC rooms and start streaming captions to viewers in sync with the audio.
Let’s Build APP Together
Start building with real-time video, voice & chat SDK for apps today!
