There is an open-source toolkit that wants to replace Whisper for live, on-device speech recognition, and its latency numbers are hard to ignore.
There is an open-source toolkit that wants to replace Whisper for live, on-device speech recognition, and its latency numbers are hard to ignore.
Moonshine Voice is an open-source toolkit built for real-time, on-device voice applications. Unlike Whisper, which was designed primarily for batch transcription of audio files, Moonshine is engineered around streaming speech: it processes audio incrementally, caches what it has already heard, and only computes on the new audio tail as the user keeps talking.
The project bills itself as "Voice Interfaces for Everyone," and it ships as a single library covering speech-to-text, text-to-speech, voice cloning, speaker diarization, intent recognition, and conversational agent APIs. The core is written in C++ using ONNX Runtime, with bindings for Python, Swift, Java, and C++, plus examples for desktop, mobile, and Raspberry Pi. It also offers a Vietnamese speech recognition model with 58 million parameters, reporting an 8.82% word error rate on the FLEURS benchmark.
The key architectural difference is streaming with caching. Whisper uses a fixed 30-second input window and does not cache intermediate state. When a user says a few more words, Whisper reprocesses the entire audio segment from the beginning. For batch transcription this is fine. For live conversation, it creates compounding latency.
Moonshine takes a different approach. Its streaming models support flexible input windows and cache both encoder input and part of the decoder state. When new audio arrives, only the new tail is processed; the cached state is reused. According to the project's own benchmark on a MacBook Pro, Moonshine Tiny Streaming achieved 34 milliseconds of latency, compared to 277 milliseconds for Whisper Tiny. The Medium Streaming model reportedly reached 6.65% WER at 107 ms, compared to Whisper Large V3 at 7.44% WER and 11,286 ms.
Getting started is straightforward: install the Python package with pip install moonshine-voice, then run to start live microphone transcription. The first run downloads the model; subsequent runs work offline.
moonshine-voice mic --language enThe benchmarks are published by the project itself and depend heavily on hardware, model selection, and test data. They should be read as project-reported figures, not independent validation.
More importantly, non-English models are licensed under the Moonshine Community License, which permits only non-commercial use. If you are building a commercial product targeting Vietnamese or any other non-English language, the MIT license does not apply.
The Vietnamese text-to-speech pipeline also has significant room for improvement. The project reports a grapheme-to-phoneme (G2P) character error rate of 79.0% for Vietnamese, compared to a 36.5% reference. That means the TTS system mispronounces Vietnamese words frequently enough that it is not production-ready. The Vietnamese STT model is usable for experimentation, but roughly one in eleven words is wrong.
Moonshine is best suited for developers building live voice interfaces that run on-device: voice notes, hands-free device control, private assistants, or edge applications where sending audio to the cloud is not an option. If you need batch transcription of large audio files on a GPU server, Whisper remains the more established choice.
The takeaway: Moonshine is a serious attempt at making on-device, streaming voice interfaces practical, and its latency advantage is real for live use cases. But read the license carefully, and do not assume the Vietnamese TTS is ready for your users.