Ultravox

Ultravox is Fixie AI's multimodal speech model that processes human speech directly into an LLM's embedding space, skipping the separate automatic speech recognition stage that traditional voice stacks need. A multimodal projector converts audio into the high-dimensional space of the backbone LLM, which cuts latency to about 150ms time-to-first-token on the 8B variant. Versions have been trained on Llama 3, Mistral, and Gemma backbones, with the default on Llama 3.3 70B and an 8B variant available. It takes audio in and streams text out; speech output still needs a separate TTS step. Fixie reports a 60% transcription accuracy improvement from v0.4 to v0.5 and 42 supported languages, measured on their own eval sets. Weights are open on Hugging Face, and Fixie runs a managed Ultravox Realtime platform for voice-to-voice agents. For developers building voice agents where latency matters more than anything, the collapsed ASR-plus-LLM architecture is the main draw.

Team-reported: 60% transcription accuracy gain v0.4 to v0.5; Big Bench Audio 82.7 (0.5 70B)