Whisper Speech Recognition

What it is

Whisper is an open-source automatic speech recognition model from OpenAI. It transcribes speech in many languages, translates non-English speech into English, and identifies which language is being spoken. OpenAI released it on September 21, 2022, with later versions large-v2 in December 2022, large-v3 in November 2023, and the faster large-v3-turbo in October 2024.

It became the default transcription engine for countless applications because it was free, MIT licensed, and remarkably robust to accents, background noise, and jargon compared with what came before. Podcast transcription, meeting notes, subtitle generation, and voice interfaces all got dramatically cheaper to build.

How it works

Whisper is an encoder-decoder transformer trained with weak supervision on 680,000 hours of audio paired with transcripts collected from the web, covering 96 languages at launch and around 100 in the current tokenizer, with Cantonese added in version 3. Weak supervision means the training pairs were noisy and imperfect, and the model learned to be robust anyway, which turned out to be the key insight.

Audio is converted to a Mel spectrogram, processed by the encoder through convolutional layers and transformer blocks, and a standard transformer decoder predicts the transcript autoregressively. The elegant part is the unified format. Special tokens switch the model between tasks: transcribe, translate to English, identify the language, predict timestamps, and detect voice activity, all through the same sequence-to-sequence interface. One model, many jobs, no task-specific heads.

Model sizes range from tiny at 39 million parameters through base at 74M, small at 244M, and medium at 769M, up to large at 1.55 billion. The turbo variant trims the decoder from 32 layers to 4 to reach roughly eight times the speed with a modest accuracy cost. Input is 16kHz audio, and the tokenizer follows the GPT-2 byte-pair encoding style.

Key details

The weights are free under the MIT license and run locally through the openai-whisper package, whisper.cpp for efficient CPU inference, or faster-whisper. Hugging Face download figures put it at millions of monthly pulls, which reflects how thoroughly it displaced commercial transcription APIs for many use cases.

For those who prefer an API, OpenAI launched hosted Whisper transcription on March 1, 2023 at $0.006 per minute, based on the large-v2 model. That price undercut most existing transcription services on day one.

Limitations

The most serious documented problem is hallucination. An AP News investigation in October 2024 found hallucinated content in 8 of 10 public-meeting transcripts it examined. A 2024 academic study found hallucinations in 1.4 percent of short audio segments, with 38 percent of those potentially harmful, including false references to race, medications, and violent events. This makes Whisper risky for exactly the high-stakes uses people reach for it: medical notes, legal proceedings, meeting records. A transcript that invents a medication name is worse than no transcript.

Word error rates are higher in under-represented languages, and it does not beat specialist models on narrow benchmarks like LibriSpeech. OpenAI's own model card cautions against transcribing people without consent and against using it in high-risk decision-making. Free, open, and multilingual made Whisper ubiquitous; the hallucinations mean its output still needs a human in the loop, and for critical applications that review step is not optional.

Where it runs

Whisper's MIT license produced a flourishing ecosystem of runtimes. The reference implementation runs on PyTorch, but whisper.cpp ported the model to efficient C++ inference that runs on CPUs and phones, and faster-whisper reimplemented it on the CTranslate2 engine for substantially faster GPU transcription. These variants are what most production deployments actually use; the original package is the reference, not the workhorse.

The applications span from mundane to critical. Podcasters generate show notes and subtitles. Journalists transcribe interviews. Developers build meeting assistants that summarize calls. Accessibility tools provide real-time captioning. Researchers transcribe archival audio in low-resource languages that commercial services never supported. The 680,000 hours of multilingual training data made Whisper the first transcription tool that worked reasonably well for many languages out of the box, which mattered enormously outside the English-speaking world.

The large-v3-turbo release in October 2024 addressed the main practical complaint, speed, by trimming the decoder while keeping accuracy close to the full model. For real-time and near-real-time applications, turbo made the difference between feasible and not. The API at $0.006 per minute remains the path of least resistance for teams that would rather not manage infrastructure. Whisper's story is ultimately about commoditization: transcription went from a paid service to a solved-enough open problem, and the remaining work is about reliability at the edges rather than core accuracy.

The research community keeps building on Whisper's foundation. Distilled variants trade a little accuracy for much faster inference, fine-tuned versions adapt it to specific domains like medical dictation, and alignment techniques improve timestamp precision for subtitling. Competitors have released their own open speech models, but Whisper's combination of license, quality, and ecosystem keeps it the reference implementation. For a model released in 2022, its staying power is unusual in a field that usually discards models within a year, and it speaks to how thoroughly the training-scale bet paid off.