ESPnet
An open-source speech AI toolkit that runs pretrained models locally and supports training for transcription, translation and voice synthesis.
ESPnet is a speech AI toolkit for researchers and developers who want to run pretrained models on their own hardware or train speech systems. Its recipes cover data preparation through evaluation, with published results for comparison.
- Transcription and translation: Recognize speech with streaming models, Whisper or OWSM, whose multilingual tasks include speech translation, language identification and utterance timestamps.
- Voice synthesis: Generate speech with Tacotron 2, FastSpeech 2, VITS or JETS, with support for multiple speakers and languages, plus separate tasks for singing synthesis and voice conversion.
- Audio cleanup and speakers: Reduce noise, separate overlapping speech, verify speakers and identify who spoke when through diarization.
- Speech model research: Train speech language models and neural audio codecs, or compare systems using a shared recipe structure across tasks.
The open-source Python toolkit uses PyTorch and carries the Apache 2.0 license. It supports Docker and distributed training across multiple GPUs and machines. Pretrained models are available through Hugging Face and the ESPnet model zoo.
Local inference runs through Python or the command line. Hugging Face Spaces provide hosted cloud demos for transcription, translation, synthesis, enhancement and forced alignment. An MCP server lets AI agents call speech capabilities.
