Transcribing audio with AI using Speech Note

One of the joys of writing about technology is the opportunity to cover interesting talks on open‑source and free‑software topics. One of the pains is creating transcriptions of said talks, or continually referring back to a recording, to be able to write about them. Speech Note is an open-source application that uses machine-learning models, running locally, to translate speech to text and take the pain out of transcription. It also handles text to speech, and language translations. While not perfect, its transcriptions are better than one might expect, even when handling jargon, accents, and less-than-perfect audio.

Speech Note is a desktop application, licensed under the MPL-2.0, that is distributed as a Flatpak (via Flathub) and available for x86_64 and aarch64. Packages are also available for Arch Linux and Sailfish OS. It also has optional add-ons that provide support for AMD ("Speech Note AMD") or NVIDIA ("Speech Note NVIDIA") GPUs. It will work without hardware acceleration, but at a substantially slower pace.

Models and engines

Speech Note acts as a front-end for various open-source processing engines and their models. A full list of the processing engines, models, and languages supported by Speech Note is available that also has a table that shows the support each engine has for specific languages. So, for example, if one wants to translate audio of spoken Czech to text, it shows that DeepSpeech / Coqui STT, whisper.cpp, and Vosk have support, but that the april-asr engine does not.

The application has a limited number of actions that can be controlled via the command line, or by other applications using D-Bus, when Speech Note is running. For example, users can use Speech Note to read text aloud from the clipboard (in X11 only) using this command:

    $ flatpak run net.mkiol.SpeechNote --action start-reading-clipboard

Currently, only a few of Speech Note's actions are supported via the command line. Specifically, users can control text to speech, or listening for text to transcribe from the default system audio input. It does not support passing files to transcribe or translate via the command-line interface at this time.

After installing Speech Note, the next step is to add models. The model files can be rather large (in some cases, more than 1GB), so they are not distributed with Speech Note by default. Models are installed by clicking the "Languages" button, and then selecting the language that Speech Note will transcribe from, speak in, or translate from. For example, to transcribe audio in English to text, select English. To translate from Spanish text to English text, select Spanish. Speech Note has support for quite a few languages, even Esperanto, though the number of models available for each language varies. (Esperanto has only one speech-to-text model, and two text-to-speech models.) Sadly, Speech Note has no Klingon support, at least not yet.

Each model is listed by name with an information button next to its download button. Clicking the information button will display some extended information, including the model type, its processing speed, supported hardware acceleration (if any), quality, license, download size, and files that will be downloaded.

Speech to text

Unless one is already familiar with the models, it may be a bit confusing to decide which to download. For English audio to text, I can recommend the WhisperCPP-Distil Large-v3 and FasterWhisper-Distil Large-v3 models. Both are available under the MIT license. These are slow processing models, but they offer high-quality transcriptions with punctuation inserted with a reasonable degree of accuracy. The WhisperCPP model has support for hardware acceleration with NVIDIA GPUs (but not AMD), and the FasterWhisper model has support for CPU hardware acceleration on some Intel CPUs using the OpenVINO toolkit.

Using Speech Note without hardware acceleration meant processing times at least twice as long as the audio being processed. With hardware acceleration using an NVIDIA GPU or Intel CPU with OpenVINO support, Speech Note could chew through a transcription in less time than the actual length of the audio. Of course, even slow transcription is better than having to do it manually. Simply feed Speech Note something to transcribe in the background while off doing other things and come back to a finished transcription.

The fast models are, in my testing at least, not worth the tradeoff of speed versus accuracy. The Mozilla DeepSpeech model was indeed much faster than WhisperCPP's, but its results were much less accurate and included little punctuation. A short example of FasterWhisper versus DeepSpeech is available here. Neither transcription is perfect, and both lack paragraph breaks, but FasterWhisper clearly does a much better job and its text could be cleaned up to be wholly accurate in little time.

After downloading one or more models, users can select the model they wish to use and then go to the File menu to import an audio or video file, or click "Listen" to process speech from the system's sound input. There is no listing of supported file types or codecs, but Speech Note has happily transcribed from MP3, FLAC, and MP4 (video) files so far. While transcribing, Speech Note displays a status message at the bottom of the screen with an estimate of how much of the file has been transcribed. Transcription results are displayed in the Notepad area as they are finalized.

Once the audio file is fully processed, it can be exported to a plain-text file. Speech Note can also produce SRT files, which would be useful for producing subtitles for videos.

While some of the models offer impressively accurate transcription and punctuation, none of the models distinguish speakers. So, for example, if Speech Note is given an MP3 of a podcast with multiple speakers using the WhisperCpp-Distil Large-v3 model, it will likely do a credible job of generating a mostly accurate transcript, but the transcript will not reflect that more than one person was speaking.

Text to speech and translation

Speech Note has a dizzying array of model options for text-to-speech conversions. It will read text in real time from its text window, or it can export text to an audio file. The options are MP3, WAV, Opus, or Ogg Vorbis. The pace it uses to read text can be adjusted from regular playback speed up to 2.0x speed or down to 0.1x speed. The fastest and slowest speeds are not recommended except for comedic effect.

The text-to-speech models that I have tried are clear and crisp, but they are obviously artificial voices that have odd pauses and pronunciations. It is unlikely that someone would listen to a recording from Speech Note and mistake it for a human speaker, but its output is serviceable.

Unfortunately, my fluency in other languages is too weak to effectively judge Speech Note's machine translations. However, using Speech Note to translate English to German (for example) is fairly fast, and putting the German translation into Google Translate returns an English version that is almost identical to the original version.

The most recent feature release, version 4.6.0, came out on August 3. It included a number of new translator models, new text-to-speech voices, and added separate settings for speech-to-text engines in the preferences. Version 4.6.1 included a few bug fixes and new translator models. The project is hosted on GitHub and GitLab; contributors are invited to report issues or submit pull/merge requests on the platform they prefer most.

In my limited experience, Speech Note with the Whisper models is more accurate than services like Otter.ai or Amazon Transcribe, and is much more accurate than YouTube's automatically generated transcripts for videos. It is not perfect, and tends to stumble most on acronyms, names, and jargon, in particular, which is to be expected. However, it does a much better job with those than one might expect.

Speech Note is a useful tool for anyone who needs to convert audio to text (or vice-versa) without depending on a third-party service. It is a particularly appealing option if one wants to convert audio (such as company meeting notes) that should not be shared with third-parties. It is, of course, also much more cost-effective than using subscription services and far faster than typing up a transcription manually.

Note: you can avoid this step in the future by logging into your LWN account.