Inspiration
What it does
It takes in voice input from any person speaking into it, and transforms it into an UTAU voicebank for use with UTAU compatible software. This means that an artist can use this software in their DAW to make unique sounding vocals.
How we built it
We use standard browser technologies with webaudio feeding into a flask backend using fastapi and uvicorn. It also uses httpx to communicate with the elevenlabs API, which is used for better voices compared to a local model.
For speech, we use Vosk and Praat to convert the words into samples that can then be used to convert to the format UTAU voicebanks use. elevenlabs can be optionally used here to clear up the voice before using.
Challenges we ran into
Trying to split the voice would end up in a very static and robotic sound, and not actually sounding like the person it was supposed to. This is because vosk can't do things like split syllables and mark edges. We can't do neural synthesis due to not having the hardware required, so we had to stick with this method.
Accomplishments that we're proud of
It can be imported into UTAU and used as a voicebank. The demo "Daisy Bell" also retains the speaker's accent, which is really cool.
What we learned
Audio programming is hard :P
What's next for GenVB
Make the generated voice sound more realistic