Inspiration

What it does

It takes in voice input from any person speaking into it, and transforms it into an UTAU voicebank for use with UTAU compatible software. This means that an artist can use this software in their DAW to make unique sounding vocals.

How we built it

We use standard browser technologies with webaudio feeding into a flask backend using fastapi and uvicorn. It also uses httpx to communicate with the elevenlabs API, which is used for better voices compared to a local model.

For speech, we use Vosk and Praat to convert the words into samples that can then be used to convert to the format UTAU voicebanks use. elevenlabs can be optionally used here to clear up the voice before using.

Challenges we ran into

Trying to split the voice would end up in a very static and robotic sound, and not actually sounding like the person it was supposed to. This is because vosk can't do things like split syllables and mark edges. We can't do neural synthesis due to not having the hardware required, so we had to stick with this method.

Accomplishments that we're proud of

It can be imported into UTAU and used as a voicebank. The demo "Daisy Bell" also retains the speaker's accent, which is really cool.

What we learned

Audio programming is hard :P

What's next for GenVB

Make the generated voice sound more realistic

Built With

Share this project:

Updates

Submission history