Inspiration

Alex Chen has ALS and is slowly losing his speech. At 9 AM he reads at 165 words per minute and is 98% intelligible; by 4 PM it's 118 words per minute and 91%. His speech will keep declining, and the window to preserve it is now.

His care team has suggested voice banking. He hasn't started: "it feels like giving up." Traditional voice banking asks you to sit down and read hundreds of scripted sentences, a chore that turns every session into a reminder of what's coming. So people put it off, and by the time they're ready, the voice they wanted to keep is already changing.

Alex asked for one thing: "I want my kids to remember me as their dad and not as a patient." We wanted a way for him to preserve his voice without ever having to sit down and do it, just by talking to his students, his kids and Jordan the way he already does.

What it does

JawTalk turns everyday conversation into a permanent record of a person's voice and how they speak, then uses that record to give their voice back when they need it.

1. Bank: effortless, in the background. Alex wears a small jaw sensor and talks normally. Whisper runs on the laptop, transcribes what he says with word timestamps, and automatically pairs each word with the matching jaw movement. Every sentence he says becomes:

  • Banked voice: clean recordings (we keep his clearest, morning speech) for building a voice clone.
  • Labelled movement data: a personal dataset of how he forms each word, with no recording sessions.
  • A timeline: everything is timestamped and stored in Tiger Data, so the record grows week after week.

There are no scripts and no sessions. He never has to stop being a teacher, a partner or a dad to do it.

2. Speak: when his voice can't keep up. As speech gets harder, Alex mouths or attempts a word. A model trained on his banked data recognizes it, chains words into a phrase, and speaks it in his own cloned voice. Mouth "help" and "Jordan," pause, and it says "Jordan, I need help." The vocabulary comes from his life: Jordan is his partner, and "help," "water," "yes" and "no" cover the afternoons he spends alone.

3. Preserve: records that outlast the device. The bank belongs to Alex, not to our app. His voice recordings, transcripts and movement data are kept as ordinary audio files and database rows that he can export. When he moves to eye gaze or another communication aid, his banked voice can come with him. And his clinic can use the record to see how his speech and jaw movement have changed between visits.

How we built it

Hardware. An ESP32 reads two Adafruit BNO085 IMUs: one on the chin (articulation) and one behind the ear (a head-motion reference). Each streams gyro, accelerometer and the chip's on-board Game Rotation Vector (a 6-axis quaternion that fuses accel and gyro, with no magnetometer) at 100 Hz over USB serial, all timestamped on the ESP32's clock.

Backend (Python, FastAPI).

  • Reader: parses the serial stream into a ring buffer for each sensor.
  • Word detection: a motion detector watches jaw movement after subtracting head movement. A word is the stretch between the jaw starting to move and a short pause. The noise threshold adapts to each user's resting jitter, captured during a 2 s rest pose.
  • Inference: each word is trimmed to the part with motion, resampled to 100 time steps and classified. Low-confidence predictions and the "nothing" class are rejected, so swallows and head turns don't trigger speech.
  • Phrase builder: after about 1.5 s of stillness, the recognized words are mapped to a natural sentence and spoken through ElevenLabs streaming TTS in the user's cloned voice.
  • Learn mode: local Whisper (faster-whisper) produces word-level timestamps. We align each word with the matching IMU window and save it as a labelled clip automatically.
  • WebSocket pushes live words, confidence and phrase progress to the frontend.

Orientation calibration ("tare"). Each sensor reports its orientation in its own reference frame, which depends on how the hat sits that day. At the start of a session the user holds still for 1.5 s. We average each sensor's quaternion, aligning signs first because q and −q are the same rotation, and compute the jaw-to-head rotation R = R_headᵀ R_jaw. Every later jaw sample is rotated into the head's frame, so jaw-minus-head features mean the same thing every day. Every tare attempt is logged to the database.

ML.

  • Feature sets compared:
    • raw gyro/accel after tare, plus jaw-minus-head gyro (15 channels);
    • a rotation-invariant set (rotation speeds, motion along the main rotation axis, tilt relative to gravity);
    • a jaw-relative-to-head set built only from quaternions: R_rel = R_headᵀ R_jaw, its change since the word began as a rotation vector, and the relative angular velocity. Head motion and the jaw sensor's mounting rotation cancel out of it by construction.
  • Models compared: random forests on per-segment summary statistics, DTW template matching, and a 1D CNN (PyTorch).
  • Evaluation: we hold out an entire recording session, not random clips, so the score reflects a new day and a re-seated hat.
  • Personal calibration: before use, the user records about 5 clips per word, and these are weighted 4× in a quick retrain (about one minute).

Data (Tiger Data / TimescaleDB). The raw IMU stream, every labelled clip, tares, rest poses, detections and spoken phrases are stored in hypertables, giving each user an exportable timeline of their speech.

Frontend (React, TypeScript). Train and Speech tabs, a clip library read from Tiger Data, live predictions, a Tare button and a voice selector.

Challenges we ran into

  • Every day looks different. Our first sessions had no tare, and the jaw sensor sat almost upside down relative to the head sensor. A model trained on one day dropped to near chance (about 28%) on another. Adding tare and orientation-aware features fixed the axis problem.
  • Orientation wasn't the whole story. Even rotation-invariant features fell from about 77% within a session to about 50–60% on a new session. Most of that gap comes from how a word is mouthed that day, posture and strap tightness, which is why per-session calibration matters more than clever features.
  • Hardware dropouts. The head IMU intermittently stopped sending data, in some clips for 0.3–0.8 s. We wrote automatic quality checks (sample ratio, gap length) that exclude broken clips before training.
  • Honest numbers. Random train/test splits made accuracy look great. Holding out whole sessions showed the real problem and changed what we worked on.
  • Look-alike words. "Water" and "Jordan" move the jaw in similar ways and were our most confused pair.

Accomplishments that we're proud of

  • An end-to-end system from jaw movement to a spoken sentence in the user's own voice, running live.
  • Labelling with no effort from the user: Whisper aligns natural speech with IMU data, so the dataset grows from ordinary conversation.
  • About 77% accuracy on 6 classes (5 words plus "nothing", chance 17%) within a session, with "nothing" recognized 100% of the time, so idle movement doesn't trigger speech.
  • A 10-minute calibration step that raises accuracy on a new session from about 58% to about 70%.
  • A quaternion-based jaw-relative-to-head pipeline and an auditable per-session calibration log.

What we learned

  • For wearables, consistent placement plus quick personal calibration beat model complexity. More real sessions helped more than augmentation, CNNs or extra features.
  • Evaluate the way the device will be used. Only held-out-session accuracy predicts how it performs tomorrow.
  • Math alone doesn't fix drift. The quaternion features cancel sensor rotation exactly, but people don't mouth a word the same way twice.
  • Vocabulary design is part of the model. Choosing words that are easy to tell apart is as important as the classifier.

Being honest about ALS

ALS affects the jaw and tongue too. Alex already has tongue wasting and slurring by late afternoon, so jaw signals will weaken over time, and silent speech won't work forever. That's exactly why banking comes first: what's recorded now stays useful after the sensor stops being enough. His voice, his words and a timeline of how his speech changed are preserved for whatever comes next, whether that's eye gaze, a future treatment, or simply his kids hearing him.

What's next for JawTalk

  • One-click export: a "download my bank" button that packages voice recordings, transcripts and the voice clone for any other communication aid or for family.
  • Actions, not just speech: have "help" also text Jordan, so Alex stays "reachable and useful" when he's alone in the afternoon.
  • Privacy by design: banking runs on-device; audio from other people in the room is discarded, and only the user's own speech is kept.
  • Grow the vocabulary using words that are easy to tell apart, add sequence models once there's enough data, and build a smaller, fixed-position mount to test with people living with ALS.
Share this project:

Updates

Submission history