Every student should be welcomed into the lab with open arms.
You're the scientist. Ollie's the hands.
Ollie is a robot lab partner for students who can't hold the beaker. The student talks; Ollie looks at the bench, plans from moves tested on a real arm, says its plan and waits for a yes. A camera then checks what actually happened, and the student decides what to test next. Every task Ollie gets right becomes a saved skill, so the next time is a single tool call.
Built at HackPrinceton Fall 2026 by Victor Jimenez, Duc Duong, Joseph Roche and Stefanie Nguyen.
One in five US undergraduates reports a disability, and only 8% register one with their college (NCES, PNPI). In a national survey of students with physical disabilities, about half had been given passive roles in science labs and three in four relied on classmates to finish lab tasks (Jeannis et al., 2019). When someone else has the hands, they usually make the calls too. Ollie gives the calls back to the student. More evidence and its limits.
Question → Predict → Direct Ollie → Observe → Explain.
- The student asks in their own words: "Ollie, pour beaker A into flask C." Voice, typing, a switch, eye gaze or voice control all work.
- A vision-capable expert model checks a fresh camera frame and composes a plan from a fixed set of typed, tested actions.
- Ollie states the exact plan. Nothing moves until the student confirms that exact version. Stop always wins.
- A deterministic executor runs the bounded action on the SO-101 arm (or its simulation).
- An independent observer reads fresh pixels and reports what happened, or says "unknown".
- A plan that succeeds is replayed by an evaluator on the normal case, six fault cases, the consent check and the Stop check, then saved as a skill for that exact setup.
In Duc's logged bench run, reusing a saved skill cut request-to-plan from 58.8 s to 0.41 s and model calls for the run from 6 to 0, with consent and the camera check still in place (evidence).
| Part | What it does | Where |
|---|---|---|
| Agent runtime | One session and task loop: Gemini Live conversation, separate expert planner, exact proposals and consent, deterministic executor, independent observer, skill registry and evaluator | agentic/ |
| Student app | Ollie: guided labs, accessible by keyboard, switch, eye gaze and voice control; reports and Ask Ollie | ui/student/ (served at /student; arriving from victor/user-flow-ui) |
| Developer view | The same session with expert/tool traces, observations, skill evaluations and simulator frames | ui/lab/ (served at /developer) |
| Hardware | SO-101 arm on LeRobot, calibration, recorded pours, overhead camera localization, base plate with ArUco markers | firmware/, firmware/fixtures/cup_dock/ |
| Simulation | MuJoCo model of the same arm and bench, lined up with a real recorded pour; Smartivity kit experiments | simulation/ |
| iPhone voice | SwiftUI client sharing the same session | research/joseph/iphone-voice-bridge/ |
![]() |
![]() |
| Erupting Volcano · simulation | Magic Potions · simulation |
Python 3.12 and uv. The simulation backend needs no hardware.
uv venv --python 3.12 .venv-lab
uv pip install --python .venv-lab/bin/python -r agentic/requirements-lab.txt -e 'simulation[dev]'
OPENARMS_EXECUTION=mujoco PORT=8010 .venv-lab/bin/python -m agentic.openarms_labOpen http://127.0.0.1:8010/student/ for Ollie and /developer for the traces. Put GEMINI_API_KEY in an untracked .env for voice, and sign in to the expert CLI you choose (model profiles). The real bench, camera and arm options are in the runbook; every physical motion needs an operator at the arm.
Tests:
PYTHONPATH=agentic:simulation:. .venv-lab/bin/python -m pytest agentic/tests simulation/tests --import-mode=importlib -o asyncio_mode=auto -q- The arm, fixture, camera and recorded pour are real. Motion on the arm always runs under an operator.
- Experiments marked Simulation are MuJoCo models; their reactions are illustrative, not measured chemistry.
- Timings above are from the simulator backend.
- We have not yet tested with students with disabilities or measured learning. That pilot is the next step.
Hugging Face LeRobot and The Robot Studio SO-101, MuJoCo, Google Gemini Live, OpenAI Codex, Anthropic Claude, Devin, FastAPI, OpenCV. Experiments adapted from Smartivity's My First Science Kit. Fonts: Bagel Fat One and Atkinson Hyperlegible Next (SIL Open Font License).
Start with the architecture, the project brief, interfaces, current tasks and the integration ledger. Work in your own worktree and branch; check the ownership table first. Pitch and submission (everything is in pitch/): film plan, team directives, facts and sources, Devpost copy.



