-
-
ITHACA. When you're pinned, call a friend.
-
Odysseus came home, and he wants his house back.
-
Real chopsticks become a weapon: Gemini names them, Stability AI makes the 3D model, and the Twin Palms of Ithaca land in Adam's hands.
-
Left, the friend's iPhone view: the Quartermaster sends live game video with captions. Right, its face, made with Stability AI
-
How it works: the game, the Quartermaster agent, Chert FaceTime, the server hub, Gemini and Stability AI, and MongoDB Atlas.
💡 Inspiration
In Odyssey Book XXII, Odysseus holds the doorway of his hall against the suitors while Telemachus runs to the storeroom for spears and shields. We wanted to play that scene for real. You defend the hall with your own body, and your friend is Telemachus. Their storeroom is their kitchen. 🏛️🍳
We also loved the idea of a game that actually phones someone you know and asks them for help.
⚔️ What it does
Ithaca Calling is a first-person fighting game in the browser, and your webcam is the controller.
- 🖐️ Fight with your body. The webcam tracks your arms and hands. Reach toward a weapon and make a fist to grab it. Swing to slash, thrust to stab, put both fists up to guard, lean to dodge, and open your hand mid-swing to throw. Spears lodge in wooers and pin them to pillars, which triggers a slow-motion killcam.
- 📞 Phone a Friend. When you're pinned, make the phone sign by your ear. The game places a real outbound FaceTime call to your friend's iPhone.
- 🤖 Meet the Quartermaster. On the call, an AI agent shows your friend the fight live, narrates it in an ElevenLabs voice, and passes your voice through so they hear you shouting for a spear.
- 🍳➡️🗡️ Anything can be a weapon. Raise both hands to ask for one. Your friend holds up a real object, and the AI identifies it, crops it, builds a 3D model of that exact object and drops it into your hands. Its physics match what it is: a broom fights like a spear, a pan hits like a mace, a pot lid blocks like a shield, and a water bottle heals you when you drink from it.
- 🏆 Every forged weapon goes into The Armory with a photo of the real object and its kill count. There's also a leaderboard and saved killcams.
You face three waves of wooers, then Antinous behind a table-top shield.
🛠️ How we built it
We were four people building four parts. All of them shared one typed contract (TypeScript + zod) in a pnpm monorepo, and each part had a mock mode, so nobody had to wait for anyone else.
- 🎮 The game (React, three.js, Rapier): a firelit hall with physics ragdolls, stabs that lodge and run through, procedural Late Bronze Age weapons with PBR textures, wounds and blood, and wooers who adapt and fight with shields. The game also draws a separate 640×360 "friend view" at 15 fps for the call.
- 🖐️ Motion tracking (MediaPipe Pose + Hands): pose-guided hand crops track your hands from across the room, and finger curl drives the grip. Gestures cover the phone sign, guard, drink and weapon request. A One-Euro filter adapts its cutoff to how fast the hand moves, so a slow hand stays steady and a fast swing doesn't lag: $$f_c = f_{c,\min} + \beta\,|\dot{x}|$$
- 📞 The Quartermaster (Node, LiveKit, Chert): the agent joins two LiveKit rooms. Chert's FaceTime bridge allows only one audio publisher, so the agent sends the game feed and a single mixed audio track (ElevenLabs TTS, ducked under the player's mic) into the FaceTime room. It also sends the friend's camera and voice back into the game.
- 🔨 The Forge (Fastify, Gemini, Stability AI): the agent takes frames from the friend's camera. Gemini returns the object, a bounding box, a weapon class, a heroic name and a one-liner. We crop the object, key out the friend's hand, generate the mesh with Stable Fast 3D and store the GLB in MongoDB Atlas GridFS. The weapon spawns in your hand and the Quartermaster announces it.
- 🗄️ MongoDB Atlas: players, friends (stored with consent; full phone numbers never reach the browser), sessions, weapons, matches and killcams.
🧗 Challenges we ran into
- 📵 One voice per call. A second audio publisher in Chert's FaceTime room ends the call. That rule is why we built the two-room relay and the single audio mixer.
- ☎️ A shared FaceTime line. Other teams were using the same line, so we built queueing: wait for our turn, retry with the same idempotency key, and take over a call that Chert placed from the queue.
- ⏱️ 5-minute calls, 50 minutes a day. Every real test call used up minutes, so we built a fake hub, a fake phone page and a fake Chert to test the whole loop for free.
- ✋ Hands in the photo. Your friend's hand is on the object they hold up, and the 3D model included it. We paint the hand green and key it out before generating the mesh.
- 🔍 Vision misses. In one test call, Gemini missed a clearly held mug in 5 of 23 frames, so the agent sends another frame before giving up. Call frames also have black side bars, which took a few rounds of crop fixes.
- 🥢 Scale. A chopstick stretched to spear length looked like a plank, so thin sticks now become swords.
- 🎥 Webcam hands from across the room. Standing back from a laptop, your hands are tiny in the frame. Pose-guided crops, fair retries between hands and a hint for dark rooms made grabbing reliable.
- 📺 FaceTime crops our video to about 3:2, so the HUD stays out of the corners.
🏅 Accomplishments that we're proud of
- 📞 A browser game that places a real FaceTime call. It uses an outbound call, agent video out, caller video in, and voice both ways, all through one agent.
- 🍳 A friend holds up a frying pan in their kitchen, and a few seconds later that pan is in your hand, with a photo of the real one in the Armory.
- 🗡️ Combat that feels physical: spears pin wooers to pillars, and you can call a thrown weapon back to your hand with a fist.
- 🛡️ It keeps working when parts fail. If the server is down, the game runs on a mock hub and shows it. If the forge stalls, a stock spear flies in after 16 seconds. If tracking drops, the world slows to 0.2×. If a model won't load, a prefab takes its place.
📚 What we learned
- Real-time media has hard rules (one audio publisher, fixed aspect ratios, call time limits). Designing around them early saved the build.
- A shared typed contract plus mock modes let four people build in parallel.
- With vision models, validated JSON, retries and good fallbacks matter more than a perfect prompt.
- Webcam controls only feel fair with constant feedback: hand indicators, a framing guide and visible grip states.
🚀 What's next for Ithaca Calling
- 👥 Several friends on one call, each raiding their own "storeroom"
- 🌊 More books of the Odyssey: the Cyclops' cave, Scylla and Charybdis
- ⚡ A faster forge with more detailed meshes
- 🏛️ Multiplayer, so Telemachus can fight beside you
Built With
- chert
- claude
- elevenlabs
- facetime
- fastify
- gemini
- gridfs
- livekit
- mediapipe
- mongodb-atlas
- node.js
- rapier
- react
- react-three-fiber
- sharp
- stability-ai
- stable-fast-3d
- three.js
- typescript
- vite
- webrtc
- websockets
- zod
- zustand