Web / XR
Four distinct browser-based hand tracking experiences built with MediaPipe Hands and Three.js, all deployed to VIVERSE. Each experience works on desktop with a webcam. No installation required.
Play synthesizer sounds using finger gestures detected in real-time. Individual finger positions and angles map to notes, chords, and filter parameters — turning your hand into a playable instrument directly in the browser.
3D Tetris played inside a glowing tunnel controlled by hand position. Move and rotate your hand in space to steer falling tetrominoes through a neon corridor rendered in Three.js.
Anti-gravity racing game with hand-lean steering. Tilt your open hand left or right to bank through procedurally generated track sections at escalating speed. The physical gesture mirrors the in-game vehicle lean for an intuitive body-game connection.
WebRTC peer-to-peer multiplayer frisbee. Two players connect via a shareable room code — hand position controls throw angle and power. Real-time hand positions are synchronised between peers with no dedicated game server.
MediaPipe Hands is itself a neural network model running in-browser inference — detecting 21 hand landmarks per frame at up to 30fps without a server. Every interaction in the suite is built on top of this real-time ML inference layer. The AI is not an add-on; it is the foundational input layer.
Claude wrote the gesture recognition layer that maps raw hand landmark coordinates to meaningful game actions. Translating a floating-point skeleton — 21 (x, y, z) points — into discrete, reliable game inputs required careful threshold design, joint angle calculations, and per-experience tuning. The gesture vocabulary for each game was designed iteratively with Claude's help.
Claude architected the WebRTC signalling layer for DISC DUEL's peer-to-peer connection — handling offer/answer exchange, ICE candidate negotiation, and reconnection logic. The multiplayer game state synchronisation protocol was written from scratch with Claude, using delta compression to keep the data channel within the WebRTC bandwidth budget.
The Hand Synth audio engine was built entirely with Claude — a Web Audio API node graph connecting oscillators through filter envelopes, gain stages, and a reverb convolver. Claude mapped specific finger landmark angles to oscillator frequencies and filter cutoff values to produce an expressive, musical response to gesture.
The Three.js scenes across all four experiences — the TETRAVOID tunnel, VOIDRUNNER track, and DISC DUEL arena — were written with Claude handling geometry creation, material configuration, lighting rigs, and the per-frame render loop that integrates hand tracking data.
In-browser ML inference, 3D rendering, real-time audio synthesis, and peer-to-peer networking — all running client-side with no backend beyond a lightweight signalling server.