# MediaPipe Joy Lab Research Note

This note records the research basis for the TronVibes Joy Lab. The goal is simple: use camera-based motion in ways that feel magical, responsive, accessible, and fun while keeping support free.

## Official Sources Checked

- Google AI Edge Hand Landmarker for Web: https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/web_js
- Google AI Edge Gesture Recognizer for Web: https://ai.google.dev/edge/mediapipe/solutions/vision/gesture_recognizer/web_js
- Google AI Edge Face Landmarker for Web: https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker/web_js
- Google AI Edge Pose Landmarker for Web: https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker/web_js

## What MediaPipe Is Best At For TronVibes

### Hand Landmarks

Best for fine interaction: paddle control, conducting sound, shaping water, popping bubbles, moving chess pieces, selecting UI elements, and expressive two-hand play.

Evidence from the official docs:

- Web and JavaScript support through `@mediapipe/tasks-vision`.
- Image, video, and live camera workflows.
- Up to configurable multiple hands.
- 21 hand landmarks in normalized image coordinates.
- 21 hand world landmarks in 3D coordinates.
- Handedness detection.

TronVibes use:

- Tongg hand paddle.
- HandVibe Orchestra two-hand conducting.
- Living Waters flow shaping.
- Bubble Mercy hand popping.
- Finger Chess pinch-and-release.

### Gesture Recognition

Best for explicit commands: open palm, closed fist, pointing, thumbs up, and other recognized gesture categories. This is useful when a game needs a reliable command layer rather than only continuous motion.

Evidence from the official docs:

- Real-time hand gesture recognition.
- Returns gesture categories, landmarks, world landmarks, and handedness.

TronVibes use:

- Future Joy Lab menus without keyboard.
- Game start/pause/confirm gestures.
- Accessibility commands for players who prefer symbolic gestures over buttons.

### Face And Head Landmarks

Best for head steering, nod-like up/down control, facial expression effects, avatar-like reactions, and gentle no-hands accessibility.

Evidence from the official docs:

- Detects face landmarks and facial expressions in images and videos.
- Can output blendshapes for expressions.
- Can output facial transformation matrices for effects rendering.

TronVibes use:

- Tongg head tracking.
- Light Asteroids head steering.
- Bubble Mercy head current.
- Future calm-breath or expression-responsive experiences.

### Pose Landmarks

Best for bigger movement: posture, shoulders, arms, whole-body rhythm, and future full-room play. This should be optional because it needs more space and more user comfort with camera framing.

Evidence from the official docs:

- Detects body landmarks in images and video.
- Outputs 33 pose landmarks in normalized and world coordinates.
- Can output segmentation masks.

TronVibes use:

- Future movement rituals, rhythm games, stretching games, and full-body light trails.
- Optional large-screen modes.

## Product Rules

- Every camera game must have mouse/touch fallback.
- Camera permission should be user-triggered, never automatic.
- Every camera game should tell users that camera frames stay in the browser and are not sent to TronVibes.
- Games should explain controls briefly and avoid pressure.
- Keep spiritual copy mostly out of games; let the games carry joy, beauty, light, water, rhythm, and play.
- Joy Lab games, Support, and support stay free at launch.
- Do not send camera frames to the TronVibes server unless a future feature clearly explains why and obtains explicit consent.
- Do not market simple hand signals as full ASL or sign-language translation. Silent Access is a respectful command layer until Deaf and sign-language users help validate a real language model.
- Expression cues should adapt pacing and comfort only. They should not claim to diagnose emotion, intent, truthfulness, distress, or disability.

## Current Game Fit

| Game | Best MediaPipe Fit | Reason |
| --- | --- | --- |
| Tongg | Hands + Face Landmarks | Hand paddle and head up/down steering with mouse fallback. |
| HandVibe Orchestra | Hands | Pinch, openness, two-hand position, and expressive conducting. |
| Living Waters | Hands | Palm/wrist motion and pinch strength map naturally to flow control. |
| Light Asteroids | Face + Hands | Head steering plus hand shield. |
| Bubble Mercy | Hands + Face | Hands pop bubbles while head motion guides current. |
| Finger Chess | Hands | Index/thumb pinch maps clearly to piece selection and release. |
| Silent Access | Hands + Face | Browser-only hand signals, expression cues, and tai chi-inspired receive/release motions for users who cannot or do not want to speak. |

## Launch Latency Choice

- Joy Lab prototypes process camera input at 480x270 where practical. The canvas still renders full-screen, but the tracking frame is smaller so MediaPipe has less work per frame.
- The game scripts warm MediaPipe with a tiny offscreen frame before starting live camera play. This helps reduce the first-real-frame pause that can make hand or head control feel like it is not listening.
- A public calibration and comfort guide is available at `/camera-calibration.md`.

## Next Best Improvements

- Move newer games from legacy CDN scripts toward `@mediapipe/tasks-vision` where practical.
- Add a shared in-game camera calibration overlay.
- Add a shared low-latency controller module.
- Add a permissions/privacy prompt before every camera start.
- Add adaptive smoothing sliders for users with tremor, low light, or slower devices.