DustinYe/gesture-media-controller

简体中文 · Build guide · Version history · Validation

A working embedded media-controller prototype built with an M5Stack AtomS3R and Unit Gesture / PAJ7620U2. Hand movements become standard BLE HID Consumer Control commands for a connected computer, phone, or tablet. Normal use requires power and a compatible BLE host; a PC relay, cloud service, or project-specific host application is not required.

Published baseline: v24. The main sketch is byte-for-byte identical to the archived v24 sources that were compiled, flashed, and user-tested. The later fist/slap filtering draft is retained separately as an unverified v25 experiment, not an implemented fix.

The front screen is pressed as a physical button. It is distinct from the side Reset button. A single click is decided about 350 ms after release so a double-click does not also pause control. Volume step size and previous-track behavior are determined by the host/player.

• AtomS3R: ESP32-S3, 128 × 128 display, front button, USB-C power/programming.
• Unit Gesture U127: PAJ7620U2 infrared optical sensor.
• Grove cable for I²C and power; sensor address 0x73.
• Existing calibrated mounting: sensor face vertical, toward the user. Start around 8–10 cm away and keep the motion in front of the sensor.

Official references: AtomS3R, Unit Gesture.

flowchart LR
  Hand[Hand movement] --> Sensor[PAJ7620U2]
  Sensor -->|I²C: center, area, presence, flags| Filter[Median filtering]
  Filter --> Gestures[Sweep / Grip / Circle trackers]
  Gestures --> Guard[Event arbitration and connection guards]
  Guard --> HID[BLE HID Consumer Control]
  HID --> Host[Host OS and media player]
  Gestures --> UI[Screen and diagnostic serial output]

The C++/Arduino firmware polls the sensor about every 4 ms and samples trajectories about every 16 ms. These are scheduling intervals, not measured end-to-end latency.

• SweepTracker estimates horizontal motion and suppresses held-hand withdrawal.
• GripTracker combines recent area contraction, centroid changes, endpoint settling, and a one-event latch.
• CircleTracker integrates changes in motion heading, checks circular extent, and counts turns. Native sensor hints assist a clipped final arc.
• MediaHid exposes a consumer-only, two-byte media input report, with encrypted report access and automatic key release.
• Host switching preserves bonds, rejects the previous host during switching, and completes after a different host authenticates and subscribes.

The firmware uses an existing BLE stack; it does not implement a new Bluetooth protocol. It transmits media commands, not audio.

Pinned baseline: M5Stack ESP32 core 3.3.9, board M5AtomS3R.

# Requires Arduino CLI in PATH, or -Cli with its executable path.
./scripts/setup.ps1
./scripts/build.ps1
# Choose the actual USB serial port; COM4 is only an example.
./scripts/upload.ps1 -Port COM4

The scripts also locate the bundled CLI in the standard Windows Arduino IDE installation. See BUILD.md for versions, IDE instructions, and configuration overrides. Uploading replaces the program on the board. No firmware upload or bond clearing is performed by the build script.

Historical hardware checks include Windows media control, Xiaomi phone volume control, iPad media control after HID permission fixes, v23 grasp/withdrawal/slap checks, and v24 switching to another host. They were performed at different development stages; they are not a full platform-by-function certification matrix.

The public sensor fixtures are allowlisted numeric excerpts from those experiments. Run supplementary Python replay checks:

python -m unittest discover -s tests -v

The Python replay is a separate reference implementation, not execution of the C++ binary. One expected-failure case documents the unresolved fist-sweep false positive; it must not be read as a passing fist-rejection feature. See validation details.

Important limitations

• PAJ7620U2 cannot directly count fingers or classify a fist versus an open palm. Gesture recognition is movement-based approximation, without a camera or a trained ML model.
• A clenched-fist lateral movement can still trigger a track command in v24.
• Thresholds depend on the present mounting, distance, motion, and user. No general accuracy percentage or universal latency is claimed.
• BT: READY means authentication and report subscription succeeded; it is not proof that a player acted.
• One host is controlled at a time. Select/connect Gesture Media on the new host after double-clicking.
• Xiaomi Home / Home Assistant integration was an initial goal and is not implemented.

src/MediaGestureController/  v24 firmware
scripts/                    Arduino setup, build and optional upload
tools/                      Python trajectory replay reference
tests/                      Behavioral checks against recorded samples
validation/                 Numeric fixtures, source manifest and evidence summary
history/                    Available v13–v23 source snapshots
experiments/v25/            Unverified slap-filter patch
patches/                    Historical BLE library compatibility fix
docs/                       Build and design explanations

Full flash/NVS backups, credentials, development-machine paths, installers and raw unfiltered logs are deliberately omitted.

This is a student-led interaction-design and hardware-testing project developed with open-source libraries and substantial AI-assisted programming/debugging. The project includes a locally diagnosed BLE library compatibility fix; no upstream merge is claimed. See third-party notices.

No project-wide license has been selected. Dependencies retain their own licenses.