DWGAL-04
Synesthesia
A microphone-array cap and AR app that let deaf and hard-of-hearing people see sound: what it is, and where it came from. Winner at HackGT.
Role
Team leader
When
September 2026, one hackathon weekend
Where
HackGT, Georgia Tech
Tools
Raspberry Pi PicoESP32BluetoothAnalog microphonesYAMNetSpeech recognitionMeta Quest
Status
Winner, HackGT
The problem
A lot of everyday information travels only as sound: a knock at the door, a scooter coming up behind you, someone calling your name. If you can't hear it, you miss it. We wanted to show a deaf user two things about every sound that matters: what it is, and which direction it came from.
TRY ITClick anywhere on the page. The cap works out where it came from.
The hardware


- Analog microphones sit around the cap, with a separate rear sound sensor for the user's blind spot.
- A Raspberry Pi Pico samples each microphone, subtracts the background level and turns each reading into a weighted direction vector. The vectors are summed into one bearing.
- An ESP32 sends that bearing over Bluetooth to the AR app.
How it works
| Path | Runs on | Does |
|---|---|---|
| Where | Pico, then ESP32 over Bluetooth | Estimates the bearing from the loudness at each microphone |
| What | Backend | YAMNet classifies the sound (knock, alarm, scooter, traffic). Speech recognition catches the user's name. |
| Fusion | Backend | Matches each label to its bearing. The two paths run on separate clocks, so this step lines them up. |
| Display | Meta Quest passthrough | Shows a card with what the sound is, placed in the direction it came from |
In the headset
Decisions
- Loudness instead of time of arrival. Time-difference-of-arrival is more accurate but needs tightly synchronised sampling. Cheap analog microphones and a Pico give a rough bearing from loudness alone. We traded accuracy for something we could build and debug in a weekend.
- Two paths, then fuse. The "where" and the "what" were built and tested separately, then joined by one fusion step. That let the team work in parallel.
- A sensor just for behind. Sounds from behind are the ones a deaf user can't catch by eye, so that direction got its own sensor.
- A fake-data harness first. Before the hardware was ready, a small web hub and a phone remote sent fake alerts to the display, so the AR side didn't have to wait for the cap.
What I'd do next
- Move to time-difference-of-arrival with synchronised sampling for a tighter bearing.
- Calibrate the array properly, one microphone at a time, against a source at known angles.
- Run it on lightweight AR glasses instead of a headset.