Anish Laha/SynesthesiaSHEET 04 / 12 ›
DWGAL-04

Synesthesia

A microphone-array cap and AR app that let deaf and hard-of-hearing people see sound: what it is, and where it came from. Winner at HackGT.

Role
Team leader
When
September 2026, one hackathon weekend
Where
HackGT, Georgia Tech
Tools
Raspberry Pi PicoESP32BluetoothAnalog microphonesYAMNetSpeech recognitionMeta Quest
Status
Winner, HackGT
Demo video. If the player does not load, watch it on YouTube.

The problem

A lot of everyday information travels only as sound: a knock at the door, a scooter coming up behind you, someone calling your name. If you can't hear it, you miss it. We wanted to show a deaf user two things about every sound that matters: what it is, and which direction it came from.

TRY ITClick anywhere on the page. The cap works out where it came from.

The hardware

The cap: Pico, ESP32 and microphone wiring
The cap: Pico, ESP32 and microphone wiring
Side view, with a microphone at the brim
Side view, with a microphone at the brim
A walk around the cap

How it works

PathRuns onDoes
WherePico, then ESP32 over BluetoothEstimates the bearing from the loudness at each microphone
WhatBackendYAMNet classifies the sound (knock, alarm, scooter, traffic). Speech recognition catches the user's name.
FusionBackendMatches each label to its bearing. The two paths run on separate clocks, so this step lines them up.
DisplayMeta Quest passthroughShows a card with what the sound is, placed in the direction it came from

In the headset

Someone calls your name
A knock at the door
A scooter passing from behind

Decisions

What I'd do next

X 0000 Y 0000 HDG 000 ALT 35000 FT