It’s often said that orchestra musicians function on a sixth sense, knowing when to play louder or softer with the rest of the group.
However, one way musicians achieve this is by relying on quick signals from a conductor which are often visual and non-verbal, making it challenging for blind or low-vision musicians to follow along.
“How is someone with vision loss supposed to step into the environment where people are signaling each other with maybe, a raise of an eyebrow or just a little nod?” asked Dalia Sakas, director of music studies at the Filomen M. D’Agostino Greenberg Music School (FMDG) for blind and low-vision musicians in New York City.
“In the case of someone with vision loss, that should be a tactile signal, or something that could be felt,” she said.
Carleton PhD candidate Leon Lu targeted this need for accessible, non-verbal communication through a wearable device strapped to a musician’s ankle. The project, coined the tap-tap device, stemmed from his conversations with blind and low-vision musicians about the barriers to learning music.
“Knowing when to start, knowing when to bow [or] knowing when to stand up and acknowledge people in the crowd, [blind and low-vision musicians] would rely on other folks around them to tap them on their shoulder,” he said.
The tap-tap device is wireless and instantaneously sends and receives vibration signals between two users. It’s activated by a button pressed with a tap of the heel.
“[A] person can say, ‘I want two quick vibrations to tell me to play quieter.’ Or somebody else would say, ‘I want one long vibration to mean take a breath,’” Crispin said. “There’s no fixed-use case.”
After creating the device with Carleton students Aino Eze-Anyanwu and Rodolfo Cossovich, Lu gave FMDG students and teachers free reign of the device for eight weeks of testing.
Sakas said device uses varied depending on the musician’s skill level. Many amateur musicians used the device to fix bad habits, while more proficient musicians used it to elevate their performance.
“[A violinist] and her teacher found an innovative way to use the device to remind her of dynamic markings, when she was slowing down or tempo fluctuations,” Sakas said. “Rather than shouting above the music or interrupting the music, [the teacher] could just signal her with a little code.”
Crispin said there’s only ever been two options for blind musicians to learn music: learning by Braille music notations, or by ear. The tap-tap device provides an alternative.
“There’s never, as far as I know up to this point, been research or products geared toward a middle ground or toward an alternative way to learn,” he said. “In a lot of ways, it’s uncharted territory.”
The device is currently a prototype, but Sakas said she’s excited for how it can be refined and used in large musical settings with multiple musicians in the future.
“The first level of experimentation was there, so now it needs to go to the next level,” Sakas said.
The device even has potential uses outside of music. Lu said he plans to upgrade the device to help deafblind researcher Caroline Pakėnaitė communicate non-verbally with her guide to summit Mount Everest. Lu also said he plans to make the project publicly available, so anyone can build their own tap-tap device.
“There’s a lot that’s communicated nonverbally through visuals, it’s not just music,” Crispin said. “We don’t know [all] the ways it can be used, and that’s a good thing.”
Featured image provided by Leon Lu.
