A team led by Cheng Zhang, assistant professor of information science, and François Guimbretière, professor of information science, both in the Cornell Ann S. Bowers College of Computing and Information Science, designed the system, named EarIO. It transmits facial movements to a smartphone in real time and is compatible with commercially available headsets for hands-free, cordless video conferencing.

Figure 1: An earable device to detect facial expression.

Figure 1 shows that researchers have developed a wearable earphone device – or “earable” – that bounces sound off the cheeks and transforms the echoes into an avatar of a person’s entire moving face. Facial tracking through acoustic technology can offer better privacy, affordability, comfort and battery life, he said. [1]

Devices that track facial movements using a camera are “large, heavy and energy-hungry, which is a big issue for wearables, importantly, they capture a lot of private information,” said Zhang.

The EarIO works like a ship sending out pulses of sonar. A speaker on each side of the earphone sends acoustic signals to the sides of the face and a microphone picks up the echoes. As wearers talk, smile or raise their eyebrows, the skin moves and stretches, changing the echo profiles.

A deep learning algorithm developed by the researchers uses artificial intelligence to continually process the data and translate the shifting echoes into complete facial expressions.

“Through the power of AI, the algorithm finds complex connections between muscle movement and facial expressions that human eyes cannot identify, we can use that to infer complex information that is harder to capture – the whole front of the face.” said Ke Li. [2]

The team tested the EarIO system on 16 participants, running the algorithm on a regular smartphone. And sure enough, the device was able to reconstruct facial expressions about as well as a regular camera could. Background noise like wind, talking or street noise didn’t interfere with its ability to register faces.

Sonar has a few advantages over using a camera, the team says. Acoustic data requires far less energy and processing power, which also means the device can be smaller and lighter. Cameras can also pick up a huge amount of other personal details a user might not intend to share, so sonar could be more private.

As for what you might use this technology for, it could be a handy way to replicate your physical facial expressions on a digital avatar for games, VR or the metaverse. [3]

References:

1. https://indiaeducationdiary.in/cornell-university-earable-uses-sonar-to-reconstruct-facial-expressions/
2. https://news.cornell.edu/stories/2022/07/earable-uses-sonar-reconstruct-facial-expressions
3. https://newatlas.com/wearables/wearable-sonar-facial-expressions-eario/

Cite this article:

Sri Vasagi K (2022), An Earphone Device Can Track Facial Movements, AnaTechMaz, pp.159