Communication is a fundamental human right, and many individuals need augmentative and alternative communication (AAC) approaches or tools, such as a notebook or electronic tablet with symbols the user can select to create messages, to communicate effectively. While access to speech-language therapies and interventions that promote successful communication outcomes can help some, many existing AAC systems are not designed to support the needs of individuals with motor or visual impairments. By integrating movement sensors with artificial intelligence (AI), researchers at Penn State are finding new ways to further support expressive communication for AAC users.
Led by Krista Wilkinson, distinguished professor of communication sciences and disorders at Penn State, and Syed Billah, assistant professor of information sciences and technology at Penn State, researchers developed and tested a prototype application that interprets body-based communicative movements into speech output using sensors. This initial test included three individuals with motor or visual impairment who served as community advisors to the project. All participants said that the prototype improved their ability to communicate quickly and with people outside their immediate social circle. The theory behind the technology and initial findings were published in Augmentative and Alternative Communication.
Aided and unaided AAC
There are two different types of AAC individuals can use. Aided AAC is typically technology assisted — pointing at pictures or selecting symbols in a specialized app on an electronic tablet. For example, a person might be presented with three different food options via images on their tablet and will point to the choice they want, communicating it to their communication partner. While aided AAC can be understood easily, even by individuals not familiar with the user, it can be physically taxing for those with visual or motor impairments, according to Wilkinson, who is also a Social Science Research Institute cofunded faculty member.
The other form of AAC is unaided, or body-based AAC — facial expressions, shrugs or gestures that are specific to the individual. For example, a person with little to no speech who also has motor impairments, but can move their arms and hands, may raise their hand when shown a specific object signaling, “I want.”
“Unaided AAC is fast, efficient and often less physically taxing for individuals as the movements and gestures are routinely used in their everyday lives,” Wilkinson said. “The downside is these gestures are typically only known by people familiar with the individual and cannot be understood by those they may interact with on a less frequent basis, making it more difficult for AAC users to be independent.”
According to Wilkinson, the goal of developing the prototype was to begin breaking down the wall between aided and unaided AAC, giving individuals the tools they need to open more of the world and communicate freely with those outside their immediate circles.
How AI can help
Current technologies have already begun incorporating AI for natural gesture recognition. However, mainstream technologies are based on large numbers of movements produced by people without disabilities. For individuals with motor or visual disabilities, it is necessary to make the technologies capable of learning idiosyncratic movements — movements and gestures with specific meaning to individuals — and map them to specific commands. The ability of these systems to adjust to individual movement patterns reduces the potential for error and the demands placed on the individual to perform specific pre-assigned movements, according to Wilkinson.
The utility and user experience of AI algorithms, however, is largely unexplored. There are gaps in the understanding of how these algorithms are developed, how they can be adapted for AAC users with diverse disabilities and how they can be seamlessly integrated into existing AAC, according to Wilkinson.
Building the prototype
When developing and testing the prototype, Wilkinson said it was important to her and her team to gather input and feedback from individuals who would be likely to use, and benefit from, this technology.
Emma Elko is one of three “community advisers” the researchers worked with, along with her mother, Lynn Elko — Emma’s primary communication partner. Emma has cortical visual impairment — a visual disability caused by damage to the brain’s visual pathways rather than the eyes themselves — and uses aided AAC to communicate. She also has specific gestures she makes to say, “I want” and “come here.”
Using a sensor worn on Emma’s wrist, the researchers captured her communicative movements. The sensor detected the kinematics — how an object moves, focusing on position and speed — of each movement, allowing it to distinguish between different gestures like an up and down motion versus a side-to-side motion.
Emma was prompted to repeat a movement three times, with Lynn signaling the beginning and end of the movement for the algorithm to capture. The researchers found three repetitions of a gesture gathered sufficient data and minimized user fatigue.
Once the AI algorithm captured the gesture and an associated communicative output was assigned, a connected smartphone application translated the gesture into speech output to be produced any time the sensor recorded the gesture being made. In this way, Emma could communicate directly with someone who was unfamiliar with the specific meaning of her gestures.
“The idea is that we can create a small dictionary of an individual’s most commonly used gestures that have communicative meaning to them,” Wilkinson said. “The great thing about it is the sensor technology allows individuals to be disconnected from their computer or tablet AAC, allowing them to communicate with people more freely.”
Bringing this technology to the people who need it
While the technology is still in the prototype stage, Lynn said she has already seen it make a positive impact on Emma’s life.
“It has been exciting to see a lightweight, unobtrusive sensor detect Emma’s communicative movements and speak them for her, allowing people less familiar with her to understand her instantly,” Lynn said.
While this initial testing proved that this idea works on a conceptual level, questions remain around fine tuning the sensor technology. The next step for Wilkinson and her team is to get this technology in the hands of more people with motor or visual impairment who use AAC for more widespread testing and data collection. The researchers’ goal is to determine not only how well the algorithm does when identifying target motions, but how well it can disregard involuntary movements and how to refine it to distinguish between similar gestures that have different communicative meanings.
“Each individual will have different priorities and different communication needs,” Wilkinson said. “While the sensor is great for capturing movements that are very distinct from one another, we need to develop a way to capture gestures that require more precision. The next step for us is to develop camera-based algorithms that will work in tandem with the sensor, ultimately making this technology accessible for as many people as possible.”
Lynn and Emma are continuing to work with the sensor and integrated app and can see it making a larger impact in Emma’s life as the technology continues to evolve.
“We’re looking forward to skiing season when Emma can wear the sensor to communicate on the slopes instead of only communicating with her paper-based AAC on the chair lift,” Lynn said. “Living without spoken words can bring isolation and a limited social circle. This technology will widen Emma’s world, and I look forward to witnessing the impact of that on her life.”
Additional authors include Syed Billah, assistant professor of information sciences and technology at Penn State; Dawn J. Sowers, assistant professor at Florida State University; Lynn R. Elko, community adviser to the research team; Kevin Williams, community adviser to the research team; Mitchell Case, community adviser to the research team and undergraduate student at Penn State; Sharon Redmon, doctoral candidate at Penn State; and Tanuj Namdeo, software developer.
The Center for Biodevices at Penn State and the Penn State Center for Socially Responsible Artificial Intelligence supported this work.