The story

A team at the University of California, San Francisco has developed a brain computer interface that allows people with severe paralysis to communicate using both speech and upper body gestures at the same time. The system is the first to enable both modes of communication at once. Three patients with amyotrophic lateral sclerosis or brainstem stroke paralysis participated in the study, each using a vocabulary of 10 words and 10 gestures that could be combined in 100 ways.

Researchers implanted a thin strip of sensors called an electrocorticography array onto patients' motor cortices. Computer decoders translated brain signals into commands controlling a full body virtual avatar. One patient achieved 100% accuracy for both speech and gesture decoding in three assessments. The team found that training decoders on simultaneous words and movement worked better than translating speech and gestures independently and then combining them.

ALS and brainstem strokes often cause severe paralysis that interferes with upper body movement and speech. Current eye tracking technologies like the one used by Stephen Hawking are slow, exhausting, and offer limited expression. Dr. Debara Tucci of the NIH said the results give hope that patients with severe paralysis will be able to recapture the holistic nature of human communication. Researchers said testing in more participants with larger sets of words and gestures will be needed before the technology can be used by patients.