AI lets man with paralysis type by just thinking about handwriting

An artificial neural network can interpret signals from the brain of a person who is imagining that they are writing with a pen, and convert them into text. The device converts words accurately at 90 characters per minute, more than twice the previous record for typing with a head- or eye-tracking system.

These trackers allow people to move a mouse cursor and slowly type messages, but Jaimie Henderson at Stanford University in California says they are all-consuming for the operator. “If you’re using eye tracking to work with a computer then your eyes are tied to whatever you’re doing,” he says. “You can’t look up or look around or do something else. Having that additional input channel could be really important.”

To solve this problem, he and his colleagues implanted two small arrays of sensors just under the surface of the brain of a 65-year-old man who has a spinal cord injury that left him paralysed below the neck since 2007. Each sensor array was able to detect signals from around 100 neurons – a fraction of the estimated 100 billion neurons in the human brain.

As the man imagined writing letters and words on a piece of paper, the signals were fed to an artificial neural network. Team member Krishna Shenoy, also at Stanford University, says that the sensors don’t target exact neurons because many thousands or millions may be involved in hand movement, but with the two arrays monitoring around 200 neurons there are enough clues within the data for the artificial neural network to build up a reliable interpreter of brain signals.

Often a neural network is trained with several thousand pieces of example data, which in this case would be a recording of a brain signal while writing a certain letter. That works fine when large data sets already exist or are provided by automated systems, but in this case generating an archive that large wasn’t practical because the man would have had to think about writing thousands of letters. Instead, the team took examples of signals from the man’s brain while writing certain letters and generated additional copies with random noise added to build a synthetic data set.

The model the team created won’t translate to another person because the neural network is trained only on data from one individual, with sensors placed in an unrepeatable location.

Using this system, the man was able to type at 90 characters per minute, approaching the average of people his age when using a smartphone, which is 115 characters per minute. The output had a 94.1 per cent accuracy, which increased to more than 99 per cent when an autocorrect tool was used.

Previous brain-computer interfaces have been able to interpret large signals, such as those for arm movements, but until now haven’t been able to pick up on those for fine, dextrous movements like handwriting.

The team hopes to build on the work to create a speech decoder for use by someone who can no longer speak but is likely to still have the neural pathways to do so.