Spray-on smart skin uses AI to rapidly understand hand tasks

Spray-on smart skin uses AI to rapidly understand hand tasks

A novel, electrically active smart skin can rapidly decipher typing, sign language, even the shape of a familiar object from the movements of a human hand even with limited data.

Image: Spray-on sensory system which consists of printed, bio-compatible nanomesh directly connected with wireless Bluetooth module and further trained through meta-learning

Credit: Kyun Kyu “Richard” Kim, Bao Group, Stanford U.

By Andrew Myers

A new smart skin developed at Stanford University might foretell a day when people type on invisible keyboards, identify objects by touch alone, or allow users to communicate by hand gestures with apps in immersive environments.

In a just-published paper in the journal Nature Electronics the researchers describe a new type of stretchable biocompatible material that gets sprayed on the back of the hand, like suntan spray. Integrated in the mesh is a tiny electrical network that senses as the skin stretches and bends and, using AI, the researchers can interpret myriad daily tasks from hand motions and gestures. The researchers say it could have applications and implications in fields as far-ranging as gaming, sports, telemedicine, and robotics.

So far, several promising methods, such as measuring muscle electrical activities using wrist bands or wearable gloves, have been actively explored to enable various hand tasks and gesturing. However, these devices are bulky as multiple sensory components are needed to pinpoint movements at every single joint. Moreover, a large amount of data needs to be collected for each user and task in order to train the algorithm. These challenges make it difficult to adopt such devices as daily-use electronics.

This work is the first practical approach that is both lean enough in form and adaptable enough in function to work for essentially any user—even with limited data. Current technologies require multiple sensor components to read each joint of the finger, making them bulky. The new device also takes a leaner approach to software to allow faster learning. Such precision could be key in virtual reality applications to convey finely detailed motions for a more realistic experience.

The enabling innovation is a sprayable electrically sensitive mesh network embedded in polyurethane—the same durable-yet-stretchable material used to make skateboard wheels and to protect hardwood floors from damage. The mesh is comprised of millions of nanowires of silver coated with gold that are in contact with each other to form dynamic electrical pathways. This mesh is electrically active, biocompatible, breathable, and stays on unless rubbed in soap and water. It conforms intimately to the wrinkles and folds of each human finger that wears it. Then a light-weight Bluetooth module can be simply attached to the mesh which can wirelessly transfer the signal changes.

“As the fingers bend and twist, the nanowires in the mesh get squeezed together and stretched apart, changing the electrical conductivity of the mesh. These changes can be measured and analyzed to tell us precisely how a hand or a finger or a joint is moving,” explained Zhenan Bao, a K.K. Lee Professor of Chemical Engineering and senior author of the study.

The researchers chose a spray-on approach directly on skin so that the mesh is supported without a substrate. This key engineering decision eliminated unwanted motion artifacts and allowed them to use a single trace of conductive mesh to generate multi-joint information of the fingers.

The eWEAR-TCCI awards for science writing is a project commissioned by the Wearable Electronics Initiative (eWEAR) at Stanford University and made possible by funding through eWEAR industrial affiliates program member Shanda Group and the Tianqiao and Chrissy Chen Institute (TCCI®).generate multi-joint information of the fingers.

Read more on the Stanford Wearable Electronics Initiative website