EIS: A Wearable Device for Epidermal American Sign
Language RecognitionAuthors: Zijie Zhu, Xuewei Wang, Aakaash Kapoor, Zhichao Zhang,
Tingrui Pan, Zhou Yu
Presentation: Joe Sirrianni
American Sign Language
• American Sign Langaugs (ASL) is a communication language often used by the hearing impaired.
• ASL has its own grammer rules different from English. • There are around 1 million ASL users. • Most words have their own gestures, however, finger spelling (used
for names and technical terms) is also used and contains the familiar 26 letter alphabet.
• These finger spelling gestures involve both finger and hand movement.
Contribution
• The authors present a epidermal-iontroinic sensing (EIS) based wearable devices that wears on finger joints to for 35 fingerspelling ASL recogntions.
Other ASL recognition Device Sensors
Optical Signal Sensing:• Uses video/motion
capture to detect hand movements/recognize letters/words
• Ex: Microsoft Kinect, Leap Motion Devices
• They don’t work well in low lighting situations
Electrical Signal Sensing:• EMG: Uses muscle
contractions to monitor muscle activity
• IMU: Uses accelerometer, gyroscope, and magnetometer to model hand movement.
• Devices achieve good accuracy but incur a lot of noise when used by users.
Mechanical Signal Sensing• Uses Strain Sensors to
capture skin deformation to recognize ASL.
• Sensors can be put in a glove which has very high accuracy.
• Devices tend to not be long-term wearable and are bulky. They are also sensitive to outdoor and body temperature.
Iontronic Capacitive Sensing
• A liquid electrolytes have been used in semiconductors to control the electron transport in a device.
• These electrolytes are ionic-conducting and electronically insulating, which creates an electrical double layer (EDL).
• EDL is model of an interface holding a charge separation between ions and electrons at the nanometer scale distance. Their behavior is analogous with capacitors.
• Using this type of layer, enough capacitive signal can be generated, which can avoid noise from the human body, the air, and other environmental variables.
EIS Sensor
• An EIS Sensor uses iontronic capacitive sensing and is applied to the skin and used to measure pressure.
• The Sensor is imperceptible and wearable.• EIS Sensors can be easily attached to the body and have been used to
measure blood pressure, respiration rates, and muscle activities. • For this application, the sensor areas for their EIS sensor was 4mm x 4
mm.
EIS Sensor
EIS Sensor Device
Complete Device• In addition to the EIS sensors (1 per finger), they used an IMU
(internal measuring unit) to capture movement gestures.• The IMU is a 6-axis IMU comprised of an accelerometer (for
acceleration in 3D space) and a gyroscope (for orientation and angular velocity).
• Outputs from the sensors where sent to a LabVIEW program on a computer via USB.
Data processing Pipeline
Feature Extraction
• Time-Domain Features: 56 total features including:• Max & Min value for each sensor• Mean value for signals. Used to alleviate influence of hand shaking. • Standard Dev of signals to distinguish steady movement• Distance between adjacent sensors on adjacent fingers.
• Frequency Domain Features: • Power Spectral Density (PSD) feature on acceleration and gyroscope data.• Divide signal into separate frequency bands, calulcate mean on each band.
Classifiers
• They tried three types of classifiers:• Support Vector Machine (SVM)• Random Forest (RF)• Neural Network (NN) – Cosine normalization, 4 layers, ReLU activation
function.
Data collection
• Used 8 participants• Each of the 35 gestures were done by every participant 10 times. • Collected 315,000 data points
Results
Results - Classification
• Results on all of the data
Results - Classification
• Results on a per user basis (8-training samples, 2-testing samples)
Results - Classification
• Cross-user recognition (Different users for training and testing)
Results
User Study
My Analysis
Positives• Interesting use of EIS sensors. • It’s good that they tried different
combinations of features. • I like that they used a variety of
different classification techniques and compared them.
Criticisms• The researchers had to assist the
users in putting on the device. It does not seem easy to put on/take off as a glove or camera.
• They did not try any words, which were used in the Mechanical Sensing studies.
• EIS sensors have a short life-span because of wearing (addressed in future work).