Touché: Enhancing Touch Interaction on Humans, Screens,

Liquids, and Everyday ObjectsMark Howard

Who?

• Munehiko Sato, Ivan Poupyrev, Chris Harrison

• Disney Research Pittsburgh

• Graduate School of Engineering at the University of Tokyo

• HCI Institute at Carnegie Mellon University

Purpose of the Research

• Demonstrate a variety of applications and interactions enabled by Touché

• Demonstrate immediate feasibility of Touché

• Explore the potential richness of gesture vocabularies supported by the system

• Establish the baseline performance of the recognition engine.

Overview• Touché is a scalable capacitive touch sensing and gesture sensitive

technology- only a single electrode is required for sensing purposes (conductive materials serve as intrinsic electrodes)

• Scalable- may be used for touch/gesture detection on many different objects

• Door knob

• Water

• Table

• Not limited to inanimate objects- People!

• Safe, low power, inexpensive, compact

Overview Cont.’d

• Traditional Touch Detection

• Conductive object is excited by an electrical signal at a fixed frequency

• Sensing circuit monitors return signal and uses changes in this signal to determine touch events (human body as a conductor)

• Swept Frequency Capacitive Sensing (SFCS)

• Return signal is monitored over a range of frequencies

• Objects excited by an electrical signal respond differently at differing frequencies; return signal also changes

• Allows measurement of multiple data points at different frequencies to classify gesture/touch using machine learning

Overview Cont.’d

• SFCS can determine HOW a touch event occurred

• Signal frequency sweeping has been used for wireless comm., but not for touch interaction

• Touché is proven as feasible for immediate real-world applications with this research

Related Work• Touch sensing with respect to Human-Computer Interaction (HCI) is central

to touch interaction (via cameras, pressure sensors, acoustics, etc.)

• Like Humantenna

• Capacitive Touch Sensing- family of sensing techniques

• Based on Capacitive Coupling

• Electrical signal in an electrode forms an oscillating electrical field that is altered as a user’s hand nears it

• Degree of signal change is used for classification (signal phase or signal amplitude)

Limitations of Capacitive Sensing

• Not very expressive- can only detect whether something is touched; maybe some proximity info

• Matrices of electrodes (more data collection) can result in more meaningful results (rough 3D shapes, spatial gestures)

• Resulting increased complexity limits potential applications

Swept Frequency Capacitive Sensing

• In this work, single electrode analyzed at varying frequencies – frequency multiplexing

• Data collected at different frequencies used to construct a capacitive profile

• Human skin has high resistance (~1M Ohm), so weak DC signals would be blocked.

• AC signals pass through the body; the resistive qualities of the human body changes the phase and amplitude of an AC signal

• This AC-human interaction has been used since the 60’s for medical applications

Signal Change in SFCS• How the electrode is touched (amount of contact), body connection to

ground (shoes vs bare feet)

• Strongly dependent on signal frequency

• Different frequencies = different paths through the body (path of least impedence)

• Different anatomical parts of the body become more or less impeding depending on the frequency of the AC signal

• Generates information about: how the object is being touched, how the user is touching the ground, and the human body configuration/individual body properties

Touché Implementation

Implementation Cont.’d

• ARM Cortex-M3 microprocessor on a custom board• Runs at 120MHz, AD5932 wave generator (1KHz to 3.5MHz)

• Signal generator on the board excites an electrode and measure return signal at varying frequencies- generate a capacitive profile• Single sweep takes 33ms; sampling rate of ADC is limiting speed factor

• No measurement of phase changes for now

• Capacitive profile sent to computer via Bluetooth for classification (could classify on single board, computer allowed more rapid development)

• Classification- Support Vector Machine (SVM) on a conventional computer

Sensing Configurations

• Sensor touching one (a and c) and two (b and d) locations of the user’s body

Touché Applications

• Broad applications warrant categorization

• Making everyday objects touch gesture sensitive

• Sensing human bimanual hand gestures

• Sensing human body configuration (i.e. pose)

• Enhancing traditional touch interfaces

• Sensing interaction with unique materials (i.e. liquids)

Objects Sensitive to Touch and Grasp

• Doorknobs!

• Already in usual path, not currently computational

• Already conductive; sensor could be connected with single wire

• Features

• tight grasp = lock, pinch = away message, “grasp password”

Sensing Body Configuration• Sense configuration of entire human body without instrumenting the user

• Tables, chairs, etc. may be conductive in nature; otherwise a single electrode is needed

• Applications in gaming, smart offices, rehabilitation, etc.

• Touché Table

Enhanced Touchscreen Interaction

• Sensing hand posture (hand is a fist vs open palm, single finger vs five)

• “right click” functionality on touch interfaces like traditional computers

• Future applications include 3D drawing/sculpting and music composition/performance

• Improvement upon mobile touchscreen interaction seems viable

Sensing On-Body Gestures• Human body is “input device”- must be minimally invasive instrumentally

• Human body is conductive

• Source is placed near the hands, and the other electrode can be placed many locations

• As users touch their body differently, impedence between the electrodes will vary, resulting in differing capacitive profiles

• Example application is making “shh” gesture to silence a phone

Gesture Sensing in Liquids

• Can determine user interaction with regards to liquids

• Not touchscreens in liquids

• Touching the surface of water vs dipping a finger or hand

• Track indirect interactions (touching water via a conductive object)

Study Participants

• 2 groups of 12 participants

• First group- first four studies

• Second group- liquid study

• Studies were independently run to allow real-world environmental variation

Procedure of Studies• Participants shown gesture set to be performed sequentially

• Training

• 10 gesture instances repeated 3 times for 30 instances per gesture per user

• Useful for data analysis and capturing environmental variability

• Testing

• Participants perform random gesture and the classified gesture is compared to the intended gesture

• Five trials for each gesture

Accuracy• Per-user classification (classifier trained on user specific data)

• Accuracy improves as the gesture set decreases

• Strive for gesture sets with at least 95% accuracy

• General Classifier

• Classifier trained with data from 11 participants; 12th participant was tested

• More difficult with regards to classifier accuracy because of differences in the way participants make gestures

• Per-user classification isn’t always feasible

Results

• Study 1- Doorknob:

• Real-time per-user: 96.7% accuracy (dropping circle gesture => 98.6% accuracy)

• Walk-up: 76.8% accuracy (dropping circle => 95.8% accuracy)

• Study 2- Table to Sense Body Configuration:

• Table with thin copper plating

• Real-time per-user: 92.6% accuracy (dropping two elbows gesture => 96% accuracy)

• Walk-up: 81.2% accuracy (dropping two gestures => 91.6%; dropping 4 => 100%)

Results Cont.’d• Study 3- Improved Touchscreen Interaction

• Mobile device used for testing; two electrodes on front and back of device

• Impedance measured between user’s hand; 5 gestures

• Per-user: 93.3% accuracy (removing two finger pinch => 97.7 % accuracy

• Walk-up: 76.1% (reducing to 3 gestures => 100% accuracy)

• Study 4- On-body Gesture Sensing

• 5 gestures

• Per-user: 84% accuracy (94% accuracy with 4 gestures)

• Walk-up: 52.9% accuracy (87.1% with only 3 gestures)

Results Cont.’d

• Study 5- Touching Liquids

• Best of the 5 studies

• Per-user: 99.8% accuracy

• Walk-up: 99.3% accuracy (removing three finger tips gesture => 99.9% accuracy)

• Post Hoc Tests were conducted to ensure no linkage between accuracy and user weight, height , or gender

Conclusions

• Touché works with 200 samples between 1KHz and 3.5 MHz • Good tradeoff between speed and accuracy

• Decreasing sweep resolution would make classification faster while sacrificing accuracy

• Future work is optimizing Swept Frequency Capacitive Sensing for specific applications (how many samples are needed and what frequency bands should be sampled)

• Difficult to determine which frequencies are ideal for specific interactions, users, applications, materials, etc.

• Mark Weiser- “the most profound technologies are those that disappear”• How do we plan for the future of interacting with invisible computers?

Strengths

• Explored broad range of Touché applications (scalability)

• Immediate feasible applications (phone detecting camera mode by hand placement)

• Opens the door for many future research opportunities

• Decent real-world testing

• Walk-up classification

Critiques

• Discussed decreasing gesture set to improve accuracy

• How do smaller gesture sets limit potential applications?

• Does there seem to be a limit for the amount of gestures/options for classifiers?

• Security Issues- Location/body configuration information

• As these computers become more “invisible,” will data collection become more widespread? What does this mean for personal privacy?

• Discussed ideal sample sizes and frequencies depending on application

• Extrapolate sample size/frequency for potential application realms:

• Why is this extrapolation difficult?