Human InterfaceHuman Interfacewhere the bits meet the fleshwhere the bits meet the flesh

bits meets eyes and earsbits meets eyes and ears

Tom ZimmermanTom ZimmermanIBM Almaden Research CenterIBM Almaden Research Center

San Jose, CaliforniaSan Jose, Californiatzim@almaden.ibm.comtzim@almaden.ibm.com

October 2007October 2007

Think beyond the box

Research Theme Research Theme

Transducers; Transducers; Converting one thing into anotherConverting one thing into another User Interface, User Experience, First PersonUser Interface, User Experience, First PersonHooking people to gadgetsHooking people to gadgets– Bits Meet the FleshBits Meet the Flesh– Bits Meet Eyes & EarsBits Meet Eyes & Ears

Transducer ExamplesTransducer Examples– Dance into Music (dance synthesizer, suction cup)Dance into Music (dance synthesizer, suction cup)– Hand into Finger Spelling (DataGlove)Hand into Finger Spelling (DataGlove)– Sound into Light (Singing Coach)Sound into Light (Singing Coach)

Violin Home Health Monitoring Contact Microscope

VR Data Glove Personal Area Network Honda Smart Seat

Fantasy & Necessity Are the Mothers of Invention

Discrete EventsDiscrete Events

Converting continuous input into Converting continuous input into discrete events.discrete events.

Continuous input examplesbrightness, sound level, pressure

Discrete output; 0/1, off/on, mouse click

Comparator- Ideal SignalComparator- Ideal Signal

SensorContinuous Input

Quantized DigitizedOutput

Threshold

Sensor +_

Threshold

Digital Output

Comparator- Real World SignalComparator- Real World SignalNoise in input causes output jitterNoise in input causes output jitter

threshold

cross high for on cross low for off

Triggering Discrete Events Triggering Discrete Events with Hysteresiswith Hysteresis

Continuous Input (e.g. brightness,

sound level, pressure)

Quantized/DigitizedOutput

High threshold

Low threshold

Hysteresis, “play”

cross high for on cross low for off

TV Synth InterfaceTV Synth Interface

Hacking Remotes, Mice, Kbd

Circuit Board

4066Quad CMOS Switch

Photo Transistor

Comparatorw/ hysteresis

60 HzFilter

MouseKeyboardIR RemoteSynthesizer

TV

Sensor Signal Processing Switch

Mouse SensorsMouse Sensors• Microphone; Speaker Radiation Pattern• Light; Water Ripple Tank• Ultrasonics; Wavelength, Horn Radiation Pattern• Temperature; Weather Station• Wind speed; See Jet Stream, Car Aerodynamics• Pressure• Tilt• Rpm• Vibration• Pendulum; Simple Harmonic Motion• Acceleration; Seismograph, Earthquakes, Cars/Trucks• Voltage• Current• Resistance• Capacitance• Magnetic field • Electric field

• Left button • Right button• X direction• X increment• Y direction• Y increment

1 to 100 Hz

Wireless Doorbell Wireless Doorbell

Wireless Motion DetectorWireless Motion Detector

Working with Continuous OutputWorking with Continuous Output

PowerGlove Protoype (VPL 1983)PowerGlove Protoype (VPL 1983)

DataGlove DataGlove (Fiber Optic)(Fiber Optic)

Fiber Optic Polhemus Magnetic Fiber Optic Polhemus Magnetic Flex SensorsFlex Sensors Position & OrientationPosition & Orientation

sensor

source

Young Harvill

Head Mounted DisplayHead Mounted Display(NASA 1986)(NASA 1986)

PowerGlove (Mattel 1987)PowerGlove (Mattel 1987)

Computer Data Entry And 1.3 Million SoldManipulation Apparatus And Method $89 Retail

The Spirit Chair (MIT 1994)The Spirit Chair (MIT 1994)

Mediums use spirit cabinets to guide fieldsto contact spirits who make sounds ...

Data In Data Out

Near-Field Communication (MIT/IBM 1995)

Contagious Information Contagious Information (MIT 1995)(MIT 1995)

7/11Fred

Jan Bill

Car Seat Sensor (NEC 1998)Car Seat Sensor (NEC 1998)

Musical InstrumentsMusical Instruments

Human Actions

Musical

InstrumentPitchAmplitudeTimbreDuration

Vocalizer (Breakaway 1981)Vocalizer (Breakaway 1981)

Singing Coach (Exploratorium 1990)Singing Coach (Exploratorium 1990)

RePerformer (Zim/Wantman 1992)RePerformer (Zim/Wantman 1992)

Pen InputPen Input

Pen That Remembers Everything You Write Pen That Remembers Everything You Write (IBM 2000-2002)(IBM 2000-2002)

Premiere UI unobtrusive, low cognitive load, small, portable, quick, cheap

Universal any language

popular input methodclassic form factor

Modelesstext, graphics, doodle comment on document

direct manipulation

BUT hard to index, search,

archive, distribute

Other Pen SystemsOther Pen Systems

PDA; writing on plastic LCD; writing on glass

Tablet; invisible writing CrossPad; writing on 2 lbs.

Toy PensToy Pens

Battery (AAA)

InkCPU and

Flash Memory

CMOS ImagerLens

Normal or dot code paper

Pen Research (IBM 2002)

Ultrasonic Optical

IRDA

Electric Field Pen (IBM 1997)Electric Field Pen (IBM 1997)

Dynamic Signature VerificationDynamic Signature Verification“Sign and Go” (2003)“Sign and Go” (2003)

• Biometric verification using handwriting cadence (timing)• A means to electronically verify customer signatures • Customer signs on a digital signature capture station• Software checks signature against reference signatures and checks for major changes in signature

BiometricsBiometrics

Classification of BiometricsClassification of Biometrics

Physical Feature Physical Feature –Iris, face, finger, hand, DNAIris, face, finger, hand, DNA–Can alter presentationCan alter presentation

Behavior Trait Behavior Trait –Signature, voice, gait, keystrokeSignature, voice, gait, keystroke–Can change behaviorCan change behavior

Identification & VerificationIdentification & VerificationIdentificationIdentification– ““Who am I?”Who am I?”– 1:N matching (difficult, big search)1:N matching (difficult, big search)– Example:Example: 100k customers @ 0.1% error = 100 errors 100k customers @ 0.1% error = 100 errors– UseUse: Discover identity, Watch lists: Discover identity, Watch lists

VerificationVerification– ““Am I who I claim to be?”Am I who I claim to be?”– 1:1 matching (easier)1:1 matching (easier)– Requires identifier (not secure but unique)Requires identifier (not secure but unique)

token (loyalty card) token (loyalty card) memory (phone number)memory (phone number)

– Use:Use: Verify identity Verify identity

Performance MetricsPerformance MetricsFalse Accepts (FA) False Accepts (FA) – likelihood that someone may be falsely accepted likelihood that someone may be falsely accepted bad guy gets inbad guy gets in

False Rejects (FR)False Rejects (FR)– likelihood that a genuine user may be rejected likelihood that a genuine user may be rejected good guy gets rejectedgood guy gets rejected

Lower the FA and FR, the better the system performanceLower the FA and FR, the better the system performanceAcceptance threshold trade-off FA to FRAcceptance threshold trade-off FA to FREqual Error Rate when FA=FREqual Error Rate when FA=FRFailure to enroll, Failure to acquireFailure to enroll, Failure to acquire– Light eyes (iris), thin skin (finger), simple/changing signature, noisy Light eyes (iris), thin skin (finger), simple/changing signature, noisy

environment (voice)environment (voice)– Need backup/alternativeNeed backup/alternative

Example FA=2% FR=1%Example FA=2% FR=1%– 2 out of 100 good customers get rejected2 out of 100 good customers get rejected– 1 out of 100 bad guy attempts get in1 out of 100 bad guy attempts get in– If 1 in 1000 customers are bad guys, 1 forgery per 100,000 transactionsIf 1 in 1000 customers are bad guys, 1 forgery per 100,000 transactions

Biometric ComparisonBiometric Comparison

BiometriBiometricc

EER*EER* AdvantageAdvantage DisadvantageDisadvantage

IrisIris < 0.1%< 0.1% Most accurate, no contactMost accurate, no contact Critical alignment, Critical alignment, expensive reader expensive reader

FingerFinger 0.1-10%0.1-10% Wide use, standard Wide use, standard databases, competitive databases, competitive market, legal precedencemarket, legal precedence

Special reader, criminal Special reader, criminal association, gummy association, gummy attack, modest cost readerattack, modest cost reader

SignatureSignature 1.7-3%1.7-3% Uses existing POS signature Uses existing POS signature capture hardware, social capture hardware, social acceptance, signature dual acceptance, signature dual purpose (agreement & purpose (agreement & biometric)biometric)

Behavior (requires Behavior (requires cooperation), name cooperation), name changes, ergonomicschanges, ergonomics

VoiceVoice 2-10%2-10% Use with cell phonesUse with cell phones Sensitive to noisy Sensitive to noisy environment, requires environment, requires talkingtalking* Equal Error Rate estimates for relative comparison only. Actual results are highly

dependent on system design, equipment, user population and experience.

Tangible Programming Tangible Programming

Logo CardsLogo CardsBar code

Instruction

Usage

Definition

Example code

35mm slide holder

Students Programming With LOGO Playing Cards

Queuing and Scanning Programs into Computer

Programming Robots with Programming Robots with Playing CardsPlaying Cards

The Robosapian™ robot comes with a remote control to program robot movements.

The remote control is replaced with a deck of instruction cards and bar code reader.

Students choreograph movements by arranging the cards, then program the robot by scanning the cards with the bar code reader.

TelepresenceTelepresence

When you need to be there … but you don’t want to travel there

Dummy Head Electronics (IBM 2003)Dummy Head Electronics (IBM 2003)

Frame and Handle

VGA Camera

MegaPixel Camera

Tilt Gimble

Yaw Motor

Speaker Phone

Battery

Tiny PC w/WLAN

Ethernet AC PowerAnalog Phone Line

Ethernet

DIGITAL HEAD BASE STATION (x-ray view) (100 m range to head)

Digital Head ComponentsDigital Head Components

Techno HeadTechno Head

QVGA ScreenBinaural Mics

20 W SpeakerSeat Clamp

Digital Camera

Head AppearancesHead Appearances

Standards Committee Symposium Trade Show Booth

Breakfast Keynote Board Meeting Talk Conference Panel

Shopping BuddyShopping Buddy

Location-aware shopping cart provide real-time “you are here” navigation map and location-based content-Location-aware shopping cart provide real-time “you are here” navigation map and location-based content-driven offers such as coupons and recipes. driven offers such as coupons and recipes.

Infrared Receiver

Infrared Beacon

Wireless (Bluetooth)Barcode Scanner

WLAN

Museum InterfacesMuseum Interfaces

Sizzling JelloSizzling Jello

Bubble MachineBubble Machine

Bubble Strobe MachineBubble Strobe Machine

Questions???Questions???

Badge Computer (IBM 1998)Badge Computer (IBM 1998)

NZ Airport Travel Card (IBM 1998)NZ Airport Travel Card (IBM 1998)

PDA running XML Browser, linked by 802.11b WLAN

Keypad overlayTouch Pad

Camera with 270 deg swivel Microphones/Structured Light

FM antenna along perimeter Speakers

802.11 WLAN antennas running along each sideColor Screen (no touch)

Multi-Function PDA (IBM 2001)Multi-Function PDA (IBM 2001)

BioPDA Wireless Biometric TerminalBioPDA Wireless Biometric Terminal

Iris CameraFace & Document Camera

InfraRed Lights

802.11b WLAN

Intel XScale Processor

Numeric Keypad

Field Replaceable Biometric Module

Symbol 8000 Handheld Terminal

Fingerprint Reader

Using the Biometric Using the Biometric Wireless TerminalWireless Terminal

Capturing Face and Iris

Capturing Fingerprint

Lining up Iris Camera

lines up and frames Face Camera