ISTD 2003

Interactive Systems Technical Design

Lecture #4

Context-aware interaction with mobile devices

ISTD 2003

Context• Lenat et al.: ”There are a dozen mostly-independent

dimensions along which context vary (Absolute time, Type of time, Absolute place, Type of place, Culture, Sophistication/Security, Granularity, Epistemology, Argument-preference, Topic, Justification and Let’s) Conversely, each region of that 12-dimensional space implicitly defines a context”

• Abowd et al.: ”Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computation object”

• Schmidt et al.:” Interrelated conditions where something exists or occurs”

• Simplified: ”Description of the situation”, ”Snapshot of the world”

ISTD 2003

Context Space Dimensions (Lenat et al.)

1. Absolute time : a certain time interval in which events occur

2. Type of time: a non-absolute type of time period, e.g. ”after breakfast”

3. Absolute place: a particular location, where events occur, e.g. ”Oulu”

4. Type of place: a non-absolute type of place, e.g. ”in bed”

5. Culture: linguistic, religious, ethnic, age-group, wealth etc. of typical actors

6. Sophistication/Security: who already knows this, who could learn it, etc.

7. Granularity: phenomena and details which are (not) ignored

8. Epistemology (Modality/Disposition): who wants/believes this context to

be true?

9. Argument-Preference: local rules for resolving pro-con argument disputes

10. Topic/Usage: aspects and applications

11. Justification: are things in this context generally proven, observed, on faith, …

12. Let’s: local bindings of variables etc. that are true in that context

ISTD 2003

Context Categories (Dey)

• Categorization helps to uncover the relevant pieces of context wrt. a given application

• Who’s, Where’s, When’s and What’s of entities• Incoming context is used to encode some

action in the application• Most common contexts

• Location• Time• Identity• Activity

ISTD 2003

Context-Awareness

• Abowd et al.: ”Context-awareness or context aware computing can be defined as the use of context to provide task-relevant information and/or services to a user”

• Pascoe et al.: ”The ability of a device or program to sense, react or adapt to its environment of use”

• Simplified: ”Use of context”

ISTD 2003

Context-Awareness Categories (Dey)

• Attempt to describe how context is used

• Categories• Presentation of information and services to a

user• Automatic execution of a service• Tagging of context information for later retrieval

• Show primary types of features to consider wrt.a particular context-aware application

ISTD 2003

Context (Engineer’s View :)

• Time (absolute, relative, …)• Location (absolute, relative, symbolic, …)• People (profile, presence status, location, …)• Physical environment (objects, weather, …)• Virtual environment (data, …)• Operation/activity (task, mission, objectives,

…)• Device context

• Note!• Absolute vs relative context information• Physical vs symbolic/logical context information

ISTD 2003

Mobile Device Context

• Environment• Illumination, temperature, humidity, noisiness, …• People, machines, networks, services, …

• Location & position• Coordinates, places, indoors, outdoors, …• Near/far wrt. user, tilted, orientation, …

• Activity• Touched, held, moving, shaking, looked at, talked to,

power on/off …

• Note! A mixture of absolute/physical and relative/symbolic/logical context information

ISTD 2003

Context-Awareness and Mobile Devices

• Goal: Improved interaction• Adaptive services and interfaces• Automated context recognition preferred over

manua

• Adaptive interfaces• Modality (how the user uses the device)• Visual appearance• Functions (different functionalities in different

contexts)

• Presentation of context-relevant information• Decrease information overload• Selective context sharing

ISTD 2003

Context-Awareness and Mobile Devices (cont.)

• Challenges for multidiciplinary research• Data (context) acquisition & hardware issues• Context recognition• Distributed computing & protocols to handle

mobility• Human Computer Interaction issues for creating

usable applications• Knowledge engineering issues for representing

and manipulating context information and utilizing background information

ISTD 2003

Context-Awareness for Improved Interaction

Operation sequence Example (proximity sensor)

1. Context recognition IR emitter & receiver• Sensors 1. IR emitter emits light• External information 2. IR receiver measures the amount of

reflection,which depends on the surface type, size of

objectand distance

2. Context representation Proximity representation(a) Direct (sensor) values (a) Distance to the user in cm(b) Inferred states (b) Proximity state (Close, InRange, OutOfRange)

3. Context utilization Adapt UI according to distance(loudspeaker volume, font size, …)

ISTD 2003

Case Study

”Sensitive mobile phones”Microsoft Research, USA

Hinckley K, Pierce J, Sinclair M & Horvitz E (2000) Sensing techniques for mobile interaction. Proc. UIST'00, 91-100. (Best Paper Award)

Hinckley K & Horvitz (2001) Toward more sensitive mobile phones. Proc. UIST’01, 191-192.

http://research.microsoft.com/users/kenh/

ISTD 2003

Overview

• Context of interaction is recognized with simple sensors• Particularly, information on how the device is

being held

• Context is used to improve interaction techniques• Holding the device in certain ways indicates

what action is being performed (or a small set of possible / likely actions)

• Complete automation of particular functions• Speedups and improvements of particular

functions

ISTD 2003

Mobile Device: Casio Cassiopeia E105Augmented with Sensors

ISTD 2003

Sensor #1: Proximity Range Sensor

• IR emitter (light-emitting diode)• IR light modulated at 40Khz

• IR receiver (tuned to sense 40Khz only)• Measures the amount of reflection

• Different readings for different surfaces and sizes of object• Pretty consistent, since we are mostly looking at

the same thing (skin)

• Estimates the distance to an object• Range 5 – 40 cm

ISTD 2003

Sensor #2: Touch Sensors

• Two separate sensors (“plates”)• “Holding” sensor on back and sides• “Bezel” sensor at the left edge of screen

• Sensor ”plates”• Two regions: ”active” area and ”ground”• Conductive paint

• Sensor “plates” detect capacitance changes• Capacity changes when the user’s hand touches

the “plate“ vs. not touching

ISTD 2003

Sensor #3: Tilt Sensor

• Two-axis linear accelerometer (in plane of device)• Gravity provides constant acceleration (1g)• Direction of acceleration vector x,y tilt• Direction of gravity vector can’t be recognized

• Screen up vs. screen down can’t be recognized• Could be solved with a gravity-activated switch

• Can also detect patterns of movement

• Realized with a single chip (ADXL05)• Cheap and relatively easy

ISTD 2003

Sensor Architecture

• PIC micro-controller (Microchip 16C73A)• Constantly reads sensors (~400 samples / sec)• Reports values to PDA via serial port

• Context server (“Broker”) application on PDA• Receives and processes values• Makes information available to other

applications via API• Polled values• Subscribed events

ISTD 2003

Processed Values From Sensors

• Things taken directly from processed sensor data without recognition / inference

• Tilt sensor• TiltAngleLR, TiltAngleFB• HzLR, MagnitudeLR, HzFB, MagnitudeFB

• Dominant frequency and magnitude of movements (separate x,y measures)

• Obtained via FFT over short window

ISTD 2003

Processed Values From Sensors (cont.)

• Proximity sensor• Proximity

• Estimate in cm

• ProximityState• Close (< ~7cm)• InRange (~7-25cm)

• OutOfRange• Duration in

that state

ISTD 2003

Contexts Inferred From Touch Sensors

• Holding (duration)• From back / side touch sensors• Is user holding the device?• If so how long he has been holding it?

• TouchingBezel (duration)• Similar for bezel touch sensor• Not considered to be touching it until

duration>0.2 sec• Thresholds like this is used in a number of places

ISTD 2003

Contexts Inferred From Tilt Sensor

• LookingAt (duration)• Small range of angles appropriate for typical viewing

• Moving (duration)• Any movement and how long since last still period

• Walking (duration)• Detected by repetitive motion in 1.4-3Hz range

• Shaking• Device is being shaken vigorously

ISTD 2003

Contexts Inferred From Tilt Sensor (cont.)

Angle

Y (FB) Tilt

X (LR) Tilt

Walking

Holding at side

Looking at display

Operation sequence: ”entering an elevator, looking at display, holding the device down at side, walking to a meeting”

ISTD 2003

Context-Aware Interaction Techniques

• Automatic power on

• Voice memo recording

• Portrait / Landscape display mode selection

• Tilt Scrolling• LCD contrast compensation

• Display changes for viewing while walking

ISTD 2003

Automatic Power On

• PIC processor is on even when PDA is off

• PDA is powered up when you pick it up to use it• Holding in orientation for use• Specifically

• Holding + LookingAt • In portrait orientation (but not Flat) • For 0.5 seconds

Extremely natural interaction• Pick it up and its ready to use• Avoids most false positives

ISTD 2003

Voice Memo Recording

• Interaction: Pick it up like a cell phone and talk into it”• Holding + Proximity + proper orientation (tilt)

• Audio feedback (critical)• Click when pickup gesture recognized

• Beep to start recording

• Double beep at stop• Stop via loss of preconditions

Again, very natural interaction: pick it up as necessary to do a particular thing and it does it

ISTD 2003

Display Mode Selection

• DisplayOrientation event / data• Flat, Portrait, PortraitUpsideDown,

LandscapeLeft, LandscapeRight• Refresh event when changed

• Dead bands importantto remove jitter• Must cross all the way

through zone to count

• Flat is important • Don’t change orientation

• Interacts/interfereswith scrolling

• Input controls remapped tocurrent orientation

ISTD 2003

Tilt Scrolling

• Must explicitly touch bezel touch area to enable• Not quite as nice, but still pretty easy

• Then tilt up, down, left, right to scroll• Rate controlled (exponential)• Dead band of 4 degrees to prevent jitter

• Application menu and Start bar hidden during scrolling

• LCD contrast compensation• Contrast adjusted according to the viewing angle

ISTD 2003

Tilt Scrolling (cont.)

• Interference: Tilt scrolling interferes with display orientation selection• Hence explicit clutch• “I’m scrolling now” event for explicit disabling

across applications• BUT… what about when you let go

• Can’t tilt back prior to releasing clutch • Not scrolling anymore once released

• Handled by basically disallowing display change in direction of scroll tilt immediately after a scroll

ISTD 2003

Future Work: Display Changesfor Viewing while Walking

• If you hold the device in reading orientation (Holding + LookingAt) and walk (Walking), it will increase font size, etc.• Walking detected by repetitive motion in 1.4–3

Hz range

ISTD 2003

Lessons and Issues

• Sensor fusion• Typically need multiple evidence for a situation

in order to avoid false positives• E.g., “looking at” via both angle and touching instead

of just angle

• Need recognition error tolerant designs• False positive or negative is not a catastrophe • In general recognition errors can easily ruin a

good interaction• Recovery costs can easily destroy benefits

ISTD 2003

Lessons and Issues (cont.)

• Interaction/interference between techniques• Handled by explicit disabling

• Event saying “We’re using that now” • Used to disable other effect• Somewhat ad hoc

• General issue that needs work• Probably needs some sort of general conflict resolution

mechanism• But not clear what that is, or how to do it

ISTD 2003

Lessons and Issues (cont.)

• Big take away point• Knowing context of operation (e.g. how held)

allows space of possible current interactions to be constrained

• Less explicit actions required to invoke an action or provide a input

• E.g., typically avoid button presses or other mode entry

• Can result in simpler and more natural interaction• Best case: Picking it up to do something causes the

something to “just happen”

ISTD 2003

Video

Video