1

User Interfaces for Mobile and User Interfaces for Mobile and Wearable ComputingWearable Computing

Augmented Reality 2Augmented Reality 2

COMS E6176Prof. FeinerColumbia UniversityApril 22, 2004

2

The Ultimate DisplayThe Ultimate Display

“The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming, such a display could literally be the Wonderland into which Alice walked.”

— I. Sutherland, The Ultimate Display, Proc. IFIP 65, 506–508, 1965

3

Sutherland’s SeeSutherland’s See--Through Through HeadHead--Worn Display Worn Display Late 60’sLate 60’s

4

SizeWeightDurabilityRun-timeComfortAppearance

Research Challenges:Research Challenges:Ubiquity/Ubiquity/WearabilityWearability

Columbia “Touring Machine”

5

Research Challenges:Research Challenges:DisplaysDisplays

Minolta “forgettable display”

MicroOptical eyeglass display

Eyeglass displays

[www.microopticalcorp.com]

6

Technology:Technology:Spatial TrackersSpatial Trackers

7

Spatial Tracking BasicsSpatial Tracking Basics

Usually a sensor determines position/orientation relative to a signal it detects from a sourceSource or sensor is fixed in a known position/orientationTechnologies:

electromagnetic, ultrasonic, mechanical, video/optical, earth-relative, inertial, satellite, hybridcoarse approaches: radio, RFID, IR

8

Electromagnetic (EM) TrackingElectromagnetic (EM) TrackingThree orthogonal source coils emit a radio frequency electromagnetic signal in sequenceThree sensor coils simultaneously measure the signal from each of the three source coilsNine measurements provide position and orientation

To host computer

Sensor

SourceE.g., Polhemus [www.polhemus.com],Ascension [www.ascension-tech.com]

9

EM Tracking Advantages and EM Tracking Advantages and DisadvantagesDisadvantages

Advantagesrelatively unencumberingdo not need line-of-sight from source to sensor

Disadvantageslarge error in tracking, particularly dynamic errorshort usable range (< 20’ for long-range xmtr)highly sensitive to electromagnetic radiation and metal in environment

10

Ultrasonic TrackingUltrasonic TrackingAn emitter emits an ultrasonic burst, and three receivers pick up the burstThe time between emission and reception by each receiver gives emitter–receiver distanceCan reconstruct 3D position of emitter if positions of receivers are knownThree emitters allow full orientation measurement Sensor (receiver)

Source (emitter)E.g., Logitech “Red Baron”

11

Ultrasonic Tracking Advantages Ultrasonic Tracking Advantages and Disadvantagesand Disadvantages

Advantageslow costeasy to build

Disadvantagesrequires clear line-of-sight from emitters to receiversvery noisy, high dynamic distortionvery short rangeSimple designs are susceptible to echos and environmental noise (e.g., jangling keys)

12

Mechanical Linkage TrackingMechanical Linkage TrackingTypically a jointed structure that measures the angle at each jointUsing known angles and lengths of arms, compute relative positions/orientations of end

Base

trans(L1)

trans(L2)trans(L3)

rot(J1)

rot(J2)

rot(J3)

13

Mechanical Linkage Tracking Mechanical Linkage Tracking Advantages and DisadvantagesAdvantages and Disadvantages

Advantagesvery high quality tracking with low dynamic distortioncan support heavy displays (e.g., CRTs for head tracking)mature off-the-shelf technology

Disadvantagesrestricted motion

14

Video/Optical TrackingVideo/Optical Tracking

Video cameras monitor environmenttrack active LEDs or passive retroreflectivemarkers

“Outside-in”: tracked user/object wears targets“Inside-out”: tracked user/object wears cameras

E.g., ARToolkit [www.hitl.washington.edu/shared_space], Northern Digital [www.ndigital.com], Phoenix Technologies [www.ptiphoenix.com], 3rdTech [www.3rdtech.com]

15

Video/Optical Tracking Video/Optical Tracking Advantages and DisadvantagesAdvantages and Disadvantages

Advantagesvery high qualitycan use head-worn cameras for video see-through displaypotentially supports object recognition

Disadvantagesrequires clear line-of-sightmulti-camera configurations can be difficult to calibrate

16

““SourcelessSourceless” Tracking” Tracking

17

EarthEarth--Relative TrackingRelative Tracking

magnetometer determines yaw from earth’s magnetic fieldinclinometers determine roll and pitch from earth’s gravitational field

E.g., Precision Navigation [www.precisionnav.com]

18

EarthEarth--Relative Tracking Relative Tracking Advantages and DisadvantagesAdvantages and Disadvantages

Advantagesno source needed (besides the earth!)lightweight (< 2 oz.)inexpensive

Disadvantagessusceptible to magnetic field anomalies (large metal objects) and changes over timeangle limits caused by inclinometer and/or magnetometer gimbals

19

Inertial TrackingInertial TrackingLinear accelerometersAngular rate gyroscopesAdvantages

unlimited range

Disadvantagescurrently very difficult/expensive to make accurate systemserror accumulates over time

20

Satellite Position TrackingSatellite Position TrackingConstellations of US (GPS)/Russian (GLONASS) satellitesRadio receiver(s) determine distance from multiple satellites (ca. 10m accuracy)Differential GPS uses correction signal broadcast from receiver at preciselyknown location (ca. submeter accuracy)RTK carrier-phase measurement GPS(ca. cm accuracy)

E.g., Trimble [www.trimble.com], Ashtech [www.ashtech.com]

21

Satellite Position Tracking Satellite Position Tracking Advantages and DisadvantagesAdvantages and Disadvantages

AdvantagesVery long range

Disadvantagesmust be within line-of-sight of satellites (not indoors or in “urban canyons”)poor altitude trackinggovernments own satellites (encryption, “selective availability”, politics)low update rate

22

Hybrid TrackersHybrid Trackers

Combine different technologiesExamples

Inertial tracker (fast, but drifts) andultrasonic tracker (–slower, +ground truth, →position and orientation): InterSense IS 900magnetometer (–slower, +ground truth, +sourceless, → orientation): InterSense IS 300www.isense.com

Orientation tracker, GPS, and dead reckoning: Point Research Systems

23

Research Challenges:Research Challenges:UI DesignUI Design

What should we see, hear, feel, . . .?How should we interact with it?

24

Windows on the World Windows on the World 19931993

X11 serverTrack head/body/hand and selected objectsUser “wears” virtual information surroundWindows fixed to display, surround, worldHypermedia links between physical and virtual objects

25

Architectural Anatomy Architectural Anatomy 19951995Webster, Krueger, MacIntyre, & KellerWebster, Krueger, MacIntyre, & Keller

User examines building support systems in situ3D graphics reveal hidden physical elements2D graphics overlay output of analysis tools

26

Field Service Field Service 19961996

“Crossbox” telecom junction panel connects clients to main switch

27

Field Service Field Service 19961996

3D graphics identify groups of connection posts2D windows describe connection status2D windows can be pulled in/out to avoid clutter

28

Field Service: Field Service: Telecom Telecom CrossboxCrossboxNormal viewNormal view

29

Field Service: Field Service: Telecom Telecom CrossboxCrossboxAugmented view: Blocks of postsAugmented view: Blocks of posts

30

Field Service: Field Service: Telecom Telecom CrossboxCrossboxAugmented view: Selected group connected to noteAugmented view: Selected group connected to note

31

Field Service: Field Service: Telecom Telecom CrossboxCrossboxAugmented view: Note “pulled in” for inspectionAugmented view: Note “pulled in” for inspection

32

Field Service: Field Service: Telecom Telecom CrossboxCrossboxAugmented view: Additional notesAugmented view: Additional notes

33

ARC: Augmented Reality for ARC: Augmented Reality for Construction Construction 19961996––9898(Joint with Building Technologies Lab)(Joint with Building Technologies Lab)

Assist worker in assembling spaceframe buildingGoal

Right pieceRight placeRight sequence

34

SpaceframeSpaceframe Construction Construction TaskTaskfor each spaceframe piece in sequence {

select piece and verify identityinstall pieceverify location and identity

}

35

User InterfaceUser Interface

See-through head-worn display

provides visual andaudio instructions

Display and reader are tracked

Bar code readerverifies part identity and placement

36

Real WorldReal World

37

Virtual WorldVirtual World

38

Real World +Real World +Virtual WorldVirtual World

39

Touring Machine Touring Machine 19961996––

Assist mobile user in exploring an unfamiliar environmentUser interface

See-through head-worn displayHand-held pen computer/tablet

Backpack containscomputer with 3D graphics carddifferential GPS system spread-spectrum radio modem

40

UI: Overlaid LabelsUI: Overlaid Labels

41

UI: Overlaid LabelsUI: Overlaid Labels

42

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

43

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

44

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

45

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

46

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

47

Touring Machine: Columbia CampusTouring Machine: Columbia Campus

48

Research Challenges:Research Challenges:Environment ManagementEnvironment Management

Task of managinglarge numbers of objects onlarge numbers of displays forlarge numbers of users

Develop methods toorganize, filter, modify, navigate

interactive information spaces that arelarge, shared, 3D, multimedia

49

Environment Management Environment Management ArchitectureArchitecture

World,User,Hdw,

Situation,Tasks

ARUI

InformationFiltering

Choose data to display

UI ComponentDesign

Map data to UI components

ViewManagement

Lay out UI components

Dynamically redesign UI

50

Information Filtering Information Filtering 20002000(Imaged with Naval Research Lab BARS)(Imaged with Naval Research Lab BARS)

Unfiltered: All objects shown Filtered: Only objects relevant to route-finding task are shown

51

UI Component Design UI Component Design 20012001

Precise position tracking: labels registered with real world

Imprecise position tracking: scaled world model displayed

52

Arbitrary viewpointArbitrary scene configurationDynamic environment

ProblemProblem

→Annotation soup

53

View Management View Management 20012001

Dynamic layout of 3D information to avoid occlusion, overlap:

LabelsAnnotationsUI components

54

View ManagementView ManagementNaïve AnnotationNaïve Annotation

Annotation based on centroidsMisplaced, overlapped, ambiguous

55

View ManagementView ManagementAutomated AnnotationAutomated Annotation

Annotation based on centroidsMisplaced, overlapped, ambiguous

Annotation using new approachCorrectly placed, overlap and ambiguity avoided

56

View ManagementView ManagementCampus OverviewCampus Overview