new department of computer science, columbia university -...

1

Augmented Reality Interfaces for Procedural Tasks

Steven J. Henderson

Department of Computer Science - Columbia University

April 14, 2011

2

Procedural Task

A task whose learning requires the integration of two kinds of

human capability–intellectual skills and motor skills [Gagné-77]

Evolution of Procedural Task instruction

- Printed technical manuals

- Interactive Technical Manuals (IETMs) [Connell-78]

- Task Guidance Systems [Ockerman-98] (mobile or wearable IETMs)

1943 Harley-Davidson

Maintenance Manual

Task guidance system for telecom

troubleshooting

Example IETM Wearable IETM [Siegel-01]

Augmented Reality (AR)

AR view of starter installation

Integration of virtual content with a user’s natural view of the

environment, combining real and virtual objects interactively, at

real-time frame rates, and geometrically aligning them with each

other [Azuma-01]

User wearing AR display

4

Research Questions

What are the benefits of using an AR interface to support

procedural tasks?

- Benefits during informational activities

- Benefits during psychomotor activities

How can we develop effective user interaction techniques for

an AR interface supporting procedural tasks?

- Minimize interference with the task environment

- Minimize interference with the worker

What are the general design consideration in developing an

AR interface for supporting procedural tasks?

5

Contribution Overview Evaluation of AR in Informational

Phases of Procedural Tasks (Chapter 3)

- Articulated benefits of AR in informational phase

Evaluation of AR in Psychomotor Phases

of Procedural Tasks (Chapter 4)

- Articulated benefits of AR in psychomotor phase

Opportunistic Controls (Chapter 5)

- Proposed and evaluated novel class of interaction

techniques for procedural tasks.

Architecture for Constructing AR Interface for

Procedural Tasks (Chapter 6)

6

Talk Structure

Related Work exploring Procedural Tasks (Chapter 2)

AR Assistance in Informational

Phases of Procedural Tasks

(Chapter 3)

AR Assistance in Psychomotor


(Chapter 4)

Opportunistic Controls supporting

User Interaction with Procedural Tasks

(Chapter 5)

ARMAR Architecture (Chapter 6)

Conclusions & Future Directions (Chapter 7)

Talk Structure


Procedural Task Instructions How do people think about tasks?

8

Nature of Procedural Tasks

- [Gilberth-24], [Drury-90], [Vujosevic-97]

Required Abilities and Skills

- [Fleishman-84],[Bloomfield-03]

Teaching and Learning

- [Gagné-69, Bloom-76, Wetzel-83, Tannenbaum-93]

Fleishman’s Taxonomy

of Psychomotor abilities

Gilbreth and Gilbreth’s Therbligs

Bipartite Nature of Procedural Tasks

9

Neumann & Majoros [1998] proposed two-phase cognitive model for AR

applications in Maintenance and Manufacturing:

‒ Informational Phase: Read, comprehend, understand

‒ Work piece Phase: Psychomotor, align, orient, adjust, manipulate

Richardson and colleagues [2004] confirm similar model for assembly tasks:

comprehension

Phases follow egocentric vs. allocentric reference systems [Klatzky-1998]

Ability to visualize within each reference system affects how people

perform tasks [Kozhevnikov-2006]

Egocentric : Ability to imagine taking a different perspective in

space. Required for navigation, tracking, orienting

Allocentric: Ability to mentally manipulate objects from a stationary

point of view. Required for operation, repair, mechanical devices

Procedural Task Instructions What are the ingredients of high-quality instruction?

10

Use of Pictures

- [Booher-75],[Ellis-96]

Use of Text

- [Wright-77, Wright-81],[Smith-84]

Use of Animation

- [Tversky-02]

Cognitive design principles

- [Heiser-04]

[Heiser-04]

[Booher-75]

Computerized Instructions

11

IBIS [Seligmann-91]

Computer Generated Assembly

Instructions [Agrawala-03]

IBIS [Seligmann-91]

[Agrawala-03]

Creation

IETMs

- [Connel-78], [Rainey-91],[Boose-03]

Wearable task guidance systems

- [Siegel-96, 97, 01]

- [Ockermann-98]

Presentation

IETM

Talk Structure



(Chapter 3)

13

Maintenance and Repair Procedural Tasks

Well-formed design space for

application of AR

Current maintenance and repair

systems feature paper or Interactive

Electronic Technical Manuals (IETMs)

- Spatially disconnected assistance

- Static views and diagrams

- Cumbersome user interface

- Requires hands and job space

Leverage AR for

- Localization

- Hidden information

- In-situ Instructions and Training

IETM Interface

14

Related Work

Boeing Wireframe Bundle

- [Caudell-93],[Curtis-98]

Printer Servicing

- [Feiner-92, Feiner-93]

Toy Block Assemblies

- [Tang-03],[Robertson-08]

Dedicated Research Consortia

- 1999-2003 : ARVIKA

- 2001-2004 : STAR

- 2004-2006 : ARTESAS

ARVIKA

Printer Servicing [Feiner-93]

Boeing Wireframe Bundle [Curtis-98]

15

Informational Prototype Target Domain

LAV25A1 Turret Interior

LAV25A1 Turret (Extracted)

LAV25A1 Armored Personnel Carrier

LAV25A1 Turret Entry Hatches

16

Informational Prototype

17

Informational Prototype Software

Built using ARMAR Architecture

3D scene generation

- Managed by Valve Source game engine

- Video merged via external application

Content

- 3D models (static & animated)

- 2D close-up drawings and photos

- 2D Text in screen-fixed HUD

- Attention-directing graphics

18

Informational Prototype Hardware

Localization Sequence


Displays

- Custom-built video see-through (VST)

Head-worn display (HWD)

- NVIS optical see-through (OST) HWD

Tracking

- OptiTrack optical tracker w/ active markers

- Fiducial-based optical tracking

Interaction

- Android phone-based wrist worn controller

19

User Study Tasks and Conditions

Preceded by 2 pilot studies

18 LAV25A1 maintenance tasks

- Actual tasks from IETM

- Arbitrary ordering to mitigate

familiarity and learning effects

- Installations/Removals

- Placement of Switches & Levers

- Inspections

Baseline comparison

techniques

- Fixed LCD Display (LCD)

"Improved" IETM

- Untracked HWD (HUD)

HUD Condition LCD Condition

20

Within-subject design

Counterbalanced start condition

Fixed, arbitrary task sequence

6 subjects (male, age 18-28),

in USMC LAV mechanic course

- 4 additional subjects in pilot study

Included subjective evaluation

Hypotheses

H1: AR faster completion than HUD, LCD

H2: AR faster localization than HUD, LCD

H3: AR produces fewer errors than HUD, LCD

H4: AR less head rotation than HUD, LCD

H5: AR less head translation than HUD, LCD

H6: AR faster head movement than HUD, LCD

User Study Experiment Design

21

User Study Results Localization and Completion

Localization time:

- Display condition produced significant

effect on task localization time

- Post-hoc comparison:

AR localization time 53% of LCD*

AR localization time 44% of HUD*

- Confirms H2

Task completion times:

- No significant effect of display condition

on task completion time

- Fails to confirm H1

*Statistically significant (p < 0.05) Completion Time

Localization Time

22

User Study Results Rotational Head Movement and Velocity

AR rotational movement:

- Pitch: 38% that of LCD*

- Roll: 25% that of LCD*

- Yaw: 35% that of LCD*

- Partially confirms H4 (AR vs. LCD)

AR rotational velocities:

- Roll: 1.48 times that of HUD*

- Yaw: 1.95 times that of HUD*

- Partially confirms H6 (rotational velocity)

LCD rotational velocities:

- Yaw: 1.74 times that of AR*

*Statistically significant (p < 0.05)

23

User Study Results Translational Head Movement

Mean translational head exertion:

AR 37% that of LCD*

AR 69% that of HUD

Partially confirms H5 (AR vs. LCD)

Mean translational head velocity:

AR 1.6 times that of HUD

LCD 1.7 times that of AR

Fails to confirm H6

(translational velocity)


2D Histograms of Head Position

Supporting Task Focus

24

25

User Study Results User Survey

Median responses to survey questions:

- AR most satisfying

- LCD easiest to use

- AR as intuitive as LCD Difficult Easy

Satisfaction

Ease of Use

Intuitiveness

Most Preferred Condition:

- 4 of 6 subjects selected LCD

- Significant main effect (Friedman test)

- Only LCD-HUD significant (Wilcoxon test)

Most Intuitive Condition:

- 4 of 6 subjects selected AR

- No significant effect (Friedman test)

User comments:

"Enjoyed [AR] system the most..easy to navigate"

"I liked [AR] system..all I had to do was follow the

red line"

"Will be successful with better picture"

"Display gets in the way"

26

Summary of Findings

1. AR allowed mechanics to localize more quickly than LCD

- Mechanics can begin work more quickly

- Reduces transition time between tasks

2. AR allowed mechanics to localize more quickly than HUD

- AR visualization adds value beyond untracked HUD displays

3. AR reduced head movement compared to LCD

- Could mean less stress and fatigue

- Must adjust for effects of wearing HWD

4. Mechanics found AR system intuitive and satisfying

5. Mechanics made few errors under any condition

Talk Structure



(Chapter 4)

28

Related Work

AR Systems that track user and objects

- [Feiner-93],[Zauner-03],[Salonen-08]

Limited use of object tracking data

- Needle Biopsy Systems

[State-96],[Rosenthal-02],[Wacker-06]

Track needle throughout task for

internal visualization

Not prescriptive in nature

- Obstetrician trainer [Blum-07]

Depicts expert motions to emulating trainee

Offline learning tool; Not evaluated

29

Psychomotor Prototype

30

Psychomotor Prototype Software


3D scene generation

- Managed by Goblin XNA

Content - Dynamic, prescriptive 3D arrows

- Dynamic Highlights

- Dynamic Billboard labels

- Motion Paths

- Assistance updated based on

changes to task environment

Dynamic, Prescriptive Arrows

Dynamic Highlights

31

Psychomotor Prototype Hardware


Display

- NVIS optical see-through (OST) HWD

Tracking

- OptiTrack optical tracker w/ active markers

- Fiducial-based optical tracking

Interaction

- Proactive computing model

- Custom USB button, Wiimote controller

32

User Study Task

Assembly Task

Dart 510 Combustion Chamber

- 7 Chambers on engine

- Each chamber consists of:

Upper section “cone”

Lower section “can”

- Each chamber requires unique

pairing and alignment of cone and can

Workbench Assembly Environment

- Designed to generate statistical power

- Supports multiple iterations of assembly

Dart 510 Engine

Cone

Can Assembled

Chamber

Workbench Assembly Environment

33

User Study Task

Step Description Activity Type

1 Locate Can X in Bin W Locate

2 Move Can X to Turntable Position

3 Locate Cone Y in Bin V Locate

4 Place Cone Y on Can X Position

5 Align Cone Y with Can X; Insert pins Align & Pin (Psychomotor)

6 Move assembly XY to Bin Z Position

Single trial: 6-step procedure attaching Can X and Cone Y

Analyzed by major activity type

Align & Pin step represents psychomotor phase

Trial repeated 14 times in fixed combinations of cans & cones

- Each combination appears at least once

- Pseudo-random selection of paired holes

34

User Study Display Conditions

AR Condition LCD Condition

35

User Study Experiment Design

Within-subject design

Counter-balanced start condition

Recruited 28 subjects

- 7 female, 21 male; Age 18–44, 𝑋 =26

- First 6 participants comprised pilot study


Hypotheses

- H1: AR faster technique during psychomotor activities

- H2: AR more accurate than LCD during psychomotor activities

- H3: AR most preferred technique

- H4: AR ranked as most intuitive

36

User Study Results Completion Time

Display condition significant main

effect on Align Activity completion

time.

- AR 21.3 seconds faster than LCD

Statistically significant (p < 0.05)

- Confirms H1

37

User Study Results Accuracy

Display condition significant main

effect on mean alignment error as

measured at task completion

Accuracy rating

- Binary measure of correct alignment

AR: 95.3% mean accuracy rate

LCD: 61.7% mean accuracy rate

Significant difference in mean accuracy rate

(p < 0.001)

Angular displacement

- Average angular difference between can and cone at task completion

AR: 0.08 radians (0.25 inter-hole widths)

LCD: 0.36 radians (1.15 inter-hole widths)

- AR average displacement 22% that of LCD; Statistically significant (p < 0.05)

Confirm hypothesis H2

38

User Study Results Qualitative Results

Participants assigned higher ratings for

the AR condition in Ease of Use,

Satisfaction, and Intuitiveness

compared to LCD ratings

Most Preferred Condition

- 20 of 22 participants ranked AR as most preferred

(Significant, p < 0.001)

- Confirms hypothesis H3

Most Intuitive Condition

19 of 22 participants ranked AR as most preferred

(Significant, p < 0.001)

Confirms hypothesis H4

39

Comparison to Physical Labels

Follow-up pilot study to test AR

against idealized baseline

Printed labels attached to

assembled components

Not always possible or

realistic

Study

- Same task; Substituted PRINTED for LCD

- 6 participants; all male; Age 19–27, 𝑋 =23.5

Results:

- Display condition failed to exhibit significant main effect on task

completion time for task completion time

Printed Condition

40

Findings

1. AR allowed participants to complete align and pin activity

more quickly than LCD

2. AR allowed participants to complete align and pin activity

more accurately than LCD

3. AR most preferred technique

4. AR most intuitive technique

5. No evidence to suggest AR technique differs from idealized

condition

Talk Structure



(Chapter 5)

42

Motivation

Many procedural tasks pose competing constraints:

- Constrained use of eyes and/or hands

- Hands not visible

- Cannot modify environment

Photo courtesy European Space Agency

43

Tangible UI harvested from the environment

Comprised of:

-A physical affordance

-A 3D widget

-One or more gestures

Opportunistic Control

44

Related Work 2D haptically discriminable widgets

- [Buxton-85]

3D virtual buttons on undifferentiated

surface

- [Weimer-89]

Tangible bits

- [Ishii-97]

Passive real-world interface props

- [Hinckley-94]

Light Widgets

- [Fails-02]

[Weimer-89]

[Buxton-85]

[Fails-02]

45

Opportunistic Control Components

A naturally occurring affordance

A 3D Widget

Mapping between 3D widget and the affordance

A grammar of hand gestures

Mapping between gesture grammar and 3D Widget

Transformation between gesture and affordance spaces

46

Affordances Supporting OCs

Button

Based

OCs

Valuator

Based

OCs

Movable

OCs

47

Affordance Rules of Thumb

Do not overload meaningful affordances

Do not endanger user in another context

Do not damage affordance in another context

48

Gesture Recognition

Each affordance is defined by a convex polyhedron

bounding the usable geometry of the OC

Vision algorithm locates user's hand in camera coordinates

using appearance-based techniques

Hand location mapped to affordance coordinate space

Hand’s placement/movement in affordance space identifies

gesture

Tracked segmentation:

- Gesture to affordance mapping in real time

- Reduced segmentation workload

- Entire OC can move

49

Widgets

Provides visual feedback linked to gesture/affordance

Geometry of widget tightly coupled with affordance

Modeled as 3D Widgets [Conner-92]

Augmented transition network (ATN) of widget responds to

user gestures and alters 3D model

Slider Widget ATN 3D Model of Slider Widget

50

User Observation Study

Examined 3 questions:

- How do users perceive potential OC affordances?

- How to redirect user thinking to view affordances as OCs?

- What heuristics determine the best affordances for OCs?

Study

- Fifteen subjects (11 male, 4 female)

- Subjects presented with common interface tasks [Dachselt-2005]

- Tasks associated with two domains

- Subjects selected any affordance to create a UI for accomplishing tasks

- “Wizard of Oz” feedback presented on hand held AR display

Using a connector for 3D

Object manipulation

(view through AR display)

51


Maintenance Domain Home Entertainment

Domain

Discrete Valuator Task Menu Selection Task 3D Object Selection Task

52


53

User Observation Study Results

Subjects selected a plurality of valuator-based affordances in

both domains

Subjects used button-based and valuator-based affordance

interchangeably

Subjects combined affordances to accomplish tasks

General omission of movable OCs

Task Type Button Based Valuator Based Moveable Button Based Valuator Based Movable

3D object selection 13% 60% 27% 47% 40% 20%

3D object manipulation 7% 87% 20% 20% 67% 13%

3D scene control 13% 73% 27% 20% 53% 33%

2D document visualization 7% 67% 27% 20% 67% 13%

Discrete valuators 67% 20% 13% 67% 33% 7%

Continuous valuators 33% 40% 33% 47% 53% 13%

Menu selection 47% 53% 13% 53% 33% 0%

All Tasks 27% 57% 23% 39% 50% 14%

Maintenance Interaction Domain (MA) Home Entertainment Domain (HE) Task Type Button Based Valuator Based Moveable Button Based Valuator Based Movable

3D object selection 13% 60% 27% 47% 40% 20%

3D object manipulation 7% 87% 20% 20% 67% 13%

3D scene control 13% 73% 27% 20% 53% 33%

2D document visualization 7% 67% 27% 20% 67% 13%

Discrete valuators 67% 20% 13% 67% 33% 7%

Continuous valuators 33% 40% 33% 47% 53% 13%

Menu selection 47% 53% 13% 53% 33% 0%

All Tasks 27% 57% 23% 39% 50% 14%

Maintenance Interaction Domain (MA) Home Entertainment Domain (HE)

54

Preferred Physical Features

Top 3 features in each

domain were all located

roughly at eye level

Suggests importance of

location in selecting OC

affordances (minimize

physical exertion by user)

MA Domain HE Domain

20%

18%

11%

20%

11%

8%

55

Additional Findings

Influence of Surrounding Context

- Some users verbalized hesitancy to respond to task. Examples:

“I don’t know how to hook up a VCR”

“I’m not a mechanically inclined person”

- Suggests surrounding context might cloud OC perception

- OC implementations could/should use virtual content to mitigate this effect

(e.g., hide backgrounds, highlight affordance)

User Suggested Designs

56

Prototype

User’s view through HMD

Segmentation from overhead camera

Built using ARMAR architecture

Tracked overhead camera for gesture recognition

Tracked stereo video see-through HWD

Button, valuator, and movable OCs

Close up of Affordances

57

Movable OC

58

Interface Technique

User Study

Selection task

Five button based OCs used for selection

Baseline comparison technique (BL) :

Virtual buttons on flat, undifferentiated surface

Hypotheses

H1. OC faster than BL

H2. OC more accurate than BL


15 subjects (11 male, 4 female)

Counterbalanced, within-subject design

10 inspections (trials) x 5 locations

BL

OC

59

Interface Technique User Study (OC Condition)

60

Performance User Study Results

Quantitative Results

- OC completion time 86% that of BL

(statistically significant, =0.0125)

- Did not identify significant effects on error rates

Completion Time Errors

61

Performance User Study Results

Qualitative Results

- 73% of users preferred OC over FL (Significant ranking, p=0.02)

- Users liked ability to do

“eyes-free” interactions

- No significant differences in

mean response to ease of use,

satisfaction, or intuitiveness Likert

scale questions

Talk Structure


ARMAR Architecture

63

Practical Rules of Thumb

Tracking & Registration

- Head tracking sufficient when accuracy requirements > 1cm

Object tracking recommended when requirement < 1cm

- Use soft edges to hide registration errors

- Sharing tracking information from networked cameras tracking

fiducial markers highly susceptible to distortion

Requires robust calibration whenever cameras change position

Displays

- Stereoscopic viewing preferred when hand tools are required

- Individualized calibration of head worn displays important when

requirement < 2cm

64

Talk Structure


66

Summary of Contributions

Evaluation of AR in Informational Phases of Procedural Tasks

- Explored ability of AR to support task focus during procedural tasks

IEEE TVCG 2011 (In press)

- Showed AR is faster at localizing mechanic during procedural tasks

- Showed AR reduces some types of head movement

IEEE ISMAR 2009 (Best Paper)

Evaluation of AR in Psychomotor Phases of Procedural Tasks

- Showed users aided by active, prescriptive assistance rendered using AR

were faster in completing psychomotor portion of realistic assembly task

- Showed users preferred AR over LCD condition

Submitting to IEEE ISMAR 2011 (Deadline May 2011)

67

Summary of Contributions

Opportunistic Controls for Procedural Tasks

- Explored user affinities for OC affordances

IEEE TVCG 2010

- Showed OCs are faster than undifferentiated baseline in supporting

selection task

ACM VRST 2008 (Best Paper)

Architecture for Constructing AR Interface for Procedural Tasks

- Showed architecture is sufficient and useful for constructing AR interfaces

for procedural tasks

USAFRL Tech Report 2007, ACM VRST 2008 , IEEE ISMAR 2009,

IEEE TVCG 2010, IEEE TVCG 2011

Future Directions View Pose Management

68

Attention directing without View Pose Management

Attention directing with View Pose Management

Future Directions

Psychomotor assistance

- What are the set of possible visualization techniques to aid common

psychomotor activities? Which are optimal?

69 Extract of taxonomy proposed by Guo and Tucker [96] with added estimates for psychomotor activities

Future Directions

Opportunistic Controls

- Add depth filter to allow hovering and clutching

- Feature-based tracking

- Automatically detect and analyze affordances to use as OCs

- Allow user to quickly indicate affordances for OCs

Training vs. Assistance

- Interaction

How best to promote learning while assisting?

Gradually relaxed assistance; remain as sentinel

- Which to apply?

Can one jump directly into assistance without any prior learning?

What types of tasks demand training? Which can suffice as assisted?

70

71

Acknowledgments

Steve Feiner

Peter Allen, John Kender

Barbara Tversky, Mark Livingston

Thanks to:

- Maria, Eva, Anna

- Ohan Oda, Sean White, Lauren Wilcox, Nick Dedual, Mengu Sukan,

Christian Holz, Hrvoje Benko, Eddie Ishak, Dale Henderson, Paul Blaer, Quy O

- Department of Systems Engineering, USMA (West Point)

- Bengt-Olaf Schneider (NVIDIA) for technical assistance

- USMC Cadre and students at Aberdeen Proving Ground, MD

- Engineers at the Marine Corps Logistics Base, Albany, GA

- David Madigan, Kyle Johnsen, and Magnus Axholt

- Generous gifts from NVIDIA and Google

This research was funded in part by:

Office of Naval Research Grant N00014-04-1-0005

US Air Force Research Lab Grant FA8650-05-2-6647

US Army Advanced Civil Schooling Program

Questions?

72

Backup

73

User Study Results Completion time by

Task Type

76

User Study Results Localization and Completion

Mean localization times:

- AR 47% faster than LCD*

- AR 56% faster than HUD*

- Confirms H1

Mean task completion times:

- AR 23% faster than HUD

- LCD 18% faster than AR

- Fails to confirm H2


Completion Time

Localization Time

77

User Study Results Translational Head Movement

Mean translational head exertion:

AR 62% less movement than LCD*

AR 29% less movement than HUD

Partially confirms H3 (AR vs. LCD)

Mean translational head velocity:

AR 60% faster than HUD*

LCD 40% faster than AR*

Partially confirms H5 (AR vs. HUD)


Translational Head Exertion

Translational Head Velocity

78

Procedural Tasks

A task whose learning requires the integration of two kinds

of human capability–intellectual skills and motor skills

[Gagne-77]

Characteristics:

- Ordered set of prescribed activities

- Varying:

Level of required planning

Number of steps

Number of decision points

Required cuing

Flexibility of step ordering

Type of goal

Sandwich Assembly

Aircraft Assembly

Procedural Task Instructions Non-automated forms of Instruction

79

Workcard Lube chart Single-sheet Instructions

Manual

80

User Study Results Rotational Head Movement

Rotation about neck (yaw) greatest source of rotational head

movement

AR resulted in smaller ranges in head yaw in 15 of 18 tasks when

compared to LCD

Could equate to less stress on neck and shoulders (depending on

effects of HWD)

-70 -20 30 80 130

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

T11

T12

T13

T14

T15

T16

T17

T18

Ta

sk

Normalized Yaw Direction (degrees)

LCD

HUD

AR-70 -20 30 80 130

T1

T2

T3

T4

T5

T6

T7

T8

T9

T10

T11

T12

T13

T14

T15

T16

T17

T18

Ta

sk

Normalized Yaw Direction (degrees)

LCD

HUD

AR

APG Errors?

81

82

Talk Structure




(Chapter 3)



(Chapter 4)



(Chapter 5)



new department of computer science, columbia university -...

Documents