1

The Impact of the Combination of Real Object Interaction and Visual Overlay on Learning

By John Quarles

2

Presentation Outline Introduction Previous Work in Mixed Reality Anesthesia Machine Training My Previous Work Research Plan Timeline

3

Introduction Mixed Reality (MR) combines the

real world with a virtual world. MR is used in many training

applications Example: Flight Simulator

Can MR technology effectively improve learning? MR technology needs to be evaluated

4

Introduction Fused Worlds (FWs)

Visual Overlay (i.e. 3D graphics) Real and virtual object interaction

How do FWs impact learning?

5

Introduction 3 interdependent aspects of

learning: Learning Efficiency

Rate of learning Learning Quality

Mental models Learning Flexibility

Generalization

6

Introduction

LearningFused Worlds

Real and virtual object visualization

Real and virtual object interaction

Efficiency

Quality

Flexibility

How do FWs impact learning?

How do aspects of FWs impact learning?

7

Introduction My Previous Work

Studies comparing Real + Virtual Object Interaction to Virtual Object interaction

Real object interaction improved task learning

8

Introduction Proposed Work

Real world app: Anesthesia Machine Training

Focused on learning Can benefit from FWs

Plan: Augment an anesthesia machine with a FW Observe students as learning with the augmented

machine Framework: Combining Simulation and Realilty

9

Introduction Broader Impact

Establish the efficacy of FWs for learning.

Show where MR tech needs improvement.

Contribute to literature on MR, simulation, visualization and HCI.

Directly affects Anesthesiologists and others in related fields

10

Presentation Outline Introduction Previous Work in Mixed Reality Anesthesia Machine Training My Previous Work Research Plan Timeline

11

Previous Work in MR Mixed Reality:

Combines real world with virtual world

12

Previous Work in MR Registration Tracking Displays Interfaces

13

Previous Work in MR Displays

Magic Lenses Originated as 2D GUI manipulators Later Implemented as a 3D TUI

14

Future Work in MR Evaluation

How does MR impact the user? Real world applications

Common Frameworks Portability Code Reuse Authoring

15

Presentation Outline Introduction Previous Work in Mixed Reality Anesthesia Machine Training My Previous Work Research Plan Timeline

16

The Virtual Anesthesia Machine (VAM)

Simulation of a generic Anesthesia Machine

Interactive Web disseminated

17

The VAM Interaction with the VAM

Users interact with animated icons that represent components of the real machine.

18

Learning Tools The VAM The Opaque VAM The Augmented Anesthesia

Machine Real Anesthesia Machine

19

Presentation Outline Introduction Previous Work in Mixed Reality Anesthesia Machine Training My Previous Work Research Plan Timeline

20

My Previous Work Investigated the impact of FWs on

learning an abstract task in a lab environment.

Specifically: Benefits of Real Object Interaction

21

My Previous Work Task Description

3D spatial task Metrics

Errors Time

22

My Previous Work Overall Results

When interacting with real objects participants:

made fewer errors (0.33 errors/min vs 2.04 errors/min)

completed the task faster (108 sec vs 133 sec).

Real object interaction improved participants’ learning quality and learning efficiency.

23

Presentation Outline Introduction Previous Work in Mixed Reality Anesthesia Machine Training My Previous Work Research Plan Timeline

24

Research Plan Thesis Statement:

MR’s combination of overlaid virtual objects and real object interaction (fused worlds) will significantly improve user learning efficiency, quality and flexibility more than overlaid virtual objects or real objects alone.

This contextualization (the FW) of abstract simulation and that which is being simulated improves overall user learning of the application.

25

Research Plan Application: Anesthesia Machine

Training Research Overview

Augment an anesthesia machine with a FW

Compare student’s learning with and without the FW

Create a Framework that aids in combining simulation with reality

26

Research Plan Augmented Anesthesia Machine

(AAM)

 

Fusing World

s

• Fuses the VAM and the real machine

27

Research Plan AAM Visualization

Superimpose the VAM simulation over the real machine.

Use Computer Vision based tracking systems

28

Research Plan AAM Interaction

Interact naturally with the real machine

29

Research Plan Study: Discover if and how FWs

impact learning. Investigate two main aspects of FWs:

Virtual objects shown in context with the real world

Real object interaction Compare students who learn using a

FW (the AAM) to students who learn without the FW

30

Study Procedure

Pretest90 minlecture Posttest

Hands-on test

1st year Anesthesiology

Residents

AAM

Real Machin

e

3D Viz

VAM

Training Exercises

24 Hours

31

Research Plan Metrics

Multiple choice pre/post tests Questions come from review materials for

the American Board of Anesthesiology Exam

Hands-on test A fault is induced in the real machine and

students must find and correct it. Number of lessons completed

32

Research Plan Many other potential applications

VAM-like simulations prevalent in training

Code/Modeling Framework Library to help users

define the mapping between simulation and reality

visualize the mapping define and model interaction methods

33

Presentation Outline Introduction Previous Work in Mixed Reality The Virtual Anesthesia Machine My Previous Work Research Plan Timeline

34

Timeline Previous Publications

ISMAR, 3DUI Anticipated Publications:

Simulation: TOMACS, SIMULATION MR/HCI: ISMAR, VR, CHI Visualization: IEEE Viz Med/Psych Journals

35

Timeline Workflow

36

Acknowledgements Committee Chair: Ben Lok Committee Members:

Ira Fischler Paul Fishwick Jeff Ho Sem Lampotang

37

Fin.

38

Detailed Timeline VR’08, TVCG Journal “The Impact of the

Augmented Anesthesia Machine on Learning about Anesthesia Machines”

VR’09 or ISMAR’08: “How Real Object Interaction Impacts Learning in Anesthesia Machine Training”

CHI’09:“How Visualization Superimposed Over the Real World Impacts Learning in Anesthesia Machine Training”

CHI Journal or Presence Journal: “The Impact of Fused Worlds on Learning about Anesthesia Machines”

Medical/Psychology Journals.

39

Detailed Timeline Anticipated Publications

TOMACS: "An Approach to Contextualizing Simulation Models with Physical Phenomena with an Application to Anesthesia Machines"

ISMAR’07: “Using Mixed Reality to Effectively Combine Real Object Interaction and Abstract Simulation”

ISMAR’07: “The Impact of Real Object Interaction on Spatial Memory and Learning”

40

Summary Purpose: To discover if and how

FWs impact learning efficiency, quality, and flexibility?

Build the AAM and Framework Run a major study

Combined aspects of FWs are greater than individual aspects?

41

Summary Completing this work will:

Establish the efficacy of MR for learning efficiency, flexibility, and quality objectives

Highlight the areas of MR technology that need to be improved upon in future MR research.

Contribute to the literature on haptics, and MR interaction.

42

Introduction Broader Impact

Benefits VR/MR community Researchers App Developers

Benefits Anesthesiology community Doctors Students

43

Summary Innovations

Creating a way to combine 2D abstract simulation with the real world

Determining how FWs impact user learning

44

Research Plan AAM Visualization Method 2

Superimpose the real machine over the VAM.