Virtual Worlds and Real World

By the real… Of the Real… For the Real…

Contents

• What to capture from the real world?

• How to simulate in the virtual world?

• How to measure transfer in the real world?

A model of the real world..

IT MAY BE ARGUED THAT ACTIVITIES, INTERACTIONS AND COGNITION IS WHERETHE BIGGEST BANG FOR THE BUCK IS….

Visual Appearance

• Need 3D Models: www.3dcafe.com, www.exchange3d.com, www.turbosquid.com

• Modification of Models: Need Maya, 3D Studio Max… free solutions Vorpal, MeshMan (www.ryanholmes.net) 3D Object converter.

• Need textures… get a camera and click pics of real world. Make images small in size <100K in size.

• Need space: www.secondlife.com, www.activeworlds.com www.forterra.com

Entities.

• Poser® allows for clicking front and side images and enables building of a model.

• For non-human entities search the 3D models websites…

Activities and Interactions

• This is where conventional off-the-shelf approaches don’t work.

• We need a fly on the wall… a system which much like blackbox captures interactions in real environment without much interventions.

• We can develop models of normative behavior and simulate that and…

Interactions in a Complex Environment

• High performance, high complexity environment with high interpersonal variations and interactions that are unstructured.

• Interactions are multimodal and not just verbal– Speech– Movement– Proximity – Gestures

THE PROBLEM

BehavioralManifestations

BehavioralManifestations

Cognitive Foundations

Cognitive Foundations

Capturing Interactions….DetailedBut HumanIntensive andCan missDynamic Events

Data DrivenToo CoarseNot cognitively grounded

Laxmisan et al. 2007 Malhotra et al. 2007

Alwan et al. 2006Ostbye et al. 2003

Proposed Hybrid Method

Qualitative Data

OntologyBy Zhang et al.

Quantitative Data

Movement/Proximity Data Speech/Voice Data

Capturing Movement Data• A common approach is to

focus on <x,y,z> location of entities and then developing activities as deterministic models of location.

• The issue: sensors such as RID sensors suck when it comes to movement by themselves.

• What can we do?• Well the end product is

activities.. Why not develop probabilistic models of activities from noisy data.

Activity Recognition• Activity Recognition is a

burgeoning area..• Routinely done through

computer vision, sensor processing and data mining.

• We use temporal modeling techniques such as Hidden markov Models for detecting activities (gestures).

Scenario

But I really really want location• A multisensor approach is a

valid method of finding exact locations.

• Multisensor fusion provides multiple streams of data and enables a correction mechanism to combine multiple noisy streams to yield a single noiseless stream.

• A possible mechanism for sensor fusion is a predictro corrector mechanism called Kalman Filter.

Kalman Filter

Extended Kalman FilterDiscrete Kalman Filter

Assumed noise

Integration of Audio Data

• HIPAA regulations: security and privacy• Time synchronization• Audio Analytics– Number of words spoken– Tone/Amplitude of the signal– Directionality of Verbal Interactions.

Movement Analysis Results

UT HoustonBanner Health

Location Detection• Kalman filter to combine accelerometer data

with RID based location data

Virtual Playback and Analysis Tool

Demo

Building Learning Environments

Persuasive Collaborative FrameworkExternal motivatorsInternal motivators

Feedback on patient conditionsdue to decisions

Encouraging group consensus, Dissent.Shared mental models

Tabletop exercises

*courtesy ICT USC

Supplemented by offline sessions on web 2.0 tools

Case Study Design• Develop a decision tree for

case solving stopping at different times during vignettes and asking questions.

• Using custom software it is possible to group answers based on clinical specialtys or groups

• Word clouds can be shown for answers by these different groups to enable visualization of differences in mental models and then promotion of shared mental models.

Validation Strategy

Validation Strategy

• Actual usage statistics

– Track usage statistics

– Track number of times community option is used.

– Track number of times alternative path of care is used.

• User preference models

– User preference for tools

• Reveal the effect of different tools on diagnostic and clinical abilities.

• Differentiate between levels of expertise.

• Reveal underlying trends on technology adoption with regards to attitudes to

technology, demographic information etc.

• Reveal difference in teams physically co-located and physically separated during

training.

Some Interesting Studies.

• Discussion on what can be done.

Take Home Simulators• Re-use of existing resources such

as simulation gaming platform has several advantages

• Can provide practice on psychomotor and cognitive skills

• Engaging and fun for trainees • Several students can study

together• Connectivity proficiency scores can

be transmitted to database over the Internet

• Can be deployed anywhere, remote areas as well as developing countries

Methodology For Choosing ExercisesCognitive task analysisSuturing->{setting the needle->passing suture->tying}

Matching observationalParameters in the real world And virtual world

Monitor progressthrough mechanism that work in an ambient manner

Adapt gaming scores to our needs

Wii and fine motor skills

Fine motor skills based games are very suitable

Very high correlation with basic gestures of surgery

Quantitatively we found that hand movement acceleration, and joint angles showed 0.78 to 0.91% correlation.

Cons: doesn’t have the fulcrum effect and significant weight.

Apparatus

• Gaming Extensions to Wii can be modified for surgical probe based interactions.

• WiiMote Extension• Movement Constrainer

Location of wiimote

Full System in Action

Study

Robotic Surgery Simulator

Analysis• Exploring the error innovation

continuum• Focus in analysis on best practices

and validity of checklists in Trauma and critical care settings.

• We performed analysis on ATLS training and execution and used the tool to analyze if activities as cited in checklists are followed.

• Hypothesis: Experts assign criticality to steps in a procedure and tend to follow high criticality steps with precision but innovate in low criticality steps.

• Criticality obtained by focus group sessions with physicians, nurses and residents with checklists.

Adherence to Best Practices and Accuracy

Perc

enta

ge

Future Work: Technical

• An activity segmentation algorithm: using the accelerometer, we aim to define a method to segment data into activities and then recognize them.

• Based on Kahol (2003) , we propose to use Bayesian techniques to identify activity boundaries automatically.

• Pilot data gathered at ASU and UT Houston.

Future Work: Analytical

• Employ the tool to further explore error-innovation continuum in critical care environment.

• One mans error is another mans innovation.• This evolves extensively in critical care

environments and ASU and BannerHealth will develop methodology to explore how errors and innovations evolve in critical care environments.

Future Work Interventions

• The developed tools will be employed with new year residents for training in a control group experimental group paradigm.

• We expect that the group with exposure to workflow will be better integrated in the trauma unit workflow as compared to the group that doesn’t.

Conclusions

• Virtual from real is fun and effective…

• Focus on the activities and interactions visual brilliance is over-rated and your main aim is education anyway…

• Go with environments that allow programming…

• What are you waiting for… build it!

thanks

Partner

LABORATORY