Clinical Virtual Reality: Virtual Humans to Populate Virtual Worlds

Skip Rizzo, Ph.D. University of Southern California

Associate Director, Institute for Creative Technologies Research Professor, Dept. of Psychiatry & School of Gerontology

Los Angeles, California

2

USC Institute for Creative Technologies

What about Virtual Humans?

N = 36. All p < .001. 90 and 91% panic-free at follow-up.

0

0.5

1

1.5

2

2.5

PRE POST 12-mo F-UP

PDSS

VR In vivo WL

0

5

10

15

20

PRE POST 12-mo F-UP

FQ-Ago

VR In vivo WL

Botella et al. (2007)

Panic disorder and agoraphobia

Virtual Human Exposure as good as

the real thing!

0

20

40

60

80

100

Pre Post

Liebowitz (total)

VR In vivo

N = 36, 12 sessions. CBT in group

Social Phobia

Fea r o f Publ ic Spea k in g

North et al

1997

Slater et alSlater et al

19991999

Kim et al.

Kim et al

20002000(Grillon, Riquier, Herbelin & Thalmann 2009

Anova Type of audience, p < .05 Neutral > Positive = Negative, p < .05

0102030405060708090

100

Sa

tis

fa

ctio

n t

ow

ard

s t

he

pe

rfo

rm

an

ce

Neutral Positive Negative

Type of audience

0

2

4

6

8

10

12

14

MP

RC

S

Neutral Positive Negative

Type of audience

ANCOVA (estimated from data in the paper) Type of audience, p < .05 Negative > positive = neutral.

Fear of Public Speaking

Phobics React to Variations in

Virtual Human States

Eye Tracking Pre and Post VR Exposure (Grillon, Riquier, Herbelin & Thalmann, 2006-2010)

Using Virtual Humans with

integrated biometrics to better

understand clinical outcomes…

Baumann et al. (2008)

VR Cue Reactivity for Addiction Tx

Baumann et al. (2008)

Changes in restaurant with

and without smoking stimuli

Virtual Casino

VR Cue Reactivity for Addiction Tx

• Craving higher in VR Smoking, Alcohol and Cannabis

situations vs. neutral cues settings • First use of scent in VR cue research • High levels of immersion/presence = higher rated realism • Increased level of attention to cues sight, smell, thoughts • Increased reactivity GSR • Higher confidence to resist relapse with VR

(Bordnick, et al., 2005, 2006. 2008, 2009, 2012)

Addiction therapy is being supported by scenarios

that involve Virtual Humans

Bordnick, 2013 UH GCSW

New versions of the Virtually Better Inc. Social Phobia/Addiction Applications

17

22

27

32

37

42

47

52

57

PRE TX POST TX

Pre-Treatment

Post-Treatment

3 Month FU

p < .001 (n=20)

16 of 20 No Longer meet DSM criteria for PTSD at Post-TX

p < .001 (n=14)

1 Week

Post TX

3 Month

Post TX Pre-TX

Naval Med Center SD/Camp Pendleton PTSD Checklist-Military (PCL-M)

PreTreatment, PostTreatment & 3 Month Follow-up

Average # of Sessions < 11

Treatment Completers n=20

40

35

25

20

15

30

Assessment over Time

45

50

55 60

65

70

75 80

PCL-M

Sco

re -

Sym

pto

m S

everi

ty

Average # of Sessions < 11

16 Successful

Treatment Completers n=20

17 = No

Symptoms

Endorsed

4 Unsuccessful

Naval Med Center SD/Camp Pendleton PTSD Checklist-Military (PCL-M)

PreTreatment, PostTreatment & 3 Month Follow-up

17

22

27

32

37

42

47

52

57

62

VRET

Pre-Treatment

Post-Treatment

Pre-TX Post TX

p <.001

(n=24)

Average # of Sessions < 7.4

Treatment Completers n=24

Madigan Army Medical Center-Ft. Lewis (Reger et al)

PTSD Checklist-Military (PCL-M) PreTreatment & PostTreatment

Effect

Size

d = 1.17

Reliable Improvement Index = 62%

20

25

30

35

40

45

50

55

60

65

70

75

80

Prolonged Exposure VR Exposure

Pre-Treatment

Post-Treatment

3 Month FU

Pre-TX 1 Week

Post TX

3 Month

Post TX Pre-TX 1 Week

Post TX

3 Month

Post TX

Naval Med Center SD/Camp Pendleton Clinician Administered PTSD Scale (CAPS)

PreTreatment, PostTreatment & 3 Month Follow-up

VR Produced Significantly Better Follow-Up Outcomes Compared to

Traditional Prolonged Exposure

20

STRIVE: STress Resilience In Virtual Environments

Skip Rizzo, Ph.D. Associate Director Medical VR

Institute for Creative Technologies University of Southern California

J. Galen Buckwalter, Ph.D. Research Scientist

Institute for Creative Technologies University of Southern California

21

”To be clear, CSF will serve as a catalyst for changing Army culture—from a culture in which behavioral health was once stigmatized to a culture in which psychological fitness is recognized as every bit as important as physical fitness.” Gen. George W. Casey Jr. (American Psychologist, Jan. 2011)

”…we are moving beyond a “treatment-centric” approach to one that focuses on prevention and on the enhancement of the psychological strengths already present in our soldiers.” Gen. George W. Casey Jr. (American Psychologist, Jan. 2011)

STRIVE: The movement towards resilience

Virtual Human Agents

Autonomous virtual characters that can have meaningful interactions

with human users

Reason about environment Understand and express emotion

Communicate through speech & gesture Play the role of teachers, peers, adversaries

Marketing

Training Research

Dr. Perez SASO-ST, SASO-EN

Elder-Al-Hassan SASO-EN

C3IT Cultural training

ELECT Bi-Lat

Hassan Emotional Dialog Modeling

Virtual Humans

Raed Tactical Questioning

Sgt. Star

Sgt. Blackwell Clinical Diagnosis

Radiobots JFETS Training

Rapport Agent Study

ICT’s Virtual Humans Portfolio

Justina Virtual Patient

Gunslinger

Justin Virtual Patient

Museum Guide & Coach

Support

Veterans Center SimCoach

Virtual Patients – Military and Civilian Applications

Virtual Patients w/USC Psychiatry

USC Social Work Military MSW Program

SickCall

Justin Justina

Collaboration with the USC School of Social Work Masters in Military Social Work Program

Virtual Patients Lab

(Rizzo, Parsons &

Kenny, 2010)

• Can provide guidance, support & encouragement

• Can model exercise activities for user

• Uses natural language recognition or text input

• Builds a model of the user

– NOT JUST A CHAT BOT

– Holds conversation in ‘memory’

– Has range of answer options to choose from

• Web-based authoring tools and delivery

27

The Future

28

Detection and Computational Analysis of Psychological Signals

USC Institute for Creative Technologies

Skip Rizzo, Ph.D. Associate Director - Institute for Creative Technologies Research Professor, Psychiatry and School of Gerontology University of Southern California Los Angeles, California

Louis-Philippe Morency, Ph.D. Research Scientist - Institute for Creative Technologies Research Assistant Professor, Computer Science University of Southern California Los Angeles, California

Within DARPA Program: Detection and Computational

Analysis of Psychological Signals

Alesia Egan & Josh Williams Project Managers - Institute for Creative Technologies

Jon Gratch, Ph.D. Associate Director - Institute for Creative Technologies Research Professor, Computer Science University of Southern California Los Angeles, California

Stacy Marsella, Ph.D. Associate

Director - Institute for Creative Technologies Research Professor, Computer Science University of Southern California Los Angeles, California

David Traum, Ph.D. Research Scientist - Institute for Creative Technologies Research Professor, Computer Science University of Southern California Los Angeles, California

Mark Bolas, M.S. Research Scientist - Institute for Creative Technologies Professor, School of Cinema University of Southern California Los Angeles, California

Eric Forbell, M.S. Institute for Creative Technologies Lead Engineer University of Southern California Los Angeles, California

Detection and Computational Analysis of Psychological Signals

USC Institute for Creative Technologies

In Partnership with

Cogito Health (Pentland, Azarbayejani,

Zhang)

31

A Kiosk-Based

Intelligent

Healthcare

Guide that can

Sense your

State

DARPA: Detection and Computational Analysis of Psychological Signals

(DCAPS)

SimCoach Sensei

An Online

Intelligent

Healthcare

Guide

A Kiosk-Based

Intelligent

Healthcare

Guide that can

Sense your

State Using a WebCam, Kinect and a Microphone – to give

“eyes” and “ears” to a SimCoach!!!

Automatic Behavior Analysis (using MultiSense)

Gaze and head pose Eye and head gaze Head tilt

Facial expressions Smile intensity Eye brows Facial action units

Affect Negative: sadness, anger Positive: joy, happy

Acoustic Voice quality Pitch Intensity Variability

Conversational Response latency Turn-taking patterns

Speech recognition Sentiment analysis

Association between distress and fidgeting

0

0.02

0.04

0.06

0.08

0.1

Smile ratio

not-distressed distressed

*

2

4

6

8

not-distressed distressed

Hand fidgeting duration (s)

**p < .01

p < .05 p < .05

Distressed subjects fidget longer

Association between distress and smiles

Distressed subjects use shorter smiles.

0.1

0.3

0.5

Smile ratio

-10

0

10

20

30

Average vertical gaze

not-distressed distressednot-distressed distressed

***

Automatic

p < .01

Association between distress and gaze

0.1

0.3

0.5

Smile ratio

-10

0

10

20

30

Average vertical gaze

not-distressed distressednot-distressed distressed

***

Distressed subjects look towards the ground more on average.

p < .05

Association between distress and audio features

0.1

0.2

Voice Quality (NAQ)

not-distressed distressed

***

4

8

12

Energy Std.

not-distressed distressed

***

Distressed subjects vary their speech intensity less, and use more breathy voice.

p < .001 p < .001

12 Distress Indicators

Affect: Joy display Anger display Sadness display Smile intensity/duration

Engagement: Vertical eye/head gaze

Variability: Speech intensity Speech dynamic variation

Agitation: Self-adaptors Legs fidgeting Voice quality

Latency: Response delay Interaction balance

… and more to come!

38

A Telemedicine

Decision

Support Tool

TeleCoach

DARPA: Detection and Computational Analysis of Psychological Signals

(DCAPS)

Military Applications are Driving the Technology Soon to become a Job Interview Training System for

Persons with Asperger’s Syndrome

Mobile SimCoach

15

Braveheart

The Future

The Digital Homestead Low Cost Interaction

Advanced Displays

The future of Home-based Rehab

Or maybe an AI Virtual Agent as a Therapist Guide and Friend

A Copy of this talk is available for all attendees. Please cite the source if you use any of the

materials from this talk.

http://www.icdvrat.reading.ac.uk/

50

Contact Information:

Albert “Skip” Rizzo, Ph.D.

Associate Director - Institute for Creative Technologies

Research Professor - Dept. of Psychiatry and School of Gerontology

University of Southern California

Los Angeles, CA., USA

arizzo@usc.edu 213-610-4737 ICT MedVR: http://medvr.ict.usc.edu/

"It would be strange, and embarrassing, if clinical psychologists, supposedly

sophisticated methodologically and quantitatively trained, were to lag behind

internal medicine, investment analysis, and factory operations control in

accepting the computer revolution."

- Paul Meehl, 1987