creation of the simulator value index tool adapted from workshop on 4.21.14 presented by american...

Creation of the Simulator Value Index ToolAdapted from workshop on 4.21.14 presented byAmerican College of Surgeons Accreditation Education Institutes, Technologies & Simulation Committee)

Deborah Rooney PhD

James Cooke MD

Yuri Millo MD

David Hananel

MEDICAL SCHOOLUNIVERSITY OF MICHIGAN

Disclosures

o David Hananel, No Disclosures

o Yuri Millo, No Disclosures

o James Cooke, No Disclosures

o Deborah Rooney, No Disclosures

Overview of Main Topics

o Introduction of project

o Overview of 2014 IMSH Survey results

o Summary of 2014 ACS Consortium results

o Working meeting to refine the AVI algorithm

o Apply AVI algorithm in group exercise

o Discuss next steps

Introduction: How it all started

o ACS AEI, Technologies and Simulation Committeeo Guidelines for Simulation Development

(Millo, George, Seymour and Smith)

o University of Michigano Need to support faculty in sim purchase/decision-making

process(Cooke)

o Discourse o Definition of “value”o Differences across stakeholder role

(institution, administration, clinician, educator, researcher...)

Introduction: How it all started

o Reached consensus on factors used when considering a simulator purchaseo Survey 1

o IMSH general membership, N=2800o January, 2014

o Workshop 1, n=16o IMSH, January, 2014

o Survey 2o ACS AEI Consortium membership, N = 455o March, 2014

o Workshop 2, n = ?o ACS AEI-March, 2014

Introduction: The Instrument

o Began with 31-item survey accessed via www (Qualtrics)

o 4-point rating scale o (1 = not considered/not important 4= critical to me when I

consider a simulator purchase)o 6 Domains

o Cost, Impact, Manufacturer, Utility, Assessment, Environment/Ergonomics)

o Demographicso Country/Institutiono Stakeholder role o Involvement o Follow-up

44

5

2

1= Grenada

1= Chile1= Peru

1=Czech Republic

2

2=Singapore

3 = NewZealand

11

95 total respondents, 72 individuals completed survey approximately 2+% of IMSH membership (2,800), 7 undesignated/16 incomplete

IMSH Survey Sample: 67 institutions x 12 Countries

6

1

1

3

4

4

1 1

1= Massachusetts3 = Rhode Island1 = New Jersey

1

3

12

1

3

1

1

1

1

1

50 participants from US

IMSH Survey Sample: 44 institutions x 22 States/US

Acad./U

niv. H

ospita

l

Medica

l Sch

ool

Teachin

g Hosp

ital

Healthca

re S

yste

m

Comm

ercia

l Skil

ls Cente

r

Gov.

/Milit

ary C

enter

Indust

ry05

101520253035404550

4658%

2835% 26

33% 2025%

68%

45%

11%

n = 791 undesignated

IMSH Survey Sample: Institution Affiliation

o Costo Commercial Skills Centers (CSC) rated C1 (Purchase cost) lower than

each of the other institutions, p = .001.

o Manufacturero CSCs rated M1 (Reputation of manufacturer) lower than each of the other

institutions, p = .001.

o Utilityo CSCs rated U3 (Ease of data management) and o U11 (portability) lower than each of the other institutions, p = .001.

o Ergonomicso Medical Schools rated item E2 (Ergonomic risk factor) much higher thank

other institutions), p = .05. CSCs rated E3 (Ease of ergonomic setup) lower than each of the other

institutions, p = .001.

IMSH Survey Results: Rating Differences by Institutional Affiliation

AdministrationClinicianInstitutionTechnicianEducator

3139%

1924%

79%

810%

1418%

n = 791=undesignated

IMSH Survey Sample: Stakeholder Role

o Costo Clinicians rated C2 (Cost of warranty) lower than the

other stakeholders, p = .048.

o Utilityo Clinicians rated U11 (portability of simulator) higher than

other stakeholders, p = .037.

IMSH Survey Results: Rating Differences by Stakeholder Role

Contribute to DecisionLead/ResponsibleApproval ProcessNot Involved

3746%37

46%

45%

23%

n = 80

IMSH Survey Sample: Involvement in Decision

o Although there are no differences across level of involvement,

o There are different considerations during simulator purchasing process across;o Country o Institutional affiliation (commercial skills center may

have unique needs)o Stakeholder role (Clinicians may have unique needs)

o Keeping this in mind, let’s review the top factors considered

IMSH Survey Results: Summary

Average Factor (survey item number, item description) Domain

3.8 21- Technical stability/reliability of simulator Utility3.7 10- Customer service Manufacturer

3.4 16- Ease use for instructor/administrator Utility

3.4 19- Ease of use for learner Utility

3.3 6- Relevance of metrics to real life/clinical setting  Impact

3.2 11- Ease of delivery and installation, orientation to sim Manufacturer

3.2 26- Reproducibility of task/scenario/curriculum Assmnt/Res

3.2 1- Purchase cost of simulator Cost

3.2 9- Reputation of manufacturer Manufacturer

3.1 8- Scalability Impact

3.1 20- Quality of tutoring/feedback from sim to learners Utility

3.1 7- Number of learners impacted Impact

3.0 2- Cost of warranty Cost

3.0 3- Cost of maintenance Cost

3.0 17- Ease of configuration/authoring sim's learning management system Utility

- Physical durability Utility

The SVI Factors: Top 15+1 Factors Ranked

ACS Consortium Survey: Introduction

o Identical Survey items, ratingso Added durability of simulator questiono 31 32-item survey accessed www (Qualtrics)o 4-point rating scale

o (1 = not considered/not important 4= critical to me when I consider a simulator purchase)

o 6 Domainso Cost, Impact, Manufacturer, Utility, Assessment,

Environment/Ergonomics)o Demographics

o Country/Institutiono Stakeholder role o Involvement o Follow-up

49

1

2

1=UK1=France1=Italy

1

65 total respondents, 54 individuals completed survey approximately 12% of ACS membership (455), 2 undesignated

ACS Survey Sample: 41 institutions x 7 Countries

1=Greece

1=Sweden

8

1

1

3

5

4

1 1

8 = Massachusetts1 = Rhode Island1=Delaware1 = Maryland

1

1

44

1

3

2

1

2

1

3

49 participants from US47 indicated institution

ACS Survey Sample: 36 institutions x 17 States/US

1

1

1

Acade

mic

(Univ

ersit

y) H

ospit

al

Teach

ing H

ospit

al

Med

ical S

choo

l

Health

Car

e Deli

very

Sys

tem

Gover

nmen

t or M

ilitar

y Cen

ter

Comm

ercia

l Skil

ls Cen

ter

Indu

stry

0

5

10

15

20

25

30

35

40

3767%

2851% 24

44%

1629%

24% 0

0%

n = 55

ACS Survey Sample: Institution Affiliation

00%

InstitutionAdministrationClinicianTechnicianEducatorResearcherCoordinator

916%

2748%

1<2%

3<6%

1323%

n = 56

ACS Survey Sample: Stakeholder Role

2<4%1

<2%

Contribute to Decision

Lead/Responsible

Approval Process

Not Involved

2545%29

52%

23%

n = 56

ACS Survey Sample: Involvement in Decision

o Although there are no differences across;o institutiono stakeholder role

o There are different considerations during simulator purchasing process across;o Level of involvemento (Self-reported “Responsible” folks are more concerned

about number of learners impacted and Scalability)

ACS Survey Results: Summary

But are there differences across

IMSH and ACS membership?

ACS Survey Results: Summary

1. C1

4. C4

7. I2 10. M2

13. U2

16. U5

19. U8

22. U11

25. A1

28. A4

31. E2

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1. IMSH2. ACS

Aver

age

Obs

erva

tion

Survey Results: IMSH v. ACS

4 (C2)

7 (I2) 11 (M3) 15 (U4) 22 (U11)

o Costo ACS members rated C2 (Cost of warranty) higher

than the IMH members, bias = .40, p = .04.

o Impacto ACS members rated I2 (Number of learners) higher than other

stakeholders, bias = .53, p = .01.

o Utilityo ACS members rated U4 (Ease of report generation) higher than

the IMH members, bias = .43, p = .02.o ACS members rated U11 (Portability of simulator) higher than

other stakeholders, bias = .48, p = .01.

Survey Results: Rating Differences by Conference

The SVI Factors: Top 15+1 Factors Ranked

Applying the SVI Toolo General impressions? What stood out?

o What worked well?

o What could have gone better?

o Any surprises?

o Usefulness? How might you use the SVI Tool at your institution?

o Please complete the questions on “Feedback” Tab on the SVI Worksheet

Thank you: Our Contact Information

o Deb Rooney University of Michigandmrooney@med.umich.edu

o Jim Cooke University of Michigano cookej@med.umich.edu

o David Hananel SimPORTAL & CRESTUniversity of Minnesota Medical Schoolo dhananel@umn.edu

o Yuri Millo Millo Groupyuri.millo@millo-group.com

o Olivier Petinaux ACS American College of Surgeon, Division of Education opetinaux@facs.org