thesis defense - tommy may
TRANSCRIPT
Development of a Situational Judgment Test for Teamwork in
MedicineThesis Defense for Tommy May
Central Michigan University
Department of Psychology
Introduction Teamwork is vital in the medical community, a report by the
Institute of Medicine finds that the majority of accidents in medical facilities are due to a break down in teamwork (Kohn, Corrigan, & Donaldson, 1999)
Currently, teamwork is assessed using behavioral checklist (which is costly) and self report (which is not very accurate in determining abilities) (DeNisi & Shaw, 1977).
Situational Judgment Tests offer a more cost effective way to assess an individuals ability to apply knowledge of a subject (Christian, Edward, and Bradley, 2010).
The current study addresses this need for a more effective measure of teamwork through the development of a Situational Judgment Test.
Teamwork dimensions were based off of the research used to develop and implement the TeamSTEPPS training program. The teamwork dimensions are:
Communication – The sharing of task-important information with the team, including information about patient condition and medical history, actions one should be taking, has taken, or are currently taking.
Team Structure – Knowing the responsibilities of yourself and other team members and ensuring that your responsibilities are performed according to standard operation procedure.
Teamwork Dimensions
Mutual Support - Making team members aware of their mistakes/performance, helping others learn from mistakes, or appropriate reactions to errors in treatment. Helping other team members perform their physical tasks; offering social/emotional support to upset or frustrated team members.
Leadership - Providing clear direction, initiating structure, and/or taking command of a chaotic situation. It also includes appropriately handling objections or suggestions from subordinates.
Environmental Support - How cognizant team members are of the medical conditions of the patient, the supplies, resources, and current demand in the emergency department, and what actions other team members have taken.
Teamwork Dimensions, cont.
Assessment of Teamwork Assessment of Content
Skill-Based: motor or technical skills
Behavior Observation/Checklist
Affective-Based: change in attitude or motivation
Self-Report
Cognitive-Based: knowing how to acquire, apply, and organize knowledge of content
Not readily assessed in Teamwork among Medicine
Who is to be Assessed
The team as a whole
Individuals within the team
Emergency room teams frequently change
Competencies of interest are defined and frequently trained at the individual-level
(Kraiger et al., 1993)
Situational Judgment Test (SJT)
Describe a situation in which respondents must select the appropriate action to take Test knowledge of construct
Test application of knowledge
Carry greater degree of job relevance Situations are developed using real examples collected from
experienced professionals
SJT in Teamwork and Medicine SJT’s to assess Teamwork
Predict team member role performance above and beyond general mental ability (Mumford et al., 2008)
SJT’s measuring teamwork (r=.38) and leadership (r=.28) have demonstrated modest relationships with job performance (Christian et al., 2010)
SJT’s in the Medical Field
SJT’s in selection for medical school candidates predicted above and beyond general mental ability (Lievens et al., 2005 & Patel et al., 2012)
SJT’s have been found to be effective at assessing interpersonal skills (Lievens, 2013’ Lievens & Sackett, 2012; & Patterson et al., 2012)
Method Collect examples of effective and ineffective teamwork
Using an online survey, experts in emergency medicine gave examples of effective and ineffective teamwork. Describing why they felt the example was or was not effective.
Developed items based off of examples The examples provided by professionals were used to
develop situational judgment items. Items were written with the intent to fit in to one of the 5
teamwork dimensions Sort items in to teamwork dimensions
Items were sorted into the five dimensions by a 4 professionals Remove items with low agreement
Items that had low agreement were removed to ensure that each item was measuring the intended dimension
Rater Agreement
Table 1. Inter-rater Agreement
Raters Kappa1 and 2 11 and 3 0.881 and 4 0.52 and 3 0.882 and 4 0.53 and 4 0.5Average 0.71
Results indicate that rater agreement was respectable with the average Kappa at .71
Table 2. Rater AgreementItem Number Item Dimension Agreement
1 Environmental Awareness 50 %2 Environmental Awareness 100%3 Communication 100%4 Communication 100%5 Team Structure 75%6 Team Structure 75%7 Mutual Support 100%8 Team Structure 100%9 Leadership 75%10 Environmental Awareness 100%11 Leadership 100%12 Communication 100%13 Environmental Awareness 50%14 Leadership 75%15 Leadership 100%16 Communication 100%17 Team Structure 75%18 Mutual Support 100%19 Mutual Support 75%20 Mutual Support 100%
Note. Percentage is computed from 4 raters.
Development of Alternate SJT Forms
Item isomorphism was used following Lievens and Sackett (2007)
radicals are features that determine the items difficulty
incidentals are only surface level characteristics that do not impact difficulty
In item isomorphism, the radicals are the domain of the stem, the critical incident, and the context of the incident. The incidentals are the language and grammar found in the stem or response options.
Hypothesis
Hypothesis 1A: Medical students’ scores on the T-TAQ will not significantly differ from medical professionals’ scores on the T-TAQ.
Hypothesis 1B: Medical students’ scores on the developed SJT will significantly differ from medical professionals’ scores on the developed SJT. Such that medical professionals will outperform the medical students.
Hypothesis 2: The developed SJT will have weak to modest correlations with the T-TAQ.
Hypothesis 3: Form A of the developed SJT will have a strong and positive correlation with Form B.
Procedures Medical Students recruited through CMED
Given $7 Amazon gift card for their help
Medical Professionals recruited through personal contacts and SEMPA
Given $20 Amazon gift card for their help
Survey order
Form A or B
T-TAQ
Form B or A
Few demographic items
Participants 75 participants volunteered to complete survey
17 participants removed
Could not compare forms A and B
Could not compare SJT and T-TAQ scores
Could not identify level of medical experience
58 complete surveys
Sample was 51.7% male and 81% Caucasian
35 individuals were medical professionals
23 were medical students
Developing Test Key Participants rank ordered each response option from 1-least effective
to 5-most effective
Mean ranking of the professionals response options were used as the correct ranking
Would bring into account more opinions of professionals than using the mode
Deviations score was computed from professionals’ mean ranking of response options
K – R = D the absolute value of D was used as the deviation score
K = the mean ranking of professionals
R = respondents ranking
The greater an individuals deviation score the poorer they performed on the measure
0 5 10 15 20 25 30 350
10
20
30
40
50
60
70
Professional's SJT Score
Years Experience in Emergency Department
SJT
Devi
ati
on S
core
Test ReliabilityTable 3. Correlations Between Test Forms by Item.
Item Construct Correlations with Form B Item
1 Environmental Awareness 1 .682 Environmental Awareness 2 .623 Communication 1 .524 Communication 2 .695 Team Structure 1 .626 Team Structure 2 .427 Mutual Support 1 .528 Team Structure 3 .549 Leadership 1 .6210 Environmental Awareness 3 .4411 Leadership 2 .4812 Communication 3 .5913 Environmental Awareness 4 .6514 Leadership 3 .5415 Communication 4 .6216 Leadership 4 .3117 Team Structure 4 .4618 Mutual Support 2 .6019 Mutual Support 3 .6320 Mutual Support 4 .61
Note. Items in bold were later removed
Table 5. Correlations Among SJT Forms Total Scores. 1 2 3 41. Form A All Items 2. Form A Items removed
.99
3. Form B All items .87 .87 4. Form B Items removed
.87 .88 .98
The high correlation between forms (r =.88) and the great correlations between items suggest that the measure is reliable, supporting hypothesis 3.
Comparing the SJT and T-TAQTable 6. Correlations Between T-TAQ and SJT by Constructs and Total Scores. 1 2 3 4 5 6 7 8 9 10 11 12
1. Team Structure TAQ
2. Leadership TAQ .71
3. Situational Monitory TAQ
.76 .72
4. Mutual Support TAQ .43 .48 .31
5. Communication TAQ .67 .57 .54 .49
6. Team Structure SJT -.05 -.15 -.14 -.27 -.17
7. Leadership SJT .00 -.10 -.03 -.14 -.04 .51
8. Environmental Awareness SJT
-.12 -.26 -.19 -.41 -.28 .63 .39
9. Mutual Support SJT -.19 -.20 -.15 -.33 -.36 .46 .40 .50
10. Communication SJT -.11 -.08 -.18 -.31 -.13 .48 .39 .44 .35
11. TAQ total .88 .87 .84 .69 .82 -.19 -.11 -.31 -.32 -.21
12. SJT total -.12 -.22 -.19 -.39 -.26 .84 .69 .82 .84 .71 -.30 Results support hypothesis 2
Performance on the SJT and T-TAQ
Table 7. Means and Standard Deviation for T-TAQ Scores Between Professionals and Students. Medical
StudentsMedical
Professionals
Mean SD Mean SD Effect SizeTeam Structure TAQ 4.38 .33 4.26 .67 -.21Leadership TAQ 4.45 .38 4.56 .59 .21Situational Monitory TAQ
4.22 .53 4.41 .61 .33
Mutual Support TAQ 4.22 .51 4.35 .54 .25Communication TAQ 4.10 .51 4.16 .59 .11T-TAQ Total 21.24 1.59 21.72 2.67 .21
To test hypothesis 1A, a one-tailed t-test demonstrated that the total score on the T-TAQ between medical students (M= 21.24 SD=1.59) and medical professionals (M=21.71 SD=2.67) was not significant, t(50)= -.74, p= .233. Results support hypothesis 1A.
Table 8. Means and Standard Deviation for SJT Scores Between Professionals and Students.
Item Medical Students Medical Professionals Effect Size 95% Confidence Interval
Mean SD Mean SD
1 5.44 1.65 4.56 1.29 -.61 -1.15 -.072 3.55 2.20 2.94 1.73 -.32 -.85 .213 2.96 1.91 1.65 1.07 -.9 -1.45 -.355 4.81 2.18 3.54 1.66 -.68 -1.22 -.136 3.21 1.89 2.35 1.09 -.59 -1.13 -.057 3.77 1.00 3.47 1.41 -.28 -.77 .298 5.64 1.95 4.15 1.10 -1.00 -1.55 -.449 3.85 2.16 3.07 1.62 -.42 -.95 .1110 3.95 2.68 2.62 1.21 -.69 -1.23 -.1511 3.75 1.64 3.23 1.04 -.4 -.93 .1313 4.61 2.22 2.73 1.24 -1.11 -1.67 -.5514 2.93 1.54 1.94 1.00 -.8 -1.34 -.2515 4.22 1.82 3.41 1.36 -.52 -1.05 .0116 3.48 2.69 1.90 1.07 -1.12 -1.68 -.5517 3.93 1.71 2.81 1.34 -.74 -1.29 -.2119 4.33 1.92 2.83 0.90 -1.08 -1.64 -.5220 2.91 1.57 2.45 1.17 -.34 -.87 .19Total 67.34 17.36 49.65 4.88 -1.54 -2.13 -.94
Table 9. Means and Standard Deviations for Total and Average Scores for the SJT and T-TAQ
Medical Students Medical Professionals
Means SD Means SD
SJT Total 67.34 17.36 49.65 4.88
SJT Average 3.96 1.02 2.92 .29
T-TAQ Total 21.24 1.59 21.72 .32
T-TAQ Average 4.25 2.67 4.34 .53
A one-tailed t-test demonstrates that medical students had a significantly higher deviation score (M=67.34 SD= 17.36) on the SJT than medical professionals (M=49.65 SD=4.88), t(56)=5.72, p< .01, supporting hypothesis 1B.
SJT Measure0
10
20
30
40
50
60
70
80
SJT Performance
Students Professionals
Limitations
Low sample size, leaves the study vulnerable to error and extraneous variables
Representativeness of the sample
All medical students were selected from one university
Would other medical professionals (i.e. nurses, anesthesiologist, etc.) respond in the same way?
A more thorough validation
Using teamwork training
Concurrent validity study
Predicting success in teamwork training courses
Discussion
The T-TAQ is best used for assessing team members attitudes towards teamwork when team members have little reason to distort their
responses.
SJT assess teamwork knowledge, students and professionals differ with regards to medical knowledge and teamwork knowledge.
Further validation of the SJT measure is needed, ideally testing a group of medical students before and after teamwork training would
be a great start.
With the measure further validated it could be used in assessing training needs in medical facilities, in employee hiring for medical professionals, and even selecting medical students into a program.
Questions?
Thank you
A big thank you to the many professionals and students who dedicated their time and energy to helping complete this project.
Specifically:
Dr. Matthew Prewett, Committee Chair & Advisor
Dr. Neil Christiansen and Dr. Steven Vance, Committee Members
Spencer Thronock, CMED student
Most importantly
Chelsea (my wife), Zach, & Harvey (my boys), for motivation