1
Federation of State Physician Health Program
2012 Annual Meeting
FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 2FSPHP April 23-26, 2012
THE DISRUPTIVE PHYSICIAN: THE DEVELOPMENT AND USE OF A 360 INSTRUMENT AS A MONITORING TOOL
Betsy White Williams PhD MPHAssistant Professor Director of Outcomes and ResearchOffice of Continuing Medical EducationRush University Medical CenterClinical Program DirectorProfessional Renewal CenterLawrence, KS
William H. Swiggart, MS, LPC/MHSPAssistant in MedicineVanderbilt Department of MedicineCo-DirectorCenter for Professional HealthVanderbilt University School of MedicineNashville, TN
Marine V. Ghulyan, MA
Research Analyst
The Center for Professional Health
Vanderbilt University School of Medicine
Nashville, TN
Kayci Vickers
Research Assistant
Professional Renewal Center
Lawrence, KS
Michael V. Williams, Ph.D.
Wales Behavioral Assessment
Principal
Wales Behavioral Assessment
Lawrence , KS
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 3
Learning Objectives
Understand the results of 360 degree assessment in monitoring of intervention effectiveness.
Understanding the likely changes in the results from 360 degree evaluations over time.
Understanding the interpretation of outliers in utilizing a 360 degree evaluation to determine intervention efficacy
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 4
Presentation - Context
Physicians identified as disruptive practitioners are increasing being referred to Physician Health Programs. Heretofore it has been difficult to monitor the progress of these physicians post intervention.
This presentation discusses the development of a 360 survey instrument. While we were interested in evaluating and comparing
the data between cases and a comparison sample, we were particularly interested in the use of the 360 for
both identification and monitoring. The 360° survey was recently developed based on
input from experts and a review of the literature.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 5
3 Core Competency Survey (3CC)
It is not enough to have good motives; others respond to our behavior.
Physicians are often not given essential feedback about their behavior.
The Three Core Competency Survey (3CC) is designed to provide feedback from those we work with.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 6
Disruptive behavior
“Intimidating and disruptive behaviors can foster medical errors, contribute to poor patient satisfaction and to preventable adverse outcomes, increase the cost of care, and cause qualified clinicians, administrators and managers to seek new positions in more professional environments.”
Issue 40: Behaviors that undermine a culture of safety | Joint Commissionhttp://www.jointcommission.org/assets/1/18/SEA_40.PDF
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 7
Aggressive
Anger Outbursts
Profane/Disrespectful Language
Throwing Objects
Demeaning Behavior
Physical Aggression
Sexual Comments or Harassment
Racial/Ethnic Jokes
PassiveAggressive
Derogatory comments about institution, hospital, group, etc.
Refusing to do tasks
Passive
Chronically late
Not responding to call
Inappropriate or inadequate chart notes
Spectrum of Disruptive Behaviors
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 8
Disruptive behavior
“The term “disruptive behavior” is changed in the standardsThe term “disruptive behavior” in two elements
of performance (LD.03.01.01, EPs 4 and 5) has been revised to “behavior or behaviors that undermine a culture of safety.” ”
Joint Commission online November 11, 2011 http://www.jointcommission.org/assets/1/18/
jconline_Nov_9_11.pdf
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 9FSPHP April 23-26, 2012
Disruptive Behavior Leads to Communication Problems…Communication Problems Lead To Adverse Events1
Communication breakdown factored in OR errors
50% of the time2
Communication mishaps were associated with
30% of adverse events in OBGYN3
Communication failures contributed to 91% of
adverse events involving residents4
Gerald B. Hickson, MDJames W. Pichert, PhD
Center for Patient & Professional AdvocacyVanderbilt University School of Medicine
1. Dayton et al, J Qual & Patient Saf 2007; 33:34-44. 3. White et al, Obstet Gynecol 2005; 105(5 Pt1):1031-1038.
2. Gewande et al, Surgery 2003; 133: 614-621. 4. Lingard et al, Qual Saf Health Care 2004; 13: 330-334
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 10FSPHP April 23-26, 2012
Failure to Address Disruptive Conduct Leads To
Team members may adopt disruptive person’s negative mood/anger (Dimberg & Ohman, 1996)
Lessened trust among team members can lead to lessened task performance (always monitoring disruptive person)... effects quality and patient safety (Lewicki & Bunker, 1995; Wageman, 2000)
Felps, W et al. 2006. How, when, and why bad apples spoil the barrel: negative group members and dysfunctional groups. Research and Organizational Behavior, Volume 27, 175-222.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 11FSPHP April 23-26, 2012
Disruptive Behavior Creates fear confusion or uncertainty vengeance vs. those who
oppose/oppress them hurt ego/pride grief (denial, anger,
bargaining) apathy burnout unhealthy peer pressures
ignorance (expectations, behavior standards, rules, protocols, chain of command, standards of care)
distrust of leaders dropout: early retirement
or relocation errors disruptive behavior
begets disruptive behavior
Vanderbilt University and Medical Center Policy #HR-027
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 12
“The Perfect Storm”
PhysicianHospital/Clinic
The external system The internal system
FSPHP April 23-26, 2012
Two systems interact
Good skills
Poor skills
Functional & nurturing
Dysfunctional
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 13FSPHP April 23-26, 2012
Etiologies
Institutional Factors Scapegoats System Reinforces Behavior Individual Pathology may over-shadow institutional
pathology
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 14FSPHP April 23-26, 2012
Etiologies
Personal Factors Individual pathology Life Stressors Lack of knowledge and skills
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 15FSPHP April 23-26, 2012
2004 AAMC Council of Deans
“Physicians are often poorly socialized and enter medical school with inadequate social skills for practice.”
“There is a growing body of literature documenting residency programs do notprepare resident physicians adequately for the practice of medicine.”
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 16
16FSPHP April 23-26, 2012
Potential Resources for Healthy Coping
Courses Coaches,
counselors Comprehensive
Evaluation 360° Evaluations Risk Managers Physician
Wellness Treatment
Centers
Office of General Counsel
State BME Professional Societies QI Officers EAP Others State Physician Health
Program
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 17
CME Remediation of “disruptive behavior”
Development of insight
Development of Skills
Development of implementation strategy
Feedback and monitoring
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 18
18FSPHP April 23-26, 2012
General Trends
At 3 months, significant improvements in 20 of the 22 physicians Increased motivating behaviors and motivating
impact Decreased disruptive behaviors and disruptive
impact Changes in behavior reported by “others”
Samenow CP, Swiggart W, Blackford J, Fishel T, Dodd D, Neufeld R, Spickard A. A CME Course Aimed at Addressing Disruptive Behavior. Physician Executive; 34 (1) Jan/Feb 2008: 32-40.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 19
360 Survey to Provide Feedback and Monitor Behavior
BASED on CORE COMPETENCY AREASInterpersonal and Communication SkillsProfessionalismSystem based practice
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 20
Survey Development
Review of 300+ publications of the evidence focused articles on physician professional behavior;
Abstraction of assessment items with a expert based Delphi process to yield candidates for the final scale.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 21
Survey Development
Interpersonal and Communication SkillsUses verbal communication to provide appropriate feedback to others.
Makes others feel comfortable approaching to ask questions or make suggestions.
Communicates effectively with patients.
© Williams, Swiggart, and WilliamsFSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 22
Survey Development
Professionalism Willingly performs all tasks, functions, or responsibilities that are typically expected of him/her.
Responds promptly to telephone and pages.
Reports timely to hospital/clinical duties.
Responds quickly and appropriately to administrative communications.
© Williams, Swiggart, and WilliamsFSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 23
Survey Development
System-based practice Creates a sense of teamwork and
valued contribution by team members.
His/her behavior makes others comfortable in their work.
Creates an accepting work environment.
© Williams, Swiggart, and WilliamsFSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 24FSPHP April 23-26, 2012
Importance of Monitoring
Necessity of ensuring the behavior does not recur,
Anecdotal evidence of a significant level of recidivism,
Prior behavioral issues are a significant risk factor for later disruption (Papadakis and colleagues, see for example, Papadakis, Arnold et al. 2008)
Facilitates earlier identification
25FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
Importance of Feedback
Form of guided self-assessment,Reinforces behavioral changes,Provides a standard by which to assess
gains.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 26
Methods-Subject Group
Participants in the The Course for Distressed Physicians, a remedial CME course developed at the Center for Professional Health at Vanderbilt University. Cases: referred for workplace difficulties
that relate to team behavior Comparison: Physicians of similar
specialties as the cases .
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 27
Methods-Data Analysis
Specificity and sensitivity using the two classes of participants were analyzed.
The means and distributions were analyzed for consistency with other measures of performance more consistent with process measurement.
The outcome of an analysis of outliers is reported.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 28
Reliability and Validity
Neither reliability or validity is a simple issue in the context of measures across time Consider the issue of the WAIS, while
valid and reliable, serial tests can be problematic due to lagged time effects;
In the case of “disruptive behavior”, serial measures are core to the value of the assessment instrument.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 29
Measures of Validity
Types of validity: Face validity; Construct validity; Predictive validity
In general these are summarized by Messick: “ … the degree to which the empirical evidence
and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores.”
Expanded by Kane to four domains: Scoring, generalization, extrapolation, and
interpretation/decisions.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 30
Today’s focus
While we touch briefly on our first two elements: Scoring, and, Generalization.
Most of the focus of this discussion is on: Extrapolation, and, Interpretation and decisions.
The interpretation and decisions element most clearly differentiate useful instruments from interesting academic exercises
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 31
Scoring
An extension of face validity Is the item appropriate to the construct
of interest; We selected a scale shown to be valid in
other 360 medical applications
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 32
Generalization
Convergence: Our scale(s) generate a Cronbach’s α in excess
of 0.9 the degree of exceeding depending on the item set.
Our scales, using underlying factor structures, demonstrate factor invariance across at least 3 sets of raters
The degree of coherence is clear, individual differences – the contribution to formal error – are being examined but two seem theoretically appropriate: time of remedial training and identification as disruptive.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 33
Extrapolation
Extrapolation – Do the scores predict real-world outcomes of interest, a broad restatement of predicative validity. Four groups
distressed physician class participants at Vanderbilt
Distressed physician class participants at PRC
Comparison sample at Vanderbilt Comparison sample from PRC.
.FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 34
Extrapolation
No significant different was found between the two comparison samples.
The data were pooled in subsequent analyses
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 35
Extrapolation
The 360 demonstrates known group discrimination:
The method is valid as far as discriminating between professionals identified as demonstrating behavioral issues.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 36
Extrapolation
Interprofessional Behavior
Note the reduction in mean difference across time is minimal;
However, the reduction in variance, and particularly skew is marked.
Extreme performance, particularly low performance is reduced over time.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 37FSPHP April 23-26, 2012
Disruptive behavior – Social systems
Preliminary results suggest that disruptive physicians may not differ significantly from normal physicians in mean performance but may differ significantly in skew.
37
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 38
Extrapolation
Wave analysis for inter-professional behavior
Again the change is mostly in the reduction of variance across time, not in means.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 39FSPHP April 23-26, 2012
Disruptive Behavior and Institutional Functioning
The presence of the system disruption ultimately results in breakdown:Communications;Affiliation;Roles; and,Protocols and
duties.
39
40FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
Disruptive behavior – Social systems
Results suggest that disruptive physicians may not differ significantly from normal physicians in mean performance but may differ significantly in skew.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 41FSPHP April 23-26, 2012
Disruptive behavior – Social systems
Over time as the disruptive behavior is extinguished the pattern of data will modify.
Results suggest the proportion of extreme reports falls and moderate to good reports increase.
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV
Competencies and Team Function
As outlying performance decreases team cohesion increases.
FSPHP April 23-26, 2012 42
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 43
Discussion
The instrument discriminates between participants in the remediation exercise and the comparison sample.
The instrument shows appropriate sensitivity and specificity and appears to be valid.
Analysis of outliers and serial results means and distributions appear to be
consistent with expectation means and distributions change over time
appropriately. FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 44
Discussion
The tail of the distribution appears very sensitive to behavioral
change as reported by other observers.
The 3C 360° survey is a promising measure of CME efficacy in changing practice patterns.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 45
Interpretation/Decisions
The data support the discrimination between: Physicians identified as being disruptive,
and, Physicians from comparison samples.
As well as: Physicians identified as being disruptive,
and, Physicians in a general remediation
program. These data suggest that general
interpretation is valid.FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 46
Interpretation/Decisions
The instrument is intended to address: Application within referral sources
(hospitals, clinics, academic medical centers); and,
A methodology that both tracks improvement and indicates adequate performance.
These elements are met through a core competency structure and sensitivity to changing outliers.FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 47
Discussion
The instrument provides a consistent measurement with the
literature and our experience of those areas of functioning related to interpersonal skill and communications, professionalism, and team behavior for healthcare professionals.
demonstrates appropriate sensitivity and specificity
provides the basis of effectively assessing intervention efficacy.
shows promise as a monitoring instrument and as a mean of identifying relapse behaviors.
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 48
Discussion
Sample Case Report
FSPHP April 23-26, 2012
Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 49
Discussion
Sample Comment page
FSPHP April 23-26, 2012