measuring consumer perception of care challenges & opportunities john bartlett, m.d.,m.p.h....
TRANSCRIPT
![Page 1: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/1.jpg)
Measuring Consumer Perception of Care
Challenges & Opportunities
John Bartlett, M.D.,M.P.H.March 20-21 Meeting with California’s Division of Alcohol and Drug Programs
![Page 2: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/2.jpg)
Goals for Presentation Review consumer perception of care as a
concept
Review success criteria for its measurement
Review the development & testing of CSAT’s Modular Survey
![Page 3: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/3.jpg)
The Concept
Measuring consumer response =
core business function In healthcare tied to growth of consumerism
& CQI initiatives A NOMS domain Consumer perception of care ≠ satisfaction
![Page 4: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/4.jpg)
Measuring Consumer Perception of Care
Approaches to measurement differ
depending on scope & purpose For purposes of comparability, improvement
over time, & benchmarking measures must be :
Meaningful Scientifically-sound Actionable
![Page 5: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/5.jpg)
The Problems with Satisfaction
No evidence linking the measurement of
satisfaction to client outcomes Few satisfaction surveys scientifically
validated Data is not actionable (ceiling effect)
![Page 6: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/6.jpg)
The Modular Survey
SAMHSA –supported initiative Conducted under the auspices of the Forum
on Performance Measurement &
the Washington Circle Conducted in 2 phases
Phase 1 in conjunction with mental health Rx Phase 2 substance abuse – specific
![Page 7: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/7.jpg)
Design Requirements
Phase 1 focus on commonality,
not comprehensiveness Short Scientifically sound Actionable Use of existing, widely-used, non-proprietary
surveys Consensus-driven
![Page 8: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/8.jpg)
“Field-level”CommonMeasures
Adult Common Measures
Child/AdolescentCommon Measures
AdultMental Health
Core Measures
Adult Substance
Abuse Core Measures
AdolSubstance
AbuseCore Measures
C/AdolMental HealthCore Measures
Common DesignTemplate
Modular Survey Flow of Common Questions for Individual Respondent
![Page 9: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/9.jpg)
Approach to Phase 1
4 workgroups to develop consensus Selection of instruments Identification of concerns Identification of potential items Ranking of items Final item selection (modified Delphi) Pilot testing
![Page 10: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/10.jpg)
Phase 1 Pilot Testing
Conducted during summer/fall 2004 Primary data collection in Cincinnati
United Way agencies (N = 1157) Secondary analysis using MHSIP data
(16 state & LA County data sets) Final N > 22,000 respondents Pool of items reduced from 28 to 11 All items common to both fields, both populations
![Page 11: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/11.jpg)
Approach to Phase 2
Stand-alone SUD Rx initiative Under Washington Circle with Forum as
“subcontractor” New item development (no existing SUD survey) Content work group co-chaired by
Tom McLellan (TRI) & Doreen Cavanaugh (Georgetown)
Support from Forum Methods Work Group
& Ann Doucette (George Washington Public Provider & Consumer Advisory Groups
![Page 12: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/12.jpg)
Phase 2 SUD Initiative
Closely coordinated with NOMS Identification of concerns
Relationship to treatment program Self awareness of problem/commitment to
change Perceived outcomes Social connectedness
Generation of items (35 in testing pool)
![Page 13: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/13.jpg)
Phase 2 Pilot Testing OMB & IRB approval spring 2006 Conducted in 3 rounds
Round 1 – Adult & Adolescent (summer 2006) 14 programs, N = 1207
Round 2 – Adult & Adolescent (winter 2006 – 07) 6 programs, N = 585
Round 3 – Adolescent (spring-summer 2007) 8 programs, N = 268
Final adult N = 1549 (2 samples) Final adolescent N = 492 (1 sample) All demographic groups covered except Native American
![Page 14: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/14.jpg)
Phase 2 Completion
Analysis & recommendations by Ann Doucette PH.D. Use of IRT For a copy of the technical report, e-mail
[email protected] Review by Forum Methods Work Group
(November 2007) Review & Approval by SUD Content Committee
(November 2007)
![Page 15: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/15.jpg)
Final SUD Modular Survey
21 items (11 from Phase 1, 10 from Phase 2) Quality – 6 items Perceived Outcomes – 6 items Social Connectedness – 7 items Commitment to Change – 2 items
10 demographic & background items Spanish translation available
![Page 16: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/16.jpg)
Modular Survey Flow of Common Questions for Individual RespondentFinal Version
Phase 1Common Items *
Phase 2MentalHealth Items
Phase 2SUD
Items
Common DesignTemplate
* All populations, all fields
![Page 17: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/17.jpg)
Convergence with NOMS
In Spring 2006 NOMS Technical Consulting Group convened
Recommended 17 items from 8 different instruments
9 of the 17 from the Modular Survey 5 are in the Final Modular Survey
![Page 18: Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and](https://reader030.vdocuments.site/reader030/viewer/2022032722/56649f4a5503460f94c6c210/html5/thumbnails/18.jpg)
In summary……
Consumer perception of care key measurement domain
Its measurement must meet certain criteria in order to be worth the effort
The Modular Survey is the only current instrument measuring consumer perception of care that is:
SUD Rx – specific Product of both consensus and empirical
analysis Short and actionable