fy 2019 quality management annual report · 2020-01-27 · quality management annual report fy 2019...
Post on 03-Aug-2020
3 Views
Preview:
TRANSCRIPT
January 10, 2020
FY 2019
Quality Management
Annual Report
Quality Management Annual Report FY 2019
1 | P a g e
Table of Contents
Table Listing ...............................................................................................................................................4
Figure Listing .............................................................................................................................................5
Section 1: Executive Summary ....................................................................................................................7
Reviews Completed by Fiscal Year ................................................................................................................ 7
FY 2019 Provider Performance ..................................................................................................................... 8
FY 2019 Quality Management Activities ...................................................................................................... 9
FY 2019 Key Findings .................................................................................................................................. 13
FY 2020 Areas of Focus ............................................................................................................................... 18
In Conclusion ............................................................................................................................................... 21
Section 2: Introduction ............................................................................................................................. 23
Section 3: Behavioral Health Quality Reviews ........................................................................................... 25
Background ...................................................................................................................................................... 25
Sampling Method ............................................................................................................................................. 26
Individual Records and Billing Review ........................................................................................................ 26
Individual and Staff Interviews ................................................................................................................... 29
BHQR Overall Review Scores ............................................................................................................................ 30
BHQR Overall Scores ................................................................................................................................... 30
BHQR Overall Scores by Tier and Provider Size ........................................................................................... 32
BHQR Billing Validation .............................................................................................................................. 35
BHQR Assessment and Planning ................................................................................................................. 44
BHQR Service Guidelines............................................................................................................................. 47
Office of Deaf Services Special Reviews ...................................................................................................... 55
BHQR Programmatic (Non-scored) ............................................................................................................. 57
BHQR Focused Outcome Areas (FOA) ......................................................................................................... 59
BHQR Staff and Individual Interviews ......................................................................................................... 64
Assertive Community Treatment (ACT) Quality Reviews ................................................................................. 65
ACT Quality Review Overall Scores ............................................................................................................. 66
ACT Billing Validation ................................................................................................................................. 68
ACT Assessment and Planning .................................................................................................................... 71
ACT Service Guidelines ................................................................................................................................ 72
ACT Focused Outcome Areas ...................................................................................................................... 75
Crisis Stabilization Unit Quality Reviews .......................................................................................................... 76
Crisis Stabilization Unit Sampling Method ................................................................................................. 77
Crisis Stabilization Unit Quality Review Overall Scores .............................................................................. 79
Quality Management Annual Report FY 2019
2 | P a g e
CSUQR Individual Record Review (IRR) ....................................................................................................... 80
CSUQR Service Guidelines ........................................................................................................................... 87
CSU Focused Outcome Area ....................................................................................................................... 90
CSUQR Staff Interview and Individual Interview ......................................................................................... 94
New Provider Reviews ...................................................................................................................................... 96
Reassessment Frequency Reviews ................................................................................................................. 101
BHQR Reassessment Review Findings ...................................................................................................... 101
CSU Reassessment Review Findings ......................................................................................................... 107
Technical Assistance/Exit Conference ............................................................................................................ 108
Summary of Findings and Recommendations for Behavioral Health Providers ............................................ 109
Provider Performance ............................................................................................................................... 110
FY 2019 BH Accomplishments .................................................................................................................. 113
BH Systems Strengths and Recommendations for Improvement ............................................................. 115
Section 4: Intellectual and Developmental Disabilities ............................................................................ 121
Background .................................................................................................................................................... 121
Sampling Method ........................................................................................................................................... 122
Person Centered Review (PCR) .................................................................................................................. 122
Quality Enhancement Provider Review (QEPR) ........................................................................................ 123
Review Processes ........................................................................................................................................... 124
PCR and QEPR ........................................................................................................................................... 124
Quality Technical Assistance Consultation (QTAC) .................................................................................... 126
Person Centered Review ................................................................................................................................ 127
PCR Scores by Tool .................................................................................................................................... 127
PCR Scores by Focused Outcome Area (FOA) ............................................................................................ 128
Low-Scoring Indicators by FOA (PCR) and Opportunities for Improvement ............................................. 129
PCR Scores by Tool and Focused Outcome Area (FOA) ............................................................................. 143
Low-Scoring FOAs by Perspective (Tool) ................................................................................................... 144
FOAs by Demographics ............................................................................................................................. 146
Individual Service Plan Quality Assurance Checklist (ISP QA) ................................................................... 149
ISP Expectations ........................................................................................................................................ 152
Results Using Original ISP QA Checklist .................................................................................................... 152
Results Using New ISP QA Checklist .......................................................................................................... 154
PCR Results by Service .............................................................................................................................. 155
PCR Strengths and Recommendations ..................................................................................................... 157
Quality Enhancement Provider Review (QEPR) ............................................................................................. 158
QEPR Scores by Size .................................................................................................................................. 158
Overall QEPR Score by Tool and Year ........................................................................................................ 159
Overall Crisis Provider Scores.................................................................................................................... 161
Quality Management Annual Report FY 2019
3 | P a g e
Qualifications and Training (Q & T) and Service Specific (DDSS) .............................................................. 161
QEPR Scores by Provider Size .................................................................................................................... 163
QEPR Scores by Focused Outcome Area (FOA) ......................................................................................... 164
QEPR Scores by FOA and Tool ................................................................................................................... 165
QEPR Scores by FOA and Provider Size ..................................................................................................... 167
Quality and Technical Assistance Consultation (QTAC) .................................................................................. 168
Intellectual and Developmental Disability Summary of Findings and Recommendations ............................ 170
FY 2019 IDD Accomplishments ................................................................................................................. 170
IDD System Strengths and Recommendations ......................................................................................... 173
Section 5: Feedback Survey Results ........................................................................................................ 183
Section 6: Conclusion ............................................................................................................................. 185
Appendix A: Abbreviations and Acronyms .............................................................................................. 189
Appendix B: Score Distributions ............................................................................................................. 191
Quality Management Annual Report FY 2019
4 | P a g e
Table Listing (Links available by ctrl + clicking on table name)
Reviews Completed by Fiscal Year
Table 1. BHQR Overall Scores by Tier
Table 2. BHQR Category Scores by Provider Size
Table 3. FY 2019 Amount Reviewed by Funding Source
Table 4. Non-Intensive Outpatient Services and Percent of Funds Justified
Table 5. Specialty Services and Percent of Funds Justified
Table 6. BHQR Billing Discrepancy Reasons by Fiscal Year
Table 7. BHQR Assessment & Planning Indicator Scores by Fiscal Year
Table 8. BHQR Service Guidelines Scores by Service Type
Table 9. FY 2019 Lowest Scoring Indicator(s) by Service
Table 10. Progress Note Indicator Results by Year and Service Type
Table 11. Office of Deaf Service Specialty Review Results
Table 12. BHQR and ACT Results by Category by Year
Table 13. ACT BHQR Billing Discrepancy Reasons by Year
Table 14. ACT Service Guidelines Scores
Table 15. Lowest Scoring Individual Record Review Indicators by Category
Table 16. Highest-Scoring Individual Record Review Indicators by Category
Table 17. FY 2019 Review Results by Category of New Providers
Table 18. Intensive Family Intervention Results by Indicator for New Providers
Table 19. FY 2019 Review Results by Category of Reassessed Providers
Table 20. FY 2019 Technical Assistance/Exit Conference Details
Table 21. BHQR, ACT, CSUQR Overall Averages by Year
Table 22. BHQR and CSUQR FOA Scores
Table 23. FY 2019 Reassessment Review Results …
Table 24. FY 2019 PCR Sample by Region
Table 25. QEPR Provider Sample by Size and by Year
Table 26. FY 2019 Number of Records by Review Tool and Review Type
Table 27. Low-Scoring Whole Health Indicators (PCR)
Table 28. Low-Scoring Safety Indicators (PCR)
Table 29. Low-Scoring Person Centered Practices Indicators (PCR)
Table 30. Low-Scoring Community Life Interview Indicators (PCR)
Table 31. Low-Scoring Community Life Record Review Indicators (PCR)
Table 32. Low-Scoring Choice Indicators (PCR)
Table 33. Low-Scoring Rights Indicators (PCR)
Quality Management Annual Report FY 2019
5 | P a g e
Table 34. ISP QA Life Indicator by Region and Year
Table 35. FY 2019 ISP QA Checklist Ratings by Expectation
Table 36. Average Rating by Expectation (0 - 4) by Year
Table 37. FY 2019 “New” ISP QA Checklist Ratings
Table 38. PCR PRR Results by Service and Year
Table 39. FY 2019 PCR Record Review Results by FOA and Service
Table 40. FY 2019 Top Strengths Identified During a PCR
Table 41. FY 2019 Top Recommendations Identified During a PCR
Table 42. FY 2019 FY 2019 Qualifications and Training Low-Scoring Indicators
Table 43. QTACs by Referral Source and Type
Table 44. QTAC Referral Reasons: FY 2018
Table 45. FY 2017 Summary by Tool and Review Type
Table 46. The Collaborative Provider Feedback Surveys
Figure Listing (Links available by ctrl + clicking on figure name)
FY 2019 Provider Performance
Figure 1. FY 2019 Diagnostic Categories Reviewed
Figure 2. BHQR Fiscal Year Results by Category
Figure 3. Overall Score by Provider Size by Year
Figure 4. Billing Score by Provider Size by Year
Figure 5. BHQR Billing Validation Amount Reviewed by Year Figure 6. BHQR Amount Reviewed by Funding Source and Year
Figure 7. FY 2019 Percent of Discrepancy Reasons
Figure 8. FY 2019 Percent of Dollars Reviewed by Non-Intensive Outpatient Service
Figure 9. FY 2019 Percent of Dollars Reviewed by Specialty Service Figure 10. BHQR Focused Outcome Areas Scores by Year
Figure 11. BHQR Focused Outcome Area by Year – Whole Health
Figure 12. BHQR Focused Outcome Area by Year – Safety
Figure 13. BHQR Focused Outcome Area by Year – Person Centered Practices
Figure 14. BHQR Focused Outcome Area by Year – Community Life
Figure 15. BHQR Focused Outcome Area by Year - Choice
Figure 16. BHQR Focused Outcome Area by Year - Rights
Figure 17. BHQR Interview Data by Year Figure 18. ACT Fiscal Year Results by Category and Year
Figure 19. ACT Billing Validation by Year
Figure 20. ACT Assessment and Treatment Planning Scores by Year
Quality Management Annual Report FY 2019
6 | P a g e
Figure 21. ACT FOA Scores by Category by Year
Figure 22. FY 2019 Diagnostic Categories Observed by Review Type
Figure 23. CSU Overall Scores by Year
Figure 24. Individual Record Review Category Scores
Figure 25. CSUQR Service Guidelines Staffing Adherence Indicators by Year
Figure 26. CSUQR Service Guidelines Policy Adherence Indicators by Year
Figure 27. CSUQR FOA Results by Year
Figure 28. CSUQR Individual and Staff Interview Results by Year
Figure 29. FY 2019 Overall and Billing Scores by New Provider
Figure 30. FY 2019 BHQR Overall Score Distribution of Reassessed Providers
Figure 31. FY 2019 BHQR Billing Score Distribution of Reassessed Providers
Figure 32. Overall Score Results by Reassessed CSU
Figure 33. PCR Scores by Tool and Year
Figure 34. PCR Scores by FOA and Year
Figure 35. FY 2019 PCR Scores by Tool and FOA
Figure 36. FY 2019 PCR Scores by FOA and Region
Figure 37. FY 2019 PCR Scores by FOA and Residential Setting
Figure 38. FY 2019 PCR Scores by FOA and Funding Source
Figure 39. PCR ISP QA Life Indicator by Year
Figure 40. FY 2016 - FY 2019 QEPR Samples by Provider Size and Type
Figure 41. Overall QEPR Scores by Tool and Year
Figure 42. Overall Crisis Provider Scores by Tool and Year
Figure 43. QEPR Qualifications and Training and DDSS Results by Year
Figure 44. Overall and Qualifications and Training Score by Provider Size and Review Year
Figure 45. Overall QEPR Scores by FOA and Year
Figure 46. FY 2019 QEPR Scores by Tool and Focus Outcome Area
Figure 47. FY 2019 PRR Low-Scoring Indicators - QEPR
Figure 48. FY 2019 QEPR Provider Scores by FOA and Size
Figure 49. FY 2019 QTAC Technical Assistance Provided
Quality Management Annual Report FY 2019
7 | P a g e
Section 1: Executive Summary
Since FY 2016, the Georgia Collaborative Administrative Services Organization (the Collaborative) has
assisted the Georgia Department of Behavioral Health & Developmental Disabilities (DBHDD) in its
management of services and supports for individuals receiving Community Behavioral Health and
Rehabilitation Services (CBHRS), New Options Waiver (NOW), Comprehensive Supports Waiver
(COMP), and state funded behavioral health (BH) and intellectual and developmental disabilities
(IDD) services. DBHDD delegates BH, Crisis Stabilization Unit (CSU), and IDD quality reviews to the
Collaborative. June 30, 2019 marked the Collaborative’s Quality Management Department’s fourth
year of the contract with DBHDD. See the table below for the number of BH and IDD quality reviews
and interviews conducted.
Reviews Completed by Fiscal Year
Fiscal Year Number of Reviews Individuals Interviewed Staff Interviewed
BH/CSU* IDD** BH/CSU IDD BH/CSU IDD
2016 177 584 643 1,151 657 1,186
2017 208 581 735 921 774 702
2018 197 584 616 920 652 759
2019 201 584 780 1,228 832 888
Total 783 2,333 2,774 4,220 2,915 3,535
*The additional Assertive Community Treatment (ACT) Quality Reviews are included in the BH/CSU totals (FY17: 20;
FY18: 21; FY19: 20). Calculations specific to the ACT Quality Reviews are in a separate section of the report.
** Crisis providers are included in the total number of reviews, but are not included in the figures/graphs below
(FY16: 1; FY17: 4; FY18: 1). Separate calculations of Quality Enhancement Provider Reviews (QEPR) and Person
Centered Reviews (PCR) are included below. A separate QEPR calculation for crisis providers can be found
throughout the report.
Quality Management Annual Report FY 2019
8 | P a g e
FY 2019 Provider Performance
Behavioral Health Quality Reviews Crisis Stabilization Unit Quality Reviews
IDD: Quality Enhancement Provider Reviews1
1 The QEPR sample is designed to ensure all providers are reviewed at least once every three to four years.
Overall Score
FY 17: 83.7% FY18: 84.8%FY19: 86.2%
Individual Interview
FY17: 90.7%FY18: 89.6%FY19: 92.3%
Staff Interview
FY17: 95.2%FY18: 95.3%FY19: 95.7%
Observation
FY17: 95.7%FY18: 96.0%FY19: 97.2%
Record Review
FY17: 69.6%FY18: 72.4%FY19: 73.2%
Qualifications and Training
FY17: 82.6% FY18: 84.6% FY19: 83.2%
DD Service Specific
FY17: 99.8% FY18: 99.8% FY19: 99.8%
Quality Management Annual Report FY 2019
9 | P a g e
IDD: Person Centered Reviews by Focused Outcome Area and Year
FY 2019 Quality Management Activities
Based on Fiscal Year (FY) 2018 analysis and dialogue with DBHDD, the following areas of focus were
determined for FY 2019:
Behavioral Health
Based on FY 2018 findings statewide, the minimum score threshold for the Behavioral Health
Quality Reviews (BHQR) and Crisis Stabilization Unit Quality Reviews (CSUQR) was increased
to 90 percent for both Billing (not applicable to CSUQR) and Overall scores. Any provider not
reaching the 90 percent threshold is scheduled for a second review (reassessment). Thus,
behavioral health Quality Assessors continue to assist the network in reaching or surpassing
the threshold through technical assistance, education, and various training methods and
materials.
Rights ChoiceCommunity
Life
PersonCenteredPractices
SafetyWhole Health
FY 2017 (N = 481) 90.2% 78.7% 71.7% 82.6% 90.3% 84.7%
FY 2018 (N = 484) 92.0% 82.3% 76.9% 84.4% 91.6% 87.4%
FY 2019 (N = 484) 93.4% 80.9% 77.3% 85.2% 92.5% 86.3%
0%
25%
50%
75%
100%
Quality Management Annual Report FY 2019
10 | P a g e
Behavioral Health Quality Review (BHQR) and Crisis Stabilization Unit Quality Review
(CSUQR) annual tool revisions were completed throughout FY 2019 with several additions for
clarity, alignment, and formalized criteria including:
o Addition of Substance Abuse Intensive Outpatient Program as a measurable service
o Inclusion of a non-scored category titled Programmatic, which measures various
program needs based on service type
o Differentiation between Peer Support Whole Health and Wellness - Individual versus
Group services
o Inclusion of a billing discrepancy entitled “No valid, verified diagnosis on date of
service provided”
o Addition of question/indicator detail to include any reason(s) an indicator(s) was not
met
The Collaborative met contractual requirements and conducted the FY 2019 Quality Training
Series calendar events related to BH and CSUs. The topics presented were preapproved by
DBHDD.
Successful collaboration occurred with the Collaborative’s Quality department in developing
Final and Exit summaries and assessments specific to the IDD review process for public
website viewing. This development occurred simultaneously with the alignment of IDD tool
revision changes and needs with categories reviewed within the BHQRs/CSUQRs.
Completion of the DBHDD requested Office of Deaf Services Specialty Review.
An additional forty percent of reassessments were completed in FY 2019 (N = 28) as
compared to FY 2018 (N = 20), due to the increased threshold for Overall and Billing scores to
90 percent from 85 percent with supplemental technical assistance provided by Assessors.
Quality Management met all quality review reporting contractual requirements as set forth
by DBHDD including but not limited to; quarterly provider score reporting, semi-annual
review data and score presentation, Medicaid discrepancy reporting, as well as any ad hoc
requests made by DBHDD.
Quality Management Annual Report FY 2019
11 | P a g e
Quality Management conducted various ad hoc reporting and data analyses which were
provided to DBHDD and included the following:
o Sample size Power Analysis (BHQR/CSUQR/ACT/ICM)
o Supported Employment Study (IDD Specific but included BH collaboration)
o Intensive Case Management (ICM) Analysis
o Individual/Staff Interview Analysis
o Yale RSA-R Report
o Intensive Family Intervention (IFI) Review/Claim Analysis
o Office of Deaf Services Review pilot
o Suicide Study proposal and sampling methodology
Intellectual and Developmental Disabilities
The Collaborative’s continuous improvement efforts for the IDD system focused on three key areas:
training for stakeholders, the new IDD Connects system, and review process changes including tools,
procedures and web-based application. Following are the initiatives for each of these areas.
Early in FY 2019, the Collaborative’s Quality Management team and DBHDD met to discuss
review findings from FY 2018 to identify training needs for the stakeholder community. As a
result, the following training sessions were developed and provided to stakeholders in the
2019 fiscal year:
o A total of 11 training sessions were developed centered around the Focused Outcome
Areas (FOA) and presented throughout the year:
Whole Health: How to Identify Potential Health Risks; Monitoring Health Risks
and Protocols
Person Centered: CMS Expectations for the Support Planning Process;
Creating a Person-Centered Culture in Any Setting
Community: Networking with the Community to Build Relationships and
Natural Supports; Community Life: How to Support Community Relationships
Quality Management Annual Report FY 2019
12 | P a g e
and Natural Supports; Supporting Individuals to Gain Competitive
Employment
Rights: How to Identify Rights Violations; How to Support Self Advocacy
Choice: What is Informed Choice?
Safety: Educating Individuals on Abuse, Neglect & Exploitation
The Human Services Research Institute (HRSI) developed and presented an
additional training focused on National Core Indicator (NCI) survey results and
the development of quality improvement initiatives.
Qlarant, a subcontracting partner within the Collaborative, created an additional 16 separate
training modules, with PowerPoint presentations and User Guides to assist with
implementation of the new IDD Connects system. Each session included information for the
different user roles developed to access the system. As a part of the training team, Qlarant
participated in numerous meetings and collaborative sessions with Beacon, its Information
Technology sub-contractor and DBHDD. Qlarant also helped train Beacon staff on the
presentations in preparation for training stakeholders. At the end of the fiscal year, one
training session was conducted for regional staff.
In collaboration with Qlarant, DBHDD conducted analysis on review tools. Based upon the
results of the analysis, efforts to help reduce administrative burden on providers, and
eliminate redundancy between tools, DBHDD requested the review processes be modified.
These changes included discontinuation of the Person Centered Review as well as several
review processes from the QEPR: the individual interview, provider staff interview, Support
Coordinator interview and observations. Consequently, in the last quarter of the year,
Qlarant’s efforts focused on modifying the record review tools, review procedures, the web-
based application and QEPR reports.
Quality Management Annual Report FY 2019
13 | P a g e
FY 2019 Key Findings
Based on data collected over fiscal year 2019, in conjunction with review and analysis of trends
associated with previous year’s data, the following areas of strength and improvement opportunities
have been identified.
Behavioral Health Key Findings
The statewide average Overall BHQR score improved from 88 percent to 90 percent from FY
2018. In other words, the behavioral health Provider Network, in aggregate, improved their
performance, as aided by the associated technical assistance provided during the reviews and
ongoing training. Additionally, each category demonstrated significant increases in its
statewide average with the exception of Service Guidelines which remained high at 90
percent for both FY 2018 and FY 2019.
The BHQR Assessment and Planning category showed significant improvements related to
the three indicators that scored the lowest in the prior year:
o “Co-occurring health conditions being addressed/included in the IRP/NCP”
(FY 2018 60%; FY 2019 70%)
o “Discharge plan defining criteria”
(FY 2018 66% FY 2019 76%)
o “All assessed needs are addressed”
(FY 2018 67%; FY 2019 77%)
The FY 2019 Assertive Community Treatment (ACT) scores remained consistent to the
previous year’s results related to Focused Outcome Areas (93%), Service Guidelines (84%),
and Overall score (89%) with a statistically significant three-point increase for Assessment
and Planning to 90 percent.
Fifty-three percent of services reviewed with the BHQR Service Guidelines demonstrated
improvement from FY 2018 to FY 2019. For example, Nursing Assessment and Health Services
scores continuously improved from year to year since FY 2017 to reach 89 percent. Three
indicators for Nursing Services that demonstrated the greatest improvement were those
Quality Management Annual Report FY 2019
14 | P a g e
indicating that education was being provided to individuals related to their identified health
issues, specific service goals being individualized with progress notes indicating individuals’
progress (or lack thereof), and individuals’ responses to treatment and planned
interventions.
All BHQR FOA categories exceeded 90 percent with the exception of Safety. For the second
year in a row, of the three BHQR FOA Safety indicators, the one most often scored “No” was
“Individuals (or their legal guardians) signed medication consent forms along with the
prescriber.” Although results of this indicator significantly increased by 5 points, the score for
this indicator only reached 73% for FY 2019.
All CSUQR Whole Health FOA results improved significantly demonstrating documented
safeguards for utilized medications known to have substantial risks or side effects (93%),
ongoing assessment to determine external referral needs (99%), as well as communication
with external referral sources and providers as needed (99%).
While no significant improvements were identified specifically in CSUQR Service Guidelines,
several measured standards or indicators related to staff access, staff ratios, and staff
availability were met at 100 percent.
Intellectual and Developmental Disabilities Key Findings
Person Centered Review results showed the following:
Individuals receiving services through the DBHDD system were satisfied with those services,
felt safe, and had access to supports and services to address health needs (98% or higher).
Approximately 99 percent or more of individuals had access to supports and services to
address health needs and access to needed medications.
Individuals reported being free from all types of abuse, staff providing services appeared to
be aware of how individuals self-preserve, and the individuals served felt safe in their work
and living environments (99% or higher).
Over 90 percent of Individuals reported they participate in life’s decisions, were offered
choices, or were involved in the development of their ISP.
Quality Management Annual Report FY 2019
15 | P a g e
Most individuals interviewed indicated having privacy (99.6%), did not have any rights
restrictions (99.9%), and were treated with respect (99.6%).
Since FY 2017, Safety and Rights have generated the highest scores, over 90 percent each year, which
suggests providers have systems and practices in place to help individuals be safe in their
environments and exercise and learn about their rights. Individuals receiving services have indicated
they do feel safe and their rights are upheld. In FY 2019, the Individual Interview, Staff Interview and
Observation tools for both the PCR and QEPR scored over 90 percent. Consistent with previous years,
scores on the interviews and observations were higher than documentation by providers and
Support Coordinators (record reviews).
Behavioral Health Assessed Areas of Need
While most categories of the BHQR improved, the category of Service Guidelines maintained
the FY 2018 statewide average, 90 percent. Twenty services were reviewed with
improvements noted in only half of those services; thus, additional focus is to be dedicated in
the upcoming fiscal year to various services and ongoing analysis of the additional
“Programmatic” section piloted in FY 2019.
Although Billing scores increased statewide by a minimal one point for BHQRs, 57 percent of
reassessed providers (or providers not meeting the required 90 percent threshold at review)
had a decline from their first FY 2019 review to their second FY 2019 review with several
providers more deficient in “content does not support units billed” than the previous year’s
discrepancy of “missing or incomplete service order.”
While the BHQR Assessment and Planning category resulted in significant improvements
related to three lowest indicators; “co-occurring health conditions being addressed/included
in the IRP/NCP”, “discharge plan defining criteria”, and “interventions/objectives being goal-
linked and service-consistent;” the exact opposite was found in the case of CSUQRs by which,
the same exact indicators declined to a result of 36 percent (FY18 54%), 48 percent (FY18
69%), and 90 percent (FY18 96%), respectively.
All BHQR and CSUQR FOA categories exceeded 90 percent in FY 2019 except for Safety. For the
second year in a row, the BHQR FOA Safety indicator most often scored “no” was “Individuals
Quality Management Annual Report FY 2019
16 | P a g e
(or their legal guardians) signed medication consent forms along with the prescriber”.
However, results of this indicator have continued to trend upward yet again by five points to a
result of 73 percent for FY 2019.
Specific to the CSUQR IRR, only Admission/Initial Evaluation/Initial Screening and Course of
Stay had categorical improvement; yet declines were evident in Treatment Planning (70%) and
Discharge/Transition Planning (56%) by four points and 36 points respectively.
Within the Transition/Discharge Summary subcategory of the Individual Record Review (IRR),
two new indicators were added in FY 2019 with both resulting in exceptionally low numbers of
records providing such evidence that “A discharge summary was entered into the
Collaborative’s Provider/Connect/batch system,” and “If individual was discharged to a
homeless shelter, documentation reflects alternative were explored;” 24 percent and 36
percent respectively. These indicators, although new to the measurement review tool for FY
2019, were not new standards set forth by DBHDD. The indicators were added to address
deficits identified throughout the past three years via assessor reviews and additional
comments identified.
Although part of a non-scored element within CSUQR IRR category, follow-up and connection
to continuing care statistically decreased (p < .01) by 12 points to a low result of 39 percent.
Moreover, vital signs being documented, also a non-scored indicator, improved (p < .01) from
62 percent in FY 2018 to 72 percent in FY 2019. Records pertaining to individuals aged 17 or
below met this standard in 57 percent of the records reviewed compared to those aged 18
and over at seventy three percent).
During the CSUQR, policy adherence declined from the previous year’s results for both
theraputic blood level monitoring (90% to 77%) and safe storage of medication (86% to 59%).
Five new providers were reviewed and analyzed in FY 2019 that had never before had a quality
review completed. Of the five providers, three scored within the ten lowest scoring reviews
for Overall score, with two of those providers receiving a zero percent for the Billing category.
Twenty-eight providers were reviewed for a second time in FY 2019 due to either low scores in
Overall, Billing, or both. While this was an influx due to increased thresholds, it was still less
Quality Management Annual Report FY 2019
17 | P a g e
than the number of additional reviews conducted in FY 2017 (N = 35). While it is anticipated
that providers who are reassessed are likely to obtain an elevated score when measured on
the same indicators, only 36 percent (N = 10) actually received an increased score by year end.
Intellectual and Developmental Disabilities Assessed Areas of Need
Since FY 2018, staff supporting individuals receiving services was significantly less likely to
describe how individuals are supported to learn about medications (57.3% compared to
68.4%) and significantly less likely to document how education is provided on all prescribed
medications (27.9% compared to 48.3%), their risks and side effects (37.5% compared to
52.1%), or show evidence psychotropic medications were prescribed by a psychiatrist or
psychiatric nurse practitioner (86.9% compared to 96.5%).
Approximately half of providers had at least one staff member who did not complete training
within the required timeframes, specific to medications and their side effects.
Over 40 percent of PCRs included recommendations related to providing education regarding
abuse, neglect and exploitation and helping individuals receiving services learn how to
manage safety situations by conducting “what if” scenarios based on the following reasons:
o Approximately 35 percent of individuals receiving services were not aware of what
constitutes exploitation or neglect.
o Approximately 54 percent of providers had not offered education on how to self-
preserve or develop effective resiliency skills according to the individual’s learning
style.
Fewer than half of Support Coordinators interviewed demonstrated a knowledge of talents or
strengths of people they served, or regularly reviewed goals with individuals receiving
services. In addition, providers and Support Coordinators both demonstrated a significant
decrease, since FY 2018, of documenting how individuals respond to services or supports.
Community Life remains the lowest scoring area in the PCR and QEPR:
o Developing new social roles and maintaining those valued roles are instrumental ways
to increase community involvement and integration. Information from individual
interviews suggests approximately 30 percent to 40 percent of individuals receiving
Quality Management Annual Report FY 2019
18 | P a g e
services had not been given the opportunity to learn about or develop new social
roles in the community, or had support to maintain valued social roles.
o Provider and Support Coordinator documentation often does not show support for
individuals to seek employment in integrated settings (only 27.9% and 44.1% met this
expectation respectively).
o Only 8.4 percent of ISPs were written to promote full participation in the community.
o The ISP Likert Scale indicators demonstrated 46.9 percent of ISPs scored in the lower
two ratings, Emerging or Needs Improvement, showing summaries often did not
reflect the person’s community life. In addition, 41.8 percent of the relationship maps
scored in the lowest two ratings, which shows a lack of demonstration of how the
person is connected to the community.
Approximately 50 percent to 70 percent of provider and Support Coordinator records lacked
evidence that informed choice was provided for competitive or supported employment, living
situations, environments, community participation/social interactions, or for opportunities
for education, exploration or experiences in the community. This seems related to findings
that half of provider staff interviewed had not actually provided options for competitive
employment.
Over the past several years, documentation by providers and Support Coordinators has
shown consistently lower scores when compared to other information gathering techniques
(interviews or direct observations).
FY 2020 Areas of Focus
Behavioral Health
Throughout the year, Quality Management monitors review scores, specific service details,
and indicators to evaluate provider trends. With these findings, the Collaborative identified
and recommended changes to the BHQR and CSUQR tools to further supplement the quality
review process and reporting. Therefore, a new section entitled “Quality Risk Items” is being
added to the behavioral health review tools for FY 2020. Quality Risk Items would result in a
Quality Management Annual Report FY 2019
19 | P a g e
reduction to the Overall score by two percentage points for each identified item with a 10-
point maximum reduction. For a complete list of Quality Risk Items, please refer to the
Quality Management chapter in The Georgia Collaborative Provider Handbook.
The Collaborative suggested the introduction of additional reasons claims are not justified
following FY 2019 analysis. These additional reasons would better align the Collaborative’s
scoring policies with other auditing entities. These additional discrepancy reasons are also
intended to increase the quality of documentation within the individuals’ records by asking
providers to align their documentation more fully with existing standards.
Continue annual review and evaluation of existing BHQR and CSUQR tools and requirements,
especially for new or recently-added indicators to align with current state requirements and
DBHDD recommendations in accordance with the DBHDD Provider Manual.
Develop and continue the FY2020 Quality Training Services based on FY2019 data analysis.
Three of the five new providers who have never been reviewed by Quality Management,
scored within the ten lowest of all 157 reviews conducted. Furthermore, analysis at the
indicator level provided an additional level of detail by which it became evident new
providers require further training or technical assistance prior to their initial review occurring.
Consequently, Quality Management is recommending a collaboration and inclusion of a
training model specifically tailored to first time provider reviews.
In FY 2019, DBHDD released its FY 2017 BH Suicide Mortality Report. Through review of this
document and analysis of provider review findings, the Collaborative began a Quality
Improvement Study on suicidal ideation in FY 2020. Utilizing a tracer methodology, the study
will track individuals who have been readmitted to a CSU and with suicidal ideation at
admission.
It is notable that all individual and staff interviews averaged above 96 percent since FY 2016.
As a direct result of such high four-year findings, it has been recommended by the
Collaborative and approved by DBHDD to eliminate staff interviews beginning in FY 2020.
Additionally, Quality Management collaborated with DBHDD’s Office of Recovery to revise
the individual interviews to a set of ten questions.
Quality Management Annual Report FY 2019
20 | P a g e
The Programmatic category pilot proved successful through FY 2019, supplying supportive
supplemental information related to certain services. Quality Management, at the request of
DBHDD will continue to collect data specific to these indicators and requirements as a non-
scored element.
Intellectual and Developmental Disabilities
Based on changes to the reviews outlined in the IDD section of Quality Management
Activities, the Collaborative made several improvements to the review processes, tools and
reports, which will be implemented in FY 2020. These modifications will allow the
Collaborative to review all providers over a two-year period, which is a substantial increase
from 100 reviewed each year previously. Other modifications included the following:
o Publicly posting QEPR Final Assessment reports to the Collaborative’s website.
o The Exit Conference and Final Assessment QEPR reports have been extensively
modified to present results of the QEPR with graphics and data tables. The reports
now include sections for information related to the following:
Immediate Action Items: areas the provider should address immediately and
typically relate to significant documentation issues or health, safety or rights
of the individual.
Key Findings: a new section which provides justification for the scores on the
tools.
Recommendations: identifies, by review tool, areas where the provider needs
to focus efforts related to documentation and compliance requirements.
Additional Consultation and Suggestions for Quality Improvement: allows the
Quality Assessor to document technical assistance provided during the QEPR
and offer suggestions on how the provider can improve practices, processes
and documentation.
o Review tools were reformatted to provide additional guidance to Quality Assessors,
specifically in the documentation, and the specific reasons an indicator may be scored
not present.
Quality Management Annual Report FY 2019
21 | P a g e
o New tools to score Service Guidelines were developed and include the format
mentioned above.
The Collaborative will work with the DBHDD Office of Health and Wellness to identify
initiatives promoting independence in maintaining health and staff training relating to the
individuals’ medications. The sessions should include ways to determine if individuals
receiving services understand, to the best level possible, how to maintain personal health
and the benefits/risks of prescribed medication.
Quality Management is recommending, with support from DBHDD, the Georgia Learning
Community could develop “communities of practice” and not only focus on person centered
practices but, also, social role development and community integration. Advocacy groups
and providers across all the regions could be solicited to help generate ideas of how to
improve an understanding of what social roles are, as well as how to explore and develop
new roles. Collaborative Quality Assessors could attend meetings and add additional support
and ideas as needed.
Quality Management recommends developing a training to address how to document
choices offered and how providers can support individuals to make informed decisions. This
training would target support staff and Support Coordinators.
New providers have joined the DBHDD service delivery team, new staff have joined existing
providers, and new requirements for providers may have impacted the quality of provider
documentation. Quality Management will identify key areas from this report where
documentation is clearly falling behind interview findings and recommends targeting these
specific to training for providers and training for Support Coordinators.
In Conclusion
During FY 2019, the continued collaboration between DBHDD and Quality Management resulted in
an increased awareness and education to the BH and IDD network specific to:
Justification of billing (BHQR)
Focused Outcome Areas documentation of Whole Health (BHQR/CSUQR)
Focused Outcome Areas documentation of Rights (BHQR/PCR/QEPR)
Quality Management Annual Report FY 2019
22 | P a g e
Education on health (QEPR)
Both IDD and BH quality reviews indicate areas of focus such as safety/crisis planning and medication
education. It has been the goal of DBHDD and the Collaborative to integrate quality measurement
processes relating to BH and IDD services. Side-by-side comparisons are difficult to evaluate due to
the use of different types of sampling methods for each discipline, as well as differences in the tools
and standards utilized. During FY2019, Quality Management, in conjunction with the DBHDD Office
of Quality and divisional subject matter experts, reviewed the behavioral health and intellectual and
developmental disability record review tools to identify modifications that will enhance the ability to
make these types of comparisons. Recommendations and tool revisions have been approved to be
implemented FY2020.
Quality Management Annual Report FY 2019
23 | P a g e
Section 2: Introduction
The Georgia Collaborative Administrative Services Organization (the Collaborative), contracted by
Georgia’s Department of Behavioral Health and Developmental Disabilities (DBHDD), continued to
employ 30 Quality Assessors to complete more than 3,100 quality reviews and interview over 13,400
individuals and staff across the state of Georgia since contract inception. The mission of Quality
Management within the Collaborative is to provide DBHDD with valuable and beneficial data for use
in decision-making regarding the quality of services, funding, and development of programs (among
other quality improvement initiatives).
This annual report is the result of assessing, gathering, compiling, analyzing, and measuring the
quality of the service-delivery system through assessment of and technical assistance provided to
DBHDD’s behavioral health and intellectual and developmental disability providers. Furthermore,
this report contains suggestions for modifications in some processes, methods, approaches, and
tools to measure the quality and impact of services. Quality Management recognizes that quality
assessment or measurement processes must have a quality review periodically to examine and
determine whether current tools are measuring those things that are most important and
meaningful to those receiving services and DBHDD.
This annual report includes behavioral health, intellectual and developmental disability, and crisis
stabilization unit (CSU) findings as well as a detailed explanation of the review processes, analysis,
and comparisons of network performance across providers and information from individuals about
the services received. A difference in proportions test was used to determine statistical significance
and, where applicable, determine change from year to year.2 Areas highlighted in blue within tables
reflect significance at a p < .01 or have the p-value defined. While there are similarities in the review
processes, there are also distinct differences; therefore, behavioral health, intellectual and
developmental disability, and CSU results are reported separately. The report is divided into the
following sections:
2 Bohrnstedt, George W. & Knoke, David. (1988). Statistics for Social Data Analysis, 2nd Edition. Itasca, Illinois. F.E. Peacock Publishers,
Inc., pgs. 198-200.
Quality Management Annual Report FY 2019
24 | P a g e
Behavioral Health Quality Reviews
o Assertive Community Treatment
o Crisis Stabilization Unit Quality Reviews
o New Provider Reviews
o Reassessment Frequency Reviews
Intellectual and Developmental Disabilities
o Person Centered Reviews
o Quality Enhancement Provider Reviews
o Quality Technical Assistance Consultations
Summaries of findings and recommendations after each service delivery system section to
address areas needing improvement or training/education programs to help improve the
quality of services provided to Georgians
Provider and Individual Feedback Surveys
Two appendices are attached
o Acronym and abbreviation list, Appendix A
o Distribution graphs for all tools used in the BHQR, CSUQR, and intellectual and
developmental disability reviews, Appendix B
Quality Management Annual Report FY 2019
25 | P a g e
Section 3: Behavioral Health Quality Reviews
Background
The purpose of the Behavioral Health Quality Review (BHQR) is to determine providers’ adherence to
DBHDD’s standards and to assess the quality of the service delivery system through individual record
and claims reviews.3 Review indicators are based on DBHDD and Medicaid requirements, and are
organized into four categories: Billing Validation, Assessment and Planning, Service Guidelines, and
Focused Outcome Areas (FOA). The score for each category represents the percent of relevant
indicators met or present. The BHQR Overall score is calculated by averaging the scores for the four
categories.4 Each category accounts for 25 percent of the Overall score. In collaboration with DBHDD,
minor alterations were made before and throughout the fiscal year (FY) 2019 review process to
adhere to DBHDD and Medicaid requirement changes (if any), as well as language modifications
specific to tools, providing more clarity for Quality Assessors and providers. Additionally, the BHQR
contains a fifth category that is for data gathering purposes only, BHQR Programmatic. Please refer
to the BHQR Programmatic section of this report for more details.
Consistent with previous fiscal year reviews, when a BHQR is completed the lead assessor convenes
the team to analyze data gathered and plan the exit conference. The exit conference provides
immediate, preliminary feedback of the BHQR findings to the provider. A report of these preliminary
findings is left with the provider in the form of an exit conference report outlining the provider’s
identified strengths and any opportunities for improvement from the four primary categories of the
review. Quality Assessors include any items falling outside the parameters of the review determined
to be an area of concern or risk. Extensive technical assistance is provided during the review and exit
conference.
3 Please refer to the following link to access a full description of the review process and review tools. The Georgia Collaborative - Quality Management BH 4 The FOA subcategories are individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all “Yes” or “Present” responses of all FOA subcategories and dividing by the total “Yes” or “No” responses of the combined subcategories for each review.
Quality Management Annual Report FY 2019
26 | P a g e
Within 30 days of completion of a BHQR, a final assessment report is posted on the Collaborative’s
website and the provider is notified via email of the posting, along with the agency’s final score(s).
Like the exit conference report, the final assessment identifies strengths and opportunities for
growth in the four categories of FOA, Billing, Service Guidelines, and Assessment and Planning. The
final assessment highlights areas of concern falling outside the scope of scoring. The final
assessment also includes specific recommendations for improvement and comparisons to a
statewide average based on the previous fiscal year’s results.
Providers are offered an opportunity to appeal the BHQR findings. Appeal information, including
timeframes for submission, is provided upon notification of the final report and is made available to
providers by visiting the Collaborative’s website or reviewing the Georgia Collaborative Provider
Handbook.
Sampling Method
Individual Records and Billing Review
During the 2019 fiscal year, 131 providers were eligible for review, as determined by DBHDD, and
received a BHQR. This sample represents a slight reduction from the previous year in which 136
providers were reviewed. In FY 2019, five reviews were conducted for providers new to the DBHDD
network. An analysis of these new providers is contained within this report.
Whether an active DBHDD provider received a BHQR during a fiscal year is based on multiple factors,
including: volume of individuals served, claim volume, and type of services provided. Additionally,
while efforts to review all providers within the behavioral health (BH) network are made, some
providers may be classified as ineligible due to site closure, changes to location or site, or at the
direction of DBHDD. A sample of individuals was selected for each of the record reviews, and a
sample of those individuals’ claims was used for the billing review. To be included in the sample,
each individual selected must have had at least three claims within the three months (or longer, if
necessary) preceding the BHQR. Providers deemed ineligible because of minimal claim availability or
individuals served (< 5) within the timeframe are monitored periodically for reevaluation of
eligibility.
Quality Management Annual Report FY 2019
27 | P a g e
To select the individuals as part of the record review, an unduplicated list of individuals receiving
services from the provider was stratified by service and payer source: Medicaid, state-contract, and
fee-for-service paid claims. The sample was selected proportionate to the providers’ ratio of
individuals served by payer source, ensuring each service provided and approved for review by
DBHDD was represented. The sample for each provider consisted of up to 30 records for non-
intensive and specialty service providers, with an additional 15 service-specific records for providers
who also offered Assertive Community Treatment (ACT). The number of records selected per
provider was based on the number of individuals served during the six months prior to review,
thereby grouping a provider into one of the following categories.
Small Providers (serving 5-50 individuals) = 5-10 records selected
Medium Providers (serving 51-100 individuals) = 20 records
Large Providers (serving 101+ individuals) = 30 records
Crisis Stabilization Unit Quality Reviews = 15 records
Assertive Community Treatment Providers = 15 additional records
Providers serving fewer than five (< 5) individuals are monitored and added to the review process
once at least five individuals have been served; however, these providers can be selected for an ad
hoc review at the direction of DBHDD. Additional varying analysis occurred throughout the year by
Quality Management to evaluate appropriate service representation, which too is impacted by
individuals served (by the provider) as well as number of claims billed and is discussed further in the
Service Guidelines section of this report.
A representative sample of the population is preferred using standard sampling calculations. The
calculated sample size (even when calculated at the individual provider level based on consumer
utilization) was, in some instances, beyond the scope and resources available to both the provider
and the Collaborative (i.e. one provider may be required to have 200+ records reviewed). Thus, in
quarter one of FY 2019, a formal Power Analysis5 was conducted utilizing past provider results and
5 The Power Analysis was conducted utilizing a difference in Means test, within G*Power version 3.1.9.2 accessible via http://www.gpower.hhu.de, standardizing Power at .80 and alpha at .05.
Quality Management Annual Report FY 2019
28 | P a g e
scores to decipher a statistically relevant sample size needed and to ensure the groupings
established above were, in fact, sufficient in having the sample represent the population.
Statistically relevant sample sizes by groupings (Small, Medium, Large, and CSU) were confirmed
after removing outliers.
Oversampling of individuals at each quality review began in FY 2018 to preserve the integrity of the
sample and to ensure a full sample is reviewed. The oversample includes additional, unique
individuals following the above outlined sampling method. Providers receive the list of individuals
(including the oversample) at the start of the review process. Individuals listed on the oversample,
but not reviewed, are deleted from the review.
During FY 2019, 3,353 individuals were sampled for record reviews, 66 (2%) of which were age(s) 17
and under. The total number of records is nearly 300 more than the previous year. This is most likely
due to the standardization and confirmation in sampling methods and requirements specific to
provider size, as well as results of past reviews. Furthermore, an increase in Overall score and Billing
score thresholds occurred for the beginning of FY 2019, thereby rendering additional review
frequency to the network of having a review completed every six months.
With the increase in the FY 2019 threshold to 90 percent, 28 providers were required to have a
reassessment. This is a 40 percent increase compared to the previous year (N = 20). However, this is
still fewer than the initial baseline year in which thresholds were introduced (FY 2017, 80 percent
Overall and Billing scores) in which an additional 35 providers were reassessed See the Reassessment
Frequency Review section for more information.
FY 2019 data also provided an additional level of detail specific to type of diagnosis reviewed by
individual, where 85 percent of all records had a confirmed mental health diagnosis. While the
population is not stratified at sampling specific to age, some differences are clearly identifiable
between those aged 17 and under compared to those 18 and over and are mentioned accordingly
throughout the report. Furthermore, individuals may have more than one diagnosis selected during
Quality Management Annual Report FY 2019
29 | P a g e
the record review process. The following information shown in Figure 1, demonstrates the diagnostic
categories.
Figure 1. FY 2019 Diagnostic Categories Reviewed
All records (N = 3,353) 17 and Under (n = 66)
For each record, a random sample of up to 10 paid claims was selected for a billing review for both
BHQR and ACT reviews and the total number of claims reviewed for FY 2019 was 28,935 compared
to 25,598 in FY 2018. The number of records reviewed at a provider directly affects the total number
of claims reviewed. As there was an increase in the number of records reviewed (298), there was
also an increase in the number of claims reviewed. When providers did not have adequate claims
submissions in the three months preceding their review, the claims selection timeframe was
extended, but did not precede the provider’s previous BHQR timeframe (when applicable).
BHQR Billing Validation claim(s) reviews focused on specific services. The services included are listed
in Table 4 and Table 5. All eligible providers were reviewed at least once during the fiscal year and
had at least one claim per billed service included in the claims review, ensuring the complete array of
services provided were included in the BHQR.
Individual and Staff Interviews
In FY 2019, 572 Individual Interviews and 620 Staff Interviews were completed. Samples used for the
Individual and Staff Interviews were selected by the provider and Quality Assessors conducting the
BHQR; services received or provided were not considered in the selection of interviewees. Interview
sampling methods remained the same for FY 2019 with no changes since inception. Quality
Assessors attempted to complete a minimum of five Individual and five Staff Interviews per BHQR.
However, the actual number interviewed fluctuated based on individual and staff availability, their
IDD
7%
Mental Health
85%
Substance Use
29%
Physical
41%
IDD
27%
Mental Health
96%
Substance Use
6%
Physical
17%
Quality Management Annual Report FY 2019
30 | P a g e
agreement to participate in the interview process, the number of employees, and the number of
individuals the provider served at the time of the review. If an individual or staff declined an
interview, the Assessors would select an additional individual or staff to be interviewed, when
possible. When reviewing providers who served individuals for both non-intensive outpatient
services (NIOP) and ACT services, Assessors would attempt to interview at least one individual
receiving ACT services and one staff member from the ACT team.
There was a decrease in both Individual and Staff Interview participation by approximately 76
interviews from the previous year. Analysis has shown that fewer interviews were conducted at
smaller provider locations (serving 5-50 individuals). Many Small providers (n = 7, 12%) had only one
interview completed, while several reviews (n = 47, 30%) had only three Individual/Staff Interviews
conducted. The primary reason(s) for fewer interviews completed was due to the lack of availability
of individuals served and staff availability. Results from interviews conducted are not included in the
BHQR Overall scores; however, results have provided valuable, qualitative feedback to promote
quality improvement activities. As an additional quality initiative, the results of interviews were
shared with providers and DBHDD to provide direct communication of individuals’ perceptions and
experiences of receiving services and staff’s knowledge and expertise related to providing services.
BHQR Overall Review Scores In this report, data is aggregated and presented by overall provider scores as well as by category. The
four main categories of Billing Validation, Assessment and Planning, Service Guidelines and Focused
Outcome Areas (FOA) each account for 25 percent of the Overall score. Each FOA (Whole Health,
Safety, Person Centered Practices, Community Life, Choice, and Rights) also has scores displayed and
discussed.
BHQR Overall Scores
FY 2017
84%
FY 2018
88%
FY 2019
90%
Quality Management Annual Report FY 2019
31 | P a g e
Figure 2 shows a graphical representation of the average Overall score for the 157 reviews and the
scores for each category compared to the previous two fiscal years to demonstrate areas of
improvement and decline. The mean, or statewide average, of Overall scores for FY 2019 was 90
percent. This is a two-point increase compared to FY 2018 and a six-point increase from FY 2017.
While the FOA category demonstrated the most consistent improvements across fiscal years (FY19
94%), Assessment and Planning continued to trend upwards with a FY 2019 result of 88 percent.
Moreover, where Assessment and Planning demonstrated a historically low result across years with a
final FY 2019 result of 88 percent, Billing became the FY 2019 lowest-scoring category with a result of
86 percent (Figure 2) .
The category of Service Guidelines, historically the highest category since inception (FY 2016),
continued to maintain a steady statewide average at 90 percent. The following are key findings of the
overall category scores:
Assessment and Planning had the greatest improvement from the previous fiscal year by four
points. With an eight-point difference since inception, and 11-point increase from FY 2017,
Assessment and Planning quality indicator results continued to rise from year to year.
Providers have demonstrated being receptive to education and technical assistance from
Assessors specifically in the areas of documentation that supports the incorporation of whole
health and wellness goals into Individual Recovery/Resiliency Plans (IRPs), addressing co-
occurring health conditions, creating clear and identifiable discharge planning criteria, and
addressing all assessed needs. See Assessment and Planning section for indicator results.
Billing Validation consistently trended upward for the fourth year with scores of 81, 84, 85,
and 86 percent respectively as depicted in Figure 2.
The FOA category continued to demonstrate an upward trend in results through FY 2019.
Given the relatively high scores in this category (exceeding 90 percent for the second year in
a row), it seems providers may be more attentive to the standards and measured outcomes
in this area than they were four years ago. For example, Whole Health scored 63 percent in
FY 2016, but demonstrated significant improvement over the years (FY17 74%; FY18 84%;
FY19 91%).
Quality Management Annual Report FY 2019
32 | P a g e
Service Guidelines’ statewide average was equal to the previous year’s score of 90 percent.
Nearly three quarters of the reviews (73%, N = 114) met or exceeded 90 percent with ten
reviews receiving a perfect score.
Figure 2. BHQR Fiscal Year Results by Category
BHQR Overall Scores by Tier and Provider Size
In 2014, DBHDD implemented a community behavioral health provider network structure in which
providers were classified using a four-tiered structure.6 Tiers are defined as follows:
Tier 1: Comprehensive Community Providers
Tier 2: Community Medicaid Providers
Tier 2+: Community State Funded Providers
Tier 3: Specialty Providers
6 Policies regarding the implementation and definition of the DBHDD Community Behavioral Health Provider Network Structure can be found at DBHDD’s PolicyStat website: https://gadbhdd.policystat.com
Overall Score FY17: 84%
(N = 167)
FY18: 88%(N = 156)
FY19: 90%(N = 157)
Billing Validation
FY17: 84%FY18: 85%FY19: 86%
Focused Outcome
Areas FY17: 89%FY18: 92%FY19: 94%
Assess. &Planning
FY17: 77%FY18: 84%FY19: 88%
Service Guidelines
FY17: 88%FY18: 90%FY19: 90%
Quality Management Annual Report FY 2019
33 | P a g e
Table 1 provides a snapshot of the BHQR Overall scores by tier and a distribution of scores by each
tier is presented in detail in Appendix B. Results increased from FY 2017 to FY 2018 across tier levels
1, 2, and 2+ but declined for Tier 1 providers from 90 to 88 percent in FY 2019. Both Tier 2 and 2+
providers continued to demonstrate improvement, scoring at 91 and 92 percent respectively. Lastly,
consistent with results of previous years, scores remained unchanged and generally lower for the
group of Tier 3 providers (84%) compared to the other tiers.
Table 1. BHQR Overall Scores by Tier*
Fiscal Year Tier 1 Tier 2 Tier 2+ Tier 3 Statewide Average
FY 2017 85%
(n = 28)
85%
(n = 115)
86%
(n = 6)
83%
(n = 18)
84%
(N = 167)
FY 2018 90%
(n = 29)
89%
(n = 93)
87%
(n = 6)
83%
(n = 28)
88%
(N = 156)
FY 2019 88%
(n = 27)
91%
(n = 102)
92%
(n = 6)
84%
(n = 22)
90%
(N = 157) *N represents the total number of reviews conducted statewide; n represents the subset number of reviews per Tier level. Some providers may
have had a reassessment review thereby increasing the number as compared to previous years.
Provider sizes are established as follows with the Overall score results by size grouping displayed in
Figure 3;
Small Providers (serving 5-50 individuals)
Medium Providers (serving 51-100 individuals)
Large Providers (serving 101+ individuals)
Quality Management Annual Report FY 2019
34 | P a g e
Figure 3. Overall Score by Provider Size by Year
Consistent with FY 2018 findings, Medium providers maintained the result of 91 percent and
exceeded the statewide average of 90 percent. Large providers improved in the year to also exceed
the statewide average and obtain a mean result of 91 percent as well. Small providers improved
across the year finishing with a mean result of 88 percent. While there is noted improvement of
three points for the Small group, the group remained three points below the other groups and two
points below the 90 percent Overall score.
Figure 4. Billing Score by Provider Size by Year
80%(n = 71)
90%(n = 29)
90%(n = 56)
84%(n = 58)
88%(n = 31)
88%(n = 68)
70%
75%
80%
85%
90%
95%
Small Medium Large
2018
2019
2019
2019
2018
2018
85%(n = 71)
91%(n = 29) 89%
(n = 56)88%(n = 58)
91%(n = 31)
91%(n = 68)
70%
75%
80%
85%
90%
95%
Small Medium Large
2018
2019
2019
2019
2018
2018
Statewide
Average
FY 2019
90%
(N = 157)
Statewide
Average
Billing
Score
FY 2019
86%
(N = 157)
Quality Management Annual Report FY 2019
35 | P a g e
In Figure 4, Small providers struggled to meet the threshold or statewide average of a 90 percent
Overall score, Medium and Large providers in FY 2019 exceeded the Overall result and the same was
the case specific to Billing scores. Small providers, while improving from FY 2018 to FY 2019 by four
points, continued to fall below the FY 2019 statewide result of 86 percent, where Medium and Large
providers, although declining both by two points, continued to exceed the network result of 86
percent. Further details regarding billing result declines will be explained further in the Billing
section of this report.
Table 2. BHQR Category Scores by Provider Size FY 2018 / FY 2019
Provider Size Overall Billing Service
Guidelines Assessment &
Planning FOA
Small 85% / 88% 80% / 84% 89% / 88% 82% / 85% 91% / 94%
Medium 91% / 91% 90% / 88% 92% / 92% 88% / 90% 95% / 94%
Large 89% / 91% 90% / 88% 91% / 92% 85% / 89% 92% / 93%
Statewide 88% / 90% 85% / 86% 90% / 92% 84% / 88% 92% / 91%
In review of all category scores across provider sizes, Table 2, Small providers had increased results of
three to four points across all categories with the exception of Service Guidelines. Medium and
Large provider improvement/declines were minimal if at all. However, Medium providers resulted in
a one-point average decline in their FOA score whereby Large providers increased in their average
FOA score by one point. While Small providers make up 45 percent of the reviews conducted (N =
71, or 61 unique providers), they have fewer individuals reviewed (5-10); thereby increasing the
impact of missing elements or quality indicators scored negatively.
BHQR Billing Validation
The Billing score for each BHQR is the percent of justified paid dollars divided by the total paid
dollars for the reviewed claims. Billing scores are averaged across the network of annual reviews to
FY 2017
84%
FY 2018
85%
FY 2019
86%
Quality Management Annual Report FY 2019
36 | P a g e
obtain an annual statewide average. This category continued to trend upward from contract
inception, beginning in FY 2016 with a statewide average of 81 percent and ending FY 2019 with an
average of 86 percent. Improvement at the provider level is evident as in FY 2019 where 13
providers achieved a perfect 100 percent, compared to nine providers in FY 2018. One provider, not
previously reviewed, scored 0 percent in FY 2019. See the distribution of BHQR Billing scores in
Appendix B. Since an increase in Billing scores continued in FY 2018, the billing threshold was raised
to 90 percent for FY 2019 to align with DBHDD’s contractual Key Performance Indicators (KPIs). Only
56 percent of reviews (N = 88) met this threshold in FY 2019 and, interestingly, the average Billing
score of reassessed providers declined from 87 percent at the start of the year to 80 percent at year-
end. See Reassessment Frequency Review section for more details.
Figure 5 shows the total dollar amount reviewed through an analysis of paid claims for all providers
reviewed during FY 2019 ($2,892,206.46) compared to FY 2018 ($2,430,951.50) and FY 2017
($2,934,560.52) as well as the dollar amounts found to be justified and unjustified. In FY 2019, the
increase in funds reviewed may be due to several factors, including but not limited to: provider
sample size, claims availability, or the types of service reviewed. Although the total funds reviewed
for FY 2019 were nearly $500,000 more than the previous year, the amounts unjustified and
susceptible to recoupment continued to remain steady with a one-point increase for the year.
Quality Management Annual Report FY 2019
37 | P a g e
Figure 5. BHQR Billing Validation Amount Reviewed by Fiscal Year7
Figure 6 provides the percent of reviewed claim amounts by funding source. In FY 2019
$2,417,070.35 in Medicaid funds were reviewed compared to the state funded amount of
$475,136.10. The proportions of claims reviewed for FY 2019 by funding source is comparable to
previous fiscal years.
7 The percent of justified and unjustified dollar amounts, in Figure 5, are total dollar amounts justified or unjustified divided by the total amount of funds reviewed in total for the year. The annual statewide average Billing score is not equivalent to the percent justified/unjustified depicted in the figure; statewide Billing score is calculated based on averaging all provider scores at year-end.
$0.00
$500,000.00
$1,000,000.00
$1,500,000.00
$2,000,000.00
$2,500,000.00
$3,000,000.00
FY 2017FY 2018
FY 2019
$2,471,510.59 84% $2,136,207.68
88%
$2,513,252.73 87%
$463,049.93 16%
$294,743.82 12%
$378,953.72 13%
Total $ Justified
Total $ Unjustified
Total Amount Paid Claims Reviewed in FY 2017: $2,934,560.52Total Amount Paid Claims Reviewed in FY 2018: $2,430,951.50Total Amount Paid Claims Reviewed in FY 2019: $2,892,206.46
Quality Management Annual Report FY 2019
38 | P a g e
Figure 6. FY 2018 BHQR Amount Reviewed by Funding Source
For the first year since inception, state funded claims were found to be significantly (p < .01) more
justified than Medicaid claims. Table 3 depicts the justifiable and unjustifiable amounts by funding
source in which 87 percent of Medicaid claims reviewed were found justified compared to 89
percent of state funded claims.
Table 3. FY 2019 Amount Reviewed by Funding Source State Funds Medicaid Total
Amount Justified $422,116.12 $2,091,136.60 $2,513,252.72
(89%) (87%) (87%)
Amount Unjustified $53,019.98 $325,933.75 $378,953.73
(11%) (13%) (13%)
Total Amount Reviewed $475,136.10 $2,417,070.35 $2,892,206.45
The highest funded service(s) reviewed during FY 2019 were Individual Counseling ($507,716.34)
followed by Assertive Community Treatment ($318,879.82), and lastly, Intensive Family Intervention
($312,947.42). While most services had both sources of funding reviewed, some services had
substantially less state funds reviewed for FY 2019 compared to Medicaid funds. However, services
such as Substance Abuse Intensive Outpatient, Community Support Team (CST), and Group
Counseling had over $5,000 more in state funding claims reviewed than Medicaid funded claims.
Medicaid, 84%
State Funded,
16%
Total Amount Reviewed
$2,892,206.45
Quality Management Annual Report FY 2019
39 | P a g e
Table 4 and Table 5 provide the amount and percentages of funds justified by service, non-intensive
outpatient and specialty services of all funds combined, for fiscal years 2018 and 2019, respectively.
Table 4. Non-Intensive Outpatient Services Percent of Funds Justified
FY 2018 FY 2019
Service Amount Justified
Amount Reviewed
Justified Amount Justified
Amount Reviewed
Justified
Individual Counseling $380,030.45 $409,683.46 93% $475,262.18 $507,716.34 94%
Psychiatric Treatment $67,178.02 $75,386.25 89% $72,447.94 $78,804.19 92%
Diagnostic Assessment $37,270.98 $39,762.22 94% $37,230.03 $40,854.73 91%
Community Support $128,832.09 $146,364.10 88% $176,844.96 $197,659.23 89%
Behavioral Health Assessment
$75,895.66 $83,962.92 90% $96,288.18 $108,559.58 89%
Group Outpatient Services $107,754.77 $121,168.70 89% $83,356.02 $95,383.66 87%
Family Counseling/Therapy $140,012.07 $166,185.24 84% $178,706.41 $204,926.79 87%
Service Plan Development $51,843.00 $59,530.24 87% $69,778.54 $80,834.59 86%
Medication Administration $3,494.80 $3,876.51 90% $2,821.40 $3,430.62 82%
Case Management $109,814.09 $134,652.93 82% $148,094.90 $184,138.38 80%
Nursing Services $44,389.22 $51,337.63 86% $50,784.83 $64,149.30 79%
Psychological Testing $29,166.90 $35,869.25 81% $19,853.01 $25,261.70 79%
Crisis Intervention $9,301.10 $12,123.05 77% $7,822.65 $11,418.59 69%
Community Transition Planning
$3,103.01 $3,312.21 94% $1,936.18 $3,777.14 51%
Total $1,188,086.16 $1,343,771.74 88% $1,421,227.23 $1,606,914.84 88%
*Areas highlighted in blue within table reflect significance at a p < .01.
Quality Management Annual Report FY 2019
40 | P a g e
Table 5. Specialty Services Percent of Funds Justified*
FY 2018 FY 2019
Service Amount Justified
Amount Reviewed
Justified Amount Justified
Amount Reviewed
Justified
Opioid Maintenance $9,430.70 $11,063.50 85% $4,809.60 $4,809.60 100%
Community Support Team (CST)
$13,042.33 $14,985.88 87% $16,742.65 $17,901.24 94%
MH Peer Support - Individual $19,806.24 $26,289.03 75% $27,820.63 $30,251.62 92%
Assertive Community Treatment (ACT)
$295,649.45 $325,851.60 91% $281,284.86 $318,879.82 88%
Addictive Disease Support Services (ADSS)
$54,703.17 $64,477.32 85% $94,439.45 $108,062.11 87%
MH Peer Support Program $83,508.23 $90,261.59 93% $99,132.88 $113,524.56 87%
Intensive Case Management $16,751.46 $19,101.80 88% $26,504.41 $30,534.94 87%
Psychosocial Rehabilitation Program
$66,408.84 $78,011.70 85% $71,760.93 $83,863.64 86%
Psychosocial Rehabilitation - Individual
$151,633.45 $172,628.40 88% $198,060.49 $233,412.49 85%
Peer Support Whole Health & Wellness - Individual
$5,606.28 $5,606.28 100% $3,849.18 $4,553.81 85%
Substance Abuse Intensive Outpatient
Not Reviewed $19,495.66 $24,674.84 79%
Intensive Family Intervention $221,099.15 $267,624.22 83% $246,963.16 $312,947.42 79%
AD Peer Support - Individual $399.30 $1,052.70 38% $1,161.60 $1,875.52 62%
Peer Support Whole Health & Wellness - Group
$10,082.92 $10,225.74 99% Not Reviewed
Grand Total $948,121.52 $1,087,179.76 87% $1,092,025.50 $1,285,291.61 85%
*Areas highlighted in blue within table reflect significance at a p < .01.
When a claim was found to be unjustified, Assessors selected all applicable reasons a reviewed claim
was identified as a discrepancy; therefore, one unjustified claim may have multiple reasons. Figure 7
provides the percent of discrepancies within the year. Of reviews completed that incorporated a
review of the service Community Transition Planning (51% justified funds), the most common
discrepancies identified were “Content does not support units billed,” “Content does not support
Quality Management Annual Report FY 2019
41 | P a g e
code billed,” and “Content of note does not match service definition.” Additionally, reviews
conducted for the service of AD peer Support – Individual (62%) also had high discrepancies of
“Content does not support units/code billed,” but also “More contacts/units per day than allowed,”
and “Missing/incomplete service order.” Figure 7 provides the percent of discrepancies within the
year of all service types reviewed.
Figure 7. FY 2019 Percent of Discrepancy Reasons
Other discrepancy reasons not included in Figure 7 include:
Individual does not meet admission criteria for service billed (1.8%)
Date of entry missing (1.6%)
Intervention outside the scope of practice for staff (1.3%)
Non-billable activity (1.3%)
Diversionary activities billed (< 1%)
Printed name missing (< 1%)
Multiple services billed at same time (< 1%)
Signature missing (2.2%)
Location missing (out-of-clinic) (3.1%)
Units billed exceeded time and/or units (3.4%)
Progress note is missing (3.6%)
Billing code is missing or different from code billed (3.9%)
Staff credential missing (5.9%)
Content is not unique to the individual (6.2%)
Content does not support code billed (8.7%)
Content of note does not match service definition (9.3%)
No valid, verified diagnosis on date service provided (9.4%)
Missing/incomplete service order (10.0%)
Staff credential not supported (11.4%)
Content does not support units (11.8%)
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
Quality Management Annual Report FY 2019
42 | P a g e
More contacts/units per day than allowed (< 1%)
Consistency requirements missing (< 1%)
Date of service incorrect/missing (< 1%)
Time in/Time out missing (< 1%)
Mutually exclusive services billed (< 1%)
Record not submitted within timeframe (< 1%)
While there were nearly 500 fewer unjustified claims identified in FY 2019 some of the most
prevalent discrepancy reasons identified in FY 2019 were similar to FY 2018 and include:
Content did not support units billed – For example, billed eight units for a medication check
with no further documentation of interventions provided.
Content did not support code billed – For example, the out-of-clinic modifier was billed but
the session took place in the office.
Content of note not did not match the service definition – For example, the provider billed
Case Management, but the documentation indicated the practitioner provided Psychosocial
Rehabilitation - Individual.
Historically, common billing discrepancy reasons were individuals not meeting admission criteria,
missing progress notes, and missing signatures. However, improvements have been made in these
areas from FY 2018 to FY 2019 by 72 percent, 56 percent, and 53 percent respectively. Table 6
provides the number of all discrepancy reasons, in ranking order for FY 2019 by the number of
instances, and the change in percent of the instance count from year to year. A negative difference
demonstrates improvement from year to year with highlighted areas demonstrating a statistically
significant change. Overall, 14 (58%) of the observed indicators (N = 24) improved from the previous
fiscal year. One new discrepancy reason was added for the FY 2019 measurement year specific to a
valid/verifiable diagnosis being on file for the specific service date reviewed. For this discrepancy,
there were 417 instances of 28,935 claims or 1.4 percent in FY 2019.
Quality Management Annual Report FY 2019
43 | P a g e
Table 6. BHQR Billing Discrepancy Reasons by Fiscal Year*
FY 2017 FY 2018 FY 2019
Instances Instances Instances
Claims Reviewed 29, 602 25,598 28,935
Missing/incomplete order 496 (1.7%) 285 (1.1%) 444 (1.5%)
No valid/verified diagnosis on service date N/A N/A 417 (1.4%)
Does not meet admission criteria 748 (2.5%) 254 (1.0%) 82 (1.8%)
Quantitative
Progress note is missing 676 (2.3%) 326 (1.3%) 161 (0.6%)
Code is missing/different than code billed 379 (1.3%) 244 (1.0%) 172 (0.6%)
Staff credential missing 295 (1.0%) 236 (0.9%) 263 (0.9%)
Signature missing 194 (0.7%) 182 (0.7%) 98 (0.3%)
Date of entry missing 185 (0.6%) 160 (0.6%) 71 (0.2%)
Units billed exceed time/units documented 112 (0.4%) 90 (0.4%) 152 (0.5%)
Consistency requirements missing 57 (0.2%) 31 (0.1%) 28 (0.1%)
Credential not supported by documentation 276 (0.9%) 211 (0.8%) 505 (1.7%)
Record not submitted within timeframe 19 (0.1%) 10 (< 0.1%) 2 (< 0.1%) Location missing (out-of-clinic) 155 (0.5%) 106 (0.4%) 139 (0.5%)
Time in / time out missing 68 (0.2%) 14 (0.1%) 13 (<0.1%)
Date of service incorrect / missing 42 (0.1%) 34 (0.1%) 25 (0.1%)
Printed name missing 2 (<0.1%) 8 (<0.1%) 38 (<0.1%)
Performance Standards
Content does not match service definition 489 (1.7%) 413 (1.6%) 412 (1.4%)
Content does not support code billed 397 (1.3%) 403 (1.6%) 386 (1.3%)
Content does not support units billed 518 (1.8%) 441 (1.7%) 525 (1.8%)
Intervention outside staff's scope/practice 79 (0.3%) 120 (0.5%) 59 (0.2%)
Content is not unique to the Individual 243 (0.8%) 140 (0.5%) 273 (0.9%)
Multiple services billed at the same time 79 (0.3%) 45 (0.2%) 32 (0.1%)
Non-billable activity 59 (0.2%) 34 (0.1%) 56 (0.2%)
Diversionary activities billed 26 (0.1%) 111 (0.4%) 82 (0.3%) *Areas highlighted in blue within table reflect significance at a p < .01 and indicators with n < 90 were not tested due to low volume.
Over the course of the quality review process, providers have become acclimated to the billing
discrepancy reasons and recoupment process that has been established and implemented since
contract inception as evidenced by the consistent declines in discrepancy instances recorded. As
scores specific to Billing continued to increase statewide, the Collaborative and DBHDD have agreed
Quality Management Annual Report FY 2019
44 | P a g e
to introduce other areas of policy specific to Billing and to incorporate additional discrepancies in FY
2020 to include the following:
High utilization without clinical justification
Intervention unrelated to IRP without clinical justification
Monthly contacts not met per DBHDD Service Guidelines
No overall progress documented
Note does not include response to intervention
Progress note was not filed within seven calendar days
BHQR Assessment and Planning
Assessment and Planning consisted of ten indicators answered once per record reviewed. Table 7
contains the indicators and percent “Yes” in ranking order. All indicators increased or remained equal
for the second year in a row with areas of growth having continued through FY 2019 to include at
least a ten point, significant increase in each of the four lowest-scoring indicators of Assessment and
Planning:
All assessed needs are addressed
o C&A 62 percent, Adult 77 percent
Co-occurring health conditions addressed in the IRP
o C&A 47 percent, Adult 70 percent
Discharge plan defines criteria
o C&A 86 percent, Adult 76 percent
Whole health and wellness in IRP
o C&A 68 percent, Adult 82 percent
FY 2017
77%
FY 2018
84%
FY 2019
88%
Quality Management Annual Report FY 2019
45 | P a g e
Table 7. BHQR Assessment & Planning Indicator Scores by Fiscal Year*
Indicator FY 2017 FY 2018 FY 2019
Current medical screening is present 97% 98% 99%
Individual meets admission criteria 97% 97% 98%
Interventions/objectives are goal-linked & service-consistent 91% 92% 96%
Current behavioral health assessments of needs 94% 94% 96%
IRP is individualized in personalized language 87% 94% 94%
Goals/objectives honor hopes, choice, preferences, outcomes 89% 94% 94%
Whole health & wellness in IRP 56% 69% 82%
All assessed needs are addressed 48% 67% 77%
Discharge plan defines criteria 58% 66% 76%
Co-occurring health conditions addressed in IRP 34% 60% 70%
Assessment & Planning Statewide Average 77% 84% 88% *Areas highlighted in blue within table reflect significance at a p < .01 as tested from the previous year.
Data continues to suggest that, although records contained required assessment documentation
such as medical screenings (99%) and current behavioral health assessments (96%), a vast
percentage of recovery/resiliency plans lacked the documentation of discharge planning, co-
occurring health conditions being addressed, or a plan for increased whole health and wellness. On
July 1, 2018, an additional level of data collection was implemented and tailored to provide
additional detail for reason(s) that indicators within Assessment and Planning were “not met.” In
the case of the one percent of medical screenings not present, 36 records (88%) did not have an
annual medical/wellness update. Also, 77 percent of 137 records did not have an annual update of
behavioral health assessment of needs. The lowest-scoring indicators in the category of Assessment
and Planning, all with significant improvements (p < .01) for the first time in FY 2019, held a wide
array of reasons (new for FY 2019) for having not been met and have the reasons outlined below.
Quality Management Annual Report FY 2019
46 | P a g e
• Plan does not address all areas/types of need (60%, n = 464)
• Psychiatric issues not addressed (21%, n = 164)
• Substance use issues not addressed (18%, n = 140)
• Developmental issues not addressed (14%, n = 112)
• Plan is expired (10%, n = 81)
Assessed needs are addressed (23% "Not Met", N = 776)
• Areas identified in assessment but not addressed in IRP: Physical (56%, n = 426) IDD (18%, n = 133) Substance Use (17%, n = 126) Psychiatric illness (16%, n = 120)
• Plan is expired (9%, n = 67)
Co-occurring health conditions addressed (30% "Not Met", N = 756)
• No whole health & wellness on plan (82%, n = 501)
• Plan is expired (12%, n = 75)
• Whole health and wellness on plan; not individualized (5%, n = 32)
Whole health & wellness in IRP (18% "Not Met", N = 611)
• No clinical benchmarks indicated (44%, n = 352)
• Specific step-down service(s) not indicated/planned (35%, n = 285)
• Discharge Plan is expired (13%, n = 102)
• Date of discharge/transition no documented/planned (12%, n = 96)
• No plan at all (11%, n = 85)
• Discharge criteria does not align with goals/objectives (5%, n= 43)
Discharge/Transition Planning (24% "Not Met", N = 805)
Quality Management Annual Report FY 2019
47 | P a g e
BHQR Service Guidelines
Indicators within the Service Guidelines section of the BHQR were answered once per record
reviewed. The number of indicators answered varied, specific to the service. The service reviewed
for each record was dependent upon the services contained in the billing claims sample for the
respective individual; therefore, multiple services could be reviewed within one individual’s record.
A total of 16 NIOP services and 11 specialty services were identified as measurable services for FY
2019 as shown in Figure 8 and Figure 9. However, some services (Opioid Maintenance or
Medication-Assisted Treatment, MAT) may not have been reviewed for specific providers based on
lack of services billed within the eligible period or at the direction of DBHDD, thereby excluding
service(s) from some reviews.
The percent of dollars reviewed per individual service in FY 2019 was comparable to FY 2018 for both
non-intensive outpatient and specialty services. For example, in FY 2019 Individual Counseling
represented 26 percent of all sampled services, compared to 27 percent in FY 2018.
In analysis of funds reviewed specific to NIOP services, Individual Counseling had the most funds
reviewed (see Figure 8). Psychological Testing, Crisis Intervention, Community Transition Planning,
and Medication Administration all had the least amount of funds reviewed per service, each with
one percent or less reviewed in FY 2019.
FY 2017
88%
FY 2018
90%
FY 2019
90%
Quality Management Annual Report FY 2019
48 | P a g e
Figure 8. FY 2019 Percent of Dollars Reviewed by Non-Intensive Outpatient Service
Figure 9 indicates that Assertive Community Treatment (ACT) had the greatest amount ($318,879.82
or 34%) of funds reviewed specific to specialty services; hence ACT is considered a subset sample
with an additional 15 records sampled per ACT rendering provider/review. Intensive Family
Intervention (IFI) ranked second in FY 2019 with 33 percent of all specialty funds reviewed
($948,817.01). However, year-end analysis of data and its findings provided evidence that 37
providers offered IFI services some time during the year, yet only 27 providers had the service
reviewed by Assessors. Again, this was due to either a directive (prepayment review) made by
DBHDD or limited unique individuals being served/claims billed to generate a sample extraction in
line with sampling methods.
Individual Counseling (26%)
Psychosocial Rehabilitation -Individual (12%)
Family Counseling/
Therapy (11%)
Community Support (10%)
Case Management
(9%)
Individual Counseling (26%)
Psychosocial Rehabilitation - Individual (12%)
Family Counseling/ Therapy (11%)
Community Support (10%)
Case Management (9%)
Addictive Disease Support Services (ADSS) (6%)
Behavioral Health Assessment (6%)
Group Counseling/Therapy (5%)
Service Plan Development (4%)
Psychiatric Treatment (4%)
Nursing Assessment & Health Services (3%)
Diagnostic Assessment (2%)
Psychological Testing (1%)
Crisis Intervention (1%)
Community Transition Planning (<1%)
Medication Administration (<1%)
Quality Management Annual Report FY 2019
49 | P a g e
Figure 9. FY 2019 Percent of Dollars Reviewed by Specialty Service
Table 8 provides the scores by service type and year of review and represents the number of reviews
assessed for the service, based on the claims sample where “r” references the number of reviews,
and “n”, the number of records.
Assertive Community Treatment (ACT) (34%)
Intensive Family Intervention (33%)
MH Peer Support Program (12%)
Psychosocial Rehabilitation Program (9%)
Assertive Community Treatment (ACT) (34%)
Intensive Family Intervention (33%)
MH Peer Support Program (12%)
Psychosocial Rehabilitation Program (9%)
Intensive Case Management (3%)
MH Peer Support - Individual (3%)
Substance Abuse Intensive Outpatient Program (3%)
Community Support Team (CST) (2%)
Opioid Maintenance (1%)
Peer Support Whole Health & Wellness - Individual (<1%)
AD Peer Support - Individual (<1%)
Quality Management Annual Report FY 2019
50 | P a g e
* Areas highlighted in blue, represent significant differences where p < .01 from the previous year. Additionally due to low response volumn, Peer Support Whole Health & Wellness, AD Peer Support – Individual, and Community Support Team (CST) were not evlauted for statisical significance. Also, “r” represents the number of reviews whereas “n” respresnts the number of records.
Service Guidelines scored highest of the four categories following the first year of contract inception
at 90 percent. Although it declined slightly in FY 2017 by two points, it has since returned to 90
percent and remained there throughout FY 2019. Moreover, nearly three quarters of the reviews
Table 8. BHQR Service Guidelines Scores by Service Type*
Service Type FY 2017 FY 2018 FY 2019
AD Peer Support – Individual 48% (r = 1, n = 13) 80% (r = 2, n = 9) 84% (r = 4, n = 16)
Addictive Disease Support Services 85% (r = 69, n = 293) 80% (r = 61, n = 252) 90% (r = 62, n = 337)
Assertive Community Treatment 88% (r = 20, n = 334) 84% (r = 21, n = 336) 84% (r = 20, n = 313)
Case Management 84% (r = 90, n = 746) 88% (r = 85, n = 644) 89% (r = 84, n = 745)
Community Support 83% (r = 114, n = 637) 86% (r = 102, n = 446) 85% (r = 105, n = 550)
Community Support Team 92% (r = 7, n = 13) 99% (r = 10, n = 22) 93% (r = 10, n = 29)
Community Transition Planning N/A 100% (r = 17, n = 18) 94% (r = 17, n = 18)
Family Counseling/ Training 91% (r = 129, n = 904) 94% (r = 109, n = 625) 95% (r = 114, n = 786)
Service Type FY 2017 FY 2018 FY 2019
Group Counseling/ Training 90% (r = 78, n = 561) 95% (r = 75, n = 535) 92% (r = 74, n = 509)
Individual Counseling 94% (r = 150, n = 1980) 96% (r = 132, n = 1390) 97% (r = 137, n = 1583)
Intensive Case Management 94% (r = 13, n = 53) 94% (r = 11, n = 46) 91% (r = 17, n = 81)
Intensive Family Intervention 85% (r = 37, n = 185) 90% (r = 33, n = 149) 88% (r = 32, n = 159)
MH Peer Support - Individual 95% (r = 6, n = 38) 90% (r = 20, n = 92) 95% (r = 27, n = 138)
MH Peer Support Program 86% (r = 25, n = 199) 91% (r = 36, n = 210) 93% (r = 38, n = 238)
Nursing Assessment & Health 80% (r = 116, n = 897) 87% (r = 104, n = 649) 89% (r = 110, n = 734)
Opioid Maintenance Treatment 98% (r = 3, n = 69) 92% (r = 6, n = 52) Not Reviewed
Peer Support Whole Health & Wellness (Group)
68% (r = 12, n = 67)
80% (r = 2, n = 3) Not Reviewed
Peer Support Whole Health & Wellness (Individual)
93% (r = 2, n = 28) 89% (r = 6, n = 25)
Psychiatric Treatment 92% (r = 128, n = 950) 97% (r = 111, n = 717) 97% (r = 119, n = 749)
Psychosocial Rehabilitation Program 93% (r = 26, n = 179) 96% (r = 26, n = 153) 93% (r = 26, n = 804)
Psychosocial Rehabilitation - Individual 85% (r = 98, n = 904) 90% (r = 83, n = 679) 91% (r = 89, n = 143)
Substance Abuse Intensive Outpatient Program Not Reviewed Not Reviewed 88% (r = 9, n = 61)
Quality Management Annual Report FY 2019
51 | P a g e
(73%, N = 114) met or exceeded 90 percent with ten reviews receiving a perfect score, all of whom
were classified as a Small provider (N = 5-10 records per provider).
Individual Counseling was evaluated in 87 percent (r = 137) of all BHQRs conducted in FY 2019 and
continued to demonstrate the highest rate of compliance for the second year compared to all
services measured, with 97 percent of records meeting the measured indicators. Improvements in
scores were found evident for both MH Peer Support – Individual (p < .01) and AD Peer Support –
Individual in FY 2019, increasing results by at least five percentage points. Interestingly, the MH Peer
Support Program was evaluated more than any other specialty service spanning 38 of the 157 BHQRs
while AD Peer Support – Individual as a specialty service was reviewed the least (r = 4). Service areas
demonstrating the greatest decline(s) from FY 2018 to FY 2019, were Peer Support Whole Health
and Wellness – Individual (-4.0%), Community Transition Planning (-5.6%), and Community Support
Team (-6.1%). Although not significant, Intensive Case Management (ICM) had over a three percent
decline in records meeting indicator standards at 91 percent compared to the previous year’s 94
percent, with an additional six reviews (r = 17) conducted in the year.
Concerns over appropriate representation of ICM records across reviews completed in FY 2018 (r =
11) prompted Quality Management to evaluate statewide utilization reports and claims submission.
Through said process, it was evident the current BHQR process did not adequately include medical
records with ICM services. Only one percent of records (n=81) in FY 2019 with ICM services were
included in the BHQR samples, yet 15.3 percent of unique individuals received ICM services
statewide. In FY2020, Quality Management will begin including a set number of ICM records in each
BHQR.
Substance Abuse Intensive Outpatient Program (IOP), a newly reviewed service for FY 2019, resulted
in 88 percent compliance (r = 6, n = 61) with four of the six indicators fully meeting the compliance
with Service Guidelines (100%) necessary standards related to:
Clinical coordination/planning to avoid duplication of intervention evident for those
receiving ACT, CST, or Crisis Residential
Individual meets admission or continued stay criteria
Quality Management Annual Report FY 2019
52 | P a g e
Individual receiving services/support/skill development necessary for sobriety/use
reduction/recovery maintenance
Progress notes document individual response to the staff intervention provided
Although more than half (11 of 19, 57.9%) of the reviewed services met or exceeded the statewide
average of 90 percent for this category, specific indicators at the service level suggest areas for
improvement. Table 9 provides for the lowest-scoring indicator(s) identified, by service, through the
FY 2019 review process.
Table 9. FY 2019 Lowest-Scoring Indicators by Service* “r” represents the number of reviews and “n” the number of records
Service Name and FY 2019 lowest-scoring indicator(s): FY 2017 FY 2018 FY 2019
Assertive Community Treatment 88% (r = 20) 84% (r = 21) 84% (r = 20)
The ACT Team completes a Treatment Plan Review with the staff, the individual, and his/her family/informal supports prior to the reauthorization of services
59% (n = 307)
28% (n = 321)
28% (n = 304)
There is documented evidence that the ACT Team is working with informal support systems/collateral contacts at least 2-4 times per month (with or without the individual present) to provide support and skills training to assist the individual in his/her recovery.
39% (n = 283)
33% (n = 306)
47% (n = 273)
The ACT Team has all required staff. (List missing staff in comment box.)
91% (n = 334)
72% (n = 336)
65% (n = 312)
Addictive Disease Support Services 85% (r = 69) 80% (r = 61) 90% (r = 62)
Coordination with family and significant others is documented and
with adult individual’s permission. (Review authorization period.)
42% (n = 187)
59% (n = 143)
60% (n = 224)
Community Support 83% (r = 114) 86% (r = 85) 85% (r = 105)
There is evidence of service and resource coordination.
50% (n = 572)
63% (n = 445)
62% (n = 541)
Community Support Team 92% (r = 7) 99% (r = 10) 93% (r = 10)
CST Team Meeting logs document that the individual was discussed, even briefly, at least one time per week. (New for FY19)
Not Reviewed Not Reviewed 59%
(n = 29)
Intensive Case Management 94% (r = 13) 94% (r = 11) 91% (r = 17)
There is a joint development of a crisis plan to include the Provider and individual. The Provider is listed as primarily responsible.
85% (n = 52)
65% (n = 46)
69% (n = 80)
Intensive Family Intervention 85% (r = 37) 90% (r = 33) 88% (r = 32)
Quality Management Annual Report FY 2019
53 | P a g e
Table 9. FY 2019 Lowest-Scoring Indicators by Service* “r” represents the number of reviews and “n” the number of records
Service Name and FY 2019 lowest-scoring indicator(s): FY 2017 FY 2018 FY 2019
Documentation reflects a tapering of services. 37%
(n = 63) 75%
(n = 44) 63%
(n = 65) * Areas highlighted in blue within table reflect significance at a p < .01 from the previous year.
The age at which individuals are classified as part of a “child and adolescent (C&A) population” often
depends on the type of service the individual is receiving as well as a multitude of environmental,
physical, and mental factors. Findings were evident in analysis of individuals age 17 and under as
compared to individuals 18 and over. Age group differences were observed in the services of Family
Counseling (n = 25), Group Counseling (n = 6), Individual Counseling (n = 45), Psychiatric Treatment
(n = 17), and Intensive Family Intervention (n = 8). Several indicators were met at 100 percent for
C&A individuals as compared to adults (18+). Indicators not meeting 100 percent for the C&A
individuals across the five service types identified above relate to progress notes documentation and
progress (or lack of) as well as staff intervention in the progress notes relating to interventions listed
on the treatment plan. Results of the two indicators for C&A ranged from 88 percent to 96 percent
for Family, Individual, and Group Counseling yet only 63 percent (n = 5) of IFI progress notes
contained documented progress or lack thereof. Comparatively, these same two indicators were met
at 100 percent for all individuals ages 17 and under for all reviewed records (n = 19) within Nursing
services.
All services reviewed in FY 2019 were scored based on three specific indicators related to
documentation within progress notes. The results indicated significant improvement from the
previous fiscal year (p < .01). Table 10 contains detailed results of these similar indicators and the
results of said indicators by service type for all records reviewed.
Quality Management Annual Report FY 2019
54 | P a g e
Table 10. FY 2019 Progress Note Indicator Results by Service Type* “n” represents the number of records
FY 2017 FY 2018 FY 2019
Progress notes document progress (or lack of) toward goals/objectives identified on the IRP.
86% (n = 7,916) 90% (n = 6,875) 89% (n = 7,974)
Staff interventions reflected in the progress notes are related to staff interventions listed on the IRP.
88% (n = 7,869) 92% (n = 6,897) 93% (n = 7,988)
Progress notes document individual response to staff interventions provided.
98% (n = 7,934) 99% (n = 6,812) 98% (n = 7,996)
FY 2019 Progress Note Indicator Result(s) by Service Type
Service Type
Progress notes document progress (or
lack of) toward goals/objectives
identified on the IRP.
Staff interventions reflected in the
progress notes are related to staff
interventions listed on the IRP.
Progress notes document individual
response to staff interventions
provided.
AD Peer Support - Individual 69% 75% 100%
Addictive Disease Support Services 91% 94% 100%
Assertive Community Treatment 95% 96% 99%
Service Type
Progress notes document progress (or
lack of) toward goals/objectives
identified on the IRP.
Staff interventions reflected in the
progress notes are related to staff
interventions listed on the IRP.
Progress notes document individual
response to staff interventions
provided.
Case Management 90% 93% 98%
Community Support 91% 93% 98%
Community Support Team 93% 100% 100%
Community Transition Planning N/A N/A 100%
Family Counseling/ Training 89% 91% 98%
Group Counseling/ Training 81% 90% 93%
Individual Counseling 94% 96% 99%
Intensive Case Management 90% 93% 100%
Intensive Family Intervention 82% 88% 97%
MH Peer Support - Individual 86% 91% 100%
MH Peer Support Program 78% 93% 99%
Nursing Assessment & Health Services 88% 95% 98%
Quality Management Annual Report FY 2019
55 | P a g e
Peer Support Whole Health & Wellness – Ind. 83% 84% 100%
Psychiatric Treatment 94% 97% 99%
Psychosocial Rehabilitation - Individual 88% 92% 99%
Psychosocial Rehabilitation Program 81% 92% 98%
Substance Abuse Intensive Outpatient Program 69% 70% 100% * Areas highlighted in blue within table reflect significance at a p < .01 from the previous year.
As providers have continued to maintain a high statewide average in the category of Service
Guidelines over the past several years with the existing array of services, Quality Management began
evaluating specific service data as it relates to both services reviewed and services not historically
under review. Over the course of FY 2019, Quality Management consistently reviewed various
detailed reports, such as the Service Mix Report, as well as weekly claim files to ensure accurate
representation of service type and sampling occurred. Quality Management also provided specific
service information and data related to Supported Employment, Crisis Respite Apartments, and
Intensive Family Intervention to DBHDD. Additional data collection was provided to DBHDD related
to individuals who are identified as deaf and hard of hearing (see Office of Deaf Services Special
Review). As a direct result of the reports made available, DBHDD has requested the Collaborative
proceed in FY 2020 to include a set of indicators within the behavioral health tool to include the
additional and previously non-reviewed services: Supported Employment and Crisis Respite
Apartments. Furthermore, Intensive Case Management (ICM) will be evaluated, starting in FY 2020,
as a “carve out” or subset service review (similar to the treatment of Assertive Community
Treatment) using a sample of 15 records unique to that service.
Office of Deaf Services Special Reviews
Included within the Collaborative’s review process, DBHDD may request a “Special Review” that does
not fit into the review categories previously defined. In October 2018, DBHDD requested a special
review of records and services for individuals identified as deaf or hard of hearing using both the
provider manual and DBHDD policies8 15-111, 15-112, and 15-114 as references for review tool
8 DBHDD policies can be viewed in their entirety via DBHDD PolicyStat website, https://gadbhdd.policystat.com/.
Quality Management Annual Report FY 2019
56 | P a g e
development. The purpose of these special reviews was to gather information regarding the
provision of Non-Intensive Outpatient (NIOP) services to deaf and hard of hearing individuals. As the
purpose was to gather data, no scores or final report was provided and no exit conference was
conducted with the provider. Additionally, the results of these special reviews were not posted to the
Collaborative’s public website. Results of the special review were provided to DBHDD.
Sixteen Tier 1 providers were selected to be included in the special review. A separate notice specific
to this review was sent to providers via email 24 hours prior to the start of the review. When
applicable, the special review occurred in conjunction with a regularly-scheduled BHQR or CSUQR to
reduce administrative burden for the provider.
The DBHDD Office of Deaf Services (ODS) special review consisted of a review of individual records,
review of provider policy and procedures related to services for individuals identified as deaf or hard
of hearing, and site visits for each outpatient site. The results were informative in that results of the
questions (compiled across all 16 agencies and 69 individual records) were low for all record review
questions and agency-level programmatic questions as illustrated in Table 11.
While reviewing records, it was apparent that providers had not been consistently informing ODS
when an individual who is deaf or hard of hearing presented for services. None of the medical
records reviewed had a copy of the Communication Assessment Report filed; however, a handful of
IRPs contained information regarding the individual’s communication preference.
Table 11. Office of Deaf Service Special Review Results
ODS Agency Programmatic Review Results (N = 16)
The DBHDD Office of Deaf Service’s (ODS) American Sign Language Poster prominently displayed in all community behavioral health provider’s lobbies or waiting rooms (NIOP sites only).
81% (n = 16)
ODS Record Review Results (N = 69)
Provider notified ODS w/in two business days of first contact w/individual served (for individuals who entered into services after March 2017).
21% (n = 33)
Communication Assessment Report (CAR) is in the medical record. 0% (n = 60)
If individual indicated a change in preference to utilize (or not) ASL-fluent services, provider contacted ODS.
NA
Communication Assessment Report is addressed on IRP to include individual's preferred mode of communication.
3% (n = 60)
Quality Management Annual Report FY 2019
57 | P a g e
As a direct result of findings, the Collaborative, with DBHDD’s approval, has crafted a set of
indicators to collect and report data regarding services for individuals who are deaf or hard of
hearing to be added to the Overall Programmatic and Service Guidelines tools for use in scoring
BHQRs.
BHQR Programmatic (Non-scored)
In FY 2019, a category entitled “Programmatic” was piloted as part of the quality review process as a
non-scored element. This was as a direct result of commonly asked indicators across various services
that vary based on the standards set specifically for the service (i.e. policies and procedures being
present, staff caseloads, and credentialing requirements). This Programmatic section encompasses a
separate series of indicator(s) answered once per BHQR and based on the service(s) reviewed and
provided by a provider. By implementing indicators reviewed only once per service rather than per
record, Quality Management avoids possible redundancy in deductions or increased inflation in
Service Guidelines scoring.
Programmatic-level indicators exist for 11 different services and are answered based upon whether a
provider offers that particular service. There is also a set of three indicators answered at each
review for all providers. The results from FY 2019 are as follows, noting in parentheses the number
of reviews (N) and the percentage of met:
All Provider Indicators (N = 157)
Registered Nurse (RN) is present 10 hours per week (91%)
Staff safety and protection policies and procedures are present (96%)
Where applicable, all services provided at approved Medicaid sites (94%)
Addictive Disease Support Services, ADSS (N = 56)
• Individual to staff caseload ratio is met (98%)
Case Management, CM (N = 80)
• Oversight of Case Manager (CM) is provided by an independently licensed practitioner (99%)
Notification of Right to Free American Sign Language Services and Accommodations form is in the individual’s medical record (for individuals who have been in contact with the provider since August 24, 2018).
8% (n = 24)
Quality Management Annual Report FY 2019
58 | P a g e
Community Support Team, CST (N = 11)
• CST has all required staff (64%) • CST individual to staff ratio is within parameters (100%)
Peer Support Whole Health & Wellness Individual (N = 8)
• Minimum monthly team meetings are documented and include the Certified Peer Specialist
(CPS), Certified Peer Specialists in Whole Health (CPS-WH) and Registered Nurses (RN) (38%)
• The CPS-to-individual ratio is not exceeded (100%)
• The CPS-WH is supervised by a licensed practitioner (100%)
• The service is provided by a RN or CPS-WH (100%)
Intensive Case Management, ICM (N = 17)
• Minimum staff requirements are met (47%)
• Oversight of ICM is provided by an independently licensed practitioner (94%)
• Staff-to-individual ratios for ICM are met (75%)
• The organization has policies that govern the provision of services in natural settings… (75%)
Intensive Family Intervention, IFI (N = 32)
• All team members are dedicated to this IFI organization and one specific team (88%)
• No more than 50% of staff are "contracted"/1099 team members (78%)
• The IFI team's family-to-staff ratio has not been exceeded (100%)
• The organization has policies that govern the provision of services in natural settings (94%)
MH Peer Support Program (N = 36)
• At least two Georgia CPS' are on staff (92%)
• Individual-to-CPS ratio is not exceeded (100%)
• Individual-to-direct service/program staff ratio is not exceeded (100%)
• Program leader is present at least 75% of the time (94%)
• The CPS Program leader is supervised by an appropriately-credentialed professional (92%)
• The Program is operated the required amount of time (97%)
• The Program Leader is a CPS who is also a Certified Peer Recovery Specialist (CPRP) (66%)
Assertive Community Treatment, ACT (N = 19)
• ACT has policies and procedures governing outreach services and staff safety (79%)
• At least one full-time RN is employed by the ACT team 32-40 hours per week (95%)
Quality Management Annual Report FY 2019
59 | P a g e
• Emergency procedures/protocols present including 24/7 availability of staff skilled in crisis
intervention (100%)
• On-call coverage is provided by ACT staff (100%)
• The ACT team maintains a small consumer-to-clinician ratio (78%)
• The psychiatrist participates in weekly team meetings (85%)
Psychosocial Rehabilitation Program (N = 25)
• At least one CPRP is on site face-to-face at all times (88%)
• Individual-to-direct service/program staff ratio is not exceeded (91%).
• The program is operated the required amount of time (96%)
• The program is supervised by an appropriate-licensed practitioner (100%)
While success is evident that several services received a perfect 100 percent “met”, some programs
hold deficits in areas such as employing all required staff for a service or program (CST, MH Peer
Support Program, and ICM) or maintaining the prescribed staff to consumer ratio (ACT). Since the
pilot project proved successful and informative through FY 2019, Quality Management is continuing
to collect this information as non-scored data through FY 2020.
BHQR Focused Outcome Areas (FOA)
The indicators within the Focused Outcome Areas tool are answered once per record reviewed. Each
FOA has a different number of indicators for a total of 21 indicators. Tool revisions implemented in
the first quarter of FY 2019 included adding an additional element of data collection for reasons
within the FOA subcategories for indicators “not met.” Figure 10 displays results of the overall FOA
results for FY 2019 compared to previous years. Additionally, each respective section contains a
figure specific to the subcategory FY 2019 results as compared to previous years and a synopsis of
highlights and trends occurring as a result of FY 2019 reviews.
FY 2017
89%
FY 2018
92%
FY 2019
94%
Quality Management Annual Report FY 2019
60 | P a g e
Figure 10. BHQR Focused Outcome Area(s) Scores by Year
Figure 11. BHQR Focused Outcome Area by Year
Whole Health indicators address whether the records reviewed demonstrated individuals were
treated holistically, with their physical health needs being assessed, documented, and monitored. Of
the six FOAs, Whole Health (with results shown in Figure 11 and historically the lowest-scoring) has
surpassed the subcategory of Safety for the second year in a row. Whole Health, for FY 2019, had a
statistically significant (p < .01) increase in the average score of 91 percent compared to the 84
percent result in FY 2018. Important to note is that one indicator was eliminated from FY 2019 data
collection due to redundancy as the area is intentionally addressed in the Assessment and Planning
FY17: 89%(N = 167)
FY18: 92%(N = 156)
FY19: 94%(N = 157)
Whole Health FY17: 74%FY18: 84%FY19: 91%
Safety FY17: 83%FY18: 78%FY19: 80%
Person Centered FY17: 91%FY18: 95%FY19: 94%Community
FY17: 93%FY18: 96%FY19: 97%
Choice FY17: 96%FY18: 97%FY19: 97%
Rights FY17: 93%FY18: 93%FY19: 94%
91%
84%
74%WholeHealth
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
Quality Management Annual Report FY 2019
61 | P a g e
category: The person has current medical conditions and documentation demonstrates these
conditions are assessed, monitored, and recorded. As the lowest-scoring indicator of Whole Health
(FY 2018, 77%), it is likely that the removal of this indicator from the set generated a slight
improvement in the overall provider FOA scores as it pertains to Whole Health.
Figure 12. BHQR Focused Outcome Area by Year
Safety indicators address whether providers were attending to certain risk factors for individuals
such as their tendency to experience a crisis, have suicidal or homicidal thoughts or actions, and
receive information and education about the risks and benefits of prescribed medications. Safety
represents the lowest-scoring FY 2019 FOA, yet slightly increased by two points to 80 percent; see
Figure 12.
For FY 2019, the indicator most often scored “No” was “Individual (or their legal guardians)
has been educated on the risk/benefits of all medication prescribed and there is a signed
consent form.” Results of this indicator have trended upward (p < .01) since contract
inception to a result of 73 percent (58% in FY 2017; 68% in FY 2018). Of the 27 percent of
records not meeting this criteria, 44 percent, or 272 of the 622 responses identified the
“form was not present” while 259 records (42%), had a form present but, “lacked at least one
medication prescribed within the last six months” (C&A 85 percent, Adult 72 percent).Results
of documentation as it relates to how providers ensured that a, “Safety/Crisis plan was
developed, as needed, that directs, in advance, the individual's
desires/wishes/plans/objectives in the event of a crisis,” significantly improved (p < .01) from
FY 2018 to FY 2019 with a score of 86 percent. Of those scored not met, thirty percent were
80%
78%
83%
Safety
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
Quality Management Annual Report FY 2019
62 | P a g e
missing the plan and forty percent did not have the necessary signatures of both the
individual and a clinician (C&A 85 percent, Adult 86 percent).
Figure 13. BHQR Focused Outcome Area by Year
Person Centered Practices measure whether documentation indicates individuals actively
participated in creating or modifying the plans as needed and desired. While the score of Person
Centered Practices improved in FY 2018, a slight decline of one point occurred in FY 2019. Figure 13
provides results for fiscal years 2017 through 2019. The indicator, “Plan reassessed based upon any
changing needs, circumstances or response by the individual” had the lowest score in the
subcategory for the second year in a row at 88 percent. Only 76 percent of records of individuals 17
and under met this requirement as compared to 89 percent of record of individuals 18 and over (p
< .01). Furthermore, of the 445 records reviewed specific to this indicator, 71 percent or 335 records
did not meet the standard as the plans had not been updated based on the individuals’ current
needs/circumstances. Additionally, documentation demonstrating the individual received
individualized services also declined (p <.01) from 99 percent to 96 percent in FY 2019. This decline
was due to a variety of reasons such as duplicated notes or repeated interventions (n = 55), the plan
did not address identified needs (n = 25), interventions were not meeting needs (n = 22), there were
significant duplicated elements (n = 21), and planned services or service intensity was not meeting
individuals’ needs (n = 20).
94%
95%
91%PersonCenteredPractices
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
Quality Management Annual Report FY 2019
63 | P a g e
Figure 14. BHQR Focused Outcome Area by Year
Community Life indicators address how individuals were engaged in their communities of choice and
whether they held valued social roles. Figure 14 graphically presents the annual fiscal year result for
the Community Life FOA. All indicators within Community Life exceeded 90 percent again in FY 2019
with significant improvement noted (p < .01) in “Documentation demonstrates the individual has
age-appropriate responsibilities/valued roles in the community as desired.” Minimal differences
were found in Community Life indicators for C&A versus adult records reviewed.
Figure 15. BHQR Focused Outcome Area by Year
Choice indicators address how, and if, providers offered individuals options of services and
encouraged individuals to make educated choices concerning supports and services provided. Figure
15 provides results of the Choice FOA by fiscal year. Results at the indicator level remained equal to
FY 2018 results wherein:
If a barrier was identified, documentation demonstrated alternatives were explored (C&A 94
percent, Adult 95 percent).
97%
96%
93%Community
Life
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
97%
97%
96%
Choice
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
Quality Management Annual Report FY 2019
64 | P a g e
The individual’s known preferences and differences were followed (98% both populations)
Documentation demonstrated how the individual was provided with options of supports and
services (98% both populations)
Figure 16. BHQR Focused Outcome Area by Year
Rights indicators address whether individuals were apprised of their rights while receiving services at
the point of admission to services and on an annual basis thereafter, and whether they had been
informed of their rights under Federal HIPAA laws. There was minimal change from FY 2017 to FY
2018 with an overall subcategory score of 93 percent; however, an additional percentage point
increase occurred in FY 2019, see Figure 16. Nearly all records in FY 2019 (99%) demonstrated,
“HIPAA Privacy and Security Rules were reviewed with the individual,” and 97 percent of records
“had a signed formal acknowledgment of rights and responsibilities at the onset of services,
supports, and treatments.” Attention is still needed specific to “Individuals informed of rights at
least annually,” as only 79 percent of Adult records and 85 percent of C&A records reflected annual
updates; however, this was an increase (p < .01) of six points to the indicator for FY 2019.
BHQR Staff and Individual Interviews
Six FOA categories comprise the Staff and Individual Interviews. Individual Interviews served to
assess an individual’s perception of care with the provider, services rendered, and support in
working toward personal goals. Staff Interviews helped determine whether a person-centered
approach was used in providing services and empowering individuals. Data in Figure 17 represent
the average scores for the individual and staff FY 2019 interviews.
94%
93%
93%
Rights
FY 2017 (N = 167) FY 2018 (N = 156) FY 2019 (N = 157)
Quality Management Annual Report FY 2019
65 | P a g e
Figure 17. BHQR Interview Data by Fiscal Year
It is notable that all FOA areas, when reviewed separately, scored in the 90th percentile since contract
inception, indicating a high level of satisfaction for individuals served by the providers. Furthermore,
little difference was identified between scores of similar indicators asked of both individuals and
staff. As a direct result of such positive findings related to the individual and staff interviews over a
four-year period, it has been recommended by the Collaborative (and approved by DBHDD) that staff
interviews be eliminated beginning FY 2020 and to edit and minimize the Individual Interview to a
set of ten questions. This recommended change to the interview processes establishes a new aim of
continuing to assess/identify the individual’s perception of care and satisfaction with their treatment
provider while simultaneously decreasing any administrative burden to the staff and individual
served that the original interview process may have caused.
Assertive Community Treatment (ACT) Quality Reviews Quality Management conducted 20 ACT reviews in FY 2019 specific to 18 unique providers. The ACT
reviews contained 313 records and 3,249 billing claims. Seventeen BHQRs were completed for
providers who offered both Non-Intensive Outpatient (30 records) and ACT services (additional ACT-
specific 15 records). One provider offered only ACT services and the sample was thirty individuals.
FY 2017
96.2%
(N = 735)
FY 2018
97.6%
(N = 616)
FY 2019
97.8%
(N = 572)
INDIVIDUALINTERVIEW
FY 2017
98.1%
(N = 774)
FY 2018
98.9%
(N = 652)
FY 2019
99.0%
(N = 620)
STAFF INTERVIEW
Quality Management Annual Report FY 2019
66 | P a g e
Two providers offering ACT services fell below the 90 percent threshold for Overall and
Billing score and required a second review over the course of the year. The two providers
offered both ACT and Non-Intensive Outpatient (NIOP) services. Both were on the once per
year frequency in FY 2018 (Three different ACT providers fell under the threshold in FY
2018).
Eighty-eight percent of ACT records reviewed documented a mental health diagnosis ( the
same percentage as FY 2018). Overall, the ACT records documented the possibility of co-
occurrence of other conditions as well within the population sampled:
o Fourty-four percent identified a substance use diagnosis
o Fourty-four percent identified a physical diagnosis
o Four percent of records confirmed an IDD diagnosis
ACT Quality Review Overall Scores
Figure 18 provides ACT statewide averages, by category, for the past three years of quality reviews.
The Overall score of 89 percent in FY 2019 was the same as in FY 2018.
Quality Management Annual Report FY 2019
67 | P a g e
Figure 18. Assertive Community Treatment (ACT) Results by Category and Year
OverallFY17: 87%(N = 20)*
FY18: 89% (N = 21)*
FY19: 89%(N = 20)*
Billing Validation
FY17: 90%FY18: 91%FY19: 88%
Focused Outcome
Areas FY17: 90%FY18: 93%FY19: 93%
Assessment &Planning
FY17: 80% FY18: 87%FY19: 90%
Service GuidelinesFY17: 88%FY18: 84%FY19: 84%
*ACT only providers: FY17: 2; FY18: 1; FY19: 1
Quality Management Annual Report FY 2019
68 | P a g e
Table 12. BHQR and ACT Results by Category by Year
Billing Validation FOA
Assessment & Planning
Service Guidelines Overall
BHQR FY 2017 84% 89% 77% 88% 84%
BHQR FY 2018 85% 92% 84% 90% 88%
BHQR FY 2019 86% 94% 88% 90% 90%
ACT FY 2017 90% 90% 80% 88% 87%
ACT FY 2018 91% 93% 87% 84% 89%
ACT FY 2019 88% 93% 90% 84% 89%
Table 129 provides ACT review results by category for FY 2017 through FY 2019 compared to the
BHQRs to demonstrate how ACT quality review results align with trending of all BHQR results. Of
note is that results of the ACT FOAs and Service Guidelines remained static when compared to FY
2018. Furthermore, Assessment and Planning, the lowest-scoring area since inception, has steadily
trended upward since FY 2017 reaching a result of 90 percent, which is comparable to BHQR’s
Assessment and Planning result of 88 percent. Lastly, while Billing results trended upward from
contract inception in BHQR’s averages, ACT results and averages as a subset of BHQRs decreased
from a 91 to 88 percent in FY 2019.
ACT Billing Validation
Figure 19 shows the total dollar amount reviewed through ACT claims analysis, as well as the dollar
amount found to be unjustified and justified across the years. While fewer funds were reviewed in FY
2019 ($318,879.82), the rate of justified amounts has declined from previous years to 88 percent.
9 ACT services are reviewed as a part of the BHQR process. For the purpose of this annual report, ACT scores have been extracted as an additional
subset population only for specific service evaluation, monitoring, and analysis.
Quality Management Annual Report FY 2019
69 | P a g e
Figure 19. ACT Billing Validation by Fiscal Year
Specific billing discrepancy reasons found during the ACT reviews are provided in Table 13 for the
3,249 claims reviewed in FY 2019. Assessors select all applicable discrepancy reasons for all reviewed
claims; therefore, one claim may have multiple discrepancy reasons. While the prevalence of
previously identified common discrepancies continued to decline, other reasons for discrepancies
became more pronounced. These include failure to document location of services or “Missing
location” (97 instance in FY 2019 [3.0%]), “Content does not support code billed” (35 instances in FY
2019, [1.1%]) or “Content does not support units billed” (52 instances in FY 2019 [1.6%]) as well as
“Staff credential not supported by documentation” (100 instances [3.1%]). Moreover, several
discrepancies became null for the second year in a row with zero instances identified and include:
Individual receiving services does not meet admission criteria for service billed
Missing/incomplete order
Time in/out missing
$0.00
$50,000.00
$100,000.00
$150,000.00
$200,000.00
$250,000.00
$300,000.00
FY 2017 FY 2018 FY 2019
$307,720.24 90%
$296,038.97 91%
$281,284.86 88%
$32,707.80 10%
$29,806.63 9%
$37,594.96 12%
Amount Justified
Amount Unjustified
Total Amount Reviewed in FY 2017: $340,428.04Total Amount Reviewed in FY 2018: $325,845.60Total Amount Reviewed in FY 2019: $318,879.82
Quality Management Annual Report FY 2019
70 | P a g e
Table 13. ACT BHQR Billing Discrepancy Reasons by Year
FY 2017 FY 2018 FY 2019 Percent Change FY17–FY18
Percent Change FY18-FY19
Claims Reviewed 3,221 3,427 3,249
Not meeting admission criteria for service 20 (0.6%) 0 (0.0%) 0 (0.0% -100% 0%
Missing/incomplete order 12 (0.4%) 0 (0.0%) 0 (0.0%) -100% 0%
No valid, verified diagnosis NA NA 3 (0.1%) NA NA
Quantitative Instances Instances Instances Change Change
Staff credential not supported NA NA 100 (3.1%) NA NA
Location missing (out-of-clinic) 32 (1.0%) 32 (1.0%) 97 (3.0%) 0.00% > 100%
Progress note is missing 17 (0.5%) 35 (1.0%) 12 (0.4%) -80.0% -63.1%
Code is missing / different than code billed 19 (0.6%) 13 (0.4%) 6 (0.2%) -33.3% -53.8%
Staff credential missing 40 (1.2%) 15 (0.4%) 5 (0.2%) -66.70% -61.5%
Units billed exceed time / units documented 24 (0.7%) 10 (0.3%) 3 (0.1%) -57.1% -69.2%
Date of service incorrect/missing 3 (0.1%) 1 (<0.1%) 2 (0.1%) -69% > 100%
Date of entry missing 22 (0.7%) 4 (0.1%) 1 (<0.1%) -85.7% -90.0%
Signature missing 25 (0.8%) 4 (0.1%) 1 (<0.1%) -87.5% -74.4%
Time in / time out missing 10 (0.3%) 0 (0.0%) 0 (0.0%) -100% 0%
Performance Standards Instances Instances Instances Change Change
Content does not support units billed 51 (1.6%) 32 (0.9%) 52 (1.6%) -43.8% 77.8%
Content does not support code billed 24 (0.8%) 34 (1.0%) 35 (1.1%) 25.0% 7.7%
Intervention outside staff's scope of practice 0 (0.0%) 42 (1.2%) 31 (1.0%) 100% -21%
Content is not unique to the individual 5 (0.2%) 56 (1.6%) 29 (1.6%) > 100% -44.2%
Non-billable activity 8 (0.2%) 8 (0.2%) 11 (0.3%) 0.0% 69.3%
Multiple services billed at the same time 2 (0.1%) 6 (0.2%) 9 (0.3%) 100% 38.5%
Content does not match service definition 5 (0.2%) 3 (0.1%) 7 (0.2%) -50.0% > 100%
* Due to the low response volume, Quality did not conduct statistical difference in proportions testing on the above table.
The Billing Validation score is assigned to each provider as the percent of justified billed dollars
divided by the total paid dollars for the reviewed claims. The statewide average ACT Billing score was
88 percent in FY 2019. This remained consistent with the FY 2017 and FY 2018 results of provider
scores demonstrating ACT providers had minimally fewer funds identified as unjustified (12%)
compared to all combined services of the BHQR reviews overall (13%).
Quality Management Annual Report FY 2019
71 | P a g e
ACT Assessment and Planning
Figure 20 provides the ACT Assessment and Planning score results by specific indicator annually since
FY 2017. Similar to BHQR results, these indicators remain the lowest-scoring in FY 2019:
Co-occurring health conditions addressed on IRP
All assessed needs are addressed on the IRP
Whole health and wellness addressed on IRP
Discharge plan defines criteria
Whole health and wellness improved by eight points, co-occurring condition documentation
increased by 25 points, while both assessed needs and discharge criteria significantly improved (p
< .01) by 26 and 31 points respectively. All other indicators reviewed specific to ACT met or
exceeded 90 percent for a consecutive third year.
Quality Management Annual Report FY 2019
72 | P a g e
Figure 20. ACT Assessment and Treatment Planning Scores by Year
ACT Service Guidelines
The ACT Service Guidelines score contained 16 indicators in FY 2019 with the following indicators
added or eliminated from the tool for FY 2019, based on the DBHDD Provider Manual and
discussions with DBHDD clinical leadership:
Indicator added: “Documentation reflects individual is seen face to face at least 8 times per
month (excluding those individuals transitioning to a lower level of care or enrolled in CTP
during the month)” (96%)
99%
99%
97%
97%
94%
91%
46%
76%
51%
49%
100%
100%
97%
96%
97%
89%
66%
86%
68%
70%
100%
99%
97%
95%
94%
90%
87%
84%
77%
74%
Individual meets admission criteria and the diagnosisis verified at least annually.
Current medical screening is present
Interventions/objectives are goal-linked & service-consistent
IRP is individualized in personalized language
Goals/objectives honor hopes, choice, preferences,outcomes
Current behavioral health assessment
Discharge plan defines criteria
Whole health & wellness in IRP
All assessed needs are addressed
Co-occurring health conditions addressed in IRP
FY 2017 (N = 20) FY 2018 (N = 21) FY 2019 (N = 20)
Quality Management Annual Report FY 2019
73 | P a g e
Indicator added: “Following admission to a psychiatric facility, the ACT Team is involved in
each individual’s discharge planning” (88%)
Indicator added: “If provided in conjunction with SAIOP, documentation includes evidence
that, without SAIOP, the individual is unable to maintain sobriety and there is clinical
coordination/ planning to avoid duplication of service interventions” (89%)
Indicator eliminated: “There is documentation of individual's responses and a discussion to
the agreement of services identified in the treatment planning/individual recovery planning”
Table 14 shows the item-level detail to illustrate the percent change across similar indicators and
compare fiscal year results within the ACT Service Guidelines. The descending ranked order of
indicators generated the same exact ranked order for similar indicators from FY 2018 to FY 2019.
Furthermore, the ACT Service Guidelines category score decreased significantly (p < .01) from FY
2017 (88%) to FY 2018 (84%) yet remains unchanged from FY 2018 to FY 2019. Yet again in FY 2019,
the same three indicators scored at 100 percent for all three fiscal years:
“Individual meets admission or continuing stay criteria”
“One of the contacts per month addresses the symptom assessment and management of
medications (once a month)”
“For individuals who were discharged, there are multiple documented attempts to locate and
make contact with the individual prior to discharge (over a 45-day period)”
While success is evident with 11 of the 16 indicators exceeding 90 percent, minimal fluctuations did
occur by approximately one percent either positively or negatively for indicators resulting in 90-99
percent. Important to note is that the ACT team with all required staff substantially declined (p < .01)
in FY 2018 by 21 percent and further declined by an additional six points in FY 2019. “ACT team
completes a treatment plan review with the staff, the individual, and his/her family/informal
supports prior to the reauthorization of services” was the lowest-scoring indicator for the second
consecutive year, with evidence in only 28 percent of records. Furthermore, evidence the “ACT team
is working with informal support/contacts at least two to four times per month prior to
reauthorization” showed improvement (p < .01) by thirteen points to a result of 47 percent.
Quality Management Annual Report FY 2019
74 | P a g e
Table 14. ACT Service Guidelines Indicator Scores
FY 2017 FY 2018 FY 2019
Percent Change
FY17-FY18
Percent Change
FY18-FY19
The individual meets admission / continuing stay criteria / diagnostic criteria.
99% 100% 100% 1.0% 0.0%
For discharged individuals, there are multiple documented attempts to locate and make contact with the individual prior to discharge (over a 45-day period).
100% 100% 100% 0.0% 0.0%
One of the contacts per month addresses the symptom assessment and management of medications (1/month).
99.7% 100% 100% 0.3% 0.0%
The progress notes document individual response to the staff intervention provided.
99% 98% 99% -1.0% 1.0%
The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.
97% 96% 96% -1.0% 0.0%
There is documentation of individual’s involvement in transition planning.
90% 96% 95% 6.7% -1.0%
Progress notes contain documentation of the individual’s progress (or lack of) toward specific goals/objectives on the treatment plan.
99% 94% 95% -5.1% 1.1%
The ACT team is working with the individual toward educational or vocational needs, interests, per IRP (once per authorization).
97% 94% 96% -3.1% 2.1%
There is documentation to support when substance use services are needed and are integrated into the treatment plan.
93% 92% 92% -1.1% 0.0%
Following admission to a psychiatric facility, the ACT team is involved in each individual’s discharge planning.
95% 91% 88% -4.2% -3.3%
The ACT team has all required staff. 91% 72% 66% -20.9% -8.3%
There is evidence the ACT team is working with informal support systems/collateral contacts at least 2-4 times per month with/without the individual present to provide support and skills training to assist the individual in his/her recovery.
39% 34% 47% -12.8% 38.2%
The ACT team completes a treatment plan review with the staff, the individual, and his/her family/informal supports prior to the reauthorization of services.
59% 28% 28% -52.5% 0.0%
ACT Service Guidelines Result 88% 84% 84% -4.5% 0.0%
Quality Management Annual Report FY 2019
75 | P a g e
Table 14. ACT Service Guidelines Indicator Scores
FY 2017 FY 2018 FY 2019
Percent Change
FY17-FY18
Percent Change
FY18-FY19
(N = 20) (N = 21) (N = 20)
ACT Focused Outcome Areas
Focused Outcome Area (FOA) indicators are answered once per record, with the same FOA indicators
(21) at the BQHR. The FY 2018 Overall score for ACT FOA increased (p < .01) from FY 2017 by three
points (93%) and remained steady at 93 percent in FY 2019. The same FOAs met or exceeded 95
percent in FY 2019 as compared to prior years (see Figure 21). This included the FOAs of Whole
Health, Person Centered Practices, Community, and Choice. A minimal decline in the subcategories
of Safety and Rights occurred in FY 2018 for ACT specific services, yet in FY 2019 results slightly
improved for the two FOAs. Indicators identified as requiring improvement due to low scores across
all ACT reviews in these subcategories include “Annual updates to rights” and “Documentation of
medication education and consent.” In FY 2018, both of these indicators resulted in 70 percent of
ACT records meeting the necessary criteria and while annual rights updates declined to 67 percent,
medication education and consent documentation improved from the previous year to a FY 2019
result of 71 percent. The greatest improvement in the FOA subcategory scores specific to ACT
reviews continues to be Whole Health. This FOA, specific to individuals receiving ACT services,
continued to significantly (p < .01) trend upward for the fourth year. Whole Health significantly (p
< .01) increased from 90 percent in 2017, to 93 percent (p < .01) in FY 2018, and further
demonstrated positive efforts to a FY 2019 result of 96 percent.
Quality Management Annual Report FY 2019
76 | P a g e
Figure 21. ACT FOA Scores by Category by Year
Crisis Stabilization Unit Quality Reviews Crisis Stabilization Unit Quality Reviews (CSUQR) began in January of FY 2016. The CSUQR assesses
the provider’s overall practices, quality of service delivery, as well as determine adherence to DBHDD
standards through individual record reviews.10 When possible, Quality Management conducted the
CSUQR and BHQR simultaneously; Crisis Stabilization Unit (CSU) providers received separate CSUQR
scores and final assessment reports.
The CSUQR indicators are based on DBHDD policies and the Provider Manual for Community
Behavioral Health Providers and are organized into three review categories: Individual Record Review
(IRR), Service Guidelines, and FOAs. The provider review score for each category represents the
percent of applicable indicators met or present within that category. The CSUQR Overall score is
10 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html
FY17: 90%(N = 20)
FY18: 93%(N = 21)
FY19: 93%(N = 20)
Whole Health FY17: 90%FY18: 95%FY19: 96%
Safety FY17: 82%FY18: 80%FY19: 81%
Person Centered FY17: 91%FY18: 96%FY19: 95%Community
FY17: 94%FY18: 97%FY19: 96%
Choice FY17: 97%FY18: 97%FY19: 96%
Rights FY17: 87%FY18: 86%FY19: 88%
Quality Management Annual Report FY 2019
77 | P a g e
calculated by averaging the three categories, with each category accounting for 33.3 percent of the
Overall score.
Quality Management conducted reviews at 19 eligible CSU providers with three of the 19 providers
requiring a second review during the measurement year as a result of not meeting the threshold (90
percent Overall score). Results for the reassessment reviews were included within the overall results
for a total of 22 reviews. Five of the 19 providers offered services to children and adolescents (C&A)
(age[s] 17 and under) as well as adults; the remaining CSU providers served adults only.
Crisis Stabilization Unit Sampling Method
A random sample of 15 individuals who had received services within the three months preceding the
review was selected for each record review. However, when providers did not have an adequate
number of individuals served in the three months (a minimum of three days’/nights admission)
preceding the review, samples were pulled from individuals served up to six months preceding the
review. In FY 2019, all CSUQRs had the maximum 15 records reviewed per review. While
stratification of CSUQR samples does not occur to account for specific diagnoses type or other
demographics, the information is collected throughout the year with findings reported here within
the annual report. Figure 22 demonstrates the co-occurrences of diagnoses identified within the
sample for FY2019 compared to co-occurrences of diagnoses identified within BHQRs, as significant
differences (p < .01) have been identified. Some of these differences (p < .01) included:
Mental health diagnoses were most prevalent in 85 percent of the BHQR sample, whereas a
mental health diagnosis was evident in 80 percent of CSUQR individuals.
Substance use diagnoses were nearly double in CSUQRs (52%) than in BHQRs (29%).
Important to note: Individuals are often admitted to a CSU for detoxification purposes.
Co-occurring physical diagnoses identified were evident in 41 percent of BHQR records
compared to 17 percent of CSU records sampled.
Diagnoses related to IDD were more evident in the BHQR sample than the CSUQR sample,
with only 2% contained in the CSUQR.
Quality Management Annual Report FY 2019
78 | P a g e
Figure 22. FY 2019 Diagnostic Categories Observed by Review Type
Also noted throughout the CSUQR section of this report are differences noted between adults (ages
18+) and C&A. CSUs serving both populations of adults and C&A have an attempted sample of up to
ten adults and five C&A selected. FY 2019 CSUQRs consisted of 330 individual records, 316 adult
records, and only 14 C&A records. The limited number of C&A records sampled was likely due to
limited individuals served respective to the population as only five CSUs provide services to the C&A
population. Additional reasons for the limited number of C&A records included minimal claims
billed; use of oversample needed for reasons such as individuals’ length of stay was less than three
days, or for other reasons established during sample development or at the direction of DBHDD.
The CSUQR mirrors the BHQR scoring and sampling processes for the Individual Interviews and Staff
Interviews (selected by the provider and Quality Assessors). Results from the interviews were not
included in calculating the provider’s overall CSUQR score. Quality Assessors attempt at each review
to complete a minimum of five Individual and five Staff Interviews per CSUQR; however, the actual
number fluctuated based on individual and staff availability, their agreement to participate in the
interview process, the number of staff, individual’s stability or acuity level, and the number of
individuals the provider served at the time of the review. Individuals selected for interviews were
currently at the CSU, and the staff selected were providing services on the CSU. If an individual or
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
FY 2019 BHQR
FY 2019 CSUQR
IDD, 7%
IDD, 2%
Physical, 41%
Physical, 17%
Substance Use, 29%
Substance Use, 52%
Mental Health, 85%
Mental Health, 80%
Mental Health Substance Use Physical IDD
Quality Management Annual Report FY 2019
79 | P a g e
staff declined an interview, CSU staff selected a different individual or staff, if possible. In FY 2019,
104 interviews of individuals receiving CSU services and 106 staff interviews were conducted.
Crisis Stabilization Unit Quality Review Overall Scores
Figure 23 shows the Overall and category scores for reviews completed the past three years (FY
2017, FY 2018, and FY 2019). While CSUQRs category scores have progressively increased over the
years since contract inception, FY 2019 showed declines in both the Individual Record Review (p
< .01) as well as the Service Guidelines score(s), thereby decreasing the Overall statewide score
result from an 88 percent (FY 2018) to 87 percent (FY 2019). See distributions in Appendix B. The
FOA category continued to trend positively to be the highest amongst all categories reviewed across
all review types (BHQR, ACT, and CSUQR), for the third consecutive year. In FY 2019, the CSUQR
result for FOAs siginificantly improved (p < .01) 93 percent compared to 91 percent in FY 2018.
Figure 23. CSU Overall Scores by Fiscal Year
Overall ScoreFY17: 86%
(N = 23)FY18: 88%
(N = 21)FY19: 87%
(N= 22)
Individual Record Review
FY17: 80%FY18: 83%FY19: 79%
Focused Outcome
Areas
FY17: 91%FY18: 91%FY19: 93%
Service Guidelines
FY17: 87%FY18: 91%FY19: 89%
Quality Management Annual Report FY 2019
80 | P a g e
Three CSU providers had a second review during FY 2019 due to Overall scores that were below the
threshold (90 percent). The results of these three specific CSUQR reassessment reviews are
contained within the reassessment review section of this report in Figure 32.
CSUQR Individual Record Review (IRR)
Individual Record Review (IRR) indicators were answered once per record reviewed. The IRR is
comprised of six subcategories: Assessment and Planning, Admission and Initial Evaluation, CSU
Course of Stay, Documentation, Transition/Discharge Summary, and CSU Planning. Each of the six
subcategories had a unique number of indicators, with 32 total scored within the IRR category.
Additionally, there is a non-scored subcategory that includes four important elemental standards
used for data collection and reporting. In FY 2019, significant change (p < .01) was identified for all
subcategories except Documentation which did in fact increase by one point. Figure 24 identifies the
subcategories and results for each by fiscal year.
Figure 24. Individual Record Review Category Scores
* The IRR subcategories were individually scored and are not averaged for the final overall IRR score. The final overall IRR score is calculated by adding all “yes” or “present” responses of all IRR indicators and dividing by the total “yes” or “no” responses combined for each record review.
86%
89%
69%
85%
72%
62%
85%
85%
78%
88%
74%
92%
90%
86%
84%
82%
70%
56%
Crisis StabilizationCourse of Stay
Documentation
Admission/Initial Evaluation/Screening for Risk
Assessment/Treatment Planning
Crisis StabilizationSpecific Treatment Planning
Transition/Discharge Planning
FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)
Quality Management Annual Report FY 2019
81 | P a g e
The overall IRR statewide average increased over the past two years to 83 percent in FY 2018, and
yet declined for FY 2019 to the baseline measurement year (FY 2016) of 79 percent. Several
indicators significantly (p < .01) declined within each subcategory regardless of sampled age
grouping (C&A versus Adult) lending to additional room for improvement, highlighted in the section
below. Moreover, Transition/Discharge Planning resulted in significant declines (p < .01) from FY
2018 to FY 2019 as a direct result of FY 2019 indicator changes, as seen in Table 15. Differences
between the populations sampled (Adult and C&A) are included with the corresponding indicator,
although statistical testing could not differentiate between significant changes in populations due to
the low volume of C&A records sampled. Differences were reviewed at the indicator level for each
indicator between populations and also between changes from the previous year. The lowest-scoring
indicators by each IRR category are listed in Table 15 which includes the following information:
current and previous year result, result of indicator by population (C&A versus Adult), statistical
significance of any at the p < .01 level (highlighted in blue), and the total number of instances
(percentage instances per indicator) per reason the indicator was not met.
Table 15. Lowest-Scoring Individual Record Review Indicators by Category*
Category Name and FY 2019 lowest-scoring indicator(s): FY 2018 FY 2019
Crisis Stabilization Course of Stay 85% 90%
Indicator: Co-occurring disorders as assessed and addressed simultaneously
82% C&A 33%, Adult 83%
79% C&A 33%, Adult 83%
FY 2019 reason(s) not met:
Medical, 21 instances (37%)
MH/Psychiatric, 18 instances (32%)
SU Diagnosis, 16 instances (28%)
IDD diagnosis, 2 instances (4%)
Documentation 85% 86%
Indicator: Community Transition Plan is present if readmitted within 30 days of discharge
19% C&A N/A, Adult 19%
16% C&A N/A, Adult 16%
FY 2019 reason(s) not met: • Of the nineteen individual(s) re-admitted within 30 days, 16 did not contain a Community
Transition Plan.
Indicator: Individual's MAR has a legend with all required criteria
65% C&A 68%, Adult 64%
61% C&A 79%, Adult 61%
Quality Management Annual Report FY 2019
82 | P a g e
Table 15. Lowest-Scoring Individual Record Review Indicators by Category*
Category Name and FY 2019 lowest-scoring indicator(s): FY 2018 FY 2019
FY 2019 reason(s) not met:
Missing staff credentials, 99 instances (56%)
Missing staff signature, 40 instances (23%)
Missing staff initials, 13 instances (7%)
MAR was illegible, 13 instances (7%)
Means to link signature and initials missing, 8 instances (5%)
Name of medication was omitted without explanation, 4 instances (2%)
Admission/Initial Evaluation/Screening for Risk 78% 84%
Indicator: Comprehensive Nursing Assessment present upon admission
80% C&A 88%, Adult 80%
82% C&A 93%, Adult 82%
FY 2019 reason(s) not met:
Missing fall risk assessment, 39 instances (61%)
Incomplete assessment, 10 instances (16%)
Missing nursing assessment, 6 instances (9%)
Missing flight risk assessment, 5 instances (8%)
Missing Infectious disease assessment, 2 instances (3%)
Nursing assessment was completed after the first 24 hours of admission, 2 instances (3%)
Assessment & Planning 88% 82%
Indicator: Co-occurring health conditions are included in the IRP or NCP
54% C&A 25%, Adult 55%
36% C&A 33%, Adult 36%
FY 2019 reason(s) not met:
Medical, 76 instances (43%)
MH/Psychiatric, 46 instances (26%)
SU Diagnosis, 30 instances (17%)
Missing IRP/NCP, 21 instances (12%)
IDD diagnosis, 4 instances (2%)
Indicator: Discharge plan defines criteria 69% C&A 56%, Adult 71%
48% C&A 29%, Adult 49%
FY 2019 reason(s) not met:
Step-down service missing, 99 instances (44%)
Clinical benchmarks missing, 80 instances (35%)
Missing discharge plan, 48 instances (21%)
Indicator: Interventions/objectives are goal-linked 96% C&A 100%, Adult 96%
90% C&A 100%, Adult 90%
FY 2019 reason(s) not met:
Missing IRP/NCP, 25 instances (78%)
Interventions /objectives not related to goals, 5 instances (16%)
Quality Management Annual Report FY 2019
83 | P a g e
Table 15. Lowest-Scoring Individual Record Review Indicators by Category*
Category Name and FY 2019 lowest-scoring indicator(s): FY 2018 FY 2019
Interventions are outside the CSU scope of service, 2 instances (6%)
Crisis Stabilization Specific Treatment Planning 74% 70%
Indicator: IRP/NCP is reviewed following seclusion or restraint
100% C&A 100%, Adult 100%
50% C&A N/A, Adult 50%
FY 2019 reason(s) not met:
Lack of a needed update of the IRP/NCP after seclusion/restraint of the three individuals sampled
Discussion of/with the individual in treatment team did not occur in 2 of the 3 individuals
Indicator: IRP/NCP is individualized and in personalized language
82% C&A 84%, Adult 82%
72% C&A 79%, Adult 72%
FY 2019 reason(s) not met:
Individualized and specific to the individual and includes duplicated IRP/NCP, 63 instances (61%)
Written in individual’s language (not devoid of clinical terms), 32 instances (31%) Inclusive of a specific preference of the individual, 8 instances (8%)
Indicator: IRP/NCP incorporates medical updates as indicated
68% C&A 100%, Adult 65%
46% C&A N/A, Adult 46%
FY 2019 reason(s) not met:
Urinary tract infection, 6 instances (27%)
Wound care as indicated, 2 instances (9%)
Post-trip to emergency room, 1 instance (5%)
New medical diagnosis after physical on CSU, 1 instance (5%)
Irregular labs, 1 instance (5%)
Updates as appropriate due to medical condition, 11 instances (50%)
Indicator: IRP/NCP addresses safety issues 55% C&A 88%, Adult 51%
43% C&A 58%, Adult 42%
FY 2019 reason(s) not met (missing safety issues):
Falls, 75 instances (41%)
Flight, 50 instances (27%)
Suicide precautions, 36 instances (20%)
Physical aggression risk, 9 instances (5%)
Infection precautions, 8 instances (4%)
Sexual aggression risk, 4 instances (2%)
Indicator: Plan of care discussed every 72 hours 60% C&A 96%, Adult 61%
60% C&A 92%, Adult 58%
FY 2019 reason(s) not met:
Quality Management Annual Report FY 2019
84 | P a g e
Table 15. Lowest-Scoring Individual Record Review Indicators by Category*
Category Name and FY 2019 lowest-scoring indicator(s): FY 2018 FY 2019
Treatment team not held every 72 hours, 64 instances (42%)
Documentation of treatment team missing, 50 instances (33%)
Unable to determine which treatment team members were present, 30 instances (20%)
Individual not present/no documentation of refusal, 8 instances (5%)
Transition/Discharge Planning 92% 56%
Indicator: Discharge summary/note documentation includes listed criteria
92% C&A 96%, Adult 92%
89% C&A 92%, Adult 89%
FY 2019 reason(s) not met (missing):
Post-discharge appointment, 26 instances (46%) Living situation at time of discharge, 10 (18%) Specific instructions for ongoing care, including appointment date/time, 7 instances (13%)
Method of discharge, 6 instances (11%) Date, time, and method of discharge, 3 instances (5%)
Discharge summary incomplete or missing, 3 instances (5%) Specific instructions, 1 instance (2%)
Indicator: A discharge summary was entered into the Collaborative’s ProviderConnect/batch system
N/A 24% C&A 43%, Adult 23%
FY 2019 reason(s) not met (missing):
See below for more details
Indicator: If individual was discharged to a homeless shelter, documentation reflects alternatives were explored
N/A 36% C&A 100%, Adult 35%
FY 2019 reason(s) not met (missing):
See below for more details *C&A and Adult data was not tested in FY 2019 for significance due to low C&A volume. Areas highlighted in blue were identified as having a significant difference in year comparisons.
Specific to the Transition/Discharge Planning subcategory, two new indicators were added in FY
2019. These indicators, although new to the CSUQR review tool for FY 2019, were not new standards
set forth by DBHDD but have been standards within the DBHDD Provider Manual for several years.
The questions were added to address consistent deficits identified throughout the past three years
via additional observations during quality reviews. Specific details for each new indicator are as
follows:
Quality Management Annual Report FY 2019
85 | P a g e
A discharge summary was entered into the Collaborative’s ProviderConnect/batch
system (24%)
o One CSU provider had entered all discharge summaries
o FY 2019 reason(s) not met:
No discharge summary found in record at the time of review (65%)
Discharge summary submitted after 48 hours (35%)
If an individual was discharged to a homeless shelter, documentation reflects
alternatives were explored
o Forty-four individuals were discharged to a homeless shelter from the CSU
(within the records reviewed)
o Of these, 16 records documented alternatives were explored with individuals
within the medical record (36% met)
CSUQR Individual Record Review (IRR) FY 2019 Highlights
Albeit several indicators within various IRR subcategories declined, several records reviewed met
indicators at or above 90 percent. These indicators are outlined in Table 16 by IRR category with
statistical differences (p < .01) from the previous year shaded in blue.
Table 16. Highest-Scoring Individual Record Review Indicators by Category
Service Name and FY 2019 highest-scoring indicator(s): FY 2018 FY 2019
Course of Stay 85% 90%
Individuals participated in training/therapy 86% 90%
Individual is offered groups as needed 88% 95%
Documentation 85% 86%
Date of transfer to a transitional bed documented by physician/physician extender
100% 100%
Progress notes document individual response to interventions provided
94% 96%
Medical progress notes document progress toward goals/objectives on IRP/NCP
90% 92%
RN documents status of individual at least once per day 93% 92%
Staff interventions in progress notes are related to interventions on IRP/NCP.
94% 91%
Quality Management Annual Report FY 2019
86 | P a g e
Table 16. Highest-Scoring Individual Record Review Indicators by Category
Service Name and FY 2019 highest-scoring indicator(s): FY 2018 FY 2019
Admission/Initial Evaluation 78% 84%
*No indicators met this criteria (>90%) Null Null
Assessment and Planning 88% 82%
Individual meets admission criteria for CSU 100% 100%
Interventions/objectives are goal linked 96% 90%
Crisis Stabilization Treatment Planning 74% 70%
Individual evaluated by physician/physician extender within 24 hours
99% 99%
Transition/Discharge Planning 92% 56%
*No indicators met this criteria (> 90%) Null Null
While the subcategory of Documentation showed minimal change from FY 2018 to FY 2019 (with a
one-point increase for a FY 2019 result of 86 percent), several indicators demonstrated progress for
the second year in a row. These indicators remained at 90 percent or above and were related to
progress notes documenting the individual’s response to interventions (96%), status of individual
documentation by a registered nurse at least once per day (92%), and that staff interventions were
related to interventions on IRP/NCP (91%).
Lastly, four indicators were collected as non-scored elements of the IRR.
One hundred percent of one-on-one observation occurred, when required, while the
individual was in seclusion or restraint/experiencing severe withdrawal/detox or psychiatric
symptoms/behaviors that could cause harm to self or others (n = 124)
Individuals were provided with a reconciled list of medications upon discharge in 98 percent
of records reviewed (n = 315)
Individuals had vital signs documented every eight hours in 72 percent of records reviewed (n
= 308)
o Important to note is that although no significant difference could be found between
the C&A population as compared to the Adult population, only 57 percent of C&A
Quality Management Annual Report FY 2019
87 | P a g e
records (n = 14) compared to 73 percent of adult records (n = 301) had vital signs
documented as required.
Only 39 percent of the records contained evidence of follow-up and continuing care (n = 319)
o Increased from FY 2018 (27%)
o C&A population (77%, n = 13); adult population (38%, n = 306)
While the overall IRR category decreased from FY 2018 to the baseline year (FY 2016) result of 79
percent in FY 2019, there are areas of significant improvement such as those outlined previously in
the categories of Admission/Initial Evaluation, Course of Stay, as well as the non-scored indicators
section across the CSU network. It is clear, however, that deficits or the introduction of new
indicators such as those within the Transition/Discharge planning section likely require additional
monitoring or tailoring of the elements within the tools as well as ongoing education and technical
assistance by the Collaborative.
CSUQR Service Guidelines
The 16 CSU Service Guidelines indicators (14 scored and 2 non-scored) were answered once per
review to assess the CSU program, see Figure 25 and Figure 26 where “n” represents the number of
reviews evaluated.
Quality Management Annual Report FY 2019
88 | P a g e
Figure 25. CSUQR Service Guidelines Staffing Adherence Indicators by Fiscal Year
Adult CSU staffing requirements were met in 95 percent of the reviews conducted. Of the CSUs
serving children and adolescents, all met the staffing requirements and ratios for FY 2018 and FY
2019. An indicator was added to the FY 2017 review process specific to the “Availability of a
psychiatrist for consultation in cases in which the CSU physician(s) is/are not specialized in
psychiatry” has scored 100 percent since inception. In FY 2019, while more than half of individuals
sampled had a substance use diagnoses identified (52%), having access to an addictionologist was
the lowest-scoring indicator of Service Guidelines (82%) demonstrating room for improvement.
Further analysis and ongoing CSU education and technical assistance will continue through FY 2020.
100%
100%
83%
100%
100%
100%
96%
96%
100%
100%
100%
100%
100%
100%
95%
86%
100%
100%
100%
100%
100%
100%
95%
82%
C&A Minimum Staff Present(FY17 n = 5, FY18 n = 5, FY19 n = 4)
C&A Nursing Staff Ratio(FY17 n = 5, FY18 n = 6, FY19 n = 4)
C&A Psychiatrist (Non-scored)(FY17 n = 5, FY18 n = 5, FY19 n = 4)
C&A Staff Ratio Met(FY17 n = 5, FY18 n = 5, FY19 n = 4)
Physician Availability for 3.7Withdrawal Management
(FY17 n = 22, FY18 n = 21, FY19 n =22)
Psychiatrist Available for Consultation(FY17 n = 18, FY18 n = 21, FY19 n = 22)
Adult CSU Staffing Requirements Met(FY17 n = 23, FY18 n = 21, FY19 n = 22)
Access to Addictionologist(FY17 n = 23, FY18 n = 21, FY19 n = 22)
FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)
Quality Management Annual Report FY 2019
89 | P a g e
Figure 26. CSUQR Service Guidelines Policy Adherence Indicators by Fiscal Year
While the majority of policies reviewed improved from the previous year(see Figure 26), both
“Medication notification policy adherence” and “Model/Curriculum for substance use treatment
(non-scored)” continued to excel to reach a perfect score of 100 percent across all FY 2019 reviews.
Policy adherence related to Infection control plan (86%), handling licit and illicit drugs (95%), as well
as seclusion and restraint (86%) all remained quite static when compared to the previous year (FY
2018).
83%
91%
87%
87%
91%
83%
83%
86%
90%
95%
86%
85%
90%
86%
100%
100%
95%
86%
86%
77%
59%
Adherence to Medication Notification Policy(FY17 n = 23, FY18 n = 21, FY19 n = 22)
Model/Curriculum for SU treatment (Non-scored)
(FY17 n = 22, FY18 n = 21, FY19 n = 22)
Protocols for Handling Licit and Illicit Drugs(FY17 n = 23, FY18 n = 21, FY19 n = 22)
Infection Control Plan Adherence(FY17 n = 23, FY18 n = 21, FY19 n = 22)
Seclusion & Restraint Policy Adherence(FY17 n = 23, FY18 n = 20, FY19 n = 22)
Therapeutic Blood Level Monitoring(FY17 n = 23, FY18 n = 21, FY19 n = 22)
Adherence to Safe Storage of Medication Policy(FY17 n = 23, FY18 n = 21, FY19 n = 22)
FY 2017 (N = 23) FY 2018 (N = 21) FY 2019 (N = 22)
Quality Management Annual Report FY 2019
90 | P a g e
Policy measurement indicators related to “Adherence to safe storage of medication” and
“Therapeutic blood level monitoring” declined to results of 59 percent and 77 percent, respectively,
with absolutely no changes in measurement or criteria of these two indicators since inception.
However, the additions made to the CSUQR tools in FY 2019 supplied an additional level of detail as
to why the indicator had not been met.
Adherence to safe storage of medications (59%):
o Substances other than medication had been found in the medication refrigerator (n =
5)
o The refrigerator temperature was out of the recommended range (n = 3)
o The documentation reviewed on the temperature log indicated staff did not follow
the policy (n = 1)
Therapeutic blood level monitoring policy adherence standard (77%):
o The blood glucose monitors were not calibrated (n = 5)
o Documentation in reviewed records indicate staff did not follow policy regarding lab
levels (n = 1)
Overall, the statewide average for Service Guidelines increased by four points (91%) in FY 2018 to
decline by three points to a FY 2019 result of 89 percent. From contract inception, scores typically
increased across the CSU providers specific to Service Guidelines with each year, with more than half
of reviews scoring a perfect 100 percent through FY 2018. However, now with the Collaborative in its
fourth year of the contract, scores at the provider level are declining with a median score of 90
percent compared to the previous year’s median score of 100 percent. The Collaborative will
continue to monitor, provide feedback, and provide additional technical assistance and training to
the CSU network specific to the standards and requirements surrounding policy and staffing
adherence.
CSU Focused Outcome Areas
Focused Outcome Area (FOA) indicators were answered once per record reviewed. Each FOA (Whole
Health, Safety, etc.) had a unique number of indicators for a total of 20 indicators assessed overall.
Please refer to the BHQR FOA section for a definition of the six FOAs.
Quality Management Annual Report FY 2019
91 | P a g e
The Overall FOA score for CSU providers increased from contract inception result of 88 percent to 91
percent in FY 2017 and remained at 91 percent through FY 2018. An additional two-point significant
(p < .01) increase for CSUQRs (93%) overall became evident in FY 2019 likely as a direct result of
increases in each of the subcategories with the exception of Rights which remained unchanged at 97
percent. The highest subcategories in FY 2018 were Choice (97%), Rights (97%), and Community
(92%). Although Choice scores increased significantly (p < .01) further to 99 percent in FY 2019,
Rights remained unchanged. The Whole Health FOA (96%), significantly improved (p < .01) from FY
2018, thereby replacing Community as the third highest CSUQR FOA following the completion of the
FY 2019 review process. See Figure 27 for FOA results by year.
Figure 27. CSUQR FOA Results by Fiscal Year*
* The FOAs were individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all “yes” or “present” responses of all FOAs and dividing by the total “yes” or “no” responses combined for each review.
The following were findings based on each FOA, at the indicator level, for FY 2019:
Whole Health documentation mirrored BHQR results related to this FOA for FY 2019. This
FOA has continually improved across all indicators since contract inception to the highest
result yet (p < .01), 96 percent, compared to last year’s 88 percent. Both indicators related to
external resource communication and external coordination needs significantly increased (p
FY17: 91%(N = 23)
FY18: 91%(N = 21)
FY19: 93%(N = 22)
Whole Health
FY17: 89%FY18: 88%FY19: 96%
Safety FY17: 85%FY18: 82%FY19: 84%
Person Centered FY17: 88%FY18: 88%FY19: 90%
Community FY17: 91%FY18: 92%FY19: 93%
Choice FY17: 98%FY18: 97%FY19: 99%
Rights FY17: 92%FY18: 97%FY19: 97%
Quality Management Annual Report FY 2019
92 | P a g e
< .01) for CSUQRs by eight and five points, respectively. Additionally, there was significant
improvement (p < .01) identified in 93 percent of records reviewed in FY 2019 versus FY 2018
(83%) having documented safeguards being utilized for medications known to have a
substantial risk for undesirable effects. Most oftentimes records reviewed not meeting this
requirement did not include the necessary Abnormal Involuntary Movement Scale (AIMS) (16
instances). Moreover, 100 percent of records reviewed pertinent to C&A populations:
o Demonstrated ongoing assessment to determine external referrals (n = 13),
o Documented communication with external resources to obtain results of testing,
treatment, and follow-up (n = 14), and
o Documented safeguards for medications known to have a substantial risk for
undesirable effects (n = 11).
Safety documentation demonstrated improvement, although not significant, by two points
overall for this FOA in FY 2019 resulting in 84 percent.
o Signed medication consent forms documenting education on the risks and benefits of
all medications prescribed declined in FY 2019 to a result of 70 percent. This was
mainly as a direct result of medications not being listed on the consent form (35%),
required signatures (physician, nurse, or individual/guardian) not being included
(33%), or the entire consent form missing from the record (32%). Also, fewer C&A
records had a signed consent as having been educated on the risks & benefits of all
medications prescribed (64%) compared to adult records (70%).
o Thirty-nine records did not contain evidence of a safety/crisis plan developed, as
needed, that directs in advance the individual’s desires/wishes/plans/objectives in
the event of a crisis. However, this has improved by two points to 88 percent from FY
2018. Of the records not meeting criteria to justify the evidence needed, 51 percent
of the records had the crisis/safety plan completely missing while 46 percent may
have had a crisis/safety plan on file but lacked an individual/guardian signature.
o Services offered in an environment that ensures individual safety significantly
improved (p < .01) from FY 2018 from 87 percent to 95 percent. Environmental safety
Quality Management Annual Report FY 2019
93 | P a g e
concerns/observations are on a case-by-case basis at all CSU locations. All safety
concerns identified for FY 2019 were from one CSU. Environmental observations
included: lack of security camera in the outside common area, dim lighting due to
missing lightbulbs, furniture in the day rooms was not bolted down, and lack of
individual’s access to bathroom/bedroom. DBHDD was notified of the specific
concerns noted from this review.
Person Centered Practices in FY 2019 continues to depict a deficiency in documentation as it
relates to plans being reassessed based on changing needs of the individual. This has resulted
in a three-point increase to 79 percent. This indicator remains the lowest of the measured
indicators.
o Specific to the C&A population, 100 percent of records documented evidence of
individual/guardian participation in planning of services (n = 14), the
individual/guardian participation in the modification of plan/services (n = 4) and
documentation within C&A records to support the plan is reassessed upon changing
needs (n = 3).
o However, only 62 percent of C&A records demonstrated evidence that the individual
was receiving individualized services as compared to 93 percent of adults. This was
mostly identified within the IRP/NCP by which Assessors found duplication across
individuals of similar IRP/NCP documentation or that some other type of assessed
need was identified elsewhere within that individual’s record, but not documented
within the IRP/NCP.
Community documentation supported a seven-point improvement in the individual and
provider discussion of aftercare placement for discharge during the course of stay (97%).
However, a decline of six points, although not statistically significant, was identified in the
individual’s informed choice driving selection of housing options (89%, n = 190). This was
most often found in cases by which the individual was discharged to a homeless shelter.
o No significant differences were found between the C&A population when compared
to the indicators of the adult population on this FOA (due to low C&A response
Quality Management Annual Report FY 2019
94 | P a g e
volume). However, 100 percent of C&A records reviewed for this FOA were met across
all four Community indicators.
Choice reflected all records reviewed (N = 330) had appropriate documentation
demonstrating how the individual is provided with options of supports/services for FY 2019.
This is a significant improvement from the FY 2018 result of 97 percent.
o It is important to note that although not statistically significant (due to low volume of
responses), 100 percent of C&A records reviewed as compared to Adult records met
the criteria for:
• Documentation demonstrates individual’s known preferences and differences
are followed to the extent possible (Adult 99%)
• When barriers are identified, documentation demonstrates that alternatives
are explored (Adult 97%)
Rights was supported by 98 percent of records demonstrating, for the second year in a row,
that “Rights and responsibilities acknowledgement is readily available, well-prepared, and
written in language understandable to individual.” The lowest-scoring indicator from FY 2018,
“Individual/guardian has signed formal acknowledgement of rights and responsibilities at the
onset of services, supports, and treatment” did demonstrate a slight improvement to a FY
2019 result of 95 percent.
The C&A records reviewed (n = 14) demonstrated 100 percent compliance in all areas
of Rights for the second year as analyzed, FY 2019.
CSUQR Staff Interview and Individual Interview
The CSUQR interview indicators were similar to the BHQR interview indicators and divided into the
six FOAs. Interviews were analyzed and a score derived for reporting purposes; however, the score
obtained does not calculate into a provider’s “Overall” score or any other scored category. Individual
Interviews were used to assess the individual’s perception of care with the provider, services
rendered, and support in working toward personal goals. Staff Interviews helped determine if a
person-centered approach was used in providing services and empowering individuals. The data and
Quality Management Annual Report FY 2019
95 | P a g e
anecdotal information gathered from the Individual and Staff Interviews supplemented the record
review and enhanced the review process by providing a tailored perspective to illustrate the quality
of care the agency provided using both a quantitative and qualitative approach.
Data in Figure 28 illustrates the results of Staff and Individual Interviews by fiscal year. Individual
Interview scores declined slightly from FY 2018 (95.1%) to FY 2019 (94.6%) while Staff Interview
scores increased minimally by three-tenths of a percentage point to 99.5 percent for FY 2019.
Figure 28. CSUQR Individual and Staff Interview Results by Fiscal Year
Similar to BHQR interview results, all but one Interview FOA scored in the 90th percentile. The
lowest-scoring FOA for Individual Interviews for the fourth consecutive year was Whole Health
(85%). Moreover, it is evident over the course of all BHQRs since contract inception, both individual’s
and staff’s perception of care received, offered, and or coordinated is substantiated at a high level.
Areas of deficits can be and are addressed from interview information collected based on the
supplemental data. However, as results continue to remain high and the process of interview
scheduling, completion, and collection of indicators being somewhat timely and burdensome on the
part of individuals/staff, it has been recommended by DBHDD and Quality Management for FY 2020
that Staff Interviews be eliminated from the review process. Additionally, the collection of Individual
Interview data be reduced to a set of only 10 carefully-crafted questions.
FY 2017
93.0%
(N = 114)
FY 2018
95.1%
(N = 102)
FY 2019
94.6%
(N = 104)
INDIVIDUAL INTERVIEW
FY 2017
98.7%
(N = 115)
FY 2018
99.2%
(N = 105)
FY 2019
99.5%
(N = 106)
STAFF INTERVIEW
Quality Management Annual Report FY 2019
96 | P a g e
The recommendation mentioned above eliminates 56 separate Staff Interview indicators and a total
of 47 Individual Interview indicators at a rate of approximately 30 minutes per interview. Staff
Interview elimination would allow for a reallocation of resources for additional ICM records, new
services to be reviewed, and inclusion of the Office of Deaf Services questions to the BHQR. The
reduction of Individual Interviews indicators from 47 to 10 questions also reduces time and burden
while still providing supplemental information and allowing for the individual to provide their
perception and satisfaction with the care and services supplied by the agency.
New Provider Reviews New Provider Review Findings
In FY 2019, 131 unique BH providers were reviewed. Each fiscal year, it is anticipated that all
behavioral health providers have at least one quality review. As stated previously in this report,
exclusions are a possibility and provider eligibility is based on multiple factors including volume of
individuals served, volume of submitted claims, and type of services provided.
Of the 131 providers reviewed in FY 2019, five new providers (never before reviewed in prior fiscal
years by Quality Management) received a BHQR. New providers are reviewed approximately three
times within the first two years. All five providers were classified as “Small” serving less than 50
individuals per year, with 5-10 records reviewed per agency. Four of the five providers were Tier 3
(Specialty) providing Intensive Family Intervention services only. The fifth was a Tier 2 provider who
offered Non-Intensive Outpatient Services (NIOP). Results for each of the five providers’ Overall and
Billing scores are demonstrated in Figure 29 with remaining category scores listed in Table 17.
Quality Management Annual Report FY 2019
97 | P a g e
Figure 29. FY 2019 Overall and Billing Scores by New Provider
Overall Score: The average Overall Score for the group of new providers reviewed in FY 2019 was 64
percent; the overall statewide average was 90 percent. See Table 17. Only one of these new
providers met the 90 percent threshold as it relates to Overall Score (Tier 2 provider of NIOP
services). This provider scored a 94 percent and a 93 percent for the Overall score and Billing score,
respectively.
Important to note: Of the 157 reviews completed in FY 2019, three new providers were within the 10
lowest-scoring providers for both Overall and Billing scores. Due to this finding, rankings were
assigned to each category score in FY 2019, where “1” = lowest Overall score and “157” = highest
Overall score. See Table 17 for the New Providers’ scores by category, including the rank if the score
was within the 10 lowest scores for each category across all BHQRs.
Table 17. FY 2019 Review Results by Category of New Providers
Overall Score Billing Assessment &
Planning Service FOA
Provider 1 32%
Lowest Statewide
0% Lowest
Statewide
35% Lowest
Statewide
34% 2nd Lowest Statewide
60% Lowest
Statewide
Provider 2 36%
2nd Lowest Statewide
0% Lowest
Statewide
46% 2nd Lowest Statewide
23% 1ST Lowest Statewide
75% 2nd Lowest Statewide
Provider 3 70% 38% 85% 79% 79%
32%0%
36%
0%
70%
38%
86% 87%94% 93%90% 86%
0%
20%
40%
60%
80%
100%
Overall Score Billing
New Provider 1 New Provider 2 New Provider 3
New Provider 4 New Provider 5 State Average
Quality Management Annual Report FY 2019
98 | P a g e
Table 17. FY 2019 Review Results by Category of New Providers
Overall Score Billing Assessment &
Planning Service FOA
7th Lowest Statewide
7th Lowest Statewide
9th Lowest Statewide
Provider 4 86% 87% 63%
7th Lowest Statewide
92% 100%
Provider 5 94%
93% 95% 98% 89%
New Providers 64% 44% 65% 65% 81%
State Average 90% 86% 88% 90% 94%
Billing Score: Of the four categories, Billing was the lowest for the five providers. New providers
averaged a score of 44 percent with a median of 38 percent. The only providers in FY 2019 that
received a zero percent Billing Score for FY 2019 were two new providers. The top three billing
discrepancies replicate the initial baseline year results (FY 2016) of the entire network in which
providers were most often cited for the following;
Progress note is missing
Billing code is missing or different than code billed
Missing/incomplete service order
Assessment and Planning Score: New providers averaged 65 percent for Assessment and Planning.
The highest-scoring indicators were similar to the statewide findings related to, “individual meets
admission criteria with the diagnosis verified at least annually,” (93%) and “current medical
screening present” (100%). Additionally, the lowest-scoring FY 2019 indicators across the new
providers were the same improvement areas identified across the entire network. Results of the
initially reviewed providers for FY 2019 were:
All assessed needs are addressed (43%; Statewide BHQR 77%)
Whole health & wellness in IRP (50%; Statewide BHQR 82%)
Discharge plan defines criteria (47%; Statewide BHQR 76%)
Quality Management Annual Report FY 2019
99 | P a g e
Focused Outcome Areas (FOA) Score: The average for the five new providers specific to the Focused
Outcome Areas category was 81 percent for FY 2019; however, one provider did score 100 percent
for every FOA (Tier 3 Intensive Family Intervention specialty services provider). Both Whole Health
and Safety had fewer indicators met than any other FOA; 60 percent and 53 percent, respectively.
The lowest-scoring indicators for this set of providers in FY 2019 across all FOAs are listed below.
Whole Health
Documentation of communication with external referral sources and providers to obtain
results of testing, treatment, and follow up (55%; Statewide BHQR 91%).
There are documented safeguards utilized for medications known to have substantial risk or
undesirable effects (lab work, assessments, AIMS, etc.) (50%; Statewide BHQR 88%).
Safety
Individual/guardian & prescriber has signed consent as having been educated on the risks &
benefits of all medications prescribed (50%; Statewide BHQR 73%).
Safety/Crisis plan developed, as needed, that directs, in advance, the individual's
desires/wishes/plans/objectives in the event of a crisis (57%; Statewide BHQR 86%).
Person Centered Practices
Documentation demonstrates the plan is reassessed based upon any changing needs,
circumstances or response by the individual (63%; Statewide BHQR 88%).
Rights
Documentation reflects that individuals were informed of their rights and responsibilities and
HIPAA (83%; Statewide BHQR 99%).
Choice
Documentation demonstrates individual's known preferences and differences are followed to
the extent possible (77%; Statewide BHQR 98%).
Quality Management Annual Report FY 2019
100 | P a g e
Community
Documentation of transition planning (including individual, family, other supports) is evident
throughout service delivery and includes specific objectives to be met prior to discharge or
decrease in intensity of services (78%; Statewide BHQR 94%).
Service Guidelines Score: The FY 2019 result of Service Guidelines for newly reviewed providers was
the same as the Assessment and Planning score of 65 percent. As four of the new providers
reviewed offered Intensive Family Intervention (IFI) services (Providers 1 – 4), Table 18 supplies
results of each indicator and the percent met for these providers. The lowest statewide Service
Guidelines score was 23 percent. This is the same new provider that ranked as second out of 157 as
based on Overall score. Eight indicators scored 50 percent or below with the lowest indicators
related to IFI team and team leader minimum contacts as well as documenting assessed progress
within progress notes. See Table 18 for the complete results of IFI indicators for the four new
providers offering only the IFI service.
Table 18. Intensive Family Intervention Results by Indicator for New Providers
Intensive Family Intervention (n = 4) % Met
The individual meets admission / continuing stay criteria / diagnostic criteria. 90%
The progress notes document individual response to the staff intervention provided. 90%
There is evidence that the provider is helping the parents/responsible caregivers increase capacity to care for their children. (Review authorization period.)
85%
The Team Leader is licensed/credentialed or CAC-II or equivalent. 75%
The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.
60%
Safety planning with the family and all parties involved evident in the record from the onset of services.
50%
The IFI team has all required staff. 50%
A team approach is used, as evidenced by more than one person and at least one licensed team member. (Review authorization period.)
45%
Services are a mix of individual/family counseling and skill development according to the needs of the individual/family. (Review authorization period.)
45%
The Team Leader is meeting with families at least 2x/month. (Review authorization period.) 45%
Quality Management Annual Report FY 2019
101 | P a g e
Table 18. Intensive Family Intervention Results by Indicator for New Providers
Documentation reflects a tapering of services. Answer N/A if not appropriate. 40%
Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan.
30%
The team is making at least three (3) contacts a week and at a frequency that is clinically appropriate. (Review authorization period)
25%
Services over six (6) hours are related to a crisis and has supporting documentation signed by the Team Leader.
NA
Reassessment Frequency Reviews
BHQR Reassessment Review Findings
Beginning in FY 2017, the frequency of a BHQR and CSUQR was based on minimum scoring
thresholds for both Overall and Billing scores. Providers scoring above the minimum threshold
received one BHQR/CSUQR per fiscal year; providers scoring below the threshold received two
reviews. Providers scoring below the threshold remain on a frequency schedule of approximately
every six months until two subsequent review scores meet threshold requirements. The six months’
time frame between reviews allow ample time for claims submission and documentation to reflect
any changes made by the provider based on previous review findings.
Based on the analysis each fiscal year, the thresholds increased, per DBHDD direction:
FY 2017 thresholds (baseline year)
o BHQR: 80 percent Overall score or 70 percent Billing score
o CSUQR: 80 percent Overall score
FY 2018 thresholds
o BHQR: 80 percent for both Overall score and Billing score
o CSUQR: 80 percent Overall score
FY 2019 thresholds
o BHQR: 90 percent for both Overall score and Billing score
o CSUQR: 90 percent Overall score
Quality Management Annual Report FY 2019
102 | P a g e
Fifteen percent of providers required a reassessment in FY 2018 (N = 20) and 22 percent (N = 28)
required a reassessment in FY 2019. Five of these providers were also on the twice per year
frequency schedule in FY 2018. As in FY 2018, most reassessed providers in FY 2019 were Tier 2 (N =
19) and there was an average of 191.1 days between reassessed reviews.
Figure 30 displays the distribution of Overall scores obtained during the first and second reviews in
FY 2019 for the reassessed providers. The average change in Overall score was six points (either
increase or decrease) from the first to second review. The most visible change is in the 81 percent to
90 percent range. For the first review of FY 2019, 43 percent of providers scored in the 81 to 90
percent range, which reduced to 39 percent for the second FY 2019 review.
Figure 30. FY 2019 BHQR Overall Score Distribution of Reassessed Providers
Overall Score: The average result for the reassessed providers at first review of FY 2019 was 89
percent compared to 88 percent later in the year. When compared to FY 2018, more providers were
placed on the reassessment frequency in FY 2019 due to an Overall score below the threshold at the
initial annual review. A third of providers (n = 7) reassessed in FY 2018 were on the twice per year
frequency as a direct result of falling below the FY 2018 Overall score threshold as compared to half
4% 4%
43%50%
7% 4%
39%50%
0%
20%
40%
60%
80%
100%
1st BHQR (N = 28; Mean = 89%; Median = 90%)
2nd BHQR (N = 28; Mean = 88%, Median = 91%)
Quality Management Annual Report FY 2019
103 | P a g e
(n = 14) of the reassessed providers in FY 2019. The following are details specific to the FY 2019
reassessed providers and their Overall results:
Eight providers exceeded the 90 percent threshold for both FY 2019 reviews, thus removing
them from the bi-annual frequency.
The greatest increase in Overall score for a provider was a 15-point change from initial to
subsequent review. This provider’s Billing score increased from initial review (50%) to
subsequent review (94%).
Thirty-six percent (N = 10) of providers increased the Overall score upon subsequent review.
Three reassessed providers (11%) had the exact same Overall score for both their initial and
subsequent FY 2019 reviews.
Fifteen providers (54%) had a decline in Overall score from time of initial review to
subsequent FY 2019 review.
The largest decrease in Overall score was 26 percent and the average decrease was 5.7
percent in Overall score for these fifteen providers. Quality Management did not have review
tool changes between the first and second reviews.
Table 19 provides the average result by category for the group of reassessed BHQR providers,
both at the initial review as well as at the subsequent review.
Table 19. FY 2019 Review Results by Category of Reassessed Providers
Overall Score Billing
Assessment & Planning Service FOA
Initial Score Average
89% 87% 84% 89% 93%
Subsequent Score Average
88% 80% 87% 91% 94%
State Average 90% 86% 88% 90% 94%
Billing Score: Unlike FY 2018 where 80 percent of reassessed providers had an increase in their
Billing score, only 43 percent of providers reassessed in FY 2019 saw an increase from their initial
Quality Management Annual Report FY 2019
104 | P a g e
review. The subsequent BHQR Billing score fell below the State Average of 86 percent and is the
lowest category score. Overall, the average Billing scores for providers reviewed twice during FY 2019
decreased from 87 percent (at the initial review) to 80 percent subsequently (see Figure 31 for the
distribution scores of reassessed providers). As in previous years, the two most prevalent reasons
claims were unjustified: “Missing or incomplete service orders” and “Content does not support the
units billed.” The most substantial improvement seen in a provider’s subsequent Billing score was a
44 percent increase (initial to subsequent review results from 50 to 94 percent).
Figure 31. FY 2019 BHQR Billing Score Distribution of Reassessed Providers
Assessment and Planning was the lowest category for the reassessed providers in FY 2017 and FY
2018; however, this changed during FY 2019. The initial review result in this category maintained this
pattern with a score of 84 percent when averaged across providers, but with an increase of three
points (87%) by second review. Assessment and Planning was no longer the lowest category score.
Similar to providers who met the threshold requirement for FY 2019 and did not require a
reassessment, the lowest scored indicators were “All assessed needs addressed on the IRP”, “Co-
occurring health conditions addressed in IRP”, “Whole health and wellness within the IRP”, and
“Discharge plan defines criteria.” These indicators were also the lowest-scoring in this category in FY
2018. Of the 61 percent of providers who demonstrated improvement in Assessment and Planning,
0% 4% 0%7%
50%
32%
11%0%
7% 11%
43%29%
0%
20%
40%
60%
80%
100%
1st BHQR (N = 28; Mean = 87%; Median 89%)
2nd BHQR (N = 28; Mean = 80%; Median = 85%)
Quality Management Annual Report FY 2019
105 | P a g e
41 percent increased by 10 points or more. One provider had an increase of 29 points, with an
increased Billing score of 66 percent at initial review to 95 percent by year-end. When compared to
the statewide average of 88 percent, 54 percent of providers scored at or above this result during
their first FY 2019 review and then increased to 61 percent at the subsequent FY 2019 review.
Extensive technical assistance was provided during the quality reviews related to Assessment and
Planning. During subsequent reviews, it was found that providers implemented recommended
changes based on their previous BHQR.
Service Guidelines: The BHQR Service Guidelines state average score remained the same at 90
percent for FY 2019. The average score for reassessed providers at initial review was 89 percent, just
slightly below the statewide average. The average score improved to 91 percent at second review;
thereby exceeding the statewide average by one percent. Eighteen services were reviewed
throughout the reassessments. Services not included in these reviews were due to the agency either
not providing these services or claims not available in the date range sampled for either one or both
of the FY 2019 reviews. Reassessments did not include a review of:
AD Peer Support Program
AD Peer Support - Individual
Opioid Maintenance
Peer Support Whole Health and Wellness – Group
Peer Support Whole Health and Wellness – Individual
Most Service Guidelines scores experienced minimal change between the initial and subsequent FY
2019 review with 59 percent of services experiencing an increase or decrease of two percent or less.
A new service reviewed in FY 2019, Substance Abuse Intensive Outpatient Program (SAIOP),
demonstrated an increase of 26 percent from initial to subsequent review for applicable reassessed
providers. This was mostly due to improvement in progress note documentation being reflective of
the individual recovery plan (IRP) as SAIOP was not included on many IRPs in the initial BHQR review.
Improvement was noted within Intensive Family Intervention (IFI) Service Guidelines from the
subsequent FY 2018 review (88%) to the initial FY 2019 review (94%). However, the score reduced by
Quality Management Annual Report FY 2019
106 | P a g e
the subsequent FY 2019 review to 91 percent. The most significant reduction was with the indicator
“The Team Leader is meeting with families at least 2x/month” as scoring for this indicator dropped
45 percentage points from 93 percent to 48 percent by the FY 2019 subsequent review. Of note,
changes to scoring methodology or questions reviewed did not change for IFI during FY 2019.
Three providers of Intensive Case Management (ICM) services were reassessed in FY 2019 with the
Service Guidelines score for ICM dropping from 88 percent (initial review) to 86 percent (subsequent
review). Three ICM Service Guidelines indicators decreased by 18 percentage points or more from
initial to subsequent review:
“Documentation consists of interventions that provide environmental supports and care
coordination that promotes recovery as identified in the Individual Recovery Plan (IRP).” This
indicator scored 92 percent for the initial FY 2019 review, and dropped to 70 percent at the
subsequent review.
“Documentation reflects referral and linkage to services and resources identified on the
Individual Recovery Plan (IRP) including housing, social supports, family/natural supports,
entitlements (SSI/SSDI, Food Stamps, VA), income, transportation, etc.” scored 100 percent
for the initial FY 2019 review, but dropped to 80 percent by the subsequent review.
The lowest-scoring indicator within ICM was, “There is a joint development of a crisis plan to
include the provider and individual; the provider is listed as primarily responsible,” scored 50
percent for the initial FY 2019 review and dropped to 32 percent by the subsequent review.
Two Assertive Community Treatment (ACT) providers were reassessed in FY 2019. While there was
no change in the Overall ACT score from initial to subsequent review, two indicators had a more than
15 percentage point difference. “The ACT Team is working with the individual towards educational or
vocational needs, interests, per IRP” increased from 83 percent to 100 percent; while “There is
documented evidence that the ACT Team is working with informal support systems/collateral
contacts at least 2-4 times per month” decreased 22 points from 33 percent to 11 percent.
Focused Outcome Areas: As in FY 2018, providers reviewed for a second time in FY 2019
demonstrated improvement in FOA scores, from 93 percent to 94. Seventeen providers (61%)
Quality Management Annual Report FY 2019
107 | P a g e
demonstrated either no change or improvement in this area and 11 providers had a decline in score
at an average of five points. Of all FOAs reviewed during a reassessment in FY 2019, Safety was the
lowest-scoring category both at first review (76%) and at second review (80%). The Safety FOA
continued as an area for growth and improvement from FY 2018 to FY 2019 among not only
reassessed providers but also the entire network.
CSU Reassessment Review Findings
Three of the 19 unique CSU providers had a reassessment during FY 2019 due to either a low Overall
score in FY 2018 (< 90%) or a low score early in FY 2019. Figure 32 provides a graphic representation
of each of the CSU’s progress from the FY 2018 result to the subsequent FY 2019 reassessment
review. One of the three CSU providers (CSU #2) was also on the twice a year frequency in FY 2018.
This CSU provider (CSU #2) was the only provider to consistently improve from review to review
since FY 2018. The other two CSU’s, CSU #1 and CSU #2, ended the 2019 fiscal year having declined
in Overall scores to 75 percent and 78 percent, respectively.
Figure 32. Overall Score Results by Reassessed CSU
87%
78%
84%
75%
87%
87%
79%
82%
88%
85%
72%
96%
50% 60% 70% 80% 90% 100%
Statewide Average
CSU #3
CSU #2
CSU #1
FY 2018 Initial FY 2019 Subsequent FY 2019
Quality Management Annual Report FY 2019
108 | P a g e
Conclusion
Eighty-five percent of BHQR reassessed providers experienced an increase in Overall score from
initial to subsequent FY 2018 review, but only 36 percent of providers increased the Overall score
during FY 2019. Additionally, two of the three CSU’s reassessed in FY 2019 did not improve from
initial to subsequent review. Distributions of reassessed providers (based on each scored category)
can be found in Appendix B. Those providers who fell below the established threshold at either of
the two reviews during FY 2019 will be required to have an initial and subsequent review scheduled
for FY 2020 regardless of improvement in one or both categories (Overall and Billing score). Based on
all BHQR and CSUQR data and findings obtained from FY 2019 reviews, it is anticipated that the
threshold for FY 2020 BHQRs is to remain at 90 percent for both Overall and Billing scores and
CSUQRs remain at 90 percent Overall.
Technical Assistance/Exit Conference
While technical assistance occurred throughout the review process via interactions between the
Collaborative and provider staff, one measurable component of the volume of technical assistance
provided was collected from the exit conferences. Upon completion of all BHQRs and CSUQRs, the
lead assessor completed a formal exit conference. The exit conference(s) supplied providers with
tentative scores, provider strengths, and opportunities for growth including quality
recommendations made by the Collaborative. Table 20 provides details on the technical assistance/
exit conferences completed during FY 2019.
Table 20. FY 2019 Technical Assistance/Exit Conference Details
Fiscal Year 2019 Quarter 1 Quarter 2 Quarter 3 Quarter 4 Total
Year End
Total BHQR Exits 50 29 45 33 157
Total Minutes 2,766 1,574 2,729 2,008 9,077
Total Attendees 405 254 418 383 1,460
Total CSUQR Exit 2 5 5 10 22
Total Minutes 65 270 315 660 1,310
Total Attendees 12 64 76 215 367
Quality Management Annual Report FY 2019
109 | P a g e
Table 20. FY 2019 Technical Assistance/Exit Conference Details
Fiscal Year 2019 Quarter 1 Quarter 2 Quarter 3 Quarter 4 Total
Year End
Total Exits 52 34 50 43 179
Through the 179 exit interviews conducted in 2019, technical assistance was provided to 1,827
provider staff members for an approximate total of 173 hours. In FY 2019, the total number of
attendees increased by more than 200 from 2018. The CSUQR average attendance per exit was 17
(FY18 average attendance per exit was 15) and the BHQR average attendance was nine (exactly the
same BHQR average attendee count per exit as in FY 2018). The average exit conference length was
approximately 58 minutes. The length of exit conferences varies by type of exit conference and may
include ACT reviews, which included an additional 15 records per review and details specific to ACT
services discussed and reported. Exit conferences and technical assistance will continue throughout
FY 2020 as through the provider feedback survey, providers identified them as helpful assistance
from the Collaborative.
Summary of Findings and Recommendations for Behavioral Health Providers Quality Management continually reassesses review processes to ensure we are capturing results and
analyzing outcomes that give the best information and truest picture of service quality in Georgia.
This includes reviewing feedback received from both providers and individuals, as well as revision of
review tools as needed to ensure accurate and measurable reporting of results.
June 2019 marked the completion of the fourth year of the Collaborative’s Quality Management
program. Quality Assessors completed 157 BHQRs and 22 CSUQRs in FY 2019 with 179 exit
conferences conducted across all review types. Exit conferences may have occurred jointly across the
BHQR, CSUQR, and ACT reviews depending on whether the provider rendered more than one of
those services.
Quality Management Annual Report FY 2019
110 | P a g e
Provider Performance
Table 21. BHQR, ACT, CSUQR Overall Averages by Year
Billing Validation FOA
Assessment/ Planning
Service Guidelines Overall
BHQR FY 2017 84% 89% 77% 88% 84%
BHQR FY 2018 85% 92% 84% 90% 88%
BHQR FY 2019 86% 94% 88% 90% 90%
ACT FY 2017 90% 90% 80% 88% 87%
ACT FY 2018 91% 93% 87% 84% 89%
ACT FY 2019 88% 93% 90% 84% 89%
CSUQR FY 2017 NA 91% IRR – 80% 87% 86%
CSUQR FY 2018 NA 91% IRR – 83% 91% 88%
CSUQR FY 2019 NA 93% IRR – 79% 89% 87%
Table 21 summarizes the Overall averages by fiscal year:
While BHQRs resulted in an average improvement in Overall score from 88 percent to the
threshold of 90 percent, CSUQRs declined in Overall score from the previous year to 87
percent.
ACT remained consistent to the previous year’s results related to FOAs, Service Guidelines,
and Overall. However, ACT services improved in the area of Assessment and Planning (90%)
and declined in Billing validation by three points from the previous year (88%).
Five new providers were reviewed and analyzed in FY 2019 that had never before had a
quality review completed. When grouped together, the average Overall score for these five
providers was 64 percent. Of the five providers, three scored within the top ten lowest
scoring reviews for Overall score, with two of those providers receiving a zero percent for the
Billing category. Interesting to note, the most cited Billing discrepancies in FY 2019 for new
providers were the same as the first year of the BHQRs (FY 2016). Furthermore, new
Quality Management Annual Report FY 2019
111 | P a g e
providers’ Whole Health FOA score (60%) was similar to the baseline year (FY 2016) results of
the entire network (63%).
Twenty-eight providers were reviewed for a second time in FY 2019 due to either low scores
in Overall, Billing, or both (< 80% in FY 2018; < 90% in FY 2019). While this resulted in a
greater number of reviews completed for FY 2019, it is still less than the number of additional
reviews conducted in FY 2017 (N = 35). While it is anticipated providers whom are reassessed
are likely to obtain an elevated score when measured on the same indicators, only 36 percent
(N = 10) actually received an increased score by year end.
All category results had statistically significant (p < .01) changes, except Service Guidelines.
The Service Guidelines scores remained static from FY 2018 to FY 2019 for both BHQRs (90%)
and ACT (84%) reviews, but decreased by two percentage points in CSUQRs (89%).
Focused Outcome Areas (FOAs) were introduced for the first time to the Georgia Behavioral Health
network in June 2016. With results continuing to significantly improve (p < .01) and most providers
excelling in meeting the measured standards, Table 22 provides further details and summarizes the
FOAs by subcategory, review type, and year to further support the improvements identified.
Table 22. BHQR and CSUQR FOA Scores
Focused Outcome Areas BHQR
FY 2017 BHQR
FY 2018 BHQR
FY 2019 CSUQR FY 2017
CSUQR FY 2018
CSUQR FY 2019
Whole Health 74% 84% 91% 89% 88% 96%
Safety 83% 78% 80% 85% 82% 84%
Person Centered Practices 91% 95% 94% 88% 88% 90%
Community 93% 96% 97% 94% 92% 93%
Choice 96% 97% 97% 98% 97% 99%
Rights 93% 93% 94% 92% 97% 97%
Overall Score 89% 92% 94% 91% 91% 93%
The Overall score for the FOAs met or exceeded 93 percent in FY 2019 for both BHQR and CSUQRs.
However, five new providers reviewed for the first time in FY 2019 averaged a low score of 60
Quality Management Annual Report FY 2019
112 | P a g e
percent for this FOA, a similar finding to the FY 2016 introduction of this FOA result of 63 percent.
Minimal changes were found across the entire network in many of the FOA categories including
Person Centered Practices, Community, Choice and Rights (all of which exceeded 90 percent). Whole
Health generously improved from FY 2018 to FY 2019 for both BHQRs (91%) and CSUQRs (96%)
whereby results increased by seven points and eight points, respectively. Lastly, while Safety
demonstrated improvement in providers’ scores reaching a result of 80 percent and 84 percent, for
BHQRs and CSUQRs respectively, improvement needs are evident. This is especially the case specific
to safety/crisis plans being developed when needed with all necessary elements (BHQR 86%, p < .01;
CSUQR 88%) as well as records containing a signed consent for being educated on medication risks
and benefits when prescribed (BHQR 73%, p < .01; CSUQR 70%).
Specific to reassessed providers who fell below the minimum threshold for Billing and Overall scores,
Table 23 reflects the number of providers who improved from the first FY 2019 review to the second
review within the year. Moreover, while 14 reassessed providers met or exceeded the 90 percent
statewide Overall score at first review, only 12 providers exceeded the 90 percent threshold after the
second review. Providers who obtained low FY 2018 Overall or Billing scores were scheduled for
review and reassessment in FY 2019. While many of the reassessed providers demonstrated
improvements on an individual level from first FY 2019 review to second FY 2019 review, those that
continued to fall below the established thresholds are to be scheduled for an additional two reviews
in FY 2020. Additionally, providers were supplied with additional technical assistance during reviews
to assist with performance improvement.
Table 23. FY 2019 Reassessment Review Results Number (%) of Providers Who Increased in Score
from FY 2019 Review #1 to FY 2019 Review #2
Category BHQR
(N = 28) CSU
(N = 3)
Billing 12 (43%) NA
Individual Record Review NA 2 (67%)
Service Guidelines 17 (61%) 0 (0%)
Focused Outcome Areas 12 (43%) 1 (33%)
Quality Management Annual Report FY 2019
113 | P a g e
Table 23. FY 2019 Reassessment Review Results Number (%) of Providers Who Increased in Score
from FY 2019 Review #1 to FY 2019 Review #2
Category BHQR
(N = 28) CSU
(N = 3)
Assessment & Planning 17 (61%) NA
Overall Scores 10 (36%) 1 (33%)
FY 2019 BH Accomplishments
Throughout the year, the Quality Management Department partners with DBHDD Quality
Improvement, DBHDD subject matter experts, and others to discuss and review findings, concerns,
and areas of need across the provider network based on monthly, quarterly, and annual review
results. It is through this partnership the Collaborative derives recommendations and revisions to
the review process.
Below is a listing of accomplishments achieved by the Collaborative that have occurred throughout
FY 2019.
BHQR and CSUQR tools were revised for clarity, alignment, and formalized criteria including:
o Addition of Substance Abuse Intensive Outpatient Program as a measurable service
o A new “non-scored” Programmatic review category was piloted to collect, monitor,
and report various issues to DBHDD specific to programmatic standards of specific
service types.
o Differentiation between Peer Support Whole Health and Wellness - Individual versus
Group services.
o Inclusion of a billing discrepancy “No valid, verified diagnosis on date of service
provided.”
FY 2019 Quality Training Series
o Safe Storage, Handling and Administration of Medication
o Making the Most of Your Crisis Stabilization Unit Quality Review
Quality Management Annual Report FY 2019
114 | P a g e
o Making the Most of Your Behavioral Health Quality Review
o Medicaid Assisted Treatment (MAT) Provider Training
o Documentation for Paraprofessionals – A Guiding Light
o Person-Centered Approach to Crisis and Safety Planning
o Recovery and Rehabilitation – The Ultimate Selfie
o CSU Transition/Discharge Planning & Overview of Care Coordination
o Tier 1 & 2 Information Session “BH Provider Standards & KPIs
Successful collaboration with the Collaborative’s IDD department in developing Final and Exit
summary and assessments specific to the review process, while also assisting in alignment of
IDD tool revision changes and needs with categories reviewed within the BHQRs/CSUQRs.
Annual review of all BHQR and CSUQR tools in preparation for the new fiscal year.
Completion of the requested “Office of Deaf Services Specialty Review.”
A forty percent increase in reassessments were completed in FY 2019 as compared to FY
2018, likely as a direct result of the increased threshold for Overall and Billing scores to 90
percent.
Quality Management met all review reporting contractual requirements as set forth by
DBHDD including but not limited to quarterly provider score reporting, semi-annual review
data and score presentation, Medicaid discrepancy reporting, as well as any ad hoc request
made by DBHDD.
Quality Management conducted various ad hoc reporting and data analysis which was
provided to its client either as a direct result of previous year’s analysis or at the request of
DBHDD and included:
o Sample size Power Analysis (BHQR/CSUQR/ACT/ICM)
o Supported Employment Study (IDD specific but included BH collaboration)
o Intensive Case Management Analysis
o Individual/Staff Interview Analysis
o Yale RSA-R Report
o Intensive Family Intervention Review/Claim Analysis
Quality Management Annual Report FY 2019
115 | P a g e
o Office of Deaf Services Review pilot
o Suicide Study proposal and sampling methodology
BH Systems Strengths and Recommendations for Improvement
In FY 2019, both BHQR and CSUQR scores fluctuated either positively or negatively dependent upon
review and category type. While BHQRs Overall score increased to 90 percent, CSUQR overall scores
declined by one point to 87 percent. Differences (p < .01) were found in each of the review type
categories following scoring as based on statewide average results compared to the previous year,
except for Service Guidelines. Thus, significant differences were further evaluated at the indicator
level to aid in deciphering strengths and improvement needs. These included the following:
The BHQR Assessment and Planning category resulted in significant improvements related to
the three indicators with the lowest scores in the past, co-occurring health conditions being
addressed/included in the IRP/NCP (FY18 60%; FY19: 70%), discharge plan defining criteria
(FY18 66%; FY19 76%), and interventions/objectives being goal-linked and service-consistent
(FY18 92%; FY19 96%). The exact opposite was found in the case of CSUQRs by which the
same exact indicators declined (p < .01) to a result of 36 percent (FY18 54%), 48 percent
(FY18 69%), and 90 percent (FY18 96%).
While ten of the 18 services reviewed with the BHQR Service Guidelines demonstrated
improvement from FY 2018 to FY 2019, Nursing continuously progressed (p < .01) year to
year since FY 2017. Nursing documentation reflected education to individuals related to
identified health issues, individualized goals, individuals’ progress (or lack thereof), and
response to treatment/interventions.
For the second year in a row, of the three BHQR FOA Safety indicators, the one most often
scored “no” was “Individuals (or their legal guardians) signed medication consent forms along
with the prescriber”. However, results of this indicator have continued to trend upward yet
again by five points to a result of 73 percent for FY 2019.
The BHQR Choice FOA indicators all exceeded 90 percent in FY 2019, so too was the case for
the FOA, Community Life. However, while four of the five Community Life indicators declined
Quality Management Annual Report FY 2019
116 | P a g e
by one point, documentation across the network with the individual having age-appropriate
responsibilities/valued roles in the community improved significantly (p < .01) to 98 percent.
Specific to the CSUQR IRR, only Admission/Initial Evaluation/Initial Screening and Course of
Stay had categorical improvement. Course of Stay (90%) had a six percent (p < .01) increase
for FY 2019 compared to FY 2018 with 90 percent of records having documented the
individual’s participation in training/therapy with even more records demonstrating the
individual being offered groups as needed (95%, p < .01). One hundred percent of C&A CSU
records (ages 17 and under) had clear documents to support the individual participating in
training/therapy as well as being offered groups as needed.
Although part of a non-scored element within CSUQR IRR category, follow-up and connection
to continuing care statistically decreased (p < .01) by 12 points to a low result of 39 percent.
Moreover, documentation of vital signs (non-scored indicator) improved (p < .01) from 62
percent in FY 2018 to 72 percent in FY 2019. Fifty-seven percent of records for individuals
aged 17 or under met this standard compared to those aged 18 and over (73%).
While no significant improvements were identified specifically in CSUQR Service Guidelines,
several measured standards or indicators related to staff access, staff ratios, and staff
availability were met at 100 percent.
All CSUQR Whole Health FOA indicators improved significantly; demonstrating documented
safeguards for utilized medications known to have substantial risks or side effects (93%),
ongoing assessment to determine external referral needs (99%), as well as communication
with external referral sources and providers as needed (99%).
While several providers remained consistent or demonstrated improvements in their results from FY
2018 to FY 2019, indicator level declines provide for opportunities for improvement, especially in the
case of targeted BHQR services as well as CSUQR categories. Several targeted training efforts have
occurred over FY 2019 to address low scoring, downward-trending indicators and or service specific
trainings resulting from BHQR or CSUQRs scoring as well as via interactive technical assistance
interactions that occurred throughout the year with providers directly. Additionally, Quality
Quality Management Annual Report FY 2019
117 | P a g e
Management has devised several recommendations identified below based on results supplied
throughout the FY 2019 annual report.
Recommendation I: Due to fifty-four percent of reassessed providers scoring lower on the
subsequent review in FY 2019, a recommendation is being made to adhere to the existing threshold
requirements to remain at 90 percent for Overall and Billing scores at the provider level.
Recommendation II: Throughout the year, Quality Management monitors various provider findings,
specific service details, and indicator level review results to evaluate continuous upward/downward
trends. Additionally, provider reassessment (providers reviewed for a second time for not meeting
the annual threshold) data and findings are analyzed and reported. With these findings the
Collaborative has identified and recommended changes to both the BHQR and CSUQR tools to
further supplement the quality review process and reporting. Therefore, an added section entitled
“Quality Risk Items” is being recommended for FY 2020 to include an Overall score point reduction (2
percentage points) for each identified item. Quality Risk Items may result in a provider review
reduction with up to a 10-point maximum reduction for items that continue to be recommended at
each review for the specific provider it applies. Reductions also may be applied in cases of high-level
concerns including, but not limited to:
Columbia Suicide Severity Rating Score not in three or more records
Duplicated documents in three or more records
Provider lacks proof of background check on employee, staff, or contractor
Required staffing is incomplete for more than 90 days in a program
Repeat quality improvement recommendations on subsequent reviews (due to low
reassessment scores of similar indicators)
For a complete listing of Quality Risk items for FY 2020, access the BHQR/CSUQR Tool – Quality Risk
Items at the Collaborative’s website.
Recommendation III: The Collaborative recommends the introduction of additional reasons that
claims are not justified. These additional reasons would better align the Collaborative’s scoring
policies with other entities such as Department of Community Health. These additional discrepancy
Quality Management Annual Report FY 2019
118 | P a g e
reasons are also intended to increase the quality of documentation within individuals’ records by
asking providers to align their documentation more fully with existing standards. These discrepancy
areas include:
High utilization without clinical justification
Intervention unrelated to IRP w/o clinical justification
Monthly contacts not met per DBHDD Service Guidelines
No overall progress documented
Note does not include response to intervention
Credentials not submitted within required timeframe
Progress note not filed within seven calendar days
Recommendation IV: Continue annual review and evaluation of existing BHQR and CSUQR tools and
requirements, especially for recently added indicators to align with current state requirements and
DBHDD recommendations in accordance with the DBHDD Provider Manual.
Recommendation V: Based on analysis of FY 2019 review and reassessment data, continue the
Quality Training Series in FY 2020 and participate in the 2020 Behavioral Health Symposium, for all
BH and CSU agencies to include, but not limited to, the following:
Partnering with Individuals for Effective Recovery Planning
Behavioral Health Assessment: It’s More Than You May Think
Professional Self-Care and Boundaries
New Provider Trainings
o Overview of Collaborative/Intro to Quality Management (QM) /DBHDD
Provider Manual
o Clinical Care Coordination/QM
Targeted training based on service, category, or QM recommendations:
o Billing and Documentation
o Medication Assisted Treatment (MAT) Provider Training
o Certified Peer Specialist (CPS) Documentation Training
Recommendation VI: In FY 2019, DBHDD released its FY 2017 BH Suicide Mortality Report. Through
review of this document as well as various provider review findings and DBHDD collaborative
discussions, the Collaborative has been approved to begin data collection utilizing a tracer
Quality Management Annual Report FY 2019
119 | P a g e
methodology to follow individuals who have been readmitted to a CSU and have some form of
suicidality at the time of first admission, known as the Quality Management Suicidal Ideation Quality
Improvement Study. Additionally, a specific question is to be included as a scored element beginning
with Tier 1 and Tier 2+ providers specific to suicide risk screening, and the Columbia Suicide Severity
Risk Scale (CSSRS) in the Assessment and Planning (BHQR) or IRR (CSUQR) sections as appropriate.
Recommendation VII: Findings of ad hoc reporting throughout the fiscal year, led to findings in
which it was identified some services may be underrepresented compared to utilization reports (e.g.
Payment Service Mix Report, internal claims review/analysis). Furthermore, as some services
become more familiar or available to individuals, changes to BHQR/CSUQR tools to include these
services may be required. Thus, in conjunction with DBHDD, services or changes to services
reviewed may occur on an annual basis and it is being recommended that Quality Management
begin reviewing the following service types for FY 2020:
Intensive Case Management subsample (15 additional records)
Office of Deaf Service Review
Crisis Respite Apartment
Supported Employment
Recommendation VIII: As the Programmatic category pilot proved successful through FY 2019,
supplying supportive supplemental information related to certain services, it is recommended that
Quality Management at the request of DBHDD will continue to collect data specific to these
indicators and requirements as a non-scored element.
Recommendation IX: It is notable that all Interview, individual and staff, areas scored in the 90th
percentile since contract inception, indicating a high level of satisfaction for both individuals served
by the providers and the services received, as well as staff knowledge in providing service to
individuals. Furthermore, little difference was identified between scores of similar indicators asked
of both individuals and staff. As a direct result of such four year findings related to the individual and
staff interviews, it has been recommended by the Collaborative and approved by DBHDD to
Quality Management Annual Report FY 2019
120 | P a g e
eliminate staff interviews beginning in FY 2020, and reduce the individual interview to a set of ten
questions.
Recommendation X: Three of the five new providers, never before having been reviewed by Quality
Management, scored within the 10 lowest of all 157 reviews. Furthermore, analysis at the indicator
level provided an additional level of detail by which it became evident that new providers require an
additional level of training or technical assistance prior to their initial review occurring. Therefore,
Quality Management is recommending for FY 2020 a collaboration and inclusion of a training model
specifically tailored to new providers.
Recommendation XI: Inclusion of Immediate Actions/Recommendations at BHQR and CSUQR exit
conferences will continue in FY 2020. Furthermore, Quality Management will tailor the formal Exit
and Final Assessment documents to include any Quality Risk Items identified documented within the
first two pages for accessibility and visibility.
Quality Management Annual Report FY 2019
121 | P a g e
Section 4: Intellectual and Developmental Disabilities
Background Person Centered Reviews (PCR) and Quality Enhancement Provider Reviews (QEPR) are used to
monitor the portion of the Quality Management system that offers services to individuals with
intellectual and developmental disabilities (IDD). Assessors evaluate providers’ service delivery
systems to determine the quality of services offered and the quality of life of people receiving those
services.
The purpose of the PCR is to assess the extent to which individuals with IDD are supported by the
services they receive, are satisfied with those services, and achieve outcomes important to them.
The purpose of the QEPR is to review providers’ systems and practices to ensure they meet
requirements set forth by the Medicaid waiver and DBHDD, and to evaluate the effectiveness of their
service delivery system. Follow-up review activities offered through the Quality Technical Assistance
Consultation (QTAC) provide targeted technical assistance to help providers improve service delivery
systems and address any issues specific to individuals served.
Quality Assessors use various tools to collect data from interviews, observations, and record reviews
to compile a well-rounded picture of individuals receiving services: how supports and services are
provided by their circle of supports, how involved individuals are in the decisions and plans
developed for them, as well as the use of person centered practices and quality of services provided.
In this section of the report, results are presented for each review tool for both the PCR and QEPR.
Aggregate scores of IDD quality reviews are a weighted average, based on the total number of
standards scored [total met / (total met + total not met)]. Except for the Administrative Staff
Qualifications and Training (Q&T) and Developmental Disability Service Specific (DDSS) tools,
indicators within each tool are grouped into six Focused Outcome Areas (FOAs) important to achieve
and maintain:
Whole Health—individuals receiving services are healthy, aware of their health-related
needs, and direct their own health care regimen
Quality Management Annual Report FY 2019
122 | P a g e
Safety— individuals receiving services are safe in their home and work environments and in
their communities; they understand or are learning how to self-preserve in all environments
Person Centered Practices—supports and services are provided based on personal
preferences and direction given by the individual receiving services
Community Life—individuals receiving services are actively participating and developing
social roles in their communities as desired
Choice—information needed to make informed choices on life decisions, such as where to
live, where to work, and which supports, services and providers to use, is available and
individuals receiving services are included in these decisions
Rights—rights are upheld and information and education is provided to ensure
understanding of rights
Sampling Method Person Centered Review (PCR) The PCR uses a random sample of 484 eligible adults, age 18 and over, who had not received a PCR
during the previous year and were receiving services reviewed through this contract. The PCR sample
was stratified and sampled proportionate to each of the six DBHDD regions; therefore, the number
of PCRs per region is proportionate to the number of individuals receiving services within the region.
Table 24 shows the number and percent for the eligible population (sampling frame), as well as the
number and percent of PCRs completed within each region.
Quality Management Annual Report FY 2019
123 | P a g e
Table 24. FY 2019 PCR Sample by Region
Population PCR Sample
Region N Percent N Percent
1 2,456 20.2% 92 19.0%
2 2,076 17.0% 88 18.2%
3 3,034 24.9% 119 24.6%
4 1,324 10.9% 56 11.6%
5 1,628 13.4% 62 12.8%
6 1,665 13.7% 67 13.8%
Total 12,183 100% 484 100%
Quality Enhancement Provider Review (QEPR)
Providers offering waiver or state funded services are eligible for a QEPR, a total of 100 reviewed
each year. The QEPR sample for FY 2019 was selected as follows:
Forty-four providers who had not participated in a QEPR during the first three years of the
contract (FY 2016 – FY 2018) were selected for review.
An additional 48 providers were randomly selected from active providers who had been
reviewed previously, but not since the first year of the contract (FY 2016), for a total of 92
service providers.
Three Support Coordination agencies were randomly selected from agencies that had not
been reviewed in the previous year (FY 2018).
Five providers were selected by DBHDD, generating a total sample of 100 providers for the
QEPR.
Three of the 100 selected providers offered crisis services. Because these services are very different,
they are monitored with a separate QEPR tailored to crisis services. Therefore, QEPR scores are
presented separately for 97 service providers and three crisis providers.
Quality Management Annual Report FY 2019
124 | P a g e
Based on the number of eligible individuals served, providers were stratified into three categories by
size: “Small,” “Medium,” and “Large.” The number within each size category is shown by year in
Table 25.
Table 25. QEPR Provider Sample by Size and by Year
Provider Size FY 2017 FY 2018 FY 2019
Small (caseload ≤ 30) 52 72 47
Medium (30 < caseload < 100) 19 12 22
Large (caseload ≥ 100) 24 13 25
Support Coordination Agency 1 2 3
Crisis Stabilization Unit 4 1 3
Total 100 100 100
As part of the QEPR, a sample of staff working directly with people receiving services is selected to
complete the review of staff qualifications and training. The selection is random, stratified to ensure
all services the provider offers are included.
Review Processes
PCR and QEPR The focus of the PCR is on the quality of life people receiving services have and the quality of
services received, from the perspective of the individual receiving services. The focus of the QEPR is
on the provider’s overall practices, quality of services offered to all individuals served by the
provider, and level of compliance with Medicaid waiver and state requirements. Both the PCR and
QEPR use the Individual Observation Staff Assessment (IOSA), which includes an Individual Interview
(II) with individuals receiving services, Staff Interview with their support staff (SI), and on-site
observation(s) (OBS) at residential and day programs, as applicable.
In addition to the IOSA, the PCR and QEPR include an evaluation of the Individual Service Plan using
a quality assurance checklist (ISP QA), a Provider Record Review (PPR) of the provider’s record
maintained for individuals sampled, and review of documentation specific to service delivery
requirements using the Developmental Disabilities Service Specific (DDSS) tool. The number of PRR
Quality Management Annual Report FY 2019
125 | P a g e
and DDSS reviews completed for each process depends upon the number of services received by the
individuals receiving services.
While the PCR and QEPR share most of the same tools, there are a few exceptions. During the PCR,
the Support Coordinator Record Review (SCRR) and Support Coordinator Interview (SCI) tools are
completed to assess services provided by the Support Coordinator (SC) working with the individual
receiving services. The NCI Adult In-Person Survey is also conducted as part of the PCR. Data
collected for the survey are entered in the ODESA system (Online Data Entry Survey Application)
developed and maintained by Human Services Research Institute (HSRI)). HSRI analyzes and
generates annual reports comparing Georgia’s results to other participating states and the NCI
average.11
During the QEPR, each provider organization receives one administrative review to monitor
compliance with requirements through the Qualifications and Training (Q&T) component of the
review. This includes a review of a random sample of staff records to determine if staff has the
required qualifications for specific services rendered (e.g., background screenings, level of
education), and whether a required training was completed within specified timeframes. The total
number of records reviewed or interviews completed in FY 2019 for the PCR and QEPR is listed in
Table 26 by review tool.
11 Go to https://www.nationalcoreindicators.org/ for more information.
Quality Management Annual Report FY 2019
126 | P a g e
Table 26. FY 2019 Number of Reviews by Tool and Review Type
Review Tool PCR
(N = 484) QEPR12 (N = 97)
Total (N = 581)
IOSA - Individual Interview (II) 484 744 1,228
IOSA – Observation (OBS) 356 364 720
IOSA - Staff Interview (SI) 475 359 834
ISPQA Checklist13 479 739 1,218
Support Coordinator Interview (SCI) 484 11 495
Support Coordinator Record Review (SCRR) 484 59 543
Provider Record Review (PRR) 844 1,006 1,850
Staff Qualifications and Training (Q&T) N/A 914 914
DD Service Specific 1,004 1,382 2,386
When a PCR is completed, a report is given to the individual’s Support Coordinator that includes an
evaluation of the supports and services provided, the strengths of the support team (including the
Support Coordinator, provider, and family), and recommendations for the individual interviewed and
the support team. The providers who participated in the PCR receive a report of record review
results specific to the services they provide. A provider who participates in a QEPR receives a
comprehensive report that identifies strengths of the service delivery systems, recommendations for
improvement, record review results and several performance scores: the Overall score, Q&T score,
and DDSS score.
Quality Technical Assistance Consultation (QTAC)
The QTAC is conducted approximately 90 days after completion of the QEPR, based on any service
concerns identified during the PCR or QEPR, or if the provider requests technical assistance.
However, if there is a health, safety or rights Quality of Care (QOC) concern identified during the
QEPR, the provider is notified and a QTAC is conducted approximately 30 days after completion of
the QEPR. During the QOC QTAC, the Quality Assessor will specifically address the concern to
ensure it has been addressed by the provider and whether actions are being taken to prevent further
12 Numbers do not include Crisis providers, who were analyzed separately. 13 Five individuals who were interviewed for the PCR and five for the QEPR had an Employment Express Support Plan and therefore, did not have an ISP QA Checklist completed.
Quality Management Annual Report FY 2019
127 | P a g e
issue. This provider will also receive anther QTAC approximately 90 days after completion of the
QEPR to follow up on all of the other recommendations provided during the QEPR. Technical
assistance is provided to offer suggestions and guidance to improve service delivery systems for a
specific individual receiving services or to address overall provider systems. The process uses a
consultative approach to address specific issues and concerns related to someone receiving services
or systems and practices that need improvement. The QTAC supplements the PCR and QEPR
processes by affording providers the opportunity to solicit technical assistance for specific needs
within their service delivery system. During FY 2019, Quality Management completed 170 QTACs.
Person Centered Review
PCR Scores by Tool
Figure 33 shows the average score for each tool used during the PCR, comparing FY 2016 through FY
2019. Findings within each year show a similar pattern, with scores for the Individual Interview, Staff
Interview, and Support Coordinator Interview higher than scores for provider or Support Coordinator
documentation (record reviews). Scores across all tools decreased from FY 2016 to FY 2017, with
results remaining approximately the same in FY 2018 and FY 2019, with one exception. Interviews
with the Support Coordinator showed a decrease from 83.5 percent in FY 2018 to 79.4 percent in FY
2019. However, a revised Support Coordinator Interview tool was implemented July 1, 2018, and
direct comparisons are not appropriate. Analysis of the new tool will be provided in the Indicator
Analysis by FOA and Opportunities for Growth section.
Quality Management Annual Report FY 2019
128 | P a g e
Figure 33. PCR Scores by Tool and Year
PCR Scores by Focused Outcome Area (FOA)
Indicators in the PCR tools (II, SI, OBS, PRR, SCI, and SCRR) are organized around the six FOAs. Results
for each FOA by fiscal year are shown in Figure 34. While scores for the FOAs had decreased from the
first to the second year of the contract, in FY 2018 (third contract year) scores increased for each
FOA. In the fourth year (FY 2019), FOA scores remained similar to the FY 2018 results.
79.4%
94.3%
93.0%
83.5%
95.6%
92.2%
83.3%
94.3%
91.9%
50% 60% 70% 80% 90% 100%
SupportCoordinator Interview
Staff Interview
IndividualInterview
FY 2017 (N = 481) FY 2018 (N = 484) FY 2019 (N = 484)
76.8%
74.9%
98.6%
77.4%
74.8%
98.2%
73.7%
70.2%
96.8%
50% 60% 70% 80% 90% 100%
SupportCoordinator
RecordReview
ProviderRecordReview
Observation
Quality Management Annual Report FY 2019
129 | P a g e
Figure 34. PCR Scores by FOA and Year
Low-Scoring Indicators by FOA (PCR) and Opportunities for Improvement
Each FOA uses several standards, or indicators, to measure how well the area is being addressed.
Most FY 2019 findings showed relatively high scores across all perspectives of the PCR, particularly
for Individual and Staff Interviews, and on-site observations, averaging over 88 percent for each.
However, there are specific areas within each FOA where scoring was relatively low, has remained
low over the past two or three years, or has decreased significantly since FY 2018. These are
presented in this section by FOA in Tables 27 through 33. All p values, based on a two-tailed test, are
shown for the difference of proportions between FY 2018 and FY 2019 (p value FY18/19). DBHDD has
determined statistical significance for this report to be p < .01 (highlighted in blue).
Whole Health The average Whole Health score in FY 2019 was 86.3 percent, close to the same score as in FY 2018
(87.4%). Support Coordinators scored over 95 percent in documenting how they monitor the health
and welfare of individuals receiving services, how the frequency of contacts increases when health
issues are identified, and how supports and services are provided based on the documented needs
85.2%
92.5%
86.3%
84.4%
91.6%
87.4%
82.6%
90.3%
84.7%
50% 60% 70% 80% 90% 100%
PersonCenteredPractices
Safety
Whole Health
FY 2017 (N = 481) FY 2018 (N = 484) FY 2019 (N = 484)
93.4%
80.9%
77.3%
92.0%
82.3%
76.9%
90.2%
78.7%
71.7%
50% 60% 70% 80% 90% 100%
Rights
Choice
CommunityLife
Quality Management Annual Report FY 2019
130 | P a g e
of the person. Most individuals (99% or more) who were interviewed had access to supports and
services to address whole health needs, access to needed medications, and indicated staff followed
up on their specific needs; over 99 percent also agreed with the medications they were taking and
felt those medications were effective.
For the previous three years, (FY 2016 – FY 2018), individuals receiving services had reported a
continued decline in several areas related to medication use. Each year, results indicated individuals
were less likely to be aware of what medications they were taking and why, the side effects of the
medications, and what the medication should look like. Individuals receiving services were also less
likely each year to be aware of their diagnosis. While remaining the lowest-scoring areas as reported
through the individual interviews, information in Table 27 indicates some improvement in FY 2019,
particularly in ensuring an awareness of the diagnosis.
Staff support for the individual to help them learn about medications and their side effects remained
relatively low in FY 2019. Almost 43 percent of staff working with individuals had not helped them
learn about medication they were taking (57.3% met), down over 11 points since FY 2018. The
proportion of staff unable to describe the medications people were taking or their advanced
directives has also decreased significantly since FY 2018. These findings appear to be integrated, as it
is difficult to help someone learn about medications if not familiar with them (unable to describe
them). Many Support Coordinators, when interviewed, were unable to describe how needs related
to specialty health assessments are addressed (59.0% Met).
Of the 79 indicators in the PRR used to measure Whole Health, the greatest gain was shown in
maintaining documentation that a wheelchair evaluation had been completed, up 10 points since FY
2018 to 71.7 percent (p = 0.087). However, in both years close to 14 percent (n = 11) of the 79
indicators showed a score of less than 70 percent. Approximately 40 percent to 70 percent of
providers did not have documentation to support if the following preventative or specialty
assessments had been completed: Dental (54.9%), Pap/Pelvic exam (30.8%), Mammogram (43.9%),
Quality Management Annual Report FY 2019
131 | P a g e
PSA (44.4%), Bone Density (49.3%), Vision (50.7%) or Hearing (36.9%) Evaluation, and AIMS testing
(69.7%).14
Table 27 displays PRR indicators with low scores that reflected a decrease of more than nine
percentage points since FY 2018. Education on all medications and the risk/benefits of medication
were offered significantly less often in FY 2019; and providers were less likely to maintain
documentation of PSA tests, bone density tests, or evidence of the person’s ability to manage
personal healthcare. While providers are likely to document how medications were properly
prescribed by a psychiatrist or psychiatric nurse practitioner (86.9%), this had decreased significantly
by close to 10 points since FY 2018.
Table 27. Low-Scoring Whole Health Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18- FY1915 Individual Interview Individual receiving services was aware of:
The side effects of the medications 52.4%
(n = 410) 43.6%
(n = 433) 47.2%
(n = 422) p = .291
Why medications are prescribed 68.5%
(n = 410) 62.5%
(n = 435) 67.8%
(n= 425) p = .103
What medications he/she is taking 68.3%
(n = 435) 63.5%
(n = 458) 69.3%
(n = 453) p = .060
His or her diagnoses 71.9%
(n = 473) 64.1%
(n = 482) 71.4%
(n = 482) p = .015
The medication's color, shape, when it is taken 80.4%
(n = 408) 72.6%
(n = 431) 77.1%
(n = 423) p = .130
Staff Interview Staff could describe:
How to support the individual to learn about medications
67.7% (n = 195)
68.4% (n = 307)
57.3% (n = 335)
p = .004
The side effects of medications taken 78.7%
(n = 211) 75.4%
(n = 353) 68.8%
(n = 375) p = .048
Advanced health directives 80.6%
(n = 268) 89.3%
(n = 365) 79.3%
(n = 416) p = .000
14 PSA is Prostrate-Specific Antigen and AIMS is the Abnormal Involuntary Movement Scale 15 Shows the difference in percentages from FY 2018 to FY 2019.
Quality Management Annual Report FY 2019
132 | P a g e
Table 27. Low-Scoring Whole Health Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18- FY1915 Provider Record Review Documentation demonstrated how/included:
Individuals and families are provided education on all prescribed medications
25.7% (n = 723)
48.3% (n = 621)
27.9% (n = 380)
p = .000
Education is provided to the individual on the risks and side effects of the medication
33.9% (n = 610)
52.1% (n = 511)
37.5% (n = 651)
p = .000
Copies of the individual’s current preventative healthcare reports for a PSA test
31.3% (n = 147)
55.6% (n = 166)
44.4% (n = 153)
p = .046
Copies of the individual’s current preventative healthcare reports for a bone density test
36.5% (n = 74)
60.7% (n = 84)
49.3% (n = 67)
p =.161
The person’s ability to maintain personal health including any necessary supports
58.0% (n = 531)
59.9% (n = 690)
49.3% (n = 801)
p = .000
Antipsychotic medications were prescribed by a psychiatrist or psychiatric nurse practitioner
96.0% (n = 373)
96.5% (n = 342)
86.9% (n = 297)
p = .000
Support Coordinator Interview:
Needs related to additional specialty health assessments are addressed by the Support Coordinator.
NA NA 59.0%
(n = 144) NA
Safety The average FOA score for Safety in FY 2019 was 96.5 percent, compared to 91.6 percent in FY 2018.
Almost every individual interviewed (> 98.8%) felt safe in all environments and free from all types of
abuse. Most providers (> 99%) and Support Coordinators (> 96%) reviewed had emergency
preparedness plans in place and had documented specific emergency contact information for the
individual receiving services. Support Coordinators advocated to ensure follow-up occurred for
identified safety concerns (97.0%) and reported critical incidents as per the policy (95.2%).
Table 28 includes the lower scoring Safety indicators for FY 2019, indicating some had changed
significantly since FY 2018. Appropriate communication and training are critical components to
helping ensure individuals with IDD understand and are safe from exploitation and neglect, and how
to report abuse or ask for assistance in the event any type of abuse occurs. Findings suggest many
Quality Management Annual Report FY 2019
133 | P a g e
individuals who received services did not understand what constitutes exploitation or neglect. At the
same time, significantly fewer staff provided abuse, neglect and exploitation training according to
the person’s learning style. The crisis hotline is available if there is a behavioral crisis or event that
produces a risk of hospitalization; however, staff were significantly less likely to provide education to
individuals receiving services on how to use the crisis hotline.
Support Coordinator Record Review standards all showed scores above 90 percent; however,
interviews with coordinators using the newly implemented interview tool indicated most had not
evaluated the effectiveness of behavioral or safety/crisis plans and many had not coordinated
services to prevent the recurrence of a previously identified crisis or safety concern.
While most provider documentation did not show evidence of education on self-preservation or
effective resiliency skills was offered according to the person’s learning style, this standard showed
some improvement since FY 2018, up 4.6 percentage points. The greatest increase for providers was
in documenting evidence of ongoing efforts to identify, address, and seek prevention of abuse,
neglect, and exploitation, up 16.4 percentage points to 55.9 percent.
Table 28. Low-Scoring Safety Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Individual Interview
Individual is aware of or recognizes what constitutes exploitation
76.6% (n = 480)
70.5% (n = 482)
64.1% (n = 482)
p = .034
Individual is aware of or recognizes what constitutes neglect
81.7% (n = 480)
77.0% (n = 483)
71.8% (n = 483)
p = .064
Individual is aware of how to respond in an emergency/safety situation if supports are incapacitated
78.7% (n = 468)
71.5% (n = 481)
74.5% (n = 483)
p = .294
Individual is aware of how to call 911 in an emergency/safety situation
79.8% (n = 470)
75.0% (n = 472)
75.3% (n = 478)
p = .915
Staff Interview
Staff is providing education on how to use the crisis hotline
52.8% (n = 91)
59.9% (n = 352)
45.4% (n= 443)
p = .000
Quality Management Annual Report FY 2019
134 | P a g e
Table 28. Low-Scoring Safety Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Staff is not training the person according to a person’s learning style, on abuse, neglect, and exploitation
83.4% (n = 302)
85.3% (n = 441)
74.0% (n = 473)
p = .000
Provider Record Review Documentation demonstrated:
How education is offered to the person on how to self-preserve or develop effective resiliency skills according to the person’s learning style
41.1% (n = 893)
41.5% (n = 843)
46.1% (n = 840)
p = .057
Ongoing evidence of identifying, addressing, and seeking prevention of abuse, neglect, and exploitation
22.9% (n = 893)
39.5% (n = 838)
55.9% (n = 832)
p = .000
Evidence of a written order by the physician including the rationale and instructions for the use of adaptive supportive devices or medical protective equipment
72.3% (n = 101)
66.1% (n = 165)
59.0% (n = 166)
p = .182
Monitoring and analysis of Safety/Risk assessments16
69.6% (n = 332)
71.5% (n = 522)
66.6% (n = 580)
p = .079
Monitoring and analysis of the Positive Behavioral Support Plan
72.9% (n = 85)
93.5% (n = 92)
84.3% (n = 83)
p = .051
Support Coordinator Interview
Support Coordinator is aware of safety/crisis or behavioral plan and evaluates effectiveness of the plan
NA NA 44.7%
(n = 107) NA
When informed of a crisis or concern, the Support Coordinator coordinates supports and services to prevent recurrence of a similar behavioral crisis or safety concern
NA NA 55.9%
(n = 127) NA
Person Centered Practices
Person Centered Practices showed an average score of 85.2 percent, similar to FY 2018 (84.8%).
Almost all individuals interviewed (> 99%) felt their preferences and cultural beliefs were solicited
and their preferred communication styles were offered to them. All but five individuals indicated
16 In FY 2017, the indicator was written to only include safety assessments and did not include safety/risk assessments.
Quality Management Annual Report FY 2019
135 | P a g e
they were involved in the development of their Individual Service Plan (ISP). Support Coordinators
almost always had the ISP in the record (98.1%) and almost all providers (99.3%) had evidence the
ISP was reviewed and updated as warranted and needed. These findings help support the underlying
essence of person-centered practices: knowing the person well and ensuring the person is involved
in decision making.
Person Centered Practices also mean the person has control of decisions, goals, and activities and is
provided options to support informed choice for these. Findings each year (Table 29) suggest many
individuals receiving services were unable to describe something new they had experienced or
learned in the past six months. Data also indicate many individuals did not regularly review progress
toward their own goals or determine when a goal had been achieved. At the same time, findings
suggest most providers and many Support Coordinators were not regularly reviewing goals with
individuals they served, 41.8 percent and 43.2 percent met respectively, and approximately 20
percent of Support Coordinators could not describe how they evaluate progress on goals. Compared
to FY 2018, providers and Support Coordinators were significantly less likely to consistently include
documentation of the individual’s responses to care, services and treatment (61.7% and 53.1%
respectively).
Table 29. Low-Scoring Person Centered Practices Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Individual Interview
In the past 6 months, the Individual was able to identify something:
New or experienced 60.3%
(n = 471) 62.6%
(n = 479) 62.9%
(n = 477) p = .924
Had been learned 65.8%
(n = 474) 67.2%
(n = 478) 71.1%
(n = 478) p = .192
Person is involved in regular review of progress toward all goals
72.7% (480)
73.6% (n = 467)
70.1% (n = 469)
p = .234
Individual determines when progress on goals is achieved or goals are met
81.8% (n = 479)
75.3% (n = 482)
79.4% (n = 480)
P = .129
Staff Interview
Staff formally reviews progress on goals/objectives with the individual
72.2% (n = 302)
68.4% (n = 434)
66.5% (n = 465)
p = .545
Quality Management Annual Report FY 2019
136 | P a g e
Table 29. Low-Scoring Person Centered Practices Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19 Provider Record Review
Documentation reflected:
The individual’s hopes and dreams 42.0%
(n = 893) 31.5%
(n = 847) 40.6%
(n = 834) p = .000
Review of progress and benefit of goals occurs regularly with the individual
40.4% (n = 891)
44.0% (n = 848)
41.8% (n = 837)
p = .362
The individual’s talents/strengths17 NA 44.2%
(n = 847) 44.8%
(n = 841) p = .804
The individual’s response to the services, supports, care and treatment as a consistent theme
63.3% (n = 894)
68.7% (n = 853)
61.7% (n = 841)
p = .003
Support Coordinator Record Review
Documentation demonstrated:
The individual's talents 28.8%
(n = 479) 35.3%
(n = 484) 38.9%
(n = 483) p = .247
Review of progress and benefit of goals occurs regularly with the individual
56.1% (n = 472)
49.8% (n = 484)
43.2% (n = 482)
p = .040
The individual's strengths 41.0%
(n = 480) 44.2%
(n = 484) 47.0%
(n = 483) p = .32
The individual’s response to the services, supports, care and treatment as a consistent theme
61.6% (n = 479)
66.7% (n = 484)
53.1% (n = 481)
p = .000
Progress notes describe progress toward goals including response to the intervention or activity, based on data
57.8% (n = 481)
62.7% (n = 483)
56.7% (n = 483)
p = .057
Support Coordinator Interview:
Support Coordinator evaluates progress of person’s goals and objectives
NA NA 79.4%
(n = 480) NA
Community Life Community Life, interaction with and integration in the surrounding community, showed the lowest
average score among the FOAs each year of the ASO contract, with 77.3 percent in FY 2019;
however, during interviews with individuals, most indicated they go out in (99.0%) and actively
17 Two separate indicators were combined into one for FY 2018.
Quality Management Annual Report FY 2019
137 | P a g e
participate in (97.7%) the community. They chose their community activities (98.1%) and indicated
they have supports needed when participating in the community (98.8%). Compared to FY 2018, a
significantly higher proportion of individuals felt they had opportunities to learn about social roles
(82.5%) (Table 30). Providers showed evidence (PRR) community services were actually provided in
the community as intended (98.6%) and they were aware of people’s interests and preferences for
community activities (98.9%).
Additional information from the PCRs regarding integration into the community is shown in Table 30.
Individuals indicated they were often not getting exposure to new activities in the community or
pursuing preferences related to employment goals, 67.6 percent and 79.9 percent respectively.
As noted above, many individuals receiving services felt they had opportunities to learn about social
roles; however, fewer were able to identify a social role that is important to them. In addition, many
staff members were not supporting individuals to develop new social roles, and Support
Coordinators were often not coordinating supports to develop or maintain social roles.
Table 30. Low-Scoring Community Life Interview Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Individual Interview The individual was:
Exposed to new community activities ( in the past 6 months)
63.2% (n = 473)
70.1% (n = 478)
67.6% (n = 463)
p = .408
Actively pursuing preferences related to goals of employment (not a readiness model)
70.4% (n = 338)
71.5% (n = 312)
79.9% (n = 364)
p = .011
Able to identify a valued social role 75.5%
(n = 478) 76.7%
(n = 480) 73.9%
(n = 482) p = .314
Provided opportunities to learn about social roles in the community
71.5% (n = 478)
75.8% (n = 476)
82.5% (n = 479)
p = .011
Staff Interview
Staff was able to describe how the individual is provided opportunities to:
Develop community employment 58.3%
(n = 235) 60.7%
(n = 338) 62.6%
(n = 396) p = .598
Develop new social roles 60.4%
(n = 298) 70.6%
(n = 429) 71.4%
(n = 447) p = .794
Quality Management Annual Report FY 2019
138 | P a g e
Table 30. Low-Scoring Community Life Interview Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Support Coordinator Interview The Support Coordinator was:
Coordinating services and supports to promote opportunities for the person to develop new valued social roles
NA NA 62.6%
(n = 478) NA
Coordinating services and supports to promote opportunities for the person to maintain valued social roles.
NA NA 67.7%
(n = 477) NA
Documentation from Support Coordinators (SCRR) and providers (PRR) reflected several consistently
low-scoring areas (Table 31). Many were not documenting how they supported people to seek
employment in competitive settings, develop social roles that reflect the person’s interests, or to
experience the community. Approximately 53 percent to 64 percent of providers and Support
Coordinators supported individuals to participate in the community the same as citizens without
disabilities, a key component in the Centers for Medicaid and Medicare Services (CMS) Settings Rule,
and have responsibilities in the community as desired. 18
Table 31. Low-Scoring Community Life Record Review Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
Documentation demonstrated:
Opportunities to seek employment in competitive integrated settings
Provider Record Review 24.9%
(n = 794) 24.3%
(n = 729) 27.9%
(n = 721) p = .107
Support Coordinator Record Review 41.9%
(n = 394) 43.7%
(n = 373) 44.1%
(n = 442) p = .990
Development of social roles and natural supports that reflect the individual’s interests
Provider Record Review 25.4%
(n = 881) 28.9%
(n = 829) 36.4%
(n = 782) p = .001
Support Coordinator Record Review 46.4%
(n = 481 52.9%
(n = 482) 49.3%
(n = 483) p = .263
18 See the Medicaid website for more information on the CMS Settings Rule: https://www.medicaid.gov/medicaid/hcbs/guidance/hcbs-final-regulation/index.html
Quality Management Annual Report FY 2019
139 | P a g e
Table 31. Low-Scoring Community Life Record Review Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19
How the individual is supported to learn about, explore and experience the community
Provider Record Review 39.5%
(n = 891) 44.6%
(n = 827) 51.1%
(n = 788) p = .009
Support Coordinator Record Review 39.0%
(n = 480) 45.7%
(n = 481) 47.1%
(n = 482) p = .663
How the individual is supported to have or has responsibilities in the community as desired
Provider Record Review 42.8%
(n = 883) 50.7%
(n = 825) 53.2%
(n = 785) p = .316
Support Coordinator Record Review 59.5%
(n = 477) 63.5%
(n = 479) 60.9%
(n = 483) p = .406
How the individual is supported to/able to participate in community activities and employment the same as individuals without disabilities
Provider Record Review 40.4%
(n = 854) 55.6%
(n = 806) 53.4%
(n = 776) p = .380
Support Coordinator Record Review 62.2%
(n = 458) 62.6%
(n = 462) 64.8%
(n = 477) p = .483
Choice Choice is the second, lowest-scoring FOA, showing an average score of 80.9 percent for FY 2019,
similar to FY 2018 (82.3%). Of the 22 standards measured through interviews with individuals
receiving services, 20 reflected a score of over 94 percent or higher, indicating people felt they were
generally satisfied with services, involved in life’s decisions, and offered choices. Of note is a
significant positive increase (p < .001) of 11 points since FY 2018 to 85.6 percent (Table 32), showing
individuals felt they were much more often involved in decisions about work and supported
employment.
Choice is intimately linked to ensuring services are person-centered, based on preferences and
controlled by the individual receiving services, specifically informed choice where viable options are
provided. As indicated in Table 32, staff was often not able to describe how options for competitive
employment were provided (51.2% met) or for integrated living settings, the latter showing a
significant decrease of 11 points since FY 2018, to 53.8 percent. Findings from record reviews often
Quality Management Annual Report FY 2019
140 | P a g e
indicated a lack of documentation for how individuals receiving services were making meaningful
and informed choices in different aspects of their lives, including living environments (e.g., how
personal the environment is), living situations (e.g., where, roommates), competitive employment
and education options, and community participation.
Education, exploration, and experiences (the “3Es”) are critical aspects of enhancing people’s
awareness of options available to them. These activities were often not documented by providers
and Support Coordinators. In addition, Support Coordinator documentation showed a decrease in 11
of the 14 SCRR Choice indicators, and coordinators who were interviewed were often not able to
describe how the individual receiving services expresses dissatisfaction with those services. Not
understanding when someone is unhappy with services may inhibit processes of seeking out other
options and opportunities to better serve individuals receiving services.
Table 32. Low-Scoring Choice Indicators (PCR)
FY 2017 FY 2018 FY 2019 p value
FY18 - FY19 Individual Interview
The individual is involved in decisions about work and supported employment
73.7% (n = 327)
74.5% (n = 310)
85.6% (n = 365)
p = .000
Staff Interview Staff is presenting options of:
Competitive/supported employment 50.5%
(n = 220) 49.2%
(n = 313) 51.2%
(n = 383) p = .560
Living situations are integrated into local community (i.e., full continuum of housing options, roommate)
65.5% (n = 905)
65.5% (n = 284)
53.8% (n = 356)
p = .003
Provider Record Review Providers demonstrated through documentation how individuals:
Exercise meaningful choices about living environments
21.1% (n = 715)
26.3% (n = 759)
26.5% (n = 791)
p = .929
Are provided a choice of living situations 26.4%
(n = 708) 26.0%
(n = 739) 29.1%
(n = 787) p = .176
Make informed choices about competitive or supported employment options
30.8% (n = 827)
26.5% (n = 759)
32.1% (n = 795)
p = .015
Quality Management Annual Report FY 2019
141 | P a g e
Table 32. Low-Scoring Choice Indicators (PCR)
Are provided with information to make informed choices (education, exploration and experiences)
27.4% (n = 891)
27.8% (n = 845)
36.6% (n = 835)
p = .000
Make informed choices about community participation and social interaction
42.8% (n = 888)
47.5% (n = 832)
49.3% (n = 819)
p = .464
Support Coordinator Record Review Support Coordinators demonstrated through documentation how individuals:
Are provided with education, exploration and experiences
31.2% (n = 481)
35.8% (n = 483)
31.0% (n = 484)
p = .114
Are offered employment or educational options
49.3% (n = 423)
52.7% (n = 438)
51.2% (n = 463)
p = .652
Exercise meaningful choices about the living environment
50.3% (n = 441)
53.6% (n = 476)
50.7% (n = 477)
p = .370
Are provided choices of living situations 47.1%
(n = 433) 55.6%
(n = 466) 50.1%
(n = 475) p = .091
Exercise meaningful choices about community participation
50.3% (n = 441)
66.3% (n = 483)
67.6% (n = 482)
p = .668
Support Coordinator Interview
Support Coordinator addresses any dissatisfaction the person expresses relating to supports and services, including Support Coordination
NA NA 55.6%
(n = 135) NA
Rights Scores surrounding Rights have remained relatively high each year since FY 2016, showing an
average score of 93.4 percent in FY 2019. Individuals interviewed felt they had privacy (99.6%), did
not have rights restrictions (99.9%), and were treated with respect (99.6%). In addition, staff was
aware of individuals’ rights and preferences for exercising those rights (99.3%), providers (99.7%)
had a signed consent form to authorize the release of information, and almost all (99.5%) provided
evidence funds were not co-mingled between the individual and the provider. During the interview,
most Support Coordinators (86.3%) could describe how they advocate for individuals receiving
services to exercise their rights.
Quality Management Annual Report FY 2019
142 | P a g e
Findings in Table 33 show the lowest-scoring indicators for the Rights FOA. While staff scored quite
high in most areas of rights, staff was also significantly less likely, compared to FY 2018, to evaluate
individuals’ understanding of how to exercise their rights. On average, PRR findings were highest on
the Rights FOA; however, the lowest-scoring areas indicate many providers were not offering
ongoing rights education for people receiving services or ensuring individuals were protected, when
applicable, with a lease or other legal agreement to protect them from eviction.
Support Coordinators’ documentation showed improvement on seven of eight Rights indicators
scored through the SCRR. However, 37.3 percent showed evidence of a signature on the notification
of rights and responsibilities, and only 48.4 percent had a signed informed consent form for
psychotropic medications. While documentation often did not show evidence the coordinator
shared the complaints/grievance policy according to the individual’s learning style, this indicator
showed an 18-point increase since FY 2018.
Table 33. Low-Scoring Rights Indicators (PCR)
FY 2017 FY 2018 FY 2019
p value FY18 - FY19
Staff Interview Staff is presenting options of:
Staff has a means to evaluate/assess the person’s understanding of and preferences for exercising rights.
78.3% (n = 299)
84.5% (n = 432)
76.5% (n=464)
p = .003
Provider Record Review Provider records demonstrated evidence of:
Ongoing education about rights 32.6%
(n = 891) 43.0%
(n = 849) 49.3%
(n=827) p = .010
For Provider-Owned or Controlled Residential Settings, there was documentation demonstrating the individual has a lease or other legally enforceable agreement to protect from eviction
44.3% (n = 201)
49.8% (n = 227)
53.0% (n = 185)
p = .518
Support Coordinator Record Review Support Coordinator documentation demonstrated evidence of:
Quality Management Annual Report FY 2019
143 | P a g e
Table 33. Low-Scoring Rights Indicators (PCR)
FY 2017 FY 2018 FY 2019
p value FY18 - FY19
The individual or legal guardian's signature on notification the individual has been informed about rights and responsibilities, at least annually
22.7% (n = 476)
23.6% (n = 461)
37.3% (n = 458)
p = .72
Informed consent for taking psychotropic medications prescribed by a psychiatrist or psychiatric nurse practitioner
35.5% (n = 251)
41.1% (n = 248)
48.4% (n = 225)
p = .19
The complaints/grievance policy is shared in a manner accommodating the individual’s learning style
34.7% (n = 479)
42.0% (n = 483)
60.1% (n = 476)
p = .02
Support Coordinator Interview Support Coordinator advocates to ensure:
Plans are in place to reinstate any rights that have been restricted for safety or legal reasons.
NA NA 24.7%
(n = 73) NA
PCR Scores by Tool and Focused Outcome Area (FOA)
PCR results for FY 2019 are presented by FOA and tool in Figure 35. Findings for the review
components varied across each FOA, and indicated the following:
Record review results (PRR and SCRR) were at least 15 points lower than other tools for
Choice.
The Provider Record Review results reflected the lowest score in Whole Health, Person
Centered Practices and Choice.
Provider Record Review results were highest for Rights but, similar to previous years, less
than 90 percent met (86.3%).
SCRR was greater than 93 percent in areas of Whole Health and Safety; however, SC
Interviews were among the lowest scores for these FOAs.
For Rights, the Record Reviews for Support Coordinators showed the lowest score; over 10
points lower than any other tool.
Quality Management Annual Report FY 2019
144 | P a g e
PRR and SC Interview results were more than 10 points lower than other tools measuring
Safety.
Figure 35. FY 2019 PCR Scores by Tool and FOA
Low-Scoring FOAs by Perspective (Tool)
Each tool used in the PCR process represents a different perspective on quality of services provided,
and the extent to which goals and outcomes are addressed through those services. Interviews,
implemented as a conversation, offer the person receiving services, the provider and the Support
Coordinator the opportunity to describe how services are offered and received and what their
impact may be. Documentation review (PRR and SCRR) offers the opportunity for providers and
77.9%
81.8%
71.9%
79.1%
95.6%
93.8%
97.2%
99.9%
97.5%
94.7%
95.5%
91.9%
85.4%
76.1%
83.0%
89.6%
92.3%
89.1%
50% 60% 70% 80% 90% 100%
Person Centered Practices
Safety
WholeHealth
Individual Interview SC Interview Staff Interviews Observation SCRR PRR
86.3%
58.5%
70.3%
67.0%
66.9%
67.0%
99.0%
99.0%
97.5%
97.6%
91.9%
88.4%
79.7%
83.4%
74.2%
98.4%
96.6%
88.0%
50% 60% 70% 80% 90% 100%
Rights
Choice
Community Life
Quality Management Annual Report FY 2019
145 | P a g e
Support Coordinators to show evidence of activities and how policies and procedures are
implemented through their service delivery systems.
The following graphic displays the lowest-scoring FOA from each tool or perspective. Conversations
with people involved in the quality management system indicated community inclusion (FOA –
Community Life) was the lowest score from data collected through all the different interviews. On
the other hand, documentation from providers and Support Coordinators was least likely to have
evidence of how choice (PRR) and rights (SCRR) are offered to individuals receiving services.
Findings may suggest that while providers and Support Coordinators document community activities,
the people receiving services do not always feel it is present in their lives to the extent desired.
Analysis of these areas at the indicator level is detailed in the Indicator Analysis by FOA (PCR) and
Opportunities for Improvement section.
Quality Management Annual Report FY 2019
146 | P a g e
FOAs by Demographics
Analysis was completed comparing FOA results across regions, residential settings, and funding
source.19 Results are presented for FOA by region, residence and funding source (Figures 36 through
39). To enhance the readability of the side-by side demographics, the start point of the axis is 50
percent. Please note some of the demographic categories have a relatively small sample. For
example, the sample for host homes is 28 and for state funded services (SFS) the sample is 47.
Findings indicate the following:
In FY 2018, Region 1 showed the lowest scores for all the FOAs; however, in FY 2019 the
region was the highest scoring in Choice and second highest scoring for Community Life and
Rights.
Individuals receiving services and living in Region 6 were significantly less likely to be
participating in their Community Life as desired (p = .000), compared to their counterparts
living in other regions.20
Individuals living in Region 2 were significantly less likely to have Choice in their lives (p
< .001), compared to their counterparts living in other regions.
Individuals receiving services and living with a parent scored significantly lower in Whole
Health than their counterparts in any other residential settings (p < .001).
Receiving services while living in a group home showed a significantly lower likelihood of
being integrated into the community as desired when compared to other residential settings
(p<.000).
Results for the COMP waiver indicated individuals were more likely to have Whole Health
indicators met compared to NOW or SFS funding sources (p < .01).
19 Analyses by age group and disability did not generate discernable differences across categories. The “Other” residential category had only three PCRs and results are not shown. 20 It should be noted the difference of proportions test was based on the number of indicators scored; therefore, the n sizes were relatively large, increasing the likelihood of statistical significance.
Quality Management Annual Report FY 2019
147 | P a g e
Figure 36. FY 2019 PCR Scores by FOA and Region
85.2%
92.5%
86.3%
83.5%
94.0%
89.7%
87.7%
94.8%
88.0%
88.8%
92.5%
86.9%
83.7%
92.0%
85.0%
83.1%
90.9%
85.9%
86.3%
92.3%
84.9%
50% 60% 70% 80% 90% 100%
PersonCenteredPractices
Safety
Whole Health
Region 1 (n = 92) Region 2 (n = 88) Region 3 (n = 119) Region 4 (n = 56)
Region 5 (n = 62) Region 6 (n = 67) State (N = 484)
93.4%
80.9%
77.3%
93.8%
81.3%
70.9%
95.1%
83.0%
79.8%
93.7%
82.3%
83.6%
93.5%
79.6%
74.4%
91.2%
75.6%
75.4%
94.0%
85.3%
81.7%
50% 60% 70% 80% 90% 100%
Rights
Choice
CommunityLife
Quality Management Annual Report FY 2019
148 | P a g e
Figure 37. FY 2019 PCR Scores by FOA and Residential Setting
85.1%
92.5%
83.6%
86.8%
92.6%
87.8%
83.5%
91.6%
87.6%
85.2%
92.7%
88.7%
50% 60% 70% 80% 90% 100%
Person Centered Practices
Safety
Whole Health
Group Home (n = 138) Host Home (n = 28)
Own Place (n = 54) With Parents (n = 261)
93.9%
81.4%
77.9%
93.7%
84.0%
83.3%
90.4%
78.1%
80.4%
93.3%
80.2%
74.0%
50% 60% 70% 80% 90% 100%
Rights
Choice
CommunityLife
Quality Management Annual Report FY 2019
149 | P a g e
Figure 38. FY 2019 PCR Scores by FOA and Funding Source
Individual Service Plan Quality Assurance Checklist (ISP QA)
During PCRs, Assessors review the content of the ISP to assess adherence to requirements and the
extent to which the plan addressed specific needs, goals, and desires.
ISP Written to Support a Meaningful Life The ISP QA checklist provides an overall rating for each service plan, based upon the degree to which
the ISP is written to provide a meaningful life for the individual receiving services. Each ISP was
identified as written to support one of the following lifestyles:
Service Life: The ISP supports a life with basic paid services and paid supports. Needs “important
for” the individual were addressed, such as health and safety. However, there is not an organized
effort to provide support in obtaining other expressed desires “important to” the individual, such as
getting a driver’s license, having a home, or acting in a play. The individual receiving services was not
connected to the community and has not developed social roles but expresses a desire to do so.
82.0%
93.1%
82.8%
86.1%
92.7%
84.1%
85.1%
92.3%
87.6%
50% 60% 70% 80% 90% 100%
Person Centered Practices
Safety
Whole Health
COMP (n = 259) NOW (n = 178) SFS (n = 47)
93.5%
79.5%
77.0%
94.2%
82.1%
79.2%
93.1%
80.5%
76.4%
50% 60% 70% 80% 90% 100%
Rights
Choice
CommunityLife
Quality Management Annual Report FY 2019
150 | P a g e
Good but Paid Life: The ISP supports a life with connections to various supports and services (paid
and non-paid). Expressed goals “important to” the person are present, indicating goals and desires
were obtained beyond basic health and safety needs. Individuals may go out into the community but
with only limited integration into community activities. For example, someone may go to church but
not have the opportunity to participate in Sunday school or sing in the choir. Community connections
are lacking, and there is an indication of a desire to achieve more.
Community Life: The ISP supports a life with the desired level of integration in the community and in
various preferred settings. Friends and support beyond providers and family members are
demonstrated, as is the development of meaningful social roles, such as belonging to a book club or
having employment in a competitive rather than segregated environment. Rather than just going to
church, the individual receiving services may be an usher at the church or sing in the choir.
Relationships developed in the community are reciprocal. The ISP is written with goals that help
support moving toward a Community Life, as she or he chooses.
Overall findings from this Life Indicator are presented in Figure 39. A majority of ISPs reviewed each
year was written to support a “Good but Paid Life.” Findings in FY 2019 reflect a significant shift from
Community Life (decreasing 10 percentage points; p=.000) to a Good but Paid Life (increasing close
to 10 points; p =.001), indicating ISPs were significantly less likely to support people receiving
services.
Quality Management Annual Report FY 2019
151 | P a g e
Figure 39. PCR ISP QA Life Indicator by Year
The ISP QA Life Indicator findings by region and year are presented in Table 34. While N sizes are
relatively small across regions, each had over 50 ISPs reviewed, suggesting the following:
The overall downward trend in Community Life seen in Figure 39 appeared to be significant in
three regions (shaded in blue): Region 2 (drop of 23.5 points; p < .001), Region 3 (drop of 9.1
points; p < .018) and Region 5 (drop of 9.5 points; p < .001).
Region 3 showed a steady increase in the proportion of ISPs written to support a Service Life
and a steady decrease in the proportion written to support a Community Life.
ISPs in Region 6 were consistently most likely to represent a Service Life.
Quality Management Annual Report FY 2019
152 | P a g e
Table 34. ISP QA Life Indicator by Region and Year
Service Life Good but Paid Life Community Life
Region FY 2017 FY 2018 FY 2019 FY 2017 FY 2018 FY 2019 FY 2017 FY 2018 FY 2019
1 6.3% 15.7% 11.1% 55.8% 58.4% 74.4% 37.9% 25.8% 14.4%
2 10.0% 5.1% 14.8% 75.6% 67.9% 81.8% 14.4% 26.9% 3.4%
3 8.2% 17.1% 19.3% 74.6% 67.9% 74.8% 17.2% 15.0% 5.9%
4 9.8% 15.9% 3.6% 86.3% 77.8% 92.7% 3.9% 6.3% 3.6%
5 20.6% 8.9% 14.8% 68.3% 75.0% 78.7% 11.1% 16.1% 6.6%
6 23.7% 25.9% 27.3% 69.5% 55.2% 56.1% 6.8% 19.0% 16.7%
ISP Expectations
In December 2017, DBHDD completed modifications to the checklist template and expectations, and
the new form went into effect in January 2018. While the original form has slowly been phased out
as each ISP is renewed, 53 ISP QA Checklists were completed in FY 2019 using the original form with
426 completed using the new ISP form. All the expectations and indicators have been modified, with
the exception of the ISP QA Life Indicator Rating of how the ISP is written, shown in Table 34 for all
479 checklists completed. For the remaining results, data are presented for the original and new ISPs
separately.
Results Using Original ISP QA Checklist
Quality Assessors reviewed 12 different indicators to measure the quality of the ISP. Each indicator is
listed in Table 35 and each has four requirements that must be met. The indicator is rated 0 - 4, “0”
meaning the ISP did not meet any of the requirements and “4” meaning all four of the requirements
were addressed. The percent of ISPs at each rating on 12 different indicators is shown. For example,
56.6 percent of ISPs reviewed in FY 2019 had all four requirements present for the indicator
regarding the communication chart. Data indicate:
On average, 66.4 percent of ISPs reviewed addressed all elements in each section.
The Health and Safety Review section was most likely to have all four indicators met (83.0%).
The ISP indicators measuring how well the individual’s hopes and dreams were addressed and
whether goals were person-centered, were most likely to have two or fewer indicators addressed,
Quality Management Annual Report FY 2019
153 | P a g e
over 26 percent each with a rating of “0”, “1” or “2”. These indicators speak to the extent to which
the Support Coordinator knows and understands the specific needs and desires of people they serve.
Table 35. FY 2019 ISP QA Checklist Ratings by Expectation (N = 53)
Ratings ISP QA Checklist Description 0 1 2 3 4 Average
Person-centered Important to/for 1.9% 0.0% 3.8% 15.1% 79.2% 3.7
Health and Safety Review Section completed accurately and thoroughly. (HRST information should be noted throughout ISP.)
1.9% 3.8% 1.9% 9.4% 83.0% 3.7
SIS completed and support needs are addressed in the ISP. (SIS Information should be noted throughout ISP.)
1.9% 0.0% 0.0% 26.4% 71.7% 3.7
Meeting Minutes 3.8% 3.8% 1.9% 15.1% 75.5% 3.5
Service Summary 1.9% 3.8% 9.4% 15.1% 69.8% 3.5
Communication Chart (this includes traditional and non-traditional forms of communication)
1.9% 0.0% 11.3% 30.2% 56.6% 3.4
Training Goal Action Plan 1.9% 1.9% 3.8% 39.6% 52.8% 3.4
Relationship Map & discussion on ways to develop relationships
1.9% 0.0% 13.2% 28.3% 56.6% 3.4
Rights, Psychotropic Medications, Behavior Supports Section
13.2% 0.0% 1.9% 7.5% 77.4% 3.4
Action Plans/Objectives 1.9% 3.8% 9.4% 30.2% 54.7% 3.3
Hopes and Dreams: If you could be, do or include anything in your life what would it be?
11.3% 3.8% 11.3% 9.4% 64.2% 3.1
Goals are Person Centered. (Include the discussion/rationale, desired outcome and goal to make this determination.)
5.7% 5.7% 15.1% 18.9% 54.7% 3.1
Total 4.1% 2.2% 6.9% 20.4% 66.4% 3.4
Table 36 shows the average rating (0 - 4) by indicator for FY 2017 through FY 2019. Information in the
table indicates:
The average rating for all three years was 3.4 but showed some change by year at the
indicator level.
Indicators measuring how well the individual’s hopes and dreams were addressed and
whether goals were person-centered were the lowest rated expectations all three years.
Quality Management Annual Report FY 2019
154 | P a g e
The greatest difference between FY 2018 and FY 2019 was in scoring the Rights, Psychotropic
Medications and Behavior Supports section, a drop from an average of 3.9 to 3.4.
Table 36. Average Rating by Expectation (0 - 4) by Year
ISP QA Checklist FY 2017
(N = 481) FY 2018
(N = 397) FY 2019 (N = 53)
Rights, Psychotropic Medications, Behavior Supports Section
3.9 3.9 3.4
Health and Safety Review Section completed accurately and thoroughly (HRST information should be noted throughout ISP)
3.9 3.9 3.7
Person-centered Important To/For 3.6 3.7 3.7
Supports Intensity Scale (SIS) completed and support needs are addressed in the ISP
3.6 3.6 3.7
Communication Chart 3.4 3.4 3.4
Service Summary 3.3 3.4 3.5
Relationship Map & discussion on ways to develop relationships
3.3 3.3 3.4
Meeting Minutes 3.2 3.3 3.5
Training Goal Action Plan 3.2 3.2 3.4
Hopes and Dreams 3.0 3.1 3.1
Action Plans/Objectives 3.1 3.1 3.3
Goals are Person-centered 2.9 3.0 3.1
Overall Average 3.4 3.4 3.4
Results Using New ISP QA Checklist
The New ISP QA Checklist (effective date 1/1/2018) uses seven Likert scaled indicators with the
following scoring options: Needs Improvement, Emerging, Achieving and Exceeding. Average
statewide ratings are presented by indicator in Table 37, suggesting almost all the ISPs were
Achieving or Exceeding when documenting, in the Health and Safety section, how all current issues,
needs and/or risks are addressed (95.6%). Several indicators measuring how people are connected
to their communities and how person-centered their services are, were most likely to be scored as
Needs Improvement or Emerging (percent shown in parentheses):
Summary reflects the individual’s community life (approximately 47%)
Quality Management Annual Report FY 2019
155 | P a g e
The relationship map demonstrates the person is connected to the community (close to 42%)
Goals and objectives were person centered (31.9%)
Table 37. FY 2019 New ISP QA Checklist Ratings (N = 426)
Indicator
Needs Improvement Emerging Achieving Exceeds
Communication Chart provides meaningful information staff can use to best support the person.
1.4% 25.1% 40.8% 32.6%
Goals and objectives are person centered. 5.6% 26.3% 31.9% 36.2%
Health and Safety section includes how all current issues, needs and/or risks are addressed.
1.7% 0.0% 62.3% 36.0%
Map demonstrates the person is connected to the community.
6.3% 35.4% 20.4% 37.8%
Profile is detailed and includes person centered information.
5.9% 18.8% 35.9% 39.4%
Summary describes changes in the person’s life in the last year and the supports/plans needed to address the change with the person.
3.5% 27.7% 48.6% 20.2%
Summary reflects the person’s community life. (Maximizes opportunities for community living and receiving services in the most integrated setting.)
10.6% 36.4% 42.7% 10.3%
Average New Checklist Ratings 5.5% 24.4% 40.2% 30.3%
PCR Results by Service
The PCR includes a record review for every service received, i.e., services eligible for review by the
Collaborative. Information in Table 38 shows results for record reviews completed during the PCRs,
by service and year. The N is the number of records reviewed for each service and the percent met is
based on the total number of indicators reviewed. When reviewing the findings, it is important to
note several services had fewer than 20 records reviewed, listed in a separate section in the table.
Among services with 32 or more records, listed first in the table, Supported Employment reflected
the highest score in FY 2019 (81.2%), the same as the FY 2018 results.
Quality Management Annual Report FY 2019
156 | P a g e
Table 38. PCR PRR Results by Service and Year
FY 2017 FY 2018 FY 2019 Service N % Met N % Met N % Met
Provider Record Review 894 70.2% 856 74.8% 844 74.9%
Community Access (Group) 378 67.9% 361 74.6% 371 74.2%
Community Access (Individual) 95 72.9% 92 74.4% 87 74.7%
Community Living 72 66.8% 104 70.1% 74 72.9%
Community Residential 195 73.4% 195 77.1% 164 75.9%
Prevocational 74 64.9% 51 71.6% 71 72.5%
Supported Employment 71 77.8% 32 81.0% 48 81.2%
Behavioral Supports 2 91.7% 8 83.9% 18 81.9%
Nursing Services 1 87.8% 2 68.8% 8 78.3%
Respite 6 61.8% 8 67.6% 3 69.6%
Occupational Therapy - - 1 83.3% - -
Physical Therapy - - 1 82.8% - -
Speech/Language Therapy - - 1 82.8% - -
Support Coordination Record Review 481 73.6% 484 77.5% 484 76.8%
Table 39 displays scores by FOA for the services reviewed through record reviews (PRR and SCRR)
during PCRs competed in FY 2019.21 Among providers with more than 30 records reviewed, the first
seven listed in the table, results indicated the following:
For a majority of services, the lowest score was for Choice, ranging from 50.2 percent for
Community Residential Alternative (CRA) to 75.1 for Supported Employment.
Providers of CRA services scored lowest in Choice and Community Life.
Community Access, offered at the individual level, and Community Living Supports reflected
the lowest Safety scores.
Support Coordinators scored approximately 95 percent in documenting Whole Health and
Safety, but had the lowest average Rights score (66.8%).
21 Note these results are not comparable to FOA findings presented earlier as only the PPR or SCRR are used here and previous results included all tools in the PCR.
Quality Management Annual Report FY 2019
157 | P a g e
Table 39. FY 2019 PCR Record Review Results by FOA and Service
Service Whole Health Safety PCP
Comm Life Choice Rights Average
Community Access(Group) (N = 371) 65.6% 83.8% 79.5% 69.3% 59.0% 88.7% 74.2%
Community Access(Individual) (N = 87) 66.1% 79.2% 79.5% 77.3% 61.6% 85.3% 74.7%
Community Living Support (N = 74) 70.5% 77.2% 74.0% 70.1% 59.6% 84.6% 72.9%
Community Residential Alternative (N = 164) 86.0% 80.2% 73.3% 61.7% 50.2% 84.5% 75.9%
Prevocational (N = 71) 63.6% 82.4% 77.8% 66.9% 56.0% 86.8% 72.5%
Supported Employment (N = 48) 65.4% 85.5% 83.1% 91.2% 75.1% 88.0% 81.2%
Support Coordination (N = 484) 94.0% 95.7% 79.4% 67.2% 66.9% 66.8% 76.8%
Services with fewer than 30 records reviewed
Behavioral Supports (N = 18) 73.7% 85.9% 87.4% 87.8% 81.8% 75.4% 81.9%
Respite (N = 3) 66.7% 73.1% 70.8% 55.6% 55.6% 90.6% 69.6%
Nursing Services (N = 8) 82.0% 77.3% 79.3% 100.0% 43.5% 90.7% 78.3%
PCR Strengths and Recommendations
During the PCR, Assessors identified strengths about services offered and provided
recommendations to help improve services and overall quality of life for people receiving services.
The strengths and recommendations are not included in calculating the provider’s score but are
included in the report as information for the provider. Table 40 displays strengths identified in
approximately half or more of the PCRs completed in FY 2019 and Table 41 displays
recommendations provided during approximately 40 percent or more of PCRs completed during FY
2019.
Staff strengths most often cited included an awareness of unique safety needs, achievements, and
an understanding of how individuals they serve communicate everyday decisions. People receiving
services felt valued, possibly in part because staff promoted their independence. Challenges that
generated recommendations surrounded areas of safety education and skill building to help with
various safety situations, and developing ways to identify new experiences in the community and in
daily activities.
Quality Management Annual Report FY 2019
158 | P a g e
Table 40. FY 2019 Strengths Most Often Identified During a PCR
Strength Instances Percent of PCRs
Staff is aware of the unique safety needs of the person. 328 67.8%
Staff acknowledges the person's achievements. 276 57.0%
Staff has a clear understanding how the person communicates choice making in everyday decisions.
271 56.0%
Person feels valued. 270 55.8%
Staff consistently promotes independence. 245 50.6%
Table 41. FY 2019 Recommendations Most Often Provided During a PCR
Recommendation Instances Percent of PCRs
Provide education, according to the appropriate learning style, on the different types of exploitation and how to keep safe, or what to do in these situations.
208 43.0%
Identify ways to expose the individual to new experiences in the community.
202 41.7%
Conduct "what if" scenarios to determine the individual's skills in various safety situations.
201 41.5%
Ensure daily schedules and activities promote exposure to new things (new places and new faces) and are not stagnant in nature.
193 39.9%
Quality Enhancement Provider Review (QEPR)
QEPR Scores by Size
The QEPR Overall score is based on findings from the Individual Interview, Staff Interview,
Observation and Provider Record Review (PRR). In addition, the Administrative Qualifications and
Training (Q&T), based on a sample of staff rendering services, is used to determine if staff has all the
required education, certifications and training as per services rendered. The Developmental
Quality Management Annual Report FY 2019
159 | P a g e
Disability Service Specific (DDSS) component is used to monitor compliance specific to each service.
Scores from these review components are given to the provider but not included in the provider’s
Overall score. Three of the providers offer crisis services and received a separate crisis specific QEPR
for those services. Because crisis services are very different than typical services, results are
presented separately.
In FY 2019, 97 providers were selected for a QEPR: 47 Small, 22 Medium, and 28 Large. Three
additional QEPRs completed for crisis services providers resulted in a total of 100 QEPRs.
Comparisons by year are not appropriate as the QEPR samples were not selected to be
representative of the state – the proportion of Small providers increased each year through FY 2018.
See Figure 40 for distributions of QEPRs by provider size for FY 2016, FY 2017, FY 2018 and FY 2019.
Figure 40. FY 2016 - FY 2019 QEPR Samples by Provider Size and Type
Overall QEPR Score by Tool and Year
Figure 41 shows the average Overall scores for providers reviewed from FY 2017 to FY 2019 and
scores for each tool used in the review process. Several characteristics of the QEPR samples impact
the ability to track trends and comparisons across years must be made with caution:
47%
72%
52%
47%
22%
12%
19%
36%
28%
15%
25%
16%
3%
1%
4%
1%
0% 20% 40% 60% 80% 100%
FY 2019
FY 2018
FY 2017
FY 2016
Small (caseload ≤ 30) Medium (30 < caseload < 100)
Large (caseload ≥ 100) Crisis Provider
Quality Management Annual Report FY 2019
160 | P a g e
The samples each year were not selected to be representative of all providers.
Some providers were reviewed more than once, particularly for QEPRs completed in FY 2019.
As shown in Figure 40, the proportion of reviews completed by size varied each year.
The Overall score for FY 2019 was 86.2 percent, with Observations and Staff Interviews showing the
highest scores of 97.2 percent and 95.7 percent respectively. As in previous years, the PRR reflected
the lowest-scoring area (73.2%). The Q&T score (83.2%) is similar to providers reviewed during FY
2018 (84.6%).
Figure 41. Overall QEPR Scores by Tool and Year
Overall Score
FY 17: 83.7% FY18: 84.8%FY19: 86.2%
Individual Interview
FY17: 90.7%FY18: 89.6%FY19: 92.3%
Staff Interview
FY17: 95.2%FY18: 95.3%FY19: 95.7%
Observation
FY17: 95.7%FY18: 96.0%FY19: 97.2%
Record Review
FY17: 69.6%FY18: 72.4%FY19: 73.2%
Qualifications and Training
FY17: 82.6%
FY18: 84.6%
FY19: 83.2%
DD Service Specific
FY17: 99.8%
FY18: 99.8% FY19: 99.8%
Quality Management Annual Report FY 2019
161 | P a g e
Overall Crisis Provider Scores
Nine crisis provider QEPRs have been completed, one in FY 2016, four in FY 2017, one in FY 2018,
and three in FY 2019. Figure 42 shows the Overall score and scores by tool and year. Compared to
other providers reviewed with the QEPR, crisis providers showed similar scores across all tools, with
the exception of the PRR on which crisis providers scored higher than other providers, 88.9 percent
and 73.2 percent respectively.
Figure 42. Overall Crisis Provider Scores by Tool and Year
Qualifications and Training (Q & T) and Service Specific (DDSS)
Every QEPR includes a review of staff qualifications and training (Q&T) and service specific reviews
(DDSS). In FY 2019, a sample of 914 staff records was reviewed, stratified by service to ensure all
services offered by each provider were included. A total of 33 staff records were reviewed for the
three providers offering crisis services. The primary purpose of the Q&T record review is to confirm
Overall Score
FY17: 86.8% FY18: 93.1%FY19: 92.3%
Individual Interview
FY17: 86.0% FY18: 90.2%FY19: 91.1%
Staff Interview
FY17: 89.1% FY18: 97.8%FY19: 97.8%
Observation
FY17: 92.5% FY18: 97.9%FY19: 96.9%
PRR
FY17: 84.1% FY18: 91.1%FY19: 88.9%
FY17: N = 4 FY18: N = 1 FY19: N = 3
Quality Management Annual Report FY 2019
162 | P a g e
relevant staff information is accurate and up to date (e.g., driver’s license, performance evaluations,
background screening) and staff had received all required training specific to services provided. DDSS
reviews are completed to ensure services are provided as specified by DBHDD (Figure 43).
Scores for the DDSS component of the QEPR have remained close to 100 percent each year.
Qualification and Training scores varied little for providers reviewed each year, from 82.6 percent to
84.6 percent, with an average in FY 2019 of 83.2 percent. Providers of crisis services showed higher
Q&T scores each year, with an average in FY 2019 of 96.0 percent for the three providers reviewed.
Figure 43. QEPR Qualifications and Training and DDSS Results by Year
As noted previously, the Q&T review component uses a sample of employees from each provider to
determine compliance with standards. The number of employee records varies based upon the total
number of staff and services rendered. Rates for each indicator measured can be based on the
number of staff records reviewed or the number of providers who have ensured each staff record
reviewed as part of their QEPR has the requirement met. For example, most staff followed DBHDD’s
policy 04-104 to ensure a DBHDD approval letter identifying criminal check screening results are
Providers (Non-Crisis)
Q & T
• FY 2017: 82.6% (N = 947)
• FY 2018: 84.6% (N = 780)
• FY 2019: 83.2% (N = 914)
DDSS
• FY 2017: 99.8% (N = 1,667)
• FY 2018: 99.8% (N = 1,101)
• FY 2019: 99.8% (N = 1,382)
Providers (Crisis)
Q & T
• FY 2017: 91.8% (N = 51)
• FY 2018: 94.5% (N = 5)
• FY 2019: 96.0% (N = 33)
DDSS
• FY 2017: 100% (N = 48)
• FY 2018: 100% (N = 6)
• FY 2019: 100% (N = 37)
Quality Management Annual Report FY 2019
163 | P a g e
available in the personnel record (98.5%); however, only 88.0 percent of providers had the letter
present in all staff records that were reviewed. To help guide possible quality improvement
initiatives, it is important to understand how many providers have the requirements in place for all
staff. In the following table, Table 47, low-scoring Q&T indicators are shown with the percent of staff
records for which the indicator was met and the percent of providers for which all staff records
reviewed for that provider on the indicator were met.
Table 42. FY 2019 Qualifications and Training Low-Scoring Indicators
Required training in the following areas has been completed: % Met
(Staff Records) % Met
( Providers)
Skill development to work with individuals who have co-occurring diagnoses
50.8% 27.8%
Proper body mechanics (lifting/transferring/positioning) needed to assist with activities of daily living within the first 60 days of hire;
59.0% 33.3%
or annually thereafter 52.4% 21.3%
Proper body mechanics (lifting/transferring/positioning) specific to the individual’s care plan prior to working with each individual
63.3% 35.5%
or annually thereafter 68.0% 34.5%
Utilization of Communication Skills; Behavioral Support and Crisis Intervention techniques to de-escalate challenging and unsafe behaviors and/or Nationally benchmarked techniques for safe utilization of emergency interventions of last resort, and the Georgia Crisis Response System (GCRS) to access crisis services
69.7% 47.1%
Utilization of suicide prevention skills 61.6% 36.0%
Specific medications and their side effects 71.7% 49.4%
Fire Safety 87.8% 40.0%
QEPR Scores by Provider Size22
Figure 44 displays the QEPR Overall and Q&T scores by size of the organization and year.23 In FY
2019, Medium sized providers showed the lowest score for both review components, an Overall
score of 84.6 percent and a Q&T score of 80.4 percent, but differences among the provider sizes
were relatively small.
22 Crisis provider scores were excluded because these services are different than typical services. 23 See Figure 40 for details regarding provider size categories.
Quality Management Annual Report FY 2019
164 | P a g e
Figure 44. Overall Score and Qualifications and Training Score
by Provider Size and Year
QEPR Scores by Focused Outcome Area (FOA)
Each year, the Overall score is divided into six FOAs and FOA scores are calculated with the combined
results from the Individual Interview, Staff Interview, Observation (IOSA), and Provider Record
Review.24 Results for QEPRs (except for crisis providers) are shown by FOA and year in Figure 45.
Scores reflecting community integration were lowest among all the FOAs each year.
24 FOAs calculated and presented in the PCR section also included the Support Coordinator Interview and SCRR, which are not part of the QEPR process. Comparisons between the PCR and QPER are not appropriate.
87.2%
84.6%
85.6%
84.4%
82.0%
86.3%
82.7%
83.2%
86.1%
0% 25% 50% 75% 100%
Large
Medium
Small
Overall Score
FY 2017 (N = 96) FY 2018 (N = 99) FY 2019 (N = 97)
84.0%
80.4%
84.5%
85.8%
81.1%
85.3%
86.4%
76.0%
81.5%
0% 25% 50% 75% 100%
Large
Medium
Small
Qualifications and Training
Quality Management Annual Report FY 2019
165 | P a g e
Figure 45. Overall QEPR Scores by FOA and Year
QEPR Scores by FOA and Tool
Results by FOA are presented in Figure 46 for each tool used in calculating the Overall score for the
QEPR process – Individual Interview, Observation, Staff Interview and PRR.25 Findings for the
providers reviewed this year indicate the following:
Provider documentation (PRR) was the lowest-scoring tool across all the FOAs, ranging from a
low of 55.0 percent for measures surrounding Choice to 84.2 percent for Rights. This pattern
is similar to results for providers reviewed in FY 2018.
In the Choice FOA provider documentation was approximately 40 percentage points lower
than results for other tools.
25 See Table 26 for the number of interviews and records completed for each QEPR component.
FY17: 83.7%(N = 96)
FY18: 84.8%(N = 99)
FY19: 86.2%(N = 97)
Whole Health
FY17: 83.2%FY18: 84.9%FY19: 84.8% Safety
FY17: 90.9%FY18: 91.0%FY19: 91.8%
PCP
FY17: 81.6%FY18: 80.4%FY19: 83.3.%Community
FY17: 70.4%FY18: 72.0%FY19: 75.4%
Choice
FY17: 75.9%FY18: 78.2%FY19: 80.2%
Rights
FY17: 91.2%FY18: 92.4%FY19: 93.7%
Quality Management Annual Report FY 2019
166 | P a g e
Figure 46. FY 2019 QEPR Scores by Focus Outcome Area and Tool
Provider Record Review results were lowest for Choice (55.0%), Community Life (66.0%) and Whole
Health (71.5%). The lowest-scoring indicators from each of these FOAs are listed in Figure 47. Over
half the records reviewed this year had not documented, as applicable, various preventative health
reports or the risk and benefits of medications (Whole Health); opportunities for competitive
employment or support to develop social roles, natural supports, or to explore and experience the
community (Community Life); or informed choices for living situations, employment options,
community participation and social interactions (Choice).
75.7%
81.4%
71.5%
93.2%
99.1%
96.3%
94.7%
96.8%
95.5%
88.2%
91.9%
89.3%
0% 25% 50% 75% 100%
PersonCenteredPractices
Safety
Whole Health
Individual Interview (N = 744) Staff Interview (N = 359)
Observation (N = 364) PRR (N = 1,006)
84.2%
55.0%
66.0%
96.7%
98.1%
94.9%
97.6%
94.6%
88.8%
97.9%
95.7%
85.1%
0% 25% 50% 75% 100%
Rights
Choice
CommunityLife
Quality Management Annual Report FY 2019
167 | P a g e
Figure 47. FY 2019 PRR Low-Scoring Indicators - QEPR
QEPR Scores by FOA and Provider Size
Overall scores by FOA and size are presented in Figure 48. Large provider organizations reviewed
during FY 2019, compared to smaller organizations, were somewhat more likely to have indicators
met that measure Person Centered Practices and Community Life. Most other differences across the
FOAs and provider size were relatively small.
• Education provided on risks/benefits of medications (32.1%) or on all prescribed medications (29.1%)
• Preventative test reports are in the record for: Pap (25.6%), Mammogram (41.3%), Bone Density (46.2%), vision (48.6%) or hearing (35.9%) evaluations
Whole Health
• Opportunities to seek employment in competitive settings (24.9%)
• Supporting development of social roles/natural supports reflective of person's interests (29.3%)
• Supporting people to learn about, explore, and experience community (43.3%)
• Participation in community as citizens without disabilities (48.0%)
Community Life
• Choice of living situations (24.9%) or environments (23.3%)
• How the person is making informed choices in general (education, exploration and experiences) (29.8%), about competitive or supported employment options (25.2%) or about community participation and social interactions (39.7%)
Choice
Quality Management Annual Report FY 2019
168 | P a g e
Figure 48. FY 2019 QEPR Provider Scores by FOA and Size
Quality and Technical Assistance Consultation (QTAC) A total of 170 QTACs were completed, of which 137 were at the provider level and 33 at the
individual level. A provider level QTAC focuses on the overall provider practices and processes versus
an individual level QTAC that focuses on services and supports for one specific person. Table 43
shows the number of QTACs by referral source and type for FY 2019. Because more than one referral
type may be used, totals do not sum to 170. Percentages are calculated from the total for each type
of QTAC (137 or 33). Most QTACs were completed at the provider level (80.6%) and the majority
(63.5%) for a QEPR follow-up. These are initiated internally for the QEPR follow up but once onsite,
the provider may request additional technical assistance.
86.3%
92.7%
84.1%
78.7%
91.7%
85.0%
80.7%
90.0%
86.9%
0% 25% 50% 75% 100%
Person CenteredPractices
Safety
Whole Health
Small (n = 47) Medium (n = 22) Large (n = 28)
95.0%
82.1%
77.6%
92.5%
76.0%
73.7%
92.2%
80.5%
71.8%
0% 25% 50% 75% 100%
Rights
Choice
Community Life
Quality Management Annual Report FY 2019
169 | P a g e
Table 43. FY 2019 QTACs by Referral Source and Type
Provider (n = 137) Individual (n = 33)
Referral Source Number Percent Number Percent
Internal 122 89.1% 33 100%
Provider 15 10.9% 0 0.0%
Referral Type:
Quality of Care (PCR/QEPR) 27 19.7% 33 100%
Provider Training 11 8.0% - -
Provider Request 17 12.4% - -
QEPR Follow Up 87 63.5% - -
Table 44 provides a list of detailed reasons for the QTACs completed in FY 2019. More than one
reason can be identified for a QTAC and percentages are based on the total number of QTACs (137 or
33). The most frequently cited reasons at the provider level were related to the QEPR and follow-up
to a Quality of Care Concern. Individual level QTACs were most likely due to a Quality of Care
Concern follow-up, with an additional eight related to a concern about a provider’s safety practices.
Table 44. QTAC Referral Reasons: FY 2019
Provider (n = 137) Individual (n = 33)
Reason Number Percent Number Percent
QEPR Follow Up 88 64.2% - -
Quality of Care Concern follow up 26 19.0% 23 69.7%
Review of person centered documentation 19 13.9% - -
Safety practice concern(s) for the provider 0 0.0% 8 24.2%
Request for 2 day Person Centered Thinking Training 5 3.6% - -
Human rights concern for the person(s) 2 1.5% 2 6.1%
Follow up to previous QTAC 1 0.7% 1 3.0%
Assistance with criminal background checks 1 0.7% - -
Person centered training needed 1 0.7% - -
Total 143 34
Technical assistance (TA) is provided at every QTAC. Approximately 23 to 26 percent of provider level
QTACs offered brainstorming or group discussion. The most common type of technical assistance
offered at the individual level was individual discussion (50.0%) with staff (Figure 49). Discussions
Quality Management Annual Report FY 2019
170 | P a g e
typically cover the problems or barriers facing the provider and tactics tried, with further discussion
on other ways to address the issue. Ideas were presented to improve the quality of service delivery
or documentation. Other types of TA provided included different formats for training, brainstorming,
and resources to help address areas that needed improvement.
Figure 49. FY 2018 QTAC Technical Assistance Provided
Intellectual and Developmental Disability Summary of Findings and Recommendations June 2019 marked the completion of the fourth year of the Collaborative contract. Quality Assessors
completed 100 QEPRs, 484 PCRs, and 170 QTACs. Findings throughout this report were used to
address a wide range of requirements and promising practices.
FY 2019 IDD Accomplishments
Several opportunities were provided throughout the year for the IDD Quality Management team to
work in partnership with DBHDD. This collaboration focused on three key initiatives: training for
10.9%
50.0%
10.9%
17.4%
10.9%
4.5%
18.7%
14.9%
9.0%
25.7%
22.8%
3.4%
0% 20% 40% 60% 80% 100%
Role Play
Resources
Individual Discussion
Group Training
Group Discussion
Brainstorming
1:1Training
Provider(N =137)
Individual(N = 33)
Quality Management Annual Report FY 2019
171 | P a g e
stakeholders, the new IDD Case Management system, as well as tool and process changes for the
upcoming fiscal year.
Training The Collaborative’s Quality Management team and DBHDD’s Quality Department met bi-weekly and
with DBHDD’s Provider Relations and IDD Leadership monthly to discuss review findings from the
current fiscal year and other identified areas of need across the provider network. Based on
discussion of review findings, training sessions on the six Focused Outcome Areas were developed
and presented throughout the year. The following is a list of topics of trainings provided for
stakeholders in the 2019 fiscal year:
Whole Health
How to Identify Potential Health Risks
Monitoring Health Risks and Protocols
Person Centered
CMS Expectations for the Support Planning Process
Creating a Person Centered Culture in Any Setting
Community Networking with the Community to Build Relationships and Natural Supports
Community Life: How to support community relationships and natural supports
Supporting Individuals to Gain Competitive Employment
Rights How to Identify Rights Violations
How to Support Self Advocacy
Choice What is Informed Choice?
Safety Educating Individuals on Abuse, Neglect & Exploitation
Quality Management Annual Report FY 2019
172 | P a g e
Human Services Research Institute (HSRI), a sub-contractor of Qlarant, developed a presentation
from the FY 2016 to 2018 National Core Indicators (NCI) data collected by Collaborative Assessors.26
The focus was on strategies for quality improvement, based on findings from the analysis. The
presentation was provided to DBHDD leadership, with examples of how the NCI data could be
analyzed and used to develop quality improvement initiatives.
IDD Case Management System For two years the Collaborative has assisted in developing a new web-based IDD case management
system (IDD Connects). In preparation for the implementation of the new system, Qlarant was asked
to develop the training to be used to train all external stakeholders who will be utilizing the system.
During this fiscal year, in collaboration with DBHDD, sixteen separate training modules and user
guides were developed for the new IDD Case Management System project, called IDD Connects. The
training modules included PowerPoint presentations and guides, and were developed based upon
different user roles for the new IDD Connects system. Qlarant also trained Beacon staff on the
training modules to help support training efforts across the state. The Collaborative and DBHDD
conducted the first training at the end of the fiscal year for Regional Field Office staff.
Quality Tool and Process Revisions During the year, Qlarant and DBHDD conducted analysis of the tools used for the review processes.
Analysis were conducted for the PRR and included creating a discrimination index, to help determine
which indicators may best predict high vs low scoring providers, and Point-Biserial Correlation to
correlate each indicator with the overall score on the tool.27 Results from these analyses were used
by DBHDD to guide modifications to the tools and processes. On March 29, DBHDD requested the
review processes be modified to meet the following criteria:
Increase reliability and validity of the tools and eliminate redundancy between tools.
26 National Core Indicator website ( https://www.nationalcoreindicators.org/states/GA/) includes specific state reports including Georgia’s survey data since FY 2015. 27 Findings were sent to DBHDD in the excel spreadsheet (Discrimination Analysis revised 021219) and information from this analysis was appended to the Factor Analysis report provided to DBHDD in FY 2018, completed by Qlarant: Discrimination Analysis Intellectual and Developmental Disabilities (IDD) Provider Record Reviews: Quality Enhancement Provider Reviews, July 2016 - June 2018.
Quality Management Annual Report FY 2019
173 | P a g e
Focus on Provider Record Review as the primary source of information while keeping the
voice of the individual through participation in National Core Indicators.
Removal of several review activities (interviews and observations) to decrease administrative
burden on providers.
Apply weights to the eight main components of the new review process when calculating an
overall score for the provider, weighting areas of safety, whole health and person-centered
practices heavier than community life, rights, choice, qualifications and training, and service
guidelines.
Consequently, the Person Centered Review process will be discontinued in FY 2020 as well as several
review processes from the QEPR: the Individual Interview, Staff Interview, Support Coordinator
Interview, and Observations. Workgroups were developed during the last quarter of FY 2019 to
ensure implementation of new tools and processes by July 1, 2019. The workgroups included
representatives of the DBHDD Quality Improvement Office, DBHDD IDD program staff and the
Collaborative and were used to modify record review tools, review procedures, the web-based
application and QEPR reports. Tool revisions included:
Qlarant presented these changes to the provider network during three separate face-to-face
provider meetings around the state and through one webinar.
IDD System Strengths and Recommendations
Findings in this report indicate people receiving IDD services through the DBHDD system were
satisfied with those services, felt safe, and had many health needs met. They (99% or more) had
Tool Updates • Provider Record Review
• Staff Qualifications and Training
Tool Redesign• Support Coordinator Record Review
• DDSS (renamed to Service Guidelines)
New Tool Creation
• Administrative Review
Quality Management Annual Report FY 2019
174 | P a g e
access to supports and services to address whole health needs, access to needed medications, and
were free from all types of abuse. Staff providing services appeared to be aware of how individuals
self-preserve and if individuals served felt safe in their work and living environments (99.6%).
The service delivery system appears to promote an atmosphere to enhance aspects of person-
centered practices. Individuals participated in life’s decisions, with all but five of 484 individuals
indicating they were involved in the development of their ISP. They felt they were offered choices (>
94%) and most individuals interviewed indicated they had privacy (99.6%), did not have any rights
restrictions (99.9%), and were treated with respect (99.6%).
Findings on many of the tools for both the PCR and QEPR showed scores of over 90 percent (Table
45). Similar to previous years, documentation from the provider and Support Coordinator record
reviews was the lowest scoring area, during both the PCR and QEPR.
Table 45. FY 2019 Summary by Tool and Review Type
Tool PCR
(N = 484) QEPR 28 (N = 97)
Crisis29 (N = 3)
IOSA - Individual Interview 93.0% 92.3% 91.1%
IOSA - Observation 98.6% 97.2% 96.9%
IOSA - Staff Interview 94.3% 95.7% 97.8%
Provider Record Review 74.9% 73.2% 88.9%
SC Record Review 76.8% 74.4% NA
SC Interview 79.4% 99.7% NA
Admin Q&T NA 83.2% 96.0%
DDSS 100% 100% 100%
FOA scores from both review types (PCR and QEPR) suggest the service delivery system appears to
do relatively well across the six critical areas, showing scores of 75 percent and higher. Since FY 2017,
Safety and Rights have generated the highest scores - over 90 percent each year - suggesting
providers have systems and practices in place to help individuals be safe in their environments and
28 QEPRs completed for the one Support Coordinator agency included 59 SCRRs and 11 Support Coordinator Interviews. 29 Crisis provider QEPRs do not include the SCRR or SCI.
Quality Management Annual Report FY 2019
175 | P a g e
exercise and learn about their rights. Individuals receiving services have indicated they do feel safe
(92.4%) and their rights are upheld (98.4%).
Opportunities for Improvement
While overall FOA scores remained relatively high, “drill down” to the standards/indicators
measuring each FOA provides insight into opportunities for developing quality improvement
initiatives or training programs.
Whole Health
While some improvements have been seen in FY 2019, knowledge and education surrounding
medications have remained among the lower scoring areas in Whole Health. Close to half of
individuals interviewed did not understand the side effects of the medications they were taking, and
approximately 31 percent did not know what medications they were taking or why. In addition,
providers were significantly less likely (compared to FY 2018) to document how they offer education
to individuals receiving services on their prescribed medications (27.9%), including their risks and
side effects (37.5%), and were significantly less likely to be able to describe how individuals they
serve were supported to learn about medications (57.3%) or their side effects (68.8%). At the same
time, approximately half of providers had at least one staff member who had not completed training
within required timeframes, specific to medications and their side effects.
Documentation demonstrated Individuals receiving services were also significantly less likely to be
able to maintain personal health, including any necessary supports (49.3%). A lack of knowledge and
education surrounding medications, in combination with an inability to understand how to maintain
health, could lead to serious consequences.
Recommendation XII: The DBHDD Provider Manual for Community Developmental Disability
Providers requires education be provided to individuals on the risks and benefits of anti-psychotic
medications and all other types of prescribed medication and this must be documented in the
clinical record. Training in this area was conducted by the Collaborative in June of 2018 and it is
recommended the training be conducted again and specifically target nursing staff of providers.
Quality Management Annual Report FY 2019
176 | P a g e
Recommendation XIII: It is recommended that training sessions for providers include techniques to
effectively offer health education for individuals with all types of communication styles, particularly
on how to maintain their own health. The sessions should include ways to determine if individuals
receiving services understand, to the best level possible, how to maintain personal health and
understand different aspects of the medications they are taking.
Recommendation XIV: It is recommended the Collaborative work with the DBHDD Office of Health
and Wellness to identify initiatives around independence in maintaining health and staff training
around individuals’ medications.
Safety
Without effective training, it is difficult for individuals being served to recognize abusive situations
and without an understanding of how to report them to someone, the abusive activities could go
unreported and cause harm. Information from interviews indicates approximately 35 percent of
individuals receiving services were not aware of what constitutes exploitation or neglect. In addition,
approximately 54 percent of providers had not offered education on how to self-preserve or develop
effective resiliency skills according to the person’s learning style. Over 43 percent of PCRs included a
recommendation related to offering education, according to the person's learning style, on different
types of exploitation, on how to keep safe, and possible actions to take in these situations. In
addition, only 42 percent of PCRs included a recommendation to help individuals receiving services
learn how to manage safety situations by conducting “what if” scenarios.
Support Coordinators are an important advocate for individuals receiving services. If there is a crisis
or behavioral plan in place, it is essential for the coordinator to evaluate the effectiveness of the
plan. If a crisis occurs, it is critical to coordinate supports to prevent a recurrence. Interviews with
Support Coordinators indicated these activities were often not occurring, 44.7 percent and 55.9
percent, respectively.
Quality Management Annual Report FY 2019
177 | P a g e
Recommendation XV: The Collaborative and DBHDD could consider developing and providing
training specific to how Support Coordinators ensure systems are in place to evaluate crisis and
behavioral plans, as well as how to evaluate those plans to ensure they are effectively preventing
crisis or if there is a need for additional supports.
Person Centered Practices
Person centered practices should be an essential part of each provider’s service delivery system. To
the extent possible, the individual receiving services should be at the center of all decisions, plans,
and goals. It is important for providers to understand who each individual is and what he or she may
want, hope for, and dream of. What is actually important to anyone receiving services should be
known and incorporated into supports as much as possible. Unfortunately, approximately 32 percent
of ISPs scored in the lowest two rating categories, Needs Improvement or Emerging, demonstrating
they did not ensure all the goals and objectives were person centered or the supports to help with
changes in a person’s life were secured. In over a quarter of the ISPs reviewed, the Communication
Chart did not provide personal information that could be used to best support the individual
receiving services.
As noted, to offer person centered services it is essential to understand what someone wants, strives
for, is particularly good at doing, or finds important. As key advocates for people receiving services,
Support Coordinators are often on the “front line” helping ensure needs and desires are known and
understood. Unfortunately, fewer than half of Support Coordinators interviewed demonstrated a
knowledge of talents or strengths of people they served, or regularly reviewed goals with individuals
receiving services. In addition, providers and Support Coordinators both demonstrated a significant
decrease from FY 2018, documenting how individuals respond to services or supports. A lack of
knowledge about individualized responses to supports will also impede the ability of providing truly
person-centered services.
Recommendation XVI: DBHDD and the Collaborative will be providing training on the changes to the
ISP being implemented in FY 2020. It is recommended additional training be conducted for
providers, specifically on the goals section of the new ISP and how they can help support individuals
Quality Management Annual Report FY 2019
178 | P a g e
in developing goals and action plans that are person centered and updated based upon the review of
progress with the person.
Recommendation XVII: One of the key components to implementing a person-centered approach to
planning is to ensure individuals receiving services have the opportunity to review their goals and
make changes as deemed necessary. With implementation of the new IDD Connects system (new
electronic record and ISP) in FY 2020, it will be possible to monitor how and when the person’s ISP
goals are reviewed and changed as needed. A report could be generated from the system to track
how this process is being implemented by region. Findings from this report could guide training the
Collaborative provides, specific to each region as needed.
Recommendation XVIII: FY 2020 will be the first year all ISPs have been transitioned to the “new”
format. Some of the low scoring areas demonstrated in this report may be partially due to a learning
curve for Support Coordinators, individuals and families based upon the new ISP format
implemented in November 2018. If there is no improvement shown in the FY 2020 annual report,
further analysis conducted by the Collaborative is recommended to help determine where quality
appears to be lacking, particularly in ensuring person centered planning is used, ensuring the person
truly has a voice during planning meetings and community engagement is optimized.
Community Life
According to CMS expectations, individuals with intellectual and developmental disabilities should be
able to participate in their communities in the same manner as individuals who do not have a
disability. Community Life remains the lowest scoring area in the PCR and for providers reviewed
through the QEPR. Developing new social roles, such as being a member of a swim team, dance
group or book club, and maintaining those valued roles are instrumental ways to increase
community involvement and integration. Information from individual interviews suggests
approximately 30 percent to 40 percent of individuals receiving services had not been given the
opportunity to learn about or develop new social roles in the community, or had support to maintain
valued social roles.
Quality Management Annual Report FY 2019
179 | P a g e
When we first meet someone often the first question we ask is, “what do you do” or “where do you
work?” Many people not only identify themselves and others through their employment but also
expand social connections through contacts at work. Individuals receiving IDD services can also
expand social connections through integrated community employment, enhancing natural or paid
supports. On average, over a three-year time period (FY 2017 – FY 2019) approximately 10 percent of
individuals received Supported Employment services. Data from PCR reviews indicates provider and
Support Coordinator documentation often does not show support for individuals to seek
employment in integrated settings (27.9% and 44.1% respectively).
The ISP should be written to help ensure integration into the community in various settings, as
desired. However, only 8.4 percent of ISPs were written to reflect full participation in the community.
The Likert Scale indicators demonstrated close to 47 percent of ISPs scored in the lower two rating
categories, Emerging or Needs Improvement, showing summaries often did not reflect the person’s
community life. In addition, close to 42 percent of the relationship maps scored in the lowest two
ratings, lacking demonstration of how the person is connected to the community.
Recommendation XIX: DBHDD began contracting with an agency called Wise as an initiative to
increase the number of individuals with IDD who have integrated employment. It is recommended
DBHDD share progress on this initiative through its monthly newsletter and ensure the actions and
plans being made are shared with all stakeholders. This move toward increased awareness could
hopefully lead to increased support from the provider network.
Recommendation XX: There is a new emphasis on employment in the new ISP in IDD Connects, with
additional questions required to be answered. It is recommended the Collaborative ensure data
related to this section are analyzed and reported to DBHDD during regular quality management
meetings. This will help determine if there is improvement in the number of people who are
interested in work who also have goals and action plans in place to address that interest.
Choice
Informed choice is the cornerstone of helping anyone understand and achieve meaningful goals and
direct supports and services. Information from documentation indicates between approximately 50
Quality Management Annual Report FY 2019
180 | P a g e
percent and 70 percent of provider and Support Coordinator records lacked evidence informed
choice was provided for competitive or supported employment, living situations, environments,
community participation/social interactions, or for opportunities for education, exploration or
experiences in the community. Close to half of staff interviewed had not provided options for
competitive employment. Individuals interviewed often indicated they were involved in
employment decisions (85.6%); however, provider and Support Coordinator documentation appears
to be what was primarily affecting the low Choice FOA score.
Recommendation XXI: It is recommended the Collaborative’s Quality Management team develop a
training to address how to document choices offered and how providers can support individuals to
make informed decisions. This training would target support staff and Support Coordinators.
Rights
The proper use of restrictive interventions is strictly monitored and enforced by DBHDD. Individuals
interviewed during the review processes did not indicate any violations regarding the improper use
of any type of restrictive intervention or any unauthorized restrictions in their home, community,
work, or day program. They generally feel their rights are upheld. However, records maintained by
providers and Support Coordinators indicated there may be some issues regarding proper
documentation of addressing rights, including: offering ongoing rights education; signed informed
consent forms for medications; and a signed form of notification about rights and responsibilities.
All of these were scored met on fewer than half the records reviewed.
Recommendation XXII: DBHDD could consider requiring Support Coordinators to provide education
to individuals and families on the importance of giving and receiving copies of signed consent forms
for psychotropic medications. Through this, Support Coordinators could ensure consent forms are
being completed and subsequently document this in their own support notes. This should include
education about each individual’s rights and responsibilities.
Recommendation XXIII: DBHDD may want to consider requiring the informed medication consent
document/form be uploaded into the IDD Connects system. If the person receives psychotropic
medication(s) and the form is missing, the system could trigger a “Task” be sent to the Support
Quality Management Annual Report FY 2019
181 | P a g e
Coordinator which would be monitored by the Support Coordination agency and DBHDD for
completion.
Additional Recommendations
Provider Documentation
Throughout the findings in this report, documentation by providers and Support Coordinators has
shown consistently lower scores when compared to other information gathering techniques
(interviews or direct observations). This pattern has persisted for several years. For example,
individuals receiving services:
Indicated they had seen a doctor for an annual physical but records often did not include
corresponding documentation
Were free from abuse, neglect and exploitation (ANE) but many providers had not documented
how they identify or address ANE
Indicated they had choice across most areas of their lives but records did not include
corresponding documentation
Seemed to understand their own talents, strengths and goals but most providers did not
document those talents/strengths or how they review progress toward achieving goals with each
individual
Recommendation XXIV: The Collaborative’s Quality Management leadership could develop a
training to specifically address how Support Coordinators can improve support notes and
documentation specific to areas identified throughout the intellectual and developmental disability
section of this report.
Recommendation XXV: Documentation training sessions were developed several years ago and
presented across the state. Over time, however, new providers have joined the DBHDD service
delivery team, new staff have joined existing providers, and new requirements for providers may
have impacted the quality of provider documentation. The Collaborative’s Quality Management
team could identify key areas from this report where documentation is clearly falling behind
Quality Management Annual Report FY 2019
182 | P a g e
interview findings and target these specific to training for providers and training for Support
Coordinators.
Quality Enhancement Provider Reviews: Qualifications and Training
Since FY 2017, the average Qualifications and Training score has remained between approximately
83 percent and 84 percent. Some analysis at the indicator level shows that fewer than half of
providers reviewed this year ensured all staff (for the staff records reviewed) had required training
for fire safety, specific medications and their side effects, utilization of suicide prevention skills, and
skill development to work with individuals who have co-occurring diagnoses. In addition, fewer than
half of the providers had received training on proper body mechanics for lifting, transferring or
positioning individuals. Knowledge from these sessions can help ensure the health and safety of
individuals receiving services, and also help protect staff from potential injury.
Recommendation XXVI: It is recommended the Collaborative continue to monitor these results in FY
2020 for future training needs. The Collaborative should continue to provide technical assistance
during QEPRs related to staff training.
Quality Management Annual Report FY 2019
183 | P a g e
Section 5: Feedback Survey Results
Following completion of quality reviews for both behavioral health and intellectual and
developmental disabilities, providers are offered the opportunity to complete a feedback survey. For
intellectual and developmental disability services, individuals who participate in the interview are
also offered the opportunity to provide feedback about the process. Surveys are optional and may
be completed by the same agency more than once in a fiscal year. In addition, it is not known who
completes the survey, as individuals and staff are not required to submit their name or provider
agency. A five-point Likert scale is used: strongly agree, somewhat agree, neither agree nor disagree,
somewhat disagree, or strongly disagree.
Table 46 provides information from providers for data entered into the system during FY 2019.
Response rates were generally low, but findings were, again, overwhelmingly positive across all the
review types and from both providers and individuals when compared to previous years. The percent
was calculated as follows: (Strongly Agree + Somewhat Agree) / Total Responses. Results of
indicators for each of the two systems are identified below for annual comparison.
Table 46. FY 2019 Collaborative Provider Feedback Surveys Percent: Strongly Agree + Somewhat Agree/Total Responses
Survey Indicators Result
(N = 221)
Overall, you are satisfied with the review/consultation process. 97%
The Collaborative staff interacted with you and your staff in a professional manner. 98%
The Collaborative staff interacted with the individuals you support in a professional manner.
98%
The Collaborative staff answered your questions and concerns clearly and consistent with DBHDD manual.
98%
The Collaborative staff facilitated an environment which was collaborative and positive. 100%
You would contact the Collaborative staff for technical assistance, training, and resource support, if needed.
99%
The process provided constructive feedback. 98%
Quality Management Annual Report FY 2019
184 | P a g e
Table 46. FY 2019 Collaborative Provider Feedback Surveys Percent: Strongly Agree + Somewhat Agree/Total Responses
Survey Indicators Result
(N = 221)
The process helped identify the strengths of your supports and services. 98%
The feedback you received will help provide supports and services that meet the desired outcomes of the individuals you support.
98%
The recommendations generated from this process can be used to make a positive contribution to the individuals served.
99%
The recommendations generated from this process can be used to make a positive contribution to your organization overall.
99%
The feedback provided will assist your organization with making quality improvements to systems and practices.
99%
Overall Survey Result 98%
Only seven responses were received specific to the individual survey completed following an IDD
quality review, all responses resulted in “strongly agree,” a 100 percent score. Comments from
individuals include:
Overall, I am pleased with how the interview went.
The purpose of the interview was explained to me.
I was treated with respect.
The person who interviewed me seemed interested in what I said.
The person who interviewed me was pleasant.
My questions were answered.
The length of the interview was good.
Feedback obtained from the satisfaction surveys provides a qualitative approach to individual and
staff perceptions related to review processes and yielded positive results in FY 2019. Quality
Management will continue to use feedback from the surveys to review processes and provide
professional, effective, and constructive approaches to ensure providers are equipped with the
necessary tools and opportunities they need to best support individuals.
Quality Management Annual Report FY 2019
185 | P a g e
Section 6: Conclusion
FY 2019 marked the fourth complete year of the Collaborative Quality Management review process.
During FY 2019, Quality Management completed 3,116 reviews for behavioral health and intellectual
and developmental disability providers. The BHQR Overall score for FY 2019 was 90 percent, a two-
point increase from FY 2018. Additionally, all four BHQR categories increased from the previous year
with Assessment and Planning and Focused Outcome Areas continuing an upward trend since
contract inception.
While there were demonstrated successes noted in all of the different BHQR categories,
opportunities for growth similar to past years continued, especially related to addressing all assessed
needs, co-occurring issues being evaluated and addressed, as well as appropriate discharge planning
criteria documentation. Furthermore, the ACT Service Guidelines score remained steady at 84
percent as a subset of the BHQR reviews with ACT reviews resulting in an average Overall score of 89
percent. Several declines in ACT indicators specific to Billing (FY18 91%; FY19 88%) counteracted the
improvement of the Assessment and Planning score (FY18 87%; FY19 90%), rendering an ACT Overall
score average equivocal to the previous year’s result of 89 percent.
Additionally, FY 2019 marked the third year reassessments were completed for providers who fell
below the revised threshold criteria (90%). The threshold had been raised because of high statewide
averages in FY 2018 and progress at the specific provider level. Twenty-eight (22%) providers were
reviewed for a second BHQR during the year compared to 22 (15%) in the year prior. Only three of
the 19 CSU’s required a reassessment in FY 2019. In review of BHQR scores for these specific
providers at the time of their first FY 2019 review compared to their second FY 2019 review, the
Overall score fell slightly from 89 percent to 88 percent. However, all category scores of the
reassessed BHQRs increased slightly except for Billing, which declined from the time of initial FY
2019 review to the subsequent FY 2019 second review.
While the reassessment of providers is expected to lead to elevated results from year-to-year as a
direct result of ongoing feedback and technical assistance provided, some providers continued to
decline in scoring from first to subsequent review. This is evident in the case of CSUQR
Quality Management Annual Report FY 2019
186 | P a g e
reassessments in which two of the three CSUs fell to less than 80 percent by year-end. Moreover, 85
percent of reassessed BHQR providers experienced an increase in the Overall score from initial to
subsequent FY 2018 review, but only 36 percent of providers increased the Overall score during FY
2019. As FY 2019 resulted in fewer providers obtaining improve scores from initial to subsequent
review, it is recommended that the threshold for FY 2020 remain unchanged from FY 2019.
IDD Quality Assessors completed 100 QEPRs, 484 PCRs, and 170 QTACs. On average, findings in this
report reflected the greatest strengths in the areas of rights and safety, and people appeared to be
satisfied with those services. PCR and QEPR interviews with individuals and staff showed scores of
over 90 percent; however, like previous years, provider and Support Coordinator documentation
reflected through record reviews remained on average 76 percent or less. Interviews with the
Support Coordinator showed a decrease from 83.5 percent in FY 2018 to 79.4 percent in FY 2019.
However, a revised Support Coordinator Interview tool was implemented July 1, 2018, and direct
comparisons are not appropriate.
Average statewide PCR FOA scores remained fairly consistent since FY 2017, with the greatest gains
shown in Community Life, up to 77.3 percent. Record review results (PRR and SCRR) were at least 10
points or more lower than other tools for Community Life, Choice and Person Centered Practices and
PRR and SC Interview results were 10 points or more lower than other tools measuring Safety.
Results for FOA by region, residence and funding source produced the following findings:
Individuals living in Region 6 were significantly less likely to be participating their Community
Life as desired (p< .001), compared to their counterparts living in other regions.30
Individuals living in Region 2 were significantly less likely to have Choice in their lives
(p< .001), compared to their counterparts living in other regions.
Individuals receiving services and living with a parent scored significantly lower in Whole
Health than their counterparts in any other residential setting (p < .001).
30 It should be noted the difference of proportions test was based on the number of indicators scored; therefore, the “n” sizes were relatively large, increasing the likelihood of statistical significance.
Quality Management Annual Report FY 2019
187 | P a g e
Receiving services while living in a group home, showed a significantly lower likelihood of
being integrated into the community as desired when compared to other residential settings
(p<.001).
Results for the COMP waiver indicated individuals were more likely to have Whole Health
indicators met compared to NOW or SFS funding sources (p < .01).
The overall QEPR scores were also fairly consistent across the same timeframe, 83.7 percent to 86.2
percent. The Qualifications and Training component of the QEPR has reflected scores of 83 percent
to 84 percent since FY 2017.
To reduce administrative burden to providers, Quality Management conducted joint BHQR, CSUQR
and QEPRs as often as possible. Fourteen joint reviews were completed during FY 2019, four more
than the previous year, with 11 conducted at Community Service Board locations. By combining
BHQR and IDD activities, the Collaborative has promoted additional education for the network and
also among Assessors, further strengthening expertise and collaboration. Through the joint
discussions offered at exit conferences, that include both BHQR and IDD providers and Assessors,
joint reviews enhanced valuable integration of information and feedback and therefore this
approach will continue throughout FY 2020.
Provider trainings in FY 2020 will continue to occur both in person as well as via webinar through the
formalized Quality Training Program based on collaboration and partnership with DBHDD. Quality
Management shall use findings from the behavioral health, CSU, and intellectual and developmental
disability reviews to recommend topics for trainings. Such trainings will be prioritized based on
identified system needs.
Throughout the revisions to the IDD tools, the Collaborative explored alignment of both the BH and
IDD review processes. The revision to the FY 2020 IDD tools will further align indicators between BH
and IDD. Previously, similarity between the two review types (BHQR/IDD) included only Focused
Outcome Areas categories, but item-level indicators remained different between the populations.
An addition of an IDD Service Guidelines section is anticipated for FY 2020. The Service Guideline
Quality Management Annual Report FY 2019
188 | P a g e
section of the IDD tool will extrapolate data similar to BHQRs/CSUQRs whereby a set of
indicators/standards are to be measured only once per service.
Annual analyses and recommendations discussed throughout this report drive statewide BH and IDD
quality initiatives for the department’s internal and external review processes. Analysis also drives
efforts specific to targeted provider training and technical assistance/support to those who serve and
treat the individuals in need. Therefore, Quality Management will continue to collaborate and work
diligently with both its client, DBHDD, as well as the Georgia provider network to continue facilitating
the delivery of whole-health, person-centered, and quality supports and services to individuals and
their families throughout the state.
Quality Management Annual Report FY 2019
189 | P a g e
Appendix A: Abbreviations and Acronyms
Acronym Definition ACT Assertive Community Treatment
AD Addictive Diseases
ADSS Addictive Disease Support Services
AIMS Abnormal Involuntary Movement Scale
ANE Abuse, neglect, and exploitation
BH Behavioral Health
BHQR Behavioral Health Quality Review
C&A Child and Adolescent
CIS Consumer Information System
CMS Centers for Medicaid and Medicare Services
CPRP Certified Peer Recovery Specialist
CPS Certified Peer Specialist
CPS-WH Certified Peer Specialist – Whole Health
C-SSRS Columbia Suicide Severity Rating Scale
CST Community Support Team
CSU Crisis Stabilization Unit
CSUQR Crisis Stabilization Unit Quality Review
DBHDD Department of Behavioral Health and Developmental Disabilities
DDSS Developmental Disability Service Specific
FOA Focused Outcome Area(s)
FY Fiscal Year
GLC Georgia Learning Community
HSRI Human Services Research Institute
II Individual Interview
IDD Intellectual and Developmental Disability
IFI Intensive Family Intervention
IOP Intensive Outpatient
IOSA Individual Observation Staff Assessment
IRR Individual Record Review
IRP Individual Recovery / Resiliency Plan
ISP QA Individual Service Plan Quality Assurance Checklist
ISP Individual Service Plan
KPI Key Performance Indicator
MAR Medication Administration Record
MAT Medication-Assisted Treatment
MH Mental Health
Quality Management Annual Report FY 2019
190 | P a g e
N “n” Number in sample (number in subset of sample)
NA Not Applicable
NCI National Core Indicators
NCP Nursing Care Plan
NIOP Non-Intensive Outpatient
NR Not Reviewed
ODESA Online Data Entry Survey Application
OBS On-site Observations
ODS Office of Deaf Services
PCP Person Centered Practices
PCR Person Centered Review
PHI Protected Health Information
PRR Provider Record Review
QEPR Quality Enhancement Provider Review
QM Quality Management
Q&T Qualifications and Training
QTAC Quality Technical Assistance Consultation
R “r” Number of reviews
RN Registered Nurse
SC Support Coordinator
SCI Support Coordinator Interview
SCRR Support Coordinator Record Review
SFS State Funded Services
SI Staff Interview
SIS Supports Intensity Scale
SU Substance Use
TA Technical Assistance
Quality Management Annual Report FY 2019
191 | P a g e
Appendix B: Score Distributions
Graphical distributions for each category show the percent of providers who scored within the specified range of scores,
as indicated on the horizontal/x-axis. With each distribution, the median, mean, and mode may be provided. The median
represents the score that falls in the middle of distribution (50th percentile) and the mean represents the average score.
Standard deviation refers to the amount of variation or dispersion there is in a distribution of scores, or how much scores
tend to spread-out from the mean. Standard deviations are sometimes used to determine significant differences between
scores within a distribution; however, this application requires a distribution to be normally distributed (similar to a bell
curve). Most of the distributions presented in this report do not meet the requirements of a normal distribution, meaning
they do not look like a normal bell curve; therefore, standard deviations have been excluded from the report at this time.
Table of Contents (Links available by clicking on Figure Name)
Behavioral Health Quality Review Distributions
BHQR Overall Score Distribution
BHQR Tier 1 Overall Score Distribution
BHQR Tier 2 Overall Score Distribution
BHQR Tier 2+ Overall Score Distribution
BHQR Tier 3 Overall Score Distribution
BHQR Billing Score Distribution
BHQR Assessment & Treatment Score Planning Distribution
BHQR Service Guidelines Score Distribution
BHQR Focused Outcome Areas Score Distribution
Assertive Community Treatment (ACT) Quality Review Distributions
ACT Overall Score Distribution
ACT Billing Score Distribution
ACT Assessment & Planning Distribution
ACT Service Guidelines Score Distribution
ACT Focused Outcome Areas Score Distribution
Quality Management Annual Report FY 2019
192 | P a g e
CSU Quality Review Distributions
CSU Overall Score Distribution
CSU Service Guidelines Score Distribution
CSU Focused Outcome Areas Score Distribution
CSU Individual Record Review Score Distribution
Reassessment BHQR Distributions
BHQR Overall Score Distribution of Reassessed Providers
BHQR Billing Score Distribution of Reassessed Providers
BHQR Assessment & Treatment Planning Distribution of Reassessed Providers
BHQR Service Guidelines Score Distribution of Reassessed Providers
BHQR Focused Outcome Areas Score Distribution of Reassessed Providers
PCR Distributions: Scores by Tool and Year
PCR Individual Interview Scores by Year
PCR Observation Scores by Year
PCR Staff Interview Scores by Year
PCR Provider Record Review Scores by Year
PCR Support Coordinator Review Scores by Year
PCR Support Coordinator Interview Scores by Year
QEPR Distributions: Scores by Tool and Year
FY 2019 Overall QEPR Score
FY 2019 QEPR Individual Interview Scores
FY 2019 QEPR Observation Scores
FY 2019 QEPR Staff Interview Scores
FY 2019 QEPR Provider Record Review Scores
FY 2019 QEPR Qualifications and Training Scores
FY 2019 QEPR Service Discrepancy Scores
FY 2019 QEPR Overall Scores by Provider Size
FY 2019 QEPR Qualifications and Training Scores by Provider Size
Quality Management Annual Report FY 2019
193 | P a g e
BHQR Overall Score Distribution by Year
BHQR Tier 1 Overall Score Distribution by Year
0% 1% 2%5%
19%
49%
25%
1% 2% 3%6%
42%47%
1% 1%3% 3%
32%
61%
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY 2017 (N = 167; Mean = 84%; Median = 86%)
FY 2018 (N = 156; Mean = 88%; Median = 90%)
FY 2019 (N = 157; Mean = 90%; Median = 92%)
21%
58%
25%
3%
48% 48%
4%
59%
37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY 2017 (N = 28; Mean = 85%; Median = 85%)
FY 2018 (N = 29; Mean = 88%; Median = 90%)
FY 2019 (N = 27; Mean = 88%; Median = 89%)
Quality Management Annual Report FY 2019
194 | P a g e
BHQR Tier 2 Overall Score Distribution by Year
BHQR Tier 2+ Overall Score Distribution by Year
2%5%
19%
50%
24%
1% 2%6%
41%
49%
1% 1% 4%
25%
70%
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY 2017 (N = 115; Mean = 85%; Median = 86%)
FY 2018 (N = 93; Mean = 89%; Median = 90%)
FY 2019 (N = 102; Mean = 91%; Median = 93%)
83%
17%
83%
17%
33%
67%
0%
20%
40%
60%
80%
100%
FY 2017 (N = 6; Mean = 86%; Median = 85%)
FY 2018 (N = 6; Mean = 87%; Median = 86%)
FY 2019 (N = 6; Mean = 92%; Median = 92%)
Quality Management Annual Report FY 2019
195 | P a g e
BHQR Tier 3 Overall Score Distribution by Year
BHQR Billing Score Distribution by Year
6% 6%11% 11% 11%
56%
7% 7% 7% 7%
29%
43%
9%13%
30%
43%
0%
20%
40%
60%
80%
100%
FY 2017 (N = 18; Mean = 83%; Median = 91%)
FY 2018 (N = 28; Mean = 83%; Median = 89%)
FY 2019 (N = 22; Mean = 84%; Median = 91%)
1% 2% 1% 1% 2% 4% 5%8%
28%
48%
1% 2% 1% 1% 1% 1% 3%
11%
26%
53%
1% 1% 3% 1% 2% 3%8%
29%
54%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 167; Mean = 84%; Median = 90%)
FY 2018 (N = 156; Mean = 85%; Median = 92%)
FY 2019 (N = 157; Mean = 86%; Median = 92%)
Quality Management Annual Report FY 2019
196 | P a g e
BHQR Assessment & Treatment Planning Score Distribution by Year
BHQR Compliance with Service Guidelines Score Distribution by Year
1% 1%5%
23%
31% 31%
9%
1% 1% 2%
7%
21%
40%
29%
1% 1% 1%4%
8%
38%
47%
0
0.1
0.2
0.3
0.4
0.5
FY 2017 (N = 167; Mean = 77%; Median = 78%)
FY 2018 (N = 156; Mean = 84%; Median = 86%)
FY 2019 (N = 157; Mean = 88%; Median = 90%)
1% 1%7% 7%
31%
53%
1% 1%4% 3%
29%
62%
1% 1% 1% 1% 3% 4%
27%
64%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
FY 2017 (N = 167; Mean = 88%; Median = 91%)
FY 2018 (N = 156; Mean = 90%; Median = 93%)
FY 2019 (N = 157; Mean = 90%; Median = 92%)
Quality Management Annual Report FY 2019
197 | P a g e
BHQR Focused Outcome Areas Score Distribution by Year
ACT Overall Score Distribution by Year
2%8%
41%50%
2% 3%
28%
67%
1%5%
11%
83%
0%
20%
40%
60%
80%
100%
FY 2017 (N = 167; Mean = 89%; Median = 90%)
FY 2018 (N = 156; Mean = 92%; Median = 93%)
FY 2019 (N = 157; Mean = 94%; Median = 95%)
26% 25%
50%
5%
48% 48%
10%
40%
50%
0%
10%
20%
30%
40%
50%
60%
FY 2017 (N = 19; Mean = 87%; Median = 91%)
FY 2018 (N = 21; Mean = 89%; Median = 90%)
FY 2019 (N = 20; Mean = 89%; Median = 90%)
Quality Management Annual Report FY 2019
198 | P a g e
ACT Billing Score Distribution by Year
ACT Assessment & Planning Score Distribution by Year
5%
15%20%
60%
5% 10%
19%
67%
5%
20%25%
50%
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY 2017 (N = 19; Mean = 90%; Median = 96%)FY 2018 (N = 21; Mean = 91%; Median = 95%)FY 2019 (N = 20; Mean = 88%; Median = 90%)
20%
35%40%
5%
19%
43%38%
10%
0%
30%
60%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 19; Mean = 80%; Median = 80%)
FY 2018 (N = 21; Mean = 87%; Median = 87%)
FY 2019 (N = 20; Mean = 90%; Median = 92%)
Quality Management Annual Report FY 2019
199 | P a g e
ACT Service Guidelines
Score Distribution by Year
ACT FOAs Score Distribution by Year
5%10%
50%
35%33%38%
29%
10%15%
55%
20%
0%
10%
20%
30%
40%
50%
60%
FY 2017 (N = 19; Mean = 88%; Median = 89%)
FY 2018 (N = 21; Mean = 84%; Median = 86%)
FY 2019 (N = 20; Mean = 84%; Median = 87%)
10%
35%
55%
19%
81%
30%
70%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
FY 2017 (N = 19; Mean = 90%; Median = 91%)
FY 2018 (N = 21; Mean = 93%; Median = 95%)
FY 2019 (N = 20; Mean = 93%; Median = 93%)
Quality Management Annual Report FY 2019
200 | P a g e
CSU Overall Score Distribution by Year
CSU Compliance with Service Guidelines Score Distribution by Year
4%
17%
43%
35%
5% 5%
48%43%
18%
55%
27%
0%
10%
20%
30%
40%
50%
60%
FY 2017 (N = 23; Mean = 86%; Median = 88%)
FY 2018 (N = 21; Mean = 88%; Median = 90%)
FY 2019 (N = 22; Mean = 87%; Median = 87%)
4%
17%
43%
35%
5% 5%
48%
43%
14%
23%
18%
45%
0%
10%
20%
30%
40%
50%
60%
FY 2017 (N = 23; Mean = 87%; Median = 91%)
FY 2018 (N = 21; Mean = 91%; Median = 100%)
FY 2019 (N = 22; Mean = 89%; Median = 90%)
Quality Management Annual Report FY 2019
201 | P a g e
CSU Focused Outcome Areas Overall Distribution by Year
CSU Individual Record Review Score Distribution by Year
9%
35%
57%
48%52%
23%
77%
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY 2017 (N = 23; Mean = 91%; Median = 92%)FY 2018 (N = 21; Mean = 91%; Median = 91%)FY 2019 (N = 22; Mean = 93%; Median = 95%)
4% 4%
26%
65%
5% 5%
29%33%
29%
18%
27%
50%
5%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 23; Mean = 82%; Median = 82%)
FY 2018 (N = 21; Mean = 83%; Median = 83%)
FY 2019 (N = 22; Mean = 79%; Median = 83%)
Quality Management Annual Report FY 2019
202 | P a g e
FY 2019 BHQR Overall Score Distribution of Reassessed Providers
FY 2019 BHQR Billing Score Distribution of Reassessed Providers
4% 4%
43%50%
7% 4%
39%
50%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1st BHQR (N = 28; Mean = 89%; Median = 90%)
2nd BHQR (N = 28; Mean = 88%; Median = 91%)
4% 7%
50%
32%
11%7%
11%
43%
29%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1st BHQR (N = 28; Mean = 87%; Median 89%)
2nd BHQR (N = 28; Mean = 80%; Median = 85%)
Quality Management Annual Report FY 2019
203 | P a g e
FY 2019 BHQR Assessment & Planning Score Distribution of Reassessed Providers
FY 2019 BHQR Service Guidelines Score Distribution of Reassessed Providers
4% 7% 7% 11%
32%39%
7% 7%
39%46%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1st BHQR (N = 28; Mean = 84%; Median = 89%)
2nd BHQR (N = 28; Mean = 87%; Median = 90%)
4% 4% 4%
32%
57%
7%4%
21%
68%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1st BHQR (N = 28; Mean = 89%; Median = 93%)
2nd BHQR (N = 28; Mean = 91%; Median = 92%)
Quality Management Annual Report FY 2019
204 | P a g e
FY 2019 BHQR FOAs Score Distribution of Reassessed Providers
PCR Individual Interview Scores by Year
4%
18%
79%
4%
18%
79%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1st BHQR (N = 28; Mean = 93%; Median = 95%)
2nd BHQR (N = 28; Mean = 94%; Median = 95%)
1%6%
22%
66%
5%2% 5%
20%
62%
11%
1% 5%
17%
66%
11%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 481; Mean = 91.9%; Median = 93.7%)
FY 2018 (N = 484; Mean = 92.2%; Median = 95.1%)
FY 2019 (N = 484; Mean = 93.0%; Median = 94.9%)
Quality Management Annual Report FY 2019
205 | P a g e
PCR Observation Scores by Year
PCR Staff Interview Scores by Year
4% 6%
52%
38%
<1% 1%
42%
57%
3%
41%
57%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 181; Mean =96.8%; Median = 98.7%)
FY 2018 (N = 276; Mean =98.2%; Median = 100%)
FY 2019 (N = 274; Mean = 98.6%; Median = 100.0%)
<1% 1% 2%
11%
69%
16%
3%6%
67%
24%
1%
12%
66%
21%
0%
10%
20%
30%
40%
50%
60%
70%
FY 2017 (N = 224; Mean = 94.3%; Median = 96.0%)
FY 2018 (N = 312; Mean = 95.6%; Median = 97.1%)
FY 2019 (N = 323; Mean = 94.3%; Median = 96.2%)
Quality Management Annual Report FY 2019
206 | P a g e
PCR Provider Record Review Scores by Year
PCR Support Coordinator Record Review Scores by Year
<1%1%
5%
14%
27%29%
18%
6%
<1%1%
4%
9%
16%
28% 29%
11%
2%<1%
2%
8%
24%
28%25%
11%
2%
0%
5%
10%
15%
20%
25%
30%
FY 2017 (N = 478; Mean = 70.2%; Median = 70.5%)
FY 2018 (N = 483; Mean = 74.8%; Median = 75.9%)
FY 2019 (N = 478; Mean = 74.9%; Median = 74.3%)
1%1% 3% 5%
10%
16%
20%
24%
17%
2%1%
<1%1% 2% 2%
9%
13%
19%
25% 24%
3%
1% <1%2%
3%
7%
12%
25%
27%
19%
2%
0%
5%
10%
15%
20%
25%
30%
FY 2017 (N = 481; Mean = 73.7%; Median = 76.4%)
FY 2018 (N = 484; Mean = 77.4%; Median = 75.9%)
FY 2019 (N = 484; Mean = 76.8%; Median = 78.9%)
Quality Management Annual Report FY 2019
207 | P a g e
PCR Support Coordinator Interview Scores by Year
FY 2019 Overall QEPR Score
11%
1% 2%7%
45%
33%
12%
2%5%
39%42%
13%
1% 1% 1%6%
12%18%
48%
0%
10%
20%
30%
40%
50%
60%
FY 2017 (N = 479; Mean = 83.3%; Median = 96.7%)
FY 2018 (N = 484; Mean = 83.5%; Median = 98.4%)
FY 2019 (N = 484; Mean = 79.4%; Median = 96.4%)
12%
60%
28%
0%
20%
40%
60%
80%
100%
(N = 97; Mean = 86.2%; Median = 86.8%)
Quality Management Annual Report FY 2019
208 | P a g e
FY 2019 QEPR Individual Interview Scores
FY 2019 QEPR Observation Scores
2%
30%
66%
2%0%
20%
40%
60%
80%
100%
(N = 96; Mean = 92.3%; Median = 92.4%)
1% 1%5%
83%
10%
0%
25%
50%
75%
100%
(N = 82; Mean = 97.2%; Median = 97.6%)
Quality Management Annual Report FY 2019
209 | P a g e
FY 2019 QEPR Staff Interview Scores
FY 2019 QEPR Record Review Scores
1%8%
80%
10%
0%
20%
40%
60%
80%
100%
(N = 97; Mean = 95.3%; Median = 96.3%)
1% 1%4%
12%
28%34%
14%
5%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
(N = 97; Mean = 73.2%; Median = 70.1%)
Quality Management Annual Report FY 2019
210 | P a g e
FY 2019 QEPR Qualifications and Training Scores
FY 2019 QEPR Service Discrepancy Scores
1% 2% 3%10%
16%
28%33%
6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
(N = 97; Mean = 83.2%; Median = 85.9%)
8%
92%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
(N = 97; Mean = 99.8%; Median = 100.0%)
Quality Management Annual Report FY 2019
211 | P a g e
FY 2019 QEPR Overall Scores by Provider Size
17%
60%
23%
18%
73%
9%
4%
68%
29%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Small (n = 47; Mean = 85.6%; Median = 86.2%)
Medium (n = 22; Mean = 84.6%; Median = 84.7%)
Large (n = 28; Mean = 87.2%; Median = 87.4%)
Quality Management Annual Report FY 2019
212 | P a g e
FY 2019 QEPR Qualifications and Training Scores by Provider Size
2% 2%
4%
9%
15%
26%
32%
11%
23% 23%
32%
23%
4% 4% 4%
18%
29%
39%
4%
0%
10%
20%
30%
40%
50%
Small (n = 47; Mean = 84.5%; Median = 86.9%)
Medium (n = 22; Mean = 80.4%; Median = 83.2%)
Large (n = 28; Mean = 84.0%; Median = 84.7%)
top related