measurement for improvement. turn to your neighbor what have been your biggest learnings or...
TRANSCRIPT
Turn to your neighbor• What have been your biggest learnings or
challenges regarding data gathering and measurement for your Early Years work?– Definitions– Collection– Reporting– Frequency– Analysis– Other?
Components of a Learning System 1. System level measures2. Explicit theory or rationale for system changes3. Segmentation of the population4. Learn by testing changes sequentially5. Use informative cases: “Act for the individual learn for
the population” 6. Learning during scale-up and spread with a production
plan to go to scale7. Periodic review8. People to manage and oversee the learning systemFrom Tom Nolan PhD, IHI
What are we trying toAccomplish?
How will we know that achange is an improvement?
What change can we make that will result in improvement?
The Model for Improvement
Act Plan
Study DoSource:
Langley, et al. The Improvement Guide, 1996.
Our focus today
The three questions
provide the strategy
The PDSA cycle provides the tactical approach to work
AIM (How good? By when?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Case Study• Promoting Bedtime Reading• Lochrin and Grassmarket Nursery Schools• Edinburgh• Lead: Donna Murray
AIM – Improve bedtime routines, attachment, literacy
Concept – Increase bedtime reading
Measures – Percentage of children receiving a bedtime story
Operational Definitions – Number read story/Total Children
Data Collection Plan – daily; no sampling
Data Collection – teacher asks parents at drop off, spreadsheet
Analysis – run chart PDSA
The Quality Measurement Journey
Children have all the developmental skills and abilities expected at the start of primary school
Societal Issues
Child’s physical & mental health and emotional development
Improved sharing of information
Improved management, planning and quality of services
Improved joint working
Improved identification
Detailed Aim:
90% of all children within each CPP have reached all of the expected developmental milestones at the time the child starts primary school, by end-2017
Health
Improved child’s dental health
Improving child nutrition
Attachment
Improved stability / permanence for LAC
Improved uptake of benefits
Improved family centred response
Improved leadership, culture & planning`
Improved teamwork, communication and collaboration
Improving brain development and physical play
Early Learning & Play
WORKSTREAM 3 (30 months to start of primary school)
Identification & reasons for current resilience
Aim 1⁰ 2⁰
Theory of what drives developmental milestones
Theory of what actions will ensure developmental
milestones are reached at the start of primary school
Poverty
Quality Of Home Environment
Domestic Abuse & Violence
Workforce Issues
Transport, Community Capacity & Cultures
Access To Services
Employment
Carer’s physical & mental health and skills
Level of education
Misuse of alcohol & drugs
Nutrition
Disabilities & Mental health
Parenting skills & knowledge
Additional Support
Version: 06/03/2013
AIM (How good? By when?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Project Aim Statements
• What, by when, by how much.–“By the end of June, 90% of
children at Grassmarket Nursery will receive a bedtime story at least 3 times a week.”
AIM (How good? By when?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
PDSA
The Quality Measurement Journey
Children have all the developmental skills and abilities expected at the start of primary school
Societal Issues
Child’s physical & mental health and emotional development
Improved sharing of information
Improved management, planning and quality of services
Improved joint working
Improved identification
Detailed Aim:
90% of all children within each CPP have reached all of the expected developmental milestones at the time the child starts primary school, by end-2017
Health
Improved child’s dental health
Improving child nutrition
Attachment
Improved stability / permanence for LAC
Improved uptake of benefits
Improved family centred response
Improved leadership, culture & planning`
Improved teamwork, communication and collaboration
Improving brain development and physical play
Early Learning & Play
WORKSTREAM 3 (30 months to start of primary school)
Identification & reasons for current resilience
Aim 1⁰ 2⁰
Theory of what drives developmental milestones
Theory of what actions will ensure developmental
milestones are reached at the start of primary school
Poverty
Quality Of Home Environment
Domestic Abuse & Violence
Workforce Issues
Transport, Community Capacity & Cultures
Access To Services
Employment
Carer’s physical & mental health and skills
Level of education
Misuse of alcohol & drugs
Nutrition
Disabilities & Mental health
Parenting skills & knowledge
Additional Support
Version: 06/03/2013
Drivers
Drivers
Drivers
Concept:Bedtime Reading
Driver Concept and Measure
• Measures to provide feedback on the concept– % receiving a story– % of stories read at bedtime– % of parents reporting improved bedtime routine– % enjoying the bedtime story– % reporting increase in bedtime story reading
AIM (Why are you measuring?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Operational Definitions• What does reading a story mean?• When is bedtime versus other time?• What is improved bedtime routine?• What is the definition of enjoyed?
AIM (Why are you measuring?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
The Quality Measurement Journey
Data Collection Planning & Doing
• All children are included, no sampling. 7 days a week.
• Donna will inquire with each parent at drop off Monday through Friday.
• Data will be captured on a paper form. • Donna inputs daily data into spreadsheet
following drop off.
AIM (Why are you measuring?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
The Quality Measurement Journey
Analysis – Run Charts
Children receiving a bedtime story
MedianGoal
0
20
40
60
80
100
120
M T W TH F M T W TH F M T W TH F M T W TH FDay of the week
Perc
en
tag
e
of
ch
ild
ren
.
Parents survey
Grassmarket changes
introduced.
Books available at collection time.
Research information handed to
parents.
weekly average displayed for parents
AIM (Why are you measuring?)
Concept Measures
Operational Definitions Data Collection Plan Data Collection
Analysis PDSA
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
The Quality Measurement Journey
Background
• NHS Lanarkshire Clinical Quality Service, supporting both North and South Lanarkshire CPPs from a health perspective
• 27-30 month reviews commenced June 2013 in Lanarkshire – measurement data for stretch aim 2
• But we don’t have baseline data…….• ….so we need early access to our review data to
develop a baseline…….• ISD Proposed quarterly reports…….November
2013 before we see our first set of results
Accessing our data – lots of questions....
• Where do the forms go?• Does anyone in Lanarkshire have access to the
CHSP system?• Can we access the system?• Technical challenges accessing the
system....patience.......patience........we’re in!!• We’re in but what do all these codes mean?• Expertise to interrogate the data
How did we access CHSP Business Objects Universe (AKA – our data)?
• Contact the ATOS Helpdesk [email protected]
• Receive a form to complete via email, needs authorising by your Child Health Administrator
• Receive complex instructions from ATOS on how to set-up your PC/laptop – stick with it, you’re getting there!!
• Receive your Business Objects Universe login and password from your local Child Health Administrator
Benefits of accessing our data
• Earlier access to our data – some delays come from the time to return the forms for data entry and data entry itself (approx 4 weeks)
• Early identification of quality issues with form completion enabled us to feed this back and engage with staff
• We have early baseline data• Stratify the data as we like (unit, locality, team)• Results are helping us identify early tests of
change
Early data focus• Meaningful reviews (9 developmental milestones
reviewed and recorded)• Stretch aim 2 (meeting all developmental
milestones)• Future actions (request for assistance/referral)• Looked After Status• Breeches• Health Plan Indicator (HPI) – status change• Attendance / DNA – obtained locally
Coverage – A work in progress:
Number of Eligible Children
Number Invited for
Review
Number Attending for Review
Number of Meaningful
Reviews
A national definition of ‘eligible children’ in a reporting period is still in development.
Eligible = denominator/starting pointInvited: coverage measure 1Attended: coverage measure 2Meaningful: coverage measure 3
Coverage: An early example
July August September0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
27-30 Month Review - Attendance July-Sep 13 (NHSL 1 Unit)
Locality 1
Locality 2
Locality 3
Examples of our early data
Jun-13 Jul-13 Aug-13 Sep-13 (p)0
50
100
150
200
250
300
350
400
450
257
309
392
289
NHSL Number of Reviews - Jun to Sep 2013
Lanarkshire
Month
No
. o
f re
view
s
(p) Provisional data - incomplete
Jun-13 Jul-13 Aug-13 Sep-13 (p)0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
76% 76%72% 74%
% of Meaningful Reviews - Jun to Sep 2013
Meaningful Reviews %
Month
%
(p) Provisional data - incomplete
Jun-13 Jul-13 Aug-13 Sep-13 (p)0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
22.39
17.15
23.72
29.76
Future Actions - Rate per 100 reviews Jun to Sep 2013
Lanarkshire
Month
Rat
e p
er 1
00 r
evie
ws
(p) Provisional data - incomplete
Jun-13 Jul-13 Aug-13 Sep-13 (p)0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
85% 87% 86%
79%
No Concerns Identified (Stretch aim 2) - Jun to Sep 2013
No Concerns %
Month
%
(p) Provisional data - incomplete
Limitations of our early data• We’re still developing a robust process to
measure uptake - so results are only representative of those attending
• Non attenders are potentially our more vulnerable children and this may be skewing our results – particularly stretch aim 2
• Non-meaningful reviews are being excluded, we need to reduce this (approx 25%)
Stage 2 data focus
• What other data is available from the review?– Age of child at review– SIMD quintiles– Ethnicity– Place of review (home, GP
practice, clinic)– Exposure to smoke– Registered with a dentist– Bi/Multi-lingual
Next steps• Continue to work with colleagues to develop reporting
that is meaningful and facilitates/guides improvement• Continue to feedback real time issues with data
completeness and quality• Further work to link other reviews and develop data
intelligence over time for each child e.g. HPI• Further work to consider how best to share this data
with partners out with health to benefit children and families
I hope it did make you feel like this.....
Thank you to Scott Purdie, Clinical Quality Co-ordinator, NHS Lanarkshire for his work in this area.
And
My niece Emily, age 9 months for brightening up the data talk!