a framework for statistical performance
DESCRIPTION
Presentation at the Royal Statistical Society's International Conference, Newcastle, September 2013. Paul Askew speakingdata.org.uk [email protected] A Framework Understating Statistical Performance This paper presents a framework for understanding, managing and presenting statistical performance data. It provides a manageable, but multidimensional way of organising a range of possible analysis and assessment to provide a rounded and balanced picture for effective understanding and communication. This is founded on a broad scope of assessment, based around three key analytical elements: trend, benchmark and target. That is the first multidimensional perspective. Each of those three key elements are then further disaggregated in a range of appropriate ways to provide a second, more disaggregate, multidimensional analytical assessment. There is then a third more tailored and specific multidimensional disaggregation to provide additional detailed statistical insight. This draws on practical application of this framework in a range of sectors including health, policing and education, and across the public sector spectrum including Central Government, regulation and local service delivery, and in both strategic and operational environments. This provides an overall framework to manage the communication and understanding of statistical performance, and focussed on public data. It provides a framework to build a balanced communication of statistical analysis and messages, and at the same time it provides the user and recipient of statistics and statistical analysis with a framework to both understand and question the scope and content of statistical communication.TRANSCRIPT
![Page 1: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/1.jpg)
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
![Page 2: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/2.jpg)
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
![Page 3: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/3.jpg)
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
![Page 4: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/4.jpg)
1. Introduction
Operational Delivery
Methodological Leadership
![Page 5: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/5.jpg)
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
![Page 6: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/6.jpg)
![Page 7: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/7.jpg)
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
![Page 8: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/8.jpg)
Data
Eviden
ce
Decisi
ons
![Page 9: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/9.jpg)
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
![Page 10: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/10.jpg)
“Burglary is down compared to last month”
“Yes but it’s up compared the same month last year”
“Yes but it’s down overall for the financial year to date”
“Yes but its’ up for the calendar year so far”
“Yes but we’re still less better than our neighbours”
“Yes but they are reducing faster than we are this year”
“Yes but we’re still under (over) target”.
etc………….
![Page 11: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/11.jpg)
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique,
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
![Page 12: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/12.jpg)
Smoothed Data or Real Data
Highs and lowsHigh and low
DecreasingIncreasing Increasing convergence
Decreasing convergence
Three month step Six month stepTwo month step
Smoothed Data – 12 month rolling average
Smoothed Data
Example Real Data
Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).
This smoothed data is derived from any of these underlying
raw data examples.
![Page 13: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/13.jpg)
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
![Page 14: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/14.jpg)
Scope - Detail - Volume
Data.gov…10K
![Page 15: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/15.jpg)
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
![Page 16: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/16.jpg)
Words
Numbers
Data
Eviden
ce
Decisi
ons
![Page 17: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/17.jpg)
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
![Page 18: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/18.jpg)
Skills for Life Survey 2011 (England)Department for Business Innovation and Skills
• Literacy Improving while Numeracy declining
Numeracy • 26% to 22% (7.5m
adults) with GCSE+
• 17m adults at primary school level
The numeracy challenge is big and getting bigger…%
Adu
lts a
t GC
SE
+ L
evel
s
![Page 19: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/19.jpg)
A Framework for Understanding Statistical Performance
Paul Askew
![Page 20: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/20.jpg)
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
![Page 21: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/21.jpg)
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
![Page 22: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/22.jpg)
3. How - Macro
DATA - inputs -
ANALYSIS- process -
PRODUCTS- outputs -
INSIGHT- outcomes -
![Page 23: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/23.jpg)
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
![Page 24: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/24.jpg)
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
![Page 25: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/25.jpg)
II = (d)D x (t)T x s(S) x (c)C x (q)Q x (i)IThe factors:
D Data: Right data? Enough of it? Good enough?T Tools: Have any? Right ones?S Skills: Have any? Right ones?C Capacity: How much? Realistic?Q Question: Specific question to answer or issues to address I Inclination: Desire and drive to want to address the issues
d,t,s,c,q,i Relative weights
Askew Analytical Insight Index
![Page 26: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/26.jpg)
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
![Page 27: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/27.jpg)
3.Target
2. Bench-mark
1. Trend
Comp-aritors
Time Periods
Time Periods
3. How – Analytical Level
0. Snapshot
![Page 28: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/28.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 29: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/29.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 30: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/30.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 31: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/31.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 32: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/32.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 33: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/33.jpg)
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
![Page 34: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/34.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
0. Snapshot – we have a number which is important to us
![Page 35: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/35.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
1. Trend – what’s happening over time
![Page 36: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/36.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2. Benchmark – how this measures compares to others
![Page 37: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/37.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2a. Trend for the comparison to others
![Page 38: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/38.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3. Target - the trajectory for our measure
![Page 39: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/39.jpg)
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3a. Target - Trajectory for the comparison to others
![Page 40: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/40.jpg)
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro
Analytical
Outline
![Page 41: A Framework for Statistical Performance](https://reader033.vdocuments.site/reader033/viewer/2022052621/558a5509d8b42a851e8b467d/html5/thumbnails/41.jpg)
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
Thank You