performance monitoring with feedback: a holistic approach performance monitoring using value added...
TRANSCRIPT
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
PERFORMANCE MONITORING USING
VALUE ADDED DATA (Post – 16)
Keith Murdoch
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Overview of Presentation
• Context: The College, the locality and some political drivers
• An overview of Woodhouse College’s Performance Monitoring approach
• Performance Monitoring– College– Departments– Students
• Concluding musings
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Woodhouse College:
1,145 16-19 full time students
• 99% A – level provision
• 58% Female
• 54% BME
• Enrolments from 140+ schools
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Value Added & the Common Inspection Framework
• Learners’ attainment and progressOutcomes for Learners
Quality of Provision• Effectiveness of teaching, training and assessment in
supporting learning and development
• Effectiveness of the care, guidance and support learners receive
Leadership and Management• Raising expectations and promoting ambition
• Actively promoting equality and diversity to narrow the achievement gap
• Effectiveness of self assessment
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
PERFORMANCE MONITORING
• Monitoring Student Progress / Student Reviews
• Lesson Observation
• Internal Inspections
• Every Child Matters
• Diversity and Equality
• Self Assessment: Department / College
Key Elements: Using value added data
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch SELF ASSESSMENT
• How well do you know your Department / College?
• How well do you know your data?
• Can you accurately identify your weaknesses?
• Do you have the capacity to make improvements?
• Can you provide evidence to demonstrate improvements?
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
How are we doing and how do we know?
Departments
Whole College
Achievement
Achievement & Attitudes
What are we doing about the poor bits?
ALIS helps us measure:
Students Progress on Course
e.g. Target Setting, Action Plans, Operational Plans, Strategic Plans
with
Performance Monitoring
Using ALIS to monitor achievement & attitudes with feedback to 3 levels
Qua
lity
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Aug & Sept
Oct Nov Dec Jan Feb Mar Apr May Jun Jul
Data Capture GCSE A/AS
Achievement Feedback
Strategic & Operational Planning
Dept. SAR Achievement + Action Plans
Whole College
SAR
Input Feedback
Yr 12 Reviews
Dept SAR Phase 1 +
Action Plans
Yr 13Data Capture: Attitudes
Yr 12 Reviews
MAG
Curriculum Quality Monitoring & Internal Inspections
On-course student assessment and monitoring, subject by subject
Yr 12 & 13 Reviews
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Monitoring the overall performance of the COLLEGE
Summative monitoring by Senior Leaders and Governors
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
-0.2
-0.1
0.0
0.1
0.2
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Ave
rage
Sta
ndar
dise
d R
esid
ual
Exam Year
-0.2
-0.1
0.0
0.1
0.2Final_Result
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
-0.2
-0.1
0.0
0.1
0.2
0.3
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Ave
rage
Sta
ndar
dise
d R
esid
ual
Exam Year
-0.2
-0.1
0.0
0.1
0.2
0.3Final_Result
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
-0.2
-0.1
0.0
0.1
0.2
0.3
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Ave
rage
Sta
ndar
dise
d R
esid
ual
Exam Year
-0.2
-0.1
0.0
0.1
0.2
0.3Final_Result
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
-0.6
-0.4
-0.2
-0.0
0.2
0.4
0.6
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Ave
rage
Sta
ndar
dise
d R
esid
ual
Exam Year
-0.6
-0.4
-0.2
-0.0
0.2
0.4
0.6Final_Result
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch A Level results 2010No of students on a given MAG achieving A*-U grades A* A B C D E U Grand TotalA/B 56 112 40 7 2 218B 67 330 302 139 36 15 2 891B/C 3 20 28 26 10 2 1 90C 8 47 87 96 60 22 19 339C/D 2 4 3 2 1 12D 1 4 6 10 6 2 29D/E 1 1 1 3E 2 5 1 2 2 12Grand Total 134 512 469 280 124 49 25 1594
% of students on a given MAG achieving A*-U grades A* A B C D E U Grand TotalA/B 26% 51% 18% 3% 1% 100%B 8% 37% 34% 16% 4% 2% 100%B/C 3% 22% 31% 29% 11% 2% 1% 100%C 2% 14% 26% 28% 18% 6% 6% 100%C/D 17% 33% 25% 17% 8% 100%D 3% 14% 21% 34% 21% 7% 100%D/E 33% 33% 33% 100%E 17% 42% 8% 17% 17% 100%
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Report to Governors
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Monitoring the Performance of DEPARTMENTS
SELF ASSESSMENT: summative monitoring by departments
of their own performance using fair comparisons of achievement levels.
Analysing student achievement by subject
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Success rates
% Grade A* - B at A2 and A - B at AS
Pass rate
Value added (A2, AS): exam year / 3 year
Performance compared to MAG
Retention rate
Learner satisfaction
Attendance
Enrolment number
Gender and Ethnicity monitoring
Lesson Observation grade profile
Internal Inspection grade
Average class size
Key Performance Indicators
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Year Institns in GCSE ITDA Av. A Residual Error Attitude Attitude to Advising Extra- LSEcohort to Subject College College Mural
2004 6.4 50.2 7.4 0.36 10% (+/ -)0.2 3.6 3.7 70%
0.1
2005 6.2 48 5.7 -0.4 10% (+/ -)0.18 3.6 3.7 82%
0.09
2006 6.4 49 93.1 0.04 (+/ -)0.22 3.4 3.6
0.11
2007 6.5 48 95 -0.04 (+/ -)0.2 3.3 3.9
0.1
2008 6.5 48 101 0.31 10% (+/ -)0.2 3.5 3.7
0.1
2009 6.3 94 0.1 25% (+/ -)0.18 3.3 3.7
0.09
2010 6.5 102 0.29 10% (+/ -)0.2 3.5 3.7
0.1
2011 (+/ -)0.
Year Analysis of Residuals by GCSE scores Enrol Complete Retention
"A" "A" "A"8 - 6.2 8 - 6.9 6.8 - 6.2 6.1 - 5.5 5.4 - 4.7 4.6 - 4
2002 0.2 56 0.3 26 0.5 8 0 95 90 95%
2003 0.3 63 -0.1 24 1.6 6 -0.9 1 102 94 92%
2004 0.55 48 0.29 21 0.76 27 0.9 32 0.85 14 -0.6 1 110 95 86%
3yr. 0.35 0.37 0.98 -0.50
2005 -0.9 54 -0.2 19 -1.3 35 -0.7 38 -0.6 13 -0.6 1 115 106 92%
3 yr. -0.02 0.03 0.62 -0.70
2006 1.4 59 1.4 27 1.4 32 -2.4 20 2.6 11 0 92 90 98%
2007 -1 76 -1.3 36 -0.7 40 2.7 22 -7.8 5 13.7 1 109 104 95%
2yr. 0.20 0.05 0.35 0.15 -2.60 6.85
2008 5.6 80 4.6 32 6.2 48 -2 11 12.6 12 8 1 105 104 99%
3yr. 2.00 1.57 2.30 -0.57 2.47 7.23
2009 1.5 65 2.2 27 1.0 38 2.4 42 -4.4 7 31.1 1 120 117 98%
3yr. 2.03 1.83 2.17 1.03 0.13 17.60
2010 4.2 78 1.8 34 6 44 7.1 23 7.9 3 23.7 1 108 105 97%
3yr. 3.77 2.87 4.40 2.50 5.37 20.93
20113yr.
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Year GCSE TDA Av. A Residual Error Attitude Attitude to Advisingto Subject College College
2010 6.5 102 0.29 10% (+/-)0.2 3.5 3.70.1
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
chSELF-ASSSESSMENT: DEPARTMENTS
ALIS DATA ANALYSIS
1. Summary of Raw Results• Comparison of raw subject A-level results with the national percentage for the subject
2. Analysis by GCSE Score• In terms of residuals how have students in different bands been performing?• Does a preponderance of students in any one band help explain the overall residual?
3. Analysis by Ethnic Minority and Gender• In terms of actual scores and standardised residuals, how have students from different ethnic minority/gender
groups performed?• Are the differences significant?
4. Analysis of Extreme Cases• Can we identify common features within the high and low achieving groups?• Do the extremes distort the overall picture of the subject performance?
5. Analysis of Variance • An explanation of data points which lie outside the control lines• A comment on the moving average
6. Analysis by Teaching Group• Is there any apparent correlation between set and residual or, over the 3 year period, between member(s) of staff
teaching a set and the residuals achieved, after sets have been analysed for the range of ability?
7. Students’ Attitudes and learning and Teaching Processes• Analysis in trends in students’ attitudes to subject• Issues raised by analysis of perceived learning activities.
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
1. Analysis by GCSE Score– In terms of residuals, how have students in
different bands been performing?– Does a preponderance of students in any one
band explain the overall residual?– How does your analysis impact on strategies
for teaching and learning?
Analysis of Residuals by GCSE scores
8 - 6.2 8 - 6.9 6.8 - 6.2 6.1 - 5.5 5.4 - 4.7 4.6 - 42010 4.2 78 1.8 34 6.0 44 7.1 23 7.9 3 24 1
3yr. 3.77 2.87 4.40 2.50 5.37 20.93
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
2. Analysis by Gender & Ethnic Minority
– In terms of standardised residuals, how have students from different e-m/gender groups performed?
– Are the differences significant?– What impact will these differences have on your
teaching and learning strategies?
Year Gender No AvGCSE AvS AvR StdR Difference
2010 F 54 6.8 95.6 4.8 0.2M 45 6.7 100.4 12.8 0.6
08-10 F 178 0.35M 117 0.49
0.14
0.40
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
3. Analysis of Extreme Cases– An extreme is a student with a raw
residual of + or – 30 (15 for an AS)
– Can we identify common features within the low and high achieving groups?
– Do the extremes distort the overall picture of the subject performance?
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Student Level Residuals
Surname Forename Average GCSE Score
Actual Grade
Predicted Grade
Raw Residual
Chandaria
Jaymin
5.7
60
50
10
Davy-Martin
Lee
6.7
120
85
35
Lai
King-Ho
6.1
0
62
-62
Mansigani
Ilaha
6.0
120
60
60
Taylor
Hannah
6.3
100
75
25
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
4. Analysis of Variance– An explanation of data points which
lie between control lines
– A detailed explanation of data points which lie outside 3SD control line
– A comment on the moving average
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Statistical Control Chart
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Yr Yr Yr Yr Yr YrYr Yr Yr
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Identifying strengths and weaknesses understanding your weaknesses and
identifying actions for improvement impact on strategies for teaching and
learning and supporting students
KEY ELEMENT of
SELF ASSESSMENT ANALYSIS
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Using ALIS data to identify and ‘unpack’ weaknesses
AS Physics
-0.28 Standardised Residual
-0.7 Female
-0.1 Male
-0.64 without Maths
-0.19 with Maths
•Work with Maths Department on creating resources to support student not taking AS Maths
• Introduce problem-solving consolidation sessions
• Further investigation of girls underachievement – Institute of Physics, Standards Unit, focus groups etc
e.g.
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
-0.23 Standardised Residual
• Restructure external assessment of course – January module
• Increase hours for Applied AS students, formalise requirements, ‘integrate’ into main programme
• Review appropriateness
e.g. … contd.
AS Critical Thinking
3 distinct groups
1. Additionality: 0.1
2. Full programme : -0.8
3. Applied AS: -0.5
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Monitoring the Performance
of STUDENTS
to raise aspirations and achievement
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
You know you’ve gone to Woodhouse when.....
MAG/CAG. They do mean something.
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
PROBLEMMotivating, and monitoring student progress is a FORMATIVE process
ALIS is RETROSPECTIVE SUMMATIVE STATISTICAL
HOW CAN WE ‘SQUARE THE CIRCLE?’
Possible because correlations are high and the variation in the association between AVGCSE and the statistically ‘PREDICTED’ A-Level grade, from one year to the next, in any given subject, tends to be very SMALL
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
KEY PROCESSES Departmental Monitoring and Assessment
Practices which feed into
Student Reviews Scheduled monitoring of a student’s progress across their
programme Mutually supporting academic and pastoral functions
using COMMON DATA
MINIMUM ACCEPTABLE GRADES
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch BENCHMARKING and TARGET SETTING: Defining Grades
1. MAG: Minimum Acceptable Grade
2. CAG: Current Achievement Grade
The statistically predicted grade for each subject (Scale A/B – E [E] based on the ALIS trend line) which will not change during the year. Provides an initial benchmark (with associated health warnings) against which a student’s progress can be judged.
The grade a student is currently working at.
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Defining Grades (contd.)
3. TAG: Target Achievement Grade
4. PG: Predicted Grade
The grade (above the MAG) a student is considered capable of working at and should aim to achieve. Student Review discussions (where appropriate) would be focused on negotiating strategies to enable students to attain this grade.
The grade that is written on a student’s UCAS form and subsequently changed or confirmed as part of the return to Examination Boards
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
CALCULATING THE MINIMUM ACCEPTABLE GRADE
The student MAG for each subject is based on the
previous year’s ALIS ‘trend-line’ for that subject
Principle = ‘good enough for purpose not statistically flawless’
Example:
Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C
Total Points = 66 (i.e. 2x8 + 3x7 + 4x6 + 1x5)
AVGCSE = 66 / 10 = 6.6
MAP for subject (15.78 x 6.6) – 62.65 = 41.5
MAG = C (at AS)
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Technical problem!!!calculation creates number and decimal points which need to be translated into UCAS grades
College Conversion Table
• No student can be targeted to attain A* or A grade
• No student can be targeted to ‘fail’, i.e. below grade E
A-Level AS Level
MAG ‘Predicted’ Point Score ‘Predicted’ Point Score
A/B 109 – 140 53 – 60
B >91 - <109 >44 - <53
B/C 89 – 91 43 – 44
C >71 - <89 >34 - <43
C/D 69 – 71 33 – 34
D >51 - <69 >24 - <33
D/E 49 – 51 23 – 24
E <49 <23
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
College policy on the monitoring of student progress and achievement
Framework for
• setting and marking students’ work
• monitoring students’ progress (target setting)
• use of subject tutorial period
Department Policies
Supporting Teaching and Learning
Progress in Individual Subjects
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Student Name MAG: B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
100 90 80 74 72 71 71 72 70 67 63 63
65 67 65 60 55 58 56 59 55 50 48 40 30 20 10 0
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Nayan
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5 6 7 8 9 10 11
AS
MAG
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Dept Tutor Review Parents
SeptSet Lists Programme
Interview2nd Yr Review
MAG calculated
New Parents
2nd Yr meeting
Oct 1st Yr Subject Assessment
Programme Interview
1st Yr Review & Case Conference
Grade Report
Dec2nd Yr Subject
Assessment
Grade Report
Jan1st Yr Subject Assessment
1st & 2nd Yr Programme Interviews
1st & 2nd Yr Reviews & Case
Conferences
2nd Yr meeting
Feb 1st Yr G-Report & meeting
Mar 2nd Yr Subject Assessment
2nd Yr Reports
Jun 1st Yr Subject Assessment
1st Yr Reports (July)
Student Review Cycle
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Start of the Student Review Process
September: AVGCSE score used to calculate MAG (AS) subject by subject : included on set lists and programme record card
October: Autumn Review – with Personal Tutors
Scale: 1 = performing outstandingly
2 = satisfactory
3a/b = study skills &/or conceptual problems
4 = ‘alarm’ signal
5 = very recently joined subject
+ Effort Grade 1 - 4
AS MAG is provided as benchmark
Case Conference
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
Categories Scales Source
MAG A/B - E MIS
Module Result % Teacher
CAG A - F Teacher
Effort 1 - 4 Teacher
Quality of Work 1 - 4 Teacher
Meeting Deadlines 1 - 4 Teacher
Involvement 1 - 4 Teacher
Attendance % present MIS
Student Review Data
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch
MAG RES CAG EFF QoW MD INV ATT
310765 Student Name (98%)
Eng-s-ld AS WAL B 57 B 2 2 1 2 97
Hist-s-le AS THO B 60 B 2 2 1 2 100
Law-s-lc AS SFR B 49 B/C 2 2 2 2 96
Psyc-s-la AS STE B 54 B/C 1 2 1 2 100
329368 Student Name (88%)
Geog-s-lb AS MCA C 37 E/F 4 4 4 2 90
Germ-s-ld AS PAR D 28 F 4 4 4 4 88
Pol-s-le AS GRI C 30 F/E 4 4 3 3 83
PSM-s-la AS ALN E 30 E 3 3 3 3 91
313492 Student Name (95%)
Tutor Programme Review Data
Per
form
ance
Mon
itor
ing
wit
h fe
edba
ck: a
hol
isti
c ap
proa
ch Some musings……
• measurement gives messages
• evolve your own
• invest in the time
• transparency as a tool not a threat
• know the health warnings
• embed
• never take understanding for granted
• trust the tribe