using cem data in practice: f eedback from cem secondary assessments
Post on 23-Feb-2016
95 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using CEM Data in Practice:Feedback from CEM Secondary
Assessments
Glasgow Conference13th Feb 2013
Peter Hendry: CEM Consultant
Peter.Hendry@cem.dur.ac.ukSecondary@cem.dur.ac.uk
Assessment for Excellence: 2 systems
S1/S2 Baseline Assessment (MidYIS):A Test of Developed Ability at start of S1 or S2
• 4 sections: Vocabulary, Maths, Non-Verbal and Skills (Proof Reading, and Perceptual speed and accuracy
• Computer adaptive* or paper-based• Any time Term 1 + catch ups• Value Added measures to SOSCA and to Scottish Qualifications
S2 Curriculum-based Assessments (SOSCA):A curriculum assessment of knowledge, skills and understanding at end of S2
• Assesses what has been taught in the classroom• Covers, Maths, Science, Reading (English)• Computer adaptive*• Four week assessment window mid April to mid May + catch ups• Value Added measures from PIPs, InCAS and MidYIS and to Scottish
Qualifications
• School and students’ test performances are compared to a nationally representative sample
• Assessment data from the tests provides school and pupil data to inform teaching and learning
• ‘Predictive’ data indicates future potential performance to SQ, and will to National 4 and 5, to inform target setting and subsequent monitoring
• ‘Value-added data’ provides measures of relative performance at SQ (and will for National 4 and 5) and SOSCA from earlier baselines to inform self-evaluation
AfE: key elements
AfE Feedback: MidYIS and SOSCAAssessment Feedback:
• Standardised score system: 50-150, mean 100, SD 15
• School data including component averages and band profiles (historical and current)
• Pupil data in the form of spreadsheets and Individual Pupil Records (IPRs)
AfE Feedback: MidYIS and SOSCA‘Predictions’ Feedback:
• Currently to S2 SOSCA from S1/S2 baseline (MidYIS) and to SQ from MidYIS and SOSCA
‘Value-Added’ Feedback:
• To SQ from curriculum baseline (SOSCA), and from S1/S2 baseline (MidYIS)
• To S2 curriculum assessment (SOSCA) from P7 baseline (PIPS or InCAS) and/or S1/S2 baseline (MidYIS)
www.cemcentre.org
Data for your school.
Detailed Information about the assessment. Well worth a look.
Bands, percentiles, standardised scores…
0
50 60 70 80 90 100 110 120 130 140 150
Standardised scores
DC B
A
105 20 301 40 90 958070 996050Percentiles:
25%
25% 25%
25%
98
Mean 100, Standard Deviation of 15
MidYIS Year 8/S1 Test Academic Year 2010/2011Scores Standardised On A Nationally Representative Sample of Schools
Standardised Scores Sort Data By...
Mean Score: 100
Standard Deviation: 15
Vocabulary Maths Non Verbal Skills MidYIS Score
Average 93.3 89.9 90.9 97.3 90.7
St. error 1.2 1.2 1.2 1.2 1.2
Surname Forename CEM Id Form Sex CustJAMES M 81 D 69 D 75 D 68 D 72 DCOREY M 76 D 61 D 64 D 57 D 65 DJOHN M 93 C 87 D 101 B 92 C 89 DSAMUEL M 88 D 91 C 82 D 91 C 88 DDANIELLE F 117 A 98 C 107 B 105 B 109 BFRAZER M 98 C 89 D 98 C 117 A 93 C
Sort Data By...
SCHOOL A: S1 baseline dataMidYIS Year 8/S1 Test Academic Year 2010/2011
Standardised Scores Sort Data By...
Mean Score: 100
Vocabulary Maths Non Verbal Skills MidYIS Score
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
Average 93.3 89.9 90.9 97.3 90.7
St. error 1.2 1.2 1.2 1.2 1.2
Surname Forename CEM Id Form Sex CustJAMES M 81 D 69 D 75 D 68 D 72 DCOREY M 76 D 61 D 64 D 57 D 65 DJOHN M 93 C 87 D 101 B 92 C 89 DSAMUEL M 88 D 91 C 82 D 91 C 88 DDANIELLE F 117 A 98 C 107 B 105 B 109 BFRAZER M 98 C 89 D 98 C 117 A 93 C
Sort Data By...
SCHOOL A: S1 baseline dataIntake profiles: overall and test component bands
A school with a ‘completely average’ intake would have 25% of pupils within each band.
MidYIS Year 8/S1 Historical Band ProfilesScores Standardised on a Nationally Representative Sample of Schools
Year Band A Band B Band C Band D2013 5 12 17 652012 5 18 28 492011 6 18 30 462010 11 20 29 412009 8 18 38 36
Percentage of pupils falling into each MidYIS Band over time.(Band A = Top 25% of the sample, Band D = Bottom 25% of sample.)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2009 2010 2011 2012 2013
Perc
enta
ge
MidYIS Baseline Testing Year
Historical Band Profiles: Overall Score(Nationally Representative Standardisation)
Band DBand CBand BBand A
SCHOOL A: S1 Baseline
End Year2009 2013OverallTest Section Start Year
Show Band Percentage Values
AfE S2 Curriculum based Standardised Feedback 2012Scores Standardised On AfE Curriculum based Sample 2012
Standardised Scores
Mean Score: 100
Standard Deviation: 15
Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9
St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
Surname Forename AfE Id Form Sex CustJames M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 CCorey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 DJohn M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 ASamuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 DDanielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 AFrazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C
Sort Data By...
SCHOOL A
AfE S2 Curriculum based Standardised Feedback 2012
Standardised Scores
Mean Score: 100
Standard Deviation: 15
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
St. S
core
Ban
d
Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9
St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3
Surname Forename AfE Id Form Sex CustJames M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 CCorey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 DJohn M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 ASamuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 DDanielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 AFrazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C
Text
Co
mpr
ehen
sion
Pass
age
Com
preh
ensi
on
Ove
rall
Read
ing
Num
ber,
Mon
ey &
M
easu
rem
ent
Info
rmat
ion
Hand
ling
Reading Mathematics Science
Shap
e, P
ositi
on &
M
ovem
ent
Ove
rall
Mat
hem
atic
s
Livi
ng T
hing
s &
the
Proc
esse
s of
Li
fe
Earth
& S
pace
Ener
gy &
For
ces
Ove
rall
Scie
nce
Spee
d Re
adin
g
Sort Data By...
SCHOOL A: S2 curriculum (SOSCA) data
School no.1
School no.2
50 60 70 80 90 100 110 120 130 140 150 Standardised Test Score
Stanine 1
2 3 4 5 6 7 8 9
4% 4 % 7% 7% 12% 12% 17% 17% 20%
STANINES
Individual Pupil Record DANIELLE Date Of Birth: 22/02/99 Student Number:Sex: F Form:
Band Stanine Percentile Standard Score
Vocabulary A 7 87 117
Maths C 5 45 98
Non-Verbal B 6 67 107
Skills B 6 63 105
MidYIS Score B 6 73 109
Band 25% of nationally representative sample is in each band (A = highest)
Stanine One of nine divisions of the Normal distribution (9 = highest)
Percentile Percentage of the nationally representative sample scoring less than this student
Standard Score Score standardised against a nationally representative sample
50
60
70
80
90
100
110
120
130
140
150
Vocabulary Maths Non-Verbal Skills MidYIS Score
Natio
nally
Sta
ndar
dise
d Sc
ores
Nationally Standardised Scores with 95% Confidence Band
S1 baseline IPR
Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:
AfE Band Stanine Percentile AfE Standardised Score
Speed Reading A 7 84 115Text Comprehension A 8 90 119Passage Comprehension B 6 74 110Overall Reading A 7 87 117Number, Money & Measurement B 6 62 105
Information Handling B 5 51 100Shape, Position & Movement C 4 26 90
Overall Mathematics C 5 46 98Living Things & the Processes of Life A 9 99.3 137
Earth & Space A 9 99.5 139Energy & Forces A 8 92 121Overall Science A 9 99.4 138
S2 curriculum (SOSCA) IPR
Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:
S2 curriculum (SOSCA) IPR
• Currently available to Standard Grade from AfE baselines
• In the future to National 4 & 5 Qualifications from AfE baselines
• Also to AfE S2 curriculum (SOSCA) from S1 and/or S2 baseline (MidYIS)
• Includes Chances Graphs
Predictions Feedback
BASELINE SCORE
G
RA
DE
G
RA
DE
***********************
*******************************
*******************************************
*************************************
**************************************
********************************
*****************************
*****************
Trend line (regression line)
3
2
1
4
How is a ‘prediction’ generated?50% on or above the trend line
50% on or below the trend line
‘PREDICTION’ (expected grade)
Data for 2012 examinationsG
rade
Predictions & Chances Graphs to Standard Grade subjects based on 2012 AfE scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Forename Sex Form AfE Id Cust AfE
Rea
ding
Sco
re
AfE
Mat
hs S
core
AfE
Sci
ence
Sco
re
Ove
rall
AfE
Sco
re
Adm
inis
tratio
n
Art &
Des
ign
Bio
logy
Bus
ines
s M
anag
emen
t
Che
mis
try
Com
putin
g St
udie
s
Cra
ft &
Des
ign
Dra
ma
Eng
lish
Fre
nch
Geo
grap
hy
Ger
man
James M 84 84 92 85 3.8 3.1 4.3 3.3 3.8 3.8 3.3 3.0 3.4 4.0 4.2 4.2 3.7 3.9 3.1 4.5 3.7 2.5 2.8 4.1 3.7Corey M 75 66 50 54 5.9 4.2 6.6 5.4 5.6 5.6 4.7 4.3 4.8 5.9 6.4 6.0 5.2 5.9 4.5 6.8 5.7 3.5 4.0 6.0 5.2Kieran Roland M 68 60 72 63 5.3 3.9 5.9 4.8 5.1 5.1 4.3 3.9 4.4 5.4 5.8 5.5 4.8 5.4 4.1 6.2 5.2 3.2 3.7 5.5 4.8Megan F 100 89 93 93 3.3 2.8 3.6 2.7 3.3 3.3 2.9 2.6 3.0 3.5 3.6 3.7 3.3 3.3 2.7 3.9 3.2 2.2 2.5 3.6 3.3Danielle F 117 98 138 120 1.5 1.8 1.5 1.0 1.7 1.7 1.7 1.4 1.8 1.9 1.7 2.1 2.0 1.6 1.4 1.8 1.4 1.3 1.4 1.9 2.0Frazer M 92 95 99 95 3.2 2.7 3.5 2.6 3.2 3.2 2.8 2.5 2.9 3.4 3.5 3.6 3.2 3.2 2.6 3.7 3.1 2.2 2.4 3.5 3.2
Teacher's Adjustment
(Levels)
Standard Grade Point Predictions Standard Grade Point PredictionsPoints To Grades Adjust
based on 2012 AfE scores
Please refer to the Information Sheet for details of
AfE S2 Curriculum based Assessments 2012
James M 84 84 92 85 4 3 4/5 3/4 4 4 3/4 3 3/4 4 4 4 3/4 4 3 4/5 3/4 2/3 3 4 3/4Corey M 75 66 50 54 6 4 6/7 5/6 5/6 5/6 4/5 4/5 5 6 6/7 6 5 6 4/5 7 5/6 3/4 4 6 5Kieran Roland M 68 60 72 63 5/6 4 6 5 5 5 4/5 4 4/5 5/6 6 5/6 5 5/6 4 6 5 3 3/4 5/6 5Megan F 100 89 93 93 3/4 3 3/4 2/3 3/4 3/4 3 2/3 3 3/4 3/4 3/4 3/4 3/4 2/3 4 3 2 2/3 3/4 3/4Danielle F 117 98 138 120 1/2 2 1/2 1 1/2 1/2 1/2 1/2 2 2 1/2 2 2 1/2 1/2 2 1/2 1/2 1/2 2 2Frazer M 92 95 99 95 3 2/3 3/4 2/3 3 3 3 2/3 3 3/4 3/4 3/4 3 3 2/3 3/4 3 2 2/3 3/4 3
Predictions: Chances Graphs50% chance of a grade 2 – the most likely single grade. 50% chance of a different grade
Chances Graphs based on Pupil’s SOSCA Test Score
Point Prediction =1.8Grade Prediction = 2
Not a label for life...just another piece of information
• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.
• In working with students these graphs are more useful than a single predicted or target grade
• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores
Latest information as at 13 Feb 2102• Replacing SG with Nat 4 & 5 predictions:
CEM statisticians have put together a proposal.
• This has been looked at by colleagues in Scottish Authorities.
• The CEM Technical team have now published the new predictive data.
• This data is now available on the CEM secondary+ website alongside the existing SG predictive data
The equivalence we have assumed between CfE outcomes and Standard Grade outcomes isoutlined below:
Standard Grade
National Level 3National Level 2
N3N2
Where a subject is only available at National Level 4 (e.g. Business), only grades N4, N3, N2will be reported.
Credit Level 1Credit Level 2
General Level (3 and 4)Foundation Level (5 and 6)
Standard grade level 7
National Level 4D
N4
For each student in each CfE subject, a predicted grade is given. The range of predictedgrades is as follows
Grade
National LevelNational Level 5 A,BNational Level 5 C,D
National Level 4National Level 3National Level 2
For subjects only available at National Level 5 (e.g. Accounting), the full range of grades willbe reported. Whilst we recognise that grades N4, N3, N2 are not available we have includedthem in the chances graphs to aid the teacher in supporting the student in making realisticsubject choices.
National Level 5 ANational Level 5 BNational Level 5 CNational Level 5 D
CEM Predicted GradeABC
AfE
Rea
ding
Sco
re
AfE
Mat
hs S
core
AfE
Sci
ence
Sco
re
Ove
rall
AfE
Sco
re
Acc
ount
ing
(N5)
Adm
inis
trat
ion
& It
Art
& D
esig
n
Bio
logy
Bus
ines
s (N
4)
Bus
ines
s M
anag
emen
t (N
5)
Car
e
Che
mis
try
Com
putin
g Sc
ienc
e
Dan
ce (N
5)
Des
ign
& M
anuf
actu
re
Dra
ma
CfE Grade Predictions
New slide!!
95 104 99 99 D N4 N4/D N4/D N4 D N4/D N4/D N4/D N4/D N4/D N4/D102 103 116 108 C D D D N4 C D D D D D D115 112 123 119 B C D C N4 B C C C C D C125 135 117 129 A B C B N4 A B B B B C B
Predictions & Chances Graphs for CfE National Qualifications based on 2012 AfE scores
The grade 'predictions' and chances data presented here are simulated based on the relationship between AfE S2 Curriculum based,Standard Grade and Curriculum for Excellence levels, in the absence of any actual CfE examination data being available. They should therefore be treated with caution. The relationships will be updated once the first cohort are awarded CfE qualifications in 2014.
AfE S2 Curriculum based Assessments 2012
.
4 4.5 5 5.5 6 6.5 7 7.5 820
40
60
80
100
120
Subject X
Out
com
e
0 2 4 6 8
-ve VA+ve VA
Trend Line/Regression Line
Raw Residual
Measuring Value-Added – Terminology
Exam
gra
de
BASELINE SCORE
50
60
70
80
90
100
110
120
130
140
150
160
170
50 60 70 80 90 100 110 120 130 140 150 160 170
AfE
Sco
re
MidYIS Test Score (Standardised on a Nationally Representative Sample)
Year 8 MidYIS Score 2010/2011 Against AfE Overall Mathematics 2012
SCHOOL A: S2 curriculum (SOSCA) data
SCHOOL C:AfE data
Standardised Residual Bar Chart
99.7% confidence limit
95% confidence limit
ANY VALUE IN THE INNER SHADED AREA SHOWS THE EXPECTED RESULTS i.e. AVERAGE VALUE ADDED
SCHOOL C:AfE data
SCHOOL C:SQ data
top related