2008 star interpreting and using results august/september 2008
TRANSCRIPT
2008 STAR2008 STARInterpreting and Using Interpreting and Using
ResultsResultsAugust/September 2008
August/September 2008 Post-Test Workshop 2
ObjectivesObjectives
Workshop participants will be able to:
• Describe the purposes of STAR reports
• Interpret STAR results
• Explain key statistics
• Compare and contrast types of reports
• Identify proper uses of reports
August/September 2008 Post-Test Workshop 3
AgendaAgenda
• Quiz• Post Test Guide• What’s New?• Results and Statistical Analysis• Using Results• Summary and Internet Reports• Individual Student Reports• Teacher Reports
August/September 2008 Post-Test Workshop 4
Post-Test Guide LayoutPost-Test Guide Layout
• Part I: General Info– New in 2008– Interpreting Reports– Comparing Results
• Part II: Report Descriptions– Summary– Individual– Teacher– Internet
• Part III: Appendices– Reporting clusters– Scale score ranges– Statistical tables– Test Score Verification
procedures– Demographic Data
Correction procedures– California reading list– Glossary
Post-Test Guide
Pages (Pp) i–iii
August/September 2008 Post-Test Workshop 5
What’s New in 2008What’s New in 2008
• CMA reports– Percent correct– Grades 3–5
• STS reports– Percent correct– Grades 5–7 added
• CAPA Science percent correct• Ethnicity for Economic Status subgroup
summary (CSTs only)• STAR Student Reports sorted for district
– All without addresses– All with addresses
Post-Test Guide
P 2
August/September 2008 Post-Test Workshop 6
Results: Results: Purposes of STAR ReportsPurposes of STAR Reports
• Report progress of students toward proficiency on the state’s academic content standards
• Inform where improvement needed– To help students’ achievement– To improve educational programs
• Provide data for state and federal accountability programs
Post-Test Guide
P 3
August/September 2008 Post-Test Workshop 7
Results: Performance LevelsResults: Performance Levels
• State goal: PROFICIENT or higher– Scale score of 350 or higher for CSTs– Scale score of 35 or higher for CAPA
• Other performance levels:– Advanced– Basic (300 CSTs, 30 CAPA)– Below Basic– Far Below Basic– CST and CAPA cut points vary for
Advanced, Below Basic, and Far Below Basic by
• Subject• Grade
Post-Test Guide
P 7
August/September 2008 Post-Test Workshop 8
Results: Scale ScoresResults: Scale Scores
• Scale scores allow same score to mean same thing across test forms within grade and content area and across time
• Scale score ranges by program:– CSTs: 150–600 for each grade and subject– CAT/6 Survey: 1–999 for each grade and
subject– CAPA: 15–60 for each grade and subject– CMA, STS, CAPA Science: Not available until
2009
Post-Test Guide P 7–8
August/September 2008 Post-Test Workshop 9
Results: Equating Results: Equating
• Psychometric procedure
• Adjusts for test difficulty • Additional information in CST Technical Report
on CDE Web site• For 2008
– CSTs and CAPA (except science) are equated– CMA (3–5), CAPA (science), and STS (2–4) and
will be equated starting in 2009• CAT/6 nationally norm-referenced; not equated
for STAR
Post-Test Guide Page 7
August/September 2008 Post-Test Workshop 10
Results: Reporting ClustersResults: Reporting Clusters(Content Areas)(Content Areas)
• Four to six clusters for each content area (subject)
• May be useful as indicators of individual or group strengths and weaknesses
• But… Reporting clusters should be interpreted with caution
Post-Test Guide
P 36
Appendices A & B
August/September 2008 Post-Test Workshop 11
Results: Reporting ClusterResults: Reporting ClusterCautionsCautions
• Based on small numbers of items; therefore may not be reliable or generalizable
• NOT equated from year to year
• Should not compare reporting cluster percent correct from year to year
Post-Test Guide
P 8–10
Appendices A & B
August/September 2008 Post-Test Workshop 12
Interpreting Reporting Clusters or Interpreting Reporting Clusters or Content Areas Content Areas
in the in the Same YearSame Year• CSTs—Performance on the clusters for
students statewide scoring from– Lowest score for proficient to– Lowest score for advanced
Post-Test Guide
P 8–10; 84
Appendices A & B
August/September 2008 Post-Test Workshop 13
CSTs 2008 Reporting Clusters — Number of CSTs 2008 Reporting Clusters — Number of Questions and Average Percent CorrectQuestions and Average Percent CorrectPost-Test Guide
Page 138(Appendix A)
August/September 2008 Post-Test Workshop 14
ExamplesExamples—Interpreting Reporting —Interpreting Reporting Clusters for CST for Geometry Clusters for CST for Geometry Post-Test Guide
P 9
0%10%20%30%40%50%60%70%80%90%
100%
Logic/GeomProofs
Volume & Area AngleRelationships
Trigonometry
Avg
% C
orr
ect
My Class 2007 Just Proficient Students 2007 Just Advanced Students 2007
0%10%20%30%40%50%60%70%80%90%
100%
Logic/GeomProofs
Volume & Area AngleRelationships
Trigonometry
Av
g %
Co
rre
ct
My Class 2008 Just Proficient Students 2008 Just Advanced Students 2008
August/September 2008 Post-Test Workshop 15
Using ResultsUsing Results
• For instructional decisions with other data: CSTs, CMA, CAPA, STS
• Used in Academic Performance Index (API) calculations: CSTs, CAT/6 Survey, CAPA
• Used in adequate yearly progress (AYP) calculations for grades 2–8 in ELA and math for CSTs and 2–8 and 10 in ELA and math for CAPA
Post-Test Guide
P 3
August/September 2008 Post-Test Workshop 16
Year-to-Year ComparisonsYear-to-Year ComparisonsDoDo Compare Same Grade and Same Compare Same Grade and Same
Content AreaContent Area
• Mean scale score – Same content and grade, varying years
• Percent in each performance level– Same content by grade across years
• e.g., 2007 ELA grade 10 with 2008 ELA grade 10
Post-Test Guide
Pp 11–14
August/September 2008 Post-Test Workshop 17
Year-to-Year ComparisonsYear-to-Year ComparisonsDoDo Compare Same Grade or Same Compare Same Grade or Same
Content AreaContent Area
• Percentage of students scoring at PROFICIENT and above– For a given grade and subject, e.g., Percent
proficient and above for grade 3 math in 2007 & 2008
– For a given subject and aggregated grades, e.g., % proficient and above for grades 2 to 6 math in 2007 & 2008
– Across grades and a subject, e.g., % proficient and above in all courses and all grades
Post-Test Guide
Pp 11–14
August/September 2008 Post-Test Workshop 18
Year-to-Year ComparisonsYear-to-Year ComparisonsDON’TDON’T Compare Compare
• Individual scale scores or statistics based on scale scores for different grades or content areas– Subjects by grade are independently scaled – Different content standards are measured in
different grades
• Cohorts across grades
• CMA, STS scores because notscaled
Post-Test Guide
Pp 11–14
August/September 2008 Post-Test Workshop 19
ExampleExample—Using CST Results to —Using CST Results to Compare Grade Results from Year to Compare Grade Results from Year to
YearYear
Post-Test Guide
P 12
2007 CST for ELA
2008 CST for ELA
Grade % Prof or Above
% Prof or Above
Difference
Grade 2 31% 35% 4%
Grade 3 33% 33% 0%
Grade 4 29% 31% 2%
Grade 5 34% 32% -2%
Grade 6 31% 32% 1%
All Grades
32% 33% 1%
August/September 2008 Post-Test Workshop 20
Aggregate (Summary) ReportsAggregate (Summary) Reports
• What are they?– Student Master List Summary– Student Master List Summary EOC– Subgroup Summary– Group Summary: CAT/6 Survey
• Report Emphasis: California Standards Tests (CSTs) – Criterion-referenced tests– Progress is measured in percent of students
scoring proficient and advanced
Post-Test Guide
Pp 17–21
August/September 2008 Post-Test Workshop 21
Student Master List SummaryStudent Master List Summary
• By grade• CSTs, CMA, CAPA, and STS • Lists subjects• % and # at each Performance Level
– Not for CMA, CAPA Science, STS• Mean Scale Score
– Not for CMA, CAPA Science, STS• Reporting Cluster: Mean Percent Correct
– Not for CMA, CAPA, and STS grades 5–7
Post-Test Guide
Pp 17; 26–37
August/September 2008 Post-Test Workshop 22
Student Master List SummaryStudent Master List SummaryPost-Test Guide
P 27
August/September 2008 Post-Test Workshop 23
Student Master List SummaryStudent Master List SummaryPost-Test Guide
P 27
August/September 2008 Post-Test Workshop 24
Who Counts?Who Counts?
• Number Enrolled– All CST, CMA, CAPA, and STS multiple-
choice answer documents submitted as scorable
– Minus• Documents marked as “Student
enrolled after the first day of testing and was given this test”
Post-Test Guide
P 28
August/September 2008 Post-Test Workshop 25
Who Counts?Who Counts?
• Number, Percent Tested– All CST, CMA, CAPA, STS answer documents with one
or more answers – Plus
• Z = Tested but marked no answers– Not included
• P = Parent/guardian exemptions• A = Students absent• T = Enrolled first day, not tested, tested at previous
school• E = Not tested due to significant medical emergency• Students with inconsistent grades
Post-Test Guide
P 28
August/September 2008 Post-Test Workshop 26
Who Counts? Who Counts?
• Number Valid Scores
– For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct).
– Not included:
• Incomplete tests
• Modified tests
• Non-English learners who took the STS
• Unknown EOC math or science tests
• Inconsistent grades
Why does STS not have 100% valid scores?
Post-Test Guide
P 27
August/September 2008 Post-Test Workshop 27
Who Counts? Who Counts?
• Number Tested with Scores– All tests taken, including those taken with
modifications, that receive a score – All STS and CMA tests taken– Not included
• Incomplete tests• Non-English learners who took the STS• Unknown EOC math or science tests• Inconsistent grades
Post-Test Guide
P 28
August/September 2008 Post-Test Workshop 28
Who Counts? Who Counts?
• Performance Level Counts for CSTs– Advanced, Proficient, Basic, Below Basic
• All valid scores falling in the performance level
– Far Below Basic• All valid scores falling in the performance
level • Modified tests (in aggregate reporting and
accountability only)
Post-Test Guide
P 29
August/September 2008 Post-Test Workshop 29
Student Master List SummaryStudent Master List SummaryPerformance LevelsPerformance Levels
Post-Test Guide
P 30
August/September 2008 Post-Test Workshop 30
Student Master List SummaryStudent Master List SummaryReporting ClustersReporting Clusters
Compare to:
Average percent correct range for students statewide who scored proficient on the total test
Post-Test Guide
P 30
August/September 2008 Post-Test Workshop 31
Student Master List Summary:Student Master List Summary:WritingWriting
B = BlankC = Copied PromptI = IllegibleL = Language other
than English
R = RefusalT = Off TopicW = Wrong prompt
(Prompt from an earlier administration)
Post-Test Guide
P 30
August/September 2008 Post-Test Workshop 32
Student Master List Student Master List Summary: End-of-CourseSummary: End-of-Course
• By Subject• Lists each grade eligible to take test • Math (Grades 7–11)
– General Math, Algebra I, Geometry, etc.• Science (Grades 9–11)
– Earth Science, Biology, Chemistry, etc.• History–Social Science (Grades 9–11)
– World History• Same statistics as grade-level Student Master
List Summary
Post-Test Guide
Pp 32–37
August/September 2008 Post-Test Workshop 33
Subgroup Summary:Subgroup Summary:CSTs, CMA, CAPA, and STSCSTs, CMA, CAPA, and STS
• Disability Status– Based on “Disability Code”
• Economic Status– Based on “NSLP” & “Parent Ed Level”
• Gender• English Proficiency• Primary Ethnicity• Ethnicity for Economic Status: CSTs and
CAPA (ELA and Math) only
Post-Test Guide
Pp 38–64
August/September 2008 Post-Test Workshop 34
Subgroup Summary: Subgroup Summary: Ethnicity for Economic Status Ethnicity for Economic Status
Post-Test Guide
Pp 60–64
August/September 2008 Post-Test Workshop 35
Subgroup Summary: Subgroup Summary: Ethnicity for Economic StatusEthnicity for Economic Status Post-Test
Guide Pp 60–64
August/September 2008 Post-Test Workshop 36
Subgroup Summary: Subgroup Summary: Ethnicity for Economically Ethnicity for Economically
DisadvantagedDisadvantagedPost-Test Guide
Pages 60–64
August/September 2008 Post-Test Workshop 37
Internet ReportsInternet Reports
• Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS
• Available to public online• More subgroups than paper reports
– English-language fluency– Parent education– Special Program Participation
• Access from http://star.cde.ca.gov
Post-Test Guide
Pp 123–134
August/September 2008 Post-Test Workshop 38
Internet Reports:Internet Reports:Available ReportsAvailable Reports
• CSTs– Mean scale scores– Percents by
performance levels• CST Summary
– Percent proficient and advanced
• CMA– Average percent
correct– Grades 3–5 only
• STS– Average percent
correct– Grades 2–7 only
• CAPA– Mean scale scores – Percents by
performance levels– Average percent
correct for science• CAT/6 Survey
– Mean scale scores– NPR for “Avg.
Student”– Percent above
percentile cuts– Grades 3 and 7 only
Post-Test Guide
Pp 123-134
August/September 2008 Post-Test Workshop 39
Internet ReportsInternet ReportsAccess ReportsAccess Reports
Post-Test Guide
Pp 124
August/September 2008 Post-Test Workshop 40
Internet Reports:Internet Reports:CST SampleCST Sample
Post-Test Guide
P 127
August/September 2008 Post-Test Workshop 41
Internet Reports:Internet Reports:CST Summary SampleCST Summary Sample
Post-Test Guide
P 128
August/September 2008 Post-Test Workshop 42
Internet Reports:Internet Reports:CMACMA
% Correct% Correct
Post-Test Guide
P 129
August/September 2008 Post-Test Workshop 43
Internet Reports:Internet Reports:STSSTS
% Correct% Correct
Post-Test Guide
Pp 133
August/September 2008 Post-Test Workshop 44
ExerciseExercise
• Work with a partner or in small groups
• Plan what data you’ll present to leaders in your district
• Remember the Do’s and Don’ts and Cautions
Break — Break — 15 minutes15 minutes
August/September 2008 Post-Test Workshop 46
Individual ReportsIndividual Reports• STAR Student Record Label
– Adhesive label to affix to student’s permanent school record
• STAR Student Master List– Alphabetical list of students and their scores– Tests listed in order within grade
• CSTs-CAT/6 Survey • CMA-CAT/6 Survey • CAPA • STS
• STAR Student Report: individual’s scores– 2 two-sided color copies– For parents/guardians– For each test
Post-Test Guide
Pp 70–112
August/September 2008 Post-Test Workshop 47
Student Record Label:Student Record Label:Student Name and Identification Student Name and Identification
Post-Test Guide
Pp 70–73
August/September 2008 Post-Test Workshop 48
Student Record Label:Student Record Label:CST for Grade 10 Results CST for Grade 10 Results
Post-Test Guide
Pp 70–73
August/September 2008 Post-Test Workshop 49
Student Master List:Student Master List:CSTs/CMA-CAT/6 for Grade 3 SampleCSTs/CMA-CAT/6 for Grade 3 Sample
Post-Test Guide
Pp 74–79
August/September 2008 Post-Test Workshop 50
Student Master List:Student Master List:CSTs/CMA-CAT/6 for Grade 3 SampleCSTs/CMA-CAT/6 for Grade 3 Sample
Post-Test Guide
Pp 74–79
August/September 2008 Post-Test Workshop 51
Student Report:Student Report:CST for Grade 11 SampleCST for Grade 11 Sample
Post-Test Guide
Pp 81–90
August/September 2008 Post-Test Workshop 52
Student Report:Student Report:CST for Grade 11 SampleCST for Grade 11 Sample
Post-Test Guide
Pp 81–90
August/September 2008 Post-Test Workshop 53
Student Report:Student Report:CSTs for Grade 10 SampleCSTs for Grade 10 Sample
Post-Test Guide
Pp 81–90
♦ = Percent correct obtained by the student on the reporting cluster/content area
▬▬▬= Range of percent correct scores on the reporting cluster for students who scored proficient on the total test
StudentNameOn Back
August/September 2008 Post-Test Workshop 54
Reporting Cluster RangeReporting Cluster Rangefor Each Content Area by Gradefor Each Content Area by Grade
Reporting cluster average percent correct from all minimally proficient scores (350)
Reporting cluster average percent correct from all minimally advanced scores minus one (e.g., 401, grade 4 math)
August/September 2008 Post-Test Workshop 55
Explain to ParentsExplain to ParentsScale Score Scale Score Average % correct cluster Average % correct cluster
score x 600score x 600
• Reporting Clusters not comparable– Different difficulty
– Varying number of questions
– Average % correct of clusters % correct of total scale score
• Scale scores– Use conversion tables and other statistical
techniques
– Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient)
August/September 2008 Post-Test Workshop 56
Student Report:Student Report:CST for Grade 7 with CAT/6 SampleCST for Grade 7 with CAT/6 Sample
Post-Test Guide
P 85
August/September 2008 Post-Test Workshop 57
Student Report:Student Report:CST for Grade 11 with EAP SampleCST for Grade 11 with EAP Sample
Post-Test Guide
Pp 81–90
August/September 2008 Post-Test Workshop 58
Other Student ReportsOther Student Reports
• CMA (Pp 91–98)
– Percent correct, no performance levels– Back, “About the CMA,” P 93– No cluster reporting
• CAPA (Pp 99–104)
– Back, performance level descriptors for ELA, math P 104
• STS (Pp 105–112)
– In Spanish– Percent correct, no performance levels– Reporting clusters percent correct, grades 2–4– Back, how to use report P 108
August/September 2008 Post-Test Workshop 59
ExerciseExercise
• Work with a partner or in small groups
• Use a Student Report sample and explain content reporting diamonds and bars on back
August/September 2008 Post-Test Workshop 60
Unmatched ReportUnmatched Report
• Grade 3– CST but no CAT/6 Survey– CMA but no CAT/6 Survey– CAT/6 Survey but no CSTs or CMA
• Grades 4 and 7– Unmatched Multiple-Choice Report
• CST Multiple-choice but no writing test– Unmatched Writing Report
• Writing but no CSTs multiple-choice test
August/September 2008 Post-Test Workshop 61
Unmatched Report:Unmatched Report:
August/September 2008 Post-Test Workshop 62
Reports for TeachersReports for Teachers
• Summary of results for previous year’s class(es)• Two types of summary reports
– By grade level (e.g., grade 11 ELA) or course (e.g., Algebra I)
– By teacher name or group name (e.g., GATE)• Separate reports for:
– CST ELA– CST Math
• Not available for:– CST science or history–social science– CMA – CAPA– STS
Post-Test Guide
Pp 113–122
August/September 2008 Post-Test Workshop 63
Reports for TeachersReports for Teachers
• Individualized reports (teacher name or group name)– if name on SGID or Pre-ID– SGID priority over Pre-ID
• Group reports (grade level or course): all ELA and math teachers should receive
• EOC Math– Grades 8–11 by subject only– Not grade: e.g., All Algebra I in a school
Post-Test Guide
Pp 113 – 122
August/September 2008 Post-Test Workshop 64
Report for Teachers:Report for Teachers:SampleSample
Post-Test Guide
Pp 114–122
August/September 2008 Post-Test Workshop 65
Report for Teachers:Report for Teachers:2008 Performance2008 Performance
By Reporting ClustersBy Reporting Clusters
Reporting clusters defined under this section.
Post-Test Guide
Pp 114
August/September 2008 Post-Test Workshop 66
Report for Teachers:Report for Teachers:Cluster ReportingCluster Reporting
• For each reporting cluster, the average percent correct achieved by:– Statewide students– District students– Your students– Proficient range
• Definition of symbols = average % correct for teacher’s students
▬▬▬ = average % correct range for students statewide who scored proficient on total test
August/September 2008 Post-Test Workshop 67
Report for TeachersReport for TeachersBackBack
• School % at each performance level by subgroups
• % Proficient and above– 2007 vs. 2008– School; district; state– By subgroup
• Resources
Post-Test Guide P 117
August/September 2008 Post-Test Workshop 68
Early Assessment Program (EAP)Early Assessment Program (EAP)
• Reporting Results – Student Statuses are on the STAR Report– Ready for CSU college-level English/mathematics
courses– Ready for CSU college-level mathematics courses –
Conditional• This status in conjunction with a grade of “C” or better in
an approved math course or a supervised e-learning program can be exempt from the ELM
• A list of courses and e-learning programs can be found at www.csumathsuccess.org
– Not yet demonstrating readiness for CSU college-level English/mathematics courses
August/September 2008 Post-Test Workshop 69
SummarySummary
• Do’s and Don’ts – Do compare mean scale score, percent at
performance levels within same grade, same content area
– Don’t compare across grades, content area• Summary reports• Individual reports• Teacher reports• Quiz answers• Evaluations
August/September 2008 Post-Test Workshop 70
Quiz AnswersQuiz Answers
• 1. B
• 2. D
• 3. A
• 4. C
• 5. D