2008 star interpreting and using results august/september 2008

2008 STAR2008 STARInterpreting and Using Interpreting and Using

ResultsResultsAugust/September 2008

August/September 2008 Post-Test Workshop 2

ObjectivesObjectives

Workshop participants will be able to:

• Describe the purposes of STAR reports

• Interpret STAR results

• Explain key statistics

• Compare and contrast types of reports

• Identify proper uses of reports


AgendaAgenda

• Quiz• Post Test Guide• What’s New?• Results and Statistical Analysis• Using Results• Summary and Internet Reports• Individual Student Reports• Teacher Reports


Post-Test Guide LayoutPost-Test Guide Layout

• Part I: General Info– New in 2008– Interpreting Reports– Comparing Results

• Part II: Report Descriptions– Summary– Individual– Teacher– Internet

• Part III: Appendices– Reporting clusters– Scale score ranges– Statistical tables– Test Score Verification

procedures– Demographic Data

Correction procedures– California reading list– Glossary

Post-Test Guide

Pages (Pp) i–iii


What’s New in 2008What’s New in 2008

• CMA reports– Percent correct– Grades 3–5

• STS reports– Percent correct– Grades 5–7 added

• CAPA Science percent correct• Ethnicity for Economic Status subgroup

summary (CSTs only)• STAR Student Reports sorted for district

– All without addresses– All with addresses

Post-Test Guide

P 2


Results: Results: Purposes of STAR ReportsPurposes of STAR Reports

• Report progress of students toward proficiency on the state’s academic content standards

• Inform where improvement needed– To help students’ achievement– To improve educational programs

• Provide data for state and federal accountability programs

Post-Test Guide

P 3


Results: Performance LevelsResults: Performance Levels

• State goal: PROFICIENT or higher– Scale score of 350 or higher for CSTs– Scale score of 35 or higher for CAPA

• Other performance levels:– Advanced– Basic (300 CSTs, 30 CAPA)– Below Basic– Far Below Basic– CST and CAPA cut points vary for

Advanced, Below Basic, and Far Below Basic by

• Subject• Grade

Post-Test Guide

P 7


Results: Scale ScoresResults: Scale Scores

• Scale scores allow same score to mean same thing across test forms within grade and content area and across time

• Scale score ranges by program:– CSTs: 150–600 for each grade and subject– CAT/6 Survey: 1–999 for each grade and

subject– CAPA: 15–60 for each grade and subject– CMA, STS, CAPA Science: Not available until

2009

Post-Test Guide P 7–8


Results: Equating Results: Equating

• Psychometric procedure

• Adjusts for test difficulty • Additional information in CST Technical Report

on CDE Web site• For 2008

– CSTs and CAPA (except science) are equated– CMA (3–5), CAPA (science), and STS (2–4) and

will be equated starting in 2009• CAT/6 nationally norm-referenced; not equated

for STAR

Post-Test Guide Page 7


Results: Reporting ClustersResults: Reporting Clusters(Content Areas)(Content Areas)

• Four to six clusters for each content area (subject)

• May be useful as indicators of individual or group strengths and weaknesses

• But… Reporting clusters should be interpreted with caution

Post-Test Guide

P 36

Appendices A & B


Results: Reporting ClusterResults: Reporting ClusterCautionsCautions

• Based on small numbers of items; therefore may not be reliable or generalizable

• NOT equated from year to year

• Should not compare reporting cluster percent correct from year to year

Post-Test Guide

P 8–10

Appendices A & B


Interpreting Reporting Clusters or Interpreting Reporting Clusters or Content Areas Content Areas

in the in the Same YearSame Year• CSTs—Performance on the clusters for

students statewide scoring from– Lowest score for proficient to– Lowest score for advanced

Post-Test Guide

P 8–10; 84

Appendices A & B


CSTs 2008 Reporting Clusters — Number of CSTs 2008 Reporting Clusters — Number of Questions and Average Percent CorrectQuestions and Average Percent CorrectPost-Test Guide

8(Appendix A)


ExamplesExamples—Interpreting Reporting —Interpreting Reporting Clusters for CST for Geometry Clusters for CST for Geometry Post-Test Guide

P 9

0%10%20%30%40%50%60%70%80%90%

100%

Logic/GeomProofs

Volume & Area AngleRelationships

Trigonometry

Avg

% C

orr

ect

My Class 2007 Just Proficient Students 2007 Just Advanced Students 2007

0%10%20%30%40%50%60%70%80%90%

100%

Logic/GeomProofs

Volume & Area AngleRelationships

Trigonometry

Av

g %

Co

rre

ct

My Class 2008 Just Proficient Students 2008 Just Advanced Students 2008


Using ResultsUsing Results

• For instructional decisions with other data: CSTs, CMA, CAPA, STS

• Used in Academic Performance Index (API) calculations: CSTs, CAT/6 Survey, CAPA

• Used in adequate yearly progress (AYP) calculations for grades 2–8 in ELA and math for CSTs and 2–8 and 10 in ELA and math for CAPA

Post-Test Guide

P 3


Year-to-Year ComparisonsYear-to-Year ComparisonsDoDo Compare Same Grade and Same Compare Same Grade and Same

Content AreaContent Area

• Mean scale score – Same content and grade, varying years

• Percent in each performance level– Same content by grade across years

• e.g., 2007 ELA grade 10 with 2008 ELA grade 10

Post-Test Guide

Pp 11–14


Year-to-Year ComparisonsYear-to-Year ComparisonsDoDo Compare Same Grade or Same Compare Same Grade or Same

Content AreaContent Area

• Percentage of students scoring at PROFICIENT and above– For a given grade and subject, e.g., Percent

proficient and above for grade 3 math in 2007 & 2008

– For a given subject and aggregated grades, e.g., % proficient and above for grades 2 to 6 math in 2007 & 2008

– Across grades and a subject, e.g., % proficient and above in all courses and all grades

Post-Test Guide

Pp 11–14


Year-to-Year ComparisonsYear-to-Year ComparisonsDON’TDON’T Compare Compare

• Individual scale scores or statistics based on scale scores for different grades or content areas– Subjects by grade are independently scaled – Different content standards are measured in

different grades

• Cohorts across grades

• CMA, STS scores because notscaled

Post-Test Guide

Pp 11–14


ExampleExample—Using CST Results to —Using CST Results to Compare Grade Results from Year to Compare Grade Results from Year to

YearYear

Post-Test Guide

P 12

2007 CST for ELA

2008 CST for ELA

Grade % Prof or Above

% Prof or Above

Difference

Grade 2 31% 35% 4%

Grade 3 33% 33% 0%

Grade 4 29% 31% 2%

Grade 5 34% 32% -2%

Grade 6 31% 32% 1%

All Grades

32% 33% 1%


Aggregate (Summary) ReportsAggregate (Summary) Reports

• What are they?– Student Master List Summary– Student Master List Summary EOC– Subgroup Summary– Group Summary: CAT/6 Survey

• Report Emphasis: California Standards Tests (CSTs) – Criterion-referenced tests– Progress is measured in percent of students

scoring proficient and advanced

Post-Test Guide

Pp 17–21


Student Master List SummaryStudent Master List Summary

• By grade• CSTs, CMA, CAPA, and STS • Lists subjects• % and # at each Performance Level

– Not for CMA, CAPA Science, STS• Mean Scale Score

– Not for CMA, CAPA Science, STS• Reporting Cluster: Mean Percent Correct

– Not for CMA, CAPA, and STS grades 5–7

Post-Test Guide

Pp 17; 26–37


Student Master List SummaryStudent Master List SummaryPost-Test Guide

P 27


Who Counts?Who Counts?

• Number Enrolled– All CST, CMA, CAPA, and STS multiple-

choice answer documents submitted as scorable

– Minus• Documents marked as “Student

enrolled after the first day of testing and was given this test”

Post-Test Guide

P 28


Who Counts?Who Counts?

• Number, Percent Tested– All CST, CMA, CAPA, STS answer documents with one

or more answers – Plus

• Z = Tested but marked no answers– Not included

• P = Parent/guardian exemptions• A = Students absent• T = Enrolled first day, not tested, tested at previous

school• E = Not tested due to significant medical emergency• Students with inconsistent grades

Post-Test Guide

P 28


Who Counts? Who Counts?

• Number Valid Scores

– For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct).

– Not included:

• Incomplete tests

• Modified tests

• Non-English learners who took the STS

• Unknown EOC math or science tests

• Inconsistent grades

Why does STS not have 100% valid scores?

Post-Test Guide

P 27



• Number Tested with Scores– All tests taken, including those taken with

modifications, that receive a score – All STS and CMA tests taken– Not included

• Incomplete tests• Non-English learners who took the STS• Unknown EOC math or science tests• Inconsistent grades

Post-Test Guide

P 28



• Performance Level Counts for CSTs– Advanced, Proficient, Basic, Below Basic

• All valid scores falling in the performance level

– Far Below Basic• All valid scores falling in the performance

level • Modified tests (in aggregate reporting and

accountability only)

Post-Test Guide

P 29


Student Master List SummaryStudent Master List SummaryPerformance LevelsPerformance Levels

Post-Test Guide

P 30


Student Master List SummaryStudent Master List SummaryReporting ClustersReporting Clusters

Compare to:

Average percent correct range for students statewide who scored proficient on the total test

Post-Test Guide

P 30


Student Master List Summary:Student Master List Summary:WritingWriting

B = BlankC = Copied PromptI = IllegibleL = Language other

than English

R = RefusalT = Off TopicW = Wrong prompt

(Prompt from an earlier administration)

Post-Test Guide

P 30


Student Master List Student Master List Summary: End-of-CourseSummary: End-of-Course

• By Subject• Lists each grade eligible to take test • Math (Grades 7–11)

– General Math, Algebra I, Geometry, etc.• Science (Grades 9–11)

– Earth Science, Biology, Chemistry, etc.• History–Social Science (Grades 9–11)

– World History• Same statistics as grade-level Student Master

List Summary

Post-Test Guide

Pp 32–37


Subgroup Summary:Subgroup Summary:CSTs, CMA, CAPA, and STSCSTs, CMA, CAPA, and STS

• Disability Status– Based on “Disability Code”

• Economic Status– Based on “NSLP” & “Parent Ed Level”

• Gender• English Proficiency• Primary Ethnicity• Ethnicity for Economic Status: CSTs and

CAPA (ELA and Math) only

Post-Test Guide

Pp 38–64


Subgroup Summary: Subgroup Summary: Ethnicity for Economic Status Ethnicity for Economic Status

Post-Test Guide

Pp 60–64


Subgroup Summary: Subgroup Summary: Ethnicity for Economic StatusEthnicity for Economic Status Post-Test

Guide Pp 60–64


Subgroup Summary: Subgroup Summary: Ethnicity for Economically Ethnicity for Economically

DisadvantagedDisadvantagedPost-Test Guide

Pages 60–64


Internet ReportsInternet Reports

• Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS

• Available to public online• More subgroups than paper reports

– English-language fluency– Parent education– Special Program Participation

• Access from http://star.cde.ca.gov

Post-Test Guide

Pp 123–134


Internet Reports:Internet Reports:Available ReportsAvailable Reports

• CSTs– Mean scale scores– Percents by

performance levels• CST Summary

– Percent proficient and advanced

• CMA– Average percent

correct– Grades 3–5 only

• STS– Average percent

correct– Grades 2–7 only

• CAPA– Mean scale scores – Percents by

performance levels– Average percent

correct for science• CAT/6 Survey

– Mean scale scores– NPR for “Avg.

Student”– Percent above

percentile cuts– Grades 3 and 7 only

Post-Test Guide

Pp 123-134


Internet ReportsInternet ReportsAccess ReportsAccess Reports

Post-Test Guide

Pp 124


Internet Reports:Internet Reports:CST SampleCST Sample

Post-Test Guide

P 127


Internet Reports:Internet Reports:CST Summary SampleCST Summary Sample

Post-Test Guide

P 128


Internet Reports:Internet Reports:CMACMA

% Correct% Correct

Post-Test Guide

P 129


Internet Reports:Internet Reports:STSSTS

% Correct% Correct

Post-Test Guide

Pp 133


ExerciseExercise

• Work with a partner or in small groups

• Plan what data you’ll present to leaders in your district

• Remember the Do’s and Don’ts and Cautions

Break — Break — 15 minutes15 minutes


Individual ReportsIndividual Reports• STAR Student Record Label

– Adhesive label to affix to student’s permanent school record

• STAR Student Master List– Alphabetical list of students and their scores– Tests listed in order within grade

• CSTs-CAT/6 Survey • CMA-CAT/6 Survey • CAPA • STS

• STAR Student Report: individual’s scores– 2 two-sided color copies– For parents/guardians– For each test

Post-Test Guide

Pp 70–112


Student Record Label:Student Record Label:Student Name and Identification Student Name and Identification

Post-Test Guide

Pp 70–73


Student Record Label:Student Record Label:CST for Grade 10 Results CST for Grade 10 Results

Post-Test Guide

Pp 70–73


Student Master List:Student Master List:CSTs/CMA-CAT/6 for Grade 3 SampleCSTs/CMA-CAT/6 for Grade 3 Sample

Post-Test Guide

Pp 74–79


Student Report:Student Report:CST for Grade 11 SampleCST for Grade 11 Sample

Post-Test Guide

Pp 81–90


Student Report:Student Report:CSTs for Grade 10 SampleCSTs for Grade 10 Sample

Post-Test Guide

Pp 81–90

♦ = Percent correct obtained by the student on the reporting cluster/content area

▬▬▬= Range of percent correct scores on the reporting cluster for students who scored proficient on the total test

StudentNameOn Back


Reporting Cluster RangeReporting Cluster Rangefor Each Content Area by Gradefor Each Content Area by Grade

Reporting cluster average percent correct from all minimally proficient scores (350)

Reporting cluster average percent correct from all minimally advanced scores minus one (e.g., 401, grade 4 math)


Explain to ParentsExplain to ParentsScale Score Scale Score Average % correct cluster Average % correct cluster

score x 600score x 600

• Reporting Clusters not comparable– Different difficulty

– Varying number of questions

– Average % correct of clusters % correct of total scale score

• Scale scores– Use conversion tables and other statistical

techniques

– Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient)


Student Report:Student Report:CST for Grade 7 with CAT/6 SampleCST for Grade 7 with CAT/6 Sample

Post-Test Guide

P 85


Student Report:Student Report:CST for Grade 11 with EAP SampleCST for Grade 11 with EAP Sample

Post-Test Guide

Pp 81–90


Other Student ReportsOther Student Reports

• CMA (Pp 91–98)

– Percent correct, no performance levels– Back, “About the CMA,” P 93– No cluster reporting

• CAPA (Pp 99–104)

– Back, performance level descriptors for ELA, math P 104

• STS (Pp 105–112)

– In Spanish– Percent correct, no performance levels– Reporting clusters percent correct, grades 2–4– Back, how to use report P 108


ExerciseExercise

• Work with a partner or in small groups

• Use a Student Report sample and explain content reporting diamonds and bars on back


Unmatched ReportUnmatched Report

• Grade 3– CST but no CAT/6 Survey– CMA but no CAT/6 Survey– CAT/6 Survey but no CSTs or CMA

• Grades 4 and 7– Unmatched Multiple-Choice Report

• CST Multiple-choice but no writing test– Unmatched Writing Report

• Writing but no CSTs multiple-choice test


Unmatched Report:Unmatched Report:


Reports for TeachersReports for Teachers

• Summary of results for previous year’s class(es)• Two types of summary reports

– By grade level (e.g., grade 11 ELA) or course (e.g., Algebra I)

– By teacher name or group name (e.g., GATE)• Separate reports for:

– CST ELA– CST Math

• Not available for:– CST science or history–social science– CMA – CAPA– STS

Post-Test Guide

Pp 113–122


Reports for TeachersReports for Teachers

• Individualized reports (teacher name or group name)– if name on SGID or Pre-ID– SGID priority over Pre-ID

• Group reports (grade level or course): all ELA and math teachers should receive

• EOC Math– Grades 8–11 by subject only– Not grade: e.g., All Algebra I in a school

Post-Test Guide

Pp 113 – 122


Report for Teachers:Report for Teachers:SampleSample

Post-Test Guide

Pp 114–122


Report for Teachers:Report for Teachers:2008 Performance2008 Performance

By Reporting ClustersBy Reporting Clusters

Reporting clusters defined under this section.

Post-Test Guide

Pp 114


Report for Teachers:Report for Teachers:Cluster ReportingCluster Reporting

• For each reporting cluster, the average percent correct achieved by:– Statewide students– District students– Your students– Proficient range

• Definition of symbols = average % correct for teacher’s students

▬▬▬ = average % correct range for students statewide who scored proficient on total test


Report for TeachersReport for TeachersBackBack

• School % at each performance level by subgroups

• % Proficient and above– 2007 vs. 2008– School; district; state– By subgroup

• Resources

Post-Test Guide P 117


Early Assessment Program (EAP)Early Assessment Program (EAP)

• Reporting Results – Student Statuses are on the STAR Report– Ready for CSU college-level English/mathematics

courses– Ready for CSU college-level mathematics courses –

Conditional• This status in conjunction with a grade of “C” or better in

an approved math course or a supervised e-learning program can be exempt from the ELM

• A list of courses and e-learning programs can be found at www.csumathsuccess.org

– Not yet demonstrating readiness for CSU college-level English/mathematics courses


SummarySummary

• Do’s and Don’ts – Do compare mean scale score, percent at

performance levels within same grade, same content area

– Don’t compare across grades, content area• Summary reports• Individual reports• Teacher reports• Quiz answers• Evaluations


Quiz AnswersQuiz Answers

• 1. B

• 2. D

• 3. A

• 4. C

• 5. D

2008 star interpreting and using results august/september 2008

Documents

results augustseptember

posttest workshop11

posttest workshop7 results

posttest workshop10

posttest workshop8 results

posttest workshop6 results

posttest workshop9 results

addresses posttest guide