2013 star interpreting and using results august 7, 2013 webcast webcast starts at 9 a.m

2013 STARInterpreting and Using

ResultsAugust 7, 2013

WebcastWebcast starts at 9 a.m.

Objectives

Workshop participants will be able to:• Describe the purposes of STAR reports• Interpret STAR results• Explain key statistics• Compare and contrast types of reports• Identify proper uses of reports

August 2013 Post-Test Workshop2

August 2013 Post-Test Workshop

• What’s New?• Results and Statistical Analysis• Using Results• Summary and Internet Reports• Data CDs• Individual Student Reports• Early Assessment Program

Agenda

3


• Standards-based Tests in Spanish (STS) performance levels are now also reported for students in grades 8–11who took the grade-level STS for RLA, STS for Algebra I, and STS for Geometry.

• The score for the California Standards Test (CST) for Writing is no longer doubled; possible scores are 1, 2, 3, and 4.

Manual (M) 2 – 3

What’s New in 2013


• ELA scale scores for the grades 4 and 7 CSTs and California Modified Assessment (CMA) will be provided based on the multiple-choice items only and the reporting cluster “Writing Applications” is no longer part of the CST and CMA ELA cluster groups; instead, the writing score is provided as a standalone score called “Writing Response Score.”

M 2 – 3



• Enrollment and exit code data to determine which students are counted as “continuously enrolled” for accountability purposes, previously collected for (CBEDS), were not collected on STAR answer documents or in Pre-ID. Instead, these data were extracted from the California Longitudinal Pupil Achievement Data System (CALPADS).• Please refer to pages 2 and 3 in the 2013 Post Test

Guide for a list of all changes.


M 2 – 3

Which of these tests had scale scores reported for the first time in 2013?

A.CST for World History

B.STS for Science

C.CMA for Algebra I

D.STS for Algebra I


Quiz Question 1


Which of these tests had scale scores reported for the first time in 2013?

A.CST for World History

B.CAPA for Science

C.CMA for Algebra ID.STS for Algebra I

Quiz Question 1


• Report progress toward proficiency on the state’s academic content standards

• Notify where improvement needed– To help students’ achievement– To improve educational programs

• Provide data for state and federal accountability programs

Results: Purposes of STAR Reports

M 4


• State goal: All students score at proficient or above • 350 or higher scale score

– CST– CMA– STS

• CAPA proficient: 35 or higher scale score

Results: Performance Levels

M 8 – 11


• Advanced• Basic cut score

– CST: 300 – CMA: 300– STS: 300– CAPA: 30

• Below basic• Far below basic• For each testing program, cut points

vary for advanced and below basic by – Subject– Grade

Results: Other Performance Levels

M 8 – 11;Appendix B


• Scale scores allow the same score to mean the same thing across test versions within grade and content area.

• Scale scores account for differences in difficulty.• Scale score ranges by program are:

– CST, CMA, STS: 150–600 for each grade and subject

– CAPA: 15–60 for each level and subject

Results: Scale Scores

M 8 – 11


• Psychometric procedure • Adjusts for test difficulty • Additional information in the technical report on

the CDE Web site

Results: Equating

M 8


• Three to six clusters for each subject

• May be useful as indicators of individual or group strengths and weaknesses

• But. . . reporting clusters should be interpreted with caution

Results: Reporting Clusters (Content Area)

M 9 – 11;Appendix A


• Cluster percent correct available for CSTs, CMA, STS

• Based on small numbers of items; therefore, may not be reliable or generalized

• NOT equated from year to year• Should not compare reporting cluster percent

correct from year to year

Results: Reporting ClustersCautions



• Compare to percent-correct range of proficient students statewide

Interpreting Reporting Clusters or Content Areas in the Same Year



2013 Post-Test Guide, Appendixes A and C as posted on startest.org, will be finalized with complete data on August 15.

2013 CST Reporting Clusters:Number of Questions and Average Percent Correct

M Appendix A


0%

20%

40%

60%

80%

100%

Logic/Geom Proofs

Volume & Area Angle Relationships

Trigonometry

Avg

% C

orre

ct

My Class 2012 Just Proficient Students 2012 Just Advanced Students 2012

0%

20%

40%

60%

80%

100%

Logic/Geom Proofs

Volume & Area Angle Relationships

Trigonometry

Avg

% C

orre

ct

My Class 2013 Just Proficient Students 2013 Just Advanced Students 2013

Examples—Interpreting Reporting Clusters for the CST for Geometry

M 10

What is a scale score?

A.Percent correct of all questions

B.Mean percent correct of all questions

C.An adjustment of this year’s and last year’s raw scores to show changes

D.An adjustment of the raw score to account for differences in difficulty


Quiz Question 2


What is a scale score?

A.Percent correct of all questions

B.Mean percent correct of all questions

C.An adjustment of this year’s and last year’s raw scores to show changes

D.An adjustment of the raw score to account for differences in difficulty

Quiz Question 2


• For instructional decisions in conjunction with other data

• Used in Academic Performance Index (API) calculations, all grades and subjects: CST, CMA, CAPA

• Used in adequate yearly progress (AYP) calculations, ELA and mathematics:– CST — grades 2–8– CMA — grades 3–8 – CAPA — grades 2–8, 10

Using Results

M 4


• Mean scale score – Same content and grade, varying years

• Percent in each performance level– Same content by grade across years e.g., 2012 grade 10 ELA with 2013 grade 10

ELA

Year-to-Year ComparisonsDo Compare CSTs: Same Grade

and Same Content Area

M 12 – 15


• Percentage of students scoring at PROFICIENT and above– For a given grade and subject, e.g., Percent

proficient and above for grade 3 math in 2012 and 2013

– For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2–6 mathematics in 2012 and 2013

– Across grades and a subject, e.g., Percent proficient and above in all courses and all grades

Year-to-Year ComparisonsDo Compare CSTs: Percent Proficient and Advanced

M 12 – 15


• Individual scale scores or statistics based on scale scores for different grades or content areas– Subjects by grade are independently scaled – Different content standards are measured in

different grades• Cohorts across grades• Across tests• Scale scores to percent correct scores • CAPA to years prior to 2009, due to new standard

setting then

Year-to-Year ComparisonsDon’t Compare

M 12 – 15


Example—Using CST Results to Compare Grade Results from Year to

Year

M 14

2012 CST for ELA 2013 CST for ELA

Grade % Prof or Above % Prof or Above Difference

Grade 2 31% 35% 4%

Grade 3 33% 33% 0%

Grade 4 29% 31% 2%

Grade 5 34% 32% –2%

Grade 6 31% 32% 1%

All Grades 32% 33% 1%

Which is the best comparison for CST scores of students within a middle school?

A.2012 mean scale scores for ELA of a cohort of grade 7 students with 2013 scale scores for ELA of the same students in grade 8

B.2012 mean scale scores for ELA for grade 8 students with 2013 mean scale scores for ELA for grade 8 students

C.2012 mean percent correct scores for ELA with 2013 mean percent correct scores for ELA for the same students

D.2012 mean percent correct scores for ELA for grade 8 students with 2013 mean percent correct for ELA for grade 8 students


Quiz Question 3


Quiz Question 3

Which is the best comparison for CST scores of students within a middle school?

A.2012 mean scale scores for ELA of a cohort of grade 7 students with 2013 scale scores for ELA of the same students in grade 8

B.2012 mean scale scores for ELA for grade 8 students with 2013 mean scale scores for ELA for grade 8 students

C.2012 mean percent correct scores for ELA with 2013 mean percent correct scores for ELA for the same students

D.2012 mean percent correct scores for ELA for grade 8 students with 2013 mean percent correct for ELA for grade 8 students

Which is the best comparison of cluster scores for a single student? Compare. . .

A.To proficient students statewide

B.One cluster to another, same year

C.The same cluster to the same cluster, different years

D.To the average percent correct of all students in a class


Quiz Question 4


Which is the best comparison of cluster scores for a single student? Compare. . .

A.To proficient students statewide

B.One cluster to another, same year

C.The same cluster to the same cluster, different years

D.To the average percent correct of all students in a class

Quiz Question 4


• What are they?– Student Master List Summary– Student Master List Summary End-of-Course (EOC)– Subgroup Summary

• Report emphasis: CSTs – Criterion-referenced tests– Progress is measured in percent of students scoring

proficient and advanced• Note: Back of reports provides guide to

abbreviations, score codes

Aggregate (Summary) Reports

M 18 – 22


• By grade• Results by program (CSTs, CMA, CAPA, and STS)

and subject • # and % at each performance level• Mean scale score• Reporting cluster: mean percent correct (except

CAPA)

Student Master List Summary

M 18–19;M 27 – 33


• By program (CSTs, CMA, and STS) and subject

• Results for each grade, and for all grades combined

• # and % at each performance level• Mean scale score• Reporting cluster: mean percent correct

Student Master List Summary End-of-Course

M 19–20;M 34 – 39


Grade 7 Example

Student Master List Summary

M 33


Student Master List SummaryBasic Statistics

M 28 – 31


• Total CST/CMA and CAPA multiple-choice answer documents submitted as scorable

• Minus– Documents marked as “Student enrolled after the

first day of testing and was given this test”

Who Counts? Number Enrolled

M 29


• All CST, CMA, CAPA, STS answer documents with one or more answers

• Plus– Z = Tested but marked no answers

• Not included– A = Students absent– E = Not tested due to significant medical emergency– P = Parent/guardian exemptions– T = Enrolled first day, not tested, tested at previous school– Students with inconsistent grades– Non–English learners who took the STS

Who Counts? Number Tested

M 29


• Number Valid Scores– For the subject, number of students tested at grade level who

received a score for the test.– Not included:

Incomplete tests Modified tests Non–English learners who took the STS Unknown EOC mathematics (except grade 7

mathematics) or science tests Inconsistent grades

• Percent Valid Scores– For the subject, number of valid scores divided by the number of

students tested.

Who Counts? Number and Percent Valid Scores

M 29


• All tests taken, including those taken with modifications, that result in a score

• Not included:– Incomplete tests– Non–English learners who took the STS– Unknown EOC mathematics or science tests– Inconsistent grades

Who Counts? Number Tested with Scores

M 29


Student Master List SummaryPerformance Levels

M 30


• All CSTs, CAPA, CMA, STS

• Advanced, proficient, basic, below basic– All valid scores falling in the performance level

• Far below basic– All valid scores falling in the performance level

– CSTs taken with modifications (in aggregate reporting [CSTs and STS] and accountability [CSTs] only)

Who Counts? Performance Levels

M 30

• Average of valid scale scores• Can be used to compare results for the same

content/grade across years


Who Counts? Mean Scale Scores

M 30


Compare to:

Average percent correct range for students statewide who scored proficient on the total test (See the Post-Test Guide, Appendix A.)

Student Master List Summary: Reporting Clusters

M 30


B = BlankC = Copied promptI = IllegibleL = Language

other than English

R = RefusalT = Off topicW = Wrong prompt

(prompt from an earlier administration)

Student Master List Summary: Writing

M 30 – 31


• Disability status– Based on disability status for CST, CMA, STS– CAPA: each disability type

• Economic status– Based on NSLP eligibility or parent education level

• Gender• English proficiency• Ethnicity• Ethnicity for Economic Status (only for CSTs, CMA,

and CAPA)

Subgroup Summary:CSTs, CMA, CAPA, and STS

M 40 – 54


Subgroup Summary:Ethnicity for Economic Status

M 51 – 54


Example: Economically disadvantaged for each ethnicity


M 51 – 54



M 51 – 54

Break — 10 minutes10 minutes


• Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS

• Available to the public online for school, district, county, and state

• “Students with Scores” = number tested with scores

• CST summaries of % advanced and proficient• More subgroups than paper reports

– Parent education– Special program participation

• Access from http://star.cde.ca.gov/

M 95 – 108

Internet Reports

Internet Demonstration


August 2013 Post-Test Workshop51 M 95 – 106

Internet Reports:CST Sample


Only available for CSTs

M 99 – 100

Internet Reports:CST Summary Sample

• CST (M 98−100)

• CMA (M 100–101)– Same as CST

• CAPA (M 101−104)– State level: same as CST; separate Level I

– County, district, school Mean scale score Percent proficient or above

• STS (M 105−106)– Same as CST


Other Internet Reports

Which subgroup can only be accessed from the Internet?

A.Parent Education Level

B.CAPA by individual disability status

C.Ethnicity for Economic Status

D.English Proficiency


Quiz Question 5


Which subgroup can only be accessed from the Internet?

A.Parent Education Level

B.CAPA by individual disability status

C.Ethnicity for Economic Status

D.English Proficiency

Quiz Question 5


• What are they? – Lists of information from answer documents and scores

of every student in the district

– In .txt format

• What are they used for?– Searching for specific data

– Creating unique reports

– Verifying paper reports

• What else is needed?– Text editor

– or Desktop application

– or Student Information System

Data CDs


• As .txt, word wrap on

• With text editor, word wrap off

View of Data


• Two files:– Demographics, special conditions, and test scores

– Accommodations, modifications, English learners, and irregularities

• Data Layout = guide to location of data on files– Position

– Number of characters

– Whether numeric or alpha

Organization of Data


Data Layout Sample


• STAR Student Record Label– Adhesive label to affix to student’s permanent school record

• STAR Student Master List– Alphabetical list of students and their scores– Tests listed in order within grade

CSTs CMA CAPA STS

• STAR Student Report: individual’s scores– 2 two-sided color copies for each test– For parents/guardians, school– Per regulations, district must forward within 20 business days

M 55 – 93

Individual Reports

August 2013 Post-Test Workshop61 M 56

Student Record Label Grade 10 Sample: Student Name and Identification


CST/CMA Grade 10 Example

M 56

Student Record Label


CST/CMA Grade 3 Example

M 57 – 60

Student Master List


CST Grade 11 Example

M 61 – 93;71 – 72

Student Report


CST Grade 11 Example

M 70

Student Report


♦ = Percent correct obtained by the student on the reporting cluster/content area

▬ = Average percent-correct range on the reporting cluster for students statewide who scored proficient on the total test scores

CST Grade 11 ExampleStudent’s name on back

M 71

Student Report


Explain to ParentsScale Score Average % correct

cluster score x 600

• Reporting clusters not comparable– Different difficulty

– Varying number of questions

– Average % correct of clusters % correct on total test (or total number of test items)

• Scale scores– Use conversion tables resulting from statistical procedures

– Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient)

M 8 – 11


• CMA (M 72−80)– Front: Performance levels and scale scores– Back: Cluster percent correct, About the CMA

• CAPA (M 81−85)– Front: Performance levels and scale scores– Back: About the CAPA

• STS (M 86−93)– In Spanish Front: Performance levels and scale scores Back: Cluster percent correct, how to use the report

Other Student Reports


Unmatched ReportsWriting CST, CMA—Grs 4, 7

• Only multiple-choice

• Only writing

• Students receive 2 reports if writing score not matched to multiple-choice score

CST, CMA—Grs 8, 11

• Only CMA and no CST

• Grades 8 and 11 require CST for History–Social Science


• Do’s– Do compare mean scale score, percent at performance

levels within same grade, same content area– Do compare cluster scores to the corresponding

proficient range provided for this year • Don’ts

– Don’t compare cluster scores to each other within test, nor across years, grades, content areas

– Don’t compare mean scale scores across grades, content areas

• Summary and Internet reports• Data CDs• Individual reports

Summary


Upcoming Dates

Event Date

Post Test Guide posted on startest.org July 24State Superintendent's public announcement August 8Appendixes A & C of the Post-Test Guide posted on startest.org August 8Web Reports available August 8Research files available August 8Post-Test Guide distributed August 15Data Corrections window September 16 through October 25

1. Return the 2014 Superintendent’s Designation form and Security Agreement.

2. Finalize instruction schedule.

3. Set up admin in STAR management system prior to December 1.

4. Enter and approve orders prior to December 1.

After order approval, test administration dates are NOT changeable.


2014 Setting Up Administration


• http://www.cde.ca.gov/ta/tg/sr/• http://star.cde.ca.gov/• http://www.startest.org/ • STAR Technical Assistance Center

– 800-955-2954• CDE Accountability

– [email protected] – 916-319-0863 • CDE STAR office: 916-445-8765

For more information see:


Next Webcasts

2013 Demographic Data CorrectionsSeptember 25, 2013, 9 a.m.

2014 STAR for Students with Disabilities Webcast

September 25, 2013, 1 p.m.

Early Assessment Program Early Assessment Program ResultsResults

August 7, 2013Connie Grueter

• EAP results are being accepted at participating California Community Colleges (CCC).

– List of participating colleges is available on the CCC website at www.CollegeEAP.org. Check the site often for an updated list of campuses.


PowerPoint slides – www.CollegeEAP.org, Educators, About EAP.

Early Assessment Program (EAP)

• An EAP informational web site has been developed: www.CollegeEAP.org.

– The site provides EAP information for both CSU and CCC

– It is a resource for Students, Parents, and Educators





www.CollegeEAP.org


• Reporting Results – New this year – Conditional Exemptions for EAP English

– Ready for CSU and participating CCC college-level English/mathematics courses – Conditional This status in conjunction with a grade of “C” or better in an

approved English/math course or a supervised e-learning program can be exempt from the EPT or ELM for CSU or other English or math placement tests at participating California Community Colleges (CCC).

A list of courses and e-learning programs can be found at: www.csumathsuccess.org and www.csuenglishsuccess.org .




Sample EAP Box on the STAR Student Report

The EAP box can be found on page 67 of the Post-Test Guide



• Web Sites– www.CollegeEAP.org

Links to CSU and CCC informational sites for Students, Parents, and Educators

– www.CalState.edu/EAP Provides a list of EAP Coordinators, EAP test

blueprints, and informational materials

– www.CCCCO.edu/EAP Provides a list of participating Community Colleges




2013 star interpreting and using results august 7, 2013 webcast webcast starts at 9 a.m

Documents

posttest workshop whats

writing score

higher scale score results

star results

standalone score

ela scale scores

appendix b slide

purposes of star reports