final technical report for 2015 administration 2015 tech report.pdf · final technical report for...

FinalTechnicalReportfor2015Administration

EducationalTestingServicePearson

MeasuredProgress

April18,2016

2015TechnicalReport

April18,2016Pagei

TableofContents

ExecutiveSummary.......................................................................................................................................1

Section1:Introduction.................................................................................................................................7

1.1 Background...................................................................................................................................7

1.2 PurposeoftheOperationalTests.................................................................................................8

1.3 CompositionofOperationalTests................................................................................................8

1.4 IntendedPopulation.....................................................................................................................8

1.5 GroupsandOrganizationsInvolvedwithPARCC..........................................................................9

1.6 OverviewoftheTechnicalReport...............................................................................................10

1.7 GlossaryofAbbreviations...........................................................................................................13

Section2:TestDevelopment......................................................................................................................15

2.1OverviewofthePARCCAssessment,Claims,andDesign.................................................................15

2.1.1EnglishLanguageArts/Literacy(ELA/L)AssessmentsClaimsandSubclaims..........................15

2.1.2MathematicsAssessmentsClaimsandSubclaims..................................................................16

2.2TestDevelopmentActivities.............................................................................................................17

2.2.1ItemDevelopmentProcess........................................................................................................17

2.2.2ItemandTextReviewCommittees............................................................................................18

2.2.3OperationalTestConstruction...................................................................................................20

2.2.4LinkingDesignoftheOperationalTest......................................................................................24

2.2.5GraphicalRepresentationofPARCCOperationalTestLinkingDesign......................................25

2.2.6FieldTestDataCollectionOverview..........................................................................................27

Section3:TestAdministration....................................................................................................................28

3.1TestingWindows...............................................................................................................................28

3.2TestSecurityandAdministrationPolicies.........................................................................................29

3.3AccessibilityFeaturesandAccommodations....................................................................................31

3.3.1ParticipationGuidelinesforPARCCAssessments......................................................................31

3.3.2PARCCAccessibilitySystem........................................................................................................31

3.3.3WhatareAccessibilityFeatures?...............................................................................................32

3.3.4AccommodationsforStudentswithDisabilitiesandEnglishLearners......................................32

3.3.5UniqueAccommodations...........................................................................................................34

3.3.6EmergencyAccommodations....................................................................................................34

2015TechnicalReport

April18,2016Pageii

3.3.7StudentRefusalForm.................................................................................................................34

3.4TestingIrregularitiesandSecurityBreaches.....................................................................................34

3.5DataForensicsAnalyses....................................................................................................................36

3.5.1ResponseChangeAnalysis.........................................................................................................37

3.5.2PlagiarismAnalysis.....................................................................................................................37

3.5.3InternetandSocialMediaMonitoring.......................................................................................37

3.5.4OffHoursTestingMonitoring....................................................................................................38

3.6QualityofTestAdministrationStudies.............................................................................................38

Section4:ItemScoring...............................................................................................................................40

4.1MachineScoredItems......................................................................................................................40

4.1.1KeyBasedItems.........................................................................................................................40

4.1.2RuleBasedItems........................................................................................................................40

4.2HumanorHandscoredItems............................................................................................................41

4.2.1ScorerTraining...........................................................................................................................42

4.2.2ScorerQualification....................................................................................................................45

4.2.3ManagingScoring.......................................................................................................................46

4.2.4MonitoringScoring....................................................................................................................46

Section5:TestTakerCharacteristics..........................................................................................................50

5.1OverviewofTestTakingPopulation.................................................................................................50

5.2CompositionofOperationalForms...................................................................................................50

5.3RulesforInclusionofStudentsinAnalyses......................................................................................51

5.4TestTakersbyGrade,Mode,andGender........................................................................................51

5.5Demographics...................................................................................................................................53

Section6:ClassicalItemAnalysis................................................................................................................54

6.1Overview...........................................................................................................................................54

6.2DataScreeningCriteria.....................................................................................................................54

6.3DescriptionofClassicalItemAnalysisStatistics................................................................................55

6.4SummaryofClassicalItemAnalysisFlaggingCriteria.......................................................................57

6.5ClassicalItemAnalysisResults..........................................................................................................57

Section7:DifferentialItemFunctioning.....................................................................................................69

7.1Overview...........................................................................................................................................69

7.2DIFProcedures..................................................................................................................................69

2015TechnicalReport

April18,2016Pageiii

7.3OperationalAnalysisDIFComparisonGroups..................................................................................71

7.4OperationalDifferentialItemFunctioningResults...........................................................................73

Section8:Reliability....................................................................................................................................74

8.1Overview...........................................................................................................................................74

8.2ReliabilityandSEMEstimation.........................................................................................................75

8.3ReliabilityResultsforTotalGroup....................................................................................................76

8.4ReliabilityResultsforSubgroupsofInterest.....................................................................................78

8.5ReliabilityResultsforEnglishLanguageArts/LiteracySubscores...................................................102

8.6ReliabilityResultsforMathematicsSubscores...............................................................................105

8.7ReliabilityofClassification..............................................................................................................107

8.8InterraterAgreement....................................................................................................................112

Section9:Validity.....................................................................................................................................113

9.1Overview.........................................................................................................................................113

9.2EvidenceBasedonTestContent.....................................................................................................113

9.3EvidenceBasedonInternalStructure.............................................................................................115

9.3.1Intercorrelations......................................................................................................................115

9.3.2Reliability..................................................................................................................................128

9.3.3LocalItemDependence...........................................................................................................128

9.4EvidenceBasedonRelationshipstoOtherVariables.....................................................................133

9.5EvidencefromtheSpecialStudies..................................................................................................139

9.5.1BenchmarkingStudy................................................................................................................139

9.5.2ContentAlignmentStudies......................................................................................................140

9.5.3ModeComparabilityStudy......................................................................................................141

9.5.4DeviceComparabilityStudy.....................................................................................................142

9.6EvidenceBasedonResponseProcesses.........................................................................................142

9.7InterpretationsofTestScores.........................................................................................................143

9.8EvidenceBasedontheConsequencestoTesting...........................................................................143

9.9Summary.........................................................................................................................................144

Section10:IRTCalibrationandScalinginOperationalYearOne.............................................................146

10.1Overview.......................................................................................................................................146

10.2IRTDataPreparation.....................................................................................................................146

10.2.1Overview................................................................................................................................146

2015TechnicalReport

April18,2016Pageiv

10.2.2StudentInclusion/ExclusionRules.........................................................................................147

10.2.3ItemsExcludedfromIRTSparseMatrices.............................................................................148

10.2.4Omitted,NotReachedandNotPresentedItems..................................................................148

10.2.5QualityControloftheIRTSparseMatrixDataFiles..............................................................148

10.3DescriptionoftheCalibrationProcess..........................................................................................149

10.3.1SpecialStudiestoInformtheOperationalCalibrationandScaling.......................................149

10.3.2IRTItemExclusionRules(BeforeCalibration)........................................................................153

10.3.3PARSCALECalibrationProceduresandConvergenceCriteria...............................................153

10.3.4CalibrationQualityControl....................................................................................................154

10.4ModelFitEvaluationCriteria........................................................................................................155

10.5ItemsExcludedfromScoreReporting..........................................................................................157

10.5.1ItemReviewProcess..............................................................................................................157

10.5.2CountandPercentageofItemsExcludedfromScoreReporting..........................................159

10.6ScalingParameterEstimatesfromPapertoOnline......................................................................162

10.6.1ScalingProcedures(PBTtoCBT)............................................................................................162

10.6.2ComparabilityacrossSpanishandEnglishVersions..............................................................163

10.6.3ScalingQualityControl...........................................................................................................164

10.7ItemsExcludedfromSpring2015PapertoOnlineLinkingSets...................................................164

10.8CorrelationsandPlotsofScalingItemParameterEstimates.......................................................165

10.9ScalingConstants..........................................................................................................................173

10.10SummaryStatisticsandDistributionsfromIRTAnalyses...........................................................174

10.10.1IRTSummaryStatisticsforEnglishLanguageArts/Literacy.................................................174

10.10.2IRTSummaryStatisticsforMathematics.............................................................................177

10.11MeanLinkingItemAISEffectSizeComparedtoOverallScaleScoreEffectSize........................181

Section11:PerformanceLevelSetting.....................................................................................................185

11.1PerformanceStandards................................................................................................................185

11.2PerformanceLevelsandPolicyDefinitions...................................................................................185

11.3PerformanceLevelSettingProcessforthePARCCAssessmentSystem.......................................187

11.3.1PARCCResearchStudies........................................................................................................188

11.3.2PARCCPrePolicyMeeting.....................................................................................................188

11.3.3PerformanceLevelSettingMeetings.....................................................................................188

11.3.4PARCCPostPolicyReasonablenessReview...........................................................................190

2015TechnicalReport

April18,2016Pagev

Section12:ScaleScores............................................................................................................................191

12.1OperationalTestContent(ClaimsandSubclaims).......................................................................191

12.1.1EnglishLanguageArts/Literacy..............................................................................................191

12.1.2Mathematics..........................................................................................................................193

12.2EstablishingtheReportingScales.................................................................................................193

12.2.1FullSummativeScoreScaleandPerformanceLevels............................................................194

12.2.2ELA/LReadingandWritingClaimScale.................................................................................195

12.2.3SubclaimsScale......................................................................................................................196

12.3CreatingConversionTables..........................................................................................................196

12.4ScoreDistributions........................................................................................................................202

12.4.1ScoreDistributionsforELA/L.................................................................................................202

12.4.2ScoreDistributionsforMathematics.....................................................................................211

12.5InterpretingClaimScoresandSubclaimScores............................................................................216

12.5.1InterpretingClaimScores.......................................................................................................216

12.5.2InterpretingSubclaimScores.................................................................................................216

Section13:QualityControlProcedures....................................................................................................218

13.1QualityControloftheItemBank..................................................................................................218

13.2QualityControlofTestFormDevelopment..................................................................................219

13.3QualityControlofTestMaterials..................................................................................................220

13.4QualityControlofScanning..........................................................................................................221

13.5QualityControlofImageEditing...................................................................................................221

13.6QualityControlofAnswerDocumentProcessingandScoring.....................................................222

13.7QualityControlofPsychometricProcesses..................................................................................223

13.7.1ETSPsychometricQualityControlProcess............................................................................223

13.7.2PearsonPsychometricQualityControlProcess.....................................................................225

13.7.3MeasuredProgressPsychometricQualityControlProcess...................................................226

References................................................................................................................................................229

Appendices................................................................................................................................................232

Appendix5:TestTakersbyGradeandMode,forEachState..............................................................232

Appendix7:SummaryofDifferentialItemFunction(DIF)Results.......................................................270

Appendix8:ReliabilityofClassificationbyContentandGradeLevel..................................................292

Appendix10.1:IRTResultsforSpring2015EnglishLanguageArts/Literacy(ELA/L)...........................313

2015TechnicalReport

April18,2016Pagevi

Appendix10.2:IRTResultsforSpring2015Mathematics....................................................................325

Appendix12.1:FormComposition.......................................................................................................349

Appendix12.2:ScalingConstantsandAssociatedInformation...........................................................355

Appendix12.3:RawtoScaleConversionTablesforPerformanceLevelSetting(PLS)Forms.............360

Appendix12.4:IRTTestCharacteristicCurves,InformationCurvesandCSEMCurves.......................423

Appendix12.5:SubgroupScaleScorePerformance.............................................................................486

Addendum:StatisticalSummaryoftheFall2014Administration............................................................516

Addendum5:TestTakerCharacteristics..............................................................................................516

Addendum6:ClassicalItemAnalysis....................................................................................................523

Addendum7:DifferentialItemFunctioning.........................................................................................525

Addendum8:Reliability........................................................................................................................529

Addendum9:Validity...........................................................................................................................538

Addendum10:IRTCalibrationandScalingfortheFall2014Tests......................................................542

Addendum12:ScaleScores..................................................................................................................553

2015TechnicalReport

April18,2016Pagevii

ListofTables

Table1.DefinedSummativeScaleScoresandCutScores...........................................................................5Table2.DefinedScaledScoresandCutScoresforReadingandWritingClaimScores...............................6Table1.1GlossaryofPARCCAbbreviationsandAcronyms........................................................................13Table2.1NumberofCoreOperationalFormsperGrade/SubjectbyComponentandModeforELA/LandMathematics...............................................................................................................................................21Table3.1PARCCFall2014andSpring2015TestingWindows...................................................................28Table4.1TrainingMaterialsUsedDuringScoring......................................................................................44Table4.2MathematicsQualificationRequirements..................................................................................46Table4.3ScoringValidityAgreementRequirements.................................................................................48Table4.4InterraterAgreementExpectationsandResults........................................................................49Table5.1ELA/LTestTakersbyGradeandMode:AllStatesCombined.....................................................52Table5.2MathematicsTestTakersbyGradeandMode:AllStatesCombined.........................................52Table5.3SpanishLanguageMathematicsTestTakers,byGradeandMode:AllStatesCombined..........53Table6.1SummaryofpValuesforELA/LOperationalItemsbyGrade,Component,andMode..............59Table6.2SummaryofpValuesforMathematicsOperationalItemsbyGrade,Component,andMode.61Table6.3SummaryofItemtotalPolyserialCorrelationsforELA/LOperationalItemsbyGrade,Component,andMode...............................................................................................................................64Table6.4SummaryofItemtotalCorrelationsforMathematicsOperationalItemsbyGrade,Component,andMode....................................................................................................................................................66Table7.1DIFCategoriesforDichotomousSelectedResponseandConstructedResponseItems............71Table7.2DIFCategoriesforPolytomousConstructedResponseItems.....................................................71Table7.3TraditionalDIFComparisonGroups............................................................................................72Table8.1SummaryofELA/LTestReliabilityEstimatesforTotalGroup....................................................77Table8.2SummaryofMathematicsTestReliabilityEstimatesforTotalGroup........................................78Table8.3SummaryofTestReliabilityEstimatesforSubgroups:Grade3ELA/L........................................81Table8.4SummaryofTestReliabilityEstimatesforSubgroups:Grade4ELA/L........................................82Table8.5SummaryofTestReliabilityEstimatesforSubgroups:Grade5ELA/L........................................83Table8.6SummaryofTestReliabilityEstimatesforSubgroups:Grade6ELA/L........................................84Table8.7SummaryofTestReliabilityEstimatesforSubgroups:Grade7ELA/L........................................85Table8.8SummaryofTestReliabilityEstimatesforSubgroups:Grade8ELA/L........................................86Table8.9SummaryofTestReliabilityEstimatesforSubgroups:Grade9ELA/L........................................87Table8.10SummaryofTestReliabilityEstimatesforSubgroups:Grade10ELA/L....................................88Table8.11SummaryofTestReliabilityEstimatesforSubgroups:Grade11ELA/L....................................89Table8.12SummaryofTestReliabilityEstimatesforSubgroups:Grade3Mathematics.........................90Table8.13SummaryofTestReliabilityEstimatesforSubgroups:Grade4Mathematics.........................91Table8.14SummaryofTestReliabilityEstimatesforSubgroups:Grade5Mathematics.........................92Table8.15SummaryofTestReliabilityEstimatesforSubgroups:Grade6Mathematics.........................93Table8.16SummaryofTestReliabilityEstimatesforSubgroups:Grade7Mathematics.........................94Table8.17SummaryofTestReliabilityEstimatesforSubgroups:Grade8Mathematics.........................95Table8.18SummaryofTestReliabilityEstimatesforSubgroups:AlgebraI..............................................96

2015TechnicalReport

April18,2016Pageviii

Table8.19SummaryofTestReliabilityEstimatesforSubgroups:Geometry............................................97Table8.20SummaryofTestReliabilityEstimatesforSubgroups:AlgebraII.............................................98Table8.21SummaryofTestReliabilityEstimatesforSubgroups:IntegratedMathematicsI...................99Table8.22SummaryofTestReliabilityEstimatesforSubgroups:IntegratedMathematicsII................100Table8.23SummaryofTestReliabilityEstimatesforSubgroups:IntegratedMathematicsIII...............101Table8.24DescriptionsofELA/LClaimsandSubclaims...........................................................................102Table8.25AverageELA/LReliabilityEstimatesforTotalTestandSubscores.........................................104Table8.26AverageMathematicsReliabilityEstimatesforTotalTestandSubscores.............................106Table8.27ReliabilityofClassification:SummaryforELA/L......................................................................108Table8.28ReliabilityofClassification:Grade3ELA/L..............................................................................109Table8.29ReliabilityofClassification:SummaryforMathematics.........................................................111Table8.30InterraterAgreementExpectationsandResults....................................................................112Table9.1AverageIntercorrelationsandReliabilitybetweenGrade3ELA/LSubclaims..........................117Table9.2AverageIntercorrelationsandReliabilitybetweenGrade4ELA/LSubclaims..........................117Table9.3AverageIntercorrelationsandReliabilitybetweenGrade5ELA/LSubclaims..........................118Table9.4AverageIntercorrelationsandReliabilitybetweenGrade6ELA/LSubclaims..........................118Table9.5AverageIntercorrelationsandReliabilitybetweenGrade7ELA/LSubclaims..........................119Table9.6AverageIntercorrelationsandReliabilitybetweenGrade8ELA/LSubclaims..........................119Table9.7AverageIntercorrelationsandReliabilitybetweenGrade9ELA/LSubclaims..........................120Table9.8AverageIntercorrelationsandReliabilitybetweenGrade10ELA/LSubclaims........................120Table9.9AverageIntercorrelationsandReliabilitybetweenGrade11ELA/LSubclaims........................121Table9.10AverageIntercorrelationsandReliabilitybetweenGrade3MathematicsSubclaims...........121Table9.11AverageIntercorrelationsandReliabilitybetweenGrade4MathematicsSubclaims...........122Table9.12AverageIntercorrelationsandReliabilitybetweenGrade5MathematicsSubclaims...........122Table9.13AverageIntercorrelationsandReliabilitybetweenGrade6MathematicsSubclaims...........123Table9.14AverageIntercorrelationsandReliabilitybetweenGrade7MathematicsSubclaims...........123Table9.15AverageIntercorrelationsandReliabilitybetweenGrade8MathematicsSubclaims...........124Table9.16AverageIntercorrelationsandReliabilitybetweenAlgebraISubclaims................................124Table9.17AverageIntercorrelationsandReliabilitybetweenGeometrySubclaims..............................125Table9.18AverageIntercorrelationsandReliabilitybetweenAlgebraIISubclaims...............................125Table9.19AverageIntercorrelationsandReliabilitybetweenIntegratedMathematicsISubclaims.....126Table9.20AverageIntercorrelationsandReliabilitybetweenIntegratedMathematicsIISubclaims....126Table9.21AverageIntercorrelationsandReliabilitybetweenIntegratedMathematicsIIISubclaims...127Table9.22ConditionsusedinLIDReliabilityInvestigationandResults...................................................131Table9.23SummaryofQ3ValuesforELA/LGrade4andIntegratedMathematicsII.............................131Table9.24AverageCorrelationsbetweenELA/LandMathematicsforGrade3.....................................134Table9.25AverageCorrelationsbetweenELA/LandMathematicsforGrade4.....................................134Table9.26AverageCorrelationsbetweenELA/LandMathematicsforGrade5.....................................135Table9.27AverageCorrelationsbetweenELA/LandMathematicsforGrade6.....................................135Table9.28AverageCorrelationsbetweenELA/LandMathematicsforGrade7.....................................136Table9.29AverageCorrelationsbetweenELA/LandMathematicsforGrade8.....................................136Table9.30AverageCorrelationsbetweenELA/LandMathematicsforHighSchool...............................137

2015TechnicalReport

April18,2016Pageix

Table9.31AverageCorrelationsbetweenELA/LReadingandMathematicsforHighSchool.................137Table9.32AverageCorrelationsbetweenELA/LWritingandMathematicsforHighSchool..................138Table10.1NCounts,PercentofStudents,andNumberofItemsintheELA/LIRTCalibrationFiles......148Table10.2NCounts,PercentofStudents,andNumberofItemsintheMathematicsIRTCalibrationFiles..................................................................................................................................................................149Table10.3NumberandPercentageofELA/LItemsExcludedfromIRTCalibration................................160Table10.4NumberandPercentageofMathematicsItemsExcludedfromIRTCalibration....................161Table10.5WRMSDFlaggingCriteriaforInspectionandPossibleRemovalofLinkingItems..................163Table10.6NumberofELA/LItemsExcludedfromtheCBT/PBTLinkingSets..........................................164Table10.7NumberofMathematicsItemsExcludedfromtheCBT/PBTLinkingSets..............................165Table10.8NumberofItems,NumberofPointsandCorrelationsforELA/LLinkingItems.....................165Table10.9NumberofItems,NumberofPointsandCorrelationsforMathematicsLinkingItems.........166Table10.10ScalingConstantsSpring2015PBTtoCBTforELA/L............................................................173Table10.11ScalingConstantsSpring2015PBTtoCBTforMathematics................................................174Table10.12CBTIRTSummaryParameterEstimatesforAllItemsforELA/LbyGrade............................175Table10.13PBTIRTSummaryParameterEstimatesforAllItemsforELA/LbyGrade............................175Table10.14CBTIRTStandardErrorsofParameterEstimatesforAllItemsforELA/LbyGrade.............176Table10.15PBTIRTStandardErrorsofParameterEstimatesforAllItemsforELA/LbyGrade..............176Table10.16CBTIRTModelFitforAllItemsforELA/LbyGrade..............................................................177Table10.17PBTIRTModelFitforAllItemsforELA/LbyGrade..............................................................177Table10.18CBTIRTSummaryParameterEstimatesforAllItemsforMathematicsbyGrade/Subject..178Table10.19PBTIRTSummaryParameterEstimatesforAllItemsforMathematicsbyGrade/Subject..179Table10.20CBTIRTStandardErrorsofParameterEstimatesforAllItemsforMathematicsbyGrade/Subject...........................................................................................................................................179Table10.21PBTIRTStandardErrorsofParameterEstimatesforAllItemsforMathematicsbyGrade/Subject...........................................................................................................................................180Table10.22CBTIRTModelFitforAllItemsforMathematicsbyGrade/Subject.....................................180Table10.23PBTIRTModelFitforAllItemsforMathematicsbyGrade/Subject.....................................181Table10.24AverageItemScoreEffectSizeComparedtoOverallScaleScoreEffectSizeforELA/L.......183Table10.25AverageItemScoreEffectSizeComparedtoOverallScaleScoreEffectSizeforMathematics..................................................................................................................................................................184Table11.1PARCCPLSCommitteeMeetingsandDates...........................................................................190Table12.1FormCompositionforELA/LGrade3.....................................................................................192Table12.2ContributionofProseConstructedResponseItemstoELA/L................................................193Table12.3MathematicsFormCompositionforGrade3.........................................................................193Table12.4DefinedSummativeScaleScores............................................................................................194Table12.5DefinedScaledScoresforReadingandWritingClaimScores................................................195Table12.6CalculatingScalingConstantsforReadingandWritingClaimScores.....................................195Table12.7SubgroupPerformanceforELA/L:Grade3.............................................................................208Table12.8SubgroupPerformanceforELA/L:Grade9.............................................................................210Table12.9SubgroupPerformanceforMathematicsScaleScores:Grade3............................................214Table12.10SubgroupPerformanceforMathematicsScaleScores:AlgebraI........................................215

2015TechnicalReport

April18,2016Pagex

Table12.11SubgroupPerformanceforMathematicsScaleScores:IntegratedMathematicsI..............216TableA.5.1ELA/LTestTakers,byState,Grade,andMode......................................................................232TableA.5.2MathematicsTestTakers,byState,Grade,andMode..........................................................236TableA.5.3SpanishLanguageMathematicsTestTakers,byState,Grade,andMode............................240TableA.5.4AllStatesCombined:ELA/LTestTakersbyGrade,Mode,andGender.................................244TableA.5.5AllStatesCombined:AllMathematicsTestTakersbyGrade,Mode,andGender...............245TableA.5.6AllStatesCombined:SpanishLanguageMathematicsTestTakers,byGrade,Mode,andGender......................................................................................................................................................247TableA.5.7DemographicInformationforGrade3ELA/L,OverallandbyState......................................249TableA.5.8DemographicInformationforGrade4ELA/L,OverallandbyState......................................250TableA.5.9DemographicInformationforGrade5ELA/L,OverallandbyState......................................251TableA.5.10DemographicInformationforGrade6ELA/L,OverallandbyState....................................252TableA.5.11DemographicInformationforGrade7ELA/L,OverallandbyState....................................253TableA.5.12DemographicInformationforGrade8ELA/L,OverallandbyState....................................254TableA.5.13DemographicInformationforGrade9ELA/L,OverallandbyState....................................255TableA.5.14DemographicInformationforGrade10ELA/L,OverallandbyState..................................256TableA.5.15DemographicInformationforGrade11ELA/L,OverallandbyState..................................257TableA.5.16DemographicInformationforGrade3Mathematics,OverallandbyState.......................258TableA.5.17DemographicInformationforGrade4Mathematics,OverallandbyState.......................259TableA.5.18DemographicInformationforGrade5Mathematics,OverallandbyState.......................260TableA.5.19DemographicInformationforGrade6Mathematics,OverallandbyState.......................261TableA.5.20DemographicInformationforGrade7Mathematics,OverallandbyState.......................262TableA.5.21DemographicInformationforGrade8Mathematics,OverallandbyState.......................263TableA.5.22DemographicInformationforAlgebraI,OverallandbyState............................................264TableA.5.23DemographicInformationforGeometry,OverallandbyState..........................................265TableA.5.24DemographicInformationforAlgebraII,OverallandbyState...........................................266TableA.5.25DemographicInformationforIntegratedMathematicsI,OverallandbyState.................267TableA.5.26DemographicInformationforIntegratedMathematicsII,OverallandbyState................268TableA.5.27DemographicInformationforIntegratedMathematicsIII,OverallandbyState...............269TableA.7.1DifferentialItemFunctioningforELA/LGrade3...................................................................270TableA.7.2DifferentialItemFunctioningforELA/LGrade4...................................................................271TableA.7.3DifferentialItemFunctioningforELA/LGrade5...................................................................272TableA.7.4DifferentialItemFunctioningforELA/LGrade6...................................................................273TableA.7.5DifferentialItemFunctioningforELA/LGrade7...................................................................274TableA.7.6DifferentialItemFunctioningforELA/LGrade8...................................................................275TableA.7.7DifferentialItemFunctioningforELA/LGrade9...................................................................276TableA.7.8DifferentialItemFunctioningforELA/LGrade10.................................................................277TableA.7.9DifferentialItemFunctioningforELA/LGrade11.................................................................278TableA.7.10DifferentialItemFunctioningforMathematicsGrade3.....................................................279TableA.7.11DifferentialItemFunctioningforMathematicsGrade4.....................................................280TableA.7.12DifferentialItemFunctioningforMathematicsGrade5.....................................................281TableA.7.13DifferentialItemFunctioningforMathematicsGrade6.....................................................282

2015TechnicalReport

April18,2016Pagexi

TableA.7.14DifferentialItemFunctioningforMathematicsGrade7.....................................................283TableA.7.15DifferentialItemFunctioningforMathematicsGrade8.....................................................284TableA.7.16DifferentialItemFunctioningforAlgebraI..........................................................................285TableA.7.17DifferentialItemFunctioningforGeometry........................................................................286TableA.7.18DifferentialItemFunctioningforAlgebraII.........................................................................287TableA.7.19DifferentialItemFunctioningforIntegratedMathematicsI...............................................288TableA.7.20DifferentialItemFunctioningforIntegratedMathematicsII..............................................289TableA.7.21DifferentialItemFunctioningforIntegratedMathematicsIII.............................................290TableA.7.22DifferentialItemFunctioningforMathematics:SpanishLanguagevs.EnglishLanguageForms........................................................................................................................................................291TableA.8.1ReliabilityofClassification:Grade3ELA/L............................................................................292TableA.8.2ReliabilityofClassification:Grade4ELA/L............................................................................293TableA.8.3ReliabilityofClassification:Grade5ELA/L............................................................................294TableA.8.4ReliabilityofClassification:Grade6ELA/L............................................................................295TableA.8.5ReliabilityofClassification:Grade7ELA/L............................................................................296TableA.8.6ReliabilityofClassification:Grade8ELA/L............................................................................297TableA.8.7ReliabilityofClassification:Grade9ELA/L............................................................................298TableA.8.8ReliabilityofClassification:Grade10ELA/L..........................................................................299TableA.8.9ReliabilityofClassification:Grade11ELA/L..........................................................................300TableA.8.10ReliabilityofClassification:Grade3Mathematics..............................................................301TableA.8.11ReliabilityofClassification:Grade4Mathematics..............................................................302TableA.8.12ReliabilityofClassification:Grade5Mathematics..............................................................303TableA.8.13ReliabilityofClassification:Grade6Mathematics..............................................................304TableA.8.14ReliabilityofClassification:Grade7Mathematics..............................................................305TableA.8.15ReliabilityofClassification:Grade8Mathematics..............................................................306TableA.8.16ReliabilityofClassification:AlgebraI...................................................................................307TableA.8.17ReliabilityofClassification:Geometry.................................................................................308TableA.8.18ReliabilityofClassification:AlgebraII..................................................................................309TableA.8.19ReliabilityofClassification:IntegratedMathematicsI........................................................310TableA.8.20ReliabilityofClassification:IntegratedMathematicsII.......................................................311TableA.8.21ReliabilityofClassification:IntegratedMathematicsIII......................................................312TableA.10.1CBTIRTSummaryParameterEstimatesforAllItemsforELA/LbyGrade..........................313TableA.10.2PBTIRTSummaryParameterEstimatesforAllItemsforELA/LbyGrade..........................315TableA.10.3CBTIRTStandardErrorsofParameterEstimatesforAllItemsforELA/LbyGrade............317TableA.10.4PBTIRTStandardErrorsofParameterEstimatesforAllItemsforELA/LbyGrade............319TableA.10.5CBTIRTModelFitforAllItemsforELA/LbyGrade.............................................................321TableA.10.6PBTIRTModelFitforAllItemsforELA/LbyGrade.............................................................323TableA.10.7CBTIRTSummaryParameterEstimatesforAllItemsforMathematicsbyGrade/Subject.325TableA.10.8PBTIRTSummaryParameterEstimatesforAllItemsforMathematicsbyGrade/Subject.329TableA.10.9CBTIRTStandardErrorsofParameterEstimatesforAllItemsforMathematicsbyGrade/Subject...........................................................................................................................................333

2015TechnicalReport

April18,2016Pagexii

TableA.10.10PBTIRTStandardErrorsofParameterEstimatesforAllItemsforMathematicsbyGrade/Subject...........................................................................................................................................337TableA.10.11CBTIRTModelFitforAllItemsforMathematicsbyGrade/Subject.................................341TableA.10.12PBTIRTModelFitforAllItemsforMathematicsbyGrade/Subject.................................345TableA.12.1FormCompositionforELA/LGrade3..................................................................................349TableA.12.2FormCompositionforELA/LGrade4..................................................................................349TableA.12.3FormCompositionforELA/LGrade5..................................................................................349TableA.12.4FormCompositionforELA/LGrade6..................................................................................350TableA.12.5FormCompositionforELA/LGrade7..................................................................................350TableA.12.6FormCompositionforELA/LGrade8..................................................................................350TableA.12.7FormCompositionforELA/LGrade9..................................................................................351TableA.12.8FormCompositionforELA/LGrade10................................................................................351TableA.12.9FormCompositionforELA/LGrade11................................................................................351TableA.12.10FormCompositionforMathematicsGrade3....................................................................352TableA.12.11FormCompositionforMathematicsGrade4....................................................................352TableA.12.12FormCompositionforMathematicsGrade5....................................................................352TableA.12.13FormCompositionforMathematicsGrade6....................................................................352TableA.12.14FormCompositionforMathematicsGrade7....................................................................353TableA.12.15FormCompositionforMathematicsGrade8....................................................................353TableA.12.16FormCompositionforAlgebraI.........................................................................................353TableA.12.17FormCompositionforGeometry.......................................................................................353TableA.12.18FormCompositionforAlgebraII........................................................................................354TableA.12.19FormCompositionforIntegratedMathematicsI..............................................................354TableA.12.20FormCompositionforIntegratedMathematicsII.............................................................354TableA.12.21FormCompositionforIntegratedMathematicsIII............................................................354TableA.12.22ThresholdScoresandScalingConstantsforELA/LGrades3to8.....................................355TableA.12.23ThresholdScoresandScalingConstantsforMathematicsGrades3to8.........................356TableA.12.24ThresholdScoresandScalingConstantsforHighSchoolELA...........................................357TableA.12.25ThresholdScoresandScalingConstantsforHighSchoolMathematics............................358TableA.12.26ScalingConstantsforReadingandWritingGrades3to8.................................................359TableA.12.27ConversionTableforPerformanceLevelSettingForm:ELA/LGrade3............................360TableA.12.28ConversionTableforPerformanceLevelSettingForm:ELA/LGrade4............................363TableA.12.29ConversionTableforPerformanceLevelSettingForm:ELA/LGrade5............................366TableA.12.30ConversionTableforPerformanceLevelSettingForm:ELA/LGrade6............................369TableA.12.31ConversionTableforPerformanceLevelSettingForm:ELA/LGrade7............................373TableA.12.32ConversionTableforPerformanceLevelSettingForm:ELA/LGrade8............................377TableA.12.33ConversionTableforPerformanceLevelSettingForm:ELA/LGrade9............................381TableA.12.34ConversionTableforPerformanceLevelSettingForm:ELA/LGrade10..........................385TableA.12.35ConversionTableforPerformanceLevelSettingForm:ELA/LGrade11..........................389TableA.12.36ConversionTableforPerformanceLevelSettingForm:MathematicsGrade3................393TableA.12.37ConversionTableforPerformanceLevelSettingForm:MathematicsGrade4................395TableA.12.38ConversionTableforPerformanceLevelSettingForm:MathematicsGrade5................397

2015TechnicalReport

April18,2016Pagexiii

TableA.12.39ConversionTableforPerformanceLevelSettingForm:MathematicsGrade6................399TableA.12.40ConversionTableforPerformanceLevelSettingForm:MathematicsGrade7................401TableA.12.41ConversionTableforPerformanceLevelSettingForm:MathematicsGrade8................403TableA.12.42ConversionTableforPerformanceLevelSettingForm:AlgebraI.....................................405TableA.12.43ConversionTableforPerformanceLevelSettingForm:AlgebraII....................................408TableA.12.44ConversionTableforPerformanceLevelSettingForm:Geometry...................................411TableA.12.45ConversionTableforPerformanceLevelSettingForm:IntegratedMathematicsI..........414TableA.12.46ConversionTableforPerformanceLevelSettingForm:IntegratedMathematicsII.........417TableA.12.47ConversionTableforPerformanceLevelSettingForm:IntegratedMathematicsIII........420TableA.12.48SubgroupPerformanceforELA/LScaleScores:Grade3...................................................486TableA.12.49SubgroupPerformanceforELA/LScaleScores:Grade4...................................................488TableA.12.50SubgroupPerformanceforELA/LScaleScores:Grade5...................................................490TableA.12.51SubgroupPerformanceforELA/LScaleScores:Grade6...................................................492TableA.12.52SubgroupPerformanceforELA/LScaleScores:Grade7...................................................494TableA.12.53SubgroupPerformanceforELA/LScaleScores:Grade8...................................................496TableA.12.54SubgroupPerformanceforELA/LScaleScores:Grade9...................................................498TableA.12.55SubgroupPerformanceforELA/LScaleScores:Grade10.................................................500TableA.12.56SubgroupPerformanceforELA/LScaleScores:Grade11.................................................502TableA.12.57SubgroupPerformanceforMathematicsScaleScores:Grade3.......................................504TableA.12.58SubgroupPerformanceforMathematicsScaleScores:Grade4.......................................505TableA.12.59SubgroupPerformanceforMathematicsScaleScores:Grade5.......................................506TableA.12.60SubgroupPerformanceforMathematicsScaleScores:Grade6.......................................507TableA.12.61SubgroupPerformanceforMathematicsScaleScores:Grade7.......................................508TableA.12.62SubgroupPerformanceforMathematicsScaleScores:Grade8.......................................509TableA.12.63SubgroupPerformanceforMathematicsScaleScores:AlgebraI.....................................510TableA.12.64SubgroupPerformanceforMathematicsScaleScores:Geometry...................................511TableA.12.65SubgroupPerformanceforMathematicsScaleScores:AlgebraII....................................512TableA.12.66SubgroupPerformanceforMathematicsScaleScores:IntegratedMathematicsI..........513TableA.12.67SubgroupPerformanceforMathematicsScaleScores:IntegratedMathematicsII.........514TableA.12.68SubgroupPerformanceforMathematicsScaleScores:IntegratedMathematicsIII........515TableADD.5.1StateParticipationinELA/LFall2014OperationalTests,byGrade.................................516TableADD.5.2StateParticipationinMathematicsFall2014OperationalTests,byGrade.....................517TableADD.5.3AllStatesCombined:Fall2014ELA/LTestTakersbyGradeandGender.........................517TableADD.5.4AllStatesCombined:Fall2014MathematicsTestTakersbyGradeandGender.............517TableADD.5.5Fall2014ELA/LTestTakersbyState,Grade,andGender................................................518TableADD.5.6Fall2014MathematicsTestTakersbyState,ContentArea,andGender........................519TableADD.5.7DemographicInformationforFall2014Grade9ELA/L,OverallandbyState.................519TableADD.5.8DemographicInformationforFall2014Grade10ELA/L,OverallandbyState...............520TableADD.5.9DemographicInformationforFall2014Grade11ELA/L,OverallandbyState...............520TableADD.5.10DemographicInformationforFall2014AlgebraI,OverallandbyState........................521TableADD.5.11DemographicInformationforFall2014Geometry,OverallandbyState......................521TableADD.5.12DemographicInformationforFall2014AlgebraII,OverallandbyState.......................522

2015TechnicalReport

April18,2016Pagexiv

TableADD.6.1SummaryofpValuesforFall2014ELA/LandMathematicsOperationalItems,byGradeandComponent........................................................................................................................................523TableADD.6.2SummaryofPolyserialCorrelationsforFall2014ELA/LandMathematicsOperationalItems,byGradeandTest..........................................................................................................................524TableADD.7.1DifferentialItemFunctioningforFall2014HighSchoolELA/L.........................................525TableADD.7.2DifferentialItemFunctioningforFall2014HighSchoolMathematics.............................527TableADD.8.1SummaryofTestReliabilityEstimatesforFall2014TotalGroup.....................................529TableADD.8.2SummaryofTestReliabilityEstimatesforFall2014Subgroups:Grade9ELA/L..............530TableADD.8.3SummaryofTestReliabilityEstimatesforFall2014Subgroups:Grade10ELA/L............531TableADD.8.4SummaryofTestReliabilityEstimatesforFall2014Subgroups:Grade11ELA/L............532TableADD.8.5SummaryofTestReliabilityEstimatesforFall2014Subgroups:AlgebraI......................533TableADD.8.6SummaryofTestReliabilityEstimatesforFall2014Subgroups:Geometry....................534TableADD.8.7SummaryofTestReliabilityEstimatesforFall2014Subgroups:AlgebraII.....................535TableADD.8.8Fall2014AverageELA/LReliabilityEstimatesforReadingTotalandSubscores..............536TableADD.8.9Fall2014AverageELA/LReliabilityEstimatesforWritingTotalandSubscores...............536TableADD.8.10Fall2014AverageMathematicsReliabilityEstimatesforTotalTestandSubscores......537TableADD.9.1AverageIntercorrelationsandReliabilitybetweenGrade9ELA/LReadingSubclaims...538TableADD.9.2AverageIntercorrelationsandReliabilitybetweenGrade10ELA/LReadingSubclaims.538TableADD.9.3AverageIntercorrelationsandReliabilitybetweenGrade11ELA/LReadingSubclaims.539TableADD.9.4AverageIntercorrelationsandReliabilitybetweenAlgebraISubclaims..........................539TableADD.9.5AverageIntercorrelationsandReliabilitybetweenGeometrySubclaims........................539TableADD.9.6AverageIntercorrelationsandReliabilitybetweenAlgebraIISubclaims.........................540TableADD.9.7AverageCorrelationsbetweenELA/LandMathematicsforHighSchool.........................540TableADD.9.8AverageCorrelationsbetweenReadingandMathematicsforHighSchool.....................540TableADD.9.9AverageCorrelationsbetweenWritingandMathematicsforHighSchool......................541TableADD.10.1NumberofStudentsandNumberandPercentofFallblockItemsExcludedfromCalibration.................................................................................................................................................543TableADD.10.2NumberofFallBlockItemsExcludedfromtheFallBlock/SpringLinkingSets...............544TableADD.10.3NumberofItems,Points,andCorrelationsforFallBlock/SpringLinkingItems.............544TableADD.10.4ScalingConstants2014FallBlocktoSpring2015...........................................................545TableADD.10.52014FallBlockIRTSummaryParameterEstimatesbyGrade/Subject..........................545TableADD.10.62014FallBlockIRTStandardErrorsofParameterEstimatesbyGrade/Subject............545TableADD.10.72014FallBlockIRTModelFitbyGrade/Subject.............................................................546TableADD.10.8PBTIRTSummaryParameterEstimatesforAllItemsforELA/LbyGrade.....................547TableADD.10.9PBTIRTStandardErrorsofParameterEstimatesforAllItemsforELA/LbyGrade.......548TableADD.10.10PBTIRTModelFitforAllItemsforELA/LbyGrade......................................................549TableADD.10.11PBTIRTSummaryParameterEstimatesforAllItemsforMathematicsbyGrade/Subject..................................................................................................................................................................550TableADD.10.12PBTIRTStandardErrorsofParameterEstimatesforAllItemsforMathematicsbyGrade/Subject...........................................................................................................................................551TableADD.10.13PBTIRTModelFitforAllItemsforMathematicsbyGrade/Subject............................552TableADD.12.1Fall2014SubgroupPerformanceforELA/LScaleScores:Grade9................................553

2015TechnicalReport

April18,2016Pagexv

TableADD.12.2Fall2014SubgroupPerformanceforELA/LScaleScores:Grade10..............................555TableADD.12.3Fall2014SubgroupPerformanceforELA/LScaleScores:Grade11..............................557TableADD.12.4Fall2014SubgroupPerformanceforMathematicsScaleScores:AlgebraI...................559TableADD.12.5Fall2014SubgroupPerformanceforMathematicsScaleScores:Geometry.................560TableADD.12.6Fall2014SubgroupPerformanceforMathematicsScaleScores:AlgebraII..................561

2015TechnicalReport

April18,2016Pagexvi

ListofFiguresFigure2.1PARCCELA/LCBTLinkingDesign(Grades68,and11)..............................................................26Figure2.2PARCCMathematicsCBTLinkingDesign(Grades38,andAlgebraII)......................................26Figure3.1ConfigurationofUnitswithinthe20142015PARCCAssessments(SampleforGrade7andHighSchool)................................................................................................................................................29Figure3.2ClaimsfromtheAdministrationPhaseofthePARCCTheoryofAction....................................39Figure9.1ComparisonofScoreReliabilitybyItemReliabilityandCluster(Testlet)ReliabilityApproaches..................................................................................................................................................................130Figure9.2DistributionofQ3ValuesforGrade4ELA/L............................................................................132Figure9.3DistributionofQ3ValuesforIntegratedMathematicsII.........................................................132Figure10.1Grade3ELA/L3CategoryItem,2PL/GPCModel,Ncount124,090,chisquare305.0,adjustedfit=0.050...................................................................................................................................156Figure10.2AlgebraIDichotomous(2Category)Item,2PLModel,Ncount42,630,chisquare5167.6,adjustedfit=0.329...................................................................................................................................157Figure10.3AlgebraIDichotomousItem,afterthe3PLmodelwasappliedusingtheWeightedLeastSquaresapproach,Ncount42,630..........................................................................................................159Figure10.4ELA/LGrade3TransformedNewavs.ReferenceaParameterEstimates..........................167Figure10.5ELA/LGrade3TransformedNewbvs.ReferencebParameterEstimates..........................167Figure10.6ELA/LGrade3TransformedNewandReferenceTCCforLinkingItems...............................168Figure10.7ELA/LGrade11TransformedNewavs.ReferenceaParameterEstimates........................168Figure10.8ELA/LGrade11TransformedNewbvs.ReferencebParameterEstimates........................169Figure10.9ELA/LGrade11TransformedNewandReferenceTCCforLinkingItems.............................169Figure10.10MathematicsGrade8TransformedNewavs.ReferenceaParameterEstimates............170Figure10.11MathematicsGrade8TransformedNewbvs.ReferencebParameterEstimates............170Figure10.12MathematicsGrade8TransformedNewandReferenceTCCforLinkingItems.................171Figure10.13IntegratedMathematicsIIITransformedNewavs.ReferenceaParameterEstimates...171Figure10.14IntegratedMathematicsIIITransformedNewbvs.ReferencebParameterEstimates...172Figure10.15IntegratedMathematicsIIITransformedNewandReferenceTCCforLinkingItems.........172Figure12.1TestCharacteristicCurvesforELA/LGrade3........................................................................199Figure12.2InformationCurvesforELA/LGrade3...................................................................................200Figure12.3ConditionalStandardErrorofMeasurementCurveforELA/LGrade3.................................201Figure12.4DistributionsofELA/LScaleScores:Grades3to11..............................................................202Figure12.4(continued)DistributionsofELA/LScaleScores:Grades3to11..........................................203Figure12.5DistributionsofReadingScaleScores:Grades3to11..........................................................204Figure12.5(continued)DistributionsofReadingScaleScores:Grades3to11.......................................205Figure12.6DistributionsofWritingScaleScores:Grades3to11...........................................................206Figure12.6(continued)DistributionsofWritingScaleScores:Grades3to11........................................207Figure12.7DistributionsofMathematicsScaleScores:Grades3to8....................................................212Figure12.8DistributionsofMathematicsScaleScores:HighSchool.......................................................213FigureA.12.1TestCharacteristicCurvesELA/LGrade3...........................................................................423FigureA.12.2TestCharacteristicCurvesELA/LGrade4...........................................................................424

2015TechnicalReport

April18,2016Pagexvii

FigureA.12.3TestCharacteristicCurvesELA/LGrade5...........................................................................425FigureA.12.4TestCharacteristicCurvesELA/LGrade6...........................................................................426FigureA.12.5TestCharacteristicCurvesELA/LGrade7...........................................................................427FigureA.12.6TestCharacteristicCurvesELA/LGrade8...........................................................................428FigureA.12.7TestCharacteristicCurvesELA/LGrade9...........................................................................429FigureA.12.8TestCharacteristicCurvesELA/LGrade10.........................................................................430FigureA.12.9TestCharacteristicCurvesELA/LGrade11.........................................................................431FigureA.12.10TestCharacteristicCurvesMathematicsGrade3............................................................432FigureA.12.11TestCharacteristicCurvesMathematicsGrade4............................................................433FigureA.12.12TestCharacteristicCurvesMathematicsGrade5............................................................434FigureA.12.13TestCharacteristicCurvesMathematicsGrade6............................................................435FigureA.12.14TestCharacteristicCurvesMathematicsGrade7............................................................436FigureA.12.15TestCharacteristicCurvesMathematicsGrade8............................................................437FigureA.12.16TestCharacteristicCurvesAlgebraI.................................................................................438FigureA.12.17TestCharacteristicCurvesGeometry...............................................................................439FigureA.12.18TestCharacteristicCurvesAlgebraII................................................................................440FigureA.12.19TestCharacteristicCurvesIntegratedMathematicsI......................................................441FigureA.12.20TestCharacteristicCurvesIntegratedMathematicsII.....................................................442FigureA.12.21TestCharacteristicCurvesIntegratedMathematicsIII....................................................443FigureA.12.22TestInformationCurvesELA/LGrade3...........................................................................444FigureA.12.23TestInformationCurvesELA/LGrade4...........................................................................445FigureA.12.24TestInformationCurvesELA/LGrade5...........................................................................446FigureA.12.25TestInformationCurvesELA/LGrade6...........................................................................447FigureA.12.26TestInformationCurvesELA/LGrade7...........................................................................448FigureA.12.27TestInformationCurvesELA/LGrade8...........................................................................449FigureA.12.28TestInformationCurvesELA/LGrade9...........................................................................450FigureA.12.29TestInformationCurvesELA/LGrade10.........................................................................451FigureA.12.30TestInformationCurvesELA/LGrade11.........................................................................452FigureA.12.31TestInformationCurvesMathematicsGrade3...............................................................453FigureA.12.32TestInformationCurvesMathematicsGrade4...............................................................454FigureA.12.33TestInformationCurvesMathematicsGrade5...............................................................455FigureA.12.34TestInformationCurvesMathematicsGrade6...............................................................456FigureA.12.35TestInformationCurvesMathematicsGrade7...............................................................457FigureA.12.36TestInformationCurvesMathematicsGrade8...............................................................458FigureA.12.37TestInformationCurvesAlgebraI....................................................................................459FigureA.12.38TestInformationCurvesGeometry..................................................................................460FigureA.12.39TestInformationCurvesAlgebraII...................................................................................461FigureA.12.40TestInformationCurvesIntegratedMathematicsI.........................................................462FigureA.12.41TestInformationCurvesIntegratedMathematicsII........................................................463FigureA.12.42TestInformationCurvesIntegratedMathematicsIII.......................................................464FigureA.12.43CSEMCurvesELA/LGrade3.............................................................................................465FigureA.12.44CSEMCurvesELA/LGrade4.............................................................................................466

2015TechnicalReport

April18,2016Pagexviii

FigureA.12.45CSEMCurvesELA/LGrade5.............................................................................................467FigureA.12.46CSEMCurvesELA/LGrade6.............................................................................................468FigureA.12.47CSEMCurvesELA/LGrade7.............................................................................................469FigureA.12.48CSEMCurvesELA/LGrade8.............................................................................................470FigureA.12.49CSEMCurvesELA/LGrade9.............................................................................................471FigureA.12.50CSEMCurvesELA/LGrade10...........................................................................................472FigureA.12.51CSEMCurvesELA/LGrade11...........................................................................................473FigureA.12.52CSEMCurvesMathematicsGrade3.................................................................................474FigureA.12.53CSEMCurvesMathematicsGrade4.................................................................................475FigureA.12.54CSEMCurvesMathematicsGrade5.................................................................................476FigureA.12.55CSEMCurvesMathematicsGrade6.................................................................................477FigureA.12.56CSEMCurvesMathematicsGrade7.................................................................................478FigureA.12.57CSEMCurvesMathematicsGrade8.................................................................................479FigureA.12.58CSEMCurvesAlgebraI......................................................................................................480FigureA.12.59CSEMCurvesGeometry....................................................................................................481FigureA.12.60CSEMCurvesAlgebraII.....................................................................................................482FigureA.12.61CSEMCurvesIntegratedMathematicsI...........................................................................483FigureA.12.62CSEMCurvesIntegratedMathematicsII..........................................................................484FigureA.12.63CSEMCurvesIntegratedMathematicsIII.........................................................................485

2015TechnicalReport

April18,2016Page1

ExecutiveSummary

ThePartnershipforAssessmentofReadinessforCollegeandCareers(PARCC)isastateledconsortiumdesignedtocreatenextgenerationassessmentsthat,comparedtotraditionalK12assessments,moreaccuratelymeasurestudentprogresstowardcollegeandcareerreadiness.ThePARCCassessmentsarealignedtotheCommonCoreStateStandards(CCSS)andwereadministeredoperationallyforthefirsttimeinthe20142015academicyear.PARCCcomprisesassessmentsinbothEnglishLanguageArts/Literacy(ELA/L)andMathematicsingrades3to8andhighschool.

Theinformationprovidedinthistechnicalreportisintendedforusebythosewhoevaluatetests,interpretscores,orusetestresultsinmakingeducationaldecisions.Itisassumedthatthereaderhastechnicalknowledgeoftestconstructionandmeasurementprocedures,asstatedinStandardsforEducationalandPsychologicalTesting(AmericanEducationalResearchAssociation[AERA],AmericanPsychologicalAssociation[APA],&NationalCouncilonMeasurementinEducation[NCME],2014).ThepurposeofthistechnicalreportistodescribethefirstoperationaladministrationofthePARCCassessmentsandincludesthefollowingtopics:

Backgroundandpurposeoftheassessments, Testdevelopmentofitemsandforms, Testadministration,security,andscoring, Testtakercharacteristics, Classicalitemanalysesanddifferentialitemfunctioning, Reliabilityandvalidityofscores, Itemresponsetheory(IRT)calibrationandscaling, Performancelevelsetting, Developmentofthescorereportingscalesandstudentperformance,and Qualitycontrolprocedures.

BackgroundandPurposeAssessmentsforthefirstoperationaladministrationwereconstructedin2014.ElevenstatesandtheDistrictofColumbiaparticipatedinthefirstadministrationofthePARCCassessmentsduringthe20142015schoolyear.Asmallsubsetofstudentsweretestedinfall2014.ELA/Lgrades9,10,and11,andAlgebraI,Geometry,andAlgebraIIwereadministeredinthefall;theseassessmentswereadministeredonpaperonly.Themajorityofstudentstestedduringthespring2015windowwhenallgradesandcontentareaswereadministeredonlineandonpaper.

ThePARCCassessmentsaredesignedtoachieveseveralpurposes.First,thetestsareintendedtoprovideevidencetodeterminewhetherstudentsareontrackforcollegeandcareerreadiness.Second,thetestsarestructuredtoaccessthefullrangeofCCSSandmeasurethetotalbreadthofstudentperformance.Finally,thetestsaredesignedtoprovidedatatohelpinformclassroominstruction,studentinterventionsandprofessionaldevelopment.

2015TechnicalReport

April18,2016Page2

Thetwooperationaladministrations,Fall2014andSpring2015,ofthePARCCassessmenteachincludedtwoseparatecomponents:thePerformanceBasedAssessment(PBA)andtheEndofYear(EOY)assessment.Bothcomponentswereadministeredcomputerbasedtests(CBT)andpaperbasedtests(PBT)duringthespringtestingwindow;testsadministeredinthefallwerepaperbasedonly.AvalidscoreinboththePBAandEOYassessmentswasrequiredforastudenttoreceiveasummativescore.ItemTypesThetestscontainselectedresponse,briefandextendedconstructedresponse,technologyenabled,andtechnologyenhanceditems(TEI),aswellasperformancetasks.Technologyenableditemsaresingleresponseorconstructedresponseitemsthatinvolvesometypeofdigitalstimulusoropenendedresponseboxwithwhichthestudentsengageinansweringquestions.Technologyenhanceditemsinvolvespecializedstudentinteractionsforcollectingperformancedata.Therefore,theactofperformingthetaskisthewayinwhichdataarecollected.Studentsmaybeasked,amongothertasks,tocategorizeinformation,organizeorclassifydata,orderaseriesofevents,plotdata,generateequations,highlighttext,orfillinablank.OneexampleofaTEIisaninteractioninwhichstudentsareaskedtodragresponseoptionsontoaVenndiagramtoshowtherelationshipamongideas.

ClassicalandIRTItemAnalysisClassicalitemanalysesanddifferentialitemfunctioninganalyseswereperformedonthedatatoevaluatethepsychometriccharacteristicsoftheoperationaltestitemsafteritemswereadministeredandbeforescoreswerereported.Thetwoparameterlogistic/generalizedpartialcredit(2PL/GPC)IRTmodelswereusedforcalibrationsandscaling.1MultipleoperationalcoreformswereadministeredforeachgradeinELA/Landmathematics.Theformsincludedsetsofembeddedcommonitemstoprovidedatatosupporthorizontallinkingacrosstestformswithinagradeandcontentarea.

ThepurposeoftheIRTcalibrationandscalingwastoplacealloperationalitemsforasinglegrade/subjectontoacommonscale.Theresultsofthe2014fieldtestdimensionalitystudyindicatedthatmultidimensionalmodels,basedonpredeterminedteststructures(e.g.,PBAversusEOY,andELA/readingversusELA/Lwriting),didnotprovidesignificantlybettermodelfitcomparedtoaunidimensionalmodel,forbothELA/Landmathematics.Amodecomparabilitystudybasedonthe2014fieldtestdatadidnotprovideevidencetoassumethatscoresresultingfromPBTandCBTformswerestrictlycomparablebetweenmodes,particularlyforPBA.Basedonthefindingsfromthesetwostudies,theoperationaldatawerecalibratedconcurrentlyacrossthePBAandEOYcomponents,andcalibrationswereconductedseparatelyforPBTandCBTresponsedatausingIRTmodelsconsistentwithmixedformatdata.

1IRTmodelcomparisonanalysesconductedusingthe2014fieldtestdataclearlyindicatedthattheitemfitwasbetterforthe2PL/GPCthanthe1PL/PCmodelcombination,forbothELA/Landformathematics.Consequently,therecommendationbyETSandPearsontousethe2PL/GPCmodelsforoperationalcalibrationsandscalingwasapprovedbyPARCC.NoELA/Litemsandrelativelyfewmathematicsitemsweresingleselectmultiplechoiceitems(approximately15%to23%).Consequently,the3PL/GPCmodeldidnotmarkedlyimprovefitwhencomparedtothefitofthe2PL/GPCmodel.

2015TechnicalReport

April18,2016Page3

Aftercalibrationandmodelfitevaluationwascompleted,amasterlistofallitemsflaggedasproblematicwascompiledandbroughttothePARCCPriorityAlertTaskForce.2TheTaskForcereviewedeachitem,itscontentandthestatisticalproperties,andmadedecisionsaboutwhethertoincludetheitemintheoperationalscores.Sometimes,anitemwasrejectedbecauseitappearedtohavecontentissues,andsometimesanitemwasexcludedbecauseitcouldnotbecalibratedorshowedextremelypoorIRTmodelfit.UltimatelythedecisionaboutwhethertokeeporexcludeeachflaggeditemwasmadebytheTaskForce.ThegoalsoftheTaskForcewereto:a)minimizethenumberofitemsexcludedfromtheoperationaltestforms,andb)avoidadvantagingordisadvantaginganytesttakers.

Oncetheitemresponsedatafromthecomputerbasedtests(CBT)andthepaperbasedtests(PBT)werecalibratedforallgradesandcontentareas,allavailableitemparameterestimatesofcommonitemsacrossmodes,wereusedtotransformthePBTitemparameterestimatesontotheCBTscales.ThesoftwareprogramSTUIRT(Kim&Kolen,2004)wasusedtoobtainStockingandLord(1983)transformationvaluestolinkthePBTscalestotheCBTscales.

ThePBTformsforallgradesandcontentareasweregeneratedusingitemsfromtheCBTforms.Inresponsetoseveralpracticalconstraintsbasedonthenumberofformsconstructedforeachmodeandtomeettheblueprints(e.g.,inclusionofTEIonCBTforms),therewasnosingleCBTformthatwasadministeredintactinthepaperdeliverymodeatanygradelevel.Forexample,TEIfromonlineformswerereplacedinthepaperformswithitemshavingsimilarcontent,butappropriateforpaperbasedtesting.However,forbothELA/Landmathematics,thecontentonPBTformssignificantlyoverlappedcontentontheCBTforms.Amodecomparabilitystudywasconductedin2015andtheresultsarepresentedasaseparatespecialreport.ThestudyevaluatedtheextenttowhichscoresfromCBTandPBTformscouldbeconsideredascomparablewithregardtopsychometriccharacteristics.Amajorfindingwasthatscorecomparabilitywasinconsistentacrossthecontentdomainsandgradelevelsinvestigated.OverallScaleScores,ClaimScores,andSubclaimScoresThePARCCELA/Landmathematicsscoresareexpressedasvarioustypesofscalescores(bothtotalscoresandclaimscores,relatedtotheclaimsstructuresdescribedbelow),aswellasbyperformancelevelsusedtodescribehowwellstudentsmeettheacademicstandardsfortheirgradelevel.Onthebasisofastudentstotalscore,aninferenceisdrawnabouthowmuchknowledgeandskillinthecontentareathestudenthasacquired.ThetotalscoreisalsousedtoclassifystudentsintermsofthelevelofknowledgeandskillinthecontentareaasstudentsprogressintheirK12education.Theselevelsarecalledperformancelevelsandarereportedas:

Level5:Exceededexpectations Level4:Metexpectations Level3:Approachedexpectations Level2:Partiallymetexpectations Level1:Didnotyetmeetexpectations

2ThePriorityAlertTaskForcecomprisedParccInc.staff,stateleads,andstatestaff.

2015TechnicalReport

April18,2016Page4

StudentsclassifiedaseitherLevel4orLevel5aremeetingorexceedingthegradelevelexpectations.Additionally,informationonmorespecificskillsisprovidedandisreportedasBelowExpectations,NearlyMeetsExpectations,andMeetsorExceedsExpectations.PARCChasdevelopedperformanceleveldescriptors(PLDs)toassistwiththeunderstandingandinterpretationsoftheELA/Landmathematicsscores(http://www.parcconline.org/assessments/testdesign/elaliteracy/elaperformanceleveldescriptorsandhttp://www.parcconline.org/assessments/testdesign/mathematics/mathperformanceleveldescriptors).Additionally,resourceinformationisavailableonlinetoeducators,parents,andstudents(http://avocet.pearson.com/PARCC/Home#10829),whichincludesinformationonunderstandingandinterpretingtheELA/Landmathematicsscorereports.

TheclaimstructuresforELA/Landformathematics,groundedintheCommonCoreStateStandards,informsthedesignanddevelopmentofthesummativeassessments.

ClaimStructureforELA/LMasterClaim.ThemasterclaimistheoverallperformancegoalforthePARCCELA/LAssessmentSystemstudentsmustdemonstratethattheyarecollegeandcareerreadyorontracktoreadinessasdemonstratedthroughreadingandcomprehendingofgradeleveltextsofappropriatecomplexityandwritingeffectivelywhenusingand/oranalyzingsources.

MajorClaims:1)readingandcomprehendingarangeofsufficientlycomplextextsindependently,and2)writingeffectivelywhenusingand/oranalyzingsources.

Subclaims:ThesubclaimsfurtherexplicatewhatismeasuredonthePARCCassessmentsandincludeclaimsaboutstudentperformanceonthestandardsandevidencesoutlinedinthePARCCevidencetablesforreadingandwriting(http://www.parcconline.org/assessments/testdesign/elaliteracy/testspecificationsdocuments).Theclaimsandevidencesaregroupedintothefollowingcategories.

1. Vocabulary,Interpretation,andUse2. ReadingLiterature3. ReadingInformationalText4. WrittenExpression5. KnowledgeofLanguageandConventions

ClaimStructureforMathematicsMasterClaim.Thedegreetowhichastudentiscollegeorcareerreadyorontracktobeingreadyinmathematics.Thestudentsolvesgradelevel/courselevelproblemsalignedtotheStandardsforMathematicalContentwithconnectionstotheStandardsforMathematicalPractice.

2015TechnicalReport

April18,2016Page5

Subclaims:ThesubclaimsfurtherexplicatewhatismeasuredonthePARCCassessmentsandincludeclaimsaboutstudentperformanceonthestandardsandevidencesoutlinedinthePARCCevidencestatementtablesformathematics(http://www.parcconline.org/assessments/testdesign/mathematics/mathtestspecificationsdocuments).Theclaimsandevidencearegroupedintothefollowingcategories.

SubclaimA:MajorContentwithConnectionstoPractices

SubclaimB:AdditionalandSupportingContentwithConnectionstoPractices

SubclaimC:HighlightedPracticeswithConnectionstoContent:ExpressingMathematicalReasoningbyconstructingviablearguments,critiquingthereasoningofothers,and/orattendingtoprecisionwhenmakingmathematicalstatements

SubclaimD:HighlightedPracticewithConnectionstoContent:Modeling/Applicationbysolvingrealworldproblemsbyapplyingknowledgeandskillsarticulatedinthestandards.

ScoreScalesScalescoresweredefinedforeachtestasalineartransformationoftheIRTtheta()scale.ThetestcharacteristiccurvesassociatedwiththeperformancelevelsettingformswereusedtoidentifythethetavaluesassociatedwiththeLevel2andLevel4pointscores.BydefiningLevel2and4scalescorestobe700and750,respectively,thelinearrelationshipbetweenthetaandscalescoreswasestablished.

Theresultis201definedfullsummativescalescorepointsforeachELA/Landmathematicsassessment,rangingfrom650to850.Ascalescoreof700isalwaystheminimumforLevel2performance,ascalescoreof750isalwaystheminimumforLevel4performance.

ThethresholdsforsummativeperformancelevelsonthescalescoremetricrecommendedbythescalescoretaskforcearedescribedinTable1.

Table1.DefinedSummativeScaleScoresandCutScores Lowest

ObtainableScaleScore

CutScoreLevel2

CutScoreLevel4

HighestObtainableScale

ScoreFullSummative 650 700 750 850

Aswiththefullsummativescores,scalescoresforReadingandWritingweredefinedforeachtestasalineartransformationoftheIRTtheta()scale.ThesameIRTthetascalewasusedforReadingandWritingaswasusedfortheELA/Lfullsummativescores.ThethetavaluesassociatedwiththeLevel2andLevel4performancelevelswereidentifiedusingthetestcharacteristiccurvesassociatedwiththeperformancelevelsettingforms.Paralleltothefullsummativescores,therelationshipbetweenthetaandscalescoreswasestablishedwithLevel2and4thetascoresandthecorrespondingpredefinedscalescores.

2015TechnicalReport

April18,2016Page6

Theresultwas81definedscalescorepointsforReading,rangingfrom10to90.Ascalescoreof30isthecutscoreforminimumLevel2performance,ascalescoreof50isthecutscoreforminimumLevel4performance.Thereare51definedscalescorepointsforWriting,rangingfrom10to60.Ascalescoreof25isthecutscoreforminimumLevel2performance,ascalescoreof35isthecutscoreforminimumLevel4performance.ThethresholdReadingandWritingperformancelevelsonthescalescoremetricrecommendedbythescalescoretaskforcearedescribedinTable2.

Table2.DefinedScaledScoresandCutScoresforReadingandWritingClaimScores Lowest

ObtainableScaleScore

CutScoreLevel2

CutScoreLevel4

HighestObtainableScale

ScoreReading 10 30 50 90Writing 10 25 35 60

Regardingthesubclaimscores,theLevel4cutisdefinedasMeetsorExceedsExpectationsbecausehighschoolstudentsatLevel4orabovearelikelytohavetheskillsandknowledgetomeetthedefinitionofcareerandcollegereadiness.Subclaimoutcomescenteronthatperformancelevelandarereportedatthreelevels:

BelowExpectations; NearlyMeetsExpectations;or MeetsorExceedsExpectations.

QualityControlToensureIRTcalibrations,scalingandconversiontableswereproducedaccurately,PearsonreplicatedtheIRTcalibrationsandscalescoretransformationscarriedoutbyETS,andthegenerationofthescoreconversiontables.WhileETSusedPARSCALE,PearsonindependentlyconductedthesamecalibrationsusingIRTPRO(Cai,Thissen&duToit,2011)calibrationsoftware.ETSandPearsonbothusedSTUIRTsoftwaretotransformtheirPBTitemparameterestimatesontotheCBTscalesforeachgrade/subject.PearsonsscalingconstantswerecomparedtothosegeneratedbyETSandfoundtobeconsistent.MeasuredProgress(MP)performedindependentqualitycontrolcomparisonsbetweentheETSandPearsonitemparameterestimatestoidentifyanydifferences.Inaddition,MPindependentlymadecertainthatthesameitemswereexcludedfromtheCBT/PBTlinkingsets,andcomparedtransformedPBTparameterestimatescomputedbyETSandPearson.Ifitemshadlargedifferencesacrossmodes,theitemswerediscussedandanyremainingissuesresolved.MeasuredProgresspreparedreportsdocumentingtheirfindings.ExactmatcheswerefoundbetweenallETSandPearsonconversiontablesbeforescoreswerereported.

2015TechnicalReport

April18,2016Page7

Section1:Introduction1.1 Background

StatesassociatedwiththePartnershipforAssessmentofReadinessforCollegeandCareers(PARCC)cametogetherinearly2010withasharedvisionofensuringthatallstudentsregardlessofincome,familybackgroundorgeographyhaveequalaccesstoaworldclasseducationthatwillpreparethemforsuccessafterhighschoolincollegeand/orcareers.ThePARCCgoalwastodevelopnewassessmentsthattieintomorerigorousacademicexpectationsandhelppreparestudentsforsuccessincollegeandtheworkforce,aswellastoprovideinformationbacktoteachersandparentsaboutwherestudentsareontheirpathtosuccess.Callingontheexpertiseofthousandsofteachers,highereducationfacultyandothereducatorsinmultiplestates,thePARCCassessmentsystemisahighqualitysetofsummativeassessments,diagnosticassessments,formativetasks,andothersupportmaterialsforteachersincludingprofessionaldevelopmentandcommunicationstools.

ThePARCCconsortiumisdesignedtodevelopandadministernextgenerationassessmentsthat,comparedtotraditionalK12assessments,moreaccuratelymeasurestudentprogresstowardcollegeandcareerreadiness.TheassessmentsarealignedtotheCommonCoreStateStandards(CCSS)andincludebothEnglishLanguageArts/Literacy(ELA/L)assessments(gradesthreethrougheleven)andmathematicsassessments(gradesthreethrougheight,andhighschool).Comparedtotraditionalstandardizedtests,theseassessmentsareintendedtomeasuremorecomplexskillslikecriticalthinking,persuasivewriting,andproblemsolving.

In2013,thePARCCGoverningBoardlaunchedParccInc.,anonprofitorganizationdesignedtosupportthesuccessfuldeliveryofthetestsin201415,andthelongtermsuccessofthemultistatepartnership.Statescontinuetogoverndecisionsabouttheassessmentsystem;thenonprofitorganization

final technical report for 2015 administration 2015 tech report.pdf · final technical report for...

Documents