Using Data to Improve Using Data to Improve Instruction:Instruction:

Building on Models that WorkBuilding on Models that Work Council of Chief State School Officers

St. LouisSeptember 12-14, 2007

Education Leaders Conference

Tools for Aligning Instruction, Standards, Tools for Aligning Instruction, Standards,

& Assessments& Assessments Orientation for Educators

Surveys of Enacted Curriculum

Conference ObjectivesConference Objectives

Present & discuss: Strategies/designs/tools for using data to improve instruction– that have been applied with schools

Analyze findings/evidence about effectiveness of strategies/designs/tools, and discuss potential as model for others

Provide leadership training on how to implement and use data-driven approaches for improving instruction

Panel of State LeadersPanel of State Leaders: Policy to : Policy to Practice Question—Practice Question—

How can States effectively integrate How can States effectively integrate Education Improvement Initiatives Education Improvement Initiatives with Use of Data?with Use of Data?

PANEL Gene WilhoitGene Wilhoit, CCSSO Executive Director, (Facilitator) Gerald Zahorchak, Pennsylvania Secretary of Education Susan Castillo, Oregon Superintendent of Education Todd Flaherty, Rhode Island Department of Education Sally Wherry, Minnesota Department of Education

New members: Session topicsNew members: Session topics What is SEC? Why? How? -- RB What Collaborative does; Tools --RB How to use SEC data charts– Carolyn

Karatzas How to plan an SEC project – Diana

Nunnaley SEC online survey system/alignment –

John Smithson

New Member OrientationNew Member OrientationWhat? … are the Survey of Enacted

Curriculum tools?How?… are data collected, analyzed,

reported?Why?… are SEC data useful to educators,

leaders, researchers?Now what?…how do we implement locally

and lead the use of SEC in schools?

STATE ofSTATE of

MissourA

MissourEE

SEC State Collaborative Members Planning Meeting

St. Louis September 11, 2007

Surveys of Enacted Curriculum

RationaleRationale

AssessmentStandards

Curriculum

Applications Applications

Alignment analysis --instruction, standards, assessments

Instructional improvement in schools Needs assessment/ Evaluation Indicators – monitoring change over time

Multi-State CollaborativeMulti-State Collaborative

Build Knowledge of leaders Share Strategies Produce joint products, tools

SEC Collaborative Members2007-08

•Delaware•Idaho•Illinois•Indiana•Iowa•Kansas•Maine•Michigan

•Mississippi• Montana• Ohio• Oklahoma• Oregon• Wisconsin• Vermont*• Minnesota*

Social Studies SEC Development2005-2007

• Arizona• Delaware• Idaho• Maine•Ohio•Oregon•Vermont•Wisconsin• Minnesota• Nevada

•U of Wis-WCER• NCSS• Civics Education•National Geographic•GENIP•NCEE•CCSSO

SEC Collaborating OrganizationsSEC Collaborating Organizations

•Council of Chief State School Officers www.SECsurvey.org

•Wisconsin Center for Education Research www.SEConline.org

•Learning Point Associates/NCREL www.SECsupport.org

•TERC Regional Alliance DEC Project www.ra.terc.edu/DEC

SEC Collaborative offersSEC Collaborative offers

Core Components of SEC tools/services1) Surveys with teachers – report instruction2) Alignment analysis – content code standards

and assessments3) How to use Data – knowledge, skills4) Leader development – 2-3 meetings/yr4) In-state workshops: a) orientation b) use of data

SEC CollaborativeSEC CollaborativeStates, Districts

CCSSO – SEC Collaborative Management Rolf Blank, Carlise Smith

WCER – Surveys, Reporting, Alignment analysis John Smithson, Alissa Minor, Eric Osthoff

Training & Technical AssistanceCarolyn Karatzas, Consultant, CCSSODiana Nunnaley, TERC Gary Money, Maren Harris, Learning Point Associates

Lani Seikaly, Hillstreet & MainJennifer Unger, The Groupworks

Michael Erhlinghaus, Measured Progress

SEC by the Numbers: ’06-07SEC by the Numbers: ’06-07

10,393 SEC Surveys 06-07 4674 Math

1993 Science 3646 ELAR

80 Soc Stud 131 Standards/Assess. Content

analysis 1000 est. schools--Data use

SEC by the Numbers (2)SEC by the Numbers (2)

• By Comparison 1996 207 Science Teachers (5 states)

1999 626 M/S Teachers (11 states) 2005 5,414 E, M, S teachers 2006 10,200 E, M, S teachers

2007 10,400 E, M, S, SSt (27 states)•

SEC CollaborativeSEC Collaborative

How SEC Collaborative works for members Training and Assistance to project leaders Project Planning w/ budget– meet local objectives Each state/district makes decisions on services &

project plan Multi-state benefits of collaborative

e.g., learning from experience, sharing ideas, training

AccomplishmentsAccomplishments06-0706-07

SEC CollaborativeSEC Collaborative In-state Leader workshops, Project Planning PPTs for Orientation, Survey Admin, Data

use Leader Development Standards document Website revisions, improvements

(SECsurvey.org; SEConline.org)E.g., State Projects, All Alignment results,

Report generator

More AccomplishmentsMore Accomplishments SEC CollaborativeSEC Collaborative

“Using Data to Improve Instruction” Boulder, CO conference – 200 leaders, 32 sessions (30 states)

Social Studies Survey development – 10 states, 5 prof. orgs.,

NAEP Alignment analysis – 2007 Math items Research/evaluation:

MSP study of professional development– multi-site study Ohio case study of district/school implementation of SEC Follow-up data tool –to track school level data Validity study—longitudinal data for teachers and students

Ohio reading study John demonstrated SEC comparisons in Feb 07

Goals SEC 06-07Goals SEC 06-07

State/local projects—From pilots to broad application, local leadership training

Integrate SEC tools/data with Improvement Initiatives, especially data-driven

Research/Evaluation --Broader use of Surveys, Alignment

Assist States in planning with Service delivery agencies; SEC team with CACs & Labs

Social Studies SEC complete & implement

Key Question -- SEC ToolsKey Question -- SEC ToolsResearch into PracticeResearch into Practice

How can Educators obtain reliable, valid data to determine Alignment of instruction with required standards and assessments?

• School & Class Description• Instructional Activities

• General• Problem Solving Activities

• Pairs & Small Group Work• Use of Hands-on Materials• Use of Calculators/Computers & other Ed. Tech.

• Assessment Use• Instructional Influences

• Instructional Readiness• Teacher Opinions• Professional Development

• Types, Frequency• Content , Active,• Collegial, Coherence

• Instructional Content•Topic x Cog. Demand

Survey Sections

Steps in SEC DevelopmentSteps in SEC Development R&D studies curriculum-- Early ‘90s Models:TIMSS, NAEP, Analyze assessment OTL interest of States, Porter/ Smithson research, CCSSO Science Assessment project State collaborative –Develop surveys-’98-’01 (NSF)

11-State field study, Reports, Alignment, USI Use of data experiment – DEC (NSF/ROLE:‘01-04) Evaluation: MSP RETA PD Study (‘02 – 05, NSF) English/ Language Arts survey (‘03 – 04) Current projects: 15 States, 4 Districts, 5 MSPs

Questions Addressed by SEC DataQuestions Addressed by SEC Data

How can in-depth data on content of instruction be collected and reported–not topic checklists, to analyze teaching content in relation to standards, assessments, achievement?

How can methods of teaching practices be compared across classrooms, schools, districts, and states?

How can enacted curriculum data be reported in a manner to encourage use by teachers to improve instruction?

MSP PD Study SitesMSP PD Study SitesBrockport/Rochester – SUNYCleveland Municipal SDCorpus Christi – AIMS - TX A&MEl Paso – UTEP State MSPs: ID, IL, OR, OK, ME, MT

NSF MSPs: Boston NJ MSP -- 12 districts

NC MSP -- 17 districts SW PA MSP

SCALE – Los Angeles

Key Education Questions Key Education Questions (cont’d)(cont’d)

How can we measure the effects of standards-based initiatives on instructional practices and curriculum in classrooms?

How can we analyze effectiveness of professional development on changes in teachers’ instructional practices? (i.e. determine the quality of professional development)

What if…What if…

[for Intro of SEC to Educators]You could use data on instructional quality and content to guide professional development?

You could have consistency across grade levels?

You could know how well aligned your state standards were to the state assessment?

You could compare how you teach content compared to how others across the nation teach?

You could use anonymous teacher data to start a powerful school discussion about what the teacher needs are?

Uses a multi-dimensional language fordescribing instructional content

SEC Instructional Content

Topics

by Cognitive Demand(Expectations for Student Learning)

Categories of Cognitive Demand

Topics Memorize Peform

Procedures Communicate Understanding

Sovle non-routine

problems

Conjecture/ Generalize/

Prove Multiple

Step Equations

Inequalities

Literal Equations

Lines / Slope and Intercept

Operations on

Polynomials

Quadratic Equations

Content MatrixContent Matrix

Categories of Cognitive Demand

Topics Memorize

facts, defintions

Conduct Invest/ Perf Procedures

Communicate Understanding of sci concepts

Analzye information

Apply concepts/ Connnect

Nature of Science

Meas/Calc. in Science

Life Science

Physical Science

Earth Science

Chem/Biol/ Physics

(HS)

Science content matrixScience content matrix

MemorizePerform

CommunicateConjecture

Connect0

0.020.040.060.08

0.1

Operations

Number Sense

Measurement

Geometric Concepts

Algebraic Concepts

Data Analysis

Instructional Technology

MemorizePerform

CommunicateConjecture

Connect

State J Grade 8Mathematics Instruction

Content Maps

Surveys of Enacted Surveys of Enacted CurriculumCurriculum

A neutral content grid

with cognitive demand

The intended curriculum: State content standards—What students should learn

The enacted curriculum: What teachers teach

The learned curriculum: Student outcomes based on school learning

The assessed curriculum: State (and other) assessments—tested learning

The The EnactedEnacted Curriculum Curriculum

SEC Websites SEC Websites

CCSSO, SEC Collaborativewww.SECsurvey.org

WCER, SEC Online Survey and Reports www.SEConline.org