data interpretation workshop 2007. advancing assessment literacy modules: data interpretation...

Data Interpretation Workshop

2007

Advancing Assessment Literacy Modules: Data Interpretation Workshop (February 2008) 2

Purposes for the Day

• Bring context and meaning to the math and reading assessment project results;

• Initiate reflection and discussion among school staff members related to the math and reading assessment results;

• Encourage school personnel to judiciously review and utilize different comparators when judging math and reading assessment results;

• Model processes that can be used at the school-and division-level for building understanding of the data among school staff and the broader community; and,

• Provide an opportunity to discuss and plan around the data


Agenda

• Understanding data—sources, categories & uses

• Saskatchewan Assessment for Learning Program

• Working with the AFL Reports– Standards and Cut Scores– Predicting– Sharing– Analysis of Strengths & Areas for Growth– Creating & Testing Hypotheses

• Action Planning


32°

• What might the above piece of data mean?

• While 32° is data, the meanings you provided were interpretation.

• All data is meaningless until interpreted.

Wellman, B. & Lipton, L. (2004). Data driven dialogue. Mira Via, LLC.


A Data-Rich Environment

Wellman & Lipton (2004) state:

Schools and school districts are rich in data. It is important that the data a group explores are broad enough to offer a rich and deep view of the present state, but not so complex that the process becomes overwhelming and unmanageable.



International Data Sources

• Programme for International Student Assessment (PISA)

http://snes.eas.cornell.edu/Graphics/earth%20white%20background.JPG


National Data Sources

• Pan-Canadian Achievement Program (PCAP)

• Canadian Test of Basic Skills (CTBS)

• Canadian Achievement Tests (CAT3)http://www.recyclage.rncan.gc.ca/images/canada_map.jpg


Provincial Data Sources

• Assessment for Learning (AFL)– Opportunity to Learn

Measures– Performance

Measures

• Departmentals

http://regina.foundlocally.com/Images/Saskatchewan.jpg


Division Data Sources

• Division level rubrics

• Division bench mark assessments

http://www.sasked.gov.sk.ca/branches/ed_finance/north_east_sd200.shtml


Local Data Sources

• Cum Folders

• Teacher designed evaluations

• Portfolios

• Routine assessment data


Nature of Assessment Data

From Understanding the numbers. Saskatchewan Learning

Definitive Indicative

Individual Classroom School Division Provincial National International

Student Evaluations System Evaluations

12

.

Depth and Specificityof Knowledge

From Saskatchewan Learning. (2006). Understanding the numbers.

Little knowledge ofspecific students

In-depth knowledge of specific students

Individual NationalSchoolClassroom InternationalDivision Provincial

Assessments

In-depth knowledge of specific students

Individual NationalSchoolClassroom InternationalDivision Provincial

Assessments

In-depth knowledge of systems


Assessment for Learningis a Snapshot

• Results from a large-scale assessment are a snapshot of student performance.– The results are not definitive. They do not tell the

whole story. They need to be considered along with other sources of information available at the school.

– The results are more reliable when larger numbers of students participate and when aggregated at the provincial and division level, and should be considered cautiously at the school level. Individual student mastery of learning is best determined through effective and ongoing classroom-based assessment. (Saskatchewan Learning, 2007)


Using a Variety of Data Sources

• Thinking about the data sources available, their nature and the depth of knowledge they provide, how might the information in each impact the decisions you make?– What can you do with this data?– What is its impact on classrooms?

• Please refer to the “Using a Variety of Data Sources” template in your handout package as a guide for your discussion.


Local Level Sources of Data

While international, national and provincial sources of data can provide direction for school initiatives, the data collected at the local level is what provides the most detailed information regarding the students in classrooms.


Four Major Categories of Data: Demographics

• Local Data– Descriptive information

such as enrollment, attendance, gender, ethnicity, grade level, etc.

– Can disaggregate other data by demographic variables.

• Assessment for Learning– Opportunity-to-

Learn Data• Home support for

– learning

– reading,

– and learning math.

Bernhardt, V. L. (2004). Data analysis for continuous school improvement, 2nd Edition. Larchmont, NY: Eye on Education.


Four Major Categories of Data: Student Learning

• Local Data– Describes outcomes in

terms of standardized test results, grade averages, etc.

• Assessment for Learning Data:– Opportunity-to-Learn

Data• Know and use reading

strategies

– Student performance outcomes

• Math 5,8,20• Reading 4,7,10



Four Major Categories of Data:Perceptions

• Local Data– Provides information

regarding what students, parents, staff and community think about school programs and processes.

– This is data is important because people act in congruence with what they believe.


Learn Data• Preparation for and

commitment to learn• Persistence



Four Major Categories of Data:School Processes

• Local Data– What the system and

teachers are doing to get the results they are getting.

– Includes programs, assessments, instructional strategies and classroom practices.


Learn Data• Instruction and learning• Availability and use of

resources• Approaches to problem

solving (Math)



What Data are Useful & Available?

• Think about the goals/priorities set within your school and/or school division – they might be achievement, growth, behavioural, etc.

• Using the supplied template, begin to catalogue the data you already have and the data you need in order to better address the goals that have been set.

• An example follows on the next slide.

Questions

What data do you have answer questions?

What other data do you need to gather?

Demographics •Enrollment by subject • Number of teachers teaching reading skills

Perceptions • Student profiles • School Community Council feedback

•Student perception of their success

Student Learning •CAT3

•AFL

•Student use of reading skills

•Reading skills explicitly taught in each subject

School Processes • Current instruction in reading skills

• Impact of a focused reading skills program


Goal: Students will experience a greater success in all subject areas as we focus on reading skills.


Principles of the Saskatchewan AFL Program

1. Cooperation and Shared Responsibility2. Equity and Fairness3. Comprehensiveness4. Continuous Improvement that Promotes

Quality and Excellence5. Teacher Professionalism6. Authenticity and Validity7. Honesty and Openness


Principles of the Saskatchewan AFL Program

• Read and Connect– Everyone at the table reads the first

numbered statement.• After reading, one person at the table offers an

insight or connection they are making to that statement.

– Repeat the process with all seven statements.

• At the end, discuss the key ideas and concepts within the principles.


Comparators: Types of Referencing

• Criterion-referenced: Comparing how students perform relative to curriculum objectives, level attribution criteria (rubrics) and the level of difficulty inherent in the assessment tasks.(Tables 8.8 & 8.12)

• Standards-referenced: Comparing how students performed relative to a set of professionally or socially constructed standards.(Figure 8.2c, Table 8.3, Figure 8.4a, Figure 8.6a, and others.) From: Saskatchewan Learning. (2006). Understanding the Numbers



• Experience or self–referenced: Comparing how students perform relative to the assessment data gathered by teachers during the school year.(E.g.. Comparing these results to current school data. The standards set by the panel.)

• Norm-referenced: Comparing how students in a school performed relative to the performance of students in the division, region or project.(E.g.. Tables comparing the school, division and province.)



• Longitudinal-referenced: Comparing how students perform relative to earlier years’ performance of students.

(Table 8.1, Figure 8.3 and others.)


What Data are Collected and Reported for Reading?

• Reading Comprehension Skills: 60-item multiple-choice test (organized by reading strategies). The six categorized reading strategies are:– Using Cueing Systems– Connecting to Prior Knowledge– Making Inferences, Predictions, and Drawing

Conclusions– Noting Key & Finding Supporting Ideas– Summarizing, Recalling, synthesizing and Organizing

Information– Recognizing Author's Message & Craft

28

What Data are Collected and Reported for Reading?

• Explicit Comprehension: a subset of the 60-item multiple-choice test involving responses to ideas or information stated directly in the text. The answers are “right there” in the text.

• Implicit Comprehension: a subset of the 60-item multiple-choice test requiring the reader to apply background knowledge to interpret or infer ideas or information in the text. Interpreting vocabulary or visuals and making predictions are forms of inference.

• Critical Comprehension: a subset of the 60-item multiple-choice test involving responses to ideas and information that require inferences and critical analysis. Looking at author's purpose and point of view, distinguishing facts from opinions and recognizing persuasive techniques are all components.

– Numbers of questions in the subsets vary according to grade level

• Reader Response: written-response question(s) assessing students' ability to make meaning from text by making connections to personal knowledge or experience (extending and applying new understandings


Opportunity to Learn Measures for Reading

• Data was collected from students in 3 areas:– Preparation and commitment to learn– Knowledge and use of reading strategies– Home support for reading

• Data was collected from teachers reported on 2 classroom-related elements:– Availability and use of resources– Instruction and learning

30

What Data are Collected and Reported for Math?

• Math Content Skills: 40-item multiple-choice test (organized by mathematical strands and linked to curriculum objectives)... The five strands vary by grade level.

Grade 5 Grade 8 Grade 11•Whole Numbers•Fractions•Geometry•Measurement•Data Management

•Data Management•Numbers & Operations•Ratio & Proportion•Geometry & Measurement•Algebra

•Irrational Numbers•Consumer Math & Problems•Polynomials & Rationals•Quadradic Functions & Equations•Angles, Polygons and Circles


What Data are Collected and Reported for Math?

• Applications & Problem Solving• Concepts, Procedures & Relationships

• Challenges – Performance on challenges is reported on a 5 level scale.

• Calculator Skills • Computation Skills• Estimation Skills


Opportunity-to-Learn Measures in Math

• Data was collected from students in 4 areas:– Preparation and commitment to learn– Persistence when experiencing difficulty– Home support for learning in general– Home support for learning math

• Data was collected from teachers reported on 3 classroom-related measures:– Availability and use of resources– Instruction and learning– Approaches to problem solving


Standards

To help make meaningful longitudinal comparisons in future years, three main processes will be implemented.

1. Assessment items will be developed for each assessment cycle using a consistent table of specifications.

2. The assessment items will undergo 3 rounds of field-testing - one of which is intended to inform the comparability of the two assessments.

3. A process for setting of standards for each of the assessment items, so that any differences in difficulty between two assessments are accounted for by varying standards for the two assessments.


Opportunity-to-Learn and Performance Standards

• In order to establish Opportunity-to-Learn and Performance standards for the 2007 Reading Assessment, three panels were convened (one from each assessed grade), consisting of teachers and post-secondary academics including Education faculty.

• The panelists studied each assessment item from the 2007 assessment in significant detail established cut-scores for each of the assessment components.


Cut Scores

• On page 4 of the detailed reports you will find the cut scores detailing the percentage correct required for students to be classified at one of two levels:– Threshold of adequacy– Threshold of proficiency

• Reader response and Math challenge scores are presented on a five-level scale (1-low to 5-high).


Locating Cut Scoresin the Report

• Turn to pg. 4 in the detailed report for your grade level.

Opportunity-to-Learn Elements

Performance Component

Excellent Standard

Sufficient Standard

Proficient Standard

Adequate Standard

A score out of 5 (1-low to 5-high) % correct required to reach the standard

• You will need to refer to these scores during the following prediction activity.

39

PredictingAFL Math OTL – R & M AFL Reading

On the chart for your grade level, predict how many students achieved the ADEQUATE Standard on the 2007 AFL in Math for each strand:•Math Content Skills•Integrated Applications•Estimation Skills•Calculator Skills•Computation Skills

On the reverse are the Math Content Skills.

On the charts, predict what percentage of students had SUFFICIENT opportunity to Learn in the following areas:

Reading:•Student preparation and commitment to learn•Student knowledge & use of reading strategies•Home support for reading

Math:•Student preparation and commitment to learn•Student persistence•Home support for learning•Home support for learning math

On the charts, predict what percentage of students met the ADEQUATE Standard in the following strands:•Reading Comprehension•Explicit Comprehension•Implicit Comprehension•Critical Comprehension•Reader Response

On the reverse are the Reading Comprehension Skills.

AFL MathPredictions of Percentage Correct

0

10

20

30

40

50

60

70

80

90

100

Num & Op Ratio & Prop Geom & Meas Data Mng't Algebra CPR Q's APS Q's

AFL Math

Percentage of Students who met the Adequate Standard set by Saskatchewan Educators

AFL MathPredictions of Percentage Correct

0

10

20

30

40

50

60

70

80

90

100

Num & Op Ratio & Prop Geom & Meas Data Mng't Algebra CPR Q's APS Q's

Shade in your prediction on the supplied prediction chart.


AFL Math

Percentage of Students who met the Adequate Standard set by Saskatchewan Educators


Predicting

Based on your predictions, create a set of hypotheses for some or all of them. As you create each prediction, identify the underlying assumptions.

• Prediction – ‘X’ will contain the highest scores.• Assumption – we created a common assessment for ‘X’ in 2005.

• Prediction – students will report higher on geometry because we moved that unit earlier in the year.

• Assumption – there are fewer classroom interruptions earlier in the year and students have more time to learn the material.

Write each prediction and its accompanying assumption on the cards provided. Please write legibly.


Sharing

• Gather the cards together at your table and discuss the predictions and assumptions.– Do these statements ring true for everyone at your

table? School? Division?– Considering all of the predictions, are there any

themes or patterns emerging?• Why might this be?


Comparisons

The completed bar graphs are in the Summary Report on page 3.

– What are you noticing about the data?– What surprised you?

•What other data would you like to see to better inform the results you’ve seen so far?



Using the Prediction & Comparison Process

• What are the benefits of approaching data in this manner?

• Hints


Examining the Report

• Take a few minutes to look through the entire AFL report. Use the chart below to guide your thinking and conversation.

Performance Data

OTL Data

What pops out? Strengths, Areas of Improvement

Questions???


Designing Interventions

• Assumptions must be examined because our interventions will be based on them.

• We must strive to correctly identify the causal factors.

• Don’t fall in love with any theory until you have other data.

• Use a strength-based approach to interventions.


Team Action Plan/Fishbone

Team Action Plan• What are some areas of

strength indicated within your data?

• What are some areas for improvement indicated within your data?

• Please consider all aspects of both reports including the Opportunity to Learn Measures.

Fishbone• At your table, analyze

one strength and consider all contributing factors that led to that strength.

• Consider one area of improvement and transfer those elements from your area of strength (as applicable) that could contribute to improvement in this area.


Focusing on Improvement

• Thinking of the area of weakness you identified earlier, create a hypothesis that uses some of the positive elements from the area of strength that addresses an area for improvement.– Similar to process lists in math, providing

detailed strategy lists for student reading will increase the number of strategies used and therefore increase comprehension.


Different Lenses

• What other types of data might be required to gain a clearer picture of how specific groups of students are doing?

• How might this data shed a different light on what has been discovered so far?

• To do this, Bernhardt’s triangulation process will be used.


Applying Data Collectionand Triangulation

• Using the hypotheses created from the initial examination of the data, complete a more detailed analysis thinking about the four categories of data available – demographics, perceptions, student learning and school processes.

• You will be furnished with a data intersections template.

53

What types of strategies are used in Reading?

What data is available or needed to answer

the question?

What data collection tools will be required?

Questions

Intersections

(Colour in and name the

intersection)

D

P

SL

SP

Adapted from: Bernhardt, V. L. (2004). Data analysis for continuous school improvement, 2nd Edition. Larchmont, NY: Eye on Education.

Hypothesis: Similar to process lists in math, providing detailed strategy lists for student reading will increase the number of strategies used and therefore increase comprehension.

Place the question in the questions column

54

Available Data:

•Processes from the curriculum guide.

•Processes outlined in the AFL documents.

Data Needed:

•Survey teachers.

What types of strategies can be used in Reading? Which are teachers using? (School Processes-SP)


the question?


Questions

Intersections


intersection)

D

P

SL

SP


Identify the intersection within the question


55

Available Data:•Student reading journal responses•Student questionnaire data from AFL

Data Needed:•Student opinions gathered via questionnaire.

What Reading strategies do students seem to favour? (School Processes by Perceptions)


the question?


Questions

Intersections


intersection)

D

P

SL

SP


What other intersections and data sources might enrich the analysis?


56

Available Data:

•Student reading journal responses.

Data Needed:

•Pre and post test data for all reading strategies.

Which Reading strategies from the curriculum produce the greatest gains in comprehension for students? Which of those were favoured by students? (School Processes by Perceptions by Student Learning)


the question?


Questions

Intersections


intersection)

D

P

SL

SP




57

Available Data:

•Student reading journal responses by gender.

Data Needed:

•Pre and post test data for favoured reading strategies by grade level.

Which of the Reading strategies from the curriculum produce the greatest gains in comprehension for students? Which of those were favoured? Are certain strategies more powerful at different grade levels? (School Processes by Perceptions by Student Learning by Demographics)


the question?


Questions

Intersections


intersection)

D

P

SL

SP





Refining Questions

• Take each hypotheses and write it at the top of a worksheet.• Write questions of the hypothesis in the questions column.• In the left column, use the diagram to identify the intersections

or triangulations implied within the question.– What other intersections would increase the specificity of

this question?– If necessary, rewrite the question to reflect the new

intersections or triangulations.– Including an “over time” element to a question expands the

data that might be accessed to answer the question.• For each question identify the existing data source available

or what tool will be required to collect it. • Create new questions using a variety of the intersections

identified earlier.


Refining and Finalizing Questions

• Complete the process for each question created.• When finished, evaluate the quality of the

questions then decide which should go forward and which should be abandoned.

• On the sheet provided, write the hypothesis and the refined questions that your group has decided to keep.

• These questions will be used to gather data as more in depth goal statements are created.

• These questions will be used to guide the work in your schools to improve student results.


Advancing Assessment Literacy Modules

• 17 Modules designed to faciliate conversations and work with data for improvement of instruction.

• www.spdu.ca– Publications

• Advancing Assessment Literacy Modules

• Download a PDF of a PowerPoint and accompanying Lesson Plan for use by education professionals in schools.

http://www.spdu.ca/

Advancing Assessment Literacy - Module Selection Matrix The intended audience of each module is indicated below. Modules marked Informational Activity are to demonstrate the processes to groups that will not be making school level decisions.

Module

Total Time* Teachers In School

Administration

School Community

Council School Boards

Central Office Staff

Setting the Stage: I Engaging Stakeholders 3 hours ● ● ● ● ● II Understanding Data Purposes and Uses 3 hours ● ● ● ● ● III Freedom of Information and Protection of

Privacy Act for Central Office 2 hours 20 minutes

●

III Freedom of Information and Protection of Privacy Act for General Audience

45 minutes ● ● ● ●

IV Building Learning Communities 3.5 – 6 hours ● ● ● Data Gathering:

I Establishing Outcomes 3 – 3.5 hours

● ● Informational

Activity Informational

Activity

II Creating Questions 1 hour



Activity

III Identifying and Valuing Different Types of Data 1.5 – 2 hours



Activity

IV Collecting and Collating Data 1 – 1.5 hours ● ● Data Analysis: I Summarizing, Representing and Sharing Data 1.5 hours ● ● ● ● ● II Examining and Interpreting Data 2.5 hours ● ● III Extending the Assessment 1 – 1.5 hours ● ● Data Informed Decision Making: I Building a Collaborative Culture 2 hours ● ● ● ● ● II Goal Setting 2 hours ● ● III Creating Action Plans 2.5 – 3 hours ● ● IV Monitoring and Assessing Progress 2.5 – 3 hours ● ● ● Designing Interventions 3 – 3.5 hours ● ● ● Continuing the Conversation 3 – 3.5 hours ● ● ● ● ● * All modules are broken into smaller units of time so that the material can be delivered in a variety of formats – staff meetings, PLC meetings, half-day, full-day, etc.


Reflection

Individually, complete the following stems:

– Key concepts I have learned . . .

– Implications for me . . .

– Questions I have . . .


Assessment: Linking Purposes to Outcomes

• Bring context and meaning to the math and reading assessment project results;

• Initiate reflection and discussion among school staff members related to the math and reading assessment results;

• Encourage school personnel to judiciously review and utilize different comparators when judging math and reading assessment results;

• Model processes that can be used at the school and division level for building understanding of the data among school staff and the broader community; and,

• Provide an opportunity to discuss and plan around the data.

data interpretation workshop 2007. advancing assessment literacy modules: data interpretation...

Documents