Video

Video

Assessment 2.0: New & Next-Gen Measurements of

Learning

Jonathan MartinHead of School

St. Gregory College PrepTucson, AZ

www.21k12blog.net

How do we assess and measure student learning in ways that advance, rather than stifle, student creativity and innovation?

“How do we measure what we value rather than value what we measure?”

How do we ensure we don’t constrain our teachers?

What about Students?

Let’s commit ourselves to experimentation and

innovation, and to maintaining focus on strong

learning outcomes.

Let’s use assessment and measurement to

lead and manage educational innovation.

Less Actually, More Uncertainty

Measurement isn’t going away: it isn’t likely to diminish in importance in the coming years. Funders, Foundations, Parents/Consumers, Boards

Commission on Accreditation Criterion 13

The standards require a school to provide evidence of a thoughtful process, respectful

of its mission, for the collection and use in school decision-making of data (both

external and internal) about student learning

Broaden the Gauge beyond mandated testing, SAT scores & the College List.

Balance the Portfolio

Broadening Internal Assessments of Learning

Demonstrations of Learning: “What you do, not what you know, is the ultimate test of education.” ~PFB

Tweet

Conduct a fluent conversation in a foreign language about of piece of writing in that language.

Declaim with passion and from memory a passage that is meaningful, of one’s own or from the culture’s literature or history.

Invent a machine or program a robot capable of performing a difficult physical task.

Character and 21st c. skills report cards

www.21k12blog.net

St. Gregory’s “egg”

• Character– Integrity

• Leadership– Positive role model– Facilitating Collaboration

• Scholarship– Inquisitiveness– Analytic thinking– Synthetic thinking– Critical thinking

• Innovation – Adaptability – Initiative – Experimentation www.21k12blog.net

Digital Portfolios• High Tech High Digital Portfolios• New Tech Network Digital Portfolios

New and Next Gen. External Measurements of learning

The questions

• Does the assessment measure what we value?• Does it constrain creativity, narrow curriculum,

or restrict teacher latitude?• Are our students meaningfully engaged in the

experience of the assessment? • Does it provide data we can use for

accountability, communications, and continuous improvement of learning?

• Is it affordable?

In “Crime Reduction” students act as a consultant to Mayor Pat Stone who is running for reelection.

As Mayor Stone’s consultant, students are asked to weigh evidence to analyze the strengths and weaknesses of two policy initiatives.

Crime Statistics

Research Brief

Critical thinkingAnalytic reasoning

Problem solvingWritten communication

Three MetricsInternal growth at a school

Comparison to other participating schoolsCollege readiness

CWRA/CLA median percentile, College Freshman Normed

http://www.youtube.com/watch?v=Y4qgDXUrc4w

CWRA: the good and the bad

• Great Goals: A test worth teaching to; measuring what we value.• Does Not Constrain or Narrow Teaching or Diminish creativity &

innovation. • Authentic assessment: Positive testing experience for students. • Student motivation can be an issue. • Results: Good for communications and accountability, not

detailed enough for continuous improvement or student differentiation.

• Pricy

Note

Experiment with developing internally designed, administered, and assessed performance task assessments.

Some schools and districts are finding this enormously valuable for teachers, such as Virginia Beach.

CLA in the Classroom provides workshops on this.

Measures of Academic Progress™

(MAP)

• Traditional/Conventional Standardized Testing Updated

• Computer Adaptive Assessment• More frequent (3x) than once a year• More rapid feedback to teachers• More detailed analysis of student proficiency

Sample Math Question

• Notice the calculator.

• To use it, click the numbers with your mouse.

Sample Math Question

• Notice the calculator.

• To use it, click the numbers with your mouse.

Sample Reading Question

• Read the passage and select the best answer

• Click the <Go on> button

Sample Reading Question

• Read the passage and select the best answer

• Click the <Go on> button

• Measures what we value? Yes and no. More basic skills than higher order thinking skills.

• It can narrow teaching and curriculum, perhaps diminish creativity and innovation, by its emphases.

• Test does conform itself to each student’s individual proficiency level.

• Student experience of testing, 3x a year, certainly wears on kids. • Results are very comprehensive and detailed, and can be very

useful for continuous improvement and differentiation, but big learning curve.

• Moderately priced.

STG 2009 STG 2010 All Schools (Averaged)

8690

75

I have opportunities to be creative in the classroom

Percentage Strongly Agreeing

All Schools St. Gregory 09 St. Gregory 10

7282

92

We regularly discuss questions with no clear answers

What Excites & Engages Me?

Lecture Discussion and Debate

Group Projects Projects Involving Technology

28

6560

55

All HSSSE students

• Measures what we value. • Does Not Narrow or Constrain• Not painful for students, but does fatigue in

subsequent administrations. • Results useful for accountability and communications,

but limited in value for continuous improvement. • Highly affordable.• Middle School Survey (MSSSE) coming soon.

student, teacher and parent surveys collect data and generate reports focused on school climate, classroom conditions, and student engagement

42

Care

Control

Clarify

Challenge

Captivate

Confer

Consolidate

Control• Students in this class treat the teacher with

respect• My classmates behave the way the teacher wants

them to• Our class stays busy and doesn’t waste time

Challenge• My teacher asks students to explain more about

the answers they give.• My teacher doesn’t let people give up when the

work gets hard.• In this class, we learn to correct our mistakes.• My teacher wants us to use our thinking skills, not

just memorize things.

Tripod Seven Cs Sample Survey Items

Student Engagement Targets(Both Individual and Collective)

1. Trust (vs. Mistrust)

2. Cooperation (vs. Resistance)

3. Ambitiousness (vs. Ambivalence)

4. Diligence (vs. Discouragement or Disengagement)

5. Satisfaction and Efficacy (vs. Disappointment and Failure)

The Seven C’sWhat Teachers Do (What Students Experience)

1. Caring about students (Encouragement and Support)

2. Captivating students (Learning seems Interesting and Relevant

3. Conferring with students (Students Sense their Ideas are Respected)

4. Controlling behavior (Culture of Cooperation and Peer Support)

5. Clarifying lessons (Success Seems Feasible)

6. Challenging students (Press for Effort, Perseverance and Rigor)

7. Consolidating knowledge (Ideas get Connected and Integrated)

Teacher Professional Learning (PLCs) Content Knowledge

Pedagogic Skill Relationship-Building Skills

The Tripod

STUDENT ACHIEVEMENT OUTCOMES

Agreement with selected statements by students in 25th and 75th percentile classrooms from the MET secondary school sample.

( From among 2985 classrooms, each with at least 5 students responding)

25th Percentile

75th Percentile

1. CARE: My teacher in this class makes me feel that s/he really cares about me

40 73

2. CONTROL: Our class stays busy and doesn’t waste time.

36 69

3. CLARIFY: My teacher explains difficult things clearly. 50 794. CHALLENGE: My teacher wants me to explain my answers – why I think what I think.

59 83

5. CAPTIVATE: My teacher makes learning enjoyable. 33 726. CONFER: My teacher wants us to share our thoughts. 47 797. CONSOLIDATE: My teacher takes the time to summarize what we learn each day.

38 67

Beyond test scores: Healthy dispositions and experiencing a high quality of life in school

Happiness in classGood behavior in classHealth vs. unhealthy responses to social pressures (e.g., holding back, hiding effort, help avoidance, etc.) Self-consciousness about being perceived as smart or not

by peersAcademic engagement/effort Satisfaction with performance and progress

46

What Works in the Classroom? Ask the Students

Teachers whose students described them as skillful at maintaining classroom order, at focusing their

instruction and at helping their charges learn from their mistakes are often the same teachers whose students learn the most in the course of a year, as measured by

gains on standardized test scores, according to a progress report on the research.

Example: Happiness Index

1. This class is a happy place for me to be.2. This class feels like a happy family.3. I feel respected in this class.4. Being in this class makes me feel angry.

The next slide shows the relationship to the Seven C’s for this measure of happiness.

48

1 2 3 4 5-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

Happiness in Class(1260 secondary school classrooms in an urban school

district.)

GPA_DGPA_CGPA_BGPA_A

Classroom Quintiles in the Seven C's Distribution

S. D

. Uni

ts

Tripod Thoughts

• Measures What We Value• Does not narrow. • Not painful for students, but a large task to

administer surveys. • Seems it can provide valuable data for both

communications and continuous improvement.

• Pricing varies.

Information Literacy: “To be information literate, a person must be able to recognize when information is needed and have the ability to locate, evaluate and use effectively the needed information …” (ACRL, 1989)

Digital Fluency = information literacy + digital environments

CommunicateCommunicate

CreateCreate

IntegrateIntegrate

ManageManage

EvaluateEvaluate

AccessAccess

DefineDefine

Digital Fluency:

Skillful use of information via

technology

Digital Fluency:

Skillful use of information via

technology

Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are all registered trademarks of Educational Testing Service (ETS).

Featureso Interactive simulations – not multiple choiceo Vendor neutralo 14 real-world, problem-solving scenarios

The ETS iSkills™ Assessment

Copyright © 2011 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are all registered trademarks of Educational Testing Service (ETS).

• Task Length: 4-minute duration

• Task Type Integrate: Interpret and represent information in an ICT context, including synthesizing, summarizing, comparing, and contrasting information from multiple sources.

• Within the task, the test taker is asked to…– Summarize information from three different types of sources– Compare the information to reach a conclusion

Measuring How We Integrate

In this task, examinees summarize information from a variety of sources and then draw conclusions from their summary.

Examinees are presented with an information need and three different types of information sources.

After filling in the table, examinees must interpret the summary to rank the three sources correctly.

58

Institutional Skill Area Report

59Confidential and Proprietary. Copyright ©2006 Educational Testing Service.

Aggregate Task Performance Feedback Report

• Measures what we value? Open question. • Narrows teaching: No• Experience for students: Challenging, but the online dynamic

may be engaging for some. • Detailed data seems very useful for communications and

continuous improvement. • Price moderate.

PISA-Based Testing for Schools

Have you ever wanted to compare your school’s effectiveness with the world’s best national educational systems?

Your school’s results are statistically significantly above

Your school’s average score is not significantly different

Your school’s results are statistically significantly belowCompared to Averages

Country or Economy

Reading literacy

Mathematical literacy

Scientific literacy

Shanghai-China

Korea

Finland

Canada

United States

United Kingdom

Germany

Turkey

OECD Average

Reading: 512 Mathematics: 513 Science: 507

EXAMPLE: International comparisons of your school’s performance: PISA 2009 Countries and Economies

Section II. Your School’s Results in an international context

School ABCD

AdvantagePISA Index of socio-economic background

Disadvantage

School performance and schools’ socio-economic background

Student performance and students’ socio-economic background within schools

Private school Public school in rural area Public school in urban area

700

EXAMPLE: Your school’s performance in the context of PISA results for Canada (with socio-economic background)

Section II. Your school’s results in an international context

School ABCDReading: 512

Index calculated

from student questionnaires

-2 -1 0 1 2200

493

Score

Stu

den

t p

erf

orm

an

ce

493

Your School

300

BLOOD DONATION NOTICE

The instruments for taking the blood are sterile and single-use (syringe, tubes, bags).

There is no risk in giving your blood.

Blood donation is essential.

There is no product that can fully substitute for human blood. Blood donation is thus irreplaceable and essential to save lives.

In France, each year, 500,000 patients benefit from a blood transfusion.

Blood donation:

It is the best-known kind of donation, and takes from 45 minutes to 1 hour.

A 450-ml bag is taken as well as some small samples on which tests and checks will be done.

- A man can give his blood five times a year, a woman three times. - Donors can be from 18 to 65 years old.

An 8-week interval is compulsory between each donation.

Question 1An eighteen-year-old woman who has given her blood twice in the last twelve months wants to give blood again. According to “Blood Donation Notice”, on what condition will she be allowed to give blood again?

.................................................................................................................................

.................................................................................................................................

SCORING

Full Credit

Identifies that enough time must have elapsed since her last donation. Depends whether it has been 8 weeks since her last donation or not.

She can if it has been long enough, otherwise she can’t.

Level 2 item – 81.2% of students across OECD can perform tasks at least at this level

Will review option for schools to electronically explore their confidential results online

Proposed Structure of School Reports (Preliminary)

I. Understanding your school’s results from the PISA-Based Test for Schools assessment

II. Your school’s results in an international context

III. Additional insights from international PISA results

AnnexesA. Summary of PISA-Based Test for Schools assessment questions (items,

units and response-types)B. Summary Description of PISA Assessment Frameworks

Contents of print-ready reports will include:

PISA-Based Test for Schools Pilot:

is...• An assessment tool to support

international benchmarking efforts of schools and school networks

• Meant as a tool to assess competencies, skills and application of knowledge of 15-year-old students - comparable to international PISA results

• Based on the same assessment frameworks as the main PISA assessment

• To be available by choice and ‘on-demand’

• A tool that links schools and practitioners to comparative and international policy discussions

is not...• A mandated standardised assessment • Meant as a substitute or alternative to

the main PISA assessment• Meant to inform day-to-day instruction

of teachers• Meant to align 100% with country-

specific standards or curriculum (i.e. “Common Core” although considerable alignment is expected

• Meant as a formative evaluation tool

PISA-Based Test for Schools Pilot and Post-Pilot

Pilot Post-Pilot On-Demand

Timeline July 2011 – September 2012 September 2012 - ongoing

Assessment Window May 2012 On-demand – may be taken anytime during academic year

Test administration, scoring, and reporting

services

International service provider based on Call for Tenders process

by OECD

Local service providers accredited by OECD

School recruitment and communication services

Supported by U.S.-based team and local partners in Canada and the

U.K.

Demand-driven from schools and networks

Number of participating schools

Approx. 120 – 150 in three countries (approx. 15 - 25 in

Canada)

Demand-driven – no annual minimum required

Number of participating students

Approx. 10,000 (75-100 per school)

Demand-driven – no annual minimum required

Testing costs Approx. 80 Eur/Student. TBC by local vendor(s)

Find out more about the PISA-Based Test for Schools:pisabasedtestforschools@OECD.org

It is important to distinguish clearly between the PISA-Based Test for Schools pilot and the main PISA 2012 assessment also taking place in the following months. The two assessments are separate and independent exercises with different goals.

The next cycle of the main PISA assessment will occur in 2012. Results for participating countries and economies are expected to be published in December 2013. Schools cannot participate in both the main 2012 PISA assessment and the pilot round of PISA-Based Test for Schools. Participating in the national sample of the main PISA 2012 study should be the priority.

PISA Based Testing for Schools

The good and the bad:

• Measures what we value: application of knowledge.

• Don‘t think it constrains. • Great general data for

communications and accountability, but maybe not enough detailed reporting.

• Student experience: unsure. • May be expensive.

www.21k12blog.net

Twitter: @JonathanEMartin