measuring what we value - lyons and niblock presentation

Measuring what we value: 21st Century Assessments for Independent Schools

Douglas Lyons, Executive Director, CAIS-CTAndrew Niblock, Lower School Head, Hamden Hall (CT)

Jim Collins

• Determine what you value most, then find a way to measure it.

• Success can be a powerful disincentive: it may be hard to become a great school if you are a very good school

Good to Great

Goals of this workshop

1. Identify measures of school quality that have historically been valued by educators and/or the public.

2. Suggest new ways to report achievement in those measures.

Why?• To better tell our story (the Value Proposition)• To make certain that these measures have stature

appropriate to their significance

Goals, cont.

3. Provide a quick preview of new and emerging assessment tools –instruments designed to measure skills that are increasing in demand in the new century

4. Describe the assessment practices in a select group of schools that define themselves – and are recognized by others – as “Schools of the Future”

Criterion 13:

The Standards require a school to provide evidence of a thoughtful process, respectful of its mission, for the collection and use in school decision making of data (both external and internal) about student learning.

“Not everything that can be counted, counts, and not everything that counts can be

counted”

“It would be easier to change the course of history…”

Powerpoint presentation is available atwww.caisct.org

look for ADVIS presentation on home page

Jim Collins – “whenever possible, use the language of metrics to define what you value”

What is the language of metrics?

4 “ways of knowing”

• Data: raw input, no context, facts, figures, symbols

• Information: organized, processed, analyzed data• Knowledge: information with higher context -

accurate, relevant, current• Wisdom: evaluated knowledge; merged with life

experience

• Heads letter in viewbook; wisdom• Description of school history and mission in viewbook:

knowledge• Course catalogue: information• SAT scores, college placement stats: data

What are people most interested in ?

Data!• conveys a lot of information - quickly• Is viewed as objective, “no spin”• Can be benchmarked, used for comparison

Data is the language of metrics

Risks / Misuses of Data

• Garbage in; garbage out• Data is easily manipulated, corrupted:

Harvard Business School caution If you torture data long enough, it will admit to anything

Data Management / Data Creation in the independent school community

The Challenge:• To frame the data that define us, or have defined

us in the past, in ways that do not elevate modestly valuable information

• To gather and/or to present new data that is beneficial to educators in our planning for the future and is data that measures performance in relation to the achievement of our highest goals.

The S.A.T.

• Decreasing in stature, but still powerful• Has poor validity statistics• Does not measure 21st century Skills• Historically, did not provide faculty with

instructionally useful information• Consider giving the School Day SAT with

Enhanced Scoring

Standardized Achievement Tests

These test are increasing in stature

What happened to elevate these tests above all other forms of data, in public education and to a lesser degree, in independent schools?

“Effects” of standardized tests today:from educating the whole child to educating

the whole test-taker

• Hyper-focus on scores, minor fluctuations• Unprecedented “score chasing” • M.D.I “measurement-driven instruction”• Mind numbing drill and practice

Most popular form of data presentation: percentiles

Math concepts

Math applications

Reading comprehension

Reading Skill

1 91 87 83 83

2 88 84 85 83

3 92 90 88 88

4 84 89 80 90

5 84 86 78 82

6 89 80 81 79

7 90 89 90 87

8 87 87 81 83

Second most popular form of data presentation: Grade Equivalence

Math concepts

Math applications

Reading comprehension

Reading skill

1 1.8 2.1 2.3 2.5

2 2.6 2.8 3.3 3.6

3 3.5 4.0 3.9 4.0

4 4.5 4.9 4.8 4.7

5 5.8 5.2 6.3 6.0

6 6.6 7.3 7.0 7.5

7 8.2 9.0 8.8 8.8

8 9.0 9.6 9.9 10.4

A better way to present achievement data

• Determine and defend your norming group• Determine a worthy and realistic goal

within the norming group• Publish data relative to the goal

Goal: to score within the top third of norming group on all subtests

Reading comprehension

Reading skills

Math applications

Math skill

1 √ √ √ √

2 √ √ -4 √

3 √ √ √ √

4 √ √ √ √

5 √ √ √ √

6 √ √ √ √

7 -2 √ √ √

8 √ √ √ √

The International Database• Most schools administer one or more normed tests

that compare American student achievement with American peers

• Is there a way to assess our international competitiveness?

• Would this data be valuable to us?

Benefits of the New York State “Truth in Testing” Law

• Thousands of released items available to educators• Released items available for NAEP tests• Released items available for TIMMS tests• Released items available for PISA tests• Construct your own “replica test” or form a

research partnership to develop replica tests

CAIS score reports for TIMMS replica test

ABC Country Day School

TIMMS “released item” test results

95% of students scored in the top 1/2 of I.A.

92% of students scored in the top 1/3 of I.A.

90% of students scored in the top 10% of I.A.

Course of study guides: how we describe our program

US HISTORY HONORS GREENWICH HIGH SCHOOL

This course addresses the events and experiences that comprise American history from the period of European colonial settlement through the Civil War (1st semester) And from the period of Reconstruction through the advent of the Second World War. The goal of the course is to provide for our students substantial opportunity to develop the ability to make informed and reasoned decisions as citizens concerned with the public good. (More text follows)

US HISTORY HONORSA study of the events and critical changes that took place from the first American settlement to the present day.We will focus on these events in the context of larger themes; including the shift from an agricultural to an industrial society, the recognition and cultural identification of different groups of people, the transition to a stronger national government, immense territorial expansion, technological change and globalization.

History at Lakeland Prep:

In the four year History sequence at Lakeland Prep, all students will complete the following Demonstrations of Learning:

• 24 research based position papers (4 to 7 pages) in which an analysis, synthesis and/ or evaluation of both original and modern sources is offered in answer to a provocative question in history.

• 6 research based position papers (10 to 15 pages) focused on a students original response to one of identified Essential Questions in American History.

• 12 oral presentations• 8 collaborative projects, • 3 projects completed in collaboration with students in other schools

and/ or countries• 4 interviews with elected officials• 6 Letters to the Editor written on a current topic in local and/or state

government

HSSSE - what do you believe your high school

emphasizes most?

• 21% memorizing facts and figures• 32% understanding information and ideas• 22% analyzing ideas in depth

• 68% my school challenges me academically• 35% my school challenges me intellectually

HSSSE: what instructional methods do you find

exciting or engaging?

• 60% discussion and debate• 60% group projects• 44% student presentations• 24% teacher lectures

Using an Engagement Survey

• Use the HSSSE and contribute your school data for national benchmarking;

• Or, create your own survey, then compare your results to the national HSSSE 2009 data, where applicable .

The Emergence of Longitudenal Data

• The National Clearinghouse has 93% of all US colleges collecting and providing longitudenal data

• Two independent school associations require student tracking for accreditation (freshman GPA):

• CAIS Canada ntp@cais.ca • ISASW www.isasw.org

Measuring teacher engagement, professionalism, attachment to school

Can that be quantified?• Longevity statistics• % faculty with advanced

degrees• % faculty participating

in Annual Giving

Occasional Teacher Absenteeism

• Reported in “school district report cards”• Measures only consecutive days absent, less than

5, 7 or 10 days. After the threshold, absence considered “long term illness”, removed from calculation

• O.C.A considered an indicator of faculty commitment, professionalism, attachment

Greenwich CT High School: State Report Card

Measure your O.T.A.

• Compare it to your local or state public school average

• Inform your Board of Trustees• Inform your Parents Association

Measuring teacher effectiveness: in the best public schools, teacher evaluation is an

informed, professional process

How can independent school leaders become more skilled in the clinical observation and evaluation of teaching?

How can we measure what we value in teaching practice?

The TIMSS video study – teaching

practices in 7 select countries

New and emerging assessment instruments – workshop goal 3

• We have just reviewed ideas for improving the use of existing data

• What about new tools and techniques? • Are the assessment instruments and practices of

earlier generations obsolete or incomplete?• If so, why is that true?

Did something occur on January 1, 2001 that changed everything?

First successful HTTP communication (modern WEB)1990

Netscape, easy to use browser 1995

Google, as a research project 1996

LiveJournal, Blogger, hosting sites 1999

Ericsson smartphone 2000

Wikipedia 2001

Facebook 2004

MIT Open Courseware 2004

YouTube 2005

Skype 2005

The World is Flat first edition 2005

Internet/Multimedia Smart phone (iphone) 2007

Global financial crisis 2008

Khan Academy (2600 videos and growing) 2009

Turn of the millenium events

100 most influential people of the 2nd Millenium

• Jonas Salk 97• Steven Spielberg 91• Elvis Presley 57• Gregor Mendel 42• Martin Luther King 33• Henry Ford 29• Michelangelo 19• Galileo 10• Columbus 6• Charles Darwin 4• Who is Number 1?

Johann Gutenberg

The printing press was information technology

What about modern day visionaries

in information technology?

What number is Bill Gates?

Steve Jobs?

What’s Past is Prologue vs The Future is Not What it Used to Be

• “What does an educated person need to know?”

• Education is defined by a remembered body of knowledge, the “canon”

• Critical, sarcastic view of the canon: Education as inoculation:” American history? I had that, Tetanus shot? I had that…)

20th century technology:

• Radio, television and film had great promise, but no demonstrable effect on schooling

What 20th century technology had a revolutionary effect on teaching and learning?

SCANTRON! Bubble answer sheets!

• 1948 – Scantron Corporation revolutionized the speed and efficiency of data collection and advanced the notion that student proficiency and school quality can be determined through mass-produced, multiple choice metrics

• Scantron, to this day, has had a greater impact on k-12 curricular design than any other technology in history.

in post scantron decades: “What gets measured is what gets

taught”.

• Tests “drive” instruction in ways that mimic both the content and format of the test.

• What gets measured is almost exclusively content• In the Information Age, we measured recall of

information

In 2012, in The Conceptual Age

· There are no books, conferences, op-ed pieces on “21st Century Content”.

· The canon has been buried under the information explosionHowever, · There is near universal agreement on a short list of 21st century

skills. There is near universal agreement on the need to employ technology in a thoughtful but robust manner

Nicholas Negroponte on applying technology in a robust

manner:

• “When you drop a penny into a glass of clear water, you get a glass of clear water with a penny in it; the change is additive.”

• “When you place a drop of red dye in a glass of clear water, you get a glass of pink water. The change is ecological.”

• Technology in education needs to be ecological; pink water

The i generation

• Defined mostly by their use of technology• Accustomed to learning things on their own and learning

from peers• Expect technology to be interactive and customizable• Non-linear thinkers; web thinkers, scanners, multi-taskers

Clay Shirke, futurist describing the i generation:

“A father sets up a new television in the living room. His 4 year old daughter is seen rummaging through the box. What is she looking for?”

Passive media experiences

will hold less appeal for this

generation

The 21st century skills movement, the Schools of the Future movement, focus on the

development of these skills:

Communication Collaboration Critical/Analytical Thinking Creativity Problem-solving

Content is still important; but content in these areas will need to be acquired through active exploration as well as through instruction.

American work in the 21st century

Non–Routine Tasks defined in the Journal of Economics, volume 118

• Gathering, synthesizing, and analyzing information.• Working autonomously to a high standard with minimal

supervision.• Leading other autonomous workers through influence.• Being creative and turning that creativity into action.• Thinking critically and asking the right questions.• Striving to understand others’ perspectives and to

understand the entirety of an issue. Communicating effectively, often using technology.

Current assessment tools do not measure these skills.

You cannot have 21st century schools using 20th century assessments.

Ideal Assessment:

Provides accurate demonstration of student proficiency

Yields information for faculty planning· Is valid as a learning experience in and of itself

· An assessment of, for and as learning

What is a performance task? Students assume roles in a scenario that is based in the "real world" and contains the types of problems they might need to solve in the future. The task requires critical thinking, analytical reasoning and problem solving. Communication skills are used in describing the solution.

Ohio Mastery Test, Grade 9

• Ms. Johnson installs new insulation to save money on heating costs, but then learns that her bills have not declined by much from the previous year. Her contractor points out that heating costs have risen and weather has been colder. Ms. Johnson wants to find out how much she has actually saved due to the insulation she installed. On the basis of the situation painted above, details about Ms. Johnson’s heating bills (rates, units of heat used), temperature changes, and some initial information to help them begin to research “heating degree days” on the internet, students are given two tasks:

• (1) Assess the cost-effectiveness of Ms. Johnson’s new insulation and window sealing.

• (2) Create a short pamphlet for gas company customers to guide them in making decisions about increasing the energy efficiency of their homes.

Students get answers to questions THEY ask

• For example – Family history – Is this breast cancer

possibly caused by abnormal oncogene expression? If so, certain types of hormonal therapy or receptor antagonists are more effective treatments.

– What level of stage III cancer, A, B or C?

CBAL

• Extended, constructed-response tasks that are delivered  by computer and automatically scored.

• Pilot testing occurred in 2010 and 2011, spring of 2012. • Tests should be available for use in 2012. • Sample tests available online

Website information is in your folders.

features real-time, scenario-based tasks that measure an individual's ability to navigate, critically evaluate and understand the wealth of information available through digital technology

Ken Robinson

Age and education:

• Increase routines of behavior and habits of thought (left brain logical thinking )

• Decrease divergent thinking (free association of ideas. Right brain, creative thinking)

Creativity Index: the newstate mandate?

• Gov. Deval Patrick has made Massachusetts the first state in the country to call for the formation of a creativity index aimed at rating public schools statewide based on their ability to teach, encourage and foster creativity in students.

• Similar legislation is pending in California and Oklahoma

Torrence Test of Creative Thinking

Verbal Activity 4: Product Improvement Look at the stuffed toy elephant in the

drawing. It is six inches tall and weighs a half pound. In the space provided, list the cleverest, most interesting and unusual ways you can think of for changing this toy so that children will have more fun playing with it. Do not worry about how much the change would cost.Think only about what would make it more fun to play with.

Activity 2 and 3: Guessing Causes and Guessing Consequences measures “idea fluency”

What do you get if you solve this problem and visit the website?

The 4th goal of this workshop: Examples of assessment practices in a select group of

schools that define themselves – and are recognized by others as “Schools of the Future”

Schools that define themselves asSchools of the future

• Who are the pioneers?• What do these schools have in common?

Science Leadership AcademyPhiladelphia, Pennsylvania

All of these schools have 2 common characteristics

• urban public charter schools.

• experimenting with a dramatically different view of teaching and learning; A Collaborative, Conceptual Model

21st Century education: from coverage model to

conceptual model

• Recall of information (content) is still important• Skill in accessing and selecting information (internet

searches now deliver 2000 hits) vitally important• Ability to use or apply information in new and/or novel

settings most important (Its not what you know, but what you can do with what you know)

• Expanded role of the teacher: guide, coach, facilitator

Schools of the Future: characteristics

• Performance tasks• Project-based learning, individually and in groups• Capstone projects, individually and in groups• Independent study• Online learning, online tools (courseware, Skype, You

Tube, Ning, Moodle, Web 2.0 etc…)• Students given choices in assignments and in

demonstrations of mastery

Schools of the future, cont.• Extensive use of essential questions relating to content

area

why, how and what if questions• Computer-adaptive learning (program adjusts to student

skill level)• E-portfolios, published within the community or on the

web – seeking Facebook-type conversations in the academic community, on academic topics

• Flipped classroom strategies – routinely or occasionally• Partnerships, learning experiences beyond the school

campus• RUBRICS used to assess performance

RubricBuilderV2.xhtml

A New Definition of School

“we need to invert the conventional classroom dynamic: instead of teaching information and content first, and then asking students to answer questions about it second, we should put the question/problem first, and then facilitate students with information and guidance as they seek the answer and hold them accountable for the excellence of their solutions and of their presentation of their results”.

-Ted Mccain Teaching for Tomorrow

What about independent schools?

• lead the nation in communication skills; writing, speaking, the performing arts

• Engagement has been supported by very strong student-faculty relationships

• An incremental approach to the challenges of the future; preserving strong, successful, traditional programs while expanding collaborative learning, online learning, project-based assessments, exhibitions of learning and use of digital portfolios

• Growing interest in “Essential Questions” theory of learning

Independent schools

http://stolaf.edu/depts/cis/wp/ebouvier/index.html

Lessons from our research:

Schools in the 21st century will define success in much broader terms

Great Schools in the 21st century will include some that have far fewer resources than independent schools. What they have is the freedom to take big risks in designing innovative cultures

“Measuring What We Value” Sites Referenced in Presentation

Hechinger Article containing multiple links of sample questions on new 2012 assessmentshttp://hechingered.org/content/are-new-online-standardized-tests-revolutionary-decide-for-yourself_5655/

Information on Torrance Testhttp://www.ststesting.com/ High School Survey of Student Engagementwww.indiana.edu/~ceep/hssse/ College Student Experiences Questionnairehttp://cseq.iub.edu/cseq_generalinfo.cfm National Student Clearinghousewww.studentclearinghouse.org/ The Self-Regulation Questionnairewww.psych.rochester.edu/SDT/measures/SRQ_text.php MHISC: Mental Health in Independent Schoolshttp://www.harthosp.org/InstituteOfLiving/OtherServices/MHISC/default.aspx 

C-bal Cognitively Based Assessment for Learninghttp://www.ets.org/research/topics/cbal/initiative/ The CWRA: College to Work Readiness Assessmentwww.cae.org/cwra/ Science Leadership Academywww.scienceleadership.org/ High Tech Highhttp://www.hightechhigh.org/ New Tech Highhttp://newtechhigh.org/ Big Picture Learninghttp://www.bigpicture.org/ NYC i schoolhttp://www.nycischool.org/ Microsoft School of the Futurehttp://www.microsoft.com/education/en-us/leadership/partners_in_learning/Pages/School-of-the-Future.aspx

Performance Assessment Group of NYC Schools (check out the rubics!)http://performanceassessment.org/ Rubics – Association of American Colleges and Universities (rubics on critical thinking, creative thinking, problem-solving and others!)www.aacu.org/value/rubics Avenueshttp://www.avenues.org/ Haverford School (Decision Education)http://www.haverford.org/ Decision Education (critical thinking/character education program)http://www.decisioneducation.org/ Independent Curriculum Grouphttp://www.independentcurriculum.org/ Greens Farms Academyhttp://www.gfacademy.org/RelId/607374/ISvars/default/Capstone.htm

Hotchkiss/Loomis Collaborative Learning Projecthttp://tinyurl.com/3kq8v8f

Project-based Learning (450 sample projects – all subjects and grade levels – templates for organizing projects)http://pbl-online.org/ Siemens Challenge (sample award-winning projects)http://www.wecanchange.com/ American Society for Engineering Education (ASEE)http://teachers.egfi-k12.org/ Exploravision (sample award-winning student projects)http://www.exploravision.org/ Toyota Tapestry Grants for Science Teachers (sample grant-winning ($10,000) projects)http://www.nsta.org/pd/tapestry/

aniblock@hamdenhall.org

lyons@caisct.org