How do we know what they know?

FASS MeetingOrlando, FLApril 5, 2004

Arthur Eisenkraft (eisenkraft@att.net)

Are we listening?

• The optometrist

• The Duracell competition

• Two objects falling in a vacuum

• The cord of wood

No Child Left Behind

• Enormous concern about NCLB and other high stakes assessments.

• NCLB – a potential nightmare – – AYP - “need of improvement”– public embarrassment– best students leaving and the scores dropping more,

closing of schools. • What can we do?

– This year alone 26,100 of the nation’s 91,400 have been labeled “schools that need improvement.” (Sam Dillon, 1 in 4 Schools Fall Short Under Bush Law, N.Y. Times, January 27, 2004 at A21)

NCLB - Advice• Assessment in Support of

Instruction and Learning: Bridging the Gap Between Large-Scale and Classroom Assessment - Workshop Report (2003)Board on Testing and Assessment, Mathematical Sciences Education Board, Center for Education

• www.nap.edu

The Deborah Meier Amendment

• A basic test should first be taken by the folks we honor by electing to office.

• The people who legislate or mandate a test should be required first to take it themselves to ensure that it's measuring what they think it is. It's a form of validity checking.

• They might even have their scores posted!

• Seeking Alternatives to Standardized Testing • By Jay Mathews, Washington Post Staff Writer

Tuesday, February 17, 2004; 10:46 AM

Today’s Discussion

• Formative classroom assessment can positively impact instruction and therefore is our best approach to students performing better on all tests.

• What can teachers do?

• What can we do to support teachers?

Brief History of Assessment

• When did it all begin?

– No Child Left Behind– FCAS– New York State Regents

How do we assess?

• Please discuss– Observation– Portfolio– Projects– Questioning– Paper and pencil– Interview– Presentation– Checklist– Skills– Self assessment– Quizzes– conferences

Classroom Assessment The Grade Book

TestsQuizzesHomeworkClass participation (?)Lab reportsAttendance (X)Projects

The Final ExamLocalState – High stakes

These are often treated as summative though they do inform as formative.

Other formative assessments include:–Questions in class–Practice tests

Get tests back immediately

• They can then be used for formative assessment.

• How can anyone continue instruction when you have a tool that informs you of student understanding?

• Easily measured by supervisors and students alike

Formative Assessment

• The value of formative assessment (Paul Black): students often have limited opportunities to understand or make sense of topics because many curricula have emphasized memory rather than understanding. Textbooks are filled with facts that students are expected to memorize, and most tests assess students’ abilities to remember the facts.

What Goes Wrong?

Tests that do not correlate with understanding•Force Concept Inventory (FCI)•Regents exam question on moving galaxies•Private Universe videotapes

What Goes Wrong?

Tests that do not correlate with understanding•Force Concept Inventory (FCI)•Regents exam question on moving galaxies•Private Universe videotapes

We’re not testing what we teach•Harris cartoon of mouse and maze

What Goes Wrong?

Tests that do not correlate with understanding•Force Concept Inventory (FCI)•Regents exam question on moving galaxies

We’re not testing what we teach•Harris cartoon of mouse and maze

We’re not teaching what we test•“Waldo” phenomenon

Improvements

Rubrics•Clearly defined grading schema – matrix•B++++ and A-----

•Have students help create rubric•Ownership•Motivation

•Have students self – evaluate with rubric

The grading rubric +/-

Student Grade Teacher Grade

A A

A C

C C

Improvements

Rubrics•Clearly defined grading schema – matrix•B++++ and A-----•student and teacher comparisons: A,A or C,C or A,C or C,A

•all require very different discussions

Saphier effective instruction•testing for understanding

•how do you know what the students know?

Cognitive Empathy

With references from

The Skillful Teacher

-Jon Saphier

Checking for Understanding

Knowing when students don’t understand suggests that teachers have means for checking for understanding.

What means do we have for checking for understanding?

Checking for understanding

• Presses on– No test return for 3 weeks– After math lesson, here’s your 25 problems– Take a clean sheet, we’re going on– No clue there are kids in the room– Never asks students to explain– Incorrect response, - can anyone else answer

Checking for understanding

• Presses on

• Reads cues– Their looks– Eye contact– Nodding heads– Asleep or awake– Misbehavior– I can see it in their eyes

Checking for understanding

• Presses on• Reads cues• “dipsticks”

– White-boarding– Short quiz– Raise the hand– Choral answers– Cards with A or B– Raise fingers with 1(index) or 2– List answers on board – which answer is the best

Checking for understanding

• Presses on• Reads cues• “dipsticks”• Uses recall questions

– What do we already know– List examples– Who invented– Where’s waldo– Definitions– Name the parts of the microscope– The scientific method– Restating what is already known

Checking for understanding

• Presses on• Reads cues• “dipsticks”• Uses recall questions• Uses comprehension questions

– Explain– Justify– Compare– Apply– Calculate– Why– Summarize

Checking for understanding

• Presses on• Reads cues• “dipsticks”• Uses recall questions• Uses comprehension questions• Anticipates confusion

– Photosynthesis, Kreb’s cycle– Understanding that some kids are literal

• Underground railroad

– Misconceptions research– Look at prior knowledge– Teacher examining their own assumptions

A TEST for Checking for Understanding

How do you know that a student understands?

What evidence do you have?

How often should you be able to answer this question?

The National Science Education Standards (NSES)

Less Emphasis On: Assessing what is easily measured

More Emphasis On:Assessing what is most highly valued

Less Emphasis On: Assessing to learn what students do not know

More Emphasis On:Assessing to learn what students understand

Instructional Models

• Karplus– three-phrase learning cycle

• exploration, invention and discovery

• Lawson– exploration, term introduction, and concept

application

• Bybee 5E– Engage, explore, explain, elaborate, evaluate

• 7E clarification of 5E

5 E

Engage

Explore

Explain

Elaborate

Evaluate

5 E 7 E

--------------------------- Elicit | |

Engage ----------------- | |

---------------------------- Engage

5 E 7 E

--------------------------- Elicit | |

Engage ----------------- | |

---------------------------- Engage

Explore ------------------------------------------- Explore

5 E 7 E

--------------------------- Elicit | |

Engage ----------------- | |

---------------------------- Engage

Explore ------------------------------------------- Explore

Explain ---------------------------------------------- Explain

5 E 7 E

--------------------------- Elicit | |

Engage ----------------- | |

---------------------------- Engage

Explore ------------------------------------------- Explore

Explain ---------------------------------------------- Explain

-------------------------- Elaborate |

Elaborate --------------- |

-------------------------- Evaluate |

Evaluate ----------------- |

--------------------------- Extend

4 Q Assessment Model

• What does it mean?

4 Q Assessment Model

• What does it mean?

• How do we know?

4 Q Assessment Model

• What does it mean?

• How do we know?

• Why should I believe?

4 Q Assessment Model

• What does it mean?

• How do we know?

• Why should I believe?

• Why should I care?

The other four questions

• What did you say?

• Should we take notes?

• When is class over?

• Will this be on the test?

Challenges

Identify who are we testingoStudentsoTeachersoSchools and districts

Identify for what purpose (from Classroom Assessment and the NSES, NRC)

oHelp students learn oTo illustrate and articulate the standards for quality workoTo inform teaching oTo guide curriculum selectionoTo monitor programsoTo provide a basis for reporting concrete accomplishments to interested partiesoFor accountabilityoCertification

Reporting individual achievementGradingPlacementPromotionAccountability

oParents to taxpayers(from High Stakes Assessments, NRC)

Challenges

Are we trying to use ONE instrument for all (students, teachers, schools)? for all purposes?

Understanding vs. beliefoMazur student taking FCI

Challenges

Classroom Assessment and the National Science Education

Standards (2001)Center for Education

Knowing What Students Know: The Science and Design of Educational Assessment (2001)Center for Education

www.nap.edu

The Assessment Triangle

cognition, observation, and interpretation—must be explicitly connected and designed as a coordinated whole. If not, the meaningfulness of inferences drawn from the assessment will be compromised.

C O

I

Assessment #1

• Question: What was the date of the battle of the Spanish Armada?– Answer: 1588 [correct].

• Question: What can you tell me about what this meant?– Answer: Not much. It was one of the dates I

memorized for the exam. Want to hear the others?

Assessment #2

• Question: What was the date of the battle of the Spanish Armada?– Answer: It must have been around 1590.

• Question: Why do you say that?– Answer: I know the English began to settle in Virginia

just after 1600, not sure of the exact date. They wouldn’t have dared start overseas explorations if Spain still had control of the seas. It would take a little while to get expeditions organized, so England must have gained naval supremacy somewhere in the late 1500s.

Comparison

• Most people would agree that the second student showed a better understanding of the Age of Colonization than the first, but too many examinations would assign the first student a better score.

• When assessing knowledge, one needs to understand how the student connects pieces of knowledge to one another. Once this is known, the teacher may want to improve the connections, showing the student how to expand his or her knowledge.

Formative Assessment Research

• Black and Wiliam (1998) provide an extensive review of more than 250 books and articles presenting research evidence on the effects of classroom assessment.

• They conclude that ongoing assessment by teachers, combined with appropriate feedback to students, can have powerful and positive effects on achievement.

• They also report, however, that the characteristics of high-quality formative assessment are not well understood by teachers and that formative assessment is weak in practice. High-quality classroom assessment is a complex process

Return to bad practice

• We revert back to old, failed strategies when under pressure

• Comfort food

• In dealing with our children• You’ll poke your eye out• What if everybody jumped off the Empire State

Building?

• Prison, drug addiction

Responding to the pressure (NCLB, FCAS)

• If you go back to the worksheets and it doesn’t work, you won’t be blamed in the same way because this is what everybody does.

• If you continue with something that is closer to the edge, you are more open to criticism

• This could be the reason for the blame• Fail forward as Thomas Edison did

Support our teachers

• Help them to not succumb to the pressures.

• Encourage them to improve their instruction, not to revert to poor pedagogy.

• Give them permission to be better teachers.

Questions that foster deep understanding rather than questions that ask for repetition of memorized information

and conclusions. (Jim Minstrell and Emily van Zee)

• Ask or promote questions to open an inquiry and elicit students’ initial understanding and reasoning.

• Ask or promote questions to interpret and make sense of data in order to generate new knowledge and understanding.

• Ask or promote questions to clarify or elaborate on observations and inferences.

• Ask or promote questions to encourage learners to justify their answers and conclusions or to explain their reasoning to go beyond a mere stating of an answer.

• Ask or promote questions to extend or apply learned ides.• Ask or promote questions that help learners monitor their own

learning.

Understanding by Design

• When do we generate test questions

• UbD (Grant Wiggins, Jay McTighe)– Enduring understandings– Evidence– Instruction

Not in vain

• When studying about veins and arteries, for example, students may be expected to remember that – arteries are

• thicker than veins,• more elastic, and • carry blood from the heart;

– veins carry blood back to the heart.

Sample test item

• Arteries– a. Are more elastic than veins– b. Carry blood that is pumped from the heart– c. Are less elastic than veins– d. Both a and b– e. Both b and c

The new science of learning

• does not deny that facts are important for thinking and problem solving.

• Research on expertise in areas such as chess, history, science,and mathematics demonstrate that experts ’ abilities to think and solve problems depend strongly on a rich body of knowledge about subject matter (e.g.,Chase and Simon,1973;Chi et al.,1981;deGroot,1965).

Facts are not enough

• However,the research also shows clearly that “usable knowledge ” is not the same as a mere list of disconnected facts.

• Experts’ knowledge is – connected and organized around important concepts

(e.g., Newton ’s second law of motion); – “conditionalized” to specify the contexts in which it is

applicable;– supports understanding and transfer (to other

contexts) rather than only the ability to remember.

Vein and artery experts

• Know the facts in the mc question• also understand why veins and arteries have particular

properties.– They know that blood pumped from the heart exits in spurts – That the elasticity of the arteries helps accommodate pressure

changes.– They know that blood from the heart needs to move upward (to

the brain) as well as downward and that the elasticity of an artery permits it to function as a one-way valve that closes at the end of each spurt and prevents the blood from flowing backward.

• They are better able to transfer– Design an artificial artery strong enough to handle pressure with

or without elasticity (Bransford and Stein,1993).

Classroom environmentsFormative assessments—ongoing assessments designed to make

students’ thinking visible to both teachers and students are essential.

They permit the teacher to • grasp the students’ preconceptions,• understand where the students are in the “developmental cor-

ridor” from informal to formal thinking, and • design instruction accordingly.

In the assessment-centered classroom environment, formative assessments help both teachers and students monitor progress.

Learner friendly assessments

• Provide students with an opportunity to revise and improve their thinking (Vye et al.,1998b),

• help students see their own progress over the course of weeks or months

• help teachers identify problems that need to be remedied (problems that may not be visible without the assessments).

• Problem based learning model– Sport for the moon

Defining Science Content

• Facts, process, knowledge– Order of the planets

• I went to a two day history seminar and there was no content – I didn’t learn any new dates.

• Where is the knowledge we have lost in information. (T.S. Eliot)

Wait time study at King’s College (Black & Wiliam, 2000)

• Questions opened up discussion that helped expose and explore students’ assumptions and reasoning.

• At the same time, wrong answers became useful input, and the students realized that the teacher was interested in knowing what they thought, not in evaluating whether they were right or wrong.

• As a consequence, teachers asked fewer questions, spending more time on each.

Wait time study at King’s College (Black & Wiliam, 2000)

• In addition, teachers realized that their lesson planning had to include careful thought about the selection of informative questions.

• They discovered that they had to consider very carefully the aspects of student thinking that any given question might serve to explore.

• This discovery led them to work further on developing criteria for the quality of their questions.

Grades vs. comments

• Simply giving grades on written work can be counterproductive for learning (Butler, 1988)

• In response, teachers began instead to concentrate on providing comments without grades—feedback designed to guide students’ further learning.

• Students also took part in self-assessment and peer-assessment activities, which required that they understand the goals for learning and the criteria for quality that applied to their work.

• In these ways, assessment situations became opportunities for learning, rather than activities divorced from learning.

Sliding toward Socrates

• Follow-up research– Grades vs. comments vs. grades & comments

• No distinction between instruction and assessment

Beliefs and Practice

1 2 3 4 What you value

(e.g. inquiry, content)

Yo

ur

text

an

d p

rog

ram

1

2

3

4

+

+ +

Beliefs and Practice

1 2 3 4 What you value

(e.g. “telling and memorizing”)

You

r te

xt a

nd p

rogr

am

1

2

3

4

+

Beliefs and Practice

1 2 3 4 What you value

You

r te

xt a

nd p

rogr

am

1

2

3

4 +

+

+

+

Beliefs and Practice

1 2 3 4 What you value(e.g. inquiry, content)

You

r te

xt a

nd p

rogr

am

1

2

3

4

o o

o o

Beliefs and Practice

1 2 3 4 What we value

(e.g. telling and memorizing)

You

r te

xt a

nd p

rogr

am

1

2

3

4 o o

o

Beliefs and Practice

1 2 3 4 Content, pedagogy, assessment factors

Our

tex

t an

d pr

ogra

m

1

2

3

4 +

+

+

+

Summary

• Return tests promptly• Rubrics• Understanding by Design

– Enduring undestandings, evidence, instruction• Testing for undestanding

– Know what each student knows at the end of every class• 7E learning model• Assessment triangle: Cognition, Observation,

Interpretation• Don’t revert to non-productive behavior• Define science content to include inquiry and process

– Facts in meaningful context

Assessment in the movies

• Do we recognize this quality of assessment?

In Conclusion

• The best way in which to deal with NCLB is to improve instruction in the classroom.

• This can be done with improvements in formative assessment– Formative assessment has been shown to be effective at

improving student achievement.– Help teachers learn “Testing for understanding” techniques.

• Provide teachers with the confidence and support to continue to improve their teaching and resist the temptation to revert to poor pedagogy and worksheets in attempts to deal with NCLB.

• Give them permission to be good teachers.

Common Advice

• ….the wisdom to know what I can change

• My husband handles the important issues

“We must be the change we want to see in the world.”

- Gandhi

Get inside a student’s head.

Cognitive empathy for the workings of the learners’ minds -- an ability to put themselves in the learners’ shoes

Unscrambling Confusion

Determining what students don’t understand implies that teachers have ways of unscrambling confusions that identify the specific point(s) of misunderstanding and deal with them.

What means do we have for unscrambling confusions?

Unscrambling Confusions

• None

Unscrambling Confusions

• None

• Re-explains

Unscrambling Confusions

• None

• Re-explains

• Isolates point of confusion with pinpoint questions

Unscrambling Confusions

• None

• Re-explains

• Isolates point of confusion with pinpoint questions

• Perseveres and returns

Unscrambling Confusions

• None

• Re-explains

• Isolates point of confusion with pinpoint questions

• Perseveres and returns

• Has student explain own current thinking

Three pillars of assessment

• a model of how students represent knowledge and develop competence in the subject domain

• tasks or situations that allow one to observe students’ performance,

• an interpretation method for drawing inferences from the performance evidence thus obtained.

Improvements

Rubrics•Clearly defined grading schema – matrix•B++++ and A-----•student and teacher comparisons: A,A or C,C or A,C or C,A

•all require very different discussions

Eliciting prior understandings•Fish is fish

Standards State exams (good instruction) (good preparation)