assessment for learning, not just of learning

On Assessment for Learning Fall 2004

© 2004 Grant Wiggins

1

© 2004 Grant Wiggins 11/20041

Grant Wiggins on assessmentFall 2004

Assessment for Learning,not just of learning


Assessment Reform5 key ideas:

Helping everyone understand excellence and improveis the primary goal of assessment. The secondary goal isto formally report out.

There is a moral imperative: the student is entitled totests that teach—greater transparency, authenticity &feedback

Feedback is central to learning—so, assessment (andunit) design must optimize feedback and its use

Known Core Tasks and Rubrics (along with “anchors”)represent the key to an effective and transparentfeedback system, with agreed-upon models and criteria

The system must be credible to key constituencies:teachers, parents, students, receiving institutions


To be transparent it must beopen and demystified:

Transparency means –

Key “tests” known in advance, as in the widerworld

Rubrics for all key standards, known andregularly used in local assessment - includinglongitudinal rubrics for charting progress overtime; and local grades linked to standards

Public and studied samples of Work along twocontinua – both quality of work & sophistication –established system-wide

Students taught the way we train AP or stateportfolio readers



2


To be credible it must be“triangulated”:Credibility through being mindful

of the need for –

Disinterested AssessorsVariety of evidenceReliable/consistent scoringApt tasks, criteria, & standards


Core Premise:The primary aim of assessment

is to improve student performance,not merely audit it via grades on

simplistic tests


Reform mantra

Assess what we value,and value what we assess

Moving beyond -”Test what is merely easy and

uncontroversial to test and grade" "Teach, test, and hope for the best"



3


The mantra of BackwardDesign in UbD:

“Think like an assessor,not a ‘teacher’ or activity designer!!”

The key to better design is deriving lessons fromlearning goals, and feedback against goals in transfertasks; not merely interesting work, focused on content

Ask: What do the desired learnings/abilities imply forthe evidence we need to collect, provide feedbackabout, and ‘teach to’?


1. Identify desired results

2. Determine acceptable evidence

3. Plan learning experiences & instruction

3 Stages of(“Backward”) Design


Identify content

Brainstorm activities

Come up with an assessment andlink it to some general goal

Typical Error in Design

Without checking for alignment



4


Backward Design logic STAGE 1: If the desired results are for learners

to...Understand that...Be able to handle such challenges as...

STAGE 2: then, you need evidence of thestudent’s ability to... [General evidence needs, regardless of task specifics]

STAGE 3: so, the learning activities mustinvolve...


Stage 2 is the essence ofbackward design & alignment“Measure what we value,value what we measure”Derive the required

assessments from thecomplex performancesexplicit or implicit in the‘big ideas’ and ‘coretasks’ at the heart of thediscipline


An implication for the natureof feedback given

We must ensure that students getfeedback against the ultimate tasks of

transfer, not primarily on our testsand quizzes of content

Contrast arts & athletics with mostclassroom feedback: the performance, notthe sum of the exercises



5


to teach for understanding is tocoach for transfer

Transfer requires… tasks with minimal cues and scaffolding, unlike

typical test items: can the learner consider and judgewhich knowledge & skills are required here? Local/classroom assessment is consistently too low-level

and narrow – not focused on transfer, but ‘plugging in’

Learning how to adapt, grapple with new orunfamiliar elements, uses, or obstacles - i. e. teachthem how to learn, transfer – “to know what to do when theydon’t know what to do”

We still fail to honor what Bloom et al say aboutanalysis and synthesis.

There must be local assessment standards and oversight


What assessment for‘understanding’ implies

Bloom and common sense say: Your thinking and support, not just your answer - “show

your work”, the dissertation and its defense The ability to apply what you have learned to a novel

problem or situation - transfer-ability Being able to perform, on your own, with minimal

prompting - to do the subject The ability to adjust, as needed, in situations, against

feedback - as part of the assessment The ability draw inferences, on your own: generalize,

compare & contrast, etc. The ability to argue/critique/evaluate the work of others as

well as one’s own


What Bloom actually said… "Application is different from simple comprehension: the

student is not prompted to give specific knowledge, nor isthe problem old-hat."

"Ideally we are seeking a problem which will test the extentto which the individual has learned to apply an abstraction ina practical way.”

The evaluation of analysis abilities and skills requires thatstudents demonstrate the appropriate behavior in a newproblem or situation. Otherwise they would be doing nomore than revealing their memory or knowledge.

"In 'synthesis'...the performer wishes to achieve a given effectin some audience.... The student should have considerablefreedom...to determine the materials or elements...andfreedom to determine the specifications which the synthesisshould meet."



6


Analysis, in reading… In a real sense, analysis requires the student to

‘see’ the ideas and devices employed in adocument, which can only be inferred from whatthe author has done…

The ability to infer the author’s purpose, or traitsof thought and feeling as exhibited in the work

The ability to recognize tone, mood, attitude of theauthor

The ability to recognize form and pattern inliterary works

Ability to recognize point of view or bias of anauthor


Core tasks are key toprioritized learning for transfer

Defined: “The most important complexperformances, in realistic contexts, in

each field” Ask:

What does it mean to do the subject, to have yourknowledge ‘tested’ in the world?

What are authentic options, constraints, andopportunities available in such work?

What are the key genres of performance in yoursubject(s)? What might be the ‘decathlon’ in yourprogram area, that might anchor the curriculum?


Consider goals related to bigideas in math

Essential Questions: Which measures of central tendency are most

appropriate in this situation? How do you know? What is fair? How can math help us answer the

question (and not help us)?

Transfer Task: Transfer your knowledge to an interesting and

contextualized problem, related to measures ofcentral tendency

Knowledge: Students will know the meaning of mean, median, mode, standard

deviation, etc.



7

Class rank ClassA

ClassB

ClassC

ClassD

1 4 6 1 22 9 7 3 53 11 10 14 84 12 13 18 155 20 16 19 176 21 22 23 317 25 24 28 338 26 27 30 369 29 34 32 3710 35 39 41 3811 43 40 44 4612 45 42 47 5113 49 48 50 5514 54 52 56 2715 61 53 60 5816 65 62 63 5917 69 66 64 6718 70 72 * 6819 71 * * 7320 * * * 74

Intro problem: Four7th-grade classes had arace of all the students.

• Devise as many waysas you can to determinea fair ranking of the 4classes, given theindividual runnerresults in the table.

• Summarize the 2-3 topways you think wouldbe most fair, and beprepared to discuss… Individual ranking of runners in a 2-mile race

of all 7th-grade students in the school

UNIT: What is Fair?


Final task in the unit onmean/median/mode

So, what is a fair grade? Based on our study in this unit of various

measures of central tendency, and the pros andcons of using “averages” (and other suchmeasures) in various situations, propose anddefend a grading system for use in this class. Howshould everyone’s grade in class be calculated tobe most ‘fair’? Why is that system more fair thanthe current system (or: why is the current systemmost fair?) Your grade for the term will becalculated by the system you propose, if you makea good case that it is a fair measure of yourperformance!


Framing a MS course inWorld History - by tasks:

1. The design of a tour of the world’s most holy sites 2. The writing of a Bill of Rights for use in Afghanistan, Iraq, and

other new democracies 3. Maps of population and resources: what’s the relation between

resources, wealth, and health? 4. Collect and analyze media reports from the Internet on other

countries’ views of US policies in the Middle East. Do we understandthe issues? Brief the president…

5. Take part in a model UN on the issue of terrorism: you will be partof a group of 2-3, representing a country, and you will try to pass aSecurity Council resolution on terrorism

6. India and outsourcing: to what extent is the global economy agood thing for America? India? India’s neighbors?



8


Assessment of Understandingvia the 6 facets (UbD)

i.e. You really understand when you can: explain interpret apply & adapt see from different perspectives show empathy reveal self-understanding

155ff –from Understanding by Design, Wiggins & McTighe, ASCD


Assessment of Understandingvia the facets

i.e. You really understand when you can: explain, connect, systematize, predict it show its meaning, importance

apply or adapt it to novel situations see it as one plausible perspective among others, question its

assumptions see it as its author/speaker saw it avoid and point out common misconceptions, biases, or simplistic

views

Thus, our assessments and feedback must refer backto these performance goals

155ff


For Reliability & Fairness:Use a Variety of Assessments

Varied types, over time: Photo Album authentic tasks and projects

academic exam questions, prompts, and problems

quizzes and test items

informal checks forunderstanding

student self-assessments



9


Authenticity Defined“Realistic tasks, contexts, and standards

worth mastering”Ask:

What challenges do expertsand citizens actually face?

What does it mean to do science,history, etc.?

What tasks require core content, usedwisely?


Scenarios for Authentic TasksBuild assessments anchored inauthentic tasks using GRASPS:

What is the Goal in the scenario? What is the Role? Who is the Audience?

What is your Situation (context)?

What is the Performance challenge? By what Standards will work be judged

in the scenario?

SPS

GRA

T


Authenticity As defined by Newmann et al.:

construction of knowledge disciplined inquiry value beyond school

"A Guide to Authentic Instruction and Assessment”

As defined by UbD: realistic task, context, standard requires judgment and adjustment, not plugging in minimal secrecy and arbitrary constraint on resources,

assistance



10


The Importance of context intransfer”These programs are lacking in “realworld” scenarios and result innonthinking performance, where theability of the student to demonstrate amastery of complex problems, goodjudgment, situationalawareness…have all been removed."

-- from a government report


inauthentic vs. authentic(examples)

inauthentic fill in the blank

select an answer from aset of given choices

answer the ?s at end ofchapter

solve pat problems withsimple answers

authentic purposeful writing scientific

investigation issues debate primary research solve “real-world”

problems interpret texts


Authenticity and PurposeAuthenticity requires scoring results,not just good-faith effort: "It's not

done 'til it's done right”

Key Questions: "Did the performer accomplish the purpose?" "Did the performance achieve the desired

impact?" "Was the client or audience satisfied?"



11


Task example - GeographyCivic planning. You are a geographer hired tomake predictions about population trends overthe next 50 years, based on current maps, climateand trade data. Based on your understanding ofthe link between geography, settlement,migration, trade, and transportation, advise theplanning agencies of the government on whatthey can expect.

(Students can be presented with modern maps of counties/states -or maps of colonial times that they have not yet studied)

168ff


An entire AP history courseframed by such tasks:

Your goal is to determine why the urbanriots of the late 60's happened. You areone of many august members of an LBJappointed panel, the Kerner Commission,who must report to the president and thecountry on why the violence happenedand what can be done about it. You willproduce a collective report that must bethoughtful, thorough, and clearlypresented. Your personal contributionwill be judged through journal entries,observations of work and discussion, andsections of writing you produce.


Scenario Examples Happy Cola has hired you as a consultant to advise

them on how they can increase the fizz in their cansof soda pop. A technician, inspired by watchingoxygen in an aquarium, had tried bubbling carbondioxide gas into soda pop. Unfortunately, testsshowed no difference between this sample and theoriginal soda. Explain why this method wasunsuccessful. Then, formulate three differentoptions by which the company can accomplish theirgoal, and evaluate the efficacy and cost-effectiveness of each. Assemble your findings intoa coherent proposal in which you present yourrecommendations to the company.



12


Research - Learning andAssessment Newmann et al. (1996) measured how well 24

restructured schools implemented authentic pedagogyand authentic academic performance approaches inmathematics and social studies.

Students with high levels of authentic pedagogy andperformance were helped substantially whether theywere high- or low-achieving students. Anothersignificant finding was that the inequalities betweenhigh- and low-performing students were greatlydecreased when normally low- performing studentsused authentic pedagogy and performance strategies andassessments.


Chicago research Assignments were rated according to the degree to which

they required “authentic” intellectual work: “Studentswho received assignments requiring morechallenging intellectual work also achievedgreater than average gains on the Iowa Tests ofBasic Skills in reading and mathematics, anddemonstrated higher performance in reading,mathematics, and writing on the Illinois GoalsAssessment Program…”


Chicago, cont. “Contrary to some expectations, we found high-

quality assignments in some very disadvantagedChicago classrooms and [found] that all students inthese classes benefited from exposure to suchinstruction. We conclude, therefore, [that]assignments calling for more authentic intellectualwork actually improve student scores onconventional tests. (p. 29)

The complete research is available online athttp://www.consortiumchicago.org/publications



13


Part 2: EducativeAssessment:

Feedback


feedback and its use is key togreat gains

Black & Wiliam meta-analysis:“There is a body of firm evidence thatformative assessment is essential... We knowof no other way of raising standards for whichsuch a strong prima facie case can be made.”

Black and Wiliam (1998) “Inside the Black Box: RaisingStandards through Classroom Assessment,” Phi Delta Kappan,

volume 80, 2 (October), pp. 139 ff.

Cf. Working Inside the Black Box: Assessment for Learning inthe Classroom, by Paul Black, Christine Harrison, Clare Lee,Bethan Marshall, and Dylan Wiliam Phi Delta Kappan, Volume 86,#1 (September, 2004)


Feedback and Improvement

"Feedback is the realbreakfast of champions”The 60-Minute Manager



14


Feedback, hence self-adjustment,depends upon concrete models

based on: Exemplars/anchors Ideal specifications made practical Performance targets, not

vague verbalized expectations andexhortations

Rubrics that summarize the key traits ofexemplars (based on clear and apt criteria)and the full range of work, to enable self-assessment along a continuum


Feedback: Harvard’s “mosteffective” courses

from Making the Most of College: "The big point—it comes up over and over again as

crucial—is the importance of quick and detailedfeedback. Students overwhelmingly report that thesingle most important ingredient for making acourse effective is getting rapid response onassignments and quizzes.

"Students suggest that it should be possible incertain courses to get immediate feedback. Theysuggest that the professor should hand out anexample of an excellent answer.

- Richard Light


Feedback as key (cont.): "Secondly... an overwhelming majority are

convinced that their best learning takesplace when they have a chance to submitan early version of their work, get detailedfeedback and criticism, and then hand in afinal revised version...

Many students observe that their mostmemorable learning experiences havecome from courses where suchopportunities are routine policy."



15


Feedback is vital to all designwork “Faculty members at Harvard were

asked what single change mostimproved their teaching. Two ideasswamped all others. One is enhancingstudent awareness of the big picture,‘the big point of it all’. The second is theimportance of helpful and regularfeedback from students so a professorcan make midcourse corrections.”

- Harvard Assessment Seminar, 1993


On-going feedback is key toeffective self-adjustment

Learning to comparethe actual against the optimal;

the intent vs. effect Feedback confirms or disconfirms against

a specific standard or goal—no personalvalue judgment is madeFeedback is descriptive, not evaluativeFeedback is not praise or blameFeedback is not guidance/advice


FEEDBACKThink back...

•What was the most effectivefeedback system you have everbeen in as a learner? What madeit so?•Share examples, then generalize:“the best feedback systems…”



16


Zen Mantras... It’s never done right the 1st or 40th time -

you need feedback, not more teaching Practice makes permanent You can only get there by continual

attempts to learn and do - self-adjustment Teaching cannot cause results. Attempts

by the learner to learn and ‘do it right’causes results


Feedback versus Guidance Feedback: what you did or did not do, given a

standard; a helpful description of yourperformance or product e.g. the audience reported that it could not hear you

speak, you saw people straining to hear

Guidance: what you might do to honor thefeedback - good advice e.g. "Speak louder or use a microphone”

We give too much advice and not enoughfeedback!!!


Implied PolicyAs in sports, arts, and other

performance areas, students mustknow how their daily work compares to

exit standards

Give grades against school standards, not justclassroom norms

School-approved tasks, work samples, and scoringguides to be used routinely



17


Formative & Self Assessment

Self-Adjustment is Key!


What is self-monitoring andhow can it be fostered?

The ability to size up strengths and weaknesses,and adjust accordingly

The ability of performers to know with precisionwhat they have done to standard and what theyhave not done to standard in their work based on good models, rubrics, feedback, guidance, and

on-going opportunities to use results This is a skill that must be central to the

curriculum and instruction


Self-assessment: variousmeanings

1. Awareness of our own goals, style, interests, andpreferred approaches (self-profile)

2. Awareness of our strengths and weaknesses inpreparation for achieving goals

3. Ability to dispassionately look at and analyzeour strengths and weaknesses in recent pastperformance – against specific goals, standards,criteria (e.g. reflection on a portfolio)

4. Ability to self-consciously monitor ourperformance as it happens and adjust, as needed -to take in and give oneself feedback and advice,and to act on it (self-assess and self-adjust)



18


13 tips for developing self-monitoring (1)

1. For every rubric or criterion you use, ask students to self-assess their workbefore you or peers provide feedback.

2. Place an emphasis in self-assessment on the result of the work, not just theprocess and content: Was the audience engaged? Did the product work? Wasthe client satisfied? Did the paper achieve its purpose? Was the problemsolved? etc.

3. Make sure that you carefully distinguish effort from achievement, and qualityfrom quantity in working on self-assessment with students. They often conflatethe two in their minds.

4. The anchors/samples/models are more important than a rubric. The rubric is anabstraction drawn from groups of samples. So, don’t hide the samples: givestudents practice in self-assessing their work against models, not against onlyrubrics. (Like AP readers!!)

5. Provide diverse models, so that students don’t merely copy. Ask them togeneralize from the various models. Studying varied models at the differentlevels of quality opens the door to creativity rather than closing it: you makeclear that creative excellence, not imitation, is the goal.


The 13 Tips (2)6. Help the student develop an increasingly accurate profile of their

own strengths and weaknesses, likes and dislikes, talents andinterests. Use self-reporting surveys and reflection promptsthroughout the year.

7. Self-assessment may be more revealing of student understandingthan the performance itself, especially if the students are relativenovices.Reward them for their perception and honesty

8. To avoid simplistic reactions, require every self-assessment torefer to multiple specific criteria provided upfront.

9. Provide all rubrics & models right from the start, and designlessons around their meaning and use. But only highlight asmany criteria/rubrics at a time as can reasonably be addressed.


The 13 Tips (3)10. Focus students on personal improvement over time,

against specific and achievable goals, based ondevelopmental rubrics.

11. Make clear that being extremely self-critical or self-praising have little to do with accurate self-assessment.The former is parent-fed guilt/shame, and the latter iswishful thinking. The goal is neutral analysis againstgoals.

12. Make the thoughtfulness and accuracy of student self-assessment (prior to your feedback) part of the grade.

13. Make the quality of their self-adjustment, based on self-assessment (and feedback) part of the grade.



19


The research jibes with thiscommon sense:To achieve any goal you must learn how to

self-assess your learning and yourperformance

One of 3 chief findings in How People Learn Even the youngest learners can (and must) learn to

self-monitor & self-adjust Success means you can self-assess on your own,

and self-adjust, with minimal prompting fromothers


From the research: one of 3chief findings -

“The teaching of metacognitive skills should beintegrated into the curriculum. Becausemetacognition often takes the form of an internaldialogue, many students may be unaware of itsimportance unless the processes are explicitlyemphasized by teachers.

“Research has demonstrated that children can betaught these strategies, including the ability topredict outcomes, explain to oneself, note failuresto understand, activate background knowledge,plan ahead, and apportion time and memory…

How People Learn, p. 14, 21


Research on metacognition,part 2

The model for using the meta-cognitivestrategies is provided initially by theteacher, and student practice and discussthe strategies as they learn to use them.

Ultimately, students are able to promptthemselves and monitor their owncomprehension without teacher support.[transfer]

How People Learn, pp. 18-19



20


Beware making student think thatobeying rules is all that mattersSelf-assessment can never be effective

if the student does not focus on thepurpose of the performance

Did the piece work? Was my speech convincing? Was my proof valid? Was my story intriguing? Was my question answered?


Feedback/Self-assessmentSystem - Healthy

Performers seek feedback on their own and knowthat it is in their interest - even if the news is bad

Performance improves at all levels; there isobvious “value added”

Improved performance occurs more rapidly thanis typical or expected

Few quarrels about the results Feedback use opportunities are central to the

curriculum and assessment Norms and standards rise over time: what was

once considered extraordinary performancebecomes more common


Feedback/Self-assessmentSystem - Unhealthy Learners fear, resist, do not seek, or ignore feedback Learner performance rarely improves much beyond what is

typical Novices struggle to improve; they do not know “what you want.”

Their self-assessment is very inaccurate Many quarrels about the credibility and meaning of the results;

anecdotes and effort trusted more than data Teaching is too linear and prone to coverage of content and

activities, with little opportunity to get feedback and use itrepeatedly for core tasks and big ideas

Norms stay the same, standards rise - and expectations are thuslowered



21


No gains = poor feedbacksystem

Flat scores over time show the problem isnot teachers or students but assessment

The state primarily audits; schools anddistricts must, therefore, require localassessment to be more balanced andeffective:better incentives and requirements in support of

higher quality teacher assessment and use ofresults


Excellent feedbackSome criteria:

Timely user-friendly - in approach and amount Descriptive & specific re: performance Consistent Expert Accurate Honest, yet constructive Derived from concrete standards On-going


Needed: High-Quality LocalAssessmentLocal control with quality control: A balance between:

Teacher controlLocal and external audits as well as

feedback from key constituencies

Assessment design & use standardsneeded

Peer review of both assessment designand scoring/grading of work



22


ResourcesEducative Assessment, Jossey Bass.

(1998)Understanding by Design, ASCD,

(1998)The Understanding by Design

Workbook (2004)


for further information...Contact us:

[email protected]

Grant WigginsAuthentic Education4095 Route 1Monmouth Junction, NJ 08852732.329.0641

assessment for learning, not just of learning

Documents