Performance Assessment

Sue Brookhart

November 5, 2014

Introductions

• Sue Brookhart, Ph.D.

• Juliette Lyons-Thomas, Ph.D. (Fellow, Regents Research Fund)

2

3

Webinar Norms

• All phones will be placed on mute

• If you have a question, you can type into the chat box, and your question will be addressed during a break

The chat box icon is located at the top right hand corner of your screen (remember to direct your chat to “Everyone”)

• At the end of the webinar, you will be asked to fill out a survey based on your experience today

Reporting expectations

• The deadline to post the announcement of the award and the proposed plan for the use of the funds on the district website is December 1, 2014 (new date)

• Each participating district must post the results of their review on their website during the grant period, on or before April 1, 2015  (page 11 and page 36 of the RFP) 

• Each participating district must post their action plan on their website during the grant period, on or before June 1, 2015  (page 11 and page 36 of the RFP) 

4

Learning outcomes of today’s webinar

• The TITC grant has made numerous references to performance assessment as a strategy for assessing students, in lieu of other approaches that may not contribute effectively to teaching and learning

• The purpose of this webinar is to help attendees better understand how to use performance assessment, including tasks and rubrics, and will utilize examples for demonstrative purposes

5

6

What is performance assessment?

Performance assessment requires a student to perform a process or create a product. The process or product is scored by observation and judgment.

PROCESS EXAMPLES PRODUCT EXAMPLES Play a C major scale Diorama depictingCount by 2s to 100 everyday colonial life

Report on frogs

Example

Writing to a prompt is a type of performance assessment. So are reports, projects, and oral presentations.

7

Performance Assessment

• Task(s) or assignment must tap the student learning outcome

Knowledge Skills Level of Thinking

• Rubrics or other scoring scheme must accurately describe student performance along a continuum of quality

8

Sampling Aspects of Tasks

• Wide enough range of performance in the domain to represent the entire learning outcome? (goes to validity)

• Enough student performances to be confident in scoring? (goes to reliability)

9

NYSED K-2 Assessment Guidance

• Annual Professional Performance Review (APPR) Assessment Guidance

• Or visit https://www.engageny.org/resource/guidance-on-the-approved-regulatory-amendments-to-appr-to-help-reduce-local-testing - See Item #6

• Similar guidance is appropriate for other grades, as well

10

11

Example:KindergartenMathematics

Task features that can vary

• Time• Group/individual• Product or process• Cognitive level• Task structure• Difficulty

12

Aspects of Tasks that Interfere with Measurement

• Reading

• Writing

• Artistic ability

• Stage presence or “ham”

• Personality

(If these are not the things being measured)

13

Control the Cognitive Level

14

Bloom’s Taxonomy Description

Analyze • Break a text, problem, or task into PARTS• Use the PARTS to draw a conclusion, develop a

thesis, or solve a problem

Evaluate • Identify relevant elements of a situation for a particular purpose

• Specify values on which to base judgments• Decide on the worth of the elements for the

purpose

Create • Bring together two or more unlike things to make something new

• Envision quality criteria (is the “something new” worthwhile?)

Example

15

Grade 5 Reading

Example

16

Grade 3 Title I Reading

Example

17

Example

18

7th grade Learning Support mathematics

Cognitive Level

• Solving a routine problem is not considered higher order thinking. It is also an example of a very structured task.

• Solving a routine problem and explaining one’s reasoning in words is usually considered analysis because you have to analyze your own solution.

• Inventing your own problem is usually considered synthesis or creation. This is a less structured task than the others.

19

Critique this

Wanted: Explorer

We have been studying explorers. Now it is your turn to be an explorer. Design and draw your sailing vessel. Chart your voyage on the map. Include a compass rose, a map key, and a map scale. Complete a trip log (journal) to record the events of your voyage. Write a report about your adventures. Tell why you decided to go on your voyage, what you took with you, what you expected to find, and what you actually found. Be descriptive, creative, and imaginative. The final report can be typed, printed, or in cursive (use ink). Your report should include the following:

•Cover Page

•Title Page (title, date, name)

•Written report (four paragraphs minimum)

•Drawing of your vessel

•Map of your voyage

•Trip Log

20

A question for you

Which of the following knowledge and/or skills is this performance assessment (as written) MOST LIKELY to assess?

A.Map construction and map reading knowledge and skills

B.Understanding of the purposes and motives of explorers

C.Understanding of the contribution and legacy of explorers

D.Persuasive writing knowledge and skills

21

A question for you

What is the MOST IMPORTANT CRITICISM you would note about the criteria for students?

A.The parts of the report are clearly spelled out for students, and they could use these elements for self-assessment

B.The criteria seem to be what is observable (whether product has required elements) and not what indicates competent use of knowledge and reasoning.

C.The quality levels are not specified with descriptions of performance at each level.

22

Control the Level of Task Structure

23

More Structured Less Structured

The task clearly defines the problem.

Student is free to select and/or define the problem.

The task gives a lot of detailed directions.

Task directions give little guidance; student must figure out how to proceed.

There are very few correct or appropriate approaches to successful task completion.

There are many correct or appropriate approaches to successful task completion.

There is one “right” final performance.

There is no one “right” final performance.

A Framework for Varying Task Structure

24

Task Feature Open <- - -> Closed

Problem,Question, or

Task

NotProvided Guided Provided

Strategies and Materials

NotProvided Guided Provided

Solution, Answer,

Final Product or Performance

NotProvided Guided Provided

Control the Level of Difficulty

• What makes something difficult?

• Difficulty depends on both the task and the student.

25

Difficulty – Student features

• To adjust difficulty, create questions or tasks about the concept that differ in their requirements for

Prior knowledge Prior experiences and relationships Interests and expectancies

26

Difficulty – Task features

• To adjust difficulty, vary elements needed to accomplish the task.

Materials used in doing the task Strategies for doing the task The problem or question

• Note the order!

27

Language Arts Example

• Write a short poem about night. Use at least two forms of imagery. Then, explain in an essay how you used the images in your poem to convey a message or feeling about night.

28

Mathematics Example

Mei baked 12 cookies. She will give each of her friends 2 cookies. How many friends will eat Mei’s cookies?

Show your work in two of these ways:

Numbers Pictures Words

29

Science Example

• Design and conduct an investigation to determine which of three brands of microwave popcorn leaves the fewest un-popped kernels. Be sure to state your research question, the method and materials you will use, the data you will collect, and how you will analyze it and draw your conclusions. Write up your results in a lab report [using a conventional format students have been taught].

30

Social Studies Example

• Compare this 1910 one-room school with your classroom. Find at least five things that are different today. Choose one of them, and write an essay about how the aspect you chose affects your own learning.

31

32

Both the performance and how you observe or score it defines what kind of evidence you get.

Some people call any scoring scheme a “rubric.”

• Checklists

• Rating scales

• Point schemes

• True rubrics

33

34

My sentence has__ a capital letter.__ a period.__ a complete thought.

Math problem solving

I described what the problem was asking. EXCELLENT GOOD FAIR POORI wrote an equation, or used a diagram or model, to solve the problem. EXCELLENT GOOD FAIR POORI calculated the correct answer. EXCELLENT GOOD FAIR POORI labeled my answer. EXCELLENT GOOD FAIR POOR

Did I reason from examples to arrive at a clear, accurate description of physical and chemical changes?

2 = Completely and clearly – response gives clear evidence of reasoning from the examples1 = Partially – response is accurate, but reasoning from examples isn’t clear or is only partial0 = No – response does not demonstrate reasonable conclusions from the examples

True rubrics have 2 elements

• A coherent set of criteria• Descriptions of levels

of performance quality

35

Why is this important?

Rating scale and rubric examples follow, for mathematics problem solving.

•Which provides more useful information to the student as he or she is doing the work?

•What type of judgments do you make to score the same work each way?

•What type of feedback does the student get with the rating scale? With the rubric?

36

37

Parker & Breyfogle, 2011

38

Math Problem Solving

I can do the problem correctly and get the correct answer.

I can figure out a strategy to use to solve the problem.

I can explain my work – tell what I did and why I did it.

Criteria

• Should be about the learning to be demonstrated, not about the requirements for the assignment

• Matched to the learning goal or achievement standard you want to measure and report on

39

Effective Criteria

• Appropriate

• Definable

• Observable

• Distinct from one another

• Complete [for your purposes]

• Able to be described along a continuum of quality

40

Which is the better set of criteria?

41

Learning goal: Students will analyze how particular lines of dialogue or incidents in a story propel the action, reveal aspects of a character, or

provoke a decision

• Content • Mechanics

• Thesis (main point) about the action or story

• Use of dialogue or story elements to support the thesis

• Clarity of expression

Effective Performance Descriptions

• Descriptive

• Clear

• Cover the whole range of performance

• Distinguish among levels

• Center the target performance (mastery, passing) at appropriate levels

• Feature parallel descriptions from level to level

42

• How should you choose the language you use for performance level descriptions in rubrics? [Use examples on the next slides, and explain your reasoning.]

43

44

Example #1 Poor example [one criterion for a primary story writing rubric]

Good example

4 3 2 1

Gram-mar & Usage

Grammar, punctuation, spelling and capitalization are correct or nearly so, and minor errors do not detract from reading the story.

Grammar, punctuation, spelling and capitalization are mostly correct. Readers can still figure out what is meant in the story.

Many errors in grammar, punctuation, spelling and capitalization detract from the story.

Numerous errors in grammar, punctuation, spelling and capitalization severely detract from the story.

4 3 2 1Gram-mar & Usage

No errors in grammar, punctuation,spelling or capitalization.

1-3 errors in grammar, punctuation,spelling or capitalization.

4-5 errors in grammar, punctuation,spelling or capitalization.

6 or more errors in grammar, punctuation,spelling or capitalization.

45

Example #2 Poor example [one criterion from science lab report rubric]

4 3 2 1Intro-duction

1. Includes question to be answered.

2. States hypothesis based on research and/or reasoning

3. Title is relevant

4. Hypothesis (prediction) is testable

One of the “excellent” conditions is not met.

Two of the “excellent” conditions are not met.

Three of the “excellent” conditions are not met.

46

Example #2Good example

4 3 2 1

Introduction—Stating Research Questions and Hypotheses

States an hypothesis that is based on research and/or sound reasoning and is testable. Report title reflects question or hypothesis.

States an hypothesis that is based on research and/or sound reasoning and is testable. Report title may not reflect the question or hypothesis.

States an hypothesis, although basis for the hypothesis is not clear or hypothesis is not testable. Report title may not reflect the question or hypothesis.

Does not state an hypothesis. Introduction may be a general statement of the topic or the assignment or may be missing or unclear.

47

Example #3 Poor example [one criterion from science oral report rubric]

Good example

  4 3 2 1Props and Visuals

Props and/or visuals (pictures, diagrams, maps) were used throughout the presentation.

A few props and/or visuals were used.

One props or visual was used.

No props and/or visuals were used.

  4 3 2 1Props and Visuals

Props and/or visuals (pictures, diagrams, maps) give listeners an exceptionally clear understanding of the concepts.

Props and/or visuals mostly illustrate the concepts for listeners.

Some props and/or visuals illustrate the concepts and some do not.

Props and/or visuals were irrelevant to the concepts, inaccurate, and/or confusing.

48

6 Points 4 Points 2 Points 0 Points

Title of poster

Title is evident on poster, correctly spelled and capitalized

Title is on poster, but with errors or it is hard to read

No title or heading

Order of life cycle stages

All the stages of the life cycle are in the correct order. Stages are correctly labeled.

One or more stages of the life cycle are in the wrong order.

Not included

Illustrations of life cycle stages

Illustrations of each stage are evident.

One or two illustrations of the life cycle stages are missing.

More than 2 illustrations of the life cycle stages are missing.

Not included

Description of life cycle stages

Stages are described with at least 2 details.

Stages are described with one detail. One or more stage is missing.

Stages are incomplete missing. Stages have one or zero supporting details.

Not included

Overall appearance of poster

Poster is very neat and organized. Title and all sentences have correct spelling, capitalization, and punctuation.

Poster is somewhat neat and organized. Some correct spelling, punctuation, and capitalization. Poster shows signs of little effort.

Poster is messy, many errors, not colored, or unfinished. Poster shows no signs of effort.

First draft, Life Cycle Project rubric

49

Revised version, Life Cycle Project rubric,Used with a checklist for assignment requirements

Advanced ProficientNearing

ProficientNovice

Order of life cycle stages

All the stages of the life cycle are in the correct order and correctly labeled.

One or more stages of the life cycle are in the wrong order.

No order is specified, or order is incorrect.

Illustrations of life cycle stages

Each stage has an illustration that gives an especially clear or detailed view about what happens to the animal then.

Each stage has an illustration that helps show what happens to the animal then.

Some stage illustrations do not show what happens to the animal then.

Illustrations do not help show what happens to the animal during its life cycle.

Description of life cycle stages

Stages are described accurately. Descriptions are especially complete and detailed.

Stages are described accurately.

Stages are described with some inaccurate or incomplete information.

No stages are described, or stages are described inaccurately.

Checklist for assignment requirements

50

• Not used for grading – used formatively

• By students (self and/or peers)

My Poster Checklist

___ My poster has a title.___ My poster is neat.___ My poster is well organized.___ My poster has correct spelling.___ My poster has correct capitalization.___ My poster has correct punctuation.

Critique these Rubrics

51

CATEGORY 5 4 3 2 1 0

Student Info Included (Name, Date, Period)

All items included   One item missing   Two items missingNo info provided

General Info Included

All 8 items included

One item missing or inaccurate

Two items missing or inaccurate

Three items missing or inaccurate

More than 3 items missing or inaccurate

No info provided

Eruption Information

All 6 items included

At least 4 items are included and accurate

Half the information is included or accurate

One 2 items included and accurate

Minimal or no information included or accurate

No info provided

Volcano DiagramClear, accurate diagram; with all 15 parts shown.

Diagram is included; 11-14 of parts clear and accurately shown.

Diagram is included; 6-10 parts clear and accurately shown.

Diagram is included; 3-5 labeled parts.

Diagram has less than 3 parts labeled.

No info provided

Overall Presentation

Clear, neat, organized. Layout well planned.

Layout is planned and organized. Writing is not neat

Info could be better organized. Writing is sloppy

Not organized. Not all info fits properly. Some attempt to make it work. Writing and lines are hastily done.

Very disorganized and poorly prepared. Lines not straight. Spacing is sloppy. Writing is hastily done. May have been done in homeroom.

No suggestions followed for organization and neatness

Use of Creativity

Various materials are used for effect. Attention to detail obvious. Good use of color.

Some use of materials, attention to detail, and/or use of color.

Moderate use of varied materials, attention to detail, and/or color.

Minimal use of either varied materials, attention to detail, or color.

No use of either varied materials, attention to detail, or color.

No creativity

Only half of this score is about understanding volcanoes, and all of those points have to do with counting facts (requiring copying, not even recall)!

52

CATEGORY 5 4 3 2 1 0Student Info Included (Name, Date, Period)

All items included   One item missing   Two items missingNo info provided

General Info Included

All 8 items included

One item missing or inaccurate

Two items missing or inaccurate

Three items missing or inaccurate

More than 3 items missing or inaccurate

No info provided

Eruption Information

All 6 items included

At least 4 items are included and accurate

Half the information is included or accurate

One 2 items included and accurate

Minimal or no information included or accurate

No info provided

Volcano DiagramClear, accurate diagram; with all 15 parts shown.

Diagram is included; 11-14 of parts clear and accurately shown.

Diagram is included; 6-10 parts clear and accurately shown.

Diagram is included; 3-5 labeled parts.

Diagram has less than 3 parts labeled.

No info provided

Overall Presentation

Clear, neat, organized. Layout well planned.

Layout is planned and organized. Writing is not neat

Info could be better organized. Writing is sloppy

Not organized. Not all info fits properly. Some attempt to make it work. Writing and lines are hastily done.

Very disorganized and poorly prepared. Lines not straight. Spacing is sloppy. Writing is hastily done. May have been done in homeroom.

No suggestions followed for organization and neatness

Use of Creativity

Various materials are used for effect. Attention to detail obvious. Good use of color.

Some use of materials, attention to detail, and/or use of color.

Moderate use of varied materials, attention to detail, and/or color.

Minimal use of either varied materials, attention to detail, or color.

No use of either varied materials, attention to detail, or color.

No creativity

Performance Assessment

• Task(s) or assignment must tap the student learning outcome

Knowledge Skills Level of Thinking

• Rubrics or other scoring scheme must accurately describe student performance along a continuum of quality

53

54

Thank you

• The slides and a video of this webinar will be posted at www.engageny.org/video-library

• Next webinar: Test construction 3:30pm-5:00pm on December 1st, 2014

• Feedback: https://www.surveymonkey.com/s/performanceassessmentwebinar