Presentation at the 8th Annual Games for Change FestivalNew York, New York—June 22, 2011

Girlie C. Delacruz

Games as Formative Assessment Environments: Making Assessment Criteria Explicit and Incentivizing Use of Feedback

2 / 322 /

Background

• Games as context for formative assessment

• Two issues

Make assessment criteria explicit

Motivate student use of instructional feedback and help

3 / 323 /

What is Formative Assessment?

Formative Assessment:Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967)

4 / 324 /

Same Concept, Different Context

Formative Assessment:Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967)

Games:Use and interpretation of game performance information with intent to adapt learning, such as provide feedback.

Assessment Architecture: General Model

5 / 27

Assessment Architecture: Save Patch

6 / 27

7 / 327 /

The Game: Save Patch• The size of a fraction is relative to a whole unit

8 / 328 /

The Game: Save Patch• Like whole number integers, fractions with like

denominators can be added together to produce a given quantity

9 / 329 /

Scored Events: Choosing the Coil Size

Math Knowledge Required Game Knowledge RequiredIn mathematics, one unit is understood to be one of some quantity (intervals, areas, volumes, etc.).In our number system, the unit can be represented as one whole interval on a number line.Positive integers are represented by successive whole intervals on the positive side of zero.

The interval between each integer is constant once it is established.

The vertical red bars denote the whole unit.

Positive non-integers are represented by fractional parts of the interval between whole numbers.

Grid: The spaces between the green dots are the parts of the whole unit.

Coil: The coil pieces are parts of a whole unit coil. The denominator of a fraction represents the number of identical parts in one whole unit. That is, if we break the one whole unit into “x” pieces, each piece will be “1/x” of the one whole unit.

Grid: The number of spaces between the green dots is the denominator.

Coil: The number of coil pieces the whole unit is broken into is the denominator.

10 / 3210 /

Scored Events:Adding Coils

Math Knowledge Required Game Knowledge Required

Only identical (common) units can be added to create a single numerical sum.

If given different coils with different units, the coils must be changed so that they are the same unit before they can be added together.

11 / 3211 /

Scored Events: Patch Reaches the Goal

Math Knowledge Required Game Knowledge RequiredPositive integers can be broken (decomposed) into parts that are each one unit in quantity.

The length of the jump is the number of pieces between the blocks.

All rational numbers can be represented as additions of integers or fractions.To add quantities, the units (or parts of units) must be identical.Identical (common) units can be added to create a single numerical sum.

Add the correct number of coils that match the length of the jump.

The numerator of a fraction represents the number of identical parts that have been combined. For example, ¾ means three pieces that are each ¼ of one whole unit.

Grid: The top number of the jump distance equals the total number of spaces to jump over.

Coils: The top number of the sum of the coil pieces on the trampoline represents the number of coil pieces that have been added together.

12 / 3212 /

Making Assessment Criteria Explicit

13 / 3213 /

Hypothesis A

• Externally representing the learning objectives and the criteria by which students will be assessed can support learning

14 / 3214 /

Incentivizing Use of Feedback

15 / 3215 /

Hypothesis B

• Incentivizing the use of additional feedback will have a positive impact on math achievement measures, game performance measures, and use of feedback

16 / 3216 /

Research Questions

• Are there differential effects based on the amount of explanation of the scoring rules provided and incentivizing the use of feedback on

Math achievement performance

Game performance measures

Access of feedback help

17 / 3217 /

Full Explanation of Scoring Rules

Click here for help

18 / 3218 /

Scoring Rules Explanation Plus Incentive

To earn back 50 points, click here to get help on choosing the right denominator!

19 / 3219 /

Points-Only Feedback

Click here for help Click here for help

20 / 3220 /

No Scoring Rule Information

Click here for help

21 / 3221 /

Example of Feedback

22 / 3222 /

Independent Variables

23 / 3223 /

Dependent Measures

• Math Achievement:

Pretest (31 items)

Posttest (44 items, additional items in game context)

• Game Performance Measures

Number of coils added together, wrong-sized unit coils, resets, and failed attempts

Maximum level reached

• Help-Seeking Behavior

Proportion of times feedback was accessed

Accessing general help menu

24 / 3224 /

Methodology

• Participants

Data collected from 112 4th-6th grade students

After-school context

• Treatment

20-minute interval

Randomly assigned to treatment conditions

• Procedure

PretestGame playPosttest and Survey

25 / 3225 /

Math Outcomes: Overall Sample

• Incentive + Scoring Explanation condition had higher normalized change scores (d = .53)

• Interaction between pretest scores and treatment variation on game-like item scores

For students with:

low prior knowledge, Incentive + Scoring Explanation more beneficial

high prior knowledge, just the Scoring Explanation was more effective

26 / 3226 /

Math Outcomes: Low Academic Motivation

• For students with low self-efficacy:

Both (1) Incentive + Scoring Explanation and (2) Incentive + Scoring Explanation had higher posttest scores than minimal scoring rules information (d = .88)

• For students with low game experience, low math self-concept, and low preference for cooperative learning:

Incentive + Scoring Explanation resulted in higher change scores (d = .88 – 1.17)

27 / 3227 /

Game Outcomes

• Incentive + Scoring Explanation was superior to minimal scoring information for the within-game outcomes

Added fewer coils with different denominators

Had lower number of resets

28 / 3228 /

Use of Feedback by Incentive Group

• Accessed feedback less overall and spent less time on feedback screen

• However… accessed the same topic in general help menu more often, and when feedback was accessed, solved the level more quickly and with fewer mistakes

29 / 3229 /

Summary: Making Assessment Criteria Explicit

• Providing just the scoring rules explanation resulted in better game performance

• Providing just the explanation of scoring rules did not lead to better performance on math outcomes

Except for students with low self-efficacy

30 / 3230 /

Summary:Incentivizing Use of Feedback

• Incentive + Scoring Information is superior to minimal scoring information, with better performance on:

Math achievement measures

Game play

31 / 3231 /

Next Steps: Theoretical Explanations

• Examine cognitive and motivational processes that underlie relation between incentivizing use of feedback and learning

What is incentivized, rewarded or punished signals the norms that

are valued or to be avoided

Accesses feedback

Cognitive Engagement:

Deeper processing of information

Achievement:Improves

performance

Utility Value: Increases

perceived value of feedback

32 / 3232 /

Next Steps: Game Design Studies

• Feedback modifications

Increase opportunities to receive treatments

Prior to re-attempting solution rather than after a mistake

Provide feedback on a randomized schedule

• Incentive modifications

Make incentives more relevant to the game

Rewards for performance rather than contingent on behavior

cats.cse.ucla.edu

Girlie Delacruzgdelacruz@cse.ucla.edu

CONTACT INFORMATION