design for learning and assessment in virtual worlds

Brian C. Nelson

Arizona State University

August 2016

Design for Learning and Assessment in Virtual Worlds

Why Virtual Worlds?

Early take: teaching/training is about information transmission and uptake by individuals

More recent view: learning is primarily situated, social activity of collaborating to make sense of and apply content and concepts in specific contexts

Many commercial virtual world-based games are based on social networks and collaborative problem solving…in and outside of the game environment

Virtual world-based games are really good at collecting data about player activities

Why Virtual Worlds (2)?

Virtual Worlds and other digital media fill every moment of a learner’s life…until they enter the classroom

To students in a media-rich world, the classroom can feel like a museum

Virtual worlds engage learners beyond a novelty effect

Virtual worlds may support demonstrations of learning different/beyond that supported by traditional assessments

Educational Virtual Worlds: A study in Contrasts

1. Well -designed virtual worlds are good for learning Many educational virtual worlds are poorly designed

2. Virtual worlds can support innovative assessment of “21st century skills”

Many educational virtual worlds use traditional measures and approaches to assess learning

3. Virtual worlds can support innovative thinking and multiple ways of knowing

Policy and cultural issues result in virtual worlds that guide learners toward homogenous thinking and simple answers

My Goals Related to these Issues

1. Virtual Worlds are poorly designed

2. Virtual Worlds use ill-suited assessments

1. Theory-based design for learning in Virtual Worlds

2. Embed meaningful assessments

Issue Goal


1. Virtual worlds are poorly designed

2. Virtual worlds use ill-suited assessments

1. Theory-based design for learning


Issue Goal

Challenge: Education researchers focus (understandably) on curricular and pedagogical issues Need: Study design of virtual worlds as implemented in educational settings

Better Design for Learning in Virtual Worlds

Complexity in Virtual Worlds

Visual and interaction complexity boosts immersion/embodiment but may hinder learning in school settings

large body of literature on design principles from a cognitive processing perspective Summary: cut the cognitive fat

Yet...successful virtual worlds are frequently highly complex, but players can cope with that complexity and learn

Pragmatic approach: investigate the use of cognitive processing based design to balance complexity, short and long-term engagement, and efficiency

Chris Dede, Diane Ketelhut, Ed Dieterle, Jody Clarke-Midura, Cassie Bowman, and a whole bunch more people!

River City

River City

An Multi-player world to teach scientific inquiry and content skills to middle school students

Students work in teams to discover why people in River City are getting sick

Students gather data, form and test hypotheses More than 20,000 students have taken part

River City Origins

National policy focus on real-world science practices

Push to include science inquiry into the classroom But…realistic inquiry is difficult to teach and

difficult to learn in the classroom Challenge: Create collaborative, situated inquiry

experiences that engage more students in science, particularly those underrepresented in STEM fields

River City Interface: a mess

Cognitive Design

Keep learner focus in the 3-d environment Reduce reading through “natural narration” Use visual and audio signalling techniques to focus

attention Apply spatial contiguity principle Design for “essential complexity”

Individual investigations of context-based science problems in a virtual world

Example 1: Simlandia

Voice vs. Text Chat collaboration

Ben Erlandson (former ASU PhD student) led study

people learn better when words are presented as audio

narration rather than as on-screen text (Mayer, 2005)

helps reduce a “split attention” effect

Do students completing a science inquiry curriculum in a game using voice chat for collaborative communication...

self-report lower levels of cognitive loadshow better performance on a science learning measure

...than students collaborating via text chat?

Results

Cognitive load Voice-based Chat: lower levels of perceived cognitive

load

Learning

almost identical performance for both groups

Why?

Everyone did well on the pre-test

Assessment-performance mismatch

Diane Ketelhut, Catherine Schifter, Younsu Kim, Uma Natarajan, Kent Slack, Angela Shelton, (and many more folks)

SAVE Science

SAVE Science

situated assessment using virtual worlds for science content and inquiry

Virtual world-based Game to assess learning of classroom curriculum in science

Collect data evolving levels of understanding Enable students who don’t do well with

standardized tests to better show understanding

SAVE Science Design Studies (so far)

Visual Signaling Avatar Personalization Spatial Contiguity

Visual Signaling in Virtual Worlds

Virtual worlds work well for learning, especially over long(er) periods of engagement

Virtual worlds are initially confusing, especially for novice student gamers

Low efficiency poses challenges to in-school implementations

Low efficiency challenges assessment reliability and validity

Visual signaling is used to guide players to relevant objects and locations

Sheep Trouble Module

Sheep Trouble Module

https://www.youtube.com/watch?v=uurufkuXu3s Assess students’ knowledge of beginning

speciation/adaptation as well as aspects of scientific inquiry.

New and old flocks of sheep Determine why recently imported sheep are

getting sick and dying Apply understanding of speciation and

adaptation

https://www.youtube.com/watch?v=uurufkuXu3s

Investigating the sheep

Measuring sheep

Visual Signaling: Cognition

Visual Signaling: using visual cues (such as arrows) to direct learner attention to relevant information on the screen or page (or virtual world)

Visual Signaling may lower extraneous cognitive load and/or increase germane load…letting learners focus on tasks rather than on interface (Merrienboer, 2008)

Mixed record in past studies: often found to reduce self-reported cognitive load, but not always coupled with improved learning (Morozov, 2009; Chen & Fauzy, 2008)

Signaling Questions

Can the use of visual signaling techniques reduce perceived extraneous cognitive load in a short game-based assessment?Can use of signaling increase the efficiency in a game-based assessment?

Signaling Study

193 7th graders Sheep Trouble: Assessment of Beginning Speciation Random assignment: signaling/no signaling Lower overall perceived cognitive load (p<.05) Increased interactions with sheep (p<.01) More measurements taken (p<.001) More records entered in notebook (p<.001)

Presenter

Presentation Notes

In the signaling study, 193 7th grade students were randomly assigned either to a version of Sheep Trouble with visual signaling or a version without it. After finishing the module, students completed a cognitive load survey containing questions related to their perception of the difficulty they experienced in using the virtual world, including world navigation, object locating, data collection, and communication with in-world characters. The cognitive load survey was based on the one developed by Paas (1992) and included questions (see below for examples) through which students rated their own levels of mental effort in dealing with the virtual world and its curriculum. One-way analysis of variance (ANOVA) showed students in the visual signaling group reporting significantly lower overall cognitive load than those in non-visual signaling group (p < .05). The effect of visual signaling was significant on the question "How hard did you have to work to communicate with people you met in Scientopolis (p < .05). Non-significant but suggestive differences were seen on two additional questions “How hard did you have to work to find things in Scientopolis you wanted to interact with?” (p = .09) and “How much effort did you have to invest in order to navigate in Scientopolis?” (p = .07). In all cases, students in the visual signaling group reported lower levels of perceived cognitive load. For the task completion rates question, a tally of interactions with curriculum-related objects in the virtual world was automatically recorded. Results showed that students in the signaled version of the world showed significantly more interactions with in-world objects overall (p < .05), more interactions with sheep (p < .01), more measurements taken from the sheep (p < .001), and more records entered into the student’s electronic notebook (p <.001). Thus, it appears that use of the signaling principle allowed students to spend more time on the items crucial to solving the problem presented in the virtual world, giving them a greater chance of successfully completing the assessment task.

Implications and Questions

Use signaling! Why did signaling have a ‘sticky’ effect on

interacting with objects? Would the value of signaling for efficiency be

greater in a high search environment? (One with more visual objects on the screen at once?)

Presenter

Presentation Notes

In the signaling study, 193 7th grade students were randomly assigned either to a version of Sheep Trouble with visual signaling or a version without it. After finishing the module, students completed a cognitive load survey containing questions related to their perception of the difficulty they experienced in using the virtual world, including world navigation, object locating, data collection, and communication with in-world characters. The cognitive load survey was based on the one developed by Paas (1992) and included questions (see below for examples) through which students rated their own levels of mental effort in dealing with the virtual world and its curriculum. One-way analysis of variance (ANOVA) showed students in the visual signaling group reporting significantly lower overall cognitive load than those in non-visual signaling group (p < .05). The effect of visual signaling was significant on the question "How hard did you have to work to communicate with people you met in Scientopolis (p < .05). Non-significant but suggestive differences were seen on two additional questions “How hard did you have to work to find things in Scientopolis you wanted to interact with?” (p = .09) and “How much effort did you have to invest in order to navigate in Scientopolis?” (p = .07). In all cases, students in the visual signaling group reported lower levels of perceived cognitive load. For the task completion rates question, a tally of interactions with curriculum-related objects in the virtual world was automatically recorded. Results showed that students in the signaled version of the world showed significantly more interactions with in-world objects overall (p < .05), more interactions with sheep (p < .01), more measurements taken from the sheep (p < .001), and more records entered into the student’s electronic notebook (p <.001). Thus, it appears that use of the signaling principle allowed students to spend more time on the items crucial to solving the problem presented in the virtual world, giving them a greater chance of successfully completing the assessment task.


1. Virtual Worlds are poorly designed

2. Virtual Worlds use ill-suited assessments

1. Theory-based design for learning


Issue Goal

Emerging research on: Data-mining Statistical methods for making sense of learner

actions

Less research on: Design of tasks and “Work Products” of assessment

supported by highly immersive virtual worlds

Virtual world-based Assessment

Evidence Centered Design for Assessment in Virtual Worlds

…and/or the Presentation Model aspect of ECD(Robert Mislevy)

Assessment Tasks and Work products

Assessment in Game-based learning environments: many researchers and designers focusing mainly on ‘black box’ analysis of data output

Virtual worlds are designed spaces Need full spectrum design for more meaningful

data

Assessment tasks in Virtual Worlds

Virtual world-based tasks support multiple evidence channels in isolation and in combination

Provide complex and interwoven collection of work tools for assessment activities

Example: Global Evidence Channels

Location/Movement (LM) Object Interaction (OI) Communication Activities (CA)

Location tracking•X location visited•Time spent at X•Coordinates

Movement tracking•Direction•Speed•Acceleration/deceleration•Teleporting

Movement patterns•Order of movement•Movement as response•Movement strings over time

Objects:•View•Select•Click•Manipulate•Pickup•Release

Object Types:•Artifacts and inventory•Tools•NPCs•Humans•“intangibles”

•Type•Speak•Response selection•Emote

•In and out of character•Human and NPC•Goal-oriented vs. social

DATA-MINING BASKETBALL TROUBLE

Shanshan Zhang and Slobodan Vucetic

Department of Computer and Information Sciences Temple University

Basketball Module

http://youtu.be/hrZVa2i-e5I Assess students’ knowledge of gas laws and

related properties as well as aspects of scientific inquiry.

Mid-winter basketball tournament Determine why balls at outdoor game don’t

bounce well compared to indoor setting Apply understanding of gas laws (air

pressure/temperature link)

http://youtu.be/hrZVa2i-e5I

Basketball Module

Study

1. Create automated grading models that predict the number of embedded assessment questions a student will answer correctly based on her/his actions in the module

2. 187 students’ records analyzed3. Analyze correlations between multiple-choice scores,

within-game behavior, and free-text answers4. Correlation of .5 (Pearson’s p) with human graders

on predicting performance on in-game multiple choice and short-answer questions

Study

4 important and non-redundant features found: Distinct interactions with in-game objects Number of NPCs talked to Number of objects whose air pressure was measured Number of temperature measurements recorded in e-

notebook

Key task: discover that a decrease in the temperature of several gas systems (basketballs and balloons filled with air) is causing their pressure to decrease.

ExampleA graphical illustration of how our Classification Techniques are able to learn to distinguish between Advanced, Proficient, and Basic student evaluations (x indicates incorrect prediction)• Good predictors for grades are highly non-linear• Spherical boundaries approximately indicate the student groups

Brian C. [email protected]

Questions?

design for learning and assessment in virtual worlds

Technology