Assessing Motivation and Capacity to Arguein Gamified Seminar Setting

Shigeki Ohira† Kousuke Kawanishi‡ Katashi Nagao‡

†Information Technology Center, Nagoya University

‡Graduate School of Information Science, Nagoya University

E-mail: ohira@nagoya-u.jp

10/2/2014 TEEM2014

Technological Ecosystems for Enhancing Multiculturality (TEEM’14)Track 4. Gamification Ecosystems

Introduction

• Focus on Gamification

“The use of game design elements in non-game contexts”

– Successful cases primarily in business sector

– Application to e-learning education

• Require considerable advance preparation in real-world classroom situation

10/2/2014 TEEM2014

Events in classroom

Recording

Game mechanisms

Mapping

Critically important!for

- providing continuous support- measuring effects

Purpose and Approach

• Importance of discussion in laboratory seminars

– Understand a wide range of academic topics

– Express own opinions and brainstorm with others

→ Motivation to participate in discussion is not so high…

10/2/2014 TEEM2014

Sustain and improve motivation to participate in seminarsusing gamification techniques

Purpose

…

…

Student(Presenter)Teacher

Students(Participants)

• Recording semantic structure of seminars+ Gamification framework= Gamified Discussion (GD) System

Integrate gamification framework into seminar environmentApproach

Discussion Recorder

• Capturing video data with multiple cameras and microphones

• Presenter uses a browser-based interface to operate slides

• Each participant uses a special device to input metadata

• Secretary uses a browser-based interface to input statements

• Discussion structures and metadata are semi-automatically recorded

10/2/2014 TEEM2014

Gamification Framework

• Seven game design elements

10/2/2014 TEEM2014

Goal

Goal

Goal

Goal

Goal

Goal

Goal

End

Start

Difficulty adjustment:Adjust degree of difficulty and complexity of goals

Goal

GoalVisualization:Show a changing capacity to argue in real-time and permit users to track status

Rules:Enable participants in a discussion to evaluate one another

Design:Provide rewards and a sense of accomplishment

Social:Promote competition and cooperation among users

Goals:Step-by-step goal setting

Tutorial:Familiarize novices with how to use the system

Self

Others

Argumentative skills：LowMotivation: Low

Argumentative skills: HighMotivation: High

Capacity to Argue Classification and Goal Setting

• Five “capacity to argue” into 83 argumentative skills

– 10 reasoning skills

– 28 analytical skills

– 11 compositional skills

– 26 communication skills

– 8 system use skills

• Goal setting

– Seminar participants are able to set or modify goals as required using a tablet device

10/2/2014 TEEM2014

More difficult and complex toward the top

Higherskill

Lower skill

Argumentative skill graph

Real-time Evaluation of Argumentative Skills

• Statement evaluation interface

– Goal attainment level : five-point

– Statement quality : “+1” or “+2”

– Feedback to the Speaker

10/2/2014 TEEM2014

Participants

Participate in seminar with one tablet device per person

Screen

Other participantsevaluate speaker’s goal

4 53

5 5

Utterance Goal： “Valid criticism”

Average 4.4

Presenter

Feedback

Provide a sense of accomplishment

Level, Exp.

Ranking

• Rank– 10 levels of capacity to argue

– 10 most recent seminars

• Passion! Ranking– Last 3 seminars

– Commend student’s tenacity

– Strongly motivate new students and beginners

10/2/2014 TEEM2014

• Latest Ranking– High score award

– Show on shared displayat the end of each seminar

Show top 3 ranking to prevent users from losing motivation

Loud voice

Talk with a smile

With gestures

Rank 1st

Rank 2nd

Rank 3rd

Visualizing Capacity to Argue

• GD MyPage

– Top, Skill, Status, History, Ability, Item, and Badge tab

– Verify own progress and set next goals

– Badges are earned by pursuing recommended actions

10/2/2014 TEEM2014

Verify their own progress in comparison with others by referring to a radar chart

Set next goals while browsing argumentative skill graph

Radar chart of capacity to argue

Badge“Access to MyPage”, “Achieve 10 goals”, “Get ranking 1st”, etc.

Status tab:

Argumentative skill graph

Skill tab:

Evaluation Experiment

• Participants and experimental period

– 11 students and 3 faculties

– November 14, 2012 to December 25, 2013 (62 seminars)

• Assessment measure

– Motivation

• Number of statements

– Capacity to argue

• Speaking skills : quantity and quality of statements (3 criteria)

• Listening skills : whether he/she is listening closely (5 criteria)

10/2/2014 TEEM2014

11 12 1 2 3 4 5 6 7 8 9 10 11 12

1st period 2nd period 3rd period

Goal setting andmutual evaluation

Mutual evaluation of statement quality and the concepts of Exp. and Levels

Rewards

New school year

Improvements of GD

Summervacation

Springvacation

Details of Participants of Students

Student A M2 M2 M2 M2 M2 GraduationStudent B M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student C M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student D M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student E M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student F M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student G B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student H B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student I B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student J B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student K B4 B4 B4 B4 B4 GraduationStudent L

Prior to joining laboratoryB4 B4 B4 B4 B4 B4 B4 B4 B4

Student M B4 B4 B4 B4 B4 B4 B4 B4 B4

10/2/2014 TEEM2014

11 12 1 2 3 4 5 6 7 8 9 10 11 12

1st period 2nd period 3rd periodNew school year

Total 11 students

（M2 : 1, M1 : 5, B4 : 5）Total 11 students

（M2 : 5, M1 : 4, B4 : 2）

B4 : undergraduate seniorM1 : first year graduate studentM2 : second year graduate student

Evaluating Motivation

• Comparison of the number of statements• Before GD (Apr. 1, 2010 to Nov. 13, 2012; 135 seminars)

• After GD (Nov. 14, 2012 to Dec. 25, 2013; 62 seminars)

• Average number of statements per hour of all students

• Increasing tendency

– Before and after introduced GD system

– Implementation in 2nd period and 3rd period

– Questionnaire: all of the students felt improvement of motivation10/2/2014 TEEM2014

1.07

1.40

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Significantly differentt-test (α=0.05)

1.010.86

1.110.98

1.22

1.73

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

第1期 第2期 第3期

GD導入前

GD導入後

Before GD After GD

Before GD

After GD

1st period 2nd period 3rd period

rewards- Evaluation of statement quality- Concepts of exp. and levels

Evaluating Motivation (cont.)

• Number of B4 statements of each school year

• The number of B4 statements in school year 2013 is greatly increased as compared to other years

• GD contributes to increase motivation of students unfamiliar with seminars

10/2/2014 TEEM2014

0

0.2

0.4

0.6

0.8

1

1.2

1.4

2010 2011 2012 2013

Average number of statements per hour

45% up

(After GD)

Evaluating Capacity to Argue

• Graph of capacity to argue

– Capacity to argue improved in all grades

• B4 students maybe concentrate on listening to the other students while unfamiliar with discussions

• Questionnaire: GD system and mutual evaluation scheme contributes to better “capacity to argue”

10/2/2014 TEEM2014

0

10

20

30

40

50

60

0 10 20 30 40 50 60

List

enin

g sk

ill

Speaking skill

1st period

2nd period

3rd period

B4

M1

M2

Summary and Future Work

• Purpose and approach

– Sustain and improve motivation to participate in seminar

• Incorporate gamification framework into seminar setting

• Analyze one year’s seminar records

• Results

– Improvement of student’s motivation and capacity to argue

• Future work

– Continued operation and assessment of our system

• What happens when removing gamification framework?

– Detailed analysis and refinement in each game element

• What elements contribute to attain educational goals?

– Expansion of application to other research activities

• How smoothly everything flows?10/2/2014 TEEM2014

Gracias por su atención

Ask your question

Preliminary Slides SLIDES

10/2/2014 TEEM2014

Game elements introduced to GD

10/2/2014 TEEM2014

Game element Main mechanism introducing the element

Goals Acquire segmentalized capacity to argueLevelsRankAcquire badgesMissions

Visualization Immediate evaluation feedbackVisualization of capacity to argue on MyPage

Rules Step-by-step learning capacity to argueMutual evaluation of goals

Design AbilityItemProfile image

Social Compare capacity to argueRanking

Tutorial Tutorial seminar

DRIP Cycle

• A cycle of four phases representing knowledge activities focused on discussion

10/2/2014 TEEM2014

Investigation

Recording new knowledgebased on past discussion

Preparation

Creating of presentationmaterials using records

Reflection

Arranging ofdiscussion content

Discussion

Creating of reusablediscussion content

Discussion Browser

10/2/2014 TEEM2014

Video View

Statement View

Layered Seek Bar http://dm.nagao.nuie.nagoya-u.ac.jp

Slide View

Evaluating Capacity to Argue

• Scoring method of speaking skills

– Harmonic mean between two scores for quantity and quality of statements (3 criteria: S1 to S3)

• Quantity score “𝑄𝑡𝑦”:

– S1 : number of statements 𝑁 0,… , 𝑡ℎ𝑒 𝑙𝑎𝑟𝑔𝑒𝑠𝑡 𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑎 𝑠𝑒𝑚𝑖𝑛𝑎𝑟

𝑄𝑡𝑦 = 𝑃(𝑁)

• Quality score “𝑄𝑙𝑡”:

– S2 : evaluated goal score 𝐺 (1,… , 5) and learning difficulty 𝐷 (1, … , 100)

– S3 : evaluated quality score 𝑄 0 𝑜𝑟 1 𝑜𝑟 2

𝑄𝑙𝑡 =𝑆2 + 𝑆3

2=𝑃 𝐺 ∙ 𝐷 + 𝑃(𝑄)

2

– Speaking skill score: 𝑆𝑠𝑐𝑜𝑟𝑒 =2∙𝑄𝑡𝑦∙𝑄𝑙𝑡

𝑄𝑡𝑦+𝑄𝑙𝑡

10/2/2014 TEEM2014

𝑃 𝑎 : function 𝑃 converts 𝑎 to perfect score of 100

Learning difficulty of goal is determined by pre-questionnaire

Evaluating Capacity to Argue (cont.)

• Scoring method of listening skills

– Whether he/she is listening closely when another person is talking (5 criteria: L1 to L5)

– L1 : Press agreement/marking button with special device

– L2 : Assess quality of statement as 1 or 2 points

– L3 : Make a follow-up statement in response to a statement

– L4 : Evaluation timing in response to a goal (0, …, 100)

» Within 5 seconds after the end of statement

– L5 : Reliability of evaluation score in response to a goal (0, …, 100)

» Comparison with evaluation by faculties

» Goal score and quality score for each statement

a) All statements 𝑠𝑎 within discussion segment includes actions L1 to L3 … perfect score

b) For all other statements 𝑠𝑏, add L4 and L5 scores at the ratio of 1:1 and convert to 100

– Listening skill score:

10/2/2014 TEEM2014

- if 𝑠𝑖 ∈ 𝑠𝑎 then 𝑃(𝑠𝑖)=100

- else if 𝑠𝑖 ∈ 𝑠𝑏 then 𝑃 𝑠𝑖 =𝐿4+𝐿5

2

𝐿𝑠𝑐𝑜𝑟𝑒 =1

𝑁𝑎𝑙𝑙

𝑖=1

𝑁𝑎𝑙𝑙

𝑃(𝑠𝑖)

Motivation based on Rewards

• Intrinsic motivation

– Derived from ambition and curiosity of human

– Gamification framework

• Goals, Visualization, Rules, Social

– What someone wants to do, what one wants to become

• Extrinsic motivation

– Derived from rewards and punishments

– Gamification framework

• Design: Experimental point, Level, Bonus point, Rank, Ability, Item, Coin, Badge

– Fun and pleasure

10/2/2014 TEEM2014

Player Type : Interest Graph [Bartle 96]

10/2/2014 TEEM2014

WorldPlayers

Acting

Interacting

AchieversKillers

ExplorersSocializers

Earn Points, Level upCompetition

Discovery of new oneCommunication

Tutorial

• Handling of structural remote control

– Pointing on screen, reservation of statement, etc.

– Mutual evaluation of goals

10/2/2014 TEEM2014

Item of tutorial:“Underline appropriate places”

Current goal score

Total records

• Number of seminars: 96

• Number of statements: 6,812

• Number of assessments: 44,979

• Number of badges earned: 1,012

• Number of goal achievement: 769

10/2/2014 TEEM2014

Metadata about Discussions

• Statement segment

– Start and end times of statements

– Each participant uses a special deviceto declare start/end of his/her statement

• Discussion segment

– Division discussion into units of topicby type of statement

• Start-up statement: To begin a new topic

• Follow-up statement : To continue previous topic

– Input type by changing direction of the device when user begins utterance

– Each discussion segment has tree structurewhich begins the start utterance

• Attitude towards statement

– Agreement/Disagreement, Marking

10/2/2014 TEEM2014

Start

Follow

Start

Follow

FollowFollow

Approach

• Recording semantic structure of seminars+ Gamification framework= Gamified Discussion (GD) System

10/2/2014 TEEM2014

Integrate gamification framework into seminar environment

Game design elements

Meeting

Active participation

Enjoy game elements along with the improvement of capacity to argue

Goal of fulfillment

Attractive interface

Assessmentand

feedback

Evaluating Capacity to Argue (exactly)

• Scoring method of speaking skill

– Harmonic mean between two scores for quantity and quality of statements

• Quantity score “𝑄𝑡𝑦” :

– S1 : number of statements 𝑁 0,… , 𝑡ℎ𝑒 𝑙𝑎𝑟𝑔𝑒𝑠𝑡 𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑎 𝑠𝑒𝑚𝑖𝑛𝑎𝑟

𝑄𝑡𝑦 = 𝑃 𝑁 = 100 ∙𝑁

𝑁𝑚𝑎𝑥

• Quality score “𝑄𝑙𝑡” :

– S2 : evaluated goal score 𝐺 (1,… , 5) and learning difficulty 𝐷 (1, … , 100)

– S3 : evaluated quality score 𝑄 0, 1, 2

𝑄𝑙𝑡 =𝑆2+𝑆3

2=𝑃 𝐺∙𝐷 +𝑃(𝑄)

2

= 0.5 ∙ 100 ∙1

𝑁

𝑖=1

𝑁𝐺(𝑖) ∙ 𝐷(𝑖)

500+ 0.5 ∙ 100 ∙

1

𝑁

𝑖=1

𝑁𝑄(𝑖)

2

– Speaking skill score: 𝑆𝑠𝑐𝑜𝑟𝑒 =2∙𝑄𝑡𝑦∙𝑄𝑙𝑡

𝑄𝑡𝑦+𝑄𝑙𝑡10/2/2014 TEEM2014

𝑃 𝑎 : function 𝑃 converts 𝑎 to perfect score of 100

Learning difficulty of goal is determined by pre-questionnaire

Real-time Evaluation of Argumentative Skills

• Statement evaluation interface

– Goal attainment level : five-point

– Statement quality : “+1” or “+2”

– Feedback to the Speaker

10/2/2014 TEEM2014

Participants

Participate in seminar with one tablet device per person

Screen

Other participantsevaluate speaker’s goal

4 53

5 5

Utterance Goal： “Valid criticism”

Average 4.4

Presenter

Feedback

Provide a sense of accomplishment

Level, Exp.

Reward commensurate with the number of statements

during a seminar

Mission

MISSION2 state. bonus → coin+3003 state. bonus → exp+604 state. bonus → coin+1500

Discussion Room

10/2/2014 TEEM2014

サーバ室

dms01dms02

ss02

DM-VIS

DM-MAIN

書記PC

発表PC

canopus

NICE

サーバ・PC

スイッチングハブ

無線LAN AP

情報コンセント

研究室内LAN (1系)

Wiiポインタ用LAN (3系)

gw

Bluet01-05

IB南395

Gamification 11 elements

10/2/2014 TEEM2014

1 Step-by-step great goal

2 Tutorial

3 Autonomy and independence

4 Rules

5 Direct feedback

6 Design and production

7 Difficulty control

8 Competition

9 Cooperation

10 Visualization of the situation

11 Title and growth

http://gamificationlab.jp