(1)

Makahiki: An Extensible Open-Source

Platform for Energy Challenges

George LeeCollaborative Software Development

LaboratoryInformation and Computer Sciences

University of Hawaii

(2)

Energy Competitions

What is an energy competition?• Groups try to reduce energy usage as much as possible• Can involve participation in events

Why have competitions?• Save money on energy costs• Opportunity to educate people about saving energy

Colleges have done this for over 20 years

(3)

Dorm Energy CompetitionsHodge, Chelsea, Dorm Energy Competitions, Presented at the Behavior, Energy, and Climate Change Conference, November 2010

Over 160 colleges in the US planned to have one in 2010-2011 academic year

Reduce energy use during the competition by 9% on average

(4)

Harvard Green Cup

(5)

Bowdoin

(6)

Energy Challenge Competitions focus on energy reduction

Energy literacy and sustained behavior change

Platform for creating energy challenges and researching behavior change

(7)

MakahikiEnergy competitions• Supports near-real time energy feedback• Event and activity tracking• Personalized feedback

Energy challenge• Pedagogical approach• Social network integration• A serious game

(8)

Serious Games Zyda, Michael, From Visual Stimulation to Virtual Reality to Games, IEEE Computer, September 2005

Definition: “a mental contest, played with a computer in accordance with specific rules that uses entertainment to further government or corporate training, education, health, public policy, and strategic communication objectives.”

(9)

Serious Games as MotivationMcGonigal, Jane, Reality is Broken, Penguin Press, January, 2011

Created several “Augmented Reality Games” that involve real world tasks

Games provide a sense of accomplishment that rewards effort

(10)

Serious Games and Productivity

Reeves, Byron, Total Engagement, Harvard Business School Press, 2009

People spend hours on online games

How can we keep people in the workplace engaged?

Harnessing techniques from games can make businesses more competitive

(11)

Serious Games and Learning Murphy, Curtiss, Why Games Work and the Science of Learning, http://www. goodgamesbydesign.com/Files/WhyGamesWork_TheScienceOfLearning_ CMurphy_2011.pdf

Examined how the Navy used a game to train recruits

Saw a 50-80% increase in individual performance after 1 hour of playing

(12)

“Smart Grid” GameWant to educate users on how to be “green”• Present tasks that help them be more aware• Declare commitments• Hold events like movie nights

Reward users in the form of points

A “Serious Game” based around energy literacy

(13)

Integration with WattDepot

Dorm MetersWattDepotSensors

WattDepotServer Makahiki

WattDepotCloud Cache

(14)

Near Real-Time Feedback Peterson et al, Dormitory residents reduce electricity consumption when exposed to real-time visual feedback and incentives, International Journal of Sustainability in Higher Education, 8(1):16–33, 2007.

Experiment at Oberlin College in 2005

Users with near real-time feedback outperformed those with “low-resolution” feedback

(15)

Go Low

(16)

Evaluating Makahiki Evaluation of the alpha version

Three phases of evaluation:• Mockup Evaluation• In-lab Evaluation• Beta evaluation

Actual competition in October 2011

(17)

Goals of the evaluation How can we provide a system that supports researchers interested in users participating in an energy competition?

How can we effectively test the website and the overall design of the competition?

(18)

Mockup Evaluation Created mockups using Balsamiq Mockups

Brought in friends and family to go through scenarios

Scenarios:• Initial set up• Getting points• Viewing energy use• Viewing prizes

(19)

Mockup Landing Page

(20)

Landing Page

(21)

Mockup Get Nutz Page

(22)

Get Nutz Page

(23)

Mockup Evaluation Results Major changes:• Energy Goal Voting removed• Simplification of the “Get Nutz” page• Altruistic raffle prizes

Points were a significant motivator

On the right track

(24)

Discount Usability Engineering

Nielsen, Jakob, Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier. Cost-justifying Usability, pages 245–272, 1994.

Think aloud protocol, no need to capture video

Scenarios?

(25)

Hutchworld Cheng et al, Hutchworld: Lessons learned. Virtual Worlds, 1834/2000:12–23, 2000.

Developed by Microsoft to help cancer patients

Three dimensional interface

5 minute test period before the evaluation starts

(26)

Onboarding Users are unlikely to know how the system works beforehand

System has to guide the user and expose them to the various mechanics

The quest bar:• List of things users could do on the site• If completed, the quest does not appear• Not worth any points

(27)

Onboarding Evaluation Representative users

Procedure:• Think aloud protocol• Captured computer screen using software• Let the subject use the site for 45 minutes• Post-evaluation questions• $10 bonus and Gift Certificate quests

Two rounds of onboarding evaluations

(28)

April 2011 Results 5 first-year students from the Hale Lehua tower

Issues found:• Many users missed the quest bar• Rejected admin notifications were missed• Individuals frequently guessed at answers• Initial confusion on Go Low and Get Nutz

Positive feedback:• Found it addictive and would recommend it• One wished they could participate

(29)

July 2011 Results 5 first-year students from the Hale Lehua tower

Many improvements:• Time through setup improved• More questions answered correctly• More quests attempted and completed

Issues:• Quests were not set up properly• Timing of rejection notification

(30)

Beta Evaluation Evaluate the game over a period of time

Evaluation of logging system

Beta evaluation:• Brought in friends and family• Held a 1 week mini competition• Surveyed individuals after the competition• Also analyzed database and logs.

(31)

Beta Evaluation Results 4 teams with 5 subjects each

18 of the subjects logged in at least once

Results:• Averaged 915 points• Completed 16.2 activities

Issues:• Must detect Internet Explorer• Restrict the Daily Energy Goal Game’s points

(32)

2011 Kukui Cup 418 people logged in over the course of the competition

850 hours were spent on the site

42 individuals completed the round 3 survey• 25 found the system easy to use•Understood how to play the game

(33)

2011 Kukui Cup Survey Results

(34)

2011 Kukui Cup Survey Results

(35)

2011 Kukui Cup Survey Results

(36)

ContributionsMakahiki is an open source framework for energy competitions

Makahiki acts as a platform for researchers who are interested in user behavior during energy competitions

To evaluate the system, we created a methodology for testing Serious Games.

(37)

Questions? Kukui Cup Website: http://sites.google.com/site/kukuicup

Makahiki project website: http://github.com/csdl/makahiki/