usability of in-home energy displays - etcc · usability of in-home energy displays ht.11.sce.013...

Design & Engineering Services

Usability of In-Home Energy Displays

HT.11.SCE.013 Report

Prepared by:

Design & Engineering Services

Customer Service Business Unit

Southern California Edison

December 2012

Usability of In-Home Energy Displays HT.11.SCE.013

Southern California Edison Page i Design & Engineering Services December 2012

Acknowledgements

Southern California Edison’s Design & Engineering Services (DES) group is responsible for

this project. It was developed as part of Southern California Edison’s HVAC Technologies

and System Diagnostics Advocacy Program (HTSDA) under internal project number

HT.11.SCE.013. DES project manager Alvaro Mendoza conducted this technology evaluation

with overall guidance and management from Jerine Ahmed. For more information on this

project, contact [email protected].

Disclaimer

This report was prepared by Southern California Edison (SCE) and funded by California

utility customers under the auspices of the California Public Utilities Commission.

Reproduction or distribution of the whole or any part of the contents of this document

without the express written permission of SCE is prohibited. This work was performed with

reasonable care and in accordance with professional standards. However, neither SCE nor

any entity performing the work pursuant to SCE’s authority make any warranty or

representation, expressed or implied, with regard to this report, the merchantability or

fitness for a particular purpose of the results of the work, or any analyses, or conclusions

contained in this report. The results reflected in the work are generally representative of

operating conditions; however, the results in any other situation may vary depending upon

particular operating conditions.


Southern California Edison Page ii Design & Engineering Services December 2012

EXECUTIVE SUMMARY Southern California Edison directed the Western Cooling Efficiency Center (WCEC) of the

University of California, Davis, to conduct a research project on the usability of In-Home

Energy Displays (IHEDs). In two related studies conducted during 2011-2012, this project

evaluated IHEDs available in the consumer and utility markets. It also developed a mockup

prototype of an IHED that was then experimentally tested by varying a set of common

display features.

IHEDs are used to save energy by providing feedback to users on their energy consumption.

In addition, they have been used as a part of demand response programs, and have

facilitated home automation. The success of residential IHEDs as key players in engaging

and influencing users to reduce energy consumption relies heavily on the ability and

motivation of people to read the displayed information and respond to it with energy saving

actions. For these reasons, the ease of using energy feedback is key to success. Ease of use

is affected by design characteristics of the IHED interface, such as numerical vs. graphic

information, size of fonts, and ease of navigation. Effectiveness of these devices is also

influenced by other environmental and social variables, such as the energy saving goals of

users, social comparisons with neighbors and peers, and incentives to monitor and decrease

energy usage. The studies in this project address both interface design and social variables.

The, ability of occupants to perform certain tasks and successfully extract information using

IHEDs has not been explicitly examined.

The project evaluates the usability of a device to help reduce energy use by utility

ratepayers by providing them with detailed information on energy use that they can use to

make informed decisions. The project explores currently available IHED device interface

features, as well as promising potential features that have not yet been demonstrated; and

evaluates their effectiveness in a controlled experimental setting.

Energy savings provided by IHEDs depend on their usability that depends on the

characteristics of the devices, and how well their interfaces are designed. To design the

appropriate evaluations and usability tests for IHED interfaces, the WCEC research team

conducted first a market review of currently available IHEDs. A list of available displays was

developed to characterize their features, along with a characterization table to summarize

available IHEDs, organized by feature.

Additionally, expert heuristic usability testing was conducted with a sample of four IHEDs.

This market review was used to design the simulation prototypes used later in the test of

IHEDs usability. The main part of the study examined users’ reactions and responses to

simulated energy feedback interfaces that were representative of devices available on the

market. The IHED interfaces used in the study were abstracted from the heuristic expert

review of market devices from the first part of the study. The interfaces present increasing

levels of complex information to a user. Participants were questioned on their ability to

retrieve information from the IHED experimental interface, put that information in a

meaningful energy use and reduction context, and use that information to make energy use

decisions. If participants can retrieve and understand the information and develop a plan to

save energy based upon this information, then energy savings should result from this

technology. Recommendations on desirable specs for IHED interfaces are based on the

results.

A population that is interested in energy savings can be encouraged to adopt energy

efficient behaviors with an IHED, but issues of installation and design must be considered in

more detail by producers, utilities, and other consumers.


Southern California Edison Page iii Design & Engineering Services December 2012

The market review indicates the presence of an in-flux market, with low penetration and

wide variation in device offers and features. There are no set standards for features,

installation, compatibility, and connectivity. This fluid market probably confuses individual

consumers about the product, its function, and its effectiveness, making adoption less

likely. Despite the confusion in the marketplace, one can still purchase and install IHED

devices, and obtain feedback on energy consumption.

The usability study indicates that users can have difficulty interpreting basic information

presented in graphs, and do better when simple numbers and messages are provided. This

finding should be considered when evaluating graphic and information rich interfaces that

may be confusing to users.

Results also indicate that there are some pitfalls in using social data comparisons

(comparisons with neighbors); users seem to lose motivation to save energy when

neighbors are seen as using more.

Finally, there is strong evidence for the effectiveness of general energy saving tips and

specific energy usage diagnostic messages as a type of IHED information. IHEDs in the

market today do not present such messages, but our findings suggest that they would be

very effective in making IHED information actionable.

The following recommendations are based on the results from this project:

Gadget vs. Tool Framing: IHEDs or other energy consumption feedback devices

should be provided in the context of a planned energy savings action program such

as providing goals to reduce energy use.

Conduct additional research on the impact of general energy saving tips and

diagnostic information and messages that are provided directly in the IHED while

people are checking their energy usage.

Engage manufacturers to develop and incorporate diagnostics algorithms into their

devices, since this was found to be quite beneficial, but it is not possible with today’s

technology.

Expand research on the effects and desirability of social comparison information,

including possible “boomerang” effects.

Expand research on the impact of complex, graphic and information rich IHED

devices, focusing on users’ ability to extract correct and actionable information.

Evaluate simple, clear messages as part of the IHED information. These may be

more useful and satisfying to users.

IHED Usability scores, like the one proposed here, should combine traditional

usability measures with an effectiveness usability measure, gauging whether the

IHED provides information that facilitates energy saving actions.

Conduct further research into deployability and usability of particular models prior to

including them in any incentive programs.


Southern California Edison Page iv Design & Engineering Services December 2012

ACRONYMS

CO2 Carbon Dioxide

HAN Home Area Network

HTSDA HVAC Technologies and System Diagnostics Advocacy Program

IHED In-Home Energy Display

kWh kilowatt Hours

UCD University of California, Davis

WCEC Western Cooling Efficiency Center

WV West Village Residential Complex

ZNE Zero Net Energy


Southern California Edison Page v Design & Engineering Services December 2012

CONTENTS

EXECUTIVE SUMMARY ______________________________________________________ II

INTRODUCTION __________________________________________________________ 1

BACKGROUND __________________________________________________________ 2

ASSESSMENT OBJECTIVES __________________________________________________ 3

TECHNOLOGY/PRODUCT EVALUATION ________________________________________ 4

Review of Market Available IHEDs ................................................................. 4

IHED Spectrum ...................................................................................... 5 Characterization of Market IHEDs ........................................................... 10 Heuristic Evaluation .............................................................................. 11

TECHNICAL APPROACH/TEST METHODOLOGY _________________________________ 17

Basic Information ...................................................................................... 17

Contextualizing Information ........................................................................ 17

Decision Making Tools ................................................................................ 17

Test Plan .................................................................................................. 18

Information ......................................................................................... 19 Meaning/Context .................................................................................. 19 Actions/Decisions ................................................................................. 19 Mockup Designs ................................................................................... 20

RESULTS_______________________________________________________________ 22

A final evaluation of the devices Initial Evaluation .................................... 23 Information Accessibility ....................................................................... 25 The Effect of Social Information ............................................................. 27 Taking Action ....................................................................................... 30

DISCUSSION ___________________________________________________________ 34

CONCLUSIONS _________________________________________________________ 37

RECOMMENDATIONS ____________________________________________________ 39

APPENDIX A ___________________________________________________________ 40

Research Team ......................................................................................... 40

APPENDIX B ___________________________________________________________ 42

APPENDIX C ___________________________________________________________ 43


Southern California Edison Page vi Design & Engineering Services December 2012

APPENDIX D ___________________________________________________________ 44

Usability Controlled Experiment .................................................................. 44

Questionnaire Items ............................................................................. 44

APPENDIX E ___________________________________________________________ 49

APPENDIX F ___________________________________________________________ 67

GLOSSARY ____________________________________________________________ 69

REFERENCES ___________________________________________________________ 70


Southern California Edison Page vii Design & Engineering Services December 2012

FIGURES Figure 1. IHED Hardware (Clamp-on sensor, transmitter, display)

(source: http://www.electricity-monitor.com/wireless-

energy-monitors-c-37.html) .......................................... 6

Figure 3. Classification of the HAN Market (Source: GreenTech

Media) ........................................................................ 9

Figure 5. Sample Mockup Display with three navigation screens.

The first two screens were available for navigation when

participants were first presented with the study. The

third screen was available for the second part of the

study, and appeared as a “pop-up” on the display. ......... 21

Figure 6. Demographics: Age distribution of participants. (n=249) ... 22

Figure 7. Demographics: education level of Usability Testing

Controlled Experiment participants. (n=249) ................. 23

Figure 8. Evaluations of the mockup IHED interface along a series

of hedonic and utility dimensions. ................................ 24

Figure 9. Frequency of respondents’ subjective valuation of IHEDs,

in US Dollar ranges. (N =249) ..................................... 24

Figure 10. Percentages of correct, incorrect, and “unsure”

responses to 4 multiple choice questions testing ability

of respondents to extract information from IHED

interface. .................................................................. 26

Figure 11. Participants did better at responding to questions in

screen 1 about cost (1.1kW; top left corner screen 1)

and cost (16 cents; data point in screen 1), than to

questions in screen 2 requiring responding to a data

point (500kW; bottom screen 2) and projecting a trend

(116 dollars, bottom screen 2). .................................... 26

Figure 12. Comparison of responses to 2 information questions,

comparing participants who saw displays with vs.

without budgetary information. Budgetary information

included a cost projection trend drawn in red in the

graph, but it was a more complex screen. Questions are

“hourly cost” and “projected cost” in Table 4 above. ....... 27

Figure 13. Comparison of response patterns to the question on

comparative energy usage (“The household ‘s energy

use seems to be”….), across three conditions of social

information available. ................................................. 28


normative energy usage, across three conditions of

social information available. ........................................ 29


Absolute judgment on energy usage, across three

conditions of social information available. ...................... 30


Southern California Edison Page viii Design & Engineering Services December 2012

Figure 16. t-test p <.001 from no extra info to Tips or Diagnostics.

No significant difference found between Tips and

Diagnostics ................................................................ 31



Diagnostics. ............................................................... 31



Diagnostics. ............................................................... 32



Diagnostics ................................................................ 32

Figure 20. Respondents’ liking and usefulness ratings of a series of

IHED mockup features. ............................................... 34

Figure 21. Usability scores compared across kind of actionable

information provided (none, general tips, specific

diagnostics), and kind of social comparison provided

(usage higher than neighbors, Usage lower than

neighbors, No social information) . ............................... 35

Condition 1. Basic Information Only x Specific Diagnostic

Information ............................................................... 49

Condition 2. Basic Information Only x General Tips ......................... 50

Condition 3. Basic + Budget x Specific Diagnostic Information ......... 51

Condition 4. Basic + Budget x General Tips ................................... 52

Condition 5. Basic + Social Information Higher Than Neighbor x

Specific Diagnostic Information .................................... 53


General Tips .............................................................. 54

Condition 7. Basic + Budget + Social Information Higher Than

Neighbor x Specific Diagnostic Information .................... 55


Neighbor x General Tips .............................................. 56

Condition 9. Basic + Social Information Lower Than Neighbor x

Specific Diagnostic Information .................................... 57

Condition 10. Basic + Social Information Lower Than Neighbor x

General Tips .............................................................. 58

Condition 11. Basic + Budget + Social Information Lower Than

Neighbor x Specific Diagnostic Information .................... 59


Neighbor x General Tips .............................................. 60

Condition 13. Basic x No Energy Saving Suggestions ...................... 61

Condition 14. Basic + Budget x No Energy Saving Suggestions ........ 62


No Energy Saving Suggestions ..................................... 63


Southern California Edison Page ix Design & Engineering Services December 2012


Neighbor x No Energy Saving Suggestions .................... 64

Condition 17. Basic + Social Information Lower Than Neighbor x No

Energy Saving Suggestions ......................................... 65


Neighbor x No Energy Saving Suggestions .................... 66

TABLES Table 1. Multiple choice questions gauging ability of participants to

extract correct information from IHEDs. ........................ 25

Table 2. Complete wording of questions evaluating users’ liking and

perceived usefulness of IHED features .......................... 34


Southern California Edison Page 1 Design & Engineering Services December 2012

INTRODUCTION Currently, homeowners have very little access to information about how much energy they

are consuming. Paper utility bills have limited usefulness in allowing a customer to track

their energy use and compare it with earlier usage or with others. Most utilities have on-line

functionality that allows users to view historical energy use, by bill (or month). The recent

advent of “smart meters” (meters that collect usage data at more frequent intervals and can

be configured to provide this information to utility or third-party applications) makes it

possible to provide hourly historical energy consumption data. This still falls short of the

desire to view current energy use.

Energy feedback in-home displays and web interfaces allow residential occupants to monitor

their energy use at home or check their energy use remotely. In-Home Energy Displays

(IHEDs) are increasingly being developed, produced, and offered in the market as a tool to

reduce energy consumption and energy waste by residential users, as well as to facilitate

demand response and home automation. Past research on displays shows a wide range of

effectiveness rates in producing energy savings1, that may partly depend on the ease of use

of specific devices and on their different available features such as feedback format,

information and content provided, ability to set individual goals, comparisons of energy use

with other users2, or past use, etc. The usability of these displays, the ability of occupants

to perform certain tasks and successfully extract information, has not been explicitly

examined. However, similar studies of usability in thermostats have revealed a significant

amount of confusion leading to inability to perform simple tasks in a reasonable amount of

time3

Additionally, many energy feedback displays are now offered as integrated systems that not

only provide information, but also allow the user to remotely control appliances and other

energy - consuming services. An even more complex feature of energy feedback displays

that holds promise is the ability to provide specific diagnostic information about household

energy use at the appliance/service level. Specific diagnostic information may be key,

especially considering that general “tips” or advice for saving energy can be ignored by

consumers, even when they have asked for them4 . The ideal, of course, is that the feature

offered by an energy feedback interface both motivates and enables energy conservation

choices by users.



BACKGROUND There are obstacles to the successful implementation of energy feedback devices and

interfaces. The market of energy feedback devices –whether in-home displays, networked

systems with online dashboard access and apps, or whole house integrated control systems,

is currently in a state of flux and confronts the potential consumer with a number of

uncertainties. There are no standardized formats for the content provided by an energy

feedback display, nor is there a single standardized way for the device to connect with home

appliances, smart-meters, or household electrical panels. (Standards such as Zigbee, Wi-Fi,

SEP have all been used with different systems.) Residential household consumers motivated

to acquire an energy feedback device for their household may find it difficult to determine:

Whether a device works with their electrical system or utility

What information is available from the display

What features are available

How the device helps in energy savings.

If a consumer makes the decision to buy a device, its installation can prove quite

challenging. If consumers are expected to proactively seek and purchase market available

devices, it may be important to evaluate how they respond to the scope of existing offers

and their characteristics.

Confusing market offers and usability problems may be significant obstacles to the adoption

of these devices and their success in the market. In the current state of development and

innovation it may well be that the increasing complexity of device features is making

consumers hesitant to engage with these products, and frustrated when they use them for

the first time5.

This project used a controlled usability experiment to explore differences in effectiveness

between different features available in market IHEDs. Researchers created a mockup

simulated IHED and varied the amount, complexity, and kinds of information offered. The

study then measured users’ ability to extract correct information and use the information to

make energy saving decisions.



ASSESSMENT OBJECTIVES The goal of the project was to evaluate the usability of a device that can help reduce the

energy consumed by SCE ratepayers by providing them with detailed information on energy

use that they can use to make informed decisions. The project explores IHED device

interface features currently available in the market, as well as promising potential features

(such as diagnostic messaging) that have not yet been demonstrated, and evaluates their

effectiveness in a controlled experimental setting.

This study intends to advance our understanding of IHED usability barriers, and provide

recommendations on preferred usability design for this kind of device. The study also

evaluates whether different forms of information provided in IHEDs are helpful in motivating

and enabling residential end users to make decisions that may lead towards energy savings.

The results of the review of available models and characterization of capabilities are

summarized in Table 1.

Findings of this study are followed by recommendations about:

The most effective IHED features for design

Future research projects

How to choose devices for pilot programs

It also suggests the research and development of features that, while not currently available

in the market, appear to hold promise as enablers of effective consumer choice regarding

energy use.



TECHNOLOGY/PRODUCT EVALUATION The first step in evaluating the usability of IHEDs, was to examine the products available in

the consumer and utilities’ market and to identify common characteristics of IHEDs that can

be subject to a usability evaluation.

The wide range of offers in the IHED market, and its fluidity (with products being constantly

dropped or updated), makes this a particularly difficult task. The continuum extends from

first generation IHEDs that offer only limited data on energy used and cost, to automated

home control systems that go way beyond energy in their information and control tools.

This makes it almost impossible to determine a single standard usability evaluation protocol

that is valid across the spectrum. This report provides an extensive list of IHED and energy

monitoring systems available in the market for individuals, utilities, or developers, and

provides a characterization table of available features for each IHED identified. This list

serves, mostly, as an indication of the heterogeneity of the IHED market.

The second step involved a heuristic examination of four devices acquired by the research

team. Given the market conditions, the research team restricted the heuristic evaluation of

displays to the subset of those available for end user purchase, since they were the only

ones available. Thus, the heuristic evaluation focused only on devices that were available to

be purchased and installed by individual residential customers, and that focused exclusively

on electricity feedback6. It is the characterization of this subset of IHEDs, used to develop

the usability testing protocol, to be used in the controlled experiments.

REVIEW OF MARKET AVAILABLE IHEDS An extensive list of IHEDs and other energy monitoring systems available in the

market was compiled by the research team (see Appendix A). First, the authors of

this report who have expertise in feedback and control devices reviewed the recent

literature on the topic (e.g., Darby 2006, Fischer 2008, Neenan 2009, Froehlich

2010, Ehrhardt-Martinez, 2010, Karlin 2011, LaMarche 2012). Some of the devices

included in this list come from these reports. The list was supplemented with devices

mentioned in technical magazines, online e-zines consulted during fall/winter 2011

and from online searches for energy feedback displays. Examples of the e-zines and

magazines consulted are:

http://www.greentechmedia.com/

http://news.cnet.com/greentech/

http://gigaom.com/cleantech/

http://www.smarthome.com/_/index.aspx

http://www.automatedbuildings.com/

http://www.ashrae.org/resources--publications/periodicals/ashrae-journal

http://www.homeenergy.org/

Researchers also contacted manufacturers, energy consultants and other energy

feedback researchers to help complete the inventory. The authors tried to be

comprehensive in the survey, gathering all the information to choose the best

devices based on range of capabilities and range of different feedback formats. This

market is in rapid and constant change and this list is a snapshot in time. WCEC

http://www.greentechmedia.com/

http://news.cnet.com/greentech/

http://gigaom.com/cleantech/

http://www.smarthome.com/_/index.aspx

http://www.automatedbuildings.com/

http://www.ashrae.org/resources--publications/periodicals/ashrae-journal

http://www.homeenergy.org/



found a wide variety of devices with different functions and interface options that

currently have almost no market penetration.

After collecting extensive information on all kinds of devices and systems available in

the market, the team created an extensive list of market available IHEDs, provided

in Appendix B. This list shows the broad spectrum of devices and their features, as

they are described by the vendors. The following section analyzes the variability in

the IHED spectrum in more detail.

IHED SPECTRUM

IHEDs are devices whose main function is to show the instantaneous and cumulative

energy consumption of a home or an appliance. Most of them also display the cost of

the energy used, calculated based on either data input by users or information

coming from Utilities. IHEDs have been investigated by utilities in several Energy

Efficiency (EE) and Demand Response (DR) projects in the past few years. Examples

include, SMUD (small commercial thermostat and IHED for DR and EE, 2010), TXU

(Smart thermostats BrightenSM iThermostat for DR and EE, 2010), Bluebonnet

Electric Cooperative (Control 4 home automation for DR and EE, 2009), Reliant of

Texas (Nest, 2012), and BC Hydro (2009).

Goals of these projects were to increase customers’ awareness and reduce energy

consumption and peak demand. Through price signals (variation of the cost of

electricity in peak periods) sent to the displays, users were encouraged to voluntarily

decrease their energy consumption. This approach is significantly different from

taking direct control of customers’ equipment to match Utility’s needs and arguably

more acceptable for consumers. Traditionally IHEDs have been designed to monitor

electricity, but recent models also display gas and water consumption. The first

generation of these devices simply displayed power (kW) being used instantly and

cumulative energy (kWh) used in a period of time. An example of such architecture

is shown in Figure 1. The device records power directly from the utility meter

through a clamp-on sensor.



FIGURE 1. IHED HARDWARE (CLAMP-ON SENSOR, TRANSMITTER, DISPLAY) (SOURCE:

HTTP://WWW.ELECTRICITY-MONITOR.COM/WIRELESS-ENERGY-MONITORS-C-37.HTML)

The Smart Grid is a class of technology used to bring utility electricity delivery

systems into the 21st century, offering many benefits to utilities and consumers --

mostly seen in big improvements in energy efficiency on the electricity grid and in

the energy users’ homes and offices. With the deployment of the Smart Grid and

availability of less expensive communication components, new companies started

developing products that are more sophisticated.

The second generation of IHEDs is part of a network of connected devices creating a

Home Area Network (HAN in Figure 2). Sensors gather information on energy use

from the Utility meter and appliances; and that information is displayed through

multiple interfaces. These systems are usually connected to the Internet and have a

web interface, as well as a physical interface in the house. Some more modern

products also have smart phone/tablet applications. Figure 2 shows how IHEDs are

now part of a more complex network that partially resides inside the wall of the

house but more and more relies on remote services allowed by Internet connection.

These services span from remote climate and lighting control to security and

entertainment.



IHEDs are sometimes integrated with controls (e.g., smart thermostats) to share the

hardware in order to be less expensive. The new network now has all the capabilities to

provide users with comprehensive information and a convenient way of acting on them.

Connection to the web and availability of interval meter data also allows third parties to

interact with the system, offering services for the customer such as demand response

(e.g.,EnerNoc), energy analysis and benchmark (e.g., Opower, Efficiency 2.0 ), fault

detection and performance diagnosis (EcoFactor, NEST). Some companies also plan on

extending service including remote health monitoring. Integration and interoperability are

important challenges for developers.

FIGURE 2. HAN ARCHITECTURE

The main features offered by more recent devices (introduced since 2010) are listed

in Table 1. Frequency is shown to be Common if it appears in almost all devices. It is

shown as Uncommon if it appears in only 25% or fewer devices, and it is shown as

Rare if it is only in a few devices.

TABLE 1: CHARACTERISTICS OF IHEDS ON THE MARKET

FEATURE FREQUENCY

ENERGY MONITORING

-Energy and Cost monitoring Common

ENERGY CONTROL

-Climate Control (smart thermostat) Common

-Lighting control Uncommon

-Smart Windows Rare

-Appliance load control (through smart plugs, load control modules and smart appliances)

Common (smart plugs)

to Rare (appliances)

-Automatic load shift depending on energy price Rare

ENERGY/BEHAVIORAL SERVICES



-General or Tailored Tips on how to save energy Rare

-Energy Budget Rare

-Social Context (comparison with neighbors or friends)

Rare

-Remote Home Auditing Rare

-Continuous commissioning (...) Rare

ENTERTAINMENT

-Entertainment (integration with TV, audio, wireless audio distribution, etc.)

Uncommon

SECURITY

-Security Uncommon

-Remote house monitoring (alarm, cameras, remote door opening, system status, intercom)

Uncommon

HOME HEALTH CARE:

-Remote health care (monitoring, assisted living, emergency aids, medication schedules)

Rare

-Baby monitor and intercom Rare

INTEGRATION W/GENERATION

-Integration with Solar/Wind systems Rare

DEMAND MANAGEMENT (FOR UTILITIES)

-Demand Response (for Utilities) (price signal, load control, text messages)

Uncommon

OTHERS

-Pool/Spa pump control Uncommon

-Irrigation (schedule the irrigation system) Rare

-Electrical Vehicle charging Rare

Devices differ significantly in terms of features offered, choices in interface design

and market strategy. Most of the players actively look for collaboration with Utilities

as the first channel to increase sales. A few others target high end users (home

automation companies) or plan to use their existing sales channels (e.g., security or

thermostat companies).

Despite continuous activity and attraction of capital from different investors, the

market penetration of these devices is still very low. In 2010-2011, several start-ups

were acquired by big players such as Verizon, Motorola and Comcast. Other players

in the Demand Response (demand aggregators) started a process of vertical

integration with HAN manufacturers. At the same time GE, Whirlpool and other

manufacturers are working on a new generation of smart appliances (less expensive

and integrated with the smart grid).

However, the lack of a single standard protocol, the unclear value proposition for the

customers and the high cost of these systems ($300-$1000 according to the survey)

are casting shade on the potential market for these products in the next few years.

The current market (2011-12) is changing fast with big players entering (GE, 3M,

and Cisco) and some leaving it (Google and Microsoft) because the market is so

immature. Figure 3 shows the classification of this market according to a report of

the web magazine GreenTech Media.



FIGURE 3. CLASSIFICATION OF THE HAN MARKET (SOURCE: GREENTECH MEDIA)


Southern California Edison Page 10

Design & Engineering Services December 2012

CHARACTERIZATION OF MARKET IHEDS

The information provided above and in Appendices B and C provides a systematic

overview of the most representative IHEDs and HAN systems available in the

market, and the features they describe in their websites or other promotional

materials and in product reviews. It is difficult to characterize each system

exhaustively, because only specs that each company decides to promote in their site

are available, and the amount of information provided can vary widely across

different companies.

At this point, this is a very fluid market, and there is no standard set of specifications

that companies are required or expected to provide. For example, if a product

indicates that it can provide “current usage information,” this can be either

instantaneous kW, kWh for the last hour, cost for the last hour, cumulative energy so

far today, etc. In addition, it is not always clear whether the company is describing

an existing product’s features, promising future features, or describing features that

are available only with access to other technologies or products.

It is extremely difficult to keep track of all the companies and products (probably

more than a hundred) being developed. The technical specs are listed in a

spreadsheet in Appendix C. The spreadsheet includes the following categories:

Market: Company Headquarters

Sells to Utilities

Sells to Consumers

Sector: Residential (R), Small Commercial (SC), Large Commercial (LC)7

Product and Service Offering

Display

Mob App

Web App

Circuit-Box monitor

Meter Reader

Smart Plug/Strip

Load Control Module

Smart Thermostat

Lighting Control

Security

Entertainment

Healthcare

Demand Response: Yes-unspecified (x), Load Control (LC), Price Signal (PS)

Integration with Generation

Integration with EV

Architecture and communication:

Gateway: proprietary, ADSL gateway




Communication Meter-Device: Zigbee (ZB), Z-Wave (ZW), RDS (RD), Wi-Fi

(WF), Wi-Max (WM), Powerline (PW), Others (O)

Communication Device-Utilities: Radio (R), Web (WB), Power Line (PW), AMI

(AMI),

Communication Device-3rd Parties: Radio (R), Web (WB)

Meter Connection: clamp-on (CO), CT

Interface:

Displays energy: Whole house (W), single appliance (A), circuit breaker

(C)

Displays Real time Energy Information

Displays Current Price

Displays historical data

Displays also: gas (G), water (WT)

Displays other messages: temperature, weather forecast, text

messages

Goal setting

Provides suggestions on how to reduce energy

Social context

Data Storage and Analysis:

Customer Data stored: locally (L), cloud (C)

Partners

The team also developed a summary of the main features available in IHEDs,

grouped by frequency (common features vs. rare features).

TABLE 2. SUMMARY OF COMMON, UNCOMMON AND RARE FEATURES IN MARKET IHEDS

Hardware and Software IHED Feature Summary

Real-Time Time Averaged

(Day) Time Averaged

(month) Projection

(month) Thresholds &

Alerts Display Styles Data Collection

Common $/hr, kW $,kWh $, kWh, CO2 Numeric only

(hardware devices), bar charts

Zigbee, CT-Clamp

Uncommon CO2/hr Day-day

comparison, CO2, social comparison

Social comparisons, comparison to

previous month, comparison to

target

$, kWh Preset color

changes Line graphs

Rare Change in

consumption (delta kW)

$ & CO2 savings, peak power

demand Peak demand

Comparison to target

User-settable color changes, budget based threshold,

user advice

Pie charts, speedometers

HEURISTIC EVALUATION

In order to develop a usability test procedure for IHEDs, the research team

heuristically evaluated a sample of market-available residential IHEDs. Four energy

feedback displays were acquired and examined in detail by the research team,




adapting usability guidelines as developed by Peffer et al., 20128 developed for

programmable thermostat usability. These guidelines include evaluating the device

for

Walkup usability/visibility: User must be able to see most important features

from the initial view of the device.

Feedback: The device offers confirmation of settings or other operations conducted

by the user.

Standards: Terms, icons, and common actions available correspond to standard

information and possible operations that are expected by the user.

Terminology: Terms and abbreviations are comprehensible to the user, or

definitions are easily available.

Broad and shallow decision tree: The device does not present the user with too

many layers (information screens) that make it difficult to find information or

controls.

Hierarchy of display: The device presents important and more frequently used

features more prominently than less used features and information.

Affordances: Affordances of the device are easy to identify by the user (no hidden

navigation buttons, or covers). Navigation options and buttons are clearly

identifiable.

Recovery from errors: The device promptly offers feedback and capability to

correct errors or recover settings.

Researchers decided to focus on IHEDs that were available for purchase and

installations by individual residential consumers, and that focused exclusively on

electrical usage feedback. The four IHEDs purchased correspond to three different

kinds of devices:

1. first generation IHED that display numerical information of energy use at a

whole house level (TED 5000-C and Envi Energy Monitor);

2. Smart power strips that allow for energy monitoring at the plug-level and

present information via an online or mobile interface (UFO Power Center);

and

3. home systems that allow for power strip, thermostat and high voltage plug

monitoring, and present the information through a color and graphics-rich

IHED. The last two kinds also include the possibility of remote controlling of

plug loads.

The 4 IHEDs were acquired and evaluated by the research team.

TED 5000-C Home Electricity Monitor

Envi Energy Monitor

UFO Power Center

Energy Hub Home Base




TED 5000-C Envi Energy

Monitor

UFO Power Center Energy Hub

FIGURE 4. THE FOUR EVALUATED IHEDS : TED5000, ENVI, UFO WITH ONLINE INTERFACE, ENERGYHUB SYSTEM

Research team members installed each of the energy saving devices in their homes

or offices, and activated as many device features as possible. The team members

then accessed the device’s features, while evaluating them according to the criteria

described above (when applicable). Researchers also noted additional usability issue

not covered by the guidelines above, especially regarding installation problems and

interpretations problems.

First, however, the devices were categorized according to the feature offered, as

seen in the table below. (The UFO is described twice: UFO HW refers to the hardware

plug strip that provided some feedback on site, UFO SW refers to the mobile app

software that provided feedback that is more sophisticated).

Below is a characterization of the features available for each of the devices acquired.

(Blank spaces indicate that the feature is not available).

TABLE 3. CHARACTERIZATION OF IHEDS ACQUIRED AND HEURISTICALLY EVALUATED

DEVICE TED HW UFO HW UFO SW ENVI-R HW

Real-Time Cents-per-hour; Watts

Watts (color, high. Med. Low threshold)

Watts per plug & total ;

watts

Time Averaged (Day)

Cumulative Cost; Cumulative kwh

cost; energy cost; energy

Time Averaged (Week)

cost; energy energy

Time Averaged (month)


cost; energy cost

Time Averaged (short-term)

15 minute cost; energy

yesterday

night/day/evening bars

Projection (month) Cumulative Cost;

Cumulative kWh ; Remaining days

Detail (day - kw) current; lowest; highest

Detail (day - $) current; lowest; highest





Detail (day-CO2 lbs/hr)

current; lowest;

highest; Month cumulative

Other Voltage high/low,

time & day of month

time and temperature

Alerts / thresholds no user settable thresholds

notification of

momentarily higher or lower usage

display styles numeric light numeric and graphical (bar charts dial graphs)

numeric and graphical for retrospective data

notes sensitive to WiFi easy setup

granularity house plug plug house + appliances with additional

plug-level devices

controls none none plug-level; scheduling

none

meter interface CT clamps, power

cables & power line communications

none none CT clamps +

wireless broadcast

Real-Time Cents-per-hour; Watts

Watts (color, high. Med. Low threshold)

Watts per plug & total ;

watts

Time Averaged (Day)


cost; energy cost; energy

Time Averaged (Week)

cost; energy energy

Time Averaged (month)


cost; energy cost

Time Averaged (short-term)

15 minute cost; energy

yesterday

night/day/evening bars

Projection (month) Cumulative Cost;

Cumulative kWh ; Remaining days

Detail (day - kw) current; lowest; highest

Detail (day - $) current; lowest; highest

Detail (day-CO2 lbs/hr)

current; lowest;

highest; Month cumulative

Other Voltage high/low,

time & day of month

time and temperature

Alerts / thresholds no user settable notification of





thresholds momentarily higher or lower usage

display styles numeric light numeric and graphical (bar charts dial graphs)

numeric and graphical for retrospective data

notes sensitive to WiFi easy setup

granularity house plug plug house + appliances

with additional plug-level devices

controls none none plug-level; scheduling

none

meter interface CT clamps, power

cables & power line communications

none none CT clamps +

wireless broadcast

After acquiring and installing the devices, researchers became aware of usability

issues unique to each device. Those unique findings are presented here by device,

and common issues are addressed in the discussion section. Several noteworthy

issues affected the acquisition and installation of the devices, and are described here

because they are relevant to the market penetration of IHEDs. Consumers motivated

to purchase IHEDs probably find the same set of barriers, if they engage in the IHED

market as it stands today.

The market penetration of these devices is still very low (fewer than 1% of

homes), due in part to the lack of standard protocols, the unclear value

proposition for the customers and the high cost of these systems.

Devices in the market vary significantly in terms of featured offered, choices

in interface design and market strategy. At this point, this is a very fluid

market.

It is not always clear whether the company is describing an existing product’s

features, or promising future features, or features that are available only with

access to other technologies or products.

Sales of IHEDs online are most commonly targeted to utilities and building

developers. Consumers may invest significant time researching a given

product online before it becomes evident that they may not actually be able

to purchase it for their homes.

For IHEDs that do or do not allow purchase by individual consumers, it may

be hard for consumers to understand what they are purchasing. Research

found that some heavily promoted features were not actually available on the

products purchased, but rather were “upcoming”, or would be available

“soon”. Examples of such features were remote control of plugs, or web

portals for feedback.

Consumers may also find it difficult to know whether the device they are

purchasing works with their system. Product websites were often vague about

Smart Meter connectivity (“works with most SmartMeters”), and failed to




inform consumers that compatible Smart Meters might still not work if

wireless capabilities were not activated by the utilities.

Consumers may be confused about the scope of energy monitoring possible.

IHEDs based on power strips fail to inform consumer that large energy users

are left out of the monitoring (Air conditioners, washers, dryers).

Product websites were also ambiguous about the need for electrician assistant

for installation (“we recommend an electrician assist you, but you can also do

it yourself”). Installation usability should be evaluated if planning involves

user acquiring their own devices.

Most IHEDs are not designed to work with other IHEDs or other devices within

the home. Interoperability is not addressed.

There is no proof that these systems save energy, especially for new players.

Usability, efficacy, energy savings are all currently unstudied.




TECHNICAL APPROACH/TEST METHODOLOGY This study examines, in controlled experimental conditions, users’ reactions and responses

to simulated energy feedback interfaces that are representative of devices available on the

market. The energy feedback interfaces used in the pilot and the full study were abstracted

from the heuristic expert evaluation of representative market devices described earlier in

this report, and correspond to varying levels of complexity. Experiment participants then

respond to questions that gauge their ability to retrieve basic information, put that

information in a meaningful energy use context, and use that information to make energy

use decisions.

As part of the experimental design (described in more detail below), different participants

are randomly assigned to one of eighteen different energy interface conditions. Each

presents a different combination of features. Features presented in the device fall into three

categories:

BASIC INFORMATION Display of basic energy use and cost features, corresponds to the “common” set of

features in our heuristic evaluation. Information provided is almost never absent,

from even the simplest of devices. All experimental conditions include this basic

information, and the condition that includes only basic information is the control

condition in this study. Basic information includes current power usage (W),

cumulative day usage (kWh), pattern of hourly and daily usage (graph with daily

hours on x axis, and cents-dollars and kWh on y axes), and cumulative monthly

usage (graph with days of the month on x axis, and dollars and kWh in y axes).

CONTEXTUALIZING INFORMATION The display’s information allows the user to compare basic information with an

external standard, potentially leading to a motivation to reduce energy use. In this

study, contextualizing information is presented in the form of either a social

comparison with similar households, or a pre-set energy usage goal. Research

indicates that social comparisons, or “anchors”9 (Kahneman, 2011) act as descriptive

social norms that provide a motivation for people to act in accordance with the

norm1011. Goals act as strong motivators for individual action and change, and they

have been shown to be effective motivators in the domain of household energy

behaviors121314.

DECISION MAKING TOOLS The display presents general tips for energy saving that may be used if users are

motivated to reduce energy usage. The display also includes diagnostic tips to alert

the user that a hypothetical appliance is wasting energy because of some form of

malfunction.

This controlled randomized trial test experiment is designed to determine whether

and/or how much each layer of information (Basic, Contextual and Diagnostic)

increases the individual’s perception of the need to reduce energy use or the

magnitude of any actions they feel are appropriate.




TEST PLAN The experiment was a randomized controlled trial. The test varied the display

characteristics of a simulated energy feedback interface, including available basic

information, set budgets, and social information. Participants were randomly

assigned to either a device with basic information alone (control), a set budget, or

social comparison information (usage is higher or lower than neighbors). In a second

part of the experiment, subjects were randomly assigned into three groups: those

shown no additional information, those shown general tips for energy saving, and

those shown specific diagnostic information for energy saving.

Using the energy feedback interface they are assigned to, participants are asked to

complete several rounds of information retrieval tasks (finding and reporting

information provided in the feedback interface) and decision making tasks (choosing

appropriate actions to reduce energy usage depending on the feedback). After

completing three rounds of tasks, participants are asked questions about their

experience with the interface and some demographic questions.

Participants in the study were recruited in three different locations in the cities of

Davis and Fairfield, in California. One location was a campus cafeteria, one was a

grocery store and one was a large mall. The study was conducted at each of these

locations using a stand or table in a public high-traffic spot. Passers-by were asked

to participate in exchange for a five dollar gift card. Researchers stayed at a table

with six small tablet computers that were programmed to display the mockups as

well as an integrated questionnaire. Participants used these tablets to access the

energy feedback displays and responded to the questions and tasks required by the

study.

Participants were randomly assigned to one of six energy feedback display formats,

and to one of three energy reduction information conditions (see design summary in

Table 3 below). Devices and questions were presented using MediaLabTM

experimental software that allows display of the information, automatic recording of

the questions, and tracking of device navigation (hyperlinks followed and time spent

at each screen).

After reading the consent form (see Appendix H) and accepting to participate in the

study, participants read the first set of instructions that asked them to familiarize

themselves with the energy feedback display. The feedback display then appeared on

the screen, and participants had some time to explore the display by clicking through

the device’s screens and seeing what information was provided, and what were the

different feedback navigation paths. After the participant had explored the display,

he/she was directed to the first round of questions. The full questionnaire is available

in Appendix F.




TABLE 4. FULL EXPERIMENTAL DESIGN SCHEMATIC TABLE.

FIRST PART: KIND OF ENERGY FEEDBACK PROVIDED

BASIC

ONLY

BASIC

+

BUDGE

T

BASIC

+SOCIAL

INFORMATION

HIGHER THAN

NEIGHBOR

BASIC

+SOCIAL

INFORMATION

LOWER THAN

NEIGHBOR

BASIC

+BUDGET +

SOCIAL

+INFORMATION

HIGHER THAN

NEIGHBOR

BASIC

+BUDGET

SOCIAL

+INFORMATION

LOWER THAN

NEIGHBOR

SECOND

PART:

Energy Saving Suggestions Provided

No Energy Saving Suggestions

13 14 15 17 16 18

General Tips 2 4 6 10 8 12

Specific

Diagnostics

1 3 5 9 7 11

The numbers in the cells represent the 18 different conditions that participants were

randomly assigned to (See Appendix G).

Participants responded to three types of questions: Informational, Meaning/Context

and Actions/Decisions.

INFORMATION

Questions such as “according to the display, how much energy has the household

spent so far this month?” or “what is the projected electricity bill for the household?”

These questions are intended to gauge whether users can identify the kind of

information provided in the display and how hard or easy it is for them to access it.

Right and wrong answers to the tasks allowed the researchers to determine whether

users can obtain accurate information from the device. Ease of use is gauged by

measuring how quickly respondents are able to answer the questions, and how many

navigation screens they had to go through to reach the answer.

MEANING/CONTEXT

Questions such as, “is the energy use of the house shown in the previous screen low,

average, or high?” or, “does the energy use of the house suggest to you that energy

use should be reduced?” These questions are intended to gauge whether the

information provided in the feedback interface can be put into a meaningful context

by the participants.

ACTIONS/DECISIONS

Participants were provided with a variety of energy saving options with an increasing

cost or effort (the options were “change your thermostat setpoint”, “call a

professional to maintain your air conditioner”, and “replace your air conditioner”),

and asked which they felt was appropriate given the feedback. These questions are

intended to gauge whether information in the device can be used to inform and

motivate energy saving decisions.

After responding to this first round of questions, participants were allowed to access

the feedback interface again. The interface now included extra information on energy

reduction in either tip or diagnostic format. After seeing this info, they answered a




second round of questions on information, meaning and action. Finally participants

answered questions (five-point scale from strongly disagree to strongly-agree) rating

their experience with the experimental device (was it fun, frustrating, easy, hard,

boring, interesting, etc.) as well as demographic questions.

MOCKUP DESIGNS

As described in the design summary, participants were randomly assigned to view

different sets of IHED screens. The mockup screens were created based on common

ways of presenting features in market IHEDs. Figure 5 below presents two possible

sets of screens that participants were able to see. The first set corresponds to the

control condition, in that participants saw only basic information. That was the

simplest set of screens available. The second set of screens corresponds to the basic

+ budget + social information higher than neighbor usage + general tips condition.

This set of screens is an example of the most complex possible IHEDs. Simpler

versions would have presented less features, in the same kind of visual organization.

A full set of screens for all 18 conditions of the study is provided in Appendix G.

A description of each category (basic, budget, social information) is provided below,

with color references that can be used to look for the features in the sample screens.

That said, use of color was not one of the variables of interest in the study. The

colors chosen to present information (purple, orange and gray) were the ones

considered to be least likely to convey an evaluative judgment of consumption. In

general, cultural norms are that green indicates positive and environmentally friendly

behaviors, whereas red represents behaviors that are dangerous or must be stopped.

The use of green was avoided, and red only used in alerts regarding budget

information. Colors were uniform across all conditions, and, again, were not tested in

this study.

The basic information included numerical information (see large white numbers)

describing current power use, and energy usage for a day. It also included a graph

with “yesterday’s hourly usage” (gray color) and “today’s hourly usage” (orange

color). By clicking on “more info” the user can access a second screen, showing daily

usage for the month, and cumulative usage for the month (in orange). Basic

information corresponds to common features found in most of the IHEDs available in

the market.

Budget information included all of the above, plus a red alert on the top left of the

screen, alerting user to being “over budget” by 15%. It also included a red projected

line, in the bottom half graph of the second screen that represented the budget.

Social information presented included purple color graph information on the second

screen, representing neighbors’ usage. The purple overlay indicated similar

neighbors’ usage, whereas the orange indicates self usage. Social information was

presented in two ways: consumption higher than that of neighbors (as in the screen

sample below), or consumption lower than that of neighbors.

Finally, the “take action” screen included three messages recommending either

general tips for action, or suggestions based on specific measurements for each

users’ system (example shows general tips). The three “general tips” were

Replace your air conditioner! Older, less efficient air conditioners can use as

much as 30% more energy.

Call a professional to maintain your air conditioner! Not providing regular

maintenance for your air conditioner wastes 30% of its energy use




Change your thermostat setpoint! For each degree the thermostat is set

below 75°, 3% to 5% more energy is used.

The three “diagnostic tips” were:

Replace your air conditioner! You air conditioner is 17 years old, and uses

30% more energy than a new Energy Star model. According to our

measurements, important components of your air conditioner are worn or

near breakage, and it may only last two more years.

Call a professional to maintain your air conditioner! According to our

measurements, your air conditioner is wasting 30% of its energy use because

of airflow restrictions produced by lack of regular maintenance.

Change your thermostat setpoint! According to our measurements, your

thermostat is currently set at 68°. 21-35% more energy is being used than if

you changed it to 75°.

FIGURE 5. SAMPLE MOCKUP DISPLAY WITH THREE NAVIGATION SCREENS. THE FIRST TWO SCREENS

WERE AVAILABLE FOR NAVIGATION WHEN PARTICIPANTS WERE FIRST PRESENTED WITH

THE STUDY. THE THIRD SCREEN WAS AVAILABLE FOR THE SECOND PART OF THE STUDY, AND

APPEARED AS A “POP-UP” ON THE DISPLAY.




RESULTS Participants were recruited at three field sites in Davis and Fairfield, CA. After removing

ineligible participants, 249 responses were saved to a final data table. The respondents

were 50% female, and typically college age as shown in the demographic figures.

FIGURE 6. DEMOGRAPHICS: AGE DISTRIBUTION OF PARTICIPANTS. (N=249)




FIGURE 7. DEMOGRAPHICS: EDUCATION LEVEL OF USABILITY TESTING CONTROLLED EXPERIMENT

PARTICIPANTS. (N=249)

The experiment included four broad sections addressed in subsections below:

An initial evaluation of the devices;

A section to test the participants’ abilities to access important numerical information

from the device;

The effect of social comparison information;

An assessment of intent to take action;

A FINAL EVALUATION OF THE DEVICES INITIAL EVALUATION

The initial evaluation was performed after allowing each subject to be familiarized

with a mockup and explore its features. This evaluation is therefore a “first look”

evaluation of an IHED, similar to the type of response one might have while browsing

similar products in a home-improvement store. The evaluation is completed using a

series of scale (1-6) responses to statements, where a 1 represented “strongly

disagree”, 2 represented “moderately disagree”, 3 represented “slightly disagree”, 4

represented “slightly agree”, 5 represented “moderately agree” and 6 represented

“strongly agree. ”

As shown in the figure, the subjects generally thought that the mockup designs were

easy and even fun to use, they would like to have the information at home, and

would find it useful. They did not find the screens frustrating or too simple. There

were also mixed responses on their plans to purchase a similar device, although 60%

of responses indicated that they were positively disposed towards purchasing such a

device. When asked about how much such a device might be worth, 60% of subjects

felt that it was worth more than $50, as shown in the figure below. Only 22% of




respondents felt that the display was worth over $100 probably the lowest possible

cost for a mass produced product. These results indicate that only about 20% of

demographically similar people might purchase an unsubsidized energy display in the

future.

FIGURE 8. EVALUATIONS OF THE MOCKUP IHED INTERFACE ALONG A SERIES OF HEDONIC AND UTILITY

DIMENSIONS.

FIGURE 9. FREQUENCY OF RESPONDENTS’ SUBJECTIVE VALUATION OF IHEDS, IN US DOLLAR RANGES.

(N =249)




INFORMATION ACCESSIBILITY

One of the primary functions of an IHED is to provide residents with pertinent energy

usage and cost data. The ability of individuals to extract simple information about

their home energy use and spending on energy is therefore a crucial test of IHED

usability. To test subject’s ability to extract information we constructed four multiple

choice questions. The subject was shown the display and then asked a question

about the energy usage shown on the display. The subject was free to press the

“back” button to revisit the energy display as many times as they wished to find the

information before answering the multiple choice question. To analyze the data the

answers were coded as Correct, Incorrect, and Unsure. The responses are shown in

the pie chart matrix below.

Subjects were much better at determining hourly cost and power than energy used

or projected cost. The relatively poor performance on the quiz indicates the

importance both of clear IHED design, and improved energy education of IHED users

to ensure that they can easily extract pertinent information from their IHED.

Participants were the most successful with the question where the answer can be

found as a numerical cypher in the screen (power = 1.1 kW). Participants also did

well at one question involving a direct reading of a graph’s data point (hourly cost =

16 cents), but failed at a question that required the same skill (energy use =

500kWh). The difference between these two questions, other than the unit of

measurement (cents vs. kWh), is that the one that subjects performed poorly on,

required an extra navigation screen to get to. It is possible that participants fail to

navigate to the screen with the correct energy information.

The other question with poor performance required projecting an imaginary trend

from a graph (projected cost this month = $116). This result is an indication that

simplicity in information provided is significantly clearer to consumers than

aesthetically pleasing colorful graphics. Participants may have particular difficulty

projecting trends from graphs and navigating screens to reach all information

necessary.

Referring back to the IHED mockups, the questions and correct answers were:

TABLE 1. MULTIPLE CHOICE QUESTIONS GAUGING ABILITY OF PARTICIPANTS TO EXTRACT CORRECT INFORMATION FROM

IHEDS.

QUESTION WORDING CORRECT ANSWER

Power How much power is the household using right now?

1.1 kW

Energy Approximately, how much energy

has been used so far this month in kWh?

500kWh




FIGURE 10. PERCENTAGES OF CORRECT, INCORRECT, AND “UNSURE” RESPONSES TO 4 MULTIPLE CHOICE

QUESTIONS TESTING ABILITY OF RESPONDENTS TO EXTRACT INFORMATION FROM IHED

INTERFACE.

FIGURE 11. PARTICIPANTS DID BETTER AT RESPONDING TO QUESTIONS IN SCREEN 1 ABOUT COST

(1.1KW; TOP LEFT CORNER SCREEN 1) AND COST (16 CENTS; DATA POINT IN SCREEN 1), THAN TO QUESTIONS IN SCREEN 2 REQUIRING RESPONDING TO A DATA POINT (500KW;

BOTTOM SCREEN 2) AND PROJECTING A TREND (116 DOLLARS, BOTTOM SCREEN 2).




Budgetary information was hypothesized to help individuals make the monthly

projection estimates with more ease. The budget line is shown in the display, for the

budget condition. However, as shown below, although the budget did help subjects

answer the monthly cost question, more individuals were unsure of their answer, and

importantly, the added complexity of the screen led to a reduced percentage of

correct responses for the prior question of the energy used to-date this month. Low

rate of correct responses may be due to difficulties interpreting projections in graphs,

increased confusion as more information is presented in a screen, and difficulties in

reaching the information in navigation, since it was presented in the second screen

FIGURE 12. COMPARISON OF RESPONSES TO 2 INFORMATION QUESTIONS, COMPARING PARTICIPANTS

WHO SAW DISPLAYS WITH VS. WITHOUT BUDGETARY INFORMATION. BUDGETARY

INFORMATION INCLUDED A COST PROJECTION TREND DRAWN IN RED IN THE GRAPH, BUT IT

WAS A MORE COMPLEX SCREEN. QUESTIONS ARE “HOURLY COST” AND “PROJECTED COST”

IN TABLE 4 ABOVE.

THE EFFECT OF SOCIAL INFORMATION

Social information plays a potentially important role in IHED design. Neighbors (or

similar households) can provide contextualizing information that is otherwise

unavailable from a household’s energy data on its own. For example, although

individual-level data can answer the question: “Has my usage gone up over time?” it

cannot answer the question, “is my usage higher than average, or particularly high?”

This social information can play a role both in making individuals aware of the

potential to reduce their energy use, as well as motivating reductions due to a social

normative effect.

Currently some existing services for utilities, are already offering services that

contrast energy usage with sets of similar households near a customer.




To test the importance of social information three social variants were tested in the

experiment. The three variants were no social information, an overlay showing

neighbors that are 15% higher than the home, and an overlay showing neighbors

that are 15% lower than the home (see Appendix G for screens). A series of

questions in the survey then probed the subject’s interpretation of the presented

home energy use to determine if the usage of neighbors had an effect.

FIGURE 13. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON COMPARATIVE ENERGY USAGE

(“THE HOUSEHOLD ‘S ENERGY USE SEEMS TO BE”….), ACROSS THREE CONDITIONS OF

SOCIAL INFORMATION AVAILABLE.

The first question tested the ability of subjects to correctly interpret the screen, and

shows a clear impact of the additional information for most subjects. Neighbors using

higher energy levels resulted in the median subjects indicating that this was the

case, whereas lower neighbors resulted in a shift in responses. No neighbors shown

resulted in many subjects indicating that they believe the usage was either similar or

they cannot tell. In general, participants correctly understood the comparative

information provided when they saw screens where the neighbor was using more

energy, most said they were using less than neighbor, and vice versa.

Once we establish that participants correctly understood the information, we asked a

second question to test whether they were interpreting neighbors’ energy use as a

norm to follow. The second question, then, probed the normative value of the

neighbor information by asking if the household was using more or less energy than

it “should be.” This distinction indicates that the peer effect can play a role in




motivating reductions: if my neighbor is using less, and I take that as a norm to

follow, I might use less too. Again, subjects responded as hypothesized to the

information, and notably responded most strongly when their house was shown as

using more than their neighbors.

FIGURE 14. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON NORMATIVE ENERGY USAGE,

ACROSS THREE CONDITIONS OF SOCIAL INFORMATION AVAILABLE.




FIGURE 15. COMPARISON OF RESPONSE PATTERNS TO THE QUESTION ON ABSOLUTE JUDGMENT ON

ENERGY USAGE, ACROSS THREE CONDITIONS OF SOCIAL INFORMATION AVAILABLE.

The final question in this set asked for an overall evaluation of energy use in the

presented household. The results indicate that when the IHED showed the house as

using more energy than the neighbors the subjects felt that the usage was high, but

the reverse was not true when the house was shown using less than the neighbors –

in this case the subjects felt that the usage was average (not low).

TAKING ACTION

One purpose of an IHED is to motivate resident action in saving energy, by either

taking conservation measures or purchasing or maintaining their equipment. To test

the efficacy of the IHED mockup designs in motivating this type of behavior a series

of hypothetical questions probed each subject’s belief that it would be appropriate to

take action to save energy due to the presented information, were it shown in their

own home.

In addition, subjects were split into two groups to receive either general energy

saving tips or specific energy related diagnostic information. The tips and diagnostic

groups are additionally compared to test the hypothesis that diagnostic information

is more relevant to residents than generic energy saving tips. In these questions,

subjects were allowed to choose as many energy saving options as they wanted, to

better reflect reality.




FIGURE 16. T-TEST P <.001 FROM NO EXTRA INFO TO TIPS OR DIAGNOSTICS. NO SIGNIFICANT

DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS


DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS.





DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS.


DIFFERENCE FOUND BETWEEN TIPS AND DIAGNOSTICS




The results of the action questions show the importance of tips and diagnostic

information in converting the meaning of energy information (high vs. low) into

intent to actually change a household behavior or improve energy using equipment.

In A) subjects with diagnostics were the most certain that the house is using more

energy than it should. In B) those with diagnostics felt that they knew what actions

to take to save energy. In C) those with diagnostics were more likely to choose more

effortful and costlier conservation measures, and in D) those with diagnostics felt

that ideally (without some of the constraints of their real situation) they would be

even more aggressive with conservation measures.

The trend in each question indicates that tips were more motivating than no

information and diagnostics were more motivating still. The distributions were tested

using an unpaired Welch Sample t-test that tests whether the mean value of the

distributions is different. To test the groups unsure answers were removed, and

multiple-selection answers were added up to form a score for each individual. For

each question, each information type was tested against no additional information

and also against each other. The results were very consistent: the addition of either

Tips or Diagnostic information led to a statistically significant increase in the mean

score for the question, indicating that they have a real effect on knowledge and

motivation. The difference between Tips and Diagnostics was apparent in the mean

values but not statistically significant.

It is worth noting that there are currently no residential IHEDs that display diagnostic

information. This section of the study tested a hypothetical possibility that the

researchers thought might potentially encourage energy saving actions. It is quite

interesting to see that both kinds of diagnostic information can significantly increase

motivation to engage in energy savings.




DISCUSSION To evaluate the effectiveness of the individual measures as well as the performance of the

IHEDs overall, two measures were taken. The first measure is from subject evaluations of

the information they were presented with in terms of whether they liked the information,

and if they found it useful. The questions and answers are presented below.

TABLE 2. COMPLETE WORDING OF QUESTIONS EVALUATING USERS’ LIKING AND PERCEIVED USEFULNESS OF IHED

FEATURES

QUESTION NAME WORDING

kWH_EVAL I liked that I could see the electricity usage expressed in kWh.

kWh_USEFUL I think that seeing the electricity usage expressed in kWh was useful.

kW_EVAL I liked that I could see the power being used expressed in kW.

kW_USEFUL I think that seeing the power being used, expressed in kW, was useful.

DOLLARS_EVAL I liked seeing the electricity consumption expressed in dollars and cents.

DOLLARS_USEFUL I think that seeing the electricity consumption being expressed in dollars and cents was useful.

BUDGET_EVAL If I had a display at home, I would like it to mark a budget goal that I can see.

BUDGET_USEFUL If I had a display at home, I think a budget marker would be useful.

COMPARISON_EVAL If I had a display at home, I would like it to show my neighbors' energy information.

COMPARISON_USEFUL If I had a display at home, I think seeing my neighbors' energy information would be useful.

FIGURE 20. RESPONDENTS’ LIKING AND USEFULNESS RATINGS OF A SERIES OF IHED MOCKUP

FEATURES.




The experimental subjects on average both liked and found all of the information useful. In

summary, 80% had positive views of energy and power information. Over 90% of subjects

found financial (costs and budgeting) good (and more than half of those strongly agreed

with the statements), whereas only 60% found the social comparisons to be liked or useful

information types.

To test the projected usability of similar IHEDs, a combined usability score was generated

for each screen type. The usability score follows the basic concept that the most usable

IHED is easy to use as well as provides the required knowledge to transform the information

into action, termed “actionable knowledge”, and the motivation to act. In the case of our

study “actionable knowledge” corresponds to the question asking people whether they

would know what to do if they wanted to save energy, and “actionable norms” corresponds

to scores on the question asking whether they think, based on the IHED, that they should

be saving energy.

The score is then based on the formula:

Translated into the measures recorded in the survey this becomes:

The average usability scores are shown below organized by display.

FIGURE 21. USABILITY SCORES COMPARED ACROSS KIND OF ACTIONABLE INFORMATION PROVIDED

(NONE, GENERAL TIPS, SPECIFIC DIAGNOSTICS), AND KIND OF SOCIAL COMPARISON

PROVIDED (USAGE HIGHER THAN NEIGHBORS, USAGE LOWER THAN NEIGHBORS, NO

SOCIAL INFORMATION) .

The usability score shows two main trends- an increase in usability with additional tips and

diagnostics (as shown on the x-axis), and an increase in usability with additional social

information (shown in the side-by-side bars). The chart shows that the two information

)__(Re_

__ NormsActionableKnowledgeActionable

quiredEffort

GainednInformatioScoreUsability

)__(_

__ NormsActionableKnowledgeActionable

SpentTime

ScoreQuizScoreUsability




types also interact to improve usability – with the highest scores having both social

information and diagnostics. When the house was shown to have a higher energy use than

its neighbors it was always more usable than the opposite – most likely due to the

normative effect. The case with no neighbors was much more usable in the diagnostic

scenario. This is likely because the diagnostic information largely plays the same role as

social information in motivating improvements, but it was also easier to complete the quiz

when there was less social information on the screen. Finally, the highest usability score

(.34) is more than a 50% increase over the lowest score (.22), suggesting that IHED design

can have a large impact on resident actions.

This experiment provided valuable insight into the cognitive and sociological processes that

take place between the data underlying IHED design and a resident’s intention to take

action to reduce their energy use. The summary findings are:

80% of subjects enjoyed trying out the IHED, and would like to have one at home.

However, only 20% were seriously interested in buying one, and only 22% thought

that the device was worth at least $100.

70% of subjects can correctly answer questions about the instantaneous power draw

based on a numerical value, but only 30% can correctly answer questions about

monthly data using a chart. This indicates that both clarity of design and user

education will play an important role in the success of IHEDs.

Social (similar neighbor) information was successful in creating a performance norm

that subjects used to evaluate the house. However, in an evaluation of social

information, subjects had mixed responses to it, and 40% reported that they did not

like it or find it useful.

Direct evaluations of many common energy metrics showed that subject preferred

financial metrics (current cost and budgetary information) by a wide margin over

energy data or social information.

An ad-hoc usability measure indicated that some display styles are much more

effective than others are. In general, the added complexity of diagnostic information

and social information was worth the additional effort required to extract the

information. The usability measure also showed that the addition of diagnostic

information makes social comparisons less important to the overall usability.

Social information showing that the household was using less energy than the

neighbors were, consistently caused subjects to reduce their motivation to take

action, and led to a decrease in usability.




CONCLUSIONS The market survey found the following problems with usability of the IHED:

Penetration of these devices is still very low (fewer than 1% of households possess

an IHED), in all likelihood because of the fluid state of the market itself, resulting in

confusing information for consumers.

There is no evidence that using the information in an IHED can help save energy,

and consumers may find it hard to know how to use an IHED to save energy, and

how to verify that the IHED feedback has resulted in savings for them.

Consumers may be confused about the scope of energy monitoring possible, since

promotion of the devices is vague regarding availability, installation compatibility and

features.

It may be hard for consumers to purchase or to understand what they are

purchasing.

Because of technical installation issues, and issues of compatibility with utilities’

meters, other wireless home systems, and price, it is likely that, in the long term,

the IHED market will drift towards mass installation on the part of building

developers or the utilities, rather than individual purchase on the part of consumers.

Respondents in the usability study were more likely to interpret the information

incorrectly when it was presented in graphic form. This should be a warning flag,

since many of the new generation IHEDs are graphics and color rich, and rely on

timeline graphs to present information. More research is needed to determine

whether these graphs are actually usable and useful, besides making the device

“look cool” or “fun”.

The controlled usability experiment sheds some light on features that are more likely to

result in appropriate use of these devices. That said, because of the high potential for

confusion and failure, the researchers stress the need for further research in these areas

before choices on devices are made, or mass deployments are considered.

The usability controlled experiment found that both simplicity of design and user

education play an important role in the success of IHEDs.

Social (similar neighbor) information was successful in allowing residents to

understand the meaning of their energy usage numbers. However, subjects had

mixed responses to it, and a significant fraction reported that they did not like it or

find it useful. Comparisons can also be a double edged sword: when the household

was using less energy than the neighbors, subjects’ motivation to take energy saving

actions was decreased.

The study also found that general tips are more motivating than no information and

specific diagnostics are more motivating still.

It may be worth it to invest time and resources in developing and testing energy

feedback technologies that provide specific energy saving diagnostics to residents.

Diagnostic algorithms exist, although they have never been applied to residential

systems.

The specific diagnostics that would need to be implemented probably includes

detection of when things are left on too long (including the air conditioner) and a loss

of efficiency of the air conditioner and furnace over time. This would require

additional infrastructure (communications with the thermostat, for example) that




currently does not exist. Although these algorithms do not exist, this study makes

the case that they should be developed.

Overall, the study indicates that IHED development, marketability and effectiveness are at

an immature state. The information provided by some systems on the market is interesting

and complex, but users may have a hard time understanding what the goal of the IHED is,

what its information means, and how they are supposed to use it. If the market is to

coalesce into a feedback system that is useful for energy savings, we may need to know

more about how to educate communities and consumers about IHED capabilities and goals,

and more about how to design a device that provides consumers with simple and direct

information that they can act upon.




RECOMMENDATIONS

Gadget vs. Tool Framing: Make sure that IHEDs or other energy consumption

feedback devices are provided in the context of a planned energy savings action

program such as providing goals or other motivations to reduce energy use.

Conduct additional research on the impact of general energy saving tips and

diagnostic information and messages that are provided directly in the IHED while

people are checking their energy usage. We found that they can be more effective

than simply providing energy use, although there are currently no systems that

provide diagnostic messages

Engage manufacturers to develop and incorporate diagnostics algorithms into their

devices since this was found to be quite beneficial, but it is not possible with today’s

technology.

Expand research on the effects and desirability of social comparison information,

paying particular attention to the possibility of “boomerang” effects where a

comparison decreases motivation to save energy.

Expand research on the impact of complex, graphic and information rich IHED

devices, focusing in particular on users’ ability to extract correct and actionable

information from it.

Evaluate simple, clear messages as part of the IHED information. Results suggest

that these may be more useful and satisfying to users.

IHED Usability scores, like the one proposed here, should combine traditional

usability measure (ease of navigation, ability to gather correct information) with an

effectiveness usability measure, gauging whether the IHED provides information that

facilitates energy saving actions.

Conduct further research into deployability and usability of particular models prior to

including them in any incentive programs.




APPENDIX A

RESEARCH TEAM

Kristin Heinemeier Ph.D., P.E.

Dr. Heinemeier has over 20 years of experience in building operation-phase efficiency issues. Previously, she worked with Portland Energy Conservation, Inc., a non-profit firm that is an established leader in the area of building commissioning, was the technical director of the Brooks Energy and Sustainability Laboratory at Texas A&M University, and developed products for Honeywell Home and Building Controls. Prior to

that, she was a researcher at Lawrence Berkeley National Laboratory. She was formerly the administrator of the California Commissioning Collaborative, and the

chair of ASHRAE’s technical committee on Building Commissioning. Kristin has a Ph.D. in Building Science, and is a Licensed Mechanical Engineer.

Tai Stillwater, Ph.D.

Dr. Stillwater is a Postdoctoral Scholar at the UC Davis Institute of Transportation Studies. He works in the Behavior and Sustainability Feedback Systems group that he helped develop as a PhD student in Transportation Technology and Policy. He has been an Emerging Venture Analyst at the Energy Efficiency Center and worked at the Plug-in Hybrid Demonstration Project, focusing on vehicle interfaces and energy use.

His dissertation examined the impact of novel vehicle interfaces on driving behavior

and fuel economy. Tai was the 2007-8 CH2M Hill Fellow in transportation and received the 2007-8 Achievement Rewards for College Scientists Scholarship. He holds an M.S. in Transportation and Technology Policy from UC Davis and a B.S. in Mechanical Engineering from UC Berkeley. At Berkeley, he worked on composite parts for the ATLAS particle detector at LBNL, and he participated in the Cal Human Powered

Vehicle Team, breaking multiple world speed records with a carbon fiber vehicle he helped design and manufacture called the "Bearacuda", that was accepted into the U.S. Bicycle Hall Of Fame. Tai has also worked at the California Energy Commission and with Chevron Energy Solutions, where he organized the 2008 West Village Contributor’s Forum.




Claudia Barriga Ph.D.

Dr. Barriga is the WCEC's Behavioral Research Associate. In collaboration with Kristin Heinemeier, she is in charge of coordinating the center's new Behavioral Research Initiative. She is responsible for social science research that looks at people's

motivations, goals, and values as they impact their behavior and choices on energy consumption and use of HVAC systems. Claudia is currently leading a project to look at the behavior of homeowners and small-business owners on HVAC maintenance decision-making processes. She also collaborates in research that examines the usability of IHEDs and the impact of attitudes and goals on residential energy use and response to energy use feedback. Claudia holds a Ph.D. and an M.S. in Communication from Cornell University, and a B.S. in Psychology from Universidad

Diego Portales, Chile. While at Cornell University she collaborated in research on barriers that prevent people from practicing well-known food safe behaviors, and she conducted research on the mental processing of moral, gender and science information in narrative form.

Marco Pritoni M.I.E.

Marco Pritoni is a Ph.D. student in Mechanical and Aeronautical Engineering at UC

Davis. Prior to joining the program, Marco received his Masters’ Degree in Industrial Engineering from the University of Bologna, Italy and after working in industry for 6 years joined the LBNL as a senior research associate for a year. He is currently a graduate student researcher for UC Davis Western Cooling Efficiency Center and an Emerging Ventures Analyst for the UC Davis Energy Efficiency Center. He is a Southern California Edison Fellow and Johnson Controls Fellow.

Dina Biscotti Ph.D.

Dr. Biscotti is a Postdoctoral Scholar at the University of California, Davis Energy

Efficiency Center. She completed her Ph.D. in Sociology at UC Davis in 2010 and her research specialization is in economic and organizational sociology. In collaboration with EEC Director Nicole Biggart, Dina is leading a research project to study the diffusion of energy efficiency technologies through social institutions like churches, schools, and community-based organizations. With the UC Davis California Lighting Technology Center and the UC Davis Western Cooling Efficiency Center, she is working

on several interrelated research projects to identify structural barriers to the adoption of energy efficiency technologies and practices, along with strategies for overcoming those barriers.




APPENDIX B Appendix B, the full list of market available IHEDs, is attached separately as a pdf

document.

Appendix B.pdf




APPENDIX C Appendix C, the characterization spreadsheet of market IHEDs, is attached separately as an

excel document.

Appendix C.xlsx




APPENDIX D

USABILITY CONTROLLED EXPERIMENT

QUESTIONNAIRE ITEMS

Variable names are presented in brackets

1. Previous Familiarity with Energy Consumption

[KNOW_MONEY] I spend approximately $ _______ per month on energy (average).

[KNOW_kWh] I use approximately _____kWh monthly. 2. IHED mockup display evaluation. Respondents state their agreement with a series of statements

using a 6 point scale, where

1 = Strongly Disagree

2 = Moderately Disagree

3 = Slightly Disagree

4 = Slightly Agree

5 = Moderately Agree

6 = Strongly Agree

[EASY] The energy display seemed easy to use and understand.

[STRAIGHTFWD]The energy display was straightforward to use.

[LIKETOHAVE]I would like to have an energy display like this at home.

[PLANTOBUY]I plan to buy such an energy display eventually.

[LIKE]I like this energy display.

[FUN]The energy display was fun to use.

[ENTERTAIN]The energy display was entertaining.

[COOLTOHAVE]It would be cool to have an energy display like this one at home.

[INTERESTINGHOME]The energy display’s information would be interesting to have at home.

[SAVEENERGY] I think I could save energy if I had this energy display.

[FRUSTTECH] I find new technologies like this display frustrating.

[LOOKGOOD] The energy display looked nice

[TOOSIMPLE] The energy display was too simple

[USEFUL] I think this energy display could be very useful.

3. [WORTH] To me, this device is worth ___________ 4. Respondents were then instructed that they could consult the display to respond to the next set

of questions. The following questions were presented, and respondents were allowed to go back and look at the display in order to choose their responses. These items were multiple choice.

[POWER] How much power is the household using right now? o 1.1kW 16kWh 26kWh 16 cents $80 DK

[ENERGY] Approximately, how much energy has been used so far this month, in kWh?

o 4 26 500 800 1.1 DK




[HOUR_COST] How much has this household spent in the last hour? o 1 dollar 16 cents 26 dollars 3 cents 11 dollars DK

[PROJ_COST] If the household continues using energy at the current rate, how much will they spend this month?

o 16 25 80 500 116 DK

[ENERGY_Overall_Meaning] Overall it appears that this household’s energy use is: o Extremely Low o Low o Average o High o Extremely High o I cannot tell

[ENERGY_Compared_Meaning] The household’s energy use seems to be: o Much lower than most homes o Lower than most homes o Similar to most homes o Higher than most homes o Much higher than most homes o I cannot tell

[ENERGY_Normative_Meaning] The household appears to be using: o Much less energy than it should o Less energy than it should o As energy as it should o More energy than it should o Much more energy than it should o I don’t know




[ACTIONABLE_Norm] If you saw this information for your household, you would think:

o You need to use much less energy o You need to use slightly less energy o You are using an appropriate amount of energy o You could afford to use a little more energy o You could afford to use a lot more energy o Would not know how to decide

[ACTIONABLE_Possibilities] Which of the following actions do you think would be more useful in getting this household’s usage to an appropriate level?

o A small easy action, like turning off electronics when not using them o A moderate, no cost action, like cleaning your refrigerator or AC coils. o A moderate, low cost action, like changing incandescent light bulbs for

efficient ones o A moderate, costly action, like hiring a maintenance contractor to optimize

your air conditioner o A big, costly action, like replacing your appliances with new Energy Star

efficient ones o No change is needed

[ACTIONABLE_Knowledge] If you saw this energy display, and you WANTED to SAVE ENERGY, you would know exactly what to do to save energy

o Strongly Disagree o Moderately Disagree o Slightly Disagree o Slightly Agree o Moderately Agree o Strongly Agree

5. After participants were introduced to the second phase display, that included either general tips or specific diagnostics, participants were asked the following questions.

[ACTIONABLE_ NORM_ AFTERTIPDIAG] The new information in the display makes me think that the household is using:

o Much less energy than it should o Less energy than it should o As energy as it should o More energy than it should o Much more energy than it should o I don’t know




[ACTIONABLE_KNOWLEDGE_AFTERTIPDIAG] If you saw the device with this new information and you WANTED to SAVE energy, you would know exactly what to do to save energy

o Strongly Disagree o Moderately Disagree o Slightly Disagree o Slightly Agree o Moderately Agree o Strongly Agree

[ACTIONABLE_OPTIONS_REALISTIC] Please think of what you realistically might do: which of these options would you choose to implement if you saw the same information on your IHED at home?

o No action o Raise my thermostat 2-4 degrees o Raise my thermostat 5-7 degrees o Clean my AC o Hire someone to maintain my AC o Replace my AC

[ACTIONABLE_OPTIONS_IDEAL] Now please think of what your “ideal/best” self would do: which of the options provided would you choose to implement if you saw the same information on your IHED at home?

o No action o Raise my thermostat 2-4 degrees o Raise my thermostat 5-7 degrees o Clean my AC o Hire someone to maintain my AC o Replace my AC

6. Participants were then asked a set of questions about the usefulness and desirability of features in the IHED mockup. (Respondents used the 1-6 agreement scale.)

[kWh_EVAL] I liked that I could see the electricity usage expressed in kWh.

[kWh_USEFUL] I think that seeing the electricity usage expressed in kWh was useful.

[kW_EVAL] I liked that I could see the power being used expressed in kW

[kW_USEFUL] I think that seeing the power being used, expressed in kW, was useful

[DOLLARS_EVAL] I liked seeing the electricity consumption expressed in dollars and cents

[DOLLARS_USEFUL] I think that seeing the electricity consumption expressed in dollars and cents was useful.

[BUDGET_EVAL] If I had a display at home, I would like it to mark a budget goal that I could see.

[BUDGET_USEFUL] If I had a display at home, I think a budget marker would be useful.

[COMPARISON_EVAL] If I had a display at home, I would like it to show my neighbors’ energy information.

[COMPARISON_USEFUL] If I had a display at home, I think seeing my neighbors’ energy information would be useful.




7. Finally, participants were asked a set of demographic questions

[AGE] I am ________ years old.

[SEX] You are (male) (female)

[EDUCATION] What is your highest level of education? o Elementary School o High School o Some College o Completed College o Graduate Degree

[HOMEOWN] Do you (or your family) own the place you live in? o Yes o No

[BILLPAYER] Are you in charge of paying energy bills at your home? o Yes o No

[MOTIVATION_SAVE] Overall, how interested are you in using less energy at home? o Not interested at all o A little interested o Moderately interested o Very interested o Extremely interested




APPENDIX E Appendix E, the full set of experimental condition screens, is provided below.

CONDITION 1. BASIC INFORMATION ONLY X SPECIFIC DIAGNOSTIC INFORMATION




CONDITION 2. BASIC INFORMATION ONLY X GENERAL TIPS




CONDITION 3. BASIC + BUDGET X SPECIFIC DIAGNOSTIC INFORMATION




CONDITION 4. BASIC + BUDGET X GENERAL TIPS




CONDITION 5. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X SPECIFIC DIAGNOSTIC

INFORMATION




CONDITION 6. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X GENERAL TIPS




CONDITION 7. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X SPECIFIC

DIAGNOSTIC INFORMATION




CONDITION 8. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X GENERAL TIPS




CONDITION 9. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X SPECIFIC DIAGNOSTIC

INFORMATION




CONDITION 10. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X GENERAL TIPS




CONDITION 11. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X SPECIFIC

DIAGNOSTIC INFORMATION




CONDITION 12. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X GENERAL TIPS




CONDITION 13. BASIC X NO ENERGY SAVING SUGGESTIONS




CONDITION 14. BASIC + BUDGET X NO ENERGY SAVING SUGGESTIONS




CONDITION 15. BASIC + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X NO ENERGY SAVING

SUGGESTIONS




CONDITION 16. BASIC + BUDGET + SOCIAL INFORMATION HIGHER THAN NEIGHBOR X NO ENERGY

SAVING SUGGESTIONS




CONDITION 17. BASIC + SOCIAL INFORMATION LOWER THAN NEIGHBOR X NO ENERGY SAVING

SUGGESTIONS




CONDITION 18. BASIC + BUDGET + SOCIAL INFORMATION LOWER THAN NEIGHBOR X NO ENERGY

SAVING SUGGESTIONS




APPENDIX F UNIVERSITY OF CALIFORNIA, DAVIS

ONLINE CONSENT TO PARTICIPATE IN A RESEARCH STUDY

STUDY TITLE: Energy Feedback Device Evaluation

This is a research study conducted by Kristin Heinemeier from the Western Cooling

Efficiency Center at UC Davis, and funded by Southern California Edison. You have the right

to know about the procedures, risks, and benefits of the research study.

Participating in research is your choice and voluntary. You have the right to know about the

procedures, risks, and benefits of the research study. If you decide to take part, you can

change your mind later and leave the study. No matter what decision you make, there will

be no penalty to you.

If you decide to take part in this study, you can decide to stop at any time.

ABOUT THIS RESEARCH STUDY

We hope to learn more about the way in which energy feedback devices are used. About

200 people will take part in this study.

If you decide to participate in this study, you will be asked to familiarize yourself with a

simulated energy feedback device (presented in a tablet) and answer questions about the

information provided in the device. The whole process is expected to take no more than 10

minutes.

Discomfort and Risks

There is minimal risk to participating in this study.

We will not ask for your name or any other identifiable information, so your privacy and the

confidentiality of your responses are protected. For more information about risks, ask the

Researcher.

Benefits

You will not directly benefit from taking part in this research. The information we get from

this study will help us understand how people use energy feedback and determine the best

way to design these devices in the future, so that they can effectively help residents save

energy.

costs




There is no cost to you beyond the time and effort required to complete the procedure(s)

described above.

COMPENSATION

To show our appreciation for your help, you will receive a five dollar gift card once you

complete the study.

WHO CAN ANSWER MY QUESTIONS ABOUT THE STUDY?

If you have questions, please ask us. You can talk to the Researcher about any questions or

concerns you have about this study at:

Claudia Barriga at phone number: 530-758-1882

Tai Stillwater at email: [email protected]

For questions about your rights while taking part in this study call the Institutional Review

Board at (916) 703-9167 or write to IRB Administration, CTSC Building, Suite 1400, Room

1429, 2921 Stockton Blvd., Sacramento, CA 95817. Information to help you understand

research is on-line at www.research.ucdavis.edu/IRBAdmin.

By clicking on “Accept” below, you indicate that you have read the consent document, and

agree to participate in the study,

Accept

http://www.research.ucdavis.edu/IRBAdmin




GLOSSARY

IP Internet Protocol

OpenADR A protocol that standardizes, automates and simplifies demand response to

enable utilities to cost-effectively meet growing energy demand, and

customers to control their energy use.

SEP Protocol that offers IP-based control for advanced metering infrastructure and

home area networks

WiFi A common protocol for wireless data transfer

ZigBee A specification for a suite of high level communication protocols using small, low-power

digital radios based on an IEEE 802 standard for personal area networks.

http://en.wikipedia.org/wiki/Specification_(technical_standard)

http://en.wikipedia.org/wiki/Digital_radio

http://en.wikipedia.org/wiki/IEEE_802.15.4

http://en.wikipedia.org/wiki/Personal_area_network




REFERENCES

1 Ehrhardt-Martinez, K., Donnelly, K.A., Laitner, J.A. (2010) Advanced Metering Initiatives

and Residential Feedback Programs: A Meta-Review for Household Electricity-Saving

Opportunities. American Council for an Energy-Efficient Economy Washington, DC.

2 Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007). The

Constructive, Destructive, and Reconstructive Power of Social Norms. Association for

Psychological Science, 18(5), 429-434.

3 Meier, A., Aragon, C., Pfeffer, T., Perry, D., & Pritoni, M. (2011). Usability of residential

thermostats: Preliminary investigations. Building and Environment, 46(10), 1891-1898.

4 Haakana, M., Sillanpaeae, L., & Talsi, M. (1997). The effect of feedback and focused advice

on household energy consumption. European Council for an Energy Efficient Economy

ECEEEE Summer Study

5 Lewis, S. C. R. (2011). Energy in the Smart Home. In R. Harper (Ed.), The Connected

Home: The Future of Domestic Life. London: Springer-Verlag.

6 Note that although we focused exclusively on electricity feedback, utilities and third-party

suppliers/integrators are also interested in learning about the possibilities for using a

common system for different applications or to monitor different types of energy, furthering

the concept of integrated demand side management and the Smart Grid. Customers would

not want to pay for separate systems, e.g., electrical energy monitoring, gas monitoring,

home security, entertainment, automation, etc. if the systems can be

integrated/consolidated. This emerging technology is not yet prevalent in the market, and

thus it was beyond the scope of our study.

7 Distinction between Small and Large commercial is as defined by vendor.

8 Peffer, T., Perry, D., Pritoni, M., Aragon, C. & Meier, A. (2012). Facilitating energy savings with programmable thermostats: evaluation and guidelines for the thermostat user interface. Ergonomics, (1-17). DOI:10.1080/00140139.2012.718370

9 Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.

10 Jacobson, R. P., Cialdini, R. B., & Mortensen, C. R. (2010). Bodies Obliged and Unbound:

Differentiated Response Tendencies for Injunctive and Descriptive Social Norms. Journal

of Personality and Social Psychology, 100(3), 433-448.

11 Schultz, P. W., Nolan, J. M., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2007). The

Constructive, Destructive, and Reconstructive Power of Social Norms. Association for

Psychological Science, 18(5), 429-434.




12 Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2007). The effect of tailored

information, goal setting and tailored feedback on household energy use, energy-related

behaviors, and behavioral antecedents. Journal of Environmental Psychology, 27(4), 265-

276.

13 McCalley, L. T., deVries, P. W., & Midden, C. J. H. (2010). Consumer response to product

integrated energy feedback: behavior, goal level shifts, and energy conservation.

Environment and Behavior, 43(4), 525-545.

14 Wilson, C., & Dowlatabadi, H. (2007). Models of individual decision making and behavior:

Lessons for designing interventions to reduce residential energy use. Environment, 32, 1-

59.

usability of in-home energy displays - etcc · usability of in-home energy displays ht.11.sce.013...

Documents