i n t e r a c t i o n s / n o v e m b e r + d e c e m b e r 2 0 0 6: / 26

EVALUATION IS A MAINSTREAM ACTIVITY IN HCI. For many years we saw the

emergence of a plethora of techniques to measure user-orientated quality assessment of

technology: usability, satisfaction, efficiency, effectiveness, learnability, usefulness, and so

on. In recent times, however, the discussion seems to have moved on. Issues surrounding

the wider relationship between people and technology or the user experience are popular.

And we’ve seen a number of new concepts emerging, such as engagement, pleasure, pres-ence, and fun, to name a few. Their proponents suggest that these concepts represent

important aspects of usage that are omitted by traditional approaches to evaluation. We’ve

heard it all before, of course, but the words are new.

However, does the emperor actually have new clothes? In their efforts to explore con-

cepts related to the user experience, researchers have been slow to articulate how their

proposals should be measured or indeed if they can be measured at all. Clearly what is

needed is an organized discussion on what aspects of use need to be investigated when

the time comes for evaluation and how they could be assessed. We propose that when

evaluating technology, there are three primary elements that need to be considered, name-

ly, the product, the interaction between the user and the product, and the experience of

using the product. Each of these three elements represents a unique but interdependent

aspect of usage. These are Functionality (product), Usability (interaction), and Experience(user experience). Each area asks a different question about usage using a different lan-

guage of discourse.

WHAT SHOULD WE BE EVALUATING? Someone sending a text message with a

numeric keyboard may well agree that the usability of the interface is awful, but their expe-

rience may be very immersive and compulsive, and, as we see, different mobile-phone

operators pack in different kinds of functionalities with their product, hoping to attract dif-

ferent market sectors. This is why we’ve suggested

that Functionality, Usability, and Experience represent

three aspects of usage that need to be considered

when designing and evaluating technology [9].

Functionality is a technical issue and refers solely to

the product. In this instance, the goal of evaluation is

to answer the question, “What does the product

do?” The usefulness of device features, maintainabili-

ty, and reliability are some of the issues that could be

addressed in such an evaluation.

Usability is a characteristic of the interaction

between the user and the product. It is a user issue;

therefore the product needs to be tested with real

Functionality, Usability, and User Experience: Three Areas of ConcernNiamh McNamara | University College Cork, Ireland | n.mcnamara@ucc.ieJurek Kirakowski | University College Cork, Ireland | jzk@ucc.ie

EXPERIENCE

FUNCTIONALITY

USABILITY

Fig. 1: Three aspects of technology usage (©2005 IEEE)

: / 27i n t e r a c t i o n s / n o v e m b e r + d e c e m b e r 2 0 0 6

S P E C I A L S E C T I O N

WAITS&MEASURES

users. The criteria of support and goal facilitation are important here in answering the

question, “Can I make the product do what I want it to do?”

Finally, the user experience considers the wider relationship between the product and

the user in order to investigate the individual’s personal experience of using it. According

to McCarthy and Wright [7, 8] these questions might include “how the person felt about

the experience, what it meant to them, whether it was important to them, and whether it

sat comfortably with their other values and goals.”

These three elements are not strictly independent of each other. For example, although

usability is not a product characteristic, physical features of the product such as the level of

functionality and the way in which features are implemented can have an impact on usabil-

ity. It has even been suggested recently that device aesthetics can influence perceived

usability prior to actual use [12]. The device’s appearance can also affect the user experi-

ence, since people express themselves through the consumer products they own. Finally,

usability also influences the user experience. Poor usability would no doubt contribute to a

negative user experience, which in turn would possibly discourage further use of the prod-

uct or the inclination to buy another from the same source.

We saw some preliminary support for this framework emerging during the development

of a questionnaire to measure user satisfaction with electronic consumer products—the

Consumer Products Questionnaire (CPQ). (We’re hoping to bring more information into the

public domain about this questionnaire in the near future.) Respondents completing the

CPQ were asked to describe in their own words helpful and unhelpful aspects of devices

they were evaluating. We gathered nearly 1,000 comments in the course of time, which

we pooled and analyzed using Krippendorff’s Content Analysis method [6]. We were sur-

prised that only 24 percent of the comments related to usability; 47 percent of the com-

ments addressed functionality, performance, hardware limitations, and durability; and 23

percent of comments were related to general experience with the device, including aes-

thetics, usefulness, concerns about health and safety, cost of using the device, and enter-

tainment value. The Content Analysis method has its own standards and criteria for relia-

bility and objectivity. Doing this analysis was a learning experience for us and helped us see

things more clearly.

HOW SHOULD WE APPROACH MEASUREMENT? The question of how best to

assess each area of concern is not an easy one to answer. In terms of evaluation, it is impor-

tant to realize that each area of investigation incorporates its own unique set of method-

ologies and assumptions and the significance of choosing an appropriate methodology

when carrying out an evaluation in any of the three areas cannot be overemphasized. It

should be clear that one could not use established usability evaluation methods (UEMs) like

time on task, or number of errors to assess the user experience. Likewise, one cannot use

UEMs to assess a product’s functionality. It is vital that experience is not reduced method-

ologically to usability in just the same way that it would be a “category mistake” [11] to

attempt to measure functionality in UEM terms.

Assessing Functionality. This might include determining what features should be pro-

vided by the device, as well as evaluating device performance, reliability, and durability. The

analysis of user comments revealed that, in general, respondents appreciated the functions

provided by electronic devices, but they were intolerant of unnecessary functions. The com-

ments also indicated that users felt some devices lacked certain functions that they would

find useful. This suggests that designing for a market position is important, and designers

should take the trouble to ask for a broad description of what users would like to be able

to do with a particular product rather than adding additional features for their own sake or

because it is inexpensive to do so [10, 1, 2]. Of course it should be noted that this is not a

foolproof option. It would be difficult for users to decide beforehand exactly what features

they do and do not want, and to some extent the development of electronic devices is gov-

erned by an attempt on the part of the designers to “try it and see if there’s a use for it.”

Assessing Usability. User comments suggest that issues such as transparency, learnabil-

ity, and the support offered to users through guides, manuals, and clear and informative

device feedback are areas that require particular consideration: all the “traditional” preoc-

cupations of usability engineering. The measurement of usability should be based on the

definition of usability as quality in use, as enshrined in ISO 9241-11, as “the extent to which

a product can be used by specified users to achieve specified goals with effectiveness, effi-

ciency and satisfaction” [4]. There are numerous well-established usability evaluation meth-

ods, but be warned: Evaluators will have to determine the success with which these can be

applied to new technology before they can be used with any degree of confidence.

Assessing Experience. As this is a relatively new area of investigation within HCI, there

aren’t any well-developed assessment methods for evaluators. Some people advocate

broadening the usability construct to include more-subjective dimensions, while others call

for the creation of completely new constructs that are distinct from usability. Interestingly,

most approaches tend to characterize experience as dependent on product features, a

method insightfully dubbed design reductionism by Hassenzahl, Beu, and Burmester [3]. A

more suitable approach, McCarthy and Wright’s Felt-Life framework, argues against design

reductionism, suggesting that user experience cannot be deduced from product features.

They propose that usability professionals should not be concerned with designing an expe-

rience but rather designing for experience.

Advances in technology change the relationship between people and technology, and

we have to keep up. But let’s not just leap in. Successful measurement in HCI depends on

a clear understanding of what we are measuring and how we should measure it. Without

this understanding evaluators can offer little guidance to designers to assist them in the

creation of technology that truly supports the user.

ABOUT THE AUTHOR Niamh McNamara graduated with a first-class honors degree in

Applied Psychology from University College Cork. She has undertaken research into the

usability of the WAP (Wireless Application Protocol) service and evaluating Web site

usability including a Web site for adults with limited literacy skills. She is currently pursu-

ing a PhD on the development of a tool to evaluate user satisfaction with electronic consumer products.

Jurek Kirakowski specializes in quantitative measurement for human-computer inter-

action. His major research goal has been to show how the quality of use of IT products

can and should be quantitatively measured in an objective manner. Dr. Kirakowski

and the Human Factors Research Group in University College Cork have contributed

the SUMI (Software Usability Measurement Inventory), and the WAMMI (Web site Analysis and

Measurement Inventory) questionnaires which are by now de-facto standards in their respective areas.

His personal Web page is http://www.ucc.ie/hfrg/jk.

REFERENCES 1. R. Den Buurman, “User-centereddesign of smart products,” Ergonomics vol. 40,no.10, pp.1159-1169, 1997. 2. R.R. Hall,“Prototyping for usability of new technology,”International Journal of Human-Computer Studies,vol. 55, pp. 485-501, 2001. 3. M. Hassenzahl, A.Beu, & M. Burmester, “Engineering joy,” IEEESoftware, vol.18, no.1, pp.70-76, 2001.4. ISO 9241, Ergonomic Requirements for OfficeWork with Visual Display Terminals, Part 11:Guidance on Usability, 1998. 5. P.W. Jordan,Designing Pleasurable Products: An Introduction tothe New Human Factors. London: Taylor & Francis,2000. 6. K. Krippendorff, Content analysis: An intro-duction to its methodology. Beverly Hills, CA: SagePublications, 1980. 7. J. McCarthy & P. Wright,“Putting ‘felt-life’ at the centre of human-computerinteraction (HCI),” Cogn Tech Work, vol.7, pp. 262-271, 2005. 8. J. McCarthy & P.C. Wright, Technologyas experience. Cambridge, MA: MIT Press, 2004.9. N. McNamara & J. Kirakowski, “Defining usabili-ty: Quality of use or quality of experience?,” in Proc.IEEE International Professional CommunicationConference, Limerick, Ireland, 10-13 July, pp.200-204, 2005. 10. D.A. Norman, The InvisibleComputer. Cambridge, MA: MIT Press, 1998.11. G. Ryle, The Concept of Mind. London:Flutchinson, 1949. 12. N. Tractinsky, A.S. Katz, & I.Ikar, “What is beautiful is usable,” Interacting withComputers, vol. 13, pp.127-145, 2000.

: / 28 i n t e r a c t i o n s / n o v e m b e r + d e c e m b e r 2 0 0 6

Permission to make digital or hard copies of all or part of this

work for personal or classroom use is granted without the fee,

provided that copies are not made or distributed for profit or

commercial advantage, and that copies bear this notice and the

full citation on the first page. To copy otherwise, to republish, to

post on services or to redistribute to lists, requires prior specific

permission and/or a fee. © ACM 1072-5220/06/1100 $5.00