a framework for evaluating the usability of mobile phones based on multi-level, hierarchical model...

Interacting with Computers 21 (2009) 263–275

Contents lists available at ScienceDirect

Interacting with Computers

journal homepage: www.elsevier .com/locate / intcom

A framework for evaluating the usability of mobile phones basedon multi-level, hierarchical model of usability factors

Jeongyun Heo a, Dong-Han Ham b,*, Sanghyun Park c, Chiwon Song c, Wan Chul Yoon d

a MC Product Planning Strategy Group, LG Electronics, South Koreab School of Engineering and Information Sciences, Middlesex University, UKc MC R&D Centre, LG Electronics, South Koread Department of Intelligent Service Engineering, KAIST, South Korea

a r t i c l e i n f o

Article history:Received 19 August 2008Received in revised form 26 May 2009Accepted 26 May 2009Available online 2 June 2009

Keywords:Mobile usabilityUser interfacesUsability evaluationEvaluation framework

0953-5438/$ - see front matter � 2009 Elsevier B.V. Adoi:10.1016/j.intcom.2009.05.006

* Corresponding author. Address: School of Enginences, Middlesex University, Town Hall, The Burrough+44 20 8411 5284; fax: +44 20 8411 6943.

E-mail addresses: [email protected] (J. Heo), donHam), [email protected] (S. Park), [email protected](W.C. Yoon).

a b s t r a c t

As a mobile phone has various advanced functionalities or features, usability issues are increasingly chal-lenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods andheuristics, most of which are relevant to a software system, might not effectively be applied to a mobilephone. Another point to consider is that usability evaluation activities should help designers find usabil-ity problems easily and produce better design solutions. To support usability practitioners of the mobilephone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis ofa set of collected usability problems and our previous study on a conceptual framework for identifyingusability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobilephone from a particular perspective. As there are goal-means relationships between adjacent levels, arange of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantageis that it supports two different types of evaluation approaches: task-based and interface-based. To sup-port both evaluation approaches, we developed four sets of checklists, each of which is concerned,respectively, with task-based evaluation and three different interface types: Logical User Interface(LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifiesan approach to quantifying usability so that several usability aspects are collectively measured to give asingle score with the use of the checklists. A small case study was conducted in order to examine theapplicability of the framework and to identify the aspects of the framework to be improved. It showedthat it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study,we improved the framework in order that usability practitioners can use it more easily and consistently.

� 2009 Elsevier B.V. All rights reserved.

1. Introduction

It is well known that usability is a significant quality attribute ofmobile phones and thus usability evaluation is becoming increas-ingly important in the mobile phone industry (Lindholm et al.,2003; Duh et al., 2006). Although there are a range of usability def-initions, the usability concept specified in ISO/IEC 9126-1 (2001) isnow widely accepted (Folmer and Bosch, 2004). ISO/IEC 9126-1 de-fines usability as ‘the capability of the software product to beunderstood, learned, used and be attractive to the user, when usedunder specified conditions’. This definition is primarily concerned

ll rights reserved.

eering and Information Sci-s, London NW4 4BT, UK. Tel.:

[email protected] (D.-H.(C. Song), [email protected]

with a software product; however, it can be applied to mobilephones taking into consideration features specific to mobilephones.

User experience is another term widely mentioned in contrastto usability. Although some people use both the terms inter-changeably, user experience is much broader concept than usabil-ity (Saffer, 2007). Usability is mainly concerned with the exhibiteddesign features of interactive products in terms of how easy it is touse. But user experience looks at the individual’s entire interactionwith products, as well as the thoughts, feelings, and perceptionsthat result from that interaction (Tullis and Albert, 2008). The con-cept of user experience thus encompasses other issues such ascredibility and engagement as well as usability. Referring to theterminology from the field of software quality, it can be said thatuser experience is more related to the concept of quality-in-use,whereas usability is more to external quality (Bevan, 1999).

Usability evaluation is an essential activity for securing a highlyusable mobile phone, which should be conducted during all the

mailto:[email protected]





http://www.sciencedirect.com/science/journal/09535438

http://www.elsevier.com/locate/intcom

264 J. Heo et al. / Interacting with Computers 21 (2009) 263–275

phases of design life cycle (Kangas and Kinnunen, 2005). Varioususability evaluation methods have been developed and can be clas-sified into three types: usability testing, usability inquiry, andusability inspection (Karat, 1997; Zhang, 2003). Usability testingemploys representative users on typical tasks using a system or aprototype and then evaluates how user interface supports theusers to do their tasks. Typical methods include co-discovery learn-ing, question-asking protocol and shadowing method. Usability in-quiry talks to users, observes how they use a system in real worksettings, and let them answer questions in order to understandusers’ feeling about the system and their information needs. Fieldobservation, focus groups, and questionnaire survey are typicalusability inquiry methods. In usability inspection, usability expertsexamine usability-related aspects analytically. Typical methods arecognitive walkthrough and heuristic evaluation. Ivory and Hearst(2001) distinguish five groups of usability methods by addingtwo more types: analytical modelling and simulation, both ofwhich are focused on predicting usability by using user modeland interface model.

At present, methods for the effective evaluation of mobilephones are an open question (Jones and Marsden, 2006). However,it cannot be said that one method is the best in all situations. It isnecessary to choose an appropriate method, taking into accountevaluation purposes, available time, measures to be collected,and so on (Lavery et al., 1997; Hartson et al., 2003). Therefore sev-eral types of methods for evaluating mobile usability need to bedeveloped.

In this study we are concerned with a method pertaining tousability inspection. We aim to develop a framework for helpingusability experts evaluate the usability of mobile phones and pre-dict likely usability problems in an analytical manner. In analyticalevaluation, usability experts apply their knowledge of typical usersand tasks, guided by heuristics, to predict usability problems (Pre-ece et al., 2002). As users need not be present in this method andactual use situations are not tested, this evaluation method hassome limitations, particularly in terms of context of use.

Even though usability is mainly related to the design attributesof mobile phones, it needs to be regarded as an emergent propertywhich is formed by complicated interaction among a mobile phoneuser interface, user and task characteristics, and other environ-mental factors (Folmer and Bosch, 2004; Lee and Grice, 2004). Thisimplies that usability is a relative concept, not absolute concept,which is dependent on several factors (Jokela, 2004). Another pointto keep in mind is that usability is not static but dynamic in that itcan be changed along time (Dillon, 2001; Ling et al., 2006a,b).These two points imply that mobile phones should be evaluatedin proper consideration of context of use (Coursaris and Kim,2007) and impose challengeable issues on analytical evaluationmethods.

But despite such a drawback, analytical evaluation is very pop-ular in the industry because it is quick and does not require actualusers. For this reason, it has been required to develop some effec-tive analytical methods for evaluating mobile usability (Kiljander,2004; Jones and Marsden, 2006). Although a lot of heuristics andchecklists have been developed to be used for analytical evaluationof software systems, they are not sufficient to address mobileusability issues. It is thus necessary to develop methods that reflectuser interfaces and tasks specific to mobile phones and deal withusability factors of mobile phones.

1.1. Mobile user interfaces and tasks

As described above, we need to understand user interfaces andtasks of mobile phones, which should be systematically consideredin analytical evaluation (Bautsch et al., 2001; Holtzblatt, 2005;Kjeldskov et al., 2005). Mobile phones are portable, self-contained

information and communication systems. They are characterizedby three features affecting the design of user interfaces: (1) theyare used primarily in a user’s hands, (2) they are operated withoutcables, and (3) they support the addition of new application andinternet connection (Weiss, 2002). Ketola and Röykkee (2001) di-vided user interface elements into seven categories from the view-point of their function: input (e.g. softkey, alphanumeric keys, andnavigation tools), display (e.g. icons and indicators), audio andvoices (e.g. ringing tones and microphone), ergonomics (e.g. touchand feeling, and slide), detachable parts (e.g. SIM card and battery),communication method (e.g. Bluetooth), and applications (e.g.making a call, games).

There are three features or constraints of user interfaces thatcan influence the usability of mobile phones (Lee et al., 2006; Linget al., 2006a,b). The first is that mobile phones have too little screento display a lot of information at the same time; therefore, infor-mation organization and navigation can be critical usability issues.The second is that a physical button or key has generally more thanone control function. Thus the functions of a single key are depen-dent on the types of modes. The third is the limit in processingpower and available memory. Other features include multimodal-ity, different display resolutions, and restrictive data entry meth-ods (Zhang and Adipat, 2005).

The increasing number of functions of mobile phones allows usto do various tasks. However, typical functions of mobile phonesinclude making a call, sending a message, managing personal infor-mation, listening music, setting a phone configuration, taking apicture, and playing a game (Kiljander, 2004). Other advancedfunctions are watching TV, remote controlling of home automa-tion, internet banking, personal computing, and so on. In the forth-coming ubiquitous computing environment together with thecontinuing pattern of convergence, mobile phones are expectedto play a key role in accessing a wide range of services and willthus be an essential device of daily life (Jones and Marsden, 2006).

Although our study is mainly concerned with mobile phones,the interface features and task characteristics are very similar tothose of other kinds of mobile devices or hand held informationappliances, such as Personal Digital Assistant (PDA) and smart-phones (Weiss, 2002; Lee and Grice, 2004). But there are still somedifferences between mobile devices; for example, mobile phonesdo not have miniature QWERTY keyboard and do not support thefunction of reading business documents in a variety formats, incontrast to smartphones. But it is expected that the unique featuresof each mobile device will disappear and the boundary of mobiledevices will become ambiguous in the near future. Our stance isthus that the evaluation framework and other methods presentedin this paper can be reasonably applied to other mobile devices.

1.2. Usability factors

Whichever usability evaluation method is used, it is necessaryto consider several kinds of usability factors in a systematic man-ner. Moreover, in the case of expert evaluation using analyticalmethods, comprehensive consideration of usability factors are veryimportant to overcome its limitation (Love, 2005).

Many studies have examined various kinds of factors character-izing the usability of IT artefacts (Frøkj�er et al., 2000; Klockaret al., 2003; Folmer and Bosch, 2004; Hornb�k, 2006). Typicalexamples of the usability factors can be found in design principlesor guidelines, which have been developed for software user inter-faces. Design principles are high-level design objectives that arenot dependent on specific tasks or interface objects. But designguidelines are detailed means to realize a design principle underspecific task conditions or interface objects (Peuple and Scane,2004). Typical examples of principles include affordance, consis-tency, visibility, proper feedback, and so on (Nielsen, 1993; Hix

J. Heo et al. / Interacting with Computers 21 (2009) 263–275 265

and Hartson, 1993; Hartson, 2003). One example of guidelines re-lated to consistency is ‘to locate home button at the right top of ascreen’. But usability factors coming from design principles andguidelines do not reflect interface features or functions in detail,which are another aspect influencing the usability. Examples ofusability factors related to interface features are: size and colourof font, array of navigation buttons, screen size, and so on. Thusthere is a problem of how to bridge the semantic gap betweentwo different groups of usability factors.

Such a problem can also be found in the studies dealing with theusability factors of mobile phones (Ketola and Röykkee, 2001;Coursaris and Kim, 2006). Ketola and Röykkee (2001) proposed se-ven usability factors of mobile phones: integration of functionality,availability, utility and ease of using services, readiness for use,informativity, usefulness of support material and interoperability.In contrast, some studies considered the usability factors by focus-ing on the features or functions of mobile phones, size of soft keysand screen, layout of navigation keys, and menu structure, withouttheir clear connection to the conceptual, abstract usability factorsat the level of design principles (e.g., Klockar et al., 2003).

To resolve this problem of bridging conceptual usability factorsand feature-related factors, several studies proposed a hierarchicalmodel of usability factors (ISO/IEC 9126-1, 2001; Keinonen, 1998;Donyaee et al., 2002; Folmer et al., 2003; Scholtz, 2006). For exam-ple, Donyaee et al. (2002) developed Quality in Use IntegratedMeasurement (QUIM), which is a hierarchical model for measuringusability of interactive software. The top level has ten factors,which are not directly measurable, such as portability, efficiency,satisfaction, learnability, accessibility. They are decomposed intoa total of 26 measurable sub-factors, which are further brokendown into 126 specific metrics. For example, efficiency can bemeasured by using several sub-factors such as time behaviour, re-source utilization, and minimal action.

1.3. Research problem

Previous studies on usability factors contributed towardsunderstanding what could constitute the usability factors; how-ever, very few attempts have been made at identifying and orga-nizing them in a systematic way (Frøkj�er et al., 2000). Usabilitypractitioners actually need a structuralized model that can helpunderstand the relationships among usability factors. Additionally,most of earlier studies are concerned with software products, notmobile phones; therefore they do not reflect the features of mobilephones appropriately (Lee and Grice, 2004).

The above problems are true of analytical usability evaluationmethods that are now widely used in the mobile phone industry.Most of the methods are based on traditional methods and heuris-tics, which are more relevant to a software system; therefore, thecurrently used methods might not effectively be applied to a mo-bile phone (Beck et al., 2003; Hagen et al., 2005). Additionally,usability evaluation methods should help usability practitionersidentify critical usability problems systematically and generatebetter design ideas (Blandford et al., 2004; Lee et al., 2006). In thisregard, one critical role of an evaluation method is to lead usabilitypractitioners to consider various usability factors from multiplepoints of view (Hartson et al., 2003). It is, however, not easy to findan evaluation method for mobile phones, which satisfies therequirements described here.

With this issue in mind, this study proposes a framework foranalytically evaluating the usability of mobile phones on the basisof a hierarchical model of usability factors, which connects the per-ceived usability to the physically designed features of mobilephones in terms of goal-means relations. The remainder of this pa-per is organized as follows. In Section 2, we describe a researchmethod we adopted to develop the framework. Next, we propose

the evaluation framework in Section 3, paying particular attentionto the hierarchical model of usability factors. Section 4 deals withhow to use the proposed framework and explains a set of check-lists, a quantification scheme, and evaluation process that havebeen developed to support the use of the framework. Section 5 de-scribes a small case study in which we evaluated the proposedframework in terms of its usefulness and ease to use and explainshow we improved the framework based on the results of the casestudy.

2. Research method

We identified eight requirements that a usability evaluationframework needs to satisfy. For this, we inquired about the proper-ties that good evaluation framework should have, from six usabil-ity testers working in a mobile phone manufacturing company. Weconducted literature review to examine the properties as well. Col-lectively considering the opinions and the review results, the eightrequirements are established as shown below:

– Fact-based approach: Usability problems collected in earlier eval-uations need to be referenced to establish a framework as theymight be a standard usability problem list (Hartson et al., 2003).

– Modularization: A framework needs to allow usability practitio-ners to consider only a certain part of user interfaces, which canbe redesigned without influencing other parts (Treu, 1994; Car-liner, 2003; Hartson, 2003).

– Hierarchical approach: A framework needs to take account of thefact that usability factors and problems have different abstrac-tion levels (Dillon, 1999).

– Optimization: A framework needs to reflect that there might bean optimized way for economically evaluating a usability issuein terms of what user interface features to be examined andwhat kind of tasks to be performed (Kwahk and Han, 2002).

– User-oriented: It should be easy for usability practitioners tolearn and use a framework (Hertzum and Jacobsen, 2001).

– Implementable: It should be easily implemented through adetailed set of checklists or guidelines (Zhang and Adipat, 2005).

– Context-based: Usability evaluation based on a frameworkshould help usability practitioners understand and predict con-texts of use (Bautsch et al., 2001).

– Design-oriented: A framework should help generate betterdesigns after identifying potential usability problems (Hartsonet al., 2003; Kadyte and Tétard, 2004).

These requirements serve as the bases of adopting researchmethods and accompanying research activities for establishingand improving the evaluation framework proposed in this study.This section describes three elementary research activities con-ducted to build the framework.

2.1. Collection and analysis of usability problems

Examination of pre-reported usability problems in associationwith task analysis results can provide a useful reference point foridentifying and organizing usability factors to be considered inthe evaluation framework. We therefore collected 136 usabilityproblems that were reported in earlier usability tests conductedat one mobile phone manufacturer in Korea. We also gatheredusers’ opinions on the usability factors of mobile phones throughweb sites or focus group interviews. These opinions were used toclassify and abstract the usability problems and select representa-tive 28 mobile phone tasks.

The first step of analyzing the usability problems was to classifythem according to two criteria: whether they are task-independent


or task-dependent problems, and which of the three interface areas(PUI: Physical User Interface, LUI: Logical User Interface, GUI:Graphical User Interface) are primarily related to them. As de-scribed in the Section 1, seven categories are useful to understandthe functional characteristics of mobile user interfaces. However,these categories can further be generalized into three user inter-face design areas that are more helpful to study users’ interactionwith mobile phones (Kiljander, 2004). This is the reason why weused three interface types as a criterion. Next, employing a cardsorting method, we could generalize the 136 problems into 31high-level problem groups. Secondly, typical 28 mobile phonetasks were selected in consideration of their frequency of useand the degree of relevance to the usability of mobile phones.Thirdly, as Fig. 1 shows, we associated the 136 usability problemswith the 28 mobile phone tasks (mapping between usability prob-lems and tasks). Thus we could understand the usability problemsfrom two perspectives: interface design features and user tasks.The analysis results of these usability problems were referencedto determine the relationship between evaluation areas and usabil-ity indicators in the framework and to develop the items of thechecklists to be explained in Section 4.

2.2. Analysis of user interface design principles

Usability is a conceptual construct that should be designed aswell as evaluated through all the phases of development life cycle.Although they are primarily developed to support the design of agood user interface, user interface design principles can be effec-tively referred to for usability evaluation as well. In particular, theyare helpful to organize various factors affecting the usability. Addi-tionally, they can give some insights about how to translate evalu-ation results into design improvement. For this reason, wecollected and analyzed several user interface design principles,

Navigating menu

Searching call history

Makincall fr

contac

… …

…… …

Style guide

Text editing

High-level problems Usability problems

Vicual cue

Fig. 1. Mapping between usability pr

which include Nielsen’s heuristics (Nielsen, 1993), ISO/IEC 9241-11 (1998), and Hix and Hartson’s principles (1993). We analyzedthem taking account of cognitive processing steps of performinga mobile phone task, which are composed of planning, translation,execution, and assessment. As a result, we developed several eval-uation criteria corresponding to each cognitive processing step.They were used as basic information for determining usability cri-teria and connecting usability indicators to usability criteria in thehierarchical model to be explained in next section.

2.3. Identification of evaluation items and strategy

To establish the framework, it was necessary to determine whatinterface features to evaluate and how to evaluate them in a mean-ingful way. Fig. 2 shows the interface features that we thought tobe considered for evaluating the usability of mobile phones. In thisstudy we classified them into three interface design areas: LUI, GUI,and PUI. We define LUI (e.g. menu and navigation structure) as aninterface related to information contents and structure for taskexecution. GUI (e.g. icon and font) is defined as an interface con-cerned with graphical or visual items presenting task-relevantinformation. PUI (e.g. keypad and microphone) means tangiblephysical interface properties supporting users’ physical operationsneeded to carry out tasks. However one interface feature can haveproperties related to two or three interface areas at the same time.For example, menu is one of LUI and has some GUI elements likeicons, fonts, and colour as well.

Fig. 3 illustrates the basic strategy of evaluating the usability ofmobile phones. We propose to conduct four kinds of evaluations:task-based (dependent) evaluation, LUI evaluation, GUI evaluation,and PUI evaluation. The latter three evaluations focusing on threeinterface areas can be regarded as a task-independent evaluation.Each interface area is usually designed by different design teams

g a omt list

Receiving a call … Adding new

contacts

Sending text

messages…

Sending file through

bluetooth…

…

Typical mobille phone tasks

…

oblems and mobile phone tasks.

Keypad

Screen layout

Information Architecture

Soft key

Icon, wording

Accessory, Battery

Function,Spec

Task Flow

PUI GUI

General Elements of Mobile Phones

•Overall size-length, width, height

•LCD•Key-size, shape,layout, Type etc

•(H/W) Icon•(H/W) Label•Backlighting•Receptacle •Accessory•Battery•Form Factor•H/W spec

•Object GUI-Font-Icon-indicator-indicator-header etc

•Screen GUI-Top menu-list-pop-up-idle etc.

•Function GUI-phone book-message-camera etc

LUI•Information architecture-menu structure-menu label-menu icon

•Soft key•UI style•Operation sequence

•Screen layout -input field -title-etc

•sound •spec

Fig. 2. Identification of design features to be evaluated.

Task-based Evaluation

LUI Evaluation

PUI Evaluation

GUI Evaluation

Efficiency of procedure

Support of operation sequence

Stability of use

Cognitive burden of execution

Information architecture

Wording (Menu label)

Soft key

Functional options or details

Ergonomic considerationof buttons, grip, and accessory, etc.

Contextual considerationof position and manipulation

Icon (meaning and aesthetics)

Font type and size

Display style and colour

1

2

3

4

- Icon (meaning and aesthetics)

- Font type and size

- Display style and colour

- Ergonomic considerationof buttons, grip, and accessory, etc.

- Contextual considerationof position and manipulation

- Information architecture

- Wording (Menu label)

- Soft key

- Functional options or details

LUI-basedChecklist

PUI-basedChecklist

GUI-basedChecklist

- Efficiency of procedure - Support of operation sequence- Stability of use - Cognitive burden of execution

Task-basedChecklist

Evaluation Type

Fig. 3. Evaluation strategy using the framework.


or at different design phases. In this regard, such a differentiationof evaluation types can be useful to usability practitioners becauseit can support both modular evaluation and holistic evaluation ofuser interface features.

Task-based evaluation needs a set of task scenarios and shouldbe conducted in the context of actual task execution. For example,efficiency of task procedures and cognitive load for task executionwould not be evaluated without actually conducting tasks. Whenwe conduct task-based evaluation for one task, it is assumed thatat least one of three interface areas is concerned with carrying

out the task. In contrast, if we think of logical information structureand menu label in LUI, they can be evaluated by several methodswithout task execution. In this study we propose the use of check-lists, which will be explained in Section 4.

3. Proposed evaluation framework

Based on the results described in Section 2, we developed aframework for evaluating the usability of mobile phones. One core


element of the framework is a multi-level, hierarchical model ofusability factors, which organizes a range of factors affecting theusability of mobile phones in terms of their abstraction levels.The usability problems, users’ opinions, and user interface designprinciples explained in Section 2 were all referred to when buildingthis new hierarchical model. In this section we firstly explain thenewly developed hierarchical model and then describe two levelsof usability factors-usability indicator and usability criteria in moredetail. Lastly, we propose the new framework specifying what andhow to evaluate the usability of mobile phones, based on the hier-archical model.

3.1. Hierarchical model of usability factors

The proposed hierarchical model has four abstraction levels, asshown in Fig. 4. One point characterizing this model is that thereexist goal-means relations between adjacent levels. Thereforeone element at the higher-level is connected to one element atthe lower-level in terms of a goal-means relation. The top-levelis the usability of a mobile phone, which we aim to evaluate ulti-mately. As it is an emergent concept, usability cannot be either di-rectly or exactly measured. Instead it could be indirectly indicatedas a summation of some usability factors underlying usability con-cept. The second abstraction level in which these usability dimen-sions lie is called usability indicator. Our stance is that fiveusability indicators are meaningful to the usability of mobilephones: visual support of task goals, support of cognitive interac-tion, support of efficient interaction, functional support of userneeds, and ergonomic support. Although these five indicators cantogether form a usability index depicting usability concept, theyare still too abstract to be measured directly. So they need to befurther down specified in their meaning in order to be measuredmore directly. The next lower abstraction level, which is calledusability criteria, identifies several usability factors that can bemeasured directly by using several methods (e.g. checklists). Asone usability indicator has a goal-means relation to at least oneusability criteria, five indicators can be indirectly measuredthrough usability factors at the criteria level. The lowest level be-low the criteria level is called usability property, which exhibitsthe actual states or behaviours of several interface features of amobile phone. Thus the usability factors at this level provide an ac-tual usability value to the criteria level.

The proposed hierarchical model is similar to the models pro-posed by Donyaee et al. (2002) and Folmer and Bosch (2004). Butthe main difference between those models and ours is that they fo-

Usability Property Observable, measurable

Usability CriteriaUsability factors that canmethods (e.g. checklist)property

Usability Indicator Hypothetical, abstract coused for indicating the u

Usability Emergent feature to be

Fig. 4. Hierarchical mode

cused on software, not mobile phones. Therefore they do not re-flect the interface features specific to mobile phones at the lowerlevels of the hierarchy. Another difference lies in that our modelhas explicit goal-means relationships between adjacent levels,which are not specified in them.

3.2. Usability indicators

As explained previously, usability indicators provide a basis formaking a decision on the degree of usability. Usability indicatorsaim to connect observable and measurable usability factors atthe criteria level to the human perceived usability. Accordinglythese indicators are all concerned with explaining how users per-ceive a system without being influenced by some technical pecu-liarities of a particular system. They are not limited to theusability features specific to mobile phones. It can therefore be saidthat these indicators are reasonably applied to any other informa-tion appliances like personal digital assistant.

The hierarchical model has five usability indicators: visual sup-port of task goals, support of cognitive interaction, support of effi-cient interaction, functional support of user needs, and ergonomicsupport. In order to identify these indicators, we abstracted a rangeof design principles reviewed in Section 2.2 and made a referenceto another earlier study of ours that identified mobile usability fac-tors and collected eighteen users’ opinions on their perception ofmobile usability (Ham et al., 2008).

When we derived the five indicators, we considered two mainthemes of human–computer interaction (HCI): relationship be-tween usefulness and usability, and three aspects of human inter-action with systems (Te’eni et al., 2007). It is certain thatusefulness is a prerequisite to guarantee usability. For this reasonwe selected functional support of user needs as an indicator. Hu-man interaction with systems can be examined from three view-points: physical, cognitive, and affective. To reflect the physicalaspect of interaction, we selected ergonomic support as an indica-tor. In the case of cognitive interaction, we considered three stepsof information processing: perception of system states, under-standing and planning, and execution. These three steps weretranslated, respectively, into the remaining three indicators: visualsupport of task goals, support of cognitive interaction, and supportof efficient interaction. Because affective aspect of interaction istoo much influenced by actual users, we did not consider it inour framework for supporting analytical evaluation.

Table 1 illustrates the meaning of the five indicators when theyhave a high value or low value. For example, if ‘visual support of

properties of a mobile phone

be directly measured by using several , which are applied to the factors of usability

nceptual constructs that can be collectively sability and are not directly measured

characterized with a wide range of factors

l of usability factors.


task goals’ has a high value, it means that user interface is so intu-itive as to allow novice users to use a mobile phone without anydifficulty (no entry barrier). In contrast, its low value indicates thatnovice users can have a great deal of difficulty using the interfacefeatures.

Combination of the values of these five indicators provides aholistic as well as multi-faceted picture of usability and thus en-ables usability practitioners to identify which aspects of a mobilephone has strong points or drawbacks in terms of usability. Insome cases of testing a mobile phone, a single value of usabilityis reported so that one can judge only whether it is usable or notusable. But such a single value might not be useful for usabilitypractitioners to identify the points that should be improved andto compare different design solutions. In this regard, more detailedanalysis of the single value of usability with the five indicators canbe useful to lead usability evaluation results into better designsolutions. Fig. 5 shows three cases of same usability score, eachof which has different combinatorial pictures according to the val-ues of the five indicators. In the first case, we can know that thismobile phone needs to be improved from the viewpoint of ‘visualsupport of task goals’ and ‘support of cognitive interaction’. Thesecond case indicates that experienced users can feel some diffi-

Table 1Meaning of five usability indicators.

Usability indicator High value

Visual support of task goals UI is much intuitive (low entry barrier)It is easy for novices to use

Support of cognitive interaction Operation sequences for executing a task are simple

Support of efficient interaction Various ways of making task execution efficient are p

Functional support of user needs Functions and manipulation style reflect user needs

Ergonomic support Physical ergonomic consideration is well given

Relatively(

Visual supportof task goals

Support of cognitive interaction

Support of efficient

interaction

Functional support of user

needs Ergonomic support

70

Beginners’Operation uncomforta

ErgonomicsatisfiedEspecially lower the u

[case1]

70

[case2]

70

[case3]




interaction






interaction



Fig. 5. Interpretation of usability in

culty using a phone more efficiently whereas it is reasonablyusable at the other four perspectives. Lastly, the third case can besaid that it has relatively low usability at two perspectives (func-tional support of user needs and ergonomic support) in compari-son to the other three indicators.

The five indicators are not completely independent but interde-pendent. It thus might be that enhancing the level of one indicatorreduces or increase the level of other indicators. As shown in thestudy of cognitive dimension (Green and Blackwell, 1998), specifi-cation of trade-off relationships between the usability factors inrelation to design activities is very helpful to usability designers.Our framework, however, does not address the trade-offs betweenthe five indicators.

3.3. Usability criteria

Usability criteria make it possible to obtain the value of fiveusability indicators by measuring the usability aspects of mobilephones with traditional testing methods, such as checklist and in-quiry. Table 2 shows some of usability criteria and their relation-ships with the five indicators. All of these criteria weredetermined on the basis of a review of collected design principles,

Low value

It is difficult for novices to use (high entry barrier)Particularly, the difficulty becomes higher for the first users of amobile phone

Operation sequences are somewhat complexOperation sequences can be more complex to the users who areused to the sequences of other mobile phones

rovided It is not easy to find out an way of enabling efficient task execution

well There is a lack of functions and manipulation style satisfying user needs

Physical ergonomic consideration is not well given

high entry barrier

entry barrier is not high, but there is a usage barrier.is inefficient and inconsistent; so expert user could feel ble

aspect is not good or user functional needs are not fully

the item which has large impact on product utility could sers’ satisfaction

terms of usability indicators.

Table 2Partial list of usability criteria and their relationships with indicators.

Usability indicator Usability criteria

Visual support oftask goals

Do the users think that they can achieve a task using themobile phone?Does the mobile phone make users know whatoperations should be done to accomplish a task goal at thelevel of operation?

Support of cognitiveinteraction

Can the users assess the current state of operation or getthe feedback on task progress?Are visual cues are shown effectively?

Can the users understand the meaning of the cuesexactly?

Is the amount of information delivered appropriately?Are there any measures for preventing error

occurrence?Is the information relevant to task execution providedat right time?

Support of efficientinteraction

Does the mobile phone provide functions making theusers achieve a task more efficiently?Can the users easily perceive the functions designed tosupport efficient interaction?When the users make a mistake, can they recover easilyfrom it?

Functional supportof user needs

Are the users’ needs reflected in the designedfunctionalities?Are the functions designed in consideration of taskcontexts?

Ergonomic support Is the physical manipulation of PUI comfortable?


which was described in Section 2.2, and usability testers’ opinions.In this study these criteria are measured by using a set of checkliststo be explained later.

3.4. Usability evaluation framework

As explained in Section 2.3, the basic strategy for evaluating theusability of mobile phones is to distinguish task-independent eval-uation from task-based evaluation. Task-independent evaluation isagain classified into three groups, depending on which interfacearea is focused: LUI evaluation, GUI evaluation, and PUI evaluation.The proposed evaluation framework stipulates that the value of

Visual support oftask goals

Supporcogniti

interact

Previewing (entry)Capturing (using)

Saving (exit)Searching a number (entry)

Calling (using)Beginning a call (exit)Entering alarm screen

(entry)Setting alarm (using)

Saving alarm setting (exit)……

PUI-based Checklist

GUI-based Checklist

Task-based Checklist

Taking a picture

Making a call from

contact list

Setting alarm

LUI-based ChecklistInformation architecture

Soft key…

Fig. 6. Relationship between eva

five indicators should be obtained to judge the usability of mobilephones. As described previously, it can be obtained by measuringseveral usability criteria. Then next question is how to measurethe usability criteria, pursuing the basic strategy-conducting thedistinguished evaluations. We propose to use a set of detailedchecklists for all the criteria and to identify meaningful relation-ships between usability indicators and the items of the checklists.Fig. 6 shows a part of the relationships between evaluation itemsand indicators. A checklist for task-based evaluation has severaltask and subtask items to be evaluated, and a checklist for evalua-tion of each interface area has several design features to be evalu-ated. In reality one indicator can be related to all of the evaluationitems; therefore we need to consider all the relationships to givemore comprehensive evaluation results. In this case, however,evaluation activities can require a great deal of time and effort,so that such an evaluation with full coverage can be impractical.To avoid this problem, we instead identified certain relationshipswhich are considered to be meaningful to evaluate the usabilityof mobile phones. For this identification, three usability expertsin a mobile phone company were gathered, and they kept discuss-ing until a final relationship was determined, on the basis of theirexperience and knowledge about usability factors. We agree thatsuch relationships shown in Fig. 6 would be incomplete to examinethe usability of mobile phones. Nonetheless we believe that theyare sufficient to reveal most of usability problems and make itmore practical to implement the proposed evaluation approach.Referring to what we discussed in this section, we can depict ourproposed evaluation framework as shown in Fig. 7.

4. Usability evaluation using the framework

4.1. Checklists for implementing the framework

To support implementation of the proposed framework, wedeveloped four sets of checklists, each of which, respectively, helpsconduct four different evaluations. Thus four sets of checklist arecalled, respectively, task-based checklist, LUI-based checklist,GUI-based checklist, and PUI-based checklist. The framework canbe used at anytime during design process by using a relevantchecklist if at least one of three interface design areas is designed.

t of veion


interaction


needs

Ergonomic support

luation areas and indicators.

Visual support of task goals

Support of cognitive

interaction

Support of efficient interaction

Functional support of user needs Ergonomic support

Mobile Usability

Indicator level

Property level

GUILUI PUI

Task-Dependent Task-Independent

Criteria level

- Icon (meaning and aesthetics)

- Font type and size

- Display style and colour

- Ergonomic considerationof buttons, grip, and accessory, etc.

- Contextual considerationof position and manipulation

- Information architecture

- Wording (Menu label)

- Soft key

- Functional options or details

LUI-basedChecklist

PUI-basedChecklist

GUI-basedChecklist

- Efficiency of procedure - Support of operation sequence- Stability of use - Cognitive burden of execution

Task-basedChecklist

Fig. 7. Usability evaluation framework.


Here it is necessary to consider the meaning of task-based eval-uation and task-independent evaluation, referring to three evalua-tion aspects of user interfaces. Evaluation aspect means theviewpoint in relation to the type of evaluative evidence sought.Rouse (1991) discriminated three aspects meaningful to the evalu-ation of human performance: compatibility, understandability, andeffectiveness. Compatibility is the extent to which user interfacesare physically compatible with human abilities and limitations ofinput-output information processing. Understandability is the ex-tent to which humans can understand the meaning of informationprovided through the user interfaces. Effectiveness is the extent towhich user interfaces lead to improved human performance. Therecan be several methods, which range from paper-based analyticalevaluation to full-scope simulation experiment to in-field observa-tion, for evaluating each evaluation aspect. However, consideringthe usefulness and efficiency of an evaluation method, non-exper-imental/empirical methods are suitable to evaluate understand-ability and compatibility. But effectiveness issues need to beevaluated with more empirical methods in which certain task sce-narios are conducted. Associating our proposed evaluation typeswith the three evaluation aspects, it can be said that task-indepen-dent evaluation is more related to the evaluation of compatibilityand understandability, whereas task-based evaluation is moreclose to effectiveness.

Task-based checklist is useful to find out usability problemsduring task execution process and generate design alternativesthat can promise better task performance. This checklist addressesall the 28 tasks which were described in Fig. 1. Fig. 8 shows anexample of task-based checklist for ‘making a call from contact list’task. Interface features of three interface areas can be divided intotwo groups depending on whether they can be evaluated withoutcertain task scenarios or not. Task-based checklist is concerned

with the interface features of which evaluation must accompanyconducting task scenarios. It is unusual that a task scenario is con-ducted with only one interface areas used. Accordingly, contrary toother types of checklist, task-based checklist does not focus on onlyone interface area but considers all of the three areas in associationwith task scenarios. Fig. 8 illustrates that each subtask or its oper-ation steps have their corresponding interface features and evalu-ation questions.

The other three types of checklists related, respectively, to oneof three interface areas addresses some features of each interfacearea, which are judged to be task-independent. Strictly speaking,there is no interface feature that can be completely evaluated with-out considering task situations that users can experience. There-fore task-independent interface features should be interpretedhere as the meaning that they can be reasonably evaluated onlywith evaluators’ knowledge and experience on user interfacesand tasks. One example of interface features regarding LUI is word-ing or labelling. This interface feature, which is highly related tounderstandability, can be evaluated without actually conductinga task. Other examples regarding GUI include font and iconfeatures.

4.2. Quantification of usability

Although there is much controversy on the usefulness and thevalidity of quantified value of usability, many usability practitio-ners in the industry actually find several situations where usabilityquantification could be useful, including benchmarking and designimprovement (Sauro and Kindlund, 2005). For this reason, wepropose a three-step quantification approach (Fig. 9). Firstly, theevaluation items of all the checklists need to be measured byfive-point semantic grades like Likert-scale. Secondly, these

Task Task step Corresponding area Related phone component Checking question

GUI input field, label, cursor Is the cue for entering search words clear?GUI label, UI style, highlighting Is the cue for connecting to 'calling' clear?

GUI scroll Can the users guess the end of scrolling from the location information on contact list?

LUIscreen of contact list, screen of detailed information, screen

of editing

When a contact information is selected, is it possible to directly call by pressing 'sending' key?

GUI checkbox, highlighting, different colour coding

Are there several focusing points within a screen, which can hamper users' attention?

LUI soft keyDo the soft keys on the contact list have a coherentmapping with hypothetical mental model of users?

Making a call from contact

listSearching a number

Fig. 8. Example of task-based checklist.

Usability level

Each grade is transformed a quantified value

Quantified value

Each evaluation item is organized in terms of its related usability indicators

Single usability value of a mobile phone


Support of cognitive interactionSupport of

efficient interaction



The scores of each usability indicator are aggregated

Fig. 9. Usability quantification process.


measured usability grades are transformed into a quantified valuein a way that the fourth grade (there is no problem in terms ofusability) lies at around three-fourths (75%) of the highest grade.Thirdly, all the quantified usability values assigned to each evalu-ation item should be organized by five usability indicators so thateach usability indicator can be compared each other. The values ofeach indicator are again integrated to yield a final single usabilityvalue of a mobile phone.

4.3. Evaluation process using the framework

Fig. 10 summarizes the overall evaluation process using theproposed framework. In the first phase, planning evaluation, threepoints should be determined: the purposes of evaluation, mobile

phones to be evaluated, and places where evaluation activities willbe carried out. The scope of evaluation is decided in the next phase.From this scope, one can identify which part of four sets of check-lists should be used for the evaluation. In the third phase, oneshould conduct evaluation by using the checklists and the quanti-fication method. Evaluation sheets using Microsoft Excel are pro-vided to support the use of the checklists. Lastly, based on theresults of the third phase, rigorous analysis of the causes of usabil-ity problems and generation of better design solutions should befollowed.

5. Evaluation and improvement of the framework

In the field of HCI, as Sommerville and Dewsbury (2007)pointed out, it is practically unrealistic to conduct comparativeevaluations of any design methods and frameworks. This problemholds for usability evaluation methods and frameworks as well.Particularly if they are developed to support usability evaluationprocess, it could be more difficult to prove the effectiveness ofnew methods and frameworks quantitatively. However it wouldbe possible to examine the effectiveness of our framework in termsof quantitative measures such as the number of identified usabilityproblems. But, as Molich et al. (2004) pointed out, the nature ofidentified usability problems and the linking between usabilityproblems and their relevant design features would be more impor-tant factor to assess the value of evaluation methods. Additionally,a sort of special knowledge and skills are sometimes needed forusability evaluation of mobile phones, which makes it meaninglessto conduct evaluations with students as participants. For these rea-sons, we instead conducted a small case study to examine qualita-tively the usefulness of the framework – how useful they are toidentify and understand usability problems. Another purpose ofthe case study was to identify the points that need to be improvedto make them easier to use, based on the opinions of participants inthe case study.

5.1. Method and results of case study

Eight usability practitioners working for a mobile phone com-pany in Korea were recruited for this case study. The criterion ofrecruiting them was their expertise and experience in the mobilephone industry because main users of the framework will beusability experts. All of them had industrial experience of no lessthan three years and have been involved in designing and evaluat-ing the user interfaces of mobile phones in the company.

In this case study, two mobile phones were evaluated; the firstphone is manufactured in the company and the second phone inanother competing company. The two phones are very similar interms of their functionalities. The usability practitioners

Phase 1Planning the Evaluation

Defining the evaluation purposeSelecting the target phone, target area, and target users

Phase 2Preparing the Evaluation

Defining the evaluation items: Identifying the scope of evaluation

PUI/GUI/LUI/Task-based checklist Select items

Phase 3Conducting the Evaluation

Evaluating the selected items: Use of checklist and evaluation sheet prepared

Quantification:Calculating UI Score based on the evaluation

Phase 4Use of the Evaluation Results

Diagnosing usability issuesDefining areas to be improvedEstablishing action plan for improvement

Fig. 10. Evaluation process using the framework.


sometimes need to be involved in benchmarking mobile phonesmanufactured in competing companies. The proposed evaluationframework can also be used for the purpose of benchmarking.

As an experimental task, ‘taking a picture’ task and its relatedoperations were carried out. The task-based checklist for this taskwas used, and PUI-based checklist was additionally used for eval-uating the PUI of the phones. Regarding the task context and oper-ations, we used the task scenario developed on the basis of usermanual and interview with users. It took, on average, one hourfor a participant to do this evaluation.

Fig. 11 shows some examples of usability issues found with theuse of task-based checklist. Each usability issue is categorized intoone of three status groups: immediate redesign, inferior, and supe-rior. An issue judged as immediate redesign needs immediate ac-tion to improve or modify its relevant design areas, which arealso shown in the figure. In the case of inferior status, a usabilityissue is not a critical problem but needs to be improved to securebetter usability. An issue classified as superior status does not haveany usability problem and thus can be considered one of advanta-geous features. Additionally, each issue has a relationship with ausability criteria and a usability indicator in terms of the hierarchi-cal model of usability factors. Such an explicit relationship be-

Phone Model Status Issues from Checklist items Con

Immediate redesign Photo grip is unstable All the ta

taking a

Immediate redesign

Lens is hidden when operating hot keys

Taking astyle p

Immediate redesign

Side key should be used when taking a landscape style photo

Takinlandsca

Immediate redesign

Unable to see preview after recording video

Recordinand revie

Inferior Unable to preview on screen when taking a self photo Taking a s

Superior Change between photo modes is clearly inidcated

Change photo m

Immediate redesign Inconsistent menu for mode change Change

photo m

Immediate redesign

The direction of navigation keys and the moving of top menu icons are

different

Moving btop menusing na

Inferior Unnatural mapping between menu items and soft keys in preview Using so

The first phone

Fig. 11. Examples of usability issues

tween different abstraction levels of usability factors and issuescould help understand how and why a usability issue impacts onthe usability.

All of the participants in the case study agreed that the pro-posed evaluation framework could be a useful tool for evaluatingthe usability of a mobile phone and for comparing it with alter-native phones. Additionally, they gave several opinions thathelped us improve the contents and formats of the checklistsand the measurement method. Examples of their opinions areas follows:

– There are multiple and redundant items evaluating samefeatures.

– There are terminologies difficult to be understood.– Positive and negative questions are mixed, which makes it con-

fusing to use the checklists.– There are evaluation items that are too abstract to be directly

measured.– Evaluation sheet should contain some space where testers can

describe exceptional things.– There are ambiguous criteria for giving a usability score for an

evaluation item.

text Related Design Areas Related Criteria Related Indicator

sks for picture

Position and size of keys for taking a

picture

Comfortability of physical device Ergonomic support

portrait hoto

Position of keys for taking a picture


g a pe style

Mapping of key functions


g video w before

Addition of functions required

Provision of functions users need

Functional support of user needs

elf photo Addition of functions required

Provision of functions users need


between odes -

Change to another task when doing a

task


between odes Mode change Consistency of task

execution flowsSupport of efficient

interactionetween u items vigation

Menu icons Provision of visual cues


ft menu Soft keys Provision of visual cues


found by task-based evaluation.


From these opinions and the observation results during the casestudy, we identified five points to improve the proposed frame-work, which include enhancing the consistency of evaluation andrefining the terminologies used in the checklists.

5.2. Improvement of the framework

The first point of improvement that we focused on was how tomake evaluators identify as many common usability problems aspossible. As is usual with usability evaluation, there may be a dif-ference in evaluators’ ability and attitude of identifying usabilityproblems and judging the degree of their severity (Hertzum andJacobsen, 2001; Molich et al., 2004). Such a difference sometimescan degrade the validity of evaluation results. In this regard, theproblem of how to minimize evaluators’ difference in using theframework needs to be addressed to enhance its effectiveness. Va-gue evaluation criterion is one of main causes resulting in the dif-ference of evaluators’ judgement (Molich et al., 2004). If theabstraction level of a checklist item is high, evaluation criteria forthe item could be more ambiguous. Thus we identified the check-list items of which abstraction level is considered too high at thelevel of usability criteria and made the meaning of these itemsclearer and more detailed. Additionally, we believe that classifica-tion of usability issues into one of three groups in terms of theirseverity level, as shown in Fig. 11, is helpful to minimize the differ-ence between evaluators’ judgment.

Secondly, we improved the way of evaluating a checklist itemand giving a score-change from one-step measurement to three-step grading. A checklist item should be evaluated in collectiveconsideration of three criteria: whether there is a function relatedto a checklist item, whether the checklist item has usability prob-lems or not, and how critical the problematic item is. Before theimprovement, evaluators had to think of three criteria simulta-neously, which could make it difficult for them to make a judgmentreasonably. For this reason, four of eight participants pointed outthe need for improving the way of giving a score. Thus we changedit into three-step grading method. Each of three steps is concernedwith one of the three evaluation criteria above. The improvedmethod thus guides evaluators to measure a checklist item stepby step. We believe that it can lead them to measure an item ina more logical way.

Thirdly, we changed unclear terminologies and format of thechecklists and devised an evaluation sheet where evaluators canwrite their judgment process, detailed description of usability is-sues, and so on. To make it easier to use the checklists, we addedmore detailed examples into the checklists. We reflected the im-proved method of giving a score into the format of the evaluationsheet as well.

After finishing these improvement works, we informally in-quired to usability practitioners, some of which were participantsin the case study, about the usefulness of the framework and thechecklists. Their opinions were positive, and the participantsagreed that much improvement was made.

6. Concluding remarks

This study developed an evaluation framework for supportingusability practitioners to test the usability of mobile phones inan analytical way, which involves a hierarchical model of usabilityfactors, four sets of checklists, a quantification method, and anevaluation process. To develop the framework, we derived eightdesirable features of an evaluation framework on the basis of com-prehensive literature review and the opinions of usability testers inthe real industry. These features served as a reference point fordeveloping the framework and the checklists, as well as improving

their quality. With regard to how we addressed these features inthis study, we can summarise as follows.

– Fact-based approach: The analysis results of the pre-reported138 usability problems were much reflected in selecting andorganizing usability criteria and identifying the relationshipsbetween usability indicators and usability criteria.

– Modularization: The framework support not only task-basedevaluation but also task-independent (interface-based) evalua-tion, providing four sets of checklists (task-based checklist,LUI-based checklist, GUI-based checklist, PUI-based checklist).Such distinctive checklists make it possible to evaluate someinterface design features, independently of other features or toevaluate them altogether with certain tasks conducted.

– Hierarchical approach: The hierarchical model of usability factorsis an essential component of the framework. We developed thismodel in order that it can organize a range of usability factors interms of their abstraction levels. This model supports usabilitypractitioners in assessing and diagnosing the usability problemsof mobile phones in a hierarchical way.

– Optimization: Combined consideration of the modularization ofthe framework and the first and second phases in Fig. 10 helpsdetermine an optimized way of evaluating the usability ofmobile phones under specific conditions.

– User-oriented: The case study provided us with various opinionsand data from usability practitioners, though it was a smallstudy. They allowed us to correct and refine the deficiencies ofthe framework, thereby making it more usable from the view-point of usability practitioners.

– Implementable: The philosophy and principles underlying theframework and the hierarchical model of usability factors wererealized into the form of four sets of checklists and the quantifi-cation method, which can be easily used.

– Context-based: Task-based checklist for task-based evaluationwas developed to help evaluators consider task contexts, thoughit could not be complete.

– Design-oriented: We believe that the modularization of theframework and the hierarchical model helps usability practitio-ners identify design areas that should be improved, referring tothe usability problems found.

This study has some points to be further studied. Firstly, onewould not agree with the mapping relationships between evalua-tion areas and usability indicators and its consequent evaluationitems of the checklists. As described previously, it is ideal to testall the possible relationships between evaluation areas and usabil-ity indicators; however, it might be difficult in actual usability test-ing situations. Accordingly it is necessary to select a partial set ofall the relationships which are considered to be important to revealcritical usability problems. Such a selection may be dependent onseveral things such as company’s policy, designers’ experience,and customers’ opinions. Thus our suggested mapping relation-ships should be subject to change. But we need to establish someprinciples or guidelines that help usability practitioners selectmapping relationships in consideration of their testing situations.Secondly, the goal-means relationships between adjacent levelsin the hierarchical model need to be further examined. We definedthe goal-means relationships in an analytical and deductive way;each of them has not been examined yet empirically. Lastly, theframework and the checklists were developed to support mainlyexpert evaluation and benchmarking. However, it would be inter-esting whether they can be used for other design activities duringdesign life cycle. Particularly task-based evaluation and LUI-basedevaluation may be used to address the problem of how to integratesoftware engineering activities and usability engineering activities,which is one of critical issues in the industry.


References

Bautsch, H., Granger, J., Karnjate, T., Kahn, F., Leveston, Z., Niehus, G., Ward, T., 2001.An Investigation of Mobile Phone Use: A Socio-Technical Approach. Socio-Technical Systems in Industry, Summer Session 2001. Department of IndustrialEngineering, University of Wisconsin, Madison.

Beck, E., Christiansen, M., Kjeldskov, J., Kolbe, N., Stage, J., 2003. Experimentalevaluation of techniques for usability testing of mobile devices in a laboratorysetting. In: Proceedings of the International Conference of the AustralianComputer–Human Interaction Special Interest Group, Brisbane, Australia.

Bevan, N., 1999. Quality in use: meeting user needs for quality. Journal of Systemsand Software 49 (1), 89–96.

Blandford, A. E., Hyde, J. K., Connell, I., Green, T. R. G., 2004. Scoping analyticalusability evaluation methods: a case study. Technical report of UniversityCollege Interaction Centre (<http://www.uclic.ucl.ac.uk/annb/CASSM>).

Carliner, S., 2003. Physical, cognitive, and affective: A three-part framework forinformation design. In: Albers, M., Mazur, B. (Eds.), Content and Complexity:Information Design in Technical Communication. Lawrence Erlbaum Associates,New Jersey.

Coursaris, C.K., Kim, D.J., 2006. A qualitative review of empirical mobile usabilitystudies. In: Proceedings of the 12th Americas Conference on InformationSystems, Acapulco, Mexico, 2006.

Coursaris, C.K., Kim, D.J., 2007. A research agenda for mobile usability. In:Proceedings of the ACM CHI 2007 Conference on Human Factors inComputing Systems, San Jose, USA, 2007, pp. 2345–2350.

Dillon, A., 1999. TIME – a multi-levelled framework for evaluating and designingdigital libraries. International Journal of Digital Libraries 2 (2–3), 170–177.

Dillon, A., 2001. Beyond usability: process, outcome and affect in human–computerinteractions. Canadian Journal of Information Science 26 (4), 57–69.

Donyaee, M., Seffah, A., Kline, R., Striva, J., 2002. An integrated framework forusability measurement. In: Proceedings of the 12th Conference on SoftwareQuality, Ottawa, Canada, 2002, pp. 1–10.

Duh, H.B.-L., Tan, G.C.B., Chen, V.H.-h., 2006. Usability evaluation for mobile device:a comparison of laboratory and field tests. In: Proceedings of the 8th Conferenceon Human–Computer Interaction with Mobile Devices and Services, Helsinki,Finland, 2006, pp. 181–186.

Folmer, E., Van Gurp, J., Bosch, J., 2003. A framework for capturing the relationshipbetween usability and software architecture. Software Process Improvementand Practice 8 (2), 67–87.

Folmer, E., Bosch, J., 2004. Architecting for usability: a survey. Journal of Systemsand Software 70 (1–2), 61–78.

Frøkj�er, E., Hertzum, M., Hornb�ek, K., 2000. Measuring usability: areeffectiveness, efficiency, and satisfaction really correlated? In: Proceedings ofthe ACM CHI 2000 Conference on Human Factors in Computing Systems, TheHague, The Netherlands, 2000, pp. 345–352.

Green, T., Blackwell, A., 1998. Cognitive dimensions of information artefacts: atutorial. <http://www.cl.cam.ac.uk/~afb21> (retrieved 30.12.2008).

Hagen, P., Robertson, T., Kan, M., Sadler, K., 2005. Emerging research methods forunderstanding mobile technology use. In: Proceedings of 2006 AnnualConference of the Australian Computer–Human Interaction Special InterestGroup (OZCHI 2005), Canbera, Australia, 2005, pp. 1–10.

Ham, D.-H., Heo, J., Fossick, P., Wong, W., Park, S., Song, C., Bradley, M., 2008.Framework and models of usability factors of mobile phones. In: Lumsden, J.(Ed.), Handbook of Research on User Interface Design and Evaluation for MobileTechnology. IDEA Group.

Hartson, H.R., 2003. Cognitive, physical, sensory, and functional affordances ininteraction design. Behaviour & Information Technology 22 (5), 315–338.

Hartson, H.R., Andre, T.S., Williges, R.C., 2003. Criteria for evaluating usabilityevaluation methods. International Journal of Human–Computer Interaction 15(1), 145–181.

Hertzum, M., Jacobsen, N.E., 2001. The evaluator effect: a chilling fact aboutusability evaluation methods. International Journal of Human–ComputerInteraction 13 (4), 421–443.

Hix, D., Hartson, H.R., 1993. Developing User Interfaces: Ensuring Usability throughProduct and Process. John Wiley & Sons, New York, USA.

Holtzblatt, K., 2005. Designing for the mobile device. experiences, challenges, andmethods. Communication of the ACM 48 (7), 33–35.

Hornb�k, K., 2006. Current practice in measuring usability: challenges to usabilitystudy and research. International Journal of Human–Computer Studies 64 (2),79–102.

ISO/IEC 9126-1, 2001. Software engineering – software product quality – Part 1:quality model, International Organization for Standardization.

ISO/IEC 9241-11, 1998. Ergonomic requirements for office work with visual displayterminal – Part 11: guidance on usability, International Organization forStandardization.

Ivory, M., Hearst, M., 2001. The state of the art in automating usability evaluation ofuser interfaces. ACM Computing Survey 33 (4), 470–516.

Jokela, T., 2004. When good things happen to bad products: where are thebenefits of usability in the consumer appliance market? Interactions 11 (6), 29–35.

Jones, M., Marsden, G., 2006. Mobile Interaction Design. John Wiley & Sons, UK.Kadyte, V., Tétard, F., 2004. The role of usability evaluation and usability testing

techniques in the development of a mobile system. In: Proceedings of theWorkshop on Improving the Interplay between Usability Evaluation and UserInterface Design, NordiCHI 2004, Aalborg University, Denmark, 2004.

Kangas, E., Kinnunen, T., 2005. User-centred design to mobile applicationdevelopment. Communication of the ACM 48 (7), 55–59.

Karat, J., 1997. User-centred software evaluation methodologies. In: Helander, M.G.,Landauer, T.K., Prabhu, P.V. (Eds.), Handbook of Human–Computer Interaction.Elsevier Science, The Netherlands.

Keinonen, T., 1998. One-dimensional usability: Influence of usability on consumers’product preference. University of Art and Design Helsinki, UIAH A21.

Ketola, P., Röykkee, M., 2001. The three facets of usability in mobile handsets. In:Proceedings of the Workshop of ACM Conference on Human Factors inComputing Systems (ACM CHI 2001), Seattle, USA, 2001.

Kiljander, H., 2004. Evolution and usability of mobile phone interactionstyles. Unpublished Doctoral Dissertation, Helsinki University of Technology,Finland.

Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S., Davies, J., 2005.Evaluating the usability of a mobile guide: the influence of location, participantsand resources. Behaviour & Information Technology 24 (1), 51–65.

Klockar, T., Carr, D.A., Hedman, A., Johansson, T., Bengtsson, F., 2003. Usability ofmobile phones. In: Proceedings of the 19th International Symposium on HumanFactors in Telecommunication, Berlin, Germany, 2003, pp. 197–204.

Kwahk, J., Han, S.H., 2002. A methodology for evaluating the usability of audiovisualconsumer electronic products. Applied Ergonomics 33 (5), 419–431.

Lavery, D., Cockton, G., Atkinson, M., 1997. Comparison of evaluation methods usingstructured usability problem reports. Behaviour & Information Technology 16(4–5), 246–266.

Lee, K. B., Grice, R.A., 2004. Developing a new usability testing method for mobiledevices. In: Proceedings of the International Professional CommunicationConferenceI, Piscataway, USA, 2004, pp. 115–127.

Lee, Y.S., Hong, S.W., Smith-Jackson, T.L., Nussbaum, M.A., Tomioka, K., 2006.Systematic evaluation methodology for cell phone user interfaces. Interactingwith Computers 18 (2), 304–325.

Lindholm, C., Keinonen, T., Kiljander, H., 2003. Mobile Usability: How NokiaChanged the Face of the Mobile Phone. McGraw-Hill, New York, USA.

Ling, C., Hwang, W., Salvendy, G., 2006a. A survey of what customers want in a cellphone design. Behaviour & Information Technology 26 (2), 149–163.

Ling, C., Hwang, W., Salvendy, G., 2006b. Diversified users’ satisfaction withadvanced mobile phone features. Universal Access in the Information Society 5(2), 239–249.

Love, S., 2005. Understanding Mobile Human–Computer Interaction. Elsevier Ltd.,Oxford, UK.

Molich, R., Ede, M.R., Kaasgaard, K., Karyukin, B., 2004. Comparative usabilityevaluation. Behaviour & Information Technology 23 (1), 65–74.

Nielsen, J., 1993. Usability Engineering. Academic Press, Oxford, UK.Peuple, J.L., Scane, R., 2004. User Interface Design. Crucial, Exeter, UK.Preece, J., Rogers, Y., Sharp, H., 2002. Interaction Design: Beyond Human–Computer

Interaction. John Wiley & Sons. New Jersey, USA.Rouse, W.B., 1991. Design for Success: A Human–Centered Approach to Designing

Successful Products and Systems. John Wiley & Sons, New York, USA.Saffer, D., 2007. Designing for Interaction: Creating Smart Applications and Clever

Devices. New Riders, Berkeley, USA.Sauro, J., Kindlund, E., 2005. A method to standardize usability metrics into a single

score, In: Proceedings of the ACM CHI 2000 Conference on Human Factors inComputing Systems, Portland, Oregon, 2005, pp. 401–409.

Scholtz, J., 2006. Metrics for evaluating human information interaction systems.Interacting with Computers 18 (4), 507–527.

Sommerville, I., Dewsbury, G., 2007. Dependable domestic systems design: A socio-technical approach. Interacting with Computers 19 (4), 438–456.

Te’eni, D., Carey, J., Zhang, P., 2007. Human–Computer Interaction. John Wiley &Sons, New Jersey, USA.

Treu, S., 1994. User Interface Evaluation: A Structured Approach. Plenum Press, NewYork, USA.

Tullis, T., Albert, B., 2008. Measuring the User Experience. Morgan Kaufmann,Burlington, USA.

Weiss, S., 2002. Handheld Usability. John Wiley & Sons, Chichester, UK.Zhang, Z., 2003. Overview of usability evaluation methods. <http://

www.usabilityhome.com> (retrieved 30.12.2005).Zhang, D., Adipat, B., 2005. Challenges, methodologies, and issues in the usability

testing of mobile applications. International Journal of Human–ComputerInteraction 18 (3), 293–308.

http://www.uclic.ucl.ac.uk/annb/CASSM

http://www.cl.cam.ac.uk/~afb21

http://www.usabilityhome.com

http://www.usabilityhome.com

a framework for evaluating the usability of mobile phones based on multi-level, hierarchical model...

Documents