customer experience requirements for e-commerce web sites

Int. J. Web Engineering and Technology, Vol. 3, No. 4, 2007 441

Copyright © 2007 Inderscience Enterprises Ltd.

Customer experience requirements for e-commerce websites

Oscar de Bruijn*, Antonella De Angeli and Alistair Sutcliffe Business Systems Division Manchester Business School The University of Manchester Booth Street West Manchester M15 6PB, UK E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] *Corresponding author

Abstract: With the emergence of highly interactive applications on the World Wide Web has come a realisation that customer engagement is an increasingly important requirements consideration. It is currently not clear, however, what kinds of requirements websites have to adhere to in order to offer the right kinds of customer experience. In this paper, we report on a study in which we explored the relationships between content, presentation and usability of websites and how these relate to the overall satisfaction of the user and their willingness to purchase goods and services. Based on the results we can make recommendations about requirements considerations for content and presentation in the form of trade-offs that need to be considered by requirements-engineers.

Keywords: web requirements; electronic commerce; customer experience; usability; heuristics; aesthetics.

Reference to this paper should be made as follows: de Bruijn, O., De Angeli, A. and Sutcliffe, A. (2007) ‘Customer experience requirements for e-commerce websites’, Int. J. Web Engineering and Technology, Vol. 3, No. 4, pp.441–464.

Biographical notes: Oscar de Bruijn is a Lecturer on human-computer interaction with particular research interests in information visualisation for customer decision making and web navigation.

Antonella De Angeli is a Lecturer on human-computer interaction with particular research interests in customer behaviour, e-commerce web design and social interaction.

Alistair Sutcliffe is a Professor of Interactive Systems Design with particular research interests in multimedia design and design for attention.

442 O. de Bruijn, A. De Angeli and A. Sutcliffe

1 Introduction

With the emergence of highly interactive applications on the World Wide Web (or web for short) has come a realisation that usability is no longer the only, or even the main, user experience requirements consideration (e.g., Bolchini and Paolini, 2004). In the days when the web was used primarily as a source of information it was indeed appropriate to focus on the usability and, more specifically, the navigability of websites. In recent years, however, users’ goals and expectations have changed from utilitarian information-seeking to enjoyment, fun and entertainment (e.g., Blythe et al., 2004; Norman, 2004). Even activities such as online shopping are now expected to be pleasurable experiences in their own right and not merely time- and effort-saving substitutes for shopping in the high street or mall. Thus, it is now also increasingly clear that if businesses are to compete on the web they need to be able to provide experiences that engage the customer (e.g., Chen and Sockel, 2004). It is less clear, however, what the requirements considerations are for building engaging web applications, how engagement relates to users’ overall satisfaction with a website, and how this influences customers’ buying behaviour.

Considering customer engagement as a requirement for e-commerce web design constitutes a departure from the traditional concept of requirements engineering which focuses first of all on content and functionality, then considers usability, and finally pays lip-service to the importance of graphic design (Overmyer, 2000). Indeed, engagement is likely to constitute a trade-off between the user’s motivation to explore a website and costs of this exploration in terms of usability (Sutcliffe, 2002). Moreover, whether a user can be motivated to explore a website depends not only on the content and functionality offered, but equally on the way in which this content and functionality is presented. The requirements for content, functionality, usability and graphical design must therefore be considered with equal importance in website development.

In this paper, we report on a study in which we explored the relationships between the content, presentation and usability of websites and how these relate to the overall satisfaction of the user and their willingness to purchase goods and services. In particular, we tested a number of high level requirements for content, presentation and usability in the form of website design heuristics. The heuristics, which embody the high level requirements, were applied to two websites which were considered similar in purpose and functionality, but offered different kinds of user experiences through differences in the choice of content and styles of presentation. By investigating how the heuristics apply to these websites separately and then asking users to express preferences for one site over the other we were able to assess the relative importance of content, presentation and usability for the relative effectiveness of these e-commerce websites.

2 Related work

It has been previously noted that the requirements considerations for web applications are not necessarily the same as for more traditional interactive applications, or even hypermedia applications (e.g., Bolchini and Paolini, 2004; Ha and James, 1998; Overmyer, 2000; Sanchez, 2003). For example, Sanchez (2003) suggested that the ability

Customer experience requirements for e-commerce websites 443

of a website to not only capture the user’s attention but also to hold it for a period of time extending beyond a single visit is an important consideration in the light of the many competing websites usually available to customers. From this requirement to make websites ‘sticky’ arise a number of design recommendations. These include such issues as frequency of updating contents to make revisiting a worthwhile experience (Ha and James, 1998), creating relationships through reciprocal communication (Chen and Sockel, 2004), and the personalisation of pages based on user-action logs (Bolchini and Paolini, 2004).

Branding has been highlighted as a particularly important requirement for e-commerce websites. However, it is not yet clear what determines customer’s perception of brand image. Merrilees and Fry (2002) propose a structural model of branding which links the perception of brand image to the amount of interactivity included in the website, its usability and the presence of trust-building components such as the completeness of information and use of secure transaction mechanisms. Bolchini and Paolini (2004) offer a taxonomy of requirements for hypermedia intensive web applications which also highlights the importance of the interactivity of a website, together with considerations for content, navigation, and presentation. Sutcliffe (2002) proposed a process model for website effectiveness in which the overall satisfaction with a website depends on its usability, but also on its initial attractiveness and, for e-commerce sites, its power to persuade users to buy. From these studies, it becomes clear that we need to investigate the importance and inter-relationships of three factors underlying the customer experience:

1 usability, which includes navigation

2 content, which includes interactivity and choice of media

3 presentation, which includes graphical design considerations.

When we consider requirements for presentation, the aesthetic quality of a website constitutes an obvious consideration. However, few experimental investigations into the aesthetics of interactive systems have been carried out apart from those by Tractinsky (1997), who demonstrated that users’ perception of aesthetic qualities was an important, and culturally variable, component in rating the quality of Automatic Teller Machine (ATM) user interfaces. Further evidence for the importance of aesthetics can be found in the study by Hassenzahl et al. (2000), who asked users to compare six different designs of a process control application, with questionnaire inventories for experience, hedonic and appeal qualities. Hassenzahl et al. (2000) concluded that both experience and hedonic qualities contributed approximately equally to the overall judgement of appeal. However, these studies either did not specify which design features they varied to test aesthetic quality or varied on simple aspects such as colour and layout consistency as in Tractinsky’s studies.

More recently, Lavie and Tractinsky (2004) found clear evidence for a two-dimensional structure of perceived website aesthetics. The first dimension was characterised by qualities embraced by classical notions of what constitutes aesthetic design, such as ‘clean’, ‘clear’ and ‘symmetrical’. The second dimension was


characterised by qualities that capture the user’s perception of creativity and originality of the site’s design. They labelled these apparently separable dimensions of aesthetics as ‘classical’ and ‘expressive’ aesthetics respectively. Hassenzahl (2004) argued that judgements about the aesthetic quality of MP3 player interfaces are based on three types of considerations; pragmatic ones relating the usability of the site, how much stimulation is provided and how much the site reflects the user’s tastes and attitudes. The latter two can be considered to constitute very similar concepts as those proposed by Lavie and Tractinsky (2004), reflecting qualities of content and presentation.

The validity of separating qualities of perceived aesthetics into ‘classical’ and ‘expressive’ dimensions was tentatively confirmed by Sutcliffe and De Angeli (2005) when they used the heuristics for attractiveness developed by Sutcliffe (2002) to evaluate a traditional static menu-based website and its metaphor-based equivalent which used Flash™ animation. The sites did not differ on users’ judgements on items measuring classical aesthetics, but on items measuring expressive aesthetics the menu-based site scored significantly lower than the metaphor-based site. It could therefore be argued that by including elements contributing to this expressive dimension of aesthetics interacting with e-commerce websites becomes a ‘fun’ activity (cf. Chao, 2004).

In this paper, we aim to extend the findings reported in this section by shedding more light on the relationships between the antecedents of users’ experiences of websites, i.e., usability, content and presentation, in order to develop a more complete picture of the customer requirements considerations for e-commerce websites.

3 Method

Two live e-commerce websites were used in this study, www.hmv.co.uk (HMV; see Figure 1) and www.cd-wow.com (CD-WOW!; see Figure 2), both selling CD-ROM music. The structure and content of the websites remained constant throughout the study, apart from minor changes to details of product listings. The websites were chosen because they both offered comparable utility, but differed in their look and feel. CD-WOW! is superficially more youth oriented in appearance with stronger colours and bolder elements, compared to the more minimalist look and feel of HMV. An initial expert evaluation of the two sites indicated that HMV had a better information layout. In particular, the HMV site was less cluttered. In addition, the dialog with the HMV site seemed to be much more logical and direct. Selecting a particular music CD on the HMV site, for example, resulted in a page with information about just that CD including links to music samples. The same action on the CD-WOW! site resulted in a page that presented mostly other items than the one selected. Information about each item was very basic and did not include music samples. In contrast, however, CD-WOW! had the more strident design in terms of the use of colours and advertising.


Figure 1 The home page of hmv.co.uk as it appeared at the time of this study in November 2004

Figure 2 The home page of cd-wow.com as it appeared at the time of this study in November 2004


3.1 Heuristics questionnaire

The heuristics questionnaire had to be completed twice, once for each website. It contained questions based on the heuristics and, in addition, questions asking users to directly rate the website for attractiveness, usability and persuasiveness. Two sets of heuristics were used, heuristics for attractiveness proposed by Sutcliffe (2002), which distinguish between content and presentation, and heuristics for usability proposed by Perlman (1997). In addition, it contained questions about the representation of brand image on the websites (see Appendix A).

The heuristics included in the questionnaire were originally suggested by Sutcliffe (2002) as a way to measure the perceived attractiveness of websites as a measurable parameter of the users’ motivation to explore a website. In this view, attractiveness can be divided into generic qualities of a user interface, such as aesthetic design and use of media to direct attention, and content related issues of linking visual style, brand image and messages to users’ knowledge of the organisation. The heuristics suggested by Sutcliffe (2002) extended existing advice on website design (Nielsen, 1999; IBM, 2000) and partially formulated principles for aesthetic design (Mullet and Sano, 1995; Kristof and Satran, 1995), based on users’ requirements.

3.2 Comparative questionnaire

The comparative questionnaire consisted of three sections. Users were asked which of the two websites was more effective in Section 1, which was more attractive in Section 2, and which they would use to buy online in Section 3. In each section they were asked to rank a number of factors according to the importance these had on their decision (see Appendix B).

3.3 Participants

One hundred seventy-nine undergraduate students of the School of Informatics, University of Manchester, participated in the experiment as part of their coursework for a course module in Human-Computer Interaction (HCI). In the course module they were given basic knowledge of HCI including heuristics-based evaluation. In addition, they were briefed about the correct interpretation of the heuristics used in the experiment, which removed as far as possible any of their ambiguity. Each participant (P) was required to explore both websites in any order and to complete one heuristics questionnaire for each site before completing the comparative questionnaire. It was anticipated that the order in which the sites were explored would vary more or less randomly between Ps, which ensured that each site was explored before the other approximately equal number of times due to the large sample of subjects.

4 Results

Results are summarised in three main sections. Section 1 reports a psychometric validation of the research instruments, Section 2 concentrates on the inter-site comparison and Section 3 presents a predictive model for the role of attractiveness, usability and persuasiveness in the overall satisfaction with websites.


4.1 Psychometric validation

Responses to the 11 heuristics for attractiveness were subjected to a reliability analysis, to evaluate their internal consistency. Cronbach’s alpha1 values were moderately low (α = .69 for the HMV site and α = .74 for the CD-WOW! site), suggesting that the scale could be composed of different factors, rather than reflecting a single homogeneous dimension. To further investigate the psychometric properties of the scale, a factor analysis was conducted applying the principal-axis factoring method with Varimax rotation2 on the questionnaires collected from each website. On the basis of a scree-plot analysis and factor interpretability, two main factors were extracted from the questionnaires for both sites. These factors accounted for 39% of the total variance in the HMV site and 45% of the variance in the CD-WOW! site. Each factor had an eigenvalue index greater than 1.4.

The rotated factor matrix for both websites is reported in Table 1. The configuration of the factors was very consistent in the two websites, with only one small difference as regards H4 (Depth of field with Layers), which presented low mixed saturations in the HMV website, while in CD-WOW! it clearly loaded on a single factor.

Table 1 Rotated factor matrix for the attractiveness heuristics in both websites

HMV CD-WOW!

Attractiveness heuristics Classical aesthetic

Expressive aesthetic

Classical aesthetic

Expressive aesthetic

H1 Judicious use of colour .506 .740 .201

H2 Symmetry .448 .153 .631 –.197

H3 Style .472 .220 .421 .349

H4 Depth of field with layers .290 .292 .544 .220

H5 Structured and consistent layout .550 .188 .490

H6 Choice of media to attract attention .194 .544 .686

H7 Use of personality and media to attract and persuade

.589 .448

H8 Visibility of identity and organisation image

.622 .114 .601 .101

H9 Design of unusual or challenging images

.129 .479 .248 .548

H10 Matching arousal to users’ mood and motivation

.408 .482

H11 Stimulating users’ interest by secondary motivation

.240 .215 .239

Note: Absolute values less than .10 are not presented.

The interpretation of the factors was based on the semantic content of all the items presenting a saturation superior to .35 on one and only one factor, with the only exception of H4 which was included in the classical aesthetic factor even though it did not meet this criterion in the HMV analysis. The first factor concerned a description of classical aesthetic. It addressed dimensions such as judicious use of colours, symmetry, style and visibility of identity. The second factor reflected the expressive aesthetic dimension. It


included items such as ‘choice of media to attract attention’ and ‘use of personality and media to attract and persuade’. For each participant and website, a classical and an expressive index were computed averaging scores to individual items according to the factor configuration. The result is consistent with the two-dimensional structure of perceived website aesthetics proposed by Lavie and Tractinsky (2004).

A reliability analysis was run on the 13 usability heuristics for both websites. This returned an alpha value of .86 for CD-WOW! with all heuristics showing a corrected item-total correlation superior to .41, reflecting a very high internal consistency of the instrument. Results for HMV yielded an alpha of .77, with heuristic 1 (Help and documentation) showing a low correlation with the rest of the scale (r = .17). Nevertheless, due to results in the analysis of CD-WOW! questionnaires, we decided not to discard the item and computed two overall usability indexes averaging scores to all 13 items.

4.2 Intersites comparison

4.2.1 Heuristics questionnaire

The classical and expressive indexes of aesthetics were entered as dependent variables in a repeated measure Anova with dimension (2) and site (2) as the within-subjects factors. The main effects of dimension and site were highly significant [F(1, 142) = 406.79, p < .001 and F(1, 142) = 67.83, p < .001, respectively], with no interaction effect. As can be seen in Figure 3, the classical dimension scored better than the expressive one, and HMV scored better than CD-WOW!.

Figure 3 This graph shows that the HMV site received higher scores on heuristics contributing to both classical and expressive aesthetics than the CD-WOW! site. Both sites scored higher on classical aesthetics than on expressive aesthetics.

1

2

3

4

5

Classic Expressive

Aesthetics

Rat

ing

HMV

CD-WOW


Descriptive statistics (mean and standard errors) for the individual items of each aesthetic factor in the two experimental conditions are reported in Table 2. The first seven items (labelled C-H) refer to the classical aesthetic dimension. The remaining three items (E-H) refer to the expressive aesthetic dimension. With the exception of E-H9, all heuristic evaluations were found to be significantly better for HMV than for CD-WOW! (p < .001).

Table 2 Means and standard errors for the heuristics for attractiveness

HMV CD-WOW! Attractiveness heuristics Mean S.E Mean S.E

C-H1 Judicious use of colour * 3.84 .07 3.27 .09

C-H2 Symmetry * 4.14 .05 3.82 .07

C-H3 Style * 3.61 .07 2.93 .08 C-H4 Depth of field with layers * 3.59 .07 3.24 . 08 C-H5 Structured and consistent layout * 4.02 .06 3.68 .07

C-H8 Visibility of identity and organisation image * 4.12 .06 3.48 .08

C-H10 Matching content to users’ mood and motivation * 3.95 .06 3.42 .08

E-H6 Choice of media to attract attention * 2.81 .11 1.95 .08

E-H7 Use of personality and media to attract and persuade * 2.50 .09 2.16 .08

E-H9 Design of unusual or challenging images 2.61 .08 2.45 .08

Note: Asterisks indicate heuristics showing a significant difference on a repeated measure t-test (*p < .001).

The better evaluation of HMV with respect to attractiveness, was matched by a better evaluation on usability, t (141) = 7.14, p < .001. The average usability score for the HMV site was 3.6, while CD-WOW! scored 3.2. Descriptive statistics for individual heuristics are summarised in Table 3. All the differences tended to favour HMV over CD-WOW!.

Table 3 Comparison of the scores on the usability heuristics between HMV and CD-WOW!

HMV CD-WOW! Usability heuristics Mean S.E Mean S.E

U1 Help and documentation *** 3.84 .07 3.23 .08 U2 Adopt the user’s viewpoint *** 4.02 .05 3.71 .07 U3 Simple and natural dialogue 3.95 .06 3.78 .06 U4 Design for advancement *** 3.24 .09 2.89 .09 U5 Provide maps and a trail ** 3.18 .08 2.93 .08 U6 Show the user what is (not) possible * 2.91 .07 2.73 .07 U7 Intuitive mappings *** 3.56 .06 3.33 .07 U8 Minimise memory load *** 3.57 .07 3.24 .08 U9 Consistency in the system and to standards *** 4.01 .06 3.68 .07 U10 Feedback * 3.35 .08 3.16 .08 U11 Prevent errors *** 3.81 .07 3.42 .08 U12 Error messages ** 3.49 .08 3.24 .08 U13 Clearly Marked Exits and Error Recovery ** 3.50 .07 3.24 .08

Note: Asterisks indicate heuristics showing a significant difference at a repeated measure t-test (***p < .001; **p < .01; *p < .05).


4.2.2 Comparative questionnaire

The trend emerging so far from the heuristics questionnaire, showing a clear advantage of HMV over CD-WOW!, is reflected in the comparative evaluation questionnaire. The HMV site was judged to be more effective than the CD-WOW! site by 69.8% of Ps and more attractive by 70.4% of Ps (both significant in a binomial test with equal expected probabilities, ps < .01). The HMV site would have been used by 57.5% of Ps to buy online, but this was not significantly different from the percentage of Ps who would have used CD-WOW! (p > .05).

Table 4 Average ranks in order of diminishing importance for factors used to decide upon the most effective, attractive and persuasive of the two websites

Effective Attractive Buy

Factors HMV CD-

WOW! Factors HMV CD-

WOW! Factors HMV CD-

WOW!

Ease of use 1 2.96

1 2.41

Colours 1 2.50

1 (2.62)

Ease of use 1 4.44

3 4.82

Content 2 3.45

2 3.49

Content 2 2.50

3 (3.26)

Security/ Reputation

2 4.78

5 6.07

Security/ Reputation

3 3.83

4 4.68

Graphics 3 3.29

2 (2.70)

Range 3 5.63

4 5.18

Beauty 4 4.93

5 4.96

Functions 4 3.53

4 (3.91)

Brand 4 5.79

11 9.18

Price 5 5.03

3 3.55

Sound 5 5.75

8 (6.18)

Discounts 5 6.09

2 4.18

Brand 6 5.34

8 6.82

Animation 6 5.90

5 (5.40)

Price 6 6.13

1 2.48

Functions 7 5.56

6 5.05

Ads 7 6.24

6 (5.96)

Search 7 6.20

6 6.08

Help 8 6.25

7 6.44

Video 8 6.29

7 (5.97)

Quality 8 6.66

7 7.36

It is cool 9 7.64

9 7.60

Beauty 9 7.33

5 7.76

Readability 10 7.50

8 7.73

Download time

11 9.33

10 8.51

Feedback 12 9.66

12 10.03

Ads 13 11.46

13 11.60

Participants were asked to explain their preference by ranking a number of factors in order of descending importance. We calculated the average rank of each factor separately for Ps who preferred HMV and those who preferred CD-WOW!. The results are displayed in Table 4. The results suggest that Ps who found HMV more effective, based their choice on more or less the same qualities of the sites as did Ps who found


CD-WOW! more attractive. The same was the case for their choices of the more attractive site. Important factors for deciding on the more effective of the two websites were their ease of use, content, and reputation/security, whereas the least important factors were their coolness, the availability of help, and their functionality. Important factors for deciding which site was more attractive were their colours, content and graphics. Least important were the presence of advertisements and video. The unimportance of video is obvious when it is considered that neither site included any video.

Some striking differences were found in the reasons given for deciding which site to use to buy CDs from. Important factors for deciding to buy from the HMV site were ease of use, reputation and the range of items offered, whereas deciding to buy from CD-WOW! was based primarily on price and discounts. Notice that brand image was moderately important for people who would buy from HMV, but absolutely unimportant for those who would buy from CD-WOW!.

4.3 Predictive model

The relationships between the three judgements on the comparative questionnaire were assessed by calculating their correlations using Pearson’s Product Moment Correlation statistic. We found that when a website was judged to be more effective, it was also likely to be judged more attractive and to be used to buy music CDs (r = .40 and .49 respectively, ps < .01). Relative judgements of attractiveness and use to buy were only weakly related (r = .16, p < .05), and partial correlations confirmed that this relationship was entirely mediated by their mutual relationships with effectiveness judgements (r = .05, n.s.).

The differences in the scores on the heuristics for attractiveness and usability between HMV and CD-WOW! were computed. These difference scores were then entered as covariates in a binary logistic regression in order to predict Ps preferences in the Comparative Questionnaire. 70% of all the cases were randomly selected for the model building phase of the regression analysis using both Forward(Likelihood Ratio) and Backward(Likelihood Ratio) methods. It was determined that both methods yielded the same results and only those obtained using the Forward(LR) method will be reported here. Once the model was determined, it was used to separately predict the outcome of the selected and unselected cases. The results are shown in Table 5.

It appears that there is a relationship between judgements of effectiveness and attractiveness, as both can be predicted by heuristic H8, and H4, H10 and H1 which were classified in Section 4.1 as contributing to the same dimension addressing classical aesthetics. The heuristic that appears to be predictive for a website’s persuasiveness, H6, was classified as contributing to the other dimension (expressive aesthetic). It has to be noted, however, that it has a very low predictive power. With regards to the usability heuristics, it appears that they have some predictive power only with respect to the effectiveness judgement, as they explain very little of the variance in attractiveness judgement and willingness to buy. Most of the variance in the judgement of website effectiveness is explained by the usability heuristics dealing with the dialog aspect of interacting with a site (U3 and U1). This included the provision of relevant information for actions.


Table 5 Results of using scores on the heuristics for attractiveness and usability to predict users’ choices of the most effective, attractive and persuasive of the two websites

Dependent variable Effective Attractive Persuade to buy

Attractiveness heuristics

dH8 dH8 dH6

dH4 dH1

dH10

R-square .51 .47 .13

Predicted selected cases (%) 82.1 84.2 68.4

Predicted unselected cases (%) 76.6 78.7 61.7

Usability heuristics

dU3 dU7 dU13

dU1 dU7

dU7

R-square 0.35 0.08 0.15

Predicted selected cases (%) 73.1 70.4 65.7

Predicted unselected cases (%) 76.5 73.5 67.6

Notes: The heuristics with a significant amount of predictive power are listed in bold (e.g., dH8 stands for the difference in the scores on heuristic 8 for the two websites). R-square is a measure of the amount of variance explained by the heuristics.

5 Discussion

We found clear evidence for strong relationships between the overall effectiveness of a website and its attractiveness, usability and persuasiveness (Figure 4). Furthermore, we have confirmed the suggestion that aesthetic judgement is mediated by two dimensions corresponding to a classical and an expressive evaluation (cf. Lavie and Tractinsky, 2004). Indeed, it appeared that the relationship between the relative attractiveness of the two sites used in this study and their overall effectiveness was mediated by a clear difference in compliance with principles of classical aesthetics between the sites. The HMV site was deemed to be more aesthetically pleasing based on the use of colours, and consistency of style. The way this relates to site’s effectiveness may lie in the importance of clear visual style and layout in allowing users to quickly determine the quality of the content and decide to explore the site. Further research will have to determine whether that is indeed the case, for example, by tracking users’ eye gaze to see if increasing classical aesthetics allows a more direct and effective scanning of the page.

Looking at the constituent considerations underlying each factor it seems to us that the classical dimension is more related to the look and feel of a website (presentation), whereas the expressive aesthetic involves the ability of a site’s content to induce arousal in a user (content). This may explain why expressive aesthetics appeared not to be a predictive factor in the preference for the two websites (as illustrated by the absence of connections with expressive aesthetics in Figure 3), as the content of both sites was very


similar and neither of the two websites used any media that would increase the user’s state of arousal such as Flash™ or video. Instead they were almost completely static, with no significant visible marks of an individual style of design. Confirmation of this hypothesis was found in the clear difference in the evaluation of the two dimensions, in the order of half a point out of a five-point scale for both sites (see Figure 3). The two sites used in the current study did differ on expressive aesthetics not enough to affect, or be affected by, users’ other judgements of the sites. In contrast, Sutcliffe and De Angeli (2005) found a clear distinction, based on differences in judgements for expressive aesthetics, between two websites with entirely different interaction styles, one classic menu-based site and a dynamic metaphor-based site using Flash™ animation. Again, we need further research to establish the relationships between expressive aesthetics, attractiveness and effectiveness of websites using sites which incorporate a range of media and styles.

Figure 4 A model showing the inter-relationships between the comparative judgements, ratings on heuristics for attractiveness and usability, and ratings on brand image obtained in this study

Finally, we would like to highlight the importance of considering individual differences in users’ judgements for the evaluation of websites. Evidence for their importance can be found in this study when it is considered that different users may express preferences for different sites based on exactly the same considerations. Indeed, heuristics such as ‘judicious use of colour’ may result in entirely different results when it is applied to a website by different users. A purple website may be aesthetically pleasing to some but not to others. Further evidence for the importance of individual differences can be found in the fact that users who said they would buy from CD-WOW! thought price was the most important consideration, while those who would buy at HMV thought that brand and reputation were more important. Unfortunately, the use of only two websites did not allow us to usefully assess the intra-class reliability of the heuristic judgements across participants. Further research using a wider range of websites will have to be done in order to validate this aspect of the heuristics scales.

Expressiveaesthetic

Classicaesthetics

Usability

BrandEffectiveness

PersuasivenessAttractiveness


6 Conclusion

From the results of this study it appears that both attractiveness (in content and presentation) and usability are important requirements considerations for e-commerce websites. The fact that the reasons users gave for preferring one site over the other were independent of the actual preference they expressed points to their importance as requirements considerations. That is, the relative effectiveness and attractiveness of websites can be predicted using the heuristics applied in this study. The heuristics have thereby been validated to some extent as constituting important requirements considerations. This is particularly true for the heuristics that contributed to the classical aesthetics factor which particularly relates to graphical design considerations. The heuristics can therefore be used to guide the requirements gathering process by concentrating the efforts of the requirements engineering onto the factors that really make a difference for the effectiveness of the website.

Interesting conclusions can also be attached to the importance of brand image in persuading customers to buy goods and services online. The results of the current study appear to indicate that having and portraying a strong brand image does indeed persuade customers to buy. This was particularly the case with the HMV site, which is backed by a strong brand image that is furthermore linked to brick-and-mortar retail outlets. A strong brand image may instil a notion of trust in the customer (cf. Merrilees and Fry, 2002). However, in the absence of a strong brand image, as is the case with CD-WOW!, customers can still be persuaded to buy by offering discount prices.

At this time, the conclusions attached to this study will have to be used with some degree of caution due to the idiosyncratic nature of the participants used in the investigation. Students are an easily available but somewhat unrepresentative sample of the entire population, which may restrict the generalisability of some of the findings. In particular, the predictive model will need to be further validated using more representative samples in future research. For example, studies show that users’ preferences for e-Commerce design change with age, with younger users attaching more importance to appearance than older users (Lightner, 2003). Nevertheless, our initial assumption that the CD-WOW! site would appeal more to our sample of users than the HMV site was not confirmed by our findings, although one might speculate that the design of the CD-WOW! site was simply not radical enough to appeal to our sample of users based on expressive aesthetics. What this paper does not provide, therefore, is a recipe for e-commerce website design. What we have shown however with this study is that the heuristics used in the evaluation represent genuine requirements considerations for e-commerce website design, each of which represents a trade-off that has to be resolved with respect to the prospective users of the site.

Acknowledgements

We like to thank all those who took part in this study. Special thanks go to Murni Mahmud and Jan Hartmann for their help and contributions and to four anonymous reviewers for their comments on an earlier version of this paper.


References

Blythe, M., Hassenzahl, M. and Wright, P. (2004) ‘More funology: introduction to a special section on funology’, Interactions, Vol. 11, No. 5, p.37.

Bolchini, D. and Paolini, P. (2004) ‘Goal-driven requirements analysis for hypermedia-intensive web applications’, Requirements Engineering, Vol. 9, pp.85–103.

Chao, D.L. (2004) ‘Computer games as interfaces’, Interactions, Vol. 11, No. 5, pp.71–72.

Chen, K. and Sockel, H. (2004) ‘The impact of interactivity on business website visibility’, International Journal of Web Engineering and Technology, Vol. 1, No. 2, pp.202–217.

Cronbach, L.J. (1951) ‘Coefficient alpha and the internal structure of tests’, Psychometrika, Vol. 16, pp.297–333.

Ha, L. and James, E.L. (1998) ‘Interactivity re-examined: a baseline analysis of early business websites’, Journal of Broadcasting and Electronic Media, Vol. 42, pp.457–474.

Hassenzahl, M. (2004) ‘The interplay of beauty, goodness and usability in interactive products’, Human-Computer Interaction, Vol. 19, pp.319–349.

Hassenzahl, M., Platz, A., Burmester, M. and Lehner, K. (2000) ‘Hedonic and ergonomic quality aspects determine a software’s appeal’, in T. Turner, G. Szwillus, M. Czerwinski and F. Paterno (Eds.) CHI 2000: Human Factors Computing Systems, New York: ACM Press, pp.201–208, Conference on Human Factors in Computing Systems, The Hague, 1–6 April.

IBM (2000) ‘Ease of use: design principles’, http://www.ibm.com/ibm/easy/eou_ext.nsf/Publish/6 (accessed 2 October 2005).

Kaiser, H.F. (1958) ‘The varimax criterion for analytic relation in analysis’, Psychometrika, Vol. 23, pp.187–200.

Kristof, R. and Satran, A. (1995) Interactivity by Design: Creating and Communicating with New Media, Mountain View, CA: Adobe Press.

Lavie, T. and Tractinsky, N. (2004) ‘Assessing dimensions of perceived visual aesthetics of websites’, International Journal of Human-Computer Studies, Vol. 60, No. 3, pp.269–298.

Lightner, N.J. (2003) ‘What users want in e-commerce design: effects of age, education and income’, Ergonomics, Vol. 46, pp.153–168.

Merrilees, B. and Fry, M-L. (2002) ‘Corporate branding: a framework for e-retailers’, Corporate Reputation Review, Vol. 5, pp.213–225.

Mullet, K. and Sano, D. (1995) Designing Visual Interfaces: Communication Oriented Techniques, Englewood Cliffs, NJ: SunSoft Press.

Nielsen, J. (1999) Designing Web Usability: The Practice of Simplicity, New Riders Press.

Norman, D.A. (2004) Emotional Design: Why we Love (or Hate) Everyday Things, Cambridge, MA: Basic Books.

Overmyer, S.P. (2000) ‘What’s different about requirements engineering for web sites?’, Requirements Engineering, Vol. 5, pp.62–65.

Perlman, G. (1997) ‘Practical heuristics for usability evaluation’, http://www.acm.org/~perlman/ question.cgi?form=PHUE (accessed 2 October 2005).

Sanchez, M. (2003) ‘Eight ways to sticky sites’, http://www.efuse.com/Plan/sticky-sites.htm (accessed 30 November 2005).

Sutcliffe, A.G. (2002) ‘Heuristic evaluation of website attractiveness and usability’, in C.W. Johnson (Ed.) DSV-IS 2001, Berlin: Springer-Verlag, LNCS 2220, pp.188–198.

Sutcliffe, A.G. and De Angeli, A. (2005) ‘Assessing interaction styles in web user interfaces’, in M.F. Costabile and F. Paterno (Eds.) Human Computer Interaction – INTERACT 2005, Berlin: Springer-Verlag, LNCS 3585.

Tractinsky, N. (1997) ‘Aesthetics and apparent usability: empirically assessing cultural and methodological issues’, in S. Pemberton (Ed.) CHI’97 Human Factors in Computing Systems, New York: ACM Press, pp.115–122, Proceedings of ACM Conference on Computer-Human Interaction, Atlanta, GA, 22–27 May.


Notes

1 Cronbach’s alpha reflects the degree of inter-correlation between the heuristics. If all heuristics measure the same underlying variable then one would expect a high value (i.e., >.80) for Cronbach’s alpha (Cronbach, 1951).

2 Varimax rotation was chosen to simplify the factor space (i.e., fewer factors associated with each variable and fewer variables loading on each factor) such that the result were more interpretable (Kaiser, 1958).


Appendix A

Heuristic Questionnaire

Q1. Please rate the extent to which the design of this website is in accordance with the following heuristics for designing effective websites.

HEURISTICS FOR ATTRACTIVENESS AND CONTENT

ATTRACTIVENESS Poor Fair Avg Good Excellent

1. Judicious use of colour

Is colour balanced?

Are low and fully saturated colours used appropriately?

1 2 3 4 5

2. Symmetry

Is the visual layout symmetrical?

Are the elements arranged to convey balance and consistency?

1 2 3 4 5

3. Style

Are curved and rectangles shapes contrasted in order to provide an attractive visual style?

Are curved shaped used for menus instead of rectangles to convey a different style?

1 2 3 4 5

4. Depth of field with layers

Are images used to illustrate the site’s content?

If so, do they provide a peaceful effect?

Are there images that provide a depth of field (layers)?

1 2 3 4 5

5. Structured and consistent layout

Is the web layout structured and consistent?

Are grids used to structure images and do not exceed a 5:3 to width radio?

1 2 3 4 5

6. Choice of Media to attract attention

Are media elements such as audio, video or speech used?

If so, are these elements used appropriately?

Do media elements increase the users’ attention?

1 2 3 4 5


7. Use of personality and media to attract and persuade

Do media elements have personality to greet and praise with politeness the users’ choices?

Are images or speech of personalities such as celebrities, leaders, etc., used?

1 2 3 4 5

CONTENT Poor Fair Avg Good Excellent

8. Visibility of identity and Organisation Image

Is visual style consistent?

Does visual style reflect the brand image, corporate identity and values?

Is identity of website visible?

If so, does this identity illustrate the corporate values?

1 2 3 4 5

9. Design of unusual or challenging images

Are there any unusual or challenging images on the website?

If so, are they used in an appropriate manner?

1 2 3 4 5

10. Matching arousal to users’ mood and motivation

Does website’s content suit users’ requirements and needs?

Is content clearly stated, in simple language?

Does content provide clear cues on how to find the things?

Does content take into account different target markets?

1 2 3 4 5

11. Stimulating users’ interest by secondary motivation

Is there any extra functionality, apart from the original purpose of the web? (e.g., games, e-mail, etc.)

1 2 3 4 5


Please give an overall rating of this website for:

Poor Fair Avg Good Excellent

Effectiveness 1 2 3 4 5

Overall attractiveness 1 2 3 4 5

Persuade to buy 1 2 3 4 5

Usability and navigation 1 2 3 4 5

Q2. Please rate how clear and easy to understand the ‘Heuristics for Attractiveness and Content’ were and provide some comments to explain your rating.

Poor Fair Average Good Excellent

Comments:

_______________________________________________________________ _______________________________________________________________

Q3. How clear and easy to understand were the examples provided for these Heuristics?

Poor Fair Average Good Excellent

Please give us some comments:

_______________________________________________________________ _______________________________________________________________

Q4. How do you rate the following features for this particular website?

Poor Fair Avg Good Excellent

Help and documentation

Design for use without documentation. Provide easy-to-use task-oriented help.

1 2 3 4 5

Adopt the user’s viewpoint

Speak the user’s language (avoid jargon). Make use of existing knowledge (familiar mental models).

1 2 3 4 5

Simple and natural dialogue

Avoid extraneous information, steps, actions. Information should be in a logical, natural order.

1 2 3 4 5


Design for advancement

Provide shortcuts (quick keys, customisation).

1 2 3 4 5

Provide maps and a trail

Give the user a way to preview where to go, what will happen. Give the user a way to review/return to previous contexts.

1 2 3 4 5

Show the user what is (not) possible

Provide affordances to indicate what can be done.

1 2 3 4 5

Intuitive mappings

Design good response compatibility between controls/actions.

1 2 3 4 5

Minimise memory load

Remove the need to remember across dialogues. Provide multiple views for easy comparisons.

1 2 3 4 5

Consistency in the system and to standards

Make sure the same term/action has one meaning. When there is no better way, conform to a standard.

1 2 3 4 5

Feedback Provide timely feedback about all processes, system status.

1 2 3 4 5

Prevent errors

Make it difficult to make errors.

1 2 3 4 5

Error messages

Diagnose the source and cause of a problem and suggest a solution.

1 2 3 4 5

Clearly marked exits and error recovery

Make sure the user can get out of an undesirable state easily. Design assuming that people will make errors and need to recover previous states.

1 2 3 4 5

Q5. Did you know this website before you do the test?

Yes No


Q6. How attractive did you find this website?

Very Attractive Attractive Neither attractive nor unattractive

Unattractive Very Unattractive

Q7. Why? _______________________________________________________________ _______________________________________________________________

Please, rate in order of importance: (from 1 to 7, 1, being the most important, 7 being the least important).

Colours

Images

Advertisements

Content

Videos

Music

Interactivity

Other: ……………………..

Q8. How clear was the brand represented on this website?

Very Unclear Unclear Clear Very Clear

Q9. Did you know the brand name before you visited this website?

Yes No

Q10. If the previous answer is yes, how did you recognise it? (Tick all that apply)

Logo

Images

Website Colour

Advertisements

Content

Page Title

Web Address

Other: ………………………..


Q11. If you remember the Web address of the site, please write it down:

Q12. Would you buy any product on this site?

Definitely will not buy

Might or might not buy

Probably will buy

Very likely will buy

Definitely will buy

Q13. Would you visit this site again?

Not all likely Very likely

1 2 3 4 5

Q14. Why?

_______________________________________________________________ _______________________________________________________________


Appendix B

Comparative Questionnaire

Q1. Which of the websites that you tested was more effective? (Tick one box only)

HMV CD WOW

Q2. Why?

(Please, rate in order of importance: (from 1 to 9, 1 being the most important, 9 being the least important).

Interface Beauty/Aesthetics

Has a lot of functions

Its cool

Low Prices

Brand Name

Ease of use

Site Content

Help/FAQ

Website reputation/security

Other: …………………………………..

Q3. Which of the websites that you tested was the most attractive? (Tick one box only)

HMV CD WOW

Q4. Why?

Please, rate in order of importance: (from 1 to 8, 1 being the most important, 8 being the least important).

Interface colours

Video

Animation

Sound

Functions

Advertisements

Site Content

Images/Graphics

Other: …………………………………..


Q5. If you were to buy a CD, which of the websites will you use it to buy online? (Tick one box only)

HMV CD WOW

Q6. Why?

Please, rate in order of importance: (from 1 to 13, 1 being the most important, 13 being the least important).

Brand Name

Low Prices

Bargains/Discounts

Site easy to use

Website Aesthetics/Attractiveness

Website reputation/security

Search Facilities

Feedback/Help/FAQ

Wide range of items (CD’s)

Download Time

Advertisements

Information Quality

Content Readability

Other: …………………………………..

Q7. For this website, how important is the brand name for a purchase decision?

Don’t care Not important at all

Somehow important

Very important

Extremely important

customer experience requirements for e-commerce web sites

Documents