automating assessment of web site usability

Automating Assessment of

Web Site Usability

Marti Hearst

Melody Ivory

Rashmi Sinha

University of California, Berkeley

ASIS IA Summit, Feb 2001

The Usability Gap

196M new Web sites in the next 5 years [Nielsen99]

~20,000 user interface professionals [Nielson99]


The Usability Gap

Most sites have inadequate usability [Forrester, Spool, Hurst]

(users can’t find what they want 39-66% of the time)

196M new Web sites in the next 5 years [Nielsen99]

A shortage of user interface professionals [Nielson99]


The Problem

NON-professionals need to create websites

Guidelines are helpful, but Sometimes imprecise Sometimes conflict Usually not empirically founded


Ultimate Goal: Tools to Help Non-Professional Designers

Examples: A “grammar checker” to assess guideline

conformance Imperfect Only suggestions – not dogma

Automatic comparison to highly usable pages/sites

Automatic template suggestions


A View of Web Site Structure (Newman et al. 00)

Information design structure, categories of

information

Navigation design interaction with

information structure

Graphic design visual presentation of

information and navigation (color, typography, etc.)

Courtesy of Mark Newman


Information Architecture includes management

and more responsibility for content

User Interface Design includes testing and

evaluation

A View of Web Site Design(Newman et al. 00)

Courtesy of Mark Newman


The Goal

Eventually want to assess navigation structure and graphic design at the page and site level.

Farther down the line: information design and scent

Note: we are NOT suggesting we can characterize: Aesthetics Subjective preferences


The Investigation

Can we place web design guidelines onto an empirical foundation?

Can we build models of good design by looking at existing designs?


Example Empirical Investigation

Is it all about the content?


Webby Awards 2000

6 criteria 27 categories

We used finance, education, community, living, health, services

100 judges International Academy of Digital Arts & Sciences 3 rounds of judging

2000 sites initially


Webby Awards 2000 6 criteria

1. Content2. Structure & navigation3. Visual design4. Functionality5. Interactivity6. Overall experience

Scale: 1-10 (highest) Nearly normally distributed across judged sites What are Webby judgements about?


Webby Awards 2000 The best predictor of the overall score is

the score for content The worst predictor is visual design


So … Webbys focus on content!


Comparing Two Categories

news

arts


Guidelines

There are MANY usability guidelines A survey of 21 sets of web guidelines

found little overlap (Ratner et al. 96) Why?

Our hypothesis: not empirically validated So … let’s figure out what works!


Web Page Metrics

Web metric analysis tools report on what is easy to measure Predicted download time Depth/breadth of site

We want to worry about Content User goals/tasks

We also want to compare alternative designs.


Another Empirical Study:

Which features distinguish well-designed web pages?


Quantitative Metrics

Identified 42 attributes from the literature

Roughly characterized: Page Composition (e.g., words, links, images) Page Formatting (e.g., fonts, lists, colors) Overall Page Characteristics

(e.g., information & layout quality, download speed)


Metrics Used in Study

Word Count Body Text Percentage Emphasized Body

Text Percentage Text Positioning Count Text Cluster Count

Link Count Page Size Graphic Percentage Graphics Count Color Count Font Count


Data Collection

Collected data for 1898 pages from 163 sites Attempted to collect from 3 levels within each site

Six Webby categories Health, Living, Community, Education, Finance,

Services Data constraints

At least 30 words No pages with forms Exhibit high self-containment (i.e., no style sheets,

scripts, applets, etc.)


Method

Collect metrics from sites evaluated for Webby Awards 2000

Two comparisons Top 33% of sites vs. the rest (using the overall

Webby score) Top 33% of sites vs. bottom 33% (using the Webby

factor) Goal: see if we can use the metrics to predict

membership in top vs. other groups.


Questions:

Can we use the metrics to predict membership in top vs. other groups?

Do we see a difference in how the metrics behave in different content categories?


Findings

We can accurately classify web pages Linear discriminant analysis For top vs. rest

67% correct for overall 73% correct when taking categories into account

For top vs. bottom 65% correct for overall 80% correct using categories


Why does this work?

Content is most important predictor of overall score

BUT there is some predictive power in the visual design / navigation criteria

Also, it may just be that good design is good design all over Film making analogy This happens in other domains – automatic essay

grading for one


Deeper Analysis

Which metrics matter? All played a role

To get more insight: We noticed that small, medium, and large

pages behave differently We subdivided pages according to size and

category to find out which metrics matter and if they should have high or low values


Small pages (66 words on average)

Good pages have slightly more content, smaller page sizes, less graphics and employ more font variations

The smaller page sizes and graphics count suggests faster download times for these pages (corroborated by a download time metric, not discussed in detail here).

Correlations between font count and body text suggest that good pages vary fonts used between header and body text.


Medium pages (230 words on average)

Good pages emphasize less of the body text Text positioning and text cluster count indicate

medium-sized good pages appear to organize text into clusters (e.g., lists and shaded table areas).

Negative correlations between body text and color count suggests that good medium-sized pages use colors to distinguish headers.


Large pages (827 words on average)

Good pages have less body text and more colors (suggesting pages have more headers and text links)

Good pages are larger but have fewer graphics


Future work

Distinguish according to page role Home page vs. content vs. index …

Better metrics Separate info design, nav design, graphic

design Site level as well as page level Compare against results of live user

studies


Future work

Category-based profiles Can use clustering to create profiles of good

and poor sites for each category These can be used to suggest alternative

designs More information: CHI 2001 paper


Ramifications

It is remarkable that such simple metrics predict so well Perhaps good design is good overall There may be other factors

A foundation for a new methodology Empirical, bottom up

But, there is no one path to good design!


In Summary

Automated Usability Assessment should help close the Web Usability Gap

We can empirically distinguish between highly rated web pages and other pages Empirical validation of design guidelines Can build profiles of good vs. poor sites Are validating expert judgements with usability

assessments via a user study Eventually want to build tools to help end-users

assess their designs


More information: http://webtango.berkeley.edu http://www.sims.berkeley.edu/~hearst

automating assessment of web site usability

Documents

foundedasis ia summit

sites initiallyasis

visual designasis ia

years nielsen99asis

web design guidelines

view of web site designnewman

sets of web guidelines

information designstructure