A Walk on the Web SideA Walk on the Web Side

Differences and Differences and Similarities Between Web Similarities Between Web and non-Web Application and non-Web Application

TestingTesting

WISQA Special WISQA Special PresentationPresentation

June 11, 2003June 11, 2003

OverviewOverview

Functionality and Reliability testing: Functionality and Reliability testing: making sure a website works as requiredmaking sure a website works as required

Compatibility: making sure a website works Compatibility: making sure a website works as required for the users’ many operating as required for the users’ many operating systems and browserssystems and browsers

Usability: making sure a website works to Usability: making sure a website works to meet users’ goals with effectiveness, meet users’ goals with effectiveness, efficiency, and satisfactionefficiency, and satisfaction

Functionality and Reliability Testing

Functionality & Functionality & Reliability: The Usual Reliability: The Usual

SuspectsSuspects Forms and form validationForms and form validation Error messagesError messages Proofreading the contentProofreading the content Checking calculationsChecking calculations Verifying that new entries are saved Verifying that new entries are saved

correctlycorrectly Verifying that changes are saved correctlyVerifying that changes are saved correctly Verifying that deletions are really deletedVerifying that deletions are really deleted Find the weaknesses in system under testFind the weaknesses in system under test Regression testingRegression testing

Functionality & Functionality & Reliability: Web-wiseReliability: Web-wise

Update releases: when live, users see Update releases: when live, users see instantlyinstantly Navigation: users have choices outside of what Navigation: users have choices outside of what

the site itself offers; page orders aren’t strictthe site itself offers; page orders aren’t strict Browser settings: users can control many things Browser settings: users can control many things

about their browser… how many combinations about their browser… how many combinations will you test? how many directly affect the site?will you test? how many directly affect the site?

Cookies: should the site work without them? Cookies: should the site work without them? Search: what can Search: what can this site’sthis site’s search functions do? search functions do? Are there web-unique functions for the site… like Are there web-unique functions for the site… like

uploading or downloading files? (snapfish.com) uploading or downloading files? (snapfish.com) or sending email? (cnn.com)or sending email? (cnn.com)

Functionality & Functionality & Reliability: Complications Reliability: Complications

of eCommerceof eCommerce Business rules vary from online seller to Business rules vary from online seller to

online seller, so understanding them quickly online seller, so understanding them quickly is vitalis vital

Is registration (saving of name, address, and Is registration (saving of name, address, and possibly credit card number) required before possibly credit card number) required before checkout? Can it be done during or even checkout? Can it be done during or even after checkout?after checkout?

Are shipping and billing addresses saved Are shipping and billing addresses saved separately or together? Can they be edited, separately or together? Can they be edited, or must they be deleted and re-entered?or must they be deleted and re-entered?

What are the rules for sales tax calculation?What are the rules for sales tax calculation? What parts of the site must be secure?What parts of the site must be secure?

Functionality & ReliabilityFunctionality & ReliabilityPlaying with Test CasesPlaying with Test Cases

www.rei.comwww.rei.com

Let’s buy a roof-mounted bike rack for the Let’s buy a roof-mounted bike rack for the car. What test cases do we see along the car. What test cases do we see along the way?way?• NavigationNavigation• SearchSearch• RegistrationRegistration• CheckoutCheckout• Site-specific featuresSite-specific features

Functionality & ReliabilityFunctionality & ReliabilityHandy Tools We UseHandy Tools We Use

Many tools for automated testing of web Many tools for automated testing of web sites exist (see sites exist (see www.testingfaqs.orgwww.testingfaqs.org) )

Defect tracker: TeamTrack by TeamShareDefect tracker: TeamTrack by TeamShare SnagIt by TechSmith, super screen captureSnagIt by TechSmith, super screen capture Realistic Random Data generatorRealistic Random Data generator Microsoft ExcelMicrosoft Excel Paper techniques:Paper techniques:

• Page flows – helps plan workflow casesPage flows – helps plan workflow cases• Page hierarchy diagrams – helps plan navigation Page hierarchy diagrams – helps plan navigation

casescases• Page mockups – identifies page elements Page mockups – identifies page elements

requiring casesrequiring cases

Compatibility Testing

Compatibility TestingCompatibility Testing

It’s a browser jungle out there:It’s a browser jungle out there:• Internet Explorer 6, 5.5, 5, 4…Internet Explorer 6, 5.5, 5, 4…• Netscape Navigator 7, 6.2, 6.1, 6, 4.79, 4.78…Netscape Navigator 7, 6.2, 6.1, 6, 4.79, 4.78…• OperaOpera• MozillaMozilla• SafariSafari• AOLAOL• More! More! www.download.comwww.download.com offers 192 other offers 192 other

browsers!browsers! Don’t forget the operating systems: Don’t forget the operating systems:

Windows (95, 98, 2000, ME, XP, NT), Mac Windows (95, 98, 2000, ME, XP, NT), Mac (OS 9, OS X), Linux…(OS 9, OS X), Linux…

How could you possibly test them all?How could you possibly test them all?

Compatibility TestingCompatibility TestingMaking Good ChoicesMaking Good Choices

First step is to help the customers choose First step is to help the customers choose which OS/browser combinations to supportwhich OS/browser combinations to support• Do they have web logs for a current site?Do they have web logs for a current site?• Do they have a specialized customer base?Do they have a specialized customer base?• Get national statistics, watch the newsGet national statistics, watch the news

Consider support for more users vs. Consider support for more users vs. technology: if you technology: if you mustmust use a technology use a technology (for example, Cascading Style Sheets), who (for example, Cascading Style Sheets), who might this disenfranchise? (And does the might this disenfranchise? (And does the customer care?)customer care?)

Compatibility TestingCompatibility TestingCreating a Test LabCreating a Test Lab

Create a planCreate a plan• What browsers and operating systems do you What browsers and operating systems do you

need?need?• What limitations are there (e.g., can’t install What limitations are there (e.g., can’t install

multiple IE versions on one machine)?multiple IE versions on one machine)?• What hardware do you need? (machines, browsers, What hardware do you need? (machines, browsers,

hubs, KVM switches for monitors & keyboards)hubs, KVM switches for monitors & keyboards)• What software do you need? (browsers, screen What software do you need? (browsers, screen

capture, instant messenger, etc.)capture, instant messenger, etc.)• Try for a modem and phone lineTry for a modem and phone line

Go scavengingGo scavenging• Old PCs useless to company employees are Old PCs useless to company employees are

invaluable to your labinvaluable to your lab

Compatibility TestingCompatibility TestingMuppet LabsMuppet Labs

““Muppet Labs” lives in a corner of our buildingMuppet Labs” lives in a corner of our building• 9 PCs9 PCs• 3 Macs3 Macs• 9 Operating Systems9 Operating Systems• 21 Browsers21 Browsers

Lab area is labeled with useful information:Lab area is labeled with useful information:

IE VersIE Vers Netsc Netsc VerVer

AOL AOL VerVer

OSOS NameName NotesNotes

5.005.00 4.614.61 66 W98W98 KermitKermit scannescannerr

5.55.5 NoneNone 77 W98W98 Miss Miss PiggyPiggy

No No netscnetsc

Usability Testing

First, a few definitions…First, a few definitions…

User InterfaceUser Interface - The language by which a - The language by which a user and system communicate with each user and system communicate with each other.other.

UsabilityUsability - The extent to which a product - The extent to which a product can be used by specified users to achieve can be used by specified users to achieve specified goals with effectiveness, specified goals with effectiveness, efficiency, and satisfaction in a specified efficiency, and satisfaction in a specified context of use.context of use.

(ISO 9241-11 - Ergonomic requirements for office work with visual (ISO 9241-11 - Ergonomic requirements for office work with visual display terminals (VDTs) - guidance on usability)display terminals (VDTs) - guidance on usability)

More definitions…More definitions…

User Centered Design (UCD)User Centered Design (UCD) - any of a wide - any of a wide variety of methods and techniques for variety of methods and techniques for designing usable products. Key features:designing usable products. Key features:

• Understand users - who they are & what they Understand users - who they are & what they needneed

• Understand the context of useUnderstand the context of use• Involve users in the (iterative) design processInvolve users in the (iterative) design process• Evaluate designs against requirementsEvaluate designs against requirements

(ISO 13407: Human-centered Design Processes for Interactive Systems)(ISO 13407: Human-centered Design Processes for Interactive Systems)

And more definitions…And more definitions…

Usability Evaluation Usability Evaluation - any of a class of methods - any of a class of methods used to assess/evaluate/review how usable a used to assess/evaluate/review how usable a particular product is.particular product is.• Expert evaluationExpert evaluation• Competitive evaluationCompetitive evaluation• Task analysisTask analysis• Cognitive walkthroughCognitive walkthrough• User testUser test• etcetc

User Test User Test - Facilitated evaluation of the usability - Facilitated evaluation of the usability of a product by asking real users to perform of a product by asking real users to perform real tasks with real products in real contexts.real tasks with real products in real contexts.• Or as close as you can get to “real”Or as close as you can get to “real”

Web vs. DesktopWeb vs. Desktop

Ease-of-learning vs. ease-of-useEase-of-learning vs. ease-of-use

Much more holistic user experienceMuch more holistic user experience

Nobody reads on the webNobody reads on the web

““use, then buy” vs. “buy, then use”use, then buy” vs. “buy, then use”

What to Use WhenWhat to Use When

Competitive EvaluationCompetitive Evaluation• Product definitionProduct definition• Planning for redesignPlanning for redesign• Helps identify areas of concern, strategic Helps identify areas of concern, strategic

advantages, etc.advantages, etc.• Define Define whatwhat you want to build you want to build

Task Analysis, et. alTask Analysis, et. al• Analysis & designAnalysis & design• Helps ensure you’re solving the problem the Helps ensure you’re solving the problem the

right wayright way• Define Define howhow you want to build it you want to build it

What to Use When (cont)What to Use When (cont)

Expert EvaluationExpert Evaluation• Early (and often) during initial design phasesEarly (and often) during initial design phases• As soon as you have anything to evaluateAs soon as you have anything to evaluate• Very large bang for the buck - find most of big Very large bang for the buck - find most of big

gotchas earlygotchas early

User TestingUser Testing• Not-quite-as-early (but still early) and oftenNot-quite-as-early (but still early) and often• Certainly before launchCertainly before launch

You’re going to user test it anyway, might as well You’re going to user test it anyway, might as well do it before launchdo it before launch

• A bit harder, but only way to get at some issuesA bit harder, but only way to get at some issues

Lo-fi vs. Hi-fiLo-fi vs. Hi-fi

Don’t wait until you have a working system Don’t wait until you have a working system (or even prototype) to test(or even prototype) to test

You can test a piece of paperYou can test a piece of paper Sometimes, low-fidelity is better to test Sometimes, low-fidelity is better to test

(fewer misconceptions)(fewer misconceptions) Fixing mistakes during construction costs Fixing mistakes during construction costs

ten times as much as fixing during designten times as much as fixing during design You wouldn’t dream of not testing what you You wouldn’t dream of not testing what you

build… why would you ever NOT test what build… why would you ever NOT test what you design?you design?

Let’s Practice!Let’s Practice!

www.intellicast.comwww.intellicast.com• What’s the forecast for Buffalo, NY be tomorrow?What’s the forecast for Buffalo, NY be tomorrow?• Find a current radar image for HonoluluFind a current radar image for Honolulu

www.usatoday.comwww.usatoday.com• Find the box score for the Milwaukee Brewers Find the box score for the Milwaukee Brewers

baseball game yesterdaybaseball game yesterday• Find some current news about the NHL playoffsFind some current news about the NHL playoffs

Recommended ReadingRecommended Reading

Basics of web testing: Basics of web testing: The Web Testing The Web Testing HandbookHandbook by Steven Splaine and Stefan P. by Steven Splaine and Stefan P. JaskielJaskiel

For software with a web front end: For software with a web front end: Testing Testing Applications on the Web Applications on the Web by Hung Q. Nguyenby Hung Q. Nguyen

Usability: Usability: Don't Make Me Think: A Common Don't Make Me Think: A Common Sense Approach to Web UsabilitySense Approach to Web Usability by Steve Krug by Steve Krug

Handbook of Usability Testing: How to Plan, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective TestsDesign, and Conduct Effective Tests by Jeffrey by Jeffrey Rubin Rubin

See See www.stickyminds.comwww.stickyminds.com for more books and for more books and articlesarticles