fedex institute of technology university of memphis systems testing in the sdlc dr. robin poston,...
TRANSCRIPT
FedEx Institute of TechnologyUniversity of Memphis
Systems Testing in the SDLC
Dr. Robin Poston, Associate Director of the System Testing Excellence Program
1
FedEx Institute of TechnologyUniversity of Memphis
Agenda
Testing in Structured Design Tester in the SDLC Testing in Rapid Application and Agile
Development
2
FedEx Institute of TechnologyUniversity of Memphis
Objectives
Understand Testing in Structured Design Understand the role of the Tester in the
SDLC Understand Testing in Rapid Application
and Agile Development Understand Testing in Enterprise
Systems Development Learn the System Testing Phases
3
FedEx Institute of TechnologyUniversity of Memphis
Testing in Structured Design The Waterfall Model The V Model
Source: Modern Systems Analysis and Design by J. A. Hoffer , J. F. George, J. S. Valacich, Prentice Hall 2005
4
Communicationproject initiation &
requirements gathering PlanningEstimating, scheduling,
tracking Modelinganalysis & design Construction
code & testDeployment
delivery, support, feedback
4
FedEx Institute of TechnologyUniversity of Memphis
SDLC
SDLC Phases Planning Analysis Design Implementation & Maintenance
5
FedEx Institute of TechnologyUniversity of Memphis
6Source: http://www.coleyconsulting.co.uk/testtype.htm
FedEx Institute of TechnologyUniversity of Memphis
Testing in Structured Design (cont’d.)
Common life-cycle model of software development Development work goes in sequential phases down left
side of the V Test execution phases occurs up the right side of the V Tests are planned and developed as soon as
corresponding development phase begins. Ex: requirements are the basis for acceptance testing so
preparing for testing can start immediately after requirements are captured.
7
Source: Pressman, R.S. Software Engineering: A Practitioner's Approach, Sixth Education, McGraw Hill, New York, 2005.
FedEx Institute of TechnologyUniversity of Memphis
The SDLC Generally, the SDLC consists of 3 major stages
1. Analysis: Definition2. Design: Development & Installation3. Implementation: Operation
These stages of the SDLC can be further decomposed into 10 phases.
1. Definition consists of 3 Phases: service request/project viability analysis (SR/PVA), system requirements definition (SRD) and system design alternatives (SDA).
2. Development consists of 4 Phases: system external specifications (SES), system internal specifications (SIS), program development (PD) and testing (TST)
3. Testing is interspersed throughout the entire SDLC.
4. Installation and Operation begins with a conversion (CNV) phase, then implementation (IMPL) and ends with a post-implementation review/maintenance (PIR/MN) phase.
8Source: Li, 1990
FedEx Institute of TechnologyUniversity of Memphis
9Source: Li, E., in Journal of Systems Management (1990)
FedEx Institute of TechnologyUniversity of Memphis
SDLC Considerations Sequential Orientation: Only when a SDLC phase is completed and signed
off on can the next phase proceed. How does this impact testing? SDLC a.k.a. Waterfall model is generally a document–driven model: How
can these documentation orientation be used to facilitate testing. Tremendous time and cost is incurred by a "major" change in the document
of an earlier SDLC phase: Is this a testing issue? Testing forces changes to earlier phases and documents. Advanced tools such as CASE tools, automated code generators etc. have
facilitated the SDLC. Has testing stayed abreast of these technologies. Variations of the SDLC to include agile concepts, rapid application methods,
joint development techniques have failed to reconsider the role of testing.
10Source: Li, 1990
FedEx Institute of TechnologyUniversity of Memphis
11Source: Li, 1990
FedEx Institute of TechnologyUniversity of Memphis
Roles Across SDLCRole Activities Performed
Test Practitioner Execute defined tests within an established test environment, and document the results.
Test Analyst Run front-end of testing processes, which includes determining testing needs. Check that the requirements are adequate for starting testing processes.
Test Designer Using requirements, design test cases following published and agreed to techniques.
Test Builder Assemble the equipment needed to run the test cases.
Test Inspector Confirm that testing tasks have been performed correctly according to standards and procedures.
Test Environmentalist Manage and support the test environment.
Test Specialist Execute testing tasks across various phases of the testing life cycle.
12
FedEx Institute of TechnologyUniversity of Memphis
Tester Tasks in the SDLCSDLC Life Cycle Activities Performed Deliverables
Analysis Identify scope and strategy for testing, including refining requirements.
Test plan, updated requirements.
Design Create test cases and organize test effort. Test cases, test data sets.
Implementation Assemble the equipment needed to conduct tests.
Test procedures/scripts, drivers.
Execution Execute tests and capture test results. Test results.
Evaluation Verify results against expectation and follow-up as needed, including reporting, debugging, etc.
Test log updates, modified code.
13
FedEx Institute of TechnologyUniversity of Memphis
Tester Involvement in SDLC
Combine system testing and acceptance testing Test group does functional testing--people with domain expertise Begin testing early, as soon as first build is complete Testers test recent completed build while developers move on to next Developers are responsible for low level of integration testing of each
build before delivered to testers At end testers perform end-to-end test
DesignCode
Build NCleanup
CodeBuild1
CodeBuild2
…..
Prepare Test Plans
….. Test FullSystem
TestBuild1
TestBuild1&2
Developers
Testers
Acceptance Testing
Source: NASA’s Software Engineering Laboratory 14
FedEx Institute of TechnologyUniversity of Memphis
Thought Exercise
As a tester, what potential problems do you see with the Li-model? Also think about potential solutions to these problem.
15
FedEx Institute of TechnologyUniversity of Memphis
Risks of Structured Design
Difficult to plan accurately that far in advance Requirements or resources for the project
may change by the time testing starts20% of medium and 70-90% of large sized
projects are delayed or cancelled Model is driven by schedule and budget risks,
with quality taking back seatIf get behind schedule, test time shortens
Source: Estimating Software Costs by Capers Jones, McGraw-Hill Companies, July 22, 1998, ISBN: 0079130941.
16
FedEx Institute of TechnologyUniversity of Memphis
Risks of Structured Design Testing should start early with groups
cooperating and giving feedback Test cycles must be planned realistically
including the number of cycles needed to find and fix errors/defects
Entry/exit criteria are critical for project status
Revisit features/requirements at entry points or use change controls
17
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Testing in Rapid Application Development The Incremental Model (below) The Spiral Model
Source: Modern Systems Analysis and Design by J. A. Hoffer , J. F. George, J. S. Valacich, Prentice Hall 200518
FedEx Institute of TechnologyUniversity of Memphis
19(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Testing in RAD
Process progresses in spiral with iterative path More complete software is built as iteration
occurs through all phases The first iteration is the most important, as
possible risk factors, constraints, requirements are identified and the next iterations get to more complete software
Evolution of the software towards a complete system
20
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Advantages of Testing in RAD RAD solves The Structured Design problem of
predicting the future Features are not committed to until we know what
is possible User interface prototypes get the users involved
in early testing which mitigates usability risks Users get confused because RAD prototypes
lead to redesigns not a final product at the end of each test period
21
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Testing in Agile Development
Extreme Programming (XP) (below) (aka) Agile Development
Source: Systems Analysis and Design: An Applied Approach, A. Dennis, B. H. Wixom, and R. M. Roth, 2006.22
FedEx Institute of TechnologyUniversity of Memphis
Testing in Agile Development
Agile Testing:Treats developers as the customer of testing Emphasizes a test-first way to create softwareIs incremental because software expands from
core features that are delivered on a predetermined date
Features are added in chunks to a stable coreAs new features are added, new system is
stabilized through testing and debugging
23
FedEx Institute of TechnologyUniversity of Memphis
Testing in Agile Development
Agile testing:Abandons the notion that testers get
requirements and design documents, and give back test plans and bug reports
The documents used for testing are usually flawed - incomplete, incorrect, and ambiguous
Testers in the past have insisted that the documents be produced better
But "better" will never be good enough
24
FedEx Institute of TechnologyUniversity of Memphis
Testing in Agile Development
Design documents can't be an adequate representation of working code Agile methods encourage ongoing project
conversations, where typically:• Testers and developers sit in the same bullpen, share
offices, or work in neighboring cubicles• Many testers are assigned to help particular developers,
rather than being assigned to test pieces of the product Test plan grows through a series of short, low-
preparation, informal discussions Results in short memos about specific issues
25
FedEx Institute of TechnologyUniversity of Memphis
Risks of Agile Development Need more testers Testing starts earlier because need to thoroughly test the
core before adding new increments Vs. Structured Design Model:
Entry criteria for system testing for Structured Model is all features complete & unit tested and requirements set
Agile model can only ask that core features are complete and unit tested. Requirements won’t be set until the end
Automated regression tools can reduce need for staff It is important to develop organized but flexible test
processes and best practices
26
FedEx Institute of TechnologyUniversity of Memphis
Thought Exercise Test-driven programmers create tests in technology facing
language. Their tests talk about programmatic objects, not business concepts. They learn about business needs through conversations with business experts. But it is hard to learn everything through these conversations.
Very often business experts are surprised as something was left out that was obvious. There is no way to eliminate surprises and agile projects make it easier to correct, but it is disappointing.
How can we improve these conversations?
27(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
References Black, R., “Managing the Testing Process.” Wiley Publishing, New York, NY.,
(2002). Kaner, C., Bach,J., and Pettichord, B., “Lessons Learned in Software Testing: A
Context-Driven Approach.” John Wiley & Sons, Inc., New York, NY., (2002). Camarinha-Matos, L. M., and Afsarmaneshz, H. 2008. "On Reference Models for
Collaborative Networked Organizations," International Journal of Production Research (46:9), pp.2453-2469.
Chapurlat, V., and Braesch, C. 2008. "Verification, Validation, Qualification, and Certification of Enterprise Models: Statements and Opportunities," Computers in Industry (59), pp. 711-721.
Kim, H. M., Fox, M.S., and Sengupta, A. 2007. "How to Build Enterprise Data Models to Achieve Compliance to Standards Or Regulatory Requirements," Journal of the AIS (8:2), pp. 105-128.
Kim, T. Y., Lee, S., Kim, K., and Kim, C. H. 2006. "A Modeling Framework for Agile and Interoperable Virtual Enterprises," Computers in Industry (57), pp. 204-217.
28
FedEx Institute of TechnologyUniversity of Memphis
Systems Testing in the Review of Requirements
Dr. Robin Poston, Associate Director of the System Testing Excellence Program
29
FedEx Institute of TechnologyUniversity of Memphis
Agenda Business Requirements Specification Role of Requirements in Testing Tester Involvement Obtaining Requirements in Testing Heuristics and Other Test Tools Other Sources of Requirements Test-Driven Development Software Requirements Specification (SRS) Validation and Verification
30
FedEx Institute of TechnologyUniversity of Memphis
Objectives
Understand what Business Requirements Specification are
Understand the Role of Requirements in Testing Understand Tester Involvement in
Requirements Learn How to Obtain Requirements in Testing Learn about Test-Driven Development Understand what Software Requirements
Specification are
31
FedEx Institute of TechnologyUniversity of Memphis
Statement of what the system must do Statement of characteristics system must have Focus is on business user needs during
analysis phase Requirements will change as project moves
from analysis -> design -> implementation
Business Requirements Specification
What is a Requirement?
Source: Dennis, Wixom and Roth, Systems Analysis and Design, 2006, Wiley
3232
FedEx Institute of TechnologyUniversity of Memphis
Role of Requirements in Testing Requirements “a quality or condition that
matters to someone who matters” (Kaner et al., 2002)
Interesting fictions – useful but never sufficient Won’t automatically receive all the requirements Don’t ask for items unless you will use them Helps guide the testing process
3333
FedEx Institute of TechnologyUniversity of Memphis
Tester Involvement with Requirements Analysis Requirements “analysis” occurs early
in the SDLCIt is a representation of user’s needs
but usually is not completely accurate After requirements are set, designers
build a picture of a solution through a “design” step
34
FedEx Institute of TechnologyUniversity of Memphis
Tester Involvement (cont’d.)
At the requirements analysis stage, testers:Design a few tests but add more after the design
phaseAdd requirements because they are incompleteMake clerical checks for consistent,
unambiguous and verifiable requirements yet designers can do this
• I.e., ‘the transaction shall be processed within three day’ is ambiguous—workdays or calendar days?
35
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Tester Involvement (cont’d.)
At requirements analysis stage, testers should: Understand business domain and create user stories If untrained users create requirements, tester is needed If designers are weak or scarce, testers can compensate As long as testers are trusted and respected by
developers, are convincing when finds issues, and are able to envision the architectural implications ok, if not wait to get involved
36
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Obtaining Requirements in Testing
3 primary ways testers obtain requirements informationConferenceInferenceReference
It is your job to seek out the information you need for testing
37
FedEx Institute of TechnologyUniversity of Memphis
Obtaining Requirements (cont’d.)
Explicit Requirements Gathering Acknowledged as authoritative by users
Implicit Requirements Gathering Useful source of information not acknowledged
by users
38
FedEx Institute of TechnologyUniversity of Memphis
Obtaining Requirements (cont’d.)Implicit Requirements Gathering Authority comes from persuasiveness and credibility
of content, not from the users Competing & Related products Older versions of the same product Email discussions about the project Comments by customers Magazine articles (old reviews of product) Textbooks on related subjects GUI style guides O/S compatibility requirements Your own experience
39
FedEx Institute of TechnologyUniversity of Memphis
Heuristics
Use heuristics to quickly generate ideas and testsTesting Examples:
• Test at the boundaries• Test every error messages• Test configurations that are different than the
programmer’s• Run tests that are annoying to set up• Avoid redundant tests
40
FedEx Institute of TechnologyUniversity of Memphis
Other Test Tools Manage bias Confusion is a test tool
Is the requirement confusing? Is the product confusing? Is the user documentation confusing? Is the underlying problem difficult to understand?
Fresh eyes find failure Watch for other people’s bias One outcome testing is a better, smarter tester Can’t master testing unless you reinvent it Use the user as a tester
41
FedEx Institute of TechnologyUniversity of Memphis
Other Sources of Requirements If you don’t have a good requirements take advantage of
other sources of information User manual draft Product marketing literature Marketing presentations Software change memos (new internal versions) Internal memos Published style guide and user interface standards Published standards & regulations Third party product compatibility test suites Bug reports (responses to them) Interviews (development lead, tech writer, etc.) Header files, source code, database table definitions Prototypes and lab notes
42
FedEx Institute of TechnologyUniversity of Memphis
Thought Exercise A new jet fighter was being tested. The test pilot strapped
in, started the engines, and flipped the switch to raise the landing gear. The plane wasn’t moving, but the avionics software dutifully raised the landing gear. The plane fell down and broke. It is reasonable to assume the bug in the code was that some code was missing—”if the plane is on the ground, issue an error message”.
This is an error/fault/bug of an omission of a requirement and thus lack of adequate testing.
Discuss with a partner how you would go about catching these types of “bugs” before the code is released to users.
(Source: www.testing.com, Michael Hunter, Microsoft Corporation)43
FedEx Institute of TechnologyUniversity of Memphis
Test-Driven Development Unit tests performed before coding Before developer builds new function, first
creates test case that verifies itTest all inputs, outputs, boundary cases, error
conditionsUnexpected inputs or error conditions
Planning for ways code might fail first results in catching more when coding software
44
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
Test-Driven Development (cont’d.) Developer understand requirements better
But more fun to jump in & start coding! Creating unit tests firms up requirements in
developer’s mind and see what’s missing Ensures testing gets done Poor interface designs become apparent early Complete suite of unit test make it easier to
refactor software Unit test are run after refactoring to see if new
defects were added in updating the code
45
FedEx Institute of TechnologyUniversity of Memphis
Software Requirements Specification (SRS)
Description of the behavior of the software to be developed, includes: Use cases describe interactions users have
with the software Functional requirements define internal
workings of software (calculations etc.) Non-functional requirements define constraints
on design and implementation (performance etc.)
Coding and testing work is based on the SRS
46
FedEx Institute of TechnologyUniversity of Memphis
SRS (cont’d.) Performed by the system analyst Process followed to create the SRS document
Source: Pfleeger and Atlee, Software Engineering: Theory and Practice, 2006, Pearson/ Prentice Hall
47
FedEx Institute of TechnologyUniversity of Memphis
SRS Checklist
Fit criteria form objective standards for deciding whether a solution satisfies the requirementsEasy to set fit criteria for objective requirementsHard for subjective quality requirements
Three ways to help make requirements testable Identify a quantitative description for each adverb and
adjectiveReplace pronouns with specific names of entitiesEnsure every noun is defined in just one place in the
requirements documents
Making Requirements Testable
48
FedEx Institute of TechnologyUniversity of Memphis
Validation and Verification
Examine the original goals / objectives of the system Compare the requirements with the goals / objectives Examine the environment of the system Examine the information flow and proposed functions Assess and document risks, discuss alternatives Assess how to test the system: how will the
requirements be re-validated as they change
Requirements Review
49
FedEx Institute of TechnologyUniversity of Memphis
Thought Exercise
What do you think it takes to be a great tester?
50
FedEx Institute of TechnologyUniversity of Memphis
Summary
You should Understand what Business Requirements Specification are
You should Understand the Role of Requirements in Testing
You should Understand Tester Involvement in Requirements
You should Know How to Obtain Requirements in Testing
You should Know Test-Driven Development You should Understand what Software
Requirements Specification are
51
FedEx Institute of TechnologyUniversity of Memphis
52
Testers
Dr. Robin Poston, Associate Director of the System Testing Excellence Program
FedEx Institute of TechnologyUniversity of Memphis
53
Agenda
Critical Tester Skills Types of Testers Test Staffing Variations Attitude Matters
FedEx Institute of TechnologyUniversity of Memphis
54
Critical Tester Skills
General qualificationsEducation and good verbal and written
communication skills in formal and informal settings
Ability to read closely with significant attention to detail
Testing skills Application domain knowledge Computer Technical expertise
(Source: Michael Hunter, Microsoft Corporation)
FedEx Institute of TechnologyUniversity of Memphis
55
Critical Tester Skills (cont’d.)
TestAdmin-istrators
TestTool Experts
Automated Test
Specialist
BusinessAnalysts
TestTechnicians
Manual Test
Specialist
Application Domain Knowledge
Testing Skills
Computer Technical Expertise
FedEx Institute of TechnologyUniversity of Memphis
56
Types of Testers
Test Engineers (Leads) Skilled, educated, and experienced in all areas of
testing (planning, design, implementation, and execution)
Possesses skills, expertise and knowledge Test Technician and Specialist
Less skilled, educated, and experienced in critical skills areas (testing, application domain and technology).
Qualified for scripted manual test execution and other straightforward technician-level tasks.
FedEx Institute of TechnologyUniversity of Memphis
57
Test Participants & RolesThe participants of testing vary with the size of the project. There are five possible roles played :
1. Test Data Administrator: Controls file structure and data base contents; manages availability and recoverability of data; supports developing and implementing the test data requirements strategy.
2. Test Execution Lead: Prepares test-case conditions; manages test data and files; executes desk checking and test runs; reports test results and discrepancies (errors).
3. Test Team Leader: Supports test planning; manages test preparation, execution and evaluation; creates test plan and test-case summaries; manages testing support; oversees activities of test team; provides technical assistance to test team; participates in quality assurance reviews and inspections.
4. Test Team Member: Supports defining test conditions for test-case designs and reviewing test-case specifications and test results.
5. Test Supervisor/Coordinator: Assigns tests to test teams; reviews and approves all the relevant test materials.
Source: National Occupational Classification; Human Resources & Skills Development Canada 2006
FedEx Institute of TechnologyUniversity of Memphis
58
Systems Testing TechniciansSystems testing technicians execute test plans to evaluate the performance of software applications and information and telecommunications systems. They are employed in information technology units throughout the private and public sectors.
Related Titlesapplication testerapplication testing techniciansoftware test coordinatorsoftware testersystems testersystems testing technicianuser acceptance tester
Main dutiesSystems testing technicians perform some or all of the following duties:
Develop and document software testing plans Install software and hardware and configure operating system software in preparation for testing Execute, analyze and document results of software application tests and information and telecommunication systems tests Develop and implement software and information system testing policies, procedures and scripts.
Employment requirementsCompletion of a college program in information systems, computer science, computer programming or network administration is usually required. College or other courses in computer programming or network administration are usually required. Certification or training provided by software vendors may be required by some employers.
Source: National Occupational Classification; Human Resources & Skills Development Canada 2006
FedEx Institute of TechnologyUniversity of Memphis
59
Test Staffing Variations
Temporary Assignment—people employed elsewhere join testing team Users or technical support staff Add application domain expertise Add personal workflows & scenarios
Rotation—developers, testers and technical support all rotate among their roles Management and HR must agree Must be positioned as positive & supported Minimizes test skills and has high turnover
FedEx Institute of TechnologyUniversity of Memphis
60
Test Staffing Variations (cont’d.)
Archetype Definition
Sequence Development is a effort based on a linear set of discrete tasks. People work in specialized functions with formalized interactions. People are valued for their particular specialized skills.
Group Development is development and production combined where a set of discrete tasks are repeated until the software is complete. Developers and testers work in interdependent groups and are rewarded for their skills and ability to work together.
Network Development is a process of constant development with a focus on the outcome. Tasks are not sequential and are tied to individuals (or small groups) whose participation is based on interaction. Group members are rewarded for what they produce. This involves a network of people and hub-and-spoke management.
Source: Software Development Teams: Three Archetypes and Their Difference, Steve Sawyer.
FedEx Institute of TechnologyUniversity of Memphis
61
Test Staffing Variations (cont’d.)
Source: Software Development Teams: Three Archetypes and Their Difference, Steve Sawyer.
Archetype
Aspect
Sequence Group Network
Perspective Process first Process first Product first
Belief mode Control Conflict Interaction
Orientation Prescriptive Normative Descriptive
View of task Production Production & development
Development
Implied method
Linear & sequential Iterative & sequential Emergent & non-linear
Tie to context Prescribed boundary Permeable boundary Embedded
People’s actions
Prescribed Role & goal driven Individual & linked
Examples SDLC, SEI/CMM Spiral, RAD, JAD Open source, Chief programmer
FedEx Institute of TechnologyUniversity of Memphis
62
Attitude Matters Professional pessimism—approach testing with
assumption that bugs exist and we will find them Advocate for quality in a non-adversarial, impersonal,
non-disruptive way Curiosity—find issues when writing & running tests
Avoid perfunctory action and explore more Focus—understand quickly changing priorities General and supportive role—steady, consistent test
process which may not have glory and glamour Work hard—schedules and budgets compress testing Advocate quality
(Source: www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
63
Thought Exercise
A good tester understand how users spend their days. Why is this useful?
FedEx Institute of TechnologyUniversity of Memphis
64
Summary
Know Critical Tester Skills Know the Types of Testers Know Test Staffing Variations Know that Attitude Matters
FedEx Institute of TechnologyUniversity of Memphis
65
Users and Testers
Dr. Robin Poston, Associate Director of the System Testing Excellence Program
FedEx Institute of TechnologyUniversity of Memphis
66
Agenda Who are Users Communicating with Users User Testing Test Coverage Testing With 5 Users Why Not With 1 User When To Test With More User Testing Process
FedEx Institute of TechnologyUniversity of Memphis
67
Objectives
Understand Who are Users Understand how to Communicate with
Users Know how Users are involved in Testing Understand Test Coverage Learn about Testing With 5 Users and
not With 1 User Know When To Test With More Users Understand the User Testing Process
FedEx Institute of TechnologyUniversity of Memphis
68
Who are Users
Business analysts Help desk Customer support Technical support End customers and business
users
FedEx Institute of TechnologyUniversity of Memphis
69
Communicating with Users
Traditional focus groups Electronic focus groups Iterative surveys Exploratory surveys Scenario-building
FedEx Institute of TechnologyUniversity of Memphis
70
User Testing
Acceptance—performed by the customer prior to the customer accepting delivery of the system, using black-box testing.
Pilot—establishes readiness for release, in terms of starting to use the system to do real work.
FedEx Institute of TechnologyUniversity of Memphis
71
User Testing (cont’d.)Alpha—actual operational testing by users/
customers or an independent test team at the company site.
• Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before it goes to beta testing.
Beta—software is released to a limited audience. • Released so further testing ensures it has few faults. • Available to the public to increase feedback received
from a lot of future users.
FedEx Institute of TechnologyUniversity of Memphis
72
Test Coverage
Workflows Data sets Configurations Field conditions Gives usability issues that might not
occur to testers Quality can be inconsistent
FedEx Institute of TechnologyUniversity of Memphis
73
Testing With 5 Users
Usability is costly and complex and user tests may be reserved for web design projects with large budgets and long time schedules.
Highly structured usability tests are a waste of resources.
The best results come from testing with no more than 5 users with as many small tests as possible.
See next slide for graph
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
74
Testing With 5 Users (cont’d.)
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
75
Testing With 5 Users (cont’d.) Test with 15 users to find usability problems 5 good to use limited resources across many small tests
instead of a big one Budget for 15 representative customers and they test
system spending time on 3 tests with 5 users each Run multiple tests because the goal of testing is to
improve the design and document weaknesses After first test with 5 users find 85% of the problems,
redesign Test again Second test with 5 users will discover remaining 15% of
the problems not found in the first test
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
76
Why Not With 1 User Is 15 tests with 1 user better than 3 tests with 5
users? Curve shows we learn much more from the first
user than from any subsequent users, so why use more than 1 user to test? Two reasons: Risk of being misled by 1 person who may perform
certain actions by accident or in an unrepresentative manner. 3 users are enough to get diversity in behavior
Cost-benefit of user testing provides optimal ratio around 5 users. Better to use investment in setting up to get multiple findings.
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
77
When To Test With More
More users when several highly distinct groups of users For example, children and parents, the two groups of users
have different behavior Even when they different, there will be similarities in behavior
across the groups Many problems are related to the way people use the system There is no need to include as many members of each group
as you would in a single test of a single group Overlap will ensure a better outcome from testing with a
smaller number of people in each group, for example: 4 users from each category if testing with two groups of users 3 users from each category if testing three or more groups of users
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
78
User Testing Process Evaluate software using a variety of methods:
Usage statistics, Analysis of feedback provided, and Structured interviews with users.
Testing with users is effective. Testing provides useful information used to make effective
modifications. User testing typically done with minimal resources to
produce useful results. Other methods supplement the results providing more
accurate assessments expert evaluation, heuristic review, usability walk-through, surveys
and monitoring software. User testing involves users working through a set of tasks
using the application. (Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
79
User Testing Process (cont’d.)
Usability testing sessions conducted in a lab or at the person's desk
One person per test session, lasting 30-60 minutes Outcome is to evaluate the ease of use and
intuitiveness of the system Users provide feedback about what they like and
dislike and about difficulties they may have Use feedback to revise the application
(Source: Deming 1986 and Ishakawa 1985; www.testing.com, Brian Marick)
FedEx Institute of TechnologyUniversity of Memphis
80
Thought Exercise
Who does the testing team serve and how does it do that?
FedEx Institute of TechnologyUniversity of Memphis
81
Summary
You should Know Who are Users Know how to Communicate with Users Know how Users are involved in Testing Know Test Coverage Know about Testing With 5 Users and
not With 1 User Know When To Test With More Users Know the User Testing Process