risk driven testing
DESCRIPTION
Software organizations that want to maximize the yield of Software Testing find that choosing the right testing strategy is hard, and most testing managers are ill-prepared for this. The organization has to learn how to plan testing efforts based on the characteristics of each project and the many ways the software product is to be used. This tutorial is intended for Software professionals who are likely to be responsible for defining the strategy and planning of the testing effort and managing it through its life cycle. These roles are usually Testing Managers or Project Managers.TRANSCRIPT
1
ITMPI005
Webinar: Risk Driven Testing
May 5th, 201011:00 AM CST
Please note: The audio portion of this webinar is only accessible through the telephone dial-in number that you received in your registration confirmation email.
2
Jorge BoriaSenior VP International Process Improvement
Liveware [email protected]
Michael MilutisDirector of Marketing
Computer Aid, Inc. (CAI)[email protected]
3
About Presenter’s Firm
Liveware is a leader among SEI partners, trusted by small, medium and large organizations around the world to increase their effectiveness and efficiency through improving the quality of their processes.
With an average collective experience of over 20 years in software process improvement we know how to make our customers succeed.
We partner with our clients by focusing on their bottom line and short and long term business goals.
With over 70 Introduction to CMMI classes delivered and 40 SCAMPI appraisals performed, you will not find a better consultant for your process improvement needs.
4
• CAI is a global IT outsourcing firm currently managing active engagements with over 100 Fortune 1,000 companies and government agencies around the world.
• CAI is a leader in IT Best Practices for legacy support and new development application management.
• CAI’s focus is directed toward practical implementations that track and measure the right activities in software activity management
• CAI consistently promises and delivers double digit productivity in its outsourcing and consulting engagements.
• CAI makes all of this possible through the use of:
• Standard processes
• Management by metrics
• SLA compliance management
• Detailed cost, resource, and time tracking
• Capacity management
• Standard estimation
• A unique, metrics based methodology along with a proprietary, real time data repository and management system (TRACER®).
About Computer Aid, Inc. (CAI)
5
NOW AVAILABLE!
ONLINE WEBINAR RECORDINGS
ANYTIME ACCESS!
WWW. ITMPI.ORG / LIBRARY
6
Today’s Agenda
– State testing goals and objectives – Identify risks with regard to product and
project characteristics – State testing activities with acceptance
criteria – Select a testing lifecycle to match the
products risks and the project's schedule
Our Objectives
• Identify testing project risks and refine the plan to include mitigation and contingency activities– State testing goals and objectives – Identify risks with regard to product and project
characteristics – State testing activities with acceptance criteria – Select a testing lifecycle to match the products risks
and the project's schedule
8
The Test Categories Map ©
black box
crystal box
UNIT
INTEGRATION
SYSTEM
ALPHA
USER ACCEPTANCE
BETA
time
goal
construction
functionality
perform
ance
usability
string
volume
integratio
n
stress
error h
andling
readiness
configura
tion
memory le
aks
regre
ssion
path coverage
decision coverage
statement covera
ge
data flow covera
ge
Disclaimer: This is just indicative. Individual project’s needs vary
9
UAT Execution(SDS)
Test ReportSys Test Execution(SDS)
Test Report
Acceptance
The V Model Applied
UAT Test Planning and Preparation
System Test Planning and Preparation
Unit Test Planningand Preparation
Acceptance
Requirements(SRD)
Acceptance
Specifications(TSD)
Coding(SDS)
Unit Test Execution(SDS)
Hand OffDeveloped Components
(SDS)`
Phase End Review
Phase End Review
Post Mortem Project Review
10
Common Testing Problems
– we ship before we have finished testing– our customers find most of our defects– we don’t have the testers when we need them– testing is ad-hoc and results are irreproducible– we don’t test for the critical success factors– we don’t test versus requirements from customer– there are no acceptance criteria for any phase and
deciding when the product is ready becomes a point of contention
– ...
11
Risk Management Preventable Problems
– We anticipated lateness but accepted sending out an unfinished product
– We have too many defects to fix too late in the cycle
– We have too many tests to run in a short period of time
– We have tested for the simple defects and the customer gets to test for the “killer” bugs
– …
Mission and goals Decision drivers Organization management Customer / end user Budget / cost Schedule Project parameters Development process Development environment Personnel and relationships Operational environment New technology
Cost overruns Schedule slips Inadequate functionality Canceled projects Sudden personnel changes Customer dissatisfaction Loss of company image Demoralized staff Poor product performance Legal proceedings
Risk SourcesRisk Sources Project ConsequencesProject Consequences
testing concernstesting concerns
13
Critical Success Factors
• A product’s critical success factors (CSFs) are related to:– business needs for its development– coverage of all its intended user constituencies– fitness for use in the intended context– lack of dissatisfiers– competitive advantage
• price?• delighters?
14
CSF: Business Needs (Why?)• Typical business reasons for new systems or changes:
– cost reduction • reducing personnel time in the field reduces costs
– increased efficiency of work or resource• a better interface makes one person capable of doing the job of
two– developing new markets
• using a “smart card” allows very small business to get into the credit card market
– improved resource management• electronic data management (EDM) systems allow insurance
claims to be processed in hundreds of different locations, depending on work load
• What Is The Customer Going To Get Out Of The Use Of The Product?
15
CSF: User Constituencies (Who?)• Answer the questions:
– given the business needs, what goals will the product help achieve?
– who will have to use the product (different user constituencies) to achieve those goals?
• At least three levels of need:– need to increase the bottom line
• typical user constituency: upper management– need to gather supervisory data
• typical user constituency: middle management– need for an efficient interface
• typical user constituency: end user– other?
16
CSF: Fitness for Use (How?)• “Fitness for use”
– the product may meet all stated requirements but not support solving a real need or problem for a given user constituency
– testing features does not cover the workflows
• Test in the context of the job
• Test the product as if you would have to perform the job yourself
• Use your expertise in testing to extend the possibilities of use cases
17
High-Level Testing Goals
• Effectiveness:– Defect detection
• percentage of total found in some time frame• severity found after testing
AND• Efficiency:
– Resource usage• time• people• machines
– Reporting• time spent reproducing the defect by the developers
18
Risks from the Project• Risks from the project come from:• Project Plan Schedule Constraints
– testing might be cut short if development is late– testing might have all its tasks in the critical path
• Lifecycle being followed by the project– Simple Waterfall, – Parallel Waterfall, – Evolutionary, – Prototyping, – Spiral
• Design Architecture• Resources
– missing critical skills– budgetary shortcomings
19
Design Architecture
• Figure out what the Architecture is going to be– Ask the designer– Request information on the design elements early
• If not traditional, plan accordingly– Use scenarios profusely– Try having testers join teams early– Use testers insight of design to develop test cases– Push for updated documentation– Push for consistency reviews across documents
• Requirements traceability• Formal reviews
20
Testing Deliverables• Planning Assets
– Test Plan
• Testing Assets– Testing procedures– Testing suites– Testing templates
• Reporting Assets– Individual tests results– Test statistics– Acceptance report
21
Strategy Problems
• Problems that faulty test strategies can cause– we spend too much time on just one phase– we place all our effort at the end of the project– we ignore regressions– we test in the wrong configuration– we receive work products that are unfit to test– we get into endless arguments about product fitness to ship– ...
• Good testing strategies help!
22
Selecting a Strategy
• Decide on– how much testing is required– of what type– when will it happen– who will do it– how will it happen, e.g.:
• Will integration be top-down or bottom-up?• Will clear box be manual or automatic?
23
Identifying Test Tasks
• Review the V-Model
• Pick and choose the adequate phases to meet the goals
• You have twenty days to visit all of Europe• You may never be able to go back• How do you budget your time?
• Testing is always under-budgeted and over-committed
24
Defining a Strategy• Review and rework the business goals• Review project variables and rework the Testing
Phases – Consider cost, time to market, personnel, product life
expectancy, users, constraints• Emphasize the tasks that tie with the goals
– try to probe weak areas of the project• Points to ponder:
– regression (when, how much, which)– coverage (how much, what type, when)– deadlines (slipping deadlines, schedule compression)– integration (how, when, by whom)– assignment of responsibility (developers, testers, users)
Test Strategies (1)
• Analyze business case– where is the payoff?
• think in terms of customer satisfaction• identify particular functionality or killer faults
• Probe the quality of the product with regard to it
Note: Overriding assumption is
that you will not have time to do it all
Test Strategies (2)• Analyze users
– by frequency of use• sporadic, heavy, etc.
– by organizational level• general managers, middle managers, end users, etc.
– by their knowledge of the software• experts, newcomers, etc.
• Build a profile of system usage– sketch scenarios– assign probabilities for each scenario
• e.g.., one out of eight times the plan will be run unchanged
• Use this data to design the test cases to optimize testing coverage of most frequent paths
Test Strategies (3)
• Analyze time to market– are you going to be pressed for time?
• => focus on existing functionality• => test system rather than component• => test typical rather than exceptional
• Decide which tests can be run within the time constraints– If some of the fundamental tests cannot be
run, move tests forward
Test Strategies (4)• Analyze cost
– how many staff/hours can you pay? • Figure out if the tests you have so far selected fit
within the budget• If so, if you have room for more…• Analyze constraints
• performance• precision• volume
– find critical quantities– analyze or set stress testing
• …then include tests for them
29
Test Strategies (5)
• Analyze personnel– who can you count on
• reinforce testing of the products of the project’s weaker links
– which of your documents are going to be weak for lack of experts
• requirements or specs <=> functional tests• high-level design <=> integration• modules <=> module testing
30
Test Strategies (6)
• Analyze product life expectancy– find the payoff of documented procedures,
test case suites, results, etc..• don’t overspend in a product that has a short life
expectancy• don’t under spend in a product that will be
around long
31
Example Strategy (1)
• Business goal– Keep and expand customer base
• Internal translation to project– Make sure that the user finds the entire
current version’s functionality works as usual
• Testing strategy– Test old before new, in every phase
32
Example Strategy (2)
• Business goalExceed customer’s expectations of product
quality
• Internal translation to project– Fewer than 10% of total bugs caught by the
user
• Testing strategy– Move functional test suites to developers
before and during unit code and testing
33
Acceptance Criteria
• For each test selected, define:– Environment tests would have had to run on– Regression suites covered by the tests – Functionality covered by the tests– Performance testing goals reached– Volume Testing goals achieved– Reliability Testing (when applicable)– Usability Testing (when applicable)
• How can we tell that the work product is safe for releasing it to the user?
34
System Acceptance Criteria• Probably only main deployment environment
– but NOT the development environment only• Usually all regression suites for the entire tests
– usually automated• Most, if not all, functionality
– focus on functionality not tested or changed after unit code• Performance Testing, Volume Testing
– goals MUST be met, no excuses– deployment environment used in tests
• Reliability Testing– if applicable, statistically controlled
• Usability Testing – if critical, or if it leaves development organization
35
User Acceptance Test
• Purpose is to detect remaining defects– focus might change
• Different things to different people– another level of system acceptance– emphatically focused on usability– performed by users exclusively– performed by user proxies, exclusively– performed by the IV&V of the organization– other? ...
• Which is yours?
36
User Acceptance Criteria• Some general rules
– Cover all deployment environments– Only do regression if regression suites are different
• or significant fixes and / or changes have happened after system acceptance
– Functionality focus should be on usual rather than exceptional– Performance Testing should focus on throughput – Volume Testing might be skipped
• unless significant fixes and / or changes have happened after system acceptance
– Reliability Testing• only when thoroughly planned from day one
– Usability Testing• probably the most emphatic effort goes here
37
Limiting the Testing Effort (1)
• Some things cannot be tested– quality– user-friendliness– timeliness– …
38
Limiting the Testing Effort (2)
• Some things you might not want to test– regression on a new or relatively small
enhancement– performance– stress– …
39
Limiting the Testing Effort (3)
• Some things you might not have the ability to test– reliability (e.g. MTTF)– availability (e.g. (MTTF/(MTTF+MTTR)))– …
40
Constraints on the Lifecycle
• Review the phases against the Project’s constraints– Can you accommodate the project plan schedule constraints?– Is the model being followed by the project
• Simple Waterfall, • Parallel Waterfall, • Evolutionary, • Prototyping, • Spiral
allowing for the testing tasks you’ve set?• e.g. might not have integration testing
– Can the tests selected be adjusted to the design architecture?• three tiers might bring a whole set of problems
– Are the goals compatible with the project’s shortcomings?
Risk Action Planning
• Deal with high exposure risks first– Research: Do we know enough about this risk? – Accept: Can we live with it and do nothing about it?– Manage: Can we take action? – Avoid: Should we cancel the project or change the
approach?
• Balance the threat of the risk against the effort to avert it– How great is the threat?– How much does it cost to avert?
Risk Contingency Plans
• Devise contingency plans for– High exposure risks, in case the mitigation strategy fails– Any risk for which there is no possible mitigation action
• Specify risk measures and trigger values– Measures of time, resources to handle risk– Measures of risk impact– Trigger values that tell it is time to use contingency approach
now
• Agree with customer and management at project start how contingency plans will be funded and handled
Example Contingency Triggers
• For risks leading to schedule slips– Latest date to allow you to use alternative platforms– Latest date to select another vendor
• For risks requiring additional effort or time– Latest date to have time to locate the resources– Greatest amount of penalty or fine to incur– Greatest amount of investment available for
overrun• Limit for extra cost to the customer• Limit for learning time
44
Example of Action and Contingency (1)
• Risk Statement– Since the project team is already behind schedule
by two full weeks, we might not have time to cover all the high yield tests before the cutover date and the quality of the product will be seriously imperiled.
• Risk Action– Provide one of our test engineers as a testing
consultant to the development team, so they can test and fix before they send the product to the Testing Team.
45
Example of Action and Contingency (2)
• Contingency Plan– If by the first of April, we do not have an
integration test version of the product, drop the web testing suites and focus on regression so our customer can use a significantly improved current version for six months without a Web interface.
46
Example Contingency for Lateness• Plan your testing activities in passes
– First Pass (mandatory):• test all modules/components• use most frequently used scenarios• use few test cases• use selected error cases
– Second Pass (supplementary):• test only modules with fixes, do first pass on components• cover most scenarios• use test cases covering all data equivalence classes• include test cases for “bad values”
– Third Pass (complementary):• test only modules with fixes from second pass• cover all test suite• go for 100% clear case coverage
47
Summary• Key Activities in Defining the Testing Strategy
– identify test tasks and their goals– rank test tasks by their goals– define the limits of the testing effort– detail testing tasks– define testing tasks entry criteria– define testing tasks acceptance criteria
• Outputs to the process include– individual test task goals – phase acceptance criteria– detailed testing project tasks– all in the updated test plan
48
Questions?
Your PDU CODE: S010-ITMPI0xxxx
4949
CAI Sponsors
The IT Metrics & Productivity Institute:
• Clearinghouse repository of best practices: WWW.ITMPI.ORG
• Weekly educational newsletter: WWW.ITMPI.ORG / SUBSCRIBE
• Weekly webinars hosted by industry leaders: WWW.ITMPI.ORG / WEBINARS
• ACCESS WEBINAR RECORDINGS ANYTIME AT WWW.ITMPI.ORG / LIBRARY
• Online Expert Training Through CAI University (CAIU): WWW.CAIU.COMPAID.COM
• Follow us on TWITTER at WWW.TWITTER.COM / ITMPI
50
Software Best Practices Conferences Around the World
WWW.ITMPI.ORG / EVENTS
Feb. 23 Tampa, FL
Mar. 18 San Antonio, TX
Mar. 23 Philadelphia, PA
Mar. 30 El Segundo, CA
Apr. 15 Philadelphia, PA
Apr. 20 Detroit, MI
Apr. 29 Chicago, IL
May 4 Trenton, NJ
May 11 New York, NY
May 20 Albany, NY
May 25 Toronto, ON
Spring 2010
Sep. 14 Baltimore, MD
Sep. 21 Sydney, AU
Sep. 28 Detroit, MI
Oct. 7 Tallahassee, FL
Oct. 13 Orlando, FL
Oct. 21 Philadelphia, PA
Nov. 16 Miami, FL
Fall 2010
51
Jorge BoriaSenior VP International Process Improvement
Liveware [email protected]
Michael MilutisDirector of Marketing
Computer Aid, Inc. (CAI)[email protected]