Trends in Software Testing

- Ajay Kumar Chhokra

2Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Content

• About UnitedHealth Group• Factors Contribute in Trends• Shift in Company Technology Priorities• Business Drivers and Expectations• Key Drivers• Test Reduction Techniques

– Risk based Testing– Pair-wise Testing– Cycle based Testing– Adaptive Testing

• How to Apply Adaptive Techniques• Benefits and Challenges of Adaptive Techniques• Case Study on Adaptive Testing• Optimization• Test Automation - Effort ROI Computation• Summary

3Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

About UnitedHealth Group

• UHG is ranked Number 21 in the Latest Fortune 500 rankings for 2009 (revenues wise)

• Headquartered in Minnetonka, Minnesota, USA • Serves 71 million lives • Annual revenue greater than US # 80 Billion

• Total number of employees: 60,000+

4Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Factors Contribute in Trends

Trends change regularly - Software Testing Today is different than what it was 10 years ago and will be different 10 years from now……..

Factors contribute in Trends are

TechnologySOA, Distributed

Technology, Web

TechnologySOA, Distributed

Technology, Web

MethodologyRapid Development,

Agile

MethodologyRapid Development,

Agile Customer /Business ExpectationCost, Time, Reliability and Robustness

Customer /Business ExpectationCost, Time, Reliability and Robustness

Business

Complexities

Business

Complexities

GlobalizationGlobalization

5Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Shift in Company Technology Priorities

• UHG 2009 Priorities Strengthen Release Quality – On Time, On Quality, on Cost, On Benefit, On Scope Improve Our Service – Operational Improvements Simplify our Technology Landscape – System Investments Invest in our Future – People and Leadership

• Shift is happening– No more Technology Focus– No more Methodology Focus– No one is talking about Zero Defect……but releasing with Known

Defects– Demanding more Dashboards - Progress Visibility

It is more and more of Business Focus

6Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Business Drivers and Expectations

Drivers Expectations Effects onTime to Market Reduce Cycle Time •Revenue

•Cost

•Competitive Edge

Quality Predictability, Defect Density, and High Test Coverage

•Customer Satisfaction

•Market Share

•Revenue

Cost Manage Peaks and Valleys •Pressure on reducing Testing Effort

•Profitability

Employee Smart Testing •Employee Morale

Governance Quality Gates, Metrics and Dash Boards

• Visibility

•Confidence in Release Quality

7Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time

8Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time- Smart Testing

9Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time- Smart Testing- Visibility

10Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time- Smart Testing- Visibility

Test Reduction Techniques

Test Automation

Optimization

11Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time- Smart Testing

- Visibility

Test Reduction Techniques

Test Automation

Optimization

People

12Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time

- Smart Testing

- Visibility

Test Reduction Techniques

Test Automation

Optimization

PeopleProcess

13Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time

- Smart Testing

- Visibility

Test Reduction Techniques

Test Automation

Optimization

PeopleProcess

Tools

14Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Reduction Techniques

Risk based Testing Pair-wise Testing Cycle Based Testing

15Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Reduction Techniques -Risk based Testing

Benefits Challenges Focus on the key business areas/critical business processes

Eliminates the major risks

Quick to apply if the tester and business have good exposure to business domain

Identifying the business risks precisely is difficult (Requires clear understanding of applications and

business and detailed knowledge of requirements)

Technique focuses on the key risks and often ignores the low impact areas.

Technique is not foolproof; it is more or less intuitive in nature.

16Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Reduction Techniques - Pair-wise Testing

Benefits Challenges

Reduce the test cases drastically at 2 and 3-level interactions

Helpful for complex systems that involve large number of parameters and their combination

Suitable for regression cases where coverage is to be maintained high with minimal cases count

The defect detection rate versus the applied effort is high.

Chance of undetected defects at lower level interaction is more than higher order interaction

With number of interaction levels increases, the number of cases increases very significantly

Pairwise testing fails when the parameters and levels have not been appropriately identified.

Pairwise testing (primarily focusing on 2 –way implementation) does not cover all input space.

17Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Reduction Techniques - Cycle Based Testing

Benefits Challenges Gives good visibility on Testing progress

Helps to define logical milestone for testing. The defect fixes or patches applied are tested in subsequent cycles.

Very helpful when applying Incremental Integration techniques

Technique is discouraged when individual test cycles are too long.

May involve overheads if repeated data setup is required in every subsequent cycle.

Too many cycles in single release delays the process.

Tracking the execution, defects and cross-impact is sometimes difficult when using this technique.

Sample Cycle Based Testing Criteria

Cycle 1 70% Executed 80% pass

Cycle 2 100% Executed 90% pass

Cycle 3 100% Executed 100% pass

18Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Adaptive Testing

A Technique developed to “adapt” to various project needs. Well suited for specific project needs – reducing time or increasing

test coverage etc. Adaptive techniques can be combination of various market

proven techniques such as Risk Based testing, Pairwise testing etc

tailored per project needs.

Adaptive Testing

19Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

How to Apply Adaptive Techniques ?

The Mix-n-Match of the individual techniques can help project have different FLAVORS of adaptive testing. Each Case below is project specific and business objective based.

Case 1: Risk Based + Pairwise Test• Define the parameters and levels of interaction • Identify the high impact condition using RBT• Apply Pairwise technique on less/ low impacting parameters

Case 2: Risk Based + Cycle Based Test• Identify the high impact and low impact business condition• Define multiple cycles • Distribute the high impact and low impact functionalities over cycles based on

priorities

Case 3: Pairwise Test + Cycle Based Test• Identify parameters and levels • Apply Pairwise technique to arrive at reduced test set• Prioritize the test cases• Distribute the test cases in various cycles – sanity, priority and retest

20Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Benefits and Challenges of Adaptive Techniques

BenefitsBenefits

Can be applied to varied project scenarios or type of testing – regression, enhancement, system, integration testing etc

Adaptive techniques are fool proof and help to identify hidden defects

Focus is on high coverage and critical business functions simultaneously

Can be universally applied in product based releases cycles

High probability of defect detection.

ChallengesChallenges

Requires good knowledge of system & market techniques (Pairwise, combinatorial, risk based techniques etc)

Need analytical capabilities in determining how to Mix-n-Match or tailor the techniques

Changes in business process necessitate revision of test cases and re-assess risk associated

21Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Reduction Techniques – Adaptive Testing Case Study

Risk-Based Risk-Based TechniqueTechnique

Pairwise Pairwise TechniqueTechnique

Adaptive Adaptive TechniqueTechnique

Reduced Test CasesReduced Test Cases 32 16 32

Defects FoundDefects Found 8 5 9

Test Reduction (%)Test Reduction (%) 97.2% 98.6% 97.2%

Defects Comparisons

9

45

8

0

2

4

6

8

10

Release

Defe

ct

Co

un

t

Series1

Series1 9 5 8 4

Adaptive Pairw ise Risk Based Ad-Hoc

Exhaustive Test Case Count = 1152

UHG India Experience – Case Study

22Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time

- Smart Testing

- Visibility

Test Reduction Techniques

Test Automation

Optimization

PeopleProcess

Tools

23Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

App 3App 3App 1App 1 App 2App 2

Business Group 1 Business Group 2

Current State

Attributes• Heavy onshore• Lot of productive time going in

interfacing and coordination Challenges • Test Teams working in Silos• Have to struggle with peaks and Valleys• No leverage of Best Practices across Applications• Limited Test Governance• Hardly any Test Standardization• High Cost of Operation• Limited Career Growth• Lot of dependency on Vendor partners

App 6App 6App 4App 4 App 5App 5

Onsite EmployeeOffshore EmployeeOnsite ContractorOffshore Contractor Standards Governance

Optimization

24Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

App 3App 3App 1App 1 App 2App 2

Business Group 1End State

Onsite EmployeeOffshore EmployeeOnsite ContractorOffshore Contractor Standards Governance

Automation Regression Functional Testing

Test StrategyEnd-to-EndTesting UAT

Testing Center of Excellence

K IN M O PW AL RE TD IG NE G

App 6App 6App 4App 4 App 5App 5

DB TestingPerformance Testing

DW Testing

UI TestingProcessSpecialist Facet Testing

T GE OS VT E R N A N C E

Business Group 2

Trainer

Attributes• Lean onshore and focusing on Business Interface and defining Test Requirements

• TCOE owns Test Delivery Responsibility

• TCoE coordinating with Vendor partners

• Optimal utilization of offshore

• Standardization and Test Governance • Good Career Opportunity Challenges • Resource Mapping• Cross Training

Optimization

25Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

CoE Benefits

Capture and Drive Best PracticesAssign SMEs to train and mentor other Testing associates across

projects on Assets, Best Practices and Tools.

Reduce Cost of OperationPromote and help in reusing the valuable Assets created in various engagements thus giving high ROI and lower rework effort.

Give a Jump StartProvide skilled resources for Architecting test solutions or solving complex problems.

Minimize IT RiskSuggesting and Reviewing Testing Approach, solutions and Frameworks, Helping in establishing Quality Gates; Metrics Collection thus giving predictable results and quality delivery

Upgrade Skills of Test professionalsDevelop SMEs on Applications, Technology, Processes and Tools

26Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Key Drivers

- Reduce Cycle Time

- Smart Testing

- Visibility

Test Reduction Techniques

Test Automation

Optimization

PeopleProcess

Tools

27Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Test Automation

– Pros► Frees testers for more intelligent types of testing► Low cost of Execution► More coverage► High efficiency and consistency for repeated execution► Improved Quality of testing

– Cons► Increased up-front investment in Test Development► Easy to waste time in automating wrong tests or right tests in wrong

way► Require more technical expertise► Specialized tool support required than manual testing

28Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Effort ROI Computation

Effort ROI =

Item Definition Effort

Man Effort to Execute Test Cases Manually 81 Hrs for 270 Test Cases

Auto Automation Execution Effort

Data Preparation

Result Analysis

10 Hrs

Main Time spent on the Maintenance of the scripts, generally considered from the second iteration cycle onwards

Dev Effort Spent on Script Development, Automation Strategy Development effort, Debugging

750(450+300)Hrs

i∑[(Man-Auto)+Main]-Dev

Dev

88% Test Execution Saving

-100

-80

-60

-40

-20

0

20

40

1 2 3 4 5 6 7 8 9 10 11 12 13

IterationBreak Even

+ve ROI

RegressionIteration

Effort ROI

1* -90.27

2* -81.87

3* -72.13

4* -62.40

5* -52.67

6 -42.93

7 -33.20

8 -23.47

9 -13.73

10 -4.00

11 5.73

12 15.47

13 25.20

29Any use, copying or distribution without written permission from UnitedHealth Group is prohibited.

Summary -

Trend Setter

– Shorter Time to Market by Adaptive Techniques

– Resource Optimization and Specialization by Center of Excellence

– Test Governance and Dash Boards for Visibility and Control

– Doing More with Less Cost by Reusing Assets

– Test Automation if positive RoI

Thank You