creating sanity amidst the test methodology madness - webinar presentation

© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 1 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only

Creating Sanity Amidst Test

Methodology Madness

Right-sizing your test methodologies

© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 2

Agenda

hen hat ow

Identify the

symptoms

Key aspects

to right-size

MindTree

Tools &

Frameworks

Processes should work for you. You should not work for processes.

Rightsizing?

© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 3 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 3

Poll Question #1

● Are your processes taking too much effort and time?

● Yes

● No

● Not able to measure


When to Rightsize?

Acknowledge the need. Don’t wait until you loose your sleep.

• Test processes taking too much effort and time?

• Metrics not helping you take decisions?

• Not able to justify the investments in testing?

• Test process not helping you identify the critical

defects early?

• Difficulty in quantifying the quality in your test

processes?

hen

Identify the

symptoms


What to Rightsize?

Tailor your processes for right fit.

hat

Key aspects

to right-size

• Quality goals / objectives

• Software life cycle methodologies

• Engagement models

• Governance mechanism (metrics,

reviews)

• Tools and frameworks


Poll Question # 2

● Do you know what metrics should you be tracking for your testing

needs?

● Yes

● No

● I do, but not sure if they are exhaustive


Test Metrics Analysis and Decisions (TMAD) Model

Metrics

Based

Governance

• Business goals

• Quality goals

• Governance

• Metrics, reviews,

baselines

• Forecast

• Risks, best practices

• Corrective & preventive

actions


Statistical

Models

Historical

Data

Org / Industry

Baselines

Enablers

ow

MindTree

Tools &

Frameworks



Proposed Metrics Goal Associated Queries (RCA) Undesired Behaviour

(Risks)

Defect Containment Effectiveness

1. Improve testing quality

2. Prevent defect leakage

1. Was the criteria to classify defect detection phase clear from a leakage perspective?

2. Are there clear entry and exit criteria between phases 3. Was the team's knowledge adequate to uncover

leaked defects? 4. Was the scope of testing for each phase clear? 5. Do we have requirement traceability matrix?

1. Blame game 2. Manipulation of defect

classification criteria

Decision Support (Scope) Corrective Actions

Raw Data Requirements Suggestions (feedback)

1. Improve test case review effectiveness

2. Improve process compliance 3. Improve product and test design

knowledge

1. Clear entry and exit criteria between phases

2. Process adherence 3. Effective knowledge management 4. Update test suite 5. Get test strategy reviewed by

customers

1. Defects detected in test phase

2. Leaked defects

Do RCA for every release and identify preventive actions

Metrics Analysis

Metrics Decision Table


Test Quality Index (TQI) Model

● Bivalent questionnaire; Y or N

responses

● “Productivity per tester” vs.

“quality of deployed resources

and resource deployment”

● Contextual metrics vs.

comprehensive score; across

30 process areas

● Scoring based on self

assessment

1 Identify

areas of

improvement

2 Customize

the scored

checklist 3 Improve

scores with

each

deliverable

Winner of the Testing Leadership Award at QAI Software Testing Conference


Poll Question # 3

● Do you have an integrated framework which helps you rightsize the

test processes and methodologies?

● Yes

● No

● We have point solutions, but not an integrated solution


MindTest™ - Integrated Test Methodology

Feedback

Analysis Audits

Test Estimates

TQI TMAD

Test Management

TestLink

Test Measurement and

Metrics - ROI

Test Dashboard

Lifecycle Processes

ProcessNet

Knowledge

Management

ProjectSpace

Defect Management

Radar

Tools Technologies Domain Type of Testing Stage of Testing

Customization

Processes Templates Guidelines Checklists Best Practices

Continuous

Processes

Metrics &

Measurement

Lifecycle

Processes


• Executed in-depth analysis to understand

• Quality objectives and alignment between test process

and application's business goals

• Used Test Metrics Analysis and Decisions (TMAD) Model

and Test Quality Index (TQI) to come up with

appropriate metrics and processes

• Collected key metrics as governance enablers to measure and

drive decisions based on the health of the projects

MindTree Solution

The Business Challenge

The customer is the world's leading specialist in air transport

communications and information technology (IT) solutions.

• 90% coverage on test cases executed, a significant

improvement over the earlier metric

• 100% achievement on acceptable open defects for severity 1

and 2

• Customer overhead of test case reviews reduced by 60%

• Constant review of the test case execution helped us to

evaluate and report how far/near is the application from the

acceptance criteria at different phases of testing

• Enabled development team to prioritize the defects

that needs to be fixed for the immediate build

• Customer is now able to ascertain the quality of the

application

• Reduced risks and enabled effective decision making through

test dashboard

• No testing guidelines

• Inadequate process

• Lack of measures for impact of testing

• No single snapshot of engagement level governance

The Customer Business Impact

“MindTree's expertise in metrics is paramount in managing

the engagement and seeing if process changes have made the

hoped for improvement and if the application is ready for

release. The metrics are also a great way to communicate

progress across all the groups in this engagement.”

Senior Manager, Quality Integration

Case Study


In Short…

You don’t have to search for clues in the

labyrinth

Identify the need

Evaluate the key aspects

Optimize the process

Measure and improve


MindTree Testing Services Overview

Testing Size and

Focus

• Dedicated unit of 2200+

specialists focused on

testing

• Contributes 25% of the

MindTree’s global

business

Testing Depth and

Breadth

• Tested over 2000

applications & products

from chip to cloud

• Services ranging from

point solutions to long

term CoEs

• Domain specific testing

services and solutions

Testing Process and

Delivery Excellence

• Testing pillars of

excellence - MindTest™,

TestLabs,

TestingAcademy

• Virtual practices and

CoEs for effective

delivery

• Part of ISO working

group in defining new

testing standard – ISO /

IEC 29119

• President of India’s

leading testing forum -

STeP-IN

Thought Leadership

1500+ 95% 700+ 20+

delivering PREDICTABLE QUALITY through INDEPENDENT TESTING


ARE YOU TRACKING THE RIGHT TEST METRICS

FOR YOUR BUSINESS DECISIONS?

Complimentary Workshop at your Doorstep

REGISTER NOW

or email to [email protected]

www.mindtree.com/events/testingwebinar/index.html

MindTree Workshop

© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 16 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only

Sylvia Veeraraghavan

[email protected]

www.mindtree.com

creating sanity amidst the test methodology madness - webinar presentation

Documents

limited circulation

mindtree limited confidential

test processes

test metrics analysis

test phase

test methodologies

test strategy

test design knowledge