testing as a managed service using slas and kpis

CONNECT WITH US:

Testing as a Managed Service Using SLA’s and KPI’s

Testing Conference, Chicago

September 2014

CONNECT WITH US: 2

What is it?

Why should I care? Knowing the skillset increases

your worth to the business Not all Outsource Groups are

created equal The business will still look to

you to ensure over all quality

Your tools for success? Proper Governance

Specific KPIs Custom SLAs

How do you successfully manage outsourced testing?

Outsource testing is when a company decides to do all or part of there testing from a 3rd party specialist either onsite, near shore or offshore.

Manage Services is when the 3rd party handles everything from Strategy through Sunset

CONNECT WITH US:

Accelerate Problem

Identification and

contribute to the

development process

Make Process Improvements and ensure accountability on both sides

Governance both for

internal QA team and/or

Outsourced Vendors

Improve Quality of

Software as well as Service Delivery

Adhere to

Internal

and

External

Complianc

e

Standards

Measure to avoid the common mistakes

7

The Aberdeen Group in 2010 estimated that nearly 50% of all outsource projects fail outright or fail to meet expectations

CONNECT WITH US: 4

KPIs

Guidelines

Key Performance Indicators vs Metrics

KPIs highlight the things that are important to your business and should tie directly to business goals Financial Goals - QA costs as a % of total IT

costs and as a % of revenue Staffing Goals – QA FTE to total FTE Ratio Operational Goals – Business Process Quality

and Continuity as a % of Up Time

Guidelines for creating KPIS The success measure must show clear, specific

and measureable performance indicators Costs to monitor can not exceed the value that

will known from measuring Linked to a strategic objective and easily

understood within its context Credible and under the control of the team

attributed to

CONNECT WITH US: 5

SLA’s are vital signs. You can tell where a testing organization is headed and also measure it’s maturity level.

Service Level Agreements

CONNECT WITH US: 6

Understand your KPIs and determine what you need specific to your environment

Remember to include items you will typically delegate to internal staff to self manage

Service Level Agreements (SLA’s)

Before you begin…..

CONNECT WITH US: 7

What are some of the Key go to items to have in your SLAs?

Product quality measurements DRE (Defect Removal

Efficiency) Traceability, test case

efficiency and code coverage Defect identification quality

(valid bugs identified % of the time)

Test plan quality and strategy will require no more than X reviews

Reporting and project management timelines

Defects will be reported within a specified time window by severity

Defects assigned for re-test are regressed within x hours

Testers will design X number of test cases per day per tester based on complexity

Manual testers will execute X number of test cases per day

Automation testers will automate X number of test cases a day

Knowledge Transfer and Core business understanding

Vendor staff are required to maintain X experts and Y intermediates in our core business process

CONNECT WITH US: 8

DRE Process Compliance Business Process Knowledge

General Incident Mgmt. Automated Regression Library - Execution

Regression Library - Updates

100%

98%

100%

91%

100%

100%

Snapshot of a SLA Metrics Dashboard

CONNECT WITH US: 9

Metrics

Cost On-Cost Delivery (Accuracy

of cost estimation) Estimated Effort Index

(Actual vs. planned hours) Time

On-Time Delivery (Accuracy of schedule estimation)

Schedule Variance Productivity

Percentage Rework Quality

Weighted Average Defect Density

Defect detection rate (based on function points or person hours)

Defect Management Defects detected by phase

and by type Root cause distribution

SLAs

Delivery within budget Defect Removal Effectiveness Estimation and Sizing

Turnaround Test Design Efficiency

Process Compliance Score Staff’s knowledge of business

process Accuracy of Metrics General Incident Management

Automated Regression Test Library

Regression Library Maintenance

KPIs

Cost of testing as a % of overall project cost

Test Case Effectiveness

Test Progress – Plan vs. Actual

Skill Set Matrix

Quality of Development

Quantifiable Regression Execution

Number of regression test cases updated by release/ project

Illustrative list of typical performance measures

How KPIs, SLAs, and Metrics work together

Individual Project PerformanceMain Project DriversBusiness Goals

CONNECT WITH US: 10

Our Governance Process at McKesson

We have a three-tier model for leading the McKesson -Prolifics partnership and providing comprehensive program

oversight.

Set strategic direction Review long term goals Review areas of

improvement Optimize alignment

across partners

Steering Committee Meeting

Operational planning Account management SLA Reviews Escalation management New Initiatives

Leadership Team Meeting

Project progress review Resource management Prioritization Knowledge management Issue management

Test Managers Meeting

McKesson

Head of testing practice

Account Executive

VP McKesson IT Generics

Director SV&T Senior

Director SV& T Director

Frequency

Quarterly Meetings

Weekly Meetings

Monthly Meetings

Account Executive

Onsite Test Manager

Generics Director

SVCS Sr. Director

SV&T Senior Director

SV& T Director

Account Executive

Onsite Test Manager

Project /Program Manager

Prolifics

McKesson Prolifics

McKesson Prolifics

CONNECT WITH US: 11

Automated SLA Reporting to Track Events in Real Time

CONNECT WITH US: 12

Sample SLA Report

testing as a managed service using slas and kpis

Technology