key strategies to survive the mega test program

W7 Concurrent Class

10/2/2013 1:45:00 PM

"Key Strategies to Survive the

Mega Test Program"

Presented by:

Robert Goetz

Kaiser Permanente

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Robert Goetz

Kaiser Permanente

With more than twenty years of leadership experience in all aspects of IT, Bob Goetz has

focused on leading QA organizations for the last several years. As part of this, Bob has led the

testing of several mega programs—more than $500 million. The first mega program turned

Bob’s hair grey as he gained the experience to survive and thrive. That multi-year effort

implemented new core business software while transforming business processes and

introducing new products to the market.

1

Key Strategies to Survive the Mega Test ProgramBob Goetz, Executive Director, Test Services

Kaiser Permanente - Health Plan BIO

2 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013

Topics

About Kaiser Permanente

What is a Mega Program?

Key Strategies, Methods & Processes

Final Takeaways

Q & A

2

Kaiser Permanente: The Largest Not-for-Profit Health Plan

• A Durable Partnership

– Kaiser Foundation Health Plans and Hospitals– Eight Permanente Medical Groups

• More than 9 million members • Serves 9 states and District of Columbia• 37 medical centers; 611 medical office buildings • 175,000+ employees• 17,000+ physicians • $50.6 billion operating revenue• Largest labor management partnership in history• Largest non-academic research organization in the U.S.• Largest private user of electronic medical records in the world

Data as of year-end 2012

July 29, 2013 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.3

The Kaiser Permanente Mission

Kaiser Permanente’s mission is to provide high-quality, affordable healthcare services and to improve the health of our members and the communities we

serve.


3

Mid-Atlantic Region Washington, DC Maryland Virginia

Georgia Region Atlanta, GA

Colorado Region Denver / Boulder, CO Colorado Springs, CO

Ohio Region• Cleveland, OH• Akron, OHNorthern

California Region

Hawaii Region

Northwest Region Portland, OR Vancouver, WA

SouthernCalifornia Region

The Markets We Serve


Kp.org and My Health Manager –World’s Leading Personal Health Record

YE 2012 –

– 116 million visits

– 32 million lab test results viewed

– 13 million secure messages through Email My Doctor

– 12 million Rx refills

4



Or how did my hair turn grey?



5





• Huge Opportunity… Huge Risks

• Bets your company’s future

• Has Board of Director (BOD) Oversight

• External Review / Audit

• Your job is on the line

6



• Huge Opportunity… Huge Risks

• Bets your company’s future

• Has Board of Director (BOD) Oversight

• External Review / Audit

• Your job is on the line

How can you SURVIVE and even THRIVE a Mega Program?

Set the Stage


7

Set the Stage

Everything is a business problem– What is the mega program trying to solve?

– Why is it important?

Discover the situation you are facing– Who are the stakeholders?

– Who holds the decision rights?

– Who decides “quality”? What is “highest level of quality”?

– What are the Risks? Issues?


Standardize Processes


8



• Do a very quick analysis of process AND compliance• Don’t assume your teams are following your “standards”


Test Strategy

Test Plan

Test Suites/Cases

Data Creation

Data Usage

Naming Standards

Component Names

Automation

Test Closeout Report

Defect Standards


9



• Do a very quick analysis of process AND compliance• Don’t assume your teams are following your “standards”

• Create New Tester Orientation Guide• Test Suite/Test Case Standards

• Defect Standards



Defect Severity Severity Level Description1. Critical Complete system outage and/or is detrimental to the

majority of the development, user, and/or testing efforts and there is no known workaround. This would also be known as an emergency defect.

2. High Serious adverse impact to the user and there is not an effective work-around, however work can continue.

3. Medium Moderate adverse impact to the user but there is an effective workaround.

4. Low No immediate adverse impact to the user. Normally cosmetic.

10

19 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.

Build Allies

July 29, 2013

Build Allies


Understand the history of your partners– Do they trust you?

– Find out who they trust and why – learn from that

Be transparent– Admit what you know AND (more importantly!) what you don’t know

Give them a seat at the table– Treat them as they deserve: as professionals

– Loop them in on decisions

– Find out what they need and work their problems for them

11

Don’t Forget Your “Friends”


Don’t Forget Your “Friends”


Truism– A divided house falls

– Never throw development (or UAT) under the bus – even as you shake off their dust!

– To the business I.T. is one org – they don’t care who does what

Establish early in the project your accountability for metrics– Own it – if at all possible

– Be transparent – share all data possible – with everyone

Get data to development early – Give them a chance to consume it and be able to respond

12

Measure Progress


Measure Progress


Treat Defect Data as a Precious Commodity– Be consistent in how you log defects (e.g. root cause)

Make it public and protected– Daily reports, open SharePoint Site

– Converted to PDF format

Formats

– Static “point in time” reports

– Trending Charts

13


Burn Up Chart


Burn Down Chart

14


Backlog Chart


Defect Metrics

15


Root Cause

Speak in the Language of Business


16



Remember– This is never a technology project – it is a business project using

technology.

Speak in their terms

– Call center?

Impact to Average Speed to Answer

– Ordering System?

Impact to Average Processing Time

– Manufacturing?

Impact to Work in Process (WIP)



Know your audience– Do they want a short “Yes” / “No” answer or long story?

Don’t conflate your answer

Answer the question behind the question

Back it up with data – current data

If you don’t know – that is OK – Tell them you’ll find out and get back to them

Remember to tell the story behind the data

17


“Even though the overall defect backlog increased during the last month, we see the Critical defect backlog decreasing.

This is a good sign that Critical defects are getting the priority they need. We still have many older then 90 day Critical defects, but this is a good sign that work is being done to address this.”

Create a safe space


18

Create a safe space


Testing / QA is an easy target– We are at the “end” of the project

– We are easily measured

Find a weekly time for the team to vent and support each other

– What is said in the room, stays in the room

Double check the pulse of the team:– What is working? What is not?

– What can we change? What should I (as test leader) change?

– What is blocking that I need to unblock?

Final Take Aways


19

Final Take Aways


Top Things to Remember1. You are not alone

2. Standardize, Standardize, Standardize

3. Build Allies and Monitor your Friends

4. Measure your Progress and Treat Data as Gold

5. Tell the Story in Business Teams

6. Protect the Team

Q & A


key strategies to survive the mega test program

Technology