the pathologies of failed test automation projects

W3 Test Automation

5/1/2013 11:30:00 AM

The Pathologies of Failed Test

Automation Projects

Presented by:

Michael Stahl

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Michael Stahl

Michael Stahl is a software validation architect at Intel, working with a team that validates Intel's graphics hardware drivers. In this role, Michael defines testing strategies and work methodologies for test teams, and tests part of the product himself - the work he enjoys most. As a twenty-two year veteran and senior engineer, he has been witness to what works - and what doesn’t - in test strategies, communications, and automation. An avid teacher, Michael enjoys sharing his observations with others. Contact Michael at michael.stahl@intel.com and review previous conference presentations on his website testprincipia.com.

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)

The Pathologies of failed

Test Automation Projects

Michael Stahl, Intel Apr 2013

• Automation failure patterns

• What can we do?

• Summary

On the Menu… 2

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)

Disclaimers

Names and brands referenced herein may be claimed as the property of

third parties

The views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer

Information in this presentation is provided “AS IS” without any warranties or representations of any kind

Once upon a time…

A common automation story… 4

Mr. Otto Mate Cool!

Can you make it…

Otto’s Team

I can automate my work!...

Mr. Otto Mate

Perfect!

Can you add…

Otto’s Team

Piece of cake!

Mr. Mann a. Ger

What a wonderful

world!...

A common automation story…

Mr. Otto Mate

Otto! The

last release

fails!... Otto’s Team

Arggghhh! Fix Fix Fix

Mr. Mann a. Ger

Makes sense… Sir! I need time! I need help!

… and mates

Mr. Otto Mate … and mates

… and mates

Mr. Oto Mate … and mates

Let’s REDESIGN!!!

#$&@***!!!

Mr. Mann a. Ger

A pattern emerges…

Pattern #1: Mushrooming

Stage 1 – Small & Local

Single User / Developer

Simple Tool

A pattern…

Stage 2 – Generalization

Multiple Users / Single Developer

Enhanced Tool

A pattern…

Multiple Users &

Developers

Complicated Tool

Stage 3 – Staffing

A pattern…

Test Case

Management

Multiple Users &

Developers

Stage 4 – Non-core features

A pattern…

Stage 5 – Overload

Arghhh!

Pattern #2: The Competition

Team B

Pattern #2 – The Competition (ver. 1) 17

Team A

Team B

Team A

Team C

Team D !!!

Team B

Team A

Team C

Team D

!!! …

Pattern #3: The Night Run Fallacy

Pattern #3 – The Night Run Fallacy 21

Night Time

Test Time

Add a snapshot – or a movie snippet – of PAVE

Corporate Truism:

It’s easier to get budget for machines than for more testers

Add a snapshot – or a movie snippet – of PAVE

Test Automation Truism:

Machines create work for more testers

The Tragedy of the Commons

multiple individuals…

will ultimately deplete a shared limited resource…

even when it is not in their long-term interest

Garrett Hardin, Science, 1968

Pattern #4: Going for the Numbers

Pattern #5 – Going for the Numbers

Robustness is Invisible

Pattern #4 – Going for the Numbers

Pattern #5: The Magician Apprentice Syndrome

Pattern# 5 – The Magician Apprentice 32

Recap: The Patterns

Mushrooming

The Competition

The Night time Fallacy

Going for the Numbers

The Magician Apprentice

So... 34

What can we do?

Are the patterns… 35

Unavoidable ??

Counter measures 36

Mushrooming 37

How to use this information?

GPS Locate the stage you are at

Get directions for the way out

Start right and avoid

the wrong turns

Alert Signals & Counter Measures

Alert Signals 39

Alert Signals

Failing the project

Alert Signals

Single feature test tool?

The creator is the user?

“Skunk works”?

Key words: “Tool”

“Utility”

Alert Signals: Stage 1 Small, local, feature-centered

Ensure the following… and relax:

Code control

Documentation User Manual (Usage line…)

“Green” in the code

High level design

Counter Measures: Stage 1 Small, local, feature-centered

Additional features?

Multiple users?

Automation web site?

>25% of the tester’s time?

Key words: “Use by other testers”

“Common Libraries”

Alert Signals: Stage 2 Generalization

Counter Measures: Stage 2 Generalization

“The hardest part of building a software system is deciding

precisely what to build... No other part of the work so cripples

the resulting system if done wrong. No other part is more

difficult to rectify later” - Fred P. Brooks (author of “The Mythical Man-Month”)

Stage 1 measures

Strategy

Architecture

Lightweight PM Version control

Scope control

Bugs & Requests database

Counter Measures: Stage 2 Generalization

Alert Signals: Stage 3

Requests for additional heads?

Automation F2F?

Tool-related delays in execution?

Key words: “Tool Owner”; “Automation team”

“Framework”; “Infrastructure”

“Roll back”

“Bug fix release”

47 Institutionalization and staffing

Stage 1, 2 counter measures

Management level decision time Programming language

Framework focus

Code and release management

Acceptance tests

Skillset development

Metrics

Counter Measures: Stage 3

Metrics suggestions

Automation framework quality Number of false fails

Framework’s test results; Bug trends

ROI Number of runs

Invested effort by type (new,

maintenance, rewrite)

Number of bugs found by Automation (?)

Generic features?

Test log wading?

False fails?

Key words: “test suite / cycle generation”

“robustness enhancement”

“setup issues”

Alert Signals: Stage 4 Change of focus: Technology Management

Stage 1, 2, 3 counter measures

Build VS Buy

Re-architect Core / Non-Core

Solid infrastructure

Influence upstream Testability hooks

Design for Test (automation)

Counter Measures: Stage 4 Change of focus: Technology Management

Stage 4 – Non-core features

Build VS Buy?

(hint: Buy)

See: http://www.stickyminds.com/s.asp?F=S17601_COL_2

Build (VS Buy) if… Competitive edge

Existing expertize

Core competency

Cheaper; Faster

Good use of resources

Acceptable risk

Long term support

Counter Measures: Stage 4 Change of focus: Technology Management

Main source: Allen Eskelin http://www.informit.com/articles/article.aspx?p=21775

Maintenance & logistics overload?

Limitations overplay?

Loss of credibility?

Stage 1 initiatives?

Key words “Did it fail in manual test?”

“Architecture limitation”

“refactoring”; “redesign”

“…I can write a small program…”

Alert Signals: Stage 5 Maintenance overload; Re-design

Loss of credibility

Alert Signals: Stage 5 Maintenance overload; Re-design

Options:

Continue…

Give up problematic areas

Partial return to Stage 1

“We value Robustness over New Features”

Prepare for re-design – with a new map...

Counter Measures: Stage 5 Maintenance overload; Re-design

Locate the stage you are at

Get directions for the way out

Be alert for Alerts

Identify your stage

Analyze your situation

Implement Counter

Measures

How to Use this information? 58

Plan your trip

Be alert for Alerts

Analyze your situation

Implement Counter

Measures

How to Use this information? 59

Start right and avoid

the wrong turns

The Competition 60

Salvageable up to stage 2

Stage 3 and up: Merge the teams

EOL both tools

Start a 3rd

Plugin Architecture

Fix the main problem

Accept, encourage Stage 1 stuff…

Maintaining vigil:

Avoid ver.1 pattern

The Night Run Fallacy 64

Never forget test time, test efficiency

Balance test skills VS automation skills

Allocate machines or machine time

Going for the Numbers 66

Pattern #4 – Going for the Numbers 67

Change how you count “Done”

The Magician Apprentice Syndrome 68

http://www.youtube.com/watch?v=zlnz1rSJj7Y

Automation should not necessarily mimic humans…

… it should get the job done

http://www.youtube.com/watch?v=qDVYIT85ntg

Holistic approach VS “test after test”

Re-think, re-strategize, re-evaluate before automating

Identify Actions, Keywords (KDT)

Summary 73

The Patterns

Mushrooming

The Competition

The Night time Fallacy

Going for the Numbers

The Magician Apprentice

The patterns are pervasive

Almost inevitable

Awareness is key (engineers; managers)

Driven by organizational and human nature

The solution is only partially technical

Summary

Thank You!

Questions time…

michael.stahl@intel.com

www.testprincipia.com

the pathologies of failed test automation projects

Technology

pathologies infectieuses pathologies rachidiennes...

common gi pathologies

pathologies oculaires courantes

wargame pathologies - dtic

pathologies de chaussées

pathologies digestives

cardiovascular pathologies

differencial pathologies khbw

pathologies associées à la grossesse pathologies...

pituitary pathologies simplified

why test automation projects are failing?...•re-run failed...

pathologies chroniques

piscine collective pathologies

cours n°13 : pathologies bronchiques, pathologies

pathologies métaboliques - fac.umc.edu.dz

tracheal pathologies

les pathologies routières

pathologies general post

bone pathologies - eng.tau.ac.il

skin pathologies