the pathologies of failed test automation projects

W3 Test Automation

5/1/2013 11:30:00 AM

The Pathologies of Failed Test

Automation Projects

Presented by:

Michael Stahl

Intel

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Michael Stahl

Michael Stahl is a software validation architect at Intel, working with a team that validates Intel's graphics hardware drivers. In this role, Michael defines testing strategies and work methodologies for test teams, and tests part of the product himself - the work he enjoys most. As a twenty-two year veteran and senior engineer, he has been witness to what works - and what doesn’t - in test strategies, communications, and automation. An avid teacher, Michael enjoys sharing his observations with others. Contact Michael at [email protected] and review previous conference presentations on his website testprincipia.com.

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)

The Pathologies of failed

Test Automation Projects

Michael Stahl, Intel Apr 2013

© Michael Stahl, 2013, all rights reserved

• Automation failure patterns

• What can we do?

• Summary

On the Menu… 2

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)


Disclaimers

Names and brands referenced herein may be claimed as the property of

third parties

The views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer

Information in this presentation is provided “AS IS” without any warranties or representations of any kind


Once upon a time…

A common automation story… 4

Mr. Otto Mate Cool!

Can you make it…

Otto’s Team

I can automate my work!...


5

Mr. Otto Mate

Perfect!

Can you add…

Otto’s Team

Piece of cake!

Mr. Mann a. Ger

What a wonderful

world!...

A common automation story…


6

Mr. Otto Mate

Otto! The

last release

fails!... Otto’s Team

Arggghhh! Fix Fix Fix

Mr. Mann a. Ger

Makes sense… Sir! I need time! I need help!

… and mates



7

Mr. Otto Mate … and mates

… and mates

… and mates



… and mates

… and mates

8

Mr. Oto Mate … and mates

Let’s REDESIGN!!!

#$&@***!!!

Mr. Mann a. Ger



A pattern emerges…

9


Pattern #1: Mushrooming


11

Stage 1 – Small & Local

Single User / Developer

Simple Tool

A pattern…


Stage 2 – Generalization

12 12

Multiple Users / Single Developer

Enhanced Tool

A pattern…


13 13

A pattern…

Multiple Users &

Developers

Complicated Tool

Stage 3 – Staffing


14 14

A pattern…

Test Case

Management

Multiple Users &

Developers

Stage 4 – Non-core features


15 15

A pattern…

Stage 5 – Overload

Arghhh!


Pattern #2: The Competition

16


Team B

Pattern #2 – The Competition (ver. 1) 17

Team A


Team B


Team A

Team C

Team D !!!


Team B


Team A

Team C

Team D

!!!

!!!

!!! …

…

…


Pattern #3: The Night Run Fallacy


Pattern #3 – The Night Run Fallacy 21



Night Time

Test Time



Add a snapshot – or a movie snippet – of PAVE

Corporate Truism:

It’s easier to get budget for machines than for more testers



Add a snapshot – or a movie snippet – of PAVE



Test Automation Truism:

Machines create work for more testers


The Tragedy of the Commons

multiple individuals…

will ultimately deplete a shared limited resource…

even when it is not in their long-term interest

Garrett Hardin, Science, 1968

http://upload.wikimedia.org/wikipedia/commons/8/8f/Lacanja_burn.JPG


Pattern #4: Going for the Numbers


28


29

Pattern #5 – Going for the Numbers


Robustness is Invisible

30

Pattern #4 – Going for the Numbers


Pattern #5: The Magician Apprentice Syndrome


Pattern# 5 – The Magician Apprentice 32


33

Recap: The Patterns

Mushrooming

The Competition

The Night time Fallacy

Going for the Numbers

The Magician Apprentice


So... 34

What can we do?


Are the patterns… 35

Unavoidable ??


Counter measures 36


Mushrooming 37


How to use this information?

GPS Locate the stage you are at

Get directions for the way out

Map

Start right and avoid

the wrong turns

38

Alert Signals & Counter Measures


Alert Signals 39


40

Alert Signals


Failing the project

Failing the project

Failing the project

Failing the project

Failing the project

41

Alert Signals


Single feature test tool?

The creator is the user?

“Skunk works”?

Key words: “Tool”

“Utility”

42

Alert Signals: Stage 1 Small, local, feature-centered


Ensure the following… and relax:

Code control

Documentation User Manual (Usage line…)

“Green” in the code

High level design

43

Counter Measures: Stage 1 Small, local, feature-centered


Additional features?

Multiple users?

Automation web site?

>25% of the tester’s time?

Key words: “Use by other testers”

“Common Libraries”

44

Alert Signals: Stage 2 Generalization


45

Counter Measures: Stage 2 Generalization

“The hardest part of building a software system is deciding

precisely what to build... No other part of the work so cripples

the resulting system if done wrong. No other part is more

difficult to rectify later” - Fred P. Brooks (author of “The Mythical Man-Month”)


Stage 1 measures

Strategy

Architecture

Lightweight PM Version control

Scope control

Bugs & Requests database

46

Counter Measures: Stage 2 Generalization


Alert Signals: Stage 3

Requests for additional heads?

Automation F2F?

Tool-related delays in execution?

Key words: “Tool Owner”; “Automation team”

“Framework”; “Infrastructure”

“Roll back”

“Bug fix release”

47 Institutionalization and staffing


48 48


Stage 1, 2 counter measures

Management level decision time Programming language

Framework focus

Code and release management

Acceptance tests

Skillset development

Metrics


Counter Measures: Stage 3



Metrics suggestions

Automation framework quality Number of false fails

Framework’s test results; Bug trends

ROI Number of runs

Invested effort by type (new,

maintenance, rewrite)

Number of bugs found by Automation (?)



Generic features?

Test log wading?

False fails?

Key words: “test suite / cycle generation”

“robustness enhancement”

“setup issues”

51

Alert Signals: Stage 4 Change of focus: Technology Management


Stage 1, 2, 3 counter measures

Build VS Buy

Re-architect Core / Non-Core

Solid infrastructure

Influence upstream Testability hooks

Design for Test (automation)

52

Counter Measures: Stage 4 Change of focus: Technology Management


Stage 4 – Non-core features

Build VS Buy?

(hint: Buy)

See: http://www.stickyminds.com/s.asp?F=S17601_COL_2


http://www.stickyminds.com/s.asp?F=S17601_COL_2

http://www.stickyminds.com/s.asp?F=S17601_COL_2


Build (VS Buy) if… Competitive edge

Existing expertize

Core competency

Cheaper; Faster

Good use of resources

Acceptable risk

Long term support

54

Counter Measures: Stage 4 Change of focus: Technology Management

Main source: Allen Eskelin http://www.informit.com/articles/article.aspx?p=21775


Maintenance & logistics overload?

Limitations overplay?

Loss of credibility?

Stage 1 initiatives?

Key words “Did it fail in manual test?”

“Architecture limitation”

“refactoring”; “redesign”

“…I can write a small program…”

55

Alert Signals: Stage 5 Maintenance overload; Re-design


Loss of credibility

56

Alert Signals: Stage 5 Maintenance overload; Re-design

Pre-check-in tests - external version.docx


Options:

Continue…

Give up problematic areas

Partial return to Stage 1

“We value Robustness over New Features”

Prepare for re-design – with a new map...

57

Counter Measures: Stage 5 Maintenance overload; Re-design


GPS

Locate the stage you are at

Get directions for the way out

Be alert for Alerts

Identify your stage

Analyze your situation

Implement Counter

Measures

How to Use this information? 58


Plan your trip

Be alert for Alerts

Analyze your situation

Implement Counter

Measures

How to Use this information? 59

Map

Start right and avoid

the wrong turns


The Competition 60



Salvageable up to stage 2

Stage 3 and up: Merge the teams

EOL both tools

Start a 3rd



Plugin Architecture



Fix the main problem

Accept, encourage Stage 1 stuff…

Maintaining vigil:

Avoid ver.1 pattern


The Night Run Fallacy 64



Never forget test time, test efficiency

Balance test skills VS automation skills

Allocate machines or machine time

65

65

65


Going for the Numbers 66


Pattern #4 – Going for the Numbers 67

Change how you count “Done”


The Magician Apprentice Syndrome 68



http://www.youtube.com/watch?v=zlnz1rSJj7Y



Automation should not necessarily mimic humans…

… it should get the job done



http://www.youtube.com/watch?v=qDVYIT85ntg



Holistic approach VS “test after test”

Re-think, re-strategize, re-evaluate before automating

Identify Actions, Keywords (KDT)


Summary 73


74

The Patterns

Mushrooming

The Competition

The Night time Fallacy

Going for the Numbers

The Magician Apprentice


The patterns are pervasive

Almost inevitable

Awareness is key (engineers; managers)

Driven by organizational and human nature

The solution is only partially technical

75

Summary


Thank You!

Questions time…

[email protected]

www.testprincipia.com