(almost) everything i know about testing i learned playing poker - matt eakin

NOTICE: Proprietary and ConfidentialThis material is proprietary to Centric Consulting, LLC. It contains trade secrets and information which is solely the property of Centric Consulting, LLC. This material is solely for the Client’s internal use. This material shall not be used, reproduced, copied, disclosed, transmitted, in whole or in part, without the express consent of Centric Consulting, LLC. © 2013 Centric Consulting, LLC. All rights reserved

(Almost) Everything I know about testing I learned playing poker

Matthew Eakin, National Automation [email protected]@MatthewEakinMatthewEakin

At Centric, we deliver unmatched client experiences that center on collaboration. Working together, we implement lasting solutions that work. Solutions that matter.

mailto:[email protected]

Does anyone want to play some poker?

Need 6 “volunteers”To play Texas Hold ‘Em

.

Texas Hold ‘Em & Software Testing

Some testing concepts we are going to see in action:

1.Knowledge of the domain

2.Risk Analysis

3.Planning vs. Writing a Test Plan

4.Adjusting your efforts as you go

5.Knowledge gathering

6.Estimation

7.Resource allocation

What we are going to do…

At least 2 hands of Texas Hold ‘Em

Hand 1: “Agile”•Constant evaluation of risk•Constant knowledge gathering•Constant planning and adjusting

Hand 2: “Waterfall”•Risk evaluation conducted up-front (even though our knowledge is limited)•Knowledge is gathered up-front (even though our knowledge is limited)•Planning conducted up-front (even though our knowledge is limited)•Adjusting is discouraged. You MUST follow the Plan!!

Hand 3 and beyond: You choose the approach…

Hand 1:Agile

Knowing your Table/Team

What is their Knowledge of the Domain?

Relationship to testing

What is your testing teams knowledge of each domain?•Knowledge of Testing

•Knowledge/experience with the process (agile or waterfall)

•Technical Knowledge

•Knowledge of the Application

Assessing Risk

Risk Assessment: Key concept to both Poker & Software Testing•In Poker: assess your probability of winning the hand with your cards

•Probability of your cards•Probability of opponents cards (what you think they have)•Probability with chip stacks

•Conduct risk analysis on every card drop

•Conduct risk analysis on every bet

•Conduct risk analysis on every player action

Rules of Texas Hold ‘Em (in a nutshell)

•All players are dealt 2 cards face down. You can look at them, but don’t show anyone.

•A round of betting will follow

•5 additional cards will be dealt into the “Community” face up.•3 cards will be dealt first followed by a round of betting•1 card will be dealt followed by a round of betting•1 card will be dealt followed by a final round of betting

•Make the best 5 card hand

•Hand order:•Royal flush – 10-A all same suit•4 of a kind – 4 cards match, i.e. 9-9-9-9-3•Full House – 3 of one card, 2 of another, i.e. K-K-K-5-5•Flush – all cards same suit, i.e. 2-3-6-8-Q all clubs•Straight – all cards in a row, i.e. 4-5-6-7-8•3 of a kind – 3 cards match, i.e. K-K-K-5-7, 2-2-2-7-9•2 pair – i.e. K-K & 6-6 in your hand•Pair – 2 cards match, i.e. K-K, 6-6•High card in hand

The Deal

•Everyone is dealt 2 cards

•At this point you know very little about the deck of cards or what cards other players could potentially have.

10

Assessing Risk

Risk Assessment: Key concept to both Poker & Software Testing•In Poker: assess your probability of winning the hand with your cards

•Probability of your cards•Probability of opponents cards (what you think they have)•Probability with chip stacks

•Conduct risk analysis on every card drop

•Conduct risk analysis on every bet

•Conduct risk analysis on every player action

Relation to testing

Testing is Risk Minimization•In Testing: use risk analysis to determine what to build/test first

•Conduct risk analysis on product backlog

•Conduct risk analysis on sprint backlog

•Conduct risk analysis on every User Story

•Conduct daily risk analysis on work in progress

Betting

•As each player bets you begin to know more about:

• Their skill level• Their aversion to risk• What they have in their hand

•Hint: players won’t bet unless they think they can win the hand, right? WRONG!!!

• Beware the bluff

13

• Everyone bets

Relation to Testing

14

Resource Allocation• As a project progresses you learn more about:

• the risks associated with these changes of the application, • your testing team• the entire project team

• You might want to allocate more/less resources to one area of the app

• You might want to focus more on automation

Bluffing• With every code drop you learn more about the code

base/application • Is the project really 80% done?• Are the developers bluffing?

• Does this change the risk? Change resource allocation?

The Flop

•The flop•3 cards are dealt face-up. All players can use these cards to create the best 5 card hand possible.

•You now know a little more about the deck of cards.

•You conduct another risk assessment of your now 5 cards and determine if you should bet or fold.

15

Relation to Testing

16

Planning“Planning is everything. Plans are nothing.”–Field Marshal Helmuth Graf von Moltke

• Plan for (almost) every possible outcome. When it happens you are ready for it

• Do you have the tools/equipment/skills to handle each outcome?

• What will you do when only 50% of your scripts pass? What if 100% on first pass?

• What if development delivers code late? What if scope changes? What if your test environment isn’t available on day 1 of testing?

• Many “risks” are identified in a test plan. But what are you going to do if they actually happen?

• Can you plan out the rest of the hand?

The Flop - betting

17

• Everyone bets– As each player bets continue to conduct

risk assessments on them. Does their betting indicate they have a good hand? A bad hand?

The Turn & the Turn Bet

•The Turn•1 card is dealt face-up (4 total in the community cards).

18

• Everyone bets– Can you figure out what hand everyone has yet?

– Can you predict your final hand?

Relation to Testing

19

Estimating & Predicting• As a project progresses can you predict how/when it will end?

• Can you predict your final hand?• Can you predict opponents final hand?• Can you estimate, based on your predictions, how many

chips you will end up with?

• Tools• Burn-up/Burn-down charts• %pass or # of defects with every code drop

• Very important for testers to be able to predict the future & estimate

• Discuss bad news early• Critical team discussions

The River & the Final Bet

•The river•The final card is dealt face-up into the community cards (there are now 5).

20

• Final bets– Your last chance to get everyone's $$$

The Show

•Before we show our hands…•Can you guess each players hand?•Who do you think won the hand?

•The show•Everyone shows their cards.•Best hand takes the pot.

21

What we learned…

Understand your teams knowledge of the domain

Constant risk analysis

Adjust resources as needed

Planning is critical

Can you predict the outcome of a project accurately?

22

Hand 2:Waterfall

The 2nd Hand – the “Waterfall Hand”

We are going to play this hand as if we were testing in a waterfall project.

All planning will be done up-front rather than as we go.

All risk analysis will be conducted up-front rather than as we go.

Resources will be allocated up-front rather than as we go.

Your knowledge of the requirement is assumed to be 100%.

24

The Test Plan

Each player is given a Test Plan which they must follow EXACTLY for the entire round.

Plan will include:•What every other players cards will be•Cards in the Board will be revealed.•Did you win?

25

Sample Test Plan

Deal: Player 1 will be dealt Q, 10Player 2 will be dealt 9, 4Player 3 will be dealt 3, 9 Player 4 will be dealt 6, 7Player 5 will be dealt 8, 10Player 6 (you) will be dealt A, K

Flop: Cards will be 5, 8, 9

Turn: Card will be Q

River: Card will be J

You will end up with a Straight A, K, Q, J, 10You will lose to Player 4 who has a Straight Flush 5 , 6, 7, 8, 9

Relation to Testing

27

Estimating & Predicting in Waterfall• Every Test Manager has had to provide a resource estimate for

a project they know almost nothing about. • Have your estimates every been accurate? +_ 50%?• Estimates are very difficult when there is so much unknown

• Based on past experiences you can start getting closer on you estimates

• Tools can help

Betting

•As each player bets you begin to know more about:

• Their skill level• Their aversion to risk• What they have in their hand

•Hint: players won’t bet unless they think they can win the hand, right? WRONG!!!

• Beware the bluff

28

• Everyone bets for all rounds

The Deal

•Everyone is dealt 2 cards•Players must stick to their Test Plan.

29

• Are there any problems with the Test Plan?– Too bad, you must stick to the Test Plan!!!

• Did anything happen that was not covered in the Test Plan?– How did you react?

The Flop

•3 cards are dealt to the Board•Players must stick to their Test Plan.

30

• Are there any problems with the Test Plan?– Too bad, you must stick to the Test Plan!!!

Change Requests

31

A change request has been introduced.

Since the code delivered was not as expected, numerous questions were asked of the business/BA.

Some requirements have been changed and new ones have been introduced.

Result:

• Each player can now change their Test Plan– Conduct a new Risk Assessment on each Player.

– Evaluate your own stack and create a new betting plan for the Turn and River.

The Turn & the Turn Bet

•The Turn•1 card is dealt face-up (4 total in the community cards).

32

• Everyone bets– Can you figure out what hand everyone has yet?

– Can you predict your final hand?

The River & the Final Bet

•The river•The final card is dealt face-up into the community cards (there are now 5).

33

• Final bets– Your last chance to get everyone's $$$

The Show

•Before we show our hands…•Can you guess each players hand?•Who do you think won the hand?

•The show•Everyone shows their cards.•Best hand takes the pot.

34

What we learned…

Writing out a Test Plan based on critical assumptions we have very little information about is dangerous.

A Test Plan does not fit every situation.

Following a Test Plan, even though it doesn’t match reality, will result in disaster for the tester.

35

NOTICE: Proprietary and ConfidentialThis material is proprietary to Centric Consulting, LLC. It contains trade secrets and information which is solely the property of Centric Consulting, LLC. This material is solely for the Client’s internal use. This material shall not be used, reproduced, copied, disclosed, transmitted, in whole or in part, without the express consent of Centric Consulting, LLC. © 2013 Centric Consulting, LLC. All rights reserved

Thank you

Test Plan Player 1

Deal: Player 1 will be dealt K, 10Player 2 will be dealt 9, 4Player 3 will be dealt 3, 9 Player 4 will be dealt 6, 7Player 5 will be dealt 8, 10Player 6 will be dealt A, K




You will end up with a Straight K, Q, J, 10, 9You will lose to Player 4 who has a Straight Flush 5, 6, 7, 8, 9

Test Plan Player 2

Deal: Player 1 will be dealt Q, 10Player 2 will be dealt 6, 7Player 3 will be dealt 3, 9 Player 4 will be dealt 9, 4Player 5 will be dealt 8, 10Player 6 will be dealt A, K




You will end up with a Straight Flush 5 , 6, 7, 8, 9You WIN!!!!!

Test Plan Player 3





You will end up with a Pair 9, 9, Q, J, 8You will lose to Player 4 who has a Straight Flush 5 , 6, 7, 8, 9

Test Plan Player 4






Test Plan Player 5

Deal: Player 1 will be dealt Q, 10Player 2 will be dealt 9, 4Player 3 will be dealt 8, 10Player 4 will be dealt 6, 7Player 5 will be dealt 3, 9 Player 6 will be dealt A, K




You will end up with a Pair 9, 9, Q, J, 8You will lose to Player 4 who has a Straight Flush 5 , 6, 7, 8, 9

Test Plan Player 6

Deal: Player 1 will be dealt Q, 10Player 2 will be dealt 9, 4Player 3 will be dealt 3, 9 Player 4 will be dealt A, KPlayer 5 will be dealt 8, 10Player 6 will be dealt 6, 7





(almost) everything i know about testing i learned playing poker - matt eakin

Technology