MODEL-BASED TESTING:

TODAY AND TOMORROW

Robert V. Binder

ETSI Model-Based Testing User Conference

Tallinn, Estonia

September 27, 2012

© 2012, Robert V. Binder. All Rights Reserved

Overview

• Is software eating the world?

•Bugs

•Black Swans

•Dragon Kings

•Multi-dimensional Testing

•Challenges

2

“Software is Eating the World”

-Marc Andreessen

3

Wall Street Journal, Aug 20 2011

4

Internet Trends. Mary Meeker, D10 Conference. May 2012

5

Internet Trends. Mary Meeker, D10 Conference. May 2012

6

Internet Trends. Mary Meeker, D10 Conference. May 2012

Embedded and Connected

2015 Connected Device Landscape: The Expanding Computing Continuum.

David Perlmutter, Intel. 2009

7

IT Disruption

Gartner Top Predictions for 2012: Control Slips Away, Daryl Plummer. March 2012

8

9

Internet Trends. Mary Meeker, D10 Conference. May 2012

Is Software really eating the world? Methods/Tools since Y2K

• 2nd gen IDE

• MBT, MDD

• Aspect OP

• Open Source

• UML 2.0

• Cont. Integration

• Web 2.0

• Agile

• ISO 12027

• LAMP Stack

• Many protocols

• HTML 5

• iOS, Android

• Big Data

• Software Def Netwks http://langpop.com/

Relative rank from ~2M data points

10

Engineering Change to Accelerate Product and Systems Development.

Christopher Rommel, VDC Research. August 2012

Is Software really eating the world?

11

Hardware is eating the world

Everything Computes & Connects

• 10x to 100x

• ten years

• 100x to 1000x

• twenty years

Software Methods and Tools

• ?X Productivity

• ?X Quality

• Last ten years?

• Last twenty years?

• Cheaper, better, faster

• Disrupt or die

• Cheaper, better, faster

• “Choose two”

12

Old bugs, new bugs

• No fundamental or

transformative

improvements in

software

• End of the uni-stack

• Multi-stack, mobile,

connected

• Inherently unstable

• Bugs still as likely, more code

• New failure modes

What do you think will happen?

13

Old Bugs

• Incorrect output

• Missing feature

• Abnormal end

• Missed deadline

• Bohr Bugs Predictable, repeatable

• Mandel Bugs Obscure, complex trigger

• Heisen Bugs Disappears if you look

14

The Jargon File

http://www.catb.org/jargon

Bohr Bugs

15

Mandel Bugs

16

Heisen Bugs

17

New Bugs

•Power Law Models

•Black Swans

•Dragon Kings

18

Power Law Models

• Many natural

phenomena

• “Fat Tails”

Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette

http://www.youtube.com/watch?v=FlTSbzOvKZI

19

Reliability Growth

The Lognormal Distribution of Software Failure Rates: Application to

Software Reliability Growth Modeling. Robert E. Mullen

20

Black Swans • Very rare

• Surprise

• Cannot predict

• Power law outliers

21

Software Black Swans

• Total eclipse

• Once every 360 years

• Multi-cycle race

conditions in hard

realtime systems

22

Dragon Kings

• Extreme Outlier

Beyond fat tail

• Latent Positive Feedback

• “Phase Shift”

Earthquakes

Material Fractures

Epileptic Seizures

Market Crashes

23

Sornett calls these exceptional events dragon

kings “to stress that we deal with a completely

different kind of animal, beyond the normal,

whose extraordinary characteristics [have]

profound significance.”

Dragon Kings

24

Latent Positive Feedback = Dragon King

25

New Failure Modes:

Amazon Cloud Outage

• Amazon web service Elastic Compute Cloud crashes (power failure)

• The Elastic Load Balancing (ELB) system frantically tries to assign workloads to available servers.

• As Amazon’s cloud rebooted, “a large number of ELBs came up in a state which triggered a bug we hadn’t seen before”

• ELB reaction: try to allocate more larger servers.

• Backlog in the “control plane” results.

• Demand from customers in unaffected availability zones continues

• System swamped and crashes again.

26

What kind of failure was this?

•Why didn’t test find it?

•What would we have to do to find

these kind of bugs?

27

Hic Sunt

Dracones

28

Here be Dragons

Latent positive feedback

External disruptors

Malicious attack target

Partial degradation

Provoked panic

Low testability

Multi-stack

Field-infeasible

Uncontrollable inputs

29

Looking under the Model-based Lampost?

• 9 in 10 MBT testers do

functionality only

• Climbs a tree, not the twigs

• Typically no load test

Logic errors and incorrect

assumptions are inversely

proportional to a path’s

execution probability.

Boris Beizer

30

Multi-Dimensional Testing

Behavior

• Realistic variation

• Markov event/data

• Include

• Fat finger

• Abuse cases

• Sneak paths

• Panics

Interfaces

• Drive all interfaces

• Disrupt resources

Load

• Vary input rate wrt time

Not possible without full automation

31

Test suite interleaves three kinds of variation

Interleave Behavior, Load, Interfaces

1

10

100

1000

10000

100000

1000000

10000000

X

-500.000

0.000

500.000

1000.000

1500.000

2000.000

2500.000

3000.000

3500.000

-5000 0 5000 10000 15000 20000 25000

Ev

en

ts P

er

Se

con

d

Time (seconds)

X

SUT

Test Verdict

Run Reports

Adapter

Adapter

Adapter

Adapter

Oracle

Comparator

Splainer

Simulator

=

32

Use Case A B C D E F G H I J K L

Hunting the Dragon King

• 0 You must be joking

• 1 Write if you get work

• 2 Even a broken clock is right…

• 3 One Sigma

• 4 Michael Jordan free throw (82%)

• 5 Sargent Preston

Common

Bugs

Black

Swans

Dragon

Kings

Fragile 1 0 0

Massively Manual 2 0 0

Test Driven Dev 3 1 0

Exploratory 3 1 0

Crowdsourced 2 0 0

Functional MBT 5 3 1

Multi-dimensional MBT 5 4 3

33

What have we learned?

• Information Technology is eating the world

• Common bug rates will not decline

• More new systems will have dragon kings

• Functional/model coverage MBT will not find

dragon kings

• Multi-dimensional MBT can find dragon kings

• Opportunity for test engineering to lead

34

May you live in interesting times …

Challenges for the MBT Community

Tool Providers

• Start shipping multi-

dimensional

solutions

MBT Test Engineers

• Learn to lead

• Start hunting your

dragon kings

35

Thank you !

36

rvbinder@gmail.com

www.robertvbinder.com

Content and Image Sources

37

First Generation of Re-Imagination

Android Phone Adoption has ramped even faster

Despite Tremendous Ramp so Far, Smartphone User Adoption…

Magnitude of Upcoming Change Will be Stunning…

Internet Trends. Mary Meeker, D10 Conference May 2012

http://kpcb.com/insights/2012-internet-trends

Embedded and Connected

2015 Connected Device Landscape: The Expanding

Computing Continuum. David Perlmutter, Intel. 2009

http://www.intel.com/pressroom/kits/events/idffall_2009/video.htm

IT Disruption

Gartner Top Predictions for 2012: Control Slips Away, Daryl

Plummer. March 2012

Is Software really eating the world?

Engineering Change to Accelerate Product and Systems

Development. Christopher Rommel, VDC Research. August

2012

Mandel Bug

http://www.incamouflage.com/2009/05/bug-camouflage/

Heisen Bug

http://www.adriennehedger.com/2009/07/june-bug-situation.html

http://openclipart.org/detail/145735/cartoon-fly-by-printerkiller

Power Law Models

Dragon-Kings, Black Swans and the Prediction of Crises.

Didier Sornette. International Journal of Terraspace Science and

Engineering 2009

http://www.youtube.com/watch?v=FlTSbzOvKZI

Reliability Growth

R. E. Mullen, “The Lognormal Distribution of Software Failure Rates:

Application to Software Reliability Growth Modeling,” International

Symposium on Software Reliability Engineering, 1998, p 134.

Black Swans

The Black Swan: the Impact of the Highly Improbable. Nassim

Taleb.

http://www.publicdomainpictures.net/view-

image.php?image=8248&picture=black-swan&large=1

Software Black Swans

http://apod.nasa.gov/apod/ap060404.html

Dragon Kings

http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/

Latent Positive Feedback = Dragon King

http://www.everythingselectric.com/forum/index.php?topic=244.0

Amazon Outage

Hidden bugs that made Amazon Web Service outage worse. Cnet

News, July 3, 2012.

http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-

made-amazon-web-service-outage-worse/

Hic Sunt Dracones

http://tomlytle.com/

Looking under the Model-based Lampost?

http://stconsultant.blogspot.com/2010/11/old-saying.html