the search for the perfect program

THE SEARCH FOR THE PERFECT PROGRAM

NATALLIE BAIKEVICH

@LU_A_JALLA

A ‘PERFECT PROGRAM’ IS …

- Maintainable

- Efficient

- Correct

A ‘PERFECT PROGRAM’ IS …

- Maintainable

- Efficient

- CorrectDoes anything else really matter without correctness?

THE PLAN- Assertions

- Random Testing

- Symbolic Execution

- Types

THE PLAN- Assertions

- Random Testing

- Symbolic Execution

- Types

1.Do something

2.Check it

3.Go to 1.

THE FAST TRACK

Write code -> Release

WELL, NOT THAT FAST

Dear <…>

…

We see XYZException!!!.............!

where n = enough for you to feel miserable

n times

OK, LET’S TEST IT

1. Choose your favourite library

(good chances it’s .*Unit)

2. Write some tests

aand

3. XYZException has gone

THE PROBLEM, PART 1

But…

- they only check what you think a program should do, not what it actually should do

- ... on what you think is all possible inputs

RANDOMIZE IT

Throw the thousands of inputs into your program with

- QuickCheck (Haskell)

- FsCheck (F#)

- ScalaCheck (Scala)

- or something else

DEMO

RANDOMIZE IT

FSCHECK- Expressive and

easy to use

- Checks your spec

- … in a large number

- … of randomly generated cases

“500 checks can’t be wrong!” © don’t remember who said that

THE PROBLEM, PART 2- Expressive and

easy to use

- Checks your spec

- … in a large number

- … of randomly generated cases

“500 checks can’t be wrong!” © don’t remember who said that

- Writing good tests might be tricky

- Checks your spec

- … in a large number

- … of randomly generated cases

500 checks can easily be wrong!

But:

DEMO

RANDOMIZE IT – WHAT’S MISSING?

I WILL BUILD MY OWN TEST GENERATOR

let f x y =

if x < 10 then x

else if x = 42 then

failwith “42”

else 1 / y

x < 10 or x >= 10

x = 42 or x <> 42

y = 0 or y <> 0

DEMO

QUOTATIONS

THE PROBLEM, PART 3

But it’s too complicated to do by hands:

- too many cases,

- lots of overlapping paths,

- difficult to solve when the number of variables grows,

- not everything is a quotation…

IDEA: USE A CONSTRAINT SOLVER

- Check out Z3 homepage

- Try Z3 in your browser

- LINQ to Z3 (ch9 video and a post by Bart de Smet)

Specially for fsharpers:

- Z3Fs on github

THERE’S A TOOL FOR THAT – MEET PEX

- Dynamic symbolic execution

- Analyses .NET instructions

- Uses constraint solver to find the inputs

PEX REFERENCES

- Project homepage

- Pex for fun in your browser

- Code Digger, addin for VS

- Code Hunt website

- Documentation, videos

and more

DEMO

PEX

TEST GENERATIONStatic:

- Conditional statements

- Check the formulas satisfiability

Dynamic:

- Collect the information during the program execution

- Unknown environments

- Enhanced values generation

- Better Performance

DYNAMIC EXECUTION EXAMPLE

int obscure(int x, int y) {

if (x == hash(y)) return -1; // error

return 0;

}

“Compositional Dynamic Test Generation”, Patrice Godefroid (paper)

THE PROBLEM, PART 4

Limitations:

- Nondeterministic cases (e.g. native code)

- Concurrency

- Constraint solver limitations

MORE AND LESS TYPES WITH F*

F* - an ML-like verification-oriented language

- F* project homepage

- GitHub repo

- Try F* in your browser

DEMO

TRYING F*

THE PROBLEM, PART 5

- currently under development

- issues on mono

- in more complex cases, the errors become quite cryptic

- you still need to come up with a way to define the requirements, at the type level

CONCLUSION

If you don’t know where you’re going, any road’ll take you there.

Lewis Caroll

QUESTIONS?