property based testing - less is more
DESCRIPTION
Introduction to property-based testing.TRANSCRIPT
Why we write tests
• Programs are implemented based on specifications
• Tests are used to verify against specification
• Fail fast
Unit test
• First line of defend
• Test small modules in isolation
• Fast, repeatable
• Facilitates changes
• Documentation
Unit test - Example
• Testing sorted method on a List of Int
• empty list => []
• already-sorted list => [1,2,3,4,5,6,]
• reversed-sorted list => [6,5,4,3,2,1]
• arbitrary list => [1,4,3,6,5,2]
Problems• Data is hard-coded
• Are we missing out edge cases?
- long list?
- What about boundaries? (-21474836478 2147483647)
• Are we biased towards the implementation?
• Spend considerable amount of time writing/fixing tests
The assertion hell
“Testing shows the presence, not the absence of bugs” E.W. Dijkstra
http://googleresearch.blogspot.sg/2006/06/extra-extra-read-all-about-it-nearly.html
Property-based test
• Testable Specification + Random Testing
• Focus on the specification
• Let the testing library take care of generating test cases
Specification as code
Properties of sorted
• every element is less than or equal to the next element in the sorted list sorted[i] <= sorted[i+1] for i in 1..n-1
Properties of sorted
• should retain all elements of original list sorted[i].size == original[i].size sorted.contains(x) for x in original
Random Testing
• Random generator with emphasize on zero, extremities, +/- unity, special values, etc
• Allow custom distribution
Generator
• DSL for generating random test data
- forAll, chooseFrom, oneOf, constant, etc
• Allow custom-type
• Composable - allow complex data structure e.g. Tree
Test auto-shrink
• When encounter a failed test, the PBT library will try to minimize it
• Reduce effort on debugging
Advantages of PBT
• Let developer focus on specifications
• Reduce testing effort
• More coverage
• Make testing FUN!
Testing stateful system
• Stateful system is hard to test
- testing 1 component O(n)
- testing 2 components O(n^2)…
• Often focus on how components behave instead of the system as a whole
=> Need better abstraction
Testing stateful system with PBT
• Abstraction in PBT
- State: information about the system
- Command: interface that changes system from one state to another
- System Under Test(SUT) - the system we want to test
Property = SUT.command(preState) == postState
Testing stateful system
Testing a Counter
class Counter(n: Int)
+ inc() = { n += 1 }
+ dec() = { n -= 1}
+ get() = n
Testing a Counter
• State := n
• SUT := Counter(0)
• Commands
- Inc => State = n + 1; SUT.Inc()
-Dec => State = n - 1; SUT.Dec()
-Get => State = n;
-postCondition := SUT.Get() == n
Example - Restful Web Services
• 4 Commands: Create, Read, Update, Delete
• System under test: Resources
• State: Your Database Model
Example - Redis Client
• Testing a Redis-client https://github.com/rickynils/scalacheck/tree/master/examples/commands-redis
Example - LevelDb
• https://code.google.com/p/leveldb/issues/detail?id=44 => It is impossible for human to come up with a test case of 17 steps
Concurrency/Distributed System
• Have been used successfully in many projects
- Riak
- Akka
• Often discover bugs that Unit tests couldn't
PBT library
• Haskell - QuickCheck
• JVM(Scala/Java/Clojure) - Scalacheck (Java/Scala) - test.check (Clojure)
• Python - hypothesis
• Erlang - QuickCheck
• Javascript, C++, etc
PBT is not a silver bullet
• Writing PBT can be really hard if
- your program has side-effect
- it’s not trivial to verify the output correctness (e.g. optimal function)
- there are lots of duplication between specifications and implementations
Summary
• PBT let developer focus on and express specification in a concise format
• PBT reduces testing effort
• PBT can be used to test stateful system
• PBT works well with functional Programming Languages
Thank you!
https://github.com/hotienvu/pbt-scalacheck
@hotienvu