Property-based Testing for Everyone

Dmitriy Morozov / Zoomdata Tech Lunch

Why is testing hard?

Example-based Testing

• set up one input scenario • run the code under test • check the output

(and any other effects the code is supposed to have)

Imagine we needed to implement a custom ADD

function

@Testpublic void testSum() { assertThat(sum(1, 2), equalTo(3));}

private int sum(int a, int b) { if (a == 1 && b == 2) { return 3; } else { return 0; }}

@Testpublic void testSum() { assertThat(sum(2, 2), equalTo(4));}

private int sum(int a, int b) { if (a == 1 && b == 2) { return 3; } else if (a == 2 && b == 2) { return 4; } else { return 0; }}

Enterprise Developer

@Testpublic void testSum3() { // When I add two numbers I expect their sum for (int i = 0; i < 100; i++) { int x = randomInt(); int y = randomInt(); assertThat(sum(x, y), equalTo(x + y)); }}

@Testpublic void testSum3() { // When I add two numbers I expect their sum for (int i = 0; i < 100; i++) { int x = randomInt(); int y = randomInt(); assertThat(sum(x, y), equalTo(x + y)); }}

@Testpublic void testSumUsingCommutativityProperty() { // Changing the order of arguments // shouldn’t change their sum for (int i = 0; i < 100; i++) { int x = randomInt(); int y = randomInt(); assertThat(sum(x, y), equalTo(sum(y, x))); }}

// unfortunately that's not enoughprivate int sum(int a, int b) { return a * b;}

@Test// `x + 2` is the same as `x + 1 + 1` public void testSumWithAdditiveProperty() { for (int i = 0; i < 100; i++) { int x = randomInt(); int result1 = sum(sum(x , 1), 1); int result2 = sum(x, 2); assertEquals(result1, result2); }}

// WTF?! private int sum(int a, int b) { return 0;}

@Test// `x + 1` = `x ̀public void testSumWithIdentityProperty() { for (int i = 0; i < 100; i++) { int x = randomInt(); assertEquals(x, sum(x, 0)); }}

Testing Specification with properties

• Commutativity property • Associativity property • Identity property

Testing Specification with properties

By using specifications, we have understood the requirements in a deeper way

QuickCheck

QuickCheck in a nutshell

John Hughes - Testing the Hard Stuff and Staying Sane

Benefits

• Less time spent writing test code - One property replaces many tests

• Better testing- Lots of combinations you’d never test by hand

• Less time spent on diagnosis - Failures minimized automatically

3,000 pages of specifications 20,000 lines of QuickCheck 1,000,000 LoC from 6 suppliers 200 bugs 100 bugs in the standard

junit-quickcheck

@RunWith(JUnitQuickcheck.class) public class AdditionPropertyTest { @Property public void commutativityProperty(int x, int y) { assertThat(sum(x, y), equalTo(sum(y, x))); } @Property public void additiveProperty(int x) { int result1 = sum(sum(x , 1), 1); int result2 = sum(x, 2); assertEquals(result1, result2); } @Property public void identityProperty(int x) { assertEquals(x, sum(x, 0)); } }

java.lang.AssertionError: Property identityProperty falsified via shrinking: expected:<1> but was:<0> Shrunken args: [1] Original failure message: [expected:<98> but was:<0>] Original args: [98] Seeds: [7329045779044418918]

@RunWith(JUnitQuickcheck.class) public class MoneyChangerTest { @Property public void sumOfCoinsEqualsAmount( @InRange(min = "0", max = "500") int amountToChange, Set<@InRange(min = "1", max = "100") Integer> denominations) { assumeThat(denominations.size(), greaterThan(0)); denominations.add(1); MoneyChanger moneyChanger = new MoneyChanger(); List<Integer> coins = moneyChanger.change(amountToChange, denominations); int sum = coins.stream().mapToInt(coin -> coin).sum(); assertThat(sum, equalTo(amountToChange)); }}

JSVerifynpm install jsverify

describe("sort", function () { jsc.property( "idempotent", "array nat", function (arr) { return _.isEqual(sort(sort(arr)), sort(arr)); });});

How to handle state?

How to handle state?• Generate actions changing the state • Apply each action to the state, if possible • After each application, verify the model

Where can we benefit from

property-based tests?

Spark Proxy on DataFrame API

• Generate random DataReadRequest

• Execute request with RDD-based code

• Execute same request with DataFrame based code

• Compare the results

SDK

• Send START_VIS

• Generate random request, change metric, group-by, filter, sort, etc

• Validate responses based on the test model

• Send STOP_VIS

• QuickCheck • Testing Telecoms Software with Quviq QuickCheck • Testing the hard stuff and staying sane • The Mysteries of Dropbox • Jepsen IV: Hope Springs Eternal • Java Examples from this talk