property-based testing for godly tests

Property-Based Testing for Godly Tests - Gabe Scholz @unbounce

!

Property-Based Testing for Godly Tests - Gabe Scholz @unbounce 3

! Gabe from Vancouver

🐙 @garbles

📬 [email protected]


careers.unbounce.com

4


Property-Based Testing for Godly Tests

5


Why do we write tests?


"Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence."

- Edsger Dijkstra


Tests prevent regressions!


Tests prevent regressions!(that you know about)


Example-Based Tests


Set of all possible inputs


Fuzz Tests


it('does not throw', () !=> { const arr = arrayOfRandomNumbers(); expect(() !=> sort(arr)).not.toThrow(); });


it('sorts unsorted arrays', () !=> { const arr = arrayOfRandomNumbers(); const sorted = !// ?????? expect(sort(arr)).toEqual(sorted); });


Property-Based Tests


it('keeps the same length', () !=> { const arr = arrayOfRandomNumbers(); expect(sort(arr).length).toEqual(arr.length); });

it('is idempotent', () !=> { const arr = arrayOfRandomNumbers(); expect(sort(sort(arr))).toEqual(sort(arr)); });

it('every member is less than or equal to the next one', () !=> { const arr = arrayOfRandomNumbers(); const sorted = sort(arr);

for (let i in sorted) { const current = sorted[i]; const next = sorted[i + 1];

if (next !!!== undefined) { expect(current).toBeLessThanOrEqual(next); } } });


"Example-Based tests are very narrow in their input and broad in their expectations."

- Jessica Kerr


Quickcheck


Testcheck Testcheck.js


Deliberate randomness


arrayOfRandomNumbers(); !// [1, 500, 6012], [45, 12, 579, 77, -8]!!...


Generators


import { gen } from 'testcheck';

gen.string; !// 'a1x', '', 'e4r'!!...

gen.int;

gen.bool;



gen.string;

gen.int; !// 0, -1, 3, 1!!...

gen.bool;



gen.string;

gen.int;

gen.bool; !// true, true, false!!...



const personGen = gen.object({ name: gen.alphaNumString, age: gen.posInt, }); !// {name: 'er54', age: 2}!!...

gen.array(gen.int);

const peopleGen = gen.array(personGen);



const personGen = gen.object({ name: gen.alphaNumString, age: gen.posInt, });

gen.array(gen.int); !// [3], []!!...

const peopleGen = gen.array(personGen);




gen.array(gen.int);

const peopleGen = gen.array(personGen); !// [{name: 'e', age: 1}]!!...


import { sample, gen } from 'testcheck';

sample(gen.array(gen.int), 20);


[ [], [], [2], [-1], [], [5, 0], [-6, -4, -5], [-2, -7, -6], [6, 7, -8, 7], [], [4, 2, 7], [-5, -2], [-10, -7, 0, 5], [8, -7, 3, 5, 11, -4, 5, -1, -6, -12, 9], [-6, 0, 4, 9, -2, 3, -4, -6, -6, 5, 3, 3], [-13, -11, -6, -13], [-16, 5, 0, 8, 16], [-6, -17, 14], [0, 13, -18, -15, -1, 0, 12, 1, 4, -15, 16, 5, -2, 6, -3], [7, 19, -1, 1, 12, 18], ];


import { sample, gen } from 'testcheck';


sample(personGen, 20);


[ { name: '', age: 0 }, { name: 'D', age: 1 }, { name: 'nn', age: 0 }, { name: 'r8', age: 3 }, { name: 'el1H', age: 0 }, { name: 't04', age: 2 }, { name: '47A15bF', age: 3 }, { name: 'N', age: 2 }, { name: 'DF18GqP', age: 7 }, { name: 'iKuySuF4', age: 5 }, { name: 'bU6', age: 0 }, { name: '1Zd', age: 11 }, { name: 'dR62wPY6K4jo', age: 5 }, { name: 'c57Rv3y', age: 5 }, !// !!... ];



check.it('is idempotent', gen.array(gen.int), arr !=> { expect(sort(sort(arr))).toEqual(sort(arr)); });



check.it('keeps the same length', gen.array(gen.int), arr !=> { expect(sort(arr).length).toEqual(arr.length); });


check.it('every member is less than or equal to the next one', gen.array(gen.int), arr !=> { const sorted = sort(arr);




Shrinking


{ result: false, 'failing-size': 2, 'num-tests': 3, fail: [-12, 1, -12, 4], shrunk: { 'total-nodes-visited': 2, depth: 1, result: false, smallest: [-2, -2] } };


Super useful!


Where does it fall apart in JavaScript?


type Person = { name: string, age: number, };

function savePerson(person: Person): Person




type ActivationRule = { id: string, schemaVersion: 10, ubCode: string, embUuid: string, trigger: Trigger, meta: RuleMeta, urlTargets: UrlTargets, displaySettings: DisplaySettings, published: PublishedState, frequency: Frequency, cookieTargets: CookieTargets, geoTargets: GeoTargets, referrerTargets: ReferrerTargets, event: Event, version: string, parentVersion!?: string, dimensions: Dimensions, integrations: Integrations };


type TriggerName = 'exit' | 'welcome' | 'timed' | 'scroll' | 'scrollUp' | 'click';

type $Trigger<N: TriggerName, PN, PV> = { name: $Subtype<N>, parameters: [{ name: $Subtype<PN>, value: $Subtype<PV> }] };

type Trigger = $Trigger<'exit', 'exit' | 'topMargin', '20px'> | $Trigger<'welcome', 'delay', '0'> | $Trigger<'timed', 'delay', string> | $Trigger<'scroll', 'scrollPercent', string> | $Trigger<'scrollUp', 'scrollUp', ''> | $Trigger<'click', 'clickId' | 'clickClass' | 'clickSelector', string>;


import type {ActivationRule as ActivationRuleV0} from './v00'; import type {ActivationRule as ActivationRuleV1} from './v01'; import type {ActivationRule as ActivationRuleV2} from './v02'; import type {ActivationRule as ActivationRuleV3} from './v03'; import type {ActivationRule as ActivationRuleV4} from './v04'; import type {ActivationRule as ActivationRuleV5} from './v05'; import type {ActivationRule as ActivationRuleV6} from './v06'; import type {ActivationRule as ActivationRuleV7} from './v07'; import type {ActivationRule as ActivationRuleV8} from './v08'; import type {ActivationRule as ActivationRuleV9} from './v09'; import type {ActivationRule as ActivationRuleV10} from './v10';

type AnyActivationRule = ActivationRuleV10 | ActivationRuleV9 | ActivationRuleV8 | ActivationRuleV7 | ActivationRuleV6 | ActivationRuleV5 | ActivationRuleV4 | ActivationRuleV3 | ActivationRuleV2 | ActivationRuleV1 | ActivationRuleV0;


const upgradeActivationRule = (version: number, rule: AnyActivationRule): AnyActivationRule

61


it('is backwards compatible', () !=> { const activationRule = createActivationRule(); const { schemaVersion } = activationRule;

const upgradedRule = upgradeActivationRule( MAX_SCHEMA_VERSION, activationRule );

const originalRule = upgradeActivationRule( schemaVersion, upgradedRule );

expect(originalRule).toEqual(activationRule); });


const activationRule10Gen = gen.object({ id: gen.string, schemaVersion: gen.return(10), ubCode: gen.string, embUuid: gen.string, trigger: gen.oneOf([ gen.object({ name: gen.return('welcome'), parameters: gen.object({ name: gen.return('delay'), value: gen.number }) }), !// !!... ]), !// !!... });


yarn add babel-plugin-transform-flow-to-gen !--dev

(https:!//github.com/unbounce/babel-plugin-transform-flow-to-gen)

https://github.com/unbounce/babel-plugin-transform-flow-to-gen


!// .babelrc

{ "presets": ["es-2015"], "plugins": ["syntax-flow"], "env": { "development": { "plugins": ["strip-flow-types"] }, "test": { "plugins": ["flow-to-gen", "strip-flow-types"] } } }


import { AnyActivationRule } from './types';

const activationRuleGen = AnyActivationRule.toGen();


import { AnyActivationRule, MAX_SCHEMA_VERSION } from './types';

check.it('is backwards compatible', AnyActivationRule.toGen(), (activationRule) !=> { const { schemaVersion } = activationRule;



expect(originalRule).toEqual(activationRule); } );

72


import { AnyActivationRule, MAX_SCHEMA_VERSION } from './types';

check.it('is backwards compatible', AnyActivationRule.toGen(), (activationRule) !=> { const { schemaVersion } = activationRule;



expect(originalRule).toEqual(activationRule); } );

73


Generating generators generates success

😀


Locks down type definitions

😀


function someFunc(arr: string[]): string

function someFunc(arr: [string, string, string]): string

76


function someFunc(arr: string[]): string

function someFunc(arr: [string, string, string]): string

77


Fuzz Testing😀


type State = { !// !!... };

type Action = {type: 'DO_A'} | {type: 'DO_B'};

function reducer(state: State, action: Action): Action;

createStore<State>(reducer, initialState);

79


check.it('is idempotent', sort.toGen(), (args) !=> { expect(sort(sort(!!...args))).toEqual(sort(!!...args)); });

80


Can't type check test code and non-standard

☹


Very, very, very slow☹


No silver bullet☹


TS and Flow can now force declarations to sync

☹😀


Property-Based Testing finds flaws in your code.

Lets make it easy to do in JavaScript.


Thank you!🎉

📬 [email protected]


References:

https:!//fsharpforfunandprofit.com/posts/property-based-testing/

https:!//fsharpforfunandprofit.com/posts/property-based-testing-2/

https:!//!!www.youtube.com/watch?v=shngiiBfD80

http:!//propertesting.com/

https:!//github.com/clojure/test.check

http:!//leebyron.com/testcheck-js/

Background Image:

https:!//!!www.toptal.com/designers/subtlepatterns/memphis-colorful/

https://fsharpforfunandprofit.com/posts/property-based-testing/

https://fsharpforfunandprofit.com/posts/property-based-testing-2/

https://www.youtube.com/watch?v=shngiiBfD80

http://propertesting.com/

https://github.com/clojure/test.check

http://leebyron.com/testcheck-js/

https://www.toptal.com/designers/subtlepatterns/memphis-colorful/


import uuid from 'uuid/v4'; import type { $Gen } from 'babel-plugin-transform-flow-to-gen/Gen';

!// type $Gen<T, _U> = T;

type User = { id: $Gen<string, uuid>; name: string; }


!// use higher order type in JS const janitorGen = Worker.toGen(gen.object({type: 'janitor'}));

!// OR create type in Flow type Janitor = Worker<{type: 'janitor'}>; const janitorGen = Janitor.toGen();


import { generator } from 'my-lib'; import { AnyActivationRule } from './types';

const activationRuleGen = generator<AnyActivationRule>();

!// function generator<T>(): T;


const arr = arrayOfRandomNumbers();

const sorted1 = sort(arr.concat(-Number.MIN_VALUE));

const sorted2 = [-Number.MIN_VALUE].concat(sort(arr));

expect(sorted1).toEqual(sorted2);

property-based testing for godly tests

Software