IFL2002 Madrid1

a generic test-system

Pieter Koopman, Artem Alimarine, Jan Tretmans, Rinus

Plasmeijer

Nijmegen, NL

IFL2002 Madrid2

overview

testing:•why •what will be tested•how do we test

by an automatic systemgeneric testing by a generic systemautomatic and systematic generation of test data

•research question: can we make such a system

•conclusions / related & future work

IFL2002 Madrid3

why testing•even in functional programming we make errors

even errors that the type system does not spot

•to improve quality and confidence the software•proving is often too much work, or not feasible

manual testing is•much work (up to 50% of project cost)

•dull•to be repeated after changes make an automatic system

encourages writing specificationsmakes testing easier, faster and more reliable

IFL2002 Madrid4

what will be tested •there is a rich palette of quality aspects

suitability: validatingobeying the specification: functional testingefficiency...

we restrict ourselves to: obeying the specification, functional testing

•specification can be:executable specification inefficient but obviously correct

reference implementation older version

relation between input and output...

IFL2002 Madrid5

specification•specification by functions in Clean

prop_DeMorgan :: Bool Bool -> Boolprop_DeMorgan x y = x && y == not (not x || not y)

semantics: x, y • x && y == not (not x || not y)

prop_DropDrop :: Int Int [Int] -> Boolprop_DropDrop m n xs = drop n (drop m xs) == drop (n+m) xs

prop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==> let r = sqrt x in r*r == x

•monomorph in order to generate test datafor polymorphic functions this tests all typesfor overloaded functions the instance does matter

IFL2002 Madrid6

how: basic idea

•generate test data, finite and •check property for all these values

test prop | and [ evaluate prop x \\ x <- generate ] = "Passed" = "Counter-example found"

•requirementsone generate for all typesinformation about the counter-exampleinformation about test data used

•what test data should be generated ?

IFL2002 Madrid7

what test data?

•random ? (as in QuickCheck by Claessen & Hughes)+ easy to implement+ easy to use+ works pretty well for small programs- are all relevant cases covered?- duplicates (which are useless)

•systematic !+ coverage of standard cases (guaranteed)

Int: 0, 1, ...List Int: [ ], [ 0 ], [ 1 ], ...

+ no duplicates+ more confidence, better results- harder to implement

IFL2002 Madrid8

systematic generation of test data

what do we want?finite types:: Bool = True | False:: Colour = Red | Yellow | Blue–all values can and should be tested

very large types (always basic types)Int, Real, Char, String, ...–common border values (like 0, 1) and random values

recursive typesList a = Nil | Cons a (List a)Tree a = Tip | Node a (Tree a) (Tree a)–always handled by recursive functions–systematic generation from small to large

IFL2002 Madrid9

systematic generation 2•basic types (Int, Real, ...) are handled separately •for all other types systematic generation from small to large is better than randomcovers interesting casesavoids duplicates

•how to define one general generate function ?one function for all typesno new instance for each new typesystematic generation

– no duplicates– small to large

•use generics !

IFL2002 Madrid10

what are generics

•uniform tree representation of types•conversion between actual type and tree representation by system

•tree representation can be manipulated in generic functions

•we use standard genericsSee Hinze, Alimarine, ...

•enables us to define one generate functionsystematic generation of generic treesautomatic conversion of trees to needed types

IFL2002 Madrid11

generic trees:: UNIT = UNIT // tip of tree:: EITHER a b = LEFT a | RIGHT b // choice:: PAIR a b = PAIR a b // product type

example::: Colour = Red | Yellow | Blue

generic representation of this type:: Colour° = EITHER UNIT (EITHER UNIT UNIT)

generic representation of constructors

Red° = LEFT UNITYellow° = RIGHT (LEFT UNIT)Blue° = RIGHT (RIGHT UNIT)

Red°

Yellow° Blue°

tree representation of all colours

IFL2002 Madrid12

generic trees 2

representation of [] (which is Nil):LEFT UNIT note: same representation as Red!

representation of [Red] ( Cons Red Nil):RIGHT (PAIR (LEFT UNIT) (LEFT UNIT))

[]Cons

Red

Yellow Blue ...

[]Cons

...

tree of all possible values of type [ Colour° ]°

IFL2002 Madrid13

generic functions

•Basic idea:instead of a function F :: A -> B for each A and Bwe define a generic function F°

we implement F as GenToB o F° o AToGen

transformations are generated by the compiler

A BF

A° B°F°

toGen fromGen

IFL2002 Madrid14

Example a generic equalitygeneric eq a :: a a -> Bool

instances for generic type: eq{|UNIT|} UNIT UNIT = Trueeq{|PAIR|} eqx eqy (PAIR x1 y1) (PAIR x2 y2) = eqx x1 x2 && eqy y1 y2eq{|EITHER|} eql eqr (LEFT x) (LEFT y) = eql x yeq{|EITHER|} eql eqr (RIGHT x) (RIGHT y) = eqr x yeq{|EITHER|} eql eqr e1 e2 = False

eq{|Int|} n m = n == m // similar for other basic types

for each type T we either use the generic version:derive eq T

or define an own instance:eq{|T|} t1 t2 = ...

T T Booleq

T° T° Booleq

IFL2002 Madrid15

function generate

•using generics we can define one generate function

algorithm:•systematic generation of generic trees•automatic conversion to desired type

IFL2002 Madrid16

systematic generation of generic trees

•preorder traversalremember current tree

– first all subtrees starting with LEFT– then all subtrees starting with RIGHT

•works fine for Colour, [ Colour], ...•problems if left generic subtree is infinite

Tree a = Tip | Node a (Tree a) (Tree a)using :: T = CTree T generates: Tip, Node C Tip Tip,

Node C (Node C Tip Tip) Tip, Node C (Node C (Node C Tip Tip) Tip) Tip, ...never generates Node C Tip (Node C Tip Tip),...

[[T]] generates: [ ], [[ ]], [[C]], [[C, C]], [[C, C, C]],...never generates [[ ], [ ]], [[ ], [C]], [[C], [ ]],...

IFL2002 Madrid17

generation of generic trees 2

•breadth first traversalorder trees with respect to number of LEFT/RIGHT nodes, or constructors

yield trees with increasing depthwithin same depth: left to right

•problemsnot easy to generate trees with increasing depth efficiently

– not every tree represents a valid data value– adding a constructor requires adding its arguments

large administration needed

IFL2002 Madrid18

generation of generic trees 3

•preorder does not work with left recursion•breath first is nasty•use mixed approach

remember generated branches of treeextend tree for next test-data itemat each EITHER node choose randomly between extending LEFT or RIGHT branch

•Properties:systematicno duplicatesinteresting cases occur soon (with high probability)

also bigger values earlier in list of test-data

IFL2002 Madrid19

testing

•administrate arguments used in a record:: Result = { ok :: Maybe Bool, args :: [String] }

•define class Testable:class Testable awhere evaluate :: a RandomStream Result -> [Result]

instance Testable Boolwhere evaluate b rs result = [ {result & ok = Just b} ]

instance Testable (a->b) | Testable b & TestArg a where evaluate f rs result = let (rs, rs2) = split rs

in forAll rs2 f result (generate rs)

forAll rs f r list= diagonal [apply (genRand seed) f a r \\ a<-list & seed <- rs ]

apply rs f a r = evaluate (f a) rs {r & args = [show a:r.args]}use generic functions generate and show

IFL2002 Madrid20

conditional testsprop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==> let r = sqrt x in r*r == x

•implementation

:: Property = Prop (RandomStream Result -> [Result])

(==>) infixr 0 :: Bool p -> Property | Testable p(==>) b p | b = Prop (evaluate p) // continue testing

= Prop (\rs r = [r]) // stop testing, dummy result

instance Testable Propertywhere evaluate (Prop p) rs result = p rs result

IFL2002 Madrid21

example resultsprop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==> let r = sqrt x in r*r == x

counter-example found after 2 tests: .1191576

prop_DropDrop :: Int Int [Int] -> Boolprop_DropDrop m n xs = drop n (drop m xs) == drop (n+m) xs

counter-example found after 13 tests: -1 1 [0,0]

prop_RevRev :: [Int] -> Propertyprop_RevRev xs = classify (isEmpty xs) xs ( reverse (reverse xs) == xs )

Passed 1000 tests[]: 1 (0.1%)

prop_DeMorgan :: Bool Bool -> Boolprop_DeMorgan x y = x && y == not (not x || not y)

counter-example found after 3 tests: False True

with correction specification: Passed 4 tests

IFL2002 Madrid22

conclusions

•a test system is usefulwriting specifications is encouragedtesting improves quality and confidence

•a generic test system is usefulproperty is arbitrary Clean functionsystem applies predicate to test-datasystem generates test-data automaticallyuse generics to generate, show and compare values

•systematic generation of test datano duplicated testscovers interesting casesstops if all cases are tested

IFL2002 Madrid23

related work

•for FPL: QuickCheck•advantages of our generic approach

one generate function for all types– no user defined instances needed

» generate» show» equal

systematic generation of data– no duplicates– covers interesting cases– stops when all cases are tested

IFL2002 Madrid24

future work•handling •generating better functions

prop_Map :: (Int->Int) (Int->Int) [Int] -> Boolprop_Map f g xs = map f (map g xs) == map (f o g) xs

•easier user control over test-data•better info about test-data used•case studies•GUI•testing application specified in Clean but written in some other programming language

•integration with proof-system•...