Zombie-chaser 0.1Going beyond skin deep

Andrew GrimmUniversity of New

South Wales

Overview

● Gratuitous violence and copyright violation● Unit testing● Mutation testing● Heckle● Chaser● Zombie-chaser● Gratuitous violence and copyright violation

Gratuitous violence and copyright violation

● See above

We don't test code

● I get a specification from my boss● I spend a couple of weeks writing on my typewriter● Fax it to the client● They scan it in● It runs first time● And does what they wanted● This DOESN'T happen

We all test code somehow

● SFTP code.php, make a few edits, SFTP it back to the production server, view it in your browser

– This DOES happen

● svn update, make a few changes, try it on a development web server, don't spot anything wrong, push to production

– Better

● git clone, add new test, add new code, check all tests pass, push to production

– Yay!!

Automated testing

● Term “Unit testing” sometimes misused● Technically means testing a small portion of your

code● Nowadays used to mean automated testing● Sometimes automated testing is the lazier option

than manual testing even in the short-term

Automated testing = exomemory

● Your code is thousands of lines long– You can't memorize your code

● Requirements change from the original– You can't write once, try it out, and never touch it

again

● You can't email past-self– The code itself needs to know what's right and wrong

An example of a test

● Given I am a fake Canadian motorcade at the 2007 APEC summit

● When I supply obviously bogus credentials to security

● Then I should not be let in to the green zone

Mutation testing

● Evaluates thoroughness of testing● If your code is modified, it ought to fail some of its

tests

Example of a mutation

● Modifying an APEC credential verification method, such that it regards an invalid pass as valid and/or vice versa

Heckle

● Main mutation tester in ruby● Name derived from Jester (Java mutation tester)● Uses ParseTree and ruby2ruby● Mutates literals (eg strings) and branches (eg swaps

“if”s with “unless”s)● Parses the program, makes changes to the parse

tree, and builds the new program

Heckle pros and cons

● Best mutator out there● Except when it's not available

– Doesn't work with default version of ruby on Windows

– ParseTree doesn't work on ruby 1.9

– Plans to replace ParseTree with RubyParser, which does work on ruby 1.9

Chaser

● Modified version of heckle● Named after the ABC comedy team, best known for

a penetration test that went horribly right● Created to deal with unavailability of heckle on

Windows (which I no longer need)● Uses metaprogramming to modify methods● Intended to run under anything that's ruby● Runs test/unit and my fork of exemplor

Chaser metaprogramming

def modify_instance_method chaser = self @mutated = true @old_method = @klass.instance_method(@method_name) @klass.class_eval do remove_method(chaser.method_name) define_method(chaser.method_name) do (*args) original_value = chaser.old_method.bind(self).call(*args) chaser.mutate_value(original_value) end endend

Chaser disadvantages

● Only modifies return values (and yielded parameters)

● Methods with side-effects will work as normal● Ability to not call a method at all could be added● Doesn't mutate every line● Currently doesn't describe the mutations being done

– May be difficult task

Availability of chaser

● Chaser (actually zombie-chaser) works on:● *nix 1.8 and 1.9● Windows 1.8 and 1.9● JRuby 1.8 only

– Haven't got to the bottom yet for 1.9 (see next slide)

● MacRuby: We broke it good! (see following slide)● Haven't tried IronRuby, Rubinius, or other platforms

Availability of chaser: JRuby 1.9Andrew-Grimms-MacBook-Pro:zombie-chaser agrimm$ jruby --1.9 -Ilib bin/zombie-chaser /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `attribute'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:76:in `register_fixture'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:24:in `setup'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:9:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `class_eval'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/priority.rb:8:in `included'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:82/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/testcase.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:1/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:7/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/ui/testrunnermediator.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:4/Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'bin/zombie-chaser:4/Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/fixture.rb:11:in `included': undefined method `unregister_setup_method' for nil:NilClass (NoMethodError)

from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `call'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:64:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `each'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:61:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `each'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:58:in `set_attributes'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:42:in `attribute'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/gems/1.8/gems/test-unit-2.0.7/lib/test/unit/attribute.rb:41:in `each' ... 16 levels...from /Users/agrimm/ruby/zombie-chaser/lib/zombie-chaser/zombie_test_chaser.rb:31:in `require'from /Users/agrimm/ruby/jruby/jruby-1.4.0/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'from bin/zombie-chaser:4

Availability of chaser: MacRuby

Not even close!

[1] % ruby -Ilib -Itest -rrubygems test/test_unit.rb ~/Code/zombie-chaser/Users/pete/Code/zombie-chaser/test/test_unit.rb:1:in `<main>': unrecognized runtime type `' (TypeError)[1] % ruby -v ~/Code/zombie-chaserMacRuby version 0.6 (ruby 1.9.0) [universal-darwin10.0, x86_64]

And no, we don't recognize the runtime type either!

Thanks to Peter Hollows for running zombie-chaser on his box

Zombie-chaser

● Graphic(al) user interface for mutation testing● Incorporated code from Chris Lloyd (et al)'s brains● Ryan Bigg demonstrated zombino, a brains-based

game he created because brains wasn't working● psDooM, a Doom interface to sysadmin, was also an

inspiration

Gooey metaphor

● Human running across the screen represents running of your unit tests with unmodified code

– Equivalent to a “control” in a scientific experiment

– Fail any tests, and the human dies before zombies appear (boring!)

● Zombies represent mutations to your code– Each successful test is represented with a step closer

– Kill them with failing unit tests

Interfaces available

● Nethack like representation– @ human

– Z zombie

– * dying zombie

– + dead zombie

● GUI interface– Gosu used

Disadvantages with Zombie-chaser

● Issues shared with chaser:– Only modifies return values and yielded parameters

– Doesn't describe the mutations

● Additional issues– Doesn't explicitly describe which test it failed, or

how

– Stops running after the first failed test

Possible TODOs

● Add color to console based output (make dying zombies red)

● Make it work on more platforms● Make it work with more testing frameworks● Fix performance and threading bugs● Provide information that's currently not provided● Improve GUI metaphor● Sound and music

Gratuitous violence and copyright violation

● See above