1. Warming up to modular testing

• • YAPC::NA::2009 June 23rd

• Brad Oaks of Plus Three, LP

2. Who is this guy

First time YAPC speaker.

Not an expert on the subject.

Learned from an existing code base.

3. What we'll cover

The parts you'll need to move from a t/ directory full of flat .t files to a more modular setup.

A simple harness

An example of putting helper methods in their own module for reuse.

How Test::Class and friends ease inheritance in your tests to parallel inheritance in your main code.

4. The Harness 5. Harness to run the show

Runs the tests

Collects TAP formatted output from the tests.

We'll see a simplified example.

6. ./bin/micro_test

#!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");

my $harness = TAP::Harness->new();

$harness->runtests(@files);

7. ./bin/micro_test

#!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");

my $harness = TAP::Harness->new(

{verbosity => 1, merge => 0}

); $harness->runtests(@files);

8.

BEGIN {

# Find a ARCOS_ROOT based on path to harness

my @dir = splitdir(canonpath($RealBin));

$ENV{ARCOS_ROOT} ||=

catdir(@dir[0 .. $#dir - 1]);

}

9.

BEGIN {

# Find a ARCOS_ROOT based on path to harness

my @dir = splitdir(canonpath($RealBin));

$ENV{ARCOS_ROOT} ||=

catdir(@dir[0 .. $#dir - 1]);

# use $ARCOS_ROOT/lib for modules

my $lib = catdir($ENV{ARCOS_ROOT}, "lib");

$ENV{PERL5LIB} =

$ENV{PERL5LIB}

? "$ENV{PERL5LIB}:${lib}"

: "${lib}";

unshift @INC, $lib, "$lib/" . $Config{archname};

}

10.

BEGIN {

# Find a ARCOS_ROOT based on path to harness

my @dir = splitdir(canonpath($RealBin));

$ENV{ARCOS_ROOT} ||=

catdir(@dir[0 .. $#dir - 1]);

# use $ARCOS_ROOT/lib for modules

my $lib = catdir($ENV{ARCOS_ROOT}, "lib");

$ENV{PERL5LIB} =

$ENV{PERL5LIB}

? "$ENV{PERL5LIB}:${lib}"

: "${lib}";

unshift @INC, $lib, "$lib/" . $Config{archname};

eval { require Arcos };

die cannot find Arcos if $@

}

11.

BEGIN {

# choose the first instance if not set.

unless ($ENV{ARCOS_INSTANCE}) {

require Arcos::Conf;

$ENV{ARCOS_INSTANCE} =

(Arcos::Conf->instances())[0];

}

}

12.

BEGIN {

# choose the first instance if not set.

unless ($ENV{ARCOS_INSTANCE}) {

require Arcos::Conf;

$ENV{ARCOS_INSTANCE} =

(Arcos::Conf->instances())[0];

}

}

use Arcos;

use Arcos::Script;

13.

use Arcos;

use Arcos::Script;

our ($help, $man, $archive);

my $v = 0;

my $files;

GetOptions('help'=> $help,

'man'=> $man,

'verbose'=> $v,

'make_verbose=i'=> $v,

'files=s'=> $files,

'archive'=> $archive,

);

14. TAP::Harness can

• try to emit results with color

• be verbose or varying levels of quiet

• run multiple test jobs at once (aggregate_tests)

see "perldoc TAP::Harness"

Test::Harness is for backwards compatibility.

For new work, you should be starting with TAP::Harness.

15. Common test functions 16.

Pull commonly used functions out into their own module.

Don't need inheritance for this.

contains:

Convenience methods.

Affecting the environment.

Checking the environment.

Arcos::Test 17.

can_reach_url() - uses LWP::UserAgent and checks response.

js_escape() - when searching for strings in HTML that has already been escaped.

get_server_uri() - composes url from host and port in Arcos::Conf

Arcos::TestConvenience Methods 18.

start/stop daemons

• apache

• Arcos queue

• test SMTP server

These are basically system() calls to ctl scripts

Arcos::TestAffecting the Environment 19.

spread_is_running() -- used in t/island_mode.t

SKIP: {

skip('Spread is not running', 16) unless

Arcos::Test->spread_is_running();

# 16 tests is, isa, ok, like, etc.

}

Arcos::TestChecking the Environment 20.

spread_is_running() -- used in t/island_mode.t

SKIP: {

skip('Spread is not running', 16) unless

Arcos::Test->spread_is_running();

my $island_pp = catfile(ArcosRoot, 'bin',

'arcos_island_postprocessor');

my $result = `$island_pp 2>&1`;

like($result, qr/Island Mode Post-Processing Complete/);

ok($result =~ /Submitted job (d+) : (S+) from (S+)./);

my ($job_id, $amount, $email) = ($1,$2,$3);

. . .

}

Arcos::TestChecking the Environment 21.

apache_is_running() -- used in t/open_handler.t

BEGIN {

if (Arcos::Test->apache_is_running) {

Test::More::plan('no_plan');

} else {

Test::More::plan(skip_all => "Arcos Apache is not running skipping all tests.");

}

}

Arcos::TestChecking the Environment 22. Test::Class 23.

The methods with Test attribute will be run.

Test::ClassMethod AttributesTest(4) 24.

The methods with Test attribute will be run.

The Test attribute specifies the number of tests for this method.

Test::ClassMethod AttributesTest(4) 25.

The methods with Test attribute will be run.

The Test attribute specifies the number of tests for this method.

The order they're run is determinedalphabetically by name.

Test::ClassMethod AttributesTest(4) 26.

The methods with Test attribute will be run.

The Test attribute specifies the number of tests for this method.

The order they're run is determinedalphabetically by name.

sub b_register : Test(15) { . . . }

sub d_short_pass : Test(4) { . . . }

sub e_mismatched_pass : Test(4) { . . . }

sub f_invalid_email : Test(4) { . . . }

sub h_duplicate_email : Test(4) { . . . }

sub h_duplicate_username : Test(4) { . . . }

Test::ClassMethod AttributesTest(4) 27.

The methods with Test attribute will be run.

The Test attribute specifies the number of tests for this method.

The order they're run is determinedalphabetically by name.

sub b_register : Test(15) { . . . }

sub d_short_pass : Test(4) { . . . }

sub e_mismatched_pass : Test(4) { . . . }

sub f_invalid_email : Test(4) { . . . }

sub h_duplicate_email : Test(4) { . . . }

sub h_duplicate_username : Test(4) { . . . }

sub b_submit_empty_form : Test(no_plan) { . . . }

You can specify no_plan for a specific sub.

Can localize the uncertainty instead of using no_plan for the entire module.

Test::ClassMethod AttributesTest(no_plan) 28.

startup / shutdown

• run before /after the whole test class

Test::ClassMethod AttributesTest(startup) 29.

startup / shutdown

• run before /after the whole test class

setup / teardown

• run before / after every test method

Test::ClassMethod AttributesTest(startup) 30.

startup / shutdown

• run before /after the whole test class

setup / teardown

• run before / after every test method

sub tc_startup : Test(startup) { my $self = shift; $self->{td} = Arcos::TestData->new;

}

Test::ClassMethod AttributesTest(startup) 31.

startup / shutdown

• run before /after the whole test class

setup / teardown

• run before / after every test method

sub tc_startup : Test(startup) { my $self = shift; $self->{td} = Arcos::TestData->new;

}

sub tc_shutdown : Test(shutdown) { my $self = shift; $self->{td}->cleanup(ignore_deleted => 1);

}

Test::ClassMethod AttributesTest(startup) 32.

startup can have tests that count towards the plan

sub start : Test(startup => 1) { my $self = shift;

$self->{mail_tmp_dir} =

Arcos::Test->start_test_smtp_server;

use_ok('Arcos::JobQueue::Handler::CommunityInvitation');

}

Test::ClassMethod AttributesTest(startup) 33.

startup can have tests that count towards the plan.

sub start : Test(startup => 1) { my $self = shift;

$self->{mail_tmp_dir} =

Arcos::Test->start_test_smtp_server;

use_ok('Arcos::JobQueue::Handler::CommunityInvitation');

}

startup can also be specified on more than one method.

Test::ClassMethod AttributesTest(startup) 34. Inheritance in your tests 35.

A parent class for many of our tests.

Arcos::Test::Class 36.

Inherit from Test::Class

use base 'Test::Class';

Arcos::Test::Class 37.

Pull in some helper modules.

use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;

Arcos::Test::Class 38.

Store an Arcos::TestData reference.

use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;

sub tc_startup : Test(startup) { my $self = shift; $self->{td} = Arcos::TestData->new;

}

Arcos::Test::Class 39.

Call cleanup method.

use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;

sub tc_startup : Test(startup) { my $self = shift; $self->{td} = Arcos::TestData->new;

} sub tc_shutdown : Test(shutdown) { my $self = shift; $self->{td}->cleanup(ignore_deleted => 1);

}

Arcos::Test::Class 40. Arcos::Test::Class

We also provide convenience methods at this level:

• contains_url()

• lacks_url()

• json_contains()

• json_lacks()

• json_success()

• json_failure()

• input_value_is()

41. Arcos::Test::Class

help Test::Builder give the right line number for failures

• sub json_success {

• my $self = shift;

• {

• local $Test::Builder::Level =

• $Test::Builder::Level + 1;

• $self->json_contains('success');

• }

• }

We want the error to show up where json_success() is called, not way down here in the helper module.

42. Inherited Forms, with Inherited Tests

Arcos::Form::Test is the parent of

• Arcos::Form:: ContactOfficial ::Test

• Arcos::Form:: Register ::Test

They each have

use base 'Arcos::Form::Test';

43. Arcos::Form::Test the parent

Startup and Shutdown

sub tc_startup : Test(startup) { . . . } sub tc_shutdown : Test(shutdown) { . . . } sub x_create_form : Test(startup => 3 ){ . . . }

Intended to be overridden

sub additional_elements { '' } sub form_db_class { '' }

Shared methods for these forms sub mech_shows_custom_fields { . . . }

sub person_has_custom_fields { . . . }

44. An example of inherited tests

tc_startup can be completely overridden

or augment by calling SUPER

sub tc_startup : Test(startup) {

$self->{td} = Arcos::TestData->new();

return $self->SUPER::tc_startup;

}

45. Inheritance Example 46. We still have .t filest/register_form.t

use strict; use warnings; use Arcos::Test; use Arcos::Test::Script; use Arcos::Form::Register::Test; Arcos::Test->needs_running_krang(); Arcos::Test->needs_running_apache(); Arcos::Form::Register::Test->new->runtests;

47. We still have .t filest/contact_official_form.t

ContactOfficial forms need a little more at startup.

use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) { Test::More::plan(skip_all =>

'geocoding databases not available.');

exit;

}

48. We still have .t filest/contact_official_form.t

ContactOfficial forms need a little more at startup.

use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) { Test::More::plan(skip_all =>

'geocoding databases not available.');

exit;

} unless (Arcos::Test->stop_queue()) { Test::More::plan(skip_all =>

'Could not stop Queue daemon in time.');

exit;

}

49. We still have .t filest/contact_official_form.t

ContactOfficial forms need a little more at startup.

use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) { Test::More::plan(skip_all =>

'geocoding databases not available.');

exit;

} unless (Arcos::Test->stop_queue()) { Test::More::plan(skip_all =>

'Could not stop Queue daemon in time.');

exit;

} Arcos::Form::ContactOfficial::Test->new->runtests;

50. We still have .t filest/contact_official_form.t

ContactOfficial forms need a little more more at startup.

use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) { Test::More::plan(skip_all =>

'geocoding databases not available.');

exit;

} unless (Arcos::Test->stop_queue()) { Test::More::plan(skip_all =>

'Could not stop Queue daemon in time.');

exit;

} Arcos::Form::ContactOfficial::Test->new->runtests;

Arcos::Test->restart_queue;

51. return to skip

Section of a method not yet applicable

return 'The default templates do not currently have security code; skipping for now.';

Whole method not yet implemented

return 'The deleted form test is not implemented for this form type yet.';

Fail safe check

return 'You have no test payment account.'

unless $self->test_account;

52. plan to skip

use Arcos::Conf qw(ContributionTestMode);

if (!ContributionTestMode) {

plan(skip_all => 'contributions not in test mode');

} elsif (!Arcos::Test->can_reach_url(

'https://www.vancodev.com:443/cgi-bin/wstest.vps')) {

plan(skip_all => "Can't reach https://www.vancodev.com:443/cgi-bin/wstest.vps");

} else { Arcos::ContributionForm::Test::Vanco->runtests(); }

53. TEST_METHOD

TEST_METHOD=a_good_file

bin/arcos_test --files=t/admin-listupload.t

TEST_METHOD=e_failed_contributions

bin/arcos_test --files=t/report-contribution.t

TEST_METHOD=k_listing_checkbox

bin/arcos_test --files=t/admin-mailingmessage.t

54. TEST_METHOD

TEST_METHOD=a_good_file

bin/arcos_test --files=t/admin-listupload.t

TEST_METHOD=e_failed_contributions

bin/arcos_test --files=t/report-contribution.t

TEST_METHOD=k_listing_checkbox

bin/arcos_test --files=t/admin-mailingmessage.t

TEST_METHOD='(a|b)_.*'

bin/arcos_test --files=t/warehouse-loader.t

55. a lot of reuse

use base 'Arcos::Report::SiteEvent::Test';

Arcos::Report::Volunteer::Test

Arcos::Report::Contribution::Test

Arcos::Report::ContactOfficial::Test

Arcos::Report::Petition::Test

Arcos::Report::Tellafriend::Test

Arcos::Report::Subscription::Test

Arcos::Report::RadioCall::Test

Arcos::Report::LTE::Test

Arcos::Report::ContactUs::Test

56. Bonus slides 57. TODO tests

sub live_test : Test{

local $TODO =

"live currently unimplemented";

ok(Object->live, "object live");

};

Skip is so darn easy we haven't used TODO like this.

58. pausing to look around

warn 'pausing to look around;

hit return to continue:';

my $PAUSE = ;

This will keep the test from cleaning up before you've had a chance to look around.

Add a few warns of URL values and you can bring up the reports in a browser to get a better idea of what's going on.

Litter this within a specific method, or in your method with the Test(shutdown) attribute.

59. calling plan from Arcos::Test

I added aneeds_run_dailymethod to bail out gracefully.

if ($ENV{'RUN_DAILY_TESTS'}) { if (

$ENV{'OVERRIDE_DAILY_TESTS'}

or $class->_needs_daily_run()

) { $class->_update_run_daily_timestamp(); Test::More::plan(@args); }

} else { Test::More::plan( skip_all => 'RUN_DAILY_TESTS environment variable not set; skipping daily tests.'); }

60. Additional Resources 61. Additional Resources

#perl-qa on the irc.perl.org server

http://search.cpan.org/~adie/Test-Class-0.31/lib/Test/Class.pm

http://search.cpan.org/~mschwern/Test-Simple-0.88/lib/Test/Builder.pm

Organizing Test Suites with Test::Class

http://www.modernperlbooks.com/mt/2009/03/organizing-test-suites-with-testclass.html

62. Thank you for your time!