• by Alex Chaffee

• alex @ pivotallabs.com

• Pivotal Labs

Test-Driven

• Developers or QA Engineers

• Familiar with JUnit

• Want more detail on Automated Testing in general

• Want to know the case for Test-Driven Development

• Want to know style tips and gotchas for Testing and TDD

Intended Audience

Part I: The Blank Page

Let's test-drive a utility class...

Red-Green-Refactor

• Arrange (set up preconditions)

• Act (call production code)

• Assert (check the results)

3A

• The heart of a unit test

• An Assert is a declaration of truth

• Failed assert -> false (incorrect) behavior

• “assert your postconditions”

• Example:Set set = new MySet();set.add("Ice Cream");assertTrue(set.contains("Ice Cream"));

Assert

• Don't be over-ambitious

• Each test -- especially each new test -- should add one brick to the wall of knowledge

• Code:Brick :: Test:Mortar

• Pick tests (features) in order of growth

One Step At A Time

• A great first test to write

• Input and output are trivial

• Helps you focus on infrastructure and API

The Null Test

• When you get stuck on a test, try starting with the assertion(s) and then work your way backwards to the setup

• Start with the assert

assertTrue(set.contains(“alex”));

• Then add the code above it

Set set = new MySet();set.add(“alex”);

• Helps focus on goal

Assert First

• Start with hardcoded results and wait until later tests to force them to become real

Fake It 'Til You Make It

• Make the code abstract only when you have two or more examples

Triangulate To Abstraction

public void testSum() {

assertEquals(4, plus(3,1));

}

---

int plus(int x, y) {

return 4;

}

public void testSum() {

assertEquals(4, plus(3,1));

assertEquals(5, plus(3,2));

}

---

int plus(int x, y) {

return x + y;

}

• Before you begin, make a TODO list

• Write down a bunch of operations

• For each operation, list the null test and some others

• Also put down refactorings that come to mind

• Why not write all the tests in code first?

• Could box you in

• Interferes with red-green-refactor

Test List

• aka Don't Be Stupid

• If you really, really, honestly know the right way to implement it, then write it that way

• But keep track of how many times your "obvious implementation" was broken or untested

• Edge cases, off-by-one errors, null handling... all deserve tests and often the Obvious Implementation is not covered

Obvious Implementation

Part II: Testing Philosophy

• Unit

• Integration

• Acceptance

• QA

• UI

• Performance

• Monitoring

Automated Testing Layers

• Automated

• Isolated

• Repeatable

• Fast

• Easy to write and maintain

• Focused

• Easy to read

• Robust (opposite: Brittle)

A Good Test Is...

• Someone should be able to understand your class by reading the tests

• Live documentation (better than dead trees)

• “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”

– Martin Fowler

Tests are“Executable

Specifications”

Why do you test?

• Prevent bugs

• Regress bugs ("bug repellant")

• Localize defects

• Understand design

• Document (or specify) design

• Improve design

• Support refactorings

• Enable experimentation and change

• Confidence

• Catch errors the language can't

• Long-term sustainability and maintainability

Why do you test?

When do you test?

• Before checkin

• After update

• Before deploy

• While coding

• In the background

When do you test?

• All the time

When do you test?

• Never

• After coding

• During coding

• Before coding

When do you write tests?

Why test first?

• Gets tests written

• Easier than retrofitting tests onto an existing system

• Guarantees 100% test coverage

• In practice, you never have time after the code is written, but you always have the time before

• Go figure :-)

Why test first?

Why test first?• Reduces scope of production code

• Less scope -> less work

• Proves that your objects have usable interfaces

• more useful methods and fewer useless ones

• Guarantees testability

• Test-last code is often hard to test

• Sustainable Feature Velocity

• Think of tests as examples or specifications

• One trick: write code in a test class, then extract into the production class

"If you can't write a test, then you don't know what the code should do. And what business do you have writing code in the first place when you can't say what it's supposed to do?" - Rob Mee

How can you write tests for

code that doesn't exist?

• Simple Rule:

• Test everything that could possibly break

• Depends on definitions of “everything” and “possibly”

• (and “break”)

• This means, don’t test things that couldn’t possibly break

• E.g. Getters and Setters

• Unless you think they could fail!

• Better safe than sorry

• Full-Range Testing (positive, negative, boundary, null, exception)

What to test?

• Personal judgement, skill, experience

• Usually, you start by testing too little, then you let a bug through

• Then you start testing a lot more, then you gradually test less and less, until you let a bug through

• Then you start testing too much again :-)

• Eventually you reach homeostasis

How much to test?

• Not too big, not too small

• Same concept as high coherence, low coupling

Test for "essential complexity"

• Write the tests first

• Design for testability

• Use the front door first

• Communicate intent

• Don't modify the SUT

• Keep tests independent

• Isolate the SUT

• Minimize test overlap

• Minimize untestable code

• Keep test logic out of production code

• Verify one condition per test

• Test separate concerns separately

• Ensure commensurate effort and responsibility

Meszaros' Principles of Test Automation

• Every time you write code, you write tests that exercise it

• That means that if you change the code, and the tests break, you must either

• Change the tests to match the new spec

• Change the code to meet the old spec

• Do not remove the failing tests

• Unless they no longer apply to the new code’s design or API

• Do not work around the failing tests

• Test code is not "wasted" or "extra" -- tests are first-class citizens

• If you feel like they're too much work, examine your process

• Maybe you're writing the wrong tests, or your tests are too brittle, or they're not refactored enough

Tests Are An Extension of

Code

• It forces you to really understand the code

• It forces you to really understand the tests

• It forces you to create code that is truly reusable and modular and testable

• “put your money where your mouth is”

• These forces drive you to keep your code and your tests simple and easy to understand

Unit Testing Is Hard

• Need to spend time on infrastructure, fixtures, getting comfortable with TDD

• Business case for TDD: sustainable velocity

• for feature velocity, stabilty > early oomph

• Famous graph

Test-Driving Is Slower At First

• Test-Driven Development

• Good old-fashioned coding, now with tests!

• Test-Driven Design

• Free your mind and the code will follow

• Quite a lot of overlap, but worth keeping difference in mind

• Lots of XP gurus are all about the Zen, but you don't need to buy into that

• But it's actually pretty cool to try Zen Testing

Two D's

Part III: Advanced Techniques

• Positive Tests• exercise normal conditions (“sunny day scenarios”)• E.g. Verify that after adding an element to a set,

that element exists in the set

• Negative Tests• Exercise failure conditions (“rainy day scenarios”)• E.g. verify that trying to remove an element from an

empty set throws an exception

• Boundary Conditions• Exercise the limits of the system (“cloudy day”)• E.g. adding the maximum number of elements to a set• E.g. test 0, -1, maximum, max+1

Full Range Testing

• instead of SetTest.testEmpty

• how about SetTest.testShouldHaveSizeZeroWhenEmpty

• or EmptySetTest.testHasSizeZero

Verbose Test Naming

• Optional first parameter to JUnit asserts is "message"

• Assertion messages can be confusing

• Example:assertEquals(“set is empty”, set.isEmpty());

• Does it mean “the set must be empty” or “the test is failing because the set is empty”?

• Solution: should statements

assertEquals(“set should be empty”, set.isEmpty())

• or even better:

assertEquals("a newly-created set should be empty", set.isEmpty())

Should Statements

• Philosophy: a test is a valid client of an object

• Therefore don't be ashamed of adding a method just because it would make a test easier to write

• Used -> Useful

• Remember, tests are examples of use

Test-Only Methods

• Spend time refactoring your tests

• It'll pay off later, when writing new tests or extending/debugging old ones

• Refactor for readability, not necessarily for removing all duplication

• Different priorities than for production code

• Extract methods

• Shorter lines

• Break up long tests (scenario tests) into several short tests (feature tests)

• One technique: "Refactor production code on green, Refactor test code on red."

• for complex cases, break the code, make sure the refactored tests still reveal the breakage, then fix it

Refactor Test Code

• assertEquals(86400, new Day().getSeconds())

• vs.

• assertEquals(60 * 60 * 24, new Day().getSeconds())

• vs.

• secondsPerMinute = 60

• minutesPerHour = 60

• hoursPerDay = 24

• assertEquals(secondsPerMinute * minutesPerHour * hoursPerDay, new Day().getSeconds())

Evident Data

• Problem: several axes of variability, combinatorial explosion

• Solution: Loop through a matrix of data in your test, call a "check" function on each row

Matrix Tests

• aka "Golden Data Tests"

• Grab the complete output of a routine, put it into the test

• Not amenable to test-driven development

• Effective for large or risky refactorings

• Quite brittle, so often thrown away after the refactoring is done

Characterization Tests

public void testUnknownCountry() {

try {

currencyConverter.getRate("Snozistan");

fail("Should have thrown an exception for unknown country");

} catch (UnknownCountryException e) {

// ok

}

}

Exception Tests

• A pair's job is to keep you focused

• "Wait, let's write a test first."

• "Wait, let's refactor first."

• "Wait, let's discuss this."

• "Can I drive?"

Pair Programming

• One pair writes a test

• The other pair makes it pass and writes the next test

• Repeat

• Good way to get out of a rut, or cure a keyboard hog

Ping-Pong Pairing

• When a defect is reported...

• The first step is to write a (failing) test that reproduces the bug

• Fix the bug by writing code to make the test run successfully

• Verify the bug in the running application• Add the bug test to the automated suite• Check in the bugfix code and test• Now it’s always run – instant regression test!

• "Regression tests are test that you would have written originallly." - Kent Beck

• May also want to write a failing Acceptance Test, but that's optional -- you definitely want a failing unit test

Regression Test

• Often the best thing to do is throw away your work and start again

Do Over

• At the end of the day, write a failing test and leave it there for tomorrow

• Based on writer's trick: start a sentence and leave it unfinished

Leave One For Tomorrow

• Tests are only valuable if they're run all the time

• If they're slow, people will not want to run them all the time

• So keep them fast!

• Difficult quest, but worth it

• Don’t get stuck in molasses!

• Refactor your code to be easier to write fast tests on

• Replace slow tests with (one or more) fast tests that cover the same area

The Need For Speed

Retrofitting• What to do when you have an existing untested

codebase?

• Start small

• Write one test, make it pass, check it in

• Write tests for all new code

• Write tests for all new bugs

• Write tests before attempting refactoring

• Usually easier to write characterization tests (UI/integration)

• But don’t fall into the slow test trap

• Any time all the tests are green, you can check in

• Run all the tests all the time

• Don’t check in until all tests pass

• If you broke “someone else’s” tests, you are responsible for fixing “their” code

• Remember, they are in the room, so go get them if you need help

• Learn to Love the Orb

• ccmenu

Continuous Integration

• Suites are a pain to maintain

• Write code to automatically scan for tests and run them together

• Possible to do in JUnit, but annoying

Automatic Suites

• Matter of preference

• Both are useful at times

Outside-in vs. Inside-out

• Start with domain objects

• Next layer of tests

Inside-out

• Start with customer story or user interface

• Makes you think like a user

• Tests capture these requirements

• Lower layers implemented with Test Doubles (mocks)

• After you're done, either replace mocks with real objects, or leave them there (perhaps at higher maintenance cost)

Outside-in

• Write a bunch of UI-level tests

• Leave them there while you test-drive inside-out

• When they all pass, you're done

Outside-in design,

inside-out development

• A Test Double replaces the "real" instance of an object used by the production code with something suitable for the currently running test, but with the same interface

• Stubs

• Hard-coded values

• Mocks

• Pre-programmed with expectations

• Fail-fast

• Test Doubles in general are often called Mock Objects, so be careful about terminology

• Fakes

• Can store values across calls, but don't really do what the live object would do

• E.g. in-memory database

Test Doubles

More Test Doubles

• Spies

• Remember what methods were called with what values

• Tests can inspect these lists after code returns

• Saboteurs

• Blow up in ways that would be difficult to make happen for real

• To test what would happen when, e.g., the database goes away, or the disk is full

• Self Shunt

• The test itself declares methods or classes implementing the above, and passes in a pointer to itself

• Two types:

• DI Frameworks

• Complete Construction

• This is the one I'm talking about

• Pass in dependencies to the constructor (or, if necessary, to setters)

• An object under test will receive references to all external services

• Allows tests to inject Test Doubles at will

• Forces objects to be isolated

• Example: TBD

Dependency Injection

• Changes the language of tests to emphasize that they're specifications or examples

• Replaces "assert" with "should"

EDD/BDD (specs)

• A natural progression of refactoring your test data

• literals

• constants

• local variables

• instance variables (defined in setUp())

• creation methods

• parameterized creation methods or objects ("object mothers")

• Other patterns

• test objects / graphs ("fixtures" or "cast of characters" or "menagerie")

• external fixture files

Fixtures and Object Mothers

Mock Clock

Part IV: Test-Driving UI

Part V: Q&A

Thanks!