Testing Tools and Strategies

Scott RyanMarch 2007

Agenda

• Business Drivers• Unit Testing• Coverage• Code Quality Management (CQM)• Integration Testing• Load Testing• Test Data• Automated Test Generation• Demo

Business Drivers

• Errors found early in the development lifecycle cost less to fix

• Cost varies by Industry and Software usage• Cost is not linear• Testing improves quality and reduces cost• Testing costs money too and there is a

tradeoff

Business Drivers

• Cost of running integration and system tests again. • Cost of recertification (if you're in that kind of

environment). • Cost of deploying the software again. • Support costs when only half your customers deploy

the new version. • Indirect costs caused by using resources to fix bugs

rather than implement revenue-generating features. • Liability for damages caused by the bug.

Testing Levels

• Unit Testing• Integration Testing• System Testing

Test Design

• Black Box• White Box

Testing Techniques

• Manual Testing• Automated Testing• Regression Testing• Environment Testing• Stress Testing• Load Testing• Performance Testing

Unit Testing Frameworks

• Junito HttpUnito StrutsUnito XMLUnito DBUnit

• TestNG

JUnit

• Version 3 and 4• Extend TestCase Class• Validate through Asssertions• Setup and TearDown Methods• Testing Suites• Lots of Automation

Test NG

• Java 5 (Primarily)• No need to extend base

class• Good use of Java 5

Annotations• DataDriven Tests• Embedded Beanshell• Embedded Logging, JDK

• Parameter Support• Supports Test

Distribution• New Execution Model

o No Suiteso Groupso Relationships

• Application Server Testing support

TestNG Annotations

• @Test• @Parameters• @DataProvider• @BeforeSuite• @AfterSuite• @BeforeTest• @AfterTest

• @BeforeMethod• @AfterMethod• @BeforeClass• @AfterClass• @BeforeGroup• @AfterGroup

Unit Testing Tips

• Keep tests in same package as tested code• Separate tests for packaging• Start small and work up• Give useful names and assertions• Run often and review the results• Build tests around bugs you are fixing

Coverage

• Cobertura• Clover• Emma• Jtest (Parasoft)• Jester• GroboUtils

• Djunit• Hansel• Gretel

Coverage Types

• Statement Coverage• Branch Coverage (if and implied else)• Condition Coverage• Coverage Percentage

Coverage Hints and Tips

• Use at the beginning of test development• Strive for branch over line coverage• Build tests to increase coverage• Hard to tell what is important so the

developer is key• Review reports often as code base grows

Code Quality Management

• Used to raise the quality of code base• Range from trivial to high impact• All improvements are important• Use a wide variation of tools and tune

around your team’s best practice• Use to drive refactoring

CQM Tools

• Checkstyle• Findbugs• PMD• CPD• Jtest• JDepend

• Jalopy• Eclipse• NetBeans• JavaDoc

CQM Hints and Tips

• Automate and run often• Make incremental improvements over time• Quality is important• Use tools to improve quality at the source• Use as a guide to refactoring

Integration Testing

• Tests the assembled deployed unit• Usually requires automation• Developed after Unit tests• Least used technique but most effective• Fairly complex with many moving parts• Require mature development process

Integration Testing Frameworks

• Selenium• Cactus• Canoo• JWebUnit• HttpUnit• Jmeter• Cargo• Eclipse• Maven

• Dumbster (SMTP)• WinRunner (HP)• SilkTest (Borland)• TeamTest (Rational)

Integration Testing Hints and Tips

• Start small • Automate as soon as practical• Strive for breadth over depth• Work with QA and UAT groups• Tie tests to use cases• Use as smoke test to insure proper deployments• Understand data needs and system impact

Test Data Hints and Tips

• Understand the data needs of your tests• Start the system in a known state• Return system to known state when done• Watch out for protected data• Need to understand the impact of your tests• Keep data current• Understand system impact (i.e. database keys)• Leverage Automation

Test Data Tools

• Setup and teardown methods• Database Scripts• DBUnit• XMLUnit• DDSteps• JTestcase

Mock Testing

• Objects act as mediators• Simulates complex or unfinished systems• Allows access to “closed” systems• Quicker Testing• Mocks are not stubs they are dynamic• Mocks offer behavior verification• Stubs offer state verification

Mock Toolsets

• Easy Mock• Spring Mock• JMock• Jmockit• MockRunner

Automated Test Generation

• JunitDoclet• TestGen4J• Jtest (Parasoft)• Appfuse/Freemarker Templates

Load Testing

• Grinder• Jmeter• JunitPerf• OpenSTA• Eclipse

Other Testing

• Slim (JSP)• J2ME Unit• JFCUnit (Swing)• JSTester (Javascript)• TagUnit (JSP Tags)• SOATest (Parasoft) (Web Services)• XMLUnit (XML)

Demos

Demos

• Junit• TestNG• Coverage• Selenium• Parasoft Jtest• Eclipse

• TestGen4J• Grinder