Automated Testing in Sakai

Testing applications and services in isolation and in context

Josh Holtzman, UC Berkeley

David Haines, University of Michigan

Goals• Discuss developer-centric testing in Sakai

– Purposes– Concepts– Tools

• Demonstrate a simple running example of application and service testing

Why Formal Testing?

• Ensures that you know what the code is supposed to do

• Ensures that code does what it should do

• Ensures that the code doesn’t regress when changes are introduced

• Developer testing is not to avoid QA

Types of Testing

• Object (Logic/Unit) Testing• Integration Testing• Functional Testing• System Testing• Load Testing (Capacity, Stress, Performance)

Load Testing (Ops)

• Does it perform realistically?– Capacity – Can it deal with X users /

transactions per hour?– Stress – What happens when it gets

overloaded?– Performance – What model describes the

behavior of the system?– Tools - Loadrunner, jmeter, grinder

System Testing (QA)

• Can the entire system perform the required task?– Integrates the full software stack, including

the web server– Tools include httpunit, jmeter, jwebunit,

people, others

Functional Testing (Dev, QA)

• Test specific behavior in deployed systemCan a user really add a new site?

• Slowest method of developer testing• Tools

– Selenium-IDE, httpunit, jwebunit

Integration Testing (Dev)• Tests an object with a real environment

– Uses a test harness to bootstrap the application’s environment

– Test the object using true collaborators – Slow. Must start up the entire environment– Not a replacement for logic testing!

• Tools– Sakai test harness

Logic/Object/Unit Testing (Dev)• Test object behavior• Fastest kind of developer testing• Create an object, setup its local environment,

call a method, and verify its result is what as expected– Tests a single unit of code at a time– Does not test an object in a runtime environment– Must simulate collaborators as needed– Tests can and should be written before the

implementation– Write a failing test first!

Logic Testing (continued)• Tools

– Behavior - Junit– Runtime Environment

• Mock objects can help with collaborators (jmock, easymock)

• Stubs can help with collaborators (but get messy quickly)

• Dynamic proxies– Ecology - Coverage tools help assess how

much code has not been tested (jcoverage, clover)

Logic Testing (continued)• JUnit Architecture

– Test method– Test Cases– Test Suites

Types of Developer Testing

• Logic– Does this logic work correctly?

• Integration– Does it talk nice?

• Functional– Demonstrate that works in a deployment.

Logic Testing Demo

• Test– Test a single assertion

• TestCase– Collection of related tests– Uses fixtures for setup and teardown

• TestSuite– Collection of TestCases– Uses fixture for one time setup and teardown

Integration Testing

• Add real service and test to make sure data gets through

Functional Testing

• Does it work in a deployed system?• Get it up and deployed and running.• Click through and make sure it works.

Summary

• Anything not tested before production gets tested in production

• Fail first• Test what you are writing, not the whole

system• Changing code to be able to test is

often necessary• Refactoring is good

Resources

• JUnit– http://junit.org/– http://junit.sourceforge.net/doc/cookbook/cookbook.htm

• Mock objects– http://www.mockobjects.com– http://www-128.ibm.com/developerworks/library/j-mocktest.html

• Test harness– https://source.sakaiproject.org/svn/test-harness/trunk/xdocs/README-INTEGRATION-TESTING.txt

• Sample code– https://source.sakaiproject.org/playground/trunk/unittesttalk/hello-world/