Automated Testing of Web Applications using XML

dIon Gillard,Multitask Consulting Pty Ltd

Agenda

What is a Web Application? Testing Web Apps

Code-based testingData driven testing

Tools Automating test case creation Automating tests Comparison to Commercial Tools

What is a Web Application We’ll be concentrating on J2EE Web

Applications A lot of the tools and concepts can be

applied to other technologies Web Applications consist of:

Server side resources• JSPs, Servlets, Java code, configuration

Client side resources• HTML, images, JavaScript

Testing Web Applications – ‘Field Experience’

Developers test localised changes have had the desired effect

Without automated testing a complete regression test is impractical

Application yo-yos from good to bad, working to broken frequently

Users have little confidence in the application and wait for a ‘release’

Large effort goes into QA for a release

Testing Web Applications – Why?

There are lots of ways to test, we aren’t going to cover all of them

Testing usually has a purposeDoes the code workDoes the application perform wellDoes it respect security requirementsCan it handle loadDoes it match business requirements….

Testing web applications - How

We’re going to be focussing on ‘functional testing’, a.k.a. Black-box testing

“Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution condition”

IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries

Testing Web Applications - Implications

Looking at web applications from a user’s perspective - Does it do what I want, how I want it done?

What the code does is a side-effect for the user

Users expect a web application to be an ‘application’, regardless of how the data is displayed or the functionality accessed

User interface is still very important, however, so layout, along with look and feel must still be tested

Testing Web Applications – What to test

Interactions between the user and the application Expected Inputs

input fields, hidden fields etc, query string or URL additions Unexpected inputs

no password, field lengths, numerical domains etc Error conditions

Messages to the user, code ‘failure’ Error 500 Expected processing

Processing succeeds or fails Expected output

Results of next page, database updates, messages queued etc

Code based Testing Tools such as JUnit rely on developers

creating ‘test cases’ in code to ensure correctness

This has all the advantages and disadvantages of developers writing code

Several tools have been released that allow this development to be more productive, less tedious and the code more reusable

Writing low-level code makes it hard to focus on what the testing is for

HttpUnit Freeware ‘Browser emulator’ for use in JUnit

test cases (and other places) Written in Java, provided as a jar developers

would use Takes care of authentication, cookies, page

redirection Provides resulting pages as a DOM via

NekoHTML HTML pages don’t have to have valid XML

syntax to be parsed

HttpUnit - Sample

WebConversation conversation = new WebConversation();WebRequest request = new GetMethodWebRequest(

"http://www.meterware.com/servlet/TopSecret");

WebResponse response = conversation.getResponse(request);WebForm loginForm = response.getForms()[0];request = loginForm.getRequest();response = conversation.getResponse( request );assertTrue( "Login not rejected",

response.getText().indexOf("Login failed") != -1 );

HttpUnit – API Classes are available for creating requests in

various ways (WebRequest) Working with the HTTP response

(WebResponse) return code, text, headers, cookies etc

Working with HTML artifactsgetDOMgetForms(), getFormWithName(name),

getTables() getLinks(), getLinkWith(text) Forms can be ‘submitted’ after retrieval from

the response

HttpUnit – Pros and Cons Code can be repetitive and tedious Working at a low level, you can lose sight of

what is being done Each WebRequest is created in isolation in a

test Testing a web application is typically a single

host and URI, code is needed to unify tests Based on JUnit, is easy to automate Simple clear API is easy to understand

HtmlUnit Similar in scope to HttpUnit, but with a

focus on HTML documents rather than the HTTP protocol details like requests, responses etc

Better JavaScript support API is easier to relate to user interaction,

e.g. clicking links and pressing buttons Wrapper objects for every html element Browser specific features can be used to

fake the browser version and vendor

HtmlUnit - continued Can track frames, iframes, pop up windows Listener to handle new windows being

opened as part of interaction or JavaScript Can easily access the other windows Tab-order can be retrieved and checked Assertion helper methods for unique ids,

accessibility Full access to the actual DOM i.e. editable

HtmlUnit – Sample form submit

WebClient client = new WebClient();URL url = new URL("http://htmlunit.sf.net/");HtmlPage page = (HtmlPage) client.getPage(url);

// find form by idHtmlForm form = page.getFormById(“mainForm”);

// ‘type’ text into the userId fieldHtmlInput input = form.getInput(“userId”);input.setValue(“dion”);

// submit – note existing values provided in the page// are ‘automatically’ filled inform.submit();

ServletUnit Companion tool shipped with HttpUnit For when you need to do less black box

and more white box testing Simulates a servlet container by

providing Mock Objects Provides API for accessing the

‘Invocation Context’ Other tools such as Cactus provide a

fully-featured in-container environment

ServletUnit – Sample code

ServletRunner sr = new ServletRunner();sr.registerServlet("myServlet", StatefulServlet.class.getName());// create a client and request ServletUnitClient sc = sr.newClient();WebRequest request = new PostMethodWebRequest(

"http://test.meterware.com/myServlet" );request.setParameter( "color", "red" );InvocationContext ic = sc.newInvocation( request ); // call serviceic.getServlet().service(request, ic.getResponse());// test response, session, servlet instance variables etcassertNotNull("Session was not created",

ic.getRequest().getSession( false ) );

StrutsTestCase – What is it?

JUnit extension for testing code based on the Apache Struts web app framework

Provides Mock Objects for simulating the servlet container

Also provides an in-container Cactus extension (CactusStrutsTestCase)

Switching approaches means changing parent class to/from StrutsTestCase

StrutsTestCase – How does it work?

Tests are based around Struts Actions There are methods available for

hooking into the Struts request lifecycle

Extra assert-like methods for checking the Struts environment, like forwards and action errors

StrutsTestCase - Sample…public void testFailedLogin() {

// setup parameters to pass to the actionaddRequestParameter("username","deryl"); addRequestParameter("password","express"); // set up the path to identify the actionsetRequestPathInfo("/login"); // perform action processingactionPerform(); // check that the user has been forwarded to “login”verifyForward("login"); // make sure the error for key ‘error.password.mismatch’// is present in the requestverifyActionErrors(new String[]

{"error.password.mismatch"}); // make sure the session state is correctassertNull((String) getSession().getAttribute("authentication"));

} …

Apache’s Cactus A framework for unit testing server side

Java code JUnit extension to make it easier to test

Servlets, EJBs, Tag Libraries, Servlet Filters etc

Comes with ant tasks for automation Uses a ‘proxy’ mechanism to redirect

test case requests to a live servlet container

Apache’s Cactus - API Provides 3 Test classes to subclass from

ServletTestCase, JspTestCase and FilterTestCase

Code testXXX methods as per usual, except you need to instantiate the object to be tested (e.g. servlet)

beginXXX and endXXX methods can be coded to setup or post-process HTTP related parameters such as headers, return code, the response etc

Apache’s Cactus - Features

Provides IDE integration for JBuilder, VAJava

EJB unit testing using Servlet ‘proxy’ As it runs in-container, there is good

documentation on setting up the proxy Can integrate with HttpUnit and

StrutsTestCase

XML-based Tools These tools rely on tests written in an

XML format rather than java code Output varies – can be HTML They can be more easily generated They are more ‘user’ friendly Take away the reliance on technical

skills to build tests Work at a higher level and allow more

focus on what is being tested and how

Apache’s Latka Functional testing tool that uses XML to

define a series of HTTP(S) requests Validations for the responses are defined

as child elements of the requests Has a command prompt interface Can run tests via JUnit Can run tests interactively via a web-app Latka is moving to use Jelly as it’s XML

processor

Apache’s Latka – Document Definition

Suite top level element for a set of testsProvides default host, port and a label

SessionWrapper for a set of requests that share HTTP

state Request

A HTTP request to be executedPath, method (“get”), host, port, label

ValidateHolds all validations for a specific request

Apache’s Latka - Validations

Bytelength – length of the response Cookie – presence of a cookie maxRequestTime Regexp – tests the response matches a

regular expression statusCode – e.g. 200, 404 XPath – test the response matches an

XPath expression

Apache’s Latka - Sample<suite defaultHost="jakarta.apache.org" label="Taglibs">

<session><request path="/taglibs/" label="home page"> <validate> <statusCode /> <regexp pattern="Welcome to Jakarta Taglibs!"/> </validate></request><request path=“/logon”>

<param><paramName>user</paramName><paramValue>dion</paramValue></param>

<param><paramName>password</paramName><paramValue>dionspwd</paramValue></param>

<validate><statusCode /></request>

</session></suite>

Apache’s Latka – Running Tests

Command promptLatka.bat file:./TestSite.xml

JUnitpublic static Test suite() {

TestSuite suite = new TestSuite();String fileName = "tests/samples/TestCommonsWebsite.xml"; suite.addTest(

JUnitTestAdapter.createTestFromFile(fileName));return suite;

}

Apache’s Latka Also provides validators as tags for Jelly Automate via JUnit Ant task Listeners can be attached to pick up

test events as they happen XML and Simple (console) reporter

provided XSL Style sheet provided to produce

text from report

Apache’s Jelly Java and XML based scripting and

processing engine Jelly ‘scripts’ are XML documents which

get parsed Scripts are then ‘executed’ to produce

XML, HTML, text etc XML elements can be bound to java

code similar to JSP custom tags and Ant tasks

Apache’s Jelly - Taglibs Core: catch, choose, expr, forEach, if,

file, import, include, jelly, new, otherwise, set, thread, when, whitespace

Xml: attribute, copy, copyOf, element, expr, forEach, if, parse, set

Define: attribute, bean, invoke, invokeBody, jellyBean, script, tag, taglib,

Sql: dateParam, driver, param, query, setDataSource, transaction, update

Apache’s Jelly – Taglibs (Continued)

JSL: applyTemplates, style, stylesheet, template

Ant: ant, filescanner Werkz: attain, attainGoal, goal,

postAction, postGoal, preAction, preGoal, project

Jeez: target, tagDef Log: debug, error, fatal, info, trace, warn Ojb: broker, store

Apache’s Jelly – Taglibs (Continued)

JMS: connection, destination, mapEntry, mapMessage, message, objectMessage, property, receive, send, textMessage

Validate: verifier, validate, assertValid Http: delete, get, header, post, put Interaction: ask Antlr: antlr, grammar Util: tokenize Html: parse

Apache’s Jelly – Taglibs (Continued)

JUnit: assert, assertEquals, case, fail, run, suite

Swing: action, component, windowListener

Quartz: cronTrigger, job, waitForScheduler

Betwixt: introspector, parse

Apache’s Jelly - Testing Provides

a JUnit tag librarya HTTP tag librarya ‘cron’ like facilityan expression language

Integration with JMS, SQL etc allows page results to be checked against the database or queue

Apache’s Jelly - Sample<?xml version="1.0"?><j:jelly xmlns:j="jelly:core" xmlns:log="jelly:log"xmlns:v="jelly:org.apache.commons.latka.jelly.validators.HttpValidatorTagLibrary"

xmlns="jelly:org.apache.commons.latka.jelly.HttpTagLibrary" trim="false">

<session><get var="mtc" uri="http://www.multitask.com.au/"/><j:if test='${mtc.statusCode == 200}'>

request ok </j:if>

Results for mtc url are:http return code = ${mtc.statusCode}http status text = '${mtc.statusText}'size of result = ${mtc.responseBodyAsString.length()}response time = ${mtc.responseTime}

</session></jelly>

Apache’s Jelly - Sample<?xml version="1.0"?><j:jelly xmlns:j="jelly:core“ xmlns:log="jelly:log"xmlns:v="jelly:org.apache.commons.latka.jelly.validators.HttpValidatorTagLibrary" xmlns="jelly:org.apache.commons.latka.jelly.HttpTagLibrary“>

<session> <post var="jdc“

uri="http://developer.java.sun.com/servlet/SessionServlet"> <parameter name="action" value="login" /> <parameter name="url" value="/developer/index.jshtml" /> <parameter name="UserId" value="XXXX" /> <parameter name="Password" value="XXXX" /> </post> jdc login result = ${jdc.statusCode} good login = ${jdc.responseBodyAsString.indexOf("Sorry!") == "-1"} <v:regexp pattern="Sox.y!" var="jdc"> <request-failed var="jdc">bad pattern 1</request-failed> </v:regexp></session>

</jelly>

XMLUnit JUnit extension providing comparison

between XML docs XML can come from lots of places:

Strings, DOM, Reader Accepts ‘browser-friendly’ HTML as

input Can validate against a DTD

JXWeb XML Script ‘language’ and extension to

JXUnit Uses HttpUnit under the covers No JUnit integration Not as easily extendable as Jelly

JXWeb – Scripting Language

jxw – top level element httpGet, postXml – to retrieve data setRequestParameter, setCookie,

setHeader getLink, getTableValue,

getFormParameter, getForm, getTable ifEqual, isNull, loop, doLoop, exception,

ifNull, save, subst, ifEqualLength, isEqual

Echo, set, loadProperties

JXWeb - Sample

<jxw><set name="req" value="www.testsite.com"/> <httpGet response="response1"/> <save name="respText" file="list.html"/> <getTable name="myTable" tableIndex="0"/> <getTableValue name="tableValue" table="myTable“

row="0" column="0"/> <isEqual name=“tableValue"

value="Every Good Beer Deserves Froth“ message=“page is missing froth"/>

</jxw>

AntEater – Ant Functional Testing

Testing framework designed around Ant

Supplied as a set of Ant tasks and types

Integrates easily into an existing build or IDE

Comes with a servlet container built-in to allow for ‘call backs’

Ant Eater - Tasks Action Tasks

httpRequestsoapRequestListener

Match – groups multiple tests Tests

Listening: method, parameter, sendResponse

Header, contentEquals, regexp, responseCode, xpath, relaxng

Ant Eater – Other tags Session Logger – minimal, plain, colour, xml forEach Group – a way of grouping variables,

sessions and loggers servletContainer – start internal tomcat Deploy – deploy web apps

Ant Eater - Sample… <target name="checkonline"> <httpRequest path="/"> <match assign="online"><responseCode value="200"/> </match> </httpRequest> </target>

<target name="hitserver" if="online"> <echo>We're online!</echo> <httpRequest path="/"> <match assign="apache"> <header name="Server“

value="Apache/1.3.24 (Unix) Debian GNU/Linux"/> </match> <match assign="unknownserver“/> </httpRequest> </target>

Limitations XML as a programming language isn’t

Not as expressiveTedious syntax for non-techiesUsually not as extensible

Limits the interaction with external data sourcesOther than Jelly, most don’t have a

mechanism for dealing with java objects outside their framework

Automating Test Case Creation

One of the main reasons developers give for limited testing is limited time

Developers are actually happy to test if it isn’t a significant drain on achievements

Testing pages or servlets helps developers think of error conditions that could easily crop up

Automated Creation - JMeter

JMeter is often thought of as a load testing tool only

It also has a ‘proxy server’ mode where you can record browser activity

It can filter out requests for images or other resources using a regexp

It can also be used to ‘assert’ text is present on a request

Automated Creation – ‘Homer’

Microsoft freebie tool in IIS 5.0 resource kit

Under windows acts as a proxy and records requests coming in

Usually used for load testing Stores data in an Access .mdb file Simply queries can transform this into

XML and then into Latka or Jelly etc Merged into VS.NET toolset

Automated Testing Once the tests are created the next

thing is to get them run on a regular basis

This means application and developer confidence grows

Depending on the tool this can be achieved using the operating system and a build tool

Apache’s Ant Java based build tool Has JUnit tasks as part of the ‘optional’

tasks Can be scheduled using operating

system constructs Can provide email about failures

CruiseControl Open source tool from ThoughtWorks Provides a way to kick off Ant builds

based on source repository changes This allows the automation of the tests

to be done when the source code changes

Ensures that the application passes all tests

Flags immediately when the application breaks

Commercial Tools LoadRunner from Mercury Interactive

Provides access to XML data from the scripting language

More geared toward load testing Astra QuickTest from Mercury

InteractiveWeb testing, built-in script creation via

browser, cross browser tests from same test case, extract details from browser objects

Supports Flash, Real etc

Commercial Tools Rational RobotJ

Eclipse based toolUses proprietary scripting languageDoes auto-record, regular expressionsApprox 1.3GB space

WebLoadSimilar features to higher cost toolsMultiple server support, higher load focusSupports JavaScriptGUI and Command line driven

Commercial Tools - continued

High end Java IDEs such as WebSphere Studio Application Developer 5 now include testing toolsSome can not be automatedCompete with other vendors in same tool

space

Summary There are simple tools available now that

can alleviate the drudgery of functional testing

Users can help write the functional tests Automation and repetition are key Choose how far you want to take testing Be consistent Allow for learning curve on new tools and

technologies Mike Bowler’s Session on Test Driven

Design

Resources Ant

http://jakarta.apache.org/ant/ HttpUnit

http://httpunit.sourceforge.net/ JMeter

http://jakarta.apache.org/jmeter/ JUnit

http://www.junit.org Latka

http://jakarta.apache.org/commons/latka

Resources (Continued)

StrutsTestCasehttp://strutstestcase.sourceforge.net/

JXWeb http://qare.sourceforge.net/web/2001-12/products/jxweb/

Cactushttp://jakarta.apache.org/cactus/

XMLUnithttp://xmlunit.sourceforge.net

Resources (Continued)

Homer http://webtool.rte.microsoft.com HtmlUnit

http://htmlunit.sourceforge.net WebLoad http://www.webload.dk Jelly

http://jakarta.apache.org/commons/sandbox/jelly/

Contact Here all week Q ‘n’ A sessions dion@multitask.com.au