324

Teaching with a Unit TestingFramework

Dr Mike LanceSchool of Computing

Christchurch Polytechnic Institute of TechnologyChristchurch, NZlancem@cpit.ac

In the NACCQ curriculum desk checking is used to introducenovice programmers to testing. It is argued that using a unittesting framework for subsequent teaching of testing is an equallyeffective teaching technique.

KeywordsTesting, desk checking, Unit Testing Framework, xUnit.

1. INTRODUCTIONTesting is a major time and cost factor in com-

mercial software development - up to 50% of cost(Jones 2001). Because a wide range of practicesand skills are required to produce well-roundedsoftware testers a strong case can be made for wide-spread integration of testing topics throughout thewhole undergraduate curriculum (Jones 2000).Many report however that testing is perceived verynegatively by novice programmers (Canna 2001,Edwards 2003, Jones 2001).

In the NACCQ New Zealand Qualifications inInformation and Communications Technology mod-ule prescriptions software testing is first addressedwith an emphasis on desk checking. (PP490,PD500). Subsequent programming modules stipu-late more general testing skills in learning objectivethat are variations of "design test data that will checkthat a program works to a specification" (PP590,PR50n, PR51n, PR60n, PR61n, PR62n, PR65n,PR72n). There is also a specialized module (QA600)with a comprehensive coverage of testing.

This paper will argue that the strategy of empha-sizing a specific testing technique (desk checking)should be extended by also putting an emphasis inall programming modules on writing tests with a unittesting framework (xUnit). There is a strong need toget novice programmers competent and enthusias-

tic about testing and xUnit provides a good vehiclefor doing this.

2. DESK CHECKINGIntroducing novice programmers to testing via

desk checking is a first experience which will leavea strong and lasting impression. Desk checking isvery effective at teaching novice programmers tosimulate and check the flow of logic in an algorithm.(The alternative to students proving the soundnessof their own work is letting them handing code to atutor and leave it up to the tutor to sort out if thecode passes or not. This is not very empoweredand the novice ends up being dependant on the tu-tor as tester.) It is a concern is that having learneddesk checking in subsequent programming coursesstudents will seldom use it unless they are requiredto. (In the authors 'other life' as a programmer heonly resorts to desk checking when encountering acomplex logical algorithm, written by somebodyelse, in a programming language he don't have ac-cess to. By preference he will use a debugger andset watch and break points.) Limited use of deskchecking by advanced students may be an encour-aging indication that the method effectively teachesmental simulation of the execution of a programme.There is a danger that students learn a negative mes-sage when the first formal software testing methodthey meet is not one they will actually use to writesoftware. It is worthwhile spending the time to makeclear to novice programmers that mentally deskchecking is acceptable in many situations and is theeventual learning goal.

An emphasis on desk checking can also leavethe impression testing comes after writing code.

325

Before a desk check can be carried out, an algo-rithm needs to be designed (in pseudocode, prob-lem description language or a teaching language suchas Pascal). After the algorithm is written down, atest situation must be defined and expected inputsand outputs must be determined. Only then can thealgorithm be desk checked. If students repeatedlyhave the experience of designing a correct algorithmand then have going through a time consuming for-mal process which detects no mistakes they maylearn that desk checking is an unnecessary nuisanceand testing is an annoyance. Setting problems withsubtle logical traps in the requirements or getting stu-dents to locate subtle bugs in code can help makethe value of desk checking clear. The teacher oftesting needs to also ensure that students continueto encounter situations where formally testing of non-trivial logic is required.

The wrong message about testing can also con-veyed by the format in which students are requiredto carry out desk checks. Manual desk checking isan awkward exercise with rubbing out of incorrectresults, repetitive writing of the same text, and muchtaping of pages together. The implicit lesson whichthis teaches is that testing is a dated and painful ex-perience. (Programming students will seldom use apencil, rubber and ruler except when doing manualdesk checking.) Enforcing the use of awkward 1960stechnology is also not necessary when it is so easyto create and record the results of a desk check in aspreadsheet. Using a spreadsheet for desk check-ing reduces many of the taxing mechanical compo-nents of the task and allows the student to concen-trate on the actual testing.

3. THE JOY OF XUNITThe biggest problem with desk checking as an

introduction to testing is that it may teach studentsthat testing is not fun. One important educationalbenefit of testing with a unit testing framework ('xUnit'from now on) is that the associated enthusiastic atti-tude towards testing. The claim is made that pro-grammers will become 'test infected' and will 'lovewriting tests' and be 'amazed at how much more funprogramming is'. (Beck & Gamma, 1998a) Part ofthis positive attitude towards testing is due to ad-vantages that accrue when any sort of automatedtests exist and help find bugs in code. With a firmfoundation of tests, programmers become 'faster' and

'more productive'. (ibid) A particular reason whydevotes of xUnit have a positive attitude towardstesting is an altered emphasis on the place of testingwithin the software development life cycle. The rec-ommended practice is to write the tests before cod-ing to define requirements: "Add a test, get it to fail,write code to pass the test. Remove duplication"(Beck 2002 p5). Testing after coding will invari-ably identify failures and is a negative experience forthe programmer. Coding after writing tests in con-trast is a positive experience because the number offailing tests decreases and the increasing number ofpassing tests confirms that the code is working. Thispositive attitude associated with testing is of greatvalue to educators seeking to teach testing.

'Test driven development' is also claimed to havea subtle impact on the quality of software design.Code has to be used by two 'clients': the systemunder development and the testing framework. Thisattention to developing a testable interface forcesdesign for reuse. Code produced in this way is saidto be 'reusable', 'less coupled' and 'highlydecoupled'. (Beck & Gamma, 1998a) Once thebreak through of using code in two different situa-tions has been achieved, subsequent reuse is said tobe much easier. The final recommended step in thetest driven development process is 'remove dupli-cation'. The existence of unit tests and correctlyworking code is an 'essential precondition' forrefactoring (Fowler, 1999, p89). The support andreassurance of working tests allows aggressiverefactoring which can incrementally improve thedesign of code. The potential is that introducing xUnitappropriately into the software development life cyclecan foster other positive changes to the quality ofstudents' coding practices.

Another advantage of xUnit for the educator isthat it is freely available for down load, is availablein many different programming languages and hasan associated large number of worked examples withextensive commentary. (eg Beck 2002, Pilgrim2004). There are also discussion groups and Wikiswhere industry based xUnit users enthusiasticallyrecount their experiences. This amounts to a richand readily accessible set of teaching resources forteaching testing.

326

4. USING XUNITUsing xUnit for testing involves setting up

"microworlds" with test vocabularies specific to theapplication being tested. This is a 'test fixture' ofknown objects that can be accessed from every testmethod. A setup method gets called by the frame-work's main 'engine' before each test to create theobjects to be tested. A teardown method is alsocalled by the framework after each test method andthe code put there should destroy all objects andgenerally clean up memory usage. This means thateach test method can be assured a known set ofobject of known state. Test methods can then ex-ercise functionality on these objects and check ifmethods are performing appropriately.

As a mature framework should (Foote & Yoder1997) xUnit 'works out of the box' exposing its serv-ices with a very compact interface and generally hasa 'gentle learning curve'. A lot of effort has been putinto minimizing the amount of code that needs to bewritten to create a test. A student can start usingxUnit to write tests after being given only a few linesof boiler-plate code. Pilot studies of students usingxUnit in this manner have shown a marked reduc-tion in code defects (Edwards 2003).

There are also a 'down side' to using xUnit. Aframework by its very nature is complicated. Pow-erful and subtle object-oriented mechanisms areused as a matter of course when writing tests withxUnit. These techniques include extension by sub-classing from inheritance-based class hierarchies,conforming to public interfaces, understanding andusing run-time reflection mechanisms, the placementof code in different methods which will be calledfrom a coordinating super class Template Methodand generally programming according to the 'Open-Closed Principle' (Meyers 1988, p25, Martin, 1998)This level of understanding of object-orientation isnot what was envisaged in the lower level NACCQprogramming modules. To use xUnit requires eithera major step of faith ('it just works') or a sophisti-cated understanding of object-orientation. Mosttutorials about xUnit take a pragmatic approach andinclude lots of applied examples (eg Beck & Gamma1998b). There are also more detailed articles ex-plaining how xUnit works (Beck & Gamma 1999)but these are not promoted as introductory docu-mentation.

The original vision for xUnit was of a simple andlight weight framework (Beck 1994). Although theprogramming language roots of xUnit were inSmalltalk, the most popular implementation is JUnitwhich is written in Java. Alternative implementationsin other programming languages also proliferate.Because these ports are created as open-sourcecommunity projects the standard of implementationcan vary widely. It is telling that Kent Beck, the 'fa-ther' of xUnit resorts to Python when discussingxUnit's underlying architecture (Beck 2002, Ch20).JUnit has accrued many additions over time thatmay be off putting to students. There are for exam-ple 37 variations of the assert statement to choosefrom. This complexity has ironically prompted someto refactor JUnit and produce simpler versions(Venners et al, 2003).

Even though xUnit may be off putting because ofits apparent complexity, the alternatives are ultimatelymuch more complex. Manual testing, by enteringassorted values into a graphical interface, is timeconsuming, error prone, difficult for a tutor to verifyand downright boring for the student to do. Auto-mated testing tools which use a record and play-back mechanism are expensive, difficult to use andoften produce scripts which are 'too britle to be use-ful' (Kent 1994). Creation of rigorous automatedtesting software from scratch requires a reasonablyhigh level of programming sophistication. Write good'testware' from scratch is a difficult task for noviceprogrammers still grappling with basic programminglanguage syntax. The focus of the student's effortscan move rapidly from doing testing to writing thetestware and the overall experience will be that testingis too difficult to be worth the effort.

5. CONCLUSIONIt would be a "Good Thing" if novice program-

mers learned early on that testing is an easy processand not just an annoying 'add-on' to be done afterwriting code. Careful teaching of desk checkingbegins the educational process by teaching studentshow to simulate and verify the execution of an algo-rithm. Desk checking is a tried and proven tech-nique. XUnit in contrast is a new and technologi-cally innovative tool for testing. If novice program-mers are taught test driven development with xUnitthey may well discover the fun of testing and alsobecome better programmers.

327

REFERENCESBeck, K. (1994) Simple Smalltalk Testing: With

Patterns. Smalltalk Report, October 1994,.Available on line at http://www.xprogramming.com/testfram.htm

Beck, K. & Gamma, E. (1998a) Test Infected:Programmers Love Writing Tests, Java Report,July 1998, Volume 3, Number 7) available online at http://members.pingnet.ch/gamma/junit.htm

Beck, K. & Gamma, E. (1998b) JUnit Cookbook.Java Report, 1998. Available on line at http://junit.sourceforge.net/doc/cookbook/cookbook.htm

Beck, K. & Gamma, E. (1999) JUnit A Cook's Tour.Java Report, May 1999. Available on line athttp://junit.sourceforge.net/doc/cookstour/cookstour.htm

Beck, K. (2002) Test Driven Development: ByExample, Addison-Wesley, Boston, MA. Pre-publication copies available for download fromh t t p : / / g r o u p s . y a h o o . c o m / g r o u p /testdrivendevelopment.

Canna, J. (2001) Testing, fun? Really? Using unitand functional tests in the development process.http://www-106.ibm.com/developerworks/library/j-test.html

Edwards, S.H.(2003) Teaching software testing:Automatic grading meets test-first coding. InAddendum to the 2003 Proceedings of theConference on Object-oriented Programming,Systems, Languages, and Applications. 2003.Available on line at http://people.cs.vt.edu/~edwards/pos27-Edwards.pdf

Foote, B. & Yoder, J. (1997) The Selfish Class inChapter 25, Pattern Languages of ProgramDesign 3 edited by Robert Martin, Dirk Riehle,and Frank Buschmann. Addison-Wesley, 1998

Jones, E L. ( 2001) Integrating testing into thecurriculum - arsenic in small doses. Proceedingsof the Thirty-second

Jones, E. L. (2000) The SPRAE Framework forTeaching Software Testing in theUndergraduate Curriculum. Proceedings ofADMI 2000( June 1-4, 2000).

Martin, R.C. (1996) The Open-Closed Principle.C++ Report Available for download from http://www.objectmentor.com/resources/articles/ocp.pdf

Meyer, B (1988) Object Oriented SoftwareConstruction, Prentice Hall.

Noonan, R. E. & Prosl, H. R. (2002) Unit TestingFrameworks. Thirty-third SIGCSE TechnicalSymposium on Computer Science Education,(February 2002), pp. 232-236.

Pilgrim. M. (2004) Dive Into Python. Accessed fromhttp://diveintopython.org/ on 20 May 2004.

Testing Framework http://c2.com/cgi/wiki?TestingFramework

Venners, B. Gerrans, G. & Sommers, F. (2003) WhyWe Refactored Junit: The Story of an OpenSource Endeavor. Available for download fromhttp://www.artima.com/suiterunner/why.html