SOFTWARE TESTING, VERIFICATION AND RELIABILITYSoftw. Test. Verif. Reliab. 2008; 18:191–192Published online inWiley InterScience (www.interscience.wiley.com). DOI: 10.1002/stvr.403

Editorial: Software testing isan elephant

This issue contains two papers. The first, An analysis technique to increase testability of object-oriented components, by Kansomkeat and Rivepiboon, examines the problem of testability ofobject-oriented software components when the source is not available but something like bytecodeis. OO components have low testability because information hiding obscures the state that needs tobe controlled and monitored during testing. This research uses a clever idea of extracting a controland data flow graph, which is then used to increase both controllability and observability, makingit easier to detect faults. The second paper, The determination of optimal software release timesat different confidence levels with consideration of learning effects, by Ho, Fang and Huang, usesstochastic differential equations to build a software reliability model. This model is validated ondata that were published in six previous papers. The results will help project managers decide whento release software to maximize its reliability.We all have probably heard the parable about the blind men and the elephant. Each blind man

touches a different part of the elephant and describes what he feels. The men get into an argument,which leads to physical violence, and they resolve the conflict with outside help. In some versionsthe conflict is never resolved.Last spring I attended a workshop on software system testing and was lucky enough to find a wise

person who helped dispel some of my blindness by understanding several types of test activities.Most of my research has focused on using test criteria to help design software tests. Test design

is amenable to objective, quantitative assessments of the tests, as well as automatic generation oftest values, my first research love. Criteria-based test design requires knowledge of mathematicsand of programming—it is an engineering approach. An equally important way to generate testsis from human intuition. Objective test criteria can overlook special situations that smart peoplewith knowledge of the domain, testing and user interfaces will easily see. Both approaches areintellectually stimulating, rewarding and challenging, but they appeal to people with different back-grounds. These two approaches are also complementary and, in most projects, help equally to createhigh-quality software.For efficient and effective testing, especially as software evolves through hundreds or thou-

sands of versions, we also need to automate our tests. Test automation involves programming thatis usually relatively straightforward, involving scripting languages, frameworks such as JUnit orcapture/replay tools. Test automation requires little knowledge of theory, of algorithms, or aboutthe domain, but test scripts must often solve tricky problems with controllability.Many companies focus heavily on test execution. If we combine test design with test execution

and do not use automation, test execution is very hard. This approach is also inefficient and usually

Copyright © 2008 John Wiley & Sons, Ltd.

192 EDITORIAL

ineffective. This is like expecting programmers to take software requirements and immediately startto code without designing or outlining the algorithms. Especially when not fully automated, propertest execution requires meticulous care and attention to detail.A challenge in testing is deciding whether the output was correct or not. Test evaluation is more

difficult than most researchers think and often requires deep domain knowledge and the ability tosolve problems of observability. Test evaluation is logical and similar to scientific experimentation,so a background in fields such as law, psychology, philosophy and math can be very helpful.These four different engineering activities—test design, automation, execution and evaluation—

are essential to testing. But they do not cover all aspects of testing software. Any process requiresmanagement to set policy, organize a team and communicate with external groups. Tests also mustbe well documented and put into revision control. This requires input from all four prime activitiesas we need to know why each test was designed and how it relates to other software products. Thedocumentation must be included as a part of automation, leading us to maintenance of our tests.Tests must be reused as software evolves, allowing the cost of creating a test to be amortized overhundreds or thousands of versions of the software. A related, and very difficult problem is trimmingthe test suite. As its size grows, the suite can become redundant and too large for efficient execution.People in our field sometimes see our own piece of this elephant as the only important piece. A

couple of years ago I sat on an NSF panel reviewing software testing proposals. As the assessmentsprogressed, it became apparent that one participant was on a mission to block all criteria-basedresearch. A direct quote was ‘Only fools say the human should not be involved in testing.’ Oddly,nobody ever said humans should not be involved! At least, nobody on the panel said that; neitherdid any of the proposals. That is, the panelist was raising a false conflict.Why was this conflict false, and more importantly, why was it raised? As a field, we often view

testing as one monolithic activity and do not see the disparate pieces. Recognizing them allow us tounderstand and respect the activities of different researchers, consultants, authors and practitioners.It also allows managers to better organize their teams; for example, if engineers who are goodat criteria-based test design are asked to do test automation or evaluation, they will usually startlooking for development jobs. Treating testing as one monolithic activity:• wastes resources,• blinds us to differences,• creates unnecessary, unpleasant and wasteful conflicts.Software engineering grew out of this attitude with respect to development years ago. Software

development teams have personnel who specialize in requirements, architecture, design, integrationand coding, among others. It is high time we applied similar specialization in testing. This will allowour testing activities to be more effective and efficient, as well as helping managers to organize,train, and retain testers.The first step is the hardest. We must open our eyes, see the entire elephant in its full glory, and

quit thinking that our piece is the only piece. If we do, we can welcome research on all aspects oftesting (such as the first paper in this issue, which is on test automation), teach better classes andexpand the research community’s ability to help industry with real problems.

JEFF OFFUTT

E-mail: offutt@gmu.edu19 October 2008

Copyright q 2008 John Wiley & Sons, Ltd. Softw. Test. Verif. Reliab. 2008; 18:191–192DOI: 10.1002/stvr