wit magazine april_2008

Risk basedTesting

Reduce number of testing scenarios with this insight on -

Pg

14

Inside this issue

Putting the “Test” back in “Test plan”Some things I have learned in Software TestingMorning IdeasHave I got a deal for you?Finally Usability Testing

Pg 57Are you TEAM ready?

Pg 44

Pg 04Pg 28Pg 40 With little Preparation & Teamwork

you can pave the way for new Team Memers.

and more...

April’08, Issue

Test ngWhat Is

India’s 1st Software Testing Magazine

Cover StoryPg 21

Welcome to

is the first conference being organized by PureConferences in India.

We intend making an experience for all participants by creating a platform for professionals and experts from the testing community of the Globe

will revolve around ’Agility in Testing’ The theme will explore and expand on the need for testing teams to adapt to the changing demands of release and quality.

will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known nationally and internationally will be one of the highlights of the conference.

Let’s mark the conference program in our calendar today itself.

will offer opportunities to the partners to associate themselves with this endeavor and showcase their products in the conference.

We are sure you want to participate and/or present a paper or tutorial.

Do visit us at http://www.test2008.in

ContentsIssue1: Vol 1: April 2008

Cover Story

WhatIsTesting.com 1

Is bad testing

eating your

Pr fits?pg 21

04 FROM THE DESK: Morning Ideas by Sunil Gupta

Good design is key for a good and reliable product

08 THE PROCESS ROOM: Software Testing: ~ A fi eld par excellence by Sudhir & Anjan

Software testing as a fi eld has brought in a high level of confi dence in customers and users and has thus far provided far more challenges to the new age engineers and academicians.

12 TEAM MANAGEMENT: Are You Ready? by Michael Bolton

With a little preparation and teamwork, you can pave the way for a new team member.

21 COVER STORY: Is Bad Testing Eating Your Profi ts? by Rex Black

Testing should focus on mitigating specifi c risks to the quality of the system. Sequence of test execution should be driven by associated risk...

28 THAT TIME OF YEAR: Have I Got A Deal For You by Matthew Heusser

The methods and extend of our testing are all choices; you could even say they are trade-offs.

33 LEARN HOW TO TEST: What a Tester Should Know, even After Midnight

by Hans Schaefer Testing The Normal Way is Not Enough.... Where you fi nd defects, dig deeper!

40 LEARN HOW TO TEST: Finally Usability Testing? by Erik van Veenendaal

Will it fi nally happen!

Other Stories44 FROM DEV DESK: Some Things I’ve Learned in Software Testing

by Jonathan Kohl

50 CONTROVERSY CORNER: QA is More Challenging than Development by Mrityunjay

53 Is scripted testing bad? by Vipul Kocher

57 Putting the “Test” back in Test Plan by Paul Carvalho

Tea testers look ataspects of dried & brewedtea leaves and the liquor

Tea testers look ataspects of dried & brewedtea leaves and the liquor

Likewise, PureTesting goes into variousaspects & nuances of so�ware testing

Likewise, PureTesting goes into variousaspects & nuances of so�ware testing

We build innovative, end-to-end solutions, and manage critical testing processes,

and reduce total cost of producing quality so�ware

• • • • • •

•• Banking & Finacial Services • • Datacom & Telecom• • Pharmaceuticals • • Embedded Systems• • eLearning • • EAI & ISVs

•• ••Test Consulting • Testing Services • Testing Products

Global Software Test Consulting & Services company

India • USA • UK • NZIndia • USA • UK • NZwww.puretesting.com

+91 (120) 4621010; [email protected]

Editor’s Note

WhatIsTesting.com 3

Release of any software is a moment of mixed emotions, pleasure and pain, joy and anxiety -- joyousness of seeing the hard work resulting in a release and the anxiety of the unfound bugs that could have been found and fi xed.

After a lot of effort, this magazine is also being released with similar emotions. We take great pleasure in introducing Version 1.0 of WIT magazine. We hope to have many more versions of this magazine, both major and minor, in years to come and hopefully with minimum number of patches. We also hope that the future versions of the magazine will have more features, will be more robust, will have an expanded user base and establish itself as a dominant player.

Instead of the usual Editorial about the contents of the magazine which you anyway can fi nd out from the table of content, we take this opportunity to share with you the reasons for this magazine and who are the people who have contributed to it.

Many of the technical magazines “grow-up” with time. The articles become more “philosophical” and more abstract. Readers who are regular subscribers too grow with the magazine but most of the new comers start fi nding the magazine “too abstract” for their liking. WIT magazine hopes to keep itself relevant to both old readers who grow up with the magazine and new readers who are either new to the testing profession or new to the magazine.

Our sincere thanks to the initial group of people who helped brainstorm the concept of the magazine. They are – Bernard Homès, Danny R. Faught, Johanna Rothman, Kamesh Pemmaraju, Matthew Heusser, Michael Kelly, Rex Black, Scott Barber, Stefan Steurs. Without them the magazine would not have become a living reality. All good things about the magazine are because of this group, the authors and other well wishers. All bugs are because of us.

For putting together this version of the WIT magezine, I acknowledge the efforts of Ashwin Razdan for collaborating with the authors and collecting all the articles; Parag Sapre for designing the elaborate graphics; Anushree Tewari for providing editing support, and Satish Thakur for giving it the fi nal shape in which you see the WIT magezine. Hope you fi nd this version readable and useful and we hope to have a better V 1.1.

~ Vipul

Daily Life

WhatIsTesting.com4

One morning when I was engrossed in my newspaper, my daughter created panic as her school bag had given way at one of the shoulder belts.

I was annoyed at her letting me know only when I was completely torn while she would have seen this happening over the last few days but my bigger reason of annoyance was due to the fact this was the 3rd or 4th time the different school

bags had torn almost at the same place.

Since it was not possible to repair the bag in ten minutes, thus she carried her stuff in another bag but this episode left me thinking.

I realized that the bag was made to meet the functional requirements of the student such as adequate space to carry stuff, compartments for lunch box, pencil box and water bottle. The bag was also providing a good aesthetics but lacked the robustness in terms of handling the stress that comes on the bag with daily usage....

WhatIsTesting.com4

Morning

Ideas

Though a good design is key for a good and reliable product, a robust test and evaluation of the product only ensures that the product meets all-explicit and implicit requirements and is ready for deployment and usage.

Morning IdeasFromthe Desk

WhatIsTesting.com 5

by

Two things emerged when I extended my thought process further.

One, product making whether it is a school bag or a large and complex telecom product requires good understanding and research in terms of features, stress areas, usability, etc without which the product would give numerous problems to the user as the school bag gave to my daughter.

Second, testing plays a very crucial role to ensure that the product meets all requirements including the ability of the product to handle normal and abnormal stress conditions.

If we take the school bag example, it is very important for the product manufacturer to

ensure that the various stitches in the bag take up the full bag load else there are going to be numerous dissatisfi ed customers like me.

Though a good design is key for a good and reliable product,

a robust test and evaluation of the product only ensures that the product meets all-explicit and implicit requirements and is ready for deployment and usage.

It is seen across the IT and software industry that the organizations give a lot of focus to hiring and developing software development teams but do not focus enough on developing testing teams. The test team is put in place either very late on the programs or the test teams are not organized well enough to provide the vital value expected from the Test team.

For product organizations, the composition and organization of the test team plays a very crucial role in the success of the product. Enough time and energy needs to be spent by senior test folks to

• Visualize the product (software and hardware)

• Look at the various fi eld confi gurations

• Identify the tools required for testing

• Look at the FOUR views:

o User view o Maintenance View o Functional/Operational View o Performance ViewIn order to achieve a good understanding of the product before the exhaustive test cycle commences, a few people from the test team need to be dedicated to carry out the Test Engineering tasks (as defi ned above). This set of people should have the mandate to carry out the verifi cation and validation tasks.

In the school bag example, the User View would be to look at all the ways in which the school bag would be used by a user. There would be several user scenarios such as:

• Student lifting the bag from one belt

• Student lifting the bag using the side belt and loading on its back

• Student fully loading the bag with books but weight may not be too much

• S t u d e n t p a r t i a l l y loading the books but the weight may have exceeded the

WhatIsTesting.com 5

by Sunil Gupta, Head of Testing Practice : Flextronics Software Systems

Composition and organization of the test team plays a very crucial role in the success of the product.

WhatIsTesting.com

recommended weight

Maintenance View would be to look at aspects such as ease of cleaning and washing the bag; ease of repairing the bag and how easy it is to fi nd the spares such as zippers, buckles, etc.

Functional or Operational view is another important aspect of testing. Tester has to consider all the functional aspects of the product. In this example, aspects such as following would be tested:

• Ease of keeping books in the bag

• Access to all the pockets of the bag

• Ease of closing and opening the bag

• Ease of lifting the bag on shoulder and on the back

• Enough space to keep books, lunch box, and water bottle, etc.

Last but not the least, the Performance view of the product in terms of load handling, ability to bear stress due to regular use through load balancing is considered. In the example, tester has to take following aspects into consideration:

• Testing the bag for various load and stress situations

• Testing the bag for full load capacity

• Testing the key joints/stitches under the stress situations

• Identify the weak areas and strong areas of the product

Product testing and deployment is a very involved and rigorous task and involves the best of people and the best of the practices.

Suddenly I realized that it was 9:00 hrs by the watch and I had a 9:30 am meeting with the Quality Group to review the Post Release Defect Density of the products that we shipped in the last quarter and had only 30 minutes to reach offi ce. Thanks to god, for my home is only a 7 minutes drive from offi ce.

Bye for now……

Morning Ideas

6

Product testing and deployment is a very

involved and rigorous task

Neilsoft is a specialist engineering services & solutions

company focused on helping our clients enhance their

product engineering efficiency.

Our services in the software product engineering domain

span the entire product lifecycle including development,

testing, and localization engineering.

Pune Bangalore Canton Chicago Cambridge Dalian

Focus verticals: Construction Industrial Machinery Software Transportation Energy

T E S T I N G

Globe as lit dr ee lp ivx ee ryni ma om do eD l

egarotS SecurE itA y

C

CompatibilityTesting

Ieg

ratio

n

nt Test

nig

Perf

orm

an

ce

Testi

ng

Aut

ed

omat

Test

nig

Fu

nctio

nality

Testin

g

Installtion

a

Testign

CertificationTesting

Mlti-b

te

u

y

Testing

[email protected] | www.neilsoft.com/testing.htm

E X P E R T I S E

Partn

ersh

ips

with

industry sle esa sd ee cr os rptsubor

&nevorP

CA M

LM P

CAD

+91-20-2605 3003

�

Software Testing: ~ A fi eld par excellence

WhatIsTesting.com8

IntroductionAwareness on Software Quality has, over the last couple of decades, become one of the defi ning features not only in Indian software sector but also from a global perspective. Thanks to the dotcom crash which was an eye opener for everyone in the fi eld to look back and focus on enhancing the quality of the products that are already in the market for many years. The quality initiatives taken through the last couple of decades, have now become all pervasive and thus given rise to a new avenue within the IT

and ITeS segment known as Software Testing. Renewed growth of the Software Industry has helped testing achieve the position of being called the watchdog of quality. Apart from giving fi llip to the economy, Software testing as a fi eld has brought in a high level of confi dence in customers and users and has thus far provided far more challenges to the new age engineers and academicians. Looking at some of the astounding fi gures:

1. Bangalore has seen a 24% growth in its recruitment drive in software testing. It is

Software Software TestingTesting

~ a Field Par Excellence~ a Field Par Excellence

Apart from giving fi llip to the economy, Software test-ing as a fi eld has brought in a high level of confi dence in customers and users and has thus far provided far more challenges to the new age engineers and acad-emicians.

WhatIsTesting.com8


WhatIsTesting.com8

TheProcessCorner

WhatIsTesting.com 9

by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL, Bangalore

forecasted that India needs about 18000 test engineers in the year 2005-2006.

2. With India having more than 89 companies at SEI CMM Level 5 assessment; 275 Indian software and BPO companies acquired quality certifi cation; and more and more IT-ITES companies have dedicated quality departments responsible for developing and deploying the quality policies and reviewing them.

3. NASSCOM survey has indicated that the road ahead for the Indian software and services companies within the quality arena was largely dictated and tuned to the developments taking place in the global marketplace.

4. The future would see organizations move from quality assurance to business assurance and focus on information security complying with international legislations, developing new global delivery capabilities and enhancing quality processes in new business areas.

This is testimony to the fact that software testing is poised to grow at a phenomenal rate in future. With all these happenings, no one can deny the conclusion that software testing has grown as an independent fi eld within IT/ITeS service segments. And it has given a clarion call to everyone within the professional arena, specifi cally academicians, to closely monitor, analyze and push forward the new horizons of the technology movement that software testing as a fi eld is going to provide. Looking at some more interesting statistics on testing industry:

Facts from Industry1. $3.0 billion of $4.6 billion in outsourced

testing is sent offshore, throwing up opportunities for companies in India that thrive on low-cost knowledge workers. ~ Source Aztech Software - Bangalore

2. Testing could make up to 25-50 percent of software budgets. Independent testing is growing at 50-65 percent while the part of work done offshore is growing at 35-40 percent. ~ Source - Partha Iyengar, vice-president at industry researcher Gartner

3. Testing service team at Wipro has jumped four-fold to 2,400 in two years. In the nine months to December, revenue grew 90 percent to $64 million, three times the industry average. ~Source - C.P. Gangadharaiah, VP Testing services, Wipro

4. The global market for software testing is around $13 billion. In India alone, the demand for software testing professionals is expected to touch 20,000 to 30,000 by December 2006. . ~Source – Internet Search

Recent TrendsWith this background, we can ask ourselves - what conclusion we want to drive from it? Why suddenly is everyone so statistical about gathering all these fi gures? To take a close look below we see from 50’s onwards till 80’s either the products were not tested or if they were then it was out of compulsion.

Quality as a term itself took the real shape under the leadership of Borris Beizer who gave the awareness to the industry and the world about its very existence. The role of quality has become a necessity than a need in the software industry.

If we start looking into history, we will fi nd thousands of such cases where a small mistake in the software has brought havoc to life and money. The most recent one is the Columbia Space Shuttle crash. All such incidents have

Composition and organization of the test team plays a very crucial role in the success of

the product.

WhatIsTesting.com 9WhatIsTesting.com 9



WhatIsTesting.com10

espoused within the IT-geeks eagerness towards “checking before delivering” approach. On the contrary, there was time when products were released without or with minimal testing done. Of course, many a time, these are dictated by how the market wants them. Time-to-market was probably more important than quality of the product. But, time has changed post dotcom crash era and software quality has the maximum thrust in the market today, be it consumer products, electronics, manufacturing or software. And quite rightly, software testing is enjoying its share of this quality conscious market.

Testing as a fi eld has now been more strategic and approach-centric thus adding new dimensions as well as many challenges to it. Previously testing used to be an afterthought activity in the post release phase when the clients

call and say “we found something wrong in your delivered software”. But with time, it is no longer the case. Any small error today is a big issue from a client perspective. With more and more third party software testing companies grooming overtime, the availability of a talented resource pool, making timely deliveries and the commitment to meet the quality expectations of the customer has added many feathers to India’s software testing market. Needless to make a note that Software testing as a fi eld within IT/ITeS has now emerged as a strong area of business solutions and a source of major revenue generator and emerging more as a serious business goal as well an intrinsic part within the IT/ITeS organization culture.

Future Challenges

Domain and Technology PerspectiveThe growing demand from within the customer groups as well the growing demand of the organizations to meet and extend their client support has made software testing as a fi eld nothing but a paramount knowledge pool. There is software in almost every aspect of

human life such as Insurance, Healthcare, Banking, Financial services, Pharmaceuticals, logistics etc. What this means is that, software developed in each of these Industry verticals need testing – some rigorously and some which can give adequate confi dence in the product. At the end, software testing too is spanning across all these Industry verticals. When it comes to domains, the fi eld looks even more lucrative by the amount of challenge each one of them provide to the individuals involved in testing. Therefore, as software testing started to support the development activities in these domain areas, it became increasingly necessary for the test engineers to gain knowledge and specialize within each of these domain areas. This is another area where there is a huge potential for test engineers to grow and make an impact. This led to different teams being formed, some heterogeneous and some homogeneous groups to cater to testing software in these domains. The requirements of the client keep on growing and we as organizations need to meet the demands of the end users to stay in this business. This translates to the whole gamut of technology issues starting from software, hardware to third-party products. Software can be legacy application running on a mainframe or the latest technologies such as .NET, J2EE etc... Technology as we know has advanced from the days of mainframes to today’s applications running on a PC based Linux systems. However, at the same time, applications developed on the legacy technologies have not gone away or have not been replaced with new technology; they are still being used in Banks, Insurance companies

There is software in almost every aspect of human life.... software developed... need testing -- some rigourously and some which can give adequate confi dent in the

product.

WhatIsTesting.com 11


and many others. This even complicates the scenarios wherein the vendors have to support both old applications as well as new applications. This tells us how complex it can be for testing to handle such complex set-ups. Besides, the applications run on varied platforms from different vendors including: Windows, Linux, Macintosh, IRIX, AIX, unix & Mainframe. We are currently looking from a macro level. Since with the growing numbers of different fl avors of an individual OS (e.g. Linux as such has Linux Workstation2.0/3/0, Advanced Server 2.0/3.0, Enterprise server 2.0/3.0 etc.) the coverage from testing point of view becomes immeasurable. Does that not give a test engineer enough challenge to work on so many platforms? Of course yes, it’s not that developers only code for making an application run over different platforms but the test engineers too have their own way of reaching end to end. Apart from this we can also predict a good deal of exploration in areas like Client-Server, Web (2-tier, 3-tier) applications, Mainframe application and an endless list of such applications.

What we discussed as far as technology is concerned is only the tip of the iceberg. There seems to be an endless list and mushrooming application areas to be explored in the time to come.

ConclusionMoving a little off track to give a comparative analysis we see in the past Public Administration has struggled a lot to give itself an individual identity and for a long time shared a part of its glory with either Political science or Sociology. But today it has gained a public reputation and has become a part of everyone’s life.

So does it hold true for software testing which has remained cocooned within Software Engineering or Software Development Life Cycle. But now the fi eld has not only gained reputation but also has stood on its own. Software testing as a fi eld has come of age bearing the pain of “We can do without you” to share the glory of “With you we are right through”. A fi eld that is still growing, shining and awaiting to fl ourish still needs a lot of planning & strategizing from all sections in the Industry and academicians. As we need to remember the following lines:

“The Road to Excellence is a never ending Road” ~H. James Harrington, The Improvement Process

Are You Ready?

WhatIsTesting.com12 WhatIsTesting.com12

AreAreYouYouReady?Ready?With a little preparation and With a little preparation and teamwork, you can pave the way teamwork, you can pave the way for a new team member.for a new team member.

Use a checklist to make sure you have everything ready before the new team member arrives.● Compile all necessary documents into one binder

that can be given to any new arrivals.● Be sure to provide some context about the company

and the team to help the newbie get acclimated.

WhatIsTesting.com12

TeamMana-gment

If your company is typical, it is hiring more and more temporary workers while scrambling for every bit of effi ciency and value. Hiring a contractor to help with

temporary problems can be cost-effective, but wasting his or her time undermines the purpose of the exercise and costs hundreds of dollars per day. Moreover, people simply don’t work as well when they feel uninformed, frustrated, or stymied. On the other hand, a contractor who can hit the ground running will make better relationships, will be more effective, and will use your time and hers productively.

This article describes ways to prepare for a new contractor; however, the tips here could just as easily be applied to a temporary employee, a new permanent employee, or in some cases, a transfer into your department. I don’t assume any particular job description or title for the person, nor do I assume anything about the person’s technical skill in any given category. Neither should you make such assumptions. A crackerjack developer may be completely oblivious to network confi guration issues — especially in a new environment — and a brilliant documentation writer might be fl ummoxed by a cryptic voicemail system. Your contractor’s time is best used to solve the problem for which you’re hiring her, not her own infrastructural or contractual issues.

Contracts and AdministriviaBefore beginning work, your contractor will typically sign a contract outlining the scope of the work to be done, the deliverables expected, and a schedule detailing when each element of the work should be completed. The more specifi c the contract, the less opportunity there is for miscommunication and disagreement, and the better your interests are protected. Your contractor may have requirements that are outside the scope of your company’s standard contract, or she may have to provide certain kinds of documents relating to work eligibility. Make sure these kinds of issues are ironed out before the contractor is slated to begin work. Most contracts come with a non-disclosure

agreement. Make sure this, too, is prepared in advance, and that all agree on each of its points before the contractor’s fi rst day. If your human resources department has policies and procedures manual appropriate for your hire, make a copy available to the contractor early in the game. Human resources will typically require various kinds of information from your new hire. Be sure to ask in advance what information they’ll need so that you can pass on any questions to the contractor prior to her arrival. Human resources or security will typically supply ID badges, pass cards, access codes, and keys to the offi ce, restrooms, or other secure facilities. These items should be available and tested.

If the contractor is arriving from out of the country, make sure that your human resources department and the contractor have fulfi lled all requirements related to visas, work permits, and the like. The accounting people will need to deal with administrative and taxation issues. Make their jobs easier by coordinating and relaying the required information in advance, both from your staff and from the contractor. Issues related to eligibility, tax numbers, tax forms, and withholding information should be sorted out concurrently with the contract.


Restrooms with locks but no keys are scourges upon the land. When I started at one small outfi t, it was a week before security could produce a restroom key

for me. It wasn’t only a bother for me to ask my new colleagues for a key—I was interrupting their work, too.

At this company, the restrooms were locked, but the door to the server room was always open.


by Michael Bolton, DevelopSense, a Toronto-based consultancyand training fi rm.

Are You Ready?

WhatIsTesting.com14

There are few things that make a contractor happier than being paid on time and with a minimum of hassle. Ask your accounting department what they’ll need from the contractor to pay invoices promptly.

Ensure that expense and reimbursement policies are clear from the get-go. If your company uses a standard expense reconciliation form, make sure that a few copies or an electronic template are available to your contractor. Again, your most friendly and helpful contact person in accounting should be available to the contractor to help sort out confusion or diffi culties.

IT IssuesNotify the IT support department that a new hire is on her way, and request the specifi c things you’ll need and the date you’ll need them. Most new arrivals will need at least one computer set up, linked

to the network, loaded with tools, attached to printers, and tested. If you’re hiring a developer or tester, additional terminals or workstations may be necessary. Don’t forget to tell the IT department about this, and above all, don’t assume that they’ll know what the new arrival needs. Your new arrival will need access to various areas of the network, so make sure she has the appropriate set of access rights. If your company’s security is very restrictive—for example, limited

Internet access—notify your contractor beforehand so that other arrangements can be made for Net-based research tools and resources. An analog line might be required for some purposes. If specifi c tools—examples include compilers, confi guration management programs, and testing utilities—are required, have these loaded and tested on the contractor’s machine in advance. Be sure to follow up, verifying that the work has been done and that resources, confi guration settings, network policies, and default passwords are documented. Make sure that email accounts have been set

Elizabeth, a European contractor of my acquaintance, was hired to work in an American company. She submitted an invoice, covering salary and expenses, which was

rejected on the basis of a couple of questions from the accounting department about her tax withholding status. She dropped in to investigate the problem by speaking with Mark, the accountant who had sent her the mail rejecting the invoice. “I don’t have time,” said Mark. “Not even for just a couple of questions?” asked Elizabeth. “No,” Mark replied. “It would be a different story if you were an employee rather than a contractor. But,” he said imperiously, “I have more important strategic things to do.” This incident occurred at a company where more than half the staff was made up of contractors.

Eventually the problem was sorted out, but it meant inconvenience and exasperation for Elizabeth and her manager, and embarrassment for Mark’s manager. On the other hand, Elizabeth’s fi nal report included a subtle recommendation—which Mark’s manager followed—to give Mark less strategic work to do. Whether Mark is currently doing strategic work or not, he’s doing it somewhere else.

We challenge you.

At GrapeCity, we challenge you with assignments on the latest .NET and Java technologies, enabling you to emerge a winner in this game called life.

GrapeCity is a software development multinational company

with over 650 employees across Asia and the United States. For

more than 25 years, we have brought optimized software solutions

and services to enterprises around the world.

We rely on four key principles to help our clients achieve their goals:

thoroughly understanding our clients’ business objectives, providing

highly personalized experiences, maintaining a strong emphasis on

quality, and adhering to the highest ethical standards.

Send your resume [email protected]

now!

GrapeCity India Pvt. Ltd. A-15, Sector 62, Noida 201307 Phone: +91 120 247 0123 Fax: +91 120 247 0124www.grapecity.com

Lines of Business

Business Solutions

Custom Applications

Mobile Applications

CRM

ERP

Mobile Porting

Quality Assurance

Technical Support

Component Development

Technical Services

Technical Tools

Are You Ready?

WhatIsTesting.com16

up and checked by sending a message to the new arrival’s address, and then check with the IT department to be sure the message has been received in the new mailbox. In addition, obtain a document from the IT department on how to change network login and email passwords.

Everyone needs a working telephone and

voicemail system. Have the telephone installed, and a number ready for the contractor when she arrives. Make sure that this number is listed in the company directory — and remember that directories can exist on the network, in the voicemail system, and with the receptionist.

Despite everyone’s best intentions, a company’s information infrastructure can be complex and confusing to a new arrival. Have the IT department provide the name of a specifi c person—a guide or sponsor who is helpful and knowledgeable about the company—to help the contractor navigate through network access, email confi guration, and printer problems. In addition, make sure that your contractor will be able to fi nd her default printer, email address, network login name and password, and IP address. Try to counsel colleagues against sending email announcements saying things like “This week’s build can be found in the usual places.” Instead, make the details explicit.

Providing ContextThe contractor has been hired to accomplish some task or solve some problem. If people within your company are having diffi culty sorting out an issue, imagine the diffi culty that a stranger will face!

Even people who have worked inside your company and who know the general environment need background information when they arrive in a new position. This is yet another motivating force for your company to develop clear specifi cations, design documents, plans, schedules, budgets, and so on. Start by writing a brief description of what the company and your division do, and prepare an organizational chart or departmental roster. At the very least, identify yourself and your direct superiors and reports, along with the other people with whom you expect the contractor will be working. As you’re

Note the people who are most likely to be helpful in answering questions or fi xing

problems, and provide their contact information.

The new arrival can often help people to notice old bumps in the road. I was brought into a test organization to help identify problems in testing

effi ciency. There were two types of network drops in the test lab—one for the company’s internal network and one for the test network. These weren’t labeled, since everyone who used the lab knew which was which. I suggested that this shouldn’t be taken for granted as one of the locals helped me fi nd the right connection, and I used a bit of masking tape to label the drops nearest to my workstation. The next day, a fairly senior tester reported a defect that was later found to be spurious, at a fairly signifi cant waste of time and effort. The misunderstanding happened because he had plugged the test machine into the wrong network connection. The day after that, labels magically appeared on all of the network drops.



preparing this information, note the people who are most likely to be helpful in answering questions or fi xing problems, and provide their contact information, so as to minimize the amount of time that the contractor must wait—unproductively—for answers. Document the key business processes and workfl ows associated with the new arrival’s task at hand. If you provide answers early, there will be less time spent wondering and asking questions.

Don’t get hung up on tools or formatting: A document composed using a text editor or even a pen and paper is infi nitely superior to a document that doesn’t exist at all. When offered a choice between a detailed Notepad fi le and an empty Gantt chart, I’ll pick the former every time. Simply make sure that the information exists, and that it can be found and read easily. Make it a mandate from the beginning, and check with the new arrival often. If the new arrival feels that she is being forced to do detective work, encourage her to keep a log of what she needed to know and what she found. Make sure that one of her deliverables is a list of the things that will be useful to her successor, and be sure to budget time for her to prepare it.

City GuideIf your new arrival is coming in from out of town, she’ll feel more welcome with a guide and some idea of where important services can be found. If your company is providing accommodations, make sure that the contractor’s lodgings are quiet, secure, comfortable, and reliable, and that they’re ready for the contractor by the end of the fi rst day of work, at the latest. If arranging accommodations is the contractor’s responsibility, provide a list of facilities in a variety of price ranges. While it’s nice to be close to work, don’t strand the contractor near an industrial park without food or services nearby. Prepare a list of restaurants in the area that cater to a variety of tastes and budgets.

Ask your department for recommendations,

and you’ll quickly have a list that is easily maintained and printed.

One of the benefi ts of traveling, from the point of view of many contractors, is using free time

The processes and documents thatyou prepare for the arrival of one

contractor will be highly recyclable.

At one company where I was a full-timer, we hosted a number of developers during an outsourced project. We created a list of local restaurants and

tourist attractions by sending out email to everyone in the department, soliciting one suggestion from each person. I compiled the suggestions into a single document, removed the duplicates, and posted the results to an internal discussion board. A few of the suggestions inspired still more. Some might suggest that discussion of restaurants on company time and company bulletin boards is wasteful. Our experience was that the one-suggestion technique cost next to no time and was a way of getting to know people—and people who knew each other better almost always worked smarter and harder.

We also found that a handy list made it easy to book company gatherings, to provide directions, and so on.

Are You Ready?

WhatIsTesting.com18

to explore their surroundings. Survey your staff for a number of interesting— and possibly undiscovered— places to visit in the area. If those places are somewhat off the beaten path, ask the person making the recommendation to provide a link to one of the several useful Web-based mapping services. If everyone pitches in a suggestion or two, compiling them into a helpful guide shouldn’t be burdensome. When you’re fi nished, you should have a complete package of general information that can be passed to any new employee or visitor on paper, via email, or on an internal Web site. Ask the new arrival frequently for feedback on the checklist, and refi ne it accordingly.

On the JobMake sure that you schedule some time on the contractor’s fi rst day for introductions and a tour of the essentials. You should be prepared to walk around for at least an hour with the new arrival, introducing her to the

people she’ll need to know. An introduction from you is likely to carry more signifi cance than one from a subordinate, so be prepared to do this yourself. Even if you are routinely busy, getting your employees and contractors to work together is a tremendously worthwhile investment. A contractor’s effectiveness often depends upon being able to meet and connect with other people. Most contractors meet their new colleagues in business meetings, but more informal circumstances can lead to friendlier relationships. On the introductory tour, take your time. Cover the important places and people, but be relaxed and try to avoid a hard-and-fast agenda. Spend a few moments introducing the new contractor to each staff member. Explain quite generally what the new arrival is there to do, and describe your staff member’s role, and how each may be able to help the other. Don’t just concentrate on peers and colleagues. Spend a few moments chatting with other useful contacts with whom your new arrival will need to interact—people in network services, reception, the mail room, and accounts payable. As you’re walking around, show the new person all of the useful and necessary places around the offi ce. An early stop should be the restroom. If keys or pass cards are required, make sure that the contractor is given them upon her arrival. The IT department should have set up a default printer and informed you which one it is. Include this on the tour, along with specialty printers and other equipment that might be useful. Also on the tour, include visits to the offi ce supply cupboard, the photocopier, the fi rst aid kit, and the lunchroom. For the duration of the contract, the contractor is a member of your company and your department, and thus should be accorded at least the same kind of assistance and respect given to your permanent employees. If you are the contractor’s supervisor, you must be prepared to go to bat for her, and the rest of your organization should be prepared to take the same approach.

No matter what the task, there’s nothing worse than having to research and answer the same questions over and over again. Keep a list of

The kitchen or lunchroom is not only the place for refueling but also a good place to meet and converse with others. Linger for a while and encourage chatting, even

if it’s not directly related to the task at hand. Spend a little petty cash on group lunches, outside coffee breaks, or visits to the pub with your contractor and the rest of your staff. If these informal approaches seem of questionable value, consider that productivity and effectiveness are often based on personal relationships and rapport—do what you can to foster them.



Checklist for a New Team MemberPrepare for Arrival* Signed contract* HR info from contractor* Keys, badges, and codes ready and tested* Visa, work permit, and eligibility confi rmed* Resolution of tax issues* Telephone and voicemail set up* Phone number listed in directory* Set up computer:

networkedall applications installed and testedemail accountloginpasswordsprinter connectionDocuments to Provide

* Policies and procedures manual* Expense and reimbursement policy* Expense forms* Telephone and voicemail instructions* Telephone directory, with key contacts highlighted* Internet policy* Password and login instructions* Software resource manuals* Names of key contact people in IT, HR, Accounting, etc.* Organizational chart* Division responsibilities* Overview of company* Key business processes and workfl ows related to job* Guide to city, if applicable* List of frequently asked questions (with answers)

frequently asked questions, their answers, and where someone can go for more information. In many cases the FAQ information can be posted on an internal Web site, captured in a database within workgroup software, or collected in a piece of email. If you don’t have such things set up, consider making the collection of FAQs a task for the contractor, making such a document part of the contractor’s package of deliverables, since your contractor will have plenty of insight on what’s important for a new arrival to know.

Reuse and RecycleAll this might sound like a lot of work— and the fi rst time you do it, it will be.

Luckily, the processes and documents that you prepare for the arrival of one contractor will be highly recyclable; they can be used again and again, regardless of whether the new arrival is a temporary or full-time employee. Since you’ll be consulting with other departments, you can exchange ideas and plans, and improve the lot of new arrivals companywide. The work, once

complete, can be reused and refi ned, but rarely will you have to start it again from scratch. You’ll need to review things periodically, but the benefi ts will be immediate: happier, better-prepared, and—most importantly— more productive workers.

CORPORATE OVERVIEWHexaware is a leading global provider of IT and BPO services. Over 40 of our clients are Fortune 100 companies. Weenable clients achieve competitive advantage by co-developing innovative IT/Process capabilities delivered throughflexible business models. We focus and have achieved leadership positions in domains such as HR, Banking andFinancial Services, Insurance, Leasing and Transportation. On the technology front we specialize in BusinessAnalytics, Enterprise Applications, Application Modernization/Management and Independent Testing. Founded in1990, we are a US$ 153 Million company with over 4000 employees in 16 locations worldwide. We are currently rankedas No. 11 Software Company in India.

HITS(HEXAWARE INDEPENDENT TESTING SERVICES) OVERVIEW Team of 250+ with varied skill sets Domain focused, Automation driven E2E testing solutions Independent testing labs based in Chennai and Mumbai Showcases projects spanning a diverse spectrum of Domains – BFS, Insurance, ERP and Airlines Technologies - Mainframe, J2EE, Siebel, People soft and SAP

Expertise in test automation tools like Win runner, Quick Test Pro, Load Runner, Test Director, Rational suite.

More than 35% have hands on experience in Mercury tools. Proven Test Methodologies from end-user perspective to meet the customers expectations under real-world

business scenario. Offshore – Onsite Delivery model providing significant cost savings More than 1000 Man years experience

Hexaware Technologies

TECHNICAL EXPERTISE Technology – Legacy

Systems, Client Server, J2EEand .NET. Operating Systems –

Multiple Virtual System(MVS),

UNIX variants and Windowbased.

AUTOMATION TOOLS EXPERTISE Win Runner, Load Runner

and Test Director - MercuryInteractive. Functional, Performance

and Test Management Tools- Rational Software. Silk Test and Silk Performer

from Segue Software.

Our Key Offerings

Fail Over / Availability Testing

BR / FS/SR SpecificationGap Analysis

Functional Support

Black Box - Functional TestingWhite Box - Interface TestingGrey Box - Regression Testing

Existing Methodology ReviewExisting Tools Evaluation Process Recommendations

Load TestingStress/Volume Testing

Analysis and Interpretation

Multi-tier AvailabilityLow Resource

Endurance Tests

UAT Preparation & Execution Roll Out / Production Support

User Training

Test ManagementFunctional AutomationLoad Test Automation

Performance Testing

UATSupport

Test Automation

Software Testing Process Consulting

System Integration Testing (SIT)

Business Requirement Analysis

Airlines

Manufacturing & High technology

Banking & Securities

Insurance

Industry SpecificSolutions

IndustryIndustry SpecificSpecificSolutionsSolutions

Technology PracticesTechnology PracticesTechnology Practices

Application Management

Enterprise Application Integration

People Soft

Oracle

Siebel

E-Business solutions

Hexaware Towers, 51/3, G.N Chetty Road,

T N

Bldg. No. 152, Millennium Business Park, TTC Industrial Area, S 3 "A" Bl k M h


by Rex Black, President, RBCS, Inc., Bulverde, TX

Since the 1980s, when Bill Hetzel and Boris Beizer published their infl uential books on software testing, we have known that testing should be

risk-based. Testing should focus on mitigating specifi c risks to the quality of the system. The sequence of test execution and the total effort expended on any given test should be driven by the level of risk associated with that test. In the 1990s, people like Rick Craig, Paul Gerrard, and I independently created some ways to achieve systematic risk-based testing, often by adapting ideas from other forms of engineering, such as insurance and medicine. In this article and the next one, we’ll talk about ways to perform risk based testing.

Categories of Quality RisksComputer software, hardware, and systems can fail in the most amazing and various ways. Some bugs are not functional problems, but fall into other quality risk categories. We’ll take a look at some categories of quality risks to stimulate our thinking about what to test.

FunctionalityThe most obvious quality risk category is that of functionality. There’s always a chance that the system does not provide some function it should. The function can include a capability, feature, processing, input or output.

For example, if you’re testing a calculator

Is bad testing

eating your

Pr fits?Using Quality Risks to Guide Testing Eff ortTesting should focus on mitigating specifi c risks to the quality of the system. Sequence of test execution should be driven by associated risk...

WhatIsTesting.com21

Cover Story

Using Quality Risks to Guide Testing Effort

WhatIsTesting.com22

program, the lack of an add capability would fall into this category. So would a situation where the add capability was implemented, but the “+” key didn’t cause the function to be activated. The add capability might be implemented and accessible, but might work only on integers, not real numbers. The add operation, when carried out, might give the wrong result, as in 2+2=5. Or there could be some strange side effect where the operation is handled but the result is not at all as expected, as in a divide function where 2 divided by 2 returned one, but in Roman numeral format.

Performance and ReliabilityIn Figure 1, you see a graphical representation of three types of quality risks in the area of

performance. The shaded area at the bottom of the fi gure represents the required performance, which is one second or less transaction processing time at a load of up to 1,000 transactions per minute.

Risks to system quality in the area of performance include the possibility that the system responds

to input, produces output, or processes data too slowly under all levels of load. The straight

dashed line shows this possibility. The system might perform fi ne up

to some level of load, but have an unacceptable non-linearity in the

performance curve, as shown by the steep curved dotted line. Finally, the system

might perform within specifi cations during an initial test run, but subsequent tests—when the

Figure 1: Performance bugs


1000Transaction Arrival Rate

Tran

sact

ion

Pro

cess

ing

Tim

e

Unacceptable performance load above 500

Unacceptable performance degeneration above time

Unacceptable performance at any load

Acceptable performance

4

3

2

1



system is not rebooted between tests—might reveal unacceptable performance degradation, as shown in the family of three gently curved lines.

The risk of performance degradation—and indeed, behavioral degradation of any sort—over time brings us to the reliability category of risks to system quality. Reliability problems exist when the system fails to function normally each and every time. Such a reliability bug can exhibit itself as an intermittent functional problem. Sometimes, such bugs appear after a long period of runtime or after running under heavy load. I had such a problem with Windows 2000 while writing this article. Extended uptime between reboots usually resulted in characters in some applications taking on the wrong font or the wrong font size.

You might have seen another type of reliability bug, too, where the system functions normally when it functions, but crashes or hangs unpredictably. Such bugs can occur after some long period of runtime or after running under heavy load. Alternatively, such bugs can be truly random. I recently saw such a bug when a security patch caused reboots on a system every third or fourth time the system booted.

Stress, Capacity, and VolumeRisks to system quality in the area of capacity include seeing functionality, performance, or reliability problems due to resource depletion. For example, the performance of operating systems often starts to degrade once the system consumes more than 80% of the available hard drive or memory space.

Risks to system quality in the area of volume

include seeing functionality, performance, or reliability problems due to the rate of computational, data, and communication fl ows. For example, the performance of database management systems often start to degrade when the database management system load exceeds 80% of the rated transactions per minute or the allowed number of simultaneous connections.

Installation and DeinstallationVarious things can go wrong when installing programs. These include situations where the program won’t install or the installation process causes damage to the system.

The latter kind of problem once was ubiquitous on Windows systems, with the

appropriate nickname “DLL hell.” Here, one program overwrote

libraries used by another program during its installation process.

Deinstallation can also create problems. Sometimes, deinstallation

does not completely remove fi les and undo changes. Sometimes, deinstallation causes damage to the system.

OperationsWe also have those risks associated with recurring operating activities.

These include activities carried out at the end of some operational period, such as end-of-quarter or end-of-month closing out of accounts. Problems can arise especially with failures to archive inactive logical records in a safe and recoverable manner, as well as archiving of data that should remain active. In addition, the

Reliability problems exist when the system fails to

function normally each and every time.

Various things can go wrong during installing & deinstalling

programs.


WhatIsTesting.com24

timing of archiving and the calculation of when a period has ended and a new period begun can fail.

For systems with databases, there can be regular requirements to compact or repair the databases. In addition, for such systems, operators often must back up data and confi gurations regularly, along with verifying the restore process. There’s a strong risk of side effects like performance problems related to backup or restore operations, the inaccessibility of some or all system functionality during backup or restore operations, and the failure of backup or restore operations under full system capacity conditions. Sometimes, operations activities happen automatically, in the background. For example, rollback logs for a database often need to be purged automatically to avoid major performance and reliability bugs.

RegressionBy regression, I mean when a previously-working feature, function, capability, or behavior fails following a change. The problem we have in system testing is that software/hardware systems, being digital, tend to be discrete rather than continuous in their failure modes.

The effects of changes, even small, localized, isolated changes, are not always small, localized, or isolated. How many times have you heard a statement like, “How could that have happened? I only changed one line of code.”

A small change, even one line of code, can affect the behavior of the rest of the system. A small change can cause incompatible data to enter shared data sets, including dynamic datafl ows and static databases, resulting in bugs that

exhibit their effects a long way from their origin. Side-effects of changes can impair or even break cohabiting, interacting, or underlying software. The adding of yet-another-feature can lead to systemic performance and capacity issues.

Usability and User InterfaceOne special category of non-functional quality risks relates to the usefulness and usability of the system by the intended user or operator. Specifi c usability and user interface bugs can vary. In some cases, systems present cumbersome interfaces that do not follow normal or expected workfl ows, leading to user frustration, confusion, and mistakes. In other cases, functionality, while present in the system, is practically inaccessible to the user

or operator. Systems can be inappropriately diffi cult for the users to learn, leading

to abandonment, mistakes, and ineffi ciency. Finally, users and operators can fi nd instructional,

help, and error messages that are misleading, confusing, or misspelled.

Data QualityMany systems exist primarily to do interesting things with data. Inputs are processed, transformed, and stored. Databases are queried, records linked through foreign keys, and integrity constraints enforced. Outputs are displayed, printed, and transmitted to other system.

So, data quality poses a major category of quality risks for many systems. The system might corrupt or lose data. The system might store bad or nonsense data in a database without integrity constraints. Databases shared across multiple systems can allow data which is

The effects of changes, even small, localized, isolated changes, are not always

small, localized, or isolated.

Data quality poses a major category of quality risks for

many systems.



valid for one system to be accessed by another system which does not know how to handle that data, resulting in failures that are removed in time and feature space from the source of the problem.

Further complicating these situations, it is sometimes diffi cult to restore the system to working state after a badly-handled error condition. In some cases, the system, in the course of succumbing to the error, damages confi guration fi les or static data stores in a way that’s hard to fi x later. The data quality bug in the expense reporting package I mentioned earlier is an example of this kind of misbehavior.

Date and Time HandlingIn addition to having to handle common errors, many systems must also handle dates. This has created signifi cant information-technology problems, including the infamous “Y2K bugs” that consumed huge proportions of company and government IT budgets in the late 1990s and contributed to the depression in the high-technology sector in the early 2000s. In addition to having to handle once-in-a-hundred-lifetimes type of events like a new millennium, systems must frequently handle events like leap years.

Many systems must deal with expiration dates. Licenses expire. Credit cards expire. Insurance policies expire. After some period of time, the right of a bank to disqualify a borrower based on a derogatory like a bankruptcy can also expire. Failure to handle such expiry events properly can expose a company to serious fi nancial and legal risks.

Localization Localization refers to the ability of a system to support local customs, languages, norms, and laws. There are two broad categories of quality risks related to localization. One category relates to the user interface, and the other to operations.

Of course, if the system does not support the character sets used by the local language, we have a localization problem. Those languages that use the Roman alphabet, such as English, German, and Spanish, have single-byte character sets. Some languages that use other alphabets, such as Russian and Hebrew, have single-byte character sets, too.

Localization also has operational implications. For example, time zones and time formats vary based on locale. Does the time change in the summer? (In the United States, this is called “daylight savings times,” but it goes by different names around the world.) If so, when does

adjusted time begin and end? Are dates written “month-day-year” (as in the United States) or in the more logical “day-month-year” format (as

in much of the rest of the world)?

Confi guration and Compatibility

Another family of risks to system quality lives in the areas of confi guration and compatibility. Different users confi gure both the hardware and software of systems differently. A family of systems may support various hardware confi gurations. Often, the number of potential combinations of confi gurations is huge and testing inappropriate combinations can result in risk of failure in the fi eld.

Standards and Regulatory ComplianceMore risks to the quality of the system exist in the areas of standards and regulations. A system can work properly but be excluded from the market by failing to meet standards, either

Quality risks related to Localization may be in the categories of GUI or

operations.


WhatIsTesting.com26

effectively (no one will buy it) or legally (the government won’t let anyone buy it or sell it).

SecurityWith the growth of the Internet, awareness of the risks associated with security has increased. Security risks are legion. They include viruses and worms, criminals breaking into servers, vandals causing denial of service attacks, bugging and intercepting e-mail and Internet communications, and more.

Timing and CoordinationWhen you have communicating components, or shared components, timing and coordination between those components is a concern. For example, with an e-commerce system, how long should the system wait between events like mouse-clicks and submitted screens? At what point can it safely conclude that the customer has abandoned the transaction? As another example, consider the automated teller machine. What happens when the central computer times out or the network goes down? What if that happens in the midst of a transaction? If we’re testing an inventory system, what happens when two salespeople try to sell the same item at the same time?

DocumentationFor many people the quality of the documentation signifi cantly affects their experience of quality when using the system. Documentation quality problems include being technically incorrect, of course, especially the examples. The user can fi nd the documentation insulting or offensive, teeming with grammatical or spelling errors, or affl icted with distracting cosmetic or formatting problems.

Documentation refers not only to hard copy, but also to electronic documentation. Help screens, installation instructions, error messages, and wizards are a form of documentation.

Can You Think of Other Quality Risks?Yes, and so could I. But isn’t this list already too long? What if you tried to cover all the quality risks discussed so far? Could you test them all? Would they fi t into the budget and schedule context of the project? Would you end up wasting time and effort on unimportant problems? Surely any attempt to test this whole set of quality risks would be neither effective nor effi cient!

So, what shall we do? In the sequel to this article, we will discuss some

ideas of how to trim the infi nite number of things you could test

into a fi nite list of risks you should mitigate.

About the Author:

With a quarter-century of software and systems engineering experience, Rex Black is president and principal consultant of RBCS, Inc., a leader in software, hardware, and systems testing.

This article is based on an excerpt from Rex Black’s book, Pragmatic Software Testing, published by Wiley

When you have communicating components,

timing and coordination between them is a concern.

Have I got a deal for you?

WhatIsTesting.com28

As this is our fi rst issue, we are in a unique place—: This issue will set the ground rules for every issue that follows. Future

issues will build on this one, so we have to lay the right foundation. I would like to start that foundation with a simple assertion:

The methods and extend of our testing are all choices; you could even say they are trade-offs.

Life is all about trades. We trade a “work week” of our lives for a pay check, then we trade that pay check in for a home, for food, and utilities or clothing. We trade some of our freedom for a family.

At each stage of the game, we are

giving something up – time, money, energy, or effort, in exchange for something we value more. If you don’t think it’s like that in the world of software testing … think again.

Having an independent test group splits management attention and costs money, but it also decreases risk and provides an impartial assessment of the quality of the software. Using bug-tracking software costs money and creates the risk of “information overload”, but also ensures that every incident is logged and

tracked. With a complete list of incidents, a change control board (CCB) can determine which defects should be fi xed in the next release, by priority, but that invariably slows down bug fi xes and decreases the personal touch.

Have I got Have I got a DEALa DEALfor you …?for you …?

ThatTime of Year


by Matthew Heusser, Socialtext, California, US

Documented test cases are great, right? We couldn’t live without documented test cases, could we? Well, maybe. Time spent documenting test

cases is time that could be spent testing. There is also some evidence to suggest that following the same test scripts again and again leads to thinking about problems in the same way, which could cause a miss. Many times, fi nding a bug (or a repeatable error condition) is like playing a game of twenty questions; how likely are those scripted test cases to get the right answer if they lack feedback?

Requirements-driven testing requires written requirements, and “locks in” the feature set at the beginning of the cycle, when we have the least information about what the users actually need and how the software will support them. Reviews and Inspections of documented test cases slow down the test process, again taking time away from test execution. Exploratory testing requires talent and practice; exploratory testing in the hands of someone without experience could just be a waste of time.

How can your team get “better?” Mentoring, Coaching, and Skill Development are all very hard to measure and hard to evaluate. Test Process Improvement is easier to measure, but also of more questionable value.

Not all trade-offs are equal; the trick is to carefully weigh the pros and cons before making a decision – to consciously realize what you are trading away, what you are trading for, and if the switch is worth it. The question is, what do you value more? Which outcome is “better” for your company, with your customers, selling your product, at this point in time?

I’m harping on this point for a reason. Not because it’s so blatantlyzing obvious (it is), but because so few people are talking about it. Instead of talking about trade-offs, they are talking about “Best Practices”, which, apparently, always solve all of your problems without introducing any new problems. The fact that this defi es common sense doesn’t seem

to enter into the equation, and that worries me a bit.

Okay, more than a bit. It worries me a lot. It’s downright irresponsible to suggest that there is

one single way to do it – a single one-dimensional scale for goodness. The real suggestion that the “Best Practice” crowd is that you listen to them, doing things the way they suggest, instead of thinking for yourself. The trade-off is your brain for their brain. Keep in mind, the guru never set foot in your offi ce and doesn’t even know what product you make.

When pressed on this issue, most gurus admit that, of course, “best” doesn’t really mean “best.” To them, “Best Practice” is just a short way of saying “A practice that is good

enough for enough people enough of the time that it’s probably worth considering.” If that were really the case, then there are a number of alternatives; James Bach of Satisfi ce recently listed several on his blog, which you can fi nd at www.satisfi ce.com:

• “Here’s a practice I fi nd interesting”

• “Here’s what I would recommend for this situation”

• “Here’s my favorite practice for dealing with {x}”

• “Person {x} attributes Practice {Y} for his success. Maybe you’d like to learn about it”

Notice that all of those examples included conditionals, and that best practices are free of all conditionals. When someone consistently

Not all trade-offs are equal; the trick is to carefully weigh

the pros and cons before making a decision....

Have I got a deal for you?

WhatIsTesting.com30

uses the phrase “best practices” and never seems to use any conditionals, my radar goes up … then I go to protect my wallet.

Some gurus admit that they don’t know your situation. They explain when a practice might fi t and when it might not; they talk about the pros and cons of the practice. Some experts and consultants admit that all new practices involve a trade-off, and they are up-front about it.

At WhatIsTesting, we want to follow that example. Every issue of WhatIsTesting will contain some ideas, strategies, techniques, practices, and perhaps some philosophy. The ideas might fi t into your workplace, they might not, or perhaps you can use them later, but not now.

As I mentioned at the beginning of this essay, this issue will set the ground rules for everything that follows, so I want to start it out right. I consider myself a member of the context-driven school of testing, that has the following basic principles:

1) The value of any practice depends on its context.

2) There are good practices in context, but there are no best practices.

3) People, working together, are the most important part of any project’s context.

4) Projects unfold over time in ways that are often not predictable.

5) The product is a solution. If the problem isn’t solved, the product doesn’t work.

6) Good software testing is a challenging intellectual process.

7) Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products

- http://www.context-driven-testing.com/

The greek philosopher Socrates is credited with saying that the unexamined life is not worth living. We spend thousands of hours a year at work. Examining our work life and striving to constantly improve would seem a natural extension of that, and it is what this magazine is all about.

Of course, the choice is yours. Each issue will cost a little bit of your money, and a little bit more of your time. Would you like to trade?

About the Author:

Matt Heusser is a Member of the Technical Staff at Socialtext, where he is a lead in the QA group.

You can read Matt by email at [email protected] <mailto:[email protected]> or read his blog online at xndev.blogspot.com <http://xndev.blogspot.com>

At each stage of the game, we are giving something up...

in exchange for something we value more.


by Hans Schaefer, Cairman : Norwegian Software Testing Qualifi cations Board

Testing The Normal Way is Not Enough.Systematic testing of software or systems can be learned, just like any engineering discipline. There are tester knowledge certifi cation schemes like the ISTQB certifi cation, which is an internationally recognized, independent and non-profi t organization, and there are standards (ISTQB Glossary, BS 7925, IEEE 829, 1008, 1012). At least the books and most of the standards have been around for a long time, and many techniques are widely accepted. This means testing can actually be studied and then executed in some systematic way. This does not mean that the typical testing methods described in the

literature are widely practiced.

As test execution is generally one of the last activities in a project, it is very often run under severe budget and time pressure. This means that the testers have a hidden or open motivation to compromise on the thoroughness of their job. For example, as detecting defects always

delays both testing and the delivery or acceptance of the product, there is a hidden pressure not to fi nd any defects, or at least not serious ones. Testing is just “done”. This job is then not very motivating, investment in learning is scarce, people try to fi nd a different job, and testing does not improve over time.

What a

Tester Tester Should

Know, even

After Midnight

Learn How To Test

What a Tester Should Know, even After Midnight

WhatIsTesting.com34

Can we Do Testing in a Better Way?Glenford Myers, in 1979, defi ned testing differently: The aim of Testing is to fi nd defects. He used this defi nition because it motivates people. Having a negative focus, trying to break the product, leads to fi nding more and more serious defects than just checking whether the product “works”.

Most people do as they are told. If they are told to fi nd as many defects as possible, they will try

to do so. If they are told to get the job done (and explicitly or inherently passing the message that defects delay the progress), people will try not to fi nd defects, or they will overlook many.

Thus the fi rst problem is to clearly defi ne the purpose of testing, and make the purpose perfectly clear to the testers.

Second problem relates to change. Software products are growing more and more complex. The focus of the requirements is changing, for example emphasizing more security, interoperability and usability. This leads to changes in the requirements on the testing job. Thus, a tester should continuously try to learn more.

The next problem is the mindset of testers. As one of the purposes of testing is fi nding

problems and defects, a mindset that questions everything is likely to result in better testing.

Part of the tester’s task is to report incidents. This is not easy because it requires describing the problem in such a way that the other side accepts it as important enough to do something about it. It means not only collecting more information about the problem but also to think about how to “sell” the bug report to the responsible person.

There is defi nitely more to it. A tester should always be on the outlook for more challenges in fi eld of testing. This paper is only the beginning of what you may look for.

The Purpose of TestingThere are a lot of possible goals for testing. One main, but possibly boring, purpose is to measure the quality of a product. Testing is then considered just a usual measuring device. Just usual measuring is not much fun, but a necessary job, which must be done well. However there are questions a tester should ask, in order to measure optimally. The main question is which quality characteristics are most important to know, and how precisely the measurement must be done.

Another defi nition of testing is trying to create confi dence in the product. Confi dence, however, is best built by trying to destroy it (and not succeeding in doing so). It is like scientifi c work: Someone proposes a new theory, and all the educated specialists in the area try all they can

to show that it is wrong.

Defects clump together: they are social!

Defects often have a common cause!

Try to fi nd common causes, and then follow them!

Where you fi nd defects, dig deeper!

Test in a better way:- Clearly defi ne the purpose of testing.- Continuously try to learn more.- A mindset that questions everything is likely to result in better testing.- Report the bug with a view to “sell” it to responsible person.



After trying this for some time, unsuccessfully, the new theory is slowly adopted. This view supports Myers’ defi nition of software testing: Find defects! The approach is a pessimist approach. The pessimist believes «this probably does not work» and tries to show it. Every defect found is then a success.

People function by motivation. The defi nition of testing as actively searching for bugs is motivating, and people fi nd more bugs this way. It works in two ways: One is by designing more destructive or just more test cases. The other one is by having a closer look at the results, analyzing details a not-so-motivated tester would overlook. In the latter case this may mean to analyze results that are not directly visible at the screen, but are deep inside some

fi les, databases, buffers or at other places in the network.

Defects tend to cluster together. We should take a deeper look in areas where we fi nd defects. The trouble is that this could be a self fulfi lling prophecy. We are likely to fi nd more defects because of our focus in that are.

One more defi nition of testing is «measuring risk». This means that testing is a kind of risk mitigation, part of risk management. Testers should have an idea about product risk, as well as risk management.

A very basic method for this is looking at the usage profi le, and at possible damage. A tester should ask which kind of user would do what and how often. How will different users use the system? How will this vary over time? And it is very important to take into account wrong

input and misuse.

There are three kinds of tests: The good, the bad and the ugly tests: Good means everything is fi ne, all inputs are right. Bad means inputs are wrong. The ugly tests is where all wrong things happen simultaneously: Someone restarts the server while you do your reservation and at the same time sets the machine date wrong, …

Another way to look at risk is to asses the possible damage of the failure. This may be diffi cult to analyze but one can make a beginning by asking “what is the worst thing that can happen if this function, feature or request fails?”

Better tests forces developers to do better work, inform management about risks, and lead to lower cost of software creation. Testers bring bad News, but that is their job. Nobody loves speed checks on the motorway! But speed checks make our roads safer, and we all benefi t.

Continuous LearningContinuous learning is required in nearly any job. But for testers it is absolutely essential.

In most cases, testing is done somehow systematically, using some black box approach. In addition, test design may follow heuristics. Any black box approach may leave important areas un-covered. Any heuristic is incomplete, as it is dependent on personal experience (or on learnt experience from others. And white box testing does not uncover errors of omission. It all comes down to: If there is some aspect the tester doesn’t know, it is not tested. Hence the tester should know as much as possible. But how?

A tester should try to fi nd defects!

Defects may appear at places where you do not see them easily, i.e. not on screen output!

Testing is risk mitigation.

What determines risk?

What happens if some input is wrong?


WhatIsTesting.com36

A tester needs programming experience. There are lots of programming bugs, even after unit testing by programmers. The tester should have an idea of problems posed by a particular programming language. Most testers who have tested software written in C or C++ know that they should check for memory leaks or memory overwriting. Those who used early versions of MFC (Microsoft Foundation Classes) know that the program would crash for any date greater than a particular date “We need to check what that date was and with which version of MFC”

The tester needs design experience. Much design is about contracts and communication amongst various modules. If the tester has no idea about architectures and their inherent problems, she will have trouble planning integration tests.

The tester needs domain experience. System testing is about implementing the right solution, doing what the domain requires. Can you test a railway interlocking system? (Eh, what does interlocking mean, by the way...?). At least some domain experience is a must to test the system and communicate with different stakeholders.

The trouble is that this may not be enough. Testing for today’s stakeholders may defi nitely be not enough. There are totally new ways a system may be used or interfaced with other systems and a tester should try to anticipate at least part of this! A tester should always try to fi nd new ways to look at the object under test, new approaches, and new viewpoints.

And fi nally: testers should try to use the newest approaches and technology. You have to learn them. Read testing books, look for and learn tools, study journals, participate in discussion groups, special interest groups, discuss with

your colleagues, and go to testing conferences!

A Critical MindsetDon’t believe anything!

As a tester, don’t assume anything. It may be wrong! Designer, specifi ers, and programmers assume a lot. It may be diffi cult to ask because you may look stupid asking. Someone who could answer may be far away or not easily available. You wouldn’t even realize that there may be another interpretation. Sometimes people you ask also don’t know, or you get some sarcastic answers.

Using the pessimist view, you may as well assume that any assumption is probably wrong.

There are ways to overcome the trouble that you may seem stupid. Learn how to deal with people, learn how to interview, learn how to be self-confi dent. Ask someone else. Read, review, sleep over it, and try to fi nd different interpretations. You may need a lawyer’s mindset.

If you don’t get an answer, have it on your issues

list. But don’t just assume anything! Don’t take things for granted! And especially don’t believe that the system is probably right

DefectsNobody loves the messenger with bad News!As a tester, most of what you report is bad news. The bad News is the bugs, or issues to call them in a neutral way. Textbooks generally handle this area well. There is issue reporting, registration,

Learn more, about everything!

Programming, architecture, new domains, users, tools, anything!

Don’t assume! Ask!

If nobody else asks the right question, you might do so!

Think about new possibilities, unknown problems, and the stuff you learn.

Think «out of the box»!



handling, removal, retesting, regression testing. We know all this. But there is something extra to it, and that is not there in the books:

1 - An issue is only interesting for a tester if it is accepted as a defect and repaired.

2 - There are defects, which are the result of running many test cases in a row in some very special order.

The fi rst problem is one of salesmanship and discipline. As a tester, one has a sales job. Nobody is interested in spending any money on repairing defects. They will only be repaired if they are important enough. Thus, as a tester you have to report an issue in such a way that the developer understands that it must be repaired. The damage must look big, the probability of it occurring must look big, and the issue must be easy to repeat.

Thus the tester should not just write an issue on the issues list. The tester should think: Are there any other related scenarios, which would make this problem worse? Are there any more stakeholders interested in this? Is this really the whole problem or is there more to it? It may also mean to invent and run some more test cases. Cem Kaner has presented some excellent material on this cause which is available at his website at http://www.kaner.com/pdfs/BugAdvocacy.pdf

Finally, there are the human issues, about diplomacy, politeness etc. A tester should make sure not to hurt anyone personally when reporting an issue.

The second problem is worse: Sometimes we experience failures, and we cannot recreate them. These issues are called intermittent bugs. They are especially diffi cult if they introduce system crashes. Upon restarting the system, any corrupted data in the memory may be deleted, destroying the evidence. In many cases, intermittent bugs are the result of long-term corruption of some resource or memory. An example is memory leaks. Some function in the program does not return unused memory when fi nishing. But because there is a lot of available memory, this can go on for a long time, until the memory is depleted. Other resources may also be depleted. As an example, the Mars Explorer ceased working after 18 days due to too many fi les accumulated. In many real time embedded systems, the tasks are restarted at certain intervals, in order to cancel out possible corruption of resources.

The trouble is that ANY shared resource can be corrupted. It comes down to checking the not very easily visible outputs such as fi les, memory pointers, buffers, databases, messages to remote systems, registry and many other resources. It could even be the result of a race condition, which depends on the exact timing of some parallel tasks. And intermittent bugs normally require a whole sequence of test cases, not just one input and output.

However, if intermittent bugs occur, it is a good idea to be able to rerun the same sequence of test cases, maybe even with the same timing, and do more checking than before. James Bach has a good guide to investigating such problems available at http://blackbox.cs.fi t.edu/blog/james/archives/000197.html.

One fi nal problem: You may be wrong yourself. Humans err. Testers are humans. This means you overlook problems; you misunderstand outputs, and some of the problems you think you have found are actually not problems. Be self-critical: Sometimes it is your fault. Mistrust your memory. It is restricted. This means it is better to take notes, to log what you did, what the system did, what

For every issue (or bug), research more about it!

Make sure you report it as a risk, and as the whole risk!

Defect reporting is a sales job!

Be diplomatic when reporting issues!


WhatIsTesting.com38

is installed etc. You can trust notes more than your memory.

The Late Night Tester’s ToolboxHow should a tester work? What should a tester always keep in mind when working?

One main trouble is to test “everything”. This is far too much. It can never be achieved. But the tester should have some ideas about what is tested and what not, or what is tested more or less. That means test coverage.

There are three very basic concepts of coverage, and they can be applied to any notation for

example, a control fl ow diagram, a data fl ow diagram, a state transition diagram, a call graph, system architecture or a use case.

• Basic coverage is executing every box.

• The next level is testing every connection.

• This should be the minimum when testing. If there is more time, the next level is combining things, for example pairs of connections.

Next, a test should follow the usage profi le. This is diffi cult, especially in module and subsystem testing. But as a tester, one should at least try to get some idea about how the object under

test will be used. If nothing can be said, the test should be directed at any use, testing robustness. This means especially test cases for wrong inputs are interesting.

One technique is the basis of most black box testing: Equivalence partitioning. It helps to save

test effort, and it can be applied to derive other test techniques. As a tester, you should know it, but also be aware that it has caveats: Black box testing may leave out important aspects. You should also be aware that combination testing is interesting. Lee Copeland (Copeland 2004) has published a good introduction.

Another big problem is test environment that has not been validated. The test environment should be prepared and tested early. Waiting for the test environment to work can kill any testing

effort. Also remember that the defect may not be in the object under test, but in the test data or the output analysis. Be self-critical!

And fi nally, there is test automation. Running tests using automated tools helps regression testing. But test automation is more than that:

Analyze even intermittent problems!

Log everything you do in testing!

Log everything you see, and look at more remote details!

Make it possible to rerun your tests, with more logging and analysis tools switched on!

You may be wrong – don’t trust yourself 100%!

Take notes of what you do!

A tester must be able to state what coverage a test has achieved!

Follow the usage profi le if possible!

If not possible, test for robustness against wrong input.

Equivalence partitioning is a good basic technique!

Remember combination testing!



Tools may read specifi cations and automatically generate test cases. Tools may automatically create or restore test environments. Tools may be used to manage the testing effort and the test material. Remember that automaton can be applied to any part of the software testing life cycle and not just test execution.

Selected References • Bach 2005: James Bach. A blog note about

possible causes of intermittent bugs: http://blackbox.cs.fi t.edu/blog/james/

• Beizer 95: Boris Beizer, Black Box Testing, John Wiley, 1995

• Better Software Magazine, www.bettersoftware.com. www.stickyminds.com Very practical!

• BS7925: British Standard: www.testingstandards.co.uk/bs_7925-1.htm

• Copeland 2004: Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House, 2004.

• Crispin: Lisa Crispin, Tip House, Testing Extreme Programming, Addison-Wesley, 2002, also http://home.att.net/~lisa.crispin/XPTesterBOR.htm

• Gilb 2003: “Testers Rights: What Test should demand from others, and why?”., Keynote at EuroSTAR 2003

• GTB: German Testing Board: www.german-testing-board.info The German Testing Board developed an earlier version of the nowadays-existing ISTQB certifi cation.

• IEEE Standards: See www.ieee.org

• ISEB: Information Systems Examinations Board of British Computer Society. http://www.bcs.org/BCS/Products/Qualifi cations/ISEB/ has run a certifi cation scheme for software testers since 1999.

• ISTQB: www.istqb.org International Software Testing Qualifi cations Board. Develops and runs an international software tester certifi cation scheme.

• ISTQB Glossary: www.istqb.org/fi leadmin/media/glossary-current.pdf

• Kaner 99: C. Kaner, J. Falk, H. Q. Nguyen, “Testing Computer Software (3rd ed.), John Wiley, 1999.

• Kaner bugadvoc: A presentation about how a tester should report issues. http://www.kaner.com/pdfs/BugAdvocacy.pdf

• Myers 79: Glenford Myers: The Art of Software Testing, John Wiley, 1979.

• Schaefer 2004; Hans Schaefer, “What Software People should have known about Software Testing 10 years ago - What they defi nitely should know today. Why they still don’t know it and don’t use it”, EuroSTAR 2004

Test the test environment – well before test execution!

Check you test data!

Automate testing tasks!

Be aware that there is more to automation than using test robots!

Shipping, 5%Retail, 5%

Financial, 30%

Telecom, 15%Insurance, 15%

Publishing, 15%

Healthcare, 15%

Offering Various domain & Tools Expertise

State Of The Art SQA Labs

About Virtusa

Virtusa is a global provider of custom IT solutions and outsourcing services to Global 2000 enterprises and software vendors in

the financial services, insurance, media and communications, high technology and other industries. Our deep domain strength,

custom solutions and services, and a unique platforming methodology enable our clients to reduce costs, improve business

performance and accelerate revenue generation. We deliver these results by integrating a consultative approach with our global

delivery model.

© 2006 Virtusa Corporation. All rights reserved.

Enterprises know well that quality can make or break customer loyalty and brand reputation. However, the growing complexity,

constantly changing market needs, and time to market pressures make software quality an elusive challenge.

Virtusa QA lifecycle services allow you to take control of your QA processes, projects, people, and priorities, while strengthening

your QA capabilities, minimizing risk and reducing cost.

Leverage Virtusa’s eight years of experience in

testing and certifying hundreds of commercial

products and mission-critical solutions to rapidly

improve QA effcienceis. Reduce cost and time to

market by leveraging Virtusa’s global delivery

model, world-class QA professionals, and state-

of-the-art labs, tools and technologies.

QA Offerings

Mission-Critical Software Product QA Heritage

Take Control Of Your Quality

uality Assurance ServicesVirtusa

2000 West Park Drive, Westborough, MA 01581 Phone: 508 389 7300 Fax: 508 366 9901 www.virtusa.com Contact: [email protected]

QA

- QA Outsourcing Evaluation- Tools Evaluation- Quick Wins Workshops

- Process Improvement Framework- Health Check Assessment & Scorecard

- SQA Outsourcing and Management- Test Team Establishment/Augmentation- ‘Quick Win’ Test Project Consultancy

- Functional Automated Testing Manual Testing- Performance- Certification- Internationalization- Usability

Assessment

QA ProcessOptimization

Testing &Test Automation

QAOutsourcing

QA Assessment

QA Process optimization

Testing & Test Automation

QA Outsourcing


by Erik van Veenendaal, Improve Quality Services BV : The Netherlands

A new testing magazine that has chosen web testing as its central theme for the fi rst issue! I assume that this also means attention is provided to one of

the most critical success factors for websites: usability and thus usability testing. In fact usability has been identifi ed in various surveys as one of the main reasons for project success in any IT project! For a number of years I’ve been lecturing on this topic at test conferences. At EuroSTAR, I have received the best tutorial award for Discount Usability Testing and our company, Improve Quality Services, regularly runs courses on Usability Testing. Yet, I don’t have the impression that all this pioneering work has had much impact on the testing community. During the last EuroSTAR conference, an usability expert from South Africa stated that “usability (testing) will only start to be performed and receive the right level of attention, when testing for functionality is under control”. An interesting thought, which as far as I’m concerned, is very much true. Perhaps we were just not ready for it, but are we ready now…?

It remains strange that when no less than 40% of the

software code that is being developed world-wide is directly or indirectly related to the user-interface, we still only dedicate a minimum percentage of our test effort to usability testing. Yet, there are many so-called best practices available. Last week I was involved in a usability test that very clearly showed that thorough attention for usability throughout the development process (requirements, standards, expert reviews, usability testing) can deliver great results. The users were enthusiastic regarding the new system that supported their tasks in an effi cient, effective, and user-friendly manner. This is also possible in real-life IT-projects!

As far as I’m concerned every (senior) tester that operates at system level should, in addition to the conventional functional test design techniques (boundary value analysis, equivalence partitioning, decision tables etc.), also have the

level of knowledge and skills that allows them to apply a number of relatively easy to use usability test techniques. The fi rst things that come to my mind in this context are the Heuristic Evaluation technique developed by Jacob Nielsen (www.useit.

Finally

Usability

Testing?

will it fi nally happen?

Learn How To Test

Finally Usability Testing? Will it fi nally happen?

WhatIsTesting.com42

com) and the SUMI questionnaire method (www.improveqs.nl). Both techniques are relatively easy to learn, do not require more than one or two days of testing effort and therefore have a great cost/benefi t ratio. Recently, the British Computer Society (BCS) Special Interest Group in Software Testing developed a testing standard in the area of usability testing that is certainly worthwhile to get acquainted with (www.testingstandards.co.uk). Suffi cient possibilities exist, therefore, to make a good, low-cost but thorough start with usability testing; preferably in co-operation with developers. Prevention is always better than cure.

During a recent conference, a number of enthusiastic former participants of my usability courses told me they were now using the techniques mentioned above in their test projects with good results. Usability testing, will it fi nally happen?

For queries or comments you can contact Erik van Veenendaal ([email protected])

A paper on discount usability testing can be downloaded from www.improveqs.nl

Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922

[email protected]; [email protected]

Canarys is glad to inform that we are Partners to Radview Software, providing Sales & Support Services to customers in India. Please find a brief on the Black Box Testing Solutions offered by Radview:

Only RadView provides software-testing tools that feature JavaScript test-agenda creation for performance testing, load testing, and functional testing of website applications. Throughout the web application development lifecycle, the seamless integration between RadView's performance testing / load testing products (WebLOAD and WebRM) and functional testing product (WebFT)provides you with the necessary tools to make certain your web application is scalable and robust.

Award-winning WebLOAD can help companies deploy high performing e-business applications by modeling and anticipating the real-life demands on their applications. By accurately simulating Internet user behavior and predicting capacity requirements, WebLOAD reports bottlenecks, constraints, and weak links within an application – before they cost you downtime, lost sales, or more importantly, lost customers.

WebFT is the industry's first Web-centric testing solution that efficiently verifies the functional accuracy of Web applications and ensures applications perform as expected.

Speed implementation time — use of open standards scripting language reduces the amount of staff training requiredImprove quality — resulting from higher quality Web applications Reduce test script development costs — leverage single transferable scripts for fast, easy reuse of test scenarios between development teams Speed Web application deployment — streamline the testing process through the use of single transferable scripts.

For more information please contact:

Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011 Ph: +91-80-26539915, +91-80-26642922

Some Things I’ve Learned in Software Testing

WhatIsTesting.com44

Several years ago, when I started out as a software tester, my goals for learning software testing were clear. I needed to learn how a software company

operated, learn about the software I was assigned to test, and learn how to test software. After a while, I enjoyed fi nding and reporting diffi cult-to-reproduce bugs; I felt pleased that we were fi nding and fi xing issues that could negatively affect customers.

How did I go from a nervous student to becoming a confi dent tester? At fi rst, I read everything I could about testing, the product we were working on, and the systems we used. When I couldn’t fi nd anything to read, I asked coworkers to teach me things. I also learned a lot through mentoring from more senior developers and testers.

In the end, I learned the most by doing. (I still do.) When I fi rst started out in testing, my view of what a bug looked like was quite narrow. Like many users, I felt

that any error messages that appeared were due to mistakes I made using the software. Then, a senior tester encouraged me to follow my instincts: he told me that if something bothered me, I should log it as a potential problem instead of letting it go by. As I tested, I learned that when I had a nagging feeling that something was wrong, there usually was a problem in the software. Sometimes it was a usability issue, or an awkward spot in the application that needed more investigation, but whatever the issue, it was almost always worth taking the time to investigate further. Even now, it is tempting not to listen to that voice inside that tells me something is wrong with

the software, but every time I don’t pay attention, I miss at least one bug that I

regret later. Occasionally, my instinct is wrong, but more often than not, that

sense of feeling uncomfortable with the software helps me uncover

important information.

I also learn a lot from my

Some Things Some Things

I’ve Learned in I’ve Learned in

Software Software

TestingTesting


by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada

mistakes, especially when I try something, fail and try something else. Do you learn from your mistakes and from trying new techniques? Who can you talk to in the organization that can help you learn more about testing?

There is More Than One Way to Test SoftwareIt didn’t take long for my experience to show me that much of popular software testing information is really folklore. It was interesting to read, but I soon discovered that when I tried to apply popular testing wisdom (frequently called “Quality Assurance”) on real projects, my efforts failed. One practice involved writing all the test cases fi rst (pre-scripting test cases) based on the project documentation, and then testing using these procedural test scripts later. Trying this practice caused me a lot of problems. I found that testers would run out of time to actually test because they were so busy writing about what they were going to test. As things changed over the life of the project, we got bogged down maintaining documentation. And, when we started testing, these pre-scripted test cases really narrowed our focus, often making us miss entire classes of tests.

In the software start-up I was working for, it was important for each member of the team to provide value to the goals of the business. A development manager then complained to me that we spent so much time developing and maintaining test cases that we didn’t do nearly enough actual testing. Once I took that to heart, the business was pleased with the results I provided by using exploratory testing to focus on what was important to the business and to the customer. Previously, I had focused on writing lots of documentation (test plans and test cases) before testing. At fi rst it felt a bit wrong, like we weren’t following “the rules,” but the development team and our customers were happier with our results. That carried more weight with me.

I also read statements like: “Quality Assurance is much more than software testing.” In this line of thinking, it is common to suggest that QA should be responsible for the development process, that quality is only as good as the process it follows. However, in my testing experience working for small companies whose survival depended on satisfying paying customers, the process took a back seat to the product. We strove for product excellence over process excellence. If a process helped us deliver a product that met business goals, we followed it. If a process got in the way of those business goals, we scrapped it.

Everything we did needed to be aligned with the goals of the business if we were to survive.

This key lesson shaped my thinking as a software tester.

My desire to learn more about software testing grew because I learned the most and provided the best business results

from actually using and testing software. I wanted to fi nd more techniques

compatible with what I was learning about software testing. So I reread the fi rst book I’d ever read on software testing: Testing Computer Software by Cem Kaner, Jack Falk and Hung Nguyen. Again it resonated with me, and I began to read everything I could fi nd that was written by Cem Kaner. Through him, I discovered James Bach and later Brian Marick. That was when I found my niche. These were people who thought about testing the way I did. My own experiences and ideas were echoed in their work. Their writing was true to the point and backed by experience, but most importantly, when I tried what they recommended, it worked. These three testing thinkers, along with Bret Pettichord, went on to found the Context-Driven Testing School1.

The Context-Driven Testing School has identifi ed different schools of thought in software testing. Bret Pettichord has outlined four of them in his Four Schools of Software Testing2 presentation:

• Analytical School

• Factory School


WhatIsTesting.com46

• Quality School

• Context-Driven School

The Analytical School draws heavily on mathematics, uses a lot of tools and techniques for code coverage and uses metrics to try to measure conformance to a specifi cation. This school proposes that software reliability can be calculated. Some testers in this school use complex mathematical models for quality prediction.

The Factory School generally expresses that that testing needs to be predictable, repeatable and software is validated from written requirements. This school of thought is often interpreted so that testing becomes a clerical task. (Note: many Agile and Lean testing ideas are variations of this school of thought. This shouldn’t be surprising. Many of the ideas in this movement come straight from manufacturing where automation, repeatable processes and eliminating waste are important factors.)

The Quality School encourages conformance to a particular process. Often, the tester’s job is to police the process, and make sure that everyone is following it. The belief is that if we follow the process properly, our product quality goals will succeed as well.

The Context-Driven School believes testing is a skilled activity, and the usefulness of a practice is dependant on the context it is used in. Not all companies developing software are the same, and it is irresponsible to try to standardize processes for all software development. Different companies have very different needs and those needs are far too diverse to standardize to one true way of testing. Testers in this school use whatever testing tools or techniques are the most effective to help them reach their testing goals.

Since Four Schools of Software Testing was published, Cem Kaner has identifi ed Test-Driven Development as another school of

testing thought3. Bret Pettichord identifi es this as the “Agile” school of thought. There may be other schools as well. What is important as a tester is to understand there are different ways of thinking about and communicating testing ideas, and recognize which ones you identify with. That will help you get a focus on what to learn, and will help you understand where others are coming from.

Some people fi nd the identifi cation of schools of thought in testing objectionable, and this work by the contextualists is somewhat

controversial. When I fi rst discovered it I was reminded of different schools of

thought in other disciplines that I studied in university. For example, in philosophy classes, when

I learned of another school of thought in logic, I wanted to learn as

much as I could to see if it altered my thinking. If I found ideas in

a particular school of thought that resonated with me, I added them to my

own repertoire. The schools of thought in testing idea shouldn’t be something that bothers us as testers; instead, it can be used as a model for learning. If I get stuck studying

the same old techniques and processes, I can use this model to identify other schools of thought that I am less familiar with, and begin researching them.

As a tester, what works for you, in a given situation? Have some techniques worked well on some projects, but not so well on others? Is there a particular school of thought that you haven’t studied before that might help you on what you are testing right now?

Software Testing is MultidisciplinaryI once worked on an article on tester roles with my colleague Michael Kelly, who is also a software tester. It became clear to us how many different disciplines software testers draw from. Different testers described the following disciplines as infl uencing their work:



general systems thinking, history, software development, construction, law, psychology, family therapy, adult education theory, writing, writing and performing music, experimental science, among others. Older, more established disciplines have often already dealt the same kinds of problems we are trying to deal with in software development. Applying successful principles from other disciplines can be a rewarding source of learning.

I often explain software testing ideas by using an intersection of ideas from different fi elds. Three that I think of frequently are philosophy, business and technology. Testing, like philosophy, involves asking questions. Not only do I question the assumptions made in the product I’m testing, in our development efforts, and in the process we adhere to, but I constantly challenge my own assumptions about testing. Studying philosophy and practicing critical thinking skills especially help me when I’m generating test ideas.

Business considerations help me focus the work I’m doing. Am I adding value on a team? Is what I’m doing helping or hindering the goals of the organization? Is my testing focusing on what is important to the goals of the company, or am I blindly following a process? Understanding how businesses work, how they market themselves, and how they stay functioning is important to testing.

Technology and testing are related because we use tools, and interact with designers, developers and users in order to test. We rely on computer technology not only in software development, but in most businesses situations. As testers, we must have a certain level of skill to be able to use technology. Since technology moves so quickly, it is diffi cult to keep up and it seems like there are infi nite technological areas to explore as testers. We might specialize in testing a particular technology, or we may depend on certain tools for our testing.

Philosophy, business and technology are areas

that infl uence me, and provide enormous possibilities for learning. My background in music and sports, and my education also infl uences the way I approach learning and testing. What are your infl uences in life, and how much do you draw on these other infl uences when testing software? What other disciplines remind you of your work in software testing?

Testers Can Learn From Other RolesI’ve worked in roles other than software tester, and each of these taught me something important about software testing.

Working as a programmer, I learned how to look at applications differently. Understanding the application from a source code perspective helped me develop a catalogue of potential areas in a particular programming language that are more susceptible to bugs. It also provides a better understanding of where to fi nd useful information for reporting a bug. For example, web applications usually have a server log that may have debugging information useful

for reporting problems. As a tester wearing a developer hat, I was the recipient of bug reports, rather than a generator. It helped me understand how a poor bug report

can frustrate developers.

In a technical writing role, I learned about design and how design (UI or architectural) can affect the quality of a product. I noticed a correlation between complicated designs and the amount of bugs found. I learned that if I had trouble writing user documentation for a particular feature, it would likely be a source of bugs. Writing down requirements and design ideas from the point of view of a user exposed complexity.

Working in sales taught me about the big picture, about how a software tester can serve the business goals of an organization. A company does not necessarily survive because they are such-and-such CMM level, or follow the 12 practices of Extreme Programming, or because


WhatIsTesting.com48

they use .NET or Java, or other technical or procedural issues we fi nd so important in development. When I worked in sales, the harsh reality was that a company survives by fulfi lling an important equation:

Revenues – Expenses = Profi t

This equation must be greater than zero in order for the company to survive.

I learned customers cared that the product helped them solve a particular problem. They didn’t care what technology was used to solve it. They cared that the product was easy-to-use, that it helped them in their work, and that it was affordable. It was interesting to see that they would put up with application failures, but that they were not as tolerant of usability issues. When you deal with customers, you get a different view of the product. Testers serve customers, and it is in our best interests to be sure that they are satisfi ed and impressed with our product. I learned that it is much more important to be testing what is important to the customer rather than trying to fulfi ll some generally-accepted testing practice. It is absolutely paramount to do software testing from a user’s perspective. Testers need to understand the company, what its goals are, and to fi gure out how testing work maps to those goals.

Understanding how an application works from a technical perspective, and being able to describe that in a document helps me test more effectively. It helps me ask questions that people I work with fi nd useful. Working directly with potential and existing customers has helped me focus on their needs in my testing. Understanding the goals of the business helps me prioritize my mission in testing on a particular task.

What other jobs have you had other than in software development and testing? What have you learned on those jobs that you could apply to testing? Are there specifi c skills you developed in other jobs that give you an advantage as a tester?

Practicing Software Testing is ImportantWe can also learn by practicing software testing skills. For example, I regularly practice my testing skills, and ask my testing friends to help me develop my skills. One of these friends is James Bach, who taught me how to be more consistent with my testing approach. He found that when he gave me a number of practice testing problems, my test approach was inconsistent. James helped me develop my testing ideas into a consistent system when solving different kinds of testing problems. Now when I am faced with a testing problem, I have a strategy and a game-plan instead of an ad-hoc approach. Practicing testing with someone else helped identify an area of testing weakness for me to address and improve.

I’ve also worked with programmers and asked for feedback on how to provide better information when logging bugs. Early in my career when I was fi rst testing web applications, I wasn’t investigating beyond the error I saw on the screen. One of my programmer colleagues taught me how to fi nd more information in log fi les. I practiced developing my web testing investigation skills by:

• Finding relevant server logs on different web application architectures

• Reading error messages in the server logs

• Changing the level of logging to get more information on the server

• Saving relevant information and talking about possible causes with administrators and developers

Because I practiced, when I next tested the web application, gathering relevant information from server logs to use for a bug report was second nature. It took much less time to do after I had practiced those skills.

This is just one very simple example, but it helped my testing effectiveness. There are many skills we can develop, and many techniques we can practice. What specifi c skills you rely



on as a tester? Can you name them? Can you identify technical skills, investigation and problem-solving skills and softer skills related to communication, and working with others?

ConclusionAt the end of each section of this article, I have posed questions for you, the reader. If you would like to learn more about software testing, I urge you to answer these questions as best you can. Take out a pencil and paper, and write down your answers to the questions. Read them over, and see if there are any new areas for study for you to pursue in testing. You should have some new ideas about software testing you’d like to learn more about. Do you want to learn more about testing techniques? How about different schools of thought or defi nitions on testing? Are you interested in developing your technical or programming skills? Would you like to learn to use a test automation tool? How about learning more about the end users of your software? With software testing, there are many avenues for learning, and the more you learn, the more you realize how much more there is to learn.

I constantly learn more about software testing and I hope I have spurred your thinking on how you can learn more about software testing too. Even though I am no longer a beginner I still do the following:

• Read about software testing and other disciplines

• Attend and contribute to workshops and conferences on software testing

• Identify skills and techniques I am using when testing, and explain them to colleagues

• Participate in mailing lists, forums and other communities of practice related to software testing

• Work with others skilled practitioners who constantly challenge me to do better

• Practice, practice, practice

Learning software testing is a rewarding, career-long journey. We all learn differently, and I encourage you to try different ideas and see what works for you. Don’t be afraid to fail when you try something new; we often learn more from our failures than successes. Quality guru W. Edwards Deming said: “Drive out fear.” Don’t be afraid to try something new, if you have an open mind, you are guaranteed to learn something from it.

About the AuthorJonathan Kohl is the founder and principal software testing consultant with Kohl Concepts Inc., based in Calgary, Alberta, Canada. A noted testing thinker, Jonathan is recognized as a leader in the exploratory testing community. He is a popular author and speaker who believes that testing is a challenging intellectual craft. Jonathan’s blog on software development and testing issues is one of the most well-read testing blogs in the industry. Jonathan is a regular contributor to Better Software magazine, as an author and technical editor. Check out other writings at www.kohl.ca.

Notes 1 Kaner, Bach, Pettichord, Marick. The

Seven Basic Principles of the Context-Driven School http://www.context-driven-testing.com/

2 Pettichord, Bret.(2003-2007.) Four Schools of Software Testing http://www.io.com/~wazmo/papers/four_schools.pdf

3 Kohl, Jonathan. (2005) TDD - A Fifth School of Software Testing http://www.kohl.ca/blog/archives/000087.html

Sierra Atlantic Testing Competency Center

Our Testing Competency With over 12 years of experience, Sierra Atlantic is an established offshoring services company using CMM Level 5, ISO 9001:2002, and BS7799 certified process. Using our testing competency center SATIN (Sierra Atlantic Testing Intelligence), we create speedy, low cost and high-quality software. This in turn is delivered through our well-defined engagement models and process methodologies and incorporates the industry’s best practices.

State-of-the-Art Facilities Our state-of-the-art Interoperability Lab provides compatibility / interoperability testing services to software product companies. The scope of this service includes:

Hardware / OS platforms Enterprise Applications Middleware platforms Application Servers Web Servers Databases Browsers

With over 250 person years of experience in software testing services, our resources have in-depth understanding of various testing methodologies and technologies. The following graph illustrates this:

The following instances help demonstrate our competency.

A leading Enterprise Application product vendor Sierra Atlantic’s Interoperability Lab enabled the testing of a leading Enterprise Application vendor’s product on platforms such as Sun Solaris, HP-UX and IBM-AIX. This testing allowed the customer to remotely log in and execute tests using its staff without having to build its own test environments, an expensive and time-consuming process.

A leading portal product vendor This client of ours required its product to be tested and certified across a variety of web servers and platforms. The product also required to be tested for compliance under Rule 508. Sierra Atlantic’s Interoperability Lab created multiple functional and compliance test cases and an automation strategy based on Mercury test automation tools. As a result, the client was able to cut in-house costs, and the testing rendered its product totally compatible and fully interoperable with other leading enterprise applications and infrastructure products.

About Sierra AtlanticFounded in 1993, Sierra Atlantic is a leader in offshoring enterprise applications, helping our customers optimize their investments in Oracle, PeopleSoft, SAP and Siebel. We offer complete lifecycle application management solutions -- development, implementation, integration, upgrade and support -- using our NShore™ methodology. We integrate these point solutions into Application Networks® that enable seamless business processes within and outside the enterprise.

8%

28%

17%

23%

10%

7%7% Only manual testing

WinRunnerTest DirectorQTPLoadRunnerSilk TestRational Robot


by Mrityunjay :

I come from a background in product development and have been in QA for about a year or so, that too in a senior management capacity and not necessarily an engineer.

It saddens me when I interview some good QA engineers and they seem almost apologetic about the fact that they are into testing, and more so when they are into manual testing and not automation. This probably stems from the fact that writing code is considered as the best (and smartest) aspect of software development and anyone not doing that is considered less smart.

I started thinking about whether this is really the case, that writing code is the smartest part of the software development lifecycle and what I came out was the reverse; that writing code is probably the easier part of the development, it is testing and Quality Assurance (QA) which is tougher. This is based on my discussions with the QA engineers and leads, looking at the

product functionality and issues fi rst hand, and delivering a complex product to the market. I am very convinced that in a real world, it is easier to don a developer hat than a QA hat. In this article, I attempt to show in a very simplistic way what I mean by this statement.

Of course I say real world, because in an ideal world, a Dev will be much more like a QA and QA will be much more like a Dev, and they have joint ownership of quality of the product. The company I work in doesn’t claim to be the ideal company, but I fi nd it much closer to it than many other companies I have seen.

Let me start with an example. We are given a functionality to implement: List all prime numbers between two given numbers X and Y. Now here is what a developer has to deal with:

● Find/implement an algorithm to check if a given number is a prime number.

● Implement a logic which cycles through

QAQAis More is More

Challenging Challenging

than than

DevelopmentDevelopment

QA is More Challenging than Development

WhatIsTesting.com52

every number between X and Y, and apply the algorithm in #1 above.

● Unit test this with some combinations of X and Y and make sure output makes sense.

A QA engineer who relies on black box testing and hasn’t looked at the implementation will know that there are infi nite inputs to this implementation and so he/she will try to pick some intelligent inputs which are more likely to produce issues (think equivalent partitioning, boundary values, etc). However, this is decidedly

risky since some of the options are being left out and hence some issues may still remain.

A QA engineer who is a white box tester may look at the code and decide there are 232 options only because the developer used a long variable. This is nowhere better than infi nity. Some choices have to be exercised, probably based on decision points in the code. Again a risky affair, as QA is likely to miss many, including non-function, parts, like dependency on CPU and memory, impact of running on say

F

I

G

U

R

E

-

1

Test Entry

Test Entry

Test Entry

Test Entry

Test Entry

Test Entry

Test Entry

Test Entry

Design

Design

Design

Design

Functional Requirement

Product Requirement

Functional Requirement


by Mrityunjay :

Japanese OS, etc.

Note some of the differences in the roles in a developer and QA here:

● A developer decides on an algorithm and unambiguously sticks to it, implements and delivers to QA. For a QA, life is full of ambiguities and risk-taking. A week-at-heart QA is not a good QA.

● A QA has to second-guess Dev to fi gure out what are the likely error scenarios if he/she wants to avoid running an infi nite-input problem. This is analogous to 20-questions game, where you think of a word and someone has to guess it by asking only 20 questions. This is tough.

● Think of the implication if a developer or a QA goofs up. If a Dev goofs up, it is supposed to be caught by QA, causing bugs, design changes, etc, a costly affair indeed. However, if a QA does so, customers hit those defects, and this is a much higher cost to the company, in terms of money, credibility and market share. Which role is easier?

In a complex product, locating a bug is like searching for a needle in the haysack; you can apply heuristics, but it still is a hard and challenging work, scary too, when you know that a missed needle can cause immense customer frustration, and management backlash.

Let’s look at it from another perspective. The waterfall model of product development is a tree structure. A product requirement spawns many functional requirements, each functional requirement spawns many design and algorithm pieces, which in turn spawn many test cases (see fi g 1).

Thus, even in such a simplifi ed scenario (where next phase only doubles the entities), a product requirement generates 8 test entities for verifi cation. No wonder then that the QA has to play the game of probabilities: trying to fi nd maximum number of high impact defects

in the given time and money through various heuristics and techniques, none of which are scientifi cally proven to discover all the defects. This is decidedly tougher job than development, since the choices and the implications of choices are much better known and understood when a developer chooses a specifi c algorithm and its implementation. Given the fact that a functional requirement is broadly defi ned, it is easier for a developer to meet those requirements and verify they are met, than for a QA to verify developers’ output, since they are very precisely defi ned (algorithms and code) and hence so many more ways of proving failure exist.

To summarize my arguments above, there are two basic points I have made:

● It is easier to drop a needle in a haysack (or bug in a product) than to fi nd it (discover defect before release). The latter requires you to be much smarter, customer-focused, with good problem-solving abilities.

● QA problem is an undefi ned problem and hence solving it takes very different skills

and lots of smartness in addition to hard work.

However, don’t get me wrong: these arguments take the example of a typical developer role and a typical QA role and not of individuals who play them. As I have mentioned before, a good developer indeed dons the QA hat many times, verifi es the algorithms he/she has produced and only then allows it to be given to QA. So do not

use these arguments against any individual; I have been a developer, and I would not love to hear these arguments because I thought I donned QA hats many times to produce good quality code. Same goes for many of my developer friends out there.

Now that we know doing QA takes a lot of smartness and skill, what does it take to be a great QA? Well, the trick is to think like a developer sometimes, and that will be the topic for a forthcoming issue.

Is scripted testing bad?

WhatIsTesting.com54

When I started my testing career over a decade ago, my manager gave me two directives:

● Raise thirty bugs every day

● 80% of your bugs should be coming from test cases that you would have written prior to the test execution

While I was able to accomplish the task of raising thirty bugs every day, I have not been able to achieve the second target even after twelve years of trying really hard. To fi gure out what was going wrong I analyzed the bugs, created checklists, got my test cases reviewed, learnt about various test design techniques and used them.

Sure, I was able to improve the percentage of bugs being found but I never could achieve a fi gure anywhere close to 80%. This made me question the assumption that most of the bugs can be caught by pre-written test cases. Many of

the bugs I caught were the result of conditions that occur in the real world but are impossible to think about while designing test cases. And there were many bugs that I did not fi nd rather the bugs found me. With these fi ndings my respect for writing test cases took a beating.

Exploratory testing had begun getting talked about when I had formed my own philosophy and technique of mixing test cases and ad-hoc testing and it struck a chord immediately. Exploratory testing described what I was doing all my life. Now I had a name for it.

However, there was another trend that I noticed - scripted testing, which I had been doing all my life, started getting painted as a bad practice. To set the context right, let me describe what is generally meant by scripted and exploratory testing.

“Exploratory testing is simultaneous learning, test design, and test execution”www.satisfi ce.com/articles/et-article.pdf

0

5

10

15

20

25

Is scripted testing

bad?


by Vipul Kocher, CEO, PureTesting, NOIDA, India

“Scripted testing means that learning and test design happens prior to test execution.” http://en.wikipedia.org/wiki/Software_testing

Before you read on the rest of the article, please pause for a moment and answer honestly if you consider scripted testing a bad practice.

If you are careful about the contexts you are likely to have answered “It depends… scripted testing might be useful in some situations” and that is the truth. Unfortunately, contexts in which scripted testing is useful do not often get elaborated resulting in a message “scripted testing is bad.”

Today scripted testing is taking a beating from exploratory testing and to take a stance against it is seen as a sign of enlightenment. The stance may vary from “scripted testing is bad” to “amount of scripted testing to be done depends on context” but in all cases the underlying message that gets conveyed is “scripted testing is bad.” It is unfortunate that there are very few advocates of scripted testing who are actively writing in favor of scripted testing or explaining the contexts in which scripted testing is good.

Is scripted testing really bad?Before we can answer the question “is scripted testing bad” we fi rst need to agree on the defi nitions of scripted testing and also understand the contexts in which scripted testing is good or bad.

Scripted and Exploratory Testing: defi nition is importantWithout going into defi nitions of scripted and exploratory testing the only questions I would like to ask are:

● Who has created the defi nitions of scripted testing?

● Who has defi ned exploratory testing?

If the defi nition of scripted testing comes from the exploratory camp or vice-versa, the defi nition would naturally portray one as more negative than the other. The defi nitions put scripted

testing at one extreme and exploratory testing at the other extreme. Then, there is this idea of scripted-exploratory testing continuum which means that scripted and exploratory testing can be mixed in any percentage

Unfortunately there exists no defi nition for this type of testing. But isn’t this nameless, defi nition-less continuum what most testers, at least the good ones, have been using? The process being ‘create scripted tests, use exploratory methods to create more tests, run them, learn from them, create more tests and run purely exploratory tests...’

To me it would be more natural if somebody provided a defi nition for this mix of “scripted” and “exploratory” testing. Let me call this type of testing as Natural testing. With this defi nition, I can call myself a “natural tester” and not necessarily a “scripted” or “exploratory” tester because in real life I am neither. This is the position most of the testers I interact with, take. They are neither pure exploratory nor pure scripted testers. We start our testing whenever our organizational goals permit us to which can be at the requirements stage or late in the cycle when development is about to be completed. We write test cases which help us understand the system and in the process of writing test cases we uncover bugs, raise uncomfortable questions and help improve the product. When we execute tests, we use our pre-written tests, come up with more test and scenarios to explore the product; we document some of these discovered tests and discard the rest. We mix testing using test cases (scripted testing) with guided ad-hoc (exploratory) testing. This is “natural testing” to

Can there be scripted-exploratory testing

continuum: meaning thereby, scripted and exploratory

testing can be mixed in any percentage ?

Is scripted testing bad?

WhatIsTesting.com56

us and we fall in neither the exploratory nor the scripted camp.

Painting scripted testing as a bad practiceSome of the arguments against scripted testing include:

● Scripted testing does not promote thinking about the problem in different ways. Scripted test cases mechanize the act of testing, taking intelligence away from testing.

Statements like above make people believe that scripted testing promotes stilted thinking. If you defi ne scripted testing as “following the pre-written test cases VERBATIM” then the statement would make sense. If this is how scripted testing was to be defi ned, I would not like to be found siding with scripted testing. Of course, there is no need to defi ne scripted testing in this manner because that is not how most testers test. Unless there is a client demand, most testers do not follow the same test script again and again. They use test script only as a guide and not the gospel that has to be followed verbatim. Testers vary their data, their actions, and the sequence of actions.

● Scripted testing wastes time. Time is wasted in writing test cases whereas it could be spent testing. Also requirements change often, resulting in time wastage when test cases have to be reworked.

The difference between scripted and exploratory testing is said to be that in exploratory testing test design, execution and learning happen at the same time whereas scripted testing uses

pre-designed test cases.

There are numerous advantages of “scripted” testing over “exploratory” testing, some of them being:

• The act of writing test cases promotes logical and more complete thinking.

• Written test cases can be reviewed. This is not a small advantage. The review is likely to result in better test cases and consequently better testing.

• When test cases are written along with the requirements being written, many requirement bugs can be found and removed even before any code is written.

• Test cases can be shared with developers resulting in removal of bugs even before they can be “created”.

• Written test cases can be used as an aid for training new testers.

• Even if requirements change, the impact of requirement change as related to testing can be addressed in a better way with written test cases.

In my experience of testing, many complex products including some which are used by millions of users, review of written test cases has resulted in better understanding of testing and consequently better execution of testing by all testers involved in the review and not only the person who reviewed the test cases.

Of course, there are situations where one does not have the time to write test cases. But enough people write about these contexts. My purpose to write this article was to defend what most people today would not defend. I do not favor exploratory testing over scripted testing or vice-versa. I favor using whichever style gives the most benefi t in a given situation, rather than clinging to one end or the other of the continuum.

In exploratory testing test design, execution and

learning happen at the same time whereas scripted testing uses pre-designed test cases.

Putting the “Test” back in Test Plan

WhatIsTesting.com58

Until fairly recently, many people who called themselves Software Testers came by their jobs by accident rather than by design. This placed many

of us in the diffi cult position of having to learn how to do our job while we were trying to do that job. One of my biggest challenges was trying to fi gure out what goes into a Test Plan.

In the fi rst company that I worked for, the answer was straightforward enough. A Test Plan was simply a commonly-grouped set of test cases that was executed by a Tester. For example, there was a Print Test Plan, a GUI Test Plan, a Math Test Plan, and so on. No problem.

Doubt in that defi nition began to creep into my mind when I started to network with other Testers in town and found out that Test Plans meant different things to different people. When I got my fi rst Team Lead role in the next company I worked for, I discovered that the Test Plan I was being asked to produce was something completely different. It was supposed to be a planning document of some kind.

What to do when Confusion sets inI suddenly found myself in the uncomfortable position of not really knowing something that I thought I already understood. What do I do now? Where do I start?

I began by asking people I worked with. I networked a bit and asked other testers for advice, but most of the time no one would say much because they thought it was confi dential to share information like that. I even went to a day-long workshop on “Managing Systems Testing Projects”. While I found the workshop to be very insightful, I didn’t believe everything the instructor had said because a lot of it went against what I had learned so far on my own. For example, a Test Plan was not supposed to have any test cases in it. This just increased my doubt and anxiety.

Back at my desk, faced with an empty Microsoft Word document in front of me, I searched the Internet for Test Plan templates and examples. I needed more opinions and references than

Putting the Putting the

“Test” “Test” back in back in

Test PlanTest Plan


by Paul Carvalho

just my word against my workshop notes. It was at that time that I stumbled upon a few mailing lists focussed on Software Testing and Quality Assurance. People on those lists were very friendly and helpful and in no time I had several examples to help me develop my fi rst test plans.

The biggest hurdle I had to overcome was acknowledging that my prior experience and knowledge were based upon someone else’s incomplete knowledge. Until that moment, I thought that I knew what a test plan was. Then how was it that the defi nition changed when I changed companies? I had to unlearn the ways of the Dark Side.

Test Plan 1.0Many of the examples and templates that I had were very similar to each other with some minor variations between them. In general, I found that a Test Plan contained elements like the following:

● Overview and Objectives

● Testing Scope and Strategy (Features To Be Tested, Not Tested, etc.)

● Approach, Entry and Exit Criteria, etc.

● Responsibilities and Staff

● Schedule and Deliverables (Test Reports, etc.)

● Resource Requirements (Test Environment, Tools, etc.)

● Risks and Contingencies

I became quite profi cient at creating these Test Plan documents and customising them for the needs for each project. I even became familiar with some industry standards for Test Documentation1.

Then I started to notice an interesting pattern emerge with the more projects I worked on. For one thing, no one from any other department ever read any of these documents, even when they asked me to produce one. For another thing, no matter how I modifi ed this document, I never really found it useful in the day-to-day management of the test projects I worked on. Finally, creating and maintaining the test documentation always seemed to take a signifi cant amount of time and effort away from the actual testing activity itself.

Naturally I tried to fi nd ways to improve the effi ciency of creating and maintaining these documents, but after a while I just started to question the usefulness of the document entirely. One day I realised that I had fallen into the trap of forgetting to question why I even had to create one. This time when I asked the question the response disturbed me: “Because it is expected that you create this documentation.”

Wait a minute! Hold the phone! Who expects me to create this documentation? Why do I have to create this documentation? Where’s the value in that? I’m the one doing the job here and I’m seeing that I’m wasting my time creating and maintaining documents and procedures that don’t actually help me do my job any better. Where’s the sense in that?

A Brief Glimpse into Project ManagementBy sheer coincidence, it was at that moment that I attended a Project Management workshop. I

had requested it because I was struggling to manage several different test groups and projects simultaneously. An interesting thing happened after attending that workshop: I realised that I

Acknowledging that prior experience and knowledge based upon someone else’s incomplete knowledge is the

biggest hurdle

Putting the “Test” back in Test Plan

WhatIsTesting.com60

never wanted to create a Test Plan again.

During the course I learned about the Project Management Institute2, we covered the Project Management Body of Knowledge (PMBOK), and worked through several examples in order to understand how to apply the correct knowledge, tools and techniques in order to successfully manage a project to completion. I highly recommend such a course for any Test Manager. One of the most signifi cant aspects of the course for me was learning about the elements of a Project Work Plan:

● The Project Team

● Project Charter

● Scope Management Plan

● Risk Management Plan

● Communications/Reportin Plan

● Work Breakdown Structure

● Master Budget

● Task Responsibility

● Critical Path Schedule

Wow. That was it! I fi nally understood what it was that a Test Plan must have originally tried to accomplish. At some time, before Project Management became a formally recognised role in a Software Development organisation, it must have been commonplace for each of the teams to be required to produce their own little mini project work plans. Since that need doesn’t exist anymore, it was time to get with the program and stop wasting my time creating test documentation that clearly no one needed anymore.

Nature abhors a vacuum, and Software

Development is no different

When you stop creating Test Plans, you run into all kinds of resistance, especially from Management. There are some good reasons to create extra documentation and there are many bad reasons. To be clear, “Because it’s expected” is not a good reason.

One valid reason for producing some documentation is to help you learn from what you did on a test project. Being faced with all of the missed bugs now being reported by Customers through the Support department after a product ships is a harsh wake-up call. If you don’t have a record of the who, what, where, when, how, or why you did or didn’t test certain things, it is more diffi cult for you to learn how to improve and reduce the number of escaped bugs in the next project.

It is important to note that a Project Plan doesn’t completely replace a Test Plan though. While the project plan may outline all the players, the general work breakdown structure and so on, there is at least one thing not captured to any useful detail by a Project Manager. One of the single most important things that a Test Team needs to own on any project is the Test Strategy.

It took me a few years to become reasonably good at developing and implementing an effective Test Strategy. I think it is a key skill for any Tester or Test Manager. My learning path involved “going back to square one” so that I could really know what I needed to be effective. Nowadays there are some good references and workshops that can help jumpstart you to reasonable success.

Test Plan 2.0 – A Fresh StartIn the end, it comes down to this: you will never be able to test everything; you will never fi nd all the bugs; and you will never have as much time, people, or resources as you really want. Starting with that in mind, your Test Strategy needs to help you work with what you have, in the time you have, to do the best job you possibly can.

Nature abhors a vacuum, and Software Development is

no different


by Paul Carvalho

For me, the best way to accomplish this kind of “plan” was to use a Risk-Based Testing approach3. A few authors in particular had the most infl uence on me during my on-the-job education: James Bach4, Cem Kaner5, and Ross Collard. It was at that time that I also came to learn about the Context-Driven School of Software Testing.6

Context-Driven Testing was an interesting paradigm shift that helped me to understand the last piece of the test management puzzle: why do certain test practices or approaches seem to be successful sometimes but not always? Because that’s the way it is. Success depends on a great number of factors that change with almost every project. To be a successful Test Manager requires that you are a keen observer of these infl uencing factors and can adapt your team’s dynamics and approaches to suit the needs of your current situation. Darwin had it right when he said that things in nature adapt or die. We should too.

Armed with this new knowledge, I am now quite comfortable developing Test Strategies tailored to suit the needs of each project that I work on. I take no offence when I am asked to produce a “Test Plan”. Instead of providing a document that someone else thinks I need, I create something that I fi nd useful – a usable, working Test Strategy. Instead of worrying about templates, cover pages, and document control systems, I worry about Quality Criteria, Technical Risks, and Test Techniques.

A typical Test Strategy document of mine is about one or two pages in length, and I don’t worry about formatting or templates anymore. It sits on a network drive that is accessible to the whole development team, and we update

our strategy as the risks and priorities change throughout the development phase. This is Agile Development. Things change and we need to adapt our testing strategy along with the new information as it arises, whenever it arises.

The next time you are asked to provide a “Test Plan”, remember to focus more on the “Test” than the “Plan”. If you fi nd you are putting too much effort on the latter, you should take a serious look at the value that the documentation effort is providing your organisation. Time is money, and the last time I checked, I’m getting paid to test.

References 1. An example of an Industry Standard for

Test Documentation is IEEE Standard 829 - see http://standards.ieee.org/

2. To learn more about the Project Management Institute, you can fi nd them on the web at http://www.pmi.org/

3. For an excellent article to get you started on Risk-Based Testing, read James Bach’s article “Heuristic Risk-Based Testing”, available on his web site at http://www.satisfi ce.com/articles/hrbt.pdf

4. James Bach: to fi nd out more about his work and for excellent resources on Test Methodology, explore his web site at http://www.satisfi ce.com/

5. Cem Kaner: author, professor, and advocate of great software testing practices, he maintains several useful web sites. His personal site is http://www.kaner.com/ but most of his work and Testing educational materials can be found for free at http://www.testingeducation.org/

6. To fi nd out more about Context-Driven School of Software Testing, check out http://www.context-driven-testing.com/ or pick up the book “Lessons Learned in Software Testing, A Context-Driven Approach” by Kaner, Bach and Pettichord.

Why do certain test practices or approaches seem to be

successful sometimes but not always?

Accelrys has over twenty years of innovation and technology leadership in

delivering solutions that transform discovery and development research.

The company provides market-leading modeling, simulation and informatics

software, data pipelining and platform technology, and solutions consulting

and support services. A unique combination of expertise in both life and

materials science lets Accelrys serve a diverse range of clients, including some of

the world’s leading pharmaceutical, biotechnology, chemical, nanotechnology,

government and academic research organizations.

Accelrys is a global company. Our office in Bangalore, India undertakes

engineering and quality control tasks.

San Diego, CA USA

Cambridge, UK

Bangalore, India

Burlington, MA USA

Tokyo, Japan

Paris, France

For more information, visit our website:

EXPRESS YOURSELF

“WhatIsTesting” magazine is written for software and application development mangers, project managers, team leaders and Test & QA managers. The magazine is not written for the entry level tester running test scripts all day, but rather has a larger more strategic view of the entire application life cycle.

The reader is a decision maker within an IT department. Articles in the magazine should provide useful information to help him/her understand trends and emerging technologies, come to grip with new and timeless challenges, adopt best practices and ultimately make better decisions to improve software quality.

PresentationArticles should be in the range of 1500 to 4000 words, and be written for practitioners in the fi eld; with a style that is down to earth, practical, interesting and original. Avoid jargon, speculative theory and abstract concepts; instead, focus on practical information that can help our readers today or in the near term.

The purpose of an article is to transfer useful knowledge to the magazine’s readers, rather than to promote the author’s accomplishments or the products, services and technologies offered by the author’s employer.

Don’t start the article with long fancy anecdote, a “fi rst-the-earth-cooled” historical timeline or an academic-style abstract that says, “This article will do such and such”. Just get right into the subject matter. Also, don’t include footnotes or references. Refer to other sources, such as book, magazine articles or Web sites, within the main body of the article. Be sure to defi ne terms and acronyms that may not be familiar to the reader; incase of doubt, defi ne.

Include, annotated source code, only when it helps explain the concepts in the article. Also, add copious fi gure, tables, listings, screen captures and other graphics to make the article easier to understand. Please avoid bulleted points within our prose; it is hard to read such lists. If you feel that a list can best be expressed using bullets, consider pulling it into a separate tale or textbox.

Do not assume that the reader has specialized experience or knowledge about particular methodologies, platforms or tools. If the article is written about a specifi c methodology, platform or tool, present that dependency in the initial proposal. In the manuscript, include a short introduction or overview of prerequisite background concepts, if appropriate.

Article ReviewsAll proposals and manuscript submissions should be sent to the magazine by the article’s author. If there are multiple authors, one author will be designated as the lead author. The editors may send article, proposals and fi nalized manuscripts for peer review. In some cases, the editors might ask perform a peer technical review of the manuscript.

Once the necessary review has been completed’ the article will be sent back to the article’s author for fi nal approval.

Welcome to TEST 2008Test2008 is the fi rst conference being organized by PureConferences in India. Our conference will provide a platform for international and national test professionals to interact and participate. Speakers from around 10 countries, such as USA, UK, France, Sweden, Canada, Italy, Netherlands, including India will deliverkeynotes, tutorials, and papers during the conference. We intend to involve academia and institutions of learning with our conference. We also intend to make Test2008 a ‘green’ conference as far as possible. Additionally, we will institute two test scholarships for educating promising students unable to fi nance their studies.

Animated discussions among testers frequently threw up the requirement of agility in testing and the challenges associated with it. Creating a balance between confl icting and changing demands of quality by customers and timely releases by organizations is a real challenge.

Agility in Testing is the theme of Test2008--Test Excellence through Speed and Technology.

Agility represents nimbleness, resourcefulness, and adaptability. In the world of testing,

agility is synonymous with the testing team’s resourcefulness and ability to respond quickly to changing contexts. Organizations need to catch the ‘window of opportunity’ when releasing a product to remain profi table. At the same time, we, as customers only want to use ‘quality’ products.

We are hoping to have intense discussions around the theme to make it a learning experience for all participants. The theme is broad enough and will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known internationally and in our country will be the highlight of the conference.

Please mark theTest2008 conference program in your calendar. Our conference will offer partners theopportunities to associate themselves with this endeavor and showcase their products and participate in the conference.

If you want to present a paper or tutorial, please fi ll in theSpeaker Submission Form. To attend the conference, please Register.

We welcome all delegates to our conference.

wit magazine april_2008

Documents

testing teams

rex black testing

scripted testing bad

sudhir anjan software

critical testing processes

magazine willhave

new readers

test plan pg