test magazine - march-april 2009

Inside: Making agile fly | The value of training | Testing and cloud computing

Rosie Sherry tackles the negative view of software testing

A call to arms

I n T o u c h W I T h T e c h n o l o g y

T h e e u r o p e a n S o f T W a r e T e S T e r

Supported by

Volume 1: Issue 1: March 2009

T.E.S.T

TH

E E

UR

OP

EA

N S

OF

TW

AR

E T

ES

TE

Rv

Ol

Um

E 1

: iS

SU

E 1

: mA

RC

H 2

00

9

T.E.S.T | March 09

Full-Time Quality Assurance Manager—Immediate Opening

Tes

tTra

ck® P

ro

Test

Trac

k® T

CM

Test

Trac

k® S

tudi

o Su

rrou

nd S

CM®

Seap

ine

CM®

QA

Wiz

ard®

Pro

Te

stTr

ack®

TCM

Iss

ue M

anag

emen

t Te

st C

ase

Man

agem

ent

Test

Pla

nnin

g &

Trac

king

Co

nfigu

ratio

n M

anag

emen

t Ch

ange

Man

agem

ent

Auto

mat

ed Te

stin

g Te

st C

ase

Man

agem

ent

www.seapine.com/testmag Satisfy your quality obsession.[ [

© 2009 Seapine Software, Inc. All rights reserved.

Don’t work yourself to death. Use TestTrack® TCM to manage your testing effort.

TestTrack TCM

T.E.S.T | March 09 March 09 | T.E.S.T

Leader | 1

Welcome to the first regular issue of T.E.S.T magazine. Following the launch of our

‘toe-in-the-water’ issue last October the response was little short of overwhelming. One thing was certainly very clear; the testing industry wanted a voice – so you can now consider T.E.S.T a permanent quarterly business to business journal. As such it is your magazine, so please contact me at the email address adjacent to this column with any ideas, comments, suggestions that you may have. We are more than happy to help.

One issue that crops up in a few of the features inside is cloud computing – not so much an ‘old chestnut’ as a slightly shop-soiled concept, given a new look by the gurus of Web 2.0. Thomas J Watson, the president of IBM who oversaw the company’s rise to prominence in the 1950s, notoriously said: “I think there is a world market for maybe five computers.” Whether or not Watson actually said those words has been hotly disputed for some time (he was apparently referring to one particular model of computer), but one thing is clear, he may well have overestimated – by as many as four!

Much as I’d like to lay claim to this humorous and forward-thinking observation, any number of far more

eminent minds have pointed out the irony of Watson’s alleged gaffe that is now not such a gaffe. With the advent of cloud computing and Web 2.0 we may indeed be going back to the era of the main-frame.

I fondly remember the novelty of sending out round-robin rude messages to all and sundry on my college’s Vax system from a dumb terminal in a computer lab (surely I’m not the only one?) Although of course the web of interconnection takes in far more of the world these days than a few dusty college library rooms, the concept is at least superficially similar.

All well and good you say, but what does this mean for the software tester? While the issue is touched on by a couple of our contributors, Frank Puranik of iTrinegy gives it his full attention on page 48 in the first of our ‘The Last Word’ columns on the last page of the magazine. I invite any other test professionals out there with a view to get in touch.

We live in interesting times for IT professionals in general, although you may think the real seismic changes engendered by something like cloud computing are some way off, it’s always amazing how quickly things can change once the ball is rolling – especially if there is a clear financial incentive and especially in a global economic crisis. Necessity is the mother of invention. These concepts seem to be impinging on all our areas of interest at the moment and hitting the news consistently too, especially in terms of nervousness over security. But, I suppose a degree of suspicion is to be expected with any novelty.

A flash in the pan or the next big thing? Time will tell.

Matt Bailey, Editor

Back to the future...

the web of interconnection takes in far more of the world

these days than a few dusty college library rooms, the concept

is at least superficially similar. Matt Bailey, Editor

Editor Matthew [email protected] Tel: +44 (0)1293 934464

Group Advertising managerLorretta [email protected]: +44 (0) 1293 934465

Advertising SalesIan [email protected]: +44 (0)1293 934463

Production & DesignDean Cook [email protected] Barrington [email protected]

Editorial & Advertising Enquiries 31 Media, Crawley Business Centre, Stephenson Way, Crawley, West Sussex, RH10 1TNTel: +44 (0) 870 863 6930Fax: +44 (0) 870 085 8837Email: [email protected] Web: www.testmagazine.co.uk

Printed by Pensord, Tram Road, Pontllanfraith, Blackwood. NP12 2YA

© 2009 31 Media Limited. All rights reserved.

T.E.S.T Magazine is edited, designed, and published by 31 Media Limited. No part of T.E.S.T Magazine may be reproduced, transmitted, stored electronically, distributed, or copied, in whole or part without the prior written consent of the publisher. A reprint service is available.

Opinions expressed in this journal do not necessarily reflect those of the editor or T.E.S.T Magazine or its publisher, 31 Media Limited.

ISSN 2040-0160

Published by:


T.E.S.T | March 09

Simply visit www.testmagazine.co.uk/Subscribe.html Or email [email protected]

Quoting reference: 0309ISEB

The European Software Tester

SUBSCRIBE TO T.E.S.T.

Published by 31 Media Ltd

www.31media.co.uk

In Touch With Technology

First 200 subscriptions 1/2 price* courtesy of

Inside: Delivering with agile; The future of software testing; Anarchy in the QAGetting it right with Risk-based testing Handling the risk

I n T o u c h W I T h T e c h n o l o g y


Sponsored by

Telephone: +44 (0) 870 863 6930

Facsimile: +44 (0) 870 085 8837

Email: [email protected]

Website: www.31media.co.uk

*Please note that subscription rates vary depending on geographical location


Contents | 3

1 Leader column Editor Matt Bailey goes back to the future with cloud computing.

6 Cover story – A call to arms! Rosie Sherry says it’s time for testers to tackle the negative misconceptions

about the industry.

8 The best vehicle to drive quality Matt Bailey talks to Sally McCowan-Wright system tester at British Car Auctions,

Europe’s largest automotive auctioneer.

12 Crisis! What crisis? Nothing is normal anymore and it would take a brave person to predict what the future

holds. How is this new world order impacting testers?

16 Investment and yield: The value of tester training Although testing is gradually being recognised as a profession and a natural career path

within the IT industry, not all know the real art of testing. Senior tester Don Mills reports.

20 Making agile performance testing fly When EasyJet adopted agile software development it was unclear how much time would

be available to conduct performance testing, especially after the number of software

releases increased from four per year to one per month.

26 Mind the gap Closing the gap between what the customer wants and what the developer writes

is a matter of good communication according to software architect Joseph Wilk.

30 The testers’ guide to masking, obfuscating and scrambling The issue of what data to use for testing and development – especially when this crucial

work is increasingly being off-shored is an important one, masking, obfuscating and

scrambling can offer a very practical solution.

36 Penetration testing is dead, long live penetration testing Some of the world’s top software security experts have come to the conclusion that

penetration testing has been reborn as part of a tightly integrated approach

to security.

40 Quality assured Helping customers reduce complexity allowing them to focus on their core

business of producing quality software, Seapine company founder Rick Reccetti

speaks to T.E.S.T.

43 T.E.S.T Directory

48 The Last Word – Testing with your head in the clouds Cloud computing is the latest buzz phrase in the industry but is it more marketing

hype to drive sales, or something worth actually paying attention to? Frank Puranik

sees the light through the clouds.

CONTENTSMAR 09

26

36

4

48

SUBSCRIBE TO T.E.S.T.

T.E.S.T | March 09

4 | The testing community

There's no escaping the negative perceptions of the software testing industry. We are told testing is boring; advised to use QA or testing jobs as a stepping stone to another career; and to top it all, they tell us that testing is a piece of cake. Rosie Sherry, founder and community manager of The Software Testing Club says it’s time to set the record straight.

A call to arms!

People like us, who are serious and passionate about software testing know that the negative stereotypes are simply not true.

Yes, it can at times be boring. Yes it can be a stepping stone to another career and of course it can be a great big piece of cake. But if it is consistently any of these then you are probably in the wrong job or perhaps you need some help.

All of these perceptions are really bad for our industry. While I'm not here to analyse it in great detail, the simple and obvious fact is that if these are the common perceptions of a QA or testing job then we are highly unlikely to attract the smart, enthusiastic and dedicate people that we need to the profession. Until we tackle these myths we can’t expect the situation to improve.

So here, I present some ideas (not solutions) generated with the help of members of the Software Testing Club that may help improve the perception of the software testing industry. I don’t claim that these are ground-breaking ideas, but sometimes we just need to be reminded of them.


The testing community | 5

The perception of test automation as being a holy grail to quality software is a myth that we testers understand, however the outside world often sees things from a different angle. The imbalance of test automation compared with the manual approach is still clear. The constant promise that automation can solve many testing problems provides the unrealistic hope that manual testers are not needed.

Education, education, educationThat old chestnut again! How important is education and what should testers look to study? Many employers have a clear bias towards qualifications of all sorts (not just testing certification, but things like degrees as well). What priority should qualifications be given? Do they really make a great candidate? Which learning activities should testers consider?

Admittedly, education is part of modern culture. We often judge people's skills according to their education level. This culture is difficult to shift as it applies to the entire modern world. If this thought is pondered, most of us will easily accept that having certain qualifications does not guarantee being a great tester. In fact, many skills required for being a top notch tester are not covered in most courses. A good and typical example would be having good communication skills.

A lot of learning happens without courses. Can this be encouraged more in your work place? How about

simple knowledge sharing sessions? Or maintaining internal blogs about what a variety of people do and discover on a daily basis. Social technology – social networking sites like Facebook, MySpace and Twitter – looks as though it is here to stay and if used in a positive way can really help our learning process and improve company communication.

The myth of automationThe perception of test automation as being a holy grail to quality software is a myth that we testers understand, however the outside world often sees things from a different angle. The imbalance of test automation compared with the manual approach is still clear. The constant promise that automation can solve many testing problems provides the unrealistic hope that manual testers are not needed.

Perhaps we should take our hats off and credit the great marketing and sales efforts from test tool vendors. Or maybe we should start our own grass roots marketing effort to show the real value of good old manual testing

T.E.S.T | March 09T.E.S.T | March 09

6 | The testing community

Let’s face it, any job can be boring, it's when people choose to stand still that things get mundane. As a testing community we often stand still, accept things as they are, lose faith in the system or complain to each other. It's happened to the best of us.

and how important it is to implement the right strategy.

Testing is not boringYou only have to do a Google search to see how many people think testing is boring. It's often related to mindless repetitive actions. While testing may be boring in some projects it doesn't mean that it always has to be. Let’s face it, any job can be boring, it's when people choose to stand still that things get mundane. As a testing community we often stand still, accept things as they are, lose faith in the system or complain to each other. It's happened to the best of us.

We need to do more than simply state that testing is not boring! The question is how can we pump excitement back into our industry? Why do we only seem to discuss testing deeply within the testing community? Why isn't there someone cool ‘bigging-up’ what we do at web conferences? We seem to be locked into our own gated community.

Let’s talk and communicate moreThe web has become a great tool for people to become more social on and offline. It has never been so easy to meet and hold conversations with a wide variety of people. We can connect via social networks, blogs,

forums, Twitter, etc. While the online social space for testers seems to have gained momentum, connecting offline is a different thing. The real-world social interaction seems to be lagging behind, yet it is crucial to the development of our testing community.

The social web over the past few years has led to a massive increase in events happening across the globe. Some of the events have been commercial, but a vast majority have been grassroots. The disappointing thing, from my point of view, is that I'm still to meet another tester at one of these events. Meeting up face to face has great value and is often a source of great inspiration, ideas and conversations. How about we all put in a little effort into organising, creating or attending some of these types of events? Whether or not they are testing-focused is not the issue. The point is that we should all get out more and learn from one another through valuable knowledge and social sharing opportunities.

Value your reputationI prefer to use the term ‘whuffie’ or social capital. Go on, Google it. Reputation goes a long, long way in business or social life. If you get by just doing your job or doing the bare minimum to keep your boss happy – well, frankly you would not gain much

whuffie. In addition, it's not just about working hard. It's about working hard at the right things. It can be as simple as being nice to people or starting a conversation. Perhaps proving a point or creating some kind of perceived testing magic to show how valuable you are to the project team. Or the act of positive creativity, helping people out, proving your capability, doing a presentation, openly sharing your knowledge.

All this 'intangible yet positive’ stuff leads to unexpected opportunities and results. Get known for being a great person professionally and socially and all of a sudden people will start listening and respecting you. Imagine the difference a tester could make if important people listened. If only all us testers had a bit more whuffie!

What do you think?These are just ideas that frankly mean nothing if no action is taken. It would be nice if you agreed entirely with what I am saying, though to be honest I think this is a bit of wishful thinking! I am hoping it will give you some food for thought and perhaps a bit of inspiration to do something for our community. I welcome your feedback on the topic and encourage discussion over at the Software Testing Club which is where I hang out on a daily basis.

Rosie Sherry

Founder and Community Manager

The Software Testing Club

www.softwaretestingclub.com


t.


8 | T.E.S.T interview

The best vehicle to drive qualityEurope’s biggest automotive auctioneer, BCA has a host of complex systems under continuous development at its Farnham headquarters in Surrey. T.E.S.T editor Matt Bailey discussed business with BCA system tester Sally McCowan-Wright.

British Car Auctions (BCA) says it brings more vehicle buyers and sellers together across Europe than any other

organisation. In Austria, Belgium, Denmark, France, Germany, Holland, Hungary, Poland, Portugal, Spain, Sweden and the United Kingdom BCA is making vehicle remarketing more profitable and successful with physical auction sales, online sales, logistics and preparation, and many other services.

In the UK, BCA has 100 weekly auctions – plus a range of online auctions – with over 15,000 used vehicles consigned for sale at any one time. Stock is entered direct from the

vehicle manufacturers, from fleet and leasing companies, finance houses, dealer groups, government and local authorities and the motor trade as well as from the general public. From cars, vans, heavy commercials, plant and equipment, to caravans and motorhomes and motorcycles the BCA claims to have the widest available range.

Clearly, keeping this major organisation on the road takes sophisticated systems. Maintaining and keeping track of the inventory alone is a serious undertaking and the company has its own in-house development and testing groups to keep all systems running smoothly.

“The builders of software generally don’t use it. So obviously there needs to be stages in the development process where functionality is thoroughly put through its paces before it reaches the end user.”

T.E.S.T | March 09T.E.S.T | March 09 March 09 | T.E.S.T

T.E.S.T interview | 9

Fully user acceptedSally McCowan-Wright is a system tester in the company’s IT Services Core Systems team in Farnham, Surrey. She started out working in IT for a major national bank. As an end user she was often less than satisfied with the performance of the applications she had at her disposal. When she moved on to work for a national airline, her expertise in a particular

application lead to her becoming a ‘champion user’ and from there it was a logical step to become a fully fledged software tester and seven years ago she became a user acceptance (UA) tester at what had by this time become a contractor to the airline.

“Generally I used to be one of the first users of the software,” explains McCowan-Wright. “What I was doing was basically real-world testing of the early releases of the application in my everyday job. It was mainly desk-top applications, some on IBM green screen-type systems, typically running database and ticketing applications.”

Drugability testingSally left the airline contractor having built up a large amount of testing experience in this specialised area. She moved on to a pharmaceuticals software solutions provider as a quality assurance analyst, replacing a software tester in their team.

In this highly technical biochemical field – the company produced data-processing applications to assess the effectiveness of various drug

“It is still a very young industry compared to software development. We have really only been visible for the last 15 years and it’s only in the last 10 years that the powers-that-be have really sat up and taken notice and testing has gone from something which is nice to have to being recognised as a critical part of the business. With the current economic meltdown, it is more than ever all about beating the competitor and getting your product to market quicker and with higher quality. It’s all about survival.”

formulations on proteins and DNA – she redefined the software release structure where the company had four releases a year and introduced testing phases all the way through in place of the existing structure which saw just one test process at the very end of the schedule.

The motor tradeTwo and a half years ago Sally

McCowan-Wright joined BCA as a systems tester. “BCA has a significant amount of systems,” she says. “We act as the middleman between vendors and buyers, but we look after all the financial and inventory details and with thousands of vehicles all around the country and in Europe there is a lot of data to manage. And all has to take into account a range of languages and legislation in all the countries involved.”

McCowan-Wright works very closely with the software development team on the projects they are running. “I work on two to three projects at a time, estimating, planning, physically running the tests, managing and


10 | T.E.S.T interview

“A lot of money has been wasted because automated systems either were not needed, used or maintained. It will never take the place of manual testing because products are very user-driven these days so it takes a human to spot faults.”

flagging defects and analysing them,” she explains. “The analysis can and often does lead to debate and discussion between the testing and development groups. But we generally agree on a course of action to fix the defects and work out if this affects or even stops the next set of tests and what impact it has on the finish date for the project.”

Sally sums up the reason for software testing as follows: “The builders of software generally don’t use it. So obviously there needs to be stages in the development process where functionality is thoroughly put through its paces before it reaches the end user.”

A typical work day for McCowan-Wright could involve discussing the levels of testing required throughout life-cycles, writing test scripts and actually conducting tests as well as all the usual office business. “My day-to-day work varies considerably,” she says. “Some days it’s nearly all testing, but we also have to consider planning,

managing, reporting and working on improving what we do as well.”

A young industryAs Sally McCowan-Wright points out, testing as an industry is still very much in its infancy in many organisations. “It is still a very young industry compared to software development,” she says. “We have really only been visible for the last 15 years and it’s only in the last 10 years that the powers-that-be have really sat up and taken notice and testing has gone from something which is nice to have to being recognised as a critical part of the business. With the current economic meltdown, it is more than ever all about beating the competitor and getting your product to market quicker and with higher quality. It’s all about survival.”

There is a necessity for testers to do as much as possible in the timescales to deliver a quality product. “It’s called the quality conundrum,” says McCowan-Wright. “It’s a case of cost


T.E.S.T interview | 11

“Awareness of testing needs to be increased. Its crucial place in the product lifecycle and how valuable it really is need to be communicated. There is still a stigma attached to testing. It is seen as a bottleneck in production that holds-up progress. This is compounded by the fact that testing is a relatively young discipline, managers say, ‘we didn’t do it before, so why do we need to do it now?’ But if they want to deliver quality products better and faster testing is a crucial part of the process. ”

versus benefit versus quality. We have to maximise quality and benefit with minimum cost. And this manifests itself as risk-based testing, where the issues that are most likely to occur are tackled first and those least likely to be a problem are not immediately prioritised. But some testing is always better than none.” In this way the high and medium risks are concentrated on and if possible the low risks are tackled if they can be within time and budget constraints.

Automation and offshoringAnother way of tackling the constraints of budget and time is through automation, but Sally McCowan-Wright thinks that it isn’t the answer in most cases. “Automation has been pushed into testing incorrectly in the past,” she asserts. “Non-code-based testers are unable to use and can find it hard to understand the implications of what’s thrown up by automated systems because the faults are generated out in code that a functional tester wouldn’t necessarily understand. A lot of money has been wasted because automated systems either were not needed, used or maintained. It will never take the place of manual testing because products are very user-driven these days so it takes a human to spot faults.”

There is now a problem with scepticism about automated testing because systems were sold badly in the first place, but she isn’t entirely

hostile to the concept. “It does have its place,” she says. “In the right place and time and in the correct project and application it can be very effective.”

Another potential challenge for Europe’s software testers is offshoring. In an early testing job McCowan-Wright had some experience of an offshored job and decided it was not all it was cracked up to be. “Communication is key,” she argues. “If communication fails it can be catastrophic to a project, but this is true for any project, people working in different locations across the UK could have exactly the same problems. Clear precise and timely communication is what’s needed and if the channels are open then offshoring does have its place.”

Raising the profileLooking ahead, Sally McCowan-Wright sees the need to get the message out. “Awareness of testing needs to be increased,” she argues. “Its crucial place in the product lifecycle and how valuable it really is need to be communicated. There is still a stigma attached to testing. It is seen as a bottleneck in production that holds-up progress. This is compounded by the fact that testing is a relatively young discipline, managers say, ‘we didn’t do it before, so why do we need to do it now?’ But if they want to deliver quality products better and faster testing is a crucial part of the process.”

T.E.S.T | March 09

12 | Testing times

Crisis! What crisis?The past year has seen unprecedented shifts in the power balance of the business world and previously thought of scenarios that would not so long ago have seemed laughable, have come to fruition. Nothing is normal anymore and it would take a brave person to predict what the future holds. Dietmar Wand, account manager at Merit AT has a look at how this new world order is impacting testers.

March 09 | T.E.S.T

Testing times | 13

T.E.S.T | March 09

There is no doubt that belts and budgets are being tightened and that those nice to have projects are being put

back on the shelf to gather dust until happier days. This does not mean that IT has gone away, how could it? So for now, the impact does not seem to have been as severe in IT as it has in other areas. It’s certainly not unscathed but it has not been decimated.

Organisations still have to operate, software still has to be delivered and more importantly it has to work. In fact, the nice twist to this is that the software has to work better than ever before to minimise the costs of failures and increase customer confidence.

A classic example of this is the very industry that has been at the root cause of the problems our economy is facing, banking. When two banks merge it’s not just their balance books, it’s everything that must be joined, staff, buildings and inevitably IT. This presents a mountain of opportunity for the testing industry, from individuals through to testing service providers. But we have to be realistic. These opportunities will not have money thrown at them. This isn’t 1999 and people will want to stretch their pound more than ever before. But it still has to happen in one form or another. Our responsibility is making sure that we continue to provide a valuable business service at a cost that can be accommodated in the current financial climate.

Testing now has to be seen as not just a necessary evil, but a fundamental part of the development lifecycle that, if done properly, will not just increase quality but drive down the cost of developing and maintaining software. But you might just have to spend to save money. To the credit of many organisations we are seeing this change in attitude. Testing does seem to be changing – in the past testing was often the first activity to be culled in hard times as it is often seen as a pure cost centre ie, testing doesn't 'produce' anything. This now appears to be changing. No one seems to be stopping testing like they have in the past. They are simply reducing the amount of testing activity as costs are cut across the board. Basically, the view of testing seems to have changed and matured further to be seen as an essential function across the board.

The challengeSo the challenge is this: if project cycles are just as aggressive as before and there is an inevitable reduction in budget, how do we maintain the quality of what we produce with ever-reducing resources? The moves made now will separate the winners and losers in the months and years ahead. Companies are laying-off staff across the board, and this will hit testing and testers. Both contractors and permanent workers will suffer as this is the easiest and quickest way a company can reduce costs.

March 09 | T.E.S.T

Organisations still have to operate, software still has to be delivered and more importantly it has to work. In fact, the nice twist to this is that the software has to work better than ever before to minimise the costs of failures and increase customer confidence.


14 | Testing times

Dietmar Wand Account manager merit AT www.merit.org.uk

This brings inherent threats to organisations. A likely scenario is a slow degradation in quality over time as not enough testing is being covered. Issues will then start to gradually come to the surface until they become unmanageable, costly and public. At this point, the crisis point, the slow painful realisation that all of this was avoidable will be as painful as signing the order to bring back a small army of people to fix the issues. Organisations need to look inwards and outwards for ideas and solutions as to how they can be more efficient in testing with the specific problems they face.

Software testing has a unique opportunity to flex its muscles and show people why it is so fundamentally important. The slow changes we see in attitudes can become an avalanche. The powers that be, basically the men who sign the cheques, are looking right now for ways to save money in the short, mid

and long term. For organised and vocal test teams now is the time to say what you can bring to the party and what you need in terms of helping your business through the tough times.

As a service provider we are seeing a shift in our business in the current climate. Our traditional services of helping companies structure and deliver improvements and changes still exist – although these are much more focused on delivering cost savings than before. However we are also seeing new opportunities start to appear as companies try to get the process changes (that they were forced into through the reductions in staffing and costs) right. In this situation, change was required quickly, but no additional help could be brought in. For many companies the harsh reality that this cannot and has not worked is starting to dawn, opening up more opportunities for us testers.

Testing now has to be seen as not just a necessary evil, but a fundamental part of the development lifecycle that, if done properly, will not just increase quality but drive down the cost of developing and maintaining software. But you might just have to spend to save money.


Users crashing on performance issues?

Not having your application subjected to tests can have serious consequences: high

repair costs, dented customer relations, bad publicity, loss of face and a lot of extra

work. At Computest it is possible to have your tests executed on site or remotely in a

professional manner against lower costs. No more steep investments, a fixed price

tailor-made to your needs. Lower costs and less risk, what more can you ask for?

www.computest.nl • Tel: +31(0)79 - 750 1779 • E-mail: [email protected]

T.E.S.T | March 09

16 | Training

Investment and yield: The value of tester trainingAlthough testing is gradually being recognised as a profession and natural career path within the IT industry, not all know the real art of testing. It’s one of the reasons why e-testing Consultancy has adopted the ISEB tester certification scheme to run a series of structured courses leading to recognised certificates in software testing. Senior tester Don Mills reports.

The initial ISEB (ISTQB) Foundation Certificate course has been a resounding success. More than

48,000 people have now completed these three day courses, run by ISEB accredited training providers. The certification scheme is effectively backed up by the British Computer Society (BCS) and Special Interest Group in Software Testing, which defines career paths for IT professionals.

QC, QA or risk management?I used to teach people that testing was all about software quality control. This would often start arguments about whether it was quality control or quality assurance (see below), but the point was that I saw its purpose as enabling a project to keep the quality of its work products under control. This was and is done by periodically measuring (testing) work-product quality, and reporting back on aspects that fall below par. And this is necessary because, if you don’t keep a constant eye on it, quality will rapidly drift below par.

But there’s certainly something to be said for the point of view that testing can mean quality assurance. One of my mentors in testing, almost twenty years ago, was Boris Beizer, from whom I learnt that: “The proper use of testing isn’t finding bugs, but preventing them.” This is indeed a

T.E.S.T | March 09

T.E.S.T | March 09 March 09 | T.E.S.TT.E.S.T | March 09

‘quality assurance’ activity: work done to assure quality, by assuring a development environment which doesn’t create bugs, but prevents them instead. From early days, my testing courses have identified several ways in which testing can contribute to creating such an environment.

Later on (I’m a slow learner), I came to associate testing with risk management. It’s a happy fact that a well-designed test case that finds a bug, thereby eliminates a risk, just as surely as one that doesn’t find a bug (since a risk is an uncertainty: something that might go wrong, but hasn’t yet). Testing is important, I tell people, because of the many risks that users of untested software may face. If testers can do nothing else, we can turn those risks into certainties, known problems! (But mostly it’s even better to get them fixed.)

But why do testing?All of these interpretations of testing carry a degree of truth, but none of them really has the force of a business imperative. As we’re all only too well aware, we are in straitened economic times in 2009 (belt-tightening, and all that). Why should businesses throw scarce money into testing their software before releasing it? And why should they throw yet more money into training people how to do testing? Isn’t it something that anyone can do, from clerical staff taken from behind

their desks, to failed developers that you want to put somewhere where they can’t do any more damage?

Well, we’ll deal with that set of ideas below. But first, let’s come to the real purpose of doing testing. Why does software quality need controlling? Why do we need development environments that minimise bug-generation? Why do the risks of faulty software products need to be managed by testing them?

Quality, bugs, and risk can certainly look after themselves without help from testing, but the result will be low quality, bad bugs, and high risk, as sure as eggs is eggs and night follows day. And what really matters to businesses is that poor quality, harmful defects, and high failure rates, are all likely to cost money, whereas testing can save huge gobs of it.

“We compromised on the testing”Queen Elizabeth II officially opened Heathrow’s Terminal 5 in a ceremony on 14 March 2008. On that day it quickly became apparent that the new terminal was not operating smoothly, and British Airways cancelled 34 flights and was later forced to suspend baggage check-in.

In the first five days of operation, BA is reported to have lost sixteen million pounds. Part of the blame can be attributed to overruns on the construction work, and other problems

not directly related to software. But, according to an item in Computer Weekly (“Lack of software testing to blame for Terminal 5 fiasco, BA executive tells MPs” 9 May 2008), significant contributors to the loss included: “a software filter being left on the BAA baggage system,” which had apparently been included to ensure that only test data would be processed (!), and the inability of the servers to “cope with the number of messages the baggage system generated”.

British Airways' chief executive, Willie Walsh, commented: “If I were to pick one issue I would have done differently, it is that, having recognised the importance of [software] testing and having designed six months of testing, we subsequently compromised on that.”

A context for good testingOver the past five years, I’ve spent a lot of time guiding certification exam candidates through the material necessary to obtain the ISEB/ISTQB Foundation Level Certificate in Software Testing. As a testing course, it’s something of a disappointment, if you were expecting to be trained in “how to test”. Its real objectives are revealed by its target audience, which includes testers, for sure, but also includes “people in roles such as… project managers, quality managers, software development managers,

Training | 17

Quality, bugs, and risk can certainly look after themselves without help from testing, but the result will be low quality, bad bugs, and high risk, as sure as eggs is eggs and night follows day. And what really matters to businesses is that poor quality, harmful defects, and high failure rates, are all likely to cost money, whereas testing can save huge gobs of it.


18 | Training

business analysts, IT directors and management consultants,” not to forget “software developers”.

The truth is that software testing is a much-misunderstood set of activities, and the people listed above, generally speaking, understand it even less well than testers do. This is a pity, since they are the people who create the context within which good testing happens—or doesn’t happen, as is more usually the case. Willie Walsh may have had an excellent team of testers waiting to leap into action, but (like the testers on the infamous Ariane V project, which was a 40-second-long seven billion dollar software disaster) they never really got the chance. It’s significant that Computer Weekly filed their article under “Project

Management”, not under “Software Testing”.

In some respects, then, it’s the “project managers, quality managers, software development managers, business analysts, IT directors and management consultants” who need some basic training in testing: what it is, what it can and can’t do; how to use it rather than misuse it, and so on. Sadly, we see very few of them on our training courses. The industry needs to find some way to get the message to them in a clear, unmistakeable, and above all helpful, way.

But what about the testers?Meanwhile, I had about 250 testers, or would-be testers, in front of me last year for training at the ISEB/ISTQB Foundation level, plus around 100 others for different varieties or levels of testing. The costs of ISEB/ISTQB Foundation training differ from training supplier to training supplier, and from time to time (they’ve mostly seen a respectable decrease this year), but a typical price might be between £675 and £900, including the examination fee. The obvious temptation, in times when funds are scarce, is to save money by eliminating tester training. After all, anyone can do it, no?

But as Tom Millichamp pointed out (Can you afford not to train, T.E.S.T, October 2008), “a test team that is not properly trained to do its job is at best not working to its optimum capacity, and at worst a financial disaster waiting to happen.” Many, many other organisations besides BA and the European Space Agency can attest to this. Testing isn’t something that just anyone can do, or at least, not if it’s going to be successful.

It’s certainly true that schemes such as ISEB/ISTQB Foundation certification have brought about “an accepted formal training structure for learning the key foundational skills of testing”

British Airways' chief executive, Willie Walsh, commented: “If I were to pick one issue I would have done differently, it is that, having recognised the importance of [software] testing and having designed six months of testing, we subsequently compromised on that.”


Training | 19

Don mills Senior Tester e-testing Consultancy Limitedwww.etesting.com

(Millichamp again), and that the ISEB Intermediate and Practitioner, levels to some extent build upon them. And there is no doubt that well-applied testing – especially, testing applied to the prevention of bugs – can generate huge savings both in the development of software, and its use in live environments.

And what about the future?The ISEB/ISTQB Foundation syllabus is not (I believe) really intended to teach the skills that Millichamp refers to, so much as to make all participants aware that such skills exist, are very advantageous, and need to be learned. This is useful stuff indeed, but I think that some valuable opportunities have been missed, which the future might bring in to a tester training programme.

For example, the original Foundation syllabus, included mention of “cyclomatic complexity”. This has been dropped from all current syllabuses, Foundation, Intermediate and Practitioner – partly (I suspect) because their authors didn’t really understand it. (I beg their pardons if there other, better, reasons.)

This is a great pity, because it’s not really a difficult concept to understand, nor a difficult measurement to make, and neither is it just an attribute of program source code, as the syllabus portrayed it. Rather, it’s a fundamental property of “test models” such as flowcharts, decision tables, state transition diagrams, cause-effect graphs, activity diagrams, and even use case narratives; a property which can be used to help design powerful test sets that achieve high levels of coverage, and disclose high numbers of bugs, with relatively low numbers of test cases.

Such techniques have been shown to find approximately the same number of bugs, as found by double the number of test cases developed

by less ‘scientific’ means. Studies of this phenomenon have been reported for both code-based testing, and specification-based testing. The techniques are collectively known under the label, High-Yield Testing (google it if necessary), because they return a high yield of bugs for a relatively low investment in the number of test cases run.

ConclusionsWhat can we conclude from this? Here are the highlights:• Software development, and the

business use of software, are both risky undertakings, where the risk is that the software may let you down badly, and expensively.

• Failures of software projects, or of deployed software, can be very expensive indeed, and can usually be traced back to failures of or in testing.

• Good testing practices can be learnt the hard way – through raw experience – but it’s much less uncomfortable, safer, and cheaper, to learn the lessons from someone else.

• Tester training costs money, but it’s an investment that can repay itself many times over, provided it hits the right nails on the head.

• But even the best-trained testers can’t work their magic to real effect if the project structure is against them. Those who create the environment in which good or bad testing takes place also need some training.

And finally,• The ISEB tester certification

programme, adopted by e-testing Consultancy, have made a big difference, but not necessarily big enough. More opportunities for providing testers with skills that are of direct value to the business need to be created.

Let’s look to the future, folks.


Making agile performance testingflyWhen EasyJet adopted agile software development it was unclear how much time would be available to conduct performance testing. Paul Seaton-Smith, principal consultant at Capacitas describes how it met the challenges of testing after the number of software releases increased from four per year to one per month with an agile approach.

Before describing how we met the challenge of performance testing in an agile environment it is

worth explaining how performance testing should fit within the general development and service management methodology. The first thing to note is that performance testing should not be conducted in isolation. It is one of a group of performance assurance activities that can reduce uncertainty about the risk of performance issues.

EasyJet’s head of software delivery, Colin Rees comments: “Capacitas delivered a cost-effective and rapid performance assurance process to support our adoption of agile development. The performance assurance process has proven to be robust and prevented critical performance defects being released into our business-critical e-commerce systems.”

EasyJet’s test team leader James Mackay ensured that its performance assurance framework aligned with the EasyJet agile development methodology. “Capacitas provides a performance testing consultation and outsourced testing service for EasyJet. The rapid agile development methodology requires a very short window for performance testing each month. Capacitas and the EasyJet testing team engaged early on in the project design stage, ensuring a smooth transition from conceptual design, risk assessment, test planning, scripting and performance testing to producing results for release decisions. By ensuring all phases of the performance testing are planned upfront it is possible for risk mitigation performance testing to be conducted within very short testing window. The power in the model is the two-pronged testing strategy of using

20 | Agile testing


Agile testing | 21

baseline metrics and targeting testing as this ensures metric-based continuity in untouched areas of the code and performance results against new areas of development. This low-cost testing process, driven by risk, provides test results that give us confidence in making our release decisions.”

The EasyJet System

The EasyJet system has several interfaces to the main database. These include the web interface (easyJet.com), call centre/airport staff interface which uses Telnet servers and a B2B XML over HTTP interface. EasyJet.com has the highest volume of traffic out of all these interfaces. The production system has interfaces to a number of external systems including those for credit card authorisation, car hire, insurance and hotel bookings.

Performance test environmentEasyJet.com has a dedicated environment for performance testing. EasyJet recognises that the majority of its revenue is generated via the web platform. Therefore it views performance testing as an essential

“Capacitas provides a performance testing consultation and outsourced testing service for EasyJet. The rapid agile development methodology requires a very short window for performance testing each month. Capacitas and the EasyJet testing team engaged early on in the project design stage, ensuring a smooth transition from conceptual design, risk assessment, test planning, scripting and performance testing to producing results for release decisions.

function of the development cycle. The performance test environment is significantly smaller than the production system and the hardware in the performance test environment differs in specification from the production system. Modelling techniques are used to understand what the results will mean for the production system resource utilisation and service performance.

Moving to agileWhen developing software using traditional methods there were four major software releases per year. This ensured approximately three weeks of performance testing time. The contents of each release were very

well defined, meaning a low likelihood of environmental or functional defects during the performance testing cycle.

With the introduction of an agile development methodology EasyJet does one release per month. The diagram overleaf shows the agile approach used by EasyJet. The development, functional and performance testing teams and project owners are all involved during the Sprint planning meeting. The exact contents for a release are decided during this meeting. Time slots are assigned to the development, functional and performance teams.

The Capacitas performance team produces a performance risk assessment after the Sprint planning

EASYJET USE HP MERCURY LOAD RUNNER AS ITS LOAD GENERATION TOOL


22 | Agile testing

meeting. This is reviewed and agreed by the EasyJet test manager. As the Sprint cycle begins, the process of writing the detailed performance test plan is started. As the Sprint progresses, the performance test plan is updated with changes to the test cases, revised timescales or other information to support the performance tests.

In EasyJet’s case the code is typically released into the performance test environment after the Sprint process and functional testing has occurred. However, it is also possible to have code released into the performance test environment during the Sprint process, resulting in more than one code build dropped into the performance test environment during a Sprint. The time window for performance testing is usually very short, only a matter of a few days where the Sprint has taken longer than expected. Typically, Capacitas gets two weeks to conduct the performance testing for easyJet.com.

During this short time window Capacitas uses multiple performance testers to create test scripts, execute tests and analyse test results to maximise the use of parallel working where possible and hence meet the short timescales. The limiting factor in the case of EasyJet is a single performance test environment, therefore only a single test can be executed at any one time.

The challengesPerformance testing in an agile environment has presented many

challenges. These largely arise from working within a shorter test time frame than in traditional development approaches. Aspects of agile development, such as the stability of the code base, can shorten this time window even further. In addition there are technical challenges from a performance perspective.

The reduced testing window is a significant challenge. Capacitas typically has a maximum of two weeks to conduct performance testing within EasyJet. This has required a greater emphasis on planning performance tests to ensure that they are ‘right first time’. The performance risk assessment and performance test plan need to be clear on the objectives and purposes of each test to ensure

EasyJet recognises that the majority of its revenue is generated via the web platform. Therefore it views performance testing as an essential function of the development cycle.

THE AGILE APPROACH USED BY EASYJET


Agile testing | 23

that we are maximising the use of our performance test window and not wasting valuable test time on ‘nice to haves’.

In agile development there is an increased likelihood of functional and environmental defects in the performance test environment. The code may still be under development during the Sprint process. Therefore the code released into the performance test environment has a greater probability of having defects. Resolving these defects can consume a significant amount of the performance test time window and leave less time for actual performance testing.

The frequent code changes to accommodate smaller functionality changes are also challenging. The small changes can result in re-writes of previous test scripts and hence increase the overall scripting time and leave less time for actual performance testing. This can also increase the overall testing costs.

The overlap in monthly releases may cause resourcing issues, eg planning for the next release while conducting testing for the current release. This can potentially take away valuable resource from the performance testing of the existing release and further impact the time frames.

In addition to the challenges around time and resources, there are additional challenges in terms of the potential performance degradation per release. As new features are introduced, these new features will inevitably use resource of the

infrastructure that supports the application. The cumulative effect of performance degradation over monthly releases results in a more significant annual performance degradation.

Meeting the challengeCapacitas adapted its approach in a number of ways to meet the challenge of agile development. More detailed planning is done during the production of the performance risk assessment and test plan. The performance risk assessment is used to determine what needs to be tested depending on the risk to the production system. This ensures that low risk changes are not tested while medium to high risk code changes are tested, therefore maximising the usage of the test window.

The performance test plan typically has fewer tests in recognition of the reduced testing time window but the tests may include greater coverage, eg a customer selects all available options rather than a selection. Capacitas may only aim to do three to four tests in an agile release. Potentially, there will be multiple iterations of each of these tests depending on the frequency of code drops into the performance test environment.

During a test all capacity and performance metrics are collected but not all are necessarily analysed. This ensures that, as long as high level performance and capacity metrics such as response times, error rates and CPU service times are acceptable, then no further analysis is required. If

Performance testing in an agile environment has presented many challenges. These largely arise from working within a shorter test time frame than in traditional development approaches.


24 | Agile testing

Types of performance testing

Load testing determines the

response time and throughput

during typical forecast load.

Stress Testing determines the

peak throughput, gradually

increasing the load beyond that

expected until the maximum

achievable load is reached.

Volume Testing or Soak Testing

determines the problems that

occur during long-term activity

(eight hours to one week).

Single User Testing determines

the system resources consumed

by a single transaction on an

unloaded system, ensuring

queuing does not occur and

allowing collection of service

times (typically required for

production of performance and

capacity models). Paul Seaton-SmithPrincipal consultantCapacitaswww.capacitas.co.uk

these high level metrics appear to be significantly higher than expected then further analysis of the other metrics can be carried out. This reduces the time spent on analysing tests and targets time and resources on problem tests.

A performance test environment checklist is produced and then updated after every release. This will reduce the amount of time lost due to environmental defects.

Because of the frequency of change in an agile development process, a significantly lower level of accepted performance degradation per release needs to be set. The performance degradation needs to be tracked in conjunction with the capacity and performance modelling work. The diagram below shows how the cumulative impact in small changes in performance for every agile release can have a significant impact. Therefore the acceptable degradation per release should be three-times lower for agile, if we assume each agile release is equivalent to a ‘traditional’ release.

In order to assist the evaluation of the performance degradation, Capacitas has defined a standard test in the performance test strategy which is executed on every agile release. This is called a ‘baseline test’ and is executed before and after the new code release. A more detailed level of reporting is required by the performance testers, who will formally raise all environmental issues as defects to ensure that they are attended to promptly, thereby reducing the loss of valuable test time.The performance testers and analysts

need to provide frequent interim performance test reports to ensure that any results are suitably escalated and dealt with in time for the planned release date. The performance testing teams must be willing to challenge the release contents if there is not sufficient time to test out the performance risks.

The performance modellers need to work more closely with the performance testers in order to understand what the results of the performance testing will mean for the production system. This needs to be done rapidly during the performance testing process in order to make judgements in ‘go’ or ‘no-go’ decisions. The capacity plan may need to be reissued to understand the impact of the changes, eg what is the impact of a significant increase in database CPU time for a new feature on the production system?

An agile resultThe shift from more traditional development to an agile development model requires a change to the performance testing methodology. This includes the need to:•Work smarter in order to cope with the reduced timescales and higher pressure environment;• Have a proven methodology which

will help to ensure that tests are designed well and meet their objectives;

• Be more flexible in the approach to resourcing;

• Maintain a longer term view of performance rather than just focussing on changes from one release to the next.


Agile testing | 25

While there is clearly no ‘one size fits all’ solution, the following proven approach for performance assurance proved successful at EasyJet.

The Performance Test Strategy is an evolving document that underpins all performance testing activities and defines an approach for performance testing. The goals are to establish good-practice, to clarify the scope and goals of future performance testing phases and to provide a solid foundation and point of reference for any future tactical decisions.

The Performance Test Procedures document includes scripting guidelines and describes how to test, how to prepare test data and how to log defects. This may be updated after the performance testing is completed.

The Performance Risk Assessment is produced after the performance team has gained an understanding of the code changes planned for the release. The objective of the assessment is to assign a risk to each element of the release and also to provide a high-level estimate for man days and associated costs for the performance assurance activities. Typically this activity is carried out by a more senior member of the performance team who has the experience to make judgements on what elements of the release are risky.

The risk is based on factors such as the expected demand for the application and also how the changes impact the architecture. This helps to avoid testing for the sake of it as everything is risk-based. It also allows us to decide the priority and execution order of tests in the Performance Test Plan.

The Performance Test Plan defines the test environment and provides further detail regarding the specific tests to be conducted. Each test case should be referenced against a risk in the Performance Risk Assessment. One key aim is to ensure that performance testing is not on the critical path for delivery of the software. In this example we were fortunate in that separate functional testing was required for all eighteen language translations while performance testing was only required to be conducted against the English version of the site.

Performance Testing is the execution of the tests described in the Performance Test Plan. The goal of performance testing is the same as all testing: to ensure the system meets the stated requirements. Performance testing can be conducted across the development lifecycle. In an agile development methodology the development lifecycle is shorter and therefore the time window for performance testing will also be shorter.

The Performance Test Report is the main deliverable from the performance testing activities. This documents all the tests executed, how they were executed and the detailed results. The report also lists the defects identified during the performance testing. This includes environmental, functional and performance related defects.

modelling activities are typically cheaper than performance testing as they don’t require large-scale test environments, costly automated test tools or even stable code. EasyJet’s performance test environment is significantly smaller than the production system, therefore modelling provides a cost-effective approach to understanding what the performance test results mean for the production system. The results from the performance testing are inputted into the capacity and performance model of the easyJet.com platform in order to understand the impact of the release on the production system resource utilisation and service performance. One weakness is that capacity modelling cannot determine whether soft bottlenecks such as database locks may occur.ITIL describes the Capacity Plan as a document produced at predefined intervals which reports the current and anticipated future levels of resource utilisation and service performance. Capacitas produces a regular capacity plan for EasyJet. This document is reissued, if required, at the end of an agile release to determine the impact of the software changes on the easyJet.com infrastructure capacity based on the capacity and performance modelling work.

A performance assurance methodology

Capacitas typically has a maximum of two weeks to conduct performance testing within EasyJet. This has required a greater emphasis on planning performance tests to ensure that they are ‘right first time’


26 | Testing communication

Closing the gap between what the customer wants and what the developer writes is a matter of good communication. Joseph Wilk, software architect at eShopworks advises on how to avoid the chasm of despair.

Mind the gap

What we want is a ubiquitous language that spans the divide between the business and the technology in describing the behaviour of a system. The business vocabulary permeates right into the tests and code base. Building the ubiquitous language is a balance between the customer’s domain and the software team’s understanding, ultimately converging on a shared model.


Testing communication | 27

Writing software that ‘does the right thing’ is dependent on good communication with

the customer. We accept that our understanding and our customers’ understanding of the ‘right thing’ is continually changing during the course of a project. So how can we help prevent this communication gap from turning into a chasm of despair?

Getting the words rightBehaviour Driven Development (BDD) an evolution from Test Driven Development (TDD) and Domain Driven Development (DDD) (see side bar for an explanation of these terms) can help us move towards a consistent way of expressing behaviour with the customer’s domain terminology.

What we want is a ubiquitous language that spans the divide between the business and the technology in describing the behaviour of a system. The business vocabulary permeates right into the tests and code base. Building the ubiquitous language is a balance between the customer’s domain and the software team’s understanding, ultimately converging on a shared model. If we can sit down with the client and talk about the interesting behaviour of their system we have a good start at ensuring we converge on the right solution.

Driving out the business valueWhile the customer may have a number of feature requests it may not always be apparent exactly why the customer wants a feature. From discussions and driving questions about why a feature is required we can help extract the business value for

all to see and dispose of extraneous features. We can then use this to continually help the customer decide what is the most important value and hence features for them.

Having a definition of ‘done’To deliver the right features we need a target to hit. By getting a definition of ‘done’ from the customer we have a goal we can aim at. We also help avoid the common “that is not what I asked for” or “I forgot to tell you about this other thing” comments.

Having continuous feedback through User Acceptance TestsIn order to know when we are done and what we have left to do we want our definitions of done to act as automated tests against our system.

Plain text User Acceptance TestsBy using plain-text we enable non-technical users to write collaboratively with developers/business about the interesting behaviour of a feature using their ‘own words’. Our definitions of behaviour, given as scenarios, are made executable as tests through developers mapping the plain text to test code. Using these tests in a continuous integration server you can achieve constant feedback on your progress towards ‘done’. You retrospectively know that changes to the system do not break pre-existing features.

Tools such as Fitnesse take a website wiki-style approach to allow plain text tests. Through a browser you can edit webpages which contain the plain text of your test. One up-and-coming framework called Cucumber expands on some of the ideas of Fitnesse while using a style of plain text user story to capture system behaviour.

While the customer may have a number of feature requests it may not always be apparent exactly why the customer wants a feature. From discussions and driving questions about why a feature is required we can help extract the business value for all to see and dispose of extraneous features. We can then use this to continually help the customer decide what is the most important value and hence features for them.

March 09 | T.E.S.T


28 | Testing communication

Joseph WilkSoftware ArchitecteShopworkswww.eshopworks.co.uk

Explaining the terms:Behaviour-Driven DevelopmentBehaviour-Driven Development (or BDD) is an agile software development technique that seeks to encourage collaboration between software developers, QA and the management or business participants in a software project. Originally conceived in 2003 it was a response to Test Driven Development (TDD). The language and interactions used in the process of software development are the focus of BDD. Behaviour-driven developers use their native language in combination with the language of Domain Driven Design to describe the purpose and benefit of their code, this allows the developers to focus on why the code should be created, rather than the technical details, and minimises translation between the technical language in which the code is written and the domain language spoken by the business, users, stakeholders, project management etc.

Test-Driven DevelopmentTest-driven development (TDD) is a software development technique that uses short development iterations based on pre-written test cases that define desired improvements or new functions. Each iteration produces code necessary to pass that iteration's tests. Finally, the programmer or team refactors the code to accommodate changes. A key TDD concept is that preparing tests before coding facilitates rapid feedback changes. Note that test-driven development is a software design method, not merely a method of testing.

Domain-Driven DesignDomain-driven design (DDD) is an approach to the design of software, based on the two premises; firstly that complex domain designs should be based on a model, and secondly that, for most software projects, the primary focus should be on the domain and domain logic (as opposed to being the particular technology used to implement the system). The term was coined by Eric Evans in his book of the same title.

CucumberCucumber is a tool, written in the Ruby programming language, which takes plain text feature documentation and makes them executable as tests. It can be used to ‘test’ code written in Ruby or other languages with the help of some extra tools:JRuby and JavaironRuby and .NETFunFX and Flex

Cucumber feature exampleFeature: Search coursesIn order to ensure better utilisation of coursesPotential students should be able to search for coursesScenario: Search by topicGiven there are 240 courses where none have the topic ‘biology’And there are 3 courses A, B, C that each have "biology" as one of the topicsWhen I search for ‘biology’Then I should see the following courses:| title || A || B || C | These are similar to User Stories but we refer to them as Features. The very first part of our feature is our narrative. This represents the business value of this feature. A scenario represents a description of some behaviour of the system. Within the

scenarios we use the language of Given/When/Then to describe the behaviour. Given – are used to set-up the stateWhen – to perform some action and Then – to test the expected outcomes.

In Cucumber we break from using the typical convention seen in User Stories:As a [stakeholder] i want [feature] so that [goal]Instead we use a convention which comes from Feature Injection.in order to [goal] [stakeholder] wants [behaviour]

While it may seem a small change, it can help drive out the value, or lack of it, from a feature. A great example of this is when you have to enter your email address twice when registering on a typical web page.As a user, I want to fill in my email twice, so that... wait, I don’t want to waste my time doing that.

Switching to the Feature Injection style we get:In order to avoid lots of registered users unable to access their accounts, users will need to fill in their email address twice.

Requirements through real-world data examplesCucumber introduces scenario tables

which allow us to express data sets to be used in different runs of a scenario. These Scenario tables were inspired from FIT (Framework for Integrated Test) tables. These are seen in frameworks such as Fitnesse.

These tables allow the customer to communicate the expected output to us through concrete data examples that exemplify the function of the system. This helps us to flush out incorrectly assumed rules and to discover real business rules.

Example:Feature: Stock keepingIn order to avoid interruption of salesAs a jeweller I want to know how many rubies I have leftScenario Outline: SellingGiven there are <start> rubiesWhen i sell <sell> rubiesThen i should have <left> rubies

Examples:| start | sell | left || 12 | 5 | 7 || 20 | 5 | 15 |

Closing the gapWhile there are no Silver bullets by using tools like Cucumber, getting developers, designers, testers, business and the customer using the right words, we can help close the gap between the software that was written and the software the user actually wanted.


VitAL Focus Group Ad FP 1108.indd 1 17/10/08 15:25:18


There is unrest brewing in IT QA departments all over the globe. A frustration borne out of a lack of recognition within

their own organA major area of interest is the question of what data to use for development and testing, especially in offshore applications. Currently, most companies use copies of production data. This has obvious security issues but also has several disadvantages familiar to all CIOs. Copied data is usually out of date by the time it is used for testing making time specific tests irrelevant and new functionality will not have any pertinent data with which to test it. Multiple users will set up specific test scenarios which will be destroyed every time production is re-copied to testing. Large copies of production data on less powerful testing hardware make queries and searches run slowly and take up lots of expensive disk space.

In addition to full-sized copies, most companies will have additional approaches to building testing environments, which include: a small development database in which users create data by hand, this usually contains a large amount of invalid data; an extract subset of production data for use in development using tools such as GT Subset. They also use capture playback tools such as QTP, Forecast Studio etc to populate transactions using the

online applications and also use data generation tools such as Datamaker to build accurate test data.

At some stage during the development and testing lifecycle, users will access production data. Here we outline some of the simpler techniques to obfuscate or mask the

data where you cannot identify an original customer, account or secure entity from the masked data and the overall data trends cannot be easily identified.

Where to scrambleThe first consideration when designing a scrambling architecture is where you want to physically scramble the data. Is it good enough to copy production, move it into development, run a few scripts and off you go? I would put this into the ‘doing as little as possible’ category, as there are a few specific problems with this approach. For instance, the live data lives unscrambled in a development environment for a time; the scripts to scramble the data tend to get forgotten, are not kept updated and tend to be built by a single DBA who may move on and there is no traceability.

Better and more systematic approaches to data scrambling will depend upon your specific infrastructure. Many sites, for example, will already have copies of production data for use as reporting databases or for access by data warehousing toolset. These copy databases are protected by security layers and access control. The reporting databases can then be used as a source for scrambling extracts without impacting

Copied data is usually out of date by the time it is used for testing making time specific tests irrelevant and new functionality will not have any pertinent data with which to test it. Multiple users will set up specific test scenarios which will be destroyed every time production is re-copied to testing.

What data do you use for testing and development – especially when this crucial work is increasingly being off-shored? Huw Price of Grid-Tools says that the trio of masking, obfuscating and scrambling offer a practical solution.

The testers’ guide to masking, obfuscating and scrambling

30 | Test data


Test data | 31

on production performance. The main secure methods of

scrambling are:• Extract the data through scrambling

functions either on a live copy of production or preferably a reporting copy.

• Build a set of views that use scrambling functions to mask the data. Data will be extracted through these views. The access to these views can be granted using normal database security. As an indicator the initial data retrieval through the views is usually 5 to 10 times slower than against a native table.

• Create a secure environment and take a copy of production data, update the data in situ and when complete copy this to development. The same functions outlined in this paper can be used in update scripts to scramble the data in situ.

Ad hoc or systematic?It is very easy to write a few scripts, change a few customer names or alter a few characters of an account ID, but there are problems with this approach. The scripts tend to fall outside normal programming control and are written in SQL and non standard languages such as PERL. They may well be perfect but tend not to be documented, not incorporated in source control systems

and are not subject to testing by the test department.

Database structures tend to change regularly and the scrambling functionality needs to be upgraded with each release. After a while the scrambling routines tend to be forgotten. In an attempt to be more systematic, the use of tools can be helpful as well as turning the scrambling task into a normal IT project. The scrambling project would be subject to the infrastructure, testing and control used in your normal development lifecycle. The benefits of this are:

• Traceability – extremely important if data loss occurs.

• The scrambling tends to be more thorough and more useful to testing teams.

• Scrambling for new releases of software is automatically upgraded as part of the normal life cycle.

Know your dataBefore beginning a scrambling project you will need to understand your data. A request from management may be as simple as “make sure no one can recognise a customer”; however, understanding what a customer consists of is the first task before you can begin scrambling. To begin building up a picture, you will need to gather all of the available ‘free’ information

The first consideration when designing a scrambling architecture is where you want to physically scramble the data. Is it good enough to copy production, move it into development, run a few scripts and off you go?

March 09 | T.E.S.T


surrounding the data to be scrambled, this includes:• Foreign Keys. How are tables related

in the database?• Documentation. This is usually

held in a variety of formats and applications; however, they are rarely current.

• User knowledge. What is the user’s understanding of how and where key data is held and displayed?

• Naming standards. A surprisingly good source of information, column names in tables can give a strong hint to their use and relationship to other columns.

Once you have gathered a basic picture you will need to investigate the data itself to verify any documentation and try to understand in detail where the data is held and how it relates to other data.

There are a number of problems common across most systems, including data columns being used for multiple purposes. It is quite common for limitations in an application to be overcome by creative use of fields. Thus a field used for one purpose contains data to identify data for other uses. Examples of this type of usage are comment fields being used to hold structured information; these comments may contain data that is

sensitive for example a temporary address or phone number.

As applications and databases evolve and merge with other systems data may be created that is invalid. Users usually have an idea that this invalid data exists, however, but have made the decision to ignore the data problems as there is no critical problem that would justify the time to clean up the data. These data issues must be understood prior to scrambling a database.

Documentation and traceabilityIt should be obvious that the ability to demonstrate that efforts have been made to scramble data requires a documented trail. Turning the task into a normal IT project will allow you to use your normal change control, testing and delivery methodologies. These are usually very mature in most organisations. The documentation and control should include:• Which columns are sensitive and

need scrambling?• Who has access to any scrambling

functions, ie the code that scrambles should be protected as well.

• A before and after report of what the data has been changed from and to. You can use database compare

32 | Test data

It is very easy to write a few scripts, change a few customer names or alter a few characters of an account ID, but there are problems with this approach. The scripts tend to fall outside normal programming control and are written in SQL and non standard languages such as PERL. They may well be perfect but tend not to be documented, not incorporated in source control systems and are not subject to testing by the test department.


Test data | 33

tools such as Datamaker for this or generate triggers to update audit tables.

• Who has access to any working schemas or files used in the scrambling process?

Scrambling methodsThere are numerous methods used to scramble data, however I shall break them down into three categories:1 Simple independent functions to put

in random text, dates and numbers.

2 Multi-table column values, for example, an account number is used in lots of tables and as an identifier in other applications.

3 Offset values, for example, if a date is adjusted then other related dates must shift in line with the original date; if a post code is changed then corresponding address lines must also shift.

When building up your library of functions remember that there

are lots of powerful ones readily available from sources like database functions – every RDBMS comes with a vast library of built in functions many of which can be built up to scramble data quite easily; toolsets – Tools come with many pre built functions; your own code – Some of the scrambling you need will be very specific to your systems, for example, customer numbers can be built up of combinations of locations, dates of birth and partial names. There will be code in your system already that builds these numbers so use the same function as part of your scrambling strategy. And the internet provides a vast array of free code snippets which can be easily used.

Using seed tablesA very effective technique to scramble data is to use one or many static or temporary tables to hold data that can be included as part of your scrambling routines. These tables can include a list

of made up customer names, product names, addresses etc.

Static tables – It only takes a simple piece of SQL or, even better, a simple database function call to replace the data being extracted with data from the seed table. The seed tables contain data familiar to testers and can be added to very easily and be customized to specific locales. For example, it is very easy to create specific groups of addresses for each country. Seed tables should be populated prior to beginning the scrambling and verified that they contain no production data.

Dynamic tables – an easy to use and effective technique is to build tables that are exclusively used for a scrambling build. The next time data is extracted you would simply drop and recreate the tables. These tables tend to be used when data identifiers are used across multiple tables and a number must be changed to the same number across each of these tables.

These cross reference tables can be very useful as they ensure that even if someone knows an internal ID, they will not be able to find the specific detail of a customer. So, for example if you scramble customer names and addresses and you shift the internal customer ID field, the

Figure 1 – Datamaker stores and audits which scrambling functions have been applied to each column.


34 | Test data

data will still retain full integrity, however it will be difficult to identify a specific customer. This can also be used for detaching address IDs from Customer IDs allowing separating customer transaction details from address details; and detaching sign on information from personal information.

Independent functionsA library of simple functions to apply to data as it is extracted should be built up. It should include the adding of a small decimal increment to transaction values to mask individual transactions; and the adding of a number of days to all dates, a very simple transformation to implement, assuming all your dates are identified as date data type. This also has the advantage of allowing time-dependant process testing to be more accurate.

Bear in mind end of month processing can be affected by this. You may be better off using a cross reference table to match up periods.

Multi-table column valuesMany column values are repeated in tables across the system. These values can be external identifiers, for example, an account number may be used extensively across a system and across other applications, or they could be internal identifiers for example an account ID. While changing external values is obvious,

changing the internal identifiers should be considered. If a user can identify an internal ID, sometimes they are displayed in reports, XML messages, error messages etc then data is no longer masked.

The first problem is tracking down all of the links between tables and columns once this is done you need to make sure that the same scramble functions are applied to all of the related columns. Identifying these columns may need a tree walk across your model.

Functions that are used in multi-column scrambling should include:

• A simple character by character replacement. Basically shift character five and six in a string identifier to one more less and one more respectively.

• Hashing, which is a key component in multi-column replacement. It allows values to be transformed to the same value every time dependant on a hash key. For example, 1 would be transformed to 7, 2 to 6, 3 to 1 etc. Each value has a unique hashed value and is repeatable based on the hash key.

• Using dynamic seed tables will build an exact replacement value for each identifier. You need to protect the seed table as it contains the ‘key to crack the code’ and you must also protect the offset algorithm, as this can be used to identify data.

Using the above techniques, it is possible to apply incremental updates to your development environment. So, for example, if you extract sub sets of production transaction data to keep

Figure 2 – A tree walk using Datamaker to identify internal IDs


Test data | 35

Huw Price DirectorGrid-Tools ltd www.grid-tools.com

your development data up-to-date, you can run an extract and import the new data which will still makes referential sense. You must, however from time to time change any offset values, hash keys or simple algorithms and refresh the test databases as users will eventually work out the old and new values.

Offset valuesManaging offset values can be more complicated as data in one column/table must be related to the other column/table’s values for the system to operate in the same way as production. Obviously understanding where these types of relationships exist is crucial. The types of offset values include:• Micro aggregation – The values of

prior rows in a set of transactions refer to each other.

• Application process driven aggregations – Many systems have application components that calculate balances based on transaction throughput. These tend to be separate processes which can be run as stand alones. For example, if you are adjusting transaction amounts the customer balances may not match. Running the balance adjustment process may be required to reset these values.

• Offset dates – If you adjust dates then other values may well have to be reset. An example of this is dates of birth; adjusting this by a few days will alter the age and also the age bracket so a person may move into a different insurance premium.

The aggregation and setting of offset values can be the hardest part of any scrambling project. In my experience, the extensive use of dynamic tables

beforehand, to create offset lookups is an effective technique, especially with date-type data. In addition, the ability to use portions of application code may need to be incorporated into the scrambling routines to reset end of day balances, customer totals etc.

Data creationMany organisations have very strict data usage restrictions and may well need to create data from scratch to use in their development and testing environments. Having no access to production data is common in many government departments as well as more and more health care and financial sectors. At first, creating data from scratch may seem like a difficult task, however when you consider some of the difficulties with scrambling and the basic problem of developing with full size databases that are often out of date it may well be an easier option.

The basic procedures to do this are:

What a scrambling project should include• Understand your data – Its peculiarities, quality issues and interfaces.• Agree up front what data is sensitive – Don’t forget to include internal IDs.• Discuss the different methods and how they apply to the application as a whole and in

detail at the column level.• Build up a library of functions – Consider reusing some of your existing code which may

handle non standard situations.• Consider buying in a toolset to scramble data.• Consider creating data from scratch – It is often easier than you think and has many

benefits in terms of the amount of code covered in testing.• Document and audit everything you are doing.• Use normal project control mechanisms.• Don’t forget that the scrambling has to be kept up to date – design your scrambling

application with this in mind.

• Create a standard data object, for example a Customer.

• Tokenise or parameterise those objects so that you can vary them when you create them.

• Inherit objects so that you can make your own edits without affecting the original data objects.

Library of functionsOnce you have completed your analysis and built a library of standard functions you can use these functions across the enterprise to scramble data in other systems and linked applications.

For each table in the application you must produce and store some level of documentation to verify the scrambling is effective and sufficient. Spend some time looking at each column deciding on which function or transformation should be applied to each column. You must also note any pre processing required to build cross reference lookups and any post-processing balance aggregations etc.

New builds and releasesAn often forgotten problem with scrambling is that minor and major new releases and builds may come thick and fast so the scrambling toolset must be able to handle the addition of new, possibly sensitive columns, the addition of cross column relationships and offset values and any new functional requirements which may need new seed tables.

It is important that the scrambling methods are updatable quickly and you do not have to spend time waiting, for example, for a DBA to update some scripts. If this happens the initial work will be quickly lost and production data will begin being used again in development and testing.

Scrambling projects often start out as a request to the DBA team to “do something to make sure production data is not being used in development”. While the DBAs may be able to do a quick job to scramble some columns, the responsibility for data ownership cannot live with the DBA team. A scrambling project should come under normal application control and be the responsibility of the development team in conjunction with the application ‘data owners’. Too many projects have failed to deliver even basic data security by going down this route.

Before beginning a scrambling project you will need to understand your data. A request from management may be as simple as “make sure no one can recognise a customer”; however, understanding what a customer consists of is the first task before you can begin scrambling.


36 | Testing security

Brian Chess, co-founder and chief scientist of Fortify Software and his colleagues Gary McGraw and Sammy Migues interviewed executives at some of the world’s top software security programmes in 2008. He says discussions were truly enlightening and is now in the process of using the data collected to build a maturity model. However, one insight was just too good to wait…

Penetration testing is dead, long live penetration testing


Testing security | 37

I believe it to be universally true that if you’re not paying attention to security, then you have security problems and for this very reason penetration testing has been able to firmly establish its position in software security.

Why the need to evolve?I believe it to be universally true that if you’re not paying attention to security, then you have security problems and for this very reason penetration testing has been able to firmly establish its position in software security. Historically, many organisations have written code that they recognise will be insecure and, once complete, their first action is to deploy penetration testing to prove this premise, paying for the privilege! Am I the only one to see the futility of this exercise? Not anymore.

I was not surprised that every one of the programmes we talked to back in 2008 practiced penetration testing (most using external firms) at one time or another. After all, there's nothing like a smoking hot pen testing report to get an organisation to admit it has a problem. However, as activities earlier in the SDLC are adopted, penetration testing begins to lose some of its lustre. We found evidence that the role of penetration testing lessens (but does not go to zero) as an organisation gets a handle on the software security problem. So here’s the eureka moment: as organisations get better at software security in the first instance, so their emphasis on penetration testing to look for the holes that they know exist diminishes.

Pen testing will get wrapped into a much larger and far more

comprehensive approach to improving security. The best initiatives balance the yin and the yang of attack and defence.

2008 saw us pass an inflection point.People are now spending more money on getting code right in the first place than they are on proving it’s wrong. However, this doesn’t signal the end of the road for penetration testing, nor should it, but it does change things. Rather than being a standalone ‘product’, it’s going to be more like a product feature. Penetration testing is going to cease being an end unto itself and re-emerge as part of a more comprehensive security solution.

This kind of thing happens all the time in high-tech. The first PC spell checkers were standalone programs, but the market for stand-alone spell checkers died when they became a standard part of any word processor. These days spell checkers are everywhere (even in my IDE), but there is no market for a standalone spell checker. Proof positive: there aren’t even any Web 2.0 or iPhone spell checker startups!

So why now?Alright, so why 2009? The time is right because back in 2007, IBM bought a company named WatchFire and HP bought a company named SPI Dynamics. The acquired

March 09 | T.E.S.T


38 | Testing security

companies both made web application penetration testing products. IBM and HP spent serious money for these companies – not crazy dotcom prices, but even at HP and IBM you have to tell a good story before you get to spend upwards of seventy million dollars. The good story was that the acquired technology would work together with other products and services to fuel a broad entrée into a rapidly-growing software security market. It takes a little while to digest any acquisition, but by now it’s been long enough. 2009 will be the year this strategy comes together, and when we look back, it will be the year when most of the world began thinking about penetration testing as part of a larger offering.

There will always be boutique security consulting companies with funny names and exotic services, but the industry will grow by integrating security yin and yang. If you’d like

a sneak preview of what the future holds, check out the work White Hat Security has done to integrate their vulnerability measurement service with Web application firewalls. This is attack and defence working together in a creative new way.

Evolve or dieMore than ever before, people understand the software security challenge, and penetration testing deserves credit for helping spread the word. But knowing a security problem exists is not the same as knowing how to fix it. In other words, penetration testing is good for finding the problem but doesn’t help in finding the solution – and that’s why it must take a long hard look at itself and then make a change. Just like the venerable spell checker, it’s going to die and come back in a less distinct but more pervasive form and I, for one, can’t wait.

People are now spending more money on getting code right in the first place than they are on proving it’s wrong. However, this doesn’t signal the end of the road for penetration testing, nor should it, but it does change things. Rather than being a standalone ‘product’, it’s going to be more like a product feature. Penetration testing is going to cease being an end unto itself and re-emerge as part of a more comprehensive security solution.

Brian Chess Chief Scientist Fortify Software www.fortify.com


Supplier profile | 39

A stalwart supplier to the testing industry with global coverage, Seapine Software promises to help its customers reduce complexity allowing them to focus on their core business, producing quality software. T.E.S.T magazine spoke to company founder Rick Riccetti.

Quality assured


40 | Supplier profile

After fourteen years in the business of supplying solutions and training to software testers, Cincinnati, Ohio-based Seapine

Software can lay claim to being one of the pioneers in this sector. The company specialises in helping companies identify the problems they have developing quality products and implementing cost-effective solutions to those challenges. These solutions can come in the forms of consulting or tools, or a combination of both.

T.E.S.T: What are the origins of the company; how did it start and develop; how has it grown and how is it structured?

Rick Riccetti: Seapine Software Inc is a privately held corporation headquartered in Cincinnati, Ohio in the USA. My wife, Kelly, and I founded Seapine in 1995, and operated the company out of our home for the first three years. Seapine’s mission at the time was to reduce the complexity of software development tools, allowing developers and quality assurance

professionals more time to focus on their task at hand—producing quality software. That remains Seapine’s mission today.

Seapine’s first product was TestTrack for the Apple Macintosh platform, which was introduced in 1996. We followed this with a Windows version in 1997 and a web browser version in 1998. From there we moved to an office and began to grow the company.

In 2002, we began to expand our product portfolio by developing and releasing Surround SCM, a software change management tool, and acquiring QA Wizard, an automated testing tool. In 2006, we released TestTrack TCM, a test case management tool developed internally. In 2007, we released QA Wizard Pro, a more robust functional and regression testing tool than the original QA Wizard.

Seapine has grown from two employees to over 100, and has added offices in London, Munich, and Melbourne. With all of this growth, the company has remained faithful to its

Seapine’s mission at the time was to reduce the complexity of software development tools, allowing developers and quality assurance professionals more time to focus on their task at hand-producing quality software. That remains Seapine’s mission today.


Supplier profile | 41

internal culture and a philosophy of exceeding customer expectations.

T.E.S.T: What range of products and services does the company offer?

RR: As an application lifecycle management (ALM) solution provider, Seapine offers many tools to support the application development lifecycle. Specifically, we offer QA Wizard Pro for automated functional and regression testing, Surround SCM for software change management, TestTrack Pro for issue and development process management, and TestTrack TCM for test case planning and management. We also offer multiple levels of training for all products, custom software development, data conversion, and process consulting services.

Integration is a key feature of our products. TestTrack Pro and Surround SCM support linking issues and change requests with source code changes for improved traceability of software changes. QA Wizard Pro integrates with TestTrack Pro to automate pushing test failures into users’ defect tracking workflow. QA Wizard Pro also integrates with TestTrack TCM so users can link test scripts with TestTrack TCM tests cases and automate running tests. TestTrack Pro and TestTrack TCM are integrated in a single application so defects can be traced to the test that generated them. Defects can also be promoted to new test cases to help improve test coverage.

Surround SCM and the TestTrack products are cross-platform client/server applications, running on Windows, Mac OS X, Linux, and Solaris. All products support industry-standard databases for data storage. We have also embraced open standards where possible to make it easier for customers and our services team to extend our products.

Finally, we offer the Quality-Ready Assessment (QRA). This online assessment helps companies measure their product development capabilities in the four keys areas of testing, change control, tracking, and automation. After completing the assessment, the survey results and recommendations for improving any noted deficiencies are emailed to the respondent within a few minutes.

T.E.S.T: Does the company have any specialisations within the software testing industry?

RR: We specialise in helping companies identify their deficiencies as they relate to developing quality products and implement cost-effective solutions to address those deficiencies. These solutions can come in the forms of consulting or tools, or a combination of both.

More specifically, Seapine helps quality assurance organisations create, enforce, automate, track, and measure QA processes. Beginning with

Seapine helps quality assurance organisations create, enforce, automate, track, and measure QA processes. Beginning with TestTrack TCM, we give companies a specialised tool to help them manage the thousands of test cases needed to test today’s complex applications.

TestTrack TCM, we give companies a specialised tool to help them manage the thousands of test cases needed to test today’s complex applications. With TestTrack Pro, companies have a solution for managing issues found during software testing to ensure they are resolved or at least considered in future builds. QA Wizard Pro is a platform for automating functional and regression testing and we can bring expertise to companies to help them implement a formal automated testing programme.

T.E.S.T: Who are the company’s main customers today and in the future?

RR: Financial services, health care, computer/console gaming, aerospace/defence, and government customers produce software that is held to a very high standard. Specific needs of these customers include compliance auditing, traceability, transparency, risk management, and addressing the challenges of geographically distributed teams. The feature sets of our solutions directly address the needs of these customers, and by extension, customers in less quality-demanding industries. For example, TestTrack Pro has features that help companies automate workflows, enforce processes, and produce detailed audit trails for compliance audits. The cross-platform client/server architecture uses encrypted data communications to provide secure connectivity between geographically dispersed teams.

To measure whether Seapine and a prospective customer are a good fit, we ask how important quality is to the organisation. If quality is critical or management-driven, then we are a great fit for the organisation. However, if quality is not that important to the prospect, then it is difficult to sell them on the virtues of process


42 | Supplier profile

control, traceability, change control, etc. Fortunately, more companies are becoming aware of the impact of quality on development cost and delivery time, and the link between quality and customer satisfaction, which drives future sales.

T.E.S.T: What is your view of the current state of the testing industry and how will the recent global economic turbulence affect it? What are the challenges and the opportunities?

RR: The testing industry (or at least the demand for testing tools) remains strong; however, the vendor choices are a little less clear for companies than in the past. Prior market leaders of QA tools have been absorbed into larger companies and some products have either been deprecated or have taken a less prominent role in the acquiring companies’ portfolios. That creates an opportunity for companies like Seapine that remain independent and focused exclusively on the quality problems of our customers.

Economically turbulent times are a wakeup call for companies to improve efficiencies across the organisation. For software development companies and IT teams, test automation can have the biggest impact in helping companies accomplish more with their QA resources. So there is a real opportunity to introduce automated testing to companies. However, these tools have a higher cost-per-seat than many other tools and cash flow is being watched closely. The challenge (and opportunity) is to demonstrate to companies how quickly an ROI can be realised on the automated testing investment.

T.E.S.T: What are the future plans for the business?

RR: We plan to keep the focus on helping companies deliver quality software products, while reducing costs. Providing integrated, cost-effective products and services will remain our competitive advantage.We will continue to expand the capabilities of our ALM solution by adding innovative tools to our product line while providing complementary services that help companies improve their processes as well as get the most out of their software tool investment.

By opening additional offices within and outside of North America, we will continue to expand our ability to provide the best possible purchasing,

We know there are many, many more opportunities to help our customers with the software development and quality assurance challenges they face, and so our journey as a company is always closer to the beginning than the end.

support, and services experience regardless of a customer’s location. Maintaining high customer satisfaction rates will continue to dominate our culture.

Finally, we will continue to expand the number of development tools that our tools integrate with and provide a rich set of open interfaces into our products and data stores.

T.E.S.T: How do you see the future of the software testing sector developing in the medium and longer terms?

RR: We expect to see more integration between tools, better support for developer testing, improvements in test planning tools, and new tools to help report on quality assurance.

More integration between tools will improve testing efficiency by moving data automatically between QA and development. QA tools will also be better integrated into integrated development environments, making them more accessible to developers.

Development environments will likely include unit testing tools as a standard offering. Security testing as a part of both unit testing and system testing will become standard for the developer as well.

Test planning tools are relatively new and will become more adept at creating test cases from requirements,

capturing and managing automation scripts, and helping QA teams schedule and deploy resources, both people and software, more efficiently than ever. Techniques that improve test efficiency, such as all-pairs testing, will become standard features of these tools.

Finally, dashboard-like products will become more commonplace in QA, pulling from multiple data sources to provide the QA manager and senior managers with instant visibility into the state of the testing effort.

T.E.S.T: Is there anything else you would like to add?

RR: The opportunity to help companies deliver better products to their customers is what drives our organisation. The feedback we get from our customers says we are materially impacting their ability to deliver quality products more efficiently. We know there are many, many more opportunities to help our customers with the software development and quality assurance challenges they face, and so our journey as a company is always closer to the beginning than the end.

T.E.S.T: Thank you very much.

www.seapine.com


T.E.S.T company profile | 43

ISEB is part of the British Computer Society (BCS) and is a leading worldwide exam body for IT professionals.

With over 380,000 exams delivered worldwide in over 50 countries, including Australia, Japan, South Africa, USA, Brazil and many others, ISEB continues to lead the way in exams for IT professionals.

Our qualifications are internationally recognised and cover eight major subject areas in: Software Testing, ITIL/ IT Service Management, IT Assets and Infrastructure, Systems Development, Business Analysis, Project Management, Information and Security and IT Governance.

These are available at Foundation, Practitioner and Higher Level to suit each individual candidate. ISEB Professional Level is also available. For more information visit www.iseb-exams.com.

These qualifications allow candidates to learn new skills in specific business and IT areas which measure competence, ability and performance. This helps to promote career development and provide a competitive edge for employees.

Delivered via a network of accredited training and examinations providers, the breadth and depth of ISEB’s portfolio encourages knowledge, understanding and application in various disciplines.

BCS

BCS is the leading professional body for those working in IT and communications. Established in 1957and now with over 66,000 members in more than 100 countries, BCS aims to lead the development of the IT profession and make IT the profession of the 21st century.

BCS is the industry body for IT Professionals and Chartered Engineering Institute for IT. We are responsible for setting standards for the IT profession. Our vision is to see the IT profession recognised as being a profession of the highest integrity and competence.

BCS membership for software testers

BCS membership gives you an important edge; it shows you are serious about your career in IT and are committed to your own professional development, confirming your status as an IT practitioner of the highest integrity.

Our growing range of services and benefits are designed to be directly relevant at every stage of your career.

Industry recognition

Post-nominals – AMBCS, MBCS, FBCS & CITP - are recognised worldwide, giving you industry status and setting you apart from your peers.

BCS received its Royal Charter in 1984 and is currently the only awarding body for Chartered IT Professional (CITP) status, also offering a route to related Chartered registrations, CEng and CSci.

Membership grades

Professional membership (MBCS) is our main professional entry grade and the route to Chartered (CITP) status. Professional membership is for competent IT practitioners who typically have five or more years of IT work experience. Relevant qualifications, eg a computing-related degree, reduce this requirement to two or three years of experience. Associate membership (AMBCS) is available for those just beginning their career in IT, requiring just one year’s experience.

Joining is straightforward – for more information visit: www.bcs.org/membership where you can apply online or download an application form.

Best practice

By signing up to our Code of Conduct and Code of Good Practice, you declare your concern for public interest and your commitment to keeping pace with the increasing expectations and requirements of your profession.

Networking opportunities

Our 44 branches, 16 international sections and over 40 specialist groups including Software Testing (SIGIST) and Methods & Tools, provide access to a wealth of experience and expertise. These unrivalled networking opportunities help you to keep abreast of current developments, discuss topical issues and make useful contacts.

Specialist Group in Software Testing (SIGIST)

With over 2,500 members SIGIST is the largest specialist group in the BCS. Objectives of the group include promoting the importance of software testing, developing the awareness of the industry’s best practice and promoting and developing high standards and professionalism in software testing. For more information please visit: www.sigist.org.uk.

Information services

The BCS online library is another invaluable resource for IT professionals, comprising over 200 e-books plus Forrester reports and EBSCO databases.

BCS members also receive a 20 percent discount on all BCS book publications. This includes Software Testing, an ISEB Foundation. As well as explaining the basic steps of the testing process and how to perform effective tests, this book provides an overview of different techniques, both dynamic and static, and how to apply them.

Career development

A host of career development tools are available through BCS including full access to SFIA (the Skills Framework for the Information Age) which details the necessary skills and training required to progress your career.

ISEB

BCS, First Floor, Block D, North Star House, North Star Avenue, Swindon, SN2 1FA, United KingdomTel: +44 (0) 1793 417655 Fax: +44 (0) 1793 417559 Email: [email protected] Web: www.iseb-exams.com


44 | T.E.S.T company profile

Leysen has specialised in software and system testing recruitment for over 13 years. One of the first recruitment companies dedicated to the testing and QA market, we are still one of the market leaders.

Testing RecruitmentThe Team – Our extensive experience in this niche market, allows us to keep one step ahead of the competition. Over time we have built excellent relationships with both our clients and candidates. This means we can efficiently assess and understand your requirement, and either through the database, or by using our network of contacts, quickly respond to your needs. Forty percent of our placements last year were people that were not actively looking for a new role ie, their CVs were not active in the market, but matched with suitable roles by our team.

Our teams keep up to date with current developments by attending testing conferences and have had an involvement with the running of the BCS SIGIST for the last ten years.

Whatever the extent of your testing requirement, leysen can help –Whether you require a permanent individual to carry out system testing, or a complete team of contractors, call our resource team now on 01483 211888 or email your vacancy detail to [email protected]

Our database boasts –• Over 15,000 pre-screened testing candidates, updated and increasing daily;• Candidates with a minimum of two years’ recent solid testing experience, the

majority with more experience; • A significant number of candidates who are ISEB qualified in the Foundation,

Intermediate or Practitioner certificates in software testing;• Candidates with a wide and varied range of skills and abilities to suit your needs;• Candidates with experience in testing on all platforms and testing tools.

Clients –As you can imagine our client list is extensive. We have placed candidates and built teams of testers across the UK, Europe and in the US from multinational blue chips to local software houses.

Testing training –In addition to our main business of recruitment, Leysen believes in promoting testing as a profession and getting testers qualified to ISEB/ISTQB standard. We offer training for Foundation, Intermediate or Practitioner level. This training is either CDROM self-study, online or classroom-based.

The self-study CDROMs for Foundation and intermediate are available at £150* and £195* respectively (*plus £2.50P&P, plus VAT). For free evaluation please see: www.leysen.com/selfstudyiseb.asp

Email: [email protected] Web: www.leysen.com Tel: +44 (0) 1483 211888 Fax: +44 (0) 01483 211887

Leysen Associates

31 Media is a business to business media company that publishes high quality magazines and organises dynamic events across various market sectors. As a young, vibrant, and forward thinking company we are flexible, proactive, and responsive to our customers' needs.

VitAL MagazineVitAL is a journal for directors and senior managers who are concerned about the business issues surrounding the implementation of IT and the impact it has on their customers. Today senior management are starting to realise that implementing IT effectively has a positive impact on the internal and external customer and it also influences profitability. VitAL magazine was launched to help ease the process.

VitAL Focus GroupsThe VitAL Focus Groups are a specifically designed programme of Focus Groups that bring together senior decision makers for a series of well thought-out debates. In our role as information provider we spend a lot of time speaking and listening to our customers and then seeking out innovative ways to meet their needs. It has become apparent that senior decision makers wish to discuss their challenges in a structured manner with a view to finding pragmatic and workable solutions to what are invariably complex issues. So in September 2009 VitAL Magazine will be launching the inaugural VitAL Focus Groups.

Customer MagazineCustomer Magazine was launched to address and assist with the various challenges senior professionals face when establishing a customer-centric business. Our editorial takes a pragmatic approach to what has become a series of complex issues and delivers dynamic, provocative, and insightful articles, case studies, opinion pieces, and news stories that not only challenge our readers but also bring clarity and vision to the many challenges they face.

Customer Focus GroupsThe Customer Focus Groups are a specifically designed programme of Focus Groups that bring together senior decision makers for a series of well thought-out debates. In our role as information provider we spend a lot of time speaking and listening to our customers and then seeking out innovative ways to meet their needs. It has become apparent that senior decision makers wish to discuss their challenges in a structured manner with a view to finding pragmatic and workable solutions to what are invariably complex issues. So in November 2009 Customer Magazine will be launching the inaugural

Customer Focus Groups.

31 Media www.31media.co.uk [email protected] Crawley Business Centre, Stephenson Way, Crawley, West Sussex, RH10 1TN, United Kingdom

Phone: +44 (0) 870 863 6930 Fax: +44 (0) 870 085 8837

31 Media

INSPIRING CUSTOMER CENTRICITY



F O C U S G R O U P S




T.E.S.T company profile | 45

With over 8,500 customers worldwide, Seapine Software Inc is a recognised, award-winning, leading provider of quality-centric application lifecycle management solutions. With headquarters in Cincinnati, Ohio and offices in London, Melbourne and Munich, Seapine is uniquely positioned to directly provide sales, support, and services around the world. All customers receive the same consistently high level of responsiveness and customer care.

Built on flexible architectures using open standards, Seapine Software’s cross-platform ALM tools support industry best practices, integrate into all popular development environments and run on Microsoft Windows®, Linux®, Sun Solaris®, and Apple Macintosh® platforms.

Seapine Software's integrated software development and testing tools streamline your development and QA processes – improving quality, and saving you significant time and money.

TestTrack TCM

A scalable, cross-platform test case management solution, manages all areas of the software testing process including test case creation, scheduling, execution, measurement, and reporting. TestTrack TCM's powerful workflow and extensive customisation capabilities make it easy to bring traceability to the testing process. Reporting and graphing capabilities, along with user-definable data filters, give you the tools you need to easily measure the progress and quality of your testing effort.

TestTrack Pro

A powerful, configurable, and easy to use issue management solution. Its timesaving communication and reporting features keep team members informed and on schedule. TestTrack Pro supports MS SQL Server, Oracle, and other ODBC databases, and its open interface is easy to integrate into your development and customer support processes. Cross-platform client and server support provides full functionality for Windows, Mac OS X, Linux, and Solaris users, and enables hosting on your preferred operating system.

QA Wizard Pro

Automates the functional and regressions testing of Web, Windows, and Java applications, helping quality assurance teams increase test coverage and deliver high quality solutions faster. Featuring a next-generation scripting language and an easy-to-use Grid View, QA Wizard Pro includes advanced object searching, smart matching a global application repository, data-driven testing support, validation checkpoints, and built-in debugging. It also includes batch file support, a real-time status tool, and remote execution support.

Surround SCM

Controls access to source code files and other development assets, and tracks changes over time. All data is stored in industry-standard relational database management systems for greater security, scalability, data management, and reporting. Surround SCM’s change automation, caching proxy server, custom metadata, labels, and virtual branching streamline parallel development and provide complete control over the software change process.

www.seapine.com

United Kingdom, ireland, and Benelux: Seapine Software Ltd. Building 3, Chiswick Park, 566 Chiswick High Road, Chiswick, London, W4 5YA UKPhone:+44 (0) 208-899-6775 Email: [email protected]

Americas (Corporate Headquarters): Seapine Software, Inc. 5412 Courseview Drive, Suite 200, Mason, Ohio 45040 USA Phone: 513-754-1655

Seapine SoftwareTM


46 | T.E.S.T company profile

Network emulation & application testing toolsiTrinegy is Europe’s leading producer of network emulator technology which enables testers and QA specialists to conduct realistic pre-deployment testing in order to confirm that an application is going to behave satisfactorily when placed in the final production network.

Delivering more realistic testingIncreasingly, applications are being delivered over wide area networks (WANs), wireless LANs (WLAN), GPRS, 3G, satellite networks etc, where network characteristics such as bandwidth, latency, jitter and packet error or loss can have a big impact on their performance. So, there is a growing need to test software in these environments.

iTrinegy Network Emulators enable you to quickly and easily recreate a wide range of network environments for testing applications, including VoIP, in the test lab or even at your desktop.

Ease of useOur network emulators have been developed for ease of use:

• No need to be a network expert in order to use them• Pre-supplied with an extensive range of predefined test network scenarios to get you started • Easy to create your own custom test scenarios • All test scenarios can be saved for subsequent reuse• Automated changes in network conditions can be applied to reflect the real world• Work seamlessly with load generation and performance tools to further enhance software testing.

A comprehensive range to suit your needsiTrinegy’s comprehensive range of network emulators is designed to suit your needs and budget. It includes:

• Software for installation on your own desktop or laptop (trial copies available)• Small, portable inline emulators that sit silently on the desktop and can be shared amongst the test team• Larger portable units capable of easily recreating complex multi-path, multi-site, multi-user networks for full enterprise testing• High performance rack-mount units designed to be installed in dedicated test labs• Very high performance units capable of replicating high speed, high volume networks making them ideal for testing

applications in converged environments.

If you would like more information on how our technology can help you ensure the software you are testing is ‘WAN-ready’

and going to work in the field, please contact iTrinegy using the details below:

Email: [email protected] Tel: +44 (0)1799 543 345 Web: www.itrinegy.com

iTrinegy

SOA Quality as a continuous processParasoft empowers organisations to deliver better business applications faster. We achieve this by enabling quality as a continuous process across the SDLC–not just QA. Our solutions promote strong code foundations, solid functional components and robust business processes. Parasoft's SOA solution provides an automated infrastructure that enables SOA quality as a continuous process, allowing you to reap the full benefits of your SOA initiative.

Error preventionParasoft's SOA solution allows you to discover and augment expectations around design/development policy and test case creation. These defined policies are automatically enforced, allowing your development team to prevent errors instead of finding and fixing them later in the cycle. This significantly increases team productivity and consistency.

Continuous regression testingParasoft's SOA solution assists you in managing the complex and distributed nature of SOA. Given that your SOA is more than likely to span multiple applications, departments, organisations and business partners, this requires a component-based testing strategy. With the Parasoft solution set, you can execute a component-based testing strategy that ultimately allows you to focus on the impact of change. Parasoft's continuous regression tests are applied to the multiple layers throughout your system. These tests will then immediately alert you when modifications impact application behaviour providing a safety net that reduces the risk of change and enables rapid and agile responses to business demands.

Functional auditParasoft's continuous quality practices promote the reuse of test assets as building blocks to streamline the validation of end-to-end business scenarios impacted by changing business requirements. Functional test suites can be leveraged into load tests without the use of scripting, allowing you to track performance metrics throughout the SDLC. This enables your team to execute a more complete audit of your business application, reduces the risk of business downtime, and ensures business continuity.

Process visibility and controlParasoft's SOA solution enforces established quality criteria and policies, such as security, interoperability, and maintainability, on the various business application artefacts within an SOA. Adherence to such policies is critical to achieving consistency as well as ensuring reuse and interoperability. As a result, you evolve a visible and sustainable quality process that delivers predictable outcomes.

Please contact us to arrange either a one-to-one briefing session or a free evaluation.

Web: www.parasoft.com Email: [email protected] Tel: +44 (0) 208 263 6005 Fax: +44 (0) 208 263 6100

Parasoft


48 | The Last Word

Cloud computing is the latest buzz phrase in the industry being pushed by many including computing HP, IBM and Sun. Is it simply marketing hot air to drive sales, or something with a little more substance? Itrinegy’s Frank Puranik looks for the truth in the clouds.

Testing with your head in the clouds

The cloud is the Internet and cloud computing refers to accessing applications over the internet in a ‘pay as you

use’ model with web browsers or thin clients. It’s not new, it’s just ASP (Application Service Provision), which was itself reinvented as SaaS (Software as a Service). Third time lucky?

Cloud computing definitely includes SaaS products (salesforce.com, Google apps etc) but also takes in IaaS (Infrastructure as a Service – delivery of computer infrastructure, possibly virtualised, as a service) and PaaS (Platform as a Service – develop, test, deploy, host and maintain applications in the same integrated development environment as a service).

The first of these two (IaaS) seems like an implication of SaaS though you can also use IaaS to deliver on your own developed software in-house or out-house. PaaS goes further by providing you with services to develop, test, deploy, host and maintain applications in the same integrated development environment – a sort of total DIY SaaS toolkit.

No one completely agrees on these definitions, partly because they're evolving and partly because as they become ‘hot’ and drive the product sales bandwagon the terms become stretched and diluted. What you can take away from all this though is that cloud computing implies Internet network delivery.

So, does it matter to you and how will it affect the way you test? It feels like cloud computing should succeed and become ubiquitous, but it felt like that with ASP and SaaS. Certainly

some companies like salesforce.com are very successful. In general though, the appeal has been more limited in enterprise companies, while smaller SMBs have shown a stronger adoption. Perhaps it’s a matter of trusting other people with your data; enterprises are used to protecting their data themselves after all.

On the face of it, many of us have accepted cloud computing, with Facebook, Twitter etc being very popular and many using Webmail. These are connected applications, so that’s natural, but I don’t know a single person who uses Google Docs (Google apps) in preference to MS Office or OpenOffice.

It will matter to you if your organisation is going to embrace cloud computing, which it will if you work for a software house that wants to provide their applications as a service, or a company that wants to move to a third-party infrastructure provider (IaaS) or take advantage of PaaS to reduce costs.

Everything you know about testing still applies in cloud computing, but now your network environment during the testing will be critical. Your cloud application will be delivered over the Internet, not your LAN or WAN and the end users could be anywhere, at work, at home, abroad, on 3G, even on public WiFi.

The quality of the Internet varies greatly, from reasonable to very poor bandwidth and good to very slow network response times. So your applications will need to be able to function in adverse conditions, and what were considered extreme

network conditions in the corporate network will now just be normal. Also, there is a user expectation of richer, more interactive clients, eg Web 2.0, which don’t just send data when you press ‘submit’. These may be communicating with the server all the time and so it’s clear that assumptions about the ‘network lightness’ of browser-based clients (without an obvious ‘fat’ Java or ActiveX functionality) no longer holds and we cannot ignore the ‘network effect’.

You also shouldn’t restrict testing in the network to just the performance testing phase alone, as the network impact, even on a single client functional test may deem the application unusable. The problem is often seen as: “That’s all very well but how do we put the Internet into the test environment in any meaningful, repeatable manor?” The answer to this lies in the use of network emulators which will give you poor Internet experience repeatably and on-demand.It’s not just third time lucky, the first two goes never really died, they just morphed into the next. ASP was launched to an enormous fanfare and overpromised, but slowly, successful applications like salesforce.com have emerged. The power and scalability of the infrastructure is far better today, and our organisations don’t really want to provision, install, maintain and backup applications (provided they and their data are kept secure) that are not of the competitive advantage type. So as a tester it’s worthwhile having an awareness of cloud computing as at some point it may impact your application testing.

Frank Puranik Product directoritrinegywww.itrinegy.com

the last word...

T.E.S.T | March 09T.E.S.T | March 09■

ViTAL Sub FP 0908.indd 1 26/8/08 16:56:31

ACCREDITED BY

GLOBALLY QUALIFIED?

VISIT US ON STAND 536

AT THE SERVICE DESK

AND IT SUPPORT SHOW

With ISEB your qualifications will be recognisedin over 50 countries worldwide.

ISEB is part of the British Computer Society (BCS) and is a worldwide exam body. With over 380,000 exams delivered worldwide in over 50 countries, ISEB continue to lead the way in exams for IT professionals.

Our qualifications allow candidates to learn new skills in specific business and IT areas which measure competence, ability and performance. This helps to promote career development and provide a competitive edge for employees.

For further information: www.iseb-exams.com

BUSINESS ANALYSIS

PROJECT MANAGEMENT

INFORMATION AND SECURITY

IT GOVERNANCE

ITIL/IT SERVICE MANAGEMENT

IT ASSETS AND INFRASTRUCTURE

SOFTWARE TESTING

SYSTEMS DEVELOPMENT

ToPDF:Layout 1 04/03/2009 10:47 Page 1

test magazine - march-april 2009

Documents

t magazine

t h t e c h n o

regular issue of t

n s o f t w

o g yt h e e u r o p

armsi n t o u c h w

water issue

seapine software