Quality assurance & testing

Karel Kravik

Who am I?

� Tartu University� Applied Informatics

� Assistant lecturer “Software Project”

� Webmedia 2000-2007� Programmer

� Analyst/project manager

� Product manager for Changelogic (~4 years)

� Quality manager (~2 years)

� Playtech 2007-...� Integration manager

What I will be talking about?

� Unpredictability of bugs

� Bugs and their effects

� Testing methods & aspects

� Test automation

� Test plans and reports� Test plans and reports

� Bug tracking

� Testing as a career

What one must know about software quality & bugs?software quality & bugs?

What is a software bug?

� Nonconformance to specification� Software doesn’t do something that is required by specification

� Software does something that is prohibited by specification

� Software does something not mentioned in � Software does something not mentioned in specification

� Software does not meet end user needs� Slow, not usable, etc

Nontrivial software has bugs

� You don’t know what nor where the bugs are

� You can’t tell there are no bugs after testing:� You can observe many times when the software behaves right

� You cannot conclude that the software behaves right alwaysright always

� Bug are the software reality� Client: “Please confirm that we don’t make any more bugs!”

According to observations all swans were white. That is, until Australia was discovered.

The bug and it’s effect

� What are the preconditions for a bug to occur?� Only in Vietnamese Windows, when user click ctrl four times in a row

� When it occurs, how bad can it get?� How many users are affected?

� How many transactions are affected?� How many transactions are affected?

� Browsers’ example

� If you could predict how many bugs you have left to discover, you surely couldn’t predict their impact

� Asymmetry between a bug and it’s effect – the less you expect something to happen, the more impact it has

According to my calculations, there are no more P1 bugs in the

software

What to do with bug?

� When having a bug:� You might not be able to reproduce it

� Fix it

� Leave it as is

� Document the current behaviour

It might be a feature☺� It might be a feature☺

� What to do if there are no bugs?� Most probably the software is not tested

Software ships with bugs

� There’s nothing unusual about it

� Known bug is always better than unknown bug� You should know which bugs are left

Economic considerations

� For technical people a bug is a bug and it must be fixed for the sake of perfection

� For business people a bug is another things that wastes developers’ time

� Fixing a bug might introduce new bugs

� Conclusions: � It might be OK to add feature instead of fixing a bug

� It might be OK to go with a hack instead of correct solution

� It might be OK to leave the software as is

� Which leads us to...

Good enough quality

� Software is not written for beauty contest

� Software is written to earn or save money

� Software should earn or save more money than spent on developing it

� Conclusions:� Conclusions:� Software doesn’t have to be perfect (developers are never satisfied with the code, that’s for sure)

� Software must be good enough

� What is good enough?� There’s no simple answer

Practical testing

Difference between testing and QA

� Quality assurance is a broader term, it includes:� Testing

� Test management

� Test process management

Bug tracking� Bug tracking

� Bug prevention

� What is the difference between tester and QA specialist?

Preconditions for testing

� Testing environment� Should be separate from development

� Must be separate from production

� Update procedure

� Installation manual

� Specification or even better, test plan

� Testability features:� Software version – test known state

� Log or debug info

� References to specifications

Types of testing

� Verification� Is the software implemented according to specification?

� But what is the specification is wrong?

� ValidationDid we implement the right thing?� Did we implement the right thing?

� But what if the user cannot express what he needs?

Testing different aspects

� Functionality� Does the application behave like expected?

� Usability� Is it convenient to use?

� Performance & stress� Continuous load & break point

� Security

� Reliability

� ...

Testing methods

� Manual testing� Done mostly in India ☺

� Automated testing� Haven’t seen an actually working and useful solution

I still believe � I still believe they exist

Automated tests rarely find new bugs

When to use automated tests?

� Automated tests are good for:� Testing performance, throughput, response times

� Testing machine interfaces (API, web services)

� Automated tests are not good for:� Functionality

� Usability

� You can automate parts of tests� For example, when testing security, you can use scripts for scanning ports or cracking password

� Test results must be interpreted

Automated testing tools

� Mercury tools (now owned by HP)� Quality Center

� Test Director

� Rational Tools (now owned by IBM)� Functional Tester

� There are apecialized freeware tools

� Tools don’t save you from thinking!

Testing in software lifecycle

Testing in software lifecycle

� (Typical business application example)

� Document inspection – requirements, analysis, design

� Testing prototype/helping developer

� Testing new functionality� Testing new functionality

� Testing changes

� Testing for acceptance

� Handling client feedback

Document inspection

� Completeness� If A, then ..., but what is A is not true?

� Correctness� Human language is vague

� Sometimes, usually, most

Good, effective, stable� Good, effective, stable

� Etc,

� Can it be implemented?

� Can it be tested?

� Is it optimal?

Testing prototype

� Not all projects use prototype, but sometimes developers ask feedback to their very early solution

� Is the concept right?

� Can the concept be implemented?

� Complications

Testing new functionality

� Input validation� Date field accepts YYYY-MM-DD

� Email must contain @ and .

� Business rules� When user has debt, send warning

� Is the data saved correctly? Is it displayed correctly?

� Privileges

� Messages

� Documentation

Testing changes

� What has been changed with this release?

� You need changelog:� Test new features

� Verify fixed bugs

� Are there any dependencies that might be affected?

� Is documentation updated?

� Are machine interfaces changed?

Acceptance testing

� Verify that criteria for production have been met� At first, you have to set these criteria

� Test the main use cases

� Test critical paths

� Test functionality that has been broken before

� Verify that there are no open issues that should have been fixed

Handling client feedback

� Clients should not give feedback directly to programmers, unless you’re doing XP, but then you don’t even have tester’s

� Testers should find out:� If it is a bug at all

If the bug can be reproduced� If the bug can be reproduced

� If there are no duplicates

Tester’s inputs and outputs

Tester’s input and output

� Tester’s input� Application

� Specifications

� Tester’s output� Test plans & test reports

� Bug reports

Test plans & test reports

� Test plan� What should we test?

� How should we test?

� What is the expected outcome?

� Test report� Test plan X was tested

� On version x.y.z

� On dd-mm-yyyy

� By John Smith

� And the result was “test failed”.

� The following bugs were reported: x, y, z

Bug lifecycle

� Reporting

� Bug analysis� Bug is localized, impact and root cause is analyzed

� Fix is scheduled

� Fixing

� Verification� Tester verifies that the bug is really fixed

� And no new bugs are introduced

Reporting bugs

� Bug report should contain:� Steps for reproducing

� Current behaviour

� Expected behaviour

� As much context as possible

As general as possible� As general as possible

� As specific as possible

� No feelings included

Bug analysis (example)� Viga kliendi detailvormil

� Üldistame:� Kas ükski kliendi detailvorm ei tööta? � Kas ükski detailvorm ei tööta?� Kas terve rakendus ei tööta?

� Konkretiseerime:� Osad kliendi detailvormid ei tööta. Mis on vahe � Osad kliendi detailvormid ei tööta. Mis on vahe töötaval ja mittetöötaval?

� Hmm – täpitähed!� Üldistame: Kas ükski täpitähti sisaldav vorm ei tööta?

� Konkretiseerime: “õ”-tähte sisaldavate nimedega klientide detailvormil tekib viga teatega …

Root cause analysis (example cont)

� Kus selline viga veel võib esineda?� Kas õ-tähega asutuse detailvormid on korras?

� Mis on selle vea tegelik põhjus?� Andmebaas kasutab sellist charset’i mis ei saa hakkama õ-tähega?

Kuidas välistada terve vigade klass?� Kuidas välistada terve vigade klass?� Konverteerida andmebaas?

Bug tracking tools

� Bugzilla – freeware, open source� Big user community

� Jira – commercial, site license

� Fogbugz – per user license� Read www.joelonsoftware.com and you might end up buying it

� High-end tools – Rational ClearQuest, etc

Testing as a career

Testing as a career

� It’s a good introduction to software development world

� It’s a good starting point when you want to become analyst or project lead

� How can one be a tester more than 3 years?

What makes a good tester?

� Imagination – a lot of it

� Healthy paranoia – do not assume anything

� Curiosity & exploratory thinking

� Always polite

Why should I want to be a tester?

� Common misunderstandings:� It’s a monkey job – if programming the same app is a monkey job, then testing it probably is too, sorry

� It’s not creative

� It’s not constructive

� It pays less� It pays less

� It’s not a job people usuallydream about, but manyend up doing

Reading

� “Lessons Learned in Software Testing” by Cem Kaner, James Bach, Bret Pettichord

� Read everything that you find useful� Not only about testing, but about software development in generaldevelopment in general

� Be sure to have enough domain knowledge

� And by the way, read Härmel Nestra’s lecture materials from 2005: � http://courses.cs.ut.ee/2005/tvt/Main/Lectures

Anyway, how to test?

Just do it.Just do it.

Just do it!