ethics of software testing thomas latoza cs 210 final presentation 12 / 2 / 2002

Ethics of Software Testing

Thomas LaToza

CS 210 Final Presentation

12 / 2 / 2002

© Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign

Overview

Background - facts of testing and bugs Education, training, best practices, and a

commitment to reliability All help to improve reliability Don’t solve the problem and still leave

Research How much to spend on testing When to release software Most organizations should spend more on testing


Facts of Testing Testing is an activity to evaluate quality and find bugs

Testing only proves presence of bugs Testing NEVER proves absence of bugs

(unless you could run every line of code for all possible inputs for all possible states in all possible environments with an oracle that knows the correct output and state)

Testing is underappreciated None of the top 5 CS departments offer an undergraduate

course on testing or building software systems for reliability (other than formal verification) – only taught as part of a software engineering course

Often not perceived as challenging and fun Admission of failure – people make mistakes


Finding Bugs

Not all bugs are equal.

Bugs may be Easier to find than other bugs More important than other bugs (bugs that

crash or destroy data vs. bugs that make the interface look bad)

Invisible until other bugs are fixed


Fixing Bugs

Fixing a bug always has a chance of introducing new bug(s) which may be worse

Overall effect of testing depends on ratio of bugs fixed to bugs introduced


Bugs aren’t cheap

For users Bugs and glitches cost the economy $59.5

Billion a year (NIST estimate)

For software projects Testing cost can easily be 25 – 50% cost of

project and can be much more for certified systems


Testing bug curve

Bugs

Time / Effort / Testing resources


Overview





Engineering practices can help prevent bugs Code reviews – highly effective

Team members proof read the code and discuss

Writing for readability There are two ways of constructing a software design; one way is to make it so simple

that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare

Design by test Write the interface and test case first, then write the functionality

Statistical testing Commercial off the shelf software / Open Source /

Components Separate Quality Assurance team Betas

But none of these replace testing…


Research

Problem has (recently) sparked great research interest

Recovery Oriented Computing Formal Methods Program Comprehension Software Engineering


Testing Review

It’s expensive and time consuming

Doesn’t guarantee lack of bugs

Developers often don’t like doing it

But it’s really important…


Overview





Review – Applying Ethical Theories

1. Evaluate information2. Consider how decision may affect stakeholders3. Consider what ethics [ethical values] are relevant to

situation4. Determine the best course of action that takes into

account relevant values and stakeholders’ interests.

(from Lockheed Martin Decision Making Model)


How much to spend on testing (1. Evaluate Information)

Most organizations should spend more on testing

But have to stop somewhere – can’t spend all of your budget on testing

Question of where on the curve is best There are advantages for being further left

and advantages for being further right


More testing / Less bugs(2. Consider how decision may affect stakeholders)

Advantages Developers / Customers – less time working around bugs Customers – less frustration dealing with software Managers – know more about bugs in product and can deliver a higher quality

product Company – quality makes software more valuable to customer Company – sooner bugs can be fixed the cheaper they are to fix

Disadvantages Customers – more expensive products Company – less features, less compelling reasons for customer to spend money

upgrading


How much to spend on testing(3. Consider what ethics [ethical values] are relevant to situation)(4. Determine best action based on relevant values and stakeholders)

Utilitarian – maximize cost / benefit for company and / or customers

Value – have a responsibility to provide a quality product by setting a quality standard and sticking to it

Libertarian – just enough that it helps company the most (customer fends for himself)

Altruism – so that it is best for the customer


When to release (1. Evaluate Information)

Would like to have a higher quality product But have to stop somewhere – may not ever

find all of the bugs Again question of where on the curve is best There are advantages for being further left

and advantages for being further right


Release later / Less bugs(2. Consider how decision may affect stakeholders)

Advantages All of the advantages of more testing / less bugs Managers – know more about quality of the product being delivered Company / customers – higher quality product

Disadvantages All of the disadvantages of less testing / more bugs Company – may lose marketshare to competitors Company – no income from product Customer – can’t use product at all Company – may lose credibility for missing deadline


When to release(3. Consider what ethics [ethical values] are relevant to situation)(4. Determine best action based on relevant values and stakeholders)

Utilitarian – maximize cost / benefit for company and / or customers

Value – set quality standard vs. keeping a ship date vs. money for company

Libertarian – whenever the most customers will buy it

Altruism – so that customer will have optimum balance of most time to use it and the best product


Conclusions

Many other ethical questions Should a bug be fixed? How much information about testing to release? Do developers and testers have professional

standards to uphold? Expectations for quality – too high or too low?


Further Reading

Economics of Software Verification http://cm.bell-labs.com/cm/cs/who/gerard/gz/paste01.pdf

http://cm.bell-labs.com/cm/cs/who/gerard/gz/paste01.pdf

ethics of software testing thomas latoza cs 210 final presentation 12 / 2 / 2002

Documents