1CEN 4072

Software Testing

PPT12: Fixing the defect

2

From defect to failure

How does a defect in a program become a failure?•When a defect causes an infection (erroneous program state) that is visible to the user•Not all defects lead to failures

– Nevertheless a defect is an erroneous piece of code and should be fixed

How to locate the defect that leads to a given failure?1.Start with the infected value that defines the failure2.Determine the possible origins of the infected value, following dependencies in the code3.Observe each origin to determine whether it is infected or not

– Restart at step 2 with the infected origin(s)

The above process will loop until an infection is found that has all sane origins•The code that produces this infection is the defect that caused the failure

3

Techniques for locating defects

Infections•If an assertion fails

Dependences•Identify what the value of a variable is dependent on

– Example: average = sum / count• The value of average is dependent on sum and count

Causes•Identify causes through experimentation

– The likelihood of finding a defect near a cause is very high

Anomalies•Identify states that only occur in failing runs

– Example: if function getSum() only run on failing runs• the likelihood that the defect is closely related to this anomaly is high

Code Smells•Identify constructs in the code that are known to cause errors

– Example: initialized variables• These do not necessarily cause errors, but they do at a very high rate

4

Focusing on errors

Reduce defect search time•Focus search on origins that are more likely to cause errors

Assertions•Guarantee the sanity of data over time on any program run

– If an assertion fails then an infection has been identified– Else anything covered by the assertion can be ignored in the search

• Because it is known to be in a sane stateAnomalies•Aspects of the program that correlate with failure

– Example: a certain function only runs on failing runs– This correlation to failure means that it is wise to focus on these

Causes•Not only correlate to failure, but have been experimentally proven to cause the failure

– These origins are even more likely than anomalies to indicate the defect

5

Focusing on errors

Focus on Infections•If an origin is known to be faulty then focus on it first

– Check whether or not the infection causes the failure

Focus on Causes•If some form of experimentation has shown some state or input to be a failure cause

– Check if the cause is infected

Focus on Anomalies•Any origin that is associated with an anomaly has a good chance to contain errors

– Check if these origins are infected

Focus on Code Smells•If code smells have been identified and the origin is one of these code smells

– Check if the code smell causes the infection and/or error

Focus on Dependences•If all the above fail to find the defect, then the defect must be found in some dependency

– For all of the possible origins, check the backward slice for an infection• Start with the origins nearest the failure

6

Techniques: dependences

Starting from the failure, follow dependences to find possible origins of the defect

7

Techniques: observation

Observe the state of an origin and decide whether or not it is infected

8

Techniques: assertion

Use assertions to determine whether a given origin is sane or not•Note that a single assertion can cover multiple origins as seen below

9

Techniques: anomaly

Here there are two origins to choose from•One of them is observed to be an anomaly so it is focused

10

Techniques: cause transition

Here a cause transition is identified•This is known because of the fact that the target is infected but the origin is sane

11

Techniques: defect found

The defect has now been identified

12

Techniques: TRAFFIC strategy

The techniques explained above follow the TRAFFIC strategy•Specifically, the “focus on likely origins” step

TRAFFIC strategy•Track the problem•Reproduce•Automate•Find origins•Focus on likely origins•Isolate Correct

13

Techniques: ordering

The order in which to apply the aforementioned techniques is as follows1.Focus on infections2.Focus on causes3.Focus on anomalies4.Focus on code smells5.Focus on dependences

This ordering is chosen so that the techniques that are more likely to find a defectare applied before the techniques that are less likely to find a defect

14

Is the error a cause?

Just because an error has been found does not mean that the error causes the failure

When tracing an infection chain it must be shown at each step that1.The origin is infected2.The origin causes the infection chain

Example:a = compute_value();printf("a = %d\n", a);

If the output of the program is always “a = 0”•Then it may be assumed that the defect is related to the compute_value() function•This can be shown to be false by manually setting a = 1 right before the print

– If the program still outputs “a = 0” when a is known to not be 0• Then the problem must be with the print statement

•If this manual check had not occurred – Then the tester would search through origins where the cause does not exist

• Some error may be found in this search; however, this error is not the cause

15

Does making symptoms disappear fix the error?

Simply making the symptoms of an error disappeardoes not mean that the error has been fixed

Example:balance[account] = 0.0;for (int i = 0; i < n; i++) balance[account] += deposit[i]

Assume that account 123 has an incorrect balance•The following makes the symptom disappear but does not fix the true error

if (account == 123) balance[123] += 45.67

16

Debugging into existence

Debugging into existence•Changing the code to make it work without understanding the causeExample:static void shell_sort(int a[], int size){ int i, j; int h = 1; do { h = h * 3 + 1; } while (h <= size); do { h /= 3; for (i = h; i < size; i++) { int v = a[i]; for (j = i; j >= h && a[j - h] > v; j -= h) a[j] = a[j - h]; if (i != j) a[j] = v; } } while (h != 1);}

Assume that the fix for this program was to change the highlighted line to instead be the following

for (i = h; i < size - 1; i++)

This change was made based off a guess that the loop bound was off by one. There was no testing to prove this to be the case.

This is wrong because it is unknown whether or not the true error was fixed. This technique of debugging with guesses is dangerous because it can introduce new bugs into the program.

17

Devil’s guide to debugging

Find the defect by guessing•Scatter debugging statements throughout the program•Try changing code until something works•Do not back up old versions of the code•Do not bother understanding what the program should do

Do not waste time understand the problem•Most problems are trivial anyways

Use the most obvious fix•Just fix what you see•Example:

x = compute(y)x = compute(y)// compute(17) is wrong – fix it// compute(17) is wrong – fix itif (y == 17)if (y == 17) x = 25.15x = 25.15

•Why bother tracing the error through compute()? Just fix it!

These are all techniques that should be fully avoided

18

Correcting the defect

Before applying the correction to the code you must always first BACKUP THE CODE

After correcting the code there are four problems that must be looked at1.Does the failure no longer occur?

– If the failure no longer occurs then the correction was successful– If the failure is still present then there are two possible reasons

1. The failure was caused by more than one defect and only one was fixed2. Your understanding of the infection chain was wrong

2.Did the correction introduce new problems?– A few practices that are useful for this question

1. Have corrections peer reviewed2. Have a regression test ready to detect unwanted changes

19

Correcting the defect

3. Was the same mistake made elsewhere?– Check for other defects that are caused by the same mistake

• Focus on – Code from the same developer– Code involving the same API

4. Did I commit the change?– Assign resolution to the problem (such as FIXED)– Integrate the fix into the production code

20

Workarounds

Why might workarounds be needed?1.Unable to change

– The source code may not be available to be changed2.Risks

– The correction may induce huge risks• Such as if the fix implies large changes throughout the system

3.Flaw– The problem may not be in the code but the overall design of the program

• In which case the system design must undergo a major revision

Under what circumstances are workarounds allowed?•If no better solution is known

– Workarounds are meant to be temporary fixes– Be sure to keep the problem open so that a true fix may one day be found

21

Famous workarounds

1. Spam filters– Workaround for solving a flaw in the email system

• Anyone can forge messages and conceal their true identity– Proper solution would involve reworking the entire email system

2. Virus scanners– Privilege level flaw in operating systems

• By default a user has admin rights• Means that any download can gain complete control of the machine

– Proper solution would be to limit user rights• Problem is that too many regular programs assume admin privileges

3. Date windowing– Inability of many legacy systems to deal with four-digit years– Resolves ambiguity of two-digit year by defining a 100-year window– Proper solution would require adapting legacy systems

• Too much cost and risk involved

22

Learning from mistakes

• Improve quality assurance– Introduce systematic processes to obtain program requirement and design– Check specifications early

• Find differences in what the developers thinkthe program should do and what the clients want the program to do

• Resolving these issues early leads to requirements being more precise• Improve test suite

– If the test suite failed to find the defect• Once the defect has been found the test suite should be extended

to test for the new defect• Setup assertions

– Can narrow down the infection chain of a failure– Will catch similar infections in the future

• Improve the software process– Plan the software carefully before starting

• Improve analysis tools– Verify whether common tools could have found the defect early

• Particularly tools that detect code smells or verify system assertions

23

Facts on fixes

• The size of fixes with respect to other changes– Fixes are typically two to three times smaller than other changes

• Fixes and failures– Fixes are more likely to induce failures than other changes

• Fixes and older modules– A module that is one year older than another has 30 percent less errors

• Newly written code versus old code– Newly written code is 2.5 times as likely to contain a defect than old code

24References

Zeller, Andreas. Why Programs Fail. Second edition. Chapters 15 and 16.