qsic2010 shihab

Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy

Systems

Emad Shihab, Zhen Ming Jiang, Bram Adams, Ahmed E. Hassan and Robert Bowerman

Queen’s University and Research In MotionCanada

http://images.google.ca/imgres?imgurl=http://www.coderetard.com/wp-content/uploads/2009/03/rim-blackberry-logo07.jpg&imgrefurl=http://www.coderetard.com/category/uncategorized/&usg=__h8m4owH4y6m5J7Fiq1t8Dv1obTI=&h=303&w=450&sz=15&hl=en&start=1&sig2=xbr6Xl1OgMqa4mOt6NhatA&um=1&tbnid=aofJS58tAWo8iM:&tbnh=86&tbnw=127&prev=/images?q=RIM&hl=en&rlz=1C1GGLS_enCA313CA313&sa=G&um=1&ei=ryEOSqL-LKXYswO41Lj_Ag

http://images.google.ca/imgres?imgurl=http://research.cs.queensu.ca/~gradseries/help/images/queens.gif&imgrefurl=http://www.cs.queensu.ca/~gradseries/help/doc_writing/latex.htm&usg=__nwrOm8UJAGI0ubfUwqsTz-4EQlg=&h=413&w=600&sz=51&hl=en&start=3&sig2=ufvdIolYRgYrnF79bjHIYA&um=1&tbnid=hT2BJ9hJukVnFM:&tbnh=93&tbnw=135&prev=/images?q=Queen's&hl=en&rlz=1C1GGLS_enCA313CA313&sa=G&um=1&ei=0iEOSvaBL5-etAOx_5D4Ag

Test Driven Development (TDD)

Write unit test before

writing new code

What about already written code

Test Drive Maintenance (TDM)

Adopting Test Driven Development (TDD)

for Legacy Applications

But time and resources are limited!

Prioritizing Unit Test Creation

Use the rich history of the legacy system to prioritize the writing of unit tests

Avoid the most bugs effectively!

Write unit tests for functions with best Return on Investment (ROI)

How can we avoid the most bugs given limited resources?

Testing Writing Prioritization Heuristics

Most Frequently Modified (MFM)Most Recently Modified (MRM)

Most Frequently Fixed (MFF)Most Recently Fixed (MRF)

Largest Fixed (LF)Largest Modified (LM)

Change Risk (CR)Size Risk (SR)

Random

Modification

Fix

Size

RiskRandom

UsefulnessWas writing the unit test useful?

Time to write unit testA

B

C

6 bug fixes

2 bug fixes

0 bug fixes

Usefulness = 2/3 = 66.67%

POP: Percentage of Optimal Performance How close are we to the optimal performance?

Time to write unit testA

B

C

6 bug fixes

2 bug fixes

0 bug fixes

POP = 8/13 = 61.5%

DE

4 bug fixes

3 bug fixes

Case StudySimulation Parameters

Calibration Period: 6 months

Simulation time: 5 years

List Size: 10 functions

Effort: 100 lines per day

Study SetupExtracting Historical Data

1. Search modification record comments for keywords and bug identifiers

2. Extract source code of modified file(s) and compare to previous version to identify changed functions

3. Combine data from 1 and 2 to identify changed/fixed functions

main() { int a; /*call help*/ helpInfo(); }

helpInfo() { errorString!} main() { int a; /*call help*/ helpInfo(); }

helpInfo(){ int b;} main() { int a; /*call help*/ helpInfo(); }

V1:Undefined func.(Link Error)

V2:Syntax error

V3:Valid code

Mapping Historical Changes to Functions

Study SetupMeasuring the Performance of a Heuristic

Based on a heuristic, generate list of X functions to write unit tests for

Use size of function to measure effort required to write unit test

Test Writing Heuristics





Random

Modification

Fix

Size

RiskRandom

Best Test Writing Heuristics





Random

Modification

Fix

Size

RiskRandom

120 240 360 480 600 720 840 960 1080120013201440156016801800192020400

102030405060708090

100

Time (Days)

Usef

ulne

ss (%

) Most Frequently Modified (MFM)

Most Frequently Fixed (MFF)Largest Fixed (LF)

Change Risk (CR)

Random

UsefulnessWas writing the unit test useful?

POP: Percentage of Optimal Performance How close are we to the optimal performance?

120240

360480

600720

840960

10801200

13201440

15601680

18001920

20400

10

20

30

40

50

60

70

80

90

100

Time (days)

Perc

enta

ge o

f opti

mal

per

form

ance

(%)

Most Frequently Modified (MFM)

Most Frequently Fixed (MFF)

Largest Fixed (LF)

Change Risk (CR)Random

Largest Fixed (LF)

Largest Modified (LM)



Most Recently Fixed (MRF)

Change Risk (CR)

Size Risk (SR)

Most Recently Modified (MRM)

Random

0 10 20 30 40 50 60 70 80 90 100

87

84.7

83.8

80

56.9

55

48.8

43.1

27.7

32.4

32.2

22.2

21.8

7

5.5

4.3

4.9

1.7

Overall Performance of HeuristicsPOPUsefulness

5 10 200

5

10

15

20

25

30

35

40

List size

Perc

enta

ge o

f opti

mal

per

form

ance

(%)

Effect of Varying ParametersVarying List Size

Most Frequently Modified (MFM)Most Frequently Fixed (MFF)

Largest Fixed (LF)

Change Risk (CR)

Random

50 100 2000

5

10

15

20

25

30

35

40

45

50

Effort (lines per day)

Perc

enta

ge o

f opti

mal

per

form

ance

(%)

Effect of Varying ParametersVarying Writing Effort



Largest Fixed (LF)

Change Risk (CR)

Random

Conclusion

%

TDM ApproachOverview

Performance of Testing Heuristics

Simulation Case Study

Which heuristic is the best?

Simulation Case StudySimulation Parameters

Calibration Period: 6 months

Simulation time: 5 years

List Size: 10 functions

Effort: 100 lines per day

Case Study ResultsModification Heuristics

%

Case Study ResultsFix Heuristics

%

Case Study ResultsSize Heuristics

%

Case Study ResultsRisk Heuristics

%

qsic2010 shihab

Documents