msr2012 adams

Shahed Zaman Ahmed E. HassanSAIL, Queen’s University

A Qualitative Study onPerformance Bugs

Bram%AdamsMCIS,&École&Polytechnique&de&Montréal

Bugs, Bugs, Bugs!

2

Costly Affect%

Reputa4on

Bugs, Bugs, Bugs!

2

482%bugs/week

Most%research%treats%all%bugs%equally!%

Does%this%make%sense?

Bugs, Bugs, Bugs!

4

Performance%Bugproblem&in&efficiency/speed&at&which&a&system&performs&at&runDEme=

5hHp://www.pcmag.com/arEcle2/0,2817,2325772,00.asp

systemEkilling%

performance%problems

Security versus Performance Bugs:

A Case Study on

Shahed&Zaman,&Bram&Adams,&Ahmed&E.&HassanSoUware&Analysis&and&Intelligence&Lab&(SAIL),&Queen’s&University

6

MSR%2011%Quan4ta4ve%Study

Performance&bugs&are&fixed&by&more&experienced&

developers

People

Performance&bug&fixes&are&more&complex

Bug%FixTime

&&&&&Performance&bugs&take&more%4me&to&fix

7

Why?

8

Shahed Zaman Ahmed E. HassanSAIL, Queen’s University

A Qualitative Study onPerformance Bugs

Bram%AdamsMCIS,&École&Polytechnique&de&Montréal

Chrome®

44,997%bugs

510%performance%bugs

295,198%bugs

7,603%performance%bugs

Performance%Bugs NonEperformance%bugs

100%Bugs

100%Bugs

=

200%Bugs

+

100%Bugs

100%Bugs

=

200%Bugs

+

Study Setup

11

bug%reports

‘slow’,%‘perf’,%

‘hang’

Our%heuris4cs%have:

100&±&10%%precision83&±&10%%recall

Our%heuris4cs%have:

90&±&10%%precision97&±&10%%recall

12

analyze

13

comments

bug%report

+%a]achments

14

update&taxonomyanalyze

think

14

IterativeProcess

update&taxonomyanalyze

think

4 Dimensions(20 subdimensions)

impact&on&the&user

context&of&the&bug

bug&fix

bug&fix&validaEon

Findings

is&regression

blocks&release

WFM&after&a&long&Time

People&Threaten&to&switch

0 5 10 15 20 25

PerformanceNonDPerf.

%

sta4s4cally%

significant%

difference

No Traceability

17

WorksForMe

Findings

is&regression

blocks&release

WFM&after&a&long&Time

scares&users&away

0 5 10 15 20 25


%


impact&on&the&user

context&of&the&bug

bug&fix

bug&fix&validaEon

Findingshas&measurements

has&test&cases

has&stack&traces

Replication&Problems

Reported&by&Member

Duplicate&Bugs

0 10 20 30 40 50 60


%

21

automa4c%performance%

regression%tes4ng

21

automa4c%performance%

regression%tes4ngperformance&

regression&idenEfied&automaEcally

solicits%comparison%to%

other%browsers

22

Findingshas&measurements

has&test&cases

has&stack&traces

has&replication&problems

Reported&by&Member

Duplicate&Bugs

0 10 20 30 40 50 60


%

Replication Problems

24

incomplete&report

<empty><moreempty>

plugins/addDons

Hard to Track Bugs ofExternal Applications

25

Findingshas&measurement&data

has&test&cases

has&stack&traces

has&replication&problems

is&reported&by&developer

is&a&duplicate

0 10 20 30 40 50 60


%


impact&on&the&user

context&of&the&bug

bug&fix

bug&fix&validaEon

Findings

has&discussion

Depends&on&Other&Bugs

Blocks&Other&Bugs

0 10 20 30 40 50


%

Performance Bugs Require Collaboration

29

understand&the&problem

idenEfy&bug&fix

reach&consensus

me&too

Findings

has&discussion

depends&on&other&bug

blocks&other&bug

0 10 20 30 40 50


%

Rebasing Performance

31

adjust&baseline


impact&on&the&user

context&of&the&bug

bug&fix

bug&fix&validaEon

33

NO&difference&between&patch&discussions&and&reviews&of

performance&and&nonDperformance&bugs

is&regression + +blocks&release + +WFM&aUer&a&long&Eme + +scares&users&away + +

has&measurements + +has&test&cases + +contains&stacktrace + +has&replicaEon&problems + +is&reported&by&a&developer ? +is&a&duplicate E E

has&discussion + +depends&on&other&bug + +blocks&other&bug + E

impact&on&the&user

context&of&the&bug

bug&fix

MSR%2011%Quan4ta4ve%Study%(reprise)

Performance&bugs&are&fixed&by&more&experienced&

developers



36

replication problems

more dependencies

more discussion

WFM after a long time

more release blocking

users leaving

Threats to Validity

37

human&bias

only&4&dimensions&&&20&subDdimensions

only&2&subjectsystems

‘slow’,%‘perf’,%

‘hang’

heurisEcs&for&performance&bug&idenEficaEon

Chrome®

44,997%bugs510%performance%bugs

295,198%bugs7,603%performance%bugs

Performance%Bugs NonEperformance%bugs

100%Bugs

100%Bugs=

200%Bugs

+100%Bugs

100%Bugs=

200%Bugs

+

Study Setup

no traceability

no reproducabilitydiscussion!


impact&on&the&user

context&of&the&bug

bug&fix

bug&fix&validaEon

is&regression + +blocks&release + +WFM&aUer&a&long&Eme + +scares&users&away + +has&measurements + +has&test&cases + +contains&stacktrace + +has&replicaEon&problems + +is&reported&by&a&developer ? +is&a&duplicate E Ehas&discussion + +depends&on&other&bug + +blocks&other&bug + E

impact&on&the&user

context&of&the&bug

bug&fix

MSR%2011%Quan4ta4ve%Study%(reprise)Performance&bugs&are&fixed&

by&more&experienced&developers



37

replication problems

more dependencies

more discussion

WFM after a long time

more release blocking

users leaving

msr2012 adams

Documents