how much time it takes for my feature to arrive?
TRANSCRIPT
An Empirical Study of Delays In The Integration of Addressed Issues
Daniel A. Costa Shane McIntosh Uirá Kulesza Ahmed E. Hassan Surafel L. Abebe
The term issue can refer to a bug fix, a new feature or an enhancement
Bug fix
New feature
Enhancement
1
We may think that to address an issue is enough to satisfy an user
2
Finds a bug
We may think that to address an issue is enough to satisfy an user
2
Finds a bug
We may think that to address an issue is enough to satisfy an user
2
Previous research has focused on the time to fix a bug
Opened
3
Previous research has focused on the time to fix a bug
Resolved-Fixed Opened
3
Previous research has focused on the time to fix a bug
Opened Resolved-Fixed
How much time?
3
Techniques and approaches were invented
4
Techniques and approaches were invented
Prediction Models
4
Techniques and approaches were invented
Prediction Models Bug Repots Similarty
4
But an addressed issue may take some time to be integrated into a release
#883554
5
So when does this stuff get added? Will it be applied to the next FF23 beta? A 22.01 release? Otherwise?
#883554
But an addressed issue may take some time to be integrated into a release
5
An empirical study to analyze the integration delay
6
3,121 issues 17 releases
An empirical study to analyze the integration delay
6
14,530 issues 15 releases
3,344 issues (JDT)
11 releases
3,121 issues 17 releases
3,121 issues 17 releases
An empirical study to analyze the integration delay
6
14,530 issues 15 releases
3,344 issues (JDT)
11 releases
3,121 issues 17 releases
20,995 issues
3,121 issues 17 releases
An empirical study to analyze the integration delay
6
14,530 issues 15 releases
3,344 issues (JDT)
11 releases
3,121 issues 17 releases
20,995 issues
Rapid Traditional Traditional
Release Notes
We collected data from release notes and issue tracking systems
7
Release Notes
We collected data from release notes and issue tracking systems
7
#14 #15 #16
List of Integrated Bug Ids
extract
Release Notes
We collected data from release notes and issue tracking systems
7
#14 #15 #16
List of Integrated Bug Ids
extract
Issue Tracking System (ITS)
get info.
Integration delay
8
Prediction models
Influential variables
Integration delay
8
Prediction models
Influential variables
Integration delay
8
Prediction models
Influential variables
Integration delay
8
Prediction models
Influential variables
Integration delay
8
Prediction models
Influential variables
Measuring integration delay
Time
9
Measuring integration delay
Time
9
Measuring integration delay
Time
9
Measuring integration delay
1
Time
9
Measuring integration delay
1 2
Time
9
Measuring integration delay
1 2
Time
After 2
9
34% to 60% of addressed issues in traditional release cycle were delayed
10
34% to 60% of addressed issues in traditional release cycle were delayed
34% 60%
10
Addressed issues are usually delayed in rapid relasing cycle
98%
10
Measuring the release frequency 600
400
200
0
Days b
etw
een
rele
ases
11
Measuring the release frequency 600
400
200
0
Days b
etw
een
rele
ases
11
Measuring the release frequency 600
400
200
0
Days b
etw
een
rele
ases
11
How long an issue was addressed before a release?
1
Time
12
How long an issue was addressed before a release?
1
Time
12
How long an issue was addressed before a release?
1
30 days
Time
12
How long an issue was addressed before a release?
1
30 days
40 days
Time
12
How long an issue was addressed before a release?
30 days
40 days
1
Time
30 40
12
How long an issue was addressed before a release?
30 days
40 days 0.75
1
Time
30 40
12
How long an issue was addressed before a release?
30 days
40 days 0.75
1
Time
30 40
Addressing Stage 12
Delayed issues were addressed well before the next release
Ad
dre
ssin
g St
age
1.00 0.75 0.50 0.25 0.00
13
Delayed issues were addressed well before the next release
Ad
dre
ssin
g St
age
13
1.00 0.75 0.50 0.25 0.00
14
Integration delay
Prediction models
Influential variables
Issues are usually delayed in rapid releasing cycle
14
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
14
Integration delay
Prediction models
Influential variables
Issues are usually delayed in rapid releasing cycle
14
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
When will an addressed issue be integrated?
Time
...
15
When will an addressed issue be integrated?
Time
Next
...
15
When will an addressed issue be integrated?
Time
After 1
...
15
When will an addressed issue be integrated?
Time
After 2
...
15
When will an addressed issue be integrated?
Time
After 3 or more
...
15
We built our prediction model using random forests
16
We built our prediction model using random forests
16
It is robust to noisy data
We built our prediction model using random forests
It is robust to noisy data
It deals with several types of attributes
16
We evaluated our models using precision, recall and ROC areas
Precision
Recall
ROC Areas 17
We evaluated our models using precision, recall and ROC areas
Precision
Recall
ROC Areas 17
We evaluated our models using precision, recall and ROC areas
Precision
Recall
ROC Areas 17
We evaluated our models using precision, recall and ROC areas
Precision
Recall
ROC Areas 17
Studied metrics
18 Process metrics
Reporter metrics Issue metrics
Project metrics
Studied metrics
Process metrics
Reporter metrics Issue metrics
Project metrics 18
Reporter Experience
Number of comments
Issue priority
Integration workload
Studied metrics
18 Process metrics
Reporter metrics Issue metrics
Project metrics
Reporter Experience
Number of comments
Issue priority
Integration workload
Measuring the integration workload of a given addressed issue
Time
19
Measuring the integration workload of a given addressed issue
Time
19
Measuring the integration workload of a given addressed issue
Time
500 ? We are still waiting as well!
19
Our models perform better than random guessing
Precision
Recall
ROC Areas 20
Our models perform better than random guessing
Precision
Recall
ROC Areas 20
0.59 to 0.88
0.62 to 0.88
All above 0.74
Our models perform better than random guessing
Precision
Recall
ROC Areas 20
0.59 to 0.88
0.62 to 0.88
All above 0.74
21
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
21
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
Our models perform better than random prediction
21
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
Our models perform better than random prediction
We computed the variable importance of our methods
Experience Workload #Comments
55 340 30
66 100 70
78 540 22
Accuracy: 0.8
22
We computed the variable importance of our methods
Accuracy: 0.8
22
Experience Workload #Comments
55 340 30
66 100 70
78 540 22
We computed the variable importance of our methods
Accuracy: 0.8
Experience Workload #Comments
66 340 30
78 100 70
55 540 22
Accuracy: 0.79
22
Experience Workload #Comments
55 340 30
66 100 70
78 540 22
We computed the variable importance of our methods
Accuracy: 0.8
22
Experience Workload #Comments
55 340 30
66 100 70
78 540 22
We computed the variable importance of our methods
Accuracy: 0.8
Experience Workload #Comments
55 100 30
66 340 70
78 540 22
Accuracy: 0.3
22
Experience Workload #Comments
55 340 30
66 100 70
78 540 22
The integration workload has a bigger influence in our models
Var
iab
le Im
po
rtan
ce
23
The integration workload has a bigger influence in our models
Var
iab
le Im
po
rtan
ce
23
The integration workload has a bigger influence in our models
Var
iab
le Im
po
rtan
ce
24
The integration workload has a bigger influence in our models
Var
iab
le Im
po
rtan
ce
24
The integration workload has a bigger influence in our models
Var
iab
le Im
po
rtan
ce
24
Javascript Engine is the component with the most delayed issues in firefox
0
10
20
30
40
50
60
70
80
90
Javascript Engine General
Next
After 1
After 2
>= 3
Perc
ent
of
del
ayed
is
sues
30.4 28
16
4.5
79
18 22
52
25
Javascript Engine is the component with the most delayed issues in firefox
0
10
20
30
40
50
60
70
80
90
Javascript Engine General
Next
After 1
After 2
>= 3
Perc
ent
of
del
ayed
is
sues
30.4 28
16
4.5
79
18 22
52
25
Javascript Engine is the component with the most delayed issues in firefox
0
10
20
30
40
50
60
70
80
90
Javascript Engine General
Next
After 1
After 2
>= 3
Perc
ent
of
del
ayed
is
sues
30.4 28
16
4.5
79
18 22
52
25
Surprisingly, priority and severity have little influence on our models
0
20
40
60
80
100
120
ArgoUML Eclipse Firefox
Priority 1 (P1)
Priority 2 (P2) 35 32
57 61
97 97
26
Perc
ent
of
pri
ori
ty
del
ayed
issu
es
Surprisingly, priority and severity have little influence on our models
0
20
40
60
80
100
120
ArgoUML Eclipse Firefox
Priority 1 (P1)
Priority 2 (P2) 35 32
57 61
97 97
26
Perc
ent
of
pri
ori
ty
del
ayed
issu
es
Surprisingly, priority and severity have little influence on our models
0
20
40
60
80
100
120
ArgoUML Eclipse Firefox
Priority 1 (P1)
Priority 2 (P2) 35 32
57 61
97 97
26
Perc
ent
of
pri
ori
ty
del
ayed
issu
es
Surprisingly, priority and severity have little influence on our models
0
20
40
60
80
100
120
ArgoUML Eclipse Firefox
Priority 1 (P1)
Priority 2 (P2) 35 32
57 61
97 97
26
Perc
ent
of
pri
ori
ty
del
ayed
issu
es
27
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
Our models perform better than random prediction
27
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
Our models perform better than random prediction
Integration Workload is the most influential variable in our models
Summary
So when does this stuff get added? Will it be applied to the next FF23 beta? A 22.01 release? Otherwise?
#883554
But an addressed issue may take some time to be integrated into a release
3,121 issues 17 releases
An empirical study to analyze the integration delay
14,530 issues 15 releases
3,344 issues (JDT)
11 releases
3,121 issues 17 releases
20,995 issues
Rapid Traditional Traditional
34% to 60% of addressed issues in traditional release cycle were delayed
Integration delay
Prediction models
Influential variables
Issues are adressed well before a release
Issues are usually delayed in rapid releasing cycle
Our models perform better than random prediction
Integration Workload is the most influential variable in our models
An Empirical Study of Delays In The Integration of Addressed Issues
Daniel A. Costa Shane McIntosh Uirá Kulesza Ahmed E. Hassan Surafel L. Abebe
How ROC should be interpreted?
16
How ROC should be interpreted?
16
How ROC should be interpreted?
16
We calculated the ROC areas in a pair basis
Next
After 3
After 2
After 1
17
We calculated the ROC areas in a pair basis
Next
After 3
After 2
After 1
Next
17
We calculated the ROC areas in a pair basis
Next
After 3
After 2
After 1
Next
Other
17
We calculated the ROC areas in a pair basis
Next
After 3
After 2
After 1
Next
Other
17
We calculated the ROC areas in a pair basis
After 1
After 3
After 2
Next
After 1
Other
17