predicting build failures using social network analysis developer ... · social network analysis...

Predicting Build Failuresusing Social Network Analysis

on Developer Communication

Timo Wolf Adrian Schröter Daniela Damian Thanh NguyenSIEMENS AGCorporate TechnologySoftware & Engineering

Software Engineering Global interAction Lab – SEGALComputer Science DepartmentUniversity of Victoria, BC Canada

SIEMENS AG [email protected]

Introduction

2

mailto:[email protected]



Introduction

2

Build Failure- Compile Error- Test Failure




IntroductionThe damn build broke again! WHY ?

2





IntroductionThe damn build broke again! WHY ?

DB changed!But that’s transparent.Analyses takes time!

We should have talked earlier!

2





Introduction

• Collaboration essential in software development

• Communication problems lead to coordination and integration failures

The damn build broke again! WHY ?



2





Introduction

• Collaboration essential in software development

• Communication problems lead to coordination and integration failures

Do specific communication behaviors enable effective and successful coordination?

The damn build broke again! WHY ?



2





Communication for collaboration

3





• Communication:

• Task related content

Task relatedContent &

Information

3





• Communication:


Task relatedContent &

Information

• Structure

3





• Communication:


Our focus:Communication Structure and relationship to coordination outcome

• Structure

3



SIEMENS AG [email protected] 4

Study Settings: IBM Jazz





IBM Research&





151 developers

47 functional teams




7 development sites in USA, Canada, Europe





Coordination & Integration in Jazz

8





Team A

Stream

Build 1

Team A Contributor 1 Contributor 2 Contributor 3

8





Team A

Stream

Build 1


8

!! !! !!





Team A

Stream

Build 1


8





Team A

Stream

Build 1


Team B

Stream

Build 1

commits changes

Team B Contributor 4 Contributor 5 Contributor 6

ChangeSets

8





Team A

Stream

Build 1


Team B

Stream

Build 1

commits changes


ChangeSets

ProjectBuild 1Project

IntegrationStream

publish changesget changes

8





Team A

Stream

Build 1


Team B

Stream

Build 1

commits changes


ChangeSets

ProjectBuild 1Project

IntegrationStream

publish changesget changes

Coordination Outcome: Build Result8




Research Questions

9




! OK ERROR

Research Questions

9




! OK ERROR

Research Questions

SNA Measurese.g. Density = 0.8

SNA Measurese.g. Density = 0.4predictmeasure predictmeasure

9




! OK ERROR

Research Questions

• RQ1: Can individual measures of communication structure predict integration failure?



9




! OK ERROR

Research Questions

• RQ1: Can individual measures of communication structure predict integration failure?

• RQ2: Can the combination of communication structure measures predict integration failure?



9




Constructing Social Networks in Jazz

10





10

time

Stream

t0 t1

Build N

Failed

Previous

Build

Build N

Failed

Previous

Build





10

Build N Network

c1

c2

c3

c4

time

Stream

t0 t1

Build N

Failed

Previous

Build

Build N

Failed

Previous

Build





10

Build N Network

c1

c2

c3

c4

time

Stream

t0 t1

Build N

Failed

Previous

Build

Build N

Failed

Previous

Build

c1c2 c3 c4





10

Build N Network

c1

c2

c3

c4

time

Stream

t0 t1

Build N

Failed

Previous

Build

Build N

Failed

Previous

Build

w1 w2

c1c2 c3 c4





10

Build N Network

c1

c2

c3

c4

SNA

quantitativemeasures

of communication structure

time

Stream

t0 t1

Build N

Failed

Previous

Build

Build N

Failed

Previous

Build

Social NetworkAnalysis (SNA)

w1 w2

c1c2 c3 c4




SNA Measures

11




SNA Measures

12




SNA Measures

13




SNA Measures

14




SNA Measures

15




Data Collection & Selection

• Time Range: Nov. 5, 2007 - Feb. 26, 2008

• 1288 Build Results

• 25713 Work Items

• 71019 Comments

• Selected Build Results of 5 Teams

• Selected 3 Integration Builds (nightly, weekly, beta)

16




RQ1: individual measure prediction?

17





SN Build 1

SN Build 3

SN Build 4

SN Build 2

SN Build 5

SN Build 6

17





SN Build 1

SN Build 3

SN Build 4

SN Build 2

SN Build 5

SN Build 6

Social Network Measures

• Densityfor Completeness

• Degree Centralityfor Activity

• Betweenness Centralityfor Control

• Structural Holesfor Redundancy

17





SN Build 1

SN Build 3

SN Build 4

SN Build 2

SN Build 5

SN Build 6

Social Network Measures

• Densityfor Completeness

• Degree Centralityfor Activity

• Betweenness Centralityfor Control

• Structural Holesfor Redundancy

No statistically significant difference!

17




RQ2: multi measures prediction?

18




?


18




?


Bayesian Classifier

1. Train withall measures

18




?


2. Input

Bayesian Classifier


18




?


2. Input3. Predict

Bayesian Classifier


18




Prediction Results

19




Prediction ResultsERROR Team Builds Project Builds

B C F P W nightly weekly beta

Recall 55% 75% 62% 66% 74% 89% 100% 92%

Precision 52% 50% 75% 76% 66% 73% 92% 92%

19




Prediction Results

• Recall:

• 62 of 100 failed builds are classified correctly

ERROR Team Builds Project Builds


Recall 55% 75% 62% 66% 74% 89% 100% 92%

Precision 52% 50% 75% 76% 66% 73% 92% 92%

19




Prediction Results

• Recall:

• 62 of 100 failed builds are classified correctly

• Precision:

• 75 of 100 failure predictions are actual failures

• 75% confidence that a prediction is correct

ERROR Team Builds Project Builds


Recall 55% 75% 62% 66% 74% 89% 100% 92%

Precision 52% 50% 75% 76% 66% 73% 92% 92%

19




Summary

20




Summary

• Quantitative empirical evidence:Communication structures make a difference!

20




Summary


• Communication structures:

• impact integration build results

• are key for successful collaboration

20




Summary





• Single measures do not predict build results

20




Summary





• Single measures do not predict build results

• Combination measures do predict build results

20




Practical Implications

• Communication assessment for future builds

• Notification system for build failures

• Communication structure improvement

21




Future Work

• Factor analysis of combined measures

• Include technical information

• Prediction effectiveness case study

22



Questions?

Timo [email protected]

Adrian Schrö[email protected]

Daniela [email protected]

Thanh [email protected]

http://segal.uvic.ca

SIEMENS AGCorporate TechnologySoftware & Engineering
















Backup Slides


Descriptive Statistics I

25




Descriptive Statistics II

26



predicting build failures using social network analysis developer ... · social network analysis...

Documents