modeling the effect of size of defect proneness for open-source software
DESCRIPTION
A. Gunes Koru and Donsong Zhang and Hongfang LiuTRANSCRIPT
MODELING the EFFECT of SIZE on DEFECT PRONENESS for OPEN-SOURCE SOFTWARE
A. Güneş Koru1, Donsong Zhang1, and Hongfang Liu2
1Department of Information SystemsUMBCBaltimore, MD, USA
2Georgetown Medical CenterDepartment of Bioinformatics, Biostatistics, and BiomathematicsGeorgetown University, Washington, D.C., USA
E-mails: [email protected], [email protected], [email protected]
PROMISE at ICSE’07
UMBC
• UMBC, University of Maryland, Baltimore County (http://umbc.edu/~gkoru)
• Public research university with a focus on graduate education.
• Theoretically, all campuses belong to the University of Maryland but practically they look like different universities.
• UMBC is located in Baltimore in a small suburban neighborhood called Catonsville. UMBC is not
• University of Maryland, College Park
• University of Baltimore (Business school)
• University of Maryland Baltimore (Medical School)
• Hongfang Liu is with the Georgetown University located in Washington, D.C., Interested in Bioinformatics and Health Care.
Size--Defect Relationship
• Size is perhaps the oldest measure. Mostly, measured by lines of code (sometimes function points).
• Several studies found size to be associated with defect count. Earliest: A linear model in [Akiyama 71].
• Many other measures (e.g. cyclomatic complexity [McCabe 76], software science measures [Halstead 77]) are also correlated with size. There is some consensus that these are also size measures [Fenton and Pfleeger 96].
“May be size does not explain everything, but it explains a lot.”Bojan Cukic, PROMISE 2007
• Functional form of this relationship is still not understood well.
• Commonly, practitioners assume a linear relationship [El Emam 05].
• Only general conclusion is that there is a continuously increasing relationship between the two [Fenton and Ohlsson 00, El Emam et al. 01].
size size size
defects defects defects
(a) (b) (c)
Size--Defect Relationship: Alternative Forms
• Implications:
• (a) Linear: Smaller and larger modules are proportionally equally problematic
• (b) Quadratic: Larger modules are proportionally more problematic
• (c) Logarithmic: Smaller modules are proportionally more problematic
“Things are linear is open to questions”
Tim Menzies, PROMISE 2007
• Theoretical and Practical Importance
• Decomposition
• Focused quality assurance
• Functional Enhancements
Why the relationship is still unclear...
• Many earlier studies did not fully explore alternative functional forms or test the deviation from linearity significantly.• Linear models [Akiyama 71] or correlations [Andersson and Runeson 07] were
found sufficient.• A study stated that linear models could be good as first approximations and
there was better tool support [Shen 85]• Number of data points were very limited in the earlier studies (e.g. Akiyama 71).• Deriving models analytically and then fitting data to validate those models [Lipow
82].• Accepting correlations as a sign of a linear relationship [Schneidewind and
Hoffman 78]. Correlations do not imply proportionality.• Focus shift on defect density. Observations for optimal module size that
minimizes defect density. U-shaped curve (Goldilock’s conjecture) [Withrow 90, Hatton 97, Hatton 98, etc]. See [El Emam 02] for a detailed review.• This approach can mask the plain size--defect relationship and mislead us. [El
Emam 02, Fenton and Neil 00, and Rosenberg 97]• Gets more difficult to understand from multivariate and sophisticated machine
learning models (e.g. from Neural Networks in [Khoshgoftaar 97]).
Conventional Approach toInvestigate Size--Defect Relationship
• All these studies share a common characteristic
• A software system is measured at a snapshot time, then the obtained measurements are associated with the future defect count (note this might be pre-release or post release) For ex: [Koru and Tian 03] [Khoshgoftaar 96]
• Usually, measurement and analysis performed at module level.
• A common problem is the availability of data [Fenton and Ohlsson 02].
• Publicly available Open Source Software (OSS) repositories: Source code, change data, and defect data [Koru and Tian 04].
Challenges with Using Conventional Method in OSS Context
• Evolutionary aspects of OSS. Continuous and concurrent functional enhancements, defect fixes, all other changes (perfective, adaptive, etc.) Bazaar model rather than cathedral model [Raymond 99].
• OSS, usually, developed by volunteers, not too much planning, no requirements or design documents, source code is the main artifact. [Mockus et al. 00, Mockus et al. 02].
• Quality assurance activities are not systematic in OSS (see Zhao and Elbaum 03, Koru et al. 07])
• So far, research using conventional approach focused on relatively better planned, analyzed, designed, and tested closed source products.
• Internal validity problems caused by the dynamic OSS context:
• Deleted classes
• Size changes
• There might be closed source products developed in an evolutionary manner and vice versa. Such comparisons are outside of the scope here (see [Paulson et al. 04]))
In this study...
“If developers play with a file, it can change its defect proneness”Elaine Weyuker, PROMISE 2007
• To gain a better understanding of the size--defect relationship, we used both
• Novel approach that adopts Cox Proportional Hazards Modeling with Recurrent Events (Cox Modeling) [Cox 72].
• The data comes from a large-scale long-lived OSS product Mozilla (http://www.mozilla.org).
• The evolutionary aspects of the Mozilla project was shown in other studies:
• Gyiomothy et al. [04] found that size of Mozilla increased significantly during successive releases.
• Mockus et al. [02] found that there was no particular development process in Mozilla.
In the rest of this presentation...
•Methodology
•Demonstrating the evolutionary aspects of Mozilla
•Cox Modeling
•Data Collection
•Modeling and Results
•Future Work
•Conclusion
Results:Demonstrating Evolutionary Aspects of Mozilla
●●●●●●●●●●●●●●●●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ● ●●●●●●●●●●●
●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●● ●●●● ● ●●●●●●●●●●●●● ●●●●●●●●●●●● ● ●●●● ●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●● ●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●●●●●●● ● ● ●●●●●●●●●
●●●●●●●●●●●●●● ● ● ●●●● ● ●●●●●●●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●
●●●●●●●●●●●●●●● ●●●●● ● ●●●●●●●●● ● ● ● ● ● ● ●●●●●● ● ● ●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●● ● ●●●● ●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
020
040
060
080
010
00
(a)
Cum
ulat
ive N
umbe
r of D
elet
ed C
lass
es
2003 2004 2005 2006Years
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
050
0000
1000
000
1500
000
(b)
Cum
ulat
ive C
ode
Chur
n (in
LO
C)
2003 2004 2005 2006Years
• For only Mozilla 1.0 classes
• (a) Cumulative number of deleted classes
• (b) Cumulative code churn
Cox Modeling( A non-parametric approach)
• The instantaneous relative risk (hazard) of defect fix, also called event, becomes the response variable. Note that it can recur.
• A complete size history is obtained for each class by measuring size at each change and corrective changes are marked.
• Time of change is also noted. At each unique time, the hazard is calculated by dividing the events at that time by the classes at risk at that time.
• Hazard function:
• Relative hazard:
• Note that the relative hazard is proportional to the difference in covariate values. This is called proportional hazards assumption and needs to be checked.
e!(xj(t)!xk(t))
!i(t) = !0(t)e!xi(t). (1)
" is the regression coe!cient for xi(t) and !0(t) is an unspecified non-negativefunction of time called the baseline hazard function. It is the instantaneoushazard of having an event without any covariate e"ect (i.e., when " = 0).
• Relative log risk is noted by f(size) (for median size, it is set to zero).
• Examine the functional model with Cubic Spline Functions using four knots
• Examined the alternative model visually
• Tested whether the deviation from linearity was statistically significant
Methodology
f(size) = !0+!1size+!2(size!k1)3++!3(size!k2)3++!4(size!k3)3++!5(size!k4)3+(1)
where,
(size! kn)+ =!
(size! kn), if (size! kn) > 00, otherwise
"(2)
H0 : !2 = !3 = !4 = 0
Methodology - Data Layout and Collection
• We developed PERL scripts to extract source code, analyze CVS changes, and to find whether a class is affected or not
• (a) What would the data look like if conventional approach was used.
• (b) Novel Approach: Classes between added to the system after Mozilla 1.0 release date were measured until Feb 22, 2006.
• Each change resulted in an observation
• 15,545 observations
• Events were identified by searching the CVS logs for words ‘bug’, ‘defect’, and ‘fix’. When we sampled 100 logs randomly, we saw that this automated approach was correct for 98 of them.
(A)class name size defect count
A 75 0B 250 2C 300 2D 600 2E 800 3F 220 0G 300 0. . .. . .
(B)class name start end event size state
Y 0 50 0 75 0Y 50 100 1 200 1Y 100 200 0 300 1Z 0 200 1 250 0Z 200 800 0 180 1Z 800 1400 1 400 1Z 1400 1800 0 300 1. . . . .. . . . .
0 2000 4000 6000 8000 10000 12000
−1.0
−0.5
0.0
0.5
1.0
1.5
2.0
Size (LOC)
Inst
anta
neou
s re
lativ
e ris
k of
def
ect f
ix
Results - Functional Form
• When we use cubic spline functions the logarithmic form is also obvious. The curve down at the end is only for less than 0.3% of the data points. We can use log(size) directly in the Cox model
Results -- Modeling results
MANUSCRIPT SUBMITTED TO TSE 23
coef exp(coef) se(coef) robust se z p
log(size) 0.368 1.44 0.00732 0.018 20.4 0
Rsquare= 0.152 (max possible= 1)
Likelihood ratio test= 2565 on 1 df, p=0
Wald test = 416 on 1 df, p=0
Score (logrank) test = 2565 on 1 df, p=0,
Robust Score = 142 p=0
Fig. 5. Modeling results using logarithmic transform of size
The entire model is also very significant as shown by the Likelihood ratio, Wald, score, and525
robust score tests. Both normal and robust estimates show this significance. The estimate526
of the coe!cient, !̂, for log(size) was 0.368, and its standard error estimate was 0.00732.527
The robust sandwich estimate of the standard error, which took the intra-subject correlation528
into account, was 0.018. Both of these standard error estimates were small, therefore we can529
safely accept !̂ = 0.368.530
As discussed in Section III-C, R2 is not an appropriate measure for Cox models because531
of censored observations and non-normal distribution of residuals. However, there are other532
fitness checks for Cox models. The first step is to check whether the model satisfies the533
proportional hazards assumption. For this purpose, usually, a time–covariate interaction is534
tested to make sure the coe!cient, !, is not time dependent [69]. For example, if a drug is535
e"ective only in the first hour after it is taken, then the Cox model developed would have536
a time dependent coe!cient, which would violate the proportional hazards assumption. We537
tested whether log(size)’s interaction with time is significant. This test produced:538
rho chisq p
log(size) -0.000859 0.0433 0.835
As shown by p = 0.835, the result of this test was highly insignificant. Therefore, the539
null hypothesis that log(size) had no interaction with time was accepted; ! was constant540
over time, which satisfied the proportional hazards assumption of the Cox model. For visual541
inspection of this assumption, it is common to plot a graph of the scaled Schoenfeld residuals,542
along with a smooth curve that represents !(t). As shown in Figure 6, the smoother is543
May 2, 2007 DRAFT
!
!
!!!
!!!!!!!!!!
!
!!
!
!!!!
!!
!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!!
!!!!!!!
!!!!!
!!
!!!!!!!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!
!!!!
!
!!!
!
!!!!!!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!!
!!!
!
!!!!!!!!
!
!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!
!!!!
!!!!!!
!
!
!
!!!!
!
!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!!
!
!!
!
!!!!!
!
!!
!!!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!!!!!
!
!
!!
!!
!
!
!
!
!
!!!!!!!!
!
!
!!
!
!!
!
!!
!
!
!!
!!!
!
!!!!!!!!!
!
!
!!!!!
!!!!
!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!!
!
!
!!
!
!
!
!!!!!
!
!!!!!
!
!!!!!!!!!!!
!!
!
!!
!
!
!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!
!
!!!!!
!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!
!!!!!!!!!
!
!!!!
!
!
!!
!
!!!!!!!!
!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!
!!!!!!!!!!!!!!!!!!
!
!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!
!!!
!
!!!!
!
!
!
!!!!!
!
!
!
!!
!
!!!!!!
!
!
!
!
!!
!
!!
!
!
!!!
!
!!!
!
!
!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!
!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!!!
!
!!!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!!
!
!
!
!
!
!!!!!!!
!!!!
!!!!
!!!!!!!
!
!!!!
!
!
!
!!!!!!
!
!
!!!
!!!!!
!!!
!!!!!!
!!!
!
!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!
!
!!!!!!!!
!
!!!!!!!!!!!!!!!
!!!!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!
!!!!!!
!!!!!!!!!!!!!!!!!!!
!!
!
!
!
!!
!
!!
!!!
!
!!!!
!!
!!
!
!!!
!
!!!!!!!!!
!
!!!!!!!!!
!!!
!!!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!
!!
!!!
!
!!!!!!!!!!
!
!!
!!!!!
!!!
!
!
!
!
!!
!
!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!
!
!!!!!!!
!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!
!
!
!!!!
!
!!!!!!!
!!!!!
!
!!!
!!
!
!
!!
!!!
!
!!
!!
!
!!!!!
!!
!
!!!
!!!
!!!!!
!
!
!
!
!!!!
!!
!
!!!
!!!
!
!
!!!!
!!
!
!
!
!
!!
!!
!
!
!!!
!!
!
!
!
!
!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!
!!
!
!!!!!
!
!
!!
!!!!!
!
!
!
!!!
!
!!!!!
!
!
!!
!
!
!
!
!!!!!!
!!
!
!
!!
!!!!!!!
!
!
!!
!!
!
!
!
!!
!
!
!!
!!!
!
!
!
!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!
!
!!!
!
!!!
!
!!!!!!!!
!!!
!
!!!
!
!!
!
!
!!!
!
!
!
!
!!!
!
!!!!!!
!
!
!!!!!!
!!!!!!!
!!!!!!!!!!!!!
!
!
!
!!!!!
!
!
!
!
!!!!!!
!
!!
!!
!!
!!
!
!!
!
!!!!
!
!
!
!
!!!!!!!
!!!!!!!!!!!
!
!!!!!!!!!!!!!
!
!!!
!
!!
!
!!!!
!!
!!
!!!
!
!
!
!
!!
!
!!
!!!!!!
!
!!!!!!!!!!!!!
!
!
!!!!!
!!!
!
!!
!!
!
!
!!!!!!
!
!!!!!!
!
!
!
!!!!!
!
!!!!!
!
!!!!!!!
!
!!!
!!
!
!!
!!
!
!!
!
!!
!
!!!!
!
!!!!!
!
!
!
!
!
!!!!!!
!
!
!!!!!!!!!!!!!
!!
!
!!!!!!
!
!!!!!!
!!
!!!!!!!!!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!
!
!!!!!!!!
!
!!!!!!!!!!
!!
!
!
!
!
!!
!!
!!!
!
!
!
!
!
!
!!!!
!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
!
!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!
!
!!!!!!
!
!!!!!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!
!
!!!!!!
!
!
!
!!!!
!
!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!
!!
!!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!!
!!!
!!!!!
!
!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!!!
!!!
!!!!!
!!
!!
!!
!
!!!!!
!
!!
!
!!!!!
!
!!!!!!!!!!!!!!!
!!!
!!!!!!!!!!
!
!!!!!!!!!
!
!!!!!!
!
!!
!!!!!!!!!!!!!
!!!
!!!!
!
!
!!!
!!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!
!!!!!!!!!!!!!
!
!
!
!!
!!
!!!!!!!!!!!!!!
!!!!!!!!!!
!
!
!!
!!
!
!!
!
!!!!!
!!!!!!!!!!!
!
!
!
!!!!!
!!!
!
!!!
!!!!!!!!!!!!
!
!!!!!!!!!!
!
!!
!!!
!!!!!!!!
!
!!
!
!
!
!!!
!
!!!
!!
!
!
!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!
!!
!
!
!
!
!
!
!!!!
!
!!
!
!
!
!
!!!!!!!!
!!!
!
!
!
!!!!!!!!!
!
!!!!!!!!!!!!
!
!
!
!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!
!!
!
!!!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!!!!
!!!!!!!!!
!!!!
!
!
!!!!!!!!!!!!
!!!
!
!!!!!!!!!!!!!!!
!
!
!
!
!
!
!!!!
!!
!
!!!
!
!
!!!!!!!!!!!!
!
!!!!!
!
!!!!
!!
!
!!!
!!
!!
!
!
!!!
!!!!!!!
!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!
!
!!
!!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!
!!!!!
!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!
!
!
!!
!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!!!!
!!!!
!
!!!!!
!
!!!!!
!!!!!!!!!!!
!
!!
!!
!
!!!
!
!!
!
!!!!!!
!!
!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!!
!!!
!!
!!!!!
!
!!!!!!
!!!!
!!!
!
!
!!!!!!!!
!
!
!
!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!
!!
!
!!
!
!!!!!!!!
!!!!!!!!!
!
!!!!!!!!!!!!!!
!
!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!
!!!!!!
!
!!!!!
!!
!!!
!
!!!!!!!!!!
!
!!!!!!!
!!!!!!!!!!
!!!
!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!!!!
!
!
!!!!!!
!!!!!!!!!!
!
!!!!
!!
!
!
!
!!!!!!!!!!
!
!
!!!!!!!!
!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!
!
!!
!!!!!!!!
!
!!!
!!!!!!!
!!!!!!!
!
!!!!!!!!!!!
!
!!!
!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!!
!!!
!
!
!
!!
!
!
!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!!!!
!!!
!!
!
!
!
!
!!!!!!!
!!
!!!
!
!!
!
!
!
!
!!!
!!!!!!
!!!!!!
!!!!
!!!!
!
!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!
!
!!!!!!!!!!!!!!!!
!
!
!
!!!!
!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!
!
!
!!
!
!!!!!!!!!!!!!!!!
!!
!!!!!!
!!
!
!!!!!!
!!!
!
!!!
!
!
!!!!!!!!!!!!!!!!!!!
!!!
!
!!
!!
!
!
!
!!
!
!!
!
!
!!
!
!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!!!
!
!!!!
!
!!!!
!
!!!!
!!
!
!
!
!!!
!
!!!!!
!
!
!!!
!!!
!
!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!
!
!!
!
!!!!
!!!!!!
!!!!!!
!!!!!
!
!
!
!
!
!!!!!!!!!
!!!!!!!!!!!
!
!
!
!!
!
!!!!!!!!!
!
!
!!!!!!!!!!!!
!
!!!!!!!!
!
!
!!!!!!!!!
!!!!!!!!!!!!!!
!!
!!!!!!!
!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!
!!
!
!
!!!!
!!
!!
!
!!!
!
!
!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!
!!!
!
!!!!!
!!
!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!!
!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!!
!!
!
!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!!!!
!
!!!!!!
!
!!!
!
!!
!!
!
!!!!!!!!!!!!
!
!!!
!!
!
!
!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!!!!!!!!!
!
!!!!!
!!!!!!!
!!!!!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!!
!
!!!!!!!
!
!
!
!
!
!
!!!
!!
!!
!
!
!
!
!
!
!
!
!!
!!!
!!!
!
!
!!!
!
!
!
!!!
!
!
!
!!
!
!
!!
!
!!!
!!
!
!
!
!
!!!!
!
!!!!!!
!
!!!!
!!!!!!!!
!
!!!!!!!!!!!!!!
!
!
!!!!!!!
!!!!
!!!!
!
!!!!
!!
!!!!
!!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!
!!!!!
!
!!!!!
!
!!!!!
!
!!!!!
!
!!!!!
!
!
!!!!!!!!!!!
!!!!!!!!
!
!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!!!
!!!!!!
!
!!!
!!
!!!!!!!!!!!!!
!
!!
!
!
!
!!!!!!!!!!!!
!
!!!
!!
!
!!!!!!!!!!!
!!!!!!!
!!!!!
!
!
!
!!!!!!
!
!
!
!!!!!!!!
!
!
!!
!!!!
!!!!!!!!!!!
!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!
!
!!!
!!!!!!!!!!!
!
!!!!!!
!!!!!!!!!!!!!!!!
!!
!
!!!!
!!!
!
!
!
!
!
!!
!!!
!
!!!!!
!!
!!!!
!
!!!!!!!!!!!!!!
!
!
!
!
!
!!!
!!
!!!
!
!
!
!!!
!
!
!
!!!
!
!
!
!!!!
!!!!
!!!!!
!!!!!!!!!!!!!!!!!!!
!
!!
!!!!!!!!!!
!!!
!
!!!
!
!
!!
!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!!
!
!!
!!!!
!
!!!
!
!!!
!!!
!!!!!!!!!!!
!!!!!!!!!!!!!!!!
!
!!!!!!
!
!
!!!
!
!!!!!!!!!
!!
!
!!
!!
!!
!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!
!!!!!!!!
!!!!!!
!
!!!!!
!
!!!
!!
!!
!!
!!!!!!!
!
!!
!!!!
!!!
!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!!
!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!
!
!!!!!!
!
!!!
!
!!!!
!!!!!
!
!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!
!
!
!!!
!
!!
!!!
!!
!!!
!!
!
!
!!
!
!!!!!
!!!
!
!
!!!
!!!
!
!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!
!!!
!
!!!!!!!!!!!
!
!!
!!!!!!
!!
!!!!!!!!
!!!!
!
!
!!!!!!!
!
!
!
!!
!
!
!
!!
!
!!!!!
!
!!
!
!
!
!!!!!
!
!
!!
!!!!!
!!
!
!!
!
!
!
!
!
!
!
!!
!
!!!!
!
!!!
!
!!
!!!!!
!!!!
!
!!!
!
!!
!
!!!!
!
!!
!
!!!
!
!!
!
!!
!!!!!!!!
!!
!!!!
!
!!
!
!!!!
!
!
!
!!!!
!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!
!
!!!!!!!!
!!!
!!!!
!
!
!
!
!!!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!
!
!
!!!!
!!!!!!!
!!
!!
!
!
!!!!
!
!
!
!!!
!!
!!!
!
!
!!!
!!
!!
!!
!
!!
!
!!
!!!!!!!
!
!!!!!!!!!!
!!!!!!!!!!!
!!
!
!
!!!!!!!!!!
!!!!!!!
!!!!!!!!!!!!!
!
!!!
!
!!!!!
!
!!!!!!!
!
!!!!!!
!
!!!
!
!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!!!!
!!
!!
!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
!!!!!
!!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!!
!
!!
!!!!!!!!!!!
!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!
!!!!!!!
!!
!
!!
!
!!!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!!!!!!!!!
!!!!!!!!!!
!
!!!!!!!!
!
!
!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!
!!!!!!!
!!
!
!!
!
!
!
!!!!!!!!!!
!
!!!!!!!!!
!
!
!
!!!!!
!
!!!!!!
!!
!
!!!!!!!!!!!
!!
!!!!!!!!!!
!!!!!!!!
!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!
!!
!
!
!!!!!
!
!
!!
!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!
!
!!!!!!!!
!!!!!!!!!!!
!
!!
!!
!
!!!!!!!!
!
!
!!!!!!!!!!
!!!!!!!
!
!!
!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!
!
!
!!
!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!
!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!!!
!
!!
!
!
!
!
!!
!!
!
!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!
!!!!!
!!
!!!
!
!
!!
!!!!!!!!!
!
!
!
!
!!!!!
!
!!!!!!!!!!!!!
!!!
!!!!
!!!!!!!
!
!!!!!!!!!!!!!!!!!
!
!
!
!!
!!!!
!
!!!!!!!!!
!
!!!!!
!!!!!
!!
!
!
!
!
!
!!
!
!
!!
!
!!!!
!
!
!
!!!
!
!!!!!!!!!
!
!!!!!!!!!!
!
!!
!
!
!!!!!!!!!!!!!!!!
!
!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!!
!
!
!
!
!!!!!!
!
!!!!!
!
!!
!
!!
!
!!
!!!
!
!!!!!!!
!
!!!!
!!!!
!!!
!
!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!
!
!
!
!!!!!
!!!!
!!!!!!!!!!!!!!!
!
!!!!
!
!!!!!!!!!!
!
!
!
!
!!!!!!!!!!!!!!!
!!!!
!
!!!!!!
!
!
!
!!!!!
!
!!!!!!
!
!!!!!!!!!
!!
!!!!!!!!!!!!!
!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!!!
!
!
!
!
!
!!!
!
!
!
!!!!!!!!!!!
!
!!!!!!!!!
!
!!!!!!!!
!!!
!!
!
!!!
!
!!
!
!
!!!!
!
!
!!!!!
!
!!!!!!
!!
!!!!!!!!!
!!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!
!!
!
!!!!!!!!!
!
!!!!!
!
!!!!!!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!
!!!!!
!
!
!!
!!!
!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!
!
!!!!!!!
!
!!
!
!!!
!
!
!!
!
!!!!!!
!
!!!
!!
!!!
!
!!!!!!!
!!
!
!!!!
!!!!!!!!!!!!!!!!!!!
!!!
!!!!!
!
!
!
!!!!!!
!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!
!
!
!
!!
!
!!!!!!!!!!!!!!
!
!!!
!
!!!!!
!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!
!
!!!!!!!!!!!!
!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!
!
!!!
!!!
!!!!
!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!
!!!!!!
!!
!
!!!
!
!!!
!!
!
!
!!!!
!
!!!!!!!!!!!!!!!
!!!!!!!!!!!
!
!
!!!!!!!!
!
!
!!!!!!!!!!!
!
!!!!!!!
!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!
!
!!!!!!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!
!!!
!
!!!!!
!
!
!!
!!!
!
!!!!!!!!!!!
!!
!!!!!
!
!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!!
!!!!!!!
!
!!!!!!!!!!!
!!!!
!
!
!
!
!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!
!!!!
!!
!!
!!
!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!!!!!!!
!
!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!!!!!!!!!!!!
!
!
!
!
!
!!!!!!!!!!!!
!!!!!!
!!!!!!
!!
!
!
!!!!!!!!!!!
!!!!
!
!!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!
!!!!!
!
!!!!!!!!!!!
!
!
!
!!!!!
!
!!
!
!!
!!
!!
!
!
!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!
!!
!
!
!
!!!!!!
!!!!!!!!!!!!
!!
!!
!
!!
!
!
!!!
!
!!
!
!!
!
!!
!
!
!
!!
!!
!!!!!!!!!!!!
!
!!!!!!!!!
!!
!!!!!
!!!!!!!!!!!!!!!
!!!!!!
!!!!!!
!
!!
!!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!
!
!!!!
!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!
!
!
!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!
!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!!
!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!
!
!
!
!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!
!!
!!!
!!!!
!!!!!!!!
!!!!!
!
!!!!!!
!!
!
!!!!!!!
!
!!!!!!!!!!!!
!
!
!
!!!!
!
!
!
!!!!!
!
!!!
!
!
!!
!!
!
!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!!!!!
!
!!!
!
!
!!!!
!
!!!
!
!
!!
!!!!!!!!
!
!
!
!!!!!!
!
!
!
!
!
!
!!!!!!!!!!!!!
!
!!!!!!!!!!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!
!!!!!!!!!!
!!
!
!!!!!!!!!!!
!
!
!!
!
!!!!!!!!!!
!
!
!!
!
!
!!!!!!!!!!!
!!!!!!!!!!!
!!
!
!!
!
!!
!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!
!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!
!
!!!!!
!!
!!
!
!!
!
!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!
!!
!!!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!
!!
!!
!
!!!!!!!!
!
!!!
!
!!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!
!!!
!
!
!
!!!
!
!!!
!!!!!!!!!!!!!!!
!
!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
!
!!!
!!
!!!!!!
!!
!!!!!!!
!!!!!!
!
!!!
!!!!!!!!!!!!!!!
!
!
!!!!
!
!!!!!!
!
!!!!
!
!!!!!
!!!!!!!!!!!!
!!
!
!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!
!
!!!!
!!!!!
!!!!
!
!!!
!!!!
!!!!!!!!!!!!!
!!!!!!
!!!!
!
!!!
!!!!
!!!!!!!!!!!
!!!!
!
!!!!!!!
!
!!!!
!
!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!
!
!!!!!
!!!!
!!
!
!!!!!!!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!
!!
!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!!
!!!!!!!!!!!!
!
!
!!!!!!!!!!!!
!
!!!!!!!!!!!!
!
!!!!!!
!
!
!!!!
!
!!!!!!
!
!
!
!
!!!
!
!!!!!
!
!!!
!!!!
!!!!
!
!!
!
!
!!!!!!
!
!!!
!!!!!!!!!!
!
!!!!!!!
!!!!!!!!!!!!
!
!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!
!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!
!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!!!
!
!!!
!!!!!!!!!!!!!!!
!!!!!!!!
!!!!!!!!!!!!
!
!
!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!
!!!!
!!!!!!!!!
!
!!!!!!
!!!!!!!!!
!
!
!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!!!!!
!
!!!
!
!!!
!!!!!!!!!!!!!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!
!
!!!!
!!!!!!!!
!!!!!
!
!!!
!!
!!!!!
!!
!!!!!!!!!!!!!!!!!!!
!
!!!!
!
!
!
!!!!
!
!
!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!
!!!!!!!!
!!!!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!
!
!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!
!!!
!
!
!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!
!
!
!!!!!!!
!!!!!!!!!!!!!!!!!!!!!
!!!!!
!
!!
!
!!!!!
!
!!
!
!!!
!!
!
!
!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
!
!!
!!!
!
!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!!
!
!
!
!
!!!!!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!!!!
0 5000 10000 15000
!0.0
025
!0.0
015
!0.0
005
0.0
005
Observation id
Influence
outliers
Outlier Analysis - Checking for overly influential data points
Outliers are still valid observations
Removing them only brings the unit effect of log size to 45 %
(small change)
Decided to keep them.
Plotted Martingale Residuals
0 500000 1000000 1500000 2000000
!20
!10
010
20
Time
Beta
(t)
for
log(s
ize)
Test of Proportional Hazards
• Commonly, interaction with time is tested
• Example: A drug only effective in the first hour.
• Note: This test can also become significant when a wrong functional form is used.
• Result: p = 0.835 highly insignificant.
• A smooth plot of Schönfeld residuals show almost a perfectly straight line.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
0 2000 4000 6000 8000
02000
4000
6000
8000
Cumulative Sum of Actual Events
Cum
ula
tive S
um
of E
xpecte
d E
vents
Model Fitness - Arjas plot
• Arjas plot shows cumulative sum of expected versus cumulative sum of actual events.
• Should follow 45-degree line.
• Ours follow the line closely.
• Spearman correlation between actual vs. expected: 0.79
• The model passes all the tests
Interpretation of the results - bootstrapping
• For Mozilla classes, one unit of increase in the natural log of size caused the rate of defect fix to increase by 44%.
• We run bootstrapping to derive an estimate of the 95% confidence interval.
•Sampled 1,000 classes 1,000 times.
•For each sample, a different Cox model was produced using all observations in the sample, and a point estimate was made for the log(size) effect.
•The point estimate was 44% and the 95% confidence interval calculated via this procedure was [39% -- 49%].
Overall Results
• Note that the results using both approaches showed that the functional form of the relationship was close to a logarithmic form.
• Implies that smaller modules are proportionally more troublesome.
•Note that they do not imply that smaller modules have less defects or less probability of having defects.
•There is no threshold effect of size on defect proneness. The plot curves down for very large classes but the confidence band gets larger too.
Implications
• Practical: • A 1,000 LOC class, although 10 times bigger, is estimated to
be only 2.33 times more defect-prone (95% CI [2.13 , 2.54]).• If one has resources to inspect 10,000 LOC, it is better to
pick 100 classes of size 100-LOC as opposed to picking 10 classes of 1,000 LOC. The first approach would be estimated to be 329% times more effective (95% CI [293.70%,369.48%]).
• Theoretical:• Decomposition might have side effects• If the interface defects are responsible as suggested in
[Basili and Perricone 84], the extent of decomposition needs to be reconsidered.
Related Work
• Similar earlier observations were reported, however, by focusing on the size-defect density relationship [Hatton 97] [Hatton 98] and [Withrow 90].
• Such studies observed a U-shaped curve and identified an optimum size that minimized defect density.
• Later El Emam et al. [El Emam 02] reported that such an approach can mask the true relationship between size and defects and mislead us by showing some threshold effects.
• Indeed same points were made earlier in [Fenton and Neil 99] and even earlier in [Rosenberg 98]
• In our study, we focused on the basic size--defect relationship.
Directions for Future Research
• Replicated Studies for validation
• Studying Modularity: Is the reason interface defects as suggested earlier [Basili and Perricone 84]? What is the interplay between coupling and size?
• Studying people aspects: Experts develop larger modules?
• Studying process aspects: Larger modules inspected and tested better? Systematic or non-systematic reuse? Copy-paste into larger classes?
Conclusion
Our empirical results using a large-scale product that offered thousands of data points showed that:
•Size--defect relationship took a logarithmic form
•Defect-proneness increased as size increased
•There is no threshold value; a continuously increasing relationship
•Smaller modules are proportionally more troublesome
•Results can be immediately useful for Mozilla project
•Results also trigger many interesting research questions for the future.