large scale characterization of software vulnerability ... · muhammad shahzad, m. zubair shaﬁq,...

1545-5971 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TDSC.2019.2893950, IEEETransactions on Dependable and Secure Computing

1

Large Scale Characterization ofSoftware Vulnerability Life Cycles

Muhammad Shahzad, M. Zubair Shafiq, Alex X. Liu

Abstract—Software systems inherently contain vulnerabilities that have been exploited in the past resulting in significant revenue

losses. The study of various aspects related to vulnerabilities such as their severity, rates of disclosure, exploit and patch release, and

existence of common vulnerabilities in different products can help in improving the development, deployment, and maintenance

process of software systems. It can also help in designing future security policies and conducting audits of past incidents. Furthermore,

such an analysis can help customers to assess the security risks associated with software products of different vendors. In this paper,

we conduct an exploratory measurement study of a large software vulnerability data set containing 56077 vulnerabilities disclosed

since 1988 till 2013. We investigate vulnerabilities along following eight dimensions: (1) phases in the life cycle of vulnerabilities, (2)

evolution of vulnerabilities over the years, (3) functionality of vulnerabilities, (4) access requirement for exploitation of vulnerabilities, (5)

risk level of vulnerabilities, (6) software vendors, (7) software products, and (8) existence of common vulnerabilities in multiple software

products. Our exploratory analysis uncovers several statistically significant findings that have important implications for software

development and deployment.

Index Terms—vulnerability; disclosure; patch; exploit; diversity

✦

1 INTRODUCTION

In computer software, a vulnerability is a loophole in thesoftware code that enables an attacker to circumvent thedeployed security measures [1]. Each software vulnerabilityhas a life cycle that consists of distinct phases characterizedby the events of its discovery, disclosure, exploitation, andpatching. Each phase has a certain level of security riskassociated with it. The first phase of the life cycle of avulnerability starts when it is discovered by the vendor, ahacker, or any third-party software analyst. The risk asso-ciated with a vulnerability is particularly high if it is firstdiscovered by malicious hackers. The next phase starts withthe public disclosure of the vulnerability, which can be doneby the vendor, a hacker, or any third-party software analyst.After disclosure, the information about a vulnerability ispublicly available. Therefore, the level of risk increasesfurther because the malicious hacker community is active indeveloping and releasing zero-day exploits [2]. The aim of thevendor is to release a patch for the vulnerability as soon aspossible. The life cycle of a vulnerability ends when all usersinstall the patch to fix it. A vulnerability can be exploited byhackers at any time during its entire life cycle.

• The preliminary version of this paper titled “A Large Scale ExploratoryAnalysis of Software Vulnerability Life Cycles” was published in theproceedings of the International Conference on Software Engineering(ICSE), Zurich, Switzerland. June, 2012.

• Muhammad Shahzad ([email protected]) is with the Department ofComputer Science, North Carolina State University, Raleigh, NC, USA.M. Zubair Shafiq ([email protected]) is with the Department ofComputer Science, University of Iowa, Iowa City, IA, USA. Alex X. Liu([email protected]) is with the Department of Computer Science andEngineering, Michigan State University, East Lansing, MI, USA.

• Alex X. Liu is the corresponding author of this paper.• This work was done when Muhammad Shahzad and M. Zubair Shafiq

were PhD students under Dr. Liu.

The exploratory analysis of various aspects of vulnerabil-ities and their life cycles can uncover interesting patterns forvendors and software products that are helpful in followingways. First, a thorough analysis is helpful in the deploymentof best practices in the software development processes. Sec-ond, such analysis is useful to develop the security policiesthat can handle future attacks and threats more effectively.Third, an exploratory analysis provides insights about theprevious security incidents that are helpful in their audit.Fourth, it helps customers assess the risks associated withthe software products of a particular vendor. Finally, it helpsin implementing failure diversity by parallel use of multiplesoftware that do not have common vulnerabilities.

To the best of our knowledge, no prior work exists thatanalyzes the evolution of life cycle of different types ofvulnerabilities for different software products and vendors.The only work in this direction was from Frei et al. [3], [4]. In[3], Frei et al. studied the performance of the software indus-try as a whole but did not characterize individual vendorsbehavior. In [4], the authors only compared the vulnera-bility handling process of two vendors. Some researchershave focused on the modeling of vulnerability discoveryprocess [2], [5], [6]. The goal of such work is to estimatethe number of vulnerabilities in new software products.Another direction of work aims to study the changes in thepatching behavior of vendors in response to vulnerabilitydisclosures and the existence of competitors [7], [8]. Thesestudies analyze only small vulnerability data sets and donot cover the behavior of individual vendors. Garcia et al.studied the presence of common vulnerabilities in operatingsystems (OSes) using a set of 1887 vulnerabilities [9]. Theyconcluded that for some combinations of OSes, there areno common vulnerabilities while for others, there are. They,however, did not analyze the types of vulnerabilities thatare usually common across different OSes.



2

In this paper we make following three contributions. (1)We have aggregated a large software vulnerability data setfrom three vulnerability repositories: (a) National Vulnera-bility Database (NVD) [10], (b) Open Source VulnerabilityDatabase (OSVDB) [11], and (c) the vulnerability data col-lected by Frei et al. (FVDB) [3]. Our aggregated software vul-nerability data set contains 56077 vulnerabilities between1988 and 2013. (2) We have comprehensively analyzed soft-ware vulnerabilities along the eight dimensions mentionedin the abstract. Our observations are supported by statisticaltests for significance. (3) To systematically analyze patternsin our vulnerability data set, we have utilized associationrule mining to extract rules that represent exploitation be-havior of hackers and the patching behavior of vendors.

Our study reveals several interesting observations aboutsoftware vulnerabilities. For example, the trend of exponen-tial increase in the vulnerability disclosure rate has discon-tinued since 2006. The types of vulnerabilities prevalent inearly 2000s were mostly denial of service and executablecode (such as buffer overflow); however, around 2008, crosssite scripting, SQL injection, and PHP vulnerabilities be-came dominant. Vulnerabilities in open-source software areexploited more quickly and patched more slowly comparedto closed-source software. Although the response of vendorsto vulnerability disclosures has improved over time, thehackers are still faster than vendors in finding and exploit-ing vulnerabilities. OSes belonging to different classes suchas Windows, Mac, Linux, and Solaris contain only a fewcommon vulnerabilities, whereas OSes such as WindowsXP, Windows Vista, and Windows 7 belonging to sameclass contain a large number of common vulnerabilities.These common vulnerabilities exist probably because ven-dors carry significant amounts of code from older versionsto the newer versions of their OSes even when they claimthat the newer version is completely redesigned.

Paper Organization: In Section 2, we explain the ter-minology and notations used in the paper and providedetails about our vulnerability collection process and theaggregated data set. In Section 3, we analyze the evolutionof vulnerability disclosure rates, access methodology forvulnerability exploitation, impact of the exploitation, riskassociated with vulnerabilities, and evolution of differenttypes of vulnerabilities. In Sections 4 and 5, we study theexploitation and patching behavior of hackers and vendors,respectively. In Section 6, we cross examine the exploitationbehavior of hackers and the patching behavior of vendors.In Section 7, we study the vulnerabilities that are commonamong different OSes with a focus on failure diversity andcode reuse. In Section 8, we present the implications of ourwork followed by the related work and conclusion.

2 PRELIMINARIES

In this section, we first explain the terms and notations usedin this paper and then present the data set used for analysis.

2.1 Terminology and Notations

Vendor is an entity (an individual, a group of individuals,or an organization) that develops a software product andis responsible to keep it secure. An ideal vendor woulddiscover and patch all the vulnerabilities in its productsbefore they are exploited.

Hacker is an entity that releases exploits for thevulnerabilities in the software products. Hackers caneither be benign or malicious. Benign hackers write exploitsto show how vulnerabilities can be exploited and usuallydo not make them publicly available without first informingthe vendors. Malicious hackers write and release exploitswith the objective of adversely affecting all or a group ofpeople using the vulnerable software.Independent organization is an entity that independentlydiscovers and discloses vulnerabilities as well as theircorresponding exploits and patches but is not involved inthe development of patches or exploits.Disclosure Date (td) refers to the date when informationabout a vulnerability is made publicly available afterestablishing that the vulnerability poses a potential risk.Vulnerability disclosure always takes place after the actualdiscovery of the vulnerability or at most on the day ofactual discovery.Patch Date (tp) is the date when a vendor provides asolution (i.e. patch) for a vulnerability to neutralize thethreat posed by it. We consider only those patches that arereleased by the corresponding vendor.Exploit Date (te) is the earliest date when a hacker exploits avulnerability. An exploit can be in the form of an automaticscript, a virus, a tool, or any such thing that can breach thesecurity of a software.Exploit – Disclosure (ted) is the duration (in days) betweenthe date an exploit for a given vulnerability was providedby hackers and the date the vulnerability was disclosed.Patch – Disclosure (tpd) is the duration (in days) betweenthe date a patch for a vulnerability was released by thevendor and the date the vulnerability was disclosed.Risk Score is assigned to a vulnerability by CommonVulnerability Scoring System (CVSS) [12] and establishesthe magnitude of risk associated with that vulnerability.The numerical values of CVSS scores lie in the range [0, 10].Based on CVSS scores, we divide vulnerabilities into threecategories. Low: [0, 4); Medium: [4, 7); High: [7, 10].Access Vector (AV ∈ {Local, Adjacent Network, Network})indicates whether local access to the hardware is requiredto exploit the vulnerability or network access is sufficient.Access Complexity (AC ∈ {Low, Medium, High}) is ameasure of the complexity of the attack required to exploitthe vulnerability.Integrity Impact (Ii ∈ {None, Partial, Complete}) measuresthe potential impact of a successfully exploited vulnerabilityon the integrity of the system. Integrity refers to thetrustworthiness of information.

2.2 Data Set

In this section, we provide details of our aggregate data andits basic statistics. We have collected vulnerability informa-tion from three sources: (1) NVD [10], (2) OSVDB [11], and(3) FVDB [3].

2.2.1 Data Aggregation

NVD and FVDB identify each vulnerability with CommonVulnerability and Exposures Identifier (CVE-ID) [13]. OS-VDB also provides CVE-IDs of about 70% of vulnerabilities.We leverage the CVE-IDs to aggregate the vulnerability datafrom the three sources. We take CVSS scores, CVSS-vector



3

metrics (i.e., access vector, access complexity, and integrityimpact), vendor and product names, text description, anddisclosure dates from NVD. From OSVDB and FVDB, wetake disclosure dates, exploit dates, and patch dates.

The total number of vulnerabilities in our aggregate dataset is 56077 and the number of vulnerabilities for which wehave the disclosure dates, patch dates, and exploit dates are56077, 9666, and 15363, respectively. Due to the sheer size ofthe data set, it is infeasible to find the unknown exploit andpatch dates manually. To systematically conduct our study,we make following two subsets of our aggregate data set:ED-subset consists of 15363 vulnerabilities and containsthose vulnerabilities for which both exploit and disclosuredates are known. PD-subset consists of 9666 vulnerabilitiesand contains those vulnerabilities for which we have bothpatch and disclosure dates. A total of 1424 vulnerabilitiesare common in both ED- and PD-subsets. Note that all datesin the two subsets are taken from OSVDB and FVDB.

2.2.2 Selection of Vendors and Products

The aggregate data set contains vulnerabilities from morethan 12000 vendors and over 24000 software products.Figure 1 plots the number of vulnerabilities of each vendorin descending order. It can be seen that over 95% of the ven-dors have fewer than 10 vulnerabilities. To make statisticallysound observations, we focus our attention on the top 8 ven-dors, each of which has at least 500 vulnerabilities. We selectclosed-source vendors including Microsoft, Apple, and Ora-cle, open-source vendors including Linux, Mozilla, and RedHat, and hybrid vendors including Sun and Google. Linuxis not a vendor rather it represents generic Linux kernel. Wealso study popular software products of these vendors thatinclude Internet Explorer, Safari, Firefox, Chrome, Windows,MAC OS X, Solaris, and several Linux distributions.

3 GENERAL VULNERABILITY ANALYSIS

In this section, we study the trends in vulnerability disclo-sure and CVSS-vector metrics over the past two decades.We also categorize the vulnerabilities into groups and studytheir evolution.

3.1 Vulnerability Disclosure Trend

The rate of vulnerability disclosures experienced an expo-nential growth since 1997 and lasted till 2006 as can be seenin Figure 2. The vertical lines in the figure show the num-ber of vulnerabilities disclosed every month since January1990 and the dashed line shows the cumulative numberof vulnerabilities. The number of vulnerability disclosureshas not been increasing since 2006, rather it decreased from2006 to 2011 despite the ever increasing use of softwareproducts. In 2012, the number of vulnerability disclosuressaw a significant increase again.

3.2 General Trend of CVSS Scores

Recall from Section 2 that every vulnerability has an asso-ciated risk quantified by CVSS score. Figure 3 shows thebox plots of CVSS scores for vulnerabilities in the productsof the selected vendors. We note that CVSS scores of mostvulnerabilities in our study lie in medium to high range. Themedian CVSS scores for closed-source vendors are greaterthan the median scores for open-source vendors.

3.3 Evolution of CVSS-Vector Metrics

Figures 4(a) to 4(c) show the evolution of three metricsof CVSS-vector. For each metric, we have calculated thepercentage of vulnerabilities corresponding to each of itsthree values for every month since January 1990. We observefrom Figure 4(a) that the percentage of remotely exploitablevulnerabilities has been increasing since 1998. The fact thatmost computer systems are connected to Internet has madeit possible for hackers to exploit these systems remotely.Figure 4(b) shows the change in access complexity of vul-nerabilities over the years. We observe that the percentageof low complexity vulnerabilities has decreased over timeindicating that the hackers have to use more sophisticatedtechniques to exploit new vulnerabilities. From Figure 4(c),we observe a reduction in the percentage of vulnerabilitieshaving complete integrity impact. This shows that modernsoftware are more robust even when they are compromised.

3.4 Evolution of Different Types of Vulnerabilities

Our data sources do not specify the type of vulnerabilitiesin the data set. To determine the prevalent types of vul-nerabilities and to study their evolution, we utilized unsu-pervised k-means clustering and agglomerative hierarchicalclustering to cluster the vulnerabilities of different typesinto groups. For this, we leveraged the text description ofvulnerabilities in NVD.

We compared words in the text descriptions with wordsin everyday non-technical news articles and removed thematching ones. This left us with keywords that describe thevulnerabilities. We manually went through them, removedthe irrelevant ones, and used the remaining as attributes forclustering.

The keyword attributes are binary in nature, where an at-tribute value of 0 represents the absence of a keyword in descrip-tion of a vulnerability and 1 represents the presence. To quantifysimilarity between text descriptions during clustering, we useJaccard’s coefficient [14]. To define Jaccard’s coefficient betweena given pair of vulnerabilities, let Mxy represent the number ofattributes whose value for the first vulnerability is x ∈ {0, 1} andfor the second vulnerability is y ∈ {0, 1}. The Jaccard coefficientbetween a given pair of vulnerabilities is defined as the ratio of thenumber of attributes that are present in both vulnerabilities to thenumber of attributes that are present in at least one of them, i.e.,

J =M11

M01 +M10 +M11

(1)

The motivation behind using Jaccard coefficient is that it ignoresall those attributes that are not present (i.e., their values are 0) ineither of the two vulnerabilities in the given pair of vulnerabilities.It is important to ignore all such attributes because such attributesdo not provide any information about the “similarity” betweenthe two vulnerabilities. This is also the reason behind not usingEuclidean distance to calculate similarity because in Euclideandistance, if the values of a pair of corresponding attributes in twovulnerabilities are 0, that pair of attributes contributes as much tothe similarity measure between the two vulnerabilities as a pair ofcorresponding attributes whose values are both 1.

It is well known that k-means clustering algorithm iswell suited for large data sets with large number of at-tributes, where k represents the number of clusters it makes.The k-means clustering algorithm produces the best clusters



4

0 20 40 60 80 10010

0

101

102

103

104

Percentage of vendors (total: 12482)

Nu

mb

er

of

vu

lne

rab

ilit

ies

Fig. 1. # of vulns. for each vendor

0

10000

20000

30000

40000

50000

60000

0

250

500

750

1000

1250

1500

1990

1991

1992

1993

1994

1995

1997

1998

1999

2000

2001

2002

2004

2005

2006

2007

2008

2009

2011

2012

Monthly

vulnerability

disclosures

Year

Monthly Disclosures

Cummulative Disclosures

Cummulative

disclosed

vulnerabilities

Fig. 2. Vuln. disclosure trend

0

2

4

6

8

10

12

CVSS

Scores

Fig. 3. Boxplots of the CVSS scores

0

10

20

30

40

50

60

70

80

90

100

1990

1991

1992

1994

1995

1996

1998

1999

2000

2002

2003

2004

2006

2007

2008

2010

2011

2012

Access

Vector

Year

Local Access

Adjacent Network

Network

(a) Access Vector Evolution

0

10

20

30

40

50

60

70

80

90

100

1990

1991

1992

1994

1995

1996

1998

1999

2000

2002

2003

2004

2006

2007

2008

2010

2011

2012

Access

Complexity

Year

Low Complexity

Medium Complexity

High Complexity

(b) Access Complexity Evolution

0

10

20

30

40

50

60

70

80

90

100

1990

1991

1992

1994

1995

1996

1998

1999

2000

2002

2003

2004

2006

2007

2008

2010

2011

2012

Integrity

Impact

Year

None

Partial

Complete

(c) Integrity Impact Evolution

Fig. 4. Evolution of Access Vector, Access Complexity, and Integrity Impact of vulnerabilities

when a correct value of k is used and the algorithm isinitialized with appropriate values of the k centroids. Todetermine the correct value of k, we performed agglomer-ative hierarchical clustering with Ward’s linkage [15] on 10bootstrapped samples of the data set, sampled at a samplingrate of 20%. Hierarchical clustering with Ward’s linkage isthe hierarchical equivalent of the k-means clustering andis often used to generate initial centroid values for the k-means algorithm [16]. We generated different number ofclusters, ranging from 1 to 10, for each bootstrapped sampleand extracted dominant keywords from resulting clustersto determine the type of vulnerabilities in each cluster.The dominant keywords of a cluster are the keywords thatappear in the text description of over 80% of vulnerabilitiesin that cluster. We observed that when we increase thenumber of clusters from 7 to 8, the two new clusters thatresult from the splitting of one of the 7 clusters represent thesame vulnerability type. Therefore, we chose k = 7. We usedthe average of centroid values from the 10 bootstrappedsamples as the initial values of the k centroids for the k-means algorithm.

Table 1 tabulates the dominant keywords extracted fromthe centroids of the 7 clusters. From the observed keywords,we label the clusters as PHP vulnerabilities (PHP), exe-cutable code (EXE), denial of service (DoS), directory traver-sal (DT), SQL injection (SQL), cross-site scripting (XSS),and miscellaneous vulnerabilities (Misc). A major portion ofEXE is constituted by buffer overflow vulnerabilities. Figure5 shows the number of vulnerabilities belonging to eachcluster disclosed each year since 1999.

To illustrate the usefulness of augmenting agglomerative hier-archical clustering with the k-means clustering, Table 2 tabulatesthe dominant keywords extracted from the centroids of the 7clusters obtained using k-means clustering without augmentingit with agglomerative hierarchical clustering. On comparing Table

TABLE 1Results of vulnerability clustering: k-means with hierarchical clustering

Keywords Label Size1 php, parameter, execute, file, code, url, inclusion PHP 4.37%2 execute, code EXE 14.1%3 service, denial DoS 17.7%4 directory, files, traversal DT 4.52%5 injection, sql, execute, commands SQL 10.5%6 cross, scripting, site, script, html, inject, web, xss XSS 12.4%7 – MISC 36.5%

2 with Table 1, we see that out of the seven labels and theircorresponding keywords shown in the two tables, six labels arethe same while the seventh label is different: directory traversal(DT) in Table 1 and buffer overflow (BO) in Table 2. If wecompare the keywords corresponding to BO in Table 2 with thekeywords of EXE in the same table, we see that they both contain“execute” and “code”, i.e., all vulnerabilities clustered by simplek-means in BO cluster actually are EXE vulnerabilities. On theother hand, when we used the more sophisticated method, i.e.,k-means augmented with hierarchical clustering, we obtained acluster that kept all buffer overflow vulnerabilities inside the EXEcluster, and made a new cluster containing vulnerabilities of typedirectory traversal (DT). This identification of a new cluster andkeeping the vulnerabilities of the same type in a single clusterdemonstrates the supervisor performance of k-means augmentedwith hierarchical clustering compared to the simple k-meansclustering.

TABLE 2Results of vulnerability clustering: simple k-means

C# Keywords Label Size

1 php, parameter, execute, file, code, url PHP 8.32%2 execute, code EXE 7.25%3 buffer, execute, code, overflow BO 10.2%4 service, denial DoS 14.2%5 injection, sql, execute, commands SQL 11.2%6 cross, scripting, site, script, html, inject XSS 12.3%7 – MISC 36.6%



5

From Table 1, we observe that only DoS, EXE, andDT vulnerabilities were prevalent till 2001. DoS and EXEconstitute a major portion of software vulnerabilities eventoday which indicates that the vendors have not been ableto devise effective strategies to limit these types of vulnera-bilities. Since 2002, we observe an increase in the XSS vul-nerabilities, which peaked in 2006. PHP vulnerabilities wereprevalent in 2006 and 2007 and SQL vulnerabilities wereprevalent between 2005 and 2010. These trends highlightthe shift in focus of hackers to exploit new services as theybecome popular.

0

200

400

600

800

1000

1200

1400

1600

'99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11 '12

PHP

Exe

DT

SQL

DoS

XSS

Number of vulnerabilities of each

type

Years

Fig. 5. Evolution of vulnerability clusters over the years

4 EXPLOITATION BEHAVIOR

In this section, we study the behavior of hackers in releasingexploits for vulnerabilities. We analyze trends in ted val-ues (defined in Section 2.1) of vulnerabilities. The analysispresented in this section is done on ED-subset. We studyfollowing three ranges of ted values.

ted < 0 shows that an exploit for a given vulnerabilitywas released before its public disclosure. The vulnerabilitiesfalling in this range represent a big threat to the security ofend-users as the vendor could be oblivious about them. Atotal of 2.8% vulnerabilities fall into this range.

ted = 0 refers to the case when an exploit for a givenvulnerability was released on the day it was disclosed. Atotal of 88.2% vulnerabilities fall into this range. The exploitscorresponding to vulnerabilities with ted ≤ 0 are called zero-day exploits.

ted > 0 means that the exploit for a vulnerability wasreleased after its public disclosure. The vulnerabilities forwhich ted > 0 represent the case where a vulnerability isdisclosed by the vendor or an independent organization andthe hackers used this information to release an exploit inmore than a day. 9.7% vulnerabilities fall in this range. To domore detailed analysis, we subdivide this range into threeparts: (1) 0 < ted ≤ 7 gives us the percentage of exploitsreleased within a week of disclosure, (2) 7 < ted ≤ 30gives us the percentage of exploits released after a weekand within a month of disclosure, and (3) ted > 30 gives usthe percentage of exploits released more than a month afterthe disclosure.

4.1 Evolution of Exploitation

To extract the dominant trends in the exploitation behaviorof hackers, we first divided the vulnerabilities in the ED-subset into groups, where each group contains vulnerabili-ties disclosed in a distinct year. Then we subdivided the vul-nerabilities in each group into five subgroups corresponding

to the five ranges of ted. We then calculated the percentageof vulnerabilities in each subgroup (called the percentage sizeof the subgroup) in its respective group and plotted theresults in Figure 6 in the form of stacked bars where each barcorresponds to the group of vulnerabilities disclosed eachyear and each block in every bar represents the percentagesize of the corresponding subgroup in its respective group.The number inside each block is the value of the percentagesize of the corresponding subgroup. The number at the topof each bar represents the total number of vulnerabilitiesin the corresponding group. All figures in rest of the paperhave been made using similar methodology.

5 4 4 6

91 94 93 88

86

71

80 86 85 86

98 97 89 91

4 8

15

9 6 9 8 4 4

5 4 4 7

43 156 243 291 619 483 1471 2215 3022 1982 2782 1400 612 44

0

20

40

60

80

100

'98 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Exploited

vulnerabilities

Years

Fig. 6. Yearly change in exploitation behavior for different ted ranges

It can be seen from Figure 6 that for majority of vul-nerabilities, exploits are released on their disclosure dates(having ted = 0). This is primarily because independentorganizations often disclose vulnerabilities along with theircorresponding exploits. Till 2004, the percentage size of thesubgroup of ted < 0 was non-negligible which shows thatthe hackers were finding a significant number of vulnerabil-ities themselves and exploiting them. At the same time weobserve a decrease in the percentage size of the subgroupof ted = 0. This does not mean that hackers were gettingsluggish because we also observe a significant increase inthe total number of exploited vulnerabilities. Since 2004, weobserve a decrease in the percentage size of the subgroupof ted < 0 and an increase in the percentage size of thesubgroup of ted = 0. This shows that benign hackers arebecoming more active compared to malicious hackers indiscovering vulnerabilities.

4.2 Exploitation of Types of Vulnerability

We now consider the exploitation of different types ofvulnerabilities. Figure 7 has been made in the same wayas Figure 6 except that now the groups are the types ofvulnerabilities. It can be seen that for over 80% of vulner-abilities of each type (except EXE), exploits are released onor before the day of disclosure. In case of EXE, a significantpercentage of vulnerabilities is exploited several weeks afterthe disclosure. One possible reason for slow exploitation ofEXE vulnerabilities can be that EXE vulnerabilities are oneof the oldest types of vulnerabilities, as seen in Figure 5. Thesoftware industry has evolved to make the products moresecure against exploitation of such vulnerabilities, making itharder for the hackers to find and exploit them.

4.3 Exploitation Trend for Vendors and Products

We study the behavior of hackers in releasing exploits fordifferent vendors and their respective products. Figures 8and 9 show the exploit data for the selected vendors and



6

5 3

93

70

93 94

81

93

3

15

4 3

9

4

5

5 5

1741 1498 1398 3157 1421 2623

0

20

40

60

80

100

PHP Exe DT SQL DoS XSS

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Exploited

vulnerabilities

Vulnerability Type

Fig. 7. Exploited vulnerabilities w.r.t vulnerability types

products respectively. These figures have been made forvendors and products in the same way as Figure 7 was madefor vulnerability types.

6 3 2 3 8 10

5

70 76 70

59

61 62

58

13 11

11

17

13

23

19

6 5

7 10 4 8

5 5 10 10 14

4 10

602 235 121 88 76 127 79

0

20

40

60

80

100

Microsoft Apple Sun Oracle Linux Mozilla Redhat

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Exploited

vulnerabilities (Vendors)

Fig. 8. Exploited vulnerabilities w.r.t vendors

9 8 2 3 6 8 3 5 8 5

13

48 48

78 71 62 61

63 65

73 74 57

21 21

12

15

12 13 20 16

13

7 24 12 13

6

4 4

3 5 7

10 10 5 6

16 14 10 8 7 5

128 116 121 72 50 76 30 37 192 42 76

0

20

40

60

80

100

Win XP Win

2000

OS X OS X

Server

Solaris Linux

Krnl

Entp

Linux

RedHat

Linux

Int Exp Safari Firefox

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Exploited

vulnerabilities (Products)

Fig. 9. Exploited vulnerabilities w.r.t products

Let’s first compare the vulnerability exploitation in openvs. closed-source vendors. In comparison to closed-sourcevendors, for open-source vendors e.g., Linux, Red Hat etc.,a comparatively larger percentage of vulnerabilities is ex-ploited before the day of disclosure. This shows that themalicious hackers are faster in exploiting the vulnerabilitiesin open-source vendors. This is probably due to the accessto the source code of open-source software products.

To make statistically significant observations, we do sta-tistical hypothesis testing. As our samples for open-sourceand closed-source vendors contain large number of datapoints, the most appropriate statistical test for this scenario(and all the subsequent scenarios) is the standard one-tailedt-test [17]. The t-test is considered to be the most appropriatewhen the number of data points in the samples are large(>50) regardless of the distributions they come from.

To remove any bias in testing, we state the null hy-pothesis as: the mean value of ted for open-source ven-dors, µted(O), is equal to the mean value for closed sourcevendors, µted(C). The alternative hypothesis is: µted(C) isgreater than µted(O). We apply the right tailed t-test to thenull hypothesis. If the null hypothesis is rejected, it would be

statistically sound to claim that the average time to exploit avulnerability in closed-source software is larger comparedto open-source software. We give a general equation forhypothesis testing that will be used for all the subsequenttests: H0 : µA(X) = µB(Y )

H1 : µA(X) > µB(Y ) (2)

where X=C represents closed-source vendors, Y=O repre-sents open-source vendors, and A=B=ted represents thatthe data points of ted are being considered. We do thehypothesis testing for a 95% confidence interval i.e., α=0.05.Our test resulted in a p-value of 0.003 which is much smallerthan α, thus we reject H0 to accept H1. Thus, it is statisticallysound to state that the exploitation of vulnerabilities (bymalicious and benign hackers collectively) in closed-sourcesoftware is slower compared to open-source software.

Figure 8 shows that hackers release most exploits on orbefore the disclosure dates for Microsoft and Apple. This isprimarily because malicious hackers find it more rewardingto exploit the products that have wider market capitaliza-tion. Similarly, benign hackers consider it more important toapprise such vendors about potential exploitation becausemalicious exploitation of their products can adversely affecta large number of users.

For the selected products, we see the similar trend inFigure 9 as for vendors in Figure 8 except for Windows.The percentage of exploited vulnerabilities for Windowstill disclosure date is less than that of OS X but at thesame time the percentage of exploited vulnerabilities forWindows before disclosure is greater than that for OS X.In fact, the mean value of ted for Windows is negative whilethat for OS X is positive. The t-test with X= “OS X”, Y=“Windows”, and A=B=ted yields p=0.031 proving that theexploitation in Windows is quicker compared to OS X.

Among web browsers, Firefox has the smallest percent-age of vulnerabilities exploited till disclosure date comparedto Internet Explorer and Safari but at the same time hasthe highest percentage of vulnerabilities exploited before thedisclosure. The t-test with X = “Safari” and Y = “InternetExplorer” yields p = 0.05 showing that exploitation inInternet Explorer is quicker compared to Safari. The t-testwith X = “Safari” and Y = “Firefox” yields p = 0.09, andtherefore, fails to reject the null hypothesis.

4.4 Exploitation Behavior: CVSS Scores

Recall from Section 2.1 that we divide vulnerabilities intothree risk categories based on CVSS scores: low, medium,and high.

Figure 10 has been generated in the same way as Figure8 except that we plotted the vulnerabilities belonging to low,medium, and high risk categories separately. The white lineswith round markers represent the percentage of total vulner-abilities belonging to low, medium, or high categories.

It is intuitive to think that hackers would be less in-terested in exploiting low risk vulnerabilities because suchvulnerabilities cause less damage. This is exactly what themarkers for low risk vulnerabilities show in Figure 10. Thebars in Figure 10 show that the percentage of mediumrisk vulnerabilities for which exploits are released on orbefore the disclosure date is greater than that for high riskvulnerabilities for all closed-source vendors and some open-source vendors.



7

7 6 4 13

4

16 6

19 9 9 8

81 76

62

85

78 72 78 78

63 50 65

50

48 69

63

63

58

71

50

55

63

11 11

15

9 15

22 10

10 13

19

15

12

9

21

19

26

17

30

26

10 12

6 6

8 13

7

15

12 10 11

5 4 7 4 6 4

16 13 6

19 12 16 16

5 10 13 8

0

20

40

60

80

100

L M H L M H L M H L M H L M H L M H L M H


> 30 days

+30 days

+7 days

0 day

< 0 days

percentage

Percentage

of Exploited

vulnerabilities (CVSS

Scores)

Fig. 10. Exploited vulnerabilities w.r.t CVSS scores

4.5 Interesting Exploitation Rules

Now we present some interesting association rules aboutthe exploitation behavior in the products of the short-listedvendors. We used the implementation of the Apriori associ-ation rule mining algorithm [18] in WEKA [19] to extract therules with confidence greater than 95%. For association rulemining, we used following attributes of each vulnerability:Vendor Name <vnd>, Product Name <prd>, VulnerabilityType <typ>, Severity <sev>, ted, and tpd. For the rulespresented in this section, we used ted as class attribute.

We found that in case of Microsoft, for the majority ofthe vulnerabilities including DoS, XSS, and EXE, exploitsare released on the day the vulnerabilities are disclosed.One such rule obtained from association rule mining is:vnd=Microsoft typ=XSS sev=H → ted=0-day.

In case of Apple, the exploits for the majority of thevulnerabilities are released on or before their disclosuredates. For example, as shown in the following rule, vul-nerabilities in Safari browser are mostly exploited on theday of disclosure: vnd=Apple prod=Safari typ=EXE sev=H

→ ted=0-day.For Solaris, association rules show that high risk vulnera-

bilities are exploited on the day of disclosure while mediumrisk vulnerabilities are mostly exploited within a week aftertheir disclosure. The latter trend is shown by the followingrule: vnd=Sun prod=Solaris sev=M → 0<ted≤ +1 week.

For Mozilla, we get interesting rules showing thathackers do not exploit a vulnerability that has alreadybeen patched while they quickly exploit those that havenot been patched. Two rules stating this observation are:(1) vnd=Mozilla prod=Firefox typ=EXE tpd=0-day → ted>

+1 month, (2) vnd = Mozilla prod=Firefox typ=EXE +1 week

<tpd≤+1 month → ted=0-day.

5 PATCHING BEHAVIOR

Now we study the behavior of vendors in providing patchesfor vulnerabilities in their products. For this, we studythe trends in tpd values of vulnerabilities. The analysispresented in this section is based upon PD-subset. The threeranges for tpd that we study are described below.

tpd < 0 shows that the patch for a given vulnerabilitywas released before its public disclosure. A total of 10.1%vulnerabilities have tpd < 0 which is greater than thecorresponding value for ted < 0. A possible reason is thatthe independent organizations inform the vendors about thevulnerabilities they discover and give them enough time torelease a patch before disclosing the vulnerabilities. Another

possible reason is that the vendors discover the vulnerabili-ties themselves and release patches without disclosing them.

tpd = 0 means that the patch for a vulnerability wasreleased on the disclosure day. A total of 62.2% vulnerabili-ties have tpd = 0. The patches corresponding to tpd ≤ 0 arecalled zero-day patches.

tpd > 0 refers to the case where the patch for a givenvulnerability was released after its public disclosure. In ourPD-subset, 27.7% of all the vulnerabilities are patched morethan a day after their disclosures. We further subdivide therange tpd > 0 into the same three parts as in Section 4.

The t-test with A = tpd, B = ted, and X = Y =“aggregate data set” yields p ≈ 0 which leads us to ac-cepting the alternative hypothesis that, compared to hackers(benign and malicious collectively), vendors take more timeon average to patch a vulnerability.

5.1 Evolution of Patching Behavior

In Figure 11 we observe that till 2005, the percentage ofvulnerabilities patched on or before disclosure dates consis-tently decreased. Keeping in view the fact that independentorganizations inform the vendors about vulnerabilities wellbefore disclosing them [20], such a poor patching behaviorof vendors indicates that security was not a major concernfor vendors at that time. However, we see a significantimprovement after 2005. Since 2008, vendors have been pro-viding patches for more than 80% of total vulnerabilities tilltheir disclosure dates. A possible reason for this can be thatit has become more common to not report vulnerabilitiespublicly, rather, the vendors pay for vulnerabilities.

10 14 16 17

11 11 7 6 6

21

12 7 7

47

61 50 41

30

34 31

31 36

54

66 80 84

89 16

8

10 17

10 12 12

13

22

14

6

4

5

14

13 5

12 13 16 21

9

9

4

32

7 13

21

31 30 29 28 27

16

19 71 212 240 399 336 507 762 854 867 883 1624 2429 463

0

20

40

60

80

100

'98 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of patched

vulnerabilities

Years

Fig. 11. Yearly change in the patching behavior for different tpd ranges

5.2 Patching of Types of Vulnerabilities

From Figure 12, we observe that vendors are generallyslower in patching the PHP vulnerabilities. A possible ex-planation is that for the majority of PHP vulnerabilities, theexploits are released by the benign hackers as implied bythe large percentage size of the subgroup corresponding toted = 0 in Figure 7. The vendors are quicker in patchingthe EXE vulnerabilities as these vulnerabilities pose highsecurity risk because they usually have higher CVSS scores.

5.3 Patching Trend for Vendors and Products

Here we study the behavior of the selected vendors inpatching the vulnerabilities in their products. Figures 13 and14 show the patch data for selected vendors and products.

Closed-source vendors are typically profit based organi-zations and have more resources to secure their productsas compared to open-source vendors. Therefore, we ex-pect better patching behavior from closed- source vendors.Figure 13 confirms this intuition as Microsoft, Apple, and



8

23

9

21 26

7

16

35 65 48

50

63

67

17

6 13

15

6

6

13 8 5

3

8

3 12 12 13

6

15 7

52 2486 207 301 2190 825

0

20

40

60

80

100

PHP Exe DT SQL DoS XSS

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Patched

vulnerabilities

Vulnerability Type

Fig. 12. Patched vulnerabilities w.r.t vulnerability types

4 5 7 4 10

5

19

3

76 78

78

64

16

55 27

93

3

5 4

29

13

14

12

5

5 2

17

10

16

12 6 8

45

16

27

1531 1003 298 666 327 392 281 178

0

20

40

60

80

100

Microsoft Apple Sun Oracle Linux Mozilla Redhat Google

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Patched

vulnerabilities (Vendors)

Fig. 13. Patched vulnerabilities w.r.t vendors

Oracle release patches for about 70% or more of all thevulnerabilities on or before disclosure dates. In comparison,we observe significantly smaller percentages and quantity ofpatched vulnerabilities for open-source vendors. Applyingthe t-test with X = O, Y = C, and A = B = tpd, we ob-tained p ≈ 0 which statistically justifies the observation thatopen-source vendors are slower in patching as compared toclosed-source vendors.

We see the similar trend for the selected products inFigure 14 as for vendors in Figure 13. We also see thatover 85% of the vulnerabilities in Windows are patchedon or before the disclosure dates. If we compare Figure 14with Figure 9, we observe that the percentage of zero-daypatches for Windows is greater than the percentage of zero-day exploits.

Among web browsers, Figure 14 shows that GoogleChrome is the fastest patched web browser followed byApple’s Safari. t-test with Y = Chrome and X = (In-ternet Explorer, Safari, Firefox) respectively yields p =(0, 0.024, 0) confirming our observation about Chrome. t-test with Y = Safari and X = (Internet Explorer, Firefox)yields p = (0.009, 0.0786) confirming that Safari is patchedmore quickly compared to Internet Explorer but the test failsto reject the null hypothesis of Safari against Firefox.

5

18 10 8

36

4 5

82 80 75 80

59

16

31

22

64

86

58

96

4 5 8

7

10

13

8

14

4 11

7 6 5

17

19

8 5 11

10 9 6 5 8

45

32

20 22

8 16

385 373 535 394 73 326 118 111 335 173 324 169

0

20

40

60

80

100

Win XP Win

2000

OS X OS X

Server

Solaris Linux

Krnl

Entp

Linux

RedHat

Linux

Int Exp Safari Fire!

fox

Chr!

ome

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of Patched

vulnerabilities (Products)

Fig. 14. Patched vulnerabilities w.r.t products

5.4 Patching Behavior: CVSS Scores

One would expect the vendors to be quicker in patchingthe medium and high risk vulnerabilities compared to lowrisk vulnerabilities. This is exactly what we observe inFigure 15. Open-source vendors are slower as comparedto closed-source vendors for vulnerabilities belonging to allrisk categories.

6 12 5 8 8 3 7 7

12 12 9 6 4

18 16 21

6 2

77

70

78

65

80

79

100

78

75

88

82

37

8

13

32

29

52

62

29

32

22

100

86

96

13 6 7

13 13

52

17

10

10 21

11

16

12 10 14

11 5

5 7

18 14

18 12

13

7

10 16 18

5 6

16 11 10

6 6 12

6

50 50

28 29

18 12

31 26 26

0

20

40

60

80

100

L M H L M H L M H L M H L M H L M H L M H L M H

Microsoft Apple Sun Oracle Linux Mozilla Redhat Google

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

Percentage

of Patched

vulnerabilities (CVSS

Scores)

Fig. 15. Patched vulnerabilities w.r.t CVSS scores

5.5 Interesting Patching Rules

We present association rules about patching behavior ofvendors extracted using tpd as class attribute.

Microsoft is quicker in patching vulnerabilities in Win-dows as compared to its remaining products. The follow-ing two rules show this: (1) vnd=Microsoft prod=Windows

XP typ=EXE → tpd=0-day, (2) vnd=Microsoft prod=Internet

Explorer typ=EXE→ tpd>+1 month.Apple also patches vulnerabilities in its operating sys-

tems as soon as they are disclosed. The following rulehighlights this trend: vnd=Apple prod=MAC OS typ=EXE →

tpd=0-day. Following rule shows that Apple generally takesabout a week to fix DoS vulnerabilities even if they areexploited on the day they are disclosed: vnd=Apple prod=MACOS typ=DoS → 0<tpd≤+1 week. Other rules show that Appletakes about a month after disclosure to patch the EXEvulnerabilities although it is one of the are prevalent types.

Sun is quicker in patching all kinds of vulnerabilitiesexcept XSS. Sun fixes DoS vulnerabilities before their dis-closure, which is a better performance as compared toMicrosoft and Apple. For Mozilla, EXE vulnerabilities aremostly patched on or before the day of disclosure; however,SQL vulnerabilities are not patched for months.

6 PATCHING VS. EXPLOITATION

In this section, we compare the quickness of vendors withhackers. We study the trends in tpe values of vulnerabilitiespresent in the PE-subset.

tpe < 0 shows that a vulnerability was patched beforeits exploitation irrespective of whether or not it was dis-closed. The inherent time-lag between the release of patchesby vendors and their installation by end-users motivatesthe hackers to write exploits for vulnerabilities even aftercorresponding patches have been released. In our PE-subset,31.7% of all the vulnerabilities fall in this range.

tpe = 0 means that a given vulnerability was exploitedon the day its patch was released. 21.8% of the vulnerabili-ties fall in this range.

tpe > 0 shows that an exploit for a given vulnerabilitywas released before the vendor patched it. A total of 46.4%of vulnerabilities have tpe > 0. The larger percentage of



9

tpe > 0 compared to tpe < 0 indicates that hackers havegenerally been quicker in exploiting the vulnerabilities ascompared to vendors in patching. This observation affirmsthe result of the first t-test presented in Section 5.

6.1 Patching vs. Exploitation: Over the Years

From Figure 16 we can see the same behavior as observedin Section 5.1: patching response of vendors was poor tillaround 2005 and a large percentage of vulnerabilities wasbeing exploited before being patched. In 2006, the situa-tion was so bad that the patches for about 38% of thevulnerabilities were released more than a month after theirexploitation. However, after 2007 a significant improvementcan be observed in the vendor response. It is encouragingto see that since 2008, over 70% of all the vulnerabilitieshave been patched on or before the release date of theirexploits. From the discussion in this section and Sections4.1 and 5.1, we can conclude that the security state of thesoftware industry has been improving for the last 3 years.

22

38

26 28 23 24 27 31 29

47 39 41

69

20

67 26

30 19

22 18 18 13 17

22 35 31

8

40

24

10

15 12

8

14

6

13

17

17 14

23

11 9

10

8 14

18

18

12

16

8 4

40

24 30 29 33

22

38

25

5 6 10

5 9 34 50 100 111 204 228 139 124 143 127 136 13

0

20

40

60

80

100

'98 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of vulnerabilities for different t p

e ranges

Years

Fig. 16. Yearly change in patching vs. exploitation trend for tpe

6.2 Patching vs. Exploitation: Vendors & Products

It can be seen from Figure 17 that for all vendors exceptOracle and Sun, the percentage size of the subgroups cor-responding to tpe > 0 is greater than that for tpe < 0. Themagnitude of the difference between the percentage sizesof tpe < 0 and tpe > 0 can serve as a measure to gaugethe agility of the vendors in reference to hackers. We cansee that among the vendors, only Oracle and Sun are fasterthan hackers, whereas hackers are, on average, faster thanall other vendors. From Figure 18 we can see that, comparedto hackers, Microsoft and Sun are quicker for Windows andSolaris respectively.

31

20

36

58

28

16

35

24

22

44

16

10

4

19

5

14

12

9

13

22

15 9 19

4

5

15 28

19

31 26

4 12

35 30

12

297 125 25 43 40 69 52

0

20

40

60

80

100


.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of vulnerabilities

for t p

e ranges (Vendors)

Fig. 17. Patched vulnerabilities for vendors relative to exploit dates

6.3 Patching vs. Exploitation: CVSS Scores

From Figure 19, it can be seen that for Microsoft andApple, approximately the same percentage of vulnerabilitiesbelonging to medium and high risk categories are patchedbefore the release of their exploits. However, the percentage

48 55

18 21

50

28 22

33

16

25 22

17

17

27

30

30

10 26

33

8

6

4

17

15

20

13

9

22

9

6 14

5

6

16 13

15

22

6

17

33

29

20 22 21

35

22

6

50 56

27

84 82 82 53 10 40 23 18 88 16 51

0

20

40

60

80

100

Win XP Win

2000

OS X OS X

Server

Solaris Linux

Krnl

Entp

Linux

RedHat

Linux

Int Exp Safari Firefox

.

> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

of vulnerabilities

for t p

e ranges (Products)

Fig. 18. Patched vulnerabilities for products relative to exploit dates

of vulnerabilities for which tpe = 0 is generally greaterfor medium risk vulnerabilities as compared to high riskvulnerabilities. It can be seen that closed-source vendors arequicker in patching the medium and high risk vulnerabili-ties compared to open-source vendors.

22 29 32 32

15 21

100

18

46 40

56

69

14

32

43

10

21

6

29 36 35

33

29 21 16

28 17

73 23

40 16

8

7

11

14

5

6

27

17

11 4 5

21

11

13

9

12 8

14

11

14

20

14 41 14

14

17

22

9 9

5

23

21

8 4

7

16

29

40 26

24

29

14 22

11

29 33 26

23 28

8

20 12

8

57

32 30 33

24 29

9 9

0

20

40

60

80

100

L M H L M H L M H L M H L M H L M H L M H


> 30 days

+30 days

+7 days

0 day

< 0 days

Percentage

Percentage

of vulnerabilities

for t p

e ranges (CVSS

Scores)

Fig. 19. Patched vulns. relative to exploited vulns.: CVSS score

6.4 Discussion

Latest trends point to an escalating arms race between hackerstrying to discover software vulnerabilities and vendors trying topatch them in a timely manner. On one hand, many softwarevendors have started to implement bug bounty programs [21] toimprove vulnerability discovery and timely patching. We expectthese bug bounty programs to accelerate improvement in patchingbehavior, as shown in Figure 12. On the other hand, researchershave shown that users are slow to install patches released bysoftware vendors for known vulnerabilities [22], [23]. As demon-strated by recent ransomware attacks such as WannaCry, thisleaves large unpatched user populations vulnerable to exploitationby hackers. Since our data set does not provide information aboutpatch installation, we plan to measure patch installation as partof our future software vulnerability lifecycle studies.

7 SOFTWARE DIVERSITY

In fault tolerant systems, several redundant components,that can perform the same task independently, are deployedin parallel. The redundancy ensures failure diversity i.e., ifsome components fail, the system should still be able toperform the required tasks [24]. In case of system security, ifall redundant components are instances of same software, asingle exploit can compromise them all causing completesystem failure. Such situations can be avoided by eitherusing software developed by different vendors or by thesame vendor at different points in time. The differences insoftware code reduce the chances of existence of same vul-nerabilities in all redundant components, and thus providefailure diversity.

We study the number of common vulnerabilities in dif-ferent operating systems as well as in different versions of



10

the operating systems and determine whether the use ofmultiple operating systems can provide failure diversity ornot.

We focus our attention only on the commonly usedoperating systems for two reasons. First, nearly all softwaresystems built today rely on these operating systems. Attimes, supposedly secure systems are compromised not byexploiting vulnerabilities in application software but bycompromising a critical component in the operating system.Given the variety of operating systems available, diversityat the operating system level can be a reasonable wayof providing good security. Second, our data set containsenough information about vulnerabilities only in the oper-ating systems to make statistically significant observations.

The key challenge in determining the number of com-mon vulnerabilities in multiple operating systems is toautomatically identify vulnerabilities that are same. Man-ually identifying same vulnerabilities is not possible dueto the size of the data set. We first describe our automatictechnique to identify same vulnerabilities in the data set.Second, we discuss the quantity and types of common vul-nerabilities in different combinations of operating systems.Third, we study some trends in common vulnerabilities.Last, we present insights regarding code reuse in multipleoperating systems from same vendors.

7.1 Identifying Same Vulnerabilities

Frequently in our aggregate data set, vulnerabilities thatare actually the same are reported under multiple CVE-IDs. For example, consider following two vulnerabilities:CVE-2009-0232 and CVE-2010-1883. The text description ofCVE-2009-0232 is: integer overflow in the Embedded OpenType(EOT) Font Engine in Microsoft Windows 2000 SP4, XP SP2and SP3, Server 2003 SP2, Vista Gold, SP1, and SP2, andServer 2008 Gold and SP2 allows remote attackers to executearbitrary code via a crafted name table, aka “Embedded OpenTypeFont Integer Overflow Vulnerability”. The text description ofCVE-2010-1883 is: integer overflow in the Embedded OpenType(EOT) Font Engine in Microsoft Windows XP SP2 and SP3,Windows Server 2003 SP2, Windows Vista SP1 and SP2, Win-dows Server 2008 Gold, SP2, and R2, and Windows 7 allowsremote attackers to execute arbitrary code via a crafted table in anembedded font, aka “Embedded OpenType Font Integer OverflowVulnerability”. These two text descriptions show that CVE-2009-0232 and CVE-2010-1883 are actually two entries of thesame vulnerability. There are numerous other examples ofmultiple entries of same vulnerability. For example, CVE-2008-2332 and CVE-2010-0043 are two entries for the samevulnerability in Image I/O framework of Apple. Similarly,CVE-2011-0245 and CVE-2011-1374 are two entries for abuffer overflow vulnerability in two different versions ofApple QuickTime.

Occurrence of the same vulnerability in the data setunder multiple CVE-IDs happens due to several reasons, themost prominent of which is the discovery of same vulner-ability in different software products at different times. Forexample, for CVE-2009-0232 and CVE-2010-1883, a probablereason behind NVD adding the second entry in the databaseinstead of updating CVE-2009-0232 could be that the disclo-sure dates of this same vulnerability for Windows 7 andother versions of Windows were different.

To identify and group vulnerabilities that are same, weprocessed the text descriptions of vulnerabilities in ourdata set. Specifically, we performed following four stepsto identify and group vulnerabilities that are same butreported under multiple CVE-IDs. First, we clustered allvulnerabilities into 7 groups, as explained in Section 3.4.Second, from the vulnerabilities in each cluster, we removedthe attributes corresponding to the dominant keywords forthat cluster.

We remove the dominant keywords because they existin the description of majority of vulnerabilities in a givencluster and thus, do not provide useful information todecide whether a pair of vulnerabilities is same or not.For example, in the text descriptions of CVE-2009-0232 andCVE-2010-1883 given above, the words execute, code, andoverflow are the dominant keywords and are, respectively,present in the text descriptions of 99%, 96%, and 58%EXE vulnerabilities; so we remove them. It is the commonnon-dominant keywords in the description of a pair ofvulnerabilities that indicate the extent to which the twovulnerabilities are similar because they are common onlyamong the vulnerabilities that are same. In our example, thewords EOT, embedded, and opentype are the non-dominantkeywords and are, respectively, present in the text descrip-tions of only 0.1%, 0.8% and 0.15% EXE vulnerabilities.

Third, for each cluster, after removing the attributescorresponding to dominant keywords in that cluster, we ag-glomerated the vulnerabilities with Ward’s Linkage [15] andJaccard Coefficient [14] to obtain a dendrogram. The dendro-gram obtained from agglomeration facilitates grouping thevulnerabilities that have no less than a desired percentageof common non-dominant keywords. Fourth, we cut eachdendrogram at a height of J = 0.05 resulting in small sub-clusters, where each non-singleton sub-cluster has at least95% common non-dominant keywords. Therefore, each sub-cluster contains vulnerabilities that are actually the samebut reported under different CVE-IDs. All examples that wehave given of vulnerabilities reported under multiple CVE-IDs were found in the database using these four steps.

7.2 Common Vulnerabilities in Operating Systems

After identifying and grouping the vulnerabilities thatare same, we counted number of common vulnerabilitiesamong different operating systems. Specifically, we studiedthe following four categories of operating systems: Win-dows, Linux, Mac, and Solaris. For Windows, we studiedWindows 98, 98SE, 2000, ME, XP, 2003 Server, Vista, 2008Server, and Windows 7. We don’t consider Windows 2012Server and Windows 8 because the number of reported vul-nerabilities for them are yet too few to make any statisticallysignificant observations. For Linux, we studied Enterprise,Fedora, and Ubuntu. For Mac we studied OS X and OS XServer. We also studied Solaris and OpenSolaris.

Tables 3 and 4 give several stats about common vulnera-bilities among different operating systems. The second col-umn in each table shows several combinations of operatingsystems along with the total number of vulnerabilities dis-closed for each operating system. We have placed differentcombinations in the table such that the operating systemsthat belong to the same category lie next to each other and inchronological order wherever possible. We found 33 pairs of



11

operating systems with at least ten common vulnerabilitiesand 84 pairs with at least one common vulnerability. Due tospace limitations, we do not list all of them in Table 3. Thethird and fourth columns in the tables show the total num-ber of common vulnerabilities and the percentage of com-mon vulnerabilities, respectively. In the next nine columns,each set of three consecutive columns gives the number ofcommon vulnerabilities for the three CVSS metrics: AccessVector, Access Complexity, and Integrity Impact. Recall fromSection 2.1 that each of these three CVSS metrics has threevalues. The last four columns in Table 3 give percentages ofdifferent types of common vulnerabilities. We do not showpercentages for PHP and SQL vulnerabilities because theyare all 0 for all pairs of operating systems.

From these two tables, our general observation is thatthe use of multiple operating systems from the same vendorwill not provide good failure diversity because they containa lot of common vulnerabilities between different releases.We observe this trend primarily in Mac and Windows. Forexample, there are 530 vulnerabilities that are common inMac’s consumer version and server version. Similarly, inalmost all consecutive release pairs of Windows, the per-centage of vulnerabilities that are common is quite large.For different flavors of Linux, however, the percentage ofvulnerabilities that exist in all of them or in each pair is low(less than 20%). A possible reason is that vendor of eachflavor of Linux not only customizes its GUI but also theLinux kernel.

Our analysis reveals that to implement failure diversity,ideally one should deploy operating systems from multiplevendors. This is because there are hardly any common vul-nerabilities in operating systems from different vendors. Forexample, there is only one common vulnerability in Win-dows XP and Solaris. Similarly, there is only one commonvulnerability in Mac OS and Windows Vista. Deployingoperating systems from multiple vendors may, however, notbe feasible in some scenarios such as when the applicationthat has to be run is available only for one category ofoperating systems. In such a scenario, Tables 3 and 4 canbe used to identify an appropriate combination of operatingsystems that will provide best failure diversity for givenrequirements. For example, if the requirement is to deploytwo operating systems that provide robustness against DoSattacks, a good choice would be Windows XP and Windows7 because these two operating systems have least percentageof common DoS vulnerabilities. Similarly, if the requirementis to deploy two Windows based operating systems, that areresilient against network based attacks, on a server that islocated in a physically secure location, a good choice wouldbe Windows 2008 Server and Windows 7 because althoughthey have high percentage of locally exploitable commonvulnerabilities, they have lowest percentage of remotelyexploitable common vulnerabilities.

Another interesting observation from Table 3 is thatthe majority of the common vulnerabilities are either EXEvulnerabilities or DoS vulnerabilities. A possible reason forthe non-existence of remaining four types of vulnerabilitiescan be that they are more related to internet usage onweb browsers or applications such as data bases, and donot depend on the operating system that is hosting thoseapplications.

7.3 Trends in Vulnerabilities Reported Multiple Times

Next, we study how close are the multiple CVSS scores ofthe same vulnerability that is reported multiple times underdifferent CVE-IDs. We also study how far apart in timemultiple entries of the same vulnerability lie.

We observed that for majority of vulnerabilities thatare reported multiple times, the difference between theCVSS scores in their corresponding multiple entries is 0.Figure 20(a) shows the difference between CVSS scoresin multiple entries of the same vulnerabilities. The hori-zontal axis shows the difference in CVSS scores betweenmultiple entries and the vertical axis shows the fractionof vulnerabilities corresponding to the difference values.Recall from Section 2.1 that CVSS score lies in the range[0, 10]. We observe from Figure 20(a) that for over 90%of vulnerabilities that are reported multiple times in thedatabase, the difference between CVSS scores in multipleentries is no greater than 2. This shows that CVSS is a welldesigned system that assigns consistent scores to multipleentries of the same vulnerabilities, even when the entriesare separated by several years.

0 1 2 3 4 5 6 7 8 9 100

0.2

0.4

0.6

0.8

Difference in CVSS scores

Fre

qu

en

cy

(a) CVSS scores

0 1000 2000 3000 4000 500010

0

101

102

Difference in Disclosure Dates (in days)

# o

f V

uln

era

bil

itie

s

(b) Disclosure dates

Fig. 20. Difference in values for same vulnerabilities reported undermultiple CVE-IDs

Figure 20(b) shows the difference between dates in mul-tiple entries of the same corresponding vulnerabilities. Thehorizontal axis shows the difference in disclosure datesbetween multiple entries and the vertical axis shows thenumber of vulnerabilities corresponding to the differencevalues. For approximately 50% of the vulnerabilities, subse-quent entries are reported within a year of the first entry. Forapproximately 90% of the vulnerabilities, subsequent entriesare reported in less than 5 years from the first entry. Most ofthe vulnerabilities that exceed 5 years belong to products forwhich new versions are released frequently such as AppleQuickTime and Adobe Shockwave.

7.4 Code Reuse

The existence of multiple common vulnerabilities in con-secutive releases of the same category of operating systemsshows the reuse of code from the previous release of theoperating system into the next release. For example, in caseof Windows, there are 36 vulnerabilities that are commonin Windows 98, 98SE, 2000, and XP. One such vulnerabilityis CVE-2004-0123, which is a vulnerability in the abstractsyntax notation library, ASN.1, used by these four operatingsystems. Therefore, although Windows XP was completelyredesigned, we see common vulnerabilities between it andWindows 98 due to the reuse of such libraries. Similarly,in Windows XP, 2003 Server, Vista, 2008 Server, and Win-dows 7, there are 127 common vulnerabilities, which showsthat the code from Windows XP was carried forward to



12

TABLE 3Common vulnerabilities among pairs of operating systems

Operating System PairCommon Vulnerabilities

Total %ageAcc. Vector Acc. Complexity Integ. Impact Vulnerability Type (%)

L A N L M H N P C EXE DT DoS XSS

1 Win98 (84), Win98SE (61) 56 62.9 6 0 50 45 7 4 14 28 14 50.0 0 23.2 02 Win98SE (61), Win2000 (507) 46 8.81 5 0 41 36 7 3 7 25 14 54.4 0 15.2 03 Win2000 (507), WinMe (58) 46 8.86 4 0 42 35 9 2 7 25 14 60.9 0 13.0 04 WinMe (58), WinXP (674) 49 7.17 4 0 45 37 10 2 8 27 14 61.2 0 16.3 2.045 WinXP (674), Win2003S (577) 434 53.1 124 1 309 236 174 24 73 72 289 51.2 0 16.1 1.156 Win2003S (577), WinVista (409) 228 30.1 85 1 142 112 112 4 36 11 181 44.3 0 13.6 1.327 WinVista (409), Win2008S (399) 258 46.9 90 1 167 134 121 3 46 10 202 44.2 0 16.3 1.168 Win2008S (399), Win7 (277) 177 35.5 70 1 106 96 79 2 40 9 128 32.8 0 21.5 1.69

9 Win2003S (577), Win2008S (399) 230 30.8 80 1 149 121 105 4 35 15 180 43.0 0 12.2 1.74

10 WinXP (674), WinVista (409) 236 27.9 82 2 152 114 119 3 36 13 187 46.2 0 12.7 1.2711 WinVista (409), Win7 (277) 166 31.9 64 1 101 87 76 3 33 7 126 38.0 0 18.1 1.8112 WinXP (674), Win7 (277) 138 17.0 54 1 83 71 64 3 19 6 113 42.8 0 10.9 2.17

13 Enterprise (198), Fedora (83) 40 16.6 12 1 27 35 2 3 12 13 15 47.5 0 30.0 014 Fedora (83), Ubuntu (69) 26 20.6 5 1 20 23 1 2 3 11 12 53.9 0 23.1 015 Enterprise (198), Ubuntu (69) 25 10.3 9 1 15 19 2 4 4 10 11 36.0 0 24.0 0

16 Mac OS (804), Mac OSX (606) 530 60.2 135 3 392 283 219 28 133 268 129 29.6 1.13 26.0 1.7

17 OpenSolaris (100), Solaris (526) 75 13.6 49 0 26 46 25 4 55 8 12 5.33 0 74.7 0

TABLE 4Common vulnerabilities among sets of three operating systems

Operating SystemsCommon Vulnerabilities

Total %ageAccess Vector Access Complexity Integrity ImpactL A N L M H N P C

1 Win98 (84), Win98SE (61), Win2000 (507) 43 7.60 4 0 39 35 5 3 6 26 112 Win2000 (507), WinMe (58), WinXP (674) 42 3.64 3 0 39 35 5 2 6 26 103 WinXP (674), Win2003S (577), WinVista (409) 221 18.1 79 0 142 103 115 3 28 9 1844 WinVista (409), Win2008S (399), Win7 (277) 161 21.1 66 0 95 84 74 3 28 7 1265 WinXP (674), WinVista (409), Win7 (277) 135 12.4 50 0 85 65 67 3 15 6 114

6 Enterprise (198), Fedora (83), Ubuntu (69) 17 5.38 4 1 12 14 1 2 2 8 7

Windows 2003 Server to Windows Vista to Windows 2008Server and finally to Windows 7. One such vulnerabilityis CVE-2009-0091, which is a vulnerability in Microsoft.NET framework. As all these operating systems use .NETframework, they all carry this common vulnerability. As be-fore, although Windows 7 was redesigned, we see commonvulnerabilities in all releases from Windows XP to Windows7 due to use of such common code. Similar trends exist inthe products of other operating system vendors.

While the code reuse itself is not harmful, the obser-vation that same vulnerabilities exist in multiple releasesshows that vendors do not thoroughly reanalyze the legacycode for potential security threats by using up-to-date secu-rity verification techniques before using it in next releases.

8 IMPLICATIONS

Observations from our study have important implicationsin software design, development, deployment, and manage-ment. We discuss them below.

8.1 Software Design

The analysis of access requirements, risk level, and function-ality of vulnerabilities presented in Sections 3.3, 3.2, and 3.4respectively, can reveal inherent flaws in software designprocess for specific products and vendors. For instance,if a particular software series has more than the typicalnumber of instances of buffer overflow vulnerabilities, thenthis may reflect a lack of sanity checks in socket readprocesses. From our data set, we observed that DoS is the

most exploited vulnerability type in Solaris accounting for38.85% of all its vulnerabilities. At the same time, only11.7% of vulnerabilities in OS X involve DoS, which showsthat Solaris is more susceptible to DoS attacks compared toOS X. The observation mentioned above implies that Solarisdevelopers need to take additional steps to make the designmore robust to DoS attacks.

8.2 Code Development Practices

The analysis of vulnerability life cycles during the evolutionof a given software can reveal insights about potentialflaws in its code development and testing practices. Inparticular, a correlation analysis of count of vulnerabilitiesacross different software and vendors can highlight impor-tant differences in code development practices. For instance,we observe in Figure 13 that the percentage sizes of thesubgroups corresponding to tpd > 0 for open-source ven-dors (Linux, Redhat) are significantly greater than those ofclosed-source vendors (Microsoft, Apple). This observationhighlights an important insight into the code developmentpractices of open-source vendors which typically rely oncontributions from a group of volunteer developers. On theother hand, closed-source vendors have dedicated resourcesto fix newly disclosed vulnerabilities as soon as possible.Therefore, open-source vendors tend to have a slower patchresponse compared to closed-source vendors. Our workcomplements the work done by Shin et al. in using differentmetrics to predict software vulnerabilities [25].



13

8.3 Customers’ Assessment of Software

The analysis presented in this paper also has direct implica-tions in product assessment, certification, and security rec-ommendations to consumers. Several commercial productse.g. eEye Digital Security (http://www.eeye.com) and Arel-lia (http://www.arellia.com/), can leverage the presentedanalysis for product recommendation and design of futuresecurity policies. For example, given that the exploits ofvulnerabilities have already been released, our measure-ment analysis showed that Sun releases patches for 96%of the vulnerabilities within a month; whereas, Microsoft,Apple, and Linux provide patches for only 69%, 74%, and65% of vulnerabilities in the same time period. Therefore,if the patch response of vendor is of prime importance to acustomer, then the products from Sun should be preferred.As another example, if a customer’s infrastructure has lesstolerance for DoS attacks, then it is more suitable to deployMac OS X, which has the lowest percentage of DoS vul-nerabilities compared to other operating systems. Likewise,if a customer requires more robustness to buffer overflowattacks, then it is more suitable to deploy Solaris becausebuffer overflow vulnerabilities account for about 20% of allthe vulnerabilities in Windows and Mac but only 13% inSolaris.

9 RELATED WORK

The major focus of the work on large scale analysis ofvulnerabilities has been on the development of vulnerabilitydiscovery models (VDMs). Some work has also been doneto understand the economic impacts of vulnerability disclo-sures in software. We briefly describe the work that has beendone in these areas in relation to our work.

9.1 Large Scale Vulnerability Analysis

The work most relevant to ours is that of Frei et al. [3], whopresented a large scale study of the discovery, disclosure,exploit, and patch dates of vulnerabilities. They analyzedabout 14000 vulnerabilities and showed that till 2006, thehackers were quicker than vendors. This observation isin accordance with what we presented in this paper butwe also show that in the last three years, the response ofvendors has been improving. Frei et al.’s work does notdifferentiate between vendors and types of vulnerabilities.

In [26], authors study the life-cycle of vulnerabilitiesfrom the time a software is released till the time the firstvulnerability is discovered. They show that the time till thediscovery of the first vulnerability is a function of the hack-ers’ familiarity with the system and the amount of legacycode. In [27], the authors propose to use semantic templatesto help the developers understand the vulnerabilities andtheir artifacts. This work only focuses on understanding thetechnical details of a disclosed vulnerability and does notstudy any large scale trends in vulnerabilities.

9.2 Studies on Disclosure and Patching

In [28], authors have studied the economic aspects of thequickness of vendors in releasing patches for Internet basedvulnerabilities. In [29], authors show that on average avendor loses 0.6% of the stock price with the disclosure of avulnerability. In [8], authors show that a vendor with more

competitors patches the vulnerabilities more quickly. In [7],they show that the vulnerability disclosure accelerates thepatch release. Although their work is based upon a smalldata set of just 354 vulnerabilities disclosed till 2003, theyobserve, like us, that the closed-source vendors are quickerin patching the disclosed vulnerabilities. These studies,however, do not develop any insight into the individualbehaviors of vendors and hackers.

In [30], using a small data set, authors make a claim thatthere is no difference between the patching behavior of openand closed-source vendors.

They make this conflicting observation because theyonly consider the percentage of patched vulnerabilities asa measure of goodness of a vendor. This is unreasonablebecause without analyzing the duration between disclosuredates and patch dates, one can not determine how active avendor is in fixing vulnerabilities in its products.

9.3 Modeling and Classification

The motivation behind the work on VDMs is to enable theprediction of quantity and timing of vulnerability discov-eries in new software. Four notable VDMs have been pro-posed: (1) Anderson Thermodynamic Model [2], (2) RescorlaLinear Model [6], (3) Rescorla Exponential Model [6], and (4)Alhazmi-Malaiya Logistic Model [5]. Another work focusedon modeling the time interval between disclosure date ofvulnerabilities and their corresponding exploit, patch, anddiscovery dates [31]. A recent work by Bozorgi et al. ex-tracted various features from NVD and OSVDB and usedmachine learning techniques to predict whether a recentlydisclosed vulnerability will be exploited within a given timeor not [32]. Our focus, however, is not the prediction ratherthe study of phases of vulnerability life cycle in referenceto different variables along with several aspects associatedwith the nature of vulnerabilities.

9.4 Software Diversity

In [9], Garcia et al. studied whether operating system di-versity is feasible or not. They also observed that for somecombinations of operating systems, the number of commonvulnerabilities were few. Although comprehensive, theirwork lacks two aspects that we have studied. First, theydid not consider the types of vulnerabilities (such as EXE,DoS etc.) that are common in multiple operating systemsand can thus not be used to determine the best pair ofoperating systems that provides robustness against a certaintype of vulnerability. Similarly, they did not study the accesscomplexity and integrity impact of common vulnerabilities.Second, they performed their study on only 1887 vulnera-bilities and did not do text processing to identify commonvulnerabilities. In contrast, we used all vulnerabilities re-ported until April 2013 for the four categories of operatingsystems.

10 CONCLUSION

In this paper, we presented a large scale study of variousaspects associated with software vulnerabilities during theirlife cycle. We aggregated a large software vulnerabilitydata set containing 56077 vulnerabilities disclosed till 2013.Our study showed that the number of vulnerabilities be-ing disclosed every year stopped increasing between 2008



14

and 2012. We showed that the most primitive and mostexploited form of vulnerabilities are DoS and EXE; however,the numbers of SQL, XSS, and PHP vulnerabilities haveincreased significantly. We also observed that the percent-age of remotely exploitable vulnerabilities has graduallyincreased to over 80% of all the vulnerabilities. Since 2008,vendors have been becoming increasingly agile in patchingthe vulnerabilities and the access complexity of vulnera-bilities has been increasing. Our findings also highlightedthat patching of vulnerabilities in closed-source software isfaster compared to open-source software and at the sametime the exploitation is slower. Our study also shows thatthe use of different versions of operating system from thesame vendor does not provide good failure diversity. Thebest failure diversity can be achieved by using operatingsystems from different vendors.

REFERENCES

[1] E. E. Schultz, D. S. Brown, and T. A. Longstaff, Responding to Com-puter Security Incidents: Guidelines for Incident Handling, LawrenceLivermore National Laboratory, Livermore, CA, 1990.

[2] R. Anderson, “Security in open versus closed systems – the danceof boltzmann, coase and moore,” in Proc.Open Source Software:Economics, Law, and Policy Confocuce, June 2002.

[3] S. Frei, M. May, U. Fiedler, and B. Plattner, “Large-scale vulner-ability analysis,” in Proc.2006 SIGCOMM workshop on Large-ScaleAttack Defense, September 2006, pp. 131–138.

[4] S. Frei, B. Tellenbach, and B. Plattner, “0-day patch exposingvendors (in) security performance,” in Proc.Black Hat TechnicalSecurity Conf., vol. 14, 2009.

[5] O. H. Alhazmi and Y. K. Malaiya, “Quantitative vulnerabilityassessment of systems software,” in Proc.Annual Reliability andMaintainability Symposium, 2005, pp. 615–620.

[6] E. Rescorla, “Is finding security holes a good idea?” IEEE Securityand Privacy, vol. 3, no. 1, pp. 14–19, January 2005.

[7] A. Arora, R. Krishnan, R. Telang, and Y. Yang, “An empiricalanalysis of software vendors patch release behavior: Impact ofvulnerability disclosure,” Information Systems Research, 2010.

[8] A. Arora, C. Forman, A. Nandkumar, and R. Telang, “Competitionand patching of security vulnerabilities: An empirical analysis,”Information Economics and Policy, vol. 22, no. 2, pp. 164–177, 2010.

[9] M. Garcia, A. Bessani, I. Gashi, N. Neves, and R. Obelheiro,“OS diversity for intrusion tolerance: Myth or reality?” in InProc.IEEE/IFIP DSN, 2011.

[10] (http://nvd.nist.gov/) National Vulnerability Database.[11] (http://osvdb.org/) The Open Source Vulnerability Database.[12] (http://www.first.org/cvss) Forum for Incident Response and

Security Teams.[13] (http://cve.mitre.org/) Common Vulnerabilities and Exposures.[14] P. Jaccard, “The distribution of the flora in the alpine zone. 1,” New

phytologist, vol. 11, no. 2, pp. 37–50, 1912.[15] J. H. Ward Jr, “Hierarchical grouping to optimize an objective

function,” Journal of the American statistical association, 1963.[16] A. K. Jain and R. C. Dubes, Algorithms for clustering data. Prentice-

Hall, Inc., 1988.[17] J. J. OConnor, E. F. Robertson, and F. Edmund, “Students t-test,”

MacTutor History of Mathematics archive, 2009.[18] R. Agrawal and R. Srikant, “Fast algorithms for mining association

rules,” in Proc.of 20th Int.Conf.on Very Large Data Bases, 1994.[19] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and

I. H. Witten, “The weka data mining software: an update,” ACMSIGKDD explorations newsletter, vol. 11, no. 1, pp. 10–18, 2009.

[20] S. Christey and C. Wysopal, “Responsible vulnerability disclosureprocess,” http://www. wiretrip. net/rfp/txt/ietf-draft. txt, 2002.

[21] “Bug Bounty Programs,” https://hackerone.com/bug-bounty-programs.

[22] B. Wang, X. Li, L. P. de Aguiar, D. S. Menasche, and Z. Shafiq,“Characterizing and modeling patching practices of industrialcontrol systems,” Proceedings of the ACM on Measurement andAnalysis of Computing Systems, vol. 1, no. 1, p. 18, 2017.

[23] S. Frei, T. Duebendorfer, and B. Plattner, “Firefox (in) securityupdate dynamics exposed,” ACM SIGCOMM Computer Commu-nication Review, vol. 39, no. 1, pp. 16–22, 2008.

[24] P. E. Verı́ssimo, N. F. Neves, and M. P. Correia, “Intrusion-tolerantarchitectures: Concepts and design,” in Architecting DependableSystems. Springer, 2003, pp. 3–36.

[25] Y. Shin, A. Meneely, L. Williams, and J. A. Osborne, “Evaluat-ing complexity, code churn, and developer activity metrics asindicators of software vulnerabilities,” Software Engineering, IEEETransactions on, vol. 37, no. 6, pp. 772–787, 2011.

[26] S. Clark, S. Frei, M. Blaze, and J. Smith, “Familiarity breedscontempt: The honeymoon effect and the role of legacy code inzero-day vulnerabilities,” in Proc.26th Int.Annual Computer SecurityApplications Conf., 2010, pp. 251–260.

[27] Y. Wu, H. Siy, and R. Gandhi, “Empirical results on the studyof software vulnerabilities: NIER track,” in Proc.33rd Int.Conf.onSoftware Engineering, 2011, pp. 964–967.

[28] R. Anderson, “Why information security is hard – an economicperspective,” in Proc.17th Annual Computer Security ApplicationsConf., 2001, pp. 358–365.

[29] R. Telang and S. Wattal, “An empirical analysis of the impact ofsoftware vulnerability announcements on firm stock price,” IEEETransactions on Software Engineering, vol. 33, no. 8, pp. 544–557,2007.

[30] G. Schryen, “A comprehensive and comparative analysis of thepatching behavior of open source and closed source softwarevendors,” in Proc.5th Int.Conf.on IT Security Incident Managementand IT Forensics, 2009, pp. 153–168.

[31] G. Vache, “Vulnerability analysis for a quantitative security eval-uation,” in Proc.3rd Int.Symp. on Empirical Software Engineering andMeasurement, 2009.

[32] M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker, “Beyondheuristics: Learning to classify vulnerabilities and predict ex-ploits,” in Proc.of 16th Int.Conf.on Knowledge discovery and datamining, 2010, pp. 105–114.

Muhammad Shahzad received the Ph.D. de-gree in computer science from Michigan StateUniversity in 2015. He is currently an Assis-tant Professor at the Department of ComputerScience, North Carolina State University, USA.His research interests include design, analysis,measurement, and modeling of networking andsecurity systems. He received the 2015 Out-standing Graduate Student Award, 2015 FitchBeach Award, and 2012 Outstanding StudentLeader Award at Michigan State University.

M. Zubair Shafiq received the Ph.D. degree incomputer science from Michigan State Univer-sity in 2014. He is currently an Assistant Pro-fessor at the Department of Computer Science,University of Iowa, USA. He was co-recipient ofthe IEEE ICNP 2012 Best Paper Award. He alsoreceived the 2013 Fitch-Beach Award by Col-lege of Engineering, Michigan State University.His research interests include measurement andmodeling, performance evaluation, networking,and security.

Alex X. Liu received the Ph.D. degree in com-puter science from UT Austin in 2006. He is cur-rently a Professor with the Department of Com-puter Science and Engineering, Michigan StateUniversity, USA. His research interests focus onnetworking, security, and dependable systems.He received the IEEE and IFIP William C. CarterAward in 2004 and an NSF CAREER Award in2009. He received the MSU College of Engi-neering Withrow Distinguished Scholar Award in2011.

large scale characterization of software vulnerability ... · muhammad shahzad, m. zubair shaﬁq,...

Documents