“evaluating cloud services using super- … jaiswal.pdfa summer internship project report on...

15
A Summer Internship Project Report On “Evaluating Cloud Services Using super- efficiency DEA and TOPSIS model” Carriedout at the Institute of Development and Research in Banking Technology, Hyderabad Established by ‘Reserve Bank of India’ Submitted by Akshay Jaiswal Roll No. 12400EN011 Integrated Dual Degree (Computer Science and Engineering), 2012-2017 Indian Institute of Technology (BHU) Varanasi Under the Guidance of Dr. G. R. Gangadharan Assistant Professor IDRBT, Hyderabad July, 2015

Upload: phunghanh

Post on 26-May-2019

218 views

Category:

Documents


0 download

TRANSCRIPT

A Summer Internship Project Report

On

“Evaluating Cloud Services Using super-

efficiency DEA and TOPSIS model” Carriedout at the

Institute of Development and Research in Banking Technology,

Hyderabad

Established by ‘Reserve Bank of India’

Submitted by

Akshay Jaiswal

Roll No. – 12400EN011

Integrated Dual Degree (Computer Science and Engineering), 2012-2017

Indian Institute of Technology (BHU) Varanasi

Under the Guidance of

Dr. G. R. Gangadharan

Assistant Professor

IDRBT, Hyderabad

July, 2015

CERTIFICATE

This is to certify that the summer internship project report entitled “Evaluating

Cloud Services Using super-efficiency DEA and TOPSIS model”submitted to

Institute for Development & Research in Banking Technology [IDRBT],

Hyderabad is a bonafide record of work done by “Akshay Jaiswal”, Roll no. –

12400EN011, IDD (CSE), 2012-17, Indian Institute of Technology (BHU)

Varanasi” under my supervision from “13th May, 2015” to “13th July, 2015”.

Dr. G. R. Gangadharan

Assistant Professor,

IDRBT, Hyderabad

Place: Hyderabad

Date: 13th July, 2015

ACKNOWLEDGEMENT

I would like to express my profound gratitude and deep regards to my guide Dr. G. R.

Gangadharan, Assistant Professor, IDRBT, Hyderabad for his guidance, and constant

encouragement throughout the course of this summer internship project.

I also take this opportunity to express a deep sense of gratitude to Dr A. S. Ramasastri,

Director, IDRBT, Hyderabad for his cordial support by providing excellent facilities like labs

and library.I am also grateful to IDRBT staff for their cooperation during the period of my

assignment.

Lastly, I thank my parents, and friends for their constant encouragement and moral support

without which this assignment would not have been possible.

Akshay Jaiswal

Integrated Dual Degree (2012-2017)

Computer Science and Engineering

IIT (BHU) Varanasi

Evaluating the Efficiency of Cloud Services Using

Super-efficiency DEA and TOPSIS

Abstract – With the growing demand and commercial availability of cloud services, the need

for comparison of their functionality available to customers at different prices and

performance has arisen. It is needed to be said that relevant and fair comparison is still

challenging due to diverse deployment options and dissimilar features of different services.

The aim of this paper is to rank cloud services using super-efficiency DEA and TOPSIS.

Keywords – Cloud services, super-efficiency DEA, AHP, TOPSIS

1 INTRODUCTION

Cloud computing has gained tremendous momentum in past few years as the use of

computers in our day-to-day life has increased effectively. Cloud computing offers

undeniable advantages in terms of cost and reliability compared to the traditional

computing model that uses a dedicated in-house infrastructure. Cloud customers don’t

have to pay large sums of money to register for using cloud services, they only need to pay

for what they actually use.

There is a high growth in number of companies that provide public cloud computing

services, such as Amazon, Google, Microsoft, Rackspace, and GoGrid. They offer various

options in pricing, performance and feature set. There are broadly three delivery models

that are provided:

Software-as-a-Service (SaaS), is a software distribution model in which applications

are hosted by a vendor or service provider and made available to customers over a

network, typically the Internet.

Platform-as-a-Service (PaaS), is a paradigm for delivering operating systems and

associated services over the Internet without downloads or installation.

Infrastructure-as-a-Service (IaaS), involves outsourcing the equipment used to

support operations, including storage, hardware, servers and networking

components.

The presence of these many cloud service providers awakens a question: “How good a cloud service provider performs compared to the others?” Answering this will benefit both the customers and the providers. For potential customers, the answer can help them choose a provider that best fits their performance and cost needs. For instance, they may choose one provider for storage intensive applicationsand another for computation intensive applications. For cloud providers, such answers can point them in the right directionfor improvements.

This paper discusses TOPSIS and fuzzy TOPSIS for comparative study of cloud service

providers like Amazon, HP, Azure, Rackspace, Google Compute Engine, Century Link and

City-Cloud. For each service provider, service levels which differ in Virtual Cores and

Memory are considered for the performance evaluation. The parameters considered for

evaluating the QoS is user specific with a relative preference value. The following

calculated Benchmark values are used for performance evaluation, though it can be any

user specific.

CPU Performance

Disk I/O Consistency

Disk Performance

Memory Performance

Price of a service is dependent on number of Virtual Cores and Memory, so only Price is

considered during the evaluation process. The two service level for each service

provider is selected based on number of virtual cores in which we considered virtual

core-2, virtual core- 4 and virtual core-8 for each service provider which among

themselves differs on price and memory.

2. Literature Review

In this section, we briefly describe some of the existing models to evaluate the relative

performance and ranking of cloud services.

With the increasing popularity of cloud computing,a lot of research has been done to

compare the cloud services for different type of applications such as scientific computing,

web services based on attributes including security, accountability, assurance,

performance,cost etc. However, there are research papers which have done comparison based

on properties of the alternative without the comparison of performance[Buyyaet al., 2008].

Kabir et al. (2012) hasevaluated the major factors for travel agency websites quality from the

viewpoint of users' perception and developed a systematic multiple-attribute evaluation

model using Technique for Order Preference by Similarity to Ideal Solution(TOPSIS) and

Fuzzy TOPSIS, to find out the effective travel agency websites. A comparative analysis of

TOPSIS and Fuzzy TOPSIS methods are illustrated in this paper through a practical

application from the websites of five travel agencies. We have done a similar study for cloud

services.

Aryanezhad et al. (2011) takes the values of alternatives with respect to the criteria or/and the

values of criteria weights as fuzzy numbers. In this paper, fuzzy TOPSIS (for Order

Preference by Similarity to Ideal Solution) method based on left and right scores for fuzzy

MADM problems is proposed, and its applicability is shown using two numerical examples.

3. Modelling Super-efficiency Data Envelopment Analysis for Cloud

Services

Since the early 1980s, Data Envelopment Analysis (DEA) has been used as an alternative

method of classification for evaluating the relative efficiency of independent homogenous

units which use the same inputs to produce the same outputs (Cooper, Seiford and Tone,

2000). However, a serious inconvenience in using DEA as a method of classification is the

room of having units tied with relative efficiency equal to 100 percent. That is, units at the

efficient frontier.

The objective function of the input oriented model used for super-efficiency DEA is shown

below:

Subject to

where DMUE is an efficient DMU and other symbols have their usual meanings.

This will give us a super-efficiency score of greater than one,enabling us to distinguish

between the efficient observations. In particular, the super-efficiency measure examines the

maximal radial change in inputs and/or outputs for an observation to remain efficient, that is,

how much can the inputs be increased, or the outputs decreased, while not becoming

inefficient. The larger the value of the super-efficiency measure, the higher an observation is

ranked among the efficient units. Super-efficiency measures can be calculated for both

inefficient and efficient observations, but in the case of inefficient observations the values of

the efficiency measure do not change, while efficient observations may obtain higher values.

The super-efficiency DEA model being used goes through a modification to find the

preferred efficiency using AHP. The modified objective function is shown below:

Subject to

4. Modelling TOPSIS for Cloud Services

Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) method was

introduced by Hwang and Yoon (1981). It is a Multi Attribute Decision Making (MADM)

technique that sorts the alternatives for a decision problem in order of distance from the

positive and negative ideal solutions. The best solution is the one which has minimum

distance from the positive ideal solution and maximum distance from the negative ideal

solution [Benitez et al., 2007]. The positive ideal solution is the one which maximizes the

benefits and minimizes the costs while the negative ideal solution is the one which minimizes

the benefits and maximizes the costs [Lin et al., 2008].

The procedure for TOPSIS model is described below:

Step 1: Construct the Decision Matrix for the alternatives based on the criteria with

each row representing an alternative and each column representing a criterion. So

element xijrefers to ith

alternative and jth

criterion.

The matrix D is defined as:

C1 C2 C3 . . . Cn

A1x11 x12 x13 . . . x1n

A2x21 x22 x23 . . . x2n

A3x31 x32 x33 . . . x3n

D = .. . . . . . .

. . . . . . . .

. . . . . . . .

Am xm1 xm2 xm3 . . . xmn

Where Ai (A1, A2, A3, ……., Am)represents m alternatives and Cj (C1, C2, C3, …….,

Cn) represents ncriteria.

Step 2: Find the normalized decision matrix D using the following formula:

Step 3: Compute the weighted normalized decision matrix by multiplying weights of

criteria wj for j = 1, ……, n with the associated columns. The weighted normalized

value vij is calculated as:

The weights are calculated by using Analytic hierarchy Process (AHP) which is

explained in section 5 of this paper.

Step 4: Determine the positive ideal solution ( ) i.e. best performance from each

criteria and negative ideal solution ( i.e. least performance from each criteria.

Where J is the set of benefit attributes (larger-the- better type) and J' is the set of cost

attributes (smaller-the-better type).

Step 5: Compute separation of each criterion value for each alternative from both

positive ideal and negative ideal solutions.

The formula for calculating the separation from the positive ideal solution is:

And the formula for calculating the separation from the negative ideal solution is:

Where i = 1, 2, 3, ……., m

Step 7: Determine for each alternative, the ratio which is the relative closeness to the

ideal solution or the similarities to the ideal solution.

Larger value of indicates better performance of the alternatives.

Step 8: Rank the alternatives based on the value of .

5. Analytic Hierarchy Process (AHP)

Analytical Hierarchy process (AHP) was used to modify the objective function of super-

efficiency DEA and to find the weight for the criteria which was further used to find the

weighted decision matrices in case of TOPSIS.The scale we use ranges from 1 to 9 as

presented in Table 1.

Intensity of Importance Definition

1 Equal Importance

3 Moderate Importance of one over another

5 Essential or Strong Importance

7 Very Strong Importance

9 Extreme Importance

2,4,6,8 Intermediate value between the two adjacent judgments

Table 1. Scales for comparison

Table 2 shows the relative importance among QoS attributes. For example, the relative importance of

CPU performance is 7 times as that of Disk I/O consistency and the relative importance of disk

performance is 0.2 (1/5) times as that of memory performance, and its reciprocal is in lower triangular

matrix i.e. memory performance is 5 times as important parameter as disk performance.

CPU

performance

Disk I/O

Consistency

Disk Performance Memory

Performance

CPU

performance

1 7 5 2

Disk I/O

Consistency

0.14 1 0.33 0.2

Disk

Performance

0.2 3 1 0.2

Memory

Performance

0.5 5 5 1

Table 2. Relative importance among QoS attributes

After normalizing the resultant matrix and averaging the value, we get the following weights

(as in Table 3):

QoS attributes Weights

CPU Performance 50.24%

Disc I/O Consistency 5.70%

Disc Performance 11.08%

Memory Performance 32.98%

Table 3. Weights for QoS attributes

To check the consistency of the calculated weights, we obtain consistency ratio (CR) as

0.052108. Consistency ratio tells how inconsistent the matrix is, and the result is acceptable if

CR <= 0.1. So our matrix is consistent and weights are valid.

These weights are then used to evaluatepreferred values of theperformance attributes which is

done by multiplying these fractions to the sum of normalised output parameters.

Normalisation is done to remove the units of different attributes so that all the outputs can be

added to evaluate the desired precise value. This new data set emerging from AHP can be

termed as preferred data set. Super-efficiency DEA is then applied on this preferred data set

to find the preferred efficiency.

The new efficiency score can be used directly to rank the cloud services considering priority

on performance benchmarks.

3. Data Collection Methodology and Data Set Description

We have considered 7 cloud service providers which include Amazon, HP, Azure,

Rackspace, Google, Century Link, and City-Cloud (without any order). For each service

provider, the number of virtual cores are considered for the cloud services offered by them.

The services having 2-virual cores are specified as large cloud services, those having 4-

virtual cores are specified as extra-large cloud services and the ones having 8-virtual cores

are specified as 2x-extra-large cloud service. The Dataset for the analysis is illustrated in

Table 5. The cloud service providers are coded as C1, C2....., and C7 (without any order) and

the services provided by each are further coded as S1, S2, S3, and so on. The specified data

(mentioned in column 3, 4, and 5) for Price/Hour (cents), Virtual core, Memory (GB) for

each service are taken from their respective websites. We considered the benchmark values

(mentioned in column 5 onwards) for CPU performance, Disk I/O consistency, Disk

performance, and Memory performance for evaluating the performance of the cloud services,

obtained from cloudharmony.com (a cloud benchmarking service).

During the data collection, consistency check is performed on data to identify if they are out

of range, logically inconsistent, or have extreme values. Inconsistent data for any service

provider are inadmissible and we either corrected it if possible or we did not consider the

service provider for the analysis. There are few missing values for benchmarks of the cloud

services. There is no trend in the data set for cloud services within a service provider and

among the service providers so as to substitute a neutral or mean value. Hence, we deleted

those cases with incomplete benchmark values.

One may argue that the different quantitative QoS attributes of cloud service providers

considered in this study are rather limited. However, it should be noticed that collecting the

real world data set regarding quantitative QoS attributes of cloud service providers were

extremely challenging.

Providers Service

price/Hr

(cents)

Virtual

core Memory

CPU

Performance Disk IO Consistency

Disk

Performance

Memory

Performance

C1 C1S2 28 4 15 25.86 92.89 110.33 129.03

C1S3 56 8 30 48.23 53.28 67.22 131.79

C2S1 14 2 7.5 13.89 114.44 97.38 144.86

C2 C2S2 28 4 15 23.66 119.63 100.55 131.81

C2S3 56 8 30 51.7 77.46 73.44 125.59

C3 C3S2 16 4 4 7.21 70.29 125.48 54.28

C3S3 32 8 8 15.33 57.11 111.18 55.68

C4S1 18 2 3.5 8.83 67.87 83.73 52.27

C4 C4S2 36 4 7 16.07 67.97 78.49 61.8

C4S3 72 8 14 28.4 78.72 70.91 27.33

C5S1 12 2 4 16.41 23.43 40.23 80.67

C5 C5S2 45 4 15 32.4 29.07 42.47 90.83

C5S3 90 8 30 52.82 35.35 55.07 83.92

C6S1 8 2 4 17.34 43.02 141.23 51.71

C6 C6S2 16 4 8 37.05 36.15 102.74 132.87

C6S3 32 8 16 71.11 39.66 99.15 135.88

C7S1 10.132 2 4 23.43 89.31 173.49 89.84

C7 C7S2 20.8624 4 8 42.05 59.63 174.5 97.16

C7S3 34.6528 8 16 75.89 64.64 174.12 100.14

Table 4. The collected dataset

7. Evaluation of Cloud Services using super-efficiency DEA

Our experimental evaluation is based on the input-oriented super-efficiency DEA modelthat

addresses the problem “By how much can input parameter (Price/Hour) be proportionally be

decreased without changing the output parameter (Performance Benchmark values)”. The

input parameter considered for the analysis is Price/Hour charged by the cloud service

provider for a cloud service. We employed four output parameters; however any number of

parameters could be included for both outputs and inputs. The output parameters include

CPU performance, Disk I/O consistency, Disk performance, and Memory performance.

Figure 1. Relative Efficiency Score of Cloud Services using super-efficiency DEA model

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

140.00%

160.00%

C

4S3

C

3S2

C

4S2

C

5S3

C

5S2

C

1S2

C

2S3

C

4S1

C

1S1

C

3S1

C

2S2

C

5S1

C

7S2

C

7S3

C

6S3

C

6S2

C

6S1

C

2S1

C

7S1

Super-efficiency DEA

Efficiency Efficiency (Preferred)

Table 5. Calculated Slack Value and Total slack for each cloud service.

The results of super-efficiency DEA model is shown in figure 1. The result in blue

colourindicates the efficiency score of DMUs with equal preference on QoS attribute i.e.

while calculating the efficiency score all performance benchmarks are given equal

weightage.Super-efficiency DEA is used to rank DMUs which are on the efficient frontier

line [Adler and Friedman, 2002]. The calculated slack value for each cloud services is

indicated in Table 5.

Apart from super-efficiency DEA, a modified version was also used to evaluate the preferred

efficiency based on priorities of the user. The result is shown in figure 1in red color.

C7S1 is performing relatively better among other cloud services with virtual core 2. The

service C7S1 provides better performance on the preference QoS attribute at a reasonable

service charge, however it has virtual core-2. So if a user wishes for a higher service with

higher virtual core (for example, a service with virtual core-4), then C6S2 is efficient on

relative scale. Similarly, among services with virtual core-8, C6S3 is better than other

services. Overall it can be seen that service provider C6 are relatively performing best among

considered cloud service providers.

8. Evaluation of Cloud Services using TOPSIS

For the analysis of cloud services, we can use benchmark values and also Price per

hour for the usage of cloud services as criteria. So in this case,maximum value of

performance-oriented benchmark for positive ideal solution and minimum value of

price data and time-oriented benchmarks are considered under each criteria. Whereas

in case of negative ideal solution, minimum value of performance-oriented

benchmark and maximum value of price data and time-oriented benchmark are

considered.

We use price per hour, CPU performance, disk I/O consistency, disk performance and

memory performance as criteria and cloud services C1S2, C1S3,……, C7S3 as the

alternatives. For giving the preference to the criteria, we use AHP.

Positive ideal solution is obtained by considering maximum value under each benchmark

attribute and minimum value under price per hour attribute. Similarly, negative ideal solution

is obtained by considering minimum values among benchmark attribute and maximum value

under price per hour attribute among the cloud services.

According to TOPSIS, the preference order of the cloud service providers is given below in

Figure 2.

Figure 2: Cloud services ranked in descending order according to TOPSIS

According to TOPSIS model also, service provider C6 and C7 are performing relatively better.

C6S2 service is relatively has best performance with Relative Ratio of 0.775, which indicates

that the service is 77.5% of the ideal solution. Among the virtual core-8 service also service

provider C6 with service C6S3 is performing the best with the ratio of 73.7%. For the service

among virtual core-2, service provider C7 with service C7S1 is relatively performing better.

Interestingly, it can be seen that Service Provider C4 and C5 are performing well in the

service with virtual-core-2 but the ratio decrease for their service with virtual-core-4 and 8.

This may be because the ratio of providing high quality of service to price charged is very

low which means that they are charging heavily for providing more quality service.

9. Conclusion and Future Work

With the growing demand for the cloud service there are many cloud service

providers are available with many cloud services with different prices and

performance. It has now become a challenge to cloud customers to select the best

cloud service which will satisfy their required QoS attribute. To select the best

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

TOPSIS

service customers need to have a framework to identify and measure key

performance criteria according to their requirement in the applications.

The Study discussed how to select the best cloud service model among various

service providers based on user specific QoS attributes. Although many

frameworks exist to evaluate the performance of cloud service, the study

introduced super-efficiency DEA model for the evaluation of cloud services

which has added advantage that it give projected values for increasing the

performance to a competing level. Other models that were used to evaluate the

performance of cloud services were AHP and TOPSIS. AHP was used to

evaluate the relative preference of QoS attribute. We used this preferences in

analysis with the other models.

In future we will incorporate other performance benchmarks other than

traditional High Performance Computing Benchmarks as these benchmarks focus

primarily on static system specific to performance and cost. A Paper [Binning]

propose metrics for measuring peak load handling, fault tolerance of cloud

computing, scalability and cost.

References

Aryanezhad, B., Tarokh, M. J., Mokhtarian, M. N., &Zaheri, F. (2011). A fuzzy TOPSIS

method based on left and right scores. International Journal of Industrial Engineering &

Production Research, 22(1).

Binning, C., Kossamann. D., Kraska. T., Loesing. S. (2009). How is the weather tomorrow?:

Towards a benchmark for the cloud. In: Proceedings of the Second International Workshop

on Testing Database Systems. ACM.

Cloudharmony.com. May 2014. https://cloudharmony.com/ Fathi, M. R., Matin, H. Z., Zarchi, M. K., &Azizollahi, S. (2011). The application of fuzzy TOPSIS approach to personnel selection for Padir Company, Iran. Journal of management Research, 3(2). Saaty, T., (2000). Fundamental of decision making and priority theory with analytic hierarchy process. RWS publications, USA. Yawe, B. (2010). Hospital Performance Evaluation in Uganda: A Super-Efficiency Data Envelope Analysis Model. Zambia Social Science Journal,1(1), 6. Zheng, Z., Wu, X., Zhang, Y., Lyu, M.R., Wang, J. (2013). QoS Ranking Prediction for Cloud Services. IEEE Transactions on Parallel and Distributed Systems. 24 (6). pp. 1213-1222.

Project Summary

Name of the Student: Akshay Jaiswal

Course and Year of Study: Integrated Dual Degree, 3rd

year

Name of the Institution: IIT (BHU) Varanasi

Name of the Project: Evaluating Cloud Services Using super-efficiency DEA and

TOPSIS model

Name of the Guide: Dr G R Gangadharan

Project Description: – With the growing demand and commercial availability of cloud

services, the need for comparison of their functionality available to customers at different

prices and performance has arisen. It is needed to be said that relevant and fair comparison is

still challenging due to diverse deployment options and dissimilar features of different

services. The aim of this paper is to rank cloud services using super-efficiency DEA and

TOPSIS.

Objectives: Ranking of cloud services using models such as super efficiency DEA and

TOPSIS to help the customers to choose a better service and to encourage cloud service

providers to improve their quality of services.

Deliverables: A highly organised ranking of the cloud services