Academic Cloud Computing Research: Five Pitfalls and Five Opportunities

Adam Barker, Blesson Varghese, Jonathan Stuart Ward and Ian SommervilleSchool of Computer Science, University of St Andrews, UK

Abstract

This discussion paper argues that there are five funda-mental pitfalls, which can restrict academics from con-ducting cloud computing research at the infrastructurelevel, which is currently where the vast majority of aca-demic research lies. Instead academics should be con-ducting higher risk research, in order to gain understand-ing and open up entirely new areas.

We call for a renewed mindset and argue that academicresearch should focus less upon physical infrastructureand embrace the abstractions provided by clouds throughfive opportunities: user driven research, new program-ming models, PaaS environments, and improved tools tosupport elasticity and large-scale debugging. This objec-tive of this paper is to foster discussion, and to definea roadmap forward, which will allow academia to makelonger-term impacts to the cloud computing community.

1 Introduction

Imagine a world where there was no academic cloudcomputing research. Would the field of cloud comput-ing really be any different from what it is today? Wouldit have evolved in the same way? This paper argues thatacademics are addressing the wrong class of problems tohave any lasting impact, and that cloud computing hasevolved without substantial influence from academia.

The core of this problem comes down to scale: aca-demics generally do not have low-level infrastructure(e.g., compute, storage, network) access to data centresized resources, in order to design, deploy and test al-gorithms. To circumvent this access problem academicssimulate the behaviour of data centres, and run exper-iments on small (toy) private clouds deployments. Ifthis research is then deployed at scale, there is the everpresent risk it is non representative of real world datacentres, and is therefore of limited value to the commu-nity. In addition, a large proportion of academic research

focuses on incremental improvements to existing frame-works created by industry, e.g., Hadoop optimisation fora small set of edge cases. Typically these contributionsare short-lived and have no lasting impact on the cloudcomputing community; not because these contributionsare irrelevant, but because these kinds of problems arealready being solved by industry, who do have access tolow-level infrastructure at scale [26].

Instead academics should be conducting higher riskresearch, which sheds light on more complex problems,in order to gain understanding and open up entirely newareas. We expect to see genuine cloud research focusless upon physical infrastructure and continue to em-brace abstraction through user driven research, program-ming models, and Platform as a Service (PaaS) clouds.In addition, research needs to focus on providing soft-ware engineers with better tools to program truly elasticapplications, and debug large-scale applications, whichutilise existing cloud infrastructure. The objective of thispaper is to foster discussion, and to define a roadmapforward, which will allow academia to make longer-termimpacts. Many of these observations have been derivedfrom an eight year UK research programme focused onLarge Scale Complex IT Systems (LSCITS) [25].

2 Five Pitfalls of Academic Research

Pitfall 1: Infrastructure at ScaleThe presence of unused capacity within large-scale datacentres led to the development of what became known ascloud computing. Therefore, since its inception, cloudcomputing and scale are inseparably linked. Today, rapidscalability and on demand resource provisioning are af-forded to users through the ever increasing scale of cloudproviders. Current estimates of the scale of AmazonEC2, the dominant public cloud provider, vary between156,225 and 454,400 servers [27]. Amazon Web Ser-vices (AWS) are now the largest web host on the planet,

1

which hosts numerous high profile customers includingNetflix, Instagram, DuckDuckGo and Reddit.

The scale of AWS and equivalent cloud providers (interms of servers, users, data, etc.) cannot be easily repli-cated. To produce a similar environment requires signif-icant capital expenditure, which is extremely prohibitivefor academic researchers. At best, academic researcherscan hope to reproduce a tiny subsection of the resourcesavailable through AWS. Typically this is achieved thoughthe use of a small testbed built from commodity hard-ware running, for example OpenStack [30]. Such de-ployments typically number between the tens and hun-dreds of servers, supported by gigabit Ethernet and com-modity network attached storage. As such they cannotcome close to replicating the scale of the compute, net-work or storage capacity that is expected from even thesmallest public clouds.

In order to perform an evaluation which indicates per-formance on a public cloud, there is no substitute for areal world public cloud deployment. Researchers whowish to investigate applications or phenomena runningon top of Virtual Machines (VMs) and storage from apublic cloud provider can feasibly do so. Any researchwhich must be evaluated at scale can incur significantcosts but remains a feasible means of evaluation.

If however research is focused upon low-level com-pute, storage, network, energy or other systems there isthe ever present risk of producing an evaluation whichis non representative of real world cloud deployments.Without access, academics cannot modify and evaluatelow-level cloud mechanics, making useful research inthis area all but infeasible. Work which is targeted atthe cloud infrastructure level and evaluated on a non rep-resentative private testbed can at best deliver interim re-sults, which suggest how it may behave when deployedin a real world context.

This has caused a dichotomy within academic cloudresearch: between those researchers with access andpartnership with cloud providers and those without. Tworecent examples to illustrate the point include: the latestbest paper award at IEEE Cloud 2013 [31] was awardedfor collaborative research between Princeton and NEC,the best paper at SIGCHI (non-cloud research but re-quires access to data) was awarded for research betweenFacebook and Carnegie Mellon University [12]. This ac-cess problem prevents academics from having any last-ing impact, and has significant implications for systemsresearch investigating the lower levels of the cloud stack.For academics that are used to low-level access this bar-rier to research is unprecedented.

There are efforts underway with the European UnionBONFIRE [9] project, and Open Cirrus [29] to provideacademics with access to large-scale resources. Indi-vidual academics are predominantly concerned with re-

search and not maintaining testbeds and research sys-tems. A high turnover of researchers and a emphasis onresearch (and not maintenance) often leads to the neglectand eventually the premature abandonment of testbeds.To ensure the longevity of a research system, with appro-priate documentation and maintenance, dedicated staffare necessary. This cannot be easily afforded be indi-vidual institutions. Academics need to collaborate, selforganise and build their own data centres at the nationallevel (through government funding etc.) in order to pro-vide the facilities and maintenance necessary for aca-demics to conduct meaningful cloud computing research.

Pitfall 2: AbstractionAbstraction is one of the key features of the cloud com-puting model. The main principle of public cloud com-puting models is that a user can make use of VMs as ifthey were offered as a service, and the complexities ofmanaging and maintaining hardware are abstracted froman end user. This translates into the fact that such an ab-straction results in an ‘observation-only’ or a ‘black box’model. Unlike the grid, the low-level machine detailsare concealed on the cloud. This has a number of com-pelling advantages over grid computing architectures in-cluding elastic on-demand architecture and high serviceavailability [6].

However, in academic cloud computing research, suchan abstraction in which the cloud is merely used as a ser-vice or tool for computing has been of minimal interest,e.g., deployment of a Physic’s application on the cloudto compare performance with a cluster environment. Inaddition, a lot of academic cloud computing research hasa reverse engineering focus, where low-level infrastruc-ture components are often (poorly) reimplemented forthe purposes of research prototypes. This not only con-tradicts the underlying principle of public cloud comput-ing models (to be used as a service), but such researchmeets with little support from the cloud providers them-selves. However, research which has built on top of exist-ing cloud platforms in order to provide new functionalityhas been enormously successful. RightScale [32] is anexemplar of how academic cloud computing research hasadded innovative new features and been commercialised.

Pitfall 3: Non-reproducible ResultsAs academics do not have low-level access to data cen-tre resources, why not simulate a data centre instead?This is a reasonable response from academia to the prob-lem of access to such large-scale resources, and hasworked well in the networking community, e.g., NS3[28]. There are multiple simulators, which attempt tomodel and simulate a cloud computing environment, in-

2

cluding CloudSim [13] and GreenCloud [24]. Simula-tions can be effectively used as a prototyping mechanismto provide a rough idea of how a particular algorithmmay perform. The problem with each of these simula-tion tools is that it is very difficult to verify if the simu-lation environment is an accurate representation of a realworld data centre environment. To date there have beenno attempts by the community to align simulations withreal world data from industry – refer to pitfall 5 for afurther discussion of industrial relations. Furthermore,real world data centres are constantly evolving, and aresubject to both planned and unplanned changes. There-fore even if a simulation model is verified at a particularpoint in time, this will no longer hold if the simulationis rerun at a later point in time. Research which is builton top of these simulation environments (e.g., simulationof VM migration) has next to no real world application,other than to demonstrate a prototype algorithm or appli-cation.

An alternative approach for academics without directaccess to large-scale infrastructure is to build simulationswith real world trace data. While there are a few exam-ples [8, 36] of companies releasing trace data for aca-demics, we consider this to be the exception rather thanthe rule, and traces are only available for a very limitedset of applications.

In order to get a paper accepted to one of the top-tiercloud computing conferences a thorough evaluation mustbe presented for peer-review. Usually this evaluation hasa very specific environment in terms of hardware andsoftware, which may very well be a bespoke setup upfor the purposes of getting a paper accepted for publi-cation. Reproducing these results is next to impossibleunless a replica of the evaluation environment can be de-ployed, or access is granted to the original environment.Furthermore, results of an evaluation are usually not gen-eral purpose; just because an expected behaviour was ob-served on a 10 node private cloud, does not mean that thesame results will be observed on a 10,000 node data cen-tre. There are currently no reliable mathematical modelsof how clouds scale over time, this is something that wehighlight as an immediate research opportunity, and canonly be solved through improved relationships betweenacademia and industry, discussed further in pitfall 5.

Pitfall 4: Rebranding

Academia is notoriously trend driven, and in order togain competitive funding, government PhD programmesand collaboration academics must be working in certaincurrent areas of research. Much of academic cloud com-puting research is simply grid computing, or cluster com-puting, or e-Science rebranded as cloud computing re-search.

The novelty of the cloud over preceding infrastruc-tures such as clusters and grids include features such aselasticity, scalability, on-demand availability and mini-mal user maintenance of the resources [18]. The majorityof academic cloud computing research does not requireany of these core properties, which differentiate cloudsfrom other forms of infrastructure.

Researchers within the cloud domain must embracethe avenues of research which the paradigm allows for.Research which places its focus at the bottom of thecloud stack at the level of virtualisation, OS and systemsresearch may have significant fundamental implicationsfor cloud computing, but many of these results cannotbe tested by the wider academic community due to lackof access to infrastructure at scale listed in pitfall 1. Webelieve that academic research which operates higher upthe stack is likely to have a longer-term impact.

Pitfall 5: Industrial Relations

As discussed throughout this short paper, the only effec-tive way to develop and test new cloud services is to de-ploy them at a scale. Unfortunately there are virtuallyno mechanisms which allow academics to gain accessto data centres in order to conduct low-level systems re-search. This is a completely understandable stance byindustry, why would a company like Google, or Amazonallow academics access to production, or even test datacentres in order to evaluate new academic research? Vis-iting programmes at Google such as the Visiting FacultyProgramme [20] do exist, but are not available to the vastmajority of the academics working on cloud computingresearch. There are examples of University research labsworking in tandem with cloud infrastructure providers;a recent success story is the Algorithms Machines andPeople Lab (AMPLab) [1], at UC Berkeley. However,these relationships are difficult to establish, and are onlyavailable to the very top-tier institutions with access toreliable industry contacts.

There are multiple programmes which allow aca-demics to use the cloud as a service, for example theAWS Education programme [7], Windows Azure for Re-search programme [39], and Google App Engine Re-search Awards [19]. These programmes are ideal forbuilding and testing applications which sit within an IaaSor PaaS cloud. However these programmes do not solvethe problem with low-level infrastructure access, whichis where the majority of current academic research lies.

3

3 Five Opportunities for Academic Re-search

Opportunity 1: User Driven Research

We believe the enthusiasm in the academic communityfor cloud research needs to be directed towards problemsthat genuinely require the exploitation of elasticity, scal-ability and abstractions offered by the cloud. A num-ber of such problems exist in science, but a large pro-portion can be addressed on clusters and simply do notrequire the core properties of clouds. The requirementsof such domain experts need to be understood and thencloud-based environments developed in order to facili-tate scientific discovery. Here we make the differenti-ation between building clouds for science applications,which is not what we are suggesting, but using existingcloud services effectively, which make use of elasticity,scalability and suitable abstractions. One potential areathat can be explored in this context are virtual privateresearch environments using simple cloud tools but tai-lored to suit specific domain requirements. Such an ap-proach is adopted in the ELVIRA [37] and CARMEN[38] projects.

Another area that needs to be investigated is the de-velopment of environments that support budget-limitedcomputation based on a set of user driven requirements.Let us assume that a job needs to be executed for a pro-longed period but is constrained in the total availablebudget for computation. How can the job be most effec-tively executed across clouds, and what models of pre-dicting computational workloads and calculating costsneed to be developed. Here again we make a differenti-ation between developing cost models for the cloud, andcost computation and prediction models on top of exist-ing clouds, and the latter is our suggestion for pursuingin an academic environment. User driven research on thecloud overlaps with research that needs to focus on PaaSenvironments which is considered in opportunity 4.

Opportunity 2: Programming Models

Engineers need more efficient, scalable and usable pro-gramming models, which will allow them to take fulladvantage of the core properties of cloud computing.Through the use of programming models built on topof cloud resources, developers should be able to writescalable, elastic code without having to mange low-levelcloud primitives.

At present, MapReduce is the default programmingmodel for applications which deal with large data sets,and has been a highly usable approach to dealing withdata. However, MapReduce is not always the appro-priate model and there are seldom alternatives. There

is some variation available from tools which are builtupon MapReduce including Apache Hive [4], Cloud-era Impala [17] and Apache Pig [5]. These tools of-fer programming abstractions which are not available invanilla Hadoop, however they still rely upon the under-ling MapReduce functionality.

There are several alternative programming modelswhich are beginning to emerge which deviate from con-ventional MapReduce semantics. The Apache Spark [41]project is a successful example of a project which hasmade the transition from academic research into produc-tion cluster environments in major companies. Sparkis an engine for large-scale data processing which of-fers higher levels of performance for certain applicationclasses when compared to MapReduce. Another emerg-ing programming model is Apache Drill [3], an opensource reimplementation of Google BigQuery whichis designed for high throughput queries on very largedatasets. Drill is currently an incubator project at Apacheand like many other programming models hinges on fur-ther development and mass adoption to ensure its suc-cess.

Opportunity 3: Bugs in Large-Scale Appli-cationsLarge-scale systems are inherently difficult to manage.The level of complexity and rate of change in such sys-tems presents a constantly moving target, which requiressubstantial talent and manpower to mange. Debugginglarge-scale cloud applications is exceptionally difficult,due to the scale of a deployment, the abstraction fromthe underlying hardware, and the nature of a remote de-ployment. Research should focus on a tool chain whichallows engineers to collect, visualise and inspect the stateof jobs running across many thousands of cores. Toolsshould focus on the emergent properties of large-scaleapplications; properties which can only be observedwhen applications are deployed at the scale offered bycloud resources. Research into this area can be built ontop of existing cloud platforms and wouldn’t necessarilyrequire low-level access to large-scale testbeds.

In academia, however, this area has received insuf-ficient attention hitherto the means to develop and de-bug large-scale applications are under researched. Thereremains only a small body of research in this crucialarea [42, 15], which limits the ability of academics totranslate prototypes into real world software. A lack ofknowledge forces current research to be discarded or sig-nificantly rewritten in order to be applicable to large-scale systems. Through further investigation into thesoftware engineering of large-scale systems, academicscan produce superior cloud research which supports thescale of systems that are now becoming commonplace.

4

Opportunity 4: PaaS Environments

Platform as a Service (PaaS) clouds aim to offer a higherlevel of abstraction, away from bare-bones infrastructureservices. At this level of the cloud stack the focus ison automating certain tasks, which if done by manuallywriting programs can be a time consuming task. In otherwords an abstract environment, which incorporates elas-ticity and on-demand service features offered through aneasy to use interface [11, 40].

Generic frameworks such as StarCluster [35] andCloud Foundry [16] exist to address some of the au-tomation described above. PaaS environments for sci-ence are gaining momentum (for example, Helix Neb-ula: The Science Cloud [21]). We believe this is oneavenue of research that can benefit from being pursued.Here the focus needs to be placed on building environ-ments, which make use of cloud infrastructure providedby multiple vendors, and at the same time providing ahigh-level interface that will allow scientists to quicklyand easily access the resources. Such research can reapmaximum benefit if based on user driven requirementsand directed towards developing stand-alone and inde-pendent environments, which make use of the variety ofPaaS features offered by the providers.

Opportunity 5: Elasticity

The most significant benefits of cloud computing are ob-tained by applications that scale over a large number ofresources. Traditionally, in grid and cluster environmentsthe resources requested for an application had to be pro-visioned before its execution. Cloud computing, how-ever, facilitates dynamic resource provisioning and pro-vides a high level abstract interface for doing this. Whileelasticity is often confused with scalability and work-load management, there are overlaps which should not beavoided [22]. Within this area, we believe that there arenumerous challenges, which can be addressed for drivingacademic cloud research forward.

The first challenge within elasticity is related to re-source provisioning, an area which overlaps with scala-bility [23, 33]. Through the Infrastructure as a Servicelayer large pools of abstracted resources can be quicklyprovisioned. This facilitates a wide range of research op-portunities. Notably the issues of efficient provisioningare not yet fully addressed. There is a need for researchwhich investigates how to avoid under and over provi-sioning, how to provision for performance and cost, andhow to efficiently compute requirements.

The second challenge is related to workload manage-ment, which again overlaps with the issue of scalabil-ity [2]. How can frameworks use workload informationfor provisioning resources, or is it possible to anticipate

the workload which can be used to provision resourcesthereby reducing the effects of variable times in provi-sioning resources.

The third challenge is related to modelling elasticity[14, 10]. An attractive feature of the cloud is its abstractview of elasticity and how easily resources can be provi-sioned. One important question that arises is can the ab-stract view also present different views of the system. Forexample, what level of performance can be obtained byprovisioning n resources when the workload increases bym%, what cost models can best capture these scenarios,what additional costs will be incurred when n resourcesare added. There are ongoing efforts to address theseissues (e.g., the SPEC RG Cloud Working Group [34]),but we believe this needs to be a direction for refocusingacademic cloud computing research. Addressing thesechallenges can elicit collaborations with industry.

4 Conclusion

Contemplating whether academics are addressing thewrong class of problems to have any lasting impact onthe cloud computing community motivated this discus-sion paper. Its objective is to foster a useful discussionabout how academia can focus its efforts to solve mean-ingful problems.

We have argued that there are five fundamental pitfalls,which restrict academics from conducting research at theinfrastructure level, which is where the vast majority ofcurrent academic research focuses. These pitfalls are re-lated to restricted access to data centre sized resources,the abstraction provided by the cloud, reproducing re-sults in a real cloud environment, rebranding previousresearch, and a lack of industrial relations that prohibitmeaningful cloud research in academia.

We expect to see genuine cloud research focus lessupon physical infrastructure and continue to embraceabstraction through user driven research, programmingmodels, and PaaS clouds. In addition research needs tofocus on providing software engineers with better toolsto program truly elastic applications, and debug large-scale applications, which utilise existing cloud infras-tructure.

References[1] ALGORITHMS MACHINES AND PEOPLE LAB. http://

amplab.cs.berkeley.edu/.

[2] ANDREOLINI, M., CASOLARI, S., COLAJANNI, M., ANDMESSORI, M. Dynamic load management of virtual machines incloud architectures. In Cloud Computing, D. Avresky, M. Diaz,A. Bode, B. Ciciani, and E. Dekel, Eds., vol. 34 of LectureNotes of the Institute for Computer Sciences, Social-Informaticsand Telecommunications Engineering. Springer Berlin Heidel-berg, 2010, pp. 201–214.

5

[3] APACHE DRILL. http://incubator.apache.org/drill/.

[4] APACHE HIVE. http://hive.apache.org/.

[5] APACHE PIG. https://pig.apache.org/.

[6] ARMBRUST, M., FOX, A., GRIFFITH, R., JOSEPH, A. D.,KATZ, R., KONWINSKI, A., LEE, G., PATTERSON, D.,RABKIN, A., STOICA, I., AND ZAHARIA, M. A view of cloudcomputing. Communications of the ACM 53, 4 (Apr. 2010), 50–58.

[7] AWS EDUCATION PROGRAMME. http://aws.amazon.com/education/.

[8] BENSON, T., AKELLA, A., AND MALTZ, D. A. Network traf-fic characteristics of data centers in the wild. In Proceedings ofthe 10th ACM SIGCOMM Conference on Internet Measurement(New York, NY, USA, 2010), IMC ’10, ACM, pp. 267–280.

[9] BONFIRE. http://www.bonfire-project.eu/.

[10] BREBNER, P. C. Is your cloud elastic enough?: Performancemodelling the elasticity of infrastructure as a service (iaas) cloudapplications. In Proceedings of the 3rd ACM/SPEC Interna-tional Conference on Performance Engineering (2012), ICPE’12, pp. 263–266.

[11] BUNCH, C., ARORA, V., CHOHAN, N., KRINTZ, C., HEGDE,S., AND SRIVASTAVA, A. A pluggable autoscaling service foropen cloud paas systems. In Proceedings of the 2012 IEEE/ACMFifth International Conference on Utility and Cloud Computing(2012), UCC ’12, pp. 191–194.

[12] BURKE, M. B. M., AND KRAUT, R. Growing closer on face-book: Changes in tie strength through social network site use (toappear). In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems (2014), ACM.

[13] CALHEIROS, R. N., RANJAN, R., BELOGLAZOV, A.,DE ROSE, C. A. F., AND BUYYA, R. Cloudsim: a toolkit formodeling and simulation of cloud computing environments andevaluation of resource provisioning algorithms. Software: Prac-tice and Experience 41, 1 (2011), 23–50.

[14] CHAISIRI, S., LEE, B.-S., AND NIYATO, D. Optimization ofresource provisioning cost in cloud computing. Services Com-puting, IEEE Transactions on 5, 2 (April 2012), 164–177.

[15] CHILIMBI, T. M., LIBLIT, B., MEHRA, K., NORI, A. V., ANDVASWANI, K. Holmes: Effective statistical debugging via effi-cient path profiling. In Proceedings of the 31st International Con-ference on Software Engineering (Washington, DC, USA, 2009),ICSE ’09, IEEE Computer Society, pp. 34–44.

[16] CLOUD FOUNDRY. http://cloudfoundry.com/.

[17] CLOUDERA IMPALA. https://github.com/cloudera/

impala.

[18] FOSTER, I., ZHAO, Y., RAICU, I., AND LU, S. Cloud comput-ing and grid computing 360-degree compared. In Grid Comput-ing Environments Workshop, 2008. GCE ’08 (Nov 2008), pp. 1–10.

[19] GOOGLE APP ENGINE RESEARCH AWARDS. http:

//research.google.com/university/relations/

appengine/index.html.

[20] GOOGLE VISITING FACULTY PROGRAMME. http:

//research.google.com/university/relations/

visiting-faculty/.

[21] HELIX NEBULA. http://helix-nebula.eu.

[22] HERBST, N. R., KOUNEV, S., AND REUSSNER, R. Elasticity inCloud Computing: What it is, and What it is Not. In Proceedingsof the 10th International Conference on Autonomic Computing(2013).

[23] HU, Y., WONG, J., ISZLAI, G., AND LITOIU, M. Resourceprovisioning for cloud computing. In Proceedings of the 2009Conference of the Center for Advanced Studies on CollaborativeResearch (2009), CASCON ’09, pp. 101–111.

[24] KLIAZOVICH, D., BOUVRY, P., AUDZEVICH, Y., AND KHAN,S. Greencloud: A packet-level simulator of energy-aware cloudcomputing data centers. In Global Telecommunications Confer-ence (GLOBECOM 2010), 2010 IEEE (Dec 2010), pp. 1–5.

[25] LSCITS. http://lscits.cs.bris.ac.uk/.

[26] MATT WELSH: VOLATILE AND DECENTRALIZED.http://matt-welsh.blogspot.co.uk/2011/05/

how-can-academics-do-research-on-cloud.html.

[27] NETCRAFT. Amazon web services growth unrelenting.http://news.netcraft.com/archives/2013/05/20/amazon-web-services-growth-unrelenting.html.

[28] NS3. http://www.nsnam.org/.

[29] OPEN CIRRUS. http://opencirrus.intel-research.

net/.

[30] PROJECT, T. O. http://www.openstack.org/.

[31] RAPOLU, N., CHAKRADHAR, S., HASSAN, A., AND GRAMA,A. M-lock: Accelerating distributed transactions on key-valuestores through dynamic lock localization. In Cloud Computing(CLOUD), 2013 IEEE Sixth International Conference on (June2013), pp. 179–187.

[32] RIGHTSCALE. http://www.rightscale.com/.

[33] SHARMA, U., SHENOY, P., SAHU, S., AND SHAIKH, A. Acost-aware elasticity provisioning system for the cloud. In Dis-tributed Computing Systems (ICDCS), 2011 31st InternationalConference on (2011), pp. 559–570.

[34] SPEC RG CLOUD WORKING GROUP. http://research.

spec.org/working-groups/rg-cloud-working-group.

html.

[35] STARCLUSTER. http://star.mit.edu/cluster/.

[36] SWIMPROJECTUCB. https://github.com/

SWIMProjectUCB/SWIM/wiki/Workloads-repository.

[37] VOSS, A., BARKER, A., ASGARI-TARGHI, M., VAN BALLE-GOOIJEN, A., AND SOMMERVILLE, I. An elastic virtual infras-tructure for research applications (elvira). Journal of Cloud Com-puting 2, 1 (2013), 1–11.

[38] WATSON, P. e-science in the cloud with carmen. In Parallel andDistributed Computing, Applications and Technologies, 2007.PDCAT ’07. Eighth International Conference on (Dec 2007),pp. 5–5.

[39] WINDOWS AZURE FOR RESEARCH PROGRAMME. http://

research.microsoft.com/en-us/projects/azure/.

[40] YANGUI, S., AND TATA, S. Cloudserv: Paas resources pro-visioning for service-based applications. In Proceedings of the27th IEEE International Conference on Advanced InformationNetworking and Applications (2013), pp. 522–529.

[41] ZAHARIA, M., CHOWDHURY, M., FRANKLIN, M. J.,SHENKER, S., AND STOICA, I. Spark: Cluster computing withworking sets. In Proceedings of the 2Nd USENIX Conferenceon Hot Topics in Cloud Computing (Berkeley, CA, USA, 2010),HotCloud’10, USENIX Association, pp. 10–10.

[42] ZHOU, B., TOO, J., KULKARNI, M., AND BAGCHI, S.Wukong: Automatically detecting and localizing bugs that man-ifest at large system scales. In Proceedings of the 22Nd In-ternational Symposium on High-performance Parallel and Dis-tributed Computing (New York, NY, USA, 2013), HPDC ’13,ACM, pp. 131–142.

6