CONTACT

Standard Performance Evaluation Corporation (SPEC)7001 Heritage Village Plaza, Suite 225Gainesville, VA 20155, USA

SPEC Research GroupChair: Samuel Kounev (kounev@kit.edu)Web: http://research.spec.org

®

CONTENTS OF THIS ISSUE2 SPECResearchGroupOfficers2 SPECResearchWorkingGroups2 WelcometotheSPECResearchGroupNewsletter3 SPECResearchGroupMissionStatement3 SPECAnnouncements5 Winner of the SPEC Distinguished Dissertation

Award20126 Call-for-Nominations: SPEC Distinguished Dis-

sertationAward20137 FINCoS:AframeworkforPerformanceEvaluation

ofEventProcessingSystems7 DiSL:ADomain-SpecificLanguageforInstrumen-

tation-Based Dynamic ProgramAnalysis on theJavaPlatform

8 Report of the SPEC RG IDS BenchmarkingWorkingGroup

8 AnnualReportoftheCloudWorkingGroup9 New Technical Report in RG Cloud Working

Group:CloudUsagePatterns10 Report fromtheFourthACM/SPECInternational

Conference on Performance Engineering (ICPE2013)andCollocatedEvents

10 ICPE2013:BestPaperAwards11 ShortReportsfromSPECFace-to-FaceMeetings11 SPECtacularAwards forOutstandingService to

SPECin201212 ICPE2013:AbstractsofSelectedPapers14 ICPE2014:CallforPapers

CALL FOR NOMINATIONS – SPEC DISTINGUISHED DISSERTATION AWARD TheSPECDistinguishedDissertationAwardaimsto

recognizeoutstandingdoctoraldissertationsinthefieldof computer benchmarking, performance evaluation,andexperimentalsystemanalysisingeneral.Themainprize–$1000–willbeawardedatICPE2014.

Read more on page 6

NEW TECHNICAL REPORT: CLOUD USAGE PATTERNS

The SPEC Research Group announces a newtechnicalreportonatextualandgraphicalformalismfordescribingcommoncloudusagescenarios.Thedocu-mentistargetedatawideaudiencewithpossibilitiesofapplicationinvariousareasofCloudcomputingprovid-ingacommonreferenceforusagescenarios.

Read more on page 9

THE ‘BEST PAPER’ AWARDS FROM THE ICPE 2013 CONFERENCE

At the 2013 International Conference onPerformance Engineering (ICPE 2013) in Prague,Czech Republic, five outstanding papers were dis-tinguished by the Program and Awards Chairs. Theawardswerepresentedintwodifferentcategories:BestIndustry-RelatedPaper,andBestResearchPaper.

Read more on page 10

NEW TOOLS IN PEER-REVIEWED TOOLS REPOSITORY

The SPEC Research Repository of peer-reviewedtools for quantitative system evaluation and analysisincludestwonewtools:DiSLandFINCoS.

Read more on page 7

1

SPEC ResearchSM Group Newsletter

Volume 1, Issue 2 June 2013

℠Research

SPEC, the SPEC logo and the names SPECjbb, SPECjAppServer, SPECpower_ssj, SPECvirt_sc, SPECweb, SERT, and SPEC PTDaemon are trademarks of the Standard Performance Evaluation Corporation. The SPEC Research Logo and the name SPEC Research are service marks of SPEC. Additional company, product and service names men-tioned herein may be the trademarks or service marks of their respective owners. Copyright © 1988-2013 Standard Performance Evaluation Corporation (SPEC). Reprinted with permission. All rights reserved.

2

SPEC RESEARCH GROUP OFFICERSChair: Samuel Kounev, Karlsruhe Institute of Technology (KIT), Germany, kounev@kit.eduVice-chair: Kai Sachs, SAP AG, GermanySecretary: Klaus-Dieter Lange, HP, USARelease Manager: Qais Noorshams, KIT, GermanySteering committee: J. Nelson Amaral, University of Alberta, CanadaAlexandru Iosup, TU Delft, NetherlandsLizy John, University of Texas at Austin, USASamuel Kounev, KIT, GermanyKlaus-Dieter Lange, HP, USAMatthias Müller, RWTH Aachen, GermanyMeikel Poess, Oracle Corporation, USAKai Sachs, SAP AG, GermanySeetharami R. Seelam, IBM TJ Watson Research Center, USAPetr Tůma, Charles University of Prague, Czech RepublicNewsletter Editor:Piotr Rygielski, KIT, Germany

SPEC RESEARCH WORKING GROUPS

Cloud Working GroupChair: Alexandru Iosup, TU Delft, NetherlandsVice-Chair: Erich Nahum, IBM TJ Watson Research Center, USASecretary: Aleksandar Milenkoski, KIT, GermanyRelease Manager: Nikolas Herbst, KIT, Germanyhttp://research.spec.org/working-groups/rg-cloud-working-group.html

IDS Benchmarking Working GroupChair: Marco Vieira, University of Coimbra, PortugalVice-Chair: Alberto Avritzer, Siemens Corporate Research, USASecretary: Aleksandar Milenkoski, KIT, GermanyRelease Manager: Aleksandar Milenkoski, KIT, Germanyhttp://research.spec.org/working-groups/ids-benchmarking-working-group.html

WELCOME TO THE SPEC RESEARCH GROUP NEWSLETTER

We are delighted to present to you the 2nd is-sue of the SPEC Research Group Newsletter. Thisregular publication provides information on latest de-velopments, news and announcements relevant tothe benchmarking and quantitative system evalua-tion communities. Our newsletter is part of our mis-sion to foster the exchange of knowledge and ex-periences between industry and academia in thefield of quantitative system evaluation and analysis.

Thepastsixmonthshavebeenveryintenseandex-citingfortheSPECRG.Someofthemajoractivitiesandmilestonesreached include: launchingofanewwork-inggrouponintrusiondetectionsystems(IDS);publish-ingthefirstofficialRGtechnicalreportoncloudusagepatterns–a textual and visual formalism for describingcommon cloud usage scenarios; establishing a long-time cooperation betweenACM and SPEC for futureICPEconferences;andfinalizingtheICPEconstitution.Inaddition,wehadaverysuccessfulICPE2013confer-ence andHotTopiCS 2013workshop,which attractedmorethan160attendeesfromindustryandacademia.

The SPEC Distinguished Dissertation Award2012 attracted again many high-quality nominationsmaking the selection of the winner very challeng-ing. The winner of this years’ award is Dr. ShicongMeng from Georgia Tech for his thesis “Monitoring-as-a-Service in the Cloud” supervised by Prof. LingLiu from the Georgia Institute of Technology.

Wewouldliketousethisopportunitytowelcomeournewmembers:MITLincolnLabs,HassoPlattnerInstitute,UniversityofNorthFlorida,andUniversityofStuttgart.

In thisnewsletteryouwill findmany interestingar-ticles that describe achievements and major eventsfromthelastsevenmonthsofSPECRG’sactivity.Wewant to highlight several articles thatmay be of spe-cialinteresttothereader.WithintheSPECAnnounce-ments you can find the call for newmembers of theCloudWorking Group.We also invite nominations totheSPECDistinguishedDissertationAward.Wepres-ent two new tools that were accepted for the Peer-reviewed Tools Repository. Reports from activities ofourworkinggroupswillgivetheinsight intoouractivecommitment in theCloudand IDS researchdomains.Ourfirsttechnicalreporton“CloudUsagePatterns”ishighlyrelevanttotheCloudComputingcommunity.WeclosetheissuewithreportsfromICPE2013,theannualSPECmeetingsandthecallforpapersforICPE2014.

We hope that you will enjoy reading this is-sue and welcome and encourage your contribu-tions forarticlesandsuggestions for future coverage.

Samuel Kounev (KIT), Kai Sachs (SAP AG), Piotr Rygielski (KIT)

3

SPEC RESEARCH GROUP MISSION STATEMENT

The SPEC Research Group (RG) is one of thefour groups of the Standard Performance Evalua-tion Corporation (SPEC). Its mission is to promoteinnovative research in the area of quantitative sys-temevaluation and analysis by serving as a platformfor collaborative research efforts fostering the inter-action between industry and academia in the field.

The scopeof the group includes computer bench-marking, performance evaluation, and experimentalsystemanalysisconsideringbothclassicalperformancemetricssuchas response time, throughput,scalabilityandefficiency,aswell asothernon-functional systemproperties includedunder the termdependability,e.g.,availability, reliability, and security. The conducted re-search efforts span the design of metrics for systemevaluation aswell as the development ofmethodolo-gies,techniquesandtoolsformeasurement,loadtest-ing, profiling, workload characterization, dependabil-ity and efficiency evaluation of computing systems.

Samuel Kounev (KIT)

SPEC ANNOUNCEMENTS

We Want You! SPEC RG Cloud WG is Looking for New Members

May 8, 2013

Ifyouworkincloudcomputingandhaveaninterestinperformance,youarewelcometojointheCloudWorkingGroupoftheSPECResearchGroup.Weunderstandper-formanceinabroadsense,includingcloudbenchmark-ing,quantitativeevaluation,andexperimentalanalysis.

By joining the Cloud Working Group, you will getaccess toadedicatedgroupof industryandacadem-ic experts with whom you can engage in meaning-ful discussions about how the performance of cloud-based systemscanbemeasured, youwill beable tojoin the SPEC standardization processes, and youwill find a new group of potential employees or po-tentialplaces tosendyourstudents todo internships.

WithintheCloudWorkingGroup,wearecommittedtorapidlyintegratingournewmembers,ineitherongoingoremergingactivities,inparticipatoryorleadershiproles.Wealsokeeptheoverheadlow(under10%ofthesched-uledmeetingtime),sothatavastmajorityofyourandoureffortscanbeputintomeaningfulworkandcontent.http://research.spec.org/working-groups/rg-cloud-working-group.html

Alexandru Iosup (TU Delft)

SPEC releases Server Efficiency Rating Tool (SERT)

March 23, 2013

On February 26th, 2013 SPEC released the firstversion of the Server Efficiency Rating Tool (SERT).Developed at the request of the US EnvironmentalProtection Agency (EPA) the SERT will be used to

generate power efficiency data as input to the newlylaunchedENERGYSTARforComputerServersVersion2 program. The SERT was developed by the sameteam that previously created SPECpower_ssj2008,whichhasbecometheindustrystandardformeasuringthepowerefficiencyofservers.

TheSERTofferssignificantlyenhancedeaseofuse,with automated sophisticated hardware configurationdiscovery,poweranalyzer rangesettingandagraphi-caluser interface targetingbothnaïveandexpertus-ers.Theexpertmodetargetsresearchersandprovidesconfiguration flexibility thatoffers fullcontroloverhowindividual workloads are invoked or combined. Theworkloads included in the first release test individualsubsystems;CPU,memory, storage, anda combinedworkload.

Initial use of the SERT in the ENERGY STAR forservers V2 program will provide power consumptiondata to theEPA.Thesedatawillalsobeusedby theSPECpowerCommittee to furtherenhance theSERT,and specifically will be used to aid in the creation ofpowerefficiencymetricsforsubsequentreleases.TheSERT has been designed to be highly flexible andextensible, and its use by theEPA is being reviewedfor potential useby international organizationsacrosstheworld.WidespreadadoptionofSERTwillleadtoamoreconsistentapproachtopowerefficiencymeasure-mentacrossmanycurrentandgrowingmarketsinthenextfewyears.

http://www.spec.org/sert/

Hansfried Block, SERT Release Manager

SPECjbb2013 ReleasedNovember 11, 2012

SPEC announces the release of SPECjbb2013,developedfromthegrounduptomeasureperformancebased on the latest Java application features. It isexpected to be usedwidely by all those interested inJava server performance, including JVM vendors,hardware developers, Java application developers,researchersandmembersoftheacademiccommunity.Newfeaturesincludesupportformultiplerunconfigura-tions,exerciseofnewJava7features,andsupportforvirtualizationandcloudenvironments.

http://www.spec.org/jbb2013/

www.spec.org

SPEC PTDaemon 1.6.0 EnhancementMarch 23, 2013

March2013sawthereleaseofPTDaemonV1.6.0,which includes several significant changes. Criticalenhancements offer improved support for several ac-cepted power analyzers, together with added optionsfor non-standard baud rates and better handling ofpoweranalyzerauto-zerofunctions.Testinghasraisedconcerns with the use of serial cables that do nothaveallpinsconnectedleadingtofailuresandmissedreadings;guidancehasbeenaddedtothePowerand

4

Temperature Measurement Setup Guide to enablebetter results to be achieved across the full range ofaccepteddevices.

Support for theYokogawaWT310 power analyzerandtheiButtonLinkT-Sense(R)andT-Probetempera-turesensors(with theLinkUSBi1-wireUSBinterface)have also been added.Three-phase powermeasure-ment isnowfullysupported,withappropriatelyconfig-ured and capable power analyzers, for benchmarksthatsupportthree-phasemeasurements.

This release also includes the first set of changestoprovidefutureDirect-Currentmeasurementsupport.Thesechangesarestillawork-in-progress.Therefore,theappropriateAPIshavebeenchanged(forDConly)to ensure that PTDaemon users cannot inadvertentlyuseDCforbenchmarkresults.

http://www.spec.org/power/docs/SPECpower-Device_List.html

Greg Darnell, SPEC PTDaemon Release Manager

SPEC Remembers Alan AdamsonNovember 11, 2012

The computer-performance benchmarking commu-nitymournsthelossofAlanAdamson,amemberoftheStandardPerformanceEvaluationCorporation(SPEC)BoardofDirectors.Alandied inhissleep thenightofOctober31,2012.

AlanAdamsoncametoSPECasanIBMrepresen-tative specializing in compiler and Java performance.Withinashortperiodoftime,hebecamevitallyinvolvedinalmosteveryaspectofSPECwork.Afterhis retire-mentfromIBMin2009,hecontinuedtoworkunpaidforawiderangeofSPECactivities.

AlanservedaschairoftheJavaandOSGSteeringCommittees.Hehelpedestablish,anddidmuchofthehard work to create, the SPECResearchGroup andtheICPEseriesofjointconferenceswithACM.

At the time of his death, he served as the SPECBoard’s inspector of elections, organizing the elec-tion process for the 2013 SPEC annual meeting. Hewas also a supporting contributor to theOSGPowerCommittee.

Alanwas the closest thing to SPEC’s official pho-tographer,andawiseadviserfortechnical,proceduralandmoral issues.SPEC representatives fromaroundtheworldwillmisshimasa friendandcolleague.Heis survived by hiswifeMelitta, hismother, andmanysiblings,nephews,niecesandcousins.

http://www.spec.org/spec/memories/alan/alanadamson.html

www.spec.org

Elections of Research Group’s Steering Committee and Officers

April 12, 2013

In April 2013, the members of SPEC ResearchGroup voted to elect theSteeringCommittee and of-ficers of SPEC Research. The following individualshavebeenelectedtotheSPECResearchGroupSteer-ing Committee to serve until 2014: J. NelsonAmaral

(University of Alberta), Alexandru Iosup (TU Delft),Lizy John (University of Texas, Austin), Klaus-DieterLange (HP), Matthias Müller (RWTHAachen Univer-sity).Additionally, for two-year cadence, the followingindividuals were elected to serve until 2015: SamuelKounev(KIT),MeikelPoess(Oracle),KaiSachs(SAPAG), Seetharami Seelam (IBM TJ Watson ResearchCenter), and Petr Tuma (Charles University Prague).

The steering committee elected the followingRG Officers: Chair: Samuel Kounev (KIT), Vice-Chair: Kai Sachs (SAPAG), Secretary: Klaus Lange(HP), Release Manager: Qais Noorshams (KIT).

Piotr Rygielski (KIT), David Schmidt (HP)

SPEC Releases New Server Virtualization Benchmark

May 22, 2013

The Standard Performance Evaluation Corp.(SPEC) has released a new version of its SPECvirtbenchmarkthatreflectstheheavyvolumeandsuddenpeaks of traffic faced by datacenter servers used forvirtualizedserverconsolidation.

The new SPECvirt_sc2013 uses real-worldworkloads and SPEC’s proven performance- andpower-measurementmethodologiestoenablevendors,usersandresearcherstocomparesystemperformanceacrossmultiplehardware,virtualizationplatforms,andapplications.ItwasdevelopedbytheSPECvirtualiza-tion subcommittee, whose members and contributorsincludeHP,IBM,Intel,Oracle,RedHat,andVMware.

“Organizations around the world are virtualizingservers to handle significantly heavier and moredynamic traffic,”saysPaulaSmith,chairof theSPECvirtualization subcommittee. “We are responding withworkloadsthatmodelsimilarbehavior,requiringvirtualmachineswithmorememoryandvirtualCPUstomeetthebenchmark’squality-of-servicerequirements.”

SPECvirt_sc2013 provides four workloads basedonreal-worldtrafficfacedbyawebserver,aJavaap-plicationserver,anIMAPserverandabatchserver.Aharnesson theclientsidecontrols theworkloadsandimplementsthemethodologiesforpowermeasurementderivedfromtheSPECpower_ssj2008benchmark.

ThenewSPECvirtbenchmarkbetterreplicatesthetypeofmulti-tieredapplicationsanddatabaseenviron-ments that runonenterprise-classvirtualizedservers.Each virtualized database server inSPECvirt_sc2013issharedbyasmanyasfourapplication-servervirtualmachines.

SPECvirt_sc2013 shares the general benchmarkarchitecture with SPECvirt_sc2010, including thebenchmarkharnessandmostof theapplicationwork-loadsandVMtypes.However,sinceSPECvirt_sc2010was developed, many companies are virtualizingsignificantly heavier workloads than those in the firstgeneration benchmark. In SPECvirt_sc2013, we areproviding heavier andmore burstyworkloads that re-quirevirtualmachineswithmorememoryandvCPUsand further stress the system’s ability to meet thebenchmark’s quality-of-service requirements. The listbelowsummarizessomeofthekeychangesmadefor

5

SPECvirt_sc2013:• The J2EE Application workload based on

SPECjAppServer2004 has been modified tofurther increase theburstinessof theworkloadsothattheinjectionrateaverages100IRvs20IRinSPECvirt_sc2010.

• TheDatabaseServerVMisnowsharedbyuptofourAppserverVMs(fewerifnumberoftilesisnotamultipleoffour)toensurethatthereisaVMrequiringmultiplevCPUs(SMP).

• TheWebServerworkloadbasedontheSupportworkload inSPECweb2005hasbeenmodifiedto require SSL (HTTPS) between the clientsand the web server, and the load level hasbeenincreasedto2500SimultaneousSessions(was500).

• The Mail-Server workload, based onSPECmail2008,hasbeenmodifiedtopresentaburstyworkloadby increasing the variability ofthethinktimesinthebenchmark.

• The IdleServer inSPECvirt_sc2010hasbeenreplaced by the Batch Server that periodi-callyrunsasetofjobsthatutilizethe401.bzip2benchmarkandharnessfromSPECCPU2006.Thisreplacementallowsthejobstobeparallel-izedbasedonthenumberofvCPUsinthebatchserverVM.Liketheidleserver,thisworkloadisnot scored but must meet QoS requirementsforcompletingeachsetofbatchjobswithinsetlimitsaswellasforbeingresponsivetoperiodicclientpolling.

More informationand initial benchmark results areavailableonline.

http://www.spec.org/virt_sc2013/

David Schmidt (HP)

Over 160 Attendees at ICPE 2013May 14, 2013

Over 160 participants from industry and academiaattended the4thACM/SPECInternationalConferenceon Performance Engineering (ICPE) 2013 in Prague,CzechRepublic.Thetechnicalprogramincluded20fullresearchpapers,2shortpapers,8industrial/experiencepapersand10work-in-progresspapersselected fromover 80 submissions over all tracks. It also featured3 keynote speakers of international repute, 6 poster/demopresentations,4tutorialsand1invitedtalk.

The conference collocated two workshops,HotTopiCS and MultiCloud. HotTopics attracted 54participantswithitsfocusonnewand“hot”topicsinthefieldofCloudServices.MultiCloudhad47participantsand aimed to investigate concepts,models and toolsforrunningapplicationsonmultipleandheterogeneousCloudsystems.

ThenextICPEconferencewill takeplaceinMarch2014inDublin,Ireland.

http://icpe2013.ipd.kit.edu/

Qais Noorshams (KIT)

WINNER OF THE SPEC DISTINGUISHED DISSERTATION AWARD 2012

January 24, 2013

TheResearchGroupof theStandardPerformanceEvaluation Corp. (SPEC) selected the research prizetobeawarded toaPh.D.studentwhose thesis is re-gardedtobeanexceptional,innovativecontributioninthescopeoftheSPECResearchGroupforthesecondtime. Nine nominationswere submitted and reviewedby the selection committee.The chair of the nomina-tion committee, Prof. Dr. Wilhelm Hasselbring fromKielUniversity inGermany,mentionsthat thecommit-tee had again a demanding task to select thewinneramongtheninehigh-qualitynominationsandthat it isgreattoseesomuchexceptionalworkinscopeoftheSPECResearchGroup. The criteria for the selectionare theoverall contribution in termsof scientific origi-nality, practical relevance, impact, and quality of thepresentation.

Afteranin-depth,multi-phaseselectionprocessthecommitteeselected the followingnomination:ShicongMengwhowasnominatedbyProfessorLingLiu fromthe Georgia Institute of Technology with the topic“Monitoring-as-a-Service in the Cloud”. The winnerreceived$1000,whichwasawardedattheICPE2013InternationalConferenceinPrague,CzechRepublic.

Of the nine nominations, the selection committeedecidedforanhonorablementionofthefollowingthreenominations,inalphabeticalorder:• Karthik Ganesan who was nominated by

Professor Lizy Kurian John from the Universityof Texas at Austin with the topic “AutomaticGenerationofSyntheticWorkloads forMulticoreSystems”.Mentioned for its contributions to thescopeoftheSPECResearchGroup.

• TilmannRablwhowasnominatedbyKarlHupplerfrom the Transaction Processing PerformanceCouncil with the topic “Efficiency in ClusterDatabase Systems”. Mentioned for its potentialpracticalrelevance.

• DanielSanchezwhowasnominatedbyChristosKozyrakisfromStanfordUniversitywiththetopic“HardwareandSoftwareTechniquesforScalableThousandCoreSystems”.Mentioned for its sci-entificoriginality.

Members of the selection committee for 2012 in-clude:Wilhelm Hasselbring, Kiel University, Germany(Chair);DavidA.Bader,GeorgiaInstituteofTechnology,USA;WalterBays,OracleCorp.,USA;WalterBinder,University of Lugano, Switzerland; Rema Hariharan,AMD, USA; Samuel Kounev, Karlsruhe Institute ofTechnology (KIT), Germany; Diwakar Krishnamurthy,UniversityofCalgary,Canada;Klaus-DieterLange,HP,USA;JohnMurphy,UniversityCollegeDublin,Ireland;MarcoVieira,UniversityofCoimbra,Portugal.

The SPEC Distinguished Dissertation Award wasestablished in2011torecognizeoutstandingdisserta-tions within the scope of the SPEC Research Groupin terms of scientific originality, scientific significance,practicalrelevance,impact,andqualityofthepresenta-tion. The scope of SPEC’s Research Group includes

6

computerbenchmarking,performanceevaluation,andexperimental system analysis in general, consideringboth classical performancemetrics such as responsetime, throughput, scalability and efficiency, aswell asothernon-functionalsystemproperties includedunderthe termdependability,e.g.,availability, reliability,andsecurity.

Contributionsofinterestspanthedesignofmetricsfor system evaluation as well as the development ofmethodologies, techniques and tools for measure-ment,loadtesting,profiling,workloadcharacterization,dependability and efficiency evaluation of computingsystems.

Wilhelm Hasselbring (Kiel University)

CALL FOR NOMINATIONS: SPEC DISTINGUISHED DISSERTATION

AWARD 2013May 7, 2013

The SPEC Distinguished DissertationAward aimsto recognize outstanding doctoral dissertations in thefield of computer benchmarking, performance evalu-ation, and experimental system analysis in general.Nominated dissertationswill be evaluated in terms ofscientific originality, scientific significance, practicalrelevance,impact,andqualityofthepresentation.

The scopeof theSPECResearchGroup includesthe aforementioned fields, considering both classicalperformancemetricssuchas response time, through-put, scalability and efficiency, as well as other non-functional system properties included under the termdependability, e.g., availability, reliability, and security.Contributionsofinterestspanthedesignofmetricsforsystemevaluationaswellasthedevelopmentofmeth-odologies, techniques and tools for measurements,load testing, profiling, workload characterization,dependability, and efficiency evaluation of computingsystems.Thewinnerwill receive$1000,whichwillbeawardedattheICPE2014.

Anominationshouldinclude:• A nomination letter (including the name of the

student,thetitleofthedissertation,theinstitutionwhere the dissertation was defended, and thedateofthedefense).

• A one page report that outlines the outstandingcontributionofthedissertation.

• The dissertation itself including a one page ex-tendedabstractofthedissertation.(Ifthedisser-tationiswritteninlanguageotherthanEnglish,itmaybeaccompaniedbypublications inEnglish,describingthesameresearchasthedissertation.)

TheSPECDistinguishedDissertationAwardisopento dissertations that have been defended betweenOctober1,2012,andSeptember30,2013.Ifthereareseveral outstanding submissions, the committee maysplittheawardbetweenthem.ThesubmissiondeadlineisSeptember30,2013.Nominationsarewelcomeatanytimebeforethefinalsubmissiondeadline.Nominations,or questions about the application process, shouldbe sent by e-mail to: nominations@spec.org. Self-nominationsarenotaccepted.

Selection committee (members that confirmedparticipationatthetimeofpublishing):MatthiasMüller,RWTHAachen University, Germany, (Chair); SamuelKounev, Karlsruhe Institute of Technology, Germany;WalterBinder,UniversityofLugano,Switzerland;JohnMurphy, University, College Dublin, Ireland; MarcoViera,UniversityofCoimbra,Portugal;GerhardWellein,Friedrich-Alexander-University, Erlangen-Nürnberg,EdgarGabriel,UniversityHouston,USA.

Matthias Müller (RWTH Aachen University)

Karthik Ganesan receives the distinction from Samuel Kounev.

Tilmann Rabl receives the distinction from Samuel Kounev.

7

DISL: A DOMAIN-SPECIFIC LANGUAGE FOR INSTRUMENTATION-BASED

DYNAMIC PROGRAM ANALYSIS ON THE JAVA PLATFORM

April 8, 2013

Dynamicprogramanalysis(DPA)techniquesenableessentialsoftwareengineeringtools,suchasprofilers,debuggers, and testing tools.DPA tools often rely oncode instrumentation, i.e., the insertion of analysiscodeatselectedlocationsintheanalyzedprogram.Asnowadaysmanyprogramminglanguagesarecompiledtobytecode for theJavaVirtualMachine (JVM),byte-codeinstrumentationhasbecomeacommontechniqueforbuildingDPAtoolsfortheJVM.

JVM bytecode instrumentation is supported bya variety of bytecode engineering libraries, such asASM,BCEL,Javassist,orShrikeBT, tomentionsomeof them.Generally, thesetoolsofferarather lowlevelofabstraction,whichresultsinhighdevelopmenteffortforbuildingDPA tools; thesourcesofsuch tools tendto be verboseanddifficult tomaintain and to extend.Ontheotherhand,bytecodemanipulationlibrariesgivetheexpertdevelopermanyopportunitiestooptimizetheinsertedcode,enablingefficientDPAtools.

Someresearchershaveproposedtheuseofaspect-oriented programming (AOP) for creating DPA tools.Thankstoitspointcut/advicemechanism,AOPoffersahigh abstraction level that allows to concisely specifycertain kinds of instrumentations. Consequently, theuse of AOP promises to significantly reduce the de-velopmenteffort for buildingDPA tools.Unfortunately,mainstream AOP languages such as AspectJ lackjoinpointsatthelevelof,e.g.,basicblocksofcodeorindividualbytecodes thatwouldbeneeded forcertainDPAtools.Inaddition,aspectweaversoftenintroducehigh overhead due to complex code transformationsinvolved. For example, in AspectJ, the access todynamic reflective join point information may involveobjectallocationsanditerations(e.g.,tostoretheargu-mentsofamethodinanarray)thatarenotvisibleintheinstrumentationcode.

DiSL is a new domain-specific language for JVMbytecode instrumentationthatreconcilesthestrengthsof low-level bytecode manipulation libraries (i.e., ex-pressivenessandefficiency)with thestrengthofAOP(i.e.,high-levelprogrammingabstractions).DiSLallowsprogrammers to rapidly specify custom DPA tools,which tend to be as concise as equivalent toolswrit-teninamainstreamAOPlanguage(ifthisispossible).However, in contrast to prevailing AOP languages,DiSL features an open join point model where anyregion of bytecodes in a program can become a joinpoint. Moreover, the DiSL weaver produces efficientcode that does not incur any unexpected costs; theweaver introducesneitherobjectallocationsnor loops

FINCOS: A FRAMEWORK FOR PERFORMANCE EVALUATION OF EVENT

PROCESSING SYSTEMSApril 3, 2013

FINCoS is a set of benchmarking tools for loadgeneration and performance measurement of eventprocessing (EP) systems. It provides a flexible andneutral approach through which users, researchersand engineers can quickly run realistic performancetests on one ormore EP platformswithout having tocode themselves generationof the load,performancemeasurementsandeventconversionroutines.

Theframeworkcanbeusedtoevaluateanyevent-processing platform capable of exchanging eventsthrough the standard JMSAPI. In addition, an exten-sible set of custom-code adapters permits to submitloadsdirectly toEPengines, using the client librariesprovidedbyvendors.

FINCoS is not bound to any specific product orworkload.Therefore,itcanbeusedbothbyend-userswilling toevaluatecandidateEPplatforms,andby re-searchers developing novel benchmarks for the area.For that, the framework offers great flexibility and awiderangeofoptions in thedefinitionofexperimentalevaluations.UsersofFINCoScanchoosetoconfiguresyntheticworkloads,witheventsbeinggeneratedbytheframework itself, or, instead, usedata files containingreal-world event traces as input. The framework alsoallows the evaluation of the response of EP systemstochanges.This isobtainedbybreakingperformancetestsintooneormorephases,eachwithitsownwork-loadcharacteristics(i.e.,eventtypes,inputrate,arrivalprocess,etc.)

Load generation and result processing in FINCoScanbedistributedacrossmultiplenodesandcentrallyorchestrated using a graphical application, denomi-nated FINCoSController. The results of performanceruns can then be visualized, both in real-time andafter test completion, using the FINCoSPerformanceMonitor component. The tool presents performancestats inboth tabularandgraphical formats–theformerdisplaysasnapshotofthroughputandlatencyforeachcontinuousqueryrunningattheEPsystemundertest,while the latter shows the evolution of these metricsovertime.

FINCoSdevelopmentstartedin2007,aspartoftheBiCEPbenchmarkinginitiative,atUniversityofCoimbra(http://bicep.dei.uc.pt). Its firstversionwasreleased in2008andsincethenithasbeenconsiderablyextendedandimproved.Recently,theframeworkhasundergoneathoroughreviewprocess,havingbeenacceptedtobeintegrated in the SPEC Research Group’s repositoryof quantitative evaluation and analysis tools. FINCoSis an open-source tool, and can be downloaded freeof charge.Auser guideanda tutorial explaininghowtouse the frameworkarealsoavailable in theprojectwebsite.

http://research.spec.org/en/tools.html

Marcelo Mendes (University of Coimbra)

8

thatwerenotwrittenby theprogrammer.Thismakesitespeciallyattractiveinperformanceevaluationtasks,wherehiddenoverheadsareparticularlydisruptive.

DiSLisavailableopen-sourceundertheApache2.0licenseonOW2.

http://disl.o w2.org/

Lukas Marek (Charles University, Czech Republic), Yudi Zheng (Univeristy of Lugano, Switzerland), Danilo Anasaloni (Univeristy

of Lugano, Switzerland), Aibek Sarimbekov (Univeristy of Lugano, Switzerland), Lubomir Bulej (Charles Univeristy, Czech Repub-

lic), Walter Binder (Univeristy of Lugano, Switzerland), Petr Tuma (Charles Univeristy, Czech Republic), Zhengwei Qi (Shanghai Jiao

Tong University, China)

REPORT OF THE SPEC RG IDS BENCHMARKING WORKING GROUP

May 7, 2013

TheSPECRG IDSBenchmarkingWorkingGroupwas created in July 2012 with the goal of contribut-ing towards addressing the increasing demand forrepresentative and rigorous benchmarks for intrusiondetection systems (IDSes) in cloud platforms. Topicsofinterestinclude,butarenotlimitedto,thefollowing:security- and performance-related IDS benchmarkingmetrics,workloadsbasedonvulnerabilityandattackin-jection,IDSbenchmarkingmethodologies,IDSbench-markingmethodswithrespect to featuresandproper-tiesofcloudplatforms,andattackdetectionbasedonperformance signatures. The elected officers of theWorking Group are: Chair – Marco Vieira (Universityof Coimbra, Portugal); Vice-Chair – Alberto Avritzer(Siemens Corporate Research, USA); and Secretary/ReleaseManager–AleksandarMilenkoski (KarlsruheInstituteofTechnology,Germany).

Since its inception, the group has established aresearch agenda and has produced output that ispublished or is currently under consideration for pub-lishing.Theplannedresearchactivitiesofthegrouparedescribed in awork-in-progress paper titled “Towardsbenchmarking intrusion detection systems for virtual-ized cloud environments” [1]. In addition, the groupwroteaTechnicalReport [2]analyzingstate-of-the-artintrusiondetectiontechniquesappliedinvirtualizeden-vironmentsandarchitecturesofVMM(VirtualMachineMonitor)-based IDSes. The Technical Report alsodescribes challenges that apply specifically to bench-markingVMM-basedIDSesfocusingonworkloadsandmetrics.Moreover,theWorkingGroupproducedasur-vey paper systematizing the accumulated knowledgeandcurrentpracticesintheareaofIDSbenchmarkingbyusinganIDSbenchmarkingdesignspacestructuredinto threeparts:workload,metrics, andmeasurementmethodology.Thesurveyprovidesanoverviewof thecurrent practices in IDS benchmarking by analyzingand comparing benchmarking approaches, tools, andmethodsrelatedtoeachpartofthedesignspace.

Currently,theSPECRGIDSBenchmarkingWorkingGroup is definingmetrics that take into considerationon-demandresourceprovisioninginvirtualizedenviron-ments (e.g., CPU and memory hotplugging, memoryballooning). Further, vulnerability injection techniques

for generating malicious workloads consisting of at-tacks against hypervisors are being considered. Thegroupisexpandingitsmembershipwithnewindustrialandacademicorganizations.

[1] Aleksandar Milenkoski and Samuel Kounev. To-wards Benchmarking Intrusion Detection Systems for Virtualized Cloud Environments (Extended Abstract). In Proceedings of the 7th International Conference for Inter-net Technology and Secured Transactions (ICITST 2012), London, United Kingdom, December 10-12, 2012, pages 562-563. Infonomics Society, IEEE. December 2012.

[2] Aleksandar Milenkoski, Samuel Kounev, Alberto Avritzer, Nuno Antunes, and Marco Vieira. On Benchmark-ing Intrusion Detection Systems for Virtualized (Cloud) En-vironments. Technical Report SPEC-RG-2013-002 v. 1.0, SPEC Research Group – IDS Benchmarking Working Group, June 2013. To appear.

Aleksandar Milenkoski (KIT), Marco Vieira (University of Coimbra), Alberto Avritzer (Siemens Corporate Research, USA)

ANNUAL REPORT OF THE CLOUD WORKING GROUP

May 10, 2013

TheSPECRGCloudWorkingGroupispleasedtoreport that in2012wehaverenewedourcommitmenttocloudbenchmarking,quantitativeevaluation,andex-perimentalanalysis,with relevance forbothacademiaandindustry.Injustover25onlinemeetings,wehavegatheredguestsfrom16organizations,hosted7expertlectures on relevant topics, and started 2 content-relatedworkactivities.Forthesecondpartof2012,wehavemaintainedoverheadofunder10%ofourmeetingtime, effectively maximizing our contribution to CloudWorking Group’s collective mission to facilitate andfosterleadingresearchincloudcomputingthroughex-periencesharing,constructivefeedback,agreementondefinitions, specificationofmetrics,anddisseminationofnovelmethods,techniquesandtoolsforquantitativeevaluationandanalysisofcloud-basedsystems.

ExperiencesharingwasakeyactivityfortheCloudWorkingGroupin2012andwillcontinuetobefortheupcoming years. Through online and face-to-facemeetings, we have exchanged best-practices relatedto cloud benchmarking methodologies, performanceevaluationtoolsandbenchmarkingsuites,performancemetrics, and performance evaluation results for com-mercial cloud services (e.g., the IaaS Amazon WebServices, the PaaS Google App Engine, the SaaSDropbox, etc.) Among the lecturers, we have hostedrepresentatives of the EPFL (CH), KIT (DE), and TUDelft(NL)universities;andSAP(DEandUK)andIBM(US). Our dedicated group of industry and academicexperts includesnowseveralof theserepresentativesandmore.TheCloudWorkingGrouphasalsofosteredcollaborationbetweenitsmembers,withjointacademicwork,researchvisits,andcompanyinternships.

Overthecourseof2012,theCloudWorkingGrouphasbeenengaged inmeaningful dialogueabout howtheperformanceofcloud-basedsystemscanbemea-sured. The dialogue has resulted, for example, in aninnovative formalism for cloud usage patterns, which

9

willhelpacademicsandpractitionersshareinformationabout IaaS, PaaS, andSaaS usage.Other topics forwhichpreliminaryworkwasstartedin2012includesur-veyingexistingcloudbenchmarkingmetricsandtools,and (depending on future participation) formulatingmethodologiesforbenchmarkingvariouscloudapplica-tion domains (e.g., IaaS cloud services, IaaS/PaaS/SaaSBigDataprocessingservices,andSaaSonlinegamingservices).

In2012,theCloudWorkingGrouphasbeenrepre-sentedbyitsmembersatmanyinternationalacademicandindustrialvenues,includingthroughaward-winningpublications. Internally within SPEC, our group hasmaintained contact with the OSG’s implementationefforts,andcontributed to the reviewandsubsequentpublicationofseveralbenchmarkingtools.

To conclude, 2012 was a successful year for theCloud Working Group. Our promise is to continuepursuingourmissionandstrengthenourcontributionin2013andtheyearstocome.http://research.spec.org/working-groups/rg-cloud-working-group.html

Alexandru Iosup (TU Delft)

NEW TECHNICAL REPORT OF THE RG CLOUD WG: CLOUD USAGE PATTERNS

April 4, 2013

The SPEC Research Cloud Working Group an-nounces thepublicationof the technical report “CloudUsagePatterns:AFormalismforDescriptionofCloudUsageScenarios”,which is the result of collaborationbetween representatives of multiple research and in-dustrialSPECResearchGroupmemberorganizations,such asKarlsruhe Institute ofTechnology (Germany),TU Delft (Netherlands), SAP AG (Germany), andSalesforce.com(USA).Cloudcomputingisbecominganincreasinglylucrativebranchoftheexistinginformationand communication technologies. Enabling a debateaboutcloudusagescenarioscan help with attracting newcustomers, sharing best-practices,anddesigningnewcloud services. In contrastto previous approaches,whichhaveattemptedmainlyto formalize the commonservicedeliverymodels (i.e.,Infrastructure-as-a-Service,Platform-as-a-Service, andSoftware-as-a-Service),in the technical report, theauthorsproposea formalismfordescribingcommoncloudusage scenarios referred toascloudusagepatterns.Theproposed formalism takes astructuralist approach allow-ingdecompositionofacloudusage scenario into ele-ments corresponding to the

�V�i�r�t�u�a�l�i�z�a�t�i�o�n

�I�a�a�S

�P�a�a�S

�S�a�a�S

�H�a�r�d�w�a�r�e

ni.s.e i.e (ni3.)(i2.)i.s.e

3

2 3 2

(ni3.)(i2.)s.e

Visual form of several cloud usage patterns.

common cloud service delivery models. Furthermore,the formalismconsiders several cloudusagepatternsthat have recently emerged, such as hybrid servicesandvaluechainsinwhichmediatorsareinvolved.Wepropose a simple yet expressive textual and visuallanguage for our formalism, andwe show how it canbe used in practice for describing a variety of realworldcloudusagescenarios.Thescenarios forwhichwe demonstrate our formalism include resourceprovisioning of global providers of infrastructure and/or platform resources (e.g., Amazon Web Services,GoogleAppEngine),onlinesocialnetworkingservices(e.g.,Facebook),user-dataprocessingservices,onlinecustomer and ticketing services, online asset man-agement and banking applications, CRM (CustomerRelationship Management) applications, and onlinesocialgamingapplications(e.g.,Zynga’sapplications).Forinstance,theuseofleasedinfrastructureresourcesbyGo!AnimatefromtheinfrastructureproviderAmazonWebServices for servicingusersprocessingamateurvideos, can be formally described with the string“i3.s.e6”,whereeachletterrelatestooneoftheknowncloudservicedeliverymodels(i.e.,Infrastructure-as-a-ServiceandSoftware-as-a-Serviceinthegivenstring),and/or a stakeholder (i.e., an end-user). Also, eachnumber relates to a volume of provisioned resourcesthatmaybeexpressed, forexample,asa logarithmicvalue(e.g.,3 for1000 in thegivenstring).For furtherdetailsonthecloudusagepatterns,wereferthereadertothetechnicalreport.

http://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC-RG-2013-001_

CloudUsagePatterns.pdf

Aleksandar Milenkoski (KIT), Alexandru Iosup (TU Delft)

10

REPORT FROM THE 4TH ACM/SPEC INTERNATIONAL CONFERENCE ON

PERFORMANCE ENGINEERING (ICPE 2013) AND COLLOCATED EVENTS

May 7, 2013

IntroductionOver 160 participants from industry and academia

coming from 25 different countries attended the 4thACM/SPEC InternationalConference onPerformanceEngineering (ICPE) 2013 in Prague,CzechRepublic.The technical program included 20 full research pa-pers, 2 short papers, 8 industrial/experience papersand 10 work-in-progress papers selected from over80 submissions over all tracks. It also featured threekeynote speakers of international repute, six poster/demopresentations, four tutorialsandone invited talkonSPECjbb2013.

The conference collocated two workshops,HotTopiCS and MultiCloud. HotTopics attracted 54participantswithitsfocusonnewand“hot”topicsinthefieldofCloudServices.MultiCloudhad47participantsand aimed to investigate concepts,models and toolsforrunningapplicationsonmultipleandheterogeneousCloudsystems.

Collocated WorkshopsThisyear’s ICPEwasaccompaniedwith twocollo-

catedworkshops.OnSaturday,April20,theHotTopiCSWorkshopstartedwiththekeynotegivenbyAlexandruIosup (TUDelft). The topic of the talk concerned ap-proaches,challengesandexperienceswithIaaSCloudbenchmarking.OnSunday,SamuelKounev(KIT)gavetheinvitedtalkentitled“RELATE:AResearchTrainingNetwork on Engineering and Provisioning of Service-BasedCloudApplications”.Duringtheworkshop’sses-sions,theparticipantshadanuniqueoccasiontofollowten presentations about the newest trends and ideasintheCloudComputingdomain.TheworkshopendedwithajointlunchonSunday.

On Monday morning, the second collocatedworkshop started: “MultiCloud”. This one-day eventgathered researchers and companies interested inissues, problem and ideas concerning the followingtopics:Multi-Cloudbackgroundandgeneralissues,de-velopmentofMulti-Cloudapplications,monitoringandexecutionsupport,Cloudmigration.TheworkshopwasconcludedwiththediscussionpanelandajointcoffeeattheroofoftheDorintHotel.

The ConferenceThemainconference,4thACM/SPECInternational

ConferenceonPerformanceEngineering(ICPE2013),startedwiththeTutorialstrackonSunday,April21.ThemaintracksoftheconferencestartedonMondaymorn-ing with the keynote speech “AutomatedModeling ofComplexDataCenterApplications”givenbyPrashantShenoy (University of Massachusetts). After thekeynote, all participants listened to the talks givenbycandidates to theResearchTrackBestPaperAward.The sessions that followed were a perfect occasion

ICPE 2013: BEST PAPER AWARDSMay 14, 2013

At the 2013 International Conference onPerformance Engineering (ICPE 2013) in Prague,Czech Republic, two outstanding papers were distin-guishedbytheProgramandAwardsChairs.

The Best Research Track Paper Award from re-search track was presented to Balaji Subramaniamand Wu-Chun Feng for the paper “Towards Energy-Proportional Computing for Enterprise-Class ServerWorkloads”.

The nominees to theBest PaperAward in the re-searchtrackwerethefollowingtwopapers:ChristophHeger,JensHappe,RoozbehFarahbod for thepaper“Automated Root Cause Isolation of PerformanceRegressions Variaduring Software Development”,and Jean-Michel Fourneau, Katinka Wolter, Philipp

to see the highest ranked papers of the conference.During the coffee breaks between the sessions, theparticipantswerediscussingtheirideasandproblems.

Theseconddayof theconferencewasopenedbythe second keynote, entitled “Supporting OperationsPersonnel through Performance Engineering”,given by Len Bass from National Information andCommunications Technology Australia (NICTA). TheTuesday’sresearchtracksessionsattractedtheatten-tionoflistenersuntillateafternoon,whentheVisionandWorkinProgresssessionstarted.Duringthissession,researcherspresentedten interestingtopics thatwereselectedforpublication,althoughtheresearchisstillinprogress.Wehopethattheinterestingideaspresentedduringthissessionwillbeagreatmotivationtosubmitfollow-upresultstothenextICPEinDublin.

The main conference ended onWednesday,April24,whenthelastkeynotetalkwasgiven.Thetalkwasentitled “Data Centric Computing for Internet ScaleEnterprises” and presented byYuqingGao from IBMT.J.Watson Research Center. Before the conferencewas closed, the participants listened to the presenta-tions of the candidates to the Industry Track BestPaperAward.

SPEC Face-to-Face MeetingsEach ICPE is a perfect occasion for face-to-face

meetings of researchers and industry representativesthat work in the domain of performance engineering.The members of SPEC also use this opportunity tomeetinpersonanddiscussthetopicsoftheirinterests.During this year’s ICPE, SPEC organized five face-to-facemeetings:SPECOSGCPU,SPECOSGJavaEnterprise, SPEC OSG Power, SPEC Research andthemeetingsof theSteeringCommittees.Thereportsfrom the face-to-facemeetingwithin SPECResearchareincludedinthisissueofthenewsletter(readmoreonpage11).Wehope,theywillgivethereadersanin-sightofselecteddiscussiontopicsandbeaninvitationtojoinSPECResearchoroneofitsworkinggroups.

Piotr Rygielski (KIT), Qais Noorshams (KIT), Petr Tuma (Charles University of Prague)

11

SHORT REPORTS FROM FACE-TO-FACE MEETINGS

May 14, 2013

The SPEC Research Annual MeetingThemembersoftheSPECResearchGroupmetto-

getheronMay24th,2013.ThemeetingwaschairedbySamuelKounevwhopresentedthegroupanditswork-inggroupshighlightingmanysuccessesachievedinthepreviousyear.Afterthisintroduction,theattendeesin-troducedthemselves,presentedtheirscientificinterestsandinvolvementinthegroups.Duringthemeeting,thechairmanhighlightedthattheSPECResearchGroupisopenforeveryone,hasdiverseareasofresearchfocusanditsmainaimistofosterthecollaborationbetweenindustryandacademia.Moreover, thebenefitsof join-ingthegrouphavebeenpresented.

Later, the chair of the IDSWorkingGroup,MarcoVieira,sharedhisexperiencesabouttheprocessofes-tablishinganewworkinggroupwithinSPECResearch.Themeetingwasconcludedbyan introductionof thetwoworkinggroups–CloudandIDS.Afterashortbreak,themeetingoftheCloudWorkingGroupfollowed.

Piotr Rygielski (KIT), Meikel Poess (Oracle)

The Cloud Working GroupThe SPEC RG Cloud Working Group met face-

to-face in Prague on May 24th, 2013. The meetingopened with the introduction of the chair AlexandruIosup.Heintroducedtopicsofinterestofthegroupandannounced to the attendees that the Cloud WorkingGrouplooksfornewmembers(readmoreonpage3).

After the introductory part, Aleksandar Milenkoskipresented the first Technical Report of the CloudWorking Group entitled “Cloud Usage Patterns: AFormalism forDescriptionofCloudUsageScenarios”(seepage9).Thepresentationwasconcludedbyadis-cussionthatshowedthatthedocumentisimportantforindustrialpartners.Moreover,manyideasofextendingtheformalismwereproposed.

In the next part of the meeting, Nikolas Herbstpresentedmaterialaboutanupcomingtechnicalreportthatfocusesoncloudperformancebenchmarking.The

presentationgave insight to the initial progress in thepreparationofthisdocument,especiallyontheelastic-ity benchmarking. The presentation interested manyattendeesandtheyactivelydiscussedonthepresentedtopics. The meeting was closed by Alexandru Iosupand the attendees headed to thePrague castle for ajointdinnerwherediscussionswerecontinued.

Piotr Rygielski(KIT)

The IDS Working GroupA face-to-face meeting of the SPEC RG IDS

BenchmarkingWorkingGrouptookplaceinPrague,onMay25th, 2013.Thiswas the first face-to-facemeet-ing of the group, which periodically meets online.Allmembersoftheworkinggroupwererepresentedinthemeeting,whichwasalsoattendedbyrepresentativesofotherSPEGRGworkinggroups.

Duringthismeeting,wefirstdiscussedmanagementaspectsofthegroupandapproachesforattractingnewmembers. Afterwards, we focused on technical as-pects,discussingrequirementsandapproachesforthegenerationofmaliciousworkloads,includingtheuseofvulnerabilityandattackinjectiontechniques.

Marco Vieira (University of Coimbra)

Reinecke,TilmanKraussforthepaper“MultipleClassG-NetworkswithRestart”.

The distinction–Best Industry Track PaperAward–was presented to Klaus-Dieter Lange, Jeremy A.Arnold,HansfriedBlock,NathanTotura,JohnBeckett,MikeG.Tricker for thepaper “Further ImplementationAspectsoftheServerEfficiencyRatingTool(SERT)”.

TheothernomineetotheBestPaperAward in theIndrustry/ExperineceTrackwas the paper byTilmannRabl,MeikelPoess,Hans-ArnoJacobsen,PatO’Neil,BettyO’Neil“VariationsoftheStarSchemaBenchmarktoTestDataSkewinDatabaseManagementSystems“

Congratulations to the award winners and the au-thorsofdistinguishedpapers!

Piotr Rygielski (KIT)

SPECTACULAR AWARDS FOR OUTSTANDING SERVICE TO SPEC IN 2012

May 29, 2013

During SPEC’s 2012 Quarterly Meeting in Prague(CzechRepublic), RGOfficersAlexandru Iosup,QaisNoorshams and Aleksandar Milenkoski were recog-nizedbytheSPECPresidentandtheSPECRGChairfortheiroutstandingservicetoSPECin2012.

Alexandru Iosup (TU Delft) received an awardfor his outstanding service as elected Chair drivingthe development and establishment of theSPECRGCloudWorkingGroup.QaisNoorshams(KIT)receivedanawardforhisoutstandingserviceaselectedSPECRGReleaseManager.AleksandarMilenkoski(KIT)re-ceivedanawardforhisoutstandingserviceaselectedSecretaryoftheRGCloudandRGIDSworkinggroups,amongotherthingsleadingtheworkonthefirstofficialtechnical report of SPEC RG. In addition, SamuelKounev (KIT)was recognized bySPECPresident forhiscontinuedservicedrivingthefurtherestablishmentand development of the SPEC Research Group andtheICPEconference.

Congratulationstoallawardrecipients!Piotr Rygielski (KIT)

12

ICPE 2013: ABSTRACTS OF SELECTED PAPERS

May 14, 2013

Balaji Subramaniam, Wu-chun Feng: Towards Energy-Proportional Computing for Enterprise-

Class Server Workloads (Research Track Best Paper Award)

Massivedatacentershousingthousandsofcomput-ing nodes have become commonplace in enterprisecomputing, and the power consumption of such datacenters is growing at an unprecedented rate.Addingto theproblem is the inabilityof theservers toexhibitenergyproportionality, i.e.,provideenergy-efficientex-ecutionunderall levelsofutilization,whichdiminishesthe overall energy efficiency of the data center. It isimperativethatwerealizeeffectivestrategiestocontrolthepowerconsumptionof theserverand improvetheenergy efficiency of data centers.With the advent ofIntel SandyBridge processors,we have the ability tospecify a limit on power consumption during runtime,whichcreatesopportunitiestodesignnewpower-man-agementtechniquesforenterpriseworkloadsandmakethesystemsthattheyrunonmoreenergyproportional.

In this paper,we investigatewhether it is possibleto achieve energy proportionality for an enterprise-class server workload, namely SPECpower ssj2008benchmark, by using Intel’s Running Average PowerLimit (RAPL) interfaces. First, we analyze the powerconsumption and characterize the instantaneouspower profile of the SPECpower benchmark withindifferentsubsystemsusing theon-chipenergymetersexposedviatheRAPLinterfaces.Wethenanalyzetheimpact of RAPL power limiting on the performance,per-transaction response time, power consumption,andenergyefficiencyofthebenchmarkunderdifferentloadlevels.Ourobservationsandresultsshedlightonthe efficacy of theRAPL interfaces andprovide guid-ance fordesigningpower-management techniques forenterprise-classworkloads.

Christoph Heger, Jens Happe, Roozbeh Farahbod:

Automated Root Cause Isolation of Performance Regressions During Software Development

(Research Track Best Paper Nomination)

Performanceiscrucialforthesuccessofanapplica-tion.Tobuildresponsiveandcostefficientapplications,softwareengineersmustbeabletodetectandfixper-formanceproblemsearly in thedevelopmentprocess.

Samuel Kounev (left) receives the SPECtacular Award from SPEC President Walter Bays.

Samuel Kounev and Walter Bays present the Award to Alexandru Iosup.

Aleksandar Milenkoski receives the SPECtacular Award from Walter Bays and Samuel Kounev.

Walter Bays presents the Award to Qais Noorshams.

13

Existingapproachesareeither relyingonahigh levelof abstraction such that critical problems cannot bedetectedorrequirehighmanualeffort.Inthispaper,wepresentanovelapproach that integratesperformanceregressionrootcauseanalysis into theexistingdevel-opment infrastructure using performance-aware unittests and the revision history. Our approach is easyto use and provides software engineers immediateinsightswithautomatedrootcauseanalysis.Inareal-isticcasestudybasedonthechangehistoryofApacheCommons Math, we demonstrate that our approachcanautomaticallydetectandidentifytherootcauseofamajorperformanceregression.

Jean-Michel Fourneau, Katinka Wolter, Philipp Reinecke, Tilman Krauss, Alexandra Danilkina :

Multiple Class G-Networks with Restart (Research Track Best Paper Nomination)

Restart is a common technique for improving re-sponse-timesincomplexsystemswherethecausesofdelayscaneithernotbediscerned,ornotbeaddressedbytheuser.Withrestart,theuserabortsarunningjobthatexceedsadeadline,andresubmitsittothesystemimmediately.Inmanycommonscenarios,thisapproachcan reduce the response-times that the user experi-ences. Restart has been well-studied for scenarioswhere only one user applies restart, and typically incaseswherequeueingeffectscanbeneglected.Inthispaperweapproachthequestionofrestartinascenariowhere restart is applied by many users in a systemthat can bemodelled as an open queueing network.We apply the G-Networks formalism to this problem.Weusenegativecustomerstomodeltheabortionandretry of a request.TheopenG-networkusesmultipleclasseswithphase-typedistributedservicetimes.Thisallowstheapproximationofapreemptiverepeatdiffer-entbehaviouras it isnatural formultiple restartsofarequest.We compute the response timeof a requestandshowthatanoptimalrestartintervalcanbefound.Theresultsarecomparedwithsimulation.

Klaus-Dieter Lange, Jeremy A. Arnold, Hans-fried Block, Nathan Totura, John Beckett, Mike G.

Tricker: Further Implementation Aspects of the Server Efficiency Rating Tool (SERT)

(Industry Track Best Paper Award)

The Server Efficiency Rating Tool (SERT) hasbeen developed by the Standard PerformanceEvaluation Corporation (SPEC) at the request of theUSEnvironmentalProtectionAgency(EPA).Almost3%ofallelectricityconsumedwithin theUSin2010wentto runningdatacenters.With this inmind, theEPAre-leasedVersion2.0oftheENERGYSTARforComputerServersprograminearly2013toincludethemandatoryuse of the SERT.Other governmentsworld-wide thatarealsoconcernedwithgrowingpowerconsumptionofserversanddatacentersare considering theadoptionoftheSERT.

Tilmann Rabl, Meikel Poess, Hans-Arno Jacob-sen, Patrick O’Neil, Elizabeth O’Neil: Variations of the Star Schema Benchmark to Test the Effects of

Data Skew on Query Performance (Industry Track Best Paper Nomination)

The Star Schema Benchmark (SSB), has beenwidely used to evaluate theperformanceof databasemanagement systems when executing star schemaqueries.SSB,basedonthewellknown industrystan-dardbenchmarkTPC-H,sharessomeofitsdrawbacks,most notably, its uniform data distributions. Today’ssystemsrelyheavilyonsophisticatedcost-basedqueryoptimizers to generate the most efficient query ex-ecutionplans.Abenchmark thatevaluatesoptimizer’scapabilitytogenerateoptimalexecutionplansunderallcircumstancesmustprovidetherichdatasetdetailsonwhich optimizers rely (uniform and non-uniform distri-butions,datasparsity,etc.).This isalso true forotherdatabasesystemparts,suchasindicesandoperators,and ultimately holds for an end-to-end benchmark aswell.SSB’sdatagenerator,basedonTPC-H’sdbgen,isnoteasytoadapttodifferentdatadistributionsasitsmetadataandactualdatagenerationimplementationsarenotseparated.Inthispaper,wemotivatetheneedfor a new revision of SSB that includes non-uniformdata distributions. We list what specific modificationsare required to SSB to implement non-uniform datasets and we demonstrate how to implement thesemodificationsintheParallelDataGeneratorFrameworktogenerateboththedataandquerysets.

14

CALL FOR PAPERS A Joint Meeting of WOSP/SIPEW sponsored by ACM SIGMETRICS and ACM SIGSOFT in cooperation with SPEC. The goal of the International Conference on Performance Engineering (ICPE) is to integrate theory and practice in the field of performance engineering by providing a forum for sharing ideas and experiences between industry and academia. ICPE is a joint meeting of the ACM Workshop on Software and Performance (WOSP) and the SPEC International Performance Evaluation Workshop (SIPEW). The conference brings together researchers and industry practitioners to share and present their experience, to discuss challenges, and to report state-of-the-art and in-progress research on performance engineering.

TOPICS OF INTEREST Topics of interest include, but are not limited to, the following:

Performance and software development processes Performance modeling and prediction Performance measurement and experimental analysis Benchmarking, configuration, sizing and capacity planning System management / optimization Performance in cloud, virtualized and multi-core systems Performance and Power Performance modeling and evaluation in other domains

For more details, please visit: http://icpe2014.ipd.kit.edu/call_for_papers/

RESEARCH PAPERS – PROGRAM COMMITTEE Jose Nelson Amaral, University of Alberta, Canada Alberto Avritzer, Siemens Corporate Research, USA Steffen Becker, University of Paderborn, Germany Simona Bernardi, Centro Universitario de la Defensa, Spain Steve Blackburn, Australian National University, Australia Andre Bondi, Siemens Corporate Research, USA Edson Borin, Universidade de Campinas, Brazil Thouraya Bouabana-Tebibel, ESI, Algeria Radu Calinescu, University of York, UK Lydia Chen, IBM Research Zurich Lab, Switzerland Vittorio Cortellessa, Universita’ di L’Aquila, Italy Tony Field, Imperial College London, UK Vincenzo Grassi, University of Roma "Tor Vergata", Italy Lars Grunske, TU Kaiserslautern, Germany Wilhelm Hasselbring, University of Kiel, Germany Matthias Hauswirth, University of Lugano, Switzerland Alexandru Iosup, TU Delft, Netherlands Carlos Juiz, University of the Balearic Islands, Spain Jorge Julvez, Universidad de Zaragoza, Spain David Kaeli, Northeastern University, USA Samuel Kounev, Karlsruhe Institute of Technology, Germany Anne Koziolek, University of Zurich, Switzerland Heiko Koziolek, ABB Corporate Research, Germany Diwakar Krishnamurthy, University of Calgary, Canada Doug Lea, SUNY Oswego, USA Yan Liu, Concordia University, Canada Stefano Marrone, Seconda Università di Napoli, Italy Daniel Menasce, George Mason University, USA Raffaela Mirandola, Politecnico di Milano, Italy David Pearce, Victoria University of Wellington, New Zealand Diego Perez-Palacin, Politecnico di Milano, Italy Dorina Petriu, Carleton University, Canada Jerry Rolia, HP Labs, Canada Giuseppe Serazzi, Politecnico di Milano, Italy Catia Trubiani, Universita’ di L’Aquila, Italy Petr Tuma, Charles University in Prague, Czech Republic Murray Woodside, Carleton University, Canada Peng Wu, IBM T. J. Watson Research Center, USA Thomas Würthinger, Oracle Labs, Austria Xiaoyun Zhu, VMware, USA

SUBMISSION GUIDELINES Authors are invited to submit original, unpublished papers that are not being considered in another forum. A variety of contribution styles for papers are solicited including: basic and applied research, industrial and experience reports, and work-in-progress/vision papers. Different acceptance criteria apply for each category; please visit: http://icpe2014.ipd.kit.edu/ for details. At least one author of each accepted paper is required to register at the full rate, attend the conference and present the paper. Presented papers will be published in the ICPE 2014 conference proceedings that will be published by ACM and included in the ACM Digital Library. For submission instructions, please visit: http://icpe2014.ipd.kit.edu/

IMPORTANT SUBMISSION DATES Research Papers 27 Sep 2013 Industrial / Experience Papers 25 Oct 2013 Tutorial Proposals 15 Nov 2013 Poster and Demo Papers 13 Jan 2014 Work-in-Progress and Vision Papers 13 Jan 2014

ORGANIZING COMMITTEE General Chairs

Klaus-Dieter Lange, Hewlett-Packard, USA John Murphy, University College Dublin, Ireland

Program Chairs Walter Binder, University of Lugano, Switzerland José Merseguer, Universidad de Zaragoza, Spain

Industrial Chair Raghunath Nambiar, Cisco Systems, USA

Tutorial Chair Alexandru Iosup, TU Delft, Netherlands

Demos and Posters Chairs Kirk W. Cameron, Virginia Tech, USA Anthony Ventresque, University College Dublin, Ireland

Publication Chair David Gregg, Trinity College Dublin, Ireland

Finance Chair Anja Bog, SAP, USA

Publicity Chairs Danilo Ansaloni, University of Lugano, Switzerland Bob Cramblitt, Cramblitt & Company, USA

Registration Chair Nicola Stokes, University College Dublin, Ireland

Awards Chairs Samuel Kounev, Karlsruhe Institute of Technology, Germany Meikel Poess, Oracle, USA

Local Organization Chair Patrick McDonagh, Dublin City University, Ireland

Web Chair Cathy Sandifer, SPEC, USA

INDUSTRY PAPERS – PROGRAM COMMITTEE Jeremy A. Arnold, IBM, USA Prasenjit Biswas, ARM, USA Jeffrey J. Blessing, Milwaukee School of Engineering, USA Hansfried Block, Fujitsu, Germany Andrew Bond, Red Hat, USA Wen Chen, University of North Dakota, USA Mathew Colgrove, The Portland Group, USA Malte Dreyer, Humboldt University, Germany Debo Dutta, Cisco Systems, USA Diwakar Krishnamurthy, University of Calgary, Canada Qais Noorshams, Karlsruhe Institute of Technology, Germany Kamesh Raghavendra, NetApp, USA Lisa Roderick, VMware, USA David Schmidt, Hewlett-Packard, USA Sanjay Sharma, Intel, USA Cloyce Spradling, Oracle, USA Mike Tricker, Microsoft, USA

http://icpe2014.ipd.kit.edu/