Pivot Tracing: Dynamic CausalMonitoring for Distributed Systems

Jonathan Mace Ryan Roelke Rodrigo FonsecaBrown University

AbstractMonitoring and troubleshooting distributed systems is noto-riously diõcult; potential problems are complex, varied, andunpredictable. _emonitoring and diagnosis tools commonlyused today – logs, counters, andmetrics – have two importantlimitations: what gets recorded is deûned a priori, and theinformation is recorded in a component- or machine-centricway,making it extremely hard to correlate events that crossthese boundaries._is paper presentsPivot Tracing, amonitor-ing framework for distributed systems that addresses both lim-itations by combining dynamic instrumentation with a novelrelational operator: the happened-before join. Pivot Tracinggives users, at runtime, the ability to deûne arbitrary metricsat one point of the system, while being able to select, ûlter,and group by events meaningful at other parts of the system,even when crossing component or machine boundaries. Wehave implemented a prototype of Pivot Tracing for Java-basedsystems and evaluate it on a heterogeneous Hadoop clustercomprising HDFS, HBase,MapReduce, and YARN. We showthat Pivot Tracing can eòectively identify a diverse range ofroot causes such as so�ware bugs,misconûguration, and limp-ing hardware.We show that Pivot Tracing is dynamic, extensi-ble, and enables cross-tier analysis between inter-operatingapplications, with low execution overhead.

1. IntroductionMonitoring and troubleshooting distributed systems is hard._e potential problems are myriad: hardware and so�warefailures,misconûgurations, hot spots, aggressive tenants, oreven simply unrealistic user expectations. Despite the com-plex, varied, and unpredictable nature of these problems,mostmonitoring and diagnosis tools commonly used today – logs,

Permission to make digital or hard copies of all or part of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for components of this workowned by others than the author(s) must be honored. Abstracting with credit ispermitted. To copy otherwise, or republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee. Request permissionsfrom permissions@acm.org.SOSP’15, October 4–7, 2015, Monterey, CA.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-3834-9/15/10. . . $15.00.http://dx.doi.org/10.1145/2815400.2815415

counters, andmetrics – have at least two fundamental limita-tions: what gets recorded is deûned a priori, at developmentor deployment time, and the information is captured in acomponent- or machine-centric way,making it extremely dif-ûcult to correlate events that cross these boundaries.

While there has been great progress in usingmachine learn-ing techniques [60, 74, 76, 95] and static analysis [97, 98] toimprove the quality of logs and their use in troubleshooting,they carry an inherent tradeoò between recall and overhead, aswhat gets loggedmust be deûned a priori. Similarly,withmon-itoring, performance countersmay be too coarse-grained [73];and if a user requests additional metrics, a cost-beneût tug ofwar with the developers can ensue [21].Dynamic instrumentation systems such as Fay [51] and

DTrace [38] enable the diagnosis of unanticipated perfor-mance problems in production systems [37] by providingthe ability to select, at runtime, which of a large number oftracepoints to activate. Both Fay and DTrace, however, arestill limited when it comes to correlating events that crossaddress-space or OS-instance boundaries. _is limitation isfundamental, as neither Fay nor DTrace can aòect themoni-tored system to propagate themonitoring context across theseboundaries.

In this paper we combine dynamic instrumentation withcausal tracing techniques [39, 52, 87] to fundamentally in-crease the power and applicability of either technique. Wepresent Pivot Tracing, amonitoring framework that gives op-erators and users, at runtime, the ability to obtain an arbitrarymetric at one point of the system,while selecting, ûltering, andgrouping by events meaningful at other parts of the system,even when crossing component or machine boundaries.

Like Fay, Pivot Tracing models themonitoring and tracingof a system as high-level queries over a dynamic dataset ofdistributed events. Pivot Tracing exposes an API for speci-fying such queries and eõciently evaluates them across thedistributed system, returning a streaming dataset of results.

_e key contribution of Pivot Tracing is the “happened-before join” operator, →⋈ , that enables queries to be contextu-alized by Lamport’s happened-before relation,� [65]. Using→⋈ , queries can group and ûlter events based on properties ofany events that causally precede them in an execution.

To track the happened-before relation between events,Pivot Tracing borrows from causal tracing techniques, andutilizes a genericmetadata propagation mechanism for pass-ing partial query execution state along the execution path ofeach request. _is enables inline evaluation of joins duringrequest execution, drastically mitigating query overhead andavoiding the scalability issues of global evaluation.

Pivot Tracing takes inspiration from data cubes in the on-line analytical processing domain [54], and derives its namefrom spreadsheets’ pivot tables and pivot charts [48], whichcan dynamically select values, functions, and grouping dimen-sions from an underlying dataset. Pivot Tracing is intendedfor use in both manual and automated diagnosis tasks, andto support both one-oò queries for interactive debugging andstanding queries for long-running system monitoring. It canserve as the foundation for the development of further diag-nosis tools. Pivot Tracing queries impose truly no overheadwhen disabled and utilize dynamic instrumentation for run-time installation.

We have implemented a prototype of Pivot Tracing for Java-based systems and evaluate it on a heterogeneous Hadoopcluster comprising HDFS, HBase, MapReduce, and YARN.In our evaluation we show that Pivot Tracing can eòectivelyidentify a diverse range of root causes such as so�ware bugs,misconûguration, and limping hardware. We show that PivotTracing is dynamic, extensible to new kinds of analysis, and en-ables cross-tier analysis between inter-operating applicationswith low execution overhead.

In summary, this paper has the following contributions:• Introduces the abstraction of the happened-before join (→⋈)for arbitrary event correlations;

• Presents an eõcient query optimization strategy and im-plementation for →⋈ at runtime, using dynamic instrumen-tation and cross-component causal tracing;

• Presents a prototype implementation of Pivot Tracing inJava, applied to multiple components of the Hadoop stack;

• Evaluates the utility and �exibility of Pivot Tracing todiagnose real problems.

2. Motivation

2.1 Pivot Tracing in Action

In this section wemotivate Pivot Tracing with amonitoringtask on the Hadoop stack. Our goal here is to demonstratesome of what Pivot Tracing can do, and we leave details of itsdesign, query language, and implementation to Sections 3, 4,and 5, respectively.

Suppose we want to apportion the disk bandwidth usageacross a cluster of eight machines simultaneously runningHBase, Hadoop MapReduce, and direct HDFS clients. Sec-tion 6 has an overview of these components, but for now itsuõces to know that HBase, a database application, accessesdata through HDFS, a distributed ûle system. MapReduce,in addition to accessing data through HDFS, also accesses

the disk directly to perform external sorts and to shuøe databetween tasks.

We run the following client applications:

FSread4m Random closed-loop 4MBHDFS readsFSread64m Random closed-loop 64MBHDFS readsHget 10kB row lookups in a largeHBase tableHscan 4MB table scans of a largeHBase tableMRsort10g MapReduce sort job on 10GB of input dataMRsort100g MapReduce sort job on 100GB of input data

By default, the systems expose a fewmetrics for disk con-sumption, such as disk read throughput aggregated by eachHDFS DataNode. To reproduce this metric with Pivot Trac-ing, we deûne a tracepoint for the DataNodeMetrics class, inHDFS, to intercept the incrBytesRead(int delta)method.Atracepoint is a location in the application source code whereinstrumentation can run, cf. §3. We then run the followingquery, in Pivot Tracing’s LINQ-like query language [70]:Q1: From incr In DataNodeMetrics.incrBytesRead

GroupBy incr.hostSelect incr.host, SUM(incr.delta)

_is query causes each machine to aggregate the delta argu-ment each time incrBytesRead is invoked, grouping by thehost name. Each machine reports its local aggregate everysecond, from which we produce the time series in Figure 1a.

_ings get more interesting, though, if we wish to mea-sure the HDFS usage of each of our client applications. HDFSonly has visibility of its direct clients, and thus an aggregateview of all HBase and all MapReduce clients. At best, ap-plications must estimate throughput client side. With PivotTracing, we deûne tracepoints for the client protocols ofHDFS (DataTransferProtocol),HBase (ClientService), andMapReduce (ApplicationClientProtocol), and use the nameof the client process as the group by key for the query. Fig-ure 1b shows the global HDFS read throughput of each clientapplication, produced by the following query:Q2: From incr In DataNodeMetrics.incrBytesRead

Join cl In First(ClientProtocols) On cl -> incrGroupBy cl.procNameSelect cl.procName, SUM(incr.delta)

_e -> symbol indicates a happened-before join. Pivot Trac-ing’s implementation will record the process name the ûrsttime the request passes through any client protocol methodand propagate it along the execution. _en, whenever the exe-cution reaches incrBytesRead on a DataNode, Pivot Tracingwill emit the bytes read or written, grouped by the recordedname. _is query exposes information about client diskthroughput that cannot currently be exposed by HDFS.Figure 1c demonstrates the ability for Pivot Tracing to

group metrics along arbitrary dimensions. It is generated bytwo queries similar to Q2 which instrument Java’s FileInput-Stream and FileOutputStream, still joining with the clientprocess name. We show the per-machine, per-applicationdisk read and write throughput of MRsort10g from thesame experiment. _is ûgure resembles a pivot table, wheresumming across rows yields per-machine totals, summing

Time [min]

0

50

100

150

200

0 5 10 15

HD

FS

Th

roug

hpu

t [M

B/s

] Host A Host EHost B Host FHost C Host GHost D Host H

(a) HDFS DataNode throughputper machine from instrumented

DataNodeMetrics.

Time [min]H

DF

S T

hro

ugh

put [

MB

/s]

0

50

100

150

200

0 5 10 15

MRSORT100G HSCANMRSORT10G HGET

FSREAD4MFSREAD64M

(b) HDFS DataNode throughputgrouped by high-level client

application.

DisknWritenThroughputDisknReadnThroughput

HostnAHostnBHostnCHostnDHostnEHostnFHostnGHostnH

Σcluster

HDFS Map Shuffle Reduce Σmachine

(c) Pivot table showing disk read and write sparklines for MRsort10g.Rows group by host machine; columns group by source process.Bottom row and right column show totals, and bottom-right cornershows grand total.

Figure 1: In this example, Pivot Tracing exposes a low-level HDFS metric grouped by client identiûers from other applications. Pivot Tracingcan expose arbitrary metrics at one point of the system, while being able to select, ûlter, and group by events meaningful at other parts of thesystem, even when crossing component or machine boundaries.

across columns yields per-system totals, and the bottom rightcorner shows the global totals. In this example, the clientapplication presents a further dimension along which wecould present statistics.

Query Q1 above is processed locally, while query Q2 re-quires the propagation of information from client processesto the data access points. Pivot Tracing’s query optimizer in-stalls dynamic instrumentationwhere needed, and determineswhen such propagationmust occur to process a query._e out-of-the boxmetrics provided byHDFS,HBase, andMapReducecannot provide analyses like those presented here. Simple cor-relations – such as determining which HDFS datanodes wereread from by a high-level client application – are not typicallypossible. Metrics are ad hoc between systems; HDFS sums IObytes, while HBase exposes operations per second. _ere isvery limited support for cross-tier analysis:MapReduce sim-ply counts global HDFS input and output bytes; HBase doesnot explicitly relateHDFS metrics to HBase operations.

2.2 Pivot Tracing OverviewFigure 2 presents a high-level overview of how Pivot Tracingenables queries such as Q2. We refer to the numbers in theûgure (e.g., À) in our description. Full support for PivotTracing in a system requires two basicmechanisms: dynamiccode injection and causal metadata propagation.While it ispossible to have some of the beneûts of Pivot Tracing withoutone of these (§8), for now we assume both are available.

Queries in Pivot Tracing refer to variables exposed by oneor more tracepoints – places in the systemwhere Pivot Tracingcan insert instrumentation.Tracepoint deûnitions are not partof the system code, but are rather instructions on where andhow to change the system to obtain the exported identiûers.Tracepoints in Pivot Tracing are similar to pointcuts fromaspect-oriented programming [62], and can refer to arbitraryinterface/method signature combinations. Tracepoints aredeûned by someone with knowledge of the system,maybe a

Tracepoint Tracepoint w/ advice

Execution path Baggage propagation

PT Agent

PT Agent

Pivot TracingFrontend

Instrumented System

Query{

Message busAdvice Tuples

1

2

3

45

4 6

7

8

Figure 2: Pivot Tracing overview (§2.2)

developer or expert operator, and deûne the vocabulary forqueries (À). _ey can be deûned and installed at any point intime, and can be shared and disseminated.

Pivot Tracing models system events as tuples of a stream-ing, distributed dataset. Users submit relational queries overthis dataset (Á), which get compiled to an intermediate repre-sentation called advice (Â).Advice uses a small instruction setto process queries, andmaps directly to code that local PivotTracing agents install dynamically at relevant tracepoints (Ã).Later, requests executing in the system invoke the installedadvice each time their execution reaches the tracepoint.

We distinguish Pivot Tracing from prior work by support-ing joins between events that occur within and across process,machine, and application boundaries. _e eõcient implemen-tation of the happened before join requires advice in one trace-point to send information along the execution path to advicein subsequent tracepoints._is is done through a new baggageabstraction, which uses causal metadata propagation (Ä). Inquery Q2, for example, cl.procName is packed in the ûrst in-vocation of the ClientProtocols tracepoint, to be accessedwhen processing the incrBytesRead tracepoint.

Advice in some tracepoints also emit tuples (Å), which getaggregated locally and then ûnally streamed to the client overamessage bus (Æ and Ç).

2.3 Monitoring and Troubleshooting Challenges

Pivot Tracing addresses two main challenges in monitoringand troubleshooting. First, when the choice of what to recordabout an execution is made a priori, there is an inherenttradeoò between recall and overhead. Second, to diagnosemany important problems one needs to correlate and integratedata that crosses component, system, andmachine boundaries.In §7 we expand on our discussion of existing work relativeto these challenges.One size does not ût all Problems in distributed systems arecomplex, varied, and unpredictable. By default, the informa-tion required to diagnose an issue may not be reported bythe system or contained in system logs. Current approachestie logging and statistics mechanisms into the developmentpath of products, where there is amismatch between the ex-pectations and incentives of the developer and the needs ofoperators and users. Panelists at SLAML [35] discussed theimportant need to “close the loop of operations back to de-velopers”. According to Yuan et al. [97], regarding diagnosingfailures, “(. . . ) existing log messages contain too little infor-mation. Despite their widespread use in failure diagnosis, itis still rare that log messages are systematically designed tosupport this function.”

_ismismatch can be observed in themany issues raised byusers onApache’s issue trackers: to requestnewmetrics [3, 4, 7–9, 17, 22]; to request changes to aggregation methods [10, 21,23]; and to request new breakdowns of existing metrics [2,5, 6, 11–16, 18–21, 25]. Many issues remain unresolved dueto developer pushback [12, 16, 17, 19, 20] or inertia [5, 7, 8,14, 18, 22, 23, 25]. Even simple cases ofmisconûguration arefrequently unreported by error logs [96].Eventually, applications may be updated to record more

information, but thishas eòects both in performance and infor-mation overload. Users must pay the performance overheadsof any systems that are enabled by default, regardless of theirutility. For example,HBase SchemaMetrics were introducedto aid developers, but all users ofHBase pay the 10% perfor-mance overhead they incur [21]. _e HBase user guide [1]carries the following warning for users wishing to integratewith Ganglia [69]: “By default, HBase emits a large numberofmetrics per region server. Gangliamay have diõculty pro-cessing all thesemetrics. Consider increasing the capacity ofthe Ganglia server or reducing the number ofmetrics emittedby HBase.”

_e glut of recorded information presents a “needle-in-a-haystack” problem to users [79]; while a system may exposeinformation relevant to a problem, e.g. in a log, extracting thisinformation requires system familiarity developed over a longperiod of time. For example,Mesos cluster state is exposedvia a single JSON endpoint and can becomemassive, even if aclient only wants information for a subset of the state [26].Dynamic instrumentation frameworks such as Fay [51],

DTrace [38], and SystemTap [78] address these limitations,by allowing almost arbitrary instrumentation to be installed

dynamically at runtime, and have proven extremely useful inthe diagnosis of complex and subtle system problems [37].Because of their side-eòect-free nature, however, they arelimited in the extent to which probes may share informationwith each other. In Fay, only probes in the same address spacecan share information, while inDTrace the scope is limited toa single operating system instance.Crossing Boundaries _is brings us to the second challengePivot Tracing addresses. In multi-tenant, multi-applicationstacks, the root cause and symptoms of an issuemay appear indiòerent processes,machines, and application tiers, andmaybe visible to diòerent users.A user of one applicationmay needto relate information from some other dependent applicationin order to diagnose problems that span multiple systems. Forexample, HBASE-4145 [13] outlines how MapReduce lacksthe ability to access HBase metrics on a per-task basis, andthat the framework only returns aggregates across all tasks.MESOS-1949 [25] outlines how the executors for a task do notpropagate failure information, so diagnosis can be diõcultif an executor fails. In discussion the developers note: “_eactually interesting / useful information is hidden in one offour or ûve diòerent places, potentially spread across as manydiòerent machines. _is leads to unpleasant and repetitivesearching through logs looking for a clue to what went wrong.(. . . ) _ere’s a lot of information that is hidden in log ûles andis very hard to correlate.”

Prior research has presentedmechanisms to observe or in-fer the relationship between events (§7) and studies of loggingpractices conclude that end-to-end tracing would be help-ful in navigating the logging issues they outline [75, 79]. Avariety of thesemechanisms have also materialized in produc-tion systems: for example, Google’s Dapper [87], Apache’sHTrace [58], Accumulo’s Cloudtrace [27], and Twitter’s Zip-kin [89]. _ese approaches can obtain richer informationabout particular executions than component-centric logs ormetrics alone, and have found uses in troubleshooting, de-bugging, performance analysis and anomaly detection, forexample. However, most of these systems record or recon-struct traces of execution for oøine analysis, and thus sharethe problems above with the ûrst challenge, concerning whatto record.

3. DesignWe now detail the fundamental concepts and mechanismsbehind Pivot Tracing. Pivot Tracing is a dynamicmonitoringand tracing framework for distributed systems.At a high level,it aims to enable �exible runtimemonitoring by correlatingmetrics and events from arbitrary points in the system. _echallenges outlined in §2 motivate the following high-leveldesign goals:

1. Dynamically conûgure and install monitoring at runtime2. Low system overhead to enable “always on” monitoring3. Capture causality between events from multiple processesand applications

Operation Description ExampleFrom Use input tuples from a set of tracepoints From e In RPCsUnion (⋃) Union events from multiple tracepoints From e In DataRPCs, ControlRPCsSelection (σ) Filter only tuples that match a predicate Where e.Size < 10Projection (Π) Restrict tuples to a subset of ûelds Select e.User, e.HostAggregation (A) Aggregate tuples Select SUM(e.Cost)GroupBy (G) Group tuples based on one or more ûelds GroupBy e.UserGroupBy Aggregation (GA) Aggregate tuples of a group Select e.User, SUM(e.Cost)Happened-Before Join (→⋈) Happened-before join tuples from another query Join d In Disk On d -> e

Happened-before join a subset of tuples Join d In MostRecent(Disk) On d -> e

Table 1: Operations supported by the Pivot Tracing query language

Tracepoints Tracepoints provide the system-level entry pointfor Pivot Tracing queries. A tracepoint typically correspondsto some event: a user submits a request; a low-level IO opera-tion completes; an external RPC is invoked, etc.A tracepoint identiûes one or more locations in the system

code where Pivot Tracing can install and run instrumentation.Tracepoints export named variables that can be accessed byinstrumentation. Figure 5 shows the speciûcation of one ofthe tracepoints in Q2 from §2. Besides declared exports, alltracepoints export a few variables by default: host, timestamp,process id, process name, and the tracepoint deûnition.

Whenever execution of the system reaches a tracepoint,any instrumentation conûgured for that tracepoint will beinvoked, generating a tuple with its exported variables. _eseare then accessible to any instrumentation code installed atthe tracepoint.Query Language Pivot Tracing enables users to express high-level queries about the variables exported by one or moretracepoints. We abstract tracepoint invocations as streamingdatasets of tuples;Pivot Tracing queries are therefore relationalqueries across the tuples of several such datasets.

To express queries, Pivot Tracing provides a parser forLINQ-like text queries such as those outlined in §2. Table 1outlines the query operations supported by Pivot Tracing.Pivot Tracing supports several typical operations includingprojection (Π), selection (σ), grouping (G), and aggregation(A). Pivot Tracing aggregators include Count, Sum, Max, Min,and Average. Pivot Tracing also deûnes the temporal ûltersMostRecent, MostRecentN, First, and FirstN, to take the1 or N most or least recent events. Finally, Pivot Tracingintroduces the happened-before join query operator (→⋈).Happened-before Joins A key contribution of Pivot Tracingis the happened-before join query operator. Happened-beforejoin enables the tuples from two Pivot Tracing queries to bejoined based on Lamport’s happened before relation,� [65].For events a and b occurring anywhere in the system, we saythat a happened before b and write a � b if the occurrenceof event a causally preceded the occurrence of event b andthey occurred as part of the execution of the same request.1

1 _is deûnition does not capture all possible causality, including whenevents in the processing of one request could in�uence another, but could beextended if necessary.

Execution Graph Query Query Results

a1

c1

a3

c2

b1

a2

b2

A a1 a2 a3

A→⋈B a1 b2 a2 b2

B→⋈Cb1 c1 b1 c2b2 c2

(A→⋈B)→⋈C a1 b2 c2 a2 b2 c2

Figure 3: An example execution that triggers tracepoints A, B and Cseveral times. We show several Pivot Tracing queries and the tuplesthat would result for each.

If a and b are not part of the same execution, then a /� b; ifthe occurrence of a did not lead to the occurrence of b, thena /� b (e.g., they occur in two parallel threads of executionthat do not communicate); and if a � b then b /� a.For any two queries Q1 and Q2, the happened-before join

Q1→⋈Q2 produces tuples t1 t2 for all t1 ∈ Q1 and t2 ∈ Q2 such

that t1 � t2. _at is, Q1 produced t1 before Q2 produced tuplet2 in the execution of the same request. Figure 3 shows anexample execution triggering tracepoints A, B, and C severaltimes, and outlines the tuples that would be produced for thisexecution by diòerent queries.

Query Q2 in §2 demonstrates the use of happened-beforejoin. In the query, tuples generated by the disk IO tracepointDataNodeMetrics.incrBytesRead are joined to the ûrst tuplegenerated by the ClientProtocols tracepoint.

Happened-before join substantially improves our ability toperform root cause analysis by giving us visibility into the rela-tionships between events in the system. _e happened-beforerelationship is fundamental to a number of prior approachesin root cause analysis (§7). Pivot Tracing is designed to eõ-ciently support happened-before joins, but does not optimizemore general joins such as equijoins (⋈).Advice Pivot Tracing queries compile to an intermediaterepresentation called advice. Advice speciûes the operationsto perform at each tracepoint used in a query, and eventuallymaterializes as monitoring code installed at those tracepoints(§5). Advice has several operations for manipulating tuplesthrough the tracepoint-exported variables, and evaluating →⋈on tuples produced by other advice at prior tracepoints in theexecution.

Operation DescriptionObserve Construct a tuple from variables exported by a

tracepointUnpack Retrieve one or more tuples from prior adviceFilter Evaluate a predicate on all tuplesPack Make tuples available for use by later adviceEmit Output a tuple for global aggregation

Table 2: Primitive operations supported by Pivot Tracing advice forgenerating and aggregating tuples as deûned in §3.

ClientProtocolsTracepoint

DataNodeMetricsTracepointRequestVExecution

ClientVProcesses HDFSVDataNode

A1OBSERVEV PACKV

UNPACKV OBSERVEV

A2 EMITV

Figure 4: Advice generated for Q2 from §2: A1 observes andpacks procName; A2 unpacks procName, observes delta, and emits(procName, SUM(delta))

Table 2 outlines the advice API. Observe creates a tuplefrom exported tracepoint variables. Unpack retrieves tuplesgenerated by other advice at other tracepoints prior in the ex-ecution. Unpacked tuples can be joined to the observed tuple,i.e., if to is observed and tu1 and tu2 are unpacked, then the re-sulting tuples are to tu1 and to tu2. Tuples created by this advicecan be discarded (Filter),made available to advice at othertracepoints later in the execution (Pack), or output for globalaggregation (Emit). Both Pack and Emit can group tuplesbased on matching ûelds, and perform simple aggregationssuch as SUM and COUNT. Pack also has the following specialcases: FIRST packs the ûrst tuple encountered and ignoressubsequent tuples; RECENT packs only themost recent tuple,overwriting existing tuples. FIRSTN and RECENTN generalizethis to N tuples. _e advice API is expressive but restrictedenough to provide some safety guarantees. In particular, ad-vice code has no jumps or recursion, and is guaranteed toterminate.

Query Q2 in §2 compiles to advice A1 and A2 for ClientProtocols and DataNodeMetrics respectively:A1:OBSERVE procName A2:OBSERVE delta

PACK-FIRST procName UNPACK procNameEMIT procName, SUM(delta)

First, A1 observes and packs a single valued tuple contain-ing the process name. _en, when execution reaches theDataNodeMetrics tracepoint, A2 unpacks the process name,observes the value of delta, then emits a joined tuple. Fig-ure 4 shows how this advice and the tracepoints interact withthe execution of requests in the system.

To compile a query to advice, we instantiate one advicespeciûcation for a From clause and add an Observe opera-tion for the tracepoint variables used in the query. For eachJoin clause, we add an Unpack operation for the variablesthat originate from the joined query. We recursively generate

classjDataNodeMetricsj{jvoidjincrBytesReadCintjdeltaKj{ PivotTracing.Advise("A1",delta);jj...j}}

classjGeneratedAdviceImplj{jvoidjAdviseCObject...jobservedKj{jj//jGeneratedjcodejforjadvicej}}

Weave

classjDataNodeMetricsj{jvoidjincrBytesReadCintjdeltaKj{

jjj...j}}

OBSERVEjdeltaUNPACKjprocNameEMITjprocName,jSUMCdeltaK

Advice A1

Class: DataNodeMetricsMethod: incrBytesReadExports: "delta" = delta

Tracepoint

Figure 5: Advice for Q2 is woven at the DataNodeMetrics tracepoint.Variables exported by the tracepoint are passed when the advice isinvoked.

advice for the joined query, and append a Pack operationat the end of its advice for the variables that we unpacked.Where directly translates to a Filter operation. We add anEmit operation for the output variables of the query, restrictedaccording to any Select clause. Aggregate, GroupBy, andGroupByAggregate are all handled by Emit and Pack. §4outlines several query rewriting optimizations for implement-ing →⋈ .

Pivot Tracing weaves advice into tracepoints by: 1) loadingcode that implements the advice operations; 2) conûguring thetracepoint to execute that code and pass its exported variables;3) activating the necessary tracepoint at all locations in thesystem. Figure 5 outlines this process of weaving advice forQ2.

4. Pivot Tracing OptimizationsIn this section we outline several optimizations that PivotTracing performs in order to support eõcient evaluation ofhappened-before joins.Unoptimized joins _enaïve evaluation strategy forhappened-before join is that of an equijoin (⋈ ) or θ-join (⋈ θ [80]),requiring tuples to be aggregated globally across the clusterprior to evaluating the join. Temporal joins as implementedbyMagpie [32], for example, are expensive because they im-plement this evaluation strategy (§7). Figure 6a illustrates thisapproach for happened-before join.Baggage Pivot Tracing enables inexpensive happened-beforejoins by providing the baggage abstraction. Baggage is a per-request container for tuples that is propagated alongside arequest as it traverses thread, application andmachine bound-aries. Pack and Unpack store and retrieve tuples from thecurrent request’s baggage. Tuples follow the request’s execu-tion path and therefore explicitly capture the happened-beforerelationship.Baggage is a generalization of end-to-endmetadata propa-

gation techniques outlined in prior work such as X-Trace [52]and Dapper [87]. Using baggage, Pivot Tracing eõciently eval-uates happened-before joins in situ during the execution ofa request. Figure 6b shows the optimized query evaluationstrategy to evaluate joins in-place during request execution.

∪( )∪( )

inputevents

per-machine, per-query merge

∪( )∪( ) ∪( )∪( )

cross-cluster, per-query merge,

∪( ) ∪( )→⋈

executio

n

(a) Unoptimized query with →⋈ evaluated centrally for thewhole cluster.

∪( )→⋈

∪∪

∪

inputevents

per-machine merge

cross-cluster merge

per-e

xecutio

nm

erg

e, jo

in

→⋈ →⋈ →⋈

∪( )→⋈ ∪( )→⋈

∪( )→⋈(b) Optimized query with inline evaluation of →⋈ ( ).

Figure 6: Optimization of →⋈ . _e optimized query o�en produces substantially less tuple traõc than the unoptimized form.

Tuple Aggregation Onemetric to assess the cost of a PivotTracing query is the number of tuples emitted for global ag-gregation. To reduce this cost, Pivot Tracing performs in-termediate aggregation for queries containing Aggregate orGroupByAggregate. Pivot Tracing aggregates the emitted tu-ples within each process and reports results globally at a regu-lar interval, e.g., once per second. Process-level aggregationsubstantially reduces traõc for emitted tuples; Q2 from §2 isreduced from approximately 600 tuples per second to 6 tuplesper second from each DataNode.QueryOptimizations A second costmetric for Pivot Tracingqueries is the number of tuples packed during a request’s exe-cution. Pivot Tracing rewrites queries tominimize the numberof tuples packed. Pivot Tracing pushes projection, selection,and aggregation terms as close as possible to source trace-points. In Fay [51] the authors outlined query optimizationsfor merging streams of tuples, enabled because projection, se-lection, and aggregation are distributive. _ese optimizationsalso apply to Pivot Tracing and reduce the number of tuplesemitted for global aggregation. To reduce the number of tu-ples transported in the baggage, Pivot Tracing adds furtheroptimizations for happened-before joins, outlined in Table 3.Propagation Overhead Pivot Tracing does not inherentlybound the number of packed tuples and potentially accumu-lates a new tuple for every tracepoint invocation. However,we liken this to database queries that inherently risk a fulltable scan – our optimizations mean that in practice, this is anunlikely event. Several of Pivot Tracing’s aggregation opera-tors explicitly restrict the number of propagated tuples and inour experience, queries only end up propagating aggregations,most-recent, or ûrst tuples.

In cases where too many tuples are packed in the baggage,Pivot Tracing could revert to an alternative query plan, whereall tuples are emitted instead of packed, and the baggage sizeis kept constant by storing only enough information to re-construct the causality, a la X-Trace [52], Stardust [88], orDapper [87]. To estimate the overhead of queries, Pivot Trac-

Query Optimized Query

Πp ,q(P →⋈ Q) Πp(P) →⋈ Πq(Q)σp(P →⋈ Q) σp(P) →⋈ Qσq(P →⋈ Q) P →⋈ σq(Q)Ap(P →⋈ Q) Combinep(Ap(P) →⋈ Q)GAp(P →⋈ Q) GpCombinep(GAp(P) →⋈ Q)GAq(P →⋈ Q) GqCombinep(P →⋈ GAq(Q))GpAq(P →⋈ Q) GpCombineq(Πp(P) →⋈ Aq(Q))GqAp(P →⋈ Q) GqCombinep(Ap(P) →⋈ Πq(Q))

Table 3: Query rewrite rules to join queries P and Q. We pushoperators as close as possible to source tuples; this reduces thenumber of tuples that must be propagated in the baggage from Pto Q. Combine refers to an aggregator’s combiner function (e.g., forCount, the combiner is Sum)

ing can execute amodiûed version of the query to count tuplesrather than aggregate them explicitly. _is would provide liveanalysis similar to “explain” queries in the database domain.

5. ImplementationWe have implemented a Pivot Tracing prototype in Java andapplied Pivot Tracing to several open-source systems from theHadoop ecosystem. Section §6 outlines our instrumentationof these systems. In this section, we describe the implementa-tion of our prototype.Agent A Pivot Tracing agent thread runs in every PivotTracing-enabled process and awaits instruction via centralpub/sub server to weave advice to tracepoints. Tuples emittedby advice are accumulated by the local Pivot Tracing agent,which performs partial aggregation of tuples according totheir source query. Agents publish partial query results at aconûgurable interval – by default, one second.Dynamic Instrumentation Our prototype weaves advice atruntime, providing dynamic instrumentation similar to that ofDTrace [38] and Fay [51]. Java version 1.5 onwards supports dy-namicmethod body rewriting via the java.lang.instrument

Method Descriptionpack(q, t...) Pack tuples into the baggage for a queryunpack(q) Retrieve all tuples for a queryserialize() Serialize the baggage to bytesdeserialize(b) Set the baggage by deserializing from bytessplit() Split the baggage for a branching executionjoin(b1, b2) Merge baggage from two joining executions

Table 4: Baggage API for Pivot Tracing Java implementation. Packoperations store tuples in the baggage. API methods are static andonly allow interaction with the current execution’s baggage.

package. _e Pivot Tracing agent programmatically rewritesand reloads class bytecode fromwithin the process using Javas-sist [44].

We can deûne new tracepoints at runtime and dynami-cally weave and unweave advice. To weave advice, we rewritemethod bodies to add advice invocations at the locations de-ûned by the tracepoint (cf. Fig. 5). Our prototype supports tra-cepoints at the entry, exit, or exceptional return of anymethod.Tracepoints can also be inserted at speciûc line numbers.

To deûne a tracepoint, users specify a class name,methodname, method signature and weave location. Pivot Tracingalso supports patternmatching, for example, all methods of aninterface on a class._is feature ismodeled a�er pointcuts fromAspectJ [61]. Pivot Tracing supports instrumenting privilegedclasses (e.g., FileInputStream in §2) by providing an optionalagent that can be placed on Java’s boot classpath.

Pivot Tracing onlymakes system modiûcations when ad-vice is woven into a tracepoint, so inactive tracepoints incurno overhead. Executions that do not trigger the tracepoint areunaòected by Pivot Tracing. Pivot Tracing has a zero-probeeòect:methods are unmodiûed by default, so tracepoints im-pose truly zero overhead until advice is woven into them.Baggage We provide an implementation of Baggage thatstores per-request instances in thread-local variables. At thebeginning of a request, we instantiate empty baggage in thethread-local variable; at the end of the request, we clear thebaggage from the thread-local variable. _e baggage API(Table 4) can get or set tuples for a query and at any pointin time baggage can be retrieved for propagation to anotherthread or serialization onto the network. To support multiplequeries simultaneously, queries are assigned unique IDs andtuples are packed and unpacked based on this ID.Baggage is lazily serialized and deserialized using protocol

buòers [53]. _is minimizes the overhead of propagating bag-gage through applications that do not actively participate ina query, since baggage is only deserialized when an applica-tion attempts to pack or unpack tuples. Serialization costs areonly incurred for modiûed baggage at network or applicationboundaries.

Pivot Tracing relies on developers to implement Baggagepropagation when a request crosses thread, process, or asyn-chronous execution boundaries. In our experience (§6) this

entails adding a baggage ûeld to existing application-levelrequest contexts and RPC headers.Branches andVersioning In order to preserve the happened-before relation correctly within a request, Pivot Tracing musthandle executions that branch and rejoin. Tuples packed byone branch cannot be visible to any other branch until thebranches rejoin.

To handle this, we implement a versioning scheme forbaggage using interval tree clocks [29]. Internally, baggageactually maintains one or more versioned instances each witha globally unique interval tree ID. Only one of the versionedinstances is considered ‘active’ for any branch of the execution.

Whenever an execution branches, the interval tree ID ofthe active instance is divided into two globally unique, non-overlapping IDs, and each branch is assigned one of these newIDs. Each side of the branch receives a copy of the baggagethen creates a new active instance using its half of the dividedID.

When a tuple is packed, it is placed into the active instancefor its branch. To unpack from baggage that has multiple in-stances, tuples are unpacked from each instance then com-bined according to query logic.

When two branches of an execution rejoin, a new activebaggage instance is constructed by merging the contents ofthe active instances on each side of the branch. _e ID of thenew active baggage is assigned by joining the IDs of each sideof the branch. _e inactive instances from each branch arecopied, and duplicates are discarded.Materializing Advice Tracepoints with woven advice in-voke the PivotTracing.Advise method (cf. Fig. 5), passingtracepoint variables as arguments. Tuples are implementedas Object arrays and there is a straightforward translationfrom advice to implementation: Observe constructs a tuplefrom the advice arguments; Unpack and Pack interact withthe Baggage API; Filter discards tuples that do not matcha predicate; and Emit forwards tuples to the process-localaggregator.

6. EvaluationIn this section we evaluate Pivot Tracing in the context of theHadoop stack.We have instrumented four open-source sys-tems – HDFS, HBase, Hadoop MapReduce, and YARN – thatare widely used in production today. We present several casestudies where we used Pivot Tracing to successfully diagnoseroot causes, including real-world issueswe encountered in ourcluster and experiments presented in prior work [66, 93]. Ourevaluation shows that Pivot Tracing addresses the challengesin §2 when applied to these stack components. In particular,we show that Pivot Tracing:• is dynamic and extensible to new kinds of analysis (§6.2)• is scalable and has low developer and execution overhead(§6.3)

• enables cross-tier analysis between any inter-operatingapplications (§2, §6.2)

HDFS

HBase

YARN

MapReduce

Clients

RegionServer

ClientApplication

ClientApplication

ResourceManager

NodeManager

DataNode

NameNode

Master TaskAppMaster

Figure 7: Interactions between systems. Each system comprises sev-eral processes on potentially many machines. Typical deploymentso�en co-locate processes from several applications, e.g. DataNode,NodeManager, Task and RegionServer processes.

• captures event causality to successfully diagnose rootcauses (§6.1, §6.2)

• enables insightful analysis with even a very small numberof tracepoints (§6.1)

Hadoop Overview We ûrst give a high-level overview ofHadoop, before describing the necessary modiûcations toenable Pivot Tracing. Figure 7 shows the relevant componentsof theHadoop stack.

HDFS [86] is a distributed ûle system that consists ofseveral DataNodes that store replicated ûle blocks and aNameNode that manages the ûlesystem metadata.

HBase [56] is a non-relational database modeled a�erGoogle’s Bigtable [41] that runs on top ofHDFS and comprisesaMaster and several RegionServer processes.

Hadoop MapReduce is an implementation of theMapRe-duce programming model [49] for large-scale data processing,that uses YARN containers to run map and reduce tasks. Eachjob runs an ApplicationMaster and several MapTask and Re-duceTask containers.

YARN [91] is a container manager to run user-providedprocesses across the cluster. NodeManager processes run oneachmachine tomanage local containers, and a centralizedRe-sourceManager manages the overall cluster state and requestsfrom users.

Hadoop Instrumentation In order to support Pivot Tracingin these systems, we made one-time modiûcations to prop-agate baggage along the execution path of requests. As de-scribed in §5 our prototype uses a thread-local variable tostore baggage during execution, so the only required systemmodiûcations are to set and unset baggage at execution bound-aries. To propagate baggage across remote procedure calls, wemanually extended the protocol deûnitions of the systems.To propagate baggage across execution boundaries withinindividual processes we implemented AspectJ [61] instrumen-tation to automatically modify common interfaces (Thread,Runnable, Callable, and Queue). Each system only requiredbetween 50 and 200 lines ofmanual codemodiûcation. Oncemodiûed, these systems could support arbitrary Pivot Tracingqueries without further modiûcation.

Our queries used tracepoints from both client and serverRPC protocol implementations of the HDFS DataNode

DataTransferProtocol and NameNode ClientProtocol. Wealso used tracepoints for piggybacking oò existing met-ric collection mechanisms in each instrumented system,such as DataNodeMetrics and RPCMetrics in HDFS andMetricsRegionServer in HBase.

6.1 Case Study:HDFS Replica Selection BugIn this section we describe our discovery of a replica selectionbug in HDFS that resulted in uneven distribution of loadto replicas. A�er identifying the bug, we found that it hadbeen recently reported and subsequently ûxed in an upcomingHDFS version [24].

HDFS provides ûle redundancy by decomposing ûles intoblocks and replicating each block onto several machines (typ-ically 3). A client can read any replica of a block and doesso by ûrst contacting the NameNode to ûnd replica hosts(GetBlockLocations), then selecting the closest replica as fol-lows: 1) read a local replica; 2) read a rack-local replica; 3)select a replica at random. We discovered a bug whereby rack-local replica selection always follows a global static orderingdue to two con�icting behaviors: the HDFS client does notrandomly select between replicas; and the HDFS NameNodedoes not randomize rack-local replicas returned to the client._e bug results in heavy load on the some hosts and near zeroload on others.

In this scenario we ran 96 stress test clients on an HDFScluster of 8 DataNodes and 1 NameNode. Each machine hasidentical hardware speciûcations; 8 cores, 16GB RAM, and a1Gbit network interface. On each host, we ran a process calledStressTest that used an HDFS client to perform closed-looprandom 8kB reads from a dataset of 10,000 128MB ûles with areplication factor of 3.

Our investigation of the bug began when we noticed thatthe stress test clients on hosts A and D had consistently lowerrequest throughput than clients on other hosts, shown inFigure 8a, despite identical machine speciûcations and setup.We ûrst checkedmachine level resource utilization on eachhost, which indicated substantial variation in the networkthroughput (Figure 8b). We began our diagnosis with PivotTracing by ûrst checking to seewhether an imbalance inHDFSload was causing the variation in network throughput. _efollowing query installs advice at a DataNode tracepoint thatis invoked by each incoming RPC:Q3: From dnop In DN.DataTransferProtocol

GroupBy dnop.hostSelect dnop.host, COUNT

Figure 8c plots the results of this query, showing the HDFSrequest throughput on each DataNode. It shows that DataN-odes on hosts A and D in particular have substantially higherrequest throughput than others – host A has on average 150ops/sec, while host H has only 25 ops/sec. _is behavior wasunexpected given that our stress test clients are supposedlyreading ûles uniformly at random. Our next query installsadvice in the stress test clients and on theHDFS NameNode,to correlate each read request with the client that issued it:

0

20

40

60

80

0 1 2 3 4 5

Clie

nt8T

hrou

ghpu

t8[re

q/s]

Time8[min]

Client8AClient8BClient8CClient8D

Client8EClient8FClient8GClient8H

(a) Clients on Hosts A and D experiencereduced workload throughput.

Mill

ions

0

0.5

1

1.5

2

2.5

0 1 2 3 4 5

Host4A Host4BHost4C Host4D

Net

wor

k4T

rans

fer4

[MB

/s]

Time4[min]

(b) Network transfer is skewed acrossmachines.

Host G Host HHost E Host F

Time [min]

Dat

aNod

e T

hrou

ghpu

t [op

s/s] 200

150

100

50

00 1 2 3 4 5

(c) HDFS DataNode throughput is skewedacross machines.

Host AHost BHost CHost D

Host GHost H

Host EHost F

Client AClient BClient CClient DClient EClient FClient GClient H

Hos

t A

Hos

t B

Hos

t CH

ost D

Hos

t GH

ost H

Hos

t E

Hos

t F

Hos

t A

Hos

t B

Hos

t CH

ost D

Hos

t GH

ost H

Hos

t E

Hos

t F

Hos

t A

Hos

t B

Hos

t CH

ost D

Hos

t GH

ost H

Hos

t E

Hos

t F

(d) ObservedHDFS ûle readdistribution (row) per client

(col).

(e) Frequency each client (row)sees each DataNode (col) as a

replica location.

(f) Frequency each client (row)subsequently selects each

DataNode (col).

(g) Observed frequency ofchoosing one replica host (row)

over another (col)

Figure 8: Pivot Tracing query results leading to our discovery of HDFS-6268 [24]. Faulty replica selection logic led clients to prioritize thereplicas hosted by particular DataNodes (§6.1).

Q4: From getloc In NN.GetBlockLocationsJoin st In StressTest.DoNextOp On st -> getlocGroupBy st.host, getloc.srcSelect st.host, getloc.src, COUNT

_is query counts the number of times each client reads eachûle. In Figure 8d we plot the distribution of counts over a 5minute period for clients from each host. _e distributions allût a normal distribution and indicate that all of the clients arereading ûles uniformly at random. _e distribution of readsfrom clients on A and D are skewed le�, consistent with theiroverall lower read throughput.

Having conûrmed the expected behavior of our stress testclients, we next checked to see whether the skewed datanodethroughput was simply a result of skewed block placementacross datanodes:Q5: From getloc In NN.GetBlockLocations

Join st In StressTest.DoNextOp On st -> getlocGroupBy st.host, getloc.replicasSelect st.host, getloc.replicas, COUNT

_is querymeasures the frequency that eachDataNode is host-ing a replica for ûles being read. Figure 8e shows that, for eachclient, replicas are near-uniformly distributed across DataN-odes in the cluster. _ese results indicate that clients have anequal opportunity to read replicas from each DataNode, yet,our measurements in 8c clearly show that they do not. To gainmore insight into this inconsistency, our next query relatesthe results from 8e and 8c:

Q6: From DNop In DN.DataTransferProtocolJoin st In StressTest.DoNextOp On st -> DNopGroupBy st.host, DNop.hostSelect st.host, DNop.host, COUNT

_is query measures the frequency that each client selectseach DataNode for reading a replica. We plot the results inFigure 8f and see that the clients are clearly favoring particularDataNodes._e strongdiagonal is consistentwithHDFS clientpreference for locally-hosted replicas (39% of the time in thiscase). However, the expected behaviorwhen there isnot a localreplica is to select a rack-local replica uniformly at random;clearly these results suggest that this was not happening.

Our ûnal diagnosis steps were as follows. First, we checkedto seewhich replicawas selected byHDFS clients from the loca-tions returned by theNameNode.We found that clients alwaysselected the ûrst location returned by theNameNode. Second,wemeasured the conditional probabilities thatDataNodes pre-cede each other in the locations returned by the NameNode.We issued the following query for the latter:Q7: From DNop In DN.DataTransferProtocol

Join getloc In NN.GetBlockLocationsOn getloc -> DNop

Join st In StressTest.DoNextOp On st -> getlocWhere st.host != DNop.hostGroupBy DNop.host, getloc.replicasSelect DNop.host, getloc.replicas, COUNT

_is query correlates the DataNode that is selected with theother DataNodes also hosting a replica. We remove the in-

Time [min]Req

uest

Lat

ency

[sec

]

060

120180240300

0 5 10 15

(a) HBase Request Latencies

0 60Latencyg[sec]

RSgQueueRSgProcess DNgTransfer

DNgBlockedDNgGC

Slow

Average

(b) Latency Decomposition

Host A Host EHost B Host FHost C Host GHost D Host H

Time [min]

Net

wor

k T

x [M

bit/s

] 800

600

400

200

00 5 10 15

(c) Per-Machine Network _roughput

Figure 9: (a) Observed request latencies for a closed-loop HBase workload experiencing occasional end-to-end latency spikes; (b) Average timein each component on average (top), and for slow requests (bottom, end-to-end latency > 30s); (c) Per-machine network throughput – a faultynetwork cable has downgradedHost B’s link speed to 100Mbit, aòecting entire cluster throughput.

terference from locally-hosted replicas by ûltering only therequests that do a non-local read. Figure 8g shows that hostA was always selected when it hosted a replica; host D wasalways selected except if host A was also a replica, and so on.At this point in our analysis, we concluded that this behav-

ior was quite likely to be a bug in HDFS. HDFS clients didnot randomly select between replicas, and the HDFS NameN-ode did not randomize the rack-local replicas. We checkedApache’s issue tracker and found that the bug had been recentlyreported and ûxed in an upcoming version ofHDFS [24].

6.2 Diagnosing End-to-End Latency

Pivot Tracing can express queries about the time spent by arequest across the components it traverses using the built-intime variable exported by each tracepoint. Advice can packthe timestamp of any event then unpack it at a subsequentevent, enabling comparison of timestamps between events._e following example querymeasures the latency betweenreceiving a request and sending a response:Q8: From response In SendResponse

Join request In MostRecent(ReceiveRequest)On request -> response

Select response.time − request.time

When evaluating this query, MostRecent ensureswe select onlythe most recent preceding ReceiveRequest event wheneverSendResponse occurs. We can use latency measurement inmore complicated queries. _e following example query mea-sures the average request latency experienced byHadoop jobs:Q9: From job In JobComplete

Join latencyMeasurement In Q8On latencyMeasurement -> end

Select job.id, AVERAGE(latencyMeasurement)

A query canmeasure latency in several components and propa-gatemeasurements in the baggage, reminiscent of transactiontracking in Timecard [81] and transactional proûling in Who-dunit [39]. For example, during the development of Pivot Trac-ing we encountered an instance of network limplock [50, 66],whereby a faulty network cable caused a network link down-grade from 1Gbit to 100Mbit. OneHBase workload in partic-ular would experience latency spikes in the requests hittingthis bottleneck link (Figure 9a). To diagnose the issue, wedecomposed requests into their per-component latency and

compared anomalous requests (> 30s end-to-end latency) tothe average case (Figure 9b). _is enabled us to identify thebottleneck source as time spent blocked on the network inthe HDFS DataNode on Host B. We measured the latencyand throughput experienced by all workloads at this compo-nent and were able to identify the uncharacteristically lowthroughput ofHost B’s network link (Figure 9c).

We have also replicated results in end-to-end latency diag-nosis in the following other cases: to diagnose rogue garbagecollection in HBase RegionServers as described in [93]; andto diagnose an overloadedHDFS NameNode due to exclusivewrite locking as described in [67].

6.3 Overheads of Pivot Tracing

Baggage By default, Pivot Tracing propagates an emptybaggagewith a serialized size of 0 bytes. In theworst case PivotTracing may need to pack an unbounded number of tuples inthe baggage, one for each tracepoint invoked. However, theoptimizations in §4 reduce the number of propagated tuples to1 for Aggregate, 1 for Recent, n for GroupBy with n groups,and N for RecentN. Of the queries in this paper, Q7 propagatesthe largest baggage containing the stress test hostname and thelocation of all 3 ûle replicas (4 tuples, ≈137 bytes per request).

_e size of serialized baggage is approximately linear in thenumber of packed tuples. _e overhead to pack and unpacktuples from the baggage varies with the size of the baggage– Figure 10 gives micro-benchmark results measuring theoverhead of baggage API calls.Application-level Overhead To estimate the impact of PivotTracing on application-level throughput and latency, we ranbenchmarks from HiBench [59], YCSB [47], andHDFS DF-SIO and NNBench benchmarks. Many of these benchmarksbottleneck on network or disk and we noticed no signiûcantperformance change with Pivot Tracing enabled.

To measure the eòect of Pivot Tracing on CPU bound re-quests, we stress testedHDFS using requests derived from theHDFS NNBench benchmark: Read8k reads 8kB from a ûle;Open opens a ûle for reading;Create creates a ûle forwriting;Rename renames an existing ûle. Read8kB is a DataNodeoperation and the others are NameNode operations. We com-pared the end-to-end latency of requests in unmodiûedHDFS

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2 4 8 1

6 3

2 6

4

128

256

Late

ncy (

µs)

Number of tuples in baggage

(a) Pack 1 tuple

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 4 8 1

6 3

2 6

4

128

256

Late

ncy (

µs)

Number of tuples in baggage

(b) Unpack all tuples

0

2

4

6

8

10

12

14

1 2 4 8 1

6 3

2 6

4

128

256

Late

ncy (

µs)

Number of tuples in baggage

(c) Serialize baggage

0 2 4 6 8

10 12 14 16 18 20 22

1 2 4 8 1

6 3

2 6

4

128

256

Late

ncy (

µs)

Number of tuples in baggage

(d) Deserialize baggage

Figure 10: Latency micro-benchmark results for packing, unpacking,serializing, and deserializing randomly-generated 8-byte tuples.

to HDFS modiûed in the following ways: 1) with Pivot Trac-ing enabled; 2) propagating baggage containing one tuple butno advice installed; 3) propagating baggage containing 60 tu-ples (≈1kB) but no advice installed; 4) with the advice fromqueries in §6.1 installed; 5) with the advice from queries in§6.2 installed.

Table 5 shows that the application-level overheadwithPivotTracing enabled is at most 0.3%. _is overhead includes thecosts of baggage propagation within HDFS, baggage serializa-tion in RPC calls, and to run Java in debugging mode. _emost noticeable overheads are incurred when propagating60 tuples in the baggage, incurring 15.9% overhead for Open.Since this is a short CPU-bound request (involving a singleread-only lookup), 16% is within reasonable expectations. Re-name does not trigger any advice for the queries from §6.1,but does trigger advice for the queries from §6.2. Overheadsof 0.3% and 5.5% respectively re�ect this diòerence.Dynamic Instrumentation JVM HotSwap requires Java’s de-bugging mode to be enabled, which causes some compileroptimizations to be disabled. For practical purposes, however,HotSpot JVM’s full-speed debugging is suõciently optimizedthat it is possible to run with debugging support always en-abled [57]. Our HDFS throughput experiments above mea-sured only a small overhead between debugging enabled anddisabled. Reloading a class with woven advice has a one-timecost of approximately 100ms, depending on the size of theclass being reloaded.

7. RelatedWorkIn §2 we described the challenges with troubleshooting toolsthat Pivot Tracing addresses, and complement the discussionon related work here.

Pivot Tracing’s dynamic instrumentation is modeled af-ter aspect-oriented programming [62], and extends prior dy-namic instrumentation systems [38, 51, 78] with causal infor-mation that crosses process and system boundaries.

Read8k Open Create RenameUnmodiûed 0% 0% 0% 0%

PivotTracing Enabled 0.3% 0.3% <0.1% 0.2%Baggage – 1 Tuple 0.8% 0.4% 0.6% 0.8%

Baggage – 60 Tuples 0.82% 15.9% 8.6% 4.1%Queries – §6.1 1.5% 4.0% 6.0% 0.3%Queries – §6.2 1.9% 14.3% 8.2% 5.5%

Table 5: Latency overheads for HDFS stress test with Pivot Tracingenabled, baggage propagation enabled, and full queries enabled, asdescribed in §6.3

Temporal Joins Like Fay [51], Pivot Tracingmodels the eventsof a distributed system as a stream of dynamically generatedtuples belonging to a distributed database. Pivot Tracing’shappened-before join is an example of a θ-join [80] wherethe condition is happened-before. Pivot Tracing’s happened-before join is also an example of a special case of path queriesin graph databases [94]. Diòerently from oøine queries ina pre-stored graph, Pivot Tracing eõciently evaluates →⋈ atruntime.

Pivot Tracing captures causality between events by general-izingmetadatapropagation techniquesproposed in priorworksuch as X-Trace [52]. While Baggage enables Pivot Tracing toeõciently evaluate happened-before join, it is not necessary;Magpie [33] demonstrated that under certain circumstances,causality between events can be inferred a�er the fact. Speciû-cally, if ‘start’ and ‘end’ events exist to demarcate a request’sexecution on a thread, then we can infer causality betweenthe intermediary events. Similarly we can also infer causal-ity across boundaries, provided we can correlate ‘send’ and‘receive’ events on both sides of the boundary (e.g., with aunique identiûer present in both events). Under these circum-stances, Magpie evaluates queries that explicitly encode allcausal boundaries and use temporal joins to extract the inter-mediary events.By extension, for any Pivot Tracing query with happened-

before join there is an equivalent query that explicitly encodescausal boundaries and uses only temporal joins. However,such a query could not take advantage of the optimizationsoutlined in this paper, and necessitates global evaluation.BeyondMetrics and Logs A variety of tools have been pro-posed in the research literature to complement or extend appli-cation logs and performance counters. _ese include the useofmachine learning [60, 74, 76, 95] and static analysis [98] toextract better information from logs; automatic enrichmentof existing log statements to ease troubleshooting [97]; end-to-end tracing systems to capture the happened-before rela-tionship between events [52, 87]; state-monitoring systems totrack system-level resources and indicate the health of a clus-ter [69]; and aggregation systems to collect and summarizeapplication-level monitoring data [64, 90].Wang et al. providea comprehensive overview of datacenter troubleshooting toolsin [92]. _ese tools suòer from the challenges of pre-deûnedinformation outlined in §2.

Troubleshooting and Root-Cause Diagnosis Several oøinetechniques have been proposed to infer execution modelsfrom logs [34, 45, 68, 98] and diagnose performance prob-lems [31, 63, 74, 85]. End-to-end tracing frameworks existboth in academia [33, 39, 52, 83, 88] and in industry [30, 46,58, 87, 89] and have been used for a variety of purposes, in-cluding diagnosing anomalous requests whose structure ortiming deviate from the norm [28, 33, 42, 43, 77, 85]; diag-nosing steady-state problems that manifest across many re-quests [52, 83, 85, 87, 88]; identifying slow components andfunctions [39, 68, 87]; andmodelling workloads and resourceusage [32, 33, 68, 88]. Recent work has extended these tech-niques to online proûling and analysis [55, 71–73, 99].

VScope [93] introduces a novel mechanism for honing inon root causes on a running system, but at the lasthop defers tooøine user analysis of debug-level logs, requiring the user totrawl through 500MB of logswhich incur a 99.1%performanceoverhead to generate.While causal tracing enables coherentsampling [84, 87] which controls overheads, sampling risksmissing important information about rare but interestingevents.

8. DiscussionDespite the advantages over logs andmetrics for troubleshoot-ing (§2), Pivot Tracing is not meant to replace all functions oflogs, such as security auditing, forensics, or debugging [75].

Pivot Tracing is designed to have similar per-query over-heads to the metrics currently exposed by systems today. Itis feasible for a system to have several Pivot Tracing querieson by default; these could be sensible defaults provided bydevelopers, or custom queries installed by users to addresstheir speciûc needs. We leave it to future work to explore theuse of Pivot Tracing for automatic problem detection andexploration.Dynamic instrumentation is not a requirement to utilize

Pivot Tracing. By default, a system could hard-code a set ofpredeûned tracepoints.Without dynamic instrumentation theuser is restricted to those tracepoints; adding new tracepointsremains tied to the development and build cycles of the system.Inactive tracepoints would also incur at least the cost of aconditional check, instead of our current zero cost.A common criticism of systems that require causal prop-

agation of metadata is the need to instrument the originalsystems [45]. We argue that the beneûts outweigh the costsof doing so (§6), especially for new systems. A system thatdoes not implement baggage can still utilize the other mecha-nisms of Pivot Tracing; in such a case the system resemblesDTrace [38] or Fay [51]. Kernel-level execution context propa-gation [36, 40, 82] can provide language-independent accessto baggage variables.

While users are restricted to advice comprised of PivotTracing primitives, Pivot Tracing does not guarantee that itsqueries will be side-eòect free, due to the way exported vari-ables from tracepoints are currently deûned. We can enforce

that only trusted administrators deûne tracepoints and requirethat advice be signed for installation, but a comprehensive se-curity analysis, including complete sanitization of tracepointcode is beyond the scope of this paper.Even though we evaluated Pivot Tracing on an 8-node clus-

ter in this paper, initial runs of the instrumented systems on a200-node clusterwith constant-size baggage being propagatedshowed negligible performance impact. It is ongoing work toevaluate the scalability of Pivot Tracing to larger clusters andmore complex queries. Sampling at the advice level is a furthermethod of reducing overhead which we plan to investigate.

We opted to implement Pivot Tracing in Java in order toeasily instrument several popular open-source distributedsystems written in this language. However, the componentsof Pivot Tracing generalize and are not restricted to Java – aquery can span multiple systems written in diòerent program-ming languages due to Pivot Tracing’s platform-independentbaggage format and restricted set of advice operations. Inparticular, it would be an interesting exercise to integrate thehappened-before join with Fay or DTrace.

9. ConclusionPivot Tracing is the ûrst monitoring system to combine dy-namic instrumentation and causal tracing. Its novel happened-before join operator fundamentally increases the expressivepower of dynamic instrumentation and the applicability ofcausal tracing. Pivot Tracing enables cross-tier analysis be-tween any inter-operating applications, with low executionoverhead. Ultimately, its power lies in the uniform and ubiqui-tous way in which it integrates monitoring of a heterogeneousdistributed system.

AcknowledgmentsWe thank our shepherd Eddie Kohler and the anonymousSOSP reviewers for their invaluable discussions and sug-gestions. _is work was partially supported by NSF award#1452712, as well as by a Google Faculty Research Award.

References[1] ApacheHBase ReferenceGuide. http://hbase.apache.org/

book.html. [Online; accessed 25-Feb-2015]. (§2.3).

[2] HADOOP-6599 Split RPC metrics into summary and de-tailedmetrics. https://issues.apache.org/jira/browse/HADOOP-6599. [Online; accessed 25-Feb-2015]. (§2.3).

[3] HADOOP-6859 Introduce additional statistics to FileSystem.https://issues.apache.org/jira/browse/HADOOP-6859.[Online; accessed 25-Feb-2015]. (§2.3).

[4] HBASE-11559 Add dumping of DATA block usage to the Block-Cache JSON report. https://issues.apache.org/jira/browse/HBASE-11559. [Online; accessed 25-Feb-2015]. (§2.3).

[5] HBASE-12364 API for query metrics. https://issues.apache.org/jira/browse/HBASE-12364. [Online; accessed25-Feb-2015]. (§2.3).

[6] HBASE-12424 Finer grained logging and metrics for splittransaction. https://issues.apache.org/jira/browse/

HBASE-12424. [Online; accessed 25-Feb-2015]. (§2.3).[7] HBASE-12477 Add a �ush failed metric. https://issues.

apache.org/jira/browse/HBASE-12477. [Online; accessed25-Feb-2015]. (§2.3).

[8] HBASE-12494 Add metrics for blocked updates and de-layed �ushes. https://issues.apache.org/jira/browse/HBASE-12494. [Online; accessed 25-Feb-2015]. (§2.3).

[9] HBASE-12496 A blockedRequestsCount metric. https://issues.apache.org/jira/browse/HBASE-12496. [Online;accessed 25-Feb-2015]. (§2.3).

[10] HBASE-12574 Update replication metrics to not do so manymap look ups. https://issues.apache.org/jira/browse/HBASE-12574. [Online; accessed 25-Feb-2015]. (§2.3).

[11] HBASE-2257 [stargate] multiuser mode. https://issues.apache.org/jira/browse/HBASE-2257. [Online; accessed25-Feb-2015]. (§2.3).

[12] HBASE-4038 Hot Region :Write Diagnosis. https://issues.apache.org/jira/browse/HBASE-4038. [Online; accessed25-Feb-2015]. (§2.3).

[13] HBASE-4145 Provide metrics for hbase client. https://issues.apache.org/jira/browse/HBASE-4145. [Online;accessed 25-Feb-2015]. (§2.3).

[14] HBASE-4169Add per-disk latencymetrics toDataNode. https://issues.apache.org/jira/browse/HDFS-4169. [Online;accessed 25-Feb-2015]. (§2.3).

[15] HBASE-4219 Add Per-Column Family Metrics. https://issues.apache.org/jira/browse/HBASE-4219. [Online;accessed 25-Feb-2015].

[16] HBASE-5253 Add requesting user’s name to PathBased-CacheEntry. https://issues.apache.org/jira/browse/HDFS-5253. [Online; accessed 25-Feb-2015]. (§2.3).

[17] HBASE-6093 Expose more caching information for debug-ging by users. https://issues.apache.org/jira/browse/HDFS-6093. [Online; accessed 25-Feb-2015]. (§2.3).

[18] HBASE-6292 Display HDFS per user and per group usageon webUI. https://issues.apache.org/jira/browse/HDFS-6292. [Online; accessed 25-Feb-2015]. (§2.3).

[19] HBASE-7390 Provide JMX metrics per storage type. https://issues.apache.org/jira/browse/HDFS-7390. [Online;accessed 25-Feb-2015]. (§2.3).

[20] HBASE-7958 Statistics per-column family per-region. https://issues.apache.org/jira/browse/HBASE-7958. [Online;accessed 25-Feb-2015]. (§2.3).

[21] HBASE-8370 Report data block cache hit rates apart from ag-gregate cache hit rates. https://issues.apache.org/jira/browse/HBASE-8370. [Online; accessed 25-Feb-2015]. (§1and 2.3).

[22] HBASE-8868 add metric to report client shortcircuit reads.https://issues.apache.org/jira/browse/HBASE-8868.[Online; accessed 25-Feb-2015]. (§2.3).

[23] HBASE-9722 need documentation to conûgure HBase to re-ducemetrics. https://issues.apache.org/jira/browse/HBASE-9722. [Online; accessed 25-Feb-2015]. (§2.3).

[24] HDFS-6268 Better sorting in NetworkTopol-ogy.pseudoSortByDistance when no local node is found.

https://issues.apache.org/jira/browse/HDFS-6268.[Online; accessed 25-Feb-2015]. (§6.1, 8, and 6.1).

[25] MESOS-1949 All log messages from master, slave, executor,etc. should be collected on a per-task basis. https://issues.apache.org/jira/browse/MESOS-1949. [Online; accessed25-Feb-2015]. (§2.3).

[26] MESOS-2157 Add /master/slaves and /master/frame-works/{framework}/tasks/{task} endpoints. https://issues.apache.org/jira/browse/MESOS-2157. [Online;accessed 25-Feb-2015]. (§2.3).

[27] Apache accumulo. http://accumulo.apache.org/. [Online;accessedMarch 2015]. (§2.3).

[28] Aguilera, M. K., Mogul, J. C., Wiener, J. L., Reynolds,P., and Muthitacharoen, A. Performance debugging fordistributed systems of black boxes. In SOSP (New York, NY,USA, 2003), ACM Press. (§7).

[29] Almeida, P. S., Baquero, C., and Fonte, V. Interval treeclocks: A logical clock for dynamic systems. In OPODIS (Berlin,Heidelberg, 2008), Springer-Verlag, pp. 259–274. (§5).

[30] Appneta traceview. http://appneta.com. [Online; accessedJuly 2013]. (§7).

[31] Attariyan,M., Chow,M., and Flinn, J. X-ray: Automatingroot-cause diagnosis of performance anomalies in productionso�ware. In OSDI (2012), pp. 307–320. (§7).

[32] Barham, P., Donnelly, A., Isaacs, R., andMortier, R. Usingmagpie for request extraction andworkloadmodelling. InOSDI(2004), vol. 4, pp. 18–18. (§4 and 7).

[33] Barham, P., Isaacs, R., Mortier, R., and Narayanan, D.Magpie: Onlinemodelling and performance-aware systems. InHotOS (2003), vol. 9. (§7).

[34] Beschastnikh, I., Brun, Y., Ernst, M. D., and Krishna-murthy, A. Inferringmodels of concurrent systems from logs oftheir behaviorwithCSight. In ICSE (Hyderabad, India, June 4–6,2014), pp. 468–479. (§7).

[35] Bodik, P. Overview of theWorkshop ofManaging Large-ScaleSystems via the Analysis of System Logs and the Application ofMachine Learning Techniques (SLAML’11). SIGOPS OperatingSystems Review 45, 3 (2011), 20–22. (§2.3).

[36] Buch, I., and Park, R. Improve debugging and performancetuning with ETW. MSDN Magazine (2007). [Online; accessed01-01-2012]. (§8).

[37] Cantrill, B. Hidden in plain sight. ACM Queue 4, 1 (Feb.2006), 26–36. (§1 and 2.3).

[38] Cantrill, B., Shapiro, M. W., and Leventhal, A. H. Dy-namic instrumentation of production systems. In USENIX ATC(2004), pp. 15–28. (§1, 2.3, 5, 7, and 8).

[39] Chanda, A., Cox, A. L., and Zwaenepoel, W. Whodunit:Transactional proûling formulti-tier applications. ACM SIGOPSOperating Systems Review 41, 3 (2007), 17–30. (§1, 6.2, and 7).

[40] Chanda, A., Elmeleegy, K., Cox, A. L., and Zwaenepoel,W. Causeway: System support for controlling and analyzing theexecution ofmulti-tier applications. In Middleware (November2005), pp. 42–59. (§8).

[41] Chang, F., Dean, J., Ghemawat, S.,Hsieh,W. C.,Wallach,D. A., Burrows, M., Chandra, T., Fikes, A., and Gruber,

R. E. Bigtable: A distributed storage system for structured data.ACM Transactions on Computer Systems (TOCS) 26, 2 (2008), 4.(§6).

[42] Chen, M. Y., Accardi, A., Kiciman, E., Patterson, D. A.,Fox, A., and Brewer, E. A. Path-based failure and evolutionmanagement. In NSDI (2004). (§7).

[43] Chen,M. Y., Kiciman, E., Fratkin, E., Fox, A., and Brewer,E. Pinpoint: ProblemDetermination in Large, Dynamic InternetServices. InDSN (Washington, DC,USA, 2002), DSN ’02, IEEEComputer Society, pp. 595–604. (§7).

[44] Chiba, S. Javassist: Java bytecode engineering made simple.Java Developer’s Journal 9, 1 (2004). (§5).

[45] Chow, M., Meisner, D., Flinn, J., Peek, D., andWenisch,T. F. _emysterymachine: End-to-end performance analysisof large-scale internet services. In OSDI (Broomûeld, CO, Oct.2014), USENIX Association, pp. 217–231. (§7 and 8).

[46] Compuware dynatrace purepath. http://www.compuware.com. [Online; accessed July 2013]. (§7).

[47] Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R.,and Sears, R. Benchmarking cloud serving systems with ycsb.In SOCC (2010), ACM, pp. 143–154. (§6.3).

[48] Couckuyt, J., Davies, P., and Cahill, J. Multiple chart userinterface, June 14 2005. US Patent 6,906,717. (§1).

[49] Dean, J., and Ghemawat, S. Mapreduce: simpliûed dataprocessing on large clusters. Communications of the ACM 51, 1(2008), 107–113. (§6).

[50] Do, T., Hao,M., Leesatapornwongsa, T., Patana-anake,T., and Gunawi, H. S. Limplock: Understanding the impact oflimpware on scale-out cloud systems. In SOCC (2013), ACM,p. 14. (§6.2).

[51] Erlingsson, Ú., Peinado, M., Peter, S., Budiu, M., andMainar-Ruiz, G. Fay: extensible distributed tracing fromkernels to clusters. ACM Transactions on Computer Systems(TOCS) 30, 4 (2012), 13. (§1, 2.3, 4, 5, 7, and 8).

[52] Fonseca, R., Porter, G., Katz, R. H., Shenker, S., andStoica, I. X-trace: A pervasive network tracing framework. InNSDI (Berkeley, CA,USA, 2007),NSDI’07,USENIXAssociation.(§1, 4, 4, and 7).

[53] Google Protocol Buòers. http://code.google.com/p/protobuf/. (§5).

[54] Gray, J., Chaudhuri, S., Bosworth, A., Layman, A., Re-ichart, D., Venkatrao, M., Pellow, F., and Pirahesh, H.Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub-totals. DataMining and Knowledge Dis-covery 1, 1 (1997), 29–53. (§1).

[55] Guo, Z., Zhou, D., Lin,H., Yang,M., Long, F., Deng, C., Liu,C., and Zhou, L. G2: A graph processing system for diagnosingdistributed systems. In USENIX ATC (2011). (§7).

[56] Apache HBase. http://hbase.apache.org. [Online; ac-cessedMarch 2015]. (§6).

[57] _e Java HotSpot Performance Engine Architec-ture. http://www.oracle.com/technetwork/java/whitepaper-135217.html. [Online; accessed March 2015].(§6.3).

[58] Apache HTrace. http://htrace.incubator.apache.org/.[Online; accessedMarch 2015]. (§2.3 and 7).

[59] Huang, S., Huang, J., Dai, J., Xie, T., and Huang, B. _ehibench benchmark suite: Characterization of themapreduce-based data analysis. In ICDEW (2010), IEEE, pp. 41–51. (§6.3).

[60] Kavulya, S. P., Daniels, S., Joshi, K.,Hiltunen,M., Gandhi,R., and Narasimhan, P. Draco: Statistical diagnosis of chronicproblems in large distributed systems. In IEEE/IFIP Conferenceon Dependable Systems and Networks (DSN) (June 2012). (§1and 7).

[61] Kiczales, G.,Hilsdale, E.,Hugunin, J., Kersten,M., Palm,J., and Griswold,W. G. An Overview of AspectJ. In ECOOP(London, UK, UK, 2001), ECOOP ’01, Springer-Verlag, pp. 327–353. (§5 and 6).

[62] Kiczales,G., Lamping, J.,Mendhekar, A.,Maeda, C., Lopes,C. V., Loingtier, J.-M., and Irwin, J. Aspect-Oriented Pro-gramming. In ECOOP (June 1997), LNCS 1241, Springer-Verlag.(§2.2 and 7).

[63] Kim,M., Sumbaly, R., and Shah, S. Root cause detection in aservice-oriented architecture. ACM SIGMETRICS PerformanceEvaluation Review 41, 1 (2013), 93–104. (§7).

[64] Ko, S. Y., Yalagandula, P., Gupta, I., Talwar, V.,Milojicic,D., and Iyer, S. Moara: �exible and scalable group-basedquerying system. In Middleware 2008. Springer, 2008, pp. 408–428. (§7).

[65] Lamport, L. Time, clocks, and the ordering of events in adistributed system. Communications of the ACM 21, 7 (1978),558–565. (§1 and 3).

[66] Laub, B., Wang, C., Schwan, K., and Huneycutt, C. To-wards combining online & oøine management for big dataapplications. In ICAC (Philadelphia, PA, June 2014), USENIXAssociation, pp. 121–127. (§6 and 6.2).

[67] Mace, J., Bodik, P.,Musuvathi,M., and Fonseca, R. Retro:Targeted resourcemanagement in multi-tenant distributed sys-tems. In NSDI (May 2015), USENIX Association. (§6.2).

[68] Mann, G., Sandler, M., Krushevskaja, D., Guha, S., andEven-Dar, E. Modeling the parallel execution of black-boxservices. USENIX/HotCloud (2011). (§7).

[69] Massie,M. L., Chun, B. N., and Culler, D. E. _e gangliadistributed monitoring system: design, implementation, andexperience. Parallel Computing 30, 7 (2004), 817–840. (§2.3and 7).

[70] Meijer, E., Beckman, B., and Bierman, G. Linq: Reconcilingobject, relations and xml in the .net framework. In SIGMOD(New York, NY, USA, 2006), SIGMOD ’06, ACM, pp. 706–706.(§2.1).

[71] Mi,H.,Wang,H., Chen, Z., and Zhou, Y. Automatic detectingperformance bugs in cloud computing systems via learninglatency speciûcation model. In SOSE (2014), IEEE, pp. 302–307.(§7).

[72] Mi,H.,Wang,H., Zhou, Y., Lyu,M. R., and Cai,H. Towardûne-grained, unsupervised, scalable performance diagnosis forproduction cloud computing systems. IEEE Transactions onParallel and Distributed Systems 24, 6 (2013), 1245–1255.

[73] Mi,H.,Wang,H., Zhou, Y., Lyu,M. R.-T., Cai,H., and Yin,G. An online service-oriented performance proûling tool forcloud computing systems. Frontiers of Computer Science 7, 3(2013), 431–445. (§1 and 7).

[74] Nagaraj, K., Killian, C. E., and Neville, J. Structuredcomparative analysis of systems logs to diagnose performanceproblems. In NSDI (2012), pp. 353–366. (§1 and 7).

[75] Oliner, A., Ganapathi, A., and Xu,W. Advances and chal-lenges in log analysis. Communications of the ACM 55, 2 (2012),55–61. (§2.3 and 8).

[76] Oliner, A., Kulkarni, A., and Aiken, A. Using correlatedsurprise to infer shared in�uence. In IEEE/IFIP DependableSystems andNetworks (DSN) (June 2010), pp. 191–200. (§1 and 7).

[77] Ostrowski, K., Mann, G., and Sandler, M. Diagnosinglatency in multi-tier black-box services. In LADIS (2011). (§7).

[78] Prasad, V., Cohen,W., Eigler, F. C., Hunt,M., Keniston,J., and Chen, B. Locating system problems using dynamicinstrumentation. In Ottawa Linux Symposium (OLS) (2005).(§2.3 and 7).

[79] Rabkin, A., and Katz, R. H. How hadoop clusters break.So�ware, IEEE 30, 4 (2013), 88–94. (§2.3).

[80] Ramakrishnan, R., and Gehrke, J. DatabaseManagementSystems, 2nd ed. Osborne/McGraw-Hill, Berkeley, CA, USA,2000. (§4 and 7).

[81] Ravindranath, L., Padhye, J.,Mahajan, R., and Balakr-ishnan, H. Timecard: Controlling user-perceived delays inserver-basedmobile applications. In SOSP (2013), ACM, pp. 85–100. (§6.2).

[82] Reumann, J., and Shin,K. G. Stateful distributed interposition.ACM Trans. Comput. Syst. 22, 1 (2004), 1–48. (§8).

[83] Reynolds, P., Killian, C.,Wiener, J. L.,Mogul, J. C., Shah,M. A., and Vahdat, A. Pip: detecting the unexpected indistributed systems. In NSDI (Berkeley, CA, USA, 2006),USENIX Association. (§7).

[84] Sambasivan, R. R., Fonseca, R., Shafer, I., and Ganger,G. R. So, you want to trace your distributed system? Keydesign insights from years of practical experience. Tech. Rep.CMU-PDL-14-102, Parallel Data Laboratory, CarnegieMellonUniversity, Pittsburgh, PA 15213-3890, April 2014. (§7).

[85] Sambasivan, R. R., Zheng, A. X., De Rosa,M., Krevat, E.,Whitman, S., Stroucken,M.,Wang,W.,Xu, L., andGanger,G. R. Diagnosing performance changes by comparing request�ows. In NSDI (2011). (§7).

[86] Shvachko, K., Kuang,H., Radia, S., and Chansler, R. _eHadoop distributed ûle system. In MSST (2010), IEEE, pp. 1–10.(§6).

[87] Sigelman, B. H., Barroso, L. A., Burrows,M., Stephenson,P., Plakal, M., Beaver, D., Jaspan, S., and Shanbhag, C.

Dapper, a large-scale distributed systems tracing infrastructure.Google research (2010). (§1, 2.3, 4, 4, and 7).

[88] Thereska, E., Salmon, B., Strunk, J.,Wachs,M., Abd-El-Malek,M., Lopez, J., and Ganger, G. R. Stardust: trackingactivity in a distributed storage system. SIGMETRICS Perform.Eval. Rev. 34, 1 (2006), 3–14. (§4 and 7).

[89] Twitter Zipkin. http://twitter.github.io/zipkin/. [On-line; accessedMarch 2015]. (§2.3 and 7).

[90] Van Renesse, R., Birman, K. P., and Vogels,W. Astrolabe:A robust and scalable technology for distributed system moni-toring,management, and datamining. ACM Transactions onComputer Systems (TOCS) 21, 2 (2003), 164–206. (§7).

[91] Vavilapalli, V. K.,Murthy, A. C., Douglas, C., Agarwal,S., Konar,M., Evans, R.,Graves, T., Lowe, J., Shah,H., Seth,S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B.,and Baldeschwieler, E. ApacheHadoop YARN: Yet AnotherResource Negotiator. In SOCC (New York, NY, USA, 2013),SOCC ’13, ACM, pp. 5:1–5:16. (§6).

[92] Wang, C., Kavulya, S. P., Tan, J.,Hu, L., Kutare,M., Kasick,M., Schwan, K., Narasimhan, P., and Gandhi, R. Perfor-mance troubleshooting in data centers: an annotated bibliog-raphy? ACM SIGOPS Operating Systems Review 47, 3 (2013),50–62. (§7).

[93] Wang, C., Rayan, I. A., Eisenhauer,G., Schwan, K., Talwar,V., Wolf, M., and Huneycutt, C. Vscope: middleware fortroubleshooting time-sensitive data center applications. InMiddleware 2012. Springer, 2012, pp. 121–141. (§6, 6.2, and 7).

[94] Wood, P. T. Query languages for graph databases. SIGMODRec. 41, 1 (Apr. 2012), 50–60. (§7).

[95] Xu,W.,Huang, L., Fox, A., Patterson, D., and Jordan,M. I.Detecting large-scale system problems by mining console logs.In SOSP (New York, NY, USA, 2009), ACM, pp. 117–132. (§1and 7).

[96] Yin, Z.,Ma, X., Zheng, J., Zhou, Y., Bairavasundaram, L. N.,and Pasupathy, S. An empirical study on conûguration errorsin commercial and open source systems. In SOSP (2011), ACM,pp. 159–172. (§2.3).

[97] Yuan, D., Zheng, J., Park, S., Zhou,Y., and Savage, S. Improv-ing so�ware diagnosability via log enhancement. In Proceedingsof the International Conference on Architecture Support for Pro-gramming Languages and Operating Systems (March 2011). (§1,2.3, and 7).

[98] Zhao, X., Zhang, Y., Lion, D., Faizan,M., Luo, Y., Yuan, D.,and Stumm,M. lprof: A nonintrusive request �ow proûler fordistributed systems. In OSDI (2014). (§1 and 7).

[99] Zhou, J., Chen, Z., Mi, H., and Wang, J. Mtracer: a trace-orientedmonitoring framework for medium-scale distributedsystems. In SOSE (2014), IEEE, pp. 266–271. (§7).