the cql continuous query language: semantic foundations ... · components of our abstract...

The VLDB Journal (2006) 15(2): 121–142DOI 10.1007/s00778-004-0147-z

REGULAR PAPER

Arvind Arasu · Shivnath Babu · Jennifer Widom

The CQL continuous query language: semantic foundationsand query execution

Received: 7 June 2004 / Accepted: 22 November 2004 / Published online: 22 July 2005c© Springer-Verlag 2005

Abstract CQL, a continuous query language, is supportedby the STREAM prototype data stream management sys-tem (DSMS) at Stanford. CQL is an expressive SQL-baseddeclarative language for registering continuous queriesagainst streams and stored relations. We begin by presentingan abstract semantics that relies only on “black-box” map-pings among streams and relations. From these mappingswe define a precise and general interpretation for continu-ous queries. CQL is an instantiation of our abstract seman-tics using SQL to map from relations to relations, windowspecifications derived from SQL-99 to map from streamsto relations, and three new operators to map from relationsto streams. Most of the CQL language is operational inthe STREAM system. We present the structure of CQL’squery execution plans as well as details of the most impor-tant components: operators, interoperator queues, synopses,and sharing of components among multiple operators andqueries. Examples throughout the paper are drawn from theLinear Road benchmark recently proposed for DSMSs. Wealso curate a public repository of data stream applicationsthat includes a wide variety of queries expressed in CQL.The relative ease of capturing these applications in CQL isone indicator that the language contains an appropriate setof constructs for data stream processing.

Keywords Data streams · Continuous queries · Querylanguage · Query processing

Edited by M. Franklin

1 Introduction

There has been a considerable surge of research in manyaspects of processing continuous queries over unboundeddata streams [24, 25]. Many papers include example contin-uous queries expressed in some declarative language, e.g.,

A. Arasu (B) · S. Babu · J. WidomComputer Science Department, Stanford University, Stanford, CA94305-2140, USAE-mail: {arvinda, shivnath, widom}@cs.stanford.edu

[2, 19, 20, 23, 28, 32]. However, these queries tend to besimple and primarily for illustration – a precise languagesemantics, particularly for more complex queries, often isleft unclear. Furthermore, very little has been published todate covering execution details of general-purpose continu-ous queries. In this paper we present the CQL language andexecution engine for general-purpose continuous queriesover streams and stored relations. CQL (for continuousquery language) is an instantiation of a precise abstractcontinuous semantics also presented in this paper, andCQL is implemented in the STREAM prototype data streammanagement system (DSMS) at Stanford.1

It may appear initially that defining a continuous querylanguage over (relational) streams is not difficult: take a re-lational query language, replace references to relations withreferences to streams, register the query with the stream pro-cessor, and wait for answers to arrive. For simple monotonicqueries over complete stream histories, indeed this approachis nearly sufficient. However, as queries get more complex– when we add aggregation, subqueries, windowing con-structs, relations mixed with streams, etc. – the situation be-comes much murkier. Consider the following simple query:

Select P.priceFrom Items[Rows 5] as I,

PriceTable as PWhere I.itemID = P.itemID

Items is a stream of purchased items, PriceTable isa stored relation containing the price of items, and [Rows5] specifies a 5-element sliding window. Even this simplequery has no single obvious interpretation that we know of.For example, is the result of the query a stream or a relation?What happens to the query result when the price of a recentlypurchased item – i.e., an item still within the 5-element win-dow – changes?

In this paper, we initially define a precise abstract se-mantics for continuous queries. Our abstract semantics isbased on two data types – streams and relations – and threeclasses of operators over these types: operators that produce

1 http://www-db.stanford.edu/stream

122 A. Arasu et al.

a relation from a stream (stream-to-relation), operators thatproduce a relation from other relations (relation-to-relation),and operators that produce a stream from a relation (stream-to-relation). The three classes of operators are “black-box”components of our abstract semantics: the abstract seman-tics does not depend on the actual behavior of the operatorsin these classes, but only on their input and output types.

CQL instantiates the black boxes in our abstract seman-tics: it uses SQL to express its relation-to-relation operators,a window specification language derived from SQL-99 toexpress its stream-to-relation operators, and a set of threeoperators for its relation-to-stream operators. Most of CQLis fully operational in our prototype DSMS.2 CQL has beenused to specify the Linear Road benchmark proposed fordata stream systems [4] and to specify a variety of otherstream applications in a public repository we are curating.3

In defining our abstract semantics and concrete languagewe had certain goals in mind:

1. We wanted to exploit well-understood relational seman-tics (and by extension relational rewrites and executionstrategies) to the extent possible.

2. We wanted queries performing simple tasks to beeasy and compact to write. Conversely, we wantedsimple-appearing queries to do what one expects.

3. We wanted the language to have sufficient constructs tocapture a wide variety of stream applications, without al-lowing the “feature creep” that can result in an esoteric,difficult-to-understand, or difficult-to-implement lan-guage. That is, we wanted to keep the language as simpleas possible without sacrificing too much expressiveness.

We believe these goals have been achieved to a large extent.The STREAM query processing engine is based on phys-

ical query plans generated from CQL textual queries. Oftenquery plan merging occurs, so a single query plan may com-pute multiple continuous queries. In this paper we focus onthe structure and details of the execution plans themselves,not on how initial plans are selected or how plans adapt overtime (which is the subject of other papers [10, 11, 13]).

In developing the structure of our query execution planswe had certain goals in mind:

4. We wanted plans built from modular and pluggable com-ponents based on generic interfaces, especially for oper-ators and synopsis structures.

5. We wanted an execution model that efficiently capturesthe combination of streams and relations that forms thebasis of our language.

6. We wanted an architecture permitting easy experimen-tation with different strategies for operator scheduling,overflowing state to disk, sharing state and computationamong multiple continuous queries, and other crucial is-sues affecting performance.

Here, too, we believe these goals have been achieved to alarge extent.

2 http://www-db.stanford.edu/stream3 SQR: A Stream Query Repository, http://www-db.

stanford.edu/stream/sqr/

To summarize the contributions of this paper:

– We formalize streams and relations that are updatedover time (Sect. 4), and we define an abstract semanticsfor continuous queries based on three black-box classesof operators: stream-to-relation, relation-to-relation, andrelation-to-stream (Sect. 5).

– We define our concrete language, CQL, which instantiatesthe black boxes in the abstract semantics as discussed ear-lier (Sect. 6). We define syntactic shortcuts and defaults inCQL for convenient and intuitive query formulation, andwe point out a few equivalences in the language (Sect. 10).

– We illustrate CQL using a hypothetical road traffic man-agement application proposed as a benchmark for datastream systems [4] (Sects. 3 and 7). We also compare theexpressiveness of CQL against related query languages(Sect. 11).

– We describe the query execution plans and strategiesused in the STREAM system for CQL queries. We focusspecifically on operators, interoperator queues, synopses,and sharing of components among multiple operators andqueries (Sect. 12).

2 Related work

A preliminary description of our abstract semantics and theCQL language appeared as an invited paper in [3]. That pa-per did not include query transformations, the Linear Roadbenchmark, or any of the material on query execution in-cluded in this paper.

A comprehensive description of work related to datastreams and continuous queries is given in [8]. Here we fo-cus on work related to languages and semantics for continu-ous queries.

Continuous queries have been used either explicitly orimplicitly for quite some time. Materialized views [26] are aform of continuous query, since a view is continuously up-dated to reflect changes to its base relations. Reference [29]extends materialized views to include chronicles, which es-sentially are continuous data streams. Operators are definedover chronicles and relations to produce other chronicles,and also to transform chronicles into materialized views.The operators are constrained to ensure that the resultingmaterialized views can be maintained incrementally withoutreferencing entire chronicle histories.

Continuous queries were introduced explicitly for thefirst time in Tapestry [41] with a SQL-based language calledTQL. (A similar language is considered in [14].) Concep-tually, a TQL query is executed once every time instant asa one-time SQL query over the snapshot of the databaseat that instant, and the results of all the one-time queriesare merged using set union. Several systems use continu-ous queries for information dissemination, e.g., [20, 31, 34].The semantics of continuous queries in these systems isalso based on periodic execution of one-time queries as inTapestry. In Sect. 11, we show how Tapestry queries and

The CQL continuous query language 123

materialized views over relations and chronicles can be ex-pressed in CQL.

The abstract semantics and concrete language proposedin this paper are more general than any of the languagesabove, incorporating window specifications, constructs forfreely mixing and mapping streams and relations, and thefull power of any relational query language. Recent workin the TelegraphCQ system [18] proposes a declarativelanguage for continuous queries with a particular focuson expressive windowing constructs. The TelegraphCQlanguage is discussed again briefly in Sect. 11.6. ATLas[30, 45, 46] proposes simple extensions to SQL-99 user-defined aggregates (UDAs) that make the resulting languageTuring complete, suitable for various data mining and datastream applications. Intuitively, the extensions let usersexpress initialize, iterate, and terminate parts of a SQL-99UDA specification using SQL update constructs rather thanusing procedural code. GSQL [21] is a SQL-like languagedeveloped for Gigascope, a DSMS for network monitoringapplications. GSQL is compared against CQL in Sect. 11.

Several systems support procedural continuous queries,as opposed to the declarative approach in this paper. TheAurora system [16] is based on users directly creating anetwork of stream operators. A large number of operatortypes are available, from simple stream filters to complexwindowing and aggregation operators. The Tribeca stream-processing system for network traffic analysis [39] supportswindows, a set of operators adapted from relational alge-bra, and a simple language for composing query plans fromthem. Tribeca does not support joins across streams. BothAurora and Tribeca are compared against CQL in more de-tail in Sect. 11.

Temporal query languages [35] are based on a richmodel of history, and with their many special-purpose lan-guage constructs they dramatically subsume CQL in termsof expressiveness. Based on the several different applica-tions we studied,4 the full expressive power of temporalquery languages seems unnecessary for most stream applica-tions. As mentioned in goal 3 earlier, our intention is to keepCQL much simpler than temporal languages, for ease ofimplementation and ease of understanding. While sequencequery languages [36] are not as complex as temporal querylanguages, they still largely subsume CQL in terms of ex-pressiveness, and they include many special-purpose opera-tors specifically for manipulating sequences. Finally, event-processing languages are geared largely toward matchingsingle events or specific event patterns against queries, usu-ally in a publish-subscribe setting [47]. The fine-grainedevent-matching constructs of these languages also were notneeded for the stream applications we studied.

Note that our abstract semantics and even our concretelanguage permit temporal, sequence, or event-processing ca-pabilities to be “plugged in” as black-box operators. How-ever, if we discovered that such capabilities were neededfrequently in future stream applications, it would be betterto fully incorporate them as a feature of the language.

4 http://www-db.stanford.edu/stream/sqr/

3 Introduction to running example

We introduce a running example based on a hypotheticalroad traffic management application introduced in the Lin-ear Road benchmark for DSMSs [4]. We use a simplifiedversion of the Linear Road application to illustrate variousaspects of our language, semantics, and execution plans; fulldetails can be found in the original specification [4].

The Linear Road application implements variable tolling– adaptive, real-time computation of vehicle tolls based ontraffic conditions – to regulate vehicular traffic on a high-way. To enable variable tolling, each vehicle is equippedwith a sensor that continuously relays its position and speedto a central server. The server aggregates the information re-ceived from all vehicles on the highway system, computestolls in real time, and transmits tolls back to vehicles usingthe sensor network.

Figure 1 shows the highway system used by our simpli-fied Linear Road. There is a single highway 100 miles long,which is divided into 100 1-mile segments. The highway hasentrance and exit ramps at segment boundaries. Traffic flowsin a single direction from left to right. When a vehicle ison the highway, it reports its current speed (miles per hour)and position (number of feet from the left end) to the serveronce every 30 s. (In the complete Linear Road application[4] there are ten highways with multiple lanes, and trafficflows in both directions.)

Vehicles pay a toll whenever they drive in a congestedsegment, while no toll is charged for uncongested segments.A segment is congested if the average speed of all vehiclesin the segment over the last 5 min is less than 40 miles perhour (mph). The toll for a congested segment is given by theformula 2 × (numvehicles − 50)2, where numvehicles is thenumber of vehicles currently in the segment. Note that thetoll for a congested segment changes dynamically as vehi-cles enter and leave the segment. When the server detectsthat a vehicle has entered a congested segment, the serveroutputs the current toll for the segment which is conveyedback to the vehicle.

In stream terminology, the simplified Linear Road appli-cation in this paper has:

– A single input stream – the stream of positions and speedsof vehicles.

– A single continuous query computing the tolls.– A single output stream containing the tolls for vehicles.

Seg 0 Seg 1 Seg 98 Seg99

1 Mile

100 Miles

Fig. 1 The Linear Road highway system

124 A. Arasu et al.

As the paper progresses we will illustrate how we model,express, and execute this application using our language, se-mantics, and query execution strategies.

4 Streams and relations

In this section we define a formal model of streams and re-lations. As in the standard relational model, each stream andrelation has a fixed schema consisting of a set of named at-tributes. For stream element arrivals and relation updates weassume a discrete, ordered time domain T . A time instant (orsimply instant) is any value from T . For concreteness, werepresent T as the nonnegative integers {0, 1, . . .}; in par-ticular note that 0 stands for the earliest time instant. Timedomain T models an application’s notion of time, not par-ticularly system or wall-clock time. Thus, although T mayoften be of type Datetime, our semantics only requires anydiscrete, ordered domain. (A thorough discussion of time is-sues in DSMSs appears in [37].)

Definition 1 (Stream) A stream S is a (possibly infinite)bag (multiset) of elements 〈s, τ 〉, where s is a tuple belong-ing to the schema of S and τ ∈ T is the timestamp of theelement.

Note that the timestamp is not part of the schema of astream, and there could be zero, one, or multiple elementswith the same timestamp in a stream. We only require thatthere be a finite (but unbounded) number of elements with agiven timestamp.

There are two classes of streams: base streams, whichare the source data streams that arrive at the DSMS, and de-rived streams, which are intermediate streams produced byoperators in a query. We use the term tuple of a stream todenote the data (nontimestamp) portion of a stream element.

Example 1 In the Linear Road application there is just onebase stream containing vehicle speed – position measure-ments, with schema:

PosSpeedStr (vehicleId, speed, xPos)

Attribute vehicleId identifies the vehicle, speed de-notes its current speed in mph, and xPos denotes its currentposition within the highway in feet as described in Sect. 3.The time domain is of type Datetime, and for this applica-tion the timestamp of a stream element denotes the physicaltime when the position and speed measurements were taken.

Definition 2 (Relation) A relation R is a mapping fromeach time instant in T to a finite but unbounded bag of tuplesbelonging to the schema of R.

A relation R defines an unordered bag of tuples at anytime instant τ ∈ T , denoted R(τ ). Note the difference be-tween this definition for relation and the standard one: in thestandard relational model a relation is simply a set (or bag)of tuples, with no notion of time as far as the semantics ofrelational query languages are concerned.

We use the term instantaneous relation to denote the bagof tuples in a relation at a given point in time. Thus, if Rdenotes a relation according to Definition 2, R(τ ) denotesan instantaneous relation. Often when there is no ambiguitywe omit the term instantaneous. We use the term base re-lation for input relations and derived relation for relationsproduced by query operators.

Example 2 The Linear Road application contains no baserelations, but several derived relations are useful in toll com-putation. For example, the toll for a congested segment de-pends on the current number of vehicles in the segment. Wecan represent the current number of vehicles in a segmentusing the derived relation:

SegVolRel (segNo, numVehicles)

Attribute segNo denotes the segment (0-99) andnumVehicles the number of vehicles in the seg-ment. At time τ , SegVolRel(τ ) contains the count ofvehicles in each highway segment as of time τ . Section 7shows how SegVolRel can be computed from the basestream PosSpeedStr and how it can be used to computetolls.

As this example suggests, the concept of a relation isuseful even in applications whose inputs and outputs are allstreams. It seems more natural to model “the current numberof vehicles in a segment” as a time-varying relation, ratherthan as a stream of the latest values. From an expressive-ness point of view, it is not necessary to have both streamsand relations: we could have picked just streams or relationsand designed our language around it without loss of expres-siveness; this issue is discussed further in Sect. 11.6. In ourimplementation we encode both streams and relations uni-formly as “plus-minus streams,” as discussed in Sect. 12.

5 Abstract semantics

This section presents our abstract semantics for continuousqueries. Recall from Sect. 1 that our semantics is based onthree classes of operators over streams and relations:

– Stream-to-relation operators that produce a relation froma stream,

– Relation-to-relation operators that produce a relationfrom one or more other relations,

– Relation-to-stream operators that produce a stream froma relation.

Recall that by “relation” we are referring to our formalnotion of a time-varying relation, as defined in Sect. 4.

Stream-to-stream operators are absent – they have to beconstructed from operators of the three classes above. As wewill discuss in detail in Sect. 6, the rationale for this decisionis based primarily on our goal 1 from Sect. 1: exploitingwell-understood relational semantics (and, by extension, re-lational rewrites and execution strategies) to the extent pos-sible.


Streams Relations

Relation–to–RelationStream–to–Relation

Relation–to–Stream

Fig. 2 Operator classes and mappings used in abstract semantics

First some terminology: S up to τ denotes the bag ofelements in stream S with timestamps ≤ τ , i.e., {〈s, τ ′〉 ∈S : τ ′ ≤ τ }. S at τ denotes the bag of elements of Swith timestamp τ , i.e., {〈s, τ ′〉 ∈ S : τ ′ = τ }. Similarly,R up to τ denotes the collection of instantaneous relationsR(0), . . . , R(τ ), and R at τ denotes the instantaneous rela-tion R(τ ).

1. A stream-to-relation operator takes a stream S as in-put and produces a relation R as output with the sameschema as S. At any instant τ , R(τ ) should be com-putable from S up to τ .

2. A relation-to-relation operator takes one or more rela-tions R1, . . . , Rn as input and produces a relation R asoutput. At any instant τ , R(τ ) should be computablefrom R1(τ ), . . . , Rn(τ ).

3. A relation-to-stream operator takes a relation R as inputand produces a stream S as output with the same schemaas R. At any instant τ , S at τ should be computable fromR up to τ .

Now we define our abstract semantics.

Definition 3 (Continuous semantics) Consider a query Qthat is any type-consistent composition of operators from theabove three classes. Suppose the set of all inputs to the inner-most (leaf) operators of Q are streams S1, . . . , Sn (n ≥ 0)and relations R1, . . . , Rm (m ≥ 0). We define the result ofcontinuous query Q at a time τ , which denotes the result ofQ once all inputs up to τ are “available” (a notion discussedbelow). There are two cases:

– Case 1 The outermost (topmost) operator in Q is relation-to-stream, producing a stream S (say). The result of Q attime τ is S up to τ , produced by recursively applying theoperators comprising Q to streams S1, . . . , Sn up to τ andrelations R1, . . . , Rm up to τ .

– Case 2 The outermost (topmost) operator in Q is stream-to-relation or relation-to-relation, producing a relation R(say). The result of Q at time τ is R(τ ), produced by re-cursively applying the operators comprising Q to streamsS1, . . . , Sn up to τ and relations R1, . . . , Rm up to τ .

Based on this definition, informally we can think of con-tinuous queries operationally as follows. Let time “advance”within domain T , further discussed below. First consider aquery producing a stream. At time τ ∈ T , all inputs up toτ are processed and the continuous query emits any new

stream result elements with timestamp τ . Because of our as-sumptions on operators, no new stream elements with times-tamp < τ can be produced from inputs with timestamp ≥ τ .A query producing a relation is similar: at time τ , all inputsup to τ are processed and the continuous query updates theoutput relation to state R(τ ).

Now let us understand what it means for time to ad-vance within domain T . The relationship between applica-tion time, wall-clock time, and system time is a complexissue discussed in depth in a separate paper [37]. However,for precise query semantics we need make no additional as-sumptions beyond those already made in this paper. Time“advances” to τ from τ − 1 when all inputs up to τ − 1have been processed. It appears we are tacitly assuming thatstreams arrive in timestamp order, relations are updated intimestamp order, and there is no timestamp “skew” acrossstreams or relations. In practice, to implement our semanticscorrectly, systems cope with out-of-order and skewed inputs.This issue is revisited in Sect. 8 and thoroughly covered by[37].

Example 3 Consider the query Istream(Filter(LastRow(S))) constructed from three operatorsand operating on stream S.5 Let stream S have a sin-gle attribute and consist of the elements {〈(a0), 0〉,〈(a1), 1〉, 〈(a2), 2〉, . . .}. LastRow is a stream-to-relationoperator; at any point in time the relation output byLastRow contains the last tuple that arrived on S. Filteris a relation-to-relation operator that produces its out-put relation by applying a filter condition on its input.Suppose that tuples (a0), (a2), (a4), . . . satisfy the filtercondition while tuples (a1), (a3), (a5), . . . do not. Finally,Istream is a relation-to-stream operator (defined formallyin Sect. 6.3) that “streams” every new tuple inserted into itsinput relation. Figure 3 shows the outputs produced by eachof the three operators as time progresses.

Example 4 In the Linear Road application the sequenceof operators producing derived relation SegVolRel ofExample 2 conceptually produces, at every time instant τ ,the instantaneous relation SegVolRel(τ ) containing thecurrent number of vehicles in each segment. In a DSMSimplementing our semantics, SegVolRel(τ ) cannot beproduced until it is known that all elements on input streamPosSpeedStr(vehicleId,speed,xPos) withtimestamp ≤ τ have been received. Furthermore, once theyhave been received, there may be additional lag before therelation is actually updated due to query processing time.Our semantics does not dictate “liveness” of continuousquery output – that issue is relegated to latency managementin the query processor [7, 17].

A formal specification of the abstract semantics usingdenotational semantics can be found in [5].

5 In CQL, this query is expressed as “Select Istream(*)From S [Rows 1] Where <filter>.”

126 A. Arasu et al.

Fig. 3 Output produced by operators of Example 3 as time progresses

6 Continuous query language

This section presents our concrete language, CQL, which isdefined by instantiating the operators of our abstract seman-tics. We also specify a few syntactic shortcuts and defaultsto simplify expression of some common operations.

Broadly, our approach to designing operators in CQL isas follows: support a large class of relation-to-relation oper-ators that perform the bulk of data manipulation in a typicalCQL query, along with a small set of stream-to-relation andrelation-to-stream operators that convert streams to relationsand back. The primary advantage of this approach is the abil-ity to reuse the formal foundations and huge body of im-plementation techniques for relation-to-relation languagessuch as relational algebra and SQL, instead of starting fromscratch with a heavily stream-based language. Furthermore,as we will see, queries in CQL are quite natural to express.

Technically, we cannot directly import existing conven-tional relation-to-relation operators into our concrete lan-guage since they operate on instantaneous relations whilewe operate on time-varying relations, but the mapping isobvious: Let Or denote a traditional relational operator orquery over instantaneous input relations R1, . . . , Rn . Thecorresponding relation-to-relation operator Oc in CQL pro-duces the time-varying relation R such that at each time τ ,R(τ ) = Or (R1(τ ), . . . , Rn(τ )).

An apparent drawback of our approach is that even asimple filter on a stream requires three operators: one to turnthe stream into a relation, one to perform a relational fil-ter, and one to turn the relation back into a stream. However,CQL’s defaults and syntactic shortcuts make filters and othersimple queries easy to express (Sect. 6.4).

Although we do not specify them explicitly as part ofour language, incorporating user-defined procedures, aggre-gates, and windows, as may be required for more complex,application-specific stream processing, is straightforward inCQL, at least from the semantics perspective.

Next, in Sects. 6.1–6.3 we cover the three classes of op-erators in CQL.

6.1 Stream-to-relation operators

Currently all stream-to-relation operators in CQL are basedon the concept of a sliding window over a stream: a win-dow that at any point of time contains a historical snapshotof a finite portion of the stream. We have three classes ofsliding-window operators in CQL: time-based, tuple-based,and partitioned, defined below. Syntactically, the sliding-window operators are specified using a window specificationlanguage derived from SQL-99. Other types of sliding win-dows such as fixed windows [39], tumbling windows [16],value-based windows [36], or any other windowing con-struct can be incorporated into CQL easily – new syntaxmust be added, but the semantics of incorporating a newwindow type relies solely on the semantics of the windowoperator itself, thanks to the development of our abstract se-mantics.

6.1.1 Time-based sliding windows

A time-based sliding window on a stream S takes a time-interval T as a parameter and is specified by following thereference to S with [Range T].6 We do not specify a syn-tax or restrictions for time interval T at this point, but weassume it specifies a computable period of application time.Intuitively, a time-based window defines its output relationover time by sliding an interval of size T time units captur-ing the latest portion of an ordered stream. More formally,the output relation R of “S [Range T]” is defined as:

R(τ ) = {s | 〈s, τ ′〉 ∈ S ∧(τ ′ ≤ τ) ∧(τ ′ ≥ max{τ −T, 0})}.Two important special cases are T = 0 and T = ∞.

When T = 0, R(τ ) consists of tuples obtained from ele-ments of S with timestamp τ . In CQL we introduce the syn-tax “S [Now]” for this special case. When T = ∞, R(τ )consists of tuples obtained from all elements of S up to τand uses the SQL-99 syntax “S [Range Unbounded].”We use the terms Now window and Unbounded window torefer to these two special windows.

Example 5 “PosSpeedStr [Range 30 Seconds]”denotes a time-based sliding window of 30 s over inputstream PosSpeedStr. At any time instant, the outputrelation of the sliding window contains the bag of position–speed measurements from the previous 30 s. Similarly, atany instant “PosSpeedStr [Now]” contains the (possi-bly empty) bag of position–speed measurements from thatinstant, and “PosSpeedStr [Range Unbounded]”contains the bag of all position–speed measurements so far.

6 In all three of our window types we dropped the keywordPreceding appearing in the SQL-99 syntax and in our earlier speci-fication [33] – we only have “preceding” windows for now so the key-word is superfluous.


6.1.2 Tuple-based windows

A tuple-based sliding window on a stream S takes a pos-itive integer N as a parameter and is specified by follow-ing the reference to S in the query with [Rows N]. In-tuitively, a tuple-based window defines its output relationover time by sliding a window of the last N tuples of anordered stream. More formally, for the output relation R of“S [Rows N],” R(τ ) consists of the N tuples of S withthe largest timestamps ≤ τ (or all tuples if the length of Sup to τ is ≤ N ). Suppose we specify a sliding window ofN tuples and at some point there are several tuples with theN th most recent timestamp (while for clarity let us assumethe other N − 1 more recent timestamps are unique). Thenwe must “break the tie” in some fashion to generate exactlyN tuples in the window. We assume such ties are broken ar-bitrarily. Thus, tuple-based sliding windows may be nonde-terministic – and therefore may not be appropriate – whentimestamps are not unique. The special case of N = ∞is specified by [Rows Unbounded] and is equivalent to[Range Unbounded].

Example 6 A tuple-based sliding window does not makemuch sense over stream PosSpeedStr (except the caseof N = ∞) since stream element timestamps arenot unique. For example, at any instant sliding windowPosSpeedStr[Rows 1] denotes the “latest” position–speed measurement, which is ambiguous whenever multiplemeasurements carry the same timestamp – a common occur-rence in the Linear Road application.

6.1.3 Partitioned windows

A partitioned sliding window on a stream S takes a posi-tive integer N and a subset {A1, . . . , Ak} of S’s attributes asparameters. It is specified by following the reference to S inthe query with [Partition By A1,...,Ak Rows N].Intuitively, this window logically partitions S into differentsubstreams based on equality of attributes A1, . . . , Ak (sim-ilar to SQL Group By), computes a tuple-based slidingwindow of size N independently on each substream, thentakes the union of these windows to produce the output rela-tion. More formally, a tuple s with values a1, . . . , ak for at-tributes A1, . . . , Ak occurs in output instantaneous relationR(τ ) iff there exists an element 〈s, τ ′〉 ∈ S, τ ′ ≤ τ such thatτ ′ is among the N largest timestamps of elements whose tu-ples have values a1, . . . , ak for attributes A1, . . . , Ak . Notethat analogous time-based partitioned windows would pro-vide no additional expressiveness over nonpartitioned time-based windows.

Example 7 The partitioned window “PosSpeedStr[Partition By vehicleId Rows 1]” parti-tions stream PosSpeedStr into substreams based onvehicleId and picks the latest element in each sub-stream. (Note that there is no ambiguity in picking the latestelement in each substream since position–speed reportsfor a particular vehicle are made only once in 30 s and

the granularity of Datetime is 1 s.) At any time instantthe relation defined by the window contains the latestspeed–position measurement for each vehicle that has evertransmitted a measurement.

6.1.4 Windows with a “slide” parameter

Windows can optionally contain a slide parameter, indicat-ing the granularity at which the window slides. The slideparameter is a time interval for time-based windows and apositive integer for row-based and partitioned windows.

A time-based window over stream S with window sizeT and slide parameter L is denoted as “S [Range TSlide L]”. Its output relation is:

R(τ ) =

φ if τ < L − 1

{s|〈s, τ ′〉 ∈ S ∧ (τ ′ ≥ τend) ∧ (τ ′ ≤ τstart)}otherwise

,

whereτstart = τ/L� · L

τend = max{τstart − T, 0}.(The expression (τ/L� × L) computes the largest

multiple of L smaller than τ .) Intuitively, “S [Range TSlide L]” defines its output relation by sliding an in-terval of width T time units over S, but the interval slidesonce only every L time units by an amount L . Note thatwe can treat “S [Range T]” as an abbreviation for S[Range T Slide 1], where 1 is the granularity of thetime domain. The following example illustrates the use ofthe slide parameter.

Example 8 “PosSpeedStr[Range 1MinuteSlide1 Minute]” denotes a 1-min window overPosSpeedStr that slides at a 1-min granularity. Atany point in time the window contains the speed–positionreports from the last clock minute. For example, at timeinstant 60 s, the window contains the first minute (1–60) ofspeed–position reports and continues to contain the samebag of tuples until the time instant 120 s, when it shifts tothe next minute (61–120), and so on. This type of windowin which L = T has been referred to as a tumbling windowin previous work [16, 18].

Tuple-based and partitioned windows with a slide pa-rameter are defined analogously.

6.2 Relation-to-relation operators

The relation-to-relation operators in CQL are derived fromtraditional relational queries expressed in SQL, with thestraightforward semantic mapping to time-varying relationsspecified at the beginning of this section. Anywhere a tra-ditional relation is referenced in a SQL query, a (base orderived) relation can be referenced in CQL.

Example 9 Consider the following CQL query for the Lin-ear Road application:

128 A. Arasu et al.

Select Distinct vehicleIdFrom PosSpeedStr [Range 30 Seconds]

This query is composed from a stream-to-relationsliding-window operator, followed by a relation-to-relationoperator that performs projection and duplicate elimination.The output relation of this query contains, at any time in-stant, the set of “active vehicles” – those vehicles that havetransmitted a position–speed measurement within the last30 s.

6.3 Relation-to-stream operators

CQL has three relation-to-stream operators: Istream,Dstream, and Rstream. In the following formal definitions,operators ∪, ×, and − are assumed to be the bag versions.

1. Istream (for “insert stream”) applied to relation Rcontains a stream element 〈s, τ 〉 whenever tuple s is inR(τ ) − R(τ − 1). Assuming R(−1) = φ for notationalsimplicity, we have:

Istream(R) =⋃

τ≥0

((R(τ ) − R(τ − 1)) × {τ }).

2. Analogously, Dstream (for “delete stream”) applied torelation R contains a stream element 〈s, τ 〉 whenever tu-ple s is in R(τ − 1) − R(τ ). Formally:

Dstream(R) =⋃

τ>0

((R(τ − 1) − R(τ )) × {τ }).

3. Rstream (for “relation stream”) applied to relation Rcontains a stream element 〈s, τ 〉 whenever tuple s is in Rat time τ . Formally:

Rstream(R) =⋃

τ≥0

(R(τ ) × {τ }).

A careful reader may observe that Istream andDstream are expressible using Rstream along with time-based sliding windows and some relational operators. How-ever, we retain all three operators in CQL in keeping withgoal 2 from Sect. 1: easy queries should be easy to write.

Example 10 Consider the following CQL query for streamfiltering:

Select Istream(*)From PosSpeedStr [Range Unbounded]Where speed > 65

(Note the syntax of the relation-to-stream operator in theSelect clause.) This query is constructed from three op-erators: an Unbounded window producing a relation thatat time τ contains all speed–position measurements up toτ , a relational filter operator that restricts the relation tothose measurements with speed greater than 65 mph, and

an Istream operator that streams new values in the (fil-tered) relation as a result of the query. The effect is a sim-ple filter over PosSpeedStr that outputs all input ele-ments with speed greater than 65 mph. The same filter querycan be written using the Rstream operator and a Nowwindow:

Select Rstream(*)From PosSpeedStr [Now]Where speed > 65

As we will see shortly, our defaults also permit this query tobe written in its most intuitive form:

Select*From PosSpeedStrWhere speed > 65

Example 11 The following query illustrates the use ofDstream:

Select Dstream(VehicleId)From PosSpeedStr [Range 30 Seconds]

This query is constructed from three operators. The time-based window operator produces the relation containing thespeed–position reports in the previous 30 s. The relation-to-relation operator (Select-From clause) projects thevehicleId attribute from this relation. Finally, theDstream operator produces a vehicleId in the outputwhenever a vehicle is deleted from the above relation. Inother words, the element 〈v, τ 〉 appears in the output streamwhenever vehicle v reported its position and speed at timeτ − 30 but did not do so at time τ . This query thus detectswhen vehicles exit from the highway. (Recall from Sect. 3that a vehicle on a highway reports its speed and positionevery 30 s.)

The Istream operator is used most commonly withUnbounded windows to express filter conditions as shownabove or to stream the results of sliding-window joinqueries. The Rstream operator is used most commonlywith Now windows to express filter conditions as shownabove or to stream the results of joins between streamsand relations, as we will see in Query 6 of Sect. 7. TheDstream operator is used less frequently than Istreamor Rstream; see7 for examples of its use.

6.4 Syntactic shortcuts and defaults

In keeping with goal 2 in Sect. 1, we permit some syntactic“shortcuts” in CQL that result in the application of certaindefaults. Of course there may be cases where the default be-havior is not what the author intended, so we assume thatwhen queries are registered the system informs the authorof the defaults applied and offers the opportunity to edit theexpanded query. There are two classes of shortcuts: omit-ting window specifications and omitting relation-to-streamoperators.



Default windows When a stream is referenced in a CQLquery where a relation is expected (most commonly in theFrom clause), an Unbounded window is applied to thestream by default. While the default Unbounded windowusually produces appropriate behavior, there are cases wherea Now window is more appropriate, e.g., when a stream isjoined with a relation; see Query 6 in Sect. 7.

Default relation-to-stream operators There are two casesin which it seems natural for authors to omit an intendedIstream operator from a CQL query: (1) on the outermostquery, even when streamed results rather than stored resultsare desired [33]; and (2) on an inner subquery, even thougha window is specified on the subquery result.

For the first case we add an Istream operator by de-fault whenever the query produces a relation that is mono-tonic. A relation R is monotonic iff R(τ1) ⊆ R(τ2) when-ever τ1 ≤ τ2. Since we cannot test monotonicity in the gen-eral case, we use a conservative static monotonicity test.For example, a base relation is monotonic if it is knownto be append-only, “S [Range Unbounded]” is mono-tonic for any stream S, and the join of two monotonic re-lations also is monotonic. If the result of a CQL query isa monotonic relation, then it makes intuitive sense to con-vert the relation into a stream using Istream. If it is notmonotonic, the author might intend Istream, Dstream,or Rstream, so we do not add a relation-to-stream operatorby default.

For the second case we add an Istream operator bydefault whenever the subquery is monotonic. If it is not,then the intended meaning of a window specification onthe subquery result is somewhat ambiguous, so a semantic(type) error is generated, and the author must add an explicitrelation-to-stream operator.

Example 12 Now we see why the filter query of Example 10can be written in its most intuitive form:

Select *From PosSpeedStrWhere speed > 65

Since PosSpeedStr is referenced without a windowspecification, an Unbounded window is applied by de-fault. Further, since the output relation of the window andfilter operators is monotonic, we add a default Istreamoperator to the result.

7 Linear Road in CQL

Recall that Linear Road has one base input stream,PosSpeedStr, containing speed-position measurementsof vehicles using the highway. The output is a single streamTollStr(vehicleId,toll) specifying tolls for vehi-cles. Whenever a vehicle with vehicleId v enters a con-gested segment at time τ , TollStr contains the element〈(v,l), τ 〉, where l denotes the toll for the congested seg-ment at time τ .

We incorporate two assumptions suggested in the origi-nal Linear Road specification [4] for computing tolls:

1. A vehicle is considered to have entered a segment whenthe first speed–position measurement for the vehicle istransmitted from that segment. The vehicle is consideredto remain in the segment until it exits (see Assumption 2next) or enters another segment (i.e., a speed–positionmeasurement is transmitted from a different segment).

2. A vehicle is considered to have exited the highway whenno speed–position report for that vehicle is transmittedfor 30 s.

These assumptions are necessary given that each vehicletransmits its speed–position measurement only once every30 s.

Since the continuous query producing TollStr isfairly complex, we express it using several named de-rived relations and streams. Figure 4 shows the derivedrelations and streams that we use and their interdependen-cies. For example, TollStr is produced from derivedstream VehicleSegEntryStr and derived relationsCongestedSegRel and SegVolRel. Our one baseinput stream PosSpeedStr naturally appears as thesource. We present specifications for the derived streamsand relations in topological order according to Fig. 4.For each derived stream and relation, we first describe itsmeaning, followed by the CQL (sub)query that produces it.

Query 1 SegSpeedStr (vehicleId, speed,segNo): This stream is obtained from PosSpeedStrby replacing the xPos attribute of each element withthe corresponding segment number. Since segments areexactly 1 mile long, the segment number is computed by(integer-)dividing xPos by 5,280, the number of feet in amile.

Select vehicleId, speed,xPos/5280 as segNo

From PosSpeedStr

Note the use of a default Unbounded window and a defaultIstream operator in this query.

PosSpeedStr

SegSpeedStr

ActiveVehicleSegRel

VehicleSegEntryStr SegVolRel

TollStr

CongestedSegRel

Fig. 4 Derived relations and streams for Linear Road queries

130 A. Arasu et al.

Query 2 ActiveVehicleSegRel(vehicleId,seg-No): At any instant τ , this relation contains the currentsegments of “active” vehicles, i.e., vehicles currently usingthe highway system.

Select vehicleId, segNoFrom SegSpeedStr [Range 30 Seconds]

Informally, the query uses a time-based window to identifycurrently active vehicles based on Assumption 2 above.8

Query 3 VehicleSegEntryStr(vehicleId,seg-No): A vehicle v entering a segment s at time τ producesan element 〈(v,s), τ 〉 on this stream.

Select Istream(*)From ActiveVehicleSegRel

VehicleSegEntryStr is produced by applying theIstream operator to ActiveVehicleSegRel. A ve-hicle v entering a segment s at time τ causes a new tupleto appear in ActiveVehicleSegRel at τ , which causesthe Istream operator to produce an element 〈(v,s), τ 〉 inVehicleSegEntryStr.

Query 4 CongestedSegRel(segNo): At any instantτ , this relation contains the current set of congested seg-ments. Recall from Sect. 3 that a segment is considered con-gested if the average speed of vehicles in the segment in theprevious 5 min is less than 40 mph.

Select segNoFrom SegSpeedStr [Range 5 Minutes]Group By segNoHaving Avg(speed) < 40

Query 5 SegVolRel(segNo,numVehicles): Thisrelation was introduced in Example 2. At any instant τthis relation contains the current count of vehicles in eachsegment.

Select segNo,count(vehicleId) as numVehicles

From ActiveVehicleSegRelGroup By segNo

Query 6 TollStr(vehicleId,toll): This is thefinal output toll stream.

Select Rstream(E.vehicleId,2 * (V.numVehicles-50)

* (V.numVehicles-50)as toll)

From VehicleSegEntryStr [Now] as E,CongestedSegRel as C,SegVolRel as V

Where E.segNo = C.segNo andC.segNo = V.segNo

8 In this query, we assume that a vehicle does not exit the highwayand reenter within 30 s. We could handle this case by using an addi-tional Partition By window.

At any instant τ the Now window on the streamVehicleSegEntryStr identifies the set of vehicles thathave entered new segments at τ . This set of vehicles isjoined with CongestedSegRel and SegVolRel to de-termine which vehicles have entered congested segments andto compute tolls for such vehicles. Recall from Sect. 3 thatthe toll for a congested segment is given by the formula2 × (numvehicles − 50)2, where numvehicles is the numberof vehicles currently in the segment.

This query provides an example where the defaultUnbounded window would not yield the intended behav-ior if a window specification were omitted. In general, if astream is joined with a relation in order to add attributes toand/or filter the stream, then a Now window on the streamcoupled with an Rstream operator usually provides the de-sired behavior.

Recall that the Linear Road specification in this paperis a simplified version of the original [4]. A CQL speci-fication of the complete Linear Road benchmark as wellas a number of other stream applications, such as net-work monitoring and online auctions [42], is available athttp://www-db.stanford.edu/stream/sqr/.

8 Time management

Recall from Sects. 4 and 5 that our abstract semantics as-sumes a discrete, ordered time domain T . Specifically, ourcontinuous semantics is based on time logically advancingwithin domain T . Conceptually, at time τ ∈ T all inputs upto τ are processed and the output corresponding to τ (streamelements with timestamp τ or instantaneous relation at τ ) isproduced. In this section we briefly discuss how a DSMSmight implement this semantics under realistic conditions.The topic is covered in much more depth in [37].

For exposition, in the remainder of this section let us as-sume that relations are updated via timestamped relationalupdate requests that arrive on a stream. Thus, without lossof generality we can focus on streams only. For a DSMS toproduce output corresponding to a time τ ∈ T , it must haveprocessed all input stream elements at least through τ . Inother words, it must know at some “real” (wall-clock) time tthat no new input stream elements with timestamp ≤ τ willarrive after t . Making this determination is straightforwardwhen all of the input streams are “alive” and their elementsarrive in timestamp order. However, in many stream applica-tions (including the Linear Road), input streams may be gen-erated by remote sources, the network conveying the streamelements to the DSMS may not guarantee in-order transmis-sion, particularly across sources, and streams may pause andrestart.

In the STREAM prototype, our approach is to assumean additional “metainput” to the system called heartbeats.A heartbeat consists simply of a timestamp τ ∈ T and hasthe semantics that after arrival of the heartbeat the systemwill receive no future stream elements with timestamp ≤ τ .


There are various ways by which heartbeats may be gener-ated. Here are three examples:

1. In the easiest case, a fairly common one, timestamps areassigned using the DSMS clock when stream tuples ar-rive at the system. Therefore, stream elements are or-dered, and the clock itself provides the heartbeats.

2. The source of an input stream might generate sourceheartbeats, which indicate that no future elements in thatstream will have a timestamp less than or equal to thatspecified by the heartbeat. If all the sources of inputstreams generate source heartbeats, an application-levelor query-level heartbeat can be generated by taking theminimum of all the source heartbeats. Note that this ap-proach is feasible only if the heartbeats and the streamelements within a single input stream reach the DSMS intimestamp order.

3. Properties of stream sources and the system or network-ing environment may be used to generate heartbeats. Forexample, if we know that all sources of input streamsuse a global clock for timestamping and there is an up-per bound D in delay of stream elements reaching theDSMS, at every global time t we can generate a heart-beat with timestamp t − D.

Heartbeats are also important internally within theSTREAM implementation of CQL in order to communi-cate time-related information among different operators ina query plan. These and other details related to heartbeatgeneration can be found in [37].

8.1 Operations over timestamps

Recall from Sect. 4 that the presence of timestamps is im-plicit, not part of the schema of a stream. Thus CQL doesnot permit direct references to timestamps in queries. Wedecided to make timestamps implicit for the following rea-sons:

1. Timestamps have certain properties (e.g., monotonicity)that we rely on in our semantics (and implementation).Therefore, we cannot permit queries to perform arbitrarytransformations on timestamps, and we decided againstmaking them explicit and monitoring the operations forviolation of these properties.

2. Making timestamps implicit limits the operations per-formed on them, simplifying query plan generation andoptimization.

If an application wishes to pose queries referring totimestamps explicitly, it can do so by simply mirroring thetimestamp attribute in its stream schemas. The following ex-ample illustrates this point.

Example 13 We can add an explicit timestamp at-tribute to PosSpeedStr, resulting in the schemaPosSpeedStr(vehicleId,Speed,xPos,tstamp).When a vehicle v reports its position x and speed s attimestamp τ , the element 〈(v, s, x, τ), τ 〉 arrives on

PosSpeedStr. The following query computes the delaybetween the last two speed–position reports received fromany vehicle:

Select Max(tstamp) - Min(tstamp)From PosSpeedStr [Rows 2]

This query cannot be expressed through implicit timestampsonly.

9 Common constructs

In this section, we illustrate a few constructs we found toappear frequently in CQL queries, primarily based on ourexperience with the Stream Query Repository.9

9.1 Stream filters

A filter over a stream can be expressed in two ways: us-ing an Istream-Unbounded window combination or anRstream-Now window combination. Both of these were il-lustrated in Example 10. Note the Istream-Unboundedwindow combination is the default for streams whose win-dow specification is omitted (recall Sect. 6.4).

9.2 Stream-relation joins

When a stream is joined with a relation, it is usually mostmeaningful to apply a Now window over the stream and anRstream operator over the join result. Consider a streamItem of purchased items and a relation PriceTable ofcurrent item prices. The query

Select Rstream(Item.id,PriceTable.price)

From Item [Now], PriceTableWhere Item.id = PriceTable.itemId

produces the streamed items with their current price ap-pended. Using other types of windows or other relation-to-stream operators usually does not produce “correct” results.For example, the query

Select Istream(Item.id,PriceTable.price)

From Item [Range Unbounded],PriceTable

Where Item.id = PriceTable.itemId

produces, along with new items, the (new) price for all pre-viously purchased items whenever the price for an itemchanges.


132 A. Arasu et al.

9.3 Sliding-window joins

The sliding-window join of two streams is an operation thathas received a great deal of attention, e.g., [10, 22, 38]. Ifboth streams in the join have keys (i.e., there are no duplicatetuples), this type of join can be expressed in CQL using anIstream operator. For example:

Select Istream(*)From S1 [Rows 5], S2 [Rows 10]Where S1.A = S2.A

is a sliding-window natural join of S1 and S2 with a 5-tuplewindow on S1 and a 10-tuple window on S2. A new tupleof S1 produces an output join tuple if it joins with one of thelast 10 tuples of S2; a new tuple of S2 produces an outputjoin tuple in a similar manner.

If either stream can have duplicates, the above query maynot have the expected semantics of a sliding-window join.Suppose a new tuple of S2 is identical to the tuple 10 po-sitions earlier. Then the new S2 tuple does not produce anyjoin result tuples, even if it joins with one of the last 5 tuplesof S1. The more general sliding-window join operation han-dling duplicates can be expressed in CQL, but it is somewhatmore involved.

9.4 Streaming aggregations

Aggregation produces relations, not streams, in CQL sinceaggregations are relation-to-relation operators. We describetwo commonly used ways of streaming an aggregated value:

1. Stream the value of the aggregation whenever it changes:This can be expressed using the Istream operator overthe aggregation. For example:

Select Istream(Count(*))From PosSpeedStr [Range 1 Minute]

counts the speed–position reports over the previousminute and streams the count whenever it changes.

2. Stream the value of the aggregation periodically: Thiscan be expressed using windows with a slide parameterand Istream. For example:

Select Istream(Count(*))From PosSpeedStr [Range 1 Minute

Slide 1 Minute]

streams the number of speed–position reports over thelast minute once every minute. A small subtlety with thisquery is that it will not stream the aggregation value if itremains unchanged from one minute to the next. Ensur-ing the value is streamed even when unchanged requiresa more complex query.

10 Equivalences in CQL

In this section we briefly consider syntactic equivalences inthe CQL language. As in any declarative language, equiv-alences can enable important query-rewrite optimizations;

however, the optimization process itself is not a central topicof this paper.

All equivalences that hold in SQL with standard re-lational semantics carry over to the relational portion ofCQL, including join reordering, predicate pushdown, sub-query flattening, etc. Furthermore, since any CQL query orsubquery producing a relation can be thought of as a materi-alized view that is updated over time, all equivalences frommaterialized view maintenance [26] can be applied to CQL.For example, a materialized view joining two relations gen-erally is maintained incrementally rather than by recomputa-tion, and the same approach can be used to join two relations(or windowed streams) in CQL. In fact, this equivalence isincorporated into our binary-join physical query planoperator (Sect. 12).

Here we consider two equivalences based on streams:window reduction and filter-window commutativity. Theidentification of other useful stream-based syntactic equiv-alences is a topic of future work.

10.1 Window reduction

The following equivalence can be used to rewrite any CQLquery or subquery with an Unbounded window and anIstream operator into an equivalent (sub)query with aNow window and an Rstream operator. Here, L is anyselect-list, S is any stream (including a subquery producinga stream), and C is any condition.

Select Istream(L)From S [Range Unbounded]Where C

≡Select Rstream(L)From S [Now]Where C

Furthermore, if stream S has a key (no duplicates), then weneed not replace the Istream operator with Rstream, al-though once a Now window is applied there is little differ-ence in efficiency between Istream and Rstream.10

Transforming Unbounded to Now obviously sug-gests a much more efficient implementation – logically,Unboundedwindows require buffering the entire history ofa stream, while Now windows allow a stream tuple to be dis-carded as soon as it is processed. In separate work we havedeveloped techniques for transforming Unbounded win-dows into [Rows N] windows, but those transformationsrely on many-one joins and constraints over the streams [12].

We may find other cases or more general criteriawhereby Unbounded windows can be replaced by Nowwindows; a detailed exploration is left as future work.

10 More generally, Istream and Rstream are equivalent over anyrelation R for which R(τ ) ∩ R(τ − 1) = ∅ for all τ .


10.2 Filter-window commutativity

Another equivalence that can be useful for query-rewrite op-timization is the commutativity of selection conditions andtime-based windows. Here, L is any select-list, S is anystream (including a subquery producing a stream), C is anycondition, and T is any time interval.

(Select L From S Where C) [Range T]

≡Select L From S [Range T] Where C

If the system uses a query-evaluation strategy based onmaterializing the windows specified in a query, then filter-ing before applying the window instead of after is preferablesince it reduces steady-state memory overhead [33]. Notethat the converse transformation might also be applied: wemight prefer to move the filtering condition out of the win-dow in order to allow the window to be shared by multi-ple queries with different selection conditions [33]. Finally,note that filters and tuple-based windows generally do notcommute.

11 Comparison with other languages

Now that we have presented our language we can providea more detailed comparison with some of the related lan-guages for continuous queries over streams and relations thatwere discussed briefly in Sect. 2. Specifically, we show thatbasic CQL (without user-defined functions, aggregates, orwindow operators) is strictly more expressive than Tapestry[41], Tribeca [39], GSQL [21], and materialized views overrelations with or without chronicles [29]. We also discussAurora [16], although it is difficult to compare CQL againstAurora because of Aurora’s graphical, procedural nature.

At the end of this section we discuss our choice to definea language based on both relations and streams, instead oftaking a stream-only approach. Included is a discussion ofthe stream-only query language of TelegraphCQ [18].

11.1 Views and chronicles

Any conventional materialized view defined using a SQLquery Q can be expressed in CQL using the same query Qwith CQL semantics.

The chronicle data model (CDM) [29] defines chroni-cles, relations, and persistent views, which are equivalent tostreams, base relations, and derived relations in our termi-nology. For consistency we use our terminology instead oftheirs. CDM supports two classes of operators based on rela-tional algebra, both of which can be expressed in CQL. Thefirst class takes streams and (optionally) base relations as in-put and produces streams as output. Each operator in thisclass can be expressed equivalently in CQL by applying aNow window on the input streams, translating the relational

algebra operator to SQL, and applying an Rstream oper-ator to produce a streamed result. For example, join queryS1 ��S1.A=S2.B S2 in CDM is equivalent to the CQL query:

Select Rstream(*)From S1 [Now], S2 [Now]Where S1.A = S2.B

The second class of operators takes a stream as input andproduces a derived relation as output. These operators can beexpressed in CQL by applying an Unbounded window onthe input stream and translating the relational algebra oper-ator to SQL.

The operators in CDM are strictly less expressive thanCQL. CDM does not support sliding windows over streams,although it has implicit Now and Unbounded windows asdescribed above. Furthermore, CDM distinguishes betweenbase relations, which can be joined with streams, and derivedrelations (persistent views), which cannot. These restrictionsensure that derived relations in CDM can be maintained in-crementally in time logarithmic in the size of the derivedrelation.

11.2 Tapestry

Tapestry queries [41] are expressed using SQL syntax. Attime τ , the result of a Tapestry query Q contains the set oftuples logically obtained by executing Q as a relational SQLquery at every instant τ ′ ≤ τ and taking the set-union ofthe results. This semantics for Q is equivalent to the CQLquery:

Select Istream(Distinct *)From (Istream(Q)) [Range Unbounded]

Tapestry does not support sliding windows over streamsor any relation-to-stream operators.

11.3 Tribeca

Tribeca is based on a set of stream-to-stream operators, andwe have shown that all of the Tribeca operators specified in[39] can be expressed in CQL; details are omitted. Two ofthe more interesting operators are demux (demultiplex) andmux (multiplex). In a Tribeca query the demux operator isused to split a single stream into an arbitrary number of sub-streams, the substreams are processed separately using other(stream-to-stream) operators, and the resulting substreamsare then merged into a single result stream using the muxoperator. This type of query is expressed in CQL using acombination of partitioned window and Group By.

Like chronicles and Tapestry, Tribeca is strictly less ex-pressive than CQL. Tribeca queries take a single stream asinput and produce a single stream as output, with no notionof relation. CQL queries can have multiple input streams andcan freely mix streams and relations.

134 A. Arasu et al.

11.4 Gigascope

GSQL is a SQL-like query language developed for Gigas-cope, a DSMS designed specifically for network monitoringapplications [21]. GSQL is a stream-only language, but re-lations can be created and manipulated using user-definedfunctions. Over streams GSQL’s primary operators are se-lection, join, aggregation, and merge. Constraints on join andaggregation ensure that they are nonblocking: a join opera-tor must contain a predicate involving an “ordered” attributefrom each of the joining streams, and an aggregation opera-tor must have at least one grouping attribute that is ordered.(Ordered attributes are generalizations of CQL timestamps.)

The four primary operations in GSQL can be expressedin CQL: Selection is straightforward. The GSQL Merge op-erator can be expressed using Union in CQL. The GSQLjoin operator translates to a sliding-window join with anIstream operator in CQL. Finally, although it is nontrivialto express GSQL aggregation in CQL (requiring groupingand aggregation, projection, and join), it always is express-ible; details are omitted.

11.5 Aurora

Aurora queries are built from a set of 11 operator types [16].Operators are composed by users into a global query exe-cution plan via a “boxes-and-arrows” graphical interface. Itis somewhat difficult to compare the procedural query inter-face of Aurora against a declarative language like CQL, butwe can draw some distinctions.

The aggregation operators of Aurora (Tumble, Slide,and XSection) are each defined from three user-definedfunctions, yielding nearly unlimited expressive power. Theaggregation operators also have optional parameters set bythe user. For example, these parameters can direct the oper-ator to take certain action if no stream elements have arrivedfor T wall-clock seconds, making the semantics dependenton stream arrival and processing rates.

All operators in Aurora are stream-to-stream, and Au-rora does not explicitly support relations. To express CQLqueries involving derived relations and relation-to-relationoperators, Aurora procedurally manipulates state corre-sponding to a derived relation.

11.6 Stream-only query language

Our abstract semantics and, therefore, CQL distinguish twofundamental data types, relations and streams. At the end ofthis section we outline several motivations for choosing ourdual approach over a purely stream-based approach. Never-theless, it is worth noting that we can always derive a stream-only language Ls from our language L (either CQL or an-other instantiation of our abstract semantics) as follows.

1. Corresponding to each n-ary relation-to-relation operatorO in L , there is an n-ary stream-to-stream operator Os inLs . The semantics of Os(S1, . . . , Sn) when expressed inL is Rstream(O(S1 [Now], . . . , Sn [Now])).

2. Corresponding to each stream-to-relation operator W inL , there is a unary stream-to-stream operator Ws inLs . The semantics of S[Ws] when expressed in L isRstream(S[W ]).

3. There are no operators in Ls corresponding to relation-to-stream operators of L .

L and Ls have essentially the same expressive power.Clearly any query in Ls can be rewritten in L . Given aquery Q in L , we obtain a query Qs in Ls by performingthe following three steps. First, transform Q to an equiv-alent query Q′ that has Rstream as its only relation-to-stream operator (this step is always possible, as indicated inSect. 6.3). Second, replace every input relation Ri in Q′ withRstream(Ri ). Finally, replace every relation-to-relationand stream-to-relation operator in Q with its Ls equivalentaccording to the definitions above. As it turns out, the lan-guage Ls derived from CQL is quite similar to the stream-to-stream approach being taken in TelegraphCQ [18].

We chose our dual approach over the stream-only ap-proach for at least three reasons:

1. Reiterating goal 1 from Sect. 1, we wanted to exploit thewide body of understanding and work on the existing re-lational model to the extent possible.

2. Our experience with a large number of queries [? ] sug-gests that the dual approach results in more intuitivequeries than the stream-only approach. As illustrated inour Linear Road examples (Sect. 7), even applicationswith purely stream-based input and output specificationsmay include fundamentally relational components.

3. Having both relations and streams cleanly generalizesmaterialized views, as discussed in detail in Sect. 11.1.

Note that the chronicle data model [29] discussed inSect. 11.1 also takes an approach similar to ours – it supportsboth chronicles (closely related to streams) and materializedviews (relations).

12 CQL implementation in STREAM

In this section we describe the query plans and execu-tion strategies we use in our implementation of CQL asthe declarative query language in the STREAM prototypeDSMS [1]. Section 12.1 describes the physical representa-tion of streams and relations within query plans, Sect. 12.2describes query plan structure, Sect. 12.5 enumerates the op-erators used in query plans, and Sect. 12.6 briefly discussesquery plan generation. (Details of STREAM’s adaptive ap-proach to query optimization appear in other papers [10, 11,13].) Finally, Sect. 12.7 presents STREAM’s graphical userinterface for viewing, monitoring, and altering query plans.

Overall, STREAM’s query execution strategy is basedlargely on incremental processing: stream-to-relation op-erators transform new elements on their input streams tochanges in their output relations; relation-to-relation oper-ators transform changes in their input relations to changes intheir output relations; relation-to-stream operators transform


changes in their input relations to new elements in their out-put streams. All the components of STREAM’s architecture– the operators, operator state, and internal representationsof relations and streams – are designed to facilitate incre-mental processing. Our execution strategy is somewhat re-lated to work on incremental view maintenance [26, 27], butthere are important differences: in STREAM, timestampsplay an important role in the generation and processing ofincrements, and there are several new operators in STREAMthat are not part of traditional view maintenance.

12.1 Internal representation of streams and relations

Recall the formal definitions of streams and relations fromSect. 4. A stream is a bag of tuple–timestamp pairs, whichcan be represented as a sequence of timestamped tuple “in-sertions.” A relation, which is a time-varying bag of tuples,can also be represented as a sequence of timestamped tuples,except now we have both insertion tuples and deletion tuplesto capture the changing state of the relation.

STREAM exploits this similarity between streams andrelations and uses a common physical representation forboth of them: sequences of tagged tuples. The tuple part con-tains a value for each attribute in the schema of the stream orrelation; the tag contains a timestamp and indicates whetherthe tuple is an insertion or a deletion. The tagged-tuple se-quences are append-only and are always in nondecreasingorder by timestamp. (Base streams and relation updates thatarrive out of timestamp order can be converted into sortedsequences, as described in Sect. 8, and operators always emitsorted sequences.)

12.2 STREAM query plans

When a continuous query specified in CQL is registered withthe STREAM system, it is compiled into a query plan. Thequery plan is merged with existing query plans wheneverpossible in order to share computation and state. Plan gener-ation itself is not a focus of this paper, although we discussit again briefly in Sect. 12.6. Each query plan runs contin-uously and is composed of three different types of compo-nents: operators, queues, and synopses.

12.2.1 Operators

Each query plan operator reads from one or more inputqueues, processes the input based on its semantics, andwrites its output to an output queue. Section 12.5 de-scribes the current set of operators supported in STREAM.Since queues (described next) encode both streams and rela-tions, query plan operators can implement all three operatortypes in our abstract semantics and CQL: stream-to-relation,relation-to-relation, and relation-to-stream.

It is important to distinguish between the logical oper-ators of the previous sections, which are used primarily to

specify precise query semantics, from the physical opera-tors that occur in query plans. As is typical in all DBMSs,physical operators are related to logical operators, but thereexists no one-to-one correspondence between them. In therest of this section, operators refer to physical operators.

12.2.2 Queues

A queue connects its input operator OI to its output oper-ator OO . At any time a queue contains a (possibly empty)sequence representing a portion of a stream or relation inour physical representation as described in Sect. 12.1. Thecontents that pass through the queue over time correspondto the stream or relation produced by OI . The queue buffersthe insertions and deletions in the output of OI until they areprocessed by OO .

12.2.3 Synopses

Synopses store the intermediate state needed by continuousquery plans. In our query plans a synopsis is always “owned”by a single operator O , and the state the synopsis containsis that needed for future evaluation of O . For example, toperform a windowed join of two streams, the join operatormust have access to all tuples in the current window on eachinput stream. Thus a join operator maintains one synopsis(e.g., a hash table) for each of the joined inputs. On the otherhand, operators such as selection and duplicate-preservingunion do not require a synopsis since they do not requiresaved state.

The most common use of a synopsis in our system is tomaterialize the current bag of tuples of a relation, such asthe contents of a sliding window, or of a relation derivedby a subquery. Synopses also may be used to store a sum-mary of the tuples in a stream or a relation for approximatequery answering. Example synopsis types for this purposeare reservoir samples [44] over streams and Bloom filters[15].

12.2.4 Memory management

Currently queues and synopses are stored entirely inmemory, although we are in the process of implementingthe capability for them to spill to disk. A common pool ofmemory is allocated to queues and synopses on demand atthe page granularity. To minimize copying and proliferationof tuples, all tuple data are stored in synopses and are notreplicated. Queues contain references to tuple data withinsynopses, along with tags containing a timestamp and aninsertion/deletion indicator. In addition, some synopsesare simply “stubs” that point to data in other synopses, asdiscussed in Sect. 12.4.

12.3 Example query plans

Figure 5 illustrates a merged STREAM query plan for twocontinuous queries, Q1 and Q2, over input streams S1 and

136 A. Arasu et al.

Syn 1

aggregate Syn4 Syn5binaryjoin

q8

Q2

i-stream

q7q6

q1 q2

q4q3q5

Syn 7

Syn6

Syn3

S1 S1

Syn2

seqwindowS1 S2

seqwindow

Q1

Fig. 5 STREAM continuous query plan for two queries

S2. Query Q1 is a windowed-aggregate query: it maintainsthe maximum value of S1.A for each distinct value of S1.Bover a 50,000-tuple sliding window on stream S1. In CQL:

Q1: Select B, max(A)From S1 [Rows 50,000]Group By B

Query Q2 streams the result of a sliding-window joinover streams S1 and S2. The window on S1 is a tuple-basedwindow containing the last 40,000 tuples, while the windowon S2 is a 10-min time-based window. In CQL:

Q2: Select Istream(*)From S1 [Rows 40,000],

S2 [Range 600 Seconds]Where S1.A = S2.A

The plan contains five operators – seq-windowS1 ,seq-windowS2 , aggregate, binary-join, andi-stream – seven synopses Syn1–Syn7, and eight queuesq1–q8. Operators seq-windowS1 and seq-windowS2 arestream-to-relation, aggregate and binary-join arerelation-to-relation, and i-stream is relation-to-stream.We explain in some detail how each of the five operatorsbehaves.

– Operator seq-windowS1 is a sliding-window operatorthat reads input stream S1’s tuples from input queue q1,11

updates the sliding-window synopsis Syn1, and outputs theinsertions and deletions to this window (which is a re-lation) on both queues q3 and q4. Syn1 always contains

11 In reality, a special operator stream-shepherd handles in-coming streams, placing the elements onto corresponding stream inputqueues; see Sect. 12.5.4.

the 50,000 most recent tuples in stream S1 that seq-windowS1 has processed, which is the larger of the twowindows applied to S1.

– Sliding-window operator seq-windowS2 processes in-put stream S2’s tuples from queue q2. seq-windowS2maintains synopsis Syn2, which contains all tuplesfrom S2 whose timestamps are within the last 600 s.seq-windowS2 outputs insertions and deletions to thiswindow (which is a relation) on queue q5. Specifically,for an S2 tuple s with timestamp τ , seq-windowS2 willproduce an insertion of s into this window with times-tamp τ and a deletion of s from this window with times-tamp τ + 601 s. Recall from Sect. 12.1 that all insertionsand deletions are emitted by the operator in nondecreasingtimestamp order.

– Operator aggregate maintains the maximum value ofS1.A for each distinct value of S1.B in its input rela-tion, which is the 50,000-tuple sliding window on streamS1. The aggregate values change based on insertionsand deletions to the window, which we recall are pro-vided by operator seq-windowS1 on queue q3. Operatoraggregate maintains the current aggregation result insynopsis Syn6, and it outputs insertions and deletions tothis result on queue q6. Because max is not incrementallymaintainable – when the current maximum S1.A for anS1.B value leaves the window, we may need to inspectthe entire window to find the new maximum – operatoraggregate maintains a second synopsis Syn3 with suf-ficient information to maintain the maximums. It turns outthat Syn3 is simply a time-shifted version of Syn1: Syn3contains the same 50,000-tuple window, except it is “be-hind” by the tuples in q3. Instead of duplicating informa-tion, the plan shares tuple data between Syn1 and Syn3, asindicated by the dotted arrow between them in Fig. 5.

– Operator binary-join joins two input relations, whichare sliding windows on streams S1 and S2. binary-join computes the join incrementally using the inser-tions and deletions to these windows provided on queuesq4 and q5, respectively. binary-join processes tuplesin nondecreasing timestamp order across both of its in-puts. That is, each time it is ready to process an in-put tuple it selects the tuple from queue q4 or q5 withthe lowest timestamp. Logically, binary-join main-tains a synopsis for each input: Syn4 and Syn5 for q4 andq5. An insertion (deletion) on q4 is joined with Syn5 tocompute the corresponding insertions (deletions) to thejoin result, which are output on queue q7; similarly forq5. These joins can be nested-loop or index joins de-pending on whether we choose to build indexes on thejoin attributes in Syn4 and Syn5. Logically, Syn4 con-tains the 40,000 most recent tuples in stream S1 thatbinary-join has processed. In our implementation,actual tuple data are shared rather than duplicated betweensynopses Syn4 and Syn1, as indicated by the dotted ar-row between them in Fig. 5. Similarly, the plan sharestuple data between Syn5 and Syn2. Furthermore, the sub-plan rooted at seq-windowS1 is shared by aggregate


and binary-join, thus sharing the window computa-tion on S1 between queries Q1 and Q2. Our query plansalways perform window sharing on a stream, with a sin-gle physically stored synopsis automatically containingall stream elements needed by any window on that stream;see Sect. 12.5.1.

– Operator i-stream converts the relation produced bythe join into a stream of relation insertions using the se-mantics of the Istream operator from Sect. 6.3. Sincei-stream receives the insertions and deletions to thejoin result in queue q7, it might appear that i-streamcan simply pass along all insertions and drop all dele-tions. However, if a tuple happens to be both inserted anddeleted from the join result with the same timestamp, thenfor correct semantics i-stream must detect this caseand not pass along the insertion. (This case can occur, forexample, when the window on S1 slides and the tuple val-ues that enter and leave the window happen to be iden-tical.) To handle this case correctly, i-stream buffersinput insertions for a timestamp τ in synopsis S7 until itknows that it will see no further tuples with that timestamp(recall Sect. 8).

It is important for timestamps on the tuples output by aquery operator to properly reflect our semantics as specifiedearlier in this paper. In most operators, one or more outputtuples are produced as the result of an input tuple being con-sumed, and it turns out that our semantics is maintained cor-rectly by copying the input tuple’s timestamp to the outputtuple. For example, when an input tuple with timestamp τcauses a tuple-based window to slide, both the insert and thedelete generated by the slide are timestamped τ . When joinresults are produced from a new input tuple with timestampτ , the new join results have timestamp τ , which always is thelater of the timestamps on the two joining tuples since weprocess join inputs in nondecreasing timestamp order acrossinputs. Aggregation is similar: a new input tuple with times-tamp τ produces one or more insertions and deletions to theaggregate result also timestamped τ . Time-based windowsare trickier, as illustrated above. For a time-based windowof size T , a stream tuple s with timestamp τ generates an in-serted tuple with timestamp τ and generates a deleted tuplewith timestamp τ + T + 1, where the timestamp arithmetic,and particularly the meaning of “+1,” depends on the timedomain to which the timestamps belong. Recall from Sect. 4that our semantics assumes a discrete and ordered time do-main, so here “+1” denotes an increment in this domain,e.g., 1 s in the Datetime domain.

The execution of query plans is controlled by a globalscheduler, which currently runs in the same thread as allof the query operators in the system. Each time the sched-uler is invoked, it selects an operator to execute and calls aspecific procedure defined for that operator, passing as a pa-rameter the maximum number of input tuples that the opera-tor should process before returning control to the scheduler.We currently use a simple round-robin scheduler, but we in-tend to incorporate a more appropriate scheduling algorithmbased on recent research [7].

12.4 A note on synopses

From the example STREAM query plan of Fig. 5 we see thatour query plans tend to be overloaded with synopses. Forexample, the plan for the windowed-join query Q2 in Fig. 5uses five synopses – one each for the two seq-windowoperators, two for the binary-join operator, and one forthe i-stream operator. Our technique of generating nu-merous synopses made it much easier to implement a plan-generation algorithm that works for arbitrary CQL queries.We could add a postprocessing step that traverses queryplans and physically merges some synopses. However, ourapproach so far has been to leave all the synopses in placebut make many of them logical “stubs” that primarily pointto other synopses. For example, the plan for query Q2 inFig. 5 materializes only three synopses instead of five sincethe two synopses of the binary-join operator are sharedwith the synopses of the corresponding seq-window op-erators. Synopses Syn4 and Syn5 in Fig. 5 are stubs pointingto synopses Syn1 and Syn2, respectively.

Currently, we have implemented only simple strategiesfor sharing synopsis state across different operators. Asdescribed above, our most common strategy is to sharestate between an operator that produces a relation (e.g., asliding window) and an operator that accesses the relation(e.g., a join over the window). In related work [6], wepresent more sophisticated strategies for sharing state,specifically algorithms for sharing state across differentaggregations over sliding windows. These algorithms couldbe incorporated into the STREAM prototype, although wehave not yet done so.

12.5 STREAM query operators

In this section we specify all of the query operators currentlyimplemented in the STREAM prototype and used in queryplans. Every operator is either a data operator or a systemoperator. Data operators are the main data-processing units.They can be categorized into three classes – stream-to-relation, relation-to-relation, and relation-to-stream – justlike the logical operators of CQL. However, we note againthat all operators perform incremental processing, so ef-fectively they are all performing a type of stream-to-streamprocessing.

All five operators discussed in the previous section aredata operators. System operators isolate the data operatorsfrom lower-level issues such as asynchronous and out-of-order arrival of streams, load shedding [9, 40], and externalstream data formats. The operators are listed in Table 1.

12.5.1 Stream-to-relation operators

The sliding-window operator seq-window is the onlystream-to-relation operator implemented in STREAM. Itsupports the tuple-based, time-based, and partitioned win-dow specifications introduced in Sect. 6.1. (STREAM does

138 A. Arasu et al.

Table 1 Operators in STREAM query plans

Name Operator Type Description

seq-window Stream-to-relation Implements time-based, tuple-based, and partitioned windowsselect Relation-to-relation Filters tuples based on predicate(s)project Relation-to-relation Duplicate-preserving projectionbinary-join Relation-to-relation Joins two input relationsmjoin Relation-to-relation Multiway join from [43]union Relation-to-relation Bag unionexcept Relation-to-relation Bag differenceintersect Relation-to-relation Bag intersectionantisemijoin Relation-to-relation Antisemijoin of two input relationsaggregate Relation-to-relation Performs grouping and aggregationduplicate-eliminate Relation-to-relation Performs duplicate eliminationi-stream Relation-to-stream Implements Istream semanticsd-stream Relation-to-stream Implements Dstream semanticsr-stream Relation-to-stream Implements Rstream semanticsstream-shepherd System operator Handles input streams arriving over the networkstream-sample System operator Samples specified fraction of tuplesstream-glue System operator Adapter for merging a stream-producing view into a planrel-glue System operator Adapter for merging a relation-producing view into a planshared-rel-op System operator Materializes a relation for sharingoutput System operator Sends results to remote clients

not yet support windows with a slide parameter, as describedin Sect. 6.1.4.) As illustrated in Fig. 5, STREAM instantiatesone sliding-window operator, denoted seq-windowS , foreach stream S that has at least one continuous query specify-ing a window over S. Operator seq-windowS has one in-put queue providing the tuples in S in nondecreasing times-tamp order. For each sliding window on S used in a queryplan, seq-windowS maintains an output queue containingthe insertions and deletions to that window. By using a singlewindow operator per stream S, the system is able to share thecomputation and memory required for window maintenanceacross all queries referencing S.

12.5.2 Relation-to-relation operators

Each relation-to-relation operator processes insertions anddeletions in timestamp order from one or more input queues,computes its output incrementally, and writes the output in-sertions and deletions to an output queue in timestamp or-der. As listed in Table 1, STREAM supports relation-to-relation operators corresponding to all standard relationaloperators. Notice that STREAM supports two join opera-tors: binary-join, a binary join operator as illustrated inFig. 5, and mjoin, the multiway join operator proposed in[43]. Consequently, a multiway join can be processed in twoways – using mjoin, which does not materialize interme-diate results [43], or using a tree of binary joins. Decidingwhich strategy to use is a query-optimization issue not cov-ered in this paper.

12.5.3 Relation-to-stream operators

STREAM supports three relation-to-stream operators –i-stream, d-stream, and r-stream – correspondingto the Istream, Dstream, and Rstream operators de-fined in Sect. 6.3. Each operator processes insertions and

deletions in timestamp order from its single input queue andwrites the output stream insertions to an output queue intimestamp order. The i-stream operator was describedin Sect. 12.3, and the d-stream operator is very similar.The r-stream operator maintains the entire current stateof its input relation in a synopsis and outputs all of the tu-ples as insertions at each time step. While this may appearexpensive, recall from Sect. 6.3 that the Rstream operatoris used most commonly with Now windows, so the “currentstate of the relation” is generally very small.

12.5.4 System operators

The primary purpose of the system operators in STREAMis to isolate the data operators from various lower-level is-sues. For completeness and for understanding the STREAMquery plans in Sect. 12.7, we briefly discuss some of theseoperators. The stream-shepherd operator for a streamS serves as the source of S to all query plans accessing S:primarily it receives tuples arriving asynchronously over thenetwork, transforms them to STREAM’s internal represen-tation, and writes them to the appropriate input queues. Inthe future, this operator also will be responsible for buffer-ing input tuples for proper ordering, generating heartbeatswhen source applications don’t provide them [37], and per-forming load shedding under overload [9].

The stream-sample operator currently is used onlyfor system-managed load shedding [9], although we also in-tend to implement a sample clause in our query language[8]. stream-sample drops a specified fraction of streamtuples from its input queue based on a uniform random sam-ple (“coin toss”). Other implemented system operators serveas materialization points for relations (shared-rel-op),enable plans for views to be merged into new query plans(rel-glue and stream-glue), and send query resultsto remote clients (output).


Fig. 6 Plan for aggregate query over join

12.6 Query plan generation

Most of the CQL language is operational in the STREAMsystem. Our query plan generator is fairly simple, usinghard-coded heuristics to generate initial query plans. Ourcurrent approach is to generate simple plans, which are thenmonitored and possibly restructured automatically usingSTREAM’s adaptivity component, StreaMon [10, 11, 13].Given the promise of this approach, currently we do not ex-pect to build a complex query optimizer for plan generation.

However, we do apply heuristics to generate our initial sim-ple plan, including:

1. Push selections below joins.2. Maintain and use indexes for synopses on

binary-join, mjoin, and aggregate opera-tors.

3. Share synopses and operators within query plans when-ever possible.12

12 So far the sharing techniques implemented in our systemare simple, but this is an important area of future research. For

140 A. Arasu et al.

Fig. 7 Plan for TollStr query

We are actively moving toward one-time and dynamiccost-based optimization of CQL queries. Since CQL usesSQL as its relational query language, we can also leveragemany of the one-time optimization techniques used in tradi-tional relational systems. In addition, we are exploring adap-tive query optimization techniques that are coarser-grainedthan Eddies (as used in the Telegraph project [18]). Our ap-proach relies on two interacting components: a monitor that

example, more sophisticated techniques such as [6, 20, 32] should beapplicable.

captures properties of streams and system behavior and anoptimizer that can reconfigure query plans and resource al-location as properties change over time.

12.7 Example STREAM plans

Lastly, we present two snapshots of query plans taken fromSTREAM’s graphical query plan visualizer. Through thevisualization interface users can inspect the plan generatedfor a continuous query as soon as the query is registered,


monitor plan behavior during execution, and even alter planstructure and attributes of plans such as memory allocationfor the purpose of experimentation.

Figure 6 shows the query plan for the following simpleillustrative CQL query over streams S1 and S1:

Select S2.name, max(S1.num)From S1 [Rows 50,000],

S2 [Rows 50,000]Where S1.name <= ’i’ and

S1.num = S2.numGroup By S2.name

This query is a windowed two-way join with a fil-ter predicate on S1, followed by an aggregation. Thesystem operators used in the query plan in Fig. 6 arestream-shepherd operators for streams S1 and S2 andan output operator to send the query result continuouslyto the client that submitted the query. The data operators areseq-window for S1 and S2, select, binary-join,aggregate, and (duplicate-preserving) project. No-tice that synopses are shared between the window operatorsand the join, and the selection has been pushed below thejoin. The selection cannot be pushed below the window op-erator since in general tuple-based windows and selectiondo not commute. The binary-join operator material-izes its output relation in a synopsis that is shared with theaggregate operator, similar to our example in Sect. 12.3.

Figure 7 shows the complete query plan for theTollStr Linear Road query from Sect. 7, which incorpo-rates plans for all subqueries that were used to write thisquery in CQL. We do not expect readers to examine everydetail of this complex query plan.

13 Conclusions

This paper specified the CQL language for continuousqueries over data streams, including a formal abstract se-mantics on which CQL is based. Most of the CQL languageis operational in the STREAM prototype data stream man-agement system, including the “Linear Road” benchmarkused as examples throughout this paper. This paper also de-scribes the structure and implementation of query executionplans for CQL in the STREAM system.

The STREAM system is available for experimentationover the Internet: for each user, a dedicated server is startedon a machine at Stanford and a client is started on the user’smachine. Through the graphical user interface depicted inFigs. 6 and 7, users may register streams and continuousqueries and view the streamed (or stored) results. Users mayalso inspect and alter query plans and perform visual sys-tem monitoring through “introspection”: query componentswrite statistics (such as throughput, selectivity, etc.) ontoa special system stream. Graphical system monitors obtaintheir plotted values by registering standard CQL queries onthe special system stream.

Please try out the system, available from theSTREAM home page at http://www-db.stanford.edu/stream/.

Acknowledgements We are especially grateful to Keith Ito for hissignificant contributions to implementing the STREAM prototype.We are also grateful to STREAM participants Brian Babcock, JohnCieslewicz, Mayur Datar, Rajeev Motwani, Itaru Nishizawa, UtkarshSrivastava, and others who have influenced the development of theCQL language and STREAM query processor.

This work was supported by the National Science Foundationunder grants IIS-0118173 and IIS-9817799, and by a 3Com StanfordGraduate Fellowship.

References

1. Arasu, A., Babcock, B., Babu, S., Datar, M., Ito, K., Nishizawa,I., Rosenstein, J., Widom, J.: STREAM: The Stanford StreamData Manager. In: Proceedings of the 2003 ACM SIGMODInternational Conference on Management of Data, p. 665 (2003)[demonstration description]

2. Arasu, A., Babcock, B., Babu, S., McAlister, J., Widom, J.:Characterizing memory requirements for queries over continuousdata streams. ACM Trans. Database Syst. 29(1), 162–194 (2004)

3. Arasu, A., Babu, S., Widom, J.: CQL: A language for continuousqueries over streams and relations. In: 9th Interantional Workshopon Database Programming Languages, pp. 1–11 (2003)

4. Arasu, A., Cherniak, M. et al.: Linear road: a stream datamanagement benchmark. In: Proceedings of the 30th Interna-tional Conference on Very Large Data Bases, pp. 480–491 (2004)

5. Arasu, A., Widom, J.: A denotational semantics for continuousqueries over streams and relations. SIGMOD Rec. 33(3), 6–12(2004)

6. Arasu, A., Widom, J.: Resource sharing in continuous sliding-window aggregates. In: Proceedings of the 30th InternationalConference on Very Large Data Bases, pp. 336–347 (2004)

7. Babcock, B., Babu, S., Datar, M., Motwani, R.: Chain: Operatorscheduling for memory minimization in data stream systems.In: Proceedings of the 2003 ACM SIGMOD InternationalConference on Management of Data, pp. 253–264 (2003)

8. Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.:Models and issues in data stream systems. In: Proceedings ofthe 21st ACM SIGACT-SIGMOD-SIGART Symposium onPrinciples of Database Systems, pp. 1–16 (2002)

9. Babcock, B., Datar, M., Motwani, R.: Load shedding foraggregation queries over data streams. In: Proceedings of the20th International Conference on Data Engineering, pp. 350–361(2004)

10. Babu, S., Motwani, R., Munagala, K., Nishizawa, I., Widom,J.: Adaptive ordering of pipelined stream filters. In: Proceed-ings of the 2004 ACM SIGMOD International Conference onManagement of Data, pp. 407–418 (2004)

11. Babu, S., Munagala, K., Widom, J., Motwani, R.: Adaptivecaching for continuous queries. In: Proceedings of the 21stInternational Conference on Data Engineering (2005)

12. Babu, S., Srivastava, U., Widom, J.: Exploiting k-constraints toreduce memory overhead in continuous queries over data streams.ACM Trans. Database Syst. 31(3) (2004)

13. Babu, S., Widom, J.: StreaMon: an adaptive engine for streamquery processing. In: Proceedings of the 2004 ACM SIG-MOD International Conference on Management of Data (2004)[demonstration description]

14. Daniel, B.: The characterization of continuous queries. Int. J.Coop. Inf. Syst. 8(4), 295–323 (1999)

15. Bloom, B.: Space/time trade-offs in hash coding with allowableerrors. Comm. ACM 13(7), 422–426 (1970)

142 A. Arasu et al.

16. Carney, D., Centintemel, U. et al.: Monitoring streams—Anew class of data management applications. In: Proceedings ofthe 28th International Conference on Very Large Data Bases,pp. 215–226 (2002)

17. Carney, D., Centintemel, U., Rasin, A., Zdonik, S., Cherniack, M.,Stonebraker, M.: Operator scheduling in a data stream manager.In: Proceedings of the 29th International Conference on VeryLarge Data Bases, pp. 838–849 (2003)

18. Chandrasekharan, S., Cooper, O. Deshpande, A., Franklin,M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S.,Raman, V., Reiss, F., Shah, M.: TelegraphCQ: continuous dataflowprocessing for an uncertain world. In: Proceedings of the 1st Con-ference on Innovative Data Systems Research, pp. 269–280 (2003)

19. Chandrasekharan, S., Franklin, M.J.: Streaming queries overstreaming data. In: Proceedings of the 28th InternationalConference on Very Large Data Bases, pp. 203–214 (2002)

20. Chen, J., DeWitt, D.J., Tian, F., Wang, Y.: NiagaraCQ: a scalablecontinuous query system for internet databases. In: Proceedingsof the 2000 ACM SIGMOD International Conference onManagement of Data, pp. 379–390 (2000)

21. Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.:Gigascope: a stream database for network applications. In:Proceedings of the 2003 ACM SIGMOD International Confer-ence on Management of Data, pp. 647–651 (2003)

22. Das, A., Gehrke J., Reidewald, M.: Approximate join processingover data streams. In: Proceedigs of the 2003 ACM SIGMODInternational Conference on Management of Data, pp. 40–51(2003)

23. Dobra, A., Garofalakis, M.N., Gehrke, J., Rastogi, R.: Processingcomplex aggregate queries over data streams. In: Proceed-ings of the 2002 ACM SIGMOD International Conference onManagement of Data, pp. 61–72 (2002)

24. Gehrke, J.: Special issue on data stream processing. IEEEComput. Soc. Bull. Tech. Commun. Data Eng. 26(1) (2003)

25. Golab, L., Ozsu, M.T.: Issues in data stream management.SIGMOD Rec. 32(2), 5–14 (2003)

26. Gupta, A., Mumick, I.S.: Maintenance of materialized views:Problems, techniques, and applications. IEEE Comput. Soc. Bull.Tech. Commun. Data Eng. 18(2), 3–18 (1995)

27. Gupta, A., Mumick, I.S., Subramanian, V.S.: Maintaining viewsincrementally. In: Proceedings of the 1993 ACM SIGMODInternational Conference on Management of Data, pp. 157–166(1993)

28. Hammad, M.A., Franklin, M.J., Aref, W.G., Elmagarmid, A.K.:Scheduling for shared window joins over data streams. In:Proceedings of the 29th International Conference on Very LargeData Bases, pp. 297–308 (2003)

29. Jagadish, H.V., Mumick, I.S., Silberschatz, A.: View maintenanceissues for the chronicle data model. In: Proceedings of the 14thACM SIGACT-SIGMOD-SIGART Symposium on Principles ofDatabase Systems, pp. 113–124 (1995)

30. Law, Y.-N., Wang, H., Zaniolo, C.: Query languages and datamodels for database sequences and data streams. In: Proceedingsof the 30th International Conference on Very Large Data Bases,pp. 492–503 (2004)

31. Liu, L., Pu, C., Tang, W.: Continual queries for internet scaleevent-driven information delivery. IEEE Trans. Knowl. Data Eng.11(4), 610–628 (1999)

32. Madden, S., Shah, M.A., Hellerstein, J.M., Raman, V.:Continuously adaptive continuous queries over streams. In:Proceedings of the 2002 ACM SIGMOD International Confer-ence on Management of Data, pp. 49–60 (2002)

33. Motwani, R., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar,M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Queryprocessing, approximation, and resource management in a datastream management system. In: Proceedings of the 1st Conferenceon Innovative Data Systems Research, pp. 245–256 (2003)

34. Nguyen, B., Abiteboul, S., Cobena, G., Preda, M.: MonitoringXML data on the web. In: Proceedings of the 2001 ACMSIGMOD International Conference on Management of Data,pp. 437–448 (2001)

35. Özsoyoglu, G., Snodgrass, R.T.: Temporal and real-timedatabases: a survey. IEEE Trans. Knowl. Data Eng. 7(4), 513–532(1995)

36. Seshadri, P., Livny, M., Ramakrishnan, R.: SEQ: A model forsequence databases. In: Proceedings of the 11th InternationalConference on Data Engineering, pp. 232–239 (1995)

37. Srivastava, U., Widom, J.: Flexible time management in datastream systems. In: Proceedings of the 23rd ACM SIGACT-SIGMODSIGART Symposium on Principles of DatabaseSystems, pp. 263–274 (2004)

38. Srivastava, U., Widom, J.: Memory-limited execution ofwindowed stream joins. In: Proceedings of the 30th InternationalConference on Very Large Data Bases, pp. 324–335 (2004)

39. Sullivan, M.: Tribeca: A stream database manager for networktraffic analysis. In: Proceedings of the 22nd InternationalConference on Very Large Data Bases, p. 594 (1996)

40. Tatbul, N., Cetintemel, U., Zdonik, S., Cherniack, M.,Stonebraker, M.: Load shedding in a data stream manager.In: Proceedings of the 2003 International Conference on VeryLarge Data Bases, pp. 309–320, 2003

41. Terry, D.B., Goldberg, D., Nichols, D., Oki, B.M.: Continuousqueries over append-only databases. In: Proceedings of the 1992ACM SIGMOD International Conference on Management ofData, pp. 321–330 (1992)

42. Tucker, P.A., Tufte, K., Papadimos, V., Maier, D.:NEXMark – A benchmark for querying data streams,2002. Available at: http://www.cse.ogi.edu/dot/niagara/NEXMark/

43. Viglas, S., Naughton, J.F., Burger, J.: Maximizing the output rateof multi-way join queries over streaming information sources. In:Proceedings of the 29th International Conference on Very LargeData Bases, pp. 285–296 (2003)

44. Vitter, J.: Random sampling with a reservoir. ACM Trans. Math.Softw. 11(1), 37–57 (1985)

45. Wang, H., Zaniolo, C.: ATLaS: a native extension of sql fordata mining. In: Proceedings of the 3rd SIAM InternationalConference on Data Mining (2003)

46. Wang, H., Zaniolo, C., Luo, C.: ATLaS: A small but completesql extension for data mining and data streams. In: Proceedingsof the 29th International Conference on Very Large Data Bases,pp. 1113–1116 (2003) [demonstration description]

47. Widom, J., Ceri, S. (eds.): Active database systems: triggers andrules for advanced database processing. San Francisco: MorganKaufmann (1996)

the cql continuous query language: semantic foundations ... · components of our abstract...

Documents