Memory Requirements of Data Streams

Reynold Cheng

19th July, 2002

Data Stream ModelUser/ApplicationUser/ApplicationUser/ApplicationUser/Application

Register QueryRegister Query

Stream QueryStream QueryProcessorProcessor

ResultsResults

Data Stream ModelUser/ApplicationUser/ApplicationUser/ApplicationUser/Application

Register QueryRegister Query

Stream QueryStream QueryProcessorProcessor

ResultsResults

Scratch SpaceScratch Space(Memory and/or Disk)(Memory and/or Disk)

DataStream

ManagementSystem

(DSMS)

Impact of Limited Memory

Continuous streams grow unboundedly Question: Can a continuous query be

evaluated using a finite amount of memory?

Query Model

SPJ (Select-Project-Join) Queries L(P(S1 x S2 x … x Sn)) L: List of projected attributes P: Selection Predicate S1, S2,…, Sn: Input Streams : duplicate-eliminating; ’: duplicate-

preserving

Query Model: Predicate P

P: conjunction of atomic predicates Atomic Predicate

Si.A Op Sj.B (i = j or i != j)

Si.A Op k

Op: {<, =, >}

Query Model: Attributes

All attributes have discrete, ordered domains All attributes are of type integer

Query Model: Monotonic and Exact Answers

The queries are monotonic Any tuple that appears in the answer at any

point in time continues to do so forever. The answers produced are exact i.e., no

approximation.

Motivating Examples

When does a SPJ query require only a bounded amount of memory?

Assume 2 streams: S(A, B, C) and T(D, E)

Query 1

’A(A>10(S)) Is a simple filter on S Tuple-at-a-time processing No extra memory for saving stream tuples

Query 1 (again)

A(A>10(S)) Keep track of each distinct value of A > 10 To eliminate duplicates Requires unbounded memory

Query 2

A(A=D(S x T)) Save each tuple s in S, since s may join with

any tuples in T that arrive in the future Also need to save each tuple in T Requires unbounded memory

Query 3

’A(A=D ^ A > 10 ^ D < 20 (S x T)) Can be evaluated with bounded memory! For each integer v in [11,19], keep:

current # of tuples in S with A = v (v.S) current # of tuples in T with D = v (v.T)

For an incoming tuple from stream S with A = v, output v for v.T times.

For an incoming tuple from stream T with D = v, output v for v.S times.

Query 4

A(B < D ^ A > 10 ^ A < 20 (S x T)) Can be evaluated with bounded memory! For each integer v in [11,19], keep:

Current min value of B among all tuples in S with A = v (v.B_min)

Current max value of D among all tuples in T with A = v (v.D_max)

For an incoming tuple from stream S, check: S.B < v.D_max.

For an incoming tuple from stream T, check: T.D > v.B_min.

Our Goals

Some queries involving join can be answered using finite amount of memory

Under what conditions can a query be answered using finite memory over all possible instances of streams?

Identify a class of queries that can be evaluated with a bounded amount of memory

Consider both duplicate-preserving and duplicate-eliminating projections

Bounded Memory Computability

An SPJ query is bounded-memory computable if we can find: A constant M and, An algorithm that evaluates the query using fewer

than M units of memory A unit of memory can store one attribute

value or a count

Memory boundness testing (Outline)

Rewrite Q as a union of Locally Totally Ordered Queries (LTO queries), based on the following theorem:

Any query Q can be rewritten as Q1 U Q2 U … U Qm, where each Qi is an LTO query and unions are duplicate-preserving

Memory boundness testing (Outline)

Q is computable in bounded memory if and only if all its decomposed LTO queries are computable in bounded memory.

Key: Key: Develop a theorem for checking the Develop a theorem for checking the memory boundness of a LTO querymemory boundness of a LTO query

Definitions

Can write a query Q as Q(P), when only P is important to discussions

Element: constant / attribute of a stream C(Q): Set of constants appearing in Q S(Q): Set of streams appearing in Q A(S): Set of attributes in stream S (S) = A(S) U C(Q): set of elements in Q

potentially relevant to stream S

Total Ordering

P+: transitive closure of P A = 10 and B < A => B < 10

Set of elements E is totally ordered by a set of predicates P if for any elements e1 and e2 in E, Exactly one of e1<e2, e1=e2, or e1>e2 is in P+

Order-Inducing Predicates

A set of predicates P is order-inducing if a set of elements E is totally ordered by P.

TO(E): Set of all order-inducing sets of predicates for elements E

Example E = {A,B,5} Order-inducing sets: {A < B, 5 < A}, {A = B, B < 5}. These two sets belong to TO(E).

LTO Query

A query Q(P) is LTO query if for every S S(Q), (S) is totally ordered by P Where S(Q): Set of streams appearing in Q (S) = A(S) U C(Q): set of elements in Q

potentially relevant to stream S

Decomposition of a Query into LTO Queries

Q = ’L(P(S1 x S2 x … x Sn)) can be decomposed into Q1 U Q2 U … U Qm, where Qi is an LTO query.

Let TO((Si)) = {Ti1, Ti

2,…, Timi},

Tij is a local total ordering – one possible

ordering of Si and query constants.

Decomposition Theorem

An exhaustive union of m1 x m2 x mn queries:

Q(P U T11 U T2

1 U … U Tn1) U

Q(P U T12 U T2

1 U … U Tn1) U

… U

Q(P U T11 U T2

2 U … U Tn1) U

Q(P U T12 U T2

2 U … U Tn1) U

… U

Q(P U T1m1 U T2

m2 U … U Tnmn)

Boundedness of Attributes

An attribute A is lower-bounded by a set of predicates P if there is an atomic predicate A > k P+ for some constant k

An attribute A is upper-bounded by a set of predicates P if there is an atomic predicate A < k P+ for some constant k

An atomic attribute is bounded if it is both upper-bounded and lower-bounded

An attribute is unbounded if it is not bounded

MaxRef and MinRef

MaxRef(Si) is the set of all unbounded attributes A of Si that participate in join of the form Sj.B < Si.A, i j

MinRef(Si) is the set of all unbounded attributes A of Si that participate in join of the form Si.A < Sj.B, i j

Boundness Testing of LTO Query (Duplicate-Preserving)

Let Q = ’L(P(S1 x S2 x … x Sn)) be an LTO query where n > 1. Q is bounded memory computable iff:

1. Every attribute in L is bounded

2. For every equality join predicate Si.A=Sj.B where i j, Si.A and Sj.B are both bounded

3. |MaxRef(Si)| = |MinRef(Si)| = 0 for i = 1,…,n

Proof Outline

We create synopses of n data streams For each stream Si, partition the tuples that

satisfy the total order condition on Si into distinct buckets based on the values of the bounded attributes

Tuples with the same values of all bounded attributes are placed in the same bucket

Tuples that differ on at least one bounded attribute are placed in different buckets

Proof Outline (2)

For each bucket, store The values for the bounded attributes Total number of tuples falling into the bucket

The total size of these synopses is bounded by constant

Can be proved that if the 3 conditions are satisfied, these synopses suffice to evaluate the query.

Proof (If):

Attributes not in project list / join condition can be ignored, because all local selection conditions must be implied by the total order

C1 and C2 guarantee that all attributes in equijoin and project list are bounded

Each synopsis maintains full information about the values of all bounded attributes for every tuple

Equijoin and projections can be handled properly

Proof (If):

C3 asserts For every Si.A < Sj.B, i j, either one is true:

1. Both attributes are bounded --- Synopses have full information for A & B

2. The total order implies Si.A < c < Sj.B where c is some constant in Q --- No need to store actual attribute values, since all tuples from Si that satisfy the total order on Si join with all tuples from Sj that satisfy the total order on Sj

No relevant information is lost by consolidating tuples into buckets

Proof (Only if):

If one of the conditions C1, C2 and C3 does not hold, then Q cannot always be evaluated in bounded memory.

For each condition, if the condition is violated, one can construct instances and presentations of the input streams that require more than M memory to correctly evaluate Q.

Boundness Testing of LTO Query (Duplicate-Eliminating)

Let Q = L(P(S1 x S2 x … x Sn)) be an LTO query where n > 1. Q is bounded memory computable iff: Every attribute in L is bounded For every equality join predicate Si.A=Sj.B where i

j, Si.A and Sj.B are both bounded

|MaxRef(Si)|eq + |MinRef(Si)|eq 1 for i = 1,…,n

|E|eq denotes the number of P-induced equivalence classes in the element set E.

Comments

Q is computable in bounded memory if and only if all its decomposed LTO queries are computable in bounded memory.

The checking algorithm needs O(|A(Q)|4) times. If a query is not memory-bounded computable,

approximation algorithms are necessary. For a memory-bounded computable query,

computation is costly in the evaluation of each LTO query. Need to evaluate the query for every arrival of tuples.

Comments

Only spatial requirement, not query speed, is considered.

Can we extend the memory-boundness checking to general (including aggregate) queries?

It is not always possible to provide exact answers to queries, so approximation query algorithms have to be developed.

For these approximation queries, need to consider memory-boundness, execution speed and approximation quality.

Outline of this Talk

Memory Requirements of QueriesMemory Requirements of Queries Approximation QueriesApproximation Queries Other Research Issues

Approximate Query Evaluation Why? Handling load – streams coming too fast Data stream is archived in a off-site data

warehouse, expensive access of archived data

Avoid unbounded storage and computation Ad hoc queries need approximate history Try to look at the data items only once and

in a fixed order

Approximate Query Evaluation

How? Sliding windows, synopsis, samples Major Issues?

Metric for set-valued queries Composition of approximate operators How is it understood/controlled by user? Integrate into query language Query planning and interaction with resource

allocation Accuracy-efficiency-storage tradeoff and global

metric

Synopses

Queries may access or aggregate past data Need bounded-memory history-approximation Synopsis?

Succinct summary of old stream tuples Like indexes/materialized-views, but base data is unavailable

Examples Sliding Windows Samples Sketches Histograms Wavelet representation

Sketching Techniques

Self-Join Size Estimation Stream of values from D = {1,2,…,n} Let fi = frequency of value i Consider S = Σ fi

2, or Gini’s index of homogeneity.

Useful in parallel DB applications, error estimation in query result size estimation and access plan costs.

Equivalent query: count (R |><|D R)

Evaluating S = Σ fi2

To update S, keep a counter fi for each value i in the domain D (n) space

Has to be kept for each self-join Question – estimating S in sub-linear space?

(O(log n))

Self-Join Size Estimation

AMS Technique (randomized sketches) Given (f1,f2,…,fN)

Zi = random{-1,1}

X = Σ fiZi (X incrementally computable)

Theorem Exp[X2] = Σ fi2

Cross-terms fiZi fjZj have 0 expectation

Square-terms fiZi fiZi = fi2

Space = log (N + Σ fi)

Independent samples Xk reduce variance

Estimation Quality

How can independent samples Xk improve the quality of estimation?

Keep s1 x s2 samples for Xk

s1 reduces variance, s2 boosts confidence

Avg(X1j2)

Avg(X2j2)

Avg(X3j2)

Avg(X4j2)

Avg(X5j2)

Median(sketch)

s1

s2

AtomicSketch

Sample Run of AMS

3 6 2 5 7V =

Z1 = -1 1 -1 1 11 1 -1 1 -1 Z2 =

X1= 5, X12 = 25 X2= 14, X2

2 = 196 Est = 110.5Σvi2 = 123

4 6 2 5 7V =

Z1 = -1 1 -1 1 11 1 -1 1 -1 Z2 =

X1= 6, X12 = 36, X2= 12, X2

2 = 144, Est = 90Σ vi2 = 130,

Comments on AMS

The self-join size can be computed on-line Sufficiently small variance (controlled by s1 and s2) Can this method be extended to answer other

queries?

Complex Aggregate Queries

A. Dobra et al. extend the idea of AMS to provide approximate answers to complex aggregate queries.

SELECT AGGAGG FROM R1,R2,…,Rr where EE AGG: COUNT/SUM/AVERAGE E: conjunction of (Ri.Aj = Rk.Al) It is proved that the error of these estimates is at

most ε with probability 1-δ.

Basic Notions of Approximation For aggregate queries (e.g., SUM, COUNT), approximation

quality can be measured by relative error: (Estimated value – Actual value) / Actual value

Open question: for queries involving more than simple aggregation, how should we define approximation?

Consider S |><|BT: (S: {A,B}, T: {B,C})

A B C

10 20.5 Doctor

8 10.3 Lawyer

3 10.2 Teacher

A B C

8 10.3 Lawyer

3 10.2 Teacher

Actual Result Approximate Result

Basic Notions of Approximation (2)

Can we accept this kind of approximation?

A B C

10 20.5 Doctor

8 10.3 Lawyer

3 10.2 Teacher

Actual Result Approximate Result

A B C

11 21.6 Doctor

8 10.3 Student

3 10.2 Teacher

Basic Notions of Approximation (3)

Can we provide useful (semantically correct) but stale results?

A B C

10 20.5 Doctor

8 10.3 Lawyer

3 10.2 Teacher

Actual Result (at time t)Approximate Result (correct result at time t - )

A B C

10 20.5 Doctor

8 10.3 Lawyer

Outline of this Talk

An Overview of Streams Data and Query Model Approximation Queries Other Research IssuesOther Research Issues

Data Mining

High-Speed Stream Data Mining Association Rules Stream Clustering Decision Trees

Single-pass algorithms for infering interesting patterns on-line (as the data stream arrives)

Useful for mission-critical tasks like telecom fraud detection

Conclusion: Future Work

Query Processing Stream Algebra and Query Languages Approximations Blocking Operators, Constraints, Punctuations

Runtime Management Scheduling, Memory Management, Rate Management Query Optimization (Adaptive, Multi-Query, Ad-hoc) Distributed processing

Synopses and Algorithmic Problems Systems

UI, statistics, crash recovery and transaction management System development and deployment

References

B. Babcock, S. Babu, M. Datar, R. Motwani, J. Widom. Models and Issues in Data Stream Systems, PODS ’02. (Paper and Talk Slides)

A. Arasu, B. Babcock, S. Babu, J. McAlister, J. Widom. Characterizing Memory Requirements for Queries over Continuous Data Streams, PODS ’02.

A. Dobra, M. Garofalakis, J. Gehrke, R. Rastogi. Processing Complex Aggregate Queries over Data Streams, SIGMOD ’02.

N. Alon, Y. Matias, M. Szegedy. The Space Complexity of Approximating the Frequency Moments, STOC ’96.

Thank You!