maintaining variance and k-medians over data stream windows paper by brian babcock, mayur datar,...
TRANSCRIPT
Maintaining Variance and k-Medians over
Data Stream Windows
Paper by Brian Babcock, Mayur Datar, Rajeev Motwani and Liadan
O’Callaghan.
Presentation by Anat Rapoport December 2003.Presentation by Anat Rapoport December 2003.
Characteristics of the data stream
Data elements arrive continually Only the most recent N elements
are used when answering queries Single linear scan algorithm (can
only have one look) Store only the summery of the data
seen thus far.
IntroductionTwo important and related
problems: –Variance
–K-median clustering
Problem 1 (Variance)
Given a stream of numbers, maintain at every instant the variance of the last N values
where denotes the mean of the last N values
N
iixVAR
1
2)(
N
i ixN 1
1
Problem 1 (Variance)
We cannot buffer the entire sliding window in memory
So we cannot compute the variance exactly at every instant
We will solve this problem approximately. We use memory and provide
an estimate with relative error of at most ε The time required per new element is
amortized O(1)
)log1
(2
NO
Extend to k-median Given a multiset X of objects in a metric
space M with distance function l the k-median problem is to pick k points c1,…,ck M∈ so as to minimize
where C(x) is the closest of c1,…,ck to x. if C(x)=ci then x is said to be assigned to ci
and l(x, ci) is called the assignment distance of x
The objective function is the sum of the assignment distances.
XxxCxl ))(,(
Problem 2 (SWKM)
Given a stream of points from a metric space M with distance function l, window size N,
and parameter k, maintain at every instant t a median set c1,…,ck M minimizing∈
where Xt is the multiset of N most recent points at time t
tXxxCxl ))(,(
Exponential Histogram
From last week: Maintaining simple statistics over sliding windows
The exponential histogram estimates a class of aggregated functions over sliding windows
Their result applies to any function f satisfying the following properties for all multisets X,Y:
Where EH goes wrong
))()(()(
)()()(
)()(
0)(
YfXfCYXf
YfXfYXf
XpolyXf
Xf
f
EH can estimate any function f defined over windows which satisfies:
Positive: Polynomialy bounded: Composable: Weakly additive:where Cf ≥1 is a constant
“Weakly Additive” condition not valid for variance, k-medians
Failure of “Weak Additivity”
Cannot afford to neglect contribution of last bucket
Time
Value
Variance of each bucket is small
Variance of combinedbucket is large
The idea Summarize intervals of the data stream
using composable synopses For efficient memory use adjacent intervals
are combined, when it doesn’t increase the error significantly
The synopsis of the last interval in the sliding window is inaccurate. Some points have expired…
HOWEVER– We will find a way to estimate this interval
Timestamp
Corresponds to the position of an active data element in the current window
We do not make explicit updates We use a wraparound counter of logN bits Timestamp can be extracted by comparison
with the counter value of the current arrival
Model We store the data elements in the buckets of
the histogram Every bucket stores the synopsis structure
for a contiguous set of elements The partition is based on arrival time The bucket also has a timestamp, of the
most recent data element in it When the timestamp reaches N+1 we drop
the bucket
Model
Buckets are numbered B1,…,Bm
– B1 the most recent
– Bm the oldest
t1,…,tm denote the bucket timestamp All buckets but Bm have only active data
elements
Maintaining variance over sliding windows
algorithm
Details
We would like to estimate the variance with relative error of at most ε
Maintain for each bucket Bi, besides it’s timestamp ti, also:– number of elements ni
– mean μi
– variance Vi
Details
Define: another set of buckets B1*,…, Bj* that represent the suffixes of the data stream.
The bucket Bm* represents all the points that arrived after the oldest non-expired bucket
The statistics for these buckets are computed temporarily
data structure: exponential histogram
X1XN X2
……
Window size N
timestampmost recentoldest
timestampmost recent
……
B1Bm-1Bm
oldest…… B2
Bm*
Combination rule In the algorithm we will need to combine adjacent buckets Consider two buckets Bi and Bj that get combined to form a
new bucket Bij
The statistics for Bij are
2
,,
,,
,
)( jiji
jijiji
ji
jjiiji
jiji
n
nnVVV
n
nn
nnn
Lemma 1
The bucket combination procedure correctly computes ni,j, μi,j, Vi,j for the new bucket
Proof
Note that ni,j, μi,j, are correctly computed from the definitions of count and average
Define δi=μi,-μi,j δj=μj,-μi,j
2
,
22
22
22
2222
22
2,,
)(
))(
())(
(
)()(
)(2)(2
)(2)()(2)(
)()(
)(,
jiji
jiji
ji
jiij
ji
ijjiji
jjiiji
Bxj
Bxji
Bxi
Bxiijjiiji
jBx
jljjliBx
iliil
Bxjjl
Bxiil
Bxjilji
n
nnVV
nn
nn
nn
nnVV
nnVV
xxnnVV
xxxx
xx
xV
jiljililil
jlil
jlil
jil
Main Solution Idea
More careful estimation of last bucket’s contribution
Decompose variance into two parts– “Internal” variance: within bucket
– “External” variance: between buckets
2ji
ji
jijiji, )μ - (μ
n + n
nn + V + V = V
Internal Varianceof Bucket i
Internal Varianceof Bucket j
External Variance
Estimation of the variance over the current active window
Let Bm’ refer to the non-expired portion of the bucket Bm (the set of active elements)
The estimation for nm’, μm’, Vm’ :– nm’
EST=N+1-tm (exact)– μm’EST= μm
– Vm’ EST= Vm/2
The statistics for Bm’,m* are sufficient for computing the variance at time t.
Estimation of the variance over the current active window
The estimate for Bm’ can be found in o(1) time if we keep statistics for Bm
The error is due to the error in the estimation statistics for Bm’
Theorem: Relative error ≤ ε, provided Vm ≤ (ε2/9) Vm*
Aim: Maintain Vm ≤ (ε2/9) Vm* using as few buckets as possible
))(()( 2*'
*'
*'*' mm
mm
mmmm nn
nnVVtVAR
Algorithm sketch
for every new element:– insert the new element
• to an existing bucket or to a new bucket
– if Bm‘s timestamp > N delete it
– if there are two adjacent buckets with small combined variance combine them to one bucket
Algorithm 1 (insert xt)
1. if xt=μ1 then insert xt to B1, by incrementing n1 by 1. Otherwise, create a new bucket for xt. The new bucket becomes B1 with v1=0 μ1= xt, n1 =1. An old bucket Bi becomes Bi+1.
2. if Bm‘s timestamp>N, delete the bucket. Bucket Bm-1 becomes the new oldest bucket. Maintain the statistics of Bm-1* (instead of Bm*), which can be computed using the previously maintained statistics for Bm* and Bm-1. (“deletion” of buckets also works…)
Algorithm 1 (insert xt)
3. Let k=9/ε^2 and Vi,i-1 is the variance combination of buckets Bi and Bi-1. While there exist an index i>2 such that kVi,i-1≤Vi-1* find the smallest i and combine the buckets according to the combination rule. The statistics for Bi* can be computed incrementally from the statistics for Bi-1 and Bi-1*
4. Output estimated variance at time t according to the estimation procedure. Vm’,m*
Invariant 1 For every bucket Bi, 9/ε2Vi ≤Vi*
– Ensures that the relative error is ≤ ε
Invariant 2 For each i<1, for every bucket Bi, 9/ε2Vi,i-1 > Vi-1*
– This invariant insures that the total number of buckets is small O((1/ε2)log NR2)
– Each bucket requires constant space
Lemma 2
The number of buckets maintained at any point in time by an algorithm that
preserves Invariant 2 is
O(1/ε2logNR2 )
where R is an upper bound on the absolute value of the data elements.
Proof sketch
From the combination rule: the variance of the union of two buckets is no less then the sum of the individual variances.
Algorithm that preserves invariant 2, the variance of the suffix bucket Bi* doubles after every O(1/ε2) buckets.
Total number of buckets: no more then O(1/ε2 logV) where V is the variance of the last N points. V is no more than NR2. O(1/ε2 log NR2)
Running time improvement The algorithm requires O(1/ε2logNR2
) time per new element.
Most time is spent in step 3 where we make the sweep to combine buckets.
The time is proportional to the size of the histogram O(1/ε2logNR2
). The trick: skip step 3 until we have seen Θ(1/ε2logNR2
). This ensures that the time of the algorithm is amortized
O(1). May violate invariant 2 temporarily, but we restore it every
Θ(1/ε2logNR2 ) data points, when we execute step 3.
Variance algorithm summery
O(1/ε2logNR2 ) time per new element
O(1/ε2 log NR2) memory with error of at most ε
Clustering on sliding windows
Clustering Data Streams
Based on k-median problem:– Data stream points from metric space.– Find k clusters in the stream such that the sum
of distances from data points to their closest center is minimized
Clustering Data Streams Constant factor approximation algorithms
– A simple two step algorithm:
step1: For each set of M=nτ points, Si, find O(k) centers in S1, …, SM
-- Local clustering: Assign each point in Si to its closest center
step2: Let S’ be centers for S1, …, SM with each center weighted by number of points assigned to it.
Cluster S’ to find k centers The solution cost is < 2*optimal solution cost τ<0.5 is a parameter which trades off space bound with
approximation factor of 2O(1/τ)
One-pass algorithm: first phase
1
3
2
4 5
1
3
2
4 5
S1 S2
S
Original data:
k=1
we take: M=3
One-pass algorithm: second phase
Original data:
M=3
k=1
we take:1
3
2
4 5
1
5
w=3
w=2
S’
Restate the algorithm…
Nτ points
input data stream
find O(k) mediansstore it with weightdiscard Nτ points
level-1 medians
level-0 medians
Nτ medians with associated weight
find O(k) medians level-2 medians
Repeat 1/τ times
The idea In general, whenever there are nτ medians at
level i they are clustered to form level (i+1) medians.
level-i medians
level-(i+1) medians
data points
timestampmost recent
……
B1Bm-1Bm
oldest……
X1XN X2
……
Window size N
timestampmost recentoldest
each bucket consists of a collection of data points or intermediate medians.
data structure: exponential histogram
Point representation Each point is represented by a triple:
(p(x),w(x), c(x)).p(x) - identifier of x (coordinate)
w(x) - weight of x, the number of points it representsc(x) - cost of x. An estimate of the sum of costs l(x,y)
of all the leaves y in the tree which x is the root of x.
w(x) = Σ(w(y1), w(y2),…,w(yi))c(x) = Σ(c(y)+w(y) l(x,y)) ,for all y assigned to x‧
if x is a level-0 median: w(x) = 1, c(x)=0
Thus, c(x) is an overestimate of the “true” cost of x
Bucket cost function
We maintain medians at intermediate levels When ever there are Nτ medians at the same
level we cluster them into O(k) medians at the next higher level
Each bucket can be spilt into 1/τ groups
where each contains medians at level j.
Each group contains at most Nτ medians
1/10 ,..., ii RR
jiR
Bucket cost function
Bucket’s cost function is an estimate of the cost of clustering the points represented by the bucket.
Consider bucket Bi. Let be the set of medians in the bucket
Cost function for Bi
Where C(x) ∈{c1,…ck} is the median closest to x
1/1
0
j
jii RY
iYxxCxlxwxc ))(,()()()f(Bi
Combination Let Bi and Bj be two adjacent buckets that need to
be combined to form Bi,j
Let be the groups of medians from the two buckets. Set:
if then cluster the points from and set it to be empty.
C0 set of O(k) medians obtained by clustering
and so on… After at most 1/τ unions we get Bi,j
Now we compute the new bucket’s cost
1/10 ,..., ii RR 1/10 ,...,
jj RR000
, jiji RRR NR ji 0
,
0111
, CRRR jiji
0, jiR
0, jiR
Answer a query
Consider buckets B1…Bm-1
Each contain at most 1/τ Nτ medians, all contain at most 1/τ Nτ medians
Cluster them to produce k medians Cluster bucket Bm to get k additional
medians Present the 2k medians as the answer
algorithm Insert xt
if number of level-0 medians in B1<k, add the point xt as a level-0 median in bucket B1. else create a new bucket B1 to contain xt and renumber the existing buckets accordingly.
if bucket Bm‘s time stamp > N, delete it; now, Bm-1 becomes the last bucket.
Make a sweep over the buckets from most recent to least recent
while there exists an index i>2 such that f(Bi,i-1)≤2f(Bi-1*), find the smallest such i and combine buckets Bi and Bi-1 using the combination procedure described above.
Invariant 3. For every bucket Bi f(Bi)≤2f(Bi*) Ensures a solution with 2k median whose
cost is within multiplicative factor of 2O(1/τ)
of the cost of the optimal k-median solution.
Invariant 4. For every bucket Bi (i>1), f(Bi,i-1)>2f(Bi-1*)Ensures that the number of buckets never
exceeds O(1/τ+logN)We assume that cost is bounded by poly(N)
O(1/τlogN) in the article
Running time improvement
After each element arrives we check if invariant 3 holds.
In order to reduce time we can execute bucket combination only after some amount of points accumulated in bucket B1, Only after it fills we check for the invariant.
We assume that the algorithm is not called after each new entry. Instead, it maintains enough statistics to produce statistics when a query arrives.
Producing exactly k clusters
With each median, we estimate within a constant factor the number of active data points that are assigned to it.
We don’t cluster Bm* and Bm’ separately but cluster the medians from all the buckets together. However the weights of medians form Bm are adjusted so that they reflect only the active data points.
Conclusions
The goal of such algorithms is to maintain statistics or information for the last N set of entries that is growing over real time.
The variance algorithm uses O(1/ε2logNR2) memory and maintains an estimate of the variance with relative error of at most ε and amortized O(1) time per new element
The k-median algorithm provides a 2O(1/τ)
approximation for τ<0.5. It uses O(1/τ+logN) memory and requires O(1) amortized time per new element.
