Adaptive Stream Adaptive Stream ResourceResource Management Management Using Kalman FiltersUsing Kalman Filters

Ankur JainAnkur Jain٭٭, Edward Y. Chang and Yuan-, Edward Y. Chang and Yuan-Fang WangFang Wang

Univ. of California, Santa BarbaraUniv. of California, Santa Barbara

SIGMOD 2004SIGMOD 2004

06/15/2004 2SIGMOD 2004

OutlineOutline

Data StreamsData Streams Introduction to data streams and Introduction to data streams and

common applicationscommon applications Resource management in data streamsResource management in data streams

Application of Kalman FiltersApplication of Kalman Filters Introduction to Kalman FiltersIntroduction to Kalman Filters Adaptive Stream Resource Management Adaptive Stream Resource Management

using Kalman Filtersusing Kalman Filters

06/15/2004 3SIGMOD 2004

Data StreamsData Streams

A Data stream is a continuous A Data stream is a continuous sequence of tuplessequence of tuples Unbounded in sizeUnbounded in size Tuples arrive onlineTuples arrive online Unpredictable/variable data arrival Unpredictable/variable data arrival

characteristicscharacteristics Real-time requirementsReal-time requirements Imprecise/noisy data (from sensors)Imprecise/noisy data (from sensors)

06/15/2004 4SIGMOD 2004

Applications Applications

Sensor networksSensor networks Monitor temperature in a nuclear reactorMonitor temperature in a nuclear reactor

Network monitoring & traffic Network monitoring & traffic engineeringengineering Monitor HTTP traffic on a network linkMonitor HTTP traffic on a network link

Financial tickersFinancial tickers Find stocks gaining more than 5% in last Find stocks gaining more than 5% in last

30 minutes30 minutes

06/15/2004 5SIGMOD 2004

A Data Stream Management A Data Stream Management SystemSystem

DSMS

Streaming Query Result

User Query

Streaming Data Sources

Query ProcessingQuery EvaluationResource ManagementStorage

Query PrecisionSampling RateSliding Window Size

Data FilteringData SamplingData ForwardingStream Synopsis

06/15/2004 6SIGMOD 2004

Resource ManagementResource Management

CommunicationCommunication Limited bandwidth and high variance in availabilityLimited bandwidth and high variance in availability

PowerPower Processing and transmitting data at remote sourceProcessing and transmitting data at remote source

CPUCPU Processing data at the server and the remote Processing data at the server and the remote

sourcesource MemoryMemory

Limited memory for processing unbounded streamsLimited memory for processing unbounded streams

06/15/2004 7SIGMOD 2004

Communication Resource Communication Resource ManagementManagement

Adaptive data filtering Adaptive data filtering STREAM [OJW03]STREAM [OJW03]

Adaptive load sheddingAdaptive load shedding Aurora [TCZAurora [TCZ++03], STREAM[BDM03]03], STREAM[BDM03]

Adaptive data samplingAdaptive data sampling TinyDB[MFH+03]TinyDB[MFH+03]

06/15/2004 8SIGMOD 2004

Adaptive Filtering of Data Adaptive Filtering of Data StreamsStreams

Some applications do not require Some applications do not require exact precision for the queries exact precision for the queries

Tradeoff between query precision Tradeoff between query precision and resource usageand resource usage

Data filtering according to the query Data filtering according to the query precisionprecision

06/15/2004 9SIGMOD 2004

Caching Data ModelCaching Data Model Prediction ModelPrediction ModelTimeTime

Va

lue

Va

lue

11 22 44 5533

8844

2266

Caching vs. PredictionCaching vs. Prediction

Time Time ValueValue Hit/Hit/MissMiss

00 00 --

11 00 MissMiss

22 22 MissMiss

33 44 MissMiss

44 66 MissMiss

55 88 HitHit

Time Time ValueValue Hit/Hit/MissMiss

00 00 --

11 00 MissMiss

22 44 HitHit

33 66 HitHit

44 88 HitHit

55 1010 MissMiss

Kalman Kalman

FilterFilter??

06/15/2004 10SIGMOD 2004

Outline of the Remaining Outline of the Remaining TalkTalk

Data StreamsData Streams Introduction to data streams and Introduction to data streams and

common applicationscommon applications Resource Management in data streamsResource Management in data streams

Data Streams and the Kalman FilterData Streams and the Kalman Filter Introduction to Kalman FiltersIntroduction to Kalman Filters Adaptive Stream Resource Management Adaptive Stream Resource Management

using Kalman Filters using Kalman Filters

06/15/2004 11SIGMOD 2004

Introduction to Kalman Introduction to Kalman Filter (KF)Filter (KF)

A prediction/correction algorithm A prediction/correction algorithm used for state estimation (developed used for state estimation (developed in 1960 by R.E. Kalman)in 1960 by R.E. Kalman)

KF is used forKF is used for PredictionPrediction – based on previous – based on previous

measurements and given state modelmeasurements and given state model EstimationEstimation – when measurements are – when measurements are

made in noisy environmentmade in noisy environment

06/15/2004 12SIGMOD 2004

Common Applications of Common Applications of the KF the KF

Tracking missiles Tracking missiles Tracking moving objectsTracking moving objects

Computer visionComputer vision Extracting lip motion from videoExtracting lip motion from video

Data fusion/integrationData fusion/integration Integration of spatio-temporal video Integration of spatio-temporal video

segmentssegments RoboticsRobotics

Robust estimation and sensor data noise Robust estimation and sensor data noise reductionreduction

06/15/2004 13SIGMOD 2004

The Discrete Kalman The Discrete Kalman FilterFilter

k kz H x w : Observation Relation

: Observation

w: Gaussian Noise (0, )

H

z

R

Measurement ModelMeasurement Model

: State Transition Matrix

: State Vector

: Gaussion Noise (0, )

F

x

v Q

1k kx F x v State ModelState Model

kk

k k

06/15/2004 14SIGMOD 2004

The Discrete Kalman The Discrete Kalman Filter Filter

ˆ : state estimate

ˆ : state estimate

k

k

x a posteriori

x a priori

ˆ ˆ ˆ( )k k k k kx x K z Hx State EstimateState Estimate

1( )T Tk k kK P H HP H R

: Kalman Gain

: estimate error covariance

k

k

K

P a priori

Kalman GainKalman Gain

06/15/2004 15SIGMOD 2004

The KF cycleThe KF cycle

Time Update(Predict)

Measurement Update(Correct)

Adjusts the current state estimate

Projects the current state estimate

06/15/2004 16SIGMOD 2004

A Simple Example - A Simple Example - TrackingTracking

1k kx Fx v

k kz Hx w

1

1 0 0

1 0 0 0

0 0 1

0 0 0 1

x x

x x

y y

y yk k

p ptp p

Qp pt

p p

1 0 0 0

0 0 1 0

x

xx

yy k

y k

p

ppR

pp

p

X

Y( , , , )k x x y yx p p p p

06/15/2004 17SIGMOD 2004

Tracking ExampleTracking ExampleActual

Estimate

06/15/2004 18SIGMOD 2004

KF and Data StreamingKF and Data Streaming

Capability to model wide range of Capability to model wide range of state transition functionsstate transition functions

Robustness during unavailability of Robustness during unavailability of measurementsmeasurements

Low computational complexity for Low computational complexity for simple problemssimple problems

06/15/2004 19SIGMOD 2004

Dual Kalman Filter Dual Kalman Filter (DKF)(DKF)

Central Server (Running KFs) Remote Source (Running KFc)

Drop thedata tuple

NO

Update available from remote source ?

YES

Forward update received fromthe remote source

NO

Forward prediction

from KFs

YES

Update the central serverwith new value

Streaming Source

Prediction at central server outsidethe query precision ?

06/15/2004 20SIGMOD 2004

Design goals of DKFDesign goals of DKF

Develop DKF as a general and Develop DKF as a general and adaptive stream filtering solutionadaptive stream filtering solution Static precision thresholdsStatic precision thresholds

Make tradeoff between query Make tradeoff between query precision and resource usageprecision and resource usage

Test performance on real and Test performance on real and synthetic data setssynthetic data sets Compare against data caching modelCompare against data caching model

06/15/2004 21SIGMOD 2004

DKF ArchitectureDKF ArchitectureSource value when estimation

error is large

KF state transition/error covariance matrices (rare)

Streaming Sources

V1

V2

Vn

1cKF

Central Server

nsKF

1

sKF

2sKF

Continuous Q

ueryE

valuator

Continuous Query

Continuous query results

06/15/2004 22SIGMOD 2004

Tracking - DatasetTracking - Dataset

06/15/2004 23SIGMOD 2004

Results - TrackingResults - Tracking

06/15/2004 24SIGMOD 2004

Results - Monitoring Results - Monitoring Electric Load in a Power Electric Load in a Power

ZoneZone

06/15/2004 25SIGMOD 2004

Issues and ChallengesIssues and Challenges

Setting sampling rates and Setting sampling rates and thresholdsthresholds

Avoiding too much computation at Avoiding too much computation at sensors sensors

Sensitivity vs. Precision vs. Sensitivity vs. Precision vs. Adaptability!Adaptability!

06/15/2004 26SIGMOD 2004

Future WorkFuture Work

Adaptive update of state transition Adaptive update of state transition matrices can further improve matrices can further improve performanceperformance

Evaluation of more complicated Evaluation of more complicated filters (e.g. particle filters) that can filters (e.g. particle filters) that can improve effectivenessimprove effectiveness

Models for non-linear systems can Models for non-linear systems can increase generalityincrease generality

06/15/2004 27SIGMOD 2004

Central Server (Running KFs)Remote Source (Running KFc)

Drop thedata tuple

NO

Is update available from remote source ?

YES

Forward update received fromthe remote source

NO

Forward predictionfrom KFs

YES

Update the central serverwith new value

Streaming Source

Is prediction at central server outside

the query precision ?

Thank You !