Adaptive Stream Adaptive Stream ResourceResource Management Management Using Kalman FiltersUsing Kalman Filters
Ankur JainAnkur Jain٭٭, Edward Y. Chang and Yuan-, Edward Y. Chang and Yuan-Fang WangFang Wang
Univ. of California, Santa BarbaraUniv. of California, Santa Barbara
SIGMOD 2004SIGMOD 2004
06/15/2004 2SIGMOD 2004
OutlineOutline
Data StreamsData Streams Introduction to data streams and Introduction to data streams and
common applicationscommon applications Resource management in data streamsResource management in data streams
Application of Kalman FiltersApplication of Kalman Filters Introduction to Kalman FiltersIntroduction to Kalman Filters Adaptive Stream Resource Management Adaptive Stream Resource Management
using Kalman Filtersusing Kalman Filters
06/15/2004 3SIGMOD 2004
Data StreamsData Streams
A Data stream is a continuous A Data stream is a continuous sequence of tuplessequence of tuples Unbounded in sizeUnbounded in size Tuples arrive onlineTuples arrive online Unpredictable/variable data arrival Unpredictable/variable data arrival
characteristicscharacteristics Real-time requirementsReal-time requirements Imprecise/noisy data (from sensors)Imprecise/noisy data (from sensors)
06/15/2004 4SIGMOD 2004
Applications Applications
Sensor networksSensor networks Monitor temperature in a nuclear reactorMonitor temperature in a nuclear reactor
Network monitoring & traffic Network monitoring & traffic engineeringengineering Monitor HTTP traffic on a network linkMonitor HTTP traffic on a network link
Financial tickersFinancial tickers Find stocks gaining more than 5% in last Find stocks gaining more than 5% in last
30 minutes30 minutes
06/15/2004 5SIGMOD 2004
A Data Stream Management A Data Stream Management SystemSystem
DSMS
Streaming Query Result
User Query
Streaming Data Sources
Query ProcessingQuery EvaluationResource ManagementStorage
Query PrecisionSampling RateSliding Window Size
Data FilteringData SamplingData ForwardingStream Synopsis
06/15/2004 6SIGMOD 2004
Resource ManagementResource Management
CommunicationCommunication Limited bandwidth and high variance in availabilityLimited bandwidth and high variance in availability
PowerPower Processing and transmitting data at remote sourceProcessing and transmitting data at remote source
CPUCPU Processing data at the server and the remote Processing data at the server and the remote
sourcesource MemoryMemory
Limited memory for processing unbounded streamsLimited memory for processing unbounded streams
06/15/2004 7SIGMOD 2004
Communication Resource Communication Resource ManagementManagement
Adaptive data filtering Adaptive data filtering STREAM [OJW03]STREAM [OJW03]
Adaptive load sheddingAdaptive load shedding Aurora [TCZAurora [TCZ++03], STREAM[BDM03]03], STREAM[BDM03]
Adaptive data samplingAdaptive data sampling TinyDB[MFH+03]TinyDB[MFH+03]
06/15/2004 8SIGMOD 2004
Adaptive Filtering of Data Adaptive Filtering of Data StreamsStreams
Some applications do not require Some applications do not require exact precision for the queries exact precision for the queries
Tradeoff between query precision Tradeoff between query precision and resource usageand resource usage
Data filtering according to the query Data filtering according to the query precisionprecision
06/15/2004 9SIGMOD 2004
Caching Data ModelCaching Data Model Prediction ModelPrediction ModelTimeTime
Va
lue
Va
lue
11 22 44 5533
8844
2266
Caching vs. PredictionCaching vs. Prediction
Time Time ValueValue Hit/Hit/MissMiss
00 00 --
11 00 MissMiss
22 22 MissMiss
33 44 MissMiss
44 66 MissMiss
55 88 HitHit
Time Time ValueValue Hit/Hit/MissMiss
00 00 --
11 00 MissMiss
22 44 HitHit
33 66 HitHit
44 88 HitHit
55 1010 MissMiss
Kalman Kalman
FilterFilter??
06/15/2004 10SIGMOD 2004
Outline of the Remaining Outline of the Remaining TalkTalk
Data StreamsData Streams Introduction to data streams and Introduction to data streams and
common applicationscommon applications Resource Management in data streamsResource Management in data streams
Data Streams and the Kalman FilterData Streams and the Kalman Filter Introduction to Kalman FiltersIntroduction to Kalman Filters Adaptive Stream Resource Management Adaptive Stream Resource Management
using Kalman Filters using Kalman Filters
06/15/2004 11SIGMOD 2004
Introduction to Kalman Introduction to Kalman Filter (KF)Filter (KF)
A prediction/correction algorithm A prediction/correction algorithm used for state estimation (developed used for state estimation (developed in 1960 by R.E. Kalman)in 1960 by R.E. Kalman)
KF is used forKF is used for PredictionPrediction – based on previous – based on previous
measurements and given state modelmeasurements and given state model EstimationEstimation – when measurements are – when measurements are
made in noisy environmentmade in noisy environment
06/15/2004 12SIGMOD 2004
Common Applications of Common Applications of the KF the KF
Tracking missiles Tracking missiles Tracking moving objectsTracking moving objects
Computer visionComputer vision Extracting lip motion from videoExtracting lip motion from video
Data fusion/integrationData fusion/integration Integration of spatio-temporal video Integration of spatio-temporal video
segmentssegments RoboticsRobotics
Robust estimation and sensor data noise Robust estimation and sensor data noise reductionreduction
06/15/2004 13SIGMOD 2004
The Discrete Kalman The Discrete Kalman FilterFilter
k kz H x w : Observation Relation
: Observation
w: Gaussian Noise (0, )
H
z
R
Measurement ModelMeasurement Model
: State Transition Matrix
: State Vector
: Gaussion Noise (0, )
F
x
v Q
1k kx F x v State ModelState Model
kk
k k
06/15/2004 14SIGMOD 2004
The Discrete Kalman The Discrete Kalman Filter Filter
ˆ : state estimate
ˆ : state estimate
k
k
x a posteriori
x a priori
ˆ ˆ ˆ( )k k k k kx x K z Hx State EstimateState Estimate
1( )T Tk k kK P H HP H R
: Kalman Gain
: estimate error covariance
k
k
K
P a priori
Kalman GainKalman Gain
06/15/2004 15SIGMOD 2004
The KF cycleThe KF cycle
Time Update(Predict)
Measurement Update(Correct)
Adjusts the current state estimate
Projects the current state estimate
06/15/2004 16SIGMOD 2004
A Simple Example - A Simple Example - TrackingTracking
1k kx Fx v
k kz Hx w
1
1 0 0
1 0 0 0
0 0 1
0 0 0 1
x x
x x
y y
y yk k
p ptp p
Qp pt
p p
1 0 0 0
0 0 1 0
x
xx
yy k
y k
p
ppR
pp
p
X
Y( , , , )k x x y yx p p p p
06/15/2004 17SIGMOD 2004
Tracking ExampleTracking ExampleActual
Estimate
06/15/2004 18SIGMOD 2004
KF and Data StreamingKF and Data Streaming
Capability to model wide range of Capability to model wide range of state transition functionsstate transition functions
Robustness during unavailability of Robustness during unavailability of measurementsmeasurements
Low computational complexity for Low computational complexity for simple problemssimple problems
06/15/2004 19SIGMOD 2004
Dual Kalman Filter Dual Kalman Filter (DKF)(DKF)
Central Server (Running KFs) Remote Source (Running KFc)
Drop thedata tuple
NO
Update available from remote source ?
YES
Forward update received fromthe remote source
NO
Forward prediction
from KFs
YES
Update the central serverwith new value
Streaming Source
Prediction at central server outsidethe query precision ?
06/15/2004 20SIGMOD 2004
Design goals of DKFDesign goals of DKF
Develop DKF as a general and Develop DKF as a general and adaptive stream filtering solutionadaptive stream filtering solution Static precision thresholdsStatic precision thresholds
Make tradeoff between query Make tradeoff between query precision and resource usageprecision and resource usage
Test performance on real and Test performance on real and synthetic data setssynthetic data sets Compare against data caching modelCompare against data caching model
06/15/2004 21SIGMOD 2004
DKF ArchitectureDKF ArchitectureSource value when estimation
error is large
KF state transition/error covariance matrices (rare)
Streaming Sources
V1
V2
Vn
1cKF
Central Server
nsKF
1
sKF
2sKF
Continuous Q
ueryE
valuator
Continuous Query
Continuous query results
06/15/2004 22SIGMOD 2004
Tracking - DatasetTracking - Dataset
06/15/2004 23SIGMOD 2004
Results - TrackingResults - Tracking
06/15/2004 24SIGMOD 2004
Results - Monitoring Results - Monitoring Electric Load in a Power Electric Load in a Power
ZoneZone
06/15/2004 25SIGMOD 2004
Issues and ChallengesIssues and Challenges
Setting sampling rates and Setting sampling rates and thresholdsthresholds
Avoiding too much computation at Avoiding too much computation at sensors sensors
Sensitivity vs. Precision vs. Sensitivity vs. Precision vs. Adaptability!Adaptability!
06/15/2004 26SIGMOD 2004
Future WorkFuture Work
Adaptive update of state transition Adaptive update of state transition matrices can further improve matrices can further improve performanceperformance
Evaluation of more complicated Evaluation of more complicated filters (e.g. particle filters) that can filters (e.g. particle filters) that can improve effectivenessimprove effectiveness
Models for non-linear systems can Models for non-linear systems can increase generalityincrease generality
06/15/2004 27SIGMOD 2004
Central Server (Running KFs)Remote Source (Running KFc)
Drop thedata tuple
NO
Is update available from remote source ?
YES
Forward update received fromthe remote source
NO
Forward predictionfrom KFs
YES
Update the central serverwith new value
Streaming Source
Is prediction at central server outside
the query precision ?
Thank You !