exact and stable covariance estimation from quadratic …...exact and stable covariance estimation...
TRANSCRIPT
Exact and Stable Covariance Estimation from Quadratic Sampling
Yuxin Chen? Yuejie Chi† Andrea J. Goldsmith?? Electrical Engineering, Stanford University † Electrical and Computer Engineering, Ohio State University
• Covariance Estimationin – second-order statistics Σ ∈ Rn×nin – cornerstone of many information processing tasks
• Quadratic Samplingin – obtain measurements of the form
y ≈ a>Σa (1)
Objectives
binary data stream by Kazmin
• Covariance Sketchingin – data stream: real-time data {xt}∞t=1 arrivingsequentially at a high rate...
• Challengesin – limited memoryin – computational efficiencyin – a single pass over the data
Application: Data Stream Processing
• Proposed qudratic sketching method! Quadratic)Sketching)Scheme!
!x1"
!x2"
!x3"
!x4"
!x5"
!x6"
!x7"
!xN!
a1T"
!xi!
amT"
random))quadratic)sampling:)"a1T xixi
Ta1
xiT" !a1"
y1 =1N
a1T xixi
Ta1i=1
N
∑
aggregate)all)sketches"
N"data"instances"!"m"sketches"
random)sampling)"
n6dim)
1. sketching:
– at each time t, randomly choose a sketching vector ai– observe a quadratic sketch (aTi xt)
2
! Quadratic)Sketching)Scheme!
!x1"
!x2"
!x3"
!x4"
!x5"
!x6"
!x7"
!xN!
a1T"
!xi!
amT"
random))quadratic)sampling:)"a1T xixi
Ta1
xiT" !a1"
y1 =1N
a1T xixi
Ta1i=1
N
∑
aggregate)all)sketches"
N"data"instances"!"m"sketches"
n6dim)
2. aggregation:
– all sketches are aggregated into m measurements
yi ≈ E(a>i xtx
>t ai
)= a>i Σai (1 ≤ i ≤ m)
3. goal (estimation)
– estimate the covariance matrix Σ from y := {yi}mi=1
• Benefits
– one pass
– minimal storage (as will be shown)
Sketching*
Aggrega.on*
Es.ma.on*
• High-frequency wireless and signal processing
– Stationary processes / signals (possibly sparse)
t1 t2 t3– In high-frequency regime, energy measurements are more reliable
– Goal: recovery the power spectral density from energy measurements
∗ Frequency spikes can assume any continuous value (off-the-grid).
Application: Spectral Estimation
• Suppose that rank(Σ) = r � n
The Task
C = A* B*+
Low-rank Matrix
Unknown rank, eigenvectors
Sparse “Errors” Matrix
Unknown support, values
GivenComposite
matrix
• Convex Relaxation (TraceMin)
minimize trace (M )
s.t. ‖A (M )− y‖1 ≤ ε1,
M � 0.
– This coincides with PhaseLift (Candes et. al.) forrank-1 cases.
Formulation
• Given m (� n2) quadratic measurements y = {yi}mi=1
yi = a>i Σai + ηi, i = 1, · · · ,m, (2)
we wish to recover Σ ∈ Rn×n.
– ai : sketching/sampling vectors
– η = {ηi}mi=1: inaccuracy / noise terms
– More concise operator form: y = A(Σ) + η
• Sampling model
– sub-Gaussian i.i.d. sketching / sampling vectors
Algorithm: general low-rank structure
• Convex Relaxation (ToeplitzMin)
minimize M1,1
s.t. ‖A (M )− y‖1 ≤ ‖η‖1 ,M � 0,
M is Toeplitz.
– Can acommodate off-grid frequencies.
Algorithm: spectrally sparse processes
Piet Mondrian
Theoretical Guarantee
Theorem. (Non-stationary) If ‖η‖1 ≤ ε1, then the solution to TraceMin obeys∥∥∥Σ̂−Σ∥∥∥
F≤ C1
‖Σ−Σr‖∗√r
+ C2ε
m, (3)
where Σr is the best rank-r approximation of Σ, provided that m > Θ(nr).
(Stationary) If ‖η‖2 ≤ ε2, then the solution to ToeplitzMin obeys∥∥∥Σ̂−Σ∥∥∥
F≤ C3ε2√
m, (4)
provided that rank(Σ) ≤ r and m > Θ(r · poly log(n)).
m / (n*n)
r/n
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
information theoretic limit
m: number of measurements
r: r
ank
5 10 15 20 25 30 35 40 45 500
5
10
15
20
25
30
35
40
45
50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
theoretic sampling limit
(Left) Non-stationary case; (Right) Stationary case.
• Exact and Universal Recovery: from minimal noiseless measurements;
• Stable Recovery: inaccuracy proportional to noise level;
• Robust Recovery: robust to imperfect structural assumptions.
Discussion
• Work for other structural models
– Sparse covariance matrix
– Sparse phase retrieval (rank-1)
– Simultaneous sparse and low-rank covariance model
• Paper: Exact and Stable Covariance Estimation from Quadratic Samplingvia Convex Programming (http://arxiv.org/abs/1310.0807)