chebyshev polynomial approximation for distributed signal … › ... › www › talks ›...
TRANSCRIPT
Chebyshev Polynomial Approximation forDistributed Signal Processing
David Shuman, Pierre Vandergheynst, and Pascal Frossard
Ecole Polytechnique Federale de Lausanne (EPFL)Signal Processing Laboratory
{david.shuman, pierre.vandergheynst, pascal.frossard}@epfl.ch
International Conference onDistributed Computing in Sensor Systems (DCOSS)
Barcelona, Spain
June 28, 2011
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Motivating Application: Distributed Denoising
Sensor network with N sensors
Noisy signal in RN : y = x+ noise
Node n only observes yn and wants toestimate xn
No central entity - nodes can only sendmessages to their neighbors in thecommunication graph
However, communication is costly
Prior info, e.g., signal is smooth or
piecewise smooth w.r.t. graph structure
� If two sensors are close enough to
communicate, their observations are
more likely to be correlated
?>=<89:;1 //
��
?>=<89:;2 //oo ?>=<89:;3
AAAAAAAAoo
?>=<89:;4
AAAAAAAA
OO
?>=<89:;5
``AAAAAAAA
��?>=<89:;6
��
``AAAAAAAA ?>=<89:;7
~~}}}}}}}
OO
?>=<89:;8 //
��
?>=<89:;9
��
OO
oo // ?>=<89:;10oo
>>}}}}}}}
AAAAAAA
?>=<89:;11
OO
// ?>=<89:;12
OO
oo ?>=<89:;13
``AAAAAAA
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 2 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Outline
1 Introduction
2 Graph Fourier Multiplier Operators
3 Chebyshev Approximation of Graph Fourier Multipliers� Chebyshev Polynomials
� Centralized Computation
� Distributed Computation
4 Distributed Denoising Example
5 Summary, Ongoing Work, and Extensions
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 3 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Spectral Graph Theory Notation
Connected, undirected, weighted graphG = {V ,E ,W }
Degree matrix D: zeros except diagonals,which are sums of weights of edges incidentto corresponding node
Non-normalized Laplacian: L := D −W
Complete set of orthonormal eigenvectors andassociated real, non-negative eigenvalues:
Lχ` = λ`χ`,
ordered w.l.o.g. s.t.
0 = λ0 < λ1 ≤ λ2... ≤ λN−1 := λmax
?>=<89:;1.3 //
.1��
?>=<89:;2
.5��
oo
.2
���������
?>=<89:;3
@@�������
OO
.7//?>=<89:;4
OO
oo
W =
0 .3 .1 0.3 0 .2 .5.1 .2 0 .70 .5 .7 0
D =
.4 0 0 00 1 0 00 0 1 00 0 0 1.2
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 4 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Graph Laplacian Eigenvectors
Values of eigenvectors associated with lower frequencies (low λ`) changeless rapidly across connected vertices
χ0 χ1
χ2 χ50
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 5 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Graph Fourier Transform
Fourier transform: expansion of f in terms of the eigenfunctions of theLaplacian / graph Laplacian
Functions on the Real Line
Fourier Transform
f (ω) = 〈e iωx , f 〉 =∫R
f (x)e−iωx dx
Inverse Fourier Transform
f (x) = 12π
∫R
f (ω)e iωx dω
Functions on the Vertices of a Graph
Graph Fourier Transform
f (`) = 〈χ`, f 〉 =N∑
n=1
f (n)χ∗` (n)
Inverse Graph Fourier Transform
f (n) =N−1∑
=0
f (`)χ`(n)
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 6 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Fourier Multiplier Operator (Filter)
f (x) // FT // f (ω) // g // g(ω)f (ω) // IFT // Φf (x)
Fourier multiplier (filter) reshapes functions’ frequencies:
Φf (ω) = g(ω)f (ω), for every frequency ω
We can extend this to any group with a Fourier transform, includingweighted, undirected graphs:
Φf = IFT(g(ω)FT(f )(ω)
)
Functions on the Real Line
Φf (x) = 12π
∫R
g(ω)f (ω)e iωx dω
Functions on the Vertices of a Graph
Φf (n) =N−1∑
=0
g(λ`)f (`)χ`(n)
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 7 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Chebyshev Polynomials
Tn(x) := cos(n arccos(x)
),
x ∈ [−1, 1],n = 0, 1, 2, . . .
T0(x) = 1
T1(x) = x
Tk (x) = 2xTk−1(x)− Tk−2(x)
for k ≥ 2
Source: Wikipedia.
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 9 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Chebyshev Polynomial Expansion and Approximation
Chebyshev polynomials form an orthogonal basis for L2
([−1, 1], dx√
1−x2
)� Every h ∈ L2
([−1, 1], dx√
1−x2
)can be represented as
h(x) =1
2c0 +
∞∑k=1
ckTk (x), where ck =2
π
∫ π
0cos(kθ)h(cos(θ))dθ
M th order Chebyshev approximation to a continuous function on aninterval provides a near-optimal approximation (in the sup norm) amongstall polynomials of degree M
Shifted Chebyshev Polynomials
� To shift the domain from [-1,1] to [0,A], define
T k (x) := Tk
( x
α− 1), where α :=
A
2
� T k (x) = 2α
(x − α)T k−1(x)− T k−2(x) for k ≥ 2
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 10 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Fast Chebyshev Approx. of a Graph Fourier Multiplier
Let Φ ∈ RN×N be a graph Fourier multiplier with Φf =
(Φf )1
...(Φf )N
Approximate Graph Fourier Multiplier Operator
(Φf )n =N−1∑`=0
g(λ`)f (`)χ`(n) =N−1∑`=0
[1
2c0 +
∞∑k=1
ckT k(λ`)
]f (`)χ`(n)
≈N−1∑`=0
[1
2c0 +
M∑k=1
ckT k(λ`)
]f (`)χ`(n)
=
(1
2c0f +
M∑k=1
ckT k(L)f
)n
:=(
Φf)
n
Here, T k(L) ∈ RN×N and(T k(L)f
)n
:=N−1∑
=0
T k(λ`)f (`)χ`(n)
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 12 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Fast Chebyshev Approx. of a Graph Fourier Multiplier
Φf = 12c0f +
M∑k=1
ckT k(L)f ≈ Φf
Question: Why do we call this a fast approximation?
Answer: From the Chebyshev polynomial recursion property, we have:
T 0(L)f = f
T 1(L)f =1
αLf − f , where α :=
λmax
2
T k(L)f =2
α(L − αI )
(T k−1(L)f
)− T k−2(L)f
=2
αLT k−1(L)f − 2T k−1(L)f − T k−2(L)f
Does not require explicit computation of the eigenvectors of the Laplacian
Computational cost proportional to # nonzero entries in the Laplacian
This corresponds to the number of edges in the communication graph
Large, sparse graph ⇒ Φf far more efficient than ΦfDavid Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 13 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Distributed Computation
(Φf)
n=
(12c0f +
M∑k=1
ckT k(L)f
)n
Node n’s knowledge:
1 (f )n
2 Neighbors and weights of edges toits neighbors
3 Graph Fourier multiplier g(·), whichis used to compute co , c1, . . . , cM
4 Loose upper bound on λmax
Task: Compute (T k(L)f )n, k ∈ {1, 2, . . . ,M} in a distributed manner
(T 1(L)f )n = 1α
(Lf )n − (f )n = 1α
f0 Ln,2 0 0 0 Ln,6 0 0 0 −(f )n
(T k (L)f
)n
=(
2αLT k−1(L)f
)n−(
2T k−1(L)f)
n−(T k−2(L)f
)n
To get (T 2(L)f )n, suffices to compute (LT 1(L)f )n = `T1(L)f0 Ln,2 0 0 0 Ln,6 0 0 0
2M|E |scalar
messages
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 15 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Distributed Denoising - Method 1
Prior: signal is smooth w.r.t the underlying graph structure
Regularization term: fTLf = 12
∑n∈V
∑m∼n
wm,n [f (m)− f (n)]2
� fTLf = 0 iff f is constant across all vertices
� fTLf is small when signal f has similar values at neighboring vertices
connected by an edge with a large weight
Distributed regularization problem:
argminf
τ
2‖f − y‖2
2 + fTLf (1)
Proposition
The solution to (1) is given by Ry, where R is a graph Fourier multiplieroperator with multiplier g(λ`) = τ
τ+2λ`.
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 17 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Distributed Denoising Illustrative Example
Graph analog to low-pass filtering
Modify the contribution of each Laplacian
eigenvector
� f∗(n) = (Ry)n =N−1∑
=0
[τ
τ+2λ`
]y(`)χ`(n)
Use Chebyshev approximation to compute Ryin a distributed manner
Over 1000 experiments, average mean squareerror reduced from 0.250 to 0.013
0 5 10 150
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
λ
Exact Multiplier
Chebyshev Polynomial Approximation, M=5
Chebyshev Polynomial Approximation, M=15
Original Signal Noisy Signal Denoised Signal
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 18 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Unions of Graph Fourier Multipliers
So far, just a singlegraph Fouriermultiplier
Can easily extend thisto unions of graphFourier multipliers:
(Φηf) 1
f
(Φ1f) 1
Φ2
Φη
1N 1
NηNη
N
=...
…
(Φ1f) N
Φ1
…
(Φηf) N
.
.
.
(Φ2f) 1
(Φ2f) N
…
Example: Spectral Graph Wavelet Transform (Hammond et al., 2011)
� (Φf )(j−1)N+n =N−1∑
=0gj (λ`)f (`)χ`(n) for j ∈ {1, 2, . . . , η}, n ∈ {1, 2, . . . ,N}
� gj (λ`) = g(tjλ`) for j ∈ {1, 2, . . . , η − 1}, where g(·) is a band-pass filter
� gη(·) is a low-pass filter; coefficients represent low frequency content of signal
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 19 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Distributed Denoising - Method 2
Prior: signal is p.w. smooth w.r.t. graph ⇔ SGWT coefficients sparse
Regularize via LASSO (Tibshirani, 1996):
mina
12‖y −W ∗a‖2
2 + µ‖a‖1
Solve via iterative soft thresholding (Daubechies et al., 2004):
a(k) = Sµτ(a(k−1) + τW
(y −W ∗a(k−1)
)), k = 1, 2, . . .
D-LASSO (Mateos et al., 2010) solves in distributed fashion, but requires2|E | messages of length N(J + 1) at each iteration
Communication cost of Chebyshev polynomial approximation:� One computation of W y (2M|E | messages of length 1)
� At each soft thresholding iteration, distributed computation ofW W ∗a(k−1) (2M|E | messages of length J + 1 and 2M|E | messages oflength 1)
� One final computation of W ∗˜a to recover signal (2M|E | messages of
length J + 1)
Key takeaway: communication workload only scales with networksize through |E |, otherwise independent of N
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 20 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Summary
Graph Fourier muliplier operators are the graph analog of filter banks
� They reshape functions’ frequencies through multiplication in the graph
Fourier domain
A number of distributed signal processing tasks can be represented asapplications of graph Fourier multiplier operators
We approximate graph Fourier multipliers by Chebyshev polynomials
The recurrence relations of the Chebyshev polynomials make theapproximate operators readily amenable to distributed computation
The communication required to perform distributed computations onlyscales with the size of the network through the number of edges in thecommunication graph
The proposed method is well-suited to large-scale sensor networks withsparse communication graphs
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 22 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Ongoing Work and Extensions
Reviewer: “This seems to be a very interesting technique looking for a
problem”
� Other possible applications we are working on include distributed
smoothing, deconvolution, classification, and learning
� More thorough comparisons of communication costs with alternative
distributed methods for these applications
Extension: use the eigenvectors of other symmetric positive-semidefinitematrices as bases
Robustness issues
� Sensitivity to quantization and communication noise - how do they
propagate?
� Effect of a sensor node dropping out of the network or losing synchronicity
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 23 / 24
Intro Multiplier Operators Chebyshev Approximation Distributed Denoising Conclusion
Further Reading
Spectral Graph Theory and Laplacian Eigenvectors
F. K. Chung, Spectral Graph Theory. Vol. 92 of the CBMS Regional Conference Series in Mathematics,AMS Bokstore, 1997.
T. Bıyıkoglu, J. Leydold, and P. F. Stadler, Laplacian Eigenvectors of Graphs. Lecture Notes inMathematics, vol. 1915, Springer, 2007.
Chebyshev Polynomials
J. C. Mason and D. C. Handscomb, Chebyshev Polynomials. Chapman and Hall, 2003.
G. M. Phillips, Interpolation and Approximation by Polynomials. CMS Books in Mathematics,Springer-Verlag, 2003.
T. J. Rivlin, Chebyshev Polynomials. Wiley-Interscience, 1990.
Spectral Graph Wavelet Transform
D. K. Hammond, P. Vandergheynst, and R. Gribonval, “Wavelets on graphs via spectral graph theory,”
Appl. Comput. Harmon. Anal., vol. 30, no. 2, pp. 129–150, Mar. 2011.
David Shuman (EPFL) Distributed Chebyshev Approximation DCOSS 2011 24 / 24