ill-posedness and regularization of linear operators (1 lecture)
DESCRIPTION
Ill-Posedness and Regularization of Linear Operators (1 lecture). Singular value decomposition (SVD) in finite-dimensional spaces Least squares solution; Moore-Penrose pseudo inverse Geometry of a linear inverse Ill-posed and ill-conditioned problems - PowerPoint PPT PresentationTRANSCRIPT
1IPIM, IST, José Bioucas, 2007 IPIM, IST, José Bioucas, 2007
Ill-Posedness and Regularization of Linear Operators (1 lecture)
Singular value decomposition (SVD) in finite-dimensional spaces
Least squares solution; Moore-Penrose pseudo inverse
Geometry of a linear inverse
Ill-posed and ill-conditioned problems
Tikhonov regularization; Truncated SVD
SVD of compact operators
2IPIM, IST, José Bioucas, 2007
Basics of linear operators in function spaces (Appendix B of [RB1])
Operator A from to is a mapping that assigns to each (domain) an element (range)
Hilbert spaces (finite or infinite)
A is defined every where
A is on operator on
Linear operators
We write
Null space of a linear operator (it is a subspace)
3IPIM, IST, José Bioucas, 2007
Basics of linear operators in function spaces (Apendix B of [RB1])
A linear operator A is continuous iff it is bounded
Adjoint operator depends on the inner product. Examples:
A is bounded if there exists a constant M:
is a matrix and
Norm of A:
Adjoint operator: is the unique operator such that
is a matrix and
Note:
4IPIM, IST, José Bioucas, 2007
Eigenvalues and eigenvectors of symmetric matrices
symmetric (self-adjoint )
is equipped with the standard Euclidian inner product
Eigen-equation
are real
may always be chosen to form an orthogonal basis
Let
(U is an unitary matrix)
Eigen-equation in terms of U
5IPIM, IST, José Bioucas, 2007
Spectral representation of symmetric matrix A
1. projects the input vector along
2. synthetizes by the linear combination
Action of a real symmetric matrix on an input vector
6IPIM, IST, José Bioucas, 2007
Functions of a symmetric matrix
From the following properties of an unitary matrix:
1.
2.
3.
It follows that
3. If A is non-singular
2. If h(A) is a power series
1.
4. If (A is positive semi-positive - PSD), we can define
7IPIM, IST, José Bioucas, 2007
• but A is not self-adjoint ( ), the eigendecomposition does not have the nice properties of self-adjoint matrices. The cyclic matrices are an exception
Singular value decomposition of a real (complex) rectangular matrix
• the eigenvalue problem is meaningless. The Singular value decomposition provides a generalization of the self-adjoint spectral decomposion
equipped with the standard Euclidian inner product
8IPIM, IST, José Bioucas, 2007
Singular value decomposition
and are equipped with the standard Euclidian inner products
isometric
left singular vectors
right singular vectors
singular values
9IPIM, IST, José Bioucas, 2007
matrix norms:
Singular value decomposition: consequences
range and null-space of
range and null-space of
10IPIM, IST, José Bioucas, 2007
Singular value decomposition
Action of a real symmetric matrix on the vector
1. projects the input vector along
2. synthetizes by the linear combination
11IPIM, IST, José Bioucas, 2007
Singular value decomposition: illustration
50 100 150 200 250
20
40
60
80
100
1200
0.2
0.4
0.6
0.8
1
0 50 100 150-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
u1
u10
0 50 100 15010
-20
10-15
10-10
10-5
100
105
i
0 50 100 150 200 250 300-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
v1
v10
12IPIM, IST, José Bioucas, 2007
Inversion methods:
A is not invertible
b)a)
A is invertible:
13IPIM, IST, José Bioucas, 2007
Least-squares approach
Orthogonal components
14IPIM, IST, José Bioucas, 2007
Generalized inverse
is the least-squares solution of minimum-norm, orthe generalized solution
15IPIM, IST, José Bioucas, 2007
Moore-Penrose pseudo-inverse and Minimum-norm solution
Moore-Penrose pseudo-inverse (r = n · m
Minimum-norm solution (r = m < n
16IPIM, IST, José Bioucas, 2007
Moore-Penrose pseudo-inverse: a variational point of view
is invertible
Minimization of the observed data misfit
Normal equations
17IPIM, IST, José Bioucas, 2007
Effect of noise
The boundary of is an ellipse centered at with principal axesaligned with . The lenght of the k-th principal semi-axis is
18IPIM, IST, José Bioucas, 2007
corresponding singular vectors
Classification of the linear operators
• If n m A is Ill-posed
• In any case “small” singular values are sources of instabilities.
Often, the smaller the eigenvalues the more oscilating the (high frequences)
Regularization: shrink/threshold large values of
i.e, multiply the eigenvalues by a regularizer function such that
as
19IPIM, IST, José Bioucas, 2007
Regularization
1)2 ) The larger singular are retained
as
Regularization by shrinking/thresholding the spectrum of A
Such that
Truncated SVD (TSVD)
Tikhonov (Wiener) regularization
20IPIM, IST, José Bioucas, 2007
Regularization by shrinking/thresholding the spectrum of A
Tikhonov regularization: variational formulation
TSVD
Tikhonov
unitary
Lets write the singular decomposition of A as
21IPIM, IST, José Bioucas, 2007
Tikhonov regularization
Family of quadratic regularizers
Does SVD plays a role?
Thus, the Tikhonov regularized solution is given by
which is the solution of the variational problem
for any Hilbert spaces (see Appendic E of [RB 1])
22IPIM, IST, José Bioucas, 2007
Singular value decomposition: illustration
50 100 150 200 250
20
40
60
80
100
1200
0.2
0.4
0.6
0.8
1
0 50 100 15010
-20
10-15
10-10
10-5
100
105
i
0 50 100 150 200 250 3000
0.2
0.4
0.6
0.8
1
1.2
1.4
f
g
23IPIM, IST, José Bioucas, 2007
Singular value decomposition: illustration
0 50 100 150 200 250 300-100
-50
0
50
100
150
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
24IPIM, IST, José Bioucas, 2007
Singular value decomposition: illustration
0 20 40 60 80 100 120 14010
-20
10-15
10-10
10-5
100
i
uk*w
0 50 100 150 200 250 300-20
-15
-10
-5
0
5
10
f
f
0 50 100 150 200 250 300-1.5
-1
-0.5
0
0.5
1
1.5
2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
25IPIM, IST, José Bioucas, 2007
Singular value decomposition: illustration
0 5 10 15 20
100.3
100.5
100.7
||fe-f||
10-6
10-4
10-2
100
102
100
101
102
L-curve
26IPIM, IST, José Bioucas, 2007
Medium/large systems
For medium/large systems, the SVD is impracticable
Example: Landweber iterations
The optimization problem
is solved by resorting to iterative methods that depend only onThe operators
with the Euler-Lagrange equation
27IPIM, IST, José Bioucas, 2007
3. The singular values are positive real numbers and are in nonincreasing order,
Singular system for a compact linear operator ( areHilbert spaces) is a countable set of triples with the following Properties:
Singular value decomposition in infinite-dimensional spaces
1. The right singular vectors forms an orthonormal basis for
2. The left singular vectors form an orthonormal basis for the closure of
4. For each j,
5. If is infinite dimensional,
6. A has the representation
28IPIM, IST, José Bioucas, 2007
Example of compact operators
1. Any linear operator for which is finite dimensional is compact
2. The diagonal operator on
3. The Fredholm first kind integral on (the space of real-valued square integrable functions on - a Hilbert space )
29IPIM, IST, José Bioucas, 2007
Compact linear operators in infinite dimensional spaces are ill-posed
be a compact linear operator are infinite dimensionalHilbert spaces.
1. If is infinite dimensional, then the operator equation is ill-posed in the sense that
•
• The solution is not stable
2. If is finite dimensional then the solution is not unique
30IPIM, IST, José Bioucas, 2007
Summary: SVD/least-squares based solutions
Least-squares approach
Minumum-norm solution
31IPIM, IST, José Bioucas, 2007
Truncated SVD (TSVD)
Tikhonov (Wiener) regularization
Summary: Regularized solutions
which is the solution of the variational problem
Regularizer(Penalizing function)
32IPIM, IST, José Bioucas, 2007
Summary: Medium/large systems with quadratic regularization
For medium/large systems, the SVD is impracticable. (periodic convolution operators are an important exception)
Example: Landweber iterations
The optimization problem
is solved by resorting to iterative methods that depend only on
with the Euler-Lagrange equation
Quadratic regularizer
33IPIM, IST, José Bioucas, 2007
Summary: Non-quadratic regularization
Example: discontinuity preserving regularizer
penalizes oscillatory solutions
34IPIM, IST, José Bioucas, 2007
0 50 100 150 200 250 30010
-20
10-15
10-10
10-5
100
105
vk*g
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
g
50 100 150 200 250
50
100
150
200
250
Matrix A
Example: deconvolution of a step
35IPIM, IST, José Bioucas, 2007
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
fTSVD
fTikhonov
fTikhonov(D)
fTV
Example: deconvolution of a step
36IPIM, IST, José Bioucas, 2007
Example: Sparse reconstruction
100 200 300 400 500 600 700 800 900 1000
20
40
60
80
100120
0 200 400 600 800 1000 1200-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Original data - f
0 20 40 60 80 100 120 140-0.1
-0.05
0
0.05
0.1
0.15
Observed data - g
norm
37IPIM, IST, José Bioucas, 2007
Example: Sparse reconstruction
0 200 400 600 800 1000 1200-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
f
fpseudo-inv
0 200 400 600 800 1000 1200-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
f
f
Pseudo-inverse regularization
38IPIM, IST, José Bioucas, 2007
Example: Sparse reconstruction. MM optimizationnorm
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
39IPIM, IST, José Bioucas, 2007
[Ch9.; RB1], [Ch2,Ch3; L1]
Majorization Minimization [PO1], [PO3] Compressed Sensing [PCS1]
Bibliography
Important topics
Matlab scripts
TSVD_regularization_1D.m TSVD_Error_1D.m step_deconvolution.m l2_l1sparse_regression.m