introduction to inverse problems (2 lectures) - itbioucas/ip/files/introduction.pdf · introduction...
TRANSCRIPT
1
Introduction to Inverse Problems (2 lectures)
Summary
Direct and inverse problems
Examples of direct (forward) problems
Deterministic and statistical points of view
Ill-posed and ill-conditioned problems
An illustrative example: The deconvolution problem
Truncated Fourier decomposition (TFD); Tikhonov regularization
Generalized Tikhonov regularization; Bayesian perspective.
Iterative optimization.
IP, José Bioucas Dias, 2007, IST
2
Direct/Inverse problems
Causes Effects
Direct (forward) problem
Inverse problem
Example:
Direct problem: the computation of the trajectories of bodies from the
knowledge of the forces.
Inverse problem: determination of the forces from the knowledge of the
trajectories
Newton solved the first direct/inverse problem: the determintion of the
gravitation force from the Kepler laws describing the trajectories of planets
3
An example: a linear time invariant (LTI) system
Inverse problem:
Fourier domain
high frequencies of the
perturbation are amplified,
degrading the estimate of f
A perturbation on leads to a perturbation on given by
Source of difficulties: is unbounded
Direct problem:
4
Image deblurring
Observation model in (linear) image restoration/reconstruction
observed image
original image
noise
Linear operator
(e.g., blur, tomography, MRI,...)
Goal: estimate f from g
7
Compressed Sensing (sparse representation)
0 200 400 600 800 1000-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Sparse vector f
0 20 40 60 80 100-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
Observed data y
Random matrix
0 200 400 600 800 1000-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
N=1000
M=100
Compressed Sensing
8
Deterministic observation mechanism
Classes of direct problems
+
perturbation
Operator
Original data (image) Observed data (image)
9
Classes of direct problems (deterministic)
Linear space-variant imaging systems (first kind Fredholm equation)
X-ray tomography
MR imaging
Radar imaging
Sonar imaging
Inverse diffraction
Inverse source
Linear regression
Blur (motion, out-of-focus,
Diffraction-limited imaging
atmospheric)
Near field acoustic holography
Channel equalization
Parameter identification
Linear space-invariant imaging systems
10
Classes of direct problems
Statistical observation mechanism
Original data (image) Observed data (image)
+
Ex: Linear/nonlinear observations in additive Gaussian noise
11
Classes of direct problems (statistic)
Rayleigh noise in coherent imaging
Poisson noise in photo-electric conversion
SPET (single photon emission tomography)
PET (positron emission tomography)
Linear/nonlinear observation driven by non-additive noise
Parameters
of a distribution
Random signal/
image
Ex: Amplitude in a coherent imaging system (radar, ultrasound)
Inphase/quadrature
backscattered signal
Terrain
reflectance
12
Well-posed/ill-posed inverse problems [Hadamard, 1923]
The inverse problem of solving is well-posed in the
Hadamard sense if:
1) A solution exists for any in the observed data space
2) The solution is unique
3) The inverse mapping is continuous
An inverse problem that is not well-posed is termed ill-posed
The operator A of an inverse well/ill-posed problem is termed well/ill-posed
Definition:
Let be a (possible nonlinear) operator
13
Finite/Infinite dimensional linear operators
Stability is also lacking: Take Then,
does not converge when
Consider A defined on ,
If a solution of exists, it is unique since
However, there are elements not in
Thus, A is ill-posed (point 1 of the Hadamard conditions does not hold)
Example: In infinite-dimensional spaces
The linear inverse problem is well-posed if 1) and 2) holds or,
equivalently, and
If is finite-dimensional, the corresponding inverse problem is
well-posed iif either one of the properties 1) and 2) holds
Linear Operators:
14
Ill-conditioned inverse problems
lll-posed
lll-conditioned
Many well-posed inverse problems are ill-conditioned, in the sense that
For linear operators (tight bound)
15
Example: Discrete deconvolution
N-periodic funtions
Cyclic convolution
Matrix notation
A is cyclic
Toeplitz
17
Eigen-decomposition of cyclic matrices
(unitary)
Eigenvector (Fourier) matrix Eigenvalue matrix (diagonal)
is the DFT of at frequency
19
Example: Discrete deconvolution (inferring f)
Assume that
Then is invertible and
Thus, assuming the direct model
We have
error
20
Example: cyclic convolution with a Gaussian kernel
0 5 10 15 20 25 30 350
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30 350
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25 30 35-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15 20 25 30 35-3
-2
-1
0
1
2
3
What went wrong ?
21
Example: Discrete deconvolution (estimation error)
“Size” of the error
Assume that
Thus
Which is a set enclosed by an ellipsoid with radii
22
Example: Discrete deconvolution (estimation error)
The estimation error is the vector
The components satisfy
23
Cyclic convolution with a Gaussian kernel (cont.)
0 5 10 15 20 25 30 3510
-8
10-6
10-4
10-2
100
102
Noise dominates at high frequencies
and is amplified by
1
(unit impulse function)
24
Example: Discrete deconvolution (A is ill-posed)
Assume now that
is not invertible and it may happen that
i.e, some are zero
Least-squares solution
Projection error
26
Example: Discrete deconvolution (A is ill-posed)
Invisible objects
is the minimum norm solution (related to the Moore-Penrose inverse)
27
Example: Discrete deconvolution (Regularization)
A is ill-conditioned
A is ill-posed
In both cases “small” eigenvalues are sources of instabilities
Often, the smaller the eigenvalue the more oscilating the
corresponding eigenvector (high frequences)
Regularization by filtereing: shrink/threshold large values of
i.e, multiply the eigenvalues by a regularizer filter such that
as
28
Example: Discrete deconvolution
1)
2 ) The larger eigenvalues are retained
as
Regularization by filtering (frequency multiplication time convolution)
Such that
Truncated Fourier Decomposition (TFD)
Tikhonov (Wiener) filter
29
Example: Discrete deconvolution (Regularization by filtering)
TFD
Tikhonov
Tikhonov regularization
Thus
Solution of the variational problem
30
Example: Discrete deconvolution (1D example)
0 50 100 150 200 250 3000
0.2
0.4
0.6
0.8
1
1.2
1.4
f
g
Gaussian shaped of standard deviation = 20
-4 -3 -2 -1 0 1 2 3 410
-12
10-10
10-8
10-6
10-4
10-2
100
frequency
| |
31
Example: Discrete deconvolution (1D example -TFD)
0 50 100 150 200 250 300-3
-2
-1
0
1
2
3
4
f
f
0 50 100 150 200 250 300-0.5
0
0.5
1
1.5
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150 200 250 300-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
34
Curing Ill-posed/Ill-conditioned inverse problems
Golden rule for solving ill-posed/ill-conditioned inverse problems
Search for solutions which:
• are compatible with the observed data
• satisfy additional constraints (a priori or prior information) coming
from the (physics) problem
35
Generalized Tikhonov regularization
Tikhonov and TFD regularization are not well suited to deal with data
Nonhomogeneities, such as edges
Generalized Tikhonov regularization
Data
Discrepancy
term
Penalty/
Regularization
term
Bayesian viewpoint
Negative
loglikelihood Negative
logprior
36
Dominating approaches to regularization
1)
2)
3)
4) In given circumstances
2), 3), and 4) are equivalent
37
Example: Discrete deconvolution (Nonquadratic regularization)
discontinuity preserving (robust) regularization
is nonconvex hard optimization problem
non-discontinuity preserving regularization
is convex treatable optimization problem
penalize oscillatory solutions
38
Optimization
- Quadratic
Linear system of equations
Large systems require iterative methods
- Non-quadratic and smooth
Methods: Steepest descent, nonlinear conjugate gradient, Newton,
trust regions, …
- Non-quadratic and nonsmooth
Constrained optimization (Linear, quadratic, second-order cone programs)
Methods: Iterative Shrinkage/Thesholding; Coordinate Subspace
Optimization; forward-backward splitting; Primal-dual Newton
Majorization Minimizaton (MM) class
39
Majorization Minorization (MM) Framework
Let
Majorization Minorization algorithm:
....with equality if and only if
Easy to prove monotonicity:
Notes:
EM is an algorithm of this type.
should be easy to maximize
40
Example: Discrete deconvolution (1D example – NQ Regula.)
0 50 100 150-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
Tikhonov
0 50 100 150-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 150-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
0 50 100 1500
0.2
0.4
0.6
0.8
1
1.2
1.4
f
f
0 50 100 150-0.2
0
0.2
0.4
0.6
0.8
1
1.2
f
f
41
Example: Discrete deconvolution (2D example-Total Variation)
Total variation regularization (TV)
TV regularizer penalizes highly oscilatory solutions, while it preserves the edges
where
42
[Ch1. RB2, Ch1. L1]
Euclidian and Hilbert spaces of functions [App. A, RB2]
Linear operators in function spaces [App. B, RB2]
Euclidian vector spaces and matrices [App. C, RB2]
Properties of the DFT and the FFT algorithm [App. B, RB2]
Bibliography
Important topics
Matlab scripts
TFD_regularization_1D.m
TFD_regularization_2D.m
TFD_Error_1D.m
TV_regulatization_1D.m