bmi2 ss08 – class 3 “image processing 1” slide 1 biomedical imaging 2 class 3,4 –...
Post on 30-Dec-2015
215 Views
Preview:
TRANSCRIPT
BMI2 SS08 – Class 3 “Image Processing 1” Slide 1
Biomedical Imaging 2Biomedical Imaging 2
Class 3,4 – Regularization; Time Series Analysis (Pt. 1); Image Post-processing (Pt. 1)
02/05/08
BMI2 SS08 – Class 3 “Image Processing 1” Slide 2
Well-Posedness, Ill-PosednessWell-Posedness, Ill-Posedness
• Definition due to Hadamard, 1915: Given the mapping A: X→Y, the equation Ax = y is well-posed if– (Existence) For every y in Y, there is an x in X such
that Ax = y.
– (Uniqueness) If Ax1 = Ax2, then x1 = x2.
– (Stability) A-1 is continuous• x = A-1y
• A-1(y + dy) = x + dx
• Ax = y is ill-posed if it is not well-posed
BMI2 SS08 – Class 3 “Image Processing 1” Slide 3
Well- and Ill-conditioned ProblemsWell- and Ill-conditioned Problems
• Overdetermined linear systems (more equations than unknowns) are ill-posed, strictly speaking– No exact Solution exists!– Existence is imposed by using least-squares solution
• Underdetermined linear systems (fewer equations than unknowns) are ill-posed, strictly speaking– Infinitely many Solutions exist!– Uniqueness is imposed by using minimum-norm
solution
• Can a discrete linear system be unstable?
BMI2 SS08 – Class 3 “Image Processing 1” Slide 4
Well- and Ill-conditioned ProblemsWell- and Ill-conditioned Problems
• Can a discrete linear system be unstable?– Strictly speaking, no!
• A-1(x0 + x) = A-1x0 + A-1∙x
• All elements of A-1 and of x are finite• Therefore, all elements of A-1∙x must be finite
• However, it certainly can be true that||A-1∙x||/||A-1x0|| >> ||x||/||x0||
– Small change in input → large change in output– Such a system is called ill-conditioned
BMI2 SS08 – Class 3 “Image Processing 1” Slide 5
Example of Ill-conditioningExample of Ill-conditioning
1 1 12 3 4
1 1 1 12 3 4 5
1 1 1 13 4 5 6
1 1 1 14 5 6 7
14×4 Hilbert matrix:
This is a full-rank, non-singular matrix, and so it has a well-defined inverse:
16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
BMI2 SS08 – Class 3 “Image Processing 1” Slide 6
Example of Ill-conditioningExample of Ill-conditioning
1 1 12 3 4
1 1 1 12 3 4 51 1 1 13 4 5 61 1 1 14 5 6 7
1 12
1 16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
116 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
13 4
1 1 1 12 3 4 51 1 1 13 4 5 61 1 1 14 5 6 7
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
Positive and negative products cancel in exactly the right manner:
But what happens if we change any element by even a small amount?
BMI2 SS08 – Class 3 “Image Processing 1” Slide 7
Example of Ill-conditioningExample of Ill-conditioning
1 1 12 3 4
1 1 1 112 3 4 5
1 1 1 13 4 5 61 1 1 14 5 6 7
1 16 120 240 140 1
120 1200 2700 1680 1, ,
240 2700 6480 4200 1
140 1680 4200 2800 1
A A y
1
16 120 240 140 1 4
120 1200 2700 1680 1 60
240 2700 6480 4200 1 180
140 1680 4200 2800 1 140
x A y
1 0.0043256 0.99567
1 0.016656 0.98334But what if ?
1 0.0012533 1.0013
1 0.0028768 1.0029
y
BMI2 SS08 – Class 3 “Image Processing 1” Slide 8
Example of Ill-conditioningExample of Ill-conditioning
1
1
16 120 240 140 1 4
120 1200 2700 1680 1 60
240 2700 6480 4200 1 180
140 1680 4200 2800 1 140
16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
y
x A y
x x A y
0.99567
0.98334
1.0013
1.0029
2.1725
41.981
140.03
115.41
BMI2 SS08 – Class 3 “Image Processing 1” Slide 9
Example of Ill-conditioningExample of Ill-conditioning
1 1 1
2 2 2
3 3 3
4 4 4
0,0.011 4
0,0.011 60, ~
0,0.011 180
0,0.011 140
Ny x y
Ny x y
Ny x y
Ny x y
1 1
2 2
3 3
4 4
Sample 10,000 times:
0.000142 0.010053
0.000005 0.009951,
0.000207 0.010162
0.000006 0.009986
y y
y ymean std
y y
y y
1 1
2 2
3 3
4 4
0.05 3.05
0.57 34.2,
1.36 82.3
0.88 53.5
x x
x xmean std
x x
x x
BMI2 SS08 – Class 3 “Image Processing 1” Slide 10
Example of Ill-conditioningExample of Ill-conditioning
1 1 1
2 2 2
3 3 3
4 4 4
1
2
3
4
0,0.011 4
0,0.011 60, ~ '
0,0.011 180
0,0.011 140
0.05
0.57
1.36
0.
Ny x y
Ny x y
Ny x y
Ny x y
x
xmean
x
x
1
2
3
4
3.05
34.2,
82.3
88 53.5
x
xstd
x
x
Our “image reconstruction” operator is unbiased
But it has high variance
BMI2 SS08 – Class 3 “Image Processing 1” Slide 11
Tradeoff Between Bias and VarianceTradeoff Between Bias and Variance
The numerical values in y are eventually represented as gray levels or colors in an image:
4
60
180
140
-4
-180
60
140
As long as the color pattern makes an interpretable image, do you care if the numerical values are exactly right?
That is, are you willing to give up accuracy to gain precision (i.e., decrease variance by increasing bias)?
BMI2 SS08 – Class 3 “Image Processing 1” Slide 12
RegularizationRegularization
• For overdetermined systems, we define the pseudo-inverse A+ = (ATA)-1AT.
• For underdetermined systems, we define the pseudo-inverse A+ = AT(AAT)-1.
• For the Hilbert matrix, both of the preceding reduce to the true inverse:– (ATA)-1AT = [A-1(AT)-1]AT = A-1[(AT)-1AT] = A-1
– AT(AAT)-1 = AT[(AT)-1A-1] = [AT(AT)-1 ]A-1 = A-1
• Now we introduce one additional term:– A+ = (ATA + αI)-1AT
– A+ = AT(AAT + αI)-1Regularization term
Regularization parameter
BMI2 SS08 – Class 3 “Image Processing 1” Slide 13
What Does Regularization Do?What Does Regularization Do?
• Now we introduce one additional term:– A+ = (ATA + αI)-1AT, A+ = AT(AAT + αI)-1
• This particular variety is called Tikhonov regularization– Imposes continuity on the computed y
• That is, limits the spatial scale on which solution can change (long-pass filter)
– Could replace the I in the regularization term with a discrete 1st, 2nd, etc., derivative operator
• Then continuity would be imposed on the corresponding derivative of the solution
BMI2 SS08 – Class 3 “Image Processing 1” Slide 14
Something To Watch Out forSomething To Watch Out for
• Two things that can go wrong when Tik. Reg. is used:– α is too small (under-regularized case): noise
continues to wreak havoc– α is too large (over-regularized case): ability to
capture spatial variations of interest is lost
• Is there an algorithm guaranteed to produce the optimal α?– Alas, no (when is life ever that easy?)– Special cases; Monte Carlo simulations; trial-and-
error
BMI2 SS08 – Class 3 “Image Processing 1” Slide 15
Regularized Hilbert Matrix InverseRegularized Hilbert Matrix Inverse
11 T T
16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
A A A A
1T T A A A I A
7
7.9301 29.208 21.508 2.0164
29.208 178.01 239.85 80.65310 :
21.508 239.85 556.97 349.04
2.0164 80.653 349.04 296.07
8
11.445 68.715 116.54 59.731
68.715 622.57 1309.8 776.1610 :
116.54 1309.8 3133 2023.9
59.731 776.16 2023.9 1385.1
9
15.148 110.41 216.92 124.99
110.41 1092.1 2440.1 151110 :
216.92 2440.1 5854.3 3793.2
124.99 1511 3793.2 2535.5
BMI2 SS08 – Class 3 “Image Processing 1” Slide 16
Impact of Noisy Data Impact of Noisy Data
1 1
2 2
3 3
4 4
0.05 3.05
0.57 34.2,
1.36 82.3
0.88 53.5
x x
x xmean std
x x
x x
Unregularized solution (i.e., α = 0):
1 1
2 29
3 3
4 4
0.71 2.76
7.96 30.910 : ,
19.1 74.4
12.4 48.4
x x
x xmean std
x x
x x
Conclusion: Under-regularized!
BMI2 SS08 – Class 3 “Image Processing 1” Slide 17
Impact of Noisy Data Impact of Noisy Data
1 1
2 2
3 3
4 4
0.05 3.05
0.57 34.2,
1.36 82.3
0.88 53.5
x x
x xmean std
x x
x x
Unregularized solution (i.e., α = 0):
1 1
2 27
3 3
4 4
6.25 0.38
70.5 3.1410 : ,
170 7.05
110 4.66
x x
x xmean std
x x
x x
Conclusion: Over-regularized!
BMI2 SS08 – Class 3 “Image Processing 1” Slide 18
Impact of Noisy Data Impact of Noisy Data
1 1
2 2
3 3
4 4
0.05 3.05
0.57 34.2,
1.36 82.3
0.88 53.5
x x
x xmean std
x x
x x
Unregularized solution (i.e., α = 0):
1 1
2 28
3 3
4 4
3.56 1.50
40.1 16.610 : ,
96.5 39.8
62.7 25.9
x x
x xmean std
x x
x x
Conclusion: Getting close?
BMI2 SS08 – Class 3 “Image Processing 1” Slide 19
Final Choice of α Parameter
7.8
1 1
2 2
3 3
4 4
-4
6010 : noise-free = ,
-180
140
0.32 1.16
11.3 12.8, .
62.9 30.6
63.8 19.9
x x
x xmean std
x x
x x
y
BMI2 SS08 – Class 3 “Image Processing 1” Slide 20
Other Types of RegularizationOther Types of Regularization
• Truncated singular value decomposition• Discrete-cosine transform• Statistical (Bayesian)
– Requires knowledge of the solution and noise covariances
• Iterative– Steepest descent– Conjugate-gradient descent– Richardson-Lucy– Landweber
BMI2 SS08 – Class 3 “Image Processing 1” Slide 21
Time Series Analysis…Time Series Analysis…
Definitions• The branch of quantitative forecasting in which data for one variable are
examined for patterns of trend, seasonality, and cycle. nces.ed.gov/programs/projections/appendix_D.asp
• Analysis of any variable classified by time, in which the values of the variable are functions of the time periods. www.indiainfoline.com/bisc/matt.html
• An analysis conducted on people observed over multiple time periods. www.rwjf.org/reports/npreports/hcrig.html
• A type of forecast in which data relating to past demand are used to predict future demand. highered.mcgraw-hill.com/sites/0072506369/student_view0/chapter12/glossary.html
• In statistics and signal processing, a time series is a sequence of data points, measured typically at successive times, spaced apart at uniform time intervals. Time series analysis comprises methods that attempt to understand such time series, often either to understand the underlying theory of the data points (where did they come from? what generated them?), or to make forecasts (predictions). en.wikipedia.org/wiki/Time_series_analysis
BMI2 SS08 – Class 3 “Image Processing 1” Slide 22
Time Series Analysis…Time Series Analysis…
Varieties• Frequency (spectral) analysis
– Fourier transform: amplitude and phase– Power spectrum; power spectral density
• Auto-spectral density– Cross-spectral density– Coherence
• Correlation Analysis– Cross-correlation function
• Cross-covariance• Correlation coefficient function
– Autocorrelation function– Cross-spectral density
• Auto-spectral density
BMI2 SS08 – Class 3 “Image Processing 1” Slide 23
Time Series Analysis…Time Series Analysis…
Varieties• Time-frequency analysis
– Short-time Fourier transform– Wavelet analysis
• Descriptive Statistics– Mean / median; standard deviation / variance / range– Short-time mean, standard deviation, etc.
• Forecasting / Prediction– Autoregressive (AR)– Moving Average (MA)– Autoregressive moving average (ARMA)– Autoregressive integrated moving average (ARIMA)
• Random walk, random trend• Exponential weighted moving average
BMI2 SS08 – Class 3 “Image Processing 1” Slide 24
Time Series Analysis…Time Series Analysis…
Varieties• Signal separation
– Data-driven [blind source separation (BSS), signal source separation (SSS)]
• Principal component analysis (PCA)• Independent component analysis (ICA)• Extended spatial decomposition, extended temporal
decomposition• Canonical correlation analysis (CCA)• Singular-value decomposition (SVD) an essential
ingredient of all– Model-based
• General linear model (GLM)• Analysis of variance (ANOVA, ANCOVA, MANOVA, MANCOVA)
– e.g., Statistical Parametric Mapping, BrainVoyager, AFNI
BMI2 SS08 – Class 3 “Image Processing 1” Slide 25
A “Family Secret” of Time Series Analysis…A “Family Secret” of Time Series Analysis…• Scary-looking formulas, such as
– Are useful and important to learn at some stage, but not really essential for understanding how all these methods work
• All the math you really need to know, for understanding, is– How to add: 3 + 5 = 8, 2 - 7 = 2 + (-7) = -5– How to multiply: 3 × 5 = 15, 2 × (-7) = -14
• Multiplication distributes over addition
u × (v1 + v2 + v3 + …) = u×v1 + u×v2 + u×v3 + …
– Pythagorean theorem: a2 + b2 = c2
1 2 1 2
1, ,
2
, , ,
, , , ,
x y
i t i t
i x y
x y
x y x y
F f t e dt f t F e d
F f x y e dxdy
F f x y f x y F F
a
b
c
BMI2 SS08 – Class 3 “Image Processing 1” Slide 26
A “Family Secret” of Time Series Analysis…A “Family Secret” of Time Series Analysis…
A most fundamental mathematical operation for time series analysis:
1 2 3
1 2 3
, , , ...,
, , , ...,
N
N
x x x x
y y y y
31 2
31 2
1 2 3
, , , ...,
N
N
N
xx x x
yy y y
z z zz
1 2 3 ... Nz z z z Z
The xi time series is measurement or image data. The yi time series depends on what type of analysis we’re doing:
Fourier analysis: yi is a sinusoidal function
Correlation analysis: yi is a second data or image time series
Wavelet or short-time FT: non-zero yi values are concentrated in a small range of i, while most of the yis are 0.
GLM: yi is an ideal, or model, time series that we expect some of the xi time series to resemble
BMI2 SS08 – Class 3 “Image Processing 1” Slide 27
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8Individual Frequency Components
BMI2 SS08 – Class 3 “Image Processing 1” Slide 28
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 1
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 2
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 3
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 4
BMI2 SS08 – Class 3 “Image Processing 1” Slide 29
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 5
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 6
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 7
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 8
BMI2 SS08 – Class 3 “Image Processing 1” Slide 30
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial Sum 9
time [s]
0.0 0.5 1.0 1.5 2.0F
unct
ion
Val
ue-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Partial sum 10
BMI2 SS08 – Class 3 “Image Processing 1” Slide 31
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8Individual Frequency Components
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Sawtooth Wave
= Σ
×
sin10πt
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Sawtooth*sin(10*pi*t)
BMI2 SS08 – Class 3 “Image Processing 1” Slide 32
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8First Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.4Second Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.25
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20Third Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20Fourth Frequency Component * sin(10*pi*t)
BMI2 SS08 – Class 3 “Image Processing 1” Slide 33
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.14
-0.10
-0.06
-0.02
0.02
0.06
0.10
0.14Fifth Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15Sixth Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.10
-0.05
0.00
0.05
0.10Seventh Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.08
-0.04
0.00
0.04
0.08Eighth Frequency Component * sin(10*pi*t)
BMI2 SS08 – Class 3 “Image Processing 1” Slide 34
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.08
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08Ninth Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0Fu
nctio
n V
alue
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06Tenth Frequency Component * sin(10*pi*t)
BMI2 SS08 – Class 3 “Image Processing 1” Slide 35
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.14
-0.10
-0.06
-0.02
0.02
0.06
0.10
0.14Fifth Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0
Fun
ctio
n V
alue
-1.2
-0.8
-0.4
0.0
0.4
0.8
1.2Sawtooth*sin(10*pi*t)
= ΣΣ
BMI2 SS08 – Class 3 “Image Processing 1” Slide 36
Example: Fourier AnalysisExample: Fourier Analysis
time [s]
0.0 0.5 1.0 1.5 2.0
Func
tion
Val
ue
-0.14
-0.10
-0.06
-0.02
0.02
0.06
0.10
0.14Fifth Frequency Component * sin(10*pi*t)
time [s]
0.0 0.5 1.0 1.5 2.0F
unct
ion
Val
ue-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8Individual Frequency Components
BMI2 SS08 – Class 3 “Image Processing 1” Slide 37
Another “Family Secret” of Time Series Analysis…Another “Family Secret” of Time Series Analysis…• The second operation that is fundamental to myriad forms of time-
series analysis is singular value decomposition (SVD)• Variations of SVD underlie:
– Principal component analysis (PCA)– Independent component analysis (ICA)– Canonical correlation analysis (CCA)– Extended spatial/temporal decorrelation
BMI2 SS08 – Class 3 “Image Processing 1” Slide 38
Significance of angle between x and b
Given an arbitrary N×N matrix A and N×1 vector x: ordinarily, b = Ax is different from x in both magnitude and direction.
x
b
However, for any A there will always be some particular directions such that b will be parallel to x (i.e., b is a simple scalar multiple of x, or Ax = λx) if x lies in one of these directions.An x that satisfies Ax = λx is an eigenvector, and λ is the corresponding eigenvalue.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 39
Homogeneous Linear System: Ax = 0Homogeneous Linear System: Ax = 0
Recall definition of eigenvectors and eigenvalues:
Ax = λx, x 0.
Then Ax - λx = Ax - λIx = (A - λI)x = 0.
That is, the eigenvalues are those specific values of λ for which the matrix A - λI is singular, and the eigenvectors are the corresponding nullspaces.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 40
Significance of angle between x and b
2 2
1 1 12
2 2
1 1 12
2 2
2 1
1 2
2 12 1 4 3
1 2
3,1
1 1 13: 0
1 1 1
1 1 11: 0
1 1
0
1
u u
u u
v v
v v
M
2 1 1 1
1 2
2 1 1 3
1 2 1
1 1
3
1 12 2
1 1 1 12 2 2 2
1 12 21 1
2 21 1
2 2
13
2
2 1 1 4
1 2 2 5
2 1 2 13 3 3
1 2 1 2
49 9
5
u v
u v u v
u v
BMI2 SS08 – Class 3 “Image Processing 1” Slide 41
Significance of eigenvalues and eigenvectors
An N×N A always has N eigenvalues.
If A is symmetric, and λ1 and λ2 are two distinct eigenvalues, the corresponding eigenvectors x1 and x2 are necessarily orthogonal.
If λ1 = λ2, we can always subtract off x1’s projection onto x2 from x1 (Gram-Schmidt orthogonalization).
If A is not symmetric, then its eigenvectors generally are not mutually orthogonal. But recall that the matrices AAT and ATA are always symmetric.
The square roots of the eigenvalues of AAT or ATA are thesingular values of A. The eigenvectors of AAT or ATA are thesingular vectors of A.
Computation of the eigenvalues and eigenvectors of AAT and ATA underlies a very useful linear algebraic technique called singular value decomposition (SVD).
SVD is the method that allows us to, among other things, tackle the one case we have not yet seen an explicit example of: finding the “solution” of a linear system when A is not of full rank.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 42
Significance of eigenvalues and eigenvectors
1 1 1 12 2 2 2 T
1 1 1 12 2 2 2
2 1 3 0
1 2 0 1
USV
M is symmetric, so MTM = MMT
1
1 1T T 1 1
1 1 1 112 2 2 23
1 1 1 12 2 2 2
2 13 3
1 23 3
2 1
1 2
0
0 1
USV V S U
An orthogonal matrix is very easy to invert:
X-1 = XT A diagonal matrix is very easy to invert:
just reciprocate each diagonal element
BMI2 SS08 – Class 3 “Image Processing 1” Slide 43
Significance of eigenvalues and eigenvectors
1 1 12 3 4
1 1 1 12 3 4 5 1
1 1 1 13 4 5 6
1 1 1 14 5 6 7
1 16 120 240 140
120 1200 2700 1680,
240 2700 6480 4200
140 1680 4200 2800
A A
Eigenvalues of A: 1.5002, 0.16914, 0.0067383, 9.6702×10-5
Eigenvalues of A-1: 0.66657, 5.9122, 148.41, 10341
Any arbitrary vector x is equal to a sum of the eigenvectors of A-1:
x = av1 + bv2 + cv3 + dv4, for some numbers a, b, c, d.
So A-1x = aA-1v1 + bA-1v2 + cA-1v3 + dA-1v4
= 0.66657av1 + 5.9122bv2 + 148.41cv3 + 10341dv4
BMI2 SS08 – Class 3 “Image Processing 1” Slide 44
Significance of eigenvalues and eigenvectors
1 1 12 3 4
1 1 1 1T2 3 4 5
4 4 4 41 1 1 13 4 5 6
51 1 1 14 5 6 7
1 1.5002 0 0 0
0 0.16914 0 0
0 0 0.0067383 0
0 0 0 9.6702 10
A U V
T4 3 3 4
1.5002 0 0
* 0 0.16914 0
0 0 0.0067383
1 0.5 0.33333 0.25
0.5 0.33332 0.25003 0.19998
0.33333 0.25003 0.19994 0.16671
0.25 0.19998 0.16671 0.14283
A U V
BMI2 SS08 – Class 3 “Image Processing 1” Slide 45
Significance of eigenvalues and eigenvectors
1 T4 3 3 4
0.66657 0 0
* 0 5.9122 0
0 0 148.41
7.1869 20.766 1.082 15.338
20.766 0.33332 9.8223 69.076
1.082 9.8223 3.0954 11.097
15.338 69.076 11.097 62.066
A V U
1 0.5 0.33333 0.25
0.5 0.33332 0.25003 0.19998*
0.33333 0.25003 0.19994 0.16671
0.25 0.19998 0.16671 0.14283
A
1
16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
A
BMI2 SS08 – Class 3 “Image Processing 1” Slide 46
Regularization Redux
1 1 12 3 4
1 1 1 12 3 4 5
1 1 1 13 4 5 6
1 1 1 14 5 6 7
1 1 2.0833 16 120 240 140 2.0833
1 1.2833 120 1200 2700 1680 1.2833,
1 0.95 240 2700 6480 4200 0.95
1 0.75952 140 1680 4200 2800 0.75952
1
1
1
1
16 120 240 140 2.0594 1.5351
120 1200 2700 1680 1.2986 8.776But...
240 2700 6480 4200 0.9613 28.408
140 1680 4200 2800 0.75924 18.228
7.1869 20.766 1.082 15.338 2.0833 3.2315
20.766 0.33332 9.8223 69.076 1.2833 1.8588
1.082 9.8223 3.0954 11.097 0.95 1.3319
15.338 69.076 11.097 62.066 0.75952 1.0443
BMI2 SS08 – Class 3 “Image Processing 1” Slide 47
Regularization Redux
16 120 240 140 2.0833 1
120 1200 2700 1680 1.2833 1
240 2700 6480 4200 0.95 1
140 1680 4200 2800 0.75952 1
16 120 240 140
120 1200 2700 1680
240 2700 6480 4200
140 1680 4200 2800
2.0594 1.5351
1.2986 8.776
0.9613 28.408
0.75924 18.228
7.1869 20.766 1.082 15.338 2.0833 3.2315
20.766 0.33332 9.8223 69.076 1.2833 1.8588
1.082 9.8223 3.0954 11.097 0.95 1.3319
15.338 69.076 11.097 62.066 0.75952 1.0443
7.1869 20.766 1.
082 15.338 2.0594 3.219
20.766 0.33332 9.8223 69.076 1.2986 1.8548
1.082 9.8223 3.0954 11.097 0.9613 1.3299
15.338 69.076 11.097 62.066 0.75924 1.0433
BMI2 SS08 – Class 3 “Image Processing 1” Slide 48
What happens if we try to use Gaussian elimination to solve Ax = b, but A is singular?
1 1 1 5
2 3 4 2
4 6 8 9
u
v
w
1 1 1 5
0 1 2 12
0 0 0 13
u
v
w
After second round of elimination:
There is no Solution!
These two equations are inconsistent.
Gaussian Elimination Redux
But there is a pseudoinverse, A+, which we can find by using SVD:
11 1 16 10 5
13
7 1 16 10 5
1 1 1
2 3 4 0 0
4 6 8
BMI2 SS08 – Class 3 “Image Processing 1” Slide 49
TT T
T
T T
,
1 1 1 0.13766 0.9904812.157 0 0.37568 0.55786 0.74004
2 3 4 0.44296 0.0615640 0.45053 0.83198 0.14875 0.53449
4 6 8 0.88591 0.12313
A AAA A A
AA
V xSA U x
How do we compute A+?
11 1 16 10 5
112.157 1
31
0.45053 7 1 16 10 5
T
0.37568 0.831980 0.13766 0.44296 0.88591
0.55786 0.14875 0 00 0.99048 0.061564 0.12313
0.74004 0.53449
S UV
A
BMI2 SS08 – Class 3 “Image Processing 1” Slide 50
As indicated, for this case AA+ ≠ I and A+A ≠ I:
Gaussian Elimination Redux
11 1 16 10 5
1 1 23 5 5
7 1 1 2 46 10 5 5 5
511 1 1 1 16 10 5 6 3 6
1 1 1 13 3 3 3
7 51 1 1 16 10 5 6 3 6
1 1 1 1 0 0
2 3 4 0 0 0
4 6 8 0
1 1 1
0 0 2 3 4
4 6 8
What is the pseudoinverse “solution,” and what is its significance?
227 22711 1 16 10 5 30 30
5 5 1613 3 3 5
7 127 127 321 16 10 5 30 30 5
5 1 1 1 5 5
0 0 2 2 3 4 2
9 4 6 8 9
x A b b Ax
BMI2 SS08 – Class 3 “Image Processing 1” Slide 51
We are not surprised that b+ ≠ b, because we already knew that the original system has no Solution.
Gaussian Elimination Redux
That is, that no linear combination of the columns of A is equal to b.
165
325
2 2 216 32
5 5
1695
5 5
2
9
5 5 2 9
5.8138
Ax b b b
However, the “solution” x+ gives us that linear combination of columns of A which is closest to b, in the sense of minimizing the distance between Ax and b.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 52
Example: Image Time-series Analysis via PCA
1 2
1 1 111 22 2 221 23 3 331 2
1 2
N
N
N
N
M M MMN
p p p
x x xt
x x xt
x x xt
x x xt
1 1 1 1 11 22 2 2 2 21 2
1 2 33 3 3 3 31 2
1 2 3
principal-componentimages
1 2
principal-componenttime series
1 0
0 2
N
NN
NN
M M M M MN
x x x
x x xq q q qs
x x xr r r rs
x x x
BMI2 SS08 – Class 3 “Image Processing 1” Slide 53
METHODS: Target MediumMETHODS: Target Medium
Quasiperiodic
Chaotic (Hénon attractor)
Stochastic
Chaotic (Hénon attractor)
Indicated dynamics were imposed on the inclusions’ μa, which ranged from 0.048 cm-1 to 0.072 cm-1 over time. The remainder of the target had a constant μa of 0.06 cm-1, and the entire target had constant μs = 10 cm-1. Black dots denote source/detector locations.
8 cm
0.6 cm
BMI2 SS08 – Class 3 “Image Processing 1” Slide 54
METHODS: Dynamics ModelsMETHODS: Dynamics Models
y1(t)
y3(t)
y2(t)
y4(t)
Quasiperiodic Chaos 1
Chaos 2 Uniform Stochastic
BMI2 SS08 – Class 3 “Image Processing 1” Slide 55
RESULTS: Statistics of Image Time SeriesRESULTS: Statistics of Image Time Series
-1.5
-1
-0.5
0
0.5
1
1.5x 10
-9Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
0 200 400 600 800 1000-1
-0.5
0
0.5
1x 10
-4 Time-dependent Spatial Mean
0 10 20 3010
-8
10-6
10-4
10-2
100
Singular Values
0 10 20 3050
60
70
80
90
100Cumulative % of Variability
BMI2 SS08 – Class 3 “Image Processing 1” Slide 56
RESULTS: Image Time Series PCARESULTS: Image Time Series PCA
0 20 40 60 80 100 120 140 160 180 200-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
-0.05
0
0.05
0.1
0.15
0.2
0.25
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
-0.05
0
0.05
0.1
0.15
0.2
0.25
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120 140 160 180 200
-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0 20 40 60 80 100 120 140 160 180 200
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1 Spatial and temporal parts of the first five principal components of noise-free image time series. Essentially 100% of all variability is captured in the first four PCs, each of which is a mixture of the model functions.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 57
RESULTS: Image Time Series MSARESULTS: Image Time Series MSA
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0
0.5
1
1.5
2
2.5
x 10-3
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
0
0.5
1
1.5
2
2.5
3
3.5
x 10-3
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120 140 160 180 200
-0.05
-0.04
-0.03
-0.02
-0.01
0
0.01
0.02
0.03
0.04
0.05
0
0.5
1
1.5
2
2.5
3
3.5
4x 10
-3
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
0
0.5
1
1.5
2
2.5
3
x 10-3
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
MS algorithm yields essentially perfect “unmixing” of the four modeled functions, in both the spatial and temporal dimensions.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 58
RESULTS: Detector Time Series PCA-MSARESULTS: Detector Time Series PCA-MSA
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
0 20 40 60 80 100 120 140 160 180 200
-0.05
-0.04
-0.03
-0.02
-0.01
0
0.01
0.02
0.03
0.04
0.05
0
0.1
0.2
0.3
0.4
0.5
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
0 20 40 60 80 100 120 140 160 180 200
-0.06
-0.04
-0.02
0
0.02
0.04
-0.4
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
2 4 6 8 10 12 14 16
2
4
6
8
10
12
14
16
Source
Detector
Application of PCA and MSA directly to the detector time series yields four “unmixed” sets of detector data that capture essentially all of the variability and directly correspond to the four model functions.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 59
RESULTS: Images Reconstructed from Detector MS componentsRESULTS: Images Reconstructed from Detector MS components
-2.5
-2
-1.5
-1
-0.5
0x 10
-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 2 3 4-1.5
-1
-0.5
0
0.5
1Time-dependent Spatial Mean
0 1 2 3 410
-10
10-5
100
Singular Values
0 1 2 3 4100
100
100
100
100
100
100Cumulative % of Variability
-4
-3
-2
-1
0x 10
-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 2 3 4-1.5
-1
-0.5
0
0.5
1Time-dependent Spatial Mean
0 1 2 3 410
-3
10-2
10-1
100
Singular Values
0 1 2 3 498.5
99
99.5
100
100.5
101Cumulative % of Variability
0
0.5
1
1.5
2
2.5
3
x 10-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 2 3 4-1
-0.5
0
0.5
1
1.5Time-dependent Spatial Mean
0 1 2 3 410
-20
10-15
10-10
10-5
100
Singular Values
0 1 2 3 4100
100
100
100
100
100Cumulative % of Variability
0
1
2
3
x 10-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 2 3 4-1
-0.5
0
0.5
1
1.5Time-dependent Spatial Mean
0 1 2 3 410
-11
10-10
10-9
10-8
Singular Values
0 1 2 3 4100
100
100
100
100
100Cumulative % of Variability
Note that most ICA algorithms would be unable to distinguish these two, as they have identical histograms
BMI2 SS08 – Class 3 “Image Processing 1” Slide 60
RESULTS: Image Time Series GLMRESULTS: Image Time Series GLM
0
5
10
15
x 10-5
20 40
10
20
30
40
0
500
1000
1500
2000
20 40
10
20
30
40
0 500 1000-1
-0.5
0
0.5
1
0
5
10
15
x 10-5
20 40
10
20
30
40
0
1000
2000
20 40
10
20
30
40
0 500 1000-1
-0.5
0
0.5
1
0
1
2x 10
-4
20 40
10
20
30
40
0
1000
2000
3000
4000
20 40
10
20
30
40
0 500 1000-1
-0.5
0
0.5
1
0
1
2
x 10-4
20 40
10
20
30
40
0
1000
2000
3000
20 40
10
20
30
40
0 500 1000-1
-0.5
0
0.5
1
-10
-5
0x 10
-5
20 40
10
20
30
40
-1000
-500
0
20 40
10
20
30
40
0 500 10000
0.5
1
1.5
2
Lin
ear
Mod
el
Coe
ffic
ient
st-
stat
isti
c M
aps
Mod
el
Fun
ctio
ns
BMI2 SS08 – Class 3 “Image Processing 1” Slide 61
Noise StudyNoise Study
Modeled N/S ratio increases with increasing angle (distance) between source and detector, in agreement with usual experimental or clinical experience.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 62
RESULTS: Images from Detector MS components, 5% noiseRESULTS: Images from Detector MS components, 5% noise
-1
0
1
2
3
4x 10
-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 1.5 2 2.5 3-1
-0.5
0
0.5
1
1.5Time-dependent Spatial Mean
0 1 2 310
-11
10-10
10-9
10-8
Singular Values
0 1 2 3100
100
100
100
100
100
100Cumulative % of Variability
0
1
2
3
4
5
x 10-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 1.5 2 2.5 3-1
-0.5
0
0.5
1
1.5Time-dependent Spatial Mean
0 1 2 310
-9.8
10-9.6
10-9.4
10-9.2
Singular Values
0 1 2 3100
100
100
100
100
100Cumulative % of Variability
-4
-3
-2
-1
0
1
x 10-3Position-dependent Temporal Mean
10 20 30 40
10
20
30
40
1 1.5 2 2.5 3-1.5
-1
-0.5
0
0.5
1Time-dependent Spatial Mean
0 1 2 310
-11
10-10
10-9
10-8
Singular Values
0 1 2 3100
100
100
100
100
100Cumulative % of Variability
The same qualitative result is obtained at the 3.2% noise level; the deep inclusions merge into a single object, while the peripheral pairs remain largely isolable. When the noise level is 10%, all four dynamic model functions are overwhelmed by it.
BMI2 SS08 – Class 3 “Image Processing 1” Slide 63
0
1
2
x 10-4
20 40
10
20
30
40
0
5
10
15
20 40
10
20
30
40
0
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
5
10
15
x 10-5
20 40
10
20
30
40
0
5
10
15
20 40
10
20
30
40
0
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
10
20x 10
-5
20 40
10
20
30
40
0
20
40
60
80
20 40
10
20
30
40
0
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
10
20x 10
-5
20 40
10
20
30
40
0
20
40
60
80
20 40
10
20
30
40
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
-4
-2
0x 10
-4
20 40
10
20
30
40
-100
-50
0
20 40
10
20
30
40
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 10000
1
2
RESULTS: Image Time Series GLM, 3.2% NoiseRESULTS: Image Time Series GLM, 3.2% NoiseL
inea
r M
odel
C
oeff
icie
nts
t-st
atis
tic
Map
sS
igni
fica
nce
Lev
el M
aps
BMI2 SS08 – Class 3 “Image Processing 1” Slide 64
0
5
10
x 10-5
20 40
10
20
30
40
-2
0
2
4
20 40
10
20
30
40
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
5
10
x 10-5
20 40
10
20
30
40
-2
0
2
4
20 40
10
20
30
40
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
10
20
x 10-5
20 40
10
20
30
40
0
5
10
20 40
10
20
30
40
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
0
10
20x 10
-5
20 40
10
20
30
40
0
5
10
20 40
10
20
30
40
0
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 1000-1
0
1
-10
-5
0x 10
-4
20 40
10
20
30
40
-100
-50
0
20 40
10
20
30
40
0
0.2
0.4
0.6
0.8
20 40
10
20
30
40
0 500 10000
1
2
RESULTS: Image Time Series GLM, 50% NoiseRESULTS: Image Time Series GLM, 50% NoiseL
inea
r M
odel
C
oeff
icie
nts
t-st
atis
tic
Map
sS
igni
fica
nce
Lev
el M
aps
top related