information geometry with applications in components...
TRANSCRIPT
![Page 1: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/1.jpg)
Information Geometry with Information Geometry with Applications in Components Applications in Components
Analysis and XAnalysis and X--Ray CT ImagingRay CT ImagingJoseph A. OJoseph A. O’’SullivanSullivan
Electronic Systems and Signals Research LaboratoryDepartment of Electrical and Systems Engineering
Washington [email protected]
http://essrl.wustl.edu/~jao
Supported by: ONR, ARO, NIHONR, ARO, NIH
![Page 2: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/2.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
2
CollaboratorsCollaborators
Michael D. DeVoreNatalia A. SchmidMetin OzRyan MurphyJasenka Benac
Donald L. SnyderWilliam H. SmithDaniel R. FuhrmannJeffrey F. WilliamsonBruce R. WhitingChrysanthe PrezaDavid G. PolitteG. James Blaine
Faculty Students and Post-Docs
![Page 3: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/3.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
3
OutlineOutline• Applications
- Components Analysis Information Value DecompositionHyperspectral Imaging
- Transmission Tomography- Maximum Likelihood Mixture
Model Estimation• Information Geometry
- Properties of I-Divergence- Projections
• Alternating Minimization Algorithms• Applications Revisited• Conclusions
![Page 4: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/4.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
4
OutlineOutline• Applications
- Components Analysis Information Value DecompositionHyperspectral Imaging
- Transmission Tomography- Maximum Likelihood Mixture
Model Estimation• Information Geometry
- Properties of I-Divergence- Projections
• Alternating Minimization Algorithms• Applications Revisited• Conclusions
![Page 5: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/5.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
5
Markov ApproximationsMarkov Approximations• X and Y RV’s on finite sets X and Y, p(x,y) unknown• Data: N i.i.d. pairs {Xi,Yi}
–Histogram is a matrix S
• Log likelihood function is log probability of S
• Unconstrained ML estimate
Nyxnyxs ),(),( =
∑ ∑
∑ ∑
∈ ∈
∈ ∈
=
=
X Y
X Y
x y
x y
yxpyxs
yxpyxnN
SPN
),(log),(
),(log),(1)(log1
),(),(1),( yxsyxnN
yxp ==
![Page 6: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/6.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
6
Equivalence of ML EstimationEquivalence of ML Estimationand Minimum Iand Minimum I--DivergenceDivergence
•• Define the information (I) divergenceDefine the information (I) divergence
•• Log likelihood function, Lagrange multiplierLog likelihood function, Lagrange multiplier
•• Constrained ML estimation: family of distributionsConstrained ML estimation: family of distributions
),(),(),(),(ln),()||( yxpyxs
yxpyxsyxsPSI
x y+−⎥
⎦
⎤⎢⎣
⎡= ∑ ∑
∈ ∈X Y
∑ ∑∑ ∑∈ ∈∈ ∈
−X YX Y x yx y
yxpyxpyxs ),(),(ln),(
F∈p
![Page 7: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/7.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
7
Markov Approximations:Markov Approximations:Components AnalysisComponents Analysis
•• Lower rank Markov approximation Lower rank Markov approximation X X M M YYMM in a set of cardinality in a set of cardinality KK
•• Factor analysis, contingency tables, economicsFactor analysis, contingency tables, economics•• Problem: Approximation of one matrix by another of Problem: Approximation of one matrix by another of
lower ranklower rank•• C. C. EckartEckart and G. Young, and G. Young, PsychometrikaPsychometrika, vol. 1, pp. , vol. 1, pp.
211211--218 1936.218 1936.•• SVD SVD IVDIVD
AP Φ=
)||(min ASIA ΦΦ
∑=
=K
kkk yaxyxp
1)()(),( φ
![Page 8: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/8.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
8
Spectral Components AnalysisSpectral Components Analysis• Data spectrum for each element (pixel)
• Model: linear combination of constituent spectra,
• Problem: Estimate constituents and proportions subject to nonnegativity; positivity of S assumed
• Ambiguity if α > 0, φ1 − α φ2 > 0 , − α φ1 + φ2 > 0
• Comments: Radiometric Calibration; Constraints Fundamental
∑=
=K
kkjkj as
1φ AS Φ=
Ijs +ℜ∈
0≥kja
![Page 9: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/9.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
9
Idealized Problem Statement:Idealized Problem Statement:MaximumMaximum--Likelihood Likelihood Minimum IMinimum I--divergencedivergence
•• Poisson distributed data Poisson distributed data loglikelihoodloglikelihood functionfunction
•• Maximization over Maximization over ΦΦ and A equivalent to and A equivalent to minimization of Iminimization of I--divergencedivergence
AS Φ=ˆ
∑∑ ∑∑= = == ⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
−⎥⎥⎦
⎤
⎢⎢⎣
⎡=Φ
I
i
J
j
K
kkjik
K
kkjikij aasASl
1 1 11ln)|( φφ
∑∑ ∑∑= =
== ⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
+−⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=Φ
I
i
J
j
Kk kjikijK
k kjik
ijij as
a
ssASI
1 11
1
ln)||( φφ
•• Information Value Decomposition ProblemInformation Value Decomposition Problem
![Page 10: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/10.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
10
HyperspectralHyperspectral ImagingImagingat Washington Universityat Washington University
Donald L. SnyderDonald L. SnyderWilliam H. SmithWilliam H. SmithDaniel R. Daniel R. FuhrmannFuhrmannJoseph A. OJoseph A. O’’SullivanSullivanChrysantheChrysanthe PrezaPreza
WU Team
Digital Array Scanning Interferometer (DASI)
![Page 11: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/11.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
11
HyperspectralHyperspectral ImagingImaging• Scene Cube Data Cube• “Drink from a fire hose”• Filter wheel, interferometer,
tunable FPAs• Modeling and processing:
- data models- optimal algorithms- efficient algorithms
![Page 12: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/12.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
12
CT Imaging in Presence of High CT Imaging in Presence of High Density AttenuatorsDensity Attenuators
BrachytherapyBrachytherapy applicators applicators AfterAfter--loading loading colpostatscolpostats
for radiation oncologyfor radiation oncology
Cervical cancer: 50% survival rateCervical cancer: 50% survival rateDose prediction importantDose prediction important
ObjectObject--Constrained Computed Constrained Computed TomographyTomography (OCCT)(OCCT)
![Page 13: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/13.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
13
Filtered Back ProjectionFiltered Back Projection
Truth FBP
FBP: inverse Radon transform
![Page 14: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/14.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
14
Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x• Data d(y) Poisson, means g(y:µ), log likelihood function
• Mean unattenuated counts I0, mean background β• Attenuation function µ(x,E), E energies
• Maximize over µ or ci; equivalently minimize I-divergence
)(),(),(exp),():(
):():(ln)()):(|(
0 yExxyhEyIyg
ygygydgdl
E x
y
βµµ
µµµ
+⎟⎟⎠
⎞⎜⎜⎝
⎛−=
−=⋅
∑ ∑
∑
∈
∈
X
Y
∑=
=I
iii ExcEx
1)()(),( µµ
![Page 15: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/15.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
15
Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence
Difficulties: log of sum, sums in exponent
)()()(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
10 yExcxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
E x
I
iii
y
y
βµµ
µµ
µ
µµµ
+⎟⎟⎠
⎞⎜⎜⎝
⎛−=
+−=⋅
−=⋅
∑ ∑ ∑
∑
∑
∈ =
∈
∈
X
Y
Y
![Page 16: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/16.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
16
OutlineOutline• Applications
- Components Analysis Information Value DecompositionHyperspectral Imaging
- Transmission Tomography- Maximum Likelihood Mixture
Model Estimation• Information Geometry
- Properties of I-Divergence- Projections
• Alternating Minimization Algorithms• Applications Revisited• Conclusions
![Page 17: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/17.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
17
Information Geometry:Information Geometry:Properties of IProperties of I--DivergenceDivergence
• I-divergence is nonnegative, convex in pair (p,q)• Generalization of relative entropy; not symmetric;
example of f-divergence (Csiszár)• Let P be a probability matrix. Then
• First projection property
∑ +−=i
iii
ii qp
qppqpI ln)||(
)||()||( qpIPqPpI ≤
)/||/()||()||( BqApAIBAIqpI
qBpAi
ii
i
+=
== ∑∑
![Page 18: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/18.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
18
Information Geometry:Information Geometry:Projections Using IProjections Using I--DivergenceDivergence
• Define the linear family of probability distributions
• Theorem. Suppose that q and L are given. Let p* in Lachieve
then for all p in L
{ }bAppbA n =ℜ∈= + :),(L
)||()||()||(
)||(minarg
**
),(
*
qpIppIqpI
qpIpbAp
+=
=∈L
![Page 19: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/19.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
19
Information Geometry:Information Geometry:Projections Using IProjections Using I--DivergenceDivergence
• Define the exponential family of probability distributions
• Theorem. Suppose that p and E are given. Let q* in E achieve
• then for all q in E , )||()||()||(
)||(minarg
**
),(
*
qqIqpIqpI
qpIqBq
+=
=∈ πE
( ){ }ννππ somefor ,exp:),( ∑=ℜ∈= + k kkiiin bqqBE
![Page 20: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/20.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
20
Comment on ProofsComment on ProofsDuality Theorem: the two problems below are (Fenchel) dual, with solutions q* = p*.
)||(min
)||(min
),(
),(
ππ
pI
qdI
dBBp
Bq
TTL
E
∈
∈
The resulting values of the objective functions satisfy
)||()||()||( ** ππ qIqdIdI +=
![Page 21: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/21.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
21
More Information GeometryMore Information Geometry……• Shun-ichi Amari, Imre Csiszár
• Two types of information geodesics:–Linear, m-projections–Exponential, e-projections
• Differential geometry on manifold of probability density functions
• Fisher Information is the Riemannian metric
• Exponential family e-flat manifold dually flat Riemannian space
• Dual parameterization: mean and exponential family parameter
![Page 22: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/22.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
22
VariationalVariational RepresentationsRepresentations•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then
• Special Case: f is ln
• Basis for EM; see also De Pierro, Lange, Fessler
∑
∑ ∑≥=
≤
iii
i ii
irii
rr
xfrxf
0,1
)()( 1
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
=≥=
−=⎟⎟⎠
⎞⎜⎜⎝
⎛
∑
∑∑∈
iii
i i
ii
pii
ppp
qppq
1,0:
lnminln
P
P
![Page 23: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/23.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
23
Information Value Decomposition:Information Value Decomposition:Components Analysis
• Minimize over Φ and A
• Convex Decomposition Lemma
∑∑ ∑∑= =
== ⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
+−⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=Φ
I
i
J
j
Kk kjikijK
k kjik
ijij as
a
ssASI
1 11
1
ln)||( φφ
kjikijijk
I
i
J
j
K
k kjik
ijijkijijk
PQ
PQ
asqasq
sq
ASQIASI
φφ
+−=
Φ=Φ
∑∑∑= = =∈
∈
|1 1 1
|| lnmin
)||(min)||( οο
![Page 24: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/24.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
24
Alternating Minimization AlgorithmAlternating Minimization Algorithm• Solution is result of double minimization over
exponential and linear families
∑ ∑∑
∑∑
∑
∑
= =+
=+
+
= ==
++
=
+
∈ΦΦ
=
==
=
Φ=Φ
Jj
Ii ij
lijk
Ii ij
lijkl
kj
J
jKk
ljk
lik
ijl
kjlik
J
jij
lijk
lik
Kk
ljk
lik
lkj
likl
ijk
PQAA
sq
sqa
a
sasq
a
aq
ASQIASI
1' 1 ')1(
'|
1)1(
|)1(
1 1')('
)('
)()(
1
)1(|
)1(
1')('
)('
)()()1(
|
,,)||(minmin)||(min
φφφ
φ
φ
οο
![Page 25: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/25.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
25
Comments on SolutionComments on Solution
• Blind estimation, Poisson data– T. Holmes, et al.– Katsaggelos, et al.
• Nonnegative matrix factorization–Lee and Seung, 1999, Nature.
![Page 26: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/26.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
26
Alternating Minimization AlgorithmsAlternating Minimization Algorithms
• Define problem as minq φ(q)• Derive variational representation: φ(q) = minp J(p,q)• J is an auxiliary function p is in auxiliary set P• Result: double minimization minq minp J(p,q)• Alternating minimization algorithm
),(minarg
),(minarg
)1()1(
)()1(
qpJq
qpJp
l
l
l
Pp
l
+
∈
+
∈
+
=
=
Comments: Guaranteed Monotonicity; J selected carefully
![Page 27: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/27.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
27
Alternating Minimization AlgorithmsAlternating Minimization Algorithms::II--Divergence, Linear, Exponential FamiliesDivergence, Linear, Exponential Families• Special Case of Interest: J is I-divergence• Families of Interest:
Linear Family L(A,b) = {p: Ap = b}Exponential Family E(π,B) = {q: qi = πi exp[Σj bij νj]}
)||(minarg
)||(minarg
)1(
)
)1(
)(
),(
)1(
qpIq
qpIp
l
B(q
l
l
bAp
l
+
∈
+
∈
+
=
=
,E
L
π
Csiszár and Tusnády; Dempster, Laird, Rubin; Blahut; Richardson; Lucy; Vardi, Shepp, and Kaufman; Cover;Miller and Snyder; O'Sullivan
![Page 28: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/28.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
28
Alternating Minimization ExampleAlternating Minimization Example• Linear family: p1 + 2 p2 = 2• Exponential family: q1 = exp (v), q2 = exp (-v)
0 1 2 3 4 50
1
2
3
4
5
p1, q1
p 2, q2
0 1 2 3 4 50
1
2
3
4
5
p1, q1
p 2, q2
0.6 0.8 1 1.2 1.40.2
0.4
0.6
0.8
1
1.2
p1, q1
p 2, q2
0.6 0.8 1 1.2 1.40.2
0.4
0.6
0.8
1
1.2
p1, q1
p 2, q2
)||(minmin qpILpEq ∈∈
![Page 29: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/29.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
29
Alternating Minimization AlgorithmsAlternating Minimization Algorithms• Projections and triangle equality
• Bounded sums (depending on initial condition)
• Monotonicity; limit points exist, form connected set
)||()||()||(
)||()||()||()()1()1()1()()1(
)()1()1()()()(
llllll
llllll
qqIqpIqpI
qpIppIqpI++++
++
+=
+=
∑
∑∞
=
+
∞
=
+
1
)()1(
1
)1()(
)||(
)||(
l
ll
l
ll
qqI
ppI
![Page 30: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/30.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
30
Information Value Decomposition:Information Value Decomposition:Changing Dimension
• Minimize over Φ and A:
• Change K to K+1 or K-1
∑∑ ∑∑= =
== ⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
+−⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=Φ
I
i
J
j
Kk kjikijK
k kjik
ijij as
a
ssASI
1 11
1
ln)||( φφ
)||(
)||()||()()(
)()()()(
KK
KKKK
qpI
ASQIBSI
=
Φ= οο
![Page 31: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/31.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
31
Information Value Decomposition:Information Value Decomposition:Changing Dimension
L(S)
EKEK+1
EK-1p(K+1)
p(K)
p(K-1)
q(K+1)
q(K)
q(K-1)
![Page 32: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/32.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
32
OutlineOutline• Applications
- Components Analysis Information Value DecompositionHyperspectral Imaging
- Transmission Tomography- Maximum Likelihood Mixture
Model Estimation• Information Geometry
- Properties of I-Divergence- Projections
• Alternating Minimization Algorithms• Applications Revisited• Conclusions
![Page 33: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/33.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
33
S=ΦAGiven Φ and S, estimate A.
Alternating Minimization AlgorithmsAlternating Minimization Algorithmsfor for HyperspectralHyperspectral ImagingImaging
α1
α2
α3
+ PoissonProcess
C
AlternatingMinimization
Algorithm
InitialEstimated
Coefficients
EstimatedCoefficients
![Page 34: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/34.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
34
Alternating MinimizationsAlternating MinimizationsApplied to Applied to HyperspectralHyperspectral DataData
• Downloaded spectra from USGS website• 470 Spectral components• Randomly generated A with 2000 columns• Ran IVD on result
![Page 35: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/35.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
35
Alternating MinimizationsAlternating MinimizationsApplied to Applied to HyperspectralHyperspectral DataData
![Page 36: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/36.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
36
CT Minimum ICT Minimum I--Divergence FormulationDivergence FormulationMaximize log likelihood function over µ
minimize I-divergenceUse Convex Decomposition Lemmag is a marginal over q
∑ ∑ ∑
∑
∑
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
+−=⋅
−=⋅
∈ =
∈
∈
E x
I
iii
y
y
ExcxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
X
Y
Y
10 )()(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
µµ
µµ
µ
µµµ
![Page 37: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/37.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
37
New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography
⎭⎬⎫
⎩⎨⎧
==
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
+−=
∑
∑ ∑
∑ ∑
∈ =
∈∈
E
x
I
iii
y Eqp
ydEypEyp
ExcxyhEyIEyq
EyqEypEyqEypEypqpI
)(),(:),(
)()(),(exp),(),(
),(),(),(),(ln),()||(minmin
10
L
X
YL
µ
Data determine the linear familyExponential family parameters are image(s)
![Page 38: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/38.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
38
Interpretation: Compare predicted data to measured data via ratio of backprojectionsUpdate estimate using a normalization constant
Comments: Choice for constants; monotonic convergence;Linear convergence; Constraints easily incorporated
)(~)(ˆ
ln)(
1)(ˆ)(ˆ)(
)()()1(
xbxb
xZxcxc l
i
li
i
li
li −=+
New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography
![Page 39: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/39.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
39David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000001
![Page 40: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/40.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
40David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000002
![Page 41: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/41.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
41David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000005
![Page 42: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/42.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
42David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000010
![Page 43: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/43.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
43David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000020
![Page 44: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/44.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
44David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000050
![Page 45: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/45.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
45David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000100
![Page 46: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/46.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
46David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000200
![Page 47: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/47.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
47David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000500
![Page 48: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/48.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
48David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0001000
![Page 49: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/49.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
49David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0002000
![Page 50: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/50.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
50David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0005000
![Page 51: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/51.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
51David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0010000
![Page 52: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/52.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
52David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0020000
![Page 53: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/53.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
53David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0050000
![Page 54: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/54.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
54David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0100000
![Page 55: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/55.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
55David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0200000
![Page 56: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/56.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
56David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0500000
![Page 57: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/57.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
57David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 1000000
![Page 58: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/58.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
58
Iterative Algorithm with Known Iterative Algorithm with Known Applicator PoseApplicator Pose
Truth Known Pose
![Page 59: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/59.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
59
OCCT IterationsOCCT Iterations
Known Pose
OCCT
![Page 60: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/60.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
60
TruthTruth Known
StandardFBP
OCCT
![Page 61: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/61.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
61
Magnified views around Magnified views around brachytherapybrachytherapy applicatorapplicator
Truth OCCT
FBP
![Page 62: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/62.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
62
ConclusionsConclusions
•• Applications: Applications: Components AnalysisComponents AnalysisCT ImagingCT Imaging
•• Information GeometryInformation Geometry•• Alternating Minimization Alternating Minimization
AlgorithmsAlgorithms•• Information Value DecompositionInformation Value Decomposition
![Page 63: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/63.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
63
![Page 64: Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information](https://reader034.vdocuments.site/reader034/viewer/2022042920/5f63fb52224f0d331a231e22/html5/thumbnails/64.jpg)
J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry
64
OO’’Sullivan Give a Math Talk?Sullivan Give a Math Talk?• Use Mathematics, interested, encourage
students,…
• Erdös mumber is 3
• Proof:– Szego Number is 2
Guido Weiss has Erdös number 2.