computer vision 691a: joint alignmentelm/teaching/691a_f11/...vidit jain, andras ferencz, gary...
TRANSCRIPT
![Page 1: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/1.jpg)
Computer Science
Erik Learned-Miller, with Vidit Jain, Andras Ferencz, Gary
Huang, Lilla Zollei, Sandy Wells, ….
Computer Vision 691A: Joint Alignment
![Page 2: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/2.jpg)
2 Learned-Miller
Congealing (CVPR 2000, PAMI 2006)
![Page 3: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/3.jpg)
3 Learned-Miller
MRI Bias Removal (NIPS 2005, MICCAI 2005, PAMI 2006) !
![Page 4: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/4.jpg)
4 Learned-Miller
Five Applications
Image factorizations • For transfer learning, learning from one example
Alignment for Data Pooling • 3D MR registration • EEG registration
Artifact removal • Magnetic resonance bias removal
Improvements to recognition algorithms • Alignment before recognition
Defining anchor points for registration • Find highly repeatable regions for future registrations
![Page 5: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/5.jpg)
5 Learned-Miller
Congealing
Process of joint “alignment” of sets of arrays (samples of continuous fields).
3 ingredients • A set of arrays in some class • A parameterized family of continuous transformations • A criterion of joint alignment
![Page 6: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/6.jpg)
6 Learned-Miller
Congealing Binary Digits
3 ingredients • A set of arrays in some class:
• Binary images • A parameterized family of continuous transformations:
• Affine transforms • A criterion of joint alignment:
• Entropy minimization
![Page 7: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/7.jpg)
7 Learned-Miller
Congealing
![Page 8: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/8.jpg)
8 Learned-Miller
Criterion of Joint Alignment Minimize sum of pixel stack
entropies by transforming each image.
A pixel stack
Note: Mutual Information doesn’t make sense here.
![Page 9: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/9.jpg)
9 Learned-Miller
Observed Image
“Latent Image”
Transform
(Previous work by Grenander,, Frey and Jojic.)
An Image Factorization
![Page 10: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/10.jpg)
10 Learned-Miller
A pixel stack
![Page 11: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/11.jpg)
11 Learned-Miller
Why Minimize Entropy?
Negative entropy is just the average log likelihood of points under their own distribution.
Min entropy = maximum non-parametric likelihood
![Page 12: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/12.jpg)
12 Learned-Miller
The Independent Pixel Assumption Model assumes independent pixels A poor generative model:
• True image probabilities don’t match model probabilities. • Reason: heavy dependence of neighboring pixels.
However! This model is great for alignment and separation of causes!
• Why? • Relative probabilities of “better aligned” and “worse aligned” are
usually correct.
Once components are separated, a more accurate (and computationally expensive) model can be used to model each component.
![Page 13: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/13.jpg)
13 Learned-Miller
Before After
Each pair implicitly creates a sample of the transform T.
Congealing
A transform
![Page 14: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/14.jpg)
14 Learned-Miller
Character Models
Latent Images
Transforms
Image Kernel Density Estimator
(or other estimator)
Transform Kernel Density Estimator
(CVPR 2003)
Latent Image Probability Density
for Zeroes
P(IL)
Transform Probability Density
for Zeroes
P(T) Con
geal
ing
![Page 15: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/15.jpg)
15 Learned-Miller
How do we line up a new image?
Sequence of successively “sharper” models !
…!
step 0 step 1 step N!
…!
Take one gradient step with respect to each model.!
![Page 16: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/16.jpg)
16 Learned-Miller
Digit Models from One Example
![Page 17: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/17.jpg)
17 Learned-Miller
Next Application: Alignment of 3D Magnetic Resonance Volumes
Lilla Zollei, Sandy Wells, Eric Grimson
![Page 18: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/18.jpg)
18 Learned-Miller
Congealing MR Volumes: Joint Registration
3 ingredients • A set of arrays in some class:
• Gray-scale MR volumes • A parameterized family of continuous transformations:
• 3-D affine transforms • A criterion of joint alignment:
• Grayscale entropy minimization
Purposes: • Pooling data for fMRI studies • Building general purpose statistical anatomical atlases
![Page 19: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/19.jpg)
19 Learned-Miller
Why Entropy?
Drives volumes to having mass concentrated in a small number of tissues.
Comparison to Transformed Mixture of Gaussians (Frey and Jojic).
Convexity of entropy in distribution.
![Page 20: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/20.jpg)
20 Learned-Miller
Congealing Gray Brain Volumes (ICCV 2005 Workshop)
![Page 21: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/21.jpg)
21 Learned-Miller
Aligned Volumes
![Page 22: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/22.jpg)
22 Learned-Miller
Validation: Synthetic Data
![Page 23: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/23.jpg)
23 Learned-Miller
Real Data
![Page 24: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/24.jpg)
24 Learned-Miller
Grayscale Entropy Minimization Fr
eque
ncy
of o
ccur
renc
e in
imag
e
Image intensity
White-gray separation?
![Page 25: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/25.jpg)
25 Learned-Miller
MR Congealing Challenges
Big data • 8 million voxels per patient • 100 patients • 12 transform parameters • 20 iterations
Techniques: • Stochastic sampling • Multi-resolution techniques • Don’t use visual basic
![Page 26: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/26.jpg)
26 Learned-Miller
Signal to Noise in Event Related Potentials
Before!congealing!
After!congealing!
![Page 27: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/27.jpg)
27 Learned-Miller
Next Application: Bias Removal in Magnetic Resonance Images
Parvez Ahammad, Vidit Jain
![Page 28: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/28.jpg)
28 Learned-Miller
The Problem
Ideal Image
Observed Image
Bias Field
Bias fields have low spatial frequency content
![Page 29: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/29.jpg)
29 Learned-Miller
Bias Removal in MR as a Congealing Problem
3 ingredients • A set of arrays in some class:
• MR Scans of Similar Anatomy (2D or 3D) • A parameterized family of continuous transformations:
• Smooth brightness transformations • A criterion of joint alignment:
• Entropy minimization
![Page 30: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/30.jpg)
30 Learned-Miller
Congealing with brightness transforms
![Page 31: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/31.jpg)
31 Learned-Miller
Grayscale Entropy Minimization Fr
eque
ncy
of o
ccur
renc
e in
imag
e
Image intensity
White-gray separation?
![Page 32: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/32.jpg)
32 Learned-Miller
Some Infant Brains (thanks to Inder, Warfield, Weisenfeld)
Pretty well registered (not perfect) Pretty bad bias fields
![Page 33: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/33.jpg)
33 Learned-Miller
Fourier Basis for Smooth Bias Fields
![Page 34: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/34.jpg)
Results
Original Images
Bias Corrected
Images
![Page 35: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/35.jpg)
35 Learned-Miller
Assumptions
Pixels in same location, across images, are independent. • When is this not true?
• Systematic bias fields. Pixels in same image are independent, given their location.
• Clearly not true, but again, doesn’t seem to matter. Bias fields are truly bandlimited.
![Page 36: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/36.jpg)
36 Learned-Miller
Some Other Recent Approaches Minimize entropy of intensity distribution in single image
• Viola (95) • Warfield and Weisenfeld extensions (current)
Wells (95) • Use tissue models and maximize likelihood • Use Expectation Maximization with unknown tissue type
Fan (02) • Incorporate multiple images from different coils, but same patient.
![Page 37: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/37.jpg)
37 Learned-Miller
Potential difficulties with single image method
If there is a component of the brain that looks like basis set, it will get eliminated.
Does this occur in practice? • Yes!
![Page 38: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/38.jpg)
38 Learned-Miller
![Page 39: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/39.jpg)
39 Learned-Miller
![Page 40: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/40.jpg)
40 Learned-Miller
MRI Bias Removal
![Page 41: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/41.jpg)
![Page 42: Computer Vision 691A: Joint Alignmentelm/Teaching/691A_F11/...Vidit Jain, Andras Ferencz, Gary Huang, Lilla Zollei, Sandy Wells, …. Computer Vision 691A: Joint Alignment . Learned-Miller](https://reader035.vdocuments.site/reader035/viewer/2022071212/602568393b75ad733c5015a5/html5/thumbnails/42.jpg)
42 Learned-Miller
Summary
Remove source of variability • MR bias removal • MR anatomical alignment • ERP signal alignment • Better alignment for recognition (hyper-features)
Model a source of variability • Form factorized models (learning from one example)
Define points of high saliency and repeatability (anchor points) for difficult registration problems