high-quality hyperspectral reconstruction using a spectral...
TRANSCRIPT
High-Quality Hyperspectral Reconstruction Using a Spectral Prior
Inchang Choi† Daniel S. Jeon† Giljoo Nam† Min H. Kim† Diego Gutierrez*
†KAIST * Universidad de Zaragoza, I3A
High-Quality Hyperspectral Reconstruction Using a Spectral Prior
Light and Color Imaging
2
Bayer pattern RGB imaging Continuous spectra of light
Hyperspectral Imaging (HSI)
3
Compressive hyperspectral imaging Hyperspectral imaging
Scene spectra Projection
Reconstruction is an inverse problem of optical imaging
Shear Mask Unshear
Compressive Hyperspectral Imaging
4
Total variation Ground truth
5
Straightforward Approach
• Learning a regression function using a CNN
6
CNN for regression
Scene spectra Projection
The Regression Network Fails
7
regression ground truth
Hyperspectral Reconstruction
Encoder Decoder
Nonlinear representations
8
Hyperspectral reconstruction
Our Approach
9
Our reconstruction
10
Our reconstruction
Our reconstruction
11
Related Work - Hyperspectral Imaging - Compressive Hyperspectral Reconstruction
12
HSI without Reconstruction
13
Bandpass filter [Mansouri et al. 2007]
Pushbroom [Brusco et al. 2006]
LCTF (liquid crystal tunable filter)
[Attas et al. 2003]
HSI with Reconstruction
14
Scene Spectra Compressive image
Reconstruction
CASSI [Wagadarikar et al. 2008]
DD-CASSI [Gehm et al. 2007]
SS-CASSI [Lin et al. 2014]
[Jeon et al. 2016]
Image Formation
15
Shear Spectra Mask Unshear and project Observed image
Observation(2D)
Spectra(3D)
Lightmodulation(3Dto2D)
Hyperspectral Reconstruction
16
underdetermined system
# equations # unknowns ≪ 2
2
“Find a hyperspectral image that satisfies the image formation”
Reconstruction using TV-L1 Prior
17
1
𝛁𝒙𝒚 𝝉 2
2
TwIST [Bioucas-Dias and Figueiredo 2007]
SpaRSA [Wright et al. 2009]
- TV-L1 is very common in computational photography
Reconstruction using Sparse Coding
18
2
2 1
- Use an overcomplete dictionary and a sparse code to represent a data
[Lin et al. 2014]
For all overlapping image patches
D: a dictionary
: a sparse code
Autoencoder - For Our Deep Spectral Prior
19
Autoencoder
20
[Hinton and Salakhutdinov 2006]
x1
x2
x3
x4
y1
y2
y3
y4
Nonlinear Representation
21
x1
x2
x3
x4
y1
y2
y3
y4
Nonlinear representation
Autoencoder: Encoder and Decoder
22
x1
x2
x3
x4
y1
y2
y3
y4
Encoder : generate nonlinear representation
Decoder : produce data from representations
Nonlinear representation
Hyperspectral Reconstruction - Learning a Spectral Prior - Reconstruction with Alpha-fidelity
23
Overview of Our Reconstruction
24
Convolutional autoencoder
Hyperspectral image datasets
Reconstructed hyperspectral
image
Encoder / decoder
Optimization Compressive sensor input
Learning hyperspectral image prior
Hyperspectral reconstruction
Decoder
Autoencoder of Hyperspectral Images
25
Encoder Decoder
Convolutional autoencoder of hyperspectral images
Encoder
Nonlinear representation
Autoencoder of Hyperspectral Images
26
- 64 feature maps
- 3 x 3 convolution without pooling - ReLU activation function
64 64 64
Training Data
27
[Yasuma et al. 2006] Columbia dataset
[Chakrabarti and Zickler 2011] Harvard dataset
Validating Autoencoder
28
ground truth reconstruction (44.24 dB / 0.98)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
420 470 520 570 620 670
refle
ctan
ce
wavelength [nm]
GT gray patch red flower
Our Reconstruction - Data Term
29
Observation(2D)
Spectra(3D)
Lightmodulation(3Dto2D)
Shear Spectra Mask Unshear and project Observed image
Our Reconstruction - Data Term
30
Encoder Decoder
Nonlinearrepresentation
𝝰
Our Reconstruction
31
2
2
1
𝛁𝒙𝒚 𝝉
𝝰
How can we utilize the encoder?
32
Decoder - produce h (hyperspectral images)
from (nonlinear representations) - a prior on h
- know how h looks like
Encoder - generate from h
- know how looks like - a prior on
Decoder
Encoder
Our Reconstruction with fidelity Prior
33
2
2
2
2
𝝉1 1
𝛁𝒙𝒚 𝝉2
“The nonlinear representation should be close to what the encoder knows.”
Impact of fidelity Prior
34
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
420 470 520 570 620 670
refle
ctan
ce
wavelength [nm]
GT
w/o 𝜶-prior
w/ 𝜶-prior
Yellow feather
Results - Our Dataset - Synthetic Results - With a Real Compressive Imager
35
Our High-Quality Dataset
36
Download from http://vclab.kaist.ac.kr
Synthetic Result with Our High Quality Dataset
37
groundtruth
ours
Ourdataset
34.20dB / SSIM:0.95 32.03dB / SSIM:0.95 32.29dB / SSIM:0.92 39.21dB / SSIM:0.97
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14
420 470 520 570 620 670
refle
ctan
ce GT
TwIST (0.006)
SpaRSA (0.010)
SC (0.019)
Ours (0.006)
TwIST[Kim2012]
SpaRSA[Wright2009]
sparsecoding[Lin2014]
38
groundtruth
TwIST[Kim2012]
SpaRSA[Wright2009]
sparsecoding[Lin2014] ours
28.42dB / SSIM:0.91 28.57dB / SSIM:0.92 34.89dB / SSIM:0.96 35.40dB / SSIM:0.97
Columbiadataset
0.00
0.05
0.10
0.15
0.20
0.25
0.30
420 470 520 570 620 670
refle
ctan
ce
GT
TwIST (0.058)
SpaRSA (0.062)
SC (0.039)
Ours (0.016)
Synthetic Result with Columbia Dataset [Yasuma et al. 2010]
39
CompressiveinputGroundtruthOurreconstruction
Synthetic Result with Our High Quality Dataset
Synthetic Result with Our High Quality Dataset
40
CompressiveinputGroundtruthOurreconstruction
Our DD-CASSI Result
41
camera coded aperture
prism prism objective lens
relay lens
imaging lens
[Gehm et al. 2007]
Our DD-CASSI Result
42
1
2
OurreconstructionCompressiveInput
0.0
0.2
0.4
0.6
0.8
1.0
420 470 520 570 620 670
GT Ours
0.0
0.2
0.4
0.6
0.8
1.0
420 470 520 570 620 670wavelength[nm]
1
2
Applications - Spectral Interpolation - Hyperspectral Demosaicing
43
Changing Modulation Matrix
44
2
2
2
2 1
for super-resolution: blurring + downsampling
Our reconstruction:
Note: the observation i should be modified accordingly
Spectral Interpolation
45
GroundTruth
Spectral Interpolation
46
16channels(52%)
Spectral Interpolation
47
8channels(26%)
Spectral Interpolation
48
3channels(10%)
49
Spectral Interpolation
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
420 470 520 570 620 670
refle
ctan
ce
wavelength [nm]
GT
52% (0.011)
26% (0.014)
10% (0.077)
16channels
8channels
3channels
GroundTruth
Hyperspectral Demosaicing
50
450nm 520nm
580nm 650nm
Bayer image
Hyperspectral Demosaicing
51
Ground truth
Hyperspectral Demosaicing
52
31.04dB / SSIM:0.89
0.00
0.10
0.20
0.30
0.40
420 470 520 570 620 670
refle
ctan
ce
wavelength [nm]
GT Bayer (0.020)
Demosaiced
Conclusion
53
Conclusion
• Learned a spectral prior using a convolutional autoencoder
• Proposed a novel hyperspectral reconstruction using the learned prior
• Demonstrated interesting applications
• Published a high quality hyperspectral dataset
54
Acknowledgments • Seung-Hwan Back, Incheol Kim, Adrian Jarabo, and Paz Hernando
• Min H. Kim acknowledges • Korea NRF grants (2016R1A2B2013031, 2013M3A6A6073718) • Giga Korea Project (GK17P0200) • MCST • Samsung Electronics (SRFC-IT1402-02) • ICT R&D program of MSIT/IITP of Korea (R7116-16-1035)
• Diego Gutierrez acknowledges • ERC under EU’s Horizon 2020 research • CHAMELEON project (682080) • The Spanish Ministerio de Economia y Competitividad (TIN2016-78753-P)
55