Imager Design using Object-Space Prior KnowledgeM. A. Neifeld

University of Arizona

OUTLINE

1. The Last Slot

2. Introduction

3. PSF Engineering

4. Feature-Specific Imaging

Neifeld IMA 2005

Introduction: objects are not iid pixels.- Conventional cameras are designed to image iid pixels

à impulse-like point-spread-functions (identity transformation)

à generic metrics such as resolution, field of view, SNR, etc.

- Real objects are not iid pixels so don’t estimate pixels

- This keeps the compression guys employed!

- (106 pixels)(3 colors/pixel)(8 bits/color) = 2.4x107 bits

- (1011 people)(4x109 years)(109 images/year) = 4x1029 images à <100 bits

- The set of “interesting” objects is small

- Many ways to characterize “interesting” objects: power spectra, principal components, Markov fields, wavelet projections, templates, task-specific models, finite alphabets, etc.

Information depends upon task:q Option 1 - this is a random image è I = 107 bits

q Option 2 – this is a “battlefield” image è I = ? bits

… how to quantify PDF!

q Option 3 – this image either contains a tank or not è I = 1bit

… task-specific source model

Neifeld IMA 2005

Introduction: post-processing exploits priors.- Linear Restoration: de-noising and de-blurring exploit noise statistics, object power

spectra, principal components, wavelets, …

- Nonlinear Restoration: super-resolution uses finite support, positivity, finite alphabet, power spectra, wavelets, principal components, isolated points, …

- Recognition: features, templates, image libraries, syntax, invariance, …

- Finite Alphabet Post-Processing Examples

LADAR Multi-Frame Super-Resolution

Object

10

15

20

25

30

35

40

45

50

55

largest returnrm

se= 7.3m

m

Wiener

rmse = 5.8m

m

10

15

20

25

30

35

40

45

50

55

Viterbirm

se = 0.6mm

Object Measurement IBP – 28%

IBPP – 24% 2D4 - 2%

Optical blur = 1.5 and pixel-blur = 2. Reconstruction from 2 images, s = 1%

Axial extent of target = Temporal pulse width = 30mm. Target feature size = Scan step size = 4.6mm

Neifeld IMA 2005

Imager Goals:Imager Goals:uu Estimate point source position(s): Estimate point source position(s): { }uu Conventional image may be formed as a postConventional image may be formed as a post--processing stepprocessing step

Source Source volumevolume

ImagerImager

r1

rM

r2

z

yx

Strong Object Model:Strong Object Model:

rMr1 r2 …

M : Number of point sources

Conventional image

®® Fluorescent markersFluorescent markers®® Distant “bright” objects: aircraft, missile, starsDistant “bright” objects: aircraft, missile, stars

uu EqualEqual--intensity monochromatic point sourcesintensity monochromatic point sourcesuu Scene is completely specified by sources positions:Scene is completely specified by sources positions:

rMr1 r2 …

rMr1 r2 …

Introduction: plausibility of a single pixel imager.

uu Measure only what you want to knowMeasure only what you want to know

Neifeld IMA 2005

1

2

3

Source Volume

12

3

d1 ,h1LensDetector

phasemask

q Optimize imager based on information metric.qMaximize measurement entropy.q Select detector sizes and positions based on measurement pdf.

Measurement log-pdf

random phase

Measurement log-pdf

cubic phase

Measurement log-pdf

Introduction: information-based design.

source power = 0.5mWNEP=2nW

40cm

1cm3

Neifeld IMA 2005

89%74%36%Two detectors in two apertures

74%54%30%Two detectors in one aperture

65%39%21%One detector in one aperture

RPMCPMConventionalê Detector(s) : Imager Type è

Multiple Sources in VolumeMultiple Sources in VolumeSingle Source in VolumeSingle Source in Volume

Introduction: single pixel imager results.

q Object-space prior knowledge should inform the optical designq Let’s utilize this viewpoint in a more useful problem domain

Neifeld IMA 2005

PSF ENGINEERING

Neifeld IMA 2005

w Imagers for which pixel size > optical spot size. .

w Large pixels result in under-sampling/aliasing.

w Sub-pixel shifted measurements to resolve ambiguity. spatial ambiguity

Frame 1

shift camera

Frame 2

…..

Frame K

PSF Engineering: Under-Sampled Imagers

w Optical degrees of freedom not exploited.

wWe consider engineering optical point spread function.

Neifeld IMA 2005

Sensor details:w Pixel = 7.5 µmw Under-sampling = 15xw Full well capacity = 49ke-

w Spectral bandwidth = 10nmw Center wavelength = 550nm

Optics details:w Resolution = 0.2mrad/1µmw Field of view = 0.1 radw Thickness = 5mmw Aperture = 2.75mmw F/# = 1/1.8

M = 34x34

Object: f Imaging operator: H Measurements: g

N = 512x512 Sub-pixel shifts

…..

…..

w Single frame signal to noise ratio: SNR = 10log[sqrt(Ne)] = 23.3dBw SNR can be improved via multi-frame averaging ~ sqrt(K)w Total photon-count is kept constant over multiple-frames.

Phase-mask

Imaging ModelNeifeld IMA 2005

u Linear imaging model: g = Hf + n (note: n is AWGN)

u Block-wise shift-invariant imaging operator H is M x N

u Problem: M << N (e.g., M=N/15)

u Linear minimum mean square error (LMMSE) reconstruction: f = Wg

u LMMSE operator: W = RfHt(HRfHt+Rn)-1

u No Priors = flat PSD

u Priors = power law PSD or triangle PSD

^

Exa

mpl

e tr

aini

ng o

bjec

ts

PSD model

Power Lawà PSD(f) = 1/fη

Linear ReconstructionNeifeld IMA 2005

u Root Mean Squared Error:( )

[%]255

ˆ100RMSE

2

ff −×=

u Angular resolution:

Point Objectf = δ(r)

CompositeChannel

Hc

Reconstruction to Diffraction-limited

sinc2+

gn

f̂

RMSE=8.6%

ObjectCompositeChannel

Hc

LMMSEReconstruction+

gn

2

2 ˆsincminarg

−

∆

→∆∆

fx

θ

θθ

∆θ=0.4mrad

Performance MeasuresNeifeld IMA 2005

Resolution for TOMBO

sub-pixel shift Sub-pixel shiftedmeasurements

TOMBO ImagerConventional Imager

Shift-sensor

Conventional/TOMBO Imager Results

RMSE for TOMBO

Neifeld IMA 2005

u Consider use of extended point spread function(PSF)

extended PSF

impulse-like PSF

u Design issue #1: retain full optical bandwidthu Design issue #2: tradeoff SNR for condition number

u Pseudo-Random Phase masks for extended PSF

Modulation Transfer Function

( )ρ

γσασ

φ

∆⋅=

−⋅∆⋅= ,exp

4 2

2xR x ∆ - mask roughness ρ - mask correlation length

Realization of a spatial Gaussian random process.

Pseudo-Random Phase mask Enhanced Lens (PRPEL)

Example PSF(∆=0.5λ ,ρ=10 λ)

Alternate PSFNeifeld IMA 2005

uAll designs use optimal roughness.

u Note more rapid convergence of PRPEL compared to TOMBO.

u Higher resolution achieved by PRPEL at reduced number of frames.

u PRPEL achieves 0.3mradresolution at K=5 compared to K=12 for TOMBO.

Resolution Results

Resolution for PRPEL and TOMBO

Neifeld IMA 2005

RMSE for PRPEL and TOMBO

u PRPEL makes effective use of prior knowledge at K=1

u Note more rapid convergence of PRPEL.

u PRPEL consistently out-performs TOMBO.

K=1

PRPEL

K=2

K=3

TOMBOK=1

K=2

K=3

RMSE ResultsNeifeld IMA 2005

45K

PRPELTOMBOImager Type→Number of Frames↓

4% RMSE requirement

3.9%4.2%RMSE

PRPELTOMBOImager Type →(K=4)

RMSE achieved at M=N/4

0.35mrad0.60mradResolution

PRPELTOMBOImager Type →(K=4)

Resolution achieved at M=N/4

u PRPEL imager achieves 60% improvement in resolution.

u PRPEL imager obtains 22% improvement in RMSE.

K

Imager Type→Number of Frames↓

512

PRPELTOMBO

0.3mrad Resolution requirement

PRPEL SummaryNeifeld IMA 2005

u Sine-Phase mask Enhanced Lens(SPEL) :

( )θωαφ ii

N

ii xx += ∑

=sin)(

1

Amplitude Spatial-frequency Phase offset

u Pick N=3: yields 12 free parameters for optimization.

u Optimization criteria: ( )

[%]255

ˆ100RMSE

2

ff −×=

u RMSE computed over object class using LMMSE operator.

u PSF is optimized for each value of K.

Phase-maskωα

PSF Engineering via SPELNeifeld IMA 2005

K=1

K=2

u Note smaller support of SPEL PSF compared to PRPEL PSF.

u SPEL PSF also contains sub-pixel structure.

u SPEL PSF has more efficient photon-distribution.

u PSF support reduces with increasing K.

u SPEL PSF is array of delta pulses.

Observations

Observations

Optimized PSFNeifeld IMA 2005

K=16

u SPEL PSF converges to delta pulses as K increases.

u In limit Kà16 we observe that SPEL PSF to converge to TOMBO-like PSF.

Observations

Optimized PSF: System ImplicationsNeifeld IMA 2005

RMSE for SPEL, PRPEL, and TOMBO

RMSE : Power law PSD

uSPEL provides best use of prior knowledge for K=1uSPEL outperforms TOMBO by 47% in terms of RMSE(K=8).uSPEL improves RMSE by 35% compared to PRPEL (K=8).

K=2

PRPELK=1

K=3

K=2

SPELK=1

K=3

ResultsNeifeld IMA 2005

Angular resolution Resolution for SPEL,PRPEL and TOMBO

u Note PSF optimization was performed over RMSE.

u SPEL out-performs TOMBO.

u SPEL performance compared to PRPEL improves with increasing K.

Results

q PSF engineering can exploit weak object prior knowledge to improve performanceq Stronger object prior knowledge can enable non-traditional image measurement

Neifeld IMA 2005

FEATURE-SPECIFIC IMAGING

Neifeld IMA 2005

noisy image

Feature extraction Features Task

Feature-specific optics Features Task

Conventional imaging system PCA, ICA, Fisher, Wavelet, etc.

noise

noiseFeature-specific imaging

system (FSI)

Restoration, recognition,compression, etc.

u Feature-Specific Imaging (FSI) is a way of directly measuring linear features (linear combinations of object pixels).

u Attractive solution for tasks that require linear projections of object spaceu Let’s consider a case for which task = pretty picture

Passive Feature-Specific Imaging: MotivationNeifeld IMA 2005

2

11

ˆ

ˆmin {|| || }

|| || max{ | |} 1m

iji j

E

subject to f

ε

=

= =

= −

= =∑

y Fx x My

x x

F

Noise-free reconstruction:

= pca pca pcaTF M F

PCA solution :

1( )general−= T

x xM R F FR F

is any invertable matrixpca=F AF

A

General solution :

Result using PCA features:

FSI for Reconstruction

u PCA features provide optimal measurements in the absence of noise

photon count constraint

Neifeld IMA 2005

2 -1( I) Wiener - operatoropt σ= +T Tx xM R F FR F

2 2 -1{ ( I) } { }Tr Trε σ= − + +T Tx x xFR F FR F R

MyxnFxy =+= ˆ

)}||{||

log(10 2

2

σxE

SNR=1

|| ||

pca

pca

=F

FF

• Object block size = 4x4• Noise = AWGN• We use stochastic tunneling to optimize/search

optF

RMSE = 11.8

RMSE = 124

RMSE = 12

RMSE = 12.9

Optimal Features in Noise

u PCA features are not optimal in presence of noise

Noise-free problem statement:

Note: PCA error is no longer monotonic in the number of features à trade-off between truncation error and photon count constraint

11

|| || max{ | |} 1m

iji j

subject to f=

= =∑F

Neifeld IMA 2005

u Error increases as number of feature increases for PCA solution

u Reconstructed is improved significantly by using optimal solution

u Optical implementation requires non-negative projections

Optimal Features in NoiseNeifeld IMA 2005

u Optimal FSI is always superior to conventional imagingu Non-negative solution is a good experimental system candidate

Passive FSI Result SummaryNeifeld IMA 2005

Passive FSI for Face Recognition

• Face recognition from grayscale image feature measurements

• Class of 10 faces, 600 images per face

• Training = 3000 faces and testing = 3000 faces

• Features: wavelet, PCA, Fisher, …

• Recognition algorithms:

- k – nearest neighbor based on Euclidean distance metric

- 2-layer neural networks batch trained using back-propagation with momentum

Sample images from face database [Each image is 128x96]

First Wavelet feature of the above images [Each feature is 8x6]

Comparison of PCA recognition with AWGN

0

10

20

30

40

50

60

70

80

90

100

0 250 500 750 1000 1250 1500 1750 2000AWGN standard deviation

Rec

ogni

tion

perf

orm

ance

[%]

0 mux0 conv0_1 mux0_1 conv

Conventional

FSI

Neifeld IMA 2005

Passive FSI Optical ImplementationsNeifeld IMA 2005

• What is active illumination ?

• Project known structure onto scene

• Additional degrees of freedomimprove imager performance

• Past work on active illumination focused on:• Obtain depth-information for 3D objects

• Enhanced resolution for 2D objects

• Our goals:• Improve object- and/or task-specific performance

• Simplify light collection hardware

Projector

Object

Illumination pattern

Conventional cameras

Active Feature-Specific Imaging: MotivationNeifeld IMA 2005

ece

• Illumination patterns are eigenvectors (refer as PCA - FSAI)

• Advantages

• Small number of detectors

• High measurement SNR

• Task is to produce object estimate using these values

Object GLight Collection

Photodetector noise (AWGN)

Sequence of illumination patterns

GP )]([ idiag

iidiag α̂~)](][[∑ GPH

iii ndiagr += ∑ GPH )](][[

H (optics operator)

=

Mr

rr

.

.2

1

R

Vector of Measurements

(Estimate of feature weight)

16 × 16 replication of eigenvector P1

P2PM

64 × 64

16 × 16detector

FSAI System Flow DiagramNeifeld IMA 2005

ece

• Post-processing operator W is obtained by minimizing J

[ ]

matrix. covariance noise matrix,n correlatioobject

)](][[~]~~

,]~~

[~ˆ

2

2

1,

1

==

==

+=

∑=

×

−

nG

N

njniji,NMji,

nT

GT

G

and

diagandwhere

RR

PHHH[H

RHRHHRW

=

Mr

r

r

.

.2

1

R

)(]})ˆ)(ˆ[({ errorsquaremeanRWGRWGtraceEJ T−−=

Measurementvector

Linear post-processing

W

G = W R ?

• The MMSE operator is given by:

∑ ii Pα̂

(suboptimal in noise)

• Metric to evaluate reconstructions :

∑ ∑= =

−=objectsofnumber

k

N

iikikNobjectsofnumber

RMSE1 1

22

2

)ˆ(11 GG

N 2 = number of pixels, M = number of patterns

FSAI Post-ProcessingNeifeld IMA 2005

ece

• PCA vectors are not optimal in presence of noise

i

K

1ii

2PCA PaG with|GG|iswhichJ ∑

=

=−≠ ˆˆ)(]})ˆ)(ˆ[({ noisecontainsRRWGRWGtraceEJ T−−=

• Minimize the residual MMSE (JMMSE) with respect to both Pi’s and Ti

’s

( ) ( ) ( )1

2

2

22

2

21

2

111

11

,....,)...()...()...(

),.....,,.....(ˆ

~~~−

+=

M

T

MGM

T

MG

MM

TTTdiagPPRPPPPR

TTPPWwhereσσσHHH

}~ˆ{),....,,...( 11 GGMMMMSE RHWRTraceTTPPJ −=

SNR = 26 dB

M = 4 M = 8

optimaloptimal

PCAPCA

• Optimal features depend on M, SNR

Illumination Using Optimal PatternsNeifeld IMA 2005

ece

SNR = 26 dB (LOW NOISE)

M = 4

M = 8

Original object

PCA-FSAI(uniform T)

PCA-FSAI (optimal T) Optimal FSAI

• Minimum from PCA-FSAI

RMSE = 0.0633

• Minimum from optimal FSAI

RMSE = 0.0465

0 2 2 6 8 10 12 14 160.04

0.1

0.2

N u m b e r o f f e a t u r e s

Ave

rag

e R

MS

E (

LO

G S

CA

LE

)

U n i f o r m i l l u m i n a t i o n

P C A - F S A I(u n i f o r m T)

P C A - F S A I (n o n-u n i f o r m T)O p t i m a l

f e a t u r e s

FSAI ResultsNeifeld IMA 2005

ece

54 %31 %

Improvement of optimal FSAI compared to uniform illumination

0.07 (M = 16)0.0465 (M = 16)Optimal features

0.0768 (M > 2 )0.063 (M > 4)PCA – FSAI (nonuniform T)

0.0768 (M = 2)0.063 (M = 4)PCA – FSAI (uniform T)

0.151 (M =1)0.067 (M = 1)Uniform illumination

SNR = 16 dBSNR = 26 dBAlgorithm

FSAI Results SummaryNeifeld IMA 2005

Conclusions

q Objects are not iid pixels

èPixel-fidelity should not be the goal of an imager

èNeed new non-traditional design metrics

q Design should reflect prior knowledge of objects

èObject-specific imagers (e.g., SPEL)

èJoint design of optics and post-processing

q Design should reflect prior knowledge of application

èTask-specific imagers (e.g., FSI)

Neifeld IMA 2005