CS-MUVI Video compressive sensing for spatial multiplexing cameras

CS-MUVIVideo compressive sensing for spatial multiplexing camerasAswin Sankaranarayanan, Christoph Studer, Richard G. BaraniukRice University

Single pixel camera

Digital micro-mirror devicePhoto-detectorExplain DMD and linear measurement2Single pixel cameraEach configuration of micro-mirrors yield ONE compressive measurement

Non-visible wavelengthsSensor material costly in IR/UV bands

Light throughputHalf the light in the scene is directed to the photo-detectorMuch higher SNR as compared to traditional sensors

Digital micro-mirror devicePhoto-detectorRemember resolution of the device depends on that of the DMD ,,,,Sensing in non-visible wavelengths where sensing materials are costly3Single pixel cameraEach configuration of micro-mirrors yield ONE compressive measurement

static scene assumption

Key question: Can we ignore motion in the scene ?

Digital micro-mirror devicePhoto-detectorRemember resolution of the device depends on that of the DMD ,,,,Sensing in non-visible wavelengths where sensing materials are costly4SPC on a time-varying sceneNave approach: Collect W measurements together to compute an estimate of an image

what happens ?t=1t=Wmeasurementscompressive recovery

?

time varyingsceneDoes this strategy work ?5SPC on a time-varying sceneTradeoffTemporal resolution vs. spatial resolutiont=1

Small WLess motion blurLower spatial resolutionLarge WHigher spatial resolutionMore motion blurt=W (small)t=W (large)

Does this strategy work ?6

SPC on a time-varying scene

Lower spatial res.Higher temporal res.Higher spatial res.Lower temporal res.sweet spotRecall sweet spot depends on the object motion in the scene7Dealing with MotionMotion information can help in obtaining better tradeoffs [Reddy et al. 2011]State-of-the-art video compression

Dealing with MotionMotion information can help in obtaining better tradeoffs [Reddy et al. 2011]State-of-the-art video compression

nave reconstruction

motion estimatesKey pointsMotion blur and the failure of the sparsity assumptionUse least squares recovery ?

Recover scene at lower spatial resolutionLower dimensional problem requires lesser number of measurementsTradeoff spatial resolution for temporal resolution

Least squares and random matrices Random matrices are ill-conditionedNoise amplification

Hadamard matricesOrthogonal (no noise amplification)Maximum light throughputOptimal for least squares recovery [Harwit and Sloane, 1979]

Hadamard + least sq. recoveryHadamard

Random

Hadamard + least sq. recovery

Designing measurement matricesHadamard matricesHigher temporal resolutionPoor spatial resolution

Random matricesGuarantees successful l1 recoveryFull spatial resolution

Can we simultaneously have both properties in the same measurement matrix ?Dual-scale sensing (DSS) matrices

1. Start with a row of the Hadamardmatrix2. Upsample3. Add high-freq. componentKey Idea: Constructing high-resolution measurement matrices that have good properties when downsampled

Contrast this random matrices14CS-MUVI: Algorithm outlinet=Tt=1t=t0t=t0+Wt=W1. obtain measurementswith DSS matrices2. low-resolution initial estimate

3. motion estimation4. compressive recovery with motion constraints

Simulation result

CS-MUVI on SPC

Single pixel camera setup

Object

InGaAs Photo-detector (Short-wave IR)SPC sampling rate: 10,000 sample/sNumber of compressive measurements: M = 16,384Recovered video: N = 128 x 128 x 61 = 61*MCS-MUVI: IR spectrumJoint work with Xu and KellyRecovered Videoinitial estimateUpsampled

CS-MUVI on SPC

Nave frame-to-frame recoveryCS-MUVI

Joint work with Xu and KellyCS-MUVI summaryKey ingredientsNovel Measurement matrix designExploiting state-of-the-art motion model

One of first practical video recovery algorithm for the SMCdsp.rice.eduPractical - tackles motion blur that is fundamental to the SPC

We are consulting with them to implement this technology 20CS-MUVI summaryLimitationsNeed a priori knowledge of object speedMotion at low-resolutionRobustness to errors in motion estimates

Future workDual-scale to multi-scale matrix constructionsMulti-frame optical flowOnline recovery algorithmsdsp.rice.eduPractical - tackles motion blur that is fundamental to the SPC

We are consulting with them to implement this technology 21