1. A Switchable Light Field Camera Using Angle Sensitive Pixels and Dictionary-based Space Coding Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University *Shared first-authorship

2. Cameras Today2DCameraCo. LightField CameraCo. FlexibleCamera! 3. Flexible Camera Vision Output 4. Flexible Camera Vision Visual Complexity ComputeTime High Resolution 2D Low Resolution Light Field High Resolution Light Field Prototype is grayscale only! 5. Computational Photography Touches... Optics Algorithms Sensors Lytro.com Dappled Photography, Veeraraghavan, et al. ?An angle-sensitive CMOS imager for single-sensor 3D photography, Wang et al. 6. Traditional Camera Bucket of photons Need optics to re-bin rays 7. Light Field Cameras Lens Based Mask Based barrier lenslets Veeraraghavan et al. 2007Ives, 1905Lippmann 1908Lytro 2014 Direct angle-space tradeoff Limits Resolution Direct angle-space tradeoff Limits light transmission Limits Light Field Resolution More Flexible 8. Camera Arrays Light Field Cameras Sequential Acquisition Wilburn et al. 2002,2005 Levoy and Hanrahan 1996 Liang et al. 2008 9. Redundancy Natural 4D Light Field Random Light Field 10. Captured2DImage4DReconstruction Compressive Capture Marwah et al., 2013 11. Compressive Capture Random and optimized optical codes Multiplexing & nonlinear reconstruction Marwah et al., 2013 Image Sensor Printed Mask 12. Image Sensor Circuit Pixel Array Timing/Addressing Amplifiers ADCs Silicon Dioxide: - Insulator for metal interconnects Photo Diode Silicon Substrate Metal Transistor 13. Angle Sensitive Pixel (ASP) Phase gratings Two Interleaved Diodes ~ Single Pixel Silicon Dioxide: Insulator Output 1 (D1) Output 2 (D2) 14. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel! Phase Grating Diodes 15. Plane wave on grating generates periodic diffraction pattern Operating Principle: Talbot Effect Single Pixel! 16. Angular Response Angle (degrees) Response Amplitude (V) Single Pixel! 17. ASP model , () = + + m 360o 18. Pixel Angular ResponseASP Output Conventional Pixel - + Angle Angle Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Single ASP 19. Pixel Angular ResponseASP Output - + Frequency Response Angle Angle ResponseAmplitudeResponseAmplitude ResponseAmplitudeResponseAmplitude Angle Angle 20. 2D ASP Tile ,, () = + + + Physical Layout Impulse Response (2D) Low Med 18 High 24 Low Med 18 High 24 Note: This prototype was made using a functionally similar variant of the ASP that uses a stack of metal gratings instead of phase gratings 21. ASP Sensor ASP Sensor 22. 8 8x + - + + 8 8x + - + + Row address Timing generation DataOutput 4:1 4:1 4:1 4:1 Pixel array S/H Mux Sum/Diff PGA ADC Design Layout Fabricate 23. INTUITION FROM SIGNAL PROCESSING A Fast Linear Reconstruction Method 24. Light Field Capture: 1D ASP Array 1 2 3 Angle Amplitude Angle Amplitude Angle Amplitude Note! Space Note! Angular Frequnecy 25. Spatio-Angular Frequency Domain = = 26. LINEAR RECONSTRUCTION A Fast and Simple Reconstruction Algorithm 27. Fast/Linear Reconstruction ASP responses are approximately orthonormal wavelents is used as a preconditioner for inverting capture equation Model the image capture process: ASP Sensor Tile Impulse Response (2D) 28. NONLINEAR RECONSTRUCTION High Resolution Light Fields from a Single Photo 29. Captured2DImage = ASP Projection 4DLightField Sparse Coefficients! Compressive Light Field Reconstruction ASP Sensor Tile Impulse Response (2D) 30. Overcomplete dictionary= Light field vector = Coefficient vector s.t. is sparse Can lead to fewer non-zero coefficients Dictionary Compressive Light Field Representation Courtesy Marwah et al. 31. Training light field = is sparse Sample 1,000,000 random 4D patches from training light fields i i i for all iCoefficient vectorDictionary s.t. Dictionary Learning Courtesy Marwah et al. 32. Light Field Atoms in Dictionary 5,000 atoms, each 9x9 pixels and 5x5 views Light fields can be represented by only a few of these atoms Courtesy Marwah et al. 33. Captured2DImage4DReconstruction = ASP Projection 4DLightField Basis Pursuit Denoise: Sparse Coefficients! Compressive Light Field Reconstruction 34. RESULTS Simulated and Live Results from a Prototype Camera 35. Limitations 2D View Reconstructed Light Field Central View 36. Limitations Specularities and other effects not well represented in dictionary Sensor saturation 37. Limitations ASP responses are non-linear in some regions ASP Tile 2D Image Light Field Center View 38. ANALYSIS Depth-of-Field, Noise, Resolution, and Failure Cases 39. Resolution, Noise Performance, Depth of Field 40. Conclusion We built a custom, ASP sensor New sensors, new opportunities Shoot now compute later! [Computation in] [quality out] Seeking new problems e.g. Illumination Multiplexing, Time of Flight 41. Support provided by: NSF IIS-1218411 NSF Graduate Student Fellowship (Suren) MIT Media Lab Consortia Funding NSF IIS-1116452 NSERC Postdoctural Fellowship (Gordon) Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2* Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1 1MIT Media Lab 2Cornell University A Switchable Light Field Camera 42. Depth-of-Field ? ? ? ? ? ? ? ? ? ? ? ? 43. Resolution 44. Noise Light Field Center View