Visi 3D Estereoscpica i Hologrfica Josep R. Casas

Visi 3D Estereoscpica i Hologrfica Josep R. CasasDispositius Optoelectrnics (DOE) Tardor 2014Dispositius Optoelectrnics3D Vision8.1 Introduction3D scene & 2D visual representationProjection and reconstruction principles8.2 Depth and stereoscopic perceptionRange and boundaries of stereoscopic, parallax and monocular views, stereoscopy vs panoramic, glasses-free auto-stereoscopy8.3 3D acquisition systemsStereoscopic cameras and multi-camera capture8.4 3D displaysStereoscopic displays, HMDs, auto-stereoscopic displaysHolographic displaysPrinciples of holographic imaging, transmission and reflection holograms, volumetric holograms, stereographic holograms, video-hologramsVolumetric displaysContents2Dispositius Optoelectrnics10 Nov.3D scenes & 2D images14 Nov.Project & Reconstruct Test 9 (21/Nov)17 Nov.Stereo & Depth21 Nov.Stereo & Depth (2) Test 10 (28/Nov)24 Nov.3D multi-camera 28 Nov.3D range-cams Test 11 (12/Des) 1 Dec.3D displays12 Dec.3D Holography, Control 33ProgramDispositius Optoelectrnics8.1 3D scenes and 2D imagesFrom Scenes to Images Conventional AV systems project 3D scenes onto a 2D image plane of the imaging sensor (arranged on a planar surface)

Advantage: simple AV systems just transmit and display 2D images!

Drawback: significant loss of information

2D Image3D SceneAVcaptureDispositius OptoelectrnicsDepth information is lost!HVS uses monocular hints, information from occlusions and previous experience about objects structures and shapes to retrieve depth information

Dispositius Optoelectrnics

Graphical projectionProduction of a 2D image from a 3D object, used in technical drawing

Parallel projectionLines of sight from the 3D object to 2D plane are parallel to each other OrthographicUsed for working drawingsPictorialsObject as viewed from a skew direction to reveal all 3 axes of space in one pictureAxonometric (iso-/di-/tri-metric)Oblique (cavalier, cabinet)6Planar projection (1)

Axonometric projectionIsometricWorking drawing

Dimetric

Dispositius Optoelectrnics Graphical projectionProduction of a 2D image from a 3D object, used in technical drawing

Perspective projection3D objects projected on a picture plane. Distant objects appear smaller than nearer objectsParallel lines in 3D intersect in the projected image, converging towards a single point vanishing pointPerspective projection looks most realistic, as photographic lenses and human eye work in the same way7Planar projection (2)

People mover at Frankfurt International Airport (vanishing point at the end of the tunnel)Dispositius OptoelectrnicsFrom perspective to parallel

Increasing the focal length and distance of the camera in a perspective projection results in an approximation of orthographic projection. Increasing them to infinity would result in an orthographic projection ( SharkD Wikimedia Commons)Dispositius OptoelectrnicsYou said this part of the course was about 3D Vision and, with projection, we loose 3D, right?9But

So, how will Homer get back out from the projection?Dispositius OptoelectrnicsException! 3D/stereoscopic

Dispositius OptoelectrnicsSorry? You said what?Wait a minute the projection process on the camera lost the third dimension. OK and now, you said we want to recover it?But if you already lost it!

Dispositius OptoelectrnicsWhat if?We had captured the 3D directly?or enough information to recover the 3D?or just the minimal information to simulate 3D?Dispositius OptoelectrnicsCapture 3D directly?

S.Izadi et al, 3D surface reconstruction @MS Research (2011)Dispositius OptoelectrnicsEnough information to recover 3D?

J. Salvador et al, Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction, IEEE Trans. IP 22(9), 2013Dispositius OptoelectrnicsMinimal info to simulate 3D

http://www.youtube.com/3Dvideos4U3Dvideos4U

http://www.youtube.com/watch?v=KtLj8FNx2lw Dispositius Optoelectrnics15What we need for 3D vision?Special sensors (range sensors)Multiple cameras (2 or many)Image Analysis / Computer Vision!Science developing theoretical and algorithmic bases to obtain information about the real world from one or several images (Haralick, 1992)Discipline developing systems able to interpret the contents of natural scenes (Castleman, 1996)Making computers see still an unsolved problem! (Faugeras, 2001)

Computer Vision is based in Mathematics and InformaticsDispositius OptoelectrnicsSensors & Early VisionColor and TextureSegmentation and GroupingMotion and TrackingStereo and Structure from MotionImage-based ModelingIllumination and Reflectance ModelingShape RepresentationPhysics-based ModelingObject RecognitionVideo Analysis and Event RecognitionFace and GestureStatistical Methods and LearningVision for RobotsPerformance EvaluationMedical Image AnalysisImage and Video RetrievalSource: Call for Papers ICCV 2013Vision topicsDispositius OptoelectrnicsImage Acquisition Photography X Rays Thermographyobject Restoration Filtering Segmentation Feature extraction Feature selection Detection Classification Measures InterpretationImage ProcessingImageAnalysisComputer Vision SystemDispositius Optoelectrnics18 Matching Tracking 3D Reconstruction (Calibration)Multi-viewAnalysisResultado3D Computer Vision SystemImage Acquisition Photography X Rays Thermographyobject Restoration Filtering Segmentation Feature extraction Feature selection Detection Classification Measures InterpretationImage ProcessingImageAnalysisDispositius Optoelectrnics19Perspective distortionPerspective correctionE.g. Perspective correction (1)

Dispositius Optoelectrnics20E.g. Perspective correction (2)

Dispositius OptoelectrnicsE.g. Perspective correction (3)

Dispositius OptoelectrnicsE.g. Lens correction (1)Lens distortionLens correctionDispositius Optoelectrnics

E.g. Lens correction (2)

Dispositius OptoelectrnicsE.g. 3D Reconstruction (1)3D ObjectBAMView 2View 1C1C2Optical center 2Optical center 1Dispositius OptoelectrnicsE.g. 3D Reconstruction (2)

Dispositius OptoelectrnicsE.g. Feature Matching (1)Image 3Image 2Image 1Mm1Camera3D Objectm2m3Dispositius OptoelectrnicsE.g. Matching (2)

45678910111213123Dispositius Optoelectrnics2811-X?4567E.g. Matching (2 cont)Dispositius Optoelectrnics2911-X?8910E.g. Matching (2 cont)Dispositius Optoelectrnics3011-X?111213E.g. Matching (2 cont)Dispositius Optoelectrnics31123E.g. Matching (2 cont)Dispositius Optoelectrnics32E.g. Matching (2 cont)Dispositius Optoelectrnics331591012 Does triplet 1-5-9 fulfill the trifocal constraint? Does triplet 1-5-10 fulfill the trifocal constraint? YES!NO! Does triplet 1-5-12 fulfill the trifocal constraint? YES!1-5-X?Dispositius Optoelectrnics3416151-6-X?191-9-X?1-5-X?Dispositius Optoelectrnics35E.g. Matching (2 cont)Dispositius Optoelectrnics365911-5-9-X?12E.g. Matching (2 cont)Dispositius Optoelectrnics37

59121610132E.g. Matching (2 cont)Dispositius Optoelectrnics38Ok, we need Computer Vision for 3D vision what else?Special sensors (range sensors)Multiple cameras (2 or many)Image Analysis / Computer Vision! and Displays!!Dispositius OptoelectrnicsWhat is available?

Dispositius Optoelectrnics