three-dimensional video postproduction and processing ibraheem alhashim - july 10 th 2013 cmpt 880
TRANSCRIPT
Three-Dimensional VideoPostproduction and Processing
Ibraheem Alhashim - July 10th 2013CMPT 880
2
Overview
History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
3
Overview
History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
4
A bit of history
Imaging technology
James Maxwell 1855
JC d'Almeida 1858Joseph Niépce 1826
Stereoscope 1860
Underwood 1901
The Power of Love (1922 film)
The Jazz Singer 1927
5
A bit of history
Becky Sharp (1935) 1950s
6
A bit of history
Timeline of 3D movies
1950s 1980s 2010s
Worldwide: $2,782,275,172
7
3D Feature Films
Highest-grossing films 2012: nine of the top 15 were in 3D [businessinsider.com]
Industry forecast > 20% of TVs by 2015
Year 3D Films
2005 4
2006 7
2008 8
2009 20
2010 30
2011 38
2012 38
8
Other uses
WW2 maps Virtual reality 3D Electron Microscopy Video games + virtual cinema 1968
1945
198020132013 2013
9
3D Video
Usually marketed as objects popping off screen
10
3D Video
In reality.. It’s the same old concept Present slightly different image per eye The brain combines them and perceives depth
Trick human visual system Stereo 3D content production:
technical, psychological, and creative skills
11
3D Video - Issues
Not as straightforward as 2D production Several considerations for a good 3D experience Balance between 3D effect and overall experience Minimize viewing discomfort
Stereoscopic comfort zone Scene depth adaptation Control of global and local disparity Video composition
12
Stereoscopic Comfort Zone Comfort zone
Stereographer “bring the whole real world inside this virtual space”
13
Control of Absolute Disparity Convergence is controlled by shifting the views
14
Scene Depth Adaptation
Keep in mind disparity range (screen size and resolution)
Carefully plan a scene’s 3D effect Consider transitions and provide resting periods Post-production depth adaptation
Manual changes per display
15
3D Display Adaptation
From cinema to TV Depth composition has to modified (stereographer)
Depth information allows for virtual view interpolation
16
Local Disparity Adaptation Objects should be within stereoscopic window
Intentional depth changes Physical / multiple camera rigs Synthetic / depth-editing
Objects at the border Can cause retinal rivalry Should be cropped by virtually shifting screen plane closer However, not applicable to live broadcast
17
Live auto-correction
Automatic correction & manipulation of stereo live broadcast Live sport events (big player) Close objects could abruptly appear
Open research problem Some kind of novel view synthesis
18
Video composition
Mixing and Composition of 3D Material, Real and Animated Content Graphics overlay / subtitles Cannot be simply pasted over other footage Leverage knowledge about depth range of footage
19
3D Video - Issues
Summary 3D Production has an “art” component Different medium requires different parameters Content makers / directors need to think about 3D issues Stereographer try to balance 3D effect with overall
viewing experience
20
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
21
3D Display Technologies
“Offer immersive experience” 3D Glasses (cinema + TV) Head-mounted displays Volumetric and holographic displays Autostereoscopic displays
22
Autostereoscopic displays
Best choice for mobile devices Backward compatible & closer to viewer expectation Most common
Parallax barrier Lenticular sheet
23
Autostereoscopic displays Crosstalk (ghosting) is most important parameter
“information meant for one eye intrudes into the other eye’s view”
24
Autostereoscopic displays Issues
Generally less available depth range More ghosting artifacts Also, depth information is essential
Synthesize different views
25
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
26
Basic processing
Signal processing to avoid visual artifacts Any small visual discrepancy could cause discomfort
Three main categories1. Correction of Geometrical Distortions2. Color Matching3. Adjustment of Stereo Geometry
27
Correction of Geometrical Distortions
Camera rigs might not be perfectly aligned Real lenses impose radial distortions by nature Other lens parameters might not sync E.g. geometrical lens distortions or chromatic
aberration
28
Color Matching
Color discrepancy can lead to eyestrain and visual fatigue
Manual calibration need to be done on cameras Automatic methods exist (histogram filtering)
29
Color Matching
Modern professional postproduction tools incorporate stereo color matching and grading
30
Adjustment of Stereo Geometry
Convergence need to be selected and balanced carefully to achieve good stereo content
Usually by shifting images horizontally in contrary directions, however, cropping & scaling is needed
“shift-crop-scale” Demo http://stereo.nypl.org/create
31
Adjustment of Stereo Geometry
Stereo baseline is fixed during shooting Several hardware solutions help camera team
analyze the disparity range Also help visualize result of shift-crop-scale
32
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
33
3D Depth information
Depth information is needed for novel output images in post-production adjusting the view parallax (different screens) many different uses in postproduction
34
Depth info
Extracting depth information (time of flight camera / SfS)
Structure-from Stereo (SfS) Advanced computer vision problem Stereo matching
Local – block matching, optical flow est. Global – graph cuts, simulated annealing
35
Structure-from Stereo
Global methods are more accurate Slow + don’t work well on video / motion
Local methods are more widely used Window-based methods Some system are in real-time Blocky output
36
Example of Depth-based method
Apply hybrid recursive matching (HRM) Follow by cross-trilateral median filtering (ACTMF) Semi-automatic
37
Post processing depth
Align depth discontinues to object borders Remove noise and mismatches Fill occlusions Approaches
Use image segmentation Neighborhood filtering
38
View synthesis
Synthesize new virtual stereo views by image-based rendering
Input – depth + color Output – image with new view
Two types depth-based + warping-based
39
Depth-based
Computer vision techniques Image-based rendering (IBR) Depth-based rendering (DIBR) Layered-depth images (LDI) Intermediate view reconstruction (IVR)
40
Depth-Image-Based Rendering
Need pixel-by-pixel depth maps Recent focus
Handle depth discontinues Better depth boundaries
41
Warping-based
Methods that deform the image content directly Compress or stretch by nonlinear warp function Do not need camera calibration, segmentation, fill holes Worst case, visible wobbling artifact
42
View synthesis
Summary Depth maps are computed using computer vision
techniques (still active) Generate new views by image-based rendering or warping Warping methods can potentially have less visual problems
DEMO
43
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
44
2D to 3D
3D to 2D is trivial Hot topic for 3DTV and 3D cinema Methods so far are
Manual (computer assisted) Automatic
45
Depth cues
Human visual systemMonocular cuesBinocular cues
46
Depth cues
Monocular depth cues - things can be seen by one eye (2D Camera)
Focus / defocus, perspective, relative size Light and shading and texture Motion parallax
Binocular – 2 eyes or 3D camera rigs accommodation, convergence, and binocular discrepancy
47
Manual 2D to 3D
Applicable to prerecorded video Utilize depth cues to generate a stereoscopic view
for each frame Time consuming and costly Cost vs. quality
48
Manual 2D to 3D
Three major steps Rotoscoping / segmentation Depth assignment Inpainting
Few companies provide process as a service
49
Depth Assignment
Shifting different parts of scene to simulate 3D To avoid cardboard effect hire a “3D compositor”
Create displacement maps for each pixel Use 3D primitives, spheres or cubes Use DIBR or 3D warping to synthesize view cannot handle transparencies well
50
Occlusion filling
Also known as “in-painting” One of the most challenging parts in 2D/3D
conversion
51
Automatic conversion
Automatic systems Extract depth information + synthesize stereoscopic images
52
Automatic segmentation
Automatic segmentation of background Background subtraction techniques (ML) Optical-flow-based (uses motion as cue)
Objects in scene segmentation Contour-based (color, edges, texture) “SnapCut” state of the art algorithm in After Effects
53
Automatic depth estimation 3D Structure recovery (Shape From X)
Depth from focus/defocus Depth from geometric cues
Relative sizes, gradients Depth from color and intensity
Lights and shadow Depth from motion
Motion parallax
54
Automatic in-painting
Relay on - texture synthesis or border continuation Some work is done on spatio-temporal inpainting
Enforce global temporal consistency for patches Object motion could reveal background
Not viable in practice
55
Real-time and Offline conversion
Real-time Approximate using motion parallax Could use color + intensity information Height-depth cues Hybrid approaches are better
Offline Apply structure from motion Create virtual scene using different views Slow cloudification?
56
2D to 3D
Summary Hire specialist to rebuild depth and frame objects Relay on computer vision methods to build depth info Fill background using in-painting Still an active research topic
segmentation + depth estimation + synthesis
57
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
58
3D media for mobile devices
Consider size of display Computing power + Network constraints Encoding
3D Video coding (V+D) Multiview Video Coding (MVC)
59
Issues
Stereoscopic images have different disparity Coding artifacts
Dealing with different channels Transmission artifacts
Image distortion on loss Temporal mismatch
Display artifacts autostereoscopic display problems (ghosting)
60
3D media for mobile devices Not only mobile devices, Internet connected devices Standards for 3D video encoding are still developing
Challenges for 2D video on mobile and more (depth)
3D Video conferencing demand real-time processing
Display technologies are evolving Resolution, need an adaptive and future ready format
61
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
62
Outlook
Today, 3D video technology are becoming practical Content creation process have matured
still a lot of room for improvement and extension There is a huge commercial incentive in this field A lot of areas to work on…
63
Outlook
64
Commentary
3D has been historically about generating revenue using mind tricks
Tricks are getting much better! Immersive experience ≠ quality
65
Commentary
Future of 3D Video Consumers decide it Good for short media If all devices support it
Cinema products areprofitable!
66
Commentary
Similar strategies could apply on different technology Light-field camera – Lytro first product 2011 (2006) Nokia / Pelican Imaging 16-lens array camera 2014
67 Thank you!