introduction to 360° video -...
TRANSCRIPT
![Page 1: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/1.jpg)
INTRODUCTION TO 360° VIDEO
Oliver Wang Adobe Research
![Page 2: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/2.jpg)
OUTLINE
What is 360 video?
![Page 3: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/3.jpg)
OUTLINE
Formats
What is 360 video?
How do we represent it?
![Page 4: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/4.jpg)
OUTLINE
Stitching
What is 360 video?
How do we represent it?
How do we create it?
![Page 5: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/5.jpg)
OUTLINE
Stabilization
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
![Page 6: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/6.jpg)
OUTLINE
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
?Future Areas
![Page 7: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/7.jpg)
OUTLINE
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
![Page 8: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/8.jpg)
360° VIDEO VS “FULL” VR VIDEO
Mozilla Developer Network - WebVR Concepts
3DoF 6DoF
![Page 9: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/9.jpg)
ONE OF THE MOST POPULAR FORMS OF VR
![Page 10: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/10.jpg)
360° VIDEO VS “FULL”
VS
Disp
lay
Capt
ure
![Page 11: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/11.jpg)
OUTLINE
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
![Page 12: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/12.jpg)
360° VIDEO FORMATS
![Page 13: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/13.jpg)
360° VIDEO FORMATS
![Page 14: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/14.jpg)
360° VIDEO FORMATS
https://en.wikipedia.org/wiki/Map_projection
![Page 15: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/15.jpg)
EQUIRECTANGULAR
![Page 16: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/16.jpg)
EQUIRECTANGULAR
• Positive
• Visually interpretable
• Easy to process
• Negative
• Distortion on poles
• Low pixel density on horizon
https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/
![Page 17: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/17.jpg)
CUBEMAPS
https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/
![Page 18: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/18.jpg)
CUBEMAPS
• Hard to interpret/process
https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/
• Graphics pipeline friendly
• Low distortion (25% storage reduction)
![Page 19: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/19.jpg)
https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/
EQUIANGULAR CUBEMAPS
![Page 20: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/20.jpg)
https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/
Equirectangular Projection Cubemap Equiangular Cubemap
EQUIANGULAR CUBEMAPS
![Page 21: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/21.jpg)
https://blog.google/products/google-vr/bringing-pixels-front-and-center-vr-video/
EQUIANGULAR CUBEMAPS
![Page 22: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/22.jpg)
PYRAMIDS
https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/
![Page 23: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/23.jpg)
PYRAMIDS
https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/
![Page 24: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/24.jpg)
360° VIDEO CONTENT CREATION
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
![Page 25: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/25.jpg)
MULTI-LENS CAMERAS
![Page 26: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/26.jpg)
STITCHING
Image Alignment and Stitching [Szeliski, Richard 2005]
![Page 27: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/27.jpg)
PARALLAX ARTIFACTS
Slides courtesy of Jungjin Lee, KAIST
different projection centers
![Page 28: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/28.jpg)
PARALLAX COMPENSATION
[Jiang and Gu 2015]
Optical Flow
http://sintel.is.tue.mpg.de/
Mesh Warping
![Page 29: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/29.jpg)
HIGH-LEVEL PIPELINE - INPUT
Panoramic Video from Unstructured Camera Arrays [Perazzi et al. Eurographics 2015]
![Page 30: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/30.jpg)
Slides courtesy of Federico Perazzi, Disney Research
REFERENCE PROJECTION
![Page 31: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/31.jpg)
PARALLAX COMPENSATION
Slides courtesy of Federico Perazzi, Disney Research
![Page 32: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/32.jpg)
PARALLAX COMPENSATION
Slides courtesy of Federico Perazzi, Disney Research
![Page 33: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/33.jpg)
BLENDING
Slides courtesy of Federico Perazzi, Disney Research
![Page 34: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/34.jpg)
BLENDING
Slides courtesy of Federico Perazzi, Disney Research
A Multiresolution Spline With Application to
Image Mosaics [Burt and Adelson 1983]
![Page 35: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/35.jpg)
Slides courtesy of Federico Perazzi, Disney Research
![Page 36: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/36.jpg)
Slides courtesy of Federico Perazzi, Disney Research
![Page 37: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/37.jpg)
ANOTHER WAY TO THINK ABOUT IT
Slides courtesy of Jungjin Lee, KAIST
Rich360: Optimized Spherical Representation from Structured Panoramic Camera Arrays [Lee et al. SIGGRAPH 2016]
![Page 38: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/38.jpg)
MESH DEFORMATION
Deformation of the projection sphere
Slides courtesy of Jungjin Lee, KAIST
![Page 39: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/39.jpg)
DEFORMABLE SPHERICAL PROJECTION SURFACE
Recovered 3D point
Slides courtesy of Jungjin Lee, KAIST
![Page 40: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/40.jpg)
DEFORMABLE SPHERICAL PROJECTION SURFACE
Recovered 3D point
Slides courtesy of Jungjin Lee, KAIST
![Page 41: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/41.jpg)
DEFORMABLE SPHERICAL PROJECTION SURFACE
Slides courtesy of Jungjin Lee, KAIST
![Page 42: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/42.jpg)
Comparisons with two state-of-the-art stitching methods GCW: per pixel warping using optical flow [Perazzi et al. 2015]
STCPW: mesh-based warping using sparse feature points [Jiang and Gu 2015]
Slides courtesy of Jungjin Lee, KAIST
![Page 43: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/43.jpg)
NON-UNIFORM RAY SAMPLING
Slides courtesy of Jungjin Lee, KAIST
![Page 44: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/44.jpg)
NON-UNIFORM RAY SAMPLING
Slides courtesy of Jungjin Lee, KAIST
![Page 45: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/45.jpg)
Slides courtesy of Jungjin Lee, KAIST
![Page 46: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/46.jpg)
Comparisons with the equirectangular projection
• Rendering resolution: 2K
• Resolution of the viewing screen: full HD Slides courtesy of Jungjin Lee, KAIST
![Page 47: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/47.jpg)
360° Video Processing
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
360° VIDEO STABILIZATION [Johannes Kopf, SIGGRAPH Asia 2016]
![Page 48: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/48.jpg)
Slides courtesy of Johannes Kopf, Facebook
![Page 49: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/49.jpg)
Slides courtesy of Johannes Kopf, Facebook
![Page 50: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/50.jpg)
Cropped
TRACK
2D or 3DNarrow FOV
FIT MOTION MODEL + SMOOTHING
WARP
Slides courtesy of Johannes Kopf, Facebook
![Page 51: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/51.jpg)
Not CroppedHybrid 3D-2DFull 360
TRACK FIT MOTION MODEL + SMOOTHING
WARP
Slides courtesy of Johannes Kopf, Facebook
![Page 52: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/52.jpg)
Input video Pure-rotation stabilization
Slides courtesy of Johannes Kopf, Facebook
![Page 53: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/53.jpg)
Rolling shutter/lens deformation
Translation effects
![Page 54: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/54.jpg)
Pure-rotation Deformed-rotation
Slides courtesy of Johannes Kopf, Facebook
![Page 55: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/55.jpg)
CREATING NARROW FOV VIDEOS FROM 360 VIDEO
Pano2Vid: Automatic Cinematography for Watching 360° Video [Su et al. ACCV 2016]
![Page 56: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/56.jpg)
PANO2VID
Input: 360° VideoOutput: normal-field-of-view
(NFOV) Video
Slides courtesy of Yu-Chuan Su, UT Austin
![Page 57: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/57.jpg)
PANO2VID
Slides courtesy of Yu-Chuan Su, UT Austin
ST-glimpses
yx
z
ΩTrain
C3D [Tran 2015]
![Page 58: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/58.jpg)
PANO2VID
Slides courtesy of Yu-Chuan Su, UT Austin
ST-glimpses
yx
z
ΩTrain
C3D [Tran 2015]
Evaluate capture-worthiness Construct trajectory
Test
![Page 59: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/59.jpg)
HUMAN VALIDATION
Slides courtesy of Yu-Chuan Su, UT Austin
![Page 60: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/60.jpg)
HUMAN VALIDATION
AutoCam performs the best Justified by multiple metrics
# videos Total length
360° videos 86 7.3 hours
HumanCam 9,171 343 hours
Slides courtesy of Yu-Chuan Su, UT Austin
![Page 61: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/61.jpg)
Making 360° Video Watchable in 2D: Learning Videography for Click Free Viewing [Su and Grauman CVPR 2017]
![Page 62: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/62.jpg)
FUTURE PROBLEMS
What is 360 video?
How do we represent it?
How do we create it?
What can we do with it?
What can’t we do with
it?
![Page 63: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/63.jpg)
PERCEPTION
Gaze Visualization for Immersive Video Thomas Löwe, Michael Stengel, Emmy-Charlotte Förster,
Steve Grogorick, Marcus Magnor Eye Tracking and Visualization (Springer), 2017
Saliency in VR: How do people explore virtual environments? Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala,
Diego Gutierrez , Gordon Wetzstein. Arxiv 2016
![Page 64: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/64.jpg)
CINEMATOGRAPHY RULES
• Constraint on camera motion
• Zooms
• Cuts (coming up)
• Lighting
![Page 65: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/65.jpg)
SPATIAL AUDIO
https://en.wikipedia.org/wiki/Spherical_harmonicsCore Sound TetraMic
![Page 66: INTRODUCTION TO 360° VIDEO - richardt.namerichardt.name/publications/video4vr/Video4VR-2-Wang-360-video-100... · Slides courtesy of Federico Perazzi, Disney Research. BLENDING Slides](https://reader031.vdocuments.site/reader031/viewer/2022013017/5c65b10b09d3f2916e8d2dbe/html5/thumbnails/66.jpg)
360° VIDEOOliver Wang Adobe Research