comparison between blur transfer and blur re-generation in depth image based rendering
DESCRIPTION
presentation at 3DTV-CON2014.TRANSCRIPT
![Page 1: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/1.jpg)
Comparison between Blur Transfer and Blur Re-
Generation in Depth Image Based Rendering
Norishige Fukushima†, Naoki Kodera†, Yutaka Ishibashi†, Masayuki Tanimoto‡
July 2-4, 2014 Budapest, Hungary 3DTV-CON 2014
†Graduate School of Engineering, Nagoya Institute of Technology,Japan
‡Nagoya Industrial Science Research Institute,Japan
![Page 2: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/2.jpg)
Outline
Background Related Works View Synthesis Methods
Blur erasing type Blur re-generation type Blur transfer type
Experimental Results Conclusion and Future Works
![Page 3: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/3.jpg)
Free Viewpoint
Image
Depth Image Based
Rendering
Original Image Depth Map
Background (1/3)
![Page 4: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/4.jpg)
Background (2/3)
Input view and
depth map
3D warping
Hole filling
Free viewpoint image
![Page 5: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/5.jpg)
Background (2/3)
Input view and
depth map
3D warping
Hole filling
Free viewpoint image
![Page 6: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/6.jpg)
Background (3/3)
Blurred region
FB
![Page 7: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/7.jpg)
Background (3/3)
Blurred region
FB
3D warp to left
Split
![Page 8: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/8.jpg)
Background (3/3)
Blurred region
FB
3D warp to left
Split
Filled
![Page 9: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/9.jpg)
Background (3/3)
Blurred region
FB
3D warp to left
Split
Filled
Boundary regions are degraded.
![Page 10: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/10.jpg)
View synthesis method
3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method
Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method (proposed method)
![Page 11: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/11.jpg)
View synthesis method
3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method
Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method (proposed method)
![Page 12: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/12.jpg)
Blur erasing type
3D warp
Erase
Artifact
![Page 13: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/13.jpg)
Blur erasing type
3D warp
Erase
Filled
Blurs are broken.
Artifact
![Page 14: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/14.jpg)
View synthesis method
3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method
Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method (proposed method)
![Page 15: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/15.jpg)
Blur re-generation type
Generating blurred region using alpha matting
Splitting input image into three images→ foreground, background and alpha mask
Alpha blending three images after DIBR
Input image
Split
Foreground
Alpha mask
Background
![Page 16: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/16.jpg)
Blur re-generation type
Fore
Alpha
Back
3D warp blendMatting
Blur re-generation
![Page 17: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/17.jpg)
View synthesis method
3 types of blur treatment Blur erasing type Blur re-generation type Blur transfer type
Improved blur transfer method (proposed method)
![Page 18: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/18.jpg)
Blur transfer type
Dilation
Color image
Depth map
![Page 19: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/19.jpg)
Blur transfer type
Dilation
Color image
Depth map
Foreground depth value can cover almost fuzzy/blurred region.
![Page 20: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/20.jpg)
Blur transfer type
Blur keep
3D warping
![Page 21: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/21.jpg)
Blur transfer type
3D warping
interpolating
Blur keep
![Page 22: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/22.jpg)
Canny and Gaussian Filtering
Dilation
DIBR
Blur transfer type
Improved blur transfer method (proposed method)
Adding a simple process
Improved blur transfer method
(proposed method)
![Page 23: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/23.jpg)
Improved blur transfer method (proposed method)
Generating mask by canny filter Smoothing masked region by Gaussian
filterGenerate
mask
Smooth
Masked region
![Page 24: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/24.jpg)
Comparison
Blur erasing type × Blurs are broken. ◎ Fastest
Blur re-generation type ○ Blurs are reconstructed. ×Slowest
Blur transfer type ○ Blurs are kept. ○ Faster
For better boundary treatment, we compare blur re-generation type with blur transfer type.
![Page 25: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/25.jpg)
Experimental Results (1/6)
Evaluating PSNR of these methods Basic type [1] Blur re-generation type [2] Blur transfer type [3] Improved blur transfer method (proposed
method)
: TeddyH (950×750)
[1]: Y. Mori et al., Signal Process. Image Commu., vol. 24, no. 1, pp. 65-72, Jan. 2009. [2]: N. Kodera et al., IEEE VCIP 2013. [3]: X. Xu et al., IEEE ICASSP 2012.
: Bowling1 (671×555)
: Reindeer (671×555) Input depth map is ground truth
![Page 26: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/26.jpg)
Experimental Results (2/6) PSNR
BasicRe-
generation
Transfer Proposed
average 35.62 38.85 37.93 38.54
TeddyH 32.74 35.03 34.97 35.13
Reindeer 34.37 38.14 36.75 38.15
Bowling1 34.42 39.72 35.46 35.93
. . . . .
. . . . .
. . . . .
. . . .
(dB)
#Average: using 30 data set
![Page 27: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/27.jpg)
Experimental Results (3/6)
Basic
TeddyH: 950 x 750
Re-generation Proposed
Proposed method can soften contour artifacts, look like re-generation method.
![Page 28: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/28.jpg)
Experimental Results (4/6)
Reindeer: 671 x 555
Re-generation Proposed
In proposed method, blurs around object boundary blend background color. Proposed method dilates depth value to
the unnecessary region.
![Page 29: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/29.jpg)
Experimental Results (5/6)
Bowling1: 671 x 555
Re-generation Proposed Proposed method interpolates using
foreground or mixed color. Depth maps are not dilated enough.
![Page 30: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/30.jpg)
Experimental Results (6/6)
Computational cost Proposed method ≒ Basic method (15ms) +
4ms 2 dilations for input depth map < 1ms Canny filter for depth map on the synthesized image
< 3ms Gaussian filter with a small kernel < 1ms
Blur re-generation type > 100ms Matting ≒ 55ms 3 times DIBR ≒ 45ms Blending < 2ms
![Page 31: Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering](https://reader038.vdocuments.site/reader038/viewer/2022103017/55844393d8b42abf1e8b52ae/html5/thumbnails/31.jpg)
Conclusions Comparing 3 types of DIBR method in blur
treatment. Proposed method improves subjective quality at object
boundaries, and objectively has 0.61dB improvement. This is the second best (0.31dB lower than the blur re-
generation). Proposed method reaches the state of the arts of blur
regeneration method, and computational cost is about x5 effective.
Proposed (19 ms) vs Blur re-generation (>100 ms)Future Work Investigating effect of proposed method using
estimated depth maps. Considering effect of coding distortions.