{ fast disparity estimation using spatio- temporal correlation of disparity field for multiview...
Post on 19-Dec-2015
232 Views
Preview:
TRANSCRIPT
{
Fast Disparity Estimation Using Spatio-temporal Correlation of Disparity Field for Multiview Video Coding
Wei Zhu, Xiang Tian, Fan Zhou and Yaowu ChenIEEE TCE 2010
2
Introduction Analysis of Disparity Field in MVC Proposed Fast Disparity Estimation Algorithm Experimental Results Conclusion
Outline
3
Fig. 1. MVC coding schemes among views for different sequences.
(a) For “Ballroom” sequence with general 1D camera setup. (b) For “Akko&Kayo” sequence with 2D-array camera setup. (c) For “Flamenco2" sequence with 2D-cross camera setup.
Introduction
4
Motion Estimation Temporal motion is dependent on the movement of
objects. Only moving objects have motion displacements,
objects in background often have no motion displacement.
Disparity Estimation Inter-view disparity is dependent on the depth of
object and the camera setup. Objects with no motion displacements may also have
large disparity displacements.
Introduction
5
Analysis of Disparity Field in MVC
[17] X. M. Li, D. B. Zhao, S. W. Ma and W. Gao, “Fast disparity and motion estimation based on correlations for multiview video coding,” IEEE Trans. Consumer Electron., vol. 54, no. 4, pp. 2037-2044, Nov. 2008.
object depth
camera parameters
6
Disparity is related to the depth of objects Shallow depths large disparity Distant depths small disparity
Objects in MVC have large range of depths, the range of disparity is also large.
DE requires a large search range to find optimal disparity vector(DV).
Analysis of Disparity Field in MVC
7
Analysis of Disparity Field in MVC
Fig. 2. Histogram of horizontal disparity vectors between view S1 and view S0 for “Ballroom” sequence.
Size of DV vary a lot: Most of the DV are in [0,
24], background objects. Some DV are in [24, 72],
foreground objects.
Most of DV are in the positive.
The disparity direction
is only determined by the location of views.
8
Spatial direction correlation
Fig. 3. A field of original disparity vectors for the 123th frame of view S1 in “Ballroom” sequence. [14] Y. Kim, J. Kim, and K. Sohn, “Fast disparity and motion estimation for multi-view video coding,” IEEE Trans. Consumer Electron., vol. 53, no.2, pp.712-719, May 2007.
Analysis of Disparity Field in MVC
DVs are highly correlated with neighboring vectors in spatial direction.[14]
Some irregular DVs are obvious different with their neighbors, especially in homogenous regions.
9
Temporal direction correlation
Fig. 4. Histogram of temporal difference for horizontal disparity vectors between view S1 and view S0 in “Ballroom” sequence.
Analysis of Disparity Field in MVC
DVs in the temporal direction are also highly correlated.
10
The characteristics of disparity field in MVC: The size of DVs could vary a lot because of different depths
of objects. Due to the fixed camera setup, most of DVs have a consistent direction with the real disparities.
Some DVs deviate from the real disparities, DVs of previous coded should be filtered to be consistent with real disparities.
Since DVs have a high correlation with neighboring vectors in the temporal and spatial direction, the search center of DVs and preliminary DV can be predicted.
Analysis of Disparity Field in MVC
11
Basic idea: Select the search center by using the spatio-temporal correlation of disparity field, and to predict the search range adaptively according to the temporal variation of disparity field.
Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame.
Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field.
Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field.
Part D: Overall Proposed Algorithm.
Proposed Fast Disparity Estimation Algorithm
12
Basic idea: There is a high correlation of DVs in temporal direction. Use previous coded frames to predict current frame.
Problem: Some noisy DVs are not consistent with their real disparities, these vectors should be eliminated to obtain a smooth disparity field.
Solution: Because noisy DVs have irregular directions, check every DV direction first.
Requirement: GDV is calculated by averaging all DVs of block 16x16 in the previous coded frame, and its direction is used as the reference of the real disparity direction.
Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame
13
Step1: Using GDV, the DV of block 16x16 for each MB is regularized.
Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame
90 °
number of vectors in 1
14
Step2: To obtain a smooth variation in the spatial direction, is further filtered by a spatial median filter.
A field of the :
Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame
15
Step3: After obtaining a smoothed disparity field of the previous coded frame, a temporal prediction of the DV () is calculated for each MB in current frame.
Part A: Temporal Prediction of the Disparity Vector Based on the Smoothed Disparity Field of the Previous Coded Frame
encoding order of current frame
0.5 co-located
(Zero Disparity Vector)
16
Basic idea: The disparity vectors are highly correlated in spatio-temporal directions, so the neighboring disparity vectors in spatio-temporal directions are used to determine the search center for the current block.
Select candidates: The neighboring disparity vectors are selected as candidates of
the search center. Non-anchor frame: 5-9 are selected from the forward and backward
temporal reference frame. Anchor frame: 5-9 are selected from the previous coded anchor
frame.
Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field
17
: Considering the influence of homogenous regions. : , it is a smoothed prediction of the disparity vector from spatial
direction. : It is a temporal disparity predictor.
All of the available candidates for the current block are made up the set .
To prevent digression of DE, candidates of the search center are further limited around by a given maximum search range().
Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field
18
The candidate which has minimum Lagrangian cost is selected as the search center of the disparity vector ().
The average Manhattan distance between CDV and the optimal DV estimated by the full search algorithm.
Part B: Selection of Search Center Based on the Spatial-temporal Correlation of Disparity Field
It’s the smallest, only a smaller search range is needed.
19
Temporal variation(distance) of DV: |CDV – TDV| CDV: An approximation of the DV in current frame. TDV: A temporal prediction of the DV in previous
coded frame.
Basic idea: the distance is related the DV consistency between current frame and previous coded frame.
Small distance small search range Large distance large search range
Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field.
20
Search range:
Part C: Prediction of Search Range Based on the Temporal Variation of Disparity Field.
(larger)
1.3
12
6
Calculate search range
22
JMVC4.0 Testing Configuration
Only Inter16x16 mode for inter mode Use first two or three views of the sequences. View S1 was chosen for proposed algorithm, and
disable ME S0 was chosen as the reference view for
Flamenco2(2D-cross), S0 and S2 were chosen for others.
Comparing with full search algorithm and fast search algorithm.
Experimental Results
25
In this paper, a fast DE algorithm is proposed to save the computational load of MVC.
Take into account the spatial-temporal correlation and the temporal variation of disparity.
Perform well on all test sequences. Compare with full search algorithm, achieve an
average 96% reduction of computational complexity, while RD performance remain the same.
Compare with fast search algorithm, achieve an average 43% reduction of computational complexity, while RD performance is improved.
Conclusion
top related