motion estimation multimedia systems and standards s2 if telkom university
DESCRIPTION
Motion estimation Exploiting similarities between the regions boundaries in successive video frames Predictive differential coding: to code the changes incurred on the boundaries of detected regions from one to another. Trade-off: coding efficiency vs algorithmic complexity. minimal complexity: image segmentation could only be utilised for each video frame with no consideration given to temporal redundancies of shape and texture information. 3TRANSCRIPT
Motion Estimation
Multimedia Systems and StandardsS2 IF Telkom University
2
Overviewvideo compression schemesInter-frame vs. Intra-frame Intra-frame (e.g. JPEG, DCT, Wavelet)
more space Independenteasy for editing
Inter-frame (long group of picture/ motion estimation, e.g. MPEG (1, 2, 4), H.261, and H.263) there is relation between the frames,better for compression
3
Motion estimationExploiting similarities between the regions boundaries in successive video frames Predictive differential coding: to code the changes incurred on the boundaries of detected regions from one to another. Trade-off: coding efficiency vs algorithmic complexity.
minimal complexity: image segmentation could only be utilised for each video frame with no consideration given to temporal redundancies of shape and texture information.
4
Motion estimationMotion Estimation
Motion VectorMotion CompensationMotion ModelMotion EstimationEncoding and Decoding
Matching AlgorithmSearching AlgorithmEvaluation Matrices
5
Motion Vector [1]Definition :(H.264/MPEG-4 AVC) : A two-dimensional vector used for inter prediction that provides an offset from the coordinates in the decoded picture to the coordinates in a reference picture.
6
Motion vector [2]Motion representation
7
Motion CompensationApplying the motion vectors to a frame to synthesize the transformation to the next frameMany frames on the movie only have little different between frames (camera or object moving)
8
9
Motion ModelMotion in 3-D space corresponds to displacements in the image planeMotion compensation is conducted to provide a prediction signal for efficient video compressionEfficient motion compensation often uses side information to transmit the displacementsDisplacement must be efficiently represented for video compression
10
Motion Estimation [1]The process of determining motion vectors that describe the transformation from one 2D image to another, usually from adjacent frames in a video sequenceBlock-matching with previous frameSpace for saving bit motion vector is smaller than full blockEfficient motion estimation reduces the energy in the motion-compensated residual frame and can dramatically improve compression performance.Motion estimation can be very computationally intensive and so this compression performance may be at the expense of high computational complexity.
11
Motion Estimation [2]Key performance issues�� Coding performance Complexity Storage and/or delay
The motion estimation creates a model by modifying one or more reference frames to match the current frame as closely as possible.The current frame is motion compensated by subtracting the model from the frame to produce a motion-compensated residual frame.This is coded and transmitted, along with the information required for the decoder to recreate the model (typically a set of motion vectors).
12
Motion Estimation [3]At the same time, the encoded residual is decoded and added to the model to reconstruct a decoded copy of the current frame( which may not be identical to the original frame because of coding losses).This reconstructed frame is stored to be used as reference frame for further predictions.
13
Motion Estimation [4]2-D spatial search :
14
Motion Estimation [5]Residual Error
15
Motion Estimation [6]Choice of reference frames1. Forward Prediction
o Using an older encoded frame2. Backwards Prediction
o Using a future frameo Encoder to buffer coded
frameo Out of temporal order
3. Bidirectional PredictionNote : by using multiple reference frames will make encode and decode complexity and storage requirement increase
16
Encoding and Decoding [1]
17
Encoding and decoding [2]
Encoding1. Find a block in previous frame(reference frame).2. Choose a region that have smallest error3. Subtraction the matching region with current block to get
difference block.4. Encode and transmit difference block.5. Encode and transmit motion vector that indicated position
of matching region. Decoding
1. Decoding the difference between block and motion vector2. Sum the difference block with matching region in the
reference frame. A region that was pointed by motion vector.
Fact: 60% computational complexity at encoder side. Fast algorithms are necessary for real time applications.
Matching Algorithm
19
Matching Algorithm [1]1. Dividing current and previous frame into
non-overlapping block MxN2. For each block in current frame:
• Finding in the region that have size 2p+N at the previous frame.
• Finding the nearst block by using evaluation matrices
20
Matching Algorithm [2]1. Slow Motion Object (reduce number of
search point) Cross Search Algorithm (CSA) Full Search Algorithm (FSA) Spiral Search Three Step Search Two Dimensional Logarithmic Search (TDL) Binary Search (BS) Four Step Search (FSS) Orthogonal Search Algorithm (OSA)
2. Fast Motion Object Hierarchical Search Algorithm (HSA)
21
Matching Algorithm [3]Search
Strategies I : full search
All possible displacements within the search range are comparedComputationally expensive
22
Matching Algorithm [4]Search Strategies II: Three Step SearchAlgorithm
Step 1 Select an initial step size (s) equal or
slightly larger than half of the maximum search range
Calculate the error for the block at the center of search area and 8 square neighborhood point at the distance of s from center
Step 2 Move the center to the point with
minimum distortion Reduce step size by a factor of two If the step size is greater than one,
repeat the step1, otherwise go to step 3
Step 3 Final point with minimum distortion is
the result
23
Matching Algorithm [5] Search Strategies III : 2D
logarithmic searchAnil K. JainIt can be more accurate than TSSThe search window is large
AlgorithmStep1
Select an initial step size (s) Calculate the error for the block at
the center of search area and four point at x and y axis at distance of s from center
Step2 If the position of best match is at
center keep the center unchanged and reduce the step size by half, otherwise the best match becomes the center
Then step 1 is repeatedStep3
When the step size becomes 1 all the 8 neighbor blocks around the center will be checked for finding the best match
24
Matching Algorithm [6]Search Strategies IV : Diamond Search
25
Matching Algorithm [7]No
Algorithm Advantages Disadvantages
1 Three Step Search Low Complexity in terms of selected candidate locationsGood regularity in terms of motion vector generation
Complexity factor increases with search area sizeHigh data bandwidth
2 2-D Logarithmic Search
Suitable for fast motion The accuracy of algorithm is low when the motion vector is at anangle to x-y axis
3 Four Step Search Very low comp, More accurate than TTS, memory saving, for small motion
Risk of local minima if motion far away from center
4 Orthogonal Search Very fast for small motions Local minima may stop the search very soon
5 One at Time Search
Low computational Local Maxima
6 Cross Search Low computational Performance is worse than Three-step search
26
Evaluation Matrices [1]Mean squared error (MSE)
Mean absolute error (MAE)
where :f(m,n)= a pixel on coordinate (m,n) of NxN pixel blockg(m+i,n+j) = a pixel on coordinate (m+i,n+j) that corresponding with previous frame.
N
m
N
n
wjiwjnimgnmfN
jiM1 1
22 , ,),(),(1),(
N
m
N
n
wjiwjnimgnmfN
jiM1 1
2 , ,,(),(1),(
27
Evaluation Matrices [2]
Current block
reference area
MSE at (0,0) {(1-4)2 + (3-2)2 + (2-3)2 + (6-4)2 + (4-2)2 + (3-2)2 + (5-4)2 + (4-3)2 + (3-3)2}/9 = 2.44
28
Evaluation Matrices [3]Sum of absolute differences (SAD)• Matching the template into search image• Sum all of number in same matrix Example: 20, 25, and 17