Tae-Shick Wang;   Kang-Sun Choi;   Hyung-Seok Jang;   Morales, A.W.;   Sung-Jea Ko;

IEEE Transactions on Consumer Electronics, Vol. 56, No. 2, May 2010   

ENHANCED FRAME RATE UP-CONVERSION METHODFOR UHD VIDEO

Introduction(1/2)Motion blur caused by inherent characteristic of LCDFRUC

MCFI consists of ME and MCI.

Introduction(2/2)The motion vector field(MVF) is estimated

between two successive frames.MVF should represent the actual object motion

Several ME methods for MCFI have been proposed to find the true motion based on BMA Multi-size block matching algorithmBi-directional Motion Compensate

Interpolation(MCI)

Motion analysis for UHD video(1/2)

Enlarge SR can span regions which belong to another object but are more similar to the current block

Motion analysis for UHD video(2/2)Full search, sort all MVs for each block in

UHD Increasing order of SAD

Search range(SR) should be kept as small as possible if a reliable MV is initially given

Large block size gives a more correct motion

Proposed ME algorithmThe blocks with similar motion are

merged to an entity.Introduce a segment-based ME algorithm

Divide the frame into several segments Estimate the motion for each segment

Large region can be obtained more reliablyfn: current

frame

Block-based segmentationClassify each block into three patterns:

edge, plane, and texture

Segments are obtained by mergingadjacent blocks with an identical pattern

Gradient calculationFor each block, the gradient is calculated

by using the Sobel operator [14].

Gradient

gradientimage

x

fg x

y

fg y

If large=>significant pixel

[14] R. C. Gonzalez and R.E. Woods, Digital Image Processing, 2nd Edition, Prentice Hall, New Jersey, 2002.

Gradient Direction Histogram

Largest number in the histogram # of significant pixels

gx

gy

= r

Segmentation result

MVF of fn-1

: k-th segment from fn

knS

Segment selectionFind segments whose motion can be

obtained accurately.Temporal consistency of the segment

MV of the segment with high reliability

1. Large segment is more reliably,

2. The pattern of segment is plane or texture

3. has dominant MV pass through Dominant MV : over 70% MVs are identical

4)( TSN knB

knS

knS

knS

Efficient true motion estimation(1/3)Full search with low computation

complexitySSAD (subsample SAD) as matching criterion

# of blocks in the segment

=> sub-sample rate σ

)( knB SN

Efficient true motion estimation(2/3)Determine the search range

Reliable block (RB) : SSAD < 5(B/σ)^2 => the MV’s similarity to the actual

motion of the segment

=> search range

Search range size LSR in the segment:

ε =8

)(

)(knB

knRB

SN

SN

Efficient true motion estimation(3/3)Three stage ME

Refine dominant MV within LSR

Refine the received MV (if average SSAD < 5(B/σ)^2) and within LSR

No initial MV within max SR

Experimental resultThree UHD video sequence:

Toy and Calendar, Table Setting, and TractorCompare Ha’s [2] and Huang’s [10]

methods

[2] T. Ha, S. Lee, and J. Kim, "Motion compensated frame interpolation by new block-based motion estimation algorithm," IEEE Trans. Consumer Electron., vol. 50, no. 2, pp. 752-759, May 2004.[10] A.-M. Huang and T. Nguyen, "A multistage motion vector processing method for motion-compensated frame interpolation," IEEE Trans. Image Process., vol. 17, no. 5, pp. 694-708, May 2008.

Subjective comparison

Objective comparison

Improve quality for the interpolated frame by 2~3 dB

Reduce the computation load.

RC: relativecomplexity

ConclusionThe block-based segmentation method

was confirmed to produce meaningful segment information with low complexity.

The proposed method can also be successfully employed for various applications includingDe-interlacing View interpolation for multi-view video.