digital image stabilization of video team 2 bo mu marvin boonmee yonghui zhao hongqin zhang

Digital Image Stabilization of VideoDigital Image Stabilization of Video

Team 2Bo Mu

Marvin Boonmee

Yonghui Zhao

Hongqin Zhang

OutlineOutline

The camera shake problem Comparison of stabilization methods

– Optical real-time– Electronic real-time– Mechanical real-time– Digital processing post – capture

Digital processing– Gradient-based flow estimation– Frequency-based flow estimation– Feature correspondence flow estimation– Locus smoothing image compensation– Affine transform image compensation

Our implemented algorithms– Block method Cathy– Phase method Iris– Harris corner method Bo– Corner anisotropism method Marvin

Measurement error on synthetic data Results for real data Analysis of results and future work

ProblemProblem

Types of image motions: – Intended camera motion– Independent moving objects– Egomotion

Image Stabilization of Video– Compensation of unwanted camera motions (Egomotion/camera shake)

Some sources of camera shake– Handheld vibrations (shaky hands)– Oscillating platform (running)– Vehicle mounted camera traveling over rough terrain– Vehicle / aircraft vibrations– Wind buffeting

Trivial solution for static platform: use a tripod – A tripod is not always a viable solution due to weight / size– Need solutions for handheld video and vehicle mounted cameras

OutlineOutline






Stabilization methodsStabilization methods

Optical– Vibration sensors adjust optics

Electronic– Vibration sensors adjust portion of sensor that is read

Mechanical– Physical damping of vibrations

Digital processing– Compensate with image transforms

Optical Image Stabilization (OIS)Optical Image Stabilization (OIS)

Small vertical and horizontal gyros precess with camera movement

Gyros are used to detect camera vibration and compensate by moving an optical element via servo motors

Response time from detection to the drive of the corrective optics is on the order of .002 seconds

Image processing used to detect low-frequency vibrations missed by the gyro– this data refines the optical compensation in subsequent frames

Examples:– Stabilizing lens group– Fluid prism– Sensor shifting

Optical Image Stabilization 1: Stabilizing lens group Optical Image Stabilization 1: Stabilizing lens group Example: Downward Tilt (rotation about x-axis)Example: Downward Tilt (rotation about x-axis)

Data from gyro sensors is used to shift lens group 2 in parallel along the optical axis

Optical Image Stabilization 2 : Fluid PrismOptical Image Stabilization 2 : Fluid PrismExample: Downward Tilt (rotation about x-axis)Example: Downward Tilt (rotation about x-axis)

Fluid prism Pair of glass plates surrounded by a bellows and filled with fluid Has a refractive index comparable to a glass prism

The angle of the prism is changed by tilting the plates

Optical Image Stabilization 3 Optical Image Stabilization 3 Sensor ShiftSensor Shift

•The CCD sensor rests on a movable mount

•Shifting occurs along the x, y-axes only

Electronic Image Stabilization (EIS)Electronic Image Stabilization (EIS)

EIS repositions the active area of the CCD chip by re-addressing the area of the chip that they're reading from

EIS uses a portion of the CCD to stabilize the image– The used area is smaller– Image is still moving across the chip, image shakes induce motion blur

• Faster shutter speed required to offset motion blur across the sensor• requires gain increase and results in "graininess" in the image,

especially in low light– Image degradation

Two methods– oversize CCD: have borders that the active area can be moved around in– enlarge (zoom) the image digitally

• reduces resolution

EIS example: Missile seekers, terminal guidance & EIS example: Missile seekers, terminal guidance & tracking systems tracking systems

Stabilize video from rolling airframe missiles

Real-time stabilization: translation, zoom, and rotation of a video image stream

Inertial measurement unit inputs RS-170 input video, digitized to 8-bit

resolution, and stored in frame memory for transformation on the next video frame. Four pixels of video are read in parallel and bilinear interpolation is performed for filtering

Outgoing video is synchronized with incoming video with exactly one field/frame delay.

High levels of circuit integration and high density packaging techniques such as gate arrays are used to render the system on two VME bus modules

Mechanical Image Stabilization 1Mechanical Image Stabilization 1

Gyroscopic stabilizer

A gyroscope with two perpendicular spinning wheels and a battery pack

Attaches to the tripod socket

Two gyroscopic wheels, in opposing axis to each other, are spinning at 22,000 RPM

Physically resists camera vibrations

Mechanical Image Stabilization 2Mechanical Image Stabilization 2

Gimballed system suspends the camera so that it remains level when its support is tipped

Set up the camera so that it has large rotational moments of inertia, but little reason to want to rotate

The camera is mounted on an arm or pole that's gimballed at its center of gravity

Steer the camera by light touches near the gimbal

Easy to oversteer the camera– off-level horizons are typical for beginners

Digital StabilizationDigital Stabilization

Process the video frames after capture

Estimate optical flow between frames– determine the intentional and unwanted components– determine the compensation required for the unwanted component

Apply either:– Affine transformation (Frame warping)– Zero-padded frame shift and rotation

ComparisonComparison

OIS High image quality Image is stabilized on the chip: no motion blur Vibration sensors are used, can’t be fooled by scene No image bounces or resets as with EIS

Additional power required Expensive Lens specific (can’t use converters) Limited to physical range of corrective optics

EIS Cheap Vibration sensors are used, can’t be fooled by scene Slightly faster reaction time than OIS (no optics to

stop moving) No moving parts

Additional power required Higher shutter speed needed to reduce blur Noisier due to higher gain needed If the stabilizer "runs out of chip" the image

abruptly re-centers itself Some digital artifacts

Mechanical Can be used with on any camera Can be used with other stabilization methods Widely used for aerial imaging and TV / movie

Additional power required Requires practice for best results Fatiguing for extended periods Expensive

Digital Cheap Can be used with on any camera Only option for post-capture Best option for tiny cameras (helmet cam)

Can be easily fooled Stabilization can hunt, oscillate, or bounce Some digital artifacts Computationally expensive

+ _

OutlineOutline






Basic Camera Motions

“Qualitative estimation of camera motion parameters from the linear composition of optical flow”, S. Park, H. Lee, S. Lee. Pattern Recognition 37 (2004) 767-779.

Translational Rotational

6 basic parameters:Translation in x, y, zRotation in x, y, z

General Camera Model

Camera-centered coordinate system – Z-axis being the optical axis

P = (X, Y, Z) is a 3-D coordinate of the object point P

p = (x, y) is a 2-D image coordinate of P in the image coordinate

System centered at the intersection of the optical axis Z

O is the focal point

The focal length of the camera = f– distance between the focal point of the camera and the image plane


3-D translational motion:3-D rotational motion:


Velocity Components

Relationship between the 3-D object coordinates and camera coordinates

The optical flows, u(x, y) and v(x, y), generated at a point on an image plane by the camera motion


• 2-D optical flow can be rewritten as:

• Translational and rotational components are separable

• The rotational component is independent of scene structure (depth)

• The depth Z influences the translational component only

• The vectors defined by the translational components lies on lines going through the point , which is called the focus of expansion (FOE)

Optical Flow ExampleOptical Flow Example

“Qualitative estimation of camera motion parameters from the linear composition of optical flow”, S. Park, H. Lee, S. Lee. Pattern Recognition 37 (2004) 767-779.

Optical Flow (OF)Optical Flow (OF)

Optical flow (OF)

– the perspective projection on the image plane of the 3D velocity field of a moving scene

– a 2-D velocity field with 2 components: rotation and translation

OF’s generated by the rotation of the camera are independent of the distance to the object

– OF’s generated by camera translation are dependent on object distance

– Discontinuities in the optic flow field are due to variations in depth

Optical Flow MethodsOptical Flow Methods

Differential techniques (Gradients)

– Compute image velocity from spatiotemporal derivatives of image intensity or filtered versions of the image

Correlation (features correspondence)

– extract features from each frame (gray level windows, edge detection) and match them from frame to frame

Frequency-based methods

– Use frequency and phase information to estimate the velocity between frames

CorrelationCorrelation

Correlation

– choose a k x k window surrounding a pixel, p, in frame i

– compare this window against windows in frame i+1

– The best match determines the displacement of p from frame i to frame i+1

Drawbacks of correlation

– Matching in the presence of rotation is computationally expensive (all orientations of the window must be matched)

– If motion is not constant in the k x k window then the window will be distorted by the motion

• This suggests using smaller windows within which motion will not vary much but smaller windows are more sensitive to noise

GradientsGradients

Estimate image motion by comparing spatial derivative at a point to its temporal derivative– example: spatial derivative is 10 units/pixel and temporal derivative

is 20 units/frame– then motion is (20 units/frame) / (10 units/pixel) = 2 pixels/frame

If the gradient directions are all similar, then the location of the best fitting (u,v) will be very sensitive to errors in estimating gradient directions

Gradient based methods only work when the motion is “small” so that the derivatives can be reliably computed– for “large” motions, employ multiresolution methods

Multiresolution MethodsMultiresolution Methods

fine scale– many edges are detected– computationally costly matching problem

coarse scale– relatively few edges identified– simple matching problem

Multiresolution– Detect edges at a coarse scale– Estimate motion by tracking– Use these estimates as initial conditions for matching edges at next

finest scale– Can be applied to correlation and gradient based methods

IssuesIssues

Poor resolution, low contrast, specular effects, shadows, insufficient texturing, blurring

– motion compensation using intensity values does not result in good estimation of motion parameters for some scenes due to lack of texture and low SNR

– Constant scene illumination and Lambertian reflectance are assumed for most current optical flow methods

Occlusion

– Flow fields are often noisy and partially incorrect, especially near occlusions or motion boundaries

Rotational and translational ambiguities

– i.e. ... small rotations about the y axis are easy to confuse with translations in x

IssuesIssues

Outliers– The 2-D motion field usually contains a set of noisy and partially

incorrect data (outliers). The set of incorrect data can be even larger if independent motions exist throughout the image sequence

Noise– Noise affects both the spatial and temporal components of the image

measurements To estimate flow well, the noise parameters need to be estimated accurately. In many situations this is impossible because the parameters are not static, but change with the viewing and lighting conditions, often too rapidly to collect enough data to obtain good estimates.

– A consequence of noise is bias in the flow estimates, the estimates tend to be smaller in length and closer in direction to the dominant gradient direction in the patch than the actual values

Noise in images can give high differences where there is no motion– compare neighborhoods rather than points

IssuesIssues

Independently moving objects

– will confuse the algorithms unless image segmentation is performed

– A moving object gives negative depth values when considering the egomotion parameters: this provides a method of detecting that it is moving independently

As objects move, their homogeneous interiors don’t result in changing image intensities over short time periods

– motion detected only at boundaries

– requires subsequent grouping of moving pixels into objects Non-integer pixel positions

– determine the gray levels for the pixels of the transformed image by interpolating the gray levels of the original image

Aperture ProblemAperture Problem

When a moving line is seen through an aperture only the normal flow is visible

Differential algorithms are more susceptible to the aperture problem

In complex scenes the aperture problem is not significant

A small aperture attempts to measure the velocity, V , of a contour passing through itThe aperture can only measure Vo, the component of V orthogonal to the contourThe parallel component, Vp , is invisible to the aperture

Error metricsError metrics

Error measurement

1. Compute the RMS difference between the stabilized images and the reference frame

2. Measure the angular deviation between the estimated velocity and the correct one

– where c is the correct motion vector, e is the estimate optical flow vector, and ^ denotes vector normalization

3. Absolute magnitude of the differences between the components of the estimated velocity and the correct one

Error metricsError metrics

Aperture problem– To measure how effectively the algorithm compensates for

this, use:

– where and denotes the vector perpendicular to

– A large error normal to the image gradient would suggest the algorithm does not compensate adequately for the aperture problem

OutlineOutline






Block methodBlock method OverviewOverview

The parametric motion model is obtained by fitting the optical flow information from block-matching motion estimation

Can apply to translation and rotation separately – the former is a special case of the latter

Block Motion Vectors and the histogram between frame 12 and 13

Block methodBlock method Algorithm Flowchart: Algorithm Flowchart:

Block method Block method Motion Estimation Details – TranslationMotion Estimation Details – Translation

1. Calculate block motion vectors

1.1 Block size 16x16

1.2 Search Range: –15 to 15 in both horizontal and vertical directions.

1.3 Search criterion: MSE

2. Calculate global motion vector

2.1 Taking the median of the block motion vectors

Block method Block method Motion Estimation Details – AffineMotion Estimation Details – Affine

1. Calculate block motion vectors

1.1 Block size 16x16

1.2 Search Range:

15 to 15 in horizontal and vertical directions

1.3 Search criterion: Weighted MSE

2. Calculate global motion vector

2.1 Assume no zoom or scaling, the affine transformation:

2.2 Global parameters can be obtained by least-square.

2.3 For each pair of corresponding blocks, select the center points as correspondences

Block method Block method Motion Smoother Motion Smoother

Smooth out the abrupt camera motion Assumption: unwanted motion is high frequency components in the

motion vector variations. Applying a low pass filter A moving average filter:

Convolving the original motion vectors with the average filter N can be changed

Block method Block method Motion Smoother Motion Smoother

Translation - Applying the average filter on the motion vectors Rotation: - Applying an average filter on both the angle of rotation and offsets

- The smoothed affine motion parameters:

Block method Block method Motion CorrectionMotion Correction

Each stabilized frame is obtained from previous stabilized frame and corresponding smoothed motion vectors or affine motion parameters.

Error propagation control techniques to avoid boundary problems:

Disadvantage: cause sudden motion when synchronization occurs

Block method Block method Motion EstimationMotion Estimation

Advantages:– Simple and easy to implement– Works well for small 2D translation

Disadvantages: – Only 2D motion estimation (no depth)– Results in large errors for rotational estimations if there are large

depth variations within images– Affine model didn’t have significant improvement over the translation

model

Block method Block method Motion CorrectionMotion Correction

Advantages:– Simple smooth filter– Frame warping using the smoothed motion vectors – Works well for small range scene

Disadvantages: – Big boundary problems for large range scene – If use error control techniques, sudden motion when synchronization

occurs– In case the smoothed motion is at the frequency to which human eyes

are more sensitive, the perceived motion might not be as smooth as expected

Phase-Based ApproachPhase-Based Approach OverviewOverview

Filter the image sequence with a set of quadrature Gabor filter pairs

Compute the phase response

Compute component velocity

Calculate the full velocity

Temujin Gautama and Marc M. Van Hulle, A phase-based approach to the estimation of optical flow field using spatial filtering, IEEE Trans. Neural Network, Vol. 13, No. 5, P1127-1136 (2002)

Phase-Based Approach Phase-Based Approach Gabor FilterGabor Filter

yfxfjS

y

S

x

SSyxg yx

yxyx

2exp

2

1exp

2

1,

22

Sx Sy Fx a

19.31648

39.31648

30.02156

8-0.08049

Phase-Based Approach Phase-Based Approach Filter BankFilter Bank

Sx Sy Fx Fy

1 9.316483 9.316483 0.021568 -0.08049

2 9.316483 9.316483 0.058926 -0.05893

3 9.316483 9.316483 0.080494 -0.02157

4 9.316483 9.316483 0.080494 0.021568

5 9.316483 9.316483 0.058926 0.058926

6 9.316483 9.316483 0.021568 0.080494

7 6.146587 6.146587 0.063155 -0.10939

8 6.146587 6.146587 0.109387 -0.06315

9 6.146587 6.146587 0.12631 0

10 6.146587 6.146587 0.109387 0.063155

11 6.146587 6.146587 0.063155 0.109387

Harris Corner DetectorHarris Corner DetectorOverviewOverview

Auto correlation detector.

Look for local maxima in average of {shift response } corners detected

2

, , , ,,

x y u v x u y v u vu v

E W I I

Harris Corner DetectorHarris Corner DetectorAlgorithm DetailsAlgorithm Details

Two images generated by X and Y gradient filter Smooth window applied (3 x3) on last step

Gaussian filter (sigma =3) generating three images (IX^2,IY^2,IXY) Tr = IX^2 + IY^2; Det = IX^2 * IY^2 – (IXY)^2

R = Det –k*Tr R > 0 corner

R < 0 edge R small flat

SSD (sum squared difference)tracking the corresponding edge

-50 0 50 100 150 200 2500

50

100

150

200

250

300

350

Harris Corner Detector Harris Corner Detector Motion Parameter EstimationMotion Parameter Estimation

Given corresponding feature point sets (X1, Y1), (X0, Y0), and the inter-frame displacement Δx, Δy between two frames

2D affine Transform– Assume small rotation

S is a scalar, which is inversely proportional to the ratio of the distance between two arbitrary image points at time t0, t1

θ is the rotation angle along the optical axis

Harris Corner Detector Harris Corner Detector Separate Rotation from TranslationSeparate Rotation from Translation

Difficulty on separating rotation from translation– Errors on estimating motion parameters

Translation Dominated– Global motion correction

– Rotation correction results in fluctuation

Rotation Dominated– Rotation correction

– Translation correction results in drifting

Harris Corner Detector Harris Corner Detector Separate Rotation from TranslationSeparate Rotation from Translation

Harris Corner Detector Harris Corner Detector Motion CompensationMotion Compensation

Frame to frame1. Estimate parameters from two consecutive frames in the input

image sequence2. Warping the current frame

P1 = sRP0 + T

3. Forward warping and backward warping

Pro– Easy implement

Con– Drift due to the accumulation error– Dynamic problem causes the small overlap between frames

Harris Corner Detector Harris Corner Detector Motion CompensationMotion Compensation

Frame to reference– Estimate parameters from reference frame and current frame in the

input image sequence– Warp

Pro– Reduce the drift but not all. Drifts happen between reference and

reference, instead of the frames Con

– Reference is easy to obsolete caused by small overlap• Requires reset• However, high reset rate is not undesirable

Harris Corner Detector Harris Corner Detector ApproachApproach

Optical flow or feature tracking to estimate the displacement between frames

Pre-filter video sequence by the smoothed global motion

Estimate the motion parameters from filtered sequence– 2D, three parameters. – 3D, six parameters plus zooming– Affine model or perspective model

Motion compensation– Cascade warping. (memory)

Harris Corner Detector Harris Corner Detector Error AnalysisError Analysis

Corner detection.

Outliers in feature tracking.

Error on estimating parameters. (model issue)

Error on Compensation

Corner detection based on local anisotropismCorner detection based on local anisotropismOverviewOverview

Identify corners as:

– points with a strong gradient not oriented in a dominant direction

= 15x15

Corner detection based on local anisotropismCorner detection based on local anisotropismAlgorithm Details (for each video frame)Algorithm Details (for each video frame)

1. Compute magnitude of gradient Prewitt filter for gradients Ix and Iy

2. Compute the g(x) uni-directionality Precompute each term in the function for the entire frame Then sum each 15x15 sub-windows, = 15x15, centered over

each pixel in the frame

3. Compute cornerness c(x)

Note– is a monotonic decreasing function from 1 to 0

– The use of integrated single derivatives is designed to be less sensitive to noise

Tracking using strong match in sub windowsTracking using strong match in sub windows OverviewOverview

Mutual Consent Pruning

Compute the effective dissimilarity of patches of corner points between 2 frames

Intensity normalized to 1 (per frame) to facilitate matching

Frame ‘a’ Frame ‘a+1’

L = 5

Tracking using strong match in sub windowsTracking using strong match in sub windows Algorithm DetailsAlgorithm Details

1. 41x41 window is centered over each corner point in frame ‘a’

2. Corner points in the corresponding 41x41 window in frame ‘a+1’ are potential matches

3. Calculate the deff between the corner point in frame ‘a’ and potential matching points in frame ‘a+1’

4. The potential matching point with the least deff is the best match

5. Repeat for all corner points

6. Repeat going backwards from frame ‘a+1’ to frame ‘a’

7. Reject corner points that do not have a match in both directions Mutual Consent Pruning

Video CompensationVideo Compensation Algorithm DetailsAlgorithm Details

Translation1. Compute one U & one V median velocity vector for each frame2. Compute the cumulative U & V median velocity vector vs. frame number3. Boxcar average (3 frames) the cumulative U & V vectors

Represents intended motion4. Translation correction is the cumulative U & V minus intended motion

Rotation1. Discard velocity vectors pointing into or away from the center (outliers)2. Define 5 pivot points, 4 points 1/3 into frame + center point3. Compute angle of rotation with respect to each pivot point4. Choose pivot point with least variance in angle of rotation5. Compute one median rotation angle for each frame6. Compute the cumulative rotation angle vs. frame number7. Boxcar average and remove intended rotation

Video CompensationVideo Compensation Algorithm Details 2Algorithm Details 2

Cum ulative U

-20

-15

-10

-5

0

5

10

15

20

25

Frame

Com pensation U

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

Frame

OutlineOutline






Phase-Based Approach Phase-Based Approach Translation (Move Up)Translation (Move Up)

0 100 200 300 400-10

-5

0

5

10

No. of Pixel

X

0 100 200 300 400-10

-5

0

5

10

No. of Pixel

Y

Frame 1 – Frame 0

Translation (Move down, left and right)Translation (Move down, left and right)

Rotation Rotation

Rotation + TranslationRotation + Translation

Harris Corner Detector Harris Corner Detector TranslationTranslation

Harris Corner Detector Harris Corner Detector Rotation & Rotation plus TranslationRotation & Rotation plus Translation

Corner detection based on local anisotropismCorner detection based on local anisotropism

Translation Error (pixels)Translation Error (pixels)

Horizontal Trans lation Error

0

1

2

3

4

5

6

0 2 4 6 8 10 12 14 16

Fram e

Pix

els

Corner Anisotropy

Harris

Block

Phase

Vertical Trans lation Error

0

1

2

3

4

5

6

7

8

9

0 2 4 6 8 10 12 14 16

Fram e

Pix

els

Corner Anisotropy

Harris

Block

Phase

Rotation Error (degrees)Rotation Error (degrees)

Rotation Error

0

1

2

3

4

5

6

7

1 2 3 4 5 6

Frame

Deg

rees

Corner Anisotropy

Harris

Block

Phase

AnalysisAnalysis

Block method gave better results for translation estimation– Reason is because a median vector was calculated for local 15x15

blocks, as opposed to other methods where a global median was used

Corner anisotropy gave better results for rotation estimation– Most likely due to preprocessing of multiple pivot point and removing vectors

pointing into and away from pivot points (outlier rejection)

Conclusion:– Not exactly a fair comparison yet

– Motion estimates from each algorithm need to be preprocessed as described above before a comparison can be made

– Has not been done due to time limit

OutlineOutline






Show VideosShow Videos

OutlineOutline






AnalysisAnalysis

Some residual unwanted motions not compensated– Error in estimating intended motion– Slow frame rate of 15 fps, need 30 fps (NTSC) or higher

Stabilized video allow for higher compression ratios (MPEG, AVI)– The fewer changes between frames, the smaller you can make the file– Difference is 0.56 MB using corner anisotropy (3.72 vs. 3.12 MB) AVI – Must process original video through same path as stabilized video

(without stabilizing it of course) for a fair comparison

ImprovementImprovement

Outlier removal for feature tracking

Kalman filter for intended motion

Hybrid algorithm

Metric for stabilization on videos without reference frames

Higher video frame rate (30 fps or better)

Preprocessing of motion estimates for higher accuracy:– Local median for translation

– Multiple pivot points and outlier rejection for rotation

QuestionsQuestions

digital image stabilization of video team 2 bo mu marvin boonmee yonghui zhao hongqin zhang

Documents

image compensationour

camera vibration

real dataanalysis of

camera movement gyros

portion of sensor

optical compensation

optical element

ccd chip