Video Object Tracking Video Object Tracking and Replacement for and Replacement for

Post TV ProductionPost TV Production

LYU0303 Final Year ProjectLYU0303 Final Year ProjectFall 2003Fall 2003

OutlineOutline

Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A

IntroductionIntroduction

Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.

Extensively used in video-making Extensively used in video-making industries.industries.

Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality

Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually

called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection

directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without prior knowledge about the video objects to be detectedWithout prior knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.

Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

RGB/HSV converterRGB/HSV converter

Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.

More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.

After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.

RGB/HSV converterRGB/HSV converter

RGB to HSVRGB to HSV HSV to RGBHSV to RGB

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

Edge detectorEdge detector Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or

brightness means that there exist a boundary brightness means that there exist a boundary line.line.

HSV: (0,0,0)

HSV: (0,255,255)

Edge detectorEdge detectorBefore edge highlightingAfter edge highlighting

Smoother for Edge detectorSmoother for Edge detector

Sometimes noise will affect the edge Sometimes noise will affect the edge detection result of low resolution images.detection result of low resolution images.

Include an image smoother to remove Include an image smoother to remove large noise points in the image.large noise points in the image.

In some cases performing a smoothing will In some cases performing a smoothing will greatly enhance the performance of edge greatly enhance the performance of edge detection due to the decrease in “fake” detection due to the decrease in “fake” edge points.edge points.

Smoother for Edge detectorSmoother for Edge detector

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

Edge equation finderEdge equation finder

Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.

Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.

Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.

This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.

Edge equation finderEdge equation finder

0

4590

(x1,y1)

Angle in degree Frequency0 1

45 3

90 1

135

135 1

Desired linear equation in point-slope form:

)1(*45tan1 xxyy

Edge equation finderEdge equation finder

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

Equation processorEquation processor

After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.

Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.

Equation processorEquation processorBefore edge findingAfter edge and equation findingAfter extra equation removal

Equation processorEquation processor The following criteria are currently adopted The following criteria are currently adopted

in the equation processor:in the equation processor: Distance between the equationsDistance between the equations Angle between the equationsAngle between the equations Whether the equation intersects the object or Whether the equation intersects the object or

not.not. Since equation processor is a potential Since equation processor is a potential

burden to the system, it may be replaced burden to the system, it may be replaced in the future by improving other parts.in the future by improving other parts.

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detector with smootherEdge detector with smoother Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

Translation detectorTranslation detector

A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to

accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.

Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.

Translation detectorTranslation detector

Records the approximate location of the Records the approximate location of the object in the key frame.object in the key frame.

When processing the following video When processing the following video frames, just scan within a certain boundary frames, just scan within a certain boundary near the recorded location.near the recorded location.

Will improve its efficiency later.Will improve its efficiency later.

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Texture mapperTexture mapper

Texture MapperTexture Mapper

A graphics design technique used to wrap A graphics design technique used to wrap a surface of a 3-D object with a a surface of a 3-D object with a texture texture mapmap

The 3-D object acquires a The 3-D object acquires a surface texturesurface texture similar to the texture map.similar to the texture map.

ColorsColors, , brightnessbrightness valuesvalues or or altitudesaltitudes

Texture MapperTexture Mapper

Mapping

(color)

Texture map Image

Texture MapperTexture Mapper

Definition of terms:Definition of terms: Image coordinates Image coordinates (r, c)(r, c) : location of pixel in : location of pixel in

the the imageimage Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture texture

mapmap which contains color information for which contains color information for image coordinatesimage coordinates

Mapping functionMapping function : determines : determines howhow texture texture coordinates are mapped to image coordinates coordinates are mapped to image coordinates or vice versa. or vice versa.

e.g. linear scan-line interpolatione.g. linear scan-line interpolation

Texture MapperTexture Mapper

Mapping

function

(u,v)(r,c)

Texture coordinates

Image coordinates

Texture MapperTexture Mapper

Definition of termsDefinition of terms Forward mappingForward mapping maps from the maps from the texture texture

spacespace to to image spaceimage space Inverse mapping Inverse mapping maps from the maps from the image spaceimage space

to to texture spacetexture space Scan-line conversionScan-line conversion: an area-filling technique : an area-filling technique

processing a surface processing a surface line by lineline by line. It can be . It can be applied with forward/inverse mapping.applied with forward/inverse mapping.

Texture MapperTexture Mapper

Forward mapping

Inverse mapping

Texture MapperTexture Mapper

Texture scanning

Image scanning

More important

Scan-line conversion

triangletriangle//parallelogramparallelogram scanning scanning line by lineline by line, from top to bottom, from top to bottom process process eacheach pixel on every line pixel on every line

Scan-line conversionScan-line conversion

Scanline yk

Scanline yk+1

scanning order

for every line

Scan-line conversionScan-line conversion

For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.

Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from scanline r.d from scanline r.

Scanline yk

Scanline yk+1

Coordinates differ by Δx and Δy

Scan-line conversionScan-line conversion

quadrilateral quadrilateral trianglestriangles or or parallelogramsparallelograms scan each scan each sub-polygonsub-polygon special case: only special case: only 22 sub-polygons sub-polygons

12

3

Scan-line conversion with forward Scan-line conversion with forward mappingmapping

AlgorithmAlgorithm

texture scanning

forward mapping functions

r = R(u,v)r = R(u,v)c = C(u,v)c = C(u,v)

copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)

for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix

Scan-line conversion with inverse Scan-line conversion with inverse mappingmapping

texture scanning

inverse mapping functions

u = U(r,c)u = U(r,c)v = V(r,c)v = V(r,c)

copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)

for (r,c) = polygon pixelfor (r,c) = polygon pixel

AlgorithmAlgorithm

ComparisonComparison

Possibility ofPossibility of aliasing aliasing

NoNoYesYesCalculation Calculation ofof

fractional area of pixel fractional area of pixel coveragecoverage

a bita bit complicated complicated as it as it involves image scanninginvolves image scanning

easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation

image image texture texturetexture texture image imagePrinciplePrinciple

Inverse Inverse mappingmapping

Forward Forward mappingmapping

ComparisonComparison

Aliasing – filtering/resampling techniques can be applied for inverse mapping

ComparisonComparison

Yes, but can be avoided Yes, but can be avoided with simple filtering/resawith simple filtering/resa

mplingmplingYesYesPossibility ofPossibility of aliasing aliasing

NoNoYesYesCalculation Calculation ofof

fractional area of pixel fractional area of pixel coveragecoverage

a bita bit complicated complicated as it as it involves image scanninginvolves image scanning

easyeasy if the mapping if the mapping function is knownfunction is knownEase Ease of implementationof implementation

image image texture texturetexture texture image imagePrinciplePrinciple

Inverse Inverse mappingmapping

Forward Forward mappingmapping

We used inverse mapping

Mapping functionsMapping functions

Simple linear transformationsSimple linear transformations Translation, scaling, etcTranslation, scaling, etc parallelogramsparallelograms only only very fastvery fast

Linear scan-line interpolationLinear scan-line interpolation Based on Based on proportionproportion any any quadrilateralsquadrilaterals

not suitable

feasible

Linear scan-line interpolationLinear scan-line interpolation

Idea of Idea of interpolationinterpolation

(x1, y1)

(x2, y2)

a

1-a

(x, y) = a(x2, y2) + (1-a)(x1, y1)

Linear scan-line interpolationLinear scan-line interpolation

(r4, c4)

(r1, c1)

(r3, c3) (r2, c2)

(r5, c5)

(r, c)

image

(u1, v1)(u3, v3)

texture map

(u2, v2)

(u4, v4)

(u5, v5)

For a particular scanline,

(u, v)

After mappingAfter mapping

a bit strange !

Shadow mappingShadow mapping Mapping of surface Mapping of surface brightnessbrightness Retain the brightness of the Retain the brightness of the original original

surfacesurface

Method:Method: Convert the original surface Convert the original surface HSVHSV Get the Get the VV value value Replace the Replace the VV value of the mapped surface value of the mapped surface

After Shadow mappingAfter Shadow mapping

More natural

!

Future WorkFuture Work

Anti-aliasingAnti-aliasing Mapping different shapes like cansMapping different shapes like cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers

Q & AQ & A