Video Mosaics

CS 294/283 Final Project

Steve MartinU.C. Berkeley, Fall 2003

Overview

• Previous Work

• Method Overview

• Metrics for Comparing Video

• Preliminary Results

• Future Work

• Questions?

Previous Work on Mosaics

• Image Mosaics– 1970’s: Chuck Close/Salvadore

Dali make image mosaics by hand – 1990’s: Papers at SIGGRAPH and

commercial programs create image mosaics

• Video Mosaics– 2002: Video Mosaic program

created at Princeton, work published at NPAR 2002

• Example. . .

– 2003: Wells Fargo commercial (done by hand)

• Example. . .

Method Overview

• Step 1: Get a LOT of source video– Quality of results highly dependant on quality and amount of

component clips!– In our case, problem solved!

• Got stock video companies donate video to the war effort• We have 200GB of high quality video to play with (more than 3000

clips)!

• Step 2: Preprocess video– Create a database of feature values for all video.– Each clip further subdivided into tiles

• Size determined by the user.• With 20 by 20 pixel tiles, each movie has 864 tiles.

– Metrics are run on each tile, values recorded– This is done with a separate program that crawls the video

repository

Method Overview (2)

• Step 3: Create video mosaic– Load in original video– Parse the previously generated database– For each clip in the original video

• Run the same metrics as was used to generate the database

– Search the database for the closest tile– Replace the original tile with one from another movie, continue

• Note that we can’t just brute force compare video or we’ll be computing forever.– How to come up with numbers that describe tiles?

Video Comparison Metric 1: Average Color

• Want replacement clips to have close to or same average color as original tile.

• To calculate:– Just add color values of pixels in movie tile and divide.

• Actually not terrible by itself. . .– As tiles get smaller, approach pixel size, eye blends them

• However, note that average color can be ‘fooled’– i.e. a clip that is half red and half green averages to olive brown:

• Need something else. . .

0.5 * + 0.5 * =

• In addition to average color, want some kind of measure of color frequency.– Helps solve previous problem, although not entirely

• To calculate:– For each pixel of each frame of each tile in movie:

• Convert from RGB to HSB • Take hue value and histogram

– Create histograms on a per-tile basis

• Number of histogram buckets is changeable– Usually use around 10.

Video Comparison Metric 2: Color Histograms

Vs. Vs.

Video Comparison Metric 3: Edge Histograms

• Also would like to have some kind of edge comparison between original and replacement tile.

• To calculate: – Foreach frame of each tile– Convert frame to black and white

• Don’t care about color—want edge data– Resize the image down to 10 x 10

• We want major edges only• Use a bicubic resize to blur the image slightly

– For each pixel in the new image, • Calc intensity gradient in the x and y directions• Take the arctangent and histogram.

– Do for each frame, create histogram.

• Example of this metric at work:

θ

Finding the Best Match• Currently, a linear search through a large unordered flat file.

– Parse this into a simple row/column data structure, keep it in memory.– Search not as bad as it sounds; takes about a second with 259200

entries.– Coming soon: some tricks to make this much better.

• For each tile of the source movie, compare with each tile entry in the database.– Get the absolute difference between average colors– Get the distance between color histograms– Get the distance between edge histograms

• Add difference values to get a value for how good a match this tile is for the original.– Can weight different metric to emphasize different features

• Choose entry that minimizes this.

Video vs. Photo Mosaics

• Why bother using video?– Could mosaic each frame individually, and then string the frames

together.– Example. . .

• Would like to exhibit some kind of temporal consistency– I.e. each tile is recognizable as a movie instead of a

compression artifact.– Neat effect; less distracting for the eye.

Preliminary Results

Preliminary Results

What’s Next?• Need to implement some way of dividing tile clips into arbitrary length.

– Approximation: subdivide tiles into groups of frames.

• Index the rest of the video so I can leverage the full library.

• Still working on good ways to preserve temporal locality– i.e. place video tiles in a manner that makes sense over time in addition to just

per frame

• Include metrics that look at neighboring tiles to avoid drastically different edges

– These just end up looking like compression artifacts– Princeton’s video has this problem

• Different sized tiles– I think a ‘patchwork’ effect would look interesting, albeit harder to compute.

• Optimize, optimize, optimize!– Takes forever to make one of these!

References• Klien, Grant, Finkelstein, and Cohen. “Video Mosaics” Proceedings

of NPAR 2002.

• Microsoft Video For Windows developer’s site

• Apple Quicktime SDK developer resources

• James O’Brien, Alexander Berg, Charless Fowlkes

Questions? Comments?