i n t r o d u c t i o n t o c o m p u t e r g r a p h i c s antialiasing & image processing...

I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S

Antialiasing &

Image Processing

Andries van Dam October 12, 2004 Antialiasing & Image Proc 1/86



• Images & Hardware

• Example Applications

• Jaggies & Aliasing

• Sampling & Duals

• Convolution

• Filtering

• Scaling

• Reconstruction

• Scaling, continued

• Implementation

Outline



• Is an entire subject in and of itself• Has its own publications and conferences

– Computer Graphics, Vision and Image Processing (CGVIP), IEEE Transactions on Image Processing, PAMI, Image and Vision Computing

• Was once closer to signal theory and audio processing than to graphics

• Shares hardware with other aspects of graphics– floating point processors (FPUs)– digital signal processors (DSPs)– 24-bit or 32-bit framebuffer– scanners– digital cameras– color printers– video frame grabbing

• Shares software with graphics. Many common applications include an image processing component:

– Microsoft WordTM

– Adobe InDesignTM

– Macromedia Flash MXTM

1. This unit was originally created by former head TA Lars Bishop and subsequently modified by John Hughes, former head TAs Sam Trychin and Sascha Becker, Dan Gould, and by Andy van Dam.

2. For more information, refer to Chapter 17 of the textbook, “Image Manipulation”, and Section 14.10, “Aliasing and Antialiasing.”

Antialiasing & Image Processing

Andy van Dam

don't know InDesign but isn't PhotoShop much more obvious and better known? i don't know how Word allows image manip - can you tell me?

Andi Fein

dan said the point of the slide was non-image processing programs use image processingword allows image scaling, rotation, color manipulation, and cropping



For our purposes, an image is:• A 2D domain

• With samples at regular points (almost always a rectilinear grid)

• Whose values represent gray levels, colors, or opacities (or tissue density in an MRI scan)

Common image types include:• 1 sample per point (B&W or Grayscale)

• 3 samples per point (Red, Green, and Blue)

• 4 samples per point (Red, Green, Blue, and “Alpha”, a.k.a. Opacity)

What is an Image?



Where is the image stored?• In order to look at image processing, we need to take

a short look at a computer’s graphics hardware• Graphics hardware is implemented as a separate

subsystem• The graphics subsystem communicates with the CPU

and main memory across a special high-speed bus in most modern system architectures (PCI Express)

Hardware

8.0 GB/s for the graphics card!!

Only 133.0 MB/s for PCI board

MCH: Memory Controller Hub

RAM

PCI: Peripheral Component Interconnect

ICH: I/O Controller Hub

Andy van Dam

1) i don't understand all the abbreviations - are you going to narrate the diagram?2) you show PCI Express, yet talk about AGP. Isn't PCI more common and PCI express starting to come in?

Andi Fein

added the boxes with names of different partnow only mentions pci express (not agp)



A framebuffer and more• Has local memory for the image

• Takes commands from the host CPU

• Can have on-board processing for various 2-D and 3-D operations

• Video hardware converts image in framebuffer to analog signal (DAC or RAM-DAC)

• Contains Look Up Tables (LUTs) explained in the intro lecture

Graphics Subsystem



A simple example

• Divided into bit-planes

• Each plane represents 1 bit of the color value at that pixel

• Each plane is size of display (1280x1024 for SunLab)

• For a basic framebuffer representing each of R, G, B with 8 bits, the framebuffer is 24 bits deep

The Framebuffer (1/2)

Blue

Green

Red

Andy van Dam

is this the panel rez today?!?

Andi Fein

dave says yes



A more advanced framebuffer• Double buffering doubles the planes in our framebuffer• For a window system, also want a backing store

3D requirements are even more demanding:• Z-buffer to hold depth values

– we will discuss this in the Visible Surface section• Stereo doubles our requirements yet again

– one set of buffers for each eye• Specialized buffers for various effects

– A-buffer (Accumulation buffer for transparency, etc)– stencil buffer (for masking regions to draw in)– overlay planes (for removable drawing on top of 3D

scene)• Machines like the SGI UltimateVisionTM have

framebuffers which are thousands of bits deep!– can add up to gigabytes on high-end graphics

systems

The Framebuffer (2/2)

Andy van Dam

is this the current SGI m/c?

Andi Fein

according to dan, this is current info



An expensive proposition• Tons of memory, all of which is

updated several times per second

• Typical (cheap) memory cannot do this

• Graphics hardware makers build very sophisticated memory systems

• Special types of memory– dual-ported memory

– logic-enhanced RAM

• Parallel, Interleaved memory system

• The problem is getting worse – memory speed is improving MUCH more slowly than the rest of the hardware

Framebuffer memory



Each of these planes is a “channel”. For example, the red channel of a 3 sample per pixel image would itself be a 1 sample per pixel image, consisting of all of the red values in the original image.

Channels

Blue Channel of Image,1 sample per pixel

Green Channel of Image,1 sample per pixel

Red Channel of Image,1 sample per pixel

3 samples per pixel



Combining various sorts of data• If your image contains z-values, you can

use the gradient of the z-values to add color: sudden changes in z-depth get colored black, for example. This highlights the silhouette edges of things

• There’s a whole body of material on tricks like these

Multi-channel operations



Adding “coverage” information to pixels• See Chapter 17 for details

• In addition to R, G, B channels of an image, we can add a fourth channel, called α.

• Alpha varies between 0 and 1; a value of 1 represents a “fully covered” pixel, one you cannot see through

• A value of 0 is a completely transparent pixel

• Useful for blending images: the one with the higher alpha value “shows through” more

The Alpha Channel



The general stages of image processing can be flowcharted in an idealized pipeline-not all stages need to be present and the middle two or three stages may be interlaced:

Stages of image processing



Encompasses a wide variety of techniques in two basic categories:

• Synthesis: images created in a computer– rendered from geometry (e.g., RenderManTM, Alias|

Wavefront’s MayaTM, autodesk 3ds maxTM)– painted (e.g., MacPaintTM, Fractal Design PainterTM)

• Capture: images from the “real world”– must be “captured” or digitized; turned into digital

information instead of analog signal or printed media– method of digitization is partially determined by

image’s original form– common methods include:

• digital camera• satellite data transmission• drum scanner• slide scanner• flatbed photo scanner • video frame grabber/converter

• Combination: texture-mapped polygons– the texture is essentially an image, and the polygons

part of a model.

Stage 1 – Image AcquisitionHow do we get images?

Andi Fein

i know autodesk 3ds max looks weird, but it's what the website says...

Andy van Dam

as always, are these products extant? the most uptodate? spelled correctly? e.g., 3ds for studiomax?



• Source image(s) is adjusted to fit a given tone, size, shape, etc., to match a desired quality or to match other images

• Can make a set of dissimilar images appear similar (if they are to be composited later), or make similar parts of an image appear dissimilar (such as contrast enhancement)

• Preprocessing techniques include:– adjusting color or gray scale curve– cropping– masking (cutting out part of an image to be used

in a composition, or to leave a hole in the original image)

– scaling up (super sampling)/ down (sub sampling)

– blurring and sharpening– edge/enhancement– filtering and antialiasing

• Notes:– techniques like blurring, sharpening, and edge

detection can also be a postprocessing techniques– some preprocessing algorithms are not followed

by mapping, others that involve resampling the image may be interlaced with mapping: filtering is done this way

Stage 2 – Preprocessing



• Mapping is catch-all stage where several images are combined, or geometric transformations are applied

• Transformations include:– rotation

– scale

– stretch– feature based image warp (aka morphing)

• Compositing:– opaque/transparent paste

– alpha-channel composition

Stage 3 – Mapping



• Creates global effects across an entire image or selected area(s)

• Art effects– posterizing

– faked “aging” of an image

– faked “out-of-focus”

– “impressionist” pixel remapping

– texturizing

• Technical effects– color remapping for contrast enhancement

– color to B&W conversion

– color separation for printing (convert RGB to CMYK)

– scan retouching and color/contrast balancing

Stage 4 – Postprocessing

Andy van Dam

i've forgotten: what is texturizing? can you show an image? also, is color separation calculating the CYMK components from transformations of RGB to CYMK?

Andi Fein

color separation is converting rgb to cmyk



• Choice of display/archive method may affect earlier processing stages– color printing accentuates certain colors

more than others

– colors on the monitor have different gamuts and HSV values than the colors printed out: need a mapping

• Techniques– monitors

– color printers

– film/videotapes

– disk files

– texture maps for 3D renderer

Stage 5 – Techniques for Archive And Display







• Convolution

• Filtering

• Scaling

• Reconstruction


• Implementation

Outline



Give an idea of what is done with image processing

• Image enhancement– scientific filtering

– forensic science

• Multipart composition

• Computer vision

Examples



Some filtering techniques are designed to make features in an image more apparent

• Done by using a filter that accentuates changes above certain threshold

• Causes specific features of an image to stand out• Can even calculate a new image based on some

function that takes an image to another image• e.g., define an image that is magnitude of change

of original image at each point. Thus, higher-valued pixels in new image are places where original image was changing rapidly

An Application of the Edge-Detection Filtering Technique

Just an illustration, not an MRI: see the next slide



Take slice from MRI scan of canine heart, and find boundaries between types of tissue

• Image with gray levels representing tissue density• Using filter from previous slide, compute new image.

Again, new image brighter where MRI image gray values changing faster

• Different densities of different types of matter will show up with bright boundaries in between.

MRI Image Enhancement

Original MRI Image of a Dog Heart

Edge Detection Image



Image enhancement has been used by forensic scientists for years to pull information from seemingly hopeless images.

• We have a security camera video of the back of a car that was used in a robbery

• The image is too dark and noisy for the police to pull a license number

Image processing like this in the media a lot in the last few years

• These techniques have been used to find small features in satellite images

• Image processing for forensic science is even spotlighted in popular entertainment, such as the book (and movie) Rising Sun, by Michael Crichton and the tv show CSI: Crime Scene Investigation

Forensic Science Image Enhancement

Andy van Dam

any of the forensic TV shows that use this stuff?

Andi Fein

no time to watch tv anymore, but the website says CSI: Crime Scene Investigation is about a team of Las Vegas forensic Crime Scene Investigators who solve crimes by studying the evidence



Normally, image enhancement uses many filtering steps, and often no mapping at all.

Image Enhancement Steps



Step One: Get the frame from the videotape digitized with a frame-grabber

Image Enhancement (1/4)

• Frame grabbing is no longer expensive – video capture boards for PC’s are now under $60

• Though humans can often discern an image of poor quality, filtering can make it easier for a pattern-recognition algorithm to decipher embedded symbols: in this case, alphanumeric characters



Step two: Crop out stuff that appears to be uninteresting (outside plate edges)

• This step can speed process by doing image processing steps on fewer pixels

• Can’t always be done –may not be able to tell which sections are interesting without some processing




Step Three: Use edge-sharpening filter to add contrast to plate number

• This step enhances edges by raising discontinuities at brightness gaps in image




Step Four: Remap colors to enhance contrast between numbers and plate itself

• This stage has truly brought out image - mapped low-valued pixels to bright white, and almost all others to black.

• Now, can make a printout for records, or just copy plate number down. YNN-707!

• Note that final colors do not even resemble real colors of license plate. Enhancement techniques have seriously distorted the colors!




Image composition is popular in art world, as well as in tabloid news world

• Takes parts of several images and creates single image. Hard part is making all images fit together naturally

• Artists can use it to create amazing collages and multi-layered effects

• Tabloid newspaper artists can use it to create “News Photos” of things that never happened

• Some famous examples previously mentioned:– cover of TV Guide with Oprah

– cover of Time and Newsweek with O.J. Simpson

– clips from Forrest Gump; ads w/ old stars

– university brochure altered to show ethnic diversity

Multipart Composition



Lars Bishop, former CS123 Head TA and primary author of these lectures, created a news photo of himself “meeting” with former Russian President Boris Yeltsin.

• Needless to say, Lars Bishop never met Mr. Yeltsin

• So, had to get images, cut out the parts he wanted, touch them up, paste them together, and retouch the end result

Our Example



Step 1 – Getting the images• The image of Boris was pulled from the internet

• The image of Lars was staged in the graphics lab, filmed with a camcorder, and digitized live

Image Composition (1/4)

Andy van Dam

images are dark on my brandnew LCD panels. how do they show in Lubrano? lighten up as needed...

Andi Fein

i think they looked pretty good in lecture today



For Boris, Lars did the following:• Found image of Boris and Gorbachev

on an FTP site, transferred it to Brown

For image of Lars, he did the following:

• Set up a video camera to use same type of camera angle as the shot of Boris

• Wired camera to a frame digitizer

• Lars posed in front of camera

• Grabbed image

• Could have simplified process by putting a white sheet behind himself, but could not be found in time

Step 1: The Details



Stage 2 – We must cut the pictures we want out of the original images

• Boris is cut from the internet image

• Lars is cut from the video image




For Boris and Lars• Lars brought images into Photoshop on his Mac,

and painted a region around important parts of images (outline of people) in white

• Continued touching up this outline until no background at edge of people

• Used a lasso tool that could grow until it hit the white background, thus selecting subject

• Cut subjects out of their images

Step 2: The Details



Step 3 – Filter the images to make them appear similar, and paste them together

• Boris is blurred and brightened, to get rid of the halftoning lines (must have been a magazine photo). Also scaled down to be same size as Lars

• Lars is blurred and noise is added to match quality of image to that of Boris




For Boris:

• Blurred image to get rid of the halftoning lines

• Brightened image to fit colors of other image

• Scaled him down to be same size as other subject

For Lars:

• Blurred image to fit blurring of Boris

• Added noise to lower quality of otherwise clean video grab to quality of scanned image of Boris

Then, they are composited together

• Lars’ hand is retouched to merge well with Boris’ document.

Step 3: The Details



Final Image (with retouching at edges)

BISHOP AND YELTSIN TALK PEACE

BISHOP: “I couldn’t understand a single word he said!”




For this step, Lars did several little touches to add to quality of fake

• Created a simple, two-color background and added noise to it to fit rest of the image

• Placed cutout of the two subjects on top of background

• This left a white halo about 3 pixels wide around the subjects, so used a “Rubber Stamp” tool to stamp background noise patterns over halo, making seams appear less obvious

• Additional loss of contrast and spatial resolution due to printing process

Step 4: The Details

Step 5: Print the Image



Image enhancement also done to enhance images for computer vision

• Computer must do all processing, without human intervention

• Processing techniques must be fast. If slow, will add to camera-to-reaction lag (latency) in system

• Common preprocessing techniques for computer vision include edge enhancement, region detection, contrast enhancement, etc..

• Computer must do more than enhance an image before reacting to it. Must pull specific information from image, such as position and orientation of edges

Computer Vision (1/2)



Image enhancement can also be done to enhance images for computer vision

• Image processing makes information easier to find• Pattern detection and pattern recognition are

separate fields in their own right– Pattern detection: looking for features and

describing the image’s content at a higher level– Pattern recognition: classifying collections of

features and matching them against library of stored patterns. (e.g., alphanumeric characters, types of abnormal cells, or human features in the case of biometrics)

– Pattern (feature) detection is one important component of pattern recognition.

• For more on computer vision, check out CS143 or CG120!

Computer Vision (2/2)



MRI scans and the like create 3D blocks of data. As time goes by, image processing may move furtherand further into 3D image processing.• Techniques we used in 2D often have 3D analogs• Display becomes far more difficult: voxels replace pixels• Data storage issues become problematic:

– 1024x1024 4 channel image = 4 megs– 1024x1024x1024 4 channel 3D image = 4 gigs!

• Computational issues can also become problematic – N2 becomes N3

• AutoStereoscopic displays hot research topic – most work with lenticular optics. DTI’s Virtual WindowTM (shown in image) is an example

• LightSpace Technology’s DepthCubeTM produces a real 3D display (no stereo glasses). Display uses 20 LCD slices to display image

• Holographic displays are being worked on also (e.g., MIT’s Media Lab)

• Computer vision can be used as part of a passive UI, as an alternative to intrusive (tethered) gadgetry such as 6DoF “space mice”, wands, data gloves

• Autostereoscopic monitors for sale:http://www.sharp3d.com/

The Future – 3D?

Andy van Dam

see if you can come up w/ any autostereoscopic displays that are being sold or subject of recent siggraph papers. love some pics of it and LightSpace display

Andi Fein

added blurb about DTI's virtual window and picture from website, pics of depthcube weren't very impressive, dave added last bullet







• Convolution

• Filtering

• Scaling

• Reconstruction


• Implementation

Outline



Now that you understand the big picture, let’s start focusing on specific topics

• Stages 2, 3 and 4 contain numerous important topics and will now be discussed in more depth

Start with antialiasing• What is aliasing?

• What is anti-aliasing?

• Why is it necessary?

• How is it performed?

Getting down to details



• Representing a line with discrete pixel values is sampling a continuous function

• Jaggies are a manifestation of sampling error and loss of information

• The term aliasing will be defined momentarily

• Doubling resolution in x and y only lessens the problem

• Costs 4 times memory, memory bandwidth and scan conversion time!

Jaggies-Manifestation of Aliasing

(a) Standard midpoint line on a bilevel display.

(b) Same line on a display that has twice the linear resolution

Andy van Dam

b) looks tilted a few degrees clockwise...also, first pic should come under first bullet, and then have the second pic come up under the "doubling resolution where you show them side by side.

Andi Fein

fixed by dave



• Sample unit rectangle rather than mathematical (infinitesimally thin) line, and switch to representing pixels as squares centered at x-y intersections rather than circles (they are neither!)

• Midpoint algorithm: pick single pixel closest to center line of rectangle (0 < slope < 1). This is a form of point sampling: Sample the mathematical center line at each of the integer X values. As with our original midpoint line algorithm, we pick a single pixel to represent the line’s intensity.

• Only single value of intensity used regardless of amount of overlap between each pixel and rectangle. Therefore all but one pixel in each column is “off,” and all have same (max) brightness.

Line as 1-Pixel Width Rectangle

Line (as unit-width rectangle) from point (1,3) to point (7,8)



• Set each pixel’s intensity value proportional to its area of overlap covered by primitive; thus sample sub-areas of rectangle (that represents line) covered by rectangle representing pixel.

• Note more than one pixel/column for lines of

0<slope<1

• This is a form of unweighted area sampling– only pixels covered by the primitive can contribute

– only total amount of area of overlap matters, regardless of the distance of any individual sub-area of overlap from pixel’s center

• Area sampling smoothes sharp transitions

First Antialiasing Attempt

Figure 3.36: Intensity of a pixel is proportional to its area covered by the line


• Weight function W(x, y) gives weight for incremental area dA centered at (x, y); constant here, hence the name box. Box is single pixel wide here, but could vary width (as we’ll see).

• For each pixel intersecting line, intensity contributed by each sub-area dA is (W(x, y) • dA)

• Then total intensity (a fraction between 0 and 1) integrated over area of overlap is:

• Integral is volume over area of overlap (in above figure, a rectangular wedge). Check out applet comparing point and area sampling:

http://www.cs.brown.edu/exploratories/freeSoftware/catalogs/signal_processing.html

• Note that, in two dimensions, box functions are often referred to as pulse functions.


“Box Filter” Represents Unweighted Area Sampling

A

dAyxW ),(

Andy van Dam

i rearranged bullets - the url went illegible. pls fix and check animation.

Andi Fein

animation fixed


• Area sampling, but weight overlap between filter and primitive so that sub-areas dA closer to center of pixel count more. Revolve -45 degree slope to get 3D cone shape: linear falloff with distance, and circular symmetry

• Circular cone =>change back to circular pixels; we set cone’s diameter (its support) to two pixel units. To calculate given pixel’s contribution, calculate volume of wedge of filter whose base is area of overlap between support and primitive. dA has greater weight if closer to center of pixel, Normalize so that volume = 1; subvolume subtended by intersection between line and filter is a conical wedge.


Cone Filter for Weighted Area Sampling (1/2)

Cone filter for circular pixel with diameter of two grid units.

Andy van Dam

split the animation on sentence boundaries in both bullets. Ideally i would have liked subbullets but there isn't room.

Andi Fein

not sure how to make animation occur by sentence, only wants to do it by bullet point...



2-unit circular support of filter

Pixel center (+)

Area of overlap between support and

primitiveDifferential area dA2

Primitive Differential area dA1

Weighted Area Sampling (2/2)

W(x, y) is the weight which is multiplied with dA at (x, y); normalize W to make volume under cone = 1


• 2 unit support implies up to 3 pixels/column

• Gupta-Sproull algorithm provides fast anti-aliased lines via table look-up (see book)

• Weighted area sampling does a better job of smoothing than unweighted area sampling.


Cone Filter for Weighted AreaSampling (2/2)

Fig. 3.60 One-unit-thick line intersects 3-pixel supports.



• This simplistic scan conversion algorithm only asks if a mathematical point is inside the primitive or not

– bad for sub-pixel detail which is very common in high-quality rendering where there may be many more micro-polygons than pixels!

Another Look at Problems ofPoint Sampling

Fig. 14.10 Point-sampling problems. Samples are shown as black dots. Object A and C are sampled, but corresponding objects B and D are not.



• The box filter sets intensity proportional to area of overlap

• Get winking of adjacent pixels which eye can detect

– eye is very forgiving of TV-type noisy images with small changes between frames

– eye is very sensitive to pixels winking in images of more uniform appearance

Another look at Unweighted AreaSampling (1/3)

Fig. 14.11 Unweighted area sampling. (a) All sub-areas in the pixel are weighted equally. (b) Changes in computed intensities as an object moves between pixels.

(b)



• Pyramid filter approximates circular cone to emphasize area of overlap close to center of pixel

• Support is equal to pixel area

Another look at Weighted Area Sampling (2/3)

Fig. 14.12 Weighted area sampling. (a) sub-areas in the pixel are weighted differently as a function of distance to the center of the pixel. (b) Changes in computed intensities as an object moves between pixels.

(b)



• Support of symmetric cone filter greater than pixel area for greater smoothness

Another Look at Weighted Area Sampling (3/3)

Fig. 14.13 Weighted area sampling with overlap. (a) Typical weighting function. (b) Changes in computed intensities as an object moves between pixels.

(b)



for each pixel p

place filter centered over p;

for each pixel q under filter

weight = filter value over q;

intensity_p += weight * intensity_q;

Pseudocode







• Convolution

• Filtering

• Scaling

• Reconstruction


• Implementation

Outline



• Scan converting a geometric scene or scanning in a natural image is digitizing a sequence of continuous intensity function, one per scanline

Fig. 14.8 Image (a) Graphical primitives. (b) Mandrill. © Intensity plot of scan line αin (a). (d) Intensity plot of scan line α in (b). (Part d is courtesy of George Wolberg,Columbia University.)

• Note step functions abound at the edges of geometric primitives – these represent infinite spatial frequencies that cause aliasing because we can’t sample them correctly (What is the intensity value at an edge?)

Sampling of Images



• Scanning in a photograph and displaying it:– scan in a sequence of intensity functions, 1 per scan line– digitize each by sampling at discrete points, losing both

spatial and intensity information– reconstruct the signal, e.g., with continuous analog voltage

wave form to CRT (via DAC)

The Sampling/Reconstruction/Display Pipeline

Fig. 14.9 The original signal is sampled, and the samples are used to reconstruct the signal. (Sampled 2D image is an approximation, since point samples have no area.) (Courtesy of George Wolberg, Columbia University.)

Andy van Dam

can we update last sub-bullet to say how it works on an LCD panel? is there a DAC?

Andi Fein

based on dan's explanation, lcd are more complicated. dan says to just use the crt for this example



Waveform Synthesis

Fig. 14.14 A signal in the spatial domain is the sum of phase-shifted sines. Each component is shown with its effect on the signal shown at its right. Approximation of Fig. 14.8(d) (Courtesy of George Wolberg, Columbia University.)



• Intensity of sound is pressure wave amplitude

• Frequency of pressure wave determines its “pitch”

• Lowest frequency is “fundamental frequency”

• Integer multiples of the fundamental frequency are called “harmonics”

• Power of two multiples of the fundamental frequency are called “octaves”

For more, an applet to interact with these concepts:http://www.cs.brown.edu/~dlg/gfxnotes/signal/wavesynth/

Digression: Music



• Many problems can be made easier by transforming them to another problem (must also be a way to go back!)

• May be easier to characterize a signal by its frequency spectrum –signal and frequency domains are duals and represent identical information. May be easier to filter in frequency domain than in signal domain, as we shall see.

• Take a familiar problem: multiplication of numbers. Can take logarithm of number, perform operations on log(number), then move back using antilogarithm

• Dual of multiplication is addition in logarithmic “space”:

If ab = c then

log (a) + log (b) = log (c)

• This “invertible transformation” makes slide-rules such effective tools for multiplication: manipulating sliders corresponds to manipulating numbers via their logs.

Digression: Analogous Operation

Andy van Dam

slide rule pic comes up w/ its bullet

Andi Fein

fixed


Andries van Dam October 12, 2004 Ant aliasing & Image Proc 62/86

• Sine wave is characterized by amplitude and frequency• Frequency of a sine wave is number of cycles per

second for audio, or number of cycles per unit length (e.g., inter-pixel distance) for images

• Can characterize any waveform by enumerating amplitude and frequency of all its component sine waves (Fourier transform – see chapter 14)

• This can be plotted as a “frequency spectrum”, a.k.a. power spectrum, (we ignore negative values, but they are needed for mathematical correctness)

Frequency Spectrum of a Signal

For fun with Fourier transforms see:http://www.ysbl.york.ac.uk/~cowtan/fourier/magic.html

Andy van Dam

again, animation is wrong - display images w/ their bulletalso, i don't understand the amplitudes in the frequency plot - why isn't the DC component .5? and the amp(.25) also .5?

Andi Fein

is this the animation you want? i'm not sure what picture should correspond to what bullet pointfollowing dan's suggestion, i got rid of the vertical values since we really only care about the quantitative aspect anywayadded the website dan suggested at the bottom



When we sample, we must sample at a rate that is higher than 2 times the highest frequency in thesignal (the Nyquist limit)• Here is an approximate analog sine wave

• Here is the sine wave sampled at an acceptable rate (4 times the highest frequency)

• Here is the reconstructed wave based on these samples

Sampling: The Nyquist Limit



Aliasing is a problem that can occur whenever we sample a signal at less than twice maximum frequency

• Here is our analog sine wave again

• Here is the sine wave sampled at too low a rate

• Here is the reconstructed wave based on these samples

• Uh, oh… The reconstruction isn’t even close!

Aliasing: Know Thine Enemy

Andy van Dam

last fig anim is wrong

Andi Fein

animation fixed



Sampling right at the Nyquist limit can be problematic• Here is our perfect analog sine wave again

• Here is the sine wave sampled at the Nyquist limit. This time it works fine

• Here is the same sine wave sampled at the Nyquist limit, with the sample points shifted. Now we get no signal

For an applet dedicated to illustrating the Nyquist Limit:

http://www.cs.brown.edu/~dlg/gfxnotes/signal/nyquist/

Sampling At the Nyquist Limit



• Nyquist Applet




• These low frequencies are “aliases” of high frequencies

• The low sampling rate data could not adequately represent the high frequency components, so it represented them incorrectly, as low frequencies

• So, we just sample above the Nyquist limit, right?

• Regrettably, we can’t always do that

• What about this?

The Enemy is RecognizedAliasing is shown in bottom diagrams on previous slides – Signals that are sampled at too low a rate can have their high-frequencies appearing in reconstructed signal as low frequencies



Square wave has infinite frequencies of sine waves in it, right at jumps from low to high and high to low (see Fig 14.15). How can we sample correctly?

• We can’t. Pure and simple. •2=, and we can’t sample at an infinite rate (unfortunately infinite frequencies are the rule in synthetic computer graphics with its discrete transitions between adjacent pixels)

• So, do inverse operation, which is best we can do. Instead of increasing our sampling frequency to meet signal, we:

– pre-filter out high frequencies we can’t represent – now signal we sample is guaranteed to consist only of

frequencies we can represent and reconstruct with reasonable accuracy with our samples

• This isn’t same signal that came in, but it’s closer than version that would cause aliasing

• The more high frequencies we pre-filter out, the lower the sampling frequency needed but the less the filtered signal resembles original

• Note: pre-filtering is often just abbreviated as filtering, but the prefix “pre” helps remind us that post-filtering (i.e., another stage of filtering after image computation or transformation) is also practiced.

Infinite Frequencies



Image Scaling Aliasing, or “Why do we have to prefilter?”

This doesn’t lookright at all!

no stripesblacker average

Andy van Dam

make the subbullets look like subs and come in progressively



Scale Aliasing II, or“Close, but no cigar?”

Better,butnotperfect



The scaled image with pre-filtering looked a little better, but we still couldn’t see any stripes

• The filter made scaled image have same relative brightness, but no stripes

• Filter removed “high frequencies” from image –discontinuities that were stripes

• Given number of points to represent image once scaled, not enough points to represent high frequencies

• We’ll never be able to represent frequencies higher than ½ our sampling rate. We can’t do better that this blurred approximation (remember Nyquist limit)

• Now lets see how we do pre-filtering in practice:– first show effect of low-pass filtering in both signal and

frequency domains to eliminate unwanted high frequencies

– then discuss mathematics of filtering, called convolution

Image Scaling Aliasing III, or“Why is it still wrong?”







• Convolution

• Filtering

• Scaling

• Reconstruction


• Implementation

Outline



Low-Pass Filtering(Spatial Domain)

Fig. 14.20 The sampling pipeline with filtering. (Courtesy of George Wolberg, Columbia University.)

Andy van Dam

this is probably a single image so you can't easily animate the successive pieces, right?it's ok the way it is


• Box function in frequency domain corresponds to “convolution” with sinc function in spatial domain (defined next)


Low-Pass Filtering(Frequency Domain)

Fig. 14.13 Multiplying with the box function in the frequency domain. (a) Original spectrum. (b) Low-pass filter. (c) Spectrum with filter. (d) Filtered spectrum. (Courtesy of George Wolberg, Columbia University.)

1

1



• Convolving signal f(x) with filter function g(x)

– value of h(x) at each point x is integral of product of f(x) and g΄(x) where g΄(x) is g(x) flipped about vertical axis and shifted so origin is at x: replace x by τ and g΄(x)=g(x-τ)

– τ is variable of integration that takes x as local origin – diagram shows how g΄(x) is constructed using a box filter– if filter g΄(x) has finite support, it does weighted average of all

pixels within support, centered at x– The mathematical ‘flip’ doesn’t matter for symmetric functions

like ours (see dirac-delta applet)

– note in filtered signal h(x) how discontinuities of f(x) are smoothed out, and the base is broadened

Convolution – Math!Or Just Play with the Applets

dxgfxgxfxh )()()(*)()(



• Convolution applets




Duals

Spatial Frequency

Multiplication Convolution

Convolution Multiplication

Box Sinc

BoxSinc







• Convolution

• Filtering

• Scaling

• Reconstruction


• Implementation

Outline



Low-Pass Filtering: Convolving with sinc

Fig. 14.23 Low-pass filtering in spatial domain. (a) Original signal. (b) Sinc filter. (c) Signal with filter, with value of filtered signal shown as black dot at filter’s origin. (d) Filtered signal. (Courtesy of George Wolberg, Columbia University.)

• Note that in theory sinc has infinite extent, however small the contributions, and negative lobes, but weights contributions at the center most heavily. Practically, it is decently approximated with gaussian (normal) distribution, or even triangle, with finite extents and weights greater than or equal to zero.

Andy van Dam

if possible, have label come in same time as the pics



• Select points at which to sample, e.g., pixel centers• As per 14.23, slide filter over each successive sample point

and compute convolution integral at that point (the area under product curve). This is weighted average of current pixel and its nearby neighbors (most useful graphics filters are symmetric about their origin and fall off rapidly from their center).

• Weighted average is value of sample/pixel for filtered image• Did much the same when we calculated total response of a

color receptor to incoming intensity distribution I(λ) by computing area under I(λ)•f(λ) product curve, where f(λ) is response function (i.e., filter function) of receptor.

• These illustrations are 2D, but for pixels one should think of sinc as three-dimensional surface of revolution and primitive as having area. [see slide 44] Of course, we’re not actually going to compute the integral, but will merely look up discrete values of filter function and do discrete multiplication and summing to approximate it.

• Use term “filter” strictly in signal-processing sense, while Unix™ and Photoshop ™ have their own notions of filters. Some of Photoshop’s filters are signal-processing filters, while some are arbitrary transformations of image.

For more information, check out convolution applet at:http://www.cs.brown.edu/research/graphics/research/exploratory/

research/applets/appletDescriptions/specialFunctionConvolution/home. html

What Does Filtering/Convolution Do?



• Answer: Convolution and filtering compute area under the h(x) = f(τ) • g’(x) product curve and are proportional to weighted area sampling

• Let’s sample 9 times. Then the discrete weighted area sample=

• Each box is hi tall and wide

• Total area

• Area under product curve =

• 1 ≤ length of interval ≤ 2

Relation btwn. Weighted Area sampling and Area under Curve(this data is mostly here for the mathematically inclined)

4

4 9i

ih

8

2a

WASah

ah

aa

h iii 2

92

82

8

2

interval oflength

)(')( 0

dxxgfWAS

)(f )(')( xgf

filter triangle)(' xg

)(')(sampling

xgf



• Filtering is nothing more than weighted averaging around a pixel to smooth image

• The dual of sinc function in spatial domain is box (aka pulse) function in frequency domain, and dual of box function in frequency domain is sinc in spatial domain

• The dual of box function in spatial domain is sinc function in frequency domain, and vice versa

• Multiplying frequency distribution with box function in frequency domain provides exact low-pass filtering, i.e., filtering out all high frequencies past a specified one. Hence, optimal filter is sinc in spatial domain (see figure 14.24)

• Multiplication of F(x) and G(x) in frequency domain corresponds to convolution of their duals, f(x) and g(x), in spatial domain, and vice versa

Filtering Summary (1/4)

)(*)()()( xgxfxGxF )()()(*)( xgxfxGxF



Fig. 14.24 (a) Sinc in spatial domain corresponds to pulse in frequency domain. (b) Truncated sinc in spatial domain corresponds to ringing pulse in frequency domain. (Courtesy of George Wolberg, Columbia University)




• To get low-pass filtering (i.e., filter out high frequencies), we often use convolution with triangle function in spatial domain to approximate ideal sinc

• Properties of triangle function: – easy to compute, unlike sinc, which has an infinite

support (or even a Gaussian approximation to sinc)

– its dual, sinc2, is acceptable approximation to a box function although it has infinite extent (see fig. 14.25b on next slide)

– these cause inaccurate representation of all frequencies and therefore some degree of corruption/aliasing must occur. This is not as bad as using box as a filter with sinc as its dual in the frequency domain. In other words, weighted area sampling of any kind, providing it is roughly cone-shaped, is better than unweighted area sampling with a box filter, which is better in turn than point sampling




Fig. 14.25 Filters in spatial and frequency domains. (a) Pulse – sinc. (b) Triangle – sinc2. (c) Gaussian – Guassian. (Courtesy of George Wolberg, Columbia University)




When explaining theory of filtering, we have used three-step pipeline:

1. Construct continuous function f(x) (e.g., rectangles representing lines, polygons)

• these are our primitives

2. Low-pass filter function to generate – goal: Multiply dual of f(x) with box in the

frequency domain– equivalently in spatial domain: Convolve

with sinc (i.e., evaluate integral at an infinite number of points)

3. Sample prefiltered function continuous function to generate

As programmers, we save work:1. Construct a continuous function f(x)2. Simultaneously sample and filter to

generate– equivalent to generating new function,

then evaluating it only at sample points (i.e., at pixels)

)(ˆ xf

The Filtering Pipeline

)(ˆ kf

)(ˆ kf

i n t r o d u c t i o n t o c o m p u t e r g r a p h i c s antialiasing & image processing...

Documents