I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Antialiasing &
Image Processing
Andries van Dam October 12, 2004 Antialiasing & Image Proc 1/86
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 2/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 3/86
• Is an entire subject in and of itself• Has its own publications and conferences
– Computer Graphics, Vision and Image Processing (CGVIP), IEEE Transactions on Image Processing, PAMI, Image and Vision Computing
• Was once closer to signal theory and audio processing than to graphics
• Shares hardware with other aspects of graphics– floating point processors (FPUs)– digital signal processors (DSPs)– 24-bit or 32-bit framebuffer– scanners– digital cameras– color printers– video frame grabbing
• Shares software with graphics. Many common applications include an image processing component:
– Microsoft WordTM
– Adobe InDesignTM
– Macromedia Flash MXTM
1. This unit was originally created by former head TA Lars Bishop and subsequently modified by John Hughes, former head TAs Sam Trychin and Sascha Becker, Dan Gould, and by Andy van Dam.
2. For more information, refer to Chapter 17 of the textbook, “Image Manipulation”, and Section 14.10, “Aliasing and Antialiasing.”
Antialiasing & Image Processing
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 4/86
For our purposes, an image is:• A 2D domain
• With samples at regular points (almost always a rectilinear grid)
• Whose values represent gray levels, colors, or opacities (or tissue density in an MRI scan)
Common image types include:• 1 sample per point (B&W or Grayscale)
• 3 samples per point (Red, Green, and Blue)
• 4 samples per point (Red, Green, Blue, and “Alpha”, a.k.a. Opacity)
What is an Image?
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 5/86
Where is the image stored?• In order to look at image processing, we need to take
a short look at a computer’s graphics hardware• Graphics hardware is implemented as a separate
subsystem• The graphics subsystem communicates with the CPU
and main memory across a special high-speed bus in most modern system architectures (PCI Express)
Hardware
8.0 GB/s for the graphics card!!
Only 133.0 MB/s for PCI board
MCH: Memory Controller Hub
RAM
PCI: Peripheral Component Interconnect
ICH: I/O Controller Hub
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 6/86
A framebuffer and more• Has local memory for the image
• Takes commands from the host CPU
• Can have on-board processing for various 2-D and 3-D operations
• Video hardware converts image in framebuffer to analog signal (DAC or RAM-DAC)
• Contains Look Up Tables (LUTs) explained in the intro lecture
Graphics Subsystem
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 7/86
A simple example
• Divided into bit-planes
• Each plane represents 1 bit of the color value at that pixel
• Each plane is size of display (1280x1024 for SunLab)
• For a basic framebuffer representing each of R, G, B with 8 bits, the framebuffer is 24 bits deep
The Framebuffer (1/2)
Blue
Green
Red
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 8/86
A more advanced framebuffer• Double buffering doubles the planes in our framebuffer• For a window system, also want a backing store
3D requirements are even more demanding:• Z-buffer to hold depth values
– we will discuss this in the Visible Surface section• Stereo doubles our requirements yet again
– one set of buffers for each eye• Specialized buffers for various effects
– A-buffer (Accumulation buffer for transparency, etc)– stencil buffer (for masking regions to draw in)– overlay planes (for removable drawing on top of 3D
scene)• Machines like the SGI UltimateVisionTM have
framebuffers which are thousands of bits deep!– can add up to gigabytes on high-end graphics
systems
The Framebuffer (2/2)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 9/86
An expensive proposition• Tons of memory, all of which is
updated several times per second
• Typical (cheap) memory cannot do this
• Graphics hardware makers build very sophisticated memory systems
• Special types of memory– dual-ported memory
– logic-enhanced RAM
• Parallel, Interleaved memory system
• The problem is getting worse – memory speed is improving MUCH more slowly than the rest of the hardware
Framebuffer memory
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 10/86
Each of these planes is a “channel”. For example, the red channel of a 3 sample per pixel image would itself be a 1 sample per pixel image, consisting of all of the red values in the original image.
Channels
Blue Channel of Image,1 sample per pixel
Green Channel of Image,1 sample per pixel
Red Channel of Image,1 sample per pixel
3 samples per pixel
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 11/86
Combining various sorts of data• If your image contains z-values, you can
use the gradient of the z-values to add color: sudden changes in z-depth get colored black, for example. This highlights the silhouette edges of things
• There’s a whole body of material on tricks like these
Multi-channel operations
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 12/86
Adding “coverage” information to pixels• See Chapter 17 for details
• In addition to R, G, B channels of an image, we can add a fourth channel, called α.
• Alpha varies between 0 and 1; a value of 1 represents a “fully covered” pixel, one you cannot see through
• A value of 0 is a completely transparent pixel
• Useful for blending images: the one with the higher alpha value “shows through” more
The Alpha Channel
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 13/86
The general stages of image processing can be flowcharted in an idealized pipeline-not all stages need to be present and the middle two or three stages may be interlaced:
Stages of image processing
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 14/86
Encompasses a wide variety of techniques in two basic categories:
• Synthesis: images created in a computer– rendered from geometry (e.g., RenderManTM, Alias|
Wavefront’s MayaTM, autodesk 3ds maxTM)– painted (e.g., MacPaintTM, Fractal Design PainterTM)
• Capture: images from the “real world”– must be “captured” or digitized; turned into digital
information instead of analog signal or printed media– method of digitization is partially determined by
image’s original form– common methods include:
• digital camera• satellite data transmission• drum scanner• slide scanner• flatbed photo scanner • video frame grabber/converter
• Combination: texture-mapped polygons– the texture is essentially an image, and the polygons
part of a model.
Stage 1 – Image AcquisitionHow do we get images?
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 15/86
• Source image(s) is adjusted to fit a given tone, size, shape, etc., to match a desired quality or to match other images
• Can make a set of dissimilar images appear similar (if they are to be composited later), or make similar parts of an image appear dissimilar (such as contrast enhancement)
• Preprocessing techniques include:– adjusting color or gray scale curve– cropping– masking (cutting out part of an image to be used
in a composition, or to leave a hole in the original image)
– scaling up (super sampling)/ down (sub sampling)
– blurring and sharpening– edge/enhancement– filtering and antialiasing
• Notes:– techniques like blurring, sharpening, and edge
detection can also be a postprocessing techniques– some preprocessing algorithms are not followed
by mapping, others that involve resampling the image may be interlaced with mapping: filtering is done this way
Stage 2 – Preprocessing
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 16/86
• Mapping is catch-all stage where several images are combined, or geometric transformations are applied
• Transformations include:– rotation
– scale
– stretch– feature based image warp (aka morphing)
• Compositing:– opaque/transparent paste
– alpha-channel composition
Stage 3 – Mapping
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 17/86
• Creates global effects across an entire image or selected area(s)
• Art effects– posterizing
– faked “aging” of an image
– faked “out-of-focus”
– “impressionist” pixel remapping
– texturizing
• Technical effects– color remapping for contrast enhancement
– color to B&W conversion
– color separation for printing (convert RGB to CMYK)
– scan retouching and color/contrast balancing
Stage 4 – Postprocessing
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 18/86
• Choice of display/archive method may affect earlier processing stages– color printing accentuates certain colors
more than others
– colors on the monitor have different gamuts and HSV values than the colors printed out: need a mapping
• Techniques– monitors
– color printers
– film/videotapes
– disk files
– texture maps for 3D renderer
Stage 5 – Techniques for Archive And Display
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 19/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 20/86
Give an idea of what is done with image processing
• Image enhancement– scientific filtering
– forensic science
• Multipart composition
• Computer vision
Examples
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 21/86
Some filtering techniques are designed to make features in an image more apparent
• Done by using a filter that accentuates changes above certain threshold
• Causes specific features of an image to stand out• Can even calculate a new image based on some
function that takes an image to another image• e.g., define an image that is magnitude of change
of original image at each point. Thus, higher-valued pixels in new image are places where original image was changing rapidly
An Application of the Edge-Detection Filtering Technique
Just an illustration, not an MRI: see the next slide
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 22/86
Take slice from MRI scan of canine heart, and find boundaries between types of tissue
• Image with gray levels representing tissue density• Using filter from previous slide, compute new image.
Again, new image brighter where MRI image gray values changing faster
• Different densities of different types of matter will show up with bright boundaries in between.
MRI Image Enhancement
Original MRI Image of a Dog Heart
Edge Detection Image
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 23/86
Image enhancement has been used by forensic scientists for years to pull information from seemingly hopeless images.
• We have a security camera video of the back of a car that was used in a robbery
• The image is too dark and noisy for the police to pull a license number
Image processing like this in the media a lot in the last few years
• These techniques have been used to find small features in satellite images
• Image processing for forensic science is even spotlighted in popular entertainment, such as the book (and movie) Rising Sun, by Michael Crichton and the tv show CSI: Crime Scene Investigation
Forensic Science Image Enhancement
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 24/86
Normally, image enhancement uses many filtering steps, and often no mapping at all.
Image Enhancement Steps
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 25/86
Step One: Get the frame from the videotape digitized with a frame-grabber
Image Enhancement (1/4)
• Frame grabbing is no longer expensive – video capture boards for PC’s are now under $60
• Though humans can often discern an image of poor quality, filtering can make it easier for a pattern-recognition algorithm to decipher embedded symbols: in this case, alphanumeric characters
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 26/86
Step two: Crop out stuff that appears to be uninteresting (outside plate edges)
• This step can speed process by doing image processing steps on fewer pixels
• Can’t always be done –may not be able to tell which sections are interesting without some processing
Image Enhancement (2/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 27/86
Step Three: Use edge-sharpening filter to add contrast to plate number
• This step enhances edges by raising discontinuities at brightness gaps in image
Image Enhancement (3/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 28/86
Step Four: Remap colors to enhance contrast between numbers and plate itself
• This stage has truly brought out image - mapped low-valued pixels to bright white, and almost all others to black.
• Now, can make a printout for records, or just copy plate number down. YNN-707!
• Note that final colors do not even resemble real colors of license plate. Enhancement techniques have seriously distorted the colors!
Image Enhancement (4/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 29/86
Image composition is popular in art world, as well as in tabloid news world
• Takes parts of several images and creates single image. Hard part is making all images fit together naturally
• Artists can use it to create amazing collages and multi-layered effects
• Tabloid newspaper artists can use it to create “News Photos” of things that never happened
• Some famous examples previously mentioned:– cover of TV Guide with Oprah
– cover of Time and Newsweek with O.J. Simpson
– clips from Forrest Gump; ads w/ old stars
– university brochure altered to show ethnic diversity
Multipart Composition
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 30/86
Lars Bishop, former CS123 Head TA and primary author of these lectures, created a news photo of himself “meeting” with former Russian President Boris Yeltsin.
• Needless to say, Lars Bishop never met Mr. Yeltsin
• So, had to get images, cut out the parts he wanted, touch them up, paste them together, and retouch the end result
Our Example
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 31/86
Step 1 – Getting the images• The image of Boris was pulled from the internet
• The image of Lars was staged in the graphics lab, filmed with a camcorder, and digitized live
Image Composition (1/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 32/86
For Boris, Lars did the following:• Found image of Boris and Gorbachev
on an FTP site, transferred it to Brown
For image of Lars, he did the following:
• Set up a video camera to use same type of camera angle as the shot of Boris
• Wired camera to a frame digitizer
• Lars posed in front of camera
• Grabbed image
• Could have simplified process by putting a white sheet behind himself, but could not be found in time
Step 1: The Details
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 33/86
Stage 2 – We must cut the pictures we want out of the original images
• Boris is cut from the internet image
• Lars is cut from the video image
Image Composition (2/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 34/86
For Boris and Lars• Lars brought images into Photoshop on his Mac,
and painted a region around important parts of images (outline of people) in white
• Continued touching up this outline until no background at edge of people
• Used a lasso tool that could grow until it hit the white background, thus selecting subject
• Cut subjects out of their images
Step 2: The Details
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 35/86
Step 3 – Filter the images to make them appear similar, and paste them together
• Boris is blurred and brightened, to get rid of the halftoning lines (must have been a magazine photo). Also scaled down to be same size as Lars
• Lars is blurred and noise is added to match quality of image to that of Boris
Image Composition (3/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 36/86
For Boris:
• Blurred image to get rid of the halftoning lines
• Brightened image to fit colors of other image
• Scaled him down to be same size as other subject
For Lars:
• Blurred image to fit blurring of Boris
• Added noise to lower quality of otherwise clean video grab to quality of scanned image of Boris
Then, they are composited together
• Lars’ hand is retouched to merge well with Boris’ document.
Step 3: The Details
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 37/86
Final Image (with retouching at edges)
BISHOP AND YELTSIN TALK PEACE
BISHOP: “I couldn’t understand a single word he said!”
Image Composition (4/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 38/86
For this step, Lars did several little touches to add to quality of fake
• Created a simple, two-color background and added noise to it to fit rest of the image
• Placed cutout of the two subjects on top of background
• This left a white halo about 3 pixels wide around the subjects, so used a “Rubber Stamp” tool to stamp background noise patterns over halo, making seams appear less obvious
• Additional loss of contrast and spatial resolution due to printing process
Step 4: The Details
Step 5: Print the Image
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 39/86
Image enhancement also done to enhance images for computer vision
• Computer must do all processing, without human intervention
• Processing techniques must be fast. If slow, will add to camera-to-reaction lag (latency) in system
• Common preprocessing techniques for computer vision include edge enhancement, region detection, contrast enhancement, etc..
• Computer must do more than enhance an image before reacting to it. Must pull specific information from image, such as position and orientation of edges
Computer Vision (1/2)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 40/86
Image enhancement can also be done to enhance images for computer vision
• Image processing makes information easier to find• Pattern detection and pattern recognition are
separate fields in their own right– Pattern detection: looking for features and
describing the image’s content at a higher level– Pattern recognition: classifying collections of
features and matching them against library of stored patterns. (e.g., alphanumeric characters, types of abnormal cells, or human features in the case of biometrics)
– Pattern (feature) detection is one important component of pattern recognition.
• For more on computer vision, check out CS143 or CG120!
Computer Vision (2/2)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 41/86
MRI scans and the like create 3D blocks of data. As time goes by, image processing may move furtherand further into 3D image processing.• Techniques we used in 2D often have 3D analogs• Display becomes far more difficult: voxels replace pixels• Data storage issues become problematic:
– 1024x1024 4 channel image = 4 megs– 1024x1024x1024 4 channel 3D image = 4 gigs!
• Computational issues can also become problematic – N2 becomes N3
• AutoStereoscopic displays hot research topic – most work with lenticular optics. DTI’s Virtual WindowTM (shown in image) is an example
• LightSpace Technology’s DepthCubeTM produces a real 3D display (no stereo glasses). Display uses 20 LCD slices to display image
• Holographic displays are being worked on also (e.g., MIT’s Media Lab)
• Computer vision can be used as part of a passive UI, as an alternative to intrusive (tethered) gadgetry such as 6DoF “space mice”, wands, data gloves
• Autostereoscopic monitors for sale:http://www.sharp3d.com/
The Future – 3D?
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 42/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 43/86
Now that you understand the big picture, let’s start focusing on specific topics
• Stages 2, 3 and 4 contain numerous important topics and will now be discussed in more depth
Start with antialiasing• What is aliasing?
• What is anti-aliasing?
• Why is it necessary?
• How is it performed?
Getting down to details
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 44/86
• Representing a line with discrete pixel values is sampling a continuous function
• Jaggies are a manifestation of sampling error and loss of information
• The term aliasing will be defined momentarily
• Doubling resolution in x and y only lessens the problem
• Costs 4 times memory, memory bandwidth and scan conversion time!
Jaggies-Manifestation of Aliasing
(a) Standard midpoint line on a bilevel display.
(b) Same line on a display that has twice the linear resolution
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 45/86
• Sample unit rectangle rather than mathematical (infinitesimally thin) line, and switch to representing pixels as squares centered at x-y intersections rather than circles (they are neither!)
• Midpoint algorithm: pick single pixel closest to center line of rectangle (0 < slope < 1). This is a form of point sampling: Sample the mathematical center line at each of the integer X values. As with our original midpoint line algorithm, we pick a single pixel to represent the line’s intensity.
• Only single value of intensity used regardless of amount of overlap between each pixel and rectangle. Therefore all but one pixel in each column is “off,” and all have same (max) brightness.
Line as 1-Pixel Width Rectangle
Line (as unit-width rectangle) from point (1,3) to point (7,8)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 46/86
• Set each pixel’s intensity value proportional to its area of overlap covered by primitive; thus sample sub-areas of rectangle (that represents line) covered by rectangle representing pixel.
• Note more than one pixel/column for lines of
0<slope<1
• This is a form of unweighted area sampling– only pixels covered by the primitive can contribute
– only total amount of area of overlap matters, regardless of the distance of any individual sub-area of overlap from pixel’s center
• Area sampling smoothes sharp transitions
First Antialiasing Attempt
Figure 3.36: Intensity of a pixel is proportional to its area covered by the line
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
• Weight function W(x, y) gives weight for incremental area dA centered at (x, y); constant here, hence the name box. Box is single pixel wide here, but could vary width (as we’ll see).
• For each pixel intersecting line, intensity contributed by each sub-area dA is (W(x, y) • dA)
• Then total intensity (a fraction between 0 and 1) integrated over area of overlap is:
• Integral is volume over area of overlap (in above figure, a rectangular wedge). Check out applet comparing point and area sampling:
http://www.cs.brown.edu/exploratories/freeSoftware/catalogs/signal_processing.html
• Note that, in two dimensions, box functions are often referred to as pulse functions.
Andries van Dam October 12, 2004 Antialiasing & Image Proc 47/86
“Box Filter” Represents Unweighted Area Sampling
A
dAyxW ),(
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
• Area sampling, but weight overlap between filter and primitive so that sub-areas dA closer to center of pixel count more. Revolve -45 degree slope to get 3D cone shape: linear falloff with distance, and circular symmetry
• Circular cone =>change back to circular pixels; we set cone’s diameter (its support) to two pixel units. To calculate given pixel’s contribution, calculate volume of wedge of filter whose base is area of overlap between support and primitive. dA has greater weight if closer to center of pixel, Normalize so that volume = 1; subvolume subtended by intersection between line and filter is a conical wedge.
Andries van Dam October 12, 2004 Antialiasing & Image Proc 48/86
Cone Filter for Weighted Area Sampling (1/2)
Cone filter for circular pixel with diameter of two grid units.
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 49/86
2-unit circular support of filter
Pixel center (+)
Area of overlap between support and
primitiveDifferential area dA2
Primitive Differential area dA1
Weighted Area Sampling (2/2)
W(x, y) is the weight which is multiplied with dA at (x, y); normalize W to make volume under cone = 1
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
• 2 unit support implies up to 3 pixels/column
• Gupta-Sproull algorithm provides fast anti-aliased lines via table look-up (see book)
• Weighted area sampling does a better job of smoothing than unweighted area sampling.
Andries van Dam October 12, 2004 Antialiasing & Image Proc 50/86
Cone Filter for Weighted AreaSampling (2/2)
Fig. 3.60 One-unit-thick line intersects 3-pixel supports.
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 51/86
• This simplistic scan conversion algorithm only asks if a mathematical point is inside the primitive or not
– bad for sub-pixel detail which is very common in high-quality rendering where there may be many more micro-polygons than pixels!
Another Look at Problems ofPoint Sampling
Fig. 14.10 Point-sampling problems. Samples are shown as black dots. Object A and C are sampled, but corresponding objects B and D are not.
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 52/86
• The box filter sets intensity proportional to area of overlap
• Get winking of adjacent pixels which eye can detect
– eye is very forgiving of TV-type noisy images with small changes between frames
– eye is very sensitive to pixels winking in images of more uniform appearance
Another look at Unweighted AreaSampling (1/3)
Fig. 14.11 Unweighted area sampling. (a) All sub-areas in the pixel are weighted equally. (b) Changes in computed intensities as an object moves between pixels.
(b)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 53/86
• Pyramid filter approximates circular cone to emphasize area of overlap close to center of pixel
• Support is equal to pixel area
Another look at Weighted Area Sampling (2/3)
Fig. 14.12 Weighted area sampling. (a) sub-areas in the pixel are weighted differently as a function of distance to the center of the pixel. (b) Changes in computed intensities as an object moves between pixels.
(b)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 54/86
• Support of symmetric cone filter greater than pixel area for greater smoothness
Another Look at Weighted Area Sampling (3/3)
Fig. 14.13 Weighted area sampling with overlap. (a) Typical weighting function. (b) Changes in computed intensities as an object moves between pixels.
(b)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 55/86
for each pixel p
place filter centered over p;
for each pixel q under filter
weight = filter value over q;
intensity_p += weight * intensity_q;
Pseudocode
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 56/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 57/86
• Scan converting a geometric scene or scanning in a natural image is digitizing a sequence of continuous intensity function, one per scanline
Fig. 14.8 Image (a) Graphical primitives. (b) Mandrill. © Intensity plot of scan line αin (a). (d) Intensity plot of scan line α in (b). (Part d is courtesy of George Wolberg,Columbia University.)
• Note step functions abound at the edges of geometric primitives – these represent infinite spatial frequencies that cause aliasing because we can’t sample them correctly (What is the intensity value at an edge?)
Sampling of Images
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 58/86
• Scanning in a photograph and displaying it:– scan in a sequence of intensity functions, 1 per scan line– digitize each by sampling at discrete points, losing both
spatial and intensity information– reconstruct the signal, e.g., with continuous analog voltage
wave form to CRT (via DAC)
The Sampling/Reconstruction/Display Pipeline
Fig. 14.9 The original signal is sampled, and the samples are used to reconstruct the signal. (Sampled 2D image is an approximation, since point samples have no area.) (Courtesy of George Wolberg, Columbia University.)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 59/86
Waveform Synthesis
Fig. 14.14 A signal in the spatial domain is the sum of phase-shifted sines. Each component is shown with its effect on the signal shown at its right. Approximation of Fig. 14.8(d) (Courtesy of George Wolberg, Columbia University.)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 60/86
• Intensity of sound is pressure wave amplitude
• Frequency of pressure wave determines its “pitch”
• Lowest frequency is “fundamental frequency”
• Integer multiples of the fundamental frequency are called “harmonics”
• Power of two multiples of the fundamental frequency are called “octaves”
For more, an applet to interact with these concepts:http://www.cs.brown.edu/~dlg/gfxnotes/signal/wavesynth/
Digression: Music
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 61/86
• Many problems can be made easier by transforming them to another problem (must also be a way to go back!)
• May be easier to characterize a signal by its frequency spectrum –signal and frequency domains are duals and represent identical information. May be easier to filter in frequency domain than in signal domain, as we shall see.
• Take a familiar problem: multiplication of numbers. Can take logarithm of number, perform operations on log(number), then move back using antilogarithm
• Dual of multiplication is addition in logarithmic “space”:
If ab = c then
log (a) + log (b) = log (c)
• This “invertible transformation” makes slide-rules such effective tools for multiplication: manipulating sliders corresponds to manipulating numbers via their logs.
Digression: Analogous Operation
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Ant aliasing & Image Proc 62/86
• Sine wave is characterized by amplitude and frequency• Frequency of a sine wave is number of cycles per
second for audio, or number of cycles per unit length (e.g., inter-pixel distance) for images
• Can characterize any waveform by enumerating amplitude and frequency of all its component sine waves (Fourier transform – see chapter 14)
• This can be plotted as a “frequency spectrum”, a.k.a. power spectrum, (we ignore negative values, but they are needed for mathematical correctness)
Frequency Spectrum of a Signal
For fun with Fourier transforms see:http://www.ysbl.york.ac.uk/~cowtan/fourier/magic.html
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 63/86
When we sample, we must sample at a rate that is higher than 2 times the highest frequency in thesignal (the Nyquist limit)• Here is an approximate analog sine wave
• Here is the sine wave sampled at an acceptable rate (4 times the highest frequency)
• Here is the reconstructed wave based on these samples
Sampling: The Nyquist Limit
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 64/86
Aliasing is a problem that can occur whenever we sample a signal at less than twice maximum frequency
• Here is our analog sine wave again
• Here is the sine wave sampled at too low a rate
• Here is the reconstructed wave based on these samples
• Uh, oh… The reconstruction isn’t even close!
Aliasing: Know Thine Enemy
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 65/86
Sampling right at the Nyquist limit can be problematic• Here is our perfect analog sine wave again
• Here is the sine wave sampled at the Nyquist limit. This time it works fine
• Here is the same sine wave sampled at the Nyquist limit, with the sample points shifted. Now we get no signal
For an applet dedicated to illustrating the Nyquist Limit:
http://www.cs.brown.edu/~dlg/gfxnotes/signal/nyquist/
Sampling At the Nyquist Limit
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 66/86
• Nyquist Applet
http://www.cs.brown.edu/exploratories/freeSoftware/catalogs/signal_processing.html
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 67/86
• These low frequencies are “aliases” of high frequencies
• The low sampling rate data could not adequately represent the high frequency components, so it represented them incorrectly, as low frequencies
• So, we just sample above the Nyquist limit, right?
• Regrettably, we can’t always do that
• What about this?
The Enemy is RecognizedAliasing is shown in bottom diagrams on previous slides – Signals that are sampled at too low a rate can have their high-frequencies appearing in reconstructed signal as low frequencies
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 68/86
Square wave has infinite frequencies of sine waves in it, right at jumps from low to high and high to low (see Fig 14.15). How can we sample correctly?
• We can’t. Pure and simple. •2=, and we can’t sample at an infinite rate (unfortunately infinite frequencies are the rule in synthetic computer graphics with its discrete transitions between adjacent pixels)
• So, do inverse operation, which is best we can do. Instead of increasing our sampling frequency to meet signal, we:
– pre-filter out high frequencies we can’t represent – now signal we sample is guaranteed to consist only of
frequencies we can represent and reconstruct with reasonable accuracy with our samples
• This isn’t same signal that came in, but it’s closer than version that would cause aliasing
• The more high frequencies we pre-filter out, the lower the sampling frequency needed but the less the filtered signal resembles original
• Note: pre-filtering is often just abbreviated as filtering, but the prefix “pre” helps remind us that post-filtering (i.e., another stage of filtering after image computation or transformation) is also practiced.
Infinite Frequencies
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 69/86
Image Scaling Aliasing, or “Why do we have to prefilter?”
This doesn’t lookright at all!
no stripesblacker average
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 70/86
Scale Aliasing II, or“Close, but no cigar?”
Better,butnotperfect
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 71/86
The scaled image with pre-filtering looked a little better, but we still couldn’t see any stripes
• The filter made scaled image have same relative brightness, but no stripes
• Filter removed “high frequencies” from image –discontinuities that were stripes
• Given number of points to represent image once scaled, not enough points to represent high frequencies
• We’ll never be able to represent frequencies higher than ½ our sampling rate. We can’t do better that this blurred approximation (remember Nyquist limit)
• Now lets see how we do pre-filtering in practice:– first show effect of low-pass filtering in both signal and
frequency domains to eliminate unwanted high frequencies
– then discuss mathematics of filtering, called convolution
Image Scaling Aliasing III, or“Why is it still wrong?”
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 72/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 73/86
Low-Pass Filtering(Spatial Domain)
Fig. 14.20 The sampling pipeline with filtering. (Courtesy of George Wolberg, Columbia University.)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
• Box function in frequency domain corresponds to “convolution” with sinc function in spatial domain (defined next)
Andries van Dam October 12, 2004 Antialiasing & Image Proc 74/86
Low-Pass Filtering(Frequency Domain)
Fig. 14.13 Multiplying with the box function in the frequency domain. (a) Original spectrum. (b) Low-pass filter. (c) Spectrum with filter. (d) Filtered spectrum. (Courtesy of George Wolberg, Columbia University.)
1
1
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 75/86
• Convolving signal f(x) with filter function g(x)
– value of h(x) at each point x is integral of product of f(x) and g΄(x) where g΄(x) is g(x) flipped about vertical axis and shifted so origin is at x: replace x by τ and g΄(x)=g(x-τ)
– τ is variable of integration that takes x as local origin – diagram shows how g΄(x) is constructed using a box filter– if filter g΄(x) has finite support, it does weighted average of all
pixels within support, centered at x– The mathematical ‘flip’ doesn’t matter for symmetric functions
like ours (see dirac-delta applet)
– note in filtered signal h(x) how discontinuities of f(x) are smoothed out, and the base is broadened
Convolution – Math!Or Just Play with the Applets
dxgfxgxfxh )()()(*)()(
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 76/86
• Convolution applets
http://www.cs.brown.edu/exploratories/freeSoftware/catalogs/signal_processing.html
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 77/86
Duals
Spatial Frequency
Multiplication Convolution
Convolution Multiplication
Box Sinc
BoxSinc
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 78/86
• Images & Hardware
• Example Applications
• Jaggies & Aliasing
• Sampling & Duals
• Convolution
• Filtering
• Scaling
• Reconstruction
• Scaling, continued
• Implementation
Outline
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 79/86
Low-Pass Filtering: Convolving with sinc
Fig. 14.23 Low-pass filtering in spatial domain. (a) Original signal. (b) Sinc filter. (c) Signal with filter, with value of filtered signal shown as black dot at filter’s origin. (d) Filtered signal. (Courtesy of George Wolberg, Columbia University.)
• Note that in theory sinc has infinite extent, however small the contributions, and negative lobes, but weights contributions at the center most heavily. Practically, it is decently approximated with gaussian (normal) distribution, or even triangle, with finite extents and weights greater than or equal to zero.
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 80/86
• Select points at which to sample, e.g., pixel centers• As per 14.23, slide filter over each successive sample point
and compute convolution integral at that point (the area under product curve). This is weighted average of current pixel and its nearby neighbors (most useful graphics filters are symmetric about their origin and fall off rapidly from their center).
• Weighted average is value of sample/pixel for filtered image• Did much the same when we calculated total response of a
color receptor to incoming intensity distribution I(λ) by computing area under I(λ)•f(λ) product curve, where f(λ) is response function (i.e., filter function) of receptor.
• These illustrations are 2D, but for pixels one should think of sinc as three-dimensional surface of revolution and primitive as having area. [see slide 44] Of course, we’re not actually going to compute the integral, but will merely look up discrete values of filter function and do discrete multiplication and summing to approximate it.
• Use term “filter” strictly in signal-processing sense, while Unix™ and Photoshop ™ have their own notions of filters. Some of Photoshop’s filters are signal-processing filters, while some are arbitrary transformations of image.
For more information, check out convolution applet at:http://www.cs.brown.edu/research/graphics/research/exploratory/
research/applets/appletDescriptions/specialFunctionConvolution/home. html
What Does Filtering/Convolution Do?
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 81/86
• Answer: Convolution and filtering compute area under the h(x) = f(τ) • g’(x) product curve and are proportional to weighted area sampling
• Let’s sample 9 times. Then the discrete weighted area sample=
• Each box is hi tall and wide
• Total area
• Area under product curve =
• 1 ≤ length of interval ≤ 2
Relation btwn. Weighted Area sampling and Area under Curve(this data is mostly here for the mathematically inclined)
4
4 9i
ih
8
2a
WASah
ah
aa
h iii 2
92
82
8
2
interval oflength
)(')( 0
dxxgfWAS
)(f )(')( xgf
filter triangle)(' xg
)(')(sampling
xgf
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 82/86
• Filtering is nothing more than weighted averaging around a pixel to smooth image
• The dual of sinc function in spatial domain is box (aka pulse) function in frequency domain, and dual of box function in frequency domain is sinc in spatial domain
• The dual of box function in spatial domain is sinc function in frequency domain, and vice versa
• Multiplying frequency distribution with box function in frequency domain provides exact low-pass filtering, i.e., filtering out all high frequencies past a specified one. Hence, optimal filter is sinc in spatial domain (see figure 14.24)
• Multiplication of F(x) and G(x) in frequency domain corresponds to convolution of their duals, f(x) and g(x), in spatial domain, and vice versa
Filtering Summary (1/4)
)(*)()()( xgxfxGxF )()()(*)( xgxfxGxF
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 83/86
Fig. 14.24 (a) Sinc in spatial domain corresponds to pulse in frequency domain. (b) Truncated sinc in spatial domain corresponds to ringing pulse in frequency domain. (Courtesy of George Wolberg, Columbia University)
Filtering Summary (2/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 84/86
• To get low-pass filtering (i.e., filter out high frequencies), we often use convolution with triangle function in spatial domain to approximate ideal sinc
• Properties of triangle function: – easy to compute, unlike sinc, which has an infinite
support (or even a Gaussian approximation to sinc)
– its dual, sinc2, is acceptable approximation to a box function although it has infinite extent (see fig. 14.25b on next slide)
– these cause inaccurate representation of all frequencies and therefore some degree of corruption/aliasing must occur. This is not as bad as using box as a filter with sinc as its dual in the frequency domain. In other words, weighted area sampling of any kind, providing it is roughly cone-shaped, is better than unweighted area sampling with a box filter, which is better in turn than point sampling
Filtering Summary (3/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 85/86
Fig. 14.25 Filters in spatial and frequency domains. (a) Pulse – sinc. (b) Triangle – sinc2. (c) Gaussian – Guassian. (Courtesy of George Wolberg, Columbia University)
Filtering Summary (4/4)
I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S
Andries van Dam October 12, 2004 Antialiasing & Image Proc 86/86
When explaining theory of filtering, we have used three-step pipeline:
1. Construct continuous function f(x) (e.g., rectangles representing lines, polygons)
• these are our primitives
2. Low-pass filter function to generate – goal: Multiply dual of f(x) with box in the
frequency domain– equivalently in spatial domain: Convolve
with sinc (i.e., evaluate integral at an infinite number of points)
3. Sample prefiltered function continuous function to generate
As programmers, we save work:1. Construct a continuous function f(x)2. Simultaneously sample and filter to
generate– equivalent to generating new function,
then evaluating it only at sample points (i.e., at pixels)
)(ˆ xf
The Filtering Pipeline
)(ˆ kf
)(ˆ kf