www.imgtec.com powervr is a division of imagination technologies ltd. texture compression using...

www.imgtec.com www.powervr.com PowerVR is a division of Imagination Technologies Ltd.

Texture Compression using Low-Frequency Signal Modulation

(PVR-TC)

Simon Fenney

Graphics Hardware 2003

© Imagination Technologies2PowerVR is a division of Imagination Technologies Ltd.

Presentation Overview

Why do we want Texture Compression?

Previous Research

Aims, Observations, & Overview

Decompression (4bpp) – PVR-TC

Compression (4bpp)

2bpp mode

Results & Future Research

Summary


Texture Compression – Section 1

Why do we want texture compression?


Why do we want Texture Compression?

Bandwidth! Bandwidth!! Bandwidth!!!

Memory can become a bottleneck A low-cost device will usually have a narrow bus Workstations & consumer devices: Much wider bus but… …can be flooded by complex filtering and multiple texture layers


Why? (pt. 2)

Smaller Texture Footprint

Portable devices (e.g. PDAs, 3D-Enabled phones) have relatively little memory

Reduces texture swapping – more data can stay resident

Benefit when transferring texture data


Texture Compression Issues

Beers et al. list four factors to consider when evaluating a texture compression scheme:

Speed of Decode: the faster the better!

Random Access - usually implies a fixed-rate encoding. (no JPEG/Huffman)

Compression Rate vs. Quality: Can usually tolerate loss of fidelity

Encoding time: Can be ‘asymmetric’



Previous Research


Previous Texture Compression Schemes:

Can be approximately divided into 3 main families:

Code-book based e.g. Colour Palettes/VQ ‘Global’ compression scheme

Block Truncation Coding (BTC) based Delp & Mitchell Independent blocks of pixels

Transform Coding Schemes e.g. DCT or Wavelet based Relate position to colour values


CodeBook-based systems

Colour Palettes – simple VQ Replace each pixel with index into codebook Widely employed from flight simulators through to game consoles

Pros and cons: Well researched compression schemes – e.g. Heckbert, Wu Requires indirection Modest compression rate (8bpp) or low quality (4bpp)


CodeBook-based systems - VQ

VQ: e.g. Beers, Agrawala, & Chaddha Replace 2x2 or 4x4 pixels with index

Pros and Cons High compression ratio (e.g. ~2bpp) with reasonable quality Still requires indirection Larger codebook


BTC-based schemes -Introduction

Block Truncation Coding (BTC) - Delp & Mitchell Not strictly intended for Texture compression… Monochrome data @ 2bpp Two 8-bit intensities per 4x4 pixel block Single bit per pixel choice of intensity

Pros and Cons No indirection Each block compressed/decompressed independently… …Each block compressed/decompressed independently


BTC-based schemes - CCC

Color Cell Compression: Campbell et al. Extends BTC to support colour. 8-bit intensities replaced by palette indices. Meant for image transmission but could be used for texturing (Knittel et al)

Pros and Cons 2bpp colour compression Requires indirection Discontinuities at block boundaries Some banding is evident


BTC-based schemes – S3TC

S3TC: Iourcha et al. Eliminates palette indirection by storing 2 colours per 4x4 block Two additional inferred colours Four colour choices per pixel

Pros & Cons 4bpp (opaque), 8bpp (with alpha) Typically good quality results – less banding than CCC Widely supported (DirectX, OpenGL, Nintendo GameCube) Artefacts at block boundaries Sometimes blocks don’t reduce to line in colour space


Transform Coding schemes

Less popular

Hierarchical Texture compression – Pereberin Haar Wavelets Stores 3 levels of MIP map simultaneously ‘Unusual’ compression rate

Talisman TREC – Torborg & Kajiya JPEG-like Expensive??



Aims, Observations, & Overview


Aims of Research

2~4bpp, good quality for (A)RGB

Simple decode

Simultaneous decompression of 2x2 pixels for filtering

Try to avoid discontinuities at block boundaries


Observations

Block schemes – no internal correlation of position & colour

Bicubic and Bilinear (low frequency) representations are often ‘close’ approximations of original position/colour correlation

Delta + Low Frequency = original image

‘Delta’ image often locally monochromatic

Idea: Represent low frequency and ‘filtered delta’ images using bicubics Modulate filtered delta using per-pixel scalar values – simple decode, better

with graphics


‘Initial’ Experiment

Compressed Format: Two, independent, 1/4 or 1/8

resolution images, A & B. 2-bpp ‘modulation’ image A&B and modulation data

separate 4-bpp or 2½-bpp compression

rate

Decompression Bicubic upscale of the low-

resolution images Blended according to

‘modulation’ value

Image A

Image B

Upscale

Upscale

Virtual Image Au

Virtual Image Bu

Lin

ear

Ble

nd

Modulation M

Result


Initial Experiment – 2

Quality of results was very promising but…

…system ‘too expensive’ to implement in a hand-held device (e.g. PowerVR MBX) 2.5bpp mode used 128-bit modulation blocks Bi-cubic upscale required 16 source colours + interpolation

Cost reduced by using Bilinear interpolation and 64 bit memory blocks



Practical Decompressor


Practical Decompressor - 4bpp

A & B images, 15~16bits / colour

A&B colours fully opaque (554, 555) or variable transparency (3443, 3444)

4x4 bilinear upscale

2 bits per pixel modulation {0, 3/8, 5/8, 1} or {0, ½, ½+Alpha=0, 1}

Alpha Mask/Punch-through decode scheme less prone to halos

BaseColour B

BaseColour A

Mod.Mode

ModulationData

MSBs LSBs

64bit block16 bits 15 bits 32 bits1bit


Bilinear Upscale Cost Reduction

Keep decompression cost low – target bilinear upscale

Bilinear low-colour precision data Bit replication standard method to

‘increase’ colour depth Linear operation so… …can be done after bilinear

operation

Need 2x2 pixels from each upscale Borrow from HOS techniques:

Use ‘recursive patch subdivision’ Handful of additions and MUX

operations

RepresentativePixels

Interpolated Pixels

'Bilinear Patch'

Required pixels



The tricky bit …

The Compression Algorithm


Compression Algorithm

Still a ‘work in progress’

Current method uses two main phases:

Computation of initial A & B signals

Iterative improvement of A & B given ‘closest’ modulation values


Phase 1: Initial A & B Images

‘Low pass filter’ of image

Obtain delta signal

Generate principal axis signal from delta

Obtain optimum end points, A and B


Low Pass Filter

Want best fitting bilinear approximation

Could use optimum FIR filter for bilinear reconstruction…

…but it’s much cheaper to use linear wavelets and almost as effective

'Ideal' Linear FIR

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

-12 -8 -4 0 4 8 12

Pixel Position

We

igh

t


Low Pass Filter & Delta (x2) (256x256 image)


Axis Generation

Want to find maximum axes through deltas - similar to VQ and S3TC compression approaches

To avoid discontinuities, ‘flip’ some deltas to get better alignment

Low pass filter to get ‘axis’ image

Flipped Delta Image


Generate A & B Signals – part 1

Get position of delta image relative to axes For each pixel: compute (delta dot axis) / (axis dot axis) Gives signed scalar values

Take overlapping (Gaussian) weighted windows of pixels and sort by value… Example Window Values

-6

-4

-2

0

2

4

6

8

10

12

0 4 8 12 16 20 24 28 32 36 40

Value Index

Va

lue


Generate A & B Signals – part 2

Search for ‘ideal’ representative end points Reminiscent of searching in VQ or S3TC Exhaustive search too expensive

Uses adaptation of Generalised Lloyds Algorithm (GLA) (e.g. used in VQ) + Least squares regression line fit.

Grouped into Buckets

-6

-4

-2

0

2

4

6

8

10

12

-0.125 0 0.125 0.25 0.375 0.5 0.625 0.75 0.875 1 1.125

Buckets (0, 3/8, 5/8, 1)

Va

lue


Raw A & B: Lena ‘the Goth’ & ‘In the Headlights’

Multiply ideal end point values by axes to give raw A & B images Add original low pass signal


Filtered Initial A and B


Intermediate Modulation and Result before optimisation passes…


Iterative improvement

Make several passes of the image Each pass:

Compute Modulation (dot product with scaling) from current A and B Quantise Modulation Examine 3x3 block regions - Optimise A and B colours for region Quantise colours.


Optimising region colours

Select regions equivalent to 3x3 blocks

Colours affected by 16 AB rep pairs

Outer 12 assumed fixed

Modulation values used to update inner four A&B reps…

'Fixed' A&B Reps

A&B pairs to beoptimised

TL

T120

TR

BL BR

T0

Window of ~3x3texel-blocks

4x4 texelBlock


…Optimising A and B…

“… a good dose of gratuitous differential equations is needed to meet the paper quota…”

P. Heckbert.

‘Ray Tracing JELL-O’

No DE’s but…


Region optimisation

Remove content from pixels due to fixed outer 12 reps

Evaluation of 121 pixels equivalent to matrix multiply Mw by AB reps

Mw matrix set of 121x8 weights.

Each weight is combination of position (bilinear) and modulation value

Need to ‘solve’ for AB vector. 121 equations with 8 unknowns!!

120

1

0

120120

22

111

.

P

P

P

B

A

B

A

M

ww

ww

www

wwwwwwww

M

BR

TR

TL

TL

W

BA

BA

BBA

Bo

Ao

Bo

Ao

Bo

Ao

Bo

Ao

W

BRTL

BRTL

BRTLTL

BRBRBLBLTRTRTLTL


Singular Value Decomposition

Allows us to ‘solve’ over-determined systems

Solution gives result with the Least Squares Error

SVD finds ‘equivalent’ of inverse of Mw

Multiply Mw-1 by pixel values to

obtain ‘best’ solution

But…

PixelsMsolution W .1


Pitfalls…

Despite 121 equations and 8 unknowns… System can be under-constrained! e.g. Simple Gradation Gives null space in solution matrix – take care

Garbage in, sub-optimal out… Poor choice of modulation value gives odd ‘optimum’ results A and B colours can fly off to extremes, e.g., image B


Final Result and Error image (x4)


Texture Compression – Section 62bpp mode

Similar to 4bpp

‘x’ dimension scaled by 8 rather than 4 for A and B

Modulation values either 1bpp (like CCC) or interpolated from neighbours (2bpp in chequerboard pattern)


Example Results: Lorikeet

Original

Original S3TC

4bpp 2bpp


Examples RMS

Image S3TC 4bpp

Lorikeet 9.80 8.10*

Lena 7.91 6.95

Kodak Image 02 6.60 6.13

Kodak Image 03 5.70 5.37


Future research

Apply to 3D textures

Re-investigate better interpolation

Higher or lower ‘colour’ dimensions

Remove ‘texture wrap around’ assumption

Better A&B fitting in compressor


Summary

Presented new lossy texture compression method.

Inexpensive hardware decode Avoids indirection ‘Block-like’ decode behaviour Optimised to feed bilinear texture filter ‘building block’

Typically good quality Closer match to image behaviour Avoids block boundary artefacts

Compressor open-ended research!

www.imgtec.com www.powervr.com PowerVR is a division of Imagination Technologies Ltd.

Questions?

www.imgtec.com powervr is a division of imagination technologies ltd. texture compression using...

Documents

bpp slide

texture compression

bpp colour compression

texture data slide

overview slide

researched compression

bpp pvrtc compression

evident slide