Deferred Lighting and Shading

With Implementation Details and Demos for the PC (DX9), Xbox, and PS2 Platforms

© 2004 Blue Shift, Inc.© 2004 Blue Shift, Inc.

Presenters

• Rich Geldreich, Ensemble Studios, rgeldreich@ensemblestudios.com

• Matt Pritchard, Ensemble Studios, mpritchard@ensemblestudios.com

• John Brooks, Blue Shift, Inc., jbrooks@blueshiftinc.com

Session Overview

• High-level description• PS2, Xbox demos

• Xbox demo will be shown early

• Pros and cons• Theory and concepts • Platform specific details• Advanced stuff, future research

What is it?• Deferred lighting and shading is the decoupling of geometry

rasterization from lighting and (maybe) shading.– Scene passes only write attributes, not lit pixels

• Light independent attributes are dumped to buffers in one or more deferment passes

– Some common attributes for deferred lighting:» normal, position/depth

– Common attributes for deferred shading: » albedo, specular color/power

• Basically, raw data needed for lighting (and maybe shading) is saved and reused in later passes

• Lights become 2D image space ops• From now on, when we say “Deferred Shading” we usually

mean “Deferred Lighting/Shading”.

Deferred Shading Demos

– Xbox: Gladiator 2K4• 2K3 version was shown at last year’s GDC to a

limited audience. This year’s version is a bit more optimized, packs albedo colors differently (YCbCr instead of sRGB), and implements a “metalness” hack for better specular highlights. Created by Blue Shift, Inc.:

– PS2: Bunny Demo• Object space normal maps, gloss maps• Directional lights with specular highlights• Two hemisphere area lighting• HDR accumulation, tone mapping

Deferred Lighting, Deferred Shading, or Both?

• Lighting – Light accumulation– Process can take into account a pixel’s location, normal (or

full tangent frame for anisotropic shading), and some aspect(s) of the illumination model’s BRDF function (i.e. specular power for Phong)

• Basically, “lighting” = Irradiance computation for direct light sources, diffuse and specular contributions separately accumulated

• Shading – The process of computing an output pixel’s final color (or exit radiance for HDR) using all of the pixel’s attributes (such as albedo, specular color, etc.), accumulated light, and a shading function.– Shading can be done at many places:

• During lighting passes (only final color is accumulated)• All at once as a final image space post process• During a final scene render pass

Why Bother with Deferred?• Some shadow techniques (stencil shadow volumes,

“forward” shadow buffering) require a Z pre-pass anyway.– Why not write some per-pixel attributes too, instead of

just depth? • Predictability: The smaller or further away the light, the

less processor time it takes to compute its influence, independent of the quantity of objects the light illuminates.

• Consistency: Entire scene can be lit using the same illumination model, shadowing technique, etc.

• Quality: Most of the lighting equation recomputed per-pixel– Good example is light attenuation (falloff): No per-

vertex approximations that break down on large triangles

• Simplicity: Scene rasterization passes require attribute dumping only– No need to handle a zillion combinations of

dir/omni/spot/etc. lights in your vertex/pixel programs

Conceptual Operation1) Render scene to attribute buffer(s)

Example attributes: Normal, albedo, specular color, depth

2) For each light:I. Shadow pass (stencil ops, shadow buffering, etc.)

• Shadow buffering can benefit from having the scene’s depth buffer available as a texture

II. Render light’s bounding volume, accumulate diffuse/specular light contributions, or lit/shaded pixels

• May use a screenspace quad (directional light/full screen light), CPU projected/clipped screenspace n-gon, a 3D object with front face culling and projective texturing, tilization, etc.

3) (Optional) Shade scene• Render a full-screen quad, apply same shader to all pixels• Or… Render scene again to support arbitrary material

shaders4) Perform image space post processing effects

• Particles, blooming, streaks, fog, tone mapping, etc.

Other Advantages• Eliminates hard limits on the number of lights influencing

individual objects– Number of active lights per object in forward shading engines

is usually constrained in some way by the max. size of a single vertex/pixel program

• Potentially reduced overdraw cost– Depends on the lighting complexity– Not as important on GPU’s with fast Z occlusion hardware

• Lighting passes can take advantage of the fast Z occlusion hardware on modern GPU’s– Occluded lights, or lights totally in front of stuff can be almost

free• On some platforms (PS2) it may be the only way of

achieving bump mapped, per-pixel lighting with many lights– PS2 is quite capable of rasterizing per-pixel attributes to

buffers– Lighting and shading performed at full floating point

precision!

Cons of Deferred Shading 1/2

• Can be bandwidth/memory intensive– On some platforms, attributes must be

written once, and read back many times. – Multiple attribute buffers at high

framebuffer resolutions gobble memory

• Very high hardware requirements– Especially at typical PC game resolutions,

or with FSAA

Cons of Deferred Shading 2/2

• Per-material shaders or multiple BRDF’s are impractical on some platforms – On some platforms/shader models it’s infeasible to

use multiple pixel shader programs to shade the scene

– Fully texture driven shaders take on more importance

• Alpha blending– “Real” alpha blending (over operator [6]) is difficult

• To do it “correctly”, multiple pixels must be independently lit and shaded before applying the over operator

• Excuse: Alpha blending is a total hack anyway• We’ll show some hacks that can be decent alternatives

on some platforms

What Attributes to Render?• Minimal attributes:

– Albedo (typically sRGB or YCbCr)– Gloss (specular intensity, scalar)– Normal (typically unit vector)

• Can also write Yaw/Pitch, (ZSign,Y,X), etc.– Depth (scaler, float, int, or spread over several

components)• Could also write full 3D position, but this is unnecessary as

the full position can easily be “recovered” given the pixel’s screenspace location and Z depth.

• Additional attributes:– Specular color, power, BRDF or NDF (Normal Distribution

Function) index, etc.– For anisotropic shading: Perturbed Tangent (XYZ) and/or

Binormal (XYZ)– Special effects: Object ID (int)– Additional shader function parameters (example: flags)

Deferred on DX9 Hardware

• Attribute passes can use Multiple Render Targets (MRT)

• Unfortunately, must write depth or position, so up to one whole RT is lost. Can’t alias Z-buffer to a texture like on Xbox.

• Can write higher precision normals– 10-10-10 looks substantially better

than 8-8-8

Deferred on DX9 HardwareHDR Light Accumulation

• Alpha blending to HDR surfaces isn’t supported by current DX9 class hardware

• Lighting passes are just 2D screenspace ops, so it’s easy to predict which portions of the framebuffer are affected by each light

• We’ve accumulated HDR light in-place (i.e. coherent read-modify-write to a single buffer set as a render target and texture source) on the Radeon 9800 series by rendering 32x32 tiles and switching to dummy render targets after each light– Render targets must be temporarily

switched to unrelated surfaces to flush the R3xx’s backside caches (?).

Deferred on XboxRound One: Two Scene Passes

• Two scene rendering passes:– Render Albedo and Gloss (“C Buffer”)– Render Normal and Object ID (“N Buffer”)– Alias Depth/Stencil Buffer to linear 32-bit A8R8G8B8

texture

• Tricky part: Omni lights– Use texm3x3pad / texm3x3tex to “unproject” [9] and

transform pixels to Normalized Light Space (NLS) [10]• Volume texture lookup fetches NLS light vector and

attenuation• Rotate light vector to view space, renormalize in the combiners• Dump light vector and attenuation to temp. buffer• Perform usual Phong lighting calcs. in another pass in the

combiners• Accumulate lit and shaded pixels

Deferred on XboxRound Two: One Scene Pass

• We don’t have Multiple Render Targets on Xbox, but we can fake it by tightly packing attributes:– High word = Packed albedo and gloss (4-4-6-2)– Low word = Packed normal (1-7-8)

• Other stuff doesn’t differ from the two pass technique in any major ways, except for a more refined and optimized implementation.

• See Rich’s Gladiator presentation for more details.• Xbox could be pushed further

– Should be possible to do a scene ~2x more visually complex at 30Hz.

Deferred Shading on PS2Benefits of Deferred on the PS2

• Allows per-pixel shading on PS2!• Extensive per-pixel programmability

– floating point (including divide, sqrt, random, etc) – integer– data swizzling– lookup tables– data load/store (pixel local or between pixels)– looping– branching

• Unlimited shader length• Flexible shader memory use (shader program, constants,

etc)• Enables high-end effects like normal mapping & per-pixel

lighting

Deferred Shading on PS2Drawbacks to Deferred on the PS2

• GPU fill-rate– DMA attribute buffers from VRAM– DMA shaded pixels to VRAM– Multiple render passes needed to store per-pixel attributes

• CPU performance– Based on shader length & number of pixels shaded– Focus on efficient algorithms and tight asm code

• VRAM memory– Attribute buffers– Attribute textures

• DRAM memory– Per-pixel attributes– Shaded pixel data

Alpha Blending - It’s a Pain• Stippling/screen door transparency [4] [9]

– An old school hack useful on console games. Console video encoders and most displays form a big low pass filter, so the pattern is invisible except near edges

– Can alternate the stipple pattern every frame– Stippled surfaces interact with stencil shadows in a natural way

• Can also use stippling while rendering shadow volumes, for less than fully dark shadows [11]

– No explicit sorting required• Depth peeling [5]

– For surfaces that need blending: Peel back 1-3 layers– Independently light each layer, composite everything together

• Hybrid techniques– Forward (immediate) shade surfaces that need blending– Alpha blend overtop of the deferred engine’s output

Deferred on DX9 Hardware Other Neat Stuff We’ve Tried

• 2D Normal Distribution Functions (NDFs) [8]• Anisotropic shading, direction of anisotropy maps• Half-angle rotated to Perturbed Tangent Space, XY used as

texcoords for specular & iridescence map lookup• Requires storage of per-pixel perturbed tangent vectors

• Arbitrary material shaders• Deferred lighting, but forward (immediate) shading• Rerender scene after lighting, use PS 3.0 “vpos” register or

texture projection to read accumulated light, shade fragment using any material pixel shader

• SH Encoded Irradiance Volumes [2]• Use renderer to create HDR “Radiance Probes” for any position

in a scene – output is HDR environment map (cubemap)• Radiance probes can be quickly converted to irradiance

environment maps [7] using Spherical Harmonics, completely on the GPU

• SH coefficients can be packed into multiple volume textures and reused in later passes

References1. Nicolas Thibieroz, “Deferred Shading with Multiple Render Targets”.

ShaderX2 - Shader Tips & Tricks

2. Peter-Pike Sloan, “Efficient Evaluation of Irradiance Environment Maps”. ShaderX2 - Shader Tips & Tricks

3. Dean Calver, “Photo-realistic Deferred Lighting”. http://www.beyond3d.com/articles/deflight

4. Tom McReynolds, “Advanced Graphics Programming Techniques Using OpenGL”. http://www.sgi.com/software/opengl/advanced98/notes/node145.html

5. Cass Everrit, NVIDIA Order Independent Transparency demo

6. Jim Blinn, “Jim Blinn's Corner - Dirty Pixels”

7. Ravi Ramamoorthi, Pat Hanrahan, “An Efficient Representation for Irradiance Environment Maps”

8. Jan Kautz, “Rendering with Hand Crafted Shading Models”. Game Programming Gems 3

9. Atman Binstock, private conversation, 2001

10. Alex Vlachos, John Isidoro, Chris Oat, “Textures as Lookup Tables for Per-Pixel Lighting”. Game Programming Gems 3

11. Atman Binstock, private conversation, 2003