Presented by : ……..Presented by : ……..

Ramesh Raskar, Greg Welch and Ramesh Raskar, Greg Welch and Henry FuchsHenry Fuchs

University of North Carolina at Chapel University of North Carolina at Chapel HillHill

Presented by : ……..Presented by : ……..

Ramesh Raskar, Greg Welch and Ramesh Raskar, Greg Welch and Henry FuchsHenry Fuchs

University of North Carolina at Chapel University of North Carolina at Chapel HillHill

Seamless Projection Overlaps using Image Warping and Intensity

Blending

Acknowledgements

• Office of the Future group at UNC Chapel HillOffice of the Future group at UNC Chapel Hill

• Andrei State (sketches)Andrei State (sketches)

• SupportSupport

– NSF Science and Technology Center for Graphics NSF Science and Technology Center for Graphics and Visualization, USAand Visualization, USA

– Advanced Networks and Services,National Tele-Advanced Networks and Services,National Tele-Immersion InitiativeImmersion Initiative

– Defense Advanced Research Projects Agency, USA Defense Advanced Research Projects Agency, USA

– Intel CorporationIntel Corporation

• Office of the Future group at UNC Chapel HillOffice of the Future group at UNC Chapel Hill

• Andrei State (sketches)Andrei State (sketches)

• SupportSupport

– NSF Science and Technology Center for Graphics NSF Science and Technology Center for Graphics and Visualization, USAand Visualization, USA

– Advanced Networks and Services,National Tele-Advanced Networks and Services,National Tele-Immersion InitiativeImmersion Initiative

– Defense Advanced Research Projects Agency, USA Defense Advanced Research Projects Agency, USA

– Intel CorporationIntel Corporation

Seamless Projection Overlaps

P1

P2

P3 P1

P2P3

Display Surface

Projectors

Well Defined Display Configuration

Irregular display and projector configuration

Traditional Flexible Setup

Goal• High Resolution, Wide Field of View High Resolution, Wide Field of View

DisplayDisplay

• Overlapping front projection systemOverlapping front projection system

• Seamless displaySeamless display

– Geometric Registration of Geometric Registration of overlapsoverlaps

– Intensity Blending near edgesIntensity Blending near edges

• Irregular display surface Irregular display surface configurationconfiguration

• High Resolution, Wide Field of View High Resolution, Wide Field of View DisplayDisplay

• Overlapping front projection systemOverlapping front projection system

• Seamless displaySeamless display

– Geometric Registration of Geometric Registration of overlapsoverlaps

– Intensity Blending near edgesIntensity Blending near edges

• Irregular display surface Irregular display surface configurationconfiguration

P1

P2P3

Motivation :Office of the Future (UNC ‘98)

• Irregular Irregular surfacessurfaces

• No specific No specific projector projector configuratioconfigurationn

• Irregular Irregular surfacessurfaces

• No specific No specific projector projector configuratioconfigurationn

Other Spatially Immersive Displays

1. CAVE (UIC 1. CAVE (UIC EVL)EVL)

• Non-overlapping Non-overlapping projectionsprojections

• Flat displays wallsFlat displays walls

• Well-defined Well-defined projection projection configurationconfiguration

1. CAVE (UIC 1. CAVE (UIC EVL)EVL)

• Non-overlapping Non-overlapping projectionsprojections

• Flat displays wallsFlat displays walls

• Well-defined Well-defined projection projection configurationconfiguration

Other Spatially Immersive Displays (Cont.)

2. VisionDome 2. VisionDome (Alternate Realities)(Alternate Realities)

• Single Projector with Single Projector with expensive opticsexpensive optics

• Well defined display Well defined display surfacessurfaces

2. VisionDome 2. VisionDome (Alternate Realities)(Alternate Realities)

• Single Projector with Single Projector with expensive opticsexpensive optics

• Well defined display Well defined display surfacessurfaces

Other Spatially Immersive Displays (Cont.)

3. Cylindrical Screens (Trimension, 3. Cylindrical Screens (Trimension, Panoram)Panoram)

• 3 Projectors with side-by-side rectangular 3 Projectors with side-by-side rectangular overlapoverlap

• Well defined display surfaceWell defined display surface

• Precise electro-mechanical setupPrecise electro-mechanical setup

3. Cylindrical Screens (Trimension, 3. Cylindrical Screens (Trimension, Panoram)Panoram)

• 3 Projectors with side-by-side rectangular 3 Projectors with side-by-side rectangular overlapoverlap

• Well defined display surfaceWell defined display surface

• Precise electro-mechanical setupPrecise electro-mechanical setup

Office of the Future (UNC ‘98)

• Irregular surfacesIrregular surfaces

• No specific projector configurationNo specific projector configuration

• Irregular surfacesIrregular surfaces

• No specific projector configurationNo specific projector configuration

Office of the Future Configuration

Projectors

Single User

Irregular Surfaces

User

Top View

Our Approach

• Calibration of display configuration using a Calibration of display configuration using a video cameravideo camera

• Rendering using texture mapping hardwareRendering using texture mapping hardware

• Pair-wise registration and blending of Pair-wise registration and blending of projection overlapsprojection overlaps

• Calibration of display configuration using a Calibration of display configuration using a video cameravideo camera

• Rendering using texture mapping hardwareRendering using texture mapping hardware

• Pair-wise registration and blending of Pair-wise registration and blending of projection overlapsprojection overlaps

Key Features

+ Display surfaces may not be flat wallsDisplay surfaces may not be flat walls

+ Projection axis not orthogonal to displaysProjection axis not orthogonal to displays

+ Flexible projection configurationFlexible projection configuration

+ Projector’s intrinsic or extrinsic parameters Projector’s intrinsic or extrinsic parameters not needed not needed

– Generated image correct for a single ideal Generated image correct for a single ideal viewer’s locationviewer’s location

+ Display surfaces may not be flat wallsDisplay surfaces may not be flat walls

+ Projection axis not orthogonal to displaysProjection axis not orthogonal to displays

+ Flexible projection configurationFlexible projection configuration

+ Projector’s intrinsic or extrinsic parameters Projector’s intrinsic or extrinsic parameters not needed not needed

– Generated image correct for a single ideal Generated image correct for a single ideal viewer’s locationviewer’s location

Calibration

• Step 1 : Geometric RegistrationStep 1 : Geometric Registration

– Pre-distort images so that when projected Pre-distort images so that when projected they appearthey appear

◊ Perspectively correctPerspectively correct

◊ Aligned with neighboring projectionAligned with neighboring projection

• Step 2 : Intensity NormalizationStep 2 : Intensity Normalization

– Blend images from overlapping projectionsBlend images from overlapping projections

• Step 1 : Geometric RegistrationStep 1 : Geometric Registration

– Pre-distort images so that when projected Pre-distort images so that when projected they appearthey appear

◊ Perspectively correctPerspectively correct

◊ Aligned with neighboring projectionAligned with neighboring projection

• Step 2 : Intensity NormalizationStep 2 : Intensity Normalization

– Blend images from overlapping projectionsBlend images from overlapping projections

Calibration Step 1 : Geometric Registration

Warping Images Warping Images

• For irregular display surfacesFor irregular display surfaces

– Desired image is pre-warpedDesired image is pre-warped

– Warping function is found using a wide field Warping function is found using a wide field of view camera (WFOV)of view camera (WFOV)

• For flat display surfacesFor flat display surfaces

– Projected images related by a 3x2 Projected images related by a 3x2 transformationtransformation

Warping Images Warping Images

• For irregular display surfacesFor irregular display surfaces

– Desired image is pre-warpedDesired image is pre-warped

– Warping function is found using a wide field Warping function is found using a wide field of view camera (WFOV)of view camera (WFOV)

• For flat display surfacesFor flat display surfaces

– Projected images related by a 3x2 Projected images related by a 3x2 transformationtransformation

Calibration Step 1 : Geometric Registration

Warping Images for Irregular Warping Images for Irregular Display SurfacesDisplay Surfaces

• Keep the WFOV camera at ideal viewer’s Keep the WFOV camera at ideal viewer’s locationlocation

• Project dots with the projector one by oneProject dots with the projector one by one

• Find mapping from projector image to camera Find mapping from projector image to camera imageimage

• Invert mapping to find the warping functionInvert mapping to find the warping function

Warping Images for Irregular Warping Images for Irregular Display SurfacesDisplay Surfaces

• Keep the WFOV camera at ideal viewer’s Keep the WFOV camera at ideal viewer’s locationlocation

• Project dots with the projector one by oneProject dots with the projector one by one

• Find mapping from projector image to camera Find mapping from projector image to camera imageimage

• Invert mapping to find the warping functionInvert mapping to find the warping function

Camera to Projector warping function

Display Surface

WFOV Camera

Icamera(u,v)

Observing a point on display surface

Icamera(u,v) = Which projector pixel ?

Camera to Projector warping function

P1

Display Surface Projector

P2

Iproj2(x,y)

Iproj1(x,y)

Projector

Which projector pixel illuminated the same point on display surface ?

Icamera(u,v) = Iproj1(x,y), Iproj2(x,y)

Calibration Step 2 : Intensity Normalization

Blending Projected Images Blending Projected Images

• For irregular display surfacesFor irregular display surfaces

• Projection overlap observed in camera image Projection overlap observed in camera image spacespace

• Assign intensity weights for projector pixelsAssign intensity weights for projector pixels

Blending Projected Images Blending Projected Images

• For irregular display surfacesFor irregular display surfaces

• Projection overlap observed in camera image Projection overlap observed in camera image spacespace

• Assign intensity weights for projector pixelsAssign intensity weights for projector pixels

Intensity in Projector Overlap

P1

Display Surface

Projector

P2

Iproj2(x,y)

Iproj1(x,y)

Projector

How to make intensity in overlap region same as everywhere else ?

WFOV Camera

Icamera(u,v)

Intensity Normalization

Camera Scanline

Camera Scanline

Projected Intensity

P1 P2

P1 P2

Resultant Intensity

Resultant Intensity

Before Normalization

After Normalization

Projected Intensity

Rendering

• Two pass rendering methodTwo pass rendering method

• First pass : Compute the desired imageFirst pass : Compute the desired image

• Second pass :Second pass :

– Warping with standard OpenGL texture Warping with standard OpenGL texture mappingmapping

– Intensity blending using Alpha channel of Intensity blending using Alpha channel of graphics hardwaregraphics hardware

• Two pass rendering methodTwo pass rendering method

• First pass : Compute the desired imageFirst pass : Compute the desired image

• Second pass :Second pass :

– Warping with standard OpenGL texture Warping with standard OpenGL texture mappingmapping

– Intensity blending using Alpha channel of Intensity blending using Alpha channel of graphics hardwaregraphics hardware

Results: correct view for one user

Live Panoramic Video Image captured using a WFOV camera cluster

Summary

• Robust general purpose method to create Robust general purpose method to create seamless imagesseamless images

• General configuration of projectors and display General configuration of projectors and display surfacessurfaces

• High-resolution wide-field of view display using High-resolution wide-field of view display using cluster of projectorscluster of projectors

• No expensive optics or electro-mechanical No expensive optics or electro-mechanical setupsetup

• One-time calibration procedureOne-time calibration procedure

• Robust general purpose method to create Robust general purpose method to create seamless imagesseamless images

• General configuration of projectors and display General configuration of projectors and display surfacessurfaces

• High-resolution wide-field of view display using High-resolution wide-field of view display using cluster of projectorscluster of projectors

• No expensive optics or electro-mechanical No expensive optics or electro-mechanical setupsetup

• One-time calibration procedureOne-time calibration procedure

Applications

• Office Scenarios, flexible setupOffice Scenarios, flexible setup

• Dome shaped displays using cluster of Dome shaped displays using cluster of projectors instead of expensive opticsprojectors instead of expensive optics

• Ease of setup for cylindrical projection Ease of setup for cylindrical projection screens using multiple projectorsscreens using multiple projectors

• Synthetic images or Video images on large Synthetic images or Video images on large display environmentsdisplay environments

• Office Scenarios, flexible setupOffice Scenarios, flexible setup

• Dome shaped displays using cluster of Dome shaped displays using cluster of projectors instead of expensive opticsprojectors instead of expensive optics

• Ease of setup for cylindrical projection Ease of setup for cylindrical projection screens using multiple projectorsscreens using multiple projectors

• Synthetic images or Video images on large Synthetic images or Video images on large display environmentsdisplay environments

Ten Years from now .. • Light bulbs replaced by Light bulbs replaced by

cheap projectors cheap projectors

• Sufficient number of Sufficient number of inexpensive camerasinexpensive cameras

• Widely available texture Widely available texture mapping graphics hardwaremapping graphics hardware

• Projectors, LCD panels to Projectors, LCD panels to create ‘desktop’ instead of create ‘desktop’ instead of looking at a 21’ monitorlooking at a 21’ monitor

• Light bulbs replaced by Light bulbs replaced by cheap projectors cheap projectors

• Sufficient number of Sufficient number of inexpensive camerasinexpensive cameras

• Widely available texture Widely available texture mapping graphics hardwaremapping graphics hardware

• Projectors, LCD panels to Projectors, LCD panels to create ‘desktop’ instead of create ‘desktop’ instead of looking at a 21’ monitorlooking at a 21’ monitor