photogrammetry and star wars battlefront

GDC2016_DICE_EA_PhotogrammetryStarWarsBattlefront

Photogrammetry andStar Wars Battlefront

Andrew Hamilton Lead Environment ArtistKenneth BrownTechnical Art DirectorEA DICE

Buckle Up.

Photogrammetry

Photogrammetry - the science of making measurements from photographs, especially for recovering the exact positions of surface points. Moreover, it may be used to recover the motion pathways of designated reference points located on any moving object, on its components and in the immediately adjacent environment. Photogrammetry may employ high-speed imaging andremote sensingin order to detect, measure and record complex 2-D and 3-D motion fields (see alsosonar,radar,lidar, etc.). Photogrammetry feeds the measurements from remote sensing and the results ofimagery analysisintocomputational modelsin an attempt to successively estimate, with increasing accuracy, the actual, 3-D relative motions within the researched field. (source: Wikipedia)2

Using photos for measurements and, in our case, modelling/texturing.

Photogrammetry

Photogrammetry - the science of making measurements from photographs, especially for recovering the exact positions of surface points. Moreover, it may be used to recover the motion pathways of designated reference points located on any moving object, on its components and in the immediately adjacent environment. Photogrammetry may employ high-speed imaging andremote sensingin order to detect, measure and record complex 2-D and 3-D motion fields (see alsosonar,radar,lidar, etc.). Photogrammetry feeds the measurements from remote sensing and the results ofimagery analysisintocomputational modelsin an attempt to successively estimate, with increasing accuracy, the actual, 3-D relative motions within the researched field. (source: Wikipedia)3

Photogrammetry

A detailed breakdown of our step-by-step workflow can be found here. 4

Photogrammetry

Align photos.5

Photogrammetry

Generate point cloud.6

Photogrammetry

Generate highpoly mesh.7

Photogrammetry

Generate/project textures.8

The End.

9

A RetrospectiveThe beginning: research and developmentScanning plans and goalsCapturing the planetsMentality and workflowsGame assets in a photogrammetry worldTerrainLevel Construction KitsKey learnings and takeaways

10

Big Changes

New console generationNew franchise Very high expectations to deliver after BF4A clean slate; we want to do it right!PBR on the way but not ready (in the beginning)

Opportunity to change the way we think and create contentAvailable manpower (artists transitioning from BF4)

11

Star Wars!Learning from the pastPhotogrammetry pushing the boundariesA really tight scheduleIf were saying 60, we need to be 60Keeping it togetherChallenges

Photogrammetry can make great high res, largely organic assets We still need man (alien?) made assetWe need to significantly increase our efficiencyLets look at were we are and re-evaluate everything we know. Nothing is off limits.Today well discuss some of the software we settled on12

Dev Structure3 month development cyclesContent development until the endNeed for 2 week buffer discoveredApproval lockStabilization

13

The beginning

The beginnings of photogrammetry with Battlefield 414

Into Star Wars: Battlefront

An example of the range of captured assets from the Lucas archives.15

An example of the range of captured assets from the Lucas archives.16

Supporting scanned assets with other tools such as Marvellous Designer17

Hard-Surface materials created with traditional methods18

An example of the level of detail were able to get with combining photoscan with new (to us) software. 19

An example of the level of detail were able to get with combining photoscan with new (to us) software. 20

An example of the ingame assets using the combination of techniques.21

Scan everything!..?

Bridge the gap between scanned characters and their surroundings.Make the best looking Star Wars game ever! ..at 60fps We have so little time

22

From here...Battlefield 3: Armored Kill

A screenshot from Battlefield: Armored Kill that was considered realistic during its time23

..to here!

to a screenshot from Star Wars: Battlefront, using photogrammetry on a very limited number of selective content to build up a realistic environment24

25

Capture the complete asset libraries of all planets.Focus and capture only what is essential, not everything we see.Capture meaningful variation and features.Visit the real film locations for true inspiration and understanding.Location capture goals

26

Simple kit (24mm lens, Canon 6D, tripod, color chart, etc)Ideal weather conditionsOn-location breakdownThinking in terms of usable piecesAware of Frostbite strengths/weaknessesFocus on quality over quantity of capturesCapturing on location

27

Dirt groundLeaves and twigsClover groundClover leavesFern largeFern mediumFern smallLog largeLog mediumLog smallTree largeTree mediumTree smallTree stumpVegetationACDBEFGHIJKLMNO

Overlap!!Covering all angles to reduce cleanup.Avoid cast shadow - ambient light is easily removed later in the process.Most reliable results occurred when full asset was visible in most photos.Avoid objects with too intricate details.Be sure that all capture methods are consistent and clean.Prepare yourself and equipment for various conditions weather can make the process sloppy and destroy consistent results.Best practices

29

Capturing the planets

A slideshow of locations we have visited on scanning trips, for a quick glimpse into the range and variety of settings at our disposal.

Iceland30

Iceland31

Death Valley, USA32

Utah, USA33

Death Valley, USA34

National Redwood Forests , USA35

National Redwood Forests , USA36

Finse, Norway37

Finse, Norway38

Iceland39

Iceland40

Iceland41

A small selection from our large library of scanned materials that would be used for Sullust terrain surfaces42

A small selection from our large library of scanned materials that would be used for Tatooine terrain surfaces43

A small selection from our large library of scanned materials that would be used for Endor terrain surfaces44

A small selection from our large library of scanned materials that would be used for Hoth terrain surfaces45

A small selection from our large library of scanned materials that would be used for Hoth terrain surfaces46

Vegetation and ground scattering assets captured on a blue board for easy background extraction47

An example of a scanned asset, showing how intentionally generic and tilable they are.48










Successful location scanningA small and manageable group with light equipment is ideal.Structured and focused on location.Absolutely key to connect and understand how to recreate real locations.

On location in Iceland

58

Managing the dataOver 100,000 photos to manually sortOver 2000 assets to manually reviewOver 14 hours of video to categorizeLess than ideal storage solutionA lot of room for improvement here!

59

Workflow


60

WorkflowNow we have the data, so lets go.


61

How long does an asset take?Is Photogrammetry a savings? (Straw Poll)

Data taken from a cross sampling of content creator disciplines. Straw poll not necessarily accurate 1:1, but representative of reality. 62

How long does an asset take?With Automation

Photogrammetry workflow

Remove vignette, keep lens distortion.Batch calibration to color chart.This entire step should be automated!PS Camera RAW

A basic step by step guide through the Photogrammetry workflow64

Photogrammetry workflowMost assets need to be masked very time consuming if not automatic.Baking at High gave most useable and manageable results for Objects.Baking at Ultra gave best results for Terrain files are huge though.Agisoft PhotoScan


Photogrammetry workflowMosaic rather than Average blend gives much sharper results.Shredded UVs take to Zbrush and reproject, or bake high res.

Agisoft PhotoScan


Photogrammetry workflowSome assets completely automated in MeshLab.Some needed fine tuning in Topogun ok if very limited asset library.

Topogun / MeshLab


Photogrammetry workflowDetail textures applied later means UVs demand very little stretching.UV direction important for consistent detail texture alignment, sometimes may need a second UV just for detail tiling.

Maya / UVLayout


Photogrammetry workflowColor, Tangent Normals, Object Normals, Heightmap, AO, Cavity.

XNormal

A basic step by step guide through the Photogrammetry workflow

69

Photogrammetry workflowObject Space Normals used as a mask to remove ambient lighting.AO or cavity map for fine tune lighting removal.

XNormal

A basic step by step guide through the Photogrammetry workflow

70

16 bit depth is your friendRemove as much as possible from camera rawOnce removed, move on to ambient Object space normal maps ftwPhotogrammetry workflowLight Removal

ShadowsNeed to mask out shadowsSuper important to have 16 bit depthBefore that, remove as much as possible from the camera raw (shadow slider photoshop camera raw plugin)Create mask from whats dark.Flatten luminance in the colors firstShadows tend to be colder, so you can use color to grab them. Once youve gotten the shadows removed, you can treat it like something thats been captured overcastGet rid of ambient lightTwo waysUse the normal map (baked out in object space) to get the angle.Use the occlusion, baked from the geometry. Also need to use the mask to remove the color (bounced light)Overcast is best

71

WorkflowLight RemovalShadows are generally:DarkerCooler (use color to grab)Create mask

WorkflowLight RemovalShadows are generally:DarkerCooler (use color to grab)Create mask

WorkflowLight RemovalNow were left with:Ambient Bounced

Light RemovalObject Normal In this example, shadows are not so distinctGenerate an object normal mapThis gives distinct colors for the different object normals

Light RemovalObject Normal Using Black & White mask in Photoshop you can pull out different directions

WorkflowNow we have the base source data

Lets look at other aspects of the asset creation77

Texel DensityGraphical representation of a pixelAn initial pitfall for usThe following screenshots are 1080p Illustrates texture pixel (texel) to screen pixel ratio

Texel refers to the graphical representation of a pixel on a 2d object (eg. a texture). Basically, when you look at a pixel of a texture rendered on a 3d model, you are looking at atexel. 78

In this shot ~25xs more pixels available than the screen can render. 79

~10 texture pixels per screen pixel(excluding mips)

Mips always try to keep a 1:1 ratio. So detail will get mipped away anyways!80

81

82

83

Not only is this extremely close for 1p, its an even more extreme distance for 3p. At this resolution, there is certainly an absolute difference in quality, but a trivial relative difference in quality.84

For Comparison: Two 4k Textures. 55mb

To give you a sense of the memory impact (and subsequent streaming impact) that this has. 85

For Comparison: Two 1k Textures. 3.7mb

The quality of these assets are great. This just reinforces what weve been saying focus on the craftsmanship. More resolution isnt whats going to make these assets look better the overall quality of the asset is whats going to sell it.

86

Texel DensityYoud be surprised what you can get away withDetail maps are our friendsLarge textures have a very big streaming overheadWe have an 800mb texture pool on Gen4One 8k texture is almost 90mbMESH DENSITY vs. Takeaways

87

Texture ArraysThe photoscanned asset as a base is generally quite low res.All environmental assets use a consistent preset shader.Planet-specific Texture Array instance shaders add the detail on top. Object Preset Shader

88

Texture ArraysThis texture array instance shader is set up for normal, reflectance, and color. Texture Array Instance Shader

In this example, the multiply node is set to a factor of 4, give a range from 0-4 (or 5 distinct integers). Color information (only greyscale though) is stored in the alpha. 89

Texture ArraysBrightness is painted in 5 even steps (as there are 5 detail textures).The order of these layers must correspond to the order in the Array.Tree Normals (Texture Array mask in Alpha)

Brightness %0255075100DetailLeavesBarkBurnt BarkRockMoss

90

Texture ArraysDetail textures are fading away towards the end of LOD0 with LodFade.LOD1 has a new simplified shader without detail textures.Texture Array Instance Shader

91

Texture ArraysLike most texture assets in Battlefront, we heavily pack into channels.The detail texture uses RG for normals, B for Reflectance, A for Color.Detail Texture within the Texture Array

92

Texture ArraysUnique PhotoScan base is surface blurred, while keeping important details.Drastically reduces texture memory.

93

Texture ArraysTexture Array tiling detail on top fill in the details without overpowering the base.Drastically increases texture resolution.

94

Photogrammetry workflowWorking too much with an asset quickly deteriorates its natural look.Standardized preset shaders ensure consistent in-game results.Repetitive work - there is a lot of room for procedural workflows here!Photogrammetry is a great tool, but not a magic bullet.

95

How do we ensure all assets fit together consistently?

Old Asset CreationQUALITY

TECHEFFORTPhotogrammetry

TECH + TECHNIQUESQUALITY

EFFORTNew Asset Creation

QUALITY

TECHEFFORT

NEEDED

EFFORT

TECH + TECHNIQUES

?Photoscan vs. Everything Else

- Anything we can do to increase efficiency 96

Helps us bridge that gapLessons from Criterion, Guerilla Games, Dontnod etc Trying to get everyone to switch together can backfire. Some people will adapt instantly. Let them lead!For PBR, there isnt really a middle road.PBRQuick Thoughts

PBR is a talk in and of itself, but there a few things worth mentioning.97

PBRAssets generally fit well but still needed to be balanced against the scene.Finding and trusting true PBR values at times was a challengeConstant reinforcement helps prevent inconsistencies

98

BaseColor

Render passes that compose the final image99

Normals

Render passes that compose the final image

100

Reflectance


101

Smoothness


102

SubSurfaceTranslucency


103

Shadow


104

Ambient Occlusion


105

Light + Indirect


106


107

VegetationReaching 60fps lets make a dense forest!Integrating hand-built vegetation into a PhotoScanned world was tough to reach comparable quality!Difficult to get good results from scanned vegetation complex and noisy, resulting in horrible scans.Based on photos of vegetation from location captured on blue board.

108

VegetationAll vegetation share the same texturesThere are textures created separately for close up, mid, and far distances.There is a lot of texture packing within vegetation assets to reduce complexity of shaders.Normals created by broadly modelling around leaves and branches Maya.

109

VegetationUnique PhotoScanned base blends to tiling texture that extends upwards.Tree base is generally evenly tessellated for best displacement results.

Tiling trunk textureUnique base texture

110

Terrain/asset blendingA lot of work put in to reduce the visible disconnect between assets and terrain.Most assets have terrain radiosity projected on to them.Texture Arrays world projected and details match.Terrain color and normals applied to most assets.

111

Terrain/asset blendingA lot of work put in to reduce the visible disconnect between assets and terrain.Most assets have terrain radiosity projected on to them.Texture Arrays world projected and details match.Terrain color and normals applied to most assets.

112

Mesh DisplacementGreat quality boost for close up detail.Object displacement stopped at ~7 meters.An ongoing balance between quality and performance.

113

Mesh DisplacementOFF

114

Mesh DisplacementON

115

Ability to define displacement shape (via mask)One of our Must Have featuresCritical for ATAT footprintsTerrain Decal ComponentCombined with detail displacement shader(More on this a little later)TessellationHeightfield Displacement

116

117

Quality vs. Speed ratio is extremely highFeel free to get creative

Tessellation and PhotogrammetryHeightfield Displacement

Speaking of terrain.Terrain

124

TerrainFinal big chunk of our photogrammetry puzzle.

125

Terrain

8x8km with a 1x1km playable space.Huge background terrain meshes.Terrain textures captured from real world.Layers limited to control draw calls.

126

Moving (mostly) away from WorldMachine..

Battlefield titles extensively used WorldMachine for the terrain, but we needed to support our up-close Photoscanned assets with realistic landscape in the distance127

..to real-world topography data.

We instead complemented Photoscanned assets with real-world landscape data128

Processing the data

129

Extending the horizon

100x80km

The square in the middle represents the 8x8km Frostbite terrain, where mesh extends the backdrop a further 50km in all directions130

Texture Capture

131

Best practices - TerrainWe aimed for 3x3 meter patches5x5 photo grid ensures enough overlapCapture only 1 defined element at a timeLow ISO and high F-Stop for sharp imageAvoid noisy surfaces such as grass

132

Example of a scanned surface133



Terrain TessellationGreat quality boost for close up detail on terrainMajor upgrade for us. Can never go back!Terrain displacement stopped at ~15 meters.

136

Terrain TessellationOFF

Terrain TessellationON

Level Construction KitsContent and documentation packages.Provides fast level iteration.Shifts focus to building levels rather than content during Production.Allows us to more quickly understand performance and keep track of it.

139

Level Construction Kits8x8km highly detailed terrain.Ready to paint terrain materials.Easily accessible limited library of content.Lighting presets based on the films.

140

HothEndorTatooineSullust

A Level Construction Kit was created for each planet or biome141

LCK video142

Endor

This scene appears to be comprised of many elements with much variation143

Endor

but it is only comprised of a very limited amount of generic assets that work well to populate a large area without appearing repetitive.144

Hoth


Hoth


Tatooine


Tatooine


Sullust

This scene appears to be comprised of many elements with much variation

149

Sullust


A slideshow showing the variety of locations in Star Wars: Battlefront, created with a limited set of assets using the Level Construction Kits151















Key learnings and takeawaysSolid processes and workflows in place before Production was a big win.Well planned and structured Photogrammetry trips results in clean and manageable assets.Photogrammetry is not a silver bullet it was only one part of achieving the final results in game.It was crucial to have artists own and be responsible for their locations/planets - Consistency is key when working with scanned assets.Working towards PBR was an ongoing process to success.Much of our creative process was incredibly repetitive, much of which can and should be automated.There was way too much manual labor, with levels being built by brute force without the help of workflow-improving tools.

166

167

Special ThanksJohn Troive -Pontus Ryman -Petter Skld -Tim McLeod -Simon Barsky -Darko Pracic -Daniel Cambrand -Oscar Carlen -Madina Chionidi -Bjrn Arvidsson -Robert Kihl - and of course, the entire Battlefront team at DICE!

Senior Environment ArtistSenior Environment ArtistSenior Environment ArtistEnvironment ArtistEnvironment ArtistEnvironment ArtistEnvironment ArtistSenior Lighting ArtistCharacter ArtistCharacter ArtistRendering Engineer

168

Thanks!Andrew Hamilton Lead Environment [email protected] Brown Technical Art [email protected]

169

photogrammetry and star wars battlefront

Entertainment & Humor