optimizing key parts of the dunia pipelinegood state: all-in ! early test show potential wins. 4...
TRANSCRIPT
Engine Architect - FarCry 4Ubisoft Montréal @azagoth
Rémi QUENIN
Optimizing key parts of the
DUNIA PIPELINEFast iteration for
THE DUNIA PIPELINE 250 GiB of package per Day
Official + Test Maps
Over 3.7 Million Lines Of Code2.5 M. C++ Runtime
6 studios, 450 people in MontréalGetting data everyday
~500 checkins per day350 data, 150 code (+sound)
DATA FLOW IN THE DUNIA PIPELINE
DATA FLOW IN THE DUNIA PIPELINE
DATA FLOW IN THE DUNIA PIPELINE
BinarizeBinarize
AUTOMATED TESTS IN THE DUNIA PIPELINE
BinarizeBinarize
FC3 WORST CASE SCENARIO
BinarizeBinarize
<1 min
<5 min
~40 min
~45 min
~15 min
FC3 WORST CASE SCENARIO
<1 min
2/15 min
~25 min
OPTIMIZING
1 Optimizing Compilation Time
2 Optimizing Nightly Builds
3 Optimizing Package Synchronization
4 Optimizing Local Change Testing
The Pipeline
COMPILATION TIME: FASTBUILDFrom 40 min to 4 min
1
HISTORY Editor DLL : ~40 min
3.7 Million LOC
Unity/Blob builds : ~20 minBad Iteration
Engine Architect pet projectFASTBuild by Franta Fulin
Good State: All-in !Early test show potential wins
4POINTS
PARALLELIZATIONMaximize local resources
CACHINGShare compilation results
BLOBBINGImprove iteration
FASTBuild
Share HW resourcesDISTRIBUTION
WIN
PARALLELIZATION Compiling/Linking : DLL
PARALLELIZATION Before
After
Compiling : Static Libs
WIN
PARALLELIZATION
PARALLELIZATION
Visual Studio: Context Switching File Cache eviction
32 Core Machine: >1000 Processes!
PARALLELIZATION
FASTBuild: NUM_PROCESSORS
CACHING
Write
Read
CACHING
Write
Read WIN
CACHING Before
After
BLOBBING
WIN
Reducing header compilation
BLOBBING
WIN
Edit
Modified File
StableWIN
Iteration
DISTRIBUTION
WIN
MISCELLANEOUS
Link Dependencies Prevent useless relink
BatchingBuild machine, pre-submit checks
REAL TIMINGS Editor-X64-Release DLL (before)
Editor-X64-Release DLL BLOB (before)
Incredibuild
FASTBuild
~40 mins
<4 mins (cache/distribution)
~20 mins
~10 mins (local PC!)
PS4: <1 min (cache/distribution)
~12 mins
EDIT&CONTINUE
Too Much Code: Takes minutes +Microsoft dropping support
DLL with Hot ReloadAnother story: lot of great content available online
Incremental LinkingWhere it actually helps
NIGHTLY BUILDS4 Times Faster
2
Improvements
6 hours
1.5 hours
45 Worlds
75 Worlds
PROFILINGUnderstand, optimize
RESOURCE CACHINGShare
HOW ?
Profiling Multi-Process
Process isolation
Multi-Machine Cluster of Workers
Interactions Working together
Big PictureUnderstand
Remote Profiling
Remote Profiling
On Top of Regular Tool Custom Performance Analyzer
Processes Shown as Threads Analyze Interactions
Fix ! Address Inefficiencies
Improvements: CPU Usage
Resource Caching
?
Resource Caching
Share Prevent useless work
Build MachinePopulate at night
EditorJIT Compilation (Just In Time)
Complex SystemKey: Understanding
Prevent useless workHardware is crucial
NIGHTLIES
Prevent Editor UsageCurrent Bottleneck
More IncrementalSome parts are not
FUTURE WORK
OPTIMIZE PACKAGE SYNC.From 20min to 1sec
3
FARCRY3 AC3 Network
2012 SUMMER
NETWORK
OPTIMIZE PACKAGE SYNC.3
SOFTWARE: RTPAL3.1
HARDWARE: ASSET STORE
3.2
SYNCH. PACKAGE: RTPALSoftware
3.1
PC BUILD
Differential Uploads
Sliced ISO
~50% Saving
Upload
STEAM PIPE
WE KNOW OUR DATA
Big Files File boundaries
Redundancy Same Resource, Several Worlds
Amount of new Data ?Out of 20GB package
WE KNOW OUR DATA
Textures: Main Resource80% of a package
Textures: Revisions Average 3 revisions
SOLUTION
Slice BigFiles into Parts
Hash Parts using MD5
Produce a Manifest
Package
SLICINGInto Parts
SOLUTION
LINK
SOLUTIONVERSION 1 VERSION 2
?
LINK
RESULTS
1GB new per 22GB package
40% Shrink Within Package
Reuse
95%2 Versions
RESULTS
FC3: Few weeks + Milestones
FC4: >10 000 Manifests
History
1 YEARFull Prod. Proof
THIS IS GOOD, BUT…
Regular Workflow…Gym/Test Map
20% of the package20% of 5% = 1%
THIS IS GOOD, BUT…
Linking…20 GB to write
Deploy…zzzzz…..
ON DEMAND
STREAM parts on demand
OPEN WORLD game
NETWORK is faster than crappy
HDD
FILE SYSTEM
VIRTUALIZE File Access
Regular DISK
MANIFEST Content
FILE SYSTEM STACK
FILE SYSTEM
?
FILE SYSTEM STACK
NETWORK FS
?
FILE SYSTEM STACK
MANIFEST FS
NETWORK FS
?
?
FILE SYSTEM STACK
MANIFEST FS
NETWORK FS
CACHE FS
?
?
RTPAL
FILE SYSTEMFILE SYSTEMFILE SYSTEM
RTPAL
STORAGE AND TRANSFER REDUCTIONHistory, Transfers
UNIFIED WORKFLOWFor all platforms
INSTANT PACKAGE SYNC1 click
RTPAL
Re-Invents Package Distribution
By Getting RID of Package Distribution
RTPAL
SYNCH. PACKAGE: ASSET STOREHardware
3.2
HIGH PERFORMANCEIOPS/Throughput
QUICKLY SCALABLE, HIGH CAPACITYAdapt to workload
ROBUSTFailure tolerance
NEEDS
2012 SOLUTIONS
GLUSTER FS
Distributed file systemCluster of PCs
Replication between nodesUp to 3 copies of a file
2 Network interfacesOne for internal data exchange
GLUSTER FS
8 x High End PCsHPZ420 (32GB RAM, 6 HT CPUs 3.2GHz)
8 x High End SDDOn Sata3 RAID0 Controller card
2 x Network Interfaces10 Gb/s
TESTGLUSTER FS – 8 NodesTHROUGHPUT – 30 users, 14 GiB
IOPS – 100k x 1KiB
18MiB/s/user (13min)SAN/EMC
GLUSTER 73MiB/s/user (3min)
SAN/EMC
GLUSTER 544 000 IOPS
250 000 IOPS
GLUSTER - COST
0 500 1000 1500 2000 2500 3000
Speed (MiB/s)
050100150200250300350400
Cost (k$)
GLUSTER
SAN VIOLIN
SAN EMC
USAGE SCHEMES
RTPalPackage distribution
Resources cachingTransformed assets
Code compilation cacheFASTBuild
Re-Evaluate CEPHImproved since 2012
Improve Healing ProcessIf still with Gluster
FUTURE WORK
LOCAL ITERATION: DEVPATCHERFrom a Day to Minutes
4
FARCRY3 Fixes Nightmare
2012 SUMMER
WHAT IF…
Detect CHANGES
Handles COMPILATION
Handles BIG FILE creation
Introducing
PATCHINGas a Workflow
3STAGES
DETECTING CHANGECompile Dep.
PACKAGING CHANGEPatch BigFile
APPLYING CHANGEBigFile Stack
CREATE PATCH
DETECTING CHANGES
INPUTS
1
COMPILE DEP.
PACKAGING CHANGES2
NIGHTLY BUILD
PACKAGING CHANGES
OK
OK
FAIL
2
APPLYING CHANGES?
3
RTPal Compile Dep. STACK FS
DEVPATCHER & RTPAL
DUNIA
1 FASTBuild
2 Fast Nightly Builds
3 RTPal & Asset Store
4 DevPatcher
The Pipeline
THANKS & CREDITS
Franta Fulin / FASTBuild (http://fastbuild.org/) Jean-Francois Cyr / DevPatcher Olivier Deschamps / DevPatcher Laurent Chouinard / AssetStore Jonhatan Chin / AssetStore Jocelyn Hotte / AssetStore