working well together: how to keep high-end game development teams productive

#

Frank CompagnerGuerrilla Games

Working Well Together

#

• Background• Iteration• Teamwork• Stability• Releasing

Overview

Iterate

Launch

How we learned to stop worrying and love the development cycle

#

• Killzone Shadow Fall – PS4 launch title• 200 local users, 150 remote• Head revision: 1.8 TB (1.6 M files)• Depot size: 8 TB• Peak change: 180 GB / week• 60 dedicated build servers• All in Perforce since 2004

Background

#

P4 setup - nowServer

Build Farm (60)

Proxy Proxy Proxy Proxy

Users (200)

Proxy

Proxy

Proxy

Proxy

Cambridge

Guilford

San Diego

ShanghaiAmsterdam

#

P4 setup - soonCommit

Server

Build Farm (60) Users (200)

Proxy

Proxy

Proxy

Cambridge

Guilford

San Diego

ShanghaiAmsterdam

Edge

Server

Edge

Server

Edge

Server

#

P4 scale

Submits GB / week

#

• Monitors all vital stats of server and proxies• Simple Python script, produces simple html• Works for Windows and Linux servers

• Proxies cleaned by another Python script

• Both available in the Perforce Workshop

P4 monitor

#

P4 monitor

#

Iterate, Iterate, Iterate

#

Code

Compile

Link

Test

Creation means Iteration

Programmer

#

Code

Compile

Link

Test

Creation means Iteration

Programmer

IDE and Tools

Distribute

Incremental linker

CodeHotloading

#

Model

ProcessTest

Creation means Iteration

Artist

Hotloading

#

Model

ProcessTest

Creation means Iteration

Artist

Tools

DistributeHotloading

#

Programmer

Rigger

AnimatorDesigner

Tester

Creation means Iteration

Team

#

Lots of Iteration

Programmer

Rigger

Animator

Animator

Designer

Tester

Programmer

Modeler

Programmer

VFX artist

Designer

Audio Engineer

Modeler

Designer

Animator

#

Teamwork

#

• Does not look like this

Games development

#

• It’s more like this

Games development

#

• OK, like this

Games development

#

• Personal branch is fine (sandbox / streams / Git)

• Feature branches sometimes work• Give control over your environment• But create distance to others• Team work requires frequent branch switching

Can branches restore order?

#

• For us, branch switching is always expensive:– Amount of data and change– Unmergeable files– Code-Data dependencies

• Branches add complexity, we already have that• Team effort, we don’t want distance• Almost everybody works on the trunk• So why isn’t it always broken?

TBD - light

#

Stability

#

Programmer

Rigger

AnimatorDesigner

Tester

Creation means Iteration

Team

#

Programmer

Rigger

AnimatorDesigner

Tester

Creation means Iteration

Team

Submit

Submit

Submit

Submit

Submit Sync

Sync

Sync

Sync

Sync

#

Programmer

Rigger

AnimatorDesigner

Tester

Creation needs Testing

Team

Submit

Submit

Submit

Submit

Submit Sync

Sync

Sync

Sync

SyncTest

Test

Test

Test

Test

#

• Test pre submit (user)• Test post submit (builder)• Test in parallel• Test as fast as possible• Still accidents do happen

StabilityUser B

Sync

< 30 minutes

User A

Submit Sync

Build Game SubmitSync

Test Game

…

Build Tools SubmitSync

Test GameSync

Cook DataSync

Sync

Upload

Cook Data Upload

Test

#

…Labels.xml

Build labels

Label 42: change 1200

write

Build machines

Label 41: change 1180

Safe-Sync Tool

read

Label 40: change 1155

Label 39: change 1150

• Each game build creates a label• Stored in Labels.xml file in depot• Builds add their result to the label• Builds can add artifacts to a label

• Nobody gets latest (not safe)• Everybody uses safe-sync tool

#

Safe Sync ToolSelecttests

Labels thatpass tests

Labels thatfail tests

• Custom safe-sync tool shows labels and test results• Use it to sync to a known good label

– This includes artifacts from other builds

Sync to get label

#

Releasing

#

• We branch for every release (often every week)• Nobody works on the release branch• Everybody works on the trunk• Changes are merged up from main to release

Release Branches are great!

main

release

= change

= release

#

Release Manager• Cherry picking rules!• Gives you a lot of control• But also lots of confusion• And mistakes are easy

• Good tools can fix this• Give flexibility• And control

(See whitepaper for details)

#

Conclusion• Iterate as fast as you can• Don’t branch too much• Test everything• Sync should be safe

• Release all the time• Control what you release

Speed

Control

##

Thank you!Frank Compagnerfrank@guerrilla-games.com

#

RESOURCESPerforce Workshop (P4 monitor, cache cleaner):

https://swarm.workshop.perforce.com/files/guest/frank_compagner

ReleaseManager:

http://www.perforce.com/resources/presentations/user-conference-talks/tools-for-smooth-release-cycle

http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf