mesoscon asia keynote: replacing a jet engine mid-flight
TRANSCRIPT
How to Replace a Jet Engine of Your System In-Flight
Aysylu Greenberg, Software Engineer, Google
AysyluGreenberg@aysylu22
BuildSystemStoryPhoto by imanka / CC BY
TowardsDistributedBuildSystem
TowardsDistributedBuildSystem
TowardsDistributedBuildSystem
TowardsDistributedBuildSystem
TowardsDistributedBuildSystem
TowardsDistributedBuildSystem
Continuous integration
system
BuildRabbit
Source System
Google engineers &
teams
Release infrastructure
Integration testing
infrastructure
Build artifact storage
Blaze
Scale• Engineers:>30,000developersin40+offices
Scale• Engineers:>30,000developersin40+offices
• Commits:15Kbyhumans+30Kbyrobots/day
Scale• Engineers:>30,000developersin40+offices
• Commits:15Kbyhumans+30Kbyrobots/day
• Sourcecode:2billionLOC
Scale• Engineers:>30,000developersin40+offices
• Commits:15Kbyhumans+30Kbyrobots/day
• Sourcecode:2billionLOC• Buildsandtests:5MperdaythroughBuildRabbit
Scale• Engineers:>30,000developersin40+offices
• Commits:15Kbyhumans+30Kbyrobots/day
• Sourcecode:2billionLOC• Buildsandtests:5MperdaythroughBuildRabbit
• PetabytesofoutputarRfacts
ArchitectureUpgradehttps://thetravelintern.com/hangzhou-things-to-do/
ArchitectureUpgradePhoto by vermazeren / CC BY
Continuous integration
system
BuildRabbit
Source System
Google engineers &
teams
Release infrastructure
Integration testing
infrastructure
Build artifact storage
Blaze
HowtoReplaceaJetEngineofyourSystem
In-Flight
https://en.wikipedia.org/wiki/File:Helicopter_aerial_refueling.jpg
MigraRonwithZeroDownRmePhoto by moonjazz / CC BY SA
OldArchitecture:Client/Server
Client for User
BuildRabbit Scheduler
BuildRabbit Worker
NewArchitecture:BuildService
Client for
User
Persistent Queue
BuildRabbit Worker
Build Artifacts Build Progress
Info
event RPC stream
Persistent Queue
BuildRabbit Worker
Build Artifacts Build Progress
Info
event RPC stream
NewArchitecture:BuildService
Client for
User
Client for
User
BuildRabbit Scheduler
BuildRabbit Worker
Client for
User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
BuildRabbit Worker
Build Progress Info Build Artifacts
Client for
User
Photo by goosmurf / CC BY
FOCUSONINTERMEDIATESTATE
Client for User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
Photo by goosmurf / CC BY
FOCUSONINTERMEDIATESTATE
Client for
User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
BuildRabbit Worker
Build Progress Info Build Artifacts
Client for
User
Photo by goosmurf / CC BY
FOCUSONINTERMEDIATESTATE
Client for
User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
BuildRabbit Worker
Build Progress Info Build Artifacts
Client for
User
Photo by goosmurf / CC BY
FOCUSONINTERMEDIATESTATE
ROLLOUTINCREMENTALLY
Photo by firepile / CC BY
ROLLOUTINCREMENTALLY
Photo by firepile / CC BY
Client for
User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
BuildRabbit Worker
Build Progress Info Build Artifacts
Client for
User
ROLLOUTINCREMENTALLY
Photo by firepile / CC BY
PRACTICEROLLOUT
Photo by greg_photos / CC BY SA
Client for User
BuildRabbit Scheduler
BuildRabbit Worker
Persistent Queue
PRACTICEROLLOUT
Photo by greg_photos / CC BY SA
• Topreventsimplestmistakes
• For organization resilience
PRACTICEROLLOUT
FOCUSONINTERMEDIATESTATE
ROLLOUTINCREMENTALLY
How to Replace a Jet Engine of Your System In-Flight
Aysylu Greenberg, Software Engineer, Google