Creating a World Class Build SystemScott EstesRelease Management and OpsCitrix Systems, Inc.

April 12, 2023

AGENDA

Getting Started

Planning and Design

Implementation and Execution

Optimizing the System

Maintenance and Evolution

GETTING STARTED

Creating and maintaining a world class build system involves much more than a few well-written scripts, e.g.

• Know the Stakeholders• Know the Stakeholders’ requirements• Have a vision • Engage, foster and grow support base at all levels• Be a “Center of Excellence”

GETTING STARTED – Evolution or Revolution?

Pros• Slow, stable change• Can keep pace

Cons• Can be too slow• Can lose momentum

Pros• Mitigates barriers to change• Can facilitate other changes

Cons• Can be disruptive• Unable to keep pace• Unintended side-effects

GETTING STARTED – Evolution or Revolution?

Leverage both,• Revolution in small doses• Energy from revolution can fuel evolution

across the organization

GETTING STARTED – Build or Buy?

Depends on,

• Resources• Timeline/schedule

GETTING STARTED – ONE LAST THING

More than likely there will not be enough resources within the Build team to accomplish the Build Vision.

Use others to help...

PLANNING AND DESIGN

A great build system facilitates,

• “Determinism”• Performance• Parallel development• Maintenance• Business Continuity• Self-service

PLANNING AND DESIGN – WHERE ARE THE TEAMS?

Knowing this will prevent support outages.

PLANNING AND DESIGN – WHAT IS THE PRODUCT?

Knowing the architecture ensures extensibility of the Vision.

At Citrix:• XenApp/XenDesktop use a Technology Component-

based architecture.• Components treated as “black boxes”.• Component visions are not tied to releases.• Features are released to product when ready.• Release process is much more efficient and

dynamic.

PLANNING AND DESIGN – DEVELOPER REQUIREMENTS

Iterative feedback ensures success.

• Developers are pulled in different directions; don’t add additional stress.

• Provide an “Out of the box” Build experience.• Support as needed.• Build confidence in the System and Vision.

IMPLEMENTATION AND EXECUTION

At Citrix:

Perforce Linked Automated Network Builder (PLANB)

• Determines when to build.• Creates Build environment.• Builds component binaries.• Creates installers.• Publishes product builds.

Two types of Builds,• SDK• DVD

IMPLEMENTATION AND EXECUTION - PLANB

Overview:

WatchCreate

environmentBuild

binariesPublish

SDKCreate DVD

Publish/Test

Svr 1 Svr 2 Svr n

Perforce

Save Area

IMPLEMENTATION AND EXECUTION - BUILDING

Most components can follow these steps:

InputPre-Build

StepsBuild

Post-Build Steps

Output

With these attributes:• Owner(s)• Stakeholder(s)• Input(s)• Control(s)• Output(s)

IMPLEMENTATION AND EXECUTION - INPUT

Anything that causes a new build, e.g. code submission, SDK update, etc.

At Citrix: A Watcher service is used to receive Perforce change notifications and start the build.

Perforce

Project config

changes?

Get changes

Call VM Creator

Update client specs

Cb

Yes

No

Change received

IMPLEMENTATION AND EXECUTION – PRE-BUILD

Configure the build environment.

At Citrix: A VM creator service dynamically creates virtual build machines using XenServer® technologies.

VM available

?

Wait for available

VM

Start build script

Connect drives to

VM

Cb

Yes

No

Determine VM type

Assign VM to host

IMPLEMENTATION AND EXECUTION – BUILD

Developer reviews are critical.

At Citrix: An add-on to Visual Studio provides an “out of the box” experience while preserving company mandates.

Built?

Log error and

continue

Create output

Yes

No

ID what to build

Sign, index

symbols, etc.

Get code

Perforce

Conf.proj/

search

Check/Build deps

Compile

Done?

YesNo

IMPLEMENTATION AND EXECUTION – BUILD

At Citrix: Non-Windows Builds are make-based.

Built?

Create output

Yes

No

Search for makefiles

Sign, index

symbols, etc.

Get code

Perforce

Compile/Link

Done?

YesNo

Log error and die

IMPLEMENTATION AND EXECUTION – POST-BUILD

Create a testable image and supporting build artifacts.

At Citrix: A “toolbox” of PowerShell scripts create the DVD and releases the build for testing.

Poll for changes

Create reports

Release to Test

Symbol Server

Layout config

Get layout

info

UpdateSymbolserver

IMPLEMENTATION AND EXECUTION – OUTPUT

Test the product and report.

At Citrix: A test automation framework executes Build Verification Tests and other test suites.

Execute BVT

Execute additional

testsReport

Provision test env

SkynetSkynet

Pass?

Report

Test config

Yes

No

OPTIMIZING THE SYSTEM

A World-Class build system optimizes,

• “Determinism”• Performance• Parallel development• Maintenance• Business Continuity• Self-service

A great build system facilitates,

• “Determinism”• Performance• Parallel development• Maintenance• Business Continuity• Self-service

• Standards

OPTIMIZING THE SYSTEM – WHAT’S IN THE BUILD?

Accurate, fast reporting, i.e. “Determinism”.

At Citrix:• Multiple reports based on audience,

• What binaries changed Test• What code changed Dev/Release

Mgmt/ Test

OPTIMIZING THE SYSTEM – LUDICROUS SPEED!

Do,• Facilitate Code Coverage

efforts.• Leverage incremental

building and architecture.• Use dedicated Perforce

Proxies.• Use as fast hardware as

can afford.• Isolate Build network.• Review Business

Continuity processes.

Don’t,• Compile code multiple

times.• Compile dead code.• Ignore dependencies.• Ignore the infrastructure.• Ignore Business

Continuity.

Build faster than can be consumed.

OPTIMIZING THE SYSTEM

At Citrix: a topographical overview:UK

Cam

Devs

US

Business Continuity Center

Build Farm

NAS

1GB

10GB

Servers

Build machines

P4 ProxyDataP4 Proxy

Perforce Svr 1

Builds NAS

Perforce Svr 2

Devs

1GB

Perforce Svr1/Proxy ports

Servers

Build machines (+BC for Chf)

Chf

Devs

Servers

Build machines(+BC for Cam)

1GB

Perforce Svr1/Proxy ports

Builds NAS(+BC for Chf)

Builds NAS(+BC for Cam)

MAINTENANCE AND EVOLUTION

How to do it:• Stage updates• Automate, automate, automate!• Test to trust• Keep everyone updated

When to do it:• Align with project milestones and schedules• Associate with a specific project

TAKEAWAYS

• Have a vision• Enable instead of mandate.• Talk to others and back it up.• Provide a simple and maintainable

infrastructure.