Creating a World Class Build System Scott Estes Release Management and Ops Citrix Systems, Inc.

10 June 2011

AGENDA

Getting Started Planning and Design Implementation and Execution Optimizing the System Maintenance and Evolution

GETTING STARTED

Creating and maintaining a world class build system involves much more than a few well-written scripts, e.g. •  Know the Stakeholders •  Know the Stakeholders’ requirements •  Have a vision •  Engage, foster and grow support base at all levels •  Be a “Center of Excellence”

GETTING STARTED – Evolution or Revolution?

Pros •  Slow, stable change •  Can keep pace

Cons •  Can be too slow •  Can lose momentum

Pros •  Mitigates barriers to change •  Can facilitate other changes

Cons •  Can be disruptive •  Unable to keep pace •  Unintended side-effects

GETTING STARTED – Evolution or Revolution?

Leverage both, •  Revolution in small doses •  Energy from revolution can fuel evolution

across the organization

GETTING STARTED – Build or Buy?

Depends on, •  Resources •  Timeline/schedule

GETTING STARTED – ONE LAST THING

More than likely there will not be enough resources within the Build team to accomplish the Build Vision. Use others to help...

PLANNING AND DESIGN

A great build system facilitates, •  “Determinism” •  Performance •  Parallel development •  Maintenance •  Business Continuity •  Self-service

PLANNING AND DESIGN – WHERE ARE THE TEAMS?

Knowing this will prevent support outages.

PLANNING AND DESIGN – WHAT IS THE PRODUCT?

Knowing the architecture ensures extensibility of the Vision. At Citrix: •  XenApp/XenDesktop use a Technology Component-

based architecture. •  Components treated as “black boxes”. •  Component visions are not tied to releases. •  Features are released to product when ready. •  Release process is much more efficient and

dynamic.

PLANNING AND DESIGN – DEVELOPER REQUIREMENTS

Iterative feedback ensures success. •  Developers are pulled in different directions; don’t add

additional stress. •  Provide an “Out of the box” Build experience. •  Support as needed. •  Build confidence in the System and Vision.

IMPLEMENTATION AND EXECUTION

At Citrix: Perforce Linked Automated Network Builder (PLANB) •  Determines when to build. •  Creates Build environment. •  Builds component binaries. •  Creates installers. •  Publishes product builds.

Two types of Builds, •  SDK •  DVD

IMPLEMENTATION AND EXECUTION - PLANB

Overview:

Watch Create environment

Build binaries

Publish SDK

Create DVD

Publish/ Test

Svr 1 Svr 2 Svr n

Perforce

Save Area

IMPLEMENTATION AND EXECUTION - BUILDING

Most components can follow these steps:

Input Pre-Build Steps Build Post-Build

Steps Output

With these attributes: •  Owner(s) •  Stakeholder(s) •  Input(s) •  Control(s) •  Output(s)

IMPLEMENTATION AND EXECUTION - INPUT

Anything that causes a new build, e.g. code submission, SDK update, etc. At Citrix: A Watcher service is used to receive Perforce change notifications and start the build.

Perforce

Project config

changes?

Get changes

Call VM Creator

Update client specs

Cb

Yes

No Change received

IMPLEMENTATION AND EXECUTION – PRE-BUILD

Configure the build environment. At Citrix: A VM creator service dynamically creates virtual build machines using XenServer® technologies.

VM available

?

Wait for available

VM

Start build script

Connect drives to

VM

Cb Yes

No

Determine VM type

Assign VM to host

IMPLEMENTATION AND EXECUTION – BUILD

Developer reviews are critical. At Citrix: An add-on to Visual Studio provides an “out of the box” experience while preserving company mandates.

Built?

Log error and

continue

Create output

Yes

No

ID what to build

Sign, index

symbols, etc.

Get code

Perforce

Conf. proj/

search

Check/ Build deps Compile

Done?

Yes No

IMPLEMENTATION AND EXECUTION – BUILD

At Citrix: Non-Windows Builds are make-based.

Built?

Create output

Yes

No

Search for makefiles

Sign, index

symbols, etc.

Get code

Perforce

Compile/ Link

Done?

Yes No

Log error and die

IMPLEMENTATION AND EXECUTION – POST-BUILD

Create a testable image and supporting build artifacts. At Citrix: A “toolbox” of PowerShell scripts create the DVD and releases the build for testing.

Poll for changes

Create reports

Release to Test

Symbol Server

Layout config

Get layout

info

Update Symbol server

IMPLEMENTATION AND EXECUTION – OUTPUT

Test the product and report. At Citrix: A test automation framework executes Build Verification Tests and other test suites.

Execute BVT

Execute additional

tests Report Provision

test env

Skynet

Pass?

Report

Test config

Yes

No

OPTIMIZING THE SYSTEM

A World-Class build system optimizes, •  “Determinism” •  Performance •  Parallel development •  Maintenance •  Business Continuity •  Self-service

A great build system facilitates, •  “Determinism” •  Performance •  Parallel development •  Maintenance •  Business Continuity •  Self-service

•  Standards

OPTIMIZING THE SYSTEM – WHAT’S IN THE BUILD?

Accurate, fast reporting, i.e. “Determinism”. At Citrix: •  Multiple reports based on audience,

•  What binaries changed Test •  What code changed Dev/Release Mgmt/

Test

OPTIMIZING THE SYSTEM – WHAT’S IN THE BUILD?

Accurate, fast reporting, i.e. “Determinism”. At Citrix: •  Multiple reports based on audience,

•  What binaries changed Test •  What code changed Dev/Release Mgmt/

Test

OPTIMIZING THE SYSTEM – LUDICROUS SPEED!

Do, •  Facilitate Code Coverage

efforts. •  Leverage incremental

building and architecture. •  Use dedicated Perforce

Proxies. •  Use as fast hardware as

can afford. •  Isolate Build network. •  Review Business

Continuity processes.

Don’t, •  Compile code multiple

times. •  Compile dead code. •  Ignore dependencies. •  Ignore the infrastructure. •  Ignore Business

Continuity.

Build faster than can be consumed.

OPTIMIZING THE SYSTEM

At Citrix: a topographical overview: UK

Cam

Devs

US

Business  Continuity  Center

Build  Farm

NAS

1GB

10GB

Servers

Build  machines

P4  ProxyDataP4  Proxy

Perforce  Svr  1

Builds  NAS

Perforce  Svr  2

Devs

1GB

Perforce  Svr1/Proxy  ports

Servers

Build  machines  (+BC  for  Chf)

Chf

Devs

Servers

Build  machines(+BC  for  Cam)

1GB

Perforce  Svr1/Proxy  ports

Builds  NAS(+BC  for  Chf)

Builds  NAS(+BC  for  Cam)

MAINTENANCE AND EVOLUTION

How to do it: •  Stage updates •  Automate, automate, automate! •  Test to trust •  Keep everyone updated

When to do it: •  Align with project milestones and schedules •  Associate with a specific project

TAKEAWAYS

•  Have a vision •  Enable instead of mandate. •  Talk to others and back it up. •  Provide a simple and maintainable

infrastructure.