taming ios testing at square -- juc west 2015
TRANSCRIPT
Footer
#jenkinsconfWhat do I do?
CI + Release automation for mobile app developers
I don’t write the tests I don’t know Objective-C or Swift
I’m from a Linux + Devops background
2
Footer
#jenkinsconfWhat does Square do?
Square has 4 iOS apps Dozens of iOS developers Several internal iOS apps
3
Footer
#jenkinsconfWhat’s different about iOS?
• Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem
4
Footer
#jenkinsconfWhat’s different about iOS?
5
October November December January
February March April May
June July August September
6.1
6.2 6.3
6.4 7.0?
Footer
#jenkinsconfWhat’s different about iOS?
• Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem
6
Footer
#jenkinsconfiOS CI @ Square 1 year ago
• 30 minute builds, longer queue times • Manual upgrades, no scale plans • Mysterious build failures • Jenkins difficult to manage • Lots of developer support + firefighting
7
Footer
#jenkinsconfHow do you make it better?
• Assume change will happen • Know exactly what’s changing • Control all the variables • Deliver repeatable builds • Keep up with Apple
8
Footer
#jenkinsconfWhat are we going to talk about?
• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems
9
Footer
#jenkinsconfWhat are we going to talk about?
• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems
10
Footer
#jenkinsconfMobile Jenkins @Square
11
Git (Stash) Stashkins Jenkins iOS-builder
Checkout Cocoapods Xcodebuild
iOS Simulator
Footer
#jenkinsconfStashkins
• Uses disabled jobs as templates • Substitutes values in Jenkins job XML • Runs off a config file in the app’s repo • Triggered on git-push
12
Footer
#jenkinsconfJenkins jobs
• Check out ios-builder on a build slave • Run ios-builder • Display logs, results, and artifacts
13
Footer
#jenkinsconfJenkins Plugins
• Promoted builds plugin • Send packaged .ipa files to OTA site
• Warnings parser plugin • JUnit test results plugin • Swarm plugin
14
Footer
#jenkinsconfRelease Code
• Pattern: Factor release code to another repo • Tweak things without touching Jenkins • Devs don’t need/want Jenkins write access
• Easier management • Devs edit configs on their branch
• CI + Code Review release code!
15
Footer
#jenkinsconfXcodebuild Needs
• Project/Workspace • Scheme • Configuration • Destination • Action
• build, test, and archive
18
Footer
#jenkinsconfUseful Xcodebuild
• Archive is not build • Uses more exact header search paths • SKIP_INSTALL Project setting
• Use xcodebuild -exportArchive for ipa files • Flag reference:
• git.io/vGDOd
19
Footer
#jenkinsconfXcodebuild Friends
• Loose coupling • Frontends other than ios-builder
• facebook/xctool • fastlane/gym
• Log output • supermarin/xcpretty
20
Footer
#jenkinsconfWhat are we going to talk about?
• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems
21
Footer
#jenkinsconfBuild slaves
• Virtualization :( • Hardware
• Dual core Mac-Minis • 16 GB RAM • 256 GB SSD • $1200 each
22
Footer
#jenkinsconfBuild slaves - DeployStudio
• Network boot • Base image with OS X 10.10 • Installs Xcode
23
Footer
#jenkinsconfBuild slaves - Ansible
• Basic config management • Machines ansible themselves • Allows easy config changes • Multiple Xcode Versions • Swarm plugin
• Jenkins remoting version
24
Footer
#jenkinsconfBuild slaves - OSX
• Match versions with Xcode • Never sleep (caffeinate command) • Turn off:
• Software update • spotlight • bonjour • ssh w/password + VNC
25
Footer
#jenkinsconfBuild slaves - Xcode
• Always have a GUI context! • Automatic login • swarm as a LaunchAgent
• TCC Database • git.io/osx-tcc
• DevToolsSecurity
26
Footer
#jenkinsconfBuild slaves - Xcode Packaging
• Xcode likes to install things on first-startup • iOS SDKs installed via Xcode • Command line tools are another package • We record installs with Composer
• (JAMF software)
• Side-by-side xcode versions with jenkins labels + xcode-select
27
Footer
#jenkinsconfBuild slaves - More detail
• https://corner.squareup.com/2015/07/ios-build-infrastructure.html
28
Footer
#jenkinsconfWhat are we going to talk about?
• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems
29
Footer
#jenkinsconfCode signing
• Its confusing • The error messages are unclear • Apple changes things
• Apple Watch! • Swift!
• Your app needs it to run
30
Footer
#jenkinsconfCode signing - Mobileprovision
• Message from Apple to iPhone • These devices • Can run this app (app ID) • With these permissions (entitlements)
• So long as ( ) signed the code
32
Footer
#jenkinsconfCode signing - Mobileprovision
• is a plist in CMS format
• Signing identities are just x509 certs • emonti/pliney • Parse + check for developer errors
33
Footer
#jenkinsconfCode signing - Xcode
• Xcode “helps” • Restrict Mobileprovision directory in CI
• ~/Library/MobileDevice/Provisioning Profiles
• Apple Watch apps • Multiple mobileprovision files • All on the same signing identity
34
Footer
#jenkinsconfCode signing - HSM
• Yubikey-NEO • Acts as an HSM • Signing identity available through keychain • Key never touches a hard-drive
35
Footer
#jenkinsconfCode signing - Yubikeys
• Can create on-device or import existing • Needs OpenSC (+reboot)
• OpenSC/OpenSC
• Needs yubikey-piv-tool • https://developers.yubico.com/yubico-piv-tool/Releases/
• Tutorial • https://developers.yubico.com/yubico-piv-tool/
OS_X_code_signing.html
36
Footer
#jenkinsconfWhat are we going to talk about?
• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems
37
Footer
#jenkinsconfBuild Stability
• Dozens of dev make 100s of builds • 1 in 100 failures happen every day • Builds that work sometimes are frustrating • Green builds usually result in code merging • “Sometimes” errors accumulate
38
Footer
#jenkinsconfBuild Stability
• Run passing builds at night! • Use our idle infrastructure • How green are they really?
39
Footer
#jenkinsconfBuild Stability — Statistics
• Builds are very unfair coins • Binomial distribution • Wilson score for confidence interval
40
Footer
#jenkinsconfBuild Stability — Statistics
• Count individual failure types • Allocate investigative resources • Keep an SLA (99.5%)
42
Footer
#jenkinsconfBuild Stability — Statistics
• Found Xcode bugs • Reduced failure rate to 0.5% • More detail:
• https://corner.squareup.com/2015/06/build-stability.html
44
Footer
#jenkinsconfWhere are we now?
• 3x as many tests, lower build time • 99.5% SLA. We file bugs for bad tests • Many developer requests are self-serve • Growing cluster with developer hiring • Supporting latest Apple hardware
45