taming ios testing at square -- juc west 2015

49
Taming iOS Testing at Square Santa Clara, CA, September 2-3 1 Michael Tauraso, Sr. Release Engineer

Upload: michael-tauraso

Post on 10-Jan-2017

897 views

Category:

Technology


1 download

TRANSCRIPT

Taming iOS Testing at SquareSanta Clara, CA, September 2-3

1

Michael Tauraso, Sr. Release Engineer

Footer

#jenkinsconfWhat do I do?

CI + Release automation for mobile app developers

I don’t write the tests I don’t know Objective-C or Swift

I’m from a Linux + Devops background

2

Footer

#jenkinsconfWhat does Square do?

Square has 4 iOS apps Dozens of iOS developers Several internal iOS apps

3

Footer

#jenkinsconfWhat’s different about iOS?

• Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem

4

Footer

#jenkinsconfWhat’s different about iOS?

5

October November December January

February March April May

June July August September

6.1

6.2 6.3

6.4 7.0?

Footer

#jenkinsconfWhat’s different about iOS?

• Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem

6

Footer

#jenkinsconfiOS CI @ Square 1 year ago

• 30 minute builds, longer queue times • Manual upgrades, no scale plans • Mysterious build failures • Jenkins difficult to manage • Lots of developer support + firefighting

7

Footer

#jenkinsconfHow do you make it better?

• Assume change will happen • Know exactly what’s changing • Control all the variables • Deliver repeatable builds • Keep up with Apple

8

Footer

#jenkinsconfWhat are we going to talk about?

• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems

9

Footer

#jenkinsconfWhat are we going to talk about?

• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems

10

Footer

#jenkinsconfMobile Jenkins @Square

11

Git (Stash) Stashkins Jenkins iOS-builder

Checkout Cocoapods Xcodebuild

iOS Simulator

Footer

#jenkinsconfStashkins

• Uses disabled jobs as templates • Substitutes values in Jenkins job XML • Runs off a config file in the app’s repo • Triggered on git-push

12

Footer

#jenkinsconfJenkins jobs

• Check out ios-builder on a build slave • Run ios-builder • Display logs, results, and artifacts

13

Footer

#jenkinsconfJenkins Plugins

• Promoted builds plugin • Send packaged .ipa files to OTA site

• Warnings parser plugin • JUnit test results plugin • Swarm plugin

14

Footer

#jenkinsconfRelease Code

• Pattern: Factor release code to another repo • Tweak things without touching Jenkins • Devs don’t need/want Jenkins write access

• Easier management • Devs edit configs on their branch

• CI + Code Review release code!

15

Footer

#jenkinsconfExample Config

16

Footer

#jenkinsconfExample Config

17

Footer

#jenkinsconfXcodebuild Needs

• Project/Workspace • Scheme • Configuration • Destination • Action

• build, test, and archive

18

Footer

#jenkinsconfUseful Xcodebuild

• Archive is not build • Uses more exact header search paths • SKIP_INSTALL Project setting

• Use xcodebuild -exportArchive for ipa files • Flag reference:

• git.io/vGDOd

19

Footer

#jenkinsconfXcodebuild Friends

• Loose coupling • Frontends other than ios-builder

• facebook/xctool • fastlane/gym

• Log output • supermarin/xcpretty

20

Footer

#jenkinsconfWhat are we going to talk about?

• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems

21

Footer

#jenkinsconfBuild slaves

• Virtualization :( • Hardware

• Dual core Mac-Minis • 16 GB RAM • 256 GB SSD • $1200 each

22

Footer

#jenkinsconfBuild slaves - DeployStudio

• Network boot • Base image with OS X 10.10 • Installs Xcode

23

Footer

#jenkinsconfBuild slaves - Ansible

• Basic config management • Machines ansible themselves • Allows easy config changes • Multiple Xcode Versions • Swarm plugin

• Jenkins remoting version

24

Footer

#jenkinsconfBuild slaves - OSX

• Match versions with Xcode • Never sleep (caffeinate command) • Turn off:

• Software update • spotlight • bonjour • ssh w/password + VNC

25

Footer

#jenkinsconfBuild slaves - Xcode

• Always have a GUI context! • Automatic login • swarm as a LaunchAgent

• TCC Database • git.io/osx-tcc

• DevToolsSecurity

26

Footer

#jenkinsconfBuild slaves - Xcode Packaging

• Xcode likes to install things on first-startup • iOS SDKs installed via Xcode • Command line tools are another package • We record installs with Composer

• (JAMF software)

• Side-by-side xcode versions with jenkins labels + xcode-select

27

Footer

#jenkinsconfBuild slaves - More detail

• https://corner.squareup.com/2015/07/ios-build-infrastructure.html

28

Footer

#jenkinsconfWhat are we going to talk about?

• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems

29

Footer

#jenkinsconfCode signing

• Its confusing • The error messages are unclear • Apple changes things

• Apple Watch! • Swift!

• Your app needs it to run

30

Footer

#jenkinsconfCode signing

• Provisioning Profiles

• Signing Identity

31

Footer

#jenkinsconfCode signing - Mobileprovision

• Message from Apple to iPhone • These devices • Can run this app (app ID) • With these permissions (entitlements)

• So long as ( ) signed the code

32

Footer

#jenkinsconfCode signing - Mobileprovision

• is a plist in CMS format

• Signing identities are just x509 certs • emonti/pliney • Parse + check for developer errors

33

Footer

#jenkinsconfCode signing - Xcode

• Xcode “helps” • Restrict Mobileprovision directory in CI

• ~/Library/MobileDevice/Provisioning Profiles

• Apple Watch apps • Multiple mobileprovision files • All on the same signing identity

34

Footer

#jenkinsconfCode signing - HSM

• Yubikey-NEO • Acts as an HSM • Signing identity available through keychain • Key never touches a hard-drive

35

Footer

#jenkinsconfCode signing - Yubikeys

• Can create on-device or import existing • Needs OpenSC (+reboot)

• OpenSC/OpenSC

• Needs yubikey-piv-tool • https://developers.yubico.com/yubico-piv-tool/Releases/

• Tutorial • https://developers.yubico.com/yubico-piv-tool/

OS_X_code_signing.html

36

Footer

#jenkinsconfWhat are we going to talk about?

• How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems

37

Footer

#jenkinsconfBuild Stability

• Dozens of dev make 100s of builds • 1 in 100 failures happen every day • Builds that work sometimes are frustrating • Green builds usually result in code merging • “Sometimes” errors accumulate

38

Footer

#jenkinsconfBuild Stability

• Run passing builds at night! • Use our idle infrastructure • How green are they really?

39

Footer

#jenkinsconfBuild Stability — Statistics

• Builds are very unfair coins • Binomial distribution • Wilson score for confidence interval

40

Footer

#jenkinsconf

Build Stability

41

2 failures 300 builds

Footer

#jenkinsconfBuild Stability — Statistics

• Count individual failure types • Allocate investigative resources • Keep an SLA (99.5%)

42

Footer

#jenkinsconf

Build Stability

43

2 failures 300 builds 0.66%

6 failures 900 builds 0.66%

Footer

#jenkinsconfBuild Stability — Statistics

• Found Xcode bugs • Reduced failure rate to 0.5% • More detail:

• https://corner.squareup.com/2015/06/build-stability.html

44

Footer

#jenkinsconfWhere are we now?

• 3x as many tests, lower build time • 99.5% SLA. We file bugs for bad tests • Many developer requests are self-serve • Growing cluster with developer hiring • Supporting latest Apple hardware

45

#jenkinsconf

Footer

Thanks to our Sponsors!

46

Footer

#jenkinsconfQuestions?

Michael Tauraso

mtauraso@{gmail,squareup}.com @mtauraso mtauraso

47

#jenkinsconf

Footer

Thanks to our Sponsors!

48

Taming iOS TestingSanta Clara, CA, September 2-3

49