Speed to Market in Mobile Development Finding the right solution with continuous

integration on real devices

Tina Su, Director of Development, Intuit

About me

Nearly 20 years of industry experience in R&D, quality and engineering management, with a strong passion for automation.

§  Engineering Director at Intuit, Developer Productivity §  Sr. Engineering Director at Yahoo, Mobile Platforms §  Sr. Engineering Manager at Sun, Java Standard and

Mobile Editions §  Compiler Developer at MIPS and Tandem

Tina Su, Intuit

Agenda

§  Mobile trends & developers’ challenges §  Speed & quality through continuous integration -  Intuit Virtual Device Lab -  Intuit mobile test automation

§  Takeaways

Aggressive mobile momentum

“2013 Internet Trends,” KPCB

Will surpass desktop by 2014

It should just work!

Proliferation of platforms, OS versions & device types

Developer challenges

§  Immense configuration matrix -  OS platforms (Android, iOS, Windows, phone vs. tablet, …)

-  Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)

-  Outputs (portrait/landscape, screen resolution, language)

§  Application delivery through the markets -  App signing, provision (app & device), submission

§  Offline experience -  Network, different network bandwidth/latency, no-network

-  HTML5 offline app

§  Media challenges -  Some codecs don’t work on a device

-  Streaming, sound

§  Security & privacy

Intuit mobile app portfolio today

Apple

Google

Amazon

SMS

0 5 10 15 20 490

22

15

11

3

491 mobile apps for financial institutions

Ramping up Intuit’s innovation engine

Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one … QA can’t keep up with the changes

Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices

Strategy: Speed & quality through mobile continuous integration

•  iPhone 4 •  iPhone 4S •  iPhone 5 •  iPad 2 •  iPad 3 •  iPad Mini

Devices OS versions

•  iOS 5.0 •  iOS 5.1 •  iOS 6.0 •  iOS 6.1.x •  iOS 7.0

Builds

X X

Daily builds

•  Apple •  Google •  Amazon

Platforms

X

Continuous integration (CI) What is CI? §  Automate build, test & deployment process §  Integrate & build early, often, daily, for every check-in §  Best practice of Agile development §  Jenkins is widely adopted, flexible & extensible

Benefits of CI in a mobile world §  Instant feedback on quality, functionality & system impact §  Less complex, no integration hell §  Detect and address device configuration issues early §  Involve stakeholders early §  Ultimately achieve shippable release any time

Speed and quality through mobile CI

Key enablers:

•  Mobile device lab •  Test automation

Test processor Real devices

Jenkins

SCM: source repository

Developers/IDE

Automatic test run

Test results

Submit the app to app store if passed

Check in

Check-in triggers

automated CI

Install the test app to real devices in parallel

Run tests on real devices in parallel

Analyze test results

Trigger testing in mobile device lab

Process starts here

3rd-party vendor solutions didn’t work for us

§  Couldn’t handle E2E automation of testing pre-release apps -  Dependency on internal resources & infrastructure

§  Vendor lock-in: customized CI & test automation solutions -  No flexibility to use our existing tools & frameworks

§  Disadvantages of vendors’ shared devices -  Security concern on testing on shared devices -  No access to corporate network -  Cost-prohibitive to reserve devices

§  Private cloud (on-site setup) with local devices not practical -  With sensitive customer data, we need private cloud -  Extensive use of devices for daily testing of hundreds of mobile apps is expensive

Why test on real devices?

Most mobile features can be tested on simulators, but simulators have limitations:

§  Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density

§  Real-world experience over the real connection (3G, 4G, crappy Wi-Fi, etc.)

§  True performance, security tests §  Testing on non-default Web browsers (e.g., Opera Mini) §  Not all simulator/emulators have good quality

(don’t reflect real rendering/behavior, bugs) §  Testing SMS & app integration

Intuit Virtual Device Lab

Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser

Intranet

VDL

User machine

User machine

User machine

3-click access

Always on

Integrated with dev infrastructure

Globally available

iPhone iPad

Galaxy Captivate

Optimus Nexus One Desire Incredible

Manufacturers/���devices:

Droid Milestone Moto

VDL architecture – Android

Windows PC

Tomcat

Java image transferring application Browser &

Java applet

Internet/intranet

Device interaction app

Developer’s laptop

VDL

Image transfer

USB connection

VDL architecture – iOS

Mac with Xcode installed

Tomcat

Java VNC client

Port forwarding

Browser & Java applet

Internet/intranet

VNC server

USB connection

Developer’s laptop

VDL

Open source tools & technologies used in VDL

Key capabilities Android iOS

Installing mobile applications into the real device

adb install <APK file> Fruitstrap https://github.com/ghughes/fruitstrap

Transferring device screen images, controlling device

Screencast tool http://code.google.com/p/androidscreencast

Veency VNC app

Fetching device local files adb pull <filename> sftp command (jailbroken devices only)

VDL key features & functionality today

Control the device through standard browser/RESTful APIs

Change orientation to landscape or portrait

Copy local file to the device from your PC

Get file from the device onto your local PC

Execute an ADB command on the device

See current running logs in your browser

Download history logs to your local PC

Continuous integration – how it works

Product builds

Trigger file launches test

VDL device inventory Resource

manager

Select a device

Test execution engine

Results

Monitor for new builds

Mobile continuous integration

Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.

Submit, build, test … in minutes vs. hours

Select devices & tests 1 Tests run automatically across selected devices with each build

2 See the results! 3

Next step: over-the-air replaces USB connection

Fast onboarding

Scalable

Any device

Internet/intranet

App installation

Test execution

Results retrieval

VDL

Mobile Test Automation

Mobile test automation tools

Instrumentation-based

Non-instrumentation

Android iOS

MonkeyRunner

MOET

Sikuli

Android SDK

Robotium

TestDroid (BitBar) Calabash

(LessPainful)

Soasta

iOS SDK

UIAutomation

UISpec

Frank

MonkeyTalk

DA/PerfectoMobile

eggPlant Cross-platform solution

iOS Driver Appium

•  Assessed as of 2012

Mobile test automation trends

§  Abstraction with domain-specific language (DSL) -  Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET) -  Support behavior-driven development (BDD)

§  Client/server-based approach (remotely driven tests via HTTP) §  Instrumentation & non-instrumentation solutions complement

each other § Mobile cloud testing environment

-  From hardware (cradle) to software approach to drive tests on real devices

§  Automated OTA app installation & tests

Intuit automation framework of choice/recommendation

§ Native apps: -  Cucumber + Calabash for BDD on iOS & Android

§ Mobile Web: -  Selenium/WebDriver

What is Cucumber? It lets stakeholders (customers, QE or developers) describe how software should behave in plain text.

§ BDD: Written in business domain-specific language (DSL) -  Good abstraction over different device types

§ One format for multiple purposes -  UI feature specification -  Automated tests

§ Cucumber itself is NOT a test automation tool -  Add-on to any test automation framework -  Examples: Calabash, Sikuli & Frank mobile test frameworks

What is Calabash? § Open source cross-platform mobile test framework

-  iOS + Android, simulator + real device, native + hybrid

§ Supports Cucumber for BDD § Application needs instrumentation that embeds a Calabash HTTP server

-  Tests run from a remote client (in a Jenkins slave) -  Tests can be driven via Wi-Fi (no USB wire)

§ Ruby clients in API level (Java clients in experiment) § Works better in touch simulation & real device testing § Mobile cloud test execution available by LessPainful

Intuit shared test library – rapid test development

Internal open source repository

Teams focus on creating

Use shared test lib

… and then contribute to common test library

Tests

Contribute

Tests

Common test library Product tests

Tests Tests

Globalization test framework (I18N/L10N)

Performance/load test framework

Security test framework

Native/hybrid test framework

High-level test domain-specific language (Cucumber steps)

Mobile back-end

Mobile front-end

Code build

Deploy

Test

Results

Connect device

Notify email

Get devices

Consolidate report

Upload report

From: manual

50*5 = 250

15 30

5

10

20

15

30 10

= 6 hours

Code build

Trigger Execute tests in parallel

Consolidated report in archive

Deploy on devices

0 5 10 0

= 15 mins

To: automated CI

TIME SAVED: 95% Release quality increased

*

Stunning results: reduced iteration cycle from hours to minutes

Tested on 5 devices

VDL cost saving over vendors’ private clouds (today)

Annual cost ($)

# of devices

Intuit Virtual Device Lab

150

3rd-party vendor 1 local device in cradle

Assumptions: 50 native apps & daily check-in Note: As vendor pricing changes, so do the savings

Savings

25

3rd-party vendor 2 local device w/o cradle (Software approach)

Savings

Private cloud setup behind corp firewall

Key takeaways

Building custom solution vs. 3rd-party vendor solution §  Understand your technical/platform/global spread §  Many vendor solutions provide a larger set of devices/OS versions

in the global market §  Quite a few offer mobile test services – might be tightly coupled with their

device access solution & automation framework §  Connecting to your corporate network is a challenge – some provide

private cloud (on-site setup within your network) but it’s cost-prohibitive

Invest in automation and CI §  Automate tests with robust automation framework §  Enable rapid test development through shared test library §  Adopt CI in mobile development for speed & quality

More mobile innovations at Intuit

§  Mobile crash reporter §  Mobile components & design patterns §  Mobile analytics SDK & gateway

Visit booth #601 – Accelerate Development §  Demos & conversations §  Contest §  Mobile device charging station §  Foosball §  We’re hiring !

Win a weekend with a TESLA!

Thank you!! Tina_Su@intuit.com