best practices for devops in mobile app testing

3 March 2016 – 12pm GMT

Ville-Veikko HelppiHead of Demand [email protected]

Best Practices for DevOps in Mobile App Testing

W E B I N A R

Agenda• Deconstructing ‘Mobile DevOps’

• Mobile Product Development Life-Cycle & Used Tools

• The DevOps Approach in Mobile App Development & Testing

• Best Practices for DevOps in Mobile App Testing

• Q&A

http://testdroid.com

DECONSTRUCTING “MOBILE DEVOPS”


Mobile DevOps

DevOps = Practices that emphasize the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. --Wikipedia

• Development focus on building the actual product and using relevant tools

• QA doing continuous testing on robustness, functionality, performance, and preferably use test automation

• Operations maintain the continuous build/integration/deployment/delivery environments, and do releases

3 Things in Mobile DevOps

• 3 DevOps Principles1. Develop & Test Against Real Environment

2. Deploy Frequently

3. Continuously Validate Quality Characteristics

• 3 Business-Critical DevOps Categories1. Continuous Integration & Continuous Delivery

2. App and Infrastructure (e.g. Back-End) Testing & Monitoring

3. Entire Dev&Test Process and Mobile App Delivery


DevOps Strategy & Typical Goals

• Maximize the Delivered Value• Customers, Internally between BUs and Teams

• Maximize the Efficiency and Productivity• Improved Process, The ‘Right’ Tools & Methods, Faster T2M

• Maximize the Quality and Robustness of Apps• Better Quality with Equal Effort, Investments in Quality

• Maximize the Support, Service & The Delivery• Alignment and Collaboration between Teams

The highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Agile Manifesto – Principle #1“ “http://testdroid.com

How DevOps Can Make A Difference

• Based on Testdroid Customer Survey 2015• Things That Go Wrong with App Testing

1. Too Long Dev-Test-Feedback Cycles

2. Bug Catching/Reporting Too Slow to Developers

3. Non-Functional or Not Well Utilized Test Automation

4. Use of Emulators - Setting Up & Maintaining Real Devices

5. Access to Relevant Mobile Platform / OS version / Devices

6. Use of Manual Testing – The Weakest Link in Agile and Devops Process

7. No Geographically Relevant Testbed – Devices, Networks, Back-End Integration


Where DevOps Make A Difference

• Devices / Hardware- Handling of new devices and other new HW used in the infrastructure

• Infrastructure- Ready-to-Go infrastructure means no delays for development, testing and result-

driven approach speeds things up & saves time and money

• Support- SW devs, SW devs in test (SDET), QA engineers – all need different type of support

when using the infrastructure and DevOps approach can help a lot

• Operations- Maintaining and Upgrading infrastructure, devices, networks, additional SW

• Tools Integration - Support for multiple test tools gives the users/developers freedom of choice. No

vendor, tech, tools lock-ins!http://testdroid.com

MOBILE DEVOPS & DEVELOPMENT- LIFECYCLE & TOOLS


DevOps Challenges *• Building the DevOps culture across organization• Overcoming the Dev vs. Ops mentality• Moving from legacy (and sometimes on-premise setup)

to cloud based tools and environment• Broader scope than before: SW and HW for mobile dev• Learning new skills – DevOps practice requires change

and collaboration across teams• Mobile Challenges:

• Fragmentation, Tools and Platforms, Not Enough Real Devices• Less Worrying Challenges on Mobile:

• Cross-platform, Using Cloud Efficiently

* = Based on Testdroid Customer Survey 2015http://testdroid.com

DevOps ‘Toolchain’

• Code – Code Development and Code Reviews, Continuous Integration tools

• Build – Version control tools, SCM, code merging, Build status

• Test – Test and results determine performance

• Package – Artifact repository, Application pre-deployment staging

• Release – Change management, Release approvals, release automation

• Configure – Infrastructure configuration and management, Infrastructure as Code tools

• Monitor – App perf. monitoring, UX

DevOps ‘Toolchain’

Code & Build

Continuous Integration, Continuous Development

Package, Release, Configure & Monitoring

Continuous Delivery, Continuous Deployment/Release

GOAL: End-to-End Transparency, Traceability, Seamless Workflow, Continuous Everything!

2 Valid Options for Mobile Testing: Cloud or On-

Premisehttp://testdroid.com

DevOps – Testing in the ‘Cloud’

1

2

3

DevOps – Testing On-Premise

CONTINUOUSINTEGRATION


Benefits of Test Automation• App Compatibility & Global coverage

• Reusability• Speed & Efficiency

• Test efficiency• Test effectiveness• Time-to-market


DevOps Must Rely on Automation

• DevOps is more than just a tool or a process change; it inherently requires an organizational culture shift

• Agile methodologies back up this thinking• Increase focus on test automation and CI methods• Best practices and continuous iteration

TimeResources

Infrastructure

TrainingTools


THE DEVOPS APPROACH IN MOBILE APP TESTING


Testing Philosophy


Manual AutomationTesting Philosophy


Smaller coverage, More money burnt & time wasted, Error-prone

Manual AutomationLarge coverage,

quickly completed,

Less money & time wasted, Exact results.

Testing Philosophy

Top Requirement for Testing

• Real Devices and Real Browsers, as emulators cannot help you to test...

• User Experience • Usability• Hardware• Software• Infrastructure

0 % = the percentage of your app users that use emulator to run your app!


Devices & Daily Tasks for DevOps

• In order to maintain mobile device farm, environment and all aspects of SW & HW the following daily check-ups must be done:• Are all devices connected and operating?• Are all devices WiFi connected?• Do all devices have an active USB connection?• Are battery levels adequate (at least 50%) for every device?• Are there any pop-ups, system notification or other system

dialogs open on the screen?• Are screens unlocked?


Troubleshooting Ain’t The Same


Automation vs Manual


Test Breakdown

FAILED FAILED


Mobile Test AutomationAutomatic test exercisers Record and PlaybackHand written test scripts

Benefits:Accurate, specific to your testing needs, plenty of options with frameworks, tools

Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty

Fastest & extremely automated, excellent for smoke testing/quick testing, availability

Tradeoffs:Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)

Compelling Recorder + Playback tools available for only few test automation frameworks

Not accurate as real test cases


Why Should DevOps Be Interested in …

Robotium uiautomator

Espresso Appium Calabash

Android Yes Yes Yes Yes Yes

iOS No No No Yes Yes

Mobile web Yes (Android)

Limited to x.y clicks

No Yes (Android & iOS)

Yes (Android)

Scripting Language

Java Java Java Almost any Ruby

Test creation tools

Testdroid Recorder

UI Automator viewer

Hierarchy Viewer

Appium.app CLI

Supported API levels

All 16 => 8, 10, 15- All All

Community Contributors Google Google Active Pretty quiet


BEST PRACTICES FOR DEVOPS IN MOBILE APP TESTING


Automate (Everything)• When it comes to mobile app testing, the more

you can automate repetitive things, the better• Not always possible but leave the last

‘exploratory testing’ for the final phase• Smoke, Unit, All sorts of Performance,

Regression testing should be automated• Run your test patterns against EVERY build• Start small but make sure you can scale-up


Ensure Workflow is Seamless

• Only use tools, frameworks and software that is fully compatible across used toolchain/workflow

• Ensure all processes, results, tools are entirely transparent and available for use across teams

• This has a significant impact on productivity and efficiency

• Open source is always highly recommended


Rely Only On Real & Quality HW

• Mobile device farms require lots of hardware – this is the correct place to invest in quality HW

• Make sure the mobile device coverage is adequate – globally

• Execute the upgrade and support policies relentlessly


“The Right” Testing Framework

• Variety of choices when it comes mobile test automation frameworks, ask the right questions:• App, Game, Web? Native, Hybrid?

• Mobile Back-End Integrated?

• Who needs to write, read and interpret tests?• Does test run results provide you what you

organization needs / is looking for?


THANK YOU!More information about frameworks,

devices and testing at www.testdroid.com


best practices for devops in mobile app testing

Technology