mobile test automation

Mobile Test Automation

Andreas Jakl

@andijakl

andreas.jakl@tieto.com

Mobile Developer After-Work #7 – Xamarin: April 9th, 2015

Andreas Jakl

Mobility Evangelist, Tieto

– @andijakl

– andreas.jakl@tieto.com

History

– Mopius: Startup-Founder (NFC & Mobile Apps)

– Nokia, Finland: Technology Wizard

– FH Hagenberg: Assistant Professor Mobile Computing

– Siemens / BenQ Mobile, Germany: Augmented Reality-Apps

Mobile Test Automation - Andreas Jakl2

TESTING – WHY?

Versions & More

Mobile Test Automation - Andreas Jakl3

Slow uptake of new

versions

Manufacturer-

modified UIs

Countless screen

sizes & densities

Mobile Test Automation - Andreas Jakl4

https://developer.android.com/about/dashboards/index.html

http://stats.unity3d.com/mobile/os-android.html

Fast adoption of new OS

– Design updates

– API & architecture changes

(e.g., 64 bit enforcement)

Quick update of apps required

Same OS version, but many iOS

devices in use

Mobile Test Automation - Andreas Jakl5

https://developer.apple.com/support/appstore/

http://stats.unity3d.com/mobile/device-ios.html

Similar hardware & software base

– 512 MB vs 1+ GB memory

– Processor speed

– 4 supported screen resolutions

Relevant: v8(.1) > 90%

– But: Windows 10 upcoming

• More resolutions, etc.

Mobile Test Automation - Andreas Jakl6

http://blogs.windows.com/buildingapps/2014/12/22/windows-and-windows-phone-store-trends-december-2014-update/

http://blog.adduplex.com/2015/03/adduplex-windows-phone-statistics.html

IMPACT OF TESTING

Successful?

Mobile Test Automation - Andreas Jakl7

Reviews vs Downloads

Review count & rating correlate to downloads

Mobile Test Automation - Andreas Jakl8

Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.

Carnegie Mellon University, 2013

Data from 13 million user reviews, 171,439 apps

Average rating: 3.9

(standard deviation: 1.48)

0

2 000 000

4 000 000

6 000 000

8 000 000

Review Analysis

Analysis

– Most frequent complaints

• Functional errors

• Feature requests

• App crashes

Mobile Test Automation - Andreas Jakl9

Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013

Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014

Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

Review Analysis

Analysis

– Most negative impact

• Privacy and ethical issues

• Hidden app costs

Mobile Test Automation - Andreas Jakl10

Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013

Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014

Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

Customer Retention

Quality (feature)

upgrades

– Up to 3x

improvement in

survival rates

Mobile Test Automation - Andreas Jakl11

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

1 Month after Download 6 Months after Download 1 Year after Download

Customer Retention in Mobile Apps

TESTING STRATEGIES

Manual

Mobile Test Automation - Andreas Jakl12

Capturing Crash Data

Mobile Test Automation - Andreas Jakl13

Monitoring User Reviews

Mobile Test Automation - Andreas Jakl14

Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.

Carnegie Mellon University, 2013

Static Code Analysis

Mobile Test Automation - Andreas Jakl15

Emulator / Simulator

Mobile Test Automation - Andreas Jakl16

Emulator

Emulates real

hardware

+ more accurate

– slow

Android

Simulator

Simulates device

behavior

+ faster

– less accurate

iOS / Windows

Reliable?

Mobile Test Automation - Andreas Jakl17

0%of your users will run your app on an emulator

Own Devices

Mobile Test Automation - Andreas Jakl18

How many test devices do you have?

Open Device Lab Vienna

Mobile Test Automation - Andreas Jakl19

Free testing on real devices

http://www.open-device-lab-vienna.at/

Automated UI Tests

Mobile Test Automation - Andreas Jakl20

How many tests can you run per hour / build / release?

Number of builds

Cost

Automated tests

Manual tests

TEST CLOUD

Automated Testing on 1000s of Devices

Mobile Test Automation - Andreas Jakl21

Xamarin Test Cloud

Automated tests

– UI interaction

– Device features (GPS, etc)

– Performance

1,300+ devices

– Android & iOS

– testcloud.xamarin.com/devices

Mobile Test Automation - Andreas Jakl22

Workflow

Mobile Test Automation - Andreas Jakl23

+ Continuous Integration

Mobile Test Automation - Andreas Jakl24

Calabash Xamarin.UiTest

Test Frameworks

Mobile Test Automation - Andreas Jakl25

Write tests in Ruby C#

Run with Gherkin NUnit

Executing Tests

Mobile Test Automation - Andreas Jakl26

Process

1. Write Tests

2. Setup Project for Testing

3. Test Locally

4. Create app on Xamarin Test Cloud

5. Upload & Run on Cloud via Command Line

Mobile Test Automation - Andreas Jakl27

Writing Tests: NUnit

Mobile Test Automation - Andreas Jakl28

Generic tests possible

with Xamarin.Forms

Test Setup

Mobile Test Automation - Andreas Jakl29

Platform-specific

initialization & test

differences

Setup Project

Mobile Test Automation - Andreas Jakl30

Disable shared

Mono runtime

API Key

Mobile Test Automation - Andreas Jakl31

Create team & app to

get unique API key

Run Tests

Locally

Mobile Test Automation - Andreas Jakl32

Test on your

devices first

Mobile Test Automation - Andreas Jakl33

Cloud Setup

Choose devices

Mobile Test Automation - Andreas Jakl34

Cloud Setup

System settings

Mobile Test Automation - Andreas Jakl35

Running Tests

in the cloud

Command line

only

Mobile Test Automation - Andreas Jakl36

Test results

Dashboard

Mobile Test Automation - Andreas Jakl37

Test results

Screenshots

Custom Screenshots

Take & describe screenshots

Enable local screenshots

Mobile Test Automation - Andreas Jakl38

[TestCase (TestName = "Click button and show success message")]public void TestCaseSuccess (){

AppResult[] result = _app.Query(InitialMessage);Assert.IsTrue(result.Any(), "The initial message string isn't correct");_app.Screenshot ("Initial screen");

_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();

Custom Screenshots

Take & describe screenshots

Enable local screenshots

Mobile Test Automation - Andreas Jakl39

[TestCase (TestName = "Click button and show success message")]public void TestCaseSuccess (){

AppResult[] result = _app.Query(InitialMessage);Assert.IsTrue(result.Any(), "The initial message string isn't correct");_app.Screenshot ("Initial screen");

_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();

Testing Finished App Packages

Find IDs of UI elements

Mobile Test Automation - Andreas Jakl40

if (TestEnvironment.Platform.Equals(TestPlatform.Local)){

_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();

_app.Repl ();// In Bash, enter app.Query(c=>c.Button()) or c.All()

}

Testing Finished App Packages

Find IDs of UI elements

Mobile Test Automation - Andreas Jakl41

if (TestEnvironment.Platform.Equals(TestPlatform.Local)){

_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();

_app.Repl ();// In Bash, enter app.Query(c=>c.All())

}

Alternatives

Testdroid

– testdroid.com/

Perfecto Mobile

– perfectomobile.com/solution/test-automation

Mobile Test Automation - Andreas Jakl42

Thank You!

Andreas Jakl

@andijakl

andreas.jakl@tieto.com

Mobile Test Automation - Andreas Jakl43