|

Introduction to Mobile App Testing Presented by Elena Houser May 30, 2012

| 2

1. What is Mobile?

2. Software Testing Overview

3. Mobile App Testing

4. Mobile Platforms

5. iOS vs. Android

6. Wrap Up & Questions

Agenda

The Challenge

| | 3

What is Mobile?

|

What is Mobile?

4

• Mobile Device – portable device powered by a battery that is carried

around by the user and uses wireless communications.

• Mobile activity primarily consists of two categories

– Web: Customized websites intended for viewing on smartphones and tablets over a cellular or wireless internet connection.

– Mobile/Native apps: Software programs developed specifically for

smartphones and tablets. They access data over a cellular network

or wireless internet connection.

|

Mobile Web vs. Native App

5

Some Companies use a Hybrid of Native App and Web for Mobile

– ESPN ScoreCenter

- A mobile app with updated scores within the app

- Directs users to its mobile website for further articles

– Pandora

- A native app for listening to music, with small ads

- HTML5-powered website to stream music and

eventually deliver more content

– For use on mobile and other devices

| 6

Smartphones have full operating

systems, that can be updated,

making them almost multi-purpose

computers whereas features

phones are more limited. Today’s

smartphones boast a multitude of

operating systems. The most

popular ones in the US are

Google’s Android operating

system, Apple’s iOS, and RIM's

Blackberry

Smart Phone vs. Computer

|

Mobile Market Trends

7

• Web and native apps were developed due to the

growth of the Smartphone and tablet market and

their ability to keep users connected more

consistently

• Smartphone applications are ubiquitous and

mainstream (banking/finance, games,

productivity, fitness)

• Abundance of free apps

Opportunities for testing “…because your app is free and has a built-in market doesn’t

mean you can lower your standards or ignore what end users

like. .. You need to fully understand how the program can be

useful on-the-go and focus solely on the features that would be

handy and increase productivity in a mobile, untethered setting.”

http://blog.utest.com/why-testing-your-business-app-is-

important/2012/05/

Mobile Market

| 8

Mobile App Success

To succeed Mobile should provide value to the

user based on the following types of services:

Games: Most downloaded app category -65%

(Nielsen Wire) – Ux/entertainment and

performance

Social: latest news and share content with

friends

Retail: location intelligent, m-commerce

Media: news stream, offline access to content

Travel: mobile bookings, check-ins, maps,

deals

http://www.razorianfly.com/2012/03/07/heres-the-itunes-app-stores-100-all-time-top-apps-direct-links/

Mobile Success

|

Why Test Mobile Apps?

9

Mobile Apps and Usage are a Big Deal 49.7% of US mobile subscribers owned Smartphones in

February 2012 (http://www.newmediatrendwatch.com/markets-by-country/17-usa/855-

mobile-devices)

• By 2015, International Data Corporation (IDC) predicts

that 182.7 billion mobile apps will be downloaded, close to a

1600% increase from the 10.7 billion apps downloaded in

2010

• Diverse user base, multiple devices, manufactures,

platforms and fragmentation

Our focus and goals

• First Impression for the brand/product is important in the

app stores

• Identify technical, design and functional challenges early

to help reduce the cost of failure and repair

• Evaluate readiness for release

The Challenge

| | 10

Software Testing Overview

|

Definitions & Terminology

11

• What is an App?

– Container for a product and/or service that is of value it its users.

• What is an App’s Purpose?

– It should provide value to the user based on a type of a service

(e.g., email, social media, game or productivity app)

• What is Quality?

– Value to some person (Jerry Weinberg)

Different users have different opinions about the app quality

Ratings

User feedback

• What is a Bug?

– Any threat to the value of the product to any stakeholder who

matters (James Bach)

|

What is Testing?

12

Testing - An empirical, technical investigation conducted to

provide stakeholders with information about the quality of

the product or service under test. - Cem Kaner Testers design and run tests in order to gain useful information

about the products quality.

http://www.testingeducation.org/BBST/exploratory/

Exploratory Testing - An interactive process of simultaneous

learning, test design, and test execution. - James Bach

|

What is Testing?

13

Different objectives require

different testing tools and

strategies and will yield

different tests, test

documentation and test results. Black Box Software Testing –

Foundations

Cem Kaner & James Bach

http://www.satisfice.com/tools/satisfice-tsm-4p.pdf

The Challenge

| | 14

Mobile App Testing

|

Quality & Mobile Context Driven App Testing

15

•Quality - is Value to Some Person (Jerry Weinberg)

– The value being tested may not be of value to you

– As a Test professional, you test for that value(s)

•Testing

– Whether it’s mobile, desktop or other types of testing the core approaches are

the same

•Context Driven Testing (CDT)

– There are no "best practices" in testing, but rather testing is a set of skills that

allow the tester to select or invent testing practices to suit each unique situation.

•Mobile Context Driven Testing (MCDT) - applies the mobile world

- Project Scope (Functional, Showstoppers/Crashes, Ux, Battery, etc.)

- Devices (phones, tablets)

- Tools (DDMS & Jing, LogCat)

- Mindset, difficulty, challenges

| 16

http://www.kohl.ca/articles/ISLICEDUPFUN.pdf http://moolya.com/blog/wp-

content/uploads/2012/04/COP-FLUNG-GUN1.png

Quality & Mobile Context Driven App Testing

| 17

Quality & Mobile Context Driven Testing

http://unifiedtestinginitiative.org/files/utc_android_v1.pdf

The Challenge

| | 18

Mobile Platforms

|

Overview of the Main Mobile Platforms

19

The Challenge

| | 20

iOS vs. Android

| 21

http://www.engadget.com/2010/12/07/visualized-the-real-android-fragmentation/

•15-month-old Gingerbread operating

system (63.7%)

•2nd most popular version - 23-month-old

Froyo OS (23.1%).

•1st unveiled in October 2009 and currently

at 6%, Android 2.1 Eclair is still found on

nearly as many devices as Honeycomb and

Ice Cream Sandwich combined.

Android Fragmentation

|

Android Fragmentation

Android fragmentation highlights

include:

• 5 OS versions on devices

(Donut, Eclair, Froyo,

Gingerbread, and Ice Cream

Sandwich)

• Devices in various sizes, from

small phones to TVs

• Device manufacturers branching their own versions

of Android

22

| 23

Android Fragmentation

|

iOS Fragmentation

24

http://www.marco.org/2011/11/30/more-ios-device-and-os-version-stats-from-instapaper

http://www.redmondpie.com/study-reveals-that-ios-apps-crash-more-frequently-than-android/

|

Which one is easier to test – Android or iOS?

25

http://411droid.com/humor-surrender-iphone-have-surrounded/

| 26

Which one is easier to test – Android or iOS?

| 27

iOS Android

I

n

s

t

a

l

l

a

t

i

o

n

1. iTunes or iPhone Configuration Utility 2. With UDID provisional and executable or just executable - download and drag them to your iTunes

library (1st provisional, then the executable file) or iPhone Config Utility 3. iDevice test apps can also be downloaded Over-The-Air (OTA) via a secure link to authorized devices.

1. Device: "Settings -> Applications -> Unknown sources" 2. Download -email attachments, Dropbox, a barcode scanner program, etc.

3. Accept the permissions and install the application. 4. App can be launched via shortcut or widget (needs to be added - Settings>Applications>Manage Applications available). After initially launching a test app, it can be called by pressing and holding the native device Home button and then tapping on the

respective icon under Recent Apps list.

S

c

r

e

e

n

s

h

o

t

s

Hold down the home button and press the power button at the top right. Either email or sync your device with either iTunes or a photo management program of the user's choice. Digital cameras can be used for videocaptureing.

Digital cameras can be used to take a photo or video of the Android device's screen. Also Dalvik Debug Monitor Server (DDMS), Droid Explorer Android and AtScreen programs and TechSmith's Jing to capture the steps to reproduce a problem.

L

o

g

s

Console logs - iPhone Configuration Utility; Crash Reports - sync up with iTunes (https://developer.apple.com/library/ios/#technotes/tn2151/_index.html)

Log Collector or CatLog, DDMS/Eclipse(http://developer.android.com/sdk/index.html)

O

t

h

e

r

Speedtest - for easy connection testing (Download, Upload, and Ping)

Which one is easier to test – Android or iOS?

| 28

Few Tips and Recommendations

•“Software testers who are assigned to test applications on mobile devices need to go

beyond thinking “if I tap on the icon, does the application start” or “if I press the cancel

button, does the operation/function get cancelled.”

•Functional application testing is important but, the application is a part of an entire system

and not just a singular entity. The application is completely dependent on the entire system,

to function within the system, is a part of the system.

•Temperature, memory, CPU Speed, Database searches are all affected when a device is

charging and these tests are continually not considered.

•A mobile app tester’s responsibility is not simply to find the bugs, their goal must be to

increase the quality of the app, finding bugs, is just a small part of this. The testing

performed must also take into consideration, the goal of the app, the target audience etc.

•By collaborating with the app developer, a mobile app tester can not only significantly

increase the app’s quality but also their involvement can drive forward the project with

alternative design suggestions and usability feedback. http://www.testing4success.com/t4sblog/mobile-device-testing-vs-mobile-applications-testing/

Wrap Up

The Challenge

| | 29

Thank You

Questions?