mobile testing success: real world strategies and techniques

16
T10 Concurrent Class 10/3/2013 11:15:00 AM "Mobile Testing Success: Real World Strategies and Techniques" Presented by: Clinton Sprauve HP Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] www.sqe.com

Upload: techwellpresentations

Post on 13-Jan-2015

130 views

Category:

Technology


0 download

DESCRIPTION

Today, consumers spend more time on mobile apps than on the web. With this increased demand and paradigm shift toward mobile devices, the role of the software tester is evolving and becoming more complex. Since mobile testing is a relatively new domain, software testers face the challenge of understanding not only what to test but how to test. Clint Sprauve focuses on real world strategies and techniques for mobile app testing including device provisioning, mobile network virtualization, multi-OS platform coverage, and hybrid app testing. Learn how companies across various industries—insurance, finance, and entertainment—are implementing successful mobile testing strategies and techniques to meet this growing challenge. In addition, Clint highlights what is most important when creating a mobile testing strategy for your organization—object recognition options (native, text, and image), mobile app performance, and device security.

TRANSCRIPT

Page 1: Mobile Testing Success: Real World Strategies and Techniques

T10 Concurrent Class

10/3/2013 11:15:00 AM

"Mobile Testing Success: Real

World Strategies and

Techniques"

Presented by:

Clinton Sprauve

HP

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Mobile Testing Success: Real World Strategies and Techniques

Clint Sprauve

Hewlett-Packard

Clint Sprauve is a senior solutions architect for HP Software. Previously, Clint was the director

of product strategy for the Borland Solutions Division of Micro Focus, where he created

Borland’s mobile strategy for functional test automation. He has more than twenty years of

experience in the software development and quality assurance industry. Clint has served as the

senior product marketing manager for the Silk Testing Solutions at Borland Software and Segue

Software, and served as a senior technical sales engineer for both companies.

Page 3: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

1

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Mobile TestingSuccessReal World Strategies & Techniques

Clint Sprauve

Solutions Architect & Software Innovation

Coach

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

What You’ll Learn Today

Current State of the Mobile Industry

Mobile Opportunities and Challenges

Mobile Performance

Mobile Success – Case Studies

Page 4: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

2

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3

Android – Device Fragmentation

According to Google’s own numbers, only 37.9

percent of Android devices are running Jelly Bean,

the latest version of Android. That number is only

slightly higher than that of Android 2.3

Gingerbread, which still commands a 34.1 percent

share in spite of being released in 2010.

source: http://venturebeat.com/2013/07/30/android-fragmention-2013/

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

Android – Device Fragmentation

source: http://venturebeat.com/2013/07/30/android-fragmention-2013/

“The availability of cheap Android phones (rarely

running the most recent version) means that

[Android phones] have a much greater global

reach than iOS, so app developers have a wider

audience to build for…”

Page 5: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

3

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5

Why Fragment the OS?

source: http://venturebeat.com/2013/07/30/android-fragmention-2013/

• At it’s core, it’s about choice

• Smartphone manufacturers can

sell a variety of phones

• Different configurations

• Different prices

• Different consumer market

segments

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6

Android vs. iOS

Page 6: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

4

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7

Mobile creates business disruptions

Location-aware, frictionless-

payment taxis & private cars

Accelerated check deposit

processing from image

Check in, change and monitor

flights

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8

Business challenges

The competition is going mobile

Accelerated time to market

Improving customer interaction

Transforming and streamlining processes

New sales channels

Page 7: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

5

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9

IT challenges

Chaos and change

New/unknown technology

Accelerate delivery

Complexity

Performance

Risk

Mobile drives IT to face a new reality

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10

Mobile success and failure is very visible

Very poor app

ranking in Google

play

Competitors have

4 star apps

Where is your app?

April 2012

Page 8: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

6

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11

Could better mobile testing have prevented this?

A very public launch

With very public

quality issues led to …

public apologies and

executive turn over

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12

Why is mobile testing so difficult?

1. User expectations are exceedingly high… it must work right the first time

- Incredibly competitive market

2. Real devices are critical

- Test results can be dependent on device, OS, and network provider

3. Speed and velocity is essential

- Pace of change in mobile platforms and devices is incredibly fast

- Mobile development projects must be agile in fast and frequent sprints

4. Mobile is not an isolated effort.

- Must be integrated in normal deliver processes

- Full lifecycle approach leads to quality outcomes

Page 9: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

7

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13

Cloud based service

Shared, remote access

Real smartphones

Real Tablets

Real mobile service

Device agnostic

Flexible and robust

Enterprise visibility

Short learning curve

Development

Functional

Performance

Monitoring

Five dimensions of mobile testing

Key considerations for mobile testing success

Test device

management

Reality Automation ALM

integration

End to end

lifecycle

Secure, centralized

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14

Is this your mobile test lab?Real devices, but how will it scale?

• Automation

• Security

• Location

• Change mgt

• Governance

• Risk exposure

• Cost

Page 10: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

8

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15

Mobile Performance

Mobile applications are everywhere

Impacting:

• Front and back-end economics

• Loss of revenue

• Loss of customer

• Available anytime, anywhere

• Significant business opportunity

• Additional sales channel

• Analyst influence

Why mobile performance has become a competitive differentiator

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16

What happens when customers “struggle” on a

mobile device? (TeaLeaf Industries, 2011)

43% Abandon the

transaction, try later

on computer16% Become more

likely to prefer a

competitor14% Log a

complaint with

customer service12% Go directly to a

competitor

You

Your

Competitor

Page 11: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

9

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17

Network Virtualization – Reduce Mobile Risk

Capacity > 500+ Users

Capacity < 200 Users

By not considering actual network conditions, it can be a trap!

Without real

network conditionsReal network conditions

VS

Performance test results in a false sense of security.

With real network conditions; Test results accurate and reliable.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18

HP/Shunra customer survey:

•Network Virtualization eliminated >15 performance related production incidents per year

• ROI < 90 Days

ROI: Value of Network Virtualization

5

3.7

0

1

2

3

4

5

6

Results Before vs. After Network Virtualization*Source: 2011 Shunra for HP Customer Survey

Relative Costs to Fix a Problem (very expensive to fix in production)

(Steve McConnell’s “Code Complete”, pp. 960 - Microsoft Press)

# of Performance Related

Production Incidents Per Month

Page 12: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

10

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Functional Testing

Object Recognition Types

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20

Object Recognition Types

� Native Recognition - recognize objects by their native id and work with the actual

components within the application (i.e., text box, text field, button)

� Image Recognition - captures what is seen on the screen of the device. It captures the

pictures and images and uses complex algorithms to recognize objects while

executing the test

� Optical Character Recognition (OCR) - allows the tester to extract the text from

screen (also known as “screen scrapping”)

Page 13: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

11

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21

Difficult with customer objects

Hard to keep up with OS Updates

Flexible and Robust

Adjust to UI Changes quicker

Easier to maintain

Faster test execution

More coding knowledge required

Multi-OS Updates hinder automation activities

Native Object Recognition

Advantages Disadvantages

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22

Slow Execution

Difficult to maintain

OS Independent Portable

Easier to buildWorks with what you see on the screen

Increased test maintenance

Reuse is extremely difficult

Image Recognition

Advantages Disadvantages

Page 14: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

12

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23

Slow Execution

Not always dealing with text

OS Independent Portable

Easier to buildWorks with what you see on the screen

Increased test maintenance

Reuse is extremely difficult

Optical Character Recognition (OCR)

Advantages Disadvantages

Difficult to maintain

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24

Large US Airline

Case Study

Challenges: Struggled with automation of mobile testing, leveraging existing QA

processes, tools & knowhow

Engagement

• Started with a private cloud & function test automation

• Added complementary public cloud - device coverage.

• Performance plans

Currently: Extremely fast ramp up, Running regression, functional, automation,

offshore testing team.

Page 15: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

13

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25

Large US Financial Services Firm

Case Study

Challenges: Handset procurement and selection, regression testing around new

versions and new devices, utilizing offshore resources for domestic US networks.

Engagement

• Started with private cloud and functional test based automation

• Added complementary public cloud - device coverage.

• Extended and doubled the capacity recently

• Performance testing plans

Currently: Running regression, functional automation, hundreds of scripts, constant

execution, dedicated team.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Summary

• Take time to understand your company’s

mobile strategy

• Review your team’s skill set

• Performance is as important as design

• Seek help from mobile vendors

• Mobile Testing is complex, but you can be

successful

Page 16: Mobile Testing Success: Real World Strategies and Techniques

9/19/2013

14

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27

Additional Resources

HP Mobile

Testing

HP Performance

Center

HP LoadRunnerHP Unified

Functional Testing

Resource URL

Mobile Testing @ HP .com www.hp.com/go/mobiletesting

Blog and Community www.hp.com/go/thefutureoftesting

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Thank you