SPEAKER NAMEJob Title, Organisation

Presentation Title

Testing Enterprise MobilityEneko Jon BilbaoSAP Innovation & Emerging Technology, Accenture

• Accenture ANZ SAP Mobility Lead – also get to dabble in our Innovation Centre (gadgets!)

• Recently been working with a number of large clients on their mobility journeys.

• In particular:– Two recent major implementations 2013-2015

– including SAP Mobile Platform, Fiori, Click Mobile, Device roll out

– Mostly packaged Hybrid Applications, a few Native

• This story is based on my thoughts from the journey’s we’ve been on with our customers.

Introduction & Overview

2

It is no longer enough to have a large team of in-office testers who run through “the usual” tests (unit, integration, user acceptance and performance).

Consider broader testing to gain better confidence of the benefits from your investment in mobile applications.

Enterprise Mobile Solutions are different.

3

• Lessons Learned

• Mobile Applications are Different

• New(ish) Objectives for Testing

• Skills to Learn

• Tools & Techniques

• Key Takeaways

What I’ll Cover

4

Obligatory Wor(l)d Cloud

5

*Provided by Tagul.com

Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for TestingSkills to LearnTools & TechniquesKey Takeaways

1. Development complete ≠ Go Live

2. Device emulators are not enough.

3. Network latency is generally a bigger factor on mobile application performance than bandwidth.

4. Make network and communication decisions early on in your mobility journey.

5. Design for security everywhere (encryption, SSO)

Lessons Learned

7

Mobile apps are differentNew(ish) Objectives for TestingSkills to LearnTools & TechniquesKey Takeaways

Lessons Learned

There are high expectations to meet!

Enterprise Mobile is different...

9

“everywhere and easy”versus

“only in the office and it’s OK…”

I can use it anywhere!It’s simple and easy to use so I won’t need training.

I can find my data easily.It’s fast!

User Experience doesn’t stop with User Interface design, it encompasses the whole experience of the mobilised business process.

• Locations (work anywhere)• Flexibility (work on anything)• Productivity (work fast)

These expectations drive new testing requirements

10

Anatomy of a Mobile Application:

There is much to test…

11

Your Backend Code /

Server & DB Performance

Generally well understood and

tested

Here be dragons…

External Services / End Points

Network(Latency,

Bandwidth)

Browser / Client Code

Device Feel / Response

Your Total Mobile

Experience

• A reliable and consistent wired network is difficult.

• A reliable and consistent wireless network whilst being mobile is near impossible.– Realise that your users will be moving!

– Network conditions will change, bad latency will occur and drop outs (whilst getting more infrequent) will happen.

– Be curious how your application behaves in these circumstances.

Add 100-200 milliseconds for 3G latency

Add 60-120 milliseconds for 4G latency

Diff 1: Networking Killed the Mobile Star

12

• Using APIs and others’ services

• Mash ups and extensions to standard SAP Apps are common user requirements.

• For example: show me my company’s Geospatial data, grab me some information from a document management system)– Remember to test these!

– Do the API owners know the extra users are coming? Have they agreed to tell you if something changes in their service?

– Are they optimised for Mobile?

– Can they be accessed from both the intranet and internet?

Diff 2: But I want to use yours!

13

Diff 3: Not all clients are created equal

14

• Chrome on iOS behaves differently to Chrome on Android(s) etc.:

– Security Certificates are handled differently

– JavaScript engines are different – what works in one might not work in the other.

– Different # of parallel TCP connections affects speed

– You may or may not get notification of errors! (they may only be in the console)

Diff 3: Not all clients are created equal

15

• Not all touch screens are the same.

• Factors that may make your users complain:

– Speed of response whilst tapping or dragging

– Accurate “touch points” – supporting fingers in gloves?

– Brightness / Glare of screen

– Ruggedness and how easily it breaks!

Diff 4: You’re not on a PC anymore Dorothy

16

New(ish) objectives for testingSkills to LearnTools & TechniquesKey Takeaways

Lessons LearnedMobile Applications are Different

Yay! Success!

Testing the new Mobile Expectations

18

Traditional Testing, PLUS:

• Test the varying network impacts are acceptable

• Test our multitude of devices work & perform

• Test app desirability and thus our productivity

Work on Anything

Work Anywhere

Work Fast

Consider the following questions:

1. How do your applications perform in slow, medium, fast network conditions? What is the likely responsiveness?

2. How do your applications perform when network conditions change? Do they fail gracefully?

3. Will your applications be offline for periods of time without connectivity? Can users transact without a network connection?

4. Where will your applications physically be used the most? City (basements?), Rural (cell towers?)

Working Anywhere – Network Behaviour Testing

19

Consider the following questions:1. What devices will your users be on? How do they

differ?

2. What screen real-estate will they have? What will be the main mode of interaction be? E.g. Touch, pen, mouse & keyboard?

3. What is the battery life expectation for your devices? How long will your users be away from power?

4. Are your devices likely to get dirty? Be dropped? Need to work in harsh sunlight?

5. Will hybrid applications deploy the same way on all your devices?

Work on Anything – Device Testing

20

Consider the following questions:

• What do your users find difficult? What don’t they like?

• How many interactions does it take to do what your users need to do?

• Have we minimised the inputs our users need to make?

• What are the impediments to users completing their activities quickly?

Work Fast – Productivity Testing

21

As soon as you can! Generally before User Acceptance Testing.

When to plan your new tests?

22

User Acceptance Testing

Performance Testing

Integration TestingUnit Testing

New Effort

New Test Aims

2-5 days per device

Basic Device Functions:

• Baseline Battery

• Screen Brightness

• Ruggedness

• Touch responsiveness

• Cases/Peripherals?

2-3 days per App

Productivity Testing:

• Use Prototypes if available

• # of Clicks to fulfill

• # of Data Entry Fields

• What is difficult?

• What would make you love it?

2-3 Weeks

Network Behaviour Tests

• Best case performance

• Worst case performance

• “Real world” performance

• Network failure / offline behaviour

Traditional Phases

Iterate and feedback to developers

Skills to Learn

Tools & TechniquesKey Takeaways

Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for Testing

Typical SAP Mobility Communication Paths

24

• SAP is getting more complicated.

• Test all your major service interactions both over external cellular and internal Wi-Fi (the arrows on this diagram)

– Connection Tests (firewalls)– Load Tests (performance)– Session Stickiness (moving locations)– TLS/SSL (security)– How does your code perform in these

scenarios?

• Developers, you’re going to need to better understand the Technology Architectures and Operational landscape your app will be deployed in. Understand your application END to END.

• Technology Operators, you’re going to need to get your hands dirty with some debugging (particularly in browsers) now and then.

Successful Mobility Testing involves DevOps skills

25

Don’t let this be you…

26

Successful Testing

27

Testing / Quality Assurance

DevelopersIT

Operations

FunctionalTesting

Technology Testing

MobilitySuccess

Good Design

What new Skills?

28

Developers Technology Operators / Architects

Familiarise yourself with your network design, understand where the services you are calling are hosted.

Understand what web services the developers are going to call, where they reside, what they do, and how many times they will call them.

Understand how variable latency affects the user experience you’re creating. Build with this in mind always.

Work towards minimising latency for time sensitive application purposes. Network acceleration / caching specific application patterns.

Understand communication security - Website Certificates, TLS/SSL, Cross Site Scripting problems you may encounter due to sharing resources

Provide developers with root certificates to establish trust, know what must be encrypted and where.

Understand the load balancing design and any Virtual IPs / Server Names and how they proxy your calls in case they adversely affect your application.

Design for BOTH intranet and internet enabled services and allow applications to have sessions cross intranet/internet boundaries. Don’t use different domains!

Understand browser limitations, differences in device behaviours - developing in your local IDE is not enough.

Familiarise yourself with JavaScript and browser development tools, provide developers with physical devices.

Tools and Techniques

Lessons LearnedMobile apps are DifferentNew(ish) Objectives for TestingSkills to Learn

Key Takeaways

You’re going to need a Lab.

• All mobile device types

• A network access point you control

• New tools/software

So what do I need?

30

Essential Mobility Testing Tools for Everyone

31

HTTP Debugging / Proxy

Text Editor Screen Capture Message / Web service testing

Why?Understand what’s happening “under the covers” of your apps.

Format Messages, HTML, XML, JSON etc.. Make it easier to read and understand.

Share problems with other teams, a picture is worth a thousand words.

Replay, investigate, debug discrete web services. Load test services.

Options

Wireshark Sublime Text Windows Snipping SOAPUI

Charles Proxy Notepad++ Snag IT JMeter

Fiddler TextMate LICEcap POSTMAN

(Browser Developer Tools – Chrome, IE,

Firefox)Komodo WinSnap

HTTP Watch VIM

• Testing your main Physical Devices is a must.

– Give devices to your developers and your normal test users.

– Emulators are OK for development, but many user experience problems will only manifest in the physical device. For example:

• Touch screen “hot spot” inaccuracy

• Resource loading in a wireless / high latency environment

• Bugs in app packaging / UI frameworks (Apache Cordova, JQuery etc.)

• Proxy configurations

• Certificate trust / handling TLS and encrypted traffic

• Not pixel-perfect

Device Testing Overview

32

• Recommendations for working through Device-specific problems:

– Run a Proxy to view the actual traffic coming and going to your device.

– If it’s a hybrid app (Fiori/UI5) check it in the native device browser as well as the Cordova package / app format.

Device Testing in Your Lab

33

• Do some real-world, manual testing on top of the normal test phases:– Run through a “10 minute day-in-the-life” for your

users • Key everyday functions (e.g. Report some graffiti)• Use all the native device functionality that they will

(e.g. Camera, GPS)

– Measure:• Battery drain (Enable % remaining mode and note it

down)• Ease of use – do you find yourself multiple tapping?

Do you miss touch hotspots?• Responsiveness (stop watch!) – confirm this with

automated testing later.

– Iterate and average-out your findings to discover problem areas

Device Testing in the Field

34

• Know your Application Service End Points – understand what your application talks to and when – are these end points available both on your internal network and over cellular internet?

• Understand Quirks of different Device behaviour before your users report problems.

• Understand likely battery life for your organisational use cases.

Device Testing Outcomes

35

How will my app perform in changed network conditions?

Network Testing in Your Lab

36

Options Pros Cons

In-Browser Tools(e.g. Chrome Development Mode)

• Free• Generally only a simple

bandwidth throttle• Chrome recently added in

Throttling based on average network characteristics

• Hybrid/Web Only• Physical Device may behave differently.• No Latency control• Cannot throttle non browser traffic

Network Emulation,Proxies

• Test from Device• Can change network during

transactions• Control latency, bandwidth AND

packet loss• All Device traffic

• Complex Set-up• Licensing cost depending on tool

Chrome Development Tools

37

• Try it yourself on any web app.

• Just hit CTRL+SHIFT+I in Chrome

• Then Click the device icon.

Set up your Network Virtualization on your Lab Access Point:• Test your key transactions.

Aim to test varying scenarios:• App Behaviour – Pass/Fail:

– Disconnected mid-way through a transaction (unplanned offline)– Coming back online whilst being offline for a period of time

• App Performance – Response Time:– Slowest Speed (3G Poor)– Average Speed (4G Normal)– Best Speed (Office Wi-Fi? Depends!)

Network virtualization options:• We have used Shunra (now HP) in the past – they provided global network profiles, ability to record

your own network conditions, replay transactions.• Alternatives: Charles (for single apps), Packetstorm, Dummynet.

Network Emulation in Your Lab

38

- Examples extrapolated from multiple tests and profiling of network conditions in Melbourne city and Accenture office.- Benchmark your own network data!

Example Network Conditions

39

Network

Emulated

Latency (ms) Data Rate

(Down/Up Kbps)

Packet Loss

Wi-Fi 35 13,000/9,000 0%

3G Good 104 2,500/750 0%

3G Fair 125 1,590/400 1%

3G Poor 160 700/100 1.5%

• Check your network coverage map!– Telstra, Optus, Vodafone all

publish fairly granular maps.

– Should I expect 4G Performance?

• (Optional) Actual location sampling – Provides for strange locations

(like a particular known blackspot)

– Assures business users that you’ve been where they will be.

Network Testing in The Field

40

Network Testing Outcomes - Example

41

Key Takeaways

Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for TestingSkills to LearnTools and techniques

• Be curious about your application! - Testing Mobility applications is different, there are a broader scope of factors to cover than traditional enterprise roll outs.

• Set yourself up a Lab, and grow a mobile testing capability. This will only become more important as “Mobile” grows to encompass the Internet of Things and becomes more than phones and tablets.

• Know as much about your user as possible, and test out their Total User Experience before they do. Ensure they are delighted, and you will get the highest ROI from your implementation.

Mobility Testing - Key Takeaways

43

E: eneko.bilbao@accenture.com

https://au.linkedin.com/in/eneko24

@enekob

• Smartphone Penetration Australia http://www.statista.com/statistics/257041/smartphone-user-penetration-in-australia/

• Mobile Wireless Performance Solutions and Problems http://www.gartner.com/document/2681417

• Internet Backbone Latency https://www.dotcom-tools.com/internet-backbone-latency.aspx

• ENIAC computer - http://en.wikipedia.org/wiki/Computer

• Telstra Network Coverage Map: https://www.telstra.com.au/coverage-networks/our-coverage

• Shunra User Guide: http://media.shunra.com/datasheets/ (recently replaced with HP documentation)

• Word Clouds from http://tagul.com

• Charles Proxy can be found at http://www.charlesproxy.com/ (no affiliation)

• Device Fragmentation http://www.businessinsider.com.au/android-fragmentation-2014-8

References

46