the current state of mobile testing by stephen janaway
Post on 24-Apr-2015
302 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Current State of Mobile Tes3ng
Stephen Janaway
© Stephen Janaway 2014
Who am I? Stephen Janaway
• 14 years experience in mobile soDware tes3ng.
• Mobile devices • Mobile networks • Mobile applica3ons • Have worked for companies such as
Ericsson, Motorola and Nokia.
• Work for the NET-‐A-‐PORTER GROUP.
• www.stephenjanaway.co.uk • @stephenjanaway
© Stephen Janaway 2014
© Stephen Janaway 2014
www.tes3nginthepub.com @tes3nginthepub Search for “Tes3ng In The Pub’” on iTunes
Today
• A bit of history. • An overview of mobile. • Why is mobile different? • Where are we now? • What’s next? • Your mobile tes3ng future.
© Stephen Janaway 2014
A Bit of History
"Mobile radio telephone" by Original uploader was Hackgillam at en.wikipedia -‐ Transferred from en.wikipedia; transferred to Commons by User:LiDarn using CommonsHelper.. Licensed under Public domain via Wikimedia Commons -‐ h`p://commons.wikimedia.org/wiki/File:Mobile_radio_telephone.jpg#mediaviewer/File:Mobile_radio_telephone.jpg
A Bit of History
"2007Computex e21-‐Mar3nCooper" by 2007Computex_e21Forum-‐Mar3nCooper.jpg: Rico Shenderiva3ve work: PowellS (talk) -‐ 2007Computex_e21Forum-‐Mar3nCooper.jpg. Licensed under Crea3ve Commons A`ribu3on-‐Share Alike 3.0 via Wikimedia Commons -‐ h`p://commons.wikimedia.org/wiki/File:2007Computex_e21-‐Mar3nCooper.jpg#mediaviewer/File:2007Computex_e21-‐
Mar3nCooper.jpg
Mobile devices are
• Smart Phones • Feature Phones • Tablets • E-‐readers • Big screens, small screens, touch screens and keypads.
© Stephen Janaway 2014
And now…
• Fitness devices. • Smart watches. • Wearable devices. • Beacons. • Household gadgets.
© Stephen Janaway 2014
Mobile devices are much more • Alarm Clocks • Maps • Link to loved ones • Hand held shops • Tickets • Music Players • Books • Games Players • Cameras • ……
• These are key experiences.
© Stephen Janaway 2014
WHY IS MOBILE DIFFERENT?
© Stephen Janaway 2014
MOBILE USERS ARE MORE EMOTIONALLY ATTACHED
© Stephen Janaway 2014
MOBILE USERS ARE NOT SECOND CLASS CITIZENS
© Stephen Janaway 2014
?
© Stephen Janaway 2014
?
BUT QUALITY IS CURRENTLY STILL NOT GREAT…
AND USERS INCREASINGLY DON’T TOLERATE THIS
MOBILE USERS CAN LEAVE BAD FEEDBACK MUCH MORE QUICKLY
What Makes Mobile Tes3ng Different?
• Many plaoorms. • Many OS versions in the field. • Many different hardware configura3ons. • Devices with limited memory and processing capability.
• Limited ba`ery life. • Network connec3vity. • Applica3on stores. • Low maturity of automa3on solu3ons.
© Stephen Janaway 2014
Func3onal Mobile Tes3ng
• Selec3ng categories. • Accounts. • Shopping bag. • Opening and closing. • Images.
© Stephen Janaway 2014
Usability Ma`ers
• Usability is more important in the mobile world: – Smaller screen. – More difficult to interact with and typically used one handed.
Understanding the User
• In app analy3cs. • Website analy3cs. • User persona’s. • Similar applica3ons.
© Stephen Janaway 2014
Consistency
• Think about what a mobile user would want. • Something that looks similar to desktop? • Takes advantage of mobile func3onality: – Loca3on. – Mobile search. – M-‐commerce.
• A design that can be viewed on a small screen and in bright light.
© Stephen Janaway 2014
Engagement
• People typically have more than one device. • They get distracted. • How engaging is your app or mobile website?
© Stephen Janaway 2014
Understand Gestures
© Stephen Janaway 2014
Mobile specifics
• Download and install. • Un-‐install. • Upgrade. • Rota3on. • Sensors. • Loca3on. • External media.
© Stephen Janaway 2014
Interac3ons
• While your applica3on runs, the phone is: – Talking to the network – Working out where you are – Maybe receiving calls, texts, Facebook and Twi`er updates.
• What happens when you have your applica3on running, and then get a call, and a text, whilst uploading video to YouTube? Test for it and find out.
Performance
• Limited processor speeds. • Limited on-‐board memory. • Typically not turned off very oDen. • Don’t forget -‐ most apps are just clients to web services. Use your web performance tes3ng skills.
© Stephen Janaway 2014
How Much Data?
• Check for excessive data consump3on. • Using on device data measuring tools. • Using a proxy.
© Stephen Janaway 2014
The Current State Of Mobile
© Stephen Janaway 2014
Subscrip3ons vs. People
© Stephen Janaway 2014
The mobile market is huge
© Stephen Janaway 2014
It’s Increasingly Smartphones and Tablets
© Stephen Janaway 2014
Different OS’s
© Stephen Janaway 2014
Different Mobile Browsers
© Stephen Janaway 2014
The Future
© Stephen Janaway 2014
More Devices = More Tes3ng (?)
© Stephen Janaway 2014
More Wearables
© Stephen Janaway 2014
OS’s
© Stephen Janaway 2014
Source: gs.statcounter.com
Android versions
Source: h`p://opensignal.com/reports/fragmenta3on-‐2013/
Screen Sizes
© Stephen Janaway 2014
Deciding What to Test
• Know your predicated customers (not your current ones).
• Find out what OS updates are coming up. • Get as much data on device usage that you can: – In app analy3cs. – Web searches. – Similar apps.
© Stephen Janaway 2014
Fragmenta3on
• Lots of different hardware and soDware configura3ons to support.
• Op3ons: – Don’t use them at all – use simulators or user agents.
– Buy all or some of them. – Rent all or some of them. – Use a cloud based service (Perfecto Mobile, DeviceAnywhere).
© Stephen Janaway 2014
My Recommenda3on
• Buy 2 from the top, 2 from the middle and 2 from the bo`om.
• Understand what your customers use or will use. • Test first on simulators. • Use cloud services if needs be, to fill any gaps.
• Never just use browser emula3on for mobile websites.
© Stephen Janaway 2014
Making More Use of Your Users
© Stephen Janaway 2014
Dog fooding/ Alpha/ Beta
• Get an internal user group going. • Distribute app via TestFlight, TestFairy or similar.
• Alpha/ Beta test via app stores.
© Stephen Janaway 2014
Using Cloud Based Tes3ng Services
Where Appropriate
© Stephen Janaway 2014
Not Buying All The Devices
© Stephen Janaway 2014
Open Device Labs
• Borrow devices, normally for free.
• h`p://opendevicelab.com • Open Device Labs = shared community pools of internet connected devices for tes3ng purposes.
© Stephen Janaway 2014
(Even) More Loca3on Awareness
© Stephen Janaway 2014
More Financial Transac3ons Will Happen On Mobile
© Stephen Janaway 2014
M-‐Commerce
• Google Wallet. • Paypal. • Scanning barcodes. • Contactless payments. • Displaying bar or QR codes. • Loca3on based.
© Stephen Janaway 2014
Security
• Big recent rise in the amount of malware targeted at mobile devices.
© Stephen Janaway 2014
© Stephen Janaway 2014
© Stephen Janaway 2014
© Stephen Janaway 2014
Is This Appropriate?
© Stephen Janaway 2014
Smartphones Everywhere = Your App Everywhere
© Stephen Janaway 2014
Accessibility
• How readable is the app/ website with a large font?
• Is there voice ac3va3on / control? • Are there text alterna3ves to images? • Tools: – TalkBack (Android) – Voice Over (iOS)
© Stephen Janaway 2014
Media Usage Is Changing
© Stephen Janaway 2014
We Are Consuming Content Differently
© Stephen Janaway 2014
Voice Ac3va3on
• Siri – iOS • Google Now – Android • Also on Windows Phone and Blackberry. • Is the app voice ac3vated? • What happens when you ac3vate voice with the app running?
© Stephen Janaway 2014
It’s No Longer OK to Have a Bad Mobile Presence
© Stephen Janaway 2014
© Stephen Janaway 2014
?
Responsive Design
© Stephen Janaway 2014
h`p://designmodo.com/responsive-‐test/
Your Mobile Tes3ng Future
You Can Avoid The Common Mobile Tes3ng Gotcha’s
© Stephen Janaway 2014
Only Tes3ng On Simulators
© Stephen Janaway 2014
Using Simulators and Browser Emula3on
• Simulators: – Normally found within OS SDKs. – Good to start tes3ng with. – Do not recreate exactly how a device func3ons.
• Browser Emula3on: – User Agent Switching: – Within a browser (Chrome developer tool is best). – Changes how the browser renders the page.
© Stephen Janaway 2014
Real Devices
• Keep a device library or rent devices from one. • Cannot pick all devices. • Choose a few at the top and a few at the bo`om – device ranking.
• Do not forget to find out what is being used by customers.
• Can be expensive and needs maintenance.
© Stephen Janaway 2014
Staying In The Office
© Stephen Janaway 2014
Don’t Just Test Here…
On Here, and At Here…
Out Here…
In Here…
Forgeung About Ba`ery Life
© Stephen Janaway 2014
Ba`ery Life
© Stephen Janaway 2014
Forgeung That It Takes Time To Submit Apps
© Stephen Janaway 2014
Apps Stores • Submission criteria vary:
© Stephen Janaway 2014
Store Time Details
Apple Apps Store 2 weeks Significant checking/ tes3ng
Google Play ½ day Limited tes3ng
Windows Phone Store
4 days
Blackberry World 1 day
Amazon App Store 1-‐2 weeks
Ignoring OS Updates
© Stephen Janaway 2014
OS Updates
• Always regression test your app or website before an OS update.
• Get on the developer program for early OS versions.
• Users update more quickly on iOS than Android.
• Carriers oDen control update cycles. • Decide beforehand how many OS version you will/ can support.
© Stephen Janaway 2014
Confusing Dev With Live
© Stephen Janaway 2014
Thinking Mobile Test Automa3on Is Like the Web Test Automa3on
© Stephen Janaway 2014
Automa3on
• The holy grail to solving the fragmenta3on tes3ng problem?
• Caveat – mobile test automa3on is like desktop test automa3on 6 years ago.
• You may want to consider pushing for more unit and integra3on tests before spending 3me on mobile UI automa3on.
© Stephen Janaway 2014
Some Op3ons Tool OS Comments
Appium iOS/ Android Probably the best out there, but s3ll flaky
iOS Driver iOS OK but cannot test apps on real devices
Frank iOS Cannot test Webviews
Zucchini iOS Cannot test Webviews
Calabash iOS/ Android Requires embedded server
Robo3um Android Android only
Selendroid Android Used at Facebook
Monkey Talk iOS/ Android Requires agent to be added to app.
And many, many more, some of which are even s3ll supported….
© Stephen Janaway 2014
Automa3on Sugges3ons
• Determine if you require cross plaoorm support.
• Avoid tools that use embedded servers or agents – they alter the produc3on code.
• Look for tools with ac3ve online communi3es. • Consider the future.
© Stephen Janaway 2014
My Picks • Appium: – Cross plaoorm. – Well supported user community. – About to be more closely integrated in Selenium 3.
• iOS Driver: – Well supported user community. – About to be more closely integrated in Selenium 3.
• Calabash: – Well supported user community. – Heavily used at the BBC.
© Stephen Janaway 2014
So……
Mobile Test Strategy – Top 5 Tips 1. Understand the poten3al user: – Which plaoorms and devices will they use? – How will they use the app or website?
2. Understand what OS versions are being used and when new ones are coming.
3. Test using simulators, real devices and users. 4. Understand the cost of automa3on: – It is where desktop automa3on was 6 years ago. – Best combined with extensive unit and integra3on tes3ng.
5. Use an exploratory approach to manual tes3ng: – Mobile device projects are typically short.
© Stephen Janaway 2014
“Be proud, your work will soon be in the hands of millions”
Stephen Janaway www.stephenjanaway.co.uk @stephenjanaway www.tes3nginthepub.com
Ques3ons?
top related