1. 1. Assuring Mobile Test Coverage May 28, 2015
2. 2. Answering the most common question: What should I test on to ensure efficient market coverage?
3. 3. Introduction
4. 4. Copyright Perfecto Mobile Why Apps Fail Report, 2014 Our users Our Testers
5. 5. v Why Is It Hard? New App releases once in 2 months New mobile devices every 8 months New Major OS release every 6 months Major mobile operating systems are expected to go from 10 months to six months between releases, while mobile devices will accelerate from 10 months to eight months, according to Gartner Inc. Applications updates will be cut in half from four to two months, the Stamford, Conn.-based analysis firm predicts. Source: Gartner & Stamford
6. 6. What does it take to cover the REAL end user experience? vv Device & Platform Model OS Version Screen Size Form Factors vv Environment Location Network vv Conditions Signal Strength Load Others
7. 7. Test Coverage Optimization Process 1. Coverage 2. Lab Testing Market Me My Space Popular & emerging Analytics Industry trends & insights The Index Teams Projects SDLC Cycle Requirements
8. 8. Coverage Target Minimum recommended device coverage driven by market share and device characteristics Minimum plus complementing market lead devices, and characteristics Standard plus consideration for portability, legacy and new device coverage 80%50%30% 10-16 Devices 17-25 Devices 26-32 Devices
9. 9. Discovery Process vv Analytics Data Exists? YES NO vv BU Requirements/ Weights? APP TYPE - B2C/B2E/Web/RWD? - Web, Native, Hybrid? BYOD vs. Market Special Prioritizations? - Demographics - Platforms - Geography - Devices vv Process & Maturity Related (Lab Size)? - Agile/Waterfall - Release cadence - Automation Coverage vv Customize Consideration Weights - Analytics - My Space - Market
10. 10. vv Device Related Hardware Mix OS Mix Form Factors (Screen Size) Memory What Do You Need To Test On? Key Factors vv Mix Categories Reference Devices/OS Popular Devices/OS Legacy Devices/OS New & Emerging
11. 11. Market Device/OS Categories Market Reference Devices/OS Popular Devices/OS Legacy Devices/OS Emerging Devices/OS
12. 12. OS Fragmentation & Legacy Examples: Samsung S2, Samsung S3, HTC Incredible Examples: iPhone 3GS, iPhone 4, iPhone 4S, iPods Market
13. 13. Example: Android L Ramps Up As Android M Evolves Source: Android Central Market Total of only 35 unique models got/in progress of getting Android L
14. 14. MAY OCTOBERJUNE JULY AUGUST SEPTEMBER Google IO Event Apple WWDC Enterprise Apps World UK Sony Xperia Z4 Tablet New iPhones & iPads Samsung Note 5 Motorola Moto X (New) iOS 9 Beta Windows 10 for Phones EVENTS DEVICES PLATFORM Mobile Market Calendar 2015 Android M ?
15. 15. vv Coverage % 30% 50% 80% vv Customer Data What Do You Need To Test On? Key Factors
16. 16. v My Space Index Reports Base Line Perfecto Mobile Cloud Data App Store Monitoring Categories Target Geographies OS and Devices Aging Demographics Vertical Competitive Data: Devices Features Technology
17. 17. U.S Financial Vertical Q1 2015 Snapshot (Cloud) 60-75% iOS Devices 25-40% Androids Are: P1 - Samsungs P2 - Google Nexus P3- HTC P4- Amazon Kindle
18. 18. v Me Look inside yourself and you Can see the universe. - Brian Herbert
19. 19. The mobile test coverage Index MAKE SURE YOUR APP WORKS ON THE RIGHT DEVICES Read the Mobile Test Coverage Index
20. 20. v Mobile Test Coverage Index Q1 2015 Scope Data Sources Covers market share data & Upcoming Data Devices Wearables Browsers Environments Verticals Country Breakdown Perfecto Mobile Cloud Data Market Research Analyst Insights EU5 & U.S Geos Factors Supported: 30%-80% coverage index report Not Covered OSs Screen Size Hardware
21. 21. Q1 2015 U.S Index Based on: 4000 Proprietary Devices Data 360,000 End-User Data Market Analytics
22. 22. Example Perfecto Mobile Cloud Data