mobile test automation
TRANSCRIPT
Mobile Test Automation
Andreas Jakl
@andijakl
Mobile Developer After-Work #7 – Xamarin: April 9th, 2015
Andreas Jakl
Mobility Evangelist, Tieto
– @andijakl
History
– Mopius: Startup-Founder (NFC & Mobile Apps)
– Nokia, Finland: Technology Wizard
– FH Hagenberg: Assistant Professor Mobile Computing
– Siemens / BenQ Mobile, Germany: Augmented Reality-Apps
Mobile Test Automation - Andreas Jakl2
TESTING – WHY?
Versions & More
Mobile Test Automation - Andreas Jakl3
Slow uptake of new
versions
Manufacturer-
modified UIs
Countless screen
sizes & densities
Mobile Test Automation - Andreas Jakl4
https://developer.android.com/about/dashboards/index.html
http://stats.unity3d.com/mobile/os-android.html
Fast adoption of new OS
– Design updates
– API & architecture changes
(e.g., 64 bit enforcement)
Quick update of apps required
Same OS version, but many iOS
devices in use
Mobile Test Automation - Andreas Jakl5
https://developer.apple.com/support/appstore/
http://stats.unity3d.com/mobile/device-ios.html
Similar hardware & software base
– 512 MB vs 1+ GB memory
– Processor speed
– 4 supported screen resolutions
Relevant: v8(.1) > 90%
– But: Windows 10 upcoming
• More resolutions, etc.
Mobile Test Automation - Andreas Jakl6
http://blogs.windows.com/buildingapps/2014/12/22/windows-and-windows-phone-store-trends-december-2014-update/
http://blog.adduplex.com/2015/03/adduplex-windows-phone-statistics.html
IMPACT OF TESTING
Successful?
Mobile Test Automation - Andreas Jakl7
Reviews vs Downloads
Review count & rating correlate to downloads
Mobile Test Automation - Andreas Jakl8
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013
Data from 13 million user reviews, 171,439 apps
Average rating: 3.9
(standard deviation: 1.48)
0
2 000 000
4 000 000
6 000 000
8 000 000
Review Analysis
Analysis
– Most frequent complaints
• Functional errors
• Feature requests
• App crashes
Mobile Test Automation - Andreas Jakl9
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014
Review Analysis
Analysis
– Most negative impact
• Privacy and ethical issues
• Hidden app costs
Mobile Test Automation - Andreas Jakl10
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014
Customer Retention
Quality (feature)
upgrades
– Up to 3x
improvement in
survival rates
Mobile Test Automation - Andreas Jakl11
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 Month after Download 6 Months after Download 1 Year after Download
Customer Retention in Mobile Apps
TESTING STRATEGIES
Manual
Mobile Test Automation - Andreas Jakl12
Capturing Crash Data
Mobile Test Automation - Andreas Jakl13
Monitoring User Reviews
Mobile Test Automation - Andreas Jakl14
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013
Static Code Analysis
Mobile Test Automation - Andreas Jakl15
Emulator / Simulator
Mobile Test Automation - Andreas Jakl16
Emulator
Emulates real
hardware
+ more accurate
– slow
Android
Simulator
Simulates device
behavior
+ faster
– less accurate
iOS / Windows
Reliable?
Mobile Test Automation - Andreas Jakl17
0%of your users will run your app on an emulator
Own Devices
Mobile Test Automation - Andreas Jakl18
How many test devices do you have?
Open Device Lab Vienna
Mobile Test Automation - Andreas Jakl19
Free testing on real devices
http://www.open-device-lab-vienna.at/
Automated UI Tests
Mobile Test Automation - Andreas Jakl20
How many tests can you run per hour / build / release?
Number of builds
Cost
Automated tests
Manual tests
TEST CLOUD
Automated Testing on 1000s of Devices
Mobile Test Automation - Andreas Jakl21
Xamarin Test Cloud
Automated tests
– UI interaction
– Device features (GPS, etc)
– Performance
1,300+ devices
– Android & iOS
– testcloud.xamarin.com/devices
Mobile Test Automation - Andreas Jakl22
Workflow
Mobile Test Automation - Andreas Jakl23
+ Continuous Integration
Mobile Test Automation - Andreas Jakl24
Calabash Xamarin.UiTest
Test Frameworks
Mobile Test Automation - Andreas Jakl25
Write tests in Ruby C#
Run with Gherkin NUnit
Executing Tests
Mobile Test Automation - Andreas Jakl26
Process
1. Write Tests
2. Setup Project for Testing
3. Test Locally
4. Create app on Xamarin Test Cloud
5. Upload & Run on Cloud via Command Line
Mobile Test Automation - Andreas Jakl27
Writing Tests: NUnit
Mobile Test Automation - Andreas Jakl28
Generic tests possible
with Xamarin.Forms
Test Setup
Mobile Test Automation - Andreas Jakl29
Platform-specific
initialization & test
differences
Setup Project
Mobile Test Automation - Andreas Jakl30
Disable shared
Mono runtime
API Key
Mobile Test Automation - Andreas Jakl31
Create team & app to
get unique API key
Run Tests
Locally
Mobile Test Automation - Andreas Jakl32
Test on your
devices first
Mobile Test Automation - Andreas Jakl33
Cloud Setup
Choose devices
Mobile Test Automation - Andreas Jakl34
Cloud Setup
System settings
Mobile Test Automation - Andreas Jakl35
Running Tests
in the cloud
Command line
only
Mobile Test Automation - Andreas Jakl36
Test results
Dashboard
Mobile Test Automation - Andreas Jakl37
Test results
Screenshots
Custom Screenshots
Take & describe screenshots
Enable local screenshots
Mobile Test Automation - Andreas Jakl38
[TestCase (TestName = "Click button and show success message")]public void TestCaseSuccess (){
AppResult[] result = _app.Query(InitialMessage);Assert.IsTrue(result.Any(), "The initial message string isn't correct");_app.Screenshot ("Initial screen");
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();
Custom Screenshots
Take & describe screenshots
Enable local screenshots
Mobile Test Automation - Andreas Jakl39
[TestCase (TestName = "Click button and show success message")]public void TestCaseSuccess (){
AppResult[] result = _app.Query(InitialMessage);Assert.IsTrue(result.Any(), "The initial message string isn't correct");_app.Screenshot ("Initial screen");
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();
Testing Finished App Packages
Find IDs of UI elements
Mobile Test Automation - Andreas Jakl40
if (TestEnvironment.Platform.Equals(TestPlatform.Local)){
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();
_app.Repl ();// In Bash, enter app.Query(c=>c.Button()) or c.All()
}
Testing Finished App Packages
Find IDs of UI elements
Mobile Test Automation - Andreas Jakl41
if (TestEnvironment.Platform.Equals(TestPlatform.Local)){
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").EnableLocalScreenshots().StartApp();
_app.Repl ();// In Bash, enter app.Query(c=>c.All())
}
Alternatives
Testdroid
– testdroid.com/
Perfecto Mobile
– perfectomobile.com/solution/test-automation
Mobile Test Automation - Andreas Jakl42
Thank You!
Andreas Jakl
@andijakl
Mobile Test Automation - Andreas Jakl43