prioritizing the devices to test your app on: a case study of android game apps
Post on 15-Apr-2017
62 Views
Preview:
TRANSCRIPT
Prioritizing The Devices To Test Your App On:A Case Study Of Android Game Apps
Hammad Khalid
Emad Shihab
Mei Nagappan
AhmedHassan
The Android Ecosystem has huge number of Stakeholders
1B+ users$4B+ revenues
2
1M+ apps150K+ developers
3
One Challenge faced by developers
4
Android Device Fragmentation
As of Aug 2014 – 19K devices
5
Android Device Fragmentation
As of Aug 2014 – 19K devices
94% of developers not working on Android cited fragmentation as their
primary reason
6
Which Devices should I test my App on?
7
Which Devices should I test my App on?
8
Current Solution
9
Problem with Market Share
10
Market Share does not care about what people are saying about your app.
11
Example - 100 Doors 2013
12
Examine reviews from Motorola Droid X
13
77% of the reviews from Motorola Droid X are Bad reviews
14
77% of the reviews from Motorola Droid X are Bad reviews
The rating for the app is brought down by users of Motorola Droid X
15
Samsung G
alaxy S
3
Samsung G
alaxy S2
Samsung G
alaxy S
Samsung G
alaxy Ace
Samsung G
alaxy Note
Samsung G
alaxy Y
HTC Desire HD
Asus Nexu
s 7
Samsung G
alaxy Tab 10.1
Motorola Droid RAZR02468
10
However Motorola Droid X is not in the top 10 devices by Market Share
16
17
Ratings from Reviews correlated with Downloads
Since Ratings are important
Prioritize Devices based on
Review Share
18
19
Definition
Review Share – The percentage of reviews that an app gets
from a device.
20
Approach to Calculate Review Share
21
Approach to Calculate Review Share
22
Approach to Calculate Review Share
Review Share
3/5 2/5
23
Case Study on 99 Game Apps in Google Play
24
Case Study on 99 Game Apps in Google Play
But why Game Apps?
25Books & Reference
Comics
Education
Finance
Librarie
s & Demo
Media & Video
Music & Audio
Personaliza
tion
Productivity
Socia
lTools
Travel & Lo
cal
Games0
5000100001500020000250003000035000
Most Popular - There are 35K Game Apps with > 500K downloads
26
144K+ Most Useful Reviews
27
Device info present in ~ 90K reviews
28
187 Devices – Fragmentation does exist
Device info present in ~ 90K reviews
29
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
30
Predict the devices to test
your new app on
Do some devices give worse ratings?
% of reviews from a device
31
~20% of the Devices Account for 80% of the Reviews
32
Total - Min 38, Max 132 Devices
33
80% of reviews from just 13 - 45 devices
34
On average, 33% of all devices account for 80% of reviews.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
35
On average, 33% of all devices account for 80% of reviews.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
36
% of bad ratings to all ratings from a device to an app
37
% of bad ratings to all ratings from a device to an app
38
7/10 reviews from Motorola Droid X to 100 Doors app are 1 or 2 Star Ratings
39
% of bad to all ratings from a device to an app for all apps and all devices
Statistical Test
Scott-Knott
40
% of bad to all ratings from a device to an app for all apps and all devices
Statistical Test
Scott-Knott
Grouping of Devices
41
Some devices give worse ratings
42
Some devices give worse ratings
43
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
44
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
12% - Performance
45
Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2
12% - Performance 6% - UI
46
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
47
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
% of reviews from a device
Do some devices give worse ratings?
Predict the devices to test
your new app on
48
Learn what devices review 98 of the 99 Game apps
49
Apply it to the remaining app
50
Compare Top 10 devices we predict vs Top 10 actual devices
51
For most apps 7 out of
top 10 devices with most reviews are common
Compare Top 10 devices we predict vs Top 10 actual devices
52
For most apps only 7% of reviews are
missed
Compare Top 10 devices we predict vs Top 10 actual devices
53
On average, 33% of all devices account for 80% of reviews.
Statistical evidence that some devices give worse ratings than others.
App developer can focus testing even before first release
% of reviews from a device
Do some devices give worse ratings?
Can we predict the devices?
54
Take Away
55
Take Away
1. Android Fragmentation is not as bad one would think (in practice).
56
Take Away
2. App developers could use device information from reviews to prioritize their testing efforts
1. Android Fragmentation is not as bad one would think (in practice).
57
All results Generalize to Paid Game Apps…
58
… and to 4 other categories of apps
59
Summary
60
61
62
63
top related