Prioritizing The Devices To Test Your App On:A Case Study Of Android Game Apps

Hammad Khalid

Emad Shihab

Mei Nagappan

AhmedHassan

The Android Ecosystem has huge number of Stakeholders

1B+ users$4B+ revenues

2

1M+ apps150K+ developers

3

One Challenge faced by developers

4

Android Device Fragmentation

As of Aug 2014 – 19K devices

5

Android Device Fragmentation

As of Aug 2014 – 19K devices

94% of developers not working on Android cited fragmentation as their

primary reason

6

Which Devices should I test my App on?

7

Which Devices should I test my App on?

8

Current Solution

9

Problem with Market Share

10

Market Share does not care about what people are saying about your app.

11

Example - 100 Doors 2013

12

Examine reviews from Motorola Droid X

13

77% of the reviews from Motorola Droid X are Bad reviews

14

77% of the reviews from Motorola Droid X are Bad reviews

The rating for the app is brought down by users of Motorola Droid X

15

Samsung G

alaxy S

3

Samsung G

alaxy S2

Samsung G

alaxy S

Samsung G

alaxy Ace

Samsung G

alaxy Note

Samsung G

alaxy Y

HTC Desire HD

Asus Nexu

s 7

Samsung G

alaxy Tab 10.1

Motorola Droid RAZR02468

10

However Motorola Droid X is not in the top 10 devices by Market Share

16

17

Ratings from Reviews correlated with Downloads

Since Ratings are important

Prioritize Devices based on

Review Share

18

19

Definition

Review Share – The percentage of reviews that an app gets

from a device.

20

Approach to Calculate Review Share

21

Approach to Calculate Review Share

22

Approach to Calculate Review Share

Review Share

3/5 2/5

23

Case Study on 99 Game Apps in Google Play

24

Case Study on 99 Game Apps in Google Play

But why Game Apps?

25Books & Reference

Comics

Education

Finance

Librarie

s & Demo

Media & Video

Music & Audio

Personaliza

tion

Productivity

Socia

lTools

Travel & Lo

cal

Games0

5000100001500020000250003000035000

Most Popular - There are 35K Game Apps with > 500K downloads

26

144K+ Most Useful Reviews

27

Device info present in ~ 90K reviews

28

187 Devices – Fragmentation does exist

Device info present in ~ 90K reviews

29

% of reviews from a device

Do some devices give worse ratings?

Predict the devices to test

your new app on

30

Predict the devices to test

your new app on

Do some devices give worse ratings?

% of reviews from a device

31

~20% of the Devices Account for 80% of the Reviews

32

Total - Min 38, Max 132 Devices

33

80% of reviews from just 13 - 45 devices

34

On average, 33% of all devices account for 80% of reviews.

% of reviews from a device

Do some devices give worse ratings?

Predict the devices to test

your new app on

35

On average, 33% of all devices account for 80% of reviews.

% of reviews from a device

Do some devices give worse ratings?

Predict the devices to test

your new app on

36

% of bad ratings to all ratings from a device to an app

37

% of bad ratings to all ratings from a device to an app

38

7/10 reviews from Motorola Droid X to 100 Doors app are 1 or 2 Star Ratings

39

% of bad to all ratings from a device to an app for all apps and all devices

Statistical Test

Scott-Knott

40

% of bad to all ratings from a device to an app for all apps and all devices

Statistical Test

Scott-Knott

Grouping of Devices

41

Some devices give worse ratings

42

Some devices give worse ratings

43

Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2

44

Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2

12% - Performance

45

Manual Analysis of 677 1 or 2 star reviews from Motorola Droid X2

12% - Performance 6% - UI

46

On average, 33% of all devices account for 80% of reviews.

Statistical evidence that some devices give worse ratings than others.

% of reviews from a device

Do some devices give worse ratings?

Predict the devices to test

your new app on

47

On average, 33% of all devices account for 80% of reviews.

Statistical evidence that some devices give worse ratings than others.

% of reviews from a device

Do some devices give worse ratings?

Predict the devices to test

your new app on

48

Learn what devices review 98 of the 99 Game apps

49

Apply it to the remaining app

50

Compare Top 10 devices we predict vs Top 10 actual devices

51

For most apps 7 out of

top 10 devices with most reviews are common

Compare Top 10 devices we predict vs Top 10 actual devices

52

For most apps only 7% of reviews are

missed

Compare Top 10 devices we predict vs Top 10 actual devices

53

On average, 33% of all devices account for 80% of reviews.

Statistical evidence that some devices give worse ratings than others.

App developer can focus testing even before first release

% of reviews from a device

Do some devices give worse ratings?

Can we predict the devices?

54

Take Away

55

Take Away

1. Android Fragmentation is not as bad one would think (in practice).

56

Take Away

2. App developers could use device information from reviews to prioritize their testing efforts

1. Android Fragmentation is not as bad one would think (in practice).

57

All results Generalize to Paid Game Apps…

58

… and to 4 other categories of apps

59

Summary

60

61

62

63