kyungmin lee *, jason flinn *, t.j. giuli +, brian noble *, and christopher peplin + university of...
TRANSCRIPT
Kyungmin Lee*, Jason Flinn*, T.J. Giuli+, Brian Noble*, and Christopher Peplin+
University of Michigan* Ford Motor Company+
AMC: Verifying User Interface Properties for Vehicular Applications
Where do we use mobile apps?
2Kyungmin Lee
Walking Driving Everywhere we go(Not-too-distant future)
Where do we use mobile apps?
3Kyungmin Lee
Walking Driving Everywhere we go(Not-too-distant future)
Mobile apps in a vehicle
4
Pros:- High usability standard- Good form factorCons:- Cost of deployment
- Limited applications
Pros: - Diverse applications- Easy deploymentCons:- Distracting apps- Poor form factor
Kyungmin Lee
Mobile apps in a vehicle
5
Pros:- High usability - Good form factor
Pros: - Diverse apps- Easy deployment
Kyungmin Lee
How can we get the best of both worlds?How can we get the best of both worlds?
Our approach
• Create a tool (AMC) that– Automatically explores the application– No source code & app-specific knowledge– Pinpoints UI guideline violations
6Kyungmin Lee
AMC’s benefits
• Developers– Test own apps– Early design feedback
7
• User interface experts– Fewer properties to check– Focus on hard properties
Kyungmin Lee
AMC’s benefits
• Developers– Test own apps– Early design feedback
8
• User interface experts– Fewer properties to check– Focus on hard properties
Kyungmin Lee
<AMC’s results>20X reduction in testing
False negative: 2%False positive: 0%
<AMC’s results>20X reduction in testing
False negative: 2%False positive: 0%
Outline
• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion
9Kyungmin Lee
How to use AMC
10
Run AMC Get violation report
Violation reportButton size:1) Button A at state 1 2) …Contrast ratio:1)…2)…
Violation reportButton size:1) Button A at state 1 2) …Contrast ratio:1)…2)…
Kyungmin Lee
- State exploration
- Find violations
- State exploration
- Find violations
ApplicationApplication
MonkeyMonkey
Hierarchy Viewer
Hierarchy Viewer
Modified ApplicationFramewo
rk
Modified ApplicationFramewo
rk
App to test
How does AMC explore?
11
State #0
Kyungmin Lee
How does AMC explore?
12
State #0
Current Location
State #1
Kyungmin Lee
How does AMC explore?
13Kyungmin Lee
How do we define a state
• Different screens -> Different states
14Kyungmin Lee
vs.
Fine-tuning state definition
• Naïve approach: every unique screen -> state
• Solution: Structure hash– Same structure, different content -> same state– Avoids state explosion
15
vs.
Kyungmin Lee
Fine-tuning state definition
• Structure hash is not enough…
• Solution: Consider only 0,1,or many items– If there are 2+ items, ignore the rest– Unless they transition to different states
16
vs.
Kyungmin Lee
Outline
• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion
17Kyungmin Lee
Finding UI violations
• Checks for 7 UI properties– User actions per task, Text contrast ratio,
Word count, Button size, Button distance,Animation, Scrolling
– Defined by transportation research groups– Report violation, no violation, or defer to experts
18Kyungmin Lee
Finding UI violations
• Properties of individual state
19Kyungmin Lee
Button size
Button closeness
Text contrast ratio
Word count
Animation
Scrolling
Finding UI violations
• Properties of individual state
20Kyungmin Lee
Button size ✔
Button closeness ✔
Text contrast ratio
Word count
Animation
Scrolling
Finding UI violations
• Properties of individual state
21Kyungmin Lee
Button size ✔
Button closeness ✔
Text contrast ratio ✖
Word count
Animation
Scrolling
Finding UI violations
• Properties of individual state
22Kyungmin Lee
Total of 12 words
Button size ✔
Button closeness ✔
Text contrast ratio ✖
Word count ✔
Animation
Scrolling
Finding UI violations
• Properties of individual state
23Kyungmin Lee
Button size ✔
Button closeness ✔
Text contrast ratio ✖
Word count ✔
Animation ✔
Scrolling ✔
Checking user actions per task
24Kyungmin Lee
Few complications
• Buttons in custom view– Empirically discover buttons– If fails, defer to manual verification
• Determining good vs. bad animation
25Kyungmin Lee
vs.
Outline
• Motivation• How AMC explores the app• How AMC finds UI violations• How closely do apps follow UI guidelines• Conclusion
26Kyungmin Lee
Evaluation
• Environmental setup
27Kyungmin Lee
Comparison with expert’s evaluation
28
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
Tested 12 out of 14 apps &
5 out of 7 properties
Tested 12 out of 14 apps &
5 out of 7 properties
Comparison with expert’s evaluation
29
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Comparison with expert’s evaluation
30
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Comparison with expert’s evaluation
31
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Comparison with expert’s evaluation
32
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Agreed 50 / 60 cases≈ 83%
Agreed 50 / 60 cases≈ 83%
Comparison with expert’s evaluation
33
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
One case of false negative<≈ 2%
One case of false negative<≈ 2%
Comparison with expert’s evaluation
34
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Comparison with expert’s evaluation
35
Application Word count Button size Button distance
Animation Scrolling
Google Finance MyDays
eBay Twitter
GasBuddy
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
: Violation : No violation
: Maybe
1st column:AMC’s report
2nd column: Expert’s report
Animation: 4 out of 381 statesButton size & distance: 13 states
Reduced by 95%!
Animation: 4 out of 381 statesButton size & distance: 13 states
Reduced by 95%!
Assessment of Android apps
36
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail
Google Finance
MyDays
eBay
GasBuddy
Google Maps
Google Nav.
TomTom Nav.
Waze
iRadar
Beat the Traffic
Best Parking
MPG
Kyungmin Lee
Assessment of Android apps
37
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
Assessment of Android apps
38
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
Assessment of Android apps
39
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
0.75 violations / state0.75 violations / state
Assessment of Android apps
40
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
Assessment of Android apps
41
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
0.69 violations / state0.69 violations / state
Assessment of Android apps
42
Application User actions per task
Text contrast
ratio
Word count Button size Button distance
Animation # Item to scroll
Gmail 17 2 2 0 0 0 3
Google Finance 0 1 3 1 4 0 3
MyDays 20 4 2 0 2 0 1
eBay 0 1 1 0 0 0 2
Twitter 24 28 7 0 14 0 9
GasBuddy 12 14 1 0 13 0 4
Google Maps 8 7 3 0 7 0 1
Google Nav. 0 0 0 2 1 0 0
TomTom Nav. 26 0 6 1 0 0 6
Waze 72 15 0 1 5 0 4
iRadar 0 2 1 5 4 0 0
Beat the Traffic 5 7 0 0 1 0 0
Best Parking 0 0 0 0 1 0 1
MPG 0 0 0 0 0 0 0
Kyungmin Lee
Only MPG app had no violationsOnly MPG app had no violations
Conclusion
• AMC is a tool that– Explores the application automatically– Pinpoints UI guideline violations
• Many vehicular apps with UI violations– Shows the need for a tool like AMC
• Can be extended to other UI settings
43Kyungmin Lee
44
Questions?
Kyungmin Lee
45Kyungmin Lee
Backup slides
46Kyungmin Lee
State coverage
47Avg. coverage: 80%Kyungmin Lee
States found & runtime
48
Application # of States found
Runtime(h:mm)
Application # of States found
Runtime(h:mm)
Gmail 27 0:56 Google Nav. 15 0:37
Google Finance
18 0:55 TomTom Nav.
36 4:05
MyDays 26 1:55 Waze 81 6:09
eBay 24 0:50 iRadar 23 1:05
Twitter 65 10:25 Beat the Traffic
18 0:19
GasBuddy 59 4:28 Best Parking 11 0:10
Google Maps 31 1:59 MPG 5 1:22
Avg. runtime: 2hr 31minMedian runtime: ~74 minutes
Kyungmin Lee
Non-deterministic transitions
• Caused by– Configuration option– Previous paths– Mistakes in state classification
• Solution– Avoid using non-deterministic transitions!– Use past exploration information
• Combine previous N transitions into one• Attempt to make it deterministic
49Kyungmin Lee
Non-deterministic transitions
• Caused by unaware hidden property
50
BACK
BACK
Kyungmin Lee
Non-deterministic transitions
• Caused by our heuristics
51≈ ≠
Kyungmin Lee
Related work
• Model checking to find software bugs– KLEE, EXE, eXplode, CMC
• GUI Model checking with pre-specified model– Dwyer et al., Paiva et al., Takala et al.
• GUI checking with no model– GUI Ripping, Amalfitano et al., AutoBlackTest
• Using cognitive model to predict distraction– Salvucci’s work
52Kyungmin Lee
Limitations
• Only two ways to explore the app– GUI buttons & Android actions (e.g., back button)– No voice input, multi-touch, gesture, etc
• No checkpoint & restore– Inability to completely revert back
• Relying on app’s dynamic content– Different time -> different violation result
(e.g., Random pop-ups)
53Kyungmin Lee
Handling text inputs
• Need to input password, address, etc– User pre-specifies the text– User pre-specifies the associated text field’s id
• AMC automatically inputs text
54Kyungmin Lee
How we handle an ad
• No different than a button– Look for button size, button distance violations– Look for animation
• No violation found in our evaluation
55Kyungmin Lee
Industry guideline (TRL, 02)
• Qualitative properties– “Color scheme must not be confusing”– “System should provide precise error message”
• Quantitative properties– “Text contrast ratio should be at least 3:1”– “Buttons should be at least 15mm apart”– Can be verified without human efforts
56Kyungmin Lee