exploring mobile usability testing methods for enterprise users: a case study
DESCRIPTION
Exploring Mobile Usability Testing Methods for Enterprise Users: A Case Study. Minmin Yang & Judy Keeley Boston Mini UPA Conference May 26, 2009. Outline. Background Testing approach In-person Remote Testing equipment In-person Remote Findings about our mobile testing method. 2. 2. - PowerPoint PPT PresentationTRANSCRIPT
Exploring Mobile Usability Testing Methods for Enterprise Users:
A Case Study
Minmin Yang & Judy KeeleyBoston Mini UPA Conference
May 26, 2009
May 2009 22
Outline
Background
Testing approach In-person Remote
Testing equipment In-person Remote
Findings about our mobile testing method
May 2009 33
Background
Usability testing on Verizon Enterprise Center (VEC) mobile portal before release to customers Portal for enterprise customers to manage
services from a mobile phone
Research & preparation Reviewed mobile testing literature for
selection of test approach Researched test equipment Acquired test equipment Tested selected method (approach &
equipment)
May 2009 4
Testing Approach
May 2009 5
What the research says…
Lab testing vs. field testing on mobile devices Kjeldskov et al. (2004) & Kaikkonen et al. (2005)
Field testing added little to usability evaluations Lab testing seems to be sufficient when searching user interface flaws to
improve user interaction
Nielsen et al. (2005) Field testing revealed significantly more usability problems than lab testing
overall Field testing revealed problems with interaction style and cognitive load that
were not identified in the laboratory
Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. (2004). Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Proceedings of the 6th International Mobile HCI 2004 conference. LNCS, Springer-Verlag.
Kaikkonen, A., Kekalainen, A., Cankar M., Kallio, T., & Kankainen A. (2005). Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies, 1(1), 4-16.
Nielsen, C. M., Overgaard M., Pederson M. B., Stage J., & Stenield S. (2006). It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. In Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. Oslo, Norway.
May 2009
What the research says…
Field testing vs. lab testing on mobile phone vs. lab testing with emulator Betiol & Cybis (2005)
All approaches revealed more similarities than differences in terms of results Lab testing with emulator could reveal a large percentage of usability problems
03 2
32
04
2
Lab Testing with Emulator
Lab Testing on Mobile Phone
Field Testing with Wireless Camera
Betiol, A. H., & Cybis, W. A. (2005), Usability Testing of Mobile Devices: A Comparison of Three Approaches. International Federation for Information Processing, 470 – 478.
May 2009 7
Testing Approach
In-Person Testing
Remote Testing
May 2009 8
In-Person: Field Testing?
Pros: Provides highest validity Reveals more usability problems compared
to lab testing (Nielsen et al., 2006)
Cons: Equipment is expensive Hard to find participants for field testing Difficult to find situations where following
participants is appropriate
Field Testing
May 2009 9
In-Person: Emulators?
Pros: Emulators are free Testing with emulators can reveal a large
percentage of the usability problems (Betiol & Cybis, 2005)
Cons: Emulators don’t duplicate experience of using
actual phone
Lab Testing with Emulator
May 2009
In-Person: Lab Testing on Mobile Phone?
Lab testing on mobile phone is sufficient in identifying user interface issues (Kaikkonen et al. 2005)
Compared to field testing Cheaper to set up equipment Easier to conduct testing
Compared to using emulators: Can provide a more authentic experience
Decision: use lab testing on mobile phone for in-person testing
Lab Testing on Mobile Phone
May 2009 11
Remote: Which Approach?
Remote field testing: not practical
Remote testing with participants’ own mobile phone: Expensive: webcam or sled need to be purchased
and shipped to participants Troublesome: participants need to set up
equipment and install software themselves;
Remote testing with emulators: No cost: emulators are free Much less troublesome: no installation on
participants’ end via web conferencing
Decision: use emulators for remote testing
Remote Testing with Emulator
May 2009 12
In-Person Test Equipment
May 2009 13
Fixed lab
Portable lab
In-Person Test Equipment
Fixed lab Could use existing lab in nearby office, or set up
a lab in local office Participants must come to lab for testing
Portable lab More convenient for participants - Lab can travel
to them
Decision: Use portable lab
May 2009 14
In-Person Test Equipment: Recording
How to record:
Phone screen Screen capture software or video camera?
User interaction with phone Typing on keyboard Holding phone
Audio
May 2009 15
In-Person Test Equipment: Recording
Screen capture software (short list)
Name Platform Issues Cost
BB ScreenStream
Blackberry Typing on Blackberry skips characters $0
Mobiola Blackberry, Windows Mobile, Nokia
a. Won't work on password-protected Blackberries (being fixed in future release)
b. Typing on Blackberry skips characters
$19
Soti Windows Mobile Only Windows Mobile phones $55 per device
Impatica ShowMate
Blackberry, Windows Mobile
Requires installation on phone $250
May 2009 16
In-Person Test Equipment: Recording
Blackberry Screen Capture SoftwareBBScreenStream
http://www.judykeeley.com/portfolio/presentations/UPA-2009/screenstream.avi
May 2009 17
In-Person Test Equipment: Recording
Screen capture software
Advantages Records screen clearly Cheap software available
Disadvantages Skips typed characters on Blackberry Doesn’t work with password-protected Blackberry phones Doesn’t capture user’s interaction with phone (requires camera)
May 2009 18
In-Person Test Equipment: Recording
Video Camera
Advantages Records users interaction with phone Records audio along with video
Disadvantages Need to purchase Harder to clearly capture phone screen
Decision Use video camera instead of screen capture software
May 2009 19
In-Person Test Equipment: Recording
How to use the video camera for recording? Video camera fixed to desk Video camera mounted to phone Video camera mounted to participant’s head
Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, Schusteritsch R., Wei C., & LaRosa M. (2007). CHI '07 extended abstracts on Human factors in computing systems, pp. 1839-1844.
Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03
May 2009 20
In-Person Test Equipment: Recording
Device-mounted Camera mounted on mobile device or sled
Pro: Camera moves with device, so device stays in video frame
Cons: Makes phone heavier; may feel cumbersome to participant Buying sled is expensive Takes time to make custom sled
Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03
May 2009 21
In-Person Test Equipment: Recording
Subject-mounted Camera mounted on participant’s head or eyeglasses Records participant’s field of view
Pro: Participants can walk around test area while using device
Cons: Camera on head might feel awkward or annoying Device lost from video frame when participant turns head
Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03
May 2009 22
In-Person Test Equipment: Recording
Environment-mounted Camera attached to fixed spot (e.g. desk)
Pro: Simplest and easiest to set up
Con: Must keep phone in fixed location
Decision: Use environment-mounted camera
Usability Test Equipment for Mobile Devices Kim S., Kim M., Kim S. W, & Kang H. (2003) CHII ‘03
May 2009 23
Logitech webcam Model: QuickCam Pro for Notebooks HD Video: 960 x 720 pixels 30 frames per second Built-in microphone to record sound
Webcam attached to lamp neck and base
Laptop with Logitech webcam software
Cost: $90 for webcam
In-Person Equipment: Our Portable Lab
May 2009 24
In-Person Equipment: Our Portable Lab
May 2009
In-Person Testing: Webcam Demo
http://www.judykeeley.com/portfolio/presentations/UPA-2009/webcam.wmv
May 2009 26
Remote Test Equipment
May 2009
Remote Test Equipment: Emulators
All remote participants had Blackberry phones Two had BB Curve, two had 8830, and one had Storm
Some emulator choices:
Blackberry Storm (touchscreen)
Blackberry 8830Blackberry Curve
May 2009
Remote Test Equipment: Emulators
http://www.judykeeley.com/portfolio/presentations/UPA-2009/emulators.avi
May 2009
Remote Test Equipment: Emulators
http://www.judykeeley.com/portfolio/presentations/UPA-2009/emulators.avi
This is a test; click the phone below
May 2009
Remote Test Equipment: Emulators
Goal: select emulators similar to participants’ phones but also easy for participant to control
Performed dry run testing on 3 emulators
Blackberry Curve Cursor moves pixel by pixel - painfully slow and awkward
Blackberry 8830 On older models, cursor jumps from link/field to link/field - Faster
and easier to control Selected for participants already using Curve & 8830
Blackberry Storm Touchscreen - Easy to control Selected for participant already using Storm
Blackberry 8830
Blackberry Storm
May 2009 31
Remote Test Equipment: Record/Connect
Recording/Connecting software (short list)
Name OS Issues Cost
UserVue Windows Not possible for users to take control of moderators’ screen; not possible to use in Mac
$149 per month; $1495 per year
Adobe Acrobat Connect Pro
Windows, Mac
Need to go through time-consuming purchase process
$0.32/min/user; 5 user plan: $375/month; 10 user plan: $750/month;
GoToMeeting Windows, Mac
Need to go through time-consuming purchase process
$49 per month; $468 per year
Microsoft Live Meeting
Windows, Mac
Delay in showing the keyboard input on the screen - annoying
$0 (already purchased by Verizon)
Webex Windows, Mac
None! $0 (already purchased by Verizon)
May 2009 32
Remote Test Final Equipment
Blackberry Emulators 8830 and Storm
Webex net conference to share emulator with participant
Phone conference (webex dials in)
Webex records both audio and video
Cost: $0
May 2009 33
Findings about Our Usability Testing Method
May 2009 34
Findings - 1
In-Person testing
Environment-mounted recording approach is good overall Video quality excellent - phone screen can be seen clearly Equipment simple and easy to carry around
Issue Moderator needed to remind the participant whenever the phone was far
away from the camera, which was not very frequent, but when it happened, it did interfere the performance a little bit
May 2009 35
Findings - 2
Remote Testing
Webex + emulator approach for remote testing worked out very well The video quality was good and the interface can be seen clearly The video size was pretty small vs. webcam
Issues Video conversion quality – Webex arf format was converted to wmv;
quality degraded with conversion and further saves Participants needed quick training session before using emulator
May 2009 36
Findings - 3
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8 9
Participants
Num
ber
of
Usabilit
y P
roble
ms
In-Person: 48 Problems Remote: 31 Problems
Mean = 12
In-person testing (M=16) could reveal more usability problems than remote testing (M=9), t(7) = 5.52, p = .0009
May 2009 37
Findings - 3
58%
28%14%
In-personRemote
Based on our testing (sample size: 9)
Remote testing revealed 42% of usability problems
Remote testing revealed problems not found in in-person testing
