diy mobile usability testing - ia summit 2011
DESCRIPTION
Our presentation for the IA Summit 2011 in Denver.TRANSCRIPT
we are ...
Bernard, packet core engineer at NSN
Belén, interaction designer at Intel
usability testinga process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.
Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell
please, stand up
everyone
sit down if you don’t have a US cellphone
with a data plan
sit down if you don’t like beer
sit down if you are absolutely terrified by the idea of being our
test subject
why recording?
memory aid
powerful communication tool
reactions
actions
dut = mut
where:
dut = desktop usability testingmut = mobile usability testing
dut = mut + afec
afec = a few extra challenges
which phone?
which context?
which connection?
which phone?
which context?
which connection?
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html
web task success rates
feature phones 38%
smartphones 55%
touch phones 75%
Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html
handset usability affects test results
remember ...
test with participants’ own phones
if not possible, include training and warm-up tasks
which phone?
which context?
which connection?
field vs. lab
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
experts disagreeaccording to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supportedUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
experts disagreeevaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
but they all agreeevaluations in the field (are) more complex and time-consuming
testing in the field requires double the time in comparison to the laboratory
It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005
testing in the lab is better than no testing
remember ...for most software, lab testing is fine
if you must do field testing
do it late
plan and run pilot tests
be prepared (like the Scouts)
which phone?
which context?
which connection?
!"!"#$%&'("%)*%'!+&,-.$/&01'2..'3/0"$,'3#,#)4#5 #$%&"'())67"$$89::;;;<="#$%&,"%)*%<=+*
!"#$%& '&()"*+,-*.//$0,1*")(2
US Mobile Data Market Update Q4 2010 and 2010, Chetan Sharma Technology & Strategy Consulting http://www.chetansharma.com/usmarketupdate2010.htm
remember ...
do not test over wi-fi
cover participants’ data costs
where:
dut = desktop usability testingmut = mobile usability testingafec = a few extra challenges
dut = mut + afec
tsdohoeaygtrtwt = the small detail of how on earth are you going to record the whole thing
tsdohoeaygtrtwt( )
why recording?
memory aid
powerful communication tool
4 approaches to the small detail of how on earth
are you going to record the whole thing
.
1. wearable equipment
Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007) International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)
allows testing in the field
but ...
difficult and time-consuming to set up
intrusive, uncomfortable and heavy
UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo 5http://www.digia.com/C2256FEF0043E9C1/0/405001348
2. screen-capture applications
provides high quality screen recording
but no application will support all platforms
http://www.digia.com/C2256FEF0043E9C1/0/405001348
it’s not cheap
https://shop.digia.com
and participants won’t appreciate you installing stuff on their phones
Handheld Usability, S. Weiss (page 174)
3. document cameras
Towards the Perfect Infrastructure for Usability Testing on Mobile DevicesR. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)
both video streams fed onto a laptop and recorded with Morae
webcam records participant’s face
document camera with autofocus controlled remotely from a laptop records screen and fingers
Nielsen Norman
Elmo TT-02RX Teachers Tool
popular and well documented
it’s not cheap
http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html
participants must keep within the camera range
phone must lay on a desk or be hold at a flat angle
http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/
4. mounted devices
Mobile Device Camera by Noldus http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#
Mobile Device Camera by Tracksys http://www.tracksys.co.uk/product-details.php?id=9
4a. ready-made mounted devices
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
4b. DIY mounted devices
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
Nick Bowmast http://www.bowmast.com/mob-device-cam/
4b. DIY mounted devices
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing
4b. DIY mounted devices
Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
Nick Bowmast http://www.bowmast.com/mob-device-cam/
Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing
Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)
4b. DIY mounted devices
natural interaction with the phone
but it’s not cheap
http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/
messy to build
http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/
and ...
if bulky they can prevent single-hand use
if heavy they can become tiring during long tests
easy to put togethercheaprepeatableallows holding the deviceallows one-handed usesupports all form factorsruns tests with participants’ phonescaptures screen, face and fingersgives enough video quality
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & fingers
gives enough video quality
http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/
the spirit
the ingredients
two meccano trunions (part no. A126)
two 5 and 6-hole meccano strips (part nos. 5 & 4)
one 11-hole meccano strip (part no. 2)
six meccano screws & nuts (part nos. 69 & 37h)
one 13-20mm jubilee clip
one HUE HD webcam
a second USB webcam
a USB male to female extension cable
blu tack
an allen key
a meccano wrench
a screwdriver
a Windows computer
screen recording software
You are planning a visit to the Denver Zoo after this (great) session.
Go to www.denverzoo.org and find out which bus number will bring you to the zoo from the Colorado Convention Centre.
a task
how was that?
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
15-model meccano set 18.33
Hue HD webcam 28.94
Philips webcam 52.08
additional meccano parts 4.90
blu tack 0.98
jubilee clips (x2) 1.55
USB cable 8.99
screwdriver 5.99
CamStudio 0.00
total (in GBP) 121.76
in USD: 196.40
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
weight: 125 gramsan iPhone weighs 137 grams
an iPad weighs 680 grams
I weigh 55,000 grams
a blue whale weighs 136,400,000 grams
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
Nokia E66slider with 12-key numeric keypad
Nokia E71
candybar with qwerty keyboard
BB Bold
iPhone 3Gtouchscreencandybar
Nokia N900
slider touchscreen with qwerty keyboard
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
b&b’s
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face and fingers
gives enough video quality
we expect much of our buildings: they need to have firm foundations, solid structures, pleasing aesthetics. We should expect the same of emerging mobile systems.
Mobile Interaction Design, M. Jones and G. Marsden (2005)
[email protected]@runningwithbulls.com
@belenpena @bernardtyers
http://belenpena.posterous.comhttp://www.runningwithbulls.com