diy mobile usability testing - sxsw interactive 2012

Thanks for coming!

Bernard, packet core engineer at NSN

Belén, interaction designer at Intel’s OTC

#SXdiymut

usability testinga process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.

Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

please, stand up

take out your cellphone

sit down if you don’t have a US cellphone

with a data plan

sit down if you don’t like beer

sit down if you are absolutely terrified by the idea of being our

test subject

why recording?

memory aid

powerful communication tool

reactions

actions

dut = mut

where:

dut = desktop usability testingmut = mobile usability testing

dut = mut

where:

dut = desktop usability testingmut = mobile usability testing

dut = mut + afec

afec = a few extra challenges

which phone?

which context?

which connection?

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

http://www.useit.com/alertbox/mobile-usability-study-1.html



feature phones 38%

smartphones 55%

touch phones 75%


handset usability affects test results

remember ...

test with participants’ own phones

if not possible, include training and warm-up tasks

which phone?

which context?

which connection?

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

field vs. lab


field vs. lab

0 0

The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

0 1Field Lab

according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported

Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported

Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

0 2Field Lab

evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations


evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations


1 2Field Lab

The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.

Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.

Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

2 2Field Lab


field vs. lab


field vs. lab

EXPERTS DISAGREE

... but they all agree

evaluations in the field (are) more complex and time-consumingIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006


testing in the field requires double the time in comparison to the laboratoryUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005


field-based usability studies are not easy to conduct. They are time consuming and the added value is questionable.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

testing in the lab is better than no testing

remember ...for most software, lab testing is fine

if you must do field testing

do it late

plan and run pilot tests

be prepared (like the Scouts)

which phone?

which context?

which connection?

remember ...

do not test over wi-fi

cover participants’ data costs

where:

dut = desktop usability testingmut = mobile usability testingafec = a few extra challenges

dut = mut + afec

where:


dut = mut + afec tsdohoeaygtrtwt( )

where:


dut = mut + afec

tsdohoeaygtrtwt = the small detail of how on earth are you going to record the whole thing

tsdohoeaygtrtwt( )

why recording?

memory aid

powerful communication tool

4 approaches to the small detail of how on earth

are you going to record the whole thing

.

1. wearable equipment

Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007) International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

Will Laboratory Test Results be Valid in Mobile Contexts?, A. Kaikkonen, A. Kekäläinen, M. Cankar, T. Kallio, A. Kankainen and A Field Laboratory for Evaluating in Situ, R. T. Høegh, J. Kjeldskov, M. B. Skov, J. Stage, 2011

906

��

Field tests are vulnerable to unexpected events, such as rain or bus schedules. These risks should be listed before the test is run with actual test us-ers. Since the environment cannot be controlled in the same way as the laboratory, the researchers should also have a backup plan or recruit an extra user, just in case. Running a pre-test or a pilot is ��to reduce the risks due to the technology used, but ��analysis of the results. If the user moves around during the test, for example, is there a location where the lighting makes it impossible to see the text on a screen, or the surrounding noise blocks �� focuses on software rather than hardware issues, these kinds of environmental disturbances may

make it impossible to get any meaningful results from the test.

There are several test planning issues that �� than a laboratory test—particularly if multiple moderators run the test or the tests are outsourced. Examples of these issues are moderator prompt-ing, timing between questions, how to react to external interruptions, and to what extent test user �� less predictable, specifying these details takes additional effort.

It is important to be open about the nature of the test when recruiting users. Some users may not be willing to participate when they hear the test will take place in a public location—it hap-pened with a few users. Facing this issue while

��

Table 1. Differences between locations

Laboratory Field

Total test time per user, average 35 min. 45 min.

Instructions and preparations per user, estimated time 10 min. 20 min.

All user interface problems found Yes Yes

Users easily understood the application concept Yes No

User behaviour can be observed in a natural environment No Yes

Environment can be fully controlled Yes No

Suitable for usability testing Yes Yes

Suitable for testing a concept or service idea With restrictions Yes

�(!04%2�� ),,��!"/2!4/29��%34��%35,43�"%��!,)$�).��/"),%��/.4%843��!.$"//+�/&��%3%!2#(�/.��3%2��.4%2&!#%��%3)'.�!.$��6!,5!4)/.�&/2��/"),%��%#(./,/'9�� 2%0!2%$�&/2�"%,%.�"!22/3�0%.!�).4%,�#/-��%,%.��!22/3��%.!�/092)'(4��()3�$/7.,/!$�&),%�)3�-!$%�!6!),!",%�&/2�0%23/.!,�53%�/.,9�!.$�)3�35"*%#4�4/�4(%��%2-3�/&��%26)#%��.9�/4(%2�53%�2%15)2%3�02)/2�72)44%.�#/.3%.4�&2/-�4(%�#/092)'(4�/7.%2��.!54(/2):%$�53%��2%02/$5#4)/.�!.$�/2�$)342)"54)/.�!2%�342)#4,9�02/()")4%$�!.$�6)/,!4%�!00,)#!",%�,!73��,,�2)'(43�2%3%26%$�

993

��

�� capture as described above, but now weights only 2 kg and measures only 18x14x25 cm, making it �� for longer periods of time. Powered by only one �� approximately 2.5 hours before the battery must be swapped with a spare one.

FUTURE TRENDS

�� for evaluating mobile technology use and usability in situ focus primarily on improving the quality,

reliability, and size of the cameras attached to the mobile device. As wireless video technology matures and becomes more widespread, an emer-gence of cheap high-end wireless video cameras matching the professional standard of the wireless ��laboratory are likely to be seen. Broadcast qual-ity interference-free wireless video technologies exist today, but are still rather expensive and not ��

Coming from another area of application, new camera technologies are also emerging within ��-low video signals to be transferred digitally via wireless network connections rather than over an

Figure 10. Video recording with third-person view of participants and close-up view of PDA. Note that the camera focused on the device screen is turned 90 degrees to optimize use of the Picture-in-Picture view.

��!�� cm—containing video and audio receivers, Picture-in-Picture unit, hard disk recorder, and battery.

�' /3$1��($+#�� !.1 3.17�%.1��5 +4 3(-&�(-��(34�� -#!..*�.%��$2$ 1"'�.-��2$1��-3$1% "$��$2(&-� -#��5 +4 3(.-�%.1��.!(+$��$"'-.+.&7�� 1$/ 1$#�%.1�!$+$-�! 11.2�/$- �(-3$+�".,��$+$-�� 11.2��$- �./71(&'3��'(2�#.6-+. #�%(+$�(2�, #$� 5 (+ !+$�%.1�/$12.- +�42$�.-+7� -#�(2�24!)$"3�3.�3'$��$1,2�.%��$15("$��-7�.3'$1�42$�1$04(1$2�/1(.1�61(33$-�".-2$-3�%1.,�3'$�"./71(&'3�.6-$1��- 43'.1(8$#�42$��1$/1.#4"3(.-� -#�.1�#(231(!43(.-� 1$�231("3+7�/1.'(!(3$#� -#�5(.+ 3$� //+(" !+$�+ 62��++�1(&'32�1$2$15$#�

1. wearable equipment

allows testing in the field

but ...

difficult and time-consuming to set up

intrusive, uncomfortable and heavy

Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#

2. screen capture

http://www.shapeservices.com/en/products/details.php?product=capture&platform=none%23

http://www.shapeservices.com/en/products/details.php?product=capture&platform=none%23

2. screen capture

Remote

Mac, Linux and mobile environments such as Android, Symbian, iPhone OS and Windows Mobile.

It runs on both PC,

2. screen capture

Remote

Remote

provides high quality screen recording

but participants won’t appreciate you installing stuff on their phones

no application will support all platforms

http://www.shapeservices.com/en/products/details.php?product=capture&platform=none



no application will support all platforms

http://www.ovostudios.com/



Ovo Studios screen capture application for iOShttp://www.ovostudios.com/



Ovo Studios screen capture application for iOShttp://www.ovostudios.com/

fingers are not captured ...



.. and that is a big deal

Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011http://www.remoteresear.ch/airplay/

http://www.remoteresear.ch/airplay/



Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011http://www.remoteresear.ch/airplay/

Obviously, think aloud is critical because I cannot see how the participant is interactingwith his fingers on the touch screen.

I can see using the mirroring in a lab setting to get the signal from the iPadto the observation room though you still won’t see the physical interactionwith the device, like you would with a device mounted camera (e.g. Noldus).




iPad usability testing - our equipmenthttp://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/



iPad usability testing - our equipmenthttp://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

Recently, we’ve questioned the value of capturing the device screen. It doesafter all end up being a video of the screen changing but with no senseof the participant interacting with it.



3. document cameras

Handheld Usability (page 174), S. Weiss (2002)

Towards the Perfect Infrastructure for Usability Testing on Mobile DevicesR. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

Google

Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)

both video streams fed onto a laptop and recorded with Morae

webcam records participant’s face

document camera with autofocus controlled remotely from a laptop records screen and fingers

Nielsen Norman

Elmo TT-02RX Teachers Tool

http://www.usertesting.com/mobile/ Remote

usertesting.com

http://www.usertesting.com/mobile/

http://www.usertesting.com/mobile/

good recording quality and easy to set up

http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html

but it’s not particularly cheap

Elmo TT-12 Document Camera (accessed March 4th 2012)http://www.bhphotovideo.com/c/product/843500-REG/Elmo_1331_TT_12_Interactive_Document_Camera.html

http://www.bhphotovideo.com/c/product/843500-REG/Elmo_1331_TT_12_Interactive_Document_Camera.html

http://www.bhphotovideo.com/c/product/843500-REG/Elmo_1331_TT_12_Interactive_Document_Camera.html

http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.htmlIPEVO Point 2 View USB Document Cam (accessed March 4th 2012)http://www.ipevo.com/prods/Point-2-View-USB-Camera Chea

p

http://www.ipevo.com/prods/Point-2-View-USB-Camera

http://www.ipevo.com/prods/Point-2-View-USB-Camera

participants must keep within the camera range

phone must lay on a desk or be hold at a flat angle

http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/



4. mounted devices

4. mounted devices

ready-made

4. mounted devices

DIY

ready-made

Mobile Device Camera by Noldus http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#

Mobile Device Camera by Tracksys http://www.tracksys.co.uk/product-details.php?id=9

4a. ready-made mounted devices

Ovo Studios device camerahttp://www.ovostudios.com/devicecamera.asp

http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#








http://www.ovostudios.com/devicecamera.asp

http://www.ovostudios.com/devicecamera.asp

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

4b. DIY mounted devices

http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/





Nick Bowmast http://www.bowmast.com/mob-device-cam/

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

4b. DIY mounted devices













http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/






Nick Bowmast http://www.bowmast.com/mob-device-cam/

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

4b. DIY mounted devices are blooming!!

by curiouslee

natural interaction with the phone

but they are not cheap

http://www.godigi.com/products/DigiZoom-MDC.html



Mr Tappy, a kit for filming handheld devices (accessed March 10th 2012)http://www.mrtappy.com Chea

p

http://www.mrtappy.com/index.html

http://www.mrtappy.com/index.html

messy to build


https://shop.digia.com

https://shop.digia.com

and ...

if bulky they can prevent single-hand use

if heavy they can become tiring during long tests

easy to put togethercheaprepeatableallows holding the deviceallows one-handed usesupports all form factorsruns tests with participants’ phonescaptures screen, face and fingersgives enough video quality

wearable equipment

screen capture

applications

document cameras

mounted devices

ready-made

mounted devices

DIY

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face & fingers

gives enough video quality

http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/

the spirit



the ingredients

two meccano trunions (part no. A126)

two 5 and 6-hole meccano strips (part nos. 5 & 4)

one 11-hole meccano strip (part no. 2)

six meccano screws & nuts (part nos. 69 & 37h)

one 13-20mm jubilee clip

one HUE HD webcam

a second USB webcam

a USB male to female extension cable

blu tack (I think you call it mounting putty)

an allen key

a meccano wrench

a screwdriver

a Windows computer

screen recording software

You just moved to Austin, to an old, big house in Pemberton Heights. You love it, but there is a problem: mice and rats. The house is infested!

Go to www.austintexas.gov and find out how to let the local authorities know about the infestation.

a task

http://www.austintexas.org

http://www.austintexas.org

how was that?

b&b’s


cheap

repeatable





captures screen, face and fingers


15-model Meccano set 14.99

Hue HD webcam 34.95

Philips webcam 24.32

additional Meccano parts 4.01

blu tack 0.98

jubilee clips (x2) 1.99

USB cable 8.99

screwdriver 6.29

CamStudio 0.00

total (in GBP) 96.52

in USD: 151.25

b&b’s


cheap

repeatable







weight: 125 grams

weight: 125 gramsan iPhone weighs 137 grams

an iPad weighs 680 grams

I weigh 55,000 grams

a blue whale weighs 136,400,000 grams

b&b’s


cheap

repeatable







we expect much of our buildings: they need to have firm foundations, solid structures, pleasing aesthetics. We should expect the same of emerging mobile systems.

Mobile Interaction Design, M. Jones and G. Marsden (2005)

[email protected]@runningwithbulls.com

@belenpena @bernardtyers

http://belenpena.posterous.comhttp://www.runningwithbulls.com

mailto:[email protected]














diy mobile usability testing - sxsw interactive 2012

Technology