Remote (Mobile) Usability Testing Lab

Download Remote (Mobile) Usability Testing Lab

Post on 02-Dec-2014

2.913 views

Category:

Design

5 download

Embed Size (px)

DESCRIPTION

Describes the system we developed for conducting remote usability tests. It also gives you insights on how we changed the system to perform mobile tests... .

TRANSCRIPT

<ul><li><p>Remote (Mobile) Usability Testing Lab</p><p>Human.Interaction.Design.2010Jan MoonsJan.Moons@alcatel-lucent.comNovember 23rd, 2010</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>AboutIn this presentation we describe our experiences in developing and testing a system that makes it possible to usability test (mobile) applications. Disclaimer:Work in progress</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p> Usability Testing? In a nutshell</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Usability Testing?A technique to observe people, who are representatives of the target audience, using a product to discover errors and areas of improvement.</p><p>Typically measuring how well people respond in following areas:EfficiencyEffectivenessSatisfaction</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>5 CharacteristicsEvery usability test has the following characteristics:Primary goal: Improving usability of a productParticipants represent real usersParticipants perform real tasksObserve and record (log) what participants do and sayAnalyze the data, diagnose the real problem, and recommend changes to fix those problems</p><p>A Practical Guide to Usability Testing, J.S. Dumas and J.C.Redish</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p> Usability Testing @ Bell Labs Problems and Vision</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Typical usability test @ Bell LabsParticipantObserverLogging softwareParticipants are inevitably invited to the lab!</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Problems with State-Of-The-ArtSoftware installation needed on participants computerParticipants dont appreciate installing software on their PCsLive recruitment is not possibleSoftware doesnt run on all major operating systemsTarget: MS Windows OS (XP/Vista)Problematic: Mac OS, Linux, Solaris, etcObserving and recording over the Internet/Intranet hard or even not possibleUsers cannot participate from their own premises with their own devicesRemote usability testing is an issue</p><p>Build own Remote Usability Testing system to solve these problems</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>VisionDesign a system to remotely observe and record users actions, voice and facial expressions on an Internet/Intranet connected PC.</p><p>With the ability for the observer to analyze the data and diagnose the problems</p><p>Based on Web technologies that supports all major Operating Systems</p><p>But without having to install any special software on the participants or even the observers computer</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>ArchitectureRTMPAudio/VideoStreamingScreencastStreamingRTMP A/VJDBCObservers browserTesters browserRemote Objects</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Observe / ReplayParticipants webcamParticipants screenLogging panelCurrent taskMouse pointer</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>AnalyzeEffectivenessEfficiencyData Analysis</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p> Mobile Usability Testing Popular techniques</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Look-over-shoulderhttp://www.wirelessrerc.org/for-consumersScreen recording?</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Document Camerahttp://www.easeofweb.com/mobile_usability_test.htmlFixed positionUser has to be seated</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Sled / Lamphttp://www.easeofweb.com/mobile_usability_testing.htmlMobile but very intrusive</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Pros and ConsDocument/Sled Camera:</p><p>Pros:Works with all mobile devicesNo need to install software on mobileInteraction with mobile is visibleIdeal for lab testing</p><p>Cons:Very intrusiveExperience is not naturalNo automatic key logging</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Screen-capturingJailbroken iPhone Rooted Android</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Pros and ConsScreen-Capturing:</p><p>Pros:Less/non-intrusiveMore natural experienceTesting in the field (e.g. Location-Based Services)</p><p>Cons:Software installation needed for recordingFor each OS (different) implementation needed!Doesnt show the users hands actually interacting with the mobile</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p> Mobile Usability Testing Natural Experience</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>VisionExtend our custom-built usability testing lab for the computer to mobile devices</p><p>With the ability to observe and record:The mobile screen / users actionsThe users voice (environment noise)The users face (optional)</p><p>Ultimate Goal: Making the experience as natural as possible for user testers</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>ImplementationPrioritized:</p><p>Camera capturing mobile screenAll mobile devicesAbility to capture interactions (hands)</p><p>Screen-sharing + Audio/Video streamingUltimate goal of a natural experience; non-intrusive1st focus Android mobile device</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Approach 1: Camera capturing mobile screenDIY DocumentCameraPC for transmission of streams to serverWebcam FaceCapturingAlso ideal for testing Paper Prototypes!</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Architectural ChangesRTMPAudio/VideoStreamingScreenStreamingRTMP A/VJDBCObservers browserTesters browserRemote Objects</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Approach 2: A/V Screen-CapturingScreen capturingIssues!!!</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Observer</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Architectural ChangesRTMPVideoStreamingScreencastStreamingRTMP A/VJDBCObservers browserRemote ObjectsAudio SIPADBScreen captureConnected ComputerAlso support for VNC</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>Future workMeasuring satisfactionSurveysEmotion recognition (voice, facereading or combination)Automatically generating (video) highlights of observationsCapturing the mobiles sensors:GPS-coordinates (for Location-Based Services)Accelerometers, touch events (application / device interaction)SIP/RTMP audio/video streaming</p><p>Last but not least, eat our own dog food and usability test the system ;-)</p><p>Copyright XXXX Alcatel-Lucent. All rights reserved. </p><p>www.alcatel-lucent.comwww.alcatel-lucent.com</p><p>Usability Criteria = requirements:</p><p>Efficiency:How much time and how many steps are required for people to complete basic tasks?Effectiveness:How many mistakes did people make?Satisfaction:How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend?</p><p>Learnability, memorability, errors, flexibility, user in control, etc . Improving usabilityUsability definition (ISO 9241): a product is usable if it enables users to efficiently and effectively achieve their goals in a satisfactory mannerKnow why you are testing!Know what you are testing! Real UsersUsability testing is all about seeing the design through the eyes of the test participants. As they work their way through the design, you get to see and hear what works well and where it becomes frustrating to accomplish their goals.If you've recruited the wrong participant, you'll not learn what you want. Real TasksDesigning the right tasks =&gt; dont guide users in a certain direction by using certain terminologyObserve and recordObserve: To see and hear what works and what doesntRecord: Memory and communication tool to stakeholdersSoftware installation needed:You need to have a PC ready for performing tests =&gt; you miss a lot of contextYou cannot recruit users while they are actually using the application, web site, etc. .</p><p>Not all major OSs:I am a Mac user and need to install fusion or parallels in order to use the observation softwareHard to test applications on other OSs</p><p>Remote testing:Network issuesHave to create a local network for each usability testSpread of technologies:Java: x%Flash: x%</p><p>RTMP Streaming server:Broadcasts screen and audio/videoRecords streams</p><p>RTMP = Real-Time Messaging and Presence ProtocolAdobeOpen sourceWell we got some bad news today. It was a matter of time but it looks like Apple doesnt like applications like ScreenSplitr. They now prohibits taking screenshots while in the background and crashes ScreenSplitr when it tries to do so.</p><p>This isnt a perfect solution: we couldnt capture click events and the output from the iPhone was a bit choppy. Also, the video we captured didnt show the users hands actually interacting with the phone.</p><p>ScreenSplitr on the iPhone in combination with iDemo on the MacPros: ScreenSplittr transmits the iPhone signal via USB so there is less delay, resulting in smoother animations.Cons: Having the iPhone connected to the Mac with a USB cable is less natural for user testers. It also doesnt show where the user touches the screen.</p><p>Display RecorderPros: Records where user touches the screen. Transfers live video right to a browser on a Mac, so no additional software is needed on the Mac side. It also records everything right on the iPhone leading to smooth animations in the final output.Cons: Requires iPhone 3GS. We werent able to test this out ourselves. We would need to take the recorded screencast from the iPhone and match it to the recording of the user.</p><p>Works with all mobile devicesNo need to install software on mobileInteraction with mobile is visibleBecause we are using RTMP streams it was very easy to exchange the screen-capturing Java-Applet by an extra Flash cameraProblem:Android (but also iPhone) devices prevent you from capturing screens from a separate background serviceWe didnt want to root or jailbrake devices, because you cannot do that with participants mobiles!</p><p>Solution:Connect mobile to PC, running ADB (debug) serverJava application, running on the PC, acts as a client to the debug server requesting frame buffers from the mobile deviceSip = session initiation protocolaudio / video streaming over the InternetVoIP</p><p>RTMP = real-time messaging and presence protocolAudio / video streaming over the Internet</p></li></ul>