usability test plan€¦ · process. metrics there are many different ways to categorize and rank...

USABILITY TEST PLAN

SCOPE

SESSIONS EQUIPMENT

SCHEDULE

Conduct an Usability Test for the initial prototype of the vocabulary and language learning app called ‘Talk Talk’, created for users who want to learn a new language in an efficient and fun way, in quick bite size lessons, with immediate results and applicability.

USABILITY TEST PLAN

The 4 Usability tests will be conducted between the 25th and the 29th of July.

Three out of four were done in person and one online via Skype.

Three of the sessions were done using my mobile phone and the app Prott, with pen and paper for note taking.

One of the sessions was done online, via Skype, using the Prott app as well, and pen and paper for note taking.

Each session should take approximately 10 to 15 minutes each, depending on on how quickly participants complete all requested tasks and how many questions they may have during the process.

METRICS

There are many different ways to categorize and rank the severity of errors in an Usability test. For this test we’ll refer to an adapted version of Jakob Niel-sen’s rating scale:

0 = I don't agree that this is a usability problem at all1 = Cosmetic problem only: need not be fixed unless extra time is available on project2 = Minor usability problem: fixing this should be given low priority3 = Major usability problem: important to fix and should be given high priority4 = Usability catastrophe: imperative to fix before product can be released

USABILITY TEST TASKS

DIRECT TASKS SCENARIO TASKS

1- Please create a new account and go to the homepage / dashboard of the app.

2- Access the lesson with topic #1. Take the quiz and the speaking exercises to complete the your first lesson.

3- Access your favourites / saved words and check again the definition of the word “prototype” under the label “UX”.

4- Now, make a new search on the dictionary for the meaning of the word “Bla”.

5- Finally go to “My account”, and log out from the application.

1- You are being relocated to Amsterdam, and therefore want to learn some things in Dutch to get adapted quickly to the Dutch society. To speed things up a little, you have downloaded the “Talk Talk App” and you are ready to use it. Create a new account on the app and go to the homepage / dashboard of the app.

2- Now that you have your account created, you want to start learning right away. No time to waste here! Access the lesson with topic #1. Take the quiz and the speaking exercises to complete the your first lesson.

3- There are words you need to use accurately at work and so you created a Label with the name UX (User experience) on your favourites, where you save all words respecting that topic. Now you need to access the meaning of the word “prototype” you saved there. Access your favourites, then the label “UX” and the word “Prototype”.

4- You are now attending a conference and the presenter used a word you never heard before. You want to quickly look up its definition on your “Talk Talk” app. Go to the the search tap and look for the word “Bla” on the dictionary.

5- You have bought a new mobile phone you can’t wait to use, so you are login out of all your applications on the old phone, and reinstalling everything on the new one. You have to now logout of your “Talk Talk App”. Got yo your account and logout from the application.

USABILITY TEST SCRIPT

Hi, ___________. My name is Silvana, and I’ll be walking you through this session today.

Before we begin I have some information for you, so I can explain better the context of this session. I am going to read it to make sure everything is covered accordingly.

You probably already have a good idea of why I asked you to be here, but let me go over it again briefly. We’re asking people to try using a mobile vocabulary app that we’re working on so we can see whether it works as intended. The session should take about 10 minutes total.

The first thing I want to make clear right away is that we’re testing the app, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making mis-takes.

For this session I have prepared a list of tasks I would like you to test out on this mobile application. As you use the app, I’m going to ask you as much as possible to try to think out loud: to say what you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to me.

Also, please don’t worry that you’re going to hurt our feelings. I’m doing this to improve the app, so I need to hear your honest reactions. If you have any questions as we go along, just ask them. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them to help. But if you still have any questions when we’re done I’ll try to answer them then. And if you need to take a break at any point, just let me know.

I am not going to record this session, but I am going to take as many notes as possible, so I can review your remarks and work on this prototype later, once all sessions are finalised.

Now that we have a firm understanding of what to expect in this experience, I’d like to ask you a few easy questions about yourself.

USABILITY TEST PARTICIPANTS

NameRobert W.

Age35y/o

OccupationGlobal Service Integration

Owner

NationalityBrazilian

LocationThe Netherlands

NamePati D.

Age35 y/o

ProfessionMarketing specialist

NationalityItalian


NameSusana M.

Age37 y/o

ProfessionProduct Owner

NationalityPortuguese


PARTICIPANTS

NameCharles B.

Age29 y/o

ProfessionUX Student & English

teacher

NationalityBritish

LocationGermany

USABILITY TEST NOTES

NameRobert W.

Age35y/o

OccupationGlobal Service Integration

Owner

NationalityBrazilian



Task 1:Roberto found the whole process quite standard, however, I noticed that on the on boarding screens, he was initially trying to swipe, instead of clicking on the screen.He made a few remarks though:-When choosing his learning level he noticed the order should be the other way around, with beginner on top and expert on the bottom.-When choosing the topics, he could not understand why he would choose topics, It wasn’t clear for him what were this topics about and he needed some more information.Lastly, he also disapproved having the notification request after the topics screen. For him, it would make more sense to go straight to the homepage and place the notification request somewhere else.

Task 2:Roberto found this task very easy and completed it without problems. He said though that he found weird to come back to the lesson screen after responding to the quiz exercise.

Task 3:Roberto identified the favourite tab without any problem and made a remark about how clear it was to identify where he was at a given moment, having the icon highlighted/coloured.While thinking out loud, he mentioned though that the “Play” button would be to watch a video, which is not the case. He pointed out as well that he missed an icon for a home page.This talk was extremely easy for him to complete, although he questions the difference between the buttons to “add a ew word” and “Add a new label”.

Task 4:This task was also very easy for Roberto, and he completed it without any problem. He even mentioned “I like this app!”

Task 5:Roberto found this task also very easy, but he said the logout happened a bit “all of a sudden”. It would be nice to have some sort of confirmation, instead of such an abrupt end.


NameSusana M.

Age37 y/o

ProfessionProduct Owner

NationalityPortuguese


Task 1:Susana though the sign up process was very normal, but questioned (as the other 2 members) the need for choosing topics, and the notification request screen. For her the topics could be chosen as the user was picking a lesson and the notification request should not exist. “I get quite annoyed by these notifica-tions!”Despite those remarks, Susana finished this task accordingly.

Task 2:Susana completed this task without any problem. She mentioned she found it weird to complete the quiz and go back to the course page like that (very quickly), but she thought that in real life this would not happen that way.

Task 3:Susana completed this task without any issues. It wasn’t very clear to her what labels were, she said she would probably give them another name, like “collections”, or “files”.

Task 4:Susana didn’t understand why she had to check the meaning of the word on the dictionary, instead of checking its translation first. Despite this comment, she was able to complete the task.

Task 5:While searching for My Account, she mentioned she didn’t know what the “Play” button was for. She wanted to check it out, but this option was not available on this prototype at the time. She questioned where home and the courses/lessons were then. Once she identified the 4th icon as My account, she was able to logout without any problems.


NamePati D.

Age35 y/o

ProfessionMarketing specialist

NationalityItalian


Task 1:Patrizia was a bit confused on the first screen (splash screen). She didn’t know what she was supposed to do and concluded the screen was missing an entry button. “It is not intuitive, because I don’t know where I have to press.”Also in this task, Patrizia claimed that on onboarding processes where she usually sees 3 dots by the bottom of the page, her immediate reaction is to swipe. While trying to swipe, she realised that by clicking the image would also change, but to her this was not natural.In addition, Patrizia didn’t quite understand why she would need to select topics. She didn’t know what those topics were for.At last, Patrizia questioned me why the notification request was appearing now. For her this placement doesn’t make any sense, as she was expecting to enter the app at this point.

Task 2:The second task went well and Patrizia was able to complete it very quickly. She mentioned though that the abrupt change form the quiz back to the home was a bit weird, but she understood this was probably happening because this is a prototype, and in real life she would really have to go through the exercises to complete the first lesson.

Task 3:Patrizia immediately recognised the start with her favourites, so to find this tab was very easy to her. However, she struggled a bit with the UX label concept (she was not 100% aware what UX really was and what the label UX represented). Despite of this hiccup, she was able to complete the task.

Task 4:On this task Patrizia questioned the task itself. For her, if she would be learning Dutch, she would be attending a conference in Dutch, and therefore instead of looking for a specific word in the dictionary, she would translate it first. Despite this remark, Patrizia was able to complete the task.

Task 5:To complete this task Patrizia was looking for the My account page. From the screen where she was, she was expecting a button to go back to the homepage, and from there locate her account. She understood the My account icon with an “I”, and only a while later she understood what the icon really was and com-pleted the task easily.


NameCharles B.

Age29 y/o

ProfessionUX Student & English

teacher

NationalityBritish

LocationGermany

Task 1:Charles went through this first task very smoothly without any issues or concerns, even acknowledging “The log in process was very easy and smooth.”

Task 2:Charles also completed this task very easily. However, after completing the first part of this task, he questioned me about the second part of the task. He pointed out afterwards that the way I structured this task was perhaps not the best. Despite of that he was able to compete both requests.

Task 3:Charles also understood immediately that his favourite words were under the icon “star” and it was very easy for him to find his “favourite word” Prototype.

Task 4:Charles was also very quick finding the meaning of the word “Bla” on his “Talk Talk” app dictionary, and completed this task without any hustle.

Task 5:This last task was no exception, and Charles was able to complete it without a blink.

USABILITY TEST REPORT

Task 1:Charles went through this first task very smoothly without any issues or concerns, even acknowledging “The log in process was very easy and smooth.”

Task 2:Charles also completed this task very easily. However, after completing the first part of this task, he questioned me about the second part of the task. He pointed out afterwards that the way I structured this task was perhaps not the best. Despite of that he was able to compete both requests.

Task 3:Charles also understood immediately that his favourite words were under the icon “star” and it was very easy for him to find his “favourite word” Prototype.

Task 4:Charles was also very quick finding the meaning of the word “Bla” on his “Talk Talk” app dictionary, and completed this task without any hustle.

Task 5:This last task was no exception, and Charles was able to complete it without a blink.


TASK

TASK 1

TASK 2

TASK 3

TASK 4

TASK 5

ROBERTO SUSANA PATRIZIA CHARLES

3

1

2

1

1

2

1

1

1

0

0

1

0

0

0

3

1

2

0

1

As mentioned, for this test we’ll refer to an adapted version of Jakob Nielsen’s rating scale. Following the responses received from the participants, the errors were cate-goized as showsn on the following table.

Rating scale:0 = I don't agree that this is a usability problem at all1 = Cosmetic problem only: need not be fixed unless extra time is available on project2 = Minor usability problem: fixing this should be given low priority3 = Major usability problem: important to fix and should be given high priority4 = Usability catastrophe: imperative to fix before product can be released


TASK

TASK 1

TASK 2

TASK 3

TASK 4

TASK 5

3

1

2

1

1

OBSERVATION SEVERITY RECOMMENDATION

The task was easily completed by all participants. However, there were some remarks:1- Entry screen (without any button)2- Participants expected to have the on boarding screens change by a swipe (not by a click)3- Some participants didn’t understand the need to choose topics4- Some participants didn’t agree with having the notifications request at that point

Participants completed this task without problems. Some participants mentioned they found inconsistent to go back to the lesson page after completing the quiz in such an abrupt way, but recognised this happened because we are working on a prototype.

Participants were able to complete this task without a major problem. A couple of participants did not under-stand right away what labels were though.

A few participants questioned the task itself, rather the app. Everyone was able to compete this task.

All participants were able to complete this task, and only one didn’t recognize the My account icon right away.

-Splash screen with timer (a couple of seconds)-Onboarding screens to be ‘swipable’ (not clickable)-Is there really a need to choose topics now? Probably not. I’d recommend following the suggestion of one of the participants, and choose topics when users are choosing their lessons-Move the notifications request to another point in time

-Create the lessons materials and apply properly to the prototype.-Explain to the user that each lesson is made of a learning module, followed by a quiz, a video and speaking exercise.

-Clarify on the on boarding what a label is-Instead of calling them labels, change the name to “files”, “collections”, “folders”, “categories”

-In the following usability testing, change the task. This time we can ask participants to use the translator instead

-The following usability testing could be done with more concrete wireframes, so participants don’t struggle with the icons of the app.

ADDITIONAL RECOMMENDATIONS

Although all tasks were completely in a timely fashion, a few participants pointed out while executing the required tasks that the icons on the menu were not 100% recognisable.

None of the participants who made remarks on the menu identified the “play” button as the learning page / homepage / dashboard.

Following the remarks made, it could be recommended to modify the home / dashboard / lessons page icon.

Additionally, it would be beneficial to have all links and buttons on the prototype working, to allow future usability testing participants to explore the app, while trying to complete their desired and designated tasks.

Ideally there would be other rounds of usability testing to identify if the remarks made by the four participants were solved accordingly, or if any other changes were required.

PROTOTYPE - REVISED VERSIONhttps://prottapp.com/p/c45793

usability test plan€¦ · process. metrics there are many different ways to categorize and rank...

Documents