evaluation assignment 6 usability test report · 2018. 4. 30. · 3.4.3.6 potential observations of...

1

CS5760

Graduate Human-Computer Interactions & Usability

Evaluation Assignment 6

Usability Test Report

Jiban Adhikary (Grad 9)

Team 2 (Blu Team)

April 30, 2018

2

Table of Contents:

1 Introduction 3

2 Undergraduate System 3

3 Usability Test Plan 3

4 Results 8

5 Conclusion 17

Appendix A (Undergrad student attendance) 19

Appendix B (Bug report) 20

Appendix C (Consent Form) 21

Appendix D (Pre-test Questionnaire) 22

Appendix E (Post-test Questionnaire, Scenario 1) 23

Appendix F (Post-test Questionnaire, Scenario 2) 24

Appendix G (Post-test Questionnaire, Scenario 3) 25

Appendix H (Post-test Questionnaire, General) 26

3

1 Introduction

This documentation provides a detailed usability test report for a smartphone app. First, it provides a summary of the app or software that is being developed by a team of undergraduate students, the Blu Team. Then it outlines the usability test plan. Next, it discusses the usability test results. The documentation also includes some forms in the Appendix sections. These forms were used throughout the test process. If you have read the Evaluation Assignment 4, then you can skip reading the sections 2 to 3.

2 Undergraduate System

Blu Team is working on an app which is basically a Volunteer Monitoring app. Volunteers are the backbone of Oklahoma Blue Thumb program, the eyes and ears of creeks and streams across the state. By using the app the volunteers will monitor the creeks monthly. They will assess the physical habitat, test sample of the creek’s water. The monitoring will take place at the creek, and all the data related to environmental or physical factors affecting the creeks will be collected. It will also involve collecting a sample of water which will later be chemically tested in the lab. The chemical tests are supposed to be held on a comfortable time the volunteer prefers to do. Previously all of this data were sent out to Blue Thumb using traditional media like fax, email, or snail mail. This app will ease the monitoring process by facilitating volunteers to submit data using their smartphones and email them to the quality assurance officer. The app is now ready to run some user trial and this documentation provides the procedural way to conduct those user trial.

3 Usability Test Plan

The usability test plan events included the following:

I.Welcome / Introductory Events II.Signing of the consent form

III.Pre Test questionnaire / Demographics IV.Carrying out the test tasks V.Post Test Questionnaire

3.1 Welcome / Introductory Events

The experimenter welcomed the participant and had the participant sit at a desk or table. The participant sat at this location while they completed initial paperwork.

The experimenter described the objective of the app and why it had been developed. The experimenter ascertained the participant that there were no risks that could possibly be risen from running the usability tests. The experimenter explicitly asked the participant if there was any question and answered if there was any.

http://www.csl.mtu.edu/classes/cs4760/www/projects/s18/grad9/www/Eval_Assignment4.pdf

4

The experimenter also mentioned that the data being collected will be kept confidential and any link to trace the identity of the participant will be destroyed.

3.2 Signing of The Consent Form

The experimenter handed out the consent form to the participant. The participant read the consent form and after signing it he returned it to the experimenter. The experimenter answered if there was any question asked by the participant. A sample consent form is included in Appendix C.

3.3 Pre Test Questionnaire

Before the usability task tests, the participant filled out a demographic/ pre - test questionnaire. A sample pre-test questionnaire has been included in Appendix D.

3.4 Carrying Out The Test Tasks

Three of the following three scenarios were evaluated during the test. After each of the test tasks had been completed, the participant filled out a questionnaire specific to the scenario completed.

3.4.1 Test Scenario 1: Visibility of icons and texts

3.4.1.1 Goals:

To check if the icons used in the app are of proper sizes To check if the font size of the texts in the app are appropriate To check whether the background color and identical buttons have consistent

color

3.4.1.2 Quantitative measurement:

Number of cases where the font size seemed too big or too small Number of cases where the color of buttons or texts or background seemed

inconsistent Approximate amount of time to interact and get response from the app Number of bugs encountered around the process

3.4.1.3 Scenario description: The participant was asked to launch the app from the beginning. He was asked to navigate through the pages and fill in some random fields. He was advised to keep some mental notes about the icons and text labels used in the app while he was navigating through. He was asked to locate any button or text at random saying like - “Can you locate the next button?”

3.4.1.4 Task list: The participant did the following tasks -

5

Launch the app Navigate through the pages Keep mental notes of the texts and icons Report

3.4.1.5 Qualitative measurement list:

How difficult was it to find a specific button or text? Did the user find any difficulties to read any text?

3.4.1.6 Potential observations of participant:

Facial expression of the participant Was participant moving his head back and forth occasionally? Was the distance between the smartphone and participant’s eyes inconsistent? If the participant asking questions?

3.4.1.7 Bug Report Form: A sample bug report form is presented in Appendix B. Multiple forms was used if necessary.

3.4.1.8 Post Scenario interview or questionnaire questions:

Scenario specific questions: Questionnaire for scenario 1 can be found in Appendix E

3.4.1.8 Test set up details: No specific set up was necessary. A participant was asked to use his smartphone and was given proper suggestions to find out the app icon.

3.4.2 Test Scenario 2: Alignment, text wrapping and natural flow of app pages

3.4.2.1 Goals:

To check if form labels and input fields have been properly aligned To check whether input text is wrapped within the input field To check if there is a consistent flow of pages while navigating back and forth


Number of cases where the texts or icons were not properly aligned Number of cases where user could not navigate to next or previous page Number of cases where texts were not wrapped for a large block of input text Number of events where user could not slide up and down if necessary Number of bugs encountered around the process

3.4.2.3 Scenario description: The participant was asked to launch the app. He was asked to notice the text and object alignment and was asked to enter a block of paragraph in an input field like comment section. He was also asked to navigate back and forth or up and down

6

3.4.2.4 Task list: The participant did the following -

Launch the app Notice the alignments Enter a block of texts Navigate back and forth or up and down


Which form of alignment was appropriate Was the sliding up and down bar visible


Facial expression of the participant Finger movements of the participant while sliding up and down or navigating

back and forth If the participant asking questions?

3.4.2.7 Bug Report Form: A sample bug report form is presented in Appendix D. Multiple forms will be used if necessary.


Scenario specific questions: Questionnaire for scenario 2 can be found in Appendix F


3.4.3 Test Scenario 3: Text input validation, feedback and exceptions

3.4.3.1 Goals:

To check if the app validates user input for range specific values To check what kind of feedback the app provides To check if the app has a fail-safe mechanism in case of an unusual event


Number of text fields where user can input arbitrary data Number of events a user is not provided with any feedback while trying to input

invalid data Number of bugs encountered around the process

3.4.3.3 Scenario description: The participant was asked to start up the app. He was asked to interact with the help functions. He was also asked to enter some arbitrary values in the input fields. Finally, he was asked to tilt or shake the smartphone or enable/disable the wifi.

3.4.3.4 Task list: The participant did the following -

7

Launch the app Try the help functions or labels with ‘?’ mark on it Enter arbitrary input Shake/tilt the phone, or enable/disable wifi etc


Did the app crash if it is tilted/shaken or wifi is enabled/disabled


Facial expression of the participant If the participant asking questions?

3.4.3.7 Bug Report Form: A sample bug report form is presented in Appendix B. Multiple forms will be used if necessary.


Scenario specific questions: Questionnaire for scenario 3 can be found in Appendix G


3.5 Post Test Questionnaire

The participant filled out a general questionnaire after the above scenarios had been completed. The general questionnaire is included in Appendix H.

8

4 Results

4.1 Pre Test Questionnaire Results

Pre-test questionnaire results are depicted in the following figures.

Uses smartphone for

Talking to people Texting Emailing Playing games Web browsing

Figure 4.1: Pre-test questionnaire results

4.2 Test Results on Scenario 1: Visibility of icons and texts

The post-test questionnaire conducted after the first test scenario reflects that the users liked the UI elements. According to their thought UI elements were precise and appropriate.

40%

60%

0%

Smartphone OS

iOS

Android

Others 1 1

3

smaller than4.7

4.7-5.2 larger than5.2

Screen sizes of the used smartphones (inches)

0

1

2

3

4

5

less than ayear

1-2 years 2-3 years more than3 years

Smartphone usage

Prior experience to survey water body

Yes No

Prior experience to run a chemical test

Yes No

9

Figure 4.2.1: User satisfaction on the size of the texts

Figure 4.2.2: User satisfaction on the sizes of the app icons

Figure 4.2.3: User satisfaction on the colors of elements and background

-1

1

3

5

StronglyDisagree

Disagree Neutral Agree StronglyAgree

I thought the sizes of the texts were precize

012345

StronglyDisagree


I thought the size of the icons used in the app were precise

012345

StronglyDisagree


I thought the colors of the texts and background were consistent

10

Figure 4.2.4: User satisfaction on the sizes of the buttons

4.3 Test Results on Scenario 2: Alignment, text wrapping and natural flow of app pages

In this scenario, a great user satisfaction was observed. All of the participants reported that the navigation from one page to another, using the slide bar and text wrapping were as good as it was required.

Figure 4.3.1: User satisfaction alignment of text labels and input fields

0

1

2

3

4

5

StronglyDisagree


I thought the size of the buttons were appropriate

0

1

2

3

4

5

StronglyDisagree


I thought the texts and inputs fields were properly aligned

11

Figure 4.3.2: User satisfaction on page navigation

Figure 4.3.3: User satisfaction on sliding up and down

Fig: User satisfaction on text wrapping

0

1

2

3

4

5

StronglyDisagree


I thought I could navigate back and forth quite easily

0

1

2

3

4

5

StronglyDisagree


I thought I could easily use the sliding bar to slide up and down an app page if necessary

0

1

2

3

4

5

StronglyDisagree


I thought the texts were wrapped within the input fields

12

4.4 Test Results on Scenario 3: Text input validation, feedback and exceptions

In this test scenario a mixed type of satisfaction was reported by the user. It was due to the fact that most of the input fields were not validated and there had been cases where user was able to enter invalid information in the input fields. For example, when entering a floating point number user could enter number with multiple points, user could enter negative values for volume of different chemical elements, user could submit forms without entering any value in the input fields. Also, when adding multiple volunteers in the last page, the app let the user add a volunteer without filling out the information of the previous volunteer. From a real world application’s perspective, these issues are important to resolve.

Figure 4.4.1: User satisfaction on invalid data entry. Users were not happy since many of the input fields were not validated.

Figure 4.4.2: User satisfaction on feedback. Users thought the feedback was not enough.

0

1

2

3

StronglyDisagree


I thought I could not enter invalid information

0

1

2

3

4

5

StronglyDisagree


I thought the app provided reasonable feedback

13

Figure 4.4.3: User satisfaction on application crash

Figure 4.4.4: User satisfaction on useful help information

4.5 Overall User satisfaction

This section reports the overall user satisfaction of the users after using the app. This section is based on the results from the general post-test questionnaire. User satisfaction on many aspects were great. When asked if the users liked the app or not, everyone reported that he/she liked the app.

0

2

4

6

StronglyDisagree


I thought the app did not crash suddenly

0

1

2

3

4

5

StronglyDisagree


I thought the app provided useful help information

14

Figure 4.5.1: Users’ understanding level of the app

Figure 4.5.2: Users satisfaction on app latency

Figure 4.5.3: Users’ familiarity with the app labels

0

1

2

3

4

5

StronglyDisagree


I thought I clearly understood the objective of the app

0

1

2

3

4

5

StronglyDisagree


I thought the app took precise time to respond to interactions

0

1

2

3

4

5

StronglyDisagree


I thought I am familiar with the app labels

15

Figure 4.5.4: User satisfaction on sudden crash

Figure 4.5.5: User satisfaction on app complexity.

Figure 4.5.6: User satisfaction on the app interface

0

1

2

3

4

5

StronglyDisagree


I thought the app did not behave weirdly

0

12

34

5

StronglyDisagree


I thought the design of the app was too complex

0

1

2

3

4

StronglyDisagree


I thought I was satisfied with the app interface

16

Figure 4.5.7: Whether user would use the app for volunteering purposes

Figure 4.5.8: Whether users liked the app or not

4.6 User comments

“It was easy to use”

“Sleek design, easy to use, easy to use with phone, fast”

“Going back and forth with navigation bar was simple/easy”

“It was very fast and responsive. It did well to use my native calendar and clock”

“I liked the drop down menu bar in the upper left of the screen”

0

1

2

3

4

5

StronglyDisagree


If I were a volunteer of Oklahoma Bluthumb project I would certainly use this app

Did you like the app?

Yes No

17

5 Conclusions

Based on the tests conducted the following issues are suggested to resolve

Issue 1: ‘Submit’ button does not work

Description: In the final page of the app after a user finishes entering all the information he enters the ‘Submit’ button. But when using the app there was no way to tell if the button worked. For a practical application when user enters the submit button usually a pop up message is displayed and a new form may also appear. There were no such events in our case.

Resolution: A pop up window can be displayed confirming user that their data has been submitted and a new log file can appear.

Issue 2: Focus on input field does not change on enter

Description: In the test when user was done entering a value for a text input field and pressed enter the cursor did not move to the next input field. The user had to touch outside of the

18

keyboard and then select the next input field again to enter values.

Resolution: The developers need to add some functionality to move the focus on the next field after enter or go is pressed in the keyboard.

Issue 3: Input fields were not validated

Description: Many of the input fields were not properly validated

o Invalid fractions: User could enter fractions with multiple points o Negative values: User could enter negative volumes for chemical elements such as

Oxygen, Nitrogen, Phosphorus etc. o Empty volunteer: The app would let user add multiple volunteers without checking

any information in the previous volunteer form was entered or not.

Resolution:

o Validate form fields as necessary so that invalid fractions cannot be entered o Validate form fields as necessary so that negative volume cannot be entered o Check whether a volunteer information has been added before proceeding to the

next volunteer

19

Undergraduate group member attendance at testing

Test # Date and time Name of the undergrad students

Test 1 3.00 – 4.00 PM, 4/13/18 Zachary Paris



Test 4 3.00 – 4.00 PM, 4/17/18 Nobody attended



Appendix B

20

BUG REPORT FORM

NUMBER BUG NAME UNIQUENESS LOCATION DESCRIPTION

1 Submit button

worked or not?

Consistent

through all test

Final page No popup message or a

new log form appeared

when pressed

2 User could

enter invalid

fraction

numbers

Form fields that

allow fraction

values

Invalid fractions like

12.32.25 could be

entered

3 User could

enter negative

volumes

Form fields which

required to enter

volumes

Negative volumes like -

12 could be entered

Definitions___________________________________________________________________________________________

NUMBER: A bug number is an identifier provided by the usability test administrator when a user encounters a bug.

BUG NAME: The name of an encountered bug which can easily be distinguishable and comprehendible later on.

UNIQUENESS: An asterisk is assigned if the user is encountering the bug for the first time and no other user has encountered it before.

LOCATION: The page where the bug was encountered.

DESCRIPTION: A formal description of the bug that has been encountered.

Appendix C

21

Computer User Interface Usability Testing Consent Form

You are being invited to participate in a research study to determine the usefulness and usability of computer user interfaces. This study is being conducted by Dr. Robert Pastel of Michigan Technological University Computer Science Department and students in Dr. Pastel's Human-Computer Interaction (MCI) courses. The students are performing the usability tests as part of their project and to fulfill the HCI course requirements.

There are no known risks if you decide to participate in this research study. There are no costs to you for participating in the study. The information you provide and tasks that you will perform will determine the usefulness and usability of user interfaces. The questionnaires and the tasks should take less than an hour to complete. The information collected may not benefit you directly, but the information learned in this study should provide more general benefits.

The questionnaires and test are anonymous. Do not write your name on the survey. No one will be able to identify you or your answers, and no one will know whether or not you participated in the study except for the instructor of the class that is giving you credit for participating, Should the data be published, no individual information will be disclosed.

Your participation in this study is voluntary. By completing the questionnaires and performing the tasks, you are voluntarily agreeing to participate. You are free to decline to answer any particular question you do not wish to answer or not to perform a task for any reason.

If you have any questions about the study, please contact Dr. Robert Pastel, Assistant Professor, Computer Science Department, Michigan Technological University, Houghton, MI 49931.

The MTU Institutional Review Board has reviewed my request to conduct this project. If you have any concerns about your rights in this study, please contact Joanne Polzien of the MTU-IRB at 906-487-2902 or email [email protected].

Participant signature and date:

Appendix D

mailto:[email protected]

22

PRE-TEST QUESTIONNAIRE (1 of 2)

Participant ID: _______ Age (approximate) : _______ Gender : _

I am currently studying at a university:

□ Yes □ No

If you answered YES, what is your major? _______________________________________

I normally use a smartphone with the following operating system:

o iOS

o Android

o Do not use smartphone

The physical size of my smartphone screen is (only check one):

o 3.7 ~ 4.7 inches o 5.0 ~ 5.2 inches

o > 5.2 inches o Not applicable (check only if you do not use a smartphone)

I have been using a smartphone for (only check one):

o less than a year o 1~2 years

o 2~3 years

o more than 3 years o Not applicable (check only if you do not use a smartphone)

I use my smartphone for the following activities (check all that apply):

□ talking to people □ texting □ emailing □ playing games □ web browsing □ Not applicable (check only if you do not use a smartphone)

Appendix E

23

POST TEST QUESTIONNAIRE (Scenario 1)

Please indicate your level of satisfaction with the app you have just worked on. Fill in the circle that reflects your satisfaction level:

I thought the sizes of the texts were precise

Strongly Disagree ○ ○ ○ ○ ○ Strongly Agree

I thought the size of the icons used in the app were precise


I thought the colors of the texts and background were consistent


I thought the size of the buttons were appropriate


In the space below provide any comment you like to share about this test scenario:

Appendix F

24

POST TEST QUESTIONNAIRE (Scenario 2)


I thought the texts and field were properly aligned


I thought I could navigate back and forth quite easily


I thought I could use the sliding bar to navigate up and down a app page if necessary


I thought the text were wrapped within the input field



Appendix G

25

POST TEST QUESTIONNAIRE ( Scenario 3)


I thought I could not enter invalid information


I thought the app provided reasonable feedback


I thought the app did not crash all on a sudden


I thought the app provided useful help information



Appendix H

26

POST TEST QUESTIONNAIRE (General)


I thought I clearly understood the objective of the app


I thought the app took precise time to respond to interactions


I thought I am familiar with the app labels


I thought the app did not behave weirdly


I thought the design of the app was too complex


I thought I was satisfied with the app interface


If I were a volunteer of Oklahama blue thumb project I would certainly use this app


Based on your experience please provide some feedback on the following questions:

Did you like the app? ___________________

In your opinion, is there anything that you did not like and why?

Is there anything that you liked very much and why?

Please share any comments you have about the study in general:

evaluation assignment 6 usability test report · 2018. 4. 30. · 3.4.3.6 potential observations of...

Documents