mailbox app usability test
TRANSCRIPT
1
Mailbox Usability Test
Megan McGowan Tiffany Mackins Drayton Beebe Jon D’Amato
2
Executive Summary
Mailbox is a lightweight, mobile first app that is meant to replace the stock mail app on
smartphones. The application’s main purpose is to help keep users’ email inboxes neat and
organized. Users have the ability to schedule emails to reappear in the inbox at a later, more
convenient time, and can create lists to organize their emails into predefined and personalized
categories to keep an organized inbox. A usability test was administered to determine if users
would agree with Mailbox’s claims of being simple and lightweight as well as to see if the app
could replace the native email app on the phone. The test was administered to 5 users who were
between the ages of 1823. Each participant had used mobile email in some capacity, but three
users were identified as power users. Along with an opening and closing questionnaire, users
were asked to complete 8 tasks, which were intended to mimic everyday email activities. Each
session was captured with the iRecord application and video from a Canon 7D camera.
The test results were:
∙ The average number of clicks to complete all tasks was 78 clicks.
∙ The average time for users to complete the tasks was between 4:43 and 7:33 (m:s).
∙ Users ranked the possibility of switching to Mailbox permanently at a 3 out of a
7point scale.
∙ Users’ satisfaction with the application was ranked at a 5 out of a 7point scale.
3
Introduction
Mailbox is an alternative mail app for existing Gmail, Google Apps and iCloud accounts.
It is currently available on Android, iPhone, iPad and Mac OSX. The application is designed to
keep users’ email inboxes neat and organized by making the common activities of email easier
and faster to manage. Users have the ability to schedule emails to reappear in the inbox at a later,
more convenient time. Also within the application, Mailbox users can create lists to organize
their emails into predefined and personalized categories to keep an organized inbox.
Mailbox features a clean interface design. On the ‘inbox’ screen, the only means of
navigation is a hamburger style menu in the upper left corner. The hamburger menu opens a
slideout menu that allows users to switch between accounts, view lists, archives, trash, access
settings and other email related tasks. Back on the ‘inbox’ screen, the app has four icons at the
top to access rescheduled emails, navigate back to the inbox, view archived emails and compose
a new message. The only text on the ‘inbox’ screen is the actual headings of the available emails.
Dropbox, which powers the Mailbox app, is used to add attachments to emails. Users are
required to have a Dropbox account if they wish to add attachments such as documents to
outgoing emails. Mailbox does allow the user to send photographs from the phone’s gallery
without the use of Dropbox.
The app utilizes the swipe functionality that many have become used to. Emails can be
quickly swiped left or right to be deleted, archived, added to a list, or scheduled to be read later.
From these actions, the app learns to predict future actions to take with specific emails. These
actions work to form a more intuitive app.
4
Currently, the application has between 500,000 to one million downloads on Google Play
and a rating of four stars by 23,871 reviewers (“Google Play Store”). In the Apple App Store, the
app has an overall rating of 4 starts as well. It’s important to note that the number of downloads
were not available from Apple’s App Store (“Apple App Store”). On Mailbox’s website, the
developers state that the intended users are anyone who uses email on a smartphone. The
application does not seem to be directly aimed at one demographic in particular, but instead uses
its simple and clean design to include everyone.
Because Mailbox’s main priority is to simplify the processes that go along with email,
one objective in this usability test was to determine if the application’s features were robust
enough to replace the native email app. In order to determine this, we asked the participants’ to
measure their satisfaction with Mailbox after completing a short list of tasks that explored the
main features of the app. The participants were also asked how likely they would be to replace
their native/current email app with Mailbox.
Another objective of the usability test was to determine if Mailbox’s interface was as
simple and lightweight as it was advertised to be. This measurement was determined by asking
the participants to complete a series of tasks they might encounter daily and rate the level of
difficulty of each task. The amount of errors and time it took to complete each task was also
taken into account. The usability test focused on the functions that Mailbox identified as actions
that were an improvement over other email applications.
Method
We conducted a test to measure the Mailbox app with five participants. This sample
consisted of three females, and two males between the ages of 18 and 23. The participants were
5
chosen via a sample of Elon University graduate and undergraduate students. Three participants
were in the Interactive Media graduate program, which is highly technologically centered. These
participants are considered our power users. The two undergraduate participants are nonpower
users. Alexa.com data state that the education of the site’s audience is college and graduate
school level, so we wanted to include a sample of both graduate and college students(“Alexa Site
Overview”). Since the intended Mailbox users are all email users, we were not constrained when
we were choosing participants for our study. These participants were contacted prior to the study
and agreed to participate.
Prior to the usability test, we prepared materials to ensure the same test environment for
each participant. We prepared a consent form (Appendix A), a script to read to participants prior
to the test (Appendix B), and a pre and posttest questionnaire (Appendix C). The test was
conducted in two identical editing suites on the second floor of Powell at Elon University.
Alexa.com data state that the main browsing location of the site’s audience is at work, so we
wanted to simulate more of a work environment, which is why we chose the editing suites
(“Alexa Site Overview”). We also wanted to test the participants in a lab setting to ensure more
control and to make recording the test more convenient.
For the usability test, participants were seated in the testing environment and read the
script by one of the test administrators. During this, they were asked to sign a consent form.
Participants then filled out the pretest questionnaire. We asked participants to complete realistic
and personal scenarios they are likely to encounter as a user of the Mailbox app (Appendix D).
These scenarios include “Find Megan’s email with the subject Class at 12:25. You don’t have
this class
6
until tomorrow. Set the email to be resent tomorrow.” We hoped that this wording would make it
easier for participants to navigate through the app.
The task scenarios were descriptive enough to give the participants enough information,
but not enough to lead them to the intended outcome. During the test there were two researchers
present in the room per one participant. One researcher was there to administer the test, and the
other to record on a Canon 7D. We also recorded the participant’s actions on the iPhones used in
the test to track where the participants clicked and navigated on the app. After the test, the
participants were encouraged to enjoy Dark Chocolate and Pumpkin Spice flavored Milano
cookies.
Results
Using the data collected from iRecord, video analysis, and the posttest questionnaire, we
measured the clicks per task, time per task, and difficulty rating (Appendix E). The number of
clicks and time per task were used to measure the effectiveness of the interface and whether it
met Mailbox’s goal of a “clean and lighter” interface design. The error rate is related to interface
design errors and errors within the testing environment. We chose to measure the difficulty
rating to understand how the number of clicks and time per task affected the user’s perception in
ease of use.
The average number of clicks per task allowed us to see how users maneuvered
throughout the interface to complete each task. The average number of clicks needed by our
users to complete all task was 78 clicks. It is important to note that we had a wide variety within
our dataset with the lowest number of total clicks at 38 and the highest at 114 clicks. The user
who completed the task list in 38 clicks is categorized as a power user from the results of the
7
user’s pretest questionnaire where the user stated that they use mobile email on a daily basis
across multiple platforms, which may account for the minimal number of clicks.
We also broke down each task and calculated the average clicks per task. This data
allows us to understand which task presented the most difficulty to users and compare this
information with the results of their posttest questionnaire. The results in Chart One show that
users needed a high number of clicks to complete task four and task six. For task four, we
conclude that the interface does not reach its goal to provide a clean and intuitive process when
sorting emails into categories. The high number of clicks indicates that users needed to look
around the interface and try multiple options to complete the task, which suggest that the process
has a high learning curve and is unfamiliar to new users, creating confusion.
(Chart One: Average number of clicks per task)
8
We believe that the results of task six are skewed due to an outlier. While the
average number of clicks is 19, the median is 16. There was also a participant in task six who
fell outside the first deviation results, which leads us to conclude that the average for task six is
closer to the median rather than the mean. In task six, users were asked to sort the remaining
emails to achieve mail zero. Due to errors within a previous task, participant four had a higher
number of emails in the inbox at the beginning of task six, which led to a higher number of clicks
to reach mail zero. Participant four also chose to read each email before sorting, increasing the
number of clicks for each email to be sorted.
The time for each task and overall time allows us to understand how long users spent
within the interface to familiarize themselves with the actions and complete the task list. The
average time for users to familiarize themselves with the interface and complete the six tasks is
five minutes and 56 seconds. The quickest participant completed the test in three minutes and 50
seconds while the slowest participant took seven minutes and 34 seconds to complete all of the
tasks. Both of these users are outside of the first deviation of data, which means that an average
user will have different results. Using the standard deviation and the average total time, we
conclude that an average user will use between four minutes and 43 seconds to seven minutes
and thirty seconds to complete all of the tasks.
9
(Chart Two: Average time (in seconds) per task)
The results are also broken into the average time per task to understand how users
function within different sections of the interface in Chart Two. The time it took to complete
task four is consistent with the average number of clicks needed allowing us to conclude that task
four was click and time intensive due to either its intricacy or its difficulty. Further study of task
four is noted in the study of difficulty ratings. When analyzing the time per task, we were
surprised by the time took to complete task two. Task two, which was to resend an email at a
later date, needed the lowest number of clicks to complete but had an average completion time
than expected. We believe that the discrepancy may be due to a difference in language and the
unfamiliarity with the feature. During the task, we asked the users to “resend the email to
themselves tomorrow” but the language within the interface is “save for later”. The video
recordings and iRecord data lead us to believe that users had difficulty understanding that the
10
“save for later” function was the correct button to resend the email. The time for this task
increased as users made sense of the task and found the function. Once users understood the
task, it took a lower number of clicks to complete the task.
The lower time associated with task six also shed light on the ease of use. Task six, took
users an average of 19 clicks to complete but only one minute to complete those 19 clicks. This
data allows use to conclude that while sorting the inbox uses a higher number of clicks it is not
an intensive task. Task six was also the only task where every participant fell within the first
standard deviation, allowing us to conclude that the average user will also need an average of
one minute to complete sorting their inbox.
To completely understand the user experience of Mailbox app it is important to study the
posttest questionnaire and the users own perception of difficulty. The overall difficulty rating
for Mailbox is 5.1 using the range of one as the most difficult and seven as the easiest. This
difficulty rating helps us to conclude that users had a moderately positive interaction with the
interface and judge it to be in the middle of the road for difficulty. As Mailbox aims to be the
easiest and most intuitive mail app offered there must be some changes made or a higher quality
tutorial to satisfy the users judgment of ease of use.
11
(Chart Three: Difficult Rating Per Task)
The difficulty ratings per tasks follow closely with the average number of clicks and time
needed to complete the test. Task four, which needed the highest number of clicks and the
longest average time to complete, also received the highest difficulty rating at 4.25. This allows
us to conclude that task four, which is adding an email to a category, is the hardest task to
complete within the Mailbox interface. It is also one of the two tasks that were not completed by
a participant. This data shows that Mailbox’s goal of an easy to use and lighter mailbox breaks
down when users need to add their mail to a category.
In task two, the lower difficulty rating is also reflected in the other results. We believe
that this is due to the unfamiliarity with being able to save an email for a later point in time. This
feature is unique to Mailbox and new to the participants. Understanding the function of this
12
feature increased the time of the task and the difficulty rating, but once the task was understood it
was easy to complete in a smaller number of clicks. The low number of clicks and moderate
difficulty rating lead us to conclude that saving an email for a later date is an easy function once
a user is familiar with the Mailbox interface.
Task six, also has a rating that leads us to conclude that it is a more difficult task. Task
six, utilizes a high number of clicks, average time, and lower difficulty rating, leading us to
believe that the task is too complicated. Mailbox intends for the sorting of the inbox to achieve
mailbox zero to be an intuitive and easy task, but the users found it overly complicated as
indicated by the high number of clicks and higher difficulty rating.
Discussion
While actual testing is important in gathering data, a large part of this usability test lies in
analyzing the data retrieved. In a testing group of five, two participants gave information that
skewed the average experience. We’re treating these instances as outliers not because they meet
the required third deviation, but because our testing group was small.
To gauge satisfaction with Mailbox, participants answered a posttest
questionnaire to rate different aspects of their experience. The test used a sevenpoint scale to
rate satisfaction. (Figure 1)
Figure 1
13
The overall difficulty rating for Mailbox is positive at 5.1 and participants showed overall
positive satisfaction with the app with a rating of 5. While these ratings should and do
correspond with each other, participants did not think Mailbox could replace their current email
application. The average likeliness of switching to Mailbox was a 3, a slightly negative rating on
the sevenpoint scale. This brought us to the realization that ease of use and satisfaction are not
the only factors that govern how a user chooses an interface. Habits, preferences, and prior
experience play heavily on user interaction. Also, while satisfaction was only slightly positive
with a rating of 5 and chance of replacement was equally negative with a rating of 3, it is
possible that participants were simply not impressed enough with Mailbox to change services. If
the average satisfaction rating was higher, it is possible that the likelihood of replacement would
see higher ratings as well.
While errors were made, both by participants and testers, this was an overall successful
user test. If this test influenced the design and development of the Mailbox app, the information
provided would be useful to the team in charge of making informed changes to the interface.
14
References
“Alexa Site Overview”.
<http://www.alexa.com/siteinfo/http%3A%2F%2Fwww.mailboxapp.com>
“Apple App Store”.
<https://itunes.apple.com/us/app/mailbox/id576502633?mt=8>
“Google Play Store”.
<https://play.google.com/store/apps/details?id=com.mailboxapp&hl=en>