electronic health record usability test report of dosespot...
TRANSCRIPT
1
Electronic Health Record
Usability Test Report of
DoseSpot 7.2
Michelle Kohut
DoseSpot
Jessica Maciel
Integration Division
DoseSpot
September 2019
2
Electronic Health Record Usability Test Report of
DoseSpot 7.2
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
DoseSpot Version 7.2
Date of Usability Test: 09/09/2019
Date of Report: 09/17/2019
Report Prepared By: Jessica Maciel Jessica Maciel, Integration Manager
781-355-6250 x236
18 Crawford St 2nd Floor
Needham MA, 02494
Table of Contents
1 EXECUTIVE SUMMARY
2 INTRODUCTION
3 METHOD
3.1 PARTICIPANTS
3.2 STUDY DESIGN
3.3 TASKS
3.4 PROCEDURE
3.5 TEST LOCATION
3.6 TEST ENVIRONMENT
3.7 TEST FORMS AND TOOLS
3.8 PARTICIPANT INSTRUCTIONS
3.9 USABILITY METRICS
4 RESULTS
4.1 DATA ANALYSIS AND REPORTING
4.2 DISCUSSION OF THE FINDINGS
5 APPENDICES
5.1 Appendix 1: Participant Demographics
5.2 Appendix 2: Non-Disclosure Agreement (NDA)
5.3 Appendix 3: Example Moderator’s Guide
5.4 Appendix 4: Orientation Script
5.5 Appendix 5: Task Workflows
5.6 Appendix 6: System Usability Scale Questionnaire
3
EXECUTIVE SUMMARY
A usability test of DoseSpot, 7.2 was conducted on 09/09/2019 at 18 Crawford St. Needham, MA 02494 by
DoseSpot employees. The purpose of this test was to test and validate the usability of the current user interface,
and provide evidence of usability in the EHR Under Test (EHRUT).
During the usability test, 10 healthcare providers matching the target demographic criteria served as participants
and used the EHRUT in simulated, but representative tasks. This study collected performance data on 10 tasks
typically conducted on an EHR:
E-prescribe new prescription
Change prescription
Cancel prescription
Refill prescription
Request and receive medication history info
Record a medication allergy
Change a medication allergy
Display the active medication allergy list
Trigger a drug-drug interaction by entering a new medication order
Trigger a drug-allergy interaction by entering a new medication order
During the 45 minute one-on-one usability test, each participant was greeted by the administrator
and asked to review and sign an informed consent/release form (included in Appendix 3); they
were instructed that they could withdraw at any time. Participants did not have prior experience
with the EHR.4 The administrator introduced the test, and instructed participants to complete a
series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed
the test and, along with the data logger(s) recorded user performance data on paper and
electronically. The administrator did not give the participant assistance in how to complete the task.
The following types of data were collected for each participant:
o Number of tasks successfully completed within the allotted time without assistance
o Time to complete the tasks
o Number and types of errors
o Path deviations
4
o Participant’s verbalizations
o Participant’s satisfaction ratings of the system
All participant data was de-identified – no correspondence could be made from the identity of the participant to
the data collected. Following the conclusion of the testing, participants were asked to complete a post-test
questionnaire and were compensated with $100 gift card for their time. Various recommended metrics, in
accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability
of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the
performance and rating data collected on the EHRUT.
Measure
Task
N
Task Success
Path Deviation
Task Time
Errors
Task Ratings 5=Easy
+Risk Ratings
#
Mean Total Deviations (Observed)
Mean Optimal # of Steps
Mean
Mean
1.E-prescribe New Prescription
10 100% 4 80
6 0 4.30
3
2. Change Prescription
10 100% 2 43
6 0 4.20
2
3. Cancel Prescription 10 100% 0 9
4 0 5.00
2
4. Refill Prescription 10 100% 0 33
6 0 4.20
2
5. Request and Receive Medication History Info
10 100% 0 8
3 0 4.70
1
6. Record a Medication allergy
10 100% 0 20
3 0 4.10
2
7. Change a medication allergy
10 100% 0 24
4 0 4.20
2
8. Display the active medication allergy list
10 100% 0 5
1 0 4.90
1
9. Trigger a drug-drug interaction by entering a new medication order
10 100% 2 38
5 0 4.10
2
10.Trigger a drug-allergy interaction by entering a new medication order
10 100% 1 54
5 0 4.10
2
+ Risk Ratings were scored on a scale of 1-5. The higher the risk rating for a task, the larger the impact that task may have on a client or patient:
1. Least risk- negligible impact on client/patient 2. Slight risk- minor impact on client/patient 3. Moderate risk- some impact on client/patient 4. High risk- challenge to client/patient
5
5. Severe risk- major/critical impact on client/patient
The results from the System Usability Scale scored the subjective satisfaction with the system based on
performance with these tasks to be: 72.25. This number was compiled as an average out of the 10 participants
Usability scores. The scores ranged from 62.5-80.
In addition to the performance data, the following qualitative observations were made:
Major findings
o Overall many users commented on the ease of use of the product specifically referring to the appearance and
design of the UI.
o Feedback varied across participants. Such as ensuring the location of the Patient demographic information
remained constant and did not move when navigating the rest of the UI
o In order to send a prescription a pharmacy must be added. Some found adding a preferred pharmacy difficult
as the page did not close down immediately.
o Navigation across the UI was well received, but there was some confusion on the difference between Inactive
Medications and Medication History.
o Pulling up and changing allergies went well
o Changing a pending prescription was also well received.
Areas for improvement
o Given the feedback, areas of improvement have been narrowed in on improving the flow and use of the UI.
While not all participants commented on the same thing, the patient information remaining visible was of
importance to many.
o Very few errors were made, but hesitancy was observed in knowing the right place to instinctively go for next
steps. This could be attributed to the size of the tiles and buttons. The size of everything in the screen does
have the ability to impact use. Some users found their eyes are drawn to the larger buttons/boxes which can
make it easy to overlook other areas, such as required fields.
o A few users would accidentally forget to fill in required fields when sending prescriptions. As a result we are
looking into ways to make this more obvious.
o If a task was successful the notification is small and does not stay on the screen for long, so it left some users
unsure of whether or not they had completed the task.
6
INTRODUCTION
The EHRUT tested for this study was DoseSpot 7.2. Designed to present medical information to healthcare
providers and send electronic prescriptions in facility types such as Dental Offices, Hospice facilities and
Medical facilities, the EHRUT consists of a web application with a database. It communicates with
Surescripts and receives and send messages from Surescripts to facilitate the sending of electronics
prescriptions. The usability testing attempted to represent realistic exercises and conditions. The purpose of
this study was to test and validate the usability of the current user interface, and provide evidence of
usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user
satisfaction, such as time on task, deviations from standard operating workflows and ease of use, were
captured during the usability testing.
METHOD
PARTICIPANTS
A total of 10 participants were tested on the EHRUT. Participants in the test were most RNS or in a similar
field. Participants were recruited by employees of DoseSpot, Christina Salvucci and Jessica Maciel and were
compensated $100 gift card for their time. In addition, participants had no direct connection to the
development of or organization producing the EHRUT(s). Participants were not from the testing or supplier
organization. Participants were given the opportunity to have the same orientation and level of training as
the actual end users would have received.
Recruited participants had a mix of backgrounds and demographic characteristics. The
following is a table of participants by characteristics, including demographics, professional
7
experience, computing experience and user needs for assistive technology. Participant names
were replaced with Participant IDs so that an individual’s data cannot be tied back to individual
identities.
Part
ID
Gender
Age
Education
Occupation/
role
Professional
Experience
Computer
Experience
Product
Experience
Assistive
Technology
Needs 1 F 34 Bachelor’s Degree LPN 156 180 0 No
2 F 50 Bachelor’s Degree RN 288 240 0 No
3 F 43 Associate Degree RN 60 180 0 No
4 F 32 Bachelor’s Degree RN 36 230 0 No
5 F 31 Bachelor’s Degree RN 72 192 0 No
6 F 54 Bachelor’s Degree RN 360 360 0 No
7 M 29 Bachelor’s Degree RN 50 250 0 No
8 M 39 Doctorate Degree Physician 87 220 0 No
9 M 25 Bachelor’s Degree Clinical Assistant 32 210 0 No
10 F 28 Bachelor’s Degree RN 30 230 0 No
10 participants (matching the demographics in the section on Participants) were recruited and 10
participated in the usability test. 0 participants failed to show for the study. Participants were scheduled for
[45 minute] sessions with [5 minutes] in between each session for debrief by the administrator(s) and data
logger(s), and to reset systems to proper test conditions. A spreadsheet was used to keep track of the
participant schedule, and included each participant’s demographic characteristics as provided by the
recruiting firm.
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application performed
well – that is, effectively, efficiently, and with satisfaction – and areas where the
application failed to meet the needs of the participants. The data from this test may serve
as a baseline for future tests with an updated version of the same EHR and/or comparison
with other EHRs provided the same tasks are used. In short, this testing serves as both a
means to record or benchmark current usability, but also to identify areas where
improvements must be made.
8
During the usability test, participants interacted with one EHR. Each participant used the
system in the same location, and was provided with the same instructions. The system
was evaluated for effectiveness, efficiency and satisfaction as defined by measures
collected and analyzed for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks
Number and types of errors Path deviations
Participant’s verbalizations (comments)
Participant’s satisfaction ratings of the system
Additional information about the various measures can be found in Section 3.9 on
Usability Metrics.
TASKS
A number of tasks were constructed that would be realistic and representative of the
kinds of activities a user might do with this EHR, including:
E-prescribe new prescription
Change prescription
Cancel prescription
Refill prescription
Request and receive medication history info
Record a medication allergy
Change a medication allergy
Display the active medication allergy list
Trigger a drug-drug interaction by entering a new medication order
Trigger a drug-allergy interaction by entering a new medication order
Tasks were selected based on their frequency of use, criticality of function, and those
that may be most troublesome for users.6 Tasks should always be constructed in light of
the study objectives.
PROCEDURES
9
Upon arrival, participants were greeted; their identity was verified and matched with a
name on the participant schedule. Participants were then assigned a participant ID.7
Each participant reviewed and signed an informed consent and release form (See
Appendix 3). A representative from the test team witnessed the participant’s signature.
To ensure that the test ran smoothly, two staff members participated in this test, the
usability administrator and the data logger. The usability testing staff conducting the
test was experienced usability practitioners with 1-2 years of experience, with a
Bachelors and Masters degree. The administrator moderated the session including
administering instructions and tasks. The administrator also monitored task times,
obtained post-task rating data, and took notes on participant comments. A second
person served as the data logger and took notes on task success, path deviations,
number and type of errors, and comments. Participants were instructed to perform the
tasks (see specific instructions below):
As quickly as possible making as few errors and deviations as possible.
Without assistance; administrators were allowed to give immaterial guidance and
clarification on tasks, but not instructions on use.
Without using a think aloud technique.
For each task, the participants were given a written copy of the task. Task timing
began once the administrator finished reading the question. The task time was
stopped once the participant indicated they had successfully completed the task.
Scoring is discussed below in Section 3.9.
Following the session, the administrator gave the participant the post-test questionnaire
(e.g., the System Usability Scale, see Appendix 6), compensated them for their time,
and thanked each individual for their participation. Participants' demographic
information, task success rate, time on task, errors, deviations, verbal responses, and
post-test questionnaire were recorded into a spreadsheet. Participants were thanked
for their time and compensated. Participants signed a receipt and acknowledgement
form (See Appendix 6) indicating that they had received the compensation.
10
TEST LOCATION
The test facility included a waiting area and a quiet testing room with a table, computer
for the participant, and recording computer for the administrator. Only the participant and
administrator were in the test room. To ensure that the environment was comfortable for
users, noise levels were kept to a minimum with the ambient temperature within a normal
range. All of the safety instruction and evacuation procedures were valid, in place, and
visible to the participants.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing
was conducted in the DoseSpot offices. For testing, the computer used was a Dell laptop running
Windows. The participants used a mouse and keyboard] when interacting with the EHRUT. The
[EHRUT] used a standard Dell laptop, with a screen size of 13 inches. The application was set up
by DoseSpot. The application itself was running on a Staging platform] using a [training / test
database] on a WiFi connection. Technically, the system performance (i.e., response time) was
representative to what actual users would experience in a field implementation. Additionally,
participants were instructed not to change any of the default system settings (such as control of font
size).
TEST FORMS AND TOOLS
During the usability test, various documents and instruments were used, including:
1. Informed Consent
2. Moderator’s Guide
3. Post-test Questionnaire
4. Incentive Receipt and Acknowledgment Form
Examples of these documents can be found in Appendices 3-6 respectively. The
Moderator’s Guide was devised so as to be able to capture required data. The
participant’s interaction with the EHRUT was captured and recorded digitally with
screen capture software running on the test machine.
11
PARTICIPANT INSTRUCTIONS
The administrator reads the following instructions aloud to each participant (also
see the full moderator’s guide in Appendix [B4]):
Thank you for participating in this study. Your input is very important. Our session today
will last about [30 minutes]. During that time you will use an instance of an electronic
health record. I will ask you to complete a few tasks using this system and answer some
questions. You should complete the tasks as quickly as possible making as few errors
as possible. Please try to complete the tasks on your own following the instructions very
closely. Please note that we are not testing you we are testing the system, therefore if
you have difficulty all this means is that something needs to be improved in the system.
I will be here in case you need specific help, but I am not able to instruct you or provide
help in how to use the application.
Overall, we are interested in how easy (or how difficult) this system is to use, what in it
would be useful to you, and how we could improve it. I did not have any involvement in
its creation, so please be honest with your opinions. All of the information that you
provide will be kept confidential and your name will not be associated with your
comments at any time. Should you feel it necessary you are able to withdraw at any
time during the testing.
Following the procedural instructions, participants were shown the EHR and as their first task, were
given time 5 minutes to explore the system and make comments. Once this task was complete, the
administrator gave the following instructions:
For each task, I will read the description to you and say “Begin.” At that point, please
perform the task and say “Done” once you believe you have successfully completed
the task. I would like to request that you not talk aloud or verbalize while you are doing
the tasks. 9 I will ask you your impressions about the task once you are done.
Participants were then given 10 tasks to complete. Tasks are listed in the moderator’s
guide in Appendix [B4].
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the Usability of
Electronic Health Records, EHRs should support a process that provides a high level of
usability for all users. The goal is for users to interact with the system effectively,
12
efficiently, and with an acceptable level of satisfaction. To this end, metrics for
effectiveness, efficiency and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of [EHRUT] by measuring participant success rates and errors
2. Efficiency of [EHRUT] by measuring the average task time and path deviations
3. Satisfaction with [EHRUT] by measuring ease of use ratings
DATA SCORING
The following Table details how tasks were scored, errors
evaluated, and the time data analyzed.
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.
The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.
Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.
Effectiveness:
Path Deviation The total number of errors/path deviations was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors.11 This should also be expressed as the mean number of failed tasks per participant.
On a qualitative level, an enumeration of errors and error types should be collected.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.
13
Efficiency:
Task Time
Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 12
Common convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and likeability of the [EHRUT] overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix 5.13
14
RESULTS Table [x]. Details of how observed data were scored.
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods specified in the
Usability Metrics section above. Participants who failed to follow session and task
instructions had their data excluded from the analyses.
The usability testing results for the EHRUT are detailed below (see Table). The results should be
seen in light of the objectives and goals outlined in Section 3.2 Study Design. The data should
yield actionable results that, if corrected, yield material, positive impact on user performance.
Measure
Task
N
Task Success
Path Deviation
Task Time
Errors
Task Ratings 5=Easy
Risk Ratings
#
Mean Total Deviations (Observed)
Mean Optimal # of Steps
Mean
Mean
1.E-prescribe New Prescription
10 100% 4 80
6 0 4.30
3
2. Change Prescription
10 100% 2 43
6 0 4.20
2
3. Cancel Prescription 10 100% 0 9
4 0 5.00
2
4. Refill Prescription 10 100% 0 33
6 0 4.20
2
5. Request and Receive Medication History Info
10 100% 0 8
3 0 4.70
1
6. Record a Medication allergy
10 100% 0 20
3 0 4.10
2
7. Change a medication allergy
10 100% 0 24
4 0 4.20
2
8. Display the active medication allergy list
10 100% 0 5
1 0 4.90
1
9. Trigger a drug-drug interaction by entering a new medication order
10 100% 2 38
5 0 4.10
2
10.Trigger a drug-allergy interaction by entering a new medication order
10 100% 1 54
5 0 4.10
2
The results from the System Usability Scale scored the subjective satisfaction with the system based on
15
performance with these tasks to be: 72.25. This number was compiled as an average out of the 10 participants
Usability scores. The scores ranged from 62.5-80.
DISCUSSION OF THE FINDINGS
EFFECTIVENESS
Based on the success rate of our participants we feel our system can be considered overall a success. All participants were able to complete the tasks provided within the optimal time. Out of the 10 tasks performed with 10 participants only 4 tasks experienced deviations and they were minor. For example, Task 1 saw that only a total of 4 deviations that occurred across 3 participants and they were still able to complete the task on time.
EFFICIENCY
In light of the findings we found that the average time between the 10 participants was within expected performance. Based on the observations of the task time and deviation data we consider the application to be quite efficient.
SATISFACTION
We saw great responses to the application and workflow, but could use improvement based on the SUS results data. While most users found the application easy to use we did see some lower scores around the integration of the application and whether or not the users would use the system. We believe that the second concern can more be attributed to whether or not the users would be required to send electronic prescriptions or use an application such as DoseSpot.
MAJOR FINDINGS
Overall we feel confident that the apps usability is strong. Some users naturally navigated through the app easier than others, but all were able to complete the 10 tasks on time with little to no errors. We did still find areas that could use improvement though and have detailed those below.
AREAS FOR IMPROVEMENT
After reviewing all of the feedback we have identified that we can improve the flow and use of the UI through a few areas. The first was a preference that as physicians the Patient information should always remain visible. That was a piece of feedback we heard from a few participants. We also found that the size of the buttons on the screen (while helpful) could also serve as a distraction. They clearly indicated what you needed to do to for example “Send a Prescription” but due to the size of the button it takes up a lot of space on the screen and can draw your eye away from other areas. This could result in someone overlooking a required field to move on. Also due to the size of some of the buttons and features users found they had to scroll more than expected to get to a needed or required section. While it did not impact the usability improvements to the need to scroll would be welcome.
16
APPENDICES
The following appendices include supplemental data for this usability test report.
Following is a list of the appendices provided:
1. Participant demographics
2. Non-Disclosure Agreement (NDA)
3. Example Moderator’s Guide
4. Orientation Script
5. Task Workflows
6. System Usability Scale Questionnaire
17
Appendix 1: PARTICIPANT DEMOGRAPHICS - TOTALS
The report should contain a breakdown of the key participant demographics. A representative list
is shown below.
Following is a high-level overview of the participants in this study.
Gender
Men [7] Women [3] Total (participants) [10]
Occupation/Role
RN/BSN [9] Physician [1]
Admin Staff [X] Total (participants) [10]
Years of Experience
Years experience [0] Facility Use of EHR
All electronic [10] Total (participants) [10]
18
Appendix 2: NON-DISCLOSURE AGREEMENT
See attachments
19
Appendix 3: EXAMPLE MODERATOR’S GUIDE
Only three tasks are presented here for illustration.
EHRUT Usability Test Moderator’s Guide
Administrator Jessica Maciel
Data Logger Christina Salvucci
Date 09/09/2019 Time 2 PM EST
Participant # 1-10
Location 18 Crawford St Needham, MA 02494
Prior to testing
Confirm schedule with Participants
Ensure EHRUT environment is running properly
Prior to each participant:
Reset application
Prior to each task:
Reset application to starting point for next task
After each participant:
End session
After all testing
Back up all data files
Orientation (5 minutes) Given to all Participants
Thank you for participating in this study. Our session today will last 45 minutes. During that
time you will take a look at an electronic health record system.
I will ask you to complete a few tasks using this system and answer some questions. We are
interested in how easy (or how difficult) this system is to use, what in it would be useful to you,
and how we could improve it. You will be asked to complete these tasks on your own trying to do
them as quickly as possible with the fewest possible errors or deviations. Do not do anything
more than asked. If you get lost or have difficulty I cannot answer help you with anything to do
with the system itself. Please save your detailed comments until the end of a task or the end of the
session as a whole when we can discuss freely.
I did not have any involvement in its creation, so please be honest with your opinions.
The product you will be using today is the DoseSpot ePrescription application. It is
currently being used by our clients today.
We are recording some screenshots of our session today. All of the information that you provide
will be kept confidential and your name will not be associated with your comments at any time.
Do you have any questions or concerns?
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• The simplicity of the look of the application.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• The sparseness of the application. It is not very cluttered, but nothing has been inputted
yet.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• The font is a bit large. But that makes the buttons easily visible.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• I’d like to be able to see more of the application. The font is very large and I can only see a
few of the application features.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• There is a large amount of unused space. It looks like it adapts to the screen size and the
information already filled in.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• I noticed that when you scrolled down the patient information disappeared.
You lose some information and features depending on where you scroll.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• Great visual management of colors
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• You lose the patient demographic information when looking at any other
pieces of the app.
Task 1: First Impressions (60 Seconds)
This is the application you will be working with. Have you heard of it? Yes X No
If so, tell me what you know about it.
Show test participant the EHRUT.
Please don’t click on anything just yet. What do you notice? What are you able to do here?
Please be specific.
• Looks easy to navigate. Simplistic.
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 165 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Minor Deviations / Cycles :: Describe below
• 1 deviation occurred
Observed Errors and Verbalizations:
Comments: Participant forgot to fill in instructions. Attempted to Save without all required fields filled in.
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 100 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 93 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 62 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Before going into the exam room and you want to review Patient’s chief complaint, history, and
vitals. Find this information.
Success:
Easily completed
:
Task Time: 67 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 70 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 39 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments: None
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 79 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Minor Deviations / Cycles :: Describe below • 1 deviation occurred
Observed Errors and Verbalizations:
Comments: User accidentally put a negative number in refills. Refills did not have to be filled in and could be
left at 0
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 24 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 2: E-prescribe New Prescription
Success:
Easily completed
:
Task Time: 97 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Check off Button Next to Prescription Select Approve and Send Button
Minor Deviations / Cycles :: Describe below • 2 deviations occurred
Observed Errors and Verbalizations:
Comments: Forgot to fill in instructions section and forgot to checkoff button next to prescription.
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 20 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 31 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 52 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 75 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Minor Deviations / Cycles :: Describe below
• 2 deviations occurred
Observed Errors and Verbalizations:
Comments: Before clicking the actions button the user tried to go to another screen. Also forgot to check off button to send prescription
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 41 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 56 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 30 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 32 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 58 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 3: Change Prescription (dosage or duration)
Success:
Easily completed
:
Task Time: 39 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits/changes Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 12 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 15 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 6 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 7 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 8 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 10 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 10 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 6 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 4: Cancel Prescription
Success:
Easily completed
:
Task Time: 7 Seconds
Optimal Path: Scroll to Pending Medications Actions Button Select Delete Select OK
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 30 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 25 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 25 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 40 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 9 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 45 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 35 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 39 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 32 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 5: Refill Prescription
Success:
Easily completed
:
Task Time: 45 Seconds
Optimal Path: Scroll to Pending Medications Actions Button to Edit Make edits to Refill Save
Prescription Check off Button Next to Prescription Select Approve and Send Button
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 7 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 6 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 5 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 3 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 6 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 13 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 9 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 10 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 11 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 6: Request and Receive Medication History Information
Success:
Easily completed
:
Task Time: 11 Seconds
Optimal Path: Select Medication History Check Off Box Show Medication History
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 20 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 15 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 21 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 15 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 29 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 15 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 15 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed :
Task Time: 27 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 20 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 7: Record Medication Allergy
Success:
Easily completed
:
Task Time: 19 Seconds
Optimal Path: Select Add/Edit Allergies Fill in Allergy Info Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 8 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 10 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 5 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 28 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 45 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 37 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 40 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 28 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 8: Change a Medication Allergy
Success:
Easily completed
:
Task Time: 31 Seconds
Optimal Path: Select Add/Edit Allergies Select Edit Make Changes Save
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 6 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 2 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 2 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 16 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 4 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 9: Display the Active Medication Allergy List
Success:
Easily completed
:
Task Time: 5 Seconds
Optimal Path: Expand Patient Demographic Section
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 60 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Minor Deviations / Cycles :: Describe below • 2 deviations occurred
Observed Errors and Verbalizations:
Comments: The user forgot to check off button to send prescription. Prescription initially did not trigger a drug-drug interaction
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 19 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 18 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 67 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 35 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 30 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 36 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 39 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 30 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 10: Using CPOE, Trigger a Drug-Drug Interaction by Entering a New Medication Order
Success:
Easily completed
:
Task Time: 46 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Drug Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 60 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed :
Task Time: 90 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Minor Deviations / Cycles :: Describe below
• 1 deviation occurred
Observed Errors and Verbalizations:
Comments: Prescription initially did not trigger a drug-allergy interaction
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 60 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 38 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 25 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 54 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 75 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 3
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Administrator / Notetaker Comments: The user was able to complete this task, but was unsure of a lot of
steps and took longer.
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 50 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 37 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 5
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Task 11: Using CPOE, Trigger a Drug-Allergy Interaction by Entering a New Medication
Order
Success:
Easily completed
:
Task Time: 48 Seconds
Optimal Path: Select Add Prescription Fill in all required fields Select Save Prescription View
Pending Prescriptions Section Confirm Drug/Allergy Interaction Box
Correct
Observed Errors and Verbalizations:
Comments:
Rating:
Overall, this task was: 4
Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use
this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use
this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use
this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use this
system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE
In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at
http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
1. I think that I would like to use
this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Strongly Strongly
disagree agree
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5
X
1 2 3 4 5