milestone five prototype...
TRANSCRIPT
Jeff BazerHCI 598X - Capstone Project
Iowa State UniversityFall 2015
prototype testingMilestone Five
SUMMARY
The current AgDirect Quote center is a tool that lets dealers run quotes for customers,
compare those quotes and be able to print, email and save them to be given to the customer
or used at a later time. The current tool is outdated, runs slow and has a poor user experience.
Issues with the current tool include data being lost after timing out, no ability to compare
quotes in the quote center and no easy way to email quotes to customers.
There are three types of users that will interact with the AgDirect Quote Center. The first type of
users are internal sales representatives. They are on the phones most of the day talking to
sales people at dealerships. The second type of users are sales people at dealerships. They
spend a couple hours a day in front of the computer and are typically outside of the office on
the sales lot talking and interacting with customers. The last type of users are external users
visiting the public facing website which can be divided into an older generation, Generation X
and Baby Boomers, and a younger generation, Generation Y.
Users who visit the AgDirect Quote Center should be able to complete the following three
tasks. The first task is to be able to run a basic or advanced finance or lease quote. The user
should be able to select between a finance or lease quote and be able to add in many other
options. The second task is to be able to compare that quote to multiple other quotes and
compare the differences in an easy to view side by side format. The third task is the quote(s)
should be able to printed, emailed and saved.
EVALUATION METHODS
Ferre, Juristo and Moreno (2004) define usability as “The extent to which a product can be
used by specified users to achieve specified goals with effectiveness, efficiency, and
satisfaction in a specified context os use.” What is usability testing? Usability testing is defined
�
AGDIRECT QUOTE CENTER �2
as requiring five characteristics: that the goal is to improve a product’s usability, that the
participants represent real users, that they do real tasks, that the testers observe and record
the participants and that they then analyze the data and recommend changes to fix problems
(Dicks 2002). Why should we test? Design decisions are informed by data gathered from
representative users to expose design issues so they can be remedied, thus minimizing or
eliminating frustration for users (Rubin and Chisel 2008). The evaluation method that was
chosen for this was a formative evaluation including doing a usability study and a
questionnaire.
Rubin and Chisel (2008) state that doing formative evaluation and early analysis with research
is important. This point in time is when critical design decisions set the stage for what will
follow and if certain elements are wrong it can cause many issues down the road for the whole
project. Using this and the usability test with a feedback form and questionnaire will allow more
useful data for the prototype of the AgDirect Quote Center.
An important aspect stated by Katner (1994) is “usability testing is in the details.” She states
that “details overlooked during preparation for a usability test become gaping holes in the data
collected during the actual test (Katner, 1994). This is why it is critical that the usability test is
set up correctly and efficiently so that the most accurate data is collected.
As research suggests, approximately 80% of the usability problems identified are found using
four to five users (Virzi, 1992). Virzi (1992) also states that additional subjects are less and less
likely to reveal new information and by running fewer subjects in each iteration it is a valuable
time and money saving approach. For this usability test four users will be tested - two internal
sales representatives and two AgDirect dealer sales people. Rubin and Chisel (2008) suggest
screening potential users beforehand to make sure they are representative of the correct user
profile for the usability test. The AgDirect internal sales representatives were selected after
�
AGDIRECT QUOTE CENTER �3
meeting with their leader to select individuals that are representative of the usability test
scenarios and then asking them if they would be willing to help out. Rubin and Chisel (2008)
state that a good way to find existing customers is through sales representatives. Each state
has an AgDirect Territory Manager and they were an excellent resource to sit down with and
select two dealerships that met the usability test scenarios. An informal phone screen was then
conducted and two sales people at two different dealerships agreed to volunteer.
Compensation was provided in the form of lunch for each dealership.
The two internal sales representatives that were selected were younger female users who are
computer savvy and create, edit and compare a minimal amount of quotes (0-5) in a single day.
They will be referred to as “Tester 1 and Tester 2.” The two AgDirect dealer sales people
selected include one who works for a John Deere dealership and one who works for a Case IH
dealership. They are both not very computer savvy and each creates, edits and compares
many quotes (10-15) for customers in a single day. They will be referred to as “Tester 3 and
Tester 4."
As soon as the participant enter the usability session, Rubin and Chisel (2008) mention starting
with an orientation script. Parts of this script include introductions, explaining why the
participant is there, what is expected of the participant, assuring the participant that they are
not being tested and that it is ok to ask for questions at any time. When each participant enters
the usability session, they will be given a questionnaire to fill out which includes their name, job
title along with these three questions: 1. On average, how many new quotes to do you create in
a day?, 2. On average, how any quotes do you edit in a day? and 3. On average, how many
times do you compare quotes in a day? The form then includes information at the bottom
stating that this not a test of your abilities but our ability to make something that can be useful
to you - there are no right or wrong answers. See Appendix A, figure 1 for reference.
�
AGDIRECT QUOTE CENTER �4
Each participant will go through two different scenarios with various tasks for each scenario.
They will be given a sheet that includes both scenarios to work on at their own pace. The tasks
that are to be completed for the first scenario include: participants will be instructed to
calculate the payment for a new quote with the following information: Quote Description: 2013
Planter 5 year, Amount: 215,000.00, Rate: 4.15, Term: 5 years, and Frequency: Annual. The
participant will then add the quote for comparison and run a new quote with the same
information but for a seven year term and also add it for comparison. The last step for the first
scenario is to edit the payment frequency for both quotes to be “Semi-Annual."
The tasks that are to be completed for the second scenario include adding a quote to
an existing saved quote. The participants will be instructed to open the quote “Jack Jones
2012 Tractor.” They will then add a new quote for comparison with the following
information: Quote Description: 2013 Tractor 5 year, Amount: 150,000.00, Rate: 4.15, Term: 7
years, and Frequency: Annual. The final step for this scenario is to duplicate that quote and
remove the trade-in amount and the trade-in payoff and save the changes.
This form can be found in Appendix A, figure 2 for reference.
Rubin and Chisel (2008) define a post-test questionnaire as gathering information from the
participants after the usability test in order to understand more the strengths and weaknesses
of the tool. Accordingly, a post-test questionnaire was distributed right after each session. It
includes seven questions - three questions that ask for a rating from one to ten and why and
four open ended questions. In retrospect, this would have been better done with a five point
scale as Rubin and Chisel (2008) point out.
�
AGDIRECT QUOTE CENTER �5
The questions are included below:
1. How would you rate the navigation of the new Quote Calculator Tool? (1-10, 1 = very
difficult and 10=very easy) Why?
2. How would you rate the design (screen layout and colors) of the new ADO website (1-10, 1
= hideous and 10=looks great) Why?
3. How would you rate your experience entering and editing quotes through the new
quoting told design? (1-10, 1 = very poor and 10 = perfection) Why?
4. What did you like about creating or editing quotes in the new ADO website?
5. What did you not like about creating or editing quotes in the new ADO website?
6. What additional features would you like to see when creating or editing quotes?
7. Additional Comments?
This questionnaire can be found in Appendix A, figure 3 for reference.
The testing environment for the internal AgDirect sales representatives will take place using a
simple single-room setup. Rubin and Chisel (2008) define this as the most basic type of testing
setup, both in terms of resources and the amount of space required. It will take place in a
conference room in the office and each participant will be brought in separately to run the
usability test. An advantage to using this setup is that the moderator and observers have an
excellence sense of what is going on with the participant but a disadvantage is that
the observers behavior can affect the behavior of the participant.
The testing environment for the AgDirect dealer sales people will take place at each of their
offices. Items that will be brought to each location include scripts, forms and questionnaires,
laptops with prototype on it, projector to project the participants screens for observers to take
notes, and pens to write with. This list is based off of what Rubin and Chisel (2008) suggest to
bring.
�
AGDIRECT QUOTE CENTER �6
See Appendix B for photos of each setup.
RESULTS
The results from the usability sessions ended up with interesting themes. There were a total of
four participants, two internal AgDirect sales representatives and two AgDirect Dealer sales
people. Overarching themes from the participant and observer forms that were positive include
easy flow and navigation, the payment box with illustration of payment and interest and
user friendly design.
Below are quotes from participants that did the usability study that are positive. These quotes
varied between all four participants and were collected from the post-test questionnaire.
- "Easy navigation with user friendly design. Easily can add/compare quotes when needed."
- "The payment box on the right side makes it easy to see and the ability to have the principal/
interest is very handy.
- "Like the illustration of principal and interest."
- "Smooth flow. Love the large payment."
The overarching theme that shows where improvement is needed is in editing and
updating the quotes and duplicating the quotes. Each user had some kind of issue with editing
and updating a quote.
Below are quotes from participants that did the usability study that show room for
improvement. These quotes varied between all four participants and were collected from the
post-test questionnaire.
�
AGDIRECT QUOTE CENTER �7
- "Initially editing the quote took a minute to find "edit" button. Seamless with rest of the
process though."
- "Scrolling to the bottom of the compared quotes to find "edit" was the only issue I had."
- "I needed some explanation on how to update quotes. Most of it was intuitive if your use to
the previous version but it'll take some training for salesmen to get up to speed."
- "Not sure about the compare and edit features."
It is worthy to note that while there are many positive quotes and feedback, the most helpful
feedback is the needs improvement feedback. Without this feedback, the AgDirect Quote
Center will not be able to mature and gain a better user experience from this usability study.
Even with each participant having the issues with editing the quotes and duplicating the
quotes, they still rated each of the three questions fairly high. See figure 1 for details on each
question and rating. While there were issues, each participant did not think that those issues
were bad enough to majorly affect the design.
�
AGDIRECT QUOTE CENTER �8
Figure 1: Post-Questionnaire Survey Responses
See Appendix C for feedback forms from each participant.
DISCUSSION
Overall there was very positive feedback. All the participants had positive feedback about the
flow of the AgDirect Quote Center and how each page was laid out. They also thought the
�
AGDIRECT QUOTE CENTER �9
quote section with the large payment along with the graph and the payment and interest
beneath it was extremely useful.
Items that have room for improvement include the edit at the bottom of the compared quotes.
Participants had a hard time finding that - partially due to screen size and participants not
scrolling to the very bottom - but good feedback as it has shown that the link needs to be
moved. A possible fix for this is to include the edit link beneath the description instead of at the
bottom. Participants also had a hard time finding duplicate at the bottom - one participant
didn’t even notice the duplicate link. A possible fix for this is to also move it to the top next to
edit so users will see that right away also. Having these links separate from the delete and view
amortization will also reduce the users cognitive load since they will not have to look through
four links at the bottom to find what they are looking for.
The site seems to be on the right track. It just needs to have some cleanup of the items listed
above. One other comment that users mentioned was that when they had more than a couple
quotes added to the compared section that it was difficult to see the key information which
includes interest rate, term and payment. These fields and data will be bolded to stand out to
the user more.
Surprising findings that come out of the usability sessions were items that hadn’t been
originally thought about. One of those items was one participant asked if they could rearrange
the quote order in the comparison. It wasn’t feasible in the prototype, but it wasn’t something
that was originally thought of so that will be a good addition. Another surprising finding is that
users had a difficult time finding the edit link at the bottom of the compared quotes. As this
was being designed, the thought process was that a user would look through the quote and
once they reach the bottom, they would see edit and click it, but the testing shows that is not
�
AGDIRECT QUOTE CENTER �10
the case. As discussed above, a possible fix to this is move it up beneath quote description so
it is easier for a user to see and it is not mixed up with the other links at the bottom.
One issue that may cause implications is if a user opens a collection of quotes, the compared
quotes could be below the fold on smaller screen resolutions and users may see a blank quote
at the top and not realize they need to scroll to see their saved quotes. This will need to be
revisited to see if there is a way to make it noticeable that there are quotes towards the bottom.
One test that we did not test yet due to the limitations of the prototype software is that if there
are more quotes added to the compare section than will fit on the screen - how to display that.
The current design has large arrows showing up on each side like an image carousel to let the
user know that they can scroll left and right to see more quotes.
The process of doing usability testing proved to be extremely useful. Without doing this
process, there were design flaws that would have been pushed out to users that could have
resulted in calls with questions and feedback. There would have had to been more time and
money spent on fixing those design flaws and having to push it out to users again - by doing
this usability test it eliminated all of that. The usability test also showed some ideas that had
not been thought of before and will be incorporated into the design to help future users of the
site. The test also help with feedback knowing that the site design was moving in the
right direction and that it was not off-base.
�
AGDIRECT QUOTE CENTER �11
References
Rubin, Jeffrey and Chisnell, Dana (2008) Handbook of Usability Testing: How to Plan, Design,
and Conduct Effective Tests. Indianapolis, Indiana: Wiley Publishing, Inc.
Virzi, Robert A. (1992). Refining the Test Phase of Usability Evaluation: How Many Subjects Is
Enough? Human Factors: The Journal of the Human Factors and Ergonomics Society,
34 (4). Retrieved from
http://hfs.sagepub.com.proxy.lib.iastate.edu/content/34/4/457.short
Dicks, R. Stanley. (2002). Mis-usability: on the uses and misuses of usability testing. ACM: New
York, NY Retrieved from http://dl.acm.org.proxy.lib.iastate.edu/citation.cfm?id=584960
Brown, Kenneth and Gerhardt, Megan. (2002) Formative Evaluation: An Integrative Practice
Model and Case Study. Personnel Psychology, 55 (4) Retrieved from http://
onlinelibrary.wiley.com.proxy.lib.iastate.edu/doi/10.1111/j.1744-6570.2002.tb00137.x/
abstract
Katner, Laurie. (1994) Techniques for managing a usability test. Professional Communication,
37 (3) Retrieved from http://ieeexplore.ieee.org.proxy.lib.iastate.edu/xpls/abs_all.jsp?
arnumber=317479&tag=1
Ferre, Xavier, Juristo, Natalia and Moreno, Ana (2004) Improving Software Engineering Practice
with HCI Aspects. Software Engineering Research and Applications. Springer
Berling Heidelberg.
�
AGDIRECT QUOTE CENTER �12
APPENDIX A
Figure 1: Usability Test Pre-Questionnaire
�
AGDIRECT QUOTE CENTER �13
Figure 2: Usability Test Scenarios
�
AGDIRECT QUOTE CENTER �14
Figure 3: Usability Test Post-Questionnaire
�
AGDIRECT QUOTE CENTER �15
APPENDIX B
�
AGDIRECT QUOTE CENTER �16
�
AGDIRECT QUOTE CENTER �17
APPENDIX C
Figure 4: Tester 1 Pre-Questionnaire
�
AGDIRECT QUOTE CENTER �18
Figure 5: Tester 1 Post-Questionnaire
�
AGDIRECT QUOTE CENTER �19
Figure 6: Tester 2 Pre-Questionnaire
�
AGDIRECT QUOTE CENTER �20
Figure 7: Tester 2 Post-Questionnaire
�
AGDIRECT QUOTE CENTER �21
Figure 8: Tester 3 Pre-Questionnaire
�
AGDIRECT QUOTE CENTER �22
Figure 9: Tester 3 Post-Questionnaire
�
AGDIRECT QUOTE CENTER �23
Figure 10: Tester 4 Pre-Questionnaire
�
AGDIRECT QUOTE CENTER �24
Figure 11: Tester 4 Post-Questionnaire
�
AGDIRECT QUOTE CENTER �25