sandra martinez, ph.d. apply texas team. performed spring 2010 by garrett stettler as his...
TRANSCRIPT
Usability Study of theApply TexasApplication
including Interviews with High School Counselors
Sandra Martinez, Ph.D.Apply Texas Team
Performed Spring 2010 by Garrett Stettler as his Master’s project for UT School of Information
Garrett observed, interviewed, and videotaped 15 participants
He interviewed 5 high school counselors
Usability Study
Getting a third party to perform a usability study is more effective than using developers because:
◦ They have less personal involvement in the website, which produces less emotional response.
◦ They are not biased by over-familiarity with the website
◦ They have different expectations than developers
Usability Study – the 3rd party
They evaluate a product by testing it on actual users.
They discover errors and areas for improvement by observing people using the product.
They measure efficiency, accuracy, recall, and emotional responses of test subjects.
How are usability studies helpful?
The Apply Texas Committee, in authorizing the study, charged the Apply Texas team with four goals:
◦Fix simple errors
◦Fix big, urgent problems
◦Create list of usability issues for the Apply Texas committee to prioritize
◦Create list of suggested changes for the Apply Texas committee
How the study will be used
8 postsecondary-bound Texas high school students
2 postsecondary-bound high school students from other states
2 current college students (transfer)
1 prospective graduate student
1 “non-traditional” adult returning to school
1 pilot participant
Participants
1) Navigation
2) Copy process
3) Help
4) Confusion and frustration
5) Accuracy and completion
Areas of research for Apply Texas usability study
Tasks by Group Group A (“new”
users)
1. Create account, input profile information
2. Start and complete a new application
3. Enter essay information
4. Enter scholarship information
5. Logout
Group B (“returning” users)
1. Retrieve forgotten password
2. View/edit a saved profile
3. View/edit a saved application
4. View/edit saved essays
5. Copy an existing application
Description of critical concerns:
Critical data may be lost, or
The user may not be able to complete the task, or
The user may not want to continue using the application
Critical concerns (None found)
Description of major concerns:
Users can accomplish the task but only with considerable frustration and/or unnecessary steps
Non-critical data may be lost
The user will have great difficulty in circumventing the problem
Users can overcome the problem only after they have been shown how to perform the task
Major concerns (2 found)
Error handling
◦ Error messages are grouped at the top of the page.
◦ Errors are not highlighted within the page itself
◦ Participants tended to skim over error messages and used a “trial-and-error” approach to correcting the error
Major concerns (1 of 2)
Error handling recommendations
◦ Errors should be highlighted at the top of the page and at the location in the page where the error has occurred.
◦ If possible, identify the error as the user is typing
Major concerns (1 of 2)
Home / My Account / Logout
◦ Navigating to the main account page was confusing
◦ Attempting to log out could be confusing
Major concerns (2 of 2)
Major concerns (2 of 2)Current view of top toolbar
Currently links to logout page
Currently links to logout page
Currently links to application listing page
Currently links to help page
Major concerns (2 of 2)
For new cycle, links to logout page
For new cycle, links to application listing page
Links to application listing page (no change)
For new cycle, name changed to Logoff from Home/Logoff
Changes for 2011-2012 cycle
Description of moderate concerns:
The user will be able to complete the task in most cases but will undertake some moderate effort in getting around the problem
The user may need to investigate several links or pathways through the system to determine which option will allow them to accomplish the intended task
Users will most likely remember how to perform the task on subsequent encounters with the system
Moderate concerns (5 found)
Helper text and other indicators
◦ Every participant failed to notice help text at some point during the test.
Moderate concerns (1 of 5)
Helper Text
Buttons or important information appear ‘below the fold’
Some participants had difficulty completing a task if the continue button required scrolling
Moderate concerns (2 of 5)
Confusing terminology on essay page
Participants were unsure about clicking ‘Save’ if they had not entered an essay.
Some participants were unsure about the difference between save and submit.
Moderate concerns (3 of 5)
Difficulty using Perform High School/College Search function
◦ Over one-third of participants clicked the “perform search” button before entering any text
◦ Two participants failed to select the radio button
◦ Several participants did not initially use the search feature
Moderate concerns (4 of 5)
The Apply Texas team recommends reworking this page as follows:
1. Make the “perform high school name search” button more prominent
2. Ask applicants to enter the name of the school in a new window when the button is clicked
3. When the applicant chooses a radio button, automatically close the window and fill in information on the application
Moderate concerns (4 or 5)
Top Menu and Quick Links
◦ Some participants were confused about the Menu bar and the Quick Links menu, and how they differed
◦ Participants generally did not use the navigation bar at the top for moving between Admissions/Scholarships/Essays/Submit
Moderate concerns (5 of 5)
The Apply Texas team recommends reworking the navigation and links:
◦ Make the navigation links as a side navbar
◦ Add quick links to the top menu bar with a “mouseover” dropdown.
Moderate concerns (5 of 5)
Description of minor concerns:
The concern is an irritant for the user
There is a cosmetic problem
There is a typographical error
Minor concerns ( 4 found)
Confusing/inconsistent terminology
◦ 1) Applicants were unsure what was being asked for “Mini or short” semester
Minor concerns (1 of 4)
Confusing/inconsistent terminology◦ 2) Participants were unsure about the difference
between ‘community and volunteer service’ and ‘extracurricular activities’
◦ 3) Under extracurricular activities, participants were unclear what ‘activity level’ meant
◦ 4) One participant was unclear whether ‘household size’ included only children, or all persons in household.
Minor concerns (1 of 4)
Minor concerns (1 of 4)For mini/short semester, the following paragraph was added for the new cycle as clarification:
Constraining employment form
◦ Two participants felt the employment form would be difficult to use, especially for applicants who have recurring summer jobs.
Minor concerns (2 of 4)
Custom Questions◦ Many participants found the custom question
page to be irritating, particularly for institutions with many custom questions.
◦ The page did not cause confusion, just user irritation as it requires scrolling down the page for each question in order to click the ‘save, acknowledge question button.’
Minor concerns (3 of 4)
Suggestion: Schools with many custom questions may want to consider moving some of their questions to other modules instead of grouping them all on the custom question page.
Minor concerns – 3 of 4
Yes/No/Reset Answer
◦ Several participants were confused by the Yes/No/Reset Answer response.
Minor concerns – 4 of 4
One suggestion would be to change the button labels from Reset Answer – Not Applicable to No Answer – Not Applicable
Overall, Apply Texas is a fairly usable product, with no critical concerns
Many improvements to reduce frustration could be done fairly easily
Some improvements will require reworking certain modules or processes
And now, let’s hear what the high school counselors had to say --
Usability Study
Suggestions for improving Apply Texas:
◦ Give applicants a warning that if they select to send a paper resume, schools may not process their application until the paper resume is received.
◦ Allow more space in online resumes, or provide an upload option into Apply Texas
◦ Provide another mechanism for retrieval of lost usernames, passwords, and application id numbers.
◦ Provide an online application deadline table equivalent to the one available in the paper copy of the Apply Texas application.
High School Counselor interviews
Suggestions for improving the High School Counselor System
◦ Different navigation options
◦ Different documentation on what data is available
◦ The ability to separate four year and two year information
High School Counselor interviews
The Apply Texas team would like to thank Garrett Stettler for his time, effort, and excellent results.
We would also like to thank the Apply Texas committee for authorizing this study.
We are confident that implementing his results will make the application easier and less frustrating for our applicants.
Usability Study