umbc blackboard usability testing spring 2002. overview usability testing basics umbc blackboard...
TRANSCRIPT
UMBCUMBCBlackboard Usability Blackboard Usability
TestingTestingSpring 2002Spring 2002
OverviewOverview
Usability Testing BasicsUsability Testing Basics UMBC Blackboard Usability TestingUMBC Blackboard Usability Testing Highlights VideoHighlights Video Sneak Peek: Blackboard Version 6Sneak Peek: Blackboard Version 6 Next Web Usability Study: myUMBCNext Web Usability Study: myUMBC
Usability DefinedUsability Defined
““Making whatever you’re working on easier Making whatever you’re working on easier to use for who ever is going to use it.”to use for who ever is going to use it.”
““An element of design, focusing on ‘can this An element of design, focusing on ‘can this be used when it’s done?’ rather than just be used when it’s done?’ rather than just making it look good.”making it look good.”
Steve Krug, 2001 WebTalkGuys interviewSteve Krug, 2001 WebTalkGuys interview
Must ReadMust Read
One of Amazon’s top One of Amazon’s top 200 books.200 books.
Companion site Companion site www.sensible.comwww.sensible.com Sample chapterSample chapter Sample scriptSample script Workshops & updatesWorkshops & updates
Usability BasicsUsability Basics
Instead of imagining what users want, you observe, Instead of imagining what users want, you observe, interview them and prototype their needs.interview them and prototype their needs.
Develop content, design and programming strategy by Develop content, design and programming strategy by reconciling observed user needs with sponsor’s desired reconciling observed user needs with sponsor’s desired objectives.objectives.
Test user tasks and expectations with low-cost Test user tasks and expectations with low-cost prototypes (e.g., paper, PhotoShop) prototypes (e.g., paper, PhotoShop)
User’s don’t make mistakes; websites doUser’s don’t make mistakes; websites do
Pros & ConsPros & Cons
The alternative of using designers and The alternative of using designers and programmers as “proof of concept” tools is programmers as “proof of concept” tools is expensive and demoralizing.expensive and demoralizing.
May seem to add time (and cost) to a production May seem to add time (and cost) to a production schedule, but constantly revising is expensive, schedule, but constantly revising is expensive, too.too.
BlackboardBlackboard
Blackboard @ UMBCBlackboard @ UMBC
UMBCUMBC Faculty = 616 full-time, 318 part-timeFaculty = 616 full-time, 318 part-time Students = 11,237Students = 11,237 Staff = 415Staff = 415
BlackboardBlackboard Courses & organizations = 699 (238 Sp02)Courses & organizations = 699 (238 Sp02) Number of users = 12,141Number of users = 12,141 Number of Instructors & managers = 610Number of Instructors & managers = 610 Average hits per day = 35,585Average hits per day = 35,585
User/Task AnalysisUser/Task Analysis
QuestionnaireQuestionnaire
User : Instructor 1User : Instructor 1
Interview Date: Interview Date: 2/18/022/18/02
Name: RobinName: Robin Sex: FemaleSex: Female Role: InstructorRole: Instructor
Typical Usage & Findings Typical Usage & Findings
Post syllabus, Post syllabus, handouts, handouts, announcementsannouncements
Collect and return Collect and return papers via Digital papers via Digital DropBoxDropBox
Discussion boardDiscussion board
Would like to be able to Would like to be able to do bulk deletesdo bulk deletes
Likes email discussions, Likes email discussions, students don’t use students don’t use Blackboard forumsBlackboard forums
Grade book inflexibleGrade book inflexible Likes Word editing and Likes Word editing and
feels like she could feels like she could duplicate many BB duplicate many BB features with other tools.features with other tools.
Usage Findings
User : Instructor 2User : Instructor 2
Interview Date: Interview Date: 2/18/022/18/02
Name: KristieName: Kristie Sex: FemaleSex: Female Role: InstructorRole: Instructor
Typical Usage & FindingsTypical Usage & Findings
Posting to and usage of Posting to and usage of course syllabuscourse syllabus
Posting to and usage of Posting to and usage of course schedulescourse schedules
Posting of assignmentsPosting of assignments Use of Grade BookUse of Grade Book Accessing documents in Accessing documents in
Drop BoxDrop Box Usage of discussion Usage of discussion
boardboard
Frequently uses 2 or more Frequently uses 2 or more browser windows along with browser windows along with other apps (such as Word)other apps (such as Word)
Makes heavy use of her own Makes heavy use of her own external siteexternal site More reliableMore reliable Easier to back upEasier to back up Always accessibleAlways accessible
Uses BB to post copyrighted Uses BB to post copyrighted materialmaterial
Uses MS Word for spell Uses MS Word for spell checkingchecking
Wants fewer buttonsWants fewer buttons Sees BB as an enhancement Sees BB as an enhancement
to course.to course.
Usage Findings
User : Instructor 3User : Instructor 3
Interview Date: Interview Date: 2/18/022/18/02
Name: KarinName: Karin Sex: FemaleSex: Female Role: InstructorRole: Instructor
Typical Usage & FindingsTypical Usage & Findings
Posting of assignmentsPosting of assignments Retrieval of completed Retrieval of completed
workwork Assistance in assigning Assistance in assigning
and collecting workand collecting work Support of classroom Support of classroom
instructioninstruction CommunicationCommunication Homework assignment Homework assignment
hand-inhand-in
Wants:Wants: More flexible search More flexible search Check boxes with Check boxes with
Digital Drop Box to Digital Drop Box to indicate document indicate document statusstatus
Replication from Replication from computer-to-computercomputer-to-computer
Batch processingBatch processing Additional levels of Additional levels of
admin for TA’sadmin for TA’s
Usage Findings
User : Student 1User : Student 1
Interview Date: Interview Date: 2/12/022/12/02
Name: RogerName: Roger Sex: MaleSex: Male Role: StudentRole: Student
Typical Usage and FindingsTypical Usage and Findings
Checking:Checking: AssignmentsAssignments SchedulesSchedules AnnouncementsAnnouncements
Discussion board for Discussion board for assignmentsassignments
Downloads course Downloads course docs.docs.
Checks gradesChecks grades
Interface didn’t match Interface didn’t match mental modelmental model
Primary functions:Primary functions: Discussion boardsDiscussion boards Digital drop boxDigital drop box
Not familiar with many Not familiar with many featuresfeatures
Uses Word for spell Uses Word for spell checkingchecking
Backs up replies in Word.Backs up replies in Word. Wants to save docs,not Wants to save docs,not
look at them in browser.look at them in browser. Easy to use, convenientEasy to use, convenient
Usage Findings
User : Student 2User : Student 2
Interview Date: Interview Date: 2/13/022/13/02
Name: ChrisName: Chris Sex: MaleSex: Male Role: StudentRole: Student
Typical Usage/FindingsTypical Usage/Findings
Mandatory useMandatory use CommunicationCommunication Drop BoxDrop Box Retrieving and Retrieving and
submitting submitting assignmentsassignments
Likes interface: “User Likes interface: “User friendly”friendly”
Easy to navigateEasy to navigate Quick to learnQuick to learn Got several errorsGot several errors
Usage Findings
User : Student 3User : Student 3
Interview Date: Interview Date: 2/13/022/13/02
Name: AnitaName: Anita Sex: FemaleSex: Female Role: StudentRole: Student
Typical Usage/FindingsTypical Usage/Findings
Checks course Checks course materialsmaterials
Completes homeworkCompletes homework Posts questions to Posts questions to
discussion board discussion board about homeworkabout homework
Encountered server Encountered server errorserrors
Confusion about Confusion about multiple Discussion multiple Discussion BoardsBoards
Clicks on all buttons Clicks on all buttons to find what she’s to find what she’s looking forlooking for
Does not check site Does not check site frequently/dailyfrequently/daily
Usage Findings
User : Student 4User : Student 4
Interview Date: Interview Date: 2/13/022/13/02
Name: SaraName: Sara Sex: FemaleSex: Female Role: StudentRole: Student
Typical usage/findingsTypical usage/findings
Checks Checks announcements and announcements and course materialcourse material
Do homeworkDo homework Use discussion boardUse discussion board Maximum use: 5X per Maximum use: 5X per
weekweek EmailEmail
Works from own Works from own computercomputer
Uses multiple Uses multiple windows to avoid windows to avoid going back and forthgoing back and forth
Confused by multiple Confused by multiple discussion boardsdiscussion boards
Missed Missed announcements announcements burried in lots of text.burried in lots of text.
Usage Findings
User/Task MatrixUser/Task Matrix
Typical EnvironmentsTypical Environments
University public computer lab workstation University public computer lab workstation (Students)(Students)
Home computer (students & instructors)Home computer (students & instructors) ModemModem CableCable
Office computer (instructors)Office computer (instructors)
User GoalsUser Goals
Earn a good gradeEarn a good grade Learn about course Learn about course
topictopic Complete Complete
requirementrequirement Complete courseComplete course Research topicsResearch topics
Use the Web to enhance Use the Web to enhance coursecourse
Enable learningEnable learning Design coursesDesign courses Facilitate administrationFacilitate administration Foster interactionFoster interaction Provide supplementary Provide supplementary
researchresearch Manage large classes with Manage large classes with
multiple sectionsmultiple sections Create online archiveCreate online archive Organize materialsOrganize materials Personal enrichmentPersonal enrichment
Students Instructors
Task FlowTask Flow
User TasksUser Tasks
View course workView course work Read notesRead notes Ask questionsAsk questions Share informationShare information Turn in assignmentsTurn in assignments Do researchDo research View SchedulesView Schedules View syllabusView syllabus Find required readingsFind required readings Get helpGet help Download documentsDownload documents
Deliver course workDeliver course work Track student progressTrack student progress Monitor participationMonitor participation Track attendanceTrack attendance Track gradesTrack grades Archive course infoArchive course info Communicate with studentsCommunicate with students Conduct discussions and lecturesConduct discussions and lectures Review workReview work Grade workGrade work Edit assignments, course work, Edit assignments, course work,
schedules, etc.schedules, etc. Make announcementsMake announcements Post schedules, syllabus, etc.Post schedules, syllabus, etc. Manage accessManage access
Students Instructors
User TestsUser Tests
MethodologyMethodology
Conducted at UMBCConducted at UMBC Volunteer participantsVolunteer participants
5 students5 students 4 instructors4 instructors Varying experienceVarying experience
Test team:Test team: Moderator- lead subject through tasksModerator- lead subject through tasks Observer- noted subject responses/actionsObserver- noted subject responses/actions
Protocol:Protocol: Subjects in front of computer, asked to complete tasksSubjects in front of computer, asked to complete tasks Subjects encouraged to verbalize thoughtsSubjects encouraged to verbalize thoughts Responses recorded by ObserverResponses recorded by Observer
Usability Test ScriptsUsability Test Scripts
SummarySummary
Easy to learn and useEasy to learn and use Consistent navigation helpsConsistent navigation helps Operational scalability issuesOperational scalability issues Features for “power users”Features for “power users” EfficienciesEfficiencies Menu wording/functionsMenu wording/functions
Summary: Task EfficienciesSummary: Task Efficiencies
Faculty spend more time than students, Faculty spend more time than students, complained about repetitive and boring complained about repetitive and boring tasks:tasks: Enrolling more than one student in course or Enrolling more than one student in course or
groupgroup Managing documents via the Digital Drop BoxManaging documents via the Digital Drop Box Managing own contentManaging own content Inflexibility of gradebookInflexibility of gradebook
Summary: Interface labelingSummary: Interface labeling
Students main issue: misunderstanding Students main issue: misunderstanding important menus and functions:important menus and functions: ““Tools” a vague termTools” a vague term ““Digital Drop Box” and Grades sections use Digital Drop Box” and Grades sections use
unclearunclear Send vs. Drop metaphor problematicSend vs. Drop metaphor problematic Confusion over Assessment & Pool managerConfusion over Assessment & Pool manager Editable navigationEditable navigation
SubjectsSubjects
5 students5 students Graduate, PsychologyGraduate, Psychology Undergraduate, Computer ScienceUndergraduate, Computer Science Undergraduate, AnthropologyUndergraduate, Anthropology Graduate Student, Information SystemsGraduate Student, Information Systems Undergraduate, Computer ScienceUndergraduate, Computer Science
4 Instructors4 Instructors Geography, expertGeography, expert Modern Languages, advanced beginnerModern Languages, advanced beginner English, advanced beginnerEnglish, advanced beginner Chemical Engineering, beginnerChemical Engineering, beginner
User Testing ResultsUser Testing ResultsStudentsStudents
Digital Drop BoxDigital Drop Box Observation:Observation:
Difficult for studentsDifficult for students Confused with “submit” and Confused with “submit” and
“send”“send” Confirmation/warning Confirmation/warning
ignored.ignored.
Recommendation:Recommendation: Language should be Language should be
reconsidered to clear reconsidered to clear up confusionup confusion
Submit should send Submit should send with first clickwith first clickRating: UNUSABLE
Tools LanguageTools Language Observation:Observation:
Confusion over “Tools” Confusion over “Tools” buttonbutton
Frequently ignoredFrequently ignored When used, not able to When used, not able to
locate informationlocate information
Recommendation:Recommendation: Use more descriptive Use more descriptive
language in labelslanguage in labels Allow instructors to move Allow instructors to move
frequently accessed frequently accessed functionalityfunctionality
Create more meaningful Create more meaningful name based on user insight.name based on user insight.Rating: SEVERE
Discussion BoardDiscussion Board Observation:Observation:
Confusing to studentsConfusing to students Unclear about role of Unclear about role of
discussion boarddiscussion board Overwhelmed with active Overwhelmed with active
boardsboards
Recommendation:Recommendation: Work on hierarchyWork on hierarchy Provide multiple levels of Provide multiple levels of
configurability.configurability. Clearly differentiate Group Clearly differentiate Group
Discussions from general Discussions from general class discussionsclass discussions
Be able to rename “Group” as Be able to rename “Group” as “Lab”“Lab”Rating: MODERATE to SEVERE
NavigationNavigation Observation:Observation:
Students go to “staff Students go to “staff information” instead of information” instead of “communications” for email“communications” for email
Recommendation:Recommendation: Provide clearly-labeled “email” Provide clearly-labeled “email”
or “contact instructor” buttonsor “contact instructor” buttons
Rating: MODERATE
EmailEmail Observation:Observation:
Students used external email Students used external email appapp
Did not use Blackboard email Did not use Blackboard email functionsfunctions
Recommendation:Recommendation: Provide more familiar e-mail Provide more familiar e-mail
interfaceinterface
Rating: IRRITANT
User Testing ResultsUser Testing ResultsInstructorInstructor
Adding Multiple UsersAdding Multiple Users Observation:Observation:
Batch functions not evidentBatch functions not evident Added one at a timeAdded one at a time Large amount of search returnsLarge amount of search returns Used “guessing” strategy to locate Used “guessing” strategy to locate
studentsstudents One used user IDOne used user ID Instructors have batch enroll rightsInstructors have batch enroll rights
Recommendation:Recommendation: Provide more evident labelsProvide more evident labels Separate enrollment from Separate enrollment from
creation of useridcreation of userid Make the function obvious and Make the function obvious and
easy to useeasy to use
Rating:SEVERE
Editable NavigationEditable Navigation Observation:Observation:
Mandatory use of pre-made labels Mandatory use of pre-made labels limiting, especially with foreign-limiting, especially with foreign-language instructorslanguage instructors
Labels often confusing to studentsLabels often confusing to students
Recommendation:Recommendation: Provide multiple language Provide multiple language
supportsupport Provide configuration options Provide configuration options
for interface elementsfor interface elements
Rating:MODERATE
Virtual ClassroomVirtual Classroom Observation:Observation:
Instructors did not find function to be Instructors did not find function to be usefuluseful
Unproductive with large classesUnproductive with large classes Concerned about student access& Concerned about student access&
browser compatibilitybrowser compatibility Limited space for typingLimited space for typing Little space to read messagesLittle space to read messages Loads slowly in some configurationsLoads slowly in some configurations
Recommendation:Recommendation: Provide greater stability Provide greater stability
across platforms/browsersacross platforms/browsers Larger in-chat text areaLarger in-chat text area
Rating:MODERATE
Assessment ManagerAssessment Manager Observation:Observation:
Frequently misinterpreted and Frequently misinterpreted and ignoredignored
Could not identify function from Could not identify function from labellabel
Few instructors could create Few instructors could create online quizonline quiz
Recommendation:Recommendation: Clearly defined labels based Clearly defined labels based
on user insighton user insight Clearly define function based Clearly define function based
on academic-- not technical--on academic-- not technical--languagelanguage
Rating:MODERATE
GradebookGradebook Observation:Observation:
Did not find usefulDid not find useful InflexibleInflexible Grades confused studentsGrades confused students Wanted ability to import gradesWanted ability to import grades Wanted ability to export gradesWanted ability to export grades
Recommendation:Recommendation: Provide “weighting” functionalityProvide “weighting” functionality Provide ability to factor in letter Provide ability to factor in letter
gradesgrades Provide import/export capability.Provide import/export capability.
Rating:MODERATE
Advanced OptionsAdvanced Options Observation:Observation:
Ignored option settingsIgnored option settings Expressed curiosity about Expressed curiosity about
infrequently-used functionsinfrequently-used functions Amount of choices overwhelmingAmount of choices overwhelming Often assumed red reminders Often assumed red reminders
were warnings of errorswere warnings of errors
Recommendation:Recommendation: Improve trainingImprove training Improve help functionsImprove help functions Provide multiple user levels of Provide multiple user levels of
complexity: “Power User” vs. complexity: “Power User” vs. default simple interfacedefault simple interface
Interactive “Wizards” to guide Interactive “Wizards” to guide setupsetup
Use color other than red in text for Use color other than red in text for simple confirmationssimple confirmationsRating:MODERATE
GroupsGroups Observation:Observation:
Instructors added each student Instructors added each student individuallyindividually
One instructor hired grad assistant One instructor hired grad assistant just for this functionjust for this function
Recommendation:Recommendation: Allow batch updatingAllow batch updating Create groups before adding Create groups before adding
students to themstudents to them
Rating:MODERATE
Highlights: VideoHighlights: Video
OverallOverall
Effective and easy to use for basic tasksEffective and easy to use for basic tasks Terminology/wording confusingTerminology/wording confusing Instructors crave batch functionsInstructors crave batch functions Interface inflexibleInterface inflexible Metaphors can get in the way of functionality Metaphors can get in the way of functionality
(eg. Digital DropBox)(eg. Digital DropBox) Often too much information/optionsOften too much information/options
Sneak Peek: Blackboard 6Sneak Peek: Blackboard 6
UMBC migrating to version 6 for Spring 2003UMBC migrating to version 6 for Spring 2003 LoginLogin
http://preview9.blackboard.com/http://preview9.blackboard.com/ Userid = umbctestUserid = umbctest Psswd = umbctestPsswd = umbctest
Available for 4-6 weeksAvailable for 4-6 weeks Release notes in “UMBC Test Course”Release notes in “UMBC Test Course” Send comments to Send comments to [email protected]@umbc.eduedu
Next Usability Study: myUMBCNext Usability Study: myUMBC
myUMBC has not changed much in 4 yearsmyUMBC has not changed much in 4 years PeopleSoft HR & Finance launch July ’03PeopleSoft HR & Finance launch July ’03 Need for more roles & affinitiesNeed for more roles & affinities Questions: Questions:
How do current students, faculty & staff currently use How do current students, faculty & staff currently use myUMBC vs. UMBC website?myUMBC vs. UMBC website?
What would users like to do now but can’t?What would users like to do now but can’t? How could campus make better use of its portal?How could campus make better use of its portal?
Likely redesign launch date: April ‘03Likely redesign launch date: April ‘03
LinksLinks
www.umbc.edu/oit/newmedia/blackboard/usabilitywww.umbc.edu/oit/newmedia/blackboard/usability::
PresentationPresentation Final ReportFinal Report Highlights VideoHighlights Video