Faculty Feedback on Blackboard Course Management System Spring, 2014

Cornell University Report by Kim Nicholson, Marina Tokman, and Todd Maniscalco

Background

In the Spring of 2014, Academic Technologies sought feedback from instructors regarding their experience with the campus Course Management System (CMS), Blackboard Learn 9.1 - Service Pack 11.

The primary goals of the inquiry were to:

1. Examine the degree to which faculty were using Blackboard 2. Discover what other (non-Blackboard) tools faculty are using 3. Discover what top CMS features faculty are interested in 4. Ascertain satisfaction with existing Blackboard features 5. Reveal what suggestions for improvement, if any, faculty had to offer

An online survey (Qualtrics) was sent to all faculty (2,661) in April, 2014. Those who were invited to take the survey included faculty who use the campus CMS as well as faculty who do not. This survey was also administered for the first time to a wider audience who have never had the opportunity to provide feedback on Blackboard or any other CMS. The last study of instructor satisfaction with Blackboard was restricted to those who do use the tool and was conducted in 2011. Since 2011, 2,386 unique faculty users have logged into Blackboard and are listed as active instructors in a course. At the time this survey was conducted, there were 19,559 active users in the Blackboard system.

In 2014, 390 individuals began the survey and 362 completed the instrument for a response rate of 13.6%. A broad range of Colleges were represented (Figures 1 and 2).

Figure 1. Colleges Represented

Responses Responses ARTS AND SCIENCES 102 ARCHITECTURE ART & PLANNING 7 AGRICULTURE & LIFE SCIENCES (CALS) 70 VETERINARY MEDICINE 5 ENGINEERING 38 COMPUTER INFO SCIENCE 4 HUMAN ECOLOGY 23 LEARNING STRATEGIES CTR 4 HOTEL SCHOOL 19 WASHINGTON 1 JOHNSON SCHOOL 18 PROVOST OFFICE 1 ILR 18 NYC TECH CAMPUS 1

LAW SCHOOL 7

Figure 2.

Usage

All 390 participants responded to the first question which was whether or not they use Blackboard. 82% replied that they did (Figure 3), and by far the number of semesters in use was six or more (Figure 4).

Figure 3. Do you use Blackboard?

Responses Percentages No

72 18%

Yes

318 82% Total 390 100%

0

20

40

60

80

100

120

Figure 4. Including the current semester, how many semesters have you been using Blackboard?

Responses spons Percentages 6 or more

121 77%

5

5 3% 4

9 6%

3

10 6% 2

7 4%

1

6 4% Total 158 100%

That nearly 80% of all respondents have been using Blackboard for six or more semesters is worth noting. The feedback offered in this survey is drawn from multiple years of working with different versions of Blackboard. Challenges or frustrations expressed should not be discounted as ‘typical novice issues’ that will resolve upon skill development. That said, years of working with the tool does not imply mastery. Forthcoming open comments suggest that some instructors believe they possess mastery when in fact, their statements contain errors and misperceptions. The aforementioned reluctance to obtain training may explain why some frustrations persist.

Non-Usage

An open comments field was presented to the 18% not using Blackboard. When asked why they chose to not use the service, 57 individuals answered. The top five answers ranked in order of frequency, were as follows:

Why do you choose not to use Blackboard?

#1 Blackboard is inadequate (clunky, cumbersome, inflexible, not user friendly, slow, difficult to navigate, difficult to understand) - 22 comments (39% of non-users)

#2 I do not actually have teaching assignments (this question does not apply to me) – 9 comments (16% of non-users)

#3 The Computer Science Department provides us with a CMS -8 comments (14% of non-users)

#4 I do not need a CMS for my teaching – 6 comments (11% of non-users)

#5 Our College (unspecified) provided a CMS to us/I prefer my own web pages/I don’t have time to learn Blackboard - 4 comments each

Faculty were also asked what other technologies they use. Figure 5 suggests that the course management system provided by the Computer Science Department and Piazza are used the most by non-Blackboard users. Blackboard users also frequently use Piazza and their own web sites.

Note: Currently Blackboard instructors can set up links to Piazza from their courses.

Figure 5. What Alternative technologies do you use to manage your courses (posting materials online, notify students of grades, hold online discussions)?

Rank

Faculty who do not use Blackboard

Faculty who use Blackboard

#1 “Computer Science Management System”

15 comments “Piazza” 19 comments

#2 “Piazza” 14 comments “My own website”

14 comments

#3 “My own website” 8 comments “Email/Google apps including Google Docs”

9 comments

#4 “Email” 5 comments “Excel or Access”

8 comments

#5 “Language Resource Center/Wikis not specified/GoogleDocs”

3 comments each

“Blogs/Publisher Provided Tool”

4 comments each

Other tools mentioned are not being utilized in significant numbers.

Other Tools Mentioned (less than 2% of all respondents) Use of Cornell’s WebEx Service, Adobe Connect/Skype Dropbox/Box.com/Cornell Box Sakai Jooners Qualtrics/Poll Everywhere/Survey Monkey Drupal Joomla APLIA ClassLists Confluence Facebook /Twitter Camtasia Mind jet/Mindmanager Desire2Learn Turnitin

Features

Those who use Blackboard were asked to rate the features they use. Figure 6 shows that two features that are used the most, adding materials to course documents and/or content areas, along with announcements & email, are meeting all or most of the needs of respondents. Many of the standard features available in the CMS are underutilized. Four of the features that are used the least are: Turnitin plagiarism detection, Quizzing tool, Chat and virtual collaboration tool, and Wikis, blogs, and journals. System reports show that on average 569 (roughly ¼) Blackboard instructors used Turnitin during the past year. There were no features that DID NOT meet needs for significant number of users.

Figure 6. To what degree do the following Blackboard features meet your Instructional Needs on Campus?

Meets all

Meets most

Meets some

Does not

meet

Do not use

Not aware of this feature

Total Responses

Adding materials to Course Documents and/or Content Areas

45.93% 34.85% 15.96% 1.63% 1.63% 0.00% 307

Sending notifications using Announcements, Email

50.33% 28.62% 11.18% 2.96% 6.25% 0.66% 304

Grading through Grade Center

16.01% 18.95% 17.65% 6.54% 36.27% 4.58% 306

Collecting Student Work using Assignment tool

12.79% 16.72% 8.85% 1.97% 54.10% 5.57% 305

Conducting assessments with Quizzes and Tests

4.28% 8.88% 7.24% 1.97% 72.04% 5.59% 304

Checking originality of work with Turnitin Assignments

5.92% 8.22% 3.62% 2.30% 73.68% 6.25% 304

Using Discussion Boards 7.19% 12.09% 9.48% 3.59% 64.71% 2.94% 306 Using Groups 7.54% 8.85% 9.84% 1.97% 66.89% 4.92% 305 Using Chat and/or Virtual Classroom for synchronous collaboration

1.97% 0.33% 0.00% 1.97% 85.20% 10.53% 304

Using collaborative tools such as Wikis, Blogs or Journals

2.97% 1.98% 3.63% 2.31% 77.23% 11.88% 303

Linking to CU Library Reserves

15.18% 10.23% 5.28% 3.30% 53.80% 12.21% 303

Other: please specify 7.94% 3.17% 4.76% 7.94% 55.56% 15.87% 60 Adding Videos using Kaltura

1.97% 3.29% 2.30% 1.64% 66.45% 24.34% 304

Adding Lecture Capture with Panopto

2.99% 1.66% 1.33% 1.00% 69.44% 23.59% 301

Representative comments suggest different reasons of reluctance to use Blackboard’s features. Many individuals stated they would prefer a “lite” version of Blackboard:

Blackboard's extensive menu of choices is overwhelming It is overly complicated with way too many bells and whistles that I would never consider using. Perhaps you could offer a streamlined version. Can we please have a vanilla tool for a CMS? Others do not find the features pertinent to their style of teaching:

I use Blackboard just to post assignments and messages of various sorts, and to post readings from e.g. various journals. I get along fine with pen and paper and email – seriously! I don’t need Blackboard or any other CMS.

When faculty were asked about features that are not meeting their needs because they are not perceived as sufficient, the majority of comments included improvements to the Interface and Grading. Detailed breakdown of all comments is provided below:

Announcements and Emails are problematic (auto emails a real problem, subject header too long, Unable to auto send announcements)

Groups are too confusing to set up/Group assignments do not allow individual submissions Blogs are too clunky with insufficient comment/threading format is unclear Wikis are not even close to real wikis Grade Center is convoluted and cumbersome/Import & Export do not work well/absence of sub-

gradebooks for sections Testing takes way too long to create quizzes/Quizzes often do not export accurately Folder organization is poor – The inability to drag and drop does not make sense Panopto (lecture capture) appears to have version issues and permission issues Turnitin seems unable to do group papers Discussion Board is just not up to the task (Piazza is) Peer Review is not reliable

Additional comments:

I'd like to be able to track the number of students downloading content in the current day. However, the download reports ALWAYS exclude the current day. I have asked about this before and it's still not functional.

It's difficult to allow auditors access to the site, which is unfortunate

Every year they come out with new features and that just pushes back my willingness to learn them

Respondents were asked to rank the top five features that are important to them in a CMS. Both Blackboard users and those who do not use Blackboard have selected similar features. Figures 7 and 8 show how they ranked them.

Figure 7. Please select the top five features that are important to you:

Rank

Feature

Count

#1 Adding content materials to course 287 #2 Sending Announcements, Emails 254 #3 Grading 142 #4 Collecting Student Work 116 #5 Using Discussion Board 72

Figure 8 indicates the top five features that are important to faculty who DO NOT use Blackboard.

Figure 8. Please select the top five features that are important to you:

Rank

Feature

Count

#1 Adding content materials to course 37 #2 Sending Announcements, Emails 32 #3 Collecting Student Work 29 #4 Grading 26 #5 Recording Lectures and Conducting Assessments (tied) 16 (each)

Course Management Tasks

Using the campus CMS instructors have the option to link their Blackboard course enrollment to the official University Registrar’s enrollment. In order to do this, instructors or their administrative staff need to create new course every semester and copy course materials.

Figure 2. Screen capture of Faculty Center, the environment in which faculty request Blackboard Course sites.

Figure 9 suggests that overall, the majority of respondents are “Satisfied” with Blackboard’s ability to handle course management tasks.

Figure 9. How satisfied are you with Blackboard’s ability to handle the following course management tasks?

Very Satisfied

Satisfied Neutral Not Satisfied

Not Applicable

Total Responses

Setting Up a Course for New Semester

22.92% 40.86% 18.27% 13.29% 4.65% 301

Enrolling/Removing Users 20.20% 45.36% 15.56% 12.91% 5.96% 302 Recycling Course from Previous Semester

14.38% 36.79% 21.74% 15.05% 12.04% 299

Copying/Exporting/Archiving a Course

10.74% 28.52% 23.15% 14.09% 23.49% 298

Managing Course Properties (Duration, Availability, Categorization, etc.)

11.74% 35.23% 25.50% 12.08% 15.44% 298

Other: please specify 0.00% 9.38% 6.25% 34.38% 46.88% 31

Eighty four individuals (24%) responded to an open comments question that inquired how course administrative tasks could be improved. Unfortunately, most of the comments did not answer the

question. These statements were transferred to the section that sought feedback on “Overall concerns with CMS”. Furthermore, several comments that did answer the question contained misperceptions, e.g. “If only there was a way to connect Blackboard with the Registrar’s People Soft System.” Comments that did not contain errors are summarized below.

How Can Course Management Tasks Be Improved? Please make [admin tasks] easier. /Need simpler copy [process]/ Every semester it is a pain to figure out how to do these tasks. / I have reflected multiple times [on] the chronic difficulties with enrollments in Blackboard. It has caused so many headaches. (as I have said, I hope we get rid of Blackboard for good) I want to be able to copy select files from certain weeks of a past course, instead of just blanket copying the entire course at once. I cannot figure out how./ I teach the same courses over and over…. I do not understand why I have to do anything to keep using the same material. Seems it is set up for the convenience of the administrators rather than the instructors./ As it stands, I basically start from scratch each semester. That would be fine if Blackboard were easy to use, but it isn’t. We need a batch drag [and drop] of materials from previous semester to new semester; while keeping previous semester available in archive, unchanged. Roles/permissions not specific enough. I don’t understand what is and is not seen. Identifying TAs [and manually enrolling them] is particularly complicated for no good reason/ It seems like it takes two reloads to make sure users were added correctly; this often take 10-15 minutes each semester, especially when we have TAs. I am glad this system is finally synched with the registrar, but students are not aware of this and still email me all the time to "add them to Blackboard". The protocols for [admin] tasks keeps changing/ I always have my TA do this task as the process seems to change every year. I am certain that I have not archived the courses correctly and will lose a lot of material I have had difficulty adding visiting scholars who are not enrolled in the class, but need access to the materials on blackboard. Online instructions [for Admin Tasks] weren't adequate/ I had to call for help. Too much lag-time between course adds and Blackboard enrollment and it almost always generates a student email that I have to respond to.

Training, Consultation and Documentation

The campus CMS is supported by a full range of training, consultation and documentation services. Faculty are a click away from an extensive resource “BbHelp” (bbhelp.cit.cornell.edu) which is available 24/7, email assistance is available via atc_support@cornell.edu and provides response within 12 hours and within 4 hours at the start of each semester.

Figure 10. Screenshot of Academic Technologies’ Blackboard Help Website

While Figure 11 indicates that the majority of respondents have “never used” these services, a significant count for Academic Technologies’ website documentation was noted for the answer choice: “Satisfied”.

Figure 11. How satisfied are you with training, consultations and documentation for Blackboard?

Very satisfied

Satisfied Not satisfied

Never used

Total Responses

Face-to-face training sessions 10.44% 13.13% 2.02% 74.41% 297 Online webinars 1.35% 9.12% 2.70% 86.82% 296 Custom training for a department, group or unit

2.72% 3.40% 1.70% 92.18% 294

One-on-one consultation with a staff member

24.49% 18.03% 3.06% 54.42% 294

Blackboard documentation on the Academic Technologies website (bbhelp.cit.cornell.edu)

4.75% 41.02% 16.61% 37.63% 295

When asked how these services could be improved, 35 (12% of respondents) open comments were provided. The majority of comments fell under the category of ‘reluctance to use documentation and/or training’, e.g. “Faculty don’t have the time for these services”, “I don’t want to have to set up a consultation or go to a training session” and “Good software should be intuitive and not require training.” The remainder fell into the following two categories.

• Services cannot be improved/Face to Face training is good/[name of staff member] is quite helpful/Online documentation not helpful, but the human support is very good/Rapid response to questions is extremely good.

• Documentation (bbhelp.cornell.cit.edu) is hard to find/Documentation needs an index/Documentation needs more detail/ In general, CIT documentation is poor and often out-of-date.

General Comments about Blackboard

Respondents were eager to share their comments about Blackboard. Over 100 responses were received and many survey takers wrote lengthy answers. Some comments included remarks that the product was “fine”, “okay” and “I’m getting to know it better”. The themes that emerged however, expressed concerns with Blackboard’s interface and grading. The other common theme is to minimize frequency of changes. The following comments are offered:

Overall liked Blackboard:

• Please keep it. I need it. • It is a very useful system for course management. It has also greatly reduced my expenses for

copying material for distribution to students. • I like Blackboard, and I would strongly prefer to continue using it. A lot of the textbooks offer test

banks that can be imported into Blackboard. • It is a very useful tool, and I hope we can maintain support for it (mainly, to help instructors save

time) so they can spend that time interacting with students. • STAY WITH BLACKBOARD! DO NOT CHANGE SOFTWARE! • I really love using Blackboard. It meets all my needs and then some. • It's been very useful to me and I use it for all my courses. • I am in general happy with the Blackboard, but I am not able to use many services it provides. It

may have to do with my teaching needs. In short, I am satisfied with the services provided. • Blackboard has allowed me to manage much of a course in a paperless manner. It also makes it

much easier to provide reading and other course materials to students in a very accessible way.

Generally OK with Blackboard but with some reservations or caveats:

• It's basically a good system. • It is not an intuitive program. Recent updates differ substantially from previous versions so

transitions from one to another are not smooth. I could take more advantage of all the features it has, but to do so would take a lot of effort and I'm not convinced that students would learn more as a result.

• Seems to work well, but I'm not familiar with other programs, so there may be better options available.

• Over the years BB has become easier to use but it could be streamlined further. I'd like to use all of BB's cool features, but I would have to invest too much time to learn them.

• I especially appreciate the training that you have provided for my TAs. This has done more to help me get my site up and running each semester than anything else.

• There should be a simple, bare-bones version for the many instructors who do not need the many many features that use up my time.

• It serves my needs • Straightforward and seems to be relatively fast. • This summer, when teaching obligations are low, I would like to learn more about BB features.

Blackboard is clunky, cumbersome, non-intuitive with an inflexible interface/way too many clicks/Defaults don’t make sense

• Powerful features but terrible interface • Arcane multiple step procedures with arcane terminology. I should be able to click on the

appropriate box on the grade sheet, put in the grade and drag and drop any attachments I want the student to see.

• The defaults are often bizarre (why would exams default to not showing the feedback we spend hours writing? why are there two types of feedback, which can only be seen under different circumstances?).

• The whole Blackboard interface is TERRIBLE. • Blackboard can be quite useful in many ways. However, it tends to create more and more work. • When I want to download an assignment I shouldn't have to guess at what the right steps are. I

should be able to just click on a file and drag it to my desktop. That's how every other contemporary interface works these days.

Blackboard has too many features/Difficult to understand/Please give us ”Vanilla” too

• A lighter weight platform would be more to my liking. • There are too many options that are not well explained. There should be a simple, bare-bones

version for the many instructors who do not need the many features that use up my time. • Frankly, the user interface is too chaotic. • [The features themselves] are good, but the names of the associated tasks are [non-intuitive e.g.

“Adaptive Release”] which requires reading A LOT OF instructions in order to do something simple.

• It's like you have to get a BA in Blackboard to really get much use out of it. • I sat through a focus group you guys convened a year ago. Everyone basically agreed--it is so

difficult to figure out whether or not it does what you want that we don't even try. Why are we still asking questions? What happened to [our feedback from] your focus groups?

• Look, Blackboard is just way too complicated and not well-documented. • In general, nothing is easy to use, the interface often doesn’t work. • I have to smile that this survey was done in Qualtrics and not with Blackboard itself.

Grade Center is Insufficient, particularly for sections and large classes

• Grade center does not meet the needs of large lecture courses/I have had difficulty especially understanding how Grade Center works and how to get sections working

• You won't like my saying this, but the software is written by people who don't know the first thing about university education.

• When I told learning software experts at NYU we were using Blackboard at Cornell they were shaking their heads. It has a very poor reputation.

• Grade Center should be more efficient and user-friendly. Entering grades is difficult. • Please pick another system with a logical grade book set up

Upgrades are too frequent/Time invested in the tool does not necessarily pay off

• I'd rather spend my time teaching than fussing with Blackboard. The only reason I use it is so that I can put course readings online. Otherwise it's a waste of time.

• There are so many changes each year that one has to learn a whole new way of using the site. • It’s hard [enough] to figure out how to do something, [when it will only change] with upgrades,

forcing a continuous process of relearning. • Blackboard is okay. / The course administrator did not know how to make it possible for students

in the parent course to access the supplemental materials if they wanted to.

Conclusion

Academic Technologies sought feedback from instructors regarding their experience with the campus Course Management System (CMS), Blackboard Learn 9.1. The majority of respondents have been using Blackboard for six or more semesters and indicates that they started using Blackboard 3 or more versions ago. Two of Blackboard’s features: Adding Content and Announcements/Emails are presently meeting all needs for the majority of Blackboard users. Many features are not used by the majority of respondents. Academic Technologies’ documentation, consultation and training are currently underutilized. In general, faculty either do not have the time, nor do they believe they should have to invest the time when, in their view many of software products they use (Qualtrics, Google, etc.) are intuitive, streamlined and fairly easy to learn unassisted. Overall, the majority of respondents are “Satisfied” with Blackboard’s ability to handle course management tasks. It is their wish however, that they did not have to copy over their courses each semester. While faculty are not necessarily clamoring for a switch to a new Course Management System, they are asking for a simple drag and drop interface, simpler course recycling for each semester, cleaner interface, fewer clicks and a tool that focuses almost entirely on the four features that are important to them: Adding Content Materials to Course, Grading, Collecting Students’ Work and Sending Announcements & Emails.