This article was downloaded by: [Northeastern University]On: 22 November 2014, At: 07:05Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

Medical Reference ServicesQuarterlyPublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/wmrs20

The Initiation of DistanceLearning at Thomas JeffersonUniversityAnthony J. Frisby PhD a & Susan S. Jones MS aa Academic Information Services and Research ,Thomas Jefferson University , Philadelphia, PA,19107-5587, USAPublished online: 12 Oct 2008.

To cite this article: Anthony J. Frisby PhD & Susan S. Jones MS (2000) The Initiationof Distance Learning at Thomas Jefferson University, Medical Reference ServicesQuarterly, 19:3, 19-37, DOI: 10.1300/J115v19n03_02

To link to this article: http://dx.doi.org/10.1300/J115v19n03_02

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for anylosses, actions, claims, proceedings, demands, costs, expenses, damages,and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of theContent.

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone isexpressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

The Initiation of Distance Learningat Thomas Jefferson University:The Library as Integral Partner

Anthony J. FrisbySusan S. Jones

ABSTRACT. In 1998 Thomas Jefferson University offered its firstentirely online course. Librarians and library staff were integral in thedevelopment, support, evaluation, and refinement of this course. Whilestaff members may have taken non-traditional roles in this effort, theirroles generally fell within the broad guidelines of assisting Universityfaculty with information and knowledge management. The develop-ment and support of distance course offerings will continue to be afocus at Scott Memorial Library. [Article copies available for a fee from TheHaworth Document Delivery Service: 1-800-342-9678. E-mail address:<getinfo@haworthpressinc.com> Website: <http://www.HaworthPress.com>]

KEYWORDS. Distance learning, distance education, nursing educa-tion, Web-based instruction

INTRODUCTION

In late 1997, staff from Thomas Jefferson University’s Academic Informa-tion Services and Research (AISR) and the College of Health Professions

Anthony J. Frisby, PhD, is Director of Education Services and Susan S. Jones,MS, is Senior Librarian, both with Academic Information Services and Research,Thomas Jefferson University, Philadelphia, PA 19107-5587.

The authors wish to acknowledge the significant contributions of: Molly Rose,PhD, course faculty and content expert for the course; Christopher D. Braster, Com-puter-Based Learning Developer and Webmaster; and Michael D. Hamlin, PhD,previous Director of Education Services and Marketing.

Medical Reference Services Quarterly, Vol. 19(3), Fall 2000E 2000 by The Haworth Press, Inc. All rights reserved. 19

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY20

Nursing Department initiated discussions about the development of a dis-tance learning course. The Nursing Department was interested in reachingpotential program candidates unable to avail themselves of the traditionalcampus-based course offerings. AISR’s interest in distance learning was a directoutcome of its earlier Learning Infrastructure Project report (http://aisr.lib.tju.edu/Education/LIP/), which detailed trends in education and reaching new mar-kets.Epidemiology, a graduate course required for advance practice nurses, was

selected as a pilot. A Web-based version of the course was developed over anine-month schedule. The content and objectives of the distance learningcourse and the campus-based course were identical (see Figure 1), but materi-als and activities differed. Online course materials included graphics, anima-tion, over 1200 information pages, a search utility, glossary, self-assessmentpractice quizzes, exercises, discussion boards, and case studies. The onlinecourse was first offered in the fall of 1998.This paper reviews the impetus for non-traditional campus learning pro-

grams and alternative delivery methods for distance learning, details the

Figure 1. Course Description and Objectives

Course Description

This course applies the concepts, principles, and uses of epidemiology. Population-basedcollection and analysis of health data and its relationship to health services will beemphasized. Application of epidemiological methods to infectious and chronic diseases willbe critically analyzed.

NU625 is a required part of the core curriculum for all graduate nursing students at ThomasJefferson University. Health professionals in all specialty areas use aspects of the principlesof prevention, early identification, and prompt treatment of health problems. Epidemiologyprovides the background and skills to enhance the ability of health professionals to providecomprehensive management of care for the client/public (e.g., prevention, treatment,rehabilitation, disease surveillance, planning and evaluating health services and interventions.

Course Objectives

At the completion of this course, you should be able to:

describe basic principles of descriptive and analytical epidemiology.

identify the uses and limitations of epidemiology.

apply the epidemiological model and the causal chain model to specificcommon diseases.

analyze the relationship of basic measures of disease occurrence such as incidence,prevalence, case-fatality, relative risk, odds-ratio and mortality rates.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 21

development of a course--including the critical contribution of the library andlibrary staff--and presents the results from the course’s first offering.

BACKGROUND: THE NURSING PROGRAM

Thomas Jefferson University, a medical and health care professions insti-tution, comprises three colleges: Jefferson Medical College, the College ofGraduate Studies, and the College of Health Professions. Within the Collegeof Health Professions, the Nursing Department offers both undergraduate andgraduate degree programs. Recognizing that the Philadelphia health profes-sional education market is a very competitive one, the Nursing Departmentdecided to expand its reach by offering Web-based courses for graduatestudents. This format was preferred over video-based or text-based formatsbecause it was felt that the multimedia increased student involvement withthe content and would strengthen learning, thereby offering a stronger pro-gram.1The first course selected for online development was one of several newly

designated core courses in the graduate degree program. These are requiredof all matriculating students; non-matriculating students can take several ofthese courses as well, giving candidates the option to experience the curricu-lum before committing to a degree program. Current plans include develop-ing online versions of all four core courses in the graduate nursing program,with the goal of reaching a wider pool of students from which to recruit. TheJefferson Health System, for example, employs an enormous number ofpotential students. Attracting them to the program benefits the students, theHealth System hospitals, and the University.

BACKGROUND: DISTANCE LEARNING

Although distance learning is currently a hot topic, it is hardly a newconcept. As early as 1883 distance learning appeared on the American educa-tion scene.2 Initial formats were text-based and involved students andinstructors writing back and forth, hence the name ‘‘correspondence course.’’As technology became available it was included in the instructional toolbox.Delivery options quickly moved from audiocassette tapes to broadcast TV,cable TV, videotape, and more recently the computer.The rising popularity of the Internet and World Wide Web in the last few

years increased the public’s access to learning materials. Suddenly, develop-ers saw an enormous, electronically interconnected potential audience, whichmade it both tempting and relatively easy for them to rapidly produce and

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY22

distribute instructional material. However, as experience has shown, justbecause something is easy to produce and distribute does not necessarilymean it will be well planned and effective. Many companies and institutionsimmediately produced instructional material for Web delivery, only to dis-cover that it either did not meet the needs of the users or was not deemedcredible. Although distance courses have evolved in the last five years andmany highly respected universities now offer rigorous, educationally-sounddistance degrees, the notion persists with some educators and students thatdistance learning is inferior to the traditional campus-based format. As theteaching medium continues to evolve and as more and more students andeducators become proficient distant learners and distant teachers, this percep-tion will change. Distance courses will be judged on their merits and out-comes and not solely on their format.

BACKGROUND:ACADEMIC INFORMATION SERVICES AND RESEARCH

Organizationally, Thomas Jefferson University’s information ‘‘infrastruc-ture’’ differs from that of many universities. What is commonly recognizedas ‘‘the Library’’ is actually a much larger division, Academic InformationService and Research (AISR). This division specializes in the development,distribution, evaluation, and archiving of information. It includes ScottMemorial Library, Education Services, Medical Media Services, and the Of-fice of Academic Computing. Scott Memorial Library is responsible for refer-ence, collection development, technical services, the archives, document deliv-ery, and computer labs; Education Services provides computer-based learningdevelopment (instructional design, programming, and graphic animation), in-formation literacy instruction, and marketing services; Medical Media Servicescontributes art design, photography, audio recording and videography, in bothanalog and digital formats; and the Office of Academic Computing contrib-utes database administration and technical support of all library systems.Because of its broad goal of providing a coordinated approach to informa-

tion and knowledge management, AISR is intimately involved with a varietyof tasks and projects traditionally outside the purview of librarians and li-brary staff. The development and support of distance learning courses is anexample of one such non-traditional role.

DEVELOPMENT TEAM

The literature of software development and distance learning describesnumerous benefits in using a team approach.2 AISR has successfully used

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 23

development teams to design and produce highly effective instructional ma-terial. The team contributes qualities no one person can provide:

From the standpoint of development, creating courses in a new mediumsuch as the Internet can be very exciting. Very often, however, thefaculty member, well versed in his or her own subject area, does notpossess sufficient knowledge in the areas of technology or instructionaldesign.3

Also, the Kellogg Report identifies a major obstacle to developing onlinelearning materials as ‘‘skepticism about faculty members’ expertise in teach-ing via distance education.’’4 AISR has worked to overcome this obstacle byproviding expertise in technology and learning theory, both in workshopformat and in small group counseling for those interested in developingonline materials. In addition, the use of a development team provides strong,non-threatening support for faculty inexperienced with online course devel-opment or teaching.For the production of its first distance learning course AISR used the same

team model it uses in the development of computer-based learning programs.Members of the team included professionals with diverse backgrounds andskills. First and foremost, of course, was the faculty member who was re-sponsible for the content of the course and who would also be teaching it. Tothis AISR contributed an instructional designer, an educational psychologist,a librarian, a programmer, a computer graphics artist, and a database adminis-trator. Specific team members and their primary roles are outlined in Table 1.Of course, a great deal of the work was accomplished as a team, with collabo-ration and overlapping responsibilities.

TABLE 1. Primary Responsibilities of Development Team Members

Position Responsibilities

Instructor Wrote course content, determinedappropriate exercises; reviewed all content foraccuracy.

Educational Services Librarian Organized team and kept it on schedule; editedcourse; researched and wrote historical piece.

Director, Education Services & Advised on educational principles/psychology;Marketing researched and wrote historical section.

Instructional Designer Developed course interface; identified appropriatetools for course; converted content into storyboards;developed two case studies.

Computer-Based Learning Developer Programmed course from storyboards, usingHTML, Shockwave, JavaScript, PL/SQL.

Computer-Based Learning Designer Produced graphics and animations.Database Administrator Developed database tables, login procedures,

and system for monitoring student progess.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY24

S Course faculty and subject matter expert--worked closely with the otherteam members to write the course content, determine appropriate exer-cises and review all content for accuracy.

S Educational services librarian--functioned as course editor, responsiblefor keeping the group organized and on schedule, reviewing materialfor consistency and working with the graphic artist to develop appropri-ate images and animations. Also researched and wrote material on thehistorical use of epidemiology and developed an animated tour wherelearners follow Dr. John Snow through the 1854 cholera epidemic inLondon.

S Director, Education Services and Marketing--advised on educationalprinciples/psychology. Also researched and wrote historical section onFlorence Nightingale and William Farr.

S Instructional designer--worked closely with the programmer and artistto develop the user interface for the course. Identified appropriatelearning technology tools for the course and converted the content sup-plied by course faculty into storyboards. Storyboarding allowed the de-velopment team to preview how the course would appear to a user be-fore the programmer began his task. Also developed two of the course’scase studies.

S Computer-based learning developer--programmed the course from thestoryboards, text and graphics, using a combination of HTML, FlashShockwave files, JavaScript, and PL/SQL commands.

S Computer-based learning designer--worked closely with the develop-ment team to produce graphics and animations that engage the learner.

S Database administrator--developed the database tables and login proce-dures and worked with the computer-based learning designer to recordthe students’ progress through the material and their responses to ques-tions. The unique tracking procedure allows both the students and thefaculty to monitor the students’ progress through the course.

Team members spent a combined total of nearly 1600 hours developingthe distance course. A chart depicting the breakdown of these hours is pro-vided in Figure 2.

COURSE DESIGN AND PRODUCTION

Epidemiology for the Health Professions was Jefferson’s first full-creditdistance learning course. It was offered in the fall of 1998 and ran concur-rently with the traditional campus-based course, allowing direct comparisonof performance between the two groups. Production began nine months beforethe start of the course to ensure sufficient time to develop and field test the

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 25

FIGURE 2. Breakdown of Hours Devoted to Development of Online Course

Programmer

InstructionalDesigner

Faculty

Artist

Editor

DatabaseAdministrator

course and make any necessary revisions. It should be noted that although thecourse was finished within the allotted nine months, every bit of this time wasneeded; in fact there were many weeks of frantic scrambling prior to the startof the term. Of course, the nine months included developing the look andstructure of the course, which would be used for subsequent courses. Eachmajor facet of the development process is described below.

Course Goals and Objectives

The development team met with the instructor to review the course goalsand objectives. Doing so familiarized the team with the content and desiredoutcomes of the course. The instructional designer developed a table (seeTable 2) to help the instructor identify distance learning alternatives to campus-based instruction. The table has subsequently been used in several facultydevelopment workshops to help instructors incorporate educational technolo-gy components into the curriculum. Jefferson, like other universities, firstexplored educational technology support for traditional, campus-basedcourses before considering distance learning. We continue to encourage fac-ulty to experiment with technology components in traditional courses beforeattempting to develop entire Web-based courses.

Course Content

After reviewing text materials and lecture notes, the team divided thecourse into five topical sections. Each of these was further divided intodiscrete learning modules. The faculty member began writing text for theprogram and worked with the course editor to organize the text in a formatthat would be easy to read on a computer monitor. Questions check studentlearning throughout each lesson, and practice exercises on specific case prob-

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY26

TABLE 2. Alternative Instructional Strategies

Traditional Course Web Technology Strengths Weaknesses

Appointment with Email, phone call or Shy students are more likely to use None, students may still phoneteacher or phone call appointment email. or meet with the instructor.Audio content Audio Hear actual sounds, user can reply Requires sound card and

as needed. speakers.Case Case Real world examples enhance Small group interaction isStudies/Stories/PBL Studies/Stories/PBL generalizability and retention. difficult online. Faculty

monitoring and encouragingparticipation is necessary.

Classroom Listserv Sends email to all students, archives Large number of messages canannouncements all activity, be overwhelming,

asynchronous communication. asynchronous communication.Classroom WebBoard Tool for online discussions, archives Requires students to learn anddiscussion discussions for review and analysis, use another tool, asynchronous

asynchronous communication. communication.Filmstrip or video Animation or Demonstrate complex concepts or Expensive to produce.tape digitized video actions, user can replay as needed.Glossary/Index Glossary/Index Hypertext links to definitions, links to None.

external sources, can be printed.Homework or in- Practice exercises & Practice, able to repeat. Self-discipline required.class exercises self-assessmentOffice Hours Chat, audio/video Ensures critical access to the course Speed (modems often too slow).

conferencing faculty. Equipment (not everyone has).Student Journals Student Journals Allows students to record thoughts and Recording online may feel

impressions about material. awkward, concerns aboutprivacy.

Textbooks, copied Online text or printed Online materials are inexpensive Printed materials are morearticles, handouts materials and easily updated--students can familiar.

print out materials on their own.Textbook Online exercises or Evaluate understanding and mastery Can become the focus ofquizzes/exercises, self-assessment of material, immediate feedback. learning (only study what isback exams going to be on the exam).

lems or reports in the literature encourage critical thinking and reflection onthe content. Larger case studies bridge the content among the five sections.These case studies provide students with an opportunity to collect data andsuggest solutions as practicing public health officials would. As mentionedearlier, AISR staff also researched and wrote some of the historical content ofthe course and developed two of the case studies.Concerns regarding equality between the campus-based and the online

groups prompted the use of the same grading requirements for both. Identicalcampus-based midterm and final exams were administered to all students toassure the University administration that both groups had mastered the samecontent to acceptable, measurable levels.

Course Design and Navigation

Recognizing the considerable investment in developing a distance learn-ing course, the University wanted to establish a model design that could beused as a ‘‘shell’’ for subsequent courses. From an instructional design stand-

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 27

point, having a consistent presentation would facilitate student orientation tothe interface and encourage focus on the content rather than on the technol-ogy.The team began the design development by categorizing the types of

course interactions with which students are familiar: communicating witheach other and with the faculty; accessing supplemental print or other mediamaterial; attending to the course content, either through reading a textbook orattending lectures; and completing evaluation exercises. Four navigation ele-ments were created to cover these activities: Communication, Digital Library,Syllabus, and Evaluation. Another navigation bar was added to the top of thescreen, displaying the name of the course and icons for moving forward,backward, and up within the course outline/table of contents. Links to acourse search engine, help, a glossary, and JEFFLINE, AISR’s informationalWeb site (with access to databases, the Library’s online catalog and muchmore) were included in a navigation bar along the bottom of the screen. Thedevelopment team met multiple times with the designer to evaluate variouslayouts and color combinations. Once the general course look was deter-mined, the developer loaded several screens for an evaluation with students,faculty and administration. Additional revisions led to the final course struc-ture shown in Figure 3.

Student and Faculty Communication

Much has been written about the importance of maintaining communica-tion among students and between students and faculty.5 Several courses oncampus already take advantage of e-mail lists to facilitate communicationbeyond the classroom. Previous experience with faculty and students usinge-mail convinced us this is a powerful addition to any course. E-mail givesstudents time to reflect on course content and to formulate higher-level ques-tions they may not think of during a lecture. It also provides an opportunityfor shy students to ask questions privately. The distance course design in-cluded easy e-mail links to the instructor and each student as well as a courselistserv and archive of listserv discussions. In addition, we included an elec-tronic bulletin board within several of the exercises and case studies toencourage students to discuss their opinions and share thoughts about theassignments.

Progress Monitoring

The literature also suggests distance learning students are sometimes lostin the content and may require on-going encouragement to stay on schedule.6

‘‘The constant challenge in navigation system design is to balance this flexi-

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY28

FIGURE 3. Course Homepage

bility of movement with the danger of overwhelming the user with too manyoptions.’’7 We developed a rough schedule outlining when the major coursesections should be completed. In addition, we used a combination of Web anddatabase technologies to help students track their own progress. When stu-dents log onto the course the database sends a ‘‘cookie’’ to the desktopbrowser software. The cookie holds information about the student and sendsinformation back to the database about his or her movement within the onlinecontent. Checkmarks are placed on the table of contents as the student com-pletes each activity, providing a visual reminder to the student of the sectionshe or she has completed (see Figure 4).Relevant feedback from faculty is critical to keeping some students moti-

vated. Therefore, a Web page was designed which allows the faculty andAISR staff to check individual student progress (see Figure 5). The instructorcan then contact those students who seem to be falling behind schedule.Faculty can also review all student responses to each question and practiceexercise, using this to gauge the students’ understanding of the material.

TECHNICAL SUPPORT

Technical support for the course was provided through the Library’s Infor-mation Desk. The reference librarians were familiarized with the types ofquestions they might receive and were organized to be the first point of

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 29

FIGURE 4. Course Outline/Table of Contents

response. Questions they could not answer were referred to systems or otherstaff, as appropriate. Students could contact the Information Desk by tele-phone, e-mail or voicemail.

COURSE DELIVERY

Discussions with the course faculty and concerns about responding quicklyto any unforeseen problems prompted us to limit the enrollment to fifteenstudents. Potential students were solicited through their faculty advisors whoscreened for access to the required computing equipment and interest in adistance learning course. Students meeting the computing requirements andinterested in participating were accepted on a first-come first-accepted basis.A required three-hour orientation session was held on campus to introduce

the students to the course design, layout, and to verify they could log on,navigate through the course sections, send and receive e-mail, and use theelectronic bulletin boards. Library services available to distance students andlibrary contacts for unanticipated distance needs were reviewed as well.Though the students had rated their computing skills high relative to theirpeers, we felt a detailed orientation session, with explicit exercises to demon-strate skills, was important so that we could be sure that the students wereadequately prepared to begin their course work. In addition, a study guidedeveloped by Library staff was distributed and reviewed. Students left thesession feeling confident they could use all components of the system; how-ever, later in the course a few had problems that required additional help.A great deal of planning went into the technical aspects of course delivery.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY30

FIGURE 5. Report for Monitoring Student Progress (Student IdentificationHidden)

Even so, unanticipated problems arose. In the first days of the term wediscovered that the same server used for this fifteen graduate student distancelearning course was used by 410 campus-based undergraduate students aspart of a health care informatics course exercise. The unusually high demandcreated long wait times for information moving between the server, the data-base, and back to the student. On several occasions the server locked up, asthe number of users trying to access it simultaneously exceeded the number itwas capable of handling. This was extremely frustrating to the distancestudents. Luckily, changes to the database tables and load balancing of theservers quickly resolved the problem.Course faculty monitored student progress and sent out encouraging notes

to the students. Several students ran into technical problems, most oftenresolved by upgrading the browser or changing browser settings. Part waythrough the course news stories about viruses and privacy on the Internetprompted some students to change their settings to refuse to accept cookies.Without the cookies, each time the students attempted to submit informationthe program forced them to log back on. Changing the cookie setting to ‘‘askbefore accepting’’ let students control who was allowed to place cookies ontheir systems.The course listserv was used for both announcements and discussion. The

instructor and the Library staff posted messages either to inform students orto prompt them to reflect on the application of their course work to everydayliving. For example, during the course, numerous articles and television

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 31

shows about epidemiology were published or aired. Notices and commentsabout these were sent to the course listserv with the intention of showingstudents the relevance of epidemiology to their daily lives. The number ofthese messages was perhaps six in the first few weeks of the course, a numberneither the instructor nor the Library staff considered excessive. We weresurprised to learn from a mid-course survey that students felt overwhelmedby the number of course e-mail messages they were initially getting and thatit was difficult for them to determine which messages were important. Impor-tance, as defined by the students, meant what would be on the exams.Though students seemed to pace themselves well up to the midterm exam,

there was nearly a two-week hiatus following the midterm with little activity.Concerned that students would fall behind and have difficulty completing thecontent before the final exam, the instructor sent a note to encourage thestudents to attend to their course work. The second half of the course hasmore advanced content and requires students to use a spreadsheet to produceseveral statistics and an epidemic curve. Despite the delay in returning towork all students completed the course material before the final exam.

COURSE RESULTS AND COMPARISON TO CAMPUS GROUP

Survey instruments were prepared to assess the distance course and tocompare characteristics between the students taking the course online andthose taking the traditional lecture course. All students were asked to fill outquestionnaires before the start of the term, midway through the term, andafter the completion of the courses.

Pre-Course Survey

Students were asked about their familiarity with various computer-relatedactivities, how much time they expected to spend on course work, and howcomfortable they were with various epidemiological concepts. Responseswere generally within what we anticipated. Those in the distance course ratedthemselves higher in their comfort with computer-related tasks than did thosein the lecture course. However, we were somewhat surprised at some distancestudents’ lack of computer familiarity, despite pre-course counseling. Somedid not know what browser they used or what e-mail software; some were notsure what a browser was. One had purchased a computer, but it had not yetbeen delivered.

Mid-Course Survey

Distance students were surveyed midway through their course, and librarystaff followed up the surveys by telephone. Survey questions centered around

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY32

how they felt the course was going, if it was meeting their expectations,levels of satisfaction with each course element, if they had encountered anyproblems, and how they were taking notes. At the midway point 92 percent ofthe students enrolled in the distance learning format were satisfied or ex-tremely satisfied with the course. No students indicated they were extremelydissatisfied. The chart in Figure 6 shows the distribution of student satisfac-tion.

Course Results

Fourteen of the fifteen students completed the course (one withdrew fromthe university to help her family relocate when her husband changed jobs).Quantitatively there was no significant difference between the distance learn-ing and campus-based groups. Course scores are presented in Table 3.The no-significant difference result is not unusual in alternative education

programs.8-10 Years of research studies designed to compare traditional in-struction and different computer-based learning models have suggested thatthe actual method of delivery may matter very little in the final outcome ofstudent learning. Perhaps the most reliable variable for predicting studentresults is involvement with the content. Time involved with the content ortime interacting with the content seems most related to student success. Thegoal of instructional design is to encourage the systematic application ofdocumented educational events.11-12 Gagne identified nine instructionalevents that when used can lead to predictable instructional outcomes.1,13 Thatinteraction can come from a good teacher or a good software program.

Post-Course Survey

In general, faculty and most students expressed their satisfaction with boththe lecture course and the distance learning course. Interestingly, students in

FIGURE 6. Student Satisfaction Midway Through Course

NeutralUnsatisfied

ExtremelySatisfied

Satisfied

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 33

the distance course indicated that the course took more time than they hadanticipated, while the students in the lecture course spent less time on theircourse work than they had anticipated.The distance students were, again, generally either satisfied or extremely

satisfied with the course overall. In fact, the proportion of those indicatingthey were extremely satisfied increased in the second half of the term. Figure7 shows the student responses at the conclusion of the distance course.Particular attention was paid to the two students who indicated they were

dissatisfied with the course. One had decided that distance learning wassimply not for her, that she needed the face-to-face contact with the professor.

TABLE 3. Course Grades for Campus-Based and Distance Learning Class(4.0 = A)

Measure Campus Distance

Mean 3.4 3.5

Median 3.3 4.0

Mode 3.3 4.0

Standard Dev. 0.4 0.6

Minimum 2.3 2.7

Maximum 4.0 4.0

Variance 0.12 0.33

Ho = 0

P = 0.41

FIGURE 7. Student Satisfaction After Completion of Course

Unsatisfied

Neutral

Satisfied

ExtremelySatisfied

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY34

The other had anticipated trying to complete the course work within her normalworkday, at her workplace computer, and found this untenable. For these twostudents a more extensive screening process might have been helpful.Distance learning is clearly not suitable for every student. It requires a

self-directed learner with enough discipline to interact with the content regu-larly. The student must be comfortable without face-to-face instructor sup-port, which is an adjustment for most students. Students must also understandthat distance courses are rigorous academic offerings that will require asmuch if not more time and effort than a traditional course. Distance coursescannot effectively be squeezed into an already overloaded schedule.

PLANNING FOR THE FUTURE

The first offering of the epidemiology distance course was a learningexperience for AISR and the instructor, as well as for the students. Ourexperiences prompted us to make several changes for future offerings. Pri-mary among these was the development of a series of online exercises re-quired of all students prior to the start of the course. A course ‘‘Demo’’ oroverview, an online quiz, and an FAQ were also developed to assist students inunderstanding the nature of online learning <http://aisr.lib.tju.edu/Education/distance_learning/>. All of these were designed by the education serviceslibrarian and were implemented by the computer-based learning developer.All registered students are now instructed to read this material and completethe exercises before the start of the term. Students considering distance learn-ing are encouraged to peruse the material as well. The education serviceslibrarian monitors the submission of these exercises and corresponds with allstudents prior to the start of the term to make sure students have installed thenecessary plug-ins and have demonstrated the necessary skills required forparticipation in the course.The time spent both developing and monitoring these materials was signif-

icant. Development and implementation required about seventy-five hoursspread over six months. Monitoring the exercises submitted by the incomingclass required about forty hours over two weeks, with additional hours for‘‘stragglers.’’Initial indications have been that the time invested in developing and

implementing these materials was extremely well spent. Students began theonline epidemiology course in the fall of 1999 better acclimated to learningonline and had far fewer problems to resolve while commencing their coursework.Several changes were made in the course itself and in our orientation for

the students. For example, an online calendar with recommended weeklyassignments was implemented and during the orientation we emphasized the

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 35

importance of staying on track, warning the students that the course contentbuilds in difficulty. This seems to have been effective in encouraging studentsto better pace themselves and avoid panic near exam time. A more detailedpacket was distributed to reinforce information covered in the orientationsession and to increase the students’ comfort with the online material. Whileall information in the packet was available online, the first-year studentsindicated they would have liked additional printed materials.After the development of this course, AISR staff was forced to reevaluate

and redefine its role in course content development. While it was fascinatingto research and write several content pieces, it was decided that the Divisioncouldn’t provide staff to contribute the amount of original content that wascontributed for the epidemiology course. AISR’s future content roles will bebetter aligned with the division’s information literacy goals, with the courseinstructor responsible for the other content.Changes were also instituted for technical support of distance courses.

Students are now directed to contact the Library’s Learning Resources Centerfor support, rather than the Information Desk. The Learning Resources Cen-ter has extended staffed hours as well as staff experienced in solving a widevariety of computer difficulties. Again, initial indications are that this adjust-ment has worked well, both for the students and for AISR staff. Technicalsupport closer to twenty-four hours a day, seven days a week remains achallenge for the future.Other support issues challenge us as well. For example, how can we

provide as many library privileges as possible to our distant students? AISRhas taken an aggressive initiative to provide as many online services aspossible. Even before the initiation of our first distance course, students couldaccess our catalog, databases, interlibrary loan forms, full text for manymaterials; receive articles by fax or mail; and renew materials online or overthe phone. However, they still must appear in person to check out materials.To help offset this, distance students are allowed a number of documentdelivery requests free-of-charge.One large and particularly thorny issue is how various university functions

will accommodate the distant learner. For example, students cannot currentlyregister online. When they do register in person or by mail, the software usedby the registrar’s office cannot capture an e-mail address. While the Univer-sity provides an e-mail address for each student, some do not wish to use thisaddress for their distance courses, and, in fact, some do not even check thisUniversity account. Clearly, there is still much to do to. A campus-widecommittee, on which the University Librarian serves, is wrestling with thelarger picture of how distance learning will fit into the University infrastruc-ture, how costs and revenue will be shared, and how support will be provided.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

MEDICAL REFERENCE SERVICES QUARTERLY36

CONCLUSION

There are a number of ways a university can enter the distance learningmarket. Thomas Jefferson University decided it had the in-house expertise todevelop course software and student tracking software that were commer-cially unavailable at the time. We were aware at the outset that the cost todevelop our own program could be significantly higher than purchasing acommercial product. We therefore kept detailed accounts of the time investedto help determine how to proceed with future courses.Our foray into distance learning was very successful. We have been grati-

fied to see that not only did the majority of students say they would takeadditional distance learning courses, but many already have. Since the firstcourse was offered in 1998 two additional courses have been released. Stu-dents from the first course have registered for each of these, and they haveroutinely inquired about the online availability of even more courses.The roles and responsibilities of the Library and its staff have included

spearheading the project, organizing material, researching and writing sup-plementary course content, designing course look and structure, reviewingmaterial for improvement and enhancement, and providing technical support.AISR has had to look carefully at these roles and redefine how best to supportdistant students. Although the organization of the Library within AISR mayfoster unusual librarian responsibilities in developing and supporting dis-tance courses, the goals for which we work fall into traditional librarianship,in a broad sense--we are working to support our faculty and students ineffective knowledge and information management.

Received: November 23, 1999Revised: January 5, 2000Accepted: January 12, 2000

REFERENCES

1. Gagne, Robert M. The Conditions of Learning and Theory of Instruction. 4thed. New York: Holt, Rinehart and Winston, Inc., 1985.

2. More, Michael. G., and Kearsley, Greg.Distance Education: A Systems View.Belmont: Wadsworth Publishing Company, 1996.

3. Cobb, Jeff. The Fear of ‘‘Adequate Substitutes”: Distance Learning Knocks atthe Door of Traditional Business Academe. @cademyonline. 1 (1999). <http://www.academyonline.com/archives/0399/at_issue_0399.htm>. Accessed November 4, 1999.

4. Kellogg Commission on the Future of State and Land-Grant Universities. Re-turning to Our Roots: A Learning Society. New York: National Association of StateUniversities and Land-Grant Colleges, 1999.

5. Oliver, Ron; Herrington, Jan; andOmari, Arshad. ‘‘Creating Effective Instruc-tional Materials for theWorldWideWeb.’’ Presented at AusWeb96: the Second Aus-

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014

Anthony J. Frisby and Susan S. Jones 37

tralian World Wide Web Conference, July 1996. <http://www.scu.edu.au/sponsored/ausweb/ausweb96/educn/oliver/> Accessed: November 5, 1999.

6. Hasebrook, Joachim P. ‘‘Searching theWebWithout Losing the Mind: Travel-ing the Knowledge Space.’’ WebNet Journal 1(1999):24-32.

7. Rosenfeld, Louis, and Morville, Peter. Designing Navigation Systems. In In-formation Architecture for the World Wide Web. Cambridge: O’Reilly & Associates,1998.

8. Schulman, Allan H, and Randi L. Sims. ‘‘Learning in an Online Format versusan In-Class Format: An Experimental Study.’’ T.H.E. Journal 27(1999):54-6.

9. Jackson, Gregg B. ‘‘Does Hypermedia Accelerate Learning?’’ TechKnowLo-gia 1(1999):45. <http://www.techknowlogia.com> Accessed: November 4, 1999.

10. Guy, Julia F., and Frisby, Anthony J. ‘‘Using Interactive Videodiscs to TeachGross Anatomy to Undergraduates at the Ohio State University.’’ Academic Medi-cine 67(February 1992):132-3.

11. Merrill, M. David; Drake, Leston D.; Lacy, Mark J.; Pratt, Jean A. and the ID2Research Group. ‘‘Reclaiming Instructional Design.’’ Educational Technology36(September-October 1996):5-7.

12. Merrill, M. David. Instructional Design Theory. Englewood Cliffs, NJ: Edu-cational Technology Publications, 1994.

13. Gagne, Robert M.; Briggs, Leslie J.; and Wager, Walter W. Principles of In-structional Design. 3rd ed. New York: Holt, Rinehart and Winston, Inc., 1988.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

07:

05 2

2 N

ovem

ber

2014