medical problem based learning supported by intranet technology: a natural student centred approach

International Journal of Medical Informatics 50 (1998) 225–233

Medical problem based learning supported by intranettechnology: a natural student centred approach

Simon Carlile a,b,*, Stewart Barnet a, Ann Sefton a,b, James Uther a

a Department of Educational De6elopment and E6aluation, Uni6ersity of Sydney, Sydney, N.S.W 2006, Australiab Department of Physiology, Uni6ersity of Sydney, Sydney, N.S.W. 2006, Australia

Abstract

In response to the explosion in medical information, there have been considerable recent changes in medicalcurriculum development. The move to problem based learning (PBL) is, in part, a result of these changes. The Facultyof Medicine at the University of Sydney has exploited a WWW based intranet for the development, delivery,management and evaluation of it’s problem based, graduate medical program (GMP). This system has been employedto develop the 72 medical problems that contribute to the first two years of the GMP. The activities of more than400 members of the faculty have been coordinated using the intranet to develop the wide range of resources tosupport learning in the program. Daily management of the curriculum is also enabled using Web site posting ofbulletins, e-mail and ongoing development of technology training. Coupled with the PBL problems is a formativeassessment system that provides questions and feedback that cover the whole range of learning topics. Part of thestudent and staff evaluation is supported both informally and formally through the use of a ‘Feedback’ button oneach web page and web delivered structured formal evaluations, respectively. © 1998 Elsevier Science Ireland Ltd. Allrights reserved.

Keywords: Problem based learning; Graduate medical program; Intranet

1. Introduction

In recent years there have been consider-able changes in the teaching philosophy guid-ing medical curriculum development. In

recognition of the massively increasing infor-mation load associated with current medicalpractice there is a palpable shift in emphasisfrom teaching to learning and from contentto process. This is most obvious in the moveto problem based learning [1–4]. Further-more, the recognition that effective clinicalmanagement is also highly dependent on theco-operative activity of a team has increased

* Corresponding author. Department of Physiology, F13,University of Sydney, Sydney, NSW 2006; Australia. Tel.:+61 2 93513205; fax: +61 2 9351 2058; e-mail: [email protected]

1386-5056/98/$19.00 © 1998 Elsevier Science Ireland Ltd. All rights reserved.

PII S1386-5056(98)00073-2

S. Carlile et al. / International Journal of Medical Informatics 50 (1998) 225–233226

the emphasis on group learning and also onintegrating clinical and preclinical compo-nents of medical programs. In this paper, wedescribe how we have employed a Facultywide Web based intranet exploiting standardWWW technology for the development, de-livery, management and evaluation of theproblem based, graduate medical program(GMP) at the University of Sydney.

2. PBL and autonomous learning.

A goal of the new, student-centred GMPhas been to emphasise self-directed learningto encourage students in becoming indepen-dent in learning and self-reflective so thatthey will continue their education throughouta professional life. The analysis and resolu-tion of clinical problems is central to theprogram. The structure ensures that knowl-edge across disciplines and between academicyears is integrated so that it is more readilyapplied into clinical practice. Central to theweek’s activities are the interactive problem-based tutorials which are conducted in smallgroups with a facilitator. Such a strategydevelops co-operation and teamwork.

Recent studies on the use of IT in educa-tion has revealed a number of interestingcharacteristics of the kind of learning thatthis approach can encourage. Linn [5], de-scribes the autonomous learner as someonewho takes responsibility for their own learn-ing, assesses their own learning and managestheir study patterns to meet their assessedneeds. In many respects, this descriptionmaps well onto the attributes that the newprogram aims to foster in graduating doctors.Indeed, the recognition of ‘when, how andwhy (learners) learn new material [5]’ is at theheart of ‘learning for life’ in a dynamic prac-tice environment. When properly deployedand implemented, IT can reinforce learning

as a social activity by supporting structureddiscussions involving students, teachers andcontent experts [5,6]. Ad libidum access tocore learning materials via IT encourages astudent centred focus as learners can accessthese materials as they are needed rather thanwhen it is convenient for the faculty to de-liver them. In such an environment learningcan be more selfpaced. In class projects, com-puter-based education has been found tomost effective when coupled with analysis ofstudent progress [7]. As well as providing ameans of monitoring student progress, com-puter delivery of formative assessment instru-ments in the PBL context acts to stronglyreinforce ‘‘self evaluation’’ as a core strategyin ‘‘learning for life’’ (see below).

Notwithstanding the potential educationaladvantages of integrating IT deeply into ourcurriculum, we perceived two other impor-tant motivators for this initiative. The grow-ing field of Health Informatics and itsincreased emphasis on IT is an index of howcritical the provision of IT training duringmedical education has become [8]. The com-puter is a ubiquitous tool in the modernclinical environment and encompasses thetherapeutic and diagnostic technologies, med-ical records management, data base search-ing, telemedicine, prescription and medicaldecision support systems [9]. Due to theseefficiencies in information management andthe rapidly growing power of these technolo-gies, the application of IT to health care iscertain to increase well into the next century.Thus, there is a strong educational imperativeto produce medical practitioners that are verycomfortable users of this technology. Such alevel of sophistication can only be developedby integrating these technologies into the wayin which the students learn. The technologiesshould not be seen as driving the learningprocess but rather integrated seamlessly intothe normal practice of learning so that they

S. Carlile et al. / International Journal of Medical Informatics 50 (1998) 225–233 227

become increasingly transparent to the user.One of our goals has been to introduce thesetechnologies in ways that are likely to emu-late their use in the course of future clinicalpractice. To that end, computers are pro-vided where the students need them (e.g.tutorial rooms, library desks, medical muse-ums, etc.) rather than in large computer labo-ratories to which students go for somecomputer based training component of theirprogram.

3. Infrastructure

For the GMP, the central documentrepository is a web site hosted from the cen-tral campus. This web site also supports webapplications such as the voluntary assess-ment system, web interfaces to calendar ande-mail software, the student/staff feedbacksystem, and web based CAL resources pro-vided from within the faculty. E-mail, onlineforums and other services are also providedfrom the central campus servers, they arebased on internet standards and therefore,may be used from any site with internetaccess.

Web caches are provided both centrallyand at each clinical site to ensure good re-trieval speed for often used information re-gardless of location. There areapproximately 200 PC and Mac worksta-tions both on and off campus. As well asproviding internet access, these workstationsprovide a platform for computer aidedlearning packages deployed to assist with theproblem of the week. To facilitate distribu-tion of such packages and maintenance ofthe machines, an automated software distri-bution mechanism has been implementedbased on ‘Assimilator’ for the Macintosh,and ‘PC-Rdist’ for the NT workstations.

4. Development and management of thecurriculum

The essential elements of the curriculumare the 72 medical problems which act as thevehicle for more than 500 ‘‘learning topics’’.Each topic is associated with text summariesand references provided by content expertsand a range of basic and clinical scienceresources (images, references, web sites etc.)to aid in student learning. More than 400members of the faculty, both academic andclinical teaching staff, have contributed tothese resources. Writing teams were assem-bled around each medical problem and thecontent experts associated with each of thelearning topics have been drawn from variousdepartments and clinical schools.

The contributions of these writing teamswere solicited in electronic form (e-mail or onfloppy disk) and subsequently processed by acentral group set up to publish the curricu-lum documentation using a faculty webserver. The decision to centralise this publish-ing process was motivated principally by theneed to ensure educational consistency. In anorganisation cast along academic departmen-tal lines, the outgoing undergraduate curricu-lum was underpinned by discipline basedteaching. The new curriculum required a syn-thesis of content which was coordinated by arelatively small group of academics and edu-cators. The publication on the faculty webserver of early drafts of curriculum docu-ments ensured that writers drawn from verydifferent academic departments were ap-praised of the context of their own effortsand given the opportunity to provide feed-back to team members and those authoringrelated learning topics. Feedback was facili-tated using a button on each web page thatautomatically generated an e-mail windowaddressed to the author of the current page.The faculty wide publication of the curricu-


lum documents in draft form and the subse-quent peer review has resulted, in generallyvery high quality learning materials in theform of subject summaries and lists of corereferences. Web access for staff to these doc-uments has also been a powerful means ofeducating them about the nature of the newcurriculum.

Two major technical hurdles needed to beovercome early in the development of thecurriculum to facilitate this approach: (1) de-velopment of the requisite IT literacy levels;and (2) ensuring adequate IT access for allstaff contributing to the program. While e-mail was fairly universal among the campusbased academics, the situation in the clinicalschools was far more variable. In some cases,clinical academics worked in hospital areasthat were not connected to the hospital com-munications backbone and in others, thepolicies of the hospital information depart-ments were not conducive to connecting staffto internet services and the WWW. Likewise,the proportion of academics with access toadequate computer hardware was muchgreater on campus than in the clinicalschools. A variety of staff development pro-grams were initiated and a full time telephonehelp desk was set up. In some cases, it wasnecessary to provide telephone modems toclinical teachers who were unable to accessthe faculty servers by other means. Day today management of the curriculum was alsomade possible using the Web site home pageas a bulletin board for urgent or importantchanges in the development process, toprovide notification of the ongoing develop-ment of technology training and help deskpages.

The design specification of key learningelements in the curriculum has been a majorpriority from the outset of the GMP develop-ment and continues to have a major influenceon curriculum management and presentation.

The IT requirements have been determinedfrom the instructional design and the presen-tation of documents and resources have beenstandardised to create specific styles, which inturn have given a consistent identity to thebasic educational elements of the program.This early attention to standardisation wasimportant from the faculty’s point of view tohelp keep track of the merging of traditionaldiscipline based content into the new inte-grated format. The standardisation of thepresentation of learning topic has bench-marked presentation quality and also drawnattention to the need for ongoing contentreview to ensure a continuity in ’style’. Morerecently, it has helped in trialing a range ofways in which students access the learningtopics. In the early phase of the program, thelearning topics associated with the problemof the week were linked to the problem.These learning topics were progressively re-vealed over the problem week together withother learning resources. This strategy wascalculated to reinforce the development ofthe students understanding of the problemswithout compromising the process of prob-lem based learning by revealing too muchdetail regarding the nature of the problem ofthe week.

As the students become more comfortablewith the problem based learning approach,the learning topics are ’detached’ from theindividual problems. The larger ‘data base’ oflearning topics can then be searched at anytime. One clear aim was to ensure that stu-dent learning was progressively independent.As well as acquiring new areas of knowledgein increasing detail, students should also beenhancing various skills (problem-solving; di-recting their own learning effectively; seekingout information) increasingly throughout theprogram. A current variation of this ap-proach uses the style elements as meta tagsfor matching the queries of a search. This has


lead to an analysis of how best to structurekeyword lists as search tags to take intoaccount established medical keywords, local(Australian) medical usage as well as theeasily overlooked search terms. The degree ofsearch success using the current meta tagsystem can be evaluated by both the facultyand students through an automated report,available at the end of each problem week.This automatically generates a list of the ‘‘topten’’ topics for the week in terms of hit rates.Using this checklist, the faculty can assesswhether an appropriate range of topics hasbeen accessed. For students, it provides reas-surance as to whether they have searched asimilar set of information to that found bythe rest of their cohort.

It is important that standardised designsshould not fetter flexibility rather, theyshould provide the building blocks to facili-tate variations in curriculum developmentand management strategies. It also becomeseasier with good design to encourage othersto contribute to the development processwhich has focussed on encouraging depart-mental contributions to the program. Wenow have up to six local departmental siteslinked to the program which provide substan-tial resource collections directly relevant toproblems being studied (e.g. Pathology andAnatomy museum collections have been digi-tized). More are planned or are in construc-tion, largely motivated by the fact there is aclearly defined role and ‘‘fit’’ for the resourcesthat are being generated.

5. Delivery

In terms of intranet delivery, the high vol-ume of regular site access required by theGMP called for a very direct and transparentnavigational structure. The home page pointsto resources associated with the ‘current’

PBL problem as well as ‘past problems’. Thisis achieved using a framed, constant menubar for choices to match students’ primeneeds and wherever possible, sub menus thatrequired no more than three selections toreach end resources. The navigation bar alsoprovides an entry point to data bases (e.g.Medline and Current Contents), formativeassessment modules, discussion groups and arange of help utilities (Fig. 1). These naviga-tional structures have been applied consis-tently and the result has been simple andreliable access, perhaps unspectacular bysome current internet site presentation stan-dards, however, it adequately serves our edu-cational goals of high quality in both contentand process.

Each PBL problem has a number of learn-ing topics associated with it which are linkedto the problem page with further pointers toother related resources (see http://www.gmp.usyd.edu.au/visitors/index.html).These include an audio-visual trigger to in-troduce the problem, patient examinationand test results, patient data sheet providinghistory and other details, lecture summaries,summaries of other teaching sessions and

Fig. 1. Screen shot of the home page of the GMPshowing the navigational and main frame components.


pointers to relevant web sites. Most of theresources are linked to the problem by thepatient data sheet which describes the infor-mation relevant to the medical case. How-ever, some resources are tied to likelyhypotheses. Coupled with the PBL problemsis a formative assessment system that pro-vides multiple choice questions (MCQs), ex-tended matching questions and modifiedessay questions that cover the whole range oflearning topics for the program. Once an-swered, students receive feedback regardingpossible responses and can elect using theirown password, to receive statistics on theirperformance. They can also post commentsabout the questions to a faculty coordinator.Regardless of the interaction chosen, thisform of online assessment remains anony-mous, so as to safeguard its voluntary nature.It is currently being redeveloped to includegreater feedback possibilities, and to allowquestion writers to review and edit their ques-tions online.

Establishing the relationship betweenlearning design and delivery mechanism wasan early and very important first step for us.The designs used to present the curriculumwere specified to serve the educational pur-poses of the program. In delivering thosedesigns using IT, any compromise to theirintegrity was carefully considered and if feltto be too great, postponed until the technol-ogy provided a better solution. The capacityof IT in this regard is quite reliable. Forexample, we had originally used the AdobePDF format to preserve the original designfeatures of the curriculum documents. Im-provements in the HTML specifications, thesoftware used to generate the HTML fromword processed files and the rendering by theweb browsers allowed us to move back to thenative HTML without any sacrifice in presen-tation quality. This strategy also providedadditional ease and flexibility in presenting

material. Similarly, the use of HTML‘‘frames’’ in the interface design was consid-ered carefully and adopted only when therewas a clear design possibility that could im-prove student access. The key principle ininteracting with IT in this way is to maintaina balance between evolving in a constrainedway, without allowing the adopted technol-ogy to become so embedded as to precludefuture developments. The continual referencepoint for this process is the educational intentof the curriculum.

Following the launch of the course, thereare now two parallel web sites. The studentsite provides all of the learning resourcesdiscussed above. The second is the originalfaculty curriculum development site whichstill acts as a review site for evolving curricu-lum documentation. Applying the same de-sign approach to both the ‘‘student’’ site andthe staff ‘‘curriculum review’’ site-using con-sistent navigational principles and optimisingthe presentation of materials to suit the needsof these two different groups has also been avery positive IT outcome stemming fromearly instructional design decisions.

The ultimate test of good design for deliv-ery is whether it can continue to meet theneeds of the user. There is tension here be-tween providing a secure and consistentframework for information which maintainsthe user’s confidence and responding tochanging needs of both the user and theinformation provider. The GMP has been alesson in developing strategies to maintainthis balance. So far, the original informationstructure is holding up and we have been ableto vary it in response to student’s needs; e.g.the variation in the way the learning topicswere attached or not to the problem of theweek.

To develop the software systems to allowrapid changes to the ways in which wepresent the resources, we have relied heavily


on rapid application development (RAD)tools like scripting languages and WYSI-WYG HTML editors, rather than pushingfor ultimate performance. In a similar vein,the development direction has always fol-lowed the path with the most mature andrelevant tools for the job. On the other hand,we try not to sacrifice educational objectivesif the tools are simply not available. Thelearning topic database and search enginewas developed in-house, because no searchengines that satisfied our requirements wereavailable off-the-shelf. However, in keepingwith our RAD principle, they were written ina scripting language, and utilise some capa-bilities of the document editors used to aidthe process of building the search index. Inpractical terms, this experimentation had tobe conducted within the constraints of anongoing program with prepared fall-backsand safeguards so as not to place any of thestudent body at a disadvantage. Online feed-back from students about the effect of thesevariations on their learning strategies hasbeen critical in making judgements regardingmodifications or continuation.

6. Formative assessment system

Formative assessment is now recognised tobe a responsibility of teachers. That is, stu-dents must be provided with the means ofevaluating their progress, without penalty.Particularly in PBL programs, students areuncertain of the depth of knowledge to whichthey must aspire and are uncertain whethertheir coverage is adequate. A strong forma-tive assessment strategy can offer informationand feedback on the effectiveness of the stu-dents learning. In the long term, it is hopedthat by providing models for such self evalua-tion, students will be encouraged to devisetheir own strategies later in professionalpractice.

In the GMP, there are two distinct formsof self assessment. One entirely voluntary,offers students an on-line bank of questionsmatched to the problem of the week and setat an appropriate level for mastery. They canaccess the questions through the computersat any time and they make great use of thisresource. A second pattern of formative as-sessment, once in each block of related work,is also offered and it is not computer based.Questions are associated with feedback but-tons and comments are directed electronicallyto the original writers of the questions whorespond to the students and attend to anyerror or difficulties that arise. The system isthus capable of rapid responsiveness.

The online formative assessment system iscurrently implemented in Python(www.python.org) as cgi scripts accessing anmSQL database. There have been recentchanges to the specification of the implemen-tation resulting from student and staff feed-back regarding its usability for bothauthoring and accessing questions. This pro-vides us with an opportunity to re-author thesystem in a more maintainable and scalableform using a ServerSide JavaScript imple-mentation with a Sybase database. Majordifficulties encountered in the first versioninclude creating an HTML interface efficientenough to handle entry of up to 5000 ques-tions and the structure to quickly mine infor-mation from many (currently 500000)student answers for feedback to the student.

7. Evaluation

Evaluation of the program by the studentsand staff is solicited in a number of ways. Oneach Web page generated there is a ‘Feed-back’ button which allows comments to beentered ad libidum by staff or students (Fig.1). The feedback system currently sends e-


mail to a particular member of staff whodetermines the best action to be taken (seebelow). The name and e-mail address of thesender is optionally included in this message.There is also a short formal evaluation ofeach problem week by the students which isalso entered into a web page that appears atthe appropriate time. In addition to this formof ongoing evaluation, a number of projectsare under development to facilitate trackingresource usage. Performance on the onlineformative assessment modules also providesinformation about the development ofknowledge for individuals or groups. Thiscombined with data on performance in sum-mative assessment, information from surveys,interviews and focus groups provides a verypowerful mechanism for evaluating the teach-ing efficacy of this method of supportingproblem based learning.

Having set up an ‘‘online’’ system as partof program evaluation, we were not at allsure of how students would respond or howwe would cope with their response. The‘‘Feedback’’ button was designed to be everpresent and encourage comments about theparticular resource that was in the adjacentframe at the time. In fact, in the early weeksof the program, the button was used morelike a panic button by students, to feedbackon the full range of their concerns, whetherabout timetables, computer concerns, contentunfamiliarity or more pastoral matters. Feed-back rates ran between 10 and 30 per day(from a possible 140 students) for the firstfew weeks and then settled down to a regular10 or so per day. General computer concernsdropped away and comments became morespecific about computer usage or resourceaccess and presentation. Students also be-came more involved in other forums for feed-back and we were able to cross-check theareas which were receiving most comment.

The faculty has also had to learn to live withresponses to lecture presentations—at timescritical but usually constructive—immedi-ately after their delivery.

Continuous online feedback from studentsabout their preferred sequences of learning ishelping the site to evolve into one whichbetter supports problem solving, encouragingstudents to develop their own independentlearning strategies and constructed knowl-edge. The faculty’s view of the program ispreserved on the separate review site andused for ongoing curriculum development.This evolutionary process, plus the quick re-sponse time of the e-mail feedback system,has given the site an educational life of itsown. In the future, it could well become avaluable resource for understanding some ofthe processes of learning itself.

References

[1] A.J. Sefton, Australian medical education in a timeof change: a view from the University of Sydney,Med. Educ. 29 (1995) 181–186.

[2] R.L. Henry, Curricula and courses—Implementa-tion of a philosophy at Newcastle, Australia, Ann.Community-Oriented Educ. 7 (1994) 79–92.

[3] D.C. Tosteson, S.J. Adelstein, S.T. Carver, Newpathways to medical education, Harvard Univer-sity Press, Cambridge, MA, 1994.

[4] R. Henry, K. Byrne, C. Engel (Eds.), Imperativesin medical education, Faculty of Medicine andHealth Sciences, University of Newcastle, New-castle, N.S.W., 1997.

[5] M.C. Linn, Cognition and distance learning, J.Am. Soc. Inf. Sci. 47 (11) (1996) 826–842.

[6] P. Bell, E.A. Davis, M. Linn, The knowledgeintegration environment: theory and design,CSCL, Proceedings, 1995, p. 14–21.

[7] M.C. Linn, H.C. Clark, How can assessment prac-tices foster problem solving?, in: D.R. Lavoie(Ed.),Towards a cognitive science perspective forscientific problem solving, National Associationfor the Research in Science Teaching, ManhattanKS, 1995, pp. 142–180.


[8] R. Haux, F.J. Leven, Twenty years Medical Infor-matics Education at Heidelberg/Heilbronn: Evolu-tion of a specialized curriculum for medicalinformatics, Methods Inf. Med. 33 (1994) 285–289.

[9] S. Carlile, A. Sefton, Healthcare and the informa-tion age: implications for medical education, Med.J. Aust. 168 (1998) 340–343.

.

medical problem based learning supported by intranet technology: a natural student centred approach

Documents