bibliometric indicators of university research performance in flanders

Bibliometric Indicators of University ResearchPerformance in Flanders

Herman Van den Berghe and Josee A. HoubenResearch Co-Ordination Office, Catholic University of Leuven, Universiteitshal, Naamsestraat 22,B-3000 Leuven, Belgium. E-mail: [email protected]

Renger E. de Bruin and Henk F. MoedCentre for Science and Technology Studies (CWTS), University of Leiden, P.O. Box 9555, 2300 RB Leiden,The Netherlands. E-mail: [email protected]

Andre KintOffice of the Rector, University of Ghent, Sint-Pietersnieuwstraat 25, B-9000 Ghent, Belgium

Marc LuwelMinistry of the Flemish Community, Science and Innovation Administration, Boudewijnlaan 30, 1000 Brussels,Belgium. E-mail: [email protected]

Eric H. J. SpruytOffice of the Rector, University of Antwerp (UIA), Universiteitsplein 1, B-2610 Antwerp, Belgium.E-mail: [email protected]

During the past few years, bibliometric studies were con- policy. Both limit the possibilities for financing scientificducted on research performance at three Flemish uni- research. In Belgium, universities receive a basic allow-versities: The University of Ghent, the Catholic University ance in order to organize education and to provide basicof Leuven, and the University of Antwerp. Longitudinal

research facilities. In spite of a significant increase in theanalyses of research input, publication output, and im-number of university students, the basic allowance hadpact covering a time span of 12 years were made ofbeen decreasing in real terms since 1975. Partly as thehundreds of research departments. This article outlines

the general methodology used during these studies, and result of the rigid personnel structure, the allocation ofpresents the main outcomes with respect to the faculties funds for research lessened consistently. In this situation,of medicine, science, and pharmaceutical science at the evaluating research became imperative. The restructuringthree universities involved. It focuses on the reactions

of the university system, resulting from the constitutionalof the researchers working in these faculties and of thereform of the Belgian state, reinforced this need evenuniversity evaluation authorities on the studies.further.

In 1990, the University of Ghent (RUG), was the first1. Introduction Belgian university which decided to conduct a systematic

evaluation of its research performance. This bibliometricUntil the late 1980s, very little attention had been paidstudy was completed at the end of 1991. It provided anto the efficiency of the Belgian research system. However,assessment of the research activities at the faculties ofstagnating expenditures regarding higher education on themedicine and science during 1980–1989 (de Bruin, Kint,one hand, and the growing number of students on theLuwel, & Moed, 1993a). The model for this study wasother hand, both constituted a serious problem for sciencethe measurement of research performance at Leiden Uni-versity in the Netherlands, finished in 1983 (Moed,

Renger E. de Bruin’s present address is Department of History, Burger, Frankfort, & van Raan, 1985). A basic character-Centraal Museum Utrecht, P.O. Box 2106, 3500 GC Utrecht, The Neth-

istic of the methodology applied in this study involvederlands.the combination of bibliometric analyses and a validationby the scientists involved. The close interaction betweenq 1998 John Wiley & Sons, Inc.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 49(1) :59–67, 1998 CCC 0002-8231/98/010059-09

1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

the bibliometric analysts and the researchers subjected to foreign rule, by the Spanish, the Austrians, the French,and finally the Dutch. The new country became a central-the assessment also played an important part in this RUG

study. ized unitary state. To meet the demands for autonomy byboth the Dutch- and French-speaking communities, theIn 1991, the Catholic University of Leuven (KULeu-

ven) and the University of Antwerp (UA) decided to unitary state was converted into a federation by the consti-tutional amendments of 1971, 1980, and 1988. In 1988,commission the Centre for Science and Technology Stud-

ies (CWTS), University of Leiden, to conduct a biblio- the responsibility for education and nearly all competen-cies relating to science policy were transferred to themetric analysis, applying the same methodology as the

one used in the Ghent study (de Bruin, Moed, Houben, & regional authorities.Flanders has four fully-fledged universities, as wellVan Den Berghe, 1993b; Spruyt, de Bruin, & Moed,

1996). The bibliometric assessments of the faculties of as two university centers with only a limited number ofdisciplines. These institutions have a threefold mission:medicine, science, and pharmaceutical science at these

two universities were completed in 1993 and in early To provide an academic education, to perform research,and to render scientific service to the public at large. In1994, respectively. These studies were limited to the

above-mentioned faculties for a number of reasons. First, Flanders, nearly all publicly-financed basic research isconcentrated at the universities. The Catholic Universitythe three universities had to finance the studies them-

selves. Secondly, the staff members at the faculties of of Leuven, founded in 1425, is the oldest and largestBelgian university and has 27,000 full-time students. Themedicine were familiar with bibliometric tools, especially

at the KULeuven which has already been using bibliome- University of Ghent was founded in 1816, when Belgiumwas still a part of the United Kingdom of The Nether-tric indicators for its internal management for more than

25 years. Finally, literature databases fairly adequately lands. After Belgian independence, it remained a stateuniversity until 1988. Since then, the institution has beenencompass the leading journals in the fields covered by

these faculties. under the jurisdiction of the Flemish Community, whichgranted it wide-ranging administrative autonomy in 1991.In the meantime, at the RUG, CWTS performed a

study related to the other science faculties: The faculties This university has 18,840 students. The University ofAntwerp is a confederation of three university centers. Itof veterinary medicine, agricultural science, applied sci-

ence, and pharmaceutical science (Moed, de Bruin, van has 8,550 students. The Free University of Brussels(VUB) was founded as an autonomous institution inder Velde, Luwel, & Kint, 1993). Finally, during 1995,

a more detailed analysis was made of the departments 1970. It is the smallest of the four Flemish universitiesand presently has 7,968 full-time students. Finally, theof mathematics and theoretical computer science at the

KULeuven and at the UA (Korevaar et al., 1995). two university centers are Limburg University Centre atDiepenbeek and the Catholic University of Brussels.In these five studies, bibliometric analyses were made

of the research activities of more than 400 research groups In 1991, the Flemish Parliament, being responsible forhigher education, approved a law which thoroughly re-covering all these fields of science and during a time

period of 10 to 12 years. There were interactions with vised the rules and regulations for universities. It consid-erably increased their already substantial autonomy andhundreds of scientists working at the faculties involved,

as well as with representatives from 10 different faculty introduced a new funding scheme for these institutions.Each university now receives a basic annual allowance inadministrations. In this article, we will present a synthesis

of the main conclusions reached in the first three studies, order to organize education and to provide basic researchfacilities. Since 1991, this block grant has only partiallythose related to the faculties of medicine, science, and

pharmaceutical science at the three above-mentioned uni- been based on the number of registered students. Approxi-mately 80% of the basic allowance covers personnelversities. We applied bibliometric indicators derived from

the Science Citation Index (SCI), produced by the Insti- costs. In spite of a significant increase in the number ofuniversity students, the basic allowance has decreased intute for Scientific Information (ISI) . In addition, we will

briefly introduce the university system and the research real terms over the past 20 years. Partly due to the rigidpersonnel structure (80% are appointed for life, evenevaluation procedures in Flanders. We will not discuss

methodological or technical issues, but instead will pres- though the retirement rate is relatively low due to massiverecruiting at the end of the 1960s and the beginning ofent the main outcomes, as well as focus on the views and

perceptions of the scientists subjected to the bibliometric the 1970s) , the allocation for research has decreased con-stantly. In order to develop its own research policy, toanalyses and of the university evaluation authorities. In

addition, we will review the effect of the studies on the strengthen existing centers of excellence, and to stimulatethe formation of new research groups, for example, eachscientific communities at the various faculties.university receives a rather small research grant, whichconstitutes about 5% of the basic allowance, from the

2. University Research in Flandersgovernment.

External funding of Flemish university research hasFlanders comprises the northern part of Belgium. In1830, Belgium gained independence after centuries of increased substantially during the past decades. In fact, a

60 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—January 1998

1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

recent study (Spruyt, 1995) estimates that approximately external research grants. These were obtained from anever-growing number of national and international agen-66% of Flemish university research is financed by ‘‘exter-

nal’’ funds. The National Science Foundation (NFWO), cies, programs and initiatives. The universities realizedthat a coherent research strategy with clear priority settingan autonomous organization financed by the public au-

thorities and managed by the universities, provides grants and evaluation procedures was needed. The role of theresearch council and its administrative support, namelyand finances positions at the PhD as well as at the post-

doctoral level for university researchers in all disciplines the research coordination office, which was establishedin each Belgian university at the end of the 1970s becameof fundamental research. They are selected on the basis

of an interuniversity competition. more and more important. Its main task was to developand implement research policy at the university level.As in most industrialized countries, at the end of the

1970s, the Belgian government became more aware of In the process of allocating the university’s own re-search budget, the research council evaluates individualthe importance of a strong R&D potential in order to

sustain economic development. Although the total amount research groups applying for a grant. It uses various pa-rameters upon which to base its judgment, such as theof Belgian public expenditures for R&D remained

roughly constant at 0.5% of the Gross Domestic Product number and the importance of contracts previously ob-tained; the number of PhD degrees; awards obtained in(GDP) during the 1980s, which is rather low compared

with other developed countries, a series of initiatives were Belgium, as well as abroad; and, in some cases, bibliome-tric indicators based on publications and citations. Thesetaken to strengthen the collaboration between industry

and universities, and to stimulate research in areas consid- evaluations reflect the presence of an evaluation culture.However, they were and still are performed only for thoseered critically important in the future. For disciplines such

as biotechnology and artificial intelligence, specific pro- groups applying for an internal research grant, and muchless systematically at the level of faculties and their de-grams were set up and managed by the Ministry for Sci-

ence Policy. Moreover, public authorities actively stimu- partments.lated universities to participate in international R&D pro-grams, such as the EU Framework Programme.

4. Data Collection and Methodology

The application of bibliometric indicators in the assess-3. Gradual Introduction of an Evaluation System

ment of scientific research is based on the following basicin Flemish Science Policy

assumptions. Scientific progress is achieved by research-ers who come together in order to study specific researchIn contrast with most member countries of the Organi-

zation for Economic Cooperation and Development topics and who build upon the work done by their col-leagues (Price, 1963). In this way, an international re-(OECD), the evaluation of research in Belgium remained

limited to peer review of research proposals until the end search community of scientists is created, who keep eachother informed on the scientific results they obtain. Theseof the 1980s. Neither the institutions themselves nor the

government paid much attention to investigating research results need to be published and are thus submitted forevaluation to professional colleagues (Merton, 1972). Incarried out in various sectors. Belgium’s rather compli-

cated institutional structure was undoubtedly the de- the notes and references included in their research articles,researchers refer to the previous work of others whichtermining factor to explain this quite peculiar situation.

The constitutional reform of 1988 was a turning point, the former are now working on. Consequently, scientificliterature is a reflection of scientific activity, and progressmaking evaluation a primary issue for the science policy

agenda of the Flemish government. The Flemish law con- of science can be studied through an analysis of its litera-ture (Garfield, 1979).cerning universities, approved in 1991, obliged institu-

tions to develop a policy regarding quality assessment in The number of publications is considered to be anindication of the scientific output of a group, while theexchange for a much greater autonomy. They had to set

up procedures for a systematic review of the quality of impact is assessed by using data regarding the number oftimes these publications are cited in subsequent yearsall their activities and to report their conclusions to the

government. The parliament even gave the government (Martin & Irvine, 1983). Impact and scientific qualityare not identical concepts. Impact is one aspect of quality,the authority to independently conduct a quality control

review and to impose (financial) sanctions if universities and relates to the size of the response to a certain part ofthe research project, as reflected in cited references inwould fail to do so or if certain standards were not met.

The universities had to strongly justify their use of scientific articles. Bibliometric indicators are not meantto replace peer expertise, but rather to serve as a supportpublic money and, even more so, their demand for addi-

tional support for academic research. At the same time, tool in evaluation procedures (van Raan, 1993).During the present studies, a close cooperation existedthey were confronted not only with a growing internal

competition for their own research funds, but also with between CWTS members on the one hand, and represen-tatives from the university administrations on the otherthe demand from their researchers to provide them with

more information about and assistance in the search for hand. In addition, scientists from the departments to be

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—January 1998 61

1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

evaluated were involved from the very beginning of the countries for OECD statistics, and was also applied herefor the sake of comparison. It is clear, however, that instudy. At the start, a project team was created with mem-

bers from CWTS and the university administration, and Belgium and Flanders, personnel funded by the basic al-lowance are unable to spend 40% of time on research,the following procedure was applied. In a first step, a

university administration provided CWTS with accurate 20 to 25% being more close to reality.For each department, we determined its productivity,data on scientific personnel in each department. The data

included the names of the scientists, the beginning year defined as the number of SCI articles per full-time equiva-lent spent on research. In calculating this measure, weand ending year of their appointment, the funding source,

and their rank (e.g., full professor, associate professor, applied a fractional counting scheme with respect to thenumber of SCI articles. This meant that if an article wasresearch student) .

In a second step, CWTS created a database containing co-published by scientists from two different departments,it was counted as a half for each department. Moreover,all articles published in journals processed by the Institute

for Scientific Information for the CD-ROM version of in order to obtain a measure of the adequacy of SCIcoverage, we determined the percentage of SCI articlesthe Science Citation Index. The corporate address field

contained the names of the cities where the universities relative to the total number of publications in scientificjournals.were located. In addition, data were collected on the num-

ber of times each publication was cited in journals pro- With respect to impact indicators based on citationcounts, perhaps the most significant measurement was thecessed for the SCI. In the special study on the departments

of mathematics at the KULeuven and the UA, publication impact of a department compared to the world citationaverage in the subfields in which it is active. In order toand citation data from COMPUT MATH was used, a

database produced by ISI and offering a much broader calculate this indicator, we determined the average num-ber of citations per article published by a departmentcoverage of journals in mathematics and computer science

than the CD-ROM version of the SCI. Thirdly, CWTS (symbol: CPP). On the basis of the journal in which thearticles were published, we assigned articles to subfieldsmatched the names of the scientists working in the various

departments with the names of the authors appearing in by applying a method developed by ISI, classifying jour-nals into ‘‘categories.’’ From a special datafile containingthe SCI publication database, and compiled a list of arti-

cles for each department, authored by at least one of its citation counts per journal purchased from ISI, and foreach subfield (or journal category), we calculated themembers. In the fourth step, these lists were sent to the

department heads to be verified by the scientists involved average number of citations per article in that subfield,published by scientists worldwide. If a department was(verification cycle) . The corrections were added to the

database by CWTS. active in several subfields rather than one, we calculateda weighted average citation rate, with weights equal toIn the fifth step, CWTS calculated bibliometric indica-

tors for each department, and drew preliminary conclu- the number of articles in a subfield (symbol: FCSm).Finally, we calculated the (natural) logarithm of the CPPsions at the faculty level. The overall results were dis-

cussed with the university administration and were pre- and FCSm ratio (log CPP/FCSm).sented in a special meeting to all scientists involved. Inaddition, a validation phase was conducted, in which each

5. Resultsdepartment received its own bibliometric results for com-ment, and the project team had discussions with members For each of the involved university faculties of medi-

cine, Figures 1 to 3 show the results of an impact analysis,from a limited number of departments ( typically, 2–4departments per faculty) . The last step involved a final in which the average citation rate of a department is com-

pared to the world citation average in the subfields inreport, consisting of two parts, prepared by the projectteam. The first part was made public, and presented the which it is active. Each circle represents a research depart-

ment. The horizontal axis represents the total number ofresults at the faculty level. The second part was a confi-dential Appendix, containing detailed bibliometric results articles published in SCI journals, and the vertical axis

represents the log CPP/FCSm. With respect to the KU-for each department involved. The scientists received onlydetailed results of their present department. Leuven and the UA, the data relate to 1980–1991, while

for the RUG, the time period considered is 1980–1989.We calculated a range of bibliometric indicators, com-bining data on the number of articles published in SCI Circles above the horizontal reference line represent de-

partments for which the impact (CPP) is higher than thejournals, the number of times these articles were cited,and on the research capacity, measured in full-time equiv- world citation average in the subfields in which they are

active (FCSm). The figures illustrate the impact of a fac-alents spent on research. In order to analyze trends, indi-cators were calculated for the total time period, as well ulty’s research activities in such a way that the position

of the constituent departments is still visible. As such,as annually. In the calculation of the research capacity,scientists who received funding from the basic allowance these figures constitute a synthesis between an analysis

at the micro level (a department) and one at the mesowere assumed to dedicate 40% of their time to scientificresearch. This figure of 40% is used by some European level (a faculty) .


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

FIG. 1. Impact compared to world citation average for the Faculty of Medicine at the Catholic University Leuven.

Table 1 presents the results for all faculties. For each butions to the international research front. As for the CPP/FCSm indicator, Table 1 shows that among the sciencefaculty, it specifies the years involved in the analyses, the

number of research departments included (No. depts) , faculties, the RUG has the highest score. However, thepercentage of departments at this faculty with an aboveand the number of full-time equivalents of time spent on

research per year (FTE res/year) . Next, the number of average impact is 24%, which is lower than the UA score.Apparently, the impact of the Faculty of Sciences at thearticles in SCI journals is given (No. SCI publ) , and the

degree of productivity, expressed as the number of SCI RUG is very high, though it is concentrated in a limitednumber of departments. In pharmaceutical science, thearticles per FTE spent on research, applying a fractional

counting scheme in the publication counts. CPP/FCSm KULeuven has the highest value of CPP/FCSm, followedby the UA, where half of the departments has an impactprovides the impact for all aggregated departments, com-

pared to the world citation average in the subfields in above world average. When one considers productivity,the UA has the highest score both in science and pharma-which a faculty is active. Moreover, Table 1 indicates the

percentage of departments within a faculty with an impact ceutical science, while this university ranks second inmedical science.greater than, or equal to, the world citation average. Fi-

nally, the SCI coverage provides the number of articles The SCI coverage varies between 65 and 80%, and issomewhat higher for departments in the pharmaceuticalin journals covered by the SCI, relative to the total number

of publications in scientific journals. science faculties than in the other two faculties. Table 2presents a list of the journals not covered by the CD-With respect to the faculties of medicine, the KULeu-

ven has the highest overall impact (CPP/FCSm Å 1.4) , ROM version of the SCI, in which scientists of the variousfaculties have published frequently during the time periodthe highest degree of productivity (2.0 SCI articles per

FTE), and also the highest percentage of departments considered. It should be noted that the list includes somejournals which were removed from the SCI during thiswith an impact above world average (43%, contrary to

24% for the other two faculties) . These outcomes suggest period, and one journal which was included only after1991. For instance, the journal Acta Clinica Belgica wasthat the faculty of medicine at the KULeuven is a ‘‘re-

search faculty,’’ successfully making high impact contri- removed from the SCI in 1984, while the Journal of


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

FIG. 2. Impact compared to world citation average for the Faculty of Medicine at the University of Antwerp.

Planar Chromatography was only recently included in Some departments redid the exercise themselves, andthe CD-ROM version of the SCI. Table 2 shows that, even broke down the data to the level of individual re-particularly in clinical medicine, several journals are pub- searchers. These groups and others tried to find and iden-lished by Belgian publishers. We will discuss this issue tify weaknesses, erroneous primary figures, such as thosein more detail in the next section. The table also reveals related to scientific personnel, and possible shortcomingsthat most of the journals used by scientists at the science in the method used.faculties, and not included in the SCI, cover earth science, One of these shortcomings may have been identifiedmathematics, and classical biology. by a number of clinicians. They argued that examining

In the special study on the mathematics departments hemodynamic changes and consequences in patients un-at the KULeuven and at the UA, using data from ISI’s dergoing cardiac surgery would inevitably diminish someCOMPUT MATH, the overall bibliometric results were of its real impact when evaluated within the enormouslysimilar to those obtained in the analysis of articles from broad field of physiology. In technical terms, a worldthose departments included in the CD-ROM version of citation average in the field of physiology as a whole—the SCI. Nevertheless, not only interesting but significant as applied in the studies presented in this article—maydifferences were found for several departments (see Kore- not be a representative figure for citation practices invaar et al., 1995). such a small subdomain. A similar problem arises when

clinical genetic studies are examined in the vast field ofgeneral genetics, which is dominated by more fundamen-

6. Reactions of the University Managementtal, molecular biological approaches.

and of the Scientists InvolvedIn the past, some members of the science faculty often

complained about the relatively large portion of financialAt the faculties and departments at the KULeuven, ameans allocated to the faculty of medicine, even thoughnumber of reactions were received, varying between suchthese means were awarded on a competitive basis. Theseextremes as a shock, depression, anger, as well as rejec-

tion and pride which almost seemed to resemble vanity. complaints seemed to have disappeared after the publica-


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

FIG. 3. Impact compared to world citation average for the Faculty of Medicine at the University of Ghent.

tion of the bibliometric study. Overall there were several At the other universities, reactions of the scientistsinvolved were similar to those at the KULeuven. Articlestotally unwarranted reactions of individual researchers or

leaders of research groups fighting amongst each other, by de Bruin et al. (1993a) and Spruyt et al. (1996) presenta detailed overview of the reactions at the RUG and UA,attacking each other with often partial bibliometric re-

sults. During these conversations, citation figures were respectively. Perhaps the most significant effect of thestudy in all universities was that the scientists reconsid-used as weapons, as absolute and sole parameters, even

though the authors of the final report had clearly warned ered and, in many cases, adapted their publication strate-gies, avoiding low impact journals as much as they could.everyone against such behavior.

TABLE 1. Biblometric indicators per faculty.

No. No. SCI % Dept with SCIUniversity/faculty Years depts FTE res/year publ Productivity CPP/FCSm CPP/FCSmú 1 coverage

MedicineRUG 1980–1989 42 143.5 1,467 0.8 0.9 24 75.7KULeuven 1980–1991 44 178.3 5,366 2.0 1.4 43 70.2UA 1980–1991 34 86.9 1,522 1.1 0.8 24 65.6

ScienceRUG 1980–1989 34 281.0 2,217 0.6 1.7 24 67.3KULeuven 1980–1991 24 285.0 2,619 0.6 0.8 19 70.3UA 1980–1991 25 205.6 2,563 0.8 1.0 31 81.3

Pharmaceutical scienceRUG 1980–1991 11 24.4 383 0.9 0.6 9 80.1KULeuven 1980–1991 6 20.1 342 1.1 1.4 30 75.2UA 1980–1991 8 20.5 383 1.2 1.1 50 78.2

RUG, University of Ghent; KULeuven, Catholic University of Leuven; UA, University of Antwerp.


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

TABLE 2. Journals (not covered by the SCI) in which scientists pub- ogy is applied, especially in the case of individual depart-lished frequently. ments. For larger entities, it may be considered to be

fairly representative of the international impact of theMedicine

research output. The university authorities are interestedTijdschrift voor Geneeskundein updating the studies in the future, possibly with someActa Anaestesiologica Belgica

Acta Cardiologica further refinements in the methodology. They wish toActa Chirurgica Belgica generalize the use of quantitative methods in the assess-Acta Clinica Belgica ment of research performance, and are desperately look-Acta Orthopedica Belgica

ing for a similar instrument, probably more caleidoscopicActa Otorhino-laryngologica Belgicain nature, for the social sciences and the humanities.Anatomischer Anzeiger

Bulletin de la Societe Belge d’OphthalmologieArchives Internat. de Physiologie, Biochemie et Biophysique

7. Concluding RemarksJournal Belge de RadiologieVerhaltnisse der Anatomischer Gesellschaft

It should be emphasized that the results presented inSciencethis article relate to the time span 1980–1991. Some pre-Aardrijkskunde

Annalen van het Koninklijk Museum voor Midden Afrika (Annals liminary analyses have revealed that there have been sig-of the Royal Museum for Middle Africa) nificant changes in the impact of several departments and

Annales de la Societe royale Zoologique de Belgique even faculties during recent years. An update of the studyAnnals of the Belgian Royal Academy of Arts and Sciences

would indeed provide interesting results. It would also beBiologisch Jaarboek-Dodoneaimportant to examine, in more detail, the effect of theBulletin de la Societe Mathematique Belge

Catena studies presented in this article on publication strategiesLecture Notes in Mathematics of the researchers involved.Mededelingen der Faculteit Landbouwwetenschappen van de These studies have, in fact, generated an enormous

Universiteit Gentamount of empirical data related to the publication charac-Micron and Microscopica Actateristics and research performance of scientific institu-Revue de Nematologie

Simon Stevin tions. It would be most valuable to explore these dataTijdschrift van de Belgische Vereniging van Aardrijkskundige further, by addressing policy-relevant questions of a more

Studieen general nature. The resulting answers would provide aPharmaceutical science

greater insight into trends in the functioning of the scienceBelgian Journal of Food Chemistry and Biotechnologysystem in Belgium, or even in the western world in gen-Journal de Pharmacie Belge/Belgisch Farmaceutisch Tijdschrift

Journal of Planar Chromatography eral. As indicated in the sections on the Flemish universityPharmazeutische Industrie system and evaluation procedures during the 1980s andPharmeuropa the beginning of the 1990s, significant changes have taken

place in the funding structure of scientific research per-formed at the universities involved. Such changes tookplace in other European countries as well. External fund-There were many discussions about the appropriateness

of submitting articles to journals not processed for the SCI. ing of research projects running for a limited number ofyears ( typically, 2 to 4 years) has become increasinglyFor instance, it was argued by several scientists that the

‘‘Belgian’’ clinical-medical journals such as Acta Clinica important, while the basic allowance for Flemish universi-ties has stagnated or has even decreased slightly over theBelgica and the Tijdschrift voor Geneeskunde are, in fact,

not international journals, in the sense that the editorial past 15 years.Using the data from our studies, one policy-relevantboards of such journals primarily consist of Belgian scien-

tists and that the articles published are mainly written by research question concerns the effect of these develop-ments on the scientific activities at the research depart-Belgian authors. According to this view, these journals

might indeed contain relevant and high-quality papers. ments involved, particularly on the composition of theirscientific staff and their research performance, and onHowever, such journals do not play an important role in

communication at the international research front, al- how these effects should be evaluated. By addressing thistype of question, the use of bibliometric methods couldthough they have a highly useful national function,

namely the transfer of new insights and knowledge from be broadened from simple assessments of individual insti-tutions to actual examinations of trends in the sciencethe research front to interested audiences in Belgium, such

as clinical physicians. In an analysis of the international policy system in general. We have developed plans toconduct such studies in the near future.impact of scientific work, it is appropriate to exclude such

journals. Regarding the development of bibliometric indicatorsin the social sciences and humanities, the most appro-The university research evaluation authorities consider

the methodology outlined in this article as a valuable and priate approach would be to involve scientists, active inthose scholarship domains, in projects aimed at devel-very important instrument, but definitely not the only one.

However, care has to be taken in the way this methodol- oping a classification system of scientific publications,


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

Martin, B. R., & Irvine, J. (1983). Assessing basic research. Someand aimed at making a systematic start with the inevitablypartial indicators of scientific progress in radio astronomy. Researchrather difficult process of determining the weight of thePolicy, 12, 61–90.

various types of publications authored by scientists active Merton, R. K. (1972). The institutional imperatives of science. In B. S.in those domains. Barnes (Ed.) , The sociology of science. New York: Penguin USA.

Moed, H. F., Burger, W. J. M., Frankfort, J. G., & van Raan, A. F. J.(1985). The use of bibliometric data as tools for university researchReferencespolicy. Research Policy, 14, 131–149.

Moed, H. F., de Bruin, R. E., van der Velde, J. G. M., Luwel, M., &de Bruin, R. E., Kint, A., Luwel, M., & Moed, H. F. (1993a). A study ofKint, A. (1993). De tweede Gentse bibliometrische studie [The sec-research evaluation and planning: The University of Ghent. Researchond bibliometric study at Ghent ( in Dutch)] . Leiden/Ghent: Rijksuni-Evaluation, 3, 1–14.versiteit Leiden/Universiteit Ghent.de Bruin, R. E., Moed, H. F., Houben, J. A., & Van Den Berghe, H.

Price, D. J. de Solla (1963). Little science, big science. New York:(1993b). Bibliometrische profielen van de Katholieke UniversiteitColumbia University Press.Leuven [Bibliometric profiles of the Catholic University at Louvain

Spruyt, E. H. J. (1995). Mogelijkheden tot financiering van onderzoek(in Dutch)] . Leiden/Leuven: Rijksuniversiteit Leiden/Katholiekeaan de universiteit [Possible funding of university research]. In K.Universiteit Leuven.Descheemaeker (Ed.) , Valorisatie van Onderzoek (pp. 73–121).Garfield, E. (1979). Citation indexing—Its theory and applications inBrussels: VVAC/Garant.science, technology and humanities. New York: Wiley.

Spruyt, E. H. J., de Bruin, R. E., & Moed, H. F. (1996). Are bibliome-Korevaar, J. C., Moed, H. F., Luwel, M., Spruyt, E., Houben, J. A., &tric indicators appropriate tools in a young university? Higher Educa-Van Den Berghe, H. (1995). Bibliometrische indicatoren met be-tion Management, 8, 141–154.trekking tot wiskundig onderzoek [Bibliometric indicators with re-

van Raan, A. F. J. (1993). Advanced bibliometric methods to assessspect to mathematical research (in Dutch)] . Leiden/Leuven/Antwer-research performance and scientific development: Basic principlespen: Rijksuniversiteit Leiden/Katholieke Universiteit Leuven/Uni-

versiteit Antwerpen. and recent practical applications. Research Evaluation, 3, 151–166.


1005 / 8N2F$$1005 11-06-97 10:15:00 jasa W: JASIS

bibliometric indicators of university research performance in flanders

Documents