a review of bibliometric and other science indicators and their role in research evaluation

http://jis.sagepub.com/Science

Journal of Information

http://jis.sagepub.com/content/13/5/261The online version of this article can be found at:

DOI: 10.1177/016555158701300501

1987 13: 261Journal of Information ScienceJean King

evaluationA review of bibliometric and other science indicators and their role in research

Published by:

http://www.sagepublications.com

On behalf of:

Chartered Institute of Library and Information Professionals

can be found at:Journal of Information ScienceAdditional services and information for

http://jis.sagepub.com/cgi/alertsEmail Alerts:

http://jis.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://jis.sagepub.com/content/13/5/261.refs.htmlCitations:

What is This?

- Jan 1, 1987Version of Record >>

at BROCK UNIV on September 30, 2014jis.sagepub.comDownloaded from at BROCK UNIV on September 30, 2014jis.sagepub.comDownloaded from

http://jis.sagepub.com/

http://jis.sagepub.com/content/13/5/261

http://www.sagepublications.com

http://www.cilip.org.uk/

http://jis.sagepub.com/cgi/alerts

http://jis.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://jis.sagepub.com/content/13/5/261.refs.html

http://jis.sagepub.com/content/13/5/261.full.pdf

http://online.sagepub.com/site/sphelp/vorhelp.xhtml



261

A review of bibliometric and other science indicators and their role in research evaluation.

Jean King Agricultural and Food Research Council, 160 Great Portland

Street, London WIN 6DT, United Kingdom

Received 30 January 1987

Recent reductions in research budgets have led to the needfor greater selectivity in resource allocation. Measures of pastperformance are still among the most promising means ofdeciding between competing interests. Bibliometry, the mea-surement of scientific publications and of their impact on thescientific community, assessed by the citations they attract,

provides a portfolio of indicators that can be combined to givea useful picture of recent research activity. In this state-of-the-art review the various methodologies that have been developedare outlined in terms of their strengths, weaknesses and par-ticular applications. The present limitations of science indica-tors in research evaluation are considered and some future

directions for developments in techniques are suggested.

1. Introduction ~

.

In recent years policy makers and research

managers have become increasingly interested in

the use of indicators of scientific output. The

background to this development is considered,and in Sections 2 and 3, the need for accountabil-

ity and the current pressures on the peer review

process are outlined. The types of science indica-

tors that may have value in assessment studies are

reviewed in Section 4, and the various methodolo-

gies for generating bibliometric indicators in Sec-tion 5, ranging from well-documented and widelyapplied approaches such as publication counting,citation frequency analysis and co-citation map-ping, to less-well-known methods such as journal-weighted citation analysis, co-word mapping and

bibliographic coupling. The use of records of

patents for evaluation of research at the appliedend of the spectrum is considered (Section 6),followed by a discussion of a group of indicatorscollectively termed ’measures of esteem’ (Section

7). Finally, the relatively few studies that haverelated research inputs to outputs are reviewed inSection 8. Directions for future research are sug-gested that may lead to more widely applicableand more routinely available indicators of scien-tific performance.

2. Accountability in science

Prior to the 1960’s, funding for scientific re-

search in the UK was relatively unrestricted. Re-sources were allocated largely on the basis of

internal scientific criteria (criteria generated fromwithin the scientific field concerned), which wereevaluated by the peer review process. Various

factors have resulted in a need for greater selectiv-

ity in the allocation of funds: an increase in thecapital intensity of research (e.g. the ’sophistica-tion’ factor; the growth of ’Big Science’); expan-ding objectives and opportunities, with many newfields emerging; an increase in collaborative, oftenmultidisciplinary, research projects which requirecoordination; a coalescence of basic and appliedresearch, with much research now of a strategicnature (i.e. long-term basic research undertakenwith fairly specific applications in view); finally,economic constraints require choices to be madebetween different disciplines, fields and research

proposals [73]. The peer review process has thuscome under increasing pressure.

In 1963, Weinberg, anticipating the present cri-sis, suggested that both internal and external

criteria be used in making such choices. Thesecriteria are related to the questions: &dquo;How well is

the science done?&dquo; (internal criteria) and &dquo;Whypursue this particular science?&dquo; (external criteria).While placing greater emphasis on external criteriasuch as social or technological merit, he identifiedtwo obvious internal criteria as: (1) &dquo; Is the field

ready for exploitation?&dquo; and (2) &dquo;Are the scien-

tists in the field really competent?&dquo; [71]. It is this

second question in particular that lends itself to

at BROCK UNIV on September 30, 2014jis.sagepub.comDownloaded from


262

quantitative analysis using bibliometric and otherscience indicators.

3. The peer review system

The peer review system has been criticized on

several counts:

(1) The partiality of peers is an increasingproblem as the concentration of research facilitiesin fewer, larger centres makes peers with no vestedinterest in the review increasingly difficult to find.

(2) The ’old boy’ network often results in older,entrenched fields receiving greater recognition thannew, emerging areas of research, while decliningfields may be protected out of a sense of loyalty tocolleagues. Thus the peer review process may oftenbe ineffective as a mechanism for restructuringscientific activity.

(3) The ’halo’ effect may result in a greaterlikelihood of funding for more ’ visible’ scientistsand for higher status departments or institutes.

(4) Reviewers often have quite different ideasabout what aspects of the research they are assess-ing, what criteria they should use and how theseshould be interpreted. The review itself may varyfrom a brief assessment by post to site visits bypanels of reviewers.

(5) The peer review process assumes that a

high level of agreement exists among scientists

about what constitutes good quality work. who isdoing it and where promising lines of enquiry lie,an assumption that may not hold in newer special-ities.

(6) Resource inputs to the review process, bothin terms of administrative costs and of scientists’

time, are considerable but usually ignored.Various studies have addressed these criticisms.

For example, an analysis of National Science

Foundation (NSF) review outcomes found little

evidence of bias in favour of eminent researchersor institutions [12]. However, a later study, in

which independently selected panels re-evaluatedfunding decisions by NSF reviewers, found a highlevel of disagreement among peers over the sameproposals. It concluded that the chances of receiv-ing a grant were determined about 50 ~, by the

characteristics of the proposal and principal inves-tigator and 50% by random elements i.e. the ‘ luckof the reviewer draw’ [13]. The interpretation ofthis data has been questioned, however, and the

fact that most disagreement was found in the

mid-region between acceptance and rejection em-phasized [38]. In another study, a sample of NSFreviewers and applicants was asked which of twoequally good proposals the individuals thoughtwas more likely to be funded (a) from a well-knownversus a little known institution and (b) containingmainstream versus radical ideas. The prospects ofthe proposal from the better known institute andwith mainstream views were favoured by the

majority of respondents [47]. Several studies of

journal referees also found levels of agreementlittle better than would be expected by chance,while in another report, anonymously-authoredpapers, identical except for their results, whicheither agreed or disagreed with the (presumed)standpoint of the reviewer, were less criticallyexamined and received higher ratings when theyconformed to the reviewer’s point of view [25].

Thus, while the authors of a recent NSF reportreaffirm their belief in the effectiveness of its peerreview process, (henceforth to be termed ’meritreview’ to encompass extrinsic as well as intrinsic

criteria [53]), a number of problems clearly exists.The peer review process remains an essential ele-

ment in assessing the quality of science, but im-provements to the system are warranted. These

might include:(1) The right of reply by researchers to criti-

cisms of their proposals, a system already operat-ing for grant applicants to the NSF (USA) and theZWO (Netherlands).

(2) The use of external peers, from neighbour-ing fields and other countries [32].

(3) Clear guidelines on the criteria to be em-

ployed.(4) The use of objective scientific indicators to

complement the peer review process.

4. Science indicators

Indicators of scientific activity include mea-

sures of research inputs, such as funds, re-

searchers, technical support staff, equipment etc.,and of research outputs, for example contributionsto scientific knowledge, to education and to tech-nology. For these indicators to be used for assess-ment purposes, a comparative approach is re-

quired since, in basic science, no absolute quanti-fication of research performance is possible [34].



263

Comparison may be made between similar re-

search groups or institutes, between countries

and/or over time.

Research inputsThe Organisation for Economic Cooperation

and Development (OECD) has attempted to up-grade and standardize the research input data

which it collects routinely, focusing on R& D ex-

penditures and on people working in R& D [54].However, both these indicators remain prob-lematical. For example, it is difficult to determinethe proportions of university capital expenditures,overheads and central facilities that are devoted toresearch as opposed to teaching or administration.International data are confounded by differingfunding systems and salary levels, fluctuations inthe rate of exchange and variations in the relativecosts of equipment [41].A further problem arises from inconsistencies

in the international classification of research areaswhere these can produce a distorted picture of

research expenditure within a particular field [72].With labour costs accounting for more than halfof total R& D expenditures, and average spendingper researcher for the OECD area, (including sup-port staff and equipment) standing at around

$100000 in 1981 [55], research personnel con-

stitute a major input to R&D. Yet no uniform

definition exists for international comparison, withsome countries classifying researchers by occupa-tion and others by level of formal education. ForR& D periormed by academics, it is difficult to

estimate the proportion of total working time de-voted to research as compared to teaching andadministrative duties. In the UK an earlier esti-

mate of 30%, based on a time-budget survey, hasbeen re-evaluated to around 40% [10,41].

In order to overcome the limitations of the

OECD data, such as those outlined above, the UK

Advisory Board for the Research Councils (ABRC) >recently commissioned a comprehensive interna-tional study of government funding of academicand academically-related research [41]. However,

.. due to the lack of consistent data at a disaggre-gated level, very few studies have attempted to

relate research inputs to outputs. Those that haveare discussed in Section 8. Research output data,on the other hand, have been more readily availa-ble.

Research outputsThese include bibliometric indicators (publica-

tion counts, citations), and for applied research,patents. In addition, ’measures of esteem’ (invita-tions to speak at international conferences, attrac-tion of outside funding and visiting researchers,honorific awards etc), may prove useful in assess-

ing research quality. The requirements for a meth-odology for systematically collecting data on re-

search outputs include that it should be: relativelyinexpensive (less than 1 %, say, of the research

expenditure being assessed); routinely applicable,and capable of focusing on any clearly-definedfield. The data thus generated should be: policy-relevant ; internationally comparable; capable ofidentifying time-trends (over at least 6 to 8 years);capable of disaggregation so that they can focuson individual research groups or research tech-

niques (e.g. theoretical versus practical work, useof different instrumentation); publicly accessibleand easily understood [16]. While no such datacollection system is currently in operation, interesthas focused of late on the potential role of indica-tors of research output in research evaluation andscience policy formation. Groups currently activein this area in Britain include the Science PolicyResearch Unit (SPRU) at Sussex University, theScience and Engineering Policy Studies Unit

(SEPSU) at the Royal Society, an evaluation unitat the CIBA Foundation and the Central PolicyGroup of the Agricultural and Food Research

Council (AFRC). In addition the Science PolicyUnit of the ABRC and the Economical and SocialResearch Council have both commissioned studies

of indicators of research output (e.g. [2,58]). Othergroups which may take an active interest in this

area in the future include the recently formedScience Policy Support Group and the Science

and Technology Assessment Unit at the CabinetOffice. -

5. Bibliometric indicators

Rudimentary bibliometric studies have been

undertaken almost since the beginning of scien-tific publishing and publication and citation countswere being used in the early decades of this cen-tury to assess scientific activity [51]. In the 1960sDe Solla Price’s classical essays [17] on the quanti-fication of science and in particular of scientific



264

output and communication, aroused considerableinterest in the significance of publication distri-butions. However, it was with the development inthe USA of the Science Citation Index (SCI) inthe 1960s and subsequently of the ComputerHorizons Inc (CHI) database that bibliometricdata covering the world’s leading scientific jour-nals became readily available [2,51]. Interest in

bibliometric techniques was further stimulated bythe use of literature indicators of national scien-tific performance in the Science Indicators re-

ports, sponsored by the NSF [61]. There was, inaddition, a growing perception throughout the

1970s of the need for objective measures of scien-tific productivity and merit, for reasons discussedin Section 2.

5.1. PubltCOfLOflS counts

The simplest bibliometric indicator is the num-ber of published articles that a researcher or grouphas produced. For basic research the journal articlewith its accompanying list of citations has alwaysbeen the accepted medium by which the scientificcommunity reports the results of its investigations.However, while a publication count gives a mea-sure of the total volume of research output, it

provides no indication as to the quality of the

work performed.Other objections include:

(1) Informal and formal, non-journal methodsof communication in science are ignored [18].

(2) Publication practices vary across fields andbetween journals. Also, the social and politicalpressures on a group to publish vary according tocountry, to the publication practices of the em-ploying institution and to the emphasis placed onnumber of publications for obtaining tenure, pro-motion, grants etc.

(3) It is often very difficult to retrieve all the

papers for a particular field, and to define the

boundaries of that field in order to make a com-

parative judgement i.e. the choice of a suitable

database is problematical [2].(4) Over the past few decades the number of

papers with multiple authorship has increased

considerably [17,39]. Although this is largely dueto a greater prevalence of collaborative research,the gratuitous conferring of co-authorship is notuncommon [6]. Another recent trend had been theshrinking length of papers, resulting in the emer-

gence of the ‘Least Publishable Unit’. This may beassociated with various factors including fragmen-tation of data [6]. Thus, an awareness of the use ofpublication counts for assessment may encourageundesirable publishing practices [e.g. 26].

These constraints apply equally to citation

analysis, since, in order to be cited, a researchermust first publish. However, studies have shown areasonable degree of correlation between publica-tion counts and other measures of scientific merit,such as funding and peer ranking and some ofthese have been reviewed by Jones [35]. More

recently, a correlation of 0.95 was found betweenthe amount of National Institutes of Health (NIH)funds received and the number of biomedical pub-lications produced 2 years later by 120 U.S. medi-cal schools [44]. Similarly, a peer ranking of USfaculties and graduate programmes (theRoose-Andersen ranking) was found to correlatemost highly with the research ’size’ of the institu-tion, as measured by the number of papers it

produced [4].

S.?. Citation analysis,

This involves counting the number of citations,derived from the SCI, to a particular paper for aperiod of years after its publication. The exactperiod will vary from field to field, since the timelag between publication and maximum number ofcitations received in a year differs between special-ities. One proponent of the methodology considerscitation analysis to be a very useful tool: &dquo;Meta-

phorically speaking, citations are frozen footprintsin the landscape of scholarly achievement;

footprints which bear witness to the passage ofideas. From footprints it is possible to deduce

direction; from the configuration and depth of theimprint it should be possible to construct a pictureof those who have passed by, whilst the distribu-tion and variety furnish clues as to whether theadvance was orderly and purposive. So it is with

citations in respect of the growth of human knowl-edge ; they give substantive expression to the pro-cess of innovation, and, if properly marshalled,can provide the researcher with a forensic tool ofseductive power and versatility&dquo; [15, p.25]. A re-cent UK report on the national performance ofbasic research defines four stages in the citation

analysis of a research field:(1) identification of a suitable field and defini-



265

tion of that field in usable, operational terms;(2) application of the operational definition to

construct a (preferably comprehensive) bibliogra-phy confined to appropriate types of publication(e.g. original papers in basic research journals);

(3) examination of the citation record of eachitem in the bibliography;

(4) analysis of the results [2].However, citation analysis presents a number ofserious technical and other problems beyond thosealready discussed for publication counts.

Reu.sons for citing(a) Citation analysis makes the assumption that

an intellectual link exists between the citing sourceand reference article, but the act of citing is essen-tially subjective and is probably governed by a

combination of normative (i.e. adhering to a set ofuniversal norms) and particularistic (inconsistent,personal) considerations. Cronin [15] reviews 10

different classifications of reasons for citing, mostof which favour the normative view, while one

lists a variety of particularistic citing stratagems,including ’hat-tipping’, over-elaborate reportingand citing the most popular research trends to

curry favour with editors, grant-awarding bodiesetc. Murugesan and Moravcsik [50] have produceda four-fold classification, based on citation con-text analysis which is fairly comprehensive:

(1) Conceptual/Operational( theory)/( method )

(2) Organic/Perfunctory(essential)/(non-essential)

(3) Evolutionary/Juxtapositional(development of idea)/(contrasting idea)

(4) Confirmative/Negative(supports findings)/(opposes findings)

They found that 41% of citations to 30 articles inthe PhysIcs Recnew were in the ‘perfunctorv~ cate-gory. On the other hand, a study of references inthe first 4 volumes of SCience Studies found that

80% were either substantiating a statement or

assumption, or else were pointing out further in-formation [65].

(b) Work that is incorrect may be highly cited.However, it has been argued that even incorrectwork, if cited, has made an impact or has stimu-lated further research efforts, while work that is

simply of bad quality is usually ignored by the

scientific community [22].(c) Methodological papers are among the most

highly cited, which, it is argued, reflects their

considerable ’usefulness’ to science. Conversely,many techniques and theories become assimilatedinto the science knowledge base and their origina-tors cease to be acknowledged (the ’obliterationphenomenon’ ).

(d) Self-citation may artificially inflate citationrates; in one study of psychology papers, 10% ofauthors self-cited for more than 30% of their cita-tions [57].

Datahase limitationsWhile all publications databases are subject to

various weaknesses, such as typographical errors,inconsistent or incomplete coverage of the litera-ture etc., such problems are particularly serious inthe case of the SCI since it is the primary sourceof all citation data. The following factors have

been stressed:

(a) A considerable number of citations may belost in an automated search of the SCI, due to

such problems as homographs (authors with the

same name), inconsistency in the use of initials

and in the spelling of non-English names andciting errors.

(b) There have been substantial changes in thejournal set covered by the SCI, with some journalsdropped and a larger number of new ones added,so that no consistent set exists. Also, since 1977

non-journal material e.g. conference proceedings,books etc. were included in the database [49].

(c) The SCI journal set shows a considerable

bias in favour of U.S. and other English-languagejournals and against journals from the U.S.S.R.

and other countries with non-Roman alphabets[9].

Flt!/d-dependent factors(a) Citation (and publication) practices vary

between fields and over time [49]. For example,biochemistry papers now generally contain about30 references whereas mathematics papers usuallyhave less than ten [22]. A study of one Dutch

universitv’s publications found that 40% of

mathematics articles, while written in English, werenot published in journals but in research reports,and therefore were not covered by the SCI. Cover-age of physics and astronomy publications, how-ever, was over 80% [48].

(b) The probability of being cited is also field-

dependent ; a relatively small, isolated field will



266

attract far fewer citations than either a more gen-eral field or research within a narrower field that

has a wider focus of interest.

(c) The decay rate of citation frequency will

vary with each field, and with the relative impactof the paper, i.e. a little cited paper will become

obsolete more rapidly than a more highly cited

one [19].Some of these problems, and ways of minimiz-

ing their effects, have been summarized by Martinand Irvine (Table 1)

The citation rate of a paper may be considereda partial measure of its ‘impact’ (rather than of its’quality’ or ’importance’), where impact is definedas &dquo;its actual influence on surrounding researchactivity at a given time&dquo; [40]. In this context,citation analysis may be used as one of a numberof imperfect indicators (including publicationcounts, peer review, and highly cited papers) in

Irvine and Martin’s &dquo;Method of cotitergitig partialIlldicators &dquo;. This method is based on the applica-tion of a range of performance indicators to

Table 1

Main problems with the various parUal mdicators of scientific progress and details of how their effects may be minmed



267

matched research groups using similar researchfacilities and publishing in the same body of inter-national journals subject to comparable refereeingprocedures. When the indicators all point in the

same direction, the results of the evaluation are

regarded as being more reliable than those basedon a single indicator like peer-review [34]. Thelimitations of citation counts, their use as only oneof a number of indicators, and the continuingimportance of peer review have also been stressed[23].

High citation counts have been shown to corre-late closely with recognized quality indicators suchas honorific awards, including the Nobel Prize [1].Similarly, significant positive correlations betweenthe aggregate peer ratings of departments or in-stitutes and the citation frequencies of their mem-bers have been reported [e.g. 4]. Also, studies ofnon-aggregated peer assessments of individuals

and of individual papers have been shown to

correlate positively with citation counts (e.g. [37]).Citation analysis has been used to demonstrate

that an individual who had been denied tenurehad performed better than successfull colleagues[1] and to refute the ’Ortega hypothesis’, whichstates that much of scientific progress is due to the

small contributions of a relatively large number of‘average’ scientists [11]. In fact, 15% of papers arenever cited, and the average annual citation ratefor papers that are cited is a mere 1.7 [22].

Citation frequency analysis has been increas-ingly used in the evaluation of science. Nationalperformance in eight major fields was assessed byIrvine et al [33] using aggregate, computer-gener-ated data, while manual citation analysis on severalspecialities was used by the Royal Society to assessUK performance in genetics and solid state physics[2]. Other studies in areas of ’Big Science’ (radioand optical astronomy, high energy physics) havefocused on the performance of particular researchgroups or institutions. Different techniques or in-strumentation and the performance of theoreti-

cians versus experimentalists have also been com-pared (Table 2).

In the Netherlands, [48] and in Hungary [68],research groups within departments or faculties

have been compared using publication and ci-

tation analysis. Where such small groups are con-cerned, extreme care must be taken to recover allrelevant data, since a missing highly cited paperwould greatly distort the results. It has been sug-

Tahle 2

Studies using bibliometnc indicators to assess specialities inbasic science

gested that rates of 99% and 95% coverage for

publications and citations respectively are re-

quired [49]. A further difficulty at the disaggre-gated level is the selection of groups for compari-son with the group being assessed. Ideally, field-

independent indicators are needed for internal

assessment of heterogeneous populations of re-

searchers such as university or research councilstaff. However, no such measures have yet been

developed that are reliable, although there is inter-est in the use of journal ‘impact’ or ’influence’(see section 5.3). Another problem is that, to date,most smaller-scale studies have been made manu-

ally and although this method, which involves

tedious, repetitive tasks, cannot be claimed to beerror-free, it is undoubtedly the most accurate oneavailable at present. If bibliometry in general, andcitation analysis in particular, are to become moreuseful tools in science evaluation and policy, thenmeans must be developed of generating databasesand of retrieving citations on a semi-automatedbasis with minimal omissions and errors. In a

pioneering study of online retrieval of publicationsrelated to chemical oceanography, a French groupemphasizes the need to establish a set of standardsfor writing and indexing papers, which would

facilitate the identification of relevant publica-tions using boolean searches [27].

5.3. lOUT/wi ’Impact’ or ’Influenoe’

Manual citation analysis is labour intensive andsuitable only for small-scale studies. For larger,



268

aggregated data, weighted citation counts based

on the average number of citations each journalreceives, provide an easier if less accurate estimateof citations gained. Garfield’s ’impact factor’ is

the ratio of the number of citations a journalreceives to the number of papers published over aperiod of time. The Journal Citation Report (JCR)gives yearly impact factors for the journals coveredby SCI, calculated as citations received that yearto the publications of the previous 2 years. How-ever, this approach ignores the relatively high cita-tion rates generated by review articles, the greaterweight that arguably should be accorded citationsfrom higher prestige journals, and inter-field vari-ations in citation practices. Computer HorizonsIncorporated (CHI) has developed an ’influencemethodology’ which takes these factors into

account and derives a ’total influence’ indicator

for a journal. This is the product of the ‘influenceweight’ of the journal (the weighted number ofcitations the journal receives from other journalsnormalized by the number of references that jour-nal gives to other journals) and its ’influence per

publication’ (the weighted number of citations

each article, note or review in the journal receivesfrom other journals) [52]. The advantages anddisadvantages of using such weighted citation

counts have been summarized by Rothman [58](see Table 3).

Various studies have used journal influence: forexample in the USA a comparison of peer ratingand the citation influence rating of journals in 10different fields found a strong positive relation-

ship [42]. A forthcoming UK survey of researchperformance in university departments and re-

search council institutes is using the influence

methodology to analyse publications. On a smallerscale, the journal ’packet’ of journals in which

departments published was constructed for uni-

versity research groups in the Netherlands. Usinga ’Journal Citation Score’ (JCS) analogous to SCI’s’ impact factor’ the weighted average JCS valuewas calculated for each group; this was the sum of

the JCS’s for all the journals in which the grouppublished, each weighted according to the propor-tion of the group’s publications appearing in it.

The value obtained represented the ’expected’number of citations per article for that journal set,which could then be compared with the actual

number of citations received by the research group[48]. In the micro-management system of the

Table 3

Comparison of direct citation counting and journal influenceas research productivity measures

Source: (SRI.

Hungarian Central Research Institute for Chem-

istry, papers are scored using (1) the average ofthe impact factors for the relevant subfields, ex-

cluding interdisciplinary journals and (2) the aver-age of 3 years’ impact factors, to take into accountvariations between subfields and over time. Rangesof values for impact factors are then assignedscores from 0.5 to 3, to minimize other distortinginfluences such as differences in the time requiredto produce papers and in journal citation and

publication-time practices [68]. However, the useof the journal influence or impact methodologyfor small-scale, disaggregated evaluations, as in

the Dutch and Hungarian studies, remains con-

troversial, since only approximate measures are

obtained.

5.4. Co-citation anal>’sis

Bibliographic data have also been used to con-struct maps of the structure of science using co-ci-tation (cluster) analysis. This is based on 2 as-

sumptions : (1) that when 2 papers are cited to-

gether by a third paper, then a cognitive relation-ship exists between them and (2) that the strengthof this relationship is proportional to the frequencyof the co-citation linkage (i.e. the number of papersthat co-cite them). Clusters of related papers maybe constructed for a specified threshold of co-cita-tion and the relationships between clusters dis-

played spatially using multi-dimensional scaling.



269

The clusters represent specialities or fields, whilelinks between them reveal interdisciplinary rela-

tionships.Thus, co-citation analysis may be used to map

various features including: the structure of re-

search fields or specialities; communications be-tween fields or specialities and, using time series,the development of active research fronts or thehistorical development of a particular area of

knowledge [21]. In the ISI Atlas of SCience, multi-ple levels of clusters have been used to produce’nested maps’ which provide hierarchical or re-

gionalized structures of large fields such as bio-

chemistry or biotechnology.Recent improvements in the methodology make

allowances for inter-field variations. ’Fractionalcitation counting’ overcomes the problem of theover-representation of high referencing areas suchas biomedicine and the under-representation of

low referencing ones such as mathematics. The

citing paper now carries a total strength of 1

which is divided equally between all its references.Also, whereas previously a single threshold of

co-citation was used for cluster formation often

producing ’megaclusters’, the new ‘ variable levelclustering’ sets a limit to cluster size while formingthem at the lowest possible co-citation level withinthat limit. These techniques improve the disci-

plinary balance of the maps in the SCI file [62,63].Alternatively, the Centre for Research Planning(CRP) has developed a ’stratified model’-

building algorithm. This &dquo; involves a ’top down’

procedure in which the co-citation thresholds usedto qualify pairs of specialities for clustering beginwith the highest one computed from the distribu-tion, instead of the lowest. The loss of specialitiesis avoided by returning all speciality pairs that donot cluster at a particular threshold to the distri-bution from which the next threshold will be

computed&dquo; [58].Used in conjunction with conventional histori-

cal methods, co-citation analysis was found to bea valuable tool for identifying important foci ofintellectual activity (research fronts) in the special-ity of weak interactions in physics [67]. Howeverthe authors criticised the methodology on severalgrounds, including the time lag between the actualinception of a new line of research and the forma-tion of a cluster; the over-representation of theo-retical as compared to experimental papers, due totheir greater ease of production; and the loss of

many relevant papers due to such factors as the

circulation of pre-prints, publication in Russian

journals (not covered by SCI), and the ’obliter-

ation phenomenon’.Similar objections have been raised for the spe-

ciality of spin-glass where co-citation data werecompared with a manually generated bibli-

ography. In the co-citation analysis, the authorswho cited cluster papers formed only a subset ofresearchers in the field. Also, as in the weakinteractions study, relatively few experimentalpapers were included in clusters and there was

some delay between initial research activity andthe appearance of a corresponding cluster. Otherproblems included errors in citations, the inclu-sion within clusters of papers from related butdistinct fields and the subjectivity inherent in thesetting of threshold levels although these levels

strongly affected the size and content of clusters[29]. Some of these criticisms such as time lag andsubjectivity in defining field boundaries, applyequally to citation frequency analysis.

Thus while co-citation analysis has evident use-fulness for social scientists who wish to study thestructure of science, its role in science policy re-

mains controversial. However it has been appliedby the Science Policy Advisory Council (RAWB)in the Netherlands to identify those fields in whichDutch scientists have a considerable international

impact, and both Germany and Australia are tak-ing an interest in the RAWB Study [30]. In Bri-

tain, Rothman looked at corporate activity withinco-citation models for various specialities, byidentifying author affiliations or acknowledgedfunding. He argued that companies usually con-duct or support basic research which they believeto possess strategic significance, and that a spe-ciality with a high corporate activity was thus oneof strategic importance [58].

S. S. Co-word unalvSis ..

This methodology has been developed by theCentre de Sociologie de l’Innovation (CSI) in Paris.It involves analysing papers to identify keywordsthat describe their research content and linkingpapers by the degree of co-occurrence of thesekeywords to produce a ’map index’ of a speciality.Many journals and abstracting services alreadyprovide such keywords. This approach was used ina recent ABRC study of protein digestion in rumi-



270

nants. However, experts who were asked to

evaluate the study expressed considerable doubtsabout the methodology, in particular with regardto the probable inconsistencies in keyword selec-tion by professional indexers working for differentdatabases (the ‘indexer effect’), and whether the

conceptual content of a paper could be reduced toa few words [58]. One of the main advantages ofco-word analysis would seem to lie in its indepen-dence from the SCI database, which has a consid-erable US and English-language bias, and maytherefore cover only partially the research outputof many smaller, non English-speaking countries.Also the considerable time lag associated with

citation-based analyses is avoided and the methodis not limited to scientific publications but may beapplied to all forms of scientific literary outputsuch as technical reports and patent applications,and may therefore be used to investigate appliedas well as basic research.

In order to minimize the weaknesses inherent inmanual indexing, the CSI has developed a com-

puter-aided indexing technique for full-text data-bases called LEXINET in which the controlled

vocabulary is constructed and updated interac-

tively each time a new document is analysed [14].Together with the LEXIMAPPE programme forco-word mapping, this technique is being devel-oped as a science policy tool and has already beenused to analyse publications from research on

dietary fibre and aquaculture; various documen-tary sources, including project proposals, for mac-romolecular chemistry research and patents in thefield of biotechnology. The authors emphasize thatthe method is in a developmental stage and raisevarious technical issues to be resolved [7,8].An Anglo-French study in progress on acid

rain research is using co-word analysis to identify:(a) research ‘ themes’ based on small clusters of

frequently-occurring keywords in papers (maxi-mum 10 keywords);

(b) the strength of the internal links within

themes (when strongly linked, these are said to

represent highly cohesive areas of research whichmay in fact be peripheral to the speciality understudy), and

(c) the number of external links between

themes, which may indicate centrality to the spe-ciality (where there are many links the theme is

deemed to be central to the speciality and i,i’ce-

verso

Thus a strategic map may be drawn up in

which the centrality and internal cohesiveness ofthemes are related in order to identify either

potentially neglected or peripheral themes of re-search. This methodology is termed ‘research net-work management’ [5]. There may therefore be apolicy role in the future for co-word analysis at

the level of determining priority areas of research.

5.6. Bibliographic coupling

This is based on the concept first described byKessler that if two publications share one or morereferences then they are bibliographically coupled.Kessler showed that coupled papers from the

Physics Review were often related by subject andthat the higher the coupling strength (i.e. the

greater the number of references in common), thecloser was the subject relatedness [36]. In a recentvalidation study, a random selection of 10000

articles from the SCI was coupled with papersfrom the entire SCI database and lists of the threemost strongly coupled items were compiled. Pro-fessional indexers were then asked to assess the

relatedness of these papers from their titles, for arandom sample of 300 lists. They found a closesubject relatedness in over 85% of cases whichdecreased to 81% when the frequency of occur-rence of the common references in the entire

database and the length of the reference lists wereboth taken into account [69]. Whether the contentof a scientific paper can be properly ascertainedfrom a perusal of its title is only one of severalissues raised by this methodology. Indeed, there isa lack of any defined method for relating the

conceptual content of different texts. However,the ease of analysing citations to matched sets ofdata, generated by bibliographic coupling make itan appealing approach for science policy studiesand one that warrants further investigation.

Clearly from Section 5, various bibliometric

indicators have already been used in assessmentstudies. Citation frequency analysis has been themost common, and the refined SPRU &dquo;Method of

converging partial indicators&dquo; has been notablyeffective in identifying policy issues [34,16]. Co-ci-tation analysis has mostly been confined to the

structural analysis of scientific fields and awaits

the resolution of various problems before it can

have a more prominent policy role. Newer meth-

odologies, in particular journal influence and bib-



271

liographic coupling, warrant further study becauseof their potential application to smaller or multi-disciplinary research units, while co-word analysismay have applications at a more strategic level.While the emphasis of this paper has been on

bibliometry, since to date most studies have

adopted this approach, patent analysis and mea-sures of esteem may also prove useful.

6. Patent analysis

Patent data may be used directly to indicatescientific and technological activity ie of appliedresearch and development, or the citations in

patent applications and examiners’ reviews maybe used as an additional source of material for the

bibliometric analysis of basic research. Limita-tions to the value of patent data as an indexinclude inter-industry differences in patenting ac-tivity and international differences in conditionsand charges for patenting. Also, not all inventionsare patented or patentable.

However, a study of possible biases in patent-ing activity found a close inter-country correlationbetween foreign patenting and R& D expenditureas well as a close inter-industry correlation be-

tween foreign and domestic patenting. These re-

sults suggest that the concept of foreign patentingis a useful and relatively unbiased addition to theexisting array of science and technology outputindicators [64]. Patent data were used to develop a‘ Revealed Technology Advantage ( RTA ) Index’ in10 OECD countries, showing each country’s rela-tive share of total foreign patents granted in theUSA for each of 41 sectors. Thus an RTA Index

greater than one showed a sector with a compara-tive technological advantage whereas an RTA

index less than one represented a comparativedisadvantage. Innovative success is often associ-ated with good quality research; for examplepatent applications and numbers of citations

received were strongly correlated for a group ofresearchers/ inventors at a US University of

Technology [46].

6.1. Patent Citation Analv’sis

In its study of national performance of basicresearch, the Royal Society used patents grantedin the USA to obtain a different set of citations

from that derived from journal references. Paperscited by both applicants and examiners were

analysed. While applicants were thought morelikely to select their references rigorously, since

they would presumably undergo a more thoroughscrutiny than the average journal reference, ex-

aminers on the other hand were thought to have amore limited literature knowledge base, tending tocite reference books, conference proceedings, ab-stracting journals and existing patents. Althoughpatent citation analysis was not useful for solid

state physics, due to insufficient data, in the fieldof genetics it yielded interesting information oninternational citing patterns, and the cited litera-ture was both recent and basic. The data were

found to be consistent with, and to ’usefully com-plement’ the data obtained from journal to journalcitation analysis, although the methodology wasmore labour intensive [2].

In addition to the use of patent data as indica-tors of research performance, a group of indica-tors collectively termed ’measures of esteem’ re-

lates to the recognition by the scientific commun-ity of work of outstanding quality. These measuresare discussed in Section 7.

7. Measures of esteem

The Royal Society study, discussed in section

5.2, also made a preliminary investigation into theusefulness of two indicators of esteem, namelyinvited papers at international conferences and

migration of scientists, for the evaluation of UKperformance in certain specialities [2].

7.1. ¡m’lled papers at ititei-natiotial conference

International meetings are well-established andimportant channels for the communication of sci-entific information among individuals and be-

tween nations, and are often used for the rapiddissenunation of results before their eventual pub-lication in journals. The growth of the scientificcommunity and of international travel has re-

sulted in a doubling of the number of scientificmeetings held worldwide in the late 1960s to

around 10000 by the late 1970’s [60]. An invita-tion to present a paper at an international meetingimplies that the invited scientist is held in highregard by his peers in the international commun-



272

ity. To identify trends over time, a series of con-ferences held regularly would need to be analysed.The Royal Society analysed invited papers to theCold Spring Harbour Symposia in New York byauthor nationality, and the UK papers deliveredat a series of International Physics Conferences[2]. The major constraint to the use of this indica-tor is the limited number of such conferences.

7.2. Migration

The short-term and long-term migration of sci-entists partly reflects their perceptions of the rela-tive scientific opportunities of the country to whichthey move. While permanent migration (the ‘braindrain’) has in the past received most attention, theshort-term migration of scientists is both more

common and probably more intellectually produc-tive [45]. Thus, an analysis of the numbers, lengthof stay and standing of their home research in-

stitute, for visiting researchers to UK research

groups, and a similar analysis for UK researchersvisiting abroad, might provide useful data on theesteem in which a UK research group is held

internationally.

7.3. AttractIOn of outside funding .

The willingness of outside agencies to supportresearch at a department or institute is a measureof its scientific reputation. This indicator, togetherwith a peer assessment of the top 5 departmentalpublications, was the main method used by the

University Grants Committee in its controversial

grading of research, and interdepartmental alloca-tions of resources are now being made on this

basis at some UK institutions [3].

7.4. Honours and professional status indicators

These include international prizes, membershipof professional societies, editorial boards, reviewpanels etc. Limitations to their use, as with other’measures of esteem’ include (1) the involvementof other factors besides scientific merit and (2)insufficient data. However, universities and

institutes of higher education in various countriesnow combine bibliometric indicators with such

’measures of esteem’ in their evaluation proce-dures, in recognition of the need for publicaccountability (e.g. [20,70]).

8. Input-output studies .

Some of the difficulties associated with quanti-fying research inputs were discussed in Section 4.Very few studies have attempted to correlate in-puts with outputs.A study of aggregate expenditure and publica-

tion data for US universities (allowing a two yeartime lag), found a positive, linear relationshipbetween the two, which was strongest for medicineand weakest for mathematics. There was littleevidence for ’economics of scale’. The relationshipbetween expenditure and research ’level’ (basicversus applied) was less straightforward; howeverin the ’hard’ sciences (physics and chemistry) agreater proportion of basic research tended to beat the largest institutions, a trend probably associ-ated with requirements for expensive equipment.A citation analysis showed some evidence that

research from institutions producing many paperswas more visible’ (ie attracted more citations perpaper) than that from smaller institutions, but thisheld mainly for basic research in chemistry, physicsand biology [43]. Expenditure and publicationoutput two years later were also strongly corre-lated ( r = 0.9), for research funded by the Na-tional Institutes of Health in aggregate; for indi-vidual Institutes, r ranged from 0.7 to 0.95 [24].

In US State Agricultural Experimental Sta-

tions, both publications and citations were signifi-cantly correlated with a series of current and

lagged research expenditures. Long-term, be-

tween-states differences in expenditure influencedresearch performance (with a mean gestation lagof approximately three years for citations), but notshorter-term within-states variations in spending.Controlling for inter-state differences, research

performance was found to be positively related tothe size of the graduate programme and to the

growth rate of the scientific personnel but nega-tively to the degree of fragmentation of the sub-stations (which the author suggests may be relatedto access to shared resources) [56].

In Britain a forthcoming report looks at the

cost of producing scientific papers from Univer-sity departments. Based on the results of detailedquestionnaires, a proportion of the costs of centralservices, overheads and capital expenditures are

allocated to research and a cost per publicationfor different disciplines is derived [10].

At a disaggregated level, the University of



273

Leiden bibliometric study related publications andcitations per paper to the number of researchers

(expressed as full-time equivalents (FTEs), withjunior and senior staff counting as one-half andone FTE respectively). These researcher-normal-ized indicators were used to compare research

groups (ranging in size from 4-15 FTEs) acrossspecialities but within faculties or large depart-ments (using journal impact measures as discussedin section 5.3). The results evoked explanations ofthe observed trends rather than dissent, even when

they were unfavourable. Explanations included:

changes in group leadership; concentration on

’hot’ topics; lack of effectiveness in presentingresults and lack of consensus in the field leadingto discriminatory journal acceptance behaviour

[48]. The authors conclude that this dialogue was amajor outcome of the study, raising several im-portant policy issues.An innovation in micro-level resource alloc-

ation, based on input and output indicators, hasbeen described for the Hungarian Central Re-

search Institute for Chemistry. The input-outputmanagement system was introduced in 1981 to

cover 13 departments and 200 researchers engagedin 3 main areas of research. Researclr outputs wereassessed on the basis of (1) earnings and contractsawarded for mission-oriented research (MOR) and(2) scores for publications, based on type of pub-lications, impact factors of journals (see section5.3), degree of contribution (in the case of multi-ple-author papers) and citations received. Fundlllgto departments was allocated 60~, on the basis ofnumber of researchers and 40% on average pub-lication scores over the past 3 years. The greaterthis latter allocation the less MOR and the more

basic research authorized. Costs to be met from

departments’ total incomes (funding plus earn-

ings) were estimated as follows: costs of salaries

and chemicals; running costs (electricity etc.); 5%of equipment costs (since 50% had to be repaidover 10 years); and a proportion of earnings usedto provide an incentive bonus. The bonus alloc-ation was based on MOR income and earnings,’publication score’ and citations. A peer review

input balanced any distortions in the automatic

assessment system, and measures of esteem, edu-cational activities etc. were taken into account.

The new management system led to increased

publication in international journals and to an

increase in MOR from 30% to 50% of research

activities. Institute staff recognized the need forsuch an internal control system, in the light ofsevere economic constraints; however, some dis-satisfaction was expressed over the assessment ofpublications, mainly due to variations in impactfactors and citation practices between sub-fields[68].

Clearly more studies are needed on research

input indicators, their relationship to measures ofresearch output and impact, and their role in

management systems and policy development.

Conclusion .

My purpose has been to outline the different

types of science indicator and the various method-

ologies that have been used in the assessment ofscientific performance. Most bibliometric studieshave operated at the level of the scientific special-ity or field, and the reliability of results obtainedby applying bibliometric techniques to small num-bers of publications has raised serious doubts, dueto the conceptual and technical problems outlinedin section 5.2. However, many managers are todayconfronted with the need for objective indicators.to complement the peer review process, for smallor medium-sized, often multi-disciplinary, re-

search groups. There is therefore a need to de-

velop reliable, preferably field-mdependent, indi-

cators for use at a disaggregated level-, whether

journal ’influence’ or bibliographic coupling canprovide such indicators has yet to be assessed. Inaddition, there is a need for these indicators to be

generated on a routine basis; a major constraint tosuch a system is the labour-intensive nature of the

currently accepted manual methodology. The de-velopment of online techniques. with minimal andacceptable rates of error, for both publication andcitation retrieval, is urgently required; more con-sistent inter-database formatting and indexing forpublications would greatly facilitate this process.It would also be useful to assess whether samplingsignificantly alters the results obtained from com-plete sets of publications and citations (e.g. takingpeak citation years only), since this would sub-

stantially reduce the volume of data to be han-

dled. In terms of research inputs, there is a need

for consistent and comparable data at a relativelydisaggregated level, to facilitate meaningful input-output assessments. Finally, there is an urgent



274

need for debate among research managers and

policy makers about the role that science indica-tors should play, both in terms of the weight theyshould carry relative to, for example, peer review,and the level at which they should be incorporatedinto the decision-making process.

Acknowledgements

Ben Martin (SPRU), John Lake and Don

Corbett (AFRC) kindly read and commented onthis review.

The opinions expressed in this paper are the

author’s personal views and do not necessary re-flect AFRC policy.

References

[1] S. Aaronson, The footnotes of science, Mosaic 6 (2)(1975) 22-27.

[2] ABRC, Evaluation of National Performance in Basic Re-search : a review of techniques for evaluating nationalperformance in basic research, with case studies in genet-ics and solid state physics, ABRC Science Policy StudiesNo. 1, Advisory Board for the Research Councils, London,1986.

[3] A. Anderson, Research gradings stir emotions, Nature 322(1986) 299.

[4] R.C. Anderson, F. Narin and P. McAllister, Publicationratings versus peer ratings of universities, J. Amer. Soc.

Info. Sci. 29 (2) (1978) 91-103.[5] S. Bauin, J.-P. Courtial and J. Law, Policy and the inter-

pretation of co-word data: notes on the acid rain study,Paper prepared for CNRS/SPSG Seminar on ScienceIndicators and Policy, London, September 17-18, 1986.

[6] W. Broad, The publishing game: getting more for less,Science 211 (1981) 1137-1139.

[7] M. Callon, J.-P. Courtial, W.A. Turner and S. Bauin,From translations to problematic networks: an introduc-tion to co-word analysis, Social Science Information 22 (2)(1983) 191-235.

[8] M. Callon, J. Law and A. Rip, eds., Mapping the Dy-namics of Science and Technology (MacMillan, London,1986).

[9] M.P. Carpenter and F. Narin, The adequacy of the Sci-ence Citation Index as an indicator of international scien-tific activity, J. Amer. Soc. Info. Sci. 32 (1981) 430-439.

[10] K. Clayton, Measuring the cost of research, Times HigherEducation Supplement (12th September 1986) 1.

[11] J.R. Cole and S. Cole, The Ortega Hypothesis, Science

178 (1972) 368-375.[12] S. Cole, L. Rubin and J.R. Cole, Peer review and the

support of science, Scientific American 237 (4) (1977)34-41.

[13] S. Cole, J.R. Cole and G.A. Simon, Chance and consensusin peer review, Science 214 (1981) 881-886.

[14] J.P. Courtial, M. Callon, S. Bauin and W.A. Turner,Co-Word Analysis, Paper presented at the CNRS/SPSGSeminar on Science Indicators and Policy, Science PolicySupport Group, London, September 17-18, 1986.

[15] B. Cronin, The Citation Process (Taylor Graham, London,1984).

[16] D. Crouch, J. Irvine and B.R. Martin, Bibliometric analy-sis for science policy: an evaluation of the United King-dom’s research performance in ocean currents and proteincrystallography, Scientometrics 9 (5-6) (1986) 239-267.

[17] D.J. De Solla Price, Little Science, Big Science (ColumbiaUniversity Press, 1963).

[18] D. Edge, Quantitative measures of communication in sci-ence : a critical review, History of Science 17 (1979)102-134.

[19] G. Folly, B. Hajtman, J. Nagy and I. Ruff, Methodologi-cal problems in ranking scientists by citation analysis,Scientometrics 3 (1981) 135-147.

[20] R. Fransson, Resource allocation based on evaluation ofresearch, Internat. J. Inst. Manag. Higher Educ. 9 (1985)67-71.

[21] E. Garfield, Mapping the structure of science, in: CitationIndexing, Its Theory and Application in Science, Technologyand the Humanities (Wiley-Interscience, New York, 1979)98-147.

[22] E. Garfield, Is citation analysis a legitimate evaluationtool? Scientometrics 1 (4) (1979) 359-375.

[23] E. Garfield, Uses and misuses of citation frequency, Cur-rent Contents 43 (1985) 403-409.

[24] H.H. Gee and F. Narin, An Analysis of Research Publica-tions supported by NIH 1973-76 and 1977-80, NationalInstitutes of Health Evaluation Report, NIH, Bethesda,MA, 1986.

[25] M. Gordon, Evaluating the evaluators, New Scientist (10February 1977) 342-343.

[26] D. Greenberg, Fraud and the scientific method, NewScientist (6 November 1986) 64.

[27] P. Hassanaly and H. Dou, Information systems and scien-tometric study in chemical oceanography, NATO ASISeries G9 (1986) 9-31.

[28] D. Hicks, B.R. Martin and J. Irvine, Bibliometric tech-niques for monitoring performance in strategic research:the case of integrated optics, R&D Management 16 (3)(1986) 211-224.

[29] D. Hicks, Limitations of co-citation analysis as a tool forscience policy, Social Studies of Science, to appear.

[30] R. Hooghiemstra, An Atlas of Science: sidestepping disci-pline limitations a great advantage of cluster analysis,Science Policy in the Netherlands 7 (2) (1985) 12-14.

[31] J. Irvine and B.R. Martin, Assessing basic research: thecase of the Isaac Newton Telescope, Social Studies ofScience 13 (1983) 49-86.

[32] J. Irvine and B.R. Martin, What directions for basic

research? in: M. Gibbons, P. Gummett and B.M.

Udgaonkar, eds., Science and Technology Policy in the

1980’s and Beyond (Longman, London/New York, 1984)67-98.

[33] J. Irvine, B.R. Martin, T. Peacock and R. Turner, Chart-

ing the decline in British science, Nature 316 (1985)587-90.

[34] J. Irvine and B.R. Martin, Evaluating Big Science: CERN’s



275

past performance and future prospects, Scientometrics 7

(1985) 281-308.[35] L.V. Jones, The assessment of scholarship, New Directions

for Program Evaluation 6 (1980) 1-20.[36] M.M. Kessler, Comparison of results of bibliographic

coupling and analytic subject indexing, American Docu-mentation 16 (1965) 223-233.

[37] S.M. Lawani and A.E. Bayer, Validity of citation criteriafor assessing the influence of scientific publications: newevidence with peer assessment, J. Amer. Soc. Info. Sci. 34(1983) 59-74.

[38] Letters, Science 214 (1981) 1292-1294; Science 215 (1982)344-348.

[39] D. Lindsey, Production and citation measures in the soci-ology of science: the problem of multiple authorship,Social Studies of Science 10 (1980) 145-162.

[40] B.R. Martin and J. Irvine, Assessing basic research: somepartial indicators of scientific progress in radio astron-omy, Research Policy 12 (1983) 61-90.

[41] B.R. Martin, J. Irvine and N. Minchin, An InternationalComparison of Government Funding of Academic andAcademically Related Research, ABRC Science PolicyStudies No. 2, Advisory Board for the Research Councils,London, 1986.

[42] P.R. McAllister, R.C. Anderson and F. Narin, Compari-son of peer and citation assessment of the influence of

scientific journals, J. Amer. Soc. Info. Sci. 31 (1980)147-152.

[43] P.R. McAllister and D.A. Wagner, Relationship betweenR&D expenditures and publication output for US collegesand universities, Research in Higher Education 15 (1981)3-29.

[44] P.R. McAllister and F. Narin, Characterization of the

research papers of US medical schools, J. Amer. Soc. Info.Sci. 34 (2) (1983) 123-131.

[45] R. McCulloch, International indicators of science and

technology: how does the US compare? Scientometrics 2(1980) 355-367.

[46] D.H. McQueen and J.T. Wallmark, Innovation outputand academic performance at Chalmers University of

Technology, Omega Internat. J. Management Science 12

(1984) 457-464.

[47] I.I. Mitroff and D.E. Chubin, Peer review at the NSF: adialectical policy analysis, Social Studies of Science 9(1979) 199-232.

[48] H.W. Moed, W.J.M. Burger, J.G. Frankfort and A.F.J.Van Raan, The use of bibliometric data for the measure-ment of university research performance, Research Policy14 (1985) 131-149.

[49] H.F. Moed, W.J.M. Burger, J.G. Frankfort and A.F.J.Van Raan, The application of bibliometric indicators:

important field- and time-dependent factors to be consid-ered, Scientometrics 8 (3-4) (1985) 177-203.

[50] P. Murugesan and M.J. Moravcsik, Variation of the na-ture of citation measures with journals and scientific

specialities, J. Amer. Soc. Info. Sci. 29 (1978) 141-147.

[51] F. Narin, Concordance between subjective and bibliomet-ric indicators of the nature and quality of performedbiomedical research, A Program Evaluation Report forthe Office of Program Planning and Evaluation, NIH,National Institutes of Health, US Department of Healthand Human Services, Bethesda, MA, 1983.

[52] E. Noma, Subject Classification and Influence Weightsfor 3000 Journals, Report prepared by Computer Hori-zons Inc. for the National Institutes of Health, USA andthe Advisory Board for the Research Councils, UK, 1986.

[53] NSF, Report of the NSF Advisory Committee on MeritReview, National Science Foundation, Washington, DC,1986.

[54] OECD, The Measurement of Scientific and Technical

Activities ’Frascati Manual’ 1980, Organisation for Eco-nomic Cooperation and Development, Paris, 1981.

[55] OECD, OECD Science and Technology Indicators,Organisation for Economic Cooperation and Develop-ment, Paris, 1986.

[56] P.G. Pardey, Public Sector Production of AgriculturalKnowledge, PhD Thesis, University of Minnesota, June1986.

[57] A.L. Porter, Citation analysis: queries and caveats, SocialStudies of Science 7 (1977) 257-267.

[58] H. Rothman, ABRC Science Policy Study: further studieson the evaluation and measurement of scientific research,Report for the Economic and Science Research Council,Bristol Polytechnic, 1985.

[59] H. Rothman and G. Lester, The use of bibliometric

indicators in the study of insecticide research, Scientomet-rics 8 (3-4) (1985) 247-262.

[60] A. Schubert, S. Zsindely and T. Braun, Scientometric

analysis of attendance at international scientific meetings,Scientometrics 5 (1983) 177-187.

[61] Science Indicators; An Analysis of the State of US Sci-ence, Engineering and Technology, Reports of the Na-tional Science Board, National Science Foundation,Washington, DC, 1972, 1974, 1976, 1978, 1980, 1982.

[62] H. Small and E. Sweeney, Clustering the Science CitationIndex using co-citations: I; A comparison of methods,Scientometrics 7 (1985) 391-409.

[63] H. Small, E. Sweeney and E. Greenlee, Clustering theScience Citation Index using co-citations: II; MappingScience, Scientometrics 8 (1985) 321-340.

[64] L.G. Soete and S.M.E. Wyatt, The use of foreign patent-ing as an internationally comparable science and technol-ogy output indicator, Scientometrics 5 (1983) 31-54.

[65] I. Spiegel-Rosing, Bibliometric and content analysis, So-cial Studies of Science 7 (1977) 97-113.

[66] D. Sullivan, D.H. White and E.J. Barboni, The state ofscience: indicators in the speciality of Weak Interactions,Social Studies of Science 7 (1977) 167-200.

[67] D. Sullivan, D.H. White and E.J. Barboni, Co-citation

analyses of science: an evaluation, Social Studies of Sci-ence 7 (1977) 223-240.

[68] P. Vinkler, Management system for a scientific researchinstitute based on the assessment of scientific publica-tions, Research Policy 15 (1986) 77-87.

[69] G. Vladutz and J. Cook, Bibliographic coupling and sub-ject relatedness, in: B. Flood, J. Witiak and T.H. Hogan,eds., 1984: Challenges to an Information Society, Proceed-ings of the 47th ASIS Annual Meeting (Knowledge In-dustry Publications, New York, 1984) 204-207.

[70] T. Von Waldkirch, Ten years of project-oriented alloca-tion of resources at E.T.H. Zurich: review and evaluation

of experiences, Internat. J. Institutional Management in

Higher Education 9 (1) (1985) 52-57.



276

[71] A. Weinberg, Criteria for scientific choice, Minerva 1

(1963) 159-171.[72] W.S. Wise, Agricultural R&D, Nature 313 (1985) 8.[73] J. Ziman, Criteria for national priorities in research, in:

Rise and Fall of a Priority Field, Proceedings of the

ESRC/NSF Symposium, Paris, 22-24 September, 1985(European Science Foundation, Strasbourg, 1985) 95-125.



a review of bibliometric and other science indicators and their role in research evaluation

Documents