a validation study of bibliometric indicators: the comparative performance of cum laude doctorates...

Scientometrics, Vol. 17, Nos 5-6 (1989) 42 7-435

A VALIDATION STUDY OF BIBLIOMETRIC INDICATORS: THE COMPARATIVE PERFORMANCE OF CUM LAUDE DOCTORATES

IN CHEMISTRY

A. J. NEDERHOF, A. F. J: VAN RAAN

Science Studies Unit, LISBON-Institute I University of Leiden, Stationsplein 242, 2312 AR Leiden (The Netherlands)

(Received January 17, 1989)

The validity of bibliometric indicators as a monitor of the impact and usefulness of scientific research is examined by compaiing the scientific performance of cure laude and non-cum laude degree holders in chemistry (N=237), from five y~ears before their graduation to four years afterwards. Papers of cum laudes were cited more frequently than those of non-cum laudes from three years before graduation until one year after graduation..Two to three years after graduation, the short-term impact per paper was no longer significantly different for both groups. A similar pattern was found with regard to productivity. Little evidence was found in favor of the Ortega hypothesis and the Matthew effect. The results support the concurrent validity of bibliometric indicators with peer review indicators of quality of the research project.

Introduction

During the past decade, the assessment of scientific performance has been studied

ex~ensively in scientometrics, resulting in an enhanced appraisal of the strengths

and liabilities of both peer review and bibliometric methods. Recently, limitations

of peer review have become apparent (Harnad, 1982; lrvine and Martin, 1983;

Moed, Burger, Frankfort and Van Raan, 1985a; Nederhof, 1988), while biometric

studies claim only to monitor the impact and usefulness of scientific research, and not

the quality of scientific performance (e.g., Moed, Burger, Frankfort and Van Raan, 1985b).

Critics of bibliometric methods have attempted to show that citations are only,

or in the first place, rethoric instruments used, not to convince others by the strength

of arguments but to persuade th0m by nonlogical means (Cozzens, 1989; Gilbert, 197 7 ; MacRoberts and MacRoberts, 1986). Also, MacRoberts and MacRoberts (1986:159) make a case for the importance of 'ceremonial citations', references made

to individuals whose work has actually not been influential, but to whom the citer

Scientometrics 17 (1989) Elsevier, Amsterdam-Oxford-New York Akad~rniai Kiad6, Budapest

A. J. NEDERHOF, A. F. J. VAN RAAN: VALIDATION OF BIBLIOMETRIC INDICATORS

feels obliged. In the present study, we will investigate whether or not a significant number of references are made for other reasons than persuasion or ceremonial purposes. We studied citations to the work of graduate students. When citations are purely made to persuade an audience, or for ceremonial reasons, it would be expected that the

work of graduate students, who will not be recognized very well or respected very highly by the scientific audience, would be cited rather low uniformly, especially

after removal of the effects of the presence or absence of prestigeous or powerful

co-authors. In addition, it would be expected that the post-Ph.D, work of graduates awarded with a 'cum laude' ("cumlaudes") would be cited more after their award than before, both in comparison with their own work, as well as with that of graduates not receiving their Ph.D. with a cure laude ("non-cumlaudes").

A simlar hypothesis could be deduced from the Matthew effect (Merton, 1968): before graduation, differences between cumlandes and noncumlaudes-to-be would be small, as grounds are lacking for those outside the particular university to distribute merit (i.e,, citations) according to (indirectly) observable distinctions, such as a better position. However, post-Ph.D, publications by cumlaudes should receive increasingly more citations than noncumlaudes, because of the cumulative nature

of the Matthew effect. According to the Ortega hypothesis (Ortega y Gasser, 1932), progress in experi-

mental science is due in great part to the work of mediocre and less than mediocre

scientists. Provided that scientists as a group distribute their citations according to

the contribution scientists actually have made to scientific progress, the Ortega hypothesis implies that a relatively large share of citations obtained by graduates would be received by non-cumlaudes, who should also produce a sizable share of

the highly cited publications. A fourth hypothesis assumes that both peer review and bibliometric measures

are partially valid indicators of past scientific performance. Therefore, it is expected that publications of cumlaudes are cited more often than publications of non, cumlaudes. When, additionally, the assumption is made that cumlaudes outperform

non-cumlaudes because they are better researchers, than it would be expected that

cumlaudes continue to outperform non-cumlaudes after graduation, and would also continue to produce more highly cited publications than non-cumlaudes.

A fifth hypothesis is that the quality of the research project, and not so much the quality of the particular student, is the most important factor in both productivity levels of researchers and impact levels of scientific publications (Nederhof and Van Raan, 1987: 334). Then, cumlaudes would outvie non-cumlaudes, and also be authors of more highly cited publications, as long as their publications concern their graduate work. However, when cumlaudes and noncumlaudes embark

428 Scientometrics 1771989)

A. J. NEDERHOF, A. F. J. vAN RAAN: VALIDATION OF BIBLIOMETRIC INDICATORS

upon new'research projects after graduation, initial differences may disappear. In

our study of physics graduates, we indeed found this pattern of results (Nederhof and Van Raan, 1987).

To test the various hypotheses, productivity and impact of cum-laudes and non- cumlaudes in chemistry were studied from five years before graduation until three

years after graduation.

Method

The files of the Leiden Indicators Project were used as data source, especially publication and citation data of the chemistry departments at the University of Leiden for a fourteen-year period (1970-1983). These data, used in previous

studies of research performance, are amply discussed in earlier work of our group

(Moed et al., 1985a, 1985b). In-house citations (references given by other publications with a Leiden address) were not included in the citation counts. To remove

potential differences due to type of publication, only research articles were included

in the publication file (the articles coded as a blank by ISI). Book publications were excluded. Regarding publications with multiple authors, only the status (cum

laude or not) was taken into consideration of those authors who could fulfill all

conditions posed to inclusion. Thus, if, for instance, only papers published before graduation are sonsidered in a particular analysis, the status of co-authors who have already graduated is disregarded. This was done in order to focus as much as

possible on the performance of the graduates, independently from co-authorships. The doctorate files of the University of Leiden have been searched for the data

concerning the cum laude award. In five chemistry departments, 28 cumlaudes (11,8%) and 209 non-cumlaudes (99,2%) were found.

The impact of publications was computed according tct'citation windows',

consisting of citations obtained in the year of publication of the paper and the two subsequent years.

Results

To examine the comparative changes in productivity and impact of the graduates over time, Table 1 was constructed.

From five years before graduation to three years ,after graduation, the productivity of graduates, the short-term impact of their publications, as well as the total number of citations received were computed for eight consecutive partially over lapping two-year blocks (see Figs 1, 2, and 3).

Scientometrics 17 {1989} 429


Table 1 Short-term impact and productivity of cumlaude and non-cumlaude graduates in chemistry

Cumlaudes Non-cumlaudes Publication

blocks C/a P/a C/P C/a P/a C/P

Yg-5 to Yg-4 0.076 0.615 0.125 1.574 0.327 4.818 Yg-4 to Yg-3 1.687 0.375 4.500 1.403 0.468 3,000 Yg-3 to Yg-2 5.647 0.706 8,000 /..743 0.625 2.789 Yg-2 to Yg-1 7.400 1.700 4.353 2.451 0.915 2.680 Yg-1 to Yg 8.478 2.391 3,545 3.583 1.389 2,580 Yg to Yg+l 9.178 2.178 4.213 4.071 1.565 2.602 Yg+l to Yg+2 6.885 1.538 4,475 2.780 0.995 2.795 Yg+2 to Yg+3 2.375 0.667 3.563 1.384 0.469 2.95T

C/a = average number of citations per author; P/a = average number of publications per author; C/P = average number of citations per publication; Yg = year of graduation.

A

o Cumlaudes .No cu . ~ 6 ' 0

o ~ , r I [ I I I E I -5,-4 -z4-3. -3,-2 -2,-1 -1,0 0,1 ~,2 2,3 <

2-yeors publication blocks with Yg=0

Fig. 1. Citations per author for cumlaudes and non-cumlaudes over time

The first two year-block covers the f if th and four th year before graduation,

while the eighth b lock covers the second and third year after graduation. With

regard to ci ta t ion counts , blocks were formed consisting o f two consecutive

publicat ion years, each having its own 0 - 2 year ci ta t ion window.

The product iv i ty ' o f cumlaudes and non-cumlaudes is rather similar f rom five to

two years before graduation. Two years before graduation, however , the prodttc-

t ivi ty o f cumlaudes more than doubles f rom 0.7 paper to 1.7 paper, while this

430 Scientometrics 17 (1989)


~ 2/4

a. 1.92

.u 1Az

.D

o961 "6

r~

c

o

.~ - , ,

~ 4 I t I L L l -5,-z, -4,-3 -3,-2 -2,-1 -I,0 0,1 1,2 2,3

2-years publication blocks with Yg--0

Fig. 2. Publications per author for cumlaudes and non-cumlaudes

cz 6z Q.

"6 3.2

i 1.6

o

A o Cumlaudes

/ - ' " - f - i : - ' i .... .... -5,-~ -~,-3 -3,-2 -2,-1 -1,0 0,1 1,2 2,3

2-ye~s publication blocks with Yg:O

Fig. 3. Short-term impact per paper for cumlaudes and rton-cumtaudes

increase is less than 50% for noncumlaudes compared to the previous two-year

block. Productivity reaches a peak at 2.4 papers for cumlaudes in the year preceding

graduation and the graduation year itself. The productivity of noncumlaudes also

increases, to 1.4 papers on the average, one full paper below the level of cumlaudes.

The gap begins to close, however, during the year of graduation and the year

thereafter, as prodcutivity of cumlaudes declines somewhat to 2.2 paper, while

that of noncumlaudes increases further to 1.6 paper. Thereafter, the registered

productivity of both groups declines steeply, partly no doubt because of finding

Scientometrics 17 (1989) 431


jobs outside Leiden University. In the second and third year after graduation, the "Leiden" productivity of cumlaudes is only 0.2 papers above that of non-cumlaudes.

The picture changes somewhat when the total short-term impact is examined of all papers produced during two consecutive years. From five to four and from

four to three years before graduation, cumlaudes and noncumlaudes are cited about

equally (due to small numbers, the relatively high score of non-cumlaudes in the first two-year block is not significantly different from the relatively low score of the cumlaudes). Three to two years before graduation, however, the number of citations to cumlaudes soar significantly from 1.7 to 5.7 citations, while non- cumlaudes only slightly improve from 1.4 to t.7 citations. The published research

papers of cumlandes continue to draw more citations until the two year period

covering the graduation year and the subsequent year, when they are cited 9.2 times. In the same period, the publications of non-cumlaudes are also more

frequently cited, but at a much slower pace, increasing from 1.7 to a peak of 4.1. citations. In the two subsequent blocks, covering 1 to 3 years after graduation, the number of citations to Leiden publications of the graduates decrease dramatically, although cumlaudes continue to receive somewhat more citations than non-cumlaudes.

The most important measure of impact is the average number of citations per

paper. Interestingly, the few non-cumlaudes who publish early (5 to 4 years before

graduation) are never cited more frequently during their subsequent career. Due to

small numbers, the seemingly huge difference in average short-term impact per paper between non-cumlaudes (4.8 citations) and cumlaudes (0.1 citations) is not

significantly different (p < 0.06). Although the impact per paper is higher for cumlaudes (4.5 citations) than for non-cumlaudes (3.0 citations) 4 to 3 years

before graduation, this difference is also not significant. However, 3 to 2 years before graduation, papers of currdaudes are cited 8 times

on average (versus only 2.8 times for non-cumlandes, p < 0.004), a level not to be surpassed again during the period surveyed in this study. After that, papers of cumlaudes average between 3.5 and 4.5 citations each, compared to a significantly lower 216 to 2.8 citations for papers of non-cumlaudes until the second year after

graduation. Only in the last two-year block, extending 2 to 3 years after graduation, the

average number of citations per paper of cumlaudes (3.6 times) and non-cumlaudes (3.0 times) are not significantly different. It is important to check wether this is due the structure of our data file, which contains only publications with a Leiden

address. After graduation, many leave the Leiden University to get a job elsewhere. To get a better view of what happens after graduation, we compared the output

of first-authored cited papers produced two to four years after graduation of both cumlaudes and a matched sample (as regards year of graduation and mentor) of



non-cumlaudes in chemistry. Thus, persons no longer active in (academic) research

were excluded from the samples, and only papers were included which were likely

to be produced after graduation. Included papers were published between 1974 and

1980. Data were collected concerning 24 cumlaudes and 25 non-cumlaudes.

The results were as follows. Cumlaudes produced on average 0.65 papers per

year, while noncumlaudes averaged only half that amount: 0.33 papers. Papers by

cumlaudes were cited 6 times on average during the first three year, versus only 3.7

times when non-cumlaudes were authors. Although this concerns only first-

authored papers, these results suggest strongly that cumlaudes who continue

publishing after graduation, clearly outproduce active non-cumlaudes, and that

cited papers of cumlaudes have a higher impact on average than cited papers of

non-cumlaudes.

Number of highly cited papers per author

Another important way of measuring the impact of work is to look at the

number and percentage of papers exceeding a particular level of citations. We

examined the short-term impact of papers produced as a result of graduate work

by focussing on the period from 3 years before graduation to 1 year after. The

results are presented in Table 2.

Table 2 Number of (highly) cited papers per author for cumlaudes

and non-cumlaudes from three years before graduation to one year after graduation

Citation level/> Cumlaudes Non-cumlaudes

0 3.94 (100%) 2.90 (100%) 1 3.12 ( 79.1%) 1.89 ( 65%) 5 1.29 (32.8%) 0.53 (18.2%)

10 0.41 (10.4%) 0.17 (6.0%) 15 0.29 (7.5%) 0.08 (2.9%) 20 0.12 (3.0%) 0.05 (1.7%) 30 0.06 (1.5%) 0.01 (0.2%)

As has been shown already, cumlaudes are moreproduc t ive than non-cumlaudes.

The greater productivi ty of cumlaudes has not diluted the impact of their work, as

the average impact of their papers is clearly above that of non-cumlaudes. Whereas

79% of the papers by cumlaudes are cited, only 65% of the papers by non-cumlau-

des receive at least one citation. While cumlaudes produce on the average during

433


the five year period 1.3. papers which are cited five times or more (nearly a third of their total production), non-cumlaudes only produce 0.5 papers (representing 18% of their total production). On all citation levels examined, cumlaudes have

a higher percentage of their papers included than non-cumlaudes.

Discussion

In the present study, a comparison is made between the results of a peer review process, namely the awarding of a cure laude degree to graduates in chemistry, and bibliometric indicators of productivity and impact. By monitoring the work of the graduates from five years before graduation to three years after graduation, it was

possible to compare results with various hypotheses. The results do not fit the pattern predicted by the Matthew effect: little differ-

ence in short-term impact before graduation and an increasing difference between cumlaudes and non-cumlaudes after graduation. The actual pattern, however, is nearly reversed. Large difference in impact and productivity favor cttmlaudes three to two years before graduation, differences which decrease in the following years, although remaining significant. As eumlaudes and non-cumlaudes share the same mentors, the observed pattern cannot be attributed to differences in distiction of the co-authors. An alternative explanation might be, that cumlaudes reap the benefits of very early career awards, as for instance a cure laude undergraduate degree. The nonsignificant differences in impact between 5 and 3 years before graduation do not point in this direction, however. We are therefore lead to conclude that a Matthew effect is not visible in our data. This replic~ites earlier results in a study of physics graduates (Nederhofand Van Raan, 1987). If a Matthew effect still exists, it takes longer than the period studied

here. According to the Ortega hypothesis, a relatively large share of citations and

highly cited publications are expected to be obtained by the relatively 'mediocre' non-cumlaudes. Indeed, of the 1476 citations received by graduates from three years before to one year after graduation, 1134 go to non-cumlaudes, and only 299 to cum-laudes. However, only 18% of the publications of non-cumlaudes draw 64% of all citations. These figures indicate that even among non-cumlaudes, a few persons attract the major share of citations. Thus, the Ortega hypothesis is hardly

supported by the present data. Various sceptics have claimed that bibliometric measures based on citations are

generally invalid. The present data do not offer any support for this stance. Highly significant differences in impact and productivity were obtained between two groups distinguished on a measure of scientific quality based on peer review (the cum laude award). As with other results of validation studies of citation studies, these



can no t be explained by sceptics. It is hard to see what differential ceremonial or

persuasive value a c i ta t ion towards the work o f a graduate s tudent would have.

Even though a percentage o f the ci tat ions may have been given for per func tory or

negative reasons, these results suppor t the con ten t ion that when sufficiently large

numbers o f papers are examined, ci tat ions counts may provide a useful partial

indicator o f quali ty, and can be used frui tful ly to mon i to r scientific research.

References

1. COZZENS, S. E. What do citations count? The Rhetoric-first model, Scientometrics, 15 (1989) 000.

2. GILBERT, G. N. Referencing as persuasion, Social Studies o f Science, 7, (1977) 113-122. 3. HARNAD, S. (Ed.), Peer commentary on peer review, Cambridge, Cambridge University Press,

1982. 4. IRVINE, J., MARTIN, B. Assessing basic research. Some partial indicators of scieritific progress

in radio astronomy, Research Policy, 12, (1983) 61 -90 . 5. MACROBERTS, M. H. MACROBERTS, B. R., Quantitative measures of communication in

science: A study of university research performance, Social Studies o f Science, 16 (1986), 151-172.

6. MERTON, R. K., The Matthew effect in science, Science, 159 (1986) 56-63. 7. MOED, H. F., BURGER, W. J. M., FRANKFORT, J. G., VAN RAAN, A. F. J., A comparative

study of bibliometric past performance analysis and peer judgment, Scientimetrics, 8 (1985) 149-159.

8. MOED, H. F., BURGER, W. J. M., FRANKFORT, J. G. VAN RAAN, A. F. J. The use of bibliometric data for the measurement of university research performance, Research Policy, 14 (1985) 131-149.

9. NEDERHOF, A. J. The validity and reliability of evaluation of scholarly performance, In: A. F. J. VAN RAAN (Ed.), Handbook o f Quantitative Studies in Science and Technology, Amsterdam, North Holland, 1988, pp. 193-228.

10. NEDERHOF, A. J. VAN RAAN, A. F. J. Peer review and bibliometric indicators of scientific performance: A comparison of cure laude doctorates with ordinary doctorates in physics, Scientometrics, 11 (1987) 333-350.

11. ORTEGA Y GASSET, J., The revolution of the masses. New York, Norton, 1932.

Scientometries 17 (1989) 435

a validation study of bibliometric indicators: the comparative performance of cum laude doctorates...

Documents