bibliometric performance measures

Jointly published by Elsev&r Science Ltd, Oxford

and Akad~miai Kiad6, Budapest

Scientometrics,

Vol. 36, No. 3 (1996) 293-310

BIBLIOMETRIC PERFORMANCE MEASURES

F. NARIN, KIMBERLY S. HAMILTON

CHI Computer Horizons Inc., 10 White Horse Pike, Haddon Heights, NJ 08035 (USA) r

(Received May 24, 1996)

Three different types of bibliometrics - literature bibliometrics, patent bibliometrics, and linkage bibliometric can all be used to address various government performance and results questions. Applications of these three bibliometric types will be described within the framework of Weinberg's internal and external criteria, whether the work being done is good science, efficiently and effectively done, and whether it is important science from a technological viewpoint. Within all bibliometrics the fundamental assumption is that the frequency with which a set of papers or patents is cited is a measure of the impact or influence of the set of papers. The literature bibliometric indicators are counts of publications and citations received in the scientific literature and various derived indicators including such phenomena as cross-sectoral citation, coauthorship and concentration within influential journals. One basic observation of literature bibliometrics, which carries over to patent bibliometrics, is that of highly skewed distributions - with a relatively small number of high-impact patents and papers, and large numbers of patents and papers of minimal impact. The key measure is whether an agency is producing or supporting highly cited papers and patents. The final set of data are in the area of linkage bibliometrics, looking at citations from patents to scientific papers. These are particularly relevant to the external criteria, in that it is quite obvious that institutions and supporting agencies whose papers are highly cited in patents are making measurable contributions to a nation's technological progress.

Introduction

It seems quite appropr ia te for b ib l iomet r ic techniques to play a major role in the

applicat ion o f the U.S . G P R A (Governmen t Per formance and Results Act) to scient if ic

research. Many b ib l iomet r ic techniques were deve loped with the evaluat ion o f research

as a pr imary goal , especial ly eva lua t ion o f research departments , insti tutions, and

international aggregates , for which there are vi r tual ly no other quant i ta t ive

alternatives.1 In this paper we wil l discuss three different categories o f b ib l iometr ic

analysis: l i terature b ib l iometr ics , patent b ib l iometr ics , and l inkage bibl iometr ics , all o f

which may be used in establ ishing quant i ta t ive indicators o f both the outputs and the

outcomes o f gove rnmen t programs.

0138- 9130/96/US $ 15.00 Copyright �9 1996 Akaddmiai Kiadr, Budapest All rights reserved

F. NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES

The drive to develop techniques for evaluating research programs arose in the first

years of financial stress following the post World War II period of unlimited science

growth. As a result of the development of radar, atomic energy and other technological

wonders in the war years, the essential contribution of science to technology became

apparent, and of technology to national policy, and the broad support of science

became a national goal. Inevitably, of course, the growth in support had to level off,

and the scientific community had to face the stressful prospect of limited funding.

The challenge of assuring that the output of publicly supported science was worthy

of the dollars being devoted to it was attacked in two elegant papers published by Alvin Weinberg in the early 1960's. 2,3

Weinberg set out to systematize the criteria for scientific choices.

Weinberg categorized the criteria as internal and extemal. The internal criteria deal

with the question "Is it good science?" whereas the external criteria deal more

explicitly with what kind of impact that particular area of science was having.

It is very important to note that he felt that these two sets of criteria, internal and

external, should be viewed separately, a point that seems to be lost in much of the

current discussion of program evaluation.

Weinberg is an eloquent writer and we will basically summarize what he said in his

own words, by excerpting sections from his 1963 paper.

"As science grows, its demands on our society's resources grow. It seems

inevitable that science's demands will eventually be limited by what society can

allocate to it. We shall then have to make choices. These choices are of two kinds.

We shall have to choose among different, often incommensurable, fields of science

-between, for example, high-energy physics and oceanography or between

molecular biology and science of metals. We shall also have to choose among the

different institutions that receive support for science from the government - among

university, governmental laboratories and industry. The first choice I call scientific

choice; the second, institutional choice. My purpose is to suggest criteria for

making scientific choices - to formulate a scale of values which might help

establish priorities among scientific fields whose only common characteristic is

that they all derive support from the government."

"Internal criteria are generated within the scientific field itself and answer the

question: How well is the science done? External criteria are generated outside the

scientific field and answer the question: Why pursue this particular science?

Though both are important, I think the external criteria are the more important."

"Two internal criteria can be easily identified:

(1) Is the field ready for exploitation?

294 Scientometrics 36 (1996)


(2) Are the scientists in the field really competent?"

"Three external criteria can be recognized:

(1) technological merit

(2) scientific merit, and

(3) social merit."

The first is fairly obvious: once we have decided, one way or another, that a

certain technological end is worthwhile, we must support the scientific research

necessary to achieve that end."

"The criteria of scientific merit and social merit are much more difficult: scientific

merit because we have given little thought to defining scientific merit in the

broadest sense, social merit because it is difficult to define the values of our

society."

"Relevance to neighboring fields of science is, therefore, a valid measure of the

scientific merit of a field of basic science."

"... that field has the most scientific merit which contributes most heavily to and

illuminates most brightly its neighboring scientific disciplines."

The imperative for justifying scientific investment also led to the first TRACES

study (Technology in Retrospect and Critical Events in Science) which tried to link, in

a relatively systematic way, the development of important innovations to the science

base. 4 The initiation of the Science Indicators Report Series by the National Science

Board in 19725 was another step in responding to the need for quantitative output and

outcomes indicators.

In the subsequent sections of this paper we will show how the three types of

bibliometrics, literature, patent and linkage, can all be used to address two categories

of GPR-related questions: is the work being done by an agency good science,

efficiently and effectively done, and is it important from a technological viewpoint, in

essence Weinberg's internal and external criteria.

We will discuss both some of the basic indicators that can be developed, and some

of their limitations. However, at the outset we want to make an extremely important

point about the skewness of virtually all bibliometric distribution, and how that must

be carefully taken into account in any analysis and interpretation of bibliometric data. 6

The fundamental point is that virtually every distribution of scientific productivity

or impact is highly skewed, with a small number of highly productive scientists, hot

areas, research institutions, etc., and a relatively large number of institutions and

producers of much smaller impact. The distributions are more often logarithmic than

linear, with difference factors of 10 to 100 between the most productive entities and

the least productive entities, in any scientific or technological distribution.

Scientometrics 36 (1996) 295


For example, the influence of individual scientific journals - where influences is

defined as the weighted average number of citations per paper in a journal - is distri-

buted logarithmically, over a range of more than 100 between the most influential,

most cited journals in a subfield and the least influential, least cited journals in a

subfield. 7 Similarly, recent data has shown that the number of patents being produced

by individual inventors, even within a single company, is logarithmically distributed,

just as the original Lotka's Law showed for scientific papers. 8,9

The reason this is very important GPRA related work is that for one individual, or

even a small group, a very large amount of the ultimate impact on society may be

centered around a few relatively rare, extremely high impact discoveries, papers,

inventions, etc. Further, in highly skewed distributions the means and the medians can

be quite different, and one may often want to do statistical testing in the logarithm of

the distribution, rather than in the distribution itself, in order to have something even

approaching a normal curve. Therefore, in all of the subsequent discussions of publi-

cation and citation rates, it must be kept in mind that comparisons should always take

skewness into account, and that differences of five-to-ten percent in impact between

two research institutions are almost certainly insignificant in a realm where the

differences between two research institutions are often factors of five-to-ten or more.

Within all of the bibliometric analyses, of course, there are a series of basic tenants

that are widely and almost universally accepted. The tenants include:

1. Counts of the number of papers, and the numbers of patents, provide basic

indicators of the amount of scientific and technological productivity.

2. Counts of citations to these papers and patents, and from patents to papers,

provide indicators both of the quality (impact) of research, and of the linkage

between basic and applied research, between one subfield and another, and

between technology and science.

3. Counts of coauthorships, and especially international coauthorships, are an

indicator of quality, and that scientists who cooperate with their colleagues in

other institutions and overseas are more likely to be doing quality research

than those that are relatively isolated.

4. There are natural differences in citation and publication pattems across

research and technology, unrelated to quality of the science, and one must be

extremely careful in doing comparisons of any bibliometric parameters

without fully adjusting for these differences: for example, between the citation parameters in a relatively lightly citing subfield such as acoustics and

a very heavily citing subfield such as biochemistry and molecular biology.



Literature indicators for GPRA

The bibliometric indicators which have been in use for the longest time are, of course, literature indicators: indicators of the scientific performance of organizations, agencies and countries based on counts of publications and citations in the scientific literature. These have been used most extensively in the United States in various studies done in the 1970's and 1980's for the National Institutes of Health 1~ and, of course, in the pioneering work at the National Science Foundation in the development of the Science Indicators series, starting with Science Indicators 1972, ongoing to Science and Engineering Indicators-1996, which has just been published.

Amongst the literature indicators which are relevant to Weinberg's internal criteria are the number of publications, publications per dollar, the number of citations

received by the papers, as well as various influence and impact surrogates for citation

rate. In addition to those basic productivity and impact measures, another measure applicable to the internal criteria of whether an agency is producing good science is coauthorship, since it is widely accepted that having papers coauthored with one's colleagues at prestigious universities and research institutes, and with colleagues overseas, are indicators of acceptance within the scientific community.

The number of papers produced by a given research institution is certainly a measure of its output, and a very basic one for GPRA. In most bibliometric studies of the physical, biological and mathematical science this is confined to articles, notes and reviews in refereed research journals, such as those covered in the Science Citation

Index (SCI). This measure has particular significance when it is placed within the context of an organization's budget. Figure 1 shows such a placement, adapted from a

paper published in 1976, graphing the number of biomedical papers published by the top 122 U.S. universities, compared to the amount of funds these schools received from NIH three years before. 11 The correlation, of course, is extremely high, approximately r =: 0.95, indicating that virtually all variation in the number of biomedical papers produced by those institutions is fully accounted for by the funding

received: large and small universities produce essentially the same number of papers on a per dollar of funding basis. In general, we have never found that the size of an institution is of any significance in measurement of either the productivity or the quality of its research output. Productivity and quality vary widely, but are not primarily driven by organizational size. Small, highly specialized institutions can

produce papers of just as high quality papers per funding increment as large, well known institutions.

All bibliometric indicators of the quality of research produced are normally based on either direct citation counts, or various citation surrogates such as journal influence



or journal impact factor, and all are subject to one extremely important constraint: the

data must be normalized for differences in field, subfield, and sometimes specialty

parameters. Citation densities, that is the number of references per paper, the number

of times a paper is cited, and time lags all vary widely from one field to another, and

one subfield to another, and sometimes even within a subfield by specialty area. As a

general rule, citation densities are highest in very hot and fast areas of science such as

molecular biology and genetics, and much lower and slower in some of the more

traditional areas of astronomy, descriptive biology, mathematics, and so forth. Citation

patterns are also quite light in most of the engineering fields, and very different in the

Social Sciences.

800

700

600

500

_~ 4oo

g 3oo

<

S

s �9

200 ~ �9 �9 �9 ..' ". ," t, .

100 �9 "-eeqb �9 s�9 *

0 D ! I i i i

2000 4000 6000 8000 10000

Average NIH Constant 1967 Dollars (thousands), 1965-1969

l l * s

I , i I

12000 14000 16000

Fig. 1. Publication output v e r s u s NIH funds for 122 universities

Perhaps the most fundamental challenge facing any evaluation of the impact of an

institution's programs or publications is to properly normalize and adjust for field and

subfield differences.

This point is illustrated by Fig. 2, which shows the average citations per paper

received in a five year window for papers in biomedical subfields, where subfield is, in

this case, CHI 's division of approximately 1,300 biomedical journals into 49 different

subfields. Obviously, the factor of 4 or more between the citation frequency of papers

in the most highly cited subfield biochemistry and molecular biology compared to the

~r;D,,tnmatrie~ ~6 (1996)

F, NARIN, K. S. HAMILTON: BIBLIOMETRIC PERFORMANCE MEASURES

least highly cited specific biomedical subfield pharmacy indicates that citation studies

of biomedical publication performance must be very carefully adjusted for differences

in the subfield. For example, a highly prestigious paper in a lightly citing subfield may

be less frequently cited than an average paper in a highly citing subfield, and it is

simply not valid to directly compare citation counts in one of the lightly citing clinical

subfields with citation counts in the much more highly citing basic subfields.

5 Year Citing Window)

o

o

Fig. 2. Cumulative cites per paper to 1986 Clinical Medicine & Biomedical Research papers from 1986-91 papers by subfield

One type of normalization sometimes done, with which we totally disagree, is to

divide a papers' citation count by the average citation count for papers in the specific

journal in which the paper appeared. This has the perverse effect of penalizing a

research institution that encourages its scientists to publish in the most prestigious and

best journals in its field.



The correct way to do citation normalization is on a subfield basis, dividing citation

counts by subfield analysis. In highly precise studies one should further adjust for the

citation frequencies within specialized areas.

Coauth0rship rates may be used to determine another internal indicator of

performance, based on the generally accepted notion that scientists who coauthor with

people outside their institutions, and especially with their colleagues overseas, are the

scientists who are the most widely accepted in their community. A particularly

interesting demonstration of this is contained in a study done for the European

Community, which showed that Community papers that were coauthored with

scientists outside the home country were more than twice as highly cited as those that

were authored at a single institution within a single country; papers coauthored within

one country were cited intermediate between the single institution papers and the

internationally coauthored papers. 12

Coauthorship rates should always, of course, be adjusted for the dynamic changes

that are occurring in coauthorship, and for the size of countries. International

coauthorship has been increasing steadily in the United States and everywhere else, and

is quite field dependent, in complicated ways. For example, although institutional

coauthorship rates are generally higher in the field of Clinical Medicine than in the

field of Biomedical Research, international coauthorship is higher in Biomedical

Research than in Clinical Medicine. It is extremely important that one adjust

coauthorship data for fields and subfield differences and, of course, year, since

coauthorship rates have been increasing monotonically for more than 20 years.

The final internal criteria for bibliometric performance to be discussed here relates

to the concept of research level. CHI Research has classified each of some 3,000

journals covered in the Science Citation Index (SCI) on a research level scale ranging

from 1 (most applied journals) to 4 (most basic journals). This scale allows one to

characterize the degree of basicness of a research program, or the publication of a

research institution. The four research levels and prototype journals are:

Level 1 Applied Technology J Iron & Steel Inst (Clinical Observation in Biomedicine) J Am Med Assn

Level 2 Engineering-Techno- J Nuc Sci & Tech logical Science Proc IEEE (Clinical Mix in Biomedicine) New Eng J Med

Level 3 Applied Research J Appl Phys (Clinical Investigation in Biomedicine) Cancer Res

Level 4 Basic Scientific Research Phys Rev J Am Ch Soc J Biol Chem



A particularly telling application of this is illustrated in Fig. 3, redrawn from a

paper published almost 20 years ago, which shows the average research level for the

publications of the 95 largest U.S. hospitals, medical schools, clinics, and research

universities. Also shown there is the average research level for papers published by the

U.S. National Institutes of Health (NIH) Intramural program. The figure quite

graphically illustrates the usefulness of this indicator: the average level of the papers

from the hospitals, clinics and medical schools is, for every single one, more clinical

than the average level of the papers from the major universities. Furthermore, the

papers of the NIH Intramural program lie right in the middle of the distribution: the

NIH Intramural program is more clinical than any university program, more basic than

any hospital or medical school program, and, in fact, very beautifully spans the two

constituencies which the intramural program at NIH serves so well.

The construction of indicators relevant to Weinberg's external criteria within the

scientific literature are normally based on cross-citing patterns, either citations to a

basic subfield from an applied subfield, or citations to university and other public

science from industrial, private science. An illustration of this kind of indicator is

shown in Fig. 4 which shows the trend in citation from industry papers to university

sector papers in all fields. It is quite clear that industrial science continues to heavily

use the more basic papers produced at the universities, and the universities have, if

anything, a growing role in producing the science base of industrial technology.

There are, of course, various technical problems which impact any bibliometric

evaluation. One of the most important problems, especially for program evaluation, is

that of constructing the basic program bibliography. For a support program where

there are many different grantee institutions, agency records are almost always quite

incomplete, and usually in quite non-standard form. Just establishing the journal,

volume, page and year for the bibliography of the papers supported in a typical U.S.

government agency program is a massive job, and often takes many person months of

unification, yet such a standard form is absolutely imperative for any kind of analysis,

before you can even establish the field and subfield of the papers, or whether the

publications are, in fact, papers in refereed journals, in journals covered by the SCI, and so forth and so on.

Even the construction of an institutional bibliography is not trivial. Papers from

Harvard University contain literally hundreds of different variations of the name

Harvard University, and finding all of them in the literature is a substantial challenge.

In recent years the Institute for Scientific Information (ISI) has done some standardization of these names in the SCI, and that is a help in using that data.

However, many universities contain hospitals and research labs that have names other



than that of the university, and those addresses often do not explicitly mention the

university name, and so that a substantial amount of research is necessary to obtain a

decent bibliography. In creating the U.S. Science Indicators over the years, CHI has

created a thesaurus of over 600,000 different names which are associated with 3,500

major U.S. research institutions, and it. takes a substantial amount of time, effort and

care to construct an institutions bibliography from the many names under which its

scientists will publish.

MEDICAL SCHOOLS, HOSPITALS, AND CLINICS

] UNIVERSITIES

E

15

0 v l t n i

21o

(1 = Most Clinical)

2's 37o I 3'5 ,'o NIH

INTRAMURAL (4 = Mwst Basic)

Average Level

Fig. 3. Average level for biomedical papers of NIH and the 95 largest U.S. universities, medical schools,

hospitals and clinics. Source: The Intramural Role of the NIH as a Biomedical Research Institute, F. NARIN, S. B. KEITH Federation Proceedings, Vol. 37, No. 8, June 1978

Another important factor that must be carefully normalized for are the time effects.

Specifically, of course, citations accumulate over time, and as it was mentioned

earlier, citation data must be over comparable time periods, and within the same

subfield or area of science. However, in addition to that, the time patterns of citation

are far from uniform across country, and any valid evaluative indicator must use a

fixed window and be appropriately normalized. To illustrate this Fig. 5 shows a

phenomenon we call the "Citation Time Anomaly", namely the anomaly in the percent

of references from a country's papers to earlier papers for the same country. The



ordinate is the percent of the references from each country's papers which are to the

same country's papers, published in the prior year indicated on the abscissa.

0 . 0 0 5.00 10.00 15.00 20.00 25.00 30.00 35.00

i i : = ', [ t

Industry Sector

University Sector

Federal Gov Sector

FFRDC Sector

t r---I

1985 Cixl~ 1981-83

1986 Ctt.'~ 1982-84

~] 1997 Cili,~l 1983-88

[ ] 1989 CainQ 1984-86

C~ 1989 Crting 1985-87 [

[~ 1990 Cklng 1986-88

[ ] 1991 Citing 1997-89

FFRDC = Federarly Funded Research & Development Cenler

Fig. 4. Percent of industry paper cites to various sectors' papers 1981 journal set

Clearly the effects are extremely strong, with some 52 percent of the German

physics references to German papers in the same year, only 40 percent in the first year

down to 21 percent for papers published ten years earlier. Similarly in U.S. Clinical

Medicine, 83 percent of the essentially immediate references are to U.S. papers,

whereas after ten years only percent are to U.S. papers. This effect is quite general,

and shows just as strongly for almost any country and for almost any field or subfield,

and obviously could have major effects on an evaluation.

Specifically, if one is evaluating the short time citation rate to non-U.S, papers in a

database that is as heavily U.S. as the Science Citation Index is, then the short term



indicators will give far lower citation ratios to the non-U.S, papers than long term

data, since many of the quick U.S. references are to U.S. papers. This, of course,

would effect any short term citation indicator substitute, such as the impact factor, and

argues strongly for being very careful to adjust for the citation window in any kind of

cross national Citation analysis.

For comparison of institutions within one country this is not as important, but for

cross national comparisons it is quite a striking, important phenomenon.

P e r c e n t o f C i t e s f rc~m U . S . Papecs t o U . S . C l i n i c a l M e d i c i n e

Papers

85

8o

7 0 _ _ -

6 5 l ine represen~ one d t i n o year

Ngmber e~ Ymlrs P l k x t o C ~ l g Year

- - 1 9 9 3

- - - - - - 1 9 9 1

. . . . . . . . 1 9 8 9

. . . . 1 9 8 7

. . . . . . 1 9 8 8

Pmcent of C#es h'om W. Gmmm P~oem to W. Gmman

3 0

2 0

1 0 Each l ine represents one chino yee, r

0 ~ ~ J i i i t i i i t i

Number o f Years Pdor t o CRIng Yelw

1 9 9 3

- -- 1 9 9 1 [

. . . . . . . . 1 9 8 9 [

Fig. 5. "Citation time anomaly"



Linkage indicators for GPRA

A series of very direct indicators of the impact that an agency's science is having

on industrial technology may be developed from linkage statistics, by characterizing

the citations from patented industrial technology to the agency's scientific research

papers. The analysis of linkage from technology to science has been underway for a

number of years, and is becoming a crucially important means of demonstrating the

importance of science via Weinberg's external criteria. This type of analysis of

industrial impact is now being used at a number of research agencies. 13-15 In a paper

published in 1992 we summarized the status of linkage knowledge at that time. 16

When a typical U.S. patent is issued it contains on its front page seven or eight

references to earlier U.S. patents, one reference to a foreign patent, and 1.3 non patent

references (NPR), which are reference to material other than earlier U.S, or foreign

patents. Of these non patent references approximately 43 percent or a total of 60,000

per year in recent years are to scientific research papers in the central core of 3500 or

so journals covered by the Science Citation Index.

There are a number of exceedingly important aspects of this general relationship

from a GPRA viewpoint. First, technology-to-science linkage is increasing rapidly,

with a two to three-fold increase in citations from patents to papers over the last six

years, as shown in Fig. 6, which summarizes the citations from U.S. patents to U.S.

authored papers in two 11 year citation windows. The number of papers cited has more

than doubled, the number of citations to these papers has increased similarly, and the

number of support sources on the cited papers has more than tripled.

This rapidly increasing citation is to rather basic, mainstream science. The citation

is heavily to Level 3 and 4 Applied and Basic research papers; in the Biomedical and

Chemical areas predominately to the Level 4, very basic journals; in the electrical and

computer industry more mixed, with a heavy citation to Level 3 Applied Physics journals.

This citation is also quite rapid, with the age of the cited scientific papers almost as

young as the age of the cited patents. 17 It is largely to public science, to scientific

work done at universities or government laboratories, and supported by government

agencies. The top 10 research support agencies cited in 1993 and 1994 U.S. patents

include 6 NIH Institutes, the National Science Foundation, the American Cancer

Society, the Department of Energy, and the Navy.. This data clearly demonstrates the

crucial contribution that public science makes to patented industrial technology.



# of Support 80000 Sourcas Cited *

70000

60000 ~-

50000

40000

30000 i

i ! 20000 T~

r ~ all cited papers i �9 supported papers ] ; [~ unsupported papers

10000 I

1957/88 1993/94 1987/85 1993/94 1997/85 1993/94 citLng citing citing citing citing citing

1975-85 1981-91 1975-85 1981-91 ~975-85 1981-91

�9 Includes only those papers found in the librar'/

Fig. 6. Citations from U.S. patents to U.S.-authored SCI journal papers

Patent indicators for GPRA

The patents generated by various government laboratories, and by the grantees and

contractors of government agencies, may also be used in performance assessments, just as papers are. As with papers, patents data may be used to construct indicators of

internal and external performance for an agency, internal in that their existence, per se, is an indicator of contributions to technology, and external in that the impact of those

patents on other technology, and ebpecially on U.S. private sector technology, is an indicator of the contribution of those technological inventions to society.



The most basic of the internal indicators are, of course, the numbers and distributions of patents across different technologies. As a general rule government agencies have been relatively steady in their patenting over the last few decades, whereas university patenting has been increasing quite rapidly. Figure 7, for example, shows the rate of patenting by the U.S. government agencies and U.S. universities . Clearly, 9niversities have been increasing their patenting at a very much more rapid rate than~tlae government agencies.

One general point about university patenting in the U.S. is that it is rather heavily concentrated in biotechnology, one of the very, very hot areas of U.S. technology. Other properties of those patents, particularly their relatively high science linkage, also indicate that university patents are in leading edge areas of technology.

2500

21XX)

~ 1513t3

i

m | z 1000

500

US Academic

/ / / 1 \ \

\ \

0 j ~ F ! i I I '

1997 1988 1989 1990 1991 1992 1993 1994 1995

Fig. 7. U.S. patenting trends

As a measure of the quality of patents one can compute various citation indicators. The one we tend to use most is Current Impact Index (CII), which is, in essence, the citation rate for the last five years of an agency's patents, as cited by the most current



year. For example, in computing CII for 1995 we will take all the agency's patents in

each of the five years 1990-1994, note how many times each patent is cited, divide it

by the expected citation rates for all patents in the same product group and year, and

average those ratios, so that a CII equal to 1.0 is the expected value. Science Linkage

(SL), the average number of times each patent references scientific materials, is

another indicator of quality patents.

For example, for the 5 years, 1991-1995, U.S. Navy patents have CII = 0.69 and

SL = 1.09, while U.S. Dept. of Health and Human Services (largely NIH patents)

have CII = 0.75 but SL = 8.10.

While neither agencies' patents are highly cited (expected CII = 1.00) they are

quite science-linked, especially HHS. Overall for all U.S. invented U.S. patents in the

time period the SL = 0.99, so that both NIH and NRL are in relatively highly science

linked areas with NIH, of course, far more heavily science-linked.

Evaluating the efficiency of a government laboratory or program in producing

patents is very problematic, and perhaps impossible. The R&D dollars associated with

each patent produced in a large industrial laboratory tends to be in the one to five

million dollar range, and we have observed similar ranges for some government

laboratories. However, most government laboratories have fundamental missions far

removed from the production of patentable technology, and in many cases the patents

are the product of only certain parts of the labs. Therefore, comparisons of patents per

R&D dollar are far more problematic than comparisons of papers per R&D dollar in

the university environment, where publication is a central objective of the scientific

process.

There are a number of indicators of the external contribution that agency and labo-

ratory patents make, both direct indicators in terms of licensing, and indirect in their

contribution to the advances of technology in general. The indirect contributions can

be tracked by looking at the pattern of citation to laboratories' patents, as is illustrated

by Fig. 8, which shows citations from the top assignees citing to NRL patents,

Another way of looking at the patent impact from an agency viewpoint is to look at

the licensing information, at how many licenses the laboratory has obtained, and how

much revenue is derived from those licenses.

While we don't have detailed licensing revenue information, licensing activity is a

reflection of the growing technology transfer activities of most U.S. labs and

universities, and there is an interesting interaction that we have found between the

bibliometric characteristics of patents and their licenses. In particular, we looked at the

patents which had been licensed for most of the Navy laboratories, and found that

there are two important bibliometric characteristics of licensed patents: first, these



patents are cited much more often than the other lab patents, and second, they tend to

be much more science linked. This is illustrative of a rather interesting convergence of

the internal and external criteria. Those patents which are most highly cited and most

science linked, both internal indicators of the likely impact of that technology, are also the patents that tend to be most heavily licensed, and are therefore making the most

~lirect external contribution to the economy.

I ~ oro.

t US Air . . . . '"J 16 ~

SumJtomo Elec . �9 ~1 !

i , ~ I L . _

I,owa S~. Univ ]13_j .... / . . . / AIcstel Alsthom CGE J/ " Philips / /"

;Si ..... ]/2 "/ //' GE PLC "I'/ Northrup Grurnrnan 8P ,. "

Mitsubishi Elec .~ Raytheon [ NASA Sumitorno Metals

10

ToshibaNEC ]j~/ Fujitsu

NRL 231 Patents

Fig. 8. Companies with 10 or more patents citing to 1985-89 NRL patents

Which brings us to our final point. As was perceived at the very beginning of the

modern age of science, the most important characteristic of research is the quality of the work. Highly cited, science dependent, quality patents, and the related basic scientific work, seem to have wide economic impact. This has also been Shown in the sample cases studied from the beginnings of modern bibliometric analysis, and now is



shown statistically in technology and in its interface with science: high impact, basic scientific research is the type of research directly driving our technological advancement.

References

1. F. NARIN, "Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity", Report prepared for the National Science Foundation, Contract NSF C-627, NTIS Accession #PB252339/AS, (1976), 456pp.

2. A.M. WEINBERG, Criteria for scientific choice II: The two cultures, Minerva, 3 (1964) 3-14. 3. A.M. WEINBERG, Criteria for scientific choice", Minerva, 1 (1963) 159-171. 4. IIT Research Institute, "Technology in Retrospect and Critical Events in Science (TRACES)". Report

prepared for the National Science Foundation, Contract NSF-C535, (1968), 72pp. 5. National Science Foundation, Science Indicators 1972, U.S. Government Printing Office, Reports of

the National Science Board. 6. P.O. SEGLEN, The skewness of science, Journal of the American Society for Information Science, 43

(9), (1992) 628-638. 7. G. P1NSKI, F. NARIN, Citation influence for journal aggregates of scientific publications: theory, with

application to the literature of physics, Information Processing and Management, 12 (5), (1976) 297-312.

8. F. NARIN, A. BREITZMAN, Inventive Productivity, Research Policy, 24 (1995) 507-519. 9. A.J. LOTKA, The frequency distribution of scientific productivity, Journal of the Washington Academy

of Science, 16 (1926) 317-323. 10. F. NARIN, H.H. GEE, "An Analysis of Research Publications Supported by NIH, 1973-1980", NIH

Program Evaluation Report, U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health, (1986).

11. J.D. FRAME, F. NARIN, NIH funding and biomedical publication output, Federation Proceedings, 35 (14) (1976) 2529-2532.

12. F. NARIN, K. STEVENS, E. WHITLOW, Scientific cooperation in Europe and the citation of multinationally authored papers, Scientometrics, 21 (1991) 313-323.

13. L.B. ELLWEIN, P. KROLL, F. NAR1N, Linkage between research sponsorship and patented eye-care technology, To be published (1996).

14. J. ANDERSON, N. WILLIAMS, D. SEEMUNGAL, F. NARiN, D. OLIVASTRO, Human genetic technology: Exploring the links between science and innovation. To be published in Technology Analysis and Strategic Management, (1996).

15. F. NARIN, "Linking Biomedical Research to Outcomes - The Role of Bibliometrics and Patent Analysis", Presented at The Economics Round Table, National Institutes of Health, (1995), 27pp.

16. F. NARIN, D. OLIVASTRO, Status report - linkage between technology and science, Research Policy, 21 (3) (1992) 237-249.

17. F. NARIN, E. NOMA, IS technology becoming science?, Scientometrics, 7 (1985) 369-381.


bibliometric performance measures

Documents