bibliometric measures to demonstrate impact

Prestigious Fellows SeminarThe University of Queensland

Dr Berenika M Webster13 April 2010

Using bibliometric measures to demonstrate the impact of

research

James McKeen Cattell and American Men of Science

1

Hulme’s statistical bibliography

2

Samuel C Bradford’s law of scattering

3

Derek de Solla Price’s Little Science, Big Science

4

Sociology of science at Columbia University

5

Eugene Garfield’s “association of ideas index”

6

(Bibliometric* OR Citation*) AND Evaluation

Publication year

Number of publications

Publication year

Number of publications

1963 1 1993 71967 1 1994 101972 1 1995 221976 3 1996 241979 2 1997 181980 4 1998 211981 5 1999 231982 4 2000 301983 4 2001 241984 2 2002 361985 2 2003 271986 5 2004 291987 3 2005 341988 5 2006 331989 3 2007 681990 9 2008 681991 10 2009 1091992 22 2010 25

RAE in UK

RQF in Australia

ERA and REF

Source: Web of Science, 20107

Things to look out for…

8

Can the available data answer the question?Are you comparing like with like?Relative and multiple measures.Do the results make sense?

Can the available data answer the question?

Disciplinary differencesPatterns of scholarly communicationInterdisciplinarity

Place vs. people

Source of publication informationExternal data sourceUniversity records Individual’s CV

Data set – size matters

9

Compare like with like – two case studies

Cancer research charities

Two schools of psychology

10

Absolute measure – volume of outputs

11

Relative measure: % of world output

12

Relative measures: subject profile of a researcher

Subject area Paps CitsAuthor

CPP Aus CPP Wld CPP Aus RCI WldRCIREPRODUCTIVE BIOLOGY 19 379 19.95 13.27 12.47 1.50 1.60DEVELOPMENTAL BIOLOGY 3 106 35.33 17.82 26.68 1.98 1.32OBSTETRICS & GYNECOLOGY 4 104 26.00 5.15 4.75 5.05 5.47CELL BIOLOGY 2 74 37.00 19.26 20.93 1.92 1.77AGRICULTURE, DAIRY & ANIMAL SCIENCE 1 83 83.00 4.85 6.19 17.11 13.41ENDOCRINOLOGY & METABOLISM 6 67 11.17 11.76 9.44 0.95 1.18VETERINARY SCIENCES 3 23 7.67 1.96 1.58 3.91 4.85ZOOLOGY 1 12 12.00 6.86 6.41 1.75 1.87IMMUNOLOGY 1 2 2.00 14.7 13.58 0.14 0.15PATHOLOGY 1 0 0.00 6.1 5.15 0.00 0.00Total 41 850 20.73 10.173 10.718 2.04 1.93

Source: Custom Citation Report, 201013

Multiple measures: how does my organisation compare against others?

Source: InCites (UQ data set), 2009

Publications Citations CPPAgricultural Sciences 1 2 7Biology & Biochemistry 1 1 3Chemistry 3 6 6Clinical Medicine 3 4 5Computer Science 3 1 2Economics & Business 5 4 5Engineering 5 5 5Environment/Ecology 1 1 3Geosciences 6 6 5Immunology 4 3 5Materials Science 3 1 2Mathematics 5 5 5Microbiology 1 2 5Molecular Biology & Genetics 2 1 1Neuroscience & Behavior 5 4 2Pharmacology & Toxicology 3 4 5Physics 5 4 2Plant & Animal Science 1 1 7Psychiatry/Psychology 1 3 5Social Sciences, general 2 3 3Space Science 7 6 2

Subject Area Rank

Source: InCites (UQ/Glasgow Univ data sets), 200915

Do the results make sense?

Use healthy scepticism

Is the data used relevant to the question asked?Are the methods used robust?Are the conclusions not reaching beyond the

limits of the data collected and methods applied?

Are all caveats stated clearly?

Publication and citation data should be a supplement to peer review

16