EVOLVING RESEARCH EVALUATIONRAE2008 OUTCOMES AND REF PROSPECTS

JONATHAN ADAMS

10 NOVEMBER 2009

Evaluation

• Impact– Indicators vs profiles

– Decision support, not substitution

• What variables are relevant in ...– Academic

– Economic

– Policy

– Social ... Impact

• Objective impact evaluation is constrained by a lack of reference data outside ‘academic’

Thomson Reuters and Education

• Web of Science– 250 + journal categories

– 3 relate to ‘education’• Education & educational research – 210 journals• Education, disciplines – 37 journals• Education, special – 36 journals

– Coverage is back-filled

– Regional spread is diversifying

Sample coverage• Thomson Reuters coverage has broadened

– Alberta journal of educational research

– American journal of education

– Asia Pacific journal of education

– Australian journal of early childhood

– British educational research journal

– Chinese education and society

– Didactica Slovenica - pedagoska obzorja

– Durham and Newcastle research review

– Egitim ve bilim - education and science

– European journal of Dental education

All the way through to

– Revista Brasileira de Ensino de Fisica

– Zeitschrift fur Soziologie der Erziehung und Sozialisation

UK coverage

• National Citation Report

• Evidence tracks all author addresses and reconciles to ‘real’ organisations

• Profiles for each UK university

Steady but volatile improvement

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Cita

tion

impa

ct re

lativ

e to

wor

ld

Russell Group

UK total

What is used to represent excellent research outcomes?

RAE1996Science Engineering Social sciences Humanities and arts

Outputs % Outputs % Outputs % Outputs %

Books and chapters 5,013 5.8 2,405 8.1 16,185 35.1 22,635 44.4

Conference proceedings 2,657 3.1 9,117 30.8 3,202 6.9 2,133 4.2

Journal articles 77,037 89.8 16,951 57.3 22,575 49.0 15,135 29.7

Other 1,104 1.3 1,122 3.8 4,154 9.0 11,128 21.8

RAE2001

Books and chapters 1,953 2.5 1,438 5.4 12,972 28.6 25,217 46.5

Conference proceedings 751 0.9 3,944 14.9 857 1.9 1,619 3.0

Journal articles 76,182 95.8 20,657 78.1 29,449 65.0 17,074 31.5

Other 618 0.8 408 1.5 2,008 4.4 10,345 19.1

RAE2008

Books and chapters 1,048 1.2 216 1.2 12,632 19.0 21,579 47.6

Conference proceedings 2,164 2.5 326 1.8 614 0.9 897 2.0

Journal articles 80,203 93.8 17,451 95.4 50,163 75.5 14,543 32.1

Other 2,125 2.5 301 1.6 3,018 4.5 8,287 18.3

RAE2008 grade spread

Staff FTE 4* % 3* % 2* % 1* % U

HEIs >= 20% 4* [13] 49.2 23.8 35.0 29.6 10.8 0.8

HEIs < 20% 4* [68] 15.5 5.4 20.4 33.8 30.0 10.1

Education UoA 45 1695.9 15.1 27.5 33.1 20.1 4.2

OVERALL 52409 17 37 33 11 1

Golden Triangle 5 7794 30 40 25 5 1

Russell Group excl 5 17788 18 42 32 8 1

1994 Group 9392 18 39 33 9 1

University Alliance 5633 10 30 38 19 2

Million + 3424 6 23 39 27 4

Previously among studies by Evidence

• HEFCs/UUK – Maintaining Research Excellence– Recognition of a peak supported by a research platform

– Evolution of 3 to 4 to 5 grade units

• HEFCE – Role of QR funding– Strategic significance of the block grant

• Universities UK – Excellence and Diversity– Regional network of research competence

– Importance of an evidence base for research policy

• OST/DIUS/BIS– Diversity as a critical element in research policy

RAE2008 – a novel outcome with dispersed rewards

• Gareth Roberts’ proposals on research profiling– Problem of the ‘cliff edge’

– Recognising dynamic excellence in the research base

• Unexpected differences between panels– The problem of interpreting methodology and standards

– The absence of a commonly understood standard?

– The dispersed map of reward-able research

• Decisions about funding allocations– 2* = 1 (x 3 to) 3* = 3 (x 2.33 to) 4* = 7

– Greater relative rewards for modest gains than for very expensive world-class excellence (cf 2001)

How can we explore the spread of excellence?

• We have – Panel outcomes from RAE2008 – but opinions differ, so ...

– Bibliometric data on a consistent basis from 1981

• We split the HE sector into crude groups– All HEIs

– Russell Group

– 1994 Group

– Universities with a shorter history of research investment

• And we created an elite ‘golden triangle’ band– Oxford (C12th), Cambridge (1209), UCL (1836), Imperial

(C19th) , LSE (1895)

Excellence is linked to selectivity

15

20

25

1.1

1.2

1.3

1.4

1981 1986 1991 1996 2001 2006

Share (%) of U

K research output co-authored by golden triangle

Rela

tive

impa

ct o

f UK

rese

arch

pub

licati

ons

Papers co-authored by five leading institutions

UK citation impact

5 year movingaverage

1989 1992

And excellence is concentrated

UK output Total articles and reviews, 2002-2006

Papers above world average impact

Papers with impact exceeding four times world average

Papers with at least one co-author from the HE research base

306,661112,358

37% of HE total

16,060

5.2% of HE total

Papers with at least one co-author from the Russell Group

204,307

66% of UK HEIs’ total

79,505

70% of HE output

39% of Russell total

11,734

73% of HE output

5.7% of Russell total

Papers with at least one co-author from a ’golden triangle’ HEI

87,157

28% of HEIs’ total

37,370

33% of HE output

43% of GT total

6,308

39% of HE output

7.2% of GT total

The problem with simplistic indicators

• They don’t really express the complexity of research performance

• Average impact (e.g. ‘crown indicator’) can be very misleading– Research Council studies reveal error of interpretation

– Skewed data, median much smaller than average

– Lots of papers are not cited

– The interesting bit is about how much is really, really cited lots

• So we we prefer Impact Profiles®

UK background and ‘golden triangle’

0

5

10

15

20

25

uncited RBI > 0 < 0.125 RBI ≥ 0.125 < 0.25

RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1 RBI ≥ 1 < 2 RBI ≥ 2 < 4 RBI ≥ 4 < 8 RBI ≥ 8

Pe

rce

nta

ge

of o

utp

ut

20

02

-2

00

6

UK higher education sector, all research fields - 306661 papers UK 'golden triangle', all research fields - 87157 papers

This is the small but critical excess of really highly cited research output

Even smaller differences separate the UK and USA research profiles

0

5

10

15

20

25

30

RBI = 0 RBI > 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cent

age

of o

utpu

t 199

6-20

05

UK 10 Years - 686476 Papers

USA 10 Years - 2888315 Papers

Excellence extends to other institutions

0

5

10

15

20

uncited RBI > 0 < 0.125 RBI ≥ 0.125 < 0.25

RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1 RBI ≥ 1 < 2 RBI ≥ 2 < 4 RBI ≥ 4 < 8 RBI ≥ 8

Pe

rce

nta

ge

of o

utp

ut

20

02

-2

00

6

UK higher education sector, all research fields - 306661 papers Russell Group, all research fields - 204307 papers

UK 'golden triangle', all research fields - 87157 papers

But the differences become very fine

0

5

10

15

20

uncited RBI > 0 < 0.125 RBI ≥ 0.125 < 0.25

RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1 RBI ≥ 1 < 2 RBI ≥ 2 < 4 RBI ≥ 4 < 8 RBI ≥ 8

Pe

rce

nta

ge

of o

utp

ut

20

02

-2

00

6

UK higher education sector, all research fields - 306661 papers Russell Group, all research fields - 204307 papers

1994 Group, all research fields - 58083 papers

And if we distil further, other complexity is revealed

0

5

10

15

20

uncited RBI > 0 < 0.125 RBI ≥ 0.125 < 0.25

RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1 RBI ≥ 1 < 2 RBI ≥ 2 < 4 RBI ≥ 4 < 8 RBI ≥ 8

Pe

rce

nta

ge

of o

utp

ut

20

02

-2

00

6

Russell-5, all research fields - 126885 papers 1994 Group, all research fields - 58083 papers

UK 'golden triangle', all research fields - 87157 papers

Older institutions have no monopoly on quality, but the peak is distinctive

0

2

4

6

uncited RBI > 0 < 0.125 RBI ≥ 0.125 < 0.25 RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1 RBI ≥ 1 < 2 RBI ≥ 2 < 4 RBI ≥ 4 < 8 RBI ≥ 8

Perc

enta

ge o

f out

put 2

002

-200

6

UK higher education sector - 306661 papers 1994 Group, all research fields - 58083 papers

Russell-5, all research fields - 126885 papers UK 'golden triangle', all research fields - 87157 papers

For the 1994 Group, there are relatively fewer low-cited papers and relatively more high-cited papers than for many in the Russell Group

What does this imply?

• There is a very concentrated peak of exceptional excellence

• UK international competitiveness is associated with selectivity that increased concentration

• There is a regional network of exceptional quality, much in dynamic institutions created in the 1960s

• The balance that supports diversity & dynamism and the concentrated support of international excellence is very fine

• Model first, meddle second

EVOLVING RESEARCH EVALUATIONRAE2008 OUTCOMES AND REF PROSPECTS

JONATHAN ADAMS

10 NOVEMBER 2009