1

Pride and prejudices: Problems with national and international league tables

Dr Paul Greatrix, Registrar, University of Nottingham

2

League Tables

•  Background •  Who wants to know? •  Who are the providers of this valuable

information? •  Mad, bad and dangerous •  National and international dimensions •  Measuring the unmeasurable?

3

Background

•  They sell papers...

•  ...and, some would suggest, create perverse incentives

4

Who wants to know?

•  The Government •  The State •  The Funding Councils

•  Potential Students •  Alumni •  Journalists

•  Parents, teachers, advisors

•  Employers – national and international

•  Overseas sponsors •  Jo Public •  The Universities •  Dangerous obsessives

5

Who’s responsible for providing this valuable data? •  The Times

•  Sunday Times

•  The Guardian

•  Complete University Guide

•  Financial Times

•  THE

•  Shanghai Jiao Tong University

•  QS •  US News and World Report •  HEFCE •  Others... •  and, of course,

Government

6

Invaluable information

•  For all stakeholders •  Intelligent decision-making •  Better than prejudice

•  Reflecting the realities of the market place •  We have a right to know •  Employers have right to know •  We aren’t stupid

7

League tables are a bad thing...

“The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”

8

But we’ll use them anyway!

“Having said that, I’m not ashamed to report that we came a very creditable 79th overall, with my own department rating a particularly good score for research

- and as I remarked to the Dean, you can’t get much better than that.”

9

Dangerous... •  Criteria used do not reflect quality of education •  Historical data •  Variation over time

•  Scores are institutional averages – mask strengths •  Distorting effect of weightings and scalings and

data manipulation •  Many of criteria used are inter-related

(ie not independent)

10

... extremely dangerous... •  Apples and elephants and

paperclips •  Perverse incentives •  Hugely political •  Open to manipulation •  Delivered by journalists •  Spurious precision •  Serious consequences for

universities, departments, staff and students

11

The Times 2010

1 Oxford 2 Cambridge 3 Imperial 4 St Andrews 5 LSE 6 Durham 7 UCL 8 Warwick 9 York 10 Lancaster

11 Edinburgh 12 Exeter 13 Bath 14 Bristol 15 Leicester 16= Loughborough 16= King’s 18 Sheffield 19 Southampton 20 Nottingham

12

Sunday Times 2010

1 Oxford 2 Cambridge 3 Imperial 4 UCL 5 LSE 6 Durham 7 St Andrews 8 Warwick 9 Bath 10 King’s

11 Bristol 12 Nottingham 13 York 14 Edinburgh 15 Sheffield 16 Loughborough 17 Exeter 18= Southampton 18= Birmingham 20 Newcastle

13

The Guardian 2010

1 Oxford 2 Cambridge 3 Warwick 4 St Andrews 5 UCL 6 Lancaster 7 Imperial 8 LSE 9= Loughborough 9= York

11 SOAS 12 Leicester 13 Bath 14 Exeter 15= Sussex 15= Edinburgh 17 Durham 18 Southampton 19 UEA 20 Surrey

Tuesday, April 19, 2011 14

Complete University Guide 2011

1 Cambridge 2 Oxford 3 Imperial 4 LSE 5 Durham 6 St Andrews 7 UCL 8 Warwick 9 Lancaster 10 Bath

11 Bristol 12 York 13 Edinburgh 14 Southampton 15 Exeter 16 King’s 17 Nottingham 18 SOAS 19= Loughborough 19= Sussex

15

And a completely different approach: UEL: TQ in the South East 2005

1  East London 4 2  King’s College London 2 3  Brunel 2 4  Queen Mary, London 0 5  Kent 0 6  Hertfordshire 0 7 Royal Holloway -1 8  London South Bank -3 9 Greenwich -6 10 Essex -6 11 Anglia Polytechnic -9

Surplus of good practice over recommendations (in QAA audit reports)

16

International value-added

•  HE is now a global business •  Global branding assisted by competitive

ranking •  International benchmarking increasingly

important, especially in research •  Student recruitment is increasingly

international... •  ...mobile students are increasingly

choosy •  It’s all good healthy fun

17

Just as dangerous... •  Again, the criteria used do not reflect

quality of education •  Archaic and irrelevant data •  Major biases to large, English-

speaking, research- and science-intensive universities

•  Institutional scores are extraordinarily broad brush

•  Distorting effect of weightings, scalings and data manipulation

•  Far from comprehensive surveys of peers and employers

18

Times Higher Education World Rankings 2010

6= Cambridge 6= Oxford 9 Imperial College 22 University College London 40 Edinburgh 68= Bristol 77 King’s

79= Sussex 81= York 85 Durham 86 LSE 87 Manchester 88 Royal Holloway 90= Southampton

19

1 Cambridge (2) 4 UCL (4) 6 Oxford (5=) 7 Imperial (5=) 21 King’s (23) 22 Edinburgh (20=) 27 Bristol (34) 30 Manchester (26) 53 Warwick (58) 59 Birmingham (66)

69 Sheffield (82) 73 Nottingham (91) 77 Glasgow (79) 80 LSE (67=) 81 Southampton (95=) 85 Leeds (99) 88 York (70=) 92 Durham (103=) 95 St Andrews (87=)

QS World Rankings 2010

20

5 Cambridge (4)

10 Oxford (10)

21 UCL (21)

26 Imperial (26)

44 Manchester (41)

54 Edinburgh (53)

63 King’s (65)

66 Bristol (61)

84 Nottingham (79)

88 Sheffield (81)

99 Birmingham (94)

Shanghai Jiao Tong 2010

21

1 Harvard University 2 Stanford University 3 MIT 4 University of California, Los Angeles 5 University of California, Berkeley 6 University of Michigan 7 University of Washington 8 University of Pennsylvania 9 Johns Hopkins University 10 University of California, San Diego

11 Columbia University 12 University of Minnesota 13 University of Cambridge 14 University of Toronto 15 University of Chicago 16 Cornell University 17 University of Oxford 18 University of Wisconsin, Madison 19 Yale University 20 Pennsylvania State

‘High Impact Universities’ 2010

22

15 Cranfield 17 Henley 19 London Business School 29 Bath 30 Cambridge - Judge 34 Warwick 55 Durham 61 Manchester

67 Ashridge 68 Birmingham 71 Oxford – Said 74 Aston 75 Edinburgh 77 Strathclyde 89 Lancaster 97 Leeds

Economist FT MBA Rankings 2010

23

1 Harvard 10 Ecole des Mines de Paris 11 Oxford 26 Manchester 35= Queen’s 35= Glasgow

60= Cambridge 89= Heriot-Watt 89= Bath 89= Huddersfield 89= Sheffield 89= Uni of Wales:

Swansea (and over 100 others…)

Another approach: Ecole des Mines de Paris Rankings

24

1 University of California, Berkeley

2 University of Nottingham

3 York University

4 Northeastern University

5 Cornell University

6 Universiti Putra Malaysia

7 Washington University In St. Louis

8 Georgia Institute of Technology

9 University of Wisconsin-Madison

10 University of North Carolina,

Chapel Hill

11 Bangor University

12 University of Sussex 13 Escuela Superior Politécnica del Litoral

14 University of Kent

15 Universitas Indonesia

16 Universidad de Alcalá 17 King Mongkut´s University of Technology Thonburi

18 Institut Teknologi Sepuluh November

19 Hokkaido University

20 National University of Singapore

Which league table?

25

1 Greenwich 2 Sheffield Hallam 3 Kingston 4 Westminster 5 East London 6 Central Lancashire 7 Leeds Met 8 Wolverhampton 9 Coventry 10 Middlesex

Which league table?

26

1 Loughborough 2 Sheffield 3 UEA 4 Cambridge 5 Dundee 6 Oxford 7 Glasgow 8 Leeds 9 Aberystwyth 10 Southampton

Which league table?

27

1 Webb Institute 70.9 2 Carleton College 61.3 3 Princeton University 60.3 4 Middlebury 60.1 5 Amherst 59.5 6 Williams College 57.6 7 Centre College 56.7 8 Indiana Institute of Technology 55.1 9 Davidson College 54.9 10 Thomas Aquinas College 52.5

Which league table?

28

1 Massachusetts Institute of Technology 2 Stanford University 3 Harvard University 4 University of California Berkeley 5 Cornell University 6 University of Michigan 7 California Institute of Technology 8 University of Minnesota 9 University of Illinois Urbana Champaign 10 University of Texas Austin

Which league table?

29

1 University of Cambridge, World Rank: 28 2 Swiss Federal Institute of Technology WR: 40 3 University of Oxford WR: 42 4 University of Helsinki WR: 58 5 University of Oslo WR: 60 6 University of Edinburgh WR: 65 7 University College London WR: 68 8 Utrecht University WR: 71 9 University of Manchester WR: 82 10 Royal Institute of Technology (Sweden) WR: 84

Which league table?

30

International approaches 1

Shanghai Jiao Tong: Ranking of World Universities –  6 indicators covering: quality of education;

quality of faculty; research output; performance relative to size

–  Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science; articles in citation indices.

31

International approaches 2

32

Other tables

•  ALOHA, PISA, OECD •  China – number of billionaires •  Just research performance: ranking of

scientific papers •  Most conservative, best parties, most interns •  European initiative – U-Multirank •  Islamic universities

33

An indicator too far (not covered in the UK tables - yet)

•  Alumni giving •  Academic staff pay •  Research income •  Citations •  Brand impact

But international tables will increasingly influence methodologies of UK tables

34

What is to be done •  Publish prolifically and get top RAE grades •  Be Highly Cited •  Get articles in Nature and Science •  Win a Nobel Prize or a Fields Medal •  Make sure your students all get top jobs with big multinationals (and

then win Nobels) •  Plug the institution relentlessly and cultivate peers, Headmasters/

mistresses and employers •  Improve NSS results (and SSRs) - every little helps •  Spend more on Library and IT and everything else to do with teaching

and learning •  Recruit more international students and staff (all with Nobel potential)

35

Conclusions •  They aren’t going to go away

•  The international dimension will become increasingly significant

•  Methodologies - for both national and international tables - are all dubious, at best

•  They can and will be used by many different groups – but can be dangerous in the wrong hands

•  Handle with great care!

36

Finally For a copy of the presentation and further league table commentary and observations, visit:

http://registrarism.wordpress.com/

Also on Twitter: @registrarism