Searching for Globally Feasible Indicators from Domestic Rankings
Ya Lan Xie and Ying ChengGraduate School of Education,
Shanghai Jiao Tong University, China
The Fourth Meeting of International Ranking Expert Group14 - 16 June 2009, Astana, Kazakhstan
Background
• The new global rankings have been published one after another in recent years
• Rankings have problems and global rankings have additional problems, because“Not all nations or systems share the same values and beliefs
about what constitutes ‘quality’ in tertiary institutions, …”5th Item of Berlin Principles
• Consequently, very limited indicators have been used in global rankings
Aim of the study
• Are there some indictors used by domestic rankings that can also be used for global rankings?– Analysis of their International comparability– Examination of their relevance to institutions’
quality
Samples: 11 rankings in 8 countries Title of Ranking Country
Melbourne Institute Index of the International Standing of Australian Universities 2005 Australia
Sunday Times University League Tables 2008 UK
Guardian University Guide 2008 UK
U.S. News America’s Best Colleges 2008 USA
The Top American Research Universities 2007 USA
Maclean's Rankings 2007 CanadaComprehensive Competitiveness Ranking of Chinese Key Universities 2007 China
Netbig Ranking 2008 China
Perspektywy University Ranking 2002 Poland
Slovak Ranking 2007 Slovakia
Romanian University Ranking 2007 Romania
Assignation of indicators into 63 categories
Classification of indicator categories according to their international comparability and universality
Group 1: Comparable at a global level
• Those indicators – EITHER can show universities’ global competence,
e.g.• Publications and citations in international indexes• Globally selected top scholars, such as winners of
International awards, highly cited researchers
– OR are measurements of internationalization• Proportion of international students• Research income from international source
Group 2: Fairly comparable at a global level
• The actual meaning of an indicator is relatively similar in different higher education systems, e.g.– Degrees granted (Bachelor, Master’s, Doctor)– Employment rate of graduates– Spending per student– Domestic publications and patents– Total research income– Proportion of faculties with the highest degree– Students/staff ratio– Student evaluation/satisfaction– ……
Group 3: Hardly comparable at a global level
• The effectiveness of an indicator is heavily affected by different higher education systems, e.g.– Acceptance rate– Number of Master’s or Doctoral programs– Alumni giving rate– Graduation rate– Retention rate– Proportion of full-time faculty– Assessment by domestic administrators/scholars
Group 4: Uncomparable at a global level
• Not universities in all countries have such indicator, e.g.– National Academy Membership– Faculty who won major national awards– Excellent course/textbooks, nationally selected– Excellent programs, nationally selected– Excellent research labs/centers/units, nationally selected– Excellent research products, nationally selected– Honors awarded to graduates– Performance on nationally standardized tests or benchmarks – Research income, national competitive
Number of Indicator Categories in each Group No. of Indicator
Categories %
Total 63 100%
Group 1Comparable 8
59%Group 2
Fairly comparable 29
Group 3Hardly comparable 12
41%Group 4
Uncomparable 14
Frequency of indicators by groups in 11 rankings
Weight of indicators by groups in 11 rankings
The design of indicators for global rankings: TWO PATHS
Can these indicators be actually used in global rankings?
• Group 1: Comparable at a global level– Number of international projects– Books published at international editing houses
• Group 2: Fairly Comparable at a global level– Number of Bachelors granted– Average age of professors
Group 1 & Group 2 indicators that are used by rankings in Three or More Countries
Ranking CountryMelbourne Institute Australia
Sunday Times UK
Guardian UK
U.S. News USA
Top American Res. U. USA
Maclean's Canada
RCCSE China
Netbig China
Perspektywy Poland
ARRA Slovakia
CNCSIS Romania
Number of Countries
Number of Rankings
Students/faculty ratio√√√√
√√√√√
7
9
Spending per student√
√√
√√
√
6
6
Pub. & Cit. in int’l indexes√
√√
√√4
5
Total research income
√√√
√
4
4
Number of Doctors granted√
√
√
√4
4
Employment rate of graduates
√√
√
√
3
4
Proportion of postgraduates
√√√√
3
4
Proportion of professors
√√√√
3
4
Library resources
√√√√
3
4
Proportion of Ph. D. faculty
√
√√
3
3
Proportion of Int’l students
√
√√
3
3
Commonly used indicators
Indicator No. of Countries
No. of Rankings
Also used by
Students/faculty ratio 7 9 THES-QS
Spending per student 6 6
Pub. & cit. in int’l indexes 4 5 ARWU; Taiwan; THES
Total research income 4 4
Number of Doctors granted 4 4
Employment rate of graduates 3 4
Proportion of postgraduates 3 4
Proportion of professors 3 4
Library resources 3 4 Newsweek
Proportion of Ph. D. faculty 3 3
Proportion of int’l students 3 3 THES-QS
Should some indicators be normalized to be more comparable at a global level and how?
• Employment rate of graduates – Normalized by national employment rates or not?
• Spending (of an university) per student – Normalized by GDP per capita or not?
• Total research income – Normalized by Purchasing Power Parity instead of
exchange rate?
Underlying questions
• Does a decision need to be made across countries?
• To what extent a university is a global university?
• Global rankings: To measure universities’ performance as comprehensive as possible OR only to measure their “globalized part”?
Thanks for your attention!