![Page 1: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/1.jpg)
MEASURING ACADEMIC RESEARCH IN CANADA:ALEX USHERHIGHER EDUCATION STRATEGY ASSOCIATES
IREG-7Warsaw, Poland – May 17, 2013
![Page 2: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/2.jpg)
The Problem
When making institutional comparisons, biases can occur both because of institutional size and distribution of fields of study
Can we find a way to compare institutional research output in a way that controls for size and field of study?
![Page 3: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/3.jpg)
YES
![Page 4: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/4.jpg)
Basic methodology Simple 2-indicator system: publication (H-
index) and research income (granting councils)
Data gathered at the level of the individual researcher, not institution
Every researcher given a score for his/her performance relative to the average of his/her discipline. Scores are then summed and averaged.
![Page 5: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/5.jpg)
Publication Metric: H-Index
“A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) papers have no
more than h citations each.” (i.e., the largest possible number N where a scientist has a total of N papers with
N or more citations)
Ex. 2Publication 1: 10 citationsPublication 2: 2 citationsPublication 3: 2 citationsPublication 4: 2 citations
Ex. 1Publication 1: 5 citationsPublication 2: 4 citationsPublication 3: 3 citationsPublication 4: 2 citations
H-Index: 3
H-Index: 2
![Page 6: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/6.jpg)
H-Index (pros and cons)- Pros
- Discounts publications with little or no impact- Discounts sole publications with very high impact
Cons- Requires a large, accurate, cross-referenced database
(labour)- Age bias (less concern on aggregates)- Differences in publication cultures (can be fixed)- Not very useful in disciplines with low publication
cultures
![Page 7: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/7.jpg)
The HiBar Database
Automated collection & calculation
Manual correction
Analysis
Faculty listsStandardized discipline
names
![Page 8: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/8.jpg)
Example: Dr. Joshua Barker
Barker, Joshua D.Associate ProfessorUniversity of Toronto
Social cultural anthropology, violence & power, crime & policing, theories of modernity, anthropology of technology, nationalism, urban studies; Indonesia, South East Asia
• Simple automated search
129 (1000+ pubs)
• Add advanced filtering and Boolean logic
43 (800+ pubs)
• Manual elimination of false positives, excluded publication types, etc.
2 (5
pubs)
![Page 9: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/9.jpg)
The Canadian Prestige Hierarchy
Institution ARWU/THE
Toronto 1British Columbia 2McGill 3Alberta, McMaster, Montreal, Waterloo 2nd tierDalhousie, Laval, Queen’s, Simon Fraser, Calgary, Western, Guelph, Manitoba, Ottawa, Saskatchewan, Victoria
3rd tier
Laval, Carleton, Quebec, UQAM, Concordia Other major institutions
![Page 10: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/10.jpg)
Science-Engineering H-Index
Rank Institution Score Rank
Institution Score
1 UBC 1.509 11 McMaster 1.1972 Toronto – St. G 1.504 12 Trent 1.1603 Montreal 1.500 13 Scarborough 1.1534 McGill 1.327 14 Manitoba 1.0575 Simon Fraser 1.306 15 Trois-Rivieres 1.0546 Waterloo 1.257 16 Alberta 1.0267 Ottawa 1.254 17 Western 0.9968 York 1.208 18 Concordia 0.9929 Queen’s 1.200 19 Laval 0.98910 Rimouski 1.200 20 UQAM 0.967
![Page 11: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/11.jpg)
Arts H-IndexRank Institution Score Ran
kInstitution Score
1 UBC 1.927 11 Concordia 1.2442 Toronto – St. G 1.647 12 Trent 1.2383 McGill 1.629 13 Mississauga 1.2194 Queen’s 1.533 14 Scarborough 1.1925 Alberta 1.370 15 Carleton 1.1626 McMaster 1.364 16 Manitoba 1.1307 York 1.331 17 Montreal 1.0968 Guelph 1.320 18 Calgary 1.0709 Simon Fraser 1.312 19 Saskatchewan 1.05410 Waterloo 1.289 20 Western 1.016
![Page 12: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/12.jpg)
Medicine
We did not cover medical fields
Impossible to do so because manner in which certain institutions choose to list staff at associated teaching hospitals made it impossible to generate equivalent staff lists.
![Page 13: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/13.jpg)
Research Income
Collected data on peer-evaluated individual grants (i.e. major institutional allocations for equipment, etc excluded) made by two main granting councils (SSHRC and NSERC) over a period of three years
Data then field-normalized as per process for H-Index.
![Page 14: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/14.jpg)
Research Income (pros and cons)
- Pros - Publicly available, 3rd party data, with personal
identifiers- Based on a peer-review system designed to
reward excellence
Cons- Issues with respect to cross-institutional awards- Ignores income from private sources which may
be substantial
![Page 15: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/15.jpg)
Science-Engineering IncomeRank Institution Score Ran
kInstitution Score
1 UBC 1.640 11 Guelph 1.2502 Ottawa 1.623 12 McMaster 1.2303 Montreal 1.572 13 Waterloo 1.2294 Alberta 1.465 14 Queen’s 1.2165 Toronto- St. G 1.447 15 Simon Fraser 1.2066 Calgary 1.359 16 Scarborough 1.1877 Rimouski 1.295 17 Carleton 1.1398 Saskatchewan 1.292 18 Western 1.0939 McGill 1.281 19 Sherbrooke 1.01110 Laval 1.272 20 Chicoutimi 0.969
![Page 16: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/16.jpg)
Arts IncomeRank Institution Score Ran
kInstitution Score
1 McGill 2.258 11 Calgary 1.3052 UBC 2.206 12 Dalhousie 1.2633 Montreal 1.944 13 Laval 1.2634 Guelph 1.901 14 Queen’s 1.1055 Alberta 1.895 15 Ottawa 1.0906 McMaster 1.799 16 Waterloo 1.0657 Toronto – St. G 1.733 17 Carleton 0.9918 York 1.615 18 Rimouski 0.9719 Concordia 1.582 19 Scarborough 0.95310 Simon Fraser 1.372 20 Western 0.951
![Page 17: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/17.jpg)
Science-Engineering TotalRank Institution Score Ran
kInstitution Score
1 UBC 100 11 Queen’s 76.852 Montreal 97.63 12 Scarborough 74.403 Toronto – St. G 93.97 13 Calgary 73.264 Ottawa 91.05 14 Laval 71.555 McGill 83.05 15 Saskatchewan 70.156 SFU 80.04 16 Guelph 66.887 Rimouski 79.24 17 Western 66.348 Waterloo 79.14 18 York 65.979 Alberta 78.67 19 Carleton 62.0110 McMaster 77.18 20 Concordia 59.67
![Page 18: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/18.jpg)
Arts TotalRank Institution Score Ran
kInstitution Score
1 UBC 98.84 11 Queen’s 64.252 McGill 92.26 12 Waterloo 57.033 Toronto – St. G 81.83 13 Calgary 56.654 Alberta 77.52 14 Dalhousie 54.095 Guelph 76.35 15 Carleton 51.276 Montreal 75.32 16 Scarborough 51.267 McMaster 75.22 17 Trent 48.368 York 70.29 18 Western 47.429 Concordia 67.15 19 Mississauga 47.1510 Simon Fraser 64.44 20 Ottawa 46.06
![Page 19: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/19.jpg)
Controversies (1)
The double-count issue. In an initial draft, we included a record count of staff rather than a head count (former is higher because of cross-appointments). Led to questions
The part-time professor issue. Many objected to our inclusion of part-time staff in the total. So we re-did the numbers without them…
![Page 20: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/20.jpg)
NSERC Scores (revised)New Rank
Institution Old Rank
New Rank
Institution Old Rank
1 UBC 1 11 Rimouski 72 Toronto-St. G 3 12 McMaster 103 Montreal 2 13 Queen’s 114 SFU 6 14 York 185 McGill 5 15 Guelph 166 Ottawa 4 16 Saskatchewan 157 Alberta 9 17 Manitoba 278 Waterloo 8 18 Trent 219 Laval 14 19 Western 1710 Calgary 13 20 Concordia 20
![Page 21: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/21.jpg)
SSHRC Scores (revised)New Rank
Institution Old Rank
New Rank
Institution Old Rank
1 McGill 2 11 Concordia 92 UBC 1 12 Calgary 133 Toronto-St.G 3 13 Waterloo 124 Guelph 5 14 Laval 215 Alberta 4 15 Ottawa 206 McMaster 7 16 Dalhousie 147 Montreal 6 17 UQAM 438 Queen’s 11 18 Trent 179 Simon Fraser 10 19 Carleton 1510 York 8 20 Western 18
![Page 22: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/22.jpg)
The Philosophical Part
![Page 23: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/23.jpg)
Who is a university?
Whose performance gets included in a ranking says something about who one believes embodies a university. Should it include:
FT faculty only? PT faculty? Emeritus faculty? Graduate students?
At the moment, most ranking systems decision driven by data collection methodology.
![Page 24: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/24.jpg)
Do all subjects matter equally? Field-normalization implies that they do.
But is this correct? Are some fields more central to the creation of knowledge than others? Should some fields be privileged when making inter-institutional comparisons?
![Page 25: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/25.jpg)
Does Size Matter?
Does aggregation of talent bring benefits of its own, independent of the quality of people being aggregated?
![Page 26: Measuring academic research in canada : AleX Usher Higher Education Strategy Associates](https://reader036.vdocuments.site/reader036/viewer/2022081603/56815bef550346895dc9e097/html5/thumbnails/26.jpg)
Where Does Greatness Lie?
On whose work should institutional reputation be based? Its best scholars, or all of its scholars?
Norming for size implicitly rewards schools with good average professors. Failure to norm more likely to reward a few “top” professors