the pbrf and bibliometric measures
DESCRIPTION
The PBRF and bibliometric measures. Introduction. Use of bibliometrics by the Ministry of Education Part 1 – presentation of latest bibliometric results Part 2 – what if bibliometrics was used to allocate PBRF funding? – a case study. Bibliometrics. Thomson Reuters bibliometrics dataset - PowerPoint PPT PresentationTRANSCRIPT
The PBRF and bibliometric measures
Introduction
• Use of bibliometrics by the Ministry of Education
• Part 1 – presentation of latest bibliometric results
• Part 2 – what if bibliometrics was used to allocate PBRF funding? – a case study
Bibliometrics• Thomson Reuters bibliometrics dataset
– Reasonably stable– Long time series– External to the PBRF system – can triangulate and calibrate– International benchmarking– Limitations of bibliometrics
• Useful monitoring tool– But don’t see bibliometrics as a way of dispensing research
funding at this time
Bibliometrics measures – NZ universities
– Share of world publications– Share of world citations– Relative impact:
(citations per paperNZ universities)
(citations per paperworld)
Share of world publications and citations
Source: Thomson Reuters
0.0%
0.1%
0.2%
0.3%
0.4%
0.5%
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Sha
re o
f w
orld
Publications Citations
Relative impact – NZ vs Australia
Source: Thomson Reuters
0.0
0.2
0.4
0.6
0.8
1.0
1.2
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Rel
ativ
e im
pact
(wor
ld a
vera
ge =
1)
NZ G8 Non-G8
Relative impact – by panel
Source: Thomson Reuters
0.0
0.2
0.4
0.6
0.8
1.0
1.2
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Rel
ativ
e im
pact
(wor
ld a
vera
ge =
1)
Engineering Maths Physical science
Relative impact – by panel
Source: Thomson Reuters
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Rel
ativ
e im
pact
(wor
ld a
vera
ge =
1)
Biological sciences Medicine & public heath Health
Relative impact – by panel
Source: Thomson Reuters
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Rel
ativ
e im
pact
(wor
ld a
vera
ge =
1)
Business Education
Relative impact – by panel
Source: Thomson Reuters
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
81-85 83-87 85-89 87-91 89-93 91-95 93-97 95-99 97-01 99-03 01-05 03-07
Rel
ativ
e im
pact
(wor
ld a
vera
ge =
1)
Social sciences Humanities & law
Relative impact – by panel
Source: Thomson Reuters
Engineering
Maths
Physical science
Biological sciences
Medicine & public heath
Health
Business
Education
Social sciences
Humanities & law
World 1998-2002 2003-2007
NZ vs Australian universities 2003-2007
Engineering
Maths
Physical science
Biological sciences
Medicine & public heath
Health
Business
Education
Social sciences
Humanities & law
World NZ G8 Non-G8Source: Thomson Reuters
What if citations were used to allocate PBRF funding?
• Case study – Earth Science
• Method– Compare funding allocated to each university in
2007 via the PBRF quality evaluation with what would have been allocated if based on citations
Funding via the PBRF 2007University PBRF score FTE PBRF funding
($000s)
Lincoln 3.0 8.0 $146
Massey 4.7 15.7 $453
Auckland 4.6 26.5 $742
Canterbury 5.4 21.0 $695
Otago 5.5 17.7 $598
Waikato 5.0 18.3 $555
VUW 4.9 25.8 $763
Total $3,952
Funding via relative impact in 2007
• Relative impact in five year overlapping time periods
• Use three year weighted average– 1999-2003 (15%)– 2000-2004 (35%)– 2001-2005 (50%)
• Then adjust by PBRF FTE
Relative impact
University 99-03 00-04 01-05 Weighted
Lincoln 1.04 0.56 1.04 0.87
Massey 0.73 0.50 0.60 0.59
Auckland 0.77 0.82 0.84 0.82
Canterbury 0.73 0.78 0.64 0.70
Otago 1.68 1.63 1.28 1.46
Waikato 0.97 1.06 1.05 1.04
VUW 1.10 0.96 1.02 1.01
Source: Thomson Reuters
Comparing PBRF scores and weighted relative impact
0.0
0.5
1.0
1.5
0 1 2 3 4 5 6
PBRF average quality score
Rel
ativ
e im
pact
OU
AU
VUWWU
CU
LU
MU
Comparing funding allocations 2007
$0.0
$0.2
$0.4
$0.6
$0.8
$1.0
VUW Auckland Canterbury Otago Waikato Massey Lincoln
$mill
ion
PBRF Relative impact
Issues
• Capturing all indexed research vs 4 nominated research outputs
• Size of NZ subject areas
University staffing trends
0
500
1,000
1,500
2,000
2,500
3,000
Professors AssociateProfessors
Senior Lecturers Lecturers Research OnlyStaff
2001 2002 2003 2004 2005 2006 2007
Ministry of Education monitoring
• Ministry of Education reports assessing research performance can be found at the Education Counts website:
www.educationcounts.govt.nz