policies to measure and reward hei knowledge transfer ... rossi.pdf · source: rosli and rossi,...

19/09/2016

1

Policies to measure and reward knowledge transfer performance:

Do they work for diverse HEI systems?

Federica Rossi

Centre for Innovation Management Research, Birkbeck College, University of London

Centro per la Valutazione delle Politiche Pubbliche, Universita’ di Modena e Reggio Emilia

Policies to measure and reward HEIs’ knowledge transfer (KT) performance

• Increasingly used to incentivise HEIs to engage in KT

• Using empirical evidence from the UK, I look at some of the

problems inherent in these instruments, as:

– Creating the “right” incentives is not easy: HEIs’ opportunities

for KT engagement are very diverse

– National policies can mobilize more resources and can be more

influential on practices and institutional cultures than local ones,

but they risk imposing a “one size fits all” approach

19/09/2016

2

Advantages of measuring HEIs KT performance

• Provide objective knowledge that can drive evidence-based

policy

• Signal the importance of the KT mission, thus inducing HEIs to

embrace a culture that is accepting of KT

• Help HEIs improve their performance, by enabling

benchmarking and allowing them to emulate the strategies of

more successful competitors

• Improve system’s efficiency by funding best performers

HEIs KT performance measurement:

• Must include a broad enough variety of KT activities to reflect

the diverse HEI KT profiles

• Must allow comparability between institutions: different

outcomes must reflect genuine performance differentials rather

than institutional characteristics (such as size and subject mix)

• Must avoid undesirable behavioural incentives, such as

incentivising HEIs to engage only in certain KT activities, even

when these may not be appropriate for them, or conflict with

other HEI missions

19/09/2016

3

UK case: performance-linked policy instruments to incentivise HEIs’ KT engagement

• Higher Education Business and Community Interaction (HEBCI)

survey: collects data on HEIs KT performance (1999-)

• Higher Education Innovation Fund (HEIF): awarded to HEIs based

on their KT performance (2001-)

• Research Excellence Framework (REF): assigns part of research

funding based on research impact (2014-)

• HEBCI survey includes a broad range of KT activities when

compared with similar exercises

Areas of KT activity HE-BCI (UK) AUTM

(US/Canad

a)

NSRC

(Australia)

ProTon

(Europe)

Strategy x x

Infrastructure x x x x

Intellectual property x x x x

Spin offs x x x x

Collaborations * x x x x

Education x

Facilities and equipment related

services

x x

Social, community and cultural

engagement

x

Include: collaborative research, regeneration programmes, contract research, consultancy

Source: Rosli and Rossi, 2015

19/09/2016

4



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

HE-BCI (UK) AUTM (US) NSRC

(Australia)

ProTon

(Europe)

% q

ue

sti

on

s

Strategy and

infrastructure

Intellectual property

and spinoffs

Collaborations

Education

Facilities & equipment

related services

Social, community &

cultural engagement





0

10

20

30

40

50

60

HE-BCI (UK) AUTM (US) NSRC

(Australia)

ProTon

(Europe)

N q

ues

tio

ns

Strategy and

infrastructure

Intellectual property

and spinoffs

Collaborations

Education

Facilities & equipment

related services

Social, community &

cultural engagement

19/09/2016

5

• Since mid-2000, HEBCI results are used as a basis for HEIF

allocation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

20

01-2

00

4

20

04-2

006

20

06-2

008

2008

-2011

2011-2

015

20

15-2

016

Perf

orm

nce s

co

re f

or

HE

IF

allo

ca

tio

n

Formula: KT

income

Formula: Activities

not best measured

by income

Formula: Potential

& capacity building

Competitive

Bidding

Since 2011 only institutions with £250k + income are eligible for funding


• HEIF allocates about £130 million per year to UK HEIs

– HEIF funds contribute to 34% of UK HEIs KT income (Coates Ulrichsen,

2014)

Source: Coates Ulrichsen, 2014

Knowledge Exchange Funding in England

11

3 Knowledge Exchange Funding in England

Previous research has found that HEFCE KE funding has played an important role in helping HEIs

build up their capacity and capability to engage with users to exchange knowledge and deliver

economic and social benefits from the knowledge base (PACEC/CBR, 2009). HEFCE KE funding is the

main government dedicated funding stream provided to HEIs to support their KE activities. It has

few restrictions other than to support KE with any form of external partner to achieve maximum

economic and social benefit for the country, allowing HEIs flexibility to deploy the funding to meet

their specific needs and circumstances. HEFCE KE funding was maintained in cash terms at £150

million per year over the period 2011/12–2014/15 in the last Comprehensive Spending Review,

emphasising the UK Government’s commitment to supporting KE in English HEIs (Figure 3.1). This is

pumping £600 million into the HE sector in support of KE. The amount of funding provided by HEFCE

in support of KE over the period 2000/01–2011/12 now totals £1.34 billion at constant 2011 prices.

Figure 3.1 HEFCE knowledge exchange funding evolution 2000/01–2014/15

0

20

40

60

80

100

120

140

160

Total KE income HEIF funding Other KE income

HEIF

Total KE funding

Kn

ow

led

ge

exc

ha

ng

e f

un

din

g(£

mil

lio

ns,

co

nst

an

t 2

00

9/1

0 p

rice

s)

Other HEFCE KE funding

HEIF 4 HEIF 2011-15

Source: PACEC (2012) Strengthening the Contribution of English Higher Education Institutions to the Innovation System:

Knowledge Exchange and HEIF Funding: A report for HEFCE

3.1 Distribution of HEFCE knowledge exchange funding

HEFCE KE funding was originally distributed through a range of funding schemes delivered

through a competitive bidding system, each with different objectives. Over time, these were

amalgamated into a single funding stream – HEIF, and were increasingly allocated by formula.

By 2008/09 and the HEIF4 allocation, all HEIs in England received some funding to support KE

determined by a formula based in part on the scale of the institution (reflecting a capacity

building goal) and partly on performance (providing an incentive to increase KE activity). The

latest round of HEIF – HEIF 2011-15 - saw further changes to the allocation method. Although

still allocated purely by formula, three key changes were introduced. The first was a threshold

value of KE activity below which HEIs would not receive any funding; the second was an

19/09/2016

6

Issues with policy instruments / 1: Survey provides partial view of HEIs KT activities

• Lack of activities involving students and personal interactions

• Knowledge-intensive services (prototyping, clinical testing,

design) are grouped with non-knowledge producing services

(room and equipment rental)

• Focus of the survey has narrowed over time:

– More IP & spinoffs, less strategies & infrastructure

– More quantitative indicators, less qualitative information

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

19

99/2

00

0

200

0/0

1

200

1/0

2

200

2/0

3

2003

/04

200

4/0

5

200

5/0

6

200

6/0

7

200

7/0

8

200

8/0

9

200

9/1

0

20

10/1

1

20

11/1

2

201

2/1

3

2013

/14

% q

uesti

on

s

Strategy and

infrastructure

Intellectual

property and spin

offs

Collaborations

Education

Facilities and

equipment related

services

Social, community

and cultural

engagement


19/09/2016

7

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

19

99/2

00

0

2000

/01

2001

/02

2002

/03

2003/0

4

2004

/05

2005

/06

2006

/07

2007

/08

2008

/09

2009

/10

20

10/1

1

20

11/1

2

2012

/13

2013/1

4

% q

uesti

on

s

Quantitative

information:

Qualitative

information


Issues with policy instruments / 2: KT income not a good proxy for impact

“income remains the best proxy we have for the impact of KE activities on the

economy and society; hence it is the best measure of performance and will be used

in the formula allocation of HEIF” (HEFCE, 2011)

• But:

• Not all income-producing activities are included in the survey

• Some non-income producing activities are impactful (e.g. public engagement:

lectures, performance arts, exhibitions, museum education)

• Low income does not always mean low impact (presence of externalities,

uncertainty, nature of the intended beneficiaries)

• High income does not always mean high impact (higher cost of production,

premium for reputation)

19/09/2016

8

Source: Digital Science, 2015

Source: Digital Science, 2015

19/09/2016

9

Issues with policy instruments / 3: KT income depends on institutional characteristics

• E.g. size, subject mix, types of stakeholders the HEI interacts with

• Rewarding performance only based on KT income rewards “capacity

rather than excellence” (Universities Alliance, 2011)

HEIF allocation 2010-11 Average n. academic staff

2010-11

Average n. KT staff

2010-11

correlation between amount of

HEIF allocation and size of HEI

0.7521*** 0.5676***

Source: Rossi and Rosli, 2015

VARIABLES HEIF allocation 2010-11 (ln)

N. academic staff FTE +

N. Business & Community engagement staff +

Age

% academic staff in science and medicine

% academic staff in technology

% academic staff in arts and humanities -

Governing body: % business

Governing body: % social, community and cultural

Governing body: % public sector

Mission focus: research & TT

Mission focus: education -

Mission focus: local development

Sector focus: agriculture and mining

Sector focus: manufacturing +

Sector focus: utilities +

Sector focus: KIBS -

Sector focus: other services

% income from collaborative research, regeneration

programmes

% income from contract research, consultancies, facilities &

equipment +

Constant

Regional variables significant


19/09/2016

10

• The UK system has succeeded in driving KT to the top of HEIs’

priorities

• All KT indicators including income have grown substantially across

the HEI system, indicating growing and more focused efforts of HEIs

to engage in KT

Knowledge Exchange Output Performance

22

from collaborative research, a reduction of 1.4% over the 2011 value. Income from courses grew by

5.8% to £537 million while consultancy income grew by 1.0% to £303 million. Facilities and

equipment services grew modestly by 7.5% to £115 million. IP revenues, while still by far the

smallest contributor to KE income at just 2%, grew rapidly by 24% to £64 million.

Figure 4.2 KE income by source 2003-2012 (£millions, constant 2011 prices)

Contract research

Collaborative research

Courses

Consultancy

Facilities and equipmentIP revenues

923

620

537

303

11564

34

23

20

11

42

5.8

3.4

7.4

5.5

6.38.4

Regeneration 121 5 -5.8

3.1

-1.4

5.8

1.0

7.524.1

-20.6

KE income 2,683 100 4.7 1.5

0

500

1,000

1,500

2,000

2,500

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

20

11

20

12

Kn

ow

led

ge

exch

an

ge

inco

me

(£ m

illio

ns,

co

nst

an

t 2

01

1 p

rice

s)

Year

Income stream

2012 income

(£m)

2012share of total (%)

Growth04-12

(% p.a.)

Growth11-12

(% p.a.)

KE income (excl. RDA) 2,656 99 5.0 3.2

Source: HEBCI, author’s analysis

When comparing how KE income varies across different types of HEIs, it is important to control for

the different sizes of institutions. Figure 4.3 shows how KE income per academic FTE has evolved

over the period 2003-2012 for the different clusters of HEIs. It clearly reveals the link between

research intensity and the amount of KE income generated even controlling for size. In addition,

there are signs of divergence in KE income per academic FTE between different types of HEIs,

particularly since the onset of the economic crisis in 2008, with the higher research intensive

institutions increasing their KE income per academic FTE while lower research intensives have seen it

flatline or fall.


• But KT income has grown faster in top research-intensive HEIs

(Coates Ulrichsen, 2014) and, recently, in larger HEIs,

reversing a previous trend in which smaller HEIs were growing

faster (Day and Fernandez, 2015)

• The system particularly rewards research-intensive, science-

based, larger HEIs, to some extent due to implicit biases

Knowledge Exchange Funding in England

12

increase in the cap from £1.8 million per year to £2.8 million; and the third was the removal of

the capacity-building element of the formula, with allocations now based purely on KE

performance. This reflected a maturing of the funding programme with all HEIs having had at

least five years of funding with which to experiment and build their KE capacity and move

towards performance improvement. The changes mean that 99 of the 129 HEIs now receive

HEFCE KE funding and an increased concentration of funding goes to the more research

intensive HEIs (Figure 3.2). However, the analysis also shows that there was no ‘north-south

divide’ evident in the concentration of funding. An analysis of the funding by region shows

that the share of funding received in each region has remained approximately constant.

Figure 3.2 Changing distribution of HEFCE KE funding between HEIF4 and HEIF 2011-2015

21

-24

-18

28

47

Change 2010/11 – 11/12 (%)2010/11

8%

35%

34%

20%

4%

Top 6

High

Medium

Low

ArtsRes

ear

ch in

ten

sity

cl

ust

er

11%

45%

28%

13%

3%

Eligible for HEIF2011-15

6

32

33

22

6

Number of HEIs

6

34

33

35

18

All English HEIs

Share of HEFCE KE funding(%)

HEFCE KE funding per HEI for eligible institutions

2,800

2,080

1,260

850

680

2011/12 (£000s)

2011/12

-12

-6

-1

2

6

11

13

15

21

4

-50 0 50

All HEIs 100%

7%

11%

10%

9%

8%

6%

25%

15%

8%

East of England

Yorkshire & the Humber

West Midlands

North West

South West

North East

London

South East

East Midlands

Reg

ion

100%

9%

11%

10%

10%

8%

6%

24%

14%

7%

99

7

11

8

8

8

5

29

15

8

129

10

14

12

11

12

5

40

17

9

1,490

1,860

1,510

1,910

1,870

1,450

1,820

1,230

1,380

1,300

Total (£ millions) £149.8m £147.4m

Constant 2011 prices Source: HEFCE, author’s analysis


19/09/2016

11

• Ranking of best performing institutions changes if we consider

income per academic or income per contract, rather than

income level


0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

total income

Income per

contract

Income per

academic

staff A

ve

rgae r

an

k p

osit

ion

Research and TT

General

Education

Local

• Ranking of best performing institutions changes if we consider

income per academic or income per contract, rather than

income level


-2 -1 0 1 2

Research and TT

General

Education

Local

Change in rank position with respect to income level

Income per academic

staff

Income per contract

19/09/2016

12

Summing up

• Measuring KT performance on the basis of quantitative output

indicators only, particularly income, is not well aligned with the

objective to support a system of diverse HEIs with different

approaches to KT

– Particularly rewards certain HEIs partly because of their characteristics

not just their effectiveness in KT

– May lead HEIs to move away from: (i) KT activities that do not produce

income even when they may be more socially beneficial; (ii) serving less

affluent constituencies; (iii) more risky/volatile sources of income

– May encourage HEIs to see their interactions with businesses within a

context of short-term revenue generation, rather than for longer-term

economic and public benefit

Improving KT performance measurement and reward policy for diverse HEI systems / 1

• Measure KT performance using a broader range of activities and

more varied ways to measure impact besides income (e.g.

qualitative information capturing broader social impact, etc.)

• Introduce more flexibility in aggregating performance indicators:

multidimensional measurements (van Vught and Ziegeleeds,

2012), positioning indicators (Bonaccorsi and Daraio, 2008)

19/09/2016

13

• Recognize that institutions are different and may require

different sets of indicators:

– Develop a broad range of indicators representing all possible

activities, and allow universities to choose the indicators that

best fit their KT profile

– Use information from KT surveys like the HE-BCI to identify

HEIs with different KT profiles, whose performance will be

measured with different indicators


• Introduce different policy instruments to reach different objectives

and promote experimentation and variety, for example by mixing

– Competitive project-based funding (forward looking) and

formula funding (based on past performance)

– Quantitative assessment of impact (performance indicators) vs

qualitative assessment of impact (peer review)

• Impact evaluation in the REF 2014 channels 20% research funds through

peer reviewed impact case studies so mitigates limits of HEBCI/HEIF as

allows a wide variety of impacts to be rewarded


policies to measure and reward hei knowledge transfer ... rossi.pdf · source: rosli and rossi,...

Documents