SCIENCE  UNDER  THE  MICROSCOPEMONITORING  AND  EVALUATING  IMPACT

Why  measure  impact?Not  just  because  you’re  told  to…

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 2

Impact  Assessment  Purpose

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 3

Analysis

Advocacy

Accountability

Allocation

Purpose…ADVOCACY

MAKING  THE  CASE  FOR  RESEARCH

ACCOUNTABILITY

TO  TAXPAYERS  &  DONORS

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 4

“Used  as  evidence  as  part  of  the  preparation  for  the  spending  review  and  will  be  in  the  foreseeable  future” –Science  Minister

“few  studies  that  have  made  a  genuine  attempt  to  objectively  assess  the  economic  returns  of  research” –Nature  Editorial

Purpose…ANALYSIS

UNDERSTANDING  WHAT  WORKS

ALLOCATION

REWARDING  IMPACT

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 5

Researchers  that  show  interests  in  other  field(s)  have  a  higher  

impact  and  those  focused  on  a  

single  topic  have  a  lower  wider  social  impact  

There  are  some  perennial  challenges   to  research  evaluation

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 6

NCEs  and  ImpactWhat’s  currently  happening  and  why?

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 7

What  is  an  impact  for  NCEs?Potential  contentious  issue:

• In  a  number  of  research  impact  models,  impacts  are  often  ‘things  that  arise  as  a  consequence  of  the  research’  (i.e.  outputs  and  outcomes)

• For  more  traditional  evaluation,  impacts  are  often  ‘long-­‐term  outcomes’  (i.e.  societal  /  behaviour  changes  as  a  result  of  the  research)

• For  NCEs,  the  publication  of  impacts  in  annual  reports  seems  to  suggest  the  former  definition  • The  ‘What  if’  report  catalogues  numbers  of  publications,  partnerships,   co-­‐funding   levels,  capacity  building   opportunities,   patents.

• There  is  an  argument  that  NCEs  should  be  looking  at  impacts  as  more  the  latter  definition• The  stated  goal  of  NCEs  is  to  improve  the  economy  and  quality  of  life  for  Canadians

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 8

How  to  know  if  you’re  having  an  impact?Two  questions  really:◦ 1  – How  would  I  measure  /  collate  the  impacts

◦ 2  – How  can  I  find  out  about  the  impacts?

These  two  questions  have  different  answers◦ 1  – Using  frameworks,  methods  and  approaches  around  impact  assessment

◦ 2  – By  engaging  with  the  right  people  to  know  ‘of’  the  impacts  that  occur

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 9

FrameworksCreating  a  structure  to  assess  research  impact…

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 10

Canadian  Academy  of  Health  Sciences

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 11

The  Co-­‐produced  Pathway  to  Impact  Based  on  a  knowledge  mobilization  approach◦ Combining:   university  push,  community  pull,   knowledge  exchange,  and  co-­‐production   of  knowledge

Logic  model  driven◦ Research  à dissemination  à uptake  à implementation  à impact

Developed  with  PREVNet to  take  advantage  of  their  co-­‐creation  model  for  research  (academic  and  community  input)

At  each  stage  of  the  logic  model  can  identify  ‘benefits’  of  PREVNetactivities:

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 12

Research

•Research  benefits

Dissemination

•Dissemination  benefits

Uptake

•Uptake  benefits

Implementation

•Implementation  benefits

Impact

•‘Traditional’  outcomes

Research  Excellence  Framework  (REF  -­‐ UK)The  primary  purpose  of  the  REF  is  to  produce  assessment   outcomes  for  institutions◦ To  inform  the  selective  allocation  of  research  funding  to  HEIs,  with  effect  from  2015-­‐16

◦ To  provide  accountability  for  public  investment  in  research  and  produces  evidence  of  the  benefits  of  this  investment

◦ To  provide  benchmarking  information

Research  Excellence   Framework  (REF)  collected   impact  case  studies  from  university  research

Universities   asked  to  provide  narratives  of  impacts

Based  on  expert  review  of  institution   submissions

20%  of  score  is  based  on  ‘Impact’◦ Criteria  for  assessing  impacts  are  ‘reach’  and  ‘significance’

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 13

Comparing  frameworksCAHS Co-­‐produced REF

Strengths Comprehensive,  Process and  outputs  and  impacts,  Indicator  library

KM  framework  base,  Strong narrative  of  impacts

Rich  detail,  Acrossdisciplines,  Ability   to  compare  impact  stories

Weaknesses Resource  intensive,  Requires  expertise  to  implement,  

Reliance  on  user  input,difficult   to  identify   ‘unit’  of  impact

Not  representative,highly   resource  intensive  to  collect

Opportunities International  comparisons,   flexibility,  

Ability to  describe  the  path  to  impacts,  Building  on  growing  KM  field

Developing library  of  impact  narratives,  analysis  opportunities

Threats No  implementing  owner

Could be  seen  as  relevant  only  to  co-­‐production   research

Linking   impact  to  funding   politically  challenging

June  27  2016 FOOTER  TEXT  HERE 14

Who  knows  the  impacts?Identifying  the  right  people  to  get  at  your  impacts…

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 15

Sphere  of  influence  and  knowledge

Beneficiary

User  

Researcher

Input• Research  funding

Process• Research  activity

Output• Research  product

Outcome• Changed  decisions

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 16

ControlInfluence

Tools  to  assess  impactCan  anyone  make  this  easier  for  you?

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 17

Some  examples  of  impact  toolsResearch  fish

ImpactFinder

ImpactStory

Uber  Research

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 18

Articulating  impactHow  you  say  it  can  be  more  important  than  what  you  say…

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 19

REF  examples

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 20

0 1,000+

624Ireland

4Bermuda

1CaicosIslands

25Jamaica

24Macedonia

(FYROM)

15Moldova

209Russian Federation

1,822United States

878Canada

10Greenland

48Iceland

159Mexico

5,308United

Kingdom

361Switzerland

281Denmark

141Portugal

175Finland

28Latvia

52Estonia

44Lithuania

8Gibraltar

37Morocco

29Kazakhstan

50Croatia

59Slovenia

18Mongolia

619China

7North Korea

397Sweden

321Norway

19Liberia

84Ghana

2Western Sahara

24Sierra Leone

3Guinea-Bissau

21Tunisia

38Malta 27

Lebanon

18Kyrgyzstan

18Iran

38Iraq

54Qatar59

SaudiArabia

5Turkmenistan

6Bhutan

46Nepal

20Algeria

23Libya

72Egypt

India

492

58Ethiopia

Pakistan

72

Afghanistan

55

43Sri Lanka

3Cambodia

8Laos

44Vietnam

86UAE

24Kuwait

50Jordan

2British Indian

Ocean Territory

1,078Australia

358New Zealand

1New

Caledonia

3

Fiji

10Niger

7Chad

26Sudan

14SouthSudan

15Mali

24DR Congo

4Central African

Republic

262SouthAfrica

8Swaziland

10Madagascar

15Mauritius

32Zimbabwe

28Mozambique

1Comoros

4Djibouti

35Zambia

21Namibia

26Bolivia

56Colombia

80Argentina

39Peru

28Ecuador

81Chile

4Paraguay

15Venezuela

4SaintLucia

9Haiti

6DominicanRepublic

5Puerto Rico

1British Virgin Islands

2U.S. Virgin Islands

3Anguilla

10Barbados

8Trinidad and

Tobago

1Grenada

10El Salvador

6Belize

9Guatemala

14Papua

New Guinea

56Philippines

4Tonga

1AmericanSamoa

11Brunei

1Palau

1Guam

1Cook Islands

182Poland

86Taiwan

678France

864Germany

603Netherlands

484Italy 56

Romania52

Bulgaria458Spain

1Andorra

443Japan

393Belgium

26Luxembourg

320Brazil

216Austria

165Turkey28

Syria

63Ukraine

161Israel

40Palestinian Territory

155Singapore

136Greece

128Hong Kong

4Macao

126Malaysia

108Kenya

106Czech Republic

38Slovakia

105South Korea

100Indonesia

96Thailand

95Hungary

41Serbia

77Uganda

81Nigeria

80Tanzania

45Malawi

72Bangladesh

1Cayman Islands

22Senegal

11Gambia

9Maldives

28Rwanda

8Burundi

22Bosnia &

Herzegovina13

Montenegro

20Kosovo

21Albania

22Myanmar

21Somalia

20Botswana

18Azerbaijan

7Georgia

19Armenia

19Cameroon

2Equatorial

Guinea

9Gabon

5Congo –

Brazzaville

18Oman

16Burkina Faso

16Uruguay

15Cuba

14Uzbekistan

13Bahrain

12Yemen

11Panama

10Belarus

10Guernsey

9Guatemala

8Ivory Coast

4Togo

6Benin

8Seychelles

10Vanuatu

8SolomonIslands

7Angola

7Tajikistan

7Guyana

1FrenchGuyana

1Surinam

6Lesotho

6Monaco

6Montserrat

5Dominica

4Guinea

3CapeVerde

3East

Timor

3Eritrea

3FalklandIslands

2Mauritania

1Bahamas

1Saipan

53Cyprus

15Nicaragua

2Honduras

http://impact.ref.ac.uk/CaseStudies/

REF  examples

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 21

UoA

23

UoA 5

UoA 1UoA 36UoA 35

UoA 34

UoA 33

UoA

32

UoA

31

UoA

30

UoA

29

UoA

28

UoA

27

UoA

26

UoA

25

UoA 2

4

UoA 22

UoA 21

UoA 20UoA 19 UoA 18

UoA 17

UoA 16

UoA 15

UoA 13

UoA 12

UoA 11

UoA 10

UoA 9

UoA 8

UoA 7

UoA 6

UoA 4

UoA 3

UoA 2

UoA 14

0%

4%

42%0%0%

0%

0%

1%

0%

0%

0%0%

0%6%

0%

0%

0%

0%

0%0% 1%

0%

0%

2%

0%

0%

1%3%

0%1%

0%

2%

11%

14%

29%

0%

0%

Clinical Guidelines

10%20%30%40%50%60%70%80%

100%90%

10%

20%

30%

40%

50%

60%

70%

80%

100%

90%

10%20%30%40%50%60%70%80%

100%90%

10%20%30%40%50%60%70%80%

100%90%

‘Clinical  guidance’(guidelin patient clinic treatment recommend

stroke nice risk trial)n  =  326  or  c5%  of  case  studies

‘Informing  government  policy’(develop  polici nation  plan  govern  inform  

work  strategi assess)  (n=1233  or  c20%  of  case  studies)  

UoA

23

UoA 5

UoA 1UoA 36UoA 35

UoA 34

UoA 33

UoA

32

UoA

31

UoA

30

UoA

29

UoA

28

UoA

27

UoA

26

UoA

25

UoA 2

4

UoA 22

UoA 21

UoA 20UoA 19 UoA 18

UoA 17

UoA 16

UoA 15

UoA 13

UoA 12

UoA 11

UoA 10

UoA 9

UoA 8

UoA 7

UoA 6

UoA 4

UoA 3

UoA 2

UoA 14

31

%

15%

8%15%4%

6%

1%

5%

3%

3%

1%8%

3%25

%

42%

30%

41%

18%

24%36% 15%

43%

51%

7%

3%

6%

9%11%

1%4%

35%

26%

23%

25%

36%

33%

Informing Government Strategy

0%

10%20%30%40%50%60%70%80%

100%90%

10%

20%

30%

40%

50%

60%

70%

80%

100%

90%

10%20%30%40%50%60%70%80%

100%90%

10%20%30%40%50%60%70%80%

100%90%

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 22

REF  examples

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 23

REF  examples

REF  examples

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 24

Impact  Array

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 25

The  journey  is  still  going  on…Eddy  Nason,  Assistant  Director,  Ontario  SPOR  Support  UnitE:  eddynason@ossu.ca @eddynason

June  27  2016 SCIENCE  UNDER  THE  MICROSCOPE 26