real-world impacts from research: evidence & lessons

Real-world impacts from research: Evidence & lessonsDavid PannellCentre for Environmental Economics and PolicySchool of Agricultural and Resource Economics

For this PPT see www.davidpannell.net under “Talks”

Growing interest· Perception: we need to do better at convincing

government about benefits of research

· ARC discussing how to include real-world impact in ERA

· UK’s Research Excellence Framework: 20% of funding based on “impact” from 2014.

Trial by universities, 2012· Group of Eight (Go8) and

Aust Technology Network of Universities (ATN)

· Each university submitted cherry-picked case studies (165 submissions)

· Evaluated by people from industry & government

· 24 ‘best’ selected

Plan· An example research project

Was selected in the GO8/ATN · Some evidence about impact

· Measuring impact

· Strategies for having impact

Example

2000: Salinity was a hot topic

$1.4 billion of public funding

I was shocked· Poor design of the program

· Program developers seemed to have been unaware of crucial areas of salinity research and their implications

· No chance of any significant benefits

My response· Media· Discussion papers · Presentations· Submissions

Tried to help them· Developed INFFER (Investment Framework for

Environmental Resources)A tool for integrating the science with other infoDevelop logical, evidence-based environmental

projectsAssess value for moneyPrioritise projects

INFFER strategy· Extensive input by users· Make tools as simple as possible· Provide training and help desk for users· Readable documentation· Public critiques of existing approaches· Attempt to influence gov’t agencies to change the

signals

Regional NRM application

Policy impacts· Senate inquiry (2006)

Recommended use of INFFER· NRM Ministerial Council (2007)

Endorsed new set of principles for investment in salinity

· Victorian Government, Biodiversity White Paper “INFFER will be utilised for the next five years”.

· Caring for our Country Influenced design of project template

Lessons: Use of science· If you want people to use good science, the

people issues are crucialRelationshipsCommunication

· Most prospective users were happy with current (very poor) approach

· Didn’t perceive that government would reward them for doing it better

Lessons: User capacity· Lack of capacity to formally integrate disparate

technical and socio-economic information for decision making

· Lack of expertise in economics and social science

· Lack of time to read things· People misinterpret things easily

Research versus? Impact· Has taken considerable effort beyond traditional

researchTime commitmentNew skills and knowledgeNew networks

· Satisfying but very challenging to make a difference

· Worth it?

versus?and?

Research versus? Impact· Various benefits for my research

· Interesting problems and issues arise

· Innovation - outside what’s currently in journals

· Better understanding of research relevance

· Journal papers generatedDirectly part of the INFFER work: 17Related/stimulated by: 16

· Reputation for useful research easier to get funding (unsolicited approaches offering $)

versus?and?

Evidence about impact

Evidence of high returns· Estimated rates of return to R&D are typically

very highCan indicate 30%, 50%, 100% annual rate of return

· Credible?$1 invested at 50% over 100 years = $4E17 (a

million times Australia’s annual GDP)· Sound analyses still show good returns

For both applied and basic research

Heterogeneity· The distribution of benefits is highly skewed

· Most research has low impact

· A small number of projects have huge impactMore than enough to pay for the rest

Example: CRC program· Benefits for 1991 to 2017

· The CRC program generated a net economic benefit of $7.5 billion over the study period

· Annual contribution of $278 million

· BCR = 3.1

Impact is often slow· Lags to impact usually measured in decades

· e.g. US agriculture

· From first investment to peak impact = 24 years

· Still generating benefits after 50 years

· Several lagsResearch lagCommercialisation lagAdoption lag Impact lag

Longer lags = lower net benefits· Discounting allowing for interest costs on the

up-front investment

· 30-year lag, 7% discount rate, benefits reduced by 87%

· The high measured rates of return occur despite the long time lags

Supply push vs demand pull· Science push (Bush, 1945)

· Implicit in the “linear model”Basic R Applied R Technology Benefits

· Demand pull (Schmookler, 1966)Market demand Applied R Technology Benefits

· Big debate in the 1960s

· Resolved in the 1970s – innovation is an iterative process – both push and pull matter

Measuring impact

Determinants of benefits· Scale of relevance

· Adoptability of the research

· Benefits per unit

· Probability of research success

· Share of the credit attributable to particular research

· Time lags

With vs without

0 5 10 15 20 250

10

20

30

40

50

60

70

80

90

100

With project

Without project

Year

Wel

fare

/util

ity/in

com

e

Applicability?· The theory is relatively straightforward

· It has been applied successfully in many case studiesEspecially agriculture

But …· It takes resources and skills

· Easier … for physical products than for knowledge if the benefits arise in markets if the benefits occur quickly for applied than for basic research

· Much university research is not in the categories that are relatively easy to evaluateKnowledge, public goods, long time lags, basic

What will ERA do?· Perhaps copy the UK Research Excellence

Framework

· Two componentsCase studies of impactThe submitting unit's approach to enabling impact

from its research· They won’t expect an economic evaluation

If it’s case studies, you’ll need to· Make the case/tell the story

· Link elements in chain from research to impact

· Provide evidence

· Note: in Go8/ATN trial, many nominations did this poorlyThe chain was incompleteThe evidence was weak/unconvincing

· If you can do it well, you’ll stand out

Having an impact

How to have an impact?· There is little research about this

· There are papers, but largely anecdotal

· Some resources at end of PPT

Chain from research to impact· The chain varies widely from case to case

· Can have many links

· Understanding the chain for your research helps you tochoose, design and deliver research for greater

impactcommunicate impact provide evidence

A chain from research to impact: Technology· Research and development

· Sell the IP

· Feasibility studies

· Design

· Manufacturing capacity

· Finance

· Marketing

· Sales

A chain from research to impact: Information for policy· Research

· Something useful is learned (or isn’t)

· New information influences policy (or doesn’t)

· Policy change is implemented (or isn’t)

· If policy aims to change behaviour, people respond as intended (or don’t)

· Changes (relative to no research) result – social, environmental or economic benefits (or not)

Risk of low benefits from research to influence policy· Nobody is listening

· You lack credibility with the decision maker

· The decision maker doesn’t understand

· The new results are not different enough from what we already know

· The decision depends more on other factors

· The decision options have similar payoffs

Lessons: having impact· Need some demand pull

· Understand and respect potential users

· Be prepared for opposition

· Need perseverance, continual marketing

· Need repetition – government has short memory

· Seek a product champion

Lessons: having impact· Need “absorptive capacity” in the organisation

· The political circumstances need to be right. You can’t change ideological positions of govt.

· Timing. Grasp opportunities.

· Good communicationSimplicity, brevity, clarityAvoid jargon, maths, complex graphs

· Think about impact which choosing what to research

Conclusion· We are going to be asked to demonstrate real-

world impact

· It’s not just about communicating what we do better – we may need to change what we do to have genuine impact

· Pursuing impact is exciting and worthwhile but challenging – spinoff benefits for research

· The earlier in the research process you start thinking about impact, the better

Resources· Pannell, D.J. and Roberts, A.M. (2009).

Conducting and delivering integrated research to influence land-use policy: salinity policy in Australia, Environmental Science and Policy 12(8), 1088-1099.http://dpannell.fnas.uwa.edu.au/dp0803.htm

· Pannell, D.J. (2004). Effectively communicating economics to policy makers. Australian Journal of Agricultural and Resource Economics 48(3), 535-555. http://dpannell.fnas.uwa.edu.au/j78ajare.pdf

Resources· Weible et al. (2012). “Understanding and

influencing the policy process”, Policy Science 45, 1-12. http://link.springer.com/article/10.1007%2Fs11077-

011-9143-5

Pannell Discussions (Blog posts)· 150 – Why don’t environmental managers use

decision theory?http://www.pannelldiscussions.net/2009/04/150-wh

y-dont-environmental-managers-use-decision-theory/

· 136 – Engaging with policy: tips for researchershttp://www.pannelldiscussions.net/2008/09/136-eng

aging-with-policy-tips-for-researchers/

Resources· A relevant blog post by ecologist Brian McGill on

“What it takes to do policy-relevant science” http://dynamicecology.wordpress.com/2013/05/14/

what-it-takes-to-do-policy-relevant-science/

· Video: Ben Martin (U Sussex) “Science Policy Research - Can Research Influence Policy? How? And Does It Make for Better Policy?”http://upload.sms.csx.cam.ac.uk/media/747324

For this PPT see www.davidpannell.net under “Talks”