measuring the contribution of sustainable development research to society and policy-making luke...

Measuring the Contribution of Sustainable

Development Research to Society and Policy-Making

Luke GeorghiouManchester Institute of Innovation Research

Manchester Business School

Some initial trade-offs

• Research as consumption good vs research as an investment– In first case value is cultural, in second there are

users with the expectation of a return at some point in the future

• Outputs vs Impacts– Output may be scientific publication or even new

product or service but impact only realised when output interacts with economy or society eg sales or green technology or application of new regulation

More trade-offs

• Short term vs. long term– many different time-profiles over which impacts are

manifested including • short-term effects which terminate abruptly as market conditions

change • outputs which are not used for some years and then become

very important, perhaps because complementary technologies have been developed

• extrapolation is dangerous.

• Intended vs. unintended– excessive focus on project or programme goals as the basis

for evaluation could lead to important unintended effects being missed

• eg in medical research when a drug developed for one condition turns out to be important in the treatment of another

– Unintended effects also include most of those in the negative category.

INNOVATION

Problems in calculating returns to R&D

Timing

Attribution

Appropriability

RE

SE

AR

CH

EF

FE

CT

S

ICT Technology Pillars

Nano-electronics, photonics and integrated micro/nano-system

Ubiquitous and unlimited capacity communication networks

Embedded systems, computing and control

Software, Grids, security and dependability

Knowledge, cognitive and learning systems

Simulation, visualisation, interaction and mixed realities

New perspectives in ICT drawing on other science and technology disciplines

Integration of Technologies

Personal environment

Home environments

Robotic systems

Intelligent infrastructures

Applications Research

ICT meeting societal challenges

•health

•inclusion

•mobility

•environment

•government

ICT for content, creativity and personal development

•media

•learning

•cultural

ICT supporting businesses and industry

•business processes

•Manufacturing

ICT for trust and confidence

Future and Emerging Technologies

Stronger, lasting growth More & better jobs Sustainability

FP7

A more attractive place to invest and work

•Internal market

•Improved regulation

•Open competitive markets

•Expand/improve infrastructure

Knowledge and innovation for growth

•Increase R&D

•Facilitate innovation, uptake of ICT and sustainable use of resources

•Contribute to strong industrial base

Creating more & better jobs

•More employment & modernise social protection

•Adaptable workers & flexible labour markets

•Better education & skills

Revised Lisbon

Information space

Open stable markets for electronic communications & digital services economy

Innovation & investment in ICT

Deploy services

•e government

Research leadership

Investment & improvement

Effective

adoption of ICT

Inclusion & better QoL

•Knowledge society

•Social development

i2010

Overall linkage mapping can become very dense

• Previous slide shows only a selection of the more obvious linkages

• While important not to lose the argument of connected rationale must also be able to examine elements of it in isolation to allow more detailed arguments to be developed

Different logics

• Horizontal logic– Exploring interdependency between high level objectives or

between intermediate level actions (ie the proposals for FP7) as per previous example

• Vertical logic– Exploring interdependency between an objective and the

relevant parts of FP7• Can isolate as binary link, or• Consider combined effect of all aspects of the Programme on

that objective, or • Consider multiple effects on objectives of a single Programme

activity

• Systemic logic– Considering implications of change across the whole system

Vertical logic – binary chain example

• Pair of binary relationships• Overall Lisbon strategy sees increased

R&D as necessary condition for growth by making business more innovative, productive etc

• Activity here is sponsorship of pre-competitive R&D

• Stated rationale for spending on research rests on:– “European industry lags in investment cf

major competitors”– “More intensive cooperation makes most of

current capabilities”

Growth

Increase R&D


Expected impacts and outcomes en route to increased R&D


Immediateimpacts

Intermediateimpacts

Ultimateimpacts

ActivitiesCollaborative projects

Networks of excellence

Joint technology initiatives

etc

New consortia formed

Companies invest own resources

Technology area new to participants

More ambitious projects undertaken

papers IPR products services

New markets entered

Increased sales and cost reductions

Increased R&D

Trained researchers

Higher return on R&D

Researchers employed

Problems continued

• Timing – Effects may happen long after research– Issues of discounting

• Attribution– See previous graph – what complementary inputs

needed? Linearmodel issues

• Appropriability– Who gets the benefit from a research project?– Spillovers

Inadequacy of linear model of innovation

• On the one hand we have research which is addressed specifically to solving problems of sustainability– Eg sustainable construction

• May even be the defining characteristic of a whole field– Eg environmental monitoring

• On the other hand sustainable solutions may depend upon advances in knowledge in areas of research such as nano, bio, cognitive science and complexity where initial aim was – general advancement of knowledge; or– pursuit of a different objective

• What kind of translational model is needed?

Spillovers

• Benefits accrue to agents other than the party undertaking the research

• Rationale for policy intervention social rate of return exceeds inadequate private rate– Eg We may need to subsidise for research leading to green

technologies because market alone will not produce them

• Jaffe describes 3 types– Knowledge spillovers

• Knowledge created by one agent can be used by another without compensation, or with compensation less than the value of the knowledge

• Eg reverse engineering, imitation• May be deliberate disclosure through publication or patent

Spillovers continued

• Market spillovers– Market forces cause buyers of new product or

product made with new process to get some of the benefit because not all superiority of price reduction captured in price (consumer surplus)

• Network spillovers– Arise when commercial or economic value of a

new technology is dependent upon developments in related technologies

– Eg communication systems– Firms may fail to coordinate their activities

without intervention

Nature of socio-economic effects

Intermediate outputs

• Products• Processes• Services• Standards• Knowledge and

skills• Dissemination

Impacts/effects• Competitiveness• Employment• Organisation• Quality of life• Control & care of the

environment• Regional development • Development of infrastructure• Production & rational use of

energy• Industrial development• Regulation & policy

Nature of Social Impact

• Often public goods where government is the actual or proxy customer– benefits not captured through market mechanism– paid through taxation– enforced through legislation/regulation

• Not easily reduced to monetary equivalent– may not be social consensus on valuation of

quality of life• May be trade-off or complementary relation with

wealth creation– co-production of economic and social effects

• Introduces new stakeholders to evaluation– social groups

Science and Quality of Life*

Development of techniques to detect problems

Environmental sensors, health monitoring equipment, security systems

Development of preventative measures

Cleaner technology, healthier food

Development of methods for remediation & amelioration of problems

Land reclamation, medical treatment

Knowledge to inform better policymaking

Climate change forecasts, understanding of science behind regulations & standards

* with thanks to Ian Miles

Specific Environment Issues

• Usually link to policy or regulation• May involve inter-temporal pricing• Some environmental changes are irreversible

with implications for future generations• May not be obvious who are the “end-users”• Problem if proxy-customers not fulfilling

duties

Impact gap

R&D Effect

POLICY MEASURE (regulation,

law, standard, policy,

practice)

Evaluation A Evaluation B

Practical solutions?

• UK Government allocates block funding for University research by means of an ex post evaluation every 5-7 years– Previously called Research Assessment

Exercise– Allocated funds on basis of quality of

outputs (peer review); research environment and esteem)

• Next exercise renamed Research Excellence Framework

REF and IMpact

• Major innovation to include impact and government insisting that it will count for 25% of score

• “Impact: An assessment of demonstrable economic and social impacts that have been achieved through activity within the submitted unit that builds on excellent research. This is to assess the extent to which a submitted unit has built upon its strong record of excellent research to make a positive impact on the economy and society within the assessment period. Throughout this document, where we refer to ‘impact’ or ‘social and economic impact’, we include economic, social, public policy, cultural and quality of life impacts. “

Method for assessment• One case study for every 5-10 people entered• Impact must be evident in the 2008-12 assessment

period but could come from research up to 15 years before

• Case study includes– explanation of the nature of the impact; how far-reaching it

is/who the beneficiaries are; and how significant the benefits are

– appropriate indicators of the impact – an outline of what the underpinning research was, when this

was undertaken and by whom– what efforts were made by staff in the unit to exploit or apply

the findings or secure the impact through its research expertise

– key research outputs that underpin the impact– external reports or documents, or contact details of a user,

that could corroborate the impact or the unit’s contribution

Indicators for impact on policy

• Better informed public policy-making or improved public services• Research income from government organisations• Changes to legislation/regulations/government policy (including

references in relevant documents)• Changes to public service practices/guidelines (including

references in guidelines)• Measures of improved public services (for example, increased

literary and numeracy rates) • Staff exchanges with government organisations • Participation on public policy/advisory committees• Influence on public policy debate (for example, as indicated by

citations by non-government organisations or the media)

A framework for impact and internalising spillovers

• Understanding the relation between research and innovation

• How to transfer to more sustainable modes of development – overcome lock-in to dominant sociotechnical regimes– Put in place infrastructures and policy/regulatory frameworks

• Critical role for users and link to demand-side innovation policies

• Building constituencies including researchers• Coordination, engagement and building the capacities

and incentives for users to play a major role in the process

• Embedding in a Grand Challenge initiative more likely to succeed?

Research and policy

• Similar feedback loops to innovation

• Role of policy users – Risk of purchaser lock-in

• Is there a need for coordinated Grand Challenge Initiatives to carry forward SD research?

Conclusions

• Linkages to innovation and policy play key role in constructing agenda for SD research

• Before measuring impact we must have understanding of systemic model which links – Research and innovation– Research and policy

• The pathways and complementary or intervening factors may not be the same from one sector to another

• Without evaluation and measurement we work only on theory and belief without the assurance that an evidence base can give us

• Evidence is needed not only for managers but to reassure the stakeholders who fund and use research

measuring the contribution of sustainable development research to society and policy-making luke...

Documents