what lies beneath… avoiding the risk of undervaluation
DESCRIPTION
What Lies Beneath… avoiding the risk of undervaluation. Luke Georghiou PREST, Manchester Business School http://www.mbs.ac.uk/PREST. Proposition 1. - PowerPoint PPT PresentationTRANSCRIPT
What Lies Beneath… avoiding the risk of undervaluation
Luke Georghiou
PREST, Manchester Business School
http://www.mbs.ac.uk/PREST
Proposition 1
• Present techniques, and particularly surveys, focus on predictable and linear effects while missing or recording only perfunctorily a series of other effects which may be of equal or greater value.
[Behavioural additionality] can help explain that the effects of policy interventions may be greater than those perceived through a narrower input-output lens, and that the range of effects and the means by which they are achieved are complex mechanisms requiring substantial effort for successful policy design.
Proposition 2
• Even where effects are properly identified and/or calculated there is a tendency to underestimate the contribution of public intervention.
Historical context -co-evolution…
• Evaluation approaches tend to co-evolve cumulatively with policy development, eg– 1970s modification of peer review to extend criteria– 1980s interest began in collaborative R&D programmes– 1990s rise of:
• performance indicators • emphasis on knowledge transfer indicators• Institutional evaluation using programmatic approach
– 2000s interest in;• evaluation of system capabilities eg national or regional
systems• aggregate or interactive effects of policies (“policy mix”)• effect of “soft” policy tools such as foresight• strategic and persistent effects of public support
(“behavioural additionality”)
Evaluation and situating the object in its context
• When we come to evaluate support for industrial R&D (or other research) we encounter key problem
• Project fallacy– Confusion in timing and scope between the unit of
research and the contractual entity– Research impacts are often cumulative over series
of projects– Effects of research policies result from an
interaction between the measure and the strategy of the research performer
Project fallacy
• Key problem of “project fallacy” in which policymaker assumes that a contract is equivalent to a project – In practice contracted work is often only part of a longer and
broader project
Real deliverablesContract Real project
Contract deliverables
EUREKA Initiative as Example
• Evaluation approach since mid-1990s based on “Continuous and Systematic Evaluation” – Final reporting requirements replaced by
short impact survey– Follow-up after one and three years if
market effects reported
EUREKA CSE
Typical question (1)
Typical question (2)
Gives some useful results eg turnover effects - skewed returns
% of turnover accounted by % of firms in EUREKA Initiative
Additional Annual Turnover(achieved)
0%
10%
20%
30%
40%
50%
60%
70%
0=1 1=5 5=25 more
Meuro
% of projects % of turnover
But several problems…
• Inaccurate completion of survey forms– Trivial but important – confusing currency units– Non-trivial and also important - What to measure,
when to measure, how to interpret all dependent upon the respondent’s underlying model of innovation – implicit or explicit
• Large proportion of benefits dealt with only by means of importance rating with no model of how they interact or translate to measurable effects
EUREKA 2006
• Changing methodology – keeping CSE as supplementary approach
• Main effort on panel carrying out case studies of high impact projects – use skew as basis for purposive sample
• Using methodology developed by PREST in evaluation of Japanese National R&D Programme for Medical and Welfare Equipment using Beta method as starting point
Iceberg model
Sales of innovative product
Reduced process costs
Licence income
Firm strategy, organisation and method learningUse of technology in other parts of the businessNew contacts/networks & prestigeEmployment, competence & training
Spillovers to non-participants
User and social benefits
Effects on firm strategy
• Projects normally aligned with firm strategies but also create pathways to transform them – Movement from R&D services/consultancy to
manufacture – Assisting privatised state enterprises to become
commercially viable– Improvements in organisation and methods
induced by the project, including working at a higher technological level frequently internalised by the firms.
Use of technologies in other parts of the business
Neglected aspect of benefits• Opening new technological options through
follow-on projects• Application of technologies developed to
exploit new opportunities, potentially with turnover effects equal to or greater than in the originally intended application.
Contacts, networks and prestige
New contacts and networks generated by projects Mobilising effect on large sectors eg AUTOSAR
network in the automotive industry which emerged from ITEA EAST EEA.
Prestige effects of the EUREKA label helpful to give smaller firms a foothold in the
market and visibility to investors.
Employment
• Quantitative versus qualitative important here– In quantitative treatment need to consider
jobs induced or displaced in value chain– Qualitative can include upgrading of
competences, hiring higher proportion of scientists and engineers
– Also elimination of dangerous or unpleasant work
Spillovers
• Using Jaffe’s classifications:– Knowledge spillovers appear weak and rare
because SMEs have focussed niche strategies and large consortia working in parallel with competitors
– Market spillovers very strong – quick measure is to look at payback time for customers
– Network spillovers usually exploiting complementarities of equipment or equipment/service
User benefits
• Improved environments, health, safety or simply from better value for money and functionality in everyday objects
Bringing it to reality…
• Benefits come as a package and much is lost in disaggregation
• 3 cases
Case study 1- Σ!1692 Sanifogger• Lynx Award for humidifiers that keep food fresh in supermarket
storage and refrigerated cabinets• Contronics turnover from €700k in 1999 to €3.8M in 2005 almost
all from project• Transfer of technology to vehicle air-conditioning cleaner led to
orders for 10k units p.a. (contact to new partner through EUREKA Secretariat)
• Organisational learning included introduction of quality manager in Contronics and market repositioning in UK partner Pendred leading to €375k sales
• 16 jobs created in partners, 10 in sub-contractors • Customer benefit of 50% reduction in wastage with 6 months
payback for fruit/veg and 4 week payback for meat – estimate of €42M p.a. customer benefit
• EUREKA label providing substantial marketing and visibility effects
Case study 2 – Σ!1872 P3D
• Developed passive surveillance systems for tracking aircraft 200km around airports
• ERA is SME privatised 1994 and using EUREKA project in R&D based shift from military to commercial orientation
• ERA and its spin-offs 30-50% of total ERA's sales in period of 2002-4, total value of €5.4-9.0M over 3 years.
• Fourteen new jobs in ERA and 20 in partner firms performing outsourced manufacture
• Chairing international standards setting group• Reduced costs by half compared with conventional
systems• Fundamental contribution to safety
Case study 3 - Σ!2275 Visualix
• Gendex Dental Systems developed powerful digital system for high resolution dental radiography
• Sales, since 2002, worth €28.6M to produce 8,635 units– 20% of firm’s total 2005 sales
• 3 new jobs created, 2 in R&D now developing future projects using extended knowledge base and IPR platform
• Eliminates film development and liquid disposal for dentists giving them 2-4 year payback and more time with patients
• Patient gets immediate image and possibility of better treatment decisions
Additionality – what difference does the intervention make?
• Input additionality – are resources being spent on desired target?
• Output additionality – what proportion of outputs result from particular intervention?
• Behavioural additionality – what difference in behaviour results from the intervention– Concept formulated mid-1990s to help explain consistent
evaluation findings– Rooted in question of how support interacts with strategies
and capabilities of funded organisations– Looks closely at mode of delivery of support for research– Emphasis on persistent changes– OECD project explored measurement issues in industrial
grant schemes
Effects during grant cycle
Awarenessapplication
Contract
Monitoring
ResearchPost-project
support
Failed/non-applicants stimulated IPR &
collaboration rules adopted
Feedback Research direction or linkages changed
Exploitation Route changed
EUREKA contribution to effects of its projects
• Public funding – Role of public funding in enabling, extending or accelerating
the work stressed – No success cases without public funding and some not
returned to EUREKA because funding no longer available– Advice from funding agencies helpful to smaller or less
experienced firms and large firms spoke very positively of the role of cluster offices.
• Collaboration – Complementary skills brought together, often in vertical
structure, linking users and suppliers– In clusters “pre-competitive” configurations, establishing de
facto standards or sharing cost of developing technologies to be applied in different sectors
– Often the initiative of EUREKA that brought together these networks
Under-evaluation = Under-investment
• Our failure to appreciate the full extent of both the private and the social returns to R&D is a key reason for the international decline in government investment in research
• It is possible, though not proven, that industry itself may underestimate its own benefits from R&D, particularly under shareholder pressure to show short term profits
Conclusions (1)• “Standard” questionnaire based evaluation
approaches are never a sufficient means to assess effects however sophisticated an analysis is subsequently applied– Nonetheless they can be a useful first step towards
identifying case studies• High impact cases can be a good approximation
to total effects in an R&D initiative – but not for other instruments where effects are more
evenly spread (eg information and advisory services• Behavioural additionality perspective opens the
way to a much more comprehensive appreciation of effects– Ultimately we are concerned as much with building
capacities as with short-term impacts• Evaluators need to probe what lies beneath the
surface if they are to succeed in capturing and measuring the value created
Conclusions (2)
• Evaluators need to probe what lies beneath the surface if they are to succeed in capturing and measuring the value created
• This allows us to address some of the key policy issues of today