POLICY EVALU

FOURTH FIFTH F

IRST SECOND T

HIRD FOURTH F

IFTH FIRST SEC

OND THIRD FOU

RTH FIFTH FIRS

T

ENTREPRENEUR

SHIP POLICY EVALU

ATION FIRST FIFTH F

OURTH THIRD SE

COND FIRST FIFTH FO

URTH THIRD SECOND

FIRST FIFTH FOURT

H THRID SECOND

FIRST FIF

SME ENTREPRE

NEURSHIP FIRST

SECOND THIRD

FOURTH FIFTH

FIRST SECOND

THIRD FOURTH

FIFTH FIRST SE

COND

SME POLICY EVALUAT

ION ENTREPRENEU

RSHIP D FIRST FIFTH

FOURTH THIRD SEC

OND FIRST FIFTH FO

URTH FIFTH FOURTH

THIRD SECOND FIRS

T FIFTH

ENTREPRENEUR

SHIP POLICY EV

ALUATION SME

POLICY EVALUATI

ON ENTREPRENEU

RSHIP SME POLICY

EVALUATION ENTR

EPRENEURSHIP

SME

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME

ENTREPRENEUR

SHIP SME POLICY

EVALUATION ENTRE

PRENEURSHIP SM

E POLICY EVALUA

TION ENTREPRENE

URSHIP SME POL

ICY EVALUATION ENT

REPRENEURSHI

P S

SME ENTREPRE

NEURSHIP POLI

CY EVALUATIO

N SME ENTREPR

ENEURSHIP POL

ICY EVALUATIO

N SME ENTREPR

ENEURSHI

SME POLICY EVALUAT

ION ENTREPRENEU

RSHIP SME POLICY

EVALUATION ENTR

EPRENEURSHIP

SME POLICY EVALU

ATION ENTREPREN

EURSHIP SME PO

LICY EVALUATI

ENTREPRENEUR

SHIP POLICY EV

ALUATION SME E

NTREPRENEURS

HIP POLICY EVA

LUATION SME E

NTREPRENEURS

HIP PO

POLICY EVALUATI

ON ENTREPRENEU

RSHIP SME POLICY

EVALUATION ENTR

EPRENEURSHIP

SME POLICY EVALU

ATION ENTREPREN

EURSHIP SME PO

LICY EVALUATION

ENT

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME

ENTREPRENEUR

SHIP SME POLICY

EVALUATION ENTRE

PRENEURSHIP SM

E POLICY EVALUA

TION

ENTRE

PRENEURSHIP SM

E POLICY EVALUA

TION ENTREPRENE

URSHIP SME

SME ENTREPRE

NEURSHIP POLI

CY EVALUATIO

N SME ENTREPR

ENEURSHIP POL

ICY EVALUATIO

N SME ENTREPR

ENEURSHI

SME POLICY EVALUAT

ION ENTREPRENEU

RSHIP SME POLICY

EVALUATION ENTR

EPRENEURSHIP

SME POLICY EVALU

ATION ENTREPREN

EURSHIP SME PO

LICY EVALUATI

ENTREPRENEUR

SHIP POLICY EV

ALUATION SME E

NTREPRENEURS

HIP POLICY EVA

LUATION SME E

NTREPRENEURS

HIP PO

POLICY EVALUATI

ON ENTREPRENEU

RSHIP SME POLICY

EVALUATION ENTR

EPRENEURSHIP

SME POLICY EVALU

ATION ENTREPREN

EURSHIP SME PO

LICY EVALUATION

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION

ENTREPRENEUR

SHIP SME POLICY

EVALUATION ENTRE

PRENEURSHIP SM

E POLICY EVALUA

TION ENTREPRENE

URSHIP SME POL

ICY EVALUATION ENT

REPRENEURSHI

P S

SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

POLICY EVALUA

TION SME ENTR

EP

SME POL

ICY EVALUATION ENT

REPRENEURSHI

P SME POLICY EVAL

UATION ENTREPR

ENEURSHIP SME P

OLICY EVALUATIO

N ENT

ENTREPREN

EURSHIP POLIC

Y EVALUATION S

ME ENTREPREN

EURSHIP POLIC

Y EV

POLICY EVALU

ATION ENTREPREN

EURSHIP SME PO

LICY EVALUATION

ENTREPRENEUR

POLICY EVALUA

TION SME ENTR

EPRENEURSHIP

PO

ENTREPRENEUR

SHIP SME POLICY

EVALUATION

SME ENTR

EP

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes

-:HSTCQE=UYUU]X:

The full text of this book is available on line via this link: www.sourceoecd.org/industrytrade/9789264040083

Those with access to all OECD books on line should use this link: www.sourceoecd.org/9789264040083

SourceOECD is the OECDs online library of books, periodicals and statistical databases. For more information about this award-winning service and free trials, ask your librarian, or write to us at SourceOECD@oecd.org.

ISBN 978-92-64-04008-3 85 2007 04 1 P

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes This Framework provides policy makers with a concrete, explicit, practical and accessible guide to best practice evaluation methods for SME and entrepreneurship policies and programmes, drawing upon examples from a wide range of OECD countries. It examines the benets of evaluation and how to address common issues that arise when commissioning and undertaking SME and entrepreneurship evaluations. Key evaluation principles are set out, including the Six Steps to Heaven approach, and illustrated with examples of evaluations of national, regional and local programmes that can be explored further by the reader. The publication focuses not only on the evaluation of individual policies and programmes but also on bigger picture peer review evaluations and assessment of the impact on SMEs and entrepreneurship of mainstream programmes that do not have business development as their principal aim.

OE

CD

Framew

ork for the Evaluation of S

ME

and Entrepreneurship P

olicies and Program

mes

852007041cov.indd 1 11-Dec-2007 10:43:50 AM

OECD Framework for the Evaluation of SME

and Entrepreneurship Policies and Programmes

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT

The OECD is a unique forum where the governments of 30 democracies worktogether to address the economic, social and environmental challenges of globalisation.The OECD is also at the forefront of efforts to understand and to help governmentsrespond to new developments and concerns, such as corporate governance, theinformation economy and the challenges of an ageing population. The Organisationprovides a setting where governments can compare policy experiences, seek answers tocommon problems, identify good practice and work to co-ordinate domestic andinternational policies.

The OECD member countries are: Australia, Austria, Belgium, Canada, theCzech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland,Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand,Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, Turkey,the United Kingdom and the United States. The Commission of the EuropeanCommunities takes part in the work of the OECD.

OECD Publishing disseminates widely the results of the Organisations statisticsgathering and research on economic, social and environmental issues, as well as theconventions, guidelines and standards agreed by its members.

Also available in French under the title:

Cadre de lOCDE sur lvaluation des politiques et des programmes lgard des PME

et de lentrepreneuriat

Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.

OECD 2007

No reproduction, copy, transmission or translation of this publication may be made without written permission.

Applications should be sent to OECD Publishing rights@oecd.org or by fax 33 1 45 24 99 30. Permission to photocopy a

portion of this work should be addressed to the Centre franais dexploitation du droit de copie (CFC), 20, rue des

Grands-Augustins, 75006 Paris, France, fax 33 1 46 34 67 19, contact@cfcopies.com or (for US only) to Copyright Clearance

Center (CCC), 222 Rosewood Drive, Danvers, MA 01923, USA, fax 1 978 646 8600, info@copyright.com.

This work is published on the responsibility of the Secretary-General of

the OECD. The opinions expressed and arguments employed herein do not

necessarily reflect the official views of the Organisation or of the governments

of its member countries.

FOREWORDForeword

The OECD Working Party on Small and Medium-sized Enterprises andEntrepreneurship (WPSMEE), in line with a recommendation of the 2004 IstanbulMinisterial Declaration on Fostering the Growth of Innovation and Internationally

Competitive SMEs, has prepared this report aimed at strengthening the conceptualframework for SME policy evaluation. This report seeks to be of direct practicalassistance to public administrators and politicians concerned with evidence on the

effectiveness of SME and entrepreneurship policies and programmes at a national andlocal level.

The Framework was written by Dr. Jonathan Potter, Principal Administrator,

OECD Centre for Entrepreneurship, SMEs and Local Development, and Prof. DavidStorey, Warwick Business School, UK, and prepared under the supervision of Mme

Marie-Florence Estim, Deputy Director of the OECD Centre on Entrepreneurship,SMEs and Local Development (CFE).

A Steering Group, co-chaired by Dr. Roger Wigglesworth, New Zealand and

Mr. George Bramley, United Kingdom, guided the preparation of the Framework. TheCo-Chairs along with the members of the Steering Group offered many valuablecomments during the drafting, revisions and review of the Framework: Mrs. Sue

Weston and Ms. Vicki Brown, Australia; Mrs. Laura Morin, and Ms. Kaili Levesque,Canada; Ms. Annukka Lehtonen and Mr. Pertti Valtonen, Finland; Mr. Serge Boscherand Mr. Jean-Hugues Pierson, France; Mr. Tamas Lesko and Dr. gnes Jnszky,

Hungary; Mr. Young-Tae Kim and Dr. Sung Cheon Kang, Korea; and Ms. Ana MaraLagares Prez, Spain.

Sincere appreciation is extended to the Delegates of the OECD WPSMEE for their

numerous comments and inputs during the compilation of the Framework.

Thanks also go to Mr. Kevin Williams, Principal Administrator, OECD Council andExecutive Committee Secretariat, Mr. Hans Lundgren, Head of Section, Evaluation,

Development Co-operation Directorate, and Mrs. Mariarosa Lunati, Administrator, CFE/SME and Entrepreneurship Division for their drafting suggestions and Ms. Brynn Deprey,OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 3

Mr. Jorge Glvez Mrdez, Mr. Damian Garnys, and Ms. Elsie Lotthe for their operational

support.

TABLE OF CONTENTSTable of Contents

Summary and Route Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Section 1. Evaluation Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Defining evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Why do an evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Typical objections to evaluation and responses . . . . . . . . . . . . . . . . . . 19Key evaluation debates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Doing evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Some key principles for evaluation practice . . . . . . . . . . . . . . . . . . . . . 32

Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Section 2. Evaluation of Individual National Programmes. . . . . . . . . . . . . . 37Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Evaluations of financial assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Enterprise culture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Advice and assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Section 3. Evaluation of Regional and Local Programmes . . . . . . . . . . . . . . 53Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54Advice, consultancy and financial assistance . . . . . . . . . . . . . . . . . . . . 55Clusters and local innovation systems . . . . . . . . . . . . . . . . . . . . . . . . . . 57Support to areas of geographical disadvantage. . . . . . . . . . . . . . . . . . . 59Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Section 4. The Role of Peer Review in Evaluation . . . . . . . . . . . . . . . . . . . . . 67Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68The peer review methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68OECD national SME reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70OECD regional and local entrepreneurship reviews . . . . . . . . . . . . . . . 71OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 5

OECD evaluation guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Section 5. Reviewing the Aggregate Impact of Public Policies . . . . . . . . . . 75Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Impact of mainstream policies on SMEs. . . . . . . . . . . . . . . . . . . . . . . . . 77Capturing the total policy package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

TABLE OF CONTENTSConclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Appendix B. Six Steps to Heaven: Methods for Assessing the Impact of SME Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Appendix A. The OECD Istanbul Position . . . . . . . . . . . . . . . . . . . . . . . . 103

Appendix C. Examples of Evaluation Guidance . . . . . . . . . . . . . . . . . . . 109Appendix D. Assessing the Quality of an Evaluation . . . . . . . . . . . . . . . 111Appendix E. Framework Condition Indicators: Entrepreneurship

Conditions in Denmark in 2005 . . . . . . . . . . . . . . . . . . . . . 113Appendix F. Summary of the Evaluation of State Aid to SMEs

in the Member States, European Economic Areaand the Candidate Countries . . . . . . . . . . . . . . . . . . . . . . . 120

List of tables

1.1. Qualitative compared with quantitative evaluation . . . . . . . . . . . . 231.2. The choice of internal and external evaluators . . . . . . . . . . . . . . . . 252.1. SME and entrepreneurship policy areas covered . . . . . . . . . . . . . . . 382.2. Loan guarantee scheme, Japan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.3. Loan guarantee scheme, Canada . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.4. Assistance to new enterprises started by young people, Italy . . . . 402.5. Grant assistance and small firm performance, Ireland . . . . . . . . . . 402.6. Public subsidies to business angels: EIS and VCT, UK . . . . . . . . . . . 412.7. Public subsidies to business angels: EIS, UK . . . . . . . . . . . . . . . . . . . 412.8. Assisting young disadvantaged people to start up businesses, UK . . 422.9. Graduates into business, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2.10. Investment readiness, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . . 432.11. Impact of marketing advice, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442.12. Impact of business advice, Belgium . . . . . . . . . . . . . . . . . . . . . . . . . . 442.13. Impact of advisory support, Bangladesh . . . . . . . . . . . . . . . . . . . . . . 442.14. Bank customers receiving business advice, UK . . . . . . . . . . . . . . . . 452.15. Assistance and advice for mature SMEs, UK . . . . . . . . . . . . . . . . . . . 452.16. Use and impact of business advice, UK . . . . . . . . . . . . . . . . . . . . . . . 462.17. Evaluating entrepreneurial assistance programs, US . . . . . . . . . . . 462.18. Encouraging partnerships amongst SMEs, Sweden . . . . . . . . . . . . . 472.19. Technology assistance to small firms, US . . . . . . . . . . . . . . . . . . . . . 482.20. The SBIR program, US . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.21. The UK SMART programme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.22. Impact of science parks, Greece . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.23. Impact of science parks, Sweden . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502.24. University/SME links, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . . . 50OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 20076

TABLE OF CONTENTS2.25. Impact of management training on SMEs, UK . . . . . . . . . . . . . . . . . 512.26. Small firms training loans, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.1. Regional/local policy areas covered . . . . . . . . . . . . . . . . . . . . . . . . . . 553.2. Subsidised consulting, Belgium, Wallonia. . . . . . . . . . . . . . . . . . . . . 563.3. Business advisory services, UK, South West England . . . . . . . . . . . 563.4. Enhancing the capability of the SME owner through use

of consultants, UK, Scotland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573.5. Export information and advice, Canada, Quebec . . . . . . . . . . . . . . . 573.6. Enterprise partnerships for exporting, Sweden, rebo . . . . . . . . . . 583.7. Small business grants, UK, North East England . . . . . . . . . . . . . . . . 583.8. Regional development agency grants, Ireland, Shannon . . . . . . . . 593.9. Local innovation system policy, EU regions . . . . . . . . . . . . . . . . . . . 60

3.10. Business networking, UK, North East England . . . . . . . . . . . . . . . . . 613.11. Enterprise Zone evaluation, US, Indiana . . . . . . . . . . . . . . . . . . . . . . 613.12. Enterprise Zone evaluation, US, Five States . . . . . . . . . . . . . . . . . . . 623.13. Enterprise Zone evaluation, UK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.14. Evaluation of enterprise support in disadvantaged areas, UK . . . . 633.15. Regional policy evaluation, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633.16. Regional policy evaluation, Italy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643.17. Rural policy evaluation, Canada, Quebec. . . . . . . . . . . . . . . . . . . . . . 643.18. Rural enterprise support, United Kingdom, Northumberland . . . . 65

5.1. The indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835.2. Ease of Doing Business ranking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.3. Starting a business in 1999, 2004 and 2006 . . . . . . . . . . . . . . . . . . . . 855.4. Average conversion rates young businesses/nascent

entrepreneurs, 2000-2004 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.5. Selecting policy areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92B.1. Six Steps to Heaven: Methods for assessing the impact

of SME policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106D.1. Grid for a synthetic assessment of the quality

of evaluation work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Figure

1.1. New Zealand Trade and Enterprise (NZTE) Growth RangeProgramme Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 7

ISBN 978-92-64-04008-3OECD Framework for the Evaluation of SME and Entrepreneurship

Policies and Programmes

OECD 2007

Summary and Route Map

This Framework document provides a forum for the international exchangeof knowledge on best practice evaluation of Small and Medium-sizedEnterprise (SME) and Entrepreneurship policy. Its target readership is publicadministrators and policy-makers concerned with the formulation,development and implementation of SME policy, together with professionalsconcerned with evaluation of such policies. It seeks to be concrete, explicit,practical and accessible, drawing upon examples from a wide range of OECDcountries. Almost all the evaluations documented are publicly availableonline. It is also intended that the text will assist SME policy makers in non-member countries.

In line with the OECD Istanbul Position, which underlines the need tostrengthen the culture of evaluation of SME and entrepreneurship policies(Appendix A), this document has four objectives:

To increase the awareness of politicians and public officials of the benefitsfrom having an evaluation culture.

To disseminate examples of good micro evaluation practice at national andsub national levels.

To highlight key evaluation debates: Who does evaluations? Whatprocedures and methods should be used? When to do the evaluations?What about the dissemination of findings? Should all policies bedisseminated in the same way?

To make a clear distinction between policies that operate at the micro level,i.e. SME and entrepreneurship specific policies, and those that operate atthe macro level, i.e. mainstream policies that nonetheless influence SMEsand entrepreneurship.

To achieve this end, the Framework is divided in three main parts. Thefirst deals with evaluations of micro-entrepreneurship and SME policiesformulated and delivered at the national level. The second deals with9

entrepreneurship and SME policies delivered at the local/regional level. Thethird section is rather different. It reviews approaches to establishing theaggregate impact of a range of public policies that strongly influenceentrepreneurship and SME performance, yet are rarely the responsibility of

SUMMARY AND ROUTE MAPthe main department of government responsible for SMEs. Prior to that, theFramework reviews good practice in evaluation more generally.

It should be noted that the Framework does not seek to be a handbook ormanual that sets out the steps that need to be taken to complete anevaluation. A substantial body of such handbooks and manuals exist andselected examples are provided in Appendix C. Rather the focus of theFramework is on discussing the difficult issues that arise in evaluating SMEand entrepreneurship policies and programmes, particularly with respect toquantitative impact evaluation, and providing examples of evaluationapproaches that have been used to address these issues. The Frameworkshould therefore be read in conjunction with, rather than in place of, otherevaluation guidance in this field.

This summary provides a route-map for the reader, highlighting its keyconclusions. It then moves on to setting out the key conclusions from each ofthe three parts. Finally it sets out a proposal for continuous improvement inthe evaluation of SME policy.

So, why do evaluation?

To establish the impact of policies and programmes.

To make informed decisions about the allocation of funds.

To show the taxpayer and business community whether the programme isa cost-effective use of public funds.

To stimulate informed debate.

To achieve continued improvements in the design and administration ofprogrammes.

When and how should programme evaluation be done?

Evaluation has to be integral to the policy process. Hence there is merit inundertaking prospective evaluations as policy options are beingformulated; formative evaluations as the policy is in operation; andsummative evaluations once a clear policy impact can be judged. Thesummative evaluation findings have to feed back into current policymaking.

For summative evaluations we favour a dual approach. The first is toestablish the impact of established large scale programmes by usingquantitative, statistical methods using control groups that score highly onthe Six Steps to Heaven metric.

These can be valuably complemented with qualitative approaches such ascase studies and peer reviews for more detail on how policy works and howOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200710

SUMMARY AND ROUTE MAPit may be adjusted. Qualitative approaches are also useful for smaller scaleprogrammes for which the costs of quantitative evaluation may be too high.

And by whom?

Evaluation undertaken by specialists is essential for reliable impact evaluation.Sometimes the necessary independence can only be delivered by outsidersbut independent evaluation units within government can also perform thisrole.

The bedrock of good evaluation comprises:

The programme has to have clearly specified objectives from which it ispossible to determine whether or not it succeeded.

The evaluation has to be set in progress and data collection begun as, oreven before, the programme is implemented.

The evaluation has to be able to lead to policy change.

The evaluation of national programmes

This section of the Framework provides examples of evaluations that havebeen undertaken on the following policy areas: Financial Assistance;Enterprise Culture; Advice and Assistance; Technology; and ManagementTraining.

It concludes that, whilst there are examples of high quality evaluations, thisis not the norm.

Broadly, lower quality evaluations seem to produce more favourableoutcomes for the project because they attribute observed change to the policywhen this may not be justified.

The evaluation of local and regional programmes

At the regional and local level less costly and less sophisticated approachesare often adopted because the programmes are often smaller and becauseevaluation structures in terms of information bases, professionalevaluation capabilities and understanding of evaluation methods by usersmay be weaker.

The work of the OECD Local Economic and Employment Development(LEED) Programme with city and regional governments and developmentagencies has shown that a critical issue for policy development is increasingunderstanding of the real policy needs of the region or locality andassessing the alternative options for intervention given the specific localcontext.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 11

SUMMARY AND ROUTE MAPPeer review: a tool for evaluation

Whilst evaluation of programme impacts is still required, broader peerreviews are also useful in providing big picture assessments of the fullrange of entrepreneurship and SME policies including in selected regions.

Reviewing the aggregate impact on SMEs and entrepreneurship of public policies

Although explicit and targeted SME and entrepreneurship policies influencethe creation of new firms and the development of SMEs, so also do othergovernment policies which do not have such a focus. They are also rarelythe responsibility of the main SME department of government. Thesepolicies include control of interest rate and tax policies, social policies suchas the setting of unemployment benefits, the cost and time of starting anew business and the role of immigration and emigration.

These policies represent substantial expenditures in many countries andour review shows they impact powerfully on entrepreneurship and SMEdevelopment. However, control, or influence, over that total expenditure israrely exercised by the department of government responsible for SMEpolicy. Instead, other departments or organisations of government oftenhave considerably larger budgets, but may have different priorities to that ofthe main SME department.

The challenge for SME and entrepreneurship policy makers is to identifythese macro policies and their links to enterprise. It is then to seek toensure that they work in a way which is congruent with the objectives ofenterprise support.

Evaluation approaches need to be developed that permit policy makers withSME and entrepreneurship responsibilities to be able to engage more fully incross-government discussions on priority setting.

Future work

The current document is not to be regarded as the definitive or finalstatement on how SME policy and its constituent parts should be evaluated.In our judgement there remain too few examples of top quality evaluations. Wealso have too little knowledge about the impact which these evaluations have hadupon the formulation of policy and the impact which policy changes have uponSMEs and the economy more widely for this to be the last word. We thereforepropose that this document should evolve over time to reflect the direct interestsOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200712

SUMMARY AND ROUTE MAPof policy makers, SMEs and the taxpayer. A future text could benefit from thefollowing:

More examples of high quality evaluations that can be shared betweencountries; and

More evidence, probably in a case study format, of the links betweenevaluations undertaken and policy changes. An example here might be thereview by OECD of SME policy in Mexico and the changes that subsequentlyoccurred in that country.

In short, what we are able to provide in this current text are some genericapproaches to evaluation and some examples of evaluations undertaken,some of which are better than others in terms of their technical merit.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 13

ISBN 978-92-64-04008-3OECD Framework for the Evaluation of SME and Entrepreneurship

Policies and Programmes

OECD 2007

Section 1

Evaluation Issues15

1. EVALUATION ISSUESDefining evaluation

In their review of policy evaluation in innovation and technology,Papaconstantinou and Polt (1997) provide a very helpful definition ofevaluation. They say: Evaluation refers to a process that seeks to determineas systematically and objectively as possible the relevance, efficiency andeffectiveness of an activity in terms of its objectives, including the analysis ofthe implementation and administrative management of such activity.

Several words or phrases in this definition merit strong emphasis. Thefirst key-word is process. This emphasises that evaluation is not a once-offactivity, undertaken once a particular programme has been completed.Instead it is an integral element of a process of improved policy or servicedelivery.

A second key phrase in the definition of evaluation is as systematicallyand objectively as possible. Given that evaluation traditionally takes place atthe end of the line1 there are likely to be strong entrenched interests in placeonce a programme has been in existence for a number of years. Theseentrenched interests include the direct beneficiaries of the programme, suchas the businesses receiving funds, but they will also include those who areresponsible for initiating and administering these programmes. All else heldequal, it is to be expected that all these groups will choose the programme tocontinue or expand. The task of the evaluator, however, is to systematicallyand objectively assess the merits of the programme. In this task, the evaluatormay well conflict with those committed to the programme. Only through theuse of objective techniques, discussed later in the paper, can the evaluatordemonstrate their independence to those delivering programmes.

The third key phrase in the definition is the relevance, efficiency and effectof an activity in terms of its objective. The implicit assumption in this statementis that the policy has clear objectives and that these are stated in sufficientlyclear terms for them to be used by the evaluator. In practice, this is by nomeans always the case. As will be shown later, a key role for evaluators is oftenOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200716

to formalise for the first time the objectives of programmes, often after suchprogrammes have been in operation for many years.

This definition, and the OECD Istanbul paper,2 emphasised thatevaluation has an integral role to play in the policy process. Evaluation cannotbe left at the end of the line. Instead, it has to be a key element of initial

1. EVALUATION ISSUESpolicy formulation. Once the policy is operational, all organisations andindividuals responsible for delivery have to be aware that evaluation is to takeplace. Once the evaluation has been undertaken, and sometimes as it is takingplace, it should be used as the basis for dialogue with policy makers, with theobjective of delivering better policy. The outcome of the evaluation can thenbecome an input into a debate on the appropriate ways for governments andSMEs to interact.

Why do an evaluation?

Whilst some countries have a long established tradition of undertakingevaluation, others do not. For those seeking to champion a culture ofevaluation, the following arguments summarise the case in favour. We thenalso take the arguments that are often used against evaluation and addressthem.

To establish the impact of policies and programmes against their objectives

The principal reason for doing evaluation is to establish whether or notpolicy has contributed to correcting or ameliorating the problem it set out toresolve. This is often thought of in terms of tackling market failures thatreduce economic efficiency, such as inadequate availability of finance, skills,advice and technologies, but may also encompass a desire to improve equityamong groups of people or places, for example by supporting entrepreneurshipamong unemployed youth or entrepreneurship in poor localities. Evaluationof these impacts is facilitated by a clear statement of measurable outcomesright at the start of the policy/programme design and the collection of relevantdata throughout its life.

To make informed decisions about the allocation of funds

Governments manage a portfolio of policies and programmes each withits own rationale and justification. Evaluation assists managers to assess therelative effectiveness of these policies and programmes and to makejudgements about where to place their efforts in order to obtain the greatestbenefits for given costs. Evaluation evidence can help to identify wheregovernment can make the biggest difference to its objectives and targets.

To show the tax payer and business community whether the programme is a cost effective use of public funds

The scale of tax-payers funding for entrepreneurship and small businesspolicies clearly varies from one country to another. It also varies according toprecisely what is incorporated into the definition. Nevertheless the amountsOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 17

1. EVALUATION ISSUESare usually substantial. For example, EIM (2004) reports that approximatelysix billion Euros was spent annually by EU Member States on state aid to smalland medium sized enterprises.3 However, even this may be a considerableunderestimate. One EU Member State the United Kingdom in acomprehensive review of tax-payers funding directed towards SMEs, reportedthat 2.5 billion GBP of public money was spent on direct support to SMEs inEngland alone, PACEC [2005] quoted by [National Audit Office (2006)]. A thirdexample is a programme in the United States the Small Business InnovationResearch Program. Cooper (2003) reports this programme made annual awardsof USD 1.1 billion in the 1997-1999 calendar years.4

These examples illustrate that, probably for most developed countries,public funding of SMEs is substantial, even if it is extremely difficult toquantify in aggregate and may still be relatively modest in terms of tax-payerssupport to large enterprises. Given these substantial sums of public money, itis reasonable for tax-payers to be reassured that their funding is being spentin an appropriate manner. It is reasonable for tax-payers to demand evidencethat public programmes are spending funds in accordance with their statedobjectives. This role is normally played by public auditors. A second role, butone not normally played by auditors, is to assess whether the public funds areachieving the objectives set out by politicians. This is the function ofevaluators.

To stimulate democratic debate

In democracies, it is reasonable for the electorate to question thedecisions made by governments. In order to facilitate that debate, it isappropriate for organisations to be able to have access to evidence on theimpact of policies. In this regard, SME and entrepreneurship policies are nodifferent from other areas of government expenditure. For this reason, theresults of evaluations enhance and inform public debate.

This debate only takes place when the results of evaluations enter thepublic domain. This emphasises not only the importance of undertakingevaluations, but also of their findings being disseminated.

To achieve continued improvement in the design and administration of programmes

Politicians and public servants administering SME and entrepreneurshipprogrammes should be seeking continuous improvements and there is ofcourse a need to ensure adaptation to changing conditions. Evaluation is a keytool for learning about how well policies and programmes are delivering, whatproblems may be emerging, what elements work well and less well and whatcould be done better in the future. For example, policy makers may seek toOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200718

1. EVALUATION ISSUESdeliver policies to different groups, for example by directing more resourcestowards enterprises established by the socially disadvantaged or by thoselikely to employ others, or those in high technology. They may seek to deliverpolicies using different organisational forms, to stimulate the take-up ofpolicies or to deliver them in a more cost effective manner. All these changesof focus can emerge from undertaking appropriate evaluations. Alternatively,existing policies can be delivered more effectively as a result of accumulatedevaluation experience.

Typical objections to evaluation and responses

The discussion above focussed on the positive aspects of evaluation.However, one of the barriers to spreading an evaluation practice is a resistanceto evaluation amongst a range of politicians, policy makers and practitioners.Here we discuss some of the most common objections to evaluation and thedegree to which they stand up to critical assessment. Our judgement is thatalthough the objections have some weight, on balance, they do not amount toa solid case for rejecting evaluation and hence sacrificing the benefits citedabove.

But evaluation is expensive and bureaucratic

Evaluation is not costless. Costs include the payment of consultants/evaluators, the collection of data and the time taken from those deliveringprogrammes to inform the evaluation. The United Kingdom statistical office,for example, requires the time of recipients of the programme in providingtheir opinions and information about the programme to be costed (i.e. the costof the respondents time must be explicitly included in the cost of theevaluation). Data may also have to be collected from both clients of theprogramme, and a control group of non clients.

However, the resources committed to evaluation are normally verymodest in comparison with the total size of the programme. For example, thereview by Sheikh and Steiber [2002], Evaluating Actions and MeasuresPromoting Female Entrepreneurship identified an appropriate budget ofbetween 2% and 5% for the purposes of evaluation. This may be appropriatefor small programmes but for programmes in larger countries a figure ofbetween 0.5% and 1% of annual expenditure would be more usual.

Given the opportunity which evaluation provides for using resourcesmore efficiently, and for the design of new programmes, these seem to be verymodest costs indeed.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 19

1. EVALUATION ISSUESBut evaluation does not always lead to policy improvements

Evaluations of programmes can fail to lead to policy change for severalreasons. It may be because those responsible for programme management arehostile to the concept of evaluation. It can also happen where evaluators failto engage programme managers, or where they fail to understand the detailsof the programme. Evaluators themselves may fail to express their findings ina language that is easily understandable to policy makers and thoseresponsible for policy delivery.

Although there are instances where evaluations have not led toimprovement, this is not a sufficient justification for being reluctant toundertake any form of evaluation. To minimise the potential problems,programme managers have to be persuaded that the quality of programmedelivery can be enhanced through evaluations and the consultants have toreach out to programme managers to engage them wherever possible.

But ultimately evaluation takes place for the benefit of the tax-payer, andnot for the provider[s] of the programme. Those programmes that are shownto be demonstrably ineffective have to be closed, and this has to be recognisedby programme managers.

In practice, if evaluation is to lead to change, a balance must be struckbetween, on the one hand, ensuring the independence of the evaluator whilst,on the other, engaging support of those involved with programme delivery.

And risks diverting attention away from programme delivery

It is the case that there are cultural differences between evaluators anddeliverers of programmes. The former are often analytical individuals, oftenwith an academic background, whereas the latter consider themselvespractical individuals focused upon delivering services to their clients. Becausethey are so close to their clients they view themselves as the best judge of theeffectiveness of the programme. They have difficulty seeing what value adetached consultant can provide in terms of programme improvement. Forthis reason, programme deliverers often resent the time taken in completingforms and collecting data which are, however, vital to the success of anevaluation. Programme managers and deliverers understandably can also feelthreatened by an evaluation, especially when they know they do not fullyunderstand the techniques used by the evaluators, but fear the evaluators donot fully understand the programme.

For an evaluation to be a success however, these cultural differences haveto be managed. The most effective way of achieving this, as identified above,is to demonstrate that the interests of both the evaluators and the programmemanagers/deliverers can be more closely aligned by both parties focussing onareas for programme improvement. This can be most effectively achieved byOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200720

1. EVALUATION ISSUESengaging those delivering the policy through ensuring the issues of concernare addressed in the evaluation and by being adequate opportunity tocomment upon, and offer their interpretation of, provisional findings.

It is, of course, a simplification to imply that the programme managersmost hostile to evaluation are those fearing negative feedback. Neverthelesssenior policy-makers need to be aware that evaluation, whilst it is in thetaxpayers interests, may provoke considerable hostility from programmedeliverers. The latter have to be engaged but not the ultimate voice.

But evaluation is only for advanced countries

It is the case that programme evaluation is more frequently undertakenin advanced, rather than in developing economies. In part this may be becauseit is more difficult to find sufficient numbers of individuals with the type ofanalytical skills necessary to conduct good quality evaluations in developingeconomies. Major donor organisations, such as the World Bank, can thereforeplay a role in both undertaking evaluations themselves and in training othersto perform these tasks.

Nevertheless it is not only the most developed countries that undertakeevaluation. In its review of state aid to SMEs, EIM [2004] surveyed EU MemberStates, European Economic Area and candidate countries. A total of29 countries were identified. Only Ireland, the Netherlands and Slovakiaperformed state aid evaluations on all schemes, implying that evaluation isnot simply characteristic of the more wealthy countries. EIM specifically notedthat the State Aid Act obliges the Slovak Government to evaluate all state aidusing statistical analysis of aid recipients and control groups. They also notedthat the analyses are performed on macro and micro levels. Full details of thisimportant survey are provided in Appendix F.

Mexico has also recently committed to undertaking SME evaluation. Itbelieves this will improve support systems and identify areas of opportunity,thus granting certainty to the population on the efficient use of resources.

These examples illustrate that it is not necessarily the most economicallydeveloped countries which are committed to undertaking evaluation.

But there is no history of undertaking evaluation

In countries without a tradition of evaluation it can be difficult to makethis transition. Nevertheless, it is clear that the electorates in many countriesare becoming more sophisticated, in part because of access to the media andthe internet. Countries where evaluations do not take place are likely, in thefuture, to be asked why it is that such policy assessments take placeelsewhere. The, perhaps unjustified, inference is that evaluations do not takeplace because there is something to hide. It is not sufficient to imply thatOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 21

1. EVALUATION ISSUESpolicies are being delivered efficiently because there is no information to thecontrary.

Key evaluation debates

This section reviews four key evaluation debates. The first is theappropriate technique for evaluating SME and entrepreneurship policies. Thesecond is the appropriate level of sophistication of the quantitative evaluationapproaches. The third is whether evaluation should be undertaken byinsiders or outsiders. The fourth is whether the same evaluation techniquesshould be used for all programmes.

The choice of technique

There are two basic options in undertaking summative evaluations5 thequantitative and qualitative approaches. Quantitative evaluation involvesassessment of the impact of programmes through a comparison of outcomesbetween the group in receipt of aid and some form of control group, forexample a similar group of enterprises that have not benefited from policy orthe same enterprises before and after receipt of policy support. Such data maybe collected either directly from the firms themselves or from official data.Qualitative evaluation or approaches are much more likely to rely upon theopinions of programme stakeholders including managers and beneficiariesabout the functioning and impact of the programme through techniquesincluding surveys, case studies and peer reviews. Both approaches will relyupon a careful scrutiny of programme documentation.

Table 1.1 reviews the advantages and disadvantages of the quantitativeand qualitative approaches.

The principal advantage of qualitative evaluation is the additionalinformation that it can provide beyond that associated with quantitativeevaluations. Qualitative evaluation normally involves face-to-face discussionswith those in receipt of aid, those responsible for delivering programmes andother stakeholders. These conversations help not only to obtain informationfrom stakeholders that can lead to a deeper understanding of the mechanismsby which policy impact is achieved and how policy might be adjusted but alsoto engage stakeholders in policy learning processes. The approach can alsopick up a wide range of other information of interest to policy makers, goingbeyond impact to issues such as client satisfaction, policy appropriateness,sustainability and conflict with other policies.

However, qualitative evaluation has the major disadvantage that it is notgood at providing reliable estimates of policy impact for a number of reasons.First, surveys of a sample of stakeholders run the risk of being unrepresentativeof programme participants. Increasing the numbers however either addsOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200722

1. EVALUATION ISSUESconsiderably to budgets or reduces the quality or depth of the interviews.Second, despite the best efforts of interviewers, there remains a strong risk ofinterviewer bias. Thirdly, the outcome of qualitative evaluation is more oftento describe a process rather than to evaluate an outcome. Fourthly, there is noopportunity for independent verification. Finally, programme participantsmay be asked questions that are virtually impossible to answer. The classicexample is What impact do you think this programme had on yourbusiness? Implicitly the respondent is required to hold every other influenceon their business constant and estimate how a programme which probablytook place some years previously has influenced their business in theintervening period. Even if some programme participants were able toundertake such mental gymnastics others clearly are not and there is no wayof distinguishing between the answers of the two groups.

The principal disadvantages of the quantitative approach concern itstechnical difficulties and the relatively narrow nature of the results it offers,which focus primarily on issues of effectiveness and efficiency. In terms of thetechnical issues, effective quantitative evaluation requires extensive datacollection on the performance of policy-targeted and control group firms.More importantly, however, in SME and entrepreneurship policy evaluationsituations, there may sometimes be no natural, uncontaminated controlgroup. Whilst good quantitative analysis seeks to match as closely as

Table 1.1. Qualitative compared with quantitative evaluation

Qualitative evaluation methodologies Quantitative evaluation methodologies

Advantages Disadvantages Advantages Disadvantages

Engages participants in policy learning

Respondents and interviewers may be biased or poorly informed

Clear answers on impact Cost of data collection and technical demands

Can vary the scale and hence cost

Rarely provides a clear answer

If well done will get close to true impact

Lacks information on context and mechanisms behind policy impacts

Deeper understanding of processes leading to impacts

Tends to describe rather than evaluate

Can be independently verified

Absence of pure control groups

Should be easy to interpret Risks including un-representative groups

Possible false impression of precision

Can assess against a wide range of evaluation criteria

No opportunity for independent verification

Narrow focus on effectiveness and efficiency

Picks up unintended consequences

Hard to judge efficiency and effectiveness

Difficult to use on indirect interventions that seek to influence the business environment

Better understanding of policy options and alternatives

Hard to establish cause and effectOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 23

1. EVALUATION ISSUESpossible policy-influenced and non-policy-influenced firms and seeks toaccount for possible selection bias between the two groups, there are alwayssome differences between the treatment and control groups that cannotbe taken into account. To address this, evaluators in several OECD countrieshave collaborated with their own statistical agencies to derive samples ofSMEs with prescribed characteristics so as to act as a control group. Evenso, some evaluators may be tempted to give a false impression of precisionin reporting their results. In terms of the nature of the results the maindrawback is the problem of the black box, i.e. that little information isprovided on the nature of the policy problem and how it is addressed bypolicy and hence on how policy might be adjusted to increase impact. Thiscan be reflected in an unduly narrow focus of quantitative approaches ontwo evaluation criteria, namely efficiency (impacts against expenditure) andeffectiveness (impacts against targets), that can leave other evaluationquestions unanswered.

On the other hand, the fundamental advantage of quantitative evaluationis that it should provide clear answers. If it is well done it will get as close aspossible to a value-free assessment of impact. Of course, no evaluation iswholly value-free.

Given the advantages and disadvantages of both approaches, thisFramework argues for the use of a plurality of approaches that are able togain from the complementarities in the information they can provide. Therole of the qualitative approach to evaluation is recognised and the role ofsurvey, case study and peer review approaches is outlined in this respect.However, the Framework focuses in particular on setting out the issuesinvolved in undertaking good quantitative evaluations, reflecting theoriginal concern of the OECD Working Party on SMEs and Entrepreneurship(WPSMEE) to share information on best practices in impact evaluation. Thisreflects both the perception that quantitative impact evaluations are notsufficiently used in SME and entrepreneurship policy evaluation and thepresence of some difficult issues that are not sufficiently well understood bypolicy makers, particularly in accurately establishing the counterfactual.

Assessing quantitative evaluations: The Six Steps to Heaven

A useful guide in developing robust quantitative evaluations andassessing the quality of such evaluation evidence is the so called Six Stepsto Heaven approach, (Storey, [2000]), reviewed and operationalised recentlyby Lenihan et al. [2007], Bonner and McGuiness [2007], and Ramsey and Bond[2007].OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200724

1. EVALUATION ISSUESThe Six Steps methodology is a categorisation in which Step 1 is the least,and Step 6 the most, sophisticated approach. The six steps are:

Step 1. Take up of schemes.

Step 2. Recipients opinions.

Step 3. Recipients views of the difference made by the assistance.

The above three steps tend to be associated with qualitative approaches,but the following three steps typify quantitative evaluations:

Step 4. Comparison of the performance of the assisted with typical firms.

Step 5. Comparison with match firms.

Step 6. Taking account of selection bias.

This is an approach that is mainly relevant to quantitative and ex postevaluations rather than to qualitative and ex ante evaluation. It is nonethelessa very helpful framework for assessing the former type of evaluations and isreferred to a number of times in this Framework, notably in relation to wherethe evaluation examples provided later in the report stand in relation to thedifferent levels of sophistication in establishing impact.

Fuller details of the approach are to be found in Appendix B, which istaken from OECD (2004a).

Evaluation by insiders or outsiders?

A second key evaluation debate is who should undertake the evaluation should it be insiders or outsiders? The arguments for and against are set outin Table 1.2 below.

The key argument in favour of using external evaluators is that they arenot only less likely to be influenced by the political regime, but they are alsomore likely to be seen, by others, to be independent.6 This independence islikely to provide more objectivity to the evaluation. A second argument for the

Table 1.2. The choice of internal and external evaluators

External evaluator Internal evaluator

Advantages Disadvantages Advantages Disadvantages

Less likely to be influenced by political regime

Less well informed of the real situation

More insights through understanding the realities on the ground

Lack of independence

Seen by others to be independent

Less able to drive through change as a result of the recommendations

More chance of buy-in from those delivering the programme

Less likely to be able to think outside the box

Brings new ideas and fresh approaches

More chance of really changing policyOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 25

1. EVALUATION ISSUESuse of external evaluators is that they can bring new ideas and freshapproaches not only to the evaluation but also to subsequent policydevelopment.

In contrast, the key advantage of using internal evaluators is that theyfrequently have a much better knowledge both of the policy itself and of thepolitical context in which it is undertaken. Internal evaluators therefore haveto spend less time in acquainting themselves with the detailed workings ofpolicy and can focus much more upon producing targeted recommendations.Internal evaluators also are more likely to engage the support of the managersdelivering the programmes because of their greater knowledge and becausethey are perceived to be less threatening. Finally, internal evaluators are alsomore likely to be careful about their policy recommendations, since they willhave to perhaps live with, and possibly implement, any changes theyrecommend. Unlike external evaluators they are less likely to be able to walkaway from the issue.

The OECD (2004a) recognised that the choice of internal or externalevaluators was a close call. Much might depend upon whether, in commissioningthe evaluation, the purpose was to undertake a root and branch approach inwhich case external evaluators might be preferred. In contrast, evaluationdesigned to ensure programmes were on track might favour the use ofinternal evaluators.

Ultimately, therefore, there is a broad choice between selectingevaluators who are more independent but with perhaps less policy insight andevaluators who may be less radical in their recommendations but whoperhaps are more likely to induce changes in programmes. The IstanbulMinisterial Declaration, however, made it clear that it favoured independentbut informed evaluators.

It is also possible to develop alternative models that are neither fullyinternal nor fully external. For example, some government departments andagencies create independent evaluation units that are not directly attached orresponsible to the particular units responsible for the programmes that theyevaluate. Another option is to create teams of evaluators, with some comingfrom inside and some from outside the organisation. This latter approach istypical of the peer review method described in Section 4.

Should the same evaluation techniques be used for evaluating all programmes?

A third debate is whether the same approaches should be used to reviewall programmes. The central argument favouring a similar approach toevaluation is that the ultimate purpose, if the tax-payer is to obtain value formoney from SME and entrepreneurship policies, is that all programmesOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200726

1. EVALUATION ISSUESshould have the same effectiveness at the margin. In simple terms it shouldnot be possible to transfer funds from one programme to another and increasethe benefits to SMEs and/or the wider economy. So, the economic impact ofpolicies to reduce taxes for SMEs should have the same marginal benefit aspolicies to provide export advice or management training or access to finance.

There is clear evidence from work on SME evaluation that the methodsused for evaluation appear to influence the apparent effectiveness ofprogrammes and policies. Expressed baldly, the less sophisticated theevaluation the more likely it is to apparently demonstrate benefits. Thisreflects the more simple evaluations failing to hold constant the myriad ofinfluences on outcomes and, by implication, attributing them to theprogramme. In contrast, the more sophisticated approaches strip out theother influences, and so only attribute to the programme its real effects.

This finding has major implications because it means that it is not validto compare the findings from a study which uses a Step 2 or Step 3 approachwith that which uses a Step 5 or Step 6 approach. Indeed it may even beinvalid to compare findings between Steps 5 and 6. Hence it means that onlyby using a uniform methodology can governments really ensure thatentrepreneurship and SME policy is efficiently delivered.

The opposing argument is the following: that programmes varyconsiderably in scale and budget and that if a fixed proportion of programmefunds are to be allocated to evaluation then inevitably evaluation budgets willalso vary. More sophisticated evaluations are, of course, generally both moreexpensive and with higher fixed costs than less sophisticated approaches.This means that if the same approach were used across all programmes thensmall programmes would have to devote a much higher proportion of theirfunds to evaluation than is the case for larger programmes. This is unrealistic.

Both arguments, of course, have validity, but some form of compromise ispossible. If the desirability of uniform evaluation procedures is accepted, then itstill may be possible for individual smaller programmes to be evaluated lessfrequently, or possibly as part of an evaluation of a package of small programmes.What is clear is that programmes with small budgets should not either escapefrom all evaluation or be assessed by radically different and by implication lesschallenging procedures.

Doing evaluations

This section examines the practical issues of how to prepare, manage anddisseminate evaluations. Further useful information on preparing, managing anddisseminating evaluations is provided in the evaluation guidance documentsreferred in Appendix C, including the web resources of the OECD DevelopmentAssistance Committee (DAC) Evaluation Resource Centre.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 27

1. EVALUATION ISSUESPreparing an evaluation

A number of key issues have to be addressed when preparing anevaluation: the first is to identify precisely what it is that will be evaluated.This can be a problem when one item in a package of assistance is beingassessed. For example, some SME programmes combine both financialassistance and business advice. It is therefore important, at the outset, todecide whether the whole programme is to be evaluated or whether thecomponent parts are to be evaluated separately. The advantage of examiningthe whole programme is that an overall assessment of the use of public fundscan be undertaken. But separately examining the component parts thefinance and the advice may show that it is only one or the other that is reallyeffective. For example, the evaluation might show that the impact on firmperformance is primarily due to access to finance. In that case, resourcesmight be moved away from the advice towards activities that improve accessto finance.

A second question is when the evaluation should be conducted. Hereagain there is no simple answer because some forms of assistance take longerto impact on firm performance than others. For example, a programmedesigned to network firms with one another at a trade fair might be expectedto have an impact in terms of additional sales within 3-6 months. In contrast,a programme to provide management training for SME owners might not beexpected to have significant impacts for at least 2-3 years. Finally,entrepreneurship policies such as those designed to influence the attitudesof school children to enterprise creation might not be expected to beobservable for at least 20 years. Given these varying likely outcomes the periodafter the policy is implemented after the evaluation takes place is also likely tovary. However, a broad rule of thumb is that SME policy initiatives such as theimpact of loans and grants should plan for the evaluation immediately thepolicy is introduced and begin the formal evaluation within 2-3 years.

The objectives of the programme have to be clearly specified

Unless programmes have objectives which are in principle capable ofmeasurement then a quantitative evaluation cannot be undertaken. Veryoften these objectives are set out in a logic model that provides policy makersand evaluators with a clear understanding of possible programme outcomesand how they are likely to be achieved. This is important for evaluationbecause it provides a guide to what should be assessed and measured. Logicmodels can take many forms, which will all be valid as long as they clearlyexpress what policy is seeking to achieve. An example from New Zealand is setout in Figure 1.1. The University of Wisconsin online reference guide onEnhancing Programme Performance with Logic Models provides further usefulOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200728

1. EVALUATION ISSUESinformation for the development of these tools.7 The involvement of skilledevaluators at the outset of programme formulation will help ensure thatobjectives and programme logic are clearly stated. In practice, however, this isnot always the case. This means that a key function of the evaluator has to beto infer perhaps even guess what the objectives of the policy maker werewhen the programme was designed. Although this might seem curious, it isoften in practice a very valuable role of the evaluator, and even more so forpolicy-makers and programme manager.8

Specifying the content of the evaluation

Those responsible for preparing the evaluation have to be clear,particularly when the evaluation is to be undertaken externally, about what itis expected to achieve and what the role of the evaluation manager will be.The latter may have to assist consultants in clarifying both the objectives ofthe programme and the current requirements of politicians. Howeverevaluation managers should not normally specify in detail the methodology tobe used but merely identify the questions to be addressed such asadditionality, dead weight or displacement.9 To specify the methodology indetail would be to exclude the possibility of evaluators employing new ornovel approaches. The only clear exception would be where the purpose is todirectly compare policy impacts at two points in time. Here there would bemerit in using a similar methodology, providing the chosen method wasdeemed to be satisfactory.

Ensuring good data are available

Whatever level of sophistication of evaluation is used, a minimumrequirement is that data are available. For example an evaluation of theimpact of business advice or of loans or grants requires, as a minimum, acomplete and up to date list of clients to be available. Until such informationexists no evaluation should even be contemplated.

Managing an evaluation

Six major issues arise in managing an evaluation.

Should the evaluator be internal or external?

This issue was discussed in-depth above and it was concluded that whilstthe external evaluator was more likely to be independent from theorganisation responsible for designing and delivering policy, and hence morelikely to be critical of it, the internal evaluator was likely to have greaterknowledge and political awareness. There would be, therefore, circumstancesOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 29

1. EVALUATION ISSUESin which either was appropriate but, all else equal, the independence of theevaluator was critical.

The scale of the budget

As noted earlier the scale of budget strongly influences the methodologywhich can be undertaken. It also influences the outcome of the evaluationsince it seems that inexpensive evaluations seem to produce more positivefindings. This can produce considerable pressure to undertake only the mostsimple of evaluations.

Terms of reference

Terms of reference need to be clearly specified, but not in a way whichprecludes innovative thinking.

Choosing a consultant

As noted earlier the first key choice is whether the consultant should beexternal or internal. If the choice is to restrict the consultants to those fromoutside the organisation, the key choice then is to select individuals ororganisations which have a track record in delivering timely and appropriateevaluations using their chosen methodology. So, for example, if a statisticalstudy was required, it would be inappropriate to choose consultants whoseprime track record was in the use of qualitative methods.

Timetable

A timetable for the delivery of the evaluation has to be specified. Veryoften this coincides with a wider appraisal of policies within thecommissioning department. To achieve this there has to be agreed milestonesin the form of interim and final reports to ensure that research is on track. Toachieve this it may be necessary to have a small steering committee.

In practice, however, the more sophisticated evaluations frequently tendto overrun in terms of time. This is because of the difficulty of contactingappropriate numbers of enterprises often because the lists given to theconsultants are incomplete. Hence, a crucial element of ensuring thatevaluations are on time is to ensure that the base data are of high qualitybefore the evaluators begin their work.

Quality assessment

Initially an assessment has to be made of the quality of the evaluation.One clear purpose of this Framework is to enable an accurate assessment ofthe quality of the evaluation to be made. Our overwhelming focus is upon thetechnical quality of the evaluation defined as the extent to which policOECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200730

1. EVALUATION ISSUES

Fi

gure

1.1

.N

ew Z

eala

nd

Tra

de

and

En

terp

rise

(NZ

TE)

Gro

wth

Ran

ge P

rogr

amm

e Lo

gic

Mod

el

Sou

rce:

New

Zea

lan

d T

rad

e an

d E

nte

rpri

se.

!"

!

"

"

!

"

!

"

!"

"

#$

%#

&'

#

"

$%

#$

%#

&'

#%

%

#()

*'!

&

+

!#

'

($

"

"

)

)

*

+

)*

$

,

"

'

% #

&,

%#()

#

-%)

,

!'

,

%%

&

#(

#

-

"

)

*

.

.

)

/

0

"

!

!

"

)

"

#

"

)

1

"

#

"

#

)

,

-

"

"

"

!

"

+

)OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 31

1. EVALUATION ISSUESmakers can be confident that an identified programme impacts genuinelyreflect the contribution of the programme. However other considerations mayconcern those managing evaluations and these are outlined in Appendix D.

Dissemination

A key decision is the extent to which the results of an evaluation enterthe public domain. Some evaluations normally those conducted by insiders are specifically targeted towards the interests of, for example, departmentalcommittees. Other evaluations, however, are intended to contribute to a widerpublic debate on SME or entrepreneurship policy. These are likely to beundertaken by academics with a commitment to disseminating their work.Therefore, a key decision is the nature of any dissemination which is to beundertaken.

A second issue is the extent to which the data collected by the evaluatorsis able to be accessed by other researchers in the area. Such researchers maywish to verify the interpretations and conclusions that have been placed uponthe data, but much depends upon data protection legislation. The latter alsoinfluences the opportunities for researchers to use the data for extending theresearch over time by following up these businesses or individuals or derivingsamples from similar businesses or individuals in other industries or regions.In principle therefore, subject to complying with data protection legislation, itis desirable for the data derived from evaluations to be available to bona fideresearchers.

Ultimately, however, the purpose of evaluation is to stimulate and informpublic debate. It is desirable that evaluation reports are made available to themedia and other interested parties. Only in this way can evaluation truly leadto policy change.

Some key principles for evaluation practice

Drawing on the discussion above, the following key principles forevaluation can be set out.

1. Evaluation should lead to policy change

The prime purpose of undertaking evaluation is that it informs keydecisions. Such decisions may be to change policy. For example, it may lead toa policy budget being increased, decreased or the policy itself beingabandoned. It may also lead to different objectives of the policy beingspecified and, most likely, will lead to the policy being delivered in differentways possibly to different target groups. Alternatively, the policy decisionmay be that no change is required and that the programmes are on track.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200732

1. EVALUATION ISSUES2. Evaluation should be part of the policy debate

If it is to be at the heart of policy making, evaluation cannot be confinedto providing a historical review of previous policies. Instead, evaluation has tolead to policy learning so that current policies may be amended in the light ofthis knowledge and new policies developed from such learning.

3. Evaluators should be in at the start

Evaluation is, as noted above, vital in the formulation of new policies.Those skilled in evaluation techniques can make major contributions to policydevelopment, most notably in helping policy makers to clearly formulatepolicy objectives. Without the input of evaluators, policy objectives may not bespecified at all, or be expressed in such a way that they cannot fail to beachieved, or be specified in such opaque language that it is impossible todetermine at a future point whether or not they were achieved.

The role of the evaluator is to ensure that policy makers specify clear andtangible objectives as policy is being developed. A second merit in evaluationbeing included at an early stage is that budgets for evaluation are specifiedwhen programmes are being formulated. A third advantage of evaluatorsbeing in at the start is the methods to be used to determine the success orotherwise of the programme are clearly brought to the attention of the policymakers. Finally, policy makers are made aware that a programme is to beevaluated and the criteria of success that will be used to assess effectivenessare agreed in advance by all parties.

4. Evaluation techniques should always use the most appropriate methodology

Subject to the provisos outlined below, the most appropriate evaluationtechniques should be employed. The Six Steps to Heaven is a simple methodof assessing sophistication in this area, and for impact evaluation of mediumto large sized programmes we recommend that at least a Step 4 approachshould be used. By this we mean that, as a minimum, the beneficiaries of aprogramme or policy should be compared with a control group of otherwisesimilar firms or individuals, but who did not participate in the programme. Bycomparing outcomes for the two groups a crude estimate can be made ofpolicy impact. Governments, with their substantial access to statistical data,always have the opportunity to formulate such control groups, even if on someoccasions their own statistical agencies are unwilling to collaborate in suchwork. The importance of evaluation however is such that the statisticalagencies should be mandated to develop this aspect of their evaluationculture.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 33

1. EVALUATION ISSUESHowever, in order to understand more deeply the processes of how policyworks and to involve stakeholders in policy learning qualitative methods suchas peer reviews and case studies also have their place. These methods are notsuited for robust impact estimates but are necessary for many other aspects ofevaluation.

5. Evaluation should apply to all policies and programmes

It appears that some policies and programmes are evaluated many times,and with some rigour, whereas others seem to either escape scrutinyaltogether, or are evaluated in a much less challenging manner than otherpolicies/programmes. This is extremely unfortunate since the overall purposeof policy is to ensure that, at the margin, all policies are equally effective.

6. International comparisons should be made where necessary

Finally this Framework concludes that for some policy areas, evaluationcan only be undertaken on an international basis. For example, comparing theimpact of tax regimes is best undertaken across countries. As noted above,international comparisons and peer reviews are appropriate for reviewingpolicies at the regional or local level. In this respect, OECD can serve a valuablefunction as a clearing house for information on policy effectiveness, forproducing harmonised data and in having access to experts in this field.

Notes

1. This term was coined in OECD (2004a). It refers to evaluation only beingconsidered after the objectives and targets have been set, and the programme hasbeen in operation for some time. When evaluation is at the end of the line it canonly serve as a historical auditing function. OECD (2004a) contrast this with themore preferred Integral Evaluation. Here Evaluation is integrated into the policyprocess. So, as the policy is being developed, consideration is given to how it willbe evaluated. This has four merits. The first is that it ensures that the objectives andtargets of the policy are clearly specified. Second, it ensures that the necessary datacollection can begin immediately and is built in to the programme. Thirdly, itensures that the evaluation budget is identified. Fourthly, it ensures that theprogress of the programme is monitored so that modifications and improvementscan be made in the light of evidence.

2. Evaluation of SME Policies and Programmes, background report for the 2nd OECDConference of Ministers Responsible for SMEs, Istanbul, June 2004.

3. Source: European Commission State Aid Scoreboard, Spring 2004 update.

4. The awards specified by Cooper were 1.107 in 1997, 1.067 in 1998 and 1.097 in 1999 all in billions of USDs.

5. For formative and prospective evaluations, the opportunities for undertakingquantitative/statistical approaches are generally much less, unless they drawupon previous summative evaluations.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200734

1. EVALUATION ISSUES6. Examples of external evaluators are generally those not employed as publicservants by the government. These will include private sector consultants andresearch contractors, such as academics.

7. www.uwex.edu/ces/lmcourse/#.

8. For example, when the objectives of a programme are unclear the evaluator mightask the question What outcome from the programme would you identify asfailure? Experience of asking this question is that it is initially met with hostilesilence. Frequently it is then followed by specifying outcomes that are [close to]impossible. These might include Nobody is interested in the policy. Afterconsideration, however, more realistic objections normally emerge.

9. Additionality refers to the proportion of the supported activity that would nothave gone ahead without the support, deadweight activity that would have goneahead anyway, displacement refers to reductions of activity elsewhere (e.g. inother firms) as a result of increased competition from the supported activity (e.g.the firms receiving assistance). OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 35

ISBN 978-92-64-04008-3OECD Framework for the Evaluation of SME and Entrepreneurship

Policies and Programmes

OECD 2007

Section 2

Evaluation of Individual National Programmes37

2. EVALUATION OF INDIVIDUAL NATIONAL PROGRAMMESIntroduction

This section takes examples of SME and entrepreneurship policies from anumber of countries and seeks to report, using a common format, the resultsof evaluations. The vast majority of these evaluations are in the publicdomain.

For ease of access the published evaluations are categorised by topic.These are:

Financial assistance;

Enterprise culture;

Advice and assistance;

Technology; and

Management training.

Table 2.1 shows some specific policies that are included under theseheadings.

Table 2.1. SME and entrepreneurship policy areas covered

Financial assistance Enterprise culture Advice and assistance Technology Management training

Loan Guarantee Schemes.

Programmes to encourage young disadvantaged individuals to start businesses.

Provision of Marketing Advice.

Subsidies to New Technology Based firms.

Subsidies to stimulate the take up of management training in SMEs.

Subsidising the creation of businesses and growth of SMEs perhaps by disadvantaged groups.

Programmes to encourage graduates to start businesses.

Provision of general business advice.

Creation of Science Parks.

Tax relief to business angels.

Enhancing Investment readiness of SME owners.

Encouraging SMEs to export.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 200738

The central purpose of this chapter is to provide a number of examples ofevaluation of public programmes to assist SMEs in each of the five key policyareas. Some of these evaluations can be considered to be sophisticatedaccording to the Six Steps framework but others are less so. The inclusion of

2. EVALUATION OF INDIVIDUAL NATIONAL PROGRAMMESthe less sophisticated approaches is justified partly on the grounds that thereis an absence of top quality evaluations from which to draw. It is also toemphasise that there is an opportunity for improvement in evaluationmethods. Thirdly it shows that good practice is found in evaluations in almostall SME policy areas but, in our view, is not yet the norm.

Evaluations of financial assistance

Table 2.1 shows there are primarily three types of financial assistanceprogrammes that have been evaluated. These are, firstly, loan guaranteeprogrammes, secondly subsidies to disadvantaged groups to encouragethem to start or grow a business, and thirdly, subsidies to develop the marketfor business angels.

Loan guarantee programmes

Evaluations of loan guarantee programmes from Japan and Canada areshown in Tables 2.2 and 2.3.

Subsidising the creation of businesses and growth of SMEs perhaps by disadvantaged groups

Two examples of evaluations of these policies are presented in thissection although the programme in Table 4.8 could also be taken as anotherexample. The first is of Law 44 in southern Italy where the disadvantagedgroup is young people (Table 2.4). The second is grant assistance to SMEs inNorthern Ireland and the Republic of Ireland (Table 2.5).

Table 2.2. Loan guarantee scheme, Japan

Available online http://isb.sagepub.com/cgi/content/abstract/23/1/48

Country Japan

Time period of study 1980-2002

Title of report Promoting Enterprise Development or Subsidising Tradition: The Japanese Credit Supplementation Scheme.

Date of report 2005

Author/details M. Nitani and A. Riding, International Small Business Journal, Vol. 23[1], pp. 48-71.

Objective of policy To contribute to the smooth flow of funds by guaranteeing loans that are advanced to SMEs by banks or other financial institutions.

Key findings The Japanese Credit Supplementation Scheme is much more targeted to averting the failures of weak firms and much less focussed on new and growing firms.

Sophistication of evaluation Step 1: Reviews publicly available data. No survey undertaken.

Comments Helpful review placing Japan in context of other OECD countries operating an loan guarantee scheme.OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ISBN 978-92-64-04008-3 OECD 2007 39

2. EVALUATION OF INDIVIDUAL NATIONAL PROGRAMMES

d

mall

rvive

ays:Table 2.3. Loan guarantee scheme, Canada

Available online www.ingentaconnect.com/content/els/08839026/2001/00000016/00000006/art00050Country CanadaTime period of study 1989-1997Title of report Loan Guarantee: Cost of Default and Benefit to Small Firms.Date of report 2001Author/details A.L. Riding and G. Haines, Journal of Business Venturing, Vol. 16[6], pp. 595-612.Objective of policy To increase the availability of loans for the purpose of the establishment, expansion, modernisation an

improvement of small business enterprise. To encourage lenders to provide debt financing to SMEs, that the lenders would otherwise consider too s

or risky.Key findings The SBLA is an extraordinary effective means of stimulating job creation and assisting small firms to su

and grow. The Canadian default rates are well below those for other countries.

Sophistication of evaluation 232 000 loans were examined. Comparisons made with other OECD countries.

Comments Very helpful international review of LGS findings.

Table 2.4. Assistance to new enterprises started by young people, Italy

Available online www.informaworld.com/smpp/content~content=a713693784Country ItalyTime period of study 1988-1997Title of report The Life Duration of Small Firms born within a Start-up Programme: Evidence from Italy.Date of report 2001Author/details A. Del Monte and D. Scalera, Regional Studies, 35.1, pp. 11-21.

Objective of policy

Law 44 supports the creation of new firms by young people [< 35 years old] in Southern Ita