evaluating the impact of reforms of vocational education ... · evaluating the impact of reforms of...

Evaluating the impact of reforms of vocationaleducation and training: examples of practice

Mike Coles

In:

Descy, P.; Tessaring, M. (eds)

Evaluation of systems and programmesThird report on vocational training research in Europe: background report.

Luxembourg: Office for Official Publications of the European Communities, 2004(Cedefop Reference series, 57)

Reproduction is authorised provided the source is acknowledged

Additional information on Cedefop’s research reports can be found on:http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/

For your information:

• the background report to the third report on vocational training research in Europe contains originalcontributions from researchers. They are regrouped in three volumes published separately in English only.A list of contents is on the next page.

• A synthesis report based on these contributions and with additional research findings is being published inEnglish, French and German.

Bibliographical reference of the English version:Descy, P.; Tessaring, M. Evaluation and impact of education and training: the value of learning. Thirdreport on vocational training research in Europe: synthesis report. Luxembourg: Office for OfficialPublications of the European Communities (Cedefop Reference series)

• In addition, an executive summary in all EU languages will be available.

The background and synthesis reports will be available from national EU sales offices or from Cedefop.

For further information contact:

Cedefop, PO Box 22427, GR-55102 ThessalonikiTel.: (30)2310 490 111Fax: (30)2310 490 102E-mail: [email protected]: www.cedefop.eu.intInteractive website: www.trainingvillage.gr

http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/

www.cedefop.eu.int

www.trainingvillage.gr

Contributions to the background report of the third research report

Impact of education and training

Preface

The impact of human capital on economic growth: areviewRob A. Wilson, Geoff Briscoe

Empirical analysis of human capital development andeconomic growth in European regionsHiro Izushi, Robert Huggins

Non-material benefits of education, training and skillsat a macro levelAndy Green, John Preston, Lars-Erik Malmberg

Macroeconometric evaluation of active labour-marketpolicy – a case study for GermanyReinhard Hujer, Marco Caliendo, Christopher Zeiss

Active policies and measures: impact on integrationand reintegration in the labour market and social lifeKenneth Walsh and David J. Parsons

The impact of human capital and human capitalinvestments on company performance Evidence fromliterature and European survey resultsBo Hansson, Ulf Johanson, Karl-Heinz Leitner

The benefits of education, training and skills from anindividual life-course perspective with a particularfocus on life-course and biographical researchMaren Heise, Wolfgang Meyer

The foundations of evaluation andimpact research

Preface

Philosophies and types of evaluation researchElliot Stern

Developing standards to evaluate vocational educationand training programmesWolfgang Beywl; Sandra Speer

Methods and limitations of evaluation and impactresearchReinhard Hujer, Marco Caliendo, Dubravko Radic

From project to policy evaluation in vocationaleducation and training – possible concepts and tools.Evidence from countries in transition.Evelyn Viertel, Søren P. Nielsen, David L. Parkes,Søren Poulsen

Look, listen and learn: an international evaluation ofadult learningBeatriz Pont and Patrick Werquin

Measurement and evaluation of competenceGerald A. Straka

An overarching conceptual framework for assessingkey competences. Lessons from an interdisciplinaryand policy-oriented approachDominique Simone Rychen

Evaluation of systems andprogrammes

Preface

Evaluating the impact of reforms of vocationaleducation and training: examples of practiceMike Coles

Evaluating systems’ reform in vocational educationand training. Learning from Danish and Dutch casesLoek Nieuwenhuis, Hanne Shapiro

Evaluation of EU and international programmes andinitiatives promoting mobility – selected case studiesWolfgang Hellwig, Uwe Lauterbach,Hermann-Günter Hesse, Sabine Fabriz

Consultancy for free? Evaluation practice in theEuropean Union and central and eastern EuropeFindings from selected EU programmesBernd Baumgartl, Olga Strietska-Ilina,Gerhard Schaumberger

Quasi-market reforms in employment and trainingservices: first experiences and evaluation resultsLudo Struyven, Geert Steurs

Evaluation activities in the European CommissionJosep Molsosa

Evaluating the impact of reforms of vocational education and training:

examples of practiceMike Coles

AbstractIn this study 19 evaluative reports of national reforms to vocational education and training (VET) across12 European Union countries have been selected and analysed. Criteria have been developed which clar-ifies the analytical framework. The study of these reports of reforms provides insights into the ways eval-uations have been conducted, the role of evaluations in the reform process and the kinds of impacts thereforms have made.A range of large-scale national reforms to VET is examined covering initial vocational training (IVT),continuing vocational training (CVT), VET in higher education, VET for employability, qualification devel-opment and financing. The trend away from central control of VET planning and financing is evident asis a deepening appreciation of the potential of VET for broadening the school curriculum and forimproving the life chances of individuals.Countries differ greatly in the ways they use evaluation. In some countries evaluation appears not to beused as a development tool and in others it is developed simultaneously with the reform process.Published impact studies are rare; the most common form of evaluation programme is a mid-term and/orsummative stakeholder survey. However, there are some good examples of published impact analysisand the methodological rigour in these can be used to develop confidence in tackling impact studies.The study offers evidence-based guidance on making evaluation programmes more scientific and moreimpact oriented.

1. Introduction 12

1.1. The nature of VET 12

1.2. Some definitions and limitations 13

2. Selection of reports for analysis 14

3. Criteria for judging reforms 18

4. Evaluating VET reform 19

4.1. VET reform 19

4.2. Evaluation 21

4.3. Context of reform 24

4.4. Goals 24

4.5. Planning of reforms 25

4.6. Management of reforms 25

4.7. Drivers 26

4.8. Impact 26

5. Financing VET: reforms and evaluation 27

5.1. Financing reforms 27

5.2. Evaluation of financing 28

6. Levels of evaluation 29

6.1. Establishing a baseline 29

6.2. Impact and timing of measurement 30

6.3. The individual 30

6.4. The organisation 34

6.5. The aggregate level of the economy and society 37

7. Towards better evaluation of reforms 40

8. Summary and conclusions 42

List of abbreviations 44

Annex: common evaluation criteria 45

References 49

Table of contents

TablesTable 1: Reports used as the evidence base 15

Table 2: Classification of reforms to learning and learning environments 21

Table 3: Broad review of VET reforms and evaluation methods in this study 23

Table 4: Measuring the impact of new funding arrangements 28

Table 5: Proportion of Fachhochschule leavers who feel disadvantaged by their counterparts

in ‘normal universities’ with regard to selected aspects 31

Table 6: How much an apprenticeship course is helping to… 31

Table 7: Increases in salary from the first employment in 1992 to employment in 1997 31

Table 8: The measures of success for two active youth labour market programmes 32

Table 9: Why a modern apprenticeship is attractive 32

Table 10: Motives for abandoning apprenticeship 33

Table 11: Effect of training on long-term unemployed 33

Table 12: Changes in workers as a result of VET 35

Table 13: Employers’ views on the reform 35

Table 14: Fachhochshule graduates compared to graduates from traditional universities 36

Table 15: Are you interested in employing more graduates from Fachhochschule in the future? 37

Table 16: Five years after gaining a qualification how many young people in France are registered as

unemployed or inactive? 38

Table 17: Destinations on completion of Portuguese IVT programme 38

Table 18: Labour-market situation immediately following VTOS by certification achievements 38

List of tables

The main objective of this study is to provide acritical review of research dealing with evaluationof education and training system reforms thatinfluence the organisation and outcomes ofeducation and training at macro level. To developthis review a selection of national reforms tovocational education and training (VET) has beenmade and reports on these reforms analysedagainst criteria to provide insights into the waysevaluations have been conducted, the role ofevaluations in programmes of national reformsand the impacts these reforms have made.

The task of looking at national reforms andfinding evaluative reports on them has been acomplex task. The specific nature of this studyhas meant that many reports fail to meet theexacting specification required and consequentlyresearching sources of potentially useful informa-tion has been time consuming. In the end19 reports have been identified as relevant to thestudy. Ten are what might be termed ‘core’reports as they are especially rich in data on areform, its evaluation process and the evaluationof the effects of the reform. Two of the 19 reports(from Denmark and the Netherlands) are specificcase studies in Part IV of the third researchreport, but they are not considered in great detailin this study.

1.1. The nature of VET

VET is a complex subject of research because itis so wide ranging, covering multiple forms ofactivity from parts of mainstream initial educationto short-term, highly focused job specific skilldevelopment. There are also multiple layers ofactivity on where learning takes place, whoorganises it, who pays for it and who certificatesit. The value of VET is located with individuals,communities, institutions, trade unions, busi-nesses and governments at regional, national andinternational levels. Development and reform ofVET is therefore subject to a multitude of influ-ences, controls and incentives. Setting up alarge-scale reform to a country’s VET system

demands high levels of consultation, commitmentfrom national agencies and detailed planning.There are clearly many ways to manage anational reform ranging from a carefullyconceived, multistage plan to one that sets out toachieve aims through multiple activities. In thelatter type, actions are often dependent onresponses to recent changes in the system andso method corresponds, for example, to settingin place an integrated transport system where thedevelopment of the train network depends on thefunctioning of other means of transport and thechoices people make as different ways of travel-ling become available.

Evaluation processes also take on differentroles in these different styles of reform. Somerepresent a feedback system to central plannerson progress and are mechanical in nature usingtargets or performance indicators as account-ability tools. We could label these evaluations as‘summative’. Other evaluation processes areembedded in the reform process itself and offerfeedback to a range of interested parties. Indica-tors and performance measures are less promi-nent here and are used to improve organisationallearning. The evaluation is seen as a way ofcommunicating thoughts and actions across arange of actors. This style could be labelled‘formative’. The complex nature of VET and itssocial embeddedness makes a single prestruc-tured summative evaluation of limited value. Theevaluation process might be considered to beabout learning about VET change itself and aformative evaluation process is, perhaps, essen-tial. In fact, drawing on the experience ofresearching the evidence for this study evalua-tions of national reforms are rarely singularlysummative or singularly formative but fall within aspectrum of types ranging from one to the other.

All evaluation studies respond to their objec-tives and report on these. However some gofurther and report on the implications of findingsfor the system being evaluated. This extensioncan, in some reports, lead to substantial state-ments that have implications for reformmanagers. This ‘added value’ to evaluation

1. Introduction

Evaluating the impact of reforms of vocational education and training: examples of practice 13

reports is generally associated with evaluations ofthe embedded or formative kind.

In this study a wide range of types of VETreform and evaluation processes have beenencountered. The spectrum of types of reformand evaluation processes makes the analysispresented a useful illuminative tool for VETresearchers and offers ideas for further, deeperanalysis.

1.2. Some definitions andlimitations

It is useful to state how certain key concepts areused in this report.

VET is defined as all organised forms of initialand continuing or further vocational educationand training, independent of location, age ofparticipants and their level of qualification. VET isto prepare for a certain occupation or employ-ment and could also include an element ofgeneral education, for example the developmentof basic skills.

Reforms of VET are limited in this study tothose that are national or system-wide. Reformsneed not cover the whole VET system in acountry to be included, indeed no examples ofthis root and branch scale of reform were found,however reforms were required to cover asubstantial component of the VET system.Regional reforms have also been included whenthe results of the reform are likely to impact on

national provision. A distinct part of the focus onVET reform is to consider the financing measures.Some VET reforms include reform of financingarrangements as well as the content and deliveryof VET. Some evaluation reports are not associ-ated with any single reform. They are an evalua-tion of VET provision in a country from theperspective of a particular set of users. Forexample, evaluation of effects of qualifications ona full cohort of school leavers in 1992 has beencarried out in France. Leaving out such studieswould have reduced the quality of the analysis oftypes of evaluation.

Evaluation is research that involves use ofsocial and economic methodologies to assessand improve the planning, implementation, moni-toring, effectiveness and efficiency of measuresor programmes. In this study we are concernedwith evaluation of impact. We are looking forsome idea of how changes to individuals, institu-tions, communities and systems are dependenton interventions embedded in the reformprocess. Stern (2003) refers to these evaluationsas those that bring ‘accountability for policy-makers’ and distinguishes them from evaluationswhich aim to develop a reform programme,create knowledge about the VET system or aimfor social improvement and change. The focus ofthese accountability evaluations is impact,outcomes and achievement of targets and valuefor money.

The next chapter outlines the research, givingan indication of scope, the work undertaken andthe timescale involved.

Searches for reports were protracted and oftenled to an inadequate description of the evaluationprocess, focusing instead on detailed descrip-tions of the planning and introduction of reforms.For this research there are three points of interest– the reform itself, the evaluation of the reform(process and outcomes) and how the reform andthe evaluation relate to each other. The first pointis always present, the second sometimes presentand the latter is invariably missing. The intersec-tion of the reform and the evaluation is importantfor two main reasons:(a) it gives an insight into the way in which the

reform and the evaluation methodology arematched to each other by design. Forexample, how well the objectives for evalua-tion correspond with the goals of the reform;

(b) it tells us how impact is being monitored andthen weighed against other contextualfactors.

Compiling a group of reports representative ofthe range available proved difficult; it was notpossible to be as selective as one would havewished as the range of reports available waslimited. Selection of reports was initially based on:(a) the significance of the reform in the country

concerned;(b) the breadth and depth of the evaluation; (c) the extent of the focus on evaluation of impact.

In practice, the first goal was to find at leastone report from each of the 15 EU nations. It wasdesirable to have good coverage of cultures andVET systems across countries and in some coun-tries there was more than one report eligible forinclusion.

Exchanges with researchers have suggestedthat full reports of evaluations of VET reforms arenot published as frequently as one might expect.Many reasons are cited by experts including awish for confidentiality by funding agencies, lackof political will to publish challenging evaluationsand the fairly common ‘light touch’ style of evalu-ation that yield low levels of analytical informa-tion. In the initial stages of the work over

200 references to potentially useful sources ofevaluative information were identified. Thesewere scrutinised against the broad requirementsof this study. Essentially reports were requiredthat:(a) had the subject of a major reform to VET

within a country;(b) focused on a reform that was not EU funded

(covered in other contributions to the thirdresearch report);

(c) was written and published authoritatively (byhigh status people/organisations);

(d) was evaluative in nature rather than descrip-tive;

(e) focused on the impact of reforms;(f) included reference to ways of financing VET

provision.There were many promising references to

reports that appeared to meet these requirementsbut as the reports were scrutinised in detail manyfell away. This a common pattern in metastudies;in the major ones into specific aspects of educa-tion in the UK (1) it is not uncommon for2 000 references to be reduced to 20 or so usefulreports. The filtering process in this study showedthat there are many evaluation reports but:(a) most are about the way a project worked

rather than the impact a reform has had; (b) many are linked to EU programmes and not

reforms initiated and funded by countries tomeet national goals;

(c) many are more concerned with regional orfragments of the VET system thansystem-wide reforms;

(d) many are linked to broad initial educationrather than vocational workforce develop-ment;

(e) few have specific references to financing.This paucity of outcome-based reporting is

highlighted in evaluation literature, Griffith (2001)states in his review of school-to-work transitionsmost publications quote such variables asnumbers of participants and numbers of workplacements and few chart progress through to

2. Selection of reports for analysis

(1) EPPI – Evidence based policy and practice – University of London, Institute of Education


Country Reform Report reference Justification for inclusion

outcome measures. Baker and Taylor (1998) reportthat the shortage of empirical research moregenerally is the issue; they found only 12 suchstudies published between 1983 and 1996 andthese were small scale and limited in various ways.

The first sift of reports that related to all the keywords (evaluation, impact, national reform, voca-tional, education, training) reduced the 200+ poten-tially useful reports to 33. These covered all EUcountries and most aspects of VET and financing.

In some countries it is difficult to isolate aspecific thread of evaluation on programmes. It ispossible to take two fairly extreme views of this.Either there is no discernible, independent evalu-ation or, development and evaluation are so inter-twined it is not possible to distinguish them. Inthe latter case development or reform takes placein relatively small stages. It would be useful toresearch reports of evolution of a system andtrace the drivers of change. In so doing theprocess of evaluating current provision andmaking incremental changes as a result shouldbecome clear.

Some interesting reports were identified thatwere not conventional VET research publicationsbut nevertheless had potential to contribute tothis study. For example a Cedefop review paper(Richini, 2002) on steps to enhance lifelonglearning in Italy proved useful for information onways of better coordinating VET, two French

documents (Klein, 2002; Thireau, 2001) detailedhow a single cohort of school leavers fared in thelabour market and in further education andtraining. Another report of this type comparedtwo UK methods of financing training (Felsteadand Unwin, 1999). Two Swedish active labourmarket policies are systematically compared inanother report (Larsson, 2000). These reportshave been used alongside others in forming theevidence base for this study.

In the early stages of this work potential reformswere to be classified as core, suitable for indepthanalysis, and associate, useful for supplementarymaterial for illuminating, supporting, challengingand cross referencing findings about theories,approaches, methods and results in the coremetaanalysis. This categorisation proved unten-able in the light of difficulties in finding reports and,more importantly, in trying to cover types of reformand evaluation within six to eight reports ofreforms. A decision was made to reduce the set of33 reports to a single set of reports on 19 reformsand to treat them in the same way. Table 1 liststhese and includes a justification for including thereport in the study.

Having isolated the key reforms and their asso-ciated evaluative reports, each report was anal-ysed against the criteria described earlier and isattached as an annex to this report. These criteriaare discussed in the next chapter.

Table 1: Reports used as the evidence base

Finland Valtioneuvoston selon-teko eduskunalleuuisen koululakienvaikutuksista ja laessaasetettujen tavoit-teiden toteutumisesta

Finnish Ministry ofEducation, 2000

This is a major reform to secondary education and adult education. There are changes to qualifications and new models of financing. The official evaluation report shows how evaluationis integrated with the reform process.

Austria Die ersten Fach-hochschulAbsol-ventInnen am Arbeitsmarkt

Wimmer, 2000 The development of Fachhoch schools provides a tertiary education focus. Students and enterprises were involved in the evaluationprocess.

Denmark Reform 2000 Shapiro et al., 2001Kaasbøl et al., 2001Nielsen, 1997Nieuwenhuis andShapiro, 2003

This reform is included as it is a case study in thethird research report. The reform is nationwide and involved a pilot phase and multiple evaluations. It also has a theoretical basis.

Evaluation of systems and programmes16

France Les bénéficiaires ducontrat de qualificationentrés à l’automne1994: les enseigne-ments de 4 vagues de panel

Klein, 2002 This is an example of a longitudinal study withfocus on the effects of a vocational qualification. A wide range of impact measures is covered.

Germany Personalqualifizierungin den nuen Bundes-landern

Neubert and Steinborn, 1999

It is difficult to identify a specific reform that hasbeen evaluated in German VET. Recent initiativesmay provide such reports. This report containsinformation about the impact of training andcontains a financial dimension.

Greece Evaluation of thelabour-market policiesand assessment of theinfluence of Europeanemployment strategyin Greece during theperiod 1997 to 2001.

VFA, 2002 While there is no specific VET reform in this report itis possible to study the effects of VET scientifically:case/control methodology was used on a largesample. The case group and the control group werestrikingly well matched demographically.

Ireland Leaving certificate(applied)

Irish Ministry ofEducation, 2002

This is a large-scale qualification developmentwith a strong VET dimension. A strong decentralised evaluation programme accompanies it.

Ireland Vocational trainingopportunities scheme(VTOS)

WRC Social andeconomic consul-tants Ltd., 1994

This is a detailed evaluation of a nationwideprogramme of training for the long-term unemployed that contains good impact analysis.The evaluation is integrated with the programmeof reform.

Italy Il nuovo apprendistatoe la formazione

Angeli, 1999ISFOL, 2000

This reform is system-wide and is representativeof renewed interest in apprenticeship acrossseveral countries. The evaluation focuses on theinfrastructure of the reform. Other papers outliningsystem-wide reforms provide perspective on theapprenticeship and some impact analysis.

Netherlands Adult education andvocational educationact (WEB)

Borghans and Heijke,2002;De Bruijn et al., 2001Nieuwenhuis andShapiro, 2003

This reform is included as it is a case study in thethird research report. The reform is nationwide andinvolved decentralisation of management changesto VET programmes, funding changes. The evaluation has a formative function. Several as yetunpublished papers have been made available.

Portugal Sistema das escolasprofissionais

Montavavao e Silvaet al., 1996

This is a reform of IVT. The diverse range of vocational training is to come under a morecoherent approach but includes the introduction of specialist schools The new schools areautonomous although there is some regulation of them. The evaluation is phased and commissioned from an international source.


Portugal Despacho Conjunto –do Ministério daEducação e para aQualificação e oEmprego – No 123/97de 16.6.1997

Ministério daEducação e para aQualificação e oEmprego, 1997

An annual ministry-led evaluation programme thatinvolves a ‘bottom up’ approach is included.Finance features in this evaluation.

Spain Memoria Banance – IAcuerdo Nacional deFormacion Continua1993-1996

Forcem, 1999 Developing a culture of professional developmentis the goal and evaluation of impact of the firstthree years of promotional activity is provided. The reform is wide ranging and involves manystakeholders.

Sweden Reformeringen avgymnasieskolan

National Agency forEducation, 2000Ekstrom, 2002

A system wide reform of upper secondary education (qualified vocational education, QVE)with a strong focus on IVT. A full-scale evaluationof this reform has been written and is currentlywith the sponsoring organisations. It was notavailable at the time of writing.

Sweden Swedish youth labourmarket programmes

Larsson, 2000 Two programmes involving training are compared.The evaluation has an econometric design todetermine the effects of these programmes onearnings and reemployment probability.

UK Work related furthereducationprogrammes

Metcalf and Dawson,1995

A study of reforming planning and financing of IVTand CVT in colleges. The evaluation uses an interesting comparative method which is quasiexperimental. Impact measures are included.

UK The Union learningfund

Cutter et al., 2000Antill et al., 2001

This is an evaluation of an initiative that includessocial partner involvement, VET capacity buildingand financing. It deals with basic skills training inparticular. The evaluation is a blend of survey andcase study and runs in three annual phasesthereby offering progressive reporting. Manyoutput measures discussed.

UK Technical and vocational educationinitiative

Department ofEmployment, 1994

This is a large-scale curriculum initiative to reorientate learning to more active approachesand to increase the profile of vocational education.It has been evaluated extensively.

UK Modern apprenticeships

Economic ResearchServices Ltd, 1998;Coleman andWilliams, 1998

A new apprenticeship scheme that has been evaluated for impact and over a period of threeyears. Learners and employers are surveyed indetail.

UK Funding systems andtheir impact on skills

Felstead and Unwin,1999

This paper has no VET reform at its heart butpaves the way for changing the way VET isfunded. Two funding systems for IVT arecompared and evaluated.

One of the challenges in developing a meta-studyof evaluation reports is to set analytical criteria tomake the study systematic. A criterion-basedreview makes it possible for readers to judge thevalue of the review for their purposes. If neces-sary, readers can extend the review by refining theanalytical criteria and altering judgements ofreforms accordingly. Developing criteria for thisstudy began in preparing the proposal to carry outthis work. The proposal made the point that it issimplistic to use criteria based on the conventionalanatomy of reforms (conception, consultation,initiation, pilot phase, implementation, review,refine, etc.). Instead it was proposed to look moreclosely at the evaluation procedure as well as thereform and to focus strongly on impact. For thereform this meant looking at the aims and objec-tives of it closely and, for the evaluation, lookingfor baseline assessment, comparative approachesand impact or performance criteria. It also meantlooking for unexpected outcomes andsystem-wide implications. A set of criteria wasdeveloped using a range of literature (e.g., Barrett,1998; European Commission, 1999; Plewis, 2001;World Bank, 2002) and expert sources within thegroup of researchers working with Cedefop on thethird research report and within the Qualificationsand Curriculum Authority (QCA).

The second stage was to seek the views ofother experts of the draft criteria. Amendmentswere made accordingly and a working documentproduced. The structure of this set of criteria wasbased on the chronology of an evaluationprocess – before the reform was implemented,

during implementation and post reform. To thesethree a set of general criteria were added on theevaluation process itself. Later, it was consideredadvantageous to introduce a further set of organ-ising headings to make it clear that a chronolog-ical approach was only one way of organising thecriteria. This additional set of headings is asfollows:(a) context: what external influences are likely to

influence?(b) goals: how are the aims, objectives, mile-

stones described?(c) planning: what is discussed in anticipation of

what might happen?(d) management: what is done to make the

programme function effectively?(e) drivers: who is shaping the evaluation?(f) impact: what is the effect of the evaluation?

When these headings are applied and the40 criteria within the chronological headings areallocated the coverage is as follows: context (6),goals (5), planning (4), management (13), drivers(6) and impact (6). This distribution reflects thefocus of this study with the emphasis on driversand impact being served by a higher proportionof criteria than would normally be expected. Theapplication of these criteria to the reports ofreforms is discussed later in this report.

In the next chapter the emerging observations,generalisations and issues are discussed. Theheadings used to group the common evaluationcriteria (see annex) are used to organise thediscussion.

3. Criteria for judging reforms

The selected reforms and their evaluative reportsrepresent a large body of evidence on the recenthistory of change in VET in Europe. In thischapter some review of VET reform is presentedfollowed by a synopsis of findings on changes tothe financing of VET. A general discussion onevaluating the effects of VET reform concludesthis chapter.

4.1. VET reform

The reforms studied exemplify a wide range ofVET activity. They cover vocational preparation inschools, full-scale initial vocational training (IVT)programmes, continuing vocational training (CVT)in work, CVT for those who are seeking employ-ment, retraining activities, short programmes,long programmes and qualifications from theInternational Standard Classification of Education(ISCED) levels 1 to 5. The reforms aim to changethe management of VET, financing of training, thelearning environment for VET, the teachingapproach, the content of courses and, of course,to boost outcomes of training. The image of VETgained from studying the reports is one oftremendous diversity, with multiple purposesclosely related to the dynamics in families,communities and institutions. The breadth of VETmeans many people are agents in supporting,designing, delivering or assessing programmesand these people are generally well consulted inreform development. They are also generally wellconsulted in evaluation programmes.

In Cedefop’s second research report (Descyand Tessaring, 2001) it was established that therewas a general move away from State-led VETand market-led VET to new forms of collaborativesteering arrangements, possibly through profes-sional associations and networks. There isconsiderable evidence in the sample of reformsused in this study that such a trend is continuing:examples include designing new semi-autonomous institutions for IVT in Portugal andpolicies to decentralise provision of State-fundedVET in the Netherlands and the Nordic countries.

The institutions that distribute State funds tomeet the needs of individuals and employersseem to be the key brokers in the new profes-sional associations and networks.

All reports studied described the aims ofreforms; these aims reflect the wide range of VETapplications discussed above. For this researchstudy, it is important to look at the relationshipbetween the aims of the reform and the objec-tives of the evaluation programme associatedwith it. This is key for impact analysis and isdiscussed in detail later in this chapter.

Almost all the reports studied examinedchanges to the way VET is organised. Examplesinclude new types of IVT programmes (chieflyapprenticeship schemes and schemes for makinggeneral education more vocational), CVTprogrammes (chiefly State-funded schemes andother ways of organising funding). Additionallyattention is being paid to reorganisation toimprove social welfare and the autonomy of theindividual. Innovation in VET provision is evidentand takes many forms, for example in the firstyear of developing the Union learning fund inthe UK the project had innovative ways of deliv-ering VET at its core.

The reforms chosen for analysis were primarilyselected as examples of impact studies and thiswill have skewed the representative nature of thereforms as depicting types of VET activity.However the reforms do reflect the span ofnational programmes and it is notable the focuson reforming IVT is strongly represented in thesereports. Usually this involves developing IVTalongside general education and allowing thosewith a disposition for more applied study (experi-ence of work, content of courses, teaching,learning and assessment styles) to opt into voca-tional programmes. By doing so it is hoped thesepeople will be more motivated to learn and willdevelop useful skills sought on the labour market.Consequently, they will be less likely to drop outof school and lose the advantages of a goodinitial general education. Some believe theseprogrammes result in a diminution of the volumeof general education taught to the participating

4. Evaluating VET reform


cohort. There is also evidence of some perme-ation of teaching styles and resources from IVTprovision to general provision when theseprogrammes are taught in the same institution.

Access to higher education also features in theselected reforms. Returning to the changes inupper secondary IVT programmes, a commonobjective is that students following these newprogrammes should have access to highereducation, so more time is needed to develop thevocational provision while retaining a core ofgeneral education. Several reforms include addi-tional time for study in upper secondary educa-tion – typically another year. This additional timeis needed for learning from workplace experienceand for developing and assessing projects orassignments. In some reforms studied legalchanges provide a right for individuals graduatingfrom such programmes to have a place in ahigher education institution. One reform that aimsto make higher education provision more desir-able and accessible to students who retain thisinterest in vocational programmes.

Some reforms aimed to create new learningcentres, others aimed to change participation andpedagogy in existing centres. Sometimes thereforms to content, pedagogy and VET organisa-tion are sufficiently extensive to create a newlearning environment based in an existing institu-tion. Thus it is sometimes difficult to isolatereforms for creating new environments fromthose simply changing content or pedagogy. Inthis study new learning environments werecreated by:(a) the Austrian reforms where students are

educated in new Fachhochshule whereexpertise could be concentrated on voca-tional courses;

(b) introducing apprenticeship programmes inItaly and the UK where employers andtraining providers established new ways ofblending general learning with work-basedlearning;

(c) the reforms in Portugal where new semi-inde-pendent regionally controlled schools for IVTwere established alongside existing provision;

(d) trade unions in the UK where they negotiated

new provision with a different rationale andmanagement.

Many reforms aimed to create a new purposefor learning programmes by developing existinginstitutions. This could be by altering manage-ment structure, financing or course provision.Examples of these reforms include:(a) regionalisation of management of institutions

delivering upper secondary education inDenmark, the Netherlands, Finland andSweden;

(b) improvement in CVT in former East Germanyto meet better the needs of workers andemployers;

(c) increasing emphasis on work experience inDenmark, Ireland, the Netherlands, Finlandand Sweden.

Several reforms aimed to change the contentand structure of programmes within existing insti-tutions that were not required to change signifi-cantly. Examples include:(a) experimenting with new content in the Italian

apprenticeship programme;(b) the UK reform of the general curriculum in

schools to create a more vocational focus;(c) inclusion measures in the Irish vocational

training opportunities scheme (VTOS);(d) introducing new forms of assessment in

Finland and Sweden;(e) developing new schools leaving qualifications

in Ireland.We can classify these reforms in learning and

learning environments as shown in Table 2.Many reforms allocate time to general educa-

tion and sociocultural studies as well as develop-ment of specific technical skills – the balance issometimes contested. For example, in thePortuguese reform employers and parents seeteaching and learning about general socialissues reducing the time that could be usefullyspent on work-based assignments and skillslearning. The designers of programmescontaining an element of general educationclearly have to strike a balance between theexpectation of users and the strong evidencesupporting inclusion of elements of generaleducation in IVT provision.


4.2. Evaluation

Evaluation reports generally describe the subjectof the evaluation, outline objectives, brieflydescribe the method and then report findings andsuggest implications. It is usually not possible tofind details of the evaluation process, the prac-tical constraints, the compromises, the adjust-ments, the pressures for results, the considera-tion of different ways to interpret results and howresults and conclusions are protected from misin-

terpretation. Reading reports will not reveal thisinformation. Academic research into evaluationmethods usually contains discussion of thebroader issues but this discussion will normallybe out of the context of a specific reform. It ispossible that only through discussion with plan-ners, evaluators and programme managers willwe learn about the evaluation process itself.Therefore, in reports such as this there is a gapbetween full knowledge of the reform, the inter-ventions, the evaluations and what is reported inliterature.

Table 2: Classification of reforms to learning and learning environments

Change Aims of change Examples

Create additional new institutions

Concentrate expertise,diversify VET provision,attract students, meet skills need.

• Austria (Wimmer, 2000)• Portugal (Montavavao e Silva et al., 1996; Ministério

da Educação e para a Qualificação e o Emprego,1997)

Blend existing learningenvironments into a newkind of provision

Produce more effectiveprogrammes, shift controlof training to employers,tailor courses to meet skills needs.

• Ireland (WRC Social and economic consultants Ltd.,1994)

• UK (Department of Employment, 1994; Metcalf andDawson, 1995), Spain (Forcem, 1999)

• Portugal (Montavavao e Silva et al., 1996; Ministério daEducação e para a Qualificação e o Emprego, 1997)

Engage social partners in planning and management of provision

Innovate in programmedesign (focus on learner),coordinate range ofexisting provision.

• UK (Cutter et al., 2000; Antill et al., 2001)• Spain (Forcem, 1999)• Portugal (Montavavao e Silva et al., 1996; Ministério da

Educação e para a Qualificação e o Emprego, 1997)

Develop the schoolcurriculum and assessment and qualifications

To make curriculum relevant, assessmentauthentic, recogniseinformal learning, motivatelearners, bridge initialeducation and work situations.

• Sweden (National Agency for Education, 2000;Ekstrom, 2002

• Finland (Finnish Ministry of Education, 2000)• the Netherlands (Borghans and Heijke, 2002;

De Bruijn et al., 2001)• Denmark (Shapiro et al., 2001; Kaasbøl et al, 2001;

Nielsen, 1997)• Ireland (Irish Ministry of Education, 2002)

Improvements in thecontent and teaching ofspecific programmes in an existing type of institution

Work force development,drive up productivitythrough deployment of new skills, improveprogression opportunitiesto work and in work.

• Denmark (Shapiro et al., 2001; Kaasbøl et al., 2001;Nielsen, 1997)

• UK (Department of Employment, 1994)

Develop teachingapproaches to motivateand retain learners

Reduce exclusion fromwork and costs of benefits.

• Ireland (WRC Social and economic consultants Ltd.,1994)


Evidence from evaluations of VET can beuseful for:(a) inform government decisions (including allo-

cating funds);(b) improving employer decisions on training;(c) informing individuals of their options;(d) designing new programmes;(e) refining programme design;(f) improving programme targeting;(g) identifying ineffective programmes;(h) encouraging public debate about VET.

Stern (2003) summarises the purposes of VETevaluation under four categories:(a) accountability for policy-makers;(b) development for programme improvement;(c) knowledge production and explanation;(d) social improvement and change.

Impact evaluation forms part of only the first ofStern’s categories where he looks for ‘impacts,outcomes, achievement of targets and value formoney’.

When a VET programme is created or changedit is important to know if the new programmeworks. Programme evaluations therefore require(usually quantitative) measures of impact – tojudge if an intervention has worked or not – and(usually qualitative) information to discover thereasons why the intervention has succeeded orfailed. Both these measures are important but inthis study the first measure is of prime interest.Interventions based on reform of VET can rarely, ifever, be isolated from other contextual influencesand this makes measuring impact highlycomplex. To help focus hard on impact there aretwo questions that might be asked:(a) what are the changes in VET (including

changes to those involved in VET and thecontexts in which VET learning is applied)that have occurred during the period of inter-vention and after the intervention?

(b) is there evidence to link these changesdirectly or indirectly to the VET intervention?

Restricting this study to reports that provideanswers to both these questions eliminatereports of process evaluations that analyse how aprogramme operates and focus on impact evalu-ations and causality in particular. The commonevaluation criteria (see annex) help with this sharpfocus on the two questions.

Literature shows the way evaluation is consid-ered varies greatly. In fact the range of practice is

so wide that it is difficult to understand how asingle term like ‘evaluation’ can cover allinstances. There is a need to distinguish researchaccording to purpose, i.e. accountability, effi-ciency, functioning (impact) and valorisation or asStern (2003) puts it: accountability, programmeimprovement, knowledge production and socialimprovement.

One striking difference in evaluation programmesis the distinction between those that aim to answerthe question ‘has it worked?’ rather than answer aseries of questions based on ‘is it working?’ In otherwords, end-loaded summative evaluation relative tointegrated formative evaluation. Evaluation helpedthe reform process along in just over half of thecases sampled in this study. On the potential of theevaluation to yield information for the reformprocess, the reports used are once again evenlysplit between those that have been designed toinject a sense of progress or impact at any stageduring the reform implementation and those thathave not.

Linked to the question of timely intervention isthe question of deciding when it is reasonable tomake a measurement of effect after the reformhas become established. It is rare to find an eval-uation that has measured changes to participantsin VET programmes a year (or longer) aftercompleting a VET programme. One of the bestexamples is the Swedish reform of IVT in uppersecondary education (Ekstrom, 2002). In the threeacademic years between 1988 and 1991 thecurrent Swedish upper secondary vocationalpathway (three years duration) was piloted withstudents occupying 11 200 study places. Thecareer development of these students wastracked by measuring the time spent in uppersecondary education, level of university enrolmentand the rate of educational or employment inac-tivity. However, a large proportion of Swedishyouth were not eligible for this pilot programme asthey were following the usual two-year vocationalprogramme. Consequently, ‘natural experiment’was created where important influences otherthan education could be assumed to be the samefor both groups of students. Attempts were alsomade to balance various regional differences.

Results showed that the students in pilot areas,where both two and three-year programmes wereavailable, were more likely to spend three years inupper secondary education. Higher education

22


enrolments were higher for the pilot group withthree years of vocational upper secondary educa-tion. The effect of a third year of vocational educa-tion on the rate of inactivity of individuals was notsignificant. One possible explanation is theSwedish labour market declined during the pilotphase and students on two-year programmes hadtime to establish themselves in jobs before thedecline took place. The report of this evaluationhighlights the effects of changes in context over aperiod of years following an intervention.

The low number of evaluations such as theSwedish study that go significantly beyond inter-vention has major implications for this review of

impact studies since the subject or focal point ofany effect changes with time. This effect isdiscussed later. Many reports suggest that anevaluation of the long-term effects of reforms isuseful but only six reports attempt gathering dataafter intervention is complete and analyse andreport on them. However, several evaluations,possibly five more are in progress and willcontinue to measure progress and/or impact.

Looking across a spectrum of VET reforms andevaluation methods it is useful to look at the typeof reform, the kind of desired outcome or effect,the evaluation style and the research tools used.This overview is described in Table 3.

Table 3: Broad review of VET reforms and evaluation methods in this study

Type of VET reform Intermediate impact End point impact Research tools used Evaluation style

Development ofCVTD, EL, E, NL

Increased participationof adults

Higher skills levels,economic growth

Questionnaire survey of institutions and individuals, satisfactionsurvey, case studyexemplification

Continuous monitoring andreporting programme

Reform of postcompulsory IVTDK, NL, FIN, S, UK

Content development,pedagogic development, institutional development, regionalisation

Increased participation, good progression for graduates

Questionnaire surveys, consultations withstakeholders

Monitoring andformative evaluation

New qualificationdevelopmentIRL, I, UK

Increased participation Close skills gap,higher skills levels,economic growth

Questionnaire survey of institutions and individuals. Satisfaction survey

Monitoring andformative evaluation

Cohort reviewF

Identify successes and problems

Modify provision Periodic survey Independent monitoring

VET funding review UK

Changes to funding distribution system

More efficient andresponsive system

Literature review, interview,focus groups

Single event evaluation

VET institutionalchangeA, P, FIN, S, UK

Establish popular new learning institutions,make institutions moreresponsive

Capacity building,improve skills supply

Stakeholder consultation, casestudy exemplification,satisfaction survey

Formative evaluation

Inclusion measuresD, IRL, S

High take up ofprogrammes

Lower long-termunemployment

Stakeholder consultation including use ofcontrols.Post programmereview

Mix of evaluationstyles

The type of evaluation tool does not necessarilylink with the type of reform but in this study it isclear some types of VET reform, for example VETlinked to inclusion measures draws on a broaderrange of evaluation tools than many others. Almostall reforms use some form of user response ques-tionnaire, usually postal but sometimes telephonebased, to measure effects of programmes. Theseusually have three main parts:(a) personal and demographic background infor-

mation of the subject;(b) ratings by subjects of aspects of the VET

provision they are using or have used;(c) judgements by subjects about future prospects

for employment or learning progression.Extending these questionnaire-based

approaches so analysis can have a greaterimpact focus is discussed later in this report.

In some evaluations, there was an issue oftiming of feedback from the evaluation programmeto the developing reform. This is restricted to eval-uation programmes where evaluators are notinvolved in the programme itself. Typically, data willhave been gathered, analysed and then reportedbut in the meantime the reform will have moved onand the data appear to be, at least in part, redun-dant. A good example in this study is the evalua-tion of funding of work-related collegeprogrammes in the UK. The ways in which collegesinvolved in the reform produced business plans forfuture provision soon spread to other colleges notinvolved in the study. By the time the evaluatorsreported, the ideas permeating between collegeshad become a flood and the report was moredocumentation of progress than something thatmight shape dissemination of outcomes to othercolleges not involved in the study.

Having reviewed generally the nature of VETreforms and the ways they have been evaluatedthis report now begins using the common evalua-tion criteria to examine the evidence base in moredetail. First, a review of criteria is describedfollowed by deep consideration of measuring theimpact of VET.

4.3. Context of reform

Almost all reforms were set in the context ofbroad changes brought about by governments.Whether this link to national policy was a trigger

for the reform or whether in reporting on evalua-tion the authors attempted to locate, post hoc,the reform in a wider setting is usually not clear.However, most reforms are fundamental and it isdifficult to perceive them as developing indepen-dently of central government policy, for examplethe Nordic reforms of upper secondary education.

Many reports describe the forces promotingchange in VET. These include:(a) the political wish or mandate to reorganise

VET; (b) economic necessity; (c) relief of social issues; (d) the need to find more equitable ways of

funding VET;(e) the need to overcome bureaucratic barriers; (f) feedback from enterprises about skills needs

and shortages. Legal requirements for new structures to be

established and for new individual rights totraining undoubtedly carry forward VET reformbut it is difficult to establish whether legal require-ments are a driving force. There are numerousexamples of reforms driven by law (prime exam-ples come from Spain, Italy, the Netherlands,Finland and Sweden). Research on the drivingforces for VET reform is worthy of further explo-ration on a European basis. There are insufficientdata in literature on how different agencies driveVET reform in the reports used in this study and itmay be more fruitful to gather information ondrivers of reform independently and outsideimpact studies.

4.4. Goals

All reports included some information about goalsor aims of the reforms being evaluated. Theseaims were usually stated in broad language andnot related to impact on individuals or organisa-tions. Typically, aims referred to improvingeconomic performance by raising skills levels orimproving access to the labour market for disad-vantaged groups or developing a culture ofprofessional development. Separate identifiablegoals for the evaluation process were alsopresent in half the reports but these were oftenworded to evaluate the process of reform ratherthan the outcome of reforms. For example, toreport on the future direction of the reform, to



improve reform management or resource alloca-tion. About one third of studies were clearlydefined for measuring outcomes with stated aimsof impact on specific stakeholders.

One strong differentiator of reports is theextent to which the goals of reforms and evalua-tions referenced theory-based reasoning fortaking the proposed course of action. One thirdof reforms argued for a new form of VET by usingevidence from research, another third made nosuch case and for the remaining third it is difficultto determine where the arguments for new formsof VET originated. A theory-based reform typicallyhas theoretical proposals such as ‘knowledge issituated in specific contexts’ and include mecha-nisms for translating theoretical knowledge intooutcomes such as ‘use a range of contexts forlearning ideas’ and the outcome could be ‘greaterlikelihood of employment’ or ‘better performanceat work’. Griffith (2001) provides useful informa-tion on theoretical perspectives in evaluationstudies.

It was not possible to make judgements onhow aims changed during evaluation. One studymade it clear the direction of the reform hadchanged significantly during its implementationand the purpose of evaluation had also changed.As stated earlier only a few reports describedeffects of interventions some time after the inter-vention had taken place.

4.5. Planning of reforms

Establishing a baseline for analysis is a basicrequirement for impact studies. Most reportscontain information and justification of a post hocrationale for accepting a baseline measure. It wasnot possible to judge whether any projects hadestablished the baseline data to be used to judgeeffect before a reform got underway. There was apaucity of reports describing establishment of anex ante evaluation process.

There were indications in about half the reportsthat the evaluation process was designed to aiddevelopment of the reform and was to take placesimultaneously with the reform.

Reports usually set the reform in a wider polit-ical or economic context but it was not possiblefrom the reports alone to deduce there was asense of coordination and coherence within a set

of reforms in one country. However, many reformscentred on institutional change which dependedon changes in governance and funding of institu-tions – a coordinated approach was a prerequi-site in these cases.

Dissemination of findings from evaluationprocesses was rarely discussed. This mightsignal such processes were not defined at theoutset of the project.

4.6. Management of reforms

Almost all reports described the structure of thereform process but very few described in anydetail how the reform was managed. Those thatincluded descriptions of management structuresand responsibilities used them to describe thereform itself. For example, in the Portuguesestudy the setting up of new institutions requirednew regional management structures. These wereseen in the evaluation as the subject of thereform and not as a means of managing thereform. There was little detail on managementinfrastructure, steering groups and job specifica-tions for key workers. More information isprovided on accountable organisations orcommittees: a ministerial report on thePortuguese reform offers some insight. Annualevaluation (effectively a ministerial meta-study) isbased on a hierarchy of active committees. Atlocal level, a board of teachers evaluate perfor-mance as they see it. These reports areforwarded to a regional board that analyses theteachers’ reports and prepares a regional evalua-tion. This is then passed to a national council thatsummarises regional reports and presents anannual progress report to government.

Common evaluation criteria contain referenceto the commitment of donor organisations. In factfew reforms were independent of governmentand there was almost always an unstatedassumption that the reform would be supportedto completion, that is, until a new structure was inplace. There were some exceptions, for examplein the UK Union learning fund the provider wasthe UK government and over a three-year periodthe efficacy of the scheme was to be established.After the first year negative results would have ledto abandoning the scheme. In developing IVTschools in Portugal the first phase was to estab-

lish whether it was worth developing theprogramme further. It was never evident fromreading the reports that the donor influence wasaffecting measurement of effectiveness of areform programme. However, some researchersbelieve such influences exist and they can beapplied when determining the scope and style ofevaluation and in reporting findings (Martin,2001).

Risk management of reforms was invisible inevaluation reports. This is perhaps not surprisingsince the evaluation programme itself waspossibly part of a risk management strategy.Even with government-led reforms wheresuccess is highly desirable for political reasonsrisk considerations had a low profile. It is possiblein the quest to obtain reports concentrating onimpact evaluation, that good reports assessingprocess risks were ignored. Whatever the reasonand for management appraisal in general, thereare signs that programmes are finding it difficultto establish this aspect of evaluation. One reasonmay be difficulty in judging management in suchlarge reforms. In each country there may be fewexperts with experience of such reforms.

Only one quarter of reports described a pilotphase for the reform. However, many reforms(e.g. VTOS and leaving certificate applied inIreland, modern apprenticeship in UK, and thedevelopment of Swedish vocational programmes)were voluntary for users and therefore developedover time starting with a relatively small basebefore expanding to full scale. There was alsolittle evidence of evaluation of the means bywhich reforms were expanded from pilot to fullscale or disseminated more widely.

In many cases, how other reforms (e.g. tobenefit systems, guidance services, regionalmanagement) interfaced and supported thereform were described. Many of these descrip-

tions arise because measuring effect oftendemands an explanation that goes beyond theeffect of the reform itself. For example, the Greekevaluation of training programmes explains thatthose undergoing training have a better chance ofgaining long-term employment. These peopleoften use more sophisticated job applicationtechniques than those who do not undergotraining and they find they can access othermeans of government support.

4.7. Drivers

As part of the contextualisation of a reform thekey stakeholders in developing, carrying out andbenefiting from the reform were often identified.The roles of different groups in making a project asuccess are almost always described. However,interaction between these stakeholders and thechanges in their relative authority within VET wasnever discussed. This kind of discussion rarelyfinds its way into evaluative reports. Suchchanges in the power to influence are crucial forthe long-term sustainability of VET reforms andconsequently their impacts on stakeholders.Analysis of changes in the structure anddynamics of managing VET change is more likelyto be included in process evaluation reports sothese have been intentionally excluded from theset selected for this study.

4.8. Impact

Focus of the research on impact studies, theways in which effects are measured and theproblems faced are discussed in full in the nextchapter.


Keeping the financing of VET under review isimportant. Individuals, society and the economybenefit from an effective VET system and theState, enterprises and individuals have an interestin investing in VET to gain rewards later. Themultiple funding sources for VET and the range ofprogrammes requiring financial support make thisfield highly strategic – reallocation of resourcesfrom one programme to another is usually astrong indicator of values and priorities held byfunding bodies. It can simply mean oneprogramme is more effective or more importantthan another; it can also mean policy haschanged, priorities are now different or thedemand side of training is changing. Thus, toevaluate shifts in financing is a major contributionto evaluating VET itself. It is interesting to note,however, evaluation of the effects of funding ofVET reform was discussed in any detail in only 5of the 19 reports selected. One report was specif-ically about funding and two more were onjudging whether a reform had yielded value formoney. The paucity of empirical studies of theefficacy of financing procedures has beenreported (Kath, 1999). Of particular note is theshortage of systematic longitudinal studies. Thereare major methodological difficulties in estab-lishing such studies (Barrett, 1998). First measuresof impact have to be developed and agreed asvalid. Then the methods must be capable ofisolating these effects from other influences andthe costs (direct and indirect) and expenditure(often from multiple sources) have to be charac-terised clearly and unambiguously. Kath (1999)suggests that no cross-national comparison of theeffectiveness of VET financing systems is yetpossible because of these difficulties.

5.1. Financing reforms

There were some interesting examples of using afunding review to improve the targeting and effec-tiveness of VET (for example the work relatedfurther education review in the UK and thechange in financing arrangements for IVT in

Portugal and Sweden). In the UK example, thereform required a sample of colleges to changethe way they planned their future provision, byconsulting employers more thoroughly, estab-lishing partnerships in planning and evaluatingeffectiveness collaboratively. This way of deter-mining funding needs has now become part ofthe UK system of funding VET.

Another funding reform signalling reappraisalof the way central funds for VET are distributedwas providing incentives for additional (levered)funds from independent providers. The examplebelow shows how this expectation was measuredand reported as an effect of the reform.

Some reforms include major changes in the wayVET is funded; most of these concern regionalisa-tion of control and devolution of funding responsi-bilities. Others involve using private funding inpublic provision and yet others have introducednew ways for individuals to pay more or less fortheir own training. The upper secondary reform inFinland has a specific focus on financing. In thenew Finnish system, collectives are now required todistribute funds for training – these funds werepreviously held by institutions. Unfortunately, evalu-ating the effectiveness of this change in fundingarrangements is limited to a user view of how thenew systems have worked. This has revealed, forexample, that users have found it too complex.Sustainability of funding systems is discussed inseveral reports.

Some innovative funding mechanisms arereported, one concerned the UK national govern-ment funding a union-led initiative to exploreways local networks of workers and trainerscould develop systems for training for localneeds. Essentially this was a union trainingcapacity building exercise that fits well with life-long learning objectives. The scheme expandedover the three years of evaluation and wassuccessful in creating a range of training provi-sion not previously available and as reportedabove, in levering funding from additionalsources. The scheme was also incorporated intoUK mainstream provision.

5. Financing VET: reforms and evaluation


5.2. Evaluation of financing

Using evaluation data for resource allocation is anissue for governments and large-scale funders ofVET (Summa, 2000). Using evaluation data asjustification for changing resource allocations isdifficult as evaluations tend to recommendchanges to single programmes and do notprovide evidence of the effectiveness of VETacross different contexts. Many reports simplystate costs of programmes which is of little eval-uative use, but sometimes data are given astrends over the period the reform operated. Thisis sometimes accompanied by a description(rather than an evaluation) of the funding distribu-tion system (e.g. VTOS in Ireland). It is possiblethat absence of financial data shows that analysisleading to evaluating the value-for-money ofreforms is problematic. This is likely to be partic-ularly difficult when the gains of the reform arejudged against the system that existed before thereform was introduced. The key issues are likelyto be absence of controllable variables and theconsequential lack of a baseline for measure-ment. The Irish VTOS report attempts to show thedifferences in scale of funding for VTOS asopposed to other VET for the long-term unem-ployed and discusses the numbers of partici-pants ‘processed’ by the different schemes. Thiscomparative work is helpful but does not give aclear indication of value-for-money.

The econometric studies included in thesample of reports simplify judgement on whetherone reform was more effective than another(within the bounds of how much it is possible toknow of what effectiveness means) but rarely gofurther and comment on the funding processitself or of value-for-money of VET provisionoutside the comparative study.

Some reports include information on how learnerfinance issues affect take up of VET provision. ThePortuguese report outlines how student financing isaffecting retention. A law governs how student feesare set and the extent to which a school canborrow money and sell products. Evaluationshowed the financing model to be faulty as it wastoo short term, highly bureaucratic and heavilydependent on government subsidy. The report alsohighlighted serious weaknesses in evaluationmethods used to judge the effectiveness of thefinance process. The Finnish report also detailssome research into financing measures. New uppersecondary vocational provision is funded 60 % bythe State and 40 % by regional authorities. Thereport also explains the possible commercialfunding of provision and its limitations, althoughonce again there is no evaluation detail.

The evidence in these reports on financialaspects of reforms is limited. There is someeconometric analysis with a scientific basis whichis probably more reliable for judging the effects ofVET financing.

Table 4: Measuring the impact of new funding arrangements

Year of programme Year 1 Year 2 Year 3

Number of projects with levered funds 16 13 30

Proportion of total projects 36 % 20 % 32 %

Average amount (per project) levered GBP 1000s 28 43 54

Source: Union learning fund (UK).

Measurement of impact is the key focus of thisreport. It may be important to distinguishbetween impact and outcomes. Literaturesuggests the most accepted classification ofoutcomes of training is the four-level system firstdescribed by Kirkpatrick (1998). These are:(a) reaction to training: this determines how

much the learner might learn;(b) learning: a measure of how much is learned

and therefore available to transfer to the job;(c) behaviour transfer: how much work behaviour

changes as a result of training;(d) organisational results: in what ways and how

much the organisation benefits from changedbehaviour.

These four sets of outcomes are hierarchicaland maximising each one provides a better basefor the next. Grubb and Ryan (1999) take abroader view; they describe outcomes using thefollowing categories:(a) for the individual, e.g. economic, wages,

employment pattern, progression, developinglearning and skills;

(b) outcomes of other agents, e.g. productivity;(c) welfare and social programmes: reduced

dependence;(d) social outcomes, e.g. reduced criminal

behaviour.A way of reconciling these two approaches is to

consider all that is planned and delivered as VETas ‘intermediate’ outcomes, for example, perfor-mance of the learning infrastructure, efficiency andequity of learning providers and qualification targetrealisation. As stated earlier, programme evalua-tion, as opposed to impact evaluation, will startfrom changes to infrastructure as an outcome andthen progress to changes in institutions andprogrammes and finally to impact on individuals. Infact, according to Plewis (2001), the final destina-tion of impact is with the individual receiving VET,their families, their community, organisations andsociety generally. It is this model of assessingimpact which underpins the analysis presented inthis report although it is clear that as one moves

away from the individual other variables andcontextual factors often mask the effect of thespecific VET intervention.

6.1. Establishing a baseline

There are no absolute positions for measuring theimpact of VET intervention. Established systems– no matter how they underperform – have animpact in their current form. Around half thereports described the process of establishing acontrol group or making some measurementsbefore intervention began. Some reports, notsurprisingly those using econometric techniques,made strenuous efforts to make a convincingcase for having measured an absolute change ina condition as a result of the intervention. There issome evidence that large-scale reforms do notappear to make ex ante evaluations or do notconduct them in sufficient detail to make impactevaluations later (2).

As stated earlier every report made an explicitstatement of the aims of the evaluation in relationto the reform. Most describe their objectives infairly general terms, for example:(a) to support changing requirements for training (D);(b) to develop earnings and career prospects (D);(c) to promote innovative activity in training (UK); (d) to maximise the value of study for employers

and individuals (UK); (e) to improve the supply of intermediate skills (UK);(f) to diversify VET provision (P);(g) to make the curriculum more flexible (P);(h) to experiment with the content of syllabuses (I);(i) training to help you get the job you want and

keep it (EL);(j) to develop student autonomy (S, NL);(k) to lower entrance barriers to education (FIN);(l) to create a culture of professional develop-

ment (E).Other reforms had more specific objectives:(a) to improve student results (UK);

6. Levels of evaluation

(2) See, for example, the final assessment report for assistance funded under Phare VET reform programme in the CzechRepublic, OMAS Consortium Middle Region, 1999


(b) to reduce long-term unemployment figures(IRL);

(c) to retain more students in post compulsoryschooling (IRL);

(d) to reduce youth unemployment (S).To measure impact it is probably helpful to

transform this type of aim or goal into morespecific objectives. This often takes place whenthe objectives for the evaluation programme aredrafted. Some examples from evaluation reportsdemonstrate this:(a) assess the probability of getting a job, the

rise in earnings, the probability of higher-levelstudy … all relative to a control group (S);

(b) how well VET provision meets individualdemand (NL);

(c) inspect evidence of practice in schoolsmeasured against common criteria (IRL);

(d) educational gains in qualifications (IRL);(e) comparing special VET provision and normal

VET provision for satisfaction ratings (UK).This area of evaluation design – definition of eval-

uation objectives – is where an evaluation pro-gramme can be optimised for generating impact.

6.2. Impact and timing ofmeasurement

Impact can usefully be considered in two discretedimensions. The first is the subject of impact (e.g.individuals, institutions), the second is a temporaldimension.

If we take, for example, an employer-ledtraining programme as the VET intervention wemay consider the subject of impact as any of thefollowing:(a) level of competence of an individual worker;(b) capacity of a work group to work differently

(flexibly, creatively);(c) capacity of the individual or work group to

disseminate new knowledge;(d) productivity of a work unit;

(e) stability of staffing;(f) manageability of a work unit;(g) profitability of the company.

There will be more subjects of impact fordifferent specific situations. The effect of theintervention also changes with time. The changecan be continuous, for example the content ofthe VET intervention can dissipate or the skillslearned can lead to learning new skills. Thechange can also be categoric in nature and beseen in distinct stages. For example, the effect onan individual could be focused on their reactionto learning during a training programme or aboutgetting a job based on the course in the weeksfollowing a programme or about promotion andfurther learning in the longer term.

The impact evaluations included in this studygenerally concern themselves with the immediateeffects of a programme on VET provision orcustomer satisfaction with their VET programme.They generally have a weak time dimension; theyare timed to measure effects during the interven-tion or soon afterwards. A Swedish evaluationextending several years after intervention hasalready been outlined above.

VET reform might change the situation of anindividual (e.g. employability, earnings, promo-tion), the VET structures and systems (e.g. devel-oping trainers, financing, qualifications) and asso-ciated supporting arrangements (e.g. guidance,skills needs analysis). This kind of analysis iscommon in evaluation literature (e.g. EuropeanCommission, 1999) and is a useful way of consid-ering the reports included in this study.

6.3. The individual

The individual is the focus of many reforms andthe views of individuals are sought in most of theevaluation programmes included in this study.However these views are often sought on how theindividual feels about the intervention, for example:

The level of satisfaction from vocational training programmes was found to be quite high. Out of all trainedrespondents, 90.6 % were very/quite a lot satisfied by the infrastructure of training centres, 89.3 % by thequality of training, 89.8 % by the trainers’ standards and 83.8 % by the usefulness of the training provided.Overall, satisfied respondents reached more than 90 % (91.8 %) and remained high level among all demo-graphic groups.

Source: Greek report of VET effectiveness.


Taking this further, the effect of an interventionmight best be measured by difference in opinionbetween an individual participating in the

programme being reformed and one that is not.This might yield more reliable information aboutthe relative effect of the intervention. For example:

While this information is important and usefulthere may be dominant short-term influencesoperating that, in time, may decay. Some eval-uations attempt to draw out a longer-term

perspective from participants and seek infor-mation about how individuals feel relative toothers who had not participated in the interven-tion. For example:

Both the differences in response rate and the simi-larities can be useful. Moving on from views aboutthe future, and possibly increasing the usefulness of

data on effects, another style of impact evaluationlooks at actual outcomes as opposed to views andopinion about future outcomes. For example:

Income 74.4

Prestige 58.4

Career chances 38.2

Opportunity for progression in company 24.4

Source: Austrian reform of higher education

Table 5: Proportion of Fachhochschule leavers who feel disadvantaged by their counterparts in‘normal universities’ with regard to selected aspects

(%)

Table 6: How much an apprenticeship course is helping to…

Intervention sample Comparison sample

get skills and qualifications 77 72

find out about working life 63 10

gain entry to chosen career 59 57

decide on a career 52 38

find out how firms work 48 20

find out about job opportunities 27 10

Source: UK modern apprenticeship study

(%)

Qualification Salary 1992 Salary 1997 Difference

Bac technologique industriel 5 500 7 500 2 000

Bac professionnel industriel 5 500 7 300 1 800

Bac général 5 000 6 500 1 500

Bac technologique tertiaire 5 200 6 000 800

Bac professionnel tertiaire 5 500 6 000 500

Source: French study of the 1992 cohort of school leavers

Table 7: Increases in salary from the first employment in 1992 to employment in 1997(FRF)

The financial effect on the individual ofinvesting in different levels of qualificationbecomes clear. The final and perhaps the mostrigorous approach to reporting effects on the

individual is to present information on the effectof the reform on a group of individuals comparedto a demographically matched group over aperiod of time. For example:


Non participants Programme 1 Programme 2

Earnings 1 year after programme SEK 37 750 SEK 52 110 SEK 44 120

Earnings 2 years after programme SEK 89 300 SEK 74 770 SEK 66 700

Employment within 1 year of programme 37 % 29 % 24 %

Employment within 2 year of programme 42 % 41 % 39 %

Study within 1 year of programme 11 % 10 % 5 %

Study within 2 year of programme 12 % 13 % 9 %

Source: Swedish youth labour market studies.

Table 8: The measures of success for two active youth labour market programmes

It is possible to deduce the relative effects of theprogrammes on individuals who have or have notparticipated in the alternative programme. A profileof effects on individuals emerges. Thus it is possibleto determine a range of ways of measuring effectsor outcomes ranging from the purely subjectiveview of the impact of intervention on an individual toa much more objective view of the impact of inter-vention on an individual. All of these have theirvalue. In the case of a formative evaluation of acurrent programme of reform, the immediate viewsof the participants can help managers to respondpositively and increase motivation to learn and toremove barriers to learning (3). Long-term compara-tive impact data can identify where further invest-ment can be made with optimal value for money forlong-term outcomes. Sometimes the proven effec-tiveness of the benefit of a training programme

carries little weight with potential recruits comparedto the ‘grass roots’, here-and-now opinion (espe-cially if it comes from peers) about content andpedagogy. Therefore, knowledge of the latter isimportant if recruitment levels are important.

In addition to measuring the effects of VETinterventions on the individual, it is also commonpractice to measure how the individual was moti-vated to participate and consequently how theytranslated the intervention into impact. Motivationis a complex area to examine in evaluation studies.The level of motivation does not depend solely onthe programme or future action; it dependscrucially on context and a multitude of factorsimportant to individuals. Nevertheless, thiscomplexity does not deter evaluators from seekinginformation about potential motivating features ofprogrammes, as the following examples show.

(3) For an example of this see the German PQO study.

Table 9: Why a modern apprenticeship is attractive

% Attracted a lot by ...

78 learning real skills in the workplace

76 career prospects

72 good quality training

70 being employed

69 paid to train

63 earning a wage

32 status of modern apprenticeship

32 being able to leave school

Source: UK modern apprenticeship survey


I didn’t like study 49.8

I was thrown out 12.9

I would like to study independently 9.1

I was not doing well 5.4

Family reasons 5.0

Financial reasons 3.1

Prefer another qualification 1.0

Did not like schools and work 0.9

I am studying (something else) 6.7

Source: evaluation of Italian Apprenticeship

Table 10: Motives for abandoning apprenticeship(%)

The effects on individuals of factors external to the provision can be considered when evaluating effectiveness.

Table 11: Effect of training on long-term unemployed

Self reported effect %

Increased self confidence 55.6

Made new friends 43.3

Sense of accomplishment 43.9

Clarified my objectives 20.0

Better able to occupy time 16.1

Improved level of education 42.2

Obtained qualification 30.0

Qualified for further study 20.0

Qualified for further training 11.7

Motivation for further education 29.4

Motivation for further training 21.7

Helped obtain job 10.0

Source: Irish VTOS evaluation

To the question how relevant is your employment subject to that of the vocational training programme youattended, almost 3 out of 10 trained employed respondents replied very/quite a lot relevant. Consequently,almost 1 out of 10 trained respondents work at this period of time (almost 10 months after the completion ofthe vocational training programme) in a job relevant to the subject of the vocational training programme he/shehad attended.

Source: Greek evaluation of VET

The information above shows the effect of characteristics of the programme on individuals.

These last four extracts show informationabout effects on individuals that reform managerscan use to optimise recruitment and retention.

Sometimes the main focus of evaluation is on theprincipal agents managing the VET infrastructureand the effect on individuals can be lost. Forexample, the effect on students in training linked toUK work related further education (WRFE) reformwas unclear. Colleges based their projected demandfor courses on employer data (as the main focus ofthe reform) and failed to notice that students werebecoming more conscious of employment as anoutcome of further education (FE) and thereforecourses with the highest chances of employmentwere likely to become more popular. However, mostcolleges in the reform programme continued to runcourses with low employment prospects. It wasassumed from the outset that employers were thekey subjects of the impact of the reform as theydetermined the labour market responsiveness of theplanning process. The students came lower downthe list of priorities for judging effects and somewere not well served by the reform.

Determining the layers of impact and the relativepriority given to different recipients is important inevaluation design. The objectives of evaluationusually make clear who or what is the main subjectof impact. Evidence from the set of selectedreforms suggests individuals are often the mainrecipient and providers and enterprises follow.Larger reforms have goals linked to a better socioe-conomic situation and, while difficult to measure,effects on these follow on. Returning to thetemporal dimension, it is also clear from thereforms studied that enterprises and trainingproviders may be the first to experience thechanges associated with the reform but in timethese are transformed into effects on individuals.Enterprises and training providers are usually thefocus of reform process objectives rather than ofobjectives linked to outputs or effects. In this studyexamples exist of where the focus of the evaluationobjectives are only on enterprises and trainingproviders as part of the process of reform. Othersexamples exist where these agencies are omittedfrom the evaluation because the effect of the reformis only measured as an impact on the individual.

6.4. The organisation

There are three main categories of institutionsthat are subject to impact studies:

(a) businesses (as employers of trained people);(b) providers of training (including businesses as

trainers);(c) regional or local government.

There was little information in the selection ofreports on the impact of interventions on thelatter two categories. This may be because it isassumed that the effect of the VET interventionon the individual is the prime outcome and onlyprocess evaluation offers added value for money,in that it helps providers and local officials tounderstand VET better and to place themselvesin a position to modify the process for the best.Stern (2003), sees this development of knowl-edge as a key purpose of evaluation.

A characteristic of measuring the effect of reformson enterprises is the difficulty of obtaining feedbackon impact. Where information from enterprises isavailable in reports it usually takes the form of satis-faction data rather than more specific information onthe changes to work process and productivity thathave arisen as a result of the VET intervention.Questionnaires are usually the preferred means ofdata gathering, and while this kind of survey tech-nique can reach many enterprises, the more specifictype of information about impact is difficult toextract and correlate by means of questionnaire.Some of the reports in this sample show poorresponse rates to surveys; sometimes the evaluatorsare concerned about this and signal reservationsabout the generality of findings (e.g. the UK Unionlearning fund reform). Response rates are only main-tained if the questionnaire is simple and not timeconsuming to complete, again confounding thecollection of impact data. In conclusion, the dataavailable for analysis from enterprises is likely to beshallow in terms of outcomes and effects. Anexample drawn from a report in this study illustratesthis: ‘A number of benefits are not quantifiable whichmakes an assessment of impact of a project moredifficult’ (Union learning fund – UK).

These findings are confirmed by evidence fromresearch studies that enterprises generally do notevaluate the impact of VET but almost always tryto estimate costs (Tessaring, 1999).

Reports indicate that employers were moti-vated to participate in training activities forreasons such as:(a) improving the quality of training;(b) staff replacement;(c) company growth;



The following example, drawn from the UKWRFE reform based on a large sample of enter-prises, makes seeking views, rather than harddata on outcomes or effects, a little more usefulby providing a comparable analysis of employerviews of training courses. Some colleges hadparticipated in a reform to develop training

specifically to match the needs of enterprises intheir locality, while other colleges were notinvolved. The views of enterprises in the regionsserved by the colleges which were participatingin the reform were compared to the views ofenterprises in the region where colleges were notinvolved.

(d) availability of government funding for training;(e) improving the general level of education of

the work force;(f) developing a more motivated, confident and

enthusiastic workforce;(g) developing basic literacy and numeracy skills;(h) improving the specialist skills base;(i) improving communication.

In terms of benefits derived from VET activities,which is taken to mean outcomes of the activi-ties, the following features were also identified bybusinesses:(a) project based work was more effective;(b) better industrial relations;(c) could lead in the longer term to a workforce

that was easier to train.

These headings are largely generalisations ofmore specific reasons for valuing training (4).

As in the section above on the impact on individ-uals, employers’ views are sampled for satisfactionratings, they are also sampled to identify issues, forexample the extent of bureaucratic loading andpoor relations with training providers. In many casesthey are surveyed simply for information, forexample: ‘how are trainees supervised?’.

The first stage of extending a typical satisfac-tion survey is to seek the views of training usersabout changes. The following example showshow useful this extension can be.

In the German PQO (Personalqualifizierung inOstdeutschland) study enterprises singled out thefollowing outcomes:

(4) More details on this issue can be found in a synopsis of research into how training impacts on enterprises is given in Tessaring,1998, pp. 72-77.

Table 12: Changes in workers as a result of VET

Number of enterprises

Have become more qualified 33

Are more motivated 27

Have taken on new tasks 10

Have been promoted 3

Have received better pay 6

Table 13: Employers’ views on the reform

Colleges provide training very employers’ views employers’ views well/quite well in terms of: in regions of the reform in other regions

The courses provided 76.1 42.6

Relevance of course content 77.5 39.1

Using appropriate machinery 75.5 34.7

Training for a workplace setting 69.1 34.9

The timing of the courses in the year 80.1 60.3

The mode of attendance 84.9 62.1

Cost effectiveness 84.0 63.7

(%)

This is useful material for those judging theeffectiveness of the scheme and could be usedfor any extension and dissemination of thereform. An extract from the Austrian Fach-hochschule reform shows another example of

how satisfaction ratings can be made morescientific. In this case the views of enterpriseswere sought about the qualities of graduates fromthe new vocational institutions compared to grad-uates from traditional universities.


Table 14: Fachhochshule graduates compared to graduates from traditional universities

Better Same Worse

Ability to meet practical work demands 37.3 49.3 13.4

Ability to work in a team 29.9 66.2 3.9

Relevance of qualifications 27.0 59.0 14.0

Introduction phase (length of time spent) 21.3 72.5 6.2

Ability to cope with new tasks (speed) 19.3 66.7 14.0

Creativity in problem-solving 15.6 67.5 16.9

Ability to apply their knowledge 13.0 70.1 16.9

Career chances 7.8 77.9 14.3

(%)

Surveys of employers sometimes pinpointwhere VET can be improved. Enterprises involvedin the German PQO initiative identified threeareas:(a) consider the demand side of training carefully

and customise training accordingly;(b) be proactive in marketing their training

programme;(c) be flexible about when and where to deliver

training. These features were also identified as effects

of the UK review of work related learning incolleges. Employers witnessed growth intailor-made courses and in methods used foridentifying demand. They believed colleges hadbecome more flexible.

The German PQO study sought opinion on thecharacteristics of good training. Good trainingwas rated by employers as better in the followingways:(a) breadth of courses;(b) exclusion of extraneous skills from courses;(c) more appropriate machinery used;(d) links with the company’s way of working;(e) handling of suggestions or complaints.

Only the first two had statistically significantpositive outcomes from employers.

In reforms where fundamental changes to thestructure of VET are taking place (for example inSpain and Portugal) effects on employers cantake on a different focus. The chief area of impactis involvement itself and participation measuresare used extensively to judge the effects ofreforms. This is as far as assessment of impacton enterprises extends in countries with theselarge-scale reforms.

In many of the more educationally focusedreforms, for example development of qualifica-tions systems, businesses were not considered adirect source of information about impact in thereforms selected. What would be a measure ofimpact on an enterprise of a school-basedreform? It is interesting to think about what usefulmeasures of effects, examples might be: easierrecruitment, lower initial training/induction costs,better skills supply or reduction of staff turnover.

In the light of all of the difficulties of gatheringemployer views of impact, the wish to continueinvolvement with a VET initiative may be the bestindicator of an employers’ view that the impact ispositive. However, more information would begained if values were compared for differentpathways. For example:


Trade unions have not usually been the subjectof impact evaluation, though in the case of theintroduction of the Union learning fund in the UKthe effect on unions was measured specifically.The reform aimed, amongst other things, todevelop the capacity of the union to delivertraining and to distribute funding for training.When the unions were asked to define whateffects had occurred on building capacity theyhad diverse views: some stated ‘the delivery oflearning activities’; others mentioned ‘developingcredibility’ and ‘developing sustainable resourcessuch as websites’. Clearly the definition andagreement on likely or desired effects at thestage of programme design would have helped indetermining capacity building. The evaluatorssuggest it is possible to group the diverse kindsof effects for unions under three headings:(a) developing resources and expertise;(b) securing ownership of the lifelong learning

agenda within the union;(c) developing union strategy and policy.

6.5. The aggregate level of the economy and society

It is much more difficult to measure the effects ofVET interventions on broader society. The linksbetween improved individual performance,improved commercial performance, greaterwell-being for individuals, their families and theircommunity is hard to establish in quantitativeterms and none of the reforms selected haveattempted to establish such links. However, someof the reforms, notably the German and Spanishreforms, have had as their central aim the devel-opment of the regional economy.

In their report on the impact of human capitalon economic growth, Wilson and Briscoe (2003)find that several fields of economic researchsuggest a link between investment in educationand training and general national economicperformance. For example they show that microe-conomic research suggests ‘… strong links …between education productivity and output levels.Although some have questioned the direction ofcausality and argued that much education simplyacts as a screening device to help employers toidentify more able individuals, the generalconsensus seems to be that education doesresult in higher individual productivity and earn-ings. On balance, the results suggest a strong andpositive causal link between investment in educa-tion and training and earnings. This applies bothat the level of the individual and when consideringthe broader social returns to such investments.The evidence seems to suggest substantial socialas well as private benefits to this kind of activity.The implication is that what is good for the indi-vidual at a micro level is also good for society atlarge at a macro level.’

They reach a similar conclusion when theyconsider ‘rates of return’ literature and the litera-ture on skills and organisational performance.

Green et al. (2003) reach similar conclusionsabout the immaterial benefits of VET. They alsolook at the micro level and cautiously affirm that:‘raising educational, skills and training levels isneither a necessary nor sufficient condition ofpromoting macro-social benefits. However,improving the distribution of educationaloutcomes may be one way in which educationand training can make some contribution to moregeneral economic and social redistribution.’

Many of the reforms, notably the Nordicreforms, have had the general well-being of

Table 15: Are you interested in employing more graduates from Fachhochschule in the future?

1998 2000

Yes 60.6 65.5

Maybe yes 27.2 21.4

Probably no 12.1 7.1

No 0 6.0

Source: Austrian vocational higher education initiative

(%)

society at the heart of developments. It will bedifficult and will take time to evaluate the effectsof IVT on these broad goals and it is likely thatother social and economic changes will rendersuch analysis impossible. However, proxymeasures for social change are being used and

do provide some impact data. Dropout fromupper secondary education is often reported as aproxy measure; the reasoning is that new IVTcourses will appeal to young people who mighthave rejected further non-compulsory generaleducation.


Table 16: Five years after gaining a qualification how many young people in France areregistered as unemployed or inactive?

Qualification Registered unemployed Inactive

Bac technologique industriel 13 1

Bac professionnel industriel 11 0.5

Bac général 12 6

Bac technologique tertiaire 18 4

Bac professionnel tertiaire 16 3

Source: French study of the 1992 cohort of school leavers.

(%)

Another proxy measure for evaluating thewider impact is the success of individuals

entering work and/or study. This data is oftenreported as trends, for example:

Table 17: Destinations on completion of Portuguese IVT programme

1998 1999

Continuing studies 42 29

Employment (active life) 39 41

Other 19 30

(%)

In some reforms the aim was to provide a meansfor the excluded to gain access to education andtraining so that they might return to work. Clearlythere are wider benefits for society (as well as forthe individual and their family) for reducing thenumber of those dependent on benefit payments.These proxy measures of wider impact are also

reported. For example, the Irish report on VTOSattempts to show the effect of education andtraining on unemployment. The effect on unem-ployment of VTOS participation is not reportedagainst that of non-participation but the evaluatorsfocus instead on the reduced probability of unem-ployment with higher levels of qualification.

Table 18: Labour-market situation immediately following VTOS by certification achievements

Qualification unemployed in education in job

5 or more subject leaving certification 80 20 0

4 or less subject leaving certification 70.6 17.6 8.8

Portfolio of achievements 76.9 23.1 0

Vocational qualification 68.8 3.1 28.1

No certification 64.6 6.2 16.9

(%)


The ease of entry into the labour market aftertraining is clearly a key output measure. In somereforms, for example the Portuguese IVT devel-opment and the Irish VTOS reform, there is arequirement that training organisations consultwith local employment agencies and employersbefore determining the training provision theyprovide. Unfortunately there is no comparativedata available for drawing conclusions about theimpact of this requirement.

Finally, the impact on broader society ofreforms which have a decentralisation of controlat their centre (for example the Italian, Spanishand Swedish reforms) is important. No evaluationof this kind of impact is reported nor is thereevidence of the intention to attempt to measurethis type of impact. However, there are likely tobe significant impacts arising from the effects of

local decision-making and greater opportunity tofocus on issues more important locally than theyare nationally. The driving forces for thesereforms are often located in the notion ofmaximising the effect of VET and minimisingbureaucracy.

Labour-market dynamics, training infrastruc-ture, qualifications systems, funding schemes,societal pressures, cultural values are all deeplyinter-related and to look for effects in one dimen-sion may not be the best way of informingimprovements for individuals and businesses.

Having studied and analysed the set of reportsin depth, there are some general points whichemerge which could help with improving evalua-tions in the sense of making them more impactoriented. These points are discussed in the nextchapter.

In general terms impact should be measured withthree parameters in mind:(a) qualitative measures: measures that locate

the intervention in a wider context of theissue being addressed and what theory wecan draw on to illuminate the changes in theposition of stakeholders;

(b) quantitative measures: the depth and breadthof the impact of the intervention on potentialstakeholders in relation to those of compa-rable position that are not participating;

(c) temporal measures: indication of impact overa longer timescale to see effects that mayresult from the initial intervention.

The analysis of the evaluations of VET reformshas suggested that there are some steps that canbe taken towards securing a better evidencebase for their impact. These steps do not repre-sent fresh thinking about impact analysis; theysimply emerge from the reports in response to thequestion ‘how could we discover more about theoutcomes of reforms when we evaluate them?’.

In setting up an impact evaluation the followingconsiderations might improve the measurementof the impact of a reform:(a) identify what is driving the reform; why is the

reform happening? This examination can clarifythe need for impact analysis; will the data onimpact affect the forces for change? Will theseforces continue regardless of feedback?

(b) clarify the purpose of the evaluation pro-gramme. Is the identification of outcomes onindividuals, enterprises and wider communi-ties adequately represented in the purpose ofthe evaluation?

(c) ensure the objectives of the evaluationprogramme cover the desired outcomes ofthe goals of the reform. Check that the datawhich is likely to be generated by the evalua-tion can be analysed in a way that fully meetsthe objectives of the evaluation programmeand covers the desired outcomes of thereform;

(d) consider methodology and use opportunitiesto make the process as rigorous as possible.This does not necessarily involve developingquantitative approaches but it does meanconsidering baseline measurements andtimelines carefully;

(e) if possible, analyse the effects of variablesthat cannot be controlled, for example thechanges in the vigour of the labour market orchanges in attitude towards certain workingmethods;

(f) where resources allow, use survey methodsthat are sensitive to context. These usuallyinvolve using semi-structured schedules inface-to-face interviews, telephone surveys,focus groups or on-line adaptive survey tech-niques;

(g) consider how the results will be presented tosponsoring organisations and highlight theneed for an extended timescale for trueimpact statements to emerge and to takeaccount of the tentative nature of interimimpact-related findings.

The following example illustrates how theprocess of breaking down the objectives of areform and transforming them into a plan for eval-uation can raise the level of impact measurement.

7. Towards better evaluation of reforms

From aims to enquiriesIt may be helpful if the following process is used to identify areas where impact measures may be located and profitablymeasured. It involves resolving the aims and objectives of reform into lower level questions for evaluation. In reportingimpact the answers to these lower level questions can be reported and aggregated to determine a response to the mainquestions.Main reform objective: to introduce employer-led apprenticeshipOne of the objectives for evaluation: to determine the response from employersA qualitative response and a quantitative response are sought – taking the latter first:How many employers were involved?


The impact of a reform occurs at several levels;evaluation programmes might consider the primaryimpact on individuals and institutions, secondaryeffects on user organisations and communities,and tertiary effects on society and the economy.

A key step in enhancing impact measures is toconsider the timing of data gathering from stake-holders. This usually takes place in stages: duringthe reform, immediately afterwards, one yearafterwards and three to five years afterwards.

How many were eligible?Of those involved what are the prominent sectors, what size of companies?Of those not involved what are the prominent sectors, what size of companies?Regional differences?How many are planning to be involved in the future?Etc.

For the qualitative response:Of those involved – satisfaction ratings, including longer-term plans. Changes to non-apprenticeship training plans,changes to recruitment practice and policy.Of those not involved – reasons why not, plans for the future, potential inducements required.

The 19 reports in this meta-study provide anoverview of the kinds of national reforms to VETthat have taken place over the last 10 years. Theselection of reports (the main focus being impactstudy) may have distorted the review but it seemsclear that VET is now less centrally determinedthan it once was and that its educational role isnow wider in terms of participation, content andassessment.

Key dimensions of impact evaluations havebeen highlighted: the ways in which data iscollected can facilitate more scientific analyses;in-depth study of the aims of reforms and appro-priate objectives for evaluations; careful consider-ation of levels of impact; and the timing ofmeasurements is important.

Common evaluation criteria have been devel-oped which have the potential to make studies ofthis kind more systematic. The criteria haveproven to be useful in highlighting aspects ofevaluation to be considered in setting up aprogramme. As a tool for analysing existingreports they functioned well. However, they werelimited in their capacity to create comparativeanalyses by the varied nature of the reports in thisstudy. The criteria used to evaluate reports areworthy of wider dissemination and are now readyfor application in other contexts.

This research has shown that evaluation prac-tice in the context of VET across EU countries isvariable, ranging from a wish to integrate evalua-tion into the infrastructure of reform to practiceswhich minimalise investment in evaluation. Evalu-ation of impact is also generally restricted tosatisfaction surveys of current experience of areform programme and they rarely investigate thelong-term effects of programmes. There arestrong indications that there are differences in theway countries value evaluation. Further work isnecessary to elaborate on this point but expertshave reported a dearth of evaluation, forexample, in Germany and in Greece. There seemto be thorough evaluative approaches linked toreforms in Portugal and Spain. In Finland and inSweden we see evaluation tightly entwined withthe reform process.

It is possible that this variability may also berelated in some countries to the availability ofpeople who are confident and experienced inboth VET policy planning and delivery and inevaluation methodologies. Cedefop has a role indeveloping expertise in the impact evaluation ofVET interventions. Dissemination of good prac-tice is the key and the publication of the thirdresearch report will be an important step.

There may be circumstances where cross-national studies of impact evaluation are sought.The experience of this study indicates that apanel of researchers would probably be the bestway of carrying out such studies. This wouldhave the advantage of sharing expertise as wellas ensuring sensitivity to any national context byusing experts from different countries on thepanel.

The examples in the previous chapter showthat it is relatively easy and inexpensive toimprove basic satisfaction surveys to providemore information on effects. More work can bedone to construct guidance for those embarkingon evaluation design applied to VET. This guid-ance must be sensitive to the political nature ofevaluation, to the socioeconomic context inwhich it is conducted and to the range of stake-holders with an interest in VET reform.

Such an improvement in the volume andquality of impact analyses will possibly influencethose who commission evaluations to be moreconfident that impact analysis is reliable, validand useful. Until impact analysis can be relied onfor what it can offer it will not draw the invest-ment it deserves and will not be used as much asit should. The strength of impact analysis is alsoits weakness in that it is a long-term evaluation ofa reform and reports on whether it has worked orhas not worked. Commissioners of process eval-uation are confident about it because it is shortterm and yields results that give a sense of goodhousekeeping and uses open, democraticprocesses.

The reports studied contained references tothe driving forces that create VET reform.However, there is little detail provided about how

8. Summary and conclusions


these forces operate and how feedback fromevaluation programmes might intersect withthem. One of the main reasons for conductingimpact evaluations is that this intersection is facil-itated. Research might usefully be conducted onthe dynamics of this process and could beginwith a review of the literature describing thedriving forces behind the reform of VET.

The findings of this study are soundly based inthe evidence from the sources used. However, itis clear that the research conducted could beextended. The current study is limited by suchfactors as the availability of reports and it ispossible that some useful reports of reforms havenot been included in this study. The study wouldalso benefit from a more extended understanding

of national contexts, particularly about the keydevelopments in countries undertaking reform ofVET. It may prove useful to study the evolution ofa VET system and trace the drivers of change. Inso doing the informal process of evaluating provi-sion and making incremental changes as a resultshould become clear.

The meta-study methodology used in thisstudy is limited by what is written and publishedabout a reform and its evaluation strategy. It islikely that much more evaluative evidence existsin the minds of reform managers and evaluators.Access to this information demands carefulstrategic planning, face-to-face interviews andassurances as to the purposes of the widermeta-study process.

List of abbreviations

CVT Continuing vocational training

IVT Initial vocational training

PQO Personalqualifizierung in Ostdeutschland

QCA Qualifications and Curriculum Authority

QVE Qualified vocational education

VTOS Vocational training opportunities scheme

WRFE Work related further education (programme)

Clarification of purpose of these criteria

We are looking for criteria we can apply as weread evaluations of programmes that come underour remit. These will be fairly large-scale evalua-tions completed by reputable individuals andorganisations.

While we are concerned about ‘impact’ evalua-tion, we will take account of ‘process’ evaluationif we are to make useful conclusions about evalu-ating impact and the design of reform pro-grammes. Hence the breadth of territory coveredby the criteria below.

The criteria are grouped by chronology ofproject implementation. They are further groupedby the following characteristics of evaluationprogrammes: (a) context: what external influences are likely to

influence?(b) goals: how are the aims, objectives, mile-

stones described?(c) planning: what is discussed in anticipation of

what might happen?(d) management: what is done to make the

programme function effectively?(e) drivers: who is shaping the evaluation activity?(f) impact: what is the effect of the evaluation?

Before the implementation of the programme

Context

Reform settingWas the background (political, social, economic,cultural) for the reform made clear? How muchattention was given to setting when designing theevaluation?

Policy relevanceWas the nature of any explicit link betweennational economic goals and the reform (e.g.

skills needs, workforce development and partner-ship) made clear? Was consistency of reformswith other policies evaluated?

Policy descriptionWas a description of national policy used tounderpin the reform and its evaluation? Was itintended to measure impact? If there was nopolicy description what was the purpose of theevaluation?

Goals

Reform aimsWas the aim of the reform included? What was tochange (e.g. learning, institutions, frameworks,and certification)? What was to be maintained?Was this the principal focus of the evaluationprogramme? Were there clear targets for thereform?

Purposes of evaluationWas the purpose(s) of evaluation made clear? Forexample to inform decision-making (government,employers, providers, and trainees), improvingreform management, improving resource alloca-tion, improving the quality of programmes, amechanism for stimulating debates, a process ofbringing reform designers and managers toaccount.

Theoretical basisDid the reform have an explicit link betweenhoped-for outcomes and a theoretical basis forexpecting the interventions to lead to theseoutcomes?

Planning

Baseline for analysisWas the evaluation set up in such a way that abaseline for measuring the effects of interventionswas established? Was this baseline establishedand agreed amongst stakeholders before thestart of the programme?

Annex: common evaluation criteria


Management

Management structureWas the management infrastructure of the reformdiscussed? Who was consulted? Did the evalua-tion look at this?

Donor commitment Is there a declaration of intent and continuedcommitment to the reform from any donorauthority? Did the evaluation investigate it?

Risk managementWere the risks associated with the reform madeexplicit? What was done to reduce risk? Howwas risk management organised? Did the evalua-tion include programme risks?

Legal embeddingIs there evidence in the evaluation that legalarrangements were considered as a preparatorystep towards implementing the reform?

Programme handbookWas there a publication describing the reform?Who was it aimed at? How did any donor institu-tion contribute to this publication? Was the manualevaluated for content and usefulness to users?

Preparatory studiesWere feasibility studies carried out or ex anteevaluations planned? Did the evaluation look atthe quality of these studies?

Drivers

Reform actors Was there evidence of pressure for change? Whowas opposed to it? How did they act? Was thereevidence of involvement from policy level actors?Similarly was there evidence of bottom-updemand for reform? Was a stakeholder analysispart of the evaluation?

Impact

User impactWas it planned to make interventions that couldbe measured by changes in opportunity andpractice amongst end users (learners, communi-ties, organisations, and countries? How was theevaluation organised to measure impact?

During implementation

Context

FamiliarisationDid the proposed reform become widely known?Who discussed it? Did evaluation considerdissemination throughout the duration ofprogramme? If so what issues were discussedand why?

Goals

StabilityDid the goals of the reform change during theimplementation? If so did the evaluation identifythe influences that were driving the change?

Planning

Formative influenceIs there evidence of change in interventionprogrammes as a result of information arisingfrom the evaluation process or intermediateoutcomes of the reform? Was evaluation everpresent during discussions of the reform?

Complementary activitiesIs there evidence of organisations at national,regional and local levels working jointly towardscommon targets in a complementary top-downand bottom-up approach, and were they involvedin the evaluation?

Management

Transitional arrangementsWere there transitional arrangements in placebetween existing VET systems and those intro-duced as a result of reform during the period ofreform? How were these introduced, managedand dismantled? Did evaluation consider this?

Piloting arrangementsIs there evidence that any pilot phase was usefulbecause it linked strongly to systemic change,and was effectively disseminated? Were issues ofmainstreaming and sustainability addressedduring the expansion phase, and did evaluationconsider these?


Drivers

Stakeholder motivationWere people and organisations incentivised toparticipate? Did it work? How was this incentivi-sation differentiated and assessed?

Stakeholder groups Were key groups participating in VET involved?For example, those in pre employment activities,those upgrading their skills used in their job,those retraining in another area of work, thoserequiring VET to improve employability.

Stakeholder activitiesWas the involvement of individuals and keygroups monitored? How were they used as indi-cators of success or failure in programme and theevaluation?

Impact

Timely informationWas there a means of summarising evaluationinformation and regularly feeding in to the reformmanagement activity? How is this done?

After the programme hadreached maturity, beencompleted or was terminated

Context

Programme contextWas the political, social, economic and culturalinfluence considered important for the success ofthe reform, and how did evaluation take contextinto account?

EU prioritiesWere the EU-wide common themes for VET (suchas improving access to VET and increasingparticipation) addressed in the programme, andthe evaluation?

Goals

Duration of outcomesWas there a plan to look at long-term effects of

interventions? How sustainable is the reform?Was this evaluated?

Planning

Dissemination phasingIs there evidence of the dissemination processculminating at this point at the outset of theactivity?

Management

Reform organisationDid evaluation consider the appropriateness of thebasic structure of the reform and its interventions(e.g. legal, structural, management, funding)?

Programme timingWas a timescale specified and did the reformprocess meet its time-related goals?

Programme budgetWere cost considerations (of reforms) included inthe evaluation specification? What were theintended costs?

Monitoring tools Were monitoring and evaluation tools in place toidentify, take account of and measure the impactof policy changes during the course of the reformand did evaluation analyse them?

Programme flexibilityWas the programme adjusted to take account ofchanging financial, social and political contexts?Did the evaluation programme seek evidence forthis?

Drivers

Social partnersWas there a plan for full engagement of socialpartners? Were they involved in the evaluation?

Programme spilloverIs there evidence that the programme and evalu-ation practices (introduced from the EU) resultedin activity in a Member State and a programme ofnational action?

Impact

Personal impactIs there evidence that the intervention has had animpact on individuals for job change, learningprogression, and reward?

Institutional impactIs there evidence that the intervention has had animpact on institutions in terms of behaviouraltransfer of learned skills into results?

Societal impactWas the impact of interventions on communitiesmeasured? For example, families and socialorganisations.

Systemic impactIs there evidence of a systemic link between theoutcomes of the interventions included in thereform and policy development, and how was itmeasured?

The evaluation process: someoverarching criteria

Development ProcessWas there evidence that the evaluation wasunderstood as a development process, whichaims at capacity and institution building, andcreates self evaluation skills amongst beneficia-ries? Was the evaluation integrated into thereform process?

CultureWere the perceptions of major stakeholders (EUfunders, national government, major actors, andsocial partners, evaluators themselves) madeexplicit?

Autonomy of evaluatorsWas there evidence that the evaluators had somechoice in the style and direction of the evaluationand in the choice of tools? Is the evaluating insti-tution independent of the reform?

Coordination of evaluation projectsWas the evaluation programme made of sepa-rately commissioned elements? If so how wellcoordinated was the whole programme?

MethodWere practical constraints on the evaluation processdiscussed? Was there evidence of methodologicalweaknesses in the evaluation programme – e.g.sampling, instrument design, time and constraints?Did the impact analysis include a control group? Ifso was the process experimental or quasi-experi-mental? Was there a combination of qualitative andquantitative assessment of impact?

CompletenessIs there evidence of the whole process of reformbeing fully evaluated? Were there shortfalls inevaluation effort in specific phases?

IndicatorsIs there evidence of use of quantitative and qual-itative indicators being developed for the evalua-tion process? If there were no indicators (devel-oped by, for example, benchmarking and targetsetting) what effect has this had?

Evaluation skillsWas the experience of the evaluation teamdiscussed? Were the methods and mechanismsunderstood by stakeholder organisations?

Evaluation censorshipWas there any evidence of evaluation findingsand reports being shelved or not disseminated?

Evaluation budgetWas the size of the evaluation budget appropriatein relation to the activity as a whole (as a rule,5 % of budget for small projects, and 3 % forlarge programmes should be made available)?

Reporting formatsIs there evidence that the dissemination materialsare being targeted to different audiences asopposed to a one-size-fits-all policy?

Evaluation disseminationAre evaluation results being made available to theproject partners and to all other member countries?


Angeli, F. Il nuovo apprendistato. Rapporto 1999.Rome: ISFOL, 1999.

Antill, M. et al. An evaluation of the Union learningfund in year 3. London: DfES, 2001 (YorkConsulting Report).

Baker, J. Evaluating the impact of developmentprojects on poverty: a handbook for practi-tioners. New York: World Bank, 2000.

Baker, S.; Taylor, J. Effects of career progressioninterventions: a meta analysis. Career Develop-ment Quarterly, 1998, No 46, p. 376-385.

Barrett, A. Methodological introduction. In:Elson-Rogers, S. (ed.) Approaches and obsta-cles to the evaluation of investment in continuingvocational training: discussion and case studiesfrom six Member States of the European Union.Luxembourg: Office for Official Publications ofthe European Communities, 1998, Part I,Chapter 2 (Cedefop Panorama series, 78).

Borghans, L.; Heijke, H. Does the Dutch adult andvocational education act promote the macroef-ficiency of vocational education? 2002 (unpub-lished).

Coleman, N.; Williams, J. Evaluation of modernapprenticeships: 1998 survey of young people.Sheffield: Department for Employment, 1998(Research Report, No 93).

Cutter, J. et al. A second evaluation of the Unionlearning fund. London: DfEE, 2000 (YorkConsulting Report).

De Bruijn, E.; Doets, C.; van Esch, W. Flexibility insecondary vocational education. PedagogisheStudien, 2001, Vol. 78, No 6.

Department of Employment. Technical and voca-tional initiative report. Sheffield: Department ofEmployment, 1994.

Descy, P.; Tessaring, M. (eds) Training in Europe:second report on vocational training research inEurope 2000: background report. Luxembourg:Office for Official Publications of the EuropeanCommunities, 2001 (Cedefop Reference series,3008).

Economic Research Services Ltd. Evaluation ofmodern apprenticeships: survey of employers.Sheffield: Department for Employment, 1998(Research Report, No 94).

Ekstrom, E. The value of a third year in uppersecondary education: evidence from a pilotingscheme. Uppsala: IFUA – Institute for labourmarket policy evaluation, 2002 (Working paper2001: 23).

European Commission, Monitoring and EvaluationUnit, Employment and Social Affairs. Guide-lines for systems of monitoring and evaluation ofESF assistance in the period 2000-2006, Brus-sels: European Commission, 1999.

Felstead, A.; Unwin, L. Funding systems and theirimpact on skills. London: Department foreducation and Employment, 1999 (Skills taskforce research paper, No 11).

Finnish Ministry of Education. Valtioneuvostonselonteko eduskunalle uuisen koululakienvaikutuksista ja laessa asetettujen tavoitteidentoteutumisesta. Helsinki: Ministry of Education,2000.

Forcem, Fundacion para la Formacion Continua.Memoria banance – I acuerdo nacional deformacion continua 1993-1996. Madrid:Forcem, 1999.

Freundlinger, A.; Grillitsch, S.; Schmid, K. Evalua-tion neuer Lehrberufe. Vienna: Institut fuerBildungsforschung der Wirtschaft – Schriften-reihe/Forschungsbericht, 2000.

Green, A.; Preston, J.; Malmberg, L. Immaterialbenefits of education, training and skills at amacro level. In: Descy, P.; Tessaring, M. (eds)Third research report: background report.Luxembourg: Office for Official Publications ofthe European Communities, 2003 (CedefopReference series).

Griffith, J. An approach to evaluating school-to-workinitiatives: post secondary activities of highschool graduates of work-based learning.Journal of vocational education and training,2001, Vol. 53, No 1, p. 37-60.

Grubb, N.; Ryan, P. The roles of evaluation for VET:plain talk on the field of dreams. Geneva: Inter-national Labour Organisation, 1999.

Irish Ministry of Education. Report of the nationalevaluation of the leaving certificate applied.Dublin: Department of Education and Science,2002.

References

ISFOL. Il nuovo apprendistato e la formazione – ilmonitoraggio dei progretti sperimentali. Roma:ISFOL – Istituto per lo sviluppo della formazioneprofessionale dei Lavoratori, 2000 (Pragma sr-1).

Kaasbøl, E.; Pedersen, F.; Størner, T. Eleverne ogeud-reformen: oplevelser af forsøg med grund-forløb og hovedforløb i 2000. Copenhagen:Undervisningsministeriets forlag, 2001 (Uddan-nelsesstyrelsens temahaefteserie, No 9).

Kath, F. The funding of vocational education andtraining: an international comparison of objec-tives and impact. European Journal of Voca-tional Training, 1999, Vol. 17, p. 38-44.

Kirkpatrick, D. Evaluating training programmes, thefour levels. San Francisco: Berrett-Koechler,1998.

Klein, T. Les bénéficiaires du contrat de qualificationentrés à l’automne 1994: les enseignements de4 vagues de panel. Paris: DARES, Ministère del’Emploi et de la Solidarité, 2002.

Larsson, L. Evaluation of Swedish youthlabour-market programmes. Uppsala: Depart-ment of Economics, 2000 (Working paperseries, 2000-6).

Lechner, M. An evaluation of public-sector-spon-sored continuous vocational training programsin East Germany. Journal of human resources,2000, Vol. 35, No 2, p. 347-375.

Martin, S. A critical analysis of the evaluation ofacademic performance in Spanish educationalreforms. European Journal of Education, 2001,Vol. 36, No 1, p. 77-90.

Metcalf, H.; Dawson, C. Evaluation of thework-related further education programme.Sheffield: Employment Department, 1995.(Research Series, No 55).

Ministério da Educação e para a Qualificação e oEmprego. Despacho conjunto No 123/97 de16/6/1997. In Diário da Replública II serie do7/71997. Lisbon. The 1998 version is also usedin this study. The official Ministries summariesfrom 1997 and 1998.

Montavavao e Silva, J.; Santos Silva, A;Prostes da Fonseca, J. M. Avaliacao do sistemadas escolas profissionais. Lisbon: Ministry ofEducation, 1996.

National Agency for Education. Reformeringen avgymnasieskolan – en sammanfattande analys.Stockholm: National Agency for Education,2000 (Report No 187).

Neubert, R.; Steinborn, H-C. Personalqualifizierungin den nuen Bundeslandern, Prozessanalyseund Evaluierung (PQO). Bonn: BIBB, 1999.

Nielsen, S. P. Vocational education and training inDenmark. Introductory note prepared at therequest of the European Centre for the Devel-opment of Vocational Training (Cedefop), 1997.Available from Internet: http://www2.training-village.gr/etv/vetsystems/report/dk_en.asp[cited 4/9/2003].

Nieuwenhuis, L.; Shapiro, H. Evaluating systems’change policies in VET: learning from Danishand Dutch cases. In: Descy, P.; Tessaring, M.(eds) Third research report: background report.Luxembourg: Office for Official Publications ofthe European Communities, 2003 (CedefopReference series).

Nijhof, W. J. (ed.) Evaluation of the Dutch VETsystem. Den Bosch: CINOP (unpublishedcollection of articles)

Plewis, I. Evaluating the benefits of lifelong learning.London: University of Education, Institute ofEducation, 2001.

Polder, K. Evaluation of an act: a case study ofthe Netherlands. Amsterdam: University ofAmsterdam, 2001.

Richini, P. Lifelong learning in Italy: the extent towhich vocational education and training policyis nurturing lifelong learning in Italy. Luxem-bourg: Office for Official Publications of theEuropean Communities, 2002. (CedefopPanorama, 43).

Shapiro, H. et al. Erferinger fra forsog med eudrefoemen – grundforlob og hovedforlob.Copenhagen: Danish Ministry of Education,2001.

Stern, E. Philosophies and types of evaluationresearch. In: Descy, P.; Tessaring, M. (eds) Thirdresearch report: background report. Luxem-bourg: Office for Official Publications of theEuropean Communities, 2003 (Cedefop Refer-ence series).

Summa, H. The role of evaluation in European Poli-tics. Internationales Jahrbuch der Erwachse-nenbildung, 2000, p. 49.

Tessaring, M. Training for a changing society: areport on current vocational education andtraining research in Europe. Second edition.Luxembourg: Office for Official Publications ofthe European Communities, 1999 (ReferenceDocument).


Thireau, V. Ouverture de l’accès au baccalauréat etinsertion des jeunes: enseignements et pistesouvertes par Génération 92. Marseille: Céreq,2001.

VFA – Valter Fissamer and Associates Ltd (for GreekMinistry of Labour and Social Affairs). Evalua-tion of the labour-market policies and assess-ment of the influence of European EmploymentStrategy in Greece during the period1997-2001– Final report. Athens: VFA, 2002.

Wilson, R. A.; Briscoe, G. The impact of humancapital on economic growth: a review. In:Descy, P.; Tessaring, M. (eds) Third researchreport: background report. Luxembourg: Officefor Official Publications of the European

Communities, 2003 (Cedefop Referenceseries).

Wimmer, P. Die ersten Fachhochschul-Absol-ventinnen am Arbeitsmarkt. Vienna: Institut fuerBildungsforschung der Wirtschaft-Schriften-reihe/Forschungsbericht, 2000.

WorldBank. Active labor market programs: impact eval-uation techniques, 2002. Available from Internet:http://wbln0018.worldbank.org/HDNet/hddocs.nsf/0/02BF4C7B59DE32C985256BF30067A80F?OpenDocument [cited 3/9/2003].

WRC Social and economic consultants Ltd. Anevaluation of the vocational training opportuni-ties scheme. Dublin: Department of Education,1994.


evaluating the impact of reforms of vocational education ... · evaluating the impact of reforms of...

Documents