budapest, 22 september 2015 martina lubyová miroslav...

Training of Unemployed Impact Evaluation

Budapest, 22 September 2015

Martina Lubyová Miroslav Štefánik Martin Obuch

ÚRAD VLÁDY SR

sekcia centrálny koordinačný orgán

Námestie slobody 1, 813 70 Bratislava


Subject of evaluation:Training of unemployed (co-financed from ESF) in period

2007-2013

Main evaluation questions:1. What is the net effect of the intervention?

2. Are there statistical differences in effects on various

groups?

3. What is the effect of the intervention on the state

budget?

4. What is an average unit cost for training of unemployed

resulting employment?

5. How does the intervention function?

6. Which factors influence the success/failure of the

intervention?


Methods to be applied:• Counterfactual Impact Evaluation (CIE)

• Cost-Benefit Analysis (CBA)

• Theory-based Impact Evaluation (TBIE)

Duration:• 05/2014 – 06/2015

Why another evaluation of training of

unemployed?• There were few attempts to assess the impacts of ESF

financed active labour market measures (using CIE

methods)

• It was only about the evaluation of intervention, but

equally important was the learning effect


Subject of the evaluation:• „Classic“ active labour market measure to address

structural imbalances on the labour market

• One of the traditional measures implemented by public

employment services in Slovakia

• It has been one of the active labour market measures

with the highest number of participants till 2009

• In contrast with the above mentioned, very little

information on the measure available to the public

• Despite the repeated recommendations from the

Commission, effectiveness and efficiency of the active

labour market measures have never been rigorously

assessed


2004-2006

03/2006– 06/2009

NP SPD 3a NP SOP 3a

Bratislava

2007-2013

01/2010 –06/2014

NP III2a NP III2b

All regions except Bratislava Other regions

Period

Delivery

Projects

Eligibility

Ministry of Labour, Social Affairs and Family

Central Office of Labour, Social Affairs and Family

46 regional offices covering the whole country

Legislation on PES

Management of PES

Delivery of PES


Delivery of the intervention

registration

application

assessment

acceptance refusal

selection

participation employment unemployment


Main challenges for the evaluation:• Involvement and cooperation with the institutions

responsible for active labour market policy

• Reconstruction of the intervention logic of the training

unemployed

• Internal and external validity of the evaluation design

• Data availability and data quality

• Protection of personal data

• Merging two independent databases managed by 2

independent institutions

Data sources and availability of data:

• Contrafactual impact evaluation was done on

administrative data:

• Database on registered unemployed was provided by the

Central Office of Labour, Social Affairs and Family

• Individual characteristics, participation in ALMM,

unemployment history

• Database of Social Security clients was provided by the Social

Insurance Agency

• Income, Employment status, Employment history• Outcome indicator 1 – income of participants of the training

• Outcome indicator 2 – labour market status of participants of

the training

Description of the database

Year

Unemploym

ent

registration

s

Declared

number of

participations

Participants in the

database by year of

participation

Participants with

information from

the Social

security

database

Share

observation

s lost due

to data

quality

2007 265 236 6 605 4 546 4 026 31,17%

2008 266 074 9 616 8 463 8 285 11,99%

2009 391 208 16 229 15 614 15 541 3,79%

2010 330 051 8 254 7 949 7 935 3,70%

2011 340 484 1 352 1 313 1 312 2,88%

2012 325 094 1 523 1 467 1 465 3,68%

2013 276 877 1 316 1 270 1 270 3,50%

Total 2 745 581 44 895 40 622 39 834 9,52%

Breaks in implementation of the measure 2007-2013

Propensity score variable

• I- Participation in the training(0,1)

• X- vector of observed characteristics (all information available from

the database):

– Individual characteristics (gender, age, region, level and field of

education, ...)

– Previous participation in other ALMM

– Pre-treatment unemployment (date of entering, length and no. of

previous unemployments, ...)

– Previous working experiences (days of previous working experience,

economic sector and occupation, ...)

– Family background (kids, marital status, ...)

– Declared skills (PC skills, languages, ...)

XXiIi20

)|1Pr(log

Probit model to predict the propensity score variable (PSV)

Distribution of the PSV before matching

0.2

.4.6

.81

0 1

psva

r

Graphs by p46

PSV – Balance

01

23

0 .5 1 0 .5 1

0 1

Den

sity

Pr(p46)Graphs by p46

N 1 758 123

Log likelihood 181 862,2

Prob > chi2 0,0000

Pseudo R2 0,5574

Sensitivity 28,24%

Specificity 99,75%

Positive predictive value 68,54%

Negative predictive value 98,65%

Correctly classified 98,42%

Description of the model 1 – PSM (matching with the

nearest neighbour with replacement)

• 1:1 matching was applied

• NN was applied within the region of participant (exact

matching on region)

• In 18.3% of control cases replacement was applied

• Matching on PSV and start of unemployment

• Less balance with more observations (practically all

participants were matched and included)

Probit model diagnostics Value

N 1 758 123

Correctly classified 98,42%

Pseudo R2 0,5574

Balance- Model 1

Control group Participants Before matching Index of improvement

Average

Start of unemployment 7.1.09 20.1.09 2.9.10 97,80%

Unemployment duration 511,36 530,02 312,59 91,42%

Age 38,13 37,82 34,97 89,16%

Propensity score 0,27 0,28 0,01 99,23%

Share in %

Male 45,22 47,97 54,12 55,28%

Under30 28,53 29,3 43,11 94,42%

30-49 50,38 50,41 39,66 99,72%

50+ 21,09 20,3 17,24 74,18%

Elementary education 28,53 29,3 43,11 94,42%

Secondary education 50,38 50,41 39,66 99,72%

Tertiary education 21,09 20,3 17,24 74,18%

Speaks foreign

language75,85 76,02 66,19 98,27%

Hungarian 8,54 8,24 9,36 73,21%

In the last employment

ISCO 1 3,14 2,8 1,58 72,13%

ISCO 2 4,91 4,42 3,19 60,16%

ISCO 3 13,81 13,11 7,63 87,23%

ISCO 4 7,55 7,31 4,72 90,73%

ISCO 5 13,76 13,63 11,7 93,26%

ISCO 6 0,63 0,63 0,92 100,00%

ISCO 7 14,54 15,33 13,2 62,91%

ISCO 9 15,3 14,79 17,22 79,01%

N N 32 649 39 834 2 364 453

Description of the model 2 – PSM (matching method -

caliper radius)

• A strict radius was applied (0.00075)

• Only 21 288 participants were included, excluding 46.6%

of participants because there was no match for them

within the radius

• 172 340 non participants were selected into the control

group

• Weighting was applied based on the distance to

participant

• Better balance in cost of loosing observations of

participants

Balance - Model 2

Control group ParticipantsBefore

matching

Index of

improvem

ent

Average

Start of unemployment 10.10.2008 16.11.08 2.9.10 94,35%

Unemployment duration 451,1125 568,1212 312,59 54,21%

Age 37,9147 38,07496 34,97 94,84%

Propensity score 0,3017917 0,304841 0,01 98,96%

Share in %

Male 45,75% 44,83% 54,12 99,98%

Under30 31,57% 29,60% 43,11 99,95%

30-49 57,84% 60,95% 39,66 99,92%

50+ 10,60% 9,45% 17,24 99,93%

Elementary education 20,77% 20,69% 43,11 100,00%

Secondary education 69,48% 69,42% 39,66 100,00%

Tertiary education 9,75% 9,89% 17,24 99,99%

Speaks foreign language 76,05% 76,00% 66,19 100,00%

Hungarian 8,79% 8,75% 9,36 100,00%

In the last employment

ISCO 1 2,18% 2,36% 1,58 99,88%

ISCO 2 4,07% 3,98% 3,19 99,97%

ISCO 3 11,68% 12,18% 7,63 99,93%

ISCO 4 7,26% 7,14% 4,72 99,97%

ISCO 5 13,60% 13,99% 11,7 99,97%

ISCO 6 0,68% 0,68% 0,92 100,00%

ISCO 7 14,95% 14,76% 13,2 99,99%

ISCO 9 16,35% 15,99% 17,22 99,98%

N N 20 690 20 690 2 364 453

Description of the model 3 (regression analysis):

XIY210

IIndicator Value

Number of obs 1 669 653

R-squared 0,1715

Adj R-squared 0,1713

Root MSE 372,93

• I- Participation in the training(0,1)

• X- vector of individual characteristics (all information available from the

database):

– Duration of unemployment (date of entering, length of the evidence, ...)

– Individual characteristics (gender, age, region, level and field of education,

...)

– Previous participation in other ALMM

– Previous working experiences (days of previous working experience,

economic sector and occupation, ...)

– Family background (kids, marital status, ...)

– Declared skills (PC skills, languages, ...)

Estimated coefficients by period of implementation

Model 1 Model 2 Model 3

Month ATT S.E. p. N ATT S.E. p. N β S.E. p. N

-6 36,15 22,92 0,115 6150 21,63 33,17 0,514 1917 18,48 18,22 0,311 148689

6 50,62 6,03 0,000 8104 7,07 6,97 0,311 1917 -36,99 6,54 0,000 333284

12 52,61 7,32 0,000 8104 18,94 8,85 0,033 1917 -1,36 6,99 0,846 333284

24 68,85 7,41 0,000 8104 1,87 9,20 0,839 1917 21,79 8,19 0,008 333284


Mesiac ATT S.E. p. N ATT S.E. p. N Β S.E. p. N

-6 22,49 29,78 0,450 4725 41,66 26,20 0,112 1606 12,94 22,06 0,558 62651

6 -21,05 8,89 0,018 4725 -28,56 7,90 0,000 1606 -26,00 7,06 0,000 62651

12 17,95 8,50 0,035 4725 -36,26 8,79 0,000 1606 9,37 7,51 0,212 62651

24 6,86 10,00 0,493 4725 -55,79 9,69 0,000 1606 2,61 8,33 0,754 62651

2007/01-2008/04

2008/05-2008/08


Model1 Model 2 Model 3

Month ATT S.E. p. N ATT S.E. p. N ATT S.E. p. N

-6 -18,72 8,88 0,035 22816 -45,92 7,50 0,000 7753 -40,25 6,20 0,000 439698

6 -91,11 3,83 0,000 22816 -63,56 3,26 0,000 7753 -56,22 3,13 0,000 439698

12 -92,50 4,39 0,000 22816 -41,75 3,90 0,000 7753 -53,26 3,51 0,000 439698

24 -79,15 5,71 0,000 22816 -16,42 4,91 0,001 7753 -41,27 4,71 0,000 439698


Month ATT S.E. p. N ATT S.E. p. N ATT S.E. p. N

-6 -23,82 5,08 0,000 23553 12,20 5,52 0,027 8672 -34,19 8,79 0,000 203136

6 -31,66 3,53 0,000 23553 -53,02 3,00 0,000 8672 -49,30 3,25 0,000 203136

12 -23,59 3,81 0,000 23553 -50,70 3,44 0,000 8672 -31,54 3,41 0,000 203136

24 -42,31 4,57 0,000 23553 -36,07 4,27 0,000 8672 -24,50 4,13 0,000 203136

2008/09-2009/08

2009/08-2010/12

ATT on income of participants (Model 1)

ATT on employment of participants (Model 1)

Main findings – summary table

Period/sub-group Probability to find a

job

Income

Slovakia + -

2007/01-2008/04 + +

2008/05-2008/08 0 0

2008/09-2009/07 0 -

2009/08-2010/12 0 -

2011/01-2011/12 - 0

2012/01-2012/12[1] - -

2013/01-2013/12[2] 0 0

Male - -

Female 0 -

Primary education + -

Secondary education 0 -

Tertiary education 0 0

< 30 0 -

30-54 + -

55 + + 0

Bratislava - +

Malacky 0 0

Pezinok + +

Dunajská Streda - -


Results:

• Results of 3 models applied are relatively consistent

• Different impacts on participants in pre-crisis period, during the crisis and after the crisis

• Positive effects are dominant in the pre-crisis period

• Significant negative effects on earnings as well as employment are observable in the post-crisis period

• On average for the period 2007-2013 the intervention had negative impact on earnings and employment of participants

• Different impacts on specific groups of participants is observable

Implications on the public finance (CBA):

Defining the scenarios

Positive scenario

(2007/01-2008/04)

Negative scenario

(2012/01-2012/12)

Realistic scenario

(2007/01-2013/12)

Additional

employment

Employmen

t of

participants

Additional

income

Additional

employment

Employmen

t of

participants

Additional

income

Additional

employment

Employmen

t of

participants

Additional

income

Year 1 4,92% 45,75% 29,55 -17,85% 35,28% -130,16 -7,75% 40,23% -53,12

Year 2 12,00% 61,10% 64,12 -7,63% 40,77% -97,52 -3,14% 54,60% -36,67

Year 3+ 14,00% 65,95% 68,85 0,00% 40,77% 0,00 0,00% 58,38% -35,00

Implications on the public finance (CBA), results

-40,000,000

-30,000,000

-20,000,000

-10,000,000

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Positive scenario Negative scenario Realistic scenario


Theory-based impact evaluation:• Identification of underlying factors determining how the

intervention functions

• Bring a light into the „black box“

Pri

ncip

les f

or

train

ing

of

un

em

plo

ye

d

Legislatíon, rules,

capacities and theory of

intervention

Delivery of the

intervention to clients

Interaction between

provider and clients

TB

IE


Theory-based impact evaluation – key findings

General

Planning

• Radical reduction of number of participants in the measure

without any visible change in the policy

• Implementation guidelines put emphasis on administrative

issues, on the other hand some crucial areas are not covered

• Backward looking analysis based on historical data is the only

information for the planning of training (strong inertia)

• Content planning is not linked with budget allocation what

increases uncertainty


Flexibility

Relevance

• Annual planning cycle reduces the flexibility (content, money

and delivery)

• After 2009, the training for unemployed available only to

Bratislava region – not covered by ESF

• The measure is not designed for specific target group of

unemployed, does have a specific role in the policy

• Formally any type of training could be provided, but in reality

approximately 60% of all participants attended 10 most

popular courses

• Most frequently used training courses in all regions


Provision of information

Selection

• Information on the training of unemployed available to all

officially registered unemployed (no selectivity)

• The ways how the information is provided to a client actually

differ

• The key communication channel are employment agents

(intermediators)

• The first selection (assessment of appropriateness of the

measure) performed by advisors, but not functional)

• No unified approach and criteria to selection of participants

of training courses

• Planned courses must be delivered, so sufficient number of

participants needs to be ensured


Reflection on individual needs

Training providers

• Most clients do not have specific idea on the preffered training

and reflects on the offer of the employment office

• Group training courses have obvious limitations

• Limited number of „traditional“ training courses offered till

2009, then only Bratislava region

• No comprehensive assessment of training providers

• Till 2009 employment offices responsible for selection of

providers

• After 2010 central public procurement for training providers

was unsuccessful and the measure became not available


Conclusions:• The measure only into limited extent reflected on the

dynamics on the labour market (low responsiveness), it was driven mainly by the organisational and administrative factors

• On average, the intervention had negative impacts on probability of a participant to find a job and income compared to non-participants.

• The measure had negative net effects especially during the global crisis and immediately after the crisis (2009-2012) and mixed effects before the crisis.

• The age and educational level were important factors determining the effects of the intervention on participants.

• The are significant differences in effects of the intervention in Bratislava region and other regions of Slovakia.


Recommendations – policy level:• The measure is functional when there are jobs available on

the labour market that can be taken by the unemployed - it

does not solve the low aggregate demand

• The policy and the measure should reflect on the

developments on the labour market – the provision can be

reduced during the recession

• The group training should be targeted to the specific

groups of unemployed, on which it had the positive impacts

• There should be a targets introduced for activation of

unemployed and the training should become an important

part of it

• The quality and consistency of implementation can be

increased through common methodology to be applied by

the regional offices


Recommendations – operational level:• The effectiveness of the measure can improved by better

identification of the training needs through closer contact

with demand side (employers)

• Proper timing of the intervention – offer the training within

the certain period after the registration of unemployed to

act as a part of the activation mechanism

• Harmonise the planning cycle of training and budgeting,

introduce multiannual planning cycle

• Avoid central planning of the training courses and centrally

organised public procurement of providers


Recommendations – impact evaluation:• Ownership of the evaluation and its results

• Close cooperation with the institutions responsible for the

intervention

• Understanding of the intervention is crucial, it is not a

statistical exercise with a data (academic exercise)

• Have an access to the databases or at least to know the

structure of data before

• Processing of the data is time and resource consuming

• The basic principles of evaluation needs to be respected

• Policy relevance and policy reflection (is the policy and

policy makers ready to reflect on the findings?)

• To avoid that impact evaluation becomes fashionable trend

among academics vs. learning capacity of public sector


References:

http://www.nsrr.sk/download.php?FNAME=1411

462887.upl&ANAME=EVALUATION_REPORT_20

140630_.pdf

http://www.nsrr.sk/download.php?FNAME=1411462887.upl&ANAME=EVALUATION_REPORT_20140630_.pdf

budapest, 22 september 2015 martina lubyová miroslav...

Documents