Evaluating Retraining Programsin OECD Countries: Lessons Learned

Amit Dar • Indermit S. GUI

Are retraining programs for the unemployed more effective than job search assistance?Governments of the member countries of the Organisation for Economic Co-operationand Development have considerable experience with retraining programs in a variety ofindustrial settings. Evaluations of these programs show that the results are disappointing,however. This article discusses the factors associated with retraining programs for twotypes of workers: those laid off en masse and the long-term unemployed. Evaluations indi-cate poor results for both groups: retraining programs are generally no more effective thanjob search assistance in increasing either reemployment probabilities or postinterventionearnings, and they are between two and four times more expensive than job searchassistance.

Industrial countries spend sizable amounts on labor programs for the unemployed.In 1992 member countries of the Organisation for Economic Co-operation andDevelopment (OECD) spent between 0.1 percent (Japan) and 2.6 percent (Sweden)of gross domestic product on labor programs. In several countries this training is"the largest category of active programs, and is often perceived as the principal alter-native to regular unemployment benefits" (OECD 1994a). Countries in EasternEurope—where the role of active labor programs is a topic of current debate—alsospent between 0.2 and 3 percent of gross domestic product on these programs (OECD1994a). Yet possibly because such assistance is viewed almost as a fundamental rightin Western Europe, these programs are rarely evaluated outside the United States.This article surveys evaluations of retraining programs in the OECD countries, high-lighting the shortcomings of such schemes and illustrating the payoff to investing inrigorous evaluations.

We examine only the evaluations of retraining programs, so our focus is largely onadults with previous work experience rather than on unemployed school-leavers. Wedistinguish here between retraining schemes for workers displaced by plant closures

The World Bank Research Observer, voL 13, no. I (February 1998), pp. 79-101© 1998 The International Bank for Reconstruction and Development / THE WORLD BANK 19

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

or restructuring and programs aimed at the long-term unemployed. These groupsdiffer in important ways. First, the long-term unemployed are a relatively hetero-genous group of individuals compared with those laid off en masse from a singleplant or firm. Second, although some programs are targeted at specific regions, thelong-term unemployed are generally more dispersed geographically. Third, the du-ration of unemployment is, almost by definition, greater for the long-term unem-ployed. Finally, retraining programs for the long-term unemployed are generally amix of classroom and in-plant training, while programs for displaced workers areusually confined to the classroom.

Evaluation Techniques

Techniques for evaluating the effectiveness of retraining programs can be broadlyclassified into two categories: scientific and nonscientific. Scientific evaluations canbe further divided into experimental and quasi-experimental. Experimental, or clas-sically designed, evaluations require selecting "treatment" and "control" groups be-fore the intervention: the treatment group receives the assistance, and the controlgroup does not. If large numbers of individuals are randomly assigned to each group,the average characteristics of the two groups should not differ significantly, so anydifference in outcomes can be attributed to program participation. In quasi-experimental studies treatment and control groups are selected after the interven-tion. To compute the program's effects, statistical techniques are used to correct fordifferences in characteristics between the two groups. Nonscientific techniques donot use control groups to evaluate the effect of interventions but instead rely onstatistics compiled by a program's administrators. These evaluations are of little use;without a control group, it is difficult to attribute the success or failure of the partici-pants to the intervention, because the changes in individuals' behavior might haveresulted from other factors, such as worker-specific attributes or economywide changes(Grossman 1994).

Classically Designed (Randomized) Experiments

This technique, which was originally developed to test drug effectiveness, identifiesand randomly assigns individuals to either a treatment or a control group before theintervention. Its main appeal lies in the simplicity of interpreting results—the effec-tiveness of the program is computed as the simple difference in die outcome betweenthe participants in the treatment group and the nonparticipants in the control group(Newman, Rawlings, and Gerder 1994). The main pitfalls of this method are a failureto select individuals through random assignment, changes in behavior as a result oftheir assignment to either group (for instance, enrolling in private programs or intensi-

8 O The World Bank Research Observer, voL 13, no. 1 (February 1998)

fying their job search), high costs because of the number of participants in the sample,and ethical questions about excluding a group of people from the intervention.

Although randomization is thought to eliminate selection bias in determiningwho will participate, proponents of this methodology make an important assump-tion: that random assignment does not alter the behavior that is being studied. Thismay not be the case and, in fact, the bias may be quite strong (Heckman 1992). Forexample, ambitious individuals who would have taken a training course in any casewill not apply to the program if they stand a chance of being in the nontreatmentgroup. Such individuals, who might have enrolled in a nonrandomized regime, maymake plans anticipating enrollment in a training program. With randomization theymay alter their decision to apply or undertake activities complementary to training.Thus risk-averse persons will tend to be eliminated from the program.

Quasi-Experimental Techniques

In quasi-experimental methods the treatment and control groups are selected afterthe intervention. Econometric techniques are used to correct for the differences incharacteristics between the two groups. The main appeal of this method lies in itsrelatively low cost and the ability to undertake the evaluation at any time. The maindrawback is that these techniques—if done properly—are statistically complex. Thetechniques used to adjust for differences in observable attributes (for example, sex,age, region of residence, education) are relatively straightforward but subject to speci-fication errors; correcting for unobservable characteristics (for example, motivation,family connections) requires a convoluted procedure that can yield wildly differentresults. Quasi-experimental evaluations fall into three types: regression adjusted forobservable characteristics; regression adjusted for observable and unobservable char-acteristics (selectivity corrected); and matched pairs.

REGRESSION ADJUSTED FOR OBSERVED CHARACTERISTICS. This technique is used toassess the impact of participation in a program when the observable characteristics ofthe participants and the comparison group are different. This method is appropriatefor estimating the effect of a program when the difference between participants andnonparticipants can be explained by these observable characteristics. For example, ifbetter-educated workers are more successful in finding work regardless of whether ornot they had special training, then controlling for the effect of education (usingregression techniques) will provide more reliable estimates than would a simple com-parison of the reemployment probabilities of the control and treatment groups.

REGRESSION ADJUSTED FOR OBSERVED AND UNOBSERVED VARIABLES (SELECTIVITY

CORRECTED). When selection into programs is not random, and participation in aprogram is due to both observable and unobservable characteristics, the above tech-

Amit Dar and Indermit S. Gill 8 1

nique, which corrects for observed characteristics, is likely to be biased. Even if par-ticipants and nonparticipants have similar observable characteristics, unobservablecharacteristics (such as innate ability) would lead to incorrect inferences about non-participants. This technique uses a method developed by Heckman (1979) called"sample selectivity correction" to try to control for these unobservables.

MATCHED PAIRS. Because the observed characteristics of individuals in the controland treatment groups are bound to be different to some degree, these groups arelikely to have different success rates in finding employment even in the absence ofactive labor market programs. To control for these differences, synthetic controlgroups are constructed using a matched pairs approach. The synthetic control group,which is a subset of the entire control group, is composed of individuals whose ob-servable characteristics most closely match those of the treatment group.

Relative Strengths of Techniques

Estimating the effect of an employment program on the earnings of trainees, usingrandomized and quasi-experimental techniques, LaLonde (1986) found that ran-domized experiments yielded results significantly different from those that relied onquasi-experimental techniques. Policymakers should be aware that availablenonexperimental evaluations of training programs may contain large biases. Whilerandomized experimentation is theoretically the best technique to estimate the ef-fects of interventions, quasi-experimental techniques may be superior in practice.

The main weakness of randomized experiments is their inability to ensure thatindividuals in the control group do not alter their behavior in a way that contami-nates the experiment. For example, individuals denied public job training mightenroll in private programs, which would bias the results of any evaluation of publicprograms.1 It may also be difficult to ensure that assignment is truly random. Forexample, applicants may be selected into the program because of nepotism, or pro-gram administrators may deliberately exclude high-risk applicants to achieve resultsthat reflect well on the program. A third problem concerns ethical questions abouttreating individuals as subjects in an experiment. Finally, experimental evaluationsare possible only for future programs, because the control and treatment groups haveto be selected before the program is initiated.

Using the dual criteria of rigor and feasibility then, randomized experiments arenot necessarily superior to quasi-experimental techniques. Because the decision toevaluate labor market programs often occurs after the programs are in place andbecause the costs of setting up the experiments are high, randomized evaluationsshould perhaps be the last alternative. Within quasi-experimental techniques, selec-tivity correction may not add much, especially when information is available for aconsiderable number of observable individual and labor-market characteristics (such

82 The World Bank Research Observer, voL 13, no. 1 (February 1998)

as education, age, sex, household wealth, and region of residence). In addition tobeing cumbersome and somewhat counterintuitive, this method often gives arbi-trary results depending on the selectivity-correction specification that is used.

This leaves the matched pairs and regression-adjusted techniques. Of the two, thematched pairs technique is preferred because the procedure is less arbitrary. Becausethe observed differences between the treatment and comparison group are mini-mized, the exact specification of the model becomes less important. And because theprogram measures the simple difference in the variables that policymakers want an-swered (reemployment probabilities and wages) between the control and treatmentgroups, the results are easier for nonstatisticians to interpret.

One weakness shared by both the scientific and nonscientific evaluations is thatthey do not take into account the displacement that may result from the retrainingprogram. For example, in countries where demand for labor is constrained, retrain-ees may "bump," or displace, employed workers, so aggregate unemployment maynot change despite the size of the program. In general, displacement implies that thesocial benefits from reemployment attributable to the retraining program are lowerthan indicated by the evaluation, however well done.

The Importance of Costs

For the purposes of informing policy decisions, an evaluation is not complete untilthe costs of both the retraining program and its alternatives are considered. For ex-ample, if retraining is twice as costly as job search assistance but no more effective infinding people jobs and increasing their wages, job search assistance is twice as cost-effective. At least at the margin, such a finding would constitute a case for reallocat-ing resources from retraining to job search programs. Unfortunately, costs appear tobe the least analyzed aspect of these programs in OECD countries.

Even the most careful evaluations of retraining programs cannot be used for socialcost-benefit analysis because of the displacement effects discussed above. But whendone correctly, evaluations are good guides for cost-benefit analysis of private train-ing programs, which policymakers can use to institute cost recovery in public pro-grams and to promote private provision. Evaluations may also help in deciding whetherretraining programs reduce budgetary expenditures by moving people off unem-ployment benefits into productive employment or whether the programs are a drainon the budget despite being effective.

Evaluating Retraining in OECD Countries

Retraining programs in OECD countries have been designed primarily to assist threecategories of workers: those laid off en masse; those who have lost their jobs because

Amit Dar and Indermit S. Gill 83

of plant closures; and those who have been unemployed over the long term. Thisreview of the evaluations of eleven retraining programs classifies the specific situa-tion that the program was designed to address, describes the policy intervention thatwas selected and the type of evaluation used, and reports the main results. Becausethe success of retraining programs depends on aggregate or regional labor marketconditions, such as unemployment rates and the state of the leading industry, theseindicators are reported as well.

We examine the type of retraining provided, whether in classrooms or on-the-job,and whether it was accompanied by—or in lieu of—other measures, such as jobsearch assistance. The evaluations are classified as experimental, quasi-experimental,and nonscientific, and we do not consider inferences drawn from the nonscientificevaluations to be reliable. We then look at the effects of the program on reemploy-ment and wages, both for subgroups of retrainees and type of intervention. The costsof the program are included when they are reported.

Training Programs Instituted as a Result of Mass Layoffsand Plant Closures

The results of eleven programs (three in the United States, four in Sweden, andone each in Australia, Canada, Denmark, and France) examining the effectivenessof retraining programs for workers displaced through mass layoffs and plantclosures have been reviewed (Table A.I). Five of the evaluations were nonscientificand five were quasi-experimental. One study relied on more than one techniqueto evaluate the impact of the program; no study used experimental evaluationtechniques.

The retraining programs were undertaken to assist workers in the steel, pulp, min-ing, shipbuilding, and automotive industries. The number of workers who lost theirjobs varied from about 500 to 3,000 per plant. The rationale underlying the pro-grams was apparently to assist the affected workers in any way possible. Generally,these programs were instituted during periods of high or rising aggregate unemploy-ment or during a contraction in certain manufacturing industries. For example, theevaluations in the United States and Canada covered primarily the auto and steelindustries, which were battered by competition from Japan. Between 1978 and 1980,auto production in the two countries declined 25 percent, precipitating layoffs andplant closures in the early 1980s. In Europe and Australia the retraining programsseem to have been instituted during periods of high or rising rates of unemployment.Most of the retraining programs were classroom based, and accompanied by jobsearch assistance. With only one exception, on-the-job training was not provided orfacilitated. In France retraining was accompanied by financial incentives to regionalfirms that hired trainees, so the full costs of the program were likely high (OECD1993a).

84 The World Bank Research Observer, vol. 13, no. 1 (February 1998)

Quasi-experimental and nonscientific techniques were used in evaluating all ofthese retraining programs. None of the studies was longitudinal, so they provide atbest a snapshot of the labor market benefits of the program. The longer-term ben-efits of retraining were not evaluated even by the scientific evaluations. Althoughnonscientific evaluations indicate that these programs are very effective in placinghigh numbers of male workers in wage-earning jobs or in self-employment and womenin business start-ups, more reliable quasi-experimental evaluations indicate that re-training programs are generally no more effective than job search assistance in in-creasing either reemployment probabilities or postintervention earnings. Some re-training programs resulted in modest gains in reemployment probabilities, but thewage changes were sometimes negative. Interestingly, evaluations of three retrainingprograms for U.S. auto workers showed the contrast between scientific and non-scientific techniques: in San Jose, California, a nonscientific evaluation indicatedhigh placement rates, while in Buffalo, New York, and Michigan—during the sameperiod—scientific evaluations showed that these programs were ineffective (OECD1993a; Corson, Long, and Maynard 1985; Leigh 1992).

The costs, when they are known, varied between $3,500 and $25,000 a person.Evaluations seldom report the full costs of retraining or job search programs, how-ever, so determining cost-effectiveness is difficult. Retraining programs appear to bebetween two and four times more expensive than job search programs: for example,in Buffalo, job search services cost $850 a participant, while retraining cost $3,300(Corson, Long, and Maynard 1985). If, as the findings indicate, both programs haveroughly the same success, job search assistance may be more cost-effective than re-training in assisting displaced workers get jobs.

Training Programs for the Long-Term Unemployed

There is no reason to assume that the impact of retraining on the long-term unem-ployed is the same as it is on workers laid off en masse. The results of studies exam-ining the effectiveness of retraining programs for the long-term unemployed are shownin Table A.2. Of the eleven evaluations (four in the United States, three in Germany,two in the Netherlands, and one each in Canada and Britain), four were nonscien-tific, four were quasi-experimental, and three were experimental.

The clientele of retraining programs for the long-term unemployed is relativelyheterogeneous. Because these individuals are displaced from various sectors and somehave never worked, they are likely to be more varied in age, skills, and educationthan laid-off workers. New Jersey's retraining program in 1986-87 included work-ers whose previous jobs were in manufacturing, trade, and services (Anderson, Corson,and Decker 1991); many were more than 55 years old. In contrast, half the workersin Germany's retraining program for the long-term unemployed were less than 35years old (Johanson 1994).

A mit Dar and Indermit S. Gill 8 5

These programs are generally instituted during improving conditions in industryor in aggregate employment. Interventions at this stage, if appropriately designed,are thought to enable the long-term unemployed to obtain some of the jobs that arebeing created. The programs are more comprehensive than are those for people laidoff en masse and generally provide a mix of classroom or workshop training, on-the-job training, and job search assistance.

Some experimental evaluations were conducted in the United States, and quasi-experimental techniques were used in both the United States and Europe. Some ofthese studies were longitudinal, providing an indication of the medium-term labormarket benefits of the retraining program (see Johanson 1994 and OECD 1993b forEurope; and Leigh 1992, and Anderson, Corson, and Decker 1991 for the UnitedStates). But many of the evaluations in Europe were nonscientific.

Again, the results of nonscientific evaluations, which were encouraging, were notconfirmed by evaluations based on scientific techniques. The effectiveness of pro-grams for the long-term unemployed, while not high, was better than that of pro-grams for those laid off en masse. A few programs did result, or were thought to haveresulted, in gains in either reemployment probabilities or wages; some evaluationsalso indicated that these programs were more effective in helping women. But whereparticipants did record gains in employment, longitudinal studies generally foundthat the effects of retraining dissipated within a couple of years. In this group, too,retraining programs were generally no more effective than job search assistance inincreasing either reemployment probabilities or postintervention earnings. For ex-ample, evaluations of the Texas Worker Adjustment Demonstration program indi-cated that participants were likely to be employed more rapidly than nonpartici-pants, but that over time, the employment opportunities of male participants wereno better than those of nonparticipants or of those who only had job search assis-tance (Bloom 1990). The costs, when known, varied between $900 and $12,000 aperson, about twice as much as job search services, but the lack of data makes itdifficult to determine the absolute cost-effectiveness of these programs. Still, jobsearch assistance programs appear to be somewhat more cost-effective than retrain-ing programs in finding jobs for the long-term unemployed.

Evaluation Results for Hungary

Rising unemployment and falling real wages are a vivid and costly aspect of countriesin transition from controlled to market economies. Long-run unemployment is par-ticularly pernicious, and many countries have mounted active labor programs (forinstance, public retraining programs and public service employment) to deal withthis problem. With the exception of Hungary, where the World Bank has sponsoredrigorous evaluations and collections of data on costs, little reliable information exists

86 The World Bank Research Observer, voL 13, no. 1 (February 1998)

on their effectiveness. As an early reformer, Hungary provides valuable lessons forother transition economies. Since 1989 sharp declines in the country's gross domes-tic product have been accompanied by rising unemployment and falling real wages.The unemployment rate appears to have stabilized since 1993, but employmentcontinues to decline, reflecting a continuing withdrawal of workers from the formallabor market. Since 1990 the government has offered workers retraining programsfinanced by a national employment fund. Trainees, who are either currently unem-ployed or are working but expect to become unemployed, include participants inpublic works programs. Instruction is largely classroom based.

Evaluations using different quasi-experimental techniques—matched pairs, regres-sion adjusted, and selectivity corrected—yielded different results (O'Leary 1995).Before adjusting for differences in the characteristics of program participants andcontrol groups, estimates of effectiveness indicated that retraining significantly raisedthe probability of reemployment. But when the observable characteristics of treat-ment and control groups were taken into account, retraining (and training, becauseabout 40 percent of trainees had not worked previously) was only marginally suc-cessful at best, increasing the probability of finding employment by 6 percent. Fur-ther controls for unobservable attributes led to ambiguous results. Retraining wasnot at all successful in raising earnings.

Preliminary analysis indicates that retraining was a substitute for attributes thatlead to higher reemployment probabilities in the absence of any intervention, such asbeing younger, better educated, and from more dynamic regions. That is, theprogram's value-added (in terms of improving labor market outcomes) is greater forrelatively disadvantaged job-seekers. This finding may be country specific, and othercountries or regions should determine whether it applies to them before implement-ing similar large-scale programs. For the programs evaluated in Hungary, focusingmore on job-seekers who lack these attributes would appear to serve both equity andefficiency objectives better than simply ensuring support for programs whose re-trainees have high probabilities of reemployment and gains in earnings. The analysisimplies that public retraining programs should target older men from backwardregions.2

This finding also highlights the usefulness of rigorous impact evaluations, whichnet out the effects of such attributes in determining whether a program is effective.And it underscores the need to agree upon a reliable, feasible, and easily interpretedtechnique to evaluate the efficiency and equity effects of labor programs. Because ofits analytical rigor and feasibility, we argue that the preferred evaluation technique ismatched pairs analysis, where trainees are compared with a subset of the controlgroup whose characteristics most resemble their own.

Private cost-benefit analysis of the costs of retraining (an average of approximately$900 per trainee in 1994, according to Pulay 1995) and the level of gains in reem-ployment reveal that based on reasonable assumptions about the durability of

Amit Dar andIndermit S. Gill 8 7

Table 1. Summary of OECD Retraining Programs and Evaluations

Item

Previous sector of employment

Labor market conditionsTraining venue

Other measures providedType of evaluation

NonscientificQuasi-experimentalExperimental

EffectivenessNonscientific techniquesScientific techniques

Relative to JSA

CostsRetrainingRelative to JSA

Note: JSA, job search assistance; OJT,

Displacedworkers

ManufacturingGenerally deteriorating

Mainly classroom

JSA

650

PositiveNegative

No more effective

Data are not availableAt least twice as costly

on-the-job training.

Long-termunemployed

Heterogeneous

Generally improvingClassroom and OJT

JSA

443 (all U.S.)

Generally positiveGenerally negative;

some groups benefitNo more effective

Data are not availableAt least twice as costly

program effects and the amount and duration of unemployment benefits, it wouldtake more than 30 years to recover costs of the program (Gill and Dar 1995). Itseems difficult to justify retraining programs based on economic considerations alone,even in a country such as Hungary where the government provides benefits for un-employed workers.

Summary and Conclusion

The paucity of rigorous evidence on the costs and effectiveness of retraining pro-grams does not permit a definitive conclusion on whether such interventions can bejustified economically (Table 1). The scattered evidence does not appear to justifythe indiscriminate expansion of retraining programs to cover more of the unem-ployed. These conclusions are consistent with the findings in the OECD EmploymentOutlook (1993b), which concluded that "for the broadly targeted sub-group of pro-grams, the overall impression is most troubling. Available evidence does not permitstrong conclusions, but it gives remarkably meager support of a hypothesis that suchprograms are effective."

From this discussion it appears that transition and other restructuring economiescan draw the following lessons from experience in the OECD countries:

8 8 The World Bank Research Observer, voL 13, no. I (February 1998)

First, sound techniques should be used to evaluate retraining programs (and otherpublic interventions). Although the nonscientific evaluations of retraining programspresent a rosy picture based on placement rates and other informal evidence, scien-tific evaluations are quite discouraging. Relying on nonscientific evaluations maylead to incorrect policy conclusions.

Second, rigorous evaluations, although not necessarily allowing a complete socialcost-benefit analysis, can be useful for policymakers in allocating public expenditureon labor programs. Reviews of evaluations find, for example, that job search assis-tance measures—which cost less than retraining but appear equally effective—maybe a more cost-effective device in assisting displaced workers.

Third, OECD experience of retraining programs for workers displaced en massemay be useful in designing assistance programs in transition countries such as theformer Soviet Union and liberalizing economies that expect labor shedding in themanufacturing and mining sectors. Principally, these economies should recognizethat retraining should not be the main form of assistance.

Fourth, transition and developing economies that are beginning to experiencelong-term unemployment can learn from OECD experience, which indicates that re-training programs for the long-term unemployed are more beneficial for some groupsthan for others within this relatively heterogeneous group of job-seekers. Whichgroup will benefit most from retraining is difficult to predict, however. Principally,these results call for using modest untargeted pilot programs, evaluating them rigor-ously, and then tightly targeting public retraining programs to those for whom theyare found most cost-effective.

Notes

Amit Dar is in the Social Protection Group of the World Bank's Human Development Network,and Indermit S. Gill is senior economist in the World Bank's Country Management Unit in Brasilia.The authors are grateful to Jane Armitage, Kathie Krumm, Ana Revenga, Michal Rutkowski, andanonymous referees for helpful suggestions.

1. Heckman (1992) documents other limitations of this technique when applied to social experi-ments; these limitations derive from selectivity biases.

2. These findings contrast with those of Revenga, Riboud, and Tan (1994) for retraining pro-grams in Mexico. Using quasi-experimental techniques, they find that effectiveness can be improvedif programs better target relatively educated job-seekers of both sexes with previous work experience.They also find generally more encouraging results than those for OECD countries and Hungary.

AmitDarandIruUrmitS.dll 89

o Table A. 1. OverviewLabor market problem

Ford Motor Co. plantclosure in San Jose\Calif. (1982). Some2,400 workers affected(OECD 1993a).

Automotive plantclosures in Michigan(1980-83). More than3,000 workers laid off,mosdy experienced,highly paid blue-collarmale employees (Leigh1992; 1994).

Auto and steel plantclosures in Buffalo,N.Y. (1982-83). Masslayoffs of highly paid,experienced blue-collarmale workers (Corson,

of Studies Evaluating Training Programs for Workers Displaced by Plant Closures and Mass Layofji

Relevant indicators

A 25 percent decline inauto production be-tween 1978 and 1980.

Unemployment ratesrose from 7.5 percentin 1981 to 9.5 percentin 1982 and 1983, andmanufacturing employ-ment declined by 5percent during 1981-83.

A 25 percent decline inauto production be-tween 1978 and 1980.High unemploymentrates in 1981-82 na-tionally (9.5 percent),which fell to 7.5 per-cent by 1984. Manu-facturing employmentrose by 5 percent be-tween 1983 and 1984.

High unemploymentrates in 1981-82 na-tionally (9.5 percent),which fell to 7.5 per-cent by 1984.Manufacturing

Intervention design Type of evaluation

Basic skills training as Nonscientificwell as targeted voca-tional training in mar-ketable skills.

Both JSA and classroom Quasi-training were provided experimentalprompdy after plantclosures. Retraining wasoffered in occupationsin which there wasgrowing demand,including blue-collartrades.

Displaced workers Quasi-were provided with experimentalboth JSA and eitherclassroom training orOJT. Program serviceswere provided after a

Result

High rate of success inplacing workers injobs.

Classroom training did

not significantly im-

prove the participants '

reemployment rate.

Trainees did no better

than those receiving

JSA.

JSA services had a fairly

large impact on earn-

ings for the first six

post-program months .

However, there is no

evidence that either

Comments

Perceived success wasdue to adequate re-source base ($6,000grant per worker)and a high degreeof coordinationprovided by Ford andby government.

Earnings estimates vary(ranging from negativeto significantly posi-tive). Training doesnot seem to have beenvery effective, espe-cially in light of thefact that training costtwice as much as JSA.

Classroom training andOJT were ineffective.These programs costabout four times asmuch as JSA, implyingthat JSA is potentially

Long, and Maynard

1985).

Plant closures at 13steel factories andmines in Canada in the1980s (Leigh 1992).

Steel and coal plantclosures in Creusot-Loire, France (1984).1,230 people affected(OECD 1993a).

employment rose by 5percent between 1983and 1984.

Unemployment ratesrose between 1980 and1983. Employment innonagricultural activi-ties fell slighdy duringthe period.

Steel sector contractionresulted in the loss of6,000 jobs during1984. Unemploymentrates rose from 8.1percent in 1982 to10.2 percent by 1985.In mining employmentfell by almost 40 per-cent between 1980 and1990. Manufacturingemployment fell about6 percent from 1983 to

1985.

fairly lengthy period ofunemployment.

JSA and training pro-grams were provided.

Workers received 70percent of dieir formersalaries for 10 monthswhile enrolled in re-training and job-searchactivities. Trainees (inengineering, plasticmolding, refrigeration,etc.) were promisedreemployment in theregion. Career counsel-ing and short courses inwork skills, productionmethods for small firms,

Quasi-experimental

Nonscientific

classroom training orOJT had any incremen-tal effect.

Participants had a 7percent higher rate ofemployment thancomparably displacedworkers who did notparticipate in the pro-gram. This impact wasattributed to training inwhich 28 percent ofworkers participated.But at two mining sites,the program had noimpact.

High success rate infinding jobs for partici-pants.

the only cost-effectiveprogram. No evidencewas provided on theparticipants' employ-ability.

Training seems to havehad a greater impactthan JSA. No informa-tion was available onthe cost of training. Jobcounseling has littleimpact in Canada,where such services areroutinely provided tothe unemployed by thePublic EmploymentService.

No evidence that re-training programsproduced long-termemployment benefits.

(Table continues on the following pages.)

Table A. 1. (continued)Labor market problem

Pulp plant closure inKramfors, Sweden(1977) (OECD 1991).

About 2,000 workerslaid off at the Udde-valla shipyard in Swe-den (1985) (Alfthanand Janzon 1994).

Relevant indicators

Unemployment ratesin Sweden rose from1.6 percent in 1976 to2.2 percent by 1978.Employment in manu-facturing declined by 8percent between 1977and 1979.

Volvo Co. set up amanufacturing plant inUddevalla. Unemploy-ment rates were declin-ing in this period whilemanufacturing em-ployment was fairlysteady.

Intervention design Type of evaluation

and job search tech-niques were also pro-vided. Firms were givenfinancial incentives tohire diese workers.

Participants were pro- Quasi-vided with classroom experimentaltraining.

A significant number of Nonscientificworkers joined retrain-ing programs severalmonths before theywere laid off. Coursesof varying durationwere offered in weld-ing, engineering, andcontrol engineering.These retraining pro-grams were provided bythe state-owned train-ing board, municipaleducation institutions,and other adult educa-tion services.

Result

Participants receivedlower weekly wagesthan those who did notreceive training. Dropin earnings was espe-cially significant in thefirst year; there wereno appreciable long-term gains.

By November 1987,more than 90 percentof the workers whohad completed traininghad found jobs orbecome self-employed—most of them inoccupations for whichthey had trained.

Comments

No information pro-vided on reemploy-ment rates or costs.Benefits from retrain-ing program wereinsignificant.

Two factors accountedfor the success of thetraining program:economic and labormarket conditionswere buoyant in theregion, and manage-ment, employmentoffices, and trainingagencies worked inclose cooperation.

Volvo Co. plant atGoteborg, Sweden,announced that 1,000workers would be laidoff in 1992 as it phasedout an old model andtooled up for a newone (Alfthan andJanzon 1994).

Shipyard closure inStorstrom county,Denmark (1986).2,000 people lost theirjobs (OECD 1993a).

Volvo planned torecruit 800 workers tomanufacture the newline of cars. Unem-ployment rates wererising sharply—from3.2 percent in 1991 to5.9 percent in 1992.Manufacturing em-ployment dropped by9 percent in 1992.Total employment fellby 4 percent.

High unemploymentrates in region, espe-cially among women.National unemploy-ment rates for men in1986 and 1987 were6.1 and 6.4 percent,respectively, while thecorresponding num-bers for women were10 and 9.6 percent.Employment in manu-facturing was stagnantfrom the mid-1980s.

A retraining programwas designed to help theexisting work forcemanage the changewithout job losses. Thecompany accepted theproposal under thestipulation that die costbe shared by die govern-ment. The program wasa broad competency-raising exercise, wiriispecific training toprepare the participantsfor die production ofthe new automobiles.

This program (1986-89) trained womenentrepreneurs to helpthem start their ownbusinesses. A total of200 hours of specificbusiness-orientedtraining was provided.

Nonscientific Program has not yetbeen evaluated.

Nonscientific Fifty-one businesseswere set up in 1989.Less than a third of theparticipants opened afull-time business, andfew hired any employ-ees, thus the totaladditional employmentgenerated was low.

Program costs areexpected to be about$25 million ($25,000per person), close tohalf of which will bepaid for by the govern-ment. In judging thecost-effectiveness ofthis program, it shouldbe noted that the un-employment benefitsthe government wouldhave had to paywould amount to$6.5 million.

The program did notseem to be successfuland was apparentlyquite costly (reliablecost estimates are notavailable).

«w (Table continues on the following page.)

Table A.I. (continued)Labor market problem

Sweden, 1980s and1990s: general evalua-tion of public retrain-ing programs (OECD1991).

Automobile plantclosure in Australia in1984. 445 workers laidoff (Leigh 1992).

Relevant indicators

Unemployment rosesteadily over theperiod.

Unemployment ratesrose steadily in Austra-lia, reaching a high of9 percent in 1984before falling to 8percent for the nextfew years. Between1980 and 1984 em-ployment in manu-facturing fell by 4percent.

Intervention design Type of evaluation

Various types Various types

Labor Adjustment Quasi-Training Arrangement experimentaloffered classroomtraining (averagelength 19 weeks).Main distinction be-tween courses waswhether they provideddriver training or not.

Note: JSA, job search assistance; OJT, on-the-job training.

Result

Retraining programshave become less effec-tive over time—espe-cially since theeconomy has begundeteriorating. Partici-pants have had moretrouble finding jobsthan other unem-ployed workers.

Over a nine-monthperiod, driver trainingincreased the probabil-ity of reemployment.However, fewer train-ees of other trainingcourses found jobs.

Comments

The cost-effectivenessof these programsdeclined both becausethey were less success-ful and because thecosts increased.

There were no datashowing the cost-effectiveness of differ-ent types of trainingcourses. Self-selectionwas a problem becauseindividuals who chosenot to participate werein the control group.The impact of longertraining was negative.

Table A.2. Overview of Studies Evaluating Training Programs for the Long-Term UnemployedLabor market problem Relevant indicators Intervention design Type of evaluation Result Comments

Training for workerseligible for JPTA TitleIII program (displacedworker) in Houston,Texas (1983-85).Eligible individualswere unemployed andhad a low probability ofreturning to their previ-ous occupation orindustry (Bloom 1990).

Training for workerseligible for JPTA TitleIII program (displacedworker) in El Paso,Texas (1983-85).(Criteria same asabove.) Workers laidoff from light manu-facturing plants(Bloom 1990).

A decline in petro-chemical industry ledto layoff. U.S. unem-ployment rates de-clined from 9.5 per-cent in 1983 to 7.4percent in 1985 (andmaintained this trenduntil the late 1980s).Employment in thisindustry increasedslightly between 1983and 1985.

U.S. unemploymentrates declined from 9.5percent in 1983 to 7.4percent in 1985 (andmaintained this trenduntil the late 1980s).Employment in manu-facturing fell by closeto 2 percent between1982 and 1983 butrose to about 5 percentin 1985.

Texas WAD project.Displaced workers wereprovided with JSA or amix of JSA with class-room training.

Texas WAD project.Displaced workersprovided with JSA aloneor combined withclassroom training.

Experimental

Experimental

One year after theprogram's completion,participants registeredno additional earninggain when comparedwith the sample ofworkers who had beengiven only JSA.

One year after theprogram's completion,this program recordedno effect on maleemployees' earningsbut did increase theearnings of women.

Despite the high costsof classroom training(twice as much as JSA),no additional gainsaccrued from this typeof training.

The increase in earn-ings of female partici-pants was slightly morethan the program'scosts. Male employeesreported no beneficialeffects.

(Table continues on the following pages.)

Table A.2. (continued)Relevant indicators Intervention design Type of evaluationLabor market problem Result Comments

Assistance aimed at thelong-term unemployedin New Jersey (1986-87). In general, work-ers were laid off frommanufacturing, trade,and services (Corsonand others 1989).

Long-term unem-ployed workers in theU.S. (1988), primarilyresulting from plantclosures (Leigh 1992).

Unemployment ratesin the U.S. fell from7.1 percent in 1985 to6.1 percent by 1987.While employment inmanufacturing re-mained fairly steadybetween 1985 and1988, trade employ-ment increased 7 per-cent and service-sectoremployment roseabout 5 percent.

Unemployment ratesin the U.S. fell from7.1 percent in 1985 to6.1 percent by 1987.While employment inmanufacturing re-mained fairly steadybetween 1985 and1988, trade employ-ment increased 7 per-cent and that in ser-vices about 5 percent.

New Jersey UI Reem- Experimentalployment Demonstra-tion project. Thisprogram provideddisplaced workers withJSA alone, or followedby OJT, classroomtraining (CT), or areemployment bonus.

The Trade AdjustmentAssistance Programwas intended to de-velop workers' skills innew occupations. Mostof these skills weresupplied by a voca-tional college or localcommunity college incourses that were morethan a year long.

Quasi-experimental

More than 10 quartersafter the programended, both CT andOJT significantly in-creased earnings overthose who received JSAonly. These individualswere also employed forlonger periods of timein each quarter thanthe jSA-only group.

Individuals who re-ceived training beganearning significantlymore than those whoreceived extendedincome-maintenancebenefits by the 6thquarter and continuedto earn more until the12th (last) quarter,reaching a level of$500 a quarter.

While the results oftraining were positive,the following caveatsapply: the trainees wereself-selected (thus theseresults may not applyfor a random group ofclaimants); only 15percent of those of-fered training acceptedit; and training ben-efited those who al-ready had marketableskills. Finally, analysisshowed that the costsexceeded the expectedbenefits.

The analysis was lim-ited to the manufactur-ing sector. Long-terminvestments in trainingmay be effective inincreasing earnings,but the training iscostly (each trainee wasgiven a $12,000 train-ing voucher).

Long-term unem-ployed workers inCanada (Goss andassociates 1989).

High regional unem-ployment levels inGermany in the late1970s (OECD 1991).

Individuals at risk ofbeing unemployed in1987-88 in Germany(Johanson 1994).

Compared with dieU.S., unemploymentrates have been fairly-high in Canada. How-ever diey declined from11.2 percent in die mid-1980s to 7.5 percent in1989. Spurred bygrowdi in commerceand services, employ-ment grew about 3percent annually overthe time period.

In 23 of the 142 re-gions in Germany,unemployment levelswere above 6 percent.Some firms were alsofacing serious problemsin adjusting to eco-nomic changes.

Unemployment rateswere steady between1986 and 1988. Theyfell somewhat by 1990.Employment grew atslighdy more than 1percent during thisperiod.

The Job Development Quasi-Program provided experimentalclassroom training andOJT. Wages and directcosts of classroomtraining were subsi-dized. The wage sub-sidy helped employerscover the cost of OJT.

Among other interven- Nonscientifictions, training andretraining of unem-ployed in firms. Thefirms who employedthese trainees receiveda subsidy of 90 percentof die employees'wages for two years.

This is an evaluation of Quasi-four programs, two of experimentalwhich trained workersfor new occupations(one offered furthertraining for employedand unemployed indi-viduals, while the odieroffered retraining forthe unemployed).

Employability ofwomen rose while thatof men declined.Weekly earnings ofwomen were insignifi-cant relative to that ofthe control group,while they were lowerfor males.

Training reducedunemployment some-what, but it is esti-mated that more than40 percent of thesehard-to-place individu-als had already lefttheir jobs by 1981.

No type of traininghad any significantimpact on the flowsout of short- or long-term unemployment,or on the flows intounemployment.

In view of the highprogram costs (around$9,300 per partici-pant), this training isnot cost-effective,especially for men.

In light of the extremelyhigh cost (around $500million for training andodier interventions), dieresults were very disap-pointing. No informa-tion was provided onwages.

No information onwages or costs oftraining.

(Table continues on the following pages..

Table A.2. (continued)Labor market problem Relevant indicators Intervention design Type of evaluation Result Comments

Individuals who wereeither at risk of becom-ing unemployed orwere already unem-ployed in Germany inthe early 1980s(Johanson 1994).

Help for long-termunemployed and dis-placed workers inEngland in the 1980s(Addison and Siebert1994).

Unemployment ratesremained fairly steadyat just over 10 percentin the early 1980s.

High unemploymentrates (around 10-11percent) persistedthrough much of the1980s. Between 1983and 1990 employmentrose by around 1.5percent.

Training preparedworkers for certifica-tion in one of the 375apprenticeable trades.The program requiredup to two years tocomplete. Contentsand specification ofretraining correspondto those of initial voca-tional training.

This community-basedprogram providedvocational training bylocal authorities inconjunction with localcolleges, central gov-ernment, and volun-tary organizations.Little private sectorinvolvement.

Nonscientific

Nonscientific

The overall successrate, measured in termsof retention rates, was54 percent. Employ-ability varied by age—less than half of thoseover 45 years old whowere unemployedmore than one yearfound jobs, while 86percent of 25- to 35-year-olds did.

Little impact onflows from long-termunemployment.

No data on cost wereavailable. The numberof workers who lefttheir jobs was fairlyhigh: two years aftercompleting trainingonly 60 percent of themen and 66 percent ofthe women were stillemployed.

The cost-effectivenessof these programs islikely to be negative.

Improve skills andprovide better access tojobs for the long-termunemployed inTilburg, the Nether-lands (OECD 1993a).

Job training programsfor unemployed anddisplaced workers inthe Netherlands (OECD1994b).

Unemployment ratesfell from about 13percent in the early1980s to below 10percent by the end ofthe decade.

Unemployment ratesfell from about 13percent in the early1980s to below 10percent by the end ofthe decade.

Training providedhands-on experience(through a simulatedworkshop). The empha-sis was on technicalskills (metalworking,woodworking, installa-tion techniques, andapparel trades) as wellas on good work habits.Courses ran from 4 to10 months.

Centers for adult voca-tional training.

Note: JSA, job search assistance; OJT, on-the-job training.

Nonscientific

Quasi-experimental

In 1991, of the 82individuals enrolled inthe course, 52 com-pleted it and 41 foundjobs or went on forfurther education. Themost successful trainingwas in metalworking.

Unemployed peoplewho were not in theprogram found jobs asquickly as those whowere. Even two yearslater, the employmentsituation for the twogroups was not signifi-cantly different.

Training was quiteexpensive: annualfunding of the pro-gram was about $7.5million, or $10,000per trainee.

No data were availableon wages and costs,but training seems tohave been ineffective interms of increasingthe employability ofparticipants.

References

The word "processed" describes informally reproduced works that may not be commonly availablethrough library systems.

Addison, J. T., and W. S. Siebert. 1994. "Vocational Training and the European Community."Oxford Economic Papers 46(4):696-724.

Alfthan, Torkel, and B. Janzon. 1994. Retraining Adult Workers in Sweden. Training Policy Study 3.Geneva: International Labour Office.

Anderson, Patricia, Walter Corson, and Paul Decker. 1991. "The New Jersey Unemployment In-surance Reemployment Demonstration Project: Follow up Report." U.S. Department of LaborUnemployment Insurance Survey. U.S. Department of Labor, Washington, D.C., Processed.

Bloom, Howard. 1990. "Back to Work: Testing Reemployment Services for Displaced Workers."W. E. Upjohn Institute for Employment Research, Kalamazoo, Mich. Processed.

Corson, Walter, Stephen Long, and Rebecca Maynard. 1985. "An Impact Evaluation of the BuffaloDislocated Worker Demonstration Program." Mathematica Policy Research. New Jersey. Pro-cessed.

Corson, Walter, S. Dunstan, Paul Decker, and A. Gordon. 1989. "New Jersey UnemploymentInsurance Reemployment Demonstration Project." Unemployment Insurance Occasional Paper89-3. U.S. Department of Labor, Washington, D.C. Processed.

Gill, Indermit S., and Amit Dar. 1995. "Costs and Effectiveness of Retraining in Hungary." Inter-nal Discussion Paper IDP-1 55. World Bank, Europe and Central Asia Region, Washington, D.C.Processed.

Goss, Gilroy, and Associates. 1989. "Evaluation of the Job Development Program: Final Report."Report prepared for the Program Evaluation Branch, Canadian Employment and ImmigrationOffice, Ottawa. Processed.

Grossman, Jean B. 1994. "Evaluating Social Policies: Principles and U.S. Experience." The WorldBank Research Observer 9(2): 159-80.

Heckman, JamesJ. 1979. "Sample Selection Bias as a Specification Error." Econometrica47(l):\53—62.

. 1992. "Randomization and Social Policy Evaluation." In Charles Manski and IrwinGarfinkel, eds, Evaluating Welfare and Training Programs. Cambridge, Mass.: Harvard Univer-sity Press.

Johanson, Richard. 1994. Retraining Adults in Germany. Training Policy Study 4. Geneva: Interna-tional Labour Office.

LaLonde, Robert. 1986. "Evaluating the Econometric Evaluations of Training Programs with Ex-perimental Data." American Econometric Review 76(4):604—20.

Leigh, Duane. 1992. "Retraining Displaced Workers: What Can Developing Nations Learn fromOECD Nations?" Policy Research Working Paper 946. World Bank, Population and Human Re-sources Department, Washington, D.C. Processed.

. 1994. Retraining Displaced Workers: The U.S. Experience. Training Policy Study 1. Geneva:International Labour Office.

Newman, John, Laura Rawlings, and Paul Gertler. 1994. "Using Randomized Control Designs inEvaluating Social Sector Programs in Developing Countries." The World Bank Research Observer9(2):181-201.

OECD (Organisation for Economic Co-operation and Development). 1991. Evaluating Labor Mar-ket and Social Programs: The State of a Complex Art. Paris.

I O O The World Bank Research Observer, voL 13, no. I (February 1998)

-. 1993a. Partnerships: The Key to Job Creation. Experiences from OECD Countries. Paris.

-. 1993b. Employment Outlook. Paris.

-. 1994a. The OECD Jobs Study: Evidence and Explanations. Paris.-. 1994b. Vocational Training in the Netherlands: Reform and Innovation. Paris.

O'Leary, Christopher. 1995. "An Impact Analysis of Labor Market Programs in Hungary." W. E.Upjohn Institute for Employment Research, Kalamazoo, Mich. Processed.

Pulay, Gyula. 1995. "Measuring the Efficiency of Employment Programs in Hungary." Paper pre-pared for a seminar on Evaluating Active Labor Programs. World Bank, Eastern Europe Depart-ment, Washington, D.C. Processed.

Revenga, Ana, Michelle Riboud, and Hong Tan. 1994. "The Impact of Mexico's Retraining Pro-gram on Employment and Wages." The World Bank Economic Review 8(2):247-77.

World Bank. 1995. World Development Report 1995: Workers in an Integrating World. New York:Oxford University Press.

Amit Dar and Indermit S. Gill 1O1