november 1990 prospective evaluation methods · chapter 1 what is a prospective question? 8 chapter...

United States General Accounting Office

GAO Program Evaluation and MethodologyDivision

November 1990

ProspectiveEvaluation Methods

The ProspectiveEvaluation Synthesis

GAO/PEMD-10.1.10

Preface

GAO assists congressional decisionmakers in theirdeliberative process by furnishing analyticalinformation on issues and options underconsideration. Many diverse methodologies areneeded to develop sound and timely answers to thequestions that are posed by the Congress. To provideGAO evaluators with basic information about themore commonly used methodologies, GAO’s policyguidance includes documents such as methodologytransfer papers and technical guidelines.

This methodology transfer paper on prospectiveevaluation synthesis focuses on a systematic methodfor providing the best possible information on, amongother things, the likely outcomes of proposedprograms, proposed legislation, the adequacy ofproposed regulations, or top-priority problems. Thepaper uses a combination of techniques that bestanswer prospective questions involving the analysesof alternative proposals and projections of variouskinds. As GAO receives more requests forassessments about the implications of futureoccurrences, evaluators should find this systematicapproach a beneficial tool to facilitate them in theirwork.

The present transfer paper is one of a series of papersissued by the Program Evaluation and MethodologyDivision (PEMD). The purpose of the series is toprovide GAO evaluators with guides to variousaspects of audit and evaluation methodology, toillustrate applications, and to indicate where moredetailed information is available. This paper wasoriginally authored by Lois-ellin Datta. This reissuedversion supersedes the July 1989 edition.

GAO/PEMD-10.1.10 Prospective EvaluationPage 1

Preface

We look forward to receiving comments from thereaders of this paper. They should be addressed toEleanor Chelimsky at 202-275-1854.

Werner GrosshansAssistant Comptroller GeneralOffice of Policy

Eleanor ChelimskyAssistant Comptroller Generalfor Program Evaluation and Methodology


Contents

Preface 1

Chapter 1 What Is aProspectiveQuestion?

8

Chapter 2 The Need forSystematicMethods forAnsweringForward-LookingQuestions

17

Chapter 3 ProspectiveMethods and theProspectiveEvaluationSynthesis BroadlyDefined

20When the PES Is and Is Not Appropriate 23The PES and the Recommendations GAO

Makes26

Chapter 4 The PES: InitialSteps

29Defining the Problem 30Selecting Alternatives to Evaluate 34

Chapter 5 The PES: Middleand Final Steps

41The Conceptual Analysis 41The Operational Analysis 48Testing the Model 54


Contents

Presenting the Results 70

Chapter 6 Variants of thePES

78Targeted PES 78Variants Using Other Sources of Information 85

Appendixes Appendix I: A Brief History of the PES andSome Other Prospective Methods

90

Appendix II: Data Quality Judgment Models 94Appendix III: A Project Evaluation Profile 104References 109Papers in This Series 115

Tables Table 1.1: Types of Forward-LookingQuestions and What We Are Asked to Do

11

Table 1.2: Features of Retrospective andProspective Methods

16

Table 3.1: Some Prospective Methods 22Table 3.2: Illustrations of Where a PES Might

Strengthen Our Recommendations27

Table 3.3: Situations in Which a PES Shouldand Should Not Be Considered

28

Table 4.1: Steps in the Basic PES Approachand Persons Involved

29

Table 4.2: Step 1: Defining the Problem 32Table 4.3: Step 2: Selecting Alternatives to

Evaluate36

Table 5.1: Step 3: Conceptual Analysis 42Table 5.2: Step 4: Operational Analysis 50Table 5.3: Step 5: Testing Key Assumptions

Against Existing Evidence56

Table 5.4: Step 6: Presenting Results 71Table 6.1: Targeted PES and Related Critical

Issues78

Table II.1: Advantages and Disadvantages ofFour Data Quality Judgment Models

95

Table II.2: Example of a Fatal Flaws Analysis 100


Contents

Figures Figure 3.1: The Triad of Analysis 23Figure 5.1: Underlying Conceptual Model of

the First Bill45

Figure 5.2: Underlying Conceptual Model ofProgram A in the Second Bill

46

Figure 5.3: Underlying Conceptual Model ofProgram B in the Second Bill

47

Figure 5.4: Underlying Operational Model ofProgram B in the Second Bill

53

Table 5.5: Example of Presenting PESFindings

74

Abbreviations

GAO U.S. General Accounting OfficePES Prospective evaluation synthesis


Chapter 1

What Is a Prospective Question?

Why should a GAO evaluator read a paper on theprospective evaluation synthesis (PES)? GAOevaluators must know about methods such as the PESbecause the changing nature of our work requires usto be familiar with the strengths and limitations, andthe applicability, of ways to answer questions dealingwith the future. The PES is one these methods.

GAO is increasingly being asked to answer questionsabout the future that involve analyses of alternativeproposals and projections of various kinds. Tosupport GAO’s capacity to answer these questionswell, our policy and project manuals have beenexpanded to discuss, for example, different types offorecasting and formal modeling approaches and ourstandards for carrying these out. This is becausesystematic methods for dealing with questions aboutthe future can be more efficient and yield sounder,better-documented answers than more informalmethods do.

Many methods exist to deal with forward-looking,future-oriented questions. Collectively, they arereferred to as prospective methods to distinguishthem from approaches designed to answer questionsabout what is happening now or what has happenedin the past—that is, retrospective methods.

Among the prospective methods, we have chosen tofocus here on the prospective evaluation synthesis.GAO developed the PES as a systematic method formeeting congressional requests for analyzingproposed legislation and helping identify top-priorityproblems. Other applications of the PES might be inthe analysis of recommendations in draft GAO reportsand in assessing the adequacy of proposedregulations.


Chapter 1


This paper shows how the tools of evaluationmethodology can be applied in order to provide thebest possible information prospectively on the likelyoutcomes of proposed programs. A PES may beconducted through the comparison of policy orprogram alternatives, although it is also useful whenfocused on a single policy or program. It is easiest toperform when an adequate data base already exists.Fortunately, data bases concerning proposedprograms frequently do exist, primarily becauseproblems are rarely new. Often they have beenaddressed by past programs whose experiences canbe drawn upon for the PES.

In essence, a PES is a combination of the followingactivities: (1) a careful, skilled textual analysis of aproposed program, designed to clarify the impliedgoals of that program and what is assumed to getresults, (2) a review and synthesis of evaluationstudies from similar programs, and (3) summaryjudgments of likely success, given a future contextthat is not too different from the past. In this respect,the PES resembles the evaluation synthesis approach,except that the focus of the PES is on how evaluationstudies cast light on the potential for success of theproposed programs, as opposed to reachingconclusions about the actual performance of existingprograms.

Three other points emerge from the experience withPES. First, the PES may call for a greater selectivitythan the evaluation synthesis. The latter involves acomprehensive review of all existing studies, whichcan allow us to generalize quite broadly. Thetime-driven nature of PES may restrict it to anarrower focus and the use of strategies, such assampling, to balance resources and the need forexternal validity. Second, legislators andcongressional staff who have received a PES view it


Chapter 1


as a useful tool. From the congressional perspective,a PES means that expert design assistance is availablefor a new program at the point when it is most neededand when it can help convince others of the basiclogic and likely success of the program. Third, from apublic policy perspective, providing understandingahead of time about how a program is likely to workrenders an important service by validating the basicsoundness of what is to be undertaken and therebyincreasing its chances for success.

To understand prospective questions, it can be helpfulto begin with some examples of GAO reports. GAOreported that the passage of a proposed bill, S. 581,would probably open some jobs to women that werecurrently closed and that might otherwise remainclosed after the review required by the secretary ofthe Department of Defense was finished. (U.S.General Accounting Office, 1988h) GAO also informedthe Congress about difficulties with specific Food andDrug Administration forecasts. These forecastspredicted the increase in the number ofmedical-device problems that would be reported byhospitals and the number of agency staff that wouldbe necessary to analyze the reports of those problemsunder the proposed Medical Devices ImprovementAct of 1988. We concluded that these forecasts werebiased and not representative of what would begenerated from data obtained from U.S. hospitals ingeneral. (U.S. General Accounting Office, 1988g) AndGAO found in yet another study that the InternalRevenue Service needed to review its entirerevenue-estimating process in order to validate theassumptions used to better reflect actual historicaltrends. (U.S. General Accounting Office, 1988k)

These reports illustrate the prospective, orforward-looking, questions that GAO is often asked to


Chapter 1


deal with.1 As table 1.1 shows, at least four kinds offorward-looking questions can be identified in reportswe have issued already, requests that have been metin ways other than through reports, and our ownpolicies regarding our recommendations.

Table 1.1: Types ofForward-LookingQuestions and What WeAre Asked to Do

What we are asked to do

Question type

Critiqueothers’analyses

Do analysesourselves

Anticipate the future 1. How well hastheadministrationprojectedfuture needs,costs, andconsequences?

3. What arefuture needs,costs, andconsequences?

Improve the future 2. What is thepotentialsuccess of anadministrationorcongressionalproposal?

4. What courseof action hasthe bestpotential forsuccess and isthe mostappropriate forGAO torecommend?

1GAO does not normally make forecasts, although we have done soon special request (for example, in response to our assigned dutiesunder requests related to Gramm-Rudman-Hollings). We do oftenevaluate the forecasting process and the methodology used. Ourpast work has indicated, for example, that agencies can improveforecast accuracy by using better techniques and validatingpredictions. The same points apply to modeling. It should also benoted that other agencies are frequently called upon forforward-looking analysis. The Office of Management and Budgetrequires regulatory impact analysis before any major newregulation is put into effect. And the Congressional Budget Office isrequired to “price out” all new legislation. Thus, there are manyapplications and methods in this prospective area.


Chapter 1


The use of the PES described in this paper isconsistent with GAO’s policy on forward-lookingquestions and on the methodology to be used indeveloping recommendations. This policy is set forthin the General Policy Manual, chapter 10.0, and inchapters 12.10 and 12.18 of the CommunicationsManual. These latter chapters specify, for example,the procedures that are to be followed when dealingwith programs and policies under legislativeconsideration or recommendations asserting thepossibility of budgetary savings. Particularly relevantin the General Policy Manual are the sections onformal modeling, economic optimizing, andforecasting.

1. How well has the administration projected orestimated the future needs, costs, and consequences?In responding to such a forward-looking question,GAO may need to address issues such as thefollowing:

• How well has it anticipated, for example, revenues orstaff needs or emerging problems?

• Are the methods for projection sound?• Are the data bases reliable and adequate?• Are the assumptions explicit?• Are they reasonable?• Have the projections been overgeneralized?• Are there feasible improvements to the procedures or

the reporting?• Are better estimates, or better-reported estimates,

available?

In the case of repeated or regular forecasts, we mayhave to examine whether the relevant agencysystematically evaluates their accuracy and, if so,whether the error rates are acceptable and withoutbias. Further, when the administration publishesclaims about the likely consequences of its own


Chapter 1


proposed activities, we may examine whether claimsare methodologically sound and properly presented.And, when the administration has sought to block orprevent action, using projections or estimates offuture costs or consequences, we may determinewhether these projections, too, are sound andaccurately reported.

2. What is the potential for the success of acongressional or administration proposal? Inanswering this type of inquiry, GAO could look at thefollowing questions:

• Given the characteristics of new or amendedlegislation being considered by the Congress, howlikely is it that a bill will achieve its stated objectives?

• What features might be modified to improve itschances of success?

• Are there side effects or pitfalls known from pastexperience that could be remedied prospectively?

• When the administration initiates a new policy or newlegislation by proposing a set of activities, how likelyis it that these will work?

• What changes that might be made before the proposalis put into effect would better achieve the intendedresults?

• What unidentified dangers should be consideredbefore action is taken?

3. What are future needs, costs, and consequences? Inmany areas, GAO is asked to anticipate the future inanalyses such as the costs of future illegalimmigration, the flow of future legal immigrants, thefuture costs of the AIDS epidemic, military personnelneeds, and the adequacy of stockpiles of materialscritical to the national defense. According to ourpolicies, we are expected to use state-of-the-artmethods for making any quantitatively basedforecasts or projections and to use due professional


Chapter 1


care in applying qualitative approaches, such asexpert panels. We could check on whether we haveused the technically most solid procedures, fullyconsidered alternative methods, and applied andreported properly the ranges of uncertainty inevitablein any prediction, using approaches such assensitivity analyses to test systematically the effectsof different assumptions.

4. What course of action should we recommend asmost likely to succeed in addressing the problems weidentify? Our policies require us to carefully consideralternative actions resulting from our findings and toweigh the costs of these alternatives and theirlikelihood of success before we present them asmatters for consideration or as recommendations.This requirement distinguishes GAO from othercongressional support agencies. They follow thepolicy analysis approach of presenting options but donot make recommendations. GAO goes through theanalytic steps and makes its choice of the preferredsolution. Further,

GAO systematically follows up and reports on theacceptance of the recommendations it makes in itsreports. In this context, procedures for developingalternatives and selecting recommendations can beseen as the most crucial part of our work. Have weused the most methodologically sound procedures foridentifying alternative actions and for making anddocumenting the analyses required in our policy andprocedures manuals?

While these illustrations do not exhaust the range ofprospective questions, what they say is that we areeffectively in the futures business, both through theimplications of our own policies and because theCongress is asking us to make or examine estimates


Chapter 1


of and projections about the future.2 This may beexpected to continue (1) as the effort required formembers of the Congress to push new legislationthrough the Congress and to amend existinglegislation becomes greater, (2) as evaluations of pastprograms demonstrate problems that could have beenprevented in existing programs, and (3) as themethodology and the motivation to get smarter aboutthe future improve and increase. That is, we have animportant role in helping prevent future problems andin helping promote greater success before action istaken and before program actors and stakeholdersbecome entrenched.

This role complements our mission to reportobjectively, but in retrospect, on what is happeningnow and on what has occurred in the past. It is quite adifferent one, with distinctive methods of its own. Astable 1.2 indicates, retrospective and prospectivemethods differ on such features as the source of theevaluation questions, where we get our information,and techniques for analyzing the evidence. Eachmethod has its own requirements and its ownstrengths and limitations for our work. Those of thePES will be discussed in detail in this transfer paper.The requirements of retrospective methods have beenpresented in other transfer papers.

2The Kansas City Regional Office maintains a comprehensivereview and bibliography of all GAO reports involving relativelyinnovative methodologies, providing easy access to these earlierapplications, for job planning purposes. This list includes manyreports dealing with forward-looking questions, some of which areincluded in our references to help illustrate further the range andhistory of this aspect of our work.


Chapter 1


Table 1.2: Features ofRetrospective andProspective Methods

Feature Retrospective Prospective

Source of questions Criteria andissues in existingprograms,regulations, andpolicies

Ideas andassumptionsabout problems,probable causes,and possiblesolutions

Primary sources ofinformation

Documents,administrativedata, interviews,observations,opinion surveys

Prior research,theory, andevaluations; pilotor experimentaltests of proposedapproach; expertopinion

Primary types ofanalysis

Qualitativeapproaches toempirical data,quantitativeapproaches toempirical data,informationsynthesis inrelation toprogram criteriaand issues

Simulations,modeling, andinformationsyntheses inrelation toconceptual andoperationalassumptions ofproposals (PES);Delphitechniques;analyses of likelyeffects

We have already discussed the nature offorward-looking questions, described the types ofmethodological issues they raise, and summarizedwhen a PES would and would not be appropriate.Subsequent chapters present a definition of theprospective evaluation synthesis, a detailed exampleof how to carry it out, and some of its variants.Special attention is given to the crucial issues ofjudging the quality of the information beingsynthesized and models for aggregating results acrossmany prior studies.


Chapter 2

The Need for Systematic Methods forAnswering Forward-Looking Questions

In doing our work, we should use the methodologyappropriate to the complexity of the question and tothe level of effort required by the situation. Eitheroverkill or underkill in design would be a mistake injob management. The first wastes scarce resources;the second fails to meet the need adequately.

For some questions and some circumstances, the useof highly systematic methods of dealing withforward-looking questions would be overkill. Forexample, we may be asked about one provision ofproposed legislation in an area in which we have hadmany years of experience and in which we havepublished reports whose recommendations beardirectly on the provision. Further, the idea may beone among several at early stages of considerationand it may be unclear that the legislation will moveforward in the current session. Here, the evaluatormight adequately satisfy methodological andcustomer concerns by drawing on our cumulativeexperience to discuss the issue as we have alreadyseen it and, subject to our usual reviews for billcomments, comment informally on it. That is, we mayuse professional judgment and opinion.

Where the questions are controversial, far-reaching,and sensitive, more systematic methods may be calledfor. For example, our analyses of the savings and loanproblems, and of various bailout proposals, called formore than informal methods, because of thesensitivity and long-term consequences of how thisissue is resolved.

Among the advantages of using systematic methodsare the following.

1. The full range of existing information may beefficiently brought to bear on the question. Ratherthan relying, in a somewhat happenstance way, on an


Chapter 2

The Need for Systematic Methods for

Answering Forward-Looking Questions

individual’s memory, we identify, consider, and applythe body of available knowledge to answering thequestion. Data that were costly to collect in the pastand are still relevant but that might otherwise beneglected can be used. The risk of overlookingcontradictory evidence may be notably reduced.

2. The degree of confidence we have in our ownanswers—whether analyses of other people’sforecasts, conclusions regarding the success ofproposed legislation, or our ownrecommendations—can be stated more precisely thanless-formal methods permit. When we deal with thefuture, uncertainty is part of any analysis, no matterhow sound, but the more precisely we state thedegree of uncertainty, the more complete, and themore useful, our prediction will be. Saying, “We are95-percent confident that the number of competitivelyawarded contracts will increase by between 10,000and 15,000 for each of the next 4 years” provides moreprecise information to a decisionmaker aboutlikelihood than does the statement “More contractswill be awarded competitively in the future.”

3. One method for promoting the quality ofprospective work is independent replications. Whenwe use systematic methods to review other people’sprojections or to make our own, we are better able toreplicate the analyses and thus promote quality. Thatis, when independent analysts obtain the sameresults, confidence in findings rises. In the physicalsciences, such replication in independent laboratoriesis often required before a result is accepted as sound.However, replication requires precision in describingand carrying out the analytic procedures. Similarly, inthe social sciences, of which program evaluation is apart, using systematic methods permits replicationand helps distinguish robust findings from artifacts ofdifferences in technique.


Chapter 2

The Need for Systematic Methods for

Answering Forward-Looking Questions

4. Systematic methods can help us follow high-qualitystandards of evidence and analysis in documentingthe basis for answers about the future. Much of ourwork requires an element of judgment. Prospectivejobs inherently involve a greater degree of uncertaintythan retrospective questions and, consequently, agreater element of judgment. In all such jobs, we mustbe scrupulous in identifying sources of uncertaintyand, consequently, the need for alternatives andoptions. However, using systematic prospectivemethods can reduce the qualifications we have to add.Fewer caveats may be necessary if we applystate-of-the-art methodology.

In short, systematic prospective methods hold greatpromise for strengthening our ability to speak well toemerging issues.


Chapter 3

Prospective Methods and theProspective Evaluation SynthesisBroadly Defined

Prospective questions deal primarily with what willhappen in the future. However, most prospectivemethods rely heavily on information about what hashappened in the past, primarily empirical andevaluative data. Judgments—that is, assumptions andinterpretations—enter in, particularly when wespeculate on future conditions or alternativescenarios. Methodologically, answers to thesequestions require approaches that meet specialchallenges, compared with retrospective methods.

For example, almost all evaluations have to takecontext into account if the ability to generalize is anissue. In retrospective methods, one approach thatpermits generalization is simple random samplingfrom a properly defined population. Another suchapproach is stratified random sampling, in whichrelevant subgroups are considered, such as urban andrural or rich and poor states. Where there is reason toexpect that the results of a program will depend ondifferent circumstances—the economy, the culture,human resources—stratified random sampling istypically used. For retrospective studies, what isrelevant is usually clear, and how the characteristicsof entities we could sample vary is usually known.

Not so for prospective studies. What the relevantcharacteristics of the future will be, and how entitieswill vary, encompasses a wide range of possibilities.For example, whether participants in a proposedjob-training program will be likely to find employmentin a given period may be influenced more by overalltrends in the economy than by instructional ortargeting nuances. But perhaps economic conditionswill be relatively unchanged, so that othercharacteristics of the context will be more importantto consider.


Chapter 3

Prospective Methods and the

Prospective Evaluation Synthesis

Broadly Defined

Putting this distinction somewhat more technically,generalizations in retrospective studies are fairlystraightforward, empirically based statements inwhich one moves logically from a sample to apopulation. Extrapolations in prospective analyses, incontrast, require one to move logically andconceptually, as well as empirically, by taking intoaccount how a particular finding might operate undervarying conditions and situations. We thus have tomake economic and other assumptions explicitly;otherwise, we are implicitly accepting thecontinuation of the present unchanged into the future.(See Cronbach, 1982, for a more detailed discussion.)

Despite this and other challenges, a set of prospectiveevaluation methods has been developed. As table 3.1illustrates, these include actual, empirical, logical,judgmental, and mixed approaches.1

1Economists have developed many quantitative methods forprojecting the future, particularly those involving economicforecasting, modeling, and simulations. These have in common thespecification of a theory (conceptual model in PES terms) of whatis influencing relevant outcomes, the identification of keyassumptions, quantification—on the bases of theory and pastexperience—of these assumptions, and running often very complexquantitative analyses of most likely outcomes under differentassumptions about how the future will be similar to and differentfrom the present and the past. For example, the Social SecurityTrustees Report is based on quantitative models whose keyassumptions include more and less optimistic estimates ofeconomic conditions. Our policy manuals describe some of thesetechniques and suggest appropriate uses. The PES can include theresults of these modeling and simulation studies but differs fromthem in its greater reliance on prior empirical work on relatedprograms in the past or on basic and applied research.


Chapter 3



Broadly Defined

Table 3.1: SomeProspective Methods Type Illustrative technique

Actual Experimental tests;Demonstration programs

Empirical Simulation;Forecasting

Logical Front-end analysis;Risk assessment;Systems analysis;Scenario building;Anticipatory analysis

Judgmental Delphi techniques;Expert opinion

Mixed Prospective evaluation synthesis

The prospective evaluation synthesis, or PES, is anew member of the class of prospective methods(Chelimsky, 1988). It was adapted by GAO from theevaluation synthesis in order to answer questionsabout the future more systematically than informalmethods and more rapidly than some otherprospective methods such as experimental programs.(U.S. General Accounting Office, 1983) (Appendix Ialso gives a brief history of the PES.)

Conceptually, the PES provides a way in which thelogic of evaluation methodology and its procedurescan be appropriately used in assessing the potentialconsequences either of an individual proposal or ofalternative and competing policy proposals. Itcombines (1) the construction of underlying modelsof proposed programs or actions as developed byWholey for evaluability assessment with (2) thesystematic application of existing knowledge asdeveloped in the evaluation synthesis methodology.(Wholey, 1977) That is, a PES is a prospective analysisanchored in evaluation concepts. It involves logical,conceptual, and empirical analyses, taken in thecontext of the future.


Chapter 3



Broadly Defined

As figure 3.1 illustrates, the conceptual analysesresults help focus the operational analyses andanswer the question, “Logically, should the proposalwork?” The operational analyses further scope thesearch for empirical findings and answer the question,“Practically, could the proposal work?” The empiricalanalyses can open both new conceptual andoperational possibilities and answer the question,“Historically, have activities conceptually andoperationally similar to the proposal worked in thepast?” Finally, the PES takes into account ways inwhich the past is and is not likely to be similar toplausible future conditions.

Figure 3.1: the Triad ofAnalysis

When the PES Isand Is NotAppropriate

As noted, the PES can be used either for examiningan individual proposal or for comparing two or morepolicy alternatives. In examining an individualproposal, the PES requires a criterion, or a hoped-forgood that needs to be made explicit. Developingexplicit criteria is a task familiar to GAO evaluators.Nonetheless, it is often difficult, since legislative


Chapter 3



Broadly Defined

proposals can result from greater agreement onactions than on aims or goals. Assessing two or moreproposals may be somewhat easier, because thepoints of “common cause” can serve as a proxy forthe hoped-for good. Further, it is generally simpler tomake comparative judgments (“Which is better?”)than absolute ones (“Is it good at all? How good?”).

The PES andTimeliness

Additional conditions affect the use of the PES.Although the PES has the promise of being among themost timely evaluation methods, obviously it cannotoperate instantaneously. While times vary, an analysisof two or more bills might require about 3 months onthe part of at least two evaluators in order to providefor adequate reviews of published and unpublishedliterature, consultation with technical experts, andthe thorough assessment of the resulting information.However, a PES may take longer than 3 months,especially when the competing legislative proposalsare quite complex, when there is little priorexperience with issues, or when most of the literatureis unpublished.2

This time constraint indicates that a PES should bestarted as soon as possible after a customer’s inquiry,in order to ensure that the assigned evaluators havethe requisite time for their work. For less-complexissues, or situations such as analyses of possible GAOrecommendations, where a separate report does nothave to be written, less time may be required. Asnoted earlier, a greater level of effort would be

2The unpublished literature can include reports prepared undercontract to the government, work in progress that has beenpresented as draft material or in speeches, and other relevantmaterial that may not have appeared yet in print. Searching forthese materials usually involves reviewing federal contracts andgrants, contacting project managers and principal investigators,and canvassing other experts in the field.


Chapter 3



Broadly Defined

allocated to controversial, sensitive, and far-reachingquestions.

The PES and DataAvailability

Another point affecting timeliness is that when anissue becomes extremely popular or extremelycontroversial in the legislature, it may happen thatmany different bills on the same subject areintroduced within a short time. This can cause suchlogistical and other problems that a PES may not bethe appropriate method. But if this situation shoulddevelop in the middle of the PES effort, then theevaluator would either have to resist expanding thescope of the study or obtain an extension of time.

As indicated above, the PES relies heavily on theknowledge—basic and applied—already produced byevaluators and researchers. The PES can be usedeffectively on topics for which a body of relevantliterature exists. For some mature issues that havelong attracted the attention of evaluators andresearchers, the existing literature may be abundant,containing many studies and theories concerning thebasic mechanisms involved. For others that are newor have not yet stimulated much investigation andscholarship, PES evaluators may not be able to find agreat deal that is relevant.

As mentioned earlier, this outcome tells thepolicymakers that there is little empirical basis fortheir decisions. They can then judge the merits ofmoving ahead, not moving forward, or limiting thetypes of actions they take (targeting, demonstrations,and so on). It may also be an important opportunity topresent to policymakers the research and data needsthat would have to be filled in order to make firmjudgments. The case of the PES that includesrecommendations for demonstration, experimental,or pilot projects may, therefore, be relatively frequent,


Chapter 3



Broadly Defined

since such approaches can be useful alternatives toacross-the-board changes in national policies.

The PES and theRecommendationsGAO Makes

In many situations, a full PES would be overkill as weprepare recommendations. For example, finding alack of accepted internal controls or finding a failureto report honestly information unfavorable to costlyweapon systems leads quite directly to well-supportedrecommendations. In other circumstances, however,our findings are more complex, our sense ofalternatives is broader, the results are more uncertain.In some cases, these could be presented as mattersfor consideration. In others, particularly thoseinvolving controversial, sensitive, or far-reachingconclusions, our recommendations—derived perhapsthrough other methods—could themselves properlybe the subject of a PES.

Table 3.2 illustrates some of these circumstances,which include, for example, situations in which thefederal role may be relatively complex, ourrecommendations would pose notable costs orburdens, and major structural or managementchanges might be involved. In such circumstances,investing some time in a PES might permit us to beeven more hard-hitting and convincing and to have asolid effect, leading in turn to greater savings andnonmonetary benefits. These and otherconsiderations about when an evaluator shouldconsider a PES are summarized in table 3.3.


Chapter 3



Broadly Defined

Table 3.2: Illustrations ofWhere a PES MightStrengthen OurRecommendations

General circumstance Specific example

Involves complexfederal, state, and localrelationships

What is the best way for thefederal government to encouragestate and local governments toserve handicapped persons whoare older and younger thanregular school age?

What would be the best strategyto strengthen results from federalfunds in child abuse prevention?

Nontrivial costs orburdens

How many Internal RevenueService agents should be addedto current staff or redirected fromcurrent tasks to go afterunreported income not caught bycomputer matching?

Major structural ormanagement changes

How should the responsibilitiesand roles of the Office ofManagement and Budget andother agencies be restructured tobetter identify low-qualitysurveys?

Very high nationalstakes are involved

What are the optimum ways ofdealing with thesavings-and-loan crisis?


Chapter 3



Broadly Defined

Table 3.3: Situations inWhich a PES Should andShould Not BeConsidered a

Consideration of PES as a method

Situation Probably shouldProbably shouldnot

Technical

Data basequality

High, moderate Low

Proposalcomplexityrelative to timeavailable

Complexity low ormoderate andtime short ormoderate; or,complexity highand time long

High complexity;little time

Proposalstability

High, moderate Low

Contextual

Degree offederalleverage(regulations,funds)

Moderate, high Low

National stakes Moderate, high Low

Consequencesof ourrecommendations

Far-reaching Restricted inscope

aThese considerations apply to the PES. Other prospectivemethods could be useful when it would not be appropriate todo a PES.


Chapter 4

The PES: Initial Steps

As table 4.1 shows, there are six steps in the basicPES approach, three of which closely involve thepersons who request the job or are likely to use theresults to make decisions—the customer. The sixsteps are defining the problem, selecting the optionsor alternatives to evaluate, analyzing the conceptualunderpinnings of the selected alternatives, analyzingthe operational logic of the selected alternatives,testing the key conceptual and operationalassumptions against existing evidence, and presentingthe results in relation to the key assumptions.

Table 4.1: Steps in theBasic PES Approach andPersons Involved

Step Persons involved

Defining the problem Customer, evaluatora

Selecting alternatives toevaluate

Customer, evaluator

Conceptual analysis Evaluator

Operational analysis Evaluator

Testing key assumptions

Check on assumptioncentrality

Customer, evaluator

Test against existingevidence

Evaluator

Presenting results EvaluatoraFor GAO, the customer is the congressional requester for thejob. Other persons helpful at this step might includestakeholders and experts in the field. In the catastrophic healthinsurance PES, for example, health provider and consumerorganizations provided useful input in defining the problem.Input is, of course, received in the context of the usual GAOguidance on ensuring our independence and objectivity.

While these steps are essential in using the PES forcommenting on proposed congressional oradministration actions, they also apply to the analysisof possible recommendations, with two


Chapter 4


modifications. First, generating alternativerecommendations involves either usual GAOprocedures or the application of techniques such asforecasts, assessment of likely effects, andscenario-building. Second, we need to use judgmentwith regard to how extensively we can involve thecustomer in selecting options and in checkingassumption centrality while maintaining our essentialindependence at this stage of our work.

In this chapter, we discuss the first two steps shownin table 4.1. The others are described in chapter 5. Foreach step, we first present what that step means, whyit is important, what its role is, and the kind ofactivities that would fulfill the requirements. Then weillustrate how to do the step through its application ina GAO report. The applications in both chapterscenter on a specific example, a PES conducted oncompeting legislative proposals dealing with theproblem of teenage pregnancies. (U.S. GeneralAccounting Office, 1986b)

Defining theProblem

DetailedSpecification

Table 4.2 shows the key elements of this importantfirst step. Here the evaluator works with the client todraw the target that the proposal is to hit, trying to beas clear as possible on the size and nature of theconcerns that the proposal is intended to solve. In thePES, the evaluator is trying to see if the proposedprogram will work to solve not a generic problem,necessarily, but a specific one. Thus, a program thatmay be well-aimed at one target may miss anotherwidely. For example, many programs can involveproviding food supplements, nutrition education, andhealth screening. Some, however, may be aimed at


Chapter 4


solving the problem of low birth weight babies amonglow-income women and teenage mothers; others maybe aimed at promoting age-appropriate progress inheight and weight among preschoolers. Hence, thepivotal question of this first step: What’s the target?


Chapter 4


Table 4.2: Step 1:Defining the Problem Aspect Definition

What “defining the problem”means rules

Detailed specification ofthe concern that rules inand out what will beconsidered part of theproblem. This creates the“target” to be “hit”successfully by theproposal

Why this step is important Different people may definean apparently “clear”problem broadly ornarrowly. Unless customerand evaluator agree onwhat is to be consideredpart of the problem,analyses aimed atdetermining whetherproposals will work canthemselves be off-target

The role of this step As the start of the PES, ithelps determine the scopeof the work and lays thefoundation for the use ofthe results

Activities that fulfill therequirements for this step

(1) Discussions with thecustomer and review ofhearings (if any) on theproposal with regard to thesize and nature of theproblem. (2) Independentanalysis of the evidenceregarding the size andnature of the problem. (3)Identification of points thatrequire agreement anddecisions. (4) Discussionswith the customer andothers as necessary toreach closure on thedefinition of the problem


Chapter 4


Illustration In 1984, there were about a million pregnancies and500,000 births to women under 20. In response camebipartisan congressional efforts to increase thefederal effort in this area. More than a score of billswere introduced into the Congress in 1986.

Concerned about the best way to assess the proposedlegislation, a congressional requester asked us twoquestions: (1) How effective had prior efforts been toaddress the problem? (2) What implications forstructuring future legislation might be drawn fromexisting knowledge about teenage pregnancy?

The first step of the PES was to clarify the problem inorder to focus the scope of the PES properly. In thisexample, the GAO staff determined that “teenagepregnancy” per se was not the problem, becausepolicymakers were not concerned about births tomarried women under 20. Rather, two problems wereposed in debates: (1) births to teenagers without theresources to support themselves or their children and(2) the negative health and social consequences forboth mothers and infants associated with births tounwed and poor teenagers.

Faced with a subject that has been defined in morethan one way, one can, of course, decide to restrictthe focus of the PES to one definition or another.Following discussion with the customer, we chose todeal with both problems. In effect, this decisionmeant enlarging the scope of the PES to a review ofthe literature addressing both the prevalence ofteenage motherhood and the consequences of thatprevalence. Fortunately, the literature on teenagepregnancies was not ordinarily restricted to one orthe other issue: most sources contained informationrelevant to both. Certain topics that could have beenincluded with the teenage pregnancy problem hadreceived little or no attention. The excluded topics


Chapter 4


also helped define the policy space.1 For example,congressional concern was expressed not about allpregnancies but only about those resulting in livebirths. Ignored in the discussion were the estimated50 percent of the teenage pregnancies terminated byspontaneous or induced abortion.2 Furthermore,interest centered largely on the pregnant women andnot on the presumably teenage males who hadimpregnated them.3 Whether correct or not, theimplicit legislative definition of teenage pregnancy in1986 was as a problem primarily affecting the youngwomen and their children.

Another aspect of defining the problem centered onwho is to be considered a teenager. Clearly, women18 or younger were included by everyone. But somediscussions included all women under 25, whileothers restricted the definition to persons under 20.By agreement with the customer, we focusedprimarily on women 20 or younger.

SelectingAlternatives toEvaluate

The PES does not generate proposals at thebeginning: that is, a proposal has already been made,and the issue is whether it is likely to hit the target, aswe said earlier. Not all proposals are good or equallygood candidates for a PES, however. This step doestwo things. First, it screens out proposals in which aPES is not the right evaluation tool. Perhaps, for

1“Policy space” is within the boundaries of politically acceptablepolicies. Thus, the set of policies enclosed within the policy spaceof any given period consists of all the policies that are acceptable toone or another of the principal political partisans.

2It seemed obvious that a policy of promoting induced abortions asa solution to adolescent pregnancies was clearly outside the 1986policy space.

3There was some concern in one proposal with teenage fathers, butthis was never an important center of attention, although theproblem could also be phrased as lack of family formation or ofresponsibility on the part of the young men. A PES could, at thisstage, compare alternative target definitions in terms of precision,efficiency, and so on.GAO/PEMD-10.1.10 Prospective EvaluationPage 34

Chapter 4


example, the proposal seems to change daily orperhaps we have already reviewed similar proposalsand can quickly draw on our corporate knowledge toprovide comments on likely success.

Second, of the proposals for which the PES is theright evaluation tool, this step selects the optimumideas for review. “Optimum” can include theconsideration of a variety of factors. One is, of course,the specific interest of the customer. Others mayinclude variations among proposals in cost, targetgroups, or the governmental meansproposed—regulatory, categorical, tax policy, blockgrant. For example, proposals to provide long-termnursing care to the elderly could vary notably in cost,depending on such factors as the copaymentsrequired, the conditions covered, and the duration ofcare authorized. Some proposals could cost millionsannually; others, billions. Selection on the basis ofvariation among the proposals could in turn reflectsuch factors as maximum ranges, special interests,and similarity to existing pilot work. The PES shouldbe explicit about the basis to be used, because thechoice made at the end of this step notably affects thescope of the work and the utility of the results. Table4.3 describes this step.


Chapter 4


Table 4.3: Step 2:Selecting Alternatives toEvaluate

Aspect Definition

What “selecting policyalternatives to evaluate”means

A PES usually begins with aspecific proposal whose likelysuccess is to be evaluated. Whatis actually evaluated may differ,however, as a result of activitiesconducted during this step.“Selection” means that at the endof the step, the proposal to beassessed will have beendetermined and alternatives, ifany, will have been selected

Why this step isimportant

Not all proposals are goodcandidates for a PES. Andamong the good candidates, notall may be equal in optimum useof time: it may be more useful topolicy to analyze some proposalsrather than others

The role of this step It helps ensure that the evaluatorwill not be wasting time, and itgives the analyses optimum value

Activities that fulfill therequirements for thisstep

(1) Identification of the politicallyviable alternatives. (2) Screeningto be sure there are no reasons,such as rapidly moving changesor an adequate body of analysesof similar prior proposals, toreject these as PES candidates.(3) Examination of the proposalsthat would be optimum to reviewin depth through the PES,according to criteria such asmaximum differences in proposalcharacteristics. (4) Selection ofthe PES proposals

Why the PES BeginsWith ExistingOptions

For any problem, a large number of potential policiesand programs may be relevant. However, assessingthe full range of possible alternative policies is not the


Chapter 4


concern of a PES. The PES task is constrained by twoprinciples. (1) The task must be restricted to one thatcan be examined by posing the evaluation question,“Is there evidence that a particular program or policywill or will not be likely to meet its stated objectives?”(2) The PES begins with the options thatpolicymakers are already considering in order forPES findings to be useful to them. Thus, this is aprocess that starts with the alternatives underconsideration, then looks for any evidenceconcerning the potential efficacy of thosealternatives, and, only if necessary, generates otheroptions.

It is important to understand the implications ofcentering the PES on existing alternative policies.Another way to proceed would be to make acomprehensive review of all the research andevaluation literature relevant to the problem inquestion, attempting to infer the implications it hasfor policy and designing alternatives ourselves.However, this alternative is rejected in the PESmethod for two main reasons.

First, there may be only a loose fit between researchfindings and policy. It is possible for two reviewers todraw different policy implications from the sameresearch evidence.4 Unless some obvious logical errorhas been made, neither reviewer would be correctand neither would be incorrect in his or her

4For example, given the existence of a large number of teenagepregnancies, one policy alternative would be to conduct campaignsto convince teenagers to have abortions. Another policy that fitsthe data is to conduct campaigns stressing sexual abstinenceamong teenagers. Still a third would be to provide cash bonusesand ongoing subsidies to men who would marry and supportpregnant teenage women, since the underlying problem could beconceptualized as lack of family formation. None of these policiesis “incorrect” in the sense of misinterpreting the basic finding of theexistence of a widespread problem, but, also, none would havebeen relevant to the policy formation process in 1986.


Chapter 4


projection of policy implications. But contradictory oreven equivocal recommendations are difficult to usein decisionmaking.

Second, the PES approach allows the reviewer tomake definite statements that are subject toverification. The outcome of a PES review is anassessment of whether the policy or policies underconsideration are supported or not supported by theexisting evidence. If a PES concludes that proposal Ais justified by the evidence and some othercommentator asserts that it is not, then it is possibleto compare the analytic procedures used by each ofthe disagreeing parties to determine the position thatis justified by the research evidence.

What about a situation in which none of the optionsalready on the table is likely to work? To bemaximally helpful, the PES relies on prior researchand evidence as a way of refining the policy options.If the prior research did not support the optionsunder consideration, then the PES would try toidentify the policy options that were within the mostrealistic range of the research, when the questionswere considered at appropriate levels of complexity.For example, proposed legislation on housing forphysically handicapped adults might focus onincreasing independence for single persons, but theliterature might consistently place greater emphasison group homes or family units.5

Illustration As stated earlier, the PES is intended to weigh howclosely the research and evaluation evidence supportsa proposed policy or one or another of several

5Care must be taken in using prior research to assess its technicalquality, including the independence and objectivity of theresearcher. See our discussion on recognizing threats to objectivityin our transfer paper entitled Case Study Evaluations (U.S. GeneralAccounting Office, 1987c).


Chapter 4


alternative policies. In the case of the teenagepregnancy project in 1986, several alternatives couldbe compared. Twenty-two separate bills regardingteenage pregnancy had been introduced in theCongress, twice the number proposed the year before.For the PES, which had to be completed within 4months, the selection of proposals to consider tookon some importance. Clearly, full consideration of all22 proposals was out of the question.

To aid in the selection of proposals to assess, GAOstaff performed a content analysis of each programproposal, listing its program requirements, includingsuch items as criteria for client eligibility, allowableand required services, and any required administrativearrangements. (U.S. General Accounting Office,1982) This information was presented in tabular formto facilitate identifying the elements that were similarand those that were different across proposals andhow each bill resembled or differed from the others.

With a few exceptions, most of the 22 congressionalbills proposed national programs of assistanceservices exclusively for pregnant and parenting youngwomen. However, the bills differed on the scope ofthe services to be provided, the types of clients whowould be served, and the administrative and financingarrangements that would be required. Therefore,rather than attempt to assess the feasibility andpromise of all possible program options, the decisionwas made, in consultation with the customer, to focusthe PES on those apparently key, congressionallyrelevant dimensions of difference between theproposals—that is, the choices presented to theCongress regarding scope of services, clients, andadministrative arrangements. Picking alternatives thatdiffered widely also would help in the evaluation ofother proposals that differed along the samedimensions.


Chapter 4


In order to further narrow the focus of the PES, GAOstaff, again in consultation with the customer,selected two proposals that embodied these choicesby differing substantially on each of these keydimensions.6 The first proposal was targeted topregnant and parenting teenagers, flexible regardingthe services that should be provided, andadministratively straightforward. Grants would beprovided directly to local agencies that would designand deliver services. In contrast, the second proposalwas more broadly targeted to include economicallydisadvantaged women up to age 25, was highlyprescriptive about services to provide, and wasadministratively complex, requiring coordination withfive other federal programs. This bill also included aproposed program for preventing teenage pregnancy,permitting the PES to address both of the problemsfor policymakers that had been identified at the start.


Chapter 5

The PES: Middle and Final Steps

After narrowing the focus of the problem, we have theremaining tasks of analyzing the chosen bills in termsof conceptual and operational models of the proposedprograms; identifying from those models the targetpopulations and the program features of interest;selecting the appropriate evidence; arraying thatevidence against the models to assess whether theseproposed programs are likely to meet their statedobjectives; and reporting the results.

The ConceptualAnalysis

Underlying Logic The key elements of this step are presented in table5.1. At this point, the evaluation aims at revealing theunderlying logic of the proposal: why, in theory, theproposer thinks it will work. For example, a proposalaimed at reducing urban congestion by subsidies forsatellite location of offices and businesses probably isbased on the assumption that a dispersion of people ispossible and desirable and that for a givencommunity, the primary centralization comes fromcommercial or governmental requirements. Aproposal aimed at reducing urban congestion byincreasing mass transit and reducing individualparking facilities probably is based on theassumptions that dispersion of businesses attractingpeople centrally is not possible or desirable and thatwhat will most motivate people to use mass transit isaversion to high parking-lot prices and having to walklong distances from parking lots to businesses,relative to cheaper, more readily accessible masstransit.


Chapter 5


Table 5.1: Step 3:Conceptual Analysis Aspect Definition

What “conceptual analysis”means

Identification of theassumptions, beliefs,values, and theoryunderlying the proposal:why, in principle, it is likelyto work or not work

Why this step is important Two reasons. First, it helpsset up criteria for figuringout what prior research orprogram evaluation isrelevant: it is the researchon the underlying theoriesor the program whoseunderlying assumptionswere similar. Second, thisstep can identify gaps (orstrengths) in logic thatcould lead to uncertaintyabout program success

The role of this step In scoping, this stepincreasingly targets theresearch that will and willnot have to be examined,and it increases theefficiency of the job


Content analysis of theproposed bill or idea.Graphic techniques arehelpful in efficientlydisplaying the conceptualmodels and checking theaccuracy andcompleteness of ourinterpretation. Can besupplemented byinterviews with sponsors ofthe proposals oracademicians who haveworked on the ideas


Chapter 5


Making the underlying assumptions or beliefs asexplicit as possible helps identify gaps in the logic andhelps focus the subsequent literature search onrelevant prior research or program evaluations.1 Inthe urban congestion example, the literature in thefirst instance might focus on evidence regarding thedispersion assumption and factors affecting businessrelocations. The second instance might focus ourattention on research on individual incentives anddisincentives involving money, convenience, safety,and so on in relation to using mass transit versusindividual cars.

Illustration To assess both the promise and the feasibility of thetwo teenage pregnancy bills, it was necessary tobreak them down into components that could beaddressed as subquestions. This required analyzingthe texts of the two bills to develop two types ofmodel for each proposal: (1) a conceptual model and(2) an operational model. The strategy here was


Chapter 5


similar to that of developing an evaluation design,except that a PES reviews existing evidence insteadof collecting new data.

The conceptual models would answer the followingquestions: What was the problem to be addressed?What was the treatment? (Or what actions would bebrought about by the program?) And what was theintended outcome of those actions? Figures 5.1, 5.2,and 5.3, from GAO’s report, contain the results of thatdisaggregation. (U.S. General Accounting Office,1986b) These models helped determine the previouslystudied programs that should be considered similar tothose proposed and the outcomes that should beexamined when judging their effectiveness. As can beseen from figure 5.1, the first bill had the objective ofreducing the number of unintended repeatpregnancies, while the second bill, whose structure isshown in figures 5.2 and 5.3, articulated a fairlydetailed theoretical model. It proposed to aid youngmothers to avoid welfare dependence by allowing


Chapter 5


them to complete school and gain employment and,thus, the bill specified additional intermediateobjectives.

Figure 5.1: UnderlyingConceptual Model of theFirst Bill

Source: U.S. General Accounting Office, Teenage Pregnancy:500,000 Births a Year but Few Tested Programs,GAO/PEMD-86-16BR (Washington, D.C.: July 1986), p. 16.


Chapter 5


Figure 5.2: UnderlyingConceptual Model ofProgram a in the SecondBill



Chapter 5


Figure 5.3: UnderlyingConceptual Model ofProgram B in the SecondBill



Chapter 5


The OperationalAnalysis

UnderlyingOperations

The operational model of a proposed program showshow to accomplish the goals of the program. Like theconceptual model, it is constructed by a carefultextual analysis of the legislation, but it answers thefollowing question: Who is to be served, by whom,and under what financial and operationalarrangements or constraints? An operational modeldefines the target populations, the intended serviceproviders, the funding sources and amounts, and theadministrative structures that should be the focus ofthe PES.

The details of the fourth step—operationalanalysis—are described in table 5.2. Here theemphasis is not on the “why” of the proposal. It is onthe “how” of the proposal: how the proposed programwould be carried out and how it would operate. The


Chapter 5


methods of operations research come into play in thisstep. The proposals are analyzed to determine who isdoing what, when, and under what circumstances towhom in order for the proposal to be carried out. Thisstep can identify the operational complexities (orsimplicities) in the proposal, the number ofdecisionmakers, and how contingent the final resultswill be on the agreement and coordination of many(or relatively few) actors.


Chapter 5


Table 5.2: Step 4:Operational Analysis Aspect Definition

What “operational analysis”means

Identification of themechanics of the proposal:how it is supposed to becarried out

Why this step is important Two reasons. First, it setsup criteria for determiningthe relevant prior researchor programs or the priorexperience with operationssimilar to those of theproposal. Second, this stepcan also identify gaps (orstrengths) in the proposedprocedures that could leadto more or less certaintyabout program success

The role of this step It sets limits within whichthe search for relevant priorresearch or programevaluations takes place,increasing job efficiencyand completeness


Operations analysis of theproposal. The techniquesof operationsresearch—using thecontent of the proposal toidentify the designelements- -are appropriate.Graphic presentation of theoperation helps check theaccuracy andcompleteness of ourinterpretation. Interviewswith proposal sponsors ordevelopers provide finalassurance of theoperational model’s quality


Chapter 5


The analysis in itself can reveal likely sources ofsuccess or failure for the proposal: gaps, for example,in authority for making decisions or assumptionsabout the availability of resources other than those tobe provided directly through the proposed program.The operational analysis also serves another function:it focuses the literature review on the relevantoperational issues that could affect the success orfailure of the new program. Finding, for example, thatthe operation of one proposal would requireestablishing local stakeholder groups while that of thecompeting proposal would involve using electedofficials would turn attention to relevant priorexperience of the efficiency and effectiveness of thesecontrasting modes of program management andcontrol.2


Chapter 5


Illustration Figure 5.4 shows the operational model constructedfor the second teenage pregnancy bill.


Chapter 5


Figure 5.4: Underlying Operational Model of Program B in the Second Bill

(Figure notes on next page)


Chapter 5



Testing the Model

Two Substeps Testing the model involves two substeps. The firstsubstep—checking the centrality of the assumptionsto be examined in depth—means reviewing with thecustomer the assumptions selected as the focus of thereview of prior evidence. The conceptual andoperational models usually involve many steps, and itmay not be valuable to delve into them all. The


Chapter 5


evaluator selects those that seem to be most pivotalor to offer the most useful contrasts betweencompeting proposals. Discussion with the customer(or the developers of the idea or knowledgeableacademic sources) is a final check that the best pointsof entry into tests of key assumptions have beenselected.

The second substep—testing key assumptions againstexisting evidence—is summarized in table 5.3. Thisstep uses the evaluation synthesis methodology butwith two differences. The first difference is that whatis relevant has been determined through the processof specifying the conceptual and operational modelsand through checking the importance of theassumption to the customer. A second difference is


Chapter 5


that the evaluations are synthesized with respect onlyto the chosen assumptions.

Table 5.3: Step 5: Testing Key Assumptions Against Existing EvidenceAspect Definition

What “testing key assumptions againstexisting evidence” means

A complex body of evidence from priorresearch and program evaluation iscollected, and the key conceptual andoperational assumptions are comparedwith the findings from prior studies todetermine the likelihood of new programsuccess

Why this step is important The conceptual and operational analysescan reveal gaps in logic that are likely toaffect program success. This direct testagainst prior experience, however, is themajor criterion for deciding whether theidea will work. If relevant prior researchand experience indicate that the keyassumptions have worked in the past,then, if conditions are similar, they arelikely to work in the future (similarly, if theyhave not worked in the past and conditionsare to work in the future)

The role of this step It completes the triad of analyses(conceptual, operational, empirical) to givea conclusion on the proposal’s successthat is as solid as possible

Activities that fulfill the (1) Complete identification of relevant priorresearch and program evaluation, (2)assessment of the quality of this evidence,(3) synthesis of credible findings. Theevaluation synthesis method is applied.Systematic tabular or graphic comparisonof the evidence against each keyconceptual or operational assumption aidsthe efficiency and completeness of thisanalysis. Thus, techniques of meta-analysis and multiple case studycomparisons are applicable


Chapter 5


In table 5.3, step 5 is described as completing the triadof analyses. As noted earlier, a central methodologicalpoint in the PES is that the results of three differenttypes of analyses— conceptual, operational, andempirical—are all compared and otherwise taken intoaccount in reaching conclusions, thus strengtheningwhat can be said with some confidence about thefuture. When all three approaches give the sameanswer, we can be more confident about itssoundness. When they differ, as seen fromconceptual, operational, or empirical perspectives, wemust qualify our results in terms of that lack ofreinforcing agreement. Finally, we need to considerways in which the future may differ from the past,identifying, for example, more or less optimisticscenarios for relevant factors. Where the future islikely to be similar to the past on key dimensions, wecan have more confidence about the appropriatenessof the PES to judge the likely success of proposals. Asthe scenarios differ from past or present experience,our certainty necessarily decreases, although we canstill specify conditions under which a proposal ismore or less likely to work.


Chapter 5


Illustration The review of evidence in the teenage pregnancyexample we are following started with a basicquestion: How many people would be eligible for theprograms in the proposed legislation? This was arelatively easy question to answer because of theexcellent demographic data collected by the Bureauof the Census and the National Center for HealthStatistics concerning the number of teenage womenat present, in the past, and in the near future, as wellas birth statistics. Less definitive data were availableon births by socioeconomic level, although severalsurveys were the basis for our estimates. The nextsections give further detail for the illustration.

Estimating TargetPopulation Size

Good estimates of the size of the target population fora proposed program are important for projecting


Chapter 5


program costs. However, the target population is notidentical to client population, since few programs areever able to reach all the eligible members of a targetpopulation. In general, the more complex theeligibility requirements are, the less precise theestimates of client participation can be. An importantdata source can be experience with similar existingprograms. If the clients of an existing program areidentical (or nearly so) with the target population ofsome proposed program, a good basis for suchestimates can be the existing program’s currentnumber of participants. For example, data from stateswith catastrophic illness insurance programsprovided important insights for the PES on theproposed national system. More usually, it isnecessary to synthesize population estimates,combining numbers from census and administrativedata, for example, with information from populationsurveys and research data on the degree ofassociation between eligibility characteristics.


Chapter 5


In this illustration, no existing program served allpregnant and parenting teenagers. It was necessary torely on published tables from the National Center forHealth Statistics on the characteristics and numbersof women giving birth each year by age, maritalstatus, years of school completed, and number ofprevious births. It was possible to add up the numberof first births to women under age 18 over severalyears to calculate the number of young unmarriedmothers who constituted the target population.However, this target population is too inclusive, sincesome of the young mothers are not poor and, hence,would be ineligible for program participation underthe first proposal.


Chapter 5


Unfortunately, the National Center for HealthStatistics collects no information on the incomes ofmothers. To estimate the number of poor youngmothers required using sample survey data andapplying survey findings to the vital statistics. Ofcourse, the potential client populations of proposedprograms are always problematic. Clients should notexceed in number the total target population, butparticipation rates can vary considerably, assuggested earlier.3

Some information on participation rates can beobtained by examining existing programs of a similar


Chapter 5


nature. The next task in the PES was to identify theexisting federal programs with related objectives andtarget populations. This is important for severaladditional reasons. First, there is always an impliedalternative to the proposals being considered, andthat is the status quo, consisting of all the federalprograms already in place. Second, in this instance,information on existing programs would also addressthe feasibility of both the proposed coordination ofexisting services and the proposed funding level.

For example, if a proposed program relies oncoordinating services provided under another


Chapter 5


program or programs, whether those services are infact available becomes crucial information. It iscrucial because if the services funded by these otherprograms are not available, or if the providers arealready operating at capacity and cannot take on newclients, then the new program has to find another wayof providing those services, and it will need additionalfunds to provide them. Further, if existing serviceswere apparently underutilized, a new program mightnot be needed.

The review of existing programs provided littleinformation on what could be expected as


Chapter 5


participation rates in either of the two proposedprograms. The main reason for this disappointingoutcome was that the existing programs were, withone exception, not exclusively targeted at teenagepregnancies but included other target groups as well.

Finding the Studies The next task was to conduct a search for all studiespublished in the recent past (5 years, in theillustration) that evaluated pregnancy preventionprograms and comprehensive service programs for


Chapter 5


pregnant and parenting young women.4 The searchincluded formal publications, such as professionaljournals and monographs, as well as computerizeddata bases, usually containing bibliographic citations,abstracts, and informal (or so-called fugitive)publications, including reports of limited circulationand monographs. It is especially important that everyeffort be made to obtain coverage of the last categoryas wide as possible, since informal publications oftencontain the latest studies, and to collect and notenegative findings, since studies showing positiveresults are more likely to be published than those thatdo not.

4Publication dates in journals can follow the time of data collectionby several years. The studies covered up to a decade of researchprevious to the time of the PES. This time restriction recognizesthat applied social research has only recently been used extensivelyin the evaluation of programs and that the credibility of remotedata is slight for reason of age alone. For example, data on theeffectiveness of the Great Depression programs, such as theCivilian Conservation Corps, are not likely to be viewed as relevantto similar contemporary programs. However, for some programs,time restrictions may be much looser. For example, in a PES on jobtraining, studies that are a decade or two old may not be seen asirrelevant, especially if studies over time are quite consistent intheir findings.


Chapter 5


To obtain information on all relevant evaluationstudies, it is usually necessary to rely on personalcontacts with knowledgeable persons. This cannormally be accomplished by sending out lists ofpublications already located and asking for the list tobe supplemented by other publications known to theexperts.

In this case, all the studies—whether containingoutcome evaluations or not—were reviewed foranalysis of program costs, their sources of funding,and implementation problems. The information onwhere individual projects gained their funds alsoaugmented the information on existing programs andservices collected earlier on the federal level. Articlesabout program failures can provide invaluableinformation that gives balance and perspective to theinformation gained from successes. For example, theymay give clues as to the staff, public relations, clientrecruitment, or support services required for theproposed programs to operate as intended.

Special attention was paid to publications containingoutcome evaluations. Each publication was readcarefully to ascertain how closely the programs inquestion resembled the proposed programs, and asuccinct summary of each program was prepared.The outcome variables used in the evaluation werenoted separately, particular attention being paid tothe quality of the “impact assessment” data. The endresult of this careful examination was a profile foreach evaluated program, recorded in tabular form,containing the crucial information on programdescription, outcomes, and ratings of data quality.Appendix III gives an example of one such profile.

As mentioned earlier, it is important to bring to bearon the literature the same conceptual framework usedin examining the proposed programs. For each article


Chapter 5


describing a program and its evaluation, a profileform was filled out, characterizing that project’sclients, services, and administrative arrangements.The categories were the same as those developed inthe analysis of the two legislative proposals, in orderto ensure that the derived information was directlyrelevant to the consideration of the proposedprograms.

Quality Assessment The most technically demanding aspect of the reviewof each evaluation was assessing the quality of theinformation. Since this task is essentially identical tothat confronted in the evaluation synthesis, GAO staffborrowed from criteria employed in previoussyntheses. Each evaluation outcome, as defined bythe conceptual models of the programs beingexamined, was treated separately. The evidence oneach objective was rated separately. An evaluationmight provide evidence of adequate quality on one ofthe outcomes of interest but not on another, because,for example, of the use of different data collectionmethods. Other outcomes that were not of directconcern in the conceptual models of the programsunder scrutiny in the PES were also noted, along withassessments of the quality of the evaluation evidenceused.

Criteria The quality-rating criteria used in the assessment ofeffectiveness evidence have to be tailored to someextent to the issues involved in the PES. Nevertheless,the criteria are largely the same from PES to PES. Inthis case, criteria centered primarily on the internalvalidity of the research design used in arriving at


Chapter 5


effectiveness estimates.5 Most of the evaluationstudies had used longitudinal comparison groupdesigns, making the composition of the comparisongroup critical. (Appendix II presents more detail oncriteria.)

The criteria included (1) appropriateness of thecomparison (or control) group; (2) sample sizeadequacy, including attrition among clients andcomparison group; (3) standardization of datacollection, including measures of data reliability;(4) validity of measures used to represent outcomevariables; and (5) appropriateness of statisticalmethods used, especially those used to enhance theinternal validity of effectiveness estimates, by testingfor competing explanations of estimates.

The assessment of data quality requires some trainingin evaluation design, measurement, and statistics aswell as some understanding of the substantive area.Several readings are often required. For example,sometimes the fact that there is anything wrong witha particular measure of a variable is not obvious untilanother study has been examined that is more carefuland accurate in its measurement strategy. It may

5Internal validity refers to the attribution of cause and effect;external validity, to the ability to generalize. An “ideal” designwould offer strong evidence that effects, if any, stemmed from theprogram (or event being studied) and would be obtained fromgroups and in situations as similar as possible to the whole range ofcircumstances in which the program was being applied. Further,this ideal design would be appropriately sensitive, able to detecteffects of a size believed worth the costs of the program. Someexperts believe the controls necessary for internal validity severelylimit external validity, and they argue that for policy purposes,external validity, with its implications for extrapolation, is mostimportant in judging quality. Other experts are more sanguineabout optimizing both or place heavier emphasis on internalvalidity. We thought the question with top priority for thisparticular PES was evidence of any effects, and so we focused onthat aspect of design. For some other PES, different criteria mightbe weighted more heavily, a point discussed in more detail inappendix II.


Chapter 5


often be necessary to read the set of evaluationstudies several times before a final quality reading canbe arrived at.

Reliability As in other rating tasks, it is necessary to test thereliability of the ratings (that is, their replicability, orlikelihood that our reviewers will reach the samerating conclusions) by ensuring that there will be atleast two readers for at least a subset of evaluationstudies. If a subset is used, the reliability checkratings should be done early, midway, and late in thecoding process to avoid rater-drift and generalfatigue. Discussion among raters concerning theirdisagreements on the subset often brings to lightcritical characteristics of studies that were notimmediately discerned.

Aggregation Although it is possible to arrive fairly easily at areliable and credible rating for each criterion, arrivingat an overall quality rating is usually more difficult.Much of the problem encountered in developingoverall assessments for the teenage pregnancy studyarose because many reports did not provideinformation with which to judge the adequacy of theevaluation on one or more of the criteria. In theabsence of direct evidence, it is possible to judge theevidence only questionable, unless some other pieceof information suggests that the absence ofinformation stems from some serious flaw.

In addition, many evaluation studies provide data onseveral evaluation outcomes, each outcome varying inthe quality of evidence presented. It would be amistake to discount entirely a study that contains anacceptable evaluation of one outcome and a poorevaluation of another. For these reasons, rather thanoverall quality ratings for each evaluation study, eachoutcome was presented separately along with qualityassessments of each outcome.


Chapter 5


Presenting theResults

Product Type Presenting the results of a PES differs frompresenting the results of an evaluation synthesis. In aPES, the underlying conceptual and operationalmodels have to be identified, the key assumptionshave to be highlighted, and the evidence has tosummarized in relation to these assumptions. Incontrast, an evaluation synthesis arrays the evidencein relation to the questions to be answered, and theunderlying models need not be explicated. Table 5.4summarizes the elements of step 6.


Chapter 5


Table 5.4: Step 6:Presenting Results Aspect Definition

What “presenting results”means

Presentation of theconceptual and operationalmodels (usually in graphicform) and of the results ofthe comparison of keyassumptions and evidenceconcisely and clearly

Why this step is important The PES involves anuncommonly detailedanalysis of a proposal. Thecredibility of the resultsdepends in part on thereader’s being able tofollow the PES procedureseasily and to see in detailhow the findings havedeveloped

The role of this step Promoting credibility andmaking our conclusions assimple, clear, andaccessible as possible

Activities that fulfill therequirements of this step

Development ofappropriate graphics andtables; preparation ofnecessary technicalappendixes (for example,details on procedures usedto rate the quality of priorevidence and to aggregatefindings)

Table 5.4 emphasizes the value of tabular and graphictechniques. The result of a PES might look more like abriefing report than a chapter report. This would vary,of course, in terms of length, depth, whether or notrecommendations are provided, and our other usualcriteria for deciding on product type.


Chapter 5


Illustration The results of outcome evaluations are typicallypresented in tabular form, as shown in table 5.5,where some of the findings from the teenagepregnancy PES assessment are presented. Table 5.5was designed to draw the reader’s attention to severaldifferent things. Across the top are the explicitobjectives of the legislation plus some others thatwere found to be important in the field. Along the sideare program types generated by clustering studiesaccording to similarity with regard to the servicesthey provided. In the body of the table are thedescriptions of the studies’ comparison groups andthe results, expressed as whether the program group“did better” than the comparison group at astatistically significant level. The boxes representfindings we considered to be most methodologicallycredible. All this information was transcribed fromthe rating sheets.

A summary table such as table 5.5 providesinformation on how many studies addressed eachparticular outcome, how much of those data arecredible, and the types of programs that had effectscompared to other conditions or programs. Theinformation is presented in narrative rather thannumerical form. While this was an appropriate way topresent the findings, an alternative would be to reporteffect sizes. Where there are quite a few studies withrelevant results—and particularly where theprograms’ clients can be grouped by factors such asage, race, education, and family income, which wouldbe expected to influence the outcome variables—aquantitative presentation can be efficient andeffective.

Note that in table 5.5, comparison groups aredescribed in detail. This is also critical information,because some evaluations compared the program tonothing more than ordinary prenatal health care,


Chapter 5


while other studies compared their program with onethat was only slightly different from it. The presenceand absence of effects under these types of testcondition are thus difficult to assess. That is, ahigh-quality test of a program includes the essentialelements of a high treatment strength and a strongbasis for causal attribution. This point became aconclusion of our illustrative PES: few programs hadbeen adequately tested, a wide variety of programsappeared successful, and both comprehensive andless-comprehensive programs appeared to have beensuccessful.

More specifically, the findings of the illustrative PESwith regard to the requester’s questions weresummarized as follows.

1. The pattern of credible results showed no clearpreference between the two proposed programs. Avariety of past programs appeared successful, butthere was little information on the components thatwere responsible for their apparent success. Andthere was no convincing evidence that the mostcomprehensive service packages were more effectivethan the least comprehensive.

2. Implementation analyses suggested that there werecertain avoidable operational problems associatedwith the proposed administrative structures. Forexample, program administrators as well asevaluators frequently mentioned complexcoordination arrangements as a significant obstacle toprogram success.

3. Therefore, if the Congress wanted to initiate anationwide program, then the administrativelysimpler model might have a greater chance ofsuccess. However, we concluded that the evidencewas most


Chapter 5


Table 5.5: Example of Presenting PES Findings a


Chapter 5



Chapter 5


aThis is one segment of a longer table. It illustrates anintermediate summary of findings by offered services. Thetable included verbal and graphic material. The “C7” and theother code numbers in the second column refer to fullbibliographic data for each comparison in the teenagepregnancy report from which we have taken the table.


consistent with initiating a large-scale demonstrationprogram that would systematically test the feasibility,costs, and benefits of different approaches toreducing teenage pregnancy.6

In this particular instance, the conclusions did notclearly favor the legislative proposal that wasprescriptive (given the lack of strong evaluativeknowledge) and relied on existing services (given pastexperience with complex coordination processes). Inaddition, the smaller, more flexible proposal had totake into account the need to develop informationabout which strategies work with which teenagers.Thus, no clear advantage adhered to the onecompared to the other. This is not always the case fora PES and, in fact, did not occur in another exampleof the method dealing with catastrophic health

6Two options were suggested as consistent with the analyses. (1) Ifexpansion of available services is wanted, then it would makesense to target services to the teenagers who are at highestrisk—young and unmarried teenagers—to allow flexibility in thetype of services provided and to have a simple administrativestructure. (2) In an alternative to a program of expanded services,the federal government could take the role of promoting innovationand ensuring both sound comprehensive evaluations of theinnovations and dissemination of the programs (or theircomponents) that have been shown to work.


Chapter 5


insurance proposals.7 However, the importance of theteenage pregnancy example is real in that it savedtaxpayer resources, since neither proposal had beenintroduced in a form that was likely to succeed.

7See U.S. General Accounting Office, 1987f. In this report, welooked at six legislative proposals for protecting Medicare enrolleesfrom the financial hardships that often accompany catastrophicillness. Our review, and in-depth analysis of two of these six,determined that while protection would increase, some gaps wouldremain. We further identified issues requiring additionalconsideration, such as coverage of prescription drugs. Ourconclusions played a significant role in both hearings and thesubsequent configuration of the act.


Chapter 6

Variants of the PES

Several variants of the PES are possible. They are oftwo types. The first variant derives from targeting thePES: customer interest in special aspects of aproposal. The second variant involves combining thePES with sources of information other than priorwritten evaluations. Using multiple methods would, ofcourse, notably expand the range of the PES. Further,it is typical of designs for many of GAO’s important orcontroversial jobs that we use several methods, sothat the limits of one are offset by the strengths ofanother.

Targeted PES The basic model of the PES we described in chapter 3is appropriate when relatively well articulatedproposals have been developed. However, the PEScan be helpful in other, more limited situations, aswhen a problem is being defined or when costs are ofparticular interest. In essence, aspects of the full PESdiscussed earlier become the target of more limitedwork. Table 6.1 summarizes some of the variants ofthe PES.

Table 6.1: Targeted PESand Related CriticalIssues

Target Critical issue

Problem definition Determining the fit betweenthe perceived problem andlegislative proposals

Problem characteristics Assessing data quality andnarrowing or resolvingcontradictory estimates

Relation of proposal toprevailing scientific models

Clarifying underlyingassumptions

Assessing projected costs Checking sensitivity ofprojections against varyingassumptions

The PES andProblem Definition

For many issues that come before a legislative body,some critical problem has been identified by the


Chapter 6

Variants of the PES

proposers of legislation, along with suggestedmeasures expected to resolve the problem. If theproblem is a major one, it is rare that only one pieceof legislation will be proposed. Even in such cases, asalready noted, every proposal has an implicitalternative—namely, not to enact any legislation atall. In any case, before a judgment can be made aboutwhether the proposed measure will resolve theproblem, it is important to be clear about exactlywhat the problem is.

Proposed legislation designed to address a particularproblem is necessarily based on some definition orunderstanding of the issue involved. For example,two contending legislative proposals may both beaddressed to the issue of homeless persons, oneidentifying the homeless as needy persons who haveno kin upon whom to depend and the other defininghomelessness as the lack of access to conventionalshelter. The first definition centers attention primarilyon the social isolation of potential clients, while thesecond focuses on housing arrangements. It is likelythat the ameliorative actions that follow will bedifferent, as well. The first might emphasize aprogram to reconcile estranged persons with theirrelatives, while the second might imply a subsidizedhousing program. Thus, the two definitions lead todifferent proposals.

Especially critical in problem definition is the fitbetween what is perceived to be the problem by thosewho have pressed for attention to the issue and thedefinition in the legislative proposals. In thisconnection, the PES evaluator would ordinarily referto legislative proceedings, including committeehearings and floor debates, journals, newspaper andmagazine editorials, and other sources in whichdiscussions of the problem may appear. The purposeof this review of sources is to examine how the


Chapter 6

Variants of the PES

problem has been formulated and to state as clearlyas possible the range of politically acceptablealternatives.

ProblemCharacteristics:Density andDistribution

To design a public program properly and to project itscosts reasonably well, good information is needed onthe density, distribution, and overall size of theproblem. For example, in providing financial supportfor emergency shelters for homeless persons, it wouldmake a significant difference if the total homelesspopulation is 2.5 million or 250,000 (both estimateshave been advanced). It would also make a differencewhether the problem is located primarily in centralcities or can be found in equal densities in smaller andlarger places.

An identified problem is often a complex mixture ofrelated conditions; for planning purposes, specificinformation is needed about that complexity. In theexample of homelessness, the proportions of thehomeless suffering from chronic mental illness,chronic alcoholism, or physical disabilities has to beknown in order to appropriately design the relevantmixture of programs.

It is much easier to identify and define a problem thanto develop valid estimates of its density anddistribution. For example, only a small handful ofbattered children may be enough to establish that aproblem of child abuse exists. However, to know howgreat a problem is and where it is locatedgeographically and socially involves detailedknowledge about the population of abused childrenand its distribution throughout the politicaljurisdiction in question. Such exact knowledge isordinarily much more difficult to obtain with the kindof precision that may be needed.


Chapter 6

Variants of the PES

To collate and assess whatever information exists onthe issues in question, evaluators need to use whatthey have learned from the literature (consisting ofgovernment reports, published and unpublishedstudies, and limited-distribution reports) and theirunderstanding of the designs and methods that lead toconclusive results. Equal emphasis is given in the lastsentence to “collate” and “assess.” Unevaluatedinformation can often be as worthless as noinformation at all.

For some issues, existing data sources may be ofsufficient quality to be used with confidence. Forexample, an issue on which measurements areroutinely taken by either the Current PopulationSurvey or the decennial census is typically an issueabout which accurate and trustworthy knowledgeordinarily may be obtained from those sources. Datafrom some other statistical series, such as thosepublished by the Bureau of Labor Statistics, also fallinto the trustworthy category. But when we deal withdata produced by other sources, it is necessary toexamine with care how the data were collected.

A rule of thumb is that for any subject, existing datasources provide contradictory estimates. But evenchaos can sometimes be reduced to some order.Seemingly contradictory data on the same topiccollected by opposing stakeholders can be especiallyuseful for assessment purposes. For example, boththe Coalition Against Handguns and the National RifleAssociation have sponsored sample surveys of theU.S. population concerning their approval ordisapproval of gun-control legislation. Although thetwo reports issued by the coalition and theassociation differed widely in their conclusions, theone finding much popular support for more-stringentgun-control measures and the other the opposite, aclose inspection of the data showed that many of the


Chapter 6

Variants of the PES

specific findings were nearly identical in the twosurveys. The findings upon which both surveyssubstantially agreed can be regarded as having thegreater credibility.

Relating ProposalModels andPrevailing ScientificModels

Whether explicitly intended or not, legislative andother proposals are based on some set of ideas ormodels of how the problem in question may havearisen and how it is currently sustained. For example,one welfare reform alternative suggests extending toall states the coverage of public welfare to intactfamilies with unemployed parents in order to reducethe number of households headed by women. Thisproposal may be based on a model that sees currentwelfare policies as penalizing marriage, since benefitsto a woman and her children would stop uponmarriage.

An alternative welfare reform proposal might suggestthat benefits be continued upon marriage but reducedby some proportion to avoid subsidizing parasiticmarriages. Both proposals involve extending benefitsto intact families, one to support such families whenboth parents are unemployed and the other withoutregard to the employment status of a new parent.Each proposal is based on different models of howpayments might affect marriages of householdsheaded by women. In the first case, the proposal isbased on the idea that women will avoid marriage tounemployed men because they would lose theirbenefits, and it ignores the effects that marriage to anemployed man would have. The second proposal isconcerned with the possibility that the continuationof benefits after the marriage of a woman head ofhousehold might render the woman susceptible tomarrying a man who was primarily interested insharing her benefits.


Chapter 6

Variants of the PES

Both proposals are based on models that stress therole of economic incentives in marriage formation, atopic that has received considerable attention inmicroeconomic theory, econometric research, andsocial psychology and sociology. An appropriatetactic for the PES would be to review this literature,seeking to establish two things: (1) the extent towhich experts agree and (2) the existence ofempirical evidence concerning the intended effects ofeither proposal. A thorough review of the existingliterature accompanied by consultation withsubject-matter specialists and knowledgeablepractitioners could determine that one of theproposals has more support than the other, that thereis as much evidence for one as for the other, or,alternatively, that neither proposal has much positivebacking in research and experience.

An important opportunity is presented when a PESfinds that there are very few or no previousevaluations that are relevant because the proposedprogram is a notable departure from programsevaluated in the past. A clear message can be sent todecisionmakers that their proposals go far beyondfirm knowledge and are, hence, subject to amore-than-ordinary risk of failure.1 This advice neednot be an admonition to stick to the programs of thepast. For example, the advice may be to funddemonstration projects incorporating the newproposals rather than to fund fully operationalprograms. Pointing out areas on which existingknowledge has nothing to say may be as important forthe avoidance of public policy failures as gathering arich harvest of firm knowledge.

1We would need to take into account that not acting carries its ownrisks of failure. For example, while we may have little certaintyabout effective AIDS prevention measures, not making the bestefforts we can also incurs risks.


Chapter 6

Variants of the PES

Assessing ProjectedCosts

Legislative proposals are often accompanied byprojected costs. In fact, all the bills that are reportedout of committee include a Congressional BudgetOffice cost estimate. Although any projection can beeasily upset by subsequent actual experience, it isusually possible to make a viable assessmentconcerning whether projected costs are based uponreasonable and likely assumptions. For example, theprojected cost of a proposed measure that wouldsubsidize flood insurance for structures built on floodplains can be profoundly affected by assumptionsmade about the number of structures that are to becovered and the participation rate among potentiallycovered households.

If the flood plains are defined as areas within a100-year flood zone—where a major flood is expectedat least once every century—coverage will be greaterbut flood incidence will be lower than if the limits ofthe flood plain were defined as a 20-year flood plain.If all the applicable property owners participated,anticipated costs might be more than if theparticipation rate were much lower. But there are alsoother complications that affect cost. If only theproperty owners who were close to the source offloods signed up, then the subsidy costs might begreater than if participation rates were more uniformover the flood plain.

A PES can help assess cost projections by judgingwhether the appropriate assumptions have been madein their construction, as well as by proposingalternative assumptions. Here the statistical analysistests how responsive the projections are to alterationsin the assumptions. It raises questions like how muchcosts would be changed if participation rates werechanged by a given amount or if unit prices ofservices were changed. Sensitivity analyses highlightthe assumptions concerning the costs that are the


Chapter 6

Variants of the PES

most critical to the overall cost estimate. Further, aspart of the PES analysis, estimates of the magnitudeand direction of the problems of under- orovercosting that were identified could be applied toexisting information and synthesized into ameaningful range.

Variants UsingOther Sources ofInformation

The basic PES operational model uses priorevaluations or research as the source of information.If, in reviewing this literature, tradeoffs should bemade between timeliness and comprehensiveness,strategies such as sampling and time-limited searchescould be adopted. There may be situations, however,when available information must be supplementedwith some original data collection and when it may bemore efficient to tap into existing knowledge throughpanels or expert judgments. Further, there may besituations where the PES is combined with originaldata collection and other audit work.

Combining the PESWith Some OriginalData Collection

The results of the PES may be supplemented withsome original data collection, such as examination ofagency records or surveys. That is, where existingdata are insufficient and where time and resourcespermit, evaluators may want to use PES proceduresup to the point of matching evidence and keyassumptions. At this point, the PES could proceed ondual tracks with some highly targeted new data beingcollected while other, prior work is reviewed. Severalof the reports already mentioned, such as one on theconsequences of opening more combat supportpositions and units to women, involved multiplemethods of data collection in answering a prospectivequestion. (U.S. General Accounting Office, 1988h)

For example, we were asked by the Congress todetermine what might be learned from state and local


Chapter 6

Variants of the PES

experience in addressing mandate burdens. A lawalready in place since 1981 required the CongressionalBudget Office to estimate such costs for proposedfederal legislation. Similar requirements for reviewingthe costs of proposed state legislation exist in 42states. New legislation proposed by the congressionalrequesters would have required federalreimbursement for additional costs. This approachwas already in use in 14 states that reimbursed localgovernments for burdens imposed by new state laws.(U.S. General Accounting Office, 1988n) The methodsfor answering the prospective question included areview of the literature, analysis of relevant bills, andvisits to 8 states selected by searching prior studies,plus a telephone survey. Data from the 8 states weresupplemented by questionnaires for state officials,state legislative leaders, and relevant interest groups.Using evidence from these 14 states, we found thatestimating and reimbursing costs have had only alimited effect on the burden of mandates, except insome special circumstances.

When may such original data collection beparticularly valuable? One might expect that in areassuch as defense and tax policy, our unique access todata is likely to mean we would have betterinformation than one could expect to find in thepublished literature. In other areas, however, such ascertain aspects of health that require confidentiality indealing with patients’ records, physicians who arealso evaluators and researchers might have therelative advantage and would find a richer data basein published reports than we ourselves might be ableto collect. That is, combining the PES with otherforms of audit and evaluative work is consistent withthe multimethod approach we typically use. However,evaluators planning a PES can also anticipate, to acertain extent, where we may find a relatively richdata base and where our unique authorization may


Chapter 6

Variants of the PES

suggest the need for new data collection tosupplement the PES.

Combining the PESWith ExpertJudgment

The evaluator supplementing other evidence with theviews of experts must be aware of the requirementsof systematic methods such as Delphi techniques.Properly applied, these systematic methods yieldinformation that differs in some key ways from theanecdotal evidence on which congressional testimonyis often based. First, the effects of “charisma” inpresenting testimony are ruled out. Second, since thesame questions are usually asked of many keyinformants, it is possible to determine what opinion isgenerally held. Third, the bases for opinions arebrought out and can be compared objectively withavailable evidence. Fourth, the experts or keyinformants can be selected primarily or solely byconsiderations such as knowledgeability andappropriate diversity.

We have used expert judgment and panels in a varietyof ways to answer prospective (and alsoretrospective) questions. For example,

• to assess major welfare reform proposals dealing withcase management, contracts between welfarerecipients and agencies, coordination of services, andtarget populations, HRD contracted for two panels ofexperts. One panel consisted of experts at thenational level and was convened by the NationalAcademy of Public Administration; the other panelconsisted of experts at the local level and wasconvened by the Federation for Community Planning.The findings of both panels were synthesized by GAOand the numerous concerns, observations, andrecommendations were presented to the Congress asthe insights of expert panels. (U.S. GeneralAccounting Office, 1988b)


Chapter 6

Variants of the PES

• to examine the probable effects of legislation thatwould change the conditions for legal immigration,we identified (in consultation with the customer) theissues and we brought together a panel of experts.The experts identified the highest-quality datarelevant to these issues and presented their ownconclusions. We then independently assessed theconclusions, relative to our own judgment of thequality of the evidence, in order to report thesoundest available statement on probable effects.(Chelimsky, 1989)

The use of expert judgment to supplement ourprospective work requires (1) clarity in presentationwhen we are relying primarily on the opinions ofothers and (2) careful planning when the experts are asignificant source but our own, independent judgmentis needed. In the instance of proposed immigrationlegislation, the experts helped sharpen the issue,identified relevant empirical data, and examinedpoints of consensus and dispute in the interpretationof the data. We then independently reviewed theavailable information and reached our ownconclusions by the usual standards of audit andevaluation work. (Chelimsky, 1989)

In another instance, GAO had a problem-definitionassignment—examining the nature and extent ofsweatshops in the United States and identifying thepolicy options that might help control the problem.2

In this study, which was clearly entitled opinions onthe extent of the problem and possible enforcementoptions, we reviewed the relevant literature onsweatshops, particularly with regard to their originand efforts at control; developed a working definition(since the term is not defined in federal statutes orregulations) in agreement with the customer;

2See U.S. General Accounting Office, 1988j. This was not formally aPES but illustrates a multimethod approach to analyzing a problemand possible action.


Chapter 6

Variants of the PES

interviewed federal, state, and local officials,researchers, and union and management experts;surveyed state labor departments and agencyofficials; investigated possible sweatshops in NewYork and Los Angeles; and analyzed federalinspection reports. While this required more effortthan might usually be available for a PES, it illustratesthat for certain prospective questions, GAO cannegotiate with the congressional customer the time toundertake quite extensive involvement of experts, aswell as site visits, to supplement the literature.


Appendix I

A Brief History of the PES and SomeOther Prospective Methods

This appendix helps place the PES in relation to othermethods. Traditionally, the basic concepts ofevaluation have been used primarily in theassessment of policies and programs that are alreadyin place. This ex post application has become socommonplace that it is the one most frequentlyassociated with evaluation. Less frequently,evaluation methodology has been used to assess exante the potential success of policies that are underconsideration.

The conventional approaches to prospectiveevaluations have ranged widely from relativelyfreewheeling “demonstrations” to highly controlledfield experiments. However, most proposed programsare put into operation—often nationwide—with littleevaluative evidence attesting to their potential forsuccess. (Some of the unevaluated programs thathave been put in place have to do with recent druglaws, various regulatory programs targeting improvedhealth, “deinstitutionalization,” “the strategic defenseinitiative,” “pilot cities,” “impact cities,” “modelcities,” “operation push,” and “operationbreakthrough.”)

But even when small-scale pilot efforts of anexperimental sort are implemented—and mostevaluators would agree that highly controlled fieldexperiments yield the most credible results—theexperiments have many practical drawbacks.1 Inparticular, three serious limitations must be takeninto account when they are considered for use as the

1Pilot and experimental studies can provide crucial intellectualcapital on which synthesis draws. They are among the primarysources of information which the PES relies. That is, a PES benefitsfrom having available a good fund of knowledge based onevaluations of other programs, research knowledge, and so on.Thus, the PES does not replace the new data collection forms ofprogram evaluation. Pointing out the limitations of pilot andexperimental studies should not be misconstrued as arguing againstthis valuable prospective method.


Appendix I

A Brief History of the PES and Some

Other Prospective Methods

only application of evaluation methodology to theassessment of prospective public policies. Consider,for example, three randomized public policyexperiments: the five income-maintenanceexperiments, the housing allowance experiments, andthe several experiments on demand pricing ofelectricity. First, they were costly. On this groundalone, it would not be likely that more than a smallhandful of experiments could be set under way duringany decade. That is, only a minute proportion of thepublic policies and programs that are in any currentpolicy space could possibly be assessed through fieldexperiments.

Second, these field experiments were limited to theconsideration of only a narrow band of alternativepolicies. Indeed, none of the income maintenanceexperiments came close to testing the actual publicwelfare policies that were considered by the Congressand the executive branch in the years since theircompletion. Policy space tends to be occupied bymore contenders than can easily be accommodated inthe design of the typical field experiment.2

Furthermore, with every new administration orsession of the Congress, the contending policies andprograms, as embodied in various versions ofproposed legislation, are never a static body and mayin fact be constantly changing.

Third, public policy experiments take a long time tocomplete. Legislative proposals are often decidedwithin the space of months and, at most, a few years.Clearly, field experiments that take 5 years to run andanother 3 to analyze can rarely speak directly to anyset of specific, proposed laws for the many years thattypically pass before results appear. To some degree,

2This does not mean that the field experiments were irrelevant.Almost all the proposed welfare reform measures involvedwork-leisure tradeoff issues, a topic about which the five incomemaintenance experiments have much to contribute.


Appendix I



these deficiencies are also characteristic of otherprospective efforts.

Pilot demonstrations that call for the collection oforiginal observations in the field may take almost aslong to carry through to completion as fieldexperiments. Even cross-sectional surveys takesignificant periods of time. For example, a nationalhousehold sample survey ordinarily takes from 6months to up to 2 years to complete (depending onthe complexity of sampling and analysis). In short,although “demonstrations” and quasi-experimentaltrials of prospective policies may take less time toconduct than the classical field experiments, they stillmay require more than several years to complete. Inaddition, they share the other drawbacks outlinedabove, being expensive and subject to increasingirrelevance with changes in the policy space.

In sum, the traditional ways in which evaluators havefaced the problem of providing information todecisionmakers on the potential for success ofpolicies and programs that may be underconsideration at any time are not useful to adecisionmaking process that may take no longer thana year or two from proposal to definitive action. Ifevaluations are to contribute to decisions aboutproposed new programs, the contribution should beaccomplished through procedures that are relativelyinexpensive, speak to each of the variety of proposalsunder consideration, and provide timely results.

There is nothing especially new or startling about thisidea, and many evaluators have given the problemsome thought. A relevant example is an application ofevaluative techniques to proposed legislation thatadvocated the use of national health screening foridentifying abused children. (Light, 1973) TheAmerican Evaluation Association has identified


Appendix I



front-end analysis as a major focus of attention.(“Evaluation . . . ,” 1982) Indeed, even the moreextended forms of evaluation, such as randomizedfield experiments, could benefit from a PESconducted at the point of design. And there have beenother efforts in recent years to come to grips with theproblems of timeliness that are inherent in suchfront-end analysis. Many of the specific elements ofPES have been advocated by others. In particular,evaluability assessment as advocated by JosephWholey emphasizes the construction of underlyingmodels of proposed programs in order to assesswhether a program or policy can be evaluated foroutcome effectiveness. (Wholey, 1977) In addition,many others stress the importance of the theoreticalunderpinnings of prospective programs. (Chen andRossi, 1983; Wang and Walberg, 1983; Gottfredson,1984; Finney and Moos, 1984; Weick, 1980)

The main strength of the prospective evaluationsynthesis is that because it draws upon existingknowledge and research to assess the potentialsuccess of a new proposal, it can be timely enough tobe used within the policy development process. Thatis, the PES will not necessarily provide the bestpossible information that could be obtained underoptimal conditions, but it can provide in a timelymanner the best possible information that is currentlyavailable.


Appendix II

Data Quality Judgment Models

The PES relies primarily on the results of pastevaluations of previous or existing programs. That is,the results of a PES could be notably different ifdifferent rules were used for including a given study.Because the weighing of criteria used to judge thequality of prior studies is so critical to the results of aPES, this appendix discusses in some detail a pointnot elaborated upon in our paper on the evaluationsynthesis: how criteria are aggregated in reaching adecision of whether to use (or how much emphasis togive) a specific study. There are at least four differentways to assess the quality of prior evaluation studies.Table II.1 summarizes the advantages anddisadvantages of these four approaches.1

1We also note the special case of where quantitative estimates arerequired as part of a PES. In this instance, careful attention shouldbe paid to the adequacy of our estimates of values that go into thePES analysis, including an examination of the quality of the dataand methods for checking their validity. If data are not of truly highquality, provisions for boundary or sensitivity analyses should bemade. Further, any time the functions we have to deal with arelikely to be multiplicative rather than additive, the accuracy ofvalues entered into the analysis is critical, particularly in goingfrom local to national estimates. The PES could identify points atwhich data must be aggregated and could identify the vulnerabilityto multiplicative effects, where it is not possible as part of the PESto make these better estimates ourselves.


Appendix II


Table II.1: Advantagesand Disadvantages ofFour Data QualityJudgment Models

Model Advantages Disadvantages

One criterion Maximum numberof prior reportsbrought to bear

Large number ofreports permitstests ofinteractions

Analysis may bequicker sincetime for multiplequality screens isnot taken

One strong reportmay be a betterguide than 20weak ones

One criterion isunlikely to beadequate, andinteractions ofdata of mixedquality may bemisleading

Equally weighted If all criteria are infact equallyimportant, thismodel may bestrepresent thequality of the priorevaluations

Permits direct testof whether takingquality intoaccount wouldmake a differencein the findings

When severalcriteria arerelevant, one mayhave little toanalyze if athreshold for all isset, but notsetting athreshold maypermit a modeststrength to offseta serious flaw in astudy

Rare to find allcriteria equallyimportant

(continued)


Appendix II


Model Advantages Disadvantages

Unequally weighted Better representsrelativeimportance ofdifferent criteria

Permits direct testof whether takingquality intoaccount wouldmake a differencein the findings

A modeststrength in onesignificantcriterion can stilloffset a seriousflaw in anothercriterion if thereare two or moreheavily weightedcriteria

Can becumbersome toassign andcompute weightsfor each criterionfor each study, aswell as to makeratings on eachcriterion on eachstudy

Threshold or fatalflaw

Efficient infocusing on mostcrucial criteria

Ensures that astudy with highscores on severalrelatively minorcriteria but aEnsures that astudy with highscores on severalrelatively fatalweakness in oneor more crucialcriteria is notincluded

Must be sure thefatal flaw issufficientlyserious to be ascreen ruling outstudies thatotherwise arepotentially useful


Appendix II


One CriterionOnly

In this method, the set of prior research andevaluation studies on the general topic isdeveloped—say, on food-stamp participation, militarybase closings, the effectiveness of federal programsaimed at disseminating knowledge, or the quality ofexecutive and managerial personnel. The set isexamined against a single criterion.

For example, a decision might be made that only onecriterion such as measure validity should be reallyimportant for the job. This might be true if we areasked to assess the probable cost of a certain type ofchild care. Prior evaluations of child care that did nothave information on costs that we consideredcomplete and properly measured would be rejected.Those with valid cost information would be retained.

Except for the one selected criterion, other aspects ofthe quality of the relevant reports are not assessed inthis method of synthesis. Rather, “strength throughnumbers” is the intention, with the notion that thelargest possible set of prior studies that meet theselected criterion will offer the soundest guide toanswering the question. In a variant of this method,the information in the entire set of reports can bejudged on the single criterion. The extent to which theanswer to the evaluation question would differ whenhigher-quality and lower-quality studies (as judged bythe single criterion) are used can be determined.

Among advantages of this approach are that it drawson a large possible body of data. A primedisadvantage is that it is quite rare that only onecriterion of study quality would be important. Theevaluative question, as noted, would have to be quitelimited in scope.

Equally WeightedCriteria

In this approach, a set of criteria for selecting theprior research to be synthesized is developed.


Appendix II


Typically, the set includes relevance, recency, contextsimilarity, and a variety of indicators of technicaladequacy including those appropriate tomeasurement, design, analysis, and reporting.

Each of these criteria is given equal weight indeciding whether or not to include the report, article,or book in the set of material to be synthesized. Thatis, a high score on relevance might offset a lowerscore on technical adequacy when a “total” qualityscore is derived and the cut-off established forwhether a study is included. Or, alternatively, athreshold score in all criteria may be required for thereport to be used.

GAO has many examples of this approach, includingthe criteria described in the reviews of the effect ofillegal aliens on legal workers and the effect of thedrinking-age laws on highway safety. (U.S. GeneralAccounting Office, 1986a, 1987b) An advantage to thisapproach is that the effects of various aspects ofquality can be tested empirically. A disadvantage isthat particularly if a threshold is set for all criteria,almost no studies may pass the quality screen.

UnequallyWeighted Criteria

In this approach, the criteria receive differentweights. For example, technical quality may be seenas more important than recency in deciding whetheror not to include the study. Among thetechnical-quality criteria, for some questions theextent to which the design permits strong inferenceabout causality may be seen as much more importantthan, say, the extent to which documentation ofmeasurement reliability exists. The weights are notarbitrary but are guided by the theory underlying themethods. Again, there are examples of this approach.(U.S. General Accounting Office, 1984a)


Appendix II


This approach has the advantage of betterrepresenting the importance of different criteria. It isstill possible, however, that modest strength onseveral relatively less important criteria can offset aserious flaw on a significant criterion, if scores oneach criterion are aggregated.

Threshold orFatal Flaw

In some situations, a report that does not pass musteron a specific criterion is not considered at all, andother criteria come into play only after the “fatal flaw”test has been passed. For example, in a synthesis ofstudies on the homeless mentally ill, reports that didnot attempt to estimate the size of the localpopulation of the homeless were excluded fromconsideration. Further, within the useful studies, afatal flaws criterion (sampling the range of settings)set a cap on rated quality. That is, among the studiesthat estimate population size, the quality of the reportwas judged against seven other criteria and thedirection and extent of bias were judged. Thetechnical-quality rating was the profile of whether theerrors were likely to lead to an overestimate bias oran underestimate and the size of the bias. (U.S.General Accounting Office, 1988i, 1988d)

This model is the most efficient way to ensure quality.The fatal flaw must be carefully examined, however,to be sure that no offsetting features are possible,since potentially informative studies that fail on onlyone criterion may be excluded from the review set.

Table II.2 provides a detailed example of the criteriaused and how they were applied with regard to thenumber of homeless mentally ill persons.


Appendix II


Table II.2: Example of a Fatal Flaws AnalysisWhat we did How we did it

Screening the studies In defining our universe of studies for the evaluation synthesis,we purposefully kept our inclusion criteria broad. We includedany study, regardless of methodological quality, that attemptedto estimate the size of the homeless or homeless mentally illpopulation. We did, however, have some minimum inclusioncriteria. Of our universe of 83 studies, 27 were selected asuseful. Specifically, we included a study in our universe if it meteach of the following three criteria:

1. The study was in written form. Telephone conversations,speeches, or conference proceedings without a written productwere not included.

2. The study provided a count or estimate (by whatevermethod) of the homeless or homeless mentally ill persons orassessed trends in a designated geographic area. This wouldexclude case studies of individuals or studies describingservice needs without a count or estimate.

3. The method used to make the estimate of the number ofhomeless or homeless mentally ill was sufficiently described topermit us to evaluate its merits (or shortcomings). By“sufficiently described,” we mean the study provided someinformation on

— the data used to make the estimate (for example, expertjudgments or actual counts of persons in shelters);— how those data were collected (for example, shelter-providers were interviewed over the telephone, streets werecanvassed by car, and so on);— how the estimate of the size of the homeless or homelessmentally ill population was actually computed (for example, howshelter and street counts were aggregated). That is, there wassome kind of link between the data collected and the finalpopulation estimate.

(continued)


Appendix II


What we did How we did it

Assessing the studies Next we rated the 27 relevant studies on two dimensions:technical quality and soundness (that is, the extent to which thechosen method would produce an underestimate oroverestimate of the size of the homeless population). Wediscovered that many of the studies involved multiple methodsfor counting the homeless, reflecting the various settings(shelters, streets, institutions) in which the homeless andchronically mentally ill can be found. We considered each ofthese “nested studies” for how well it met survey methodologystandards for soundness. Criteria for methodologicalsoundness encompassed such issues as adequacy of universedefinition, coverage of sampling frame, implementationprocedures, and soundness of data analysis. We developedand applied a coding form to extract data relevant to thesecriteria. Finally, two staff members rated the full studies oncriteria related to their overall sampling, measurement,implementation, and population estimation procedures.

Sampling design Did the design cover the range of settings where homelesspersons were likely to be found (shelters, streets, other publicplaces, institutions)?

Was the sample of shelters and institutions representative interms of the area’s shelter size (that is, number of beds) andtype (public or private)?

Did the sample of streets and other public places (such ascensus blocks) adequately cover the locations where thehomeless are known to congregate?

Did the sampling design account for seasonal variation inhomelessness?

Was the unit of analysis (such as municipality) clearly defined?

Measurement Was the estimate of the number of homeless based on anactual count rather than expert judgment?

Was a respondent’s homeless status determined on the basis ofscreening questions?

(continued)


Appendix II



Implementation Were survey procedures explicitly stated in the report?

Were interviewers trained to engage with and administerinterviews to homeless persons?

Were instruments pretested?

If a street survey was conducted, were canvassing proceduresconsistently applied in areas searched? Were areasenumerated before the actual street survey was conducted?

If a shelter-and-institutions survey was conducted, was thecount based upon administrative records rather than subjectiveestimates? Were procedures developed to ensure anunduplicated count of the homeless within shelters andinstitutions?

Deriving the populationestimate

Was the estimate of the number of homeless based upon aprobability sample of areas (such as a national estimate basedupon a probability sample of cities)?

Were adjustments from the sample made to estimate thepopulation (for example, was the application of ashelter-to-street ratio obtained from previous studies)appropriate and justified?

Fatal flaws analysis In applying these criteria, we gave a higher priority to thesampling dimension. That is, if a study did not adequatelysample the range of settings where homeless persons stay,there was a limit on how high the study could be rated, nomatter how strong the measurement, implementation, andestimation procedures. To illustrate, a study that had a strongsampling design (for example, surveyed many settings) butused simple estimation procedures was rated higher than astudy that had a weak sampling design (for example, surveyedonly shelters) and used sophisticated statistical adjustments toaccount for the fact that streets or institutions were notsurveyed. Accounting for sampling bias by using statisticaladjustments—in some cases the only option available—isbased on assumptions about the size of the homelesspopulation in the settings not included in the survey, not anactual count. Applying the criteria in this manner, we rated eachstudy’s technical quality very high, high, moderate, low, or verylow.

(continued)GAO/PEMD-10.1.10 Prospective EvaluationPage 102

Appendix II



Our second rating helped us distinguish where on thetechnical-quality scale (very high to very low) studies could beconsidered sound enough to provide reliable estimates. Thesoundness of studies was determined by rating each study onthe extent to which its methodology would produce, in ourjudgment, an underestimate or overestimate of the number ofhomeless persons. For example, a study that employed adesign that relied solely on the estimates of service providerswould be rated as having the potential for overestimating thesize of the homeless population. Each study was assigned arating on a 7-point scale that ranged from –3 (seriousunderestimate) to +3 (serious overestimate). A writtenjustification was given for each bias rating.

To determine a cutoff point for the methodological soundness,we selected studies that received a bias rating of –1, 0, or +1.In addition to providing a cutoff point, this second ratingindicates the direction and likely magnitude of the bias in eachstudy.

We used the information from these ratings to get an overviewof the current approaches and research designs that are beingused to count homeless and homeless chronically mentally illpersons. This information formed the basis for a closerexamination of the patterns of strengths and weaknesses thatwere evident in the various studies and was applied indeveloping our alternative approaches.


Appendix III

A Project Evaluation Profile


Appendix III



References

Chelimsky, Eleanor. “Federal Evaluation in aLegislative Environment: Producing on a FasterTrack,” pp. 73-86. In C. G. Wye and H. P. Hatry (eds.),Timely, Low-Cost Evaluation in the Public Sector,New Directions for Program Evaluation, No. 38. SanFrancisco: Jossey-Bass, Summer 1988.

Chelimsky, Eleanor. “Immigration: S. 358 WouldChange the Distribution of Immigrant Classes.”Statement before the Subcommittee on Immigrationand Refugee Affairs, Committee on the Judiciary, U.S.Senate, Washington, D.C. U.S. General AccountingOffice, GAO/T-PEMD-89-1, March 3, 1989.

Chen, Huey-Tsyh, and Peter Rossi. “Evaluating withSense: The Theory-Driven Approach.” EvaluationReview, 7:3 (June 1983), 282-302.

Cronbach, Lee. Designing Evaluations of Educationaland Social Programs. San Francisco: Jossey-Bass,1982.

“Evaluation Research Society Standards for ProgramEvaluation.” Standards for Evaluation Practice. No.15. New Directions for Program Evaluation. SanFrancisco: Jossey-Bass, September 1982.

Finney, John W., and Rudolf H. Moos. “EnvironmentalAssessment and Evaluation Research: Examples fromMental Health and Substance Abuse Programs.”Evaluation and Program Planning, 7 (1984), 564-80.

Gottfredson, Gary D. “A Theory-Ridden Approach toProgram Evaluation: A Method for StimulatingResearcher-Implementor Collaboration.” AmericanPsychologist, 39:10 (1984), 1101-12.

Hedges, L., and I. Olkin, Statistical Methods forMeta-Analysis. New York: Academic Press, 1985.


References

Light, Richard J. “Abused and Neglected Children inAmerica: A Study of Alternative Policies.” HarvardEducational Review, 43:4 (November 1973), 209-13.

Light, R., and D. Pillemer. Summing Up: The Scienceof Reviewing Research. Cambridge, Mass.: HarvardUniversity Press, 1984.

U.S. General Accounting Office. What the Departmentof Agriculture Has Done and Needs to Do to ImproveAgricultural Commodity Forecasting and Reports,GAO/RED-76-6. Washington, D.C.: August 1975.

U.S. General Accounting Office. Models, Data, andWar: A Critique of the Foundation for DefenseAnalyses, GAO/PAD-80-21. Washington, D.C.:May 1980.

U.S. General Accounting Office. Minerals Critical toDeveloping Future Energy Technologies, TheirAvailability, and Projected Demand,GAO/PEMD-81-104. Washington, D.C.: June 1981.

U.S. General Accounting Office. Content Analysis: AMethodology for Structuring and Analyzing WrittenMaterial, methodology transfer paper 3. Washington,D.C.: June 1982.

U.S. General Accounting Office. The EvaluationSynthesis, methods paper I. Washington, D.C.:April 1983.

U.S. General Accounting Office. WIC EvaluationsProvide Some Favorable But No Conclusive Evidenceon the Effects Expected for the Special SupplementalProgram for Women, Infants, and Children,GAO/PEMD-84-4. Washington, D.C.: January 30,1984a.


References

U.S. General Accounting Office. EstimatedEmployment Effects of Federal EconomicDevelopment Programs, GAO/OCE-84-4.

Washington, D.C.: August 1984b.

U.S. General Accounting Office. Simulations of aMedicare Prospective Payment System for HomeHealth Care, GAO/HRD-85-110. Washington, D.C.:September 1985.

U.S. General Accounting Office. Illegal Aliens: LimitedResearch Suggests Illegal Aliens May Displace NativeWorkers, GAO/PEMD-86-9BR. Washington, D.C.:April 21, 1986.

U.S. General Accounting Office. Teenage Pregnancy:500,000 Births a Year but Few Tested Programs,GAO/PEMD-86-16BR. Washington, D.C.: July 1986b.

U.S. General Accounting Office. Hazardous Waste:Uncertainties of Existing Data, GAO/PEMD-87-11BR.Washington, D.C.: February 18, 1987a.

U.S. General Accounting Office. Drinking Age Laws:An Evaluation Synthesis of Their Impact on HighwaySafety, GAO/PEMD-87-10 Washington, D.C.: March 16,1987b.

U.S. General Accounting Office. Case StudyEvaluations, transfer paper 10.1.9. Washington, D.C.:April 1987c.

U.S. General Accounting Office. Financing HigherEducation: Examples Comparing Existing andProposed Student Aid Programs, GAO/HRD-87-88FS.Washington, D.C.: April 1987d.


References

U.S. General Accounting Office. Patent Policy: RecentChanges in Federal Law Considered Beneficial,GAO/RCED-87-44. Washington, D.C.: April 1987e.

U.S. General Accounting Office. Medicare:Catastrophic Illness Insurance, GAO/PEMD-87-21BR.Washington, D.C.: July 1987f.

U.S. General Accounting Office. Farm Payments:Analysis of Proposals to Amend the $50,000 PaymentLimit, GAO/RCED-88-42BR. Washington, D.C.:October 1987g.

U.S. General Accounting Office. Immigration: TheFuture Flow of Legal Immigration of the UnitedStates, GAO/PEMD-88-6.

Washington, D.C.: January 1988a.

U.S. General Accounting Office. Welfare: ExpectPanels’ Insights on Major Reform Proposals,GAO/HRD-88-59. Washington, D.C.: February 1988b.

U.S. General Accounting Office. Milk MarketingOrders: Options for Change, GAO/RCED-88-9.Washington, D.C.: March 1988c.

U.S. General Accounting Office. Illegal Aliens:Influence of Illegal Workers on Wages and WorkingConditions of Legal Workers, GAO/PEMD-88-13BR.Washington, D.C.: March 1988d.

U.S. General Accounting Office. USDA’s CommodityProgram: The Accuracy of Budget Forecasts,GAO/PEMD-88-8. Washington, D.C.: April 1988e.

U.S. General Accounting Office. Welfare Reform:Projected Effects of Requiring AFDC for Unemployed


References

Parents Nationwide, GAO/HRD-88-88BR. Washington,D.C.: May 1988f.

U.S. General Accounting Office. Medical Devices:FDA’s Forecast of Problem Reports and FTEs UnderH.R. 4640, GAO/PEMD-88-30. Washington, D.C.:July 1988g.

U.S. General Accounting Office. Women in theMilitary: Impact on Proposed Legislation to OpenMore

Combat Support Positions and Units to Women,GAO/NSIAD-88-197BR. Washington, D.C.: July 1988h.

U.S. General Accounting Office. Homeless MentallyIll: Problems and Options in Estimating Numbers andTrends, GAO/PEMD-88-24, Washington, D.C.:August 1988i.

U.S. General Accounting Office. “Sweatshops” in theU.S.: Opinions on Their Extent and PossibleEnforcement Options, GAO/HRD-88-130BR.Washington, D.C.: August 1988j.

U.S. General Accounting Office. Tax Administration:Difficulties in Accurately Estimating Tax ExaminationYield, GAO/GGD-88-119. Washington, D.C.:August 1988k.

U.S. General Accounting Office. Federal LandManagement: Consideration of Proposed Alaska LandExchange Should Be Discontinued,GAO/RCED-88-179. Washington, D.C.:September 1988m.

U.S. General Accounting Office. Legislative Mandates:State Experiences Offer Insights for Federal Action,GAO/HRD-88-75. Washington, D.C.: September 1988n.


References

Wang, Margaret C., and H. J. Walberg. “EvaluatingEducational Programs: An Integrative,Causal-Modeling Approach.” Educational Evaluationand Policy Analysis, 5:3 (1983), 347-66.

Weick, Karl E. Social Psychology of Organizing, 2nded. New York: Random House, 1980

Wholey, Joseph. “Evaluability Assessment.”Evaluation Research Methods, ed. L. Rutman. BeverlyHills, Calif.: Sage Publications, 1977.


Papers in This Series

This is a flexible series continually being added to andupdated. The interested reader should inquire aboutthe possibility of additional papers in the series.

The Evaluation Synthesis. Transfer paper 10.1.2,formerly methods paper I.

Content Analysis: A Methodology for Structuring andAnalyzing Written Material. Transfer paper 10.1.3,formerly methodology transfer paper 3.

Designing Evaluations. Transfer paper 10.1.4,formerly methodology transfer paper 4.

Using Structured Interviewing Techniques. Transferpaper 10.1.5, formerly methodology transfer paper 5.

Using Statistical Sampling. Transfer paper 10.1.6,formerly methodology transfer paper 6.

Developing and Using Questionnaires. Transfer paper10.1.7, formerly methodology transfer paper 7.

Case Study Evaluations. Transfer paper 10.1.9,formerly methodology transfer paper 9.

Prospective Evaluation Methods: The ProspectiveEvaluation Synthesis. Transfer paper 10.1.10, formerlymethodology transfer paper 10.

(973317) GAO/PEMD-10.1.10 Prospective EvaluationPage 115

Ordering Information

The first copy of each GAO report and testimony

is free. Additional copies are $2 each. Orders

should be sent to the following address,

accompanied by a check or money order made

out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be

mailed to a single address are discounted

25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax number

(301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available

reports and testimony. To receive facsimile

copies of the daily list or any list from the past

30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide

information on how to obtain these lists.

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100