using probability distributions to evaluate an energy conservation program: a technique for dealing...

Pergamon

Evaluatron and Program Planning. Vol. IX. No 3. pp. 200.217. IW5 Copynghr ,I; 19Y5 Elsev~er Science Lrd Printed rn the USA. All rights reserved

0149-7189(95)00015-I 0149-7189:95 %9.50+ .@I

USING PROBABILITY DISTRIBUTIONS TO EVALUATE AN ENERGY CONSERVATION PROGRAM: A TECHNIQUE FOR

DEALING WITH CONTROVERSY

LINDA BERRY AND MARILYN A. BROWN

Oak Ridge National Laboratory

INTRODUCTION

The process of determining the cost effectiveness of both government- and utility-sponsored energy conservation programs is often fraught with controversy. This occurs, in part, because assessments ofcost effectiveness depend upon a long series of choices and assumptions. To begin, one must choose the “correct” perspective or test. Often a number of options are available. For example, there are five standard benefit/cost tests used in evaluations of utility-sponsored energy conservation programs: societal, total resource, utility, participant, and nonparticipant;’ and, a variety of new tests have been proposed (Chamberlin & Herman, 1993; Braithwait & Caves, 1993). In addition, there is a long running debate concerning the value of various perspectives, especially the merits of the total resource cost (TRC) test vs the ratepayer impact (RIM) test’ (Kahn, 1991; Ruff, 1988; Joskow, 1990; Cicchetti & Hogan, 1989).

Once a perspective is chosen, one must carefully account for a program’s benefits and costs to various subgroups. Issues of which benefits and costs to include and how to value them must be resolved. Although program costs can usually be measured with reasonable accuracy, benefits (especially for those programs that include goals designed to increase human well-being) can seldom be quantified in a satisfying manner. As Birdsall (I 987) observed, about the evaluation of human service programs, “the transformation of beneficial outcomes into dollars is often flimsy at best; arguments to double or halve the estimate are as cogent as arguments for the initial estimate.”

Although there are many uncertainties in determining program efficiency and cost effectiveness, a mandate to systematically evaluate government programs was passed by the 103rd Congress in August of 1993 (Public Law 103-62). This law is designed to improve the performance of government by “systematically holding Federal agencies accountable for achieving program results.” Various levels of evaluation activities, from continuous process monitoring to extensive and comprehensive outcome evaluations, will be required between now and the year 2000. To conduct such evaluations, program managers must define output measures and performance goals, which are expressed as tangible, measurable objectives against which actual achieve- ments can be measured. Determining program cost effectiveness will surely be an important part of these future federal evaluation activities. When the correct assumptions and inputs are uncertain, as is often the case in determinations of cost effectiveness, the technique illustrated in this paper can play a useful role.

Previous applications of the technique for dealing with uncertainty which is illustrated here are limited. One previous example involves the analysis of the cost effectiveness of utility demand-side management (DSM) programs under uncertainty with the RIM and TRC tests (Seiden, Faruqui, Chamberlin & Ellingson, 1990). Seiden et al. (1990) conclude that probabilistic simulation analysis can provide useful information for DSM planners. In particular, the probabilistic simulation results provide decision makers with a probability estimate that a program is cost effective under various types of cost-effectiveness tests.

Requests for reprints should be sent to Linda Berry, Oak Ridge National Laboratory, P.O. Box 2008, Oak Ridge. TN 37831.

‘For definitions of these perspectives see the Standard Prac~icr Manual Jbr Economic Analysis oj Demand-Side Managemmr Programs. published in 1987 by the California Public Utilities Commission and the California Energy Commission.

‘The RIM test is also called the nonparticipant, or no-losers. test. The TRC test typically results in more favorable assessments of demand-

side management programs than the RIM test.

209

210 LINDA BERRY and MARILYN A. BROWN

Although the technique for dealing with uncertainty which is presented in this paper has broad application in the evaluation of many program types, it is most appropriate for situations in which results are highly sensitive to a large number of plausible combinations of the relevant inputs. When results depend primarily on only a small number of independent inputs and assumptions traditional sensitivity analysis is less com- plex and will produce equivalent results.

EVALUATION OF THE WEATHERIZATION PROGRAM

A recent national evaluation (Brown, Berry, Balzer & Faby, 1993) of the U.S. Department of Energy’s (DOE) low-income Weatherization Assistance Program brought the problem of the controversy surrounding program benefit/cost analyses sharply into focus. In this evaluation, consensus concerning the correct choice of perspectives, assumptions, and dollar values of benefits (especially nonenergy benefits) was not an attainable goal.

The DOE’s Weatherization Program is one of the largest energy conservation programs in the nation. Its network of over 1,100 local agencies weatherizes several hundred thousand low-income dwellings annually. The Program strives to increase the energy efficiency of dwellings occupied by low-income persons in order to reduce their energy consumption, lower their fuel bills, increase the comfort of their homes, and safeguard their health. It targets vulnerable groups including the elderly, people with disabilities, and families with chil- dren. Thus, the Program has both energy conservation and human services goals.

In 1990, the DOE initiated a National Evaluation of its Weatherization Assistance Program. The major goals of this evaluation were to estimate the energy saved by the Program and to determine its cost effectiveness. Secondary goals included estimating nonenergy impacts (such as health, safety, affordable housing, employment, and environmental externalities), analyzing factors that influence energy savings and cost effectiveness, and identifying promising future Program opportunities.

Representative national samples of 18,748 weatherized and 11,795 control dwellings were used to estimate national and regional Program impacts. Fuel consumption records were obtained from almost 500 gas and electric utilities. Agency records were used to obtain information on dwelling and occupant characteristics, costs, and weatherization measures installed.

Altogether, the national evaluation design provided for:

l the most comprehensive evaluation of the Program

ever conducted (involving thousands of dwellings and hundreds of local weatherization agencies and utilities); understanding of the program across key subgroups (climate regions, primary heating fuels, dwelling types, and agency size); implementation of innovative approaches to weatherization program evaluation (e.g., retention of dwellings with occupancy changes and use of a new weather normalization model for electrically heated and cooled dwellings); a detailed description of the Program’s weatherization activities; a primary data analysis of energy savings and cost effectiveness of the Program as applied to gas- and electrically-heated homes, and an extrapolation of these findings to homes heated with other fuels; the inclusion of some nonenergy benefits in the cost- effectiveness analysis; and the involvement of representatives of all the Pro- gram’s major stakeholders in the evaluation’s design and implementation.

The National Weatherization Evaluation was guided by a 40-member advisory group that helped design the study, assisted in its implementation, and suggested interpretations of its findings. This group provided invaluable assistance throughout the evaluation. How- ever, discussions of strategies for determining Program cost effectiveness rarely led to consensus. For example, discount rates ranging from 0 to 10% were proposed. Some believed energy savings would last 10 years, others believed 25 years. Some advocated excluding certain types of expenditures (e.g. funds leveraged from non- DOE sources, funds used for housing rehabilitation instead of for energy-efficiency measures, certain administrative costs), while others argued that all costs should be included. There was agreement that nonenergy benefits (such as health, comfort, safety, affordable housing, employment, and environmental externalities) should be valued. The effort to assign dollar values to these nonenergy benefits led, however, to more controversy.

A PROBABILISTIC COST-EFFECTIVENESS ANALYSIS

Because of the lack of consensus concerning the proper perspectives, assumptions, and dollar values of benefits to use in determining the Program’s cost effectiveness, we conducted an extensive sensitivity analysis. As the analysis proceeded, the possible combinations of input values grew to unmanageable proportions. To incorporate the diversity and uncertainty about the inputs (e.g., costs, energy savings, measure lifetimes, discount

Using Probability Distributions 211

rates, fuel price escalation rates) into the cost-effec- on-site installation costs plus management and over- tiveness analysis, a technique was applied that uses dis- head costs). The third perspective, the societal, includes tributions of input variables to produce distributions of the most comprehensive set of benefits and costs: energy likely outcomes. This approach has several advantages and selected nonenergy benefits are compared to all over traditional sensitivity analysis. costs.

In typical sensitivity analyses, the values of one input vary while all others are held constant. This procedure shows the sensitivities of the results to several levels of values, but not the entire range of possibilities for all combinations of variables. The technique used to develop the results presented later in this paper, however, systematically varies all combinations of the input variables based on a priori distributions. This process is analogous to a statistical error propagation analysis. It offers a way of including the full range of plausible values, and of divergent stakeholder viewpoints, in the presentation of cost-effectiveness results. Using distributions of inputs also helps to increase the acceptance of the range of results because no single set of assumptions or benefit values has to be selected. In addition, using probability distributions of results offers a convenient way of showing the likelihood of various conclusions given the uncertainties in the inputs. Thus, this approach is useful in the evaluation of many programs, which are characterized by controversy concerning the proper perspectives, assumptions, and dollar values of costs and benefits.

For each of the perspectives, the average energy savings per dwelling were valued by: (1) measuring the first- year savings based on before and after weatherization fuel consumption records; (2) calculating the dollar value (in 1989 dollars) of the savings based on Energy Information Administration (EIA) State fuel prices that were weighted by the proportion of weatherized dwellings in each State; (3) assuming the first-year savings continued for 20 years; (4) applying EIA-recommended fuel price escalation rates; and (5) applying a discount rate of 4.7%. The formula for determining the present value of energy savings benefits is shown in equation (1). These benefits were divided by first-year costs (which are assumed to be the only costs) to yield a benefit/cost (b/c) ratio. Benefit/cost ratios were calculated for a variety of subgroups such as climate regions, dominant heating fuel, and dwelling type.

’ (I+e) ‘_-I Present value of energy savings = SP ,g,

[ 1 (1

MAJOR PERSPECTIVES

Our analysis of the cost effectiveness of the DOE’s Weatherization Program defined three major perspectives (Figure 1). The first perspective includes only energy savings benefits and on-site installation costs. In this installation perspective, which follows the usual procedure in previous low-income weatherization program evaluations, the only benefit that is valued is energy savings and the only costs included are expenditures for materials and the on-site labor that installs the weatherization measures. The second perspective, the program perspective, includes only energy savings benefits, but compares these benefits to total costs (i.e.

Perspective Benefits included

costs included

Installation

Program

Societal

Figure 1. Three perspectives used to calculate cost effectiveness.

where

e = fuel price escalation rate d = discount rate 1 = lifetime of measures

S = first-year energy savings P = first-year energy price.

In the societal perspective, which includes both energy and nonenergy benefits, energy benefits were valued in the same way that was described above. The methods used to estimate the value of the nonenergy impacts varied. Estimates of environmental benefits relied on a literature review and on information from the National Weatherization Evaluation about the proportions of weatherized dwellings using various fuel types and about the average savings by fuel type. The analysis of environmental impacts was limited to the costs associated with S02, NO,, and COZ. Estimates of employment benefits combined a literature review with data from the National Evaluation on the number of employees directly supported by DOE’s weatherization program, the skill level of workers, and managers’judg- ments concerning the structure of the job market for weatherization workers. Direct and indirect, but not induced, employment benefits were included in the estimate. Data from the National Evaluation on Program


Benefit:cost (B:C) ratios

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Cold region

4

1.92

Moderate region

Single-family detached dwellings

1.67

1.67

Single-family attached dwellings

1.42

1

Mobile homes Perspective:

Small multifamily units

Figure 2. Variations in benefit/cost ratios by perspective, climate region, and dwelling type.

expenditures for home repairs were used to quantify purposes. The installation perspective, which deter-

the benefits associated with maintaining or enhancing mines on-site installation benefits and costs is best for

property values and extending the lifetime of dwellings. comparing this study’s results to those of previous

The estimate of reductions in arrears was based on a evaluations of low-income weatherization programs.3

literature review and data on payment histories that The program perspective offers the most conservative

were collected on the dwellings included in this study. estimate of program cost effectiveness. If the program

Overall, nonenergy impacts were estimated to have a is cost effective from this perspective, it will be from the

present value of $926 (Brown, Berry, Balzer & Faby, others as well. The third approach, which uses a broad,

1993). To calculate a societal benefit/cost ratio, the non- societal perspective, is best for valuing a more com-

energy benefit value was added to the energy benefit prehensive set of program impacts, and for comparisons

value and the sum divided by the first-year costs. with alternative uses of government funds.

With the baseline assumptions described above (i.e. a 20 year lifetime of weatherization measures, a 4.7% discount rate, and the EIA-recommended fuel price escalation rates), the results (for gas-heated dwellings, the dominant heating fuel) are as shown in Figure 2. The variations across perspectives and subgroups are significant. In addition, while the program perspective always results in the lowest b/c ratio, the societal perspective typically, but not always, produces the highest b/c ratios. With the societal perspective, all subgroups achieve cost-effectiveness. With the other two perspectives, however, some subgroups do not.

PROBABILITY DISTRIBUTIONS

Because there was uncertainty and controversy about the inputs (e.g., energy savings, values of nonenergy

Each of the three perspectives is valuable for different

‘Nearly all previous evaluations of low-income weatherization pro-

grams have used the installation perspective for their analyses of cost effectiveness. This occurred because record keeping is more complete

and easier to interpret for installation costs than for administrative

costs, and because installation costs can be associated with specific dwellings, while administrative costs are more difficult to assign to

individual dwellings.


TABLE 1 CHARACTERISTICS OF TRIANGULAR INPUT DISTRIBUTIONS

Minimum Maximum Mode Mean Standard Deviation

Discount rates 2% 10% 4.7% 5.6% 0.017

Energy savings 151 193 173 172 6.576

Measure lifetimes 15 25 20 20 2.041

Fuel price escalation rates 1.9% 2.1% 2.0% 2.0% 0.0004

Nonenergy benefits” 500 1500 976 976 247

aNonenergy benefits are valued only in the societal perspective. They have a value of zero in the

installation and program perspectives.

benefits, costs, measure lifetimes, fuel price escalation rates, and discount rates) to the cost-effectiveness analysis, probability distributions of inputs were defined. All of the input variables to the cost-effectiveness analysis were initially assigned a triangular probability distribution (Technical Appendix E. p. E-30, @RISK User’s Guide). The triangular distribution is specified by a minimum, maximum, and modal value (Table 1 and Figure 3), with the modal value being most likely. For example, measure lifetimes were assumed to have a minimum, maximum, and mode of 15,25 and 20 years, respectively. In this case, as in all others where the mode is halfway between the minimum and the maximum, the mean is the same as the mode (20 years). The distribution of annual gas savings was defined as the 90% confidence interval around the net savings, with a minimum of 15 1 ccf, a maximum of 195 ccf and a modal value of 173 ccf. The distribution of discount rates was assigned a minimum of 2%, a maximum of lo%, and a modal value of 4.7%. A range of values was also specified for price escalation rates for natural gas (1.9-2.1% with a mode of 2.0%).

The choice of minimum, maximum, and modal values was generally based on available literature, and on expert opinion (represented by the National Weather- ization Evaluation’s Working Group). This was the case for the selection of the range of values for measure lifetimes, discount rates, and fuel price escalation rates. Assumptions about energy savings and costs, however, were based on measured values and their associated confidence intervals. Nonenergy benefits were assigned values of zero in the installation and program perspectives, but had the values shown in Table 1 for the societal perspective.

After all the input distributions were defined, as shown in Table 1 and Figure 3, a statistical sampling approach was implemented to produce a distribution of likely outcomes. The @RISK risk analysis and modeling program systematically selects samples of points from the input distributions4 to produce probability

‘In our analysis, we varied all of the inputs independently. The

@RISK software also will allow the analyst to build in correlations among input variables.

distributions of likely outcomes. The sampling can be completely random (Monte Carlo) or constrained to produce more efficient sampling (Latin Hypercube).

In our analysis, discrete values were sampled (with

Expected value = 20 years

Weatherization measures lifetimes inputs

0.09 0.08

,c 0.07 2 0.06

0.05 2 0.04 i 0.03

0.02 0.01

0

Expected value = 0.056%

19

years

23 25

Discount rates inputs

0.09 0.08

,h 0.07 2 0.06

; 0.05 0.04 C 0.03

0.02 0.01

0 0.02 0.04 0.06 0.08 0.10

Discount rate

Expected value = 173 ccf

Energy savings inputs

nr 0.10 r

-150 l&O li0 140

ccf

Figure 3. Triangular input distributions for measure lifetimes, discount rates, and energy savings.


the Latin Hypercube technique) from each of the input distributions and then combined to produce one output (i.e., one b/c ratio). This sampling procedure can be repeated a specified number of times to generate the

desired statistical precision in the ouputs. We chose to use one thousand samples of points from the input

distributions. With 1000 samples, the means of the sample

output distributions of b/c ratios will be within 3% of

the population values.

SIMULATION RESULTS

For each of the three major perspectives (Figure 1) examined, 1000 combinations of points from the input distributions were used to produce 1000 outcomes. These outcomes were then used to produce probability

distributions of likely b/c ratios, given the assumed distributions of the inputs. The output distributions are

shown in Figures 4-6 for the three major perspectives.

The output distribution for the installationperspective

(Figure 4) shows that all combinations of inputs pro-

duce cost-effective results. The mean of the distribution is 1.49 and all cases have a b/c ratio greater than 1 .OO. The output distribution for the program perspectit’e

(Figure 5) shows that only the more favorable combinations of assumptions produce a cost-effective result,

while less favorable ones do not. Nevertheless, the mean of the distribution was 1.02 and slightly more than 50%

of the cases produced a b/c ratio above 1.00. With the societalperspective, the mean value is 1.59, the Program

is cost effective for all of the specified input distributions in 100% of the cases sampled (Figure 6).

The results using the program perspective are sensitive to the choice of discount rates as illustrated in

Figure 7. With the program perspective, a fixed discount rate of 2%, and distributions for the other inputs, the

Program is cost effective with all combinations of inputs. With a fixed discount rate of 4.7%, it is cost effective only for the more favorable parts of the input distributions, and with a discount rate of 10% or higher

Expected value = I .4Y

Installation perspective: triangular input distributions

0.12

0.10

2 0.08

‘2 z 0.06

k 0.04

0.02

00.0 0.90 1.40 1.90 2.40

Benefit:cost ratios

Figure 4. Distribution of installation benefit/cost ratios with triangular input distributions for assumptions.

The triangular input distributions, shown in Figure 3, assume that the modal values are much more likely to occur than the maximum and minimum values. Another option is to assume that each value in the distribution, including the minimum and maximum, is equally likely to occur. In this case, all of the inputs would assume uniform distributions, and would have a shape similar to that shown in Figure 9.

Results produced with uniform distributions for

Expected value = 1.02

Program perspective

Benefit:cost ratio

Figure 5. Distribution of program benefit/cost ratios with triangular input distributions for assumptions.


0.14

0.12

Societal perspective

nn ,x 0.10 : D 0.08

2 2

0.06

a 0.04

0.02

0.00 1 .oo l.iO Lb0 l.bO 2.bo 2.5.0 2.40

Benefit:cost ratio

Figure 6. Distribution of societal benefit/cost ratios with triangular input distributions for assumptions.

it is never cost effective. With the societal perspective,

even discount rates as high as 10% always produce cost- effective results (Table 2).

B/c results also are sensitive to the assumed lifetime of the measures. The effects of varying both assumed discount rates and lifetimes are summarized in Table

2 and Figure 8. Using the program perspective (the “strictest” test), Figure 8 indicates that the Program is cost effective at an assumed discount rate of 4.7%, if

the weatherization measures are assumed to generate constant savings for at least 20 years. With the program perspective and a 2% discount rate, the Program is cost effective with an assumed measure lifetime of 15 years.

UNIFORM DISTRIBUTIONS


Expected value = 1.38 TABLE 2

2% Discount rate SENSITIVITY OF PROGRAM BENEFIT/COST RATIOS TO

DISCOUNT RATES AND LIFETIMES

0.10

x .=: 0.08

2 2 0.06

2 0.04 a

0.02

0.00 0.96 1.16 1.36 1.56 1.76

Benefit:cost ratio


4.7% Discount rate

0.10 ~

+z 0.08 P 2 0.06

i 0.04

0.02

0.00 0

Benefit:cost ratio


10% Discount rate 0.14 r

0.12

* .z 0.10

2 0.08

g 0.06

6 0.04

0.02

0.00 0.56 0.61 0.66 0.71 0.76 0.81 0.86

Benefit:cost ratio

Figure 7. Sensitivity of program perspective results to choice of discount

rates.

inputs are similar to those produced with triangular distributions. In general, the outcomes are relatively insensitive to the form of the input distributions and depend mainly on their means and standard deviations (the exception is when one of the input variables has a strong nonlinear influence on the outcomes). Because the standard deviations are higher for the uniform than for the triangular input distributions the results also vary to some extent. Nevertheless, in both cases nearly all of the b/c ratios are greater than 1.00. The relative size of the standard deviations for the uniform vs triangular distributions are as follows:

l discount rates: 0.023 (uniform) vs 0.017 (triangular) l energy savings: 12.638 (uniform) vs 8.576 (triangular)

Lifetime (years)

Discount rate

2% 4.7% 10%

Installation perspective 10

15

20

25

Program perspective 10

15

20

25

Societal perspective 10

15

20

25

1.20 0.89 0.73

1.50 1.26 0.93

2.00 1.58 1.07

2.51 1.86 1.17

0.67 0.59 0.49

1.01 0.84 0.63 1.34 1.06 0.72

1.68 1.25 0.78

1.22 1.14 1.03

1.56 1.39 1.17

1.90 1.61 1.26 2.24 1.80 1.33

.; 2.0 I- ;; x z 1.5

B $ : 1.0 ._ ij & :: “p 0.5

E 2

: k 0.0

Length of assumed lifetime

-.- 25 Years _-.-- 20 Years

2 4.7 10

Discount rate (%)

Figure 8. Sensitivity of program benefit/cost ratios to assumed discount

rates and measure lifetimes.

Expected value = 6%

0.06 Discounts rates: uniform input distribution

0.05

.e 2

0.04

2 0.03

% 0.02

0.01

0.00 ; k ‘6

Discount rates (%)

Figure 9. Example of uniform input distributions for assumptions.


Expected value = 1.45 Expected value = 1.71

Installation perspective: uniform input distributions Societal perspective: uniform input distributions

0.12 0.12

0.10 0.10

x .z 0.08 .g 0.08

‘2 0.06 2

p” p” 0.06

g 0.04 2 0.04

a a 0.02 0.02

0.00 0.00

0.80 1.‘30 IL30 2.jo

Benefit:cost ratios

Figure 10. Distribution of installation benefit/cost ratios with uniform

input distributions for assumptions.

1.20 1.70 2.20 2.70

l measure lifetimes: 2.872 (uniform) vs 2.041 (tri- SUMMARY angular)

l fuel price escalation rates: 0.0006 (uniform) vs 0.004 (triangular).

These differences in standard deviations have some effect on the results, however, conclusions about the likelihood that the Program is cost effective show little change.

For each of the three perspectives, the results with uniform input distributions are similar to those with triangular input distributions. With the installation perspective (Figures 4 and lo), mean values are nearly the same (1.49 vs 1.45), and the range is similar. Using uniform input distributions produces, however, an output distribution that is less symmetrical and more skewed to the left th?_n the results for triangular distributions. Comparisons for the program perspective (Figures 5 and 1 l), and for the societal perspective (Figures 6 and 12), also show similar results. The means differ slightly (by less than lo%), and the ranges are broader with the uniform input distributions. In addition, the distributions are less symmetrical and more skewed to the left. This occurs because of the influence of the nonlinearities that result from the exponential term in the formula for the benefits that is shown in equation (1).

The determination of the cost effectiveness of the Weatherization Assistance Program raised many issues concerning the selection of the correct perspectives, input assumptions, and benefit values. Consensus concerning the desired inputs could not be achieved among stakeholders, yet the indicators of cost effectiveness were highly sensitive to the inputs chosen. B/c ratios for the Program as a whole ranged from 0.49 to 2.24 depending upon the assumptions, perspectives, benefits (particularly nonenergy benefits), and cost values used.

Initial sensitivity analyses showed that the results were most sensitive to the choice of discount rate and the assumed lifetime of the weatherization measures. The results were less sensitive to variations in the assumptions about fuel escalation rates and energy savings. In a typical sensitivity analysis, the analyst varies the values of one input while holding the others constant. Thus, the sensitivities of the results to several levels of the input values are examined, but not the entire range of possibilities for all combinations of variables. The technique used to develop the results presented in this paper, however, systematically varies all combinations of the input variables based on a priori distributions. This process is analogous to a statistical error propagation analysis.


Program perspective: uniform input distributions

0.12 r rhl

0.10

.c 0.08

.g m 0.06

s a”

0.04

0.02

0.00 0.50 0.‘70 0.‘90 l.‘lO 1.;0 1,;o l.iO

Benefit:cost ratios

Figure 11. Distribution of program benefit/cost ratios with uniform input distributions for assumptions.

Benefit:cost ratio

Figure 12. Distribution of societal benefit/cost ratios with uniform input

distributions for assumptions.

In many evaluation contexts, the technique illustrated in this paper offers a number of advantages over typical sensitivity analyses. Most importantly, it provides a way of dealing with a lack of consensus concerning the correct perspectives, benefit values, and assumptions to use in a cost-effectiveness analysis. The approach demon- strated in this paper offers a way of including divergent stakeholder viewpoints in the presentation of cost- effectiveness results. By combining multiple sources of uncertainty in a way that demonstrates their overall effect on the range of results, the method allows one to show the likelihood that the Program is cost effective even when a consensus on assumptions cannot be reached. The technique makes it possible to include the


full range of plausible values and to incorporate the uncertainty about the proper inputs into the cost-effectiveness analysis by using probability distributions of inputs in a simple simulation modeling approach that produces a distribution of likely outcomes. The results were relatively insensitive to the form of the input distributions (triangular vs uniform), and varied mainly in response to the assumed mean values for the inputs.

Although the technique for dealing with uncertainty which is presented in this paper has broad application in the evaluation of many program types, it is most appropriate for situations in which results are highly sensitive to a large number of plausible combinations of the relevant inputs. When results depend primarily on only a small number of independent inputs and assumptions traditional sensitivity analysis is less com- plex and will produce equivalent results.

In many cases, presenting probability distributions of cost-effectiveness results offers a convenient way of showing the likelihood of various conclusions given the uncertainties in the inputs. Using distributions of inputs and outputs also helps to increase the acceptance of the results because no single set of assumptions or values of benefits must be selected. In addition, because stakeholders are presented with such a variety of combinations of perspectives and assumptions, they are likely to find one that is satisfactory. Thus, this approach should be useful to the evaluators of many other programs, which are characterized by controversy concerning the proper perspectives, assumptions, and dollar values of costs and benefits.

REFERENCES

BIRDSALL, W. C. (1987). When benefits are difficult to measure. Ezaluation and Program Planning, 10, 109-I 18,

BRAITHWAIT, S. D. & CAVES, D. W. (1993). The complete and unabridged measure of DSM net benefits: What we’ve been missing (pp. 238-242). Proceedings: 6th national demand-side management conference: Making a difference. 2426 March, Miami Beach, Florida.

BROWN, M. A., BERRY, L. G., BALZER, R. A. & FABY, E. (1993). National impacts of rhe weatherization assistance program in single-family and small multzfamily dwellings. Oak Ridge, Tennessee: ORNL/CON-326, Oak Ridge National Laboratory.

CHAMBERLIN, J. H. & HERMAN, P. M. (1993). Why all ‘good’ economists reject the RIM test (pp. 231-237). Proceedings: 6th national demand-side managemenl conference: Making a difference, 24-26 March, Miami Beach, FL.

CICCHETTI, C. J. & HOGAN, W. (1989). Including unbundled demand-side options in electric utility bidding programs. Public Vtili- ties Fortnightly, 8 June.

JOSKOW, P. L. (1990). Understanding the ‘unbundled’ utility conservation bidding proposal. Public Utilities Forinightly, 4 January.

KAHN, A. E. (1991). An economically rational approach to least- cost planning. Electricity Journal, June.

@RISK: Risk analysis and simulation add-in for Microsoft Excel, release 1.1, User’s Guide, February 1992. Newfield, NY: Palisade Corp.

RUFF, L. E. (1988). Least-cost planning and demand-side management: Six common fallacies and one simple truth. Public Utilities Formighlly, 28 April.

SEIDEN, K., AHMAD, F., CHAMBERLIN, J. & ELLINGSON, E. (1990). Cost-effectiveness analysis under uncertainty. Oakland, CA: Barakat & Chamberlin Inc.

using probability distributions to evaluate an energy conservation program: a technique for dealing...

Documents