March 5, 2002

Lessons Learned from GAO’s Evaluation of the Outcomes of R&D Programs

Presentation to ORNL’s Conference on Estimating the Benefits of

Government-Sponsored Energy R&D

Metrics for R&D Evaluation are Limited

Spending on research was the primary indicator of the investment in research -- an input measure

– Available in both the public and private sectors.– However, not a good indicator of the level of

results achieved

No single output measure captures the results of research well either. Some examples include:

– Quantitative analyses of return on investment; patents, copyrights & technology licensing; technology partnerships; and bibliometrics

– Qualitative assessments based on peer review.

Outcome Orientation Drives R&D Evaluation

Outcomes is the focus of GPRA

Outcomes have also been focus of GAO’s work at DOE

GAO reports on the Department of Energy’s R&D efforts include

– Energy Efficiency – Fossil Energy– Renewable Energy

Four Broad Themes from GAO Reports, 1991—2001

1. Would private sector do the research?

2. Will consumers buy the product?

3. Do benefits exceed costs?

4. Are efforts consistent with agency’s mission and strategic plans? (1998)

1. Would the private sector do the research?

Sample Findings:

DOE has funded research that private industry may have performed.

– Clean Coal Technology Program

Other Programs, such as the Advanced Technology Program

(1991)

(1996)

RENEWABLE ENERGY

DOE's Funding and Markets for Wind Energy and Solar Cell Technologies

GAO/RCED-99-130

GAO Report to the Congressional Requesters

RENEWABLE ENERGY

DOE's Funding and Markets for Wind Energy and Solar Cell Technologies

GAO/RCED-99-130

GAO Report to the Congressional Requesters

COOPERATIVE RESEARCH

Results of U.S.-Industry Partnershipto Develop a New Generation of Vehicles

GAO/RCED-00-81

GAO Report to the Congressional Requesters

COOPERATIVE RESEARCH

Results of U.S.-Industry Partnershipto Develop a New Generation of Vehicles

GAO/RCED-00-81

GAO Report to the Congressional Requesters

2. Will consumers buy the product?

Sample Findings:

DOE has conducted research on technologies that

– have already been developed and marketed elsewhere,

• Wind turbines, Clean Coal Technologies

– and may not be marketable.• PNGV, Advanced Batteries

(1999)

(1990)

(2000)(2000)

(1995)

3. Do benefits exceed costs?

Sample Findings:

In evaluating the results of its R&D, DOE

– has not clearly stated the costs of its research

– has overstated economic benefits• DOE’s Success Stories Report

(1996)

Sample Findings:

DOE’s Performance Plans and Reports (energy efficiency, fossil energy, and science) do not convey a clear picture of what the department is trying to do or how it plans to get there.

Problems inherent to science

4. Are efforts consistent with agency’s mission and strategic plans?

?Strategic Goal

Performance Goal

Annual Activity Annual Activity Annual Activity

Progress TowardsStrategic Goal

100%

75%

50%

25%

(2000)