1

Utilization-focused Evaluation of a Portfolio of Research, Development, & Demonstration Programs

Helen Kim, Larry Pakenas - NYSERDA Rick Ridge – Heschong Mahone Group, Inc.

Scott Albert – GDS Associates, Inc. Gretchen Jordan – Sandia National Laboratory

American Evaluation Association ConferencePortland, Oregon

November 2, 2006

2

New York State Energy Research and Development Authority

(NYSERDA)• Mission

Use innovation and technology to solve New York’s energy challenges in ways that benefit the State’s economy and environment

• Vision

Serve as a catalyst for change – enabling New Yorkers to realize affordable energy, a growing and vibrant economy, greater energy independence, and a cleaner environment

3

NYSERDA and RD&D Funding

• Statutory Funding for RD&D– Levy on interstate sales of gas and electricity– Annual funding ~$13 million for RD&D

• Public Benefits Funds– Annual budget

~175 million total~$45 million for RD&D

• NYSERDA Staff– Total 220– RD&D 40 (29 before Public Benefits Funding)

4

Public Benefits Funded RD&D Program Portfolio

Product Development

21%

Proof of Concept20%

Develop/Improve New Product

64%

Pre-deployment21%

Research for Policy18%

Demonstration45%

Test/Improve Product16%

6

Project Selection

Competitive Solicitations

Proposals reviewed by a Technical Evaluation Panel

• Mix of external and internal reviewers• Score and rank projects• Criteria for selection

– Technical merit (Does it make sense?)– Resources (Does the team have the resources to

succeed?)– Benefits (Will the project result in energy and economic

benefits for New York State?)

8

NYSERDA Select & Manage R&D Projects to:-drive portfolio changes over time to respond to current needs, and-Provide public benefits

Develop new or improved product

Study, Prove Concepts

Demonstrate products, inform markets

Study to inform policy & R&D community

Test & improve products

Dissemination builds common knowledge base-Lab prototypes-Future R&D & product options

-Investment/interest growing-Commercial scale product developed -Potential demonstrated

Product proven/ introduced in market

Informed policies & programs;R&D opportunities & standards identified, publicized

- Data from tests- Establish standards- Hands on experience (industry)-Feedback to R&D

White papers, workshops;Policy-relevant research

-Intermediate scale prototypes- Performance/cost specifications improving

Producers, consumers,

policy makers see value

New knowledge:-papers, articles -data

- Data from tests in different context- Feedback to R&D & policy makers- Visibility & data from showcases

Inputs:Funds, staff, NYSERDA competencies, partnerships

Activities

Outcomes

Outputs

Policy and Product development and pre-deployment process (5-10 years)

Knowledge for future R&D and productsFirms have credibility & market infrastructure is supportive

Products manufactured as replacement, stand alone, or part of system and purchased by early adopters

External Influences:Cost, Performance of existing technologies; Industry willingness to take risks; Uncertainty of R&D; Energy prices; Government policies

Research for Policy DemonstrationProduct Development Pre-deployment

Educate, provide incentives to supply & delivery

-Training, certification-Production incentives-Innovative designs-Other barriers lowered

Business infrastructure supports

the product

Environmental benefits Energy benefits- generation, energy/load management, efficient use

Economic benefits -cost of compliance, NY jobs

- New- Accelerated- Expanded

9

Peer-Review Methodology

• Year 1– GDS and Staff developed Accomplishment Packets for five

projects and sent to reviewers– Assessments were returned to GDS– Obtained feedback from reviewers about process

• Year 2– GDS and Staff developed Accomplishments Packets for two

Programs and sent to reviewers– Staff made presentations to the peer reviewers – Peer reviewers provided opportunity to discuss the program

amongst themselves– Peer reviewers scored assessments and returned to GDS

10

Peer-Review Methodology

Recruitment of Peer Reviewers• Staff were asked to provide names and contact

information of appropriate individuals • An email invitation was sent to recruit Peer

Reviewer candidates• “Peer Reviewer Agreement” form used to solicited

information on credentials and possible conflicts of interest

11

Peer-Review Methodology

Accomplishments Packets Consisted of• Background information including project goals and

justification • Summary of resources including funding, co-funding

and staff time• Summary the accomplishments categorized into the

five criteria: knowledge created, disseminated, commercialization progress, energy, economic, and environmental benefits

A mini case study

12

First-Year Peer Review (Five Projects)1. 21st Century HVAC Research Consortium

– $200,000; $1.7 million Co-Funding

– Goal: Drive basic research on heating, ventilation, air conditioning, and refrigeration technologies

– Activity Type: Information for Policy and Researchers

2. Aggregating Distributed Generators

– $450,000; $590,000 Co-Funding

– Goal: Demonstrate that existing backup generators can be economically aggregated and dispatched

– Activity Type: Demonstration

3. Continuous Ambient Particulate Monitor

– $500,000; $1.2 million Co-funding

– Goal: Improve measurement of ambient PM2.5 mass – Activity Type: Product Development

13

First Year Peer Review (Continued)

3. Truck Stop Electrification

– $1.5; $3.3 million Co-funding

– Goal: Evaluate potential, construct demonstration sites, establish new businesses, develop off-peak market for electricity

– Activity Type: Market Analysis, Product Development, Demonstration

5. Turnkey Pump and Compressed Air

– $370,000; $353,000 Co-funding

– Goal: Increase use of compressed air efficiency projects among manufacturers

– Activity Type: Demonstration

14

Assessment Criteria in Year One• Knowledge Creation

– Quantity – Significance

• Knowledge Dissemination– Availability of Knowledge Products– Impact on target audience

• Commercialization Progress– Capital Attraction– Technical Achievement – Market Advancement

• Realized and Potential Energy Benefits• Realized and Potential Economic Benefits• Realized and Potential Environmental and Health Benefits• Value versus Costs

15

Rater Scores by Criteria and Project

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

TruckstopElectrification

Compressed AirProgram

21st Century HVAC Particulate Monitoring Aggregating DG

Knowledge Creation Knowledge Dissemination Commercialization Progress

Energy Benefits Economic Benefits Environmental Benefits

Overall Value

Year-One Results

Not rated

18

Year-One Results

Mean of Peer Reviewer Ratings

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

AdequacyInformation

Clarity ofAssessment

Criteria

Importance ofAssessing R&D

Programs

Adequacy ofAssessment

Criteria

Usefulness

19

Year-Two Activity

• Conducted a peer-review assessment of two programs

• Combined Heat and Power Demonstration Program: ~$50 million awarded– 3-hour teleconference with presentation by

program manager• Environmental Monitoring Program: ~ $20 million

awarded– 5-hour on-site meeting with presentations by staff

on critical research findings• Reviewers held 30-minute meeting after

presentations

20

Assessment Criteria in Year 2• Knowledge Creation

– Quantity – Significance

• Knowledge Dissemination– Availability of Knowledge Products: – Impact on target audience.

• Commercialization Progress– Capital Attraction: – Technical Achievement: – Market Advancement:

• Likelihood of Realizing Significant Energy Benefits• Likelihood of Realizing Significant Economic Benefits• Likelihood of Realizing Significant Environmental and Health Benefits• Value versus Costs

21

Year-Two Results

Rater Scores for CHP and Env. Monitoring Programs

00.5

11.5

22.5

33.5

44.5

Knowledg

e Cre

ation

Knowledg

e Diss

eminati

on

Commerci

aliza

tion

Progr

ess

Energ

y Ben

efits

Econom

ic Bene

fits

Enviro

nmen

tal B

enef

its

Value

vers

us C

ost

CHP

Env. MonitoringNot Rated

24

Reviewer Comments

Knowledge (Quantity)• The program has produced a large number of published papers in

quality journals. • Staff have gone the extra mile to summarize these for those interested

in an overview of findings.• Improvements could be made to expand the amount of knowledge (as

opposed to the amount of data) created. .

25

Knowledge (Quality)• The information on sulfur, nitrogen, and mercury deposition and its

effects on surface waters and fauna as well as studies of the responses of surface waters to changes in emissions are of particular value and probably the most important contribution of this program.

Availability of Knowledge Product• The program has done an exemplary job of making results from their

projects available to multiple audiences ranging from the scientific community to policy-makers.

• To the extent possible, links to the actual papers would be very helpful as well.

• EMEP should consider more detailed evaluation of the internet contact information. That information would improve the program’s knowledge of which documents are found to be useful by web users.

Reviewer Comments

26

Reviewer CommentsTarget Audience• Given the mission of NYSERDA over all, and the focus of the program on

impacts of power industry operations, there should be greater efforts to engage the industry.

• EMEP might consider some sort of survey of the relevant target audiences to estimate the extent to which the appropriate target markets are reached.

Environmental• Determining whether the likelihood of increase is “significant” or if those

increases are likely to be “substantial” is extremely difficult to gauge. • A large number of new fish consumption advisories were found to be needed

as a consequence of the monitoring. Some health benefits are likely, but their significance is probably difficult to quantify.

Overall Value versus Cost• Estimating the value of research on a dollar basis is next to impossible,

particularly given that much of the benefit from program-supported research is not likely to be measurable for some time after the research is completed.

• I was amazed that the oversight of the program is conducted with only 2.5 full-time-equivalent employees.

27

Year-Two Results

Feedback from Peer Reviewers

00.5

11.5

22.5

33.5

4

Adequecy ofInformation

Packet

Clarity ofInstructions

Clarity ofcriteria

OverallRelevance of

Criteria

CHP

Env. Monitoring

28

Year-Two Results

Feedback on Relevance of Criteria

0

0.5

1

1.5

2

2.5

3

3.5

4

Knowled

ge Q

ty.

Knowled

ge S

ig.

Knowled

ge A

vail.

Targe

t Aud

ience

Capita

l Attr

actio

n

Tech.

Ach

ievem

ents

Mar

ket A

dvnm

nt

Impr

oved

Out

look

CHP

Env. Monitoring

29

Conclusions

• Logic model was useful in identifying project types and assessment criteria

• Peer reviewers were able to provide useful feedback• The assessment criteria appear to be clear and

relevant• Assessment activity validated the project types• Framework can be applied to a group of related

projects• Although too early to tell, R&D programs appear to

be on track to achieving long-term energy, economic, and environmental benefits

30

Next Steps

• Develop methodologies for measuring economic impacts – Economic case studies

– Macroeconomic impact analysis based on new investments and sales of new products

• Develop ratios to forecast future benefits of product development activities