National Action Plan for Energy Efficiency

www.epa.gov/ eeactionplan

Model Energy Efficiency Program Impact Evaluation Guide - Basics of EM&V

Steve Schiller. Schiller Consulting

steve@schiller.com

www.schiller.com

NARUC Workshop

February 2008

www.schiller.com 2

Topics

Evaluation objectives

Categories of programs and evaluations

Some more on the Action Plan Guide

Calculating energy savings and avoided emissions

Issues to think about when designing evaluation

efforts and issues for discussion today

Resources - references

www.schiller.com 3

Why Evaluate Energy Efficiency Programs?

Accountability. Document and measure the effects of a program in order to determine how well it has met its intended outcomes (goals).

Improvement. Understand why those effects occurred and identify ways to improve current and future programs

www.schiller.com 4

Efficiency Program Categories - by their primary objectives

Resource acquisition. Directly achieve energy savings, and possibly avoid emissions, through specific actions.

Market transformation. Change the way in which energy efficiency markets operate and achieve either naturally occurring efficiency or incremental codes/standards. Can also include support for emerging technologies.

Codes and standards. primary objective is to define and enforce mandated levels of efficiency in buildings and products.

Education and training. primary objective is to inform consumers and providers about energy efficiency and encourage them to act on that information.

www.schiller.com 5

Efficiency Evaluation Categories Impact evaluations. Determine the impacts (usually energy

savings) and co-benefits that directly result from a program. Process evaluations. Assess how efficiently a program was or is

being implemented. Market effects evaluations. Estimate a program’s influence on

encouraging future energy efficiency projects because of changes in the marketplace.

Cost-effectiveness analyses. Typically seen as an extension of impact evaluations, but may also take into account market evaluation results considering market penetration over the expected lifetime of the measures.

www.schiller.com 6

Back to the NAPEE Guide - Why a Program Guide?

Programs are different from projects

There are widely recognized protocols for the measurement and verification (M&V) of energy savings from single projects e.g., International Performance

Measurement and Verification Protocol (IPMVP at www.evo-world.org)

Policy neutral guides for determining savings from programs must be interpreted from existing specific program protocols

www.schiller.com 7

Guide Scope: Programs Addressed Primary focus (i.e., includes detailed guidance):

Resource acquisition, downstream energy efficiency programs - directly achieve energy and/or demand savings, and possibly avoid emissions, through specific actions

Secondary focus (i.e., addressed, but no detailed guidance): Other demand-side programs: Market transformation,

codes and standards, demand response, and upstream efficiency programs

Supply-side programs: renewable energy and combined heat and power (CHP) program

www.schiller.com 8

Guide Scope: Evaluation Focus

Primary focus: Impact evaluation, including: kWh, kW, therm savings

and avoided emissions • Three basic gross savings analysis approaches• Four basic net savings analysis approaches• Two basic avoided emission analysis approaches

“Very” Secondary focus: Process and market evaluations Potential studies Cost-effectiveness evaluation

www.schiller.com 9

Guide ContentsAlso includes about 40 “sidebars” of examples/clarifications and 25 figures and tables

Part 1 Executive Summary

Part 2 Chapter 1: IntroductionChapter 2: Energy Efficiency Program EvaluationChapter 3: Impact Evaluation Basics

Part 3 Chapter 4: Calculating Gross Energy and Demand SavingsChapter 5: Calculating Net Energy and Demand SavingsChapter 6: Calculating Avoided Air Emissions

Part 4 Chapter 7: Planning An Impact Evaluation

Part 5 Appendix A: Leadership Group ListAppendix B: GlossaryAppendix C: Other Evaluation TypesAppendix D: UncertaintyAppendix E: Resources and ReferencesAppendix F: Renewables and Combined Heat and Power Program EvaluationAppendix G: References

www.schiller.com 10

Guide Audience Policy-makers:

Information about the basic principles of efficiency evaluation

Program evaluation managers: Resources and background on evaluation and statistical analyses Guidance, or a roadmap, on process and key issues relating to documenting

energy/demand savings and avoided emissions

Program designers: Understanding of how their programs will be evaluated

Energy efficiency community: Common terminology definitions A central reference document References for publicly available best practices resources Understanding of efficiency’s potential value as an emissions control strategy

www.schiller.com 11

Impact Evaluation Results Reported Estimates of gross savings. Gross energy (or demand) savings are the

change in energy consumption and/or demand that results directly from program-promoted actions taken by program participants regardless of the extent or nature of program influence on their actions.

Estimates of net savings. Net energy savings refer to the portion of gross savings that is attributable to the program. This involves separating out the impacts that are a result of other influences, such as consumer self-motivation. Given the range of influences on consumers’ energy consumption, attributing changes to one cause (i.e., a particular program) or another can be quite complex.

Estimates of co-benefits. A co-benefit commonly documented and reported is avoided air emissions: the air pollution or greenhouse gases that would have been emitted if more energy had been consumed in the absence of the energy efficiency program.

www.schiller.com 12

Approaches for Determining Gross Energy Savings

One or more measurement and verification (M&V) methods from the IPMVP are used to determine the savings from a sample of projects. These savings are then applied to all of the projects in the program.

Deemed savings, based on historical and verified data, are applied to conventional energy efficiency measures implemented in the program.

Statistical analyses of large volumes of metered energy usage data are conducted.

www.schiller.com 13

Approaches for Determining Net Energy Savings

The primary, but not exclusive, considerations that account for the difference between net and gross savings are free riders and participant and non-participant spillover.

Approaches (categories):• Self-reporting surveys

• Enhanced self-reporting surveys • Statistical models that compare participants’ and

non-participants’ energy and demand patterns• Stipulated net-to-gross ratios

www.schiller.com 14

Approaches for Determining Avoided Emissions

Applying emission factors (e.g., pounds of CO2 per MWh) to net energy savings

Using emissions scenario analyses, e.g., using computer models to estimate the difference in emissions from power plants with and without the reduced electricity consumption associated with an efficiency program.

www.schiller.com 15

Issues Addressed in NAPEE Guide Defining evaluation goals and scale and which benefits to evaluate

Setting time frame for evaluation and reporting expectations

Setting spatial boundary for evaluation (i.e. what energy uses, emission sources, etc. will be included in the analyses)

Defining baseline, baseline adjustments, and data collection requirements

Establishing a budget vis-à-vis expectations for quality of reported results

Selecting impact evaluation approaches for gross and net savings calculations and avoided emissions calculations

Selecting who (or which type of organization) will conduct the evaluation

www.schiller.com 16

Evaluation Issues for Discussion Context:

Efficiency is the first resource and a critical part of cheap, reliable, clean and stable energy systems

Efficiency is the first mechanism for energy sector climate change mitigation

• 50- 80% GHG emissions reduction is required for climate stabilization

• EE has a key role according to IPCC, IEA, McKinsey and other studies

Each jurisdiction's resource and climate impact evaluation requirements are not policy-neutral (but the Guide is)

www.schiller.com 17

Evaluation Issues for Discussion How will efficiency savings be documented as real?

• What is additional? • How does one attribute the savings to a particular activity?• How accurate is accurate enough and how do you balance quality of

results and evaluation budgets

Will cross-jurisdictional, post-Kyoto, national or international requirements (standards) be established? Is that a good idea?

How will the data, tools and technology for cost-effective EM&V be developed and supported?

Who should do evaluation, the program administrator, the regulator, and/or third-parties?

www.schiller.com 18

References - Sample Resources - Program Evaluation

2007 NAPEE Model Energy Efficiency Program Impact Evaluation Guide. www.epa.gov/cleanrgy/documents/evaluation_guide.pdf

2006 California Energy Efficiency Evaluation Protocols: http://www.calmac.org/publications/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf

2004 The California Evaluation Framework. http://www.calmac.org/publications/California_Evaluation_Framework_June_2004.pdf

2007 US DOE Impact Evaluation Framework for Technology Deployment Programs. http://www.eere.energy.gov/ba/pba/km_portal/docs/pdf/2007/impact_framework_tech_deploy_2007_main.pdf

2006 International Energy Agency. Evaluating Energy Efficiency Policy Measures & DSM Programmes. http://dsm.iea.org

www.schiller.com 19

References - Sample Resources - M&V

2007 International Performance Measurement and Verification Protocol (IPMVP). (www.evo-world.org)

2000 Federal Energy Management Program M&V Guidelines. (http://ateam.lbl.gov/mv/docs/26265.pdf)

2002 ASHRAE Guideline 14 Measurement of Energy and Demand Savings. (www.ashrae.org)

www.schiller.com 20

References - Sample Resources - EM&V Websites

Evaluation reports and guidance documents, they can be found at two web-accessible databases: CALifornia Measurement Advisory Council (CALMAC):

http://www.calmac.org. The Consortium for Energy Efficiency’s Market Assessment and

Program Evaluation (MAPE) Clearinghouse: http://www.cee1.org/eval/clearinghouse.php3.

Other Sites: Proceedings of the IEPEC Conference (http://www.iepec.org) ACEEE Summer Studies (http://www.aceee.org)

Efficiency Valuation Organization (EVO) resource pages (www.evo-world.org)

www.schiller.com 21

Time for questions

Reprinted with permission from Don Piraro