contents · 2015. 2. 19. · contents sbdi final evaluation report cecony iii appendix a:...

Contents

CECONY i

1 EXECUTIVE SUMMARY .......................................................................................... 1

1.1 IMPACT EVALUATION OBJECTIVES .................................................................................. 1

1.2 INTERIM REPORT CONTEXT ............................................................................................ 2

1.3 RESEARCH APPROACH .................................................................................................. 2

1.4 RESULTS SUMMARY ...................................................................................................... 3

1.4.1 Gross Savings ........................................................................................................................... 3

1.4.2 Net-to-Gross Ratio ..................................................................................................................... 4

1.4.3 Net Savings ............................................................................................................................... 4

1.4.4 Comparison with Billing Analysis ............................................................................................... 5

1.5 RECOMMENDATIONS ...................................................................................................... 6

1.5.1 Program Recommendations ...................................................................................................... 6

1.5.2 New York Technical Manual Recommendations....................................................................... 7

1.5.3 Evaluation Recommendations ................................................................................................... 8

2 INTRODUCTION ...................................................................................................... 9

2.1 PROGRAM BACKGROUND ............................................................................................... 9

2.2 PROGRAM GOALS AND OBJECTIVES ..............................................................................10

2.3 PROGRAM DELIVERY ....................................................................................................11

3 EVALUATION METHODOLOGY ........................................................................... 14

3.1 OVERALL APPROACH ....................................................................................................14

3.2 SAMPLE DESIGN ...........................................................................................................15

3.2.1 Realization Rate Sample Design ............................................................................................. 16

3.2.2 Within-Site Sampling to Select Fixtures for Logging ............................................................... 17

3.2.3 Attribution Sample Design ....................................................................................................... 17

3.3 GROSS SAVINGS REALIZATION RATE .............................................................................20

3.3.1 Engineering Analysis ............................................................................................................... 20

3.3.2 Program Measurement and Verification .................................................................................. 22

3.3.3 Baseline ................................................................................................................................... 22

3.3.4 Measurement and Verification Site Work ................................................................................ 23

Contents SBDI Final Evaluation Report

ii CECONY

3.3.5 iPad Tool .................................................................................................................................. 24

3.3.6 Calculation of Project Level Results ........................................................................................ 24

3.4 DISCREPANCY ANALYSIS ..............................................................................................24

3.5 ATTRIBUTION ...............................................................................................................26

3.5.1 Free Ridership ......................................................................................................................... 27

3.5.2 Spillover ................................................................................................................................... 30

3.6 POTENTIAL GROSS IMPACT ERROR AND BIAS .................................................................31

3.6.1 Sample Bias............................................................................................................................. 31

3.6.2 Non-Response Bias ................................................................................................................. 31

3.6.3 Measurement Error .................................................................................................................. 32

3.7 BILLING ANALYSIS ........................................................................................................32

4 RESULTS ............................................................................................................... 33

4.1 FINAL SAMPLE DISPOSITION ..........................................................................................33

4.1.1 Realization Rate Sample Disposition ...................................................................................... 33

4.1.2 Attribution Sample Disposition................................................................................................. 34

4.2 GROSS SAVINGS REALIZATION RATE .............................................................................35

4.2.1 Engineering Based Site-Specific Savings ............................................................................... 35

4.2.2 Site-Specific Reasons for Deviations from Tracking ............................................................... 38

4.2.3 Operating Hours Discrepancy ................................................................................................. 39

4.2.4 Program Level Gross Realization Rate and Components ...................................................... 40

4.3 ATTRIBUTION ...............................................................................................................43

4.3.1 Free Ridership ......................................................................................................................... 43

4.3.2 Alternative Free Ridership ....................................................................................................... 44

4.3.3 Participant Spillover ................................................................................................................. 45

4.3.4 Combined Net-to-Gross Ratio ................................................................................................. 46

4.4 NET REALIZATION RATE ................................................................................................46

4.5 COMPARISON WITH BILLING ANALYSIS ...........................................................................47

4.6 IMPLICATIONS FOR THE NEW YORK TECHNICAL MANUAL .................................................49

5 CONCLUSIONS AND RECOMMENDATIONS ...................................................... 51

5.1 OBSERVATIONS AND CONCLUSIONS ..............................................................................51

5.2 RECOMMENDATIONS .....................................................................................................51

5.2.1 New York Technical Manual Recommendations..................................................................... 51

5.2.2 Program Recommendations .................................................................................................... 51

5.2.3 Evaluation Recommendations ................................................................................................. 53

Contents SBDI Final Evaluation Report

CECONY iii

APPENDIX A: STATISTICAL SAMPLING METHODOLOGY

APPENDIX B: RATIO EXPANSION – SAMPLE TO POPULATION

APPENDIX C: FACILITY TYPE MAPPING

APPENDIX D: ADDITIONAL LOOKS AT RESULTS

APPENDIX E: DETAILED IPAD TOOL DESCRIPTION

APPENDIX F: ATTRIBUTION METHOD

APPENDIX G: PARTICIPANT SURVEY INSTRUMENT

APPENDIX H: PARTICIPANT SURVEY FIELDING AND FINAL SAMPLE DISPOSITION

APPENDIX I: GLOSSARY OF TERMS

APPENDIX J: INDIVIDUAL SITE SUMMARIES

APPENDIX K: OPERATING HOURS HISTOGRAM

Con Edison SBDI

Final Impact Evaluation Report

CECONY 1

1 EXECUTIVE SUMMARY

Consolidated Edison Company of New York (CECONY) has completed the delivery of the

first cycle (2009 – 2011) of a portfolio of Energy Efficiency Portfolio Standard (EEPS) Utility

Administered programs, as ordered by the New York Public Service Commission. This is

the final report of the impact evaluation of the CECONY Small Business Direct Install

(SBDI) program.

CECONY is committed to independent and transparent program evaluations. Through a

competitive bid process, CECONY selected an evaluation team led by ERS to complete impact

evaluations for their EEPS programs. The ERS team for the SBDI evaluation also includes DNV

KEMA, APPRISE, and Opinion Dynamics Corp (ODC). ERS is the lead in this particular

evaluation effort. CECONY’s Section Manager for Measurement, Verification, & Evaluation is

managing the impact evaluation. This Section Manager reports directly to the Director of

Energy Efficiency Programs to maintain internal independence.

1.1 Impact Evaluation Objectives

The intent of this impact evaluation of the SBDI program is threefold:

1. To evaluate the program’s recent performance by developing gross savings realization

rates (RRs) and a net-to-gross ratio (NTGR) that measures the attribution of savings for

the SBDI program.

2. To provide information to the New York Standard Approach for Estimating Energy

Savings from Energy Efficiency Programs (from here on referenced as New York

Technical Manual [NYTM]) authors that will help them update key deemed savings

input parameters affecting the SBDI program based on New York-specific performance

data.

3. To provide actionable recommendations for improving the program’s implementation.

The program research focused solely on lighting measures, which contribute approximately

97% of program savings.

SBDI Final Impact Evaluation Report

2 CECONY

1.2 Interim Report Context

This report is the final overall program impact evaluation report. It is the fourth in a series of

reports submitted to CECONY during the 12-month evaluation period. The previous three

reports were internal interim reports provided to CECONY to progressively deliver results at

specific stages of the overall project. The reports are:

1. Interim RR analysis based on billing analysis

2. Interim RR analysis based on on-site verification and self-reported hours of use

3. Interim RR analysis based on on-site verification and 3 months of logged time-of-use

performance and NTGR analysis

4. Final Updated RR analysis based on on-site verification and 12 months of logged time-of-

use performance and NTGR analysis

The ERS team has previously delivered the first three reports. This fourth and final overall

program impact evaluation report presents results from the verification site visits and

operating hours based on 12 months of metering conducted by the evaluator. This report also

presents the Net-to-Gross (NTG) results and the facility type-specific lighting hours of use to

compare with that specified in the New York Technical Manual (NYTM).

1.3 Research Approach

Program evaluated gross savings RRs were determined through an on-site M&V sample of 133

sites, with monitoring of 1,210 unique lighting circuits, and a telephone survey sample of 450

interviews with program participants. The sample was drawn in March 2012 from the

population of projects completed in 2011.

On-site evaluators verified measure installation and installed lighting loggers to measure

operating hours and peak diversity1, key factors in the NYTM calculation algorithms.

Customer staff members were interviewed to determine operating characteristics of the

retrofitted area within the facility. Necessary HVAC information was also collected to quantify

interactive effects. All this information was used to calculate the gross savings RR.

The attribution factors were derived from self-reported information from telephone interviews

with program participants. The evaluation team relied on the self-report method to derive both

free ridership (FR) and spillover (SO) estimates. Program participants were interviewed and

1 Peak diversity is the likelihood that the lighting fixtures will be operating at the time of system peak.


CECONY 3

asked a series of structured and open-ended questions about the influence of the program and its

various components and on the decision to purchase or install energy efficient lighting fixtures.

The ERS team also performed a program-level pre-/post-retrofit billing analysis of late 2010

and early 2011 participants to compare with the sample-based metering and NTG study

results. The billing analysis used multiple techniques, including pooled fixed effects and

several variants of the PriSM-type methods.2

1.4 Results Summary

The RR quantifies the variance between the program-reported savings and the evaluation’s

estimated savings. It is defined as the evaluated savings for the population divided by the

program-reported savings for the population.

1.4.1 Gross Savings

Table 1-1 shows the aggregate gross energy savings and RRs for the Program. The gross RR

based on the metered hours and on-site verifications is 76%, and the relative precision estimate

is 9% at the 90% confidence interval.

Table 1-1. Gross Energy Savings and RR by Building Type

The three most significant reasons that engineers identified for the evaluated savings deviating

from – and in particular for being less than – the tracking savings were operating hours (17%

lower than tracking estimates), quantities (6% lower), and technology (2% lower). The report

body and appendices detail results as a function of measure type, cost type, and borough.

2 CECONY EEPS Programs SBDI Impact Evaluation Preliminary Billing Analysis, Energy & Resource Solutions team,

work led by DNV KEMA, September 19, 2012 and Orange & Rockland EEPS Programs SBDI Impact Evaluation Billing

Analysis, Energy & Resource Solutions team, work led by DNV KEMA, October 5, 2012. PriSM refers to the

Princeton Scorekeeping Method (PRISM™, Center for Energy and Environment Studies, Princeton, NJ 1995)

software, but it has become a generic term for site-level weather normalized consumption modeling using gas and

electric billing data records.

Building Type

Percent

Program

Savings

(Tracking)

Min

n*

Gross Savings

Realization

Rate

Relative

Precision at

90%

Confidence

Measured

Gross

Program kWh

Savings

Retail 30% 63 80% 14% 25,810,748

Offices 8% 73 75% 12% 6,634,822

Parking Garages 9% 57 83% 8% 8,251,934

Other 53% 38 72% 16% 41,260,504

Entire program** 100% 231 76% 9% 81,958,008

* The minimum number of sampled measures from w hich the results are based

** Overall program level numbers are savings w eighted totals/averages


4 CECONY

1.4.2 Net-to-Gross Ratio

Attribution research found that 17% of the program’s savings would have occurred in the

absence of the program due to free ridership (FR). Measured participant SO was negligible.

Initially FR of 17% seemed high for SBDI programs but in a NTG study of similar SBDI

programs in the Northeast conducted in 2011, FR was 13% and SO < 1%. So the 17% free

ridership is within the error bounds (+/- 10%) of these other programs. The combined NTGR

for CECONY is 83%. Parking garages have the lowest NTGR at 69% and highest free ridership

at 31%, while the other three building types have NTGRs of 84% or 85%. The high

concentration of parking garages that are chains in Con Edison territory could be a factor in the

higher than expected FR results. Other breakouts that had low NTGR (70%–86%) included:

compact fluorescent lamps (CFLs), customers with some free measures, and chains. Details for

other breakouts are presented in Appendix D. Table 1-2 shows the NTGR by building type and

the overall free ridership.

Table 1-2. NTGR by Building Type

1.4.3 Net Savings

The combined program net impact factor based on site-specific M&V and NTG interviews is

63% (76% RR × 83% NTGR). Table 1-3 presents the net savings by building type.

Table 1-3. Net Energy Savings by Building Type

Building Type Min n*

Combined Net-

to-Gross

Ratio

Relative

Precision at 90%

Confidence

Retail 136 86% 4%

Offices 75 85% 8%

Parking Garages 21 69% 15%

Other 311 83% 5%

Overall NTGR 543 83% 3%

Overall FR 543 17% 3%



Net

Realization

Rate

Relative

Precision at 90%

Confidence

Net Program

kWh Savings

Retail 63 69% 15% 22,261,770

Offices 73 64% 14% 5,661,715

Parking Garages 21 57% 17% 5,666,991

Other 38 60% 17% 34,383,753

Entire program** 231 63% 9% 67,974,229




CECONY 5

1.4.4 Comparison with Billing Analysis

The program-level billing analysis found RRs between 18% and 21% for CECONY and

between 52% and 54% for Orange & Rockland (O&R). At 90% confidence the relative precision

ranged from 13% to 32%, depending on method and whether or not CFL-only sites were

included in the analysis.

The evaluation team chose to use the M&V-based results alone instead of the test billing

analysis for final reporting due to the gross discrepancy between the two methods.. The

reasons for this are:

There was evidence of a strong pattern of missing account data from the billing

analysis. CECONY has complex space and metering arrangements in all boroughs but

especially in Manhattan. These complex arrangements lead to inadvertently omitted

account data. Further, there is the possibility that the implementation contractors

withheld account information in order to maintain eligibility for the SBDI program. In

either case missing account data biases billing analysis savings downwards.

The billing analysis and M&V populations overlap but are not the same. Specifically,

the billing analysis group is from an earlier time period. The early period was necessary

to collect a year’s worth of post-retrofit billing data. CECONY believes that program

tracking savings were relatively higher and thus realization rates lower in early-year

operations than later-year operations. Directives with restraints on limits for excessive

installer estimated hours were not in place during the early months of the program.

When this was realized restrictions were put in place in October 2010.

The M&V-based approach has relatively few uncertainties compared to billing analysis.

There is not enough uncertainty or potential bias in the M&V approach to justify

making a major downward change in realization rate by incorporating the billing

analysis into the final evaluated gross savings results.

Small businesses are not a very stable population. .This instability could contribute to

billing data errors due to changes in business type, ownership, etc.

These reasons are further discussed in detail in Section 4.5. In summary, the initial pilot nature

of the billing analysis, the confidence in the M&V results, the known biases in the billing

analysis, and the inability to otherwise explain the differences led the team to choose to use the

primary method of M&V alone for reporting.

In addition, the billing analysis results for a similar program in New York were informally

discussed by the program administrators. The billing analysis results and discrepancies


6 CECONY

between billing analysis and onsite results for both the programs were found to be in similar

ranges.

1.5 Recommendations

The evaluation team’s recommendations for the program, the NYTM, and any future

evaluations are given in the following section.

1.5.1 Program Recommendations

The evaluation team offers seven recommendations for increasing program realization rates,

savings and cost-effectiveness. These are listed below and then discussed in detail. The

evaluation team’s recommendations focus on driving participation toward higher energy

savings per transaction or on lowering the cost of achieving savings.

Revise operating hours – The evaluation team found that parking garages operate more

hours than that suggested by NYTM, whereas the offices and retail facilities operate

fewer hours than the NYTM-suggested values. If the DPS chooses not to update the

NYTM-deemed hour estimates in response to this research, the evaluators recommend

that the SBDI program itself apply the operating hours for the various building types

identified in the study. This will improve the savings estimation accuracy using data

most applicable to the SBDI program population and increase the RR in the next

evaluation.

Account Tracking – Identify all Con Edison primary usage accounts associated with each

facility or part of a facility that participates in the program for customers that possess

multiple accounts.

Use pre-retrofit billing data for QC – During the first EEPS cycle, the SBDI installation

contractor used pre-retrofit billing data to cross check the savings magnitude for every

project. The EM&V group flags sites with high savings to usage ratio and visits them to

verify the measures. However, no action was taken by the program contractors on such

sites with high savings to usage ratios. We recommend that the program staff or the

implementation contractor inspect all the job sites with savings to usage ratios greater

than or equal to 80% and adjust the claimed savings accordingly.

Standardize the use of NYTM hours – Standardize the use of NYTM hours as a matter of

general procedure instead of site-specific estimates. The program standardized on NYTM

hours in October of 2010 but evaluators still found some sites where installer reported

hours were used for estimating savings.


CECONY 7

Do not rely on savings from CFLs – CFLs had lower installation rates and higher FR

than the rest of the program measures. As CFLs become standard practice, net savings

from CFLs will continue to decline. CFLs may still be useful to the program as a “loss

leader” measure that can be offered to get auditors in the door, but most of the savings

from CFLs will likely not be realized.

Include install location information in the tracking data– The tracking data included all

the lighting inventory information except the installation locations. The evaluators

extracted the installation location information from the actual work orders, which was a

time-consuming process. We recommend that CECONY look into including the

installation location information in the tracking data for each measure or group of

measures installed, which will then serve as a more complete source of data for impact

evaluations and post installation inspections.

Track service addresses. Ensure that the site address matches the location where the

measures were installed and not the billing address for the participant.

1.5.2 New York Technical Manual Recommendations

The evaluators recommend that the NYTM authors either update the operating load hours for

selected building types based on this study’s research, as shown in Table 1-4, or create new

“small business” or “downstate” retail, office, and parking garage categories specifically for

SBDI programs that reflect this study’s findings. The table presents both the current NYTM

deemed hours and the recommended updated values based on this evaluation’s research.

Section 4.7 provides additional details.

Table 1-4. Recommended Updated Hours by Business Type

Description

NYTM Current

Hours

Study-Measured

Hours - CECONY

Banks 3,748 3,013

Court house 3,748 3,013

Library 3,748 3,013

Medical offices 3,748 3,013

Museum 3,748 3,013

Office/retail 3,748 3,013

Post office 3,748 3,013

Town hall 3,748 3,013

Small services 3,750 2,995

Retail 4,057 3,458

Parking garages 4,368 7,717


8 CECONY

If the DPS chooses not to update the NYTM-deemed hour estimates in response to this

research, the evaluators recommend that the SBDI program itself apply the operating hours for

specific building types as obtained through the study. This will improve the savings estimation

accuracy using data most applicable to the CECONY SBDI program population and increase

the RR in the next evaluation.

1.5.3 Evaluation Recommendations

Upon completing this evaluation, the evaluation team has some recommendations for ways to

improve future SBDI evaluations.

The error ratio used for future SBDI evaluations needs to account for the reliability of the

tracking data. The final error ratio based on the RR of this study was 0.9, indicating that

there was more variation in the sample than anticipated by the typical 0.3 or 0.4

assumption generally used for sampling.

Billing analysis – This population is not a good candidate for program-level billing

analysis of savings. We recommend that it not be attempted again for Con Edison due to

the complex account association challenges found in New York City, unless the program

is able to definitively identify all accounts affected by project activities, with emphasis on

the primary usage accounts located at the site of record. This appears to be a consistent

pattern as similar billing analysis results were observed for other PA small business

programs.

Prior to processing the twelve month logged data, the evaluators expected the numbers

to remain in the same ball park as the three month logged data. However, the overall

twelve month metered operating hours went up by 8% compared to three month metered

data. Given the observed increase in operating hours for longer duration metering, the

evaluators recommend metering for longer duration whenever possible.


CECONY 9

2 INTRODUCTION

This evaluation group consists of the CECONY and O&R Small Business Direct Install (SBDI)

programs. This report focuses on results for CECONY only.

2.1 Program Background

CECONY designed the SBDI programs for rapid deployment of energy efficiency measures to

existing small commercial and industrial (C&I) customers. Both companies defined small C&I

customers as those with facilities that have an average monthly peak demand of less than or

equal to 100 kilowatts (kW).

SBDI participants are recruited primarily through door-to-door solicitation. The prospective

participant is audited, free CFLs are sometimes installed, and additional measures are

identified for installation. The energy surveys conducted during the audit were designed for

the SBDI program to engage customers, to provide customized recommendations for energy

efficiency upgrades, and to document existing equipment. The contractors who conduct the

surveys discuss appropriate behavioral and operational energy efficiency actions, inspect the

customer’s equipment and building envelope, and provide recommendations on cost-

effective energy efficiency upgrades. Following the energy survey, a summary of

recommended energy efficiency measures is provided.

If the customer agrees to the installation of additional measures, the implementation

contractor arranges for an electrical contractor to visit the customer site and provide

complete pricing and installation services. The customer is required to pay 30% of the

installed cost; the program pays the balance.

Table 2-1 summarizes the incentives for the program energy efficiency measures (free versus

reduced cost). The CECONY SBDI program offers up to $100 of free CFLs, faucet aerators, and

high-pressure rinse valves.


10 CECONY

Table 2-1. Summary of SBDI Program Incentives

Approximately 97% of the savings to date are attributable to the installation of high efficiency

lighting. A mix of refrigeration and HVAC control measures constitutes the balance.

2.2 Program Goals and Objectives

The SBDI program is designed to cost-effectively contribute to New York State’s energy

efficiency goals.

Per the CECONY SBDI program filings (submitted August 21, 2008 to the Public Service

Commission), specific objectives associated with these programs include:

Reducing energy use, peak demand, local air pollution impacts, and carbon dioxide

emissions in CECONY service territories

Maximizing available cost-effective energy savings for every small business participant in

the program

Effectively driving the adoption of low-cost, high-value energy efficiency measures in

customer facilities

Measure Eligibility Incentives

Efficient lighting package Meets federal code 70% of installed cost

High-efficiency lighting package Above federal code by 15% 70% of incremental installed cost

Compact fluorescent lamps ENERGY STAR® Free

LED exit signs 5 watts 70% of installed cost

LED refrigeration case lights 28 watts 70% of incremental installed cost

Occupancy sensors Fluorescent 70% of installed cost

Bi-level control for stairwell lighting 50% lighting power during

unoccupied time

70% of installed cost

Refrigerated walk-in evaporator fan

controls

Constant speed always on 70% of installed cost

Refrigerated case

Door gasket replacement N/A

Anti-sweat heater controls Variable temperature control

Night case covers N/A

Strip curtains N/A

Low-flow aerators 1.5 gallons per minute (gpm) Free

High-pressure rinse sprayers 1.6 gpm Free

Water heater thermostat setback Thermostat setback and

replacement (115°F)

Free

Water pipe insulation R-4 insulation 70% of installed cost

Vending machine controls Passive infrared sensor

monitoring vacancy of area and

cycling cooling controls


HVAC retro-commissioning N/A 70% of cost

Programmable thermostat ENERGY STAR® 70% of installed cost



CECONY 11

Increasing small business customer awareness of energy efficiency opportunities

available in their facility, from both equipment upgrades and behavioral changes

Generating customer awareness of energy efficiency programs available through

CECONY, O&R, NYSERDA, and other entities to support their energy efficiency

objectives

Building higher-level customer, trade ally, and stakeholder relationships by

providing value-added energy efficiency services, training, education, hardware,

verification, and customer support

Supporting the local economy by helping to reduce small-businesses’ operational costs,

utilizing local labor, and promoting the adoption of high-quality, high-efficiency

equipment

The CECONY SBDI program has undergone significant retooling since its launch in 2009.

In particular, in 2010 CECONY reconstructed its SBDI program and re-launched it in

January of 2011. The evaluation team focused this evaluation on 2011 data because of the

program improvements made as a result of the restructuring.

2.3 Program Delivery

For the 2009 – 2011 period, two implementation contractors served the CECONY SBDI

program – Willdan Energy Services (WES) and Free Lighting Corporation (FLC). FLC was

responsible for Staten Island while Willdan covered all of the other CECONY territories,

including Westchester County. 3Both Willdan and FLC hired multiple electrical subcontractors

that were responsible for purchasing materials and installing the proposed energy efficiency

measures at each participating customer site. As discussed earlier, energy efficient lighting is

the most common measure implemented through the program. Table 2-2 provides a summary

of the savings by measure installed by the program in the 2009 – 2011 program period4.

3 As of 12/31/11, FLC’s involvement was curtailed. Subsequently Lockheed Martin was added as an Implementation

Contractor (IC). 4 The SBDI program began in 2009. The sample designs and evaluated results subsequently presented use the 2011

participants, as the evaluators wanted to avoid measuring the effects of start-up issues not indicative of longer-term

program performance, and wanted to interview participants closer to the time of decision-making than would occur

if 2009 and 2010 participants were included. The 2009–2011 population data is presented in this section for

illustrative characterization purposes only. The ERS team affirmed that the 2011 measure mix was substantially

similar to the mix of 2009 and 2010.


12 CECONY

Table 2-2. Summary of Total Gross Savings by Measure Type for Program Years 2009 – 2011

Initially the CECONY SBDI program was reporting customer-reported operating hours for

the facility lighting based on auditor interviews and calculating savings based on this

number. The customer-reported operating hours were not always accurate. As a result, in

early 2010 CECONY mandated that the contractors use standard operating hours by facility

type, per the NYTM. However, it appears that the contractors still use self-reported hours

because the evaluators found instances when tracking hours did not match the NYTM hours.

The final program accomplishments reflected in the Revised Scorecard filed April 2012 for the

2009-2011 program activity are shown in Table 2-3. The difference between the total gross

savings in Table 2-2 above and the total net savings in Table 2-3 is explained by the program’s

presumed NTGR of 90%. In summary, CECONY achieved 70% of its 2009 – 2011 kWh goal,

and program expenditures were less than the budgeted amount.

Category Measure Type Total kWh % of Total

Lighting Stairway bi-level control - 0.0%

Lighting LED exit signs (free) 17,692 0.0%

Lighting Occupancy sensors 1,614,220 0.9%

Lighting LED exit signs 1,832,575 1.1%

Lighting Lighting 2,678,475 1.6%

Lighting LED refrigeration case lighting 3,482,838 2.0%

Lighting CFL (non-free) 8,667,818 5.0%

Lighting CFL (free) 13,344,307 7.7%

Lighting Tube lighting 134,613,213 78.1%

166,251,139 96.5%

Non-lighting Water pipe insulation (free) 1,046 0.0%

Non-lighting Faucet aerators (non-free) 1,355 0.0%

Non-lighting Programmable thermostat 7,221 0.0%

Non-lighting HVAC 15,007 0.0%

Non-lighting Evaporator fan 16,981 0.0%

Non-lighting Strip curtains (free) 24,547 0.0%

Non-lighting Vending machine controls 25,773 0.0%

Non-lighting Faucet aerators (free) 29,848 0.0%

Non-lighting Water pipe insulation 60,258 0.0%

Non-lighting High pressure rinse valve 98,480 0.1%

Non-lighting Door gaskets (free) 187,428 0.1%

Non-lighting Blank 276,171 0.2%

Non-lighting Strip curtains 410,426 0.2%

Non-lighting Refrigerated case night covers 767,493 0.4%

Non-lighting EC motor retrofits 1,150,793 0.7%

Non-lighting Door gaskets 1,319,953 0.8%

Non-lighting Door heater controls 1,616,818 0.9%

6,009,598 3.5%

172,260,737 -

Subtotal

Subtotal

Total


CECONY 13

Table 2-3 Net Savings SBDI 2009 – 2011 Program Scorecard Analysis

Program Administrator (PA) and Program ID CECONY

Program name SBDI

Program type Downstream

Net first-year annual kWh acquired to date 155,127,876

Net program kWh goal 221,225,000

Net first-year annual kWh acquired to date as a percent of annual goal 70%

Net cumulative first-year annual kWh acquired to date 155,127,876

Net utility kW reductions acquired to date 37,798

Net utility peak kW reductions acquired to date as a percent of utility annual goal 74%

Total program budget $76,702,688

Total expenditures to date $56,864,989

Percent of total budget spent to date 74%

Number of program applications received to date 37,477

Number of program applications processed to date 36,053

Number of processed applications approved to date 15,057

Percent of applications received to date that have been processed 96%

Costs

Participation

Total Acquired Net First-Year Impacts To Date


14 CECONY

3 EVALUATION METHODOLOGY

The methodology section describes the methods used to calculate the evaluated net savings of

the program. Separate subsections address the overall approach; the sample designs; RR

method; the calculation of total program savings; tracking; reporting; and attribution

methodology.

3.1 Overall Approach

The evaluation team conducted a retrospective evaluation of a sample of SBDI projects

completed in the year 2011 using on-site M&V. The RR was sample based, with the results

extrapolated to all projects in the population.

This impact evaluation consisted of two major components as described below.

1. Gross savings evaluation of energy efficiency projects – On-site M&V of a sample of sites

to establish RR

2. NTG evaluation – Telephone surveys of participants to estimate NTG and verify

installation of measures

The impact evaluation needs quality data to support the effort. Most of the required data was

obtained in response to the evaluation data request of record and work orders supplied by

CECONY. This data included the following details:

Project level information, including address, contact information for the site owner and

site engineer, and type of business

Measure level information, such as a description of the measure, quantity installed, and

the energy savings

Supporting implemented measure details – installation location (through work orders)

Following data confidentiality and security protocols, CECONY provided the evaluation

team access to the scorecards’ back-up information and vendor data. The verified data was

used as the tracking database. Based on the review of this data, the following data issues

were identified:

The monthly savings for acquired (installed) projects in the data set were compared with

the monthly scorecard information as a quality cross-check to identify any discrepancies.

CECONY staff members were contacted to troubleshoot the differences and establish the

final set of tracking data. Minimal discrepancies were identified during the data review.


CECONY 15

Program data offers most of the detail required by the evaluation effort, e.g., name of

facility, measure level savings, building types, etc. However, there were a few sites with

missing building type information. These unknown building/facility types were listed as

“others” and sampled as such. This is a minor issue. For sites that were missing other

important information, the evaluation team approached CECONY to supply the missing

information.

The work orders contained the locations within the site where the fixtures were installed.

A few sites were missing this data, making it difficult to locate the fixtures and conduct

in-measure sampling. In such cases, the evaluation team counted all the installed fixtures,

in order to supplement the missing fixture-related location information.

The following sections describe how the program tracking data was combined with the

evaluation findings to calculate the savings and, subsequently, the realization rate.

3.2 Sample Design

Two samples were designed for the SBDI evaluation, an on-site sample to evaluate gross

energy savings RRs and installation of measures, and a computer-assisted telephone interview

(CATI) sample to verify installations and gather data for determining program attribution and

SO. Both sample designs used the population of projects completed in 2011 as the frame. The

primary objective of the sample designs was to target a relative precision of ± 10% at the 90%

confidence level for the program’s verified gross savings RR for each of three building types

and better than ±8% on the overall RR and ±10% on the NTGR.

The evaluation’s stratified sample design targeted customers who made a larger contribution

to the total program savings, although the sample was designed to ensure that the evaluation

team would complete surveys and on-site visits with customers with smaller contributions as

well. Targeting a larger proportion of customers with greater savings enabled a more precise

savings estimate while limiting evaluation data collection costs by reducing the number of

surveys and on-site visits. The evaluation used a model-based stratified sampling approach

that defines strata within segments (utility and building type or utility and technology) based

on kWh savings.

The evaluation team collected data from customers based on a random order within the

stratum. Interviewers attempted to contact the customer until the survey was completed or

otherwise terminated. Final disposition codes were completion, refusal, partial complete

termination, or no contact. The last outcome occurred when interviewers or recruiters failed to

make contact with an institutional representative within the designated number of six attempts


16 CECONY

(on different days at different times of the day). Customers selected for the CATI sample were

contacted by phone.

The evaluation team was unable to recruit all of the desired sample targets within each stratum,

especially for those strata where we attempted to recruit a census. When a census stratum

customer terminated without completion, the team transferred a sample point from the stratum

that we were unable to complete to the stratum with the largest contribution to total savings that

still had sites available in the population to sample..

3.2.1 Realization Rate Sample Design

The RR sample was designed to select accounts for on-site visits and engineering-evaluation of

gross energy savings. Lighting savings represents 97% of savings for the SBDI program. The

primary source of uncertainty in the NYTM lighting savings estimates is operating hours,

which were purportedly assigned based on building type. An evaluation goal was to produce

operating hour results by building type to inform the NYTM. Building type is an account level

characteristic, which led us to use accounts as the primary sampling unit for the on-site sample

design. Building type, as assigned in Appendix C, was used to stratify our on-site survey

sample. The survey sample selected customers based on the kWh savings associated with the

lighting measures for individual customer accounts. It was necessary to collapse the sixty-plus

NYTM categories into a smaller number of groups in order for this research to generate

statistically significant results by building type, without increasing the sample size to a cost-

prohibitive number of site visits.5

In total, 133 accounts were targeted in the CECONY sample design. The evaluation team

selected accounts using a model-based stratified sampling approach. This approach defines

strata within segments (program and building type) based on account kWh savings and

allocated targets. Using this technique, the overall relative precision for each of the programs

was optimized. The sample design assumed an error ratio of 0.4, resulting in an anticipated

overall precision of 90/8 by program and 90/10 for most building types as shown in Table 3-1.6

5 The evaluation’s “Offices” category is a consolidation of nine NYTM office-type activities. Eight of the nine have

the same NYTM-deemed hours and the ninth is within 0.05% of the same hours, 3,750 instead of 3,748–effectively

the same. Likewise, the evaluation’s “Retail” category is a consolidation of two NYTM categories, Retail and

Laundromats, that have 4,056 and 4,057 deemed hr/yr, respectively. 6 The error ratio used in the on-site sample design is a measure of the assumed variability in the relationship

between verified gross kWh savings and the tracking gross kWh savings. The 0.4 value assumes a strong

relationship between verified and tracking gross kWh.


CECONY 17

Table 3-1

Simplified SBDI On-Site Sample Design7

3.2.2 Within-Site Sampling to Select Fixtures for Logging

Based on the quantities of the installed fixtures/controls and the number of unique lighting

circuits, within-site sampling was required for certain measures. The evaluation team

attempted to log fixtures in all the different predominant space types at the site in order to

obtain representative operating hours by space type for the entire facility. For extremely large

quantities of fixtures/controls within the site, within-site measure sampling was conducted.

The evaluation team used a formal protocol, described in Appendix E, that was applied in the

field to randomly and representatively select samples of lighting circuits. The number of

circuits to meter representing the given space operating hours was determined based on a

sampling target of 20% relative precision at 80% confidence assuming a 0.50 coefficient of

variation.

3.2.3 Attribution Sample Design

In contrast to the RR sample which used the project account as the sampling unit, the

attribution sample design used the measure as the sampling unit. For the on-sites, the primary

area of uncertainty was lighting operating hours, which were purportedly assigned based on

building type, an account-level characteristic. For the telephone survey, the primary area of

uncertainty was the influence of the program on the decision to install each measure. The

evaluation team chose to sample at a level that would allow respondents to balance other

influences (such as measure cost) with the program influence and respond at a level detailed

enough to accurately reflect the program’s influence. Therefore, the CATI sample design

selected customers based on the kWh savings associated with individual measures rather than

individual customer accounts.

7 The percent program savings breakdown shown in the table is only for lighting measures. This excludes small

lighting measures and non-lighting measures.

Building Type

Planned

Sample

Size

2011

Population

Accounts

Percent of

2011 Program

Lighting

Savings

Expected

Relative

Precision at 90%

Confidence

Retail 43 2,171 32% 10%

Offices 41 1,104 9% 10%

Parking Garages 27 93 10% 10%

Other 22 3,567 50% 15%

Entire program* 133 6,935 100% 8%

* Overall program level numbers are savings w eighted totals/averages


18 CECONY

Analysts stratified the samples by other characteristics in order to produce more descriptive

and detailed results that may help in program planning. These characteristics included:

Measure type – The program tracking databases categorize measures using two fields:

measure type and measure description. The measure type field had twenty possible

values (with varying levels of specificity), while the measure description field had more

than 1,000 unique descriptions. The two database fields did not always agree with one

another; for example, the measure type field categorized several measures as “Tube

Lighting (linear fluorescents),” while the corresponding measure description field had a

value of “COMPACT FLUORESCENT LAMP 13W Spiral.” In cases where measure type

conflicted with measure description, the analysis team reviewed and re-categorized the

measure type based on the measure description field. For the sample design, we grouped

the survey measure types further into four sample measure types: Tube Lighting, CFL,

Other Lighting, and Non-Lighting.

Borough – Where the participating account is located.

Cost type – Once the survey measure type was assigned for each measure, we then

combined the measures into a single record, and categorized the aggregated record as

“some free” if one or more measures of the type were free. If none of the measures of the

type were free, then the measure was categorized as “all low-cost” or “not free.”

Chain – Whether the account is part of a chain with multiple locations.

The evaluators stratified first by characteristic and then by measure annual energy savings,

allocating samples to optimize the relative precision for the programs overall. The sample

design assumed an error ratio of 0.5 for free measures and 0.7 for low-cost measures.

In total, 310 measures were targeted in the Con Ed sample design, which resulted in an

anticipated overall precision of 90/9 as shown in Table 3-2.8 Within each of the segments

shown, there is further segmentation by free/low-cost measure, chain account/not, and

measure size. Altogether there were ninety-three strata in the CATI sample design.

8 The error ratios used in the CATI sample design are a measure of the assumed variability in the relationship

between net kWh savings and the tracking gross kWh savings. The 0.5 value used for free measures vs. the 0.7 value

used for low-cost measures assumes that the relationship between net and tracking gross kWh is stronger for free

measures than it is for low-cost measures.


CECONY 19

Table 3-2 Simplified SBDI CATI Sample Design

Most customers received multiple measures through the program – for example, CFLs and

tube lighting. Because measures were randomized within a stratum, a customer could be

eligible for a survey regarding their CFLs but not yet be eligible for a survey regarding their

tube lighting. However, the evaluation could complete the survey regarding the CFLs, and the

customer could later become eligible for a survey regarding their tube lighting. To avoid

customer burden and repeated attempts at reaching the same person, the evaluation asked

customers about up to three measures, regardless of where each fell within the call order. For

surveys conducted on measures that were not included in the sample or would not have come

up in the normal call order, the results were included in the analysis but given a weight of one,

meaning they represented only themselves and no other measures in the population.

Borough

Sampling

Measure

Type

Planned

Sample

Size

2011

Population

Measures

Percent of

2011 Program

Savings

Bronx CFL 5 413 1%

Bronx Non-Lighting 3 77 1%

Bronx Other Lighting 6 249 2%

Bronx Tube Lighting 22 663 9%

Brooklyn CFL 12 1,881 4%

Brooklyn Non-Lighting 7 240 1%

Brooklyn Other Lighting 13 574 4%

Brooklyn Tube Lighting 57 1,616 21%

Manhattan CFL 6 319 1%

Manhattan Non-Lighting 5 483 1%

Manhattan Other Lighting 23 523 7%

Manhattan Tube Lighting 32 411 12%

Queens CFL 9 901 2%

Queens Non-Lighting 4 167 1%

Queens Other Lighting 14 636 4%

Queens Tube Lighting 47 1,352 18%

Staten Island CFL 3 86 0%

Staten Island Non-Lighting 1 7 0%

Staten Island Other Lighting 2 87 0%

Staten Island Tube Lighting 5 151 1%

Westchester CFL 5 420 1%

Westchester Non-Lighting 2 164 0%

Westchester Other Lighting 6 376 1%

Westchester Tube Lighting 21 468 7%

310 12,264 100%Total


20 CECONY

3.3 Gross Savings Realization Rate

Site M&V activities utilized in this interim report include verification of key factors used in

the NYTM calculation algorithms, and self-reported operating hours as discussed in the

following section.

3.3.1 Engineering Analysis

The intent of this section is to describe and analyze the program implementer savings

estimation methods for lighting measures.

Gross Tracking Savings Algorithm for Lighting Fixtures

In principle, CECONY’s tracking savings utilize the NYTM algorithm in the calculations.

However, as straightforward as the equation appears (shown below), the savings calculations

are implemented in stages, using different sources for each parameter.

The first estimate of savings is made by the SBDI contractor in the field using calculations that do

not include interactive effects or coincidence factors. The estimated savings are typically

calculated for groups of like fixtures located in a discrete area (such as an individual office or

conference room) with identical pre-retrofit lamps and ballasts, identical post-retrofit lamps and

ballasts, and identical operating hours. The “direct measure savings” equations are as follows:

In the above equation, the fixture quantity reflects the number of units removed and installed.

The wattage is derived from tables of generic pre- and post-installation lamp/ballast

combination wattages, which reference the NYTM Appendix C’s prescribed wattages.

The operating hours in the equation ideally reflect the hours of operation of that particular

group of like fixtures. According to the implementation staff, the current practice at CECONY

is to have the SBDI contractors first classify the customer as one of sixty facility types defined

in the NYTM. These building types are listed in Appendix C of this report. Each facility type

has associated lighting hours of operation that are used in calculating the savings for all fixture

groups at that site, including exit signs.


CECONY 21

The NYTM savings algorithm specifies coincidence factors for demand savings calculations.

The coincidence factor is defined as the ratio of the peak lighting demand operating at the time

of the system peak to the connected load. Because not all of the fixtures in the population are

operating at all times, the peak lighting demand is often less than the connected load. The

NYTM-recommended coincidence factor is 1.0 for interior lighting and 0 for exterior lighting.

The tracking system estimates of peak demand use these assumptions as the basis.

The NYTM also specifies interactive factors to account for the effect of reduced lighting-

source heat gain on the HVAC system.

The recommended HVAC interaction factors for lighting energy and peak demand savings are

provided in Appendix D of the NYTM.

Gross Tracking Savings Algorithm for Lighting Controls

Lighting control measures include occupancy sensors, photocell controls, time clocks,

daylighting controls, dimmers, and programmable control systems. These systems save energy

and peak demand by shutting off power to lighting fixtures when the space is unoccupied or

the full level of illumination is not required.

Like the lighting fixture measures, CECONY’s tracking savings utilize the NYTM algorithm in

their calculations.

The NYTM algorithm for lighting controls is presented below.


22 CECONY

Evaluation Savings Algorithms

The evaluators used the same NYTM formulas and collected the inputs for them with the

following variations and enhancements. Relative to this report’s analysis, the evaluation

analysts did the following:

Confirmed that the customer’s facility type was correctly classified in the tracking system.

Verified the quantity of fixtures in place.

Used the site-specific installed efficient lamp/ballast models to determine the actual post-

retrofit combination wattage compared to the “generic” wattages (from NYTM), and

verified whether the tracking system correctly classifies the correct installed generic

lamp/ballast categories. The evaluators found differences in the generic wattages used in

the tracking savings which were corrected prior to the site visits.

Metered the hours of use with time-of-use lighting loggers.

Developed site-specific HVAC interactive effects estimates based on interviews about

days/months per year in heating and cooling modes, and inspection of HVAC system

types, vintage, and efficiency.

Estimated the pre-retrofit system type and operation, especially for the controls measures.

3.3.2 Program Measurement and Verification

The evaluation of the energy efficiency measures involved on-site verification and long-term

metering to supplement the available data in the project files.

3.3.3 Baseline

All the installed measures were characterized as “retrofit” to define the baseline and calculate

first-year savings. “Retrofit,” by definition means that savings are based on the difference

between equipment and operating characteristics of the new high efficiency equipment and the

old inefficient equipment that was replaced.

The baseline information was available from the project documentation (information collected

by the vendors when performing the retrofits). The evaluation team verified this baseline

information through discussions with the site staff and in some cases by observing the

surrounding non-retrofitted fixtures in the facility. Overall, the baseline information in the

tracking data was found to be consistent with that described by the site staff.


CECONY 23

3.3.4 Measurement and Verification Site Work

After receiving tracking program data, completing the sample design, and receiving detailed

project documents for the sampled projects, the engineering team evaluated the savings. The

M&V process had three major steps:

1. The evaluation team, on behalf of CECONY, sent advance letters to each participant to let

them know that they were in the sample and would receive a call from an impact

evaluation team member.

2. The evaluation team scheduled and then conducted an initial site visit, interviewed the

site contact regarding equipment hours of use, overall facility hours and other topics,

and installed metering equipment as appropriate. During the site visit the field

engineering team recorded the observations in an iPad tool. Data from the meters was

collected every 3 to 4 months. For sites that could not be scheduled, the evaluation team

conducted drop-in visits, which were found to be effective.

3. The evaluation field engineers then completed the analysis using the data collected

during the site visit.

The on-site M&V assessed the types and quantities of fixtures installed, the locations of the

measures installed, hours of operation based on discussion with the site staff (self-reported

hours), operating hours based on metering results (metered hours), calculated energy and

demand impacts, and corresponding program-level RRs and coincidence factors. When they

were on-site, the evaluators verified the quantity of lighting fixtures and the technology

against the quantities and technologies in the tracking database. Lighting loggers were

installed on the lighting fixtures to obtain the operating profiles for different space types in the

facility. In addition, through discussion with the site staff, the evaluators collected the typical

lighting start and stop times by season (if applicable) for weekdays, weekends, and holidays

for each distinct space type. This information was used to validate the logged data. The logged

data was then used to calculate the annual operating hours. In addition, the evaluators

attempted to inspect a sample of ballasts to verify the ballast make and model number. The

evaluators also collected data to enable the calculation of HVAC interactive effects through

observations of the heating and cooling systems serving the installed measures. Space type

information for the installed fixtures was also collected during the site visit. The result of the

verification effort is essentially a quantification of what was actually installed and operating at

the facility; the measured savings of the projects; and a good indicator of the overall quality of

the program implementation in general.


24 CECONY

3.3.5 iPad Tool

ERS developed an iPad tool to facilitate the data collection process during the site visits, and to

reduce the post-site visit analysis time for calculating the lighting measure savings.

The tool allowed the evaluators to efficiently collect relevant information during site visits,

including verifying the installed lighting measures (quantity, fixture types, operating schedule,

etc.) and HVAC setup to determine the interactive savings. The evaluator downloaded the pre-

loaded site inventories (from the central database on the ERS server) for the scheduled/drop-in

sites on their iPad, verified and modified the information during the site visit, and then

uploaded the information back to the central database. The tool connected to the central

database using a secure VPN connection.

The iPad tool is discussed in detail in Appendix E.

3.3.6 Calculation of Project Level Results

The RR calculation for an individual project is:

where,

= Project number (i.e., the ith project)

= Evaluated M&V kWh savings (by evaluation M&V contractor)

= tracking kWh savings for the project

The goal of the calculations presented in this report is to estimate RRs at the program level.

Expansion of sample results to the program level is discussed in Section 3.6 and Appendix B.

3.4 Discrepancy Analysis

The evaluation team calculated a number of sub-ratios that addressed important stages in the

energy savings calculation, including the number of units installed, the calculation formula,

technology (type of fixtures) and wattage assumptions, and the assumed operating hours. The

sub-ratios allow the evaluation team to pinpoint the source of the discrepancy between the

tracking and verified savings.

The evaluators identified and calculated three ratios to present the difference between the

evaluated and tracking savings. The three ratios are:


CECONY 25

1. Installation rate – This adjustment factor accounts for the difference between the

number of units that the program tracked and the number of units verified through the

CATI survey or observed during the on-site visits. The installation rate is the ratio of

installed to tracking fixture counts.

2. Calculation adjustment – This adjustment factor accounts for differences in the

technology in the tracking database and what was observed during the on-site visits. It

also corrects inconsistencies in the tracking database’s calculation approach and in the

wattages used in the calculations. It does not include corrections to the tracking

database’s assumed operating hours. In this evaluation the primary adjustments were

due to the field engineers finding different fixture types installed or different baseline

fixtures, and also to the evaluation including calculations to account for HVAC

interactive effects. This adjustment is calculated for installed measures only.

3. Operating-hours adjustment – This adjustment factor accounts for the difference

between the operating hours used to determine tracking savings and the metered hours

observed from the on-site analysis. The operating hours adjustment is the ratio of verified

to calculated weighted average annual hours of use for installed measures only.

Figure 3-1 shows each of the savings estimates included in the evaluation adjustment ratios,

calculated from the tracking data and the on-site metering or CATI survey data collection.

Green shading indicates that the input used in the calculation is evaluation verified rather than

based on the program tracking data.


26 CECONY

Figure 3-1 Savings Estimates Employed in Evaluation Ratios

3.5 Attribution

One goal of the evaluation of CECONY’s Small Business Direct Install Program9 is to

measure program net impacts – energy savings associated with equipment installations that

would not have been achieved in the absence of the program. Program net impacts are

expressed as a Net-to-Gross ratio (NTGR) that uses free ridership (FR) and participant

spillover (PSO) rates as its components. FR represents the percentage of savings that would

have been achieved in the absence of the program. PSO represents additional savings that

were achieved by program participants installing additional similar measures without

9 Referred to as Small Business Energy Efficiency Program in the survey instrument

NYTM Formula

Tracking Formula

NYTM Formula

Tracking Formula

Tracking QuantityTracking Savings Tracking Technology Tracking Operating Hours

Installed Savings Observed Quantity Tracking Technology Tracking Operating Hours

Calculated Savings Observed Technology Tracking Operating Hours

Verified SavingsObserved (Logged)

Operating Hours

Observed Quantity

Observed Quantity

NYTM Assumptions

Tracking Assumptions

NYTM Assumptions

Tracking Assumptions

=

=

=

=

Observed Technology

Input based on Tracking Data

Input verifed in Installed Savings

Input verified in Calculated Savings

Input verified in Verified Savings

Legend


CECONY 27

receiving program rebates that would not have happened in the absence of their

participation in the program.

The formula used to calculate the NTGR can be expressed as:

As part of this effort, the evaluation team derived both FR and PSO estimates using self-

reported information from telephone interviews with program participants. Using the survey

instrument developed for this evaluation, we interviewed program participants and asked

them a series of structured and open-ended questions about the influence of the program

and its various components on the decision to install energy efficient equipment.

Non-participant spillover (NPSO) was not assessed during this evaluation. From a program

theory perspective, we do not expect much NPSO associated with this program. For the SBDI

program to achieve NPSO, non-participants would need to make similar energy efficient

upgrades to their business because of the program, yet not receive an audit, free measures, or a

rebate. It is unlikely that small business owners would be motivated to make changes without

a rebate because they knew of the existence of the program and were inspired to make the

changes on their own.

In addition, the participating contractors who do similar work for non-participants have every

incentive to encourage them to participate in the program which means that it is unlikely that

participating contractors will contribute to NPSO.

However, an NPSO study may be warranted in the future if research shows that the market

has changed due to the program. At this time, because NPSO studies are typically costly and

difficult to conduct, the small expected NPSO savings did not justify the cost. NPSO will be

evaluated in a separate joint statewide research study scheduled to commence at a later date.

Below is a general overview of the method for developing FR and PSO results. Appendix F of

this report contains further detail on the NTG estimation method.

3.5.1 Free Ridership

Free riders are program participants who would have implemented the incented energy

efficient measure(s) even without the program. FR represents the percent of savings that would

have been achieved in the absence of the program. Telephone interviews with participants

were used to develop the basis of the FR score which is the percentage of savings not

attributable to the program. FR rates were estimated separately by measure.


28 CECONY

Using the survey instrument developed for this evaluation, program participants were

interviewed and asked a series of structured and open-ended questions about the influence of

the program and its various components on the decision to have high efficiency equipment

installed at their business. More specifically, program participants were asked about any pre-

existing plans to implement the program measure(s), willingness to hire someone to install the

measure(s) even if there were no program incentives, and likelihood of taking the same action

absent the program.

The goal of most incentive-based energy efficiency programs is to influence customer decision-

making regarding energy efficient improvements. Programs can do this by changing what

customers install, when they install it, and how much they install. In other words, programs

influence the efficiency, timing, and quantity of customers’ energy-using equipment installations.

The bulk of program savings is typically achieved by encouraging customers to install higher-

efficiency equipment than they would have installed on their own. Programs may also

encourage early replacement of still-functioning equipment that is less efficient, thus impacting

the timing of the installation, so that savings can be realized earlier. The incentive may also

make it more affordable for customers to install a greater number of high efficiency measures.

As such, the FR algorithm outlined here combines estimates of each of these concepts:

Program influence on the efficiency level of the installed equipment (EI)

Program influence on the timing of the installation of high efficiency equipment (TI)

Program influence on the quantity of the high efficiency equipment installed (QI)

Each concept takes a value between 0 and 1. The values are expressed in FR terms, with 0

meaning no FR and 1 meaning full FR.

Efficiency (EI), timing (TI), and quantity (QI) of the installation are distinct avenues of program

influence. However, the timing of the installation and quantity of measures installed are

conditional on efficiency. The program can only realize timing savings if the customer would

have installed the efficient equipment on their own but the program caused the installation to

happen earlier. Similarly, savings due to a quantity increase can only happen if a customer

who was already installing some energy efficient measures chooses to install additional

measures because of the program.

We believe that when the three concepts are measured as distinct yet conditional methods of

program influence, it is appropriate to combine them by using multiplication. Averaging or

using some other calculation method would overestimate FR. As such, the formula to calculate

FR through the program influences using the multiplicative approach can be expressed as:


CECONY 29

IF EI 0.50, THEN FR1=EI ×TI ×QI

ALL OTHER CASES, FR1=EI

During the New York Department of Public Service (DPS) review process, the DPS expressed

concern that the multiplicative method could produce an FR estimate that was biased

downward. After extensive discussions with the DPS, CECONY, and other stakeholders, the

evaluation team agreed to multiply the EI and QI scores, and then average this product with

the TI score. This averaging would only apply to cases when the TI score is lower than the

product of the EI and QI scores. In cases where the timing score exceeds the product of EI and

QI, the FR rate is based on just the product of the EI and QI scores. This selective averaging is

important so that the program is not penalized for having a smaller influence on the timing of

the project than the efficiency and scope of the project. The formula to calculate FR using the

selective averaging approach is expressed as:

IF EI 0.50 AND TI> (EI*QI), THEN FR2=EI*QI

IF EI 0.50 AND TI (EI*QI), THEN FR2=AVERAGE((EI*QI);TI)


As part of the analysis, the evaluation team developed two estimates of FR – one that follows

the multiplicative approach and another that follows the selective averaging approach.

Developing two FR estimates using different algorithms allowed us to compare the results and

better understand the sensitivities associated with each method.

We estimated FR at the measure level, which means that, based on the project scope,

respondents were asked about the decision-making process for more than one measure if

applicable. Some participants installed a variety of measures, and estimating FR for all of them

was not feasible given the amount of time that would be required during the interview. In

these cases, we asked about three measures, selected based on their rarity. That is, least

common measures were given priority in the selection process. This was done to ensure that

we get sufficient coverage of those measures. As part of the FR estimation, we attempted to

capture as much of the program savings as possible. To meet this goal, we identified chain or

multi-facility participants and asked them if the decision to install each measure was a single

decision across all of the facilities, or if each facility underwent its own decision-making

process. In cases where the decision-making process is the same across multiple facilities, we

applied the FR estimate to the savings resulting from measure installation at all facilities.

Because FR was estimated at the measure level, certain measures, more specifically lower

cost lighting measures, call for a different line of questioning. As such, we present the FR


30 CECONY

approach for them separately from low-cost non-lighting and for higher-cost lighting

measures.

Because respondents can sometimes give inconsistent answers, the survey instrument included

consistency checks to clarify these responses. As part of the data analysis, we carefully

reviewed those responses and adjusted either FR scores or individual component scores

accordingly.

Respondents can sometimes provide “don’t know” responses to one or more questions that are

critical to FR estimation. Sometimes, respondents can simply refuse to answer some questions.

This leads to item non-response. To overcome any biases associated with item non-response,

we imputed data based on the responses that other similar participants gave.

3.5.2 Spillover

PSO represents additional savings (expressed as a percent of total program savings) that

were achieved without the customer receiving program rebates but would not have

happened in the absence of the program. PSO was assessed through interviews with

participating customers by asking about non-program efficiency actions that participants

took as a result of participating in the program. The actions could have taken place at the

same facility that received the program-funded upgrades or at another site. Spillover

questions covered both program-like measures that were installed without program rebates

as well as energy efficiency measures not offered as part of the program design. The survey

instrument contained checks to ensure consistency of response.

While PSO can result from a variety of measures, survey length did not allow for estimation of

PSO across all possible measures or scenarios. Given the types of businesses that participated

in the program, the evaluation team included measures that experiences with the program

could reasonably influence. As such, PSO was measured for the following equipment

categories:

Lighting equipment

Cooling equipment

Refrigeration equipment

Kitchen equipment

Motors

Heating and water-heating equipment


CECONY 31

Participants were asked if they made any of the above-listed improvements but did not receive

incentives for them through the program. Those who did were asked if the Con Edison

program was of any influence and, if so, the degree of influence. Respondents were also asked

to explain in their own words exactly how the program influenced their decision to make

specific additional improvements.

Respondents were also asked a few equipment-specific questions that allowed the calculation

of savings associated with the installed equipment. The equipment details were limited by the

survey length as well as by what we believe respondents could reliably answer.

As part of the SO calculation, the evaluation team applied savings values to the measures

installed outside of the program. We estimated savings for each measure using the most recent

NYTM values supplemented by engineering assumptions. We determined the program-level

SO factor by dividing the estimated savings of the measures installed by survey respondents

outside of the program (but influenced by the program) by the savings the survey respondents

realized through the program.

NPSO was not in the scope of this evaluation.

3.6 Potential Gross Impact Error and Bias

In addition to sampling uncertainty, other sources of error discussed below were considered

over the course of this impact study, and precautions were taken to mitigate or address them.

3.6.1 Sample Bias

The sample was drawn with representativeness targets as described above in Section 3.2.

Quotas for representativeness reduce the likelihood that a random sample will misrepresent

the population.

3.6.2 Non-Response Bias

Non-response bias is always an issue when conducting surveys of voluntary participants. The

evaluation team employed industry standard techniques for mitigating the impact of non-

response. These include stratifying the sample, making phone survey calls at varying times of

day and evening, calling sampled participants at least six times before removing them from

consideration, translating surveys in several languages, offering cash incentives for

participation in on-site data collection, and offering alternative time slots for on-site data

collection, such as evenings and weekends.


32 CECONY

3.6.3 Measurement Error

Measurement uncertainty can be due to meter accuracy, extrapolation, simplifications in

HVAC interactive effects methods, and within-site sampling. To mitigate these uncertainties,

the first step taken to address measurement error was the development of standardized field

forms, detailed written field protocols, and field training materials. Rigorous field training was

conducted in order to ensure that field staff are collecting data and recording it in an accurate

and consistent manner. In addition, a rapid quality-control check of data collected was

conducted to identify issues to follow up on during the subsequent site visits. It should be

noted that none of these measurement uncertainties are believed to have inherent bias that will

affect the RR estimate in a particular direction.

3.7 Billing Analysis

The ERS team performed a program-level pre-/post-retrofit billing analysis of late 2010 and

early 2011 participants to compare with the sample-based metering and NTG study results.

The billing analysis used multiple techniques, including pooled fixed effects and several

variants of the PriSM-type methods. The methodology is described in detail in the CECONY

EEPS Programs SBDI Impact Evaluation Preliminary Billing Analysis .


CECONY 33

4 RESULTS

The results presented in this section are shown by building type, which was the primary

characteristic for the on-site data collection sample. Additional breakdowns of the results,

including results by measure type, geographic area, measure cost, and whether the customer

was part of a chain or not, are provided in Appendix D.

4.1 Final Sample Disposition

The sample designs discussed in Section 3.2 represent the optimal distribution of the data

collection targets. However, the actual data collected is limited by the willingness of the

respondents to complete the survey or participate in the on-site data collection. Respondents

may refuse to participate in the evaluation, which may result in strata that do not meet their

completion targets. In those cases, the evaluation moved targets from one stratum to another to

achieve the overall number of target completes. An overview of the sample disposition for the

CECONY evaluation sample is provided in this section.

4.1.1 Realization Rate Sample Disposition

The on-site sample targeted a total of 133 CECONY sites. The evaluation exceeded the target

with 137 sites visited. Additional site visits were conducted to allow for attrition during the

data collection period, or participants who agreed to begin the process but were unable to

finish. During recruiting, back-up sites were used to replace the original sample sites that

could not be recruited. A total of 64 sites (48%) were used from the back-up sample to achieve

the target count. Overall, the response rate including backups was 68%. Table 4-1 provides the

distribution of the final sample.

Table 4-1. Final RR Sample Disposition

The evaluation met or exceeded targets for nearly all of the strata in the on-site sample, which

limits the potential for bias along the stratification dimensions. We attempted to control for

non-respondent bias by recruiting the on-sites using two methods: pre-scheduling via

telephone or unscheduled drop-in visits when the telephone contact went unanswered. Some

of the stratification variables used in the telephone survey sample were not included in the on-

Population Sample

Retail 2,171 43 43 32% 1%

Offices 1,104 41 41 9% 1%

Parking Garages 93 27 30 10% 4%

Other 3,567 22 23 50% 1%

Total 6,935 133 137 100% 7%

Fraction of Population

Total Savings

Building Type

2011

Population

Accounts

Planned

Sample

Size

Actual

Sample

Completes


34 CECONY

site sample (borough, measure mix, chain accounts); therefore, we cannot guarantee a

representative mix of those dimensions in our on-site sample.

4.1.2 Attribution Sample Disposition

The CATI sample targeted 310 measures out of the 12,264 measures installed through the

program in 2011. The evaluation completed surveys of 597 measures, almost double our target.

Many contacted participants installed more than one measure, and the evaluation asked about

up to three measures per site on each call. To limit potential bias from “over-collecting” data

from multiple measure sites, the evaluation unit-weighted secondary measures collected from

sites with more than one measure if the measure was itself not randomly selected in the

measure-level sample design. A simplified look at the sample dispositions from the CATI

survey (additional stratification by business type, cost type, and measure size are not shown) is

displayed in Table 4-2.

Table 4-2. Simplified CATI Sample Dispositions

Population Sample

Bronx CFL 413 5 18 1% 0%

Bronx Non-Lighting 77 3 3 1% 0%

Bronx Other Lighting 249 6 9 2% 0%

Bronx Tube Lighting 663 22 41 9% 1%

Brooklyn CFL 1,881 12 54 4% 0%

Brooklyn Non-Lighting 240 7 7 1% 0%

Brooklyn Other Lighting 574 13 39 4% 0%

Brooklyn Tube Lighting 1,616 57 70 21% 1%

Manhattan CFL 319 6 16 1% 0%

Manhattan Non-Lighting 483 5 7 1% 0%

Manhattan Other Lighting 523 23 44 7% 1%

Manhattan Tube Lighting 411 32 32 12% 1%

Queens CFL 901 9 37 2% 0%

Queens Non-Lighting 167 4 4 1% 0%

Queens Other Lighting 636 14 47 4% 0%

Queens Tube Lighting 1,352 47 64 18% 2%

Staten Island CFL 86 3 2 0% 0%

Staten Island Non-Lighting 7 1 0 0% 0%

Staten Island Other Lighting 87 2 3 0% 0%

Staten Island Tube Lighting 151 5 3 1% 0%

Westchester CFL 420 5 18 1% 0%

Westchester Non-Lighting 164 2 9 0% 0%

Westchester Other Lighting 376 6 37 1% 0%

Westchester Tube Lighting 468 21 33 7% 1%

12,264 310 597 100% 8%

Fraction of Population

Total Savings

Borough

Sampling

Measure Type

2011

Population

Measures

Planned

Sample

Size

Actual

Sample

Completes

Total


CECONY 35

One source of potential bias that the evaluation identified was that 443 (9%) of non-

respondents had disconnected phone numbers. If these businesses closed the location where

the program installed measures, then our results may overestimate the installation rate. We do

not think this is the case because in conducting the on-site surveys we found only two sites

where the participating business had closed. In one case the site remained empty, but at the

other the measures installed through the program were still in use, albeit by a different

business. Participants responsible for completing improvements across multiple facilities and

multiple measure categories were interviewed in a more in-depth fashion than participants

with a simpler set of completed improvements. In-depth interviews allowed for added

flexibility and a less burdensome survey experience. The in-depth interviews that were

completed closely followed the approved survey instrument. Overall, we completed in-depth

interviews with eight unique participants across 103 facilities. While there could potentially be

some biases stemming from the survey mode, the directionality of the bias and its magnitude

are unknown.

Upon completion of the fielding efforts, the evaluation team assembled the fielding report

and calculated final response and cooperation rates. These results are presented in Appendix

H of this report.

4.2 Gross Savings Realization Rate

The gross savings RR is the evaluation’s estimate of gross savings divided by the program’s

tracking savings. An RR in excess of 1.0 indicates that participating customers are saving more

than Con Edison claimed; an RR of less than 1.0 indicates they are saving less than reported.

This section presents the gross RRs by site and for the overall program.

4.2.1 Engineering Based Site-Specific Savings

Figure 4-1 compares the evaluated annual electric energy savings with those reported by the

program. Ideally, the evaluated savings would always match the program tracking savings.

This ideal is shown as a solid black line on the charts. Actual findings are plotted as points

on the scatter graph, with program-reported savings on the x-axis and evaluated gross

savings on the y-axis. The dotted line represents the evaluated realization rate. If all the

points were to fall directly on the solid line, it would mean that the evaluated savings were

exactly the same as the program-reported savings and the RR is 1.0. A pattern of points

below the ideal line suggests an RR of less than 1.0; more points above the line suggest an RR

greater than 1.0. Each point on the plot represents a site that was sampled and evaluated.

As seen from Figure 4-1, a majority of the projects resulted in savings of less than 50,000 kWh

and are not clearly visible on the plot. These sites are better presented in Figure 4-2.


36 CECONY

Figure 4-1 Program-Reported and Evaluated Electric Energy Savings

Figure 4-1 also presents sites that have high enough energy savings (greater than 150,000 kWh)

to indicate that they may not be a small business with demand less than or equal to 100 kW.10

10A 100 kW business with 8,760 hr/yr operation would have to reduce its entire facility load by more than one-third

to save the 300,000 kWh/yr.


CECONY 37

Figure 4-2 Program-Reported and Evaluated Electric Energy Savings (Magnified View)

It should be noted that there were seven sites that had zero savings. Out of these, one site

changed business and the program-installed lighting was changed to different fixtures by the

new tenant. All the remaining six sites were small sites that did not implement the measures.

Two projects are highlighted in Figure 4-1 for which the evaluators measured exceptionally

high savings (one site) or low savings (one site) compared to tracking savings. The reasons for

such high deviations are:

High RR:

The fixtures were never switched off, resulting in double the operating hours.

Low RR:

The evaluators identified the installed fixtures to have 32 W fluorescent lamps instead

of the expected 28 W fluorescent lamps, which reduced the savings.


38 CECONY

The tracking savings estimates were relatively close to the evaluated savings for the larger

projects. Figure 4-1, above, looks like data from a program with accurate tracking estimates

and a low evaluated error ratio. However, the scatter plot close-up in Figure 4-2 reveals a much

wider distribution when focusing on the large number of smaller projects. The final error ratio

based on the RR of this study was 0.9, indicating a high variation between tracking and

evaluation savings estimates and, in particular, more variation than anticipated. The sample

design assumed a 0.4 error ratio for the RR in order to achieve 90/8 confidence/relative

precision for the program and 90/10 for most building types.

4.2.2 Site-Specific Reasons for Deviations from Tracking

Each site was reviewed to identify the factors that created discrepancies between evaluated

and tracked savings and then to quantify and categorize them. While quantitative, this is not a

statistically rigorous estimate of the factor on total program impact. The intent of this analysis

is to provide the PAs with indicators of where they may want to focus future process

improvements.

Figure 4-3 Weighted Contribution of Discrepancy

The graph shows both positive and negative impact of each discrepancy category.

Each discrepancy was estimated independently as the difference in kWh between the tracking

savings and what the savings would have been using the correct value. Residual error was

categorized as “Indiscernible”. The site’s independent discrepancy values were reconciled to

the site’s evaluated discrepancy using the ratio of the sum of the individual results divided by

the site’s total evaluated discrepancy. An alternate method of calculating discrepancies is to

cascade the calculations so that each subsequent calculation depends upon the results of the

previous calculations. However, this method’s results are highly dependent upon the ordering

of the analysis, increasing the apparent impact of the first category, decreasing the subsequent

category impacts. The cascading method makes it harder to discern the value of a proposed

process improvement action.

Category Description

Negative

Impact

on RR

Positive

Impact

on RR

-3%

1%

-6%

0%

-23%

11%

-7%

4%

HVAC interactive:

121 sites

This category captures the savings difference between the tracking HVAC interactive factor

estimates and evaluated interactive factors obtained through discussion with the site staff.

Quantity:

73 sites

This category captures the savings difference between tracking and evaluation verified

fixture counts.

Technology:

78 sites

This category captures the savings between tracking and evaluation verified fixture type

and wattage.

Operating hours:

126 sites

This category captures the difference in savings due to tracking estimated and evaluation

measured operating hours.


CECONY 39

Site weights were applied to individual site discrepancy results to provide an estimate of the

value of the discrepancy in the population.

Note that the outcome of the discrepancy characterization itself has no impact on realization

rates but only explains the sources of the discrepancies.

Based on these findings, the evaluators have recommended changes to the TRM operating

hours for the retail, parking and office sectors.

4.2.3 Operating Hours Discrepancy

The majority of the sites had operating hour discrepancies. This is to be expected, as the

program used deemed hours as a function of building type, whereas the evaluation takes the

actual measured and space type-specific operating hours into consideration. The solid line

represents the ideal case where as the dotted line represents findings from the measured data.

Figure 4-4 compares the evaluated operating hours with that reported by the program.

Figure 4-4 Program-Reported and Evaluated Operating Hours


40 CECONY

As can be seen from the plot above, the majority of sites had lower evaluated operating hours

than tracking hours. The evaluated operating hours are the weighted average of the entire

facility and use the operating hours by space type within the facility and weigh them by the

connected fixture load. The tracking hours are based on the building types and applied to all

fixtures in the building, even if the space does not operate that many hours.11

4.2.4 Program Level Gross Realization Rate and Components

The overall CECONY SBDI gross savings RR is 76%. The evaluators estimate that customers

are realizing 76% of the program tracking savings.

The gross savings RR difference is the RR minus 100%. For this program the difference is -24%.

The majority of the deviation from a 100% RR is due to operating hour differences, which

reduced savings by 17%12. The calculation adjustment further reduced savings by 2%. The

calculation adjustment includes some effects which could be positive or negative deviations

(wattage and technology corrections), but also included the addition of HVAC interactive

effects. Without the evaluation-added HVAC interactive effects, the calculation adjustment

sub-ratio would have been a larger negative deviation. Variances in the installation rates

further reduced savings by 6%. Problems in the tracking data prevented the evaluation team

from consistently identifying which building spaces received program-supported retrofits and

which did not. A total of twenty-two such sites were identified. The evaluators re-verified the

fixture counts for all sites during the final site visit. For sites that had higher counts than the

tracking savings, the evaluators reviewed documentation for all the projects done at the site in

order to be certain that the additional fixtures were not implemented through a separate

project and interviewed the site staff to verify that the additional fixtures were installed at the

same time. The evaluators reviewed the project documentation for sites that had higher fixture

counts

Table 4-3 summarizes the differences. The tables displayed in this section include the results of

the evaluation with the accompanying measure of statistical precision. Unlike the data used to

create Figure 4-3 above, these results are weighted by energy savings and sample expansion

weights. The statistics shown include:

11 The program administrators originally allowed SBDI implementation contractors to estimate site-specific hours

but found that the net effect was a significant overestimation of hours. They therefore changed the policy and

instructed contractors to use the deemed approach by building type that is currently in place. This change occurred

prior to the beginning of the evaluation period. While deeming hours increased the evaluation’s variability of

results, it has had the more important desired effect of reducing the hours estimates overall to values closer to

reality. 12 This discrepancy represents the difference in savings between the tracking and metered operating hours value.

The tracking operating hours include a combination of NYTM and contractor reported hours by building type.


CECONY 41

Min n – The number of sampled measures from which the results are based.

Impact of deviation – The evaluation’s measured result expressed as the deviation from

100%. Can be calculated as 100% minus the ratio of evaluated result to program result.

Relative precision at the 90% confidence interval.

Table 4-3 Gross Savings RR Deviation and Deviation Components

In addition to the overall results reported in this section, the evaluation team produced

breakouts by building type, measure type, cost type, chain, and borough. Appendix D shows

breakouts by measure group, cost type, business type, and borough. The breakouts show:

Building type – The overall RR deviation was relatively consistent across building types,

with deviations ranging from -17% to -28%. None of the differences were statistically

significant at the 90% confidence interval.

Measure group – The overall RR deviation was close to twice as large for CFLs (-39%) as

it was for the other measure groups. This is primarily due to a CFL deviation in

installation rate that is more than four times as large (-22%) as other measure groups and

a CFL deviation in operating hours adjustment that is almost twice as large (-25%) as

other measure groups. CFLs are easier to remove than other lighting types in the

program and thus can be expected to have lower installation rates generally. CFLs may

also be installed in low-usage areas (such as closets and storage spaces) more often than

other types of light fixtures. Despite the large deviations, the CFL results did not have an

overwhelming impact on the overall program RR because they make up only 10% of

program savings.

Factor

Min

n*

Impact of

Deviation

Relative

Precision at

90%

Confidence Description

Installation rate 882 -6% 2% Number of units installed vs. tracked

Calculation adjustment 231 -2% 3% Wattage, technology and HVAC differences

Operating hours adjustment 231 -17% 8% Operating Hours differences

Gross realization rate difference 231 -24% 9%

Gross realization rate 76% 10%

* The minimum number of sampled measures from which the results are based


42 CECONY

Table 4-4. Gross Savings RR Difference for CFLs

Cost type – The overall RR deviation was more than twice as large for measures where

some of the units were provided free of charge (-39%) as it was for all low-cost measures.

This is primarily due to a deviation in installation rate (-20%) for “some free” measures

that was four times as large as all low-cost measures, and a deviation in the operating

hours adjustment (-26%) for “some free” measures that was close to twice as large as all

low-cost measures. These results mimic what was seen for CFLs, which is not a

coincidence. 88% of the savings of the “some free” measures are from CFLs. Of CFL

savings, 95% appear at sites with “some free” measures. The heavy overlap between CFL

savings and “some free” measures suggests that CFLs were effective in getting audits

done, but did not necessarily encourage audit participants to move forward with

additional installations of the same measure type within the evaluation period. “Some

free” measures make up 11% of program savings.

Business type – The overall RR deviation was marginally larger for chain businesses (-

28%) than for non-chain businesses (-24%); however, the difference was not statistically

significant at the 90% confidence interval. This is primarily due to a chain operating

hours adjustment that was larger (-22%) than non-chain businesses (-17%). Chains made

up a smaller portion of the program savings (14%) than non-chain businesses (86%).

Borough – The overall RR deviation ranged from -11% to -38%, but none of the

differences were statistically significant at the 90% confidence interval.

The evaluation team calculated the average magnitude of the operating hours metered for

three different building types and an “Other” building type category and compared them to

the assumptions in the NYTM. The results are shown in Table 4-5.

The metering result for the “Retail” building type shows the least amount of deviation from

the NYTM estimate with a decrease of 599 hours (15%). Offices were next with a decrease of

735 hours (20%). Parking garages showed the largest deviation from the TRM estimate with an

increase of 3,349 hours (77%).

Factor

Min

n*

Impact of

Deviation

Relative

Precision at

90%

Confidence Description

Installation rate 217 -22% 13% Number of units installed vs. tracked

Calculation adjustment 48 5% 6% Wattage, technology and HVAC differences

Operating hours adjustment 48 -25% 12% Operating Hours differences

Gross realization rate difference 48 -39% 19%

* The minimum number of sampled measures from which the results are based


CECONY 43

Table 4-5. Operating Hours Magnitude by Building Type

4.3 Attribution

The Net-to-Gross ratio (NTGR) is based on responses to the CATI survey of program

participants. It is made up of two components:

1. Free ridership (FR) – Has a value between 0% and 100%.

2. Participant Spillover (PSO) – Can have any value (including negative) and is an estimate

of savings from program participant projects that were influenced by the program but

installed without a rebate. Only kWh PSO savings were included in the final NTGR.


FR was determined using the CATI survey, which was stratified by measure characteristics

(measure type, cost type) and business type (chain or not) because these characteristics were

expected to have the greatest effect on FR. We include detailed looks at FR by these

characteristics in Appendix D.

The evaluation estimated the FR two ways, once using an averaging approach and once using

the method applied in Massachusetts and evaluations in other jurisdictions, known as the

“multiplicative method” and referred to here as the “alternative method.”

FR for the CECONY program was 17% with a relative precision of 3%. This rate is higher than

expected for a program of this type. Table 4-6 shows the FR by building type.

Building Type

Min

n*

NYTM

Operating

Hours

Study

Measured

Operating

Hours

Relative

Precision at 90%

Confidence

Percent

Difference in

Measured Hours

vs. NYTM

Retail 63 4,057 3,458 13% -15%

Offices 73 3,748 3,013 10% -20%

Parking Garages 57 4,368 7,717 8% 77%

Entire program** 231 N/A 3,468 16% 100%




44 CECONY

Table 4-6. FR by Building Type

Appendix D shows breakouts by measure group, cost type, business type, and borough. The

breakouts show:

Building type – Parking garages had the highest FR at 31%, which was nearly twice as

high as the other building types. Parking garages differ from other building types in the

program in two ways: almost all of their savings came from tube lighting (95%) and

almost half of the parking garages were identified as being part of a chain (46%). Parking

garages typically operate their lights more frequently than other building types, resulting

in a faster payback on lighting projects. In addition, lighting as an end use may make up

a greater percentage of total energy use for parking garages than for other buildings. Both

characteristics would tend to make high efficiency lighting projects financially attractive

for parking garages and reduce the program’s opportunity to have an influence on the

decision to install. Parking garages account for 9% of the program savings.

Measure group – CFLs had higher FR than other measure types (30%), nearly twice as

high as the other measure groups. 95% of CFL savings come from sites with some free

measures. CFLs make up less than 10% of program savings.

Cost type – Customers who received some free measures had FR that was almost twice as

high (30%) as those customers whose measures were all low-cost. Customers who

received free measures accounted for 11% of the program savings.

Business type – Chains had higher FR than non-chains (27% vs. 16%), but the difference

was not statistically significant. Chains represent 14% of program savings.

Borough – FR was not affected by which borough the measures were installed in.

4.3.2 Alternative Free Ridership

Alternative FR was calculated based on the same survey data as the FR. In simple terms, the

difference between the two methods is that the timing component of FR is multiplied by the

Building Type

Min

n*

Direct

Free

Ridership

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Retail 136 14% 4% 30%

Offices 75 15% 8% 8%

Parking Garages 21 31% 15% 9%

Other 311 17% 5% 53%

Entire program** 543 17% 3% 100%




CECONY 45

product of the FR from efficiency and quantity rather than averaged with the product of

efficiency and quantity FR.

Alternative FR for the CECONY program was 15% compared to 17% using the NY approach.

The difference between the two methodologies was statistically significant at the 90%

confidence interval; however, at only a 2% difference it is perhaps not meaningful for this

study. The difference in approach would likely show greater differences for a program that has

higher FR. Table 4-7 shows the alternative FR by building type.

Table 4-7. Alternative FR by Building Type

4.3.3 Participant Spillover

This section reports the kWh PSO results as a ratio of attributable PSO savings to verified gross

savings (parallel in form to the complement of the FR ratios). PSO savings for savings types

other than kWh are reported in Section 3.5.

PSO savings were reported by few respondents. Of those with any reported SO, only a

portion had kWh savings associated with the SO project. For CECONY, the overall kWh SO

rate was 0.19%. PSO savings were found for projects installed by both Offices and Other

building type categories. Table 4-8 shows the PSO by building type.

Table 4-8. PSO by Building Type

Building Type

Min

n*

Direct

Free

Ridership

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Retail 136 10% 3% 30%

Offices 75 14% 6% 8%

Parking Garages 21 31% 10% 9%

Other 311 15% 4% 53%

Entire program** 543 15% 3% 100%



Building Type

Min

n*

kWh/kWh

Spillover

Ratio

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Retail 136 0.00% 75% 30%

Offices 75 0.21% 161% 8%

Parking Garages 21 0.00% 97% 9%

Other 311 0.29% 101% 53%

Entire program** 543 0.19% 93% 100%




46 CECONY

4.3.4 Combined Net-to-Gross Ratio

FR, PSO savings, and verified gross installed (VGI) savings were used to calculate the NTGR as

shown in the equation below.

( )

The combined NTGR for CECONY is 83%. Table 4-9 shows the NTGR by building type.

Table 4-9. NTGR by Building Type

4.4 Net Realization Rate

The net RR is the product of the gross savings RR and the NTGR. It provides the ratio of net

savings to program tracking savings. In other words, this ratio represents the percent of

savings tracked by the program that are both verified and attributable to program activities.

The net RR for CECONY is 63%.

The evaluation-verified gross impacts are the product of the gross savings RR and the program

tracked kWh savings. The net program impacts are the product of the net RR and the program

tracked kWh savings. For CECONY, the tracked annual kWh savings for all measures for the

2011 program period are 108,358,208 kWh. Based on the evaluation findings regarding RRs and

attribution, the evaluation-verified gross impacts are 81,958,008 kWh, while the net impacts are

67,974,229 kWh. Table 4-10 shows the gross and net impacts by building type.


Combined Net-

to-Gross

Ratio

Relative

Precision at 90%

Confidence

Retail 136 86% 4%

Offices 75 85% 8%


Other 311 83% 5%

Overall NTGR 543 83% 3%



CECONY 47

Table 4-10. Gross and Net Impacts by Building Type

4.5 Comparison with Billing Analysis

The program-level billing analysis found RRs between 18% and 21% for CECONY and

between 52% and 54% for O&R. At 90% confidence the relative precision ranged from 13% to

32%, depending on the method and whether or not CFL-only sites were included in the

analysis. The results are presented in detail in the CECONY EEPS Programs SBDI Impact

Evaluation Preliminary Billing Analysis and the O&R EEPS Programs SBDI Impact Evaluation

Preliminary Billing Analysis.

The dramatic difference between the billing analysis and the site M&V-based analysis results

requires addressing. In the Work Plan the evaluation team proposed billing analysis as a “test”

method to perform before M&V, with the attractiveness of early feedback and low cost

outweighing the recognized vulnerabilities associated with low savings fractions and the risk

that missing account data might present a fatal weakness in the analysis since comprehensive

consumption data is a threshold required for such an approach.

The gross discrepancy in results between the two methods and the probable reasons for them

led the evaluation team to choose to use the M&V-based results alone over the test billing

analysis alone for final reporting. The reasons for this are:

1. There is evidence of a strong pattern of missing account data from the billing analysis.

CECONY has complex space and metering arrangements in all boroughs but, especially

in Manhattan. These complex arrangements lead to inadvertently omitted account data.

Further, there is the possibility that the implementation contractors withheld account

information in order to maintain eligibility for the SBDI program. In either case missing

account data biases billing analysis savings downwards. This bias vulnerability was a

mild concern at the outset of evaluation and noted in the plan. The billing analysis and

M&V populations overlap but are not the same. Specifically, the billing analysis group

Building Type

Tracked

kWh

Savings

Gross

Realization

Rate

Measured

Gross

Program kWh

Savings

Net

Realization

Rate

Net

Program

kWh

Savings

Net Relative

Precision at

90%

Confidence

Retail 32,263,435 80% 25,810,748 69% 22,261,770 15%

Offices 8,846,429 75% 6,634,822 64% 5,661,715 14%

Parking Garages 9,942,089 83% 8,251,934 57% 5,666,991 17%

Other 57,306,255 72% 41,260,504 60% 34,383,753 17%

Entire program* 108,358,208 76% 81,958,008 63% 67,974,229 9%

* Overall program level numbers are savings w eighted totals/averages


48 CECONY

is from an earlier time period. The early period was necessary to collect a year’s worth

of post-retrofit billing data. CECONY believes that program tracking savings were

relatively higher and thus realization rates lower in early-year operations than later-

year operations. Directives with restraints on limits for excessive installer estimated

hours were not in place during the early months of the program. When this was

realized restrictions were put in place in October 2010. The M&V sample is of course

more representative of its own population than the billing analysis population. Based

on program design and operation changes described in the process evaluation and

observed in this impact evaluation, the evaluators are confident that it also is

representative of more recent and future program performance.

2. The M&V-based approach has relatively few uncertainties. Time of use (TOU) metering

was extensive, the analysis includes the relatively minor interactive effects with HVAC,

and post-installation equipment wattage was verified. Pre-installation fixture types

were validated by the program. Surveyors could not meter pre-retrofit TOU but did ask

site staff about them and found no patterned difference. Simply put, there is not

enough uncertainty or potential bias in the M&V approach to justify making a major

downward change in the realization rate by incorporating the billing analysis into the

final evaluated gross savings results.

3. In addition, the population is not a very stable population and could contribute to

billing data errors. For example, the businesses close, merge, expand, sell out, change

owners, or change methods of operation all of which will result in a change in the

facility energy use patterns.

4. This leaves on-site behavior increasing energy use beyond the retrofitted fixtures and

because of the program as being the only explainable reason for using the billing

results. The evaluation team interviewed about this possibility during site visits.

Respondents gave no evidence of a pattern of snapback. It is possible that unknown

non-program business activity changes (e.g. decreased vacancy rates as the economy

recovered during the billing analysis period) could have biased savings downwards,

but one of the billing analysis methods does use participants during nonparticipating

months as an effective control group and this phenomenon was not present.

In summary, the initial pilot nature of the billing analysis, the confidence in the M&V results,

the known biases in the billing analysis, and the inability to otherwise explain the differences

led the team to choose to use the primary method of M&V alone for reporting.

In addition, the billing analysis results for a similar program in New York were informally

discussed by the program administrators. The billing analysis and discrepancies between


CECONY 49

billing analysis and onsite M&V results for both the programs were found to be in similar

ranges.

4.6 Implications for the New York Technical Manual

Table 4-11 translates the results from Table 4-5 above into the NYTM categories. The

evaluation team recommends that the NYTM either update the operating hours for selected

building types based on this study’s research, or create new “small business” or “downstate”

retail, office, and parking garage categories specifically for SBDI programs that reflect this

study’s findings. The table presents both the current NYTM-deemed hours and the

recommended updated values based on this evaluation’s research.

Table 4-11. Recommended Updated NYTM Hours by Business Type

The hours of operation were determined from sites in CECONY territory and weighted both

by case weights (sample weights) and by the average kW savings of the logged lights, making

the hours of use found in this study directly applicable for use in calculating savings from

lights that are likely to be replaced in energy efficiency programs.

Table 4-12 presents the hours of operation and relative precision for the building types.

Description

NYTM Current

Hours

Study-Measured

Hours - CECONY

Banks 3,748 3,013

Court house 3,748 3,013

Library 3,748 3,013

Medical offices 3,748 3,013

Museum 3,748 3,013

Office/retail 3,748 3,013

Post office 3,748 3,013

Town hall 3,748 3,013

Small services 3,750 2,995

Retail 4,057 3,458

Parking garages 4,368 7,717


50 CECONY

Table 4-12. Operating Hours Magnitude by Business Type

Building Type

Min

n*

NYTM

Operating

Hours

Study

Measured

Operating

Hours

Relative

Precision at 90%

Confidence

Percent

Difference in

Measured Hours

vs. NYTM

Retail 63 4,057 3,458 13% -15%

Offices 73 3,748 3,013 10% -20%

Parking Garages 57 4,368 7,717 8% 77%

Entire program** 231 N/A 3,468 16% N/A




CECONY 51

5 CONCLUSIONS AND RECOMMENDATIONS

This section presents the conclusions and recommendations made by the evaluation team as a

result of the evaluation.

5.1 Observations and Conclusions

The results of this study show six main conclusions. These conclusions are listed below and

discussed in detail in the Results section.

1. The difference between the tracking and evaluated operating hours for the parking

garages sector was found to be lower than that for the retail and office sectors.

2. The evaluated parking garages operating hours were found to be significantly greater

than the NYTM deemed value.

3. Similarly, the evaluated operating hours for the offices and retail sectors were found to be

lower than the NYTM-deemed value.

4. The contractors are not consistently using the NYTM hours for all the lighting projects.

5. Parking garages were found to have a high FR.

6. Chains were found to have higher FR than non-chains.

5.2 Recommendations

The evaluation team’s recommendations for the program, the NYTM, and any future

evaluations are given in the following section.

5.2.1 New York Technical Manual Recommendations

The evaluators recommend that the NYTM authors either update the operating load hours for

selected building types based on this study’s research, as shown in Table 4-11 above, or create

new “small business” or “downstate” retail, office, and parking garage categories specifically

for SBDI programs that reflect this study’s findings.

5.2.2 Program Recommendations

The evaluation team offers seven recommendations for increasing program realization rates,

savings and cost-effectiveness. These are listed below and then discussed in detail. The

evaluation team’s recommendations focus on driving participation toward higher energy

savings per transaction or on lowering the cost of achieving savings.


52 CECONY

Revise operating hours – The evaluation found that parking garages operate more hours

than that suggested by NYTM, whereas the offices and retail facilities operate fewer

hours than the NYTM’s suggested values. If the DPS chooses not to update the NYTM-

deemed hour estimates in response to this research, the evaluators recommend that the

SBDI program itself apply the operating hours for the various building types identified in

the study. This will improve the savings estimation accuracy using data most applicable

to the SBDI program population and increase the RR in the next evaluation.

Account Identity – Identify all Con Edison primary usage accounts associated with each

participating facility or part of a facility for customers that possess multiple accounts.

Use pre-retrofit billing data for QC – During the first EEPS cycle, the SBDI installation

contractor used pre-retrofit billing data to cross check the savings magnitude for every

project. The EM&V group flags sites with high savings to usage ratio and visits them to

verify the measures. However, no action was taken by the program contractors on such

sites with high savings to usage ratios. We recommend that the program staff or the

implementation contractor inspect all the job sites with savings to usage ratios greater

than or equal to 80% and adjust the claimed savings accordingly.

Standardize the use of NYTM hours – Standardize the use of NYTM hours as a matter of

general procedure instead of site-specific estimates. The program standardized on NYTM

hours in October of 2010 but evaluators still found some sites where installer reported

hours were used for estimating savings.

Do not rely on savings from CFLs – CFLs had lower installation rates and higher FR. As

CFLs become standard practice, net savings from CFLs will continue to decline. CFLs

may still be useful to the program as a “loss leader” measure that can be offered to get

auditors in the door, but most of the savings from CFLs will likely not be realized.

Include installation locations information in the tracking data – The tracking data

included all the lighting inventory information except the installation locations. The

evaluators extracted the installation location information from the actual work orders,

which was a time consuming process. We recommend that CECONY look into including

the installation location information in the tracking data for each measure or group of

measures installed, which will then serve as a more complete source of data for impact

evaluations and post installation inspections.

Track service addresses – Ensure the site address matches the location where the

measures were installed and not the billing address for the participant.


CECONY 53

5.2.3 Evaluation Recommendations

Upon completing this evaluation, the evaluation team has some recommendations for ways to

improve future SBDI evaluations.

Error Ratio - The error ratio used for future SBDI evaluations needs to account for the

reliability of the tracking database. The final error ratio based on the RR of this study was

0.9, indicating that there was more variation in the sample than anticipated by the typical

0.3 or 0.4 error ratio assumption generally used for lighting impact evaluation sampling.

Billing analysis – This population is not a good candidate for program-level billing

analysis of savings. We recommend that it not be attempted again for Con Edison due to

the complex account association challenges found in New York City, unless the program

is able to definitively identify all accounts affected by project activities, with emphasis on

the primary usage accounts located at the site of record. This appears to be a consistent

pattern as similar billing analysis results were observed for other PA small business

programs.

Metering Duration - Prior to processing the twelve month logged data, the evaluators

expected the numbers to remain in the same ball park as the three month logged data.

However, the overall twelve month metered operating hours went up by 8% compared to

three month metered data. Given the observed increase in operating hours for longer

duration metering, the evaluators recommend metering for longer duration whenever

possible.


CECONY A-1

APPENDIX A: STATISTICAL SAMPLING METHODOLOGY

This section provides a brief summary of the model-based statistical sampling (MBSS)

methodology. This methodology has been applied to the evaluation of energy efficiency

program impacts by DNV KEMA for more than 20 years, and it has proven to be effective and

efficient. It is also widely accepted by the industry as the foundation for load research sampling.

In energy efficiency impact evaluations, the goal is typically to estimate savings RRs as the ratio

between evaluated savings and preliminary estimates of savings recorded in tracking system

data.

Background A.1

Conventional methods for sample design and estimation are documented in standard texts,

such as Cochran’s Sampling Techniques1. MBSS is grounded in theory of model-assisted survey

sampling developed by C.E. Sarndal and others2 3. MBSS methodology has been applied in load

research and impact evaluation for more than 30 years. This fusion of theory and practice has

led to important advances in both model-based theory and practice, including the use of the

error ratio for preliminary sample design, the model-based methodology for efficient stratified

ratio estimation, and effective methods for domains estimation.

MBSS and conventional methodologies are currently taught in the AEIC Advanced Methods in

Load Research seminar. MBSS methodology is also documented in The California Evaluation

Framework4. It has been used in countless load research and program evaluation studies and has

been examined in public utility hearings and in numerous EPRI studies.

The Role of the Statistical Model A.2

As we have said, MBSS uses a statistical model to guide the study planning and the sample

design. The parameters of the model, especially the error ratio, are used to represent prior

information about the population to be sampled. The model describes the nature of the

variation in the relationship between any target y variable of the study and one or more x

variables that can be developed from known tracking data and other supporting information.

The y variable can be any of the measurements taken at the evaluated site or any function

1 “Sampling Techniques,” by W. G. Cochran, 3rd Ed. Wiley, 1977. 2 “Model Assisted Survey Sampling,” by Carl Erik Sarndal, Bengt Swensson, and Jan Wretman, Springer-Verlag,

1992. 3 Wright, R. L. (1983), “Finite Population Sampling with Multivariate Auxiliary Information,” Journal of the

American Statistical Association, 78, 879-884. 4 The report can be downloaded from the site http://www.calmac.org/calmac-filings.asp.

http://www.calmac.org/calmac-filings.asp


CECONY A-2

thereof. In impact evaluations, the x variable is usually the tracking system savings estimate.

The model is used to help choose the sample size n, to assess the expected statistical precision of

any sample design, and to help formulate a sample design that is efficiently stratified for ratio

estimation.

The model is used as a guide to the sample design, but the results of the study itself are not

strongly dependent on the accuracy of the model5. Once the sample design is selected, the

subsequent analysis of the data is usually based only on the sample design and not on the

model used to develop the sample design. In particular, conventional stratified-sampling

techniques can be used to analyze the sample data collected from an MBSS sample design. The

resulting estimates will be essentially unbiased in repeated sampling and the confidence

intervals will also be valid, provided that the sample design has been followed to select the

sample customers. The results will be consistent with traditional sampling theory as found in

texts such as Cochran’s “Sampling Techniques” and consistent with standard load and market

research practice.

Stratified Ratio Estimation A.3

We assume that an impact evaluation study is to be conducted of a given population of N

projects in a given program. In the study, the sample sites will be monitored and savings will be

determined based on actual loads and observed operations. We let y denote any characteristic to

be determined from the on-site evaluation, and we let x denote any suitable characteristic of the

site.

We define the population ratio B by the equation:

N

i

i

N

i

i

x

y

B

1

1

Here the summations are over the entire N units (e.g., accounts, measures, or projects) in the

target population. We note that the population mean or total of y is equal to B times the

population mean or total of x. The latter is assumed to be known from the tracking data.

We assume that a sample of n sites is selected following a stratified sample design. For each

sample customer we define the case weight w to be equal to the number of sites in the target

5 Other methods, called model-dependent sampling, are much more dependent on the accuracy of the model. Such

methods are not commonly used in load research or evaluation applications since they would be more difficult to

defend than MBSS and conventional methods.


CECONY A-3

population within the stratum containing the given site divided by the number of sites in the

sample within the given stratum. The case weight is used to avoid any bias that might

otherwise arise from the different sampling fractions used from one stratum to another.

Using the case weight, we define the combined ratio estimator of B by the equation:6

n

i

ii

n

i

ii

xw

yw

b

1

1

Then, if desired, the population mean or total of y can be estimated as b times the population

mean or total of x, known from the tracking data.

Using the case weights, we calculate the relative precision at the 90% level of confidence in three

steps:

1. Calculate the sample residual for each unit in the sample.

2. Calculate7

n

i

ii

n

i

iii

xw

eww

bse

1

1

21

3. Calculate

b

bserp

645.1

A 90% confidence interval for B is calculated using the equation. A confidence interval for the

mean or total can be calculated in a similar way.

6 This equation gives the same result as the conventional stratum-weighted equation:

L

h

hh

L

h

hh

xN

yN

b

1

1

7 The conventional equation is

L

h h

h

h

hhL

h

hh

n

es

N

nN

xN

bse1

2

2

1

11

where,

hn

i

i

h

h een

es1

22

1

1

Our equation assumes that

hn

i

i

h

een 1

2

1

1is approximately equal to

hn

i

i

h

en 1

21in each stratum.


CECONY A-4

We can also use the sample data to estimate a measure of population variability called the error

ratio, denoted er. The error ratio, defined in the next section, is the key determinant of the

expected relative precision, along with the sample size n. We estimate the error ratio from the

sample using the following equation:

n

i

ii

n

i

ii

n

i

iii

yw

xwxew

re

1

11

2

ˆ

The parameter (gamma) is also defined in the next section. In load research and evaluation

applications it is usually taken to be 0.8. We will not attempt to interpret the preceding equation

here, but we will define both the error ratio and gamma in the following section.

A key advantage of the MBSS methodology is the ease of domains estimation. A domain is

any identifiable subset of the population, e.g., the sites in a particular region or having a

particular appliance or end use. Domain estimation is the process of obtaining the results of

interest for one or more domains. With the MBSS methodology, domains estimation is very

straightforward. We usually calculate the case weights for each sample site to reflect the

sample design and current population and then regard them as fixed for any domains

analysis. Then we simply evaluate the preceding equations for the sample sites that are

included in each domain.8

The Ratio Model A.4

The ratio model is used to choose the appropriate sample size n, to assess the expected

statistical precision of any stratified sample design, and to develop an efficiently stratified

sample design. The ratio model describes the relationship between y and x for the set of all units

in the population. The model consists of two equations called the primary and secondary

equations respectively.9

iii xy

iii xsd 0

8 In the software, a domain is any class or sector. 9 The x-variable in the primary equation is sometimes different than the x-variable in the secondary equation. In the

SAS modules, we refer to the later as the stratification variable. For simplicity, we will not make this distinction in the

theoretical discussion given here.


CECONY A-5

Here i denotes any customer, account, or premise in the target population. 0ix is usually

known throughout the population. The primary equation describes the relationship between the

y variable of interest and the x variable used in the ratio estimate, i.e., annual use. Since we

assume that 0iE , the primary equation can also be written as iii xyE . Here i

denotes the expected value of y for unit i. The primary equation says that under the model, the

expected value of iy is equal to a fixed constant times the known ix

The quantity, iii y , is called the residual. The N residuals are considered to be N

independent random variables. The standard deviation of i is denoted as i . We refer to i as

the residual standard deviation of each customer i. The secondary equation is used to estimate the

residual standard deviation and to guide the development of an efficient sample design.

To summarize, under the ratio model, the target variable iy is a random variable with expected

value i and standard deviation i . The expected value i is determined by the primary

equation of the model. The standard deviation i is determined by the secondary equation of

the model. There are three parameters in the model: (beta), 0 (sigma-naught), and

(gamma).

Figure A-1 shows an example. The points of the scatterplot represent the values of (x, y) for each

site in the population. The solid line represents the equation xy , i.e, the expected value of y

given x. This is a line through the origin with slope given by the parameter . The two dashed

lines represent the equation xy , i.e, the one-standard deviation interval around the

expected value. Here x0 so the dashed lines are determined by the two parameters 0

and .10

10 The role of gamma can be seen by rewriting this equation as )log()log( x , where )log( 0 . This shows that

for each site in the population the log of sigma is a constant plus gamma times the log of the value of x for the site.

Gamma is the slope in the relationship between the log of x and the log of sigma.


CECONY A-6

Figure A-1 The Ratio Model

Now we are finally positioned to define the error ratio. The error ratio is defined by the equation:

N

i

i

N

i

i

er

1

1

The error ratio can be regarded as an alternative parameter to 0 since under the preceding

ratio model, 0 can be calculated from the error ratio using the equation:

N

i

i

N

i

i

x

er

1

1

0

As we will see in the next section, the error ratio is the key measure of variability when

stratified ratio estimation is to be used to analyze the data. Figure A-2 shows some examples. If

the error ratio is close to zero, there is a strong relationship between x and y. If the error ratio is

larger, the relationship is weaker.


CECONY A-7

Figure A-2 Examples of Different Error Ratios

Choosing the Sample Size A.5

We assume that the ratio model provides a reasonably accurate description of the relationship

between y and x in the target population. We also assume that the sample design will be

efficiently stratified as discussed in the following section and that the analysis will use stratified

ratio estimation.

Under these assumptions and the added assumption that the population size N is large, then

the expected relative precision is given by the equation:

n

errp 645.1

If the population is relatively small, the finite population correction factor can be added, giving

n

er

N

nrp 1645.1 .

0

5,000

10,000

15,000

20,000

0 10,000 20,000 30,000 40,000 50,000 60,000

0

5,000

10,000

15,000

20,000

0 10,000 20,000 30,000 40,000 50,000 60,000

0

5,000

10,000

15,000

20,000

0 10,000 20,000 30,000 40,000 50,000 60,000

0

5,000

10,000

15,000

20,000

0 10,000 20,000 30,000 40,000 50,000 60,000

er = .2 er = .4

er = .8er = .6


CECONY A-8

If the desired relative precision D is specified, then the preceding equations can be solved to

determine the required n. If the population size N is large, we have

2645.1

D

ern

If the population is small, the sample size can be calculated in two steps. First, calculate 2

0

645.1

D

ern . Then calculate

Nn

nn

0

0

1

These equations are generally sufficient to develop a preliminary plan. However, added issues

must usually be addressed. First, there are usually many y variables of interest, i.e., annual kWh

savings, connected load savings, or savings in a peak hour. Second, it is often necessary to

consider the expected statistical precision in various segments of the target populations. Third,

there are usually limits on the sample size or other resource constraints.11

Model-Based Stratification A.6

The preceding results assume that the sample is efficiently stratified. Under the ratio model, an

efficiently stratified sample design for ratio estimation can be developed in the following steps: 12

1. Use the sampling frame and the assumed model to calculate i for each customer in the

population.

2. Choose the desired number of strata,13

11 When domains estimation is involved we may use two added results: 1) The standard error of the total of y across

two or more mutually exclusive domains is the square root of the sum of the squared standard error of the total of y

within each of the individual domains; 2) For estimating the total of y across two or more mutually exclusive

domains, the optimal allocation of the sample to each domain is proportional to the sum of the i within each

domain. The later is equal to the error ratio within the domain times the expected value of the total of y within the

domain.

12 This methodology is the model-based version of the Dalenius-Hodges method of constructing strata combined

with optimal allocation of the sample using the within-strata population standard deviation of the ie . However,

Dalenius-Hodges stratification is approximately optimal for stratified mean per unit estimation, whereas model-

based stratification is approximately optimal for stratified ratio estimation. Moreover, with conventional methods it

is common to calculate the required sample size from the within-stratum population standard deviation of ix . This

practice can yield very misleading results and, thus, cannot be recommended.

13 With MBSS methodology we can systematically assess the gain from increased stratification. These studies indicate

that five annual-use strata are usually sufficient in most load research applications. Some applications may call for

added stratification by seasonal use, customer load factor, etc.


CECONY A-9

3. Sort the sampling frame by increasing i .

4. Choose stratum cut points to divide the sum of the i approximately equally between the

strata.

5. Allocate an equal number of sample customers to each stratum.

6. Make added adjustments if the sample size exceeds the population size in any stratum.

Under the ratio model, i is determined by the x variable together with the value of .

Methods are available for estimating from a sample. Indeed, we have estimated in

numerous studies. We have found that the estimated values are clustered around 0.8. We have

also found that the key results are not very sensitive to . Therefore, in load research and

evaluation applications, we generally recommend the use of 8.0 both in constructing strata

as discussed in this section and in estimating the value of the error ratio from a given sample.

Evaluating the Precision of any Design A.7

For any sample design, we define the inclusion probability of each site in the population,

denoted i , to be the probability that the site is included in the sample. For a stratified sample

design, the inclusion probability is the sampling fraction in each stratum, i.e., hh Nn .

Under the ratio model discussed previously and any sample design, the expected relative

precision of the stratified ratio estimator is:

N

i

i

N

i

iizrp11

21 1

Here 645.1z for the 90% level of confidence.

This key result has the following mathematical implications:

1. For any given sample size n , a sample design is said to be efficient if the sample design

minimizes the expected relative precision. For any efficient sample design, iN

i

i

i

n

1

provided that the right-hand side is less than 1.

2. If the right-hand side is greater than 1, the site should be included with certainty.


CECONY A-10

3. If the sample design is efficient and the population is large, then the expected relative

precision is n

erzrp .

4. The model-based sample design, described in Section 2.2, is practically efficient as long as

the number of strata is large enough.

The preceding equation can be also used to calculate the expected statistical precision of any

sample design in any domain of interest. For example, this result can be used to calculate the

expected relative precision as the number of strata is increase, e.g., from 5 to 6 to 7, etc. This

type of analysis has led us to conclude that five strata are usually enough in most cases.

Summary A.8

Extensive experience indicates that stratified ratio estimation is very effective in almost all load

research and evaluation applications. MBSS methods are generally grounded on the same

principles as conventional sampling methods such as Dalenius-Hodges stratification and

Neyman allocation, but MBSS methods are specifically tailored to ratio estimation. Some

methods for calculating sample sizes that have been used in the past can provide badly

misleading results for ratio estimation. The MBSS approach addresses these problems and

provides a coherent, consistent approach to both sample design and analysis. The MBSS

methodology follows the life cycle of load research and evaluation studies very nicely.

A bonus of MBSS methodology is its strength for multiple y variables and domains estimation.

At the sample design stage, MBSS provides straightforward methods for assessing the statistical

precision expected for various y variables and domains of interest from the associated error

ratios. At the analysis stage, MBSS again provides straightforward methods for developing

estimates and their statistical precision for various y variables and domains, and for estimating

the associated error ratios. In the past it has been thought to be risky to report results for

domains that were not factored into the sample design. MBSS methodology has shown that

meaningful results can generally be developed for questions that arise later in the study, much

after the planning stage.


CECONY B-1

APPENDIX B: RATIO EXPANSION – SAMPLE TO POPULATION

This appendix provides the specific ratio estimation computations DNV KEMA employed to

develop estimates of evaluation verified gross and net impacts.

Ratio Estimation B.1

DNV KEMA used the statistical procedure of ratio estimation to develop estimates of

evaluation verified gross and net impacts. There are two basic steps in the process. The first step

is to verify energy savings in a sample of measures. DNV KEMA accomplished this first step via

CATI surveys and on-site visits. The second step is to expand the sample results to the

population of measures. This is accomplished by calculating the ratios of verified-to-reported

and attributable-to-verified for the sample. The ratios are also referred to in this discussion as

adjustment factors or rates.

B.1.1 Expansion of Sample Results to the Population via Ratio Analysis

The calculation of the adjustment factors for tracking system gross and net savings uses

appropriate case weights corresponding to the sampling rate. The four directly calculated

adjustment factors are the installation rate, the calculation adjustment, the operating hours

adjustment, and the net-to-gross (NTG) factor.

Each of these is calculated as a ratio estimator over the sample of interest (Cochran, 1977, p.165).

The formulas for these factors are given below.

Notation: The following terms are used in calculating the adjustment factors:

GTj = tracking estimate of gross savings for measure j

GIj = tracking estimate of gross savings for measure j, adjusted for non-installation

GCj = engineer verified estimate of gross savings for measure j, not adjusted for operating

hours

GVj = verified gross savings for measure j

NVj = net savings determined from the CATI surveys.

WOj = weighting factor for measure j used to expand the on-site sample to the full population

WCj = weighting factor for measure j used to expand the CATI sample to the full population


CECONY B-2

WAj = weighting factor for measure j used to expand the combined on-site and CATI sample to

the full population

B.1.1.1 Installation Rate

The installation rate RI is calculated using the combined on-site and CATI samples as:

∑

∑

B.1.1.2 Calculation Adjustment

The Calculation Adjustment RC is calculated from the on-site sample as:

∑

∑

B.1.1.3 Operating Hours Adjustment

The operating hours adjustment RH is calculated from the on-site sample as:

∑

∑

B.1.1.4 Net-to-Gross

The NTGR RA is calculated from the CATI sample as1:

∑

∑

B.1.2 Standard Errors

The ratio estimator is calculated using a SAS® macro provided by SAS for ratio estimation by

domains. The procedure also returns the standard error of the estimate.

The standard error is calculated recognizing the sample as drawn from a finite population: the

measures completed within the analysis period with associated energy impacts in the program-

tracking database. This calculation uses the finite population correction (FPC) factor. This factor

is a reduction to the calculated variance that accounts for the fact that a relatively large fraction

of the population of interest has been observed directly and is not subject to uncertainty. It is

appropriate to apply precision statistics, such as confidence intervals, based on the standard

error calculated in this manner when quantifying the results of the program during the study

period only.

1 For the net-to-gross ratio, the verified gross savings for measures in the CATI survey (GVj) were estimated based on

the gross savings adjustments found for measures of the same measure type in the on-site sample.


CECONY B-3

B.1.2.1 Combined Ratio Estimators

The engineering adjustment factor, RE, is calculated by chaining together the installation rate

and the calculation adjustment:

[∑

∑ ] [∑

∑ ]

This is an example of a chained ratio estimator using a nested sample. The standard error for

the chained ratio is approximated by the formula:

( ) √[( ( )

) (

( )

)

].

(This formula overstates the standard error, because it ignores the correlation between the

numerator of RI and the denominator of RC, which reduces the variance of the product.)

Likewise, the gross savings RR, Rv, is calculated by chaining together the engineering

adjustment and the operating hours adjustment:

And the net savings RR, RN, is calculated by chaining together the gross savings RR and the

NTGR:

The same standard error approximation formula allows (an overestimate of) the standard errors

of each of the RRs to be calculated from the two separate standard errors.


CECONY C-1

APPENDIX C: FACILITY TYPE MAPPING

Table C-1 presents the mapping whereby the sixty NYTM facility types were collapsed into

thirty-four business types based on similar NYTM lighting hours.

Table C-1. Mapping of Facility Types to Business Types

TRM Facility Type TRM Hours Business Type

Pump stations 1,949 Pump stations

Entertainment 1,952 Assembly

Fire station (unmanned) 1,953 Fire station (unmanned)

Convention center 1,954 Assembly

Motion picture theatre 1,954 Assembly

Sports arena 1,954 Assembly

Church 1,955 Religious

Religious building 1,955 Religious

School / university 2,187 Education

Schools (jr./sr. high) 2,187 Education

Schools (preschool / elementary) 2,187 Education

Schools (technical / vocational) 2,187 Education

College – classes / administrative 2,586 College classrooms

Gymnasium 2,586 Edu – recreational

Performing arts theatre 2,586 Edu – recreational

Refrigerated warehouse 2,602 Warehouse

Warehouse (not refrigerated) 2,602 Warehouse

Light manufacturers 2,613 Light manufacturers

College – cafeteria 2,713 College – cafeteria

Bakery 2,854 Bakery

Industrial – 1 shift 2,857 Industrial – 1 shift

Manufacturing facility 2,857 Industrial – 1 shift

Lodging (hotel / motel) 3,064 Lodging

College – dormitory 3,066 Lodging

Commerical condos 3,100 Commerical condos

Office (general office types) 3,100 General Offices

Banks 3,748 Offices

Courthouse 3,748 Offices

Library 3,748 Offices

Medical offices 3,748 Offices

Museum 3,748 Offices

Office / retail 3,748 Offices

Post office 3,748 Offices

Town hall 3,748 Offices

Small services 3,750 Offices

Workshop 3,750 Workshop

Food stores 4,055 Food stores

Auto-related 4,056 Auto-related


CECONY C-2

Table C-1 Mapping of Facility Types to Business Types, continued

Table C-2 presents the savings by business type for CECONY, with the three building types that

represent the largest proportions of savings highlighted in yellow. There were some accounts

that were not coded correctly in the tracking data as one of the original sixty facility types; these

were assigned to a business type called “missing/bad.”

TRM Facility Type TRM Hours Business Type

Laundromats 4,056 Retail

Retail 4,057 Retail

Parking lots 4,100 Parking lots

Dining – bar lounge / leisure 4,182 Dining

Dining – family 4,182 Dining

Restaurants 4,182 Dining

Parking garages 4,368 Parking garages


Mall concourse 4,833 Mall concourse

Penitentiary 5,477 Penitentiary

Exercise center 5,836 Exercise center

Nursing homes 5,840 Nursing homes

Convenience stores 6,376 Convenience stores

Fast food restaurant 6,376 Fast food restaurant

Dining – cafeteria / fast food 6,456 Fast food restaurant

Transportation 6,456 Transportation


Waste water treatment plant 6,631 Waste water treatment plant

Multifamily (common areas) 7,665 24-hour public places

Police / fire stations (24-hr) 7,665 24-hour public places

Hospitals/health care 7,666 24-hour public places

Hospitals 7,674 24-hour public places


CECONY C-3

Table C-2. CECONY Accounts and Savings by Business Type

Business Type

# of

Accounts

% of

Accounts

MWh

Savings

% of

Savings

24-hour public places 96 1.4% 2,955 2.9%

Assembly 35 0.5% 543 0.5%

Auto related 445 6.4% 5,588 5.5%

Bakery 75 1.1% 460 0.5%

College – cafeteria 2 0.0% 83 0.1%

College classrooms 2 0.0% 14 0.0%

Commercial condos 44 0.6% 313 0.3%

Convenience stores 361 5.2% 3,873 3.8%

Dining 408 5.9% 3,734 3.7%

Edu – recreational 25 0.4% 502 0.5%

Education 109 1.6% 2,280 2.2%

Exercise center 23 0.3% 758 0.7%

Fast food restaurants 148 2.1% 1,972 1.9%

Food stores 453 6.5% 5,769 5.7%

General offices 458 6.6% 4,466 4.4%

Industrial – 1 shift 129 1.9% 2,787 2.7%



Light manufacturers 49 0.7% 522 0.5%

Lodging 23 0.3% 522 0.5%

Mall concourse 3 0.0% 18 0.0%

Missing/bad 45 0.6% 1,647 1.6%

Nursing homes 4 0.1% 158 0.2%

Offices 1,104 15.9% 8,736 8.6%

Parking garages 93 1.3% 9,941 9.8%

Parking lots 5 0.1% 29 0.0%

Penitentiary 4 0.1% 26 0.0%

Pump stations 1 0.0% 6 0.0%

Religious 158 2.3% 2,388 2.4%

Retail 2,171 31.3% 32,084 31.6%

Transportation 14 0.2% 468 0.5%

Warehouse 296 4.3% 5,482 5.4%

Workshop 80 1.2% 1,148 1.1%

Total 6,935 100.0% 101,504 100.0%


CECONY D-1

APPENDIX D: ADDITIONAL LOOKS AT RESULTS

Installation Rate D.1

Table D-1 shows that the lowest installation rates deviation for CECONY measure groups was

for tube lighting, with -3%. The highest installation rate deviation for a measure group was seen

for CFLs, with -22%. CFLs, as small measures that are easy to install and remove, frequently

have lower persistence than larger measures that are harder to install and remove.

Table D-1 Installation Rate, by Measure Group, CECONY

Table D-2 shows that for the CECONY program, sites with a mix of free and low-cost measures

had a lower installation rate than sites that had only low-cost measures. This finding is not

unexpected. First, CFLs are the most likely measure to be provided for free, and as mentioned

above, CFLs are easier to uninstall than many other measures. Second, free measures commonly

have lower installation rates than non-free measures because the participant has nothing

invested in them, making them less motivated to put the measures to use.

Table D-2 Installation Rate, by Cost Type, CECONY

Chain or non-chain status for businesses did not have much effect on the installation rate for the

CECONY program, as shown in Table D-3.

Measure Group Min n*

Installation

Rate

Deviation

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Tube lighting 378 -3% 1% 68%

CFL 217 -22% 13% 10%

Other lighting 257 -6% 2% 18%

Other 30 -6% 7% 4%

CECONY overall 882 -6% 2% 100%


Cost Type Min n*

Installation

Rate

Deviation

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Some free 236 -20% 7% 11%

All low-cost 646 -5% 2% 89%




CECONY D-2

Table D-3 Installation Rate, by Business Type, CECONY

Brooklyn, the largest borough in terms of program savings, has a lower installation rate (89%)

than the other boroughs as shown in Table D-4. No other borough was below 94%.

Table D-4 Installation Rate, by Borough, CECONY

Table D-5 shows that installation rates deviation for CECONY by building type.

Table D-5 Installation Rate, by Building Type, CECONY

Chain Min n*

Installation

Rate

Deviation

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Chain 137 -4% 2% 14%

Non-chain 745 -7% 2% 86%



Borough Min n*

Installation

Rate

Deviation

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Brooklyn 244 -11% 6% 30%

Bronx 92 -5% 4% 12%

Manhattan 191 -6% 2% 20%

Queens 209 -5% 3% 25%

Staten Island 8 0% 1% 2%

Westchester 138 -3% 2% 10%




Installation

Rate

Deviation

Relative

Precision at

90%

Confidence

Percent

Program

Savings

Retail 227 -10% 6% 30%

Offices 160 -9% 3% 8%

Parking Garages 75 -9% 4% 9%

Other 420 -4% 2% 53%




CECONY D-3

Calculation Adjustment D.2

Table D-6 shows that the “tube lighting” category had the largest calculation adjustment at 97%,

which is a 3% difference from the program’s calculated savings. No “other” (non-lighting)

measures were evaluated as part of the on-site sample, and so no calculation adjustment is

available for that measure group.

Table D-6. Calculation Adjustment, by Measure Group, CECONY

Table D-7 shows that sites with some free measures had a smaller calculation adjustment (3%

different from tracked) than sites where all the measures of a type were low-cost (-2% different

from tracked).

Table D-7 Calculation Adjustment, by Cost Type, CECONY

Chain installations had a greater calculation adjustment, 97% versus 98% for non-chain

installations, as shown in Table D-8.


Calculation

Adjustment

Deviation

Relative

Precision at

90% Confidence

Tube lighting 110 -3% 4%

CFL 48 5% 6%

Other lighting 73 -1% 7%

CECONY overall 231 -2% 3%


Cost Type Min n*

Calculation

Adjustment

Deviation

Relative

Precision at

90% Confidence

Some free 53 3% 5%

All low-cost 178 -2% 4%




CECONY D-4

Table D-8 Calculation Adjustment, by Business Type, CECONY

Brooklyn had the greatest calculation adjustment of any borough at 92%, as shown in Table D-9.

Table D-9 Calculation Adjustment, by Borough, CECONY

Table D-10 presents the calculation adjsutment deviations for CECONY by building type.

Table D-10 Calculation Adjustment, by Building Type, CECONY

Operating-Hours Adjustment D.3

Table D-11 shows the operating-hours adjustments for the CECONY program by measure

group. The operating-hours adjustment is based on the on-site sample only. The largest

Chain Min n*

Calculation

Adjustment

Deviation

Relative

Precision at

90% Confidence

Chain 42 -3% 6%

Non-chain 189 -2% 3%



Borough Min n*

Calculation

Adjustment

Deviation

Relative

Precision at

90% Confidence

Brooklyn 63 -8% 8%

Bronx 16 6% 4%

Manhattan 79 1% 3%

Queens 51 2% 5%

Staten Island 2 ** <0.1%

Westchester 20 0% 6%



**Results w ith few er than 5 completes are not reported separately


Calculation

Adjustment

Deviation

Relative

Precision at

90% Confidence

Retail 63 2% 4%

Offices 73 2% 5%


Other 38 -4% 6%




CECONY D-5

adjustment was to CFLs at 75%. An operating-hours adjustment significantly different from

100% indicates that the hours assumed for the lights were different from the hours found via

light loggers at the time of the evaluation.

Table D-11. Operating Hours Adjustment, by Measure Group, CECONY

Table D-12 shows that installations in the “some free” category have a greater operating-hours

adjustment at 74%, versus 84% for installations in the “all low-cost” category.

Table D-12. Operating Hours Adjustment, by Cost Type, CECONY

Chain installations have a greater operating-hours adjustment, 78% versus 83% for non-chain

installations, as shown in Table D-13.

Table D-13. Operating Hours Adjustment, by Business Type, CECONY


Op hours

Adjustment

Deviation

Relative

Precision at

90% Confidence

Tube lighting 110 -18% 8%

CFL 48 -25% 12%

Other lighting 73 -6% 26%



Cost Type Min n*

Operating

Hours

Adjustment

Deviation

Relative

Precision at

90% Confidence

Some free 53 -26% 12%

All low-cost 178 -16% 8%



Chain Min n*

Operating

Hours

Adjustment

Deviation

Relative

Precision at

90% Confidence

Chain 42 -22% 23%

Non-chain 189 -17% 8%




CECONY D-6

Table D-14 shows that Manhattan had the least difference between tracked and logged

operating hours, with a 94% adjustment. Westchester had the largest adjustment at 64%.

Table D-14. Operating-Hours Adjustment, by Borough, CECONY

Table D-15 presents the operating hours adjsutment deviation for CECONY by building type.

Table D-15. Operating-Hours Adjustment, by Building Type, CECONY

Gross Savings Realization Rate D.4

Table D-16 shows that gross savings were adjusted the most for CFLs: due primarily to a low

installation rate and low operating-hours adjustment, only 61% of CFL savings claimed by the

program were realized. Both other measure groups with a GSRR realized close to 80% of the

program tracked claimed.

Borough Min n*

Operating

Hours

Adjustment

Deviation

Relative

Precision at

90% Confidence

Brooklyn 63 -15% 13%

Bronx 16 -32% 26%

Manhattan 79 -6% 15%

Queens 51 -19% 13%


Westchester 20 -36% 35%





Operating

Hours

Adjustment

Deviation

Relative

Precision at

90% Confidence

Retail 63 -13% 12%

Offices 73 -19% 10%

Parking Garages 57 -9% 5%

Other 38 -22% 15%




CECONY D-7

Table D-16 Gross Savings RR, by Measure Group, CECONY

Table D-17 shows that non-free installations had a smaller gross savings adjustment with 78%

against 61% adjustment for the “some free” category.

Table D-17 Gross Savings Adjustment, by Cost Type, CECONY

Table D-18 shows that measures installed at a chain location had a lower GSRR (72%) than those

installed at a non-chain location (76%).

Table D-18. Gross Savings Adjustment, by Business Type, CECONY

Table D-19 shows that Manhattan and Queens each have GSRRs between 79% and 90% (89%

and 79%, respectively). Each of the other boroughs for which we have a GSRR, the Bronx,

Brooklyn and Westchester, have GSRRs between 60% and 70% (68%, 70%, and 62%,

respectively).


Gross

Savings

Realization

Rate

Relative

Precision at

90% Confidence

Tube lighting 110 77% 9%

CFL 48 61% 19%

Other lighting 73 87% 27%

CECONY overall 231 76% 9%


Cost Type Min n*

Gross

Savings

Realization

Rate

Relative

Precision at

90% Confidence

Some free 53 61% 15%

All low-cost 178 78% 9%



Chain Min n*

Gross

Savings

Realization

Rate

Relative

Precision at

90% Confidence

Chain 42 72% 24%

Non-chain 189 76% 9%




CECONY D-8

Table D-19. Gross Savings Adjustment, by Borough, CECONY

Table D-20 shows the GSRRs for CECONY by building type.

Table D-20. Gross Savings Adjustment, by Building Type, CECONY

Attribution D.5

Free Ridership

The direct attribution for CECONY was 84% or 85% for all measure groups other than CFLs, as

shown in Table D-21. CFLs are becoming the standard screw-in light bulb, which leads to

higher FR (lower attribution).

Borough Min n*

Gross

Savings

Realization

Rate

Relative

Precision at

90% Confidence

Brooklyn 63 70% 16%

Bronx 16 68% 27%

Manhattan 79 89% 15%

Queens 51 79% 14%







Gross

Savings

Realization

Rate

Relative

Precision at

90% Confidence

Retail 63 80% 14%

Offices 73 75% 12%


Other 38 72% 16%




CECONY D-9

Table D-21 Direct Attribution, by Measure Group, CECONY

Attribution was higher for measures that were all low-cost than those that included some free,

as shown in Table D-22. This is likely because the measures offered for free are inexpensive and

installing them is becoming standard practice.

Table D-22 Direct Attribution, by Cost Type, CECONY

Table D-23 shows that non-chain installations had a higher attribution than chain installations,

with 84% and 73%, respectively.

Table D-23 Direct Attribution, by Business Type, CECONY

D-24 shows that the direct attribution for each borough is within 6% of each of the others.


Direct

Attribution

Relative

Precision at

90% Confidence


CFL 120 70% 20%


Other 28 84% 10%



Cost Type Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Some free 136 70% 18%




Chain Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Chain 75 73% 11%





CECONY D-10

Table D-24 Direct Attribution, by Borough, CECONY

D-25 shows that the direct attribution for CECONY by building type.

Table D-25 Direct Attribution, by Building Type, CECONY

The alternative direct attribution followed the same patterns (albeit with slightly higher ratios)

as the direct attribution scores as shown in Tables D-26 through D-30.

Table D-26. Alternative Direct Attribution, by Measure Group, CECONY

Borough Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Brooklyn 153 84% 6%

Bronx 60 80% 10%

Manhattan 94 84% 6%

Queens 140 84% 5%

Staten Island 7 86% 10%





Direct

Attribution

Relative

Precision at

90% Confidence

Retail 136 86% 4%

Offices 75 85% 8%


Other 311 83% 5%




Direct

Attribution

Relative

Precision at

90% Confidence


CFL 120 71% 20%


Other 28 90% 7%




CECONY D-11

Table D-27. Alternative Direct Attribution, by Cost Type, CECONY

Table D-28. Alternative Direct Attribution, by Business Type, CECONY

Table D-29. Alternative Direct Attribution, by Borough, CECONY

Cost Type Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Some free 136 71% 18%




Chain Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Chain 75 73% 11%




Borough Min n*

Direct

Attribution

Relative

Precision at

90% Confidence

Brooklyn 153 85% 6%

Bronx 60 85% 8%

Manhattan 94 85% 6%

Queens 140 86% 5%






CECONY D-12

Table D-30. Alternative Direct Attribution, by Building Type, CECONY

Spillover

Overall kWh SO savings were less than 1% (0.19%). Most SO was found in the measure group

for other lighting, for measures that were all low-cost, and installed at non-chains, as shown in

Tables D-31, D-32, and D-33. Manhattan and Westchester had the highest SO rates among

boroughs with 0.33% and 0.81%, respectively, as shown in Table D-34.

Table D-31 KWh SO, by Measure Group, CECONY

Table D-32 KWh SO, by Cost Type, CECONY


Direct

Attribution

Relative

Precision at

90% Confidence

Retail 136 90% 3%

Offices 75 86% 6%


Other 311 85% 4%




kWh/kWh

Spillover

Ratio

Relative

Precision at

90% Confidence

Tube lighting 234 0.04% 115%

CFL 120 0.02% 128%

Other lighting 161 0.75% 109%

Other 28 0.00% 84%

CECONY overall 543 0.19% 93%


Cost Type Min n*

kWh/kWh

Spillover

Ratio

Relative

Precision at

90% Confidence

Some free 136 0.02% 127%

All low-cost 407 0.20% 94%




CECONY D-13

Table D-33 KWh SO, by Business Type, CECONY

Table D-34 KWh SO, by Borough, CECONY

Table D-35 KWh SO, by Building Type, CECONY

Net-to-Gross Ratio

The distribution of combined NTGRs mirrors those found for direct attribution, with only a

modest effect seen from the addition of SO, as shown in tables D-36 through D-40.

Chain Min n*

kWh/kWh

Spillover

Ratio

Relative

Precision at

90% Confidence

Chain 75 0.00% 80%

Non-chain 468 0.21% 93%



Borough Min n*

kWh/kWh

Spillover

Ratio

Relative

Precision at

90% Confidence

Brooklyn 153 0.00% 74%

Bronx 60 0.00% 75%

Manhattan 94 0.31% 162%

Queens 140 0.09% 131%

Staten Island 7 0.00% 194%

Westchester 89 0.79% 136%




kWh/kWh

Spillover

Ratio

Relative

Precision at

90% Confidence

Retail 136 0.00% 75%

Offices 75 0.21% 161%

Parking Garages 21 0.00% 97%

Other 311 0.29% 101%




CECONY D-14

Table D-36 Combined NTGR, by Measure Group, CECONY

Table D-37 Combined NTGR, by Cost Type, CECONY

Table D-38 Combined NTGR, by Business Type, CECONY


Combined

Net-to-Gross

Ratio

Relative

Precision at

90% Confidence


CFL 120 70% 20%


Other 28 84% 10%



Cost Type Min n*

Combined

Net-to-Gross

Ratio

Relative

Precision at

90% Confidence

Some free 136 70% 18%




Chain Min n*

Combined

Net-to-Gross

Ratio

Relative

Precision at

90% Confidence

Chain 75 73% 11%





CECONY D-15

Table D-39 Combined NTGR, by Borough, CECONY

Table D-40 Combined NTGR, by Building Type, CECONY

Net Realization Rate D.6

Table D-41 shows that the net RR was highest for “other lighting” measures, with 75%. Tube

lighting measures followed close behind, at 64%. CFLs displayed the lowest RR, with 43%.

Table D-41 Net RR, by Measure Group, CECONY

Table D-42 shows that the all low-cost installations have a higher net RR of 66% than the some-

free installations with 43%.

Borough Min n*

Combined

Net-to-Gross

Ratio

Relative

Precision at

90% Confidence

Brooklyn 153 84% 6%

Bronx 60 80% 10%

Manhattan 94 84% 6%

Queens 140 84% 5%






Combined

Net-to-Gross

Ratio

Relative

Precision at

90% Confidence

Retail 136 86% 4%

Offices 75 85% 8%


Other 311 84% 5%




Net

Realization

Rate

Relative

Precision at

90% Confidence


CFL 48 43% 28%





CECONY D-16

Table D-42 Net RR, by Cost Type, CECONY

Chain installations have a lower net RR of 53% than non-chain installations, with an net RR of

64% as shown in Table D-43.

Table D-43 Net RR, by Business Type, CECONY

Table D-44 shows that three boroughs had net RRs of nearly 50%, while two had net RRs of

more than 70%. Manhattan had the highest net RR of 75%.

Table D-44 Net RR, by Borough, CECONY

Table D-45 shows the net RR for CECONY by building type.

Cost Type Min n*

Net

Realization

Rate

Relative

Precision at

90% Confidence

Some free 53 43% 23%




Chain Min n*

Net

Realization

Rate

Relative

Precision at

90% Confidence

Chain 42 53% 26%

Non-chain 189 64% 10%



Borough Min n*

Net

Realization

Rate

Relative

Precision at

90% Confidence

Brooklyn 63 58% 17%

Bronx 16 55% 29%

Manhattan 79 75% 17%

Queens 51 66% 15%







CECONY D-17

Table D-45 Net RR, by Building Type, CECONY


Net

Realization

Rate

Relative

Precision at

90% Confidence

Retail 63 69% 15%

Offices 73 64% 14%


Other 38 60% 17%




CECONY E-1

APPENDIX E: DETAILED IPAD TOOL DESCRIPTION

The evaluation involved visiting 137 CECONY sites in a short amount of time, and deploying

an average of five lighting loggers at each project location. The site visits involved verifying the

installed lighting measures (quantity, fixture types, operating schedule, etc.) and HVAC setup

to determine the interactive savings.

In order to stay on schedule, it was necessary for field engineers to schedule and perform

multiple site visits per day. This required an approach that would streamline the data collection

and minimize each site visit’s duration without compromising the quality of the data collected.

To address this concern, ERS developed an iPad tool to facilitate the data collection process

during the site visits, and to reduce the post-site visit analysis time for calculating the lighting

measure savings. Figure E-1 shows a screenshot of the iPad tool summary page developed for

data collection.

Figure E-1 iPad Data Collection and Analysis Tool Summary Page


CECONY E-2

The iPad tool was created with the program FileMaker 12, a software package designed to

create database-driven applications for the Apple iPad. The custom application was loaded onto

the field iPad tablets with Apple’s iTunes software.

The following subsections describe the different aspects of the iPad tool in detail.

Database

The tracking data provided by CECONY was prepared and imported in to the FileMaker

database for use with the iPad tool. An empty database was loaded onto each iPad by the tool.

The central database with the site data resides on the ERS server.

Connection

Prior to the site visits, the field engineer downloads project data onto the field iPad’s database

from the central database on the ERS server through a secured VPN connection. Once

connected to the server, the user opens “Upload/Download” page on the tool, shown on

Figure E-2.

Figure E-2 Upload/Download Page

The field engineer selects and downloads the scheduled sites. Once the site data is downloaded

to the iPad, the list of downloaded sites becomes available for use in the “Site Selection”


CECONY E-3

window. Figure E-3 shows the screenshot of downloaded sites on the iPad. In the example that

it shows, only Site 007 had been downloaded to the tool.

Figure E-3 Listing of Downloaded Sites

Similarly, once the field engineer is done with the site visit, he or she uploads the site data back

to the central database through the secure VPN connection. This new site information is stored

as a new record with a site visit ID marker. Once the data is uploaded back to the central

database, the local (field iPad database) copy of the record is automatically deleted.

Tool Description and Flow

During the site visit, the field engineer clicks the downloaded site, and the “Site Information”

page is displayed first. This page has all the relevant information for the project, such as name,

address, primary contact details, account number, building type, etc. This information has

already been populated from the CECONY tracking data. The screenshot for the “Site

Information” page is presented in Figure E-4.


CECONY E-4

Figure E-4 Site Information Page

After verifying the information on this page, the field engineer clicks on the “Holiday” button,

which opens the next page featuring a list of predetermined holidays. On this page, the field

engineer selects the holidays observed by the site and enters any special holiday schedules. Any

special holidays that are not listed can be also be added on this page. The “Holidays” page is

presented in Figure E-5.

Figure E-5 Holidays Page


CECONY E-5

After entering the Holiday schedule, the field engineer navigates to the “Schedule Inventory”

page. All the site area descriptions from the lighting inventory are mapped to a standard space-

type profile. On this page, the user can see all the different space types and their associated

profiles. Figure E-6 presents a screenshot of the schedule inventory page.

Figure E-6 Schedule Inventory

The field engineer has the ability to input the operating schedule for each of the profile type on

this page by clicking on the “Profile Details” button. Figure E-7 shows the “Profile Details” page.

Figure E-7 Profile Details


CECONY E-6

Once the profile types and associated operating schedules are set, the field engineer goes on to

the next page, the “Measure Inventory” page, which displays the lighting inventory for the

entire site. The “Measure Inventory” page is presented in Figure E-8.

Figure E-8 Measure Inventory Page

On this page, the installed lighting fixtures are presented by location and profile type. The field

engineer is able to make changes to the data presented on this page, such as profile types, measure

types, quantities, lighting control method, etc. Clicking the “Fixture Controls Conditioning” button

brings up the individual measure details for that that particular line item. This page displays more

details about the selected measure, such as baseline fixture types, wattages, quantities, controls

types, heating/cooling checkboxes, etc. The individual measure details page is presented in Figure

E-9. The tool also has a database of standard wattages based on the NY Standard Wattage Tables

that can be used to determine measures if changes are encountered.


CECONY E-7

Figure E-9 Individual Measure Details Page

The tool also displays the verification status for the entire inventory at the bottom of the

“Measure Inventory” page (Figure E-8). The field engineer checks off the Complete/Verified

checkbox once the measure details are verified. In this manner, the field engineer can keep

track of verified measures.

Once the lighting inventory is verified, the field engineer goes to the “Sample” page, where a

calculation is performed to determine the appropriate number of loggers that should be

installed to obtain 20% precision at 80% confidence. The sampling page is shown in Figure E-10.


CECONY E-8

Figure E-10 Sample Page

Once the number of loggers is determined, the tool also randomly generates a list of locations

where the loggers should be installed for the given profile type. The field engineer can then

enter the logger number, deployment date and time (automatic), and an exact location of where

the logger was placed in the space by clicking the “Add Logger” button. The logger information

can then be entered in the “Logger Deployment” page, shown in Figure E-11. Swapped-out

loggers are also tracked here during subsequent site visits.


CECONY E-9

Figure E-11 Logger Deployment Page

Once the logger information is entered, the “Logger List” page is updated, as shown in

Figure E-12.

Figure E-12 Logger List


CECONY E-10

The field engineer can get an overview of the data collected for the site on the “Site Summary”

page (shown on Figure E-1). This page has a status indicator for all the various individual pages

for the site. For example, if all lighting inventory has been verified and the profiles have been

updated, the summary page will indicate a complete status for the “Measure Inventory” and

“Schedule Inventory” pages.


CECONY F-1

APPENDIX F: ATTRIBUTION METHOD

One goal of the evaluation of CECONY’s Small Business Direct Install Program1 is to measure

program net impacts – energy savings associated with equipment installations that would not

have been achieved in the absence of the program. Program attribution accounts for the

portion of the gross energy savings associated with a program-supported measure or behavior

change that would not have been realized in the absence of the program. The program-induced

savings, indicated as a NTGR, is made up of free ridership (FR) and participant spillover (PSO).

The formula used to calculate the NTG ratio can be expressed as:

FR is the portion of the program-achieved verified gross savings that would have been realized

absent the program and its interventions. Spillover (SO) is generally classified into participant

and non-participant SO. PSO occurs when participants take additional energy-saving actions

that are influenced by the program interventions but did not receive program support. NPSO is

the reduction in energy consumption and/or demand by non-participants because of the

influence of the program.

As part of this evaluation, the evaluation team derived both FR and P SO estimates from self-

reported information from telephone interviews with program participants. Using the survey

instrument developed for this evaluation, we interviewed program participants and asked them

a series of structured and open-ended questions about the influence of the program and its

various components on the decision to install energy efficient equipment.

NPSO was not assessed during this evaluation. From a program theory perspective, we do not

expect much NPSO associated with this program. In addition, NPSO studies are typically costly

and difficult to conduct. The small expected NPSO savings did not justify the cost.

Below we present the algorithm for estimation of the NTG ratio based on this participant

survey. The survey instrument can be found in Appendix G of this report.

1 Referred to as Small Business Energy Efficiency Program in the survey instrument


CECONY F-2


The goal of most incentive-based energy efficiency programs is to influence customer decision-

making regarding energy efficient improvements. Programs can do this by changing what

customers install, when they install it, and how much they install. In other words, programs

influence the efficiency, timing, and quantity of customers’ energy-using equipment installations.

The bulk of program savings is typically achieved by encouraging customers to install higher

efficiency equipment than they would have installed on their own. Programs may also

encourage early replacement of still functioning equipment that is less efficient, thus impacting

the timing of the installation, so that savings can be realized earlier. The incentive may also

make it more affordable for customers to install a greater number of high efficiency measures.

The FR algorithm outlined here combines estimates of each of these concepts:

Program influence on the efficiency level of the installed equipment (EI)

Program influence on the timing of the installation of high efficiency equipment (TI)

Program influence on the quantity of the high efficiency equipment installed (QI)

Each concept takes a value between 0 and 1. The values are expressed in FR terms, with 0

meaning no FR and 1 meaning full FR.

Efficiency (EI), timing (TI), and quantity (QI) of the installation are distinct avenues of program

influence. However, the timing of the installation and quantity of measures installed are

conditional on efficiency. The program can only realize timing savings if the customer would

have installed the efficient equipment on their own but the program caused the installation to

happen earlier. Similarly, savings due to a quantity increase can only happen if a customer who

was already installing some energy efficient measures chooses to install additional ones because

of the program.

Below is further detail on how each influence score was calculated, along with the survey

questions measuring each area of influence. We estimated FR at the measure level, which means

that, based on the project scope, respondents were asked about the decision-making process for

more than one measure if applicable. Some participants installed a variety of measures and

estimating FR for all of them was not feasible given the amount of time that would be required

during the interview. In these cases, we asked about three measures, selected based on their

rarity. As part of the FR estimation, we attempted to capture as much of the program savings as

possible. To meet this goal, we identified chain or multi-facility participants and asked them if

the decision to install each measure was a single decision across all of the facilities, or if each

facility underwent its own decision-making process. In cases where the decision-making

process is the same across multiple facilities, we applied the FR estimate to the savings resulting

from measure installation at all facilities.

Because FR was estimated at the measure level, certain measures, more specifically lower cost

lighting measures, call for a different line of questioning. As such, we present the FR approach

for them separately low cost non-lighting and higher cost lighting measures.


CECONY F-3

Low Cost Non-Lighting and Higher Cost Lighting and Non-Lighting Measures FR

This group of measures was comprised of faucet aerators, linear fluorescent lighting, and other

low-cost and higher cost lighting and non-lighting measures.

To calculate the FR rate for measures in this group, the evaluation team proposed multiplying

the estimates of efficiency (EI), timing (TI) and quantity (QI) because each is a distinct avenue of

program influence. This multiplication would only take place for participants who had a

moderate to high probability installing the high efficiency equipment on their own. 2 For other

participants where the probability of a high efficiency installation is low or none, the final FR

score would default to the EI score.

We believe that when the three concepts are measured as distinct yet conditional methods of

program influence, it is appropriate to combine them by using multiplication. Averaging or

using some other calculation method would over-estimate FR. As such, the formula to calculate

FR through the program influences can be expressed as:

IF EI 0.50, THEN FR1=EI *TI * QI


During the New York Department of Public Service (DPS) review process, the DPS expressed

concern that the multiplicative method could produce a FR estimate that was biased

downward. After extensive discussions with the DPS, CECONY, and other stakeholders, the

evaluation team agreed to multiply the efficiency (EI) and quantity (QI) scores, and then

average this product with the timing (TI) score. This averaging would only apply to cases when

the TI score is lower than the product of the EI and QI scores. In cases where the timing score

exceeds the product of EI and QI, the FR rate is based on just the product of the EI and QI

scores. This selective averaging is important so that the program is not penalized having a

smaller influence on the timing of the project than the efficiency and scope of the project.

IF EI 0.50 AND TI> (EI*QI), THEN FR2=EI*QI

IF EI 0.50 AND TI (EI*QI), THEN FR2=AVERAGE((EI*QI);TI)


As part of the analysis, the evaluation team developed two estimates of FR – one that follows

the multiplicative approach and another that follows the selective averaging approach.

Developing two FR estimates using different algorithms allowed us to compare the results and

better understand the sensitivities associated with each method.

The method for calculating the program influence component scores is described in detail

below.

2 We multiplied the EI, TI, and QI scores only in cases where the EI score was 0.5 or higher (50% probability of the

high efficiency installation happening absent the program)


CECONY F-4

Program Influence on Equipment Efficiency (EI)

As part of the Small Business Direct Install Program, customers can receive select measures

(such as faucet aerators) at no cost up to a certain dollar amount ($100). For other measures

(such as energy-efficient linear lighting), the program covers 70% of project costs.

For free and low-cost non-lighting measures (e.g., faucet aerators), the evaluation team

measured program influence on equipment efficiency level by asking participants to estimate

the likelihood of the high efficiency measure installation in the absence of the program.3

For linear fluorescent lighting and other higher cost lighting and non-lighting measures that

require contractor installations, in addition to asking about the “likelihood” of installation, we

also asked a battery of questions measuring, on a seven-point scale (1-7), the influence of each of

the following program components separately:

Program incentives

Audit recommendations

Marketing and outreach

Ease of participation/measure installation (i.e. not having to search for a contractor to

install the measure)

The survey did not measure the influence of individual program components for free and lower

cost non-lighting measures (e.g., faucet aerators), as we believe such questions will be difficult,

if not impossible, for respondents to answer, given that the measures are free, are installed as

part of the audit (or follow-up audit) visit, and can be installed without a contractor.

Opinions in the industry on the use of different length rating scales vary. However, there is

research providing evidence that seven-point scale yield more reliable and valid results than

longer or shorter scales.4

The evaluation team converted survey responses from the seven-point scale to a value between

0 and 1 using linear transformation. For example, a response of 3 on a seven-point scale would

become .33 or .66, depending on how the anchor points of the scale were defined to

respondents. We do not have any reason to believe that linear transformations yield results that

are less reliable or valid than if we were to use non-linear transformations of the scale

responses. The linear transformation approach also seems intuitive, given the use of the scalars.

We therefore chose to use it in our calculations.

The questions were converted, when needed, for the low rating to represent low FR (and

therefore high level of program influence). For example, the scale for the question asking about

the influence of program incentives (uses a seven-point scale where 1 means no influence and 7

3 At the beginning of the survey, we explain to survey respondents that by program we mean all components –

incentives, the audit, marketing and outreach.

4 Lozano, M., García-Cueto, E., Muñiz, J. 2008. “Effect of the Number of Response Categories on the Reliability and

Validity of Rating Scales.” European Journal of Research Methods for the Behavioral and Social Sciences 4(2): 73-79.


CECONY F-5

means a great deal of influence) was flipped (1 would mean a great deal of influence and 7

means no influence), whereas the scale for the question asking about the likelihood to install

same efficiency measure remained unchanged.

For lower cost non-lighting measures (e.g., faucet aerators.), the score for program influence on

efficiency level (EI) was based on the question asking about likelihood of measure installation

absent the program. For higher cost measures (e.g., linear lighting), the score for program

influence on efficiency level (EI) will represent the average value of two scores:

1. Likelihood to install the measure in the absence of the program

2. Influence of individual program components on the decision to install the measure

The score for the influence of individual program components (#2 above) consisted of the

minimum rating across the four program components (marketing, incentives, energy survey

recommendations, and ease of participation/installation). Participants may have been

influenced differently by each program component and this allowed for the program to claim

the credit for the most influential of its components on the respondent decision-making process.

The resulting program influence on equipment efficiency score equals a value between 0 and 1,

with 1 being no influence and 0 being maximum influence. This ensures that the responses are

consistent with the measurement of FR.

Survey Questions:

QEI1 – Likelihood to install high efficiency measure in the absence of the program

As part of the Small Business Energy Efficiency program, a program representative came to your

business and installed <MEASURE> [IF V_NFREE=0; READ “FOR FREE”]. [IF V_NFREE>0; READ:

THE PROGRAM ALSO COVERED MOST OF THE FAUCET AERATOR COST]. If it had not been

for the program, what is the likelihood that you would have installed any <MEASURE> within the next

two years? Please use a scale of 1 to 7, where 1 is not at all likely and 7 is very likely.

QEI2a – Influence of program incentives

Using a scale from 1 to 7, where 1 means “no influence” and 7 means “a great deal of influence”, how

influential was each of the following on your decision to have <MEASURE> installed in your business? –

Utility incentives

QEI2b – Influence of information and recommendations provided as the result of the audit



Information and recommendations provided as the result of the audit

QEI2c – Influence of program marketing and outreach



Information that you learned through program marketing and outreach


CECONY F-6

QEI2d – Influence of the ease of program participation/measure installation



Not having to search for a contractor to install the measure

The table below maps each of the above-listed questions to measure-specific questions in the

survey instrument.

Table 2-7. Mapping of Measure-Specific Questions Measuring Program Influence on Measure Efficiency to the Survey Instrument (Low Cost Non-Lighting and Higher Cost Lighting and Non-

Lighting Measures)

Measure QEI1 QEI2A QEI2B QEI2C QEI2D

Faucet

aerators

Table 1. FR4b Table 2. -- Table 3. -- Table 4. -- Table 5. --

Table 6. Other free

measures

Table 7. FR5b Table 8. -- Table 9. -- Table 10. -- Table 11. --

Table 12. Non-free

measures

Table 13. FR6b/FR6c

Table 14. FR6fa

Table 15. FR6fb

Table 16. FR6fd

Table 17. FR6fh

Calculation:

[FOR FAUCET AERATORS AND OTHER FREE NON-LIGHTING MEASURES]

EI=(QEI1-1)/6

[FOR LINEAR LIGHTING AND OTHER NON-FREE MEASURES]

EI=AVERAGE(EI1; EI2)

EI1=(QEI1-1)/6

EI2=MIN(EI2A, EI2B, EI2C, EI2D)

EI2A=1-((QEI2A-1)/6)

EI2B=1-((QEI2B-1)/6)

EI2C=1-((QEI2C-1)/6)

EI2D=1-((QEI2D-1)/6)

Program Influence on Equipment Quantity (QI)

Program influence on quantity is measured by asking participants if they would have

purchased/installed fewer measures without the program. The evaluation team calculated the

quantity score by dividing the quantity that would have been installed absent the program by

the verified program rebated quantity. The score takes a value between 0 and 1.


CECONY F-7

The quantity question is conditional on at least some probability of the high efficiency

installation taking place absent the program. When asking the question, we emphasized that

we are referring to the high efficiency purchase and only used the quantity score in the

calculation of FR when the efficiency score (EI) took a value of 0.5 or higher (i.e. moderate to

high probability of the high efficiency installation taking place absent the program.5

Survey Questions:

QQIA – Program influence on quantity

You received <MEASURE QUANTITY> <MEASURE> through the Small Business Energy

Efficiency Program. If you were to install them on your own, would you have installed the same

quantity or would you have installed fewer?6

QQIB –Quantity of measures installed (if fewer)

How many <MEASURE> would you have installed?

Calculation:

QI=1 IF QQIA=WOULD HAVE INSTALLED THE SAME QUANTITY

QI=QQIB/PROGRAM QUANTITY IF QQIA=WOULD HAVE INSTALLED FEWER

QI=1 IF QQIA=WOULD HAVE INSTALLED MORE

QI=0 IF QQIA=WOULD NOT HAVE INSTALLED ANY

The table below contains the matrix of the survey questions by measure.

Table 2-8. Mapping of Measure-Specific Questions Measuring Program Influence on Measure Quantity to the Survey Instrument (Low Cost Non-Lighting and Higher Cost Lighting and Non-

Lighting Measures)

Measure QQIA QQIB

Faucet aerators Table 18. FR4c Table 19. FR4d

Table 20. Other free measures Table 21. FR5c Table 22. FR5d

Table 23. Non-free measures Table 24. FR6d Table 25. FR6e

Program Influence on Equipment Timing (TI)

Program influence on timing is measured by asking participants if the installation would have

happened later in the absence of the program, and, if so, how much later, with the resulting

5 Note that, for the ease of survey administration, only respondents who rated their likelihood to install high

efficiency equipment as 1 (not at all likely) were skipped through the questions asking about the scope of the high

efficiency project.

6 Survey respondents also had an option to answer “more” or “would not have installed any.” These categories,

however, were not read to them, given the skips employed in the survey instrument.


CECONY F-8

score taking a value between 0 and 1, with one representing no program influence and

therefore maximum FR.

Respondents were first asked if the program had any influence on the timing of the

installation. Those who said that the program sped up equipment installation were asked

when they would have installed in the measure, based on four categories: within 3 months of

when they purchased, 3 to 6 months, 6 months to a year later, and more than 1 year later.

Similar to the quantity question, the timing question is conditional on at least some

probability of the high efficiency installation taking place absent the program. When asking

the question, we emphasized that we are referring to the high efficiency purchase and only

used the timing score in the calculation of FR when the efficiency score (EI) took a value of 0.5

or higher (i.e. high probability of the high efficiency installation taking place absent the

program.7

Initially, the evaluation team planned on assigning the same timing influence score to all

participants regardless of whether they were a part of a chain or franchise. Through the

discussions with the DPS, however, we decided that a longer time frame might be more

reflective of the chain or franchise decision-making process. The DPS-recommended timing

adjustment was to give the program full timing credit (EI=0) if it accelerated the installation

by six years or more, 0.33 credit (EI=0.66) if it accelerated the installation by four or five years,

and no credit (EI=1) for any other length of time. The DPS also encouraged the evaluation

team to analyze the chains or franchises on a case-by-case basis and use the non-chain/non-

franchise timing scores if the business did install equipment that was specified by the chain or

franchise or if the franchise has a short approval process for new equipment installations.

The discussions about revising the calculation of the timing score occurred after the survey

was fielded. As a result, we were somewhat limited in our ability to better understand the

decision-making process for chains and franchises in our sample. Furthermore, the timing

question employed the acceleration period of one year. As part of the analysis, we adjusted

the timing score for chains/franchises to 1 if the program accelerated the installation of high

efficiency equipment by a year or less. There were no cases where a chain or a franchise

participant mentioned that the program accelerated the installation of high efficiency

equipment by more than a year.8

7 Note that, for the ease of survey administration, only respondents who rated their likelihood to install high

efficiency equipment as 1 (not at all likely) were skipped through the questions asking about the timing of the high

efficiency installation.

8 We determined if the participating company is a chain or franchise by carefully reviewing program tracking data as

well as participant survey responses to questions S1a “Is your business a part of chain or franchise?”


CECONY F-9

Survey Questions:

If it had not been for the Small Business Energy Efficiency Program, when would you have installed the

<MEASURE>? Would you say within 3 months of when you did, 3 to 6 months later, 6 months to a year

later, or more than a year later?9

The table below contains the matrix of the survey questions by measure.

Table 2-9. Mapping of Measure-Specific Questions Measuring Program Influence on the Timing of Installation to the Survey Instrument (Low Cost Non-Lighting and Higher Cost Lighting and Non-

Lighting Measures)

Measure QTI

Faucet aerators Table 26. FR4e

Table 27. Other free measures Table 28. FR5e

Table 29. Non-free lighting measures

Table 30. FR6g

Calculation:

For Chains/Franchises

TI=1 IF QTI= WITHIN 3 MONTHS, 3 TO 6 MONTHS, 6 MONTHS TO A YEAR LATER, MORE

THAN A YEAR LATER, AT THE SAME TIME

TI=0 IF QTI=NEVER10

For Non-Chains/Non-Franchises

TI=1 IF QTI=WITHIN 3 MONTHS

TI=0.66 IF QTI=3 TO 6 MONTHS

TI=0.33 IF QTI=6 MONTHS TO A YEAR LATER

TI=0 IF QTI=MORE THAN A YEAR LATER

TI=1 IF QTI=AT THE SAME TIME

TI=0 IF QTI=NEVER11

The FR algorithms used in this evaluation are illustrated below in Figure 2-1 and Figure 2-2.

9 Survey respondents will also have an option to say “at the same time” or “never.” These categories, however, will

not be read to them, given the skips employed in the survey instrument.

10 Note that this response option was not read to respondents and was only recorded when respondents volunteered

it.

11 Note that this response option was not read to respondents and was only recorded when respondents volunteered

it.


CECONY F-10

Figure 2-1. FR Algorithm – Higher Cost Lighting and Non-Lighting Measures

Program Influence on Timing

(Timing of high efficiency

measure(s) adoption in

absence of the program)

(TI)

Program Influence on

Quantity

(Quantity of high efficiency

measures adopted in

absence of the program)

(QI)

Survey Questions

QEI1

FR6b or FR6c

(Possible Values: 0-1)

Calculation:

EI=(QEI1-1)/6

(Possible Values: 0, 0.33,

0.66, 1)(Possible Values: 0-1)

Survey Questions

QTI

FR6g

Calculation:

For Chains/Franchises:

IF QTI=1, 2, 3, 4, 5 TI=1

IF QTI=6 TI=0

For Non-Chains/Non-Franchises:

IF EI<0.5 TI=MISSING

IF QTI=1 TI=1

IFQTI=2 TI=0.66

IF QTI=3 TI=0.33

IF QTI=4 TI=0

IF QTI=5 TI=1

IF QTI=6 TI=0

Calculation:

IF EI<0.5 QI=MISSING

IF QQIA=1 QI=1

IF QQIA=2 QI=QQIB/PROGRAM

QUANTITY

IF QQIA=3 QI=1

IF QQIA=4 QI=0

Survey Questions

QQIA QQIB

FR6d FR6e

Likelihood to adopt

same efficiency

measure(s) in absence

of the program

(EI1)

Influence of program components on

adoption of same efficiency

measure(s) (incentives, audit

recommendations, marketing, easy

installation process)

EI2=MIN(EI2A, EI2B, EI2C, EI2D)

Survey Questions

QEI2A QEI2B QEI2C QEI2D

FR6fa FR6fb FR6fd FR6h


Calculation:

EI2A=1-((QEI2A-1)/6)

EI2B=1-((QEI2B-1)/6)

EI2C=1-((QEI2C-1)/6)

EI2D=1-((QEI2D-1)/6)

Explanation:

Score decreases if participants

would have delayed the

installation without the program

Explanation:

The lower the likelihood

to install measure(s) in

absence of the program,

the lower the score

Explanation:

Score decreases if

participants would have

installed fewer units without

the program

Explanation:

The higher the influence of program-

related factors, the lower the score

IF EI ≥0.50, THEN FR1=EI *TI * QI


IF EI≥0.50 AND TI> (EI*QI), THEN FR2=EI*QI

IF EI≥0.50 AND TI≤(EI*QI), THEN

FR2=AVERAGE((EI*QI);TI)


Program Influence on Efficiency (EI)

EI=AVERAGE(EI1;EI2)


CECONY F-11

Figure 2-2. FR Algorithm – Low-Cost Non-Lighting Measures

Low Cost Lighting Measures FR

This group of measures is comprised of CFLs, LEDs and LED exit signs.

Program Influence on Equipment Efficiency and Timing (EI and TI)

Due to the unique nature of the lighting measures as well as the unique nature of the direct

installation program, the influence of efficiency and timing was measured as a single

component for CFLs, LEDs, and LED exit sign measures. The SBDI program sends team

members into small business facilities to replace working incandescent lighting measures with

high efficiency options (CFLs, LEDs, or LED exit signs). The nature of the lighting measures is

such that customers generally replace them at burnout. Accordingly, we asked participants

whether they would have replaced their working non-energy efficient lights with high efficiency

options if it had not been for the program. Customers who say they would not have replaced

Program Influence on Efficiency (EI)

(Likelihood to adopt same efficiency

measure(s) in absence of the

program)

Explanation:

Score decreases if participants

would have delayed the installation

without the program

Explanation:

The lower the likelihood to install

measure(s) in absence of the program,

the lower the score

Program Influence on Timing (TI)

(Timing of high efficiency

measure(s) adoption in absence

of the program)

Program Influence on Quantity (QI)

(Quantity of high efficiency measures

adopted in absence of the program)

Survey Questions

QEI

FR4b

FR5b


Calculation:

EI=(QEI-1)/6

(Possible Values: 0-1) (Possible Values: 0-1)

Survey Questions

QTI

FR4e

FR5e

Explanation:

Score decreases if participants would

have installed fewer units without the

program

Calculation:

For Chains/Franchises:

IF QTI=1, 2, 3, 4, 5 TI=1

IF QTI=6 TI=0

For Non-Chains/Non-Franchises:

IF EI<0.5 TI=MISSING

IF QTI=1 TI=1

IFQTI=2 TI=0.66

IF QTI=3 TI=0.33

IF QTI=4 TI=0

IF QTI=5 TI=1

IF QTI=6 TI=0

Calculation:

IF EI<0.5 QI=MISSING

IF QQIA=1 QI=1

IF QQIA=2 QI=QQIB/PROGRAM

QUANTITY

IF QQIA=3 OR QTI2=1 QI=QQIC/

PROGRAM QUANTITY

IF QQIA=4 QI=0

Survey Questions

QQIA QQIB

FR4c FR4d

FR5c FR5d

Measure Type

Faucet aerators

Other measures

IF EI ≥0.50, THEN FR1=EI *TI * QI


IF EI≥0.50 AND TI> (EI*QI), THEN FR2=EI*QI

IF EI≥0.50 AND TI≤(EI*QI), THEN

FR2=AVERAGE((EI*QI);TI)



CECONY F-12

their working lights with high efficiency alternatives are asked if the replacement would have

happened at burnout. Both questions were yes or no questions.12

We incorporated a timing adjustment factor into the efficiency score, based on when

participants said they would have replaced working measures with high efficiency options on

their own. We asked participants who said they would have replaced their working measures if

they would have done so within three months of when the high efficiency equipment was

installed through the program. Though this is a short time frame, it is appropriate given the

program structure, the cost of the measures, and the expected incandescent light bulb life in a

commercial setting.13 Customers were using less efficient measures when the program visited

their business and installed high efficiency measures. If they truly had plans to install high

efficiency alternatives, it should happen within three months, which is the anticipated time of

the bulb burning out, or we have reason to believe they would not have done so on their own.

By stating that participants would not replace their working measures for more than three

months, the stated response is called into question. We effectively used this question as a

consistency check for those respondents and gave the program full credit for those high

efficiency installations. We gave the program partial credit for participants who said they

would have installed high efficiency measures at burnout.

Participants who said that they would not have replaced their working measures with high

efficiency options were asked if they would have done so at burnout. Respondents who said

yes were assigned the program partial timing credit resulting in a FR score of 0.75. The credit

of 0.75 is reasonable given that three months is the expected life of an incandescent bulb in a

commercial setting and represents a quarter of the program year.

The program was given full credit for participants who said they would not have replaced their

working measures with high efficiency options within three months of program installation and

would not have installed high efficiency measures at burnout.

12 Due to the relatively short expected life of these measures, we believe that asking respondents to rate their

likelihood of replacing the measures on a scale is needlessly nuanced and difficult to answer.

13 Mid-Atlantic TRM references average daily usage of 9.6 hours. This usage will result in the life of an incandescent

bulb of about three months assuming 1,000 incandescent bulb lifetime.


CECONY F-13

Survey Questions

QEITI1

As part of the Small Business Energy Efficiency program, a program representative came to your

business and replaced <V_MEASD_QUANT> working <MEASURE> with energy efficient

<CFLS/LEDS/LED EXIT SIGNS> [IF MEASD_NFREE=0; READ “FOR FREE”]. [IF

MEASD_NFREE>0; READ: THE PROGRAM ALSO COVERED MOST OF THE CFL COST]. If it

had not been for the program, would you have replaced ANY of these WORKING <MEASURE> with

<CFLS/LEDS/LED EXIT SIGNS>?

QEITI2 [ASKED IF QEITI1=NO]

If it had not been for the Small Business Energy Efficiency program, would you have replaced any of these

<V_MEASD_QUANT> <MEASURE> with <CFLS/LEDS/LED EXIT SIGNS> when your old bulbs

burnt out?

QEITI3 [ASKED IF QEITI1=YES]

Do you think you would have replaced the working <MEASURE> with <CFLS/LEDS/LED EXIT

SIGNS> within the next three months if you had done it outside of the program or would it have taken

you more than three months to do it?14

The table below maps each of the above-listed questions to measure-specific questions in the

survey instrument.

Table 2-10. Mapping of Measure-Specific Questions Measuring Program Influence on Measure Efficiency to the Survey Instrument (CFLs, LEDs, LED Exist Signs)

Measure QEITI1 QEITI2 QEITI3

CFLs Table 31. FR1b Table 32. FR1f Table 33. FR1e

Table 34. LED Table 35. FR2b Table 36. FR2f Table 37. FR2e

Table 38. LED Exit Signs Table 39. FR3b Table 40. FR3f Table 41. FR3e

Calculation

Program Influence on Equipment Quantity (QI)

Participants who said that, absent the program, they would have replaced their working lights

with high efficiency alternatives within three months or would have installed high efficiency

lights at burnout, were then asked questions about quantity to determine if program can receive

some credit for influencing the scope of the installation.

14 Note that the question includes option 3 “Would not have replaced <MEASURE> with <CFLS/LEDS/LED EXIT

SIGNS> at all” and 4 “Would have replaced at burnout.” Neither of these options was read to the respondents but

recorded in cases when respondents volunteered the response. The algorithm accounts for those options.


CECONY F-14

Survey Questions:

QQIA – Program influence on quantity

Would you have replaced all working <MEASURES> with <CFLS/LEDS/LED EXIT SIGNS> or would

you have replaced just some of these <MEASURES> if it had not been for the Small Business Energy

Efficiency Program?15

QQIB – Quantity of working bulbs replaced (if fewer)

How many working <MEASURES> would you have replaced with <CFLS/LEDS/LED EXIT SIGNS>?

QQIC – Quantity of bulbs replaced upon burnout

If it had not been for the Small Business Energy Efficiency program, would you have replaced any of these

<MEASURE QUANTITY> <MEASURE> with <CFLS/LEDS/LED EXIT SIGNS> when <MEASURE>

burned out?

Table 2-11. Mapping of Measure-Specific Questions Measuring Program Influence on Measure Quantity to the

Survey Instrument (CFLs, LEDs, LED Exist Signs)

Measure QQIA QQIB QQIC

CFLs Table 42. FR1c Table 43. FR1d Table 44. FR1g

Table 45. LED Table 46. FR2c Table 47. FR2d Table 48. FR2g

Table 49. LED Exit Signs Table 50. FR3c Table 51. FR3d Table 52. FR3g

Calculation:

QI=1 IF QQIA=WOULD HAVE INSTALLED THE SAME QUANTITY

QI=QQIB/VERIFIED PROGRAM QUANTITY IF QQIA=WOULD HAVE INSTALLED FEWER

QI=1 IF QQIA=WOULD HAVE INSTALLED MORE

QI=0 IF QQIA=WOULD NOT HAVE INSTALLED ANY

QI=QQIC/VERIFIED PROGRAM QUANTITY IF QQIA=WOULD HAVE INSTALLED NONE

OR QTI2=WOULD HAVE REPLACED UPON BURNOUT

The flow diagram below visually presents the calculation of the FR scores for the low-cost

lighting measures. As can be seen from the table, the final FR score for these measures is a

product of multiplying the efficiency and timing score with the quantity score. Because of the

unique nature of the program and the measures, the efficiency and timing score was calculated

jointly eliminating the need for a selective averaging algorithm.

15 Survey respondents also had an option to answer “none.” This category, however, were not read to them, given

the skips employed in the survey instrument.


CECONY F-15

Figure 2-3. FR Scoring Algorithm for Free and Low-Cost Measures

Consistency Checks

The scoring algorithm relies on responses from multiple questions to triangulate FR rate.

Because respondents can sometimes give inconsistent answers, the survey instrument included

consistency checks to clarify these responses.

As part of the data analysis, we carefully studied those responses and adjusted either FR scores

or individual component scores accordingly.

Survey Question:

Just to make sure I understand, please explain the importance of the Small Business Energy Efficiency

Program on your decision to install <MEASURE> as opposed to less-efficient <MEASURE>.

1.1.2 Spillover

PSO represents additional savings (expressed as a percent of total program savings) that were

achieved without program rebates but would not have happened in the absence of the program.

PSO was assessed through interviews with participating customers by asking about non-program

efficiency actions that participants took as a result of participating in the program. The actions

could have taken place at the same facility that received the program-funded upgrades or at

another site. The survey instrument contained checks to ensure consistency of response.

Eff

icie

nc

y a

nd

Tim

ing

Sc

ore

(EIT

I)

QEITI1. Would have replaced ANY WORKING MEASURES with HIGH EFFICIENCY MEASURES absent the program?

Yes No

QEITI3. Would have replaced within three months or later than three months

absent the program?

Within three monthsLater than three months/would

not have replaced at all

QETI2. Would have replaced at burnout absent

the program?

Yes No

EITI=1 EITI=0

Qu

an

tity

Sc

ore

(QI)

QQIA Would have replaced all or some absent the program?

All Some

QI=1 QI=QQIB/VERIFIED QTY

QQIC. How many would have

replaced absent the

program?

Free-Ridership Score

FR=EITI*QI

QQIB. How many would have replaced

absent the program?

QI=QQIC/VERIFIED QTY

Would have replaced

at burnout*

*Response was not read but recorded if respondent volunteered it.

EITI=0.75EITI=0


CECONY F-16

While PSO can result from a variety of measures, survey length did not allow for estimation of

PSO across all possible scenarios. Given the types of businesses that participated in the

program, the evaluation team included measures that could reasonably be expected to be

influenced by program participation and are more likely to have been implemented without

program support. PSO was measured for the following equipment:

Lighting equipment

Cooling equipment

Refrigeration equipment

Kitchen equipment

Motors

Heating and water-heating equipment

Neither NPSO nor vendor off-site SO was in the scope of this evaluation.

Survey Questions:

SP1a SINCE you completed the improvements through the Small Business Energy Efficiency Program,

did you install any additional ENERGY EFFICIENT equipment at THIS facility that did NOT

receive incentives through <UTILITY>?

SP1b And, since you completed the improvements through the Small Business Energy Efficiency

Program, did you install any additional ENERGY EFFICIENT equipment at OTHER facilities

in <UTILITY>’s service territory that did NOT receive incentives through <UTILITY>?

[IF SP1A OR SP1B=YES]

SP1c Just to confirm, you made energy efficient improvements AFTER you participated in the Small

Business Energy Efficiency Program, is that correct?

[IF SP1A OR SP1B=YES]

SP1d Just to confirm, you made energy efficient improvements that did NOT receive incentives

through the Small Business Energy Efficiency Program, is that correct?

Those who did were asked about specific improvements they made.

Survey Questions:

SP2A/SP3A/SP4A/SP5A/QP6A/SP7A/SP8A

Did you install any energy efficient <ENDUSE> equipment without receiving an incentive from

<UTILITY>?

For each of the end-use categories, respondents were asked to explain why they purchased the

equipment without applying for incentives, as well as provide a rating of the program influence

on their decision to make additional improvements.


CECONY F-17

Survey Questions:

SP2B/SP3B/SP4B/SP5B/QP6B/SP7B/SP8B

Why did you purchase this <ENDUSE> equipment without applying for incentives through

<UTILITY>?

SP2C/SP3C/SP4C/SP5C/QP6C/SP7C/SP8C

How much did your experience with <UTILITY>’s Small Business Energy Efficiency Program influence

your decision to install the additional energy efficient <ENDUSE> equipment that you ended up

installing? Please use a scale from 1 to 7 where 1 means no influence and 7 means a great deal of

influence.

If any energy efficient improvements were heavily influenced by the program16, respondents

were asked in an open-ended fashion to explain how the Small Business Direct Install Program

influenced their decision to install the energy efficient equipment.

Survey Questions:

SP2M/SP3M/SP4M/SP5M/QP6M/SP7M/SP8M

How did the Small Business Energy Efficiency Program influence your decision to install the additional

energy efficient lighting equipment that you ended up installing?

Respondents were also asked a few equipment specific questions that allowed the calculation of

savings associated with the installed equipment. The equipment details were limited by the

survey length as well as by what we believe respondents could reliably answer.

As part of the SO calculation, the evaluation team applied savings values to the measures

installed outside of the program. We estimated savings for each measure using the most recent

NYTM values supplemented by engineering assumptions. We determined the program-level

SO factor by dividing the estimated savings of the measures installed by survey respondents

outside of the program (but influenced by the program) by the savings the survey respondents

realized through the program.

As indicated earlier, SO is included in the overall NTGR as (1 - FR + SO).

16A rating of 6 or 7 on a scale of 1 to 7 where 1 is no influence and 7 is a great deal of influence.


CECONY F-18

Figure 2-4. PSO Diagram

Data Imputations

Respondents can sometimes provide “don’t know” responses to one or more questions that are

critical to FR estimation. Sometimes, respondents can simply refuse to answer some questions.

This leads to item non-response. To overcome any biases associated with item non-response, we

imputed data based on the responses of that other similar participants gave. As part of the

analysis we performed the following imputations. All of the imputation decisions were driven

by the differences in results observed during analysis among the subgroups of interest.

In cases where respondents could not provide a valid answer to the timing question (TI),

we assigned measure-based savings-weighted TI average based on other responses. This

imputation was performed for a total of 20 measures units across 17 unique participants.

In cases where respondents could not provide a valid answer to the quantity question

(QI), we assigned measure-based savings-weighted QI average based on other responses.

This imputation was performed for a total of 58 measure units across 49 unique

participants.

Participant Spillover (SO)

Presence of action taken outside of the program

(SP1a; SP1b; SP1c; Sp1d)

Explanation of program influence

(Open ended question)

(SP2m) (SP3m) (SP4m) (SP5m) (SP6m) (SP7m) (SP8m)

Lighting

(SP2a)

Cooling

(SP3a)

Refrigeration

(SP4a)

>5 rating

Yes

Kitchen

Equipment

(SP5a)

Motors

(SP6a)

Heating and Water

Heating

(SP7a)

Other

(SP8a)

Yes Yes Yes Yes Yes Yes

Yes Yes Yes Yes Yes Yes Yes

Reason(s) for installing equipment without the incentive

(Open ended question)

(SP2b) (SP3b) (SP4b) (SP5b) (SP6b) (SP7b) (SP8b)

Degree of program influence on

enduse purchase/installation

(scale of 1 to 7, where 1 is no influence and 7 is a great deal of influence)

(SP2c) (SP3c) (SP4c) (SP5c) (SP6c) (SP7c) (SP8c)

>5 rating >5 rating >5 rating >5 rating >5 rating >5 rating

Enduse specific questions to quantify energy savings resulting from spillover

(SP2d-SP2h) (SP3d-SP3h) (SP4d) (SP5d) (SP6d-SP6h) (SP7d-SP7h) (SP8d)


CECONY F-19

For low-cost lighting measures (CFLs, LEDs, and LED exit signs), the imputations

were further refined to distinguish between those who said they would have

replaced WORKING light bulbs with high efficiency options vs. those who said they

would have done it AT BURNOUT.

(For low cost lighting measures only) In cases where respondents could not provide an

answer to the question whether they would have replaced working light bulbs with high

efficiency options, we assigned measure-based savings-weighted FR average across two

subgroups – those who said that they had high efficiency lighting at their business and

those who did not. This imputation was performed for a total of 11 measure units across

11 unique participants.


CECONY G-1

APPENDIX G PARTICIPANT SURVEY INSTRUMENT

The following is the customer survey instrument that will be used by ODC for the participant

surveys.

Con Edison and Orange & Rockland Small Business Direct Install Program Participant Net-

To-Gross Survey

Sample Variables

<PROGRAM CONTACT> (Name of the contact person)

<COMPANY> (Name of participating company)

<ADDRESS> (Facility address)

<MEASURE> (Measure description)

<AUDIT> (Energy survey, energy evaluation)

<UTILITY> (ConEd/O&R)

<FRANCHISE> (Is firm a franchise?)

INTRODUCTION

Hi, may I please speak with <PROGRAM CONTACT>?

My name is _____, and I’m calling from Opinion Dynamics, an independent research company, on

behalf of <UTILITY>. This is not a sales call.

[IF <PROGRAM CONTACT>=MISSING]

May I please speak to the person at your company most knowledgeable about your company’s

energy using equipment?

Our records show that <COMPANY> participated in <UTILITY>’s Small Business Energy

Efficiency Program in 2011 at [IF ADDRESS IS NONMISISNG: <ADDRESS>]. You may also

know this program as <UTILITY>’s [IF UTILITY=CONED: “GREEN TEAM”; IF

UTILITY=O&R: “LIGHTEN UP”] program. We are calling to do a follow-up study about

<COMPANY>’s participation in this program. I was told that you are the person most

knowledgeable about this project. Is this correct? [IF NOT, ASK TO BE TRANSFERRED TO

MOST KNOWLEDGEABLE PERSON OR RECORD NAME & NUMBER]

[IF NEEDED: THIS SURVEY WILL TAKE ABOUT 15 MINUTES.]


CECONY G-2

C1. Are you currently talking to me on a regular landline phone or a cell phone?

1. Regular landline phone

2. Cell Phone

8. (Don’t know)

9. (Refused)

[ASK IF C1 = 2; ELSE GO TO SURVEY START]

C2. Are you currently in a place where you can talk safely and answer my questions?

1. Yes

2. No [SCHEDULE CALL BACK]

8. (Don’t know) [SCHEDULE CALL BACK]

9. (Refused) [SCHEDULE CALL BACK

INSTALLATION VERIFICATION

To start with, I would like to confirm some information in <UTILITY>’s program tracking

database.

Energy Survey/Energy Evaluation V1a Our records show that <COMPANY> had a free <AUDIT> performed at <ADDRESS>

through <UTILITY>’s Small Business Energy Efficiency Program. Is that correct?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF V1<>1]

V1b Is it possible that someone else dealt with this project?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[IF V1B=1, ASK TO BE TRANSFERRED TO THAT PERSON OR RECORD NAME &

NUMBER]

[IF V1B=2,8,9, THANK AND TERMINATE. RECORD DISPO AS “COULD NOT

CONFIRM PARTICIPATION”]


CECONY G-3

Free Measures

[REPEAT THE LOOP FOR ALL FREE MEASURES]

[ASK IF FREE_MEASD1 IS NOT SYSMIS, ELSE SKIP TO V3]

V2a Our records indicate that as part of the <AUDIT> you received <QTY>

<FREE_MEASD1> at <ADDRESS> at no cost to you, is that correct?

1 Yes

2 (Yes, but different quantity)

3 No

8 (Don’t know)

9 (Refused)

[ASK IF V2A=2]

V2b How many <FREE_MEASD1> did you receive at no cost to you? [NUMERIC OPEN

END]

[FOR EACH MEASURE CALCULATE V_FREE_MEASD AND V_FREE_MEASD_QUANT]

[ASK IF V_FREE_MEASD_QUANT>0]

V2c And how many of <V_FREE_MEASD_QUANT> <V_FREE_MEASD> are currently

installed? [NUMERIC OPEN END]

[ASK IF V2C IS LESS THAN <V_FREE_MEASD_QUANT>]

V2d Why are not ALL <V_FREE_MEASD> installed? [OPEN END]

Non-Free Measures [REPEAT THE LOOP FOR ALL NON-FREE MEASURES]

[ASK IF NFREE_MEASD1 IS NOT SYSMIS, ELSE SKIP TO S1]

V3a Our records indicate that you received an incentive from <UTILITY> for <QTY>

<NFREE_MEASD1> installed at <ADDRESS>, is that correct?

1 Yes

2 (Yes, but different quantity)

3 No

8 (Don’t know)

9 (Refused)

[ASK IF V3A=2]

V3b How many <NFREE_MEASD1> did you receive an incentive from <UTILITY> for?

[NUMERIC OPEN END]


CECONY G-4

[FOR EACH MEASURE CALCULATE V_NFREE_MEASD AND

V_NFREE_MEASD_QUANT]

V3c And how many of <V_NFREE_MEASD_QUANT> <V_NFREE_MEASD > are

currently installed? [NUMERIC OPEN END]

[ASK IF V3C IS LESS THAN <V_NFREE_MEASD_QUANT>]

V3d Why are not ALL <V_NFREE_MEASD> installed? [OPEN END]

SCREENING QUESTIONS

S1 Is your business in the private sector, the public sector, or is it a non-profit organization?

1 Private

2 Public

3 Non-profit

8 (Don’t know)

9 (Refused)

[ASK IF S1=1]

S1a Is your business part of a chain or franchise?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF S1A=1]

S2 Does <COMPANY> take corporate policy or guidelines into account when making

purchasing decisions about energy-using equipment?

1 Yes

2 No

8 (Don’t know)

9 (Refused)


CECONY G-5

PROGRAM MARKETING AND INTERACTIONS

A1 How did you first learn about the <UTILITY>’s Small Business Energy Efficiency

Program?

01 (In-person/door-to-door outreach)

02 (<UTILITY> website)

03 (Direct mail)

04 (Email)

05 (Word of mouth)

06 (Chambers of Commerce)

07 (Business Improvement District Outreach and Communication)

08 (Newspaper advertising)

09 (Radio advertising)

10 (TV advertising)

00 (Other, specify)

98 (Don’t know)

99 (Refused)

A2 You might have mentioned it already, but I just wanted to confirm. Did you hear about the

<UTILITY>’s Small Business Energy Efficiency Program from any of the following

sources?

a. [SKIP IF A1=6] Chamber of Commerce communications

b. In-person outreach by Business Improvement District representatives

c. In-person outreach by <UTILITY> representatives

d. Business Improvement District meetings or speaking engagements

e. [SKIP IF A1=2] <UTILITY> website

f. [SKIP IF A1=8] Newspaper advertising

g. [SKIP IF A1=9] Radio advertising

[COMPUTE MARKETING_FLAG=1 IF A1=2,3,4,6,7,8 OR ANY IN A2A THROUGH

A2F=1]

[ASK IF V_NFREE_MEASD=T8 OR T5]

A6 Before today, were you aware of recent legislation that set higher energy standards for linear

fluorescent lighting that will effectively remove T12 lamps and ballasts from the market?

1 Yes

2 No

8 (Don’t know)

9 (Refused)


CECONY G-6

As you probably remember, the Small Business Energy Efficiency Program included a free

<AUDIT> of the facility, a report with recommendations for energy efficiency improvements, free

installation of the energy saving equipment, as well as incentives toward the purchase and

installation of additional energy saving equipment. For the remainder of the survey, please think of

all these components when I refer to the Small Business Energy Efficiency Program.

[ASK IS V_MEASD=CFL]

FREE-RIDERSHIP – CFLS

For the next set of questions, please think about the CFLs that you received through the Small

Business Energy Efficiency program.

FR1b As part of the Small Business Energy Efficiency program, a program representative came to

your business and replaced <V_MEASD_QUANT> working incandescent light bulbs with

energy efficient CFLs [IF MEASD_NFREE=0; READ “FOR FREE”]. [IF

MEASD_NFREE>0; READ: THE PROGRAM ALSO COVERED MOST OF THE

CFL COST]. If it had not been for the program, would you have replaced ANY of these

WORKING light bulbs with CFLs?

[NOTE TO INTERVIEWER: IF RESPONDENT SAYS WOULD HAVE INSTALLED

CFLS WHEN THE OLD LIGHT BULBS BURNT OUT, RECORD AS “NO”]

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF FR1B=1]

FR1c Would you have replaced all <V_MEASD_QUANT> working light bulbs with CFLs or

would you have replaced just some of these light bulbs if it had not been for the Small

Business Energy Efficiency program?

1 All

2 Some

3 (None)

8 (Don’t know)

9 (Refused)

[ASK IF FR1C=2]

FR1d How many working light bulbs would you have replaced with CFLs? [NUMERIC OPEN

END, 998=DON'T KNOW; 999=REFUSED]


CECONY G-7

FR1e Do you think you would have replaced the working incandescent light bulbs with CFLs

within the next three months if you had done it outside of the program or would it have

taken you more than three months to do it?

1 Within three months

2 Later than three months

3 (Would not have replaced light bulbs with CFLs at all)

4 (Would have replaced them when the light bulbs were to burn out)

8 (Don’t know)

9 (Refused)

[ASK IF FR1B=2 OR FR1C=3]

FR1f If it had not been for the Small Business Energy Efficiency program, would you have

replaced any of these <V_MEASD_QUANT> light bulbs with CFLs when your old bulbs

burnt out?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[SKIP IF FR1F=1]

FR1g How many of the <V_MEASD_QUANT> light bulbs would you have replaced with CFLs

when they burnt out? [NUMERIC OPEN END, 998=DON'T KNOW; 999=REFUSED]

FR1h Before the Small Business Energy Efficiency program installed CFLs at your business, did

you have any CFLs in any of your business’s light sockets or in storage?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IS V_ MEASD=LED]

FREE-RIDERSHIP – LEDS

For the next set of questions, please think about the LED light bulbs that you received through the

Small Business Energy Efficiency program.


your business and replaced <V_MEASD_QUANT> working incandescent light bulbs with

energy efficient LEDs [IF MEASD_NFREE=0; READ “FOR FREE”]. [IF

MEASD_NFREE>0; READ: THE PROGRAM ALSO COVERED MOST OF THE

LED COST]. If it had not been for the program, would you have replaced ANY of these

WORKING light bulbs with LEDs?


LEDS WHEN THE OLD LIGHT BULBS BURNT OUT, RECORD AS “NO”]

1 Yes


CECONY G-8

2 No

8 (Don’t know)

9 (Refused)

[ASK IF FR2B=1]

FR2c Would you have replaced all <V_MEASD_QUANT> working light bulbs with LEDs or

would you have replaced just some of these light bulbs if it had not been for the Small


1 All

2 Some

3 (None)

8 (Don’t know)

9 (Refused)

[ASK IF FR2C=2]

FR2d How many working light bulbs would you have replaced with LEDs? [NUMERIC OPEN


FR2e Do you think you would have replaced the working incandescent light bulbs with LEDs

within the next three months if you had done it outside of the program or would it have

taken you more than three months to do it?



3 (Would not have replaced light bulbs with LEDs at all)


8 (Don’t know)

9 (Refused)



replaced any of these <V_MEASD_QUANT> light bulbs with LEDs when your old bulbs

burnt out?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[SKIP IF FR2F=1]

FR2g How many of the <V_ MEASD_QUANT> light bulbs would you have replaced with

LEDs when they burnt out? [NUMERIC OPEN END, 998=DON'T KNOW;

999=REFUSED]


CECONY G-9

FR2h Before the Small Business Energy Efficiency program installed LEDs at your business, did

you have any LEDs in any of your business’s light sockets or in storage?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IS V_ MEASD=LED EXIT SIGN]

FREE-RIDERSHIP – LED EXIT SIGNS

For the next set of questions, please think about the LED exit signs that you received through the



your business and replaced <V_MEASD_QUANT> working exit signs with LED exit

signs [IF MEASD_NFREE=0; READ “FOR FREE”]. [IF MEASD_NFREE>0; READ:

THE PROGRAM ALSO COVERED MOST OF THE EXIT SIGN COST]. If it had not

been for the program, would you have replaced ANY of these WORKING exit signs with

LED exit signs?


LED BULBS WHEN THE OLD LIGHT BULBS BURNT OUT, RECORD AS “NO”]

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF FR3B=1]

FR3c Would you have replaced all <V_MEASD_QUANT> working exit signs with LED exit

signs or would you have replaced just some of these signs if it had not been for the Small


1 All

2 Some

3 (None)

8 (Don’t know)

9 (Refused)

[ASK IF FR3C=2]

FR3d How many working signs would you have replaced with LED signs? [NUMERIC OPEN



CECONY G-10

FR3e Do you think you would have replaced the working exit signs with LED exit signs within

the next three months if you had done it outside of the program or would it have taken you

more than three months to do it?



3 (Would not have replaced exit signs with LED exit signs at all)


8 (Don’t know)

9 (Refused)



replaced any of these <V_MEASD_QUANT> exit signs with LED exit signs when the

bulbs in the existing exit signs burnt out?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[SKIP IF FR3F=1]

FR3g How many of the <V_MEASD_QUANT> exit signs would you have replaced with LED

technology when they burnt out? [NUMERIC OPEN END, 998=DON'T KNOW;

999=REFUSED]

FR3h Before the Small Business Energy Efficiency program installed LED exit signs at your

business, did you have any LED exit signs in use at your business?

1 Yes

2 No

8 (Don’t know)

9 (Refused)


CECONY G-11

[ASK IS V_ MEASD=AERATOR]

FREE-RIDERSHIP – AERATORS

For the next set of questions, please think about the faucet aerators that you received through the



your business and installed faucet aerators [IF MEASD_NFREE=0; READ “FOR

FREE”]. [IF MEASD_NFREE>0; READ: THE PROGRAM ALSO COVERED MOST

OF THE FAUCET AERATOR COST]. If it had not been for the program, what is the

likelihood that you would have installed any faucet aerators within the next two years? Please

use a scale of 1 to 7, where 1 is not at all likely and 7 is very likely. [RECORD 1 TO 7;

8=DON'T KNOW; 9=REFUSED]

[ASK IF V_MEASD_QUANT>1 AND FR4B>1]

FR4c You received <V_MEASD_QUANT> faucet aerators through the Small Business Energy

Efficiency Program. If you were to install them on your own, would you have installed the

same quantity or would you have installed fewer?

1 Same quantity

2 Fewer

3 (More)

4 (Would not have installed any)

8 (Don’t know)

9 (Refused)

[ASK IF FR4C=2]

FR4d How many faucet aerators would you have installed? [NUMERIC OPEN END,



FR4e If it had not been for the Small Business Energy Efficiency Program, when would you have

installed faucet aerators? Would you say…?

1 Within 3 months of when you did

2 3 to 6 months later

3 6 months to a year later

4 or more than a year later

5 (At the same time)

6 (Never)

8 (Don't know)

9 (Refused)

FR4f Before the Small Business Energy Efficiency program installed faucet aerators at your

business, did you have any faucet aerators in use at your business?

1 Yes

2 No


CECONY G-12

8 (Don’t know)

9 (Refused)

[ASK IF MEASD_NFREE=0 AND V_ MEASD<>AERATOR OR CFL OR LED EXIT SIGN]

FREE-RIDERSHIP – OTHER FREE MEASURES

For the next set of questions, please think about the <V_MEASD> that you received through the



your business and installed <V_MEASD> for free. If it had not been for the program, what

is the likelihood that you would have installed any <V_MEASD> within the next two

years? Please use a scale of 1 to 7, where 1 is not at all likely and 7 is very likely. [RECORD

1 TO 7; 8=DON'T KNOW; 9=REFUSED]

[ASK IF V_ MEASD_QUANT>1 AND FR5B>1]

FR5c You received <V_ MEASD_QUANT> <V_MEASD> for free through the Small Business

Energy Efficiency Program. If you were to install them on your own, would you have

installed the same quantity or would you have installed fewer?

1 Same quantity

2 Fewer

3 (More)


8 (Don’t know)

9 (Refused)

[ASK IF FR5C=2]

FR5d How many <V_MEASD> would you have installed? [NUMERIC OPEN END,



FR5e If it had not been for the Small Business Energy Efficiency program when would you have

installed <V_MEASD>? Would you say..






6 (Never)

8 (Don't know)

9 (Refused)


CECONY G-13

FR5h Before the Small Business Energy Efficiency program installed <V_FREE_MEASD> at

your business, did you have any <V_FREE_MEASD> installed at your business

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF MEASD_NFREE=1]

FREE-RIDERSHIP – NON-FREE MEASURES

I now have a few questions about your decision to make <V_MEASD> improvements through

<UTILITY>’s Small Business Energy Efficiency Program for which you received the incentive

from <UTILITY>.

There is a variety of ways in which the Small Business Energy Efficiency Program might have

influenced your decision to have <V_MEASD> installed in your business. Using a scale from 1 to

7, where 1 means “no influence” and 7 means “a great deal of influence,” how influential was each

of the following?

FR6fa <UTILITY> incentives

FR6fb Information and recommendations provided as the result of the <AUDIT>

FR6fd [ASK IF MARKETING_FLAG=1] Information that you learned through program

marketing and outreach

FR6fh [ASK OF MEASURES REQUIRING CONTRACTOR INSTALLATION] Not

having to search for a contractor to install the <V_ MEASD>

Now using the same scale, please rate the influence of the following on your decision to have

<V_NFREE_MEASD> installed in your business? [REPEAT SCALE IF NEEDED]

FR6fc [ASK IF S2=1] Corporate policy and guidelines

FR6ff [ASK IF V_MEASD=T8 OR T5 AND A6=1] The federal legislation that will

affect the sale of T12 lamps

[ASK IF V_ MEASD=OCCUPANCY SENSORS, REFRIGERATION DOORS,

REFRIGERATION CURTAINS, REFRIGERATED CASE COVERS]


your business and installed <V_MEASD>. The program also covered most of the cost of

the project. What is the likelihood that you would have installed any <V_MEASD> on

your own within the next two years if it had not been for the program? Please use a scale of

1 to 7, where 1 is not at all likely and 7 is very likely. [RECORD 1 TO 7; 8=DON'T

KNOW; 9=REFUSED]

[ASK IF V_ MEASD=OTHER LIGHTING, TUBE LIGHTING]

FR6c As part of the Small Business Energy Efficiency program, a program representative came to

your business and installed <V_MEASD>. The program also covered most of the cost of

the project. What is the likelihood that you would have installed the SAME EFFICIENCY


CECONY G-14

lighting equipment on your own within the next year if it had not been for the program?

Please use a 1 to 7 point scale, where 1 is “Not at all likely” and 7 is “very likely.” (IF

NEEDED: Scale is from 1 to 7, where 1 is “Not at all likely” and 7 is “very likely”)

[RECORD 1 TO 7; 98=DON'T KNOW; 99=REFUSED]

[ASK IF V_ MEASD_QUANT>1 AND FR6B>1 OR FR6C>1]

FR6d You received <V_MEASD_QUANT> <V_ MEASD> through the Small Business Energy

Efficiency Program. If you were to install them on your own, would you have installed the

same quantity or would you have installed fewer?

1 Same quantity

2 Fewer

3 (More)


8 (Don’t know)

9 (Refused)

[ASK IF FR6C=2]

FR6e How many <V_MEASD> would you have installed? [NUMERIC OPEN END,


[ASK IF V_ MEASD_QUANT>0 AND FR5B>1 OR FR5C>1]

FR6g If it had not been for the Small Business Energy Efficiency program, when would you have

installed the <V_MEASD>? Would you say..






6 (Never)

8 (Don't know)

9 (Refused)

[ASK IF FR6G=1,2,3]

FR6ga. As you know, <UTILITY>’s Small Business Energy Efficiency program paid for 70% of

the total project cost. You just told me that in the absence of the program you would have

installed the <V_MEASD> <IF FR6G=1, READ “WITHIN THREE MONTHS OF

WHEN YOU DID”; IF FR6G=2, READ “3 TO 6 MONTHS LATER OF WHEN YOU

DID”; IF FR6G=3, READ “6 MONTHS TO A YEAR LATER OF WHEN YOU

DID”>. At the time that you participated in the program, did you already have money set

aside to pay the FULL project cost?


CECONY G-15

1 Yes

2 No

3 (Could pay for some of the project cost)

8 (Don’t know)

9 (Refused)

[ASK IF ANY IN FR6FA, B, D>5 AND FR6B OR FR5C>5]

[ASK IF ANY IN FR6FA, B, D <3 AND FR6B OR FR5C<3]

FR6i Just to make sure I understand, please explain the importance of the Small Business Energy

Efficiency Program on your decision to install the [IF <V_MEASD> IS OCCUPANCY

SENSORS, REFRIGERATION CURTAINS AND REFRIGERATION CASE

COVERS, <V_MEASD> , ELSE READ “HIGH EFFICIENCY <V_MEASD>

INSTEAD OF LESS EFFICIENT EQUIPMENT”]

00 [OPEN END]

98 (Don’t know)

99 (Refused)

FR6j Before the Small Business Energy Efficiency program installed [IF <V_MEASD> IS

OCCUPANCY SENSORS, REFRIGERATION CURTAINS AND REFRIGERATION

CASE COVERS, <V_MEASD> , ELSE READ “HIGH EFFICIENCY <V_ MEASD>”]

at your business, had you ever installed <V_NFREE_MEASD> at your business?

1 Yes

2 No

8 (Don’t know)

9 (Refused)


CECONY G-16

ADDITIONAL PROJECTS

[ASK AP1 OR AP2 AS PART OF THE FREE-RIDERSHIP MEASURE LOOP IF THEY

INSTALLED THE SAME MEASURE AT DIFFERENT FACILITIES]

Our records show that <COMPANY> also received an incentive from <UTILITY> for

<V_SMEASD_QUANT> other <V_SMEASD> project(s).


AP1 Our records show that <COMPANY> had the <AUDIT> done at <NSAME> other

facilities. Was it a single decision have audits done across all of those facilities or did each

facility go through its own decision making process?

01 (Single decision)

02 (Each facility went through its own decision process)

00 (Other, specify)

98 (Don’t know)

99 (Refused)


AP2 Our records show that <COMPANY> also received an incentive from <UTILITY> for

<V_SMEASD_QUANT> other <V_SMEASD> project(s). Was it a single decision to

complete all of those <MSAME> projects for which you received an incentive from

<UTILITY> or did each project go through its own decision process?

01 (Single decision)

02 (Each project went through its own decision process)

00 (Other, specify)

98 (Don’t know)

99 (Refused)


CECONY G-17

CHANGES TO BUSINESS OPERATIONS

Thank you for discussing the improvements that you installed through <UTILITY>’s Small

Business Energy Efficiency Program. I now have a few questions about other changes you may have

made to your business.

C1 Did you make any changes to your business before or after you participated in the Small

Business Energy Efficiency Program that increased or decreased your energy usage? Such

changes might include, among other things, expansion or reduction in space, an increase or

decrease in machinery, or a change in operating hours.

1 Yes

2 No

8 (Don't know)

9 (Refused)

[IF C1<>1, SKIP TO SPILLOVER SECTION]

C2 And did this change or changes occur before, after, or at about the same time as you

participated in the Small Business Energy Efficiency Program? [MULTIPLE RESPONSE.

ACCEPT UP TO THREE]

1 Before

2 At the same time

3 After

8 (Don’t know)

9 (Refused)

[ASK IF C2=1]

C3 Please describe the change or changes that happened before you participated in the Small

Business Energy Efficiency Program. [OPEN END]

[ASK IF C2=1]

C4 And, did this change(s) result in an increase or decrease in energy usage?

1 Increase

2 Decrease

3 (Other, specify)

8 (Don’t know)

9 (Refused)

[ASK IF C2=2]

C5 Please describe the change or changes that happened at the time of your participation in the

Small Business Energy Efficiency Program. [OPEN END]


CECONY G-18

[ASK IF C2=2]


1 Increase

2 Decrease

3 (Other, specify)

8 (Don’t know)

9 (Refused)

[ASK IF C2=3]

C7 Please describe the change or changes that happened after you participated in the Small

Business Energy Efficiency Program. [OPEN END]


CECONY G-19

[ASK IF C2=3]


1 Increase

2 Decrease

3 (Other, specify)

8 (Don’t know)

9 (Refused)

SPILLOVER

Next, I would like to discuss any energy efficient equipment you might have installed AFTER

participating in the Small Business Energy Efficiency Program that did not receive incentives from

<UTILITY>.

SP1a SINCE you completed the improvements through the Small Business Energy Efficiency

Program, did you install any additional ENERGY EFFICIENT equipment at THIS facility

that did NOT receive incentives through <UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)

SP1b And, since you completed the improvements through the Small Business Energy Efficiency

Program, did you install any additional ENERGY EFFICIENT equipment at OTHER

facilities in <UTILITY>’s service territory that did NOT receive incentives through

<UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)


CECONY G-20

[ASK IF SP1A=1 OR SP1B=1]

SP1c Just to confirm, you made energy efficient improvements AFTER you participated in the

Small Business Energy Efficiency Program, is that correct?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF SP1A=1 OR SP1B=1]

SP1d Just to confirm, you made energy efficient improvements that did NOT receive incentives

through the Small Business Energy Efficiency Program, is that correct?

1 Yes

2 No

8 (Don’t know)

9 (Refused)

[ASK IF SP1C=1 OR SP1D=1, ELSE SKIP TO END]

Lighting

SP2a Did you install any energy efficient lighting equipment without getting an incentive from

<UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)

[ASK IF SP2a=1, ELSE SKIP TO SP3A]

SP2b Why did you purchase this lighting equipment without applying for incentives through

<UTILITY>?

01 (Did not know how to get incentive after declining originally)

02 (No time to participate/needed equipment immediately)

03 (No incentive was offered)

04 (The amount of the incentive was not large enough)

00 (Other, specify)

96 (I did get an incentive)

98 (Don’t know)

99 (Refused)

[ASK IF SP2B<>96, 3, ELSE SKIP TO SP3A]


CECONY G-21

SP2c How much did your experience with <UTILITY>’s Small Business Energy Efficiency

Program influence your decision to install the energy efficient lighting equipment that you

ended up installing afterward? Please use a scale from 1 to 7 where 1 means no influence and

7 means a great deal of influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]

[ASK IF SP2C>5, ELSE SKIP TO SP3A]

SP2m How did the Small Business Energy Efficiency Program influence your decision to install

the energy efficient lighting equipment that you ended up installing afterward? [OPEN

END]

SP2d What types of energy efficient lighting equipment did you install without getting an

incentive through <UTILITY>? [MULTIPLE RESPONSE. ACCEPT UP TO THREE]

01 (Linear fluorescent lights – T8)

02 (Linear fluorescent lights – T5)

03 (High-Intensity Discharge or HID Fixtures)

04 (Compact fluorescent lamps or CFLs)

05 (LED light fixtures)

06 (LED exit signs)

07 (Halogen lighting)

08 (Lighting Occupancy Sensors)

09 (Daylighting Controls)

96 (Did not install any equipment) [SKIP TO SP3A]

00 (Other, specify)

98 (Don't know)

99 (Refused)

[ASK ABOUT FIRST MEASURE; REPEAT SP2E-SP2H FOR EACH MEASURE

MENTIONED. DO NOT LOOP IF SP2D=00, AND SKIP TO SP3A]

SP2e How many [READ IN RESPONSE FROM SP2D] did you install? [NUMERIC OPEN

END: 1-5000; 9998=DON’T KNOW; 9999=REFUSED]

[SKIP IF SP2D=6 OR 8 OR 9]

SP2f What is the average wattage of the [READ IN RESPONSE FROM SP2D] you installed?

[NUMERIC OPEN END: 1-5000; 9998=DON’T KOW; 9999=REFUSED]

SP2g What equipment did these [READ IN RESPONSE FROM SP2D] replace? [OPEN END]

[SKIP IF SP2D=6 OR 8 OR 9]

SP2h And were these [READ IN RESPONSE FROM SP2D] installed inside, outside, or in a

refrigerated space?

01 Inside (in heated or cooled space)

02 Outside (in non-heated and non-cooled space)


CECONY G-22

03 Refrigerated space (in a cooler or freezer)

00 (Other, specify)

98 (Don't know)

99 (Refused)

Cooling SP3a Did you install any energy efficient cooling equipment without getting an incentive from

<UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)


SP3b Why did you purchase this cooling equipment without getting an incentive through

<UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)



Program influence your decision to install the energy efficient cooling equipment that you

ended up installing afterward? Please use a scale from 1 to 7 where 1 means no influence and

7 means great influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]



the energy efficient cooling equipment that you ended up installing? [OPEN END]


CECONY G-23

SP3d What types of energy efficient cooling equipment did you install without getting an


01 (Unitary/Split Air Conditioning System)

02 (Heat Pump)

03 (Chiller)

04 (Compressor)

05 (Condenser)

06 (Cooling Tower)

07 (Air Handler)

08 (Room air conditioners)

09 (Packaged Terminal Air Conditioners or PTAC units)

00 (Other, specify)

96 (Didn’t install any equipment) [SKIP TO SP4A]

98 (Don't know)

99 (Refused)

SP3e How many tons of cooling does this new equipment provide? [NUMERIC OPEN END;

1-500; 996=NOT APPLICABLE, 998=DON’T KNOW, 999=REFUSED]

[ASK SC4c IF SP3e=998,999]

SP3f Approximately, how many square feet of floor space does this cooling equipment serve?

[NUMERIC OPEN END; 1-500,000; 999,996=NOT APPLICABLE; 999,998=DON’T

KNOW; 999,999=REFUSED]

SP3g Did this new equipment replace old equipment?

1 Yes

2 No

8 (Don't know)

9 (Refused)

[SKIP TO SP4A IF SP3G=2,8,9]

SP3h How old was the replaced equipment?

1 (0-4 years)

2 (5-9 years)

3 (10-14 years)

4 (15-19 years)

5 (20 years or older)

8 (Don’t know)

9 (Refused)


CECONY G-24

Refrigeration SP4a Did you install any refrigeration equipment without getting an incentive from

<UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)


SP4b Why did you purchase refrigeration equipment without getting an incentive through

<UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)



Program influence your decision to install refrigeration equipment that you ended up

installing afterward? Please use a scale from 1 to 7 where 1 means no influence and 7 means

great influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]



the energy efficient refrigeration equipment that you ended up installing afterward? [OPEN

END]

SP4d What type of refrigeration equipment did you install without getting an incentive through

<UTILITY>? [MULTIPLE RESPONSE. ACCEPT UP TO THREE]

01 (Strip curtains)

02 (Anti-sweat controls)

03 (EC motor for cooler or freezer)

04 (Solid door cooler or freezer)

05 (Glass door cooler or freezer)

06 (Case cooler or freezer)


CECONY G-25

07 (Condenser)

08 (LED case lights with motion controls)

09 (Door gaskets)

96 (Didn’t install any equipment)

98 (Don't know)

99 (Refused)

Kitchen Equipment

SP5a Did you install any energy efficient kitchen or vending equipment without getting an

incentive from <UTILITY>, such as vending machine controls, fryers, griddles,

commercial ovens, pre-rinse spray valves or hot food holding cabinets?

1 Yes

2 No

8 (Don't know)

9 (Refused)


SP5b Why did you purchase energy efficient kitchen equipment without getting an incentive

through <UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)



Program influence your decision to install energy efficient kitchen equipment that you ended

up installing afterward? Please use a scale from 1 to 7 where 1 means no influence and 7

means great influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]



the energy efficient kitchen equipment that you ended up installing afterward? [OPEN

END]


CECONY G-26

SP5d What types of energy efficient kitchen equipment did you install without getting an


01 (Electric steamers)

02 (Convection ovens)

03 (Electric griddles)

04 (Electric fryers)

05 (Pre-rinse spray valves)

06 (Food holding cabinets)

07 (Snack vending machine controls)

08 (Refrigerated cooler controls)

00 (Other, specify)

96 (Didn’t install any equipment)

98 (Don't know)

99 (Refused)

Motors SP6a Did you install any energy efficient motors without getting an incentive from <UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)


SP6b Why did you purchase energy efficient motors without getting an incentive through

<UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)



Program influence your decision to install energy efficient motors that you ended up

installing afterward? Please use a scale from 1 to 7 where 1 means no influence and 7 means

great influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]


CECONY G-27



the energy efficient motors that you ended up installing afterward? [OPEN END]

SP6d In what types of equipment did you install the energy efficient motors? [MULTIPLE

RESPONSE. ACCEPT UP TO THREE]

01 (HVAC system fans)

02 (HVAC system pumps)

03 (non-HVAC system fans)

04 (non-HVAC system pumps)

00 (Other, specify)

96 (Didn’t install any motors) [SKIP TO SP7A]

98 (Don't know)

99 (Refused)

SP6e How many motors did you install? [NUMERIC OPEN END, 1 TO 500; 998=DON’T

KNOW, 999=REFUSED]

SP6f What was the total horsepower of these motors? [NUMERIC OPEN END, 1 TO 5000;

9998=DON’T KNOW, 9999=REFUSED]

SP6g Did the new motors also have new variable frequency drives (VFDs)?

1 Yes

2 No

8 (Don't know)

9 (Refused)

SP6h Did this new equipment replace old equipment?

1 Yes

2 No

8 (Don't know)

9 (Refused)


CECONY G-28

Heating and Water Heating SP7a Did you install any energy efficient heating or water heating equipment without getting an

incentive from <UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)


SP7b Why did you purchase this heating or water heating equipment without getting an incentive

through <UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)



Program influence your decision to install the energy efficient heating or water heating

equipment that you ended up installing afterward? Please use a scale from 1 to 7 where 1

means no influence and 7 means great influence. [SCALE 1-7; 98=DON’T KNOW;

99=REFUSED]



the energy efficient heating or water heating equipment that you ended up installing

afterward? [OPEN END]


CECONY G-29

SP7d What types of energy efficient heating or water heating equipment did you install without

getting an incentive through <UTILITY>? [MULTIPLE RESPONSE. ACCEPT UP TO

THREE]

01 (Gas furnace)

02 (Electric furnace)

03 (Gas boiler)

04 (Electric boiler)

05 (Electric heat pump)

06 (Tankless water heater)

07 (Solar water heater)

08 (Gas water heater)

09 (Electric water heater)

10 (Geothermal water heater)

00 (Other, specify)

96 (Didn’t implement any heating measures) [SKIP TO SP8A]

98 (Don't know)

99 (Refused)

[ASK ABOUT FIRST MEASURE; REPEAT SP7E-SP7H FOR EACH MEASURE

MENTIONED. DO NOT LOOP IF SP7D>6 AND <11, AND SKIP TO SP8A]

SP7E Approximately, how many square feet of floor space does this heating equipment serve?

[NUMERIC OPEN END; 1-500,000; 999,996=NOT APPLICABLE; 999,998=DON’T

KNOW; 999,999=REFUSED]

SP7F Did this new equipment replace old equipment?

1 Yes

2 No

8 (Don't know)

9 (Refused)

[SKIP TO SP8A IF SP7F=2,8,9]

SP7g What type of old equipment was replaced?

01 (Gas furnace)

02 (Gas boiler)

03 (Electric resistance heater)

04 (Electric heat pump)

00 (Other, specify)

98 (Don't know)

99 (Refused)


CECONY G-30

SP7h How old was the replaced equipment?

1 (0-4 years)

2 (5-9 years)

3 (10-14 years)

4 (15-19 years)

5 (20 years or older)

8 (Don’t know)

9 (Refused)

Other Equipment

SP8a Did you install any other energy efficient equipment without getting an incentive from

<UTILITY>?

1 Yes

2 No

8 (Don't know)

9 (Refused)

[ASK IF SP8A=1, ELSE SKIP TO END]

SP8b Why did you purchase this energy efficient equipment without getting an incentive through

<UTILITY>?





00 (Other, specify)


98 (Don’t know)

99 (Refused)

[ASK IF SP8B<>96, 3, ELSE SKIP TO END]


Program influence your decision to install this energy efficient equipment that you ended up

installing? Please use a scale from 1 to 7 where 1 means no influence and 7 means great

influence. [SCALE 1-7; 98=DON’T KNOW; 99=REFUSED]

[ASK IF SP8C>5]

SP8d What type or types of energy efficient equipment did you install? [PROMPT RESPONSE

CATEGORIES IF NEEDED]

00 (Other, specify)

98 (Don't know)


CECONY G-31

99 (Refused)

These are all the questions that I have for you. Thank you very much for your time. It is greatly

appreciated. Have a nice day.


CECONY H-1

APPENDIX H: PARTICIPANT SURVEY FIELDING AND FINAL SAMPLE DISPOSITION

We fielded the survey with the SBDI program participants from January 28 through February

26, 2013. We completed a total of 448 interviews. We used a Computer-Assisted Telephone

Interviewing (CATI) system to conduct interviews with the participants whose phone number

is associated with up to three facilities. We conducted in-depth interviews with participants

whose phone numbers are associated with more than three facilities in order to capture as much

information as possible about the installed equipment and the decision making process.

To minimize the measurement error, we tested the survey internally for comprehension, as well

as pre-tested the survey with several participants to ensure that survey questions are

interpreted correctly and answered in a consistent manner.

Table H-1 shows the final survey dispositions.


CECONY H-2

Table H-1 SBDI Program Participant Survey Dispositions

Table H-2 below provides the response and cooperation rates. The survey response rate is the

number of completed interviews divided by the total number of potentially eligible respondents

in the sample. We calculated the response rate using the standards and formulas set forth by the

American Association for Public Opinion Research (AAPOR).1

We also calculated a cooperation rate, which is the number of completed interviews divided by

the total number of eligible sample units actually contacted. In essence, the cooperation rate

1 Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, AAPOR, 2011.

http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2&Template=/CM/ContentDisplay.cfm&Cont

entID=3156

Disposition N

Completed Interviews 448

Partially Completed Interviews 38

Eligible Non-Interviews 2,937

Refusals 1,210

Break off 116

Telephone answering device 537

Respondent never available 1,070

Language Problem 4

Not Eligible 1,000

Fax/Data Line 89

Non-Working 443

Wrong Number 373

Business/Government 13

No Eligible Respondent 71

Duplicate Number 11

Unknown Eligibility Non-Interview 1,073

Not dialed/worked 285

No Answer 762

Busy 21

Call Blocking 5

Total Participants in Sample 5,496


CECONY H-3

gives the percentage of participants who completed an interview out of all of the participants

with whom we actually spoke. We used AAPOR Cooperation Rate 1 (COOP1).

Table H-2 SBDI Program Participant Survey Response and Cooperation Rates

There are multiple sources of non-sampling error that can impact survey results, including non-

response error and resulting coverage bias. This type of bias is usually overcome through

comparing and, if needed, weighting the survey results to the observable characteristics

(generally demographic or household) in the population of customers targeted by the survey

effort. Since the demographic composition of the participant population is unknown to us and

may have inherent differences from the overall customer population, we could not estimate or

correct for the non-response bias. However, we tried to mitigate the non-response bias through

the fielding process by undertaking the following steps:

Calling participants multiple times at varying times of the day and week.

Extending the fielding process over a period of time to allow us to “work” the sample.

AAPOR Rate Percentage

Response Rate (RR3) 10%

Cooperation Rate 25%


CECONY I-1

APPENDIX I: GLOSSARY OF TERMS

business type – Aggregated NYTM facility types based on similar lighting hours of use.

census stratum – In a stratified sample design, the stratum with those participants with the

largest savings may have a calculated sample size which exceeds the population of the

stratum. A stratum which meets this condition is referred to as a census stratum.

coefficient of variation (CV) – A normalized measure of dispersion of a probability distribution

and defined as the ratio of the standard deviation, to the mean, :

CECONY or Con Edison (in some tables) – Consolidated Edison Company of New York.

DPS - New York Department of Public Service.

error ratio – In energy efficiency evaluation, the error ratio is a measure of the degree of

variance between the reported savings estimates and the evaluated estimates. For a sample,

the error ratio is:

√∑

∑

∑

where n is the sample size, wi is the population expansion weight associated with each

sample point i, xi is the program reported savings for each sample point i, yi is the evaluated

gross savings for each sample point i, error for each sample point ei = yi - bxi, and ɤ= 0.8.

free rider, free ridership (FR) – A program participant who would have implemented the

program measure or practice in the absence of the program. Free ridership is the overall

percentage of savings from all customers in the population that would have implemented

the program measure or practice in the absence of the program.

Energy Efficiency Portfolio Standard (EEPS) – The state-mandated utility-administered

programs.

ex ante savings estimate – Forecasted savings used for program and portfolio planning

purposes.


CECONY I-2

ex post savings estimate – Savings estimate reported by an evaluator after the energy impact

evaluation has been completed.

HVAC – Heating, ventilation and air conditioning.

interval meter – An electric utility meter that measures and stores energy use and demand in

15-minute intervals. Interval meters are required for New York customers to participate in

Independent System Operator demand response programs.

measurement and verification (M&V) – A subset of program impact evaluation that is

associated with the documentation of energy savings at individual sites or projects using

one or more methods that can involve measurements, engineering calculations, statistical

analyses, and/or computer simulation modeling.

National Action Plan for Energy Efficiency – Model energy efficiency program impact

evaluation guide abbreviated as NAPEE. This is the DPS-recommended reference guide for

impact evaluations.

New York Technical Manual (NYTM) – The DPS-mandated reference document for

calculating EEPS program savings.

net to gross, net-to-gross ratio (NTG, NTGR) – The relationship between net energy or net

demand savings, where net is measured as what would have occurred without the program,

what would have occurred naturally, and gross savings (often evaluated savings). The

NTGR is the ratio of net savings to gross savings.

O&R – Orange & Rockland Utilities.

pooled, fixed-effect model – The pooled, fixed-effect model addresses both the cross-sectional

(multiple sites) and the time-series (multiple months) aspects of consumption data in a

single regression model. For billing analysis, the pooled-fixed effects model estimates

average site change in consumption from the pre- to post-installation periods. The model

controls for weather, unobserved site-level characteristics, and a systemic time trend. The

approach has similarities to site-level billing analysis approaches (e.g., PriSM) but, because

it controls for trend over time, does not require a comparison group.

relative precision – Measures the expected error bound of an estimate on a normalized basis. It

must be expressed for a specified confidence level. The relative precision (rp) of an estimate

at 90% confidence is:

√


CECONY I-3

where n is the sample size, N is the population size, and the coefficient of variance is cv =

standard deviation/estimate mean value. The square root expression at the end of the

equation is the finite population correction factor, which becomes inconsequential and

unnecessary for large populations.

realization rate – The term is used in several contexts in the development of reported program

savings. The primary applications include the ratio of project tracking system savings data

(e.g., initial estimates of project savings) to savings that (1) are adjusted for data errors and

(2) incorporate evaluated or verified results of the tracked savings. In the Updated

Guidelines, the RR does not include program attribution.

savings realization rate (RR) – The ratio of the field evaluated energy savings to the program’s

claimed savings. The RR represents the percentage of program-estimated savings that the

impact evaluation team estimates as being actually achieved based on the results of the

evaluation M&V analysis.

self-reported approach (SRA) – A method for gathering program attribution data through

direct interviews with participants

snapback – Snapback occurs when customers actually increase their energy consumption due

to reductions in the cost of energy

spillover (SO) – Includes participant spillover (PSO) and non-participant spillover (NPSO) –

Additional savings in energy consumption and/or demand caused by the presence of the

energy efficiency program, beyond program-related gross savings of participants.

PSO occurs when additional actions similar to those for which incentives were provided are

taken to reduce energy use at the same site, but these actions are not part of the measures

paid for or included as program savings.

NPSO are additional energy savings reduction in energy consumption and/or demand from

measures installed and actions taken or encouraged at non-participating customer by non-

participating vendors or contractors because of the influence of the program.

spot measurement – This value represents the instantaneous reading at the time of the

measurement.

standard deviation – Standard deviation is a measure of the spread or dispersion of a set of

data. It is calculated by taking the square root of the variance and is symbolized by sd, or s.


CECONY I-4

stratified ratio estimator (SRE) – An efficient sampling design combining stratified sample

design with a ratio estimator. It’s most advantageous when the population has a large

coefficient of variation. (A large coefficient of variation occurs, for example, when a

substantial portion of the projects have small savings and a small number of projects have

very large savings.) The ratio estimator uses supporting information for each unit of the

population when this information is highly correlated with the desired estimate to be

derived from the evaluation, such as the tracking savings and the evaluated savings.

The RR calculation for electric energy is shown below:

where is the savings RR, is the evaluation M&V kWh savings (by evaluation

M&V contractor), and is the kWh savings claimed by program.

summer coincident peak demand period – Defined by NYSERDA as the hours between noon

and 6 p.m. on non-holiday weekdays during June, July, and August. NYSERDA reports the

demand savings as the average reduction in electric demand during the summer coincident

peak demand period.

within-site sampling – When the quantity of fixtures to be evaluated is large, a sample of the

installed fixtures targeting ±20% relative precision at 80% confidence is drawn to conduct

metering.


CECONY 1

APPENDIX J: INDIVIDUAL SITE SUMMARIES

SiteID: Site‐003

Overall Savings Ex‐Ante Ex‐Post RR

Annual energy savings (kWh/yr) 30,657 27,271 89%

Summer peak demand savings (kW) 3.5 3.8 108%

Interactive heating penalty (therms/yr) N/A 98 N/A

Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy 0% 0 0%

Technology discrepancy 0% (53) 0%

Hours discrepancy ‐13% (3,859) ‐14%

Interactive effects discrepancy 3% 820 3%

Administrative/errors discrepancy 0% 0 0%

Total discrepancy (1‐RR) N/A N/A ‐11%

Facility Observations Ex‐Ante Ex‐Post NYTM

Building type Auto Related Auto Related

Pre‐retrofit fixture quantity 33 33

Post‐retrofit fixture quantity 33 33

Pre‐retrofit connected kW 5.7 5.7

Post‐retrofit connected kW 2.2 2.2

Pre‐retrofit operating hours 8,737 7,720

Post‐retrofit operating hours 8,736 7,858 4,056

Average coincidence factor 1.00 1.00 1.00

Average kWh interactive factor 0.00 0.02 0.12

Average kW interactive factor 0.00 0.08 0.20

M&V Details Values

Number of light loggers installed 3

Discussion

This site is a taxi repair shop with a garage area, small office, and a small break room. The project consisted of

replacing T12 fluorescent fixtures with T8 fluorescent fixtures. The garage area was not heated and cooled.

The small office and break area did have heating and cooling. Evaluators found that the number of replaced

fixtures matched the application. The difference in savings came from fewer operating hours and slightly

higher interactive savings. The operating hours determined from metering were lower than the hours used in

the tracking savings.

SiteID: Site‐005





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy 0% 0 0%


Interactive effects discrepancy ‐6% (276) ‐5%




Building type Office Office










M&V Details Values


Discussion

This site is a small insurance/investment office. The project consisted of replacing T12 fluorescent fixtures

with T8 fluorescent fixtures. The space was both heated and cooled. Evaluators found that the number of

replaced fixtures matched the application. The difference in savings came from fewer operating hours and

interactive savings. The operating hours determined from metering were lower than the hours used in the

tracking savings.

SiteID: Site‐007





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Store Store










M&V Details Values


Discussion

This site is a home improvement store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. Evaluators found that the number of replaced fixtures matched those found in the

application. The area is cooled, but not heated. There is a ‐2% discrepancy in interactive effects. The

operating hours determined from metering was lower than the hours used in the tracking savings for nan

adjusted discrepancy of ‐14%. The annual energy savings realization rate is 43% for the site.

SiteID: Site‐008





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy ‐4% (1,058) ‐3%


Interactive effects discrepancy ‐12% (3,399) ‐11%




Building type Auditorium Auditorium










M&V Details Values


Discussion

This site is an auditorium. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. Evaluators found there to be one fewer fixture in the

auditorium than found in the tracking analysis resulting in a 1% discrepancy. The operating hours determined

from metering was lower than the hours used in the tracking savings accounting for a ‐52% discrepancy.

Interactive savings were overestimated for an adjusted discrepancy of ‐11%. The realization rate for the

annual energy savings is 34% for the site.

SiteID: Site‐010





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy



Hours discrepancy ‐3% (529) ‐97%

Interactive effects discrepancy 11% 1,917 353%




Building type Hair Salon Hair Salon










M&V Details Values


Discussion

This site is a hair salon. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and incandescents with CFLs. The space was both heated and cooled. Evaluators found that the number and

type of replaced fixtures matched those found in the application. The tracking analysis rated wattage for

fixtures did not match the actual rated wattage of the fixture resulting in a ‐8% technology discrepancy. The

interactive effects of the were underestimated for the site resulting in a 11% discrepancy. The operational

hours of the site was less than estimated resulting in a ‐3% discrepancy. The overall annual energy savings

realization rate is 99% of the savings found in the tracking analysis.

SiteID: Site‐011





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy ‐7% (646) ‐5%















Average kW interactive factor 0.20 #DIV/0! 0.20

M&V Details Values


Discussion

This site is a convenience store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. The space was both heated and cooled. Evaluators found that the incandescent 60W

lamps in the sales area were not replaced The difference accounted for a ‐5% technology discrepancy. All

other fixtures were installed and quantities matched. The logged operational hours was less than those

found in tracking resulting in a ‐8% discrepancy. Interactive effects were overestimated as well resulting in a ‐

13% discrepancy. The overall annual energy savings for the site had a 74% realization rate.

SiteID: Site‐012





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Dry cleaner Dry cleaner










M&V Details Values


Discussion

This site is a dry cleaner. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with CFLs. The space did not have heating and cooling resulting in a ‐11%

discrepancy from the tracking. Evaluators found that the number of replaced fixtures matched the

application. The operating hours determined from metering was lower than the hours used in the tracking

savings accounting for the ‐23% hours discrepancy. The overall annual energy savings realization rate was

67% of the energy savings estimated in the tracking analysis.

SiteID: Site‐013





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Waiting Room Waiting Room










M&V Details Values


Discussion

This site is a doctor's waiting room. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures and an exit sign incandescent with an exit sign LED. The space was natural gas heated and

had electric cooling. Evaluators found that the installed fixtures did not match those in tracking with four U‐

Tube fixtures installed and 48" T8s installed. This accounted for a quantity and technology discrepancy of 9%

and 2%, respectively. The operation hours were lower than estimated netting a ‐70% discrepancy in hours.

The annual energy savings had a realization rate of 40%.

SiteID: Site‐014





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy ‐27% (969) ‐11%

















M&V Details Values


Discussion

This site is a small dental office. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. The space was both heated and cooled. Evaluators found that the number of replaced

fixtures was fewer than indicated in the tracking data ‐three fewer fixtures were found on‐site. The operating

hours determined from metering was lower than the hours used in the tracking savings. The difference in

savings came from fewer operating hours and and fewer fixtures found on‐site by the evaluators. There were

not any fixtures installed in the restroom.

SiteID: Site‐017





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures.

The space was both heated and cooled. Evaluators found that the number and type of replaced fixtures

matched those in the application. However, the rated wattage of the exit sign LEDs and 48" 32W T8

fluorescents in the tracking did not match the actual rated wattage of the fixtures resulting in a ‐10%

technology discrepancy. Hours and interactive savings were overestimated in the tracking analysis resulting

in a ‐50% and ‐6% discrepancy in hours and interactive effects, respectively. The energy savings realization

rate for the site was 34%.

SiteID: Site‐018




Interactive heating penalty (therms/yr) N/A 1,589 N/A

Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy



Hours discrepancy 13% 10,684 13%



Total discrepancy (1‐RR) N/A N/A 13%


Building type auto dealershipauto dealership










M&V Details Values


Discussion

This site is an auto sales showroom. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. The space was both heated and cooled. Evaluators found one fewer 48" 2 lamp

fluorescent fixture in the sales area than found in the tracking analysis. The operating hours determined from

metering was greater than the hours used in the tracking savings. resulting in a 13% hours discrepancy. The

realization rate for the site's annual energy savings is 113%.

SiteID: Site‐019





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a shoe store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with fluorescent fixture. Evaluators found that the number of replaced fixtures

matched those in the application. One of the incandescent fixtures were not replaced by fluorescent fixtures

accounting for the ‐3% technology discrepancy. The operating hours determined from metering was lower

than the hours used in the tracking savings resulting in a ‐49% hours discrepancy. The interactive effects

reduction was due to an overestimation in heating and cooling savings resulting in a ‐1% interactive effects

discrepancy. The annual energy savings realization rate is 37% for the site.

SiteID: Site‐021





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Laundromat Laundromat










M&V Details Values


Discussion

This site is a laundromat. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. Evaluators found that the number and type of replaced

fixtures matched those in the application. However, the rated wattage of the 32W T8 in tracking did not

match the actual rated wattage of the fixture resulting in a ‐11% technology discrepancy. The operating hours

determined from metering was lower than the hours used in the tracking savings resulting in a ‐40% hours

discrepancy. The annual energy savings realization rate for the site is 48%

SiteID: Site‐024





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy ‐8% (88) 20%

Hours discrepancy 19% 220 ‐49%

Interactive effects discrepancy ‐11% (124) 28%














M&V Details Values


Discussion

This site is a furniture repair store. The project consisted of replacing a T12 fluorescent fixture with a T8

fluorescent fixture. The space was not cooled, but was heated. Interactive effects were overestimated

resulting in a ‐11% interactive effects discrepancy. Evaluators found that the number and type of replaced

fixtures matched those in the application. The technology difference was due to the tracking analysis having

a rated wattage different from the fixture's actual rated wattage resulting in a ‐8% adjusted discrepancy. The

operating hours determined from metering was greater than the hours used in the tracking savings causing a

19% hours discrepancy. The overall annual energy savings realization rate was 98% for the site.

SiteID: Site‐025





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy 4% 3,172 4%















Average kWh interactive factor (0.26) 0.00 0.12

Average kW interactive factor 0.20 #DIV/0! 0.20

M&V Details Values


Discussion

This site is a department store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. Evaluators found that the number of replaced fixtures was

fewer than listed in the tracking analysis causing a 4% discrepancy. The operating hours determined from

metering was lower than the hours used in the tracking savings accounting for a ‐8% discrepancy. Interactive

effects were underestimated for an adjusted discrepancy of 33%. the overall annual energy savings

realization rate for the site was 128%.

SiteID: Site‐026





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy 21% 53,263 ‐42%

Hours discrepancy ‐6% (15,697) 12%

Interactive effects discrepancy ‐14% (34,896) 28%




Building type Parking GargaeParking Garage










M&V Details Values


Discussion

This site is a parking garage. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. Evaluators found the quantity and fixture types matched those found in the application. However,

the rated wattage of the fixture did not match the actual wattage of the fixture installed resulting in a 21%

adjusted discrepancy. The operational hours and interactive effects for the site were overestimated in the

tracking analysis resulting in a ‐6% and ‐14% discrepancy in savings, respectively. The overall annual energy

savings realization rate for the site is 98%.

SiteID: Site‐29





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and exit sign incandescents with exit sign LEDs. The space was both heated and cooled. Evaluators found that

the number of replaced fixtures matched the application. The operating hours determined from metering

was 5% lower than the hours used in the tracking savings. There is also a discrepancy of ‐2% for interactive

effects. The difference in savings came from fewer operating hours and interactive savings. The final

realization rate for this site is 94%.

SiteID: Site‐30





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Retail Store Retail Store










M&V Details Values


Discussion

This site is a pharmacy/retail store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures and incandescents with CFLs. The space was both heated and cooled. Evaluators found

that the number of replaced fixtures did not match those found in the application resulting in a 2% quantity

discrepancy. The operating hours determined from metering was lower than the hours used in the tracking

savings accounting for a ‐27% discrepancy. There is also a ‐2% interactive effects discrepancy. The final


SiteID: Site‐33





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a company office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and mercury vapor and metal halides to fluorescents. The space was both heated and cooled via a

packaged rooftop unit. Additional HVAC savings resulted in a 4% discrepancy in the interactive effects.

Evaluators found that the number of replaced fixtures matched the application. There are some slight

discrepancies between the tracking and evaluated post‐retrofit lighting fixtures resulting in a ‐1% discrepancy.

The operating hours determined from metering was significantly lower than the hours used in the tracking

savings resulting in a ‐68% discrepancy. The final realization rate for this site is 34%.

SiteID: Site‐038





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy



Hours discrepancy 0% 257 1%





Building type Auto‐related Auto‐related










M&V Details Values


Discussion

This site is a parking garage that operates 24/7. The project consisted of replacing T12 fluorescent fixtures

with T8 fluorescent fixtures and replacing incandescent exit signs with LEDs. The space was not heated and

cooled, except for the small office space in the garage. Evaluators found that the number of replaced fixtures

matched the application. Metered hours of operation were very close to the hours indicated in the tracking

data.

SiteID: Site‐040





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Retail Retail










M&V Details Values


Discussion

This site is a retail hardware store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures and incandescent exit signs with LEDs. The space was both heated and cooled. Evaluators

found that the number of replaced fixtures matched the application. The difference in savings came from

greater operating hours. The operating hours determined from metering were approximately 50% greater

than the hours used in the tracking savings.

SiteID: Site‐042





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. Evaluators found that the number of replaced fixtures

matched the application. The difference in savings came from fewer operating hours. The operating hours

determined from metering were slighty lower than the hours used in the tracking savings.

SiteID: Site‐043





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy ‐14% (25,503) ‐9%







Building type Parking GarageParking Garage










M&V Details Values


Discussion


with T8 fluorescent fixtures and incandescent exit signs with LEDs. The space was not heated and cooled

except for a small office area. Evaluators found that the number of replaced fixtures did not match the

application ‐ fewer fixtures were found on‐site than indicated in the tracking data. The difference in savings

came from finding fewer fixtures on‐site than indicated in the tracking data.

SiteID: Site‐044





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Medical Office Medical Office










M&V Details Values


Discussion

This site is a doctor's office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


matched the application. The difference in savings came from fewer operating hours. The operating hours

determined from metering were slightly lower than the hours used in the tracking savings.

SiteID: Site‐045





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and replacing incandescent exits signs with LEDs. The space was not heated and

cooled, except for a small office area. Evaluators found that the number of replaced fixtures matched the

application. The difference in savings came from a slight discrepancy in the technology types between the

tracking data and what evaluators found on‐site.

SiteID: Site‐047





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a large retail gift shop. The project consisted of replacing incandescent lamps with PAR38 LEDs.

The space was both heated and cooled. Evaluators found one fewer replaced fixture on‐site than what was

indicated in the application. The difference in savings is due to differences in operating hours. Metered data

shows that the evaluated operation hours for the lights are over 35% greater than originally indicated in the

tracking data.

SiteID: Site‐048





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

The facility is a parking garge that operates 24/7. The measure involved replacing T12 fluorescent fixtures and

incandescent exit signs with T8 fixtures and LED exit signs. The space was not heated and cooled, except for a

small office area and break room. Evaluators found that the number of replaced fixtures matched the

application. The largest difference in savings came from a discrepancy in technology types. The evaluators

observed that 32W lamps were used instead of the 28W lamps indicated in the tracking data.

SiteID: Site‐051





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy




Interactive effects discrepancy 0% (3) 0%




Building type Resaturant Restaurant










M&V Details Values


Discussion

This site is a restaurant/bar. The project consisted of lighting replacements in the basement. Incadescents

were replaced with CFLS and T12 fluorescent fixutres were replaced wiht T8 fixtures. The space was both

heated and cooled. Evaluators found that the number of replaced fixtures was slightly different than what the

application indicated ‐ one fewer fixture was found on‐site compared to the tracking data. The difference in

savings came from greater operating hours (nearly double). The operating hours determined from metering

were greater than the hours used in the tracking savings.

SiteID: Site‐055





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and incandescent exit signs with LEDs. The space was not heated and cooled.

Some floors had occupancy sensors. Evaluators found that the number of replaced fixtures were fewer than

what was indicated in the application. Metered hours were close to what was indicated in the application.

The difference in savings is mostly attributable to the quantity discrepancy. The quantity of fixtures counted

during evaluation is larger than the quantity recorded in the tracking data.

SiteID: Site‐057





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures and incandescent exit signs with LEDs. The space was not heated and cooled. Evaluators found that

the number of replaced fixtures were fewer than what was indicated in the application. The difference in

savings came from fewer operating hours and fewer fixtures. The operating hours determined from metering

were slightly lower than the hours used in the tracking savings. In addition, some of the parking garage floors

had occupancy sensors.

SiteID: Site‐058





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Parking garage Parking garage










M&V Details Values


Discussion



cooled. Evaluators found that the number of replaced fixtures was different than what was indicated in the

tracking data. The evaluators found 2 fewer fixtures on site than in the tracking. The operating hours

determined from metering were similar to what was described in the tracking data ‐ the application used

8,736 hours and evaluators measured 8,760 hours.

SiteID: Site‐059





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Parking garage Parking garage










M&V Details Values


Discussion

This site is a small parking garage that operates 24/7. The project consisted of replacing T12 fluorescent

fixtures with T8 fluorescent fixtures and replacing incanscent exit signs with LEDs. The space was not heated

and cooled. Evaluators found that the number of replaced fixtures did not match the application ‐ eight fewer

fixtures were found on‐site compared to the tracking data. Metered operating hours were close to what was

used in the tracking data. Evaluators observed 8,758 opearting hours and the tracking data showed 8,736

hours. The main discrepancy arises from quantity differences and interactive effects.

SiteID: Site‐060





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small office space with several hallways and working areas. This site had undergone significant

renovation after the new lighting had been installed, and space types were difficult to match up. The original

project involved replacing halogens with 6W LEDs and incandescent with 18W and 12W LEDs. The space was

both heated and cooled. Evaluators found that the number of replaced fixtures did not match the application;

far fewer fixture were found on‐site likely due to the space being remodeled. The difference in savings came

from the quantity discrepancy, fewer operating hours and interactive savings. The operating hours

determined from metering were lower than the hours used in the tracking savings.

SiteID: Site‐063





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a parking garage that operated 24/7. The project consisted of replacing T12 fluorescent fixtures


cooled. Evaluators found that the number of replaced fixtures did not match the application ‐ 5 fewer fixtures

were found on‐site than recorded in the tracking data. In addition, the evaluators observed that 32W lamps

were used instead of the 28W lamps indicated in the tracking data. Some circuits in the garage also had

occupancy sensors. The site contact confirmed that this was part of the project. The difference in savings is

mainly a result of the discrepancy in fixture quantities and the technology type.

SiteID: Site‐064





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy 4% 1,670 3%






Building type Religious BuildinReligious Building










M&V Details Values


Discussion

This site is a church with a school connected. The project consisted of replacing T12 fluorescent fixtures with

T8 fluorescent fixtures and replacing incandescent fixtures with CFLS. The space was both heated and cooled.

Evaluators found that the number of replaced fixtures did not match the application; the evaluators found 32

fewer fixtures on‐site. The operating hours determined from metering were significantly lower than the hours

used in the tracking savings. The difference in savings comes from the fixture quantity discrepancy and the

operating hours discrepancy.

SiteID: Site‐065





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an auto shop garage operating mainly on week days. The project consisted of replacing T12

fluorescent fixtures with T8 fluorescent fixtures. The space was both heated and cooled. Evaluators found

that the number of replaced fixtures numbered four fewer than what was indicated in the tracking data. The

difference in savings came from fewer operating hours and the quantity discrepancy. The operating hours

determined from metering were lower than the hours used in the tracking savings.

SiteID: Site‐066





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail showroom. The project consisted of replacing replacing 90W PAR 38 incandescent fixtures

on the showroom floor with 18W LED PAR 38 fixtures. The space was both heated and cooled. Evaluators

found that the number of replaced fixtures was three fewer than indicated in the application. The difference

in savings came from greater operating hours. The operating hours determined from metering were greater


SiteID: Site‐067





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a parking garage. The project consisted of replacing a combination of T12 fluorescent fixtures,

mercury vapor, metal halide and incandescent exit signs with T8 fluorescent fixtures and LED exit signs.

Additionally, occupancy controls were installed. The space was not heated or cooled, except for a small

portion of the garage which was just heated. Evaluators found that the number of replaced fixtures did not

match the application. Fewer fixtures were found on‐site. Metered hours were close to what was used in the

tracking data. The difference in savings came from the quantity discrepancy and interactive effects.

SiteID: Site‐069





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures and incandescent exit signs with LED exit signs. The space was not heated or cooled. Evaluators found

that the number of replaced fixtures matched the application. Evaluators noted that occupancy sensors were

on some floors of the parking garage and that these sensors were not part of the project. The difference in

savings came from the discrepancy in operating hours.

SiteID: Site‐072





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and replacing incandescent exit signs with LED exit signs. The space was not

heated or cooled, except for a small office space. Evaluators found that the number of replaced fixtures did

not the application; nine fewer fixtures were found on‐site. One occupancy sensor controlling two fixtures

was observed in a small side room, but was likely not part of the project. Metered hours were also slightly

lower than what was used in the tracking data. The difference in savings came primarily from the quantity

discrepancy.

SiteID: Site‐074





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures. The space was not heated and cooled. Evaluators found that the number of replaced fixtures

matched the application. The difference in savings came from a discrepancy in the technology types. The

tracking data indicated that 28W lamps were used, but evaluators found that 32W lamps were installed on‐

site.

SiteID: Site‐076





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and replacing incadescent exit signs wtih LED exit signs. The space was not heated

or cooled. Evaluators found that the number of replaced fixtures matched the application. Occupancy

controls were installed as part of the project. The difference in savings came from additional control savings

realized from the logged profiles.

SiteID: Site‐79





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small florist/supply shop. The project consisted of replacing 200W incandescents with 32W CFLs.

The space was both heated and cooled. Evaluators found that the number of replaced fixtures matched the

application. The difference in savings came from fewer operating hours and interactive savings. The operating

hours determined from metering was significantly lower than the hours used in the tracking savings resulting

in a ‐53% discrepancy. There is also a slight interactive effects discrepancy. The final realization rate for this

site is 57%.

SiteID: Site‐81





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site was a small store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


matched the application. The difference in energy saving mainly comes from increased operational hours,

which came to a 15% discrepancy in the realization rate. The final realiztion rate for this site is 114%.

However, the store was out of business before the final logger retrieval and has been under construction.

SiteID: Site‐82





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures, incandescents with CFLs, and metal halides with fluorescents. The space was both heated and

cooled. Evaluators found that the number of replaced fixtures did not match those found in the application.

20 incandescents were not replaced by CFLs. There were also 6 fewer T8s found than originally stated. There

was a 10% quantity discrepancy. The operating hours determined from metering was lower than the hours

used in the tracking savings. The difference in savings came from fewer operating hours, interactive savings,

and a difference in fixtures and quantities from the application. The final realization rate for this site is 105%.

SiteID: Site‐83





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with CFLs. The space was both heated and cooled. Evaluators found that the

number of replaced fixtures matched the application. The operating hours determined from metering was

significantly lower than the hours used in the tracking savings, responsible for a ‐45% discrepancy. The final


SiteID: Site‐84





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures and exit sign incandescents with exit sign LEDs. The space was not heated or cooled aside from a

small office area heated and cooled by a unit heater and window unit, respectively. Evaluators found that the

number of replaced fixtures matched the application. The operating hours determined from metering was

lower than the hours used in the tracking savings resulting in a ‐23% discrepancy. The final realization rate for

this site is 77%.

SiteID: Site‐085





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a grocery store. The project consisted of replacing replacing 90W incandescent lamps with 18W

LED lamps. The space was both heated and cooled. Evaluators found that the number of replaced fixtures

found on‐site was 28 fewer than what was indicated in the application. Additionally, the evaluators found the

the operating hours were approximately double the hours listed in the tracking data. The difference in savings

came primarily from the greater operating hours.

SiteID: Site‐86





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Workshop Workshop










M&V Details Values


Discussion

This site is a small production/manufacturing facility and office. The project consisted of replacing T12

fluorescent fixtures with T8 fluorescent fixtures, incandescents with CFLs, and mercury vapor lamps with HID

fluorescents. The manufacturing/production area was not heated or cooled, but the offices had both heating

and cooling. Evaluators found that the number of replaced fixtures matched the application. The operating

hours determined from metering was significantly lower than the hours used in the tracking savings resulting

in a ‐55% discrepancy. The final realization rate for this site is 37%.

SiteID: Site‐87





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Laundry Laundry





Pre‐retrofit operating hours 4,056 372

Post‐retrofit operating hours 4,056 354 3,748




M&V Details Values


Discussion


fixtures. The space is not heated or cooled. Evaluators found that the number and type of replaced fixtures

did not match the those in the application. There was one fewer 4' T8 and one more 8' T8 resulting in a net

1% discrepancy. The difference in savings came from fewer operating hours, interactive savings, and slight

difference in fixtures. The operating hours determined from metering was significantly lower than the hours

used in the tracking savings resulting in a ‐80% hours discrepancy. The final realization rate for this site is 8%.

SiteID: Site‐90





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an interior building design office and show room. The project consisted of replacing 50W halogens

and PAR30 incandescents with 4W MR‐16s. The space was both heated and cooled. Evaluators found that the

number of replaced fixtures did not match those in the application. There were no PAR38 LEDs on‐site. ALL

LEDs were MR‐16s. The total fixture count for MR‐16 and proposed PAR38s matched the total number of MR‐

16s found on site. Quantity and technology discrepancies accounted for a net 9% discrepancy. The operating

hours determined from metering was lower than the hours used in the tracking saving resulting in a ‐24%

discrepancy. The final realization rate for this site is 87%.

SiteID: Site‐91





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures, PAR20 and PAR30 incandescents with PAR20 and PAR30 LEDs, MR16 halogens with MR16 LEDs, and

PAR20 incandescents with PAR20 fluorescents. The space was both heated and cooled. The interactive effects

savings were underestimated due to greater HVAC savings resulting in a 17% discrepancy. Evaluators found

that the number of replaced fixtures matched the application. The operating hours determined from

metering was slightly higher than the hours used in the tracking savings. The final realization rate for this site

is 120%.

SiteID: Site‐92





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Laundromat Laundromat










M&V Details Values


Discussion

This site is a laundromat. The project consisted of replacing 75W incandescents with 23W CFLs, T12

fluorescents with T8 fluorescents and exit sign incandescents with exit sign LEDs. The space is heated, but not

cooled. The interactive effects were overestimated in tracking resulting in a ‐18% discrepancy in interactive

effects. Evaluators found that the fixtures did not match those in the application. Incandescents were not

replaced with CFLs resulting in a ‐8% discrepancy. The operating hours determined from metering was higher

than the hours used in the tracking savings resulting in a 35% discrepancy. The final realization rate for this

site is 118%.

SiteID: Site‐93





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. The quantity of fixtures installed on site matches the

proposed number. There is a slight difference in the fixture type. The operating hours determined from

metering was higher than the hours used in the tracking savings resulting in a 12% hours discrepancy. The

final realization rate for this site is 109%.

SiteID: Site‐095





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail facility. The project consisted of replacing incandescent fixtures with CFLs. The space was

both heated and cooled. Evaluators found that the number of replaced fixtures did not match the application;

10 fewer fixtures were found on‐site. Additionally, the operating hours determined from metering were

lower than the hours used in the tracking savings. The difference in savings comes from the quantity

discrepancy, differences in operating hours, and interactive effects.

SiteID: Site‐097





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail shop. The project consisted of replacing incandescents with LEDs. The space was both

heated and cooled. Evaluators found that the number of replaced fixtures did not match the application; 13

fewer fixtures were found on‐site. The difference in savings came from fewer operating hours and interactive

savings. The site owner indicated that the ballasts and the bulbs did not match as the bulbs were too heavy

for the screw‐in fixtures. As a result some lights were taken down permanently after the installation. The

operating hours determined from metering were lower than the hours used in the tracking savings. The

primary reasons for the difference in savings were the quantity discrepancy and the hours discrepancy.

SiteID: Site‐097





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail shop. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


matched the application. The difference in savings came from slightly fewer operating hours and interactive

savings. The operating hours determined from metering were lower than the hours used in the tracking

savings.

SiteID: Site‐112





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy 0% (183) 0%

















M&V Details Values


Discussion

This site is a retail store. The project consisted of replacing a combination of T12 fluorescent fixtures and

metal halides with T8 fluorescent fixtures. Occupancy sensors were also installed in the storage areas.

Evaluators found that the number of replaced fixtures matched the application. The difference in savings

came from fewer operating hours and a technology discrepancy. The operating hours determined from

metering were lower than the hours used in the tracking savings.

SiteID: Site‐115





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

Quantity discrepancy ‐1% (128) 0%

Technology discrepancy 10% 1,789 ‐1%

Hours discrepancy 32% 5,870 ‐4%















M&V Details Values


Discussion


fixtures and replacing incandescent exit signs with LEDs. The space was both heated and cooled. Additionally

occupancy sensors were installed on select T8 fixtures. Evaluators found that the number of replaced fixtures

did not match the application. Two fewer fixtures were found on‐site. The difference in savings came from

fewer operating hours and interactive savings.

SiteID: Site‐117





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a large retail furniture showroom. The project consisted of replacing incandescent and halogen

lights with LEDs, as well as replacing some incandescents with CFLs. The space was both heated and cooled.

Evaluators found that the number of replaced fixtures matched the application. There were some questions

on the extent of the project, and evaluators attempted to verify this at each site visit. A one‐to‐one

replacement was assumed. The difference in savings came from fewer operating hours and interactive

savings. The operating hours determined from metering were lower than the hours used in the tracking

savings.

SiteID: Site‐118





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Restaurant Restaurant










M&V Details Values


Discussion

This site is a restaurant. The project consisted of replacing 75W incandescents with 15W CFLs. The space was

both heated and cooled. Evaluators found that the number of replaced fixtures were fewer than indicated in

the application. Additionally, the operating hours determined from metering were lower than the hours used

in the tracking savings. The difference in savings came from the quantity discrepancy, operating hours

discrepancy, and interactive effects.

SiteID: Site‐120





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Small Services Small Services










M&V Details Values


Discussion

This site is a dry cleaning and linen services facility. The project consisted of replacing incandescent lamps

with CFLs, T12 fixtures with T8 fixtures, and halogen lamps with LEDs. The sales are was heated and cooled,

but the workshop and storage areas were only heated. Evaluators found that the number of replaced fixtures

did not match the application; 2 fewer fixtures were found on‐site. The operating hours determined from

metering were lower than the hours used in the tracking savings. The difference in savings came from fewer

operating hours, fewer fixtures found on‐site, and interactive effects.

SiteID: Site‐121





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Food Store Food Store










M&V Details Values


Discussion

This site is a small grocery store. The project consisted of replacing 100W incandescent A bulbs with 23W

compact fluorescent spirals. The space heated, but not cooled. Evaluators found that the number of replaced

fixtures matched the application. The operating hours determined from metering were slightly lower than the

hours used in the tracking savings. The difference in savings came from interactive effects.

SiteID: Site‐122





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a nail salon. The project consisted of replacing a combination of 60W and 100W lamps and four‐

lamp T12 fluorescent fixtures with 23W and 13W CFLs and four‐lamp reduced wattage T8 fluorescent

fixtures. The entire store is heated but only the front of the store is cooled. Evaluators found the same

amount of fixtures on‐site. However, not all fixtures had lamps in them. According to the site contact, after

preexisting fixtures were removed three as‐built fixtures were never installed. The difference in savings came

from fewer operating hours and interactive savings. The operating hours determined from metering were

lower than the hours used in the tracking savings.

SiteID: Site‐123





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Small Services Small Services










M&V Details Values


Discussion

This site is a print shop. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and replacing incandescent lamps with CFLs. The space was not heated or cooled. Evaluators found

that the number of replaced fixtures was one fewer than indicated in the application. The operating hours

determined from metering were lower than the hours used in the tracking savings. The difference in savings

came from the quantity discrepancy and the operating hours discrepancy.

SiteID: Site‐127





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an autobody repair shop. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. The space was not heated or cooled, although, the tracking information indicated

otherwise. Evaluators found that the number of replaced fixtures matched the application. The operating

hours determined from metering were lower than the hours used in the tracking savings. The evaluators

found that some areas of the garage had a good amount of natural light during the day. The difference in

savings derives mainly from the discrepancy in operating hours and the discrepancy in interactive effects.

SiteID: Site‐133





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a restaurant/small grocery store. The project consisted of replacing T12 fluorescent fixtures with

T8 fluorescent fixtures and T12 refrigerator case fixtures with LED fixtures. The space was both heated and

cooled. Evaluators found that the number of replaced fixtures matched the application. The operating hours

determined from metering was significantly lower than the hours used in the tracking savings resulting in a ‐

66% hours discrepancy. The final realization rate for the site is 33%.

SiteID: Site‐136





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Pharmacy Pharmacy










M&V Details Values


Discussion

This site is a pharmacy. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and 100W incandescents with 23W CFLs. The space was both heated and cooled. Evaluators found fewer

fixtures than those found in the application resulting in a 19% quantity discrepancy. The operating hours

determined from metering was lower than the hours used in the tracking savings resulting in a ‐53%

discrepancy. The difference in savings came from fewer operating hours, interactive savings and the quantity

of lights. The final realization rate for this site is 62%.

SiteID: Site‐137





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a textile store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with CFLs. The space is not being heated or cooled. Evaluators found that several

of the replaced fixture quantities did not match those found in the application resulting in a 14% quantity

discrepancy. Several fixtures also remained unretrofitted. The difference in savings came from interactive

effects, fewer operating hours, and differences in fixture quantity and unretrofitted fixtures. The number of

fixtures pre‐retrofit were modified to match those found on site. The operating hours determined from

metering was lower than the hours used in the tracking savings resulting in a ‐70% discrepancy. Interactive

effects savings reductions, responsible for a ‐12% reduction, due to an overestimation of HVAC savings

throughout the facility. The final realization rate for this site is 32%.

SiteID: Site‐141





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail shop. The project consisted of replacing metal halides with T8 fixtures. The space was both

heated and cooled. Evaluators found that the number of replaced fixtures did not match the application; six

fewer fixtures were found on‐site than indicated in the tracking data. The operating hours determined from

metering were lower than the hours reported in the tracking savings. The difference in savings is derived from

the quantity discrepancy, fewer operating hours, and a discrepancy in interactive savings.

SiteID: Site‐142





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a parking garage that operates 24/7. The project consisted of replacing a combination of T12

flourescent fixtures, incandescent bulbs, and incandescent exit signs with T8 fluorescent fixtures, compact

fluorescent lamps, and LED exit signs. The space was not heated or cooled. Evaluators found that the number

of replaced fixtures did not match the application; 8 fewer fixtures were found on‐site. Operating hours

determined from metering were close to the hours used in the tracking savings. The difference in savings

came from the quantity discrepancy.

SiteID: Site‐144





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and replacing incandescent exit signs with LEDs. The space was not heated or

cooled, except for a small office area. Evaluators found a greater number of fixtures on‐site than indicated in

the application. The application listed approximately 3,344 linear feet of fluorescent fixtures in both the pre

and post cases. Evaluators inventoried 4,288 linear feet. Additionally, the baseline fixture wattages were

found to be lower than the tracking baseline fixture wattages and the average installed fixture wattage was

also found to be higher than the tracaking estimates. Operating hours from metering were close to operating

hours indicated in the tracking data. The difference in savings is derivied primarily from the technology

discrepancy.

SiteID: Site‐146





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and replacing incandescent exit signs with LEDs. The space was not heated or

cooled, except for a small office area. Evaluators found that the number of replaced fixtures matched the

application. Metered operating hours were also close to the hours in the tracking data. The difference in

savings came primarily from a technology discrepancy. Tracking data indicated that 28W lamps were

installed, but evaluators found 32W lamps on‐site.

SiteID: Site‐147





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures. The interior space was both heated and cooled, and the awning fixtures were not. Evaluators found

that the number of replaced fixtures were fewer than indicated in the application. The difference in savings

came from fewer operating hours and interactive savings. The operating hours determined from metering

were lower than the hours used in the tracking savings.

SiteID: Site‐150





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Religious CenteReligious Center










M&V Details Values


Discussion

This site is a religious center. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with CFLs. The space is both heated and cooled. Evaluators found that the number

of fixtures in the bridal room, replaced fixtures did not match those found in the application resulting in a 1%

quantity discrepancy. The operating hours determined from metering was less than half the hours used in

the tracking savings resulting in a ‐56% discrepancy. The final realization rate for this site is 39%.

SiteID: Site‐151





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Dry cleaner Dry cleaner










M&V Details Values


Discussion


fixtures and exit sign incandescents with LEDs. The space was heated, but not cooled. Evaluators found that

the number of replaced fixtures matched the application. The operating hours determined from metering

was lower than the hours used in the tracking savings resulting in a ‐62% discrepancy. The difference in

savings came from fewer operating hours and interactive savings. The final realization rate for this site is

324%.

SiteID: Site‐163





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fixtures. Additionally, T12 fixtures were permanently removed in the exterior canopy. The space was not

heated or cooled. Evaluators found that the number of replaced fixtures was one fewer than indicated in the

application. The operating hours determined from metering were higher than the hours used in the tracking

savings. The difference in savings came primarily from the difference in operating hours. There was also a

slight quantity discrepancy and a different in interactive effects.

SiteID: Site‐164





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small community facility with a main floor and basement area. The project consisted of replacing

T12 fluorescent fixtures with T8 fluorescent fixtures and replacing incandescents with CFLs. The space was

both heated and cooled. Evaluators found that the number of replaced numbered two fewer than what was

indicated in the application. The difference in savings came from fewer operating hours, the quantity

discrepancy, and interactive savings. The operating hours determined from metering were lower than the

hours used in the tracking savings.

SiteID: Site‐168





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small retail location. The project consisted of replacing T12 fluorescent fixtures with T8


fixtures numbered one fewer than what was indicated on the application. The difference in savings came



SiteID: Site‐170





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail discount store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. Although the tracking data indicated that some incandescent were replaced with CFLs,

this portion of the project was observed as not implemented upon site visit. It should also be noted that a

few fixtures were de‐lamped on purpose since the site manager said the lights were too bright. This

happened in a few instances throughout the site.The space was heated but not cooled. Evaluators found that

the number of replaced fixtures was fewer than what was inidcated in the application. The operating hours

determined from metering were greater than the hours used in the tracking savings. Although operating

hours were greater, the savings difference is due to a discrepancy in interactive effects. Since the space was

not cooled, there is not cooling bonus.

SiteID: Site‐171





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an auto‐body repair shop. The project consisted of replacing T12 and metal halide HID fixtures with

linear fluorescent T8 fixtures. The space was not heated or cooled, except for a small office space which was

only cooled. Evaluators found that the number of replaced fixtures was 8 fewer than what was indicated in

the application. The operating hours determined from metering were lower than the hours used in the

tracking savings. Additionally, a cooling bonus was claimed in the application, but evalutors found the the site

was not cooled except for a small office area. The difference in savings came from fewer operating hours, the

quantity discrepancy, and interactive savings.

SiteID: Site‐172





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a parking garage. The project consisted of replacing replacing 255 100W metal‐halide fixtures with

255 55W induction lights. The space was not heated or cooled. Evaluators found that the number of replaced

fixtures was fewer than what was indicated in the application. The difference in savings came from fewer

operating hours and the quantity discrepancy. The operating hours determined from metering were lower


SiteID: Site‐173





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small grocery store. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures on the sales floor and replacing T12 fluorescent fixtures with LEDs in the refrigerated and

freezer cases . The sales floor was heated but not cooled. Evaluators found that the number of replaced

fixtures matched the application. The operating hours determined from metering were greater than the

hours used in the tracking savings.

SiteID: Site‐291


Annual energy savings (kWh/yr) 3,478 710 20%



Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a pharmacy. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. The interactive effects were overestimated resulting in a ‐6%

discrepancy. Evaluators found that the number of replaced fixtures matched the application. An

overestimation of operational hours resulted a ‐73% hours discrepancy in operational hour savings. The

realization rate is 20% of the annual energy savings found in the tracking analysis.

SiteID: Site‐293


Annual energy savings (kWh/yr) 4,325 722 17%



Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a real estate office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. Evaluators found that the number of replaced fixtures did

not match the application. Fewer fixtures were found on‐site than what was indicated in the application. The


difference in savings came from fewer operating hours and the quantity discrepancy.

SiteID: Site‐294





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a small small physical therapy facility/doctor's office in a three story office building which is

centrally heated and cooled.. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. Evaluators found that the number of replaced fixtures matched the application. The difference in

savings came from fewer operating hours and interactive savings. The operating hours determined from


SiteID: Site‐295





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a pharmacy. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was both heated and cooled. the interactive savings was underestimated for the site;

there was a 12% discrepana discrepancy of 12% in interactive effects. Evaluators found that the number of

replaced fixtures matched the application. Operational hours was less than found in the tracking analysis

resulting in a ‐8% discrepancy. The interactive effects and operational hour savings resulted in a net

discrepancy of 4%. The realization rate was 4% higher than the annual energy savings in the tracking analysis.

SiteID: Site‐296





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail shop. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


not match the application; two fewer fixtures were found on site than what was indicated in the application.

The operating hours determined from metering were lower than the hours used in the tracking savings. The

difference in savings came from fewer operating hours, a quantity discrepancy, and interactive savings.

SiteID: Site‐300





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a meat‐oriented grocer. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. The space was not heated, but it was cooled. Evaluators found that the number of

replaced fixtures did not match the application; two fewer fixtures were found on‐site than what was

indicated in the application. However, conversations with site staff indicated two fixtures were purposefully

de‐lamped due to the brightness of the T8s. The operating hours determined from metering were higher

than the hours used in the tracking savings. Despite an interactive effects discrepancy, the increased

operating hours increased the savings for this project.

SiteID: Site‐303





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures. The

space was both heated and cooled. The interactive effects resulted in an 8% interactive effects discrepancy.

Evaluators found that the number of replaced fixtures matched the application. The operational hours were

overestimated resulting in a ‐6% discrepancy. The overall realization rate for the site was 102% of the energy

savings estimated in the tracking analysis.

SiteID: Site‐304





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a furniture store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The sales area was both heated and cooled. Evaluators found that the number of replaced fixtures

did not match those found in the application. The incandescent in the basement was not replaced and there

were 30 fluorescent fixtures in the sales area compared to the 24 found in the tracking analysis. The quantity

difference resulted in a ‐6% quantity discrepancy. The ‐9% technology discrepancy is attributed to a

difference in the rated kW assigned to the fluorescent fixtures in the tracking analysis. The operational hours

were lower than those found in the tracking analysis resulting in a ‐29% discrepancy. The overall annual

energy savings realization rate was 42% for the site.

SiteID: Site‐305





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a photo shop. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


not match those in the application, resulting in a 12% quantity discrepancy. The operating hours determined

from metering was lower than the hours used in the tracking savings accounting for a ‐27% discrepancy. The

interactive effects were underestimated for the site, having a 10% discrepancy. The overall annual energy

savings realization rate for the site was 74%.

SiteID: Site‐306





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures. The

space was both heated and cooled. Evaluators found that the number of replaced fixtures matched the

application. The difference in savings came from a ‐18% operational hour discrepancy. The overall annual

energy savings realization rate was 82% for the site.

SiteID: Site‐311





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Religious CenteReligious Center










M&V Details Values


Discussion

This site is a religious center. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures, incandescents with CFLs, exit sign incandescents with exit sign LEDS, and PAR30 and PAR38

incandescents with PAR30 and PAR38 CFLs. The space was both heated and cooled. The interactive effects

were overestimated in the tracking analysis attributing a ‐7% adjusted discrepancy. The operating hours

determined from metering was lower than the hours used in the tracking savings for a 26% discrepancy. The

site's annual energy savings realization rate is 116%.

SiteID: Site‐312





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Cemetery Cemetery










M&V Details Values


Discussion

This site is a cemetery. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures,

metal halides with HID fluorescents, exit sign incandescents with exit sign flLEDs, and incandescents with

CFLs. The space was both heated and cooled. Tracking analysis overestimated the interactive effects savings

resulting in a ‐41% discrepancy. Evaluators found that the number of replaced fixtures did not match those in

the application resulting in a 10% adjusted quantity discrepancy. The operating hours determined from

metering was lower than the hours used in the tracking saving, resulting in a ‐57% discrepancy. There was an

interactive savings discrepancy of ‐12% Interactive savings were reduced due to overestimation of interactive

savings in the tracking analysis. There was a total discrepancy of ‐59%. The site's realization rate for annual

energy savings was 41% of the energy savings estimated in the tracking analysis.

SiteID: Site‐321





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a blinds manufacturer. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures. Evaluators found that the number of replaced fixtures matched the application. The

difference in savings came from fewer evaluated operating hours. The operating hours determined from


SiteID: Site‐323





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is an auto body shop. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures. The space was not heated or cooled. Evaluators found that the number of replaced fixtures was

three fewer than what was indicated in the application. Additionally, the application stated that occupancy

controls were installed, but none were found on‐site. The difference in savings came from fewer operating

hours, interactive savings, and a technology discrepancy. The operating hours determined from metering

were lower than the hours used in the tracking savings.

SiteID: Site‐325





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Library Library










M&V Details Values


Discussion

This site is a Jewish temple and study hall which is open roughly 18 ‐ 20 hours every day. The project

consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures. The space was both heated and

cooled. Evaluators found that the number of replaced fixtures matched the application. The difference in

savings came from greater operating hours. The operating hours determined from metering were higher than

the hours used in the tracking savings.

SiteID: Site‐327





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Senior Center Senior Center










M&V Details Values


Discussion

This site is a senior center. The project consisted of replacing incandescents with fluorescents and exit sign

incandescents with exit sign LEDs. The space was both heated and cooled. The heating and cooling in the

tracking analysis was overestimated causing analysis resulting in a ‐19% interactive effects discrepancy.

Evaluators found that the number of replaced fixtures did not match those found in the application. There

were no lights found in the exterior of the building and fewer A‐Lamp LEDs than found in the tracking

analysis. Quantity discrepancies resulted in a ‐15% discrepancy. Operational hours were lower than

predicted resulting in a ‐25% discrepancy. The overall annual energy savings had a 41% realization rate from

the tracking analysis for the site.

SiteID: Site‐331





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

















Average kW interactive factor (0.25) 0.56 0.20

M&V Details Values


Discussion


fluorescent fixtures and refrigerator case fluorescents with refrigerator case LEDs. The sales floor was both

heated and cooled and the refrigerator cases were cooled only. Evaluators found that the number of replaced

fixtures matched the application. The difference in savings came from fewer operating hours and interactive

savings. The operating hours determined from metering was higher than the hours used in the tracking

savings resulting in a 18% hours discrepancy. Interactive effects were overestimated resulting in a ‐9%

discrepancy. The total realization rate for the site's annual energy savings was 109% of those fonud in the

tracking analysis.

SiteID: Site‐337





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Religious centerReligious center










M&V Details Values


Discussion

This site is a religious center. The project consisted of replacing PAR30 and PAR20 incandescents with PAR30

and PAR20 LEDs and MR16 halogens with MR16 LEDs. The space was both heated and cooled. Reductions in

the interactive effect savings were due to an overestimation of interactive savings in the tracking analysis.

Evaluators found that the number of replaced fixtures matched the application. Operational hours were

significantly less than estimated in the tracking analysis resulting in a ‐60% discrepancy. The site's annual

energy savings realization rate was 35% of the energy savings estimated in the tracking analysis.

SiteID: Site‐339





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Light Manufacturght Manufacturer










M&V Details Values


Discussion

This site manufactures and repairs printing machines. The project consisted of replacing T12 fluorescent

fixtures with T8 fluorescent fixtures and incandescent fixtures with LED fixtures. The space was heated

throughout, and only the showroom area was also cooled. Due to the size of the facility and the site contact's

limited availability, a sample of fixtures was inventoried and logged. This sample matched well with the

tracking savings. The difference in savings came from fewer operating hours and interactive savings. Metered

data was used to determine run hours of the fixtures and showed many fixtures were operated significantly

less than indicated on the tracking. However, after discussions with the contact, this seems reasonable as the

large facility has relatively few staff, and much of it is storage areas.

SiteID: Site‐340





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Church Church










M&V Details Values


Discussion

This site is a large religious community center featuring many different space types including school, offices,

senior center, worship areas, etc. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures, replacing incandescents with CFLS, and replacing incandescent exit signs with LED exit

signs. The space was both heated and cooled. Only two floors were counted in the site visit since the facility

was expansive. Additionally, the spaces types in the tracking info did not match up to what was found on‐site.

As a result it is difficult to say how accurate the fixture count is. The site team assumed a one‐to‐one

replacement of fixtures. Since the facility was badly damaged by Sandy some renovations took place during

the metering period. The operating hours determined from metering were lower than the hours used in the

tracking savings. The difference in savings came from fewer operating hours and technology discrepancies.

SiteID: Site‐345





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a stand‐alone structure that serves as a rental car office. The project consisted of replacing T12

fluorescent fixtures with T8 fluorescent fixtures. The space was both heated and cooled. Evaluators found

that the number of replaced fixtures matched the application. The difference in savings came from fewer

operating hours and a technology discrepancy. The operating hours determined from metering were lower

than the hours used in the tracking savings, and evaluators found that 32W lamps were used instead of 28W

lamps as indicated in the tracking data.

SiteID: Site‐347





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a car rental business with a one story garage for parking their car inventory. The project consisted

of replacing T12 fluorescent fixtures with T8 fluorescent fixtures. The space was heated and the office areas

were also cooled. Evaluators found that the number of replaced fixtures were fewer than what was indicated

in the application. The difference in savings came from fewer operating hours and a technology discrepancy.

The operating hours determined from metering were lower than the hours used in the tracking savings.

SiteID: Site‐350





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures incandescent exit signs with LED exit signs. The space was not heated or cooled,

except for a small office area. Evaluators found that the number of replaced fixtures matched the application.

The difference in savings came from fewer operating hours a technology discrepancy, and interactive effects.

SiteID: Site‐353





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy ‐3% (4,333) 6%
















M&V Details Values


Discussion


fixtures and replacing incandescent exit signs with LED exit signs. The space was not heated or cooled, except

for a small office area. Evaluators found that the number of replaced fixtures matched the application.

Additionally, the evaluators noted occupancy sensors on the upper floors which the site contact indicated

were part of the lighting upgrade. Greater savings were realized through the occupancy sensors.

SiteID: Site‐356





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a parking garage with operates 24/7. The project consisted of replacing T12 fluorescent fixtures

with T8 fluorescent fixtures and incandescent exit signs with LED exit signs. The space was not heated or

cooled except for a small office area. Evaluators found that the number of replaced fixtures did not match the

application; 5 fewer fixtues were found on‐site. The difference in savings came from fewer fixtures found on

site and a technology discrepancy. The evaluators found 32W lamps on‐site instead of 28W lamps as

indicated in the tracking data.

SiteID: Site‐358





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Garage Garage










M&V Details Values


Discussion


fixtures, metal halides with fluorescents, and exit sign incandescents with exit sign LEDs. The space was both

heated and cooled. Evaluators found that the number of replaced fixtures did not those found in the

application. Quantity discrepancies in the parking garage area resulted in a 1% discrepancy with the tracking

savings. The site's annual energy savings realization rate was found to be 101% of the tracking analysis's

annual energy savings.

SiteID: Site‐361





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


with T8 fluorescent fixtures and incandescent exit signs with LED exit signs. The space was not heated or

cooled. Evaluators found that the number of replaced fixtures matched the application. The difference in

savings came from a slight technology discrepancy since evaluators found different fixtures on‐site.

SiteID: Site‐364


Annual energy savings (kWh/yr) 215 41 19%



Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Light Manufacturght Manufacturer










M&V Details Values


Discussion

The company that installed lighting project no longer remains at this site. The new site is an events/specialty

set designer and builder. The project consisted of replacing one 60W incandescent lamp with a 13W

fluorescent lamp. The space was both heated and cooled. Evaluators found that the number of replaced

fixtures matched the application. The difference in savings came from fewer operating hours. Metered hours

were far lower than what was indicated in the application. The evaluted space is an IT closet with very limited

use during normal business hours.

SiteID: Site‐366





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a floral shop. The project consisted of replacing incandescent lamps with LEDs. The space was both

heated and cooled. Evaluators found that the number of replaced fixtures matched the application. The

savings claimed in the application was very close to what evaluators found. The slight difference is due to

slightly greater operating hours than what was indicated in the tracking data.

SiteID: Site‐369





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


Technology discrepancy ‐1% (69) 0%
















M&V Details Values


Discussion

This site is a a retail pharmacy. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and replacing halogen lamps with LEDs. The space was both heated and cooled, except the exterior

awning. Evaluators found that the number of replaced fixtures was fewer than what was indicated in the

application. The operating hours determined from metering were greater than the hours used in the tracking

savings. However, the difference in savings is derived mainly from the quantity discrepancy and less from the

interactive savings since the exterior area was not cooled.

SiteID: Site‐372





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a clothing retail store. The project consisted of replacing T12 fluorescent fixtures with T8


fixtures did not match the application ‐ fewer fixtures were found on‐site. The difference in savings came



SiteID: Site‐375





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail gift and card shop. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures and replacing incandescent lamps with LEDs. The space was both heated and cooled.

Evaluators found that the number of replaced fixtures matched the application. The difference in savings

came primarily from fewer operating hours. The operating hours determined from metering were lower than

the hours used in the tracking savings.

SiteID: Site‐377





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a retail clothing shop. The project consisted of replacing T12 fluorescent fixtures with T8

fluorescent fixtures and replacing incadescent lamps with LEDs. The sales floor was both heated and cooled,

and the basement area was only heated. Evaluators found that the number of replaced fixtures did not match

the application. Fewer fixtures were found on‐site. The operating hours determined from metering were

lower than the hours used in the tracking savings. The difference in savings came from fewer operating hours,

fewer fixtures found on‐site, and a difference in interactive savings since the basement was not cooled.

SiteID: Site‐379





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a grocery store. The project consisted of replacing 90W PAR38 and 50W MR16 lamps with 18W

and 6W LED lamps. The space was both heated and cooled. Evaluators found that the number of replaced

fixtures matched the application. Although the fixture quantities were the same, more of the 18W fixtures

were found onsite, indicating that more of the 90W fixtures had been replaced than initially anticipated. The

operating hours determined from metering were greater than the hours used in the tracking savings. The

operating hours and discrepancy in fixture types resulted in greater evaluated savings.

SiteID: Site‐382





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a general beauty and supply store. The project consisted of replacing T12 fluorescent fixtures with

T8 fluorescent fixtures and replacing halogens with LEDs. The space was both heated and cooled. Evaluators

found that the number of replaced fixtures matched the application. The difference in savings came from

fewer operating hours and interactive savings. The operating hours determined from metering were lower


SiteID: Site‐386





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion


fluorescent fixtures and PAR30 incandescents with PAR30 CFLs. Evaluators found that the number of replaced

fixtures did not fully match those found in the application accounting for a 3% quantity discrepancy. The

operating hours determined from metering was lower than the hours used in the tracking savings.

SiteID: Site‐387





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy



Hours discrepancy 0% (223) 0%





Building type Warehouse Warehouse










M&V Details Values


Discussion

This site is a warehouse/storage area for a larger clothing retailer. The project consisted of replacing T12

fluorescent fixtures with T8 fluorescent fixtures and replacing incandescent exit signs with LED exit signs. The

space was both heated and cooled, except for the basement area. Evaluators found that the number of

replaced fixtures did not match the application. 27 fewer fixtures were found on‐site by the evaluators. The

difference in savings came primarily from slightly fewer operating hours and the quantity discrepancy.

SiteID: Site‐450





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Hair salon Hair salon










M&V Details Values


Discussion

This site is a hair salon. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and incandescents with compact fluorescents. Interactive effects were overestimated for the site resulting in

a ‐3% realization rate. The operating hours determined from metering was lower than the hours used in the

tracking savings resulting in a ‐3% discrepancy. The fixture count did not fully match the count in the

application. There were fewer lights installed than listed in tracking accounting for a 1% discrepancy. The

realization rate for the site's annual energy savings is 95%.

SiteID: Site‐458





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a jewelry store. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent


matched the application. The operating hours determined from metering was lower than the hours used in

the tracking savings. There was a ‐24% hours discrepancy. There was a ‐1% difference in interactive effects

due to heating and cooling. The annual energy savings realization rate was found to be 75% of the tracking

analysis annual energy savings.

SiteID: Site‐459





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy








Building type Warehouse Warehouse










M&V Details Values


Discussion

This site is a warehouse space and office area. The project consisted of replacing T12 fluorescent fixtures with

T8 fluorescent fixtures. The space was both heated and cooled. Evaluators found that the number of replaced

fixtures did not match the application. Fewer fixtures were found on‐site. The operating hours determined

from metering were lower than the hours used in the tracking savings. The difference in savings came from

the quantity discrepancy, fewer operating hours, and interactive savings.

SiteID: Site‐460





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a nail salon. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and replacing incandescents with CFLs and LEDs. The space was both heated and cooled. Evaluators found

that the number of replaced fixtures did not match the application. Fewer fixtures were found on‐site. The


difference in savings is from the quantity discrepancy, different technology types found on‐site, fewer hours

of operation, and interactive effects.

SiteID: Site‐464





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy

















Average kW interactive factor (0.16) 0.21 0.20

M&V Details Values


Discussion

This site is a realty office. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent

fixtures and incandescents with fluorescents. Evaluators found that the number of replaced fixtures did not

match those found in the application. There were fewer fixtures installed resulting in a quantity discrepancy

of 25%. Interactive effects were underestimated for a 7% discrepancy and operational hours were

overestimated for a ‐67% discrepancy. The site's realization rate for annual energy savings was 45%.

SiteID: Site‐493





Savings Discrepancy

Individual

Discrepancy

% Savings

Difference

Individual

Discrepancy

kWh Savings

Difference

Adjusted

%

Discrepancy


















M&V Details Values


Discussion

This site is a pizzeria.. The project consisted of replacing T12 fluorescent fixtures with T8 fluorescent fixtures

and replacing halogen fixutres with LEDs. The space was both heated but not cooled. Evaluators found that

the number of replaced fixtures was one fewer than what was indicated in the application. The operating

hours determined from metering were greater than the hours used in the tracking savings. The difference in

savings came from a technology discrepancy and interactive effects.


CECONY K-1

APPENDIX K: OPERATING HOURS HISTOGRAM

For the three major building types of interest, the annual operating hours for the sampled sites

were plotted. The X-axis represents 1,000 operating hour bins. The Y-axis represents the number

of facilities for each type of building type. This plot provides an overall view of how the

operating hours are distributed for the sampled sites.

Figure K-1 shows the lighting annual operating hours distribution by facility type.

Table K-1 SBDI Lighting Annual Hours of Use Distribution by Facility Type

As seen from the plot, a majority of the retail facilities have lighting operating hours between

3,000 and 4,000. However approximately 10% of the retail facilities had higher operating hours

in the range of 6,000 to 7,000 hours.


CECONY K-2

Similarly, a majority of the office facilities as expected were in the range of 2,000 to 4,000 annual

operating hours.

The parking garages were primarily found to operate at 8,760 hours. Only a few facilities were

found to operate at lower hours due to occupancy sensors/scheduled hours.

contents · 2015. 2. 19. · contents sbdi final evaluation report cecony iii appendix a:...

Documents