SEEM 94 Calibration to RBSA DataProgress Report on Phase 2 (Digging a Little Deeper)

SubcommitteeRegional Technical Forum

March 20, 2013

2

Background• Ran SEEM on 289 RBSA houses. Full RBSA sample (n = 1404) whittled down due to

– Presence of non-gas / non-electric heat,– Poor billing fits,– More than one foundation type,– Other…

• T-Stat “Calibration” by heat source only – Aligns SEEM with PRISM kWh averages for each heating-system group.

• RTF requested further research:– Assess calibration needs related to other variables (climate, measure

parameters, etc) in addition to heating system.

• See January 23, 2013 presentation and minutes for more details

Heating System Type

HeatingHigh °F(day)

HeatingLow °F(night)

Electric Zonal64 64

Electric FAF

Gas FAF 68.6 63.9

Heat Pump 69.6 65.4

3

Research Objectives

• Identify and quantify any systematic patterns (trends) in the differences, ∆ kWh = SEEM kWh ‒ PRISM kWh. – “Systematic” means “explained by known variables.” (Example:

SEEM kWh tends to exceed PRISM kWh in cooler climates.)

– Problem is multivariate. A single underlying trend (for example, ∆ increasing with heating costs) may be apparent in multiple guises (∆ increases with HDD, or with U-value, or fuel type.)

• Develop calibration procedure to remove systematic differences. – Express calibration in terms of T-stat adjustments.

– Understand calibration impact on measure-related parameters.

4

General Approach

• Regression analysis– Compare PRISM output with SEEM output when SEEM is

run at with a constant T-stat setting. • All 289 SEEM values generated with T-stat = 68°F.

– Y-value is the percent difference between SEEM kWh and PRISM kWh.

– X-values are parameters known through RBSA. (Discovering explanatory variables is part of the goal.)

• Assumptions: – Billing (PRISM) results are generally unbiased (excepting

obvious errors)

5

Data Issues

• Heteroskedasticity. – The SEEM/PRISM differences generally increase in

magnitude in proportion to SEEM kWh (or PRISM kWh)

• Measurement error (random noise). – As estimates of heating kWh, SEEM and PRISM

both have substantial standard errors.

6

Heteroskedasticity…

7

Regression Step 0: The y-variable

Note choice of signs: means SEEM > PRISM.

Ideal denominator is “Actual kWh”. Replacing with SEEM kWh or PRISM kWh would skew the y-values.– Due to random error in SEEM and PRISM estimates. – Example: Dividing by PRISM kWh will magnify

differences where PRISM’s random error is negative (these values would have artificially small denominators).

– Better to divide by the midpoint: ?? = (SEEM + PRISM)/2.

Log-transforms (closely related) not quite right either.

8

Regression Step 1: The x-variables

• Goal: Identify variables having systematic influence on y-values.– Substantial random noise in both y-values and x-values, so “influence” is only

seen in rough trends (think: correlation).

• Prominent x-variable candidates:– Measure-related parameters (U-values, duct tightness, infiltration, …)– Heating system type – Climate parameters (esp. HDDs)– Others?

• Model development is iterative. – A variable may be weakly correlated with raw y-values but strongly correlated

with y’s that have been adjusted to account for some other variable’s influence.• Need a single regression model that includes all important x-variables at

once (different models for separate calibrations okay).

9

Limitations to keep in mind…

• Multicollinearity. When a potential x-variable closely tracks some combination of variables that are already included.– This redundancy leads to unstable model fits.– Threshold for “tracks too closely” gets low when the usual

suspects are around:

High noise / faint signal / small sample.

• Parsimony. General principle: Don’t over-fit the data (by including too many explanatory variables).

• Some variables aren’t known for many houses.

10

11

Current Status• There are substantial systematic patterns (in the y-values) related

to U-value and heating system type. Final model must account for these.– Each variable’s influence persists after controlling for the other variable. – Wall-, ceiling-, and floor- U-values are too strongly correlated to

separately include in one model. (Must combine into one or two coarser variables).

• Several other variables are influential too, but we can’t include them all.

• Still developing variables and comparing models. We present a particular linear model with variables for:

Insulation, Heating System Type, and Heating Degree Days

• Purpose is to convey trade-offs and solicit input.

12

Insulation Variable(s)

Model uses an indicator variable to capture “Poor Insulation”.– Definition: Insulation is “poor” if: (Wall u-value > 0.25) OR (Ceiling u-value > 0.25) OR both.– Judgment call in choosing this indicator variable versus the

(continuous) weighted-average u-value, U0.– Still checking options for floor- and window u-values.

Floor-u variable is tricky because of different foundation types.

13

14

Heating System Variable

Four distinct heating systems in the sample:Electric zonal Electric FAF Gas FAF Heat pump

After controlling for insulation, heating system effect mostly captured with just two groups:

“Electric Resistance” = Electric zonal / Electric FAF “Gas/HP” = Gas FAF / Heat Pump

Parsimony: two is better than four!

15

16

17

Variable Estimated Standard t-value p-value coefficient Error

Intercept -0.60 0.14 -4.1 < 0.001 Poor Insulation 0.49 0.07 7.0 < 0.001 Electric Resist. 0.26 0.05 4.9 < 0.001 HDDs 8.96e-05 2.6e-05 3.4 < 0.001

Next model development step would be to check correlations with adjusted y-values (and other fit diagnostics)…

Regression fit for this model

18

19

Variable Estimated Standard t-value p-value coefficient Error

Intercept -0.60 0.14 -4.1 < 0.001 Poor Insulation 0.49 0.07 7.0 < 0.001 Electric Resist. 0.26 0.05 4.9 < 0.001 HDDs 8.96e-05 2.6e-05 3.4 < 0.001

Example: Consider a house that is well-insulated, is heated with an electric FAF, and has 6000 HDDs in a typical year. We expect the y-value for such houses to average around

In other words, we expect SEEM (with t-stat=68) to average around 20% too high.

Interpreting the fitted model

20

0%

20%

40%

60%

80%

100%

120%

Electric Gas/HP Electric Gas/HP Electric Gas/HP

Heating Zone 1 Heating Zone 2 Heating Zone 3

Fact

or to

App

ly to

SEE

M(6

8°F)

Out

put

Well Insulated Poor Insulation

Interim Calibration Results (for this example)

21

And Finally,

• Translate percent kWh adjustments into adjustments in daytime t-stat setting (from 68 °F).

• Do we interpret t-stat calibrations literally (as in, homes like this tend to maintain lower winter temps)? (Refer to morning session!)

• No data limitations here: we can directly observe SEEM’s sensitivity to t-stat settings.– Just run SEEM multiple times, with a different setting

on each.

22

23

50

55

60

65

70

75

Electric Gas/HP Electric Gas/HP Electric Gas/HP

Heating Zone 1 Heating Zone 2 Heating Zone 3

Day

time

T-St

at S

etting

(°F)

Well Insulated Poor Insulation

Calibration Results (for this example)

• There were no cases of poor insulation in heating zone 3 within the filtered sample (289).

24

Discussion

• Is this the correct approach?• Keep in mind we’re still working on

incorporating other variables into the model• Duct tightness• Infiltration• Floor u-value• Window u-value