sampling for m&v: reference guide

Microsoft Word - 2_BPA_M&V_Sampling_Reference_Guide_093011_FINAL.docxB O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Sampling for M&V: Reference Guide
September 2011
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Sampling for M&V: Reference Guide
Version 1.0
September 2011
Prepared for
Stetz Consulting, LLC
Sampling for M&V: Reference Guide i
Table of Contents
2.6. Definitions ................................................................................................................. 5
3. The Sampling Process ......................................................................................9
3.1. Process Steps .......................................................................................................... 9
3.2. Random Samples ................................................................................................... 10 3.2.1. Simple Random Samples .................................................................................................. 12 3.2.2. Stratified Sampling ............................................................................................................ 14 3.2.3. Binomial Distributions ........................................................................................................ 16
3.3. Sample Size and Results Calculator ...................................................................... 17
4. Applications and Examples ............................................................................ 19
4.1. Application 1: Simple Random Sample .................................................................. 19
4.2. Application 2: Stratified Random Sample ............................................................... 20
4.3. Application 3: Binomial Distribution ........................................................................ 22
5. Minimum Reporting Requirements ................................................................. 25
6. References and Resources ............................................................................ 27
Appendix: Glossary of Statistical Terms ............................................................ 29
Sources: .......................................................................................................................... 32
Sampling for M&V: Reference Guide ii
Sampling for M&V: Reference Guide 1
1. Introduction
1.1. Purpose This document presents a Sampling for M&V: Reference Guide1 as a complement to the Measurement & Verification (M&V) protocols used by the Bonneville Power Administration (BPA). The Sampling Reference Guide assists the engineer in selecting how many of what types of points to monitor or meter when it would not be cost-effective to perform these actions at all points.
This document is one of many produced by BPA to direct M&V activities. The Measurement and Verification (M&V) Protocol Selection Guide and Example M&V Plan provides the region with an overview of all of BPA’s M&V protocols, application guides, and reference guides, and gives direction as to the appropriate document for a given energy efficiency project. The document Glossary for M&V: Reference Guide defines terms used in the collection of BPA M&V protocols and guides. In addition, an appendix to this Sampling Reference Guide provides a glossary specific to this guide.
Chapter 6 of this guide provides full citations (and web locations, where applicable) of documents referenced.
1.2. Background In 2009, BPA contracted with a team led by Research Into Action, Inc. to assist the organization in revising the M&V protocols it uses to assure energy savings for the custom projects it accepts from its customer utilities. The team has conducted two phases of research and protocol development under the contract, Number 00044680.
In the first phase, Research Into Action directed a team comprised of:
Quantum Energy Services & Technologies, Inc. (QuEST), led by David Jump, Ph.D., PE and assisted by William E. Koran, PE;
Left Fork Energy, Inc., the firm of Dakers Gowans, PE;
Warren Energy Engineering, LLC, the firm of Kevin Warren, PE;
Schiller Consulting, Inc., the firm of Steven Schiller, PE; and
Stetz Consulting, LLC, the firm of Mark Stetz, PE.
In the second phase, Research Into Action directed a team comprised of:
1 Hereinafter, Sampling Reference Guide.
David Jump, Ph.D., PE, William E. Koran, PE, and David Zankowsky of QuEST;
Mark Stetz, PE, CMVP, of Stetz Consulting;
Erik Kolderup, PE, LEED AP, of Kolderup Consulting; and
Kevin Warren, PE, of Warren Energy Engineering.
The Research Into Action team was led by Jane S. Peters, Ph.D., and Marjorie McRae, Ph.D. Assisting Drs. Peters and McRae were Robert Scholl, Joe Van Clock, Mersiha Spahic, Anna Kim, Alexandra Dunn, Ph.D., and Kathleen Gygi, Ph.D.
For BPA, Todd Amundson, PE, directed the M&V protocol research and development activities. Mr. Amundson was working under the direction of Ryan Fedie, PE, and was assisted by BPA engineers. Mr. Amundson coordinated this work with protocol development work undertaken by the Regional Technical Forum. In addition, Mr. Amundson obtained feedback from regional stakeholders.
Mark Stetz is the primary author of this Sampling Reference Guide; team members reviewed and provided guidance. We thank Andie Baker, Ph.D., Senior Conservation Evaluator for Tacoma Power, for her thoughtful comments on the draft document.
A Microsoft Excel file written by Mark Stetz and entitled BPA Sample Size and Results Calculator.xls accompanies this protocol and is available from BPA.
2. Overview of Sampling
2.1. Description Proper measurement and verification (M&V) requires actual measurements of the affected systems. Except in rare instances, it is not cost-effective to measure the performance of every piece of equipment. Typically, we can measure the performance of a representative sample of the population and extrapolate the results to the whole. The process of identifying which and how many items to measure, and evaluating the results to assess reliability is known as sampling.
2.2. Applicability Sampling is often used where the population of affected systems is too large to measure each point cost-effectively. The primary requirement for sampling to be effective is that the samples selected must be representative of the population as a whole.
Any end-use technology or building category can be evaluated using sampling techniques: lighting upgrades, controls upgrades, rooftop unit replacements, residential upgrades, industrial equipment retrofits, etc.
Several BPA protocols specify that sampling can be used or the protocol methodology is compatible with sampling techniques (see Table 2.1).
Table 2.1: Protocols for Which Sampling May be Appropriate
Protocol Name Application IPMVP Option
Sampling Applicable
A measure that changes electrical load, but not operating hours
B YES
B YES
Verification by Energy Indexing
Equipment whose energy use is affected by the measure and also by some driving variable (i.e., temperature)
N/A
N/A
RTF Group 1 Residential buildings with energy savings at least 5% of whole house consumption
C YES (at program level)
RTF Group 2 New residential buildings D YES (at program level)
RTF Group 3 Nonresidential buildings with a single measure or multiple measures that are not interactive
N/A YES (at program level)
Continued
Protocol Name Application IPMVP Option
Sampling Applicable
RTF Group 4 Nonresidential buildings with interactive measures generating at least 500,000 kWh annual savings
D
RTF Site Specific 1 Non-Interactive, constant load measures A YES
RTF Site Specific 2 Nonresidential sites with at least 100,000 kWh savings and/or complex weather-sensitive measures
B
New industrial processes using a baseline from a similar process or plant
N/A
2.3. Advantages of Sampling The primary advantage of sampling is cost-effectiveness. By measuring the performance of a sample of affected items rather than the entire population, the level of effort is significantly reduced
2.4. Disadvantages of Sampling The fundamental disadvantage of sampling is the introduction of uncertainty, because not every item is measured. With careful sample plan development, selection, and evaluation, uncertainty can be kept to minimal levels and be quantified.
Also, a sampled measurement process can be compromised if the selected samples are not representative of the population, if data collection problems reduce the sample size to less than acceptable numbers, or if there proves to be much greater variability in the population than originally assumed. These limitations may not be discovered until the data has been collected and evaluated, making corrective steps difficult.
2.5. Types of Sampling For most measurement and verification purposes, there are two types of sampling: simple random sampling and stratified sampling.
Simple random sampling draws representative samples from a single population where any sample is representative of the whole. One example is measuring the average power draw of several light fixtures containing two 4-foot T-8 lamps in order to establish the typical power draw where hundreds of identical fixtures exist.
Stratified sampling is used where the population is not homogeneous and needs to be segregated by some defining characteristic. While the population could be divided into discrete groups and simple random sampling applied to each group, this would result in a greater overall sample size and level of effort. Stratified sampling considers the contribution to the overall uncertainty from each group (strata) and allocates the samples to minimize the overall uncertainty and sample size. An example might be residential
energy use, where the population could be divided into three strata of single-family homes, multifamily homes, and condominiums.
2.6. Definitions For sampling to be effective, samples must be drawn from a homogeneous population. Homogeneous in this context means that all members of the population have similar characteristics. A group of single-family, three-bedroom homes of similar vintage, with four occupants, all in the same city, would be a homogenous population.
If a group contains members that have different characteristics, the population is considered heterogeneous. A group of small commercial buildings that includes offices, banks, convenience stores, and self-storage buildings is heterogeneous because the schedule, function, and internal load differ across the building types. In this example, the buildings should be grouped by function to make several small homogeneous populations.
For most projects, it is reasonable to assume that a homogeneous population can be defined by a single average value and that most of the members will have values very close to that average. Only a few members will vary significantly from the average, with some having values higher and some lower than the average value. The farther from the average value, the fewer members will be found. This type of population is considered a normal distribution. An example might be the operating hours of lighting fixtures in an office. Most will operate about 2,200 hours per year, but a few may operate for 1,100, and a few for 3,300 hours. If the population distribution were plotted as a function operating hours, it would form a bell-shaped curve centered at 2,200 hours.
However, there are projects where we need to establish the presence or absence of a condition, rather than determining a value for population members. Examples include situations that test for pass/fail, installed/not installed, operating/broken, or free-rider/not free-rider. These are referred to as binomial distributions, as each element can occupy one of two states. The sampling techniques described here can be applied to binomial distribution cases with some adjustments to the mathematics.
There are also projects where the goal is to determine specific values rather than proportions in a population. In this case, a power analysis must be conducted. Power analysis has different reporting requirements and is beyond the scope of this guide.
The coefficient of variation (CV) is a mathematical expression of the dispersion of a data set from its mean. It can be visualized as how wide or narrow the population distribution (or bell curve) is when data is plotted on a two-dimensional graph. The CV is defined as the standard deviation of the population divided by the average value (or mean) of that population, but it is more important to understand its significance.
Populations with a large range of values have a large CV; tightly clustered populations have a small CV. The previous example of lighting operating hours has a moderate distribution ranging between 1,100 and 3,300, centered at 2,200. An example of a population with a tight distribution would be the speed of cars on a crowded freeway – they all need to travel at close to the same speed for traffic to move.
For many energy efficiency projects, the CV will range in value from 0.1 to 1.0, depending on what is being evaluated. Figure 2.1, below, illustrates the distributions of two different populations, each with a mean value of 100, but with different coefficients of variation.
Figure 2.1: Illustration of Two Distributions with Different Coefficients of Variation
We expect a certain amount of confidence in our measured results. Mathematically, confidence (or confidence level) describes how repeatable the measurement process is. If the desired confidence level is 90%, we would expect that our sampled measurement process will provide the same or similar result nine times out of ten. Increasing the desired confidence level requires expending more effort and often greater expense, the trade-off for increased reliability.
The related confidence interval is the range around a measurement that conveys how accurate the measurement is. If the confidence level is 90%, then there is a 90% probability that the true value of the population lies within the stated confidence interval.
The purpose of a sampled measurement process is to quantify a value that represents the entire population. While the average value of the measured samples is known exactly, sampling introduces some uncertainty as to the true average value of the population. The acceptable difference between the sampled measurement and the population average values is expressed as the precision. If we accept a 10% precision in our measurement, we are saying that the average value of the population is within ±10% of our measured value.
It is important to understand several points:
Confidence and precision must be reported together – a specification of either one alone is meaningless. When reporting results, a statement of 10% precision at 90% confidence means that: 1) the true value of the population is within 10% of the measured value; and 2) we are 90% certain of this result. To be 99% certain of the result would
0 50 100 150 200
N um
Cv = 0.1
Cv = 0.3
require accepting a precision of 16%. We could also state a greater precision, but with a much reduced confidence.
Do not confuse precision and accuracy – they are not the same thing. Precision refers to the repeatability of the measurement by equipment or process; accuracy refers to how close to the true value the measurement is. We often assume that a precise value is an accurate value. However, a biased measurement device or process can yield a very precise, but inaccurate, result. Consider a ruler missing the first two inches – it is precise because it provides the same answer every time, but it is inaccurate because the bias introduces error into the measured value.
Checking for bias requires an independent measurement device or process – one cannot measure and calibrate with the same equipment!
However, checking for systematic bias is beyond the guidance that can be provided here. For our current purpose, we will continue to assume that a precise value is an accurate value.
3. The Sampling Process
3.1. Process Steps The following are the steps to be taken in a sampled measurement process:
1. Define the population. The first step in a sampled measurement process is to identify the population of interest. This could be all of the equipment affected by a retrofit or all of the buildings participating in a program. Usually an equipment inventory or participant database is available that defines the population.
2. Decide if the population is homogenous or heterogeneous. Consider the population and decide if the members are homogeneous or heterogeneous. Do they all have identical or nearly-identical characteristics, or are there characteristics that identify members of the population as unique? Are all fan motors the same size, or do they span 2- to 50-hp? Do all 20-hp motors serve the same purpose, or are some fire pumps and others driving conveyer belts? If the population can be considered homogenous, then use simple random sampling. If the population is heterogeneous, then either divide the population into homogeneous groups and use simple random sampling on each, or apply stratified sampling to the entire population. Where the entire population is comprised of a number of distinct categories (strata), the population can be segregated into these categories or strata. Each stratum is then sampled as an independent sub-population, out of which individual elements can be randomly selected. Strata are selected by considering which categories might have the greatest influence on the primary outcome variable one wishes to measure. Stratified sampling yields a smaller total sample size (and cost), but requires more calculation and evaluation effort.
3. Define the desired confidence and precision. BPA recommends sampled measurement to 10% precision at 90% confidence, but allows some flexibility in the target precision in certain cases. Where individual measurements are expensive or the populations have large variances (high CV), reducing the acceptable precision results in smaller sample sizes. Because precision is proportional to the square of the sample size, relaxing the precision from 10% to 20% results in a four-fold decrease in sample size. Increasing the precision from 10% to 5% requires a four-fold increase in sample size.
4. Assume an initial coefficient of variation. The sample size to achieve a specific precision and confidence depends on the population coefficient of variation CV. But the CV is often not known until after the measurements are taken, so the sample size cannot be determined in advance. To get around this paradox, we must assume a CV value in order to develop a sampling plan. If measurements have been performed previously and the CV is known for the particular characteristic of interest (e.g., residential square footage) one can initially apply the CV determined by earlier research. When estimating the true CV, a statistician should use a default value for the CV of not less than 0.5 for homogeneous samples and 1.0 for heterogeneous samples, until such a time that the population CV can be estimated from the project sample population (PJM 2010, 30). However, the actual CV must be calculated afterwards and compared to the assumed
value. (In some cases researchers can continually monitor the CV of the population as the study progresses and additional data is acquired, allowing researchers to add additional samples if necessary.)
5. Calculate the sample size. With the population(s) defined and the precision and confidence targets set, calculate the required number of samples for each population or group. Where populations are small or the desired precision is very high, the calculated sample size may be close to or even exceed the population size. The sample size can be adjusted using the Finite Population Correction equation, given below.
6. Select random samples. From the equipment or participant list, select samples at random until the desired number have been identified. A quasi-random selection process can be used, such as selecting every kth element. Alternatively, a random number generator can be used to select samples.
7. Implement the measurement process. Using the randomly selected samples, take measurements or conduct surveys. Occasionally, the identified items may not be able to be measured (people are not home, etc.). Rather than giving up a measurement and reducing the actual sample size, have a plan to provide substitutions in the field (find another room, go to another house, etc.). Researchers and evaluators will typically select an initial pool of randomly drawn elements that is larger than the sample size required for the study. While there is always a concern that substitutions can introduce bias, it is usually better to accept a substitute member than reduce sample size. If the populations are truly homogeneous, substitutions will not introduce bias.
8. Evaluate results. For each population or measurement group, calculate: the average value, the standard deviation, the coefficient of variation, and the size of the final sample. Compare the CV of each group – if the actual CV is less than the assumed value, then the desired precision and confidence targets have been met. For each group, calculate the actual precision achieved. If time and budget permit, additional measurements can be taken to increase the sample size and improve the precision. In all cases, report the actual precision achieved.
9. Learn from experience. Because the sample size is a function of the assumed CV, use the calculated CV value for next year’s sampling plan or for another, similar project researching the same primary outcome variable.
3.2. Random Samples A sampling process is used to maximize the useful information from a minimum amount of data. With the information available on each system or building, investigate the defining characteristics to determine if the population is homogeneous or heterogeneous. Consider both the performance and the usage aspects. Table 3.1 illustrates performance and usage characteristics for typical equipment, systems, or building upgrades. This list is only intended to provide guidance; it is not exhaustive.
Note that parameters like efficiency, kW/ton, and R or U value are not directly (or easily) measurable quantities. Each requires multiple (and sometimes difficult) measurements, adding to the complexity, cost, and uncertainty of the measurement process. It is better to focus on directly
measurable quantities like power (kW) rather than trying to rigorously quantify equipment performance.
If the population is reasonably homogeneous, random samples can be selected from the population. If either the usage or performance characteristics vary widely, the population should be divided into groups containing items of similar characteristics. However, too many groups with small numbers in each will be difficult to evaluate as well. The grouping should be a balance between resolution and number of groups. In general, no more than a dozen groups should be necessary.
Table 3.1: Performance and Usage Characteristics for Typical Technologies and Applications
Technology or Application Performance Characteristics Usage Characteristics
Lighting Fixtures Power (watts) Space type, operating hours
Motors hp, RPM, rated efficiency Constant speed and load, variable speed and load, operating hours
Rooftop Units Capacity (tons), EER / SEER Operating hours, local climate
Office Equipment Function, power (Watts) Operating hours, cycles
Water Heating Power (Watts / kBTUh), gallons Inlet and outlet temperatures, gallons/day
Envelope Improvements R value, U value, SHGC Indoor temperature, outdoor temperature (climate), window orientation
Homes Size, type (SF, MF, condo), number of bedrooms, overall R and U values
Number of occupants, age of occupants / hours home, local climate, vintage, type of heating equipment
Small Commercial Buildings Size, function, internal equipment
Schedule, number of occupants, local climate
If the population has been divided into groups, consider whether to use simple random sampling for each group or stratified sampling. Simple random sampling is easy to implement and the results from one group have no influence on another group. Stratified random sampling allocates resources more effectively, providing lower overall measurement costs. However, the calculation process is more complex and results from one group may affect the overall uncertainty of the entire process.
The acceptable level of confidence and precision needs to be established before the research has begun. BPA recommends measured sampling with 10% precision at 90% confidence level, but allows some flexibility in the target precision in certain cases. Where individual measurements are expensive or the populations have large variances (high CV), reducing the acceptable precision results in smaller sample sizes. (The evaluator should avoid revising downward the anticipated CV in order to reduce the required sample size.) Consider the savings expected from the project or program, the cost of each measurement, and the overall evaluation budget. Choosing an acceptable precision level may be an iterative process as the estimated measurement and evaluation cost is compared to the available budget or project/program savings. Absent a defined budget, initial M&V costs should be no more than 10% of the project budget; and annual M&V costs should be no more than 10% of the annual cost savings. The rationale is to provide enough verification to reliably assess project performance, while preventing the M&V effort from eroding its cost-effectiveness.
3.2.1. Simple Random Samples Most projects use a simple random sampling process, even when multiple groups are identified. For each group, the parameter to be measured needs to be identified, the population size known, and the relative variance of the parameter estimated.
The initial sample size for a population is given by:
Sample Size Equation:
Z = 1.645 for a confidence level of 90%
CV = assumed coefficient of variation of the item being measured
P = desired precision, typically 10%
The value of n must be an integer. The calculated value should be rounded up to the nearest integer value (e.g., 67.2 becomes 68).
Because the coefficient of variation is not known in advance, it must be estimated first and calculated afterwards to see if the estimate resulted in the correct sample size. Table 3.2 lists some typical measured values and reasonable estimates of initial CV values that can be used to develop sampling plans. It is not intended to be a prescriptive, comprehensive, or definitive list; experience with previous similar projects (if available) will provide better information on likely CV values.
Table 3.2: Technologies Categorized by Typical Amount of Variance
Items With Low Variance (Assume CV = 0.25)
Items With Medium Variance (Assume CV = 0.5)
Items With High Variance (Assume CV = 1.0)
• Lighting fixture power • HVAC operating hours under
energy management & control system or building automation system control
• Identical rooftop unit full load kW
• Lighting fixture hours under manual control
• Lighting fixture energy use under daylight harvesting control
• Constant-speed motor load factors (motors of similar hp)
• Lighting fixture hours under motion sensor control
• Variable-speed or variable-load motor load factors
• Office equipment hours/day or cycles/day
Where populations are small (< 500) and/or variances large (CV > 0.5), the calculated initial sample may be significantly greater than 10% of the population. If the initial sample size is more than 10% of the population, the sample size may be adjusted downward to account for the small population size. (This is obviously necessary if the sample size exceeds the population size!) However, the correction may still yield a sample size greater than 10% of the population. The finite population correction is:
Finite Population Correction Equation:
n = Z 2Cv
n = initial sample size
N = Population size
Table 3.3 illustrates how sample size depends on the assumed CV, all calculated at 10% precision and 90% confidence (z=1.645). Note that the assumed CV strongly influences the necessary sample size. Populations with low variances (CV ~0.25) can be measured using less than 20 samples, but populations with large variances (CV ~1.00) may need nearly 300 samples to achieve the same precision.
Table 3.3: Illustration of Dependence of Sample Size on Assumed CV
Sample Characteristics Illustrative Value
Population Size, N Sample Size, N*
3 3 3 3
5 4 5 5
10 7 9 10
25 11 19 23
50 13 29 43
100 15 41 74
250 16 54 131
500 17 60 176
Infinite 17 68 271
Once the sample size has been determined, the last step is to select the samples from the population. This can take the form of quasi-random sampling where each kth element is selected from the population until the desired number is achieved (assuming the list is random with regard to the primary variable being measured). This method is acceptable so long as k is not so small that all of the samples are concentrated at the beginning of the list. The selected samples should be distributed fairly evenly across the population list.
For true randomness, a random number generator (such as the RAND() function in Microsoft Excel) can be used to assign a random value between 0 and 1. If 10% of the population is to be sampled, then selecting all items with random values between 0 and 0.1 (or any other range spanning 0.1) will yield a random sample of the appropriate sample size. It is helpful to add a few additional observations to the sample size so that the quota will still be met if some observations are compromised or invalidated by data collection or non-response shortfalls.
Also, a process should be developed in advance for selecting alternate samples in the field. Rather than ignore a location or piece of equipment because access is restricted (or for any other valid reason), allow the field inspectors to choose an alternative but similar sample, document why an alternate was selected, and what the actual sample was. Occasional sample substitution
will yield far better results than rigid adherence to the selected random samples that cannot be measured.
3.2.2. Stratified Sampling There are several reasons to use stratified sampling instead of simple random sampling:
To cost-effectively allocate resources
To provide information about each stratum
Once the overall population has been identified, the characteristics of the members need to be investigated to see if the population is homogeneous and, if not, what groups within the population might be homogeneous. For example, a population of small commercial buildings that includes offices, banks, convenience stores, and self-storage buildings should be stratified based on building function. However, too many groups with small numbers in each will be difficult to evaluate as well. The grouping should be a balance between resolution and number of groups. In general, no more than a dozen groups should be necessary.
Next, the acceptable level of confidence and precision needs to be established. BPA recommends sampled measurement to 10% precision at 90% confidence, but allows some flexibility in the target precision in certain cases. Where individual measurements are expensive or the populations have large variances (high CV), reducing the acceptable precision (such as accepting 20% precision instead of 10%) results in smaller sample sizes.
The advantage of stratified sampling is that evaluating the precision at the project level instead of the group level reduces the overall sample size and cost relative to group level sampling. Stratified sampling achieves this cost reduction by allocating resources according to each stratum’s contribution to the overall variance. Strata that contribute little to the overall variance – either because they are small or because they have a low coefficient of variation – have a smaller sample size than larger or higher variance groups.
As in simple random sampling, an initial coefficient of variation for each stratum needs to be estimated. If all strata have the same CV, the sample size will be allocated proportional to the projected energy use or savings from each group.
For each project, the information shown in Table 3.4 is necessary:
Table 3.4: Recommended Format for a Stratified Sampling Plan
Stratum Name Population Estimated Savings Assumed Coefficient of Variation
Group 1 N1 kWh1 CV1
Group 2 N2 kWh2 CV2
Group i Ni kWhi CVi
Total N kWhT
If savings for each group cannot be estimated with a high degree of reliability, use the expected energy use in place of the estimated savings. After all, we measure energy use, not savings.
With this information, the total sample size n can be calculated as:
Sample Size Equation: n =
CVi = assumed coefficient of variation for group i
P = desired precision, typically 10%
kWhT = estimated total savings
Ni = population size of group i
While this equation looks intimidating, it is relatively straightforward to implement using a spreadsheet calculation tool. The total sample size must be an integer, so round up the result to the next highest integer. Note that for small populations (<500), it is possible that the total sample size may still exceed 10% of the population. However, the total sample size will still be less than that for group-level sampling.
Next, the total number of samples needs to be allocated to each group according to the expected contribution to the variance. Group sample sizes are calculated as:
Equation For Group Sample Sizes:
where: ni = sample size for group i
n = total sample size
CVi = assumed coefficient of variation for group i
Again, the group sample size needs to be rounded up to the next highest integer. In rare cases where a single group has a very small population and a high CV, it is possible to calculate a stratum sample size greater than the population. In this event, simply cap the sample size at the population (measure all items). Even if the group has a high CV, it will make no contribution to the overall uncertainty if all items are measured.
Once the group sample sizes have been determined, selecting appropriate samples is the same as for simple random sampling.
∑ 2 2 + ∑ 2
= ∑
3.2.3. Binomial Distributions Binomial distributions are a special case sometimes encountered in energy efficiency projects and programs. Average values of the probability of encountering a specific state (e.g., pass/fail) can be calculated by assigning a value of zero to the fail (or absent) condition and one to the pass (or present) condition. However, the calculation of the standard deviation and uncertainty need to be treated differently than for a normally distributed population.
The definition of a binomial distribution is one where there are complementary probabilities of occurrence of a state. Let p be one condition and q the other. Then, by definition:
p + q = 1
The standard deviation (s) and standard error (SE) of this type of population can be shown to be:
Standard Deviation and Error:
where: n = sample size
The precision (P) of sampled measurement of the value of p is then:
Precision of Sampled Measurement: P = Z ⋅ SE = Z pq n
where: P = achieved precision
n = sample size
Rearranging to determine sample size yields:
Sample Size: n = pq Z2
P2 = p(1− p) Z 2
P2
P = desired precision
The value of n must be an integer, so the calculated value should be rounded up to the nearest integer value (e.g., 67.2 becomes 68). In the event that p and q cannot be estimated, the safest assumption to make is p = q = 0.5, which will maximize the sample size. Once sample size has
s = pq SE = pq n
been determined, sample selection is similar to that for simple random sampling. In practice, binomial populations tend to be very large. Therefore, it is rare that the finite population correction would need to be applied, so it is not discussed here.
When selecting a desired precision, the precision must be less than ½ of the expected smaller value of p or q. For example, evaluating a situation where the expected failure rate is 10% requires a target precision of 5% or less. If the target precision were greater than the quantity being evaluated, the results would be meaningless.
A quick check to determine if the sample size is adequate is to see that the conditions np ≥ 5 and nq ≥ 5 are met. For example, if p is estimated to be 0.1, then the minimum sample size needs to be 50 to ensure that (50)(0.1) ≥ 5 is true. Failure to meet this condition will result in under- sampling and unreliable results. If p is expected to be very small, the corresponding sample will be very large (e.g., p = 0.01 results in n=500).
3.3. Sample Size and Results Calculator An Excel file entitled BPA Sample Size and Results Calculator.xls accompanies this protocol and is available from BPA.
4. Applications and Examples
4.1. Application 1: Simple Random Sample An industrial energy project replaced 50 standard-efficiency conveyer-belt motors with premium-efficiency motors. All motors are 10-hp in size, serve similar loads, and have similar operating schedules. The measurement and verification plan calls for measuring the actual energy use of a motor over a 24-hour period in the baseline case and retrofit case.
Because all baseline motors are similar in size, function, operating hours, and nameplate efficiency, they can be treated as a single homogeneous group. The same applies to the retrofit motors. Simple random sampling will be used and the selected samples will have true-power data loggers connected to them for short intervals. If the motors were different sizes or served different loads, then the population would need to be divided into different groups.
Because the incremental demand reduction from each motor is small, the energy use must be measured to a high degree of precision in order to reliably assess the savings. The proposed measurement process is to be carried out with 5% precision at 90% confidence to minimize uncertainty in the savings estimate (instead of the usual 10% precision).
Although conveyer-belt motors are subject to large instantaneous load changes, the energy use over the course of a day is expected to remain relatively constant. The expected variance on daily energy use among all motors is therefore small, as each motor serves a portion of the same conveyer line and is subject to the same variations. The coefficient of variation is assumed to be 0.25 for purposes of selecting the sample size.
Calculating, the initial sample size from the equations for P = 5%, C = 90%, CV = 0.25, and N=50 is 29. This sample size needs to be repeated twice – once in the baseline case and once in the post-retrofit case. Given the magnitude of the savings and the M&V budget, installing 29 data loggers twice is considered too expensive. It is decided to accept a reduced measurement precision, recognizing that the resulting uncertainty in the savings calculation would be significantly greater. The target precision is reduced to a more typical value of 10%. Referring to the sample-size table, the proposed sample size for P = 10%, C = 90%, CV = 0.25, and N=50 is 13. Two additional samples are added to ensure sufficient coverage, bringing the total sample size to 15. Repeating this process twice (30 measurements) falls within the proposed budget.
From the motor inventory, every third motor is selected, so that fifteen motors are identified. During the data logger installation, two of the selected motors are found to be in an overhead location that would require a scissor jack to reach. Instead, two nearby motors at ground level are substituted and noted.
After removing the data loggers, the energy use over a 24-hour period is determined for each. One data logger was misconfigured, resulting in unreliable data; the data was discarded. Table 4.1 presents the results.
Table 4.1: Example Simple Random Sample Plan
Sample Characteristics Illustrative Value
Standard Deviation, KWh 31
Actual Sample Size 14
The measured coefficient of variation is 0.35 – greater than the assumed value of 0.25 – indicating that the target precision level was not met. The actual precision is determined by “unadjusting” the sample size and then calculating the resulting precision.
With a population of N=50 and an adjusted sample size n* of 14, the unadjusted sample size n is:
n = Nn*/(N-n*) = (50)(14)/(50 - 14) = 19.4 ≈ 20
The precision for n=20 and CV = 0.35 at 90% confidence is:
p = ZCV / n1/2 = (1.645)(0.35) / (14)1/2 = 15%
To achieve the desired target precision would require
n = Z2CV2 / p2 = (1.645)2(0.35)2 / (0.10)2 = 33.1 ≈ 34
n* = Nn/(N+n) = (50)(34) / (50 + 34) = 20.2 ≈ 21 samples (adjusted for population size)
The cost of seven additional samples needs to be checked against the M&V budget and schedule to ensure that additional measurements can be taken. The measured CV for the baseline motor measurements will be applied to the new motor measurements as well, bringing the entire process to a total of 42 measurements (21 baseline, 21 post-retrofit), budget permitting. The greater variance than originally assumed increased the required sample size. The M&V plan calls for measuring the same motors after replacement rather than a new random sample. The justification for this decision is beyond the guidance of this document.2
4.2. Application 2: Stratified Random Sample A utility provided incentives for an HVAC upgrade program that offset the costs of high- efficiency rooftop units. Incentives are based on the level of improvement relative to ASHRAE 90.1-2004 specifications. To assess the actual program savings, a representative sample of replaced rooftop units is to be measured to verify the actual energy use and estimated savings from each unit and the program as a whole.
2 Measuring the same motors allows the use of a paired Student’s t-test to verify the statistical validity of the
savings reduction. Using a different random sample would allow only an unpaired Student’s t-test, which may not be as sensitive as a paired test.

where: kWhsaved = actual energy savings from each unit during the monitoring period
kWhmeasured = measured energy use of the new unit during the monitoring period
EERrated = nameplate EER rating at ARI conditions
EER90.1 = ASHRAE 90.1 EER rating at ARI conditions
The challenge is to measure the energy use on a sufficient number of rooftop units to achieve an overall precision in the savings estimate of 10% at 90% confidence. Over 400 rooftop units of various sizes have been deployed at a number of different building types; significant variation is expected in their loads and operating hours, depending on the building function and schedule. Since only total program savings need to be known with a high degree of precision, the population of rooftop units can be stratified by building type and function. This is possible because all of the rooftop units are similar in size for each building type. If rooftop units were of radically different sizes within each group, further strata would be required (e.g., Mall RTU 10 tons, Mall RTU 50 tons, etc.)
Initial savings can be estimated by making assumptions about operating hours for each space type and function. Additionally, the initial coefficient of variation for each group will have to be assumed. Based on estimated and assumed values shown in Table 4.2, values and calculated fields can be constructed based on the equations presented previously.
Table 4.2: Example Stratified Random Sample Plan
Usage Group Estimated Energy Savings
kWhi
Samples Required
Grocery Stores 150,000 0.50 75 75,000 75,000,000 7
Worship 10,000 1.00 10 10,000 10,000,000 1
Total 980,000 435 320,000 260,125,000 30
Total Samples 27
Note that although Malls has a low CV, it has the largest sample size because it has the greatest contribution to the overall variance. The Worship group has the smallest sample size because it has the lowest contribution to the overall variance despite its high CV. The total initial sample size of 27 is allocated according to each group’s variance; additional rounding upwards increases the total sample size to 30. Note that simple random sampling on each group would require a
kWhsaved = kWhmeasured EERrated
total of 106 samples to achieve 10% precision at 90% confidence. Stratified sampling has reduced the level of effort by more than a factor of three!
Based on the sample size table, data loggers are deployed at locations selected at random from the project inventory. After one month, the data is retrieved and evaluated. The energy use of each rooftop unit is used to estimate the energy savings based on its nameplate rating relative to the ASHRAE 90.1 baseline. The average energy savings and standard deviation for each group is calculated and the group CV determined. Data logger collection errors reduced the sample size from 11 to 9 in the Malls group. Based on the previously defined equations, Table 4.3 can be constructed:
Table 4.3: Example Stratified Random Sample Results
Usage Group Population Size Ni
Samples Taken
Measured Energy
Savings, kWh
Grocery Stores 75 7 170,000 80,000 0.47 8.29E+08
Worship 10 1 12,000 0 0.00 0.00E+00
Total 435 28 912,000 10.3%
The Worship group has no standard deviation because only one measurement was taken. It therefore does not contribute to the total variance despite the originally assumed CV of 1.0. The resulting overall precision of 10.3% is close enough to the target level of 10% that the precision goal has been achieved.
The measured CVs could be used to develop next year’s sampling plan if one is necessary.
4.3. Application 3: Binomial Distribution A utility company offers standardized incentives for lighting upgrades, but does not want to provide incentives for repairing failed fixtures. The utility requires a sampled survey of lighting fixtures to document that the facility contains 90% or more operating, functional fixtures. For facilities where the number of operating fixtures is less than 90%, the utility will adjust the amount of fixtures eligible for incentives, explaining that they do not offer incentives to subsidize maintenance.
Because a fixture is either operating or failed, the population distribution is binomial. (To remove ambiguity, the utility defines any fixture with a single failed lamp as failed, regardless of the number of lamps in the fixture.) At a large facility where the population of eligible fixtures exceeds 10,000, an inspector needs to determine how many fixtures are operating and how many are failed to determine whether the failure rate exceeds 10%, and needs to have 90% confidence in the results. Because the target precision needs to be smaller than the failure rate, a target
precision of 2% was selected, meaning that a measured error rate of 10% could reflect a true value between 8% and 12%.
The population of fixtures does not need to be grouped by type or schedule, so long as it is of a type eligible for an upgrade incentive. Therefore, separation into discrete groups or strata is not required.
Because the population N is very large and the estimated values of p and q are known, the initial sample size is calculated as:
n = pq Z2
P2
p = probability of state p (0.10)
q = probability of state q (0.90)
P = desired precision (0.02)
The calculation results in a sample size of 609, which also meets the requirements that np >5 and nq >5. From the facility description, 300 spaces are randomly selected, where each space has at least two fixtures. The spaces are inspected, noting at each how many fixtures are present and how many are considered operating and how many failed. Because many spaces have more than two fixtures, the actual sample size will be much greater than 609.
When the survey is completed, it is found that 800 total fixtures were surveyed, 88 fixtures failed, and 712 were functional. This yields the following results:
n = actual sample size, 800
p = probability of state p (failed), 88/800 = 0.11
q = probability of state q (operating), 712/800 = 0.89
Although the calculated failure rate of 11% exceeds the threshold of 10%, the precision needs to be calculated to see if the measured results are different from 10% in a statistical sense. The error of the estimate is calculated as:
P = Z pq n
p = probability of state p, 0.11
q = probability of state q, 0.89
n = sample size, 800
The precision is 0.0182 (1.82%). This indicates that the true value of the failure rate is 0.110 ±0.0182, meaning that we are 90% certain that the true value of the measurement lies between 0.0918 and 0.1282 (9.18% to 12.82%). Because the lower bound of this range is less than 10%, we cannot state that the measured result is statistically different from the 10% threshold, so we accept the results and allow incentives for all fixtures. With 800 samples, 96 failures (12% failure rate) would be required to exceed the uncertainty bounds and adjust the number of fixtures eligible for incentives.
5. Minimum Reporting Requirements
The user of this document should follow the minimum reporting requirements of the M&V protocol for which the sampling is undertaken. In addition, the sampling activities need to be supported in the M&V plan and verification report by a thorough statement of the assumptions made about the population and the nature of the sample. (The verification report need not duplicate sampling information reported in the M&V plan if the verification report appends the M&V plan.)
Population Characteristics: When testing for heterogeneity and determining if the sample should be stratified, the engineer must consider the defining aspects of the systems and facilities under study. Most commonly, the defining characteristics include performance and usage. These and any other defining characteristics should be described and any needs for stratification should be discussed.
Distribution: The M&V plan and verification report should state the assumed probability distribution of the population characteristics. If the engineer has reasons to assume a distribution other than the normal distribution or the binomial distribution, these reasons should be stated in support of the reported distribution.
Population Size (N): The population size of the group or groups needs to be defined.
Sample Type: The documents should specify whether a simple random sample or a stratified sample is required. If stratification is necessary, the M&V plan and verification report should state whether each group should be (or was) treated separately or if the strata should be (or were) used in a combined model.
Variability: For simple and stratified sampling, state the assumed coefficient of variation (CV)
for each group and how it will be calculated after sampling is complete. For binomial sampling, state the assumed presence / absence rate, or success vs.
failure rate.
Precision (P): State the target precision level and how it will be calculated after sampling is complete.
Confidence Level (Z): The desired confidence level (such as 90%).
Sample Size (n): The initial sample size, per strata if multiple strata, and after applying the finite population correction in the case of smaller populations.
Finite Population Correction: State whether or not the finite population correction was used.
Power Analysis: If a power analysis is warranted (see Section 2.6), its parameters should be described; however, power analysis and its reporting requirements are beyond the scope of this guide.
In addition to these elements, the M&V plan needs to discuss the planned course of action if the target precision is not met, and the verification report needs to address the approach followed. There are three options:
Report the poorer precision level in the verification report;
Increase the sample size (take more measurements during field work); or
Convert a stratified sampling plan to a simple sampling plan and report the overall precision; note that one is then no longer able to describe strata findings with any statistical confidence.
6. References and Resources
Brase, Charles Henry, and Corrinne Pellillo Brase. 2009. Understandable Statistics: Concepts and Methods (Ninth Ed.). New York, N.Y.: Houghton Mifflin Company.
Cochran, William Gemmell. 1977. Sampling Techniques (Third ed.). Hoboken, N.J.: John Wiley & Sons, Inc.
De Leeuw, Edith D., Joop Hox, and Don A. Dillman (editors). 2008. International Handbook of Survey Methodology. Mahwah, N.J.: Lawrence Erlbaum Associates.
IPMVP. 2010. International Performance Measurement and Verification Protocol Volume 1: Concepts and Options for Determining Energy and Water Savings. EVO 10000 – 1:2010. Washington, D.C.: Efficiency Valuation Organization. Available at: http://www.evoworld.org/index.php?option=com_docman&task=doc_ view&gid= 1318.
Kish, Leslie F. 1965. Survey Sampling. New York, N.Y.: John Wiley & Sons, Inc.
PJM Forward Market Operations. 2010. PJM Manual 18B: Energy Efficiency Measurement and Verification. Revision: 01, Effective Date: March 1, 2010. Norristown, Penn.: PMJ Interconnection.
Schaeffer, Richard L., William Mendenhall III, and R. Lyman Ott. 2006. Elementary Survey Sampling (Sixth Ed.). Pacific Grove, Calif.: Duxbury Press.
TecMarket Works Team. 2006. California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. April 2006. San Francisco, Calif.: California Public Utilities Commission. Available at: www.cpuc.ca.gov/PUC/energy/Energy+Efficiency/EM+and+V.
Thompson, Steven K. 2002. Sampling. (Second Ed.) New York, N.Y.: John Wiley & Sons, Inc.
Appendix: Glossary of Statistical Terms This Glossary provides definitions for the statistical terms used in this Sampling Reference Guide. Additional M&V terms are defined in the companion document Glossary for M&V: Reference Guide.
Accuracy: An indication of how close the measured value is to the true value of the quantity in question. Accuracy is not the same as precision.
Binomial Distributions: A population consisting of items that can only occupy one of two states (i.e., present/absent, on/off, pass/fail).
Coefficient of Variation (CV): An indication of how much variation or randomness there is with any given data set. It quantifies the variation relative to the average, and is dimensionless. The higher it is, the more variation there is relative to the average. It is defined as the ratio of the standard deviation to the average. It is calculated as:
CV = σ x
Confidence Interval: The range of values expected to contain the true value with a specific probability. The probability is referred to as the confidence level.
Confidence Level: The probability that the true value resides within the confidence interval.
Coefficient of Variation (CV) of the Root-Mean Squared Error [CV(RMSE)]: A measure that describes how much variation or randomness there is between the data and the model, calculated by dividing the root-mean squared error (RMSE) by the average energy use. It is calculated as:
CV (RMSE) = 1 y

Error: Deviation of a measurement from the true value.
Finite Population Correction Equation: When calculating sample size, the finite population correction equation reduces the number of samples required (n) when the population size is small (N) relative to the number of samples (n) assuming a very large population. It is calculated as:
n* = Nn N + n
Heterogeneous Population: A population in which all members have similar characteristics and can be considered of the same type.
Homogeneous Population: A population in which members have different characteristics and thus there are subpopulations of different types.
Independent Variable: A factor that cannot be controlled, and is expected to change regularly and have a measurable impact on the energy use of a system or facility.
Mean: The most widely used measure of the central tendency of a series of observations. The Mean ( ) is determined by summing the individual observations ( ) and dividing by the total number of observations ( ), as follows:
Y = 1 n
Y∑ i
Normal Distribution: A continuous and symmetric population distribution in which the frequency of occurrence decreases exponentially as values deviate from the mean (or central) value. It is also referred to as a Gaussian or bell curve. Most continuous populations are normal
p-value: The probability that a derived value is due to random chance. A small p-value indicates that a result has a low probability of being due to selecting non-representative samples. See t- statistic.
Precision: The indication of the closeness of agreement among repeated measurements; a measure of the repeatability of a process. Any precision statement about a measured value must include a confidence level. A precision of 10% at 90% confidence means that we are 90% certain the measured values are drawn from samples that represent the population and that the “true” value is within ±10% of the measured value. Because precision does not account for bias or instrumentation error, it is not an indication of accuracy.
Quasi-Random Sampling: A sampling technique in which each kth element is selected from the population until the desired number is achieved.
Random Number Generator: A computer algorithm that generates quasi-random numbers that can be used for sample selection or for other purposes. Note that computers cannot generate truly random numbers and that algorithms may generate results with either uniform or quasi-normal population distributions.
Reliability: When used in energy evaluation, refers to the likelihood that the observations can be replicated.
Sampling: The process of identifying which and how many items to measure and evaluating the results to assess reliability.
Simple Random Sampling: A sampling technique that draws representative samples from a single population where any sample is expected to be representative of the whole population.
Standard Deviation (s): The square root of the variance, which brings the variability measure back to the units of the data. (With variance units in kWh2, the standard deviation units are kWh.) The sample standard deviation (s) is defined as:
s = xi − x( )2
∑ (n −1)
Standard Error (SE): The standard deviation divided by √ . This measure is used in estimating precision. SE is defined as:
x sSE n
=
Stratified Sampling: A sampling technique used when the population is not homogeneous and needs to be segregated by some defining characteristic. Sample size for each stratum is determined by the contribution of that stratum to the overall variance such that the total sample size is minimized.
t-statistic: A measure of the probability that the value (or difference between two values) is statistically valid. The calculated t-statistic can be compared to critical t-values from a t-table. The t-statistic is inversely related to the p-value; a high t-statistic (t>2) indicates a low probability that random chance has introduced an erroneous result. The t-statistic is calculated as:
0 ˆ
ˆ ˆ. .( )
−=
Uncertainty: The range or interval of doubt surrounding a measured or calculated value within which the true value is expected to fall within some stated degree of confidence.
Variance (S2): A measure of the extent to which observed values differ from each other (i.e., variability or dispersion), such that the greater the variability, the greater the uncertainty in the mean. Variance, the most important measure of variability, is found by averaging the squares of the individual deviations from the mean. The reason these deviations from the mean are squared is to eliminate the negative values (when a value is below the mean) so they do not cancel out the positive values (when a value is above the mean). Variance is computed as follows: = ∑ −− 1
z-statistic: (Also known as the Standard Score.) The z-statistic indicates how many standard deviations an observation or datum is above or below the population mean. It is a dimensionless quantity derived by subtracting the population mean from an individual raw score and then dividing the difference by the population standard deviation.
xz μ σ −=
Sources: ASHRAE. 2010. ASHRAE Guideline 14-2002 – Measurement of Energy and Demand Savings.
Atlanta, Ga.: American Society of Heating, Refrigerating and Air-Conditioning Engineers.
Brase, Charles Henry, and Corrinne Pellillo Brase. 2009. Understandable Statistics: Concepts and Methods (Ninth Ed.). New York, N.Y.: Houghton Mifflin Company.
IPMVP. 2010. International Performance Measurement and Verification Protocol Volume 1: Concepts and Options for Determining Energy and Water Savings. EVO 10000 – 1:2010. Washington, D.C.: Efficiency Valuation Organization.
TecMarket Works Team. 2006. California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. April 2006. San Francisco, Calif.: California Public Utilities Commission.
Thompson, Steven K. 2002. Sampling. (2nd Ed.). New York, N.Y.: John Wiley & Sons, Inc.
Bonneville Power Administration DOE\BP-4353 September 2011
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles false /AutoRotatePages /All /Binding /Left /CalGrayProfile (Gray Gamma 2.2) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Warning /CompatibilityLevel 1.5 /CompressObjects /Tags /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams true /MaxSubsetPct 100 /Optimize true /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo false /PreserveFlatness true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true /Arial-Black /Arial-BlackItalic /Arial-BoldItalicMT /Arial-BoldMT /Arial-ItalicMT /ArialMT /ArialNarrow /ArialNarrow-Bold /ArialNarrow-BoldItalic /ArialNarrow-Italic /ArialUnicodeMS /CenturyGothic /CenturyGothic-Bold /CenturyGothic-BoldItalic /CenturyGothic-Italic /CourierNewPS-BoldItalicMT /CourierNewPS-BoldMT /CourierNewPS-ItalicMT /CourierNewPSMT /Georgia /Georgia-Bold /Georgia-BoldItalic /Georgia-Italic /Impact /LucidaConsole /Tahoma /Tahoma-Bold /TimesNewRomanMT-ExtraBold /TimesNewRomanPS-BoldItalicMT /TimesNewRomanPS-BoldMT /TimesNewRomanPS-ItalicMT /TimesNewRomanPSMT /Trebuchet-BoldItalic /TrebuchetMS /TrebuchetMS-Bold /TrebuchetMS-Italic /Verdana /Verdana-Bold /Verdana-BoldItalic /Verdana-Italic ] /AntiAliasColorImages false /CropColorImages true /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 150 /ColorImageDepth -1 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.40 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 150 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /GrayImageDict << /QFactor 0.76 /HSamples [2 1 1 2] /VSamples [2 1 1 2] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 15 >> /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects true /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False /CreateJDFFile false /Description << /ARA <FEFF06270633062A062E062F0645002006470630064700200627064406250639062F0627062F0627062A002006440625064606340627062100200648062B062706260642002000410064006F00620065002000500044004600200645062A064806270641064206290020064506390020064506420627064A064A0633002006390631063600200648063706280627063906290020062706440648062B0627062606420020062706440645062A062F062706480644062900200641064A00200645062C062706440627062A002006270644062306390645062706440020062706440645062E062A064406410629061B0020064A06450643064600200641062A062D00200648062B0627062606420020005000440046002006270644064506460634062306290020062806270633062A062E062F062706450020004100630072006F0062006100740020064800410064006F006200650020005200650061006400650072002006250635062F0627063100200035002E0030002006480627064406250635062F062706310627062A0020062706440623062D062F062B002E> /CHS <FEFF4f7f75288fd94e9b8bbe5b9a521b5efa7684002000410064006f006200650020005000440046002065876863900275284e8e55464e1a65876863768467e5770b548c62535370300260a853ef4ee54f7f75280020004100630072006f0062006100740020548c002000410064006f00620065002000520065006100640065007200200035002e003000204ee553ca66f49ad87248672c676562535f00521b5efa768400200050004400460020658768633002> /CHT <FEFF4f7f752890194e9b8a2d7f6e5efa7acb7684002000410064006f006200650020005000440046002065874ef69069752865bc666e901a554652d965874ef6768467e5770b548c52175370300260a853ef4ee54f7f75280020004100630072006f0062006100740020548c002000410064006f00620065002000520065006100640065007200200035002e003000204ee553ca66f49ad87248672c4f86958b555f5df25efa7acb76840020005000440046002065874ef63002> /CZE <FEFF005400610074006f0020006e006100730074006100760065006e00ed00200070006f0075017e0069006a007400650020006b0020007600790074007600e101590065006e00ed00200064006f006b0075006d0065006e0074016f002000410064006f006200650020005000440046002000760068006f0064006e00fd00630068002000700072006f002000730070006f006c00650068006c0069007600e90020007a006f006200720061007a006f007600e1006e00ed002000610020007400690073006b0020006f006200630068006f0064006e00ed0063006800200064006f006b0075006d0065006e0074016f002e002000200056007900740076006f01590065006e00e900200064006f006b0075006d0065006e007400790020005000440046002000620075006400650020006d006f017e006e00e90020006f007400650076015900ed007400200076002000700072006f006700720061006d0065006300680020004100630072006f00620061007400200061002000410064006f00620065002000520065006100640065007200200035002e0030002000610020006e006f0076011b006a016100ed00630068002e> /DAN <FEFF004200720075006700200069006e0064007300740069006c006c0069006e006700650072006e0065002000740069006c0020006100740020006f007000720065007400740065002000410064006f006200650020005000440046002d0064006f006b0075006d0065006e007400650072002c0020006400650072002000650067006e006500720020007300690067002000740069006c00200064006500740061006c006a006500720065007400200073006b00e60072006d007600690073006e0069006e00670020006f00670020007500640073006b007200690076006e0069006e006700200061006600200066006f0072007200650074006e0069006e006700730064006f006b0075006d0065006e007400650072002e0020004400650020006f007000720065007400740065006400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e00650073002000690020004100630072006f00620061007400200065006c006c006500720020004100630072006f006200610074002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e002000410064006f006200650020005000440046002d0044006f006b0075006d0065006e00740065006e002c00200075006d002000650069006e00650020007a0075007600650072006c00e40073007300690067006500200041006e007a006500690067006500200075006e00640020004100750073006700610062006500200076006f006e00200047006500730063006800e40066007400730064006f006b0075006d0065006e00740065006e0020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f00620061007400200075006e0064002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /ESP <FEFF005500740069006c0069006300650020006500730074006100200063006f006e0066006900670075007200610063006900f3006e0020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000640065002000410064006f00620065002000500044004600200061006400650063007500610064006f007300200070006100720061002000760069007300750061006c0069007a00610063006900f3006e0020006500200069006d0070007200650073006900f3006e00200064006500200063006f006e006600690061006e007a006100200064006500200064006f00630075006d0065006e0074006f007300200063006f006d00650072006300690061006c00650073002e002000530065002000700075006500640065006e00200061006200720069007200200064006f00630075006d0065006e0074006f00730020005000440046002000630072006500610064006f007300200063006f006e0020004100630072006f006200610074002c002000410064006f00620065002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /FRA <FEFF005500740069006c006900730065007a00200063006500730020006f007000740069006f006e00730020006100660069006e00200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000410064006f006200650020005000440046002000700072006f00660065007300730069006f006e006e0065006c007300200066006900610062006c0065007300200070006f007500720020006c0061002000760069007300750061006c00690073006100740069006f006e0020006500740020006c00270069006d007000720065007300730069006f006e002e0020004c0065007300200064006f00630075006d0065006e00740073002000500044004600200063007200e900e90073002000700065007500760065006e0074002000ea0074007200650020006f007500760065007200740073002000640061006e00730020004100630072006f006200610074002c002000610069006e00730069002000710075002700410064006f00620065002000520065006100640065007200200035002e0030002000650074002000760065007200730069006f006e007300200075006c007400e90072006900650075007200650073002e> /GRE <FEFF03a703c103b703c303b903bc03bf03c003bf03b903ae03c303c403b5002003b103c503c403ad03c2002003c403b903c2002003c103c503b803bc03af03c303b503b903c2002003b303b903b1002003bd03b1002003b403b703bc03b903bf03c503c103b303ae03c303b503c403b5002003ad03b303b303c103b103c603b1002000410064006f006200650020005000440046002003ba03b103c403ac03bb03bb03b703bb03b1002003b303b903b1002003b103be03b903cc03c003b903c303c403b7002003c003c103bf03b203bf03bb03ae002003ba03b103b9002003b503ba03c403cd03c003c903c303b7002003b503c003b903c703b503b903c103b703bc03b103c403b903ba03ce03bd002003b503b303b303c103ac03c603c903bd002e0020002003a403b10020005000440046002003ad03b303b303c103b103c603b1002003c003bf03c5002003ad03c703b503c403b5002003b403b703bc03b903bf03c503c103b303ae03c303b503b9002003bc03c003bf03c103bf03cd03bd002003bd03b1002003b103bd03bf03b903c703c403bf03cd03bd002003bc03b5002003c403bf0020004100630072006f006200610074002c002003c403bf002000410064006f00620065002000520065006100640065007200200035002e0030002003ba03b103b9002003bc03b503c403b103b303b503bd03ad03c303c403b503c103b503c2002003b503ba03b403cc03c303b503b903c2002e> /HEB <FEFF05D405E905EA05DE05E905D5002005D105D405D205D305E805D505EA002005D005DC05D4002005DB05D305D9002005DC05D905E605D505E8002005DE05E105DE05DB05D9002000410064006F006200650020005000440046002005E205D105D505E8002005D405E605D205D4002005D505D405D305E405E105D4002005D005DE05D905E005D4002005E905DC002005DE05E105DE05DB05D905DD002005E205E105E705D905D905DD002E002005DE05E105DE05DB05D90020005000440046002005E905E005D505E605E805D5002005E005D905EA05E005D905DD002005DC05E405EA05D905D705D4002005D105D005DE05E605E205D505EA0020004100630072006F006200610074002005D5002D00410064006F00620065002000520065006100640065007200200035002E0030002005D505D205E805E105D005D505EA002005DE05EA05E705D305DE05D505EA002005D905D505EA05E8002E05D905D505EA05E8002E002D0033002C002005E205D905D905E005D5002005D105DE05D305E805D905DA002005DC05DE05E905EA05DE05E9002005E905DC0020004100630072006F006200610074002E002005DE05E105DE05DB05D90020005000440046002005E905E005D505E605E805D5002005E005D905EA05E005D905DD002005DC05E405EA05D905D705D4002005D105D005DE05E605E205D505EA0020004100630072006F006200610074002005D5002D00410064006F00620065002000520065006100640065007200200035002E0030002005D505D205E805E105D005D505EA002005DE05EA05E705D305DE05D505EA002005D905D505EA05E8002E> /HRV (Za stvaranje Adobe PDF dokumenata pogodnih za pouzdani prikaz i ispis poslovnih dokumenata koristite ove postavke. Stvoreni PDF dokumenti mogu se otvoriti Acrobat i Adobe Reader 5.0 i kasnijim verzijama.) /HUN <FEFF00410020006800690076006100740061006c006f007300200064006f006b0075006d0065006e00740075006d006f006b0020006d00650067006200ed007a00680061007400f30020006d0065006700740065006b0069006e007400e9007300e900720065002000e900730020006e0079006f006d00740061007400e1007300e10072006100200073007a00e1006e0074002000410064006f00620065002000500044004600200064006f006b0075006d0065006e00740075006d006f006b0061007400200065007a0065006b006b0065006c0020006100200062006500e1006c006c00ed007400e10073006f006b006b0061006c00200068006f007a006800610074006a00610020006c00e9007400720065002e0020002000410020006c00e90074007200650068006f007a006f00740074002000500044004600200064006f006b0075006d0065006e00740075006d006f006b00200061007a0020004100630072006f006200610074002000e9007300200061007a002000410064006f00620065002000520065006100640065007200200035002e0030002c0020007600610067007900200061007a002000610074007400f3006c0020006b00e9007301510062006200690020007600650072007a006900f3006b006b0061006c0020006e00790069007400680061007400f3006b0020006d00650067002e> /ITA (Utilizzare queste impostazioni per creare documenti Adobe PDF adatti per visualizzare e stampare documenti aziendali in modo affidabile. I documenti PDF creati possono essere aperti con Acrobat e Adobe Reader 5.0 e versioni successive.) /JPN <FEFF30d330b830cd30b9658766f8306e8868793a304a3088307353705237306b90693057305f002000410064006f0062006500200050004400460020658766f8306e4f5c6210306b4f7f75283057307e305930023053306e8a2d5b9a30674f5c62103055308c305f0020005000440046002030d530a130a430eb306f3001004100630072006f0062006100740020304a30883073002000410064006f00620065002000520065006100640065007200200035002e003000204ee5964d3067958b304f30533068304c3067304d307e305930023053306e8a2d5b9a3067306f30d530a930f330c8306e57cb30818fbc307f3092884c3044307e30593002> /KOR <FEFFc7740020c124c815c7440020c0acc6a9d558c5ec0020be44c988b2c8c2a40020bb38c11cb97c0020c548c815c801c73cb85c0020bcf4ace00020c778c1c4d558b2940020b3700020ac00c7a50020c801d569d55c002000410064006f0062006500200050004400460020bb38c11cb97c0020c791c131d569b2c8b2e4002e0020c774b807ac8c0020c791c131b41c00200050004400460020bb38c11cb2940020004100630072006f0062006100740020bc0f002000410064006f00620065002000520065006100640065007200200035002e00300020c774c0c1c5d0c11c0020c5f40020c2180020c788c2b5b2c8b2e4002e> /NLD (Gebruik deze instellingen om Adobe PDF-documenten te maken waarmee zakelijke documenten betrouwbaar kunnen worden weergegeven en afgedrukt. De gemaakte PDF-documenten kunnen worden geopend met Acrobat en Adobe Reader 5.0 en hoger.) /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f0070007000720065007400740065002000410064006f006200650020005000440046002d0064006f006b0075006d0065006e00740065007200200073006f006d002000650072002000650067006e0065007400200066006f00720020007000e5006c006900740065006c006900670020007600690073006e0069006e00670020006f00670020007500740073006b007200690066007400200061007600200066006f0072007200650074006e0069006e006700730064006f006b0075006d0065006e007400650072002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e00650073002000690020004100630072006f00620061007400200065006c006c00650072002000410064006f00620065002000520065006100640065007200200035002e003000200065006c006c00650072002e> /POL <FEFF0055007300740061007700690065006e0069006100200064006f002000740077006f0072007a0065006e0069006100200064006f006b0075006d0065006e007400f300770020005000440046002000700072007a0065007a006e00610063007a006f006e00790063006800200064006f0020006e00690065007a00610077006f0064006e00650067006f002000770079015b0077006900650074006c0061006e00690061002000690020006400720075006b006f00770061006e0069006100200064006f006b0075006d0065006e007400f300770020006600690072006d006f0077007900630068002e002000200044006f006b0075006d0065006e0074007900200050004400460020006d006f017c006e00610020006f007400770069006500720061010700200077002000700072006f006700720061006d006900650020004100630072006f00620061007400200069002000410064006f00620065002000520065006100640065007200200035002e0030002000690020006e006f00770073007a0079006d002e> /PTB <FEFF005500740069006c0069007a006500200065007300730061007300200063006f006e00660069006700750072006100e700f50065007300200064006500200066006f0072006d00610020006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000410064006f00620065002000500044004600200061006400650071007500610064006f00730020007000610072006100200061002000760069007300750061006c0069007a006100e700e3006f002000650020006100200069006d0070007200650073007300e3006f00200063006f006e0066006900e1007600650069007300200064006500200064006f00630075006d0065006e0074006f007300200063006f006d0065007200630069006100690073002e0020004f007300200064006f00630075006d0065006e0074006f00730020005000440046002000630072006900610064006f007300200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002000650020006f002000410064006f00620065002000520065006100640065007200200035002e0030002000650020007600650072007300f50065007300200070006f00730074006500720069006f007200650073002e> /RUM <FEFF005500740069006c0069007a00610163006900200061006300650073007400650020007300650074010300720069002000700065006e007400720075002000610020006300720065006100200064006f00630075006d0065006e00740065002000410064006f006200650020005000440046002000610064006500630076006100740065002000700065006e007400720075002000760069007a00750061006c0069007a00610072006500610020015f006900200074006900700103007200690072006500610020006c0061002000630061006c006900740061007400650020007300750070006500720069006f0061007201030020006100200064006f00630075006d0065006e00740065006c006f007200200064006500200061006600610063006500720069002e002000200044006f00630075006d0065006e00740065006c00650020005000440046002000630072006500610074006500200070006f00740020006600690020006400650073006300680069007300650020006300750020004100630072006f006200610074002c002000410064006f00620065002000520065006100640065007200200035002e00300020015f00690020007600650072007300690075006e0069006c006500200075006c0074006500720069006f006100720065002e> /RUS <FEFF04180441043f043e043b044c04370443043904420435002004340430043d043d044b04350020043d0430044104420440043e0439043a043800200434043b044f00200441043e043704340430043d0438044f00200434043e043a0443043c0435043d0442043e0432002000410064006f006200650020005000440046002c0020043f043e04340445043e0434044f04490438044500200434043b044f0020043d0430043404350436043d043e0433043e0020043f0440043e0441043c043e044204400430002004380020043f04350447043004420438002004340435043b043e0432044b044500200434043e043a0443043c0435043d0442043e0432002e002000200421043e043704340430043d043d044b04350020005000440046002d0434043e043a0443043c0435043d0442044b0020043c043e0436043d043e0020043e0442043a0440044b043204300442044c002004410020043f043e043c043e0449044c044e0020004100630072006f00620061007400200438002000410064006f00620065002000520065006100640065007200200035002e00300020043800200431043e043b043504350020043f043e04370434043d043804450020043204350440044104380439002e> /SLV <FEFF005400650020006e006100730074006100760069007400760065002000750070006f0072006100620069007400650020007a00610020007500730074007600610072006a0061006e006a006500200064006f006b0075006d0065006e0074006f0076002000410064006f006200650020005000440046002c0020007000720069006d00650072006e006900680020007a00610020007a0061006e00650073006c006a00690076006f0020006f0067006c00650064006f00760061006e006a006500200069006e0020007400690073006b0061006e006a006500200070006f0073006c006f0076006e0069006800200064006f006b0075006d0065006e0074006f0076002e00200020005500730074007600610072006a0065006e006500200064006f006b0075006d0065006e0074006500200050004400460020006a00650020006d006f0067006f010d00650020006f0064007000720065007400690020007a0020004100630072006f00620061007400200069006e002000410064006f00620065002000520065006100640065007200200035002e003000200069006e0020006e006f00760065006a01610069006d002e> /SUO <FEFF004b00e40079007400e40020006e00e40069007400e4002000610073006500740075006b007300690061002c0020006b0075006e0020006c0075006f0074002000410064006f0062006500200050004400460020002d0064006f006b0075006d0065006e007400740065006a0061002c0020006a006f0074006b006100200073006f0070006900760061007400200079007200690074007900730061007300690061006b00690072006a006f006a0065006e0020006c0075006f00740065007400740061007600610061006e0020006e00e400790074007400e4006d0069007300650065006e0020006a0061002000740075006c006f007300740061006d0069007300650065006e002e0020004c0075006f0064007500740020005000440046002d0064006f006b0075006d0065006e00740069007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f0062006100740069006c006c00610020006a0061002000410064006f00620065002000520065006100640065007200200035002e0030003a006c006c00610020006a006100200075007500640065006d006d0069006c006c0061002e&g

sampling for m&v: reference guide

Documents