bmp analytical & statistical methods tm · technical support services (year 3) contract no....

Florida Department of Environmental Protection TMDL PROGRAM

TECHNICAL SUPPORT SERVICES (YEAR 3) Contract No. WM861

Task Assignment No. 004.03/06-001 FLORIDA BMP DATABASE

Prepared for the

CENTER FOR ENVIRONMENTAL STUDIES Florida Atlantic University

Prepared by:

URS Corporation 7650 West Courtney Campbell Causeway

Tampa, Florida

ANALYTICAL & STATISTICAL METHODS TM

MARCH 2006

Florida Department of Environmental Protection March 2006 FLORIDA BMP DATABASE Technical Memo Task Assignment 004.03/06-001 BMP Analytical and Statistical Methods

State of Florida Total Maximum Daily Load Program

BMP Analytical and Statistical Methods TM

For the Florida Department of Environmental Protection

The BMP Analytical and Statistical Methods Technical Memo presents a discussion of analytical methods and statistical analyses, and recommends specific approaches for use in assessing the performance of BMPs in conjunction with the Florida BMP Database. This document has been produced as a Task Assignment under the Florida Department of Environmental Protection contract with the Florida Atlantic University Center for Environmental Studies (CES) under FAU Research Subagreement #CRE14, which is authorized by FDEP Contract No. WM861. The reproduction of this document was the responsibility of CES. The Technical Elements of this document were the primary responsibility of URS Corporation as a subcontractor to CES in collaboration with Camp Dresser & McKee, Inc. (CDM).

v v v


Page 1

1. Background The objective of Task Assignment 004.03/06-001 was to finalize the development of the Florida BMP Database, incorporate modifications to the BMP performance assessments and conduct an analysis of BMP performance. Previous database development efforts provided for the solicitation, acquisition, organization and data entry of the available BMP monitoring data that had been collected in Florida. This was followed by the development of BMP characteristic data tables and event-based monitoring data tables to serve as the basis for the evaluation of BMP performance characteristics in support of FDEP’s ongoing TMDL modeling activities. At that time, a standardized statistical method was developed for the assessment of BMP performance. With this updated TM, that approach has been updated to include a matched-pairs analysis to help account for the effects of correlation between inflow and outflow concentrations.

2. Analytical & Statistical Methods The analysis of monitoring data for the purpose of estimating the effectiveness of BMPs in reducing pollutant loads is a far more complex task than it would at first appear. Historically, a number of different methods and approaches have been used, all based on different and varying assumptions and yielding significantly different results. This has been presented by others (Strecker 1998, EPA 1999 and Strecker 2001) during the development of the NSW BMP Database. Strecker (2001) made several relevant observations regarding estimating efficiency based on loads using a storm-by-storm basis:

• assumes that all storms are equal (generally untrue)

• wet ponds and wetlands often have discharges which have no relationship to the inflow for the same event

• storm-to-storm comparisons are probably not valid 2.1 Common Analytical Methods Over the history of BMP studies, a number of different analytical methods have been used to estimate the efficiency of a BMP. Four of the most commonly used methods include:

• Efficiency Ratio (ER)

• Summation of Loads (SL)

• Regression of Loads (RL)

• Mean Concentration (MC)

• Efficiency of Individual Storm Loads (ISL) None of these common methods assess the monitoring data statistically and do not examine the differences in inflow and outflow water quality for statistical significance. The Effluent


Page 2

Probability (EP) method selected for use in the NSW BMP Database is an expansion of the efficiency ratio (ER) method and describes the statistical distributions of the upstream and downstream water quality to determine if there are statistically significant differences. 2.1.1 Efficiency Ratio (ER) The efficiency ratio (ER) is defined in terms of the average event mean concentration (EMC) of a parameter from the inflow and outflow monitoring stations. It is computed as:

( )EMCInletAverage

EMCOutletAverageEMCInletAverageER

−=

The ER method provides equal weighting to all storms, but the use of log transforms tends to minimize differences between EMCs and mass balance calculations. It is most applicable when loads are proportional to storm volume. Since inflow EMCs generally do not correlate with storm volume, the use of average EMC values for the inflow should be representative. This may not necessarily be true for outflow EMCs, and the accuracy of this method is expected to vary depending on the type of BMP. 2.1.2 Summation of Loads (SL) The summation of loads (SL) is defined in terms of the ratio of all outgoing loads to all incoming loads. Load (L) is computed for each event as:

VolumeEMCL ∗= The SL ratio is computed based on the sum of the loads as:

( ) ( )( )( )∑

∑∑ −=

inlet

outletinlet

L

LLSL

The SL method is based on mass balance and is generally dominated by a few large storms in the monitoring data. For an accurate evaluation, it is important that sample collection be over a sufficiently long period so that the effects of temporary storage or export of pollutants are minimized. This is of particular concern for BMP systems that may have long residence times. 2.1.3 Regression of Loads (RL) The regression of loads (RL) is computed using least-squares linear regression of the inlet and outlet loads and constraining the y-intercept to zero. This yields a relationship where the outlet loads are defined as a multiple (β) of the inlet loads, where β is the slope of the regression line. The efficiency of the BMP is then computed as:


Page 3

β−= 1RL

Like the SL method, the RL method is based on mass balance and is often dominated by a few large storms in the monitoring data. Again, for an accurate evaluation it is important that sample collection be over a sufficiently long period so that the effects of temporary storage or export of pollutants are minimized. The RL method assumes that the outlet loads are a linear function of the inlet loads (generally observed to have poor correlation) and that treatment efficiency is the same for all events (often untrue). 2.1.4 Mean Concentration (MC) The mean concentration (MC) method is identical to the ER method, but does not require that the event sample data be flow weighted (EMC). This permits grab samples and other non-flow weighted samples to be utilized in defining the efficiency of the BMP. Since sample data that is not flow weighted is applicable to this method, the MC method may introduce significant bias due to different sampling protocols and programs that could be lumped together for assessment. 2.1.5 Efficiency of Individual Storm Loads (ISL) The efficiency of individual storm loads (ISL) computes the removal efficiency for each monitored event based on the inlet and outlet loads. These individual storm efficiencies (SE) are computed as:

( ) ( )( )( )inlet

outletinlet

L

LLSE −=

The ISL efficiency is computed as the average of the SE values:

nSE

ISL ∑=

The ISL method assigns equal weighting to all storms. Large storms don’t dominate the efficiency, which is an average performance regardless of storm size. Both an inflow and outflow value must be available for each event included in the analysis. This excludes any data points that do not have a corresponding measurement at the inflow/outflow for a particular storm. It should be noted that a storm-by-storm analysis makes no provision for events in which the outflow being measured has little or no relationship to the inflow. This often occurs for smaller events in BMPs that have a permanent pool.


Page 4

2.2 Recommended Analytical Methods The performance of BMPs should be evaluated in terms of both concentration reductions (EMCs) and mass load reductions. The approach taken in the development of the NSW BMP Database was to use statistical characterizations of the inflow and outflow concentrations to evaluate the effectiveness of the BMPs. It has been suggested that if enough sample data were available to represent the population of storms expected annually, the same approach can be taken utilizing loads (rather than concentrations) into and out of the BMP. This approach is known as the Effluent Probability (EP) method (EPA 2002). One shortcoming of the EP method is that by assuming the inflow and outflow data represent two independent distributions, any correlation between inflow and outflow EMCs is ignored and becomes part of the variance and uncertainty in the outflow distribution. Two approaches were considered to deal with this; the first was to attempt to adjust the variance of the outflow EMCs to account for any observed correlation, the second was to evaluate the EMC reduction on an individual event basis. The second approach was considered to be more direct, with fewer assumptions required to implement, and was adopted for this study. 2.2.1 EMC Assessment The Effluent Probability (EP) method is an expansion of the efficiency ratio (ER) method and describes the statistical distributions of the upstream and downstream water quality to determine if there are statistically significant differences, as well as confidence limits about the expected average values. In addition, if enough flow and EMC data are available, then the EP method can also be used to enhance the SL method in a similar fashion. Based on previous research (as well as generally accepted stormwater practices), the use of log-transform EMCs is recommended. The EP method adopted for this study provides standard descriptive statistics, box & whisker charts, normal probability charts and time-series plots of the transformed data for both the inflow and outflow data. This information is used to demonstrate the differences in the mean EMCs, including confidence intervals, as well as the performance of the BMP throughout the range of influent and effluent EMCs. Since the EP method does not account for any correlation effects that may be present in the data, a second analytical method recommended for the evaluation of BMP performance is the efficiency individual storm loads (using EMCs), with a slight modification to fit a transformed log-normal distribution to the event efficiency data series. This method was called the individual storm efficiency (ISE) method and included estimates of the average and 95% confidence intervals to illustrate the variation in load reductions and the uncertainty in a BMPs performance. One of the shortcomings of a statistical characterization is the need for data quantity. Stormwater runoff concentrations have been observed to possess notoriously high variations. The greater the variation of a population, the larger the sample needed to accurately characterize the statistics of that population. Sample sets of less than ten data points often have such large confidence intervals that any reduction in concentration provided by the BMP remains ‘hidden’ within the data variation. This results in a finding of


Page 5

“no significant difference” between the inflow and outflow mean values, when in reality the BMP may provide significant removal. 2.2.2 Mass Load Assessment Ideally, the performance of a BMP would be expressed in terms of mass load reductions. However, mass loads can be surprisingly difficult to accurately characterize. There are essentially two basic BMP types that need to be dealt with. For a substantial number of BMP types (such as detention systems, wetlands and media filters) the volume of flow leaving the BMP is essentially the same as that entering the BMP. Losses in volume are not a significant part of the performance of these BMPs in reducing pollutant loads. As a result, their mass load reductions are the same as their EMC reductions and do not require a separate analysis. Other types of BMPs (such as retention and infiltration systems) depend heavily on volume reduction as a significant part of the BMPs performance in reducing pollutant loads. In some cases there may be no significant EMC reduction, or even increases in concentrations of flows passing through the BMP. For these BMPs, both EMC reduction and the reduced volume of discharge can be important factors in the performance of the BMP in reducing mass loads. Unfortunately, monitoring of flow volumes in BMP studies can be very challenging, since accurately measuring flow rates is often difficult. In addition, trying to track all significant sources and volumes can be almost impossible in some cases. The measured data from some BMP studies occasionally revealed more outflow than inflow, presumably due to groundwater contributions! This tends to make a consistent approach to assessing mass reduction performance very difficult to implement. In addition to the challenges presented in collecting accurate flow data from all significant sources, it is also highly unlikely that any set of monitored flows and volumes would be representative of average annual conditions, since event size, frequency and inter-event time are all highly variable factors with significant influence on this data. As a result, it is recommended that a better approach for mass load assessment might be to develop a long-term water balance model of the BMP to estimate the expected annual volume reduction. In addition, if there is a statistically significant difference in the measured inflow and outflow EMCs, a total BMP mass efficiency could be determined using the modeled volumes and the EMC statistics.

3. BMP Performance Assessment In this study, the performance of the BMPs within the Florida BMP Database was limited to EMC values, due to the difficulties in evaluating mass loads (see Section 2.2.2). The assessment of the EMC performance included two statistical approaches, one of which characterizes the inflow and outflow characteristics as if they were two independent distributions. Although some BMP/parameter combinations are expected to display an effluent characteristic that is generally independent of inflow concentrations, others are expected to exhibit effluent concentrations that are highly dependent upon inflow concentrations. Therefore a second statistical assessment which could account for


Page 6

correlation was applied by using paired inflow and outflow data points to represent the observed EMC performance for each measured event. 3.1 Selection of Parameters for Assessment The Florida BMP Database supports a wide range of analytes and the current Database includes data values for nearly 80 different parameters (see the BMP & Event Characteristics TM). It should be noted that the data for many of these analytes is very scarce, and they are not commonly evaluated in the context of BMP performance. Table 1 lists the parameters proposed for performance evaluation of the BMPs in the Florida BMP Database. Table 1: Parameters Selected for BMP Assessment STORET

Code STORET Parameter Name Parameter ID

530 Residue, Total Nonfiltrable (mg/l) TSS 600 Nitrogen, Total (mg/l as N) TN 605 Nitrogen, Organic, Total (mg/l as N) Org-N 610 Nitrogen, Ammonia, Total (mg/l as N) NH3+NH4-N 625 Nitrogen, Kjeldahl, Total, (mg/l as N) TKN 630 Nitrite Plus Nitrate, Total 1 Det. (mg/l as N) NOx or TON 665 Phosphorus, Total (mg/l as P) TP 671 Phosphorus, Dissolved Orthophosphate (mg/l as P) Ortho-P

1027 Cadmium, Total (µg/l as Cd) Cd, Total 1042 Copper, Total (µg/l as Cu) Cu, Total 1051 Lead, Total (µg/l as Pb) Pb, Total 1092 Zinc, Total (µg/l as Zn) Zn, Total

3.2 Performance Calculations The performance of a particular BMP and parameter was expressed in terms of the average expected performance, along with an upper and lower 95% confidence limit to demonstrate the level of uncertainty in the statistical assessments. Given the large variance usually found in stormwater quality, these confidence limits can define a fairly wide range over which the average expected performance can vary. As a result, these confidence intervals are very important to consider when evaluating the performance of a BMP. 3.2.1 BDL Values Analytical tests used by laboratories to establish the concentration of a parameter in a sample are limited in their ability to identify very low concentrations. The method detection limit (MDL) is the minimum concentration that can be measured and reported with 99%


Page 7

confidence that it is greater than zero (but not necessarily quantify it – more on that below). The MDL is determined from the preparation and analysis of a sample in a given matrix containing the parameter and can vary with different samples and testing methods. When an analytical result is below the MDL, it is generally identified as a “below detection limit” (BDL) value, which distinguishes the result as uncertain, but below the MDL. Some laboratories report the actual analytical result (which does not meet the 99% confidence criteria), while others will only report that the test was below the MDL. The BMP studies and reports that formed the basis of the data entered into the BMP Database showed a variety of approaches in handling and reporting BDL values. These included:

• Substituting the result with zero values • Substituting the result with the MDL value • Substituting the result with ½ the MDL value • Exclusion of BDL results

None of these typical approaches is ideal and the greater the fraction of the data set represented by BDL results, the more impact they have on the resulting parametric statistics. The approach adopted for this study is known as a “robust method”, which utilizes the data above the MDL to estimate values that fall below the MDL assuming a log-normal distribution for the data series. Because the estimated values cannot be assigned to specific BDL samples, this approach can only be used for summary statistics and cannot be used for the paired-data (ISE) analysis. The robust method was applied by ranking the data and computing a least-squares regression through the log-transformed values which were above the MDL and their associated standard normal deviates. The least-squares regression was then used to extrapolate replacement values for the BDL results based on their ranked position and these replacement values were then substituted for the BDL values when computing distributional statistics. Because the assignment of replacement values would sometime cause the ranking order of the entire dataset to change, the robust method iterated until this no longer occurred (up to five iterations). It should be noted that the MDL value is the minimum concentration at which the analytical test can reliably detect the presence of the parameter, but not necessarily quantify it with any accuracy. The practical quantitation limit (PQL) is the lowest level of measurement that can be reliably achieved during routine laboratory operating conditions within specified limits of precision and accuracy. The FDEP has adopted a general guideline that if a laboratory fails to report a PQL, the PQL is estimated at four times the MDL. Due to a general lack of reported MDL values within the database (and no PQL data), the effects of the PQL were not addressed in this study. 3.2.2 Outlier Values The statistical assessments conducted in this study generally apply the assumption of normally distributed data, albeit using transformed data. Such assessments can be strongly influenced by the presence of outlier values in the data sets. The presence of outlier values is not necessarily indicative of a bad data set, though errors in the data will frequently


Page 8

appear as outliers. An outlier may be a valid data point, but is due to factors that are rare occurrences and not expected to be found in a data set of so few values. If the number of outliers identified in the data set is significant (greater than 2% to 5% of the data set population), then the assumption of a single normal distribution may not be valid and the resulting statistical analyses should be considered suspect. Outlier values were identified by assessing each data point against the computed distribution parameters for the log-transformed data set. If a log-transformed value fell above or below the expected range for that distribution it was considered an outlier. The range of expected values was defined by upper and lower fence values which are based on the hinges of the distribution (first and third quartiles) and the spread between them (inter-quartile range). The first (25% or Q1) and third (75% or Q3) quartile values of the distribution were computed based on a normal distribution of the log-transformed data. The difference between these two values is called the inter-quartile range (IQR). A “k” value of 1.5 was applied to the IQR to determine the fence values as follows:

)13(*3 QQkQFenceUpper −+=

)13(*1 QQkQFenceLower −−= A data point was considered an outlier if the transformed value was greater than the upper fence or below the lower fence. If a data set contained outliers, the parametric statistics were recomputed with the outlier values removed and the process repeated until no additional outliers were identified. 3.2.3 EMC Analysis – Effluent Probability (EP) The Effluent Probability (EP) method was recommended in Section 2 due to its previous acceptance by ASCE, its general robust application, and the ability to quantify the level of uncertainty in the expected results. This method compares the statistical distributions (in this case, log-normal) of two different data series, the inflow and outflow EMCs for a particular BMP. The data for each monitoring station were log-transformed and ranked for fitting of a log-normal distribution by computing the mean or average (U) and standard deviation (W) of the transformed values. Note that the robust method was used to develop substitute values for any BDL results. These parametric statistics were then used to estimate the following parameters, based on a normal distribution of the log transformed values:

• the estimated 25%, 50% and 75% quartile values (Q1, Q2 and Q3) • the inter-quartile range (IQR) value (75% - 25% quartile values) • the upper fence value (75% quartile plus 1.5 times the IQR) • the lower fence value (25% quartile minus 1.5 times the IQR)

The data set was evaluated for outlier values and adjusted until all outliers were identified and removed from the data set as described in Section 3.2.2. The value used to represent the average value for subsequent performance analysis was the arithmetic estimate of the mean value (M) for the outlier adjusted data set. Based on a log-normal distribution, where


Page 9

U is the average of the log transformed values and W is the standard deviation of the log transformed values, M was computed as:

+=

2exp

2WUM

The modified Cox method, presented by Olsson, was used to estimate the 95% confidence interval (CI) about the estimated arithmetic mean values, based on the use of a log-normal distribution. The Cox method uses the following to estimate the upper and lower 95% confidence limit, where n is the number of data values in the series and t is the Students-t value at the 95% level (for a given n):

−+±+=

)1(22exp

422

nW

nWtWUCL

The ANOVA (Analysis of Variance) procedure using the log-transformed data was performed to test the mean values of the two monitoring stations for significant differences. The ANOVA analysis produces a P value (probability) that the data sets representing the two stations do not have a statistically different mean value. In this study a certainty level of 95% was adopted; thus a P value of 5% or less would indicate that there appeared to be a statistically significant difference in the mean values of the two stations. The average performance of the BMP was computed as the difference between the M values of the two distributions, divided by the M value of the inflow distribution. The uncertainty in the computed average performance was estimated using the upper (UCL) and lower (LCL) 95% confidence limits of the two distributions. These calculations were made as follows:

low

outflowlow

MMM

ePerformancAverageinf

inf −=

low

outflowlow

UCLLCLUCL

CLUpperinf

inf −=

low

outflowlow

LCLUCLLCL

CLLowerinf

inf −=

3.2.4 EMC Analysis – Individual Storm Efficiency (ISE) The EP approach assumes that the two distributions represented by the inflow and outflow EMC data are independent of each other. Correlations between inflow and outflow concentrations result in increased deviation and uncertainty in the outflow distribution. While this effect could be addressed by removing any correlative effects between the two data series, a more direct approach was developed using the paired data sets to compute the


Page 10

EMC reduction for each data pair (individual storm efficiency or ISE) and the resulting efficiency data set to characterize the performance of the BMP. The ISE data set has an upper bound of 1 (can approach 100% removal efficiency) and a lower bound of negative infinity (a virtually unlimited increase in EMC values is possible). The ISE data would also be expected to be skewed to the right. A transform was applied to the data so that it could be represented by a normal distribution as follows:

• A value of 1, or 100% was subtracted from each efficiency value. This shifts the distribution to the left and creates an upper bound of 0.

• The shifted values were multiplied by (-1) to invert the distribution about the Y-axis. The transformed values now have a lower bound of 0 and an upper bound of infinity, with a left skew, the same as log-normally distributed data.

• The shifted and inverted values were then log-transformed to fit a normal distribution. The transformed efficiency data set was evaluated for outlier values and adjusted until all outliers were identified and removed from the data set as described in Section 3.2.2. The value used to represent the average efficiency was based on the arithmetic estimate of the mean value (M) for the outlier adjusted transformed efficiency data set. Based on a log-normal distribution, M was computed as:

+=

2exp

2WUM

The uncertainty in the computed average performance was estimated using the upper (UCL) and lower (LCL) 95% confidence limits of the transformed efficiency distribution using Cox’s method. These calculations were made as follows:

−+++=

)1(22exp

422

nW

nWtWUCLUpper

−+−+=

)1(22exp

422

nW

nWtWUCLLower

The robust BDL method cannot be applied to event specific data, so efficiency computations for paired events where one of the values was BDL were based on a value of ½ the MDL. This can dramatically affect the computed distribution of the efficiency values, so two assessments were conducted using the ISE approach. The first assessment included individual storm events with a BDL result at one of the two monitoring stations (but not both); the other excluded all events in which either station had a BDL result. Any event in which both the inflow and outflow values were both BDL were dropped from the data series completely.


Page 11

3.2.5 Spreadsheet Analysis Tool The tremendous volume of data in the Florida BMP Database can quickly become overwhelming when attempting to conduct the assessment of BMP performance as described above. In addition, databases are not easily adapted to perform many calculations, particularly between independent records. A specialized spreadsheet was developed early in this project to assist with data entry to the database and this tool was expanded to include the retrieval and assessment of BMP performance data for this study. The application of both the EP and ISE methods to the EMC data in the database was implemented into an Excel spreadsheet, “FL BMP Database.xls”. The spreadsheet uses a series of VBA macros to perform the statistical calculations based on user supplied input. A screen capture of the spreadsheet EMC analysis tool showing the user selectable options is included in Figure 1. Initially there are ten steps to complete (see Figure 1), but subsequent assessments can be performed with modifications to as little as one of the selection items.


Page 12

Figure 1: EMC Analysis Tool

A brief summary of the operation of the FL BMP Database Analysis Tool follows:

• The user must identify where the Database is stored (Step 1) and where the output files should be stored (Step 8). These are both initialization steps and the values can be saved in the spreadsheet. They do not need to be updated for subsequent analyses unless the location of the Database is changed (or a different Database is desired), or the location of the output files needs to be changed.

• The user must then “refresh” the spreadsheet with data from the Database (Step 2). When this button is clicked, the Database specified in Step 1 is queried to create a


Page 13

list of the BMP Test Sites and Monitoring Stations. This step needs to be done each time the FL BMP Database.xls file is opened since this data are not stored in the spreadsheet file. However, it does not need to be repeated between analyses unless the FL BMP Database.xls file was closed.

• Once the data lists have been updated to the spreadsheet, the user can select the BMP Test Site from the pull down list in Step 3. At this point the pull down list immediately below the Test Site will show the available images for the BMP. If desired, the user can click the “Show Image” button to view the selected image, which generally shows the BMP layout and the location of the various Monitoring Stations. Clicking the dotted button to the right of the image list displays the full comment associated with the image.

• Following the selection of a BMP Test Site, the Inflow and Outflow Monitoring Station lists (Steps 4 and 5) will be automatically filtered to show only the Monitoring Stations for the selected Test Site. If a pull down list is blank, then either the spreadsheet needs to be refreshed (Step 2), or no data exists in the Database for the current Test Site. Both Monitoring Station lists are identical and it is incumbent upon the user to select the appropriate inflow and outflow stations.

• Following the identification of the two Monitoring Stations, the desired parameter for analysis can be selected (Step 6) using a similarly filtered pull-down list.

• After the selection of monitoring stations and parameter, the user can apply additional filters so that only specific types of water quality records will be examined (Step 7). Available filters include flow type, sample media type, sample type and exclusion of BDL values. The current inventory for this data is shown next to each of the filter selections to help guide the user. If the Auto Update box is selected, these values update each time a selection from one of the pull-down lists is changed. If the box is not selected, the “Update Sample Counts” button will need to be clicked before the inventory counts will update (which may be preferable if the database is being accessed over a local area network rather than locally on the user’s computer).

o Flow Types: The user can select Runoff or Baseflow to limit the selected water quality data to samples which have these types of flow data associated with them. If the user wants to select water quality for samples even if no flow data exist, the Not Specified box should be selected. Unless mass calculations will be conducted, the Not Specified box should usually be selected.

o Sample Media Types: The user can filter the data request based on various media types by selecting the appropriate boxes. For a typical EMC analysis, usually only surface runoff/flow records are desirable.

o Sample Types: The user can apply these filters to select data limited to specific types of samples, such as composite, grab, etc.

• Each time the Water Quality Parameter is reselected (Step 6), the database is queried to determine the range of available dates for water quality data collected at the selected inflow and outflow monitoring sites. The user can modify these dates to select a subset of the data by date for statistical analysis.

• The name of the output file is created automatically by the analysis tool, but can be modified by the user if desired (Step 9).


Page 14

• Finally, clicking on the “Acquire & Analyze” button (Step 10) will activate the analysis macros and create the output file, which is left open for the user to examine once the macro successfully terminates.

The actions performed by the macro are outlined as follows:

• The Database is queried for the data using the selections and filters as set by the user. A copy of the resulting data is pulled into the output spreadsheet file.

• Substitute values for BDL results are computed for the inflow and outflow data sets using the robust method described in Section 3.2.1.

• Outlier values in both the inflow and outflow data sets are identified and the data sets adjusted as described in Section 3.2.2.

• An EMC analysis is performed using the EP method as described in Section 3.2.3. An example of the results is shown in Table 2.

• An EMC analysis is performed using the ISE method as described in Section 3.2.4. An example of the results is shown in Table 3.

3.3 Interpretation of Analytical Output Once the FL BMP Database Analysis Tool has completed the statistical computations, it produces a spreadsheet output file with a summary statistics tab, three data tabs and seven analytical charts:

• Summary statistics tab • Performance comparison chart • Box & whisker chart • Probability chart • Time series chart • Efficiency series chart • Efficiency probability chart • Correlation series chart • Inflow data tab • Outflow data tab • Paired data tab

The ability to successfully use statistical characterizations to evaluate BMP performance is highly dependent on the amount of data as compared to the variance in that data. Stormwater EMCs can generally be classified as having large variations, and will thus require fairly large data sets for characterization. The inspection of the graphic charts included in the output file can often help reveal insights into the performance of a BMP with statistically inadequate datasets, or to confirm the results shown in the summary statistics tab. 3.3.1 Summary Statistics Tab The summary statistics tab lists the Test Site, Inflow and Outflow Monitoring Stations, the assessment parameter, and selected query filters. It summarizes the distributional statistics of the log-transformed inflow and outflow data. The summary data shown in Tables 2 and 3


Page 15

(above) were taken from the Summary stats tab. Finally, several plotting support data tables are included for the Box & whisker chart and the Probability chart. Interpretation of the output file should begin with a review of the data on the Summary stats tab. The example output in Table 2 shows that the ANOVA test was passed, but more importantly the P Value was very small (< 0.000), indicating a distinct difference between the two means. Table 2: Example EMC Assessment (EP Method)

Summary Statistics

Site: FDEP-64 (Florida Aquarium Test Site) Inflow MS: FDEP-64-01 (Inflow)

Outflow MS: FDEP-64-02 (Outflow)

Parameter: CU-T (COPPER, TOTAL (UG/L AS CU))

Statistics for Log-transformed Data:

ANOVA

LN (Inflow)

LN (Outflow) All

Statistic Inflow Outflow Sum 155.877 52.072 207.949 Data Count 54 41 Sum^2 467.012 84.965 551.977

# BDL 1 7 Sum Sq. 17.055 18.832 # Outliers 0 1 Overall Mean 2.189

Mean 2.887 1.270 Source SS df MS Std Dev 0.567 0.686 Between groups 60.904 1 60.904

Minimum 1.386 -0.369 Within groups 35.887 93 0.386 Maximum 4.022 2.451 Total 96.791

Q10 2.160 0.391 F 157.831 PASS: MEANS ARE DIFFERENT Q25 2.504 0.807 P 0.000

Q75 3.269 1.733

Q90 3.614 2.149 Removal Efficiency (Distribution Probability)

IQR 0.765 0.926 Inflow Outflow

95% t 2.006 2.021 Arithmetic Est. of Mean: 21.063 4.506

Std Error 0.077 0.107 Modified Cox 95% UCL: 24.894 5.737

95% CI 0.155 0.217 Modified Cox 95% LCL: 17.821 3.539

Lower CL 2.732 1.053 Average Difference: 78.6%

Upper CL 3.041 1.487 Max. Diff. (Inflow UCL - Outflow LCL): 85.8%

Lower Fence 1.356 -0.581 Max. Diff. (Inflow LCL - Outflow UCL): 67.8%

Upper Fence 4.417 3.121

Review of the example results shown in Table 2 (EP Method) finds the difference between the average EMC values of the two distributions indicates a reduction of 78.6%, with a 95% confidence interval of [67.8%, 85.8%]. It should be noted that due to the use of a log-transformed distribution, the confidence interval is not symmetrical about the mean.


Page 16

Table 3: Example EMC Assessment (ISE Method)

Paired Data Summary Statistics Stat Value

(Incl BDLs) Stat Value

(Excl BDLs) Data Count 41 34

# Outliers 1 1 Average 74.6% 71.3%

Std. Dev. 80.7% 81.3% CV 24.1% 35.1%

Cox Upper 95% CL 80.0% 77.1% Cox Lower 95% CL 67.7% 63.9%

Inflow/Outflow Correlation -.025 -.072

Review of the example results shown in Table 3 (ISE Method) finds the average reduction of the EMC based on an individual storm assessment and excluding BDL events was 71.3%,with a 95% confidence interval of [63.9%, 77.1%]. 3.3.2 Performance Comparison Chart The performance comparison chart, shown in Figure 2, provides a visual summary of the expected EMC reductions based on both methods, the effluent probability (EP) method and the individual storm efficiency (ISE) method. In this particular example, both methods yield very similar results, ranging from approximately 70 to 80% average reduction estimate.


Page 17

Figure 2: Performance Comparison Chart

3.3.3 Box & Whisker Chart The box & whisker chart, shown in Figure 3, provides a graphic representation of the distribution of the two EMC data sets and Figure 4 presents the interpretive elements and nomenclature of a box and whisker chart. Examining the results shown in Figure 3 reveals the distinctive differences in the two distributions. The two distributions appear to be very similar in shape, with the outflow distribution shifted downwards. The presence of an outlier value (>100µg/l) in the outflow distribution should be noted by the user.


Page 18

Figure 3: EMC Box & Whisker Chart


Page 19

Figure 4: Box & Whisker Nomenclature

0.1

1

10Ev

ent M

ean

Conc

entra

tion

(mg/

l)

Upper Fence

Lower Fence

Outlier

3rd Quartile

1st Quartile

Geometric Mean

Upper 95% Confidence Limit

Lower 95% Confidence Limit


Page 20

3.3.4 Probability Chart The probability chart is shown in Figure 5. This chart plots the EMC values for both distributions against their computed deviation from their respective means and is another visual presentation of the EP method. The strongly linear plot supports the assumption of a log-normal distribution. This is further supported by the clustering of both data sets within 2 standard deviations of the mean. It should be noted that this chart excludes outlier values as no value greater than 60µg/l is shown. Figure 5: EMC Probability Chart


Page 21

3.3.5 Time Series Chart The time series chart is shown in Figure 6. This chart plots the EMC values for both monitoring stations against their respective dates. Although this is not a “statistical” chart, it can yield insights not otherwise provided by a statistical evaluation. For example, Figure 6 shows a gap in the monitoring data of more than six months. It should be noted that this chart also excludes outlier values. Figure 6: EMC Time Series Chart


Page 22

3.3.6 Efficiency Chart The efficiency chart is shown in Figure 7. This chart presents the computed individual storm efficiencies (ISE) over the period of record. Two ISE data series are shown, one includes BDL results, and the other excludes them. The example shown exhibits a strong clustering around the 80% reduction level. This chart excludes outlier values; though in this case outliers are based on the computed efficiency series (see the Paired Data tab of the output spreadsheet). The observed outflow EMC value 100+µg/l yielded a computed removal efficiency of -618% which was identified as an outlier value in the efficiency data series. Figure 7: Efficiency Chart


Page 23

3.3.7 Efficiency Probability Chart The efficiency probability chart is shown in Figure 8. This chart presents the ranked probabilities for both ISE series along with the expected values based on the transformed log-normal distribution described in Section 3.2.4. Again, the outlier value of -618% was excluded from this chart. Figure 8: Efficiency Probability Chart


Page 24

3.3.8 Correlation Chart The correlation chart is shown in Figure 9. This chart presents the inflow and outflow data for the individual storm events, along with a regression line and the regression coefficient (R2). The R2 value is a measure of how much the inflow EMC value affects the outflow EMC value and varies from 0 to 1, with a perfect correlation between the two yielding an R2 value of 1. The example in Figure 8 has an R2 value of 0.0006, which indicates that the variation in the inflow accounts for virtually none of the observed variation in the outflow. Figure 9: Correlation Chart


Page 25

However, it should be noted that one value appears to be an outlier, with an outflow EMC of over 100 µg/l. Unlike the previous charts and calculations, the EMC Analysis Tool does not currently adjust the correlation data sets to account for outliers! Fortunately, the Paired Data tab in the output spreadsheet file contains all the data and the outlier can be deleted from the chart, resulting in the revised correlation shown in Figure 10. The revised R2 value of 0.1043 indicates more than 10% of the variation in the outflow EMC is accounted for by the inflow EMC. It can also be seen that the values are positively correlated, thus the revised R value is 0.323 (computed using Excel’s CORREL function) as opposed to the value of -0.025 shown at the bottom of Table 3. Figure 10: Revised Correlation Chart


Page 26

3.3.9 Inflow/Outflow Data Tabs The inflow and outflow data used in the analyses are copied into their associated data tabs in the output spreadsheet. These tabs allow the user to confirm, or modify the analyses and charts, or to perform additional assessments of the data. At the top of each tab are a few brief tables providing information about the selected monitoring stations and parameter, summary statistics and distributional estimates. An example is shown in Table 4. Table 4: Inflow/Outflow Data Tab – Summary Tables

Inflow Data

Site ID: FDEP-64 Mon Sta ID: FDEP-64-01 Parameter: CU-T

Summary Statistics

Value LN

(Value) Arith. Est.

Total Count 54 Data Count 54 BDL Count 1

# Outliers 0 Average 20.829 2.887 21.063 Median 18.550 2.920 17.932

Std. Dev. 11.507 0.567 12.977 CV 0.552 0.197 0.616

Distribution Estimates Confidence Limits P Value N Dist LN Dist Trans LN EXP 0.100 6.082 8.668 2.160 95% t 2.007 0.250 13.068 12.231 2.504 CI 0.155 0.500 20.829 17.932 2.887 Std Error 0.077 0.750 28.590 26.291 3.269 Conf. Inter. 0.155 0.900 35.575 37.099 3.614 Cox LCL 2.880 17.821 IQR 15.522 14.060 0.765 Cox UCL 3.215 24.894

Following these summary tables are the data records extracted from the database and used in the analysis. Interim calculation values are also included such as the substitute value for BDL results (see Section 3.2.1), the log transformed values, computed plotting probability and standard normal deviate. Outlier EMC values are relocated below the remaining data records. An example is shown in Table 5.


Page 27

Table 5: Inflow/Outflow Data Tab – Data Tables

Sorted Data

Sample Date-time Database

Value Detection

Limit Is

BDL? Robust Value

LN (Robust Value)

Plot Prob.

Std. Norm. Dev.

Flow Type

Sample Media

Sample Type

11 07/13/01 00:00 1.000 2.000 Yes 0.691 -0.369 0.023 -1.991 R 1 1 12 07/21/01 00:00 1.000 2.000 Yes 0.904 -0.101 0.047 -1.680 R 1 1 15 08/05/01 00:00 1.000 2.000 Yes 1.077 0.074 0.070 -1.478 R 1 1 21 07/01/02 00:00 1.150 2.300 Yes 1.231 0.208 0.093 -1.322 R 1 1

(remaining rows not shown for brevity)

20 06/29/02 00:00 2.300 1.000 No 2.300 0.833 0.209 -0.809 R 1 1 31 12/12/02 00:00 9.500 1.000 No 9.500 2.251 0.907 1.322 R 1 1 30 12/09/02 00:00 11.300 1.000 No 11.300 2.425 0.930 1.478 R 1 1 35 04/25/03 00:00 11.600 1.000 No 11.600 2.451 0.953 1.680 R 1 1

Outliers

Sample Date-time Database

Value Detection

Limit Is

BDL? Robust Value

LN(Robust Value)

Plot Prob.

Std. Norm. Dev.

Flow Type

Sample Media

Sample Type

40 07/11/03 00:00 102.000 1.000 No 102.000 4.625 R 1 1

The Flow Type, Sample Media and Sample Type are shown using their code values from the BMP Database. Under Flow Type, there are three possible entries: R for surface runoff, B for baseflow, or blank (which indicates there is no flow data associated with that sample). Sample Media and Sample Type are coded based on the following: Code Sample Media Sample Type

0 Groundwater <unused> 1 Surface Runoff/Flow Flow weighted composite EMC 2 Soil Time weighted composite EMC 3 Dry Atmospheric Fallout Un-weighted composite EMC 4 Wet Atmospheric Fallout Grab Sample 5 Pond/Lake Water Sediment Sample 6 Accumulated Bottom Sediment Biological Sample 7 Biological Continuous meter 8 Other Manual meter

3.3.10 Paired Data Tabs The paired inflow and outflow data used in the ISE analyses are copied into the Paired Data tab in the output spreadsheet. This tab allows the user to confirm, or modify the analyses and charts, or to perform additional assessments of the data. At the top of this tab are a few brief tables providing information about the monitoring stations, parameter and summary statistics. An example is shown in Table 6.


Page 28

Table 6: Paired Data Tab – Summary Tables

Paired Data Inflow MSID: FDEP-64-01

Outflow MSID: FDEP-64-02 Parameter: CU-T

Paired Data Summary Statistics

Stat Value (Incl BDLs)

Stat Value (Excl BDLs)

Data Count 41 34 # Outliers 1 1

Average 74.6% 71.3% Std. Dev. 80.7% 81.3%

CV 24.1% 35.1% Cox UCL 80.0% 77.1% Cox LCL 67.7% 63.9%

I/O Correl -0.025 -0.072 Following these summary tables are the data records extracted from the database and used in the analysis. Interim calculation values are also included such as the substitute value for BDL results (½ the MDL, since robust methods cannot be used), the computed storm efficiencies and the transformed efficiency values (see Section 3.2.4). An example of this data table is shown in Table 7. Table 7: Inflow/Outflow Data Tab – Data Tables

Inflow Data Outflow Data Removal

Efficiency

Date-time Database

Value MDL Is

BDL? Value Used Date-time

Database Value MDL

Is BDL?

Value Used Eff.

Trans Eff

12/18/00 00:00 40.20 1.0 No 40.20 12/18/00 00:00 7.40 1.0 No 7.40 81.6% 0.184 01/08/01 00:00 12.90 1.0 No 12.90 01/08/01 00:00 5.50 1.0 No 5.50 57.4% 0.426 03/04/01 00:00 8.90 1.0 No 8.90 03/04/01 00:00 9.20 1.0 No 9.20 -3.4% 1.034


07/11/03 00:00 14.20 1.0 No 14.20 07/11/03 00:00 102.00 1.0 No 102.00 -618.3% 7.183 08/27/03 00:00 18.20 1.0 No 18.20 08/27/03 00:00 1.50 3.0 Yes 1.50 91.8% 0.082 09/19/03 00:00 38.70 1.0 No 38.70 09/19/03 00:00 7.35 1.0 No 7.35 81.0% 0.190


Page 29

Following the data tables are the sorted and transformed efficiency tables. These tables include the computed efficiency (transformed), the computed plotting probability and standard normal deviate along with the expected efficiency based on a log-normal distribution of the transformed efficiency values (see Section 3.2.4). There are two of these tables, one which includes the BDL values while the other excludes them. Any outlier efficiency values (again based on a log-normal distribution of the transformed efficiency values) are moved below these data tables. An example is shown in Table 8. Table 8: Inflow/Outflow Data Tab – Efficiency Probability Tables

Sorted Transformed Data (Including BDLs)

Sample Date-time Eff. Trans.

Eff.

LN (Trans.

Eff.) Plot

Prob.

Std. Norm. Dev.

Expected Eff.

10 07/13/01 00:00 94.0% 0.060 -2.821 0.024 -1.971 94.6% 15 08/07/01 00:00 93.4% 0.066 -2.715 0.049 -1.657 93.4% 11 07/21/01 00:00 93.2% 0.068 -2.681 0.073 -1.453 92.4% 23 08/17/02 00:00 92.6% 0.074 -2.601 0.098 -1.296 91.5%


29 12/09/02 00:00 36.5% 0.635 -0.454 0.902 1.296 51.4% 17 06/24/02 00:00 29.4% 0.706 -0.348 0.927 1.453 46.0% 18 06/28/02 00:00 18.5% 0.815 -0.205 0.951 1.657 38.0% 3 03/04/01 00:00 -3.4% 1.034 0.033 0.976 1.971 23.4%

Outliers (Including BDLs)

Sample Date-time Eff. Trans.

Eff.

LN (Trans.

Eff.) Plot

Prob.

Std. Norm. Dev.

Expected Eff.

39 07/11/03 00:00 -618.3% 7.183 1.972


Page 30

4. Recommendations Ideally, the performance of a BMP would be expressed in terms of mass load reductions. However, mass loads can be surprisingly difficult to accurately characterize and it is extremely highly unlikely that any set of monitored flows and volumes would be representative of average annual conditions, since event size, frequency and inter-event time are all highly variable factors with significant influence on this data. As a result, it is recommended that a better approach for mass load assessment might be to develop a long-term water balance model of the BMP to estimate the expected annual volume reduction. In addition, if there is a statistically significant difference in the measured inflow and outflow EMCs, a total BMP mass efficiency could be determined using the modeled volumes and the EMC statistics. The performance of BMPs in regards to EMC reductions should be evaluated using both the EP method and the ISE method.

• The assessment of BMP performance should be limited to the water quality parameters for which sufficient data exists in the Database, and which commonly occur in stormwater. This includes common nitrogen and phosphorous species, suspended solids, copper, lead and zinc.

• EMC reductions should be evaluated using the EP method to estimate the average difference between the inflow and outflow distributions, along with the computed 95% confidence limits as a measure of uncertainty.

• EMC reductions should also be evaluated using the ISE method to estimate the average expected EMC reduction, along with the computed 95% confidence limits as a measure of uncertainty.

• Consideration should be given to the potential for some BMPs to exhibit a consistent effluent quality rather than a consistent EMC reduction.

• Consideration should be given to the potential for some BMPs to exhibit a minimum effluent quality which may influence the observed EMC reductions.

• Mass load reductions should be estimated using long-term water balance modeling combined with the statistical assessment of EMCs.

• The data developed from the EMC reductions for each BMP should be aggregated by BMP category to determine the expected reductions and uncertainty by type of BMP.

F:\Projects\FDEP Projects\TMDL Contract\Year 3\004.04 BMP Database\3-Analytical Methods\Draft\BMP Performance Assessment Analytical & Statistical Methods TM Ver 2.doc


Page 31

References Burton, G.A. Jr., Ph.D., Pitt, R.E., Ph.D., P.E., 2001, Stormwater Effects Handbook - A Toolbox for Watershed Managers, Scientists, and Engineers. Helsel, D.R. and Hirsch, R.M., September 2002, Techniques of Water-Resources Investigations of the United States Geological Survey, Book 4, Hydrologic Analysis and Interpretation, Chapter A3: Statistical Methods in Water Resources, U.S. Geological Survey Olsson, Ulf, 2005, Confidence Intervals for the Mean of a Log-Normal Distribution, Journal of Statistics Education, Volume 13, Number 1. Strecker, E.W., 1998. Considerations and Approaches for Monitoring the Effectiveness of Urban BMPs. Proceedings of the National Conference on Retrofit Opportunities for Water Resources Protection in Urban Environments. 65-82 Strecker, E., Quigley, M. and Urbonas, B., 2001. Determining Urban Stormwater BMP Effectiveness. Journal of Water Resources Planning and Management, ASCE. 127(3), 175-185 United States Environmental Protection Agency, July 2, 1999. Determining Urban Stormwater Best Management Practice (BMP) Removal Efficiencies: Task 3.1 – Development of Performance Measures Technical Memorandum. Washington, District of Columbia. United States Environmental Protection Agency, June 25, 2000. Determining Urban Stormwater Best Management Practice (BMP) Removal Efficiencies, Task 3.4 - Final Data Exploration and Evaluation Report, Washington, District of Columbia. United States Environmental Protection Agency. July 2000. Determining Urban Stormwater Best Management Practice (BMP) Removal Efficiencies: Task 1.1 – National Stormwater BMP Database Data Elements. Washington, District of Columbia. United States Environmental Protection Agency, March 2001. National Stormwater Best Management Practices Database User’s Guide, Release Version 1.2, Washington, District of Columbia. United States Environmental Protection Agency. April 2002. Urban Stormwater BMP Performance Monitoring - A Guidance Manual for Meeting the National Stormwater BMP Database Requirements, EPA-821-B-02-001, Washington, District of Columbia.

bmp analytical & statistical methods tm · technical support services (year 3) contract no....

Documents