HARRIS COUNTY WATERSHED PROTECTION GROUP

Lessons Learned onStorm Water QualityResearch ProjectsPresented by:Trent Martin

July 15, 2008

Statistical Significance Our Holy Grail

Statistical Significance Our Holy Grail POWER

Statistical Significance Our Holy Grail POWERThings that affect POWER: * variability

Statistical Significance Our Holy Grail POWERThings that affect POWER:

* variability

* level of significance

Statistical Significance Our Holy Grail POWERThings that affect POWER:

* variability

* level of significance

* size of difference between sets

Statistical Significance Our Holy Grail POWERThings that affect POWER: * variability

* level of significance

* size of difference between sets

* number of samples collected

Quality Assurance Project Plan (QAPP)

Its OK to ask for help!

K.I.S.S. - Keep it simple. Answer one question at a time.

Any Questions?For more information: Harris County Watershed Protection Group(713) 290 - 3000 orwww.CleanWaterways.org

Physical Address: 9800 Northwest Frwy., Suite 305, Houston, TX 77092

*Most people think of field sampling as something fun, adventuresome. Its good to get out of the office for a change. But field sampling is like working on your own car. In the old days, it was easy to change the belts and hoses, or replace the thermostat. Now, its hard to even locate the thermostat! And if you try to work on the car, youre more likely to make things worse. In much the same way, field sampling has gotten much more sophisticated, and for your data to count, there are a lot of things to consider before grabbing a sample bucket and going to the field.State and federal agencies have collected most of the data upon which resource management decisions are based. Their data is scant, but important decisions rest on their data. TMDLs, water quality standards, use designations. There is a great need for more complete data to supplement existing data, and, rather than wait for the state to collect sufficient data to help you out in your situation, you might want to fill this data gap yourself. Lets talk about some of the things to consider before heading out into the field.

*Statistical significance is the Holy Grail for any study. We want to be able to say that something is different from another thing or that it is not. Statistics allows us to do that. *The core for statistical analysis is the statistical power which is the ability to distinguish differences. Power is affected by several things. *The first thing that affects power is the variability of the results. If all results fall within a narrow range, it is easy to spot a difference. But if the range is all over the board, it is almost impossible to spot trends. Guess where storm water is?*The next element of power is the level of significance required. Do we need to tell the difference between samples beyond a reasonable doubt, or is preponderance of evidence good enough? For most storm water work we look at knowing something is different with 90% confidence. Others may use a 95% level.*What is the size of difference between sample sets? Is one set very low and the other high or are both sets pretty much the same. Obviously, the greater the difference between two sets of samples, the easier it is to find a statistical difference.

*The last thing affecting the power to distinguish between samples is the number of samples collected. A sample size of 1,000 provides much more power than a sample size of 5. The more samples, the better, of course. But samples cost money, so a balance has to be struck.

*For data to be widely accepted by the regulatory and academic communities, and the community-at-large, you should develop a Quality Assurance Project Plan. This Plan includes a description of the work, outlines chain-of-command, which items you sample for, and how records will be kept. Think of this as preparing a case for trial. If there is no QAPP, the results from your work will be of little value. The QAPP should be in a format provided by your state agency and should ideally be reviewed by your state agency. It takes a long time to prepare the QAPP and have it reviewed. Be prepared to wait.

*Local governments are good at building roads, jails, and libraries, and law enforcement, but not so much at environmental monitoring. Nobody expects perfection from you. Ask for assistance from experts in this area. Each state has a land grant university through which is offered extension services. These are a great resource. Call your old professor. Call your COG. Someone will be able to direct you to the right person who will be willing and able to help design your study. If these efforts fail, there are environmental and engineering consultants who will be able to help. Just remember to add these costs into your sampling budget.

*Keep it simple. Try to answer one question at a time. Its tempting when you start sampling to say, I think we should test for how the north side is different from the south side, and how results might vary season-to-season, and how automated samplers differ from sampling with a bucket. With each variable you add to a sample regimen, you double the number of samples needed to see statistical significance, cutting your statistical power by half. The more questions you try to answer with your monitoring program, the more complicated it will get. Its easy and very frustrating to have things start to unravel, to lose track of what lab results go with which sample. Keep things tight, stay within your defined scope, and follow your QAPP exactly.

*Having your own dataset is very helpful, when it comes to understanding your storm sewer system. You can tailor your study specifically for your needs, and dont have to wait for the state, the feds, or the river authority to come up with the data. Harris County has studied the effectiveness of BMPs on improving water quality, from public education to detention basins. Weve studied how E. coli bacteria behaves, and we understand better how to address it. Weve studied sewage treatment plant discharges, in an effort to understand their impacts on our waterways. You see, if we are being asked to spend millions of dollars to fix water quality problems, wed better understand the best way to proceed. We have used results from our studies to make for more equitable permit parameters in our MS4 permit, and to help develop better models for TMDLs which will strongly affect our MS4 permit. Water quality studies have enormous benefit to us as MS4 operators, and benefit the regulatory agencies and interested stakeholders. Studies arent cheap, and, if they arent carefully designed and implemented, they are wasted effort. When they are well executed, they help policymakers and the public for many years to come.

*Having your own dataset is very helpful, when it comes to understanding your storm sewer system. You can tailor your study specifically for your needs, and dont have to wait for the state, the feds, or the river authority to come up with the data. Harris County has studied the effectiveness of BMPs on improving water quality, from public education to detention basins. Weve studied how E. coli bacteria behaves, and we understand better how to address it. Weve studied sewage treatment plant discharges, in an effort to understand their impacts on our waterways. You see, if we are being asked to spend millions of dollars to fix water quality problems, wed better understand the best way to proceed. We have used results from our studies to make for more equitable permit parameters in our MS4 permit, and to help develop better models for TMDLs which will strongly affect our MS4 permit. Water quality studies have enormous benefit to us as MS4 operators, and benefit the regulatory agencies and interested stakeholders. Studies arent cheap, and, if they arent carefully designed and implemented, they are wasted effort. When they are well executed, they help policymakers and the public for many years to come.

**To find more information about community projects and how to get involved please give us (Harris County Storm Water Quality) a call at (713) 290-3000 or visit our website www.CleanWaterClearChoice.org