Upload File

data delivery and real me data quality flags

1
Changing dynamics of water quality and nutrient monitoring through the use of long term una6ended deployments, real 9me data delivery and real 9me data quality flags Nichole Halsey , Corey Koch, Bruce Rhoades Adam Dutton Sea-Bird Scientific Doug Wilson Caribbean Wind LLC Changing Dynamics of Water Quality and Nutrient Monitoring HydroCycle-PO4 Phosphate Sensor Range: PO 4 3- : 2.3-1200 µgP/L Accuracy: PO 4 3- : ~4.6 µgP/L (extensive grab sample in field operation) Endurance: PO 4 3- : 9-18 weeks (hourly- bihourly interval) Field service: PO 4 3- : optics clean, new reagent cartridges, filters ABSTRACT The Cycle-PO4 was designed for estuarine research, but has been adopted by many water quality sensor users. The latest version, the HydroCycle-PO4, offers an improved plaEorm to provide a wet chemical instrument for con9nuous monitoring in both fresh and coastal waters. The HydroCycle-PO4 has increased reliability over its predecessor by maximizing instrument up9me. Fluidics have been advanced to provide >90% up9me free of bubbles for opera9on in high oxygen satura9on environments. Filter life has been extended to minimize clogging and increase the 9me between service intervals. These improvements permit phosphate measurement in a wider variety of condi9ons and provide an instrument be6er suited for applica9ons such as agricultural runoff monitoring. A challenge in serving and visualizing real-9me data from in-situ sensors is qualifying the data. With “Quality Assurance of Real-Time Ocean Data” as a guide, we have developed sensor-specific Quality Control (QC) flags to alert users of Suspect/High Interest, Bad, or Good data. Flags are based on algorithms on-board the sensor that use sensor specific meta-data and site defined thresholds to qualify sensor performance and data quality for each datum. Examples include out- of-range, low signal, etc. So]ware is used to inves9gate High Interest data and provide quality sta9s9cs. Qualified data can be presented to the community in real 9me. Ul9mately, flag levels can be used to help automate classifica9on of data so stakeholders can make be6er decisions and scien9sts can quickly iden9fy quality data for use in analysis and models. Pilot collec9on of automa9cally QC’ed phosphate data is happening now in a number of sites. Users are studying phosphorus from 9dal, estuarine, and terrestrial sources, requiring high frequency observa9ons to understand ecosystem impacts. Mechanisms by which automated QC are performed, how quality meta-data can be used in the data management, web- publishing, visualiza9on, and the significance of real-9me phosphate will be presented. The HydroCycle-PO4 is a wet chemical sensor engineered for environmental monitoring to enable valid, scien9fically defensible results that lead to be6er management recommenda9ons. A NIST traceable on-board standard, factory calibra9on check, a 2.3 µgP/L detec9on limit, and an accurate calibra9on mean you can trust data, line up grab sample records, and swap sensors with minimal offsets. Autonomous opera9ons, low-power, five month reagent stabilityenable extended deployments in remote loca9ons. Over 1000 samples between services, up to four samples per hour and both SDI-12 and RS232 permit capturing higher frequency phosphate data than before. The HydroCycle-PO4 has increased reliability over its predecessor, the Cycle-PO4, and many wet-chemical sensors, by maximizing data reliability and sensor up9me. Fluidics have been advanced to provide > 90% up9me free of bubbles for data quality in high oxygen satura9on environments and to ensure it stabilizes rapidly a]er deployment. Filter life has been extended to minimize clogging and enhance data quality during high sediment events. Real Time Quality Control Flags This system of automated nutrient sensors will deliver high-frequency, quality-controlled nutrient data to scien9sts, managers, and policy makers to help understand natural and anthropogenic influenced coastal nutrient dynamics throughout the region. Quality control (QC)flags provide confidence in the data quality and up9me so users can know their instrument is working properly. QC flags can help speed up troubleshoo9ng, help jus9fy published data, and permit users to spend more 9me on advanced data analysis rather than reviewing data to separate good data from bad. Using this system of QC flags each data point receives a general quality flag (good, suspect, or bad) in real 9me. This flag is a composite of the six individual quality flags. Those flags include: out-of-range measurement, op9cal noise, low signal, bubbles and calibra9on and mixing spikes. The included so]ware can be used to inves9gate high interest data and provide quality sta9s9cs. Data is automa9cally color coded Green (Good), Yellow (Suspect), Red (Bad) and Blue (Not Evaluated). TradiEonal Method: Grab Sample Site ~ 12 – 24 measurements per year Current Monitoring Trend: Real Time Site with Nutrient Sensors ~ 315,000 measurement per year Water is a finite resource vital for human life. It is a key ingredient in agriculture, manufacturing, energy crea9on, and transporta9on. Currently our aqua9c ecosystems are threatened, locally and globally by a variety of pollutants as result of popula9on growth, industrial growth and large-scale agriculture. Point and non-point sources include domes9c sewage, agricultural run-off and industrial effluents. The rate and impact of nutrient pollu9on is growing and is contribu9ng to accelerated eutrophica9on in lakes and impounded rivers. Impacts to aqua9c systems can occur in a wide range of 9me scales and based on a number of factors. NERRS Myrtle Beach Deployment A Cycle-PO4 sensor was deployed in the Withers Swash, in late winter of 2013 through the summer. It was mounted in a cage and deployed along the bo6om at a site by Family Kingdom. A LOBO system, including a Cycle-P Phoshate sensor, was deployed at Withers Swash, SC, part of the North Island – Winyah Bay Na9onal Estuarine Research Reserve. Figure 1 shows PO4 readings taken over the first half of 2013.All data are shown with red data points highligh9ng values deemed invalid by automated analysis within the Cycle-P sample processing system. Similar analysis is now included and output by Cycle instruments along with real 9me data. Conclusions • An addi9onal feature of the Quality Control flags is that they can be applied to any Cycle data - Here we post-processed data from a 2013 deployment before we had developed QC flags or improved the sensor’s filters • Over 4113 samples—6 months of con9nuous hourly data—73% of the data was good, 10% suspect, and 17% of the data was bad - Unit serviced on 3/12, 4/22, and 6/4 - Some bad data in March and June due to running out of reagent and latent reagent priming issues - The majority of the bad data is due to the bubble flag on April 1 that goes un9l May, then shorter stretches of bad data in mid-June and mid-July • Consistent bad bubble flags halfway through the deployment is a sign that filters need to be serviced. A]er changing the filters the bad data becomes good again - The event had several suspect flags and intermi6ent bad flags on 3/31 that, if available in 2013, would have alerted us to change the filters and not lose data quality for the month of April - The regular frequency of filter clogging in this system is about every 4-5 weeks during the produc9ve season, now HydroCycle-PO4's filters last over 3X longer, but we should have considered changing our service schedule! Myrtle Beach is a popular tourist site in a 60 mile stretch of beaches that support a multi-billion dollar tourism industry in South Carolina. In this area much of the storm water flows in to tidal creeks known as swashes that travers the beaches and flow directly in to the ocean. Because of this local scientists and regulators were interested in studying how swashes collect, transform, and export the nutrients from the runoff in to the coastal ocean waters. Overall QC Bubble Spike Flag QC COV Flag QC Low Signal Flag QC Out Of Range Flag QC Mixing Spike Flag QC Cal Spike Flag sample count 4113 4113 4107 4107 3737 4023 603 % Good 73 86 93 100 97 83 96 % Suspect 10 2 7 0 2 8 2 % Bad 17 12 0 0 1 9 2 Overall QC Bubble Spike Flag QC COV Flag QC Low Signal Flag QC Out Of Range Flag QC Mixing Spike Flag QC Cal Spike Flag sample count 4113 4113 4107 4107 3737 4023 603 % Good 73 86 93 100 97 83 96 % Suspect 10 2 7 0 2 8 2 % Bad 17 12 0 0 1 9 2 Why the wide data range? Withers Swash is characterized by free 9dal exchange with the Atlan9c Ocean through an open channel crossing the beach. The drainage basin is also significantly affected by stormwater runoff. Figure 2 shows the first deployment (JFM) PO 4 (as well as NO 2+3 from a coincidentally located SUNA Nitrate Sensor) as a func9on of High Water / Low Water as well as local precipita9on, showing the strong influences of both ocean flooding and stormwater runoff. With this level of variability, having access to real 9me data quality informa9on is vital to maintaining con9nuous confidence in the validity of the data.

Upload: others

Post on 09-Jun-2022

4 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: data delivery and real me data quality flags

Changingdynamicsofwaterqualityandnutrientmonitoringthroughtheuseoflongtermuna6endeddeployments,real9me

datadeliveryandreal9medataqualityflags

Nichole Halsey, Corey Koch,

Bruce Rhoades Adam Dutton

Sea-Bird Scientific Doug Wilson

Caribbean Wind LLC

ChangingDynamicsofWaterQualityandNutrientMonitoring

HydroCycle-PO4PhosphateSensor

Range: PO4

3- : 2.3-1200 µgP/L Accuracy: PO4

3- : ~4.6 µgP/L (extensive grab sample in field operation) Endurance: PO4

3- : 9-18 weeks (hourly-bihourly interval) Field service: PO4

3- : optics clean, new reagent cartridges, filters

ABSTRACTTheCycle-PO4wasdesignedforestuarineresearch,buthasbeenadoptedbymanywater

qualitysensorusers.Thelatestversion,theHydroCycle-PO4,offersanimprovedplaEormtoprovideawetchemicalinstrumentforcon9nuousmonitoringinbothfreshandcoastalwaters.TheHydroCycle-PO4hasincreasedreliabilityoveritspredecessorbymaximizinginstrumentup9me.Fluidicshavebeenadvancedtoprovide>90%up9mefreeofbubblesforopera9oninhighoxygensatura9onenvironments.Filterlifehasbeenextendedtominimizecloggingandincreasethe9mebetweenserviceintervals.Theseimprovementspermitphosphatemeasurementinawidervarietyofcondi9onsandprovideaninstrumentbe6ersuitedforapplica9onssuchasagriculturalrunoffmonitoring.Achallengeinservingandvisualizingreal-9medatafromin-situsensorsisqualifyingthedata.With“QualityAssuranceofReal-TimeOceanData”asaguide,wehavedevelopedsensor-specificQualityControl(QC)flagstoalertusersofSuspect/HighInterest,Bad,orGooddata.Flagsarebasedonalgorithmson-boardthesensorthatusesensorspecificmeta-dataandsitedefinedthresholdstoqualifysensorperformanceanddataqualityforeachdatum.Examplesincludeout-of-range,lowsignal,etc.So]wareisusedtoinves9gateHighInterestdataandprovidequalitysta9s9cs.Qualifieddatacanbepresentedtothecommunityinreal9me.Ul9mately,flaglevelscanbeusedtohelpautomateclassifica9onofdatasostakeholderscanmakebe6erdecisionsandscien9stscanquicklyiden9fyqualitydataforuseinanalysisandmodels.Pilotcollec9onofautoma9callyQC’edphosphatedataishappeningnowinanumberofsites.Usersarestudyingphosphorusfrom9dal,estuarine,andterrestrialsources,requiringhighfrequencyobserva9onstounderstandecosystemimpacts.MechanismsbywhichautomatedQCareperformed,howqualitymeta-datacanbeusedinthedatamanagement,web-publishing,visualiza9on,andthesignificanceofreal-9mephosphatewillbepresented.

TheHydroCycle-PO4isawetchemicalsensorengineeredforenvironmentalmonitoringtoenablevalid,scien9ficallydefensibleresultsthatleadtobe6ermanagementrecommenda9ons.ANISTtraceableon-boardstandard,factorycalibra9oncheck,a2.3µgP/Ldetec9onlimit,andanaccuratecalibra9onmeanyoucantrustdata,lineupgrabsamplerecords,andswapsensorswithminimaloffsets.Autonomousopera9ons,low-power,fivemonthreagentstabilityenableextendeddeploymentsinremoteloca9ons.Over1000samplesbetweenservices,uptofoursamplesperhourandbothSDI-12andRS232permitcapturinghigherfrequencyphosphatedatathanbefore.TheHydroCycle-PO4hasincreasedreliabilityoveritspredecessor,theCycle-PO4,andmanywet-chemicalsensors,bymaximizingdatareliabilityandsensorup9me.Fluidicshavebeenadvancedtoprovide>90%up9mefreeofbubblesfordataqualityinhighoxygensatura9onenvironmentsandtoensureitstabilizesrapidlya]erdeployment.Filterlifehasbeenextendedtominimizecloggingandenhancedataqualityduringhighsedimentevents.

RealTimeQualityControlFlagsThissystemofautomatednutrientsensorswilldeliverhigh-frequency,quality-controllednutrientdatatoscien9sts,managers,andpolicymakerstohelpunderstandnaturalandanthropogenicinfluencedcoastalnutrientdynamicsthroughouttheregion.Qualitycontrol(QC)flagsprovideconfidenceinthedataqualityandup9mesouserscanknowtheirinstrumentisworkingproperly.QCflagscanhelpspeeduptroubleshoo9ng,helpjus9fypublisheddata,andpermituserstospendmore9meonadvanceddataanalysisratherthanreviewingdatatoseparategooddatafrombad.UsingthissystemofQCflagseachdatapointreceivesageneralqualityflag(good,suspect,orbad)inreal9me.Thisflagisacompositeofthesixindividualqualityflags.Thoseflagsinclude:out-of-rangemeasurement,op9calnoise,lowsignal,bubblesandcalibra9onandmixingspikes.Theincludedso]warecanbeusedtoinves9gatehighinterestdataandprovidequalitysta9s9cs.Dataisautoma9callycolorcodedGreen(Good),Yellow(Suspect),Red(Bad)andBlue(NotEvaluated).

TradiEonalMethod:GrabSampleSite~12–24measurementsperyear

CurrentMonitoringTrend:RealTimeSitewithNutrientSensors~315,000measurementperyear

Waterisafiniteresourcevitalforhumanlife.Itisakeyingredientinagriculture,manufacturing,energycrea9on,andtransporta9on.Currentlyouraqua9cecosystemsarethreatened,locallyandgloballybyavarietyofpollutantsasresultofpopula9ongrowth,industrialgrowthandlarge-scaleagriculture.Pointandnon-pointsourcesincludedomes9csewage,agriculturalrun-offandindustrialeffluents.Therateandimpactofnutrientpollu9onisgrowingandiscontribu9ngtoacceleratedeutrophica9oninlakesandimpoundedrivers.Impactstoaqua9csystemscanoccurinawiderangeof9mescalesandbasedonanumberoffactors.

NERRSMyrtleBeachDeploymentACycle-PO4sensorwasdeployedintheWithersSwash,inlatewinterof2013throughthesummer.Itwasmountedinacageanddeployedalongthebo6omatasitebyFamilyKingdom.ALOBOsystem,includingaCycle-PPhoshatesensor,wasdeployedatWithersSwash,SC,partoftheNorthIsland–WinyahBayNa9onalEstuarineResearchReserve.Figure1showsPO4readingstakenoverthefirsthalfof2013.Alldataareshownwithreddatapointshighligh9ngvaluesdeemedinvalidbyautomatedanalysiswithintheCycle-Psampleprocessingsystem.SimilaranalysisisnowincludedandoutputbyCycleinstrumentsalongwithreal9medata.

Conclusions•  Anaddi9onalfeatureoftheQualityControlflagsisthattheycanbeappliedtoanyCycledata-Herewepost-processeddatafroma2013deploymentbeforewehaddevelopedQCflagsorimprovedthesensor’sfilters•  Over4113samples—6monthsofcon9nuoushourlydata—73%ofthedatawasgood,10%suspect,and17%ofthedatawasbad

-Unitservicedon3/12,4/22,and6/4-SomebaddatainMarchandJuneduetorunningoutofreagentandlatentreagentprimingissues-ThemajorityofthebaddataisduetothebubbleflagonApril1thatgoesun9lMay,thenshorterstretchesofbaddatain

mid-Juneandmid-July•  Consistentbadbubbleflagshalfwaythroughthedeploymentisasignthatfiltersneedtobeserviced.A]erchangingthefiltersthebaddatabecomesgoodagain

-Theeventhadseveralsuspectflagsandintermi6entbadflagson3/31that,ifavailablein2013,wouldhavealertedustochangethefiltersandnotlosedataqualityforthemonthofApril

-Theregularfrequencyoffilterclogginginthissystemisaboutevery4-5weeksduringtheproduc9veseason,nowHydroCycle-PO4'sfilterslastover3Xlonger,butweshouldhaveconsideredchangingourserviceschedule!

RESULTS

Measurementsofnutrientsprovidedformoreaccuratees9matesofphosphateloadings.Threemethodsare

comparedbasedon:1)Cycle-PO4measurement,2)LinearInterpola9onofgrabsamples,and3)Linearregressionofgrab

sampleswithdischarge.

Myrtle Beach is a popular tourist site in a 60 mile stretch of beaches that support a multi-billion dollar tourism industry in South Carolina. In this area much of the storm water flows in to tidal creeks known as swashes that travers the beaches and flow directly in to the ocean. Because of this local scientists and regulators were interested in studying how swashes collect, transform, and export the nutrients from the runoff in to the coastal ocean waters. Cycle-PO4 Linear

Interpolation Discharge

Regression Aug Load (kg) 5.8 7.8 8.4 Sept Load (kg) 12.5 13.1 10.4

0

2

4

6

8

10

12

14

Load

(kg/

mon

th)

OverallQCBubbleSpikeFlag QCCOVFlag

QCLowSignalFlag

QCOutOfRangeFlag

QCMixingSpikeFlag

QCCalSpikeFlag

samplecount 4113 4113 4107 4107 3737 4023 603

%Good 73 86 93 100 97 83 96%Suspect 10 2 7 0 2 8 2

%Bad 17 12 0 0 1 9 2

OverallQCBubbleSpikeFlag

QCCOVFlag

QCLowSignalFlag

QCOutOfRangeFlag

QCMixingSpikeFlag

QCCalSpikeFlag

samplecount 4113 4113 4107 4107 3737 4023 603%Good 73 86 93 100 97 83 96%Suspect 10 2 7 0 2 8 2%Bad 17 12 0 0 1 9 2

Whythewidedatarange?WithersSwashischaracterizedbyfree9dalexchangewiththeAtlan9cOceanthroughanopenchannelcrossingthebeach.Thedrainagebasinisalsosignificantlyaffectedbystormwaterrunoff.Figure2showsthefirstdeployment(JFM)PO4(aswellasNO2+3fromacoincidentallylocatedSUNANitrateSensor)asafunc9onofHighWater/LowWateraswellaslocalprecipita9on,showingthestronginfluencesofbothoceanfloodingandstormwaterrunoff.Withthislevelofvariability,havingaccesstoreal9medataqualityinforma9onisvitaltomaintainingcon9nuousconfidenceinthevalidityofthedata.

Don’t Secure Routing, Secure Data Delivery

ENES Data Activities: Delivery Plan

Red flags presented in current low back pain guidelines: a ... · ORIGINAL ARTICLE Red flags presented in current low back pain guidelines: a review Arianne P. Verhagen1 • Aron

Gerrit jenkins-big data-continuous-delivery

8339MA-Delivery Data of SHP-06

A Robinson-Schensted algorithm in the geometry of flags for

F5 Signaling Delivery Controller Transaction Data Records · F5 Signaling Delivery Controller Transaction Data ... F5 Signaling Delivery Controller Transaction Data Records ... Origin_Host_Answer

Data structures - Delivery guide

How To Comply With Collateral Data Delivery · 3/16/2010  · CDD (Collateral Data Delivery) initiative May 2009 • Requires lenders to deliver appraisal data prior to loan delivery

Reliable data delivery in large-scale low-power sensor ...mhaenggi/pubs/tosn10.pdf · Reliable Data Delivery in Large-Scale ... Reliable Data Delivery in Large-Scale Low-Power Sensor

Applying Open Source Strategies to Data Center Delivery...Applying an Open Source Model to Data Center Delivery Open Source Delivery Traditional Delivery ... employees contributing

Appraisal Report and Data Delivery Guide

Mobile Healthcare Patient Data Delivery

een the flags!Find the FUN · 2 days ago · een the flags! Find the FUN F ee game wnload! Each level helps you learn about the beach. This summer's hottest new game is Safety Beach,

Modernizing Data Architecture using Data Virtualization for Agile Data Delivery

Reporting: Data Integrity and Data Analysis for Service Delivery Sites

Everything About Tor - Ritter · Consensus Algorithm • Collect all the data • Produce a vote on what params should be, relays included, flags • Post vote to all authorities

Data Spotlight: Improving Healthcare Delivery

Stateful Data Delivery

NSW Cadastral Delivery Model Data Dictionaryspatialservices.finance.nsw.gov.au/__data/assets/pdf_file/0003/... · NSW Cadastral Delivery Model Data Dictionary NSW Cadastral Data Dictionary

DATA SHEET BROCADE … · Application Delivery Application Delivery Controllers for Next-Generation Data Centers BROCADE SERVERIRON ADX 1000, 4000, AND 10000 SWITCHES DATA SHEET

In-Time Data Delivery - Home, Schlumberger

NEON Data Delivery & Cyberinfrastructure · & Cyberinfrastructure (mission, data products, data delivery, processing, ... monitoring a number of critical ecological and environmental

Appraisal Report and Data Delivery Guide - hud.gov · FHA Single Family Housing Appraisal Report and Data Delivery Guide A. Appraisal Report and Data Delivery to FHA 1 APPRAISAL REPORT

Data Delivery Standards and Specifications Template

UNIT-I INTRODUCTION Introduction Data Warehouse Delivery Method ·  · 2013-12-09UNIT-I INTRODUCTION Introduction – Data warehouse delivery method ... Data Warehouse Delivery Method

Open Data API delivery “Open-XDX”

Sparse Matrices and Large Data Issues · – most routines are in Fortran, adapted for spam – balance between readability and overhead – flags for “expert” users. What is

Data Delivery Standards and Specifications Templatetraining.fws.gov/courses/.../geospatial/.../Data-Deliver-Template.pdf · Data Delivery Standards and Specifications Template

Yes, We Are Open - KDM · 2020. 4. 15. · Window graphics Feather flags Yard signs Sidewalk and store front signs Flags and pennants Inflatables Light-up, LED and neon signs Delivery

Unifying Data Delivery Systems Through Data Virtualization

Application Delivery Management - Tech Data

Equinox Data Delivery System

Commercial Real Estate data delivery agreement

Data Delivery - the Evolving Model