endurance array current data and qa/qc ac1vi1es · • ea began creating scripts in fall 2015 for...

EnduranceArrayCurrentDataandQA/QCAc1vi1es

ChristopherWingard,CraigRisien,RussellDesiderio,StuartPearce

EnduranceDataTeamDataTeamLead•  ChristopherWingard–Datatransfers,ingestion,QA/QC,annotations,andassetmanagement.

DataTeam•  CraigRisien–QA/QC,in-depthassessments,andcommunications.

•  RussellDesiderio–assetmanagement,QA/QC,andin-depthassessments.

•  StuartPearce–Glideroperationsanddatamanagement.

OOIFBandDDCICommitteeMeetingMay2019 2

Spending~1.0FTEeffortonDataQA/QCinOOI2.0(Oct2018-Apr2019)

EnduranceDataTeamResponsibili1es•  AssetManagement

•  Enteringrequiredmetadata(cal,deploymentinfo)intoOOINet.•  DataIngestion/Availability

•  Ingesting/re-ingestingdataintoOOINet.•  EnsuringdataUlowfromplatformtoshoretoscienceusers.

•  DataQC•  Identifying/resolvingissueswithinstruments/data.•  ImplementingautomatedandHITLdataQC.

•  Annotation•  AnnotatingdatainOOINetforinstrument/dataissues,metadatachanges.

•  SupportingCI(workinggroups,enhancements)andusers(HelpDesk)


AssetManagement•  Documentingandformalizingassetmanagementprocesses

•  EAbegancreatingscriptsinFall2015foraccuracyandefUiciency.•  Scriptshaveevolvedasprogramneedshavechanged.•  Institutedformal,two-personcheckofallmetadataenteredforEndurance11(Spring2019).

•  CollaboratingondocumentationofSOPsandcodeused,includingsharinganddocumentingautomatedprocessesforcapturingcalibrationcoefUicients.

•  CriticalMetadatareviewofallpastEndurancedeployments•  Createdformalprocessandcodeforreview.•  Endurance10(Fall2018)reviewcompleted;workingbackwards.•  WillcompletebyendofJuly2019.


DataInges1on/Availability(TimeAvailability)•  AvailabilityAssessmentfor1.0and1stQuarterof2.0

•  DevelopedcodetoassessdataavailabilityinOOIviaM2Minterface,cross-comparingwithdataavailableininternalOMS++system.

•  IdentiUiedandbackUilledseveraldatasets,complexiterativeprocessinvolving:•  Identifyingissue/causesformissingdata•  Determiningcorrectiveaction•  Implementationofcorrectiveaction•  Re-runninganalysis

•  Example:METBK–IdentiUied125daysofmissingdata(2.46%oftotal)duetoparserbug.Purgedandre-ingesteddatawithupdatedparser.

•  Example:NUTNR–IdentiUied1365daysofmissingdata(16.72%oftotal)duetomissingparser(newinstrumentversion).Setupingestsandranwithnewparser.

•  WorkwillbecompletedbyendofPY1.•  CodesharedwithallMIOs.


DataInges1on/Availability(CE02SHSM)CE02SHSM UFrame OMS++ %Difference StartDate EndDate TotalDays Difference

(days) IngestID Comments

FDCHPBUOY 0.00 0.00 0.00 4/2/2015 9/30/2018 1277 0.00

N/A

TherewasanannotationassociatedwiththeFDCHPthatpreventedanydatafrombeingdownloadedfromuFrame.IclearedthatannotationandsettheappropriateQA/QCUlagonthemotion-correctedandUluxmeasurements.Thenon-correcteddataisOKandshouldbeavailabletointerestedusers.

METBKBUOY 95.23 96.09 0.86 4/2/2015 9/30/2018 1277 10.99PCO2ABUOY 96.24 96.24 0.00 4/2/2015 9/30/2018 1277 0.00VELPTBUOY 96.32 96.32 0.00 4/2/2015 9/30/2018 1277 0.00WAVSSBUOY 90.77 90.77 0.00 4/2/2015 9/30/2018 1277 0.00

ADCPTNSIF 94.60 0.70 -93.90 4/2/2015 9/30/2018 1277 -1199.06

CTDBPNSIF 86.15 94.91 8.76 4/2/2015 9/30/2018 1277 111.91

13840

Therewasanissuewithpower-cyclingDCL27duringD00007.Thisresultedingapindatacollectionthatexceededthetelemeteredingesttimeoutwindow(30days).OnceweUixedtheissueanddatastartedUlowingagain,theingestsshouldhavebeenrestarted.Ire-rantheingestsforthisdeploymentandhaveUilledinthegap.

DOSTANSIF 86.15 94.91 8.76 4/2/2015 9/30/2018 1277 111.91

13841


FLORTNSIF 73.63 83.41 9.78 4/2/2015 9/30/2018 1277 124.90

13843


NUTNRNSIF 71.91 94.21 22.30 4/2/2015 9/30/2018 1277 284.78

13844

Therewasanissuewithpower-cyclingDCL27duringD00007.Thisresultedingapindatacollectionthatexceededthetelemeteredingesttimeoutwindow(30days).OnceweUixedtheissueanddatastartedUlowingagain,theingestsshouldhavebeenrestarted.Ire-rantheingestsforthisdeploymentandhaveUilledinthegap.AdditionalgapduetotransitiontoSUNAstartingwithD0007

OPTAANSIF 79.81 49.61 -30.20 4/2/2015 9/30/2018 1277 -385.68

13842


PHSENNSIF 83.26 82.24 -1.02 4/2/2015 9/30/2018 1277 -12.99SPKIRNSIF 98.44 98.51 0.08 4/2/2015 9/30/2018 1277 1.00VELPTNSIF 98.36 98.36 0.00 4/2/2015 9/30/2018 1277 0.00


DataInges1on/Availability(GliderDAC)“ThemissionoftheGliderDACistoprovideglideroperatorswithasimpleprocessforsubmittinggliderdatasetstoacentralizedlocation,enablingthedatatobevisualized,analyzed,widelydistributedviaexistingwebservicesandtheGlobalTelecommunicationsSystem(GTS)andarchivedattheNational

CentersforEnvironmentalInformation(NCEI).”https://gliders.ioos.us/data/

•  Assumedresponsibility(StuartPearce)in2.0fortransmittingOOIEndurancegliderdatatotheGliderDAC.

•  ReceivedGliderDACcodewhichneedednewcodetoaddadditionalsciencedatavariables(newfeaturecomingtotheGliderDAC).

•  GliderDACcodeforEnduranceOOIgliderdataiscompleteandrunning.AllEndurancegliderdeploymentswillbeuploadedtotheGliderDACbyendofmonth,May2019.

•  AutomatingrealtimeprocessinganduploadingofdatatotheGliderDACinprogress.

•  WorkingtogeneralizecodeanddocumentationforPioneer,Globalandpossiblyothergliderusers.


DataQA/QC•  OperationsandDataMonitoring:Implementingformalbi-weeklyUieldoperationsreview,assignmentofengineering&dataactions,andupdatingOMS++infrastructureandinstrumentalarming.Informaldailyassessments.•  Addressing/identifyingissuesviaOOIDataTeamworkinggroups(QARTOD,ADCP,WaterSampling,Ticketing,etc.).•  DevelopedtoolsforpullingdatafromM2MtocreatedifferentvisualizationsforHITLQA/QC(builtoffTimeAvailabilityAssessments).•  MooringmeteorologicalandsurfacewavedatapushedtoNDBC.Automatedroutinescomparedatatonearbysource(s):•  StationIDs:46097(CE02SHSM),46098(CE04OSSM),46099(CE07HSM)and46100(CE09OSSM).


DataQA/QC• DevelopedcodetoprocessdataMMPdatasets,creatingsinglemergeddatasets.WillbeusedtogenerateNetCDFUilesforOOIERDDAPserver.• UpdatingcodeusedtogeneratedatasetsforOMS++toincludeADCPdatafromallmoorings,andaddingallinductivemodemhostedinstruments.• Completedin-depthanalysisofuncabledproUilersystems,identifyingcriticalerrorsintreatmentofsourcedataand(timeandpressure)andmergingofdatasets.Redmineticketsgenerated.


EnduranceAccomplishmentsToDate

•  Transitioningrolesandresponsibilitieswithingrouptoadapttonewrole.•  AttendedOOIDataTeamWorkshopatUWinJanuary2019.•  Completedre-ingestionofbacklogassignedby1.0DataTeamandhavemadesigniUicantprogresstowardcompletionofdataavailabilityassessment.

•  Participatedincreationofacommonformat/spreadsheetforOOIWaterSamplingdataacrosstheprogram.•  Allpastdataisprocessedandenteredintostandardizedspreadsheetswithexceptionofsalinitysamples.

•  WillcompleteUinal,“secondsetofeyes”reviewofspreadsheetsbyendofJune2019withuploadtoAlfrescoascomplete.

•  SuccessfulAssetManagementupdatesandDataIngestionfor:•  Endurance10(Fall2018);Endurance11a,11b(Spring2019);multiplegliderandCSPPcruises


EndurancePlanGoingForward•  Bi-WeeklymeetingstoidentifytaskingforMooringandDataOperations.•  Completecriticalmetadatareview,endofJuly2019.•  In-depthreviewofexistingannotations.•  EvolutionofOMS++systemtoimprovemooringanddatamonitoring(alerts&alarms,datavisualization).•  EngagingSMEstoconUirm/determinebestpracticesanddatareviewmethodology.•  Continuetodevelopandimplementautomatedprocessesfordatareviews.•  ContinueOOIDataTeamworkonprogram-wideissues/processes.


endurance array current data and qa/qc ac1vi1es · • ea began creating scripts in fall 2015 for...

Documents