meteorological assimilation data ingest system (madis) and iso data quality ted habermann noaa...
TRANSCRIPT
Meteorological Assimilation Data Ingest System (MADIS) and ISO Data QualityTed HabermannNOAA National Data Centers MADIS observations on April 29, 2004
MADIS and ISO MetadataMADIS produces and distributes thousands of quality reports for stations in U.S. meteorological mesonets every week. See http://madis.noaa.gov/ for MADIS information.
The ISO 19115 Metadata Standard includes an extensive section devoted to data quality reports.
How can these fit together?
MADIS mesonet stations in the North-American domain on
May 26, 2004
Generating ISO Metadata
RichInventory
MADIS TrackingNMMR
Existing Quality Resources
Existing Metadata Resources
ISO Generator
ISOMetadata
RESTViews
This schematic shows our approach. Existing quality resources are queried using a web service (REST) for the “beef” of the quality report. The “bun” is provided from views into existing FGDC or new ISO sources. These two are combined into a complete ISO report.
The “Beef”
The “Bun”
Data Quality Granularity
MADIS integrates observations from stations in many meteorological mesonets around the U.S.
The MADIS Quality Repository includes:nine statistics for six parameters For thousands of stations from~150 data networks every week.
Quality reports can be generated at any level (or granularity) from this hierarchy.
ISO Quality GranularityDQ_Scope+ level: MD_ScopeCode+ extent [0..1] : EX_Extent+ levelDescription [0..*] : MD_ScopeDescription
MD_ScopeDescription+ attributes : Set<GF_AttributeType>+ features : Set<GF_FeatureType>+ featureInstances : Set<GF_FeatureType>+ attributeInstances : Set<GF_AttributeType>+ dataset : CharacterString+ other : CharacterString
+ nonGeographicDataset+ dimensionGroup+ feature+ featureType+ propertyType+ fieldSession
MD_ScopeCode+ attribute+ attributeType+ collectionHardware+ collectionSession+ dataset+ series
+ software+ service+ model+ tile
ISO Quality Granularity
In the geographic jargonan ISO feature is a station andan ISO attribute is a parameter
MD_ScopeDescription+ attributes : Set<GF_AttributeType>+ features : Set<GF_FeatureType>+ featureInstances : Set<GF_FeatureType>+ attributeInstances : Set<GF_AttributeType>+ dataset : CharacterString+ other : CharacterString
MD_ScopeCode = dataset (data network in this case)
We considered several options for expressing report granularity in ISO and ended up with a general approach that seems to make sense (at least as a first draft):
= list ofparameters
= list of stations
Select Report
Select threeweeks
Select threestations
Select twoparameters
http://ngdc.noaa.gov/idb/struts/form?t=102711&s=14&d=35,36,38
The HTML View
Three weeks
Two Parameters
Three Stations
The Quality Information
The “Beef”
Three weeks
Two Parameters
Three Stations
The Quality Information
The ISO Report
<scope><DQ_Scope>
<level><MD_ScopeCode>dataset</MD_ScopeCode>
</level><levelDescription>
<MD_ScopeDescription><dataset><gco:CharacterString>IODOT</gco:CharacterString></dataset>
</MD_ScopeDescription>/levelDescription><levelDescription>
<MD_ScopeDescription><attributeInstances><gco:CharacterString>SLP,THETA</
gco:CharacterString></attributeInstances></MD_ScopeDescription>
</levelDescription><levelDescription>
<MD_ScopeDescription><featureInstances><gco:CharacterString>RAMI4,RAVI4,RBUI4</
gco:CharacterString></featureInstances></MD_ScopeDescription>
</levelDescription></DQ_Scope>
</scope>
ISO Report Scope
Two Parameters
Three Stations
One Network
ISO Temporal Extent
Week One
Week Two
The ISO Quantitative Report<DQ_QuantitativeResult>
<value><gco:Record>
"Start Date","End Date","Data Network","Station","Parameter","Total Count","Bad Count","Accum_Sum","Accum_Sum2","Percent Bad","Mean","RMS“
"2007-01-22", "2007-01-28","IODOT","RAMI4","THETA",294,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RAVI4","THETA",282,5,36.96,276.1,1.77,7.39,7.43"2007-01-22", "2007-01-28","IODOT","RBUI4","THETA",308,0,0,0,0,0,0"2007-01-22", "2007-01-28","IODOT","RBUI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAMI4","THETA",314,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAVI4","THETA",291,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RBUI4","THETA",316,0,0,0,0,0,0"2007-01-29", "2007-02-04","IODOT","RBUI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAMI4","THETA",671,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAMI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAVI4","THETA",633,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RAVI4","SLP",0,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RBUI4","THETA",622,0,0,0,0,0,0"2007-02-05", "2007-02-11","IODOT","RBUI4","SLP",0,0,0,0,0,0,0
</gco:Record></value>
</DQ_QuantitativeResult>
The Quality Information
Conclusions• Yes, MADIS Quality Information can be reported as ISO 19115
• The ScopeCode is general with details in the LevelDescription
• We need to learn about describing feature and attribute sets in Feature Catalogs. At present we are using comma separated lists.
• We need to learn about how to describe the formats of reports. At present we are using CSV (comma separated values)