metadata kim owens – noaa’s ocean service mike moeller – noaa coastal services center...

Metadata

Kim Owens – NOAA’s Ocean ServiceMike Moeller – NOAA Coastal Services Center

Understanding the Valueand Importance of Proper

Data Documentation

First things first

Introductions

Logistics

Issues/questions

Presentation Outline

• The What and the Why

• The Value of Metadata

• The FGDC Content Standard for

Digital Geospatial Metadata (CSDGM)

• Writing Quality Metadata

What is Metadata?

What is Metadata?

Metadata is information about your data

Therefore, the metadata describes the characteristics (content, location,

structure, quality, condition, etc.) of the data set.

http://www.lungusa.org/data/art/data_head2.jpg

This is the metadata for this.

What’s Missing?

Emily and M

adison

This is the metadata for this.

While the card-catalog entry is a form of metadata, it does not address topics such as quality, accuracy, or scale.

Well-written geospatial metadata describes these and many more aspects of the data.

Rodale's illustrated encyclopedia of herbs ISBN: 087596964x (pbk.) : $17.95ISBN: 0878576991 : $24.95Title: Rodale's illustrated encyclopedia of herbs /

Claire Kowalchik & William H. Hylton, editors ; writers, Anna Carr ... [et al.].

Publication info: Emmaus, Pa. : Rodale Press, c1987.Physical descrip: vi, 545 p. : ill. (some col.) ; 24 cm.General Note: Includes index.Subject term: Herbs.Subject term: Herbs--Utilization.Subject term: Herb gardening.Subject term: Herbs--History.Subject term: Herbals.Added author: Kowalchik, Claire.Added author: Hylton, William H.Added author: Carr, Anna, 1955-Added author: Rodale Press.

Rodale's illustrated encyclopedia of herbs ISBN: 087596964x (pbk.) : $17.95ISBN: 0878576991 : $24.95Title: Rodale's illustrated encyclopedia of herbs /

Claire Kowalchik & William H. Hylton, editors ; writers, Anna Carr ... [et al.].

Publication info: Emmaus, Pa. : Rodale Press, c1987.Physical descrip: vi, 545 p. : ill. (some col.) ; 24 cm.General Note: Includes index.Subject term: Herbs.Subject term: Herbs--Utilization.Subject term: Herb gardening.Subject term: Herbs--History.Subject term: Herbals.Added author: Kowalchik, Claire.Added author: Hylton, William H.Added author: Carr, Anna, 1955-Added author: Rodale Press.

This is

Identification_Information Citation Citation_Information Originator: NOAA, NESDIS Publication_Date: 20030929 Title: Hurricane Isabel Storm Surge Geospatial_Data_Presentation_Form: Remote Sensing Image/Map Publication_Information Publication_Place: Camp Springs, MD Publisher: NOAA, NESDIS, SSD Larger_Work_Citation Citation_Information

Identification_Information Citation Citation_Information Originator: NOAA, NESDIS Publication_Date: 20030929 Title: Hurricane Isabel Storm Surge Geospatial_Data_Presentation_Form: Remote Sensing Image/Map Publication_Information Publication_Place: Camp Springs, MD Publisher: NOAA, NESDIS, SSD Larger_Work_Citation Citation_Information

the metadata for this.

Metadata

A Component of Data

Proper data documentation provides vital information to

interested parties.

A Component of Data

Metadata is that component of data which describes it.

Environmental Sensitivity Index Data

Metadata

RARNUM - unique combination of species, concentration, and seasonality

CONC (concentration) = Density species is found at location

Season_ID = seasonality code like to the seasonal table

Element - Biology group

A Component of Data

It’s data about a data set.

Title

Scale

Source

Content

Location

Publication

Access

Title

Scale

Source

Content

Location

Publication

Access

MetadataMetadataMetadataMetadata

GIS files

Imagery

Geospatial databases

GPS data

GIS files

Imagery

Geospatial databases

GPS data

Data setData setData setData set

A Component of Data

A Component of Data

Because metadata provides vital information about a dataset, it

should never be viewed or treated as a separate entity.

Metadata

Non-spatial orattributes

SpatialTake Home Message

Metadata is a critical and

integral component of any complete

data set.

Metadata is a critical and

integral component of any complete

data set.

The Value of Metadata

Why Bother with Metadata?


The Current ConceptPrimary external value

• Discovery

• Assessment

• Access

• Use


The Current ConceptPrimary internal value

• “Inheritance”“Properly

documenting a data set is the key to preserving its usefulness

through time.”


An Emerging Concept

An aid to data management

Internal value

• Discovery

• Assessment

• Access

• Use

Additional data management benefits


Data Currency

• Date of last edit/update

• Age of source files

Data Utility

• Track source file usage

• Track distribution frequency


Monitoring Data Development

• Data processing steps

• Status of development

Estimate Development Costs

• Data processing – time and extent

• Source file availability

Additional data management benefits


To realize the full potential of

metadata under this new concept,

metadata creation must become

integral to the data development

process.

The question is “How?”

Make metadata part of the process

Approach metadata development from a business perspective

Build administrative support


• Preserves data investment

• Limits liability

• Helps manage data resources

• Aids in external data acquisition

• Facilitates data access and transfer

• Provides for efficient data distribution

Stress the individual benefits of metadata

Build technical support


• Reduces workload over the long term

• Field fewer data inquiries

• Provides a means of documenting personal contributions

• Facilitates sharing of reliable information

Develop strong staff support


• Incorporate metadata expectations into job descriptions and performance standards

Build technical support

• Provide staff development opportunities The three “T’s”

Training Tools Time

Develop templates to facilitate efficient and consistent metadata creation

Build organizational support


• Identify pertinent fields within the metadata structure

• Populate fixed fields Use standardized language Define distribution methods Cite standards used

• Build source and contact libraries

• Map metadata fields to the work flow

• Establish and assign responsibilities

Distribute the effort


Technicians - lineage Analysts – process and methodology Field Scientists – accuracy assessments I.T. Managers – tools, automated collection methods, information management

• Mandate the use of standards and templates.

• Develop boilerplate metadata deliverable language for data contractors.

• Require publication of metadata.

• Create and publish a metadata SOP to document policies and procedures.

Establish standard policies


Why Have a Standard?Federal mandates and legislation

StandardizedMetadata

Mandates, Policy, and Legislation

The Federal Geographic Data Committee (FGDC)

• Organized in 1990 under the Office of Management and Budget (OMB)

• Promotes the coordinated use, sharing, and dissemination of geospatial data on a national basis

Background


“ All Federal agencies must document all Geospatial data that they collect or produce, either directly or indirectly, using the FGDC Content Standard for Digital Geospatial Metadata (CSDGM), and to make that standardized documentation electronically accessible to the FGDC Clearinghouse network.”

President Clinton, 1994

Executive Order 12906:

“ All Federal agencies that collect, produce, acquire, maintain, distribute, use, or archive analog or digital spatial data in the fulfillment of their mission, financed directly or indirectly, in whole or part, by Federal funds are covered by this requirement.”

OMB Circular A-16 (revised)http://www.whitehouse.gov/omb/circulars/a016/a016_rev.html

OMB Circular A-16 (revised)


The Data Quality ActSecion 515 of the Treasury and General

Government Appropriations Act for Fiscal Year 2001 directs OMB to issue government-wide

guidelines that:

“. . . provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies.”

http://www.noaanews.noaa.gov/stories/iq.htm


Why Have a Standard?

The standard for

metadata ensures a level

of consistency in

data documentation.

Standards ensure consistency.

Why Have a Standard?

Think for a moment how hard it would be to….

… bake a cake without standard units of measurement… put gas into your car without standard nozzle sizes… plug a lamp into a socket without standard electrical outlets

http://work-at-home-mom.20m.com/members/measure.jpg

http://www.rpi.edu/dept/advising/esl/chemistry/chemistry/vocabulary/chemistry_objects/electrical_outlet.jpg

http://images.google.com/imgres?imgurl=www.nr.state.ky.us/nrepc/dep/daq/images/gasnozzle.gif&imgrefurl=http://www.nr.state.ky.us/nrepc/dep/daq/outreach/AirTips.html&h=455&w=529&prev=/images%3Fq%3Dgas%2Bnozzle%26svnum%3D10%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26sa%3DN

The Content Standard utilizes...

• Common terms

• Common definitions

• Common language

• Common structure Access

constraints

Citation

currentness

entity

attrib

ute

domain

lineage

Process step

Establishing a Standard

The Content Standard helps

the user determine...

• If a set of geospatial data is available, fit for a particular use.

• How to access and transfer the data set.


WhoWho collected the data?Who processed the data?Who wrote the metadata?Who to contact for questions?Who to contact to order?Who owns the data?

WhereWhere were the data collected?Where were the data processed?Where are the data located?

WhatWhat are the data about?What project were they collected under?What are the constraints on their use?What is the quality?What are appropriate uses?What parameters were measured?What format are the data in?

WhenWhen were the data collected?When were the data processed?

HowHow were the data collected?How were the data processed?How do I access the data?How do I order the data?How much do the data cost?How was the quality assessed?Why

Why were the data collected?

Metadata written using the Content Standard answers these important

questions:


Details About the Sections and Terms

of FGDC Metadata Standard

FGDC Metadata Standard

All About the Standard

FGDC Content Standard for Digital Geospatial Metadata (CSDGM)

Defines the 334 metadata elements and their

associated production rules.

“The Workbook”

The Content Standard is organized using numbered chapters called “sections.”

There are 7 main sections3 supporting sections.

Each section is organized into series of elements that define the

content required to document your geospatial data set.

Organization of the Content Standard

Section

Data Element

Warm up Exercise

Tagging the Sections of the Standard

CSDGM- 7 Main Sections

1. Identification_Information: (p. 34)

General bibliographic information about data set:

• title, originator, data contact, status, date, abstract, purpose, keywords, geographic location

2. Data_Quality_Information: (p. 44)

Lineage and data assessments • sources, process methods, accuracy,

data processing contact


3. Spatial_Data_Organization_InformationData format: (p. 56)

• vector, point, raster

4. Spatial_Reference_InformationCoordinate system parameters: (p. 60)

• horizontal / vertical coordinate system, projection, datum


5. Entity_and_Attribute_Information: (p. 75)

Database design

• entities, attributes, domains, description of data values

6. Distribution_Information: (p. 81)

How to acquire the data

• distribution contact, available formats, online distribution website, liability, costs


7. Metadata_Reference_Information: (p. 88)

General information about the metadata record itself

• metadata contact, metadata standard used, metadata creation date, metadata review date

CSDGM- 3 Supporting Sections

8. Citation_Information: (p. 91)

• originator, title, publication date, publisher, online linkage, larger work

9. Time_Period_of_Content: (p. 95)

• single date, multiple dates, range of dates

10. Contact_Information: (p. 96)

• contact person/organization, address, phone, email

Exercise 1

Reading A Metadata File

Rules of the Metadata Game

Learning how to read the structure of the standard

DataQuality

Information

SpatialData

OrganizationInformation

SpatialReference

Information

Entityand

AttributeInformation

4 52 6 731

Metadata

The Three Supporting Sections

9 Time PeriodInformation

10 Contact

Information

8 Citation

Information

DistributionInformation

MetadataReference

Information

IdentificationInformation

Organization of the Content Standard

The Seven Main Sections

Interpreting the Graphical Production Rules

The workbook uses graphics to illustrate the production rules of the standard. These graphics include most of the information provided by the production rules, including:

How elements are grouped What is mandatory and what is not What elements can repeat and how many

times they can repeat


SectionSections are depicted by this symbol.

Compound Element

Compound elements are depicted using a 2-dimensional box.

Data Element

Data elements are depicted using a 3-dimensional box

with shadow.


Data Element

A data element is a logically primitive itemof data. Data elements are the things thatyou “fill in.”

The form for a data element is:

Data element name -- definition.Type: (choice of “integer”, “real”, “text”, “date”, or “time”)Domain: (describes valid values that can be assigned)

An example of a data element is:

Abstract -- a brief narrative summary of the data set.Type: textDomain: free text

Note: Data element definitions are containedin the text of the Content Standard,

not in the graphical production rules.


Turn to page 17 in workbook

Mandatory - must be provided.

MeaningData

ElementCompound

Element

What’s Mandatory? What’s Not?

Mandatory if Applicable - must be provided if the data set exhibits

the defined characteristic.

Optional - provided at the discretion of the data set producer.

If an element can be repeatedindependently from other elements, itwill be indicated as such below theelement name.

Repeating Elements

CompoundElement 1

(can be repeatedunlimited times)

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

This group of elements

would repeat.

Compound Element 1 Compound Element 1.1 Data Element 1.1.1 Data Element 1.1.2 Data Element 1.2

See page 34, under Keywords

Using the Graphics to Make Decisions

All elements are colored yellow, so all are mandatory and must be reported.

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

Compound Element 1 is mandatory.

Compound Element 1.1 is optional.

If yes, Data Elements 1.1.1 and 1.1.2 are mandatory.

If no, do not report Compound Element 1.1, Data Element 1.1.1 or 1.1.2, and skip to Data Element 1.2.

Data Element 1.2 is mandatory.

CompoundElement 1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2

CompoundElement 1.1


Compound Element 1 is mandatory.

Compound Element 1.1 is mandatory.

Data Element 1.1.1 is mandatory.

Data Element 1.1.2 is mandatory if

applicable.

Data Element 1.2 is optional.

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2


Compound Element 1 is mandatory if applicable. If not applicable to the data set, do not report any elements. If applicable, it is mandatory and:

Compound Element 1.1 is mandatory.

Data Element 1.1.1 is mandatory if applicable. If not applicable,

do not report it. If applicable, it is mandatory.

Data Element 1.1.2 is mandatory.

Data Element 1.2 is optional.

CompoundElement 1

CompoundElement 1.1

DataElement 1.1.1

DataElement 1.1.2

DataElement 1.2


Exercise 2

Using The Workbook

The FGDC Metadata

Clearinghouse

Metadata as aData Discovery

Tool

The FGDC metadata clearinghouse is a decentralized system of Internet

servers you can use to search for available geospatial data.

Discovering Data Through Metadata

Client

FGDCGateway

Servershousingmetadata

A Brief Look at the FGDC Clearinghouse

The FGDC has 6 gateways

to its clearinghouse,

with accessto over 300 spatial data

servers.

www.fgdc.gov/clearinghouse/clearinghouse.html


Searches can be performed by place names or by

using a map interface.

The new NSDI Search Wizard bins servers by the types of metadata they house.


Searches can be performedusing a map interface that

allows the user to define

an area of interest.


An area of interest can be defined bydragging an area of interest box

on the map interface.

The selected area defines the bounding coordinates that will be used in the

search.


You can search all the servers listed,

or you can select only those that

interest you.


Individual servers are selected by holding the

Ctrl key down and selecting with the mouse.


Search criteria can be further refined by

time period of content

and keywords.



The status of each selected

node is displayed as

search is conducted.


When the search is

complete, the status window lets you know

if you were successful in discovering

metadata that matched your

search criteria.


Select a server

to see what

metadata is

available.


Metadata

found by the

search is

displayed by

title.


Metadata record

returnedin HTML format.

Links take you to each of the seven main sections of

the record.

• Similar to the FGDC Clearinghouse

• Searches for servers that house metadata of a “Coastal” nature

Coastal Information Directory (CID) – NOAA/CSC

http://www.csc.noaa.gov/CID

For more information on the clearinghouse system, visit the FGDC Web site (www.fgdc.gov).

Here you can find information on howto establish your own clearinghouse node

using free Isite software.

On-line tutorials provide assistance for setting up and configuring this software.


Exercise 3Search for metadata

www.fgdc.gov/clearinghouse/clearinghouse.html

www.csc.noaa.gov/CID/

WritingMetadata

It’s not THAT bad!

• First records are the hardest.• Not all fields may need to be filled in.• Tools are available.• Can often be produced automatically.• Can (and should) be reviewed

for updates.

Document your data as you go.

Writing Metadata

Before you begin writing,get organized.

Writing Metadata

Write so others can understand.

Writing Metadata

Always review your document.

Writing Metadata

• Write simply but completely.

• Document for a general audience.

• Be consistent in style and terminology.

Keep your readers in mind.

Writing Metadata

• Define all acronyms.

• Avoid using jargon.• Clearly state data limitations.

Writing Metadata

Keep your readers in mind.

Write a complete Title that includes:

• What• Where• When• Scale• Who

Writing Metadata

The title is critical in helping others find your data.

Greater Yellowstone Rivers from 1:126,700 Forest Visitor Maps (1961-1983)

Writing Metadata

Be specific. Quantify when you can.

Vague: We checked our work and it looks complete.

Specific: We checked our work using 3 separate sets of check plots reviewed by 2 different people. We determined our work to be 95% complete based on these visual inspections.

Writing Metadata

Select your key words wisely.

• Use unambiguous words.

• Use descriptive words.

• Fully qualify geographic locations.

Writing Metadata

• Have someone else read it.

• If you’re the only reviewer, put it away and read it again later.

• Check for clarity and omissions.

Review your final product.

Writing Metadata

• Can a novice understand what you wrote?

• Are your data properly documented for posterity?

When you review your work, ask:

Writing Metadata

• Does the documentation present all the information needed to use or reuse the data?

• Are any pieces missing?

When you review your work, ask:

Writing Metadata

Writing Metadata

Your audience may be very diverse.

Considerwriting your metadata to reflect this diversity.

Metadata Creation

and Validation

Tools of the Trade

Metadata Tools

Some available tools for metadatacreation, validation, and publication.

• CNS and MP“Chew ‘n spit”- checks and corrects structural errors, and “Metadata Parser”- checks for errors in element compliance; “mp batch” and “mp online tool”

• Template tools CSC’s metaScribe create large numbers of similar records.

• ArcView Tools Extension for ArcView 3.x

ArcCatalog for ArcGIS 8.x

• TKMEText editor used for metadata entry.

TKME, CNS and mp are

FREE downloads!

http://geology.usgs.gov/tools/metadata

See document, “Downloading and Installing CNS and MP”

Tools of the TradeTKME - An editor for formal metadata, TKME is

intended to simplify the process of creating metadata that conform to the FGDC Standard.

- Hierarchical structure - Proper arrangement

Maintains:

of elements

Tools of the Trade – Creating metadata

• “Add” element names from dropdown menu

When right side turns white, you know to input info (import, cut & paste, etc.)


Help Menu:

- Version (how to use, helpful hints

- Element (definition of element)

- Output (final output look)

- Fonts (choose different fonts

Help Menu


• Double click on Tkme icon from desktop

• Go to File, Open, and navigate to C:\Metadata\ benthic_bad.txt

Tools of the Trade- Creating metadata

NOAA CSC ArcView ® Metadata Collector

ArcGIS metadata collector

Found in ArcCatalog, this tool allows the user

to write metadata within the Arc environment.

Tools of the Trade

Tools of the Trade

MetaScribeThis new tool was also developed by NOAA CSC to

aid in the creation of multiple sets of metadata that

exhibit a high degree of redundancy.

CNS (“Chew ‘n Spit”)

A pre-parser for formal metadata designed to assist metadata managers convert records

that cannot be parsed by mp into records that can be parsed by mp.

Tools of the Trade - Validation Tools

MP (Metadata Parser)

A compiler to parse formal metadata, checking the syntax against the FGDC Content Standard for Digital

Geospatial Metadata and generating output suitable for viewing with a web browser or text editor.

TKME, CNS, and MP are available as free downloads from the United States Geological Survey (USGS) Website. (geology.usgs.gov/tools/metadata)

TKME will run from a shortcut on the desktop

Both MP and CNS can be run from:• command line in MS-DOS or UNIX• MP Batch tool and • MP online interface

Tools of the Trade

Tools of the Trade

• MP Batch, Integraph®

Check multiple records at one time for CNS and MP compliance

http://imgs.intergraph.com/smms/default.asp

• SMMS Metadata Software

Variety of metadata tools (but it cost)

http://www.intergraph.com/gis/support/ > Free Utilities/Tools

• MP Online Tool, Peter Schweitzer (USGS)

User friendly interface for MP (no command line)

http://www.nature.nps.gov/im/units/mwr/gis/metadata/metadata_tools.htm

• NPS Metadata Tools/Extensions - ArcGIS

• ESRI Metadata Customizations

http://imgssupport.intergraph.com/Tools.asp

- ArcCatalog extension; 5 new buttons (MP, editor, organizational, input/export capabilities)

- Spell checker: understands element names, underscores

- Advanced synchronization: turn on/off different sections so it’s not “automatically updated” (Entity & Attributes)

Tools of the Trade

Optional Exercise

• MP exersise• MP Batch• MP online tool

Tools of the Trade

http://www.fgdc.gov/standards/status/textstatus.html

• NBII Biological Data Profile

• Remote Sensing Extension

FGDC Profiles and Extensions to the CSDGM

• Shoreline Data Profilehttp://www.csc.noaa.gov/metadata/shoreline_profile.html

http://www.fgdc.gov/standards/status/sub5_2.html

http://www.fgdc.gov/standards/status/csdgm_rs_ex.html

http://www.fgdc.gov/standards/status/sub5_6.html

http://www.nbii.gov/datainfo/metadata/standards/index.html

Keyword Lists / Controlled Vocabularies / Thesauri

• Global Change Master Directory's (GCMD)

http://gcmd.gsfc.nasa.gov/Resources/valids/index.html

• Integrated Taxonomic Information System

http://www.itis.usda.gov/

• Master Environmental Library

http://mel.dmso.mil/docs/metadata_guide/section_6.htm

• Aquatic Science and Fisheries Abstracts

http://www4.fao.org/asfa/asfa.htm

• Geographic Names Information System (GNIS)

http://geonames.usgs.gov/

Resources: CSC’s Metadata Website: http://www.csc.noaa.gov/metadata

• Metadata Standards

• How to start writing metadata with Metadata Bob

• Metadata tools

• Metadata Forum

• Metadata Training Materials

. . . and much, much more!

Featuring:

Resources: NOS Internal Website

• NOS Metadata Program

• Metadata in our Everyday Lives

• Metadata: What and Why?

• The FGDC Metadata Standard

• The FGDC Clearinghouse

• Metadata Tools

. . . and much, much more!

Featuring:

https://inside.nos.noaa.gov/foremployees/it/metadata/welcome.html

Finally...

Remember, metadata is a legacy document that

concisely sums up your data or data set.

Without metadata, your data set is

incomplete.

Optional Exercise:

Get Started!

Create a record using Tkme or ArcCatalog

Kim [email protected]

Mike [email protected]

metadata kim owens – noaa’s ocean service mike moeller – noaa coastal services center...

Documents