h. thiemann / 07.06.2005 / 1 world data centre for climate - wdcc - hannes thiemann 7.6.2005

H. Thiemann / 07.06.2005 / 1

World Data Centre for Climate- WDCC -

Hannes Thiemann

7.6.2005

H. Thiemann / 07.06.2005 / 2

Übersicht

• Einordung in die wissenschaftspolitische Landschaft

• Implementation des WDCC

• Inhalt des WDCC

• Zugriff auf das WDCC

• WDCC Verbund

H. Thiemann / 07.06.2005 / 3

International Council for Science (History)

• Founded 1931

• represents the evolution and expansion of two earlier bodies– International Association of Academies

(IAA; 1899-1914) – International Research Council (IRC; 1919-

1931)

H. Thiemann / 07.06.2005 / 4

International Council for Science (ICSU)

In order to strengthen international science for the benefit of society, ICSU mobilizes the knowledge and resources of the international science community to:

• Identify and address major issues of importance to science and society.

• Facilitate interaction amongst scientists across all disciplines and from all countries.

• Promote the participation of all scientists—regardless of race, citizenship, language, political stance, or gender—in the international scientific endeavour.

• Provide independent, authoritative advice to stimulate constructive dialogue between the scientific community and

governments, civil society, and the private sector.

H. Thiemann / 07.06.2005 / 5

ICSU Activities

• International Geophysical Year (1957-58) • International Biological Programme (1964-74)• International Geosphere-Biosphere Programme: A

Study of Global Change (IGBP) • World Climate Research Programme (WCRP) • DIVERSITAS: An Integrated Programme of

Biodiversity Science

• International Human Dimensions Programme on Global Environmental Change (IHDP).

H. Thiemann / 07.06.2005 / 6

ICSU

ICSU has several interdisciplinary bodies, whose principal focus is the management and use of large scientific data sets: – the Committee on Data for Science and

Technology (CODATA)– the Federation of Astronomical and Geophysical

Data Analysis Services (FAGS)– the Panel of the World Data Centres (WDC)

H. Thiemann / 07.06.2005 / 7

WDC system

• The World Data Center (WDC) system was created to archive and distribute data (…) . Originally established in the United States, Europe, Russia, and Japan, the WDC system has since expanded to other countries and to new scientific disciplines.

• The WDC system now includes 52 Centers in 12 countries. Its holdings include a wide range of solar, geophysical, environmental, and human dimensions data.

H. Thiemann / 07.06.2005 / 8

WDC principles

• WDCs are funded and maintained by their host countries on behalf of the international science community.

• They accept data from national and international scientific or monitoring programs as resources permit. All data held in WDCs are available for no more than the cost of copying and sending the requested information.

H. Thiemann / 07.06.2005 / 9

WDCCStart: Approved in January 2003

Maintenance: Model and Data (M&D/MPIMET) and German Climate Computing Centre (DKRZ)

Mission: Data for climate research are collected, stored and disseminated

ICSU Policy: long-term archiving and unrestricted data access for scientists

Restriction: Only climate data products in CERA DB, no raw data storage.

Content: Emphasis is spent on climate modelling and related data products.

Co-operation: with thematically corresponding data centres like WDC-MARE (Bremen) and WDC-RSAT (Oberpfaffenhofen)

URL: http://www.mad.zmaw.de/wdcc/

H. Thiemann / 07.06.2005 / 10

WDCC + CERA

• WDCC is built upon CERA database system at M&D / DKRZ

H. Thiemann / 07.06.2005 / 11

DKRZ's Archive Increase (Estim. 09.03)

6001200

1920

2640

3360

4080

12 40 184424 664 904

2002 2003 2004 2005 2006 2007

Years

Dat

a A

mo

un

t [T

B]

Unix-File Archive

CERA DB

H. Thiemann / 07.06.2005 / 12

Semantic Data Management

• Data consist of numbers and metadata.• Metadata construct the semantic data context.• Metadata form a data catalogue which makes data searchable.• Data are produced, archived and extracted within their semantic

context.

Data without explanation are only numbers.

Problems:• Metadata are of different complexity for different data types. • Consistency between numbers and metadata have to be

ensured.

H. Thiemann / 07.06.2005 / 13

• Data catalogue

CERA Hierarchy

H. Thiemann / 07.06.2005 / 14

Entry

Reference

Status

Distribution

Contact Coverage

Parameter

SpatialReferenceLocal Adm.

Data Access

Data Org

H. Thiemann / 07.06.2005 / 15

CERA-2 Data Model

• Complete with respect to IEEE’s Reference Model for Metadata (Bretherton, 1994)– Browse, Search and Retrieval– Ingest, Quality Assurance, Reprocessing– Application to Application Transfer– Storage and Archive

• Reference– “The CERA-2 Data Model” (DKRZ-Report No. 15,

1998)– URL: http://www.pik-potsdam.de/dept/dc/e/sdm/cera/

H. Thiemann / 07.06.2005 / 16

Interoperability

• Supports interoperability due to inclusion of international standards– Directory Interchange Format (NASA,

1998)– FGDC Metadata Content Standard (FGDC,

1996)– ISO Metadata Standard for Geographic

Information (ISO 19115)

H. Thiemann / 07.06.2005 / 17

• Data catalogue – “Yellow pages”

• Pointer to primary data

CERA Hierarchy

H. Thiemann / 07.06.2005 / 18

CERA Pointer

• Document the location of model raw data

• To allow users to locate model raw data quickly

• Provide consistency between CERA and the model raw data

H. Thiemann / 07.06.2005 / 19

CERA Hierarchy• Data catalogue – “Yellow pages”

• Pointer to primary data

• Storage of application oriented pre-processed data

H. Thiemann / 07.06.2005 / 20

CERA Data Storage

• CERA offers the storage of application oriented, pre-processed data

• Could be either the storage of– Aggregated data like monthly means– Range of variables in full resolution (1h/6h-

values)

• Can be only done for the most relevant production runs

H. Thiemann / 07.06.2005 / 21

Model Raw DataStructure

Application-orientedData Storage

5D Model Data Structure

6-hour StorageInterval

H. Thiemann / 07.06.2005 / 22

WDC-CLIMATE / Data Content (I)

• Climate Model Data (Continuous stream of new data)

• IPCC DDC (Data Distribution Centre)– Will be continued for the Fourth Assessment Report

• CEOP (Coordinated Enhanced Observing Period) Model output retention and handling Centre – Part of WCRP that was motivated by GEWEX with focus on

water and energy cycles within the climate system (01.10.2002 – 31.12.2004)

H. Thiemann / 07.06.2005 / 23

WDC-CLIMATE / Data Content (II)

• Observational Data

– Model related observations: ERA15/40 (ECMWF), NCEP 40 Y. Reanal.

– Instrumental data: WOCE (World Ocean Circulation Experiment)

– Earth observations: Access to SST's from NOAA AVHRR in cooperation with WDC RSAT (distributed archive)

• Project Support (encourage Good Scientific Practice)

• HOAPS (Hamburg Ocean Atmosphere Parameters and Fluxes from Satellite Data)

• CARIBIC (Civil Aircraft for Regular Investigation of the Atmosphere Based on an Instrumentation Container), MPI Mainz

• Different model applications

H. Thiemann / 07.06.2005 / 24

WDCC: some numbers

• Size of CERA: 170 Tbyte

• Number of experiments: 410

• Number of datasets: 57000

• No. of blobs:4.5 billion

• Downloads last 12 months 500.000

H. Thiemann / 07.06.2005 / 25

User Interface- extract data -

• Graphical User Interface – http://cera-www.dkrz.de

• Batch Interface – http://cera-www.dkrz.de/CERA/

jblob

H. Thiemann / 07.06.2005 / 26

Selection via CERA meta data:

selection of the project

selection of the experiment (=model run)

display of meta data: experiment, quality, datasets

selection of the dataset

display of dataset information

add datasets to “process list”

download to client

H. Thiemann / 07.06.2005 / 27

Batch Interface- jblob -

• Java application to allow for batch download of datasets

• jblob -datasetname name[ -options ]• jblob –showdatasets "search string"( use '%' as

wildcard )• jblob –help• Example:• jblob -datasetname EH4OPYC_SRES_A2_WIND10 -dir /tmp

• See hands on session tomorrow

H. Thiemann / 07.06.2005 / 28

WDC Verbund Erdsystemforschung

Wurde am 25.04.03 von den 3 deutschen ICSU WDC's in Oberpfaffenhofen gegründet.

• WDC for Climate: M&D / DKRZ, Hamburghttp://www.mad.zmaw.de/wdcc/ • WDC MARE (Marine Environmental Sciences): Marum, Bremen und Bremerhavenhttp://www.wdc-mare.org/ • WDC RSAT (Remote Sensing for the Atmosphere): DFD/DLR, Oberpfaffenhofenhttp://wdc.dlr.de/

Verpflichtung: Langzeit-Datenarchivierung und freier, unbeschränkter Datenzugang für alle Wissenschaftler (ICSU Rules for WDC's und Regeln zur guten wissenschaftlichen Praxis)

H. Thiemann / 07.06.2005 / 29

WDC Verbund Erdsystemforschung

Grundsatzerklärung• Datenpublikation- Die Daten selbst sollen unabhängig vom archivierenden System eindeutig Identifizierbar, referenzierbar und universell zugreifbar sein (z.B. Vergabe von DOI's).

- DFG Projekt "Publikation und Zitierfähigkeit wissenschaftlicher Primärdaten" (Beginn 01.10.03)

• Service der Datenzentren- Qualifizierte thematische Datenzentren übernehmen die Rolle für die Archivierung und Publikation von wissenschaftlichen Daten.

- Die Zentren garantieren eine langfristige und freie Verfügbarkeit archivierter Daten im Rahmen der Richtlinien der ICSU Weltdatenzentren.

- Datenzentren stehen mit ihrer Expertise den Fördereinrichtungen, den Gutachtern und der Wissenschaft beratend zur Verfügung.

H. Thiemann / 07.06.2005 / 30

CERA DB using countries

H. Thiemann / 07.06.2005 / 31

Contact

• Email: data@dkrz.de

• WDCC: http://www.wdc-climate.de

• CERA: http://cera-www.dkrz.de/CERA

H. Thiemann / 07.06.2005 / 32

Data Group maintaining the WDCC

Michael Kurtz

Hans Luthardt

Michael Lautenschlager

Heinke Höck

Hannes Thiemann

Hermann Winter

Jörg Wegner

Frank Toussaint

Peter Lenzen

H. Thiemann / 07.06.2005 / 33

Thank you for your attention