european strategy for long term eo data preservation and...
TRANSCRIPT
European Strategy for Long Term
EO Data Preservation and Access
FRINGE 2007 Workshop26th November 2007 – ESRIN
M.Albani, V.Beruti, E.Forcada
(EO Ground Segment Department, ESA/ESRIN)
2
Outline
ü Goals
ü Need and benefits
ü Current archives situation
ü Implementation strategy
ü Conclusions
3
1. Preserve on a long term basis (unlimited time-span)the EO data of all past, current and future ESA andThird Parties ESA managed missions.
2. Ensure and facilitate their accessibility andexploitation.
3. Preserve these data, and ensure the preservation ofall European EO data, as part of a European(including Canada) strategy based on an harmonizedsolution (European Long Term Data Preservationsystem, LTDP) elaborated and established:
Ø in cooperation with ESA Member States Agencies andEUMETSAT;
Ø in consultation with the European Commission and withNational and commercial mission operators.
Goals (1)
4
Goals (2)
4. Propose a mission independent long term fundingscheme in order to support efficiently the cost ofimplementation of such a strategy.
5. Possibly ensure the coherency with the archiving ofother non-space based environmental data andinternational policies.
Need and benefits
6
Source of requirements I
ü The requirements for accessing historical archivesstrongly increased in the last ten years and the trendis likely to increase in the future mainly for long termscience and environmental monitoring.
ü As the time-span of EO data archives extends from afew years to decades, their value as a scientific time-series is highly increased especially for global change.
ü In the next decade, the wealth of information currentlylocked inside the global data archives must be fullyexploited and re-analysed on a global scale.
7
Source of requirements II
ü “The goals of the science strategy will be achievedonly if the data gathered are made easily accessible toall user communities concerned - both during themission lifetime and afterwards (archives).”
[Ref.: The Changing Earth - Science challenges for ESA’s Living Planet Programme]
ü “Despite laudable efforts, the current situation withrespect to the availability of Earth observations isnot optimal. […] Moreover, there is an erodingobservational infrastructure, inadequate long-termdata archiving, and no assured continuity for manyessential observing systems.”
[Ref.: GEOSS 10 Year Implementation Plan]
8
Benefits - Main application areas
ü EC Policies with long-term perspective.
ü European and Global Environment (forest monitoring,soil monitoring, urban development, air qualitymonitoring, ecosystems monitoring and managementfor protection of resources).
ü Development and Humanitarian Aid.
ü Food Security.
ü Civil Protection and Disaster Monitoring (floodprediction and mitigation, landslides, subsidence,volcanoes monitoring).
ü Global Climate Change (monitoring the atmosphere,systematic climate observations, drought monitoring).
ü Global Security and Sustainable Development.
Current archives situation
10
ESA - Archives logistic and data volume
0
1 5 0
3 0 0
4 5 0
6 0 0
7 5 0
9 0 0
1 0 5 0
1 2 0 0
1 3 5 0
1 5 0 0
1 6 5 0
1 8 0 0
1 9 5 0
2 1 0 0
2 2 5 0
2 4 0 0
2 5 5 0
2 7 0 0
2 8 5 0
3 0 0 0
Tota
l A
rchiv
e
in
Terr
aB
ytes
(T
B)
1 9 8 6 1 9 8 8 1 9 9 0 1 9 9 21 9 9 4 1 9 9 6 1 9 9 8 2 0 0 0 2 0 0 2 2 0 0 4 2 0 0 6
Y e a r
Evolution of ESA's EO Historical Data Archives between 1986-2006
LANDSAT 2-4 MSS (75-Dec 93)
AQUA Modis (April 03-today)
ENVISAT LR (March 02-today)
ENVISAT HR (March 02-today)
TERRA Modis (June 01-today)
QUICK SCATT (01-today) /PROBA (May 02-today)
LANDSAT 7 ETM (April 99-Dec 03)
SEA STAR SeaWifs (Apr 98-today)
ERS 2 HR (May 95-today)
ERS 2 LBR (May 95-today)
JERS SAR/OPS VNIR (92-Sep 98)
ERS 1 HR (Jul 91-Mar 00)
ERS 1 LBR (Jul 91-Mar 00)
SPOT 1-4 HRV (87-today)
MOS 1, 1b MESSR (87-Oct 93)
NOAA 9-17 AVHRR (86-today)
LANDSAT 5 TM (April 84-today)
NIMBUS 7 (Nov 78-May 86), SEASAT (Jun-Oct 78)
The evolution follows a similar trend for all nationalarchives.
Distributed archive system reusing national public archivesinfrastructure and existing thematic expertise.
11
01 5 03 0 04 5 06 0 07 5 09 0 0
1 0 5 0
1 2 0 01 3 5 01 5 0 01 6 5 01 8 0 01 9 5 02 1 0 02 2 5 0
2 4 0 02 5 5 02 7 0 02 8 5 03 0 0 0
Tota
l A
rchiv
e
in
Te
rraB
yte
s
(T
B)
1 9 9 2 1 9 9 4 1 9 9 6 1 9 9 8 2 0 0 0 2 0 0 2 2 0 0 4 2 0 0 6
Y e a r
Evolution of ESA's EO Historical Data Archives between 1992-2006
ENVISAT LR (March 02-today)
ENVISAT HR (March 02-today)
ERS 2 HR (May 95-today)
ERS 2 LBR (May 95-today)
ERS 1 HR (Jul 91-Mar 00)
ERS 1 LBR (Jul 91-Mar 00)
ESA – ERS/ENVISAT data
12
ESA - Archives management
ü Periodic migration processes to convert data (differentmedia & formats) carried out to exploit the mostadvanced available technology (today automated tapelibraries and disk farms).
ü Data compacting and conversion to the ESA standardformat packaging (SAFE).
ü The Enhanced On Line data access to the long-termarchive will allow online access to all archived data.
ü Ad-hoc creation of new types of data or of information.
ü All long term data preservation activities executed at aminimum profile (part of operational maintenance).
13
National archives
ü National and EUMETSAT archives mainly centralizedin one facility.
ü Different technologies and data formats used butrather similar policies and procedures.
ü No common policy for long term data preservation.
ü Most missions do not have a committed long termdata preservation strategy.
ü Commercial missions could be interested intransferring their long term archives into publicarchives once the commercial perspective of the datais reduced.
14
Overall status
ü Under actual ESA scheme the proactive and quickadaptation of the archive management to the changingdemand for access and exploitation is not adequate enough.
ü Urgent need for a coordinated and coherent approach for aharmonized European archives management policy toguarantee the complete European EO data set preservation.
ü Missing Long Term programmes and funding to support thisstrategy.
ü ESA, within its mandate for space coordination, (reinforcedin the “Resolution on the European Space Policy” adoptedat the 4th Space Council in 2007), can coordinate and sharea common strategy among all the stakeholders.
ü ESA is currently consolidating and formalising this strategyproposal for approval by the appropriate delegate bodies.
Implementation strategy
16
Approach
ü Prepare and implement the ESA and Third PartyManaged missions Long Term Data Preservation(LTDP).
ü Prepare the European LTDP by designing atechnical, programmatic and financial cooperationscheme with other missions.
ü Start the implementation of the European (includingCanada) LTDP system for the benefit of Europeancountries.
17
ESA LTDP Elements (1)
1. Archive maintenance and archived data integrity
Ø Periodical full data transcription and eventually reprocessing.
Ø Certification of archive locations and services.
Ø Coordinated technology evolution.
Ø Two copies of data in different geographical locations.
2. Evolution of archives logistic infrastructure
Ø Enhancement of the current distributed ESA Facilitiesnetwork and the related infrastructure.
Ø Preparation for an increasing number of distributed archives(thematic applications), harmonising interfaces, technologyand standards.
Ø Agreements eventually with non European archives.
Ø Alignment of processing and distribution services for newand old missions data.
18
ESA LTDP Elements (2)
3. Consolidation of Data Security
Ø Consolidation of local risk mitigation infrastructure andensure dual copy availability of all archived data.
Ø Implementation of a secure data access (critical for GMES).
4. Evolution of Data access and interoperability
Ø Enhancement of the historical data use through offeringspecial easy data access conditions (enhanced Cat1).
Ø On-line data access through improved networkinginfrastructure at all facilities.
Ø Standard interfaces to allow easy and cost-effective accessand dissemination of heterogeneous EO mission data.
19
ESA LTDP Elements (3)
5. Archived data exploitation
Ø Reduction of the global data handling time and cost providingdata pre-processing, hosting & execution of user algorithms,etc...
Ø Enhancement of information extraction from low-level EOproducts.
6. Standardization
Ø Standardization of all critical interfaces.
Ø Realignment of products characteristics and format of oldmissions to most modern technologies.
Ø Harmonization of EO products and format of differentmissions, EO products quality certification, across-missionCAL/VAL standards.
20
European LTDP
ü Set-up of a collaborative project to increase the
synergies between National, private and ESA data
preservation leading to an integrated approach while
respecting the national initiatives and interests.
Ø Analysis (feasibility, programmatic aspects, data access and
preservation policies).
Ø Definition (LTDP system requirements, interfaces,
agreements, common data access and preservation policy,
technical aspects).
Ø Implementation Plan.
ü Implementation of the Long Term Data Preservation
system at European level.
Conclusions
22
ü Access to long time series of EO data for long termscience and environmental monitoring is needed.
ü The mandate for long term EO data preservation isscattered and an overall strategy is missing.
ü Most mission operators identify Long-term EO dataarchiving as an important issue where cooperation andsharing can bring a benefit.
ü There is an urgent need for a coordinated and coherentapproach for a harmonized European archivesmanagement policy.
ü ESA, within its mandate for space coordination, cancoordinate and share such a common strategyamong all the stakeholders.
ü The strategy here described is a first step towardsthis final goal.
Conclusions
