archival of raw and analysed radar data at eiscat and worldwide

Archival of raw and analysed radar dataat EISCAT and worldwide

Carl-Fredrik Enell, EISCAT Scientific Association

COOPEUS workshop and EGI-CC kickoff, 11 March 2015

C-F Enell, EISCAT Radar data archival Coopeus/EGI workshop 1 / 23

1 Data levels in incoherent scatter radars

2 Archival of raw data at EISCAT

3 Archival of and access to analysed data: Madrigal and data portals

4 Discussion


Outline




4 Discussion


Data from antenna to plots

Signal processing chain

EISCAT as example

Most (one-dimensional, monostatic) radars are similar


Data from antenna to plots

Channel boardsFiltering

ADC

Level 1processing

RF

Cratecomputer

Siteserver

DecodingIntegration

AnalysiscomputerParameter

fitting

Level 2 (+ 1) data

Level 2processing

Lag profiles(integrated

filtered samples)

Level 3 data

Level 3processing

Physicalparameters

Analog receiver

chain

Level 0 data

VME bus

Network

Auxiliarysampler

Level 0 data

User-defined level 0 processing User-defined level 1 storage

Level 1 data

ComputerPowermeter


Analog signal

Antenna

Receiving element

Preamplifiers

Analog receiver chain

Polarization combiner or separate chains

Local oscillators and mixers

Analogue filters

The signal is not yet data at this level — so the analogue signal can bethought of as ”level -1”


Digital signal

A/D converted voltage domain signal

Level 0 Stream of raw samples

Level 1 Signal after digital filtering and decimation. First accessiblelevel of data.

Correlated/decoded ACF domain signal

Level 2 Signal after forming lag profiles and decoding. Reduces datarates significantly but retains spectral information.


Digital signal

Level 1 vs level 2, two philosophies:

1 Analysis in waveform domain can be easier with present computercapacity. Needed for interferometry etc.

2 Correlated data have essentially the same information content whilereducing data volumes. Stored at EISCAT.


Analysed data

Level 3 Parameters estimatedby fitting a theoreticalfunction to calibrateddata (at level 1 or 2)

Level 4 Visualisations andpublished articles


Outline




4 Discussion


Data formats

Level 1 Stored only for certain needs e.g. meteors, space debris(coherent echoes), together with level 2

Level 2 Matlab v4 compatible files with metadata and data blocks(EISCAT specific)

Level 3 EISCAT specific Matlab and Madrigal NCAR format

Level 4 Published articles etc: EISCAT annual reports. Investigatebest practice for data citation standards and persistentobject identifiers


Storage at EISCAT

Level 2 data archive

Level 2 is the lowest data level routinelyarchived at EISCAT, thus representing raw data

Some users store level 1 data

File storage, 2 copies on RAID units

Data 1981–present: some 40 TB


Storage at EISCAT

Level 2 data search and retrieval

Matlab file archive on RAID

Index in MySQL database (file catalogue)

Apache-MySQL-Python interface to dataI in-house development at EISCATI part of EISCAT schedule pages


Storage at EISCAT

Level 2 data retrieval functions

Download (http) — data transfer from RAID server to web server byTCP socket (Python)

Online analysis — CGI web page writes a text file, processed by cronjob on analysis computer

I Runs MatlabI Emails results to user


SQL databases

Master database server: Data on RAID

Catalogue of “raw” (level 2) incoherent scatter data files

Dynasonde web portal

Analysed dynasonde data

Archived historical ionosonde data

Slave database server

radar data catalogue

dynasonde portal

dynasonde data

EISCAT 3D web site


Level 2 data retrieval and online analysis

Demo

http://www.eiscat.se/schedule/schedule.cgi


http://www.eiscat.se/schedule/schedule.cgi

Outline




4 Discussion


Physical parameters retrieved

All incoherent scatter radars retrieve fundamental ionospheric parameters

electron density

ion temperature

electron temperature

ion drift velocity

etc


Harmonisation of level 3 data access: Madrigal

Madrigal

URL http://www.openmadrigal.org

Developer Bill Rideout, Millstone Hill (MIT) with assistance fromEISCAT and several other sites

Use The de facto standard for distributing incoherent scatterparameters

Also used by several other instruments


http://www.openmadrigal.org

Harmonisation of level 3 data access: Madrigal

Design principles of Madrigal

NCAR format file backend holds 1-dimensional and 2-dimensionaltime series data

Per-experiment directories (e.g. one per day)

File index and other metadata: plain text files


Madrigal data access

Methods of access

Web GUI

Web services with APIs for Matlab, Python . . .

Download: ASCII text, NCAR file formats, HDF5


Madrigal data access

EISCAT results 1981– around 110 GB

http://www.eiscat.se/madrigal


http://www.eiscat.se/madrigal

Realtime Madrigal experiments

Madrigal experiments can be inserted in real time — no conceptualdifference from normal experiments, but useful to distinguish preliminaryfrom calibrated data

Realtime plot of electron density profiles from selected sites

Using Madrigal Python API for retrieval and plotting

URL http://www.eiscat.se/realtime


http://www.eiscat.se/realtime

Data portals

Data portals are search engines for data. They hold metadata only; dataare downloaded from the individual providers.

EISCAT participation

ENVRI see talk by Ingemar

ESPAS http://www.espas-fp7.eu

ESPAS and ENVRI metadata are XML files generated from Madrigal(routines implemented with the Python API).


http://www.espas-fp7.eu

Outline




4 Discussion


Pros and cons of Madrigal

+Well established

Easy-to-use web GUI

Has programming APIs for many common languages e.g. Python

–Present storage file format only 2 D

Only analysed data (no interface to levels 1 or 2)

Scaling to much larger data volumes may be difficult

No persistent data identifiers (numbering of experiments changeseach time a new experiment is added!)


Pros and cons of data portals and virtual observatories

+Online search expected by contemporary users

Provide means to search for coincident data from severalheterogeneous data sets in a consistent way

Consistent metadata (data models / ontologies)

–Multitude of virtual observatories, data portals and GUIs confusing todata users

Usually store only metadata, not addressing needs for archival andbackup

Only suitable for analysed parameters, not raw data

How secure operation after end of projects?


Harmonization of data and access at different levels

Level 1+2

Highly system-specific, but should aim for open formats (that is, notproprietary formats such as Matlab).

See http://www.openradar.org

Open questions: data access, data embargos

Level 3+4

General (same parameters everywhere)

To most users, level 3 is the incoherent scatter radar data

Existing system (Madrigal) serves existing radars well

New scalable systems needed to archive and access 3-D data

Must address data citation: data provenance, persistent objectidentifiers


http://www.openradar.org

Thanks for your attention! Questions?

Data volumes: temporary PBs, archives 100s of TB up to EBs

Data transfer protocols

Storage file formats

Home-brewn vs standards

Data provenance, Digital object identifiers

Big data in other research communities (CERN, astronomy. . . )

?C-F Enell, EISCAT Radar data archival Coopeus/EGI workshop 23 / 23

archival of raw and analysed radar data at eiscat and worldwide

Documents