madrigal databaseiri2019.frederick.ac.cy/docs/lectures/madrigal_database.pdfopen madrigal 2 • a...
TRANSCRIPT
![Page 1: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/1.jpg)
Madrigal Databasehttp://openmadrigal.org
Shunrong Zhang
MIT Haystack Observatory
September 6, 2019
Based on slides Bill Rideout prepared
![Page 2: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/2.jpg)
Open Madrigal
2
• A robust, World Wide Web based system that manages
and serves archival and real-time data, in a variety of
formats.
• Geospace data (mostly for ground-based observations),
• Distributed data centers. Each Madrigal site controls
(and updates) its data, and shares metadata with other
Madrigal sites.
• Led by MIT Haystack Observatory since 1980, and
supported by community members.
![Page 3: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/3.jpg)
The Madrigal database stores data from a wide variety of upper
atmosphere research instruments
Incoherent Scatter Radar TEC via GPS MF Radar
Examples of number of instruments in Madrigal:
• Incoherent scatter radars: 22
• MST radars: 3
• MF radars: 16
• Meteor radars: 11
• FPI: 32
• Michelson Interferometers: 6
• Lidars: 9
• Photometers: 7
Other examples:
GPS TEC
DMSP
![Page 4: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/4.jpg)
4
![Page 5: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/5.jpg)
• Continuous data since 2000
• 1x1 (latxlon); every 5 min • GPS (200-)
• GLONASS (May 2019 -)
![Page 6: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/6.jpg)
6
![Page 7: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/7.jpg)
7
![Page 8: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/8.jpg)
Madrigal is a distributed database
8
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DBMadrigal DB
Shared
metadata
![Page 9: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/9.jpg)
CEDAR Madrigal archive imports all data weekly
9
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DB
Madrigal DBMadrigal DB
Shared
metadata
![Page 10: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/10.jpg)
How can the Madrigal database be accessed?
User
Web
interface
Web services API•From anywhere on internet
•Python API
•Matlab API
•IDL API
•Other could be written
Typical use - data discoveryTypical use - data downloading of many
experiments at a time - zero clicks to data
Outputs: ASCII, HDF5, netCDF4
![Page 11: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/11.jpg)
Remote Access to
Madrigal Data
Simple Most complex
I just want to
download lots of
files to my PC
I want to
only download
filtered data with
derived parameters
I want to
write my own
script
globalDownload globalIsprint Work with API’s
![Page 12: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/12.jpg)
Wed-created script to download files
1
2
![Page 13: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/13.jpg)
Write your own script
• Use python, Matlab, IDL
• Methods
getInstrumentsWeb
getExperimentsWeb
getExperimentFilesWeb
getParametersWeb
isprintWeb
madDownloadFile
madCalculatorWeb
• Methods match Madrigal model
![Page 14: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/14.jpg)
Extending/contributing to Madrigal
• Madrigal is completely open source
See www.openmadrigal.org for CVS
• All new code is Python or C.
Imported derivation methods sometimes in Fortran.
• We appreciate all contributions
Suggestions and ideas
Finding bugs
Code
![Page 15: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/15.jpg)
Rules of the road
If you want to use data in Madrigal in a paper or talk, contact the PI
The PI is listed on every web page
1
5
Use of data without informing
PI may lead to seven years of
bad luck in grant writing -
don’t let this happen to you!
![Page 16: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/16.jpg)
1
6
![Page 17: Madrigal Databaseiri2019.frederick.ac.cy/docs/lectures/Madrigal_Database.pdfOpen Madrigal 2 • A robust, World Wide Web based system that manages and serves archival and real-time](https://reader034.vdocuments.site/reader034/viewer/2022042113/5e8fb0fde1cd917ba943173b/html5/thumbnails/17.jpg)
1
7