20161028 strahlendorff fmi experience in openness

Openness in researchExperiences at Finnish Meteorological Institute

28.10.2016

Finnish Meteorological InstituteMikko Strahlendorff

Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

01.05.2023 2

Background for my experienceMostly employed by Finnish Meteorological Institute Weatherforecasting 1995-1997 Public service development 1998--99 Commercial service development and head of it

2000-2003 Head of Informations system services 2004-

2007 Seconded Expert to European Commission

Earth Observation program 2008-2010 Ministry of Transport and Communications

Ministerial adviser 2011-2012 EU research policy and international

organisation 2012- Helping to build European Research

Infrastructure Consortias ICOS, EuroARGO, SIOS, COPAL

Executive commiittee member in Group on Earth Observation

Openness experience at FMI – Mikko Strahlendorff

Grown with the internet

01.05.2023 3

FOSS revolution and the internet• Researcher Linus Torvalds at Helsinki

university releases Linux as Free and Open-Source Software

• 25 years anniversary now and• 96% of all web-servers run Linux• 69% of all mobile devices run Android

(developed from Linux)

• Sharing ones work with a huge network of like-minded ensures high quality results faster than anything

• Motivation comes from the common utility of the result

Openness experience at FMI – Mikko Strahlendorff

• Openness requires extra efforts• Free and open doesn’t generate revenue

• How to motivate financing? How to share expenses fairly?

• Sharing efforts is making the list of contributors long and single persons are less visible

• How to divide the fame?Solutions:• Application of the common to many challenges in

many places• Analysis, understanding and education for sale• Fame to the whole system:

• Data production is publishing!• Each use of data should be rewarded!

• FOSS tools and internet technologies reduce costs

Challenges with Openness – rewarding effort

01.05.2023 4Openness experience at FMI – Mikko Strahlendorff

Data is a great common motivationo Analysis needs data and it gets better with

more datao Selling scientific data is rarely lucrative

and data lying in databases unused is only expenses - no gain

o Data is good to be collected in similar ways for similar applications – common interest naturally networks collectors

o Quality control and technical solutions are perfect tasks for sharing

o Data dissemination has costs that are usually not planned in research

o Tracking usage is good for development and for finance motivation

01.05.2023 5

01.05.2023 6

Finnish Meteorological Institute opened its data in 2013 after government decision for all the Finnish public sector.

Basically everything that FMI has property rights on was opened.

Data is provided freely in machine readable format over the internet.

Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data



Data set Description Time Interval

Estimated publish date

Weather Observations

Temperature, Wind, Humidity, Ground Temperature…

10 min Open,older data to be added

Sun Radiation UV, Short and Long Term Radiation…

1 min Open

Marine Observations

Waves, Sea Temperature, Sea Level…

1 h Open

Weather Radars Precipitation Rate, Precipitation Amount…

5 min Open, older data to be added

Lightning Thunder Strikes in Finland

5 min Open

Example Data Sets

01.05.2023 7Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari

Example of Data SetsData set Description Time Interval Estimated

publish dateReal Time Observations

Real Time Observations from specific location(s)

AWS 2010 – Soundings 1959 –Flashes 1998 – Sea Level 1971 – Waves 2005 –

Openolder data will be added

Climatological Observations

Dayly and monthly temperature mean and extreme values from weather stations

1959 - Open

Climatological Observations

Monthly temperature and precipitation rate mean values interpolated to grid

1961 - Open

Climatological Reference

Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth.

Reference seasons:1971-2000 1981-2010

Open


Example Data SetsData set Description Time Interval Estimated

publish dateWeather forecast model HIRLAM RCR

Point forecasts and grid data

Latest model run(4 times a day) 0…54 h

Open

Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data

Latest model run(4 times a day) 0...54 h

Open

Environmental Monitoring Facilities

Weather observation stations, radars…

2015

Aviation Observations

METAR 30 min open

Ground & mast observations

Special observations from ground and masts

2016 /Open


Example Data SetsData set Description Time Interval Estimated

publish dateAir Quality Observations

Air Quality Observations 1h open

Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen

Latest model run (once a day)0…96h

2015

HELMI Ice Model Ice forecast model Latest model run(4 times a day) 0...54 h

open

Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height

2 times a day open

Road Weather Observations (LIVI)

Road Weather Observations

10 min open


FMI Open Data Portal follows INSPIRE requirements.

FMI Open Data Portal

Meta data

Data Models

Services

The very same data portal works as Open Data and INSPIRE portal.


Registrationo Registration is required to use View and Download

Services o Working email address is the only mandatory

informationo After registration the user gets an API key which have to

be added into all requestso GET parameter fmi-apikey=…&o Header fmi-apikey; …o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?

o One can create several API keys with one email


Usage LimitsWith one API key it’s allowed to

o do at most 20 000 requests per day to Download Serviceo do at most 10 000 requests per day to View Serviceo do at most 600 requests per 5 minutes to both services

o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily

o So, with one API key it’s allowed to load everything onceo View service can be used for testing but can not be used as a

back end for popular clients


And a little over450 000 data downloads

per day (5.2 req/s)

At the moment about 11 700

registered users

3 year experience


Industry is happy to use standardized

services

Amateur and freelancer coders

would prefer simple JSON API

Some Experiences


Please contribute!

https://github.com/fmidev/

MetoLibo Open source JavaScript library produced by Finnish

Meteorological Instituteo Helps users to load and use the data

o Supports multi point coverage data formato Python version is on the list

Easy requests Cache

Parse the data to as JSON


01.05.2023 Open Meteorological Data with OGC and INSPIRE / Roope Tervo

17

SmartMet Server•High capacity & availability Data server for MetOcean data•Data is extracted and products generated always on-demand• INSPIRE Compliant•Operational use since 2008

• FMI client services (since 2008)• Finnish Meteorological Institute (FMI) Open Data Portal (since 2013)

•Several input sources• GRIB-, NetCDF-, etc. files via internal data format querydata

• PostGIS database (vectors)

• SQL database (point observations)

•Several output interfaces and formats• WMS, WFS 2.0

• JSON, XML, ASCII, HTML, SERIAL

• GRIB1, GRIB2, NetCDFC++

01.05.2023 Open Meteorological Data with OGC and INSPIRE / Roope Tervo

18

SmartMet Server goes Open Source•Going to be published as Open Source in Q4/2016•MIT license•https://github.com/fmidev

Fast scienceOpen Science visionScience WIKIPEDIA

o Data is collected in standard ways and published immediately

o Papers are written collaborativelyo Science is mostly about writing

scripts/models to analyze data for answers to questions

o Analysis scripts are published with the answers so readers can reproduce results and evaluate conclusions themselves

o Donors get to ask questions and use the knowledge

o Sharing the fame is visible thru linking all the scientists, from every data set and every model/script


• Governments: What data do we need for public interests?• How do private entities support data production?

• Science interest

• Commercial interest

• Research funding should be two-fold:• longer time plans and grants for data production and modeling

• short term grants for answers to societal questions for scripts and analysis

• How does a new Science approach conquer the world like Linux?

• Thanks and please comment!

What do we need to do for this vision?


www.fmi.fi

http://www.slideshare.net/MikkoStrahlendorffhttps://en.ilmatieteenlaitos.fi/open-data

http://www.slideshare.net/MikkoStrahlendorff



