fmi open data interface and data models

Open Meteorological DataUsage and Best Practices

Finnish Meteorological InstituteRoope Tervo, Mikko Visa

13.04.2023 2

Finnish Meteorological Institute opened its data in 2013.

Basically everything FMI owns was opened.

Data is provided freely in machine readable format.

FMI Open Meteorological Data Services / Roope Tervo, Mikko Visa

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data



13.04.2023 3

FMI Open Data Portal follows INSPIRE requirements.

FMI Open Data Portal

Meta data

Data Models

Services

The very same data portal works as Open Data and INSPIRE portal.


13.04.2023 4

Data set Description Time Interval

Estimated publish date

Weather Observations

Temperature, Wind, Humidity, Ground Temperature…

10 min Open,older data to be added

Sun Radiation UV, Short and Long Term Radiation…

1 min Open

Marine Observations

Waves, Sea Temperature, Sea Level…

1 h Open

Weather Radars Precipitation Rate, Precipitation Amount…

5 min Open, older and more data to be added 2015

Lightning Thunder Strikes in Finland

5 min Open

Example of Data Sets


13.04.2023 5


Data set Description Time Interval Estimated publish date

Real Time Observations

Real Time Observations from specific location(s)

AWS 2010 – Soundings 1959 –Flashes 1998 – Sea Level 1971 – Waves 2005 –

Openolder data will be added

Climatological Observations

Daily and monthly temperature mean and extreme values from weather stations

1959 - Open

Climatological Observations

Monthly temperature and precipitation rate mean values interpolated to grid

1961 - Open

Climatological Reference

Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth.

Reference seasons:1971-2000 1981-2010

Open


13.04.2023 6



Weather forecast model HIRLAM RCR

Point forecasts and grid data

Latest model run(4 times a day) 0…54 h

Open

Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data

Latest model run(4 times a day) 0...54 h

Open

Environmental Monitoring Facilities

Weather observation stations, radars…

- 2015

Aviation Observations

METAR/IWXXM 30 min Open

Ground & mast observations

Special observations from ground and masts

2014


13.04.2023 7



Air Quality Observations

Air Quality Observations 1h 2015-2016

Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen

Latest model run (once a day)0…96h

2014-2015

HELMI Ice Model Ice forecast model Latest model run(4 times a day) 0...54 h

2014/12

Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height

2 times a day 2014

Quality information

Quality information for observations


13.04.2023 8

Registrationo Registration is required to use View and Download

Services o Working email address is the only mandatory

informationo After registration the user gets an API key which have to

be added into all requestso GET parameter fmi-apikey=…&

o Header fmi-apikey; …

o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?

o One can create several API keys with one email


13.04.2023 9

Usage Limits

With one API key it’s allowed too do at most 20 000 requests per day to Download Serviceo do at most 10 000 requests per day to View Serviceo do at most 600 requests per 5 minutes to both services

o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily

o So, with one API key it’s allowed load everything onceo View service can be used for testing but can not be used as a

back end for popular clients


13.04.2023 10

Catalog Service (CSW)

o Based on GeoNetworko High-level metadatao Provides UIo Maybe more useful sources:

o http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services (based on WFS DescribeStoredQueries)

o http://www.paikkatietoikkuna.fi/web/fi/metatietohaku (based on GeoNetwork)


http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services



http://www.paikkatietoikkuna.fi/web/fi/metatietohaku



13.04.2023 11

View Service (WMS)o Based on GeoServero Visualize observation and

forecast data and provides radar images

o Only the most common layers are published

o Can be used for prototyping - not for production use


13.04.2023 12

Download Service (WFS 2.0)

o Web Feature Service (WFS) 2.0 Simple Profile

o Based on stored querieso Predefined data sets with

possibility for additional parameters (i.e. time and area)

o In-house production


13.04.2023 13

GetCapabilities:• Describes capabilities of the service• Normally start point for client applications• http://data.fmi.fi/fmi-apikey/…/wfs?request=GetCapabilities

FMI Open Meteorological Data ServicesRoope Tervo, Mikko Visa

WFS Basicsobseravtions and point forecasts

13.04.2023 14

XSLT

DescribeStoredQueries• List and describes available stored queries (pre-defined

data sets)• Stored query name and description

• Possible additional parameters for the data query

• http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-

services

FMI Open Meteorological Data Services Roope Tervo, Mikko Visa





13.04.2023 15

GetFeature• Actual data query• In FMI service, stored query id is compulsory

• &storedquery_id=..

• Other possible parameters depend on the stored query• Possible values can be fetched with DescribeStoredQueries

request.

• Location has to be often defined some how.

http://data.fmi.fi/fmi-apikey/.../wfs?

request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluep

air&bbox=21,62,22,64FMI Open Meteorological Data Services

Roope Tervo, Mikko Visa


13.04.2023 16

Location parameters in stored queries& bbox area

• By default, in same srs with the (requested) srs. Specify as

fifth element if needed (i.e.

&bbox=132249,6433579,631999,6933329,epsg:3067)

& fmisid FMI id for observation station& wmo WMO id for observation station

FMI Open Meteorological Data Services Roope Tervo, Mikko Visa


13.04.2023 17

Location parameters in stored queries& geoid http://geonames.org id for location& place human readable name

• use region to accurate your query (i.e.

&place=kumpula,tampere)

• can be used several times in one query (i.e.

&place=kumpula,helsinki&place=heinola)

& latlon coordinates (used only in marine data sets)



http://geonames.org/

13.04.2023 18

GetPropertyValue• Responses only requested part of the GetFeature response• Use ValueReference=xpath to define part of interest• Same additional parameters for stored query with GetFeature

request are still valid • For example: http://data.fmi.fi/fmi-apikey/…/wfs?

request=GetPropertyValue&storedquery_id=fmi::observations::weather::m

ultipointcoverage&place=kumpula,tampere&ValueReference=wfs:Feature

Collection/wfs:member/omso:GridSeriesObservation/om:result



13.04.2023 19

Data Modelso Observations and point

forecasts as GMLo The same data is published in

two different formats.

o Gridded data is provided in appropriate binary format (grib, NetCDF, HDF…)o WFS members contains the

metadata ‘envelope’ with a link to a actual data


o Every XML document as self-explanatory as possivble

13.04.2023 20

om:phenomenonTime• Tells start and end time of returned data


Data Modelsmost important elements

13.04.2023 21

om:procedure• Describes data producing procedure

om:parameter• Changing parameters in the process

• i.e. analysis time of the numerical model



13.04.2023 22

om:featureOfInterest• Describes location(s) or area of returned data• Several different kind of identifiers

• target:SurfaceWeatherTargetArea area (grid data)

• fmisid FMI id for observation station

• wmo WMO id for observation station

• geoid http://geonames.org id for location

• name Human readable name

• region county of the station



http://geonames.org/

13.04.2023 23FMI Open Meteorological Data Services

Roope Tervo, Mikko Visa

om:observedProperty• Describes meteorological parameters in returned data• Defined as xlink• Describes label, basePhenomenon and units of measure of

every parameter in the reponse• Tip: use the same service without parameter definitions to

query all possible parameters• http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast

• http://data.fmi.fi/fmi-apikey/.../meta?

observableProperty=observation

• Note that all parameters are not available in all data sets


13.04.2023 24

om:result• Data in required format in observations and point forecasts and link to a

external binary data in grid formatted data

FMI Open Meteorological Data Service Roope Tervo, Mikko Visa


13.04.2023 25

Data Modelsgmlcov:MultiPointCoverage

gml:rangeSetgml:doubleOrNilReasonTupleList

The data is listed for every point defined in domain set.

gml:domainSetgmlcov:simpleMultiPoint

The coverage is defined as a list of

points in 4 dimensional grid (lat,

lon, height, time).

gmlcov:rangeType

The parameters listed in range set

are defined in separate element.


13.04.2023 26

Cons- Not intuitive- No natural

structure of XML XSLT and Xpath don’t work

Pros+ Compact+ Efficient+ Small file size+ Works for many

data types

Data Modelsgmlcov:MultiPointCoverage


13.04.2023 27

Data Modelswml2:MeasurementTimeseries

wml2:MeasurementTimeseries

One member contains time series of one parameter and

one location


13.04.2023 28

Cons- Lots of repetition- Large file size- Heavy for DOM-

based parsers- Don’t work i.e. for

thunder strikes

Pros+ Intuitive+ Easy to use+ XSLT & XPath

works

Data Modelswml2:MeasurementTimeseries


13.04.2023 29

Data ModelsSimpleFeature

SimpleFeature

One member contains one time step of one parameter

and one location

Coming

soon


13.04.2023 30

Cons- Huge file size- Heavy for DOM-

based parsers

Pros+ Intuitive+ Easy to use+ Wide application

support

Data ModelsSimpleFeature


13.04.2023 31

Data Models file size comparison

Simple

Featu

re

Mea

sure

men

tTim

eser

ies

Mult

iPoin

tCov

erag

e0

20

40

60

80

10081.7

52.9

1.81.3 1.2 0.2

Document Size [MB]

Compressed DocumentSize [MB]

o13

8 ob

serv

atio

n st

atio

ns

o11

par

amet

ers

o72

tim

e st

eps

(12

hour

s)


13.04.2023 32

• Aviation weather reposts are delivered as IWXXM• New data model coming into use in aviation

• Consists of the same elements than other messages• om:phenomenonTime, om:procedure, om:featureOfInterest,

om:result

• Content of the METAR is in om:result part as

• extracted into XML elements

• original, “old fashion”, METAR string


Data Modelsaviation observations IWXXM / METARS

http://en.wikipedia.org/wiki/IWXXM

13.04.2023 33

• Once published, content and structure of IWXXM

messages will not change• Messages will contain a digital signature

• Use GetPropertyValue to download only original

METAR string• http://data.fmi.fi/fmi-apikey/…/wfs?

request=GetPropertyValue&storedquery_id=fmi::avi::observ

ations::finland::latest::iwxxm&valuereference=wfs:FeatureC

ollection/wfs:member/avi:VerifiableMessage/avi:metadata/

avi:MessageMetadata/avi:source/avi:Process/avi:input



13.04.2023 34

• Messages have been automatically created based on

the original METAR/SPECI code messages.

• Messages are NOT an authoritative aviation

weather report. They SHOULD NOT be used as a

weather report to be used in flight planning or

other aviation related use.



13.04.2023 35

• WFS response contains the same meta data information

than in observations and point forecasts• om:result contains gmlcov:RectifiedGridCoverage

• Basically the same with multipointcoverage

• domainSet defines the grid (now as regular grid)

• rangeSet contains data as an external link to the binary content

• rangeType defines the parameters


Data Modelsbinary data

13.04.2023 36

• Note that fileReference points often to an other service• Can be used directly, but

• It is always good practice to consult WFS for available

times and parameters



Client

WFS

WMSFMI Data

Server

Available radar images?Available weather model outputs?

13.04.2023 37

• Area, time and weather parameters may be defined in

the request• For weather models, intersection of requested area and

available data area is returned

• For radar images, all images which intersects requested

spatial and temporal space are returned



Requested area

Radar images

Radar images Requested area

Model coverage

Weather models

13.04.2023 38

Radar images• The reference points to a “original” gray scale GeoTiff image• Images can also be downloaded as color images

• Remove &styles=raster from the data request

• But then information is lost

• Used SLD-files can be downloaded from

https://github.com/fmidev/opendata-resources/tree/master/sld

• Consult om:parameter element for single radar measurement

parameters• Scanning angle, bin count and bin length





13.04.2023 39

Numerical models• The reference points to (often a subset) the model output in

appropriate format• GRIB2 for weather models

• NetCDF for marine models

• Note that whole model output can be large• Up to 17 GB

• Download always only area, parameters and time range of

interest



13.04.2023 40

For binary data• PanoPly

• http://www.giss.nasa.gov/tools/panoply/

• Simple

• Handle only very basic data

• IDV• http://www.unidata.ucar.edu/software/idv/

• Handle most of the data


Helpful Software

http://www.giss.nasa.gov/tools/panoply/



http://www.unidata.ucar.edu/software/idv/

http://www.unidata.ucar.edu/software/idv/

13.04.2023 41

For exploring the data• Google Earth & QGIS• Start from GetCapabilities, UI guides

thereafterhttp://wms.fmi.fi/fmi-apikey/.../geoserver/wms

?

service=WMS&version=1.3.0&request=GetCapabilities


Helpful Software

13.04.2023 42

Please contribute!

https://github.com/fm

idev/

MetoLib

o Open source JavaScript library produced by Finnish Meteorological Institute

o Helps users to load and use the data

o Supports multi point coverage data formato Python version is on the list

Easy requests Cache

Parse the data to as JSON


www.fmi.fi

fmi open data interface and data models

Engineering

open data andinspire

example of data setsdata

open meteorological

mikko visa6

new data sets

mikko visa2fmi open

o header fmiapikey o

time step