4D Data Visualisation and Quality Control
Peter Löwe, Jens Klump
GeoForschungsZentrum [email protected], [email protected]
Intention of the talk● Give a better understanding of the relevance of
„preview formats“ for quality control of complex enironmental data.
● Demonstrate how preview formats can be generated with FOSS geoinformatics applications.
● Real world example:– A soil erosion topic, relying on the processing of
„complex data“– which uses FOSS GIS for this task.
•Topic, theory •Hypothesis and approach•Technical frameworkTechnical framework•Modelling•Results / Lessons learnedResults / Lessons learned
Overview: Soil Erosion Project
Soil erosion is the process of soil destruction, a natural process, which can be initiated or amplified by human land management. ... Soil erosion can deminish the agricultural yield significantly.
Soil Erosion: The Problem
In addition to human use of the Earth's surface, climate is a key factor. It provides a means of transportation for soil material to be carried away.
How can these How can these processes be processes be modelled ?modelled ?
ClimateTerrain
Soils
Vegetation
Humans
SOIL EROSIONWind WaterTransportation
Control Enhance
Start
Water, as in ... rainfall
● We need a map of the rainfall distribution (simple task)– Regular updates – Sufficient resolution – Reliable
● The changes of this map can be used to calculate the „potential erosivity“ of the rainfall for a given area:– Total amount: how much water comes down in total ?– Temporal Pattern: Once only, repeated soaking ?– Small droplets, big droplets ?
Idea and approachHypothesis: There are small, temporally fluctuating peaks of There are small, temporally fluctuating peaks of erosiviy due to the convective weather situation. How can these erosiviy due to the convective weather situation. How can these erositiy peaks be charted?erositiy peaks be charted?A sufficiently high temporal (When ?) and spatial data coverage (Where ?), is needed, and also a measure of confidence for the data values (How much ?).
To answer „When - How much - Where“ the radar reflectivity products must be accessed and processed.
ModellingTechnical Framework
Results
Analysis and encoding of the Erosivity.
Geoinformatics,
Information-logistics,
Remote sensing,
Radarmeteorology
•Verification•Validation
•Results
Practical Approach
We use ground-based weather radar for a test site:– 5 Minutes scan rate (200 km radius, 18 km vertical)– Pixel/Voxel resolution 1 km – continous Reflectivity Data (not rain)
400 km
18 km Lower Atmosphere
X
XX
XX
ab
cd
Erosivitymaps
Reflectivity GIS-layers for defined altitudes
Rainfall maps, Pluviogram
XXXX
X X XX
1
42
Erosivity-Model
What type of weather occurs when, where?
Where does erosivity potential occur ?
When does how much rain fall where ?
-
Data flow
3D -> 2Dtransformation
18km altitude
1km altitude
Precipitation-data stream
of „radar rain“
InitState 1
Hibernate
State 2Store
State 3Pause
P: Precipitation
D: Dry
DD
D
PP
P
A „virtual rain gauge“ [state machine] is simulated for each spatial cell of the radar coverage. Erosivity values are derived according to the cell's individual input data stream.
For this reason, agent technology is used, as each „gauge-agent“ must keep its own record of previous precipitation events.
Erosivity-Modelling
Erosivity-data stream Maps
Index-value
Cell-Agent
Tools of the trade● GRASS GIS
– Raster and volume data processing– 2D Animations (flip-books)
● NVIZ (part of GRASS)– 3D visualisation and animation
● Paraview– 3D visualisation and animation
GRASS GIS
● GRASS GIS is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization.
● Oldest and largest FOSS GIS project● GRASS is official project of the Open Source
Geospatial Foundation.● Scriptable● „Backend use“ in QuantumGIS, PyWPS,
JGRASS
Calculating Totals
24h total
erosive
16:18:50 Hours
16:43:30 Hours
16:59:56 Hours
Erosivity
Reflectivity
Σ
Σ
Left: ReflectivityCentre: RainfallRight: Erosivity
Erosivity-Totals display of local erosivity pulses
Elevation: Rainfall totalColor: Erosivity total
Conclusion: The model implementation works !
Animated Time SeriesAnimated displays of reflectivity, derived “radar rainfall” and the
corresponding erosivity peak pattern were created with GRASS GIS.
Reflectivity “Radar-Rain” Erosivity
The erosivity ribbons (right) follow the rainfall fields (center): The model works
The Challenge● 2D animations and 2.5D images show that the
erosivity modelling „works well“– [the erosivitiy peaks follow the rainfall fields]
● However, the model depends on input data:● Can we trust the data ?
– Metadata appears correct.– [are the rainfall fields correct ?]
● Weather Radar provides 3D data. – [3D->2D transformation: Correctly done?]
3D: Straightforward Approach
NVIZ-Animation (GRASS): „Real Clouds“
Volume [Full Information]= Reflectivity Values (30/40/50 dBz) 2.5D Surface: MaxReflecivity(Color), Radar horizon (Elevation)
Flattening
NVIZ● „GRASS in-house“ visualisation tool● Pro: Works directly on the internal database● Pro: Scriptable● Con: small user base, bugs, missing documentation● http://grass.itc.it/nviz/
● Parallel Visualisation Toolkit● Frontend to VTK + QT● Large userbase.● GRASS-related import issues:
– Loosely coupled via file system– Ascii-VTK-Format – currently not effective for use. Improvements
are hoped for later this year● http://www.paraview.org
Garbage in, Garbage out
● Can we trust the rainfall information of the weather radar ?
● Model results are based on rainfall data.● Errors and Biases in the rainfall data will affect
all derived products.● What about transient biases which might
vary in time or space?
Flattening TrustTrust
3D data
2D information
Boredom in, Boredom out● Large data archives exist and more data are
added every day.● How can we easily identify time intervals when
„some interesting weather“ has occurred?● We could watch it all in 4D (3D over time):
– Takes too much time, is incredibly boring– Problem to watch the right things at the right time.
Introducing Preview Formats● A visual preview format provides a condensed
view with the relevant information for the current question („determinant“).
● Humans are visualy oriented: preview formats (shapes/volumes) are easier to comprehend than numbers.
Transient Phenomena Preview● What was the weather like
for a 24 h period ?● Try this: 2D Animation
(Flip-Book)
● Alternative: Stacking of the flip-books pages (just the ink, really) and look at all pages at once.
● Howto: Creation of a volume in GRASS GIS, visualisation by Paraview.
From single drawings
1 2 3
„Radar Rain“
Erosivity
Flip-book Volume
time
1
2D Space: Rainfall field
2
3
Yellow: RainfallRed: Erosivity
Data Errors(ground targets)
Rainfall field
Rainfall field
Erosivity Peaks
Not realclouds !
1
2
3
Ce n'est pas un nuage!
● Detailed top-view of the track of a precipitation field (yellow) and the derived erosivity pulses (red). Note the highly localized distribution of the erosivity pulses. This information can be used to calibrate the interaction between point-sampling rain gauge networks, weather radar calibration and soil erosion plots.
Painting of a pipe
Rainfall tracks of clouds
(+ „erosivity tracks“)
Quality Control 1.0
A precipitation field and its resulting erosivity pulses shown in side-view.
The image does not show real world clouds but precipitation- and erosivity tracks.
„Soaking“ The height of a rainfall track tells us how long it did rain at a certain location
Rods of eternal
soaking: Data errors
Dimensional Collapse
The 2D (xy) rainfall field was „squeezed“ out of the 3D (xyz) weather radar data, implicitly „collapsing“ the vertical dimension.
The stacking of the time frame „flip-book“ pages substituted the altitude (z) dimension by the time dimension.
Collapse 2.0
This approach can be followed further:● In the previous example we collapsed the z-
dimension● Now we collapse the horizontal (xy)
dimension.● The resulting diagram is a preview format
commonly used in meteorology: the „Contoured Altitude by Frequency Diagram“ (CFAD).
Contoured Frequency by Altitude Diagrams (CFAD)
● CFAD can be created from 3D radar reflectivity data (original airspace radar scan). The 3D data set is sliced vertically.
● A histogram of the reflectivities (1D) is generated for each slice/layer.
● Stacking the histograms gives us a 2D synopsis of the current situation in the scanned airspace.
● This tells us a lot about the weather and potential measurement errors.
● In our case, this task is done in GRASS.
CFAD – An Example
● Contoured Frequency by Altitude Diagram (CFAD). Numbers on contour lines give the number of voxels in the observation area with a given radar reflectivity. The CFAD gives a snapshot of weather intensity at different altitudes in the lower atmosphere.
2D Animation
Leafing through the flip-book:Weather CFAD
CFATD: One step beyond● Contoured Frequency Altitude by Time Diagram
adds the time dimension, resulting in a volume body.
● The shape of the CFATD makes it easy to identify:
● periods of high radar reflectivity, i.e. intense weather, and
● Errors in the radar or processing chain.● Done in GRASS, displayed in Paraview
Flip-book volume● Once again we can create a volume from the
flipbook.
Time
Altitude
„Droplet Size“
Iso Surfaces resemble levels of droplet counts (a few, many, lots)
Critical threshold: If the inner layer (many droplets) of the „loaf“ exceeds it, then there is heavy downpour or even hail.
Visual Quality Control● CFATD gives a convenient and reliable quality measure
for observations not to use
● If the CFATD structure appears blocky, or „non-organic“: discard the data
Faulty data
Faulty data
Better data, better models● 4D previews for „Live Quality Control“ in sensor
systems:– Weather Radar does „now-casting“
● It looks into the distance (right now) ● but not into the future
– Real-time generation of CFATD „loaves“ could be used for radar system calibration and maintenance.
What level of quality do we get RIGHT NOW ?
Applications in Grid/eScience● Dimensional collapse and 3D animation of data
can be used as a preview format for very large/complex datasets.
● The computing power needed for the generation of these preview formats can be sourced from the Grid.
Conclusion
● Complex (4D) data are not easy to interpret.● Preview formats enable identification of biases
drifting in time and space in complex data.● Preview formats save time in the selection of
useful data.● You can do it, too:
– Ukrainian Radarsystem (MRL-5 + Linux-based Operating Software (Titan))
– GRASS + Clips + Database + Paraview
Thank you for your attention !