ensemble visualization and verication in python€¦ · 7/2/2018 workshop_slides slides 1/38...
TRANSCRIPT
![Page 1: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/1.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 1/38
Ensemble Visualization and Veri�cation in PythonEnsemble Visualization and Veri�cation in Python
Tyler Wixtrom Texas Tech University [email protected] Unidata Users Workshop 25-28 June 2018 Boulder, CO
![Page 2: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/2.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 2/38
OverviewOverview
1. PlotsSpaghetti Plots and Postage StampsPaintball PlotsProbability Plots
2. Veri�cationRMSE
3. Examples
![Page 3: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/3.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 3/38
Examples and Data Repository
https://github.com/tjwixtrom/workshop2018
![Page 4: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/4.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 4/38
Why Ensembles and Fancy Plots? Ensembles are more than a collection of deterministic forecasts
Probabilistic forecastsMost likely and range of solutions
Information regarding forecast uncertaintyVariability in time, space, and intensity
![Page 5: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/5.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 5/38
Spaghetti Plots and Postage Stamps
![Page 6: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/6.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 6/38
Spaghetti plots and postage stamp plots are both concise, simple methods of viewing allmembers in a ensemble on the same plot.
Spaghetti plots are generally used for contoured �elds (e.g. 500 hPa geopotential height)while postage stamps are more commonly reserved for shaded �elds (e.g. simulatedre�ectivity).
![Page 7: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/7.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 7/38
Example 500 hPa Spaghetti Plot
![Page 8: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/8.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 8/38
![Page 9: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/9.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 9/38
Advantages
Displays information from each individual memberFull location and magnitude of each �eld is plottedSimple to interpret
![Page 10: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/10.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 10/38
Disadvantages
Can be very busyBecomes increasingly hard to interpret as spread increasesEasy to miss small differences among members (esp. postage stamps)
![Page 11: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/11.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 11/38
Paintball Plots
![Page 12: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/12.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 12/38
Paintball plots are plots where individual points greater than a speci�ed threshold (e.g.simulated re�ectivity 25 dBZ) are plotted, color-coded by ensemble member.
![Page 13: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/13.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 13/38
![Page 14: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/14.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 14/38
Advantages
Easy to quickly see spatial variations among individual member solutionsPlot remains relatively uncluttered, even for ensembles with many members
![Page 15: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/15.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 15/38
Disadvantages
Very little information regarding variations in intensityThresholds are arbitrary and may not be best for all cases
![Page 16: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/16.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 16/38
Probability Plots
![Page 17: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/17.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 17/38
If an ensemble contains a suf�cient number of members and suf�cient spread, probabilitiesof event occurence can be calculated based on the individual members. This is done at asingle point with the following method (Schwartz and Sobash, 2017):
![Page 18: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/18.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 18/38
Ensemble ProbabilityEnsemble Probability
1. De�ne the Binary Probability (BR) for each member at each point for �eld andthreshold
\begin{equation} BP(q)_{ij} = \left{
\right. \end{equation}
\begin{array}{ll} 1 & \quad f_{ij} \geq q \\
0 & \quad f_{ij} < q \end{array}
![Page 19: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/19.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 19/38
1. Calculate the average of ensemble member binary probabilities at each point tode�ne the Ensemble Probability (EP)
![Page 20: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/20.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 20/38
Example Ensemble Probability Plot
![Page 21: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/21.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 21/38
Neighborhood Ensemble Probability
![Page 22: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/22.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 22/38
Since the ensemble probability is valid for only a single point and does not account for smallspatial differences in member solutions, the Neighborhood Ensemble Probability (NEP) canbe de�ned as the probability of event occurence within a speci�ed radius of any point(Schwartz and Sobash, 2017).
![Page 23: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/23.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 23/38
To calculate the NEP for a given �eld and threshold :
Begin by calculating the EP at each pointFind all points within speci�ed radius and calculate mean of EP
\begin{equation}
\end{equation}
NEP(q)_{i} = \frac{1}{N_b}\sum^{N_b}_{j=1}EP_{ij}
![Page 24: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/24.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 24/38
![Page 25: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/25.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 25/38
EP and NEP Di�erences
NEP will have smoothed gradients when compared to EPEP is valid at each point while NEP is valid within a radius of each pointNEP better for forecasts with high spatial variability
e.g. mesoscale convection
![Page 26: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/26.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 26/38
Ensemble Probability Neighborhood Ensemble Probability
![Page 27: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/27.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 27/38
Advantages
Probabilistic forecasts of speci�c eventsQuickly shows ensemble con�denceRelatively simple to interpret
![Page 28: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/28.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 28/38
Disadvantages
Probabilities are often too high due to low model spreadAssumes that all member solutions are equally likelyLimited information regarding member differences in intensity and location
![Page 29: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/29.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 29/38
Plots Summary
![Page 30: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/30.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 30/38
1. Spaghetti Plots and Postage StampsBest for viewing individual members, ensemble spread, range ofsolutions
2. Postage Stamp PlotsBest for quickly viewing spatial differences among member solutions
3. Probability PlotsBest for probability of event occurence and ensemble con�dence
![Page 31: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/31.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 31/38
Veri�cation
![Page 32: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/32.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 32/38
There are many veri�cation metrics available which can be placed into three categories:
Grid-point (Wolff et al., 2014)Object-based (Davis et al., 2006)Neighborhood approaches (Schwartz and Sobash, 2017)
One very simple approach that is often used in the Root Mean Square Error (RMSE).
![Page 33: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/33.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 33/38
Root Mean Square Error
![Page 34: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/34.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 34/38
The Root Mean Square Error is de�ned as:
Measure of mean magnitude of forecast errorSame units as forecast valueSimple to compute and interpretUseful for comparing two models, multiple members, etc.Holmes (2000)
![Page 35: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/35.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 35/38
Examples
![Page 36: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/36.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 36/38
1. Open terminal2. Navigate to workshop2018 folder
3. source activate workshop for mac users, activate workshop for
windows users4. Type jupyter notebook
![Page 37: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/37.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 37/38
ReferencesReferences
Davis, C., B. Brown, and R. Bullock, 2006: Object-Based Veri�cation of PrecipitationForecasts. Part I: Methodology and Application to Mesoscale Rain Areas. *Mon.Weather Rev.*, **134 (7)**, 1772-1784.
Holmes, S., 2000: RMS Error. URLstatweb.stanford.edu/~susan/courses/s60/split/node60.html.
![Page 38: Ensemble Visualization and Verication in Python€¦ · 7/2/2018 workshop_slides slides 1/38 Ensemble Visualization and Verication in Python](https://reader034.vdocuments.site/reader034/viewer/2022042320/5f0a3c7e7e708231d42aabe7/html5/thumbnails/38.jpg)
7/2/2018 workshop_slides slides
http://127.0.0.1:8000/workshop_slides.slides.html?print-pdf#/ 38/38
Schwartz, C. S., and R. A. Sobash, 2017: Generating Probabilistic Forecasts fromConvection-Allowing Ensembles Using Neighborhood Approaches: A Review andRecommendations. *Mon. Weather Rev.*, **145 (9)**, 3397-3418.
Wolff, J. K., M. Harrold, T. Fowler, J. H. Gotway, L. Nance, and B. G. Brown, 2014: Beyond theBasics: Evaluating Model-Based Precipitation Forecasts Using Traditional, Spatial, andObject-Based Methods. *Weather Forecast.*, **29 (6)**, 1451-1472.