An Updated Statistical Prediction Model for Solar Energetic Particles:
New Approaches
Christopher Balch Space Environment CenterSHINE Workshop Midway, UT31 July – 4 August 2006
Outline • Abstract/Introduction• Event database• Current model verification measures• Density estimation in brief• Results: one-variable density models• Results: two-variable density models• SummaryContact Info:
email: [email protected]: 303-497-5693mailing address: 325 Broadway, Boulder, CO 80305
AbstractSolar Energetic Particle (SEP) events are an important type of space weather phenomena and have real, significant impacts on human activities and systems in space as well as communication in the polar caps. The Space Environment Center (in Boulder Colorado) has the responsibility to be the official source for alerts, warnings, and forecasts of SEP's in the U.S. Although a physics based model for these predictions would be ideal, current forecasts are limited by making use of what solar phenomena are actually observed and available in real time. This has led to an emphasis on development of empirical statistical prediction methods. This paper will discuss an updated database of SEP events and a corresponding non-SEP event database that have been constructed for the last two solar cycles, 1986-2004. The paper will focus on new approaches to predicting SEP probability based on this data, including multi-variate density estimation and discriminant analysis. The performance of these methods will be compared with the current operational model using standard forecast verification measures.
Introduction: SEP Effects & Affected Activities
• Impacts of SEP’s– Human health hazard– Spacecraft Electronics– The Polar Ionosphere
Closed
Open
Open
• Affected Industries– Manned Spaceflight– Spacecraft Operations– HF communication– Airline operations
Introduction: SEC nowcasts, forecasts, products
• GOES real-time particle flux data– Range of energies 0.6 – 500 MeV
– Event defined as flux of 10 p cm-2 s-1 ster-1 (PFU) at > 10 MeV
• Daily forecast– Three day probabilities for SEP events
• Short term warnings– Based primarily on flare observations– Thresholds: 10 PFU 10 MeV and 1 PFU 100 MeV– Prediction for onset time, maximum flux, time of maximum flux, and expected
event duration• Alerts
– Real-time reports of an observed event– Issued shortly after threshold has been attained
• Additional notices at 100, 1000, and 10000 PFU.• Event Summaries
– Important for users to have an ‘all-clear’ indicator
Introduction: Practical Considerations for Operational Prediction Models
• Forecasters must know the necessary conditions for an SEP to occur
• Input parameters for the model must be valid for a specific parent solar event
• The observations that provide the parameters must be routinely available in real-time
• Assessment of a potential SEP must be done in real-time – prior to arrival of the first particles (particles may arrive before solar event is over)
• Today’s prediction models are based on empirical and statistical analyses
Event Database
• We consider two solar cycles of proton events observed during 1986-2004. • A review of all proton events from 1986-2004 was carried out by an examination of corrected, archived 5-
minute GOES integrated particle flux values.1 • A total of 165 events were identified and parameterized• Events originating from distinct solar sources were separated (even if a pre-existing event was still in
progress)• Associated GOES XRS events were identified by comparison of 1-minute x-ray flux time series with the
GOES particle data.1
• Associated ground-based H-alpha flare reports and radio sweep events were also carefully reviewed and included as appropriate2
• Associated CME events from the CDAW CME catalog were identified and associated as appropriate3
• Characteristics of the proton event were noted and recorded (e.g. late, shock-driven maximum (or secondary maximum), multiple-injection events, well-connected events, events which began when a previous event was still in progress, suspected backside source events, slow-riser events, events with associated erupting prominence)
• Events whose maximum was driven by the energetic storm particle (ESP) effect were flagged as such
1 Source: NOAA/SEC CDF archive files and NOAA/NGDC GOES archives
2 Source: NOAA/SEC event archives (collected from USAF-SEON network). Additional contributions from the NOAA/NGDC archives are gratefully acknowledged
3 Source: We gratefully acknowledge the CDAW Data Center (The CME catalog is generated and maintained by NASA and The CatholicUniversity of America in cooperation with the Naval Research Laboratory. SOHO is a project of international cooperation between ESA and NASA)
Control Event Database
• We find 27/165 of the SEP events are suspected to be from a backside source. We drop these from analysis because the radiative signals are absent or significantly affected.
• We find that 4/165 of the SEP events resulted from the ESP effect exclusively. The occurrence of these events is related to the physics of particle trapping in interplanetary shocks – which differs from most of the other events which have contributions of particles well ahead of the shock. These events are also dropped from the analysis
• The remaining 134 events are found to establish the following necessary conditions for an SEP event:– Peak X-ray Flux ≥ C2.4 (2.4 x 10-6 W m-2)– Integrated X-ray Flux ≥ 0.008 J m-2
– Background Subtracted Integrated X-ray Flux ≥ 0.00595 J m-2
• A careful review of GOES 1-minute XRS data and XRS event archives shows that there are 4038 events that meet the necessary conditions for the same time period (1986-2004). 134 are associated with proton events and 3904 are not.
• 3.3% of events meeting the necessary conditions result in a proton event• H-alpha and Radio event data was reviewed and associated as appropriate• CME event data was also reviewed and associated as appropriate• Patrol information for Radio observations and CME observations were included for each event (to indicate
whether observations were available at the time of the event)• Association (or not) with an SEP event is indicated for each control event
Event Parameters
Onset Time when 10 MeV flux begins to rise
Threshold Time when 10 MeV flux reaches 10 PFU
Maxtime Time of 10 MeV maximum flux
Endtime Time when 10 MeV flux drops below 10 PFU
EventType Characteristics of the event
P10Max Maximum flux at 10 MeV
P30Max Maximum flux at 30 MeV
P60Max Maximum flux at 60 MeV
P100Max Maxmum flux at 100 MeV
P10Fluence Integral of 10 MeV flux from threshold to end
P30Fluence Integral of 30 MeV flux from threshold to end
P60Fluence Integral of 60 MeV flux from threshold to end
P100Fluence Integral of 100 MeV flux from threshold to end
Associated XRS event
Identifies associated XRS event
Associated CME event
Identifies associated CME event
Proton Events parameters
Onset/Max/End XRS event times (End at ½ power point)
Peak Flux Maximum 1-8 Å x-ray flux (from XRS)
OptClass H-alpha optical class of associated flare
OptLocation H-alpha location of associated flare
Type II Identify association of type II radio sweep (Yes or No)
Type IV Identify association of type IV radio sweep (Yes of No)
CME onset Time of onset of associated CME
CME speed Leading edge CME speed (linear fit)
Integrated XRS flux Integral of XRS flux from onset to end
Bkgd Subtracted integrated XRS flux
Integral of background subtracted XRS flux from onset to end (background is taken to be the pre-event level)
Temperature1 Derived Temperature using ratio of two XRS channels
Emission Measure1 Derived Emission Measure using ratio of two XRS channels
Associated SEP event
Identifies associated proton event (or set to none if there is no association)
XRS/CME event parameters
1 Temperature and Emission measure were derived using the SolarSoft library routinesGOES_CHIANTI_TEM.PRO and GOES_MEWE_TEM.PRO
• Key Measures of forecast performance: Hsu and Murphy, 1986, describe the following attributes in assessing probability forecasts: quality, skill, reliability, and resolution
• Quality:
QR is the ‘quadratic score’, where fi are the forecast probabilities and oi are the observations (1=event, 0=no event). It is also the mean square deviation of forecasts from observations.
• Skill:
Measures quality of QR relative to QR*, where QR* is the quality of forecasts that use the overall event occurrence rate as the prediction for every event (e.g. 0.033 for proton events, given the necessary conditions)
• Partitioning of QR: The quality can be separated into three components as follows:
Where the forecast probabilities are divided into T distinct intervals, Ni is the number of forecasts (and corresponding observations) for each interval, <fi> is the average forecast probability for each interval, <oi> is the average occurrence rate of events for each interval, and ō is the overall average event occurrence rate.
• Reliability: The second term of the equation measures the correspondence between the forecast probability and the occurrence rate and is referred to as the reliability (REL) of the predictions. Smaller values of REL indicate more reliable forecasts and improve the overall quality.
• Resolution: The third term measures the ability of the forecasts to separate the sample into subsamples for which the respective relative frequencies differ from the average occurrence frequency and is referred to as the resolution (RES) of the predictions. Larger values of RES indicate forecasts with better resolution and improve the overall quality
• An equivalent form for the equation is QR = QR* + REL - RES
Assessing Probability Forecasts: Verification Measures
** /)( QRQRQRSS
N
i ii ofNQR1
2)()/1(
T
i
T
i iiiii ooNNofNNQRQR1 1
22* )()/1()()/1(
Current Model Performance Attributes
• Current Operational Model based on Balch (1999) study
• Using the database, the probability predictions from the model were tested
• Attributes diagram (Hsu and Murphy, 1986) shows forecast performance along with reference lines for perfect reliability, perfect resolution, zero skill, zero resolution
• Quality: QR=0.0246(rms error = 0.157)
• Skill: SS = (QR*-QR)/QR*=0.234• Reliability: REL = 0.0006• Resolution: RES = 0.0081• Fairly good reliability with one exception
(0.25-0.35 range)• Problem forecast bin 0.25-0.35: occurrence
rate for a type of event(≥X7, Type II/IV, Xint≥0.895) from 1999 study was 3/10 but new data shows 10/12
Verification Measures: Categorical Forecasts
• Suppose we define a probability threshold pt
• Suppose that we will issue a warning whenever the forecast fi ≥ pt. However, if fi < pt then no warning is issued
• Then the forecasts and observations can be analyzed in terms of a 2x2 contingency table shown to the right, where we observe that:A = number of hitsB = number of false alarmsC = number of missed eventsD = number of correct nulls
• The following statistical measures provide information about the quality of these categorical forecasts:POD = A / (A+C) – probability of detectionFAR = B/(A+B) – false alarm ratePC = (A+D)/N – percent correctHSS = (A+D–E)/(N-E) – Heidke skill score. HSS measures the fraction of correct forecasts adjusted by E, the number of forecasts we would expect to be correct by chance.
• For probability forecasts, probability threshold can be treated as an independent variable ranging from 0.0 to 1.0, hence each of the categorical quality measures (POD, FAR, PC, HSS) can be considered to be a function of pt
Yes No
Yes A B nw
No C D nn
np nc N
Event Observed
Eve
nt F
orec
ast
Derivation of EProbability of (event = Y) = (A+C)/NProbability of (forecast = Y) = (A+B)/NProbability of a chance hit P(event=Y and forecast=Y) (A+B)*(A+C)/N2
Probability of (event = N) = (B+D)/NProbability of (forecast = N) = (C+D)/NProbability of a chance correct null P(event=N and forecast=N) (B+D)*(C+D)/N2
Probability of chance hit or chance correct null: [(A+B)*(A+C) + (B+D)*(C+D)]/N2
Number of correct forecasts by chance: E = [(A+B)*(A+C) + (B+D)*(C+D)]/N
Skill score (HSS) optimization is achieved for range of probabilities 20-30%
Categorical quality measures for the current model
At optimal point:POD = 56% (75/134)FAR = 55% (91/166)PC = 96% (3888/4038) HSS = 0.48
White – PODRed – FARGreen – PCYellow – PCHBlue – GSSCyan - HSS
FAR falls as the threshold level is increased
However, POD also decreases with increasing threshold
Raw categorical quality measures for single parameters
White – PODRed – FARGreen – PCYellow – PCHBlue – GSSCyan - HSS
• Proton prediction performance based on a single parameter: z = log (background subtracted integrated x-ray flux)
• Prediction Rule: issue a warning if z zt, where zt is a threshold level.
• Performance is calculated as a function of various threshold levels
• Skill score (HSS) optimization is achieved for zt = -0.86 (background subtracted integrated x-ray flux = 0.14 J m-2)
• At optimal point:POD = 59% FAR = 59% PC = 96% HSS = 0.46
Results for Single Parameter Categorical Forecasts
Parameter Optimal Threshold
POD FAR PC HSS
Bg Subtr Int XRS flux
0.14 J m-2 59% 59% 96% 0.46
Int XRS flux 0.17 J m-2 51% 58% 96% 0.44
Log (Emission Measure cm-3)
50.3 36% 50% 97% 0.40
Peak XRS flux X1.9 37% 58% 96% 0.37
CME speed 1532 km/s 40% 60% 92% 0.36
Type II & IV Both occur 57% 77% 92% 0.30
Derived T 22 x 106 K 34% 82% 93% 0.20
Longitude ≥W79 9% 92% 92% 0.04
Single Variable Density Estimation• Density Estimation (Silverman, 1998) is a method of
deducing continuous probability density functions from observed data
• We consider the distribution of one of the parameters in our data set, for example integrated x-ray flux
• We expect to have different probability distributions for this parameter, depending on whether it is from the set of proton associated events, or from the set of events not associated with proton events
• Once smooth distributions are found for these two cases, we can deduce a continuous function for the probability for an event as a function of the parameter
Description of the Method• Let {Xi} be the set of n observed values of the parameter, i=1, 2,
…, n• The probability density function is defined on the continuous
domain of values x for this parameter, such that:
Where we use the normal probability density function for the kernel function K:
and h is a smoothing parameter • Geometrically, each observed value contributes a ‘bump’ to the
overall density estimate, so that the resulting function is a smooth, continuous probability density for the parameter Xi
f̂
n
i
i
h
XxK
nhxf
1
1ˆ
2)(
22zezK
Density estimate for log of background subtracted integrated x-ray flux for proton events using the normal probability density with a window width of 0.15
Density estimate for log of background subtracted integrated x-ray flux for events not associated with proton events (control events)
Example
Probability model for proton events, using the density functions
Probability = np*f1/(np*f1 +nc*f2),
np - number of proton eventsf1 - density estimate - proton associated events nc - number of control eventsf2 - density estimate for the control events
The probability is set to constant once maximum probability is reached
Probability model for proton events, using the density functions
Density Model metrics:
• Accuracy = 0.0235
• Rms error = 0.153
• Skill = 0.269
• Reliability=2.59 x 10-4
• Resolution=8.94 x 10-3
For comparison - metrics for operational model:
• Accuracy = 0.0246
• Rms error = 0.157
• Skill = 0.234
• Reliability=6.0 x 10-4
• Resolution=8.1 x 10-3
Density analysis for one parameter (background subtracted integrated x-ray flux) gives better accuracy, skill, reliability, and resolution !
However, the result is preliminary since the training set is being used for the verification calculation
Performance of categorical forecasts using probability thresholds
one parameter density model
Parameter is background subtracted integrated x-ray flux
Skill score optimization is at probability threshold of 0.18
At optimal point:POD = 59% (79/134)FAR = 59% (113/192)PC = 96% (3773/4020) HSS = 0.46
White – PODRed – FARGreen – PCYellow – PCHBlue – GSSCyan - HSS
• The method can be applied using more than one variable as a prediction inputs
• In this case we consider a set of n observation vectors Xi which consist two components
• Each component represents one of the observed variables (e.g. 1st component could be integrated x-ray flux, 2nd component could be CME speed)
• The density estimate in this two-dimensional parameter space is defined to be:
where we use the standard, multivariate normal density function:
n
i
i
hK
nhf
12
1ˆ Xxx
Two Parameter Density Estimation
2
exp)( 2
1 zzz
T
K
• In order to use a scalar smoothing parameter, h, it is necessary to normalize the observation vectors – we rescale all of the parameters so that their range is restricted to the interval [0,1]
2D Density Maps for Background Subtracted Integrated X-ray flux combined with H-alpha flare longitude
Red – proton event associated
Green – control event associated
Corresponding Probability Map for Proton Events
Performance Statistics for this 2D probability model(Log Background Subtracted Integrated X-ray Flux & Longitude)
QR = 0.0263RMS = 0.162Skill = 0.308REL = 7.60 x 10-4
RES = 1.34 x 10-2
Performance for Categorical Forecasts – 2D density model
At optimal point, threshold probability = 21%, POD = 53.7%, FAR = 41.5%, PC = 96.6%, HSS = 0.542
White – PODRed – FARGreen – PCYellow – PCHBlue – GSSCyan - HSS
2D Density Maps for CME speed combined with Emission Measure
Red – proton event associated
Green – control event associated
Corresponding Probability Map for Proton Events
Performance Statistics for this 2D probability model(CME speed and Emission Measure)
QR = 0.0423RMS = 0.206Skill = 0.316REL = 1.86 x 10-3
RES = 2.14 x 10-2
Performance for Categorical Forecasts – 2D density model
At optimal point, threshold probability = 24%, POD = 51.6%, FAR = 41.8%, PC = 94.3%, HSS = 0.517
White – PODRed – FARGreen – PCYellow – PCHBlue – GSSCyan - HSS
Parameter1 Parameter2 QR RMS SS REL RES
HSS
Thresh POD FAR PC
BgSub Int Xray Flux longitude 0.0263 0.162 0.325 7.60E-04 1.34E-02 0.543 0.21 53.7 41.5 96.6
Int Xray Flux longitude 0.0269 0.164 0.308 1.19E-03 1.32E-02 0.524 0.18 53.7 45.0 96.3
EM (Mewe) CME speed 0.0423 0.206 0.316 1.86E-03 2.14E-02 0.517 0.24 51.6 41.8 94.3
CME speed longitude 0.0544 0.233 0.313 3.87E-03 2.86E-02 0.495 0.25 43.6 32.5 93.3
EM (Chianti) CME speed 0.0442 0.210 0.286 1.37E-03 1.91E-02 0.490 0.28 45.2 39.1 94.4
Log Max Xray Flux CME speed 0.0443 0.210 0.285 1.45E-03 1.91E-02 0.484 0.27 45.2 40.4 94.3
Top Performers with respect to HSS2D probability density models
All of these pairs of parameters result in a prediction model the HSS better than the existing operational model
Summary• Current practical SEP prediction models are limited by available observations and typically rely on
empirical/statistical relationships
• A comprehensive event database for 1986-2004 has been compiled for SEC proton events and corresponding control events, where the control events are selected based on the necessary conditions identified from the proton events
• The current operational model performance was evaluated using the data – standard verification measures were derived for accuracy, reliability, resolution, and relative skill
• Categorical performance measures were also calculated as a function of probability thresholds
• Categorical performance based on optimal thresholds for input parameters were derived, but none performed better than the operational model
• Density Estimation techniques were applied to selected parameters: the performance of the probability forecasts were improved for one of the parameters relative to the operational model. The categorical verification measures did not change significantly relative to the simple parameter threshold method
• 2D densities and corresponding probability maps were derived. The following pairs of parameters were found to have improved performance measures relative to the operational model:Integrated X-ray Flux and Longitude, Emission Measure and CME speed, CME speed and longitude, Peak X-ray flux and CME speed.
• The method shows promise – but a more rigorous approach needs to used to derive the verification statistics
• We expect that adding more dimensions/parameters to the analysis will also improve the quality of these important space weather predictions