providing integrity for satellite navigation: lessons learned (thus far) from the financial collapse...

Providing Integrity for Satellite Navigation: Lessons Learned (Thus Far) from the Financial Collapse of

2008 – 2009

ION GNSS 2009 Savannah, GA.

24 September 2009

Sam Pullen

Stanford University

[email protected]

24 September 2009 Lessons Learned from 2008-09 Financial Crisis 2

Overview and Motivation

• My interest in this subject comes from my background in Probabilistic Risk Assessment (PRA), which formed the basis for my Ph.D. dissertation (Stanford, 1996).

– Optimal satellite design

– Optimal design of GPS integrity augmentations

• Studying and understanding failures of the past are the key to improving risk assessment.

– Motivation for Hurricane Katrina presentation from ION GNSS 2008

• Since SatNav augmentations have been demonstrated to be safe (with substantial margin), the benefit of this work for SatNav is improving risk-assessment for future systems and upgrades.

– Find means to reduce margin against “unknown unknowns”


Origins: The Changing Debt Market

Yield (%) on 10-Year Treasury Bonds: 1964 - 2009

Year

Yield

(%)

Low Treasury yields created demand for higher-yielding

investments.

Source: http://finance.yahoo.com


A Simplified Picture of the Financial Market Collapse

“Wine Glass Pyramid” Overview of Collateralized Debt Market

Source: Paddy Hirsch, “Crisis Explainer: Uncorking CDOs,” http://soundlearning. publicradio.org/subjects/economics_finance/financial_crisis/uncorking_cdos.shtml

Also see: Jonathan Jarvis, “The Crisis of Credit Visualized,” http://www.vimeo.com/3261363

mortgage payments

overflows when full

1st Tranche (AAA, 3%)

2nd Tranche (AA, 5%)

3rd Tranche (BBB, 7%)

4th Tranche (Unrated, 10%)

can further insure via credit default swaps

http://soundlearning.publicradio.org/subjects/economics_finance/financial_crisis/uncorking_cdos.shtml




http://www.vimeo.com/3261363


The Efficient Market Hypothesis (EMH)

• Origins (“random walk”) go back to 1900, but codified and popularized by the Eugene Fama (“Chicago school” of economists, 1960s).

• Expresses the concept that today’s prices reflect all available information, properly (“rationally”) judged.

• “The (market) price is right” – the foundation of quantitative economics– Traditional linear analysis with Markov state transitions

– Gaussian (or log-Normal) market-state transition probabilities are assumed (definition of random walk).

• Despite limited supporting evidence, EMH became widely accepted (and exploited) because of its academic and mathematical elegance/convenience.


Non-Gaussian Stock Market Behavior

P DeGrauwe, et al, “How Abnormal was the Stock Market in October 2008?” Euro Intelligence, 11 Nov. 2008. http://www.eurointelligence.com/article.581+M5f21b8d26a3.0.html

chg = 1.032%

http://www.eurointelligence.com/article.581+M5f21b8d26a3.0.html


An Honest Explanation from a Leading Master of Quantitative Finance

In finance we often assume that equity returns are normally distributed. … We find ourselves using the normal distribution quite naturally for many

financial processes.

As often with mathematical ‘laws’ there is the ‘legal’ small print, in this case the conditions under which the Central Limit Theorem applies. … Of course, financial data may not satisfy all of these, or indeed, any. In

particular, it turns out that if you try to fit equity returns data with non-normal distributions you often find that the best distribution is one that has

infinite variance. Not only does it complicate the nice mathematics of normal distributions and the Central Limit Theorem, it also results in infinite volatility. This is appealing to those who want to produce the best models of financial reality but does rather spoil many decades of financial theory

and practice based on volatility as a measure of risk for example.

Paul Wilmott, Frequently Asked Questions in Quantitative Finance (Wiley, 2007), pp. 33 – 35:


One Disastrous Outcome: The Copula Model for Credit Risk Correlation

• Before 2000, debt markets were much more conservative due to the complexity of modeling default risk and the small data base of major loan defaults.

• David Li of JP Morgan RiskMetrics group “removed” this difficulty by assuming a Gaussian copula formulation with a single correlation parameter derived from comparative market prices. – Justified by EMH since market prices confer “best” knowledge

• This approach led to dramatic growth in the credit-derivatives market until its fatal flaws were revealed by housing market crash of 2007-08.– Nationwide (correlated) loan defaults were not captured by model.

See Felix Salmon in Wired Magazine (March 2009)


Wilmott Actually Predicted this Disaster Years Beforehand

Abstract (conclusion)

Unfortunately, as the mathematics of finance reaches higher levels so the level of common sense seems to drop. There have been some well-publicized cases of large losses sustained by companies because of their lack of understanding of financial instruments. In this article we look at the history of financial modelling, the

current state of the subject and possible future directions. It is clear that a major rethink is

desperately required if the world is to avoid a mathematician-led market meltdown.

Paul Wilmott, “The Use, Misuse, and Abuse of Mathematics in Finance,” Philosph Trans: Math, Phys and Eng Sci, Vol. 358, No. 1765 (Jan. 2000), pp. 63-73.


The Financial Modeler’s Manifesto (in response to the crisis)

Our experience in the financial arena has taught us to be very humble in applying mathematics to markets, and to be extremely wary of ambitious theories, which are in the end trying to model human

behavior. We like simplicity, but we like to remember that it is our models that are simple, not the world … The greatest danger is the age-old sin of idolatry. Financial markets are alive but a model, however

beautiful, is an artifice. No matter how hard you try, you will not be able to breathe life into it. To confuse the model with the world is to embrace a future disaster driven by the belief that humans obey

mathematical rules.

MODELERS OF ALL MARKETS, UNITE! You have nothing to lose but your illusions.

The Modelers' Hippocratic Oath

~ I will remember that I didn't make the world, and it doesn't satisfy my equations.

~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.

~ I will never sacrifice reality for elegance without explaining why I have done so.

~ Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.

~ I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

Excerpted from Derman and Wilmott, The Financial Modelers’ Manifesto (Jan. 2009):

http://www.wilmott.com/blogs/paul/index.cfm/2009/1/8/Financial-Modelers-Manifesto

http://www.wilmott.com/blogs/paul/index.cfm/2009/1/8/Financial-Modelers-Manifesto


Key Lessons Applicable to Engineering Risk Modeling

1. Precise modeling of the unknown is not possible.

– Therefore, probabilistic models built upon uncertainty are preferred to deterministic ones.

2. Simplified risk models may be justified for specific threats, but model limitations must be given as much weight as the results.– Study assumptions carefully and “carry them forward” with

the results so that they are not “lost in time”.

– Avoid “falling in love” with models – keep a critical mind.

3. In particular, avoid extrapolating from a flawed model into the realm of absurdity.– Initial errors may be tolerable until exploited too aggressively

by a follow-up model.


As with other engineering risk analyses, SatNav integrity shares features with financial models:

1. Deterministic equations: cause-and-effect behavior is assumed known; uncertainty introduced by random variables with known distributions.

» The SatNav solution – worst-case modeling: where significant uncertainty cannot be removed, “worst-case” simplifications are derived to bound unknown reality.

2. Gaussian distribution: most random perturbations are modeled as Gaussian (or Gaussian variations)

» The SatNav solution: theory and data are combined to determine “inflation” factors such that the unknown “true” distribution is bounded at sufficiently low probabilities.

Specific Lessons for SatNav Integrity


Summary

• The financial crisis illustrates the perils of risk assess--ment based upon hubris and over-simplified models.

• This experience provides many useful lessons:– Probabilistic models are better when uncertainty is large.

– When using deterministic models:» Emphasize assumptions when presenting results.

» Avoid over-extrapolating from results.

– Remain open to new threats and threat model changes.

• SatNav integrity models are deterministic but apply multiple levels of caution against uncertainty.– Care applied to insure proper use of Gaussian distribution

– “Worst-case” error mitigation theoretically bounds all remaining uncertainty but often impacts user performance.


Backup Slides follow…


My Background in Risk Assessment

• Research on Probabilistic Risk Analysis (PRA) formed the core of my Ph.D. dissertation (1996).

• PRA in my thesis was applied to optimal design of satellites and GPS integrity augmentations (RAIM, WAAS).

• Since then, my work has focused on optimal design and verification of GPS augmentation systems with pre-designed (and highly-constrained) architectures.– Focus on GBAS (LAAS) ionosphere and ephemeris threats

– Focus on optimal diagnosis and isolation of detected faults

• Risk analysis failures outside of GNSS reinforce basic PRA principles and provide important lessons:– Hurricane Katrina (ION GNSS 2008)

– Recent financial crisis (this paper …)


Opening Thoughts

"The experience of being proved disastrously wrong is salutary.

No economist should be denied it, and none are."

- John Kenneth Galbraith (early 1980s)

Also by Galbraith:

"The only function of economic forecasting is to make astrology look respectable."

Sources: J. A. Smith, The Idea Brokers: Think Tanks and the Rise of the New Policy Elite (1993).

http://www.fool.co.uk/news/comment/2006/c060502g.htm


Overview

• Serious flaws in mathematical modeling directly contributed to the financial collapse of 2008 2009.

• The caution and consideration applied to integrity assurance for satellite navigation stands in sharp contrast to the hubris of the financial community.

• However, elements of the faulty financial models cited above exist in most traditional forms of risk analysis.

• This briefing examines what lessons, if any, can be learned that are relevant to risk assessment in general and SatNav integrity analysis in particular.


A Serious Outcome: Value-at-Risk (VaR) Modeling

• Devised by JP Morgan in the 1990’s to provide standardized trade and company-wide risk modeling – uses traditional linear-Gaussian statistics

• Key selling point: a single “Value at Risk” output– Represents a lower confidence bound, for a given percentile

(95th or 99th) and duration, on the amount that could be lost.

– VaR results available to managers in near-real time.

• Ease of use and simplicity of results led to massive over-dependence and abuse.– Uncertainty parameters set using limited historical data.

– Used as basis for capital requirements – led to insufficient reserves when financial crisis hit.

See Joe Nocera in NYT Magazine (4 January 2009)


“Misleading Information” Error Criterion

Normalized Vertical Position Error (no. of sigmas)

10

10

10

10

10

10

-10

-8

-6

-4

-2

0P

rob

abili

ty D

ensi

ty

VPL (for this

fault state)

KFFMDKfault

PFFMD

VAL (for this

flight operation)

Nominal Error plus 4

Bias Fault

-6 -4 -2 0 2 4 6 8 10 12 14 16

Gaussian dist. model

Vertical Alert Limit

Vertical Protection

Level


Model Definition vs. Reality

Level of Hazard

Error Size (meters)VAL

Typical Model

“Real-World” Model

In practice, maximum error bounds (VPL, VAL) are very conservative

actual “hazard level” at or just above VAL is low (if not zero).

VAL + is not materially more hazardous than VAL – .


Resulting Region of “Most Threatening” Error (Snapshot View)

Source: T. Zaugg, Proc. ION 58th Annual Meeting, June 2002.

= Er / MERR (normalized range error)

Range Error = MERR (range-domain bound)

Error giving max. hazard probability is much lower than MERR!


Dealing with the Unknown in GBAS: Ionosphere Anomaly Threat Modeling

Boundaries of resulting “threat model” for LAAS in CONUS

Based upon most severe anomalies observed in CONUS since 1999

Source: S. Datta-Barua, J. Lee, et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing Systems,” Submitted to AIAA J. of Aircraft (August 2009, under review).

0 10 20 30 40 50 60 70 80 900

50

100

150

200

250

300

350

400

450

Elevation [deg]

Slo

pe

[mm

/km

]

L1-L2 and L1 CMC

L1 CMC

L1 CMC (low-elev)

Worst-case gradients along

this upper bound.


Dealing with the Unknown in GBAS: Ionosphere Anomaly Threat Modeling

Boundaries of resulting “threat model” for LAAS in CONUS

Based upon most severe anomalies observed in CONUS since 1999

Source: S. Datta-Barua, J. Lee, et al, “Ionospheric Threat Parameterization for Local Area GPS-Based Aircraft Landing Systems,” Submitted to AIAA J. of Aircraft (August 2009, under review).

L1-L2 and L1 CMC

L1 CMC

L1 CMC (low-elev)

Worst-case gradients along

this upper bound.

0 100 200 300 400 500 600 700 8000

50

100

150

200

250

300

350

400

450

Ground Speed [m/s]

Slo

pe [m

m/k

m]

high elevation

low elevation


Iono. Anomaly “Wedge” Model Geometry

Simplified Ionosphere Front Model: a linear ramp defined by constant speed,

slope, and width

Front Speed 200 m/s

Airplane Speed ~ 70 m/s

(synthetic baseline due to smoothing ~ 14 km)

Front Width 25 km

GBAS Ground Station

Front Slope 425 mm/km

LGF IPP Speed 200 m/s

Max. ~ 6 km at DH


Impact of Ionosphere Anomaly Model and Worst-Case Error

0 5 10 15 20 25 30 35 40 450

0.02

0.04

0.06

0.08

0.1

0.12

0.14

User Vertical Position Error (meters)

PD

F

Worst-case error, or

“MIEV”, is 41 m

RTCA-24 Constellation; All-in-view, all 1-SV-out, and all 2-SV-out subsets included; 2 satellites impacted simultaneously by ionosphere anomaly

Most errors (~ 75%) are exactly zero due to detection/exclusion, but all zero errors have

been removed from the histogram.

28.8-meter tolerable limit

(CAT I PA)

Vast majority of non-zero errors are well

below tolerable limit.

Parameter Inflation

(“geometry screening”)

added to remove geometries with unsafe errors, but many good geometries are

removed as well significant

availability loss.


Features of Worst-Case Mitigation

• Theoretically, mitigating worst-case errors covers all threat scenarios and removes unquantifiable risk due to “unknown unknowns.”

• Worst-case mitigation almost always protects integrity with substantial margin (as in iono. anomaly example).

• Key limitation: “worst case” model remains dependent on un-provable assumptions.– Ongoing vigilance needed to monitor validity of key assumptions.

• On the “cost” side, the difficulty of mitigating worst-case scenarios stresses the resulting system:– User benefits may be significantly degraded.

– Loss of availability may have unforeseen safety implications.


How Much do Modeling Issues Matter?

• Unlike absurdities in financial modeling, assumptions made in GNSS risk modeling are “hubris-free” and are almost always conservative.

• Furthermore, in theory, focus on and mitigation of worst-case anomalies also covers all other threats.

• Unfortunately, “worst-case” anomalies are, by definition, difficult to counter and require extensive hardware/software/personnel resources.

• As a result, risk mitigation may become mis-focused, and sources of risk that do not easily fit the above models may get “assumed away”.


Similarities to Hurricane Katrina Lessons

• A key lesson from Katrina is the importance of maintaining flexibility and adaptability in risk-mitigation systems.

– New Orleans hurricane “threat model” did not change as better information became available.

– Lengthy political battles prevented improvements to hurricane defense system to address worsening threat understanding and flaws in levees discovered over time.

• The most obvious similarity here is the consistent refusal of mainstream financial economics to consider the obvious violations of EMH and their implications.

– Even today, anecdotal evidence suggests that mainstream financial economists are mostly “sticking with their story …”

providing integrity for satellite navigation: lessons learned (thus far) from the financial collapse...

Documents

financial collapse

market price

financial crisis5

financial crisis7

financial crisis4

financial crisis3 origins

financial crisis2 overview

changing debt market