Discussion on “Diagnosing the Financial System: Financial Conditions

and Financial Stress”

by Scott Brave and R. Andrew Butters

Discussant: Chen ZhouDe Nederlandsche Bank

Views expressed do not necessarily reflect the view of DNB

The paper

• An indicator of financial condition (NFCI)• Thermometer for financial system• Based on a large scope of macro/finance data• Aggregating data with different frequency

• NFCI as an indicator/predictor for financial stress• Normal v.s. stress (crisis threshold)• ROC curve in threshold selection

An example in diagnosis

• Diagnosing a rare disease• Very rare disease Pr(Virus)=0.01%• Very reliable doctor/method

• P(Positive|Virus)=99.9%• P(Negatie|No Virus)=99.9%

• When getting a positive test result, should the patient worry?

Calculating Pr(Virus|Positive)

Calculation…

• Bayes formula

The patient does not have to worry too much!

%0.99.9999.99

9.99%99.99*%1.0%01.0*%9.99

%01.0*%9.99

)Pr()|Pr()Pr()|Pr(

)Pr()|Pr(

)|Pr(

NoVirusNoVirusPositiveVirusVirusPositive

VirusVirusPositive

PositiveVirus

What went wrong with the diagnosis?

• Why Pr(Virus|Positive) is not high• The diagnosis method seems to be reliable:

Pr(Positive|Virus) and Pr(Negative|No Virus) are both very high

• However, Pr(Virus|Positive)≠Pr(Positive|Virus)• If Pr(Positive)>P(Virus)

Pr(Virus|Positive)<Pr(Positive|Virus)• If a positive signal occurs too often, then it is

less informative.

A Close look at the formula

• As Pr(Virus) tends to zero, Pr(Virus|Positive) can be very low. Hence it is very difficult to diagnose rare disease.

)Pr()|Pr()Pr()|Pr(

)Pr()|Pr(

)|Pr(

NoVirusNoVirusPositiveVirusVirusPositive

VirusVirusPositive

PositiveVirus

)Pr(/)Pr()|Pr(/)|Pr(

)|Pr(/)|Pr(

)|Pr(

VirusNoVirusNoVirusPositiveVirusPositive

NoVirusPositiveVirusPositive

PositiveVirus

Very high

Should be very high to offset the effect that the disease is rare

999 9999

Lessons learned

• Diagnosing rare disease• Having low Type I and II errors is not sufficient• It is necessary to have a diagnosis system that

do no produce many “positive” signals.• It is necessary to have Type II error much

lower than Type I error

Rare disease is very difficult to diagnose by nature!

Back to diagnosing the financial system

• How about the NFCI as an indicator?• Financial stresses/crises are rare:

• In this paper P(Crisis)=52/1983=2.62%• With the threshold at -0.37, from ROC

• Pr(Signal|Crisis)=91%• Pr(Signal|No Crisis)=19%

• Thus Pr(Crisis|Signal)=11.4%, quite low!• Reason: Pr(Signal)=20.9%>>Pr(Crisis)

• See also Figure 3

How to improve?

• To diagnose rare events, it is necessary to have Type II error much lower than Type I error

• Currently: Type II error: 19%, Type I error 9%• Along the ROC curve, can we improve?

Here?

Type II: 2%

Type I: 38%

At the new point

• Calculations:• P(Crisis)=52/1983=2.62%• Pr(Signal|Crisis)=62%• Pr(Signal|No Crisis)=2%• Thus Pr(Crisis|Signal)=45.5%• Better, but still not satisfactory

• Threshold:• Much higher than -0.37 at the new point!

Which conditional probability?

• Comparison• In current paper:

Pr(Signal|Crisis) and Pr(No Signal|No Crisis)• We talked about:

Pr(Crisis|Signal) and Pr(No Crisis|No Signal)• Why the latter is more important than the

former?• Utility function: decisions from the signal

Can we improve?

• Absolute performance• Difficult to have Pr(Signal|Crisis) and

Pr(No Signal|No Crisis) both at high level• Not only NFCI, but also other indicators• Difficult as it is

• Relative performance• Can still be compared• Utility function based on the new sets of

conditional probability

An alternative ROC curve• Current ROC curve based on

Pr(Signal|No Crisis) v.s. Pr(Signal|Crisis)• New ROC curve based on

Pr(Crisis|No Signal) v.s. Pr(Crisis|Signal)• All existing techniques such as AUROC are still

valid for examining the relative performance between the NFCI and other indicators

• The choice of threshold: based on the new utility function and the new ROC curve

Not yet about forecasting

• Indicating crisis is already so difficult!• Forecasting is even more!• Alternative measure on forecasting performance

• Current: a “hit rate” measure• Alternative: Probability forecasting?

• NFCI might be linked to a good probability forecast of the distress

• Evaluating probability forecast

Always difficult to handle rare events!

Two main messages

When working on indicator/predictor of financial crisis (or any rare event)

• Should look at the right measures for evaluating the performance

• Should not be disappointed when looking at them