forecasting with judgment - stanford universityfinmath.stanford.edu › seminars › documents ›...
TRANSCRIPT
Forecasting with JudgmentForecasting with Judgment
Simone ManganelliDG-Research
European Central Bank(Frankfurt am Main, Germany)
Disclaimer: The views expressed in this paperare our own and do not necessarily reflect the
views of the ECB or the Eurosystem
Classical Theory of ForecastingClassical Theory of Forecasting
• Forecasts serve to make decisions about the future.• Forecast errors impose costs on the decision-maker.• Agents want to minimize expected loss associated to
forecast errors.• Classical forecast is the minimizer of the sample
equivalent of the expected loss.• See, e.g., Haavelmo (1944), Granger and Newbold
(1986), Granger and Machina (2005).
Optimal Point ForecastOptimal Point Forecast
])[()(min 210 !!
!"# +TyEQ
! =
"=
T
t tT yT1
1#̂
0][2 1 =!! + "TyEFOC: ][ 1*
+= TyE!
Ttty 1}{ = Want to forecast yT+1
Simplest case:• i.i.d. normally distributed observations with variance 1• quadratic loss function
The ProblemThe Problem
T!̂
! =
" "#T
t tT yTQ1
21 ])[()(ˆmin $$$
is the minimiser of:
]})[(])[(])[({min 211
2121 !!!
!"""+" +=
"+ # T
T
t tT yEyTyE
)(!"T
#
)}(])[({min 21 !"!
!TTyE +#+
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
OutlineOutline
0][)( 10 =!"# + $$ TyEQ
!!!~
ˆ]~
[)~(ˆ
1
1 "="#$ % =
"T
T
t tT yyTQ
maker.decision theof guess subjective a is ~
where!
Start from a Subjective GuessStart from a Subjective Guess])[()(min 2
10 !!!
"# +TyEQ
The sample equivalent of the FOCis:
NOTE: This is a random variable, which may be differentfrom zero just because of statistical error.
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
• Choose a confidence level α
• ηα/2 is the critical value
Then Do Hypothesis TestingThen Do Hypothesis Testingmean true theis
~ :0 !H
)/1,0(~~
ˆ TNyT !"
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
If null is rejected:
→ First derivatives statistically ≠ 0
→ Can be confident to decrease the true objective function
→ Only up to the point where new H0 cannot be rejected*ˆT!
2/!)~(ˆ !TQ"
)~(ˆ !TQ" )ˆ(ˆ *
TTQ !"
Graphical IllustrationGraphical Illustration
Distribution of FOC under H0
0
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
11
ˆ y=!
First derivatives not statisticallydifferent from zero
2/1
ˆ!"# $
2/1
ˆ!"# +
!~
])[( 2!"yE2
1 )( !"y
100*(1-α)%confidence interval
)(2 1y!"
!
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
IntuitionIntuition
New estimator:
!!"
!!#
$
%<%+
<%
>%%
=
2/2/
2/
2/2/
*
~ˆ if ˆ
|~
ˆ| if ~
~ˆ if ˆ
&&
&
&&
'('
'((
'('
(
TT
T
TT
T
yy
y
yy
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
InterpretationInterpretation
• α is the probability of committing type Ierrors, i.e. of rejecting the null when it is true.
• Choose low α when confident in subjectiveguess or if the cost of type I errors is high.
• Classical paradigm sets α =1:– Always FOC equal to zero;– No room for subjective guess;– It commits type I errors with probability 1.
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Problems with Pretest EstimatorsProblems with Pretest Estimators
!!~ˆ:
0=
TH
Test the following null hypothesis, for given confidence level α:
Then:
• If do not reject, keep the subjective guess
• If reject, take the maximum likelihood estimator
This is wrong!
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
11
ˆ y=!
First derivatives not statisticallydifferent from zero
2/1
ˆ!"# $
2/1
ˆ!"# +
!~
])[( 2!"yE2
1 )( !"y
(1-α)% confidenceinterval
)(2 1y!"
!
Introducing JudgmentIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Estimate the mean, given a single observation y1 (Magnus02)
)1,(...~ 0!NdiiytLet
!!"
!!#
$
%<%+
<%
>%%
=
2/12/1
2/1
2/12/1
*
1
~ if
|~
| if ~
~ if
&&
&
&&
'('
'((
'('
(
yy
y
yy
Compare the risk properties of two estimators:1) Subjective classical estimator (coincides with Magnus 02)
!"
!#$
>%
<%=
2/11
2/1
|~
| if
|~
| if ~
ˆ
&
&
'(
'(((
yy
yP
2) Pretest estimator
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Risk Associated to Risk Associated to ff((yy))
0
1
2
3
4
0 0.5 1 1.5 2 2.5 3 3.5 4
OLS (_=1) Subjective Guess (_=0)
Pretest (_=0.10) Subjective Classical (_=0.10)
0~
,]))([( 200 =! ""
"yfE
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Monte Carlo SimulationMonte Carlo Simulation
• Random draws from standard normal• Sample sizes = 5, 20, 60, 120, 240, 1000• Quadratic loss function• Two estimators: classical, new (α=0.10)• Evaluated expected loss with MC simulation• Repeat 5000 times and average
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
5 20 60 120 240 1000
Mean
0
0.05
0.1
0.5
1
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
ImplicationsImplications
• Good guess (gut feelings) are as important as goodeconometric models.
• Organization of forecasting process:– Subjective guess based on maximum likelihood estimates
can never be rejected by construction;– Clear separation b/w:
Who provides guess, based on judgment;Who tests the guess, based on econometric models.
• Shared responsibility for the quality of the forecasts:– High confidence in bad judgment results in bad forecast.
Introducing JudgmentRisk Analysis of New EstimatorRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
GeneralizationGeneralization),0()(ˆ 0
!"# NUT
d
T$$
]1,0[ ~
)1(ˆ)(*!"+# $%$%$$%
TT
0*
0 )(: null Under the !"! =T
H
))((ˆˆ))((ˆ))((ˆ *1*'* !"!"!" "" TTTTTTUUTz #$%$#&
'
kTT
TTkTT
kTT
z
Uz
z
,
*
*
]1,0[,
*
,
*
))((ˆ s.t.
))((ˆmaxargˆ ))0((ˆ if
0ˆ ))0((ˆ if
!
"!
!
#"$
"$"#$
"#$
=
=%>
=%&
'
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Mean-Variance Asset AllocationMean-Variance Asset Allocation• Mean-variance optimizers tend to produce portfolio
allocations with little or negative investment value.
• “[They] overuse statistically estimated informationand magnify the impact of estimation errors. It is notsimply a matter of garbage in, garbage out, but,rather, a molehill of garbage in, a mountain ofgarbage out” (Michaud 1998)
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Set upSet up
• N+1 assets
• yt,i return of asset i
• θi weight of asset i (they sum to 1)
!+
==
1
1 ,)(N
i itit yy "" portfolio return
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
OtpimizationOtpimization
})]([)]([{)]([
}]['{]['
)]([)]([];[
22 !!"!
!!"!
!"!!
ttt
tt
ttt
yEyEyE
yVyE
yVyEyU
##=
#=
#=
}])([)({)(];[ˆ 2
1
1
1
21
1
1 !!! =
"
=
"
=
" ""=T
t t
T
t t
T
t tt yTyTyTyU ##$##
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Implementation DetailsImplementation Details• Monthly log returns from DJIA• From January 1987 to July 2005• λ = 4 (coefficient of risk aversion)• Rolling windows (M=60 and 120 as in
DeMiguel et al. 2007)• Benchmark portfolio: equally weighted• Report out of sample differences in average
utilities between optimal and benchmarkportfolios
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
)}(])()[()({)()(1
12 !!!"!! t
T
Mt tttt yyMTyyU # +=
$$$$=
! += ""
" ""#T
Mt tttMTUUMTZ
1
*
1
1 )]~())ˆ(([)( $%$
Out of Sample EvaluationOut of Sample Evaluation
Test for statistical significance using Giacomini and White (2006)
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
00.92**0.01000.02α=0.01
1.48*4.19***0.82*00.090.48*α=0.10
2.616.47*1.82-8.40***1.00-0.30α=1
N=30N=16N=4N=30N=16N=4
M=120M=60
ResultsResultsIntroducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianConclusion
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPExample 2: Forecasting US GDPRelationship with BayesianConclusion
Implementation IssuesImplementation IssuesDifficult to formulate guess on abstract model parameters.Assume the decision-maker can formulate a guess on the variable
to be forecast (U.S. GDP, in this example). This guess can bemapped into a guess for model parameters as follows:
11~)(ˆ s.t.
)(ˆmaxarg~
++ =
=
TT
T
yy
U
!
!!!
Subjective guess
Model’s forecast
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPExample 2: Forecasting US GDPRelationship with BayesianConclusion
Implementation DetailsImplementation Details
ti itityy !"" ++= # = $
4
10
The model, AR(4):
! =
" ""=T
t ttT yyTU1
21 )](ˆ[)(ˆ ##
The objective function (quadratic):
The data:
• Quarterly U.S. real GDP growth rates
• FRED data base, seasonally adjusted
• From Q1 1983 to Q3 2005
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPExample 2: Forecasting US GDPRelationship with BayesianConclusion
ResultsResults
3.77%5%3%3%3.19%
0.01-0.040.050.050.04
-0.12-0.03-0.18-0.18-0.16
0.340.300.370.370.36
0.290.420.210.210.23
1.992.731.541.541.65
%3~
1=+Ty
0!
)ˆ(* !"T
%5~
1=+Ty
T!̂ !
~
1!
2!
3!
4!
)(ˆ1 !+Ty
!~
)ˆ(* !"T
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPExample 2: Forecasting US GDPRelationship with BayesianConclusion
Introducing JudgmentRisk Analysis of New EstimatorExample 1: Asset AllocationExample 2: Forecasting US GDPRelationship with BayesianRelationship with BayesianConclusion
Non sample information is summarised by:• priors, in Bayesian econometrics• subjective guess and confidence associated to it, in our case
Special cases:1. No uncertainty about optimal value of θ0:
– Bayesian: prior is degenerate distribution with total mass on θ0;– Classical: subjective guess=θ0, α=0 (i.e. never reject the null)
2. No information, besides the sample:– Bayesian: diffuse prior– Classical : α=1 (i.e. set FOC equal to zero)
In other intermediate cases, no clear mapping b/w the two.When non sample info if formulated:• via prior distributions, be Bayesian• via subjective guess and confidence, use subjective classical estimator
In general, the choice is dictated by the decision maker, throughthe format in which s/he provides the non sample info.
Relationship with Bayesian EconometricsRelationship with Bayesian Econometrics
ConclusionConclusion
• Ignoring non sample information and estimation errorare connected problems in the classical theory offorecasting.
• Forecasts should maximize the objective function in astochastic sense, not deterministic.
• Start from subjective guess and construct estimator(instead of the other way round)
• Test if FOC are statistically different from zero:– If not, subjective guess is retained as forecast– If yes, objective function is increased as long as FOC are ~= 0