Ontology and Methodology inStatistical Modeling and Inference:Lessons from Economics1

Aris Spanos [Dept. of Economics, Virginia Tech]

Outline:1. Statistical Inference and Reliability

how statistical misspecification leads to untrustworthy evidence2. Theory testing in Economics: revisiting the CAPM

Endemic misspecification bringing about a dead end3. Theory appraisal in the context of Error Statistics

Disentangling the statistical and substantive premises4. Questions of ontology: statistical vs. substantive models

Empirical examples: discovery of Argon, Kepler’s first law5. Summary and conclusions

1Presentation at the Ontology and Methodology conference, Virginia Tech, May 4-5, 2013.

1

1 Statistical Inference and Reliability

1.1 The Frequentist recasting of statistical induction: model-basedR. A. Fisher’s (1922)most remarkable achievementwas to recast statistical inductionby replacing generalizing observed ‘events’ relating to data z0, to modeling theunderlying ‘process’ that gave rise to z0, via the notion of a statistical model:

Mθ(z)={(z;θ) θ∈Θ} z∈R Θ⊂R (1.1.1)

Θ-parameter space, R-sample space, (z;θ)-joint distribution of the sample Z

Fisher’s devised a general way to ‘operationalize’ inferential errors by:(a) embedding the material experiment into a statistical modelMθ(z),(b) calibrating the capacity of inference procedures in terms of the relevant errorprobabilities in the context ofMθ(z). The recasting also rendered the inductivepremises testable vis-a-vis z0 Neyman and Pearson (1933) supplemented Fisher’s‘optimal’ estimation theory by proposing a theory of ‘optimal’ testing.The technical apparatus of frequentist statistical inference was largely in placeby the late 1930s, but the nature of the underlying inductive reasoning wasclouded in disagreements: ‘inductive inference’ vs. ‘inductive behavior’, etc.Moreover, neither reasoning could adequate answer the question (Mayo, 1996):‘when do data z0 provide evidence for or against a hypothesis or a claim?’

2

1.2 Statistical misspecification: how serious is the problem?What is often unappreciated is that behind every substantive modelMϕ(z)

there is (often implicit) a statistical modelMθ(z) whose validity vis-a-vis dataz0 underwrites the reliability of any statistical inferences based onMbϕ(z)H All statistical approaches are vulnerable since they invoke statistical models.invalidMθ(z)⇒ wrong likelihood (; z0)∝(z0;θ)⇒ Unreliable Inferences

Bayesian inference: erroneous posterior (|z0)=()(; z0) θ∈ΘKadane (2011): “... likelihoods are just as subjective as priors.” Nonsense! Theassumptions of (; z0) [Mθ(z)] are testable with data z0, those of () aren’t.Nonparametric: invalid heterogeneity/dependence/distributional assumptions.Frequentist: erroneous error probabilities and fit/prediction measures.A frequentist inference (estimation, testing, prediction) is rendered unreliablewhen the actual error probabilities differ from the nominal (assumed) ones!

I Applying a .05 -level test when the actual type I error probability is closer to .90,is very likely to lead to erroneous inferences. But lame excuses abound.

All models are wrong, but some are useful! Useful for what?

3

Example. Consider a simulation experiment using the Linear Regression model.

Table 1: Adequate model Table 2: Misspecified modelReplications=10000

TrueEstim: =15+05+True: =15+13+5+Estim.: =0 + 1 +

Estimated =50 =100 = 50 = 100

Parameters Mean Std Mean Std Mean Std Mean Std

[0=15] ̂0 1.502 .122 1.500 .087 0.462 .450 0.228 .315[1=5] ̂1 0.499 .015 0.500 .008 1.959 .040 1.989 .015[2=75] ̂2 0.751 .021 0.750 .010 2.945 .384 2.985 .266[R2=25]2 0.253 .090 0.251 .065 0.979 .003 0.995 .001

t-statistics Mean R(.05) Mean R(.05) Mean R(.05) Mean R(.05)

0=(̂0−0)̂0 0.004 .049 0.015 .050 -1.968 0.774 -3.531 0.9681=(̂1−1)̂1 -.013 .047 -.005 .049 35.406 1.000 100.2 1.000

Table 1: (i) estimates bθ(z0) are highly accurate and the empirical 'nominal(05), even for =50 and (ii) their accuracy improves as increases to =100Table 2: (iii) point estimates bθ(z0) are highly inaccurate and the empirical≫05, and (iv) the inaccuracies get worse as increases!

4

The intuitive explanation of what goes wrong in table 4 is that one is usingan inconsistent estimator of the means of ( ), which yields inconsistent(spurious) estimators of the variances, covariances and correlation coefficient.

1009080706050403020101

25

20

15

10

5

0

Index

y(t)

1009080706050403020101

5

4

3

2

1

0

Index

z(t)

Typical realization of Z(t)~NIID(2,1)

Inconsistent (statistically spurious) Consistent (statistically meaningful)1

P=1( − )2=3421 [=121] 1

P=1( − 2− 2)2 = 1

1

P=1( − )2=1804 [=807] 1

P=1( − 1− 14)2 = 1

1

P=1(−)(−)=2429 1

P=1(−2−2)(−1−14) = 5

1

P=1(−)(−)√

[ 1P

=1(−)2][ 1P

=1(−)2]=978

1

P=1(−2−2)(−1−14)√

[ 1P

=1(−2−2)2][ 1P

=1(−1−14)2]=5

Crucial distinction: Statistical vs. substantive meaningfulness

5

These spurious (artificially inflated) estimators will give rise to spurious teststatistics and goodness-of-fit measures, rendering them highly unreliable:

(1)=pP

=1(−)2(b1) 2=1−[P=1(−)2]−1

P=1 b2 2= 1

−2P

=1 b2 ∴ A statistically misspecifiedmodel (table 2) is useless for inference purposes.

2 Theory testing in Economics: a bird’s eye view

2.1 A key source of unreliability: foisting the theory on the dataTheory has generally held the pre-eminent role in economics, with data beinggiven the subordinate role of: ‘quantifying theories’ presumed true.Cairnes (1888) articulated an extreme version of the Pre-Eminence of Theory(PET) perspective arguing that data is irrelevant for appraising the ‘truth’ ofeconomic theories because they are infallible deductions from self-evident truths— established introspectively. In contrast to physics whose theories stem frommere inductive generalizations based on experimentation and inductive inferences,which are known to be fallible. The current PETperspective is almost as extreme:“The model economy which better fits the data is not the one used. Rather currentlyestablished theory dictates which one is used.” (Kydland and Prescott, 1991)From the PET perspective data does not so much test as allow instantiation of

6

theories: econometric methods offer elaborate (but often misleading) ways ‘tobring data into line’ with an assumed theory. Since the theory has little or nochance to be falsified, such instantiations provide no genuine tests of the theory.

Traditional Econometric Modeling Procedure [Lai and Xing (2008)]

Theory model

= +

= 0 6= 0→

Statistical model

= + +

[i] vNIID(0 2)[ii] ()=0 ∈N

Data( )

=1 ↓−→↑

Statisticalmethods

Statistical Inferenceestimation ( 2)testing ( 2)prediction: + ≥1simulation

The Capital Asset Pricing Model (CAPM), relates excess returns of a single asset=(−) and market excess returns =(−) where —returns on theparticular asset (CITI shares), —market returns (S&P500), —returns of riskfree asset (treasury bills); the CAPM assumes 1 assets.Data: monthly Aug. 2000 toOctober 2005 (=64) on log returns (:= ln(−1):

7

= 0053(0033)

+1137(089)

+ b3(0188)

2=725 =0188 =64 (2.1.2)

Lai & Xing claim that (2.1.2) provides strong evidence for the CAPM since:(a) the signs and magnitudes of the estimated ( ) are as predicted:(i) the CAPMrestriction =0 is accepted by the data: (z0;)= 0053

0033=1606[110]

(ii) the beta coefficient is statistically significant: (z0; )=1137089 =12775[000]

(b) the goodness-of-fit (2=725) provides additional support to the CAPM.IWhat potential errors could render unreliable an inferences based on (2.1.2)?[a] Substantively inadequate model: improper ceteris paribus clauses, miss-ing confounding factors, false causal claims, etc.[b] Inaccurate data: systematic errors imbued by the compilation process.[c] Incongruous measurement: data Z0 do not adequately quantify the con-cepts envisioned by the theory, e.g. intentions vs. realizations.[d] Statistically misspecified inductive premises: error term assumptions[i]-[ii] impose (indirectly) invalid probabilistic assumptions on data Z0:

[i] vNIID(0 2) ∈N[ii] ()=0 ∈N

¾À {Z:=( )vNIID(μΣ) ∈N}

The discussion that follows focuses primarily on [d]: statistical misspecification.

8

Step 1: Complete specification of inductive premises: Mθ(z).

{(|=) ∈N} {(|=) ∈N}(|=)vN( ) À [1] (|=) v N( ) (|=) = 0 À [2] (|=) =0 + 1¡2 |=

¢= 2 À [3] (|=) =

2

{(|=) ∈N} À [4] {(|=) ∈N} indep. process? [5]

¡0 1

2¢do not change with

Recasting [i]-[ii] into a complete set of probabilistic assumptions pertaining tothe observable process {(|=) ∈N} yields [1]-[5] (table 3).

Table 3: Normal, Linear Regression ModelStatistical GM: =0 + 1 + ∈N.[1] Normality: (|=) v N( )[2] Linearity: (|=) =0 + 1

[3] Homoskedasticity: (|=) =2

[4] Independence: {(|=) ∈N} indep. process[5] t-invariance:

¡0 1

2¢do not change with

⎫⎪⎪⎪⎪⎪⎬⎪⎪⎪⎪⎪⎭∈N.

0=()−1() 1=() ()

2= ()− [()]2

()

9

Step 2: Thoroughly test the inductive premises: assumptions [1]-[5](table 3). Mis-Specification (M-S) testing using pertinent auxiliary regressions:

b=10+11 +

[5]-non-constancyz }| {12 + 13

2 +

[2]-non-linearityz }| {14

2 +

[4]-non-independencez }| {15−1 + 16−1 + 1

0: 1 = 0 = 2 5

b2=20 + [5]-non-constancyz }| {22 + 23

2 +

[3]-heteroskedasticityz }| {21 + 24

2 +

[3]−[4]-non-independencez }| {25

2−1 + 26

2−1 + 2

0: 2 = 0 = 1 6b=√(−b0−b1) are the studentized residuals. Applying theseM-S tests to (2.1.2)indicate that [3]-[5] are clearly invalid; p-values in square brackets.

Table 4 - Mis-Specification (M-S) testsNormality: 0 = 1169[557]? Linearity: (1 61) = 468[496]

Homosk/city: (2 59)=4950[010]∗ Independ.: (1 59) = 615[016]∗

t-invariance: (2 60)=4611[014]∗

I These M-S test results undermine the above authors’s inferences (a)-(b)!No evidence for or against a theory can be drawn on the basis of a statisticallymisspecified model! Why? The actual type I error probabilities are likely to

10

be very close to 1.0, invaliding inferences (a)-(b)! Much worse than the simulationexample in table 4; M-S results can be guessed from the t-plots of the data!

60544842363024181261

0.10

0.05

0.00

-0.05

-0.10

t ime

y3(t

)

Time Series Plot of y3(t)

60544842363024181261

0.02

0.00

-0.02

-0.04

-0.06

-0.08

-0.10

t ime

X(t)

Time Series Plot of X(t)

60544842363024181261

10

5

0

-5

-10

time

y3(t

)

Fig. 1: t-plot of CITI excess returns

60544842363024181261

2

0

-2

-4

-6

-8

-10

time

x(t)

Fig. 2: t-plot of market excess returns

11

Step 3: WhenMbθ(z) is misspecified, pose no substantive questionsWhy? It elicits dubious answers. Consider posing the classic confounding variablequestion: is last period’s excess returns of General Motors, 1−1 an omitted butrelevant variable? Re-estimating the model with 1−1 added yields:

=0055(0031)

+1131(114)

-144(056)

1−1 + b1(0182)

2=747 =0182

Taking the t-test ( (z0)= 144056=2571) at face value, the answer is YES! A dubious

answer. Any variable that proxies the trends, shifts or cycles in fig. 1-2, is likelyto appear statistically significant, including generic trends and lags:

=0296(0116)

+1134(119)

-134(065)

+ 168(083)

2 + b2(0184)

2=745 =0184

=0034(003)

+124(094)

−175(071)

−1 + b3(0181)

2=75 =0181

Important: statistical misspecification also undermines goodness-of-fit/prediction.Intuitively, the latter calls for small but the statistical adequacy calls fornonsystematic (white-noise) residuals. It also undermines all goodness-of-fit/predictiondriven procedures, including the Akaike-type model selection:min. AIC=−2 ln(θ; z0) + 2, when (θ; z0) is erroneous?

12

What is one supposed to do next?

Strategy (?) 1: Respecify the CAPM (Mϕ(z)) to improve it. Problem: howdoes one assess improvement? Goodness-of-fit/predictionmeasures are unreliable.Strategy 2: Respecify the statistical model (2.1.2) to secure statistical adequacyand then test the CAPM and other modifications/extensions. Problem: Howdoes one respecify (2.1.2) without changing the underlying theory?Unreliable ‘error-fixing’ strategies for ‘upholding’ a theory. The PETperspective encourages modelers to uphold the original modelMϕ(z) and applyad hocmodifications to [i]-[ii] in an attempt to account for an apparent ‘anomaly’.Example 1. When the independence ([4]) assumption is rejected, one is supposedto ‘fix’ the problem by replacing 0: ()=0 for 6= with =−1+ i.e.adopting 1 of an M-S test — the fallacy of rejection; Mayo & Spanos (2004).Example 2. When the homoskedasticity ([3]) assumption is invalid a twofoldrecommendation is often prescribed. One ‘fixes’ the problem by: (i) an ad hocmodeling of heteroskedasticity using weighted least squares with b2=0+c>1 z+,or (ii) retaining the original OLS estimator bβ but replacing its Standard Errors(SEs) with Heteroskedasticity-Consistent SEs (HCSE).

INone of these ‘fixes’ brings the actual error probabilities closer to the nominal ones!

13

3 Theory appraisal in the context of Error Statistics

The problemwith the traditional approach. Foisting a theory on the databy attaching stochastic error terms to the theory model, gives rise to structuralmodel which is an amalgam of substantive and statistical assumptions. Hence,the estimatedMϕ(z) is often both statistically and substantively misspecified, butone has no principled way to distinguish between:

the theory is false or the inductive premises are invalid.

The key to circumventing this Duhemian ambiguity is to find a way to disen-tangle the statistical from the substantive premises, without compromising thecredibility of either source of information.

3.1 Disentangling the substantive and statistical premisesWhat is often not adequately appreciated in empirical research is that:I when confronting theory with data, behind every substantive model Mϕ(z),there is a statistical modelMθ(z); the two models are ontologically distinct!¥Mϕ(z) comes from the theory in light of dataZ0 but where does the statisticalmodelMθ(z) come from, if not fromMϕ(z)? From the probabilistic structureof the stochastic process {Z ∈N} underlying data Z0

14

An alternative perspective on the relationship betweenMθ(z) andMϕ(z) :

I The construction of the statistical model (premises)Mθ(z) begins with a givendata Z0 irrespective of the theory or theories that led to the choice of Z0.Once selected, data Z0 take on ‘a life of its own’ as a particular realization of ageneric stochastic process {Z ∈N} The link between data Z0 and the process{Z ∈N} is provided by a pertinent answer to the key question:what probabilistic structure, when imposed on the process {Z ∈N}would render data Z0 a typical realization thereof?’What do we mean by a typical realization of a generic process {Z ∈N}?

Figure 3 Figure 4

Fig. 3 exhibits dependence (cycles), and fig. 4 also exhibits mean heterogeneity.

15

In practiceMθ(z) is chosen with two objectives in mind.A. Typicality. The truly typical realization answer provides the apropos prob-abilistic structure for {Z ∈N}; testable using thorough M-S testing.The typicality generalizes Fisher’s (1922) original specification question: “Ofwhat population is this a random sample?” to accommodate non-random samples.B. Relevant parameterization. Mθ(z) is specified by choosing a parameter-ization θ∈Θ for {Z ∈N} to the structural modelMϕ(z) via G(θϕ)=0.This replaces the notion of a ‘population’ with that of a ‘generating mechanism’.Example 1. For the data in fig. 4, an appropriate model is the Normal AR(1)with a trend (θ:=(0 1 2)∈R×(−1 1)×R+):

Mθ(x): (|−1)vN(0+1+22+1−1 20) ∈N

The statistical model for the data in fig. 3: Mθ(x)|1=2=0.DisentanglingMθ(z) fromMϕ(z) delineates two very different questions:

[a] statistical adequacy: doesMθ(z) account for the chance regularities in Z0?[b] substantive adequacy: does the modelMϕ(z) shed adequate light(describe, explain, predict) on the phenomenon of interest?To establish [b] one needs to secure [a] and then probe for potential errors, likeceteris paribus clauses, confounding effects, false causal claims, etc.

16

4 Questions of ontology: statistical vs. substantive modelsA. A statistical modelMθ(z), when untangled fromthe substantivemodelMϕ(z), is built exclusively on the statistical information contained in data Z0and acts as a mediator betweenMϕ(z) and Z0Ontological commitments in specifyingMθ(z) concern the existence of:

(a) a rich enough probabilistic structure to ‘model’ the chance regularities in Z0(b) ∃θ∗∈Θ :Mθ∗(z)={(z;θ∗)} z∈R

could have generated data Z0.The ‘trueness’ of θ∗ is limited to whetherMθ∗(z) could have generated data Z0¥ In this sense, the ontological status ofMθ(z) is inextricably bound up witha particular frequentist methodology of inference and modeling.

I M-S testing assesses whether the familyMθ(z) θ∈Θ could have generated Z0regardless of the particular value θ∗.

I Statistical inference aims to narrow Θ down to θ∗- whatever θ∗ happens to be!

Inductive reasoning based on sampling distributions

Example. In the case of the simple Normal model:Mθ(x) : v NIID( 2) 2-known, ∈N

Estimation and testing adopt different forms of inductive reasoning, but their

17

primary aim is identical:learn from x0 about ∗ the ‘true’ (but unknown) value of ∈R

A (1−) Confidence Interval stems from the sampling distribution of a pivot:√(−)

=∗v N(0 1), where =1

P=1

evaluated under =∗ The N-P test for 0: =0 vs. 1: 6=0 uses thesampling distribution of a test statistic

√(−0)

evaluated under 0 & 1 :type I error prob.z }| {√

(−0)

=0v N(0 1),

type II error prob./powerz }| {√(−0)

=1v N(√(1−0)

1), for 1 6=0

What is often overlooked is that√([

∗]−0) is a scaled difference between ∗

[that generated ] and 0I The pre-data error probabilities calibrate the generic capacity (power) offrequentist tests: how effective a test is in detecting different discrepancies from 0 This capacity is often conflated with the empirical relative frequenciesof errors associated with the long-run metaphor, which invokes the ‘repeata-bility in principle’ [not in time or space] of the GM defined byMθ(x). Errorprobabilities are relevant for inferences with any x∈R

including data x0

18

¥Error Statistics (ER) can be viewed as a refinement/extension of the Fisher-Neyman-Pearson (F-N-P)motivated by the call to address several foundational problems.In particular, ER (a) refines the F-N-P approach by proposing a broader frame-work with a view to secure statistical adequacy, and reliability of inference, and(b) extends the F-N-P approach to provide an evidential interpretation forthe accept/reject rules and p-values. This is achieved using a post-data severityassessment that takes into account the test’s generic capacity (power) to out-put the discrepancies from =0 warranted or ruled out by data x0; Mayoand Spanos (2006). Severity also bridges the inferential gap between statistical(θ) and substantive (ϕ) parameters related via G(θϕ)=0, e.g. statistical vs.substantive significance, etc.

B. A substantive modelMϕ(z) is often viewed as aiming to approximate

the actual Data Generating Mechanism by abstracting, simplifying and focusing onparticular aspects (selecting the relevant observables Z) of the phenomenon ofinterest. In practice, however, the form ofMϕ(z) can vary widely from a highlycomplex system of dynamic (integro-differential) equations to a simple questionof interest. Consider two examples from the two ends of this spectrum.

19

Example 1. The discovery of Argon was partly based on probing thedifference in weight between nitrogen produced by two different broad methods,referred to as Atmospheric () and Chemical ( ); see table 4. The data weregenerated over a period of 18 months using different experimental techniques andprocedures to produce nitrogen and measure the weight of the end product.

Table 4 - Rayleigh data (1894) 2.31017 2.30986 2.31010 2.31001 2.31024 2.31010 2.31028 2.31163 2.30956 2.31026

2.30143 2.29890 2.29816 2.30182 2.29869 2.29940 2.29849 2.29889 2.30074 2.30054

Substantive model Mϕ(z): is there a difference in weight between the nitrogenproduced by the two procedures? TheMθ(z) is a simple Normal:

Mθ(z) : v NIID(1 2) v NIID(2 2) ∈N (4.0.3)

The substantive question is framed in terms of the statistical hypotheses:

0 : (1−2)=0 vs. 1 : (1−2) 0 (4.0.4)

in the context of Mθ(z) in (4.0.3). The material experiment is embedded inMθ(z) which is substantively fictitious. However,Mθ(z) is statistically adequatebecause the data z0 in figure 5 could have been generated by (4.0.3).

20

Inde x

Data

10987654321

2.312

2 .310

2 .308

2 .306

2 .304

2 .302

2 .300

2 .298

V ar iab lexy

Time S e r ie s P lot of x , y

Fig. 5: t-plots of z0:=( ) =1

The t-test (z0)=372678[000] rejects 0 and a severity evaluation:

SEV(; z0; ) = P( (Z) ≤ (z0);1=)=9

indicates that a substantive discrepancy ≥ 011 is warranted by data z0.It is interesting to note that the substantively true value is ∗=01186.Lesson: the statistical modelMθ(z) acting as amediator between the substan-tive question and z0, enabled us to find out (learn from data) true thingsabout the real difference in weight between atmospheric and chemical nitrogen,only to the extent that it does ‘account for the statistical regularities in z0’.

21

Example 2. Kepler’s first law states that ‘a planet moves around the sun inan elliptical motion with one focus at the sun’. The loci of the elliptical motionin polar coordinates is: (1) = 0 + 1 cos

where - the distance of the planet from the sun, - the angle between the linejoining the sun and the planet and the principal axis of the ellipse.

For :=(1) := cos this yields the substantive model:

Mϕ(z) : = 0 + 1 + =1 2 (4.0.5)

The statistical modelMθ(z) is the Linear Regression model (table 3), which whenestimated Brahe’s original data (=28) yields:

= 662062(000002)

+061333(000003)

+ b 2=9999 =00001115 (4.0.6)

(4.0.6) is statistically adequate, providing strong evidence for Kepler’s firstlaw. In this caseMϕ(z) andMθ(z) aim to approximate the actual DGM.

22

5 Summary and conclusions

We learn from data about phenomena of interest by applying reliable [actual' nominal error probabilities] and potent [optimal] inference procedures.I Foisting a theory on the data yields statistically and substantively inad-equate estimated models. A dead end! Addressing the Duhemian ambiguity:Step 1: Separate, ab initio, the substantive Mϕ(z) from the statisticalpremisesMθ(z) by viewing the latter as a parameterization of the process{Z ∈N} underlying data Z0 that nestsMϕ(z) via G(θϕ)=0Step 2: Secure statistical adequacy using trenchant M-S testing.Step 3: RespecifyMθ(z) with a view to find a statistically adequate model.

The ontological status of Mθ(z) is inextricably bound up with the error-statistical methodology of inference and modeling.

I M-S testing assesses whether the familyMθ(z) θ∈Θ could have generated Z0.I Statistical inference aims to narrow Θ down to θ∗- whatever θ∗ happens to be!

Step 4: Test the validity of the substantive model in the context of a statis-tically adequateMθ(z) WhenMϕ(z) is found empirically invalid, one canuse a statistically adequateMθ(z) to guide the search for a better theory!

23

IfMθ(z) then Q(z;θ) for z∈R

Statistical Model:Mθ(z) z∈R

Θ⊂R

(assumptions [1]-[5])

⇑

Data: z0=(1 2 )

=⇒

⇑Structural

ModelMϕ(z)

Inferential Propositions:

Q(z;θ) G(θϕ)=0⇒ Q1(z;ϕ)⇓ Z = z0

⇐ Inference results:Q(z0;θ)⇒ Q1(z0;ϕ)

Statistical adequacy- [A] ... ↓ [C] substantiveadequacy .[B] Severity

Model-based frequentist statistical inductionKey components of Error Statistics [A]-[C]

24