Download - The choice between fixed and random effects models: some considerations for educational research
![Page 1: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/1.jpg)
The choice between fixed and random effects models: some considerations
for educational research
Claire Crawford with Paul Clarke, Fiona Steele & Anna Vignoles
![Page 2: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/2.jpg)
Motivation
• Appropriate modelling of pupil achievement– Pupils clustered within schools → hierarchical models
• Two popular choices: fixed and random effects• Which approach is best in which context?
– May depend whether primary interest is pupil or school characteristics
– But idea is always to move closer to a causal interpretation
![Page 3: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/3.jpg)
Outline of talk
• Why SEN?• Fixed and random effects models in the
context of our empirical question• Data and results• Conclusions
![Page 4: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/4.jpg)
Special educational needs (SEN)
• One in four Year 6 pupils (25% of 10 year olds) in England identified as having SEN– With statement (more severe): 3.7%– Without statement (less severe): 22.3%
• SEN label means different things in different schools and for different pupils– Huge variation in numbers of pupils labelled across schools– Assistance received also varies widely
• Ongoing policy interest (recent Green Paper)
![Page 5: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/5.jpg)
Why adjust for school effects?
• Want to estimate causal effect of SEN on pupil attainment no matter what school they attend
• Need to adjust for school differences in SEN labelling– e.g. children with moderate difficulties more likely to be
labelled SEN in a high achieving school than in a low achieving school (Keslair et al, 2008; Ofsted, 2004)
– May also be differences due to unobserved factors• Hierarchical models can account for such differences
– Fixed or random school effects?
![Page 6: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/6.jpg)
Fixed effects vs. random effects
• Long debate:– Economists tend to use FE models– Educationalists tend to use RE/multi-level models
• But choice must be context and data specific
![Page 7: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/7.jpg)
Basic model
issisis euXy 10
• FE: us is school dummy variable coefficient
• RE: us is school level residual– More flexible and efficient than FE, but:– Additional assumption required: E [us|Xis] = 0
• That is, no correlation between unobserved school characteristics and observed pupil characteristics
• Both: models assume: E [eis|Xis] = 0– That is, no correlation between unobserved pupil
characteristics and observed pupil characteristics
![Page 8: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/8.jpg)
Relationship between FE, RE and OLS
issisis euXy 10
)()(1 iisiisiis eeXXyy
)()(1 iisiisiis eeXXyy
FE:
RE:
Where:22 /1
11euS
![Page 9: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/9.jpg)
How to choose between FE and RE
• Very important to consider sources of bias:– Is RE assumption (i.e. E [us|Xis] = 0) likely to hold?
• Other issues:– Number of clusters– Sample size within clusters– Rich vs. sparse covariates – Whether variation is within or between clusters
• What is the real world consequence of choosing the wrong model?
![Page 10: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/10.jpg)
Sources of selection
• Probability of being SEN may depend on:– Observed school characteristics
• e.g. ability distribution, FSM distribution– Unobserved school characteristics
• e.g. values/motivation of SEN coordinator– Observed pupil characteristics
• e.g. prior ability, FSM status– Unobserved pupil characteristics
• e.g. education values and/or motivation of parents
![Page 11: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/11.jpg)
Intuition I
• If probability of being labelled SEN depends ONLY on observed school characteristics:– e.g. schools with high FSM/low achieving intake
are more or less likely to label a child SEN• Random effects appropriate as RE assumption
holds (i.e. unobserved school effects are not correlated with probability of being SEN)
![Page 12: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/12.jpg)
Intuition 2• If probability of being labelled SEN also depends
on unobserved school characteristics:– e.g. SEN coordinator tries to label as many kids SEN as
possible, because they attract additional resources• Random effects inappropriate as RE assumption
fails (i.e. unobserved school effects are correlated with probability of being SEN)
• FE accounts for these unobserved school characteristics, so is more appropriate– Identifies impact of SEN on attainment within schools
rather than between schools
![Page 13: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/13.jpg)
Intuition 3
• If probability of being labelled SEN depends on unobserved pupil/parent characteristics: – e.g. some parents may push harder for the label
and accompanying additional resources;– alternatively, some parents may not countenance
the idea of their kid being labelled SEN• Neither FE nor RE will address the
endogeneity problem:– Need to resort to other methods, e.g. IV
![Page 14: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/14.jpg)
Other considerations
• Other than its greater efficiency, the RE model may be favoured over FE where:– Number of observations per cluster is large
• e.g. ALSPAC vs. NPD– Most variation is between clusters
• e.g. UK (between) vs. Sweden (within)– Have rich covariates
![Page 15: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/15.jpg)
Can tests help?
• Hausman test: – Commonly used to test the RE assumption
• i.e. E [us|Xis] = 0
– But really testing for differences between FE and RE coefficients
• Over-interpretation, as coefficients could be different due to other forms of model misspecification and sample size considerations (Fielding, 2004)
– Test also assumes: E [eis|Xis] = 0
![Page 16: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/16.jpg)
Data
• Avon Longitudinal Study of Parents and Children (ALSPAC)– Recruited pregnant women in Avon with due
dates between April 1991 and December 1992– Followed these mothers and their children over
time, collecting a wealth of information:• Family background (including education, income, etc)• Medical and genetic information• Clinic testing of cognitive and non-cognitive skills• Linked to National Pupil Database
![Page 17: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/17.jpg)
Looking at SEN in ALSPAC
• Why is ALSPAC good for looking at this issue?– Availability of many usually unobserved individual
and school characteristics:• e.g. IQ, enjoyment of school, education values of
parents, headteacher tenure
![Page 18: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/18.jpg)
Descriptive statistics• 17% of sample are identified as having SEN at age 10
Individual characteristics School characteristicsStandardised KS1 APS -0.104** % eligible for FSM -0.002**IQ (age 8) -0.003** H’teacher tenure: 1-2 yrs -0.044**SDQ (age 7) 0.012** H’teacher tenure: 3-9 yrs -0.046**Mum high qual vocational -0.028* H’teacher tenure: 10+ yrs -0.031Mum high qual O-level -0.021
Mum high qual A-level -0.033*
Mum high qual degree -0.019 Observations 5,417
Notes: relationship between selected individual and school characteristics and SEN status. Omitted categories are: mum’s highest qualification is CSE level; head teacher tenure < 1 year.
![Page 19: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/19.jpg)
SEN resultsFixed
effectsRandom effects
Intra-school correlation
% difference
M1: KS1 APS only -0.335**[0.025]
-0.330**[0.025]
0.175 1.5
M2: M1 + admin data -0.347**[0.025]
-0.342**[0.025]
0.161 1.4
M3: M2 + typical survey data -0.355**[0.025]
-0.349**[0.024]
0.086 1.7
M4: M3 + rich survey data -0.321**[0.024]
-0.314**[0.024]
0.076 2.2
M5: M4 + school level data -0.321**[0.024]
-0.319**[0.024]
0.064 0.6
Notes: ** indicates significance at the 1% level; * at the 5% level. Robust standard errors are shown in parentheses.
![Page 20: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/20.jpg)
Summary of SEN results
• SEN appears to be strongly negatively correlated with progress between KS1 and KS2– SEN pupils score around 0.3 SDs lower
• Choice of model does not seem to matter here– FE and RE give qualitatively similar results– Suggests correlation between probability of having SEN
and unobserved school characteristics is not important • Consistency across specifications suggests regression
assumption is also likely to hold
![Page 21: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/21.jpg)
Summary of FSM results
• In contrast to the SEN results, the estimated effects of FSM on attainment decrease as richer data is used– Suggests that the regression assumption may fail in models
with few controls, such as those based on admin data • There are also relatively larger differences between
FE and RE models until we add school characteristics – Suggests that the RE assumption is less likely to hold here
![Page 22: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/22.jpg)
Conclusions
• Approach each problem with agnostic view on model– Should be determined by theory and data, not tradition
• FE should be preferred when the selection of pupils into schools is poorly understood or data is sparse
• RE should be preferred when the selection of pupils into schools is well understood and data is rich
• Worth remembering that neither FE nor RE deals with correlation between observed and unobserved individual characteristics
![Page 23: The choice between fixed and random effects models: some considerations for educational research](https://reader031.vdocuments.site/reader031/viewer/2022013123/5681674f550346895ddbff8c/html5/thumbnails/23.jpg)
FSM resultsFixed
effectsRandom effects
Intra-school correlation
% difference
M1: KS1 APS only -0.157**[0.028]
-0.175 0.145**[0.028]
11.5
M2: M1 + admin data -0.122**[0.028]
-0.138 0.161**[0.027]
13.1
M3: M2 + typical survey data -0.089**[0.029]
-0.103 0.086**[0.028]
15.7
M4: M3 + rich survey data -0.089**[0.028]
-0.102 0.076**[0.028]
14.6
M5: M4 + school level data -0.089**[0.028]
-0.095 0.064**[0.028]
6.7
Notes: ** indicates significance at the 1% level; * at the 5% level. Robust standard errors are shown in parentheses.