![Page 1: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/1.jpg)
On the consistency of supervised learning with
missing values
Erwan Scornet, Julie Josse, Nicolas Prost, Gael Varoquaux
CMAP, INRIA Parietal
27 March 2019
https://hal.archives-ouvertes.fr/hal-02024202
Chaire Data Analytics & Models for Insurance
1
![Page 2: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/2.jpg)
Overview
1. Introduction
2. Missing data mechanisms
3. A first approach : logistic regression
4. Existing methods from the supervised learning perspective
5. Decision trees
6. Discussion
2
![Page 3: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/3.jpg)
Introduction
![Page 4: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/4.jpg)
Context : big data in health and social sciences
• More and more missing data due to :
• high dimensionality (one feature may be missing)
• difficulty of fine control on the acquisition process
• Causal conclusions from analysis challenging :
• observational data (as opposed to experiments)
• missing data induces selection biases
New data sources challenge missing-data methodology :
high-dimensional
observational
uncontrolled confounds
3
![Page 5: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/5.jpg)
Traumabase
http://www.traumabase.eu/fr_FR
15000 patients/ 250 variables/ 11 hospitals, from 2011 (4000 new
patients/ year)
Center Accident Age Sex Weight Height BMI BP SBP
1 Beaujon Fall 54 m 85 NR NR 180 110
2 Lille Other 33 m 80 1.8 24.69 130 62
3 Pitie Salpetriere Gun 26 m NR NR NR 131 62
4 Beaujon AVP moto 63 m 80 1.8 24.69 145 89
6 Pitie Salpetriere AVP bicycle 33 m 75 NR NR 104 86
7 Pitie Salpetriere AVP pedestrian 30 w NR NR NR 107 66
9 HEGP White weapon 16 m 98 1.92 26.58 118 54
10 Toulon White weapon 20 m NR NR NR 124 73
...................
SpO2 Temperature Lactates Hb Glasgow Transfusion ...........
1 97 35.6 <NA> 12.7 12 yes
2 100 36.5 4.8 11.1 15 no
3 100 36 3.9 11.4 3 no
4 100 36.7 1.66 13 15 yes
6 100 36 NM 14.4 15 no
7 100 36.6 NM 14.3 15 yes
9 100 37.5 13 15.9 15 yes
10 100 36.9 NM 13.7 15 no
⇒ Predict whether to start a blood transfusion, to administer fresh
frozen plasma, etc... 4
![Page 6: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/6.jpg)
Traumabase
15000 patients/ 250 variables/ 11 hospitals, from 2011 (4000 new
patients/ year)
Center Accident Age Sex Weight Height BMI BP SBP
1 Beaujon Fall 54 m 85 NR NR 180 110
2 Lille Other 33 m 80 1.8 24.69 130 62
3 Pitie Salpetriere Gun 26 m NR NR NR 131 62
4 Beaujon AVP moto 63 m 80 1.8 24.69 145 89
6 Pitie Salpetriere AVP bicycle 33 m 75 NR NR 104 86
7 Pitie Salpetriere AVP pedestrian 30 w NR NR NR 107 66
9 HEGP White weapon 16 m 98 1.92 26.58 118 54
10 Toulon White weapon 20 m NR NR NR 124 73
...................
SpO2 Temperature Lactates Hb Glasgow Transfusion ...........
1 97 35.6 <NA> 12.7 12 yes
2 100 36.5 4.8 11.1 15 no
3 100 36 3.9 11.4 3 no
4 100 36.7 1.66 13 15 yes
6 100 36 NM 14.4 15 no
7 100 36.6 NM 14.3 15 yes
9 100 37.5 13 15.9 15 yes
10 100 36.9 NM 13.7 15 no
⇒ Estimate causal effect : administration of the treatment
”tranexamic acid” (within the first 3 hours after the accident) on
mortality (outcome) for traumatic brain injury (TBI) patients.4
![Page 7: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/7.jpg)
Traumabase
15000 patients/ 250 variables/ 11 hospitals, from 2011 (4000 new
patients/ year)
Center Accident Age Sex Weight Height BMI BP SBP
1 Beaujon Fall 54 m 85 NR NR 180 110
2 Lille Other 33 m 80 1.8 24.69 130 62
3 Pitie Salpetriere Gun 26 m NR NR NR 131 62
4 Beaujon AVP moto 63 m 80 1.8 24.69 145 89
6 Pitie Salpetriere AVP bicycle 33 m 75 NR NR 104 86
7 Pitie Salpetriere AVP pedestrian 30 w NR NR NR 107 66
9 HEGP White weapon 16 m 98 1.92 26.58 118 54
10 Toulon White weapon 20 m NR NR NR 124 73
...................
SpO2 Temperature Lactates Hb Glasgow Transfusion ...........
1 97 35.6 <NA> 12.7 12 yes
2 100 36.5 4.8 11.1 15 no
3 100 36 3.9 11.4 3 no
4 100 36.7 1.66 13 15 yes
6 100 36 NM 14.4 15 no
7 100 36.6 NM 14.3 15 yes
9 100 37.5 13 15.9 15 yes
10 100 36.9 NM 13.7 15 no
- missing data : Not Recorded, Made, Applicable, etc.
4
![Page 8: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/8.jpg)
Missing values
0
25
50
75
100AI
S.fa
ceAI
S.te
te
Choc
.hem
orra
gique
Trau
ma.
cran
ienGl
asgo
w
Anom
alie.
pupil
laire
IOT.S
MURFC
Myd
riase
Glas
gow.
initia
lAC
R.1
Cate
chola
mine
sPA
SPA
DTe
mps
.en.
rea
SpO2 Hb
DC.e
n.re
aPl
aque
ttes
Trait
emen
t.ant
iagre
gant
s
Trait
emen
t.ant
icoag
ulant
TP.p
ourc
enta
gePA
S.m
in
Glas
gow.
mot
eur.in
itial
FC.m
axPA
D.m
in
Vent
ilatio
n.Fi
O2Sp
O2.m
inFi
brino
gene
.1LA
TA
KTV.
pose
s.ava
nt.T
DM
Dose
.NAD
.dep
art
Tem
ps.d
epar
t.sca
nner
.ou.
bloc
Dern
iere.
PAS.
avan
t.dep
art
Dern
iere.
PAD.
avan
t.dep
art
Lacta
tes
Tem
ps.lie
ux.h
op
Glas
gow.
mot
eur
PaO2
pCO2
ARDS
Coup
leAl
cool
EER
FC.S
MUR
PAS.
SMUR
PAD.
SMUR
DTC.
IP.m
axPIC
Osm
othe
rapieDV
E
Cran
iecto
mie.
deco
mpr
essiv
e
Diplo
me.
plus.e
leve.
ou.n
iveau
Lacta
tes.H
2.1
DTC.
IP.m
ax.2
4h.H
TIC
HTIC
Lacta
tes.H
2
Glas
gow.
sorti
e
Lacta
tes.p
reho
spM
annit
ol.SS
H
Hypo
ther
mie.
ther
apeu
tique
Caus
e.du
.DC
Delai
.DC
Tem
ps.a
rrive
e.po
se.P
IC
Tem
pera
ture
.min
Regr
essio
n.m
ydria
se.so
us.o
smot
hera
pie
Tem
ps.a
rrive
e.po
se.D
VE
Variable
Perc
enta
ge
variablenull.data
na.data
nr.data
nf.data
imp.data
Percentage of missing values
5
![Page 9: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/9.jpg)
How to solve this problem ?
1. Delete all missing values → bad idea !
2. Impute data with your favorite imputation method
• Replace all missing values by the mean/median/mode of the
corresponding variable.
• Good point : the mean/median/mode is unchanged !
• Bad point 1 : the variance of the imputed data is lower than reality
• Bad point 2 : structure of dependence between variable is destroyed.
• Multiple imputation : very good idea but not the focus of this
presentation.
3. Design methods that handle missing values.
6
![Page 10: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/10.jpg)
How to solve this problem ?
1. Delete all missing values → bad idea !
2. Impute data with your favorite imputation method
• Replace all missing values by the mean/median/mode of the
corresponding variable.
• Good point : the mean/median/mode is unchanged !
• Bad point 1 : the variance of the imputed data is lower than reality
• Bad point 2 : structure of dependence between variable is destroyed.
• Multiple imputation : very good idea but not the focus of this
presentation.
3. Design methods that handle missing values.
6
![Page 11: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/11.jpg)
How to solve this problem ?
1. Delete all missing values → bad idea !
2. Impute data with your favorite imputation method
• Replace all missing values by the mean/median/mode of the
corresponding variable.
• Good point : the mean/median/mode is unchanged !
• Bad point 1 : the variance of the imputed data is lower than reality
• Bad point 2 : structure of dependence between variable is destroyed.
• Multiple imputation : very good idea but not the focus of this
presentation.
3. Design methods that handle missing values.
6
![Page 12: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/12.jpg)
Missing data mechanisms
![Page 13: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/13.jpg)
Missing values
Missing values are everywhere : unanswered questions in a
survey, lost data, damaged plants, machines that fail...
The best thing to do with missing values is not to have
any” Gertrude Mary Cox.
⇒ Still an issue with ”big data”
Data integration : data from different sources
i
I
1
1
k1
1
1
kI
ki
1 J
Xi
XI
X1
?
?
?
?
?
?
?????
?
?
?
Multilevel data : sporadically - systematic (one variable missing in one hospital) 7
![Page 14: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/14.jpg)
Available observations
We assume to be given an observed sample,
D?n = ((X?i ,Yi ))1≤i≤n ∼ (X?,Y ) i.i.d.,
where X? ∈ (R× {NA})d , resulting from the (unobserved) complete
sample
Dn = ((Xi ,Mi ,Yi ))1≤i≤n ∼ (X,M,Y ) i.i.d,.
For each variable index j and each realization i ,
• Mi,j = 0 means that Xi,j is observed,
• Mi,j = 1 means that Xi,j is missing.
In a nutshell,
X? := X⊗ (1−M) + NA ·M,
where ⊗ is the term-by-term product, NA× 1 := NA, NA× 0 := 0.
8
![Page 15: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/15.jpg)
The most simple NAs
For a realization of the complete sample,
dn =
2 3 1 0 0 0 1 0 0
1 0 3 5 0 1 0 0 0
9 4 2 5 0 0 0 1 0
7 6 3 2 0 0 1 1 1
,the observed sample is
d?n =
2 3 NA 0 0
1 NA 3 5 0
9 4 2 NA 0
7 6 NA NA 1
.MCAR
The pattern of missing data is said to be Missing Completely At Random
(MCAR) if the missing pattern M and the observations X are
independent. 9
![Page 16: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/16.jpg)
Rubin-MAR / realized MAR
Let m be the realization of M. We separate the observed values and the
missing values of x : x = (xobs , xmiss). The missing data are said to be
missing at random if g(m|x) is the same for all values of xmiss , i.e.
∀x, x′, xobs = x′obs ⇒ g(m|x) = g(m|x′).
Realized MAR is the minimal property to access the likelihood of missing
data. But...
• difficult to comprehend !
• Not useful to generate data.
10
![Page 17: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/17.jpg)
MAR/MNAR
(Everywhere) MAR
The missing data are said to be everywhere missing at random if for all
realizations m of M, g(m|x) is the same for all values of xmiss , i.e.
∀m, ∀x, x′, xobs = x′obs ⇒ g(m|x) = g(m|x′).
It should NOT be interpreted as M |= Xmiss |Xobs .
MNAR
The pattern of missing data is said to be Missing Non At Random
(MNAR) if it is not MAR, i.e., the pattern of missingness depends on
unobserved values.
11
![Page 18: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/18.jpg)
A first approach : logistic
regression
![Page 19: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/19.jpg)
Binary outcome
12
![Page 20: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/20.jpg)
Binary outcome
12
![Page 21: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/21.jpg)
Binary outcome
• Logistic model :
log
(P[CHD = 1|AGE ]
P[CHD = 0|AGE ]
)= (intercept) + β × AGE .
12
![Page 22: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/22.jpg)
Binary outcome with missing inputs
Litterature : Schaefer (2002) ; Little & Rubin (2002) ; Gelman & Meng (2004) ; Kim & Shao
(2013) ; Carpenter & Kenward (2013) ; van Buuren (2015)
⇒ Modify the estimation process to deal with missing values. Maximum
likelihood : EM algorithm to obtain point estimates + Supplemented
EM (Meng & Rubin, 1991) ; Louis for their variability
Difficult to establish ?
Not many implementations, even for simple models
One specific algorithm for each statistical method...
⇒ Imputation (multiple) to get a completed data set on which you can
perform any statistical method (Rubin, 1976)
⇒ Aim : estimate parameters and their variance from an incomplete data
13
![Page 23: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/23.jpg)
Logistic regression with missing covariates : parameter estima-
tion and prediction. (Jiang, Josse, Lavielle, Gauss, Hamada, 2018)
x = (xij) a n × d matrix of quantitative covariates
y = (yi ) an n-vector of binary responses {0, 1}Logistic regression model
P (yi = 1|xi ;β) =exp(β0 +
∑dj=1 βjxij)
1 + exp(β0 +∑d
j=1 βjxij)
Covariables
xi ∼i.i.d.Np(µ,Σ)
Log-likelihood for complete-data with θ = (µ,Σ, β)
LL(θ; x , y) =n∑
i=1
(log(p(yi |xi ;β)) + log(p(xi ;µ,Σ))
).
Decomposition : x = (xobs, xmis)
Under MAR, possibility to ignore the missing value mechanism
Observed likelihood arg maxLL(θ; xobs, y) =∫LL(θ; x , y)dxmis
14
![Page 24: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/24.jpg)
Stochastic Approximation EM
• E-step : Evaluate the quantity
Qk(θ) = E[LL(θ; x , y)|xobs, y ; θk−1]
=
∫LL(θ; x , y)p(xmis|xobs, y ; θk−1)dxmis
• M-step : θk = arg maxθ Qk(θ)
⇒ Unfeasible computation of expectation
MCEM (Wei & Tanner, 1990) : generate samples of missing data from
p(xmis|xobs, y ; θk−1) and replaces the expectation by an empirical mean.
⇒ Require a huge number of samples
SAEM (Lavielle, 2014) almost sure convergence to MLE. (Metropolis
Hasting - Variance estimation with Louis).
Unbiased estimates : β1, . . . , βd - V (β1), . . . , V (βd) - good coverage
15
![Page 25: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/25.jpg)
EM and out-of sample prediction
P (yi = 1|xi ;β) =exp(
∑dj=1 βjxij )
1+exp(∑d
j=1 βjxij )After EM : θ = (β1, β2, β3, . . . , βd , µ, Σ)
New obs : xn+1 = (x(n+1)1,NA,NA, x(n+1)4, . . . , x(n+1)d)
Predict Y on a test set with missing entries x = (xobs , xmiss)
y = arg maxy
pθ(y |xobs)
= arg maxy
∫pθ(y |x)pθ(xmis|xobs)dxmis
= arg maxy
EpXm|Xo=xopθ(y |Xm, xo) ≈ arg max
y
M∑m=1
pθ
(y |xobs, x (m)
mis
).
Logistic regression y1 yM
y = 1M
∑Mm=1 y
m
16
![Page 26: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/26.jpg)
EM and out-of sample prediction
P (yi = 1|xi ;β) =exp(
∑dj=1 βjxij )
1+exp(∑d
j=1 βjxij )After EM : θ = (β1, β2, β3, . . . , βd , µ, Σ)
New obs : xn+1 = (x(n+1)1,NA,NA, x(n+1)4, . . . , x(n+1)d)
Predict Y on a test set with missing entries x = (xobs , xmiss)
y = arg maxy
pθ(y |xobs)
= arg maxy
∫pθ(y |x)pθ(xmis|xobs)dxmis
= arg maxy
EpXm|Xo=xopθ(y |Xm, xo) ≈ arg max
y
M∑m=1
pθ
(y |xobs, x (m)
mis
).
Logistic regression y1 yM
y = 1M
∑Mm=1 y
m16
![Page 27: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/27.jpg)
Supervised learning with missing
values
![Page 28: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/28.jpg)
Supervised learning
• Data : input X and a response vector Y where
X = X� (1−M) + NA�M
(takes value in R ∪ {NA})• Find a prediction function that minimizes the expected risk.
Bayes rule : f ? ∈ arg minf :X→Y
E[`(f (X),Y )
]f ?(X) = E[Y |X]
• Empirical risk minimization :
fDn,train ∈ arg minf :X→Y
(1
n
n∑i=1
`(f (Xi ),Yi
))A new data Dn,test to estimate the generalization error rate
• Bayes consistent : E[`(fn(X),Y )] −−−→n→∞
E[`(f ?(X),Y )]
Difference with classical litterature
• response variable Y - Aim : Prediction
• two data sets (out of sample) with missing values : train & test sets
• No model !
17
![Page 29: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/29.jpg)
Supervised learning
• Data : input X and a response vector Y where
X = X� (1−M) + NA�M
(takes value in R ∪ {NA})• Find a prediction function that minimizes the expected risk.
Bayes rule : f ? ∈ arg minf :X→Y
E[`(f (X),Y )
]f ?(X) = E[Y |X]
• Empirical risk minimization :
fDn,train ∈ arg minf :X→Y
(1
n
n∑i=1
`(f (Xi ),Yi
))A new data Dn,test to estimate the generalization error rate
• Bayes consistent : E[`(fn(X),Y )] −−−→n→∞
E[`(f ?(X),Y )]
Difference with classical litterature
• response variable Y - Aim : Prediction
• two data sets (out of sample) with missing values : train & test sets
• No model !17
![Page 30: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/30.jpg)
Prediction on test incomplete data with a full data model
• Let a Bayes-consistent predictor f for complete data : f (X) = E[Y |X]
• Perform multiple imputation :
f ?mult imput(x) = EXm|Xo=xo [f (Xm, xo)]
same as out-of sample EM but assuming know f
Theorem
Consider the regression model Y = f (X) + ε, where
• we assume MAR ∀S ⊂ {1, . . . , d}, (Mj)j∈S |= (Xj)j∈S | (Xk)k∈Sc
• ε |= (M1,X1, . . . ,Md ,Xd) is a centred noise
Then multiple imputation is consistent :
f ?mult imput(x) = E[Y |X = x]
18
![Page 31: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/31.jpg)
Proof
Let x ∈ (R ∪ NA)d . Without loss of generality, assume only x1, . . . , xj are
NA.
f ?mult imput(x) = EXm|Xo=xo [f (Xm,Xo = xo)]
= E[f (Xm,Xo = xo)|Xo = xo ]
= E[Y |Xo = xo ]
= E[Y |Xj+1 = xj+1, . . . , Xd = xd ]
E[Y |X = x] = E[Y |X1 = NA, . . . , Xj = NA, Xj+1 = xj+1, . . . , Xd = xd ]
= E[Y |M1 = 1, . . . ,Mj = 1, Xj+1 = xj+1, . . . , Xd = xd ]
= E[Y |Xj+1 = xj+1, . . . , Xd = xd ]
19
![Page 32: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/32.jpg)
Imputation prior to learning
Impute the train, learn a model with Xtrain,Ytrain. Impute the test with
the same imputation and predict with Xtest and ftrain
? ??
?? ?
?
Xtrain Ytrain
? ??
?? ?
?
Xtest
Same imputation itrain itrain
Xtrain Ytrain Xtest Ytest
ftrain
Prediction model 20
![Page 33: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/33.jpg)
Imputation prior to learning
Imputation with the same model
Easy to implement for univariate imputation : the means (µ1, ..., µd) of
each colum of the train. Also OK for Gaussian imputation.
Issue : many methods are ”black-boxes” and take as an imput the
incomplete data and output the completed data (mice, missForest)
Separate imputation
Impute train and test separately (with a different model)
Issue : depends on the size of the test set ? one observation ?
Group imputation
Impute train and test simultaneously but the predictive model is learned
only on the training imputed data set
Issue : sometimes not the training set
21
![Page 34: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/34.jpg)
Imputation with the same model : mean imputation is consistent
Learn on the mean-imputed training data, impute the test set with the
same means and predict is optimal if the missing data are MAR and the
learning algorithm is universally consistent
Framework - assumptions
• Y = f (X) + ε
• X = (X1, . . . ,Xd) has a continuous density g > 0 on [0, 1]d
• ‖f ‖∞ <∞• Missing data on X1 with M1 |= X1|X2, . . . ,Xd .
• (x2, . . . , xd) 7→ P[M1 = 1|X2 = x2, . . . ,Xd = xd ] is continuous
• ε is a centered noise independent of (X,M1)
• imputed entry x′ = (x ′1, x2, . . . , xd) where
x ′1 = x11M1=0 + E[X1]1M1=1
• X = X′ if X ′1 6= E[X1] and X = (NA,X2, . . . ,Xd) if X ′1 = E[X1]22
![Page 35: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/35.jpg)
Imputation with the same model : mean imputation is consistent
Learn on the mean-imputed training data, impute the test set with the
same means and predict is optimal if the missing data are MAR and the
learning algorithm is universally consistent
Theorem
f ?impute(x ′) =E[Y |X2 = x2, . . . ,Xd = xd ,M1 = 1]
1x′1=E[X1]1P[M1=1|X2=x2,...,Xd=xd ]>0
+ E[Y |X = x′]1x′1=E[X1]1P[M1=1|X2=x2,...,Xd=xd ]=0
+ E[Y |X1 = x1,X2 = x2, . . . ,Xd = xd ,M1 = 0]1x′1 6=E[X1].
Prediction with mean is equal to the Bayes function almost everywhere
f ?impute(x ′) = f ?(X) = E[Y |X = x]
(remains valid when missing values occur for variables X1, . . . , Xj)22
![Page 36: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/36.jpg)
Imputation with the same model : mean imputation is consistent
Learn on the mean-imputed training data, impute the test set with the
same means and predict is optimal if the missing data are MAR and the
learning algorithm is universally consistent
Rationale
The learning algorithm learns the imputed value (here the mean) and use
that information to detect that the entry was initially missing. If the
imputed value changes from train to test set the learning algorithm may
fail, since imputed data distribution differs between train and test sets.
⇒ Other values than the mean are possible. Mean not a bad choice
despite its drawbacks.
22
![Page 37: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/37.jpg)
Trees
![Page 38: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/38.jpg)
General problem
New features [? ] :
X = X� (1−M) + NA�M,
E[Y∣∣∣X] =
∑m∈{0,1}d
E [Y |o(X,m),M = m] 1M=m.
Empirical risk minimization with missing values
fn ∈ arg minf∈F
1
n
n∑i=1
`(f (Xi ),Yi
).
Difficulties :
• Many different missingness patterns
• X is half-discrete (difficult optimization problem)
23
![Page 39: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/39.jpg)
Where are Random Forests ?
24
![Page 40: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/40.jpg)
No random forests !
Forests are overrated ! Let us consider a single tree.
25
![Page 41: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/41.jpg)
End-to-end learning with missing values
Xtrain Ytrain Xtest Ytest
p
prediction learner
Trees natural for empirical risk minimization with NA : handle half
discrete data.
26
![Page 42: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/42.jpg)
CART (Breiman, 1984)
Built recursively by splitting the current cell into two children : find the
feature j?, the threshold z? which minimises the (quadratic) loss
(j?, z?) ∈ arg min(j,z)∈S
E[(Y − E[Y |Xj ≤ z ]
)2 · 1Xj≤z
+(Y − E[Y |Xj > z ]
)2 · 1Xj>z
].
X1
X2 root
X1 ≤ 3.3 X1 > 3.3
X2 ≤ 1.5 X2 > 1.5
27
![Page 43: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/43.jpg)
CART (Breiman, 1984)
Built recursively by splitting the current cell into two children : find the
feature j?, the threshold z? which minimises the (quadratic) loss
(j?, z?) ∈ arg min(j,z)∈S
E[(Y − E[Y |Xj ≤ z ]
)2 · 1Xj≤z
+(Y − E[Y |Xj > z ]
)2 · 1Xj>z
].
X1
X2 root
X1 ≤ 3.3 X1 > 3.3
X2 ≤ 1.5 X2 > 1.5
27
![Page 44: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/44.jpg)
CART (Breiman, 1984)
Built recursively by splitting the current cell into two children : find the
feature j?, the threshold z? which minimises the (quadratic) loss
(j?, z?) ∈ arg min(j,z)∈S
E[(Y − E[Y |Xj ≤ z ]
)2 · 1Xj≤z
+(Y − E[Y |Xj > z ]
)2 · 1Xj>z
].
X1
X2 root
X1 ≤ 3.3 X1 > 3.3
X2 ≤ 1.5 X2 > 1.5
27
![Page 45: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/45.jpg)
CART with missing values : split on available cases
X1 X2
?
?
?
root
X1 ≤ s1 X1 > s1
X2 ≤ s2 X2 > s2
E[(
Y − E[Y |Xj ≤ z,Mj = 0])2 · 1Xj≤z,Mj =0 +
(Y − E[Y |Xj > z,Mj = 0]
)2 · 1Xj>z,Mj =0
].
Propagate missing values ?
Probabilistic splits : Bernouilli( #L#L+#R ) (C4.5 algorithm)
Block propag : send all to a side by minimizing the error (xgboost,
lightgbm)
Surrogate split : search for a split on another variable that induces a
partition close to the original one (rpart)
Implicit impute by an interval : missing values assigned to the left or right
28
![Page 46: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/46.jpg)
CART with missing values : split on available cases
X1 X2
?
?
?
root
X1 ≤ s1 X1 > s1
X2 ≤ s2 X2 > s2
E[(
Y − E[Y |Xj ≤ z,Mj = 0])2 · 1Xj≤z,Mj =0 +
(Y − E[Y |Xj > z,Mj = 0]
)2 · 1Xj>z,Mj =0
].
Propagate missing values ?
Probabilistic splits : Bernouilli( #L#L+#R ) (C4.5 algorithm)
Block propag : send all to a side by minimizing the error (xgboost,
lightgbm)
Surrogate split : search for a split on another variable that induces a
partition close to the original one (rpart)
Implicit impute by an interval : missing values assigned to the left or right
28
![Page 47: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/47.jpg)
CART with missing values : split on available cases
X1 X2
?
?
?
root
X1 ≤ s1 X1 > s1
X2 ≤ s2 X2 > s2
E[(
Y − E[Y |Xj ≤ z,Mj = 0])2 · 1Xj≤z,Mj =0 +
(Y − E[Y |Xj > z,Mj = 0]
)2 · 1Xj>z,Mj =0
].
Propagate missing values ?
Probabilistic splits : Bernouilli( #L#L+#R ) (C4.5 algorithm)
Block propag : send all to a side by minimizing the error (xgboost,
lightgbm)
Surrogate split : search for a split on another variable that induces a
partition close to the original one (rpart)
Implicit impute by an interval : missing values assigned to the left or right28
![Page 48: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/48.jpg)
Split on available observations
⇒ Biais in variable selection : tendency to underselect variables with
missing values (favor variables where many splits are available)
⇒ Conditional tree (Hothorn, 2006) Ctree selects variables with a test{X1 |= X2 ∼ N (0, 1)
Y = 0.25X1 + ε with ε ∼ N (0, 1)
Frequency of selection of X1 when there are missing values on X1 :
CART selects the non-informative variable X2 more frequently 29
![Page 49: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/49.jpg)
Missing incorporated in attribute, Twala et al 2008
Selection of the variable, threshold and propagation of missing values
f ? ∈ arg minf∈Pc,miss
E[(Y − f (X)
)2],
where Pc,miss = Pc,miss,L ∪ Pc,miss,R ∪ Pc,miss,sep with
• Pc,miss,L → {{Xj ≤ z ∨ Xj = NA}, {Xj > z}}• Pc,miss,R → {{Xj ≤ z}, {Xj > z ∨ Xj = NA}}• Pc,miss,sep → {{Xj 6= NA}, {Xj = NA}}.
⇒ Missing values treated like a category (well to handle R ∪ NA)
⇒ Target E[Y∣∣∣X] =
∑m∈{0,1}d E [Y |o(X,m),M = m] 1M=m
⇒ Good for informative pattern
⇒ Implementation : duplicate the incomplete columns, and replace the
missing entries once by +∞ and once by −∞.
30
![Page 50: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/50.jpg)
Split comparison
{Y = X1
X1 ∼ U([0, 1]),
{P[M1 = 0] = 1− p
P[M1 = 1] = p,
The best split CART s? = 1/2
The split chosen by the MIA
31
![Page 51: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/51.jpg)
Risk comparison
Consider the regression modelY = X1
X1 ∼ U([0, 1])
X2 = X11W=1
,
{P[W = 0] = η
P[W = 1] = 1− η,
{P[M1 = 0] = 1− p
P[M1 = 1] = p,
where (M1,W ) |= (X1,Y ).
0.00 0.25 0.50 0.75 1.00Fraction p of missing values
0.00
0.02
0.04
0.06
0.08
0.10
0.12
Quad
ratic
risk
R(p
)
= 0.0
0.00 0.25 0.50 0.75 1.00Fraction p of missing values
0.00
0.02
0.04
0.06
0.08
0.10
0.12
Quad
ratic
risk
R(p
)
= 0.25
0.00 0.25 0.50 0.75 1.00Fraction p of missing values
0.00
0.02
0.04
0.06
0.08
0.10
0.12
Quad
ratic
risk
R(p
)
= 0.75
Probabilistic splitBlock propagation
MIASurrogate splits
32
![Page 52: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/52.jpg)
Simulations : 20% missing values
Quadratic : Y = X 21 + ε, xi. ∈ N (µ3,Σ3×3), ρ = 0.5, n = 1000
MCAR (MAR) MNAR Predictive
RANDOM FOREST
DECISION TREE
−0,2 0,0 +0.2
1. MIA2. impute mean
+ mask3. impute mean4. impute Gaussian
+ mask5. impute Gaussian6. rpart (surrogates)
+ mask7. rpart (surrogates)8. ctree (surrogates)
+ mask9. ctree (surrogates)
1. MIA
2. impute mean
+ mask
3. impute mean
4. impute Gaussian
+ mask
5. impute Gaussian
Relative explained variance
RANDOM FOREST
DECISION TREE
−0,4 −0,2 0,0 +0.2Relative explained variance
RANDOM FOREST
DECISION TREE
−0,4 −0,2 0,0 +0.2Relative explained variance
Mi,1 ∼ B(p) Mi,1 = 1Xi,1>[X1](1−p)nY = X 2
1 + 3M1 + ε 33
![Page 53: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/53.jpg)
Consistency : 40% missing values MCAR
103 104 105
Sample size
0.4
0.6
0.8
Expl
aine
d va
rianc
eLinear problem(high noise)
103 104 105
Sample size
0.4
0.6
0.8
Expl
aine
d va
rianc
e
Friedman problem(high noise)
103 104 105
Sample size
0.6
0.8
1.0
Expl
aine
d va
rianc
e
Non-linear problem(low noise)
DECISIO
N TREE
103 104 105
Sample size
0.65
0.70
0.75
0.80
0.85
Expl
aine
d va
rianc
e
103 104 105
Sample size
0.5
0.6
0.7
Expl
aine
d va
rianc
e
103 104 105
Sample size
0.96
0.98
1.00
Expl
aine
d va
rianc
e
RAND
OM
FOREST
rpart (surrogates)rpart (surrogates) + maskMean imputationMean imputation + maskGaussian imputation
Gaussian imputation + maskMIActree (surrogates)Bayes rate
34
![Page 54: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/54.jpg)
Discussion
![Page 55: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/55.jpg)
Discussion
Take-home
• Consistent learner for the fully observed data → multiple
imputation on the test set
• Incomplete train and test → same imputation model
• Single mean imputation is consistent, provided a powerful
learner
• tree-based models → Missing Incorporated in Attribute optimizes
not only the split but also the handling of the missing values
• Empirically, good imputation methods reduce the number of samples
required to reach good prediction
• Informative missing data ?Adding the mask helps imputation !
To be done
• Nonasymptotic results
• Prove the usefulness of methods in MNAR
• Uncertainty associated with the prediction
• Distributional shift : no missing values in the test set ? 35
![Page 56: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/56.jpg)
Thank you !
36
![Page 57: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/57.jpg)
References i
References
[1] S. Athey, J. Tibshirani, and S. Wager. Generalized random forests. Ann.
Statist., 47(2) :1148–1178, 2019.
[2] J. Carpenter and M. Kenward. Multiple Imputation and its Application.
Wiley, Chichester, West Sussex, UK, 2013.
[3] S. R. Hamada, J. Josse, S. Wager, T. Gauss, et al. Effect of fibrinogen
administration on early mortality in traumatic haemorrhagic shock : a
propensity score analysis. Submitted, 2018.
[4] W. Jiang, J. Josse, M. Lavielle, et al. Stochastic approximation em for
logistic regression with missing values. arXiv preprint arXiv :1805.04602,
2018.
37
![Page 58: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/58.jpg)
References ii
[5] J. Josse, J. Pages, and F. Husson. Multiple imputation in principal
component analysis. Advances in Data Analysis and Classification,
5(3) :231–246, 2011.
[6] N. Kallus, X. Mao, and M. Udell. Causal inference with noisy and missing
covariats via matrix factorization. arXiv preprint, 2018.
[7] J. K. Kim and J. Shao. Statistical Methods for Handling Incomplete Data.
Chapman and Hall/CRC, Boca Raton, FL, USA, 2013.
[8] R. J. A. Little and D. B. Rubin. Statistical Analysis with Missing Data.
Wiley, 2002.
[9] D. K. Menon and A. Ercole. Critical care management of traumatic brain
injury. In Handbook of clinical neurology, volume 140, pages 239–274.
Elsevier, 2017.
[10] J. M. Robins, A. Rotnitzky, and L. P. Zhao. Estimation of regression
coefficients when some regressors are not always observed. Journal of the
American Statistical Association, 89(427) :846–866, 1994.
38
![Page 59: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/59.jpg)
References iii
[11] J. L. Schafer and J. W. Graham. Missing data : our view of the state of
the art. Psychological Methods, 7(2) :147–177, 2002.
[12] A. Sportisse, C. Boyer, and J. Josse. Imputation and low-rank estimation
with missing non at random data. arXiv preprint arXiv :1812.11409, 2018.
[13] S. van Buuren. Flexible Imputation of Missing Data. Chapman and
Hall/CRC, Boca Raton, FL, 2018.
[14] S. Wager. Lecture notes in causal inference and treatment effect
estimation (oit 661), 2018.
39
![Page 60: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/60.jpg)
Context
Major trauma : any injury that endangers the life or the functional
integrity of a person. Road traffic accidents, interpersonal violence,
self-harm, falls, etc → hemorrhage and traumatic brain injury.
Major source of mortality and handicap in France and worldwide
(3rd cause of death, 1st cause for 16-45 - 2-3th cause of disability)
⇒ A public health challenge
Patient prognosis can be improved : standardized and reproducible
procedures but personalized for the patient and the trauma system.
Trauma decision making : rapid and complex decisions under time
pressure in a dynamic and multi-player environment (fragmentation : loss
or distortion of information) with high levels of uncertainty and stress.
Issues : patient management exceeds time frames, diagnostic errors,
decisions not reproducible, etc
⇒ Can Machine Learning, AI help ?
40
![Page 61: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/61.jpg)
Decision support tool for the management of severe trauma :
Traumamatrix
REAL TIME ADAPTIVE AND LEARNING INFORMATION MANAGEMENT
FOR ALL PROVIDERS
Audio-Visual Cockpit style Decision Support
PATIENT CENTERED PROBABILISTIC DECISION SUPPORT
41
![Page 62: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/62.jpg)
Causal inference for traumatic brain injury with missing values
• 3050 patients with a brain injury (a lesion visible on the CT scan)
• Treatment : tranexamic acid (binary)
• Outcome : in-ICU death (binary), causes : brain death, withdrawal of
care, head injury and multiple organ failure.
• 45 quantitative & categorical covariates selected by experts
(Delphi process). Pre-hospital (blood pressure, patients reactivity,
type of accident, anamnesis, etc. ) and hospital data
0
25
50
75
100
AIS.
face
AIS.
tete
Choc
.hem
orra
gique
Trau
ma.
cran
ienGl
asgo
w
Anom
alie.
pupil
laire
IOT.S
MURFC
Myd
riase
Glas
gow.
initia
lAC
R.1
Cate
chola
mine
sPA
SPA
DTe
mps
.en.
rea
SpO2 Hb
DC.e
n.re
aPl
aque
ttes
Trait
emen
t.ant
iagre
gant
s
Trait
emen
t.ant
icoag
ulant
TP.p
ourc
enta
gePA
S.m
in
Glas
gow.
mot
eur.in
itial
FC.m
axPA
D.m
in
Vent
ilatio
n.Fi
O2Sp
O2.m
inFi
brino
gene
.1LA
TA
KTV.
pose
s.ava
nt.T
DM
Dose
.NAD
.dep
art
Tem
ps.d
epar
t.sca
nner
.ou.
bloc
Dern
iere.
PAS.
avan
t.dep
art
Dern
iere.
PAD.
avan
t.dep
art
Lacta
tes
Tem
ps.lie
ux.h
op
Glas
gow.
mot
eur
PaO2
pCO2
ARDS
Coup
leAl
cool
EER
FC.S
MUR
PAS.
SMUR
PAD.
SMUR
DTC.
IP.m
axPIC
Osm
othe
rapieDV
E
Cran
iecto
mie.
deco
mpr
essiv
e
Diplo
me.
plus.e
leve.
ou.n
iveau
Lacta
tes.H
2.1
DTC.
IP.m
ax.2
4h.H
TIC
HTIC
Lacta
tes.H
2
Glas
gow.
sorti
e
Lacta
tes.p
reho
spM
annit
ol.SS
H
Hypo
ther
mie.
ther
apeu
tique
Caus
e.du
.DC
Delai
.DC
Tem
ps.a
rrive
e.po
se.P
IC
Tem
pera
ture
.min
Regr
essio
n.m
ydria
se.so
us.o
smot
hera
pie
Tem
ps.a
rrive
e.po
se.D
VE
Variable
Perc
enta
ge
variablenull.data
na.data
nr.data
nf.data
imp.data
Percentage of missing values
42
![Page 63: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/63.jpg)
Imputation assuming a joint modeling with gaussian distribu-
tion
based on Gaussian assumption : xi. ∼ N (µ,Σ)
• Bivariate with missing on x.1 (stochastic reg) : estimate β and σ -
impute from the predictive xi1 ∼ N(xi2β, σ
2)
• Extension to multivariate case : estimate µ and Σ from an incomplete
data with EM - impute by drawing from N(µ, Σ
)equivalence
conditional expectation and regression (complement Schur)
packages Amelia, mice (conditional)
43
![Page 64: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/64.jpg)
PCA reconstruction
-2.00 -2.74 -1.56 -0.77 -1.11 -1.59 -0.67 -1.13 -0.22 -1.22 0.22 -0.52 0.67 1.46 1.11 0.63 1.56 1.10 2.00 1.00
-2.16 -2.58 -0.96 -1.35 -1.15 -1.55 -0.70 -1.09 -0.53 -0.92 0.04 -0.34 1.24 0.89 1.05 0.69 1.50 1.15 1.67 1.33
X
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
x1
x2μ
X F μ
V'
≈
⇒ Minimizes distance between observations and their projection
⇒ Approx Xn×p with a low rank matrix k < p ‖A‖22 = tr(AA>) :
arg minµ
{‖X − µ‖2
2 : rank (µ) ≤ k}
SVD X : µPCA = Un×kDk×kV′
p×k
= Fn×kV′
p×k
F = UD PC - scores
V principal axes - loadings44
![Page 65: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/65.jpg)
PCA reconstruction
-2.00 -2.74 NA -0.77 -1.11 -1.59 -0.67 -1.13 -0.22 NA 0.22 -0.52 0.67 1.46 NA 0.63 1.56 1.10 2.00 1.00
-2.16 -2.58 -0.96 -1.35 -1.15 -1.55 -0.70 -1.09 -0.53 -0.92 0.04 -0.34 1.24 0.89 1.05 0.69 1.50 1.15 1.67 1.33
X
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
x1
x2μ
X F μ
V'
≈
⇒ Minimizes distance between observations and their projection
⇒ Approx Xn×p with a low rank matrix k < p ‖A‖22 = tr(AA>) :
arg minµ
{‖X − µ‖2
2 : rank (µ) ≤ k}
SVD X : µPCA = Un×kDk×kV′
p×k
= Fn×kV′
p×k
F = UD PC - scores
V principal axes - loadings44
![Page 66: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/66.jpg)
Missing values in PCA
⇒ PCA : least squares
arg minµ
{‖Xn×p − µn×p‖2
2 : rank (µ) ≤ k}
⇒ PCA with missing values : weighted least squares
arg minµ
{‖Wn×p � (X − µ)‖2
2 : rank (µ) ≤ k}
with wij = 0 if xij is missing, wij = 1 otherwise ; � elementwise
multiplication
Many algorithms :
Gabriel & Zamir, 1979 : weighted alternating least squares (without explicit
imputation)
Kiers, 1997 : iterative PCA (with imputation)
45
![Page 67: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/67.jpg)
Iterative PCA
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
46
![Page 68: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/68.jpg)
Iterative PCA
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.002.0 1.98
Initialization ` = 0 : X 0 (mean imputation)
46
![Page 69: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/69.jpg)
Iterative PCA
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.002.0 1.98
x1 x2-1.98 -2.04-1.44 -1.560.15 -0.181.00 0.572.27 1.67
PCA on the completed data set → (U`,Λ`,D`) ;
46
![Page 70: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/70.jpg)
Iterative PCA
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.002.0 1.98
x1 x2-1.98 -2.04-1.44 -1.560.15 -0.181.00 0.572.27 1.67
Missing values imputed with the fitted matrix µ` = U`D`V `′
46
![Page 71: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/71.jpg)
Iterative PCA
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.002.0 1.98
x1 x2-1.98 -2.04-1.44 -1.560.15 -0.181.00 0.572.27 1.67
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.572.0 1.98
The new imputed dataset is X ` = W � X + (1−W )� µ`
46
![Page 72: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/72.jpg)
Iterative PCA
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.572.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.572.0 1.98
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
46
![Page 73: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/73.jpg)
Iterative PCA
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.572.0 1.98
x1 x2-2.00 -2.01-1.47 -1.520.09 -0.111.20 0.902.18 1.78
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.902.0 1.98
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
46
![Page 74: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/74.jpg)
Iterative PCA
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 NA2.0 1.98
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.002.0 1.98
x1 x2-1.98 -2.04-1.44 -1.560.15 -0.181.00 0.572.27 1.67
x1 x2-2.0 -2.01-1.5 -1.480.0 -0.011.5 0.572.0 1.98
-2 -1 0 1 2 3
-2-1
01
23
x1
x2
Steps are repeated until convergence
46
![Page 75: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/75.jpg)
Iterative PCA
x1 x2
-2.0 -2.01
-1.5 -1.48
0.0 -0.01
1.5 NA
2.0 1.98
x1 x2
-2.0 -2.01
-1.5 -1.48
0.0 -0.01
1.5 1.46
2.0 1.98
-2 -1 0 1 2 3
-2
-1
0
1
2
3
x1
x2
PCA on the completed data set → (U`,D`,V `)
Missing values imputed with the fitted matrix µ` = U`D`V `′46
![Page 76: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/76.jpg)
Iterative PCA
1. initialization ` = 0 : X 0 (mean imputation)
2. step ` :
(a) PCA on the completed data → (U`,D`,V `) ; k dim kept
(b) µPCA =∑k
q=1 dquqv′q X ` = W � X + (1−W )� µ`
3. steps of estimation and imputation are repeated
⇒ Overfitting : nb param (Un×k ,Vk×p)/obs values : k large - NA ; noisy
Regularized versions. Imputation is replaced by
(µ)λ =∑p
q=1 (dq − λ)+uqv′
q arg minµ
{‖W � (X − µ)‖2
2 + λ‖µ‖∗}
Different regularization : Hastie et.al. (2015) (softimpute), Verbank, J. & Husson (2013) ; Gavish
& Donoho (2014), J. & Wager (2015), J. & Sardy (2014), etc.
⇒ Iterative SVD algo good to impute data (matrix completion, Netflix)
⇒ Model makes sense : data = rank k signal+ noise
X = µ+ ε εijiid∼N
(0, σ2
)with µ of low rank
(Udell & Townsend, 2017)
47
![Page 77: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/77.jpg)
Iterative SVD
⇒ Imputation with FAMD for mixed data :
age weight size alcohol sex snore tobaccoNA 100 190 NA M yes no70 96 186 1-2 gl/d M NA <=1NA 104 194 No W no NA62 68 165 1-2 gl/d M no <=1
age weight size alcohol sex snore tobacco51 100 190 1-2 gl/d M yes no70 96 186 1-2 gl/d M no <=148 104 194 No W no <=162 68 165 1-2 gl/d M no <=1
51 100 190 0.2 0.7 0.1 1 0 0 1 1 0 070 96 186 0 1 0 1 0 0.8 0.2 0 1 048 104 194 1 0 0 0 1 1 0 0.1 0.8 0.162 68 165 0 1 0 1 0 1 0 0 1 0
NA 100 190 NA NA NA 1 0 0 1 1 0 070 96 186 0 1 0 1 0 NA NA 0 1 0NA 104 194 1 0 0 0 1 1 0 NA NA NA62 68 165 0 1 0 1 0 1 0 0 1 0
imputeAFDM
⇒ Multilevel imputation : hospital effect with patient nested in hospital.
(J., Husson, Robin & Balasu., 2018, Imputation of mixed data with multilevel SVD. JCGS)
package MissMDA.
48
![Page 78: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/78.jpg)
Imputation methods : conditional model
Imputation with fully conditional specification (FCS). Impute with a joint
model defined implicitely through the conditional distributions (mice).
⇒ Imputation model for each variable is a forest.
1. Initial imputation : mean imputation - random category
2. for t in 1 : T loop through iterations t
3. for j in 1 : p loop through variables j
Define currently complete data set except
X t−j = (X t
1 ,Xtj−1,X
t−1j+1 ,X
t−1p ), then X t
j is obtained by
• fitting a RF X obsj on the other variables X t
−j
• predicting Xmissj using the trained RF on X t
−j
package missForest (Stekhoven & Buhlmann, 2011)
49
![Page 79: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/79.jpg)
Mechanism
M = (M1, . . . ,Md) : indicator of missing values in X = (X1, . . . ,Xd).
Missing value mechanisms (Rubin, 1976)
MCAR ∀φ, ∀m, x, gφ(m|x) = gφ(m)
MAR ∀φ, ∀i ,∀x′, o(x′,mi ) = o(xi ,mi )⇒ gφ(mi |x′) = gφ(mi |xi )(e.g. gφ((0, 0, 1, 0) | (3, 2, 4, 8)) = gφ((0, 0, 1, 0) | (3, 2, 7, 8)))
MNAR Not MAR
→ useful for likelihoods
Missing value mechanisms – variable level
MCAR M |= XMAR (bis) ∀S ⊂ {1, . . . , d}, (Mj)j∈S |= (Xj)j∈S | (Xk)k∈Sc
MNAR Not MAR
→ useful for our results
50
![Page 80: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/80.jpg)
Parametric estimation
Let X ∼ fθ? .
Observed log-likelihood `obs(θ) =n∑
i=1
log
∫fθ(x)dδo(·,mi )=o(xi ,mi )(x).
(inspired by Seaman 2013)
Example
X1,X2 ∼ fθ(x1)gθ(x2|x1)
M1,2, . . . ,Mr ,2 = 1
`obs(θ) =r∑
i=1
log fθ(x1) +n∑
i=r+1
log fθ(x1)gθ(x2|x1).
51
![Page 81: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/81.jpg)
Parametric estimation
Let X ∼ fθ? .
Observed log-likelihood `obs(θ) =n∑
i=1
log
∫fθ(x)dδo(·,mi )=o(xi ,mi )(x).
Full log-likelihood `full(θ, φ) =n∑
i=1
log
∫fθ(x)gφ(mi |x)dδo(·,mi )=o(xi ,mi )(x).
Theorem (Theorem 7.1 in Rubin 1976)
θ can be infered from `obs, assuming MAR.
51
![Page 82: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/82.jpg)
Ignorable mechanism
Full log-likelihood :
`full(θ) =n∑
i=1
log
∫fθ(x)gφ(mi |x)dδo(·,mi )=o(xi ,mi )(x).
Observed log-likelihood :
`obs(θ) =n∑
i=1
log
∫fθ(x)dδo(·,mi )=o(xi ,mi )(x).
Assuming MAR,
`full(θ, φ) =n∑
i=1
log
∫fθ(x)gφ(mi |xi )dδo(·,mi )=o(xi ,mi )(x)
= `obs(θ) +n∑
i=1
log gφ(mi |xi ).
52
![Page 83: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/83.jpg)
Parametric estimation
Let X ∼ fθ? .
Observed log-likelihood `obs(θ) =n∑
i=1
log
∫fθ(x)dδo(·,mi )=o(xi ,mi )(x).
EM algorithm (Dempster, 1977)
Starting from an initial parameter θ(0), the algorithm alternates the two
following steps,
(E-step) Q(θ|θ(t)) =n∑
i=1
∫(log fθ(x))fθ(t) (x)dδo(·,mi )=o(xi ,mi )(x).
(M-step) θ(t+1) ∈ argmaxθ∈Θ
Q(θ|θ(t)).
The likelihood is guaranteed to increase.
53
![Page 84: On the consistency of supervised learning ... - Chaire DAMIchaire-dami.fr/files/2019/04/E.Cornet-Chaire_DAMI.pdf · Variable Percentage variable null.data na.data nr.data nf.data](https://reader034.vdocuments.site/reader034/viewer/2022042114/5e90cd1285741531153bce24/html5/thumbnails/84.jpg)
Missing values
X = X� (1−M) + NA�M takes value in R ∪ {NA}
The (unobserved) complete sample Dn = (Xi ,Mi ,Yi )1≤i≤n ∼ (X,M,Y )
dn =
2 3 1 0 0 0 1 0 15
1 0 3 5 0 1 0 0 13
9 4 2 5 0 0 0 1 18
7 6 3 2 0 0 1 1 10
,The observed training set Dn,train = (Xi ,Yi )1≤i≤n
dn =
2 3 NA 0 15
1 NA 3 5 13
9 4 2 NA 18
7 6 NA NA 10
.
54