rejectinferencein credit scoring - github pages › assets › publications › ... ·...
TRANSCRIPT
Reject Inference in Credit Scoring
Adrien Ehrhardt
Christophe Biernacki, Vincent Vandewalle,Philippe Heinrich, Sébastien Beben
Crédit Agricole Consumer FinanceInria Lille - Nord-Europe
Journées de Statistique 2017 - Avignon
29 mai 2017
Table of contents
1 IntroductionData generation processAccept / Reject loan applicantsCredit Scoring in practice
2 Reject Inference methodsIntroductory exampleMaximum likelihood estimationWhat is at stake?A possible reinterpretationExperimental results
3 Conclusion
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 2 / 22
Introduction
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 3 / 22
Introduction: Data generation process
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 4 / 22
Introduction: Accept / Reject loan applicants
X : random vector of a client’scharacteristicsY ∈ {0, 1} : repayment performanceZ ∈ {f, nf} : v. a. de financement
n financed clients (Z = f)m not financed clients (Z = nf)x : observed features of clientsy : clients’ repayment
x =
(x f
xnf
); y =
(y f
ynf
)
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 5 / 22
Introduction: Credit Scoring in practice
“Classical” logistic regression:
∃ θ ∈ Rd+1 s.t. ∀ x , ln(pθ(1|x)
pθ(0|x)
)= θ · x
Parameter estimation:
`(θ; x , y)︸ ︷︷ ︸completelikelihood
=n∑
i=1
ln(pθ(yi |xi )) +n+m∑i=n+1
ln(pθ(yi |xi ))
= `(θ; x f, y f)︸ ︷︷ ︸observedlikelihood
+ `(θ; xnf, ynf)︸ ︷︷ ︸unknown
Sample selection problems
Why and how use xnf?What are the consequences of using “only” (x f, y f)?
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 6 / 22
Introduction: Credit Scoring in practice
“Classical” logistic regression:
∃ θ ∈ Rd+1 s.t. ∀ x , ln(pθ(1|x)
pθ(0|x)
)= θ · x
Parameter estimation:
`(θ; x , y)︸ ︷︷ ︸completelikelihood
=n∑
i=1
ln(pθ(yi |xi )) +n+m∑i=n+1
ln(pθ(yi |xi ))
= `(θ; x f, y f)︸ ︷︷ ︸observedlikelihood
+ `(θ; xnf, ynf)︸ ︷︷ ︸unknown
Sample selection problems
Why and how use xnf?What are the consequences of using “only” (x f, y f)?
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 6 / 22
Reject Inference methods
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 7 / 22
Reject Inference methods: Fuzzy Augmentation I
Fuzzy Augmentation can be found, among others, in [Nguyen, 2016].
y f
y nf
y1...ynNA...
NA
x f
xnf
x11 · · · xd1... ... ...x1n · · · xdnx1n+1 · · · xdn+1... ... ...
x1n+m · · · xdn+m
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 8 / 22
Reject Inference methods: Fuzzy Augmentation II
Step 1: Discard xnf and estimate θf = argmaxθ `(θ; x f, y f).
y f
y nf
y1...ynNA...
NA
x f
xnf
x11 · · · xd1... ... ...x1n · · · xdnx1n+1 · · · xdn+1... ... ...
x1n+m · · · xdn+m
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 9 / 22
Reject Inference methods: Fuzzy Augmentation III
Step 2: Impute y nf with their estimation given by θf.
y f
y nf
y1...yn
pθf(Yn+1 = 1|xn+1)...
pθf(Yn+m = 1|xn+m)
x f
xnf
x11 · · · xd1... ... ...x1n · · · xdnx1n+1 · · · xdn+1... ... ...
x1n+m · · · xdn+m
Step 3: estimate θfuzzy = argmaxθ `(θ; x , y f, y nf).
Problem : θfuzzy = θf.
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 10 / 22
Reject Inference methods: Maximum likelihood estimation
“Classical” estimation in the Credit Scoring fieldθf = argmax
θ`(θ; x f, y f).
“Oracle” estimation knowing y nf
θ = argmaxθ
`(θ; x , y).
Asymptotics ?
Model sp
aceΘ
θf opt
θopt
θ
θfEstimatio
n
bias+va
riance
Estimatio
n
bias+va
riance
ptrue(y |x)
Model bias
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 11 / 22
Reject Inference methods: Maximum likelihood estimation
“Classical” estimation in the Credit Scoring fieldθf = argmax
θ`(θ; x f, y f).
“Oracle” estimation knowing y nf
θ = argmaxθ
`(θ; x , y).
Asymptotics ?
Model sp
aceΘ
θf opt
θopt
θ
θfEstimatio
n
bias+va
riance
Estimatio
n
bias+va
riance
ptrue(y |x)
Model bias
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 11 / 22
Reject Inference methods: Maximum likelihood estimation
“Classical” estimation in the Credit Scoring fieldθf = argmax
θ`(θ; x f, y f).
“Oracle” estimation knowing y nf
θ = argmaxθ
`(θ; x , y).
Asymptotics ?
Model sp
aceΘ
θf opt
θopt
θ
θfEstimatio
n
bias+va
riance
Estimatio
n
bias+va
riance
ptrue(y |x)
Model bias
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 11 / 22
Reject Inference methods: What is at stake?
Estimators :
1 “Oracle”:√n + m(θ − θopt)
L−−−−−→n,m→∞
Nd+1(0,Σθopt)
2 Current methodology:√n(θf − θf
opt)L−−−→
n→∞Nd+1(0,Σf
θfopt)
Question 1 : asymptotics of the estimators
(Q1) θopt?= θf
opt
(Q2) Σθopt?= Σf
θfopt
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 12 / 22
Reject Inference methods: What is at stake?
Estimators :
1 “Oracle”:√n + m(θ − θopt)
L−−−−−→n,m→∞
Nd+1(0,Σθopt)
2 Current methodology:√n(θf − θf
opt)L−−−→
n→∞Nd+1(0,Σf
θfopt)
Question 1 : asymptotics of the estimators
(Q1) θopt?= θf
opt
(Q2) Σθopt?= Σf
θfopt
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 12 / 22
Reject Inference methods: What is at stake?
Estimators :
1 “Oracle”:√n + m(θ − θopt)
L−−−−−→n,m→∞
Nd+1(0,Σθopt)
2 Current methodology:√n(θf − θf
opt)L−−−→
n→∞Nd+1(0,Σf
θfopt)
Question 1 : asymptotics of the estimators
(Q1) θopt?= θf
opt
(Q2) Σθopt?= Σf
θfopt
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 12 / 22
Reject Inference methods: What is at stake?
Estimators :
1 “Oracle”:√n + m(θ − θopt)
L−−−−−→n,m→∞
Nd+1(0,Σθopt)
2 Current methodology:√n(θf − θf
opt)L−−−→
n→∞Nd+1(0,Σf
θfopt)
Question 1 : asymptotics of the estimators
(Q1) θopt?= θf
opt
(Q2) Σθopt?= Σf
θfopt
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 12 / 22
Reject Inference methods: What is at stake?
Estimators :
1 “Oracle”:√n + m(θ − θopt)
L−−−−−→n,m→∞
Nd+1(0,Σθopt)
2 Current methodology:√n(θf − θf
opt)L−−−→
n→∞Nd+1(0,Σf
θfopt)
Question 1 : asymptotics of the estimators
(Q1) θopt?= θf
opt
(Q2) Σθopt?= Σf
θfopt
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 12 / 22
Reject Inference methods: Missingness mechanism
MAR : ∀ x , y , z , ptrue(z |x , y) = ptrue(z |x)→ Acceptance is determined by the score : Z = 1{θ′X>cut}.MNAR : ∃ x , y , z , ptrue(z |x , y) 6= ptrue(z |x)→ Operators’ “feeling” X c influence the acceptance.
Y X
X c
Z
Figure: Dependencies between random variables Y , X c, X and Z
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 13 / 22
Reject Inference methods: Model specification
Well-specified model : ∃ θtrue, ptrue(y |x) = pθtrue(y |x).→ With real data ⇒ hypothesis unlikely to be true.Misspecified model : θopt is the “best” in the Θ family.→ Logistic regression commonly used for its robustness tomisspecification.
pθ(y |x)ptrue(z |x , y)
MAR MNAR
Well specified θfopt = θopt
Σfθfopt
6= Σθopt θfopt 6= θopt
Misspecified θfopt 6= θopt Σf
θfopt
6= Σθopt
Σfθfopt
6= Σθopt
Table: (Q1) and (Q2) w.r.t. model specification and missingness mechanism
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 14 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ,
Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ,Model acceptance/rejection process (i.e. pγ(z |x , y)),
Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ,Model acceptance/rejection process (i.e. pγ(z |x , y)),Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ,Model acceptance/rejection process (i.e. pγ(z |x , y)),Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxθα,βα,γα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ logistic regression,Model acceptance/rejection process (i.e. pγ(z |x , y)),Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: How to use xnf?
Question 2: How to construct a better estimator than θf?
Scope for action:Change model space Θ logistic regression,Model acceptance/rejection process (i.e. pγ(z |x , y))γ cannot be estimated,Use xnf.
Natural way to achieve all three: generative approachpα(x , y , z) = pβα(x)pθα(y |x)pγα(z |x , y).
( θα , βα, γα) = argmaxα
`(α; x , y f) = argmaxθα,βα,γα
n∑i=1
ln(pθα(yi |xi ))
+n+m∑i=1
ln(pβα(xi ))
(+
n∑i=1
ln(pγα(zi |xi , yi ))
).
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 15 / 22
Reject Inference methods: A possible reinterpretation
Reclassification1 :
(θCEM, ynf) = argmaxθ,ynf
`(θ; x , y f, ynf) where yi = argmaxyi
pθf(yi |xi ).
Problem: inconsistent estimator.
1[Soulié and Viennet, 2007, Banasik and Crook, 2007, Guizani et al., 2013]Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 16 / 22
Reject Inference methods: A possible reinterpretation
Augmentation2: MAR / misspecified model.
`Aug(θ; x f, y f) =n∑
i=1
1ptrue(f|xi )
ln(pθ(yi |xi )).
Problem: estimation of ptrue(f|xi ).
Parcelling 3:
`(θ; x , y f, ynf) where yi =
{1 w.p. αipθf(1|xi , f)0 w.p. 1− αipθf(1|xi , f)
.
Problem: MNAR assumptions hidden in ynf (αi ) impossible to test.2[Soulié and Viennet, 2007, Banasik and Crook, 2007, Guizani et al., 2013,
Nguyen, 2016]3[Soulié and Viennet, 2007, Banasik and Crook, 2007, Guizani et al., 2013]
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 17 / 22
Reject Inference methods: Experimental results
Portfolio from a consumer electronics distributor partner:
Approximately 250,000 applications
Approximately 10 discrete (or discretized) features (with interactions):Socio-professional category(-ies)Amount of rent
Number of childrenYears in current job position
Approximately 3 % of default
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 18 / 22
Conclusion
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 19 / 22
Conclusion
Fuzzy Augmentation, Reclassification (and Twins) were proveduseless.
Augmentation and Parcelling seem legitimate depending on modelspecification and missingness mechanism but difficult/impossible toset forth in practice.
Recommendation: do not practice Reject Inference since highrisk/low return of all tested Reject Inference techniques.
For the most part, Reject Inference had previously been tackled onlyexperimentally.
Research paper in progress.
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 20 / 22
Thank you for your attention!
Any questions?
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 21 / 22
Banasik, J. and Crook, J. (2007).Reject inference, augmentation, and sample selection.European Journal of Operational Research, 183(3):1582–1594.
Guizani, A., Souissi, B., Ammou, S. B., and Saporta, G. (2013).Une comparaison de quatre techniques d’inférence des refusés dans leprocessus d’octroi de crédit.In 45 èmes Journées de statistique.
Nguyen, H. T. (2016).Reject inference in application scorecards: evidence from France.Technical report, University of Paris West-Nanterre la Défense,EconomiX.
Soulié, F. F. and Viennet, E. (2007).Le traitement des refusés dans le risque crédit.Revue des Nouvelles Technologies de l’Information, Data Mining etApprentissage Statistique : application en assurance, banque etmarketing, RNTI-A-1:22–44.
Adrien Ehrhardt (CACF - Inria) Reject Inference 29 mai 2017 22 / 22