robust estimation of a mean [6pt] in a multivariate ... · robust estimation of a mean in a...
TRANSCRIPT
![Page 1: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/1.jpg)
Robust estimation of a mean
in a multivariate Gaussian model: Part 1
Frejus, December 17, 2018
Arnak S. Dalalyan
ENSAE ParisTech / CREST
![Page 2: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/2.jpg)
1. Various models of contamination
![Page 3: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/3.jpg)
3
General notation
We first introduce the notation that are common to all the models ofcontamination considered in this talk.
Number of observations : n.
Dimension of the unknown parameter µ∗: p.
Observations (X1, . . . ,Xn) ∼ P n.
Number of outliers (possibly random): s ∈ {1, . . . , n}.
Set of outliers: S ⊂ {1, . . . , n}.
Proportion of outliers: ε = E[s/n] = E[|S|/n].
Setting (informal)
Among the n observations X1, . . . ,Xn, there is a small number s ofoutliers. If we remove the outliers, all the other Xi’s are iid drawn froma reference distribution Pµ∗ .
Dalalyan, A.S. Dec 17, 2018 3
![Page 4: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/4.jpg)
4
Gaussian model with unknown mean
Assumption (model for inliers)
Throughout this presentation, we assume that the reference distributionPµ∗ is p-variate Gaussian Np(µ∗, Ip). The goal is to estimate theparameter µ∗ ∈ Rp.
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.60.5
1
1.5
inliersoutlierscontour plot
n = 30, s = 5, µ∗ = [1, 1]>
Dalalyan, A.S. Dec 17, 2018 4
![Page 5: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/5.jpg)
4
Gaussian model with unknown mean
Assumption (model for inliers)
Throughout this presentation, we assume that the reference distributionPµ∗ is p-variate Gaussian Np(µ∗, Ip). The goal is to estimate theparameter µ∗ ∈ Rp.
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.60.5
1
1.5
observationscontour plot
n = 30, s = 5, µ∗ = [1, 1]>
Dalalyan, A.S. Dec 17, 2018 4
![Page 6: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/6.jpg)
5
Huber’s contamination
Assumption (HC model for outliers)
There are unobserved iid random variables Z1, . . . , Zn ∼ B(ε) and adistribution Q, such that
L (Xi|Zi = 0) = Np(µ∗, Ip), L (Xi|Zi = 1) = Q,
the observations Xi corresponding to different i’s are independent.This is equivalent to
P n ={
(1− ε)Np(µ∗, Ip) + εQ}⊗n
.
In this model,
S = {i : Zi = 1}︸ ︷︷ ︸set of outliers
and s ∼ B(n, ε)︸ ︷︷ ︸nb of outliers
are both random.
Dalalyan, A.S. Dec 17, 2018 5
![Page 7: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/7.jpg)
5
Huber’s contamination
Assumption (HC model for outliers)
There are unobserved iid random variables Z1, . . . , Zn ∼ B(ε) and adistribution Q, such that
L (Xi|Zi = 0) = Np(µ∗, Ip), L (Xi|Zi = 1) = Q,
the observations Xi corresponding to different i’s are independent.This is equivalent to
P n ={
(1− ε)Np(µ∗, Ip) + εQ}⊗n
.
In this model,
S = {i : Zi = 1}︸ ︷︷ ︸set of outliers
and s ∼ B(n, ε)︸ ︷︷ ︸nb of outliers
are both random.
i Zi Xi ∼
1 0 Np(µ∗, Ip)
2 0 Np(µ∗, Ip)
3 1 Q4 0 Np(µ
∗, Ip)5 1 Q6 0 Np(µ
∗, Ip)7 0 Np(µ
∗, Ip)
.
.
....
.
.
.30 0 Np(µ
∗, Ip)
s = 5 ∼B(30, 0.2)
Dalalyan, A.S. Dec 17, 2018 5
![Page 8: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/8.jpg)
5
Huber’s contamination
Assumption (HC model for outliers)
There are unobserved iid random variables Z1, . . . , Zn ∼ B(ε) and adistribution Q, such that
L (Xi|Zi = 0) = Np(µ∗, Ip), L (Xi|Zi = 1) = Q,
the observations Xi corresponding to different i’s are independent.This is equivalent to
P n ={
(1− ε)Np(µ∗, Ip) + εQ}⊗n
.
In this model,
S = {i : Zi = 1}︸ ︷︷ ︸set of outliers
and s ∼ B(n, ε)︸ ︷︷ ︸nb of outliers
are both random.
i Zi Xi ∼
1 0 Np(µ∗, Ip)
2 1 Q3 0 Np(µ
∗, Ip)4 0 Np(µ
∗, Ip)5 0 Np(µ
∗, Ip)6 0 Np(µ
∗, Ip)7 1 Q
.
.
....
.
.
.30 0 Np(µ
∗, Ip)
s = 6 ∼B(30, 0.2)
Dalalyan, A.S. Dec 17, 2018 5
![Page 9: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/9.jpg)
5
Huber’s contamination
Assumption (HC model for outliers)
There are unobserved iid random variables Z1, . . . , Zn ∼ B(ε) and adistribution Q, such that
L (Xi|Zi = 0) = Np(µ∗, Ip), L (Xi|Zi = 1) = Q,
the observations Xi corresponding to different i’s are independent.This is equivalent to
P n ={
(1− ε)Np(µ∗, Ip) + εQ}⊗n
.
In this model,
S = {i : Zi = 1}︸ ︷︷ ︸set of outliers
and s ∼ B(n, ε)︸ ︷︷ ︸nb of outliers
are both random.
We write
Pn ∈MHCn (p, ε,µ∗).
for the model of Huber’scontamination.
Dalalyan, A.S. Dec 17, 2018 5
![Page 10: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/10.jpg)
6
Huber’s deterministic contamination
Assumption (HDC model for outliers)
There is a set S ⊂ {1, . . . , n} of cardinality s = [nε] and a distributionQ, such that
{Xi : i ∈ Sc} iid∼ Np(µ∗, Ip) ⊥⊥ {Xi : i ∈ S} iid∼ Q.
Similar to HC: the outliers are iid.
Different from HC: the set of outliers is determenistic.
Remark The number of outliers s should be smaller than n/2,otherwise Q would be the reference distribution and Np(µ∗, Ip) thecontamination.
We write Pn ∈MHDCn (p, ε,µ∗).
Dalalyan, A.S. Dec 17, 2018 6
![Page 11: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/11.jpg)
7
Parameter contamination
Assumption (PC model for outliers)
There is a set S ⊂ {1, . . . , n} of cardinality s = [nε] and a collection ofvectors {µi : i ∈ S}, such that
{Xi : i ∈ Sc} iid∼ Np(µ∗, Ip) ⊥⊥ {Xi : i ∈ S} ∼⊗i∈SNp(µi, Ip).
Similar to HC & HDC: the outliers are independent.
Different from HC & HDC: the outliers might have differentdistributions.
We write Pn ∈MPCn (p, ε,µ∗).
Dalalyan, A.S. Dec 17, 2018 7
![Page 12: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/12.jpg)
8
Adversarial contamination
Assumption (AC model for outliers)
For a sequence Y iiid∼ Np(µ∗, Ip), i = 1, . . . , n, and a random set
S ⊂ {1, . . . , n} of cardinality s = [nε] we have
Xi = Y i, ∀i ∈ Sc.
The set S is not independent of {Y i : i = 1, . . . , n}.
The observations {Xi : i ∈ S} may have arbitrary dependencestructure.
We write Pn ∈MACn (p, ε,µ∗).
Dalalyan, A.S. Dec 17, 2018 8
![Page 13: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/13.jpg)
9
Relation between the models
MHCn (p, ε,µ∗)
MHDCn (p, 2ε,µ∗)
MPCn (p, 2ε,µ∗)
MACn (p, 2ε,µ∗)
Dalalyan, A.S. Dec 17, 2018 9
![Page 14: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/14.jpg)
2. Problem formulation and overview of results
![Page 15: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/15.jpg)
11
Historical approachBreakdown point
Assume the unknown parameter µ∗ is in Rp.
Let µ̂ be an estimator of µ∗. Thus,
µ̂ :
∞⋃n=1
Xn → Rp.
The breakdown point ε∗n of µ̂ is defined by
ε∗n =1
nmin
{s ∈ {1, . . . , n} : sup
y1,...,ys
‖µ̂(x1:(n−s),y1:s)‖ = +∞}.
Drawbacks:
does not take into account the impact of “mild” outliers,meaningless if the parameter space is bounded,does not depend on the norm under consideration,...
Dalalyan, A.S. Dec 17, 2018 11
![Page 16: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/16.jpg)
12
Minimax approachIn expectation
A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.
Worst-case risk of an estimator µ̂n:
R?n,p,ε(µ̂n) = sup
µ∗sup
Pn∈M?n(p,ε,µ∗)
EX∼Pn [‖µ̂n(X)− µ∗‖22].
Here,M?n(p, ε,µ∗) is one of the 4 models of contamination
considered in previous slides.For instance, RHC
n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.
Minimax risk:R?
n,p,ε = infµ̂n
R?n,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 12
![Page 17: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/17.jpg)
12
Minimax approachIn expectation
A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.
Worst-case risk of an estimator µ̂n:
RHCn,p,ε(µ̂n) = sup
µ∗sup
Pn∈MHCn (p,ε,µ∗)
EX∼Pn[‖µ̂n(X)− µ∗‖22].
Here,M?n(p, ε,µ∗) is one of the 4 models of contamination
considered in previous slides.For instance, RHC
n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.
Minimax risk:RHC
n,p,ε = infµ̂n
RHCn,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 12
![Page 18: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/18.jpg)
12
Minimax approachIn expectation
A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.
Worst-case risk of an estimator µ̂n:
RHDCn,p,ε(µ̂n) = sup
µ∗sup
Pn∈MHDCn (p,ε,µ∗)
EX∼Pn[‖µ̂n(X)− µ∗‖22].
Here,M?n(p, ε,µ∗) is one of the 4 models of contamination
considered in previous slides.For instance, RHC
n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.
Minimax risk:RHDC
n,p,ε = infµ̂n
RHDCn,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 12
![Page 19: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/19.jpg)
12
Minimax approachIn expectation
A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.
Worst-case risk of an estimator µ̂n:
RPCn,p,ε(µ̂n) = sup
µ∗sup
Pn∈MPCn (p,ε,µ∗)
EX∼Pn[‖µ̂n(X)− µ∗‖22].
Here,M?n(p, ε,µ∗) is one of the 4 models of contamination
considered in previous slides.For instance, RHC
n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.
Minimax risk:RPC
n,p,ε = infµ̂n
RPCn,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 12
![Page 20: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/20.jpg)
12
Minimax approachIn expectation
A more informative way of quantifying the robustness is theevaluation of the worst-case risk and its comparison to theminimax risk.
Worst-case risk of an estimator µ̂n:
RACn,p,ε(µ̂n) = sup
µ∗sup
Pn∈MACn (p,ε,µ∗)
EX∼Pn[‖µ̂n(X)− µ∗‖22].
Here,M?n(p, ε,µ∗) is one of the 4 models of contamination
considered in previous slides.For instance, RHC
n,p,ε(µ̂n) is the minimax risk for Huber’scontamination model.
Minimax risk:RAC
n,p,ε = infµ̂n
RACn,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 12
![Page 21: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/21.jpg)
13
Minimax approachIn deviation
Most results in the literature provide bounds on the deviation, notfor the expectation.
Fix a confidence level δ ∈ (0, 1).
Worst-case deviation of an estimator µ̂n: r?n,p,ε(µ̂n) is solution to
minimize r
subject to PX∼Pn
(‖µ̂n(X)− µ∗‖22 > r
)≤ δ
∀µ∗ ∈ Rp, ∀P n ∈M?n(p, ε,µ∗).
Clearly, r?n,p,ε(µ̂n) depends on δ, but we will not be interested inthis dependence.
Minimax risk:r?n,p,ε = inf
µ̂n
r?n,p,ε(µ̂n).
Tchebychev’s inequality yields δr?n,p,ε(µ̂n) ≤ R?n,p,ε(µ̂n).
Dalalyan, A.S. Dec 17, 2018 13
![Page 22: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/22.jpg)
14
Common robust estimators of the mean
The most common robust estimators of the mean are perhaps thecoordinatewise median, the geometric median and the Huber’sestimator.
All these estimators can be defined as an M -estimator:
µ̂n ∈ arg minµ∈Rp
n∑i=1
Ψ(Xi − µ)
with
Ψ(x) =
‖x‖1, coordinatewise median,‖x‖2, geometric median,‖x‖22
2 ∧ λ(‖x‖2 − 0.5λ), Huber’s estimator.
In all the three cases, the function Ψ is convex and the estimatoris computable in polynomial time.
Dalalyan, A.S. Dec 17, 2018 14
![Page 23: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/23.jpg)
15
Overview of the resultsMinimax rates in deviation
Dalalyan, A.S. Dec 17, 2018 15
![Page 24: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/24.jpg)
15
Overview of the resultsMinimax rates in deviation
Dalalyan, A.S. Dec 17, 2018 15
![Page 25: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/25.jpg)
15
Overview of the resultsMinimax rates in deviation
Dalalyan, A.S. Dec 17, 2018 15
![Page 26: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/26.jpg)
16
Overview of the resultsTractable estimators
We will present three tractable estimators that improve on thecoordinatewise median.
1 The ellipsoid method (Diakonikolas et al., 2016).
2 The spectral method (Lai et al., 2016).
3 The iterative soft thresholding (Collier and Dalalyan, 2017).
Dalalyan, A.S. Dec 17, 2018 16
![Page 27: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/27.jpg)
3. The minimax rate
![Page 28: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/28.jpg)
18
Minimax lower bound
Theorem 1 (Chen et al., 2015)
There is a constant c > 0 such that for every ε ∈ [0, 1] and everyδ ∈ (0, 1/2), it holds that
rHCn,p,ε ≥ c
(p
n+ ε2
).
Some remarks
By Tchebychev’s inequality, pn + ε2 is also a lower bound for the
minimax risk in expectation.
By inclusion, pn + ε2 is also a lower bound for the minimax risk in
models HDC and AC.
The same lower bound pn + ε2 holds true for the model PC.
Dalalyan, A.S. Dec 17, 2018 18
![Page 29: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/29.jpg)
19
Proof of the lower bound 1
1 From the classic parametric minimax theory: rHCn,p,ε &
pn .
2 Thus, we need only to show that rHCn,p,ε & ε
2.
3 Main steps of the proof:
Reduction to dimension 1: rHCn,p,ε ≥ rHC
n,1,ε.Construct a probability density function fε such that
f⊗nε ∈MHCn (1, ε, 0)
f⊗nε ∈MHCn (1, ε,∆ε)
with ∆ε � ε.
Parameter values µ∗ = 0 and µ∗ = ∆ε are indistinguishablefrom the observations X1, . . . ,Xn ∼ f⊗nε .
Therefore rHCn,p,ε & ‖∆ε − 0‖22 � ε2.
Dalalyan, A.S. Dec 17, 2018 19
![Page 30: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/30.jpg)
20
Proof of the lower bound 2
For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.
Dalalyan, A.S. Dec 17, 2018 20
![Page 31: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/31.jpg)
20
Proof of the lower bound 2
For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.
We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).
Dalalyan, A.S. Dec 17, 2018 20
![Page 32: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/32.jpg)
20
Proof of the lower bound 2
For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.
We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).
Then, f∆ = f◦∆/S∆ is a pdf.
Dalalyan, A.S. Dec 17, 2018 20
![Page 33: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/33.jpg)
20
Proof of the lower bound 2
For a ∆ > 0, define f◦∆ = ϕ0 ∨ ϕ∆.
We have S∆ =∫f◦∆(x) dx = 1 + a∆ +O(∆2).
Then, f∆ = f◦∆/S∆ is a pdf.
We choose ∆ε so that 1/S∆ε= 1− ε
and set fε = f∆ε .
Dalalyan, A.S. Dec 17, 2018 20
![Page 34: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/34.jpg)
21
Proof of the lower bound 3
fε = (1− ε)(ϕ0 ∨ ϕ∆ε)
ll ll
(1− ε)ϕ0
+ +
εq
fε =(1− ε)ϕ0 + εq
(1− ε)ϕ∆ε+ εq′
.
Dalalyan, A.S. Dec 17, 2018 21
![Page 35: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/35.jpg)
21
Proof of the lower bound 3
fε = (1− ε)(ϕ0 ∨ ϕ∆ε)
ll ll
(1− ε)ϕ0
+ +
εq
fε =(1− ε)ϕ0 + εq
(1− ε)ϕ∆ε+ εq′
.
Dalalyan, A.S. Dec 17, 2018 21
![Page 36: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/36.jpg)
21
Proof of the lower bound 3
fε = (1− ε)(ϕ0 ∨ ϕ∆ε)
ll ll
(1− ε)ϕ0
+ +
εq
fε =(1− ε)ϕ0 + εq
(1− ε)ϕ∆ε+ εq′
.
Dalalyan, A.S. Dec 17, 2018 21
![Page 37: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/37.jpg)
21
Proof of the lower bound 3
fε = (1− ε)(ϕ0 ∨ ϕ∆ε)
ll ll
(1− ε)ϕ0
+ +
εq
fε =(1− ε)ϕ0 + εq
(1− ε)ϕ∆ε+ εq′
.
Dalalyan, A.S. Dec 17, 2018 21
![Page 38: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/38.jpg)
22
Minimax upper bound
Theorem 2 (Chen et al., 2015)There are two constants C1, C2 > 0 such that
for every ε ≤ 1/5for every p ≤ C1nfor every δ ≥ e−C1n,
it holds that
rHCn,p,ε ≤ C2
(p
n+ ε2 +
log 1/δ
n
).
Some remarks:
The upper bound is attained by Tukey’s median.
The condition ε ≤ 1/5 can be replaced by ε ≤ 1/3− c′, with anarbitrarily small c′ > 0.
The estimator does not rely on the knowledge of ε.
Dalalyan, A.S. Dec 17, 2018 22
![Page 39: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/39.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
Dalalyan, A.S. Dec 17, 2018 23
![Page 40: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/40.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
depth of µ in X1:10?
Dalalyan, A.S. Dec 17, 2018 23
![Page 41: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/41.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
depth of µ in X1:10?
Dalalyan, A.S. Dec 17, 2018 23
![Page 42: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/42.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
depth of µ in theforest ?
Dalalyan, A.S. Dec 17, 2018 23
![Page 43: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/43.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
D(µ,X1:n) = 1.
Dalalyan, A.S. Dec 17, 2018 23
![Page 44: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/44.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
depth of µ in theforest ?
Dalalyan, A.S. Dec 17, 2018 23
![Page 45: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/45.jpg)
23
Tukey’s median
The upper bound is attained byTukey’s median.
Tukey’s median is any maximaizer ofTukey’s depth:
µ̂TMn ∈ arg max
µ∈Rp
D(µ, {X1:n}).
Tukey’s (halfspace) depth is
D(µ,X1:n) = minu∈S1
n∑i=1
1(u>Xi ≤ u>µ).
µ̂TMn is computationally intractable for
large p.
D(µ,X1:n) = 3.
Dalalyan, A.S. Dec 17, 2018 23
![Page 46: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/46.jpg)
24
Summary
We introduced four models of contamination by outliers:
Huber’s contaminationMHCn (p, ε,µ∗).
Huber’s deterministic contaminationMHDCn (p, ε,µ∗).
Parameter contaminationMPCn (p, ε,µ∗).
Adversarial contaminationMACn (p, ε,µ∗).
We have defined the worst case risks in expectation and indeviation, R?
n,p,ε(µ̂) and r?n,p,ε(µ̂).
We have defined the minimax risks R?n,p,ε = infµ̂R
?n,p,ε(µ̂) .
For every ε < 1/3−�, we have r?n,p,ε �pn + ε2.
This minimax rate is obtained by Tukey’s median, which is hard tocompute for large p.
QuestionWhat is the smallest rate of the worst-case risk that can be obtained byan estimator computable in poly(n, p, 1/ε) time?
Dalalyan, A.S. Dec 17, 2018 24
![Page 47: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/47.jpg)
24
Summary
We introduced four models of contamination by outliers:
Huber’s contaminationMHCn (p, ε,µ∗).
Huber’s deterministic contaminationMHDCn (p, ε,µ∗).
Parameter contaminationMPCn (p, ε,µ∗).
Adversarial contaminationMACn (p, ε,µ∗).
We have defined the worst case risks in expectation and indeviation, R?
n,p,ε(µ̂) and r?n,p,ε(µ̂).
We have defined the minimax risks r?n,p,ε = infµ̂ r?n,p,ε(µ̂) .
For every ε < 1/3−�, we have r?n,p,ε �pn + ε2.
This minimax rate is obtained by Tukey’s median, which is hard tocompute for large p.
QuestionWhat is the smallest rate of the worst-case risk that can be obtained byan estimator computable in poly(n, p, 1/ε) time?
Dalalyan, A.S. Dec 17, 2018 24
![Page 48: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/48.jpg)
24
Summary
We introduced four models of contamination by outliers:
Huber’s contaminationMHCn (p, ε,µ∗).
Huber’s deterministic contaminationMHDCn (p, ε,µ∗).
Parameter contaminationMPCn (p, ε,µ∗).
Adversarial contaminationMACn (p, ε,µ∗).
We have defined the worst case risks in expectation and indeviation, R?
n,p,ε(µ̂) and r?n,p,ε(µ̂).
We have defined the minimax risks r?n,p,ε = infµ̂ r?n,p,ε(µ̂) .
For every ε < 1/3−�, we have r?n,p,ε �pn + ε2.
This minimax rate is obtained by Tukey’s median, which is hard tocompute for large p.
QuestionWhat is the smallest rate of the worst-case risk that can be obtained byan estimator computable in poly(n, p, 1/ε) time?
Dalalyan, A.S. Dec 17, 2018 24
![Page 49: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/49.jpg)
24
Summary
We introduced four models of contamination by outliers:
Huber’s contaminationMHCn (p, ε,µ∗).
Huber’s deterministic contaminationMHDCn (p, ε,µ∗).
Parameter contaminationMPCn (p, ε,µ∗).
Adversarial contaminationMACn (p, ε,µ∗).
We have defined the worst case risks in expectation and indeviation, R?
n,p,ε(µ̂) and r?n,p,ε(µ̂).
We have defined the minimax risks r?n,p,ε = infµ̂ r?n,p,ε(µ̂) .
For every ε < 1/3−�, we have r?n,p,ε �pn + ε2.
This minimax rate is obtained by Tukey’s median, which is hard tocompute for large p.
QuestionWhat is the smallest rate of the worst-case risk that can be obtained byan estimator computable in poly(n, p, 1/ε) time?
Dalalyan, A.S. Dec 17, 2018 24
![Page 50: Robust estimation of a mean [6pt] in a multivariate ... · Robust estimation of a mean in a multivariate Gaussian model: Part 1 Frejus, December 17, 2018 Arnak S. Dalalyan ENSAE ParisTech](https://reader034.vdocuments.site/reader034/viewer/2022042802/5f3c543bb7a3993cd06c0e7d/html5/thumbnails/50.jpg)
25
References I
M. Chen, C. Gao, and Z. Ren. Robust Covariance and Scatter MatrixEstimation under Huber’s Contamination Model. ArXiv e-prints, to appear inthe Annals of Statistics, 2015.
Olivier Collier and Arnak S. Dalalyan. Minimax estimation of amultidimensional linear functional in sparse gaussian models and robustestimation of the mean. submitted 1712.05495, arXiv, December 2017.URL https://arxiv.org/abs/1712.05495.
Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Ankur Moitra,and Alistair Stewart. Robust estimators in high dimensions without thecomputational intractability. In IEEE 57th Annual Symposium onFoundations of Computer Science, FOCS 2016, USA, pages 655–664,2016.
K. A. Lai, A. B. Rao, and S. Vempala. Agnostic estimation of mean andcovariance. In 2016 IEEE 57th Annual Symposium on Foundations ofComputer Science (FOCS), pages 665–674, Oct 2016.
Dalalyan, A.S. Dec 17, 2018 25