a comparison of three learning methods to predict n20 fluxes and n leaching
DESCRIPTION
Climate Change Unit, European Commission Ispra, Italy March 18th, 2010TRANSCRIPT
A comparison of three learning methods topredict N2O fluxes and N leaching
Nathalie Villa-Vialaneix
http://www.nathalievilla.org
Institut de Mathématiques de Toulouse &
IUT de Carcassonne (Université de Perpignan)
March 18th, 2010
UC - Climate Change Unit, Ispra
1 / 25Comparison of metamodels
N
Sommaire
1 Basics about machine learning for regression problems
2 Presentation of three learning methods
3 Methodology and results [Villa-Vialaneix et al., 2010]
2 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Sommaire
1 Basics about machine learning for regression problems
2 Presentation of three learning methods
3 Methodology and results [Villa-Vialaneix et al., 2010]
3 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Regression
Consider the problem where:
we want to predict Y ∈ R from X ∈ Rd ;
a learning set of N i.i.d. observations of (X ,Y) is already available:(x1, y1), . . . , (xN , yN);
new values of X are available from which we expect to evaluate thecorresponding Y : xN+1, . . . , xM .
Example: Predict N2O fluxes from PH, climate, concentration of Nin rain, fertilization for a large number of HSMU . . .
4 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Regression
Consider the problem where:
we want to predict Y ∈ R from X ∈ Rd ;
a learning set of N i.i.d. observations of (X ,Y) is already available:(x1, y1), . . . , (xN , yN);
new values of X are available from which we expect to evaluate thecorresponding Y : xN+1, . . . , xM .
Example: Predict N2O fluxes from PH, climate, concentration of Nin rain, fertilization for a large number of HSMU . . .
4 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Two different approaches
1 Linear model: A linear relation between X and Y is supposed:
yi = βT xi + ε, i = 1, . . . ,N
β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .
2 Non parametric model: No particular relation is supposed betweenX and Y :
yi = Φ(xi) + ε, i = 1, . . . ,N
where Φ is unknown.
From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.
5 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Two different approaches
1 Linear model: A linear relation between X and Y is supposed:
yi = βT xi + ε, i = 1, . . . ,N
β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .
2 Non parametric model: No particular relation is supposed betweenX and Y :
yi = Φ(xi) + ε, i = 1, . . . ,N
where Φ is unknown.
From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.
5 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Two different approaches
1 Linear model: A linear relation between X and Y is supposed:
yi = βT xi + ε, i = 1, . . . ,N
β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .
2 Non parametric model: No particular relation is supposed betweenX and Y :
yi = Φ(xi) + ε, i = 1, . . . ,N
where Φ is unknown.
From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.
5 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Two different approaches
1 Linear model: A linear relation between X and Y is supposed:
yi = βT xi + ε, i = 1, . . . ,N
β can be estimated from the learning set by βN = (XT X)−1XT Ywhere X = (x1 . . . xN)T and Y = (y1 . . . yN)T .
2 Non parametric model: No particular relation is supposed betweenX and Y :
yi = Φ(xi) + ε, i = 1, . . . ,N
where Φ is unknown.From (xi , yi)i , we intend to define a machine, i.e., a functionΦN : Rd → R that approximates Φ.
5 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Desirable properties for a machine
The adequacy of ΦN is measure through an error function, L .Example: Mean squared error
L(y, y) = (y − y)2
Hence, ΦN should be:
accurate for the learning data set:∑N
i=1 L(yi , ΦN(xi)) has to be
small;
able to predict well new data (generalization ability):∑Mi=N+1 L(yi , Φ
N(xi)) has to be small (but here, yi are unknown!).
These two objectives are somehow contradictory and a tradeoffhas to be found (see [Vapnik, 1995] for further details).
6 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Desirable properties for a machine
The adequacy of ΦN is measure through an error function, L .Example: Mean squared error
L(y, y) = (y − y)2
Hence, ΦN should be:
accurate for the learning data set:∑N
i=1 L(yi , ΦN(xi)) has to be
small;
able to predict well new data (generalization ability):∑Mi=N+1 L(yi , Φ
N(xi)) has to be small (but here, yi are unknown!).
These two objectives are somehow contradictory and a tradeoffhas to be found (see [Vapnik, 1995] for further details).
6 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Desirable properties for a machine
The adequacy of ΦN is measure through an error function, L .Example: Mean squared error
L(y, y) = (y − y)2
Hence, ΦN should be:
accurate for the learning data set:∑N
i=1 L(yi , ΦN(xi)) has to be
small;
able to predict well new data (generalization ability):∑Mi=N+1 L(yi , Φ
N(xi)) has to be small (but here, yi are unknown!).
These two objectives are somehow contradictory and a tradeoffhas to be found (see [Vapnik, 1995] for further details).
6 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Desirable properties for a machine
The adequacy of ΦN is measure through an error function, L .Example: Mean squared error
L(y, y) = (y − y)2
Hence, ΦN should be:
accurate for the learning data set:∑N
i=1 L(yi , ΦN(xi)) has to be
small;
able to predict well new data (generalization ability):∑Mi=N+1 L(yi , Φ
N(xi)) has to be small (but here, yi are unknown!).
These two objectives are somehow contradictory and a tradeoffhas to be found (see [Vapnik, 1995] for further details).
6 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Purpose of the work
We focus on methods that are universally consistent. Thesemethods lead to the definition of machines ΦN such that:
EL(ΦN(X),Y)N→+∞−−−−−−→ L∗ = inf
Φ:Rd→RL(Φ(X),Y)
for any random pair (X ,Y).
1 multi-layer perceptrons (neural networks): [Bishop, 1995]
2 Support Vector Machines (SVM): [Boser et al., 1992]
3 random forests: [Breiman, 2001] (universal consistency is notproven in this case)
7 / 25Comparison of metamodels
N
Basics about machine learning for regression problems
Purpose of the work
We focus on methods that are universally consistent. Thesemethods lead to the definition of machines ΦN such that:
EL(ΦN(X),Y)N→+∞−−−−−−→ L∗ = inf
Φ:Rd→RL(Φ(X),Y)
for any random pair (X ,Y).
1 multi-layer perceptrons (neural networks): [Bishop, 1995]
2 Support Vector Machines (SVM): [Boser et al., 1992]
3 random forests: [Breiman, 2001] (universal consistency is notproven in this case)
7 / 25Comparison of metamodels
N
Presentation of three learning methods
Sommaire
1 Basics about machine learning for regression problems
2 Presentation of three learning methods
3 Methodology and results [Villa-Vialaneix et al., 2010]
8 / 25Comparison of metamodels
N
Presentation of three learning methods
Multilayer perceptrons (MLP)
A “one-hidden-layer perceptron” is a function of the form:
Φw : x ∈ Rd →
Q∑i=1
w(2)i G
(xT w(1)
i + w(0)i
)+ w(2)
0
where:
the w are the weights of the MLP that have to be learned from thelearning set;
G is a given activation function: typically, G(z) = 1−e−z
1+e−z ;
Q is the number of neurons on the hidden layer. It controls theflexibility of the machine. Q is a hyper-parameter that is usuallytuned during the learning process.
9 / 25Comparison of metamodels
N
Presentation of three learning methods
Symbolic representation of MLP
INP
UTS
x1
x2
. . .
xd
w(1)11
w(1)pQ
Neuron 1
Neuron Q
OU
TPU
TS
φw(x)
w(2)1
w(2)Q
+w(0)Q 10 / 25
Comparison of metamodelsN
Presentation of three learning methods
Learning MLP
Learning the weights: w are learned by a mean squared errorminimization scheme :
w∗ = arg minw
N∑i=1
L(yi ,Φw(xi)).
Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.
Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .
11 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning MLP
Learning the weights: w are learned by a mean squared errorminimization scheme penalized by a weight decay to avoidoverfitting (ensure a better generalization ability):
w∗ = arg minw
N∑i=1
L(yi ,Φw(xi))+C‖w‖2.
Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.
Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .
11 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning MLP
Learning the weights: w are learned by a mean squared errorminimization scheme penalized by a weight decay to avoidoverfitting (ensure a better generalization ability):
w∗ = arg minw
N∑i=1
L(yi ,Φw(xi))+C‖w‖2.
Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.
Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .
11 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning MLP
Learning the weights: w are learned by a mean squared errorminimization scheme penalized by a weight decay to avoidoverfitting (ensure a better generalization ability):
w∗ = arg minw
N∑i=1
L(yi ,Φw(xi))+C‖w‖2.
Problem: MSE is not quadratic in w and thus the optimization cannot be made perfectly right. Some solutions can be local minima.
Tuning of the hyper-parameters, C and Q : simple validation hasbeen used to tune first C and Q .
11 / 25Comparison of metamodels
N
Presentation of three learning methods
SVM
SVM is also a error loss with penalization minimization:1 Basic linear SVM for regression: Φ(w,b) is of the form
x → wT x + b with (w, b) solution of
arg minN∑
i=1
Lε(yi ,Φ(w,b)(xi)) + λ‖w‖2
whereλ is a regularization (hyper) parameter (to be tuned during thelearning process);Lε(y, y) = max{|y − y | − ε, 0} is an ε-insensitive loss function
See ε-insensitive loss function
2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.
Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)
Tϕ(xj).
Common kernel: Gaussian kernel
Kγ(u, v) = e−γ‖u−v‖2
is known to have good theoretical properties both for accuracy andgeneralization.
12 / 25Comparison of metamodels
N
Presentation of three learning methods
SVM
SVM is also a error loss with penalization minimization:
1 Basic linear SVM for regression
2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.
Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)
Tϕ(xj).
Common kernel: Gaussian kernel
Kγ(u, v) = e−γ‖u−v‖2
is known to have good theoretical properties both for accuracy andgeneralization.
12 / 25Comparison of metamodels
N
Presentation of three learning methods
SVM
SVM is also a error loss with penalization minimization:
1 Basic linear SVM for regression
2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)
Tϕ(xj).
Common kernel: Gaussian kernel
Kγ(u, v) = e−γ‖u−v‖2
is known to have good theoretical properties both for accuracy andgeneralization.
12 / 25Comparison of metamodels
N
Presentation of three learning methods
SVM
SVM is also a error loss with penalization minimization:
1 Basic linear SVM for regression
2 Non linear SVM for regression are the same except that a nonlinear (fixed) transformation of the inputs is previously made:ϕ(x) ∈ H is used instead of x.Kernel trick: in fact, ϕ is never explicit but used through a kernel,K : Rd × Rd → R. This kernel is used for K(xi , xj) = ϕ(xi)
Tϕ(xj).
Common kernel: Gaussian kernel
Kγ(u, v) = e−γ‖u−v‖2
is known to have good theoretical properties both for accuracy andgeneralization.
12 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning SVM
Learning (w, b): w =∑N
i=1 αiK(xi , .) and b are calculated by anexact optimization scheme (quadratic programming). The only stepthat can be time consumming is the calculation of the kernel matrix:
K(xi , xj) for i, j = 1, . . . ,N.
The resulting ΦN is known to be of the form:
ΦN(x) =N∑
i=1
αiK(xi , x) + b
where only a few αi are non zero. The corresponding xi are calledsupport vectors.
Tuning of the hyper-parameters, C = 1/λ, ε and γ: simplevalidation has been used. To limit waste of time, ε has not beentuned in our experiments but set to the default value (1) whichensured 0.5N support vectors at most.
13 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning SVM
Learning (w, b): w =∑N
i=1 αiK(xi , .) and b are calculated by anexact optimization scheme (quadratic programming). The only stepthat can be time consumming is the calculation of the kernel matrix:
K(xi , xj) for i, j = 1, . . . ,N.
The resulting ΦN is known to be of the form:
ΦN(x) =N∑
i=1
αiK(xi , x) + b
where only a few αi are non zero. The corresponding xi are calledsupport vectors.
Tuning of the hyper-parameters, C = 1/λ, ε and γ: simplevalidation has been used. To limit waste of time, ε has not beentuned in our experiments but set to the default value (1) whichensured 0.5N support vectors at most.
13 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning SVM
Learning (w, b): w =∑N
i=1 αiK(xi , .) and b are calculated by anexact optimization scheme (quadratic programming). The only stepthat can be time consumming is the calculation of the kernel matrix:
K(xi , xj) for i, j = 1, . . . ,N.
The resulting ΦN is known to be of the form:
ΦN(x) =N∑
i=1
αiK(xi , x) + b
where only a few αi are non zero. The corresponding xi are calledsupport vectors.
Tuning of the hyper-parameters, C = 1/λ, ε and γ: simplevalidation has been used. To limit waste of time, ε has not beentuned in our experiments but set to the default value (1) whichensured 0.5N support vectors at most.
13 / 25Comparison of metamodels
N
Presentation of three learning methods
From regression tree to random forest
Example of a regression tree
|SOCt < 0.095
PH < 7.815
SOCt < 0.025
FR < 130.45 clay < 0.185
SOCt < 0.025
SOCt < 0.145
FR < 108.45PH < 6.5
4.366 7.10015.010 8.975
2.685 5.257
26.26028.070 35.900 59.330
Each split is made such thatthe two induced subsets havethe greatest homogeneity pos-sible.The prediction of a final nodeis the mean of the Y value ofthe observations belonging tothis node.
14 / 25Comparison of metamodels
N
Presentation of three learning methods
Random forest
Basic principle: combination of a large number of under-efficientregression trees (the prediction is the mean prediction of all trees).
For each tree, two simplifications of the original method areperformed:
1 A given number of observations are randomly chosen among thetraining set: this subset of the training data set is called in-bagsample whereas the other observations are called out-of-bag andare used to control the error of the tree;
2 For each node of the tree, a given number of variables arerandomly chosen among all possible explanatory variables.
The best split is then calculated on the basis of these variables andof the chosen observations. The chosen observations are thesame for a given tree whereas the variables taken into accountchange for each split.
15 / 25Comparison of metamodels
N
Presentation of three learning methods
Random forest
Basic principle: combination of a large number of under-efficientregression trees (the prediction is the mean prediction of all trees).For each tree, two simplifications of the original method areperformed:
1 A given number of observations are randomly chosen among thetraining set: this subset of the training data set is called in-bagsample whereas the other observations are called out-of-bag andare used to control the error of the tree;
2 For each node of the tree, a given number of variables arerandomly chosen among all possible explanatory variables.
The best split is then calculated on the basis of these variables andof the chosen observations. The chosen observations are thesame for a given tree whereas the variables taken into accountchange for each split.
15 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning a random forest
Random forest are not very sensitive to hyper-parameters(number of observations for each tree, number of variables foreach split): the default values have been used.
The number of trees should be large enough for the mean squarederror based on out-of-sample observations to stabilize:
0 100 200 300 400 500
02
46
810
trees
Err
or
Out−of−bag (training)Test
16 / 25Comparison of metamodels
N
Presentation of three learning methods
Learning a random forest
Random forest are not very sensitive to hyper-parameters(number of observations for each tree, number of variables foreach split): the default values have been used.The number of trees should be large enough for the mean squarederror based on out-of-sample observations to stabilize:
0 100 200 300 400 500
02
46
810
trees
Err
or
Out−of−bag (training)Test
16 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Sommaire
1 Basics about machine learning for regression problems
2 Presentation of three learning methods
3 Methodology and results [Villa-Vialaneix et al., 2010]
17 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Data
5 data sets corresponding to different scenarios coming from thegeobiological simulator DNDC-EUROPE.
S1: Baseline scenario (conventional corn cropping system): 18 794HSMU
S2: like S1 without tillage: 18 830 HSMU
S3: like S1 with a limit in N input through manure spreading at 170kg/ha y: 18 800 HSMU
S4: rotation between corn (2y) and catch crop (3y): 40 536 HSMU
S5: like S1 with an additional application of fertilizer in winter:18 658 HSMU
2 outputs to estimate from the inputs:N2O fluxes;
N leaching.
18 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Data
11 input variablesN FR (N input through fertilization; kg/ha y);
N MR (N input through manure spreading; kg/ha y);
Nfix (N input from biological fixation; kg/ha y);
Nres (N input from root residue; kg/ha y);
BD (Bulk Density; g/cm3 );
SOC (Soil organic carbon in topsoil; mass fraction);
PH (Soil PH);
Clay (Ratio of soil clay content);
Rain (Annual precipitation; mm/y);
Tmean (Annual mean temperature; C);
Nr (Concentration of N in rain; ppm).
2 outputs to estimate from the inputs:N2O fluxes;
N leaching.
18 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Data
2 outputs to estimate from the inputs:
N2O fluxes;
N leaching.
18 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Methodology
For every data set, every output and every method,
1 The data set has been split into a training set and a test set (on a80%/20% basis);
2 The machine has been learned from the training set (with a fullvalidation process for the hyperparameter tuning);
3 The performances have been calculated on the basis of the testset: for the test set, predictions have been made from the inputsand compared to the true outputs.
19 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Methodology
For every data set, every output and every method,
1 The data set has been split into a training set and a test set (on a80%/20% basis);
2 The machine has been learned from the training set (with a fullvalidation process for the hyperparameter tuning);
3 The performances have been calculated on the basis of the testset: for the test set, predictions have been made from the inputsand compared to the true outputs.
19 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Methodology
For every data set, every output and every method,
1 The data set has been split into a training set and a test set (on a80%/20% basis);
2 The machine has been learned from the training set (with a fullvalidation process for the hyperparameter tuning);
3 The performances have been calculated on the basis of the testset: for the test set, predictions have been made from the inputsand compared to the true outputs.
19 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Numerical results
MLP RF SVM LMN2O Scenario 1 90.7% 92.3% 91.0% 77.1%
Scenario 2 80.3% 85.0% 82.3% 62.2%Scenario 3 85.1% 88.0% 86.7% 74.6%Scenario 4 88.6% 90.5% 86.3% 78.3%Scenario 5 80.6% 84.9% 82.7% 42.5%
N leaching Scenario 1 89.7% 93.5% 96.6% 70.3%Scenario 2 91.4% 93.1% 97.0% 67.7%Scenario 3 90.6% 90.4% 96.0% 68.6%Scenario 4 80.5% 86.9% 90.6% 42.5%Scenario 5 86.5% 92.9% 94.5% 71.1%
20 / 25Comparison of metamodelsN
Methodology and results [Villa-Vialaneix et al., 2010]
Graphical results
Random forest (S1, N2O)
●●●●●●●●●● ●●●
●
●
●●
●●
●
●
●
●
●●
●
●●
●
●
●●●
●●●
●●●
●●●●
●●
●●
● ●●●
●
●●
●
●
●
●●●●
●
●●●
●●
●●●●●●
●
●
●●●
●
●●
●●
●
●
●
●●●
●
●●●
●
●●●
●●●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●●●●●
●●●●●
●●
●
● ●●
●●●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●●●●
●
●● ●
●
●
●●●
●
●
●●●
●
●●● ●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●●
●●
●
●●
●●●
●
●●
●●●
●●
●
●
●
●
●●●
●
●
●
●●●
●
●●●●●●
●
●
●
●●●●●●
●
●
●
●
●●
●
●
● ●
●●
●●●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●●
● ●
●●●
●
●
●
●
●
●●
●●
●●● ●
●●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●●●
●●●●
●
●●
●
●●●
●●
●
●●
●
●●
●●
●
●●
●●●
●●
●●
●
●●
●●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●●
●
●●●
●
●
●
●
●●
●●●●
●●●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●●
●●
●●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●● ●
●
●
●●
●●
● ●●
●
●●
●●●●
●●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●●●●
●●●
●
●
●
●
●●
●
●●●
●
●
●
●●●●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●●
●
● ●●
●
● ●
●●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●●
●
●
●●●●
●●●
●
●
●
●●
●
●●●
●●●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●●
●●
●
●
●●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●● ●●
●
●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●●● ●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●
●●
●
●
●
●●
●●
● ●
●●
●●
●
●
●
● ●
●●●
●●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●
●●
● ●
●
●
●●
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●●●●
●
●●
●
●
●●
●
●●
●●
●
●●●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●●●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●●
●●●●
●
●●
●
●
●●
●●●●
●
●●●●
●
●●●
●●●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●●
●
●
●
●●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●●
●
● ●
●
●
●
●
●
●●●●●
●●
●
●
●
●
●●
●
●●●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●●●●
●●●
●
●●●
●
●●●●●●●
●●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●●●
●● ●●●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●●●●●●●●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
● ●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●
●●
●
●●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●●
●
●●
●
●
●
●
●
●●●
●●●
●
●
●
●●
●●●●
●
●
●
●
●● ●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●
●
●●
● ●
●
●
●
●
●●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
● ●
●
●
●
●●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●●
●●
●●●●
●
●
● ●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●●
●
●
●●
●
●●
●
●
●●
● ●
● ●
●●
●●
●
●●
●
●●
●
●
●●●●●
●
●
●
●
●●
●
●
●
●●●
●●
●
●●
●
● ●●
●●
● ●●
●
●
●
●●
●
●
●●
●
●●
●●
●
●●●
●●
●●
●
●●
●●
●
●
● ●
●●●
●
●●
●
●●●●
●●●
●
●
●●
●●
●
●●
●
●● ●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
● ●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●●
●●●●●● ●
●●
●●●●
●●●
●●●●●●●
●●●●
●
●●
● ●
●
●
●
●●●●●
●
●
●
● ●●
●● ●
●
●●
●●●●●●
●
●
●
●●●●
●●
●
●
●●●●
●
●
●
●●
●●●
● ●●
●
●
●●●
●
●
●
● ●●
●●●
●
●●●
●●
●
●●●●●
● ●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●● ●
●●●●
●
●
●●●●●●
●●●●
●●
●●
●
●
●●●●●●●●●
●●
●●
●
●
●
●●●●●
●
●●●●
●●
●●●
●
● ●●●
●
●●
●
●●●●●●●
●●
●
●●●●●
● ●●
●●●●●●●●●
●●● ●
●●●●●●
●●●●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●●●
●●●
●●●●
●●●●●●
●
●
●
●●●●
●
●●
●
●●
●●
●●
●●
●
●●●●●
●●
●
●●
●
●●●●●
●
●
●●
●●
●
●
●
●●●●
●
●
●●●●
●
●●
●●●●
●●
●●
●
● ●●
●
●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●
●
●●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●●●●●●●
● ●●●
●
●
●
●●
●●●●
●●
●
● ●
●
●
●
●●
●
●
●● ●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●●●
●
●
●
●
●●●
●
●●
●
●●
●●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●●
●●
●● ●●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●●
●●●
●●
●
●
●●●●
●
●
●
●
●●
●●
●●●●●
●
●●
●
●
●
●
●●
●
●●
●●●
●
●●●
●
●
●●
●●●
●
●●●
●
●
●
●●
●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●
●●
●●●
●●
●●●
●
●●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●●●●
●
●●
●
●
●
●●●●
●
●
● ●
●●
●
●
●
●●●
●●●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●●●●
●●●●
●
●
●●●
●●
●●
●
●
●
●●●
●
●
●●
●●
●
●
●●
●●
●●
●
●●●●●
●●●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●●
●●●
●●●
●
●
●●
●
●●●●●●●●●●●
●●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●●
●
●●●
●●●●●
●
●
●●
●●
●
●●
●●
●
●
●●●●●●●
●●
●
●●●
●●
●
●●●●
●
●●
●
●
●
●
●●●
●●●
●●
●●
●●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●●●
● ●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●
●●●●●●●
●
●●
●
●●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
● ●●
●
●●
●
●
●
●
●
●●
●
●●●●
●
●
●●
●
●
●●
●
●●●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●
●●
●●● ●
●●●
●
●●
●
●
●
●●
● ●
●
●●
●
●
●
●
●●●
●
●●● ●
●
●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●●●●●
●
●●●●
●
●●
●●
●
●●●●
●●
●
●●●
●
●●●●
●
●
●
●●
●
●
●●
●
● ●
●●
●
●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●●
●●
●
● ●●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●● ●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●●
●
●● ●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●●
●
●
●
●
●●
●
●
●●
● ●●
●
●
●
● ●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●
●
●●
●
●
●●●●
●
●
●●
●
●●●
●
●
●●
●●●
●●●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●● ●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
● ●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●● ●
●
●
●●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●● ●
●●●
●● ●●●
●●
●●●
●●
●● ●●
●
● ●●
●
●
●● ●
●
●
●●
●●
●
●●
●●
●
●●
●●
●● ●●●●●●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
● ●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●●●
●
●●
●●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●●●●
●
●●
●
●
●●
●●
●●
●●
● ●●
●●
●
●
●●●
●
●
●
●
●●
●
●
● ●
●
●
●
● ●
●
●
●
●●
●
●●
●●
●
●
●
●●●
●
●
●●●
●
●
●
●●
●●●
●
●
●
●●●●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●●●
●●
●
● ●●
●
●
●
●
●
●●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●●
●
●●
●
●
●
●
● ●
●●
●
●
●
●
●
●●
●
●●
●●
●●
●
●●
●
●
●
●
●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
● ●
●●●
●●
●●
●
●●●
●
●●
●
●●●
●
●●●
●
●●●
●
●●
●● ●
●●
●●
●●
●
●
●
●●
●
●●●
●●●●
●●
●
●●
●
●
●
●●●
●
●
●
●●
●●●
●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●●●
●
●●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
● ●
●●●
●
●
●
●
●●● ●●
●
●
●
●●
●
●
●
●
●●
●●●●●●
●
●●
●●
●
●●
●●●●
●●
●●●●●●
●●●●
●
●
●
●●
●
●●●
●●
●●●
●
●●
●
●●●●
●
●●●●
●
●●
●●
●
●
●
●
●
●
●●●
●●●●
●
●
●●●
●●●●●
●
●●●
●●●
●●●
●
●●●●
●●
●
●●
●●●●
●
●●
●●●●●
●●●●●●
●●●
●
●●
●●●●
●
● ●●●●
●●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●●●
●●●●●
●●
●
●●●●●●
●●●●●●●●
● ●●●●●
●
●●●
●●●●
●
●
●●
●●
●●●
●
●●●●
●
●
●●
●●●
●
●●●
●
●●
●
●
●●
●●
●●
●●
●
●
●
●●●
●●
●●●●● ●●
●●●●●●●
●●●●
●●●
●
●●●
●●●●
●
●
●●●
●●●
● ●
●
●●●●●
●●
●
●●
●
●●
●●
● ●●●●●●
●●●●
●●●
●
●●
●
●●●●●
●●●
●●
●●
●●
●●
●
●●
●
●● ●●
●
●●●
●●
●
●
●
●
●
●●
●
●●●●
●
●●
●
●●● ● ●
●
●
●●●●
●●
●
●●●
●●
●
●
●●
●
●●●
●
●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●●
●●●
●●●
●
●
●●●●
●●●●
●
●●
●●
●●●
●●
●●●●●●●
●●●●
●●●●●●●
●●●
●●●●●●
●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●
●● ●
●●
●●
●
●●
●
●●●●
●●
●●●●
●
●
●●●
●●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●●●
●●
●●●●●●
●
●●
●●●
●
●
●●
●●●●
●
●●
●
●
●
●
●
●●●
●
●
●●
●●●
●
●●●●
●●
●●●●
●●●●●
●●
●●
●●
●
●●●●
●●
●●
●
●
●
● ●
●●●●
●
●●●
●
●●●
●●●●●●●●
●
●●●
●●
●
●
●●
●●●●
●
●
●●
●
●●●
●
●●●●●
●●●●
●●
●
●●
●●●
●
●●●●●●●
●●●●●●
●●●
●
●●
●
●●
●
●
●
●●●●
●●●
●●●
●●●
●●
●●●
●
●
●
●●
●
●
●
●●● ●
●
●●●●●●●
●
●●●● ●
●●
●
●●● ●
●
●
●
●
●
●● ●●●
●
●
●●●●
●●●●●●●●●●●●●●●●●●●●●
●●●●●
●
●
●●●●●●●
●
●
●●●●●
●●●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●●●●
●
●●
●
●●
●●●
●
●●
●●
●●
●●●
●●●●
●
●
●●●
●●●●●
●●
●
●
●●
●
●●
●●●●●●
●
●●●●●●●●●●●●
● ●●●
●●●
●●●●●
● ●●●
●●
●●
●●●●●●
●
●●
●●
●●●
● ●●●
●
●●●●
●
●●
●
●
●●●●
●●
● ●
●●
●
●●
●
●
●●●●●●●●●●●●●
●●
●●● ●
●●
●
●●●●
●●●●
●
●
●●● ●●●
●●●●●
●●
●●●●●
●●
●●●
●●
●●
●●●
●
●
●
●
●
●●
●●
●●●
●●
●
●●●
●
●
● ●●●
●
●●●●●●●●●●
●
●●●●●
●●●●
●●●●●
●●●●●
●●●
●●
●●●● ●
●
●●●●● ●
●
●
●●●●
●●●●●
●
●
●
●●●
●●●●
●
●●
●●
●●●●●●
●●
●●
●●
●
●
●●●
●
●●
●
●●●
●●●
●
●
●
●●●●●●●
●●●
●●●
●●●●●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●●
●
●
●●
●●●●●●●●●●●●●
●●
●
●●
●
●●●●●
●
●●●●●
●
●
●
●●
●●●
●
●
●
●●●●●●
●●
●●
●●●●●●●●
●
●
●
●●●●●
●
●
●
●●
●●●
●●●●
●
●●
●●
●
●●●
●●●●●●●●●
●
●
●●
●
●●
●
●●
●●
●●
●
●●
●
●
●●●●
●●●●●
●●●
●●●●
●
●
●●●●●
●●●
●●●●●
●●
●
●
●●
●
●
●●●●
●●
●●
●●
●●
●
●●●
●●●●●●●●●●●●●
●●●●●
●●●●
●●
●
●●●●●●
●●
●
●●●●●
●
●
●
●
●
●●●
●●
●●●●
●
●
●●●
●
●●●●●●●
●
●
● ●●●●
●
●●
●
●●●●●●●●
●●●
●●
●
●●
●●●●●
●●
●
●●●
●●
●
● ●
●
●●●●●●●
●
●●●●●●●
●●●●●●●●●
●
●●
●●●●
●●●●●●
●●
●
●●
●●●●●
●
●
●●●●●
●
●●
●
●●
●
●●
●●
●●
●●●
●●
●
●●
● ●
●●●●●
●●●●●●
●●● ●●
●●
●
●
●●●●●
●●
●●●
●●
●
●
●
●
●●●●●●
●●
●●●●●●
●●
●
● ●
●
●●
●
●
●●●
●
●
●●
●
●●● ●
●
●●●●
●
●●●●
●
●
●
●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●
●●
● ●
●●●●
●●●
●
● ●
●
●
●●●●
●
●
●●●● ●
●●●●
●●●
●●●●●
●●
●●●●●●●●●
●●
●●
●● ●●●●●●●
●●
●●
●●●●●
●
●●●●●
●●
●●
●
●
●●●
●
●●●●●●●
●
●●
●●
●●
●●
●●●●●●
●
●●●
●
●
●
●●
●
●●
●
●
●●●
●
●
●●
●
●
● ●
●
●●
●
●●
●●●
●
●●
●●
●●
●●
●●●
●
●
●
●
●
●●
●●●●●
●●●●●●
●●●
●
●●●●●
●
●
●
●●●
●
●●
●●● ●●
●●●●●●●●
●●●●●●
●
●
●●● ●●●
●●●●
●●
●
●●●● ●●
●●●●●●
●
●●
●●
●●
●●●
●
●
●●
●
●●
●
●●
●●●
●●
●●
●
●●●●
●●● ●●
●●●●
●●
●●●●
●●●
●
●
●
●●
●●●●●
●
●●●
● ●●●●●●●●●●●●●●
●
● ●●●●●●●●
●●●
●●●●●●●
●●
●
●●
●
●●
●
●●●●●●●●●●
●●●●●●
●●●●●●●●
●●
●
●●●●●●●
●
●●●
●
●●●
●●
●●
●●●●
●●
●●●●●
●●●●
●
●
●●
●●●●●●
●●
●
●
●●
●●
●●●
●●
●
●
●●●●
●
●●
●
●●●●
●●
●
●●
●
●●
●
●●●●●●
●●●● ●
●
●●●●●
●
●
●●●
●●
●●
●●●●●
●●●
●●●
●
●●●●
●●●
●
●●●●●
●●●●●
●●●
●●●
●●●
●
●●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●●●●
●●●●●
●
●●
●●
●
●
●
●●
●
●●
●●●●
●
●●
●
●
●
●●
●●●●●●●
●●
●●●
●●
●●●●●
●●
●●
●●●
●
●●
●
●●●
●●
●●
●●
●
●●●
●●●
●
●●
●
●
●
●●●●
●●●●●●●
●●●●●●●
● ●●●●
●●●●●
●
●●
●●
●●●●●●●●●●●●
●●●
●
●●
●●
●●
●
●●
●
●●
●
●●
●
●
●
●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●
●●
●●
●●●●
●
●
●
●●
●
●●●●●●●
●●●
●●
●●
●
●●●●
●
●
●
●
●
●●
●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
●●
●●●●
●●●
●
●●●
●●
●●●
●
●●
●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●
●●●●●●●
●●
●●●●●●●●●●●●●
●
●●
●
●
●
●●●●●●●
●
●●
●●●●●
●
●●●●
●
●
●●
●●
●●●
●●●
●●
●●●●●
●●
●●●●
●●
●
●●
●
●●
●
●●
●●●
●
●●
●●
●
●
●
●
●●
●●●●●●●●●
●
●
●●●●●
●
●
●●●
●●●●●
●
●●●●
●
●
●
●●
● ●●●●●●
●●●●
●
●
●
●
●
●
●●●●
●
●
●●●●●●
●
●●
●
●
●
●●●
●
●
●
●●●
●
●●●
●
●
●
●
●
●●●
●●●●
●
●
● ●●●●
●
●●●
●
●
●●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●●●●
●
●●
●
●●
●●
●●●
●
●●●●●●●
●●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●●
●
●
●●●●
●●
●●●●
●
●●
●
●●●
●●
●●●
●
●●●●
●
●●●●
●●● ●●●●●●
●●
●
●
●●●
●●●●●●●●
●●●●
●●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●●
●●
●●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●●●
●●●
●
●
●●●●
●
●
●●
●
●
●●
●
●
●●●
●●
●
●●
●●●
●●●●
●●●● ●●
●●
●●
●
●
●
●
●
●●
●●
●
●
●●●● ●●
●
●●●
●●
●
●●
●
●●
●●
●
●
● ●●●
●●●
● ●
●
●
●
●●●
●
●
●●●●●
●
●●●
●●●
●●
●●●
●
●
●
●
●
●
●●●●
●
●
●
●●●●●●●●●●●●
●●●
●●●●●●●
●●
●
●●
●●●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●●
●●●
●
●●●●●●
●●●●●●●●●
●
●●
●●●●●●●●
●●
●●●●●
●●●
●●●●●
●●●
●
●●●●●●
●●●●
●●
●● ●
●●●●
●●●●
●
●
●●
●
●●
●●
●
●●
●
●
●
●●●
●
●●
●●
●●
●●●●●
●
●
●
●●
●●
●
●●●
●●●●●●●●●
●
●●
●●
●
●
●●●●●
●
●●●●
●●●
●
●●●
●●
●●●
●● ●
●●●●
●●
●●
●●
●●
●
●●●
●●●
●●●
●
●
●
●
●●●●●
●● ●
●●●●●●
●
●
●
●
●●●
●●●●
●
●●●●
●
●●
●
●
●●●
●●
●●●●
● ●
●
●●
●
●●●
●
●●
●●●
●●●●●●●
●
●
●
●●●
●
●● ●●●●●
●●● ●
●●●●
●●●
●●
●
●●●
●
●●
●●
●
●
●●●●
●
●●
●
● ●
●●
●
●●
●●●
●●
●●
●●●●●●
●●●
●
●●●●●●
●●●●
●●●●●
●●
●
●
●●
●
●●●
●
●
●
●●
●
●●
●
●
●●
●●●●
●
●
●
●●
●
●●●●
●
●●●●●●
●●
●
●
●●
● ●
●●●
●
●
●
●
●
●●
●● ●
●●●
●
●●●
● ●●●
● ●●●●
●●
●
●
●●
●●
●
●●●●●●
●
●●
●●
●●
●●●●●●●
●
●
●
●●
●
●
●
●●
●
●●
●●●
●
●
●●●
●
●●
●
●
●
●●
●●
●
●
●●●●
●
●●
●
●●●
●
●●●●
●●
●
●●●
●●●
●●
●●
●
●
●●●●
●●
●
●
●
●
●●
●●●●●● ●●
●●●●
●●●●●●●
●●
●
●●●
●●●●
●
●●●
●
●
●
●●●
●●
●
●●
●●●
●●●●●●
●
●●
●
●
●
●●●
●
●●●
●
●●
●
●●●●●
●
●
●
●
●
●●●●
●●
●
●●
●●
●
● ●
●
●●●●
●●●
●
●
●
●●
●
●
●
●●●
●●
●●
●●
●●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●
●●●●
●
●
●
●● ●●
●●
●
●●
● ●
●
●
●
●●
●
●
●●●
●
●●●
●●●
●●
●●●
●●●
●
●●
●
●●●
●●
●●●
●
●●
●●●●
●●●●
●
●
●●
●●●●●●
●
●●●●●●●●
●●●●
●
●●●
●●●●
●
●
●●
●●
●●●
●
●●
●
●●●●●●●
●●●
●●
●●
●
●
●
●
●●●
●
●●●
●●
●
●
●
●●●●
●
●
●●●●●●
●●●●●
●
●
●●●
●●●●●●●●●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●
●●●●●●●
●●
●●
●●●
●●●
●●
●
●
●
●●●
●●
●
●
●
●
●
● ●
●●●
●
●
●
●
●
●●●
●●
●●●
●
●●●●●●●
●
●●
●●
●●●
● ●
●●●
●
●●●●
●
●
●
●
●●●●
●●●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●●●●●●
●
●
●●●●
●
●
●●●●●
●●
●●●●●
●
●●
●
●●
●
●
●●●●●●●●
●●●
●
●●
●
●●
●●●●● ●●
●
●●
●●●
●
●●●●●●●
●
●
●
●●●
●
●
●●
●
●●●●●●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●●
●●
●●
●
●●
●●
●
●●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●● ●●
●●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
● ●●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●●●●
●●
●
●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●●
●
●●
● ●●
●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●● ●●
●●●●●
●●●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
● ●●●
●
●
●
●●●
●
●●
●
●●
●
●
●●
●●
●
●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●●
●●●
●●●●●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●●●
●●
●
●
●●●
●●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●●
●●●
●●●
●●●●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●●
●●
●
●
●
●
●●●
●●
●●●
●●●●●●●●●
●
●●●●
●●●
●
●
●
●
●●●●
●●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●●●●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●●●
●●●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●●●●
●●●●
●
●●●●●●
●●●
●●
●
● ●
●
● ●
●
●●
●
●●
●
● ●
●
●
●●
●●
●●
●
●
●
●
●●●●
●●●●●●●
●
●
●
●
●
●
●● ●●
●
●●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
●●
●●
●
●●●
●●
●●●●●●●●●
● ●
●
●
●●●
●
●
● ●●●
●● ●●●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●●
●
●
●●
●●●●
●
●●●●●●●
●●
●
●
●●●
●●
●
●
●●●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●
●●●
●●●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●●●●●
●●
●
●
●
●
●
●
●●
● ●
●●
●
●
●
●●●
●●
●
●
●●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●●●
●●●
●
●●●
●●●
●●●
●
●●●●
●
●
●●
●
●●●●
●●●
●
●●
●
●
●●●
●●●●
●
●●
●
●●●●●
●
●●
●
●
●●
●
●
●
●
●
●●●●●●●
●
●
●
●
●
●●
●
●●●
●●●
●
●●●●
●●●●●●●●●●●
●
●
●
●
●
●●●●
●
●●●●●●
●
●
●
●●
●●●●●
●
●●●●
●●●
●
●
●
●●●
●●●●●●●●
●●●
●●●●●●●●●●●●●●●
●●
●●
●●
●
●
●
●●●●●●
●●
●
●●
●
●
●
●
●
●
●
●●●
●●●
●●
●
●●●
●
●●
●
●●●●
●●●
●
●●●
●
●●●●
●●
●
●
●
●●
●
●●●●
●
● ●
●
●●●
●●●●●●●●●●●●●●
●●
●
●
●●●
●
●
●
●●●
●●
●●●
●●
●
● ●
●
●
●●●
●
●
●●●
●●●
●●
●
●
●●●
●
●
●
●
●●●●
●
●●
●
●
●●● ●●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●●●●●●
●●●●●●
●
●●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●●●●
●
●
● ●●
●
●●●●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●●●
●●●
●
●
●●
●
●●●● ●●●
●●●
●●
●● ●●
●●
●●●
●
●●●●●●
●●
●●●●
●●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●● ●
●
●●●
●
●
●●
●●●
●
●●●
●●●
●●●●●●
●
●
●●
●●
●
●●●
● ●
●
●
●
●
●
●●
●●
●●●
●
●
●●
●
●●●●
●●●
●
●
●
●
●
●●●
●●
●
●●●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●●
●
●●
●●
● ●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●●●
●●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●●●
● ●
●
●● ●
●●
●●
●
●●
●●
●
●
●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●●●●
●
●●
●
●●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●
●
● ●●●
●●
●
●
●
●
●●●
●
●●
●
●
●●
●
●●
●●
●
●●●●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●● ●
●
●●●●●
●
●
●●●
●
●●
●
●
●●●●
●●
●
●
●
●●
●
●●
●
●
●
●●●
●●
●●
●
●
●●●
●
●
●●●●●
●
●●●
●●
●●
●
●●●●●
●
●
●●●●
●
●
●
●●
●
●
●
●
●●●
●
●●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●●
●●
●●
●●
●
●
●
●●●●●●
●●
●
●●●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●
●●
●●●
●●
●●
●●●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●●
●●●●●
●●●●●●
●●
●●
●
●
●●
●
●
●●
●●
●●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●●●●
●
●●●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●●●●●
●
●
●
●
●
●●
●
●●●●●●●●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●
●
●●●●
●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●●
●
●●
●●
●
●●
●
●
●
●
●●●●
●
●●
●●●
●
●●●●●
●
●
●
●●
●
● ●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●●●
●●
● ●
●●
●
●●●
●
●●
●
●
●
●●●
●●
●
●
●●●●●
●
●●
●
●
●●●
●●
●
●
●
● ●●●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●●
●
●● ●
●
● ●
●
●
●●
●
●
●
●●●
●●
●
●
●
●●
● ●
●
●●
●●●
●●●
●●
●
●
●●
●●●●●
●
●
●
●
●
●
●●
●
●
●●●●
●
●●
●●
●
●
●
●●●
●
●●
●
●
●●●
●
● ●
●●
●●●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●
●
●●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●
●
● ●
●
●●
●
●●●●●●
●●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●●●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●
● ●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●
●●●●
●
●
●●
●●
●●
●
●●
●●
●
●●
●
●
●●
●
●●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●●
●
●
●
●
●●
●●●●
●●●●
●
●●
●●●●●
●
●
●●●
●●●●
●
●
●●
●
●●●●●●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●●●
●●
● ●
●
●●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●●●
●
●
●
●●
●●
●●●
●
●
●
●●
●
●●
●
●
●●
●
●●●
●
●●
●●
●
●●●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●●●
●
●
●●●●●
●
●●
●
●●●●
●●
●●
●●
●●●
●
●●
●●
●
●
●
●
●●
●●●
●
●
●
●●●●●
●●
●
●
●●●●●●●
●
●●
●
●
●
●
●●●
●
●●●●
●
●
●●
●
●
●●
●
●●●●●
●
●●
●●●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●● ●●
●
●
●●
● ●●
●●
●●●
●●
●●●
●
●●●
●●
●
●●●●
●●
●
●
●
●
●
●
●●●
●●●●●
●●
●
●●
●●
●
●
●
●●●●
●●
●
●
●
●
●
●●
●
●
●●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●●
●
●●
●
●
●●
●●
●
●
●
●●
●
●●
●●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
● ●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●● ●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●●
●
●●
●●
●●●
●●
●●●●
●
●●
●
●
●●
●●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●●●●
●●●
●●●●●
●●
●●
●
● ●
●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●●●●●
●● ●
●●●
●●●
●●
●
●●
●●
●
●●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●●●
●●
● ●
●
●
●
●●
●
●
●
●●
●
●●
●
●●●
●
●●
●
●
●●●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●●
●
●
●●
●
●
●
●
●●
●●
●
●● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●●
●
● ●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●●●●●
●
●
●●
●
●
●
●●
●
●
●●●
●
●
●●
●
●●●
●●
●● ●
●
●●●●●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●● ●
●
●●
●
●●●
●
●●
●●
●●●●
●
●
●
●● ●
●●
●
●
●●
●●●
●●
●
●
●●●●●●●●
●
●●
●
●
●
●
●●●
●●
●●●●●●●
●
●●●
●
●
●●●
●● ●
●
●●
●● ●●
● ●●
●
●●●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
● ●●
●
●
●●
●
●●
●●
●
●
●
●
●●
●
●●
●
●●●●●
●
●●●
●
●●
●●
●
●●
●●
●
●
●●●●●●
●●
●
●●
●
●●●
●
●●
●●
●●
●
●
●
●●●●
●
●●
●●●●
●
●
●
●●
●
●
●
●●
●
●●●
●●
●
●●
●
●
●
●●
●
●
●●●
●●●
●●
●
●
●●
●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●●●●
●
●●●
●
●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●●
●●●
●
●●●
●●●
●● ●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●●
●
●●●
●●●●●●●●●
●●●
●●
●●●
●
●
●
●
●
●
●● ●
●●
●●
●
●
●
●
●
●
●●●
●●
●
●
●●●●●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●●
●
●●●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●● ● ●
●●●●
●
●●
●
●
●
●●
●●●●●
●
●●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●●
●●●
●
●
●
●
●
●
●●
●●●●●
●●
● ●●
●●
●●●
●●●
●●
●
●
●
● ●●
●●
●
●● ●●● ●
●●
●
●
●
●●●
●
●
●●
●
●● ●●
●●
●●●●
●
●●●
●
●
●●●●
●●
●
●
●●●
●
●●
●●
● ●
●● ●
●
●
●●
●
●● ●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●●●●
●
●
●●
●
●
●●● ●
●
●
●
●
●●
●●●●
●●●
●
●
●●●
●
●
●
●●
●
●●
●
●●●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●●●●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●●
●
● ●●
●●●●●●●
●●
●
●
●●●●●●●
●●
●
●●
●
●
●
●
●●●●●
●●
●●
●●●
●
●●●●
●●
●●
●●
●
●●
●
●●●
●●
●●
●
●●●●
●●
●
●●●●
●●●
●
●●
●
●
●●
●
● ●●
●●●
● ●
● ●
●
●
●●
●
●
●●
●●
●
●●●
●
●●●●●
●
●
●
●●
●
●●●●
●●●
●●
● ●
● ●●●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●●●
●
●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●
● ●
●●
●
●
●●●●●●●
●
●●●
●●
●
● ●
●●●
●
●●●●●
●●
●●
●
●
●
●●
●●
●●
●●
●
●●
●●
●
●
●●
●●●●●●
●●
●●
●
●
●
●
●●●
●
●●
●
●
●●
●●●●●●●
●●●●●
●
●●●
●●
●
●●●●●●●●●●●●●●
●●●●●● ●●●●●●
●●
●●●●
●
●
●●●●
●●
●
●
●
●
●
●●●
●●●●●
●●
●
● ●●
●● ●
●
●
●●
●●●●●●●●
●●
●●●●●●●●●●●
●
●●●●●
●●● ●
●●●●
●
●
●
●
●
●
●●● ●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●●
●
●●
●
●●●●
●
●●
●
●
●
●●
●
●
●
●●●
●●●
●●
●
●●
●
●●
●
●
●
●
●●
●●●●
●
●●
●
●
● ●●●
●
●
●
●●
●
●
●
●
●●
●
●●●
●
●●●
●●
●
●
●●
●●
●
●
●●●●
●●
●
●
●
● ●
●
●●● ●●
●●
●●●●
●●●●●
●
●
●●
●●●●
●
●
●
●●
●●●
●●
●
●
●
●●●●●●●●●
●●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ● ●
●●●●
●
●
●●●●
●
●
●
●●●●●●
●●
●●
●
●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●
●●●
●
●
●
● ●
●
●
●
●
●
●●
●
●●●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●●●●●●●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●●●
●●●
●●●
●● ●●●
●●
●
● ●
●●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●●●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●● ●
●●
●
●
●
●●
●
●
●
●●●
●
●●●
●
●●●●
●
●●
●●●
●
●
●
●●●●●
●●
●●
●
●
●
●●●
●●
●
●
●
●
●
● ●
●
●●
●
●
●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●●●● ●●
●
●●
●
●
●● ●●
●●●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●●●
●
● ●
●
●
●
●
●
●
● ●
●●●
●
●
●
●
●
●
●●
●● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●●
●
●●
●●●
●
●
●
●
●
●
●●
● ●
●●
●
●
● ●●
●●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●●
●
●
●
●●
●●●
●
●
●●●●●●
●●
●●
●
●
●●
●
●●●
●
●●●●●
●
●●●●
●●
●●●
●
●●●●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●●
●
● ●●●
●
●
●
●●●
●
●
●●
●
●
●●●
●●●●●● ●●
●●
●
● ●●●
●●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
●●●
●
●●
●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●●
●
●●
●●
●●
●
●●●
●
●●●
●
●
●●●●●●●●
●
●●●
●●●●●
●●●●●●●●●●●●●
●●●●●
● ●
●
●●
●
●
●●●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●●
●●●●●●
●●●
●
●●
●
●●
●
●●●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
● ●
●
●●●●●
●
●
●
●●
●● ●● ●●●
●
●
●●
●●●●
●
● ●●
●●● ●
●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●●●
●●●●●
●●●●
●
●●●●●●●
●
●
●●
● ●
●●●
●
●●●●●
●●
●
●●
● ●
●●
●●
●
0 20 40 60 80
010
3050
70
True value (train)
Pred
icted
val
ue (t
rain
)
●● ●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●●●
●
●
●
●
●
● ●
●
●●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●●
●●●●
●●●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●● ●
●
●
●●
●
●●
●
●
●
●●●
●
●
● ●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●
●
●●
●●
●
●
●
●●●
●
●
●
●●
● ●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●●
● ●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●
●
●
●●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●●
●●
●
●
●● ●●
●
●●
●●
●
●●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●●
●
●●
●●●●
●
●
●●
●●
●
●
●
●
●●
●
●●● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●● ●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●●●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●● ●
●●
●
●● ●
●
●●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●●
●
●
●
●
●●
●
●
●● ● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●●
●
●●
●●●
●●
●
●
●●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
● ●●●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●●●
●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●●● ●
●
●●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●●●
●
●●●●
●
●
●
●
●
●●
● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
● ●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●
●
●
●
● ●●
●
●●
●●
●
●
●●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●● ●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
● ●●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●●
●●
●
●
●
●
●
●
●
●
●
●●●●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●● ●
●
● ●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●●●
●
●
●●
●
● ●
●
●●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
● ●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
0 10 20 30 40 50 60 70
010
3050
True value (test)
Pred
icted
val
ue (t
est)
Linear model (S1, N2O)
0 20 40 60 80
010
2030
4050
True value (train)
Pred
icted
valu
e (tr
ain)
0 10 20 30 40 50 60 70
010
2030
4050
True value (test)
Pred
icted
valu
e (te
st)
21 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Further comparisons
Time for training Time for using once the model is trainedRF + (∼ 15 minutes) = (∼ 5 seconds)SVM = - (∼ 20 seconds)
sensitive to NMLP - + (∼ 1 second)
sensitive to d
Times correspond to the prediction for about 20 000 HSMU on aDell Latitude D830, Intel Core 2DUO 2.2GHz, 2GO RAM, OSUbuntu Linux 9.10 (and, for RF, to the training of about 15 000HSMU) with R.Time for DNDC: around 200 hours by using a desktop computerand around 2 days by using cluster 7!
22 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Understanding which inputs are impor-tant
Importance: A measure to estimate the importance of the inputvariables can be defined by:
for a given input variable randomly permute the input values andcalculate the prediction from this new randomly permutated inputs;
compare the accuracy of these predictions to accuracy of thepredictions obtained with the true inputs: the increase of meansquared error is called the importance.
Example (S1, N2O, RF):
●
●
●●
●●
●●
●●
●
1 2 3 4 5 6 7 8 9 10 11 12
510
1520
2530
Rank
Impo
rtan
ce (
mea
n de
crea
se m
se)
SOC
PH
Nr N_MRNfixN_FR
clay Nres TmeanBD rain
The variables SOC and PH are the most important for accuratepredictions.
23 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Understanding which inputs are impor-tant
Importance: A measure to estimate the importance of the inputvariables can be defined by:
for a given input variable randomly permute the input values andcalculate the prediction from this new randomly permutated inputs;
compare the accuracy of these predictions to accuracy of thepredictions obtained with the true inputs: the increase of meansquared error is called the importance.
This comparison is made on the basis of data that are not used todefine the machine, either the validation set or the out-of-bagobservations.
Example (S1, N2O, RF):
●
●
●●
●●
●●
●●
●
1 2 3 4 5 6 7 8 9 10 11 12
510
1520
2530
Rank
Impo
rtan
ce (
mea
n de
crea
se m
se)
SOC
PH
Nr N_MRNfixN_FR
clay Nres TmeanBD rain
The variables SOC and PH are the most important for accuratepredictions.
23 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Understanding which inputs are impor-tant
Example (S1, N2O, RF):
●
●
●●
●●
●●
●●
●
1 2 3 4 5 6 7 8 9 10 11 12
510
1520
2530
Rank
Impo
rtan
ce (
mea
n de
crea
se m
se)
SOC
PH
Nr N_MRNfixN_FR
clay Nres TmeanBD rain
The variables SOC and PH are the most important for accuratepredictions.
23 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
Understanding the relation between agiven input and the output
Evolution of N2O in function of the value of SOC
0.00 0.05 0.10 0.15 0.20 0.25
510
1520
25
SOC
N2O
pre
dict
ed
All the other variables are set to their mean values (S1, RF).
24 / 25Comparison of metamodels
N
Bishop, C. (1995).Neural Networks for Pattern Recognition.Oxford University Press, New York.
Boser, B., Guyon, I., and Vapnik, V. (1992).A training algorithm for optimal margin classifiers.In 5th annual ACM Workshop on COLT, pages 144–152. D. Haussler Editor, ACM Press.
Breiman, L. (2001).Random forests.Machine Learning, 45(1):5–32.
Vapnik, V. (1995).The Nature of Statistical Learning Theory.Springer Verlag, New York.
Villa-Vialaneix, N., Follador, M., and Leip, A. (2010).A comparison of three learning methods to predict N2O fluxes and N leaching.Submitted to MASHS 2010 http://mashs2010.free.fr.
25 / 25Comparison of metamodels
N
Methodology and results [Villa-Vialaneix et al., 2010]
ε-insensitive loss function
Go back
25 / 25Comparison of metamodels
N