performance assessment of multiobjective evolutionary

Performance Assessment of

Multiobjective Evolutionary Algorithms

Lucas S. Batista

Universidade Federal de Minas GeraisDepartamento de Engenharia Elétrica

Belo Horizonte - MG, Brasil

[email protected]

November, 2018

mailto:[email protected]

Contents

1 Introduction

2 Unary and Binary Quality Indicators

3 Performance Measures

Unary Quality Indicators

Binary Quality Indicators

4 Conclusions

Introduction Unary and Binary Quality Indicators Performance Measures Conclusions

Current Section

1 Introduction





4 Conclusions

3 / 39


Initial Presentation

State of the Art

Multiobjective Optimization Problem

minxxx

fff (xxx) ∈ Rm

subject to: xxx ∈ Ω

Ω =

gi(xxx) ≤ 0; i = 1, . . . ,phj(xxx) = 0; j = 1, . . . ,qxxx ∈ X

X =

xmin

k ≤ xk ≤ xmaxk

k = 1, . . . ,n

4 / 39



State of the Art

The global Pareto front and Approximation Sets

P = xxx∗ ∈ Ω | ∄ xxx ∈ Ω : fff (xxx) ≤ fff (xxx∗) ∧ fff (xxx) 6= fff (xxx∗)

5 / 39



State of the Art

A high-quality approximation set should:

1 Approach the true Pareto front as close as possible (con-

vergence); and

2 Be well-spread along the extension of the front found (diver-

sity).

How to evaluate the quality of estimation sets?

6 / 39



State of the Art

Comparison methods:

1 If one algorithm is better than another, can we express how

much better it is?

2 If no algorithm can be said to be better than the other, are

there certain aspects in which respect we can say the for-

mer is better than the latter?

How to best summarize approximation sets when de-

signing quality measures?

7 / 39


Current Section

1 Introduction





4 Conclusions

8 / 39


General Informations

Unary Quality Measures

The quality value assigned to an approximation set is inde-

pendent of other sets under consideration;

Unary quality indicators are most commonly used in the lit-

erature;

Assign a real number to an approximation set;

9 / 39



Unary Quality Measures

In general, are not capable of indicating whether an approx-

imation set is better than another;

Existing unary indicators at best allow to infer that an ap-

proximation set is not worse than another;

With many unary indicators and also combinations of unary

indicators, no statement about the relation between the cor-

responding approximation sets can be made.

10 / 39



Binary Quality Measures

The quality value assigned to an approximation set depends

on other sets under consideration;

Assign real numbers to ordered pairs of approximation sets;

The performance measure of MOEAs is also a multicriteria

task;

11 / 39



Binary Quality Measures

Binary quality indicators can be used to overcome the diffi-

culties with unary indicators;

However, when we compare l algorithms using a single bi-

nary indicator, we obtain l(l − 1) distinct indicator values;

This renders the analysis and the presentation of the results

more difficult.

12 / 39


Current Section

1 Introduction





4 Conclusions

13 / 39



Spacing measure (Schott 1995)

Spacing focuses on relative distance between solutions;

A value S = 0 indicates all members of the estimated front

are equidistantly spaced;

However, if all the solutions are located in the same point,

then S = 0 and in fact the diversity in this case is the worst.

S =

√√√√ 1

|A| − 1

|A|∑

i=1

(d − di)2

di = minj

(∣∣f1(xi )− f1(xj )

∣∣+

∣∣f2(xi )− f2(xj )

∣∣)

14 / 39



Error ratio (ER) (Veldhuizen 1999)

ei = 0 if vector i is in A∗ and 1 otherwise;

Lower values of the error ratio represent better nondomi-

nated sets;

ER is the proportion of non true Pareto points in A.

ER =

∑|A|i=1 ei

|A|

15 / 39



Maximum Error with respect to A∗ (ME) (Veldhuizen 1999)

Measure the distance between the found solution set and

A∗;

The more distance the solution set is from A∗, the greater

the measure ME is;

The greatest minimal distance between vectors of the solu-

tion set and their closest neighbors in A∗:

ME = maxj∈A∗

( mini∈A

dij )

16 / 39



∆ measure (Deb et al. 2002)

∆ =

∑mi=1 de

i +∑|A|

i=1

∣∣di − d

∣∣

∑mi=1 de

i + |A| d

17 / 39



γ measure (Deb et al. 2002)

γ =

∑|A|i=1 di

|A|

18 / 39



Generational distance (GD) (Nebro et al. 2006)

GD =

(∑|A|

i=1 d2i

)1/2

|A|

19 / 39



Inverted GD (IGD) (Veldhuizen and Lamont 1998)

Represent average distance from solutions in the global Pa-

reto front to the nearest solution in an approximation set;

A value of IGD = 0 indicates that all the generated elements

are in the Pareto front and they cover all the extension of the

Pareto front.

IGD =

(∑|A∗|

i=1 d2i

)1/2

|A∗|

20 / 39



Hierarchical Cluster Counting (HCC) (Guimarães et al. 2009)

Measure of both uniformity and extension of a set;

Identify the front with the best Pareto set shape;

It is an improved version of the sphere counting (SC).

21 / 39



Hyperarea and Ratio (HR) (Veldhuizen 1999)

Calculate the hypervolume of the multi-dimensional objetive

space enclosed by A and a reference point.

Consider both closeness and diversity.

HV (A) =

⋃

i

vol(yyy i) | ∀ yyy i ∈ A

HR =HV (A)

HV (A∗)

22 / 39



S-metric (HV) (Zitzler & Thiele 1999)

23 / 39



Coverage of two sets (CS) (Zitzler & Thiele 1999)

CS(X ′,X ′′)=︷︸︸︷

|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|

|X ′′|

24 / 39




CS(X ′,X ′′)=︷︸︸︷

|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|

|X ′′|

CS(X ′,X ′′) = (4/10) = 0.40

25 / 39




CS(X ′,X ′′)=︷︸︸︷

|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|

|X ′′|

CS(X ′,X ′′) = (4/10) = 0.40

CS(X ′′,X ′) = (3/10) = 0.30

26 / 39



Coverage of Many Sets (CMS) (Batista et al.)

Quantify the domination of the final population of one algo-

rithm over the union of the remaining ones.

Consider both closeness and diversity.

CMS(Xi ,Ui) =|a′′ ∈ Ui ; ∃ a′ ∈ Xi : a′ a′′|

|Ui |

Ui =

k⋃

j=1j 6=i

Xj

27 / 39



Coverage of Many Sets (CMS) (Batista et al.)

28 / 39



Pareto Dominance Indicator (NR) (Goh and Tan 2009)

NR assesses the number of non-dominated solutions in the

set;

This metric measures the ratio of non-dominated solutions

that is contributed by a particular solution set Ai to the non-

dominated solutions provided by all solutions.

NR i(A1,A2, . . . ,Ak ) =|Ai ∩ B|

|B|

B =

bi | ∀ bi ,∄ aj ∈ (A1 ∪ A2 ∪ . . . ∪ Ak) ≺ bi

29 / 39



D1R (Czyzak and Jaszkiewicz 1998)

D1R measures the mean distance, over the vectors in a

reference set R, of the nearest solution in an approximation

set A.

D1R(A,Λ) =1

|R|

∑

rrr∈R

minaaa∈A

d(rrr ,aaa)

d(rrr ,aaa) = maxk

λk (ak − rk ), Λ = λ1, λ2, . . . , λm

λk = 1/range(f Rk ), k ∈ 1, . . . ,m

30 / 39



R1 (Hansen and Jaszkiewicz 1998)

R1 calculates the probability that approximation A is better

than approximation B over a set of utility functions U:

R1(A,B,U,p) =

∫

u∈U

C(A,B,u)p(u)du

C(A,B,u) =

1 if u∗(A) > u∗(B)1/2 if u∗(A) = u∗(B)0 if u∗(A) < u∗(B)

in which p(u) is an intensity function expressing the proba-

bility density of the utility u ∈ U, and u∗(A) = maxaaa∈Au(aaa).

31 / 39



Set of utility functions

A set Up of utility functions based on weighted Lp norm is a

parametric set composed of functions of the following form:

up(xxx , fff∗,Λ,p) =

( m∑

k=1

λk (fk (xxx)− f ∗k )p

)1/p

, p ∈ 1,2, . . . ,∞

Λ = λλλ ∈ Rm |∑

k

λk = 1 and λk > 0, k = 1,2, . . . ,m

32 / 39



R1R (Hansen and Jaszkiewicz 1998)

R1R is R1 when it is used with a reference set R.

R1R(A,U,p) = R1(A,R,U,p) =

∫

u∈U

C(A,R,u)p(u)du

C(A,R,u) =

1 if u∗(A) > u∗(R)1/2 if u∗(A) = u∗(R)0 if u∗(A) < u∗(R)

33 / 39



R2 (Hansen and Jaszkiewicz 1998)

R2 calculates the expected difference in the utility of an ap-

proximation A with another one B.

While R1 just uses the function C(A,B,u) to decide which

of two approximation is better on utility function u, without

measuring by how much, R2 takes into account the ex-

pected values of the utility.

R2(A,B,U,p) = E(u∗(A))−E(u∗(B)) =

∫

u∈U

(u∗(A)−u∗(B))p(u)du

34 / 39



R2R (Hansen and Jaszkiewicz 1998)

R2R is R2 when it is used with a reference set R.

R2R(A,U,p) = R2(A,R,U,p) = E(u∗(R))− E(u∗(A))

R2R(A,U,p) =∫

u∈U(u∗(R)− u∗(A))p(u)du

35 / 39



R3 and R3R (Hansen and Jaszkiewicz 1998)

In some cases, ratios of best utility values may be more

meaningful than their differences.

R3(A,B,U,p) = E

(u∗(B)− u∗(A)

u∗(B)

)

=

∫

u∈U

u∗(B)− u∗(A)

u∗(B)p(u)du

R3R(A,U,p) = R3(A,R,U,p) = E

(u∗(R)− u∗(A)

u∗(R)

)

R3R(A,U,p) =∫

u∈U

u∗(R)− u∗(A)

u∗(R)p(u)du

36 / 39


Current Section

1 Introduction





4 Conclusions

37 / 39


Conclusions

In MOOPs, the optimization result is a set of solutions;

The evaluation of obtained solution set consists of (in gen-

eral) two aspects: convergence and diversity properties;

There exists no unary quality measure that is able to indi-

cate whether an approximation A is better than an approx-

imation B, even when we consider a finite combination of

unary measure;

Since the performance assessment of MOEAs represents

a multicriteria task, the consideration of binary quality mea-

sures is very useful.

38 / 39

Specialized Literature

Specialized Literature

M. P. Hansen & A. Jaszkiewicz (1998). Evaluating the quality of approximations tothe non-dominated set, Technical Report, pp. 1–30.

D. Van Veldhuizen & G. Lamont (2000). On Measuring Multiobjective EvolutionaryAlgorithm Performance, IEEE, pp. 204–211.

Joshua Knowles & David Corne (2002). On Metrics for Comparing NondominatedSets, IEEE, pp. 711–716.

Tatsuya Okabe, Yaochu Jin & Bernhard Sendhoff (2003). A Critical Survey of Per-formance Indices for Multi-Objective Optimisation, IEEE, pp. 878–885.

Zhenan He & Gary Yen (2011). An Ensemble Method for Performance Metrics inMultiobjective Evolutionary Algorithms, IEEE, pp. 1724–1729.

Title page

39 / 39

performance assessment of multiobjective evolutionary

Documents