University of Birmingham

R-Metric:Li, Ke; Deb, Kalyanmoy ; Yao, Xin

DOI:10.1109/TEVC.2017.2737781

License:Creative Commons: Attribution (CC BY)

Document VersionPublisher's PDF, also known as Version of record

Citation for published version (Harvard):Li, K, Deb, K & Yao, X 2018, 'R-Metric: Evaluating the Performance of Preference-Based Evolutionary Multi-Objective Optimization Using Reference Points ', IEEE Transactions on Evolutionary Computation, vol. 22, no. 6,pp. 821 - 835. https://doi.org/10.1109/TEVC.2017.2737781

Link to publication on Research at Birmingham portal

General rightsUnless a licence is specified above, all rights (including copyright and moral rights) in this document are retained by the authors and/or thecopyright holders. The express permission of the copyright holder must be obtained for any use of this material other than for purposespermitted by law.

•Users may freely distribute the URL that is used to identify this publication.•Users may download and/or print one copy of the publication from the University of Birmingham research portal for the purpose of privatestudy or non-commercial research.•User may use extracts from the document in line with the concept of ‘fair dealing’ under the Copyright, Designs and Patents Act 1988 (?)•Users may not further distribute the material nor use it for the purposes of commercial gain.

Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document.

When citing, please reference the published version.

Take down policyWhile the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has beenuploaded in error or has been deemed to be commercially or otherwise sensitive.

If you believe that this is the case for this document, please contact UBIRA@lists.bham.ac.uk providing details and we will remove access tothe work immediately and investigate.

Download date: 09. Jun. 2020

IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018 821

R-Metric: Evaluating the Performance ofPreference-Based Evolutionary Multiobjective

Optimization Using Reference PointsKe Li , Member, IEEE, Kalyanmoy Deb , Fellow, IEEE, and Xin Yao, Fellow, IEEE

Abstract—Measuring the performance of an algorithm forsolving multiobjective optimization problem has always beenchallenging simply due to two conflicting goals, i.e., conver-gence and diversity of obtained tradeoff solutions. There area number of metrics for evaluating the performance of amultiobjective optimizer that approximates the whole Pareto-optimal front. However, for evaluating the quality of a preferredsubset of the whole front, the existing metrics are inadequate.In this paper, we suggest a systematic way to adapt the exist-ing metrics to quantitatively evaluate the performance of apreference-based evolutionary multiobjective optimization algo-rithm using reference points. The basic idea is to preprocessthe preferred solution set according to a multicriterion decisionmaking approach before using a regular metric for performanceassessment. Extensive experiments on several artificial scenar-ios, and benchmark problems fully demonstrate its effectivenessin evaluating the quality of different preferred solution setswith regard to various reference points supplied by a decisionmaker.

Index Terms—Evolutionary multiobjective optimization(EMO), multicriterion decision making (MCDM), performanceassessment, reference point, user-preference.

I. INTRODUCTION

MOST real-world problem solving tasks usually involvemultiple incommensurable and conflicting objectives

which need to be considered simultaneously. Such prob-lems are termed as multiobjective optimization problems(MOPs) that have earned considerable attention in engi-neering design, modeling, and operations research. Insteadof a single solution that optimizes all objectives simulta-neously, in multiobjective optimization, we often look for

Manuscript received February 17, 2017; revised May 27, 2017; acceptedJuly 29, 2017. Date of publication September 26, 2017; date of current ver-sion November 28, 2018. This work was supported in part by EPSRC underGrant EP/K001523/1, and in part by NSFC under Grant 61329302. The workof X. Yao was supported by a Royal Society Wolfson Research Merit Award.(Corresponding author: Ke Li.)

K. Li is with the Department of Computer Science, University of Exeter,Exeter EX4 4QF, U.K. (e-mail: k.li@exeter.ac.uk).

K. Deb is with the Department of Electrical and Computer Engineering,Michigan State University, East Lansing, MI 48824 USA (e-mail:kdeb@egr.msu.edu).

X. Yao is with the School of Computer Science, University of Birmingham,Birmingham B15 2TT, U.K. (e-mail: x.yao@cs.bham.ac.uk).

This paper has supplementary downloadable multimedia material availableat http://ieeexplore.ieee.org provided by the authors.

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TEVC.2017.2737781

a set of Pareto-optimal solutions none of which can beconsidered better than another when all objectives are ofimportance.

Over the past two decades and beyond, evolutionaryalgorithms (EAs) have been widely accepted as a majorapproach for multiobjective optimization. Many efforts havebeen devoted to developing evolutionary multiobjectiveoptimization (EMO) algorithms, such as elitist nondominatedsorting genetic algorithm (NSGA-II) [1]–[3], indicator-basedEA [4]–[6], and multiobjective EA based on decomposition(MOEA/D) [7]–[9]. These algorithms, without any additionalpreference information (or intervention) from a decision maker(DM), are usually designed to obtain a set of solutions thatapproximate the whole Pareto-optimal set. However, the ulti-mate goal of multiobjective optimization is to help the DMfind solutions that meet his/her own preference information. Tofacilitate the decision making process, it is desirable to inte-grate the DM’s preference information into the search processof EMO for the following reasons.

1) Supplying a DM with a large amount of tradeoffpoints not only increases his/her workload, but alsoprovides many irrelevant or even noisy information tothe decision making process. Rather than the wholePareto-optimal front (PF), the DM is usually interestedin only a small set of tradeoff points most relevant tohim/her. A biased search, according to the DM’s pref-erence information, is able to provide more acceptablealternatives.

2) Due to the curse of dimensionality, the number of pointsused to accurately represent the whole PF increasesexponentially with the number of objectives. This notonly severely increases the computational burden ofan EMO algorithm, but also causes extra difficultiesfor the DM to comprehend the obtained solutions andthen to make decisions. Therefore, it is more practicalto search for a fine-grained resolution of a preferredregion of the PF by incorporating the DM’s preferenceinformation.

3) In a high-dimensional space, the mushrooming of non-dominated solutions, even for a randomly generatedpopulation, renders the traditional Pareto dominance-based selection useless [10]. However, by consider-ing the DM’s preference information, we can expecta necessary selection pressure additional to Paretodominance [11].

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/

822 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

In the past decade, there have been a number of studieson the preference-based EMO. Generally speaking, their ideascan be divided into four categories. The first one modifies theoriginal Pareto dominance by classifying objectives into differ-ent levels and priorities (e.g., [12]–[14]) or expresses the DM’spreference information by fuzzy linguistic terms according todifferent aspiration levels (e.g., [15]–[17]). The second sortmodifies the diversity management module so that the densityof Pareto-optimal solutions can be biased toward the region ofinterest (ROI) (e.g., [18]–[20]). The third approach combinesthe classical reference point-based method [21] with EMO(e.g., [22]–[24]). The last category, as a recent trend, combinesthe DM’s preference information with performance metrics(e.g., weight hypervolume (HV) [25], R2-indicator [26], andaveraged Hausdorff distance [27]) in algorithm design. In thispaper, our discussion focuses on the reference point-basedmethod, which has been recognized as one of most popularmethods in [28].

Despite the progress in algorithm design, few have beendone on evaluating the quality of preferred solutions obtainedby a preference-based EMO algorithm. Although a numberof performance metrics have been suggested to evaluate thequality of solutions that approximate the whole PF, includ-ing metrics for evaluating convergence (e.g., [29]–[31]) anddiversity (e.g., [32]–[34]) separately, and metrics that evaluateboth aspects simultaneously (e.g., [35]–[37]), none of themcan be directly applicable when only a partial PF is consid-ered. Some attempts to adapt the regular metrics to serve thepurpose of assessing the quality of a preferred solution sethave been reported in [38]–[41]. However, they are ad-hocand oversimplified which could make the assessments mis-leading. Possibly due to the lack of reliable metrics, manystudies, if not all, on the preference-based EMO heavily relyon the visual plot in performance comparisons. These meth-ods are rather subjective, and how to visualize data in ahigh-dimensional space is itself an open problem. This paperpresents a systematic way, denoted as R-metric, to quanti-tatively evaluate the quality of preferred solutions obtainedby a preference-based EMO algorithm using reference points.Our basic idea is to use a multicriterion decision mak-ing (MCDM) approach to preprocess the obtained solutions,according to their satisfaction to the DM’s preference informa-tion, before using a regular metric for performance assessment.It is so simple and general that any existing metric can beadapted with little modification. Note that it is a compre-hensive extension and generalization of the metric developedby the second author and his collaborators [42], while themajor differences are listed in Section I in the supplementarymaterial.

The rest of this paper is organized as follows. Section IIgives some preliminary concepts related to this paper.In Section III, the motivations of this paper are delin-eated and discussed. Section IV is devoted to the descrip-tion of the proposed method. Sections V and VI presentthe empirical studies on several artificial scenarios anda series of benchmark problems, respectively. Finally,Section VII concludes this paper and provides some futuredirections.

II. PRELIMINARY CONCEPTS

This paper considers the following continuous MOP withbox constraints:

minimize F(x) = (f1(x), . . . , fm(x))T

subject to x ∈ � (1)

where � =∏ni=1 [ai, bi] ⊂ R

n is the decision (variable) space,x = (x1, . . . , xn)

T ∈ � is a candidate solution. F : � →R

m+ constitutes of m real-valued objective functions, and Rm+

is called the objective space. The attainable objective set isdefined as � = {F(x)|x ∈ �}.

Definition 1: x1 is said to Pareto dominate x2, denoted asx1 � x2, if and only if: ∀i ∈ {1, . . . , m}, fi(x1) ≤ fi(x2) and∃j ∈ {1, . . . , m}, fj(x1) < fj(x2).

Definition 2: x∗ ∈ � is said to be Pareto-optimal if thereis no other x ∈ � such that x � x∗.

Definition 3: The set of all Pareto-optimal solutions iscalled the Pareto-optimal set (PS). The set of all Pareto-optimalobjective vectors, PF = {F(x)|x ∈ PS}, is called the PF.

For the ease of later discussion, we briefly introduce twowidely used performance metrics in the EMO literature.

1) Inverted Generational Distance Metric [35]: Let P∗ bea set of points uniformly sampled along the PF, and Sbe the set of solutions obtained by an EMO algorithm.The inverted generational distance (IGD) value of S iscalculated as

IGD(S, P∗

) =∑

x∗∈P∗ dist(x∗, S)

|P∗| (2)

where dist(x∗, S) is the Euclidean distance between thepoint x∗ ∈ P∗ and its nearest neighbor of S in theobjective space, and |P∗| is the cardinality of P∗.

2) Hypervolume Metric [31]: Let zw = (zw1 , . . . , zw

m)T bea worst point in the objective space that is dominatedby all Pareto-optimal objective vectors. HV metric mea-sures the size of the objective space dominated bysolutions in S and bounded by zw

HV(S) = VOL

(⋃

x∈S

[f1(x), zw

1

]× · · · [fm(x), zwm

])

(3)

where VOL(·) indicates the Lebesgue measure.Both IGD and HV metrics are able to give a comprehen-

sive information, including the convergence and diversity, ofS simultaneously. The lower is the IGD value (or the larger isthe HV value), the better is the quality of S for approximatingthe whole PF.

III. MOTIVATION

This section first discusses the shortcomings of some exist-ing metrics for evaluating the partial PF. Then, we develop themotivation for our proposed R-metric from the perspective ofthe achievement scalarization function (ASF) in MCDM.

A. Shortcomings of Regular Metrics

Let us start from the regular IGD and HV metrics, which areused as the baseline of R-metric, for assessing the partial PF.

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 823

(a) (b) (c)

Fig. 1. Variations of IGD and HV values with respect to a DM specified reference point zr = (0.16, 0.9). (a) Partial region versus whole region. (b) Tenpoint sets along the whole EF. (c) IGD and HV values for point sets in Fig. 1(b).

Note that although IGD and HV are not originally designed toassess the partial PF, some studies [43], [44] use them for com-paring preference-based EMO algorithms. The example PFconsidered here is a line (i.e., f2 = 1− f1) having an interceptof one with each objective axis. The DM’s preference informa-tion is specified as a reference point zr = (0.16, 0.9)T in theobjective space. Points focusing on the region closest to zr aremost relevant to the DM’s preference information. To calcu-late the IGD values, we sample 670 evenly distributed pointsalong the PF; and we set the worst point as zw = (1.1, 1.1)T

for calculating the HV value.1) In Fig. 1(a), two sets of points S1 and S2 have the same

cardinality (|S1| = |S2| = 20), but are with differentspreads along the PF. S1 crowds around zr, while S2

evenly distributes along the whole PF. From the DM’sperspective, S1 is preferable over S2. However, since S2

has a wider spread over the PF, it obviously has betterIGD and HV values than S1. Specifically, IGD(S1) =3.476E-1 and IGD(S2) = 4.610E-4; HV(S1) = 0.2910and HV(S2) = 0.6837.

2) In Fig. 1(b), ten sets of points S1 to S10 are createdalong the PF. Each set contains 40 evenly distributedpoints and has the same spread. Fig. 1(c) shows theIGD and HV values obtained by each point set. SinceS2 locates in the ROI, it was supposed to have thebest metric values. However, as shown in Fig. 1(c),S2 obtains the second worst metric values, whereas S5

and S6, far away from the ROI, obtain the best metricvalues.

In summary, neither IGD nor HV metric is reliable forevaluating the quality of a preferred solution set. A solu-tion set with additional but unwanted points may obtain abetter metric value, thereby making the IGD and HV met-rics unsuitable for performance assessment in the toy exampleshown in Fig. 1(a). On the other hand, even for differentpoint sets having the same spread along the PF, their IGDand HV values depend on their positions and the PF’s geo-metric property. This makes the IGD and HV metrics unsuit-able for performance assessment in the toy example shownin Fig. 1(b).

(a) (b)

Fig. 2. Illustration of two preference-based metrics. Mechanism of (a) [38]and (b) [39].

B. Overviews of Existing Preference-Based Metrics

In [41], a modified IGD was proposed to assess the close-ness of an approximation set S toward multiple referencepoints. It calculates the average of the shortest distancebetween a solution in S and the point in P∗ closest to areference point. This metric degenerates to the shortest dis-tance measure when only considering one reference point.Analogously, [27] proposed a preference-based EMO algo-rithm that tries to minimize the averaged Hausdorff distancebetween solutions and the DM supplied aspiration set, i.e.,a set of reference points. Although this idea is not directlyproposed for performance assessment, it can be used to assessthe quality of preferred solutions with respect to the DM’spreference information. References [38] and [39] adapted theregular HV metric for the preference-based EMO. Their basicideas are similar. At first, they merge solutions obtained by allconsidered algorithms into a composite set. Then, they spec-ify a preferred region within the composite set. Finally, onlysolutions falling within this preferred region are considered forperformance assessment. The major difference between [38]and [39] is the setting of the preferred region. As shown inFig. 2(a), [38] uses the closest point to the origin as the centerof the preferred region; while, as shown in Fig. 2(b), [39]uses the closest point to the DM supplied reference pointas the center. Both these two metrics do not require anyprior knowledge of the PF, and they work for some simple

824 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

Fig. 3. Illustration of the ASF ranking method. Considering the DM suppliedreference point zr , c is preferable over a and b; while a and b are equallyimportant.

examples. However, they have some flaws that make themmisleading.

1) It is obvious that [38] does not take the DM’s prefer-ence information into consideration. For the example inFig. 2(a), S1 is obviously preferable over S2 consider-ing the given reference point zr. However, S1 and S2

are distant from each other, and the origin is closer tothe points in S2. Therefore, S1 will be wrongly excludedfrom the preferred region for performance assessment.

2) On the other hand, although [39] considers the DM’spreference information in computation, it treats pointsoutside the preferred region equally redundant, e.g.,in Fig. 2(b), no point in S2 will be considered inperformance assessment. Considering the example inFig. 1(b), all ten point sets, except S2, cannot get anymeaningful metric value. This gives the DM a wronginformation that S1 to S10, except S2, are equally bad.

C. Intuitions of MCDM Approach

In the MCDM literature, there exists a number of methodsfor finding a preferred solution set according to the DM sup-plied reference point. In this section, we describe the basicidea of the ASF method [45], which is the foundation ofour proposed R-metric, in detail. In particular, The ASF1

considered in this paper is formulated as follows:

minimize ASF(x∣∣zr, w

) = max1≤i≤m

fi(x)− zri

wisubject to x ∈ � (4)

where zr is the reference point that represents the DM’s aspi-ration level for each objective, and w is the weight vectorthat implies the relative importance of objectives. Based onthe ASF, each objective vector has a projection, called iso-ASF point, on the reference line originated from zr and alongw, as shown in Fig. 3. Specifically, for a point a ∈ �, itscorresponding iso-ASF point al is calculated as

al = zr + δw (5)

1Here, we use the classic weighted Chebyshev function for discussion.Without loss of generality, other ASF forms can also be adapted accordingly.

where δ = max1≤i≤m [(ai − zri )/wi]. This iso-ASF point gives

us an information about the closeness of a to zr alongthe preferred direction w. Note that not all Pareto-optimalsolutions are equally important when considering the DM’spreference information. Based on the supplied reference pointand a preferred direction, ASF is able to rank all Pareto-optimal solutions. As shown in Fig. 3, for a, any point onits ASF contour line (e.g., point b) has the same ASF value,i.e., they are equally good and have the identical rank. Foranother point c, its iso-ASF point is cl. Comparing to al, cl

is closer to zr along the preferred direction. Thus, c shouldhave a better rank than a and b. Another nice property of thisASF-based ranking concept is its scalability to many-objectiveproblems.

IV. R-METRIC CALCULATION PRINCIPLE

The basic idea of our proposed method, denoted asR-metric, is to use an MCDM approach to preprocess thepreferred solution set according to the DM supplied preferenceinformation. Thereafter, regular metrics, e.g., IGD and HV, canbe applied for performance assessment. Note that the R-metricis specifically designed for evaluating the performance of apreference-based EMO algorithm using one or more referencepoints. In particular, we assume that the DM prefers the solu-tions lying toward the preferred direction, represented as adirect objective-wise weighting information or a worst point.In the R-metric calculation, the DM is required to providethree parameters relating to his/her preference information:1) a reference point zr that represents his/her aspiration levelor desired value for each objective; 2) a worst point zw ora weight vector w that specifies the relative importance ofeach objective; and 3) a relative extent of the ROI, denotedas � (0 < � ≤ 1). Note that most of these parameters areused to elicit the DM’s preference information and to help thepreference-based optimization procedure find a set of trade-off solutions in the ROI. Our proposed R-metric calculationis simple in principle and its high level flowchart is given inFig. 4. In the following paragraphs, we first describe each stepin detail. Then, we provide some further comments followedby a time complexity analysis.

A. Descriptions of Each Step

1) Prescreening Procedure: In multiobjective optimization,only the nondominated solutions are of interest to the DMs andare meaningful for performance assessment. Assume that thereare L(L ≥ 1) preferred solution sets (denoted as S1, . . . , SL),obtained by L different preference-based EMO algorithms, athand. We at first merge these L preferred solution sets into acomposite set Sc. For each Si, i ∈ {1, . . . , L}, only the non-dominated solutions, comparing to those in Sc, are retained forthe R-metric calculation. The pseudo-code of this prescreeningprocedure is given in Algorithm 1.

2) Pivot Point Identification: As the name suggests, thepivot point (denoted as zp) of a given set of preferred solu-tions (denoted as S) is used as the representative that reflectsthe overall satisfaction of S with respect to the DM suppliedpreference information. In this paper, we use the best solution

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 825

Fig. 4. Flowchart of R-metric calculation.

Algorithm 1: Prescreening Procedure

Input: Preferred solution sets S1, · · · , SL

Output: Processed S1, · · · , SL

1 for i← 1 to L do

2 Sc ←L⋃

l=1Sl\Si;

3 for j← 1 to |Si| do4 for k← 1 to |Sc| do5 if Sc(k) � Si(j) then6 Si = Si\{Si(j)};7 break;

8 return S1, · · · , SL

Algorithm 2: Trimming ProcedureInput: Preferred solution set S, ROI’s relative extent �

Output: Processed S1 for i← 1 to |S| do2 for j← 1 to m do3 if |fj(xi)− zp

j | > �2 then

4 S← S\{xi};5 break;

6 return S

with respect to (4) to serve this purpose and thus zp is

zp = argminx∈S

ASF(x∣∣zr, w

). (6)

3) Trimming Procedure: Instead of the whole PF, the ROIis a bounded region, i.e., a part of the PF, given the DM’spreference information. Only solutions located in the ROI areof interest to the DM. This paper defines the ROI approx-imated by S as the cubic that is centered at the pivotpoint and is with a side length �. Only solutions locatedin this approximated ROI are valid for performance assess-ment. The pseudo-code of this trimming procedure is given inAlgorithm 2.

4) Solution Transfer: This step is the main crux of ourR-metric by which the trimmed points are transferred toa virtual position. Then, we can assess their closeness tothe ROI along the preferred direction. To this end, we firstcompute the iso-ASF point of zp (denoted as zl) on thereference line connecting zr and zw. According to (5), thisrequires to identify the objective k that contributes to the ASFvalue

k = argmax1≤i≤m

zpi − zr

i

zwi − zr

i. (7)

Algorithm 3: Solution TransferInput: Preferred solution set SOutput: Processed S

1 k← argmax1≤i≤m

(zpi −zr

izwi −zr

i);

2 for i← 1 to m do

3 zli ← zr

i + zpk−zr

kzwk −zr

k(zw

i − zri );// iso-ASF point

4 for i← 1 to |S| do5 Shift S(i) along the direction vector zl − zp;

6 return S

Then, we can compute zl as

zli = zr

i +zp

k − zrk

zwk − zr

k

(zw

i − zri

)(8)

where i ∈ {1, . . . , m}. Thereafter, all trimmed points are trans-ferred, along the direction vector zl − zp with the distance‖zl−zp‖, to a virtual position. The pseudo-code of this solutiontransfer procedure is given in Algorithm 3.

5) R-Metric Calculation: In this paper, we choose the IGDand HV as the baseline metrics to evaluate the quality of apreferred solution set. The resulting R-metric is thus denotedas R-IGD or R-HV depending on the chosen baseline. For theR-HV, we simply compute the HV of the transferred pointswith respect to the worst point zw. For the R-IGD, we needto preprocess P∗ beforehand. More specifically, we first usethe method developed in Section IV-A2 to identify the pivotpoint of P∗. Then, we use the trimming procedure suggestedin Section IV-A3 to trim the points outside the ROI along thePF. In the end, the remaining points form the new P∗ for theR-IGD calculation.

B. Further Comments

1) In this paper, we set zw = zr + 2.0 × w for proof ofprinciple studies, where w is an unit vector. This w set-ting implies that all objectives are equally important.However, in practice, different objectives might havevarious importance to the DM. For example, if we setw = (2/

√5, 1/√

5)T , the first objective is assumed to betwice less important than the second one. In particular,the importance is in the inverse order of the weights.More detailed discussions on an unequal weight set-ting can be found from Section II in the supplementarymaterial.

2) In practice, the DM has no idea about the range ofthe whole PF, not to mention the extent of ROI. Thus,� plays as an approximate expectation of the relativeextent of ROI comparing to the whole PF. Note that theobjective space is assumed to be normalized to [0, 1].

826 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

(a) (b)

Fig. 5. Illustration of the trimming procedure. (a) Trimming procedure for S1.(b) Trimming procedure for S2.

3) The trimming procedure penalizes the solution set,in term of the diversity, for having an excessiveextent or deviating from the ROI. For example, inFig. 5(a) and (b), both S1 and S2 have the same cardinal-ity (|S1| = |S2| = 11), but S1 has a wider spread. Fromthe DM’s perspective (given � = 0.3), S2 is preferableover S1. After the trimming procedure, |S1| reduces to 5while |S2| is still the same. Accordingly, S1 will sacrificeits diversity when calculating the R-metric value. As foranother example shown in Fig. 6(a) and (b), S1 and S2

again have the same cardinality. Since S2 deviates fromthe ROI, its pivot point is identified as one of its extreme.After the trimming procedure, solutions far away fromthe ROI are excluded from further consideration.

4) Given the DM’s preference information, the convergenceis not only the closeness to the PF, but also the closenessof the transferred points to the ROI along the preferreddirection. This redefinition of convergence is fulfilledby transferring points to a virtual position along theiso-ASF line between zp and its iso-ASF point zl. Letus consider the example shown in Fig. 6 again. Thepivot point of S1 is exactly its iso-ASF point, sincethis pivot point lies exactly on the reference line con-necting zr and zw. In this case, solutions of S1 stay intheir original positions after the transfering procedure.In contrast, since the pivot point of S2 deviates fromzr, the remaining solutions of the processed S2 afterthe trimming procedure are shifted, along the directionzl−zp, to a virtual position away from the PF, as shownin Fig. 6(c).

C. Time Complexity Analysis

We now analyze the time complexity of R-metric calcula-tion. The prescreening procedure requires at most O(L2N2)

dominance comparisons. The identification of a pivot pointzp from N tradeoff points requires O(N) computations. Thetrimming procedure can be achieved in O(N) computations.Transfering at most N points toward the reference line alsorequires O(N) computations. The complexity of R-IGD orR-HV computation in last step is the same as the regularIGD or HV. In summary, the time complexity of R-IGD ismax{O(NM),O(L2N2)}, where M is size of the newly formedP∗ after the filtering procedure; the time complexity of R-HV

is max{O(Nm−1),O(L2N2)} by using the method developedin [31].

V. EMPIRICAL STUDIES ON ARTIFICIAL SCENARIOS

In this section, we verify the proposed R-metrics on severalartificial scenarios. First of all, the effectiveness of R-metrics isinvestigated on the two toy examples introduced in Section III.Afterward, four popular benchmark problems with differentpopulation distributions and different reference point settingsare used to further examine R-metrics. Next, we investigatethe applicability of R-metrics on a problem with disconnectedPFs. Finally, R-metrics are used to evaluate the quality of apreferred solution set with respect to multiple reference points.

A. Investigations on Toy Examples

For the first example discussed in Section III, as shownin Fig. 7(b), S1 and S2 are rearranged according to ourproposed R-metric calculation procedure. Since S2 does notfully meet the DM’s preference, points therein are penalizedin terms of convergence and diversity. More specifically, afterthe filtering procedure, only three points have been left inthe processed S2. Then, the processed S2 has been shiftedto an area away from the PF in the solution transfer step.Thereafter, the R-metric values of S1 should be better than thatof S2, i.e., R-IGD(S1) = 0.0222 and R-IGD(S2) = 0.0452;R-HV(S1) = 1.1550 and R-HV(S2) = 1.1108. As for the sec-ond example discussed in Section III, now shown in Fig. 7(c),all ten point sets have been shifted to their corresponding vir-tual positions along the preferred direction (the dashed arrowdenotes the transfer direction for each point set). Now, R-IGDand R-HV successfully figure out the best point set S2. Thisis because the processed S2, denoted as P2, is closest to theROI. In addition, S1 and S3, which have a similar distance tothe ROI after data preprocessing, achieve similar R-IGD andR-HV values. As for S4 to S10, their R-metric values becomeworse with their increasing distances to the ROI.

B. Investigations on Benchmark Problems

In this section, we investigate the effectiveness of R-metricson four classic benchmark problems, including two-objectiveZDT1 and ZDT2 [46], and three-objective DTLZ1 andDTLZ2 [47].2 For ZDT1 and ZDT2, ten sets of points, S1

to S10, are sampled from different regions of their PFs. Eachset contains 40 evenly distributed points. For DTLZ1 andDTLZ2, we sample 21 sets of points from different regions oftheir PFs, where each set has 25 evenly distributed points. ForR-IGD calculation, we first sample 10 000 evenly distributedpoints from the corresponding PFs. Only those located in theROI (� is set as 0.2) are chosen to form P∗ in the R-IGDcalculation.

1) Two-Objective Cases: As shown in Fig. 8(a), we inves-tigate three kinds of reference points for ZDT1.

1) Unattainable Reference Point zr1 = (0.2, 0.5)T: Fromthe results shown in Fig. 8(b), we find that both R-IGD

2Due to the page limit, the results on ZDT2 and DTLZ2, which are sim-ilar to the observations on ZDT1 and DTLZ1, are put in the supplementarymaterial.

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 827

(a) (b) (c)

Fig. 6. Illustration of the R-metric calculation principle. Note that the solution transfer is according to iso-ASF lines for the ASF with zr as the referencepoint and zw − zr as the weight vector. Trimming procedure for (a) S1 and (b) S2. (c) Solution transfer.

(a) (b) (c) (d)

Fig. 7. Illustration of R-IGD and R-HV values with respect to a DM supplied reference point zr = (0.16, 0.9). (a) Partial region versus whole region.(b) After data preprocessing. (c) Ten point sets along the PF. (d) R-IGD and R-HV values for each set.

(a) (b) (c) (d)

Fig. 8. Variations of R-IGD and R-HV with respect to an unattainable reference point zr1 = (0.2, 0.5)T , an attainable reference point zr2 = (0.6, 0.3)T , and anoutside reference point zr3 = (1.1,−0.1)T on ZDT1 problem. (a) Population distribution. (b) zr1 = (0.2, 0.5)T . (c) zr2 = (0.6, 0.3)T . (d) zr3 = (1.1,−0.1)T .

and R-HV are able to make a reasonable assessment onthe quality of a point set with respect to the DM sup-plied preference information. For example, S3 residesin the ROI with respect to zr1 . As shown in Fig. 8(b),the R-IGD and R-HV values obtained by S3 are indeedthe best. For the other point sets, the farther awayfrom the ROI, the worse the R-metric values are.

2) Attainable Reference Point zr2 = (0.6, 0.3)T: Similarto the observations in the above scenario, the point setclosest to the reference point, i.e., S6, obtain the bestR-IGD and R-HV values. And the R-metric values alsodepend on the distance to the ROI.

3) Extreme Reference Point zr3 = (1.1,−0.1)T: Since thisreference point lies on one extreme, it is expected that

the point set at the respective extreme boundary, i.e., S10,is desirable. From the results shown in Fig. 8(d), we findthat our proposed R-metrics are able to capture this factand their trajectories show a monotone property.

2) Three-Objective Cases: As shown in Fig. 9, we investi-gate two kinds of reference points for DTLZ1.

1) Unattainable Reference Point zr1 = (0.05, 0.05, 0.2)T:Different from the two-objective case, in a 3-D space,points are distributed in an ambient space, where theneighboring points can be in various directions. Thisexplains the significant fluctuations of R-IGD and R-HVcurves shown in Fig. 9(c). Nevertheless, the point setmost relevant to the DM supplied preference informa-tion, i.e., S17, obtains the best R-IGD and R-HV values.

828 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

(a) (b) (c) (d)

Fig. 9. Variations of R-IGD and R-HV with respect to an unattainable reference point zr = (0.05, 0.05, 0.2)T and an attainable reference pointzr = (0.3, 0.3, 0.2)T on DTLZ1 problem. (a) Population distribution (front view). (b) Population distribution (side view). (c) zr = (0.05, 0.05, 0.2)T .(d) zr = (0.3, 0.3, 0.2)T .

(a) (b)

Fig. 10. Variations of R-IGD and R-HV with respect to a referencepoint zr = (0.5, 0.0)T on ZDT3 problem. (a) Population distribution.(b) zr = (0.5, 0.0)T .

Furthermore, we also notice that the point sets close tothe reference point obtain similar R-metric values, e.g.,S9, S13, S14, S16, and S18 in DTLZ1 get similar R-IGDand R-HV values as shown in Fig. 9(d).

2) Attainable Reference Point zr2 = (0.3, 0.3, 0.2)T: Fromthe results shown in Fig. 9(d), we find that R-metrics areable to provide a reliable assessment on different pointsets. S9, which is closest to this reference point, obtainsthe best R-IGD and R-HV values.

C. Investigations on Problems With Disconnected PFs

Although the investigations in Sections V-B1 and V-B2are based on problems with continuous PFs, our proposedR-metrics are also effective for problems with disconnectedPFs. For proof of principle purpose, this section choosesZDT3, whose PF consists of five disconnected segments,for investigation. Five point sets, S1 to S5 as shown inFig. 10(a), are, respectively, sampled from each segment andzr = (0.5, 0.0)T . In addition, we also plot the transferred pointsets, denoted as green circles, for illustration (the dashed arrowdenotes the transfer direction for each point set). From theresults shown in Fig. 10(b), we find that the R-metrics are ableto provide a reasonable assessment for problems with discon-nected PFs. In particular, S3, which is closest to zr, obtainsthe best R-metric values. Note that the transferred points of S3

still lie on S3 and is indeed closest to zr. From Fig. 10(a), wefind that S4 is also very close to zr, the same for its transferredpoints. This explains its similar R-metric values with respectto S3. On the other hand, S1, farthest away from zr, obtainsthe worst R-metric values.

(a) (b)

Fig. 11. Variations of R-IGD and R-HV with respect to two differentreference point zr1 = (0.2, 0.5)T and zr2 = (0.6, 0.3)T simultaneously.Index 1 indicates (S3, S6), 2 indicates (S1, S2), 3 indicates (S4, S5), 4 indi-cates (S3, S4), and 5 indicates (S6, S7). (a) Population distribution. (b) Pointsets combination.

D. Investigations on Multiple Reference Points

In practice, the DM might not be sure about his/her exactpreference beforehand. The DM would like to simultaneouslyexplore several ROIs by supplying T (1 < T ≤ |S|) referencepoints at the same time. Accordingly, a small modification isrequired to adapt the R-metrics to this circumstance. Generallyspeaking, when there are more than one reference point, thepreprocessing procedure should take each reference point intoconsideration separately. At first, we use the prescreening pro-cedure introduced in Section IV-A1 to remove those dominatedsolutions from further R-metric calculation. Afterward, weapply the k-means [48] algorithm to divide the remaining solu-tions into T clusters. Then, each cluster is associated with areference point closest to its centroid. For each cluster, we usesteps 2 to 4 introduced in Section IV-A to preprocess the pointsand transfer them to a virtual position. Finally, we combineall preprocessed points together and evaluate their R-metricvalues as a whole. In particular, the worst point for R-HV cal-culation is chosen as the nadir point of the worst point foreach reference point; for R-IGD calculation, P∗ needs to bepreprocessed for each reference point separately according tothe method introduced in Section IV-A.

To validate the effectiveness of our strategy, we take theexample in Fig. 11(a) for investigation. Here, we set tworeference points zr1 = (0.2, 0.5)T and zr2 = (0.6, 0.3)T , simul-taneously, in the objective space. Five point set combinations,i.e., (S3, S6), (S1, S2), (S4, S5), (S3, S4), (S6, S7), are chosen asthe candidates for performance assessment. From the results

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 829

shown in Fig. 11(b), we find that (S3, S6) obtains the bestR-IGD and R-HV values. From Fig. 11(a), we can see that S3

and S6 are in the corresponding ROI of zr1 and zr2 , respec-tively. In contrast, (S1, S2) obtains the worst R-metric values.From Fig. 11(a), we find that both S1 and S2 are close tozr1 , but are far away from zr2 . Therefore, the R-metric valueswith respect to zr1 can be acceptable, whereas the R-metricvalues with respect to zr2 should be significantly bad. Thismakes its final R-metric values become the worst. Notice thatsometimes DMs tend to use multiple reference points to dis-cretely approximate the ROI. Therefore, the regions betweenthese supplied reference points are also very important. FromFig. 11(b), we found that the R-IGD and R-HV values obtainedby (S4, S5) and (S3, S4) are similar and are only inferior to(S3, S6). In contrast, although (S6, S7) has some part locatingin the ROI of zr2 , S7 is far away from zr1 . This makes itsR-metric values not as good as (S4, S5) and (S3, S4). In sum-mary, due to the existence of multiple reference points, a goodpoint set should have a promising satisfaction for every refer-ence point. In Section III of the supplementary material, weprovide some further discussions on the “scalability” issue ofR-metric.

VI. EMPIRICAL STUDIES ON PREFERENCE-BASED

EMO ALGORITHMS

In this section, we apply our proposed R-metrics to evaluatethe performance of the following four preference-based EMOalgorithms. Notice that all multiobjective optimizers use refer-ence points to articulate the DM’s preference information, andall of them, except g-NSGA-II, are capable of handling morethan one reference point. Here, we choose the classic ZDTand DTLZ test suites as benchmark problems. For R-IGD cal-culation, similar to Section V, we at first sample 10 000 pointsfrom the corresponding PF. Then, points located in the ROI(� is set as 0.2) are used to form P∗.

1) r-MOEA/D-STM [49]–[51]: It is an extension of ourrecently proposed MOEA/D variant based on sta-ble matching model [52]. Different from the originalMOEA/D, where the selection of next parents is merelydetermined by the ASF value of a solution, MOEA/D-STM treats subproblems and solutions as two sets ofagents and considers their mutual-preferences simulta-neously. In particular, the preference of a subproblemover a solution measures the convergence issue, whilethe preference of a solution over a subproblem measuresthe diversity issue. Since the stable matching achievesan equilibrium between the mutual-preferences betweensubproblems and solutions, MOEA/D-STM strikes a bal-ance between convergence and diversity of the searchprocess. In order to incorporate the DM’s preferenceinformation into the search process, we need to specifya population of weight vectors spread around referencepoints.

2) R-NSGA-II [22]: It is a variant of NSGA-II whichmodifies the crowding operator based on the idea ofclassic reference point-based method. More specifi-cally, solutions close to reference points have a larger

chance to survive in the selection procedure. In addi-tion, R-NSGA-II employs an ε-clearing idea to controlthe spread of the final obtained solutions in the ROI.

3) g-NSGA-II [53]: It modifies NSGA-II by replacing thePareto dominance with a new dominance relation, calledg-dominance. More specifically, g-dominance uses areference vector to represent DM’s desired value foreach objective, i.e., aspiration levels. Solutions eithersatisfying all aspiration levels or fulfilling none ofthe aspiration levels are preferable over those merelysatisfying some aspiration levels.

4) r-NSGA-II [23]: It uses a new dominance relation,called r-dominance, to replace the Pareto dominancein NSGA-II. When two solutions are nondominated interms of Pareto dominance, the one closer to the refer-ence point is preferable. Moreover, its search behavior isadjusted by two parameters: one is the non-r-dominancethreshold δ that controls the spread of the obtained solu-tions; the other is the ASF weight vector that controlsthe relative importance of different objectives.

All multiobjective optimizers use simulated binarycrossover [54] and polynomial mutation [55] as the repro-duction operators. For proof of principle purpose, all fouralgorithms assume all objectives are equally important inour empirical studies. Note that although r-MOEA/D-STM,R-NSGA-II, and r-NSGA-II are able to control the spread ofobtained solutions, there is no specific guideline to set thecorresponding parameter. Due to the page limit, the parametersettings and the specifications of reference points for differenttest instances are given in Section V in the supplementarymaterial.

A. Empirical Studies on Benchmark Problems

Each algorithm is performed 31 independent runs, and theR-metric values for two different reference point settings are,respectively, given in Tables I and II. In particular, the bestmean metric values are highlighted in bold face with graybackground, and the Wilcoxon’s rank sum test at a 0.05 signifi-cance level is used to compare the statistical significance of thedifference between the best mean metric value and the others.To have a visual comparison, we also plot the final solutionsobtained by different algorithms having the best R-IGD value.Due to the page limit, they are presented in Section VI in thesupplementary material. In the following paragraphs, we willseparately discuss the effectiveness of the R-metrics for eval-uating the performance of different algorithms on problemswith continuous and disconnected PFs.

1) Problems With Continuous PFs: ZDT1 and ZDT2 aretwo relatively simple test instances, where all four algorithmsdo not have too much difficulty in finding solutions aroundthe DM supplied reference points. However, as shown inFigs. 6(d) and 7(d) in the supplementary material, the con-vergence of solutions found by r-NSGA-II is not satisfactory.This makes most of its obtained solutions be trimmed duringthe prescreening step of the R-metric calculation. Accordingly,its R-IGD and R-HV values are the worst among all fouralgorithms. As for r-MOEA/D-STM and g-NSGA-II, their

830 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

TABLE ICOMPARISON RESULTS OF R-IGD AND R-HV VALUES ON UNATTAINABLE REFERENCE POINT

TABLE IICOMPARISON RESULTS OF R-IGD VALUES ON ATTAINABLE REFERENCE POINT

performance is visually similar on finding preferred solutionsfor ZDT1 and ZDT2. However, the R-IGD and R-HV val-ues obtained by g-NSGA-II are better than r-MOEA/D-STMin 6 out of 8 comparisons. Let us look at Fig. 6(a) and (c)in the supplementary material, for the unattainable referencepoint, all solutions found by r-MOEA/D-STM well converge

to the PF, whereas some solutions found by g-NSGA-II arenot fully converged. In this case, the R-metric values obtainedby r-MOEA/D-STM are better than g-NSGA-II. For the otherthree cases (i.e., ZDT1 with an attainable reference pointand ZDT2 with both unattainable and attainable referencepoints), although solutions found by r-MOEA/D-STM well

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 831

(a) (b) (c) (d)

Fig. 12. Variations of R-metric values for different �. (a) zr = (0.3, 0.4)T . (b) zr = (0.3, 0.4)T . (c) zr = (0.65, 0.3)T . (d) zr = (0.65, 0.3)T .

converge to the PF, their overall distributions deviate fromthe ROI a bit. In contrast, solutions found by g-NSGA-IInot only converge to the PF, but also have a well concen-tration on the ROI. Therefore, it should be preferable andour proposed R-metrics also make a reasonable assessment.ZDT4 has the same PF shape as ZDT1, but it is more diffi-cult due to the presence of many local optima. All algorithms,except r-MOEA/D-STM, cannot find solutions that fully con-verge to the PF. Accordingly, r-MOEA/D-STM obtains thebest R-metric values among all four algorithms. r-NSGA-IIobtains the worst R-metric values since it only finds solu-tions close to the unattainable reference point. ZDT6 has aconcave PF shape and a biased distribution in the searchspace. It is interesting to note that although the solutionsfound by r-MOEA/D-STM not only well converge to thePF but also have a uniform distribution, the R-metric valuesobtained by r-MOEA/D-STM are not as good as R-NSGA-IIand g-NSGA-II. This might be explained as the representativepoint found by r-MOEA/D-STM is inferior to that found byR-NSGA-II and g-NGSA-II. In this case, the solution transferstep of the R-metric calculation transfer the solutions found byr-MOEA/D-STM to a farther position. Due to the poor con-vergence property, the R-metric values obtained by r-NSGA-IIare still the worst.

The PF of DTLZ1 is a simplex having an intercept of0.5 at each coordinate. Due to the presence of 115 − 1local PFs, DTLZ1 causes difficulties for an EMO algorithmin reaching the global PF. From Fig. 10 in the supplemen-tary material, only r-MOEA/D-STM well approximates theROIs. Accordingly, it obtains the best R-metric values. It isinteresting to note that solutions found by R-NSGA-II seemto have a nice concentration on the DM supplied referencepoints, but their spreads are too narrow. In contrast, nei-ther g-NSGA-II nor r-NSGA-II finds any reasonable solution.DTLZ2 is a relatively simple test instance, where all four algo-rithms do not have much difficulty in finding solutions closeto the ROIs. As shown in Fig. 11 in the supplementary mate-rial, it is clear that solutions found by r-MOEA/D-STM arepreferable over the other candidates. Accordingly, its obtainedR-metric values are also the best. Although solutions foundby R-NSGA-II well converge to the PF, their spreads are toonarrow. This explains its unsatisfied R-metric values. DTLZ3has the same PF shape as DTLZ2, but is with 310 − 1 localPFs. From Fig. 12 in the supplementary material, it is obvi-ous that g-NSGA-II and r-NSGA-II have some difficulties in

converging to the PF. Thus, their R-metric values are extremelypoor. In contrast, solutions found by r-MOEA/D-STM andR-NSGA-II are similar to those in DTLZ2. DTLZ4 also hasthe same PF shape as DTLZ2, but it has a strong bias towardf3− f1 plane. From Fig. 13 in the supplementary material, wecan see that solutions found by r-MOEA/D-STM are signifi-cantly better than the other three algorithms. Accordingly, itsR-IGD and R-HV values are the best. DTLZ5 and DTLZ6 aretwo degenerate problems, where the latter one has a strongbias away from the PF. From Fig. 14 in the supplementarymaterial, we find that solutions obtained by r-MOEA/D-STMare preferable since they converge well to the PF and have agood approximation to the ROIs. Accordingly, its obtainedR-metric values are also the best among four algorithms.Although solutions found by R-NSGA-II converge well tothe PF, their spreads are too narrow. For DTLZ6, all fouralgorithms have difficulties in converging to the PF. Solutionsfound by r-MOEA/D-STM seem to be closer to the PF andhave a wide spread. Accordingly, it obtains the best R-metricvalues.

2) Problems With Disconnected PFs: After the empiricalstudies on problems with continuous PFs, this section inves-tigates the effectiveness of our proposed R-metrics on twoproblems with disconnected PFs. For ZDT3, as shown inFig. 16 in the supplementary material, all four algorithmsare able to find solutions close to the reference points, butthose found by g-NSGA-II are visually better, where mostsolutions converge to the PF and the spread is satisfactory.Accordingly, the R-metric values obtained by g-NSGA-II arebetter than the other three algorithms. For DTLZ7, as shownin Fig. 17 in the supplementary material, although solutionsfound by r-NSGA-II have a well focus on the ROIs, they areaway from the PF. This explains its poorest R-metric val-ues. All the other three algorithms find some solutions onthe PF segments outside the ROIs. In particular, solutionsfound by r-MOEA/D-STM spread over all four PF segments,while those found by g-NSGA-II are the visual best asshown in Fig. 17 in the supplementary material. Accordingly,g-NSGA-II obtains the best R-IGD and R-HV values on bothunattainable and attainable reference points.

B. Empirical Studies on Many-Objective Problems

Recently, problems with more than three objectives havebecome one of the hottest topics in EMO. Due to the expansion

832 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

TABLE IIICOMPARISONS OF R-IGD AND R-HV VALUES ON DTLZ2 WITH 5 AND 10 OBJECTIVES

of the objective space in size, many-objective problems causeseveral challenges to the traditional EMO algorithm design.For example, the mushrooming of nondominated solutions ina population significantly weaken the selection pressure ofPareto-based EMO methods, and the sparse distribution of alimited number of solutions in a high-dimensional space makesthe density estimation and diversity management become evenmore difficult than the two- and three-objective cases. Asdiscussed in [56], instead of searching for the whole PF,finding a preferred subregion satisfying the DM’s preferenceinformation is more practical in many-objective optimization.

In this section, we investigate the scalability of R-metrics forquantitatively assessing the quality of a preferred solution setin a high-dimensional space. DTLZ2 with 5 and 10 objectivesare chosen as the test instances. The reference point is set aszr = (0.1, 0.3, 0.2, 0.4, 0.2)T for the 5-objective test instance,and zr = (0.3, 0.3, 0.3, 0.1, 0.3, 0.55, 0.35, 0.35, 0.25, 0.45)T

for the 10-objective case. For the R-IGD calculation, weemploy the method suggested in [56] to sample 101 270 pointsfrom DTLZ2’s PF in the 5-objective case, and 3 124 550 pointsin the 10-objective case. Moreover, � is increased to 0.5 in theR-metric calculation due to the sparse distribution of solutionsin a high-dimensional space. For preference-based EMO algo-rithms, all parameters are kept the same as Section VI, exceptthe number of function evaluations. Specifically, it is set as80 000 and 150 000 for the 5- and 10-objective case, respec-tively. Table III shows the comparisons of R-metric values andthe parallel coordinates of the populations with the medium R-IGD value are plotted in Figs. 18 and 19 in the supplementarymaterial. In particular, the red dotted line represents the refer-ence point. From the results shown in these two figures, it isclear that g-NSGA-II and r-NSGA-II are the worst optimizers.This observation is also confirmed by their worst R-IGD andR-HV values. Both r-MOEA/D-STM and R-NSGA-II are ableto find solutions around the ROIs. However, solutions found byR-NSGA-II almost concentrate on the reference point, whilethose found by r-MOEA/D-STM have a wider spread. Thisexplains the better R-HV values obtained by r-MOEA/D-STMin these two test instances.

C. Further Investigations

In this section, we further investigate some other interestingproperties of R-metrics. ZDT1 and DTLZ2 are chosen as thetest instances, since all four EMO algorithms have no dif-ficulty on solving them. For each test instance, we keep arecord of R-IGD and R-HV values of an intermediate pop-ulation every ten consecutive generations. Fig. 20 in the

supplementary material plots the variations of R-IGD andR-HV values versus the number of generations on ZDT1 withzr = (0.3, 0.4)T . From this figure, we find that all algo-rithms, except r-NSGA-II, converge to their optimal R-IGDand R-HV values within a few generations. In contrast, theR-metric trajectories of r-NSGA-II grow slowly with the num-ber of generations. Fig. 21 in the supplementary material plotssome intermediate populations for different algorithms. Fromthese four subfigures, we find that r-NSGA-II approximatesthe preferred region in a layer-wise manner while the otherthree algorithms converge to the preferred region rapidly (witharound 100 generations). These observations are in accord withthe corresponding R-metric trajectories. For DTLZ2, we havea similar observation. As shown in Fig. 22 in the supplemen-tary material, the R-metric trajectories of r-MOEA/D-STM andR-NSGA-II converge to a stable value within a few genera-tions. These observations are also validated by the plots ofintermediate populations in Fig. 23(a) and (b) in the supple-mentary material. As for g-NSGA-II, we also notice somefluctuations in its R-metric trajectories. This observation isalso in line with the fluctuations of the evolutionary popula-tion as shown in Fig. 23(c) in the supplementary material. TheR-metric trajectories of r-NSGA-II are rather rugged. FromFig. 23(d) in the supplementary material, we find that theintermediate populations of r-NSGA-II vibrate significantlyduring the search process.

From the above experiments, we have another interestingobservation that some algorithms do not need the predefinednumber of generations to find preferred solutions. For exam-ple, as shown in Figs. 21(a) and 23(a) in the supplementarymaterial, r-MOEA/D-STM only uses around 100 generationsto converge to the preferred region for ZDT1 and around80 generations for DTLZ2. Furthermore, an algorithm almostconverges to the preferred region when the R-metric trajec-tories become stable. Based on these observations, we keepa record of the standard deviations of the R-IGD and R-HVvalues obtained by different algorithms for every 10 and 25consecutive generations. In addition, we set two thresholdsτ = 0.1 and τ = 0.01 and to see how many generations analgorithm needs to have a R-metric’s standard deviation lessthan τ . From the empirical results shown in Table V in thesupplementary material, we observe that the standard devia-tion of R-IGD can be reduced to the given thresholds whenthe time window is set to ten generations. More interestingly,the number of generations that makes the standard deviationof R-IGD reduce to 0.01 is similar to the required budgetsof the corresponding algorithm converges to the preferredregion. Moreover, we also notice that the standard deviation

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 833

of R-HV cannot always be reduced to the given thresholds onDTLZ2. However, if we extend the time window to 25 gen-erations, even the standard deviation of R-IGD cannot dropdown to the expected thresholds in many cases. From thisexperiment, we find that our proposed R-metrics are not onlyreliable metrics to evaluate the performance of a preference-based EMO algorithm, more interestingly, the variation ofa certain R-metric (e.g., R-IGD with a time window of tengenerations and standard deviation’s threshold τ = 0.1) canbe used as a stopping criterion in searching for a preferredsolution set.

D. Influence of �

As described in Section IV, � represents the DM’s expec-tation of the ROI’s relative extent comparing to the whole PF.It is also used to trim the irrelevant solutions for the R-metriccalculation. A large � means that the DM prefers solutionshaving a wide spread, while a small � indicates that the DMprefers solutions having a good concentration on the ROI. Thissection takes ZDT1 as an example to investigate the influenceof � on R-metric values, where � varies from 0.1 to 1.0 withan increment of 0.1. Fig. 12 shows the variations of R-metricvalues obtained by four preference-based EMO algorithms fordifferent �.

Let us start from zr = (0.3, 0.4)T . As shown in Fig. 12(a),the R-IGD value obtained by R-NSGA-II is worse thanr-MOEA/D-STM and g-NSGA-II when � is small. But itbecomes the best in case � is larger than 0.2. Moreover, theR-IGD value obtained by r-NSGA-II is the worst when �

is small. However, when � is larger than 0.4, the R-IGD val-ues obtained by r-MOEA/D-STM, g-NSGA-II, and r-NSGA-IIare almost the same. Let us refer to Fig. 6 in the supple-mentary material, as for zr = (0.3, 0.4)T , solutions found byR-NSGA-II and r-NSGA-II have a wider spread than thoseof r-MOEA/D-STM and g-NSGA-II. If the DM expects theROI to be concentrated on his/her provided reference point,i.e., � is set to be small, solutions found by r-MOEA/D-STMand g-NSGA-II are preferable. Accordingly, the R-metricvalues obtained by the previous two algorithms should bebetter. On the flip side, if the DM expects the ROI to bewidely spread, i.e., � is set to be large, solutions foundby r-MOEA/D-STM and g-NSGA-II are not satisfactory anylonger. Even though the solutions found by R-NSGA-II andr-NSGA-II are not well converged, their wide spread meetthe DM’s expectation and provide him/her more choices.This explains their better R-metric values when � becomeslarge. As for zr = (0.65, 0.3), since solutions found by g-NSGA-II not only well converge to the PF, but also havea wide spread around the DM supplied reference point, itsR-metric values are constantly better than the other competi-tors. In contrast, although solutions obtained by R-NSGA-IIstill have a wide spread along the PF, their convergence is poor.Therefore, the R-metric values of R-NSGA-II are worse thang-NSGA-II.

It is worth noting that although some preference-based EMOalgorithms (e.g., [22], [23], and [49]) claim to be able to con-trol the ROI’s extent by setting an appropriate parameter, to

the best of our knowledge, there is no rule-of-thumb to setthe corresponding parameter. We believe that �, used in theR-metric calculation, is able to provide a general guideline totune the corresponding parameters in a posterior manner.

VII. CONCLUSION

Given the DM’s preference information, approximating apartial and preferred PF, rather than the whole PF, has beenone of the most important topics in the modern EMO research.Besides developing effective algorithms that drive solutionstoward the ROI, how to evaluate the quality of a set ofpreferred tradeoff solutions is of the same importance but hasrarely been studied in this literature. In this paper, we presenteda systematic way to evaluate the quality of a set of preferredsolutions obtained by a preference-based EMO using referencepoints. More specifically, we preprocess the preferred solu-tions, according to an MCDM approach, before using a regularmetrics for performance assessment. In particular, accordingto the DM’s expectation of the ROI’s extent, our proposedR-metric has a trimming procedure that penalizes the popula-tion diversity of a preferred solution set having an excessiveextent. Furthermore, inspired by the ASF-based ranking fromthe MCDM literature, our proposed R-metric has a transfer-ring procedure that transfers the preferred tradeoff solutions toa virtual position according to their satisfaction degree to theDM supplied preference information. Extensive experimentson several artificial scenarios and benchmark problems fullydemonstrate the efficacy of our proposed R-metrics for eval-uating the quality of a preferred solution set according to theDM supplied preference information.

This paper is a very first attempt to systematically and quan-titatively evaluate the quality of a preferred solution set. Muchmore attention and effort should be required on this topic. Infuture, we want to explore the following issues.

1) Note that there is no single way of expressing theDM’s preference information, so we may not be able toexpect a universal way for performance assessment. Thispaper assumes that the DM’s preference information isexpressed in terms of a reference point. However, thereexist other types of preferences to which our proposedR-metric may not be directly useful. To solve this draw-back, one may consider the method presented in [57] toadapt the R-metric to other types of preferences.

2) As described in Section IV-B, the setting of � assumesthe objective space is normalized to [0, 1]. However, inpractice, this assumption might not always hold. It isinteresting to investigate other method to specify theDM’s expectation of the relative extent of ROI withrespect to objectives in different scales.

3) Other than the weighted Chebyshev function used inour proposed R-metric, there have been some otherASF formats [58], [59] proposed in the multiobjectiveoptimization field. Different ASFs have distinct contourlines and characteristics. It is interesting to study theinfluences and effects of using different ASFs.

4) To avoid wasting computational resources and to exam-ine the formal convergence and optimality achieved by

834 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 22, NO. 6, DECEMBER 2018

an EMO algorithm, the research on the online stop-ping criteria (OSC) has obtained increasing popularity.Empirical studies in Section VI-C shows a simpleapplication of our proposed R-metrics for designingOSC. Nevertheless, more sophisticated and advancedtechniques [60] are worth being studied in future.

5) In addition to the empirical studies, it is of impor-tance to have a rigorous analysis of the optimal archivewith respect to the R-metric. This is not only usefulfor better understanding the behavior of R-metric itself,but also for providing foundations in the case of usingthe R-metric to design an indicator-based algorithm infuture.

REFERENCES

[1] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitistmultiobjective genetic algorithm: NSGA-II,” IEEE Trans. Evol. Comput.,vol. 6, no. 2, pp. 182–197, Apr. 2002.

[2] K. Li, S. Kwong, R. Wang, K.-S. Tang, and K.-F. Man, “Learningparadigm based on jumping genes: A general framework for enhanc-ing exploration in evolutionary multiobjective optimization,” Inf. Sci.,vol. 226, pp. 1–22, Mar. 2013.

[3] K. Li and S. Kwong, “A general framework for evolutionarymultiobjective optimization via manifold learning,” Neurocomputing,vol. 146, pp. 65–74, Dec. 2014.

[4] E. Zitzler and S. Künzli, “Indicator-based selection in multiobjectivesearch,” in Proc. 8th Int. Conf. Parallel Problem Solving Nat. (PPSN),Birmingham, U.K., 2004, pp. 832–842.

[5] N. Beume, B. Naujoks, and M. Emmerich, “SMS-EMOA: Multiobjectiveselection based on dominated hypervolume,” Eur. J. Oper. Res., vol. 181,no. 3, pp. 1653–1669, 2007.

[6] K. Li et al., “Achieving balance between proximity and diversityin multi-objective evolutionary algorithm,” Inf. Sci., vol. 182, no. 1,pp. 220–242, 2012.

[7] Q. Zhang and H. Li, “MOEA/D: A multiobjective evolutionary algorithmbased on decomposition,” IEEE Trans. Evol. Comput., vol. 11, no. 6,pp. 712–731, Dec. 2007.

[8] K. Li, Á. Fialho, S. Kwong, and Q. Zhang, “Adaptive operator selec-tion with bandits for a multiobjective evolutionary algorithm based ondecomposition,” IEEE Trans. Evol. Comput., vol. 18, no. 1, pp. 114–130,Feb. 2014.

[9] K. Li, S. Kwong, Q. Zhang, and K. Deb, “Interrelationship-basedselection for decomposition multiobjective optimization,” IEEE Trans.Cybern., vol. 45, no. 10, pp. 2076–2088, Oct. 2015.

[10] H. Ishibuchi, N. Tsukamoto, and Y. Nojima, “Evolutionary many-objective optimization: A short review,” in Proc. IEEE Congr. Evol.Comput. (CEC), Hong Kong, 2008, pp. 2419–2426.

[11] K. Deb and H. Jain, “An evolutionary many-objective optimizationalgorithm using reference-point-based nondominated sorting approach,part I: Solving problems with box constraints,” IEEE Trans. Evol.Comput., vol. 18, no. 4, pp. 577–601, Aug. 2014.

[12] C. M. Fonseca and P. J. Fleming, “Multiobjective optimization andmultiple constraint handling with evolutionary algorithms. I. A unifiedformulation,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 28,no. 1, pp. 26–37, Jan. 1998.

[13] J. Branke, T. Kaußler, and H. Schmeck, “Guidance in evolutionary multi-objective optimization,” Adv. Eng. Softw., vol. 32, no. 6, pp. 499–507,2001.

[14] K. C. Tan, E. F. Khor, T. H. Lee, and R. Sathikannan, “An evo-lutionary algorithm with advanced goal and priority specification formulti-objective optimization,” J. Artif. Intell. Res., vol. 18, no. 1,pp. 183–215, 2003.

[15] Y. Jin, T. Okabe, and B. Sendhoff, “Adapting weighted aggregation formultiobjective evolution strategies,” in Proc. 1st Int. Conf. Evol. MultiCriterion Optim. (EMO), Zürich, Switzerland, 2001, pp. 96–110.

[16] Y. Jin and B. Sendhoff, “Incorporation of fuzzy preferences into evo-lutionary multiobjective optimization,” in Proc. Genet. Evol. Comput.Conf. (GECCO), New York, NY, USA, 2002, p. 683.

[17] D. Cvetkovic and I. C. Parmee, “Preferences and their application inevolutionary multiobjective optimization,” IEEE Trans. Evol. Comput.,vol. 6, no. 1, pp. 42–57, Feb. 2002.

[18] K. Deb, “Multi-objective evolutionary algorithms: Introducing biasamong Pareto-optimal solutions,” in Advances in EvolutionaryComputing, A. Ghosh and S. Tsutsui, Eds. Heidelberg, Germany:Springer, 2003, pp. 263–292.

[19] J. Branke and K. Deb, “Integrating user preferences into evolu-tionary multi-objective optimization,” in Knowledge Incorporation inEvolutionary Computation, vol. 167, Y. Jin, Ed. Heidelberg, Germany:Springer, 2005, pp. 461–477.

[20] K. Deb and A. Kumar, “Interactive evolutionary multi-objectiveoptimization and decision-making using reference direction method,” inProc. 9th Genet. Evol. Comput. Conf. (GECCO), London, U.K., 2007,pp. 781–788.

[21] A. P. Wierzbicki, “The use of reference objectives in multiobjectiveoptimization,” in Proc. MCDM Theory Appl., Hagen, Germany, 1980,pp. 468–486.

[22] K. Deb, J. Sundar, U. Bhaskara, and S. Chaudhuri, “Reference pointbased multi-objective optimization using evolutionary algorithms,” Int.J. Comput. Intell. Res., vol. 2, no. 3, pp. 273–286, 2006.

[23] L. B. Said, S. Bechikh, and K. Ghédira, “The r-dominance: A newdominance relation for interactive evolutionary multicriteria decisionmaking,” IEEE Trans. Evol. Comput., vol. 14, no. 5, pp. 801–818,Oct. 2010.

[24] L. Thiele, K. Miettinen, P. J. Korhonen, and J. Molina, “A preference-based evolutionary algorithm for multi-objective optimization,” Evol.Comput., vol. 17, no. 3, pp. 411–436, 2009.

[25] D. Brockhoff, J. Bader, L. Thiele, and E. Zitzler, “Directedmultiobjective optimization based on the hypervolume indicator,”J. Multi Criteria Decis. Anal., vol. 20, nos. 5–6, pp. 291–317, 2013.

[26] T. Wagner, H. Trautmann, and D. Brockhoff, “Preference articulation bymeans of the R2 indicator,” in Proc. 7th Int. Conf. Evol. Multi CriterionOptim. (EMO), Sheffield, U.K., 2013, pp. 81–95.

[27] G. Rudolph, O. Schütze, C. Grimme, and H. Trautmann,“A multiobjective evolutionary algorithm guided by averaged Hausdorffdistance to aspiration sets,” in EVOLVE—A Bridge between Probability,Set Oriented Numerics, and Evolutionary Computation V, vol. 288.Cham, Switzerland: Springer Int., 2014, pp. 261–273.

[28] S. Bechikh, M. Kessentini, L. B. Said, and K. Ghédira, “Chapter four—Preference incorporation in evolutionary multiobjective optimization:A survey of the state-of-the-art,” Adv. Comput., vol. 98, pp. 141–207,Jun. 2015.

[29] D. A. V. Veldhuizen and G. B. Lamont, “Evolutionary computation andconvergence to a Pareto front,” in Proc. Late Breaking Papers Genet.Programm. Conf., Stanford, CA, USA, 1998, pp. 221–228.

[30] J. Knowles, L. Thiele, and E. Zitzler, “A tutorial on the performanceassessment of stochastic multiobjective optimizers,” Comput. Eng. Netw.Lab., ETH, Zürich, Switzerland, Tech. Rep. 214, 2006.

[31] E. Zitzler and L. Thiele, “Multiobjective evolutionary algorithms:A comparative case study and the strength Pareto approach,” IEEETrans. Evol. Comput., vol. 3, no. 4, pp. 257–271, Nov. 1999.

[32] J. R. Schott, “Fault tolerant design using single and multicriteria geneticalgorithm optimization,” M.S. thesis, Dept. Aeronautics Astronautics,Massachusetts Inst. Technol., Cambridge, MA, USA, May 1995.

[33] A. Farhang-Mehr and S. Azarm, “An information-theoretic entropymetric for assessing multi-objective optimization solution set quality,”J. Mech. Design, vol. 125, no. 4, pp. 655–663, 2004.

[34] K. C. Tan, T. H. Lee, and E. F. Khor, “Evolutionary algorithms formulti-objective optimization: Performance assessments and compar-isons,” Artif. Intell. Rev., vol. 17, no. 4, pp. 251–290, 2002.

[35] P. A. N. Bosman and D. Thierens, “The balance between proximity anddiversity in multiobjective evolutionary algorithms,” IEEE Trans. Evol.Comput., vol. 7, no. 2, pp. 174–188, Apr. 2003.

[36] R. L. While, L. Bradstreet, and L. Barone, “A fast way of calculat-ing exact hypervolumes,” IEEE Trans. Evol. Comput., vol. 16, no. 1,pp. 86–95, Feb. 2012.

[37] O. Schütze, X. Esquivel, A. Lara, and C. A. C. Coello, “Using theaveraged Hausdorff distance as a performance measure in evolutionarymultiobjective optimization,” IEEE Trans. Evol. Comput., vol. 16, no. 4,pp. 504–522, Aug. 2012.

[38] U. K. Wickramasinghe, R. Carrese, and X. Li, “Designing airfoils usinga reference point based evolutionary many-objective particle swarmoptimization algorithm,” in Proc. IEEE Congr. Evol. Comput. (CEC),Barcelona, Spain, 2010, pp. 1–8.

[39] A. Mohammadi, M. N. Omidvar, and X. Li, “A new performance met-ric for user-preference based multi-objective evolutionary algorithms,”in Proc. IEEE Congr. Evol. Comput. (CEC), Cancún, Mexico, 2013,pp. 2825–2832.

LI et al.: R-METRIC: EVALUATING PERFORMANCE OF PREFERENCE-BASED EMO 835

[40] G. Yu, J. Zheng, and X. Li, “An improved performance metric formultiobjective evolutionary algorithms with user preferences,” in Proc.IEEE Congr. Evol. Comput. (CEC), Sendai, Japan, 2015, pp. 908–915.

[41] J. A. H. Mejía, O. Schütze, O. Cuate, A. Lara, and K. Deb,“RDS-NSGA-II: A memetic algorithm for reference point basedmulti-objective optimization,” Eng. Optim., vol. 49, no. 5, pp. 828–845,2017.

[42] K. Deb, F. Siegmund, and A. H. C. Ng, “R-HV: A metric for computinghyper-volume for reference point based EMOs,” in Proc. 5th Int. Conf.Swarm Evol. Memetic Comput. (SEMCCO), Bhubaneswar, India, 2014,pp. 98–110.

[43] A. Mohammadi, M. N. Omidvar, X. Li, and K. Deb, “Integratinguser preferences and decomposition methods for many-objectiveoptimization,” in Proc. IEEE Congr. Evol. Comput. (CEC), 2014,Beijing, China, pp. 421–428.

[44] J. Zheng, G. Yu, Q. Zhu, X. Li, and J. Zou, “On decomposition methodsin interactive user-preference based optimization,” Appl. Soft Comput.,vol. 52, pp. 952–973, Mar. 2017.

[45] A. P. Wierzbicki, “The use of reference objectives in multiobjectiveoptimization,” in Multiple Criteria Decision Making Theory andApplications, G. Fandel and T. Gal, Eds. Berlin, Germany:Springer-Verlag, 1980, pp. 468–486.

[46] E. Zitzler, K. Deb, and L. Thiele, “Comparison of multiobjective evo-lutionary algorithms: Empirical results,” Evol. Comput., vol. 8, no. 2,pp. 173–195, 2000.

[47] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler, “Scalable test prob-lems for evolutionary multiobjective optimization,” in EvolutionaryMultiobjective Optimization (Advanced Information and KnowledgeProcessing), A. Abraham, L. Jain, and R. Goldberg, Eds. London, U.K.:Springer, 2005, pp. 105–145.

[48] A. K. Jain, M. N. Murty, and P. J. Flynn, “Data clustering: A review,”ACM Comput. Survey, vol. 31, no. 3, pp. 264–323, 1999.

[49] K. Li, Q. Zhang, S. Kwong, M. Li, and R. Wang, “Stable matching-basedselection in evolutionary multiobjective optimization,” IEEE Trans. Evol.Comput., vol. 18, no. 6, pp. 909–923, Dec. 2014.

[50] M. Wu et al., “Two-level stable matching-based selection inMOEA/D,” in Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC), 2015,pp. 1720–1725.

[51] M. Wu, K. Li, S. Kwong, Y. Zhou, and Q. Zhang, “Matching-based selection with incomplete lists for decomposition multiobjectiveoptimization,” IEEE Trans. Evol. Comput., vol. 21, no. 4, pp. 554–568,Aug. 2017.

[52] D. Gale and L. S. Shapley, “College admissions and the stability ofmarriage,” Amer. Math. Monthly, vol. 69, no. 1, pp. 9–15, 1962.

[53] J. Molina, L. V. Santana, A. G. Hernández-Díaz, C. A. C. Coello,and R. Caballero, “g-dominance: Reference point based dominancefor multiobjective metaheuristics,” Eur. J. Oper. Res., vol. 197, no. 2,pp. 685–692, 2009.

[54] K. Deb and R. B. Agrawal, “Simulated binary crossover for continuoussearch space,” Complex Syst., vol. 9, no. 2, pp. 1–34, 1994.

[55] K. Deb and M. Goyal, “A combined genetic adaptive search (GeneAS)for engineering design,” Comput. Sci. Informat., vol. 26, no. 4,pp. 30–45, 1996.

[56] K. Li, K. Deb, Q. Zhang, and S. Kwong, “An evolutionary many-objective optimization algorithm based on dominance and decompo-sition,” IEEE Trans. Evol. Comput., vol. 19, no. 5, pp. 694–716,Oct. 2015.

[57] M. Luque, R. Caballero, J. M. Luque, and F. Ruiz, “Equivalent infor-mation for multiobjective interactive procedures,” Manag. Sci., vol. 53,no. 1, pp. 125–134, 2007.

[58] K. Miettinen, Nonlinear Multiobjective Optimization, vol. 12. Boston,MA, USA: Kluwer Acad., 1999.

[59] R. Wang, Q. Zhang, and T. Zhang, “Decomposition-based algorithmsusing Pareto adaptive scalarizing methods,” IEEE Trans. Evol. Comput.,vol. 20, no. 6, pp. 821–837, Dec. 2016.

[60] T. Wagner, H. Trautmann, and L. Martí, “A taxonomy of online stoppingcriteria for multi-objective evolutionary algorithms,” in Proc. 6th Int.Conf. Evol. Multi Criterion Optim. (EMO), Ouro Preto, Brazil, 2011,pp. 16–30.

Ke Li (M’17) received the B.Sc. and M.Sc. degreesin computer science and technology from XiangtanUniversity, Xiangtan, China, in 2007 and 2010,respectively, and the Ph.D. degree in computerscience from the City University of Hong Kong,Hong Kong, in 2014.

He was a Post-Doctoral Research Associate withMichigan State University, East Lansing, MI, USA,and a Research Fellow with the University ofBirmingham, Birmingham, U.K. He is currently aLecturer (Assistant Professor) of Data Analytics

with the Department of Computer Science, University of Exeter, Exeter, U.K.He has served as a regular reviewer and has published several research papersin renowned journals, such as the IEEE TRANSACTIONS ON EVOLUTIONARY

COMPUTATION and the IEEE TRANSACTIONS ON CYBERNETICS. His cur-rent research interests include evolutionary multiobjective optimization, largescale optimization, machine learning and applications in water engineering,and software engineering.

Kalyanmoy Deb (F’13) received the B.Sc. degreein mechanical engineering from IIT Kharagpur,Kharagpur, India, in 1985, and M.Sc. and Ph.D.degrees in engineering mechanics from theUniversity of Alabama, Tuscaloosa, AL, USA, in1989 and 1991, respectively.

He is the Koenig Endowed Chair Professorwith the Department of Electrical and ComputerEngineering, Michigan State University, EastLansing, MI, USA. He has been a VisitingProfessor with various universities across the world

including IIT Kharagpur, Aalto University, Espoo, Finland, University ofSkövde, Skövde, Sweden, and Nanyang Technological University, Singapore.He has published over 475 research papers with over 103 000 Google Scholarcitations with an H-index of 104. His current research interests includeevolutionary optimization and their application in multicriterion optimization,modeling, and machine learning.

Prof. Deb was a recipient of the Infosys Prize, the TWAS Prize inEngineering Sciences, the CajAstur Mamdani Prize, the DistinguishedAlumni Award from IIT Kharagpur, the Edgeworth-Pareto Award, theBhatnagar Prize in Engineering Sciences, and the Bessel Research Awardfrom Germany. He is on the editorial board of 20 major internationaljournals. He is a Fellow of ASME and three Indian Science and EngineeringAcademies.

Xin Yao (M’91–SM’96–F’03) received the B.Sc.and Ph.D. degrees from the University of Scienceand Technology of China, Hefei, China, in 1982 and1990, respectively.

He is a Chair Professor of Computer Sciencewith the Southern University of Science andTechnology, Shenzhen, China, and a Professorof Computer Science with the University ofBirmingham, Birmingham, U.K. He has beenresearching multiobjective optimization since 2003,when he published a well-cited EMO’03 paper on

many objective optimization. His current research interests include evolu-tionary computation, ensemble learning, and their applications in softwareengineering.

Dr. Yao was a recipient of the 2001 IEEE Donald G. Fink PrizePaper Award, the 2010, 2016, and 2017 IEEE TRANSACTIONS ON

EVOLUTIONARY COMPUTATION Outstanding Paper Awards, the 2010 BTGordon Radley Award for Best Author of Innovation (Finalist), the 2011 IEEETRANSACTIONS ON NEURAL NETWORKS Outstanding Paper Award, thePrestigious Royal Society Wolfson Research Merit Award in 2012, the IEEEComputational Intelligence Society (CIS) Evolutionary Computation PioneerAward in 2013, and many other best paper awards. He was the President ofIEEE CIS from 2014 to 2015, and from 2003 to 2008 he was Editor-in-Chiefof the IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION. He is aDistinguished Lecturer of IEEE CIS.