Automatic Configuration of NSGA-II with jMetal and irace

Antonio J. Nebro Manuel López-Ibáñez

University of Málaga University of Manchester
Málaga, Spain Manchester, United Kindom
antonio@lcc.uma.es manuel.lopez-ibanez@manchester.ac.uk

Cristóbal Barba-González José García-Nieto

University of Málaga University of Málaga
Málaga, Spain Málaga, Spain
cbarba@lcc.uma.es jnieto@lcc.uma.es
ABSTRACT ’19 Companion), July 13–17, 2019, Prague, Czech Republic. ACM, New York,
jMetal is a Java-based framework for multi-objective optimization NY, USA, 8 pages. https://doi.org/10.1145/3319619.3326832
with metaheuristics providing, among other features, a wide set of
algorithms that are representative of the state-of-the-art. Although 1 INTRODUCTION
it has become a widely used tool in the area, it lacks support for The field of multi-objective optimization with evolutionary algo-
automatic tuning of algorithm parameter settings, which can pre- rithms and other metaheuristic techniques is an active research area
vent obtaining accurate Pareto front approximations, especially since the last twenty years. Around year 2000, the most widely used
for inexperienced users. In this paper, we present a first approach algorithm since then, NSGA-II [5], was proposed, and two seminal
to combine jMetal and irace, a package for automatic algorithm books about solving multi-objective problems with evolutionary
configuration; the NSGA-II is chosen as the target algorithm to algorithms were published [3, 4].
be tuned. The goal is to facilitate the combined use of both tools A factor that has promoted the development and application of
to jMetal users to avoid wasting time in adjusting manually the multi-objective metaheuristics has been the availability of software
parameters of the algorithms. Our proposal involves the definition tools. Since the emergence of PISA [2] in 2003, many frameworks
of a new algorithm template for evolutionary algorithms, which have been proposed, being jMetal one of them. jMetal started in
allows the flexible composition of multi-objective evolutionary al- 2006 [9] as a research project to develop a Java-based software
gorithms from a set of configurable components, as well as the framework for multi-objective optimization with metaheuristic
generation of configuration files for adjusting the algorithm pa- techniques. The source code is freely available since 2008, and it is
rameters with irace. To validate our approach, NSGA-II is tuned hosted in GitHub under MIT license1 . It has become a popular tool,
with a benchmark problems and compared with the same algorithm and the papers describing it [7][9][8][13] sum up more than 1300
using standard settings, resulting in a new variant that shows a cites at the time of writing this paper according to Google Scholar.
competitive behavior. jMetal was redesigned from scratch in 2015 [13] to be improved
in a number of aspects (architecture, code quality, project organiza-
CCS CONCEPTS tion, algorithm templates, parallelism support), but an aspect that
• Computing methodologies → Optimization algorithms; • The- is becoming a hot topic, the automatic configuration of metaheuris-
ory of computation → Bio-inspired optimization; tics, was not considered then. Most of metaheuristics depend on
a number of parameters, and their performance is bound largely
by finding proper values for them. This is usually a cumbersome
KEYWORDS
task, particularly in the case of users who are expert in the problem
Multi-objective Optimization, Metaheuristics, Software Tools, Au- domain but are unfamiliar with the algorithms.
tomatic Algorithm Configuration In this paper, we present a proposal to extend jMetal with sup-
ACM Reference Format: port for automatic parameter tuning. Our approach is based, on
Antonio J. Nebro, Manuel López-Ibáñez, Cristóbal Barba-González, and José the one hand, on designing a very flexible template for evolution-
García-Nieto. 2019. Automatic Configuration of NSGA-II with jMetal and ary algorithms in such a way that a multi-objective evolutionary
irace. In Genetic and Evolutionary Computation Conference Companion (GECCO algorithm can be composed of a number of building blocks that can
be configured as parameters which can take a range of values. On
Permission to make digital or hard copies of all or part of this work for personal or the other hand, these parameters are described in a way that they
classroom use is granted without fee provided that copies are not made or distributed can be tuned by irace [11], an R-based package which has not only
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
been used for algorithm configuration, but also for the automatic
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, design of multi-objective evolutionary algorithms [1]. The goal is to
to post on servers or to redistribute to lists, requires prior specific permission and/or a facilitate the combined use of both tools to jMetal users interested
fee. Request permissions from permissions@acm.org.
GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic
in solving a given problem, but not in wasting time in adjusting
© 2019 Association for Computing Machinery. manually algorithms’ parameters.
ACM ISBN 978-1-4503-6748-6/19/07. . . $15.00
https://doi.org/10.1145/3319619.3326832 1 jMetal site in GitHub: https://github.com/jMetal/jMetal
GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic A.J. Nebro et. al.

As this is our first attempt to adapt jMetal for automatic parame- 16 List<S> offspringPopulation;
17 List<S> matingPopulation;
ter configuration, we have focused in multi-objective evolutionary 18
algorithms and, in particular, in configuring the NSGA-II algorithm 19 population = createInitialPopulation();
20 population = evaluatePopulation(population);
to solve continuous optimization problems. All the new developed 21 initProgress();
22 while (! isStoppingConditionReached()) {
code is included in an experimental package called jmetal-auto, 23 matingPopulation = selection(population);
as we wanted to avoid confusing jMetal users by modifying the 24 offspringPopulation = reproduction(matingPopulation);
25 offspringPopulation = evaluate(offspringPopulation);
existing code. 26 population = replacement(population, offspringPopulation);
To validate our approach, we have carried out an experimental 27 updateProgress();
28 }
study where NSGA-II has been tuned with a family of benchmark 29 }
problems and compared with the standard NSGA-II and also with
SMPSO [12] as a representative algorithm of the state of the art. We can observe that the template consists of a number of meth-
The rest of the paper is organized as follows. Section 2 briefly ods that are called inside the run() method, which defines the flow
describes jMetal and irace, the two tools we combine in this work. control of the algorithms. These methods can be defined in any
The proposed approach is detailed in Section 3 and the experimen- subclass of the template, and this is the way how popular multi-
tation is included in Section 4. Finally, the conclusions and lines of objective metaheuristics including NSGA-II [5], SPEA2 [14], and
future work are presented in Section 5. many others are implemented.
The jMetal source code is sub-divided into four packages:
2 SOFTWARE TOOLS • jmetal-core: Classes of the core architecture plus some utili-
In this section, we briefly describe jMetal and irace, the two software ties, including quality indicators.
tools we combine in this work. • jmetal-algorithm: Implementations of the metaheuristics in-
cluded in the framework (more than 24 multi-objective algo-
2.1 The jMetal Framework rithms, excluding variants).
jMetal is tool for multi-objective optimization with metaheuris- • jmetal-problem: Implementations of problems, including 8
tics based on an object-oriented architecture comprising four core benchmark families, summing up more than 90 instances.
entities: algorithms, problems, solutions (encodings or represen- • jmetal-exec: Executable programs to configure and run the
tations), and operators, as depicted in Figure 1. They are related algorithms.
among them following the idea that an algorithm solves a problem
by using operators that manipulate solutions. 2.2 The irace Package
irace [11] is a software tool implemented in R aimed at the auto-
Algorithm 1 Pseudo-code of an evolutionary algorithm matic configuration of optimization algorithms, i.e., to find accurate
settings of a given algorithm for a given set of training instances of
1: P(0) ← GenerateInitialSolutions() a problem. In this context, an algorithm configuration is a complete
2: t ←0 assignment of values to all required parameters of an algorithm.
3: Evaluate(P(0)) In its latest versions (irace 2.0 and higher), irace implements an
4: while not StoppingCriterion() do elitist iterated racing algorithm, where algorithm configurations
5: Q(t) ← Variation(P(t)) are sampled from a sampling distribution, uniformly at random at
6: Evaluate(Q(t)) the beginning, but biased towards the best configurations found in
7: P(t + 1) ← Update(P(t), Q(t)) later iterations. At each iteration, the generated configurations and
8: t ←t +1 the "elite" ones from previous iterations are raced by evaluating
9: end while them on training problem instances. A statistical test is used to
decide which configurations should be eliminated from the race.
A feature appeared in jMetal 5 [13] is the inclusion of algorith- When the race terminates, the surviving configurations become
mic templates. If we focus on evolutionary algorithms, they can be elites for the next iteration. A complete description of elitist iterated
described with the pseudo-code included in Algorithm 1. The cor- racing is provided in the original paper [11].
responding AbstractEvolutionaryAlgorithm abstract class in jMetal
is defined as follows (we have omitted some non relevant details): 3 APPROACH FOR EXTENDING JMETAL TO
1 public abstract class AbstractEvolutionaryAlgorithm<S,R> WORK WITH IRACE: REDESIGNING
implements Algorithm<R> {
2
3
protected List<S> population;
...
NSGA-II
4 protected abstract void initProgress(); To configure an algorithm, irace requires a file containing the pa-
5 protected abstract void updateProgress();
6 protected abstract boolean isStoppingConditionReached(); rameters to be tuned, including their type (categorical, ordinal, real,
7 protected abstract List<S> createInitialPopulation() ; and integer) and the values they can take. Additionally, the algo-
8 protected abstract List<S> evaluate(List<S> population);
9 protected abstract List<S> selection(List<S> population); rithm must be prepared to be configured externally with any valid
10 protected abstract List<S> reproduction(List<S> population);
11 protected abstract List<S> replacement(List<S> population, parameter combination.
List<S> offspringPopulation); As our plan is to use irace to configure NSGA-II, we must deter-
12
13
14
@Override public abstract R getResult(); mine what kinds of parameters can be adjusted. We can distinguish
15 @Override public void run() { the following ones:
Automatic Configuration of NSGA-II with jMetal and irace GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

Figure 1: UML class diagram of jMetal 5.0 core classes.

• Selection, crossover, and mutation operators. then based in defining a new template for evolutionary algorithms,
• Common operator parameters, such as mutation and crossover which is shown in the following code snippet:
probabilities, which are applied to all of them.
• Specific operator parameters, such the distribution index of
the simulated binary crossover. 1 public class EvolutionaryAlgorithm<S extends Solution<?>>
• Algorithm parameters, such as the size of the offspring popu- 2 implements Algorithm<List<S>>{
3 protected List<S> population;
lation size, or the size of the population if an external archive 4
5 private Evaluation<S> evaluation;
is used. 6 private InitialSolutionsCreation<S> createInitialPopulation;
• Other parameters as, for example, different strategies for 7 private Termination termination;
8 private MatingPoolSelection<S> selection;
initializing the population. 9 private Variation<S> variation;
10 private Replacement<S> replacement;
We must note that the standard NSGA-II is a generational evo- 11
12 ...
13
lutionary algorithm, which is featured by using a ranking method 14 public void run() {
based on Pareto ranking and the crowding distance density estima- 15 population = createInitialPopulation.create();
16 population = evaluation.evaluate(population);
tor, both in the selection and replacement steps of the algorithm [5]. 17 initProgress();
When it is used to solve continuous problems, NSGA-II adopts the 18 while (!termination.isMet(attributes)) {
19 List<S> matingPop ;
simulated binary crossover (SBX) and the polynomial mutation. No 20 List<S> offspringPop ;
external archive is included in NSGA-II. 21 matingPop= selection.select(population);
22 offspringPop = variation.variate(population, matingPop);
However, as we intend to configure NSGA-II in an automatic 23 offspringPop = evaluation.evaluate(offspringPop);
24
way, we need to relax the aforementioned features in order to have 25 population =replacement.replace(population, offspringPop);
enough flexibility to modify the search capabilities of the algorithm. 26 updateProgress();
27 }
This way, we are going to consider that any multi-objective evolu- 28
29
}
...
tionary algorithm with the typical parameters (selection, crossover, 30 }
and mutation) and using ranking and crowding in the replacement
step can be considered as a variant of NSGA-II.
Our solution is composed of three parts: the definition of a new
template for evolutionary algorithms, the development of a program The basic difference with the former template is that we have
to configure NSGA-II from any parameter combination to work with changed the way it can be extended by replacing inheritance by
irace, and the definition of utilities for creating file with the desired delegation: instead of methods, the EvolutionaryAlgorithm class is
parameters to be configured. These components are located in a composed by a number of objects or components. This way, an
new and experimental package named jmetal-auto, which does not evolutionary algorithm can be defined by adding the proper com-
interfere with the other existing packages mentioned in Section 2.1. ponents to the template. Consequently, we have defined a package
We explore each of the new components next. including the components that can be used to configure a multi-
objective evolutionary algorithm.
3.1 New Template for Evolutionary Algorithms At the time of writing this paper, the available components are
those included in Figure 2. We must note that the components
After analyzing the current implementation of NSGA-II in jMetal
extending Evaluation and Termination add features that do not take
we concluded that it did not cope with the defined requirements
part in an automatic configuration process, as they do not affect
in an easy way, because configuring NSGA-II and variants of it
the working of the algorithm.
require to define new subclasses. The approach we adopt here is
GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic A.J. Nebro et. al.

Figure 3: Repairing strategies when a variator operator pro-

duces a value out of the valid bounds.

parameters (lines 15-17): the size of the offspring population and the
crossover and mutation operators. The first parameter indicates the
number of solutions that the variation component has to produce.
The crossover operator is SBX, which requires values for the
crossover probability and the distribution index (lines 1-4). We
can see also that the operator has a parameter of class RepairDou-
bleSolution (line 3). This defines which strategy to follow when
the operator produces a solution having some of its decision vari-
ables out of range (i.e., as we are dealing with continuous prob-
lems, the value of each decision variable vi must be in a range
[lowerBoundi , upperBoundi ]). In jMetal we provide three repairing
strategies (they are illustrated in Figure 3):
• Random: the variable takes a random value between the
Figure 2: Components included in jMetal to be combined in lower and upper bounds. This is the default strategy.
a multi-objective evolutionary algorithm. • Bounds: if the value is lower/higher than the lower/upper
bound, the variable is assigned the lower/upper bound.
• Round: If the value is lower/higher than the lower/upper
Detailing how NSGA-II can be implemented using this new tem- bound, the variable is assigned upper/lower bound.
plate and components is out the scope of this paper; notwithstand- The polynomial mutation operator is configured in a similar way
ing, we comment next how the variation component can be config- (lines 7-13).
ured with crossover and mutation operators. Let us take a look to
this code snippet: 3.2 Autoconfiguring NSGA-II
1 double crossoverProbability = 0.9;
2 double crossoverDistributionIndex = 20.0; The template for evolutionary algorithms presented in the last
3 RepairDoubleSolution crossoverSolutionRepair = new section can be used directly so, to instantiate a particular algo-
RepairDoubleSolutionWithRandomValue();
4 CrossoverOperator<DoubleSolution> crossover = new SBXCrossover( rithm, it must be properly configured with any valid parameter
crossoverProbability, crossoverDistributionIndex, configuration. In fact, the way of working of irace implies that it
crossoverSolutionRepair);
5 call a program with different parameter settings, run the resulting
6 RepairDoubleSolution mutationSolutionRepair = new
RepairDoubleSolutionWithRandomValue(); algorithm to solve a problem, and get as a result a value repre-
7 double mutationProbability = 1.0/problem.getNumberOfVariables(); senting somehow the quality of the Pareto front approximation
8 double mutationDistributionIndex = 20.0;
9 MutationOperator<DoubleSolution> mutation = found. Consequently, with the idea of making easy to any user
10 new PolynomialMutation(
11 mutationProbability, to auto configure NSGA-II, we have developed a program called
12 mutationDistributionIndex, AutoNSGAIIConfigurator for these purposes.
13 mutationSolutionRepair);
14 This program receives as a parameter a string containing a se-
15 Variation<DoubleSolution> variation =
16 new CrossoverAndMutationVariation<>( quence of pairs < –parameter, value> that is read and analyzed to
17 offspringPopulationSize, crossover, mutation); assign the proper parameter values. To process the string we have
used Picocli2 , a tool for creating Java command line applications.
The variation component we have to configure is an instance
of the CrossoverAndMutationVariation class, which requires three 2 Picocli: https://picocli.info/
Automatic Configuration of NSGA-II with jMetal and irace GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

To show an example of how it works, let us suppose that we • offspringPopulation: this is the population containing the new
want to configure the SBX crossover operator. The corresponding solutions produced after the selection and variation steps,
configuration substring would be the following: and its size can range between 1 and 400.
"--crossover SBX • createInitialsolutions: by default, the solutions of the initial
--crossoverProbability 1.0
--crossoverRepairStrategy bounds population are randomly created, so we have add two ad-
--sbxCrossoverDistributionIndex 30.0" ditional procedures, one based on the scheme used in the
and the code to process in the AutoNSGAIIConfigurator program is scatter search algorithm and another one based on latin hy-
included in this code snippet: percube sampling.
1 @Option( • crossover and mutation: the available crossover operators are
2 names = {"--crossover"}, SBX and BLX-alpha, while the mutation operators can be
3 required = true,
4 description = "Crossover: ${COMPLETION-CANDIDATES}") polynomial or uniform.
5
6
private CrossoverType crossoverType;
• selection: the selection scheme can be random or n-ary tour-
7
8
@Option(
names = {"--crossoverProbability"},
nament (with n ranging between 2 and 10).
9 description = "Crossover probability (default: ${DEFAULT-
VALUE})")
10 private double crossoverProbability = 0.9;
11
12 @Option( 4 EXPERIMENTATION
13 names = {"--sbxCrossoverDistributionIndex"},
14 description = This section is devoted to validate our proposal of combining jMetal
15 "SBX crossover distribution index (default: ${DEFAULT- and irace by applying it to find a configuration of NSGA-II suitable
VALUE})")
16 private double sbxCrossoverDistributionIndex = 0.20; for a set of benchmark problems. It is worth noting that in no way
17
18 @Option( our goal here is to search for the best configuration of NSGA-II, but
19 names = {"--crossoverRepairStrategy"}, to carry out a proof of concept.
20 description = "Crossover repair strategy (default: ${
DEFAULT-VALUE})") The experiment we have designed consists in using the WFG [10]
21 private RepairStrategyType crossoverRepairStrategy =
RepairStrategyType.random; problems for tuning NSGA-II and then use the resulting best config-
uration yielded by irace to test it against a default configuration of
We can observe that a crossover is always required, and its NSGA-II, when solving both the WFG and DTLZ problems [6] (the
parameters have a default value in case they are not indicated. problems of both families have been configured with two objec-
Once all the parameters have been read, they are used to con- tives). We intend not only to determine the degree of improvement
figure an instance of the EvolutionaryAlgorithm class, whose after over the base NSGA-II that can be obtained, but also to know if
calling its run() method, executes the algorithm and produces a the auto-NSGAII can compete with algorithms of the state-of-the-
front of solutions. As irace needs a value about the quality of these art. With that idea in mind, we have added SMPSO [12] to the
front, the relative hypervolume [15] is computed and returned. The comparative.
relative hypervolume I H r of an approximation front P is defined as, All the algorithms have been set to return 100 solutions after
given a reference Pareto front R, as I H
r = (I (R) − I (P))/I (R).
H H H computing 25 000 function evaluations. No attempt has been done
to adjust the parameters of SMPSO, with is configured with default
3.3 Creating the irace Parameter File settings.
irace needs, as input, a parameter file containing all the parameters The first step is to run irace with the parameter file described in
and the valid values they can take. This file is a text file that can Figure 4 by using the nine problems of the WFG benchmark. After
be written with any text processing tool, but we have developed a a few hours of execution in a virtual Linux machine with 24 cores,
package including an utility that allows to indicate all the desired the best configuration found by irace is the one included in Table 1
parameters to be taken into account and to generate automatically (right). We include in the left column of the same table the default
the file. settings of the NSGA-II algorithm in jMetal.
By using this package, we have created the parameter file for We can observe that there some noticeable differences. The auto
the automatic configuration of NSGA-II included in Figure 4. As tuned NSGA-II uses the external archive and the sizes of the popu-
commented in a previous section, we feel free to configure the evo- lation and offspring populations are 20 and 200, respectively; the
lutionary algorithm template, but keeping a replacement strategy crossover is BLX-alpha instead of SBX; the distribution index of the
based on ranking and crowding distance in order to keep the most polynomial mutation is 158.05 instead of 20.0; and the size of the
characteristic feature of NSGA-II. We assume that the resulting tournament is 9 instead of 2.
algorithm always has to return a population of 100 solutions. The second step is to compare the default and auto tuned NSGA-
We comment now the main parameters described in the file: II algorithms along with SMPSO. For that purpose, we have made
25 independent runs of all of them to solve the WFG and DTLZ
• algorithmResult: the algorithm can use the typical population
problems. After that, we have computed four quality indicators for
of fixed size, but it can also use an external archive which is
measuring convergence (additive epsilon), diversity (spread), and
updated whenever a new solution is evaluated. This archive
both properties (hypervolume and IGD+). We report in the result
is bounded to 100 solutions, and the crowding distance is
tables the median and interquartile ranges, highlighting the best
applied when it becomes full. When the archive is used, the
and second best indicator values in dark and light grey background,
population size can take a value between 10 and 400) and
respectively.
the result of the algorithm is the archive’s content.
GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic A.J. Nebro et. al.

algorithmResult "--algorithmResult " c (externalArchive,population)

populationSize "--populationSize " c (100) | algorithmResult %in% c("population")
populationSizeWithArchive "--populationSizeWithArchive " o (10,20,50,100,200,400) | algorithmResult %in% c("externalArchive")
#
offspringPopulationSize "--offspringPopulationSize " o (1,5,10,20,50,100,200,400)
#
createInitialSolutions "--createInitialSolutions " c (random,latinHypercubeSampling,scatterSearch)
#
variation "--variation " c (crossoverAndMutationVariation)
#
crossover "--crossover " c (SBX,BLX_ALPHA)
crossoverProbability "--crossoverProbability " r (0.0, 1.0) | crossover %in% c("SBX","BLX_ALPHA")
crossoverRepairStrategy "--crossoverRepairStrategy " c (random,round,bounds) | crossover %in% c("SBX","BLX_ALPHA")
sbxCrossoverDistributionIndex "--sbxCrossoverDistributionIndex " r (5.0, 400.0) | crossover %in% c("SBX")
blxAlphaCrossoverAlphaValue "--blxAlphaCrossoverAlphaValue " r (0.0, 1.0) | crossover %in% c("BLX_ALPHA")
#
mutation "--mutation " c (uniform,polynomial)
mutationProbability "--mutationProbability " r (0.0, 1.0) | mutation %in% c("polynomial","uniform")
mutationRepairStrategy "--mutationRepairStrategy " c (random,round,bounds) | mutation %in% c("polynomial","uniform")
polynomialMutationDistributionIndex "--polynomialMutationDistributionIndex " r (5.0, 400.0) | mutation %in% c("polynomial")
uniformMutationPerturbation "--uniformMutationPerturbation " r (0.0, 1.0) | mutation %in% c("uniform")
#
selection "--selection " c (random,tournament)
selectionTournamentSize "--selectionTournamentSize " i (2, 10) | selection %in% c("tournament")

Figure 4: irace parameter file to auto configure NSGA-II

Default settings for NSGA-II Settings of auto-NSGAII

–algorithmResult population –algorithmResult externalArchive
–populationSize 100 –populationSizeWithArchive 20
–offspringPopulationSize 100 –offspringPopulationSize 200
–variation crossoverAndMutationVariation –variation crossoverAndMutationVariation
–crossover SBX –crossover BLX_ALPHA
–crossoverProbability 0.9 –crossoverProbability 0.9874
–crossoverRepairStrategy random –crossoverRepairStrategy bounds
–sbxDistributionIndexValue 20.0 –blxAlphaCrossoverAlphaValue 0.5906
–mutation polynomial –mutation polynomial
–mutationProbability 1/L (L : number of variables) –mutationProbability 0.0015
–mutationRepairStrategy random –mutationRepairStrategy random
–polynomialMutationDistributionIndex 20.0 –polynomialMutationDistributionIndex 158.05
–selection tournament –selection tournament
–selectionTournamentSize 2 –selectionTournamentSize 9

Table 1: Settings for NSGA-II: default (left) and auto tuned (right).

Table 2: Additive epsilon indicator. Median and Interquartile Table 3: Spread quality indicator. Median and Interquartile
Range Range

NSGAII SMPSO AutoNSGAII NSGAII SMPSO AutoNSGAII

WFG1 4.52e − 012.4e −01 4.55e − 019.8e −03 5.99e − 037.3e −04 WFG1 7.54e − 015.6e −02 1.02e + 003.6e −02 1.23e − 011.1e −02
WFG2 5.41e − 032.4e −03 6.04e − 031.1e −03 3.88e − 035.5e −04 WFG2 7.84e − 011.5e −02 8.04e − 012.9e −02 7.58e − 011.5e −03
WFG3 3.34e − 015.3e −04 3.34e − 011.9e −04 3.33e − 012.7e −07 WFG3 5.67e − 012.2e −02 3.79e − 017.9e −03 3.62e − 019.7e −03
WFG4 1.29e − 022.2e −03 2.16e − 023.4e −03 6.72e − 031.2e −03 WFG4 3.64e − 015.0e −02 4.55e − 015.1e −02 1.30e − 011.6e −02
WFG5 3.31e − 023.7e −03 2.77e − 021.9e −04 2.76e − 021.6e −04 WFG5 3.91e − 015.3e −02 1.35e − 011.4e −02 1.33e − 012.8e −02
WFG6 1.50e − 025.9e −03 6.37e − 034.9e −04 6.13e − 037.2e −03 WFG6 3.55e − 014.8e −02 1.49e − 012.5e −02 1.05e − 012.8e −02
WFG7 1.28e − 024.0e −03 6.52e − 034.9e −04 5.20e − 031.9e −04 WFG7 3.56e − 014.1e −02 1.51e − 012.5e −02 1.10e − 012.5e −02
WFG8 1.68e − 011.0e −01 1.69e − 011.7e −02 2.45e − 011.3e −03 WFG8 6.17e − 016.5e −02 7.43e − 013.5e −02 5.46e − 013.2e −02
WFG9 1.42e − 022.0e −03 1.10e − 021.9e −03 7.39e − 031.2e −03 WFG9 3.75e − 013.4e −02 2.14e − 013.1e −02 1.29e − 011.6e −02
DTLZ1 3.53e − 021.5e −01 6.30e − 035.5e −04 3.08e + 011.7e +01 DTLZ1 1.04e + 002.6e −01 6.83e − 021.4e −02 6.00e − 016.1e −03
DTLZ2 1.12e − 024.2e −03 5.53e − 033.2e −04 5.28e − 032.5e −04 DTLZ2 3.38e − 014.6e −02 1.23e − 012.2e −02 9.89e − 023.1e −02
DTLZ3 1.00e + 016.4e +00 5.97e − 033.5e −01 1.05e + 023.6e +01 DTLZ3 9.25e − 011.3e −01 1.26e − 014.8e −01 6.04e − 011.0e −02
DTLZ4 1.18e − 025.4e −03 5.61e − 033.1e −04 5.32e − 031.9e −04 DTLZ4 3.43e − 015.8e −02 1.19e − 012.4e −02 9.41e − 022.2e −02
DTLZ5 1.01e − 022.2e −03 5.12e − 033.5e −04 5.03e − 032.7e −04 DTLZ5 3.26e − 013.0e −02 1.32e − 012.1e −02 1.01e − 012.5e −02
DTLZ6 3.72e − 015.3e −02 5.15e − 034.5e −04 5.06e − 032.9e −04 DTLZ6 7.66e − 019.2e −02 1.11e − 012.6e −02 9.61e − 022.7e −02
DTLZ7 7.78e − 032.6e −03 4.30e − 032.3e −04 4.06e − 032.8e −04 DTLZ7 6.20e − 012.4e −02 5.25e − 011.9e −03 5.25e − 011.8e −03

DTLZ1, and DTLZ3) out of the sixteen problems AutoNSGAII have

Tables 2,3,4, and 5 contain the obtained values of the four quality performed poorly.
indicators. Some conclusions can be drawn from a first examination This analysis is supported by the computing of the statistical
of these results. The auto tuned NSGA-II (called AutoNSGAII in the Wilcoxon rank sum test (at a 5% level of significance), whose re-
tables) does not only outperform globally NSGA-II, but also SMPSO. sults for all the quality indicators are included in Table 6. Each cell
Although the hypervolume has been used as quality measure in the contains a symbol representing each of the 17 problems (WFG1-9
tuning step, AutoNSGAII is the best performing algorithm when and DTLZ1-7). There are three different symbols in this table: “–”
considering the other three indicators, what suggests that the found indicates that there not statistical significance between the algo-
configuration is robust; only in three problem instances (WFG8, rithms in the row and in the column, “▲” means that the algorithm
Automatic Configuration of NSGA-II with jMetal and irace GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic

Table 4: Hypervolume quality indicator. Median and In- repeat the experiment using only the DTLZ1, DTLZ3, and WFG8
terquartile Range instances.
The resulting settings found by irace are reported in Table 7
NSGAII SMPSO AutoNSGAII
WFG1 4.49e − 017.6e −02 1.16e − 017.7e −03 6.34e − 012.6e −05
(right). When comparing the obtained configuration with the for-
WFG2 5.64e − 019.5e −04 5.62e − 011.2e −03 5.65e − 015.1e −05 mer one, included in the left column of the table, we can observe
WFG3 4.41e − 013.8e −04 4.41e − 012.2e −04 4.42e − 011.1e −05
WFG4 2.17e − 017.6e −04 2.03e − 012.4e −03 2.17e − 013.0e −03 a number of differences: the offspring population size is 5 instead
WFG5 1.95e − 012.9e −04 1.96e − 017.5e −05 1.96e − 011.0e −04 of 100, the crossover operator is SBX instead of BLX-Alpha, the
WFG6 2.03e − 018.9e −03 2.09e − 014.3e −04 2.08e − 011.3e −02
WFG7 2.09e − 013.5e −04 2.09e − 013.2e −04 2.11e − 013.1e −05 mutation operator is uniform instead of polynomial, and the size
WFG8 1.48e − 012.3e −02 1.48e − 011.0e −03 1.39e − 012.3e −03 of the tournament is 4 instead of 9. The similarities of both config-
WFG9 2.37e − 012.9e −03 2.35e − 018.2e −04 2.39e − 011.9e −03
DTLZ1 4.66e − 011.6e −01 4.94e − 011.9e −04 0.00e + 000.0e +00 urations are the use of an external archive, a population size of 20
DTLZ2 2.09e − 012.7e −04 2.10e − 011.5e −04 2.11e − 014.1e −05
DTLZ3 0.00e + 000.0e +00 2.10e − 016.3e −02 0.00e + 000.0e +00
individuals, and a crossover probability around 0.98.
DTLZ4 2.10e − 017.1e −04 2.10e − 011.5e −04 2.11e − 014.2e −05 The indicator values obtained for this second configuration are
DTLZ5 2.11e − 013.5e −04 2.12e − 011.3e −04 2.12e − 014.1e −05
DTLZ6 1.89e − 051.4e −03 2.12e − 016.9e −05 2.12e − 015.6e −05 included in the last column of Table 8 (we have named AutoNSGAIIb
DTLZ7 3.29e − 012.8e −04 3.30e − 019.8e −05 3.30e − 017.3e −05 the new version of autoNSGAII in the table), which contains also the
values of the former experiment for the three considered problems.
Table 5: IGD+ quality indicator. Median and Interquartile We can observe that all the results have been improved, particularly
Range in the case of the DTLZ problems, where the second auto configured
NSGA-II outperforms NSGA-II and the first AutoNSGAII, although
NSGAII SMPSO AutoNSGAII it cannot beat the values obtained by SMPSO.
WFG1 2.39e − 011.4e −01 4.76e − 011.0e −02 1.94e − 039.1e −05
WFG2 1.61e − 036.1e −04 2.68e − 036.7e −04 9.95e − 049.9e −05
WFG3 2.15e − 022.3e −04 2.13e − 021.4e −04 2.06e − 021.7e −05 5 CONCLUSIONS AND FUTURE WORK
WFG4 3.34e − 034.9e −04 1.14e − 021.4e −03 3.42e − 031.4e −03
WFG5 2.75e − 029.8e −05 2.71e − 025.7e −05 2.70e − 023.9e −05 We have presented a first approximation to adapt the jMetal frame-
WFG6 6.84e − 035.9e −03 3.05e − 033.2e −04 3.54e − 038.5e −03
WFG7 3.47e − 034.5e −04 3.15e − 031.8e −04 2.30e − 036.1e −05
work to work jointly with the irace package to allow the automatic
WFG8 4.24e − 022.3e −02 4.26e − 022.3e −03 5.80e − 023.0e −03 parameter configuration of multi-objective evolutionary algorithms.
WFG9 4.97e − 031.6e −03 6.05e − 034.1e −04 4.07e − 031.1e −03
DTLZ1 1.93e − 029.0e −02 2.81e − 031.0e −04 4.31e + 012.1e +01 The proposal involves the creation of a flexible and configurable
DTLZ2 3.15e − 032.7e −04 2.48e − 038.0e −05 2.30e − 035.2e −05 template for this family of techniques, and NSGA-II has been used
DTLZ3 1.01e + 018.3e +00 2.09e − 031.6e −01 1.35e + 024.7e +01
DTLZ4 2.59e − 034.1e −04 2.13e − 036.9e −05 1.95e − 033.0e −05 as target algorithm. We have developed also a program able of in-
DTLZ5 2.50e − 031.7e −04 2.05e − 039.0e −05 1.81e − 035.8e −05 teract with irace in such a way that it can receive any configuration
DTLZ6 3.88e − 015.4e −02 1.91e − 039.1e −05 1.83e − 034.3e −05
DTLZ7 2.42e − 032.2e −04 1.88e − 031.2e −04 1.82e − 039.3e −05 from irace, set the corresponding NSGA-II variant, run the resulting
algorithm, and return as a result a quality measure of the produced
front. All this new functionality has been included in a new package
Table 6: Wilcoxon rank sum test results. The symbols in
called jmetal-auto.
each cell correspond to problems WFG1-9 and DTLZ1-7.
We have validated our proposal in two steps. First, we have
Additive epsilon
auto configured NSGA-II from a wide set of parameters that can be
SMPSO AutoNSGAII incorporated into the algorithm; the nine benchmark problems of
NSGAII – – ▽ ▲ ▽ ▽ ▽ – ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▲ ▽ ▲ ▽ ▲ ▽ ▽ ▽ ▽ the WFG family have been used for this auto-tuning. Second, the
SMPSO ▽▽▽▽▽– ▽▲▽▲▽▲▽▽– ▽
Spread resulting NSGA-II with the found configuration has been compared
SMPSO AutoNSGAII
NSGAII ▲ ▲ ▽ ▲ ▽ ▽ ▽ ▲ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽
against the same algorithm with default settings and with SMPSO
SMPSO ▽▽▽▽– ▽▽▽▽▲▽▲▽▽▽– on a benchmark composed of 16 problems. The reported results
Hypervolume
SMPSO AutoNSGAII
after applying four quality indicators show that the auto configured
NSGAII ▲ ▲ ▽ ▲ ▽ ▽ − − ▲ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ − ▽ − ▽ ▲ ▽ ▲ ▽ − ▽ ▽ ▽ ▽ NSGA-II clearly outperforms the other two algorithms in all the
SMPSO ▽▽▽▽− − ▽▲▽▲▽▲▽▽▽▽
IGD+
indicators.
SMPSO AutoNSGAII There are several lines for further research. We have conducted
NSGAII ▲ ▲ ▽ ▲ ▽ ▽ ▽ – ▲ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ ▽ – ▽ – ▽ ▲ ▽ ▲ ▽ ▲ ▽ ▽ ▽ ▽
SMPSO ▽▽▽▽– – ▽▲▽▲▽▲▽▽▽–
a case study, but others can be developed by considering different
scenarios involving the automatic configuration of NSGA-II, such
as fixing the stopping condition in a low number of evaluations
or using problems with more than two objectives. We plan also to
in the row has produced a best indicator value than the algorithm
extend the features of the jmetal-auto package to incorporate more
in the column with confidence, and “▽” is used when the algorithm
algorithmic components, in particular from indicator-based and
in the column is statistically better than the algorithm in the row.
decomposition-based multi-objective algorithms.
We can observe that statistical confidence has been found in most
of the results.
An issue that arises after the conducted experiment is, given that 6 ACKNOWLEDGEMENTS
the auto configured NSGA-II has yielded unsatisfactory indicator This work has been partially funded by Spanish Grants TIN2017-
values on three problems, to use irace to find a particular configu- 86049-R (Spanish Ministry of Education and Science) and P12-TIC-
ration for only those three instances and to see how it differs from 1519 (Plan Andaluz de Investigación, Desarrollo e Innovación).
the previously found configuration. We have proceeded then to Cristóbal Barba-González is supported by Grant BES-2015-072209
GECCO ’19 Companion, July 13–17, 2019, Prague, Czech Republic A.J. Nebro et. al.

Settings for auto-NSGA-II Settings of auto-NSGAII (DTLZ1, DTLZ3, WFG8)

–algorithmResult externalArchive –algorithmResult externalArchive
–populationSizeWithArchive 20 –populationSizeWithArchive 20
–offspringPopulationSize 100 –offspringPopulationSize 5
–variation crossoverAndMutationVariation –variation crossoverAndMutationVariation
–crossover BLX_ALPHA –crossover SBX
–crossoverProbability 0.9874 –crossoverProbability 0.9791
–crossoverRepairStrategy bounds –crossoverRepairStrategy random
–blxAlphaCrossoverAlphaValue 0.59.6 –sbxDistributionIndexValue 5.0587
–mutation polynomial –mutation uniform
–mutationProbability 0.0015 –mutationProbability 0.0463
–mutationRepairStrategy random –mutationRepairStrategy random
–polynomialMutationDistributionIndex 158.05 –uniformMutationPerturbation 0.2307
–selection tournament –selection tournament
–selectionTournamentSize 9 –selectionTournamentSize 4

Table 7: Settings for auto-NSGAII: first experiment (left) and second experiment (right).

Additive epsilon NSGAII SMPSO AutoNSGAII AutoNSGAIIb

DTLZ1 3.53e − 021.5e −01 6.30e − 035.5e −04 3.08e + 011.7e +01 7.06e − 032.4e −03
DTLZ3 1.00e + 016.4e +00 5.97e − 033.5e −01 1.05e + 023.6e +01 2.87e − 022.6e −02
WFG8 1.68e − 011.0e −01 1.69e − 011.7e −02 2.45e − 011.3e −03 2.44e − 011.0e −01
Spread NSGAII SMPSO AutoNSGAII AutoNSGAIIb
DTLZ1 1.04e + 002.6e −01 6.83e − 021.4e −02 6.00e − 016.1e −03 6.28e − 021.8e −02
DTLZ3 9.25e − 011.3e −01 1.26e − 014.8e −01 6.04e − 011.0e −02 2.14e − 012.1e −01
WFG8 6.17e − 016.5e −02 7.43e − 013.5e −02 5.46e − 013.2e −02 5.33e − 015.5e −02
Hypervolume NSGAII SMPSO AutoNSGAII AutoNSGAIIb
DTLZ1 4.66e − 011.6e −01 4.94e − 011.9e −04 0.00e + 000.0e +00 4.93e − 013.9e −03
DTLZ3 0.00e + 000.0e +00 2.10e − 016.3e −02 0.00e + 000.0e +00 1.76e − 013.1e −02
WFG8 1.48e − 012.3e −02 1.48e − 011.0e −03 1.39e − 012.3e −03 1.46e − 013.4e −03
IGD+ NSGAII SMPSO AutoNSGAII AutoNSGAIIb
DTLZ1 1.93e − 029.0e −02 2.81e − 031.0e −04 4.31e + 012.1e +01 3.61e − 032.1e −03
DTLZ3 1.01e + 018.3e +00 2.09e − 031.6e −01 1.35e + 024.7e +01 2.78e − 022.5e −02
WFG8 4.24e − 022.3e −02 4.26e − 022.3e −03 5.80e − 023.0e −03 5.06e − 029.1e −03
Table 8: Median and Interquartile Range of the indicator values for the DTLZ1, DTL3, and WFG8 instances experiment (Au-
toNSGAIIb is NSGA-II auto tuned with the second configuration).

(Spanish Ministry of Economy and Competitiveness). José García- Spain, 4138–4325.

Nieto is the recipient of a Post-Doctoral fellowship of “Captación [9] J.J. Durillo, A.J. Nebro, F. Luna, B. Dorronsoro, and E. Alba. 2006. jMetal: a Java
framework for developing multi-objective optimization metaheuristics. Technical
de Talento para la Investigación” Plan Propio at Universidad de Report ITI-2006-10. Departamento de Lenguajes y Ciencias de la Computación,
Málaga. University of Málaga, E.T.S.I. Informática, Campus de Teatinos.
[10] S. Huband, L. Barone, R.L. While, and P. Hingston. 2005. A Scalable Multi-
objective Test Problem Toolkit. In Third International Conference on Evolutionary
REFERENCES MultiCriterion Optimization, EMO 2005 (Lecture Notes in Computer Science), C.A.
[1] L. C. T. Bezerra, M. López-Ibáñez, and T. Stützle. 2016. Automatic Component- Coello, A. Hernández, and E. Zitler (Eds.), Vol. 3410. Springer, Berlin, Germany,
Wise Design of Multiobjective Evolutionary Algorithms. IEEE Transactions on 280–295.
Evolutionary Computation 20, 3 (June 2016), 403–417. [11] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas
[2] Stefan Bleuler, Marco Laumanns, Lothar Thiele, and Eckart Zitzler. 2003. PISA Stützle, and Mauro Birattari. 2016. The irace package: Iterated Racing for Auto-
— A Platform and Programming Language Independent Interface for Search matic Algorithm Configuration. Operations Research Perspectives 3 (2016), 43–58.
Algorithms. In Evolutionary Multi-Criterion Optimization (EMO 2003) (Lecture https://doi.org/10.1016/j.orp.2016.09.002
Notes in Computer Science), Carlos M. Fonseca, Peter J. Fleming, Eckart Zitzler, [12] A.J. Nebro, J.J. Durillo, J. García-Nieto, C.A. Coello Coello, F. Luna, and E. Alba.
Kalyanmoy Deb, and Lothar Thiele (Eds.). Springer, Berlin, 494 – 508. 2009. SMPSO: A New PSO-based Metaheuristic for Multi-objective Optimization.
[3] C.A. Coello Coello, G.B. Lamont, and D.A. van Veldhuizen. 2007. Multi-Objective In 2009 IEEE Symposium on Computational Intelligence in Multicriteria Decision-
Optimization Using Evolutionary Algorithms. John Wiley & Sons, Inc. 2nd Ed., Making (MCDM 2009). IEEE Press, 66–73.
NY, USA. [13] A.J. Nebro, Juan J. Durillo, and M. Vergne. 2015. Redesigning the jMetal Multi-
[4] K. Deb. 2001. Multi-Objective Optimization Using Evolutionary Algorithms. John Objective Optimization Framework. In Proceedings of the Companion Publication
Wiley & Sons, New York, NY, USA. of the 2015 Annual Conference on Genetic and Evolutionary Computation (GECCO
[5] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A Fast and Elitist Multi- Companion ’15). ACM, New York, NY, USA, 1093–1100. https://doi.org/10.1145/
objective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Com- 2739482.2768462
putation 6, 2 (2002), 182–197. [14] E. Zitzler, M. Laumanns, and L. Thiele. 2001. SPEA2: Improving the Strength
[6] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. 2005. Scalable test problems Pareto Evolutionary Algorithm. In EUROGEN 2001. Evolutionary Methods for
for evolutionary multiobjective optimization. In Evolutionary Multiobjective Design, Optimization and Control with Applications to Industrial Problems, K. Gian-
Optimization. Theoretical Advances and Applications, Ajith Abraham, Lakhmi nakoglou, D. Tsahalis, J. Periaux, P. Papailou, and T. Fogarty (Eds.). International
Jain, and Robert Goldberg (Eds.). Springer, USA, 105–145. Center for Numerical Methods in Engineering, Athens, Greece, 95–100.
[7] J.J. Durillo and A.J. Nebro. 2011. jMetal: A Java framework for multi-objective [15] E. Zitzler and L. Thiele. 1999. Multiobjective evolutionary algorithms: a compara-
optimization. Advances in Engineering Software 42, 10 (2011), 760 – 771. https: tive case study and the strength Pareto approach. IEEE Transactions on Evolution-
//doi.org/10.1016/j.advengsoft.2011.05.014 ary Computation 3, 4 (Nov 1999), 257–271. https://doi.org/10.1109/4235.797969
[8] J.J. Durillo, A.J. Nebro, and E. Alba. 2010. The jMetal Framework for Multi-
Objective Optimization: Design and Architecture. In CEC 2010. IEEE, Barcelona,