Performance Assessment

• Dr. Sobia Tariq Javed

1
Performance Assessment
• Assessing the performance of a multi-objective optimization
algorithm (MOA) is critical to determine its effectiveness in
solving multi-objective problems (MOPs).
• the quantitative comparison of the performance of different
algorithms
• Since MOPs involve multiple conflicting objectives, a good
algorithm should find a well-distributed, diverse, and optimal set
of solutions along the Pareto Front.

• Question: How to evaluate the quality of approximation sets?

The Need for Quality Indicators
• Ideally: quality indicators
allow both types of
statements
Quality
Measures
(Indicators)
• The performance of multi-
objective algorithms is
measured by assessing the
quality of the obtained
Pareto front approximation.
• The quality of the solutions
depends on
• Convergence to the
Pareto-optimal front
• Diversity of solutions
along the entire front
Generational Distance (GD)
• Introduced by Van Veldhuizen and Lamont, GD measures how far
the solutions in the obtained Pareto front approximation are
from the true Pareto-optimal front. It is defined as:

• where:
• n is the number of solutions in the approximation.
• di is the Euclidean distance (measured in the objective space) between
each solution in the approximation and its nearest point in the true Pareto
front.
Generational Distance (GD)
• A GD indicates better convergence, meaning the obtained
solutions are closer to the true Pareto-optimal front.
• A GD value of 0 signifies perfect convergence, where all solutions
lie exactly on the Pareto front.
Exercise-Generational Distance (GD)
• Compute GD
 • Since GD is relatively low, the approximate Pareto front
has a reasonable level of convergence to the true Pareto
Exercise- front, but there is still room for improvement.

Generational
Distance (GD)
Drawbacks of GD
• Ignores Diversity
• GD only measures how close the obtained solutions are to the true Pareto-
optimal front but does not consider how well they are distributed along the front.
• A solution set that is clustered in one region of the Pareto front can have a low
GD but still fail to represent the entire front properly.
• Sensitive to Outliers
• If a few solutions are very close to the Pareto front, GD may give a misleadingly
low value, even if many other solutions are poorly converged.
• Scaling Issues
• GD is dependent on the number of solutions in the approximation set. Larger
solution sets generally result in lower GD values, making it difficult to compare
across different algorithms with different population sizes.
Drawbacks of GD
• Does Not Differentiate Between Good and Poor Distributions
• Two solution sets with the same GD can have very different distributions;
one may be well-spread across the Pareto front, while the other may be
concentrated in only a small region.
• Requires Knowledge of the True Pareto Front
• GD requires the true Pareto front (or an accurate reference front) for
comparison, which is often unknown or computationally expensive to
determine in real-world problems.
Inverted Generational Distance (IGD)
• It is a variant of the Generational Distance.
• It measures both Convergence and Diversity.
• It measures the distances between each solution composing the optimal
Pareto front and the computed approximation and it is defined as:

• where
• |PF*| is the number of solutions in the true optimal Pareto front
• dj is the Euclidean distance between the j-th point on the true Pareto front and the
closest point in the approximation set.
• A lower IGD value indicates better convergence and diversity. A value of IGD
= 0 indicates that all the generated elements are in the Pareto front.
Exercise-IGD
• Compute IGD
Exercise-IGD
• Compute Euclidean Distance for Each p*
Exercise-IGD
• Compute Euclidean Distance for Each p*
Exercise-IGD
• Compute Euclidean Distance for Each p*
Exercise-IGD
• Compute Euclidean Distance for Each p*
Exercise-IGD
• Compute IGD
Drawbacks of IGD
1. IGD Requires a Well-Defined True Pareto Front
• IGD assumes that the true Pareto front is known and well-defined, which is not always the
case in real-world problems.
• In problems where the Pareto front is unknown or hard to compute, IGD cannot be used
effectively.
• Example: In a dynamic or uncertain environment where the Pareto front changes over time,
IGD may not be a reliable metric.

2. Computational Cost for Large-Scale Problems

• IGD requires computing the minimum distance for each true Pareto front point to all
approximate front points, making it computationally expensive for large-scale problems.
• The complexity is O(m × n) where:
• m is the number of true Pareto front points.
• n is the number of approximate front points.
• Example: If P∗ and PA both have thousands of points, IGD calculation becomes slow and
inefficient.
Drawbacks of IGD
3. IGD is Less Effective for Many-Objective Optimization
(MaOPs)
• In many-objective optimization (i.e., problems with more than 3
objectives), IGD often fails to distinguish between good and bad solutions
because the Pareto front is high-dimensional and distances become less
meaningful. This is known as the "curse of dimensionality."
• Example: In a 10-objective problem, distances become less informative,
and IGD may not reflect the true quality of solutions.
HyperVolume (HV)
• Hypervolume (HV) is a widely used
metric in multi-objective optimization to
evaluate the quality of a set of solutions
(also called the Pareto front).
• It measures the size of the space
(volume) covered by the solutions in the
objective space, relative to a reference
point.
Hypervolume enclosed by a set if 4 non-
• A larger HV indicates better diversity dominated solutions (A, B, C, and D) in a bi-
and convergence of the solutions. objective minimization problem
How HV works
• Each solution in the Pareto front is treated as a point in multi-
dimensional space.
• A reference point, often set worse than the worst objective
values, is defined.
• All points are then normalized
• HV is the volume of the region dominated by the Pareto front
and bounded by the reference point.
Exercise
• Consider a two-objective optimization problem where we
minimize both objectives:

• Solution A: (2, 5)
• Solution B: (3, 4)
• Reference Point: (5, 6)
Exercise
• Normalize the Objective Values:
Exercise
• Solution A: (2, 5)
• Solution B: (3, 4)
• Reference Point: (5, 6)

• Min f1 = 2, max f1= 5

• Min f2 = 4, max f2= 6
 Solution A: (0, 0.5)
Solution B: (0.33,0)
Exercise Reference Point: (1, 1) 1-0 =1

• Area dominated by all Reference

solutions relative to the Point
reference point. 1-0.5
= 0.5
• HVA = 0.5 x 1 = 0.5
• HVB = 0.5 x 0.67 = 0.335 A
• Total Area = HVA + HVB=
0.5 + 0.335 = 0.835 0.5- 0
= 0.5

B
1-0.33
= 0.67
Drawback
• Computational Complexity:
• HV calculation becomes computationally expensive as the number of objectives
and solutions increases. In higher dimensions, it’s especially hard to compute
efficiently.

• Sensitive to Reference Point:

• The choice of the reference point significantly affects the HV value. A poorly
chosen reference point can distort the results or misrepresent solution quality.

• Difficult Interpretation in High Dimensions:

• As the number of objectives grows, visualizing and interpreting the hypervolume
becomes increasingly difficult, even when the computation is possible.
Spread
• Spread (Δ) is a metric that
measures how evenly
distributed and diverse the
solutions are along the Pareto
front in MOP.
• The formula for Spread (Δ) is:
Spread The formula for Spread (Δ) is:

• The ideal value of Spread (Δ) is 0,

meaning the solutions are
perfectly spaced along the front.
Higher values of Δ indicate more
uneven or poorly distributed
solutions.
Exercise - Spread
• A two-objective minimization problem, where the solutions on
the Pareto front are:
S1(1,8), S2(2,6), S3(3,5), S4(5,3) with Extreme points (0,9) and (6,0)

• S1 and S4 are closest to the extreme points.

Exercise - Spread

• Mean

• Deviation from Mean

•
Interpretation:
A spread value of 0.549 shows that the solutions
are not evenly spaced, and there’s some
clustering along the Pareto front. A lower Δ value
would indicate a more balanced and diverse
distribution of solutions.
Drawback
• Sensitive to Extreme Solutions:
• The calculation depends heavily on extreme solutions (first and last points). If those points are missing
or poorly placed, the spread measure becomes less reliable.

• Assumes Ordered Solutions:

• Spread requires solutions to be ordered along the front, which can be non-trivial in higher dimensions
(3+ objectives). In complex spaces, ordering solutions consistently becomes computationally
challenging.

• Limited in High Dimensions:

• Spread works best in 2D or 3D, but as the number of objectives increases, measuring distances and
maintaining a good sense of spread becomes harder and less intuitive.

• Scale Sensitivity:
• Spread can be influenced by the scale of different objective functions. If one objective has a much larger
range than others, it can distort distance calculations and spread values.
Generalized Spread
• The previous indicator (Spread) is based on calculating the distance
between two consecutive solutions, which works only for 2-
objective problems.
• It is extended by computing the distance from a given point to its
nearest neighbor.

• Exercise
• 3-objective minimization problem with the following Pareto front
solutions
• S1(1,8,3), S2(2,6,2), S3(3,5,1), S4(5,3,0) with Extreme points (7,0,0),
(0,7,0) and (0,0,7)
Coverage
• Coverage (Zitzler et al. (2000)): for a pair (A,B) of
approximation sets the fraction of solutions in B
that are weakly dominated by one or more
solutions in A.
• The coverage measure is calculated by the
following equation:
Coverage
• a ≤ b: a dominates b
Coverage
• Find Coverage for Maximization Problem

• C(A,B) = (1+1+0)/3 = 2/3 = 0.667

• C(B,A) = (0+1)/2 = ½ =0.5
References

• Carlos M. Fonseca, Joshua D. Knowles, Lothar Thiele, Eckart

Zitzler: A Tutorial on the Performance Assessment of
Stochastic Multi-objective Optimizers.
• Juan J et. al, jMetal: A Java Framework for multi-objective
optimization. In Advances in Engineering Software, 2011.
• Zitzler, E. et. al, Comparison of multiobjective evolutionary
algorithms: Empirical results. Evolutionary Computation, 8(2),
(2000).