![Page 1: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/1.jpg)
"Tha
les
con
f ide
ntia
l. A
ll rig
hts
r ese
rved
"
Research & Technology
Multi-objective meta-parameter tuning for mono-objective stochastic metaheuristics
Johann DréoTHALES Research & Technology
![Page 2: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/2.jpg)
2
Introduction
Multi-objective method
Parameter tuning
Stochastic metaheuristics
Performance profiles
http://www.flickr.com/photos/k23/2792398403/
Dreo & Siarry, 2004
![Page 3: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/3.jpg)
3
Stochastic metaheuristics
![Page 4: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/4.jpg)
4
Examples of stochastic metaheuristics
![Page 5: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/5.jpg)
5
Parameter setting
![Page 6: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/6.jpg)
6
Meta-parameter tuning
![Page 7: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/7.jpg)
7
As a mono-objective problem
Parameter setting:Improve performance
http://www.flickr.com/photos/sigfrid/223626315/
![Page 8: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/8.jpg)
8
As a multi-objective problem
Parameter setting:
What is performance ?
→ multi-objective problem
http://www.flickr.com/photos/jesusdq/345379863/
![Page 9: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/9.jpg)
9
Multi-objective problem
Performance ? Precision
Speed
Robustness Precision
Speed
Stability (← benchmark)
http://www.flickr.com/photos/matthewfch/1688409628/
![Page 10: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/10.jpg)
10
Multi-objective problem
Performance ? Precision
Speed
Robustness Precision
Speed
Stability (← benchmark)
![Page 11: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/11.jpg)
11
Meta-parameter tuning
Mono-objective problem
Stochastic metaheuristic
![Page 12: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/12.jpg)
12
Multi-objective parameter tuning problem
Meta-parameter tuning
Mono-objective problem
Stochastic metaheuristic
![Page 13: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/13.jpg)
13
Multi-objective parameter tuning problem
Meta-parameter tuning
Mono-objective problem
Stochastic metaheuristic
Meta-optimizer
![Page 14: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/14.jpg)
14
Complexity
Multi-objective parameter tuning problem
Mono-objective problem
Stochastic metaheuristic
Meta-optimizer
Difficult
Easier1 time
![Page 15: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/15.jpg)
15
Methodology
Speed / PrecisionMedian estimation
Mono-objective problem
Stochastic metaheuristic
NSGA-2
![Page 16: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/16.jpg)
16
Methodology
Speed / PrecisionMedian estimation
Mono-objective problem
Stochastic metaheuristic
NSGA-2
![Page 17: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/17.jpg)
17
Results plots
Speed
Precision
Performance profile / front
![Page 18: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/18.jpg)
18
Some results
![Page 19: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/19.jpg)
19
Example
2 continuous EDA (CEDA, CHEDA)
Sampling density parameter
Rosenbrock, 2 dimensions
Median estimated with 10 runs
10 000 max eval.
NSGA-2
20 iter., 50 indiv.
10 runs
3 days computation
+ Nelder-Mead Search
![Page 20: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/20.jpg)
20
Example
+ simulated annealing
stable temperature parameter
Rosenbrock, 2 dimensions
Median estimated with 10 runs
10 000 max eval.
NSGA-2
20 iter., 50 indiv.
10 runs
1 day computation
![Page 21: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/21.jpg)
21
Example
+ genetic algorithm
population parameter
Rosenbrock, 2 dimensions
Median estimated with 10 runs
10 000 max eval.
NSGA-2
20 iter., 50 indiv.
10 runs
1 day computation
![Page 22: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/22.jpg)
22
SA
JGEN
CEDA
CHEDA
Speed Precision
![Page 23: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/23.jpg)
23
Behaviour exploration
Speed Precision
Genetic algorithm
Population size
![Page 24: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/24.jpg)
24
Performance front
Temporal planner, ''Divide & Evolve > CPT'', version ''GOAL''
2 mutation parameters
IPC ''rovers'' problem, instance 06
Median estimated with 10 runs
NSGA-2
10 iter., 5 indiv.
30 runs
1 week computation for 1 run
![Page 25: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/25.jpg)
25
Performance front in Parameters space
Speed
Precision
M1 M2
![Page 26: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/26.jpg)
26
Previous parameters settings
![Page 27: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/27.jpg)
27
Conclusion
![Page 28: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/28.jpg)
28
Drawbacks
Computation cost
Stochastic M.-O. algo. → supplementary bias
http://www.flickr.com/photos/orvaratli/2690949652/
![Page 29: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/29.jpg)
29
Drawbacks
Computation cost
Stochastic M.-O. algo. → supplementary bias
Valid only for: Algorithm implementation
Problem instance
Stopping criterion Error Time t steps, improvement < ε
http://www.flickr.com/photos/orvaratli/2690949652/
![Page 30: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/30.jpg)
30
Drawbacks
Computation cost
Stochastic M.-O. algo. → supplementary bias
Valid only for: Algorithm implementation
Problem instance
Stopping criterion Error Time t steps, improvement < ε
Fronts often convex→ aggregations?
No benchmarking
http://www.flickr.com/photos/orvaratli/2690949652/
![Page 31: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/31.jpg)
31
Advantages
Performance profiles Objectives space
Parameters space
Quantification of expert knowledge
![Page 32: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/32.jpg)
32
Advantages
Performance profiles Objectives space
Parameters space
Quantification of expert knowledge
Automatic parameter tuning One step before use
N parameters → 1 parameter
More degrees of freedom
![Page 33: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/33.jpg)
33
Advantages
Performance profiles Objectives space
Parameters space
Quantification of expert knowledge
Automatic parameter tuning One step before use
N parameters → 1 parameter
More degrees of freedom
Algorithms comparison Statistical tests more meaningful
![Page 34: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/34.jpg)
34
Advantages
Performance profiles Objectives space
Parameters space
Quantification of expert knowledge
Automatic parameter tuning One step before use
N parameters → 1 parameter
More degrees of freedom
Algorithms comparison Statistical tests more meaningful
Behaviour understanding
![Page 35: Multi-criteria meta-parameter tuning for mono-objective stochastic metaheuristics](https://reader034.vdocuments.site/reader034/viewer/2022042614/55a1986c1a28abe3748b45bc/html5/thumbnails/35.jpg)
35
Perspectives
Include robustness
Include dispersion estimation
Include benchmarking
Multi-objective SPO, F-Race
Regressions in parameters space Performances / parameters
Behaviour models?
Links? Fitness Landscape /
Performance profiles
Run time distribution
Taillard's significance plots
...
http://www.flickr.com/photos/colourcrazy/2065575762/