algorithm configuration: a hands-on tutorial · 2016. 2. 13. · algorithm con guration: a hands-on...
TRANSCRIPT
![Page 1: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/1.jpg)
Algorithm Configuration:A Hands-on Tutorial
Frank Hutter Marius Lindauer
University of Freiburg
AAAI 2016, Phoenix, USA
![Page 2: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/2.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 3: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/3.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 4: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/4.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 5: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/5.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 6: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/6.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 7: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/7.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 8: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/8.jpg)
What is this tutorial about?
Algorithm configuration in a nutshell: optimization of free parametersWhat kinds of parameters? any that you would otherwise tune yourself (&more)
Examples of free parameters in various subfields of AI
Tree search (in particular for SAT): pre-processing, branchingheuristics, clause learning & deletion, restarts, data structures, ...
Local search: neighbourhoods, perturbations, tabu length, annealing...
Genetic algorithms: population size, mating scheme, crossoveroperators, mutation rate, local improvement stages, hybridizations, ...
Machine Learning: pre-processing, regularization (type & strength),minibatch size, learning rate schedules, optimizer & its parameters, ...
Deep learning (in addition): #layers (& layer types), #units/layer,dropout constants, weight initialization and decay, pre-training, ...
and many more ...
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 2
![Page 9: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/9.jpg)
Hands-on!
Focus on basics
Less material, more in-depth
Target audience: focus on beginners
No special background assumed
Please ask questions
All literature references are hyperlinks
Goal: you can use algorithm configuration in your research
All demos use the virtual machine (VM) we distributed
If you downloaded the VM you can follow along live!
How many of you downloaded the VM?
- Don’t try now: it’s 3GB and you can follow what we do on the screen
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 3
![Page 10: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/10.jpg)
Hands-on!
Focus on basics
Less material, more in-depth
Target audience: focus on beginners
No special background assumed
Please ask questions
All literature references are hyperlinks
Goal: you can use algorithm configuration in your research
All demos use the virtual machine (VM) we distributed
If you downloaded the VM you can follow along live!
How many of you downloaded the VM?
- Don’t try now: it’s 3GB and you can follow what we do on the screen
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 3
![Page 11: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/11.jpg)
Hands-on!
Focus on basics
Less material, more in-depth
Target audience: focus on beginners
No special background assumed
Please ask questions
All literature references are hyperlinks
Goal: you can use algorithm configuration in your research
All demos use the virtual machine (VM) we distributed
If you downloaded the VM you can follow along live!
How many of you downloaded the VM?
- Don’t try now: it’s 3GB and you can follow what we do on the screen
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 3
![Page 12: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/12.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 4
![Page 13: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/13.jpg)
Outline
1 The Algorithm Configuration ProblemProblem StatementMotivation: a Few Success StoriesOverview of Methods
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 5
![Page 14: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/14.jpg)
Outline
1 The Algorithm Configuration ProblemProblem StatementMotivation: a Few Success StoriesOverview of Methods
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 6
![Page 15: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/15.jpg)
Algorithm Parameters
Parameter Types
Continuous, integer, ordinal
Categorical: finite domain, unordered, e.g., {apple, tomato, pepper}
Parameter space has structure
E.g., parameter θ2 of heuristic H is only active if H is used (θ1 = H)
In this case, we say θ2 is a conditional parameter with parent θ1
Sometimes, some combinations of parameter settings are forbiddene.g., the combination of θ3 = 1 and θ4 = 1 is forbidden
Parameters give rise to a structured space of configurations
Many configurations (e.g., SAT solver lingeling with 10947 )
Configurations often yield qualitatively different behaviour
→ Algorithm Configuration (as opposed to “parameter tuning”)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 7
![Page 16: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/16.jpg)
Algorithm Parameters
Parameter Types
Continuous, integer, ordinal
Categorical: finite domain, unordered, e.g., {apple, tomato, pepper}
Parameter space has structure
E.g., parameter θ2 of heuristic H is only active if H is used (θ1 = H)
In this case, we say θ2 is a conditional parameter with parent θ1
Sometimes, some combinations of parameter settings are forbiddene.g., the combination of θ3 = 1 and θ4 = 1 is forbidden
Parameters give rise to a structured space of configurations
Many configurations (e.g., SAT solver lingeling with 10947 )
Configurations often yield qualitatively different behaviour
→ Algorithm Configuration (as opposed to “parameter tuning”)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 7
![Page 17: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/17.jpg)
Algorithm Parameters
Parameter Types
Continuous, integer, ordinal
Categorical: finite domain, unordered, e.g., {apple, tomato, pepper}
Parameter space has structure
E.g., parameter θ2 of heuristic H is only active if H is used (θ1 = H)
In this case, we say θ2 is a conditional parameter with parent θ1
Sometimes, some combinations of parameter settings are forbiddene.g., the combination of θ3 = 1 and θ4 = 1 is forbidden
Parameters give rise to a structured space of configurations
Many configurations (e.g., SAT solver lingeling with 10947 )
Configurations often yield qualitatively different behaviour
→ Algorithm Configuration (as opposed to “parameter tuning”)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 7
![Page 18: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/18.jpg)
Algorithm Configuration Visualized
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 8
![Page 19: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/19.jpg)
Algorithm Configuration – in More Detail
Configuration Task
Instances IAlgorithm A andits Configuration
Space Θ
Select θ ∈ Θand π ∈ I
Run A(θ) on π tomeasure m(θ, π)
Returns BestConfiguration θ
Return Cost
Definition: algorithm configuration
Given:
a parameterized algorithm A with possible parameter settings Θ;
a distribution D over problem instances with domain I; and
a cost metric m : Θ× I → R,
Find: θ∗ ∈ arg minθ∈ΘEπ∼D(m(θ, π)).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 9
![Page 20: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/20.jpg)
Algorithm Configuration – in More Detail
Configuration Task
Instances IAlgorithm A andits Configuration
Space Θ
Select θ ∈ Θand π ∈ I
Run A(θ) on π tomeasure m(θ, π)
Returns BestConfiguration θ
Return Cost
Definition: algorithm configuration
Given:
a parameterized algorithm A with possible parameter settings Θ;
a distribution D over problem instances with domain I; and
a cost metric m : Θ× I → R,
Find: θ∗ ∈ arg minθ∈ΘEπ∼D(m(θ, π)).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 9
![Page 21: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/21.jpg)
Algorithm Configuration – in More Detail
Configuration Task
Instances IAlgorithm A andits Configuration
Space Θ
Select θ ∈ Θand π ∈ I
Run A(θ) on π tomeasure m(θ, π)
Returns BestConfiguration θ
Return Cost
Definition: algorithm configuration
Given:
a parameterized algorithm A with possible parameter settings Θ;
a distribution D over problem instances with domain I; and
a cost metric m : Θ× I → R,
Find: θ∗ ∈ arg minθ∈ΘEπ∼D(m(θ, π)).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 9
![Page 22: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/22.jpg)
Algorithm Configuration – Full Formal Definition
Definition: algorithm configuration
An instance of the algorithm configuration problem is a 5-tuple(A,Θ,D, κ,m) where:
A is a parameterized algorithm;
Θ is the parameter configuration space of A;
D is a distribution over problem instances with domain I;
κ <∞ is a cutoff time, after which each run of A will be terminatedif still running
m : Θ× I → R is a function that measures the observed cost ofrunning A(θ) on an instance π ∈ I with cutoff time κ
The cost of a candidate solution θ ∈ Θ is c(θ) = Eπ∼D(m(θ, π)).The goal is to find θ∗ ∈ arg minθ∈Θ c(θ).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 10
![Page 23: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/23.jpg)
Algorithm Configuration – Full Formal Definition
Definition: algorithm configuration
An instance of the algorithm configuration problem is a 5-tuple(A,Θ,D, κ,m) where:
A is a parameterized algorithm;
Θ is the parameter configuration space of A;
D is a distribution over problem instances with domain I;
κ <∞ is a cutoff time, after which each run of A will be terminatedif still running
m : Θ× I → R is a function that measures the observed cost ofrunning A(θ) on an instance π ∈ I with cutoff time κ
The cost of a candidate solution θ ∈ Θ is c(θ) = Eπ∼D(m(θ, π)).The goal is to find θ∗ ∈ arg minθ∈Θ c(θ).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 10
![Page 24: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/24.jpg)
Algorithm Configuration – Full Formal Definition
Definition: algorithm configuration
An instance of the algorithm configuration problem is a 5-tuple(A,Θ,D, κ,m) where:
A is a parameterized algorithm;
Θ is the parameter configuration space of A;
D is a distribution over problem instances with domain I;
κ <∞ is a cutoff time, after which each run of A will be terminatedif still running
m : Θ× I → R is a function that measures the observed cost ofrunning A(θ) on an instance π ∈ I with cutoff time κ
The cost of a candidate solution θ ∈ Θ is c(θ) = Eπ∼D(m(θ, π)).The goal is to find θ∗ ∈ arg minθ∈Θ c(θ).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 10
![Page 25: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/25.jpg)
Algorithm Configuration – Full Formal Definition
Definition: algorithm configuration
An instance of the algorithm configuration problem is a 5-tuple(A,Θ,D, κ,m) where:
A is a parameterized algorithm;
Θ is the parameter configuration space of A;
D is a distribution over problem instances with domain I;
κ <∞ is a cutoff time, after which each run of A will be terminatedif still running
m : Θ× I → R is a function that measures the observed cost ofrunning A(θ) on an instance π ∈ I with cutoff time κ
The cost of a candidate solution θ ∈ Θ is c(θ) = Eπ∼D(m(θ, π)).The goal is to find θ∗ ∈ arg minθ∈Θ c(θ).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 10
![Page 26: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/26.jpg)
Distribution vs. Set of Instances
Find: θ∗ ∈ arg minθ∈ΘEπ∼D(m(θ, π)).
Special case: distribution with finite support
We often only have N instances from a given application
In that case: split N instances into training and test set
Find θ∗ ∈ arg minθ∈Θ1
Ntrain
∑Ntraini=1 (m(θ, πi)).
Evaluation on new test instances
Same approach as in machine learning
We configure algorithms on the training instances
We only use test instances to assess generalization performance
→ unbiased estimate of generalization performance for unseen instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 11
![Page 27: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/27.jpg)
Distribution vs. Set of Instances
Find: θ∗ ∈ arg minθ∈ΘEπ∼D(m(θ, π)).
Special case: distribution with finite support
We often only have N instances from a given application
In that case: split N instances into training and test set
Find θ∗ ∈ arg minθ∈Θ1
Ntrain
∑Ntraini=1 (m(θ, πi)).
Evaluation on new test instances
Same approach as in machine learning
We configure algorithms on the training instances
We only use test instances to assess generalization performance
→ unbiased estimate of generalization performance for unseen instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 11
![Page 28: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/28.jpg)
Our Two Running Examples
Minimize the runtime of a SAT solver for a benchmark set
Optimize on training set:
θ∗ ∈ arg minθ∈Θ1
Ntrain
∑Ntraini=1 (m(θ, πi))
Minimize K-fold cross-validation error of a machine learning algorithm
A cross-validation fold k plays the role of an instanceθ∗ ∈ arg minθ∈Θ
1K
∑Kk=1(m(θ, k))
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 12
![Page 29: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/29.jpg)
Our Two Running Examples
Minimize the runtime of a SAT solver for a benchmark set
Optimize on training set:
θ∗ ∈ arg minθ∈Θ1
Ntrain
∑Ntraini=1 (m(θ, πi))
Minimize K-fold cross-validation error of a machine learning algorithm
A cross-validation fold k plays the role of an instanceθ∗ ∈ arg minθ∈Θ
1K
∑Kk=1(m(θ, k))
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 12
![Page 30: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/30.jpg)
Some use cases for Algorithm Configuration
Automatically customize versatile algorithms
New application domains
Optimize speed, accuracy, memory, energy consumption, latency, ...
All of these: possible without intimate knowledge of algorithm
Trade costly human time for cheap CPU time
Empirical studies, evaluations & comparisons of algorithms
Fairness: same tuning protocol for all algorithms
Reproducibility of protocol
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 13
![Page 31: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/31.jpg)
Some use cases for Algorithm Configuration
Automatically customize versatile algorithms
New application domains
Optimize speed, accuracy, memory, energy consumption, latency, ...
All of these: possible without intimate knowledge of algorithm
Trade costly human time for cheap CPU time
Empirical studies, evaluations & comparisons of algorithms
Fairness: same tuning protocol for all algorithms
Reproducibility of protocol
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 13
![Page 32: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/32.jpg)
Some use cases for Algorithm Configuration
Automatically customize versatile algorithms
New application domains
Optimize speed, accuracy, memory, energy consumption, latency, ...
All of these: possible without intimate knowledge of algorithm
Trade costly human time for cheap CPU time
Empirical studies, evaluations & comparisons of algorithms
Fairness: same tuning protocol for all algorithms
Reproducibility of protocol
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 13
![Page 33: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/33.jpg)
AC in Context: Computer-Aided Algorithm Design
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 14
![Page 34: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/34.jpg)
AC in Context: Computer-Aided Algorithm Design
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 14
![Page 35: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/35.jpg)
Outline
1 The Algorithm Configuration ProblemProblem StatementMotivation: a Few Success StoriesOverview of Methods
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 15
![Page 36: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/36.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
SAT (propositional satisfiability problem)
Prototypical NP-hard problem
Interesting theoretically and in practical applications
Formal verification
Software verification [Babic & Hu; CAV ’07]
Hardware verification (Bounded model checking) [Zarpas; SAT ’05]
Tree search solver for SAT-based verification
SPEAR, developed by Domagoj Babic at UBC
26 parameters, 8.34× 1017 configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 16
![Page 37: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/37.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
SAT (propositional satisfiability problem)
Prototypical NP-hard problem
Interesting theoretically and in practical applications
Formal verification
Software verification [Babic & Hu; CAV ’07]
Hardware verification (Bounded model checking) [Zarpas; SAT ’05]
Tree search solver for SAT-based verification
SPEAR, developed by Domagoj Babic at UBC
26 parameters, 8.34× 1017 configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 16
![Page 38: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/38.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
SAT (propositional satisfiability problem)
Prototypical NP-hard problem
Interesting theoretically and in practical applications
Formal verification
Software verification [Babic & Hu; CAV ’07]
Hardware verification (Bounded model checking) [Zarpas; SAT ’05]
Tree search solver for SAT-based verification
SPEAR, developed by Domagoj Babic at UBC
26 parameters, 8.34× 1017 configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 16
![Page 39: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/39.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
Ran FocusedILS, 2 days × 10 machines– On a training set from each benchmark
Compared to manually-engineered default– 1 week of performance tuning– Competitive with the state of the art– Comparison on unseen test instances
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
IBM
−B
MC
(s)
4.5-fold speedupon hardware verification
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
SW
V (
s)
500-fold speedup won categoryQF BV in 2007 SMT competition
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 17
![Page 40: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/40.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
Ran FocusedILS, 2 days × 10 machines– On a training set from each benchmark
Compared to manually-engineered default– 1 week of performance tuning– Competitive with the state of the art– Comparison on unseen test instances
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
IBM
−B
MC
(s)
4.5-fold speedupon hardware verification
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
SW
V (
s)
500-fold speedup won categoryQF BV in 2007 SMT competition
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 17
![Page 41: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/41.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
Ran FocusedILS, 2 days × 10 machines– On a training set from each benchmark
Compared to manually-engineered default– 1 week of performance tuning– Competitive with the state of the art– Comparison on unseen test instances
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
IBM
−B
MC
(s)
4.5-fold speedupon hardware verification
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
SW
V (
s)
500-fold speedup won categoryQF BV in 2007 SMT competition
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 17
![Page 42: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/42.jpg)
Configuration of a SAT Solver for Verification [Hutter et al, 2007]
Ran FocusedILS, 2 days × 10 machines– On a training set from each benchmark
Compared to manually-engineered default– 1 week of performance tuning– Competitive with the state of the art– Comparison on unseen test instances
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
IBM
−B
MC
(s)
4.5-fold speedupon hardware verification
10−2
10−1
100
101
102
103
104
10−2
10−1
100
101
102
103
104
SPEAR, original default (s)
SP
EA
R, o
ptim
ized
for
SW
V (
s)
500-fold speedup won categoryQF BV in 2007 SMT competition
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 17
![Page 43: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/43.jpg)
Algorithm Configuration is Widely Applicable
Hard combinatorial problems
- SAT, MIP, TSP, AI planning,ASP, Time-tabling, ...
- UBC exam time-tabling since 2010
Game Theory: Kidney Exchange
Mobile Robotics
Monte Carlo Localization
Motion Capture
Machine Learning
- Automated Machine Learning- Deep Learning
Also popular in industry
Better performance
increased productivity
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 18
![Page 44: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/44.jpg)
Automated Machine Learning
Machine Learning is very successful in many applications.
But it still requires human machine learning experts to
- Preprocess the data- Select / engineer features- Select a model family- Optimize hyperparameters- Construct ensembles- ...
AutoML: taking the human expert out of the loop
Deep learning helps to automatically learn features
But it is even more sensitive to hyperparameters
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 19
![Page 45: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/45.jpg)
Automated Machine Learning
Machine Learning is very successful in many applications.
But it still requires human machine learning experts to
- Preprocess the data- Select / engineer features- Select a model family- Optimize hyperparameters- Construct ensembles- ...
AutoML: taking the human expert out of the loop
Deep learning helps to automatically learn features
But it is even more sensitive to hyperparameters
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 19
![Page 46: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/46.jpg)
Automated Machine Learning
Machine Learning is very successful in many applications.
But it still requires human machine learning experts to
- Preprocess the data- Select / engineer features- Select a model family- Optimize hyperparameters- Construct ensembles- ...
AutoML: taking the human expert out of the loop
Deep learning helps to automatically learn features
But it is even more sensitive to hyperparameters
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 19
![Page 47: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/47.jpg)
Automated Machine Learning
The AutoML approach introduced by Auto-WEKA [Thornton et al, 2013]
Expose the choices in a machine learning framework
- Algorithms, hyperparameters, preprocessors, ...
Optimize C/V performance using Bayesian optimization
Obtain a true push-button solution for machine learning
Extended recently in Auto-sklearn [Feurer et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 20
![Page 48: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/48.jpg)
Automated Machine Learning
The AutoML approach introduced by Auto-WEKA [Thornton et al, 2013]
Expose the choices in a machine learning framework
- Algorithms, hyperparameters, preprocessors, ...
Optimize C/V performance using Bayesian optimization
Obtain a true push-button solution for machine learning
Extended recently in Auto-sklearn [Feurer et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 20
![Page 49: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/49.jpg)
Auto-sklearn’s configuration space
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 21
![Page 50: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/50.jpg)
Hands-on: Auto-sklearn
In your virtual machine:
Run a linear SVM and Auto-sklearn
$ cd AC-Tutorial/auto-sklearn/
$ vim baseline_svc.py
$ python baseline_svc.py
$ vim autosklearn-example.py
$ python autosklearn-example.py
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 22
![Page 51: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/51.jpg)
Outline
1 The Algorithm Configuration ProblemProblem StatementMotivation: a Few Success StoriesOverview of Methods
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 23
![Page 52: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/52.jpg)
Challenges of Algorithm Configuration
Structured high-dimensional parameter space
Categorical vs. continuous parameters
Conditionals between parameters
Stochastic optimization
Randomized algorithms: optimization across various seeds
Distribution of benchmark instances (often wide range of hardness)
Subsumes so-called multi-armed bandit problem
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 24
![Page 53: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/53.jpg)
Challenges of Algorithm Configuration
Structured high-dimensional parameter space
Categorical vs. continuous parameters
Conditionals between parameters
Stochastic optimization
Randomized algorithms: optimization across various seeds
Distribution of benchmark instances (often wide range of hardness)
Subsumes so-called multi-armed bandit problem
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 24
![Page 54: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/54.jpg)
Component 1: Which Configuration to Choose?
For this component, we can consider a simpler problem:Blackbox function optimization: minθ∈Θ f(θ)
Only mode of interaction: query f(θ) at arbitrary θ ∈ Θ
Abstracts away the complexity of evaluating multiple instances
Θ is still a structured space
Mixed continuous/discreteConditional parameters
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 25
![Page 55: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/55.jpg)
Component 1: Which Configuration to Choose?
For this component, we can consider a simpler problem:Blackbox function optimization: minθ∈Θ f(θ)
Only mode of interaction: query f(θ) at arbitrary θ ∈ Θ
Abstracts away the complexity of evaluating multiple instances
Θ is still a structured space
Mixed continuous/discreteConditional parameters
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 25
![Page 56: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/56.jpg)
Component 1: Which Configuration to Evaluate?
Need to balance diversification and intensification
The extremes
- Random search- Gradient Descent
Stochastic local search (SLS)
Population-based methods
Model-based Optimization
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 26
![Page 57: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/57.jpg)
Component 2: How to Evaluate a Configuration?
Back to the general algorithm configuration problem
Distribution over problem instances with domain I;
Performance metric m : Θ× I → R
c(θ) = Eπ∼D(m(θ, π))
Simplest, suboptimal solution: use N runs for each evaluation
Treats the problem as a blackbox function optimization problem
Issue: how large to choose N?
- too small: overtuning- too large: every function evaluation is slow
General principle to strive for
Don’t waste time on bad configurations
Evaluate good configurations more thoroughly
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 27
![Page 58: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/58.jpg)
Component 2: How to Evaluate a Configuration?
Back to the general algorithm configuration problem
Distribution over problem instances with domain I;
Performance metric m : Θ× I → R
c(θ) = Eπ∼D(m(θ, π))
Simplest, suboptimal solution: use N runs for each evaluation
Treats the problem as a blackbox function optimization problem
Issue: how large to choose N?
- too small: overtuning- too large: every function evaluation is slow
General principle to strive for
Don’t waste time on bad configurations
Evaluate good configurations more thoroughly
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 27
![Page 59: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/59.jpg)
Component 2: How to Evaluate a Configuration?
Back to the general algorithm configuration problem
Distribution over problem instances with domain I;
Performance metric m : Θ× I → R
c(θ) = Eπ∼D(m(θ, π))
Simplest, suboptimal solution: use N runs for each evaluation
Treats the problem as a blackbox function optimization problem
Issue: how large to choose N?
- too small: overtuning- too large: every function evaluation is slow
General principle to strive for
Don’t waste time on bad configurations
Evaluate good configurations more thoroughly
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 27
![Page 60: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/60.jpg)
Racing algorithms: the general approach
Problem: which one of N candidate algorithms is best?
Start with empty set of runs for each algorithm
Iteratively:
Perform one run eachDiscard inferior candidatesE.g., as judged by a statistical test (e.g., F-race uses an F-test)
Stop when a single candidate remains or configuration budget expires
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 28
![Page 61: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/61.jpg)
Saving Time: Aggressive Racing
Race new configurations against the best known
- Discard poor new configurations quickly- No requirement for statistical domination- Evaluate best configurations with many runs
Search component should allow to return to configurations discardedbecause they were “unlucky”
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 29
![Page 62: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/62.jpg)
Saving Time: Aggressive Racing
Race new configurations against the best known
- Discard poor new configurations quickly- No requirement for statistical domination- Evaluate best configurations with many runs
Search component should allow to return to configurations discardedbecause they were “unlucky”
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 29
![Page 63: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/63.jpg)
Saving More Time: Adaptive Capping
When minimizing algorithm runtime,we can terminate runs for poor configurations θ′ early:
Is θ′ better than θ?
- Example:
Can terminate evaluation of θ′ once guaranteed to be worse than θ
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 30
![Page 64: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/64.jpg)
Saving More Time: Adaptive Capping
When minimizing algorithm runtime,we can terminate runs for poor configurations θ′ early:
Is θ′ better than θ?
- Example:
Can terminate evaluation of θ′ once guaranteed to be worse than θ
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 30
![Page 65: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/65.jpg)
General algorithm configuration systems
ParamILS [Hutter et al, 2007 & 2009]
Gender-based Genetic Algorithm (GGA) [Ansotegui et al, 2009]
Iterated F-Race [Lopez-Ibanez et al, 2011]
Sequential Model-based Algorithm Configuration (SMAC)[Hutter et al, since 2011]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 31
![Page 66: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/66.jpg)
The Baseline: Graduate Student Descent
Algorithm 1: Manual Greedy Algorithm Configuration
Start with some configuration θ
repeatModify a single parameterif results on benchmark set improve then
keep new configuration
until no more improvement possible (or “good enough”)
Manually-executed first-improvement local search
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 32
![Page 67: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/67.jpg)
The Baseline: Graduate Student Descent
Algorithm 1: Manual Greedy Algorithm Configuration
Start with some configuration θ
repeat
Modify a single parameter
if results on benchmark set improve thenkeep new configuration
until no more improvement possible (or “good enough”)
Manually-executed first-improvement local search
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 32
![Page 68: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/68.jpg)
The Baseline: Graduate Student Descent
Algorithm 1: Manual Greedy Algorithm Configuration
Start with some configuration θ
repeat
Modify a single parameterif results on benchmark set improve then
keep new configuration
until no more improvement possible (or “good enough”)
Manually-executed first-improvement local search
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 32
![Page 69: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/69.jpg)
The Baseline: Graduate Student Descent
Algorithm 1: Manual Greedy Algorithm Configuration
Start with some configuration θrepeat
Modify a single parameterif results on benchmark set improve then
keep new configuration
until no more improvement possible (or “good enough”)
Manually-executed first-improvement local search
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 32
![Page 70: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/70.jpg)
The Baseline: Graduate Student Descent
Algorithm 1: Manual Greedy Algorithm Configuration
Start with some configuration θrepeat
Modify a single parameterif results on benchmark set improve then
keep new configuration
until no more improvement possible (or “good enough”)
Manually-executed first-improvement local search
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 32
![Page 71: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/71.jpg)
Going Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 72: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/72.jpg)
Going Beyond Local Optima: Iterated Local Search
(Initialisation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 73: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/73.jpg)
Going Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 74: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/74.jpg)
Going Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 75: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/75.jpg)
Going Beyond Local Optima: Iterated Local Search
(Perturbation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 76: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/76.jpg)
Going Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 77: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/77.jpg)
Going Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 78: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/78.jpg)
Going Beyond Local Optima: Iterated Local Search
(Local Search)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 79: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/79.jpg)
Going Beyond Local Optima: Iterated Local Search
?
Selection (using Acceptance Criterion)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 80: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/80.jpg)
Going Beyond Local Optima: Iterated Local Search
(Perturbation)
Animation credit: Holger HoosHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 33
![Page 81: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/81.jpg)
The ParamILS Framework [Hutter et al, 2007 & 2009]
ParamILS = Iterated Local Search in parameter configuration space
Performs biased random walk over local optima
How to evaluate a configuration’s quality?
BasicILS(N): use N fixed instances
FocusedILS: increase # instances for good configurations over time
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 34
![Page 82: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/82.jpg)
The ParamILS Framework [Hutter et al, 2007 & 2009]
ParamILS = Iterated Local Search in parameter configuration space
Performs biased random walk over local optima
How to evaluate a configuration’s quality?
BasicILS(N): use N fixed instances
FocusedILS: increase # instances for good configurations over time
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 34
![Page 83: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/83.jpg)
The ParamILS Framework [Hutter et al, 2007 & 2009]
ParamILS = Iterated Local Search in parameter configuration space
Performs biased random walk over local optima
How to evaluate a configuration’s quality?
BasicILS(N): use N fixed instances
FocusedILS: increase # instances for good configurations over time
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 34
![Page 84: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/84.jpg)
The ParamILS Framework [Hutter et al, 2007 & 2009]
Advantages
Theoretically shown to converge
Often quickly finds local improvements over default(can exploit a good default)
Very randomized → almost k-fold speedup for k parallel runs
Disadvantages
Very randomized → unreliable when only run once for a short time
Can be slow to find the global optimum
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 35
![Page 85: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/85.jpg)
The ParamILS Framework [Hutter et al, 2007 & 2009]
Advantages
Theoretically shown to converge
Often quickly finds local improvements over default(can exploit a good default)
Very randomized → almost k-fold speedup for k parallel runs
Disadvantages
Very randomized → unreliable when only run once for a short time
Can be slow to find the global optimum
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 35
![Page 86: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/86.jpg)
GGA [Ansotegui et al, 2009]
Genetic algorithm for algorithm configuration
Genes = parameter values
Population: trades of exploration and exploitation
Use N instances to evaluate configurations in each generation
- Increase N in each generation: linearly from Nstart to Nend
Advantages
Easy to use parallel resources: evaluate several population members inparallel
Disadvantages
User has to specify #generations ahead of time
Not recommended for small budgets
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 36
![Page 87: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/87.jpg)
GGA [Ansotegui et al, 2009]
Genetic algorithm for algorithm configuration
Genes = parameter values
Population: trades of exploration and exploitation
Use N instances to evaluate configurations in each generation
- Increase N in each generation: linearly from Nstart to Nend
Advantages
Easy to use parallel resources: evaluate several population members inparallel
Disadvantages
User has to specify #generations ahead of time
Not recommended for small budgets
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 36
![Page 88: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/88.jpg)
GGA [Ansotegui et al, 2009]
Genetic algorithm for algorithm configuration
Genes = parameter values
Population: trades of exploration and exploitation
Use N instances to evaluate configurations in each generation
- Increase N in each generation: linearly from Nstart to Nend
Advantages
Easy to use parallel resources: evaluate several population members inparallel
Disadvantages
User has to specify #generations ahead of time
Not recommended for small budgets
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 36
![Page 89: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/89.jpg)
GGA [Ansotegui et al, 2009]
Genetic algorithm for algorithm configuration
Genes = parameter values
Population: trades of exploration and exploitation
Use N instances to evaluate configurations in each generation
- Increase N in each generation: linearly from Nstart to Nend
Advantages
Easy to use parallel resources: evaluate several population members inparallel
Disadvantages
User has to specify #generations ahead of time
Not recommended for small budgets
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 36
![Page 90: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/90.jpg)
Iterated F-race [Lopez-Ibanez et al, 2011]
Basic idea
Use F-Race as a building block
Iteratively sample configurations to race
Advantages
Can parallelize easily: runs of each racing iteration are independent
Well-supported software package (for the community that uses R)
Disadvantages
Does not support adaptive capping
The sampling of new configurations is not very strong for complexsearch spaces
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 37
![Page 91: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/91.jpg)
Iterated F-race [Lopez-Ibanez et al, 2011]
Basic idea
Use F-Race as a building block
Iteratively sample configurations to race
Advantages
Can parallelize easily: runs of each racing iteration are independent
Well-supported software package (for the community that uses R)
Disadvantages
Does not support adaptive capping
The sampling of new configurations is not very strong for complexsearch spaces
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 37
![Page 92: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/92.jpg)
Iterated F-race [Lopez-Ibanez et al, 2011]
Basic idea
Use F-Race as a building block
Iteratively sample configurations to race
Advantages
Can parallelize easily: runs of each racing iteration are independent
Well-supported software package (for the community that uses R)
Disadvantages
Does not support adaptive capping
The sampling of new configurations is not very strong for complexsearch spaces
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 37
![Page 93: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/93.jpg)
SMAC in a Nutshell [Hutter et al, since 2011]
SMAC = Sequential Model-based Algorithm Configuration
Use a predictive model of algorithm performance to guide the search
Combine this search strategy with aggressive racing & adaptivecapping
One SMAC iteration
Construct a model to predict performance
Use that model to select promising configurations
Compare each of the selected configurations against the best known
- Using a similar procedure as FocusedILS
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 38
![Page 94: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/94.jpg)
SMAC in a Nutshell [Hutter et al, since 2011]
SMAC = Sequential Model-based Algorithm Configuration
Use a predictive model of algorithm performance to guide the search
Combine this search strategy with aggressive racing & adaptivecapping
One SMAC iteration
Construct a model to predict performance
Use that model to select promising configurations
Compare each of the selected configurations against the best known
- Using a similar procedure as FocusedILS
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 38
![Page 95: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/95.jpg)
SMAC in a Nutshell [Hutter et al, since 2011]
SMAC = Sequential Model-based Algorithm Configuration
Use a predictive model of algorithm performance to guide the search
Combine this search strategy with aggressive racing & adaptivecapping
One SMAC iteration
Construct a model to predict performance
Use that model to select promising configurations
Compare each of the selected configurations against the best known
- Using a similar procedure as FocusedILS
Theorem
Let Θ be finite. Then, the probability that FocusedILS finds the trueoptimal parameter configuration θ∗ ∈ Θ approaches 1 as the number ofILS iterations goes to infinity.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 38
![Page 96: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/96.jpg)
Bayesian Optimization
General approach
Fit a probabilistic model to thecollected function samples 〈θ, f(θ)〉Use the model to guide optimization,trading off exploration vs exploitation
Popular approach in the statisticsliterature since [Mockus, 1978]
Efficient in # function evaluations
Works when objective is nonconvex,noisy, has unknown derivatives, etc
Recent convergence results[Srinivas et al, 2010; Bull 2011; de Freitas et
al, 2012; Kawaguchi et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 39
![Page 97: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/97.jpg)
Bayesian Optimization
General approach
Fit a probabilistic model to thecollected function samples 〈θ, f(θ)〉Use the model to guide optimization,trading off exploration vs exploitation
Popular approach in the statisticsliterature since [Mockus, 1978]
Efficient in # function evaluations
Works when objective is nonconvex,noisy, has unknown derivatives, etc
Recent convergence results[Srinivas et al, 2010; Bull 2011; de Freitas et
al, 2012; Kawaguchi et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 39
![Page 98: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/98.jpg)
Bayesian Optimization
General approach
Fit a probabilistic model to thecollected function samples 〈θ, f(θ)〉Use the model to guide optimization,trading off exploration vs exploitation
Popular approach in the statisticsliterature since [Mockus, 1978]
Efficient in # function evaluations
Works when objective is nonconvex,noisy, has unknown derivatives, etc
Recent convergence results[Srinivas et al, 2010; Bull 2011; de Freitas et
al, 2012; Kawaguchi et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 39
![Page 99: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/99.jpg)
Bayesian Optimization
General approach
Fit a probabilistic model to thecollected function samples 〈θ, f(θ)〉Use the model to guide optimization,trading off exploration vs exploitation
Popular approach in the statisticsliterature since [Mockus, 1978]
Efficient in # function evaluations
Works when objective is nonconvex,noisy, has unknown derivatives, etc
Recent convergence results[Srinivas et al, 2010; Bull 2011; de Freitas et
al, 2012; Kawaguchi et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 39
![Page 100: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/100.jpg)
Bayesian Optimization
General approach
Fit a probabilistic model to thecollected function samples 〈θ, f(θ)〉Use the model to guide optimization,trading off exploration vs exploitation
Popular approach in the statisticsliterature since [Mockus, 1978]
Efficient in # function evaluations
Works when objective is nonconvex,noisy, has unknown derivatives, etc
Recent convergence results[Srinivas et al, 2010; Bull 2011; de Freitas et
al, 2012; Kawaguchi et al, 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 39
![Page 101: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/101.jpg)
Powering SMAC: Empirical Performance Models
Empirical Performance Models
Given:
Configuration space Θ = Θ1 × · · · ×Θn
For each problem instance πi: xi, a vector of feature values
Observed algorithm runtime data: 〈(θi,xi, yi)〉Ni=1
Find: a mapping m : [θ,x] 7→ y predicting performance
Which type of regression model?
Rich literature on performance prediction(overview: [Hutter et al, AIJ 2014])
Here: we use a model m based on random forests
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 40
![Page 102: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/102.jpg)
Powering SMAC: Empirical Performance Models
Empirical Performance Models
Given:
Configuration space Θ = Θ1 × · · · ×Θn
For each problem instance πi: xi, a vector of feature values
Observed algorithm runtime data: 〈(θi,xi, yi)〉Ni=1
Find: a mapping m : [θ,x] 7→ y predicting performance
Which type of regression model?
Rich literature on performance prediction(overview: [Hutter et al, AIJ 2014])
Here: we use a model m based on random forests
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 40
![Page 103: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/103.jpg)
Fitting a Regression Tree
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 41
![Page 104: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/104.jpg)
Fitting a Regression Tree
In each internal node: only store split criterion used
In each leaf: store mean of responses
Prediction for a new data point: walk down the tree, return stored value
E.g. for (param1, feature2, param3) = (true, 4.7, red): 1.65
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 42
![Page 105: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/105.jpg)
Fitting a Regression Tree
In each internal node: only store split criterion used
In each leaf: store mean of responses
Prediction for a new data point: walk down the tree, return stored value
E.g. for (param1, feature2, param3) = (true, 4.7, red): 1.65
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 42
![Page 106: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/106.jpg)
Random Forests: Sets of Regression Trees
Training
Draw T bootstrap samples of the data
For each bootstrap sample, fit a randomized regression tree
Prediction
Predict with each of the T trees
Return empirical mean and variance across these T predictions
Complexity for N data points
Training: O(TNlog2N)
Prediction: O(T logN)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 43
![Page 107: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/107.jpg)
Random Forests: Sets of Regression Trees
Training
Draw T bootstrap samples of the data
For each bootstrap sample, fit a randomized regression tree
Prediction
Predict with each of the T trees
Return empirical mean and variance across these T predictions
Complexity for N data points
Training: O(TNlog2N)
Prediction: O(T logN)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 43
![Page 108: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/108.jpg)
Random Forests: Sets of Regression Trees
Training
Draw T bootstrap samples of the data
For each bootstrap sample, fit a randomized regression tree
Prediction
Predict with each of the T trees
Return empirical mean and variance across these T predictions
Complexity for N data points
Training: O(TNlog2N)
Prediction: O(T logN)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 43
![Page 109: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/109.jpg)
SMAC: Averaging Across Multiple Instances
Fit a Model to the Algorithm Performance Data
Observed algorithm runtime data: 〈(θi,xi, yi)〉Ni=1
Random forest model m : Θ× I → R
Aggregate over instances by marginalization
f(θ) = Eπ∼D(m(θ, π))
Intuition: predict for each instance and then average
More efficient implementation in random forests
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 44
![Page 110: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/110.jpg)
SMAC: Averaging Across Multiple Instances
Fit a Model to the Algorithm Performance Data
Observed algorithm runtime data: 〈(θi,xi, yi)〉Ni=1
Random forest model m : Θ× I → R
Aggregate over instances by marginalization
f(θ) = Eπ∼D(m(θ, π))
Intuition: predict for each instance and then average
More efficient implementation in random forests
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 44
![Page 111: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/111.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeatLearn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurationsRace selected configurations against best known
until time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 112: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/112.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurationsRace selected configurations against best known
until time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 113: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/113.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurationsRace selected configurations against best known
until time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 114: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/114.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurations
Race selected configurations against best knownuntil time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 115: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/115.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurations
Race selected configurations against best knownuntil time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 116: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/116.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the default
repeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurationsRace selected configurations against best known
until time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 117: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/117.jpg)
SMAC: Putting it all Together
Algorithm 2: SMAC
Initialize with a single run for the defaultrepeat
Learn a RF model from data so far: m : Θ× I → R
Aggregate over instances: f(θ) = Eπ∼D(m(θ, π))
Use model f to select promising configurationsRace selected configurations against best known
until time budget exhausted
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 45
![Page 118: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/118.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC SystemsSpySMAC: A SAT Python Tool for ACpySMAC: A Python Interface to ACSMAC: Full Flexibility for AC
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 46
![Page 119: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/119.jpg)
SMAC, pySMAC, SpySMAC
pySMAC
SpySMAC
SMACAlgorithm
ConfigurationScenario
SMAC Configurator implemented in JAVA
pySMAC Python Interface to SMAC
SpySMAC SAT-pySMAC: an easy-to-use AC framework for SAT-solvers
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 47
![Page 120: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/120.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC SystemsSpySMAC: A SAT Python Tool for ACpySMAC: A Python Interface to ACSMAC: Full Flexibility for AC
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 48
![Page 121: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/121.jpg)
SpySMAC [Falkner et al, 2015]
User input SpySMAC run.py SpySMAC analyze.py
Instances
Training
Test
Solver
PCS file
additionaloptions
default
SMAC 1
...
SMACn
defaultperformance
configuredperformance
combinedrun data
fANOVAparameterimportance
performanceevaluation
scatter plot
CDF plot
cactus plots
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 49
![Page 122: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/122.jpg)
Example: MiniSAT [Een et al, ’03-’07]
MiniSAT (http://minisat.se/) is a SAT solver that is
minimalistic,
open-source,
and developed to help researchers and developers alike to get startedon SAT
MiniSAT has 8 (performance-relevant) parameters
Pitfall: Random seed (rnd-seed) should not be tuned!
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 50
![Page 123: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/123.jpg)
Example: MiniSAT [Een et al, ’03-’07]
MiniSAT (http://minisat.se/) is a SAT solver that is
minimalistic,
open-source,
and developed to help researchers and developers alike to get startedon SAT
MiniSAT has 8 (performance-relevant) parameters
Pitfall: Random seed (rnd-seed) should not be tuned!
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 50
![Page 124: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/124.jpg)
Example: MiniSAT [Een et al, ’03-’07]
MiniSAT (http://minisat.se/) is a SAT solver that is
minimalistic,
open-source,
and developed to help researchers and developers alike to get startedon SAT
MiniSAT has 8 (performance-relevant) parameters
Pitfall: Random seed (rnd-seed) should not be tuned!
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 50
![Page 125: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/125.jpg)
Toy AC Scenario
Solver: MiniSAT
PCS: 8 parameters
Instances: software verification of gzip
Budget: 60 seconds
Cutoff: 2 seconds
Object: runtime
Statistic: PAR10 (penalized average runtime, counting timeouts attmax as 10 · tmax)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 51
![Page 126: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/126.jpg)
Hands-on: SpySMAC
In your virtual machine:
Determine optimized configuration
$ cd AC-Tutorial/SpySMAC/
$ python ./SpySMAC/SpySMAC_run.py
-i ./SpySMAC/examples/swv-inst/SWV-GZIP/
-b ./SpySMAC/examples/minisat/core/minisat
-p ./SpySMAC/examples/minisat/pcs.txt
-o minisat-logs
--prefix "-"
-c 2 -B 60
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 52
![Page 127: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/127.jpg)
Hands-on: SpySMAC
In your virtual machine:
Validate default configuration
$ python ./SpySMAC/SpySMAC_run.py
-i ./SpySMAC/examples/swv-inst/SWV-GZIP/
-b ./SpySMAC/examples/minisat/core/minisat
-p ./SpySMAC/examples/minisat/pcs.txt
-o minisat-logs
--prefix "-"
-c 2 -B 60
--seed 0 # <-- validate default!
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 53
![Page 128: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/128.jpg)
Inputs of SpySMAC
Parameter Configuration Space
Algorithm
Instances
Options:
Target algorithm runtime cutoffConfiguration budget
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 54
![Page 129: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/129.jpg)
Specifying Parameter Configuration Spaces (PCS)
There are many different types of parameter
As for other combinatorial problems, there is a standardrepresentation that different configuration procedures can read
The simple standard format: PCS
PCS (short for ”parameter configuration space”)
human readable/writable
allows to express a wide range of parameter types
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 55
![Page 130: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/130.jpg)
Specifying Parameter Configuration Spaces (PCS)
There are many different types of parameter
As for other combinatorial problems, there is a standardrepresentation that different configuration procedures can read
The simple standard format: PCS
PCS (short for ”parameter configuration space”)
human readable/writable
allows to express a wide range of parameter types
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 55
![Page 131: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/131.jpg)
Configuration Space – Parameter Types
Continuous
Parameter can take every value in a given range:param name [lower bound, upper bound][default]
e.g., rnd-freq [0,1][0]
Integer
Parameter can take every integer value in a given range:param name [lower bound, upper bound][default]i
e.g., rfirst [1,10000000][1]i
Categorical
Parameter can take every value from an unordered finite setparam name {value1, ..., valueN}[default]e.g., ccmin-mode {0,1,2}[0]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 56
![Page 132: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/132.jpg)
Configuration Space – Parameter Types
Continuous
Parameter can take every value in a given range:param name [lower bound, upper bound][default]
e.g., rnd-freq [0,1][0]
Integer
Parameter can take every integer value in a given range:param name [lower bound, upper bound][default]i
e.g., rfirst [1,10000000][1]i
Categorical
Parameter can take every value from an unordered finite setparam name {value1, ..., valueN}[default]e.g., ccmin-mode {0,1,2}[0]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 56
![Page 133: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/133.jpg)
Configuration Space – Parameter Types
Continuous
Parameter can take every value in a given range:param name [lower bound, upper bound][default]
e.g., rnd-freq [0,1][0]
Integer
Parameter can take every integer value in a given range:param name [lower bound, upper bound][default]i
e.g., rfirst [1,10000000][1]i
Categorical
Parameter can take every value from an unordered finite setparam name {value1, ..., valueN}[default]e.g., ccmin-mode {0,1,2}[0]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 56
![Page 134: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/134.jpg)
Range Transformation
Log-Transformation
The ranges of continuous and integer parameter can belog-transformed
→ SMAC samples new configurations uniformly from thelog-transformed range
E.g., the difference between 0.9 and 0.8 is maybe not so important asbetween 0.1 and 0.2
param name [lower bound, upper bound][default]l
e.g., rinc [1.00001,1024][1.00001]l
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 57
![Page 135: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/135.jpg)
Structured Configuration Space
Conditionals
Parameter θ2 of heuristic H is only active if H is used (θ1 = H)
In this case, we say θ2 is a conditional parameter with parent θ1
θ1 currently has to be a categorical parameter
param2 | param1 in {H}
Forbidden Combinations
Sometimes, combinations of parameter settings are forbiddene.g., the combination of θ3 = 1 and θ4 = 2 is forbidden
{param3=1, param4=2}
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 58
![Page 136: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/136.jpg)
Structured Configuration Space
Conditionals
Parameter θ2 of heuristic H is only active if H is used (θ1 = H)
In this case, we say θ2 is a conditional parameter with parent θ1
θ1 currently has to be a categorical parameter
param2 | param1 in {H}
Forbidden Combinations
Sometimes, combinations of parameter settings are forbiddene.g., the combination of θ3 = 1 and θ4 = 2 is forbidden
{param3=1, param4=2}
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 58
![Page 137: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/137.jpg)
PCS Example: MiniSAT
∼/AC-Tutorial/SpySMAC/SpySMAC/examples/minisat/pcs.txtrnd-freq [0,1][1]
var-decay [0.001,1][0.001]l
cla-decay [0.001,1][0.001]l
rinc [1.00001,1024][1.00001]l
gc-frac [0,1][0.00001]
rfirst [1,10000000][1]il
ccmin-mode {0,1,2}[0]
phase-saving {0,1,2}[0]
→ For illustration, we use a bad default configuration.Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 59
![Page 138: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/138.jpg)
Decision: Instance set
Representative instances
Representative of the instances you want to solve later
Moderately hard instances
Too hard: will not solve many instances, no traction
Too easy: will results generalize to harder instances?
Rule of thumb: mix of hardness ranges
Roughly 75% instances solvable by default in maximal cutoff time
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 60
![Page 139: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/139.jpg)
Decision: Instance set
Representative instances
Representative of the instances you want to solve later
Moderately hard instances
Too hard: will not solve many instances, no traction
Too easy: will results generalize to harder instances?
Rule of thumb: mix of hardness ranges
Roughly 75% instances solvable by default in maximal cutoff time
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 60
![Page 140: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/140.jpg)
Decision: Instance set
Homogeneity of instances
Easier to optimize on homogeneous instances
There is one configuration that performs well on all instancesIndicator: Same characteristics, e.g., all instances encode the sameproblem
Harder on heterogeneous instances
A portfolio approach will perform better an these instancesIndicator: Different characteristics, e.g., instances encode differentproblems
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 61
![Page 141: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/141.jpg)
Decision: Instance set
Homogeneity of instances
Easier to optimize on homogeneous instances
There is one configuration that performs well on all instancesIndicator: Same characteristics, e.g., all instances encode the sameproblem
Harder on heterogeneous instances
A portfolio approach will perform better an these instancesIndicator: Different characteristics, e.g., instances encode differentproblems
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 61
![Page 142: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/142.jpg)
Decision: Instance set
Homogeneity of instances
Easier to optimize on homogeneous instances
There is one configuration that performs well on all instancesIndicator: Same characteristics, e.g., all instances encode the sameproblem
Harder on heterogeneous instances
A portfolio approach will perform better an these instancesIndicator: Different characteristics, e.g., instances encode differentproblems
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 61
![Page 143: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/143.jpg)
Decision: Configuration Budget and Cutoff
Configuration Budget
Dictated by your resources and needs
E.g., start configuration before leaving work on Friday
The longer the better (but diminishing returns)
Rough rule of thumb: typically at least enough time for 1000 targetruns (i.e., 1000 times the cutoff)But have also achieved good results with 50 target runs in some cases
Maximal cutoff time per target run
Dictated by your needs (typical instance hardness, etc.)
Too high: slow progress
Too low: possible overtuning to easy instances
For SAT etc, often use 300 CPU seconds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 62
![Page 144: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/144.jpg)
Decision: Configuration Budget and Cutoff
Configuration Budget
Dictated by your resources and needs
E.g., start configuration before leaving work on Friday
The longer the better (but diminishing returns)
Rough rule of thumb: typically at least enough time for 1000 targetruns (i.e., 1000 times the cutoff)But have also achieved good results with 50 target runs in some cases
Maximal cutoff time per target run
Dictated by your needs (typical instance hardness, etc.)
Too high: slow progress
Too low: possible overtuning to easy instances
For SAT etc, often use 300 CPU seconds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 62
![Page 145: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/145.jpg)
Toy AC Scenario (repeated)
Solver: MiniSAT
PCS: 8 parameters
Instances: software verification of gzip
Budget: 60 seconds
Cutoff: 2 seconds
Object: runtime
Statistic: PAR10 (penalized average runtime, counting timeouts attmax as 10 · tmax)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 63
![Page 146: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/146.jpg)
Hands-on: SpySMAC
In your virtual machine:
Analyze and show report
$ python ./SpySMAC/SpySMAC_analyze.py
-i minisat-logs/
-o minisat-report
[...]
$ firefox minisat-report/index.html &
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 64
![Page 147: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/147.jpg)
Running examples of algorithms to compare
Taken from the Configurable SAT Solver Challenge [Hutter et al, 2015]
(http://aclib.net/cssc2014/)
Example 1:Algorithm: Lingeling [Biere, 2013]
- 2 versions: default vs. configured on training instances
Benchmark set: Circuit-Fuzz [Brummayer et al, 2013]
Example 2:Algorithm: Clasp [Gebser et al, 2012]
- 2 versions: default vs. configured on training instances
Benchmark set: N-Rooks [Manthey & Steinke, 2014]
Comparing runtimes on test instances not used for configuration
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 65
![Page 148: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/148.jpg)
Running examples of algorithms to compare
Taken from the Configurable SAT Solver Challenge [Hutter et al, 2015]
(http://aclib.net/cssc2014/)
Example 1:Algorithm: Lingeling [Biere, 2013]
- 2 versions: default vs. configured on training instances
Benchmark set: Circuit-Fuzz [Brummayer et al, 2013]
Example 2:Algorithm: Clasp [Gebser et al, 2012]
- 2 versions: default vs. configured on training instances
Benchmark set: N-Rooks [Manthey & Steinke, 2014]
Comparing runtimes on test instances not used for configuration
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 65
![Page 149: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/149.jpg)
Running examples of algorithms to compare
Taken from the Configurable SAT Solver Challenge [Hutter et al, 2015]
(http://aclib.net/cssc2014/)
Example 1:Algorithm: Lingeling [Biere, 2013]
- 2 versions: default vs. configured on training instances
Benchmark set: Circuit-Fuzz [Brummayer et al, 2013]
Example 2:Algorithm: Clasp [Gebser et al, 2012]
- 2 versions: default vs. configured on training instances
Benchmark set: N-Rooks [Manthey & Steinke, 2014]
Comparing runtimes on test instances not used for configuration
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 65
![Page 150: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/150.jpg)
Running examples of algorithms to compare
Taken from the Configurable SAT Solver Challenge [Hutter et al, 2015]
(http://aclib.net/cssc2014/)
Example 1:Algorithm: Lingeling [Biere, 2013]
- 2 versions: default vs. configured on training instances
Benchmark set: Circuit-Fuzz [Brummayer et al, 2013]
Example 2:Algorithm: Clasp [Gebser et al, 2012]
- 2 versions: default vs. configured on training instances
Benchmark set: N-Rooks [Manthey & Steinke, 2014]
Comparing runtimes on test instances not used for configuration
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 65
![Page 151: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/151.jpg)
Running examples of algorithms to compare
Taken from the Configurable SAT Solver Challenge [Hutter et al, 2015]
(http://aclib.net/cssc2014/)
Example 1:Algorithm: Lingeling [Biere, 2013]
- 2 versions: default vs. configured on training instances
Benchmark set: Circuit-Fuzz [Brummayer et al, 2013]
Example 2:Algorithm: Clasp [Gebser et al, 2012]
- 2 versions: default vs. configured on training instances
Benchmark set: N-Rooks [Manthey & Steinke, 2014]
Comparing runtimes on test instances not used for configuration
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 65
![Page 152: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/152.jpg)
Comparing algorithms based on summary statistics
Clasp on N-Rooks, tmax = 300sDefault Configured
Average runtime [s] 81.8 4.68PAR10 [s] 704 4.68Timeouts (out of 351) 81 0
PAR10: penalized average runtime, counting timeouts at tmax as 10 · tmax
Lingeling on Circuit-Fuzz, tmax = 300sDefault Configured
Average runtime [s] 47.8 32.0PAR10 [s] 186 115Timeouts (out of 585) 30 18
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 66
![Page 153: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/153.jpg)
Comparing algorithms based on summary statistics
Clasp on N-Rooks, tmax = 300sDefault Configured
Average runtime [s] 81.8 4.68PAR10 [s] 704 4.68Timeouts (out of 351) 81 0
PAR10: penalized average runtime, counting timeouts at tmax as 10 · tmax
Lingeling on Circuit-Fuzz, tmax = 300sDefault Configured
Average runtime [s] 47.8 32.0PAR10 [s] 186 115Timeouts (out of 585) 30 18
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 66
![Page 154: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/154.jpg)
Comparing algorithms based on their runtime distributions
Distributions of runtime across benchmark instances(Clasp on N-Rooks)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 67
![Page 155: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/155.jpg)
Comparing algorithms based on their runtime distributions
(Lingeling on Circuit-Fuzz)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 68
![Page 156: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/156.jpg)
Comparing algorithms based on scatter plots
Each marker represents one instance; note the log-log axis!
(Clasp on N-Rooks; 81 vs. 0 timeouts)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 69
![Page 157: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/157.jpg)
Comparing algorithms based on scatter plots
(Lingeling on Circuit-Fuzz; 30 vs. 18 timeouts)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 70
![Page 158: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/158.jpg)
Comparing algorithms based on scatter plots
Scatter plots can reveal clear patterns in the data
Example: an algorithm that
first runs SLS algorithm A(good for satisfiable instances)for t seconds
then runs complete tree searchalgorithm B (good forunsatisfiable instances) untiltime is up
There are 2 instance clusters:satisfiable (left) and unsatisfiable instances (right)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 71
![Page 159: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/159.jpg)
Comparing algorithms based on scatter plots
Scatter plots can reveal clear patterns in the data
Example: an algorithm that
first runs SLS algorithm A(good for satisfiable instances)for t seconds
then runs complete tree searchalgorithm B (good forunsatisfiable instances) untiltime is up
There are 2 instance clusters:satisfiable (left) and unsatisfiable instances (right)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 71
![Page 160: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/160.jpg)
Why not always using SpySMAC?
Advantages:
Easy to set up – easier than plain SMAC
Includes an analyzing part – not part of SMAC
However, SpySMAC has some limitations right now:
Only runtime optimization
Parses only standard output of SAT solvers
However, with a little Python experience, you could solve all theselimitations.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 72
![Page 161: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/161.jpg)
Why not always using SpySMAC?
Advantages:
Easy to set up – easier than plain SMAC
Includes an analyzing part – not part of SMAC
However, SpySMAC has some limitations right now:
Only runtime optimization
Parses only standard output of SAT solvers
However, with a little Python experience, you could solve all theselimitations.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 72
![Page 162: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/162.jpg)
Why not always using SpySMAC?
Advantages:
Easy to set up – easier than plain SMAC
Includes an analyzing part – not part of SMAC
However, SpySMAC has some limitations right now:
Only runtime optimization
Parses only standard output of SAT solvers
However, with a little Python experience, you could solve all theselimitations.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 72
![Page 163: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/163.jpg)
BREAK - 30 Minutes!
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 73
![Page 164: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/164.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC SystemsSpySMAC: A SAT Python Tool for ACpySMAC: A Python Interface to ACSMAC: Full Flexibility for AC
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 74
![Page 165: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/165.jpg)
pySMAC
pySMAC
PythonFunction
SMACIPC
θ
function value
“Algorithm”; black box (Python) function
Hides most of SMAC ’s complexity (e.g., instances)
Still as powerful as SMAC
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 75
![Page 166: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/166.jpg)
Toy Example: Rosenbrock Function
source: wikipedia.org
f(x, y) = (a− x)2 + b(y − x2)2
Well-known non-convex functionOptimum at (x, y) = (a, a2)Multidimensional generalization possible (here: 4D)Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 76
![Page 167: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/167.jpg)
Optimize on Rosenbrock
Optimize Strategies:
1 SMAC (as shown before)
2 Random search
In your virtual machine
pySMAC Call
$ cd ~/AC-Tutorial/pysmac
$ python example.py
In the meantime, let us look into example.py
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 77
![Page 168: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/168.jpg)
Optimize on Rosenbrock
Optimize Strategies:
1 SMAC (as shown before)
2 Random search
In your virtual machine
pySMAC Call
$ cd ~/AC-Tutorial/pysmac
$ python example.py
In the meantime, let us look into example.py
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 77
![Page 169: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/169.jpg)
Optimize on Rosenbrock
Optimize Strategies:
1 SMAC (as shown before)
2 Random search
In your virtual machine
pySMAC Call
$ cd ~/AC-Tutorial/pysmac
$ python example.py
In the meantime, let us look into example.py
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 77
![Page 170: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/170.jpg)
Advanced pySMAC
Restricting your functions resources
Optimizing runtime instead of quality
Optimizing on a set of instances
Validation
Non-deterministic functions
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 78
![Page 171: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/171.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC SystemsSpySMAC: A SAT Python Tool for ACpySMAC: A Python Interface to ACSMAC: Full Flexibility for AC
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 79
![Page 172: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/172.jpg)
SMAC
SMAC
Core of our algorithm configuration tools
Implementation of the optimization algorithm
Further Inputs (wrt SpySMAC )
Scenario File
Call to algorithm through a wrapper
Wrapper has to honour a specified syntax
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 80
![Page 173: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/173.jpg)
Scenario File Example
algo-exec python -u minisat/SATWrapper.py --mem-limit 1024\
--script minisat/MiniSATWrapper.py
pcs-file minisat/pcs.txt
instances minisat/instances_train.txt
test-instances minisat/instances_test.txt
cutoff_time 2
wallclock-limit 120
algo-deterministic False
run-obj runtime
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 81
![Page 174: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/174.jpg)
How to Wrap a Target Algorithm
Call Format<algo executable> <instance name>
<instance specific information> <cutoff time>
<cutoff length> <seed> <param> <value> <param> <value>...
Output Format
Result of this algorithm run: <status>,
<runtime>, <runlength>, <quality>, <seed>
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 82
![Page 175: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/175.jpg)
How to Wrap a Target Algorithm
Call Format<algo executable> <instance name>
<instance specific information> <cutoff time>
<cutoff length> <seed> <param> <value> <param> <value>...
Output Format
Result of this algorithm run: <status>,
<runtime>, <runlength>, <quality>, <seed>
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 82
![Page 176: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/176.jpg)
How To Wrap the Target Algorithm: GenericWrapper
We provide a Python script with basic functionality as a wrapper,called GenericWrapper:
Limitation of runtime and memory with runsolver
Interface to configurator
Parsing of input parametersOutput string in the required format
You only have to implement two functions:
How to call your algorithm with the given parameter configuration?
How to parse the output of your algorithm?
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 83
![Page 177: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/177.jpg)
How To Wrap the Target Algorithm: GenericWrapper
We provide a Python script with basic functionality as a wrapper,called GenericWrapper:
Limitation of runtime and memory with runsolver
Interface to configurator
Parsing of input parametersOutput string in the required format
You only have to implement two functions:
How to call your algorithm with the given parameter configuration?
How to parse the output of your algorithm?
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 83
![Page 178: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/178.jpg)
GenericWrapper: CMD Call for MiniSAT
see minisat/MiniSATWrapper.py
def get_command_line_cmd(runargs, config):
cmd = "minisat/minisat -rnd-seed=%d" %(runargs["seed"])
for name, value in config.items():
cmd += " %s=%s" %(name, value)
cmd += " %s" %(runargs["instance"])
return cmd
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 84
![Page 179: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/179.jpg)
GenericWrapper: Output Parsing for MiniSAT
see minisat/SATWrapper.py (Bug in this script in the VM)
def process_results(self, filepointer, exit_code):
data = filepointer.read()
resultMap = {}
if re.search(’UNSATISFIABLE’, data):
resultMap[’status’] = ’UNSAT’
elif re.search(’SATISFIABLE’, data):
resultMap[’status’] = ’SAT’
elif re.search(’UNKNOWN’, data):
resultMap[’status’] = ’TIMEOUT’
else:
resultMap[’status’] = ’CRASHED’
return resultMap
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 85
![Page 180: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/180.jpg)
Hands-On: SMAC
$ cd ~/AC-Tutorial/minisat-smac/
$ ../smac-v2.10.03-master-778/smac
--scenarioFile scenario.txt
[...]
$ ../smac-v2.10.03-master-778/smac-validate
--scenarioFile scenario.txt
--random-configurations 1 --includeDefaultAsFirstRandom true
--numRun 1
[...]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 86
![Page 181: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/181.jpg)
Hands-On: SMAC ’s Output
SMAC has finished. [...]
Total number of runs performed: 222,
total configurations tried: 23.
Total CPU time used: 79 s, total wallclock time used: 121 s.
SMAC’s final incumbent: config 16 (internal ID: 0x34CE3),
with estimated penalized average runtime (PAR10): 0.868,
based on 24 run(s) on 24 training instance(s).
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 87
![Page 182: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/182.jpg)
Hands-On: Validation Output
Estimated mean quality
of final incumbent config 16
on test set: 0.0168,
based on 45 run(s) on 45 test instance(s).
Sample call for the final incumbent:
cd /home/ac/AC-Tutorial/minisat-smac;
python -u minisat/SATWrapper.py --mem-limit 1024
--script minisat/MiniSATWrapper.py
swv-inst/SWV-GZIP/gzip_vc1073.cnf 0 2.0
2147483647 12660129
-ccmin-mode ’2’ -cla-decay ’0.014589435384907675’ [...]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 88
![Page 183: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/183.jpg)
Hands-On: Validation of Default
$ cat validationResults-cli-1-walltime.csv
"Time","Training Performance","Test Set Performance",[...]
0.0,0.0,0.49128888888888894,[...]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 89
![Page 184: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/184.jpg)
Instance Features
Instance Features
Numerical characterizations of problem instances
Examples:
SAT: #variables, #clauses,. . .Planning: #actions, #fluents,. . .Machine Learning: #observations, #features,. . .
Can improve the accuracy of the EPM used in SMAC
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 90
![Page 185: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/185.jpg)
Instance Features
Instance Features
Numerical characterizations of problem instances
Examples:
SAT: #variables, #clauses,. . .Planning: #actions, #fluents,. . .Machine Learning: #observations, #features,. . .
Can improve the accuracy of the EPM used in SMAC
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 90
![Page 186: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/186.jpg)
Types of Instance Features
Counting Features
Compute some statistics about its characteristics
Probing Features
Running an algorithm for a short amount of time
Analyze how the algorithm behaves
For example, number of steps to the best local minimum in a run.
Please note that there are different names in literature for these types offeatures. For example, probing features are related to landmarkingfeatures.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 91
![Page 187: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/187.jpg)
Types of Instance Features
Counting Features
Compute some statistics about its characteristics
Probing Features
Running an algorithm for a short amount of time
Analyze how the algorithm behaves
For example, number of steps to the best local minimum in a run.
Please note that there are different names in literature for these types offeatures. For example, probing features are related to landmarkingfeatures.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 91
![Page 188: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/188.jpg)
Types of Instance Features
Counting Features
Compute some statistics about its characteristics
Probing Features
Running an algorithm for a short amount of time
Analyze how the algorithm behaves
For example, number of steps to the best local minimum in a run.
Please note that there are different names in literature for these types offeatures. For example, probing features are related to landmarkingfeatures.
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 91
![Page 189: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/189.jpg)
Instance Features: SAT [Hutter et al, 2014]
Problem size features (7)
Variable-Clause graph features (10)
Variable graph features (9)
Clause graph features (10)
Balance features (13)
Proximity to horn formula (7)
DPLL Probing Features (7)
LP-based feature (6)
Local search probing features (12)
Clause learning features (18)
Survey propagation feature (18)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 92
![Page 190: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/190.jpg)
Hands-On: Instance features for SAT
$ cd ~/AC-Tutorial/minisat-smac/
$ bash get_swv_minisat_features.sh > features.csv
$ ../smac-v2.10.03-master-778/smac
--scenarioFile scenario-features.txt
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 93
![Page 191: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/191.jpg)
Another Toy Example: Pyperplan
Pyperplan is a simple planner
Developed for demonstration purposes on the University of Freiburg
Only 2 parameters: search heuristic and strategy
heuristic categorical {hmax,hff,hadd,blind,hsa,lmcut,landmark}[blind]
search categorical {ehs,gbf,wastar,astar,ids,bfs}[bfs]
→ 42 parameter configurations
Instance set: 15 elevator instances (training)
Cutoff time: 10 seconds
→ Worst case of trying all configurations on all instances:10 · 15 · 42 = 6300 seconds
configuration budget: 600 seconds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 94
![Page 192: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/192.jpg)
Hands-On: Configuration of a Planner
$ cd ~/AC-Tutorial/pyperplan/
$ ../smac-v2.10.03-master-778/smac
--scenarioFile scenario.txt
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 95
![Page 193: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/193.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of ParametersAblationForward SelectionfANOVA
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 96
![Page 194: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/194.jpg)
Parameter Importance
Recommendations & Observation
Configure all parameters that could influence performance
Dependent on the instance set, different parameters matter
How to determine the important parameters?
Example
SAT-solver lingeling has more than 300 parameters
Often, less than 10 are important to optimize performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 97
![Page 195: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/195.jpg)
Parameter Importance
Recommendations & Observation
Configure all parameters that could influence performance
Dependent on the instance set, different parameters matter
How to determine the important parameters?
Example
SAT-solver lingeling has more than 300 parameters
Often, less than 10 are important to optimize performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 97
![Page 196: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/196.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of ParametersAblationForward SelectionfANOVA
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 98
![Page 197: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/197.jpg)
Ablation [Fawcett et al. 2013]
Idea
Starting from the default configuration, we change the value of theparameters
Which of these changes were important?
→ Ablation compares parameter flips between default and incumbentconfiguration
Basic Approach
Iterate over all non-flipped parameters
Flip the parameter with the largest influence on the performance in eachiteration
Ablation with Racing
To determine the best flip in each iteration, use racing with a statistical testto speed up the decision
→ do not assess the performance on all instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 99
![Page 198: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/198.jpg)
Ablation [Fawcett et al. 2013]
Idea
Starting from the default configuration, we change the value of theparameters
Which of these changes were important?
→ Ablation compares parameter flips between default and incumbentconfiguration
Basic Approach
Iterate over all non-flipped parameters
Flip the parameter with the largest influence on the performance in eachiteration
Ablation with Racing
To determine the best flip in each iteration, use racing with a statistical testto speed up the decision
→ do not assess the performance on all instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 99
![Page 199: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/199.jpg)
Ablation [Fawcett et al. 2013]
Idea
Starting from the default configuration, we change the value of theparameters
Which of these changes were important?
→ Ablation compares parameter flips between default and incumbentconfiguration
Basic Approach
Iterate over all non-flipped parameters
Flip the parameter with the largest influence on the performance in eachiteration
Ablation with Racing
To determine the best flip in each iteration, use racing with a statistical testto speed up the decision
→ do not assess the performance on all instancesHutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 99
![Page 200: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/200.jpg)
Toy example: Ablation
θ1 θ2 θ3 m
Default 1 1 1 100Conf 2 2 2 10
1st Iteration2 1 1 901 2 1 301 1 2 100
Flip θ2
2nd Iteration2 2 1 101 2 2 30
Flip θ1
3rd Iteration2 2 2 10
Flip θ3
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 100
![Page 201: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/201.jpg)
Ablation Example: Spear on SWV
Source: [Fawcett et al. 2013]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 101
![Page 202: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/202.jpg)
Ablation Example: LGP on Zenotravel
Source: [Fawcett et al. 2013]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 102
![Page 203: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/203.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of ParametersAblationForward SelectionfANOVA
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 103
![Page 204: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/204.jpg)
Forward Selection [Hutter et al. 2013]
Idea
Which parameters (or instance features) do we need to train a goodEPM (m : Θ×F → R)?
Use forward selection to identify important parameters (/instances)
Feature Selection via Forward Selection
Iterative approach
Add in each iteration the parameter (/feature) that improves thequality of the EPM the most
Details
Minimize RMSE (root mean squared error) on a validation set
Limit the maximal number of selected parameters (/features)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 104
![Page 205: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/205.jpg)
Forward Selection [Hutter et al. 2013]
Idea
Which parameters (or instance features) do we need to train a goodEPM (m : Θ×F → R)?
Use forward selection to identify important parameters (/instances)
Feature Selection via Forward Selection
Iterative approach
Add in each iteration the parameter (/feature) that improves thequality of the EPM the most
Details
Minimize RMSE (root mean squared error) on a validation set
Limit the maximal number of selected parameters (/features)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 104
![Page 206: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/206.jpg)
Forward Selection [Hutter et al. 2013]
Idea
Which parameters (or instance features) do we need to train a goodEPM (m : Θ×F → R)?
Use forward selection to identify important parameters (/instances)
Feature Selection via Forward Selection
Iterative approach
Add in each iteration the parameter (/feature) that improves thequality of the EPM the most
Details
Minimize RMSE (root mean squared error) on a validation set
Limit the maximal number of selected parameters (/features)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 104
![Page 207: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/207.jpg)
Forward Selection [Hutter et al. 2013]
Spear on SWV
Source: [Hutter et al. 2013]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 105
![Page 208: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/208.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of ParametersAblationForward SelectionfANOVA
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 106
![Page 209: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/209.jpg)
fANOVA [Hutter et al. 2014]
fANOVA [Sobol 1993]
Write performance predictions y as a sum of components:
y(θ1, . . . , θn) = f0 +∑n
i=1 fi(θi) +∑
i 6=j fij(θi, θj) + . . .
y(θ1, . . . , θn) = average response + main effects +
2-D interaction effects + higher order effects
Variance Decomposition
V =1
||Θ||
∫θ1
. . .
∫θn
[(y(θ)− f0)2]dθ1 . . . dθn
Application to Parameter Importance
How much of the variance can be explained by a parameter (orcombinations of parameters) marginalized over all other parameters?
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 107
![Page 210: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/210.jpg)
fANOVA [Hutter et al. 2014]
fANOVA [Sobol 1993]
Write performance predictions y as a sum of components:
y(θ1, . . . , θn) = f0 +∑n
i=1 fi(θi) +∑
i 6=j fij(θi, θj) + . . .
y(θ1, . . . , θn) = average response + main effects +
2-D interaction effects + higher order effects
Variance Decomposition
V =1
||Θ||
∫θ1
. . .
∫θn
[(y(θ)− f0)2]dθ1 . . . dθn
Application to Parameter Importance
How much of the variance can be explained by a parameter (orcombinations of parameters) marginalized over all other parameters?
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 107
![Page 211: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/211.jpg)
fANOVA [Hutter et al. 2014]
fANOVA [Sobol 1993]
Write performance predictions y as a sum of components:
y(θ1, . . . , θn) = f0 +∑n
i=1 fi(θi) +∑
i 6=j fij(θi, θj) + . . .
y(θ1, . . . , θn) = average response + main effects +
2-D interaction effects + higher order effects
Variance Decomposition
V =1
||Θ||
∫θ1
. . .
∫θn
[(y(θ)− f0)2]dθ1 . . . dθn
Application to Parameter Importance
How much of the variance can be explained by a parameter (orcombinations of parameters) marginalized over all other parameters?
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 107
![Page 212: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/212.jpg)
fANOVA Example
lingeling on circuit fuzz
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 108
![Page 213: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/213.jpg)
fANOVA Example
lingeling on circuit fuzz
0 seems to be a bad value for score
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 109
![Page 214: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/214.jpg)
fANOVA Example
Interesting part of the configuration space
Consider only performance values better than the default
lingeling on circuit fuzz
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 110
![Page 215: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/215.jpg)
fANOVA Example
lingeling on circuit fuzz
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 111
![Page 216: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/216.jpg)
fANOVA Example
probSAT on 3-SAT instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 112
![Page 217: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/217.jpg)
fANOVA Interaction Example
Sparrow on 5-SAT instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 113
![Page 218: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/218.jpg)
fANOVA in SpySMAC
Look again at SpySMAC report from our MiniSAT example
$ firefox ~/AC-Tutorial/SpySMAC/minisat-report/index.html
More reports in∼/AC-Tutorial/spysmac-reports
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 114
![Page 219: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/219.jpg)
Comparison Ablation, Forward Selection & fANOVA
Ablation
+ Only method to compare two configurations
- Needs a lot of algorithm runs → slow
Forward Selection
+ EPM can be trained by the performance data collected during configuration
+/- Considers complete configuration space
- Slow if the training of the EPM is slow (repeated process!)
fANOVA
+ EPM can be trained by the performance data collected during configuration
+ Considers the complete configuration space or only “interesting” areas
- Importance of interactions between parameters can be expensive
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 115
![Page 220: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/220.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best PracticesOvertuningWrapping the Target AlgorithmGeneral Advice
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 116
![Page 221: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/221.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best PracticesOvertuningWrapping the Target AlgorithmGeneral Advice
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 117
![Page 222: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/222.jpg)
The concept of overtuning
Very related to overfitting in machine learning
Performance improves on the training set
Performance does not improve on the test set, and may even degrade
More pronounced for more heterogeneous benchmark sets
But it even happens for very homogeneous sets
Indeed, one can even overfit on a single instance, to the seeds usedfor training
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 118
![Page 223: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/223.jpg)
The concept of overtuning
Very related to overfitting in machine learning
Performance improves on the training set
Performance does not improve on the test set, and may even degrade
More pronounced for more heterogeneous benchmark sets
But it even happens for very homogeneous sets
Indeed, one can even overfit on a single instance, to the seeds usedfor training
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 118
![Page 224: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/224.jpg)
Overtuning Visualized
Example: minimizing SLS solver runlengths for a single SAT instance
Training cost, here based on N=100 runs with different seeds
Test cost of θ here based on 1000 new seeds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 119
![Page 225: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/225.jpg)
Overtuning Visualized
Example: minimizing SLS solver runlengths for a single SAT instance
Training cost, here based on N=100 runs with different seeds
Test cost of θ here based on 1000 new seeds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 119
![Page 226: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/226.jpg)
Overtuning Visualized
Example: minimizing SLS solver runlengths for a single SAT instance
Training cost, here based on N=100 runs with different seeds
Test cost of θ here based on 1000 new seeds
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 119
![Page 227: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/227.jpg)
Overtuning is Stronger For Smaller Training Sets
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 120
![Page 228: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/228.jpg)
BasicILS(N) Test Results with Various N
Small N : fast evaluations → quick progress, but overtunes
Large N : slow, but overtunes less
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 121
![Page 229: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/229.jpg)
FocusedILS achieves the best of both worlds
Fast progress and no overtuning (provably, in the limit)
General principle: focus your budget on good configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 122
![Page 230: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/230.jpg)
FocusedILS achieves the best of both worlds
Fast progress and no overtuning (provably, in the limit)
General principle: focus your budget on good configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 122
![Page 231: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/231.jpg)
Hands-On: Overtuning to Fixed Seeds
Demonstration on the Simplest Case
The real issue is overtuning to subset of instances
But that would take too long to demo
In your virtual machine:
Limit SMAC to optimize on 1 seed vs. 100 seeds
$ cd AC-Tutorial/saps-overtuning/
$ vim scenario1.txt
$ ../smac-v2.10.03-master-778/smac --scenarioFile scenario1.txt
$ vim scenario100.txt
$ ../smac-v2.10.03-master-778/smac --scenarioFile scenario100.txt
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 123
![Page 232: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/232.jpg)
Hands-On: Overtuning to Fixed Seeds
Demonstration on the Simplest Case
The real issue is overtuning to subset of instances
But that would take too long to demo
In your virtual machine:
Limit SMAC to optimize on 1 seed vs. 100 seeds
$ cd AC-Tutorial/saps-overtuning/
$ vim scenario1.txt
$ ../smac-v2.10.03-master-778/smac --scenarioFile scenario1.txt
$ vim scenario100.txt
$ ../smac-v2.10.03-master-778/smac --scenarioFile scenario100.txt
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 123
![Page 233: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/233.jpg)
General advice: make solver’s randomness explicit
Several communities dislike randomness
Key arguments: reproducibility, tracking down bugs
I agree these are important
But you can achieve them by keeping track of your seeds
In fact: your tests will cover more cases when randomized
It’s much easier to get more seeds than more instances
Performance should generalize to new seeds
Otherwise, it’s less likely to generalize to new instances
If you have N instances and a budget for N runs
Do 1 run for each of the instances, with a different seed each
This simultaneously captures variance in instance & seed space
Provably yields lowest-variance estimator of mean performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 124
![Page 234: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/234.jpg)
General advice: make solver’s randomness explicit
Several communities dislike randomness
Key arguments: reproducibility, tracking down bugs
I agree these are important
But you can achieve them by keeping track of your seeds
In fact: your tests will cover more cases when randomized
It’s much easier to get more seeds than more instances
Performance should generalize to new seeds
Otherwise, it’s less likely to generalize to new instances
If you have N instances and a budget for N runs
Do 1 run for each of the instances, with a different seed each
This simultaneously captures variance in instance & seed space
Provably yields lowest-variance estimator of mean performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 124
![Page 235: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/235.jpg)
General advice: make solver’s randomness explicit
Several communities dislike randomness
Key arguments: reproducibility, tracking down bugs
I agree these are important
But you can achieve them by keeping track of your seeds
In fact: your tests will cover more cases when randomized
It’s much easier to get more seeds than more instances
Performance should generalize to new seeds
Otherwise, it’s less likely to generalize to new instances
If you have N instances and a budget for N runs
Do 1 run for each of the instances, with a different seed each
This simultaneously captures variance in instance & seed space
Provably yields lowest-variance estimator of mean performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 124
![Page 236: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/236.jpg)
Different Types of Overtuning
One can overtune to various specifics of the training setup
To the specific instances used in the training set
To the specific seeds used in the training set
To the (small) runtime cutoff used during training
To a particular machine type
To the type of instances in the training set
These should just be drawn according to the distribution of interestBut in practice, the distribution might change over time
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 125
![Page 237: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/237.jpg)
Different Types of Overtuning
One can overtune to various specifics of the training setup
To the specific instances used in the training set
To the specific seeds used in the training set
To the (small) runtime cutoff used during training
To a particular machine type
To the type of instances in the training set
These should just be drawn according to the distribution of interestBut in practice, the distribution might change over time
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 125
![Page 238: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/238.jpg)
Different Types of Overtuning
One can overtune to various specifics of the training setup
To the specific instances used in the training set
To the specific seeds used in the training set
To the (small) runtime cutoff used during training
To a particular machine type
To the type of instances in the training set
These should just be drawn according to the distribution of interestBut in practice, the distribution might change over time
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 125
![Page 239: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/239.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best PracticesOvertuningWrapping the Target AlgorithmGeneral Advice
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 126
![Page 240: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/240.jpg)
How to Wrap a Target Algorithm
Don’t trust any target algorithm
Will it terminate in the time you specify?
Will it correctly report its time?
Will it never use more memory than specified?
Will it yield correct results with all parameter settings?
Good Practice
Wrap target runs with tool controlling time and memory
E.g., runsolver [Roussel et al, 2012].
Our genericWrapper.py already does this for you
Good Practice
Verify correctness of target runs
Detect crashes & penalize them
Our genericWrapper.py already does the penalization for you
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 127
![Page 241: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/241.jpg)
How to Wrap a Target Algorithm
Don’t trust any target algorithm
Will it terminate in the time you specify?
Will it correctly report its time?
Will it never use more memory than specified?
Will it yield correct results with all parameter settings?
Good Practice
Wrap target runs with tool controlling time and memory
E.g., runsolver [Roussel et al, 2012].
Our genericWrapper.py already does this for you
Good Practice
Verify correctness of target runs
Detect crashes & penalize them
Our genericWrapper.py already does the penalization for you
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 127
![Page 242: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/242.jpg)
How to Wrap a Target Algorithm
Don’t trust any target algorithm
Will it terminate in the time you specify?
Will it correctly report its time?
Will it never use more memory than specified?
Will it yield correct results with all parameter settings?
Good Practice
Wrap target runs with tool controlling time and memory
E.g., runsolver [Roussel et al, 2012].
Our genericWrapper.py already does this for you
Good Practice
Verify correctness of target runs
Detect crashes & penalize them
Our genericWrapper.py already does the penalization for you
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 127
![Page 243: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/243.jpg)
Pitfalls in Wrappers #1
Pitfall
Blindly minimizing target algorithm runtime
Typically, you will minimize the time to crash
Anecdote 1
In 2007, with Daniel Le Berre, I configured SAT4J
I got huge improvements, emailed Daniel the configuration found
But there was a bug
- Some preprocessings were incompatible with some data structures- Leading to a very quick (wrong) result UNSAT- One solution: declare forbidden combinations
Lesson learned: verify correctness of target runs
E.g., for SAT: compare to known solubility status
E.g., for SAT: check assignment found (polytime)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 128
![Page 244: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/244.jpg)
Pitfalls in Wrappers #1
Pitfall
Blindly minimizing target algorithm runtime
Typically, you will minimize the time to crash
Anecdote 1
In 2007, with Daniel Le Berre, I configured SAT4J
I got huge improvements, emailed Daniel the configuration found
But there was a bug
- Some preprocessings were incompatible with some data structures- Leading to a very quick (wrong) result UNSAT- One solution: declare forbidden combinations
Lesson learned: verify correctness of target runs
E.g., for SAT: compare to known solubility status
E.g., for SAT: check assignment found (polytime)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 128
![Page 245: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/245.jpg)
Pitfalls in Wrappers #1
Pitfall
Blindly minimizing target algorithm runtime
Typically, you will minimize the time to crash
Anecdote 1
In 2007, with Daniel Le Berre, I configured SAT4J
I got huge improvements, emailed Daniel the configuration found
But there was a bug
- Some preprocessings were incompatible with some data structures- Leading to a very quick (wrong) result UNSAT- One solution: declare forbidden combinations
Lesson learned: verify correctness of target runs
E.g., for SAT: compare to known solubility status
E.g., for SAT: check assignment found (polytime)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 128
![Page 246: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/246.jpg)
Pitfalls in Wrappers #2
Pitfall
Blindly minimizing target algorithm runtime
Typically, you will minimize the time to crash
Anecdote 2
In 2010, I optimized several MIP solvers [Hutter et al, CPAIOR 2010]
I found many bugs
Non-default configurations are often poorly tested
You can use algorithm configuration for testing!
Anecdote 3
Some published work does indeed blindly minimize runtime
Avoid this in your own work
Ask about this in reviews etc
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 129
![Page 247: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/247.jpg)
Pitfalls in Wrappers #2
Pitfall
Blindly minimizing target algorithm runtime
Typically, you will minimize the time to crash
Anecdote 2
In 2010, I optimized several MIP solvers [Hutter et al, CPAIOR 2010]
I found many bugs
Non-default configurations are often poorly tested
You can use algorithm configuration for testing!
Anecdote 3
Some published work does indeed blindly minimize runtime
Avoid this in your own work
Ask about this in reviews etc
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 129
![Page 248: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/248.jpg)
Pitfalls in Wrappers #3
Pitfall
Trusting the target algorithm to handle its time and memory
Anecdote 4
Configuring commercial MIP solvers was an eye-opener
Some runs didn’t end when not terminated externally
Reported runtimes were incorrect (sometimes negative!)
Jobs crashed on the cluster because memory limit was not honoured
Disclaimer: code versions from 2010.
Lesson Learned
Once bitten, twice shy. Don’t trust your target algorithm!
Use runsolver (or similar) to control time & memory
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 130
![Page 249: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/249.jpg)
Pitfalls in Wrappers #3
Pitfall
Trusting the target algorithm to handle its time and memory
Anecdote 4
Configuring commercial MIP solvers was an eye-opener
Some runs didn’t end when not terminated externally
Reported runtimes were incorrect (sometimes negative!)
Jobs crashed on the cluster because memory limit was not honoured
Disclaimer: code versions from 2010.
Lesson Learned
Once bitten, twice shy. Don’t trust your target algorithm!
Use runsolver (or similar) to control time & memory
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 130
![Page 250: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/250.jpg)
Pitfalls in Wrappers #4
Pitfall
Using different wrappers for comparing different configurators
Anecdote 5
While developing SMAC, I compared it to ParamILS for tuning UBCSAT
Somehow, SMAC achieved runtimes 20% better than ‘possible’
I had even used the same wrapper
Only difference: ParamILS used an absolute path to the instance onthe file system, SMAC a relative one
Explanation:
- UBCSAT saved its callstring in the heap space before the instance data- Thus, the length of the callstring affected memory locality- Thus, more cache misses when using absolute paths→ 20% greater runtime
- Now fixed in UBCSAT
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 131
![Page 251: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/251.jpg)
Pitfalls in Wrappers #4
Pitfall
Using different wrappers for comparing different configurators
Anecdote 5
While developing SMAC, I compared it to ParamILS for tuning UBCSAT
Somehow, SMAC achieved runtimes 20% better than ‘possible’
I had even used the same wrapper
Only difference: ParamILS used an absolute path to the instance onthe file system, SMAC a relative one
Explanation:
- UBCSAT saved its callstring in the heap space before the instance data- Thus, the length of the callstring affected memory locality- Thus, more cache misses when using absolute paths→ 20% greater runtime
- Now fixed in UBCSAT
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 131
![Page 252: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/252.jpg)
Pitfalls in Wrappers #4
Pitfall
Using different wrappers for comparing different configurators
Anecdote 5
While developing SMAC, I compared it to ParamILS for tuning UBCSAT
Somehow, SMAC achieved runtimes 20% better than ‘possible’
I had even used the same wrapper
Only difference: ParamILS used an absolute path to the instance onthe file system, SMAC a relative one
Explanation:
- UBCSAT saved its callstring in the heap space before the instance data- Thus, the length of the callstring affected memory locality- Thus, more cache misses when using absolute paths→ 20% greater runtime
- Now fixed in UBCSAT
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 131
![Page 253: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/253.jpg)
Pitfalls in Wrappers #5
Pitfall
Using different wrappers for comparing different configurators
Anecdote 6
A new configurator was introduced and compared to SMAC
Wrapper for SMAC had several bugs, breaking the comparison
Authors did not notice; paper published
Bug 1: regular expressions
if (’s SATISFIABLE\n’ in lines) or (’s UNSATISFIABLE’ in lines)
Missing \n after UNSATISFIABLE
all UNSAT instances counted as TIMEOUT
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 132
![Page 254: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/254.jpg)
Pitfalls in Wrappers #5
Pitfall
Using different wrappers for comparing different configurators
Anecdote 6
A new configurator was introduced and compared to SMAC
Wrapper for SMAC had several bugs, breaking the comparison
Authors did not notice; paper published
Bug 1: regular expressions
if (’s SATISFIABLE\n’ in lines) or (’s UNSATISFIABLE’ in lines)
Missing \n after UNSATISFIABLE
all UNSAT instances counted as TIMEOUT
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 132
![Page 255: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/255.jpg)
Pitfalls in Wrappers #5
Pitfall
Using different wrappers for comparing different configurators
Anecdote 6
A new configurator was introduced and compared to SMAC
Wrapper for SMAC had several bugs, breaking the comparison
Authors did not notice; paper published
Bug 2: terminating the target algorithm
p = subprocess.Popen(cmd, shell=True)
...
os.kill(p.pid, signal.SIGKILL)
Main problem: using shell=true
- Leads to killing the shell, but not the target algorithm inside it
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 133
![Page 256: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/256.jpg)
Lessons Learned for Comparisons of Configurators
It is far too easy to introduce a subtle difference in a new wrapper
Check: solver callstrings have to be identical for same configuration
If possible, extend our genericWrapper.py from AClib
Use standard scenarios for comparing configurators: AClib
We have a benchmark library of > 300 AC scenarios
http://www.aclib.net
Based on a git repository, maintained by several research groups
As in other communities, reporting results on known benchmarksshould be mandatory
Please contribute new scenarios
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 134
![Page 257: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/257.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best PracticesOvertuningWrapping the Target AlgorithmGeneral Advice
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 135
![Page 258: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/258.jpg)
Choosing the Training Instances #1
Split instance set into training and test sets
Configure on the training instances → configuration θ
Run (only) θ on the test instances → unbiased performance estimate
Pitfall
Configuring on your test instances
→ overtuning effects – no unbiased performance estimate
Correct
Fine practice: do multiple configuration runs and pick the θ with the besttraining performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 136
![Page 259: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/259.jpg)
Choosing the Training Instances #1
Split instance set into training and test sets
Configure on the training instances → configuration θ
Run (only) θ on the test instances → unbiased performance estimate
Pitfall
Configuring on your test instances
→ overtuning effects – no unbiased performance estimate
Correct
Fine practice: do multiple configuration runs and pick the θ with the besttraining performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 136
![Page 260: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/260.jpg)
Choosing the Training Instances #1
Split instance set into training and test sets
Configure on the training instances → configuration θ
Run (only) θ on the test instances → unbiased performance estimate
Pitfall
Configuring on your test instances
→ overtuning effects – no unbiased performance estimate
Correct
Fine practice: do multiple configuration runs and pick the θ with the besttraining performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 136
![Page 261: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/261.jpg)
Choosing the Training Instances #2
AC works much better on homogeneous instance sets
Instances have something in common
E.g., come from the same problem domainE.g., use the same encoding
One configuration likely to perform well on all instances
Pitfall
Configuration on too heterogeneous sets
There often is no single great overall configuration(see advanced topics for combinations with algorithm selection)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 137
![Page 262: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/262.jpg)
Choosing the Training Instances #2
AC works much better on homogeneous instance sets
Instances have something in common
E.g., come from the same problem domainE.g., use the same encoding
One configuration likely to perform well on all instances
Pitfall
Configuration on too heterogeneous sets
There often is no single great overall configuration(see advanced topics for combinations with algorithm selection)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 137
![Page 263: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/263.jpg)
Choosing the Training Instances: Recommendation
Representative instances
Representative of the instances you want to solve later
Moderately hard instances
Too hard: will not solve many instances, no traction
Too easy: will results generalize to harder instances?
Rule of thumb: mix of hardness ranges
Roughly 75% instances solvable by default in maximal cutoff time
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 138
![Page 264: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/264.jpg)
Choosing the Training Instances: Recommendation
Representative instances
Representative of the instances you want to solve later
Moderately hard instances
Too hard: will not solve many instances, no traction
Too easy: will results generalize to harder instances?
Rule of thumb: mix of hardness ranges
Roughly 75% instances solvable by default in maximal cutoff time
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 138
![Page 265: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/265.jpg)
Choosing the Training Instances: Recommendation
Representative instances
Representative of the instances you want to solve later
Moderately hard instances
Too hard: will not solve many instances, no traction
Too easy: will results generalize to harder instances?
Rule of thumb: mix of hardness ranges
Roughly 75% instances solvable by default in maximal cutoff time
Enough instances
The more training instances the better
Very homogeneous instance sets: 50 instances might suffice
Preferably ≥ 300 instances, better even ≥ 1000 instances
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 138
![Page 266: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/266.jpg)
Using parallel computation
Simplest method: use multiple independent configurator runs
This can work very well [Hutter et al, LION 2012]
FocusedILS: basically linear speedups with up to 16 runs
SMAC: about 8-fold speedup with 16 runs
Distributed SMAC (d-SMAC) [Hutter et al, LION 2012]
Up to 50-fold speedups with 64 workers
- But so far synchronous parallelization
- Not applicable for runtime optimization
Parallel SMAC (p-SMAC) [unpublished]
Simple asynchronous scheme
Simply exectue k different SMAC runs with different seeds
Add --shared-model-mode true
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 139
![Page 267: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/267.jpg)
Using parallel computation
Simplest method: use multiple independent configurator runs
This can work very well [Hutter et al, LION 2012]
FocusedILS: basically linear speedups with up to 16 runs
SMAC: about 8-fold speedup with 16 runs
Distributed SMAC (d-SMAC) [Hutter et al, LION 2012]
Up to 50-fold speedups with 64 workers
- But so far synchronous parallelization
- Not applicable for runtime optimization
Parallel SMAC (p-SMAC) [unpublished]
Simple asynchronous scheme
Simply exectue k different SMAC runs with different seeds
Add --shared-model-mode true
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 139
![Page 268: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/268.jpg)
Using parallel computation
Simplest method: use multiple independent configurator runs
This can work very well [Hutter et al, LION 2012]
FocusedILS: basically linear speedups with up to 16 runs
SMAC: about 8-fold speedup with 16 runs
Distributed SMAC (d-SMAC) [Hutter et al, LION 2012]
Up to 50-fold speedups with 64 workers
- But so far synchronous parallelization
- Not applicable for runtime optimization
Parallel SMAC (p-SMAC) [unpublished]
Simple asynchronous scheme
Simply exectue k different SMAC runs with different seeds
Add --shared-model-mode true
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 139
![Page 269: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/269.jpg)
Further tips and tricks with SMAC
There is extensive documentation
http://aclib.net/smac
Quickstart guide, FAQ, extensive manual
E.g., resuming SMAC runs, warmstarting with previous runs, etc.
Ask questions in the SMAC Forum
https://groups.google.com/forum/#!forum/smac-forum
It can also help to read through others’ issues and solutions
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 140
![Page 270: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/270.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced TopicsAlgorithm Configuration on Heterogeneous DataMore on Automated Machine Learning
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 141
![Page 271: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/271.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced TopicsAlgorithm Configuration on Heterogeneous DataMore on Automated Machine Learning
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 142
![Page 272: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/272.jpg)
Algorithm Configuration on Heterogeneous Instances
Known Problem
Algorithm configuration only performs well on homogeneous instance sets
It only aims to find the single best configuration
For heterogeneous instances, there might not be a single greatconfiguration
- That’s why algorithm portfolios are so successful
Known Solution: Combine Algorithm Configuration & Portfolios
1 Use algorithm configuration to determine a set of complementaryconfigurations
2 Build a portfolio out of these
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 143
![Page 273: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/273.jpg)
Algorithm Configuration on Heterogeneous Instances
Known Problem
Algorithm configuration only performs well on homogeneous instance sets
It only aims to find the single best configuration
For heterogeneous instances, there might not be a single greatconfiguration
- That’s why algorithm portfolios are so successful
Known Solution: Combine Algorithm Configuration & Portfolios
1 Use algorithm configuration to determine a set of complementaryconfigurations
2 Build a portfolio out of these
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 143
![Page 274: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/274.jpg)
Manual Expert Approach
Basic Assumption
Heterogeneous instance set can be divided into homogeneous subsets
Manual Expert
An expert knows the homogeneous subsets (e.g., origin of instances)
Determine a well-performing configuration on each subset
E.g., using algorithm configuration portfolio of configurations
Use algorithm selection to select the right configuration for eachinstance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 144
![Page 275: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/275.jpg)
Manual Expert Approach
Basic Assumption
Heterogeneous instance set can be divided into homogeneous subsets
Manual Expert
An expert knows the homogeneous subsets (e.g., origin of instances)
Determine a well-performing configuration on each subset
E.g., using algorithm configuration portfolio of configurations
Use algorithm selection to select the right configuration for eachinstance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 144
![Page 276: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/276.jpg)
Instance-Specific Algorithm Configuration: ISAC[Kadioglu et al. 2010]
Idea
Training:
1 Cluster instances into homogeneous subsets (using g-means in theinstance feature space)
2 Apply algorithm configuration on each instance set
Test:
1 Determine the nearest cluster (k-NN with k = 1) in feature space
2 Apply optimized configuration of this cluster
Assumes that instances with similar features can be solved well by similarconfigurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 145
![Page 277: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/277.jpg)
Instance-Specific Algorithm Configuration: ISAC[Kadioglu et al. 2010]
Idea
Training:
1 Cluster instances into homogeneous subsets (using g-means in theinstance feature space)
2 Apply algorithm configuration on each instance set
Test:
1 Determine the nearest cluster (k-NN with k = 1) in feature space
2 Apply optimized configuration of this cluster
Assumes that instances with similar features can be solved well by similarconfigurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 145
![Page 278: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/278.jpg)
Instance-Specific Algorithm Configuration: ISAC[Kadioglu et al. 2010]
Idea
Training:
1 Cluster instances into homogeneous subsets (using g-means in theinstance feature space)
2 Apply algorithm configuration on each instance set
Test:
1 Determine the nearest cluster (k-NN with k = 1) in feature space
2 Apply optimized configuration of this cluster
Assumes that instances with similar features can be solved well by similarconfigurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 145
![Page 279: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/279.jpg)
ISAC+ [Malitsky et al. 2014]
Observations
No need to restrict selection to the configuration found on a cluster
Arbitrary algorithm selection approach possible
Idea1 Cluster instances
2 Apply algorithm configuration on each cluster
3 Build a portfolio out of all these configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 146
![Page 280: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/280.jpg)
ISAC+ [Malitsky et al. 2014]
Observations
No need to restrict selection to the configuration found on a cluster
Arbitrary algorithm selection approach possible
Idea1 Cluster instances
2 Apply algorithm configuration on each cluster
3 Build a portfolio out of all these configurations
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 146
![Page 281: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/281.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
P =
Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 282: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/282.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
P =
Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 283: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/283.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
AssessA(θ) on i
P ={}
{}Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 284: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/284.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
AssessA(θ) on i
P ={θ1}
θ1
Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 285: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/285.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
AssessA(θ||θ1) on i
P ={θ1}
{θ1}Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 286: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/286.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
AssessA(θ||θ1) on i
P ={θ1, θ2}
θ2
Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 287: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/287.jpg)
Hydra [Xu et al. 2010]
Idea
Iteratively add configurations to a portfolio P, start with P = ∅In each iteration: add a configuration that is complementary to P
Marginal contribution of a configuration θ to a portfolio P
m(P)−m(P ∪ {θ})
Configuration Task
Instances I
Algorithm A andits Configuration
Space Θ
Select θ ∈ θand i ∈ I
AssessA(θ||θ1||θ2) on i
P ={θ1, θ2}
{θ1, θ2}Return Performance
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 147
![Page 288: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/288.jpg)
Cedalion [J. Seipp et al 2014]
Idea
Optimize a schedule of configurations with algorithm configuration
Approach
Iteratively add a configuration with a time slot t to a scheduleS ⊕ 〈θ, t〉The time slot is a further parameter in the configuration space
Optimize marginal contribution per time spent:
m(S)−m(S ⊕ 〈θ, t〉)t
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 148
![Page 289: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/289.jpg)
Cedalion [J. Seipp et al 2014]
Idea
Optimize a schedule of configurations with algorithm configuration
Approach
Iteratively add a configuration with a time slot t to a scheduleS ⊕ 〈θ, t〉
The time slot is a further parameter in the configuration space
Optimize marginal contribution per time spent:
m(S)−m(S ⊕ 〈θ, t〉)t
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 148
![Page 290: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/290.jpg)
Cedalion [J. Seipp et al 2014]
Idea
Optimize a schedule of configurations with algorithm configuration
Approach
Iteratively add a configuration with a time slot t to a scheduleS ⊕ 〈θ, t〉The time slot is a further parameter in the configuration space
Optimize marginal contribution per time spent:
m(S)−m(S ⊕ 〈θ, t〉)t
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 148
![Page 291: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/291.jpg)
Submodularity
Oberservation
Performance metrics of Hydra and Cedalion are submodular
Diminishing returns: a configuration improves a smaller portfolio morethan a larger one
Definition (Submodularity of f)
For every X,Y with X ⊆ Y and every x we have thatf(X ∪ {x})− f(X) ≥ f(Y ∪ {x})− f(Y )
Advantage
We can bound the error of the portfolio/schedule(see [Streeter & Golovin ’07]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 149
![Page 292: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/292.jpg)
Submodularity
Oberservation
Performance metrics of Hydra and Cedalion are submodular
Diminishing returns: a configuration improves a smaller portfolio morethan a larger one
Definition (Submodularity of f)
For every X,Y with X ⊆ Y and every x we have thatf(X ∪ {x})− f(X) ≥ f(Y ∪ {x})− f(Y )
Advantage
We can bound the error of the portfolio/schedule(see [Streeter & Golovin ’07]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 149
![Page 293: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/293.jpg)
Submodularity
Oberservation
Performance metrics of Hydra and Cedalion are submodular
Diminishing returns: a configuration improves a smaller portfolio morethan a larger one
Definition (Submodularity of f)
For every X,Y with X ⊆ Y and every x we have thatf(X ∪ {x})− f(X) ≥ f(Y ∪ {x})− f(Y )
Advantage
We can bound the error of the portfolio/schedule(see [Streeter & Golovin ’07]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 149
![Page 294: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/294.jpg)
Further Combinations
Algorithms andConfigurations
AlgorithmSelection
AlgorithmSchedule
AlgorithmConfiguration
Selection of Configurators
Configuration of Selectors
Selection ofSchedules
Schedules ofSelectors
Schedules ofConfigurators
Configurationof Schedules
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 150
![Page 295: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/295.jpg)
Further Combinations
ACPP: Automatic construction of a parallel portfolio solver[Hoos et al. 2012]
AutoFolio: configuration of an algorithm selector[Lindauer et al. 2015]
Sunny : Predict an algorithm schedule for a given instance[Amadini et al. ’13-’15]
Predict the best configuration for a given instance (e.g., [Bossek et al.
2015])
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 151
![Page 296: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/296.jpg)
Outline
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced TopicsAlgorithm Configuration on Heterogeneous DataMore on Automated Machine Learning
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 152
![Page 297: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/297.jpg)
Auto-Sklearn
Extensions to AutoWEKA’s AutoML approach
Meta-learning to warmstart Bayesian optimization
Automated posthoc ensemble construction to combine the modelsBayesian optimization evaluated
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 153
![Page 298: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/298.jpg)
Hands-on: Auto-sklearn
In your virtual machine:
Run a linear SVM and Auto-sklearn
$ cd AC-Tutorial/auto-sklearn/
$ vim autosklearn-example_restricted_to_svc.py
$ python autosklearn-example_restricted_to_svc.py
$ vim autosklearn-example_with_cv.py
$ python autosklearn-example_with_cv.py
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 154
![Page 299: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/299.jpg)
Auto-sklearn: Ready for Prime Time
State-of-the-art AutoML framework
Best approach in ongoing ChaLearn AutoML challenge
Outperformed 100s of teams of human experts
Trivial to useimport autosklearn.classification as cls
automl = cls.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
Availabe online
https://github.com/automl/auto-sklearn
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 155
![Page 300: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/300.jpg)
Auto-sklearn: Ready for Prime Time
State-of-the-art AutoML framework
Best approach in ongoing ChaLearn AutoML challenge
Outperformed 100s of teams of human experts
Trivial to useimport autosklearn.classification as cls
automl = cls.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
Availabe online
https://github.com/automl/auto-sklearn
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 155
![Page 301: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/301.jpg)
Auto-sklearn: Ready for Prime Time
State-of-the-art AutoML framework
Best approach in ongoing ChaLearn AutoML challenge
Outperformed 100s of teams of human experts
Trivial to useimport autosklearn.classification as cls
automl = cls.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
Availabe online
https://github.com/automl/auto-sklearn
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 155
![Page 302: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/302.jpg)
Algorithm configuration also applies to deep learning
State of the art for deep network hyperparameter optimization
For few continuous hyperparameters: Bayesian optimization methodsbased on Gaussian processes perform best
Many/discrete/conditional hyperparameters: SMAC performs best
References: [Eggensperger et al, BayesOpt 2013], [Domhan et al, IJCAI 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 156
![Page 303: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/303.jpg)
Algorithm configuration also applies to deep learning
State of the art for deep network hyperparameter optimization
For few continuous hyperparameters: Bayesian optimization methodsbased on Gaussian processes perform best
Many/discrete/conditional hyperparameters: SMAC performs best
References: [Eggensperger et al, BayesOpt 2013], [Domhan et al, IJCAI 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 156
![Page 304: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/304.jpg)
Time-permitting: extrapolating learning curves ofstochastic gradient descent performance to later time steps
𝑦1:𝑛 train
𝑦𝑙𝑎𝑠𝑡 test
Observe initial part of learning curve
Probabilisticially predict remainder of learning curve
Reference: [Domhan et al, IJCAI 2015]
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 157
![Page 305: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/305.jpg)
Examples of SGD learning curves in deep learning
Learning curves for 1000hyperparameter settings
Learning curves for a randomsample of these
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 158
![Page 306: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/306.jpg)
Idea for extrapolation: fitting a parametric function
Increasing, saturating functions
E.g., pow3 function with 3 parameters c, a, αy = pow3(x | c, a, α) = c− ax−α (where x=epoch, y=accuracy)
Fitting those parameters: optimize fit using gradient-based methods
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 159
![Page 307: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/307.jpg)
Advanced variants of extrapolation
No single parametric family performs best
We fit convex combinations of 11 different parametric families
In practice, we’d also like uncertainty estimatesOptimizing parameters yields a single fitSampling parameters yields a distribution of fits
- Sample parameters according to their posterior probability(using Markov Chain Monte Carlo; beyond scope of this course)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 160
![Page 308: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/308.jpg)
Advanced variants of extrapolation
No single parametric family performs best
We fit convex combinations of 11 different parametric families
In practice, we’d also like uncertainty estimatesOptimizing parameters yields a single fitSampling parameters yields a distribution of fits
- Sample parameters according to their posterior probability(using Markov Chain Monte Carlo; beyond scope of this course)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 160
![Page 309: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/309.jpg)
Advanced variants of extrapolation
No single parametric family performs best
We fit convex combinations of 11 different parametric families
In practice, we’d also like uncertainty estimatesOptimizing parameters yields a single fitSampling parameters yields a distribution of fits
- Sample parameters according to their posterior probability(using Markov Chain Monte Carlo; beyond scope of this course)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 160
![Page 310: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/310.jpg)
Qualitative results
Learning curves for 1000hyperparameter settings
Bad curves are terminated early
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 161
![Page 311: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/311.jpg)
Quantitative results
2-fold speedup of DNN structure & hyperparameter optimization
For several network architectures, including state-of-the-art
For several optimizers (SMAC, TPE, random search)
New state-of-the-art for CIFAR without data augmentation
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 162
![Page 312: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/312.jpg)
Summary
1 The Algorithm Configuration Problem
2 Using AC Systems
3 Importance of Parameters
4 Pitfalls and Best Practices
5 Advanced Topics
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 163
![Page 313: Algorithm Configuration: A Hands-on Tutorial · 2016. 2. 13. · Algorithm Con guration: A Hands-on Tutorial Frank Hutter Marius Lindauer University of Freiburg AAAI 2016, Phoenix,](https://reader036.vdocuments.site/reader036/viewer/2022071411/6106ba7cb00d0517e94ecf0d/html5/thumbnails/313.jpg)
Take-away messages
Algorithm configuration is very versatile
Improves results, increases productivity
Enables automated machine learning
What you need to use algorithm configuration
An algorithm with exposed parameters
An instance distribution/set
A performance metric you care about
You know about field X. What can you contribute?
Case study applying algorithm configuration to X
Construct better features for X
Build a system: Auto-X (like Auto-sklearn)
Hutter & Lindauer AC-Tutorial AAAI 2016, Phoenix, USA 164