Welcome to the webinarMigrating models from other solvers to use the Gurobi solver
Renan Garcia, Ph.D.
Optimization Support Engineer at Gurobi Optimization
} Ph.D. in Industrial and Systems Engineering, Georgia Tech} Expert in optimization modeling and software development} Over a decade of experience implementing decision support systems
© 2016 Gurobi Optimization
David Nehme, Ph.D.
Principal Consultant at Abremod
} 20 years of software development experience} 15 years implementing CPLEX and Gurobi models} Ph.D. in Operations Research for The University of Texas} Top StackOverflow answerer for both CPLEX and Gurobi tags
© 2016 Gurobi Optimization
Agenda
} Switching is easier than you may think
} Common migration scenarios
} Migrating from OPL
} Migrating from CPLEX Concert API
} Best practices, limitations and considerations
© 2016 Gurobi Optimization
© 2016 Gurobi Optimization
Switching is Easier Than You May Think
Key Migration Issues
} Building the model◦ How do I build my optimization model?◦ Do I build it one constraint at a time, or do I build an entire constraint
matrix?
} Setting solver parameters◦ What solver parameters do I change?◦ What effects are these changes intended to produce?◦ Am I looking for an optimal solution, or just a good feasible solution?
} Computing and extracting the solution◦ How do I extract the solution produced by the solver?
© 2016 Gurobi Optimization
Interfacing with Gurobi
} Command-Line Tool
} Full-featured Interactive Shell
} Matrix-oriented APIs◦ C, MATLAB, R
} Object-oriented APIs◦ C++, Java, .NET, Python
} Modeling systems◦ Commercial: AIMMS, AMPL, Frontline Solvers, GAMS, MPL, …◦ Free: CMPL, JuliaOpt, OSI, PuLP, PYOMO, SolverStudio, YALMIP, …
© 2016 Gurobi Optimization
Do-It-Yourself Resources on www.gurobi.com
} Switching guidance:https://www.gurobi.com/resources/switching-to-gurobi/switching-overview
} Documentation:https://www.gurobi.com/documentation/◦ Quick Start Guides◦ Reference Manual
� APIs� Attributes� Parameters� Tuning� …
◦ Example Tour� 22 functional examples
} Seminars and videos:https://www.gurobi.com/resources/seminars-and-videos/seminars-videos
© 2016 Gurobi Optimization
© 2016 Gurobi Optimization
Common Migration Scenarios
Scenario 1: Migrating with Model Files
} Example: performance testing
} Export model file with virtually no changes needed to existing code ◦ Gurobi supports several file formats (MPS, LP, …)
} Export guidance:https://www.gurobi.com/resources/switching-to-gurobi/exporting-mps-files-from-competing-solvers
} Use Gurobi Command-Line Tool to solve the model ◦ Usage: gurobi_cl [parameters] filename◦ Ex: gurobi_cl TimeLimit=3600 misc07.mps
} "Quick-and-dirty" approach◦ Limited ability to interact with solver (parameters only)◦ For more control, try Interactive Shell
© 2016 Gurobi Optimization
Gurobi Parameters
} Control algorithmic behavior◦ Defaults validated against large
internal test set of models
} Termination criteria◦ Ex. TimeLimit, SolutionLimit
} Tolerances◦ Ex. MIPGap, BarConvTol
} Simplex and barrier◦ Ex. Method, Crossover
} MIP◦ Ex. MIPFocus, ImproveStartTime
} MIP cuts◦ Ex. Cuts, GomoryPasses
} Tuning and distributed algorithms◦ Ex. TuneJobs, WorkerPool
} General◦ Ex. Presolve, OutputFlag
© 2016 Gurobi Optimization
Scenario 2: Using a Modeling System
} Example: model is written in AMPL
} Migrating is extremely easy for solver-independent systems
◦ Obtain license◦ Set solver to Gurobi◦ Convert parameter settings
◦ Ex: in AMPL model file, addoption solver gurobi_ampl;
option gurobi_options 'mipfocus 1';
} Need to migrate your existing model code for single-solver systems◦ OPL, Mosel, …
© 2016 Gurobi Optimization
Gurobi Python Environment
} High-level optimization modeling constructs embedded in Python programming language
} Combines expressiveness of a modeling language with the power and flexibility of a programming language◦ Bring "feel" of a modeling language to the Python interface
} Requires minimal programming skills to get started
} Support all solver and programming needs
} Several seminars on this topic:https://www.gurobi.com/resources/seminars-and-videos/seminars-videos
© 2016 Gurobi Optimization
Interactive Examples
© 2016 Gurobi Optimization
Scenario 3: Porting Matrix-Oriented Code
} Example: C program which calls CPLEX Callable Library, Xpress, …
} Gurobi's C API supports sparse matrix format◦ Standard format used by many solvers
� Simple arrays represent matrix coefficients and their index positions◦ Ex: GRBaddconstrs(), GRBaddvars()◦ Minimal changes required to existing code
} Gurobi also supports advanced features◦ Callbacks◦ Advanced simplex routines (querying tableau rows)◦ …
} Must consider some Gurobi-specific modeling features when porting existing code◦ Gurobi environments◦ Lazy updates◦ Attributes
© 2016 Gurobi Optimization
Gurobi Environments
} Models are built from an environment
} Parameters are set on an environment
} Models get their own copy of the environment ◦ Once a model is created, subsequent parameter changes in parent
environment are not reflected in model environment ◦ Use getEnv() functions to get the environment from model
} Setting parameters in C ◦ Ex: set time limit of 3600 seconds for parent environment
status = GRBsetdblparam(env, "TimeLimit", 3600); ◦ Ex: set presolve level to 2 for model's environment
status = GRBsetintparam(GRBgetenv(model), "Presolve", 2);
© 2016 Gurobi Optimization
Lazy Updates
} Gurobi updates models in batch mode◦ Model creation and updates are efficient
} Must call update() functions to use model elements ◦ Ex: Call after creating a variable before using it in a constraint ◦ May require changes to code for other solvers
} UpdateMode=1 parameter setting allows you to use elements immediately
© 2016 Gurobi Optimization
Attributes
} Unified system to access model elements◦ Attributes work the same across all Gurobi interfaces
} Access via a basic set of get/set functions◦ Attribute name is specified as a parameter◦ Replaces many functions used by other solvers
} Getting/Setting attributes in C◦ Use get/set functions by type (int, double, char, string)◦ Ex: query number of nonzeros in a model
status = GRBgetintattr(model, "NumNZs", &nzs);◦ Ex: query solution vector
double x[NUMVARS];status = GRBgetdblattrarray(model, "X", 0, NUMVARS, x);
◦ Ex: modify constraint RHS to 1status = GRBsetdblattrelement(model, "RHS", cidx, 1.0);
© 2016 Gurobi Optimization
Scenario 4: Porting Object-Oriented Code
} Example: Java program which uses CPLEX Concert Technology
} Gurobi's OO APIs represent models using objects◦ Objects for variables and constraints ◦ Methods are used to create model elements◦ Ex: add simple constraint x + y ≥ 1 in C++
c1 = model.addConstr(x + y >= 1, "c1");
} All Gurobi APIs are just thin layers on top of same native C code
} Must consider same Gurobi-specific modeling features when porting◦ Subsequent parameter changes in parent environment not reflected in
model environment � Java Ex: model.getEnv().set(GRB.IntParam.Presolve, 2);
◦ Must call model's update() method to use elements (unless UpdateMode=1)◦ Use get/set methods on objects to access attributes
� Python Ex: constr.setAttr('RHS', 1.0)
© 2016 Gurobi Optimization
Don't Panic, Your Code Often Looks Similar
} CPLEX Concert C++:IloEnv env; // create empty environmentIloModel model(env); // create empty model
IloNumVar x(env, 0, 10, ILOINT); // add variables
model.add(x);
// ...
model.add(x + 2*y <= 1); // add constraints// ...
} Gurobi C++:GRBEnv env; // create empty environment
env.set(GRB_IntParam_UpdateMode, 1); // enable automatic updates
GRBModel model(env); // create empty model
GRBVar x = model.addVar(0, 10, 0, GRB_INTEGER); // add variables
// ...model.addConstr(x + 2*y <= 1); // add constraints
// ...
© 2016 Gurobi Optimization
Solver-specific Guidance
© 2016 Gurobi Optimization
© 2016 Gurobi Optimization
} Cover two disparate examples◦ OPL ◦ ILOG Concert� Have a C#, and Java Adapters� Working on C++◦ Showing Two approaches� Translating OPL� Using an Adapter for Concert
} General advice on the migrating
Agenda
© 2016 Gurobi Optimization
Switching To Gurobi
} Hurdles◦ Large code base◦ Business Logic embedded in API calls
} Mitigating Factors◦ The time to migrate is not proportional to the size of the
codebase◦ With mps files, you can see how the Gurobi will perform on
your specific models.
© 2016 Gurobi Optimization
Before You Migrate
} Current State◦ Actively adding features?◦ Maintenance only?◦ Do you have regression tests?
} MPS is your friend◦ Evaluate relative performance
} LP is also your friend◦ Name your variables and constraints◦ Testing your code
© 2016 Gurobi Optimization
Migrating From OPL
} OPL includes a domain specific language
} Locked to a Solver (CPLEX)
} Two most likely strategies◦ Move to a solver-agnostic Language� AMPL� AIMMS, GAMS, MPL◦ Migrate to Python
© 2016 Gurobi Optimization
Migrating from OPL to Python
} Why Python
◦ Best of Both worlds� Powerful as a General Purpose Language� As effective as a Domain Specific Language
� Concise� Readable� Learnable
� See Stackoverflow� (http://bit.ly/1QEF0fq)
◦ Huge user base
© 2016 Gurobi Optimization
OPL to Python
} Feature Comparison
OPL PythonTuples, Sets Tuples, SetsRead from Excel openpyxl, xlrdRead from SQL SqlalchemySlicing, Grouping PandasUI Jupyter Notebook.dat format JSONOPLScript Python
© 2016 Gurobi Optimization
} Installing Python◦ Use the Anaconda Distribution� Especially great on Windows� Easy installation for Gurobi Libraries
� http://www.gurobi.com/downloads/get-anaconda
OPL To Python
© 2016 Gurobi Optimization
OPL To Python
} Migrate a complete example
} From a book ◦ Planning and Scheduling in Manufacturing and Services◦ OPL Code is freely available� http://bit.ly/20VpATz◦ Reads from OPL dat file
} Python Code Available on github
© 2016 Gurobi Optimization
} Translate the “.dat” file to JSON
◦ JSON is a standard� Almost all programming languages have readers� OPL reads it� Native format for MongoDB� Many application have JSON exporters
◦ Simple Python script to translate� Most dat files
� without embedded logic� Using PyParsing
� Another library included with Anaconda
Approach
© 2016 Gurobi Optimization
OPL Data
Demand = #[ 2: #[
1: [ 20000 30000 15000 40000 ] 2: [ 0 50000 30000 50000 ]
]#3: #[
1: [ 10000 5000 15000 40000 ] 2: [ 0 10000 0 5000 ] ]#
]#;
"Demand": {"2": {
"Product 1": [ 20000, 30000, 15000, 40000 ],"Product 2": [ 0, 50000, 30000, 50000 ]
},"3": {
"Product 1": [ 10000, 5000, 15000, 40000 ],"Product 2": [ 0, 10000, 0, 5000 ]
}
} Can Translate with Python Script
© 2016 Gurobi Optimization
OPL Model
} Data DeclarationsLang. StatementOPL int RequiredLotSize = ...;
Python RequiredLotSize = d['RequiredLotSize']
OPL float Demand[Stages, Products, Periods] = ...;Python Demand = series_from_json(d['Demand'],
[Stages, Products, Periods])OPL range Products = 1..2;Python Products = ['Product 1', 'Product 2']
© 2016 Gurobi Optimization
OPL Model
} Variable Declarations
dvar float+ x[Factories, Products, Periods];dvar int+ y[Factories, Stages, Products, Periods] in 0..maxint;dvar float+ z[Products, Periods];dvar float+ q2[Products, ZPeriods];dvar float+ v2[Products, Periods];dvar float+ v3[Products, ZPeriods];dvar boolean yb[bnds, Periods];
© 2016 Gurobi Optimization
VariablesOPL dvar float+ x[Factories, Products, Periods];Python x = get_vars('x', Factories, Products, Periods)OPL dvar boolean yb[bnds, Periods];Python yb = get_vars('yb', bnds.index, Periods,
vtype=GRB.BINARY)
© 2016 Gurobi Optimization
Objective function
} OPL} minimize} sum (t in Periods, j in Products, } i in Factories) } ProdCost[i,j]*x[i,j,t]
} Python} model.setObjective(} grb.quicksum([ProdCost[i, j] * x[i, j, t]} for i in Factories} for j in Products} for t in Periods])
© 2016 Gurobi Optimization
Constraints
OPL if (RequiredLotSize > 0)forall (i in Factories, j in Products, t in Periods)
x[i,j,t] == RequiredLotSize*xlots[i,j,t];else
forall (i in Factories, j in Products, t in Periods)xlots[i,j,t] == 0;
Python if RequiredLotSize > 0:[addConstr(x[i, j, t] == RequiredLotSize * xlots[i, j, t])for i in Factories for j in Products for t in Periods]
else:[addConstr(xlots[i, j, t] == 0)for i in Factories for j in Products for t in Periods]
© 2016 Gurobi Optimization
Results From SolverOPL float TotProdCost = sum (t in Periods,
j in Products, i in Factories)
ProdCost[i,j]*x[i,j,t];Python TotProdCost = sum([ProdCost[i, j] * x[i, j, t].x
for t in Periodsfor j in Productsfor i in Factories])
© 2016 Gurobi Optimization
OPL to Python
} Consider moving to Python} If you still like modeling languages, try AMPL
© 2016 Gurobi Optimization
Migration Library for CPLEX Concert API
© 2016 Gurobi Optimization
Migrating From Concert
} Concert◦ C++ API developed in late 90s◦ Java and C# versions followed◦ Good option for using mainstream languages
© 2016 Gurobi Optimization
Migrating from Concert
} Two Strategies◦ Translate code◦ Use an Adapter
© 2016 Gurobi Optimization
} Pattern◦ Object Adapter Pattern
} Physical Examples◦ Electric plugs◦ CO2 Filter on Apollo 13
Adapter
© 2016 Gurobi Optimization
Adapter Approach
} Give applications written against the Concert API, access to the Gurobi solver
} Our Adapter◦ Not a complete implementation of Concert� Most applications use a small subset◦ Enough to make a model run◦ Free to use� Starting point for your application
© 2016 Gurobi Optimization
Adapter Approach
} Working Concert Application
} Unlink CPLEX / Concert libraries
} Add Gurobi Library
} Add adapter library◦ Modify adapter for your code
© 2016 Gurobi Optimization
Example
} Makefile◦ CPPLIB = … -lilo_grb … -lgurobi_c++ -lgurobi65
} Eclipse
© 2016 Gurobi Optimization
Adapter Library
} Available for Java and C#
} Working on C++
} Available on Github◦ https://github.com/abremod/concert2gurobi4cs
© 2016 Gurobi Optimization
Adapter Library
} “Object Adapter”, not a “Class Adapter”} Favor Composition over Inheritance
◦ public class IloCplex extends ilog.concert.Algorithmimplements IloMPModeler {
◦ GRBEnv env;
◦ GRBModel model;◦ private boolean vars_synced = true;◦ private boolean constrs_synced = true;
© 2016 Gurobi Optimization
Adapter Library
} C++ specific
◦ IloXXXX objects are handles� “Pimpl” ◦ Boost shared_ptr<> covers handle functionality◦ class IloRange : public IloExtractable {◦ private:◦ class Impl;
◦ boost::shared_ptr<Impl> _impl;◦ public:◦ IloRange(IloEnv env, double lb, double ub, const char *name=0);
◦ IloColumn operator()(IloNum);
◦ };
© 2016 Gurobi Optimization
Limitations
} Adapter probably won’t work out of the box
} Variables tied to models, not environments◦ No “not extracted exceptions”
} Limited Support for Callbacks
} No support for Goals
So what happens next?
} Request a free evaluation license, if you have not already done so◦ [email protected]◦ [email protected]
} Set up a free consultation with a consultant from [email protected]
} Webinar slides: Will be available in the next day or two
} Abremod tools available from: https://github.com/abremod/ilogrb
} Webinar recording: Will be available next week