Canonical Duality Theory for Solving General Canonical Duality Theory for Solving General Mixed Integer Nonlinear Programming Problems Mixed Integer Nonlinear Programming Problems

with Applications with Applications

David Gao

Alex Rubinov Prof. of Mathematics, Federation University

Research Prof. of Eng. Science, Australian National University

Supported by US Air Force AFOSR grants

Since 2008

1. Duality Gap between Math and Physics conceptual problems

3. Challenges Breakthrough

2. Canonical Duality-Triality:

Unified Modeling Unified Solutions

MINLP 2014, Carnegie Mellon, June 2-5, 2014

Gap between Math and MechanicsGap between Math and MechanicsNonlinear/Global Optimization Problem: min f (x) s.t. g(x) ≤ 0

f(x) is an “objective” functiong(x) is a general constraint.

(naive) questions: What is the objective function? target and cost? what is Lagrangian? …

“Mathematics is a part of physics. …In the middle of the twentieth century it was attempted to divide physics and mathematics. The consequences turned out to be catastrophic." — V.I. Arnold (1997)

Mathematics needs to remarry physics – A. JaffeGao-Ogden-Ratiu, Springer

Duality in mathematics is not a theorem, but a “principle” – Sir M.F. Atiyah

Duality gap is not allowed in mathematical physics!

Canonical Duality-Triality TheoryCanonical Duality-Triality Theory

A methodological theory comprises mainly

1. Canonical dual transformation

2. Complementary-Dual Principle

3. Triality Theory

Unified Modeling

Unified Solution

Identify both global and local extrema Design powerful algorithms

Unified understanding complexities

Gao-Strang, 1989 MIT and Gao, 1991 Harvard

x

PPd

min = max

max = max

min = min

Nothing is too wonderful to be true, if it be consistent with the laws of nature Michael Farady (1860 AC)

Philosophical FoundationPhilosophical Foundation I-Ching ( 2800 BC-2737 BC): The fundamental Law of Nature is the Dao : the complementarity of one yin (Ying) and one Yang

Canonical System = { (Ying, Yang) | H-Chi } = { ( X , X* ) | A }

Laozi: All things have the receptivity of the yin and the activity of the yang. Through union with the life-giving force (chi) they blend in harmony

Everything = {( Yin, Yang) ; Chi }

= { (subj. , obj.) ; verb }

Convex Canonical System: Unified ModelingConvex Canonical System: Unified Modeling

(P): min P(x) = W(Dx) - F( x)

s.t. x Xc= { xXa | Dx Ya}

W( y ) : Objective function (Gao, 2000): W( Q y ) = W( y ) QT = Q -1, det Q = 1

Exam: W(y) = ½ | y |2 , |Q y|2 = yTQTQ y = | y|2 The 2nd duality: y* = ∂ W(y) Constitutive law

F(x) = f T x Subjective function The 1st duality: x*=∂F( x) = f , action-reaction

Convex System

input

f out put

x

Xa x x* = f Xa* ∂F( x)

y* Ya*Ya yD= D D

∂W(y)

Legendre transf. W*( y*) = y y - W(y)

(Pd ): max Pd(y*) = - W*( y*) s.t. Dy* = f

Lagrangian: L(x, y*) = (Dx) y* - W*( y*) - f T x = xDy* - f ) - W*( y*)

xP

Pd

min P(x) = max Pd(y*)

frame-indifference

Objectivity, Gao 2000Objectivity, Gao 2000Objectivity is not a hypothesis, but a principle. Objectivity is not a hypothesis, but a principle. P.G. Ciarlet, P.G. Ciarlet, Nonlinear Functional AnalysNonlinear Functional Analysis, 2013, SIAMis, 2013, SIAM

Manufacturing Company System Manufacturing Company System

Workers y

CompanyProducts x

D D

Price x*

Salary y*

(P): min P(x) = W(Dx) – F( x )

cost incomeTarget(Lose)

Xa

Ya

Xa*

Ya*

F( x ) = xT x*

Unified Understanding Constraints (Unified Understanding Constraints (Gao, 1997Gao, 1997))(P): min P(x) = W(Dx) – U( x) W(y) : Ya = { y Y | g(y) ≥ 0 } physically feasible U(x) : Xa = { x X | Bx ≤ 0 } geometrically feasible

Boundary (external) constraints in Xa

external KKT conditions

0 ≥ Bx = u ┴ u* = B*x* ≥ 0 Constitutive (objective) constraints in Ya

internal KKT conditions

0 ≤ g(y) = ┴ * = g*(y*) ≤ 0

Xa

Ya*

Xa*

Ya

D=D

Dmn

( y; y*

(x, x*

0 ≤ * ≤ 0

g(y)

┴ *

g*(y*)

0 ≥ u u* ≥ 0

uBx

u ┴ u*

uB*x*

W (y) = {

∂W constitutive law and KKT conditions

W(y) if g(y) ≥ 0 ∞ otherwise

Indicator ( J-J Moreau, 1963)

(P): min P(x) = W(Dx) – U( x) , x X = Obj. – Subj.

Math = { ( X, X* ) ; A}

Canonical Duality - Triality TheoryCanonical Duality - Triality Theory(P): min P(x) = W(Dx) – x T f

Legendre Trans: V*( T–V

Canonical Dual: Pd() x - ½ f T G -1f - V*( 2. Complemenary-Dual Principle:

3. Triality Theory:

If c S - , then either P(xc ) = max P(x) = max Pd() = Pd(c ) (Gao, 1996) or P(xc ) = min P(x) = min Pd() = Pd(c )

Total complementary function (Gao-Strang, 1989) (x, ) = x) T - V*() – x T f

Gap function

1. Canonical transf.

Let S+ = {G 0 }

If c is a critical point of Pd(), then xc = G c -1f is a critical solution of (P) and P(xc ) =Pdc

G-Strang (1989) If c S+, then P(xc ) = min P(x) = max Pd() = Pd(c )

choose an objective measure =x) W(D x) = V((x)) convex in canonical dual eqn (one-to-one): ∂ V ( )

(Quadratic ) = ½ x T G()x - V*() – x T f ∂xAnalytic solution: x = G -1f

S- = {G < 0 }

D

y*y

D*

x x*

* = ∂V

y

y∂W

*

t*

Nonconvex W(y)

Example: Nonconvex in RExample: Nonconvex in Rn Convex in Convex in RR1

P(x) = W(Dx) – F(x) = ½( ½ |x|2 - 1 )2 – x T f

Complementary-Dual Principle: Analytic solutions: xk = (k) -1 f P(xk ) = Pd(k ) k =1,2,3

x

PPd =½ |x|2 V() = ½ ( – 1 )2

Pd() = -½ | f |½ | f | 2

-1- ½ 2 -

∂Pd() = 0 2 (+ 1) = ½ | f |2

3 ≤ 2 ≤ 0 ≤ 1

f

n=2: Mexican hat

y

W(y) = ½ ( ½ y 2 - 1)2

Triality Theory: P(x1 ) = Pd(1 ) P(x2 ) = Pd(2 ) P(x3 ) = Pd(3) Open Problem (2003): If dim x ≠ dim P(x2 ) = min P(x) ≠ min Pd(= Pd(2 )

n=1: double-well = ∂ V() = - 1

Solved in 2012 f = 0 Multiple solution x

P

4

Pd

Perturbation: f ≠ 0 Unique solution

Buridan’s donkey

Quadratic Boolean ProgrammingQuadratic Boolean Programming(P): min P(x) = ½ xTAx – f T x s.t. x {-1,1}n

(Pd): max Pd() = - ½ f T [G( ) ]-1 f – i s.t. S + = { Rn | ≥ 0, G( ) 0 }

Canonical transformation: i = x i 2 – 1 ≤ 0

(x, P(x) + ixi 2 - 1 ) = ½ x TG() x - i-

f T x G ()A+2 Diag (

i ≠ 0 xi2 =1 integer!

KKT: i ≥ 0 , i = xi2 - 1 ≤ 0, ( xi

2 - 1 ) i = 0

Thm (Gao,2007): For each critical point c ≠ 0 ,

the vector xc = G -1(c) f {-1,1}n is a KKT point of P(x) and P(xc ) = Pd(c )

if G(c) 0 P(xc )= min P(x ) = max Pd ( ) =Pd (c ) if G(c) 0 P(xc )= min P(x ) = min Pd ( ) =Pd (c )

x(x, ) = 0 x = G() -1 f

P(x)

Pd()

(P) Could be NP-Hard if Pd ( ) has no critical point in S +

minP(x)= max

min = min

Results for Max-Cut Problem (NP-Complete) Wang-Fang-Gao-Xing (2012) J. Global Optimization

Comparison of the running time produced by the canonical dual approach and GW’s approach (Goemans and Williamson)

max P(x) = ½ xTAx s.t x {0,1}n

– f T x linear perturbation

(Pd): max Pd() = - ½ f T [G( ) ]-1 f – i s.t. G( ) ≥ 0

Max -Cut Problem (contin.)

■ Randomly produce 50 instances on graphs of sizes 20,50, 100, 150,200 and 500. The weight of each edge is uniformly from [0,10]

■ Ave ratio is the average approximate ratio, the ratio is close to 1 when the dimension increases

The 2The 2ndnd Canonical Dual for Integer Programming Canonical Dual for Integer Programming(P): min P(x) = ½ xTAx – f T x s.t. x {-1,1}n

The second canonical dual (Gao, 2009)(Pg): min Pg() = - ½ T A-1– fi - i | s.t Rn

Thm: If cis a solution of ( Pg ) , then

xc i = {

is a feasible solution of (P) and P(xc ) = Pg(c ) .

1 if fi > c i -1 if fi < c i

Nonconvex/nonsmooth minimization DIRECT method (Deterministic )

If A 0, P(xc )= minP(x)= maxPg( )= Pg(c )

If A 0, P(xc )= min P(x)= min Pg( ) = Pg(c )

P(x)

Pg( )

P(x)

Pg( )

If A = - B T B , B Rm n , Pg() = ½ T– fi - Bjij | m < n

n.m

General MINLP ProblemsGeneral MINLP Problems

(P): min P(x,y ) = W(x,y) + aT x – bT y , x Xa , y Ya

s.t. C1 x + C2 y ≤ c , D1 x + D2 y = d , Xa = {x Rn | 0 x u }, Ya = { y Zm | 0 y v } Let z = (x, y) , assume W(z ) is objective such that an objective measure =z ) and a convex V() W(z ) = V((z )) Canonical form: min P(z ) = V((z )) – f T z s.t. z Za

Mixed Integer (fixed Cost) ProblemMixed Integer (fixed Cost) Problem(with H.D. Sherali and N. Ruan)(with H.D. Sherali and N. Ruan)

(P): min P(x,y) = ½ xTA x + cT x – f T y s.t. -y ≤ x ≤ y, y { 0 , 1 }n

(Pd): maxPd() = - ½ cTG()-1c - ½ i fi )+ s.t. ≥ 0 , G() = A +2 Diag ( ) p.d.

Thm: If cis a solution of (Pd ) , then

xc = - G (c)-1 c ,

yci = {

is a global solution of (P) and P(xc , yc ) = Pd(c )

1 if fi < c i 0 if fi > c i

Applications to scheduling and decision science x Rd x n

Problems that can be solvedProblems that can be solved

Benchmark Problems:

1. Rosenbrock function

2. Lennard-Jones potential minimization

3. Three Hump Camel Back Problem

4. Goldstein-Price Problem

5. 2n order polynomials minimizations

6. Canonical functions … New math– Nonlinear space

Nonconvex constrained problemsNonconvex constrained problems

(P): min P(x) = || y – z || 2 s.t. h(y) = ½ y A y – r ellipsoid g (z) = ½ ( || z – c || 2 - b )2 – d t ( z - c)

Thm: If G ( ) 0 , (Pd) has at least one critical solution which gives to a global optimal solution to (P).

Lagrangian: x = ( y, z ) R2n L(x,) = || y – z || 2 + h(y) + g(z)

Let = z ) = || z – c || 2 , V() = ½ (b 2

= ∂ V() = b , V*(V() =½ 2

bTotal complementary function (x,) = || y – z || 2 + h(y) +(z ) - V() – d t ( z - c) ]

(Pd): Pd() = minx(x,) = - ½ F T G () -1 F - V()

0

yz

G () =

Challenges Challenges Super-Duality Super-DualitySince 2010, Zalinescu (+ 2) has wrote 11 papers + 1 letter challenging

the Canonical Duality Theory, which can be grouped in three categories: 1. Conceptual Duality (4 papers, two published and two rejected)

• min P(x) = V((x)) – F(x)

F (x) external energy (must be linear function)∂F(x) = x* = f

V() internal (stored) energy (must be objective ) ∂V() =

2. Moral Duality (6 papers) all on the same open problem left in 2003:

If dim P ≠ dim Pd min P(x) ≠ minPd( S- 3. Multi-scale duality (1 paper): Locally correct but globally wrong

Certain condition in S+ is missing Total complementary function (x,) , x = ( y , z ) R2n

0

yz

““Counter-Example” Counter-Example” Hidden truthHidden truth

Conclusion: The consideration of the Gao-Strang function (x,) is useless, at least for the problem studied in [3]. Morales-Gao (2012): linear perturbation (x,) – k -1 xT f

Mixed Integer Optim. Supply Chain Process

Nonconvex/nonsmoothVariational/V.I. Analysis

Graph, lattice, fuzzy max-plus algebra

FEM, FDM, FVM, SDP Meshless, Wavelet, SIP

Discrete optimization

Continuous Optimization

Unified Global Optimization

Combinatorial Algebra

Numerical Analysis

Combinatorial Optim. Integer Programming

Canonical Duality-Triality

Theory

Duality in Nonconvex Systems:Duality in Nonconvex Systems:Theory, Methods and ApplicationTheory, Methods and Application

David Yang GaoDavid Yang GaoKluwer Academic Publishers, 2000, 454pp

Part I    Symmetry in Convex Systems1.   Mono-duality in static systems2.   Bi-duality in dynamical systems

Part II    Symmetry Breaking: Triality Theory in Nonconvex Systems

3.   Tri-duality in nonconvex systems4.   Multi-duality and classifications of general systems

Part III    Duality in Canonical Systems5.   Duality in geometrically linear systems6.   Duality in finite deformation systems7.   Applications, open problems and concluding remarks duality in fluid mechanics ?

All happy families are alike,

Every unhappy family is unhappy in its own way

Anna Karenina --- Leo N Tolstoy

Reason: canonical duality

Reason: different duality gaps

Philosophy = Love of Canonical Duality

Proof: 1. By Greeks: Philosophy = Love of Wisdom

2. By Confucius: The highest Wisdom = Dao

3. By I-Ching (4000BC): Dao = one Ying + one Yang = Canonical Duality ---

Open Problem:

How to correctly understand the Triality

Canonical Duality –Triality Theory:

1. Non-convex concave

2. Discrete continuous

5. Diff. eqn Algebraic eqn.

4. Rescaling: Rn Rm Rr

n > m > r

7. Challenges Breakthrough

6. Non-deterministic deterministic

Rn

Rm

Rn

Rm

Rm n

x x*

yy*

yf

Rr Rr

oxxo

Rmr

x= 0

Open Problems: (P) is NP-Hard if (Pd) has no solution in Sa

+ ?

u

3. Non-smooth smooth

The The 44thth World Congress on Global Optimization World Congress on Global Optimization Gainesville, Florida - USA, Feb 22-25, 2015Gainesville, Florida - USA, Feb 22-25, 2015

Thanks! Thanks!

Some referencesSome references

[1] Gao, D.Y. and Sherali, H.D. (2008).

Canonical duality: Connection between

nonconvex mechanics and global optimization,

in Advances in Appl. Mathematics and Global Optimization, 249-316, Springer, 2008

[2] Gao, D.Y. (2009). Canonical duality theory: Unified understanding and generalized solution for global optimization problems,

Computers & Chemical Engineering, 33:1964–1972

[3] Daniel Morales-Silva, David Gao On the minimal distance between two surfaces, http://arxiv.org/abs/1210.1618

[4] Gao, DY and Wu, C, On the Triality Theory in Global Optimization

http://arxiv.org/abs/1104.2970

The The 44thth World Congress on Global Optimization World Congress on Global Optimization Gainesville, Florida - USA, Feb 22-25, 2015Gainesville, Florida - USA, Feb 22-25, 2015

Thanks! Thanks!

International Society of Global OptimizationInternational Society of Global Optimization (www.iSoGOp.org)(www.iSoGOp.org)

WCGO 2015WCGO 2015