lecture static
TRANSCRIPT
-
7/31/2019 Lecture Static
1/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Math 693A: Advanced Numerical Analysis Numerical Optimization
Lecture Notes #1 Introduction
Peter Blomgren,[email protected]
Department of Mathematics and StatisticsDynamical Systems Group
Computational Sciences Research Center
San Diego State UniversitySan Diego, CA 92182-7720
http://terminus.sdsu.edu/
Fall 2012
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (1/31)
http://terminus.sdsu.edu/http://terminus.sdsu.edu/ -
7/31/2019 Lecture Static
2/31
-
7/31/2019 Lecture Static
3/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Academic LifeContact Information, Office Hours
Academic Life
MSc. Engineering Physics, Royal Institute of Technology (KTH), Stockholm, Sweden. Thesis Advisers:Michael Benedicks, Department of Mathematics KTH, andErik Aurell, Stockholm University, Department of Mathematics. Thesis Topic: A Renormalization Technique for Families with Flat Maxima.
PhD. UCLA Department of Mathematics. Adviser:Tony F. Chan. PDE-Based Methods for Image Processing.Thesis title: Total Variation Methods for Restoration of Vector Valued Images.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (3/31)
-
7/31/2019 Lecture Static
4/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Academic LifeContact Information, Office Hours
Academic Life
Research Associate. Stanford University,
Department of Mathematics. Main Focus: Time Reversal andImaging in Random Media (with George Papanicolaou, et. al.)
Professor, San Diego State University, Departmentof Mathematics and Statistics. Projects: ComputationalCombustion, Biomedical Imaging (Mitochondrial Structures,Heartcell Contractility, Skin Cancer Classication).
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (4/31)
-
7/31/2019 Lecture Static
5/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Academic LifeContact Information, Office Hours
Contact Information
Office GMCS-587Email [email protected]
Web http://terminus.sdsu.edu/SDSU/Math693a f2012/Phone (619)594-2602Office Hours TuTh: 10:45 11:30a, 3:00 3:45p
and by appointment
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (5/31)
h f
http://terminus.sdsu.edu/SDSU/Math693a_f2012/http://terminus.sdsu.edu/SDSU/Math693a_f2012/ -
7/31/2019 Lecture Static
6/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Math 693A: Literature
Required (A Modern Treatment of NumericalOptimization)Numerical Optimization, 2nd Edition, Jorge Nocedal and StephenJ. Wright, Springer Series in Operations Research, Springer Verlag,
2006. ISBN-10: 0387303030; ISBN-13: 978-0387303031Required (Supplemental)Class notes and class web-page.
Optional (A Classic in the eld; Source for classprojects)Numerical Methods for Unconstrained Optimization and Nonlinear Equations , J. E. Dennis, Jr. and Robert B. Schnabel, Classics inApplied Mathematics 16, Society for Industrial and AppliedMathematics (SIAM), 1996. ISBN 0-89871-364-1.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (6/31)
Th P f
-
7/31/2019 Lecture Static
7/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Math 693A: Introduction What we will cover
NW-2 Unconstrained Optimization
NW-3 Line Search Methods
NW-4 Trust Region Methods
NW-5 Conjugate Gradient Methods
NW-6 Quasi-Newton Methods
NW-7 Calculating Derivatives
NW-10 Least Squares Problems
NW-11 Nonlinear Equations
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (7/31)
The Professor
-
7/31/2019 Lecture Static
8/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Math 693A: Introduction Grading etc.
40% Homework: both theoretical, and implementation (program-ming) C/C++ or Fortran are the recommended languages,but feel free to program in 6510 assembler, Java, M$-D , orMatlab... Class accounts will be available.
60% Project: Implementation of several interacting parts of anoptimization package. By the end of the semester you should
have a working toolbox of optimization algorithms whichwill be useful in your current and future research projects.[Complete details TBA].
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (8/31)
The Professor
-
7/31/2019 Lecture Static
9/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Expectations and Procedures, I
Most class attendance is OPTIONAL Homework andannouncements will be posted on the class web page. If/whenyou attend class:
Please be on time.
Please pay attention.
Please turn off mobile phones.
Please be courteous to other students and the instructor.Abide by university statutes, and all applicable local, state, andfederal laws.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (9/31)
The Professor
-
7/31/2019 Lecture Static
10/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Expectations and Procedures, II
Please, turn in assignments on time. (The instructor reservesthe right not to accept late assignments.)The instructor will make special arrangements for studentswith documented learning disabilities and willtry to makeaccommodations for other unforeseen circumstances, e.g.illness, personal/family crises, etc. in a way that is fair to allstudents enrolled in the class. Please contact the instructorEARLY regarding special circumstances.
Students are expected and encouraged to ask questions inclass!Students are expected and encouraged to to make use of office hours! If you cannot make it to the scheduled officehours: contact the instructor to schedule an appointment!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (10/31)
The Professor Li & S ll b
-
7/31/2019 Lecture Static
11/31
The ProfessorThe Class Overview
The Class...Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Expectations and Procedures, III
Missed midterm exams: Dont miss exams! The instructorreserves the right to schedule make-up exams, make such
exams oral presentation, and/or base the grade solely on otherwork (including the nal exam).
Missed nal exam: Dont miss the nal! Contact theinstructor ASAP or a grade of WUor F will be assigned.
Academic honesty : submit your own work but feel free todiscuss homework with other students in the class!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (11/31)
The Professor Lit t & S ll b
-
7/31/2019 Lecture Static
12/31
The Class OverviewThe Class...
Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Honesty Pledges, I
The following Honesty Pledge must be included in allprograms you submit (as part of homework and/or projects):
I, (your name), pledge that this program is completely my ownwork, and that I did not take, borrow or steal code from anyother person, and that I did not allow any other person to use,have, borrow or steal portions of my code. I understand that if I violate this honesty pledge, I am subject to disciplinary actionpursuant to the appropriate sections of the San Diego StateUniversity Policies.
Work missing the honesty pledge may not be graded!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (12/31)
The Professor Literature & Syllabus
-
7/31/2019 Lecture Static
13/31
The Class OverviewThe Class...
Numerical Optimization
Literature & SyllabusGradingExpectations and Procedures
Honesty Pledges, II
Larger reports must contain the following text:
I, (your name), pledge that this report is completely my own
work, and that I did not take, borrow or steal any portionsfrom any other person. Any and all references I used areclearly cited in the text. I understand that if I violate thishonesty pledge, I am subject to disciplinary action pursuant tothe appropriate sections of the San Diego State UniversityPolicies. Your signature .
Work missing the honesty pledge may not be graded!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (13/31)
The Professor
-
7/31/2019 Lecture Static
14/31
The Class OverviewThe Class...
Numerical Optimization
ResourcesFormal Prerequisites
Math 693A: Computer Resources
You need access to a computing environment in which to writeyour code; you may want to use a combination of Matlab (forquick prototyping and short homework assignments) and C/C++or Fortran (or something completely different).
Class accounts for the computer lab(s) will be available.
You can also use the Rohan Sun Enterprise system or anothercapable system. [http://www-rohan.sdsu.edu/raccts.html ]
Free C/C++ ( gcc ) and Fortran ( f77 ) compilers are available forLinux/UNIX.
You may also want to consider buying the student version of Matlab: http://www.mathworks.com/
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (14/31)
The Professor
http://www-rohan.sdsu.edu/raccts.htmlhttp://www-rohan.sdsu.edu/raccts.htmlhttp://www.mathworks.com/http://www.mathworks.com/http://www-rohan.sdsu.edu/raccts.html -
7/31/2019 Lecture Static
15/31
The Class OverviewThe Class...
Numerical Optimization
ResourcesFormal Prerequisites
Math 693A: Introduction What you should know already
Math 524 and (Math 542 or Math 543 )
524 Linear Algebra Vector spaces, linear transformations, orthogonality, eigenvalues
and eigenvectors, normal forms for complex matrices, positive
denite matrices.542 Numerical Solutions of Differential Equations
Initial and boundary value problems for ODEs. PDEs. Iterativemethods, nite difference methods, the method of lines.
543 Numerical Matrix Analysis Gaussian elimination, LU-factorizations and pivoting strategies.
Direct and iterative methods for linear systems. Iterative methodsfor diagonalization and eigensystem computation. Tridiagonal,Hessenberg, and Householder matrices. The QR algorithm.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (15/31)
The ProfessorTh Cl O i The What? Why? and How?
-
7/31/2019 Lecture Static
16/31
The Class OverviewThe Class...
Numerical Optimization
The What? Why? and How?Concepts & TermsMathematical Formulation
Math 693A: Introduction Why???
Q: Why do we need numerical optimization methods?
A: Many problems in applications are formulated as optimizationproblems:
Optimal trajectories for airplanes, space craft, robotic motion,etc.
Optimal shape for cars, airfoils, aerodynamic bicycle wheels,etc.
Risk management investment portfolios; insurance premi-ums.
Circuit and network design.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (16/31)
The ProfessorTh Cl O i The What? Why? and How?
-
7/31/2019 Lecture Static
17/31
The Class OverviewThe Class...
Numerical Optimization
yConcepts & TermsMathematical Formulation
Numerical Optimization: Concepts and Term 1 of 2
Students optimize: minimize study time T , such that GPA isacceptable. ( :-)
Nature optimizes: Physical systems settle in a state of minimalenergy A ball rolls down to the bottom of a slope; DNA molecules fold to minimize some measure of energy; Light rays follow the path that minimizes travel time. Chemical reactions are energy-driven, etc, etc, etc...
In order to understand physical (economic, etc.) systems we mustoptimize: rst we must identify the objective (the measure of performance, or energy). The objective depends on a number of variables (the characteristics of the system).
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (17/31)
The ProfessorThe Class Overview The What? Why? and How?
-
7/31/2019 Lecture Static
18/31
The Class OverviewThe Class...
Numerical Optimization
yConcepts & TermsMathematical Formulation
Numerical Optimization: Concepts and Term 2 of 2
Our goal is to nd the values of the variables that optimize (eitherminimize, or maximize) the objective . Often the variables are
constrained (restricted) in some way (e.g. densities, and interestrates are non-negative).
The process of identifying the objective , variables , andconstraints is non-trivial and will essentially be completely ignoredin this class. (See Mathematical Modeling )
Our discussion starts after the modeling is done!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (18/31)
The ProfessorThe Class Overview The What? Why? and How?
-
7/31/2019 Lecture Static
19/31
The Class OverviewThe Class...
Numerical Optimization
Concepts & TermsMathematical Formulation
Mathematical Formulation 1 of 2
From the point of view of a mathematician, optimization is theminimization (or maximization) of a function subject to constraintson its variables.
Notation:
x the vector of variables (a.k.a. unknowns, or parameters)f (x) the objective functionc the vector of constraints that the unknowns must satisfy
The Optimization Problem can be written
minxR n
f (x) subject to c i (x) = 0 , i E c i (x) 0, i I
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (19/31)
The ProfessorThe Class Overview The What? Why? and How?
-
7/31/2019 Lecture Static
20/31
The Class OverviewThe Class...
Numerical Optimization
Concepts & TermsMathematical Formulation
Mathematical Formulation 2 of 2
The Optimization Problem
minxR n
f (x) subject to c i (x) = 0 , i E c i (x) 0, i I
Here E is the set of equality constraints , and I the set of inequality constraints .
Note that a maximization problem can be converted into aminimization problem:
maxxR n
f (x) minxR n
[ f (x)]
and a less-than-or-equal-to constraint can similarly be convertedinto a greater-than-or-equal-to constraint.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (20/31)
The ProfessorThe Class Overview The What? Why? and How?C & T
-
7/31/2019 Lecture Static
21/31
The Class OverviewThe Class...
Numerical Optimization
Concepts & TermsMathematical Formulation
Feasible Region
The set of all x R n which satisfy the constraints c is called thefeasible region , e.g. if
c 1(x 1 , x 2) = x 21 + x 22 1c 2(x 1 , x 2) = (x 21 + x 22 ) 4
then the feasible region is the annulus:
Figure: The annulus 1 r 2.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (21/31)
The ProfessorThe Class Overview The What? Why? and How?C t & T
-
7/31/2019 Lecture Static
22/31
The Class...Numerical Optimization
Concepts & TermsMathematical Formulation
Constrained vs. Unconstrained Optimization 1 of 2
Optimization problems of the form
minxR n
f (x) subject to c i (x) = 0 , i E c i (x) 0, i I
can be classied according to the nature of the function andconstraints (linear, non-linear, convex, etc.) the key distinctionis between problems that have constraints, and problems that donot:
Constrained Optimization Problems : arise from models thatinclude explicit constraints on the variables. They can be relativelysimple, or nasty non-linear inequalities expressing complexrelationships between the variables.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (22/31)
The ProfessorThe Class Overview The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
23/31
The Class...Numerical Optimization
Concepts & TermsMathematical Formulation
Constrained vs. Unconstrained Optimization 2 of 2
Unconstrained Optimization Problems arise directly in someapplications; if the constraints are natural it may be safe todisregard them during the solution process and verify that they are
satised in the solution.Further, constrained problems can be restated as unconstrainedproblems the constraints are replaced by penalizing terms in theobjective which discourage violation of the constraints.
The more complicated the constraints, the more difficult it is tond the optimal solution. The absence of constraints is the easiestcase.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (23/31)
The ProfessorThe Class Overview The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
24/31
The Class...Numerical Optimization
Concepts & TermsMathematical Formulation
Continuous vs. Discrete Optimization
In many applications, the variables (x) can only take integer values very few people would be interested in buying 3/4 of a TV set,or receive 1/3 of a package; the electrons in an atom can only existin certain quantum states, etc, etc.
The Discrete Optimization Problem is harder than the ContinuousOptimization Problem. One way to get close to solving thediscrete problem is to solve the problem as if it is continuous andthen round or truncate the solution to integer values. This will
often give a sub-optimal integer solution and/or a solution that isinfeasible .
Here we will only consider the easier Continuous OptimizationProblem.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (24/31)
The ProfessorThe Class Overview The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
25/31
The Class...Numerical Optimization
Concepts & TermsMathematical Formulation
Global and Local Optimization
The fastest optimization algo-rithms seek a local solution a point where the objective issmaller than all other feasiblepoints in its vicinity.
The best solution the globalminimum is usually hard to nd,but is often desirable.
-10 0 10-1
0
1
2
3
Figure: A function with multiple local minima, and one global minimum.
Under certain circumstances (e.g. see convexity) there is only oneminimum.
We will focus on local optimization algorithms, but note that(most) global algorithms will solve a sequence of local optimizationproblems.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (25/31)
The ProfessorThe Class Overview
Th Cl
The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
26/31
The Class...Numerical Optimization
Concepts & TermsMathematical Formulation
Stochastic and Deterministic Optimization
In many applications it is impossible to fully specify all parameters
at the time of formulation; in quantum physics, the stock market,or the game of risk some quantities are random and are bestmodeled using some probability model.
We will focus ondeterministic optimization problems, where themodel can be fully specied when we formulate the problem.
However, in many cases the solutions to stochastic optimizationproblems are formulated as sequences or collections of deterministic problems.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (26/31)
The ProfessorThe Class Overview
Th Cl
The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
27/31
The Class...Numerical Optimization
pMathematical Formulation
Convexity: Denitions
There are two types of convexity which impact optimizationproblems:
S R n is a convex set if the straight line segment connectingany two points in S lies entirely insideS . Formally, for any two
points x S and y S , we have ( x + (1 )y) S for all [0, 1].
f is a convex function if its domain is a convex set and if forany two points x and y in this domain, the graph of f lies below
the straight line connecting ( x, f (x)) to ( y, f (y)) in R n+1
. Thatis, we have
f ( x + (1 )y) f (x) + (1 )f (y) , [0, 1]
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (27/31)
The ProfessorThe Class Overview
The Class
The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
28/31
The Class...Numerical Optimization
pMathematical Formulation
Convexity: Illustrations
Figure: A convex (left) and a non-convex set (right) in R 2 .
Figure: A convex function.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (28/31)
The ProfessorThe Class Overview
The Class
The What? Why? and How?Concepts & Terms
-
7/31/2019 Lecture Static
29/31
The Class...Numerical Optimization Mathematical Formulation
Convexity: Notes
A function f is said to be concave if f is convex.
Optimization algorithms for unconstrained problems are usuallyguaranteed to converge to a stationary point (maximum,minimum, or inection point) of the objective f .
If f is convex, then the algorithm has converged to a globaloptimum.
Bottom line: Convexity simplies the problem.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (29/31)
The ProfessorThe Class OverviewThe Class
The What? Why? and How?Concepts & Terms
h l l
-
7/31/2019 Lecture Static
30/31
The Class...Numerical Optimization Mathematical Formulation
Summary
Easier HarderUnconstrained Constrained Continuous Discrete Local Optimization Global OptimizationDeterministic Stochastic Convex Non-Convex
Table: Summary of some factors impacting the difficulty of the optimization problem.
In this class we will mainly look at Local Optimization methods forDeterministic, Unconstrained, Continuous, Convex functions overConvex sets. Still, it will be a challenging semester!
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (30/31)
The ProfessorThe Class OverviewThe Class...
The What? Why? and How?Concepts & TermsM h i l F l i
-
7/31/2019 Lecture Static
31/31
The Class...Numerical Optimization Mathematical Formulation
Algorithms
Optimization algorithms are iterative and generate a sequence of successively better estimates of the solution.Three key attributes characterize each (good) algorithm:
Robustness: Algorithm performance on a wide variety of problems
(of the same type), for a range of reasonable choices of initial values.
Efficiency: We prefer fast algorithms that do not require excessiveamounts of storage.
Accuracy: The algorithm should nd the solution without beingoverly sensitive to errors in the data or roundoff errors inthe computations.
These goals are often conicting hence careful consideration of trade-off between the goals is a key part of this course.
Peter Blomgren, [email protected] Lecture Notes #1 Introduction (31/31)