jorge nocedal - opt-ml.org
TRANSCRIPT
![Page 1: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/1.jpg)
1
An Evolving Gradient Resampling Method for Machine Learning
Jorge Nocedal Northwestern University
NIPS, Montreal 2015
![Page 2: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/2.jpg)
2
Collaborators
Figen Oztoprak Stefan Solntsev Richard Byrd
![Page 3: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/3.jpg)
3
Outline
1. How to improve upon the stochastic gradient method for
risk minimization
2. Noise reduction methods Dynamic Sampling (batching) Aggregated Gradient methods (SAG, SVRG, etc)
3. Second order methods
4. Propose a noise reduction method that re-uses old gradients and also employs dynamic sampling
![Page 4: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/4.jpg)
4
Organization of optimization methods
Stochastic Gradient Batch Gradient Method Method
Stochastic Newton Method
condition number
noise reduction
![Page 5: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/5.jpg)
5
Second-order methods
Stochastic Gradient Batch Gradient Method Method
Stochastic Newton
• Averaging (Polyak-Ruppert) • Momentum • Natural gradient, Fischer • quasi-Newton • inexact Newton (Hessian-free)
![Page 6: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/6.jpg)
6
Noise reducing methods
Stochastic Gradient Batch Gradient Method Method
Stochastic Newton Method
• Dynamic sampling methods
• aggregated gradient methods •
This Talk: combine both ideas Why?
![Page 7: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/7.jpg)
7
Objective Function
Fs (w) =1| S |
fi∈S∑ (w;ξi )
Sample gradient approximation - batch (or mini-batch)
minw F(w) = E[ f (w;ξ )]
ξ = (x, y) random variable with distribution Pf (⋅;ξ ) composition of loss ℓ and prediction h
wk+1 = wk −α k∇f (wk;ξk ) stochastic gradient method (SG)
wk+1 = wk −α k∇FS (wk ) batch (mini) method
![Page 8: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/8.jpg)
8
Transient behavior of SG
To ensure convergence α k → 0 in SG method to control variance. Steplength selected to achieve fast initial progress, but this willslow progress in later stages
Expected function decreaseE[F(wk+1)− F(wk )] ≤ −α k‖∇F(wk )‖2
2 +α k2E‖∇f (wk ,ξk )‖2
Initially, gradient decrease dominates; then variance in gradient hinters progress (area of confusion)
Dynamic sampling methods reduce gradient variance by increasing batch. What is the right rate?
![Page 9: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/9.jpg)
Proposal: Gradient accuracy conditions
Consider stochastic gradient method with fixed steplength wk+1 = wk −αg(wk ,ξk )
Geometric noise reduction
If the variance of stochastic gradient decreases geometrically, the method yields linear convergence
Lemma If ∃M > 0, ζ ∈(0,1) s.t. E[‖g(wk ,ξk )‖2
2 ]−‖∇F(wk )‖22 ≤ M ζ k−1
Then E[F(wk )− F*] ≤νρ k−1
Extension of classical convergence result for gradient method where error in gradient estimates decreases sufficiently rapidly to preserve linear convergence
Schmidt et al Pasupathy et al
![Page 10: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/10.jpg)
Proposal: Gradient accuracy conditions
Optimal work complexity
Moreover, we obtain optimal complexity bounds
We can ensure variance condition E[‖g(wk ,ξk )‖2
2 ]−‖∇F(wk )‖22 ≤ M ζ k−1
by letting | Sk | = ak−1 a >1
The total number of stochastic gradient evaluations to achieve E[F(wk )− F*] ≤ ε is O(1 / ε)
with favorable constants a∈[1,1− βcµ2
]−1
Pasupathy, Glynn et al 2014 Friedlander, Schmidt 2012 Homem-de-Mello, Shapiro 2012 Byrd, Chin, N., Wu 2013
∇Fs (w) =1| S |
∇fi∈S∑ (w;ξi )
![Page 11: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/11.jpg)
Theorem: Suppose F is strongly convex. Consider wk+1 = wk − (1 / L)gk where Sk is chosen so that variance condition holds and | Sk | ≥ γ k for γ >1.Then ! E[F(wk )− F(w*)]≤Cρ k ρ <1! the number of gradient samples to achieve ε accuracy is
O κεωdλ
⎛⎝⎜
⎞⎠⎟ d = no. of variables
κ = condition number, λ = smallest eigenv of Hessian
‖Var∇ℓ(wk;i)‖1≤ω (population)
![Page 12: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/12.jpg)
12
Dynamic sampling (batching)
At every iteration, choose a subset S of {1,…,n} and apply one step of an optimization algorithm to the function
FS (wk ) =
1| S | i∈S∑ f (w;ξi ),
At the start, a small sample size | S | is chosen • If optimization step is likely to reduce F(w), sample size is kept
unchanged; new sample S is chosen; next optimization step taken • Else, a larger sample size is chosen, a new random sample S is
selected, a new iterate computed
Many optimization methods can be used. This approach creates the opportunity of employing second order methods
![Page 13: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/13.jpg)
13
How to implement this in practice?
1. Predetermine a geometric increase, tuning parameter
2. Use angle (i.e. variance test)
Popular: combination of these two strategies.
| Sk | = ak−1 a >1
Ensure bound is satisfied in expectation ‖g(wk )−∇F(wk )‖≤θ‖gk‖⦆⦆⦆θ <1
![Page 14: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/14.jpg)
14
Newton-CG method with dynamic sampling, Armijo line search Test Problem • From Google VoiceSearch • 191,607 training points • 129 classes; 235 features • 30,315 parameters (variables) • Small version of production problem • Multi-class logistic regression • Initial batch size: 1%; Hessian sample 10%
Numerical Tests:
wk+1 = wk −α k∇2FHk
(wk )−1gk α k ≈1
Numerical test
![Page 15: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/15.jpg)
15
Classical Newton-CG
New method
L-BFGS (m=2,20)
Function
Time Dynamic change of sample sizes … based on variance estimate
Batch L-BFGS
Stochastic gradient descent
Dynamic Newton-CG
![Page 16: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/16.jpg)
16
However, not completely satisfactory
More investigation is needed …. Particularly: • Transition between stochastic and batch regimes • Coordination between step size and batch size • Use of second order information (one stochastic gradient is not too
noisy) • Can the idea of re-using gradients in a gradient aggregation approach
help?
![Page 17: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/17.jpg)
17
Stochastic process gradient methods
1 − − −[ ]Sk m
twilight zone
SGD
α k = 1 / k
α k = 1
Transition from stochastic to batch regimes
Gradient aggregation could smooth transition
![Page 18: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/18.jpg)
18
Randomized Aggregated Gradient Methods (for empirical risk min)
SAG, SAGA, SVRG, etc focus on minimizing empirical risk Iteration:
Expected Risk: F (w) = E [ f (w;ξ )]
Empirical Risk: Fm (w) = 1m i=1
n
∑ f (w;ξi ) =1m i=1
n
∑ fi (w)
wk+1 = wk −α yk
yk combination of gradients ∇fi evaluated at previous iterates φ j
SAG
yk =1m[∇f j (wk )−∇f j (φk−1
j )+ 1m
∇i=1
m
∑ fi (φk−1i )] Choose j
at random
![Page 19: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/19.jpg)
19
Example of Gradient Aggregation Methods
yk =1m[∇f j (wk )−∇f j (φk−1
j )+ 1m
∇i=1
m
∑ fi (φk−1i )]
Achieve linear rate of convergence in expectation (after a full initialization pass)
m
j
SAG, SAGA, SVRG
SAG
Fm (w) = 1m i=1∑ f (w;ξi ) =
1m i=1∑ fi (w)
![Page 20: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/20.jpg)
20
EGR Method
The Evolving Gradient Resampling Method for Expected Risk Minimization
![Page 21: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/21.jpg)
21
Proposed algorithm
1. Minimizes expected risk (not training error) 2. Stores previous gradients and updates several (sk) at each 3. iteration 4. Additional (uk) gradients are computed at current iterate 5. Total amount of stored gradients increases monotonically 6. Shares properties with dynamic sampling and gradient aggregation
methods
Goal: analyze an algorithm of this generality (interesting in its own right) Finding right balance between re-using old information and batching can result in efficient method.
![Page 22: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/22.jpg)
22
The EGR method
yk =1
tk + uk( ∑ j∈Sk
[∇f j (wk )−∇f j (φk−1j )]+
i=1
tk
∑∇fi (φk−1i )+ ∇
j∈Uk
∑ f j (wk ) )
Related work: Frostig et al 2014, Babanezhad et al 2015
tk
jj
sk uk
tk number of gradients in storage at start of iteration kUk indices of new gradients sampled at wk uk = |Uk |Sk indices of previously computed gradients that are updated sk = | Sk |
How should sk and uk be controled?
Evaluated at wk
![Page 23: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/23.jpg)
23
Algorithms included in framework
tk
jj
sk uk
stochastic gradient method: sk = 0, uk = 1dynamic sampling method: sk = 0, uk = function(k)aggregated gradient: sk = constant, uk = 0
EGR lin: sk = 0, uk = rEGR quad: sk = rk, uk = rEGR exp: sk = uk ≈ a
k
![Page 24: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/24.jpg)
Assumptions:1) sk = uk = a
k a∈!+ geometric growth2) F strongly convex, fi Lipschitz continuous gradients3) tr (var [∇fi (w)])≤ v2 ∀w
Lemma:
E[Ek[ek ]]E[‖wk −w*‖]σ k
⎛
⎝
⎜⎜
⎞
⎠
⎟⎟ ≤ M
E[Ek[ek−1]]E[‖wk−1 −w*‖]σ k−1
⎛
⎝
⎜⎜
⎞
⎠
⎟⎟
ek =
1tk+1
‖∇fi (i=1
tk+1
∑ φki )−∇fi (wk )‖ σ k = v2 / tk+1
Lyapunov function
![Page 25: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/25.jpg)
M =
1−η1+η
(1+αL) 1−η1+η
αL 1−η1+η
α
αL 1−αµ α
0 0 11+η
⎛
⎝
⎜⎜⎜⎜⎜⎜
⎞
⎠
⎟⎟⎟⎟⎟⎟
η : probability that an old gradient is recomputedα : steplengthL: Lipschitz constantµ : strong convexity parameter
Lemma. For sufficiently small α the spectral radius of M satisfiesρM <1
![Page 26: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/26.jpg)
Theorem: If α k is chosen small enough E‖wk −w*‖ ≤ ckβ R-linear convergence
SAG special case: tk = m, uk = 0, sk = constant
Simple proof of R-linear convergence of SAG but with a larger constant
![Page 27: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/27.jpg)
27
Related Methods
yk =1m[∇f j (wk )−∇f j (φk−1
j )+ 1m
∇i=1
m
∑ fi (φk−1i )]
• Streaming SVRG (Frostig et al 2014) • Stop wasting gradients (Babanezhad et al 2015)
m
j SVRG
![Page 28: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/28.jpg)
28
SAG(A) initialization phase
yk =1m[∇f j (wk )−∇f j (φk−1
j )+ 1m
∇i=1
m
∑ fi (φk−1i )]
m
j SAG
1. Sample j ∈{1,...,m} at random2. Compute ∇f j (wk )3. If j has been sampled earlier, replace old gradient4. Else add new ∇f j (wk ) to aggregated gradient (memory grows)
![Page 29: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/29.jpg)
29
Numerical Result
1. Comparison of EGR with various growth rates 2. Comparison with SGD 3. Comparison with SAG-init and SAGA-init
Goal: analyze an algorithm of this generality (interesting in its own right) Finding right balance between re-using old information and batching can result in efficient method.
![Page 30: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/30.jpg)
30
Susy EGR vs SG vs Dynamic Sampling
����
� �������
������ ������������
���
�����
�����
�����
�����
�����
�����
�����
�����
�����
�����
�����
���
![Page 31: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/31.jpg)
31
Random Comparing with SAG initialization
����
� ������
������ ������� � �����
���
����
����
����
����
����
����
���
![Page 32: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/32.jpg)
32
Random Larger initial batch for EGR
����
� ������
��������� � ������ � �
���
����
����
����
����
����
����
����
����
����
����
����
���
![Page 33: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/33.jpg)
33
Alpha
����
� �������
������ ������������
���
�����
�����
�����
�����
�����
�����
�����
�����
�����
�����
�����
���
![Page 34: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/34.jpg)
34
EGR with various growth rates (Alpha)
����
� �������
������� ������������������������������� �������������������������
���
�����
�����
�����
�����
�����
�����
�����
�����
�����
�����
��������
![Page 35: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/35.jpg)
35
Random
����
� ������
������� �������� �
���
����
����
����
����
����
����
����
����
����
����
�������
![Page 36: Jorge Nocedal - opt-ml.org](https://reader031.vdocuments.site/reader031/viewer/2022012102/616a006111a7b741a34dba4b/html5/thumbnails/36.jpg)
36
The End