the landscape of non-convex empirical risk with degenerate ... · the landscape of non-convex...

The Landscape of Non-convex Empirical Risk withDegenerate Population Risk

Shuang Li, Gongguo Tang, and Michael B. WakinDepartment of Electrical Engineering

Colorado School of MinesGolden, CO 80401

shuangli,gtang,[email protected]

Abstract

The landscape of empirical risk has been widely studied in a series of machinelearning problems, including low-rank matrix factorization, matrix sensing, matrixcompletion, and phase retrieval. In this work, we focus on the situation where thecorresponding population risk is a degenerate non-convex loss function, namely, theHessian of the population risk can have zero eigenvalues. Instead of analyzing thenon-convex empirical risk directly, we first study the landscape of the correspondingpopulation risk, which is usually easier to characterize, and then build a connectionbetween the landscape of the empirical risk and its population risk. In particular,we establish a correspondence between the critical points of the empirical risk andits population risk without the strongly Morse assumption, which is required inexisting literature but not satisfied in degenerate scenarios. We also apply the theoryto matrix sensing and phase retrieval to demonstrate how to infer the landscape ofempirical risk from that of the corresponding population risk.

1 Introduction

Understanding the connection between empirical risk and population risk can yield valuable insightinto an optimization problem [1, 2]. Mathematically, the empirical risk f(x) with respect to aparameter vector x is defined as

f(x) ,1

M

M∑m=1

L(x,ym).

Here, L(·) is a loss function and we are interested in losses that are non-convex in x in this work.y = [y1, · · · ,yM ]> is a vector containing the random training samples, and M is the total number ofsamples contained in the training set. The population risk, denoted as g(x), is the expectation of theempirical risk with respect to the random measure used to generate the samples y, i.e., g(x) = Ef(x).Recently, the landscapes of empirical and population risk have been extensively studied in manyfields of science and engineering, including machine learning and signal processing. In particular, thelocal or global geometry has been characterized in a wide variety of convex and non-convex problems,such as matrix sensing [3, 4], matrix completion [5, 6, 7], low-rank matrix factorization [8, 9, 10],phase retrieval [11, 12], blind deconvolution [13, 14], tensor decomposition [15, 16, 17], and so on.In this work, we focus on analyzing global geometry, which requires understanding not only regionsnear critical points but also the landscape away from these points.

It follows from empirical process theory that the empirical risk can uniformly converge to thecorresponding population risk as M →∞ [18]. A recent work [1] exploits the uniform convergenceof the empirical risk to the corresponding population risk and establishes a correspondence of their

33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

critical points when provided with enough samples. The authors build their theoretical guaranteesbased on the assumption that the population risk is strongly Morse, namely, the Hessian of thepopulation risk cannot have zero eigenvalues at or near the critical points1. However, many problemsof practical interest do have Hessians with zero eigenvalues at some critical points. We refer to suchproblems as degenerate. To illustrate this, we present the very simple rank-1 matrix sensing andphase retrieval examples below.

Example 1.1. (Rank-1 matrix sensing). Given measurements ym = 〈Am,x?x?>〉, 1 ≤ m ≤M ,

where x? ∈ RN and Am ∈ RN×N denote the true signal and them-th Gaussian sensing matrix withentries following N (0, 1), respectively. The following empirical risk is commonly used in practice

f(x) =1

4M

M∑m=1

(〈Am,xx

>〉 − ym)2.

The corresponding population risk is then

g(x) = Ef(x) =1

4‖xx> − x?x?>‖2F .

Elementary calculations give the gradient and Hessian of the above population risk as

∇g(x) = (xx> − x?x?>)x, and ∇2g(x) = 2xx> − x?x?> + ‖x‖22IN .We see that g(x) has three critical points x = 0, ± x?. Observe that the Hessian at x = 0 is∇2g(0) = −x?x?>, which does have zero eigenvalues and thus g(x) does not satisfy the stronglyMorse condition required in [1]. The conclusion extends to the general low-rank matrix sensing.

Example 1.2. (Phase retrieval). Given measurements ym = |〈am,x?〉|2, 1 ≤ m ≤ M , wherex? ∈ RN and am ∈ RN denote the true signal and the m-th Gaussian random vector with entriesfollowing N (0, 1), respectively. The following empirical risk is commonly used in practice

f(x) =1

2M

M∑m=1

(|〈am,x〉|2 − ym

)2. (1.1)

The corresponding population risk is then

g(x) = Ef(x) = ‖xx> − x?x?>‖2F +1

2(‖x‖22 − ‖x?‖22)2. (1.2)

Elementary calculations give the gradient and Hessian of the above population risk as

∇g(x) = 6‖x‖22x− 2‖x?‖22x− 4(x?>x)x?,

∇2g(x) = 12xx> − 4x?x?> + 6‖x‖22IN − 2‖x?‖22IN .

We see that the population loss has critical points x = 0, ± x?, 1√3‖x?‖2w with w>x? = 0 and

‖w‖2 = 1. Observe that the Hessian at x = 1√3‖x?‖2w is ∇2g( 1√

3‖x?‖2w) = 4‖x?‖22ww> −

4x?x?>, which also has zero eigenvalues and thus g(x) does not satisfy the strongly Morse conditionrequired in [1].

In this work, we aim to fill this gap and establish the correspondence between the critical points ofempirical risk and its population risk without the strongly Morse assumption. In particular, we workon the situation where the population risk may be a degenerate non-convex function, i.e., the Hessianof the population risk can have zero eigenvalues. Given the correspondence between the criticalpoints of the empirical risk and its population risk, we are able to build a connection between thelandscape of the empirical risk and its population counterpart. To illustrate the effectiveness of thistheory, we also apply it to applications such as matrix sensing (with general rank) and phase retrievalto show how to characterize the landscape of the empirical risk via its corresponding population risk.

1A twice differentiable function f(x) is Morse if all of its critical points are non-degenerate, i.e., its Hessianhas no zero eigenvalues at all critical points. Mathematically, ∇f(x) = 0 implies all λi(∇2f(x)) 6= 0 withλi(·) being the i-th eigenvalue of the Hessian. A twice differentiable function f(x) is (ε, η)-strongly Morse if‖∇f(x)‖2 ≤ ε implies mini |λi(∇2f(x))| ≥ η. One can refer to [1] for more information.

2

The remainder of this work is organized as follows. In Section 2, we present our main results on thecorrespondence between the critical points of the empirical risk and its population risk. In Section 3,we apply our theory to the two applications, matrix sensing and phase retrieval. In Section 4, weconduct experiments to further support our analysis. Finally, we conclude our work in Section 5.

Notation: For a twice differential function f(·): ∇f , ∇2f , grad f , and hess f denote the gradientand Hessian of f in the Euclidean space and with respect to a Riemannian manifoldM, respectively.Note that the Riemannian gradient/Hessian (grad/hess) reduces to the Euclidean gradient/Hessian(∇/∇2) when the domain of f is the Euclidean space. For a scalar function with a matrix variable,e.g., f(U), we represent its Euclidean Hessian with a bilinear form defined as ∇2f(U)[D,D] =∑i,j,p,q

∂2f(U)∂D(i,j)∂D(p,q)D(i, j)D(p, q) for any D having the same size as U. Denote B(l) as a compact

and connected subset of a Riemannian manifoldM with l being a problem-specific parameter.2

2 Main Results

In this section, we present our main results on the correspondence between the critical points of theempirical risk and its population risk. LetM be a Riemannian manifold. For notational simplicity, weuse x ∈M to denote the parameter vector when we introduce our theory3. We begin by introducingthe assumptions needed to build our theory. Denote f(x) and g(x) as the empirical risk and thecorresponding population risk defined for x ∈M, respectively. Let ε and η be two positive constants.Assumption 2.1. The population risk g(x) satisfies

|λmin(hess g(x))| ≥ η (2.1)

in the set D , x ∈ B(l) : ‖grad g(x)‖2 ≤ ε. Here, λmin(·) denotes the minimal eigenvalue (notthe eigenvalue of smallest magnitude).

Assumption 2.1 is closely related to the robust strict saddle property [19] – it requires that any pointwith a small gradient has either a positive definite Hessian (λmin(hess g(x)) ≥ η) or a Hessian witha negative curvature (λmin(hess g(x)) ≤ −η). It is weaker than the (ε, η)-strongly Morse condition

-1.5 -1 -0.5 0 0.5 1 1.5

0.51

1.52

2.5

Ris

ks

PopulationEmpirical

-1.5 -1 -0.5 0 0.5 1 1.5

-10

-11

10

Gra

die

nts

-1.5 -1 -0.5 0 0.5 1 1.5

0102030

Hess

ians

Figure 1: Phase retrieval with N = 1.

as it allows the Hessian hess g(x) to have zero eigenvalues inD,provided it also has at least one sufficiently negative eigenvalue.Assumption 2.2. (Gradient proximity). The gradients of theempirical risk and population risk satisfy

supx∈B(l)

‖grad f(x)− grad g(x)‖2 ≤ε

2. (2.2)

Assumption 2.3. (Hessian proximity). The Hessians of theempirical risk and population risk satisfy

supx∈B(l)

‖hess f(x)− hess g(x)‖2 ≤η

2. (2.3)

To illustrate the above three assumptions, we use the phaseretrieval Example 1.2 with N = 1, x? = 1, and M = 30. Wepresent the population risk g(x) = 3

2 (x2−1)2 and the empirical risk f(x) = 1

2M

∑Mm=1 a

4m(x2−1)2

together with their gradients and Hessians in Figure 1. It can be seen that in the small gradient region(the three parts between the light blue vertical dashed lines), the absolute value of the populationHessian’s minimal eigenvalue (which equals the absolute value of Hessian here since N = 1) isbounded away from zero. In addition, with enough measurements, e.g., M = 30, we do see thegradients and Hessians of the empirical and population risk are close to each other.

We are now in the position to state our main theorem.Theorem 2.1. Denote f and g as the non-convex empirical risk and the corresponding populationrisk, respectively. Let D be any maximal connected and compact subset of D with a C2 boundary ∂D.Under Assumptions 2.1-2.3 stated above, the following statements hold:

2The subset B(l) can vary in different applications. For example, we define B(l) , U ∈ RN×k∗ :

‖UU>‖F ≤ l in matrix sensing and B(l) , x ∈ RN : ‖x‖2 ≤ l in phase retrieval.3For problems with matrix variables, such as matrix sensing introduced in Section 3, x is the vectorized

representation of the matrix.

3

(a) D contains at most one local minimum of g. If g has K (K = 0, 1) local minima in D, then falso has K local minima in D.

(b) If g has strict saddles in D, then if f has any critical points in D, they must be strict saddlepoints.

The proof of Theorem 2.1 is given in Appendix A (see supplementary material). In particular, weprove Theorem 2.1 by extending the proof of Theorem 2 in [1] without requiring the strongly Morseassumption on the population risk. We first present two key lemmas, in which we show that thereexists a correspondence between the critical points of the empirical risk and those of the populationrisk in a connected and compact set under certain assumptions, and the small gradient area can bepartitioned into many maximal connected and compact components with each component eithercontaining one local minimum or no local minimum. Finally, we finish the proof of Theorem 2.1 byusing these two key lemmas.

Part (a) in Theorem 2.1 indicates a one-to-one correspondence between the local minima of theempirical risk and its population risk. We can further bound the distance between the local minima ofthe empirical risk and its population risk. We summarize this result in the following corollary, whichis proved in Appendix C (see supplementary material).Corollary 2.1. Let xkKk=1 and xkKk=1 denote the local minima of the empirical risk and itspopulation risk, and Dk be the maximal connected and compact subset of D containing xk and xk.Let ρ be the injectivity radius of the manifoldM. Suppose the pre-image of Dk under the exponentialmapping Expxk

(·) is contained in the ball at the origin of the tangent space TxkM with radius ρ.

Assume the differential of the exponential mapping DExpxk(v) has an operator norm bounded by σ

for all v ∈ TxkM with norm less than ρ. Suppose the pullback of the population risk onto the tangent

space TxkM has Lipschitz Hessian with constant LH at the origin. Then as long as ε ≤ η2

2σLH, the

Riemannian distance between xk and xk satisfies

dist(xk,xk) ≤ 2σε/η, 1 ≤ k ≤ K.

In general, the two parameters ε and η used in Assumptions 2.1-2.3 can be obtained by lower bounding|λmin(hess g(x))| in a small gradient region. In this way, one can adjust the size of the small gradientregion to get an upper bound on ε, and use the lower bound for |λmin(hess g(x))| as η. In the casewhen it is not easy to directly bound |λmin(hess g(x))| in a small gradient region, one can also firstchoose a region for which it is easy to find the lower bound, and then show that the gradient has alarge norm outside of this region, as we do in Section 3. For phase retrieval, note that |λmin(∇2g(x))|and ‖∇g(x)‖2 roughly scale with ‖x?‖22 and ‖x?‖32 in the regions near critical points, which impliesthat η and the upper bound on ε should also scale with ‖x?‖22 and ‖x?‖32, respectively. For matrixsensing, in a similar way, |λmin(hess g(U))| and ‖grad g(U)‖F roughly scale with λk and λ1.5k inthe regions near critical points, which implies that η and the upper bound on ε should also scale withλk and λ1.5k , respectively. Note, however, with more samples (larger M ), ε can be set to smallervalues, while η typically remains unchanged. One can refer to Section 3 for more details on thenotation as well as how to choose η and upper bounds on ε in the two applications.

Note that we have shown the correspondence between the critical points of the empirical risk and itspopulation risk without the strongly Morse assumption in the above theorem. In particular, we relaxthe strongly Morse assumption to our Assumption 2.1, which implies that we are able to handle thescenario where the Hessian of the population risk has zero eigenvalues at some critical points or eveneverywhere in the set D. With this correspondence, we can then establish a connection between thelandscape of the empirical risk and the population risk, and thus for problems where the populationrisk has a favorable geometry, we are able to carry this favorable geometry over to the correspondingempirical risk. To illustrate this in detail, we highlight two applications, matrix sensing and phaseretrieval, in the next section.

3 Applications

In this section, we illustrate how to completely characterize the landscape of an empirical risk from itspopulation risk using Theorem 2.1. In particular, we apply Theorem 2.1 to two applications, matrixsensing and phase retrieval. In order to use Theorem 2.1, all we need is to verify that the empiricalrisk and population risk in these two applications satisfy the three assumptions stated in Section 2.

4

3.1 Matrix Sensing

Let X ∈ RN×N be a symmetric, positive semi-definite matrix with rank r. We measure X witha symmetric Gaussian linear operator A : RN×N → RM . The m-th entry of the observationy = A(X) is given as ym = 〈X,Am〉, where Am = 1

2 (Bm + B>m) with Bm being a Gaussianrandom matrix with entries following N (0, 1

M ). The adjoint operator A∗ : RM → RN×N isdefined as A∗(y) =

∑Mm=1 ymAm. It can be shown that E(A∗A) is the identity operator, i.e.

E(A∗A(X)) = X. To find a low-rank approximation of X when given the measurements y = A(X),one can solve the following optimization problem:

minX∈RN×N

1

4‖A(X−X)‖22 s. t. rank(X) ≤ k, X 0. (3.1)

Here, we assume that r2 ≤ k ≤ r N . By using the Burer-Monteiro type factorization [20, 21],i.e., letting X = UU> with U ∈ RN×k, we can transform the above optimization problem into thefollowing unconstrained one:

minU∈RN×k

f(U) ,1

4‖A(UU> −X)‖22. (3.2)

Observe that this empirical risk f(U) is a non-convex function due to the quadratic term UU>. Withsome elementary calculation, we obtain the gradient and Hessian of f(U), which are given as

∇f(U) = A∗A(UU> −X)U,

∇2f(U)[D,D] =1

2‖A(UD> + DU>)‖22 + 〈A∗A(UU> −X),DD>〉.

Computing the expectation of f(U), we get the population risk

g(U) = Ef(U) =1

4‖UU> −X‖2F , (3.3)

whose gradient and Hessian are given as

∇g(U) = (UU> −X)U and ∇2g(U)[D,D] =1

2‖UD> + DU>‖22 + 〈UU> −X,DD>〉.

The landscape of the above population risk has been studied in the general RN×k space with k = rin [8]. The landscape of its variants, such as the asymmetric version with or without a balanced term,has also been studied in [4, 22]. It is well known that there exists an ambiguity in the solution of (3.2)due to the fact that UU> = UQQ>U> holds for any orthogonal matrix Q ∈ Rk×k . This impliesthat the Euclidean Hessian∇2g(U) always has zero eigenvalues for k > 1 at critical points, even atlocal minima, violating not only the strongly Morse condition but also Assumption 2.1. To overcomethis difficulty, we propose to formulate an equivalent problem on a proper quotient manifold (ratherthan the general RN×k space as in [8]) to remove this ambiguity and make sure Assumption 2.1 issatisfied.

3.1.1 Background on the quotient manifold

To keep our work self-contained, we provide a brief introduction to quotient manifolds in this sectionbefore we verify our three assumptions. One can refer to [23, 24] for more information. We makethe assumption that the matrix variable U is always full-rank. This is required in order to define aproper quotient manifold, since otherwise the equivalence classes defined below will have differentdimensions, violating Proposition 3.4.4 in [23]. Thus, we focus on the case that U belongs to themanifold RN×k∗ , i.e., the set of all N × k real matrices with full column rank. To remove theparameterization ambiguity caused by the factorization X = UU>, we define an equivalence classfor any U ∈ RN×k∗ as [U] , V ∈ RN×k∗ : VV> = UU> = UQ : Q ∈ Rk×k,Q>Q = Ik.We will abuse notation and use U to denote also its equivalence class [U] in the following. LetM denote the set of all equivalence classes of the above form, which admits a (unique) differentialstructure that makes it a (Riemannian) quotient manifold, denoted asM = RN×k∗ /Ok. Here Ok isthe orthogonal group Q ∈ Rk×k : QQ> = Q>Q = Ik. Since the objective function g(U) in

5

(3.3) (and f(U) in (3.2)) is invariant under the equivalence relation, it induces a unique function onthe quotient manifold RN×k∗ /Ok, also denoted as g(U).

Note that the tangent space TURN×k∗ of the manifold RN×k∗ at any point U ∈ RN×k∗ is stillRN×k∗ . We define the vertical space VUM as the tangent space to the equivalence classes (whichare themselves manifolds): VUM , UΩ : Ω ∈ Rk×k, Ω> = −Ω. We also define thehorizontal space HUM as the orthogonal complement of the vertical space VUM in the tangentspace TURN×k∗ = RN×k∗ : HUM , D ∈ RN×k∗ : D>U = U>D. For any matrix Z ∈ RN×k∗ ,its projection onto the horizontal spaceHUM is given as PU(Z) = Z−UΩ, where Ω is a skew-symmetric matrix that solves the following Sylvester equation ΩU>U + U>UΩ = U>Z− Z>U.Then, we can define the Riemannian gradient (grad ·) and Hessian (hess ·) of the empirical risk andpopulation risk on the quotient manifoldM, which are given in the supplementary material.

3.1.2 Verifying Assumptions 2.1, 2.2, and 2.3

Assume that X = WΛW> with W ∈ RN×r and Λ = diag([λ1, · · · , λr]) ∈ Rr×r is an eigen-decomposition of X. Without loss of generality, we assume that the eigenvalues of X are indescending order. Let Λu ∈ Rk×k be a diagonal matrix that contains any k non-zero eigenvaluesof X and Wu ∈ RN×k contain the k eigenvectors of X associated with the eigenvalues in Λu. LetΛk = diag([λ1, · · · , λk]) be the diagonal matrix that contains the largest k eigenvalues of X andWk ∈ RN×k contain the k eigenvectors of X associated with the eigenvalues in Λk. Q ∈ Ok is anyorthogonal matrix. The following lemma provides the global geometry of the population risk in (3.3),which also determines the values of ε and η in Assumption 2.1.

Lemma 3.1. Define U , U = WuΛ12uQ>, U? , U? = WkΛ

12

kQ> ⊆ U , and U?s , U\U?.

Denote κ ,√

λ1

λk≥ 1 as the condition number of any U? ∈ U?. Define the following regions:

R1 ,

U∈ RN×k∗ : min

P∈Ok

‖U−U?P‖F ≤ 0.2κ−1√λk, ∀U? ∈ U?

,

R′2 ,

U∈ RN×k∗ : σk(U) ≤ 1

2

√λk, ‖UU>‖F ≤

8

7‖U?U?>‖F , ‖ grad g(U)‖F ≤

1

80λ

32

k

,

R′′2 ,

U∈ RN×k∗ : σk(U) ≤ 1

2

√λk, ‖UU>‖F ≤

8

7‖U?U?>‖F , ‖ grad g(U)‖F >

1

80λ

32

k

,

R′3 ,

U∈ RN×k∗ : σk(U)>

1

2

√λk, min

P∈Ok

‖U−U?P‖F >0.2κ−1√λk, ‖UU>‖F ≤

8

7‖U?U?>‖F

,

R′′3 ,

U∈ RN×k∗ : ‖UU>‖F >

8

7‖U?U?>‖F

,

where σk(U) denotes the k-th singular value of a matrix U ∈ RN×k∗ , i.e., the smallest singularvalue of U. These regions also induce regions in the quotient manifoldM in an apparent way. Weadditionally assume that λk+1 ≤ 1

12λk and k ≤ r N . Then, the following properties hold:

(1) For any U ∈ U , U is a critical point of the population risk g(U) in (3.3).

(2) For any U? ∈ U?, U? is a global minimum of g(U) with λmin(hess g(U?)) ≥ 1.91λk. Moreover,

for any U ∈ R1, we have

λmin(hess g(U)) ≥ 0.19λk.

(3) For any U?s ∈ U?s , U?

s is a strict saddle point of g(U) with λmin(hess g(U?s)) ≤ −0.91λk.

Moreover, for any U ∈ R′2, we have

λmin(hess g(U)) ≤ −0.06λk.

(4) For any U ∈ R′′2⋃R′3⋃R′′3 , we have a large gradient. In particular,

‖ grad g(U)‖F >

180λ

32

k , if U ∈ R′′2 ,160κ−1λ

32

k , if U ∈ R′3,584k

14λ

32

k , if U ∈ R′′3 .

6

The proof of Lemma 3.1 is inspired by the proofs of [8, Theorem 4], [3, Lemma 13] and [4,Theorem 5], and is given in Appendix D (see supplementary material). Therefore, we can setε ≤ min1/80, 1/60κ−1λ

32

k and η = 0.06λk. Then, the population risk given in (3.3) satisfiesAssumption 2.1. It can be seen that each critical point of the population risk g(U) in (3.3) is eithera global minimum or a strict saddle, which inspires us to carry this favorable geometry over to thecorresponding empirical risk.

RNk<latexit sha1_base64="kqYJ9Roav7ehVoIj5r4u2uq1l+g=">AAAbmHicxdndbuPGFQBgJU3bxG0aJ72Lb4gYBYpFurCSAm3u1rZsWbb+LFk/9soxhtSI4ppDMuSItpdQn6S37Tv1bTokj3gOOdyguaqABXS+OTNDzRwNtbQZuE4kj47+89HHv/rk17/57aef7f3u95//4Yv9L7+aRv4mtPjE8l0/nJss4q7j8Yl0pMvnQciZMF0+Mx9P0/ZZzMPI8b0b+RLwe8Fsz1k5FpOKHvb3F6OHVz8m/YV0BI+Mx+3D/uHR66PsZehvmvDmsAGv4cOXXweLpW9tBPek5bIoets8CuR9wkLpWC7f7i02EQ+Y9chs/la99Zia6D7JLn1r/EnJ0lj5ofrnSSNT2iNhIopehKkyBZPrqNqWYl3b241c/f0+cbxgI7ln5ROtNq4hfSNdB2PphNyS7ot6w6zQUddqWGsWMkuq1dorTWOa4luY6tvd5ewtWlx92pD3lI6V+G6yiNZ+KIXjbaJtskjTTcfbJse9Mdsm33z/w1YNu+QrY+HaRv5KFi7z7GyRsoYQG0Jo2Ft4/MnyhWDeMlnElty+bd4nCzXfMsqnPWxut+UsIZ/zrOwaVjUZMgq1cbJsi+UDVvO5F/m/rEvEJV7ELqcyqvCDPMcPeMikH6bF8SrPNGBNYuYGa2Yk5YkzzObMk0wumVFNSpHk2EzNW83JkCS56ruzZJWkHEmWXKfzVbIyJEnPjqFd0rNDEryNnuBtSEKwdrQEZTQjqsmIaMaSu/qlZkiSuOrj+l41DZgkhmvf0C9aaZoDSaKdLXQ1qV1eadHKLqya1SpfmehmS1/N6lY2RAwEt/XBMiVZw3XNlgzpgophVJdCV1SMHbvm42VKsib52lWyJtUVFUOnZkGHDllPeWzsvoHJcdFRnqCeoJ6inqK2UFuoZ6hnqOeo56ht1DbqBeoFage1g3qJeol6hXqF2kXtovZQe6h91D7qAHWAOkQdol6jXqOOUEeoY9Qx6g3qDeoEdYI6RZ2izlBnqHPUOeot6i3qHeodFo/66qR3jYQViSaIWYgFYhWyBFkWwkF4ISuQVSE2iF3IGmRdiAPiFPIO5F0hjyCP5FgGcvGLAyIK8UC8QnwQv5AAJCjkJ5CfCglBwkIikKiQFxBZyAZkgxcNEhfyBPJUyDPIszbySyHvQd7jrsb6tsb6vsb6xsb6zsb61sb63sb65sb67sb69sb6/sb/0wbH+g7H+hbH+h7H+ibH+i7H+jbH+j7H+kbHNTutb3Ws73Wsb3as73Zcs90ivQOoX3Xk+BcnQHj2i1MgPPhFCwhPfXEGhEe+OAfC8160gfCwFxdAeNKLDhAe8+ISCM94cQWEB7zoAuHpLnpAeLSLPhCe62IAhIe6GALhiS6ugfA4FyMgPMvFGAgPcnEDhKe4mADhES6mQHh+ixkQHt5iDoQnt7gFwmNb3AGRM1v9Wk43XP2CJhuu8ATwhOIp4CnFFmCL4hngGcVzwHOKbcA2xQvAC4odwA7FS8BLileAVxS7gF2KPcAexT5gn+IAcEBxCDikeA14TXEEOKI4BhxTvAG8oTgBnFCcAk4pzgBnFOeAc4q3gLcU7wBJiahQLxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iahILxEV1ZSIaWZ3jfQRgEmKxDRPkE8InyKfEm4htwifIZ8RPkc+J9xGbhO+QL4g3EHuEL5EviR8hXxFuIvcJdxD7hHuI/cJD5AHhIfIQ8LXyNeER8gjwmPkMeEb5BvCE+QJ4SnylPAMeUZ4jjwnfIt8S/gOOSur0jOvATyxeaVyHM8Rznu+LT/HSQrffrAne67vufMP9mShLdKHa+V+Sv+R8s92Y8+13Rj+KoukUX0kFW3Md9ySC0P6C6NI9LzieZ3nexth8nDXFPjui+vb2kDgxRCB48X5ED8ulsy2cYRdg2pJ/2P3LJO/NPG5AX/hu6nhKR/5DdZZfrDtPQ/9ctsR+ekJjfkPUNLge9zAhibOhD3S3zDYQ2CPtAF77Bznx7Zf8lw0b1g6zIaG9GFiksbFcDIsemWNMiRNzOL1TWqCR9orjXEhuFUaUsW0o03HHKVx0Tr62db+xnXpuGlcNI4D5tHGNMZatb3SFakY2xy71DGNi8ZHXB3LCS2XL1kki3XlsjSqinENWLhrK5f2lOEqWv6urPMB0niLu7Yqj05aIlnezwgnXjveSr7kD8vWTCaLLMavxNBfrQx83jx8gEEUq6Tt50WaZ9SmeTTraZ0mPTlLrqYq7qn/z0vYHcvZV2t3Movs6wNtZ7tlwzvy7g4bCnKHrb0h1d7+K7cAKNa6e+VxuiS7D3SMKzLeeX6bUxxy+jeAwbZ8h6Vtx36wzY++aAWfLBv5Yf+wWf2LlP5m+t3r5vevv7v+6+GbE/hr1aeNg8Y3jT83mo2/Nd40LhrDxqRhNeLGPxv/avz74OuDNwftg06e+vFH0OePjdLrYPRfSj31Mw==</latexit>

<latexit sha1_base64="ovUlmdxxBVgVzrehS5ywkNWt+3s=">AAAbl3icxdlbb+PGFQBgJb2l26bZtE+tX4gYBYpgu7A2D2mfGtuyJduSdbMu9spdDKkRxTWHZMgRbS+h/oq+tv+r/6ZD8mjOoYYbNE8VsIDON2eG1MzhkEvbke8l8ujoP598+pOf/uznv/jsly9+9evPf/PFyy9/O03CTezwiRP6YTy3WcJ9L+AT6Umfz6OYM2H7fGY/nObts5THiRcGN/I54veCuYG38hwmFd0tll7gZs1v32zfvTw8en1UfCzzSxO+HDbgM3j35R8eF8vQ2QgeSMdnSfK2eRTJ+4zF0nN8vn2x2CQ8Ys4Dc/lb9TVggif3WXHKW+uPSpbWKozVv0BahdIeGRNJ8ixslSmYXCf7bTnWtb3dyNVf7jMviDaSB055oNXGt2Ro5b/fWnoxd6T/rL4wJ/bUuVrOmsXMkWqWXlQOY9viFRzqFZzOq8jLz1f9uBZXPzrmPdU4Vg2hny2SdRhL4QWbZJst8l62F2yz496YbbOvvvnrVo2+5Ctr4btW+ckWPgvcYq6KhhgbYmh4sQj4oxMKwYJltkgduX3bvM8W6njLpDzsYXO7rWYJ+VRmFeewqsmQSWyMU2Q7rBxwP58HSfjjuiRc4knscvZGFWFU5oQRj5kM47xGvi4zLZiTlPnRmllZ9cAFFscsk2wumbWflCPJcZk67n5OgSTJV5fOku0llUiy5Do/3l5WgSTpybOMU3rySEKwMROCDUmI1p6RoIxmJDUZCc1Yct881QJJEld9/DDYTwMmifE6tMyTVprnQJJoFxO9n9SuzrRoFSe2n9WqnpnoFlO/n9XdWxDRF9w1ByuUZA3WNUsyoBMqBkldCp1RMfbcmp9XKMmalHO3lzXZn1Ex8GomdOCR+ZTH1u4KzI51R3mCeoJ6inqK2kJtoZ6hnqGeo56jtlHbqB3UDuoF6gXqJeol6hXqFWoXtYvaQ+2hXqNeo/ZR+6gD1AHqEHWIOkIdoY5Rx6g3qDeoE9QJ6hR1ijpDnaHOUeeot6i3qHeod1g86tLJ7xoZ04k2iK3FAXG0LEGWWjgI17ICWWlxQVwta5C1Fg/E0/Ie5L2WB5AHsi0D+XjhgAgtAUigJQQJtUQgkZbvQb7XEoPEWhKQRMsziNSyAdngSYOkWh5BHrU8gTwZIz9r+QDyAVc1NZc1Ndc1NRc2NVc2NZc2Ndc2NRc3NVc3NZc3Ndc3/Z8WODVXODWXODXXODUXOTVXOTWXOTXXOTUXOq1ZaXOpU3OtU3OxU3O105rlFvkdQD3Vke1fnADh3i9OgXDjFy0g3PXFGRBu+eIcCPd70QbCzV50gHCnFxdAuM2LSyDc48UVEG7woguEu7voAeHWLq6BcF8XfSDc1MUACHd0MQTC7VyMgHAvF2Mg3MjFDRDu4mIChFu4mALh/i1mQLh5izkQ7tziFgi3bXEHRPZs9bScL7h6giYLrvAE8ITiKeApxRZgi+IZ4BnFc8Bzim3ANsUOYIfiBeAFxUvAS4pXgFcUu4Bdij3AHsVrwGuKfcA+xQHggOIQcEhxBDiiOAYcU7wBvKE4AZxQnAJOKc4AZxTngHOKt4C3FO8ASYmo0CwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIiswSUZFZIioyS0RFZomoyCwRFdWUiG0Xd438FYBNisS2T5BPCJ8inxJuIbcInyGfET5HPifcRm4T7iB3CF8gXxC+RL4kfIV8RbiL3CXcQ+4Rvka+JtxH7hMeIA8ID5GHhEfII8Jj5DHhG+QbwhPkCeEp8pTwDHlGeI48J3yLfEv4Drkoq8o7rz68sfla5XiBJ7wPfFt9j5Np3360J3uq77nzj/ZksSvyl2vVfkr/kfMPdmNPtd0YPpUl0tp/JZVs7PfckQtLhgtLJwaBfl8XhMFG2DzeNUWh/+yHrjEQuB4i8oK0HOLviyVzXRxh16Ba8v/YPcnsz018b8Cf+e7Q8JaPPINdLD/a9oHHYbXtiDx6QmP5AEoawoBb2NDEI2GP/BkGewjskTdgj53j8bHtx7wXLRuWHnOhIX+ZmOWxHk7GulfRKGPSxBxe36QO8EB75TFOBHcqQ6qYdnTpmKM81q2jH2y93vg+HTePdeM4YgFtzGOsVTeonJGKsc1zKx3zWDc+4Ow4Xuz4fMkSqeeVy8qoKsY5YPGurVraU4az6IS7si4HyOMtrtqqOjppSWR1PRM88NoLVvK5fFm2ZjJbFDFeEoNwtbLwffPgHQyiWCVtP9dpgVWbFtCsx3We9OgtuTqUvqf+P09hty0Xl9ZuZxbF5QNtZ7tpwzvy7g4bC3KHrb0h1d7+924BUKx198rjfEp2P+gYZ2S88/I2pzjm9G8A/W31DkvbjsNoW259yQp+WTHyu5eHzf0/TJlfpm9eN795/Wb45vC7E/ij1WeNg8ZXjT81mo1vG981Oo1BY9JwGkHjn41/Nf598PuDvx2cH3TK1E8/gT6/a1Q+B8P/AqBA9VY=</latexit>

≠<latexit sha1_base64="kDpf+tr573gcNjEl/RgPTMq30EY=">AAAbl3icxdlbb+PGFQBg5dIm2TbNJn1K/ELECFAE24W1+5D2qbEtW7ItWTfrYq+cxZAaUVxzSIYc0fYSyq/Ia/O/8m8yJI94DjncoHmqgAV0vjkzpGYOh1zaDFwnkgcHv773/gcf/unPH338yZO//PXTv3329PMvppG/CS0+sXzXD+cmi7jreHwiHenyeRByJkyXz8y747R9FvMwcnzvSj4G/FYw23NWjsWkopvF0vHspPndy+3rp/sHzw+yj6F/acKX/QZ8Bq8//+p+sfStjeCetFwWRa+aB4G8TVgoHcvl2yeLTcQDZt0xm79SXz0meHSbZKe8Nb5RsjRWfqj+edLIlPZImIiiR2GqTMHkOqq2pVjX9mojV/+6TRwv2EjuWfmBVhvXkL6R/n5j6YTcku6j+sKs0FHnalhrFjJLqll6UjqMaYpncKhncDrPAic9X/XjWlz96JD3VONYNfhusojWfiiF422ibbJIe5mOt00Oe2O2Tb5++e+tGn3JV8bCtY38kyxc5tnZXGUNITaE0PBk4fF7yxeCectkEVty+6p5myzU8ZZRftj95nZbzhLyIc/KzmFVkyGjUBsny7ZYPmA1n3uR/8e6RFziSexyKqMKP8hz/ICHTPphWiPf5pkGzEnM3GDNjKR84AyzY+ZJJpfMqCalSHJspo5bzcmQJLnq0lmySlKOJEuu0+NVsjIkSQ+OoZ3Sg0MSvI2e4G1IQrB2tARlNCOqyYhoxpK7+qlmSJK46uP6XjUNmCSGa9/QT1ppmgNJop1NdDWpXZ5p0cpOrJrVKp+Z6GZTX83qVhZE9AW39cEyJVmDdc2SDOiEikFUl0JnVIwdu+bnZUqyJvncVbIm1RkVA6dmQgcOmU95aOyuwOSw6CiPUI9Qj1GPUVuoLdQT1BPUU9RT1DZqG7WD2kE9Qz1DPUc9R71AvUDtonZRe6g91EvUS9Q+ah91gDpAHaIOUUeoI9Qx6hj1CvUKdYI6QZ2iTlFnqDPUOeoc9Rr1GvUG9QaLR1066V0jYUWiCWIWYoFYhSxBloVwEF7ICmRViA1iF7IGWRfigDiFvAF5U8gdyB3ZloFcvHBARCEeiFeID+IXEoAEhfwI8mMhIUhYSAQSFfIIIgvZgGzwpEHiQu5B7gt5AHnQRn4s5C3IW1zVWF/WWF/XWF/YWF/ZWF/aWF/bWF/cWF/dWF/eWF/f+H9a4Fhf4Vhf4lhf41hf5Fhf5Vhf5lhf51hf6LhmpfWljvW1jvXFjvXVjmuWW6R3APVUR7Z/cQSEe784BsKNX7SAcNcXJ0C45YtTINzvRRsIN3vRAcKdXpwB4TYvzoFwjxcXQLjBiy4Q7u6iB4Rbu7gEwn1d9IFwUxcDINzRxRAIt3MxAsK9XIyBcCMXV0C4i4sJEG7hYgqE+7eYAeHmLeZAuHOLayDctsUNENmz1dNyuuDqCZosuMIjwCOKx4DHFFuALYongCcUTwFPKbYB2xQ7gB2KZ4BnFM8BzyleAF5Q7AJ2KfYAexQvAS8p9gH7FAeAA4pDwCHFEeCI4hhwTPEK8IriBHBCcQo4pTgDnFGcA84pXgNeU7wBJCWiQr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVFRTIqaZ3TXSVwAmKRLTPEI+InyMfEy4hdwifIJ8QvgU+ZRwG7lNuIPcIXyGfEb4HPmc8AXyBeEucpdwD7lH+BL5knAfuU94gDwgPEQeEh4hjwiPkceEr5CvCE+QJ4SnyFPCM+QZ4TnynPA18jXhG+SsrErvvPrwxuZbleN4jnDe8m35PU5S+PadPdlDfc+dv7MnC22Rvlwr91P6U8q/24091HZj+FQWSaP6SiramG+4JReG9BdGkeh5xfs6z/c2wuThrinw3UfXt7WBwIshAseL8yF+WCyZbeMIuwbVkv7H7kEm/2ziewP+yHeHhrd85BnsbPnOtrc89MttB+TRExrzB1DS4HvcwIYmHgl7pM8w2ENgj7QBe+wcj49tf+S9aN6wdJgNDenLxCSNi+FkWPTKGmVImpjF65vUAe5orzTGieBWaUgV0442HXOUxkXr6HdbLzeuS8dN46JxHDCPNqYx1qrtlc5Ixdjm2KWOaVw03uHsWE5ouXzJIlnMK5elUVWMc8DCXVu5tKcMZ9Hyd2WdD5DGW1y1VXl00hLJ8npGeOC1463kY/6ybM1ksshivCQG/mpl4PvmwWsYRLFK2n5apHlGbZpHs+7XadK9s+TqUMU99f95CrttObu0djuzyC4faDvZTRvekXd32FCQO2ztDan29l+5BUCx1t0rD9Mp2f2gQ5yR8c7z25zikNO/AfS35TssbTv0g22+9UUr+GXZyK+f7jerf5jSv0xfPG++fP5i+GL/+yP4o9XHjb3G141/NJqN7xrfNzqNQWPSsBpe4+fGfxu/7H2595+9071Onvr+e9Dn743SZ2/4G7tr9Vc=</latexit>

Æ<latexit sha1_base64="8ZaXGbhmjqhglTEE8ySItJRGj98=">AAAbl3icxdlbb+O4FQBg7/a2nXbb2fapzYuwQYFiMR3EswW2feomcWInseNbfEnG6YCSaVkTUdJKtJKM4P6Kvrb/q/+mlHTMc2RqFt2nGhjA5+MhJZNHlEaxI99L5NHRfz759Ec//slPf/bZz1/84pef/+rXL7/4zTQJN7HDJ07oh/HcZgn3vYBPpCd9Po9izoTt85n9cJq3z1IeJ14Y3MjniN8L5gbeynOYVHS3WHqBmzW/+fP23cvDo9dHxccyvzThy2EDPoN3X/z+cbEMnY3ggXR8liRvm0eRvM9YLD3H59sXi03CI+Y8MJe/VV8DJnhynxWnvLX+oGRprcJY/QukVSjtkTGRJM/CVpmCyXWy35ZjXdvbjVz95T7zgmgjeeCUB1ptfEuGVv77raUXc0f6z+oLc2JPnavlrFnMHKlm6UXlMLYtXsGhXsHpvIq8/HzVj2tx9aNj3lONY9UQ+tkiWYexFF6wSbbZIu9le8E2O+6N2Tb78uu/btXoS76yFr5rlZ9s4bPALeaqaIixIYaGF4uAPzqhECxYZovUkdu3zftsoY63TMrDHja322qWkE9lVnEOq5oMmcTGOEW2w8oB9/N5kIQ/rEvCJZ7ELmdvVBFGZU4Y8ZjJMM5r5Ksy04I5SZkfrZmVVQ9cYHHMMsnmkln7STmSHJep4+7nFEiSfHXpLNleUokkS67z4+1lFUiSnjzLOKUnjyQEGzMh2JCEaO0ZCcpoRlKTkdCMJffNUy2QJHHVxw+D/TRgkhivQ8s8aaV5DiSJdjHR+0nt6kyLVnFi+1mt6pmJbjH1+1ndvQURfcFdc7BCSdZgXbMkAzqhYpDUpdAZFWPPrfl5hZKsSTl3e1mT/RkVA69mQgcemU95bO2uwOxYd5QnqCeop6inqC3UFuoZ6hnqOeo5ahu1jdpB7aBeoF6gXqJeol6hXqF2UbuoPdQe6jXqNWoftY86QB2gDlGHqCPUEeoYdYx6g3qDOkGdoE5Rp6gz1BnqHHWOeot6i3qHeofFoy6d/K6RMZ1og9haHBBHyxJkqYWDcC0rkJUWF8TVsgZZa/FAPC3vQd5reQB5INsykI8XDojQEoAEWkKQUEsEEmn5DuQ7LTFIrCUBSbQ8g0gtG5ANnjRIquUR5FHLE8iTMfKzlg8gH3BVU3NZU3NdU3NhU3NlU3NpU3NtU3NxU3N1U3N5U3N90/9pgVNzhVNziVNzjVNzkVNzlVNzmVNznVNzodOalTaXOjXXOjUXOzVXO61ZbpHfAdRTHdn+xQkQ7v3iFAg3ftECwl1fnAHhli/OgXC/F20g3OxFBwh3enEBhNu8uATCPV5cAeEGL7pAuLuLHhBu7eIaCPd10QfCTV0MgHBHF0Mg3M7FCAj3cjEGwo1c3ADhLi4mQLiFiykQ7t9iBoSbt5gD4c4tboFw2xZ3QGTPVk/L+YKrJ2iy4ApPAE8ongKeUmwBtiieAZ5RPAc8p9gGbFPsAHYoXgBeULwEvKR4BXhFsQvYpdgD7FG8Brym2AfsUxwADigOAYcUR4AjimPAMcUbwBuKE8AJxSnglOIMcEZxDjineAt4S/EOkJSICs0SUZFZIioyS0RFZomoyCwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIiswSUVFNidh2cdfIXwHYpEhs+wT5hPAp8inhFnKL8BnyGeFz5HPCbeQ24Q5yh/AF8gXhS+RLwlfIV4S7yF3CPeQe4Wvka8J95D7hAfKA8BB5SHiEPCI8Rh4TvkG+ITxBnhCeIk8Jz5BnhOfIc8K3yLeE75CLsqq88+rDG5uvVI4XeML7wLfV9ziZ9u1He7Kn+p47/2hPFrsif7lW7af0Hzl/bzf2VNuN4VNZIq39V1LJxn7PHbmwZLiwdGIQ6Pd1QRhshM3jXVMU+s9+6BoDgeshIi9IyyH+vlgy18URdg2qJf+P3ZPM/tTE9wb8me8ODW/5yDPYxfKjbR94HFbbjsijJzSWD6CkIQy4hQ1NPBL2yJ9hsIfAHnkD9tg5Hh/bfsh70bJh6TEXGvKXiVke6+FkrHsVjTImTczh9U3qAA+0Vx7jRHCnMqSKaUeXjjnKY906+t7W643v03HzWDeOIxbQxjzGWnWDyhmpGNs8t9Ixj3XjA86O48WOz5cskXpeuayMqmKcAxbv2qqlPWU4i064K+tygDze4qqtqqOTlkRW1zPBA6+9YCWfy5dlayazRRHjJTEIVysL3zcP3sEgilXS9nOdFli1aQHNelznSY/ekqtD6Xvq//MUdttycWntdmZRXD7QdrabNrwj7+6wsSB32NobUu3tf+8WAMVad688zqdk94OOcUbGOy9vc4pjTv8G0N9W77C07TiMtuXWl6zglxUjv3t52Nz/w5T5ZfrmdfPr12+Gbw6/PYE/Wn3WOGh82fhjo9n4pvFto9MYNCYNpxE0/tn4V+PfB787+NvB+UGnTP30E+jz20blczD8L9aW9Vg=</latexit>

Figure 2: Partition of regions inLemma 3.1.

To illustrate the partition of the manifold RN×k∗ used in the aboveLemma 3.1, we use the purple (¬), yellow (), and green (®) re-gions in Figure 2 to denote the regions that satisfy minP∈Ok

‖U−U?P‖F ≤ 0.2κ−1

√λk, σk(U) ≤ 1

2

√λk, and ‖UU>‖F ≤

87‖U?U?>‖F , respectively. It can be seen thatR1 is exactly the pur-ple region, which contains the areas near the global minima ([U?]).R2 = R′2

⋃R′′2 is the intersection of the yellow and green regions.R′3 is the part of the green region that does not intersect with thepurple or yellow regions. Finally, R′′3 is the space outside of thegreen region. Therefore, the union ofR1,R2, andR3 = R′3

⋃R′′3covers the entire manifold RN×k∗ .

We define a norm ball as B(l) , U ∈ RN×k∗ : ‖UU>‖F ≤ l with l = 87‖U?U?>‖F . The

following lemma verifies Assumptions 2.2 and 2.3 under the restricted isometry property (RIP).Lemma 3.2. Assume r

2 ≤ k ≤ r N . Suppose that a linear operator B with [B(Z)]m = 〈Z,Bm〉satisfies the following RIP

(1− δr+k)‖Z‖2F ≤ ‖B(Z)‖22 ≤ (1 + δr+k)‖Z‖2F (3.4)

for any matrix Z ∈ RN×N with rank at most r + k. We construct the linear operator A by settingAm = 1

2 (Bm + B>m). If the restricted isometry constant δr+k satisfies

δr+k≤min

ε

2√

87k

14(87‖U?U?>‖F+‖X‖F)‖U?U?>‖

12

F

,1

36,

η

2(167√k‖U?U?>‖F+ 8

7‖U?U?>‖F+‖X‖F)

then, we have

supU∈B(l)

‖ grad f(U)− grad g(U)‖F ≤ε

2, and sup

U∈B(l)‖hess f(U)− hess g(U)‖2 ≤

η

2.

The proof of Lemma 3.2 is given in Appendix E (see supplementary material). As is shown in existingliterature [25, 26, 27], a Gaussian linear operator B : RN×N → RM satisfies the RIP condition (3.4)with high probability if M ≥ C(r + k)N/δ2r+k for some numerical constant C. Therefore, we canconclude that the three statements in Theorem 2.1 hold for the empirical risk (3.2) and populationrisk (3.3) as long as M is large enough. Some similar bounds for the sample complexity M underdifferent settings can also be found in papers [8, 4]. Note that the particular choice of l can guaranteethat ‖ grad f(U)‖F is large outside of B(l), which is also proved in Appendix E. Together withTheorem 2.1, we prove a globally benign landscape for the empirical risk.

3.2 Phase Retrieval

We continue to elaborate on Example 1.2. The following lemma provides the global geometry of thepopulation risk in (1.2), which also determines the values of ε and η in Assumption 2.1.Lemma 3.3. Define the following four regions:

R1 ,

x ∈ RN : ‖x‖2 ≤

1

2‖x?‖2

, R2 ,

x ∈ RN : min

γ∈1,−1‖x− γx?‖2 ≤

1

10‖x?‖2

,

R3 ,

x ∈ RN : min

γ∈1,−1

∥∥∥∥x− γ 1√3‖x?‖2w

∥∥∥∥2

≤ 1

5‖x?‖2, w>x? = 0, ‖w‖2 = 1

,

R4 ,

x ∈ RN : ‖x‖2 >

1

2‖x?‖2, min

γ∈1,−1‖x− γx?‖2 >

1

10‖x?‖2,

minγ∈1,−1

∥∥∥∥x− γ 1√3‖x?‖2w

∥∥∥∥2

>1

5‖x?‖2, w>x? = 0, ‖w‖2 = 1

7

Then, the following properties hold:

(1) x = 0 is a strict saddle point with ∇2g(0) = −4x?x?> − 2‖x?‖22IN and λmin(∇2g(0)) =−6‖x?‖22. Moreover, for any x ∈ R1, the neighborhood of strict saddle point 0, we have

λmin(∇2g(x)) ≤ −3

2‖x?‖22.

(2) x = ±x? are global minima with∇2g(±x?) = 8x?x?> + 4‖x?‖22IN and λmin(∇2g(±x?)) =4‖x?‖22. Moreover, for any x ∈ R2, the neighborhood of global minima ±x?, we have

λmin(∇2g(x)) ≥ 0.22‖x?‖22.

(3) x = ± 1√3‖x?‖2w, with w>x? = 0 and ‖w‖2 = 1, are strict saddle points with

∇2g(± 1√3‖x?‖2w) = 4‖x?‖22ww> − 4x?x?> and λmin(∇2g(± 1√

3‖x?‖2w)) = −4‖x?‖22.

Moreover, for any x ∈ R3, the neighborhood of strict saddle points ± 1√3‖x?‖2w, we have

λmin(∇2g(x)) ≤ −0.78‖x?‖22.

(4) For any x ∈ R4, the complement region ofR1,R2, andR3, we have ‖∇g(x)‖2 > 0.3963‖x?‖32.

RN<latexit sha1_base64="QsD84PYZQ5aPxGwA1e5S/b+JU2M=">AAAbkXicxdlbb+PGFQBgJW3SdJukm+atfiFiFAiCzcJKCrR5W9uy5YtulqyLvXIWQ2pEcc0hGXJE20uoP6Gv7V/rv+mQPJpzqOEGyVMELKDzzZkhNXM45NJ25HuJPDj43wcf/u73H338h0/++OxPn372+Z+ff/GXSRKuY4ePndAP45nNEu57AR9LT/p8FsWcCdvnU/v+OG+fpjxOvDC4lk8RvxPMDbyl5zCZ03z4Y+/N8/2DlwfFxzK/NOHLfgM+gzdf/PVhvgidteCBdHyWJK+bB5G8y1gsPcfnm2fzdcIj5twzl79WXwMmeHKXFSe7sf6mZGEtw1j9C6RVKO2RMZEkT8JWmYLJVbLblmNd2+u1XP7zLvOCaC154JQHWq59S4ZW/suthRdzR/pP6gtzYk+dq+WsWMwcqebnWeUwti1ewKFewOm8iLz8fNWPa3H1o2PeVY0j1RD62TxZhbEUXrBONtk872V7wSY77I7YJvvq+x82avQFX1pz37XKTzb3WeAWc1U0xNgQQ8OzecAfnFAIFiyyeerIzevmXTZXx1sk5WH3m5tNNUvIxzKrOIdlTYZMYmOcItth5YC7+TxIwl/XJeEST2KbszOqCKMyJ4x4zGQY5zXyTZlpwZykzI9WzMqqBy6wOGaZZHPJrN2kHEmOy9Rxd3MKJEm+umgWbCepRJIlV/nxdrIKJEmPnmWc0qNHEoK1mRCsSUK08owEZTQjqclIaMaC++apFkiSuOrjh8FuGjBJjFehZZ600jwHkkS7mOjdpHZ1pkWrOLHdrFb1zESnmPrdrM7Ogoi+4K45WKEka7CqWZIBnVAxSOpS6IyKkefW/LxCSda4nLudrPHujIqBVzOhA4/Mpzy0tldgdqg7yiPUI9Rj1GPUFmoL9QT1BPUU9RS1jdpGPUM9Qz1HPUe9QL1AvUS9RO2gdlC7qF3UHmoPtY/aRx2gDlCvUK9Qh6hD1BHqCPUa9Rp1jDpGnaBOUKeoU9QZ6gz1BvUG9Rb1FotHXTr5XSNjOtEGsbU4II6WBchCCwfhWpYgSy0uiKtlBbLS4oF4Wt6CvNVyD3JPtmUgHy8cEKElAAm0hCChlggk0vITyE9aYpBYSwKSaHkCkVrWIGs8aZBUywPIg5ZHkEdj5Cct70De4aqm5rKm5rqm5sKm5sqm5tKm5tqm5uKm5uqm5vKm5vqmv2iBU3OFU3OJU3ONU3ORU3OVU3OZU3OdU3Oh05qVNpc6Ndc6NRc7NVc7rVlukd8B1FMd2f7FERDu/eIYCDd+0QLCXV+cAOGWL06BcL8XbSDc7MUZEO704hwIt3lxAYR7vLgEwg1edIBwdxddINzaRQ8I93XRB8JNXQyAcEcXV0C4nYshEO7lYgSEG7m4BsJdXIyBcAsXEyDcv8UUCDdvMQPCnVvcAOG2LW6ByJ6tnpbzBVdP0GTBFR4BHlE8Bjym2AJsUTwBPKF4CnhKsQ3YpngGeEbxHPCc4gXgBcVLwEuKHcAOxS5gl2IPsEexD9inOAAcULwCvKI4BBxSHAGOKF4DXlMcA44pTgAnFKeAU4ozwBnFG8AbireApERUaJaIiswSUZFZIioyS0RFZomoyCwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIimpKxLaLu0b+CsAmRWLbR8hHhI+Rjwm3kFuET5BPCJ8inxJuI7cJnyGfET5HPid8gXxB+BL5knAHuUO4i9wl3EPuEe4j9wkPkAeEr5CvCA+Rh4RHyCPC18jXhMfIY8IT5AnhKfKU8Ax5RvgG+YbwLXJRVpV3Xn14Y/ONyvECT3jv+Kb6HifTvnlvT/ZY33Pr7+3JYlfkL9eq/ZT+K+ef7cYea7sxfCpLpLX7SipZ22+5I+eWDOeWTgwC/b4uCIO1sHm8bYpC/8kPXWMgcD1E5AVpOcSP8wVzXRxh26Ba8v/YPcrs2ya+N+BPfHtoeMtHnsHOF+9te8fjsNp2QB49obF8ACUNYcAtbGjikbBH/gyDPQT2yBuwx9bx+Nj2a96Llg0Lj7nQkL9MzPJYDydj3atolDFpYg6vb1IHuKe98hgngjuVIVVMO7p0zGEe69bhz7b21r5Px81j3TiKWEAb8xhr1Q0qZ6RibPPcSsc81o33ODuOFzs+X7BE6nnlsjKqinEOWLxtq5b2hOEsOuG2rMsB8niDq7asjk5aElldzwQPvPKCpXwqX5atmMzmRYyXxCBcLi183zx4A4MoVkmbz3RaYNWmBTTrYZUnPXgLrg6l76m/5Slst+Xi0truzKK4fKDtZDtteEfe3mFjQe6wtTek2tv/zi0AirXuXnmYT8n2Bx3ijIy2Xt7mFMec/g2gv6neYWnbYRhtyq0vWcIvK0Z+83y/ufuHKfPL5LuXze9ffnf19/1XR/BHq08ae42vGl83mo1/NF41zhqDxrjhNFaNfzf+0/jv3pd7P+y92oPcDz+APl82Kp+9y/8DCBjy5g==</latexit>

R1<latexit sha1_base64="fhf23CdZVPHjX4j+XUzHy/ZGZMg=">AAAblHicxdlbb+PGFQBgJU3bdNs0mxToQ/xCxChQBNuFlQRICvQhtiTLF90sWRd75RpDakRxzSEZckTbS6j/oa/tL+u/6ZA84jnkcIPmqQIW0PnmzJCaORxyaTNwnUgeHf3ngw9/8dEvf/Xrj3/z4re/++T3n7787PNZ5G9Di08t3/XDhcki7joen0pHunwRhJwJ0+Vz86GVts9jHkaO713L54DfCWZ7ztqxmFQ0X1rMHd83718eHr0+yj6G/qUJXw4b8Bndf/bF43LlW1vBPWm5LIreNI8CeZewUDqWy3cvltuIB8x6YDZ/o756TPDoLsnOd2f8ScnKWPuh+udJI1PaI2Eiip6FqTIFk5uo2pZiXdubrVx/f5c4XrCV3LPyA623riF9I/3xxsoJuSXdZ/WFWaGjztWwNixkllRT9KJ0GNMUr+BQr+B0XgVOer7qx7W5+tEh76vGiWrw3WQZbfxQCsfbRrtkmfYyHW+XHPcnbJd8+c1fd2r0FV8bS9c28k+ydJlnZ3OVNYTYEELDi6XHHy1fCOatkmVsyd2b5l2yVMdbRflhD5u7XTlLyKc8KzuHdU2GjEJtnCxb1UKWXs3nXuT/vC4Rl3gS+5zKqMIP8hw/4CGTfpjWyFd5pgFzEjM32DAjKR84w+yYeZLJJTOqSSmSHJup41ZzMiRJrrpuVqySlCPJkpv0eJWsDEnSk2Nop/TkkARvqyd4W5IQbBwtQRnNiGoyIpqx4q5+qhmSJK76uL5XTQMmieHGN/STVprmQJLoZhNdTeqWZ1q0sxOrZrXLZyZ62dRXs3qVBRFDwW19sExJ1mhTsyQjOqFiFNWl0BkVE8eu+XmZkqxpPneVrGl1RsXIqZnQkUPmUx4b+yswOS46yhPUE9QWagu1jdpG7aB2UE9RT1G7qF3UM9Qz1HPUc9QL1AvUS9RL1B5qD7WP2kcdoA5Qh6hD1BHqCPUK9Qp1jDpGnaBOUK9Rr1GnqFPUGeoMdY46R12gLlBvUG9Qb1FvsXjUpZPeNRJWJJogZiEWiFXICmRVCAfhhaxB1oXYIHYhG5BNIQ6IU8hbkLeFPIA8kG0ZyMULB0QU4oF4hfggfiEBSFDIjyA/FhKChIVEIFEhzyCykC3IFk8aJC7kEeSxkCeQJ23k50LegbzDVY31ZY31dY31hY31lY31pY31tY31xY311Y315Y319Y3/pwWO9RWO9SWO9TWO9UWO9VWO9WWO9XWO9YWOa1ZaX+pYX+tYX+xYX+24ZrlFegdQT3Vk+xcnQLj3ixYQbvyiDYS7vugA4ZYvToFwvxddINzsxRkQ7vTiHAi3eXEBhHu8uATCDV70gHB3F30g3NrFAAj3dTEEwk1djIBwRxdXQLidizEQ7uViAoQbubgGwl1cTIFwCxczINy/xRwIN2+xAMKdW9wA4bYtboHInq2eltMFV0/QZMEVngCeUGwBtii2AdsUO4AdiqeApxS7gF2KZ4BnFM8BzyleAF5QvAS8pNgD7FHsA/YpDgAHFIeAQ4ojwBHFK8ArimPAMcUJ4ITiNeA1xSnglOIMcEZxDjinuABcULwBvKF4C0hKRIV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iaiopkRMM7trpK8ATFIkpnmCfEK4hdwi3EZuE+4gdwifIp8S7iJ3CZ8hnxE+Rz4nfIF8QfgS+ZJwD7lHuI/cJzxAHhAeIg8Jj5BHhK+QrwiPkceEJ8gTwtfI14SnyFPCM+QZ4TnynPACeUH4BvmG8C1yVlald15DeGPzlcpxPEc47/iu/B4nKXz33p7sqb7n3t/bk4W2SF+ulfsp/UfKP9mNPdV2Y/hUFkmj+koq2ppvuSWXhvSXRpHoecX7Os/3tsLk4b4p8N1n17e1gcCLIQLHi/Mh/r5cMdvGEfYNqiX9j92TTP7SxPcG/JnvDw1v+cgz2PnqvW3veOiX247Ioyc05g+gpMH3uIENTTwS9kifYbCHwB5pA/bYOx4f237Oe9G8YeUwGxrSl4lJGhfDybDolTXKkDQxi9c3qQM80F5pjBPBrdKQKqYdbTrmOI2L1vFPtg62rkvHTeOicRIwjzamMdaq7ZXOSMXY5tiljmlcND7g7FhOaLl8xSJZzCuXpVFVjHPAwn1bubRnDGfR8vdlnQ+QxjtctXV5dNISyfJ6RnjgjeOt5XP+smzDZLLMYrwkRv56beD75tE9DKJYJe0+KdI8ozbNo1mPmzTp0Vlxdajinvr/PIX9tpxdWvudWWSXD7R19tOGd+T9HTYU5A5be0Oqvf1XbgFQrHX3yuN0SvY/6BhnZLL3/DanOOT0bwDDXfkOS9uO/WCXb33RGn5ZNvL9y8Nm9Q9T+pfZ16+b37z++urbwx9O4I9WHzcOGl82/txoNr5r/NA4a4wa04bVeGj8s/Gvxr8P/njwt4PWQSdP/fAD6POHRulzMPgvFKT0GA==</latexit>

·<latexit sha1_base64="yfLByaGqA9Kze7ECSuWRNUQnOz0=">AAAbknicxdndbuPGFQBgJWmbdNs0m6ZX9Q0Ro0ARbBdWEiBNrmJLtvwjWbJk/dgrZzGkRhTXHJIhR7S9hPoKvW0eLW/TIXk051DDDZqrClhA55szQ2rmcMil7cj3Enlw8PN773/wm9/+7sOPfv/sD3/8+E+fPP/0z5MkXMcOHzuhH8YzmyXc9wI+lp70+SyKORO2z6f2fStvn6Y8TrwwuJZPEb8TzA28pecwqWg8dxahfP18/+DlQfGxzC9N+LLfgM/g9ad/fZgvQmcteCAdnyXJq+ZBJO8yFkvP8fnm2Xyd8Ig598zlr9TXgAme3GXF2W6svylZWMswVv8CaRVKe2RMJMmTsFWmYHKV7LblWNf2ai2X/7zLvCBaSx445YGWa9+SoZX/dGvhxdyR/pP6wpzYU+dqOSsWM0eqCXpWOYxtixdwqBdwOi8iLz9f9ePaXP3omPdU40g1hH42T1ZhLIUXrJNNNs972V6wyQ57I7bJPv/q240afcGX1tx3rfKTzX0WuMVcFQ0xNsTQ8Gwe8AcnFIIFi2yeOnLzqnmXzdXxFkl52P3mZlPNEvKxzCrOYVmTIZPYGKfIdlg54G4+D5Lw13VJuMST2ObsjCrCqMwJIx4zGcZ5jXxRZlowJynzoxWzsuqBCyyOWSbZXDJrNylHkuMyddzdnAJJkq+umgXbSSqRZMlVfrydrAJJ0qNnGaf06JGEYG0mBGuSEK08I0EZzUhqMhKaseC+eaoFkiSu+vhhsJsGTBLjVWiZJ600z4Ek0SkmejepU51p0S5ObDerXT0z0S2mfjeru7Mgoi+4aw5WKMkarGqWZEAnVAySuhQ6o2LkuTU/r1CSNS7nbidrvDujYuDVTOjAI/MpD63tFZgd6o7yCPUItYXaQm2jtlGPUY9RT1BPUDuoHdRT1FPUM9Qz1HPUc9QL1AvULmoXtYfaQ71EvUTto/ZRB6gD1CvUK9Qh6hB1hDpCvUa9Rh2jjlEnqBPUKeoUdYY6Q71BvUG9Rb3F4lGXTn7XyJhOtEFsLQ6Io2UBstDCQbiWJchSiwvialmBrLR4IJ6WNyBvtNyD3JNtGcjHCwdEaAlAAi0hSKglAom0/Ajyo5YYJNaSgCRankCkljXIGk8aJNXyAPKg5RHk0Rj5SctbkLe4qqm5rKm5rqm5sKm5sqm5tKm5tqm5uKm5uqm5vKm5vun/tMCpucKpucSpucapucipucqpucypuc6pudBpzUqbS52aa52ai52aq53WLLfI7wDqqY5s/+IICPd+0QLCjV+0gXDXF8dAuOWLEyDc70UHCDd7cQqEO704A8JtXpwD4R4vLoBwgxddINzdRQ8It3ZxCYT7uugD4aYuBkC4o4srINzOxRAI93IxAsKNXFwD4S4uxkC4hYsJEO7fYgqEm7eYAeHOLW6AcNsWt0Bkz1ZPy/mCqydosuAKjwCPKLYAWxTbgG2Kx4DHFE8ATyh2ADsUTwFPKZ4BnlE8BzyneAF4QbEL2KXYA+xRvAS8pNgH7FMcAA4oXgFeURwCDimOAEcUrwGvKY4BxxQngBOKU8ApxRngjOIN4A3FW0BSIio0S0RFZomoyCwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIiswSUZFZIioyS0RFNSVi28VdI38FYJMise0j5CPCLeQW4TZym/Ax8jHhE+QTwh3kDuFT5FPCZ8hnhM+RzwlfIF8Q7iJ3CfeQe4QvkS8J95H7hAfIA8JXyFeEh8hDwiPkEeFr5GvCY+Qx4QnyhPAUeUp4hjwjfIN8Q/gWuSiryjuvPryx+ULleIEnvLd8U32Pk2nfvLMne6zvufV39mSxK/KXa9V+Sv+V8y92Y4+13Rg+lSXS2n0llaztN9yRc0uGc0snBoF+XxeEwVrYPN42RaH/5IeuMRC4HiLygrQc4of5grkujrBtUC35f+weZfaPJr434E98e2h4y0eewc4W72x7y+Ow2nZAHj2hsXwAJQ1hwC1saOKRsEf+DIM9BPbIG7DH1vH42PZr3ouWDQuPudCQv0zM8lgPJ2Pdq2iUMWliDq9vUge4p73yGCeCO5UhVUw7unTMYR7r1uEvtl6ufZ+Om8e6cRSxgDbmMdaqG1TOSMXY5rmVjnmsG+9xdhwvdny+YInU88plZVQV4xyweNtWLe0Jw1l0wm1ZlwPk8QZXbVkdnbQksrqeCR545QVL+VS+LFsxmc2LGC+JQbhcWvi+efAaBlGskjYf67TAqk0LaNbDKk968BZcHUrfU/+fp7DdlotLa7szi+Lygbbj7bThHXl7h40FucPW3pBqb/87twAo1rp75WE+JdsfdIgzMtp6eZtTHHP6N4D+pnqHpW2HYbQpt75kCb+sGPn18/3m7h+mzC+TL182v3r55dXX+98fwR+tPmrsNT5v/L3RbHzT+L5x2hg0xg2n4TX+3fhP46e9v+x9t3e41ypT338P+nzWqHz2uv8FiajznA==</latexit>



·<latexit sha1_base64="yfLByaGqA9Kze7ECSuWRNUQnOz0=">AAAbknicxdndbuPGFQBgJWmbdNs0m6ZX9Q0Ro0ARbBdWEiBNrmJLtvwjWbJk/dgrZzGkRhTXHJIhR7S9hPoKvW0eLW/TIXk051DDDZqrClhA55szQ2rmcMil7cj3Enlw8PN773/wm9/+7sOPfv/sD3/8+E+fPP/0z5MkXMcOHzuhH8YzmyXc9wI+lp70+SyKORO2z6f2fStvn6Y8TrwwuJZPEb8TzA28pecwqWg8dxahfP18/+DlQfGxzC9N+LLfgM/g9ad/fZgvQmcteCAdnyXJq+ZBJO8yFkvP8fnm2Xyd8Ig598zlr9TXgAme3GXF2W6svylZWMswVv8CaRVKe2RMJMmTsFWmYHKV7LblWNf2ai2X/7zLvCBaSx445YGWa9+SoZX/dGvhxdyR/pP6wpzYU+dqOSsWM0eqCXpWOYxtixdwqBdwOi8iLz9f9ePaXP3omPdU40g1hH42T1ZhLIUXrJNNNs972V6wyQ57I7bJPv/q240afcGX1tx3rfKTzX0WuMVcFQ0xNsTQ8Gwe8AcnFIIFi2yeOnLzqnmXzdXxFkl52P3mZlPNEvKxzCrOYVmTIZPYGKfIdlg54G4+D5Lw13VJuMST2ObsjCrCqMwJIx4zGcZ5jXxRZlowJynzoxWzsuqBCyyOWSbZXDJrNylHkuMyddzdnAJJkq+umgXbSSqRZMlVfrydrAJJ0qNnGaf06JGEYG0mBGuSEK08I0EZzUhqMhKaseC+eaoFkiSu+vhhsJsGTBLjVWiZJ600z4Ek0SkmejepU51p0S5ObDerXT0z0S2mfjeru7Mgoi+4aw5WKMkarGqWZEAnVAySuhQ6o2LkuTU/r1CSNS7nbidrvDujYuDVTOjAI/MpD63tFZgd6o7yCPUItYXaQm2jtlGPUY9RT1BPUDuoHdRT1FPUM9Qz1HPUc9QL1AvULmoXtYfaQ71EvUTto/ZRB6gD1CvUK9Qh6hB1hDpCvUa9Rh2jjlEnqBPUKeoUdYY6Q71BvUG9Rb3F4lGXTn7XyJhOtEFsLQ6Io2UBstDCQbiWJchSiwvialmBrLR4IJ6WNyBvtNyD3JNtGcjHCwdEaAlAAi0hSKglAom0/Ajyo5YYJNaSgCRankCkljXIGk8aJNXyAPKg5RHk0Rj5SctbkLe4qqm5rKm5rqm5sKm5sqm5tKm5tqm5uKm5uqm5vKm5vun/tMCpucKpucSpucapucipucqpucypuc6pudBpzUqbS52aa52ai52aq53WLLfI7wDqqY5s/+IICPd+0QLCjV+0gXDXF8dAuOWLEyDc70UHCDd7cQqEO704A8JtXpwD4R4vLoBwgxddINzdRQ8It3ZxCYT7uugD4aYuBkC4o4srINzOxRAI93IxAsKNXFwD4S4uxkC4hYsJEO7fYgqEm7eYAeHOLW6AcNsWt0Bkz1ZPy/mCqydosuAKjwCPKLYAWxTbgG2Kx4DHFE8ATyh2ADsUTwFPKZ4BnlE8BzyneAF4QbEL2KXYA+xRvAS8pNgH7FMcAA4oXgFeURwCDimOAEcUrwGvKY4BxxQngBOKU8ApxRngjOIN4A3FW0BSIio0S0RFZomoyCwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIiswSUZFZIioyS0RFNSVi28VdI38FYJMise0j5CPCLeQW4TZym/Ax8jHhE+QTwh3kDuFT5FPCZ8hnhM+RzwlfIF8Q7iJ3CfeQe4QvkS8J95H7hAfIA8JXyFeEh8hDwiPkEeFr5GvCY+Qx4QnyhPAUeUp4hjwjfIN8Q/gWuSiryjuvPryx+ULleIEnvLd8U32Pk2nfvLMne6zvufV39mSxK/KXa9V+Sv+V8y92Y4+13Rg+lSXS2n0llaztN9yRc0uGc0snBoF+XxeEwVrYPN42RaH/5IeuMRC4HiLygrQc4of5grkujrBtUC35f+weZfaPJr434E98e2h4y0eewc4W72x7y+Ow2nZAHj2hsXwAJQ1hwC1saOKRsEf+DIM9BPbIG7DH1vH42PZr3ouWDQuPudCQv0zM8lgPJ2Pdq2iUMWliDq9vUge4p73yGCeCO5UhVUw7unTMYR7r1uEvtl6ufZ+Om8e6cRSxgDbmMdaqG1TOSMXY5rmVjnmsG+9xdhwvdny+YInU88plZVQV4xyweNtWLe0Jw1l0wm1ZlwPk8QZXbVkdnbQksrqeCR545QVL+VS+LFsxmc2LGC+JQbhcWvi+efAaBlGskjYf67TAqk0LaNbDKk968BZcHUrfU/+fp7DdlotLa7szi+Lygbbj7bThHXl7h40FucPW3pBqb/87twAo1rp75WE+JdsfdIgzMtp6eZtTHHP6N4D+pnqHpW2HYbQpt75kCb+sGPn18/3m7h+mzC+TL182v3r55dXX+98fwR+tPmrsNT5v/L3RbHzT+L5x2hg0xg2n4TX+3fhP46e9v+x9t3e41ypT338P+nzWqHz2uv8FiajznA==</latexit> ·<latexit sha1_base64="yfLByaGqA9Kze7ECSuWRNUQnOz0=">AAAbknicxdndbuPGFQBgJWmbdNs0m6ZX9Q0Ro0ARbBdWEiBNrmJLtvwjWbJk/dgrZzGkRhTXHJIhR7S9hPoKvW0eLW/TIXk051DDDZqrClhA55szQ2rmcMil7cj3Enlw8PN773/wm9/+7sOPfv/sD3/8+E+fPP/0z5MkXMcOHzuhH8YzmyXc9wI+lp70+SyKORO2z6f2fStvn6Y8TrwwuJZPEb8TzA28pecwqWg8dxahfP18/+DlQfGxzC9N+LLfgM/g9ad/fZgvQmcteCAdnyXJq+ZBJO8yFkvP8fnm2Xyd8Ig598zlr9TXgAme3GXF2W6svylZWMswVv8CaRVKe2RMJMmTsFWmYHKV7LblWNf2ai2X/7zLvCBaSx445YGWa9+SoZX/dGvhxdyR/pP6wpzYU+dqOSsWM0eqCXpWOYxtixdwqBdwOi8iLz9f9ePaXP3omPdU40g1hH42T1ZhLIUXrJNNNs972V6wyQ57I7bJPv/q240afcGX1tx3rfKTzX0WuMVcFQ0xNsTQ8Gwe8AcnFIIFi2yeOnLzqnmXzdXxFkl52P3mZlPNEvKxzCrOYVmTIZPYGKfIdlg54G4+D5Lw13VJuMST2ObsjCrCqMwJIx4zGcZ5jXxRZlowJynzoxWzsuqBCyyOWSbZXDJrNylHkuMyddzdnAJJkq+umgXbSSqRZMlVfrydrAJJ0qNnGaf06JGEYG0mBGuSEK08I0EZzUhqMhKaseC+eaoFkiSu+vhhsJsGTBLjVWiZJ600z4Ek0SkmejepU51p0S5ObDerXT0z0S2mfjeru7Mgoi+4aw5WKMkarGqWZEAnVAySuhQ6o2LkuTU/r1CSNS7nbidrvDujYuDVTOjAI/MpD63tFZgd6o7yCPUItYXaQm2jtlGPUY9RT1BPUDuoHdRT1FPUM9Qz1HPUc9QL1AvULmoXtYfaQ71EvUTto/ZRB6gD1CvUK9Qh6hB1hDpCvUa9Rh2jjlEnqBPUKeoUdYY6Q71BvUG9Rb3F4lGXTn7XyJhOtEFsLQ6Io2UBstDCQbiWJchSiwvialmBrLR4IJ6WNyBvtNyD3JNtGcjHCwdEaAlAAi0hSKglAom0/Ajyo5YYJNaSgCRankCkljXIGk8aJNXyAPKg5RHk0Rj5SctbkLe4qqm5rKm5rqm5sKm5sqm5tKm5tqm5uKm5uqm5vKm5vun/tMCpucKpucSpucapucipucqpucypuc6pudBpzUqbS52aa52ai52aq53WLLfI7wDqqY5s/+IICPd+0QLCjV+0gXDXF8dAuOWLEyDc70UHCDd7cQqEO704A8JtXpwD4R4vLoBwgxddINzdRQ8It3ZxCYT7uugD4aYuBkC4o4srINzOxRAI93IxAsKNXFwD4S4uxkC4hYsJEO7fYgqEm7eYAeHOLW6AcNsWt0Bkz1ZPy/mCqydosuAKjwCPKLYAWxTbgG2Kx4DHFE8ATyh2ADsUTwFPKZ4BnlE8BzyneAF4QbEL2KXYA+xRvAS8pNgH7FMcAA4oXgFeURwCDimOAEcUrwGvKY4BxxQngBOKU8ApxRngjOIN4A3FW0BSIio0S0RFZomoyCwRFZkloiKzRFRkloiKzBJRkVkiKjJLREVmiajILBEVmSWiIrNEVGSWiIrMElGRWSIqMktERWaJqMgsERWZJaIis0RUZJaIiswSUZFZIioyS0RFNSVi28VdI38FYJMise0j5CPCLeQW4TZym/Ax8jHhE+QTwh3kDuFT5FPCZ8hnhM+RzwlfIF8Q7iJ3CfeQe4QvkS8J95H7hAfIA8JXyFeEh8hDwiPkEeFr5GvCY+Qx4QnyhPAUeUp4hjwjfIN8Q/gWuSiryjuvPryx+ULleIEnvLd8U32Pk2nfvLMne6zvufV39mSxK/KXa9V+Sv+V8y92Y4+13Rg+lSXS2n0llaztN9yRc0uGc0snBoF+XxeEwVrYPN42RaH/5IeuMRC4HiLygrQc4of5grkujrBtUC35f+weZfaPJr434E98e2h4y0eewc4W72x7y+Ow2nZAHj2hsXwAJQ1hwC1saOKRsEf+DIM9BPbIG7DH1vH42PZr3ouWDQuPudCQv0zM8lgPJ2Pdq2iUMWliDq9vUge4p73yGCeCO5UhVUw7unTMYR7r1uEvtl6ufZ+Om8e6cRSxgDbmMdaqG1TOSMXY5rmVjnmsG+9xdhwvdny+YInU88plZVQV4xyweNtWLe0Jw1l0wm1ZlwPk8QZXbVkdnbQksrqeCR545QVL+VS+LFsxmc2LGC+JQbhcWvi+efAaBlGskjYf67TAqk0LaNbDKk968BZcHUrfU/+fp7DdlotLa7szi+Lygbbj7bThHXl7h40FucPW3pBqb/87twAo1rp75WE+JdsfdIgzMtp6eZtTHHP6N4D+pnqHpW2HYbQpt75kCb+sGPn18/3m7h+mzC+TL182v3r55dXX+98fwR+tPmrsNT5v/L3RbHzT+L5x2hg0xg2n4TX+3fhP46e9v+x9t3e41ypT338P+nzWqHz2uv8FiajznA==</latexit>

R2<latexit sha1_base64="LcovMJc0WKFjPI89cfXMNlqNG/w=">AAAblHicxdlbb+PGFQBgJb2l26bZJEAf6hciRoEi2C6sTYGmQB5iS7J80c2SdbFXrjGkRhTXHJIhR7S9hPof+tr+sv6bDskjnkMON2ieKmABnW/ODKmZwyGXNgPXieTR0X8++vhnP//FL3/1ya9f/Oa3n/7us5effzGL/G1o8anlu364MFnEXcfjU+lIly+CkDNhunxuPrTS9nnMw8jxvWv5HPA7wWzPWTsWk4rmS4u54/s39y8Pj14fZR9D/9KEL4cN+IzuP//D43LlW1vBPWm5LIreNo8CeZewUDqWy3cvltuIB8x6YDZ/q756TPDoLsnOd2f8UcnKWPuh+udJI1PaI2Eiip6FqTIFk5uo2pZiXdvbrVx/e5c4XrCV3LPyA623riF9I/3xxsoJuSXdZ/WFWaGjztWwNixkllRT9KJ0GNMUr+BQr+B0XgVOer7qx7W5+tEh76vGiWrw3WQZbfxQCsfbRrtkmfYyHW+XHPcnbJd89c3fdmr0FV8bS9c28k+ydJlnZ3OVNYTYEELDi6XHHy1fCOatkmVsyd3b5l2yVMdbRflhD5u7XTlLyKc8KzuHdU2GjEJtnCxb1UKWXs3nXuT/tC4Rl3gS+5zKqMIP8hw/4CGTfpjWyNd5pgFzEjM32DAjKR84w+yYeZLJJTOqSSmSHJup41ZzMiRJrrpuVqySlCPJkpv0eJWsDEnSk2Nop/TkkARvqyd4W5IQbBwtQRnNiGoyIpqx4q5+qhmSJK76uL5XTQMmieHGN/STVprmQJLoZhNdTeqWZ1q0sxOrZrXLZyZ62dRXs3qVBRFDwW19sExJ1mhTsyQjOqFiFNWl0BkVE8eu+XmZkqxpPneVrGl1RsXIqZnQkUPmUx4b+yswOS46yhPUE9QWagu1jdpG7aB2UE9RT1G7qF3UM9Qz1HPUc9QL1AvUS9RL1B5qD7WP2kcdoA5Qh6hD1BHqCPUK9Qp1jDpGnaBOUK9Rr1GnqFPUGeoMdY46R12gLlBvUG9Qb1FvsXjUpZPeNRJWJJogZiEWiFXICmRVCAfhhaxB1oXYIHYhG5BNIQ6IU8g7kHeFPIA8kG0ZyMULB0QU4oF4hfggfiEBSFDIDyA/FBKChIVEIFEhzyCykC3IFk8aJC7kEeSxkCeQJ23k50Leg7zHVY31ZY31dY31hY31lY31pY31tY31xY311Y315Y319Y3/pwWO9RWO9SWO9TWO9UWO9VWO9WWO9XWO9YWOa1ZaX+pYX+tYX+xYX+24ZrlFegdQT3Vk+xcnQLj3ixYQbvyiDYS7vugA4ZYvToFwvxddINzsxRkQ7vTiHAi3eXEBhHu8uATCDV70gHB3F30g3NrFAAj3dTEEwk1djIBwRxdXQLidizEQ7uViAoQbubgGwl1cTIFwCxczINy/xRwIN2+xAMKdW9wA4bYtboHInq2eltMFV0/QZMEVngCeUGwBtii2AdsUO4AdiqeApxS7gF2KZ4BnFM8BzyleAF5QvAS8pNgD7FHsA/YpDgAHFIeAQ4ojwBHFK8ArimPAMcUJ4ITiNeA1xSnglOIMcEZxDjinuABcULwBvKF4C0hKRIV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iaiopkRMM7trpK8ATFIkpnmCfEK4hdwi3EZuE+4gdwifIp8S7iJ3CZ8hnxE+Rz4nfIF8QfgS+ZJwD7lHuI/cJzxAHhAeIg8Jj5BHhK+QrwiPkceEJ8gTwtfI14SnyFPCM+QZ4TnynPACeUH4BvmG8C1yVlald15DeGPztcpxPEc47/mu/B4nKXz3wZ7sqb7n3j/Yk4W2SF+ulfsp/UfKP9qNPdV2Y/hUFkmj+koq2prvuCWXhvSXRpHoecX7Os/3tsLk4b4p8N1n17e1gcCLIQLHi/Mh/r5cMdvGEfYNqiX9j92TTP7cxPcG/JnvDw1v+cgz2Pnqg23veeiX247Ioyc05g+gpMH3uIENTTwS9kifYbCHwB5pA/bYOx4f237Ke9G8YeUwGxrSl4lJGhfDybDolTXKkDQxi9c3qQM80F5pjBPBrdKQKqYdbTrmOI2L1vGPtg62rkvHTeOicRIwjzamMdaq7ZXOSMXY5tiljmlcND7g7FhOaLl8xSJZzCuXpVFVjHPAwn1bubRnDGfR8vdlnQ+QxjtctXV5dNISyfJ6RnjgjeOt5XP+smzDZLLMYrwkRv56beD75tE9DKJYJe0+LdI8ozbNo1mPmzTp0Vlxdajinvr/PIX9tpxdWvudWWSXD7R19tOGd+T9HTYU5A5be0Oqvf1XbgFQrHX3yuN0SvY/6BhnZLL3/DanOOT0bwDDXfkOS9uO/WCXb33RGn5ZNvL9y8Nm9Q9T+pfZm9fNb16/ufrL4fcn8EerTxoHja8af2o0G39tfN84a4wa04bVeGj8s/Gvxr8Pfn/w3UHroJOnfvwR9PmyUfocDP4LL870GQ==</latexit>

R2<latexit sha1_base64="LcovMJc0WKFjPI89cfXMNlqNG/w=">AAAblHicxdlbb+PGFQBgJb2l26bZJEAf6hciRoEi2C6sTYGmQB5iS7J80c2SdbFXrjGkRhTXHJIhR7S9hPof+tr+sv6bDskjnkMON2ieKmABnW/ODKmZwyGXNgPXieTR0X8++vhnP//FL3/1ya9f/Oa3n/7us5effzGL/G1o8anlu364MFnEXcfjU+lIly+CkDNhunxuPrTS9nnMw8jxvWv5HPA7wWzPWTsWk4rmS4u54/s39y8Pj14fZR9D/9KEL4cN+IzuP//D43LlW1vBPWm5LIreNo8CeZewUDqWy3cvltuIB8x6YDZ/q756TPDoLsnOd2f8UcnKWPuh+udJI1PaI2Eiip6FqTIFk5uo2pZiXdvbrVx/e5c4XrCV3LPyA623riF9I/3xxsoJuSXdZ/WFWaGjztWwNixkllRT9KJ0GNMUr+BQr+B0XgVOer7qx7W5+tEh76vGiWrw3WQZbfxQCsfbRrtkmfYyHW+XHPcnbJd89c3fdmr0FV8bS9c28k+ydJlnZ3OVNYTYEELDi6XHHy1fCOatkmVsyd3b5l2yVMdbRflhD5u7XTlLyKc8KzuHdU2GjEJtnCxb1UKWXs3nXuT/tC4Rl3gS+5zKqMIP8hw/4CGTfpjWyNd5pgFzEjM32DAjKR84w+yYeZLJJTOqSSmSHJup41ZzMiRJrrpuVqySlCPJkpv0eJWsDEnSk2Nop/TkkARvqyd4W5IQbBwtQRnNiGoyIpqx4q5+qhmSJK76uL5XTQMmieHGN/STVprmQJLoZhNdTeqWZ1q0sxOrZrXLZyZ62dRXs3qVBRFDwW19sExJ1mhTsyQjOqFiFNWl0BkVE8eu+XmZkqxpPneVrGl1RsXIqZnQkUPmUx4b+yswOS46yhPUE9QWagu1jdpG7aB2UE9RT1G7qF3UM9Qz1HPUc9QL1AvUS9RL1B5qD7WP2kcdoA5Qh6hD1BHqCPUK9Qp1jDpGnaBOUK9Rr1GnqFPUGeoMdY46R12gLlBvUG9Qb1FvsXjUpZPeNRJWJJogZiEWiFXICmRVCAfhhaxB1oXYIHYhG5BNIQ6IU8g7kHeFPIA8kG0ZyMULB0QU4oF4hfggfiEBSFDIDyA/FBKChIVEIFEhzyCykC3IFk8aJC7kEeSxkCeQJ23k50Leg7zHVY31ZY31dY31hY31lY31pY31tY31xY311Y315Y319Y3/pwWO9RWO9SWO9TWO9UWO9VWO9WWO9XWO9YWOa1ZaX+pYX+tYX+xYX+24ZrlFegdQT3Vk+xcnQLj3ixYQbvyiDYS7vugA4ZYvToFwvxddINzsxRkQ7vTiHAi3eXEBhHu8uATCDV70gHB3F30g3NrFAAj3dTEEwk1djIBwRxdXQLidizEQ7uViAoQbubgGwl1cTIFwCxczINy/xRwIN2+xAMKdW9wA4bYtboHInq2eltMFV0/QZMEVngCeUGwBtii2AdsUO4AdiqeApxS7gF2KZ4BnFM8BzyleAF5QvAS8pNgD7FHsA/YpDgAHFIeAQ4ojwBHFK8ArimPAMcUJ4ITiNeA1xSnglOIMcEZxDjinuABcULwBvKF4C0hKRIV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iaiopkRMM7trpK8ATFIkpnmCfEK4hdwi3EZuE+4gdwifIp8S7iJ3CZ8hnxE+Rz4nfIF8QfgS+ZJwD7lHuI/cJzxAHhAeIg8Jj5BHhK+QrwiPkceEJ8gTwtfI14SnyFPCM+QZ4TnynPACeUH4BvmG8C1yVlald15DeGPztcpxPEc47/mu/B4nKXz3wZ7sqb7n3j/Yk4W2SF+ulfsp/UfKP9qNPdV2Y/hUFkmj+koq2prvuCWXhvSXRpHoecX7Os/3tsLk4b4p8N1n17e1gcCLIQLHi/Mh/r5cMdvGEfYNqiX9j92TTP7cxPcG/JnvDw1v+cgz2Pnqg23veeiX247Ioyc05g+gpMH3uIENTTwS9kifYbCHwB5pA/bYOx4f237Ke9G8YeUwGxrSl4lJGhfDybDolTXKkDQxi9c3qQM80F5pjBPBrdKQKqYdbTrmOI2L1vGPtg62rkvHTeOicRIwjzamMdaq7ZXOSMXY5tiljmlcND7g7FhOaLl8xSJZzCuXpVFVjHPAwn1bubRnDGfR8vdlnQ+QxjtctXV5dNISyfJ6RnjgjeOt5XP+smzDZLLMYrwkRv56beD75tE9DKJYJe0+LdI8ozbNo1mPmzTp0Vlxdajinvr/PIX9tpxdWvudWWSXD7R19tOGd+T9HTYU5A5be0Oqvf1XbgFQrHX3yuN0SvY/6BhnZLL3/DanOOT0bwDDXfkOS9uO/WCXb33RGn5ZNvL9y8Nm9Q9T+pfZm9fNb16/ufrL4fcn8EerTxoHja8af2o0G39tfN84a4wa04bVeGj8s/Gvxr8Pfn/w3UHroJOnfvwR9PmyUfocDP4LL870GQ==</latexit>

R3<latexit sha1_base64="61aVkzh+6DfFoVCam2zBvc7Ekac=">AAAblHicxdlbb+PGFQBgJU3bdNs0mxboQ/xCxChQBNuFlS3QFuhDbEmWL7pZsi72yjWG1IjimkMy5Ii2l1D+Q17bX9Z/0yF5xHPI4QbNUwUsoPPNmSE1czjk0mbgOpE8OvrPRx//7JOf/+KXn/7qxa9/89lvP3/5xe9mkb8NLT61fNcPFyaLuOt4fCod6fJFEHImTJfPzYdW2j6PeRg5vnctnwN+J5jtOWvHYlLRfGkxd3z/5v7l4dHro+xj6F+a8OWwAZ/R/RdfPi5XvrUV3JOWy6LobfMokHcJC6VjuXz3YrmNeMCsB2bzt+qrxwSP7pLsfHfGH5WsjLUfqn+eNDKlPRImouhZmCpTMLmJqm0p1rW93cr13+4Sxwu2kntWfqD11jWkb6Q/3lg5Ibek+6y+MCt01Lka1oaFzJJqil6UDmOa4hUc6hWczqvASc9X/bg2Vz865H3VOFENvpsso40fSuF422iXLNNepuPtkuP+hO2Sr978fadGX/G1sXRtI/8kS5d5djZXWUOIDSE0vFh6/NHyhWDeKlnGlty9bd4lS3W8VZQf9rC525WzhHzKs7JzWNdkyCjUxsmyVS1k6dV87kX+T+sScYknsc+pjCr8IM/xAx4y6YdpjXydZxowJzFzgw0zkvKBM8yOmSeZXDKjmpQiybGZOm41J0OS5KrrZsUqSTmSLLlJj1fJypAkPTmGdkpPDknwtnqCtyUJwcbREpTRjKgmI6IZK+7qp5ohSeKqj+t71TRgkhhufEM/aaVpDiSJbjbR1aRueaZFOzuxala7fGail019NatXWRAxFNzWB8uUZI02NUsyohMqRlFdCp1RMXHsmp+XKcma5nNXyZpWZ1SMnJoJHTlkPuWxsb8Ck+OiozxBPUFtobZQ26ht1A5qB/UU9RS1i9pFPUM9Qz1HPUe9QL1AvUS9RO2h9lD7qH3UAeoAdYg6RB2hjlCvUK9Qx6hj1AnqBPUa9Rp1ijpFnaHOUOeoc9QF6gL1BvUG9Rb1FotHXTrpXSNhRaIJYhZigViFrEBWhXAQXsgaZF2IDWIXsgHZFOKAOIW8A3lXyAPIA9mWgVy8cEBEIR6IV4gP4hcSgASFfAfyXSEhSFhIBBIV8gwiC9mCbPGkQeJCHkEeC3kCedJGfi7kPch7XNVYX9ZYX9dYX9hYX9lYX9pYX9tYX9xYX91YX95YX9/4f1rgWF/hWF/iWF/jWF/kWF/lWF/mWF/nWF/ouGal9aWO9bWO9cWO9dWOa5ZbpHcA9VRHtn9xAoR7v2gB4cYv2kC464sOEG754hQI93vRBcLNXpwB4U4vzoFwmxcXQLjHi0sg3OBFDwh3d9EHwq1dDIBwXxdDINzUxQgId3RxBYTbuRgD4V4uJkC4kYtrINzFxRQIt3AxA8L9W8yBcPMWCyDcucUNEG7b4haI7NnqaTldcPUETRZc4QngCcUWYItiG7BNsQPYoXgKeEqxC9ileAZ4RvEc8JziBeAFxUvAS4o9wB7FPmCf4gBwQHEIOKQ4AhxRvAK8ojgGHFOcAE4oXgNeU5wCTinOAGcU54BzigvABcUbwBuKt4CkRFSol4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iKakrENLO7RvoKwCRFYponyCeEW8gtwm3kNuEOcofwKfIp4S5yl/AZ8hnhc+RzwhfIF4QvkS8J95B7hPvIfcID5AHhIfKQ8Ah5RPgK+YrwGHlMeII8IXyNfE14ijwlPEOeEZ4jzwkvkBeEb5BvCN8iZ2VVeuc1hDc2X6scx3OE857vyu9xksJ3H+zJnup77v2DPVloi/TlWrmf0u9T/tFu7Km2G8Onskga1VdS0dZ8xy25NKS/NIpEzyve13m+txUmD/dNge8+u76tDQReDBE4XpwP8c/litk2jrBvUC3pf+yeZPLnJr434M98f2h4y0eewc5XH2x7z0O/3HZEHj2hMX8AJQ2+xw1saOKRsEf6DIM9BPZIG7DH3vH42PZT3ovmDSuH2dCQvkxM0rgYToZFr6xRhqSJWby+SR3ggfZKY5wIbpWGVDHtaNMxx2lctI5/tHWwdV06bhoXjZOAebQxjbFWba90RirGNscudUzjovEBZ8dyQsvlKxbJYl65LI2qYpwDFu7byqU9YziLlr8v63yANN7hqq3Lo5OWSJbXM8IDbxxvLZ/zl2UbJpNlFuMlMfLXawPfN4/uYRDFKmn3WZHmGbVpHs163KRJj86Kq0MV99T/5ynst+Xs0trvzCK7fKCts582vCPv77ChIHfY2htS7e2/cguAYq27Vx6nU7L/Qcc4I5O957c5xSGnfwMY7sp3WNp27Ae7fOuL1vDLspHvXx42q3+Y0r/MvnndfPP6m6u/HH57An+0+rRx0Piq8adGs/HXxreNs8aoMW1YjYfGD41/Nf598IeDfxy0Djp56scfQZ/fN0qfg8F/AUr49Bo=</latexit>

R3<latexit sha1_base64="61aVkzh+6DfFoVCam2zBvc7Ekac=">AAAblHicxdlbb+PGFQBgJU3bdNs0mxboQ/xCxChQBNuFlS3QFuhDbEmWL7pZsi72yjWG1IjimkMy5Ii2l1D+Q17bX9Z/0yF5xHPI4QbNUwUsoPPNmSE1czjk0mbgOpE8OvrPRx//7JOf/+KXn/7qxa9/89lvP3/5xe9mkb8NLT61fNcPFyaLuOt4fCod6fJFEHImTJfPzYdW2j6PeRg5vnctnwN+J5jtOWvHYlLRfGkxd3z/5v7l4dHro+xj6F+a8OWwAZ/R/RdfPi5XvrUV3JOWy6LobfMokHcJC6VjuXz3YrmNeMCsB2bzt+qrxwSP7pLsfHfGH5WsjLUfqn+eNDKlPRImouhZmCpTMLmJqm0p1rW93cr13+4Sxwu2kntWfqD11jWkb6Q/3lg5Ibek+6y+MCt01Lka1oaFzJJqil6UDmOa4hUc6hWczqvASc9X/bg2Vz865H3VOFENvpsso40fSuF422iXLNNepuPtkuP+hO2Sr978fadGX/G1sXRtI/8kS5d5djZXWUOIDSE0vFh6/NHyhWDeKlnGlty9bd4lS3W8VZQf9rC525WzhHzKs7JzWNdkyCjUxsmyVS1k6dV87kX+T+sScYknsc+pjCr8IM/xAx4y6YdpjXydZxowJzFzgw0zkvKBM8yOmSeZXDKjmpQiybGZOm41J0OS5KrrZsUqSTmSLLlJj1fJypAkPTmGdkpPDknwtnqCtyUJwcbREpTRjKgmI6IZK+7qp5ohSeKqj+t71TRgkhhufEM/aaVpDiSJbjbR1aRueaZFOzuxala7fGail019NatXWRAxFNzWB8uUZI02NUsyohMqRlFdCp1RMXHsmp+XKcma5nNXyZpWZ1SMnJoJHTlkPuWxsb8Ck+OiozxBPUFtobZQ26ht1A5qB/UU9RS1i9pFPUM9Qz1HPUe9QL1AvUS9RO2h9lD7qH3UAeoAdYg6RB2hjlCvUK9Qx6hj1AnqBPUa9Rp1ijpFnaHOUOeoc9QF6gL1BvUG9Rb1FotHXTrpXSNhRaIJYhZigViFrEBWhXAQXsgaZF2IDWIXsgHZFOKAOIW8A3lXyAPIA9mWgVy8cEBEIR6IV4gP4hcSgASFfAfyXSEhSFhIBBIV8gwiC9mCbPGkQeJCHkEeC3kCedJGfi7kPch7XNVYX9ZYX9dYX9hYX9lYX9pYX9tYX9xYX91YX95YX9/4f1rgWF/hWF/iWF/jWF/kWF/lWF/mWF/nWF/ouGal9aWO9bWO9cWO9dWOa5ZbpHcA9VRHtn9xAoR7v2gB4cYv2kC464sOEG754hQI93vRBcLNXpwB4U4vzoFwmxcXQLjHi0sg3OBFDwh3d9EHwq1dDIBwXxdDINzUxQgId3RxBYTbuRgD4V4uJkC4kYtrINzFxRQIt3AxA8L9W8yBcPMWCyDcucUNEG7b4haI7NnqaTldcPUETRZc4QngCcUWYItiG7BNsQPYoXgKeEqxC9ileAZ4RvEc8JziBeAFxUvAS4o9wB7FPmCf4gBwQHEIOKQ4AhxRvAK8ojgGHFOcAE4oXgNeU5wCTinOAGcU54BzigvABcUbwBuKt4CkRFSol4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iKakrENLO7RvoKwCRFYponyCeEW8gtwm3kNuEOcofwKfIp4S5yl/AZ8hnhc+RzwhfIF4QvkS8J95B7hPvIfcID5AHhIfKQ8Ah5RPgK+YrwGHlMeII8IXyNfE14ijwlPEOeEZ4jzwkvkBeEb5BvCN8iZ2VVeuc1hDc2X6scx3OE857vyu9xksJ3H+zJnup77v2DPVloi/TlWrmf0u9T/tFu7Km2G8Onskga1VdS0dZ8xy25NKS/NIpEzyve13m+txUmD/dNge8+u76tDQReDBE4XpwP8c/litk2jrBvUC3pf+yeZPLnJr434M98f2h4y0eewc5XH2x7z0O/3HZEHj2hMX8AJQ2+xw1saOKRsEf6DIM9BPZIG7DH3vH42PZT3ovmDSuH2dCQvkxM0rgYToZFr6xRhqSJWby+SR3ggfZKY5wIbpWGVDHtaNMxx2lctI5/tHWwdV06bhoXjZOAebQxjbFWba90RirGNscudUzjovEBZ8dyQsvlKxbJYl65LI2qYpwDFu7byqU9YziLlr8v63yANN7hqq3Lo5OWSJbXM8IDbxxvLZ/zl2UbJpNlFuMlMfLXawPfN4/uYRDFKmn3WZHmGbVpHs163KRJj86Kq0MV99T/5ynst+Xs0trvzCK7fKCts582vCPv77ChIHfY2htS7e2/cguAYq27Vx6nU7L/Qcc4I5O957c5xSGnfwMY7sp3WNp27Ae7fOuL1vDLspHvXx42q3+Y0r/MvnndfPP6m6u/HH57An+0+rRx0Piq8adGs/HXxreNs8aoMW1YjYfGD41/Nf598IeDfxy0Djp56scfQZ/fN0qfg8F/AUr49Bo=</latexit>

0x?

<latexit sha1_base64="aITZbuqsTwVSCk+BuOOP18SpMN4=">AAAbo3icxdndbuPGFQBgJWnadJu0m7RX9Q0RI0ARbBZWEiDJXWzLln8kS9a/vXKMITWiuOaQDDmiZRPqs/S2faO+TYfkEc8hhxs0VxEQROebM0Nq5nDIpc3AdSJ5cPDf997/4Hcf/v4PH/3xxZ8+/uTPf3n56WeTyF+HFh9bvuuHM5NF3HU8PpaOdPksCDkTpsun5sNx2j6NeRg5vjeSTwG/E8z2nKVjMano/uXf5qbvLqInof6XbLY/zSPJwvuX+wevD7KPoX9pwpf9Bnz695/+/XG+8K214J60XBZFb5oHgbxLWCgdy+XbF/N1xANmPTCbv1FfPSZ4dJdkp781vlCyMJZ+qP7zpJEp7ZEwEaVnqDIFk6uo2pZiXdubtVx+f5c4XrCW3LPyAy3XriF9I50LY+GE3JLuk/rCrNBR52pYKxYyS6oZe1E6jGmKV3CoV3A6rwInPV/141pc/eiQd1XjMJ/JebTyQykcbx1tk3nay3S8bXLYHbJt8vk3P2zV6Au+NOaubeSfZO4yz87mKmsIsSGEhhdzjz9avhDMWyTz2JLbN827hC7gfnO7LWcJucmzsnNY1mTIKNTGybItlg9Yzede5P+6LhGXeBK7nMqowg/yHD/gIZN+mNbIl3mmAXMSMzdYMSMpHzjD7Jh5ksklM6pJKZIcm6njVnMyJEmuuowWrJKUI8mSq/R4lawMSdLGMbRT2jgkwVvrCd6aJAQrR0tQRjOimoyIZiy4q59qhiSJqz6u71XTgEliuPIN/aSVpjmQJNrZRFeT2uWZFq3sxKpZrfKZiU429dWsTmVBRE9wWx8sU5LVX9UsSZ9OqOhHdSl0RsXQsWt+XqYka5zPXSVrXJ1R0XdqJrTvkPmUh8buCkwOi47yCPUI9Rj1GLWF2kI9QT1BPUU9RW2jtlHPUM9Qz1HPUS9QL1AvUS9RO6gd1C5qF/UK9Qq1h9pD7aP2Ua9Rr1EHqAPUIeoQdYQ6Qh2jjlEnqBPUKeoUdYY6Q71BvUG9Rb3F4lGXTnrXSFiRaIKYhVggViELkEUhHIQXsgRZFmKD2IWsQFaFOCBOIW9B3hbyAPJAtmUgFy8cEFGIB+IV4oP4hQQgQSE/g/xcSAgSFhKBRIU8gchC1iBrPGmQuJBHkMdCNiAbbeSnQp5BnnFVY31ZY31dY31hY31lY31pY31tY31xY311Y315Y3194/9rgWN9hWN9iWN9jWN9kWN9lWN9mWN9nWN9oeOaldaXOtbXOtYXO9ZXO65ZbpHeAdRTHdn+xREQ7v3iGAg3ftECwl1fnADhli9OgXC/F20g3OzFGRDu9OIcCLd5cQGEe7y4BMINXnSAcHcXXSDc2sUVEO7rogeEm7roA+GOLq6BcDsXAyDcy8UQCDdyMQLCXVyMgXALFxMg3L/FFAg3bzEDwp1b3ADhti1ugcierZ6W0wVXT9BkwRUeAR5RPAY8ptgCbFE8ATyheAp4SrEN2KZ4BnhG8RzwnOIF4AXFS8BLih3ADsUuYJfiFeAVxR5gj2IfsE/xGvCa4gBwQHEIOKQ4AhxRHAOOKU4AJxSngFOKM8AZxRvAG4q3gKREVKiXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIpqSsQ0s7tG+grAJEVimkfIR4SPkY8Jt5BbhE+QTwifIp8SbiO3CZ8hnxE+Rz4nfIF8QfgS+ZJwB7lDuIvcJXyFfEW4h9wj3EfuE75GviY8QB4QHiIPCY+QR4THyGPCE+QJ4SnylPAMeUb4BvmG8C1yVlald149eGPzpcpxPEc4z3xbfo+TFL59Z0+2qe+583f2ZKEt0pdr5X5K/5nyL3Zjm9puDJ/KImlUX0lFa/Mtt+TckP7cKBI9r3hf5/neWpg83DUFvvvk+rY2EHgxROB4cT7ET/MFs20cYdegWtJ/2G1k8lUT3xvwJ747NLzlI89g54t3tj3z0C+3HZBHT2jMH0BJg+9xAxuaeCTskT7DYA+BPdIG7LFzPD62/Zr3onnDwmE2NKQvE5M0LoaTYdEra5QhaWIWr29SB3igvdIYJ4JbpSFVTDvadMxBGhetg19svVq7Lh03jYvGYcA82pjGWKu2VzojFWObY5c6pnHR+ICzYzmh5fIFi2Qxr1yWRlUxzgELd23l0p4wnEXL35V1PkAab3HVluXRSUsky+sZ4YFXjreUT/nLshWTyTyL8ZLo+8ulge+b+/cwiGKVtP2kSPOM2jSPZj2u0qRHZ8HVoYp76m95CrttObu0djuzyC4faDvZTRvekXd32FCQO2ztDan29l+5BUCx1t0rD9Mp2f2gQ5yR4c7z25zikNO/AfS25TssbTv0g22+9UVL+GXZyPcv95vVP0zpXyZfv25+8/rr62/3fzyCP1p91NhrfN74R6PZ+K7xY+Os0W+MG1bjufGvxr8b/9n7Yu9yb7A3ylPffw/6/LVR+uzd/Q8YcfrU</latexit>-x?<latexit sha1_base64="xaI6QImBW7Fk38zkKLtw2K5bOHA=">AAAbq3icxdlbb+PGFQBg5dI23TbNJn2r80DEKFAEm4WVFGj7Ftuy5YtkyZJ1sVeOMaRGFNcckiFHtLwE+9Bf09f23/TfdEge8RxyuEHzVAGL1fnmzJCaORwStBm4TiQPDv7zwYcfffyLX/7qk1+/+M1vP/3dZy8//2Ia+ZvQ4hPLd/1wbrKIu47HJ9KRLp8HIWfCdPnMfDzO2mcxDyPH927kc8DvBbM9Z+VYTCp6ePnlQvKtTL5JF6bvLqNnof5LtukPi0iy8OHl/sHrg/xj6F/a8GW/BZ/hw+d/eFosfWsjuCctl0XRm/ZBIO8TFkrHcnn6YrGJeMCsR2bzN+qrxwSP7pP8Z6TGH5UsjZUfqn+eNHKlPRImouwMVaZgch3V2zJsanuzkau/3ieOF2wk96ziQKuNa0jfyObEWDoht6T7rL4wK3TUuRrWmoXMkmrmXlQOY5riFRzqFZzOq8DJzlf9uA5XPzrkfdU4LmZyEa39UArH20Rpssh6mY6XJof9MUuTr777W6pGX/KVsXBto/gkC5d5dj5XeUOIDSE0vFh4/MnyhWDeMlnElkzftO8TuoD77TStZgm5LbLyc1g1ZMgo1MbJsy1WDFjP517k/7wuEZd4Eruc2qjCD4ocP+Ahk36Y1cjXRaYBcxIzN1gzI6keOMf8mEWSySUz6kkZkhybqePWc3IkSa66nJasllQgyZLr7Hi1rBxJ0tYxtFPaOiTB2+gJ3oYkBGtHS1BGM6KGjIhmLLmrn2qOJImrPq7v1dOASWK49g39pJVmOZAkuvlE15O61ZkWnfzE6lmd6pmJXj719axebUHEQHBbHyxXkjVcNyzJkE6oGEZNKXRGxdixG35eriRrUsxdLWtSn1ExdBomdOiQ+ZSHxu4KTA7LjvII9Qj1GPUYtYPaQT1BPUE9RT1F7aJ2Uc9Qz1DPUc9RL1AvUC9RL1F7qD3UPmof9Qr1CnWAOkAdog5Rr1GvUUeoI9Qx6hj1BvUGdYI6QZ2iTlFnqDPUOeoc9Rb1FvUO9Q6LR1062V0jYWWiCWKWYoFYpSxBlqVwEF7KCmRVig1il7IGWZfigDilvAV5W8ojyCPZloFcvHBARCkeiFeKD+KXEoAEpfwI8mMpIUhYSgQSlfIMIkvZgGzwpEHiUp5AnkrZgmy1kZ9LeQfyDlc11pc11tc11hc21lc21pc21tc21hc31lc31pc31tc3/p8WONZXONaXONbXONYXOdZXOdaXOdbXOdYXOm5YaX2pY32tY32xY32144blFtkdQD3Vke1fHAHh3i+OgXDjFx0g3PXFCRBu+eIUCPd70QXCzV6cAeFOL86BcJsXF0C4x4tLINzgRQ8Id3fRB8KtXVwB4b4uBkC4qYshEO7o4hoIt3MxAsK9XIyBcCMXN0C4i4sJEG7hYgqE+7eYAeHmLeZAuHOLWyDctsUdENmz1dNytuDqCZosuMIjwCOKx4DHFDuAHYongCcUTwFPKXYBuxTPAM8ongOeU7wAvKB4CXhJsQfYo9gH7FO8AryiOAAcUBwCDileA15THAGOKI4BxxRvAG8oTgAnFKeAU4ozwBnFOeCc4i3gLcU7QFIiKtRLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREUNJWKa+V0jewVgkiIxzSPkI8LHyMeEO8gdwifIJ4RPkU8Jd5G7hM+QzwifI58TvkC+IHyJfEm4h9wj3EfuE75CviI8QB4QHiIPCV8jXxMeIY8Ij5HHhG+QbwhPkCeEp8hTwjPkGeE58pzwLfIt4TvkvKwq77wG8Mbma5XjeI5w3vG0+h4nKT19b0+2be658/f2ZKEtspdr1X5K/57xT3Zj28ZuDJ/KImnUX0lFG/Mtt+TCkP7CKBM9r3xf5/neRpg83DUFvvvs+rY2EHg5ROB4cTHED4sls20cYdegWhJ4cdvG9wb8me8ODW/5yDPY+fK9be946FfbDsijJzQWD6Ckwfe4gQ1tPBL2yJ5hsIfAHlkD9tg5Hh/bfs570aJh6TAbGrKXiUkWl8PJsOyVN8qQNDGLNzepAzzSXlmME8GtypAqph1tOuYoi8vW0U+2Xm1cl46bxWXjOGAebcxirFXbq5yRirHNsSsds7hsfMTZsZzQcvmSRbKcVy4ro6oY54CFu7ZqaU8ZzqLl78q6GCCLU1y1VXV00hLJ6npGeOC1463kc/GybM1ksshjvCSG/mpl4Pvm4QMMolglpZ+WaZ7RmObRrKd1lvTkLLk6VHlP/X+ewm5bzi+t3c4s8ssH2k5204Z35N0dNhTkDtt4Q2q8/dduAVCsTffKw2xKdj/oEGdkvPPiNqc45PRvAIO0eoelbYd+kBZbX7SCX5aP/PByv13/w5T+Zfrt6/Z3r7+9/vP+90fwR6tPWnutr1p/arVbf2l93zprDVuTltX6R+ufrX+1/r33zd54725vUaR++AH0+X2r8tnj/wVM/v5q</latexit>

1p3kx?k2w

<latexit sha1_base64="0NJkfG3R44o0QdhLkbV6gvYHRag=">AAAbx3icxdndbuPGFQBgJelPum2aTXpX3xAxChTBdmHtFmh6F9uyZdmSJUvWj710jCE1krjmkFxyRMvLMEBepE/T2/a+b9MhecRzyNEGzVUFBNH55syQmjkccmkrcJ1IHhz856OPP/nFL3/1609/8+y3v/vs958//+LLSeSvQ5uPbd/1w5nFIu46Hh9LR7p8FoScCcvlU+vhOGufxjyMHN+7lk8BvxNs6TkLx2ZS0f3zb8xFyOykmSZm9C6Ur1Pze9Py3Xn0JNT/kk36nRlJFprf37+i/pjeP98/eHmQfwz9SxO+7DfgM7j/4o+P5ty314J70nZZFL1pHgTyLmGhdGyXp8/MdcQDZj+wJX+jvnpM8OguyX9iavxJydxY+KH6z5NGrrRHwkSUnZvKFEyuonpbhrva3qzl4pu7xPGCteSeXRxosXYN6RvZfBlzJ+S2dJ/UF2aHjjpXw14xNWdSzeqzymEsS7yAQ72A03kRONn5qh/X4upHh7ynGkfFHJrRyg+lcLx1pGY/62U5Xpoc9kYsTb56/fdUjT7nC8N0l0bxSUyXect8rvKGEBtCaHhmevzR9oVg3jwxY1umb5p3CV26/WaaVrOE3BRZ+TksdmTIKNTGybNtVgxYz+de5P+8LhGXeBLbnNqowg+KHD/gIZN+mNXI10WmAXMSMzdYMSOpHjjH/JhFksUlM+pJGZKcJVPHrefkSJJcdanNWS2pQJIlV9nxalk5kqSNY2intHFIgrfWE7w1SQhWjpagjGZEOzIimjHnrn6qOZIkrvq4vldPAyaJ4co39JNWmuVAkmjnE11PaldnWrTyE6tntapnJrr51NezurUFEX3Bl/pguZKswWrHkgzohIpBtCuFzqgYOcsdPy9XkjUu5q6WNa7PqBg4OyZ04JD5lIfG9gpMDsuO8gj1CPUY9Ri1hdpCPUE9QT1FPUVto7ZRz1DPUDuoHdRz1HPUC9QL1C5qF7WH2kO9RL1E7aP2UQeoA9Qr1CvUIeoQdYQ6Qr1GvUYdo45RJ6gT1CnqFHWGOkO9Qb1BvUW9xeJRl05210hYmWiBWKXYIHYpc5B5KRyEl7IAWZSyBFmWsgJZleKAOKW8BXlbygPIA9mWgVy8cEBEKR6IV4oP4pcSgASlvAN5V0oIEpYSgUSlPIHIUtYgazxpkLiUR5DHUjYgG23kp1Leg7zHVY31ZY31dY31hY31lY31pY31tY31xY311Y315Y319Y3/pwWO9RWO9SWO9TWO9UWO9VWO9WWO9XWO9YWOd6y0vtSxvtaxvtixvtrxjuUW2R1APdWR7V8cAeHeL46BcOMXLSDc9cUJEG754hQI93vRBsLNXpwB4U4vOkC4zYtzINzjxQUQbvCiC4S7u+gB4dYuLoFwXxd9INzUxQAId3RxBYTbuRgC4V4uRkC4kYtrINzFxRgIt3AxAcL9W0yBcPMWMyDcucUNEG7b4haI7NnqaTlbcPUETRZc4RHgEcVjwGOKLcAWxRPAE4qngKcU24BtimeAZxQ7gB2K54DnFC8ALyh2AbsUe4A9ipeAlxT7gH2KA8ABxSvAK4pDwCHFEeCI4jXgNcUx4JjiBHBCcQo4pTgDnFG8AbyheAtISkSFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoaEeJWFZ+18heAVikSCzrCPmI8DHyMeEWcovwCfIJ4VPkU8Jt5DbhM+Qzwh3kDuFz5HPCF8gXhLvIXcI95B7hS+RLwn3kPuEB8oDwFfIV4SHykPAIeUT4Gvma8Bh5THiCPCE8RZ4SniHPCN8g3xC+Rc7LqvLOqw9vbL5WOY7nCOc9T6vvcZLS0w/2ZJvdPbf+wZ4sXIrs5Vq1n9IfMv7JbmyzsxvDp7JIGvVXUtHaesttaRrSN40y0fPK93We762FxcNtU+C7T66/1AYCL4cIHC8uhvjOnLPlEkfYNqiW7B92G5n8pYnvDfgT3x4a3vKRZ7DO/INt73noV9sOyKMnNBYPoKTB97iBDU08EvbInmGwh8AeWQP22DoeH9t+znvRomHusCU0ZC8Tkywuh5Nh2StvlCFpYjbf3aQO8EB7ZTFOBLcrQ6qYdlzSMYdZXLYOf7L1cu26dNwsLhtHAfNoYxZjrS69yhmpGNucZaVjFpeNDzg7thPaLp+zSJbzymVlVBXjHLBw21Yt7QnDWbT9bVkXA2Rxiqu2qI5OWiJZXc8ID7xyvIV8Kl6WrZhMzDzGS2LgLxYGvm8e3MMgilVS+lmZ5hk70zya9bjKkh6dOVeHKu+p/89T2G7L+aW13ZlFfvlA28l22vCOvL3DhoLcYXfekHbe/mu3ACjWXffKw2xKtj/oEGdktPXiNqc45PRvAP20eoelbYd+kBZbX7SAX5aPfP98v1n/w5T+ZfLqZfP1y1dXf93/9gj+aPVpY6/xVePPjWbjb41vG2eNQWPcsBv/aPyz8a/Gv/c6e/5evLcpUj/+CPr8oVH57P34X7F8CwA=</latexit>

-1p3kx?k2w

<latexit sha1_base64="kRfkrwFMjKNUM0IfSqNcgHF/ZbA=">AAAbz3icxdlbb9vIFQBg7fa2Tbtttn2rX4g1ChSLNLCSAt19i23Z8kWybtbFDr3GkBpJjDkkQ45oOVwu+to/0l/T1xbov+mQPOI51DCL7lMFBNH55syQmjkcErQVuE4kDw7+88mnP/npz37+i89++exXv/78N799/sXvJpG/Dm0+tn3XD2cWi7jreHwsHenyWRByJiyXT62H46x9GvMwcnzvWj4F/E6wpecsHJtJRffP35iLkNmJKflGJn9Om2liRu9D+To1vzMt351HT0L9l2zSb81IstD87v4V9cf0/vn+wcuD/GPoX5rwZb8Bn/79F394NOe+vRbck7bLouht8yCQdwkLpWO7PH1mriMeMPuBLflb9dVjgkd3Sf5TU+OPSubGwg/VP08audIeCRNRdm4qUzC5inbbMqxre7uWi6/vEscL1pJ7dnGgxdo1pG9k82bMnZDb0n1SX5gdOupcDXvF1NxJNbvPKoexLPECDvUCTudF4GTnq35ci6sfHfKuahwVc2hGKz+UwvHWkZr9rJfleGly2B2xNPny9TepGn3OF4bpLo3ik5gu85b5XOUNITaE0PDM9Pij7QvBvHlixrZM3zbvErp0+800rWYJuSmy8nNY1GTIKNTGybNtVgy4m8+9yP9xXSIu8SS2OTujCj8ocvyAh0z6YVYjXxWZBsxJzNxgxYykeuAc82MWSRaXzNhNypDkLJk67m5OjiTJVZfcnO0kFUiy5Co73k5WjiRp4xjaKW0ckuCt9QRvTRKClaMlKKMZUU1GRDPm3NVPNUeSxFUf1/d204BJYrjyDf2klWY5kCTa+UTvJrWrMy1a+YntZrWqZyY6+dTvZnV2FkT0BF/qg+VKsvqrmiXp0wkV/aguhc6oGDnLmp+XK8kaF3O3kzXenVHRd2omtO+Q+ZSHxvYKTA7LjvII9Qj1GPUYtYXaQj1BPUE9RT1FbaO2Uc9Qz1DPUc9RL1AvUC9RL1E7qB3ULmoX9Qr1CrWH2kPto/ZRB6gD1CHqEHWEOkK9Rr1GHaOOUSeoE9Qp6hR1hjpDvUG9Qb1FvcXiUZdOdtdIWJlogVil2CB2KXOQeSkchJeyAFmUsgRZlrICWZXigDilvAN5V8oDyAPZloFcvHBARCkeiFeKD+KXEoAEpbwHeV9KCBKWEoFEpTyByFLWIGs8aZC4lEeQx1I2IBtt5KdSPoB8wFWN9WWN9XWN9YWN9ZWN9aWN9bWN9cWN9dWN9eWN9fWN/6cFjvUVjvUljvU1jvVFjvVVjvVljvV1jvWFjmtWWl/qWF/rWF/sWF/tuGa5RXYHUE91ZPsXR0C494tjINz4RQsId31xAoRbvjgFwv1etIFwsxdnQLjTi3Mg3ObFBRDu8eISCDd40QHC3V10gXBrF1dAuK+LHhBu6qIPhDu6GADhdi6GQLiXixEQbuTiGgh3cTEGwi1cTIBw/xZTINy8xQwId25xA4TbtrgFInu2elrOFlw9QZMFV3gEeETxGPCYYguwRfEE8ITiKeApxTZgm+IZ4BnFc8BziheAFxQvAS8pdgA7FLuAXYpXgFcUe4A9in3APsUB4IDiEHBIcQQ4ongNeE1xDDimOAGcUJwCTinOAGcUbwBvKN4CkhJRoV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iK9BJRkV4iKqopEcvK7xrZKwCLFIllHSEfET5GPibcQm4RPkE+IXyKfEq4jdwmfIZ8Rvgc+ZzwBfIF4UvkS8Id5A7hLnKX8BXyFeEeco9wH7lPeIA8IDxEHhIeIY8IXyNfEx4jjwlPkCeEp8hTwjPkGeEb5BvCt8h5WVXeefXgjc1XKsfxHOF84Gn1PU5SevrRnmxT33PrH+3JwqXIXq5V+yn9PuMf7MY2td0YPpVF0th9JRWtrXfclqYhfdMoEz2vfF/n+d5aWDzcNgW+++T6S20g8HKIwPHiYohvzTlbLnGEbYNq2b7WbeJ7A/7Et4eGt3zkGex8/tG2Dzz0q20H5NETGosHUNLge9zAhiYeCXtkzzDYQ2CPrAF7bB2Pj20/5r1o0TB32BIaspeJSRaXw8mw7JU3ypA0MZvXN6kDPNBeWYwTwe3KkCqmHZd0zGEWl63DH2y9WrsuHTeLy8ZRwDzamMVYq0uvckYqxjZnWemYxWXjA86O7YS2y+cskuW8clkZVcU4ByzctlVLe8JwFm1/W9bFAFmc4qotqqOTlkhW1zPCA68cbyGfipdlKyYTM4/xkuj7i4WB75v79zCIYpWUfl6meUZtmkezHldZ0qMz5+pQ5T31/3kK2205v7S2O7PILx9oO9lOG96Rt3fYUJA7bO0Nqfb2v3MLgGKtu1ceZlOy/UGHOCOjrRe3OcUhp38D6KXVOyxtO/SDtNj6ogX8snzk++f7zd0/TOlfJq9eNl+/fDX4y/6bI/ij1WeNvcaXjT81mo2/Nt40zhr9xrhhN/7R+GfjX41/7w32Hve+3/tbkfrpJ9Dn943KZ+/v/wVeJA6W</latexit>

R4<latexit sha1_base64="6o5qQI9pAQSKwvYR6td9ZLo2o1c=">AAAblHicxdlbb+PGFQBgJU3bdNu0mwToQ/xCxChQBNuFlQRoCuQhtiTLF90sWRd75RpDakRxzSEZckTbS6j/oa/tL+u/6ZA84jnkcIPmqQIW0PnmzJCaORxyaTNwnUgeHf3ngw9/8dEvf/Xrj3/z4re/++T3f3j56WezyN+GFp9avuuHC5NF3HU8PpWOdPkiCDkTpsvn5kMrbZ/HPIwc37uWzwG/E8z2nLVjMalovrSYO77/9v7l4dHro+xj6F+a8OWwAZ/R/adfPC5XvrUV3JOWy6LoTfMokHcJC6VjuXz3YrmNeMCsB2bzN+qrxwSP7pLsfHfGn5SsjLUfqn+eNDKlPRImouhZmCpTMLmJqm0p1rW92cr1d3eJ4wVbyT0rP9B66xrSN9Ifb6yckFvSfVZfmBU66lwNa8NCZkk1RS9KhzFN8QoO9QpO51XgpOerflybqx8d8r5qnKgG302W0cYPpXC8bbRLlmkv0/F2yXF/wnbJl9/8badGX/G1sXRtI/8kS5d5djZXWUOIDSE0vFh6/NHyhWDeKlnGlty9ad4lS3W8VZQf9rC525WzhHzKs7JzWNdkyCjUxsmyVS1k6dV87kX+z+sScYknsc+pjCr8IM/xAx4y6YdpjXyVZxowJzFzgw0zkvKBM8yOmSeZXDKjmpQiybGZOm41J0OS5KrrZsUqSTmSLLlJj1fJypAkPTmGdkpPDknwtnqCtyUJwcbREpTRjKgmI6IZK+7qp5ohSeKqj+t71TRgkhhufEM/aaVpDiSJbjbR1aRueaZFOzuxala7fGail019NatXWRAxFNzWB8uUZI02NUsyohMqRlFdCp1RMXHsmp+XKcma5nNXyZpWZ1SMnJoJHTlkPuWxsb8Ck+OiozxBPUFtobZQ26ht1A5qB/UU9RS1i9pFPUM9Qz1HPUe9QL1AvUS9RO2h9lD7qH3UAeoAdYg6RB2hjlCvUK9Qx6hj1AnqBPUa9Rp1ijpFnaHOUOeoc9QF6gL1BvUG9Rb1FotHXTrpXSNhRaIJYhZigViFrEBWhXAQXsgaZF2IDWIXsgHZFOKAOIW8BXlbyAPIA9mWgVy8cEBEIR6IV4gP4hcSgASF/AjyYyEhSFhIBBIV8gwiC9mCbPGkQeJCHkEeC3kCedJGfi7kHcg7XNVYX9ZYX9dYX9hYX9lYX9pYX9tYX9xYX91YX95YX9/4f1rgWF/hWF/iWF/jWF/kWF/lWF/mWF/nWF/ouGal9aWO9bWO9cWO9dWOa5ZbpHcA9VRHtn9xAoR7v2gB4cYv2kC464sOEG754hQI93vRBcLNXpwB4U4vzoFwmxcXQLjHi0sg3OBFDwh3d9EHwq1dDIBwXxdDINzUxQgId3RxBYTbuRgD4V4uJkC4kYtrINzFxRQIt3AxA8L9W8yBcPMWCyDcucUNEG7b4haI7NnqaTldcPUETRZc4QngCcUWYItiG7BNsQPYoXgKeEqxC9ileAZ4RvEc8JziBeAFxUvAS4o9wB7FPmCf4gBwQHEIOKQ4AhxRvAK8ojgGHFOcAE4oXgNeU5wCTinOAGcU54BzigvABcUbwBuKt4CkRFSol4iK9BJRkV4iKtJLREV6iahILxEV6SWiIr1EVKSXiIr0ElGRXiIq0ktERXqJqEgvERXpJaIivURUpJeIivQSUZFeIirSS0RFeomoSC8RFekloiK9RFSkl4iKakrENLO7RvoKwCRFYponyCeEW8gtwm3kNuEOcofwKfIp4S5yl/AZ8hnhc+RzwhfIF4QvkS8J95B7hPvIfcID5AHhIfKQ8Ah5RPgK+YrwGHlMeII8IXyNfE14ijwlPEOeEZ4jzwkvkBeEb5BvCN8iZ2VVeuc1hDc2X6kcx3OE847vyu9xksJ37+3Jnup77v29PVloi/TlWrmf0n+k/JPd2FNtN4ZPZZE0qq+koq35lltyaUh/aRSJnle8r/N8bytMHu6bAt99dn1bGwi8GCJwvDgf4u/LFbNtHGHfoFrS/9g9yeQvTXxvwJ/5/tDwlo88g52v3tv2jod+ue2IPHpCY/4AShp8jxvY0MQjYY/0GQZ7COyRNmCPvePxse3nvBfNG1YOs6EhfZmYpHExnAyLXlmjDEkTs3h9kzrAA+2VxjgR3CoNqWLa0aZjjtO4aB3/ZOtg67p03DQuGicB82hjGmOt2l7pjFSMbY5d6pjGReMDzo7lhJbLVyySxbxyWRpVxTgHLNy3lUt7xnAWLX9f1vkAabzDVVuXRyctkSyvZ4QH3jjeWj7nL8s2TCbLLMZLYuSv1wa+bx7dwyCKVdLukyLNM2rTPJr1uEmTHp0VV4cq7qn/z1PYb8vZpbXfmUV2+UBbZz9teEfe32FDQe6wtTek2tt/5RYAxVp3rzxOp2T/g45xRiZ7z29zikNO/wYw3JXvsLTt2A92+dYXreGXZSPfvzxsVv8wpX+Zff26+c3rr6++PfzhBP5o9XHjoPFl48+NZuOvjR8aZ41RY9qwGg+Nfzb+1fj3wR8Pvj9oHXTy1A8/gD6fN0qfg8F/AWYi9Bs=</latexit>

Figure 3: Partition of regions in Lemma 3.3.

The proof of Lemma 3.3 is inspired by the proofof [8, Theorem 3] and is given in Appendix F (seesupplementary material). Letting ε ≤ 0.3963‖x?‖32and η = 0.22‖x?‖22, the population risk (1.2) thensatisfies Assumption 2.1. As in Lemma 3.1, we alsonote that each critical point of the population riskin (1.2) is either a global minimum or a strict saddle.This inspires us to carry this favorable geometry overto the corresponding empirical risk.

The partition of regions used in Lemma 3.3 is illus-trated in Figure 3. We use the purple, green, and blueballs to denote the three regionsR1,R2, andR3, respectively. R4 is then represented with the lightgray region. Therefore, the union of the four regions covers the entire RN space.

Define a norm ball as B(l) , x ∈ RN : ‖x‖2 ≤ l with radius l = 1.1‖x?‖2. This particularchoice of l guarantees that ‖ grad f(x)‖2 is large outside of B(l), which is proved in Appendix G.Together with Theorem 2.1, we prove a globally benign landscape for the empirical risk. We also

define h(N,M) , O(N2

M +√

NM

)with O denoting an asymptotic notation that hides polylog

factors. The following lemma verifies Assumptions 2.2 and 2.3 for this phase retrieval problem.Lemma 3.4. Suppose that am ∈ RN is a Gaussian random vector with entries following N (0, 1).If h(N,M) ≤ 0.0118, we then have

supx∈B(l)

‖∇f(x)−∇g(x)‖2 ≤ε

2, and sup

x∈B(l)‖∇2f(x)−∇2g(x)‖2 ≤

η

2

hold with probability at least 1− e−CN log(M).

The proof of Lemma 3.4 is given in Appendix G (see supplementary material). The assumptionh(N,M) ≤ 0.0118 implies that we need a sample complexity that scales like N2, which is notoptimal since x has only N degrees of freedom. This is a technical artifact that can be traced back toAssumptions 2.2 and 2.3–which require two-sided closeness between the gradients and Hessians–andthe heavy-tail property of the fourth powers of Gaussian random process [12]. To arrive at theconclusions of Theorem 2.1, however, these two assumptions are sufficient but not necessary (whileAssumption 2.1 is more critical), leaving room for tightening the sampling complexity bound. Weleave this to future work.

4 Numerical Simulations

We first conduct numerical experiments on the two examples introduced in Section 1, i.e., the rank-1matrix sensing and phase retrieval problems. In both problems, we fix N = 2 and set x? = [1 − 1]>.

8

-2 -1 0 1 2-2

-1

0

1

2

(a)

-2 -1 0 1 2-2

-1

0

1

2

(b) M = 3

-2 -1 0 1 2-2

-1

0

1

2

(c) M = 10

Figure 4: Rank-1 matrix sensing: (a) Population risk. (b, c) A realization of empirical risk. In boththis and Figure 5, we use the red star and cross to denote the global minima and saddle points of thepopulation risk, and use blue square, circle, and diamond to denote the global minima, spurious localminima, and saddle points of the empirical risk, respectively.

-2 -1 0 1 2-2

-1

0

1

2

(a)

-2 -1 0 1 2-2

-1

0

1

2

(b) M = 3

-2 -1 0 1 2-2

-1

0

1

2

(c) M = 10

Figure 5: Phase retrieval: (a) Population risk. (b, c) A realization of empirical risk.

20 40 60 80 100

0.2

0.25

0.3

Lo

ca

l m

inim

a d

ist

Figure 6: Rank-2 matrix sensing.

Then, we generate the population risk and empirical risk based onthe formulation introduced in these two examples. The contour plotsof the population risk and a realization of empirical risk with M = 3and M = 10 are given in Figure 4 for rank-1 matrix sensing andFigure 5 for phase retrieval. We see that when we have fewer samples(e.g., M = 3), there could exist some spurious local minima as isshown in plots (b). However, as we increase the number of samples(e.g., M = 10), we see a direct correspondence between the localminima of empirical risk and population risk in both examples with a much higher probability. Wealso notice that extra saddle points can emerge as shown in Figure 4 (c), which shows that statement(b) in Theorem 2.1 cannot be improved to a one-to-one correspondence between saddle points indegenerate scenarios. We still observe this phenomenon even when M = 1000, which is not shownhere. Note that for the rank-1 case, Theorem 2.1 can be applied directly without restricting to full-rankrepresentations. Next, we conduct another experiment on general-rank matrix sensing with k = 2,r = 3, N = 8, and a variety of M . We set U? as the first r columns of an N ×N identity matrix andcreate X = U?U?>. The population and empirical risks are then generated according to the modelintroduced in Section 3.1. As shown in Figure 6, the distance (averaged over 100 trials) between thelocal minima of the population and empirical risk decreases as we increase M .

5 Conclusions

In this work, we study the problem of establishing a correspondence between the critical points ofthe empirical risk and its population counterpart without the strongly Morse assumption requiredin some existing literature. With this correspondence, we are able to analyze the landscape of anempirical risk from the landscape of its population risk. Our theory builds on a weaker conditionthan the strongly Morse assumption. This enables us to work on the very popular matrix sensing andphase retrieval problems, whose Hessian does have zero eigenvalues at some critical points, i.e., theyare degenerate and do not satisfy the strongly Morse assumption. As mentioned, there is still room toimprove the sample complexity of the phase retrieval problem that we will pursue in future work.

9

Acknowledgments

SL would like to thank Qiuwei Li at Colorado School of Mines for many helpful discussions on theanalysis of matrix sensing and phase retrieval. The authors would also like to thank the anonymousreviewers for their constructive comments and suggestions which greatly improved the quality of thispaper. This work was supported by NSF grant CCF-1704204, and the DARPA Lagrange Programunder ONR/SPAWAR contract N660011824020.

References[1] S. Mei, Y. Bai, and A. Montanari, “The landscape of empirical risk for non-convex losses,”

arXiv preprint arXiv:1607.06534, 2016.

[2] V. Vapnik, “Principles of risk minimization for learning theory,” in Advances in Neural Informa-tion Processing Systems, pp. 831–838, 1992.

[3] R. Ge, C. Jin, and Y. Zheng, “No spurious local minima in nonconvex low rank problems: Aunified geometric analysis,” in Proceedings of the 34th International Conference on MachineLearning-Volume 70, pp. 1233–1242, 2017.

[4] Z. Zhu, Q. Li, G. Tang, and M. B. Wakin, “The global optimization geometry of low-rank matrixoptimization,” arXiv preprint arXiv:1703.01256, 2017.

[5] R. Ge, J. D. Lee, and T. Ma, “Matrix completion has no spurious local minimum,” in Advancesin Neural Information Processing Systems, pp. 2973–2981, 2016.

[6] R. Sun and Z.-Q. Luo, “Guaranteed matrix completion via non-convex factorization,” IEEETransactions on Information Theory, vol. 62, no. 11, pp. 6535–6579, 2016.

[7] Q. Li, Z. Zhu, and G. Tang, “The non-convex geometry of low-rank matrix optimization,”Information and Inference: A Journal of the IMA, vol. 8, no. 1, pp. 51–96, 2018.

[8] X. Li, J. Lu, R. Arora, J. Haupt, H. Liu, Z. Wang, and T. Zhao, “Symmetry, saddle points,and global optimization landscape of nonconvex matrix factorization,” IEEE Transactions onInformation Theory, vol. 65, no. 6, pp. 3489–3514, 2019.

[9] S. Tu, R. Boczar, M. Simchowitz, M. Soltanolkotabi, and B. Recht, “Low-rank solutions oflinear matrix equations via Procrustes flow,” in International Conference on Machine Learning,pp. 964–973, 2016.

[10] Q. Li, Z. Zhu, G. Tang, and M. B. Wakin, “The geometry of equality-constrained globalconsensus problems,” in ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pp. 7928–7932, IEEE, 2019.

[11] D. Davis, D. Drusvyatskiy, and C. Paquette, “The nonsmooth landscape of phase retrieval,”arXiv preprint arXiv:1711.03247, 2017.

[12] J. Sun, Q. Qu, and J. Wright, “A geometric analysis of phase retrieval,” Foundations of Compu-tational Mathematics, vol. 18, no. 5, pp. 1131–1198, 2018.

[13] Y. Li and Y. Bresler, “Global geometry of multichannel sparse blind deconvolution on thesphere,” in Advances in Neural Information Processing Systems, pp. 1132–1143, 2018.

[14] Y. Zhang, Y. Lau, H.-w. Kuo, S. Cheung, A. Pasupathy, and J. Wright, “On the global geometryof sphere-constrained sparse blind deconvolution,” in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pp. 4894–4902, 2017.

[15] R. Ge and T. Ma, “On the optimization landscape of tensor decompositions,” in Advances inNeural Information Processing Systems, pp. 3653–3663, 2017.

[16] R. Ge, J. D. Lee, and T. Ma, “Learning one-hidden-layer neural networks with landscape design,”arXiv preprint arXiv:1711.00501, 2017.

10

[17] Q. Li and G. Tang, “Convex and nonconvex geometries of symmetric tensor factorization,” inAsilomar Conference on Signals, Systems, and Computers, 2017.

[18] S. Boucheron, G. Lugosi, and P. Massart, Concentration inequalities: A nonasymptotic theoryof independence. Oxford University Press, 2013.

[19] C. Jin, R. Ge, P. Netrapalli, S. M. Kakade, and M. I. Jordan, “How to escape saddle pointsefficiently,” in Proceedings of the 34th International Conference on Machine Learning-Volume70, pp. 1724–1732, JMLR. org, 2017.

[20] S. Burer and R. D. Monteiro, “A nonlinear programming algorithm for solving semidefiniteprograms via low-rank factorization,” Mathematical Programming, vol. 95, no. 2, pp. 329–357,2003.

[21] S. Burer and R. D. Monteiro, “Local minima and convergence in low-rank semidefinite pro-gramming,” Mathematical Programming, vol. 103, no. 3, pp. 427–444, 2005.

[22] Z. Zhu, Q. Li, G. Tang, and M. B. Wakin, “Global optimality in distributed low-rank matrixfactorization,” arXiv preprint arXiv:1811.03129, 2018.

[23] P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization algorithms on matrix manifolds.Princeton University Press, 2009.

[24] M. Journée, F. Bach, P.-A. Absil, and R. Sepulchre, “Low-rank optimization on the cone ofpositive semidefinite matrices,” SIAM Journal on Optimization, vol. 20, no. 5, pp. 2327–2351,2010.

[25] E. J. Candès and Y. Plan, “Tight oracle inequalities for low-rank matrix recovery from a minimalnumber of noisy random measurements,” IEEE Transactions on Information Theory, vol. 57,no. 4, pp. 2342–2359, 2011.

[26] M. A. Davenport and J. Romberg, “An overview of low-rank matrix recovery from incompleteobservations,” IEEE Journal of Selected Topics in Signal Processing, vol. 10, no. 4, pp. 608–622,2016.

[27] B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrixequations via nuclear norm minimization,” SIAM Review, vol. 52, no. 3, pp. 471–501, 2010.

11

the landscape of non-convex empirical risk with degenerate ... · the landscape of non-convex...

Documents