A Certifiably Globally Optimal Solutionto the Non-Minimal Relative Pose Problem

Jesus BrialesMAPIR-UMA Group

University of Malaga, Spainjesusbriales@uma.es

Laurent KneipMobile Perception Lab

SIST ShanghaiTechlkneip@shanghaitech.edu.cn

Javier Gonzalez-JimenezMAPIR-UMA Group

University of Malaga, Spainjaviergonzalez@uma.es

Abstract

Finding the relative pose between two calibrated viewsranks among the most fundamental geometric vision prob-lems. It therefore appears as somewhat a surprise that aglobally optimal solver that minimizes a properly definedenergy over non-minimal correspondence sets and in theoriginal space of relative transformations has yet to be dis-covered. This, notably, is the contribution of the presentpaper. We formulate the problem as a Quadratically Con-strained Quadratic Program (QCQP), which can be con-verted into a Semidefinite Program (SDP) using Shor’s con-vex relaxation. While a theoretical proof for the tightness ofthis relaxation remains open, we prove through exhaustivevalidation on both simulated and real experiments that ourapproach always finds and certifies (a-posteriori) the globaloptimum of the cost function.

1. Introduction

Relative pose estimation from corresponding point pairsbetween two calibrated views constitutes a fundamentalgeometric building block in most sparse structure-from-motion and visual SLAM algorithms. In structure from mo-tion, it helps to initialize both topology and initial values ofthe view-graph, which will then be used in transformationaveraging schemes [15] as well as bundle adjustment [42]to obtain the joint solution over all poses and points. Morespecifically, using an image matching technique [29], we es-tablish a hypothetical covisibility graph between neighbour-ing pairs of views, which are then verified geometrically us-ing robust relative pose computation. In visual SLAM [26],the algorithm is used to bootstrap the computation and thussolve the chicken-and-egg problem behind the mutually de-pending tracking and mapping modules.

The relative pose problem is constrained by the epipo-lar geometry [17]. The correspondence condition betweentwo views can notably be translated into a linear constraint

by employing the well-known essential matrix. Due to scaleinvariance, the essential matrix has only five degrees of free-dom. Five point correspondences are hence enough to con-strain the relative pose [23]. The solution to the linear prob-lem given by stacking five correspondence conditions firstresults in a four-dimensional nullspace. The true solutioncan then be recovered by applying the additional non-linearconstraints that exist on the coefficients of a proper essen-tial matrix. It can be solved using polynomial eliminationtechniques [28], or—more generally—the Groebner basismethod [38].

The above solution is the so-called minimal solution tothe problem as—ignoring special cases like degenerate 3Dpoint configurations or zero parallax—it always leads to anexact solution for the considered correspondences (up tonumerical inaccuracies). The minimal solver can be usedwithin Ransac in order to gain robustness against outliercorrespondences. An important observation in this regardis that—depending on the noise inside the data—using onlyfive points for each hypothesis instantiation may not nec-essarily lead to a minimum number of Ransac iterations,as those hypotheses may be hard to bring in agreement withother points. For moderate effective outlier ratios, increasednoise in the data means that sampling more points will leadto noise-cancellation effects and thus quicker convergenceand better identified inlier ratios [21]. In short, a completetheory of epipolar geometry requires a non-minimal solverto the relative pose problem.

What comes as a surprise is that, despite the maturity ofthe field and the age of the problem, a good non-minimalsolver for the calibrated relative pose problem has neverbeen discovered. This stands in harsh contrast with theabsolute pose problem, where recent solvers even achieveglobal and geometric optimality in closed form for an arbi-trary number of 2D-to-3D correspondences [20]. The expertreader may ask why we can not use the 7 or 8-point algo-rithm [24, 13] to solve this problem. The reason is againexplained by the noise in the data. Noise notably causesthe rank-defficiency of our essential matrix constraints to

vanish. The solution is henceforth found in a smaller-dimensional nullspace. However, what we are in fact do-ing here is hoping that the additional non-linear constraintson our essential matrix are implicitly enforced by the noisydata. To give a more concrete example, when solving thecalibrated relative pose problem with the 8-point algorithm,we are in fact solving for a fundamental matrix. We justhope that our data is strong enough to implicitly enforce thesolution to also be a proper essential matrix. This, of course,is only true in the noise-free case.

It is much better to find the solution to the non-minimalproblem by directly minimizing an energy defined in the5-dimensional space of scale invariant relative poses. Thisis an inherently difficult problem, and an elegant globallyoptimal solution remains an open problem. The present pa-per draws inspiration from [21] who presented a cost func-tion for the relative pose problem that is parametrized in thespace of relative rotations, and can be constructed for anarbitrary number of correspondences. However, while [21]failed to solve this problem with optimality guarantees, thepresent paper leverages convex optimization theory to—forthe first time—come up with a fast1 probably globally opti-mal solution to this problem, whose optimality gets certifieda-posteriori.

We warn the reader, this paper does not deal with outlierobservations. However, we observe that even without out-liers this problem is challenging enough to make traditionalsolvers fail in certain scenarios.

2. Further related work on global optimizationIn general, finding a guaranteed globally optimal solu-

tion for non-convex optimization problems is a hard task,most often computationally intensive [14].

One natural approach consists of characterizing the glob-ally optimal solution as one of the stationary points in theenergy functional. This can be cast as a polynomial problemand solved with similar techniques than minimal problems.However, for general non-minimal problems, the numberof stationary points and the size of the polynomial elimi-nation template may explode as the order of the equationsincreases, often rendering the application of this procedureinfeasible.

Another powerful and generic tool for NP-hard opti-mization problems is Branch and Bound (BnB). This pro-ceeds by cleverly exploring (branching) the whole opti-mization space while exploiting some relaxations on theproblem to skip certain regions (bounding). Whereas thisapproach has been successfully used in various geometriccomputer vision problems [31, 18, 21, 46], its exploratorynature often leads to slow performance (exponential time inworst-case scenario).

1At least faster than other globally optimal alternatives with guaranteessuch as Branch-and-Bound for the non-convex problem.

For the rest of this document we will focus instead on re-laxation techniques that consider approximate, simpler ver-sions of the optimization problem whose global optimum iseasier to reach. This general idea does not necessarily leadus to the original, optimal solution. In fact, an inferior relax-ation may not provide any useful information at all. On theopposite side, a tight relaxation is one that features the sameoptimal objective as the original optimization problem. Atight relaxation may provide us with the necessary informa-tion to recover the optimal solution to the original problem.If a relaxation that is tight is also convex by construction,this provides us with an appealing way to solve the originalhard problem globally, as solving a convex problem glob-ally is a much more tractable problem (typically of polyno-mial complexity).

Finding a good relaxation for a certain problem is notan exact science. Even though various recurrent tools ex-ist, such as relaxing non-convex constraints to its convexhull [36, 34, 19] or applying Lagrangian duality [5], thereremains much engineering involved in the combination ofthese tools to come up with a good relaxation.

In this work we leverage two fundamental ideas as guid-ing design principles: Firstly, many polynomial optimiza-tion problem can be reformulated as a Quadratically Con-strained Quadratic Program (QCQP), which in turn maybe relaxed onto a Semidefinite Program (SDP) via Shor’srelaxation [5, 11]. This observation alone has allowedfor great progress in global optimization for some prob-lems whose SDP relaxation turns out to be tight, e.g.Rotation Synchronization [4, 3] or Pose Synchronization[10, 9, 6, 33, 8]. Secondly, relaxations can be strength-ened (improved) by introducing redundant constraints [27,Chap. 13]. This trick has found applicability in the op-timization literature [32, 35], e.g. in QCQP problems in-volving orthonormal constraints [1]. Briales and Gonzalez-Jimenez show in [7] that properly applying this trick ishighly beneficial when working with rotation constraints inthe context of the Absolute Pose problem.

3. PreliminariesIn this work we will follow similar notation and assump-

tions to those used by Kneip and Lynen in [21]. Specifically,we consider the central calibrated relative pose case, whereeach image feature can be translated into a unique unit bear-ing vector. Pair-wise correspondences thus consist of pairsof bearing vectors (f i,f

′i) pointing to the same 3D world

point pi from the first and second camera centers.The variables of interest in the problem are the transla-

tion t and relative orientation R of the second camera framew.r.t. that of the first camera. All the variables involvedare illustrated in Fig. 1. The normalized direction t willbe identified with points in the 2-sphere S2. The 3D rota-tion will be featured as 3 × 3 orthogonal matrix with posi-

Figure 1. The relative pose problem. The measurements are givenby correspondence pairs of unit bearing vectors {fi, f ′i}, and theunknown variables are given by the relative orientation R and thedirection of the relative translation t.

tive determinant belonging to the Special Orthogonal groupSO(3). Symmetric n × n matrices are denoted by Symn,the subindex + standing for positive semidefiniteness.

Lastly, for convenience we will often refer to the list ofelements in a matrix as a vector (lowercase) stacking thecolumns into a vector, e.g. r = vec(R) or x = vec(X).

4. An eigenvalue based formulationGiven the bearing vectors f i and f ′i corresponding to

each feature i in the two (calibrated) camera projections,Kneip and Lynen proposed in [21] to address the relativepose problem by enforcing the coplanarity of the momentvectors mi = f i × Rf ′i for all the epipolar planes. Thiswould lead to a decoupled formulation where the optimalrotation is recovered from

R? = arg minR∈SO(3)

λmin(M(R)), (1)

with M(R) =∑N

i=1 mim>i standing for the covariance

matrix of all (in general non-unitary) epipolar plane nor-mals. The translation direction is recovered as the eigenvec-tor corresponding to the smallest eigenvalue of M(R?):

t? = arg mint∈S2

t>(M(R?))t. (2)

Whereas this decoupled approach may be compact and ap-pealing, dealing with the non-linearity and non-convexity ofthe rotation suproblem (1) is far from trivial.

In this work, we take a different approach and considerinstead the equivalent joint optimization problem

f? = minR,t

t>M(R)t, s.t. R ∈ SO(3), t ∈ S2. (3)

The elements of the covariance matrix of normals M(R) =[mij(R)

]3i,j=1

are quadratic on the elements of the rotationmatrix r = vec(R) [21]. This allows us to rewrite eachcomponent of M(R) as a quadratic function mij(R) =r>Cijr

> with Cij ∈ Sym9,+ a data matrix that depends

only on the problem data (see supplementary material fordetails). Using this characterization, the objective to opti-mize in (3) may be written as

f(R, t) = t>M(R)t =

n∑i,j=1

ti(r>Cijr)tj , (4)

which is a quartic function of the unknowns (R, t).A fundamental step towards our solution is the reformu-

lation of this quartic objective into a quadratic one by intro-ducing the convenient auxiliary variable

X = rt> =[t1r | t2r | t3r

]=[ritj]i=1,..,9j=1,2,3

(5)

and its vectorized form

x = vec(X) = vec(rt>). (6)

With this definition of X in mind, the optimization objec-tive (4) may be equivalently written in terms of x, and be-comes

f(R, t) =

n∑i,j=1

(tir)>Cij(tjr) = x>Cx, (7)

where the data matrix C ∈ Sym27 gathers all the data (ma-trices Cij) in the problem (3). Interested readers are in-vited to check the supplementary material for further de-tails. With this reformulation in mind, the joint optimizationproblem (3) can be rewritten as

f? = minR∈SO(3)

t∈S2,X

x>Cx, s.t. X = rv>, (8)

where we assume the already presented notations for vec-torized counterparts.

So far we have just rewritten the original problem in dif-ferent forms, all of which are equivalent and inherently hardto solve. In particular, the last proposed formulation (8) canbe seen as a Quadratically Constrained Quadratic Program(QCQP): The objective f(X) is quadratic; the unitary con-straint on the translation direction, Ct ≡ {t>t = 1}, is alsoquadratic; a rotation matrix R can be fully defined solelyby a set of quadratic constraints CR [22, 43, 7]; and, lastly,the auxiliary constraint CX on X (5) is also quadratic. Thecorresponding QCQP may be represented then as

minX,R,t

x>Cx, s.t. {CR, Ct, CX}(X,R, t). (9)

Having formulated the problem as a QCQP does notmake it any easier to solve. In fact, a QCQP is a very gen-eral kind of problem that comprises many NP-hard prob-lems. However, the interest in reaching this particular for-mulation lies in the existence of a well-known SemidefiniteProgram (SDP) relaxation for QCQPs.

Next section will provide a thorough overview of a SDPrelaxation for problem (8) that, as far as we empirically ob-served, is always tight and allows us to recover and certifythe globally optimal solution (R?, t?). An overview of thecomplete algorithmic approach is given in Alg. 1.

5. Global resolution through SDP relaxationAny Quadratically Constrained Quadratic Programming

(QCQP) problem instance, and in particular the introducedrelative pose problem (9), may be written in the followinggeneric form2.

Problem QCQP (General).

minz

= z>Q0z, z =[z>, 1

]>, (10)

s.t. z>Qiz = 0, i = 1, . . . ,m, (11)

where z is a vector stacking all unknowns involved in theproblem and z its homogeneization.

Note we homogeneize the variables and data matrices,which is a common trick to get more compact quadratic ex-pressions [43, 7]. In our problem, z stacks the elementsof our unknowns (R, t,X). For further details on homo-geneization as well as the matrices Qi corresponding to theQCQP formulation of the relative pose problem we refer theinterested reader to the supplementary material.

As previously stated, the problem (QCQP) is in generalan NP-hard problem. However, it is well-known that thiskind of problems can be relaxed to a convex SemidefiniteProgram (SDP), also known as Shor’s relaxation [37, 11]. Itis tightly connected to Lagrangian duality [44, 5]. The pri-mal version of the SDP relaxation for the problem (QCQP)reads:

Problem SDP (Primal SDP).

d? = f?SDP = minZ

tr(Q0Z) (12)

s.t. tr(QiZ) = 0, i = 1, . . . ,K, (13)

Zyy = 1, Z < 0, (14)

where Z is the lifted unknown matrix and the y-subindexrefers to the homogeneous component in the matrix.

As a relaxation, the problem above fulfills3

d? = f?SDP ≤ f?. (15)

Our interest in this relaxation resides in the well-known fact[44] that if the relaxation (SDP) is tight (d? = f?SDP = f?)

2Note that QCQPs can in general also feature inequality constraints.3We borrow the notation d? for the optimal relaxation objective, more

common for the dual of (SDP). In our context the SDP problem fulfillsSlater’s condition so we can safely state d? = f?

SDP.

and the original problem (QCQP) features a unique globalminimum, we can easily obtain the guaranteed optimal so-lution z? to the original problem from the optimal solutionZ

?to the SDP relaxation. Thus, if we are able to ensure

(either theoretically or empirically) that the SDP relaxationfor a given QCQP problem is tight in some scenario, thisprovides a highly appealing way to approach the global res-olution of the original hard QCQP problem.

Unfortunately, the SDP relaxation arising from theQCQP problem corresponding to the proposed formulation(8) for the relative pose problem is in general not tight, sowe may not benefit of the corresponding SDP relaxation asit is. Thus, our next goal will be to explore how we may at-tain this desired tight behavior in the context of the relativepose problem at hand.

5.1. Making SDP tight: Redundant constraints

In this section we are going to investigate the applicationof a well-known trick to our problem to improve the qualityof the relaxation: The introduction of additional redundantconstraints.

There are some interesting examples in the literature[45, 7] on how introducing additional redundant constraintsinto a QCQP problem may significantly improve the sub-sequent SDP relaxation (in terms of tightness). In partic-ular, Briales and Gonzalez-Jimenez show in [7] that for aQCQP problem affected solely by a (non-convex) 3D ro-tation constraint, it is possible to produce an SDP relax-ation that remains (empirically) tight at all times by clev-erly exploiting complementary descriptions of the rotationconstraints. Since our problem also features a 3D rotationconstraint R ∈ SO(3), it makes sense to apply this trickhere as well.

5.1.1 Redundant rotation constraints

Briales and Gonzalez-Jimenez argue in [7] that the full fam-ily of independent rotation constraints for R ∈ SO(3) thatmay be written as quadratic expressions is given by

CR ≡

R>R = I3, RR> = I3,

(Rei)× (Rej) = (Rek),

∀(i, j, k)={(1, 2, 3), (2, 3, 1), (3, 1, 2)}.(16)

The set CR amounts to 20 independent quadratic con-straints, which may be written in the form r>Qir = 0.

Unfortunately, whereas in the case explored in [7] thesimple introduction of these additional constraints into theproblem (QCQP) would suffice to turn the correspondingrelaxation (SDP) tight in practice, we found out that this isnot yet enough in the present problem.

5.1.2 More redundant quadratic constraints

A convenient trick to obtain additional redundant con-straints in the context of our relative pose problem (8) isto produce higher order versions of the existing constraints(R ∈ SO(3),t ∈ S2) that may be still rewritten as quadraticexploiting the available auxiliary variable X (5).

Lifted sphere constraints: Ct We saw the translation fea-tures a single quadratic constraint t>t = 1. We may buildredundant constraints by multiplying this constraint by rifor i = 1, . . . , 9. Even though this new constraint is im-plicitly cubic, we may refactor products of the form ritj viathe auxiliary variable X =

[ritj]i=1,...,9j=1,2,3

, which features

all second-order combinations of this kind. As a result weobtain 9 new constraints which are implicitly cubic but ex-pressed as quadratic constraints again.

From here on, most new constraints are built in a similarway. We multiply the quadratic constraint t>t = 1 by thequadratic factor rirj for i, j = 1, . . . , 9. This time, it re-sults in an implicitly quartic constraint that may be rewrit-ten though via the auxiliary mapping Xij = ritj into aquadratic constraint again. In this case, due to symmetry,we introduce

(92

)= 45 additional redundant constraints.

Lifted rotation constraints: CR The case for rotationconstraints is pretty much analogue to that of the sphereconstraint. For any of the quadratic constraints previouslyconsidered for the rotation set, we may multiply this con-straint by ti for i = 1, 2, 3, and again rewrite pairs of theform ritj as Xij to obtain a quadratic expression that isimplicitly cubic. Similarly, if we multiply in the quadraticterm titj for i, j = 1, 2, 3 and rewrite in terms ofXij wherepossible, we will be featuring quartic constraints written inquadratic form.

If there are 20 independent constraints in the rotationconstraint set, the cubic lift produces 3 new constraints foreach one, while the quartic lift results in

(32

)= 6 additional

redundant constraints per rotation constraint. Overall, thisresults in 20 · 9 = 180 additional quadratic constraints.

Additional constraints on auxiliary variable: CX Forthe auxiliary variable X it is also possible to feature addi-tional independent constraints. In this case, it is not througha lifting procedure but simply by observing that, through itsdefinition in (5), rank(X) = 1. While the rank constraint isa complex one that we do not want to directly employ here,it features though a wide set of quadratic constraints as any2×2 minor inside the matrix X needs to be zero (otherwiserank(X) > 1, which contradicts its definition).

These constraints are of the form

XabXij −XibXaj = 0, (17)a = 1, . . . , 9; b = 1, 2, 3; (18)a < i ≤ 9; b < j ≤ 3, (19)

which finally results in 108 additional linearly independentconstraints that depend solely on X .

Surprisingly, after including this whole redundant set ofunderlying quadratic constraints, the SDP relaxation for thecorresponding enriched QCQP problem turns out to be em-pirically tight in all tested circumstances, and we are readyto proceed further in our way towards the global resolutionof the original problem.

5.2. Solving from the tight SDP solution: Recovery

Now that we have featured a QCQP formulation for therelative pose problem (8) whose convex SDP relaxation isempirically tight, we should be ready to proceed and solvethe problem (QCQP) with global optimality guarantees.

However, we still encounter another difficulty: It is aclassical result related to Shor’s relaxation that if there isa unique global solution in the problem (QCQP), the tightSDP solution Z

?fulfills rank(Z

?) = 1 and we may re-

cover z? from the low-rank decomposition Z?

= z?(z?)>

[44, 25]. As we show next, the condition that the originalproblem has a unique global minimum is not fulfilled herethough. Nevertheless, we will show how we can adapt theclassical low-rank decomposition trick [25] and still obtainthe solution to the original problem (8) from a solution Z

?

to the relaxation (SDP).

5.2.1 The relative pose problem has 4 solutions

An important characteristic of original formulation (3) ofthe relative pose problem is that there exist symmetries inthe objective. In particular,

f(R,−t) = f(R, t), f(P tR, t) = f(R, t), (20)

where P t = 2tt>−I3 is the reflection matrix w.r.t. the axisof direction t. These symmetries are illustrated and provenin the supplementary material.

The symmetries are also consistent with the well-knownfact that the estimation of the relative pose from the essen-tial matrix also leads to 4 different solutions [24]. Indeed,our algebraic error is exactly equivalent to that minimizedby classical solvers, e.g. the 8-point algorithm by Longuet-Higgins [24, 16].

As a consequence of the equivalences above, even in thewell constrained, non-minimal situation, there exist 4 glob-ally optimal solutions to the formulated problem (3):

{(R?, t?), (R?,−t?), (P t?R?, t?), (P t?R

?,−t?)}.(21)

Of these solutions, only one is physically realizable as therest lead to geometries where the reconstructed 3D pointslie behind the cameras rather than in front of them [24].

5.2.2 The tight primal SDP solution has rank-4

Still under the assumption that the SDP relaxation is tight,a feasible (and optimal) rank-1 solution Z

?

k for the problem(SDP) can be built from any of the 4 solutions (21) to ouroriginal problem (8):

(R?k, t

?k)→ z?

k = stack(x?, r?, t?, 1) (22)

→ Z?

k = z?k(z?

k)>. (23)

All of these lifted solutions are globally optimal as they ful-fill tr(Q0Z

?

k) = f? = d? (with our tightness assumption).By linearity of the SDP objective (12), it is easy to prove

(see supplementary material) that any convex combination

Z?

=

4∑k=1

akZ?

k =

4∑k=1

akz?k(z?

k)> (24)

of the rank-1 solutions Z?

k, with ak ≥ 0 and∑4

k=1 ak = 1,is also an optimal solution to the problem (SDP). The non-negativity of ak is required to fulfill the Positive Semidefi-nite constraint (14) on Z. These coefficients a may be re-garded as barycentric coordinates that parameterize the setof all possible optimal solutions for problem (SDP).

In conclusion, the existence of 4 globally optimal solu-tions in the original problem results in the existence of in-finitely many rank-4 solutions to the SDP relaxation (SDP).These solutions coincide with the convex hull of the rank-1lifted versions of the solutions to the problem (QCQP).

5.2.3 Practical recovery of original solution

At this point, we have fully characterized the optimal so-lutions of the original problem (3) as well as their connec-tion to the infinitely many solutions of the relaxation (SDP)when this is tight. In practice, we will solve the relaxation(SDP) with some off-the-shelf Primal-Dual Interior PointMethod (IPM) solver (e.g. SeDuMi [40] or SDPT3 [41]).But these IPM approaches return one optimal solution Z

?

0

of the infinitely many available ones in the convex set Z?.

Now, the question of main practical interest is: Given a par-ticular optimal solution Z

?

0 to the SDP problem (SDP), arewe able to recover the optimal solution (R?, t?) (and cor-responding symmetries) to the original problem (3)? Theanswer to this question is yes.

In practice, a fundamental observation is that the solutionprovided by an IPM solver always fulfilled rank(Z

?

0) = 4,with an eigenvalue decomposition of the form

Z?

0 = λ1U1U>1 + λ2U2U

>2 , U>k Uk = I2, (25)

that is, there exist two numerically distinct eigenvaluesλ1, λ2 with multiplicity 2. From (24) we know both U1

and U2 together must span the same range space as all theglobal solutions to the (QCQP). In fact, another important(still empirical) observation is that each pair of eigenvec-tors Uk span the same range as the solutions with commonrotation value:

span(U1) = span([z?1, z

?2

])← {(R?, t?), (R?,−t?)},

span(U2) = span([z?3, z

?4

])← {(P t?R

?, t?), (P t?R?,−t?)}.

This relation allows us to recover the optimal solutions fromthe range space of the appropriate sub-blocks in the com-puted eigenvectors: Considering the case of U1, since4

z?1(r, :) = z?

2(r, :) = r? and z?1(t, :) = z?

2(t, :) = t?,

span(U1(r, :)) = span(r?), span(U1(t, :)) = span(t?).

As a result, if V r ∈ R9 and V t ∈ R3 are basis vectors forthe rank-1 subspaces span(U1(r, :)) and span(U1(t, :))respectively, the optimal solutions within the range of U1

can be obtained by simple normalization:

vec(R?) =√

3V r

‖V r‖, t? =

V t

‖V t‖. (26)

Similar relations hold for the solutions contained in U2.Further discussions on these empirical observations can

be found in Section 8 of the supplementary material.

5.2.4 A-posteriori global optimality guarantees

Throughout the previous subsections we have extensivelybuilt upon the assumption that the relaxation (SDP) is tight.This, however, is not known a-priori for a given problem.Thus, the way we proceed in practice is as follows: First,we proceed with the described approach as if the relaxation(SDP) was indeed tight, obtaining a set of (symmetric) can-didate solutions (Rk, tk). Then, we check the primal fea-sibility (Rk ∈ SO(3), tk ∈ S2) of these candidates. Ifthe relaxation was tight, these candidates should be indeedfeasible. Just for numerical stability, we may project thesecandidates into their closest point in their manifold (by clas-sical means), and check that d? = f(Rk, tk) as a certificateof optimality. If this property is fulfilled, this proves ourtightness assumption and these feasible candidate points areindeed the global solutions of the original problem (3).

The complete algorithm is characterized in Alg. 1.

6. ExperimentsThe main goal of this section will be to empirically prove

the surprising and desirable fact that the proposed SDP re-laxation always turns out to be tight in practice, providing a(a-posteriori) guaranteed optimal solution.

4We use (r, :) and (t, :) as Matlab-like notation to refer to the set ofindexes corresponding to r and t within z by convention (see Sec. 5).

Algorithm 1: Solving relative pose via SDP relaxation

Input: List of correspondences {(f i,f′i)}Ni=1

Output: Probably certifiable global solution (R?, t?)/* build (QCQP) constr. mat. (full set) */

1 {Qi}Ki=1 ← {CR, Ct, CX}(X,R, t);/* build (QCQP) objective data matrix */

2 Q0 ← C ← {Cij}i,j=1,2,3 ← {(f i,f′i)}Ni=1

/* solve relaxation (SDP) */

3 (Z?

0, d?)← SDP(Q0, {Qi}Ki=1) ; // IPM [41, 40]

/* compute low-rank eig-decomposition */

4∑

k λkuku>k ← Z

?

0, λk ≥ λk+1 ∀k;5 ASSERT( (λ1 =λ2) ∧ (λ3 =λ4) ∧ (λk =0, ∀k > 4) );/* recover candidate sol. (Sec. 5.2.3) */

6 (Rk, tk)k=1,2 ← {u1,u2}7 (Rk, tk)k=3,4 ← {u3,u4}/* project candidates to feasible set */

8 Rk ← projSO(3)(Rk), tk ← projS2(tk), ∀k = 1, .., 4;/* check optim. gap, certify optimality */

9 ∆k = f(Rk, tk)− d?;10 ASSERT( ∆k < tol, ∀k = 1, .., 4 ), tol ∼ 10−9;

/* disambiguate realizable solution [24] */

11 return (R?, t?)← {(Rk, tk)}

For this purpose, we evaluated the proposed approach,SDP, in an extensive batch of experiments that span a wideset of problem configurations, both on synthetic and realdata. The problem (SDP) was modeled with CVX [12] andthe IPM solver used in practice was SDPT3 [41], whichtakes around 1 second to solve on a common consumer-grade computer. We compared our performance to that ofthe state-of-the-art (local) eigensolver proposed by Kneipand Lynen [21], which we initialized using the classical8-point algorithm by Longuet-Higgins [24]. The linearclosed-form estimator is referred to as 8pt, and its refinedversion through the eigensolver as 8pt+eig. Note that allof these methods feature exactly the same equivalent opti-mization objective, so it is fair to compare all of them interms of the attained objective value.

In the subsequent evaluation, we will rely on two differ-ent metrics. First, since the SDP relaxation (SDP) providesa lower bound d? on the optimal objective f?, we definethe optimality gap for a candidate solution ∆ = f − d?.If this optimality gap is numerically zero, we can assert thecandidate is globally optimal (check Section 5). Despite itstheoretical soundness, the optimality gap measure may belittle intuitive. Thus, we also provide the more intuitive ori-entation error measure of a given candidate w.r.t. a known-optimal solution5. For space’s sake, we skip here the er-

5Since the duality gap always vanishes numerically for the solution ofSDP, this will be our optimal reference.

ror measure on the translation direction component, whichleads to similar conclusions to those from orientation error.

6.1. Synthetic data

We generate random problems by the following proce-dure: We fix the first camera at the origin (orientation isidentity). Then, we generate a set of 3D world points ran-domly distributed within a bounded region of the space thatlies fully inside the Field of View (FOV) of the first cam-era, and at least one meter away from the camera. Next, wegenerate a random pose for the second camera with boundedtranslation parallax and constrained so that the world pointsalso lie within its FOV. This way we obtain random relativeposes that are consistent with the physical constraints thatwould exist in practical situations. From this point, we pro-ceed as in Kneip and Lynen [21]: bearing vectors for each3D point are computed and corrupted with some pixel noiseassuming a focal length of 800 pixels for the camera.

The synthetic world allows us to tune at will several rel-evant parameters of the problem: The noise level in theimage features (σ, in pix), the number of observed fea-tures (N ), the FOV of the camera (θ, in deg), and the par-allax magnitude (‖t‖, in m). For the evaluation, we as-sumed a reference set of parameters σ = 0.5pix, N = 10,θ = 100 deg, and ‖t‖ ≤ 2m, and then proceed to analyzethe behavior of the evaluated methods by modifying onesingle parameter. The results of the synthetic evaluationare collected in Fig. 2. We display the evaluation met-rics both with a boxplot and a superimposed histogram ofconcrete values. A shallow boxplot collapsed in 0 pointsto a frequent global convergence, whereas outliers (whenexistent) clearly reveal failure cases with wrong converge.In the evaluation, we generated 200 different instances foreach problem configuration.

In these results we see the proposed approach SDP al-ways attained a zero optimality gap (∆SDP ∼ o(10−10) inpractice), whereas the optimality gap for the other methodsrose up to ∆8pt,∆8pt+eig ∼ o(10−3) in some cases. In-terestingly, even though these optimality gaps do not seemtoo large, the analysis of the corresponding rotation errorreveals this gap is actually important, as it incurs in largeerrors in the estimated orientation parameter. This is con-sistent with the fact that the squared error terms minimizedin the objective are bounded by 1, and in general, even iferroneous, are not too high. This seems to result in sub-optimal local minima having objective values dangerouslyclose to those of the global solution [21], which may lead tomissing the true global solution.

A close analysis on how often the alternative solver8pt+eig fails to converge provides valuable feedbackabout the inherent difficulty of the problem. The obtainedresults suggest that difficulty increases with high level ofnoise in observations and low numbers of observations,

0.1 0.5 1 2 0

0.002

0.004

0.006

0.008

0.01

0.012

optim

alit

y g

ap (

f-d)

SDP

8pt

8pt+eig

(a)

8 10 20 500

0.05

0.1

0.15

0.2

0.25

optim

alit

y g

ap (

f-d)

SDP

8pt

8pt+eig

(b)

50 90 120 1500

0.002

0.004

0.006

0.008

0.01

0.012

optim

alit

y g

ap (

f-d)

SDP

8pt

8pt+eig

(c)

0.001 0.5 1 2 0

0.2

0.4

0.6

0.8

1

1.2

optim

alit

y g

ap (

f-d)

10 -3

SDP

8pt

8pt+eig

(d)

0.1 0.5 1 2 0

20

40

60

80

rota

tion e

rror

[deg]

SDP

8pt

8pt+eig

(e)

8 10 20 500

50

100

150

200

rota

tion e

rror

[deg]

SDP

8pt

8pt+eig

(f)

50 90 120 1500

5

10

15

20

25

30

35

rota

tion e

rror

[deg]

SDP

8pt

8pt+eig

(g)

0.001 0.5 1 2 0

0.5

1

1.5

2

2.5

rota

tion e

rror

[deg]

SDP

8pt

8pt+eig

(h)

Figure 2. Optimality gap and rotation error vs. all tunable parameters. Default values: σ=0.5pix,N=10, FOV=100 deg, and ‖t‖≤2m.Note that despite choosing σ≤ 2pix for visualization, experiments were performed with virtually unbounded noise (σ ∼ 103pix), stillleading to tight relaxations in all cases, thus supporting the generality of tightness for this relaxation.

which is an expectable behavior [30, 7]. Also, an increas-ing FOV makes the problem easier, as this results in a betterconstraining of the optimization objective [21]. A largerparallax hindered the performance of 8pt and 8pt+eig.

We also evaluated our approach in the pure rotation sce-nario. Interestingly, whereas the relaxation in this case alsoremains tight, some additional challenges appear when re-covering the optimal solutions from the SDP solution, spe-cially in the noiseless case where the number of solutionsdoubles. This is an interesting case that we keep howeverfor further analysis in future work, due to its extension.

6.2. Real data

For the evaluation on real data, we took pairs of over-lapping images from the TUM benchmark sequence [39],which provides both accurate ground-truth camera poses aswell as the intrinsic camera calibration. We extract andmatch SURF features in the images, filtering outliers bythresholding on the reprojection error of triangulated cor-respondences. Finally, we run the same algorithms as in theprevious section. The results, which are displayed in thesupplementary material due to space limitations, are con-sistent with the conclusions reached in the synthetic case.In particular, our proposed approach continued to find andcertify the globally optimal solution.

The general conclusion in view of our experimental re-sults is that the concrete configuration of a relative poseproblem may have a great impact in the hardness of con-verging to the global solution by traditional means. Yet,

our proposed SDP relaxation-based approach always suc-ceeded, providing the optimal solution together with a cer-tificate of optimality based on duality theory.

7. ConclusionsThis work solves a previously unsatisfyingly solved ge-

ometric problem of fundamental interest: Direct and op-timal computation of the relative pose between two cali-brated images over an arbitrary number of correspondences.Previous solvers either solve only for sub-optimal lineariza-tions, or proceed by computationally less attractive exhaus-tive search strategies such as branch-and-bound. We haveproposed a tailored (non-trivial) formulation of the prob-lem as a Quadratically Constrained Quadratic Program forwhich Shor’s relaxation is tight in 100% of the experimen-tally evaluated problems, allowing us to recover and certifythe globally optimal solution.

We find these results very exciting, and we think thereis still significant space for improvement, both in solvingthe underlying SDP relaxation with specialized solvers thatexploit the low-rank structure of the problem as well as inleveraging these results in different scenarios, such as thedevelopment of computationally attractive Probably Certi-fiably Correct algorithms [2] for the problem at hand.

References[1] K. Anstreicher and H. Wolkowicz. On Lagrangian relax-

ation of quadratic matrix constraints. SIAM Journal on Ma-trix Analysis and Applications, 22(1):41–55, 2000.

[2] A. Bandeira. A note on probably certifiably correct algo-rithms. Comptes Rendus Mathematique, 2016.

[3] A. Bandeira, N. Boumal, and A. Singer. Tightness of themaximum likelihood semidefinite relaxation for angular syn-chronization. Mathematical Programming, 2016.

[4] N. Boumal. A Riemannian low-rank method for optimizationover semidefinite matrices with block-diagonal constraints.arXiv preprint arXiv:1506.00575, 2015.

[5] S. Boyd and L. Vandenberghe. Convex optimization. Cam-bridge University Press, 2004.

[6] J. Briales and J. Gonzalez-Jimenez. Fast Global OptimalityVerification in 3D SLAM. In Proc. Int. Conf. Intell. RobotsSyst. (IROS), Daejeon, 2016. IEEE/RSJ.

[7] J. Briales and J. Gonzalez-Jimenez. Convex Global 3D Reg-istration with Lagrangian Duality. In International Confer-ence on Computer Vision and Pattern Recognition, jul 2017.

[8] J. Briales and J. Gonzalez-Jimenez. Initialization of 3D PoseGraph Optimization using Lagrangian duality. In Int. Conf.on Robotics and Automation (ICRA). IEEE, 2017.

[9] L. Carlone, G. C. Calafiore, C. Tommolillo, and F. Del-laert. Planar Pose Graph Optimization: Duality, OptimalSolutions, and Verification. IEEE Transactions on Robotics,32(3):545–565, 2016.

[10] L. Carlone, D. M. Rosen, G. Calafiore, J. J. Leonard, andF. Dellaert. Lagrangian duality in 3D SLAM: Verificationtechniques and optimal solutions. In Proc. Int. Conf. Intell.Robots Syst. (IROS), pages 125–132, Hamburg, sep 2015.

[11] Y. Ding. On efficient semidefinite relaxations for quadrati-cally constrained quadratic programming. 2007.

[12] M. Grant and S. Boyd. {CVX}: Matlab Soft-ware for Disciplined Convex Programming, version 2.1.\url{http://cvxr.com/cvx}, 2014.

[13] R. Hartley. In Defense of the Eight-Point Algorithm.19(6):580–593, 1997.

[14] R. Hartley and F. Kahl. Optimal Algorithms in MultiviewGeometry. In Computer Vision ACCV 2007, pages 13–34.Springer Berlin Heidelberg, Berlin, Heidelberg.

[15] R. Hartley, J. Trumpf, Y. Dai, and H. Li. Rotation Averaging.International Journal of Computer Vision, 103(3):267–305,jan 2013.

[16] R. Hartley and A. Zisserman. Multiple View Geometry inComputer Vision. Cambridge University Press, New York,NY, USA, 2 edition, 2003.

[17] R. Hartley and A. Zisserman. Multiple View Geometry inComputer Vision. Cambridge University Press, New York,NY, USA, second edition, 2004.

[18] R. I. Hartley and F. Kahl. Global optimization through rota-tion space search. International Journal of Computer Vision,82(1):64–79, 2009.

[19] Y. Khoo and A. Kapoor. Non-iterative rigid 2D/3D point-setregistration using semidefinite programming. IEEE Transac-tions on Image Processing, 25(7):2956–2970, 2016.

[20] L. Kneip, H. Li, and Y. Seo. Upnp: An optimal o (n) solutionto the absolute pose problem with universal applicability.In Computer Vision–ECCV 2014, pages 127–142. Springer,2014.

[21] L. Kneip and S. Lynen. Direct optimization of frame-to-frame rotation. In Computer Vision (ICCV), 2013 IEEE In-ternational Conference on, pages 2352–2359. IEEE, 2013.

[22] L. Kneip, R. Siegwart, and M. Pollefeys. Finding the exactrotation between two images independently of the transla-tion. Springer, 2012.

[23] E. Kruppa. Zur Ermittlung eines Objektes aus zwei Perspek-tiven mit innerer Orientierung. Sitzgsber. Akad. Wien, Math.Naturw. Abt., IIa., 122:1939–1948, 1913.

[24] H. C. Longuet-Higgins. A computer algorithm for re-constructing a scene from two projections. Nature,293(5828):133–135, 1981.

[25] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang.Semidefinite relaxation of quadratic optimization problems.IEEE Signal Processing Magazine, 27(3):20–34, 2010.

[26] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. ORB-SLAM: A Versatile and Accurate Monocular SLAM Sys-tem. IEEE Transactions on Robotics (T-RO), 31(5):1147–1163, 2015.

[27] Y. Nesterov, H. Wolkowicz, and Y. Ye. Semidefinite pro-gramming relaxations of nonconvex quadratic optimization.In Handbook of semidefinite programming, pages 361–419.Springer, 2000.

[28] D. Nister. An efficient solution to the five-point relative poseproblem. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence (PAMI), 26(6):756–777, 2004.

[29] D. Nister and H. Stewenius. Scalable recognition with a vo-cabulary tree. 2006.

[30] C. Olsson and A. Eriksson. Solving quadratically con-strained geometrical problems using lagrangian duality. InPattern Recognition, 2008. ICPR 2008. 19th InternationalConference on, pages 1–5. IEEE, 2008.

[31] C. Olsson, F. Kahl, and M. Oskarsson. Branch-and-BoundMethods for Euclidean Registration Problems. IEEE Trans.Pattern Anal. Mach. Intell., 31(5):783–794, 2009.

[32] P. A. Parrilo and S. Lall. Semidefinite programming relax-ations and algebraic optimization in control. European Jour-nal of Control, 9(2):307–321, 2003.

[33] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard.SE-Sync: A Certifiably Correct Algorithm for Synchro-nization over the Special Euclidean Group. arXiv preprintarXiv:1612.07386, pages 1–49, 2016.

[34] D. M. Rosen, C. DuHadway, and J. J. Leonard. A convexrelaxation for approximate global optimization in simultane-ous localization and mapping. In Proc. Int. Conf. Robot. andAutom. (ICRA), pages 5822–5829, Seattle, 2015. IEEE.

[35] J. P. Ruiz and I. E. Grossmann. Using redundancy tostrengthen the relaxation for the global optimization ofMINLP problems. Computers & Chemical Engineering,35(12):2729–2740, 2011.

[36] J. Saunderson, P. A. Parrilo, and A. S. Willsky. Semidefinitedescriptions of the convex hull of rotation matrices. arXivpreprint arXiv:1403.4914, 2014.

[37] N. Z. Shor. Quadratic optimization problems. Sov. J. Com-put. Syst. Sci., 25(6):1–11, 1987.

[38] H. D. Stewenius, C. Engels, and D. Nister. Recent develop-ments on direct relative orientation. ISPRS Journal of Pho-togrammetry and Remote Sensing, 60(4):284–294, 2006.

[39] J. Sturm and N. Engelhard. A benchmark for the evaluationof RGB-D SLAM systems. Intelligent Robots and Systems(IROS), 2012 IEEE/RSJ International Conference on, 2012.

[40] J. F. Sturm. Using SeDuMi 1.02, a MATLAB toolbox foroptimization over symmetric cones. Optimization methodsand software, 11(1-4):625–653, 1999.

[41] K.-C. Toh, M. J. Todd, and R. H. Tutuncu. SDPT3a MAT-LAB software package for semidefinite programming, ver-sion 1.3. Optim. methods Softw., 11(1-4):545–581, 1999.

[42] B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon.Bundle adjustment - a modern synthesis. In Proceedings ofthe International Workshop on Vision Algorithms: Theoryand Practice (ICCV), pages 298–372, Corfu, Greece, 1999.

[43] R. Tron, D. M. Rosen, and L. Carlone. On the Inclusionof Determinant Constraints in Lagrangian Duality for 3DSLAM. In Proc. Robotics: Science and Systems (RSS),Workshop The Problem of Mobile Sensors, 2015.

[44] L. Vandenberghe and S. Boyd. Semidefinite programming.SIAM review, 38(1):49–95, 1996.

[45] H. Wolkowicz. A note on lack of strong duality for quadraticproblems with orthogonal constraints. European Journal ofOperational Research, 143(2):356–364, 2002.

[46] J. Yang, H. Li, and Y. Jia. Optimal essential matrix esti-mation via inlier-set maximisation. In Proceedings of theEuropean Conference on Computer Vision (ECCV), 2014.