Learning to Optimize Non-Rigid Tracking

Yang Li1,4 Aljaz Bozic4 Tianwei Zhang1 Yanli Ji1,2 Tatsuya Harada1,3 Matthias Nießner4

1The University of Tokyo, 2UESTC, 3RIKEN, 4Technical University Munich

Abstract

One of the widespread solutions for non-rigid trackinghas a nested-loop structure: with Gauss-Newton to mini-mize a tracking objective in the outer loop, and Precon-ditioned Conjugate Gradient (PCG) to solve a sparse lin-ear system in the inner loop. In this paper, we employlearnable optimizations to improve tracking robustness andspeed up solver convergence. First, we upgrade the track-ing objective by integrating an alignment data term on deepfeatures which are learned end-to-end through CNN. Thenew tracking objective can capture the global deformationwhich helps Gauss-Newton to jump over local minimum,leading to robust tracking on large non-rigid motions. Sec-ond, we bridge the gap between the preconditioning tech-nique and learning method by introducing a ConditionNetwhich is trained to generate a preconditioner such that PCGcan converge within a small number of steps. Experimen-tal results indicate that the proposed learning method con-verges faster than the original PCG by a large margin.

1. IntroductionNon-rigid dynamic objects, e.g., humans and animals,

are important targets in both computer vision and roboticsapplications. Their complex geometric shapes and non-rigid surface changes result in challenging problems fortracking and reconstruction. In recent years, using com-modity RGB-D cameras, the seminal works such as Dy-namicFusion [20] and VolumeDeform [13] made their ef-forts to tackle this problem and obtained impressive non-rigid reconstruction results. At the core of DynamicFucionand VolumeDeform are non-linear optimization problems.However, this optimization can be slow, and can also re-sult in undesired local minima. In this paper, we proposea learning-based method that finds optimization steps thatexpand the convergence radius (i.e., avoids local minima)and also makes convergence faster. We test our method onthe essential inter-frame non-rigid tracking task, i.e., to findthe deformation between two RGB-D frames, which is ahigh-dimensional and non-convex problem. The absence

Figure 1. PCG convergence using different preconditioners. Thecurves show the average convergence on the testing dataset. Notethat our final method (green curve) requires 3 times fewer PCGsteps to achieve the same residual (10−6) than the best baseline(dashed line).

of an object template model, large non-overlapping area,and observation noise in both source and target frame makethis problem even more challenging. This section will firstreview the classic approach and then put our contributionsinto context.

Non-rigid Registration The non-rigid surface motionscan be roughly approximated through the “deformationgraph” [24]. In this deformable model, all of the unknowns,i.e., the rotations and translations, are denoted as G. Giventwo RGB-D frames, the goal of non-rigid registration is todetermine the G that minimizes the typical objective func-tion:

minG{Efit(G) + λEreg(G)} (1)

where Efit is the data fitting term that measures the close-ness between the warped source frame and the target frame.Many different data fitting terms have been proposed overthe past decades, such as the geometric point-to-point andpoint-to-plane constraints [16, 31, 20, 13] sparse SIFT de-scriptor correspondences [13], and the dense color term[31], etc. The term Ereg regularizes the problem by favor-ing locally rigid deformation. Coefficient λ balances these

arX

iv:2

003.

1223

0v1

[cs

.CV

] 2

7 M

ar 2

020

two terms. The energy (1) is minimized by iterating theGauss-Newton update step [2] till convergence. Inside eachGauss-Newton update step a large linear system needs tobe solved, for which an iterative preconditioned conjugategradient (PCG) solver is commonly used.

This classic approach cannot properly handle large non-rigid motions since the data fitting term Efit in the energyfunction (1) is made of local constraints (e.g., dense geom-etry or color maps), which only work when they are closeto the global solution, or global constraints that are proneto noise (e.g., sparse descriptor). In the case of large non-rigid motions, these constraints cannot provide convergentresiduals and lead to tracking failure. In this paper, we al-leviate the non-convexity of this problem by introducing adeep feature alignment term into Efit. The deep featuresare extracted through an end-to-end trained CNN. We as-sume that, by leveraging the large receptive field of con-volutional kernels and the nature of the data-driven method,the learned feature can capture the global information whichhelps Gauss-Newton to jump over local minimums.

Preconditioning

Figure 2. Example of using the Deepest Decent to solve a 2D sys-tem. Deepest Decent needs multiple steps to converge on an ill-conditioned system (left) and only one step on a perfectly con-ditioned system (right). Intuitively, Preconditioning is trying tomodify the energy landscape from an elliptical paraboloid into aspherical one such that from any initial position, the direction ofthe first-order derivative directly points to the solution.

As illustrated in Fig. 2, preconditioning speeds up theconvergence of an iterative solver. The general idea behindpreconditioning is to use a matrix, called preconditioner,to modify an ill-posed system into a well-posed one thatis easier to solve. As the hard-coded block-diagonal pre-conditioner was not designed specifically for the non-rigidtracking task, the existing non-rigid tracking solvers are stilltime-consuming. We argue that PCG converges much fasterif the design of the preconditioner involves prior expertknowledge of this specific task. Then we raise the question:does the data-driven method learn a good preconditioner?In this paper, we exploit this idea by training neural networkto generate a preconditioner such that PCG can convergewithin a few steps.

Our contribution is twofold:

• We introduce a deep feature fitting term based on end-to-end learned CNN for the non-rigid tracking prob-lem. Using the proposed data fitting term, the non-rigid tracking Gauss-Newton solver can converge to

the global solution even with large non-rigid motions.

• We propose ConditionNet that learns to generate aproblem-specific preconditioner using a large numberof training samples from the Gauss-Newton updateequation. The learned preconditioner increases PCG’sconvergence speed by a large margin.

2. Related Works2.1. Classic Data-terms for Non-rigid Tracking

The core of non-rigid tracking is to define a data fit-ting term for robust registration. Many different data fit-ting terms have been proposed in the recent geometric ap-proaches, e.g., the point-to-point alignment terms in [16],and the point-to-plane alignment terms in [20, 13]. Besidedense geometric constraints, sparse color image descriptordetection and matching have been used to establish the cor-respondences in [13]. In additions, in [31], the potentialof color consistency assumption was studied. Furthermore,to deal with the lighting change, the reflection consistencetechnique was proposed in [11], and the correspondenceprediction using decision trees was developed in [10].

2.2. Learning based tracking

This line of research focuses on solving motion track-ing tasks from a deep learning perspective. One of thepromising ideas is to replace the hand-engineered descrip-tors with the learned ones. For instance, the Learned Invari-ant Feature Transform (LIFT) is proposed in [28], the volu-metric descriptor for 3D matching is proposed in [29], andthe coplanarity descriptor for plane matching is proposedin [23]. For non-rigid localization/tracking, Schmidt etal. [22] use Fully-Convolutional networks to learn thedense descriptors for upper torso and head of the same per-son; Aljaz et al. [3] proposed a large labeled dataset ofsparse correspondence for general non-rigidly deformingobjects, and a Siamese network based non-rigid 3D patchmatching approach. Regression networks have also beenused to directly map input sensor data to motions, includ-ing the camera pose tracking [30], the dense optical flowtracking [9], and the 3D scene flow estimation [17]. Theproblem of motion regression is that the regressors couldbe overwhelmed by the complexity of the task, therefore,leading to severe over-fitting. A more elegant way is to letthe model focus on a simple task, such as feature extractionwhile using classic optimization tools to solve the rest. Thisresulted in the recent works that combine Gauss-Newtonoptimization and deep learning to learn the most suitablefeatures for image alignment [6], pose registration [12, 19],and multi-frame direct bundle-adjustment [25]. Inspired bythese works, we integrate the entire non-rigid optimizationmethod into the end-to-end pipeline to learn the optimal fea-ture for non-rigid tracking, which requires dealing with or-

ders of magnitude more degree of freedoms than the previ-ous cases. The details are described in Section 3.

2.3. Preconditioning Techniques

Preconditioning as a method of transforming a difficultproblem into one that is easier to solve has centuries ofhistory. Back to 1845, the Jacobi’s Method [5] was firstproposed to improve the convergence of iterative meth-ods. Block-Jacobi is the simplest form of precondition-ing, in which the preconditioner is chosen to be the blockdiagonal of the linear system that we want to solve. De-spite its easy accessibility, we found that applying it showsonly a marginal improvement in our problem. Other meth-ods, such as Incomplete Cholesky Factorization, multiGridmethod [26] or successive over-relaxation [27] method haveshown their effectiveness in many applications. In this pa-per, we exploit the potential of data-driven preconditioner tosolve the linear system in the non-rigid tracking task. Thedetails are shown in Section 4.

3. Learning Deep Non-Rigid Feature3.1. Scene Representation

The input of our method is two frames that are capturedusing a commodity RGB-D sensor. Each frame contains acolor map and a depth map both at the size of 640×480.Calibration was done to ensure that color and depth werealigned in temporal and spatial domain. We denote thesource frame as S, and the target frame as T.

We approximate the surface deformation with the defor-mation graph G. Fig. 3 shows an example of our defor-mation graph. We uniformly sample the image, resulting ina rectangle mesh grid of size w × h. A point in the meshgrid is treated as a node in the deformation graph. Eachnode connects exactly to its 8 neighboring nodes. To filterout the invalid nodes, a binary mask V ∈ Rw×h is con-structed by checking if the node is from the background,holds invalid depth, or lies on occlusion boundaries withlarge depth discontinuity. Similarly, edges are filtered bythe mask E ∈ Rw×h×8 if they link to invalid nodes or gobeyond the edge length threshold. In the deformation graph,the node i is parameterized by a translation vector ti ∈ R3

and a matrix Ri ∈ SO3. Putting all parameters into a singlevector, we get

G = {Ri, ti|i=1,2,··· ,w×h}

3.2. Deep Feature Fitting Term

We use the functionF(·), which is based on fully convo-lutional networks [18], to extract feature map from sourceframe S and target frame T. The encoded feature maps are:

FS = F(S), FT = F(T) (2)

Figure 3. Our deformation graph. Left top: Uniform sampling onthe pixel grid. Let bottom: Binary mask acquired using simpledepth threshold or depth aided human annotation. Right: Masked3D deformation graph.

We apply up-sampling layers in the neural network suchthat the encoded feature map has the size w × h× c, wherec is the dimension of a single feature vector. Thus the fea-ture map and the deformation graph have the same rowsand columns. This means that a feature vector and a graphnode have a one-to-one correspondence (to reduce GPUmemory overhead and speed up the learning). We denoteDS ∈ Rw×h and DT ∈ Rw×h as the sampled depth mapfrom source and target frames. Given the translation vectorsti ∈ G, and the depth value DT(i), the projected feature forthe pixel i can be obtained by

FS(i) = FS(π(ti,DT(i))) (3)

where π(·) : R2 → R2 is the warping function that mapsone pixel coordinate to another pixel coordinate by applyingtranslation ti to a back-projected pixel i, and projecting thetransformed point to the source camera frame. The warpedcoordinate are continuous values. FS(i) is sampled by bi-linearly interpolating the 4 nearest features on the 2D meshgrid. This sampling operation is made differentiable usingthe spatial transformer network defined in [14]. Then thedeep feature fitting term is defined as

Efea(G) = λf

w×h∑i=0

Vi · ||FS(i)− FT(i)||2 (4)

Note that compared to the classic color-consistency con-straints, the learned deep feature captures high-order spatialdeformations in the scans, by leveraging the large receptivefield size of the convolution kernels.

3.3. Total Energy

To resolve the ambiguity inZ axis, we adopt a projectivedepth, which is a rough approximation of the point-to-planeconstraint, as our geometric fitting term. This term mea-sures the difference between warped depth map DS and the

Gauss-Newton update step

Warping

RGB-D

Non-Rigid Feature Extractor

Figure 4. High-level overview of our non-rigid feature extractor training method. Jacobian J’s entries for the feature term (4) can beprecomputed according to the inverse composition algorithm. Other entries in Jacobian J and all entries in residues r are recomputed ineach Gauss-Newton iteration. For simplicity, the geometric fitting term (5) and regularization term (6) are omitted from this figure.

depth map of target frame. It is defined as

Egeo(G) = λg

w×h∑i=0

Vi · ||DS(i)−DT(i)||2 (5)

Finally, we regularize the shape deformation by the ARAPregularization term, which encourages locally rigid mo-tions. It is defined as

Ereg(G) = λr

w×h∑i=0

∑j∈Ni

Ei,j ·||(ti−tj)−Ri(t′

i−t′

j)||2 (6)

Where Ni denotes node-i’s neighboring nodes, and t′

j , t′

j

are the positions of i, j after the transformation. To summa-rize the above, we obtain the following energy for non-rigidtracking:

Etotal(G) = Efea(G) + Egeo(G) + Ereg(G) (7)

The three terms are balanced by [λf , λg, λr]. The total en-ergy is then optimized by the Gauss-Newton update steps:

(JTJ)∆G = JTr (8)

where r is the error residue, and J is the Jacobian of theresidue with respect to G. This equation is further solved bythe iterative PCG solver.

3.4. Back-Propagation Through the Two Solvers

The learning pipeline is shown in Fig. 4. We inte-grate all energy optimization steps into an end-to-end train-ing pipeline. To this end, we need to make both Gauss-Newton and PCG differentiable. In the Gauss-Newton case,the update steps stop when a specified threshold is reached.Such if-else based termination criteria prevents error back-propagation. We apply the same solution as in [19, 25, 12],i.e., we fix the number of Gauss-Newton iterations. In this

project, we set this number to a small digit. There aretwo reasons behind this: 1) For the recursive nature of theGauss-Newton layer, large iterations number will induce in-stability to the network training, 2) By limiting the availablestep, the feature extractor is pushed to produce the featuresthat allow Gauss-Newton solver to make bigger jumps to-ward the solution. Thus we can achieve faster convergenceand robust solving.

Back-propagation through PCG can is done in a differ-ent fashion as described in [1]. Equation (8) need to besolved in every Gauss-Newton iteration. Let’s representJTJ by A, ∆G by x, and JTr by b, then we get the fol-lowing iconic equation:

Ax = b (9)

Suppose that we have already got the gradient of loss L w.r.tto the solution x as ∂L/∂x. We want to back propagate thatquantity onto A and b:

∂L∂b

= A−1∂L∂x

(10)

∂L∂A

= (−A−1 ∂L∂x

)(A−1b)T = −∂L∂b

xT (11)

which means that back-propagating through the linear sys-tem only need another PCG solve for Equation (10).

3.5. Training Objective & Data Acquisition

The method outputs the final deformation graph after afew Gauss-Newton iterations. We apply the L1 flow loss onall the translation vectors ti ∈ G in the deformation graph

Lflow =∑ti∈G|ti − ti,gt| (12)

where ti,gt ∈ R3 is node-i’s ground truth 3D translationvector, i.e., the scene flow.

Figure 5. Using our non-rigid tracking and reconstruction methodto obtain point-point correspondence. This method can generateaccurate correspondence when the motion is small. The long termcorrespondence between distant frames can be obtained by accu-mulating small inter-frame motions through time and space.

Collecting ti,gt is a non-trivial task. Inspired by Zenget al. [29] and Schmidt et al. [22], we realize that the 3Dcorrespondence ground truth can be achieved by running thestate-of-the-art tracking and reconstruction methods such asBundleFusion [8], for rigid scenes, or DynamicFusion [20]/VolumeDeform [13], for non-rigidly deforming scenes. Forthe rigid training set, we turn to the ScanNet, which con-tains a large number of indoor sequences with BundleFu-sion based camera trajectory. For the non-rigid trainingdataset, as shown in Fig. 5, we run our geometry based non-rigid reconstruction method (which is similar to Dynamic-Fusion [20]) on the collected non-rigid sequences. We ar-gue that non-rigid feature learning could benefit from rigidscenes. Since the rigid scenes can be considered as a subsetof the non-rigid ones, the domain gap is not that huge whenwe approximate the rigid object surface from a deformableperspective. Eventually, the feature learning pipeline is pre-trained on ScanNet and fine-tuned on our non-rigid dataset.

4. Data-Driven PreconditionerPreconditioner M−1 modifies the system Ax = b to

M−1Ax = M−1b (13)

which is easier to solve. From the iterative optimizationperspective, solving (13) is equal to finding the x that min-imizes the quadratic form

minx||M−1Ax−M−1b||2 (14)

Here, we propose the ConditionNet C(·) based on neuralnetworks with an encoder and decoder structure to do themapping:

C(·) : Rn×n → Rn×n : A→M−1

A good preconditioner should be a symmetric positive defi-nite (SPD) matrix, otherwise, the PCG can not guarantee toconverge. To this end, the ConditionNet first generates thelower triangle matrix L. Then the preconditioner M−1 iscomputed as

M−1 = LLT (15)

Empirically, we apply a hard positive threshold on M−1’sdiagonal entries to combat the situations that there exist zerosingular values. By doing this, M−1 is ensured to be anSPD matrix in our case.

The matrix density, i.e., the ratio of non-zero entries,play an important role in preconditioning. On one end, adenser preconditioner has a higher potential to approximateA−1, which is the perfect conditioner, but the matrix in-verse itself is time-consuming. On the other end, a sparsermatrix is cheaper to achieve while leading to a poor pre-conditioning effect. To examine the trade-off between ef-ficiency and effectiveness, we propose the following threeConditionNet variants. They use the same network structurebut generate preconditioners with different density, fromdense to sparse.

ConditionNet-Dense. As shown in Fig. 6, this oneuses full matrix A as input and generate the dense precon-ditioner, in which all entries can be non-zero. Intuitively,this model is trying to approximate the perfect conditionerA−1.

ConditionNet-Sparse. This one inputs full matrix A.For the output, a binary mask is applied such that any entryin L is set to zero if the corresponding entry in A is alsozero.

ConditionNet-Diagonal. The input and output are theblock diagonals of the matrices. There are w × h diagonalblocks and each block is 6 × 6. Since each block is di-rectly related to a feature in the 2D mesh grid, we reshapethe input block diagonal entries to a [w, h, 36] volume toleverage such 2D spatial correlations. The output volumeis [w, h, 21] for the lower triangle matrix L. This modelgenerates the sparsest preconditioner.

4.1. Self-Supervised Training

The straight forward way to train the ConditionNet is tominimize the condition number κ(M−1A) = λmax/λmin

i.e., the ratio of the maximum and minimum singular valuein M−1A. However, the time consuming singular value de-composition (SVD) makes large scale training impractical.

Instead, we propose the PCG-Loss for training. Asshown in Fig. 6 the learned preconditioner M−1 is fed toa PCG layer to minimize (14) and output the solution x.Training data generation for the ConditionNet is fully au-tomatic; i.e., no annotations are needed to find the groundtruth solution xgt to equation Ax = b, which we do byrunning a standard PCG solver. To obtain xgt, the standardPCG is executed as many iterations as possible till conver-gence. Then the L1 PCG-Loss is applied on the predictedsolution

Lpcg = |x− xgt| (16)

The training samples, i.e., the [A, b] pairs, are collectedfrom the Gauss-Newton update step in Eqn. (8).

PCG LayerConditionNet

Figure 6. Overview of ConditionNet-Dense. The output L is the lower triangle matrix of the preconditioner. After a few iterations in thePCG layer, the solution x is then penalized by the L1 loss (16). The whole pipeline can be trained end-to-end.

MethodEnergy Terms ScanNet (SN) Non-Rigid Dataset (NR)

3D EPE (cm) on ↓ validation/teston frame jump

3D EPE (cm) on ↓ validation/teston frame jump

AR

AP

P2Pl

ane

P2Po

int

Col

or

Feat

ure

0→2 0→4 0→8 0→16 0→2 0→4 0→8 0→16

N-ICP-0 X X 3.21/3.65 5.03/5.68 8.17/9.66 15.35/18.65 2.29/2.3 4.08/3.3 7.71/6.3 13.63/12.72

N-ICP-1 [20] X X X 2.43/2.75 4.50/5.38 8.11/9.09 14.62/17.10 1.49/1.53 2.98/2.71 6.61/6.52 11.14/12.07

N-ICP-2 [31] X X X X 2.04/2.71 3.58/4.47 6.07/7.89 10.36/14.96 1.68/1.70 3.41/2.60 5.50/5.20 11.80/10.59

Ours (SN) X X X 2.10/2.60 3.55/4.39 5.28/6.98 7.34/10.59 1.73/1.60 2.77/2.63 4.99/5.08 7.09/8.32

Ours (SN+NR) X X X – – – – 1.55/1.34 2.25/2.23 4.16/4.50 6.47/7.59Table 1. 3D End point Error (EPE) on ScanNet and our Non-Rigid dataset. The frame jumps shows the index of the indices of the sourceand target frame. The number of unkonws in the deformation graph is 1152 (16× 12× 6). Ours (SN): trained on ScanNet [7]. Ours (SN+ NR): pretrained on ScanNet and fine-tuned on the Non-Rigid dataset [3].

During the training phase, we limit the number of avail-able iterations in the PCG layer. This is to encouragethe ConditionNet to generate a better preconditioner thatachieves the same solution while using fewer steps. At theearly phase of the training, the PCG layer with limited iter-ations does not guarantee a good convergence. The back-propagation strategy described in Section 3.4 can not beapplied here, because incomplete solving results in wronggradient. Instead, we directly flow the gradient through allPCG iterations for ConditionNet training.

We train ConditionNet and the non-rigid feature extrac-tor separately. They are used together at the testing phase.

5. Experiments

Implementation Details: The resolution of the deforma-tion graph is 16 × 12. Empirically, the weighting factor[λf , λg, λr] in the energy function (7) are set to [1, 0.5, 40].The number of Gauss-Newton iterations is 3 for non-rigidfeature extractor training. The number of PCG iterationsis 10 for ConditionNet Training. We implement our net-works using the publicly available Pytorch framework andtrain it with Tesla P100 GPUs. We trained all the modelsfrom scratch for 30 epochs,with a mini-batch size of 4 us-ing Adam [15] optimizer, where β1 = 0.9, β1 = 0.999. Weused an initial learning rate of 0.0001 and halve it every 1/5of the total iterations.

5.1. Datasets

ScanNet ScanNet [7] is a large-scale RGBD video datasetcontaining 1,513 sequences in 706 different scenes. Thesequences are captured by iPad Mounted RGBD sensorsthat provide calibrated depth-color pairs of VGA resolution.The 3D camera poses are based on BundleFusion [8]. The3D dense motion ground truth on the ScanNet is obtained byprojecting point cloud via depth and 6-Dof camera pose. Weapply the following filtering process for training data. Tonarrow the domain gap with the non-rigid dataset, we filterout images if more than 50% of the pixels have the invaliddepth or depth values larger than 2 meters. To avoid imagepairs with large pose error, we filter image pairs with a largephoto-consistency error. Finally, we remove the image pairswith less than 50% “covisibility”, i.e., the percentage of thepixels that are visible from both images. Similarly, the se-quences are subsampled using the intervals [2, 4, 8, 16]. Weuse 60k frame pairs in total and split train/valid/test as 8/1/1.

Non-Rigid Dataset We use the non-rigid dataset fromAljaz et al. [3] which consists of 400 non-rigidly deform-ing scenes, over 390,000 RGB-D frames. A variety of de-formable objects are captured including adults, children,bags, clothes, and animals, etc. The distance of the objectsto the camera center lies in the range [0.5m, 2.0m]. De-pending on the complexity of the scene, the foreground ob-ject masks are either obtained by a simple depth thresholdor depth map aided human annotation. We run our track-ing and reconstruction method to obtain the ground truth

Mesh1 andMasked Color Image Initial Alignment2

Final Alignment,Transformed Source Mesh, and Alignment Error3

Source Target N-ICP-1 [20] N-ICP-2 [31] Ours

bear

pillo

wad

ult

clot

hes

Figure 7. Frame-Frame tracking results. 1 Meshes are constructed from depth images. Depth images are preprocessed by the bilateral filterto reduce observation noise. 2 Initial alignment is done by simply setting the camera poses of both frames to identity. 3 The alignmenterror (hotter means larger) measures the point to point distance between target mesh and the transformed source mesh.

non-rigid motions. We remove the drifted sequences bymanually checking the tracking quality of the reconstructedmodel. The example of this dataset can be found in thepaper [3]. Similarly to the rigid case, we sub-sample thesequences using the frame jumps [2, 4, 8, 16] to simulatethe different magnitude of non-rigid deformation. For data-augmentation, we perform horizontal flips, random gamma,brightness, and color shifts for input frame pairs. Finally,we got 8.5k frame pairs in total and split train/valid/test as8/1/1.

5.2. Non-Rigid Tracking Evaluation

Baselines We implement a few variants of the non-rigidICP (N-ICP) methods. They apply different energy termsas shown in Tab. 1. Among them, N-ICP-1 is our imple-mentation of the method DynamicFusion [20], and N-ICP-2 is our implementation for the method described in [31].The original two papers are focusing on the model to frametracking problems where the model is either reconstructedon-the-fly or pre-defined. Here all baselines are deployed

for the frame-frame tracking problem. Ours first optimizesthe feature fitting term based objective (7) to get the coarsemotion and then refine the graph with the classic point-to-plane constraints using the raw depth maps.

Quantitative Results The quantitative results on theScanNet dataset and the non-rigid dataset can be found inTable 1. The estimated motions are evaluated using the3D End-Point-Error (EPE) metric. On ScanNet, Ours(SN)achieves overall better performance than the other N-ICPbaselines, especially when the motions are large (e.g., on0→8 and 0→16 frame jump). Note that the ScanNet pre-trained model Ours(SN) even achieves better results thanthe classic N-ICPs on the non-rigid dataset, indicating agood generalization ability of the learned non-rigid fea-ture, which makes sense considering that the learnable CNNmodel focuses only on the feature extraction part, and theusing of classic optimizer disentangle the direct mappingfrom images to motion. It also proves the assumptionthat the rigid and non-rigid surfaces lie in quite close do-mains. The fine-tuned model Ours(SN+NR) on the non-rigid dataset further improved these numbers.

Qualitative Results Fig. 7 shows the frame-frame track-ing results on the non-rigid frame pairs. We selected theframe pairs with relatively large non-rigid motions. N-ICP-1 and N-ICP-2 have trouble dealing with these motions andconverged to bad local minimums. Our method managesto converge to the global solutions on these challengingcases. For instance, the clothes scene in Fig. 7 is an es-pecially challenging case for classic non-rigid ICP methodsbecause the point-to-plane term has no chance to slide overthe zigzag clothes surface which contains multiple folds,and the color consistency term could also be easily con-fused by the repetitive camouflage textures of the clothes.The learned features show an advantage for capturing highorder deformation on those cases.

5.3. Preconditioning Results

We randomly collected 10K [A, b] pairs from differ-ent iterations the Gauss-Newton step. We split them totrain/valid/test according to the ratio of 8/1/1. We com-pare with 3 PCG baselines: w/o preconditioner, the stan-dard block-diagonal preconditioner, and the IncompleteCholesky factorization based preconditioner. We also showthe ablation studies on three of the ConditionNet variants:Diagonal, Sparse and Dense. Fig. 1 shows the PCG stepsusing different preconditioners. The learned preconditioneroutperforms the classic ones by a large margin. Tab. 2shows PCGs solving results using different precondition-ers. All learned preconditioners significantly reduced thecondition numbers. ConditionNet-Dense achieves the bestconvergence rate and the least overall solving time.

Preconditioner density κ iters time (ms)None – 3442.18 46 33.43Block-Diagonal 0.46% 541.52 44 31.34Incomplete Cholesky 1.52 379.82 37 28.42ConditionNet-Diagonal (ours) 0.46% 93.55 21 12.38ConditionNet-Sparse (ours) 1.52% 125.81 23 17.80ConditionNet-Dense (ours) 100.% 34.90 13 10.32

Table 2. PCG solving results using different preconditioners(residue threshold of convergence: 10−6). density: density of pre-conditioner. κ: condition number of the modified linear system.iters: total steps for convergence. time(ms): time of solving. Allnumbers are obtained with Pytorch-GPU implementation.

6. Conclusion and DiscussionIn this work, we present an end-to-end learning ap-

proach for non-rigid RGB-D tracking. Our core contribu-tion is the learnable optimization approach which improvesboth robustness and convergence by a significant margin.The experimental results show that the learned non-rigidfeature significantly improves the convergence of Gauss-Newton solver for the frame-frame non-rigid tracking. Inaddition, our method increases the PCG solver’s conver-gence rate by predicting a good preconditionier. Overall,the learned preconditioner requires 2 to 3 times fewer itera-tions until convergence.

While we believe this results are very promising andcan lead to significant practical improvements in non-rigidtracking and reconstruction frameworks, there are severalmajor challenges are yet to be addressed: 1) The proposednon-rigid feature extractor adopted plain 2D convolutionkernels, which are potentially not the best option to han-dle 3D scene occlusions. One possible research avenue isto directly extract non-rigid features from 3D point cloudsor mesh structures using the point-based architectures [21],or even graph convolutions [4]. 2) Collecting dense sceneflow using DynamicFusion for real-world RGB-D video se-quence is expensive (i.e., segmentation and outlier removalcan become painful processes). The potential solution islearning on synthetic datasets. (e.g., using graphics simula-tions where the dense motion ground truth is available).

AcknowledgementsThis work was also supported by a TUM-IAS Rudolf

Moßbauer Fellowship, the ERC Starting Grant Scan2CAD(804724), and the German Research Foundation (DFG)Grant Making Machine Learning on Static and Dynamic 3DData Practical, as well as the JST CREST Grant NumberJPMJCR1403, and partially supported by JSPS KAKENHIGrant Number JP19H01115. YL was supported by theErasmus+ grant during his stay at TUM. We thank Christo-pher Choy, Atsuhiro Noguchi, Shunya Wakasugi, and KoheiUehara for helpful discussion.

References[1] Jonathan T Barron and Ben Poole. The fast bilateral solver.

In ECCV, pages 617–632. Springer, 2016. 4[2] Ake Bjorck. Numerical methods for least squares problems.

SIAM, 1996. 2[3] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and

Matthias Nießner. Deepdeform: Learning non-rigid rgb-dreconstruction with semi-supervised data. In CVPR, 2020.2, 6, 7

[4] Michael M Bronstein, Joan Bruna, Yann LeCun, ArthurSzlam, and Pierre Vandergheynst. Geometric deep learning:going beyond euclidean data. IEEE Signal Processing Mag-azine, 34(4):18–42, 2017. 8

[5] Jacobi Carl Gustav Jacob. Uber eine neue auosungsart derbei der methode der kleinsten quadrate vorkommenden lin-earen gleichungen. In Nachrichten, pages 22, 297, 1845. 3

[6] Che-Han Chang, Chun-Nan Chou, and Edward Y Chang.Clkn: Cascaded lucas-kanade networks for image alignment.In CVPR, pages 2213–2221, 2017. 2

[7] Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal-ber, Thomas Funkhouser, and Matthias Nießner. Scannet:Richly-annotated 3d reconstructions of indoor scenes. InCVPR, pages 5828–5839, 2017. 6

[8] Angela Dai, Matthias Nießner, Michael Zollhofer, ShahramIzadi, and Christian Theobalt. Bundlefusion: Real-timeglobally consistent 3d reconstruction using on-the-fly sur-face reintegration. ACM Transactions on Graphics (ToG),36(3):24, 2017. 5, 6

[9] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, PhilipHausser, Caner Hazirbas, Vladimir Golkov, Patrick VanDer Smagt, Daniel Cremers, and Thomas Brox. Flownet:Learning optical flow with convolutional networks. In ICCV,pages 2758–2766, 2015. 2

[10] Mingsong Dou, Sameh Khamis, Yury Degtyarev, PhilipDavidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio OrtsEscolano, Christoph Rhemann, David Kim, Jonathan Tay-lor, et al. Fusion4d: Real-time performance capture of chal-lenging scenes. ACM Transactions on Graphics (TOG),35(4):114, 2016. 2

[11] Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai,and Yebin Liu. Real-time geometry, albedo, and motion re-construction using a single rgb-d camera. ACM Transactionson Graphics (TOG), 36(3):32, 2017. 2

[12] Lei Han, Mengqi Ji, Lu Fang, and Matthias Nießner. Reg-net: Learning the optimization of direct image-to-image poseregistration. arXiv preprint arXiv:1812.10212, 2018. 2, 4

[13] Matthias Innmann, Michael Zollhofer, Matthias Nießner,Christian Theobalt, and Marc Stamminger. Volumedeform:Real-time volumetric non-rigid reconstruction. In ECCV,pages 362–379. Springer, 2016. 1, 2, 5

[14] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.Spatial transformer networks. In Advances in neural infor-mation processing systems, pages 2017–2025, 2015. 3

[15] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,2014. 6

[16] Hao Li, Robert W Sumner, and Mark Pauly. Global cor-respondence optimization for non-rigid registration of depthscans. In Computer graphics forum, volume 27, pages 1421–1430. Wiley Online Library, 2008. 1, 2

[17] Xingyu Liu, Charles R Qi, and Leonidas J Guibas.Flownet3d: Learning scene flow in 3d point clouds. InCVPR, pages 529–537, 2019. 2

[18] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fullyconvolutional networks for semantic segmentation. In Pro-ceedings of the IEEE conference on computer vision and pat-tern recognition, pages 3431–3440, 2015. 3

[19] Zhaoyang Lv, Frank Dellaert, James M Rehg, and AndreasGeiger. Taking a deeper look at the inverse compositionalalgorithm. In CVPR, pages 4581–4590, 2019. 2, 4

[20] Richard A Newcombe, Dieter Fox, and Steven M Seitz.Dynamicfusion: Reconstruction and tracking of non-rigidscenes in real-time. In CVPR, pages 343–352, 2015. 1, 2,5, 6, 7

[21] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas JGuibas. Pointnet++: Deep hierarchical feature learning onpoint sets in a metric space. In Advances in neural informa-tion processing systems, pages 5099–5108, 2017. 8

[22] Tanner Schmidt, Richard Newcombe, and Dieter Fox. Self-supervised visual descriptor learning for dense correspon-dence. IEEE Robotics and Automation Letters, 2(2):420–427, 2016. 2, 5

[23] Yifei Shi, Kai Xu, Matthias Nießner, Szymon Rusinkiewicz,and Thomas Funkhouser. Planematch: Patch coplanarity pre-diction for robust rgb-d reconstruction. In ECCV, pages 750–766, 2018. 2

[24] Robert W Sumner, Johannes Schmid, and Mark Pauly. Em-bedded deformation for shape manipulation. ACM Transac-tions on Graphics (TOG), 26(3):80, 2007. 1

[25] Chengzhou Tang and Ping Tan. Ba-net: Dense bundle ad-justment network. arXiv preprint arXiv:1806.04807, 2018.2, 4

[26] Osamu Tatebe. The multigrid preconditioned conjugate gra-dient method. 1993. 3

[27] Henk A van der Vorst. High performance precondition-ing. SIAM Journal on Scientific and Statistical Computing,10(6):1174–1185, 1989. 3

[28] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and PascalFua. Lift: Learned invariant feature transform. In ECCV,pages 467–483. Springer, 2016. 2

[29] Andy Zeng, Shuran Song, Matthias Nießner, MatthewFisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch:Learning local geometric descriptors from rgb-d reconstruc-tions. In CVPR, pages 1802–1811, 2017. 2, 5

[30] Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox.Deeptam: Deep tracking and mapping. In ECCV, pages 822–838, 2018. 2

[31] Michael Zollhofer, Matthias Nießner, Shahram Izadi,Christoph Rehmann, Christopher Zach, Matthew Fisher,Chenglei Wu, Andrew Fitzgibbon, Charles Loop, ChristianTheobalt, et al. Real-time non-rigid reconstruction usingan rgb-d camera. ACM Transactions on Graphics (ToG),33(4):156, 2014. 1, 2, 6, 7