Improving Robustness for Pose Estimation via Stable Heatmap Regression

Yumeng ZhangSchool of Software, BNRist,

Tsinghua University

Beijing, China

Li Chen *

School of Software, BNRist,

Tsinghua University

Beijing, China

Yufeng LiuSEU-ALLEN Joint Center,

Southeast University

Nanjing, China

Xiaoyan GuoY-tech,

Kuaishou Technology

Beijing, China

Wen ZhengY-tech,

Kuaishou Technology

Beijing, China

Junhai YongSchool of Software, BNRist,

Tsinghua University

Beijing, China

Abstract

Deep learning methods have achieved excellent perfor-mance in pose estimation, but the lack of robustness causesthe keypoints to change drastically between similar images.In view of this problem, a stable heatmap regression methodis proposed to alleviate network vulnerability to small per-turbations. We utilize the correlation between different rowsand columns in a heatmap to alleviate the multi-peaks prob-lem, and design a highly differentiated heatmap regressionto make a keypoint discriminative from surrounding points.A maximum stability training loss is used to simplify theoptimization difficulty when minimizing the prediction gapof two similar images. The proposed method achieves asignificant advance in robustness over state-of-the-art ap-proaches on two benchmark datasets and maintains highperformance.

1. IntroductionWith the continuous development of deep learning, the

performance of pose estimation has been improved in un-precedented ways, thus increasing its wide application inhuman-computer interaction, virtual reality and many otherfields. However, existing pose estimation models can beextremely vulnerable to small perturbations despite theirexcellent performance. These small perturbations do notchange the semantic information of an image, but theycan greatly alter the output of the networks. For example,the predictions of keypoints change considerably in simi-lar continuous frames, as illustrated in Fig 1. This problem

*Corresponding author: Li Chen

Traditionalheatmapregression

Stableheatmapregression

Figure 1. The predicted keypoints of traditional heatmap regres-sion and those of the proposed stable heatmap regression on sevensimilar images. The translucent white parts represent areas of highresponses on the heatmap and the points are the predicted key-points. These images are almost same for humans, whereas thetraditional method produces unstable predictions (yellow points).The proposed stable heatmap regression is going to alleviate thisproblem.

greatly affects the user experience. Therefore, improvingthe robustness of a pose estimation network is an urgent re-search task.

Improving the robustness of a network is challengingdue to the black-box nature of deep learning methods. Atpresent, few papers have analyzed the stability of pose es-timation task. Most of the existing studies for improvingrobustness focus on a relatively simple task-image classifi-cation, such as famous adversarial examples [5], and a largenumber of robustness-enhancing methods [2, 14, 19, 25, 29]have emerged in classification to improve the robustness ofnetworks against small perturbations.

However, most of these robustness-enhancing methodsare unsuitable for pose estimation. For example, Mixup[27] and Cutmix [26] improved the robustness by per-forming data augmentations, but these methods may causeocclusions of keypoints, resulting in meaningless training

1

arX

iv:2

105.

0356

9v1

[cs

.CV

] 8

May

202

1

samples. Some other robustness-enhancing strategies, al-though theoretically applicable to different tasks, have lim-ited or even negative effects on robustness in pose estima-tion task because of their characteristic. For example, lim-iting the Lipschitz of networks [21] or using the largest kvalues activation function [3] significantly improve robust-ness against adversarial attacks and scale to pose estimationtheoretically. However, these approaches limit the expres-sive power of networks while pose estimation models relyon rich global and local features to determine the positionof keypoints. Thus, when these methods are applied to poseestimation task, the network fails to converge or the perfor-mance becomes unsatisfactory. To the best of our knowl-edge, few methods have been proposed to analyze or im-prove the robustness of the pose estimation models. Thecurrent study is going to fill this gap.

Initially, we analyze the instability factors of pose esti-mation task, and then design corresponding modules to im-prove the robustness. The robustness analysis is based onthe heatmap regression, a commonly used method in poseestimation. The heatmap is generally the same size as theinput image and each value represents the probability of thecorresponding point to belong to a keypoint in the input im-age. The point with the highest probability is often identi-fied as the predicted keypoint. Many methods have achievedhigh performance with this strategy, but several robustnessproblems were ignored. We believe that the networks arenot robust with the current heatmap regression methods be-cause of three reasons.

The first is multi-peaks predictions. When predictingkeypoints of some difficult samples, similar high responsesmay appear in different regions of the heatmap. Thus, aslight change in the input may cause the predicted keypointto shift from one peak to another. This problem has beenillustrated in the two rightmost images in the first row ofFig. 1.

The second is the use of Gaussian heatmap as groundtruth. Current ground-truth heatmap is obtained by a Gaus-sian distribution. The maximum and second-largest valueof an output heatmap are too close in value due to the useof this Gaussian heatmap. Thus, small perturbations in theoutput may cause the change of predicted keypoints.

The third is the optimization difficulty when combin-ing heatmap regression with existing stability training(ST) methods. Many ST methods [10, 29, 31] improve ro-bustness by minimizing the distance between the output ofa clean image and its perturbed image. However, the outputprobabilities of a heatmap are different from those in theclassification. The output heatmap represents the probabil-ity that each pixel in the image belongs to a keypoint, andeach pixel of a clean image may be changed in its perturbedimage. Forcing the probabilities of all the pixels betweenthese image pairs to be the same increases difficulty in net-

work optimization.In view of these problems, we propose a stable heatmap

regression, an alternative to traditional heatmap regression,with three novel robustness-enhancing designs.

1) A Row-Column Correlation (RCC) layer to allevi-ate the multi-peaks problem. This layer utilizes thecorrelation between different rows and columns in theheatmap to detect the multi-peaks problem. Then ithelps the networks output single-peak heatmap by in-creasing the gap between different peaks and raisingthe loss of multi-peaks heatmap.

2) A Highly Differentiated Heatmap Regression(HDHR) method to make a keypoint prediction dis-criminative from surrounding points. We replace theGaussian heatmap with a multi-label map and a well-designed highly differentiated heatmap. Then we de-sign a weighted cross-entropy loss based on them toenlarge the difference between maximum and the sec-ond largest value in the heatmap.

3) A Maximum Stability Training (MST) loss to sim-plify the optimization difficulty when minimizing thegap between the outputs of two similar images. Thisloss focuses on changes of the position with the largestvalue in the heatmap. And we provide a theoreticalcertification to demonstrate the effectiveness of MST.

The proposed method is evaluated on two datasets withsix different model architectures. The experimental resultsdemonstrate that our method enhances the robustness ofpose estimation models and achieves a significant advanceover existing methods for improving robustness.

2. Related worksVery few works in the literature have explored ways to

improve the robustness for pose estimation. The task, how-ever, shares several properties with adversarial examples[5, 4] and natural corruptions [7, 16]. Some approaches thathave been proposed for these fields can be adapted for poseestimation. Hence, in the following section, we discuss therelated works for improving the robustness of networks ingeneral.

Stability training Stability training (ST) [9, 31] is an ef-fective strategy to improve robustness. Zheng et al. [31]first proposed a regularization loss to make the features oroutputs of a clean image and its perturbed image consis-tent. Li et al. [11] further provided a theoretical evidencefor Zheng’s loss by analyzing the upper bound on the tol-erable size of perturbations with Renyi divergence [22] andGaussian noise. Subsequently, a variant of Zheng’s [31]loss is proposed by Zhang et al. [29]. They replaced the L2metric with the KL-divergence metric on the basis of their

2

Weight Multi-Label

Gaussian

!"#$

!%&'

RCC

(')+,-

.')/,0123

((, .)Multi-peaks

heatmap

DRCC

perturbations set

Figure 2. Schematic of our stable heatmap regression algorithm. The proposed method takes a clean image and its perturbed image asinput. The perturbed image is selected randomly from a pre-defined perturbation set. A Row-Column Correlation (RCC) layer is designedto alleviate the multi-peaks problem. Traditional L2 or L1 loss on Gaussian heatmap are replaced with weighted cross-entropy loss Lwce

on a multi-label map, where the highly differentiated heatmap is used as the weight of the multi-label map. The difference (D) between theoutputs of two input images is used to calculate the Maximum Stability Training loss Lmst.

theoretical analysis. More recent studies by Hendrycks etal. [8] and Lopes et al. [12] focused on the constructionof the perturbation set. Hendrycks et al. [8] obtained moregeneral perturbations by combining different augmentationmethods and then applied Zheng’s loss [31] on these pertur-bations to improve robustness. Lopes et al. [12] proposed apatch gaussian augmentation method to improve robustnessand retain high performance on clean images. ST meth-ods can be easily adapted to different tasks. However, task-specific modifications should be considered to further im-prove the robustness on a certain task.

Network robustness analysis Many methods have beenproposed to identify the component that causes instabilitythrough the analysis of the network property or architec-ture. For example, Tsuzuku et al. [20] and Usama et al.[21] have found that networks with smaller Lipschitz con-stant are more robust. Then, they proposed correspondingstrategies to limit this constant. However, the Lipschitz con-stant of networks is related to the expressive power of them.Limiting Lipschitz degrades the performance on complextasks or hard datasets. Meanwhile, Azulay et al. [1] dis-covered that the downsampling operation caused the lackof translation invariant in convolutional neural networks(CNN). However, downsampling cannot be easily removedfrom CNN models, because the shape of the output is notalways consistent with that of the input. Hendrycks et al.[7] pointed out that the tricks, such as multiscale architec-tures and larger feature aggregating models, could improvethe robustness, whereas these tricks were obtained by ex-periments without considering the underlying causes of theinstabilities.

(a) Variance Case (b) Center Case (c) Equal Case

Figure 3. The role of the proposed RCC layer in reducing multi-peaks problem. Cases in (a) and (b) illustrate that RCC layer tendsto attenuate peaks with small variance and far from the center ofthe heatmap. Case (c) shows a rare situation that C1 and C2 (de-fined by Eq. (6)) are equal. In this case, the RCC layer generatestwo peaks on the diagonal. The two peaks are usually wrong andresult in a big loss. Thus, the network could realize that the single-peak output is better as the loss decreases.

To conclude, the improvement of robustness in pose esti-mation is still an underexplored problem. Our work is goingto fill this gap.

3. Method

The proposed stable heatmap regression is shown in Fig.2. It consists of three components, row-column correlationlayer, highly differentiated heatmap regression and maxi-mum stability training. The three modules are used to al-leviate the multi-peaks problem, enlarge the difference be-tween maximum and the second-largest value of a heatmap,and limit the prediction gap between a clean image and itsperturbed image respectively. Next, we will introduce thethree modules in detail.

3

3.1. Row-column correlation layer

Row-Column Correlation (RCC) layer is designed toalleviate the multi-peaks phenomenon in the predictedheatmaps. The layer can detect the multi-peaks problem andalleviate it by increasing the gap between different peaksand raising the loss of multi-peaks heatmap. In this sec-tion, we first introduce the operation of this layer and thenprovide a theoretical analysis to explain its behavior. Theoperation of RCC is shown in (1).

RCC(l,m) =

W∑i=1

HT (l, i)HT (i,m) (1)

where RCC(l,m) is the output of Row-Column Correla-tion layer at lth row and mth column, HT is the inputheatmap of RCC and W is the width of HT . Throughthis definition, the value at (l,m) of RCC represents thecorrelation between lth row and mth column of the inputheatmap. RCC layer requires that the input heatmap has thesame height and width, and this can be easily achieved bycropping or padding.

A theoretical analysis of RCC layer is provided basedon a N-peak Gaussian heatmap to demonstrate the elegantproperties of RCC. The N-peak heatmap is defined as thesum of N independent Gaussian sub-heatmaps, where thecenter and variance of nth Gaussian heatmap are (krn, kcn)and σn. The value of (i, j) position in the nth Gaussianheatmap Gn is shown in formula (2).

Gn(i, j) = e− (i−krn)2+(j−kcn)2

2σ2n (2)

Then the proposed RCC layer has following properties.

Theorem 1 If N=1, output of RCC layer is C1 times the

input heatmap, where C1 =∑W

i=1 e− (i−kc1)2+(i−kr1)2

2σ21 , W iswidth of the heatmap.

Theorem 2 If N > 1, output of RCC layer is

RCC(l,m) =∑a6=b

Ca,bGa,b(l,m)+

N∑n=1

CnGn(l,m) (3)

where a, b ∈ {1, 2, ..., N} and

Ga,b(l,m) = e−σ

2b (l−kra)2+σ2a(m−kcb)

2

2σ2aσ2b (4)

Ca,b =

W∑i=1

e−σ

2b (i−kca)2+σ2a(i−krb)

2

2σ2aσ2b (5)

Cn =

W∑i=1

e− (i−kcn)2+(i−krn)2

2σ2n (6)

Theorem 1 indicates that the output of RCC can be samewith the input Gaussian heatmap in single-peak situationby normalizing the output to [0,1]. Thus, the RCC layerdoes not affect the results of single-peak predictions. Whenthere are multi-peaks in the heatmap, Theorem 2 showsthat all the Gaussian including their joint Gaussian sub-heatmaps Ga,b(l,m) are weighted, thereby enlarging thedifference between these peaks. Generally speaking, thoseGaussian sub-heatmaps with a small σ and a (krb, kca) or(krn, kcn) far from the center of the heatmap are assigneda low weight, as illustrated in (a), (b) of Fig. 3. Thereare cases that the weights of some Gn are equal. However,Ca,b is often assigned a higher weight in this case, and thepeaks of these joint sub-heatmaps tend to be on the diago-nal. As shown in (c) of Fig. 3, these peaks may be far fromthe ground truth and result in a big loss during the training.Thus, networks would be aware of the multi-peaks situation,and avoid it as the loss decreases.

The output of RCC layer is normalized to [0,1] by a nor-malization factor on the basis of above analysis. This trans-formation makes the output of RCC remain the same as theinput when N = 1 and does not change the conclusionwhen N > 1. The normalization is shown in (7).

RCC(l,m) =

∑iHT (l, i)HT (i,m)

Z(7)

where Z is a normalization factor with a value that equals tothe maximum in the output of RCC layer.

Note that RCC is not only used during the inference, itneeds to be introduced into training. Although the prior ofthis layer is not always right, its main purpose is to let thenetwork know that single-peak output is better during train-ing. Thus, the network is optimized towards more accurateand single-peak. In this way, the network can learn to avoidmulti-peaks predictions.

3.2. Highly differentiated heatmap regression

The heatmap used as the ground truth is generally gen-erated by a Gaussian distribution. The maximum of theheatmap is too close to the values of neighboring pixels,thus a slight change in the output may change the positionof the predicted keypoint. In view of this problem, we pro-pose a highly differentiated heatmap and a weighted cross-entropy loss.

3.2.1 Highly differentiated heatmap

The highly differentiated heatmap is constructed as follows:The value of a keypoint’s position is 1 in the heatmap, andthe value of the point with a distance dp from the keypointis αdp , where α is the hyperparameter with a value in range(0, 1). When distance dp is greater than a certain thresholdThd, the value of this point is set to 0. The highly differen-tiated heatmap increases the difference between keypoints

4

2019/11/3 first_td.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/latex/images/first_td.svg 1/1

(a) TR. heatmap

2019/11/3 first_hd.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/latex/images/first_hd.svg 1/1

(b) HD. heatmap

Figure 4. The difference between the traditional heatmap (TR.heatmap) and highly differentiated heatmap (HD. heatmap). Forgenerating HD. heatmap, we use the chessboard distance metricin this paper, but other distance metrics such as Euclidean or CityBlock distance could also be suitable choices. The hyperparameterThd is set to 8 and α is set to 0.7 in the experiments.

and the other points in the heatmap, making the features ofthe keypoint more discriminative. The traditional heatmapand highly differentiated heatmap are shown in Fig. 4.

3.2.2 Weighted cross-entropy loss

It is difficult to converge when directly regressing the highlydifferentiated heatmap using L1 or L2 loss function. Sincethe features of the keypoint are similar to that of the sur-rounding points, the intention to enlarge the difference be-tween the keypoint and its surrounding points leads to con-fusion of the network. We solve this problem by regardingthe heatmap regression task as a multi-label classification.A multi-label map is built as the annotations. The numberof pixels in the highly differentiated heatmap is regarded asthe number of categories in the multi-label map. The pointsin the highly differentiated heatmap with value 0 mean theinput does not belong to this category and the others rep-resent the input has these category labels. The schematicdiagram of multi-label map can be found in Fig. 2. Themulti-label map provides an area where a keypoint shouldbe in. Commonly used loss function for multi-label clas-sification is the cross-entropy, but the use of cross-entropywith the multi-label map cannot increase the discriminationbetween the keypoint and its surrounding points. Thus, wepropose a weighted cross-entropy loss, defined in the equa-tion (8), to solve this problem.

Lwce = −∑x∈I

∑i,j

wi,jx yi,jx logri,jx (8)

where x is an image from the training set I , (i, j) repre-sents the position of column j of the ith row. wi,j

x is thevalue of (i, j) position in highly differentiated heatmap andyi,jx is the label of the (i, j) position in multi-label map.ri,jx is the value of (i, j) position in the output heatmap.The loss means that the pixel value of the highly differenti-ated heatmap is used as the weight of the cross-entropy loss.At the same time, we use a softmax function on the output

heatmap before calculating the loss. The softmax opera-tion increases the competitive relationship between differ-ent pixels, where the position of higher weight, the greaterthe predicted value. Thus, the distance between maximumand the second-largest value in the heatmap is enlarged.

3.3. Maximum stability training

In order to make the network more robust to unseen natu-ral perturbations, we propose a maximum stability trainingloss to explore the relation between a clean image and itsperturbed image. A typical approach is to make the outputsof them similar, also known as stability training [31]. How-ever, this loss is too strict for pose estimation. The outputheatmap represents the probability that each pixel in the im-age belongs to a keypoint, and each pixel of a clean imagemay be changed in its perturbed image. Forcing the prob-ability of all the pixels between these image pairs to be thesame increases difficulty in network optimization.

In fact, the change in the maximum of a heatmap con-tributes most to stability. As the keypoint is the positionwith max probabilities in the heatmap, if the maximum ofan image and that of its perturbed image remain in the sameposition, the predicted keypoint would be stable. Based onabove analysis, we propose a new regularization loss, withthe change of maximum being focused on.

Lmst =∑

x∈I,x′∈Impx

∑i,j (4ri,jx )2

H ·W+ (4r(1)x )2 (9)

where ri,jx is the value at (i, j) position of an output heatmaprx with an input x. 4ri,jx = ri,jx′ − ri,jx , x′ is a perturbedimage of x and is obtained from pre-defined multiple pertur-bations Imp

x . H,W are the height and width of a heatmaprespectively. 4r(1)x is the difference between maximum ofrx and the value of same position in rx′ . This loss strength-ens the limit on the change of r(1)x , allowing small changesin other positions, thereby reducing the difficulty of opti-mization. Meanwhile, we provide a theoretical certificationto demonstrate that our maximum stability training loss canindeed improve robustness.

Theorem 3 In the same setting in (9), with r(1)x and r(2)x be-ing the first and second largest values in the output heatmaprx.

If the following condition is satisfied,

r(1)x − r(2)x ≥

√∑i,j (4r

i,jx )2 +H ·W (4r(1)x )2

then,

argmaxi,j(ri,jx ) = argmaxi,j(r

i,jx′ ).

5

This property manifests that a larger r(1)x − r(2)x and

a smaller√∑

i,j (4ri,jx )2 +H ·W (4r(1)x )2 are benefi-

cial for improving the stability. r(1)x − r(2)x is already en-

larged by HDHR. Therefore, minimizing the proposed loss∑i,j (4ri,jx )2

H·W + (4r(1)x )2 could further enhance the robust-ness.

4. Experiments4.1. Datasets

We conduct experiments on two benchmark datasets ofhand pose estimation-RHD [32] and STB [30]. RHD isa synthetic hand pose dataset. It contains 41,258 trainingimages and 2,728 test images. Precise 2D annotations for21 keypoints are provided. STB is a real-world dataset. Itcontains 12 sequences with six different backgrounds. Fol-lowing same setting in [24, 32], 10 sequences are used fortraining and the other 2 for testing.

There are lots of differences between synthetic datasetsand real datasets in terms of background, annotation mech-anism and so on. Thus, using these two kinds of datasetsallows for comprehensive evaluation of algorithms.

4.2. Experimental setup

Following the creation of perturbed images in [7], wegenerated 15 kinds of perturbations-brightness, defocusblur, zoom blur, frost, contrast, gaussian noise, glass blur,motion blur, shot noise, snow, gaussian blur, jpeg compres-sion, saturate, spatter and speckle noise-each with five lev-els of severity. Since the perturbations in practical applica-tions cannot all be seen during training, we train the networkwith first 10 kinds of perturbations and evaluate on the oth-ers. Each network used in our evaluation consists of twoparts-a backbone network and an upsampling network. Thebackbone network is used to extract latent features of an in-put image. The upsampling network utilizes the features topredict the heatmaps.

4.3. Evaluation criteria

Accuracy The area under the curve (AUC) on the per-centage of correct keypoints are used to measure the perfor-mance of pose estimation.

Robustness Different from the classification task, the ro-bustness of a regression task has little connect to accuracy[28]. Thus, we define a new criteria R(T, Tmp) to comparethe robustness of different models, where Tmp is a pertur-bation set of validation set T . The definition of R(T, Tmp)is shown in (10).

R(T, Tmp) =∑x∈T

∑x′∈Tmpx

(Pos(r(1)x )− Pos(r(1)x′ ))2 (10)

Dn(↓) RHD STB D(1)(2)(↑) RHD STBBaseline 954.67 820.22 Baseline 0.048 0.055

RCC 870.67 749.68 HDHR 0.099 0.203Table 1. Experiments for verifying the effectiveness of RCC andHDHR. The n is set to 64 when calculating Dn.

where Tmpx is a set that contains all the perturbed images

of x. Pos(r(1)x ) and Pos(r(1)x′ ) represent the predicted key-points of x and x′ respectively.

After calculating the R(·, ·), the proportion of stablesamples, where the predictions of the perturbed images inTmp remain the same with their natural images, are calcu-lated. The predicted keypoints of an original image and thatof its perturbed image are considered to be the same whenthe difference between them does not exceed P ∈ N+ pix-els (N+ is a set of natural numbers). By taking differentP , a Robustness Under Curve (RUC) can be obtained in asimilar way to Accuracy Under Curve.

4.4. Ablation study

The proposed method can be divided into three parts,Row-Column Correlation (RCC), Highly DifferentiatedHeatmap Regression (HDHR) and Maximum StabilityTraining (MST). MobilenetV2 [17] is adopted as the back-bone to study the role of the three modules. We first con-duct experiments to demonstrate that RCC could alleviatethe multi-peaks problem and HDHR can enlarge r(1)x −r(2)x .For evaluating RCC, we use a metric Dn, which representsthe difference of positions between topn and top1 in thevalidation set T , as shown in (11).

Dn(T ) = mean(∑x∈T

n∑i=2

(Pos(r(i)x )− Pos(r(1)x ))2) (11)

where r(i)x is the ith largest value in the heatmap and

Pos(r(i)x ) represents its position. The smaller the Dn, the

more concentrated the high response of a heatmap. As forHDHR, a metric (12) is adopted.

D(1)(2)(T ) = mean(∑x∈T

r(1)x − r(2)x ) (12)

The results are shown in Table 1. Baseline representsthe traditional heatmap regression. RCC achieves a smallerDn(T ) and HDHR enlarges the D(1)(2)(T ) in both RHDand STB datasets. The effectiveness of both modules hasbeen verified. In addition, we show the predicted heatmapsof the RCC model and the baseline model in Fig. 5. Base-line model tend to output heatmaps with multi peaks whilethe RCC model tend to output single peak heatmaps. Thisphenomenon demonstrates that RCC could alleviate multi-peaks problem in real-world evaluation.

6

Baseline

RCC

Figure 5. The visual comparisons between the predicted heatmapsof the Baseline model and the RCC model.

0

5000

10000

15000

20000

25000

30000

35000

40000

45000 70000

68000

14000

12000

10000

8000

6000

4000

2000

0

81.44%77.46% 80.12%80.41% 80.06%

!"#$(& ', ')* )

82.23% 82.67% 81.75% 80.30%81.63%

!"#$(& ', ')* ) RHD STB

Baseline RCC HDHR MST RCC+HDHR+MSTMethodAUC of RHDAUC of STB

Figure 6. Robustness of different training strategies on RHD andSTB datasets. The ordinate is the average value of R(T, Tmp)(↓ is better) of each perturbation and abscissa shows the name ofperturbations. AUC (↑ is better) is calculated with thresholds 0-30pixel on clean images.

R(T, Tmp) and AUC of the three modules is shown inFig. 6 and RUC in Fig. 7(a) and 7(b). Although thereare many difference between RHD and STB, where RHDis a synthetic dataset and STB is a real-world dataset, ro-bustness against five perturbations are enhanced after us-ing RCC, HDHR and MST. However, R(T, Tmp) of MSTon RHD is slightly worse than Baseline on Jpeg and Satu-rate perturbations. As shown in Theorem 3, the robustnessimprovement of MST loss is related to r(1)x − r(2)x . How-ever, traditional heatmap regression has tiny r(1)x − r(2)x be-cause Gaussian heatmap is used as the ground truth. Andthe r(1)x − r(2)x of RHD is smaller than that of STB (Table1), thus only using Lmst is unable to reach its full potentialon RHD dataset. When MST is combined with HDHR, therobustness is significantly enhanced. In addition, the dropof AUC on clean images is reasonable, since many papersin the field of adversarial examples have demonstrated thatthe improvement in the robustness is accompanied by thedecrease in the accuracy of the clean images [29].

2020/2/28 rucablation_little.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/ECCV-main-paper/images/rhpresults/rucablation_little.svg 1/1

(a) RHD

2020/2/28 rucablation_little.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/ECCV-main-paper/images/stbresults/rucablation_little.svg 1/1

(b) STB

Figure 7. RUC (↑ is better) of different training strategies. R+H+Mis an abbreviation of RCC+HDHR+MST.

RUC

AUC

MobileNetV2

ShuffleNet

VGG19_bn

Resnet50

StackHourglass

HRNet

Network Architecture

(%)

(%)

TraditionalHeatmapRegressionStableHeatmapRegression

(a) RHD

RUC

MobileNetV2

ShuffleNet

VGG19_bn

Resnet50

StackHourglass

HRNet

Network Architecture

RUC

AUC

(%)

(%)

TraditionalHeatmapRegressionStableHeatmapRegression

(b) STB

Figure 8. AUC (↑ is better) and RUC (↑ is better) of different modelarchitectures.

4.4.1 Different model architectures

Different backbones of pytorch official realization, resnet50[6], vgg19 bn [18], mobilenetV2 [17], shufflenet [13],stackhourglass (2-stack) [15] and HRNet [23] are adoptedfor comprehensive evaluation. The results on RHD datasetare shown in Fig. 8(a) and STB in Fig. 8(b). Our methodachieves most robust models against small perturbationsand maintains high performance in both datasets on all mod-els. The experimental results demonstrate the potential thatthe proposed method can be a general substitute for tradi-tional heatmap regression methods.

4.5. Comparison with state of the art

The proposed method is compared with state-of-the-art(SOTA) methods for improving robustness, AT [14], Aug-Mix [8], Lip [21], PG [12] and ST [11, 31]. Note that ST[31] added a stability loss Lstability =‖ rx − rx′ ‖ to makethe outputs of clean images and their perturbed images sta-ble and [11] proved that using gaussian noise to constructperturbed images is sufficient to improve robustness againstmultiple perturbations. However, several papers [19, 25]demonstrated that using a diverse set of augmentations isnecessary to improve the robustness. Therefore, we use allthe perturbations in the training of MST, including the gaus-sian noise, to calculate the stability loss Lstability as a sup-plementary method. This modified version of ST is denotedas STa and the original version is STg . RUC results of thesemethods on RHD and STB are shown in Fig. 9(a) and Fig.

7

9(b), and AUC in Table 2. The proposed method achievesan advance over these SOTA methods as well as maintainshigh performance on RHD and STB datasets. In addition,RUC of MST in Fig. 7(a) and Fig. 7(b) is better than thatof STa and STg in Fig. 9(a) and Fig. 9(b), which manifeststhat focusing on the change of maximum is more efficientin pose estimation.

2020/2/28 rucmobilenet_sota.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/ECCV-main-paper/images/rhpresults/rucmobilenet_sota.svg 1/1

(a) RHD

2020/3/6 rucmobile_sota_stb.svg

file:///Users/zhangyumeng/Desktop/perturbation paper/ECCV-main-paper/images/rucmobile_sota_stb.svg 1/1

(b) STB

Figure 9. RUC comparisons with SOTA methods. The backboneis mobilenetV2.

AUC(%) Base. AT Aug. Lip PG STg STa HDHR Ours

RHD 82.2 81.2 82.1 79.7 80.3 79.5 80.4 82.6 80.3STB 80.4 79.3 80.9 81.5 80.1 79.6 80.0 80.1 80.0

Table 2. AUC of state-of-the-art methods and the proposedmethod. Base. and Aug. are the abbreviations of Baseline andAugmix. The backbone is mobilenetV2.

5. Discussion

5.1. AUC under different image corruptions

The test samples provided by public datasets are rela-tively clean samples, and the test samples in the real-worldapplications have many image corruption problems. There-fore, a higher AUC achieved with the test samples of thepublic datasets does not mean a better visual effect in real-world applications. We evaluate the AUC results of the pro-posed and the corresponding baseline models under differ-ent corruption samples, and find that models with a highRUC have better accuracy results on the corruption samples.The results are shown in Table 3. In fact, our method hasbetter visual performance even with the original test sam-ples since the keypoints predicted by our method are morestable than those predicted by the baseline model.

5.2. Loss surface

We plot loss surfaces around test data points in Fig. 10.We vary the input along a linear space defined by a direc-tion of a difference vector (dn) between a perturbed imageand its clean image and a direction of a Rademacher vector(dr), where the x and y-axes represent the magnitude of the

AUC(%) Original GB Jpeg Saturate Spatter SN

RHD Baseline 82.23 51.48 66.29 73.37 69.69 41.95Ours 80.30 64.80 67.82 72.94 73.41 66.99

STB Baseline 80.41 59.58 77.68 53.40 54.91 29.03Ours 80.06 59.79 77.16 59.82 61.73 45.21

Table 3. AUC of the proposed models and the baseline models un-der different corruptions on RHD and STB dataset. The backboneis mobilenetV2. ”GB” is the abbreviation of gaussian blur and”SN” the speckle noise.

perturbation added in each direction and the z-axis repre-sents the loss. Our method achieves smoothest loss surface,demonstrating that a true boost for robustness is gained.

LipAugmixAT

STA

Baseline

STBPG Ours

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

CDCE

0,FF

Figure 10. Loss surfaces of our method and SOTA methods onRHD dataset.

6. Conclusions

In this paper, we propose a stable heatmap regressionmethod to improve robustness for pose estimation models.The method alleviates the multi-peaks problem, makes thekeypoint discriminative in the heatmap and applies a suit-able stability loss for pose estimation, thereby achieving arobust pose estimation model without being vulnerable tosmall perturbations. The effectiveness of the method is val-idated by theoretical analysis and extensive experiments ontwo benchmark datasets with different model architectures.In the future, we will construct multi-perturbation sets moreeffectively to further improve robustness.

References[1] Aharon Azulay and Yair Weiss. Why do deep convolutional

networks generalize so poorly to small image transforma-tions? Journal of Machine Learning Research, 20:1–25,2019. 3

[2] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John CDuchi, and Percy S Liang. Unlabeled data improves adver-sarial robustness. In Advances in Neural Information Pro-cessing Systems, pages 11192–11203, 2019. 1

8

[3] Changxi Zheng Chang Xiao, Peilin Zhong. Enhancing ad-versarial defense by k-winners-take-all. International Con-ference on Learning Representations, 2020. 2

[4] Micah Goldblum, Liam Fowl, and Tom Goldstein. Adver-sarially robust few-shot learning: A meta-learning approach.Advances in Neural Information Processing Systems, 33,2020. 2

[5] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.Explaining and harnessing adversarial examples. arXivpreprint arXiv:1412.6572, 2014. 1, 2

[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition. In Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition, pages 770–778, 2016. 7

[7] Dan Hendrycks and Thomas Dietterich. Benchmarking neu-ral network robustness to common corruptions and perturba-tions. In International Conference on Learning Representa-tions, 2019. 2, 3, 6

[8] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph,Justin Gilmer, and Balaji Lakshminarayanan. AugMix: Asimple data processing method to improve robustness anduncertainty. International Conference on Learning Repre-sentations, 2020. 3, 7

[9] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adver-sarial logit pairing. arXiv preprint arXiv:1803.06373, 2018.2

[10] Jan Laermann, Wojciech Samek, and Nils Strodthoff.Achieving generalizable robustness of deep neural networksby stability training. In German Conference on PatternRecognition, pages 360–373. Springer, 2019. 2

[11] Bai Li, Changyou Chen, Wenlin Wang, and Lawrence CarinDuke. Certified adversarial robustness with addition gaus-sian noise. In Advances in Neural Information ProcessingSystems, 2019. 2, 7

[12] Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer,and Ekin D Cubuk. Improving robustness without sacrificingaccuracy with patch gaussian augmentation. arXiv preprintarXiv:1906.02611, 2019. 3, 7

[13] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.Shufflenet v2: Practical guidelines for efficient cnn architec-ture design. In Proceedings of the European Conference onComputer Vision, pages 116–131, 2018. 7

[14] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt,Dimitris Tsipras, and Adrian Vladu. Towards deep learn-ing models resistant to adversarial attacks. In InternationalConference on Learning Representations, 2018. 1, 7

[15] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hour-glass networks for human pose estimation. In Proceedingsof the European Conference on Computer Vision, pages 483–499. Springer, 2016. 7

[16] Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Ju-lian Bitterwolf, Oliver Bringmann, Matthias Bethge, andWieland Brendel. A simple way to make neural networksrobust against diverse image corruptions. In European Con-ference on Computer Vision, pages 53–69. Springer, 2020.2

[17] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh-moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted

residuals and linear bottlenecks. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,pages 4510–4520, 2018. 6, 7

[18] Karen Simonyan and Andrew Zisserman. Very deep convo-lutional networks for large-scale image recognition. arXivpreprint arXiv:1409.1556, 2014. 7

[19] Florian Tramer and Dan Boneh. Adversarial training androbustness for multiple perturbations. In Advances in NeuralInformation Processing Systems, pages 5866–5876, 2019. 1,7

[20] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama.Lipschitz-margin training: Scalable certification of pertur-bation invariance for deep neural networks. In Advances inNeural Information Processing Systems, pages 6541–6550,2018. 3

[21] Muhammad Usama and Dong Eui Chang. Towards ro-bust neural networks with lipschitz continuity. In Interna-tional Workshop on Digital Watermarking, pages 373–389.Springer, 2018. 2, 3, 7

[22] Tim Van Erven and Peter Harremos. Renyi divergence andkullback-leibler divergence. IEEE Transactions on Informa-tion Theory, 60(7):3797–3820, 2014. 2

[23] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang,Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, MingkuiTan, Xinggang Wang, et al. Deep high-resolution represen-tation learning for visual recognition. IEEE transactions onpattern analysis and machine intelligence, 2020. 7

[24] Linlin Yang, Shile Li, Dongheui Lee, and Angela Yao.Aligning latent spaces for 3d hand pose estimation. In Pro-ceedings of the IEEE International Conference on ComputerVision, pages 2335–2343, 2019. 6

[25] Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin DCubuk, and Justin Gilmer. A fourier perspective on modelrobustness in computer vision. In Advances in Neural Infor-mation Processing Systems, volume 32, pages 13276–13286,2019. 1, 7

[26] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, SanghyukChun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu-larization strategy to train strong classifiers with localizablefeatures. In Proceedings of the IEEE International Confer-ence on Computer Vision, pages 6023–6032, 2019. 1

[27] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, andDavid Lopezpaz. mixup: Beyond empirical risk minimiza-tion. International Conference on Learning Representations,2018. 1

[28] Hong Zhang and Naiyan Wang. On the stability of videodetection and tracking. arXiv preprint arXiv:1611.06467,2016. 6

[29] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing,Laurent El Ghaoui, and Michael I Jordan. Theoretically prin-cipled trade-off between robustness and accuracy. In Inter-national Conference on Machine Learning, 2019. 1, 2, 7

[30] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu,Xiaobin Xu, and Qingxiong Yang. A hand pose trackingbenchmark from stereo matching. In IEEE InternationalConference on Image Processing, pages 982–986, 2017. 6

[31] Stephan Zheng, Yang Song, Thomas Leung, and Ian Good-fellow. Improving the robustness of deep neural networks via

9

stability training. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 4480–4488, 2016. 2, 3, 5, 7

[32] Christian Zimmermann and Thomas Brox. Learning to esti-mate 3d hand pose from single rgb images. In Proceedingsof the IEEE International Conference on Computer Vision,pages 4903–4911, 2017. 6

10