Towards Unified INT8 Training for Convolutional Neural Network

Feng Zhu1 Ruihao Gong1,2 Fengwei Yu1 Xianglong Liu2∗

Yanfei Wang1 Zhelong Li1 Xiuqi Yang1 Junjie Yan1

1SenseTime Research2 State Key Laboratory of Software Development Environment, Beihang University

{zhufeng1, yufengwei, wangyanfei, lizhelong, yangxiuqi, yanjunjie}@sensetime.com{gongruihao, xlliu}@nlsde.buaa.edu.cn

Abstract

Recently low-bit (e.g., 8-bit) network quantization hasbeen extensively studied to accelerate the inference. Be-sides inference, low-bit training with quantized gradientscan further bring more considerable acceleration, since thebackward process is often computation-intensive. Unfortu-nately, the inappropriate quantization of backward propa-gation usually makes the training unstable and even crash.There lacks a successful unified low-bit training frameworkthat can support diverse networks on various tasks. In thispaper, we give an attempt to build a unified 8-bit (INT8)training framework for common convolutional neural net-works from the aspects of both accuracy and speed. First,we empirically find the four distinctive characteristics ofgradients, which provide us insightful clues for gradientquantization. Then, we theoretically give an in-depth anal-ysis of the convergence bound and derive two principlesfor stable INT8 training. Finally, we propose two universaltechniques, including Direction Sensitive Gradient Clippingthat reduces the direction deviation of gradients and Devi-ation Counteractive Learning Rate Scaling that avoids ille-gal gradient update along the wrong direction. The experi-ments show that our unified solution promises accurate andefficient INT8 training for a variety of networks and tasks,including MobileNetV2, InceptionV3 and object detectionthat prior studies have never succeeded. Moreover, it enjoysa strong flexibility to run on off-the-shelf hardware, and re-duces the training time by 22% on Pascal GPU without toomuch optimization effort. We believe that this pioneeringstudy will help lead the community towards a fully unifiedINT8 training for convolutional neural networks.

∗indicates corresponding author

𝐠𝐱

𝐠# 𝐱𝛼 𝛽

Clipping

Learning rate scaling

𝐠# 𝐱

Convergence boundary

𝐠𝐱

Convergence boundary

𝛽

Figure 1. The fundamental idea of our unified INT8 training. gx

and gx represent the original float gradient and the quantized one,respectively. α and β represent different direction deviations thatquantization brings. The red lines present crash cases when the di-rection deviation is large. The left subfigure indicates that clippinggradient properly to reduce direction deviation within the conver-gence boundary can avoid crash. The right subfigure points outthat controlling learning rate (step size) could promise a stable pa-rameter updating by counteracting negative effect of deviation.

1. Introduction

Deep convolutional neural networks (DCNNs) haveachieved remarkable success in many fields, such as com-puter vision, natural language processing, information re-trieval, etc. However, training and deploying DCNNs usu-ally require a large amount of computational cost and powerconsumption, which is greatly challenging the extensive ap-plications in industry. As a result, many recent studies havebeen focusing on how to accelerate the inference of neu-ral networks by fixed-point quantization on weights or ac-tivations [6, 3, 23, 29, 27, 62, 42, 53, 61, 55, 20], and de-sign dedicated hardware utilizing the efficient integer arith-metic [17, 5, 26, 22]. The successful progress surprisinglyshows that the bit-width can be reduced to extremely lowsuch as 4-bit while bringing quite little hurt to the accuracyfor inference [15, 57, 13].

Besides inference, low-bit training can also promise con-siderable acceleration, which further quantizes gradientsand utilizes low-bit efficient compute kernel for both the

1

arX

iv:1

912.

1260

7v1

[cs

.LG

] 2

9 D

ec 2

019

forward and backward propagation. As analyzed in [24], thecomputation of backward propagation occupies more timethan that of forward propagation. So accelerating the train-ing utilizing low-bit quantization has greater potential whenconsidering the backward process. There has existed 16-bitfloating-point (FP16) training, which proves the feasibilityof low-bit training [41, 9, 33]. But it is restricted to lim-ited advanced GPUs based on Turing or Volta architecture.Compared with FP16, the 8-bit integer (INT8) operation iswidely supported by general GPUs based on Turing, Voltaand even low-end Pascal architectures. Besides, the 8-bitinteger arithmetic is theoretically and practically 2× fasterthan FP16 and 4× faster than FP32. Therefore, INT8 train-ing enjoys better efficiency, lower power consumption andbetter versatility on off-the-shelf hardware.

Despite the attractive benefits, when quantizing gradi-ents to 8-bit, the normal training tends to become unsta-ble, since the distortion of gradients easily misleads the di-rection of training and causes crash of optimization. Thisdefinitely makes INT8 training very difficult, especially forthe deep networks. Currently only a few studies have at-tempted to solve this problem [62, 56, 58, 2, 54, 50]. Unfor-tunately, all of them just tested limited quantization-friendlynetworks with high redundancy, and usually require com-plex structure adjustment or introduce additional operationto reduce quantization error, while significantly increasingthe computational complexity. Besides, most of these workslack the theoretical analysis on the ad-hoc tricks, and evenworse, none of them reports the practical speedup in thereal-world case. All these reasons make the existing INT8training methods stay far away from the practicality withoutthe universal design.

To build a robust and unified INT8 training framework,we conduct deeper explorations in the challenges of gradi-ent quantization. We empirically find that the distribution ofgradients owns four special characteristics: sharp and wide,evolutionary, depth-specific and structure-specific. Theseunique characteristics make gradient quantization quite dif-ferent from the naive quantization on weights or activations,and INT8 training more difficult to be stabilized. It is im-portant to understand the behaviors and effects of quantizedgradient in the convergence of the training. Therefore, wetheoretically establish the convergence bound with respectto the gradient quantization error and the learning rate.

Based on the special characteristics and the theoreticalanalysis, we propose two universal techniques: DirectionSensitive Gradient Clipping and Deviation CounteractiveLearning Rate Scaling to stabilize the INT8 training. TheDirection Sensitive Gradient Clipping minimizes the direc-tion deviation by pursuing an appropriate clipping as thetraining process evolves. Sometimes even if the clippinghelps reduce the quantization error, it may still suffer fromthe accumulated gradient deviations across deep layers. To

eliminate this effect, the Deviation Counteractive LearningRate Scaling is further devised to promise stable parameterupdating. The fundamental idea of our method is shown inFigure 1. Extensive experiments on a variety of networkstructures and tasks prove the superiority and versatility ofour method.

Our contribution can be summarized as below:

• We observe four special characteristics on the gradi-ent distribution: sharp and wide, evolutionary, depth-specific and structure-specific, which cause the largerquantization error of gradients.

• We theoretically provide the convergence bound ofINT8 training, and respectively devise two universaltechniques that can stabilize the INT8 training.

• We are the first to achieve stable INT8 training of var-ious networks such as MobileNetV2/InceptionV3 andvarious tasks such as object detection, with compara-ble accuracy to full-precision training.

• We build a flexible and unified INT8 training frame-work for various tasks using various networks, whichcan easily replace the original full-precision training.

• We are the first to complete practical acceleration ofINT8 training on low-end GPUs with Pascal architec-ture, i.e., NVIDIA GeForce GTX 1080Ti, achievingabout 22% speedup without too much code optimiza-tion.

2. Related WorkCompared to huge amount of studies on accelerating

inference by model quantization [47, 60, 7, 53, 11, 40],there are few works exploring quantized training includingbackward propagation comprehensively. DoReFa-Net [62]quantizes gradients to 4 and 6 bits, but only experimentsAlexNet with low precision gradient. WAGE [56] andWAGEUBN [58] quantize gradient to 8-bit integer, but theyboth incur considerable loss of accuracy (greater than 5%).RangeBN [2] and FP8 training [54] achieve accuracy com-parable to full-precision models, but they both use floating-point number in gradients, which is not beneficial for hard-ware optimization to boost the speed. Besides quantizedtraining, most low-precision training research keeps gra-dient precision in 16-bit floating-point. Flexpoint [33],MPT [41] and DFP [9] all use 16-bit floating-point to trainDNNs with accuracy comparable to full-precision model.To perform more efficient training of neural networks, INT8training has more advantages over FP16 training.

3. Unified INT8 TrainingIn this paper, we aim to build a unified INT8 training

framework, which utilizes 8-bit integer arithmetic to accel-

erate the expensive training process of deep neural networksincluding both the forward and backward propagation.

3.1. Preliminaries

Symmetric uniform quantization is the most efficientscheme among existed quantization methods, due to itshardware-friendly computation. Therefore, to guaranteethe acceleration performance, we build the INT8 trainingframework based on it. Given the data x (i.e., weights, ac-tivations, and gradients) following in the range (l, u) and aclipping value c ∈ (0,max(|l|, |u|)], the symmetric uniformquantization can be formulated as:

q = round(clip(x, c)

s), (1)

where clip(x, c) = min(max(x,−c), c), s = c28−1−1 in-

dicates the scaling factor to project the floating-point num-ber to fixed-point 8-bit integer, and q represents the quan-tized fixed-point number. Subsequently, the correspondingdequantized data x can be calculated by:

x = q · s. (2)

Different from most prior studies that mainly focus onspeeding up the inference (i.e., the forward propagation),our INT8 training framework attempts to further acceleratethe backward propagation during the training stage, by ap-plying quantization to the gradients. Namely, we pursue thequantize-dequantized gradients g from full-precision gradi-ents g in a proper way.

To ensure the quantized gradients maintain an unbiasedexpectation compared with the original ones, we adopt thestochastic rounding following [16]:

rounds(x) =

{bxc, w.p. 1− (x− bxc)bxc+ 1, w.p. x− bxc

. (3)

Unfortunately, although the stochastic rounding tech-nique limits the quantization error to some extent from thestatistical view, the perturbation for each training iteration isstill inevitable and harmful for convergence, whose reasonswill be discussed in the following section.

3.2. Challenges of Gradient Quantization

Gradients determine the direction of optimization andthe magnitude of parameter update and thus play a criti-cal role in pursuing high accurate models. In INT8 training,after we apply quantization to gradients, the perturbation in-troduces deviation to the optimization direction. Once thedeviation accumulates to an unacceptable degree, the train-ing process may be unstable and even crash, resulting insevere performance degradation. Figure 2 shows our empir-ical observation that for some special network architectures

0 100 200 300Epoch

0

25

75

100

Accuracy

INT8FP32

(a) the accuracy curve

0 100 200 300Epoch

0

2

∞

Loss

INT8FP32

(b) the loss curve

Figure 2. Crashed training of MobileNetV2 on CIFAR-10 afterquantizing gradients to 8-bit.

−1.2 0.0 1.2 ×10−1

0.28

0.55 layers.16.conv3.W

−4.9 0.0 4.9

layers.16.conv3.A

−5.0 0.0 5.0 ×10−4

layers.16.conv3.G

(a) gradients are different from weights and activations

−1.8 0.0 1.8 ×10−4

0.45

0.90 layers.16.conv3.G

−5.0 0.0 5.0 ×10−4

layers.16.conv3.G

−1.0 0.0 1.0 ×10

−4

layers.16.conv3.GEpoch 0 Epoch 30 Epoch 300

(b) gradients keep evolving during training

−1.1 0.0 1.1 ×10−2

0.12

0.25 layers.0.conv3.G

−4.0 0.0 4.0 ×10

−4

layers.12.conv3.G

−1.8 0.0 1.8 ×10−4

layers.16.conv3.G

(c) gradients of different depths have have different patterns

−1.1 0.0 1.1×10

−4

0.33

0.66 layers.16.conv1.G

−9.3 0.0 9.3 ×10−3

layers.16.conv2.G

−1.8 0.0 1.8 ×10−4

layers.16.conv3.G

(d) gradients of different structures have different patterns

Figure 3. Distributions of activations, weights and gradients withrespect to different layers of MobileNetV2 and training iterations.

like MobileNetV2, directly quantizing gradients causes arapid crash of training.

To further investigate the essential reasons behind thisphenomenon, we conduct detailed analysis on the distribu-tion of gradients during training without gradient quantiza-tion, as shown in Figure 3. We surprisingly observe that thegradients own the following unique characteristics:

C1: Sharp and Wide. As shown in Figure 3(a), comparedto weights and activations, gradients follow an unusualdistribution that has more values concentrated around

zero while a certain number of extreme values also ex-ists. Therefore, the distribution curve is very sharpwith small values taking the majority of gradients, butthe range is relatively very wide. This makes manygradients quantized to zero and the quantization errorsignificantly large when using uniform quantization.

C2: Evolutionary. Figure 3(b) depicts how the gradientdistribution of the same layer evolves with respect tothe training iterations. We can find that as the train-ing goes on, the shape of gradient distribution becomesmuch sharper and narrower, which means it is impossi-ble to fix the quantization settings throughout the train-ing process, as we usually do for weights and activa-tions, such as assuming the same clipping range in thewhole training.

C3: Depth-Specific. Figure 3(c) compares the distribu-tion of gradients in different layers. It is obvious thatthe distributions in the shallow layers are sharper withlarger extreme values than the deeper layers. Thismeans that the preceding layers of the deep neural net-works often face more severe quantization loss.

C4: Structure-Specific. As can be seen in Figure 3(d), thegradients of layers with different structures present ap-parently different patterns. For MobileNetV2, the sec-ond convolutional layer in each block is of depth-wisestructure. Its gradients own larger range and sharpershape even in the deeper block, making MobileNetV2harder to quantize from the aspect of gradients.

Based on the above observations, we can conclude thatthe gradients differ from weights and activations largely,which inevitably causes an unstable training, when simplyadopting the common quantization techniques for weightsand activations. This means that we need certain tech-niques to take care of distinctiveness in gradient quantiza-tion, which brings great challenges to the real and unifiedINT8 training in practice.

Before turning to devise the desired techniques consider-ing the speciality of gradients, we first attempt to understandthe gradient’s effect on the training stability, by theoreticallyrevealing the connections between training convergence andgradient quantization. This will provide us a reliable clue tobuild the robust and unified INT8 training framework.

3.3. Stabilize Training: A Theoretical Perspective

As commonly used in the analysis of deep learning opti-mizers [12, 28, 48, 39], the ability of convergence is usuallyevaluated by the regret R(T ).

R(T ) =

T∑t=1

(ft(wt)− ft(w∗)), (4)

where T indicates the number of iterations. wt ∈ S is theparameter at time t in the convex compact set S, and ft(wt)denotes the corresponding loss function. The optimal pa-rameter is represented by w∗. If the average regret R(T )

Tapproaches zero quickly as T increases, the speed and abil-ity of convergence can be guaranteed.

Due to the complexity of the DCNNs, it is very difficultto directly analyze its behaviors. As the prior studies [1, 34,21, 59] do, we first make the following assumptions:

Assumption 1. ft is convex;

Assumption 2. ∀wi,wj ∈ S, ‖wi −wj‖∞ ≤ D∞.

Although the convexity assumption may not hold fordeep networks, analysis based on this can provide reason-able and valuable insights for us, which has been proved inprevious studies [12, 39, 21, 59].

Taking the standard stochastic gradient descent algo-rithm into consideration, the optimization based on quan-tized gradient gt and learning rate ηt can be formulated as:

wt+1 = wt − ηtgt. (5)

Then we have the following theoretical finding (see the sup-plementary materials for detailed proof):

Theorem 1. If define the error of quantized gradients asεt = gt − gt, then with assumption 1 and 2, we have:

R(T )

T≤ dD2

∞2TηT︸ ︷︷ ︸(1)

+D∞T

T∑t=1

‖εt‖︸ ︷︷ ︸(2)

+1

T

T∑t=1

ηt2‖gt‖2︸ ︷︷ ︸

(3)

. (6)

We can find that the bound of average regret is dominatedby three terms. Term (1) approaches zero as T increasesand thus can be ignored in gradient quantization. Term (2)indicates the quantization error of gradients greatly affectsthe ability to converge, and it is usually large, as analyzedin Section 3.2. For term (3), its magnitude is mainly influ-enced by the learning rate and l2-norm of quantized gradi-ents. Based on the theoretical analysis, to stabilize INT8training, we have two basic principles for designing betterquantization techniques: (1) reduce the quantization errorof gradients; (2) scale down the learning rate. They are alsovery intuitive since, on the one hand, a lower quantizationerror means small deviation of optimization direction andthus avoids the training crash, on the other hand, it is acommon sense that decreasing the learning rate graduallypromises a better solution in the optimization.

Now with the design principles, the question is how todevise the universal techniques for INT8 training, mean-while take the characteristics of gradients into considera-tion. We respectively present two novel techniques: Direc-tion Sensitive Gradient Clipping and Deviation Counterac-tive Learning Rate Scaling, which together lower the aver-age regret bound and guarantee stable INT8 training.

Table 1. KS-statistics of gradient and weight with respect to dif-ferent layers’ conv3 in MobiletNetV2, the last column indicatesthe maximum value that can accept the hypothesis at significancelevel of 0.05.

DataDistribution

Critical valueGaussian Laplace Student

layer0g 0.1934 0.0790 0.2005 0.0012w 0.0391 0.0721 0.1011 0.0765

layer8g 0.2061 0.1091 0.2303 0.0024w 0.0294 0.0569 0.1084 0.0110

3.4. Direction Sensitive Gradient Clipping

Considering the basic operation z = W>a in deep neu-ral networks, the gradients of weights gW actually can becalculated by gz

>a. From this aspect, the quantization errorof gW in (6) mainly stems from that of activation gradientsgz. Therefore, in our INT8 training we can mainly concernthe quantization of gz, which will help control the error ofquantized gradients in (6). For simplicity of notations, inthe following discussion we directly use g to denote gz.

To minimize quantization error, previous works mainlyseek the optimal clipping value c in (1) by assuming certaindata distribution, e.g. Gaussian distribution [3, 4, 19, 2, 21,11]. However, according to the gradient characteristics C1and C2 we discover, it is unpractical to make a common as-sumption for an evolutionary and unusual gradient distribu-tion. To further prove this point, we do the KolmogorovS-mirnov test with distribution parameter solved by maximumlikelihood estimation, and report the KS-statistics that con-sistently reject the assumption that gradients obey any com-mon distribution in Table 1.

To find the optimal clipping value c without any assump-tion, a straightforward idea is to keep the quantized gra-dient consistent with the original one by gradient descentalgorithm. Usually, one can model the consistency usingthe popular mean-square error (MSE). Unfortunately, dueto characteristics C2 and C3 of gradients with huge discrep-ancy and fluctuation in their magnitudes, MSE makes theoptimization vulnerable and unable to work under the samesimple setting across various layers.

Therefore, to pursue the desired clipping values of dif-ferent layers that promise stable training, we choose cosinedistance to guide the learning of clipping values, which notonly avoids the negative effect of the varied gradients’ mag-nitudes, but also keeps the network optimization directionsconsistent:

dc = 1− cos(< g, g >) = 1− g · g|g| · |g|

(7)

where g and g denote the original floating-point gradientand its quantize-dequantized counterpart.

The cosine distance measures the direction deviation ofquantized gradients. As shown in Figure 4, when dc in-creases to a certain level, the whole training crashes. There

0 5000 10000Training Step

0

25

50

75

100

Accuracy Accuracy

dc0.05

0.10

d c

(a) the accuracy curve

0 5000 10000Training Step

0.5

1.0

1.5

2.0

Loss

Lossdc

0.05

0.10

d c

(b) the loss curve

Figure 4. Model crashes when dc exceeds limits.

0.60 0.80 1.00Clipping Ratio

1.0

1.5

2.0

2.5

d c

1e−4γ

(a) effect of clipping

0 100 200 300Epoch

0

50

100

Accuracy η1

η2η3η4

(b) effect of scaling strategies

Figure 5. The effect of clipping and learning rates on INT8 train-ing. γ in (a) represents optimal clipping value. In (b), η1 sets ini-tial learning rate as 0.1 with φ(dc) scaling, η2, η3 and η4 choose0.01, 0.05, 0.1 as initial learning rate respectively without scale.

exists strong correlation between dc and training stability,which proves that cosine distance can effectively reflect theinfluence of gradient quantization on the convergence. Byminimizing the deviation, we subsequently reduce term (2)in (6). Figure 5(a) shows the quantization error using differ-ent clipping values, where there exists an optimal clippingvalue that substantially reduces the cosine distance.

3.5. Deviation Counteractive Learning Rate Scaling

The theoretical analysis on convergence ability of quan-tized training indicates the necessity of scaling down learn-ing rate, since the quantization error of gradients cannotvanish completely. To validate this point, we decreasethe learning rate of the original crashed training of Mo-bileNetV2 mentioned in Section 3.2 and find that it defersand even eliminates the crash with an extremely low learn-ing rate, although facing a performance degradation (see thered, green and orange lines in Figure 5(b)).

Since the gradients are backward propagated layer bylayer, the minor gradient deviation will accumulate expo-nentially after massive multiplication and addition calcula-tion. To address this issue, we further propose the DeviationCounteractive Learning Rate Scaling to balance out the er-ror by exponentially decaying the learning rate according tothe degree of direction deviation dc, the scaling function isformulated at:

φ(dc) = max(e−αdc , β) (8)

where α controls the decay degree and β limits the lower

Figure 6. Flexible INT8 convolutional layer replacement.

Table 2. Overhead reduced with Periodic Update (on ResNet-50).Period 1 10 100 1000

Average time(s/iter) 1.006 0.364 0.301 0.297

bound of scaling.This scaling function generates a factor to scale down the

original full-precision learning rate. We empirically findthat the self-adapting scaling function performs well in alayer-wise way, adaptively adjusting the learning rate ac-cording to the direction deviations in different layers. Thiscounteracts the undesired effects of the gradient deviationsacross layers, and exactly addresses the challenges of thedepth-specific and structure-specific patterns as observed incharacteristics C3 and C4 in Section 3.2. The blue line inFigure 5(b) demonstrates that the training equipped withφ(dc) scaling achieves higher accuracy than the manuallyadjusted ones (tested with MobileNetV2 on CIFAR-10).

3.6. General Purpose Training Framework

In addition to ensuring the stable and accurate conver-gence, in practice our unified INT8 training frameworkshould also satisfy the following three features:(1) Easy to plug into any DCNN architecture. To re-alize this, we implement an automatic match and replace-ment mechanism in PyTorch [46] that correspondingly sub-stitutes convolutional and fully-connected layers with 8-bitcounterpart. The whole workflow including both forwardand backward passes is shown in Figure 6.(2) No excessive extra computational overhead. To avoidthe extra time cost of calculating clipping value, we designa Periodic Update method to optimize the clipping valueperiodically. As we can see in Table 2, the Periodic Updatemethod dramatically reduces the computational overhead ofoptimizing the clipping value.(3) Easy to implement on off-the-shelf hardware. To val-

0 1000 2000 3000Step

0.0

0.2

0.4

0.6

d c

conv1conv2conv3

(a) the cosine distance

0 100 200 300Epoch

0

25

50

75

100

Accuracy dc

dc2

ϕ(dc)

(b) the accuracy curve

Figure 7. Analysis of cosine distance and learning rate scalingfunction.

idate the potential of that, we utilizes the DP4A instruc-tion (8-bit integer 4-element vector dot product) on low-endNVIDIA Pascal GPUs to implement efficient 8-bit kernelsfor calculating gradients. To the best of our knowledge,we are the first to achieve practical acceleration of INT8training including the backward propagation. The detailedspeedup will be reported and discussed in Section 4.4.

4. ExperimentsWe conduct extensive experiments to demonstrate that

our proposed framework is unified for various networkstructures on popular image classification and object detec-tion tasks with state-of-the-art accuracy, and meanwhile itcan be easily deployed on the mainstream devices (NVIDIAPascal GPU) with satisfactory speedup, compared to full-precision training.

4.1. Ablation Study

Settings. We first conduct the ablation study on CIFAR-10dataset with MobileNetV2 [51], to validate the effectivenessof the proposed techniques. We use cosine scheduler [1]with initial learning rate set to 0.1 for all experiments. Inthe Periodic Update experiment, the α and β in learningrate scaling are set to 20 and 0.1 respectively.Direction Sensitive Gradient Clipping. Figure 7(a) showsthe cosine distance with respect to the training steps. Wecan observe that conv2 (the second convolutional layer) ofeach block owns a much larger cosine distance than otherlayers of the block most of the time. This is consistent withC4 that the gradients of conv2 own sharper shape, indicatingthat our cosine distance can well reflect the gradient charac-teristics.

Moreover, as Table 3 lists, our proposed direction sensi-tive gradient clipping technique indeed prevents INT8 train-ing from crashing, which proves the fact that optimizing aclipping value of gradients to minimize direction deviationdc can certainly ensure a stable INT8 training.Deviation Counteractive Learning Rate Scaling. Weevaluate three forms of learning rate scaling strategies with-out clipping to control variable for a reasonable compari-

Table 3. Ablation study on clipping method for INT8 training.

Clipping method No clippingDirection SensitiveGradient Clipping

Accuracy (%) NaN 93.02

son. The results shown in Figure 7(b) reveal that linear andquadratic forms are too weak to control optimization direc-tion within the convergence boundary and model crashes inthe training process. Compared with linear and quadraticform, the scaling with exponential form is more powerful tocounteract the direction deviation and prevents optimizationfrom stepping out of the convergence boundary. We furtherexplore its sensitivity to the selection of hyperparameter inTable 4, and we can see that different settings of α and βachieve similar accuracy, which presents the stability of ourDeviation Counteractive Learning Rate Scaling.

Table 4. Comparison of different hyperparameters for learning ratescaling.

α 10 10 20 20β 0.1 0.2 0.1 0.2

Accuracy (%) 92.82 93.28 93.38 93.27

Periodic Update for clipping value. To reduce the extracomputational overhead, we increase the period to updateclipping value and find that it brings little hurt to the accu-racy, as shown in Table 5. This empirical conclusion bringspossibilities for the practical acceleration of INT8 training.Besides, here we apply both gradient clipping and learningrate scaling, and obtain better performance (see that withperiod 1) than those in Table 3 and 4. This further verifiesthe positive effects of the two general techniques.

Table 5. Ablation study on update period.Period 1 10 100 1000

Accuracy (%) 93.66 93.07 93.38 92.75

4.2. Image Classification

Now we consider the popular image classification taskthat most prior studies choose to evaluate the quantizationperformance. We experiment with AlexNet [31], ResNet[18], MobileNetV2 [51] and InceptionV3 [52] on CIFAR-10 [30] and ImageNet (ILSVRC2012) [10]. The CIFAR-10dataset contains a training set of 50K images and a test-ing set of 10k images. Each image is of size 3×3 with 10classes. ImageNet (ILSVRC2012) consists of 1.2 milliontraining images and 50K test images with 1000 classes.Settings. As for the hyperparameters of ResNet, we use thesame settings described in [18]. For other neural networks,we use cosine scheduler [1] with initial learning rate set to0.1. Theα and β in learning rate scaling are set to 20 and 0.1respectively. Clipping value is updated per 100 iterationsfor all experiments.

Table 6. Results on CIFAR-10 dataset.

Model MethodBit-width(W/A/G)

Accuracy(%)

ResNet-20FP 32/32/32 92.32

FP8 training [54] 8/8/8 92.21Ours 8/8/8 91.95

MobileNetV2FP 32/32/32 94.39

Ours 8/8/8 93.38

InceptionV3FP 32/32/32 94.89

Ours 8/8/8 95.00

CIFAR-10. As Table 6 shows, our method achieves com-parable accuracy on ResNet-20 to FP8 training, but takesmuch less memory and computation consumption due to thefixed-point operation. Moreover, our method performs sur-prisingly good on MobileNetV2 (1.01% accuracy drop) andInceptionV3 (even better than full precision model).

ImageNet. Table 7 lists existing state-of-the-art quantizedtraining methods including WAGE [56], WAGEUBN [58]and FP8 training [54]. For AlexNet INT8 training, ourmethod obtains 5.84% improvement over DoReFa-Net [62].Free from the extra overhead like tanh, our method enjoyshigher efficiency than DoReFa-Net. As for the 2-bit weightand 8-bit activation/gradient case, we significantly outper-form WAGE with about 3% accuracy gain. What’s more,equipped with our method, the INT8 training for ResNetarchitecture achieves almost no performance degradation,while none of the previous studies has done that. Comparedwith the FP8 training method, our method improves the ac-curacy by nearly 3%. It should be noted that we can di-rectly get a real speedup on popular off-the-shelf deviceswhile methods like FP8 training need specially designedhardware, which means that our framework is more generalfor unified training acceleration.

As analyzed in [36], the convolutional layer occupiesmost of the training time while other layers like BatchNormand ReLU are not computation-intensive. Therefore, wemainly focus on quantizing convolutional layers currentlyand do not quantize BatchNorm layer like RangeBN [2]and WAGEUBN [58]. Even so, there is still a significantspeedup for INT8 training. In addition, we could get com-parable accuracy to full precision training, much higher thanRangeBN and WAGEUBN.

Networks using INT8 training for the first time. To ourbest knowledge, we are the first to quantize gradient ofMobileNetV2, which is known to be difficult in this com-munity. Our method gets very good performance on bothCIFAR-10 and ImageNet datasets using MobileNetV2, withonly around 1% accuracy loss. We also try INT8 training onInceptionV3 for the first time, and achieve comparable ac-curacy to full precision model. Note that for InveptionV3on CIFAR-10, our INT8 training method can even achievebetter performance than the full-precision model.

Table 7. Results on ImageNet dataset.

Model MethodBit-width(W/A/G)

Accuracy(%)

AlexNet

FP 32/32/32 59.84DoReFa-Net [62] 8/8/8 53.00

Ours 8/8/8 58.84WAGE [56] 2/8/8 48.40

Ours 2/8/8 51.28

ResNet-18

FP 32/32/32 70.30WAGEUBN [58] 8/8/8 66.92FP8 training [54] 8/8/8 67.34

Ours 8/8/8 69.67

ResNet-34FP 32/32/32 73.68

WAGEUBN [58] 8/8/8 68.50Ours 8/8/8 73.29

ResNet-50FP 32/32/32 76.60

WAGEUBN [58] 8/8/8 69.07Ours 8/8/8 76.34

MobileNetV2FP 32/32/32 72.39

Ours 8/8/8 71.20

InceptionV3FP 32/32/32 72.39

Ours 8/8/8 71.20

4.3. Object Detection

To prove the versatility of our method, we further con-duct experiments with the popular object detection net-works including Faster-RCNN [49], RFCN [8] and Reti-naNet [37] on two widely used datasets: PASCAL VOC[14] and COCO [38]. The PASCAL VOC dataset consistsof 11k images with 20 classes. The COCO dataset containsmore than 20k images and 80 object categories. Note thatwe are the first to successfully achieve INT8 training on theobject detection task.Settings. As for the hyperparameters, we follow the samerules described in [35]. And α and β for learning rate scal-ing are the same as those used in image classification task.PASCAL VOC. We test RFCN and Faster R-CNN with dif-ferent backbones, and find that quantized training equippedwith our method only suffers a very slight detection accu-racy (mAP) drop. The result of RFCN shows that even fora deeper backbone such as ResNet-101, our INT8 trainingstill maintains almost the same accuracy as full-precision.COCO. On the large scale COCO dataset, we experimentwith RetinaNet (one-stage) and Faster R-CNN (two-stage).Our method performs stably with less than 1.8% accuracydegradation on both networks. We find that RetinaNet in-curs higher mAP loss than Faster R-CNN, which is incon-sistent with the conclusions in the previous study [35]. Thismay be caused by the fact that the focal loss used in onestage detector is more sensitive to gradient quantization.

4.4. Speed Result on NVIDIA GPU

None of the existing libraries can directly support thecomplete INT8 training. Thus we implement it by ourselves

Table 8. Results on PASCAL VOC Dataset.

Model Backbone MethodBit-width(W/A/G)

mAP (%)

FasterR-CNN

ResNet-50 FP 32/32/32 82.0ResNet-50 Ours 8/8/8 81.9

RFCNResNet-101 FP 32/32/32 80.8ResNet-101 Ours 8/8/8 79.1

Table 9. Results on COCO Dataset.

Model Backbone MethodBit-width(W/A/G)

mAP (%)

FasterR-CNN

ResNet-50 FP 32/32/32 36.2ResNet-50 Ours 8/8/8 34.95

RetinaNetResNet-50 FP 32/32/32 36.9ResNet-50 Ours 8/8/8 35.1

Figure 8. INT8 convolution speedup on GPU, where Y-axis indi-cates (input shape), (kernel number, kernel size) of convolution.

Table 10. End-to-end average time for a round of INT8 training.(tested with ResNet-50 on GeForce GTX1080TI, batch size 64.)

Precision Forward (s) Backward (s) Iteration (s)FP32 (cuDNN) 0.117 0.221 0.360

INT8 (ours) 0.101 0.171 0.293

on NVIDIA Pascal GPU using DP4A instruction to verifythe acceleration power of our method. The speedup of eachconvolutional layer in ResNet-50 is shown in Figure 8. Inthe forward process using our solution, INT8 can bring anaverage 1.63× speedup, while in the backward process, itcan achieve a higher 1.94× speedup. Table 10 further re-ports the time consumption and speed improvement of eachtraining round. Even if we only replace the FP32 convolu-tional layer with the slightly optimized INT8 one, the train-ing time for ResNet-50 can be reduced by about 22%.

5. ConclusionsIn this paper, we attempt to build an INT8 training frame-

work for common DCNNs. We found four distinctive char-acteristics of gradients and then gave two theoretical princi-

ples stabilizing training with the convergence bound. Basedon that, we proposed Direction Sensitive Gradient Clippingand Deviation Counteractive Learning Rate Scaling. Ex-tensive experiments prove the versatility of our method forvarious networks and tasks. We reduced the training timeby 22% on Pascal GPU with only trivial optimization. Ifeach layer is sufficiently optimized, the training will achievehigher speedup and lower memory consumption. We hopeour first successful attempt can help lead the community to-wards a fully unified INT8 training.

References[1] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and

Milan Vojnovic. Qsgd: Communication-efficient sgd via gra-dient quantization and encoding. In Advances in Neural In-formation Processing Systems 30, pages 1709–1720. 2017.

[2] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry.Scalable methods for 8-bit training of neural networks, 2018.

[3] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry.Post-training 4-bit quantization of convolution networks forrapid-deployment. arXiv preprint arXiv:1810.05723, 2018.

[4] Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconce-los. Deep learning with low precision by half-wave gaussianquantization. In CVPR, July 2017.

[5] Yunji Chen, Huiying Lan, Zidong Du, Shaoli Liu, JinhuaTao, Dong Han, Tao Luo, Qi Guo, Ling Li, Yuan Xie, et al.An instruction set architecture for machine learning. ACMTransactions on Computer Systems (TOCS), 36(3):9, 2019.

[6] Jungwook Choi, Zhuo Wang, Swagath Venkataramani,Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and KailashGopalakrishnan. Pact: Parameterized clipping activa-tion for quantized neural networks. arXiv preprintarXiv:1805.06085, 2018.

[7] Matthieu Courbariaux, Yoshua Bengio, and Jean-PierreDavid. Binaryconnect: Training deep neural networkswith binary weights during propagations. arXiv preprintarXiv:1511.00363, 2015.

[8] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Objectdetection via region-based fully convolutional networks. InProceedings of the 30th Annual Conference on Neural Infor-mation Processing Systems, Dec 2016.

[9] Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere,Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srini-vas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evange-los Georganas, Alexander Heinecke, Pradeep Dubey, JesusCorbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko,and Vadim Pirogov. Mixed precision training of convolu-tional neural networks using integer operations. In ICLR,May 2018.

[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database.2009 IEEE Conference on Computer Vision and PatternRecognition (CVPR), Jul 2009.

[11] Ruizhou Ding, Ting-Wu Chin, Zeye Liu, and Diana Mar-culescu. Regularizing activation distribution for training bi-narized deep networks. In CVPR, June 2019.

[12] John Duchi, Elad Hazan, and Yoram Singer. Adap-tive subgradient methods for online learning and stochas-tic optimization. Journal of Machine Learning Research,12(Jul):2121–2159, 2011.

[13] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani,Rathinakumar Appuswamy, and Dharmendra S Modha.Learned step size quantization. arXiv preprintarXiv:1902.08153, 2019.

[14] Mark Everingham, Luc Gool, Christopher K. Williams, JohnWinn, and Andrew Zisserman. The pascal visual objectclasses (voc) challenge. Int. J. Comput. Vision, June 2010.

[15] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li,Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differen-tiable soft quantization: Bridging full-precision and low-bitneural networks. In ICCV, October 2019.

[16] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, andPritish Narayanan. Deep learning with limited numericalprecision. In Proceedings of the 32nd International Con-ference on Machine Learning, pages 1737–1746, 07–09 Jul2015.

[17] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pe-dram, Mark A. Horowitz, and William J. Dally. Eie. ACMSIGARCH Computer Architecture News, 44(3):243254, Jun2016.

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition. CVPR, Jun2016.

[19] Zhezhi He and Deliang Fan. Simultaneously optimizingweight and quantizer of ternary neural network using trun-cated gaussian approximation. In CVPR, June 2019.

[20] Lu Hou and James T. Kwok. Loss-aware weight quantizationof deep networks. In ICLR, May 2018.

[21] Lu Hou, Ruiliang Zhang, and James T. Kwok. Analysis ofquantized models. In ICLR, May 2019.

[22] Huawei Technologies Co., Ltd. Ascend 310.https://e.huawei.com/se/products/cloud-computing-dc/atlas/ascend-310.

[23] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu,Matthew Tang, Andrew Howard, Hartwig Adam, and DmitryKalenichenko. Quantization and training of neural networksfor efficient integer-arithmetic-only inference. 2018 IEEEConference on Computer Vision and Pattern Recognition(CVPR), Jun 2018.

[24] jcjohnson. cnn-benchmarks. https://github.com/jcjohnson/cnn-benchmarks, 2016.

[25] Yangqing Jia, Evan Shelhamer, Jeff Donahue, SergeyKarayev, Jonathan Long, Ross Girshick, Sergio Guadarrama,and Trevor Darrell. Caffe: Convolutional architecture for fastfeature embedding. In Proceedings of the 22Nd ACM In-ternational Conference on Multimedia, MM ’14, New York,NY, USA, 2014. ACM.

[26] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patter-son, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, SureshBhatia, Nan Boden, Al Borchers, et al. In-datacenter perfor-mance analysis of a tensor processing unit. In Proceedingsof the 44th Annual International Symposium on ComputerArchitecture, pages 1–12. ACM, 2017.

[27] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son,Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, andChangkyu Choi. Learning to quantize deep networks by op-timizing quantization intervals with task loss. In CVPR, June2019.

[28] Diederik P. Kingma and Jimmy Ba. Adam: A method forstochastic optimization. In ICLR, May 2015.

[29] Raghuraman Krishnamoorthi. Quantizing deep convolu-tional networks for efficient inference: A whitepaper. arXivpreprint arXiv:1806.08342, 2018.

[30] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Thecifar-10 dataset. online: http://www. cs. toronto. edu/kriz/-cifar. html, page 4, 2014.

[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.Imagenet classification with deep convolutional neural net-works. In Proceedings of the 25th International Confer-ence on Neural Information Processing Systems - Volume 1,NIPS’12, 2012.

[32] Patrice Simard Kumar Chellapilla, Sidd Puri. High perfor-mance convolutional neural networks for document process-ing. In Tenth International Workshop on Frontiers in Hand-writing Recognition, La Baule, France, 2006.

[33] Urs Kster, Tristan J. Webb, Xin Wang, Marcel Nassar,Arjun K. Bansal, William H. Constable, Ouz H. Elibol,Scott Gray, Stewart Hall, Luke Hornof, Amir Khosrowshahi,Carey Kloss, Ruby J. Pai, and Naveen Rao. Flexpoint: Anadaptive numerical format for efficient training of deep neu-ral networks. In Proceedings of the 31st Annual Conferenceon Neural Information Processing Systems, Dec 2017.

[34] Hao Li, Soham De, Zheng Xu, Christoph Studer, HananSamet, and Tom Goldstein. Training quantized nets: Adeeper understanding. In Advances in Neural InformationProcessing Systems 30, pages 5811–5821. 2017.

[35] Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, JunjieYan, and Rui Fan. Fully quantized network for object de-tection. In The IEEE Conference on Computer Vision andPattern Recognition (CVPR), June 2019.

[36] Xiaqing Li, Guangyan Zhang, H Howie Huang, ZhufanWang, and Weimin Zheng. Performance analysis of gpu-based convolutional neural networks. In 2016 45th Inter-national Conference on Parallel Processing (ICPP), pages67–76. IEEE, 2016.

[37] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, andPiotr Dollar. Focal loss for dense object detection. In ICCV,Oct 2017.

[38] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,Pietro Perona, Deva Ramanan, Piotr Dollar, and C. LawrenceZitnick. Microsoft coco: Common objects in context. InDavid Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuyte-laars, editors, Computer Vision – ECCV 2014, pages 740–755. Springer International Publishing, 2014.

[39] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun.Adaptive gradient methods with dynamic bound of learningrate. In ICLR, May 2019.

[40] Jeffrey L. McKinstry, Steven K. Esser, Rathinakumar Ap-puswamy, Deepika Bablani, John V. Arthur, Izzet B. Yildiz,and Dharmendra S. Modha. Discovering low-precision net-

works close to full-precision networks for efficient embed-ded inference. arXiv preprint arXiv:1809.04191, 2018.

[41] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gre-gory Diamos, Erich Elsen, David Garcia, Boris Ginsburg,Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, andHao Wuv. Mixed precision training. In ICLR, May 2018.

[42] Asit Mishra and Debbie Marr. Apprentice: Using knowledgedistillation techniques to improve low-precision network ac-curacy. arXiv preprint arXiv:1711.05852, 2017.

[43] NVIDIA Corporation. cuDNN Documentation.https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html.

[44] NVIDIA Corporation. PTX ISA. https://docs.nvidia.com/cuda/parallel-thread-execution/.

[45] S. K. Park and K. W. Miller. Random number generators:Good ones are hard to find. Commun. ACM.

[46] Adam Paszke, Sam Gross, Soumith Chintala, GregoryChanan, Edward Yang, Zachary DeVito, Zeming Lin, Al-ban Desmaison, Luca Antiga, and Adam Lerer. Automaticdifferentiation in pytorch. 2017.

[47] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon,and Ali Farhadi. Xnor-net: Imagenet classification using bi-nary convolutional neural networks. Lecture Notes in Com-puter Science, page 525542, 2016.

[48] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On theconvergence of adam and beyond. In ICLR, May 2018.

[49] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.Faster r-cnn: Towards real-time object detection with re-gion proposal networks. In Proceedings of the 29th AnnualConference on Neural Information Processing Systems, Dec2015.

[50] Charbel Sakr and Naresh Shanbhag. Per-tensor fixed-pointquantization of the back-propagation algorithm. In ICLR,May 2019.

[51] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh-moginov, and Liang-Chieh Chen. Mobilenetv2: Invertedresiduals and linear bottlenecks. In CVPR, June 2018.

[52] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,Jonathon Shlens, and Zbigniew Wojna. Rethinking the in-ception architecture for computer vision. 2016 IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),Jun 2016.

[53] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and SongHan. Haq: Hardware-aware automated quantization. arXivpreprint arXiv:1811.08886, 2018.

[54] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-YuChen, and Kailash Gopalakrishnan. Training deep neuralnetworks with 8-bit floating point numbers. In Proceedingsof the 32nd Annual Conference on Neural Information Pro-cessing Systems, Dec 2018.

[55] Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang,Yang Liu, and Jian Cheng. Two-step quantization for low-bitneural networks. IEEE CVPR, June 2018.

[56] Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Train-ing and inference with integers in deep neural networks. InICLR, May 2018.

[57] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li,Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quanti-zation networks. In CVPR, June 2019.

[58] Yukuan Yang, Shuang Wu, Lei Deng, Tianyi Yan, Yuan Xie,and Guoqi Li. Training high-performance and large-scaledeep neural networks with full 8-bit integers, 2019.

[59] Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher,Yingyong Qi, and Jack Xin. Blended coarse gradient descentfor full quantization of deep neural networks. Research in theMathematical Sciences, 6(1):14, 2019.

[60] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and GangHua. Lq-nets: Learned quantization for highly accurate andcompact deep neural networks. In ECCV, September 2018.

[61] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, andYurong Chen. Incremental network quantization: Towardslossless cnns with low-precision weights. arXiv preprintarXiv:1702.03044, 2017.

[62] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen,and Yuheng Zou. Dorefa-net: Training low bitwidth convo-lutional neural networks with low bitwidth gradients. CoRR,abs/1606.06160, 2016.

6. Supplementary Material6.1. Proof of Theorem 1

Assumption 1. ft is convex;

Assumption 2. ∀wp,wq ∈ S, ‖wp −wq‖∞ ≤ D∞.

Proof. Considering the update for ith entry of weight,

wt+1,i = wt,i − ηt,igt,i (9)

we have

(wt+1,i − w∗i )2 = (wt,i − ηt,i gt,i − w∗i )2

= (wt,i − w∗i )2 − 2(wt,i − w∗i )ηt,i gt,i + η2t,i g2t,i

(10)

Rearrange the equation, and divide 2ηt,i on both side as ηt,iis none-zero,

gt,i(wt,i − w∗i ) =1

2ηt,i(wt,i − w∗i )2 +

ηt,i2g2t,i

− 1

2ηt,i(wt+1,i − w∗i )2

(11)

The error of quantized gradients is defined as

εt,i = gt,i − gt,i (12)

Replace gt,i in the (11) with gt,i and εt,i, and we can getthat

gt,i(wt,i − w∗i ) =1

2ηt,i[(wt,i − w∗i )2 − (wt+1,i − w∗i )2)]

+ εt,i(wt,i − w∗i ) +ηt,i2

(gt,i − εt,i)2(13)

According to assumption 1,

ft(wt)− ft(w∗) ≤ g>t (wt −w∗) (14)

So combine the (13) and (14), sum over the d dimensionsof w and the T iterations, then the regret

R(T ) ≤T∑t=1

d∑i=1

(1

2ηt,i[(wt,i − w∗i )2 − (wt+1,i − w∗i )2]

+ εt,i(wt,i − w∗i ) +ηt,i2

(gt,i − εt,i)2)

=

d∑i=1

[1

2η1,i(w1,i − w∗i )2 −

1

2ηT,i(wT+1,i − w∗i )2]

+

T∑t=2

d∑i=1

(1

2ηt,i− 1

2ηt−1,i)(wt,i − w∗i )2

+

T∑t=1

d∑i=1

[εt,i(wt,i − w∗i ) +ηt,i2

(gt,i − εt,i)2]

(15)

Combine (15) with the assumption 2, and we can furtherrelax the above (15) to

R(T ) ≤d∑i=1

D2∞

2η1,i+

T∑t=2

d∑i=1

(1

2ηt,i− 1

2ηt−1,i)D2∞

+

T∑t=1

d∑i=1

[εt,i(wt,i − w∗i ) +ηt,i2

(gt,i − εt,i)2]

(16)

Assume that all layers have the same learning rate, then

R(T ) ≤ d D2∞

2ηT+

T∑t=1

εt(wt −w∗) +

T∑t=1

ηt2(gt − εt)2

(17)

Based on Cauchy’s inequality and assumption 2, we finallyget

R(T ) ≤ d D2∞

2ηT+

T∑t=1

‖εt‖ · ‖wt −w∗‖+T∑t=1

ηt2‖gt − εt‖2

≤ d D2∞

2ηT+D∞

T∑t=1

‖εt‖+T∑t=1

ηt2‖gt‖2

(18)

Thus the average regret

R(T )

T≤ d D2

∞2TηT︸ ︷︷ ︸(1)

+D∞T

T∑t=1

‖εt‖︸ ︷︷ ︸(2)

+1

T

T∑t=1

ηt2‖gt‖2︸ ︷︷ ︸

(3)

(19)

6.2. INT8 Training Stability

We plot the accuracy and the loss curve of MobileNetV2training on CIFAR-10 dataset and ResNet-50 training onImageNet dataset to show the stability of INT8 training.From Figure 9 and Figure 10, we can see that our methodmakes INT8 training smooth and achieves accuracy com-parable to FP32 training. The quantization noise increasesexploratory ability of INT8 training since the quantizationnoise at early stage of training could make the optimizationdirection more diverse, and with properly reduced learn-ing rate, INT8 training sometimes even converge faster thanFP32 training.

6.3. INT8 Convolution Speed Up Algorithm

6.3.1 INT8 Convolution

On NVIDIA GPUs with Pascal architectures (such asGP102, GP104, and GP106), the new 8-bit integer 4-element dot product with accumulation (DP4A) [44] in-struction is supported. This enables the NVIDIA GeForce

0 100 200 300Epoch

0

25

50

75

100

Accuracy Naive INT8

Our INT8FP32

(a) the accuracy curve

0 100 200 300Epoch

0

2

∞

Loss

Naive INT8Our INT8FP32

(b) the loss curve

Figure 9. Comparison of INT8 training and FP32 training onCIFAR-10 using MobileNetV2.

0 50 100 150Epoch

0

25

50

75

Accuracy

Naive INT8Our INT8FP32

(a) the accuracy curve

0 50 100 150Epoch

2.5

5.0

7.5

Loss

Naive INT8Our INT8FP32

(b) the loss curve

Figure 10. Comparison of INT8 training and FP32 training on Im-ageNet using ResNet-50.

GTX 1080Ti (based on GP102) to achieve a peak inte-ger throughput of 44 Tera Operations Per Second (TOPS),while the peak float throughput is only 11 Tera Float Oper-ations Per Second (TFLOPS).

Since the release of cuDNN 6.0 [43], INT8 inferenceis supported but the INT8 backward process is not imple-mented. So we use the DP4A instruction to implement theINT8 backward process by ourselves. Moreover, we findthat the quantization process before INT8 convolution com-putation is pretty time-consuming as the quantization needsto read and write the whole data. In order to reduce theoverhead that quantization brings, we fuse the quantizationprocess with the convolution computation (quantization-convolution fused kernel). In Figure 11, we can see that thecombination of quantization and convolution could avoidone extra global memory read and write effectively. Thuswe rewrite the INT8 forward and backward process usingthis quantization-convolution fused kernel and achieve asignificant speed-up.

In our implementation, we transpose the data layout intoNC4HW so that we can use the DP4A instruction to conductthe convolution computation. We use the prmt instructionin Parallel Thread Execution and Instruction Set Architec-ture (PTX ISA) [44] to transpose the data efficiently. Thisprmt instruction picks four arbitrary bytes from two 32-bit registers, and reassembles them into a 32-bit destinationregister. Figure 12 shows that one thread can transpose datain 4×4 8-bit integer block by using 12 prmt instructionswith shared memory. The transpose implementation code is

GLOBALMEMORY READ

CONVOLUTION

GLOBALMEMORYWRITE

GLOBALMEMORY READ

QUANTIZATION

GLOBALMEMORYWRITE

GLOBALMEMORY READ

QUANTIZATION+CONVOLUTION

GLOBALMEMORYWRITE

OUTPUT

OUTPUT

INPUT

INPUT

Figure 11. Quantization-convolution fused kernel avoids one extraglobal memory read and write.

A0 A1 A2 A3

B0 B1 B2 B3

C0 C1 C2 C3

D0 D1 D2 D3

A0 B0 C0 D0

A1 B1 C1 D1

A2 B2 C2 D2

A3 B3 C3 D3

Figure 12. 4×4 8-bit integer block transpose in a thread usingprmt instruction.

listed below.

int regLDG[4]; int4 regPRMT; int tmp;asm volatile("prmt.b32 %0, %1, %2,

0x0040;" : "=r"(regPRMT.x) :"r"(regLDG[0]), "r"(regLDG[1]));

asm volatile("prmt.b32 %0, %1, %2,0x0040;" : "=r"(tmp) : "r"(regLDG[2]),"r"(regLDG[3]));

asm volatile("prmt.b32 %0, %1, %2,0x5410;" : "=r"(regPRMT.x) :"r"(regPRMT.x), "r"(tmp));

asm volatile("prmt.b32 %0, %1, %2,0x0051;" : "=r"(regPRMT.y) :"r"(regLDG[0]), "r"(regLDG[1]));

asm volatile("prmt.b32 %0, %1, %2,0x0051;" : "=r"(tmp) : "r"(regLDG[2]),"r"(regLDG[3]));

asm volatile("prmt.b32 %0, %1, %2,0x5410;" : "=r"(regPRMT.y) :"r"(regPRMT.y), "r"(tmp));

asm volatile("prmt.b32 %0, %1, %2,0x0062;" : "=r"(regPRMT.z) :"r"(regLDG[0]), "r"(regLDG[1]));

asm volatile("prmt.b32 %0, %1, %2,0x0062;" : "=r"(tmp) : "r"(regLDG[2]),"r"(regLDG[3]));

asm volatile("prmt.b32 %0, %1, %2,0x5410;" : "=r"(regPRMT.z) :"r"(regPRMT.z), "r"(tmp));

asm volatile("prmt.b32 %0, %1, %2,

0x0073;" : "=r"(regPRMT.w) :"r"(regLDG[0]), "r"(regLDG[1]));

asm volatile("prmt.b32 %0, %1, %2,0x0073;" : "=r"(tmp) : "r"(regLDG[2]),"r"(regLDG[3]));

asm volatile("prmt.b32 %0, %1, %2,0x5410;" : "=r"(regPRMT.w) :"r"(regPRMT.w), "r"(tmp));

After transposition, we use two kinds of algorithmsim2col plus GEMM [25, 32] and implicit GEMM [43] toimplement convolution, and choose a faster algorithm foreach convolution layer before training. Through these twoalgorithms, we convert the original convolution into dotproduct. Then we use one float load instruction to load fourINT8 data and one DP4A instruction to compute four INT8dot product operations. This can speed up the INT8 convo-lution significantly.

6.3.2 Stochastic Rounding

Due to the use of stochastic rounding in quantizing gradi-ents, we need to generate uniform random numbers dur-ing the backward process. One way to generate ran-dom numbers is using curandGenerator, but this instruc-tion needs extra global memory access, which will signif-icantly degrade our INT8 convolution performance, withtime consumption increasing over 100%. Another methodis to use curand uniform, and we need to set a uniquecurandState for each thread to get different random num-bers, which requires a large amount of gpu memory. Worsestill, this method runs as slow as the first method. Consid-ering both disadvantages above, we use Linear Congruen-tial Generator (LCG) [45] to yield a sequence of pseudo-randomized numbers instead.

The generator is defined by recurrence relation,

Xn+1 = (aXn + c) mod m, (20)

where X is the sequence of pseudo-random values, m isthe modules, a is the multiplier, c is the increment, and X0

is the random seed. The parameters a, c and m are set toconstants.

In order to get different random seeds in each thread,we set the random seed X0 to first input data and add thethread index to X0. With above settings, each thread canget a unique random seed. The LCG method generates ran-dom numbers quickly and brings slight time consumptionto INT8 convolution.