arX

iv:1

509.

0552

0v1

[cs.

CV

] 18

Sep

201

51

An Experimental Survey on Correlation Filter-basedTracking

Zhe Chen, Zhibin Hong, and Dacheng Tao,Fellow, IEEE

Abstract—Over these years, Correlation Filter-based Trackers(CFTs) have aroused increasing interests in the field of visualobject tracking, and have achieved extremely compelling resultsin different competitions and benchmarks. In this paper, ourgoal is to review the developments of CFTs with extensiveexperimental results. 11 trackers are surveyed in our work,basedon which a general framework is summarized. Furthermore,we investigate different training schemes for correlationfilters,and also discuss various effective improvements that have beenmade recently. Comprehensive experiments have been conductedto evaluate the effectiveness and efficiency of the surveyedCFTs, and comparisons have been made with other competingtrackers. The experimental results have shown that state-of-art performence, in terms of robustness, speed and accuracy,can be achieved by several recent CFTs, such as MUSTer andSAMF. We find that further improvements for correlation filte r-based tracking can be made on estimating scales, applying part-based tracking strategy and cooperating with long-term trackingmethods.

Index Terms—Visual object tracking, correlation filters, track-ing evaluation, computer vision

I. I NTRODUCTION

V ISUAL object tracking is one of the most challengingtasks in the field of computer vision and is related to

a wide range of applications like surveillance and robotics.Given the initial state of a target in the first frame, the goaloftracking is to predict states of the target in a video. However,designing a fast and robust tracker is difficult according tovar-ious critical issues in visual tracking, such as illumination vari-ations, occlusions, deformations, rotations and so on. Over thepast decade, various tracking algorithms have been proposedto cope with these challenges, some of which use generativemodels [1]–[6] while the others use discriminative models [7]–[12]. Generative trackers perform tracking by searching thebest-matching windows, and discriminative methods learn todistinguish the target from backgrounds. In [13], [14], it hasbeen found that background information is advantageous foreffective tracking, which suggests that discriminative methodsare more competing. In particular, the correlation filter-baseddiscriminative trackers have made significant achievementsrecently, and have been paid more attention by correspondingresearchers. Therefore, summarizing the developments of cor-relation filter-based tracking algorithms and comparing themwith other popular trackers are supposed to be conducive forfuture researches.

Conventionally, correlation filters are designed to producecorrelation peaks for each interested target in the scene whileyielding low responses to background, which are usually usedas detectors of expected patterns. Although localization tasks

can be effectively performed by these filters, the required train-ing needs used to make them inappropriate for online tracking.Only after the proposal of Minimum Output Sum of SquaredError (MOSSE) [15] filter, this situation has been changed.Using an adaptive training scheme, MOSSE is considerablyrobust and efficient in tracking. Based on the basic frameworkof MOSSE filter, numerous improvements have been madelater. For example, Henriqueset al. [16] improved the MOSSEfilter by introducing kernel methods, and Danelljanet al. [17]applied color-attributes to better represent the input data. Byfurther handling the scale changes, three Correlation Filter-based Trackers (CFTs), namely SAMF [18], DSST [19] andan improved KCF [20], have achieved state-of-art results andhave beaten all other attended trackers in terms of accuracyina recent competition [21]. With more CFTs developed recently[22]–[25], correlation filter-based tracking has proven its greatstrengths in efficiency and robustness, and has considerablyaccelerated the development of visual object tracking

Despite the various correlation filter-based tracking algo-rithms proposed these years, there is no work to review themwith comprehensive evaluations. To facilitate other researchersfor future contributions, our fundamental goals of this paper in-clude: 1) formulating a general framework; 2) investigating themajor developments of CFTs; 3) carrying out comprehensiveevaluations on a large scale benchmark; 4) making appropriatecomparisons; and 5) illustrating future research directions.

In this work, various important CFTs are surveyed and theircontributions are discussed in detail. Brief introductions ofthese studies can be found in Table I. In general, trainingschemes of filters are extremely crucial in correlation filter-based tracking, and CFTs can be further improved by intro-ducing better training schemes, extracting powerful features,relieving scaling issue, applying part-based tracking strategyand cooperating with long-term tracking. To evaluate the track-ing performance, we have collected source codes of 8 CFTsfrom the internet, and implemented two CFTs in a simpleversion. By running on a large scale benchmark [13], [14], theperformance of tested CFTs is compared with other popularcompeting trackers. The obtained experimental results arepresented and analyzed, proving the efficiency and robustnessof correlation filter-based tracking methods. According totheconducted experiments, latest CFTs are demonstrated to bestate-of-art trackers.

The rest of this paper is arranged as follows. In SectionII, we provide an overview of the basic framework of cor-relation filter-based tracking methods. Afterwards, theoriesand schemes for training correlation filters are introducedinSection III. For Section IV, numerous aspects of further im-

2

TABLE IMAJOR SURVEYED PAPERS

Name Published Year Major ContributionMOSSE and Regularized ASEF [15] 2010 Pioneering work of introducing correlation filters for visual tracking

CSK [16] 2012 Introduced Ridge Regression problem with circulant matrixto apply kernel methodsSTC [26] 2014 Introduced spatio-temporal context informationKCF [20] 2014 Formulated the work of CSK and introduced multi-channel HOGfeature.CN [17] 2014 Introduced color attributes as effective features

DSST [19] 2014 Relieved the scaling issue using feature pyramid and 3-dimensional correlation filterSAMF [18] 2014 Integrated both color feature and HOG feature; Applied a scaling pool to handle scale

variationsRPAC [24]1 2015 Introduced part-based tracking strategyRPT [25] 2015 Introduced reliable local patches to facilitate trackingLCT [23] 2015 Introduced online random fern classifier as re-detection component for long-term

trackingMUSTer [22] 2015 Proposed a biology-inspired framework where short-term processing and long-term

processing are cooperated with each other1This abbreviation is taken from its title:Real-time Part-based visual tracking via Adaptive Correlation filters.

provements are reviewed and discussed in detail. Afterwards,experimental results are presented and analyzed in SectionV.In the end, conclusions and future trends are summarized inSection VI.

II. CORRELATION FILTER-BASED TRACKING FRAMEWORK

According to the existing correlation filter-based trackingmethods, the general working framework can be summarizedas follows. Initially, correlation filter is trained with imagepatch cropped from a given position of the target at first frame.Then in each subsequent time step, the patch at previouspredicted position is cropped for detection. Afterwards, asshown in Figure 1, various features can be extracted fromthe raw input data, and a cosine window is usually appliedfor smoothing the boundary effects. Subsequently, efficientcorrelation operations are performed by replacing the ex-hausted convolutions with element-wise multiplications usingDiscrete Fourier Transform (DFT). In practice, the DFT ofa vector is computed by the efficient Fast Fourier Transform(FFT) algorithm. Following the correlation procedure, a spatialconfidence map, or response map, can be obtained usinginverse FFT. The position with a maximum value in this mapis then predicted as the new state of target. Next, appearanceat the estimated position is extracted for training and updatingthe correlation filter. Because only the DFT of correlation filteris required for detection, training and updating procedures areall performed in frequency domain.

To describe the workflow mathematically, letx be the inputof detection stage andh be the correlation filter. In practice,xcan be either raw image patch or extracted features. Supposethe symbol represents the Fourier transform of a vector. Ac-cording to Convolution Theorem, circulant convolution equalsto element-wise multiplication in frequency domain

x⊗ h = F−1(

x⊙ h∗)

(1)

whereF−1 is inverse Fourier transform operation,⊙ denoteselement-wise multiplication and∗ means the complex conju-gate. The results of (1) are the expected correlation outputbetweenx andh, which also form the mentioned confidencemap.

For training the filter, let us first define a desired correlationoutputy. Using the new instancex′ of target, correlation filterh should satisfy:

y = F−1(

x′ ⊙ h∗)

(2)

and thus:

h∗ =y

x′(3)

wherey is the DFT ofy and the division is computed element-wise.

In terms of computation cost, the complexity of circu-lar convolution for an image of sizen × n is O(n4) ,while the element-wise multiplications using FFT only requireO(n2 logn). Therefore the acceleration brought by FFT issignificant.

However, there are some issues that should be handled wellwhen using the correlation filter-based tracking framework.First, training schemes are extremely crucial for CFTs. Sincethe target may change its appearance continuously, correlationfilters should be adaptively trained and updated on-the-flyto adapt to the new appearance of target. Second, featurerepresenting methods also greatly influence the performance.Although raw pixels can be directly used for detection, thetracker may be affected by various noises like illuminationchanges and motion blurs. More powerful features are sup-posed to be helpful. Moreover, how to adapt to the scalesof target is another challenging problem for CFTs. Sincethe sizes of correlation filters are usually fixed in tracking,scale variations of the target cannot be handled well in thesetrackers. As a result, an effective scale estimation approachis supposed to complement this shortage of correlation filter-based tracking. Furthermore, long-term tracking is believed tobe the weakness of many CFTs since they commonly lackthe ability to re-locate the target after drifting. By cooperatingwith long-term tracking methods, CFTs can be much morerobust in tracking.

III. T RAINING SCHEMES FOR CORRELATION FILTERS

The behaviors of correlation filters can be diversified ifdifferent methods are used for training.To train a robust

3

FFT IFFT

FFT

FFT

Initialization

Current Input Extract Features Response Map Prediction

Extract

Features

Desired Output

Train and

Update

Correlation Filter

(FFT)Cosine Window

Fig. 1. General workflow for typical correlation filter-based tracking methods. At each frame after initialization, an image patch at previous estimated positionis cropped as current input. Subsequently, visual featurescan be extracted for better describe the input, and a cosine window is usually applied for smoothingthe discontinuities at window boundaries. Afterwards, correlation between current input and the learned filter is performed in frequency domain based onConvolution Theorem. The symbol⊙ in the figure denotes element-wise computation, andFFT means Fast Fourier Transform. After the correlation, a spatialconfidence map is obtained by Inverse FFT (IFFT), whose peak can be predicted as the new position of target. Lastly, appearance at the newly estimatedposition is extracted for training and updating the correlation filter with a desired output.

correlation filter for online visual tracking, numerous studieshave been proposed.

A. Traditional Training Methods

For the simplest case, template cropped from an imagecan be used to produce peaks for the target. However, theirresponses to background patterns are also relatively high.Toovercome this issue, a variety of correlation filters [27]–[31]were trained by suppressing responses to negative trainingsamples while maintaining high response to the target. Themain difference among these filters is the methods they areconstructed with the collected training samples. For example,Synthetic Discriminant Functions (SDF) [27], [32], OptimalTradeoff Filters (OTF) [28] and Minimum Average Corre-lation Energy (MACE) [29] are trained with enforced hardconstraints so that peaks would always be produced in thesame height. On the contrary, hard constraints are believedto be unnecessary in other filters, such as Maximum AverageCorrelation Height (MACH) [30] and Unconstrained MACE(UMACE) [31]. These filters are trained by relaxing the hardconstraints. More details about developments of correlationfilters can be found in the survey [33]. Recently, a correlationfilter, which is named as Average of Synthetic Exact Filters(ASEF) [34], averages all the trained exact filters to obtainageneral one. The resulted filter has shown to perform well ineye localization [34] and pedestrian detection [35]. AlthoughASEF may be robust enough to be applied in visual tracking,a large number of samples are required for training, whichmakes it too slow for online tasks.

B. Adaptive Correlation Filters

To train correlation filters more efficiently, a novel filtertermed as Minimum Output Sum of Squared Error (MOSSE)was developed by Bolmeet al. [15], together with an improvedversion of ASEF.

1) MOSSE:According to (2) and (3), a simple filter can beobtained on samplex with the corresponding desired outputy.However, more samples are needed to improve the robustnessof correlation filters. To properly map these input samples todesired outputs, MOSSE finds a filterh by minimizing the sumof squared error between actual correlation outputs and desiredcorrelation outputs. By computing in frequency domain, thisminimization problem can be expressed by:

minh∗

∑

i

‖xi ⊙ h∗ − yi‖2

(4)

wherei indexes each training image. Then the solution ofh∗

is given by:

h∗ =

∑

i yi ⊙ x∗i

∑

i xi ⊙ x∗i

(5)

whose detailed derivations can be found in [15].In general, the desired outputy can take any shape. In

MOSSE, it is generated from ground truth with a compact 2DGaussian shaped distribution whose peak is at the center. IfKronecker delta function is used for definingy, whose value attarget center is one and values elsewhere are zero, the resultedfilter is theoretically a UMACE filter mentioned above. ThusUMACE is a special case of MOSSE.

2) Regularized ASEF:By slightly modifying the originalform, ASEF is also capable of efficient tracking. Using one

4

sample at a time, a filter called exact filter can be found bysolving (4):

h∗i =

yi ⊙ x∗i

xi ⊙ x∗i

(6)

Then a more general filter can be produced by averagingall the computed exact filters:

h∗i =

1

N

∑

i

yi ⊙ x∗i

xi ⊙ x∗i

(7)

However, original ASEF can be much unstable becausethe denominator in (7) may be extremely small. To helpproduce a more stable filter, a regularization parameterǫ canbe introduced in the denominator to prevent it from beinga close-to-zero number, which has shown to be effective forstabilization.

C. Kernelized Correlation Filters

After the success of [15], [34], correlation filter-basedtracking framework has shown to be significantly efficientfor robust tracking. However, the overall performance may belimited because the ASEF and MOSSE filters can be viewed assimple linear classifiers. By taking advantage of kernel trick,correlation filters are supposed to be more powerful.

There have already been some studies [36]–[38] to applykernel methods in correlation filters. According to [36], [37],it has been found that filters which do not use the powerspectrum or image translations are easier to be kernelized.Different from these studies, Henriqueset al. [16], [20] pro-posed that correlation filters can be effectively kernelized withthe introduction of Ridge Regression problem and circulantmatrix.

1) Ridge Regression Problem:By considering correlationfilters as classifiers, they can be trained by finding the relationbetweeni-th input xi and its labelyi from a training set.Suppose the relation takes the formf(xi) = yi, trainingproblem can be viewed as minimizing the objective function:

minw

∑

i

L (f (w,xi) , yi) + λ‖w‖2 (8)

wherew denotes the parameters,λ is regularization parameterto prevent overfitting, andL(·) is loss function. In SVM,L(·) is defined by hinge lossL(f(w,xi), yi) = max (0, 1 −yif(w,xi)), while Regularized Least Squares (RLS) whichuses quadratic lossL(f(w,xi), yi) = (yi − f(w,xi))

2 canbe alternatively applied for training filters. It has been shownthat training by RLS can deliver equivalent performance withhinge loss [39]. The RLS is also known as Ridge Regression.

For the functionf(xi), it can be a linear operationf(xi) =〈w,xi〉+ b where〈·, ·〉 is dot product andb is constant offset.By solving (8), the parameterw can be given in a closed form[39]:

w =(

XTX + λI)−1

XTy (9)

whereX is a matrix whose rows are training samples,y is avector of corresponding labels, andI is identity matrix. It isworth noting that if the computation is performed in frequencydomain,XT should be replaced by the Hermitian transposeof X in (9), which isXH = (X∗)T .

To introduce the kernel functions for improving perfor-mance, input datax can be mapped to a non-linear-featurespace withϕ(x), andw can be expressed by linear combi-nation of the inputsw =

∑

i αiϕ(xi). Thenf(xi) takes theform:

f(xi) =

n∑

j=1

αiκ(xi,xj) (10)

whereκ(xi,xj) = 〈ϕ(xi), ϕ(xj)〉 is the kernel function. Sup-poseK is the kernel matrix with its elementsKij = κ(xi,xj).The solution of (8) using kernel functions can be given by [39]:

α = (K + λI)−1

y (11)

whereI is identity matrix. To avoid difficulty in computinginverse matrix of (11), circulant matrix can be introduced.

2) Circulant Matrix: Generally, samples are obtained byrandom sampling [7], [40]–[43]. With the help of circulantmatrix, however, all the translated samples around the targetcan be collected for training without sacrificing much speed.

With a base samplex = (x0, . . . , xn−1), a circulant matrixX has the following form:

X = C(x) =

x0 x1 . . . xn−1

xn−1 x0 . . . xn−2

......

. . ....

x1 x2 . . . x0

(12)

There are various interesting properties of circulant matri-ces. For example, their sums, products and inverses are alsocirculant. In addition, a circulant matrix can be made diagonalwith the DFT of its base vectorx [44]:

X = Fdiag(x)FH (13)

whereF is DFT matrix, which is used for computing the DFTof an vectorF(z) =

√nFz. Then the solution ofw can be

expressed in the form:

w = Fdiag

(

x

x∗ ⊙ x+ λ

)

FHy (14)

which is equivalent to a simpler form in frequency domain:

w =x∗ ⊙ y

x∗ ⊙ x+ λ(15)

where the division is performed element-wise . Similarly,α

can also be computed efficiently if the kernel matrixK iscirculant:

α = F(

diag(k+ λ))−1

FHg (16)

wherek is the base vector of circulant matrixK, and further:

α =y

k+ λ(17)

where the division is also element-wise.It has been proven that the kernel function of a circulant

kernel matrix should be unitarily invariant (detailed proof canbe found in [16], [20]). Since dot-product and radial basiskernel functions are found to satisfy this condition, polynomialkernels and Gaussian kernels are usually applied.

5

If the kernelk is computed betweenx andx′, a polynomialkernelkxx′

= (xTx′ + a)b can be expressed as:

kxx′

=(

F−1(x∗ ⊙ x′) + a)b

(18)

and the Gaussian kernelkxx′

= exp(

− 1σ2 (‖x− x′‖2)

)

canbe computed by:

kxx′

= exp

(

− 1

σ2

(

‖x‖2 + ‖x′‖2 − 2F−1 (x∗ ⊙ x′))

)

(19)All the derivations of equations (8) to (19) can be found in

[16], [20].3) Detection: In a new frame, the target can be detected

by the trained parameterα and a maintained base samplex.If the new sample isz, a confidence mapy can be obtainedby:

y = C(kxz)α = F−1(

kxz ⊙ α

)

(20)

Similar with ASEF and MOSSE, the position with a maxi-mum value iny can be predicted as new position of the target.

D. Dense Spatio-Temporal Context Tracker

Spatio-Temporal Context (STC) tracker proposed by [26]was developed to exploit the use of context information. Weconsider it as another CFT since it follows a similar workflowdescribed in Section II.

Context information has already been considered in varioustrackers [45]–[48]. In the majority of these studies, key pointsaround the target are first extracted and then descriptors likeSURF and SIFT are introduced to describe these consistent re-gions. However, crucial information can be ignored sometimesby these methods and they are also quite time-consuming.Therefore the fundamental goal of STC is to use contextinformation more efficiently.

Instead of training by optimizing, STC is designed to learn alikelihood distribution, which is defined as the prior possibilityof object locating in positionp (p ∈ R2):

ℓ(p) = P (p|o) (21)

whereℓ(·) means likelihood ando is the object present in thescene.

Let po denote the position of targets center, andΩc(p)denote the neighboring coordinates aroundpo. Then a contextfeature set can be defined byP c = c(p′) = (I(p′),p′)|p′ ∈Ωc(po) whereI(p′) represents the image intensity at positionpo. By marginalizing the likelihood distribution ofc(p′) giveno:

ℓ(p) = P (p|o)=

∑

c(p′)∈P c

P (p, c(p′)|o)

=∑

c(p′)∈P c

P (p|c(p′), o)P (c(p′)|o) (22)

whereP (p|c(p′), o) models the relationship between spatialcontext information and target location, andP (c(p′)|o) mod-els the appearance of object.

Since there is no direct expression ofP (p|c(p′), o), let usdefine a function to describe it:

P (p|c(p′), o) = h(p− p′) (23)

whereh can be some operations which take the difference oftwo vectorsp andp′ as its input. To relieve the ambiguitiescaused by similar objects in the neighborhood,h should not beradially symmetric. In other words,h(p−p′) andh(|p−p′|)should not equal to each other.

For P (c(p′)|o), it can be defined as:

P (c(p′)|o) = I(p′)ωσ(p′ − po) (24)

where I(·) is image intensity andωσ(·) denotes weightedGaussian function defined by:

ωσ(p′ − po) = a exp

(

− 1

σ2‖p′ − po‖2

)

(25)

where a is a normalization parameter. Given this Gaussiandistributed weights, contexts closer to the center of object areassigned with larger values while further contexts are assignedwith smaller values. Therefore the tracker pays more attentionon the central area.

For training theh(p−p′), a desired output distributionℓ(p)can be designed by hand. If the object is known to be at thecenter of scene,ℓ(p) can be defined by:

ℓ(p) = P (p|o) = b exp

(

−‖p− po

α‖β)

(26)

whereb is also a normalization parameter,α is scale parameterandβ controls the shape of this distribution. Subsequently, wehave:

ℓ(p) = b exp

(

−‖p− po

α‖β)

=∑

c(p′)∈P c

h(p− p′)I(p′)ωσ(p′ − po)

= h(p)⊗ (I(p)ωσ(p− po)) (27)

By introducing Convolution Theorem, we have:

h(p) = F−1

(

F(

b exp(

−‖p−po

α‖β))

F (I(p)ωσ(p− po))

)

(28)

where division is performed element-wise. With a trainedh(p), ℓ(p) of the new frame can be calculated by:

ℓ(p) = F−1(

F(

h(p)t−1)

⊙F(

I(p)tωσ(p− pt−1o )

))

(29)wheret represents current time step.

Similarly, a positionp with the maximum value inℓ(p) canbe viewed as the new position of the object.

E. Updating Scheme

According to the introduced training schemes, each framecan produce a correlation filter, thus the strategy of combiningit to existing trained filter is crucial for constructing a robustappearance model.

6

In CFTs, running average is usually applied for updating,though different algorithms may average over different com-ponents. For regularized ASEF, a general correlation filterisupdated by averaging every learned exact filter:

h∗t = η

yt ⊙ x∗t

xt ⊙ x∗t + ǫ

+ (1− η)h∗t−1 (30)

where t denotes thet-th frame andη is learning rate. STCalso updates its filter based on the form of (30).

Instead, MOSSE respectively averages the numerator andthe denominator of (9):

h∗t = At

Bt

At = η(yt ⊙ x∗t ) + (1− η)At−1

Bt = η(xt ⊙ x∗t ) + (1− η)Bt−1

(31)

For KCF, the dual space coefficientsα can be updated infrequency domain:

αt = ηy

kt + λ+ (1− η)αt−1 (32)

whosekt is averaged by:

kt = ηkzz + (1− η)kt−1 (33)

wherez is the new sample extracted from currently predictedposition.

Generally, CFTs use similar updating schemes describedabove, and sometimes slight modifications can be made toimprove the performance. Given an example, Danelljanetal. [17] modified the updating scheme of CSK tracker [16](original version of KCF) by using the cost function withweighted average quadratic error for training. Moreover, robustupdating schemes can also be achieved by considering long-term tracking. If the target is lost or occluded, learning theappearance model is obviously harmful. To avoid learning thefalse positive samples, some studies have introduced long-term components with failure detection schemes [22]–[25].Forinstance, the tracker of [24] stops updating if occlusions aredetected, and the tracker of [22] refreshes the correlationfilterif the prediction of long-term component is more confident.Experiments have shown that the detection of occlusions isextremely beneficial.

F. Comparisons of Different Training Schemes

Training schemes discussed in this section include ASEF,MOSSE, Kernelized Correlation Filter (KCF) and STC. Ingeneral, they all follow the workflow described in SectionII, where computations based on Convolution Theorem areemployed for detection and the filter is trained with a desiredoutput. However, there are some differences among theseCFTs.

For ASEF tracker, its filter is produced by averaging over allthe learned filters, while MOSSE filter is trained by averagingover all the images. By introducing Ridge Regression problemand circulant matrices, kernelized correlation filters canbeintroduced for tracking. Theoretically, the linear kernelin KCFcan be the same with MOSSE filter if multiple samples ofsingle channel are used for training. If multiple channels are

supported, the linear kernel, which is called Dual CorrelationFilter (DCF) [20], can be trained by only using a singlesample. The general case which use several multi-channelsamples to train filters requires expensive computation costs,it is inappropriate in online visual tracking.

The differences between STC and other introduced trainingschemes include the following aspects. First, STC is developedto model the relationships between the object and its local spa-tial contexts, while common CFTs model the input appearancewith trained filters. Second, values of the confidence map inSTC can be referred to as prior probabilities given the currentobject, while values in confidence maps of other CFTs arecorrelation scores. Third, the algorithm of STC has the abilityof estimating scale variations, which is difficult for CFTs likeMOSSE and KCF. More arguments can be found in [26].

IV. FURTHER IMPROVEMENTS

Instead of proposing a novel training scheme of correlationfilter, there are various aspects that can improve the robustnessof CFTs. Over the years, improvements have been mainlymade on representing features, handling scale variations,ap-plying part-based strategy and cooperating with long-termtracking.

A. Feature Representation

In earlier CFTs like MOSSE and CSK, raw pixels aredirectly used for tracking. However, noises brought by rawimages extremely limit the tracking performance. Althoughshifting the input data to a zero-mean distribution or mul-tiplying it with Hanning window can slightly resist thesenoises, more powerful features are still needed for furtherimprovements.

Apparently, features with multiple channels can be morerepresentative and informative. In KCF, integrating them issimple and efficient. For Gaussian kernel function, vectorsfrom different channels can be simply added together:

kxx′

= exp

(

− 1

σ2

(

‖x‖2 + ‖x′‖2 − 2F−1(

∑

cx∗c ⊙ x′

c

))

)

(34)where c denotes the number of channels. With the multi-channel kernel functions, the famous HOG feature [49] hasbeen successfully applied in KCF trackers with superior per-formance.

Besides HOG, color attributes are also believed to bebeneficial [17]. Color attributes [50], or Color Names (CN),are the names of different colors defined by humans. InEnglish, it has been concluded that there are 11 basic colorterms [50], which include white, black, blue and so on.A map between the RGB combinations and linguistic colorattributes can be found in [50], which is trained with imagesretrieved from Google-image search. Using the map, RGBvalues can be associated with a probabilistic 11 dimensionalcolor vector with unit length. By further proposing an adaptivedimensionality reduction technique, the resulted trackercanachieve state-of-art accuracy with a considerably high speed,which is over 100 FPS.

7

To some extents, two features are complementary to eachother. HOG feature is mainly applied for analyzing the imagegradients, while CN feature focuses on color representations.Based on the efficiency of integrating multi-channel data in(34), both HOG feature and CN feature can be fused togetherto facilitate robust tracking [18].

B. Handling Scale Variations

Conventional CFTs, such as MOSSE and KCF, mainlyemploy fixed-sized windows for tracking, and they are unableto deal with target changes. To handle the scale variations,numerous algorithms have been proposed.

In SAMF and DSST, a searching strategy is applied to es-timate scales of the target. In specific, windows with differentsizes are sampled around the target, and are correlated withthe learned filter. Subsequently, the window with the highestcorrelation score can be predicted as the new state.

Suppose the size of a windowi is denoted by a 2-dimensional vectorsi. Let so denote a template window size,then we can havesi = aiso whereai denotes a scale parametergiven by a scaling poolS = a1, a2, . . . , aN of N positivenumbers. As shown in Figure 2, the size of current targetcan be estimated by searching the window with a maximumcorrelation score among the sampled windows. In SAMF,S

is set by constant values ranged from 0.985 to 1.015, andin DSST, S =

an|n = ⌊−N−12 ⌋, . . . , ⌊N−1

2 ⌋

. The majordifference between the two trackers is that SAMF processesone window at a time while a 3-dimensional correlation filteris employed to search the best scale in DSST. Some recentCFTs [22] and [23] also apply the scaling pool method.

t-1

t

Scaling Pool

Fig. 2. Workflow of estimating scales using the searching strategy. Each timea new frame comes, windows with different sizes are cropped around previousposition. By being correlated with a trained correlation filter h, correspondingconfidence mapsmi(i = 1, . . . , N) can be obtained. Then the window thatcan produce maximum confidence score is estimated as the new scale.

Different from using scaling pool, scale varying issue canalso be solved by some part-based tracking methods. In [24],the whole target is determined through a Bayesian framework.In another part-based tracker [25], a statistical method isemployed, which records and sorts the variations of relativepositions between different sub-patches to estimate scales. Inpractice, the performance of these methods is also promising.

In addition, it is worth mentioning that STC has its ownscheme to deal with scale variations. According to the formu-las used in Section III-D, suppose the new estimated center of

the target ispo, andℓ(po) is its computed confidence score.Then scales can be estimated by:

s′t =

√

ℓ ((po)t)

ℓ ((po)t−1)(35)

where s′t is the predicted scale at timet. To smooth the

predictions, estimated scales are averaged overn consecutiveframes, and linear interpolation is used for prediction:

s = 1n

∑n

i=1 s′t−i

st+1 = (1 − λ)st + λs(36)

whereλ is a fixed parameter. With the estimated size of thetarget, the parameterσ of weight function (25) is also requiredto be updated2:

σt+1 = stσt (37)

While conducting the experiments, we found that the esti-mation in STC is sometimes unstable, because the computeds′t can be extremely large if the denominator of (35) is close

to zero.

C. Part-based Tracking

Instead of learning a holistic appearance model, variouspart-based tracking algorithms [51]–[56] have been proposed,in which the target is tracked by its local appearance. Ifthe target is partially occluded, its remaining visible partscan still represent the target and thus the tracker is able tocontinue tracking. In [13], [14], experimental results haveshown that the local representations are effective for objecttracking. Therefore introducing part-based tracking strategy inCFTs is supposed to be advantageous.

Recently, [24], [25] have made successful attempts to applypart-based tracking strategy to CFTs. In [24], 5 parts of thetarget are independently tracked by KCF trackers. When anew frame comes, confidence maps of these tracked parts arefirst computed. By assigning adaptive weights to these maps,ajoint map can be constructed to predict new state using particlefilter method. Another tracker, which is called Reliable PatchTrackers (RPT) [25], also exploits the use of local contextsand treats KCF as its base tracker. However, the trackedparts in RPT are automatically selected by sampling, whosereliabilities are estimated on-the-fly. A reliable patch isdefinedas being trackable and sticking on the target. If a part is nolonger reliable, it will be discarded and re-sampled aroundthetarget. After obtaining the tracking results of reliable patches,new state of the target is predicted by a Hough Voting-likescheme.

In general, part-based tracking strategy can be helpful togain robustness against partial occlusions. The main difficultyof developing a robust part-based CFT is how to designan appropriate mechanism to handle multiple results fromdifferent tracked parts. According to the introduced studies,particle filter method has proven to be an effective solution.

2Detailed derivations of (35), (36) and (37) can be found athttp://www4.comp.polyu.edu.hk/∼cslzhang/STC/STC.htm

8

D. Long-term Tracking

One other vital challenge in visual tracking is the absenceof the target. If the target partially or fully disappears from theview, conventional CFTs can be easily distracted by irrelevantobjects because they do not contain a long-term component.As a consequence, introducing long-term tracking methods isbelieved to be favorable for improving correlation filter-basedtracking methods.

For long-term tracking, there exists several studies [45],[57]–[61], some of which introduce a re-detection modulewhile the others attempt to learn conservative appearance ofthe target. For example, TLD tracker [57] trains a detectorwith an expert of false negative samples and an expert offalse positive samples. If the tracking module in TLD fails,this trained detector can then re-initialize the tracker. On theother hand, the tracker of [59] conservatively learns the targetappearance from reliable frames with a self-paced learningscheme. Regarding to CFTs, two recent studies [22], [23] havesuccessfully cooperated CFTs with long-term tracking.

Inspired by a biological memory model called Atkinson-Shiffrin Memory Model (ASMM) [62], a MUlti-Store Tracker(MUSTer) based on a cooperative tracking framework was pro-posed [22]. In ASMM, there are short-term memory and long-term memory in human brains. Short-term memory, whichupdates aggressively and forgets information quickly, storeslocal and temporal information, while long-term memory,which updates conservatively and maintains information for along time, retains general and reliable information. With short-term and long-term memory working together, both efficiencyand robustness can be achieved. By considering CFTs asefficient short-term trackers, introducing long-term trackingmethods is supposed to complement the shortage of CFTs. InMUSTer, the long-term part is a key points-based method. Inthe course of tracking, key points of the target are maintainedor discarded based on a forgetting curve, and then retrievedforlocating the target if the CFT fails. Experiments have shownthat MUSTer have surpassed various state-of-art trackers indifferent benchmarks.

Another method of introducing long-term tracking wasproposed by Maet al. [23]. In this method, a re-detectioncomponent is added into the tracking system. Similar to TLDtracker, the re-detection procedure is carried out based onan online random fern classifier, whose training samples arecollected by a k-nearest neighbour (kNN) classifier. With thiscooperation, the resulted tracker, namely LCT, has shown tobe able to handle well with long term tracking.

V. EXPERIMENTS

In this section, both quantitative and qualitative experimentshave been conducted on large scale benchmarks to evaluatethe advantages brought by correlation filter-based trackingframework. To carry out comprehensive and fair comparisons,additional 29 popular trackers are evaluated as well, andparameters of all the evaluated trackers are set as default andfixed during the experiments. The hardware we have used inthe evaluation is a cluster node (3.4GHz Intel Xeon CPU, 8cores 32GB RAM).

A. Experiment Setup

1) Compared Trackers:For CFTs, trackers with currentlyavailable source codes are selected in our evaluations (exceptASEF and MOSSE are implemented by ourselves), whichare regularized ASEF [15], [34], MOSSE3 [15], CSK (withraw pixels) [16], KCF4 (with HOG features) [20], CN5 [17],DSST6 [19], SAMF7 [18], STC8 [26], MUSTer9 [22] andRPT10 [25]. Other competing trackers used for comparisonsinclude 28 trackers from the code library of Online ObjectTracking Benchmark (OOTB)11 [13], [14] and a recent state-of-art tracker MEEM12 [63].

2) Test Sequences:All the test sequences in our evaluationcome from OOTB [13], [14]. In original OOTB, there are 51different tracking tasks with fully annotated attributes,whichinclude scale variations, illumination variations, rotations andso on. Then in later OOTB, the number of tasks has beenextended to 100. In our experiments, original OOTB is mainlyused to compare trackers since it is more representative.The later OOTB is used as extended dataset to verify theperformance of CFTs.

3) Evaluation Methods:Following the protocol proposedin [13], One-Pass Evaluation (OPE), Temporal RobustnessEvaluation (TRE) and Spatial Robustness Evaluation (SRE)are performed in our evaluation. OPE is a traditional evaluationmethod which runs trackers on each sequence for once.For TRE, it runs trackers on 20 sub-sequences segmentedfrom the original sequence with different lengths, and SREevaluates trackers by initializing them with slightly shifted orscaled ground truth bounding boxes. With TRE and SRE, therobustness of each evaluated trackers can be comprehensivelyinterpreted.

After running the trackers, precision plots and successplots are applied to present results. Instead of using averageEuclidean distances between the predicted centers to ground-truth centers, precision plots show percentages of frameswhose estimated locations lie in a given threshold distancetoground-truth centers. Regarding to success plots, an overlapscore is introduced to represent performance. Letrt denotethe area of tracked bounding box andra denote the groundtruth. An Overlap Score (OS) can be defined byS = |rt∩ra|

|rt∪ra|where∩ and∩ are the intersection and union of two regions,and | · | counts the number of pixels in the correspondingarea. Afterwards, a frame whose OS is larger than a thresholdis termed as a successful frame, and the ratios of successful

3Original implementation can be found inhttp://www.cs.colostate.edu/∼vision/ocof toolset 2012/index.php

4http://home.isr.uc.pt/∼henriques/circulant/5http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A711538&dswid=-

74926https://github.com/gnebehay/DSST7https://github.com/ihpdep/samf8http://www4.comp.polyu.edu.hk/∼cslzhang/STC/STC.htm (Note that STC

may crash during the scale estimations, thus we simply fix theparametersbefore the detected errors)

9https://sites.google.com/site/zhibinhong4131/Projects/muster10https://github.com/ihpdep/rpt11http://cvlab.hanyang.ac.kr/trackerbenchmark/12http://cs-people.bu.edu/jmzhang/MEEM/MEEM.html

9

Fig. 3. Plots of OPE, SRE and TRE in OOTB. Trackers with best 10scores are presented in the legends.

Fig. 4. Attribute-based success plots of SRE. The trackers with best 10 AUC scores are presented in the legends.

frames at the thresholds ranged from0 to 1 are plotted insuccess plots.

Furthermore, trackers are ranked in both plots, and first10 are presented. In precision plots, the precisions at thethreshold of 20 pixels are used for ranking, while Area UnderCurve (AUC) is used for ranking in success plots. Since AUS

calculates overall performance, it is more representativeforestimating the robustness of trackers.

B. Quantitative Evaluation

1) Overall Performance:Using 51 sequences from OOTB,the overall performance of all the 39 trackers are obtained and

10

Fig. 5. Extended results for 10 CFTs and 4 indicate trackers.Plots of OPE, SRE and TRE in OOTB100 are presented, as well as the scores and ranks of allthe 14 trackers.

TABLE IIADDITIONAL RESULTS OFOPE. AVERAGE CENTER LOCATION ERRORS(CLE), AVERAGE OVERLAP SCORES(OS)AND AVERAGE SPEEDS(FPS)ARE

PRESENTED. IN EACH ENTRY, SCORE IN THE TOP COMES FROMOOTB WITH 51 TASKS, WHILE THE SCORE IN THE BOTTOM COMES FROMOOTB100.THE FIRST AND SECOND BEST SCORES ARE HIGHLIGHTED BY BOLD AND UNDERLINE.

Tracker MUSTer RPT SAMF MEEM DSST KCF ASLA Struck TLD CSK MOSSE STC CN ASEFCLE 17.3

31.736.736.3

28.935.9

22.328.8

41.350.8

35.545.3

73.175.5

50.649.6

48.154.9

88.8306

82.899.6

68.380.4

64.881.8

91.6130

OS(%) 65.058.3

58.253.9

57.954.8

57.953.6

56.152.2

51.948.0

43.940.6

47.845.9

44.142.4

40.138.4

31.829.1

35.131.2

44.842.4

29.927.4

FPS 3.853.94

3.703.63

15.815.1

19.320.0

32.731.5

191183

7.484.75

20.420.7

33.340.9

288282

281284

580578

142132

324320

presented in Figure 3.According to the presented results, it can be found that

CFTs, such as MUSTer, RPT and SAMF, perform considerablywell in these plots. Particularly, the recent proposed trackerMUSTer has outperformed other trackers in all the successplots (64.1% in OPE, 56.4% in SRE and 61.7% in TRE). Inprecision plots, MUSTer can also achieve state-of-art resultsdespite that MEEM has a 1% higher score in TRE.

Moreover, there are six improved CFTs that can alwayscarry out top-10 performance, which indicates that these CFTsare considerably robust in tracking.

2) Attribute-based Evaluation:With the annotated at-tributes of each sequence, the performance of evaluated track-ers with respect to different challenges is revealed. Theseinvolved challenges are mainly caused by three factors, whichare varying appearance of the target, severe surrounding envi-ronments and the limitations of the cameras. The SRE resultsof success plots are presented in Figure 4.

The challenges brought by varying appearances of the targetare scale variation, out-of-plane rotation, in-plane rotation,deformation, and fast motion. In scale variations, MUSTer,DSST and SAMF perform the best with around 50% overlapscore, which suggests that the searching strategy introduced

in Section IV-B is effective. In the meantime, the score ofRPT (48.8%) proves that its scale estimation scheme is alsoapplicable. For rotations and deformations, MUSTer is shownto be the most robust tracker, and RPT, SAMF, MEEM alsoperform well. While in the fast motion evaluation, the bestscore is carried out by MEEM (52.7%), which is 2.4% higherthan the score of MUSTer. As a conclusion, the improvedCFTs, which are MUSTer, RPT, DSST and SAMF, can learn aconsiderably robust appearance model, while MEEM is shownto be better at locating the fast moving targets.

Another group of challenges are caused by surroundingenvironments, including illumination variations, occlusionsand background clutters. It can be found that CFTs stillproduce better results and can exclude other trackers fromthe first two ranks, which implies that background contextscan be efficiently identified to help avoid distractions usingcorrelation filter-based tracking algorithms.

The rest of the evaluated attributes are motion blur, out ofview and low resolution, which are mainly brought by the lim-itations of cameras. In motion blur and out of view challenges,the multi-expert restoration scheme has made MEEM the mostrobust among tested trackers, while MUSTer maintains the firstrank in low resolution. It is worth mentioning that RPT has

11

dropped a lot in low resolution tracking, which only achieves34.3% success rates and 6th rank in top 10. This may suggestthat tracking with local appearance need high resolution forbetter performance.

To sum up, recent CFTs such as MUSTer, RPT, SAMF,DSST and KCF can all perform well in various challengingtasks, and the strengths of using correlation filters for trackingare shown to be significant. In particular, MUSTer becomes themost robust tracker in our evaluation since it wins 8 challengesout of 11 challenges.

3) Extended Experiments:For a better interpretation oftested CFTs, a larger OOTB with 100 sequences, which canbe denoted as OOTB100, is used for extended experiments.The overall performance can be found in Figure 5. To betterrepresent the performance of CFTs, 4 other competing trackersare additionally selected as indicators. These selected trackersare Struck and ASLA, which use a single online classifier fortracking, and MEEM and TLD, which apply multiple onlineclassifiers.

According to the presented results, some aspects of suc-cessful improvements on CFTs can be revealed. First, thetraining scheme of MOSSE, which averages over all thesamples, is shown to be better than training scheme of ASEF,which averages over learned filters. Second, spatial contextsused in STC are also shown to be beneficial. Furthermore,introducing kernel methods has made CSK and KCF muchmore competing, and has helped CFTs like MUSTer andSAMF become state-of-art trackers. On the other hand, colorattributes used in CN tracker and HOG feature used in KCF arealso shown to be advantageous. By integrating both features,SAMF further improves the overall performance. Moreover,improvements based on relieving the scaling issue, applyingpart-based strategy and introducing long-term tracking areproven to be effective as well.

In comparison with other four competing trackers, improvedCFTs like MUSTer and SAMF still perform well in OOTB100,despite that the best results in precision plots of OPE andTRE are achieved by MEEM. This may because its restorationmethod can help MEEM quickly re-locate the target afterdrifting.

Besides the plots, additional statistical data of trackersunderOPE can be found in Table II, where average Center LocationError (CLE), average Overlap Score (OS) and average speedsof the tested trackers are presented. According to the results,MUSTer and MEEM deliver the most precise results, andhigher overlap scores are achieved by MUSTer, RPT andSAMF. By further observing the speeds of presented trackers,it can be found that better scores are often carried out byslower trackers, which suggests that acceleration is stillanundergoing topic.

C. Qualitative Evaluation

To evaluate the actual tracking results, we have randomly se-lected 20 video sequences from OOTB-100 and used them forqualitative evaluation. These videos includeCouple, CarScale,Bolt, Car4, Liquor, Board, Walking2, Singer1, Soccer, Car1,Girl2, Sylvester, MotorRolling, Jogging2, Rubik, Freeman1,

FaceOcc2, Deer, Basketball. In these videos, all the challengeattributes are properly included. Top-6 trackers in OOTB100,which are MUSTer, RPT, SAMF, MEEM, DSST and KCF, areselected for comparison, whose predicted boxes at the seven-eighth of the tested sequences are presented in Figure 6.

For KCF tracker, its limitation of using fixed windows canbe revealed inCarScaleand Singer1, where the predictedboxes are obviously incompatible with the target. By relievingthe scaling issue, DSST and SAMF track better in sequenceslike Car4 andCarScale. However, lacking of long-term com-ponent has made them unable to re-locate missed targets, suchas the failures inFreeman1. With long-term consideration, theresults of MUSTer are much better. For example, while othertested trackers lose the target inMotorRolling, MUSTer is ableto find the target and restart tracking. The only failed sequencefor MUSTer is theBoard, in which background objects arequite similar to the target. In addition to MUSTer, MEEMalso has the ability to re-locate the target, but its performanceis limited when dealing with light variations regarding to thefailure in Singer1. For RPT, as discussed in the attribute-basedevaluations, its tracking results become worse if the videoquality is rather low, and RPT fails inBolt from the verybeginning, which suggests that the initialization may be crucialfor RPT.

Overall, although the qualitative performance of evaluatedtrackers is promising, there is still plenty of room to improvethe robustness of CFTs as well as other trackers. The imple-mentation of MUSTer may provide a promising direction forfurther research.

VI. CONCLUSION

In this paper, we have reviewed numerous correlation filter-based tracking algorithms, and have conducted comprehensiveexperiments have been conducted. According to the experi-mental results, the efficiency and robustness of tracking withcorrelation filters have been verified. In specific, state-of-artperformance can be achieved by improved CFTs like MUSTer,RPT, SAMF and DSST. Furthermore, MUSTer has shown tobe the most robust tracker in the experiments.

To further improve CFTs, some valuable points can beconcluded based on the obtained results. First, it can be foundthat the possibilities of drifting are significantly reduced withpowerful features, as shown by improved performance of CNand KCF. It is worthy of trying more types of features. Next,solving the scaling issue is also a vital direction for improvingCFTs. Although experimental results have shown that bothscaling pool and part-based methods are helpful for handlingscale variations, approaches with less processing time arefavorable. Regarding to the part-based tracking strategy,itsadvantages have not been fully exploited in RPT according toits performance. A more effective mechanism to fuse trackingresults of different parts is desired. Lastly, introducingefficientlong-term tracking methods is another promising direction.With the outstanding performance brought by MUSTer, long-term tracking is shown to be the most helpful complement ofcorrelation filter-based tracking. Further studies can be madeon improving the accuracy of long-term tracking component,

12

Basketball Board Bolt Car1 Car4

CarScale Couple David Deer FaceOcc2

Freeman1 Girl2 Jogging-2 Liquor MotorRolling

Rubik Singer1 Soccer Sylvester Walking2

Fig. 6. Qualitative results of 6 selected trackers in 20 sequences. The name of each sequence is located on top of the corresponding figure. Frames arecollected from seven-eighth of the sequences.

developing more appropriate architectures and accelerating theoverall system.

REFERENCES

[1] J. Kwon and K. M. Lee, “Tracking by sampling trackers,” inComputerVision (ICCV), 2011 IEEE International Conference on. IEEE, 2011,pp. 1195–1202.

[2] L. Sevilla-Lara and E. Learned-Miller, “Distribution fields for tracking,”in Computer Vision and Pattern Recognition (CVPR), 2012 IEEEConference on. IEEE, 2012, pp. 1910–1917.

[3] T. Zhang, B. Ghanem, S. Liu, and N. Ahuja, “Robust visual tracking viamulti-task sparse learning,” inComputer Vision and Pattern Recognition(CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 2042–2049.

[4] T. Liu, G. Wang, L. Wang, and K. L. Chan, “Visual tracking viatemporally smooth sparse coding,” 2012.

[5] V. Belagiannis, F. Schubert, N. Navab, and S. Ilic, “Segmentation basedparticle filtering for real-time 2d object tracking,” inComputer Vision–ECCV 2012. Springer, 2012, pp. 842–855.

[6] S. Kwak, W. Nam, B. Han, and J. H. Han, “Learning occlusionwithlikelihoods for visual tracking,” inComputer Vision (ICCV), 2011 IEEEInternational Conference on. IEEE, 2011, pp. 1551–1558.

[7] S. Hare, A. Saffari, and P. H. Torr, “Struck: Structured output trackingwith kernels,” in Computer Vision (ICCV), 2011 IEEE InternationalConference on. IEEE, 2011, pp. 263–270.

[8] S. Lucey, “Enforcing non-positive weights for stable support vectortracking,” in Computer Vision and Pattern Recognition, 2008. CVPR2008. IEEE Conference on. IEEE, 2008, pp. 1–8.

[9] X. Wang, G. Hua, and T. X. Han, “Discriminative tracking by metriclearning,” in Computer Vision–ECCV 2010. Springer, 2010, pp. 200–214.

[10] Z. Kalal, J. Matas, and K. Mikolajczyk, “Pn learning: Bootstrappingbinary classifiers by structural constraints,” inComputer Vision andPattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010,pp. 49–56.

[11] K. Zhang, L. Zhang, and M.-H. Yang, “Real-time compressive tracking,”in Computer Vision–ECCV 2012. Springer, 2012, pp. 864–877.

[12] Q. Wang, F. Chen, W. Xu, and M.-H. Yang, “Object trackingwith jointoptimization of representation and classification,”Circuits and Systemsfor Video Technology, IEEE Transactions on, vol. 25, no. 4, pp. 638–650,2015.

[13] Y. Wu, J. Lim, and M.-H. Yang, “Online object tracking: Abenchmark,”in Computer vision and pattern recognition (CVPR), 2013 IEEE Con-ference on. IEEE, 2013, pp. 2411–2418.

[14] ——, “Object tracking benchmark,” 2015.[15] D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual

object tracking using adaptive correlation filters,” inComputer Visionand Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE,2010, pp. 2544–2550.

[16] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista,“Exploiting thecirculant structure of tracking-by-detection with kernels,” in ComputerVision–ECCV 2012. Springer, 2012, pp. 702–715.

13

[17] M. Danelljan, F. S. Khan, M. Felsberg, and J. v. d. Weijer, “Adaptivecolor attributes for real-time visual tracking,” inComputer Vision andPattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014,pp. 1090–1097.

[18] Y. Li and J. Zhu, “A scale adaptive kernel correlation filter trackerwith feature integration,” inComputer Vision-ECCV 2014 Workshops.Springer, 2014, pp. 254–265.

[19] M. Danelljan, G. Hager, F. S. Khan, and M. Felsberg, “Accurate scaleestimation for robust visual tracking,” inProceedings of the BritishMachine Vision Conference BMVC, 2014.

[20] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista,“High-speedtracking with kernelized correlation filters,” 2014.

[21] F. LIRIS, “The visual object tracking vot2014 challenge results.”[22] Z. Hong, Z. Chen, C. Wang, X. Mei, D. Prokhorov, and D. Tao, “Multi-

store tracker (muster): A cognitive psychology inspired approach toobject tracking,” inProceedings of the IEEE Conference on ComputerVision and Pattern Recognition, 2015, pp. 749–758.

[23] C. Ma, X. Yang, C. Zhang, and M.-H. Yang, “Long-term correlationtracking,” in Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, 2015, pp. 5388–5396.

[24] T. Liu, G. Wang, and Q. Yang, “Real-time part-based visual trackingvia adaptive correlation filters,” inProceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, 2015, pp. 4902–4912.

[25] Y. Li, J. Zhu, and S. C. Hoi, “Reliable patch trackers: Robust visualtracking by exploiting reliable patches,” inProceedings of the IEEEConference on Computer Vision and Pattern Recognition, 2015, pp. 353–361.

[26] K. Zhang, L. Zhang, Q. Liu, D. Zhang, and M.-H. Yang, “Fast vi-sual tracking via dense spatio-temporal context learning,” in ComputerVision–ECCV 2014. Springer, 2014, pp. 127–141.

[27] C. F. Hester and D. Casasent, “Multivariant technique for multiclasspattern recognition,”Applied Optics, vol. 19, no. 11, pp. 1758–1761,1980.

[28] P. Refregier, “Optimal trade-off filters for noise robustness, sharpnessof the correlation peak, and horner efficiency,”Optics Letters, vol. 16,no. 11, pp. 829–831, 1991.

[29] A. Mahalanobis, B. Vijaya Kumar, and D. Casasent, “Minimum averagecorrelation energy filters,”Applied Optics, vol. 26, no. 17, pp. 3633–3640, 1987.

[30] M. Savvides and B. Kumar, “Efficient design of advanced correlationfilters for robust distortion-tolerant face recognition,”in Advanced Videoand Signal Based Surveillance, 2003. Proceedings. IEEE Conference on.IEEE, 2003, pp. 45–52.

[31] A. Mahalanobis, B. Vijaya Kumar, S. Song, S. Sims, and J.Epperson,“Unconstrained correlation filters,”Applied Optics, vol. 33, no. 17, pp.3751–3759, 1994.

[32] D. Casasent, “Unified synthetic discriminant functioncomputationalformulation,” Applied Optics, vol. 23, no. 10, pp. 1620–1627, 1984.

[33] B. V. Kumar, J. A. Fernandez, A. Rodriguez, and V. N. Boddeti, “Recentadvances in correlation filter theory and application,” inSPIE Defense+Security. International Society for Optics and Photonics, 2014, pp.909 404–909 404.

[34] D. S. Bolme, B. A. Draper, and J. R. Beveridge, “Average of syntheticexact filters,” inComputer Vision and Pattern Recognition, 2009. CVPR2009. IEEE Conference on. IEEE, 2009, pp. 2105–2112.

[35] D. S. Bolme, Y. M. Lui, B. A. Draper, and J. R. Beveridge, “Simple real-time human detection using a single correlation filter,” inPerformanceEvaluation of Tracking and Surveillance (PETS-Winter), 2009 TwelfthIEEE International Workshop on. IEEE, 2009, pp. 1–8.

[36] R. Patnaik and D. Casasent, “Fast fft-based distortion-invariant kernelfilters for general object recognition,” inIS&T/SPIE Electronic Imaging.International Society for Optics and Photonics, 2009, pp. 725 202–725 202.

[37] D. Casasent and R. Patnaik, “Analysis of kernel distortion-invariantfilters,” in Optics East 2007. International Society for Optics andPhotonics, 2007, pp. 67 640Y–67 640Y.

[38] K.-H. Jeong, P. P. Pokharel, J.-W. Xu, S. Han, and J. C. Principe, “Kernelbased synthetic discriminant function for object recognition,” in Acous-tics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings.2006 IEEE International Conference on, vol. 5. IEEE, 2006, pp. V–V.

[39] R. Rifkin, G. Yeo, and T. Poggio, “Regularized least-squares classi-fication,” Nato Science Series Sub Series III Computer and SystemsSciences, vol. 190, pp. 131–154, 2003.

[40] B. Babenko, M.-H. Yang, and S. Belongie, “Robust objecttrackingwith online multiple instance learning,”Pattern Analysis and MachineIntelligence, IEEE Transactions on, vol. 33, no. 8, pp. 1619–1632, 2011.

[41] H. Grabner, C. Leistner, and H. Bischof, “Semi-supervised on-line boost-ing for robust tracking,” inComputer Vision–ECCV 2008. Springer,2008, pp. 234–247.

[42] S. Avidan, “Support vector tracking,”Pattern Analysis and MachineIntelligence, IEEE Transactions on, vol. 26, no. 8, pp. 1064–1072, 2004.

[43] A. Saffari, C. Leistner, J. Santner, M. Godec, and H. Bischof, “On-line random forests,” inComputer Vision Workshops (ICCV Workshops),2009 IEEE 12th International Conference on. IEEE, 2009, pp. 1393–1400.

[44] R. M. Gray,Toeplitz and circulant matrices: A review. now publishersinc, 2006.

[45] T. B. Dinh, N. Vo, and G. Medioni, “Context tracker: Exploringsupporters and distracters in unconstrained environments,” in ComputerVision and Pattern Recognition (CVPR), 2011 IEEE Conference on.IEEE, 2011, pp. 1177–1184.

[46] H. Grabner, J. Matas, L. Van Gool, and P. Cattin, “Tracking the invisible:Learning where the object might be,” inComputer Vision and PatternRecognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 1285–1292.

[47] L. Wen, Z. Cai, Z. Lei, D. Yi, and S. Z. Li, “Online spatio-temporalstructural context learning for visual tracking,” inComputer Vision–ECCV 2012. Springer, 2012, pp. 716–729.

[48] M. Yang, Y. Wu, and G. Hua, “Context-aware visual tracking,” PatternAnalysis and Machine Intelligence, IEEE Transactions on, vol. 31, no. 7,pp. 1195–1209, 2009.

[49] N. Dalal and B. Triggs, “Histograms of oriented gradients for humandetection,” inComputer Vision and Pattern Recognition, 2005. CVPR2005. IEEE Computer Society Conference on, vol. 1. IEEE, 2005, pp.886–893.

[50] J. Van De Weijer, C. Schmid, J. Verbeek, and D. Larlus, “Learning colornames for real-world applications,”Image Processing, IEEE Transac-tions on, vol. 18, no. 7, pp. 1512–1523, 2009.

[51] L. Cehovin, M. Kristan, and A. Leonardis, “An adaptive coupled-layervisual model for robust visual tracking,” inComputer Vision (ICCV),2011 IEEE International Conference on. IEEE, 2011, pp. 1363–1370.

[52] H. Izadinia, I. Saleemi, W. Li, and M. Shah, “2t: Multiple peoplemultiple parts tracker,” inComputer Vision–ECCV 2012. Springer,2012, pp. 100–114.

[53] D. A. Ross, J. Lim, R.-S. Lin, and M.-H. Yang, “Incremental learningfor robust visual tracking,”International Journal of Computer Vision,vol. 77, no. 1-3, pp. 125–141, 2008.

[54] G. Shu, A. Dehghan, O. Oreifej, E. Hand, and M. Shah, “Part-basedmultiple-person tracking with partial occlusion handling,” in ComputerVision and Pattern Recognition (CVPR), 2012 IEEE Conference on.IEEE, 2012, pp. 1815–1821.

[55] B. Yang and R. Nevatia, “Online learned discriminativepart-basedappearance models for multi-human tracking,” inComputer Vision–ECCV 2012. Springer, 2012, pp. 484–498.

[56] X. Jia, H. Lu, and M.-H. Yang, “Visual tracking via adaptive structurallocal sparse appearance model,” inComputer vision and pattern recog-nition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 1822–1829.

[57] Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,”Pattern Analysis and Machine Intelligence, IEEE Transactions on,vol. 34, no. 7, pp. 1409–1422, 2012.

[58] Y. Hua, K. Alahari, and C. Schmid, “Occlusion and motionreasoningfor long-term tracking,” inComputer Vision–ECCV 2014. Springer,2014, pp. 172–187.

[59] J. Supancic and D. Ramanan, “Self-paced learning for long-term track-ing,” in Computer Vision and Pattern Recognition (CVPR), 2013 IEEEConference on. IEEE, 2013, pp. 2379–2386.

[60] K. Lebeda, S. Hadfield, J. Matas, and R. Bowden, “Long-term trackingthrough failure cases,” inComputer Vision Workshops (ICCVW), 2013IEEE International Conference on. IEEE, 2013, pp. 153–160.

[61] F. Pernici and A. Del Bimbo, “Object tracking by oversampling localfeatures,”Pattern Analysis and Machine Intelligence, IEEE Transactionson, vol. 36, no. 12, pp. 2538–2551, 2014.

[62] R. C. Atkinson and R. M. Shiffrin, “Human memory: A proposed systemand its control processes,”Psychology of learning and motivation, vol. 2,pp. 89–195, 1968.

[63] J. Zhang, S. Ma, and S. Sclaroff, “Meem: Robust trackingvia multipleexperts using entropy minimization,” inComputer Vision–ECCV 2014.Springer, 2014, pp. 188–203.