ieee transactions on pattern analysis and machine...

Nonconvex Online Support Vector MachinesSeyda Ertekin, Leon Bottou, and C. Lee Giles, Fellow, IEEE

Abstract—In this paper, we propose a nonconvex online Support Vector Machine (SVM) algorithm (LASVM-NC) based on the Ramp

Loss, which has the strong ability of suppressing the influence of outliers. Then, again in the online learning setting, we propose an

outlier filtering mechanism (LASVM-I) based on approximating nonconvex behavior in convex optimization. These two algorithms are

built upon another novel SVM algorithm (LASVM-G) that is capable of generating accurate intermediate models in its iterative steps by

leveraging the duality gap. We present experimental results that demonstrate the merit of our frameworks in achieving significant

robustness to outliers in noisy data classification where mislabeled training instances are in abundance. Experimental evaluation

shows that the proposed approaches yield a more scalable online SVM algorithm with sparser models and less computational running

time, both in the training and recognition phases, without sacrificing generalization performance. We also point out the relation between

nonconvex optimization and min-margin active learning.

Index Terms—Online learning, nonconvex optimization, support vector machines, active learning.

Ç

1 INTRODUCTION

IN supervised learning systems, the generalization perfor-mance of classification algorithms is known to be greatly

improved with large margin training. Large margin classi-fiers find the maximal margin hyperplane that separates thetraining data in the appropriately chosen kernel-inducedfeature space. It is well established that if a large margin isobtained, the separating hyperplane is likely to have a smallmisclassification rate during recognition (or prediction) [1],[2], [3]. However, requiring that all instances be correctlyclassified with the specified margin often leads to overfitting,especially when the data set is noisy. Support VectorMachines [4] address this problem by using a soft margincriterion, which allows some examples to appear on thewrong side of the hyperplane (i.e., misclassified examples) inthe training phase to achieve higher generalization accuracy.With the soft margin criterion, patterns are allowed to bemisclassified for a certain cost, and consequently, theoutliers—the instances that are misclassified outside ofthe margin—start to play a dominant role in determiningthe decision hyperplane,since they tend to have the largestmargin loss according to the Hinge Loss. Nonetheless, due toits convex property and practicality, Hinge Loss has becomea commonly used loss function in SVMs.

Convexity is viewed as a virtue in the machine learningliterature both from a theoretical and experimental pointof view. Convex methods can be easily analyzed mathema-tically and bounds can be produced. Additionally, convex

solutions are guaranteed to reach global optima and are notsensitive to initial conditions. The popularity of convexityfurther increased after the success of convex algorithms,particularly with SVMs, which yield good generalizationperformance and have strong theoretical foundations.However, in convex SVM solvers, all misclassified examplesbecome support vectors, which may limit the scalability ofthe algorithm to learning from large-scale data sets. In thispaper, we show that nonconvexity can be very effective forachieving sparse and scalable solutions, particularly whenthe data consist of abundant label noise. We present hereinexperimental results that show how a nonconvex lossfunction, Ramp Loss, can be integrated into an online SVMalgorithm in order to suppress the influence of misclassifiedexamples.

Various works in the history of machine learning researchfocused on using nonconvex loss functions as an alternate toconvex Hinge Loss in large margin classifiers. While Masonet al. [5] and Krause and Singer [6] applied it to Boosting,Perez-Cruz et al. [7] and Xu and Cramer [8] proposedtraining algorithms for SVMs with the Ramp Loss and solvedthe nonconvex optimization by utilizing semidefinite pro-gramming and convex relaxation techniques. On the otherhand, some previous works of Liu et al. [9] and Wang et al.[10] used the Concave-Convex Procedure (CCCP) [11] fornonconvex optimization as the work presented here. Thosestudies are worthwhile in the endeavor of achieving sparsemodels or competitive generalization performance; never-theless, none of them are efficient in terms of computationalrunning time and scalability for real-world data miningapplications, and yet the improvement in classificationaccuracy is only marginal. Collobert et al. [12] pointed outthe scalability advantages of nonconvex approaches andused CCCP for nonconvex optimization in order to achievefaster batch SVMs and Transductive SVMs. In this paper, wefocus on bringing the scalability advantages of nonconvexityto the online learning setting by using an online SVMalgorithm, LASVM [13]. We also highlight and discuss theconnection between the nonconvex loss and traditional min-margin active learners.

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 33, NO. XX, XXXXXXX 2011 1

. S. Ertekin is with the Massachusetts Institute of Technology, Sloan Schoolof Management, Cambridge, MA 02139. E-mail: [email protected].

. L. Bottou is with the NEC Laboratories America, 4 Independence Way,Suite 200, Princeton, NJ 08540. E-mail: [email protected].

. C.L. Giles is with the College of Information Sciences and Technology, ThePennsylvania State University, University Park, PA 16802.E-mail: [email protected].

Manuscript received 15 Mar. 2009; revised 11 Dec. 2009; accepted 30 Mar.2010; published online 24 May 2010.Recommended for acceptance by O. Chapelle.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log NumberTPAMI-2009-03-0168.Digital Object Identifier no. 10.1109/TPAMI.2010.109.

0162-8828/11/$26.00 � 2011 IEEE Published by the IEEE Computer Society

Online learning offers significant computational advan-tages over batch learning algorithms and the benefits ofonline learning become more evident when dealing withstreaming or very large-scale data. Online learners incorpo-rate the information of recently observed training data intothe model via incremental model updates and without theneed for retraining it with previously seen entire trainingdata. Since these learners process the data one at a time inthe training phase, selective sampling can be applied andevaluation of the informativeness of the data prior to theprocessing by the learner becomes possible. The computa-tional benefits of avoiding periodic batch optimizations,however, necessitate the online learner fulfilling two criticalrequirements, namely, the intermediate models need to bewell enough trained in order to capture the characteristicsof the training data, but, on the other hand, should not beoveroptimized since only part of the entire training data isseen at that point in time. In this paper, we present anonline SVM algorithm, LASVM-G, that maintains a balancebetween these conditions by leveraging the duality gapbetween the primal and dual functions throughout theonline optimization steps. Based on the online trainingscheme of LASVM-G, we then present LASVM-NC, an onlineSVM algorithm with nonconvex loss function, which yieldsa significant speed improvement in training and builds asparser model, hence resulting in faster recognition than itsconvex version as well. Finally, we propose an SVMalgorithm (LASVM-I) that utilizes the selective samplingheuristic by ignoring the instances that lie in the flat regionof the Ramp Loss in advance, before they are processed bythe learner. Although this approach may appear like anoveraggressive training sample elimination process, wepoint out that these instances do not play a large role indetermining the decision hyperplane according to the RampLoss anyway. We show that for a particular case of sampleelimination scenario, misclassified instances according tothe most recent model are not taken into account in thetraining process. For another case, only the instances in themargin pass the barrier of elimination and are processed inthe training, hence leading to an extreme case of small poolactive learning framework [14] in online SVMs. Theproposed nonconvex implementation and selective sampleignoring policy yields sparser models with fewer supportvectors and faster training with less computational time andkernel computations, which overall leads to a more scalableonline SVM algorithm. The benefits of the proposedmethods are fully realized for kernel SVMs and theiradvantages become more pronounced in noisy dataclassification, where mislabeled samples are in abundance.

In the next section, we present a background on SupportVector Machines. Section 3 gives a brief overview of theonline SVM solver, LASVM [13]. We then present theproposed online SVM algorithms, LASVM-G, LASVM-NC,and LASVM-I. The paper continues with the experimentalanalysis presented in Section 8, followed by concludingremarks.

2 SUPPORT VECTOR MACHINES

Support Vector Machines [4] are well known for their strongtheoretical foundations, generalization performance, andability to handle high-dimensional data. In the binaryclassification setting, let ððx1; y1Þ � � � ðxn; ynÞÞ be the training

data set where xi are the feature vectors representing theinstances and yi 2 f�1;þ1g are the labels of those instances.Using the training set, SVM builds an optimum hyperplane—a linear discriminant in a higher dimensional featurespace—that separates the two classes by the largest margin.The SVM solution is obtained by minimizing the followingprimal objective function:

minww;b

Jðw; bÞ ¼ 1

2kwwk2 þ C

Xni¼1

�i; ð1Þ

with 8i yiðww � �ðxiÞ þ bÞ � 1� �i;�i � 0;

�

where w is the normal vector of the hyperplane, b is theoffset, yi are the labels, �ð�Þ is the mapping from input spaceto feature space, and �i are the slack variables that permitthe nonseparable case by allowing misclassification oftraining instances.

In practice, the convex quadratic programming (QP)problem in (1) is solved by optimizing the dual cost function:

max��

Gð��Þ �XNi¼1

�iyi �1

2

Xi;j

�i�jKðxi; xjÞ; ð2Þ

subject to

Pi �i ¼ 0;

Ai � �i � Bi;

Ai ¼ minð0; CyiÞ;Bi ¼ maxð0; CyiÞ;

8>>><>>>:

ð3Þ

where Kðxi; xjÞ ¼ h�ðxiÞ;�ðxjÞi is the kernel matrix repre-senting the dot products �ðxiÞ � �ðxjÞ in feature space. Weadopt a slight deviation of the coefficients �i from thestandard representation and let them inherit the signs of thelabels yi, permitting the �i to take on negative values. Aftersolving the QP problem, the norm of the hyperplane ww canbe represented as a linear combination of the vectors in thetraining set

ww ¼Xi

�i�ðxiÞ: ð4Þ

Once a model is trained, a soft margin SVM classifies apattern x according to the sign of a decision function, whichcan be represented as a kernel expansion

yðxÞ ¼Xni¼1

�iKðx; xiÞ þ b; ð5Þ

where the sign of yðxÞ represents the predicted classificationof x.

A widely popular methodology for solving the SVM QPproblem is Sequential Minimal Optimization (SMO) [15].SMO works by making successive direction searches,which involves finding a pair of instances that violate theKKT conditions and taking an optimization step along thatfeasible direction. The �� coefficients of these instances aremodified by opposite amounts, so SMO makes sure thatthe constraint

Pi �i ¼ 0 is not violated. Practical imple-

mentations of SMO select working sets based on finding apair of instances that violate the KKT conditions more than�-precision, also known as �-violating pairs [13]:

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 33, NO. XX, XXXXXXX 2011

ði; jÞ is a �-violating pair()�i < Bi;�j > Aj;gi � gj > �;

8<:

where g denotes the gradient of an instance and � is a smallpositive threshold. The algorithm terminates when all KKTviolations are below the desired precision.

The effect of the bias term. Note that the equalityconstraint on the sum of �i in (3) appears in the SVMformulation only when we allow the offset (bias) term b tobe nonzero. While there is a single “optimal” b, differentSVM implementations may choose separate ways ofadjusting the offset. For instance, it is sometimes beneficialto change b in order to adjust the number of false positivesand false negatives [2, page 203], or even disallow the biasterm completely (i.e., b ¼ 0) [16] for computational simpli-city. In SVM implementations that disallow the offset term,the constraint

Pi �i ¼ 0 is removed from the SVM problem.

The online algorithms proposed in this paper also adopt thestrategy of setting b ¼ 0. This strategy gives the algorithmsthe flexibility to update a single �i at a time at eachoptimization step, bringing computational simplicity andefficiency to the solution of the SVM problem withoutadversely affecting the classification accuracy.

3 LASVM

LASVM [13] is an efficient online SVM solver that uses lessmemory resources and trains significantly faster than otherstate-of-the-art SVM solvers while yielding competitivemisclassification rates after a single pass over the trainingexamples. LASVM realizes these benefits due to its noveloptimization steps that have been inspired by SMO. LASVM

applies the same pairwise optimization principle to onlinelearning by defining two direction search operations. Thefirst operation, PROCESS, attempts to insert a new exampleinto the set of current support vectors (SVs) by searching foran existing SV that forms a �-violating pair with maximalgradient. Once such an SV is found, LASVM performs adirection search that can potentially change the coefficient ofthe new example and make it a support vector. The secondoperation, REPROCESS, attempts to reduce the currentnumber of SVs by finding two SVs that are �-violating SVswith maximal gradient. A direction search can zero thecoefficient of one or both SVs, removing them from the set ofcurrent support vectors of the model. In short, PROCESS addsnew instances to the working set and REPROCESS removesthe ones that the learner does not benefit from anymore. Inthe online iterations, LASVM alternates between runningsingle PROCESS and REPROCESS operations. Finally, LASVM

simplifies the kernel expansion by running REPROCESS toremove all �-violating pairs from the kernel expansion, a stepknown as FINISHING. The optimizations performed in theFINISHING step reduce the number of support vectors in theSVM model.

4 LASVM WITH GAP-BASED OPTIMIZATION—LASVM-G

In this section, we present LASVM-G—an efficient onlineSVM algorithm that brings performance enhancements toLASVM. Instead of running a single REPROCESS operation

after each PROCESS step, LASVM-G adjusts the number ofREPROCESS operations at each online iteration by lever-aging the gap between the primal and the dual functions.Further, LASVM-G replaces LASVM’s one time FINISHING

optimization and cleaning stage with the optimizationsperformed in each REPROCESS cycle at each iteration andthe periodic non-SV removal steps. These improvementsenable LASVM-G to generate more reliable intermediatemodels than LASVM, which lead to sparser SVM solutionsthat can potentially have better generalization performance.For further computational efficiency, the algorithms that wepresent in the rest of the paper use the SVM formulationwith b ¼ 0. As we pointed out in Section 2, the bias term bacts as a hyperparameter that can be used to adjust thenumber of false positives and false negatives for varyingsettings of b, or to achieve algorithmic efficiency due tocomputational simplicity when b ¼ 0. In the rest of thepaper, all formulations are based on setting the bias b ¼ 0and thus optimizing a single � at a time.

4.1 Leveraging the Duality Gap

One question regarding the optimization scheme in theoriginal LASVM formulation is the rate at which to performREPROCESS operations. A straightforward approach wouldbe to perform one REPROCESS operation after each PROCESS

step, which is the default behavior of LASVM. However, thisheuristic approach may result in underoptimization of theobjective function in the intermediate steps if this rate issmaller than the optimal proportion. Another option wouldbe to run REPROCESS until a small predefined threshold "exceeds the L1 norm of the projection of the gradientð@Gð�Þ=@�iÞ, but little work has been done to determine thecorrect value of the threshold ". A geometrical argumentrelates this norm to the position of the support vectorsrelative to the margins [17]. As a consequence, one usuallychooses a relatively small threshold, typically in the range10�4-10�2. Using such a small threshold to determine the rateof REPROCESS operations results in many REPROCESS stepsafter each PROCESS operation. This will not only increase thetraining time and computational complexity, but canpotentially overoptimize the objective function at eachiteration. Since nonconvex iterations work toward suppres-sing some training instances (outliers), the intermediatelearned models should be well enough trained in order tocapture the characteristics of the training data, but, on theother hand, should not be overoptimized since only part ofthe entire training data is seen at that point in time. Therefore,it is necessary to employ a criteria to determine an accuraterate of REPROCESS operations after each PROCESS. We definethis policy as the minimization of the gap between the primal andthe dual [2].

Optimization of the duality gap. From the formulationsof the primal and dual functions in (1) and (2), respectively,it can be shown that the optimal values of the primal anddual are same [18]. Furthermore, at any nonoptimal point,the primal function is guaranteed to lie above the dual curve.In formal terms, let � and � be solutions of problems (1) and(2), respectively. The strong duality asserts that for anyfeasible � and �,

Gð�Þ � Gð�Þ ¼ Jð�Þ � Jð�Þ with � ¼Xi

�i�ðxiÞ: ð6Þ

ERTEKIN ET AL.: NONCONVEX ONLINE SUPPORT VECTOR MACHINES 3

That is, at any time during the optimization, the value of theprimal Jð�Þ is higher than the dual Gð�Þ. Using the equalityw ¼

Pl �l�ðxlÞ and b ¼ 0, we show that this holds as follows:

Jð�Þ �Gð�Þ ¼ 1

2kwk2 þ C

Xl

j1� ylðw � �ðxlÞÞjþ

�Xl

�lyl þ1

2kwk2

¼ kwk2 �Xl

�lyl þ CXl


¼ wXl

�l�ðxlÞ �Xl

�lyl

þ CXl


¼ �Xl

yl�lj1� ylðw � �ðxlÞ þ bÞjþ

þ CXl

j1� ylðw � �ðxlÞ þ bÞjþ

¼Xl

ðC � �lyl|fflfflfflfflffl{zfflfflfflfflffl}�0

Þ j1� ylðw � �ðxlÞÞjþ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}�0

� 0;

where C � �lyl � 0 is satisfied by the constraint of the dualfunction in (3). Then, the SVM solution is obtained whenone reaches ��; �� such that

" > Jð��Þ �Gð��Þ where �� ¼Xi

��i�ðxiÞ: ð7Þ

The strong duality in (6) then guarantees thatJð��Þ < Jð�Þ þ ".Few solvers implement this criterion since it requires theadditional calculation of the gap Jð�Þ �Gð�Þ. In this paper,we advocate using criterion (7) using a threshold value " thatgrows sublinearly with the number of examples. Letting "grow makes the optimization coarser when the number ofexamples increases. As a consequence, the asymptoticcomplexity of optimizations in online setting can be smallerthan that of the exact optimization.

Most SVM solvers use the dual formulation of the QPproblem. However, increasing the dual does not necessarilyreduce the duality gap. The dual function follows a nicemonotonically increasing pattern at each optimization step,whereas the primal shows significant up and down fluctua-tions. In order to keep the size of the duality gap in check,before each PROCESS operation, we compute the standarddeviation of the primal, which we call the Gap Target GG:

GG ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiXni¼1

h2i �

Pni¼1 hi

� �2

l

vuut ; ð8Þ

where l is the number of support vectors and hi ¼ Cyigiwith C and gi denoting the misclassification penalty and thegradient of instance i, respectively. After computing the gaptarget, we run a PROCESS step and check the new Gap Gbetween the primal and the dual. After an easy derivation,the gap is computed as

G ¼ �Xni¼1

ð�igi þmaxð0; CgiÞÞ: ð9Þ

Note that, as we have indicated earlier, the bias term b is set tozero in all of the formulations. In the online iterations, wecycle between running REPROCESS and computing the gap Guntil the termination criteria G � maxðC; GGÞ is reached.That is, we require the duality gap after the REPROCESS

operations to be not greater than the initial gap target GG.The C parameter is part of the equation in order to preventthe algorithm from specifying a too narrow gap targetand therefore prevent making excessive number of optimi-zation steps. The heuristic upper bound on the gap isdeveloped based on the oscillating characteristics of theprimal function during the optimization steps and theseoscillations are related to the successive choice of examplesto REPROCESS. Viewing these oscillations as noise, the gaptarget enables us to stop the REPROCESS operations whenthe difference is within the noise. After this point, thelearner continues with computing the new Gap Target andrunning PROCESS and REPROCESS operation on the nextfresh instance from the unseen example pool.

4.2 Building Blocks

The implementation of LASVM-G maintains the followingpieces of information as its key building blocks: thecoefficients �i of the current kernel expansion S, the boundsfor each �, and the partial derivatives of the instances in theexpansion, given as

gk ¼@Wð�Þ@�k

¼ yk �Xi

�iKðxi; xkÞ ¼ yk � yðxkÞ: ð10Þ

The kernel expansion here maintains all of the traininginstances in the learner’s active set, both the support vectorsand the instances with � ¼ 0.

In the online iterations of LASVM-G, the optimization isdriven by two kinds of direction searches. The first operation,PROCESS, inserts an instance into the kernel expansion andinitializes the�i and gradient gi of this instance (Step 1). Aftercomputing the step size (Step 2), it performs a directionsearch (Step 3). We set the offset term for kernel expansion bto zero for computational simplicity. As discussed in theSVM section regarding the offset term, disallowing bremoves the necessity of satisfying the constraintP

i2S �i ¼ 0, enabling the algorithm to update a single � ata time, both in PROCESS and REPROCESS operations.

LASVM-G PROCESS(i)

1) �i 0; gi yk �P

s2S �sKis

2) If gi < 0 then

� ¼ max�Ai � �i; giKii

�Else

� ¼ max�Bi � �i; giKii

�3) �i �i þ �gs gs � �Kis 8s in kernel expansion

The second operation, REPROCESS, searches all of theinstances in the kernel expansion and selects the instancewith the maximal gradient (Steps 1-3). Once an instance isselected, LASVM-G computes a step size (Step 4) andperforms a direction search (Step 5).

LASVM-G REPROCESS()

1) i arg mins2Sgs with �s > As

j arg maxs2Sgs with �s < Bs


2) Bail out if ði; jÞ is not a �-violating pair.3) If gi þ gj < 0 then g gi; t i

Else g gj; t j

4) If g < 0 then

� ¼ max�At � �t; gKtt

�Else

� ¼ min�Bt � �t; gKtt

�5) �k �k þ �gs gs � �Kts 8s in kernel expansion

Both PROCESS and REPROCESS operate on the instances inthe kernel expansion, but neither of them remove anyinstances from it. A removal step is necessary for improvedefficiency because, as the learner evolves, the instances thatwere admitted to the kernel expansion in earlier iterations assupport vectors may not serve as support vectors anymore.Keeping such instances in the kernel expansion slows downthe optimization steps without serving much benefit to thelearner and increases the application’s requirement forcomputational resources. A straightforward approach toaddress this inefficiency would be to remove all of theinstances with �i ¼ 0, namely, all nonsupport vectors in thekernel expansion. One concern with this approach is that oncean instance is removed, it will not be seen by the learner again,and thus it will no longer be eligible to become a supportvector in the later stages of training. It is important to find abalance between maintaining the efficiency of a small-sizedkernel expansion and not aggressively removing instancesfrom the kernel expansion. Therefore, the cleaning policyneeds to preserve the instances that can potentially becomeSVs at a later stage of training while removing instances thathave the lowest possibility of becoming SVs in the future.

Our cleaning procedure periodically checks the number ofnon-SVs in the kernel expansion. If the number of non-SVs nis more than the number of instances that is permitted in theexpansion m by the algorithm, CLEAN selects the extra non-SV instances with the highest gradients for removal. Itfollows from (10) that this heuristic predominantly selects themost misclassified instances that are farther away from thehyperplane for deletion from the kernel expansion. Thispolicy suppresses the influence of the outliers on the model toyield sparse and scalable SVM solutions.

CLEAN

n: number of non-SVs in the kernel expansion.

m: maximum number of allowed non-SVs.

~v: Array of partial derivatives.

1) If n < m return

2) ~v ~v [ jgijþ, 8i with �i ¼ 0

3) Sort the gradients in ~v in ascending order.

gthreshold v½m�4) If jgijþ � gthreshold then remove xi, 8i with �i ¼ 0

We want to point out that it is immaterial to distinguishwhether an instance has not been an SV for many iterationsor it has just become a non-SV. In either case, these examplesdo not currently contribute to the classifier and are treatedequally from a cleaning point of view.

Note that these algorithmic components are gearedtoward designing and online SVM solver with nonlinearkernels. Even though it is possible to apply these principles

to linear SVMs as well, the algorithmic aspects for linearSVMs will be different. For instance, for linear SVMs, it ispossible to store the normal vector w of the hyperplanedirectly instead of manipulating the support vector expan-sions. In this regard, the focus of the paper is more aboutkernel SVMs.

4.3 Online Iterations in LASVM-G

LASVM-G exhibits the same learning principle as LASVM,but in a more systematic way. Both algorithms make onepass (epoch) over the training set. Empirical evidencesuggests that a single epoch over the entire training setyields a classifier as good as the SVM solution.

Upon initialization, LASVM-G alternates between PRO-

CESS and REPROCESS steps during the epoch like LASVM, butdistributes LASVM’s one time FINISHING step to theoptimizations performed in each REPROCESS cycle at eachiteration and the periodic CLEAN operations. Anotherimportant property of LASVM-G is that it leverages the gapbetween the primal and the dual functions to determine thenumber of REPROCESS steps after each PROCESS (the -Gsuffix emphasizes this distinction). Reducing the duality gaptoo fast can cause overoptimization in early stages withoutyet observing sufficient training data. Conversely, reducingthe gap too slowly can result in underoptimization in theintermediate iterations. Fig. 1 shows that as the learner seesmore training examples, the duality gap gets smaller.


Fig. 1. The duality gap (Jð�Þ �Gð�Þ), normalized by the number oftraining instances.

The major enhancements that are introduced to LASVM

enable LASVM-G to achieve higher prediction accuraciesthan LASVM in the intermediate stages of training. Fig. 2presents a comparative analysis of LASVM-G versus LASVM

for the Adult data set (Table 2). Results on other data setsare provided in Section 8.

While both algorithms report the same generalizationperformance in the end of training, LASVM-G reaches abetter classification accuracy at an earlier point in trainingthan LASVM and is able to maintain its performancerelatively stable with a more reliable model over the courseof training. Furthermore, LASVM-G maintains fewer numberof support vectors in the intermediate training steps, asevidenced in Fig. 2b.

LASVM-G

1) Initialization:

Set � 0

2) Online Iterations:

Pick an example xiCompute Gap Target GG

Threshold maxðC; GGÞRun PROCESS(xi)

while Gap G > Threshold

Run REPROCESS

end

Periodically run CLEAN

In the next sections, we further introduce three SVMalgorithms that are implemented based on LASVM-G, namely,LASVM-NC, LASVM-I, and FULL SVM. While these SVMalgorithms share the main building blocks of LASVM-G, eachalgorithm exhibits a distinct learning principle. LASVM-NC

uses the LASVM-G methodology in a nonconvex learnersetting. LASVM-I is a learning scheme that we propose as aconvex variant of LASVM-NC that employs selective sam-pling. FULL SVM does not take advantage of the noncon-vexity or the efficiency of the CLEAN operation, and acts asa traditional online SVM solver in our experimentalevaluation.

5 NONCONVEX ONLINE SVM SOLVER—LASVM-NC

In this section, we present LASVM-NC, a nonconvex onlineSVM solver that achieves sparser SVM solutions in less timethan online convex SVMs and batch SVM solvers. We first

introduce the nonconvex Ramp Loss function and discusshow nonconvexity can overcome the scalability problems ofconvex SVM solvers. We then present the methodology tooptimize the nonconvex objective function, followed by thedescription of the online iterations of LASVM-NC.

5.1 Ramp Loss

Traditional convex SVM solvers rely on the Hinge Loss H1

(as shown in Fig. 3b) to solve the QP problem, which can berepresented in Primal form as

minww;b

Jðww; bÞ ¼ 1

2kwwk2 þ C

Xnl¼1

H1ðyifðxiÞÞ: ð11Þ

In the Hinge Loss formulation HsðzÞ ¼ maxð0; s� zÞ, s

indicates the Hinge point and the elbow at s ¼ 1 indicatesthe point at which ylf�ðxlÞ ¼ ylðw � �ðxlÞ þ bÞ ¼ 1. Assume,for simplicity, that the Hinge Loss is made differentiablewith a smooth approximation on a small interval z 2½1� �; 1þ �� near the hinge point. Differentiating (11) showsthat the minimum w must satisfy

w ¼ �CXLl¼1

ylH0

1ðylÞf�ðxxlÞ�ðxxiÞ: ð12Þ

In this setting, correctly classified instances outside ofthe margin (z � 1) cannot become SVs because H 01ðzÞ ¼ 0.On the other hand, for the training examples with ðz < 1Þ,H 01ðzÞ is 1, so they cost a penalty term at the rate ofmisclassification of those instances. One problem withHinge Loss-based optimization is that it imposes no limiton the influences of the outliers, that is, the misclassifica-tion penalty is unbounded. Furthermore, in Hinge Loss-based optimization, all misclassified training instancesbecome support vectors. Consequently, the number ofsupport vectors scales linearly with the number of trainingexamples [19]. Specifically,

#SV

#Examples! 2B�; ð13Þ

where B� is the best possible error achievable linearly in thefeature space �ð�Þ. Such a fast pace of growth of the numberof support vectors becomes prohibitive for training SVMs inlarge-scale data sets.

In practice, not all misclassified training examples arenecessarily informative to the learner. For instance, in noisydata sets, many instances with label noise become supportvectors due to misclassification, even though they are notinformative about the correct classification of new instancesin recognition. Thus, it is reasonable to limit the influence of


Fig. 2. LASVM versus LASVM-G for the Adult data set. (a) Test errorconvergence. (b) Growth of number of SVs.

Fig. 3. (a) The ramp loss can be decomposed into (b) a convex hingeloss and (c) a concave loss.

the outliers and allow the real informative traininginstances to define the model. Since Hinge Loss admits alloutliers into the SVM solution, we need to select analternative loss function that enables selectively ignoringthe instances that are misclassified according to the currentmodel. For this purpose, we propose using the Ramp Loss

RsðzÞ ¼ H1ðzÞ �HsðzÞ; ð14Þ

which allows us to control the score window for z at whichwe are willing to convert instances into support vectors.Replacing H1ðzÞ with RsðzÞ in (12), we see that the RampLoss suppresses the influence of the instances with scorez < s by not converting them into support vectors. How-ever, since Ramp Loss is nonconvex, it prohibits us fromusing widely popular optimization schemes devised forconvex functions.

While convexity has many advantages and nice mathe-matical properties, the SV scaling property in (13) may beprohibitive for large-scale learning because all misclassifiedexamples become support vectors. Since the nonconvexsolvers are not necessarily bounded by this constraint,nonconvexity has the potential to generate sparser solutions[12]. In this work, our aim is to achieve the best of bothworlds: generate a reliable and robust SVM solution that isfaster and sparser than traditional convex optimizers. Thiscan be achieved by employing the CCCP, and thus reducingthe complexity of nonconvex loss function by transformingthe problem into a difference of convex parts. The RampLoss is amenable to CCCP optimization since it can bedecomposed into a difference of convex parts (as shown inFig. 3 and (14)). The cost function Jsð��Þ for the Ramp Losscan then be represented as the sum of a convex part Jvexð��tÞand a concave part Jcavð��tÞ :

min��

Jsð��Þ ¼ 1

2kwwk2 þ C

Xnl¼1

RsðyifðxiÞÞ

¼ 1

2kwwk2 þ C

Xnl¼1

H1ðyifðxiÞÞ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

Jsvexð��Þ

� CXnl¼1

HsðyifðxiÞÞ|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}

Jscavð��Þ

:

ð15Þ

For simplification purposes, we use the notation

�l ¼ yl@Jscavð��Þ@f��ðxlÞ

¼ C; if ylf��ðxlÞ < s;0; otherwise;

�ð16Þ

where C is the misclassification penalty and f��ðxlÞ is thekernel expansion defined as in (5) with the offset term b ¼ 0.The cost function in (15), along with the notation introducedin (16), is then reformulated as the following dualoptimization problem:

max�

Gð�Þ ¼Xi

yi�i �1

2

Xi;j

�i�jKi;j;

with

Ai � �i � Bi;

Ai ¼ minð0; CyiÞ � �iyi;Bi ¼ maxð0; CyiÞ � �iyi;�i from ð16Þ:

8>>><>>>:

ð17Þ

Collobert et al. [12] use a similar formulation for the CCCP-based nonconvex optimization of batch SVMs, but there are

fundamental differences between optimization of batchSVMs and the online algorithms presented here. In parti-cular, the batch SVM needs a convex initialization step priorto nonconvex iterations to go over the entire or part of thetraining data in order to initialize the CCCP parameters toavoid getting stuck in a poor local optima. Furthermore,batch nonconvex SVMs alternate between solving (17) andupdating the �s of all training instances. On the other hand,LASVM-NC runs a few online convex iterations as theinitialization stage, and adjusts the � of only the new freshinstance based on the current model and solves (17) while theonline algorithm is progressing. Additionally, due to thenature of online learning, our learning scheme also permitsselective sampling, which will be further discussed in theLASVM-I section.

We would also like to point out that if the �s of all of thetraining instances are initialized to zero and left unchangedin the online iterations, the algorithm becomes traditionalHinge Loss SVM. From another viewpoint, if s� 0, thenthe �s will remain zero and the effect of Ramp Loss will notbe realized. Therefore, (17) can be viewed as a genericalgorithm that can act as both Hinge Loss SVM and RampLoss SVM with CCCP that enables nonconvex optimization.

5.2 Online Iterations in LASVM-NC

The online iterations in LASVM-NC are similar to LASVM-G

in the sense that they are also based on alternating PROCESS

and REPROCESS steps, with the distinction of replacing theHinge Loss with the Ramp Loss. LASVM-NC extends theLASVM-G algorithm with the computation of the ��,followed by updating the � bounds A and B as shown in(17). Note that while the �� do not explicitly appear in thePROCESS and REPROCESS algorithm blocks, they do, in fact,affect these optimization steps through the new definitionof the bounds A and B.

When a new example xi is encountered, LASVM-NC firstcomputes the�i for this instance as presented in the algorithmblock, where yi is the class label, f��ðxiÞ is the decision score forxi, and s is the score threshold for permitting instances tobecome support vectors.

We would like to point out that CCCP has convergenceguarantees (c.f. [12]), but it is necessary to initialize the CCCPalgorithm appropriately in order to avoid getting trapped inpoor local optima. In batch SVMs, this corresponds torunning classical SVM on the entire set or on a subset oftraining instances in the first iteration to initialize CCCP,followed by the nonconvex optimization in the subsequentiterations. In the online setting, we initially allow convexoptimization for the first few instances by setting their �i ¼ 0(i.e., use Hinge Loss), and then switch to nonconvex behaviorin the remainder of online iterations.

Note from (17) that the � bounds for instances with � ¼ 0follow the formulation for the traditional convex

LASVM-NC

SS: min. number of SVs to start nonconvex behavior.

1) Initialization:

Set �� 0, � 0


Pick an example xi


Set �i ¼C if yif��ðxiÞ < s and #SV > SS0 otherwise

�

Set �i bounds for xi to (minð0; CyiÞ � �iyi ��i � maxð0; CyiÞ � �iyi)Compute Gap Target GG



Run REPROCESSend


setting. On the other hand, the bounds for the instances with� ¼ C, that is, the outliers with score (z < s) are assigned newbounds based on the Ramp Loss criteria. Once LASVM-NC

establishes the � bounds for the new instance, it computesthe Gap Target GG and takes a PROCESS step. Then, it makesoptimizations of the REPROCESS kind until the size of theduality gap comes down to the Gap Threshold. Finally,LASVM-NC periodically runs the CLEAN operation to keepthe size of the kernel expansion under control and maintainits efficiency throughout the training stage.

6 LASVM AND IGNORING INSTANCES—LASVM-I

This SVM algorithm employs the Ramp function in Fig. 3aas a filter to the learner prior to the PROCESS step. That is,once the learner is presented with a new instance, it firstchecks if the instance is on the ramp region of the functionð1 > yi

Pj �jKij > sÞ. The instances that are outside of the

ramp region are not eligible to participate in the optimiza-tion steps and they are immediately discarded withoutfurther action. The rationale is that the instances that lie onthe flat regions of the Ramp function will have derivativeH 0ðzÞ ¼ 0, and based on (12), these instances will not playrole in determining the decision hyperplane ww.

The LASVM-I algorithm is based on the following recordkeeping that we conducted when running LASVM-G experi-ments. In LASVM-G and LASVM-NC, we kept track of twoimportant data points. First, we intentionally permittedevery new coming training instance into the kernel expan-sion in online iterations and recorded the position of allinstances on the Ramp Loss curve just before inserting theinstances into the expansion. Second, we kept track of thenumber of instances that were removed from the expansionwhich were on the flat region of the Ramp Loss curve whenthey were admitted. The numeric breakdown is presented inTable 1. Based on the distribution of these cleaned instances,it is evident that most of the cleaned examples that wereinitially admitted from (z > 1) region were removed from thekernel expansion with CLEAN at a later point in time. This isexpected, since the instances with (z > 1) are alreadycorrectly classified by the current model with a certainconfidence, and hence do not become SVs.

On the other hand, Table 1 shows that almost all of theinstances inserted from the left flat region (misclassifiedexamples due to z < s) became SVs in LASVM-G, andtherefore were never removed from the kernel expansion.In contrast, almost all of the training instances that wereadmitted from the left flat region in LASVM-NC wereremoved from the kernel expansion, leading to a muchlarger reduction of the number of support vectors overall.

Intuitively, the examples that are misclassified by a widemargin should not become support vectors. Ideally, thesupport vectors should be the instances that are within themargin of the hyperplane. As studies on Active Learningshow [14], [20], the most informative instances to determinethe hyperplane lie within the margin. Thus, LASVM-I ignoresthe instances that are misclassified by a margin (z < s) upfront and prevents them from becoming support vectors.

LASVM-I

1) Initialization:

Set � 0


Pick an example xiCompute z ¼ yi

Pnj¼0 �jKðxi; xjÞ

if (z > 1 or z < s)

Skip xi and bail out

else

Compute Gap Target GG



Run REPROCESS

end


Note that LASVM-I cannot be regarded as a nonconvexSVM solver since the instances with � ¼ C (which corre-sponds to z < s) are already being filtered out up frontbefore the optimization steps. Consequently, all of theinstances visible to the optimization steps have � ¼ 0,which converts the objective function in (17) into the convexHinge Loss from an optimization standpoint. Thus, combin-ing these two filtering criteria (z > 1 and z < s), LASVM-I

trades nonconvexity with a filtering Ramp function todetermine whether to ignore an instance or proceed withoptimization steps. Our goal with designing LASVM-I isthat, based on this initial filtering step, it is possible toachieve further speedups in training times while maintain-ing competitive generalization performance. The experi-mental results validate this claim.

7 LASVM-G WITHOUT CLEAN—FULL SVM

This algorithm serves as a baseline case for comparisons inour experimental evaluation. The learning principle of FULL

SVM is based on alternating between LASVM-G’s PROCESS

and REPROCESS steps throughout the training iterations.

FULL SVM

1) Initialization:

Set �� 0


TABLE 1Analysis of Adult Data Set at the End of the Training Stage

2) Online Iterations:Pick an example xiCompute Gap Target GG



Run REPROCESS

end

When a new example is encountered, FULL SVM computesthe Gap Target (given in (8)) and takes a PROCESS step. Then,it makes optimizations of the REPROCESS kind until the sizeof the duality gap comes down to the Gap Threshold. In thislearning scheme, FULL SVM admits every new trainingexample into the kernel expansion without any removal step(i.e., no CLEAN operation). This behavior mimics thebehavior of traditional SVM solvers by providing that thelearner has constant access to all training instances that it hasseen during training and can make any of them a supportvector any time if necessary. The SMO-like optimization inthe online iterations of FULL SVM enables it to converge to thebatch SVM solution.

Each PROCESS operation introduces a new instance to thelearner, updates its � coefficient, and optimizes the objectivefunction. This is followed by potentially multiple REPROCESS

steps which exploit �-violating pairs in the kernel expansion.Within each pair, REPROCESS selects the instance withmaximal gradient, and potentially can zero the � coefficientof the selected instance. After sufficient iterations, as soon as a�-approximate solution is reached, the algorithm stopsupdating the � coefficients. For full convergence to the batchSVM solution, running FULL SVM usually consists ofperforming a number of epochs where each epoch performsn online iterations by sequentially visiting the randomlyshuffled training examples. Empirical evidence suggests thata single epoch yields a classifier almost as good as the SVMsolution. For the theoretical explanation of the convergenceproperties of the online iterations, refer to [13].

The freedom to maintain and access the whole pool ofseen examples during training in FULL SVM does come with aprice though. The kernel expansion needs to constantly growas new training instances are introduced to the learner, and itneeds to hold all non-SVs in addition to the SVs of the currentmodel. Furthermore, the learner still needs to include thosenon-SVs in the optimization steps and this additionalprocessing becomes a significant drag on the training timeof the learner.

8 EXPERIMENTS

The experimental evaluation involves evaluating theseoutlined SVM algorithms on various data sets in terms ofboth their classification performances and algorithmicefficiencies leading to scalability. We also compare thesealgorithms against the reported metrics of LASVM andLIBSVM on the same data sets. In the experiments presentedbelow, we run a single epoch over the training examples, allexperiments use RBF kernels and the results averaged over10 runs for each data set. Table 2 presents the characteristicsof the data sets and the SVM parameters that weredetermined via 10-fold cross validation. Adult is a hard to

classify census data set to predict if the income of the personis greater than 50K based on several census parameters, suchas age, education, marital status, etc. Mnist, USPS, andUSPS-N are optical character recognition data sets. USPS-Ncontains artificial noise which we generated by changing thelabels of 10 percent of the training examples in the USPSdata set. The Reuters-21578 is a popular text miningbenchmark data set of 21,578 news stories that appearedon the Reuters newswire in 1987, and we test the algorithmswith the Money-fx category of the Reuters-21578 data set.The Banana data set is a synthetic two-dimensional data setthat has 4,000 patterns consisting of two banana-shapedclusters that have around 10 percent noise.

8.1 Generalization Performances

One of the metrics that we used in the evaluation of thegeneralization performances is Precision-Recall BreakevenPoint (PRBEP) (see, e.g., [21]). Given the definition ofprecision as the number of correct positive class predictionsamong all positive class predictions and recall as thenumber of correct positive class predictions among allpositive class instances, PRBEP is a widely used metric thatmeasures the accuracy of the positive class where precisionequals recall. In particular, PRBEP measures the trade-offbetween high precision and high recall. Fig. 4 shows thegrowth of PRBEP curves sampled over the course of trainingfor the data sets. Compared to the baseline case FULL SVM,all algorithms are able to maintain competitive general-ization performances at the end of training on all examplesand show a more homogeneous growth compared toLASVM, especially for the Adult and Banana data sets.Furthermore, as shown in Table 3, LASVM-NC and LASVM-I

actually yield higher classification accuracy for USPS-Ncompared to FULL SVM. This can be attributed to theirability to filter bad observations (i.e., noise) from trainingdata. In noisy data sets, most of the noisy instances aremisclassified and become support vectors in FULL SVM,LASVM-G, and LASVM due to Hinge Loss. This increase inthe number of support vectors (see Fig. 6) causes SVM tolearn complex classification boundaries that can overfit tonoise, which can adversely affect their generalizationperformances. LASVM-NC and LASVM-I are less sensitiveto noise, and they learn simpler models that are able to yieldbetter generalization performances under noisy conditions.

For the evaluation of classification performances, wereport three other metrics, namely, prediction accuracy (inTable 3), and AUC, and g-means (in Table 4). Prediction


TABLE 2Data Sets Used in the Experimental Evaluations

and the SVM Parameters C and for the RBF Kernel

accuracy measures a model’s ability to correctly predict theclass labels of unseen observations. The area under the ROCcurve (AUC) is a numerical measure of a model’s discrimina-tion performance and shows how correctly the modelseparates the positive and negative observations and ranksthem. The Receiver operating characteristic (ROC) curve isthe plot of sensitivity versus 1� specificity and the AUCrepresents the area below the ROC curve. g-means is thegeometric mean of sensitivity and specificity where sensitivityand specificity represent the accuracy on positive andnegative instances, respectively. We report that all algo-rithms presented in this paper yield as good results for theseperformance metrics as FULL SVM and comparable classifica-tion accuracy to LASVM and LIBSVM. Furthermore, LASVM-

NC yields the highest g-means for the Adult, Reuters, andUSPS-N data sets compared to the rest of the algorithms.

We study the impact of the s parameter on the general-ization performances of LASVM-NC and LASVM-I, andpresent our findings in Fig. 5. Since FULL SVM, LASVM-G,LASVM, and LIBSVM do not use Ramp Loss, they arerepresented with their testing errors and total number ofsupport vectors achieved in the end of training. These plotsdepict that LASVM-NC and LASVM-I algorithms achievecompetitive generalization performance with much fewersupport vectors, especially for Adult, Banana, and USPS-Ndata sets. In all data sets, increasing the value of s into thepositive territory actually has the effect of preventing

correctly classified instances that are within the margin from

becoming SVs. This becomes detrimental to the general-

ization performance of LASVM-NC and LASVM-I since those

instances are among the most informative instances to the

learner. Likewise, moving s further down to the negative

territory diminishes the effect of the Ramp Loss on the

outliers. If s! �1, then Rs ! H1; in other words, if s takes

large negative values, the Ramp Loss will not help to remove

outliers from the SVM kernel expansion.


Fig. 4. PRBEP versus number of training instances. We used s ¼ �1 forthe ramp loss for LASVM-NC.

TABLE 3Comparison of All Four SVM Algorithms

with LASVM and LIBSVM for All Data Sets

TABLE 4Experimental Results That Assess

the Generalization Performance and Computational Efficiency

It is important to note that at the point s ¼ �1, the Ramp-loss-based algorithms LASVM-NC and LASVM-I behave asan Active Learning [20], [22] framework. Active Learning iswidely known as a querying technique for selecting themost informative instances from a pool of unlabeledinstances to acquire their labels. Even in cases where thelabels for all training instances are available beforehand,active learning can still be leveraged to select the mostinformative instances from training sets. In SVMs, theinformativeness of an instance is synonymous with itsdistance to the hyperplane and the instances closer to thehyperplane are the most informative. For this reason,traditional min-margin active learners focus on the instancesthat are within the margin of the hyperplane and pick anexample from this region to process next by searching theentire training set. However, such an exhaustive search isimpossible in the online setup and computationally expen-sive in the offline setup. Ertekin et al. [14] suggest thatquerying for the most informative example does not need tobe done from the entire training set, but instead, queryingfrom randomly picked small pools can work equally well ina more efficient way. Small pool active learning first samplesM random training examples from the entire training setand selects the best one among those M examples based onthe condition that the selected instance among the top percent closest instances in the entire training set withprobability (1� ), where 0 � � 1. With probability1� M , the value of the criterion for this example exceedsthe -quantile of the criterion for all training examples

regardless of the size of the training set. In practice, thismeans that the best example among 59 random trainingexamples has a 95 percent chance of belonging to the best5 percent of examples in the training set.

In the extreme case of small pool active learning, settingthe size of the pool to 1 corresponds to investigatingwhether that instance is within the margin or not. In thisregard, setting s ¼ �1 for the Ramp Loss in LASVM-NC andLASVM-I constrains the learner’s focus only on the instanceswithin the margin. Empirical evidence suggests thatLASVM-NC and LASVM-I algorithms exhibit the benefits ofactive learning at s ¼ �1 point, which yields the best resultsin most of our experiments. However, the exact setting forthe s hyperparameter should be determined by the require-ments of the classification task and the characteristics of thedata set.

8.2 Computational Efficiency

A significant time-consuming operation of SVMs is thecomputation of kernel products Kði; jÞ ¼ �ðxiÞ � �ðxjÞ. Foreach new example, its kernel product with every instance inthe kernel expansion needs to be computed. By reducingthe number of kernel computations, it is possible to achievesignificant computational efficiency improvements overtraditional SVM solvers. In Fig. 7, we report the numberof kernel calculations performed over the course of trainingiterations. FULL SVM suffers from uncontrolled growth ofthe kernel expansion, which results in a steep increase in thenumber of kernel products. This also shows why SVMscannot handle large-scale data sets efficiently. In compar-ison, LASVM-G requires fewer kernel products than FULL

SVM since LASVM-G keeps the number of instances in thekernel expansion under control by periodically removinguninformative instances through CLEAN operations.

LASVM-NC and LASVM-I yield significant reduction in thenumber of kernel computations and their benefit is mostpronounced in the noisy data sets Adult, Banana, andUSPS-N. LASVM-I achieves better reduction of kernelcomputations than LASVM-NC. This is due to the aggressivefiltering done in LASVM-I where no kernel computation isperformed for the instances on the flat regions of the RampLoss. On the other hand, LASVM-NC admits the instancesthat lie on the left flat region into the kernel expansion butachieves sparsity through the nonconvex optimization steps.The � values of those admitted instances are set to C inLASVM-NC so that they have new � bounds. Resulting fromthe new � bounds, these instances are more likely to bepicked in the REPROCESS steps due to violating KKTconditions, and consequently removed from the set ofsupport vectors. The reason for the low number of kernelproducts in LASVM-NC is due to its ability to create sparsermodels than other three algorithms. A comparison of thegrowth of the number of support vectors during the courseof training is shown in Fig. 6. LASVM-NC and LASVM-I endup with smaller number of support vectors than FULL SVM,LASVM-G, and LIBSVM. Furthermore, compared to LASVM-I,LASVM-NC builds noticeably sparser models with lesssupport vectors in noisy Adult, Banana, and USPS-N datasets. LASVM-I, on the other hand, makes fewer kernelcalculations in training stage than LASVM-NC for those datasets. This is a key distinction of these two algorithms: Thecomputational efficiency of LASVM-NC is the result of itsability to build sparse models. Conversely, LASVM-I creates


Fig. 5. Testing error versus number of support vectors for varioussettings of the s parameter of the ramp loss.

comparably more support vectors than LASVM-NC, butmakes fewer kernel calculations due to early filtering. Theoverall training times for all data sets and all algorithms arepresented both in Fig. 8 and Table 3. LASVM-G, LASVM-NC,and LASVM-I are all significantly more efficient than FULL

SVM. LASVM-NC and LASVM-I also yield faster training thanLIBSVM and LASVM. Note that the LIBSVM algorithm hereuses a second order active set selection. Although secondorder selection can also be applied to LASVM-like algorithmsto achieve improved speed and accuracy [23], we did notimplement it in the algorithms discussed in this paper.Nevertheless, in Fig. 8, the fastest training times belong toLASVM-I where LASVM-NC comes close second. The sparsestsolutions are achieved by LASVM-NC and this time LASVM-I

comes close second. These two algorithms represent acompromise between training time versus sparsity andrecognition time, and the appropriate algorithm should bechosen based on the requirements of the classification task.

9 CONCLUSION

In traditional convex SVM optimization, the number ofsupport vectors scales linearly with the number of trainingexamples, which unreasonably increases the training timeand computational resource requirements. This fact hashindered widespread adoption of SVMs for classificationtasks in large-scale data sets. In this work, we have studiedthe ways in which the computational efficiency of an onlineSVM solver can be improved without sacrificing thegeneralization performance. This paper is concerned with

suppressing the influences of the outliers, which particu-larly becomes problematic in noisy data classification. Forthis purpose, we first present a systematic optimizationapproach for an online learning framework to generatemore reliable and trustworthy learning models in inter-mediate iterations (LASVM-G). We then propose two onlinealgorithms, LASVM-NC and LASVM-I, which leverage theRamp function to avoid the outliers to become supportvectors. LASVM-NC replaces the traditional Hinge Loss withthe Ramp Loss and brings the benefits of nonconvexoptimization using CCCP to an online learning setting.LASVM-I uses the Ramp function as a filtering mechanism todiscard the outliers during online iterations. In onlinelearning settings, we can discard new coming trainingexamples accurately enough only when the intermediatemodels are reliable as much as possible. In LASVM-G, theincreased stability of intermediate models is achieved bythe duality gap policy. This increased stability in the modelsignificantly reduces the number of wrongly discardedinstances in online iterations of LASVM-NC and LASVM-I.Empirical evidence suggests that the algorithms provideefficient and scalable learning with noisy data sets in tworespects: 1) computational: there is a significant decrease inthe number of computations and running time duringtraining and recognition, and 2) statistical: there is asignificant decrease in the number of examples requiredfor good generalization. Our findings also reveal thatdiscarding the outliers by leveraging the Ramp function isclosely related to the working principles of margin-basedActive Learning.


Fig. 6. Number of SVs versus number of training instances.Fig. 7. Number of kernel computations versus number of traininginstances.

ACKNOWLEDGMENTS

This work was done while Seyda Ertekin was with the

Department of Computer Science and Engineering at the

Pennsylvania State University and NEC Laboratories,

America.

REFERENCES

[1] O. Bousquet and A. Elisseeff, “Stability and Generalization,”J. Machine Learning, vol. 2, pp. 499-526, 2002.

[2] B. Scholkopf and A.J. Smola, Learning with Kernels: Support VectorMachines, Regularization, Optimization, and Beyond. MIT Press, 2002.

[3] J. Shawe-Taylor and N. Cristianini, Kernel Methods for PatternAnalysis. Cambridge Univ. Press, 2004.

[4] C. Cortes and V. Vapnik, “Support Vector Networks,” MachineLearning, vol. 20, pp. 273-297, 1995.

[5] L. Mason, P.L. Bartlett, and J. Baxter, “Improved Generalizationthrough Explicit Optimization of Margins,” Machine Learning,vol. 38, pp. 243-255, 2000.

[6] N. Krause and Y. Singer, “Leveraging the Margin More Care-fully,” Proc. Int’l Conf. Machine Learning, p. 63, 2004.

[7] F. Perez-Cruz, A. Navia-Vazquez, and A.R. Figueiras-Vidal,“Empirical Risk Minimization for Support Vector Classifiers,”IEEE Trans. Neural Networks, vol. 14, no. 2, pp. 296-303, Mar. 2002.

[8] D.S.L. Xu and K. Cramer, “Robust Support Vector MachineTraining via Convex Outlier Ablation,” Proc. 21st Nat’l Conf.Artificial Intelligence, 2006.

[9] Y. Liu, X. Shen, and H. Doss, “Multicategory Learning andSupport Vector Machine: Computational Tools,” J. Computationaland Graphical Statistics, vol. 14, pp. 219-236, 2005.

[10] L. Wang, H. Jia, and J. Li, “Training Robust Support VectorMachine with Smooth Ramp Loss in the Primal Space,”Neurocomputing, vol. 71, pp. 3020-3025, 2008.

[11] A.L. Yuille and A. Rangarajan, “The Concave-Convex Procedure(CCCP),” Advances in Neural Information Processing Systems. MITPress, 2002.

[12] R. Collobert, F. Sinz, J. Weston, and L. Bottou, “Trading Convexityfor Scalability,” Proc. Int’l Conf. Machine Learning, pp. 201-208,2006.

[13] A. Bordes, S. Ertekin, J. Weston, and L. Bottou, “Fast KernelClassifiers with Online and Active Learning,” J. Machine LearningResearch, vol. 6, pp. 1579-1619, 2005.

[14] S. Ertekin, J. Huang, L. Bottou, and L. Giles, “Learning on theBorder: Active Learning in Imbalanced Data Classification,” Proc.ACM Conf. Information and Knowledge Management, pp. 127-136,2007.

[15] J.C. Platt, “Fast Training of Support Vector Machines UsingSequential Minimal Optimization,” Advances in Kernel Methods:Support Vector Learning, pp. 185-208, MIT Press, 1999.

[16] S. Shalev-Shwartz and N. Srebro, “SVM Optimization: InverseDependence on Training Set Size,” Proc. Int’l Conf. MachineLearning, pp. 928-935, 2008.

[17] S.S. Keerthi, S.K. Shevade, C. Bhattacharyya, and K.R.K. Murthy,“Improvements to Platt’s smo Algorithm for svm ClassifierDesign,” Neural Computation, vol. 13, no. 3, pp. 637-649, 2001.

[18] O. Chapelle, “Training a Support Vector Machine in the Primal,”Neural Computation, vol. 19, no. 5, pp. 1155-1178, 2007.

[19] I. Steinwart, “Sparseness of Support Vector Machines,” J. MachineLearning Research, vol. 4, pp. 1071-1105, 2003.

[20] G. Schohn and D. Cohn, “Less Is More: Active Learning withSupport Vector Machines,” Proc. Int’l Conf. Machine Learning,pp. 839-846, 2000.

[21] T. Joachims, “Text Categorization with Support Vector Machines:Learning with Many Relevant Features,” Technical Report 23,Univ. Dortmund, 1997.

[22] S. Tong and D. Koller, “Support Vector Machine Active Learningwith Applications to Text Classification,” J. Machine LearningResearch, vol. 2, pp. 45-66, 2001.

[23] T. Glasmachers and C. Igel, “Second-Order SMO Improves SVMOnline and Active Learning,” Neural Computation, vol. 20, no. 2,pp. 374-382, 2008.

Seyda Ertekin received the BSc degree inelectrical and electronics engineering fromOrta Dogu Teknik Universitesi (ODTU) inAnkara, Turkey, the MSc degree in computerscience from the University of Louisiana atLafayette, and the PhD degree in computerscience and engineering from PennsylvaniaState University–University Park in 2009. Sheis currently a postdoctoral research associateat the Massachusetts Institute of Technology

(MIT). Her research interests focus on the design, analysis, andimplementation of machine learning algorithms for large-scale datasets to solve real-world problems in the fields of data mining,information retrieval, and knowledge discovery. She is mainly knownfor her research that spans online and active learning for efficient andscalable machine learning algorithms. At Penn State, she was amember of the technical team of CiteSeerX. Throughout her PhDstudies, she also worked as a researcher in the Machine LearningGroup at NEC Research Laboratories in Princeton, New Jersey. Priorto that, she had worked at Aselsan, Inc., in Ankara, and worked ondigital wireless telecommunication systems. She has also worked asa consultant to several companies in the US on the design of datamining infrastructures and algorithms. She is the recipient ofnumerous awards from the ACM, US National Science foundation(NSF), and Google.


Fig. 8. Training times of the algorithms for all data sets after one passover the training instances.

Leon Bottou received the diplome de l’EcolePolytechnique, Paris, in 1987, the Magistere enmathematiques fondamentales et appliquees etinformatiques from the Ecole Normale Super-ieure, Paris, in 1988, and the PhD degree incomputer science from the Universite de Paris-Sud in 1991. He joined AT&T Bell Labs from 1991to 1992 and AT&T Labs from 1995 to 2002.Between 1992 and 1995, he was the chairman ofNeuristique in Paris, a small company pioneering

machine learning for data mining applications. He joined NEC LabsAmerica in Princeton, New Jersey in 2002. His primary research interestis machine learning. His contributions to this field address theory,algorithms, and large-scale applications. His secondary research interestis data compression and coding. His best known contribution in this fieldis the DjVu document compression technology (http://www.djvu.org). Heis serving on the boards of the Journal of Machine Learning Research andthe IEEE Transactions on Pattern Analysis and Machine Intelligence. Healso serves on the scientific advisory board of Kxen, Inc. (http://www.kxen.com). He won the New York Academy of Sciences BlavatnikAward for Young Scientists in 2007.

C. Lee Giles is the David Reese professor ofinformation sciences and technology at thePennsylvania State University, University Park.He has appointments in the Departments ofComputer Science and Engineering and SupplyChain and Information Systems. Previously, hewas at NEC Research Institute, Princeton, NewJersey, and the Air Force Office of ScientificResearch, Washington, District of Columbia. Hisresearch interests are in intelligent cyberinfras-

tructure, Web tools, search engines and information retrieval, digitallibraries, Web services, knowledge and information extraction, datamining, name matching and disambiguation, and social networks. Hewas a cocreator of the popular search engines and tools: SeerSuite,CiteSeer (now CiteSeerX) for computer science, and ChemXSeer, forchemistry. He also was a cocreater of an early metasearch engine,Inquirus, the first search engine for robots.txt, BotSeer, and the first foracademic business, SmealSearch. He is a fellow of the ACM, IEEE, andINNS.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


ieee transactions on pattern analysis and machine...

Documents