- university of albertawebdocs.cs.ualberta.ca/~vis/asingh1/docs/online...3department of information...

www.ietdl.org

IE

d

Published in IET BiometricsReceived on 9th February 2012Revised on 11th June 2013Accepted on 14th June 2013doi: 10.1049/iet-bmt.2012.0048

T Biom., pp. 1–15oi: 10.1049/iet-bmt.2012.0048

ISSN 2047-4938

Online signature verification using segment-levelfuzzy modellingAbdul Quaiyum Ansari1, Madasu Hanmandlu2, Jaspreet Kour1, Abhineet Kumar Singh3

1Department of Electrical Engineering, JMI, New Delhi, India2Department of Electrical Engineering, IIT, New Delhi, India3Department of Information Technology, IIIT, Allahabad, UP, India

E-mail: [email protected]

Abstract: This study presents a new online signature verification system based on fuzzy modelling of shape and dynamic featuresextracted from online signature data. Instead of extracting these features from a signature, it is segmented at the points ofgeometric extrema followed by the feature extraction and fuzzy modelling of each segment thus obtained. A minimumdistance alignment between the two samples is made using dynamic time warping technique that provides a segment tosegment correspondence. Fuzzy modelling of the extracted features is carried out in the next step. A user-dependent thresholdis used to classify a test sample as either genuine or forged. The accuracy of the proposed system is evaluated using bothskilled and random forgeries. For this, several experiments are carried out on two publicly available benchmark databases,SVC2004 and SUSIG. The experimental results obtained on these databases demonstrate the effectiveness of this system.

1 Introduction

Biometrics is an emerging field of technology for theenforcement of security. Several biometric modalities havebeen proposed in the last decades [1]. These can be dividedinto two main classes, depending on whether they are basedon physical called physiological or behavioural traits of anindividual. Physical traits are related to anatomicalcharacteristics of a person and include fingerprint, face, irisand hand geometry among others. Behavioural traits refer tohow an individual performs an action, and include voice,signature and gait among the most popular ones.Signatures have been used for centuries to validate documents

and transactions. Therefore, signature is one of the most sociallyaccepted biometric traits, thus making it the most natural andestablished way of confirming an identity. It is non-invasive innature and has no undesirable health connotations [2]. In thepast few decades, digitising devices have made machine-basedsignature verification possible. Despite its wide acceptance,automatic signature verification is still a challenging task. Oneof the main challenges in signature verification is posed by thesignature variability. While signatures from the same usertaken at different times show considerable differences (highintra-class variability), skilled forgers can imitate signatureswith high resemblance (low inter-class variability). Anexperimental study of human perception of handwrittensignatures covering genuine and the forged samples isreported in [3]. The human strategies for signature checkinginfluence the automatic signature verification. Moreover,factors like the number of reference samples available,signature sample variability and complexity of signaturepatterns also affect the verification process.

Signature verification can be split up into two modes –online and offline – depending on the type of availabledata. Online or dynamic signatures are captured by specialhardware (e.g. smart pens or pressure sensitive tablets),which is capable of measuring dynamic properties of asignature in addition to its shape. Offline signatures, on theother hand, are drawn on paper with ordinary pens and thushave shape as the only available information. Onlinesignatures are typically considered more reliable than offlineones since dynamic properties like pen pressure and writingspeed make the signature more unique and difficult toforge. Each online signature is represented by a discretetime sequence of data points with each point containing thex, y coordinates, time stamp and button status. Additionalinformation like pen pressure, altitude angle and azimuthangle may also be present.Applications of online signature verification include

identity verification during electronic payments (e.g. using acredit card), authorisation of computer users for accessingthe sensitive data or programs, authentication of individualsfor accessing physical devices or buildings and protectionof small personal devices (e.g. PDA, laptop) fromunauthorised usage [4].The remainder of this paper is organised as follows:

Section 2 presents a brief review of the existing literaturealong with showcasing the novelty of this work over anexisting method [5]. Section 3 details the proposedmethodology involving preprocessing, segmentation,segment alignment, feature extraction and fuzzy modelling.Section 4 presents the details of databases andmethodologies used in the performance evaluation alongwith the obtained results and their comparison with several

1© The Institution of Engineering and Technology 2013

www.ietdl.org
state-of-the-art methods. Section 5 gives the conclusions andthe future scope.
2 Background

2.1 Related work

An extensive literature exists in the field of online signatureverification. A review of some of the more recent approacheshas been carried out in [6]. Signature segmentation isconsidered mostly as the first step in the verification process.Segmentation is defined as the detection of perceptuallyimportant points in [7] at which to divide the signature intosegments. The work presented in [8] measures the perceptualimportance of each point by the change of writing anglebetween the selected point and its neighbour. A modifiedversion of this method appears in [9], where the end pointsof pen-down strokes are considered as the significant splittingpoints. The method proposed in [10] uses the points ofgeometric extrema, both horizontal and vertical, as thesegmenting points and carries out segment-to-segmentmatching through a set of rules based on the properties of theextremum specific to each such point. Three differentsegmentation methods, using equal partitioning, strokes andlocal extrema, are implemented in [11].Since different signatures may have different lengths, a

method is needed to equalise them before they can becompared on a point-to-point or segment-to-segment basis.Dynamic time warping (DTW) is a technique that employscompression or expansion of the time axis of two timesequences, representing two signatures with possiblydifferent number of points, to obtain a correspondence thatminimises some measure of distance between them. DTWhas been employed widely in the literatures [10, 12–16] toaid the signature matching process.Most of the existing methods extract a set of features either

from the signature or from the individual segments beforeverification. A comparative study of features commonlyused in online signature verification is presented in [17].Owing to a large number of the available features, we needa method to select a subset of features with the maximumdiscriminative ability. Genetic algorithm (GA) has beenused for this purpose in [5, 18–20]. The approaches such asvelocity image model [21], fast Fourier transform [22],Rough sets [23] and Mellin transform [24] necessitatecomplex feature extraction procedures.Fuzzy logic is a powerful tool for solving complex problems

because of its ability to handle uncertainties in the inputs and italso incorporates the heuristics devised by a human expert toarrive at the most likely solution. Fuzzy inference is used forboth offline and online signature verification. Methods in theformer category include those based on modular neuralnetwork [25], snake algorithm [26], box method [27, 28] andconfidence fuzzy intervals [29]. Most of these methodsemploy the relatively simpler Takagi-Sugeno (TS) model.There are several examples in the latter category, such as[30] that employs a neural network classifier with fuzzyinference decision module; [20] that uses fuzzy network; [31]that employs neural network-driven fuzzy reasoning; and [5]that uses a rule-based Mamdani system for fuzzy modellingof optimal features selected using GA.

2.2 Novelty of the proposed method

The present work can be regarded as an extension of the work in[5], where fuzzy modelling is carried out on features extracted


from a whole signature. In this work, however, a signature isfirst divided into segments which are then aligned with thesegments of a reference signature before applying an adaptedversion of the same fuzzy model at the level of individualsegments. This extra step of segmentation can significantlyimprove the detection accuracy for reasons stated in Section3.2. In addition, a novel adaptive segmentation improves thequality of segments and produces more accuratesegment-to-segment correspondence.Another significant difference arises with regard to the use

of a threshold. A single global threshold is used in [5] toclassify a test signature as genuine or forged whereas weuse a different threshold for each user. A user-dependentthreshold captures the inherent variability in a particularuser’s signatures and thus helps to reduce both falseacceptances and false rejections. For example, a user who isnot very precise about the way he writes his signatures willexhibit greater variability among his genuine signatures ascompared to a user who is always extremely specific aboutthe way his signatures are made. This difference in writinghabit necessitates the use of a higher threshold for thesecond user than the first one to obtain optimal results.Thus, using a single threshold for both these users may leadto an increased rate of false rejections for the first user andfalse acceptances for the second one. Another advantage ofuser-dependent threshold is that the use of global thresholdrequires the entire system to be trained again to obtain anupdated threshold whenever a new user is added to thedatabase whereas with user-specific threshold, however,only the new user’s signature samples are needed in thetraining process The time required for this process istherefore independent of the number of users alreadypresent in the database and does not increase as thedatabase grows larger, which is not the case with the globalthreshold. Also, the user-specific threshold is found to bemore appropriate than the global threshold in behaviouralbiometrics because of comparatively larger variations ingenuine samples of a person than in physiologicalbiometrics where global thresholds might make more sense.

3 Proposed system

A block diagram of the proposed system is shown in Fig. 1. Allthe tasks mentioned therein are performed independently foreach user. The preprocessing of each signature sample isfollowed by segmentation along the points of geometricextrema. Next, all the training samples of the same user arepair-wise segment-aligned with each other. One of thetraining samples is then chosen as the prototype genuinesample for that user and all the test samples of that user aresegment-aligned with this sample. This is followed by thefeature extraction where both static (shape) and dynamicfeatures are extracted for each segment for all the samples(training and testing). The extracted features are subjected tofuzzy modelling using a combination of TS and Mamdaniapproaches to obtain a single match score for each segment.The scores of all the segments of each sample are thencombined to obtain an overall score for that sample. Thisscore is compared with a user-dependent threshold at thedecision stage to classify the sample as either genuine or forged.

3.1 Preprocessing

The signing process is considered as a ballistic movement thatcauses variation among the genuine signatures of the same

IET Biom., pp. 1–15doi: 10.1049/iet-bmt.2012.0048

Fig. 1 Block diagram of the proposed system

www.ietdl.org

person. As the variations in different signatures have differentdynamic ranges, min–max normalisation is applied on their xand y coordinates. The normalisation process used here shifts


the minimum and maximum scores to 0 and 1000,respectively; but it does not change the underlyingdistribution of the data except for a scaling factor.

xn =x−min(x)

max(x)−min(x)× 1000 (1)

yn =y−min(y)

max(y)−min(y)× 1000 (2)

where x and y are the original coordinates and xn and yn are thenormalised coordinates. The remaining components of eachdata point, viz., pressure and angle information (bothaltitude and azimuth) are not changed since they are fairlyresistant to noise and also contain crucial clues to the way aperson holds the pen and writes with it. Thesecharacteristics are typically specific to a person and arepractically impossible to forge without detailed knowledgeof the person’s writing manner. Thus retaining their originalvalues is necessary for deriving maximum discriminativeability.

3.2 Segmentation

When features are extracted from the whole signature sample,they are subjected to an averaging effect [10] due to whichfiner information present in localised regions of the sampleis lost, thus reducing the discriminative capability of thefeatures. This is particularly true for online signatures –since dynamic information present in specific parts of asignature is crucial in identifying the writer and it isdifficult to be forged. For example, many people have aspecific pattern in the speed with which they typicallygenerate their signatures, for instance, slow in the beginningand at the end of the signature but with the maximum speedin the middle. This is why the samples are segmentedbefore the feature extraction step.A good segmentation method should meet two main

requirements [10]: the segments generated should beconsistent across all the genuine samples and thecorrespondence between matching segments of any twosamples should be easy to find. Based on theserequirements, the points of geometric extrema are chosen tosegment the signatures since they constitute the cornerpoints of the frame of the pattern and are thus reproducedreliably in different samples of the same user. They alsohave specific properties that help in determiningsegment-to-segment correspondence [10]. Points of verticaland horizontal extrema are detected by finding the zerocrossings of the derivatives of x and y sequences,respectively. The ith point in a signature sequence is takenas a point of vertical extremum, if the following conditionholds true

yi+1 − yi( )

yi − yi−1

( ) ≤ 0 (3)

Here, yi denotes the y coordinate value in the ith data pointwhere i = 2, 3, …, k− 1 for a sample with k data points. Asimilar expression is used for the points of horizontalextrema too. In order to deal with the presence of noise inthe time sequence, which may lead to several invalidsegments, a certain threshold (δ) is taken as the minimumnumber of points in an acceptable segment. If a segment ishaving number of points <δ, it is merged with the nextsegment if possible or with the previous one if it happens to


www.ietdl.org
be too close to the end of the signature. Experiments areconducted by considering each of the x and y derivativesseparately and the best results are obtained using the latter.Fig. 2 shows a few genuine and skilled forgery samples forthe same user from SVC2004 database.During experimentation δ is fixed at 5 and 10 and the
former value was found to give better results than the lattervalue on SVC2004 while the opposite is true for SUSIG.Further investigation into this has revealed that signatures inSUSIG are on average 50% longer than those in SVC2004in terms of the number of points in the signature sequence(300 points against 208 points). This suggests that highervalues of δ should be used for longer signatures to obtainthe best results. This in turn has led us to the concept ofadaptive segmentation where, instead of using a fixed valueof δ for all samples, a different δ is used for each sample. Acertain fraction (min_fraction) of the total number of pointsin a sample is taken as the value of δ for that samplesubject to a global minimum (global_min). Thus, the valueof δ for a sample with n points is evaluated as follows

min points = floor(min fraction× n) (4a)

d = max(global min , min points) (4b)

Here floor(x) returns the largest integer not greater than x.This method constrains the total number of segments in asample to be less than a specific value irrespective of thelength of that sample and is quite effective at dealing withspurious extrema generated due to shaky hands orinstrument noise. In addition, it also makes the task offinding segment-to-segment correspondence easier and moreaccurate since the number of segments obtained fordifferent samples is likely to be roughly equal even if thesamples are of widely differing sizes.We have experimented with several values of min_fraction

between 0.01 and 0.2 and global_min between 5 and 15 andoptimal results were obtained using min_fraction = 0.03 forSVC2004 and min_fraction = 0.04 for SUSIG, withglobal_min = 5 for both databases. A comparison of theresults obtained for various values of min_fraction ispresented in Fig. 3.

3.3 Alignment of segments

Since any two signature samples may have different numberof segments, a non-trivial method is needed to find thecorrespondence between the segments of the two samples.Through the process of alignment, segments of a sample arematched with those of another sample to obtain thiscorrespondence. This matching may involve mapping inwhich a single segment of either sample matches withmultiple segments of the other. To make it more clear,consider two samples, sample1 and sample2, then a singlesegment of sample1 may match with multiple segments ofsample2 or a single segment of sample2 may match withmultiple segments of sample1, as in Fig. 4.For this purpose, a DTW-based method is used, as detailed

in [32]. It employs a dynamic programming-based approachto obtain a many-to-many mapping between the segmentsof two samples so as to minimise the accumulated distancebetween the matched segments, as given by some distancefunction (e.g. Euclidean distance). Let us call this minimumaccumulated distance between two samples (or segments) asthe DTW distance between them.


The mapping produced by this method is subject to thefollowing constraints:

1. There is always a mapping between the starting and theending pairs of segments of the two samples.2. The mapping is strictly non-decreasing in time from theperspective of either sample. This means that, whilematching two samples, samples1 and samples2, if segmentm of samples1 (Seg1m) has been matched to segment j ofsamples2 (Seg2j), then any other segment Seg1n, where n≥m, can match to a segment Seg2k only if k≥ j.3. Each segment of either sample must be a part of exactlyone mapping which may be one-to-one, many-to-one orone-to-many. A particular segment cannot be a part of twodifferent types of mappings at the same time. For instance,in the example of Fig. 4a, segments 3 and 4 of sample (iii)are paired with segment 4 of sample (i) and are thusinvolved in a many-to-one mapping. Now it would beinvalid for either of these two segments to be a part ofanother mapping with segments of sample (i) (e.g. segment3 of sample (iii) cannot have a one-to-one mapping withsegment 3 of sample (i) even though such a mapping wouldnot violate the other two constraints). This constraint alsoimplies that each segment of either sample must match withat least one segment of the other sample.

Following are the main steps involved in the matchingprocess:

1. First, a measure of distance needs to be defined betweentwo segments, each belonging to one of the two samplesbeing aligned, where each segment is represented by asequence of data points. We have used the mean Euclideandistance between the x and y coordinates of thecorresponding data points in the two segments as themeasure. Two cases may arise here:

i. If the two segments have equal number of data points, thenthe distance between them is given by the following equation:

distij =1

n

∑nk=1

��xik − x jk

( )2+ yik − y jk

( )2√(5)

Here, distij is the DTW distance between segi and segj,belonging, respectively, to samples1 and samples2 and eachhaving n data points. The x, y coordinates of segi and segjare (xik, yik) and (xjk, yjk), respectively, for k = 1, 2, …, n.ii. If, on the other hand, they have different number of datapoints, we find the minimum-distance-matching between thetwo sets of data points by recursively applying the sameDTW-based method of finding a correspondence betweentwo sets of segments except that here we assume that eachsegment has only one data point. Thus, the DTW distancebetween two such single-point-segments is simply theEuclidean distance between the points. The overall DTWdistance distij between segi and segj is then the meanaccumulated Euclidean distance between the matched points.

2. The distance obtained in Step 1 is used in a dynamicprogramming-based algorithm to find a path from the firstpair to the last pair of segments with the minimumaccumulated cost. This is the optimal DTW path and theaccumulated cost divided by the number of steps in thispath is the corresponding DTW distance between the twosamples. More details of this algorithm can be found in [32].


Fig. 2 Sample signatures of a user from SVC2004 database segmented along points of vertical extrema (min_fraction = 0.03 global_min = 5)

The segmentation points are marked by an asterisk and consecutive segments are shown by solid and dotted linesThe unmarked end is the starting point of the signature(i) Prototype sample; (ii) and (iii) training genuine samples; (iv) and (v) test genuine samples; (vi) and (vii) skilled forgery samples

www.ietdl.org



Fig. 4 Pairings obtained by the DTW method between the prototype sa

a Pairings of sample (i) with samples (ii) and (iii) of Fig. 2b Pairings of sample (i) with samples (iv) and (v) of Fig. 2c Pairings of sample (i) with samples (vi) and (vii) of Fig. 2

Fig. 3 Results obtained in terms of user-dependent EER fordifferent values of min_fraction using both SVC2004 and SUSIGdatabases

N = 5 and global_min = 5 were used for all runs

www.ietdl.org


3. The method outlined in the previous two steps is appliedpair wise to all the training samples of the same user, thusobtaining the distance of each training sample from each ofthe other training samples. Hence if there are n trainingsamples, there will be a total of ((n(n− 1))/2) pairings. Thetraining sample with the least mean distance from all othersis then chosen as the prototype (or reference) genuinesample for that user.4. Finally, Steps 1 and 2 are applied to find the segmentcorrespondence between the prototype sample and each ofthe test samples, both genuine and forged.

Since the DTW matching process produces amany-to-many mapping between the segments of the twosamples, a method of resolution is needed for cases whereone segment of the prototype sample gets paired withmultiple segments of the other sample. The properties ofthe multiple matching segments need to be combined sothat they can be compared to the corresponding property ofthe single prototype segment. Three alternative methods aretried to carry out this combination

mple and a few other samples


Table 1 Features used in the proposed system

Shape features Dynamic features

mean of x coordinate total signature timemean of y coordinate mean velocity in x directionheight mean velocity in y directionwidth mean acceleration in x directionlength

mean acceleration in y directionmean of pressure

mean of azimuth anglemean of altitude angle

www.ietdl.org
1. Extract the features of each matching segment as detailedin Section 3.4 and take the mean over all the segments toobtain a single value for each feature which is then used forfurther processing. This is the case of feature-levelcombination.2. Extract features and carry out the first two steps of fuzzymodelling (Section 3.5) to obtain the degree of match(DOM) for each of the segments independently and thentake the mean of these DOMs as the single DOMrepresenting these segments. This is the case of DOM levelcombination.3. As a consequence of the second constraint in the matchingprocess specified earlier, if multiple segments of one samplematch with a single segment of the other sample, they willnecessarily be consecutive segments. Thus, another way tocombine them is to consider all the data points from thefirst point of the first segment to the last point of the lastsegment as belonging to a single segment and then extractfeatures from this composite segment. This is the case ofcombination at the level of data points.
Experiments are carried out using each of these methodsand the second one is found to give the best resultsprobably because of least incidence of the averaging effectmentioned in Section 3.2. Thus, only the results of DOMlevel combination are mentioned in Section 4.To clarify the segment alignment process, the segment

pairs that were obtained between the prototype sample andthe other samples shown in Fig. 2 are given in Fig. 4.

3.4 Feature extraction

Two types of features, viz., shape and dynamic features areextracted from each segment at this stage. Shape featuresare useful for detecting random forgery but may fail forskilled forgeries because of the relative ease of copying theoverall shape of a signature. Dynamic features, on the otherhand, have much better discriminative power because theyare much harder to imitate [33].Several features are experimented and out of which only

those having valid values for any segment are used in thiswork. Although the main inspiration for this work is [5],but many of its features cannot be used here because ofeither divide-by-zero problems or their irrelevance at thelevel of individual segments. For example, features likeheight-to-width ratio and length-to-width ratio used in [5]cannot be used here because of the possibility of the widthbecoming zero for some segments, a problem that does notoccur when these features are extracted for the wholesignature. Moreover, some complex features like RMScentripetal acceleration, RMS tangential acceleration andRMS acceleration [5] could not be used here since they areexpensive to compute, leading to unacceptable performancewhen this computation has to be repeated for each segment.Thus, only a few relatively simple features are used in thiswork. These are listed in Table 1.

3.5 Fuzzy modelling

Fuzzy modelling of each feature of each segment of theprototype sample signature, which forms a fuzzy set asexplained below, is used in the verification of an unknownsignature. This decision is made on the basis of the fuzzymembership values of the features extracted from thissignature using the fuzzy sets that are learned during thetraining stage. In fact, these fuzzy sets constitute the fuzzy


rules framed. The concept of a fuzzy set arising from a setof features is explained as follows. A fuzzy set is formedfrom each feature of each segment of the prototype samplegathered over all the training samples. Suppose there are mtraining samples and the prototype sample has n segmentswith i features extracted from each of these. Then therewould be a total of n × i fuzzy sets with each one having mvalues since each training sample would contribute onevalue to each fuzzy set. The variation in the feature valuesover the training sample space gives rise to fuzziness.Whenever multiple segments of a sample match with a

single segment of the prototype sample during the DTWalignment process, their features will be combined and willeither be compared with (a test sample) or contribute to (atraining sample) the single fuzzy set corresponding to thatsegment of the prototype sample. For instance, in theexample of Fig. 4a, segments 3 and 4 of training sample(iii) would contribute only to the fuzzy set corresponding tosegment 4 of prototype sample (i). Along the same lines,when the test sample (iv) of Fig. 4b is passed through thissystem, the properties of its segments 4 and 5 would becombined and compared to the values in the fuzzy setcorresponding to segment 4 of the prototype sample (i).Conversely, it is also possible for one segment to contributeto multiple fuzzy sets, as demonstrated by segment 1 ofsample (iii) that contributes to the fuzzy sets correspondingto both segments 1 and 2 of sample (i).This work uses the rule-based Mamdani approach to fuzzy

modelling, which is adapted from [5]. This approach involvesthe following steps for each segment:

1. Calculate the normalised difference for each feature fromits reference set by

Disti =fi − mi��1+ s2

i

√ (6)

Here, fi is the value of the ith feature, μi and σi are the meanand standard deviation of this feature over the trainingsamples.2. Compute the DOM. This is a measure of the degree ofsimilarity of the test signature’s specific feature value for aparticular segment against the reference, expressed as apercentage. This is the output of single-input, single-outputand single rule TS model whose input is Disti from Step 1and output is the corresponding DOM, as shown in Fig. 5.This model uses two parameters Distmin and Distmax suchthat DOMi = 100 for Disti≤Distmin and DOMi = 0 forDisti≥Distmax. After experimenting with severalcombinations of these parameters, the values of 1 and 7 forDistmin and Distmax, respectively, are found to give optimalresults.


Fig. 5 Single-input, single-output, single-rule TS fuzzy system

www.ietdl.org

The above two steps are repeated to obtain the DOM foreach feature.

3. Compute the total degree of match (TDOM) for each of thetwo feature sets: static/shape-based features and dynamicfeatures. For each set, a weighted mean of all the features inthe set is called the TDOM for that set. The weightingfactors reflect the relative importance of features and aretaken to be inversely proportional to the respective standarddeviations. This is based on the reasoning that lesser thestandard deviation of a feature over the training set, more isits relative importance in discriminating the user’s genuinesignatures from the forged ones. Thus, a feature’s relativeimportance (RIM) may be expressed as

RIMi =1��

1+ s2i

√ (7)

The variances between RIMs of different features are verylarge, making them unsuitable for direct use as weightingfactors. A sigmoid function is therefore used to normalisethe RIM into the range from 0 to 1. This provides us ameasure of the feature’s normalised relative importance(NRIM) given by

NRIMi =1

1+ exp −RIMi

( ) (8)

The weight of the ith feature is then obtained as

Wi =NRIMi∑nj=1 NRIMj

(9)

Fig. 6 Triangular membership functions for the inputs as well as the o


These weights are used to obtain two values of TDOM, oneeach for the shape and dynamic features. These TDOMs arecomputed as

TDOM =∑ni=1

Wi × DOMi

( )(10)

The summations in (9) and (10) are done over all the nfeatures in the respective feature set.4. Find the degree of authentication (DOA) for each segmentexpressed as a percentage: The overall DOA of each segmentis obtained as the output of a two-input, single-output andmultiple-rules Mamdani fuzzy system for which theTDOMs of shape and dynamics are the two inputs. Thisgives an overall measure of authenticity for each segment ofthe test signature.

The two input linguistic variables used are ‘TDOM ofshape’ and ‘TDOM of dynamics’ whereas the outputlinguistic fuzzy variable is ‘DOA of segment’. The numberof linguistic terms partitioning the input and output spacesis set to 11 each so that there is one term whose triangularmembership function is centred at each of the multiples often from 0 to 100 (both inclusive). For the sake ofconvenience, these linguistic terms are assigned integers(called linguistic IDs or LIDs) from 1 to 11 rather thannames. It should be noted from Fig. 6 that higher numbersdenote lower degrees of match. The associated functions areidentical for each of the two inputs as well as the output, asshown in Fig. 6. The rule base used in this work is given inTable 2.To further elucidate the meaning of the membership

functions and the rule base, the rules that are fired for theexample of Fig. 7, with TDOM values of 92 and 87%,respectively, for dynamic and shape feature sets, are givenin the conventional If–Then rule form as follows:

i. If Shape TDOM has LID 1 and Dynamic TDOM has LID 2Then segment DOA has LID 2ii. If Shape TDOM has LID 1 and Dynamic TDOM has LID 3Then segment DOA has LID 3iii. If Shape TDOM has LID 2 and Dynamic TDOM has LID2 Then segment DOA has LID 3

utput


Table 2 Rule base for fuzzy inference

S. No. TDOM of dynamics

1 2 3 4 5 6 7 8 9 10 11

TDOM of shape 1 1 2 3 4 5 6 7 8 9 10 112 2 3 4 5 6 7 8 9 10 11 113 3 4 5 6 7 8 9 10 11 11 114 4 5 6 7 8 9 10 11 11 11 115 5 6 7 8 9 10 11 11 11 11 116 6 7 8 9 10 11 11 11 11 11 117 7 8 9 10 11 11 11 11 11 11 118 8 9 10 11 11 11 11 11 11 11 119 9 10 11 11 11 11 11 11 11 11 11

10 10 11 11 11 11 11 11 11 11 11 1111 11 11 11 11 11 11 11 11 11 11 11

Fig. 7 Graphical representation of Larson’s method

www.ietdl.org

iv. If Shape TDOM has LID 2 and Dynamic TDOM has LID3 Then segment DOA has LID 4

Larson’s method is used to combine the rule base and themembership functions to obtain a single crisp value foreach segment. A graphical representation of Larson’smethod is shown in Fig. 7, where singleton fuzzificationand mean of maximum (MOM) defuzzification methods areemployed. The defuzzified percentage DOA of the segmentis given by (refer to Fig. 7)

DOA(segment%)

= 0.2× 90+ 0.2× 80+ 0.7× 80+ 0.3× 70

0.2+ 0.2+ 0.7+ 0.3= 79.28%

5. Classify the test signature based on its TDOA: The totaldegree of authentication (TDOA) of a signature sample is


defined as the arithmetic mean of the DOAs of all of itssegments. This value is compared with a user-dependentthreshold to classify the sample as either genuine or forged.The expression for calculating the TDOA of a samplehaving k segments is

TDOA = 1

k

∑ki=1

DOAi (11)

4 Performance evaluation

For verifying a test signature, it must be given as the input tothe fuzzy model, learned during the training stage to obtain asingle similarity score (TDOA). This is then compared with athreshold, selected during the training stage, to classify thesignature as genuine or forged. For identifying an unknownsignature, it must be passed through the models of all users


www.ietdl.org
in the database and the user whose model gives the maximumTDOA relative to his/her genuine training samples subject tothe condition that it is above the threshold specific to the sameuser is identified with the unknown signature. If a signaturesample fails to cross the threshold for any of the users inthe database, it is classified as unknown.Automatic signature verification can produce two types of
errors: Type I error, which is concerned with the falserejections of genuine signatures called the false rejectionrate (FRR); and Type II error, which is concerned with thefalse acceptance of forged signatures called the falseacceptance rate (FAR). Typically, FAR decreases whileFRR increases as the threshold is increased. The equal errorrate (EER), which is another measure of the overallaccuracy of a system, arises when FRR is made equal to theFAR by adjusting the threshold [34]. The forgeries used inthe verification process are categorised as random andskilled. In the case of a random forgery, the forger haseither no knowledge of the original signature or does nottry to imitate it. A skilled forgery, on the other hand, isattempted by a professional imposter who traces over orimitates the signature as best as he can. Skilled forgeriesalready exist in the two databases whereas randomsignatures of other users serve as random forgeries.

4.1 Databases

Two publicly available databases used in this work aredescribed now.

4.1.1 SVC2004: This database [35] has two sets ofsignatures, namely, Task 1 and Task 2; out of which onlyTask 2 signatures have been used in this work. This setcontains signatures of 40 users with 40 samples for eachuser. Out of 40 samples, 20 are genuine and the rest areskilled forgeries. Each signature in Task 2 is represented asa sequence of points, containing x coordinate, y coordinate,time stamp and pen status (pen-up or pen-down), azimuth,altitude and pressure. Some sample signatures from thisdatabase are shown in Fig. 8.

Fig. 8 Sample signatures from SVC2004


4.1.2 SUSIG: This database [36] has two sets, viz., blindand visual sub corpus. We have used the second set. Thisset contains signature data of 100 users; but we have usedthe data of only 94 users with each having 20 genuine and10 skilled forgery signatures because the remaining sixusers have one or more signatures missing. Therepresentation of signature data is similar to that ofSVC2004 except that the data on azimuth, altitude and penstatus is not available in SUSIG. A few sample signaturesfrom this database are shown in Fig. 9.

4.2 Experiments and results

4.2.1 Performance evaluation methodology: Twosets of experiments are conducted to evaluate the systemperformance, with the first one using only skilled forgerieswhile the second one including random forgeries inaddition to skilled forgeries. Each set of experiments takesthe training samples of 5 and 10, with the remaininggenuine samples being used for testing along with theforgeries. To increase the reliability of the results, crossvalidation is done with five trials on each user. A differentset of genuine samples, selected randomly, comprises thetraining set in each of these trials. The random forgeries inthe second set of experiments are obtained by randomlyselecting 10 samples belonging to other users in thedatabase. Note that the experiments are conducted on eachdatabase separately.The overall system performance is evaluated using two

measures: accuracy and EER. Here, accuracy is defined asthe percentage of forgery samples whose TDOA is less thanthe TDOA of the genuine sample with the minimumTDOA. It is calculated independently for each user andthen averaged over all users in the database to obtain thesystem accuracy. EER is obtained at the point where theFAR equals the FRR, where FAR is the percent of forgerysamples whose TDOA≥ threshold, whereas FRR is thepercent of genuine samples whose TDOA < threshold. Twodifferent methods are used in this work to evaluate EER.User-dependent threshold: As stated in Section 1, this

work emphasises the user-dependent thresholds at theverification stage. In this approach, the values of FAR andFRR are obtained for each user for a range of thresholdsand these are used to draw plots of FAR against thresholdand FRR against threshold. The point of intersection ofthese two plots gives EER for that user. In the case of crossvalidation with multiple trials, EER is evaluatedindependently for each trial and mean is taken over all trialsfor a user to obtain EER for that user. Finally, the overallEER of the system for a particular database is computed asthe mean of EERs of all the users in the database. A few

Fig. 9 Sample signatures from SUSIG


Fig. 10 User-dependent threshold-based FAR/FRR curves for the same user from SUSIG database

a) Trial 1 b) Trial 2 c) Trial 3 d) Trial 4 e) Trial 5

www.ietdl.org

sample FAR/FRR curves for the same user are shown inFig. 10.Global threshold: Even though this system relies on

user-dependent threshold, results are obtained using both


global threshold and user-dependent threshold for the sakeof comparison. TDOAs obtained for all the users in thedatabase are combined and subjected to a single threshold,varying from 0 to 100, to obtain FAR and FRR values for


Table 3 Results of the first set of experiments (only skilledforgeries) on SVC2004 database where N = no. of trainingsamples

SVC2004 (onlyskilled forgeries)

Accuracy,%

EER, %

User-dependentthreshold

Globalthreshold

N = 5 proposedmethod

93.525 2.458 7.571

method of[5]

55.050 14.930 19.088


96.175 1.781 5.250

method of[5]

65.975 11.538 15.277

www.ietdl.org
each threshold. These values are used to obtain a plot of FARagainst FRR and its intersection with the line x = y gives theglobal EER. The curves due to the proposed method andthe method of [5] are shown in Fig. 11.As expected, EER obtained with the global approach
proved to be significantly higher than the mean EERobtained using user-dependent thresholds for both databases(Tables 3 and 4).

4.2.2 Comparison with signature-level fuzzymodelling: To evaluate the effectiveness of segment-levelfuzzy modelling over signature-level fuzzy modelling, wehave also implemented the method in [5] and conducted theabove experiments on this system too. Since no detailsabout preprocessing are mentioned in [5], experiments usingthis method are conducted on the original data. Althoughthe results of SVC2004 are as expected, with our systemoutperforming that of [5] by a large margin (Table 3), theresults of SUSIG are very similar on both methods(Table 4), with [5] having a slight advantage.

Fig. 11 Error tradeoff curves with global threshold using only skilled f

The curves corresponding to the proposed method are shown with solid lines wheThe point of EER is marked with a black dot on each curvea SVC2004 with 5 training samplesb SVC2004 with 10 training samplesc SUSIG with 5 training samplesd SUSIG with 10 training samples


It may be noted that genuine signatures in SVC2004 are notreal signatures, unlike those in SUSIG, because people whoprovided signatures have made up signatures for the sake ofcontribution leading to more variation within genuine

orgeries (Set 1)

reas those of [5] are shown with dotted lines


Table 4 Results of the first set of experiments (only skilledforgeries) on SUSIG database where N = no. of training samples

SUSIG (only skilledforgeries)

Accuracy,%

EER,%


Globalthreshold


94.660 1.303 5.386

method of[5]

95.340 1.319 4.116


96.064 1.037 4.021

method of[5]

98.638 0.684 2.234

Table 7 Results of the second set of experiments (both skilledand random forgeries) on SUSIG database where N = no. oftraining samples

SUSIG (skilled andrandom forgeries)

Accuracy,%

EER,%


Globalthreshold


95.904 1.234 4.567

method of[5]

92.585 3.731 6.304


97.447 0.911 3.394

method of[5]

95.574 2.712 4.733

Table 6 Results of the second set of experiments (both skilledand random forgeries) on SVC2004 database where N = no. oftraining samples

SVC2004 (skilledand randomforgeries)

Accuracy,%

EER,%


Globalthreshold


95.783 1.653 5.493

method of[5]

64.783 13.468 16.754


98.300 0.906 3.466

method of[5]

73.550 10.701 13.615

Table 8 Comparison with other methods using SVC2004database

Authors Year Method EER,%

Ong et al. [37] 2009 statistical quantisationmechanism (SQM),user-dependent threshold

5.32

Fierrez-Aguilaret al. [38]

2005 fusion of local (DTW) andregional (HMM) approach,user-dependent threshold

6.91

SVC 2004 [35] 2004 team 219b 6.90Mohammadiet al. [39]

2012 extended regression andDTW, user-dependent

6.33

www.ietdl.org

samples than would be possible in the real signature, thusmaking the task of separating the genuine and forgedsignatures more difficult. To verify this, we have calculatedthe mean difference between the TDOA of genuine samplesand that of skilled forgery samples using our method andthat in [5] on both databases and found this difference to besignificantly larger for SUSIG indicating a greater ease ofseparation than in SVC2004 (Table 5).To the best of our knowledge, the approach in [11] is the

only other existing method in the literature, apart from ours,that has been tested on both SVC and SUSIG databases.The results obtained there too are consistent with ourobservation since the EER obtained on SVC database (7.02)is nearly three times the EER on SUSIG (2.46). Owing tothis difference, SVC2004 database benefits greatly from theadditional fine information captured by segmentation levelmodelling, while the benefits are minimal for SUSIG wheresignatures differ enough for even signature-level modellingto be able to separate them with ease. In fact, the additionalinformation provided by segmentation even appears to havea slight negative impact on the results. Thus, we concludethat our method is more suitable for situations where theforgery has been executed with great skill.Since random forgeries are easier to detect than skilled

ones, we expected the results to improve when 10 randomforgeries were added to the testing set. The results of theproposed method indeed show a marginal improvement forboth SVC2004 and SUSIG databases. The results of [5],however, show an increasing trend in results only onSVC2004 (Table 6) but exhibits a significant decline inresults on SUSIG (Table 7). This could be due to the factthat the features extracted from a genuine signature matchwith those extracted from a completely different signature(i.e. random forgery). This unexpected behaviour isobserved more with the signature-level feature extractionmethod of [5] than with the proposed segment-level featureextraction method. Thus, the method of [5] is more likely toconfuse with a genuine signature as a random signaturethan the proposed method.

4.2.3 Comparison with other methods: In order tocompare the performance of our method with other

Table 5 Mean difference between the TDOAs of genuine andskilled forgery samples using five training samples

SUSIG SVC2004

proposed method 34.171 25.457method of [5] 45.561 15.432


contemporary methods, we have considered EER based onthe user-dependent threshold to represent our system’saccuracy. All results are the outcome of using five trainingsamples with no random forgeries.A comparison of the results is given in Table 8 and Table 9

for SVC2004 and SUSIG databases, respectively. Results ofthe proposed method have been obtained using min_fraction = 0.03 for SVC2004 and min_fraction = 0.04 forSUSIG, with global_min = 5 for both the databases.

thresholdWang et al. [11] 2011 segmentation and graph

matching, user-dependentthreshold

7.02

Fallaha et al. [24] 2011 Mellin transform and MFCC,neural network

3.00

proposed method 2013 segment-level fuzzymodelling, user-dependentthreshold

2.46


Table 9 Comparison with other methods using SUSIGdatabase

Method Year Method EER,%

Khalil et al.[40]

2009 multiple feature set and DTW,user dependent threshold

3.06

Khomatovet al. [36]

2009 Fourier descriptor and DTW 2.10

Ibrahim et al.[41]

2009 Fisher linear discriminant (FLD)analysis, user-dependentthreshold

1.57

Wang et al.[11]

2011 segmentation and graphmatching, user-dependentthreshold

2.46

proposedmethod

2013 segment-level fuzzy modelling,user-dependent threshold

1.30

www.ietdl.org

5 Conclusions

A novel online signature verification method is developedusing segment-level fuzzy modelling of features. The fuzzymodelling requires a simple rule-base and a few simplefeatures. The matching of the features of a test signaturewith the training features is done using a similarity scorecalled the TDOA which is based on the TDOM ofindividual segments. The results obtained by our methodare either comparable to or better than the existing methodseven though some of them are complex andcomputationally intensive. This is an indication of thepotential underlying the concept of the localised fuzzymodelling. The use of segment-specific features coupledwith a sophisticated rule base with some additionalcomputation may improve the results.

6 Acknowledgment

The authors would like to thank the anonymous reviewers fortheir valuable comments and suggestions, which have helpeda lot in improving this paper.

7 References

1 Jain, A.K., Ross, A., Parbhakar, S.: ‘An introduction to biometricrecognition’, IEEE Trans. Circuits Syst. Video Technol. Special Issueon Image- and Video-Based Biometrics, 2004, 14, (1), pp. 4–20

2 Fairhurst, M.C.: ‘Signature verification revisited: promoting practicalexploitation of biometric technology’, Inst. Electr. Eng. Electron.Commun. Eng. J. (ECEJ), 1997, 9, (6), pp. 273–280

3 Fairhurst, M.C., Kaplani, E.: ‘Perceptual analysis of handwrittensignatures for biometric authentication’, Inst. Electr. Eng. Proc. Vis.Image Signal Process., 2003, 150, (6), pp. 389–394

4 Khalmatov, A., Yanikoglu, B.: ‘Identity authentication using improvedon-line signature verification method’, Patt. Recogn. Lett., 2005, 26,(15), pp. 2400–2408

5 Wijesoma, W.S., Mingming, M., Yue, K.W.: ‘On-line signatureverification using a computational intelligence approach’, in: Reusch,B. (Ed.): ‘Fuzzy days’ (Springer, Berlin, Germany, 2001), vol. 2206,pp. 699–711

6 Zhang, Z., Wang, K., Wang, Y.: ‘A survey of on-line signatureverification’. CCBR 2011, 2011 (LNCS, 7098), pp. 141–149

7 Schmidt, C., Kraiss, K.F.: ‘Establishment of personalized templates forautomatic signature verification’. Proc. Fourth Int. Conf. DocumentAnalysis Recognition (ICDAR-4), IEEE Computer Society, Ulm,Germany, August 1997, vol. 1, pp. 263–267

8 Brault, J.J., Plamondon, R.: ‘Segmenting handwritten signatures at theirperceptually important points’, IEEE Trans. Patt. Anal. Mach. Intell.,1993, 15, (9), pp. 953–957

9 Shafiei, M.M., Rabiee, H.R.: ‘A new on-line signature verificationalgorithm using variable length segmentation and Hidden Markovmodels’. Seventh Int. Conf. Document Analysis and Recognition


(ICDAR-7), IEEE Computer Society, Edinburgh, UK, August 2003,vol. 1, pp. 443–446

10 Lee, J., Yoon, H.S., Soh, J., Chun, B.T., Chung, Y.K.: ‘Using geometricextrema for segment-to-segment characteristics comparison in onlinesignature verification’, Patt. Recogn., 2004, 37, (1), pp. 93–103

11 Wang, K., Wang, Y., Zhang, Z.: ‘On-line signature verification usingsegment-to-segment graph matching’. Int. Conf. Document Analysisand Recognition (ICDAR), 2011, School of Computer Science &Engineering, Beihang University, Beijing, China, 18–21 September2011, pp. 804–808

12 Yue, K.W., Wijesoma, W.S.: ‘Improved segmentation and segmentassociation for on-line signature verification’. IEEE Int. Conf. SystemMan Cybernetics, 2000, vol. 4, pp. 2752–2756

13 Zhang, J., Kamata, S.: ‘Online signature verification usingsegment-to-segment matching’. Int. Conf. Frontiers in HandwritingRecognition ICFHR (2008), Montréal, Québec, 19–21 August 2008

14 Lee, W.S., Mohankrishnan, N., Paulik, M.J.: ‘Improved segmentationthrough dynamic time warping for signature verification using a neuralnetwork classifier’. Proc. IEEE Int. Conf. Image Processing (ICIP),Chicago, IL, October 1998, vol. 2, pp. 929–933

15 Rhee, T.H., Cho, S.J., Kim, J.H.: ‘On-line signature verificationusing model-guided segmentation and discriminative featureselection for skilled forgeries’. Proc. Sixth Int. Conf. DocumentAnalysis and Recognition (ICDAR-6), Seattle, WA, September 2001,pp. 645–649

16 Martens, R., Claesen, L.: ‘On-line signature verification by dynamictime-warping’. Proc. 13th Int. Conf. Pattern Recognition (ICPR96),Vienna, Austria, 1996, pp. 38–42

17 Lei, H., Govindaraju, V.: ‘A comparative study on the consistency offeatures in on-line signature verification’, Patt. Recogn. Lett., 2005,26, pp. 2483–2489

18 Aguilar, J.F., Nanni, L., Penalba, J.L., Garcia, J.O., Maltoni, D.: ‘Anonline signature verification system based on fusion of local andglobal information’. IAPR Int. Conf. Audio – and Video-BasedBiometric Person Authentication, AVBPA, 2005a (LNCS, 3546),pp. 523–532

19 Kour, J., Hammandlu, M., Ansari, A.Q.: ‘Online signature verificationusing GA-SVM’. Proc. Int. Conf. Image Information Processing,ICIIP, 3–5 November 2011, pp. 1–4

20 Xuhua, Y., Furuhashi, T., Obata, K., Uchikawa, Y.: ‘Selection offeatures for signature verification using the genetic algorithm’,J. Comput. Ind. Eng. Archive, 1996, 30, (4), pp. 1037–1045

21 Khan, M.A.U., Khan, M.K., Khan, M.A.: ‘Velocity-image model foronline signature verification’, IEEE Trans. Image Process., 2006, 15,(11), pp. 3540–3549

22 Yanikoglu, B., Khalmatov, A.: ‘Online signature verification usingFourier descriptors’, Hindawi Publ. Corp., EURASIP J. Adv. SignalProcess., 2009, article id 260516. DOI: 10.1155/2009/260516

23 Al-Mayyan, W., Own, H.S., Zedan, H.: ‘Rough set approach to onlinesignature identification’, Digit. Signal Process., 2011, 21, (3),pp. 477–485

24 Fallaha, A., Jamaatib, M., Soleamanic, A.: ‘A new online signatureverification system based on combining Mellin transform, MFCC andneural network’, Digit. Signal Process., 2011, 21, pp. 404–416

25 Mirzaei, O., Irani, H., Pourreza, H.R.: ‘Offline signaturerecognition using modular neural networks with fuzzy responseintegration’. Int. Conf. Network and Electronics Engineering, 2011,(IPCSIT), vol. 11

26 Velez, J., Sanchez, A., Moreno, A.B., Esteban, J.L.: ‘A hybrid approachusing snakes and fuzzy modelling for offline signature verification’.Proc. Eleventh Int. Conf. Information Processing and Management ofUncertainty in Knowledge-Based Systems (IPMU 2006), 2006,Editions EDK, vol. I, pp. 882–889

27 Madasu, V.K., Lovell Brian, C., Kubik, K.: ‘Automatic handwrittensignature verification system for Australian passports’. Science,Engineering and Technology Summit on Counter-TerrorismTechnology, Canberra, 2005, pp. 53–66

28 Hanmandlu, M., Mohammed, M.H., Madasu, V.K.: ‘Off-line signatureverification and forgery detection using fuzzy modeling’, Patt.Recogn., 2005, 38, (3), pp. 341–356

29 Zakaria, R., Waheb, A.F., Ali, J.M.: ‘Confidence fuzzy interval inverification of offline handwriting signature’, Eur. J. Sci. Res. ISSN1450-216X, 2010, 47, (3), pp. 455–463

30 Khalid, M., Yusof, R., Mokayed, H.: ‘Fusion of multi classifiers foronline signature verification using fuzzy logic inference’, Int. J. Innov.Comput. Inf. Control, 2011, 7, (5B), pp. 2709–2726

31 Martínez-R, J., Alcántara-S, R.: ‘On-line signature verification based onoptimal feature representation and neural-network-driven fuzzyreasoning’. Proc. Fifth Int. Conf. Advances in Infrastructure for


www.ietdl.org
e-Business, e-Education, e-Science, e-Medicine on the Internet, L’Auila,Italy, 2003
32 Muller, M.: ‘Information retrieval for music and motion’ (Springer,2007), Ch. 4 (available at http://www.springer.com/cda/content/document/cda_downloaddocument/9783540740476-c1.pdf?SGWID=0-0-45-452103-p173751818)

33 Yang, L., Widjaja, B., Prasad, R.: ‘Application of hidden Markovmodels for signature verification’, Patt. Recogn., 1995, 28, (2),pp. 161–170

34 Impedovo, D., Pirlo, G.: ‘Automatic signature verification: the state ofthe art’, IEEE Trans. Syst. Man Cybern., 2008, 38, pp. 609–635

35 www.cse.ust.hk/svc200436 Khomatov, A., Yanikoglu, B.: ‘SUSIG: an online signature database,

associated protocols and benchmark results’, Patt. Anal. Appl., 2009,12, (3), pp. 227–236


37 Ong, T.S., Khoh, W.H., Teoh, A.: ‘Dynamic handwritten signatureverification based on statistical quantization mechanism’. Int. Conf.Computer Engineering and Technology, 2009, vol. 2, pp. 312–316

38 Fierrez-Aguilar, J., Krawczyk, S., Ortega-Garcia, J., Jain, A.K.: ‘Fusionof local and regional approaches for online signature verification’. Proc.IWBRS, 2005 (LNCS, 3617), pp. 188–196

39 Mohammadi, M.H., Faez, K.: ‘Matching between important points usingdynamic time warping for online signature verification’, Cyber J.:Multidiscip. J. Sci. Technol., J. Sel. Areas Bioinf., 2012 Jan, pp. 1–7

40 Khalil, M.I., Moustafa, M., Abbas, H.M.: ‘Enhanced DTW based onlinesignature verification’. 16th IEEE Int. Conf. Image Processing (ICIP),2009, pp. 2713–2716

41 Ibrahim, M.T., Kyan, M., Guan, L.: ‘Online signature verification usingGlobal features’. Canadian Conf. Electrical and Computer Engineering,2009, pp. 682–685


- university of albertawebdocs.cs.ualberta.ca/~vis/asingh1/docs/online...3department of information...

Documents