novel representations and methods in text classification

Novel representations and methods in text classification

Manuel Montes, Hugo Jair EscalanteInstituto Nacional de Astrofísica, Óptica y Electrónica, Mexico.

http://ccc.inaoep.mx/~mmontesg/

http://ccc.inaoep.mx/~hugojair/{mmontesg, hugojair}@inaoep.mx

7th Russian Summer School in Information RetrievalKazan, Russia, September 2013

2

OVERVIEW OF THIS COURSENovel represensations and methods in text classification

3

Text classification

• Text classification is the assignment of free-text documents to one or more predefined categories based on their content

• In this course we will review the basics of text classification (document preprocessing and representation and classifier construction) with emphasis on novel or extended document representations

4

Day 1 (today): Introduction to text classification

• We will provide a general introduction to the task of automatic text classification:– Text classification: manual vs

automatic approach– Machine learning approach to

text classification– BOW: Standard representation for

documents– Main learning algorithms for text

classification – Evaluation of text classification

5

Day 2: Concept-based representations

• This session elaborates on concept-based representations of documents. That is document representations extending the standard representation with –latent– semantic information. • Distributional representations • Random indexing• Concise semantic analysis• Multimodal representations• Applications in short-text classification, author profiling

and authorship attribution

6

Day 3: Modeling sequential and syntactic information

• This session elaborates on different alternatives that extend the BOW approach by including sequential and syntactic information. – N-grams– Maximal frequent sequences– Pattern sequences– Locally weighted bag of words– Syntactic representations– Applications in authorship attribution

7

Day 4: Non-conventional classification methods

• This session considers text classification techniques specially suited to work with low quality training sets.

– Self-training– PU-learning – Consensus classification– Applications in short-text classification, cross-lingual

text classification and opinion spam detection.

8

Day 5: Automatic construction of classification models

• This session presents methods for the automated construction of classification models in the context of text classification. – Introduction to full model selection – Related works– PSMS– Applications in authorship attribution

9

Some relevant material

ccc.inaoep.mx/~mmontesg/

• Notes of this courseteaching Russir 2013

• Notes of our course on text miningteaching text mining

• Notes of our course on information retrievalteaching information retrieval

10

INTRODUCTION TO TEXT CLASSIFICATION

Novel represensations and methods in text classification

11

Text classification

• Text classification is the assignment of free-text documents to one or more predefined categories based on their content

Documents (e.g., news articles)Categories/classes

(e.g., sports, religion, economy)

12

Manual classification

• Very accurate when job is done by experts– Different to classify news in general categories than

biomedical papers into subcategories. • But difficult and expensive to scale– Different to classify thousands than millions

• Used by Yahoo!, Looksmart, about.com, ODP, Medline, etc.

13

Hand-coded rule based systems

• Main approach in the 80s• Disadvantage knowledge acquisition bottleneck– too time consuming, too difficult, inconsistency issues

Experts

Labeleddocuments

Knowledgeengineers

Rule 1, if … then … elseRule N, if … then …

Classifier

New document

Document’s category

14

Example: filtering spam email• Rule-based classifier

Hastie et al. The Elements of Statistical Learning, 2007, Springer.

15

Machine learning approach (1)

• A general inductive process builds a classifier by learning from a set of preclassified examples.– Determines the characteristics associated with each one

of the topics.

Ronen Feldman and James Sanger, The Text Mining Handbook

16

Machine learning approach (2)

Labeleddocuments(training set)

Rules, trees, probabilities,

prototypes, etc.

Classifier

Newdocument

Document’scategory

Learningalgorithm

Experts

How to represent document’s information?

17

Representation of documents

• First step: transform documents, which typically are strings of characters, into a representation suitable for the learning algorithm.– Generation of a document-attribute representation

• The most common used document representation is the bag of words.– Documents are represent by the set of different words

in all of the documents– Word order is not capture by this representation– There is no attempt for understanding their content

18

Representation of documents

t1 t1 … tn

d1

d2

: wi,j

dm

All documents(one vector per document)

Weight indicating the contributionof word j in document i.

Vocabulary from the collection(set of different words)

Each different word is a feature!How to compute their weights?

19

Preprocessing• Eliminate information about style, such as html or

xml tags.– For some applications this information may be useful. For

instance, only index some document sections.• Remove stop words– Functional words such as articles, prepositions,

conjunctions are not useful (do not have an own meaning).• Perform stemming or lemmatization– The goal is to reduce inflectional forms, and sometimes

derivationally related forms.

am, are, is → be car, cars, car‘s → car Have to do them always?

20

Term weighting - two main ideas

• The importance of a term increases proportionally to the number of times it appears in the document.– It helps to describe document’s content.

• The general importance of a term decreases proportionally to its occurrences in the entire collection.– Common terms are not good to discriminate between

different classes

21

Term weighting – main approaches• Binary weights: – wi,j = 1 iff document di contains term tj , otherwise 0.

• Term frequency (tf):– wi,j = (no. of occurrences of tj in di)

• tf x idf weighting scheme:

– wi,j = tf(tj, di) × idf(tj), where:

• tf(tj, di) indicates the ocurrences of tj in document di

• idf(tj) = log [N/df(tj)], where df(tj) is the number of

documets that contain the term tj.

M

kki

jiji

w

ww

1

2,

,,

)(

Normalization?

22

Extended document representations• BOW is simple and tend to produce good results, but it has

important limitations– Does not capture word order neither semantic information

• New representations attempt to handle these limitations. Some examples are:– Distributional term representations – Locally weighted bag of words – Bag of concepts – Concise semantic analysis – Latent semantic indexing– Topic modeling– …

The topic of this course

23

Dimensionality reduction

• A central problem in text classification is the high dimensionality of the features space.– Exist one dimension for each unique word found in

the collection can reach hundreds of thousands– Processing is extremely costly in computational terms– Most of the words (features) are irrelevant to the

categorization task

How to select/extract relevant features?How to evaluate the relevance of the features?

24

Two main approaches• Feature selection• Idea: removal of non-informative words according to

corpus statistics• Output: subset of original features– Main techniques: document frequency, mutual information

and information gain• Re-parameterization– Idea: combine lower level features (words) into higher-level

orthogonal dimensions– Output: a new set of features (not words)– Main techniques: word clustering and Latent semantic

indexing (LSI)Out of the scope of this course

Learning the classification model

C1

C2

C3

C4

Categories

X1

X2

25

How to learn this?

26

What is a classifier?• A function:

• or:

• Given:

Cf d :

},...,{ 1 KCCC

}1,0{),(: Cf d

Terms (d = |V|)

Docu

men

ts

(M)

m1

m3

di x

Cyi

27

What is a classifier?• A function:

• or:

• Given:

Nii yD ,...,1)},{(x

Cyid

i ;x

Terms (d = |V|)

Docu

men

ts

(M)

m1

m3

di x Cyi

Cf d :

},...,{ 1 KCCC

}1,0{),(: Cf d

28

Classification algorithms• Popular classification algorithms for TC are:– K-Nearest Neighbors• Example-based approach

– Centroid-based classification• Prototype-based approach

– Support Vector Machines• Kernel-based approach

– Naïve Bayes • Probabilistic approach

29

Positive examplesNegative examples

KNN: K-nearest neighbors classifier

1-nearest neighbor classifier

30



3-nearest neighbors classifier

31



5-nearest neighbors classifier

32

KNN – the algorithm• Given a new document d:

1. Find the k most similar documents from the training set.• Common similarity measures are the cosine

similarity and the Dice coefficient.

2. Assign the class to d by considering the classes of its k nearest neighbors• Majority voting scheme• Weighted-sum voting scheme

33

Common similarity measures

• Dice coefficient

• Cosine measure

wki indicates the weight of word k in document i

m

k kjm

k ki

m

k kjkiji

ww

wwdds

12

121

2,

m

k kjm

k ki

m

k kjkiji

ww

wwdds

12

12

1,

2 | |,| | | |A Bs A BA B

, cos( )|| || || ||A Bs A BA B

34

Selection of K

How to select a good value for K?

35

Decision surface of KNN

http://clopinet.com/CLOP

K=1


36



K=2


37



K=5


38



K=10


39

The weighted-sum voting scheme

Other alternatives for computing the weights?

40

KNN - comments• One of the best-performing text classifiers.

• It is robust in the sense of not requiring the categories to be linearly separated.

• The major drawback is the computational effort during classification.

• Other limitation is that its performance is primarily determined by the choice of k as well as the distance metric applied.

41

Linear models

• Classification of DNA micro-arrays

?

x1

x2

No Cancer

Cancer

1 2,x x x

?1 bxw

1 bxw

0 bxw

bf xwx)(

42

Support vector machines (SVM)

• A binary SVM classifier can be seen as a hyperplane in the feature space separating the points that represent the positive from negative instances.– SVMs selects the hyperplane

that maximizes the marginaround it.

– Hyperplanes are fullydetermined by a small subsetof the training instances, calledthe support vectors.

Support vectors

Maximizemargin

43

Support vector machines (SVM)

• When data are linearly separable we have:

1min2

Tw w

( ( ) ) 1Ti iy b w x

Subject to:

{1,..., }i m

1|| ||w 1

|| ||w

44

Non-linear SVMs (on the inputs)

• What about classes whose training instances are not linearly separable?– The original input space can always be mapped to

some higher-dimensional feature space where the training set is separable.• A kernel function is some function that corresponds

to an inner product in some expanded feature space.

0 x

x2

45

Decision surface of SVMs

Linear support vector machine



46

Decision surface of SVMs

Non-linear support vector machine



47

SVM – discussion • The support vector machine (SVM) algorithm is very fast and

effective for text classification problems.

– Flexibility in choosing a similarity function• By means of a kernel function

– Sparseness of solution when dealing with large data sets• Only support vectors are used to specify the separating

hyperplane

– Ability to handle large feature spaces• Complexity does not depend on the dimensionality of the

feature space

48

Naïve Bayes• It is the simplest probabilistic classifier used to

classify documents– Based on the application of the Bayes theorem

• Builds a generative model that approximates how data is produced– Uses prior probability of each category given no

information about an item– Categorization produces a posterior probability

distribution over the possible categories given a description of an item.

Sec.13.2

A. M. Kibriya, E. Frank, B. Pfahringer, G. Holmes. Multinomial Naive Bayes for Text Categorization Revisited. Australian Conference on Artificial Intelligence 2004: 488-499

50

Naïve Bayes

• For a document d and a class cj( | ) ( )

( | )( )j j

j

P C P CP C

P

dd

d

Sec.13.2

1 | |

1 | |

( ,..., | ) ( )( ,..., )V j j

V

P t t C P CP t t

| |1 ( | ) ( )

( )

Vi i j jP t C P C

P

d

• Assuming terms are independent of each other given the class (naïve assumption)

| |1 ( | ) ( )V

i i j jP t C P C • Assuming each document is equally probable

t1

C

t2 t|V|. . . .

51

Bayes’ Rule for text classification

• For a document d and a class cj

Sec.13.2

| |

1

V

j j i ji

P C P C P t C

d

52

Bayes’ Rule for text classification

• For a document d and a class cj

• Estimation of probabilities

Sec.13.2

| |

1

V

j j i ji

P C P C P t C

d

| |

cj

j

NP C

D | |

1

1

| |

iji j V

kjk

NP t c

V N

Prior probability of class cj Probability of occurrence of word ti in class cj

Smoothing to avoid zero-values

53

Naïve Bayes classifier

• Assignment of the class:

• Assignment using underflow prevention:– Multiplying lots of probabilities can result in floating-

point underflow– Since log(xy) = log(x) + log(y), it is better to perform all

computations by summing logs of probabilities rather than multiplying probabilities

| |

C C 1

arg max arg maxj j

V

j j i jC C i

class P C P C P t C

d

| |

C 1

argmax log log |j

V

j i jC i

class P C P t C

54

Comments on NB classifier

• Very simple classifier which works very well on numerical and textual data

• Very easy to implement and computationally cheap when compared to other classification algorithms.

• One of its major limitations is that it performs very poorly when features are highly correlated.

• Concerning text classification, it fails to consider the frequency of word occurrences in the feature vector.

55

Evaluation of text classification

• What to evaluate?• How to carry out this evaluation?– Which elements (information) are required?

• How to know which is the best classifer for a given task?– Which things are important to perform a fair

comparison?

56

Evaluation – general ideas

• Performance of classifiers is evaluated experimentally

• Requires a document set labeled with categories.– Divided into two parts: training and test sets– Usually, the test set is the smaller of the two

• A method to smooth out the variations in the corpus is the n-fold cross-validation. – The whole document collection is divided into n equal parts,

and then the training-and-testing process is run n times, each time using a different part of the collection as the test set. Then the results for n folds are averaged.

57

Evaluation of text classification

• The available data is divided into two subsets:– Training (m1)• used for the

construction (learning) the classifier

– Test (m2)• Used for the

evaluation of the classifier

Terms (N = |V|)

Docu

men

ts

(M)

m1

m3

58

Performance metrics• Considering a binary problem

• Recall for a category is defined as the percentage of correctly classified documents among all documents belonging to that category, and precision is the percentage of correctly classified documents among all documents that were assigned to the category by the classifier.

What happen if there are more than two classes?

a bdc

Classifier YES

Classifier NO

Label YES Label NO

caa

(R) recall

dcbada

accuracy

baa

(P)precision RP

PRF

2

59

Micro and macro averages• Macroaveraging: Compute performance for each

category, then average.– Gives equal weights to all categories

• Microaveraging: Compute totals of a, b, c and d for all categories, and then compute performance measures.– Gives equal weights to all documents

Is it important the selection of the averaging strategy?What happen if we are very bad classifying the minority class?

60

References

• G. Forman. An Extensive Empirical Study of Feature Selection Metrics for Text Classification. JMLR, 3:1289—1305, 2003

• H. Liu, H. Motoda. Computational Methods of Feature Selection. Chapman & Hall, CRC, 2008.

• Y. Yang, J. O. Pedersen. A Comparative Study on Feature Selection in Text Categorization. Proc. of the 14th International Conference on Machine Learning, pp. 412—420, 1997.

• D. Mladenic, M. Grobelnik. Feature Selection for Unbalanced Class Distribution and Naïve Bayes. Proc. of the 16th Conference on Machine Learning, pp. 258—267, 1999.

• I. Guyon, et al. Feature Extraction Foundations and Applications, Springer, 2006.

• Guyon, A. Elisseeff. An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, Vol. 3:1157—1182, 2003.

• M. Lan, C. Tan, H. Low, S. Sung. A comprehensive comparative study on term weighting schemes for text categorization with support vector machines. Proc. of WWW, pp. 1032—1033, 2005.

61

Questions?