sentence classifier for helpdesk emails anthony 6 june 2006 supervisors: dr. yuval marom dr. david...
Post on 21-Dec-2015
215 Views
Preview:
TRANSCRIPT
Sentence Classifier Sentence Classifier for Helpdesk Emailsfor Helpdesk Emails
Anthony
6 June 2006
Supervisors: Dr. Yuval Marom Dr. David Albrecht
Outline of Topics
IntroductionDomainApproachFeature SelectionContext-based Sentence ClassificationExperiments and ResultsQuestion & Answer
Introduction
Increased usage of emails: communication, organizing workflow, managing tasksA need to process these emails to better organize themExisting tasks: Email Summarization Email Classification Spam filtering
Use of words-only
Sentence Types
Can they be useful for existing tasks?Depend on the domainExamples Invitation Instruction Suggestion Complaint
Domain
Domain: Email helpdeskSpecifically, email responses (more
structured) Availability of data Analyze the efficiency of email
responses
Sample Email<response-ack>This is with reference to your email
regarding `Service Pack 1’</response-ack>
<instruction>Download and install the latest service packs from the link provided </instruction>
<specification>Compaq insight Manager 7<specification>
<url> http://www.hp.com/ </url>
<request>Please email us in case of any queries</request>
Thesis Aim
Develop a sentence classifierFocus: Investigate several feature selection
methods Investigate the use of context in
sentence classification
Motivation
Applications Identify informative sentences to
summarize biography Classify sentences in online product
reviews Use important sentences to help
classify documents Use sentences in the email to
determine sender’s intention
Approach
Determine the sentence typesCreate training setClassification Methods: Naïve Bayes Decision Trees SVM
Class Proportion
Class %
Statement 28.5
Thanking 15.3
Request 9.8
Salutation 8.7
Instruction 8.5
Instruction-item 6.3
URL 5.4
Class %
Response-ack
4.2
Suggestion 3.7
Specification 2.8
Signature 2.2
Apology 1.5
Questions 1.5
Others 1.5
Feature Set
Sentences need to be transformed into an appropriate representation for most classification algorithmsCommon features: Bag-of-Words (“version”, “latest”,
“software”, “install”) Bigram, Trigram
(“thank you”, “we are sorry”)
Feature Selection Purpose
High feature space: (tens of) thousands of featuresA need to reduce the feature space for Computational efficiency Remove redundant features (possibly) Improve classification
accuracy
Feature Selection Methods
Feature selection methods: Stop-words removal (“of”, “the”, “a”) Lemmatization Sentence Frequency Information Gain Chi Square
Context-based Sentence Classification
Classify a sequence of sentences extracted from an emailContext of a sentence refers to its surrounding sentences
...Set serial speed at least 38.4K.Issue AT^H carriage return.Begin your ASCII file upload.…
Context-based Sentence Classification
Assume given the class of previous sentenceFind the upper bound of improvement
...
<instruction>Set the speed at least 38.4</instruction>
<instruction>Issue AT^H carriage return</instruction>
Begin your ASCII file upload
…
Evaluation Metrics (1)Evaluation for each categoryEvaluation for average of all categoriesCommon metrics Precision Recall F1-measure
Evaluation Metrics (2)
P
PA
A
PAPA
Precision =
Recall =
F1-measure =recallprecision
recallprecision
2
Experiment and Results
Classifier
F1-measure
Without Feature
Selection
With Feature
Selection
Naïve Bayes 0.666 0.814
Decision Trees
0.829 0.844
SVM 0.883 0.888
Effect of Feature Selection
Chi-Square Effect on Different Classifiers
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Number of features
F-m
easu
re NB
DT
SVM
Effect of Context
ClassifierF1-measure
Without Context With Context
Naïve Bayes 0.814 0.844
Decision Trees
0.844 0.846
SVM 0.888 0.864
Analysis on Context
Classifier Corrections
Misclassifications
Difference
Naïve Bayes 45 11 34
Decision Trees
8 6 2
SVM 25 11 14
...<instruction>Set the speed at least 38.4</instruction><instruction>Issue AT^H carriage return.</instruction><instruction>Begin your ASCII file upload.</instruction>…
Conclusion
Feature selection methods have positive effectSVM > Decision Trees > Naïve BayesContext shows minor improvement Need more data
Future Work
Parse Trees Consider the structure of the
sentence
Viterbi Algorithm Find the best sequence of classes to
map to the sequence of sentences
Forward-backward Algorithm Include next sentences as the context
to predict current sentence
top related