text mining using lda with context
Post on 07-Feb-2017
541 Views
Preview:
TRANSCRIPT
Institute for Web Science and Technologies · University of Koblenz-Landau, Germany
Text Mining Using LDA with Context
Christoph Kling, Steffen Staab
Web and Internet Science Group · ECS · University of Southampton, UK &
Text Mining Using LDA with Context 2/68Steffen Staab
Text Mining Documents
Documents are PDFs, emails, tweets,
Flickr photo tags, CVs, ...
Documents consist of bag of words metadata
- author(s) - timestamp- geolocation- publisher- booktitle- device...
Chinese food
Vegan
food
Break-
fast
dimsumduckeggs
...
vegantofu...
eggsham...
Objective:Cluster, categorize,
& explain
Text Mining Using LDA with Context 3/68Steffen Staab
Latent Dirichlet Allocation (LDA)
Text Mining Using LDA with Context 4/68Steffen Staab
Latent Dirichlet Allocation (LDA)
Document-topic distributions
Topic-word distributions
K topicsM documentsEach doc m M has length Nm
Text Mining Using LDA with Context 5/68Steffen Staab
Use Metadata to Help Topic Prediction
Improve topic detection→ Morning times may help to improve the breakfast topic Describe dependencies: metadata ↔ topics
→ breakfast topic happens during morning hours Chinese
food
Vegan
food
Break-
fast
dimsumduckeggs
...
vegantofu...
eggsham...
Text Mining Using LDA with Context 6/68Steffen Staab
Use Metadata to Help Topic Prediction
Improve topic detection→ Morning times may help to improve the breakfast topic Describe dependencies: metadata ↔ topics
→ breakfast topic happens during morning hours
Usage Autocompletion
→ From words to words Prediction of search queries
→ From metadata to words→ From words to metadata
Chinese food
Vegan
food
Break-
fast
dimsumduckeggs
...
vegantofu...
eggsham...
Text Mining Using LDA with Context 7/68Steffen Staab
Nominal
Ordinal
Cyclic
Spherical
Networked
Structures of Metadata Spaces Nejdl Staab Kling
Text Mining Using LDA with Context 8/68Steffen Staab
Challenges for Using Metadata for Text Mining
Generalizing the Text Mining ModelCreating a special text mining model for every dataset with its
kind of metadata spaces is impractical→ we need flexible models!
Text Mining Using LDA with Context 9/68Steffen Staab
Challenges for Using Metadata for Text Mining
Generalizing the Text Mining Model Efficiency of the Text Mining ModelRich metadata → complex models → complex inference, slow convergence of samplers→ analysis of big datasets impossible
Text Mining Using LDA with Context 10/68Steffen Staab
Challenges for Using Metadata for Text Mining
Generalizing the Text Mining Model Efficiency of the Text Mining Model Explaining the ResultImportance of Metadata→ learn how to weight metadata→ exclude irrelevant metadata (improves efficiency!)Complex dependencies & complex probability functions→ Learned parameters incomprehensible→ Reduced usefulness for data analysis / visualisation→ No sanity checks on parameters
Text Mining Using LDA with Context 11/68Steffen Staab
Topic Models for Arbitrary Metadata
Text Mining Using LDA with Context 12/68Steffen Staab
Topic Models for Arbitrary Metadata
Predict document-topic distributions using metadata→ Gaussian Process Regression Topic Model
(Agovic & Banerjee, 2012)→ Dirichlet-Multinomial Regression Topic Model
(Mimno & McCallum, 2012)→ Structural Topic Model (logistic normal regression)
(Roberts et al., 2013)
Text Mining Using LDA with Context 13/68Steffen Staab
Topic Models for Arbitrary Metadata
Predict document-topic distributions using metadata→ Gaussian Process Regression Topic Model→ Dirichlet-Multinomial Regression Topic Model→ Structural Topic Model (logistic normal regression)
Regression input: MetadataRegression output: Topic distribution
Text Mining Using LDA with Context 14/68Steffen Staab
Topic Models for Arbitrary Metadata
Dirichlet-multinomial regression
Metadata
Document-topic distributions
Text Mining Using LDA with Context 15/68Steffen Staab
Topic Models for Arbitrary Metadata
Gaussian process regression
Metadata
Document-topic distributions
Text Mining Using LDA with Context 16/68Steffen Staab
Topic Models for Arbitrary Metadata
Logistic normal regression
Metadata
Document-topic distributions
Text Mining Using LDA with Context 17/68Steffen Staab
Topic Models for Arbitrary Metadata
Alternating inference: Estimate topics Estimate regression model Use prediction for re-estimating topics Re-estimate regression model with new topics ...
Text Mining Using LDA with Context 18/68Steffen Staab
Topic Models for Arbitrary Metadata
Alternating inference: Estimate topics Estimate regression model Use prediction for re-estimating topics Re-estimate regression model with new topics ...
slow convergence
Text Mining Using LDA with Context 19/68Steffen Staab
Topic Models for Arbitrary Metadata
Applicable to a wide range of metadata! Estimation of regression parameters relatively expensive Learned parameters have no natural interpretation Alternating process of paramter estimation is expensive
Text Mining Using LDA with Context 20/68Steffen Staab
Topic Models for Arbitrary Metadata
Dirichlet-multinomial and logistic-normal regression do not support complex input data
(i.e. geographical data, temporal cycles, …)
Gaussian process regression topic models are very powerful with the right kernel function
...but require expert knowledge for kernel selection and efficient inference!
Text Mining Using LDA with Context 21/68Steffen Staab
Hierarchical Multi-Dirichlet Process
Topic Models
The Idea
Text Mining Using LDA with Context 22/68Steffen Staab
Topic Prediction
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Documents, e.g. emails
Text Mining Using LDA with Context 23/68Steffen Staab
Dirichlet-Multinomial Regression
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Text Mining Using LDA with Context 24/68Steffen Staab
Gaussian Process Regression
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Topi
c P
roba
bilit
y
Text Mining Using LDA with Context 25/68Steffen Staab
Cluster-Based Prediction
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Text Mining Using LDA with Context 26/68Steffen Staab
Cluster-Based Prediction
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Text Mining Using LDA with Context 27/68Steffen Staab
Cluster-Based Prediction
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Topi
c P
roba
bilit
yTo
pic
Pro
babi
lity
Topi
c P
roba
bilit
y
Text Mining Using LDA with Context 28/68Steffen Staab
Cluster-Based Prediction
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Topi
c P
roba
bilit
yTo
pic
Pro
babi
lity
Topi
c P
roba
bilit
y
Text Mining Using LDA with Context 29/68Steffen Staab
Idea
Two-step model:1)Cluster similar documents2)Learn topics for clusters and documents simultaneously
▪ Learn topic distributions of document clusters▪ Use cluster-topic distributions for topic prediction
Text Mining Using LDA with Context 30/68Steffen Staab
Performance, Complex Metadata
Cluster documents for each metadata
Text Mining Using LDA with Context 31/68Steffen Staab
Performance, Complex Metadata
Cluster documents for each metadata
Text Mining Using LDA with Context 32/68Steffen Staab
Performance, Complex Metadata
Cluster documents for each metadata
+ nominal, ordinal, cyclic, spherical data+ any data which can be clustered!
Text Mining Using LDA with Context 33/68Steffen Staab
Performance, Complex Metadata
Metadata clusters are associated with topicsGerman Beer
Party
Text Mining Using LDA with Context 34/68Steffen Staab
Mixture of Metadata Predictions
Metadata clusters are associated with topicsGerman Beer
Party
The topic prediction for a single document is a mixture of the prediction of its metadata clusters
Text Mining Using LDA with Context 35/68Steffen Staab
Smoothing of HMDP
Text Mining Using LDA with Context 36/68Steffen Staab
Cluster-Based Prediction vs Outliers and noisy data
Topi
c P
roba
bilit
y
Metadata (e.g. time)
Text Mining Using LDA with Context 37/68Steffen Staab
Adjacency Smoothing
Naive approach: Smoothed value of a cluster is the mean of the cluster and its adjacent clusters
Repeat n times
Text Mining Using LDA with Context 38/68Steffen Staab
Smoothing topics associated with metadata clusters
Documents receive topics from their own and neighboring metadata clusters
Text Mining Using LDA with Context 39/68Steffen Staab
Performance, Complex Metadata
Smooth topics associated with metadata clusters
Text Mining Using LDA with Context 40/68Steffen Staab
Nominal Ordinal Cyclic Spherical Networked
Text Mining Using LDA with Context 41/68Steffen Staab
Smoothing
Smoothing-strength is learned during inferenceSimilar clusters → stronger smoothingDissimilar clusters → softer smoothing
Smoothing-strength alternatively can be predefined by user
Text Mining Using LDA with Context 42/68Steffen Staab
Metadata Weighting in HMDP's
Text Mining Using LDA with Context 43/68Steffen Staab
Feature Weighting
One variable governs the influence of metadata cluster on documents
If η < threshold, ignore variable.
η
Text Mining Using LDA with Context 44/68Steffen Staab
Metadata Weighting
Importance of metadata is learned during inference, answering the question:
How many percent of the topics are explained by a given metadata? (e.g. time, geographical coordinates, ...)
→ Interpretable parameter! Metadata with a low weight can be removed during
inference
Text Mining Using LDA with Context 45/68Steffen Staab
Example Application
Text Mining Using LDA with Context 46/68Steffen Staab
Dataset
Linux Kernel Mailinglist3,400,000 emails with timestamps and mailinglist ID
Text Mining Using LDA with Context 47/68Steffen Staab
Dataset
Linux Kernel Mailinglist3,400,000 emails with timestamps and mailinglist ID
Timeline Yearly cycle Weekly cycle Daily cycle Mailing list
Text Mining Using LDA with Context 48/68Steffen Staab
Topics
Text Mining Using LDA with Context 49/68Steffen Staab
Topics
Text Mining Using LDA with Context 50/68Steffen Staab
Topics
Professional topics:
Hobbyist topics:
Text Mining Using LDA with Context 51/68Steffen Staab
Topics
Metadata weighting:
Text Mining Using LDA with Context 52/68Steffen Staab
Topics
Metadata weighting:
can be removed during inference
Text Mining Using LDA with Context 53/68Steffen Staab
Efficient Inference in HMDP
Text Mining Using LDA with Context 54/68Steffen Staab
Hierarchical Multi-Dirichlet Process Topic Model (HMDP)
Cluster-topic distributions
Document-topic distributions
Metadata
Text Mining Using LDA with Context 55/68Steffen Staab
Hierarchical Multi-Dirichlet Process Topic Model (HMDP)
Inference:Nearly completely collapsedinference!
Text Mining Using LDA with Context 56/68Steffen Staab
Hierarchical Multi-Dirichlet Process Topic Model (HMDP)
We only need to learn Global topic distribution Topic assignments to words
Text Mining Using LDA with Context 57/68Steffen Staab
Hierarchical Multi-Dirichlet Process Topic Model (HMDP)
We only need to learn Global topic distribution Topic assignments to words Dirichlet parameters
Text Mining Using LDA with Context 58/68Steffen Staab
Hierarchical Multi-Dirichlet Process Topic Model (HMDP)
Approximations: Variational Practical Stochastic
→ low memory consumption→ online inference
Text Mining Using LDA with Context 59/68Steffen Staab
Parameters of HMDP
Cluster-topic distributions:How many documents of a cluster contain topic x?
Text Mining Using LDA with Context 60/68Steffen Staab
Parameters of HMDP
Cluster-topic distributions:How many documents of a cluster contain topic x? Metadata-weightsHow many of the topics of documents are explainedby metadata x?
Text Mining Using LDA with Context 61/68Steffen Staab
Parameters of HMDP
Cluster-topic distributions:How many documents of a cluster contain topic x? Metadata-weightsHow many of the topics of documents are explainedby metadata x? Dirichlet process scaling parametersHow many pseudo-counts do we add to the topic
distributions?
Text Mining Using LDA with Context 62/68Steffen Staab
Properties of HMDP
Interpretable parameters Simultaneous inference of topics and metadata-topic
dependencies Efficient online inference
Text Mining Using LDA with Context 63/68Steffen Staab
Comparison of Topic Models for Arbitrary Metadata
Text Mining Using LDA with Context 64/68Steffen Staab
Comparison
Gaussian Process Topic ModelThe “perfect” model:
Can cope with arbitrary metadata Models dependencies between metadata Parameter learning is very expensive Kernel selection and inference require expert knowledge Parameters of Gaussian processes hard to interpret
Text Mining Using LDA with Context 65/68Steffen Staab
Comparison
Multinomial Regression Topic ModelThe “straight-forward” model:
Can cope with many metadata Parameter learning is cheaper than for Gaussian
processes but still expensive (due to alternating inference and repeated distance calculations)
Can not cope with complex metadata(e.g. geographical, cyclic, ...) Does not model dependencies between metadata Regression weights of Dirichlet-multinomial regression
hard to interpret
Text Mining Using LDA with Context 66/68Steffen Staab
Comparison
Hierarchical Multi-Dirichlet Process Topic ModelThe “fast” model:
Can cope with arbitrary metadata Fast inference (simultaneously for topics and topic
predictions) All parameters have natural interpretations as probabilities
or pseudo-counts Requires a (simple) pre-clustering of documents Does not model dependencies between metadata
Text Mining Using LDA with Context 67/68Steffen Staab
THANK YOU FOR YOUR ATTENTION!
top related