attribute learning for understanding unstructured social activity
DESCRIPTION
Attribute Learning for Understanding Unstructured Social Activity. Yanwei Fu, Timothy M. Hospedales , Tao Xiang, and Shaogang Gong School of EECS, Queen Mary University of London, UK Presented by Amr El-Labban VGG Reading Group, Dec 5 th 2012 . Contributions. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/1.jpg)
Attribute Learning for Understanding Unstructured Social Activity
Yanwei Fu, Timothy M. Hospedales, Tao Xiang,
and Shaogang Gong
School of EECS, Queen Mary University of London, UK
Presented by Amr El-Labban
VGG Reading Group, Dec 5th 2012
![Page 2: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/2.jpg)
Contributions
1. Unstructured social activity attribute (USAA) dataset
2. Semi-latent attribute space
3. Topic model based attribute learning
![Page 3: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/3.jpg)
Objective
Automatic classification of unstructured group social activity
Use an attribute based approach
Start with sparse, user defined attributes
Add latent ones
Learn jointly
![Page 4: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/4.jpg)
Dataset
1500 videos, 8 classes
69 visual/audio attributes manually labelled
Weak labelling
SIFT, STIP and MFCC features used
Data available (features, attributes, YouTube IDs)
![Page 5: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/5.jpg)
Semi-Latent Attribute Space
Space consisting of:
User defined attributes
Discriminative latent attributes
Non-discriminative (background) latent attributes
![Page 6: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/6.jpg)
Topic modelling
P(x|d)
d
x
x – low level features (‘words’)
y – attributes (‘topics’)
d – ‘documents’
= P(x|y)
y
x
P(y|d)
d
y
![Page 7: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/7.jpg)
Latent Dirichlet Allocation
y x
x – low level features
y – attributes (user defined and latent)
θ – attribute distribution
φ – word distribution
α, β – Dirichlet parameters
![Page 8: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/8.jpg)
Aside: Dirichlet disribution
Distribution over multinomial distributions
Parameterised by α
α = (6,2,2)
α = (6,2,6)
α = (3,7,5)
α = (2,3,4)
![Page 9: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/9.jpg)
Aside: Dirichlet disribution
Important things to know:
- peak is closer to larger values - large gives small variance <1 gives more sparse distributions
![Page 10: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/10.jpg)
Latent Dirichlet Allocation
y x
x – low level features
y – attributes (user defined and latent)
θ – attribute distribution
φ – word distribution
α, β – Dirichlet parameters
![Page 11: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/11.jpg)
Latent Dirichlet Allocation
y x
Generative modelfor each document:
Choose θ ~ Dir(α)Choose φ ~ Dir(β)for each word:Choose y ~ Multinomial(θ) Choose x ~ Multinomial(φ y)
![Page 12: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/12.jpg)
Latent Dirichlet Allocation
y x
𝑃 (𝐷|𝛼 , 𝛽)=∏𝑘=1
𝐾
𝑃 (𝜑𝑘∨𝛽)∏𝑚=1
𝑀
𝑃 (𝜃𝑚∨𝛼)∏𝑛=1
𝑁
𝑃 (𝑦𝑚 ,𝑛|𝜃𝑚 )𝑃 (𝑥𝑚 ,𝑛∨𝜑 𝑦𝑚 ,𝑛)
![Page 13: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/13.jpg)
Latent Dirichlet Allocation
y x
EM to learn Dirichlet parameters:
Approximate inference for posterior:
![Page 14: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/14.jpg)
SLAS
User defined part Per instance prior on α. Set to zero when attribute isn’t present in ground truth
Latent part First half “class conditional”
One α per class. All but one constrained to zero.
Second half “background” Unconstrained
![Page 15: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/15.jpg)
Classification
Use SLAS posterior to map from raw data to attributes
Use standard classifier (logistic regression) from attributes to classes
![Page 16: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/16.jpg)
N-shot transfer learning
Split data into two partitions – source and target
Learn attribute models on source data
Use N examples from target to learn attribute-class mapping
![Page 17: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/17.jpg)
Zero-shot learning
Detect novel class Manually defined attribute-class “prototype” Improve with self-training algorithm:
1. Infer attributes for novel data
2. NN matching in user defined space against protoype
3. For each novel class:
a) Find top K matches
b) Train new prototype in full attribute space (mean of top K)
4. NN matching in the full space
![Page 18: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/18.jpg)
Experiments
Compare three models:
Direct: KNN or SVM on raw data
SVM-UD+LR: SVM to map raw data to attributes, LR maps attributes to classes
SLAS+LR: SLAS to map raw data to attributes, LR learns classes based on user-defined and class conditional attributes.
![Page 19: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/19.jpg)
MASSIVE HACK
“The UD part of the SLAS topic profile is estimating the same thing as the SVM attribute classifiers, however the latter are slightly more reliable due to being discriminatively optimised. As input to LR, we therefore actually use the SVM attribute classier outputs in conjunction with the latent part of our topic profile.”
![Page 20: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/20.jpg)
Results - classification
SLAS+LR better as number if training data and user defined attributes decreases
Copes with 25% wrong attribute bits
![Page 21: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/21.jpg)
Results - classification
KNN and SVM have vertical bands – consistent misclassification
![Page 22: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/22.jpg)
Results – N-shot transfer learning
Vary number of user defined attributes
SVM+LR cannot cope with zero attributes
![Page 23: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/23.jpg)
Results – Zero-shot transfer learning
Two cases: Continuous prototype – mean attribute profile Binary prototype – thresholded mean
Tested without background latent attributes (SLAS(NF))
![Page 24: Attribute Learning for Understanding Unstructured Social Activity](https://reader036.vdocuments.site/reader036/viewer/2022062520/56815d39550346895dcb3c85/html5/thumbnails/24.jpg)
Conclusion
Augmenting SVM and user defined attributes with latent ones definitely helps.
Experimental hacks make it hard to say how good the model really is…