OHSU @ MediaEval 2015: Adapting Textual Techniques to

Multimedia Search

Shiran Dudy and Steven Bedrick !

Center for Spoken Language Understanding Oregon Health & Science University

Concept

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

r d

Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

Relevance feature vector

r d

Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

Relevance feature vector

r d

Diversity relationships with selected docs

Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

Relevance feature vector

r d

Diversity relationships with selected docs

Weight vectors

Features

Relevance LSA (100) user credibility - “visualScore” - “faceProportion” - “tagSpecificity” - “uniqueTags” - “locationSimilarity”

Diversity LDA (20) cosine disimilarity “csd” (L2) “hog” (Bhatacharyya) “cn” (L2) “cm” (Canberra) “lbp” (χ2) “glr” (L1)

Two Case Studies

Textual Features Best RunQuery: “concerts in Bucharest”

CR@20=0.71

Image Features Best RunQuery: “Amsterdam gay parade”

CR@20=0.67

Thank You! :)

Questions?

APX

Learning Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

To extract the weight vectors Wr and Wd we use

Learning Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

That is with the loss function:

So we compute their gradients by simply taking their derivative

Learning Algorithm

Y. Zhu et al. Learning for search result diversification. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 293–302. ACM, 2014.

Diversity Feature Vector Rij

subtopic diversity

!

!

!

!

!

Diversity Feature Vector Rij

subtopic diversity

!

!

!

!

!

Probabilistic LSA/LDA

Diversity Feature Vector Rij

subtopic diversity

!

!

!

!

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

!

!

!

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

!

!

!

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

!

!

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

!

!

content and importance

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

!

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

!

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

linked-based diversity

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

linked-based diversity Generating Diverse and Representative Image Search Results for Landmarks, 2008, Lyndon Kennedy

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

linked-based diversity

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

linked-based diversity

url-based diversity

Diversity Feature Vector Rij

subtopic diversity

text diversity

title diversity

anchor text diversity

ODP-based diversity

linked-based diversity

url-based diversity

Diversity Feature Vector Rij

However, we must include image features to determine relevance and diversity as well.

including the ability to switch on and off different features

considering internet information

incorporating credibility information