lime recommendation

© Fraunhofer IDMT

T. Köllmer†, E. Berndl‡, T. Weißgerber‡, P. Aichroth† , H. Kosch‡

A Workflow for Cross Media Recommendations based on Linked Data Analysis

† Fraunhofer IDMT ‡ University of Passau

Thomas Köllmer

[email protected]

© Fraunhofer IDMT 2

The MICO Project

http://www.mico-project.eu

FP7 STREP, GA# 610480

7 Partners: Salzburg Research, Fraunhofer IDMT, Oxford University,

University of Passau, Umeå University, Zaizi, InsideOut10

Duration: 36 Months (end: 10/2016)

Goal of MICO is to develop a platform to analyse “media in context” …

...by orchestrating different content extraction tools that can work simultaneously or in a sequence and …..

...provide valuable metadata to third party applications.


Cross Media Recommendations in MICO

The Recommendation Problem

“Providing suggestions for items to be of use to a user”

Broad Application domains:

e-Commerce, Travel, Multimedia, News, …

Two main data sources:

User behaviour data (Collaborative Filtering)

Item Metadata (Content based recommendation)


Cross Media Recommendations in MICO

Media Recommendation

“Classic” Recommendation problems (Music-Recommendation, Netflix Challenge)

Mostly based on collaborative filtering, but also content analysis

Cross Media Recommendation

Cross Media ≠ Cross Modal

Multiple Input

Media Items + Associated Media context, e.g., commentary

Multiple Output

Not necessarily more of the same thing


Editor Support Usecase

As an editor, while I create or edit articles using WordPress, I want to automatically

get related articles and videos that I might link to the article.

Project Use Case: Recommendations for Greenpeace Magazine Italia

Articles

Videos

User behaviour data


Editor Support Usecase

As an editor, while I create or edit articles using WordPress, I want to automatically

get related articles and videos that I might link to the article.

How can Mico Help?

Articles: NER components for content spotting

Videos

Text to speech + NER

Metadata Analysis

User behaviour data

Generic collaborative filtering component (prediction.io)


Chat Analysis Usecase

As a zooniverse user, I want get more information that helps me during the current classification task, e.g., by the recommendation of similar subjects or training data.

Input:

Results from animal detection

Sentiment and competence classification of talks

NER for animal spotting

Output:

Classification Hints

Hints for moderators / experts


Processing Workflow for Editor Support Use Case

1. Crawler feeds media items to platform

e.g., videos from Greenpeace channel, news articles, internal archive….

2. Analysis results are stored as Linked Data

Currently: using Apache Marmotta as triplestore


Processing Workflow for Editor Support Use Case

3. Item gets into Focus (someone is writing an article on a specific topic)

4. Focused item is fed to the same platform, using customized pipelines

5. Results are processed by LD matching, i.e Recommendation component..

6. … enriched by stored annotations …

7. And returned to editing platform


Recommendation Workflow in MICO


Ontology Matching

Semantic distance of entities important for analysis

Animal detection: “I see a bird on the tree”, “Yes it’s definitely a sparrow”

Topic detection: Wind energy, sustainability, renewables, oil, …

Non-Mico:

Genres! (e.g., currently 1400 genres on Spotify…)

Not less controversial: movie genres


Evaluation

How to evaluate a recommender system?

Forget about precision and recall!

Observing user behaviour

Online experiment

Offline: extrapolating past user behavior

Lab Studies / Questionnaire

Open problem: standardized + comparable testing of recommender systems


Conclusions

Proposed setting tries to formalize a certain subset of recommendationproblems: Cross media recommendation

Focused on Media Items

Profits from Content Analysis

New items, have a high contextual dependency on existing items, thatwill be exploited for recommendation

Work in Progress:

Stable release of Mico-platform and extractors: right now

PoC recommendation code will be integrated in the platform

Demo Services: available in June

Incorporate feedback into recommendation

© Fraunhofer IDMT

T. Köllmer†, E. Berndl‡, T. Weißgerber‡, P. Aichroth† , H. Kosch‡

A Workflow for Cross Media Recommendations based on Linked Data Analysis

† Fraunhofer IDMT ‡ University of Passau

Thank you!Questions?

lime recommendation

Technology