a new approach for adding metadata to online media - tnc2012

A new approach for adding metadata to online Media

Frans WardTNC 2012 - Reykjavik, May 22 2012

The results of two MediaMosa technology-scouting projects

Tuesday, May 22, 12

A new approach for adding metadata to online Media - TNC2012 - Reykjavik - [email protected]

• Development and exploitation of - the Dutch National Network for

Higher Education and Research - innovative ICT platforms & services

• By and for the Dutch Higher Education and Research community- 160+ connected organizations,

serving 1 million users- Not for profit, 85 employees

2

About SURFnet...

Tuesday, May 22, 12


About MediaMosa....

3

• Open Source (GPLv2)software to build a Digital Asset Management platform (DAM)

• Since 2008, MediaMosa is the basis for various successful video services that SURFnet provide to the Dutch educational sector

• Many adopters and large community

• Proven to be an excellent ‘vehicle for innovation’

Tuesday, May 22, 12


MediaMosa is based on a Service Oriented Architecture

4

• Flexibility: combining components• Scalable for future expansion• Open source and using open

source components

MediaMosai.e. SURFmedia

Video Frontend Video Backend

REST calls in XML

➡ Common base for MediaMosa is Drupal 7

Tuesday, May 22, 12

A new approach for adding metadata to online Media - TNC2012 - Reykjavik - [email protected] 5

MediaMosa API - http://mediamosa.org/api

• Play video • Authentication• Authorization (Play restriction)• Upload (PUT, POST, FTP)• Transcoding• Media Management: media

files, assets and collections• Search• Jobs• Metadata and OAI/PMH• Notification• Logging and Statistics• Stills• ....

REST based webservicesover 185, for services like...

Tuesday, May 22, 12


Introducing MediaMosa SiteBuilder

6

Tuesday, May 22, 12


MediaMosa SiteBuilder...

Key features:

• federative login (SAML2)• upload and play media• add play restrictions to media• transcode media• metadata• skinnable• easy installable

...a Drupal distribution for a media site connected to MediaMosa

Tuesday, May 22, 12


MediaMosa SiteBuilder...

Key features:

• federative login (SAML2)• upload and play media• add play restrictions to media• transcode media• metadata• skinnable• easy installable

...a Drupal distribution for a media site connected to MediaMosa

An example site can be viewed at: https://sitebuilder.mediamosa.surfnet.nl/

Tuesday, May 22, 12

https://sitebuilder.mediamosa.surfnet.nl/

https://sitebuilder.mediamosa.surfnet.nl/


Community Website & Online Demo

8

http://mediamosa.org

Tuesday, May 22, 12




Community Website & Online Demo

8

http://demo.mediamosa.org


Tuesday, May 22, 12






A growing MediaMosa Community

9

Tuesday, May 22, 12


MediaMosa Status & Roadmap

Continued Development in 2012: • July 2012

MediaMosa 3.5 ‘Flexible workflows, Speech-to-Text functionality, support for: Postgres, BagIt, JPEG2000, PDF generation, Command-line using Drush (Drupal Shell)'

• December 2012MediaMosa 4.0 'Better Drupal integration, Drupal Rules, REST call caching, Document oriented database support (ie. Mongo DB with Grid FS) offering even more scalability and flexibility’

Latest stable release on Github: Release 3.2.1https://github.com/mediamosa/

Tuesday, May 22, 12

http://www.mediamosa.org



MediaMosa Status & Roadmap

Continued Development in 2012: • July 2012

MediaMosa 3.5 ‘Flexible workflows, Speech-to-Text functionality, support for: Postgres, BagIt, JPEG2000, PDF generation, Command-line using Drush (Drupal Shell)'

• December 2012MediaMosa 4.0 'Better Drupal integration, Drupal Rules, REST call caching, Document oriented database support (ie. Mongo DB with Grid FS) offering even more scalability and flexibility’

Latest stable release on Github: Release 3.2.1https://github.com/mediamosa/

Concrete steps are taken to make MediaMosa sustainable by forming an independent MediaMosa Foundation in June 2012

Tuesday, May 22, 12




MediaMosa is ‘Networking to Services’

Generic Online Services

Realtime Collaboration

Services

Infrastructure Services

File StorageServices

Services for Education & Online Content

Services for Research

Video Services

Two service providers will offer MediaMosa based video services to the R&E market using SURFconext

Tuesday, May 22, 12


It’s all about metadata...

Categories of Metadata:

• User Generated Metadata(title, description, ..)

• Technical Metadata(file analysis by FFMPEG)

• Social Metadata(comments, tags..)

• Automatic Generated Metadata (product of ASR and OCR)

• ...and more

Tuesday, May 22, 12








• ...and more

➡Video Annotation Project

Tuesday, May 22, 12








• ...and more

➡Video Annotation Project

➡ Transcripting Project (Speech-to-Text)

Tuesday, May 22, 12


Video Annotation Project

Why? • adding metadata makes the video better searchable and usable• adding user interaction will make the video more enjoyable to

watch

Can you combine different video annotation techniques to enrich video?

Ergo: to enhance usability of video in education

Tuesday, May 22, 12


Video Annotation 1: store user comments

• Proof of Concept using HTML5 player

• User can comment on video content

• Comments made visible using SubRip subtitle format

• and stored in

Tuesday, May 22, 12


• Using Twitter to enrich video with comments

• Comments are stored as tweets

Video Annotation 2: social interaction

Tuesday, May 22, 12


• ‘Hotspots’ in video can be added

• Activating this hotspot can start a new event

• Events can be: start a new video, show additional information etc.

• Using WebVTT format

Video Annotation 3: make video interactive

Tuesday, May 22, 12


UK National Film and Television Archive, Berkhamstedhttp://www.flickr.com/people/footage/

Transcripting Technology Project

Tuesday, May 22, 12

http://www.flickr.com/people/footage/






Transcripting Technology Project

...is all about disclose of audiovisual archives

Tuesday, May 22, 12






Disclosure of audiovisual archives

18


• The number of AV-archives on the Internet increases rapidly

• Archiving is not enough: disclosure and reusing is required!

• Adding Metadata is the key component here

• The use of speech technology is needed (to reduce human effort)

Tuesday, May 22, 12


Metadata, the traditional approach

19

Adding Manually

Huge amount of workand no time-coded relations with video

Tuesday, May 22, 12


Speech Recognition(Speech-to-Text)

Indexing and Search:Search on fragment level

Audio Extraction

Metadata, the new approach

using Speech-2-Text technology is key here

Tuesday, May 22, 12


MediaMosaTranscription by Spraak

Multi-SourcePlayer

Partners:

• Enhanced Search• Optional Subtitles• Mashup info

Web lecture recording App

• Recognize the Speech• Produce time-coded

Transcript• CMU Sphinx for English

• Recording of Teacher• Recording of Slides• Reference material

• Transcode into audio• Store all into an asset

Transcripting Technology Use Case

Tuesday, May 22, 12


In fact, this is all we need...

Tuesday, May 22, 12


In fact, this is all we need...

a timecoded transcript

Tuesday, May 22, 12




There is a bonus: subtitles in SubRip format

Tuesday, May 22, 12


Demo Transcripting Technology

Online demo: http://spraak.mediamosa.surfnet.nl/

Tuesday, May 22, 12

http://spraak2.mediamosa.surfnet.nl/node/40



























































































http://spraak.mediamosa.surfnet.nl

http://spraak.mediamosa.surfnet.nl


MediaMosa Directions and Q&A

25

WWWhttp://mediamosa.org

Online Demohttp://demo.mediamosa.org

Forumhttp://mediamosa.org/forum

Issue Trackerhttp://mediamosa.org/trac

Source Codehttps://github.com/mediamosa

Slidesharehttp://www.slideshare.net/MediaMosa

Twitterhttp://twitter.com/mediamosa

MediaMosa

MediaMosa

MediaMosa

Tuesday, May 22, 12



http://mediamosa.org/forum

http://mediamosa.org/forum

http://www.mediamosa.org/trac/

http://www.mediamosa.org/trac/

http://www.slideshare.net/MediaMosa/

http://www.slideshare.net/MediaMosa/

[email protected]

Tuesday, May 22, 12

mailto:[email protected]

mailto:[email protected]