1 caspar contemporary arts testbed: caspar contemporary arts testbed: preservation of interactive...

38
1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi ICSRiM - University of Leeds [email protected] www.icsrim.org.uk/caspar

Upload: alexander-rogers

Post on 27-Mar-2015

219 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

1

CASPAR Contemporary Arts Testbed:CASPAR Contemporary Arts Testbed:Preservation of

Interactive Multimedia Performances

Kia Ng and Eleni Mikroyannidi

ICSRiM - University of [email protected] www.icsrim.org.uk/caspar

Page 2: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Overview

• Context and Motivation– CASPAR EC IST Project Contemporary Arts Testbed– Interactive Multimedia Performance (IMP)

• Music via Motion (MvM)

• i-Maestro 3D Augmented Mirror (AMIR) System

• ICSRiM Conducting Interface

• Research and Development– IMP dataset– CIDOC-CRM and FRBR ontologies

• Result and Conclusion– ICSRiM IMP Archival System

• Interface and functionalities

• Scenarios

• Validation

Page 3: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Context

• CASPAR EC IST Project–Build a framework to support end-to-end preservation

lifecycle for scientific, artistic and cultural information

• Contemporary performing arts testbed–Interactive Multimedia Performance (IMP)

• Interactive Multimedia–Interactive multimedia performances

–Performance capture/recording and analysis beyond normal AV

Page 4: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Music via Motion (MvM)

• MvM: a framework to integrate interactive multimedia, motion sensing, audio synthesis, VR and AR technologies

– To explore virtual and augmented instruments

– To provide users/performers with real-time control of multimedia events with their physical movements

• Application areas include stage performance and installation arts

Page 5: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

IMP Elements and Processes

Example of an IMP production processbased on the Music via Motion (MvM) system, www.kcng.org/mvm

Data Analysis, Classification

and Recognition

Mapping

Multimedia Generation

GUI (for monitor & control)

Data Capture & Processing

Motions

multimodal data

Multimedia output

Mapping Strategy Parameters

Page 6: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

i-Maestro 3D Augmented Mirror (AMIR)

• Technology-enhanced learning

• More in-depth recording and understanding

– playing gesture– Intangible heritage– Individual performance

style

• Data stream include: audio, video, 3D motion data, sensor data, analysis

• Multimodal feedbacks with graphical visualisation and sonification

• Max/MSP jitter, c/c++, Lua, SDIF

Excerpt from Bach's Partita No. 3 in E major. BMW 1006 (Preludio)

Page 7: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

AMIR (www.i-maestro.org)

Page 8: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

ICSRiM Conducting Interface

• A system for tracking and analysing a conductors hand movements– Uses multiple wiimotes to

capture a conductors movements

– These movements are analysed

– This information is then fed back to the user in an entertaining yet educational manner

Page 9: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

System Setup

Page 10: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

GUI

• 3D environment• Intuitive UI• Multiple modes• Multimodal data capture and

analysis

Page 11: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Complexity of the IMP Preservation

• IMP rely heavily on digital media• A recording (e.g. video) of a performance is not

sufficient for re-performance• Preserving individual components of a

performance is clearly a challenge• However, putting these components together in

one place does not make a performance• They need to be assembled in a logical and

temporal order with their inter-relationships• Preserving this knowledge through time is even

more challenging

• Preservation for– Re-performance at a later time– Historical study/analysis in performing arts in the

future– Better preparation for the future needs

Example IMP interface from the i-Maestro EC IST Project involving 3D motion data, sonification, and graphical feedbacks

Example IMP - from a Music via Motion (MvM) performance involving motion generated music with dance

Page 12: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

12

IMP Dataset

• An Interactive Multimedia Performance (IMP) work typically consists of the following digital objects: – Audio– Video– 2D/3D motion data– Sensor data– Max/MSP patches– Configuration files– and others

• For reconstruction of an IMP work– Representation Information– Inter-relationship

• Logical• Temporal

Page 13: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

CIDOC-CRM for IMPs

• The CIDOC Conceptual Reference Model (CIDOC-CRM) has been proposed as a standard ontology for enabling interoperability among digital libraries

• It defines a set concepts for physical and temporal entities

Conceptual Objects

Physical Objects

Types

Temporal Entities

Ap

pel

lati

ons

Actors

PlacesTime Spans

are referred/ refine

participate in

within at

affect/ refer to

has location

refer to/ identify

Page 14: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

CIDOC-CRM for IMPs (2)

• CIDOC-CRM can be used to describe a performance– at a high level

• More specialised vocabularies are necessary for IMPs– e.g. how different types of equipment are connected

together in the performance.

• CIDOC-CRM – lacks concepts for digital objects

– is designed primarily for documentation of what has happened, whereas in digital preservation, it is also required to document the reconstruction of a past event from preserved components.

Page 15: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

CIDOC-CRM FRBR Extension

• Two main objectives in extending CIDOC-CRM for the preservation of an IMP:– To provide a domain specific vocabulary for describing an IMP.

• e.g. details on how the archived performance was carried out and how it can be re-performed

– To provide a vocabulary for describing digital objects, their interrelationships and operations performed on them in the digital preservation context

• FRBR (Functional Requirements for Bibliographic Records) – describes a high level conceptual model of creative works and how

they are represented in the real world

• It is used as extension ontology of the CIDOC-CRM – for describing the conception process of the performance

15

Page 16: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

16

Representation Information

• The Representation Information for this performance is in the form of CIDOC-CRM and FRBR instantiation.

F52.PerformanceIMP

F8.PersonKia (Director)Frank (Performer)

E53.PlaceLeeds - UK

E52.Time-Span2hours:5PM-12/02/07

E22.Man-Made ObjectCelloSound MixerComputer System

E73.Information ObjectMusicMusic Score

P16F – use specific object

P16F – use specific object

P7F – took place atP4F – has time span

P14F – carried out by

Page 17: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

17

Mapping IMP detailsto CIDOC-CRM and FRBR

• The FRBR concept “F52.Performance” is used to model an IMP.

• The FRBR concept “F8.Person” represents the people involved in the performance, e.g. directors, performers, MAX/MSP programmers, etc.

• The digital objects used (software patches, 3D models, etc) are instances of the “E73.Information Object” CIDOC-CRM class, whilst any physical objects (instruments, computer equipment, etc) are instances of the “E22.Man-Made Object” class.

• The “E52.Time-Span” and “E53.Place” classes are used to represent the time-span and location of the performance respectively.

Page 18: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

18

Mapping IMP detailsto CIDOC-CRM and FRBR (contd.)

Page 19: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

ICSRiM IMP Archival System

• The ICSRiM IMP Archival System – for the preservation of different IMPs– based on the CASPAR Framework – Integration of the selected CASPAR components

• Communication with the Repository containing the Knowledge base and the DataStore

ArchivalSystem

Knowledge Base

Repository DataStore

communicatesRepInfo

PerformanceFilesCASPAR

Web Services

calls

Page 20: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Web Interface of the IMP Archival System

Page 21: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Cyclops tool

• A tool to capture appropriate Representation Information to enhance Virtualisation and future re-use of the digital object.

• Cyclops provides a usable interface to describe the Performances in the form of graphs

• The graph in the backend is saved as an RDF instantiation of the CIDOC and FRBR ontologies

• Defined terminologies according to various data types

Page 22: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Cyclops Interface

Page 23: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Ingestion and Retrieval of an IMP

• Scenario 1: Ingestion of an IMP– Description of the performance with the use of

Cyclops• Generation of the Representation Information

– Both the metadata and the digital files are ingested and stored in the Repository

• Scenario 2: Access of an IMP– Queries are performed on the metadata – The related objects are returned to the user– Instructions extracted from the RepInfo provide

details on how to assemble the objects

Page 24: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Scenario 1: Ingestion of an IMP

Steps in Screenshots

24

Page 25: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 1: Describe the IMP

• Follow the Ingest wizard to complete the ingestion

25

Page 26: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 1: Describe the IMP (2)• Create Representation Information with the use of Cyclops tool• It is saved in RDF format. In this way, the graph was transformed into

an RDF instantiation of the CIDOC-CRM and FRBR ontologies.

26

Page 27: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 2: RepInfo parsing

• The system detects and parses the RepInfo for defined data types

27

Page 28: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 3: Addition of the digital files

• Upload the related digital files of the performance• The files are linked to the RepInfo• The package is stored in the Repository

28

Page 29: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Scenario 2: Retrieval of an IMP

Steps in Screenshots

29

Page 30: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step1: Search for the IMP

• The search is done by the name of the performance

30

Page 31: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step1: Search for the IMP (2)

• The system returns a table with the following selected information from the RepInfo– The name of the Performance– Details on the performance– The associated with the performance files– The links to download the files of the performance– Details on the files (e.g. which one is the data file,

the patch file, how to use them etc.)– A link to the RDF description.

31

Page 32: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 2: Download Associated Performance files

• Select and download the components of the performance

32

Page 33: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Step 3: Reconstruct the performance

• The user has to understand the connection of the components

• Instructions on the connection is provided in the table• By assembling the components the user is able to see the

performance

33

Page 34: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Retrieval of an IMP

Page 35: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Validation and Feedback• Participant

– A small set of experts who work with IMP

• Validation– Ingestion of an IMP work (their own work)

– Retrieval and reconstruction of an IMP they have no information about it

• Download and connect the components using the instructions

– Fill in a questionnaire with their feedback

• Results– The system is useful to store and preserve the IMPs

– The system provided all the needed instructions for the reconstruction of an IMP

• The appropriate RepInfo has been captured

– Some difficulty in using Cyclops (complexity of the graphs)

• Demo videos and tutorials helped

• Cyclops graph templates to help to get started

Page 36: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

CASPAR IMP References

• Selected publications include:– E. Mikroyannidi, B. Ong, D. Giaretta and K. Ng, “Preservation of

Interactive Multimedia Performance with the use of Ontology Models”, in Proc. of the Digital Resources for the Humanities & Arts (DRHA) Conference, Belfast, September 2009

– K. Ng, E. Mikroyannidi, B. Ong, N. Esposito and D. Giaretta, "Interactive Multimedia Systems for Technology-Enhanced Learning and Preservation" in Proc. Of the 15th International Conference on Distributed Multimedia Systems (DMS) 2009, San Francisco, September 2009.

– K. Ng, T.V. Pham, B. Ong, A. Mikroyannidis, and D. Giaretta, "Preservation of Interactive Multimedia Performances", International Journal of Metadata, Semantics and Ontologies, 3(3), 28 February 2009 , pp. 183-196, Inderscience Publishers.

• For latest publications on CASPAR IMP Preservation, see– www.icsrim.org.uk/caspar– CASPAR project website

36

Page 37: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Conclusions

• Preserving knowledge is vital for future reconstruction of interactive multimedia performances

• Proposed IMP preservation process– Made use of standard ontology models, CIDOC-CRM and

FRBR to create Representation Information• Used to define interrelationships between the components

• Creation of a web-based archival system for the preservation of IMPs– Integrated selected components of the CASPAR

Framework

– Current prototype online:• http://www.icsrim.org.uk/imp-preserve

37

Page 38: 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi

Thank you Kia Ng and Eleni Mikroyannidi

ICSRiM – University of Leeds

www.icsrim.org.uk/caspar [email protected]

38