co-me-di-a cooperation and mediation in digital arts 2007 - 2010 andrew gerzso - ircam co-me-di-a...

CO-ME-DI-A

COoperation and MEdiation in DIgital Arts2007 - 2010

Andrew Gerzso - IRCAMCO-ME-DI-A coordinator

CO-ME-DI-A partners

• IRCAM (F) - Institut de Recherche et Coordination Acoustique Musique, Paris.

• UNIGE (IT) - Università degli Studi di Genova, Genoa.

• CIANT (CZ) - Center for Art and New Technologies, Prague.

• SARC (UK) - Queen's University, Belfast.

• IEM (A) - University of Music and Dramatic Arts, Graz.

• HFMT (DE) - Hochschule für Musik und Theater, Hambourg.

• HCMF (HU) - Hungarian Computer Music Foundation, Budapest.

Previous Experience(some examples)

Network Performance at SARC

Geographically Displaced Ensembles

[2005]

- three-site Irish Traditional Music Session (with CCRMA, Stanford/USA, SARC and the

University of Washington)

- two-site Christmas laptop concert using Frequencyliator

[2006]

- three-site Christmas concert in 2006 (with CCRMA, Stanford/USA and UEA Norwich, UK)

[2007]

- Strings Apart Piano Duo (with CCRMA, Stanford.USA)

- Disparate Bodies v. 1.: three-site acoustic piece written specifically for the network (Nime

2007)

- Disparate Bodies v. 2.: three-site distributed ensemble piece (with IEM Graz and Hoschule

für Musik und Kunst Hamburg)

[2008]

- Laptop Ensemble (BLISS (Belfast Legion for Improvised Sights and Sounds) and Göteborg’s

Academy of Music and Drama laptop ensemble, Sweden

Network Performance at SARC

Performance in Virtual Environments

[2007]- Second Life as a performance venue (Disparate Bodies v.2)

[2008]

- SL Performance with Gestural Interaction between performer and

avatars

Future Plans

- Use of binaural streaming for spatial music

- Avatar development based on multimodal performance capture

Network Performance at SARC

Distributed Installations

[2008]

2 Places Exhibition (Belfast - Limerick)Audio Feedback loop through two vibrating metal

plates

Unreal bridge

CIANT server

CIANTClient A CIANTClient B CIANTClient X

CIANT SERVER

TCP, UDP multicast

GAME ENGINE

Transatlantic Rehearsal

•Feb. ‘07 - New World Symphony

•Reinbert de Leeuw conductor

•“Correspondances” by Henri Dutilleux

•“Color” by Marc-André Dalbavie

View from Ircam

View from NWS

“Ius Lucis” of Valerio

Sannicandro•Work for two ensembles in two

halls

•Centre Pompidou and IRCAM

•Premiere June 6, 2007

•Ensemble Intercontemporain / Xavier Roth

Important Ideas

•Remote listening of “the other” hall

•Instruments are cross spatialized

•DSP cross synthesis

•Spatialization composition

•Acoustic “signature” of each hall

Hall One : CNAC-GP

Hall Two : IRCAM

EyesWeb - Casa Paganini

QuickTime™ et undécompresseur Video

sont requis pour visionner cette image.

Test for “Rew(ind)”choreography : Hervé Robbecomposition : Andrea Cera

QuickTime™ et undécompresseur Animation

sont requis pour visionner cette image.

Comedia Main Goals

• doing and learning networking through public events, workshops and residencies.

• creation of a common platform for efficient communication and expression.

• organization of residencies in order develop the technical and artistic content.

First Step : Scenarios

•plain description of event or situation

•topology : configuration and function of nodes

•data : audio, video, control, motion capture

•dramaturgy : what should be perceived by audience, participants and external observers

“Comedia bus”

•common communication infrastructure

•continuous flow of data (audio, video, data).

•eases activating network events

“Lecture or Training Network

Event”•can be a simple oral lecture or

demonstration of technology involving audio and video

•the remote audience can express feedback via a local host or spokesperson for the audience

“Robotic Ensemble Remote

Installation”•robotic instruments placed on different nodes on the network

•each robot collects data provided by the other nodes

•each robot reacts to the collected data

•each place has a specific character

“Multi-way Performance with Remote Avatars”•2 sites with 2 musicians.

•avatar : replication of relevant aspects of “the original”.

•avatar data used in interactive processes

•audio, visual, (biometric ?) data.

•audience in each site has a specific perspective of the performance.

“Provokalia Choir Audience”

•one choir with conductor on one node

•two other choirs in remote nodes

•video and gesture data used for interaction

• “etude” to identify most effective and relevant expressive cues

“Remote Performance”

•a concert is transmitted to remote place where audience is

•concert audio “signature” or “image” is rendered to the audience - how to do this ?

•concert performers get feedback from audience

•awareness in audience that their feedback is perceived by the performers - how to do this ?

“Distributed Ensemble and Real Time Composition”• each node has a performer playing scores appearing in

real time on a screen

• conductor controls central computer generates the scores

• performer’s playing is analyzed locally and the derived data acts on central computer and thereby on the composition itself

• uses concept of virtual score

• audience at each node receives a “unified” audio image

• audience sees scores as they are generated

“Bi-directional Interactive

Performance”•two musicians at two nodes

•audio, video, motion capture data used for interaction

•two nodes differentiated by spatialisation and/or dramaturgical aspects

•each audience aware of the other

“Motion Enabled Live Electronics”

• spatial interface placed on stage on each of two nodes

•musicians and dancers act on collaborative virtual instruments controlled via motion capture

•a virtual audio image is mapped out over the network

•audio image is rendered to real and virtual audience spaces

“Network as Acoustic Medium”•network inherently contains delays

(speed of light !)

•delays, reverberation phenomena emerge

•dynamic real time connections to create sound “netscape”

“Network Mediated Social Interaction”• two nodes - each with an audience

• how to build up awareness of the other audience ?

• interactivity via sound and music to increase mutual influence

• work on the expression of a sense of community, “entrainement”, competition, shared goals ...

• scenario can be built up starting with individuals

“Multi-way Performance in Virtual

Environment”•3 sites with 3 musicians

•audio, visual and motion capture data

•content of 3 sites rendered to Internet audience in a virtual environment.

“Playing Together One Computer Music

Application”•one real time computer music program

on which several musicians act

•related to phenomena of “live coding”

•can be improvised or directed

•sound played in each location with same spatialisation and/or acoustic character

•parallel performer communication (chat ...)

SAME Project

“Network Active Listening”

Important Concepts

•local & global rendering

•external observers

•avatars

•virtual performance spaces

•audience awareness

•dramaturgy

Types of Dramaturgy

• projected dramaturgy : one node retains authorship of dramaturgy using other nodes as contributors with their specific expertise.

• directed dramaturgy : each node contributes specific expertise and supports the dramaturgy developed by an invited artist.

• distributed dramaturgy : each node contributes specific expertise and retains authorship of its own dramaturgy.

Context of Comedia

That’s where we are !

Questions ?