digital lutherie - tecn.upf.essjorda/gmv2004/tesis2005def.pdf · 11/04/2007 digital lutherie -...

Digital Digital LutherieLutherie

Crafting musical computers Crafting musical computers for new for new musicsmusics’’ performance performance

and improvisationand improvisation

Sergi Jordà

Music Technology GroupInstitut Universitari de l’AudiovisualUniversitat Pompeu Fabra, Barcelona

Barcelona, June 2005

11/04/2007 Digital Lutherie - Sergi Jordà 2

When asked what musical instrument they play, few computer musicians respond spontaneously with ’’I play the computer.’’Why not? (Wessel & Wright 2002).


As a performer and composer, my goal when constructing the instruments I will play is clear. I need instruments which will allow me to enjoy playing, particularly with other musicians, which will hopefully allow these other musicians to enjoy playing with me, and which will allow me to create music that will surprise me as much as possible, that will keep showing me little hidden secrets at every new listening, that I will therefore enjoy listening back.

Music not necessarily better, not worst either, than the one I could compose on my studio; but music, in essence, that could not have been created in any other possible way.


Musical interface construction proceeds as more art than science, and possibly this is the only way it can be done (Cook 2001).


INTRODUCTIONINTRODUCTION


New digital instrumentsNew digital instruments’’ designdesign……… is a broad field, which

encompasses highly technological areas (e.g. electronics and sensor technology, sound synthesis and processing techniques, software engineering, etc.)




and disciplines related to the study of human behavior (e.g. psychology, physiology, ergonomics and human-computer interaction components, etc.)





Trying to solve separate parts of the problem is essentialessential…





Trying to solve separate parts of the problem is essential……but also clearly insufficient.


Two axioms constitute the starting point of this thesis:

New instruments for new New instruments for new musicsmusics


Two axioms constitute the starting point of this thesis:

• New musics tend to be the result of new techniques, which can be both compositional or instrumental.

• New instruments will be able to survive and succeed in the measure they are really innovative; i.e. they have something new to bring to music, and not merely because they are based on novel, previously unavailable technologies.

New instruments for new New instruments for new musicsmusics


Premises & MethodologiesPremises & MethodologiesNew musical instruments based on computers can only be

improved by exploring three parallel paths




1) identifying the quintessence of new digital instruments; what they can bring of really original to the act of music performance; how can they redefine it;





2) identifying the drawbacks or obsolescences of traditional instruments; what limitations or problems could be eliminated, improved or solved;






3) without forgetting the essential generic assets of traditional instruments; those qualities that should never be forgotten nor discarded.


Dissertation OutlineDissertation Outline

• Introduction [chap. 1]• Part I: Theory [chaps. 2-7]• Part II: Praxis [chaps. 8-10]• Part III: Teachings & Conclusions [chaps. 11-12]

• plus …§ bibliography§ discography§ appendixes (list of publications, etc.)


PART IPART I• The purpose of this part is to progressively enlighten the three

fundamental aforementioned questions: a) quintessence of new digital instrumentsb) drawbacks or obsolescences of traditional instrumentsc) qualities of trad. instruments that should not be forgotten)

• Starts providing a thorough overview of the current know-hows and conceptual frameworks in which new instruments designers and researchers are currently working [chaps. 2-6]

• Ends presenting a theoretical framework for evaluating the potential, the diversity of new digital music instruments, as well as the possibilities and expressive freedom of human performers…[chap. 7]


PartPart I : I : CriteriaCriteria & & PurposesPurposes

• Overview of contemporary thinking, bodies of knowledge and concepts, more than of implementations

• Some real examples are however also presented and discussed§ chosen either for their historical relevance of for their

paradigmatical nature (when possible, for both)• Overview is neither neutral or aseptic, but filled with personal

ideas and criteria (critical analysis & synthesis)• Serves a dual purpose:§ useful per se (comprehensive coverage of the topic)§ introduces problems, limitations of current approaches…


PartPart I : I : OrganisationOrganisation• Reflects the interdisciplinarity of the field (technology

engineering, ergonomy, cognitive science, philosophy, musicology, history...)


acoustic instruments’ prelude

inputs & gestural controllers

interactive music systems (1)

interactive instruments: old & new paradigms

mapping & research in digital instruments design

towards a conceptual framework for new instruments design


Introduces possibilities & limitations of acoustic instruments concerning both mechanical inputs & control, and sonic output, and how both aspects are hardwired by physical constraints

Acoustic instruments tend to impose by construction their own playability rules, which allow listeners to infer the type and form of the gesture from the generated sound



NON-HUMAN EXCITATION ENERGY

AUTOMATED CONTROL PROCESSES

Hammond+Leslie

Analog modular synthesizer

Human vs. non human (a) excitation energy and (b) process automationin acoustic and electric instruments


If in acoustic instruments the separation between the control interface and the sound-generating subsystems is often fuzzy, in digital instruments, any input can be linked to any output and both components can offer a virtually infinite number of possibilities

• Sound synthesizers are not instruments!• Gestural controllers are not instruments!



inputs & gestural controllers

• Introduces controllers, responsible for extracting control information coming from the performers. These input devices constitute the first component of the digital instrument chain.

• Several types and taxonomies are presented (e.g. instrument-like controllers, extended controllers and hyperinstruments, non-imitative controllers, batons, bodysuits and exoskeletons, use of biosignals, cheap and borrowed controllers …)

• Several basic concepts such as dimensionality, resolution, sampling rate, dimension coupling, bandwidth... are also introduced.


interactive music systems (1)

interactive instruments: old & new paradigms


L’instrument philosophe est sensible; il est en même temps le musicien et l’instrument. Comme sensible, il a la conscience momentanée du son qu’il rend; comme l’animal il en a la mémoire. Cette faculté organique, en liant les sons en lui-même, y produit et conserve la mélodie. Supposez au clavecin de la sensibilité et de la mémoire, et dites-moi s’il ne se répétera pas de lui-même les airs que vous aurez exécutés sur ces touches. Nous sommes des instruments doués de sensibilité et de mémoire (Diderot à D’Alembert ~1770).


From From ‘‘scoredscored’’ instruments instruments to to ‘‘intelligentintelligent’’ instrumentsinstruments

It …seemed to me that there were a number of aspects of music where the performer didn’t have any freedom of choice. The sequence of notes to be played, for example, was an area in which the computer could legitimately help a performer in making it easier to do things correctly (Roads 1989, interview with Max Mathews).

Intelligent Musical Instruments are instruments that first sensed its performer’s intent by the type and context of the performer’s actions and then extended the performer’s controls to execute the music automatically. Intelligent instruments let people play the music on a compositional level (Gagne 1993, interview with Laurie Spiegel).


J.Cage “cartridgemusic” (1961)K.Stockhaussen(1964)

Live electronicsMEV, D.Tudor, G.Mumma, D.Buchla …

First hybrid systemsMathew’s GROOVE, J.Chadabe, S.Martirano

MicrocomputersThe League, G.Lewis, M.Waisvisz…

Early MIDIK.C.S., Music Mouse, M & Jam Factory, HMSL…

1960 19901970 1980


Interactive music/instruments Interactive music/instruments definitions, metaphors, taxonomiesdefinitions, metaphors, taxonomies

• John Bischoff et al. (1978)• Joel Chadabe (1984)• George Lewis (1985)• Michel Waisvisz (1985)• Jeff Pressing (1990)• David Wessel (1991)• Miller Puckette (1991)• Laurie Spiegel (1992)• Robert Rowe (1993)• Todd Winkler (1998)• ….Concepts (not implementations) are discussed here…leading sometimes to contradictory statements…


Some musical interaction Some musical interaction questions, problems or paradoxesquestions, problems or paradoxes

• Interaction (‘surprise’) vs. instrument control• Interactive or merely reactive? (e.g. score following)• Interaction and no improvisation? Is that possible?• Improvising with computers does necessarily mean

improvising with computers that try to behave like humans? § Intelligent instruments: knowledge?

Knowing/modeling what?§ Machine listening… the “only way”?


Interactive or merely reactive?


Multithreaded instruments Multithreaded instruments & shared control& shared control

• In traditional instruments the performer is responsible for controlling every smallest detail, leaving nothing to the instrument responsibility

• In new interactive music instruments, the instrument’s ‘intelligence’ may be partially responsible for one or more musical processes,the control of which may be entirely left to the instrument’s responsibility or may be shared in different ways with the performer

• When performers delegate some control to the instrument, their role is also approaching that of composers, who do have to delegate control of their work to instrumentalists


NON-HUMAN EXCITATION ENERGY

AUTOMATED CONTROL PROCESSES

Hammond+Leslie

Analog modular synthesizer


Some multithread & Some multithread & shared control corollariesshared control corollaries

These systems (instruments) can • surpass the one gesture to one acoustic event’ paradigm• go beyond the sound and note control level• current systems (i.e. after MIDI) allow in fact to bridge a

continuum between micro (i.e. timbre) & macrostructure, micro & macrocontrol (e.g. soundprocessingà machine listening continuum)

• run multiple and parallel musical processes in a shared control between the instrument and the performer è the possibility to have multiple performers seems as a logical and promising extension


mapping & research in digital instruments design


• The physical and logical separation of the input device from the sound production, brings the necessity to process and map the information coming from the input device in a variety of ways

• Mapping, studies the connection between gesturalparameters (or any other kind on input) and sound and music control parameters (or any other output results). It is an essential element in new instruments design

• This field is probably the closest that digital lutherieever gets to science; one of the few areas in which knowledge seems to behave in an incremental, accumulative way

MappingMapping


On ergonomics, motor & cognitive loads• Fitts (1954), Miller (1956)• Pressing (1988)• Rubine & McAvinney (1990)• Vertegaal & Eaglestone (1996) …On Mapping• Pressing (1990)• Lee & Wessel (1992)• Wanderley, Schnell, Rovan, Depalle, Hunt, Kirk…(1996-200x)• Arfib (2002)• Goudeseune (2002) …On haptic feedback• Cadoz et al. (1984, 2003)…• Bongers (1994)• Gillespie (1999) …

MainMain theorethicaltheorethical contentscontents


The unfulfilled expectations of The unfulfilled expectations of mappingmapping

• Current mapping tests rely on simple systems: simple inputs devices (or imitative controllers) and simple output control, (usually pitch, amplitude and one timbre parameter)

• No attempts to affront mapping in a context different than a sound synthesizer

• Few real musical implementations as a result of studies

• Hard to evaluate the music performance of new unknown musics (Music is not ‘simple’ HCI - what’s the goal? what are the tasks? )


Problems? Solutions?Problems? Solutions?• Low-level and focused research that tries to solve

independent parts of the problem is clearly essential for any real progression in this field, but it is also clearly insufficient..

• Crafting new instruments has to be more than designing hardware controllers, studying mappings and gluing components

• Integrated designs are needed• Holistic approaches are needed (…psychology,

philosophy… & MUSIC!!)• How do we study, compare or evaluate infinitely diverse

instruments that make musics that do still not exist?


REM: Premises & MethodologiesREM: Premises & MethodologiesNew musical instruments based on computers can only be




3) without forgetting the essential generic assets of traditional instruments; those qualities that should never be forgotten nor discarded.


The reality: The reality: new instruments possibilities are endless new instruments possibilities are endless

…… anything can be done, but anything can be done, but ……

• New virtuosi and/or professional musicians who use them as their main instrument is surprisingly small

• If live electronics and laptop music is so widespread why so many performers prefer to rely on the mouse or on generic and dull midi fader boxes ?

• Commercially available new instruments are scarce and hardly imaginative and ground-breaking (e.g KAOSS)

• A new standard electronic instrument is yet to arrive (?)• Technology evolves too fast for it ?


in shortin short……• We need useful, playable, thought-provoking enjoyable

instruments, capable of interesting, surprising, enjoyable music

• What is a good music instrument, anyway?• Myriad of criteria, subjective, changeable over history…

• Before we find the hidden, implicit properties that make traditional instruments good enough (!), we will try to find in what they could get better …

• … and we will try to do it, completely independently of the types of music they can play …


Most traditional instruments take a long time to master

We should be able to design well balanced instruments that can appeal to both professionals and dilettanti; instruments that like the piano, can offer a low entry fee with no ceiling on virtuosity (Wessel 2001)

How do we create controls and interactions that feel inevitable to expert and amateur users? (Machover, 2002)


Some inspiration sourcesSome inspiration sources……

the writersthe writers……

•• Pressing (1990)Pressing (1990)•• RubineRubine & & McAvinneyMcAvinney

(1990)(1990)•• Ryan (1991)Ryan (1991)•• ChadabeChadabe (1997)(1997)•• WaisviszWaisvisz (2000)(2000)•• Cook (2001)Cook (2001)•• WesselWessel & Wright (2001)& Wright (2001)•• Robson (2001)Robson (2001)•• SettelSettel and and LippeLippe (2003)(2003)•• Blaine and Blaine and FelsFels (2003)(2003)• ….

the improvisers the improvisers ……

•• Nicolas CollinsNicolas Collins•• Michael WaisviszMichael Waisvisz•• Tim PerkisTim Perkis

……..

and above all,and above all,all the all the unpluggedunplugged and/or and/or

analogue ones !!!analogue ones !!!


• Jordà, S. (2002). Improvising with Computers: A personal Survey (1989-2001). Journal of New Music Research, 31(1), 1-10.

• Jordà, S. (2004). Digital Instruments and Players: Part I – Efficiency and Apprenticeship. Proceedings of the 2004 Conference on New Interfaces for Musical Expression (NIME-04), 70-76. Hamamatsu.

• Jordà, S. (2004). Digital Instruments and Players: Part II–Diversity, Freedom and Control. Proceedings of the 2004 International Computer Music Conference. San Francisco, CA: International Computer Music Association.

• Jordà, S. (2005). Instruments and Players: Some thoughts on digital lutherie. Journal of New Music Research, 33(3),1-21.


Covered topicsCovered topics

• Learning curve [of a music instrument]• Music efficiency …• Music diversity …• Diversity, improvisation, configurability & identity• Virtuosity, non-linearity vs. control, predictability,

confidence, explorability…• Time considerations• Expressiveness


Musical efficiency & Musical complexitiesMusical efficiency & Musical complexities

Output complexity should include• the micro-sonic richness of the violin• the mid-sonic richness of the piano• the potential macro-structural complexities that could be managed

by any kind of “intelligent” digital instrumentInput complexity will depend on• degrees of freedom of control• correlation of controls• range and resolution• mappings• ergonomics…

InputOutput

Efficiency = EffortessEffectiven

iencyHumanEffic =

tyutComplexiControlInpityputComplexMusicalOut

iencyumentEfficMusicInstr =


Out

put C

ompl

exity

Input Complexity

tyutComplexiMusicalInpityputComplexMusicalOut

Efficiency =

?


Out

put C

ompl

exity

Input Complexity

tyutComplexiMusicalInpityputComplexMusicalOut

Efficiency =

?

!!!


Playing [with] music ?Playing [with] music ?

In ancient Greece, the kithara was usually played by In ancient Greece, the kithara was usually played by professional musicians as it required some kind of professional musicians as it required some kind of exhibitionism and virtuosity that could be considered vulgar exhibitionism and virtuosity that could be considered vulgar and not adequate for freeand not adequate for free--born men. The born men. The lyralyra instead, was instead, was commonly played by amateurs, and was, according to Plato, a commonly played by amateurs, and was, according to Plato, a well suited instrument for free citizen. well suited instrument for free citizen.

Before the advent of the radio and recorded music, families Before the advent of the radio and recorded music, families who could afford it, tended to have an instrument at home who could afford it, tended to have an instrument at home (often a piano) and some of the family members were enough (often a piano) and some of the family members were enough talented to become talented to become ‘‘animated interactive CD playersanimated interactive CD players’’ every every evening.evening.


The art of making music made easy (?)

It's true, making music is an Art, traditionally demanding years of study and practice to be successful. -- until now! Enter the new Suzuki Omnicord. If you can read this text, you can play the Omnicord, and play it well. There's no need for lessons or years of study to play and sing your favorite songs right now! Just press a chord button and strum the SonicStrings. It's that easy.


good instruments make bad music!• A good instrument should not impose its music to its

player.

• A good instrument should not be able to produce only good music! (What is good music anyway?)

• A good instrument should also be able to produce “terribly bad” music, either at the player’s will or at the player’s misuse.


Musical diversity & performer freedom:Musical diversity & performer freedom:a departure from the a departure from the efficiencyefficiency engineering engineering

viewpointviewpoint

The more ‘intelligent’ an instrument becomes, when complex processes are not anymore directly controlled by the user but possibly by other processes, the less meaningful a strict structural or informational dimensionality turns out to be, and only a perceptual musical complexity acquires sense

tyutComplexiControlInpersityMusicalDivytComplexitMusicOutpu

EfficMusicInstr×

=


On diversityOn diversity……

• Vertegaal et al. (1996)• Blaine & Fels (2003)• Nishimoto et al. (2003)• Settel & Lippe (2003)• Arfib et al. (2005) …

discuss in different ways “instrument versatility or flexibility”


Our diversityOur diversity

•• microdiversitymicrodiversity•• mididiversitymididiversity•• macrodiversitymacrodiversity

Some considerations …• They do not represent different time scales• Mididiversity nothing to do with MIDI!• Related to “music”, not to sound è not to

psychoacoustical perception• A continuum could be more realistic, but also less

clarifying


MicrodiversityMicrodiversity ((MicDMicD) : Performance nuances) : Performance nuances• Subtle nuances give a measure of how much a performer

can turn a piece her ownèMicD measures how much two performances of the

same piece can differ• All professional traditional instruments offer many

possible nuances and subtleties• For acoustic instruments MicD is clearly linked to the

richness and the fine control of sonic details --psychoacoustics and just noticeable differences (Rubine& McAvinney 1990)

• In digital instruments, also related to any kind of fine structural variation (time, tempo, number of voices, density of the voices and relations between them) that could occur within a given piece, while keeping it recognizable


The PhonographThe Phonographa a MicDMicD case studycase study

• Cage’s Imaginary Landscape No.1 (1939) includesvariable-speed phonograph turntables, playing test-tones

With a … phonograph it is now possible to control …any of these sounds and give to it rhythms within or beyond the reach of imagination. Given … four phonographs we can compose and perform a quartet for explosive motor, wind, heartbeat and landslide. (Cage 1937)

• Kool Herc, DJ Grand Wizard Theodore… late 1970s


MididiversityMididiversity ((MidDMidD) : Performance contrasts) : Performance contrasts

• How distinct two different pieces or two performances played with the same instrument can be?

• very low MidD instruments seem to be “always playing the same piece”

• New instruments for novices & interactive sound installations tend to have very low MidD

• Open or interactive compositionsA highly sophisticated ‘instrument’ with a low MidDmay be a very good interactive composition, but should not be considered as an instrument, even if it comes bundled with a hardware controller


MacrodiversityMacrodiversity ((MacDMacD) : Stylistic flexibility) : Stylistic flexibility

• The ability to be played in different contexts or music styles, or assuming varied musical roles

• The capabilities of integration with different instrumental partners

• Instruments with a high MacD could be considered all-purpose, while a low MacD denotes a more specialized and less versatile instrument

• In a traditional music context, MacD does not seem to be necessarily a quality. Generic instruments that do fairly well in completely different musical contexts will probably hardly excel in any of them

• In our context, new instruments with higher MacD may be more sexy & affordable è easier “proselytism”


•• MicroMicro--diversitydiversity (performance nuances)(performance nuances)èè turning musicians into virtuositurning musicians into virtuosi

•• MidMid--diversitydiversity (performance contrasts) (performance contrasts) èè turning gamers into musiciansturning gamers into musicians

•• MacroMacro--diversitydiversity (instrument flexibility, stylistic (instrument flexibility, stylistic diversity) diversity) èè catching candidatescatching candidates

Diversities summaryDiversities summary


Low MacD instruments have more chances to give birth to new music

stylesHigh MacD instruments have more chances for

starting being used

Elastic and translation properties of the instrument

hyperspace when submitted to external influences (other

instruments, other music styles, etc.)

‘Specialized’instruments (e.g.

double bass, piccolo)Roland TB303 bassline

‘All-purpose’instruments (e.g.

guitar)Laptop+mouse+

Max/Msp

Stilistic flexibility (how the instrument adapts to different

contexts)

Macrodiversity

Can turn music gamers into music performers

Shape and measurements of the instrument hyperspace

Jaws harpPercussion and

instruments with fixed pitch

Interactive sound installationsInteractive

compositions

Most traditional instruments.

Performances’contrasts (how

distinct two pieces played with the

same instrument can be)

Mid-diversity

Can turn music perfomers into virtuosi

Hypervolume around one hyperpoint (representing the

piece)

Player piano“John Cage turntables”

Most traditional instruments

“HipHopturntables”

Performance nuances (how two

performances of the same piece can

differ)

Microdiversity

Influence on performer

Spatial metaphorLowHighDescription

Diversities summaryDiversities summary


Time, musical sharpness & inertiaTime, musical sharpness & inertia

• a perceptual shorter level ranging somewhere around 2-10 msec (relates to technical constraints latency, resolution…)

tMicD∆

∆t

MidD∆

∆t

MacD∆

∆

100 msec (Askenfeld & Jansson 1990)delay on piano (piano)

30 msec (Askenfeld & Jansson 1990delay on piano (staccato-forte)

30-50 msec (Rasch 1979)playing “together”

20-30 (Posse Lago & Kon 2004)tolerable instrumental latency

20 msec (Hirsch 1959)separate 2 musical tones

10-20 msec (Wessel & Wright 2002)tolerable control granularity

10 msec (Gurevich et al. 2004)preferred delay between musicians

4 msec (Vorberg & Hamburg 1978)∆ in keeping steady pulse

2 mseccontrol granularity (100 samples window)

2 msec (Clynes 1984; Stewart 1987)distinguish variation in duration

2 msec (Licklinder 1951; Woodrow 1951)

separate 2 clicks

20 µsec1 sample (at 50 KHz)

20 µsec (Pierce 1999)binaural azimuth


Time, musical sharpness & inertiaTime, musical sharpness & inertia

• Reacting to music (50 msec-500 msec)Sternberg et al. 1978; Lennie 1981; Pressing 1988…

• The more “intelligent”, the more inertial music systems tend to be, but maximum contrasts should be in the 250-1000 msec range

• Musical sharpness (e.g. Carl Stalling or John Zorn vs. laptop music)

tMicD∆

∆t

MidD∆

∆t

MacD∆

∆


Multithreaded instruments & diversityMultithreaded instruments & diversity• Direct access to all final parameters will favor MicD• But accuracy should not happen at the expenses of leaving

unattended the macrocontrol processes

• MidD & MacD in digital instruments are often related to the concept of presets


DiversityDiversity, , improvisationimprovisation, , virtuosityvirtuosity ……

• Configurations & presets vs. free improvisation and continuum performance space (any instrument must permit improvisation!)

• MacD vs. instrument idiomatism• Non-linearity, control, reproducibility and virtuosity (the

goldsmith vs. the torero)• Non-linearity should not interfere exploration,

learnability (confidence is essential for starting exploration)

• Technical motor virtuosity vs. Music sensibility virtuosity (e.g. Monk, Coleman, Ayler)

• Lack of repertoire, non-linearity, exploration in new digital instruments favor 2nd approach


ExpressivenessExpressiveness• Expressive Computer Music [rendering] ?§ Expressiveness as (imitation of ) deviation (e.g.

Sundberg)

We regard expression as a rhetorically shaped transfer of structural score contents by means of the ‘‘deformation’’ mapping of symbolic data into a physical parameter space (Müller & Mazzola 2003)

• Expressive instruments? è fine, grained, accurate control

• Eurocentric view è expressiveness ~ MicD


ExpressivenessExpressiveness• Expressive Computer Music [rendering] ?§ Expressiveness as (imitation of ) deviation (e.g.

Sundberg)

We regard expression as a rhetorically shaped transfer of structural score contents by means of the ‘‘deformation’’ mapping of symbolic data into a physical parameter space (Müller & Mazzola 2003)

• Expressiveness as effort (Ryan 1991)• Expressiveness as a shared idiom (Fels et. al 2002)• Expressiveness as flexibility (Arfib 2005) (~ MacD)


Instruments, as machines, can be more or less efficient. They cannot be expressive though, machines do not have anything to express. But they can be used to transmit human expressiveness. Our instruments will achieve this for better or worse, in the measure they permit the performer to transform a mental musical representation into musical gesture and sound. Efficiency and expressiveness do not always go hand by hand. Take the pencil, the pen and the word processor. The latter is surely more efficient, but I personally find the fountain pen’s smooth and continuous ink flow over the paper more inspiring and better suited for many expressive purposes.Inspiration. Music instruments are not only in charge of transmitting human expressiveness like passive channels. They are, with theirfeedback, responsible for provoking and instigating on the performer new ideas or feelings to express.


First conclusionsFirst conclusions


1. New musics tend to be the result of new techniques, which can be both compositional or instrumental.

2. New instruments will be able to survive and succeed in the measure they are truly innovative; i.e. they have something new to bring to music, and not merely because they are based on novel, previously unavailable technologies.

3. It is therefore crucial to identify the quintessence of new digital instruments; what they can bring that is essentially original to the act of music performance; how can they redefine it.

4. Moreover, new instruments will manage to be better if luthiers do not over look the history of traditional instruments.

5. Learning from the past does not merely mean emulating it. It also means to try avoiding or improving upon passed errors and limitations (the past was not perfect).


6. Efficiency can be clearly improved is that of. Traditional instruments take years to master, whereas, given the speed at which technology and fashion shift, new instruments have to hook from the first minute.

7. New instruments whose output is based on traditional paradigms, such as one-to-one control of pitch+timbre+amplitude, have a hard time competing with existing traditional instruments (can only contend on the grounds of efficiency).

8. These new ‘traditionally-modeled’ instruments with increased efficiency may appeal to non-musicians and to dilettante, but less to advanced musicians as well, who will look for new-fangled possibilities.

9. New instruments have to profit from computers’ intelligence, with their ability to remember, to make decisions on the fly, to run multiple, concurrent processes in parallel.

10. In such cases, the performer will need to share and delegate part of the control of the music being created to the instrument; such as the traditional composer delegates on the performers.


11. By running processes at different temporal and formal scales and different levels of complexity, new instruments naturally surpass the one-action to one-event model inescapable in all traditional instruments. New instruments are not only sound producers, they become music producers too.

12. Performing with such powerful instruments should not mean leaving all the musical decisions to the instrument. Playing music is a ‘serious’ activity, different from ‘playing with music’ (also very important but not studied here).

13. An instrument of these characteristics will probably involve many internal dimensions or control parameters. Otherwise it will always seem to play ‘the same composition’.

14. The feeling of power and macrocontrol these instruments can bring to their performers, should not let us forget about the expressive nuances and the time precision mechanisms, which traditional instruments have always incorporated.

15. The performer must thus be able to affect all these dimensions: both the metacontrol of ongoing processes and the microcontrol of final parameters should be permitted. All of them with the maximum simplicity, flexibility and speed.


PartPart II : II : CriteriaCriteria & & PurposesPurposes

• Describes (in 3 chaps.) 5 implementations I’ve developed between 1989-2001

• They illustrate different paradigms exposed in [the first chapters of] Part I

• They are the origin of the ideas exposed in [in the last chapter of] Part I

• If digital lutherie is not a science, experience & maturity are important for validating empirical & intuitive mechanism è my “credentials”


PITEL machine listening & improvisation (chap 8)

The QWERTYCaster, JoAn, Epizoo, Afasiascore-based systems & multimodality (chap 9)

FMOL (chap 10)


PITEL machine listening & improvisation (1989-91)

Jordà, S. (1991). A Real-Time MIDI Composer and Interactive Improviser by Means of Feedback Systems. Proceedings of the 1991 International Computer Music Conference, 463–466.


Based on two-term non-linear feedback relations:

xa,i = f (xa,i-N, xb,i-k)


PitelPitel’’ss drawbacksdrawbacks

• Lack of serious interface design• Human performers didn’t share the machine’s logic

• Only macro/meta control / not direct access• Diversity: performances of the “same” piece as diff. as

diff. pieces! (MicD & MidD)• All non-linear processes at symbolic (not sonic) level

è big inertia è more “conductor” than “player”paradigm è driving a truck


Other topics covered in this chapterOther topics covered in this chapter……

• Machine listening• Musical knowledge• Markov chains• 1/f noise, non-linearity & feedback• Generative vs. subtractive (corrective) algorithmic music


The The QWERTYCasterQWERTYCaster, , JoAnJoAn, , EpizooEpizoo, , AfasiaAfasia

• QWERTYCaster Simplicity (the “moped”)• JoAn Expresiveness• Epizoo Interaction illusion• Afasia Score-based systems


JoAnJoAn, , ll’’homehome de de carncarnJordà & Antúnez 1992

Mimicking human expressivity has nothing to do with being

expressive !


EpizooEpizooAntúnez. Jordà & Olbeter 1994

The fallacies of interaction:

• Interaction does not make you more creative or more free!

• The illusion of control


• One-man interactive multimedia (& wordless) performance based on Homer’s Odyssey

• 3rd collaboration between M.Antúnez (visual artist & performer), Roland Olbeter (mechanical sculptor) and S.Jordà (music, interactivity and software design)

• Premiered in Barcelona 1998, presented in many European and American countries, and several international awards

• A robot quartet, an exoskeleton fitted with sensors, interactive visuals, films & music all controlled by the performer

Jordà, S. (2002). Afasia: the Ultimate Homeric One-man-multimedia-band. Proceedings of the 2002 Conference on New Interfaces for Musical Expression (NIME-02), 132-137. Dublin.

AfasiaAfasia ((AntAntúúneznez, , JordJordàà, , OlbeterOlbeter 1998)1998)


Computer Computer

Effects racks

MIDI controlled

Audio Mixer

MIDI

MIDI

OUT

Internal PC

CD Audio

Internal PC

Sampler AudioTo parallel D/A cards

Rack mounted

D/A parallel

cards and relays

Parallel

communication

Switches to

electro-valves and

step motors

Serial

RS-232

selects

projector

input

Screen fills the whole backstage

Air Compressor

Compressed air to

electro-valves

Violin Guitar Bagpipes

S-Video

from

internal

DVD

SVGA with

2D

animations

486 with

custom card

for decoding

wireless

transmission

Pentium II

with CD

Audio,

internal

sampler,

DVD and

MIDI OUT,

running

Afasia

Software

Projector

Drum

Performer with

exoskeleton

sends wireless

data to 486

4 Turn

potenciometers

continous

controllers

4 Mercury

tilt sensors

“Keys”

Video

Input

switcher

PedalsAux


AfasiaAfasia Musical interactivityMusical interactivity

• Limited semantics of sensors employed è score-driven interactive model, which can in a continuum go from 100% deterministic sequences to 100% generative

• Each of Homer’s islands is a diff. state with its own file, reflecting particular behavior and mapping

• Format 1 Standard MIDI Files expanded with 40 text meta-events: tell the custom sequencer how to process the data according to the performers inputs

e.g. TRANSPOSE=4,3,12$


Some possibilitiesSome possibilities• Switch between blocks of tracks• Mute/unmute tracks• Transpose tracks• Loop sequences - modify their

length• Modify current play position• Modify any MIDI control (by

value, by increment, by random range…)

• Quantize/delay tracks• Change tempo• Define chords or scales (for

correcting generated notes)• Gestures and solo modes …

• Open form compositions (J. Cage, Earle Brown, Boulez)

• Expands the possibilities of some early 80s MIDI sequencers (e.g. KCS, MusicX)


Drawbacks Drawbacks –– Multimodality: too much is not enoughMultimodality: too much is not enough

• The temptation given by digital technologies to control ‘everything’, including lightning, image, machinery, etc. cohabits with the danger of ending up with a sort of theme park ‘one-man-band’ (Puckette1993; Tanaka 2000…)

• Within all that digital power in his hands and below his knees, Antúnez can do little to convey his expression. He may feel like Homer, Ulysses and Zeus all-in-one; he surely can express his feelings and ideas as a director, but he is too busy as a performer, he has too many things to think about.

• Afasia is not indeed a music instrument.Besides, it cannot be played without compositions (albeit open ones)


FMOL (FMOL (F@ustF@ust Music On Line) 1997Music On Line) 1997--20032003• Jordà, S. & Aguilar, T. (1998). A graphical and net oriented approach to

interactive sonic composition and real-time synthesis for low cost computer systems.” Proceedings of COST G6 Conference on Digital Audio Effects, 207-210, Institut Universitari de l’Audiovisual, Barcelona.

• Jordà, S. (1999). Faust Music On Line (FMOL): An approach to Real-time Collective Composition on the Internet. Leonardo Music Journal 9.

• Jordà, S. (2002). FMOL: Toward User-Friendly, Sophisticated New Musical Instruments. Computer Music Journal, 26(3), 23-39.

• Jordà, S. (2003). Sonigraphical Instruments: From FMOL to the reacTable*. Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03), 70-76. Montreal.

• Jordà, S. (2003). Interactive Music Systems For Everyone: Exploring Visual Feedback as a Way for Creating More Intuitive, Efficient and Learnable Instruments. Proceedings of Stockholm Music Acoustics Conference (SMAC03) 681-684. Stockholm.


FMOL Decalogue (1997)FMOL Decalogue (1997)Project for la Fura dels Baus, F@ust 3.0 show (based on Goethe’s)

“Given the important role symbolized by the Internet in our play , we would like part of its music to be composed by cybercomposersaround the world”

• Collaboration Collective composition on the Net• Aesthetics “Sound over notes” priority (NO General MIDI,

no controllable pitches, no obvious rhythms…)• Proselytism Introduce newcomers into “experimental” electronic

music• Scalability Attractive to both trained and non-trained electronic

musicians, intuitive & complex, learnable• Affordability Cheap (free) and available (no special hardware)

èMouse driven client software for RT synthesis and RT composition with a peculiar “visual feedback” interface


FMOLFMOL’’ss Interface main characteristicsInterface main characteristics

• The interface works at a very low level• Developed in parallel to the sound engine• All functionalities are visible (D.Norman’s affordance?)• Audiovisual feedback, visuals work both as input & as

output (abacus)• No indirect mappings between what you see & what you

hear• As music needs time, visuals need animations

11/04/2007 Digital Lutherie - Sergi Jordà 99Bamboo screenshot


• 6 audio channels • For each channel: 1 generator & 3

processors (selectable from more than 100 algorithms)

• For each generator-processor: 4 LFOs

• For each LFO, dynamic control on frequency, amplitude & shape (sin, square, saw, triangle, random)

• LFOs are fundamental for all time evolution (no use of prerecorded sequences)

• Strings are plucked and fretted• They draw the sound they generate• Horizontal lines oscillate according

to their LFOs


FMOL OutputFMOL Output

• More than 1,100 Internet pieces by more than 100 authors (Jan-April, 1998), 50 used in the play F@ust 3.0 by La Fura dels Baus

• Collective CD published in 1998• More than 600 pieces (September 2000), used in the opera DQ by

la Fura dels Baus• Several workshops for musicians & non-musicians (visual artists,

kids, teenagers, “pro” improvisers… in Lisbon, Berlin, Dresden, Barcelona, Galicia…)

• FMOL Trio (1999-2003): The only instrument I designed for “everybody” is now the only one I personally use live -Improvised electronic music + visuals (3 live CDs released, 1 long-feature film soundtrack…)

• Continues to be my main music instrument


Reasons for FMOL Reasons for FMOL ““successsuccess”” ??Is rich & “interesting”• Non-linearity, feedback and ‘complexity’ (at the audio level…)• Allows to play sound & form simultaneously• Allows development of both mental & physical virtuosity• ‘Sharpness’• Passes the 3 Diversity tests

Is efficient• Visual feedback• No symbolic interface indirection• “N-dimensional” interface (graphic & mental model reflects the

engine structure) è not thin & not merely wide (hold, skip, through, random access…) (150 parameters at a time)


RecouplingRecoupling & Visual feedback& Visual feedback• By this seamless integration of output and input, of

representation and control, already present in the abacus, FMOL manages to recouple the typically decoupled new digital instruments’ two main components: the controller and the generator.

• The ‘audio’ feedback presented to the performer in an interactive visual form, intuitively helps the understanding and the mastery of the interface, enabling the simultaneous control of a high number of parameters that could not be possible without this visual feedback.

• In multithreaded instruments, the performer is permanently delegating and shifting control to the instrument, and thus needs all the affordable ways for monitoring the instrument’s processes activities.


PART III PART III CONCLUSIONS &CONCLUSIONS &FUTURE WORKFUTURE WORK


15. The performer must thus be able to affect all these dimensions: both the metacontrol of ongoing processes and the microcontrol of final parameters should be permitted. All of them with the maximum simplicity, flexibility and speed.

16. The many internal dimensions of the instrument, should not be controlled by four or five input dimensions, leaving all the dirty work to some clever mappings.

17. To allow this type of control, new instruments have to be ‘wider’ rather than ‘deeper’. ‘Wide’ instruments permit a better direct access to all of their complexity.

18. Since not all the parameters can be controlled simultaneously, these instruments need to integrate intuitive and efficient multiple sample/hold/return/bypass options.


19. These mechanisms favor a type of selective or asynchronous playing (not all the parameters are permanently modified; not all the processes are equally active).

20. In a ‘wide and concurrent instrument’, multi-user possibilities, though not a must, can represent an appealing added value. A multi-user approach can enrich the instrument’s bandwidth without provoking a single-user overload.

21. Wide instruments demand wide controllers or interfaces.

22. Moreover, sophisticated and complex concurrent instruments will benefit from interfaces that reflect and help to understand the instrument’s ongoing internal mechanisms.


23. The potential of computer graphics for representing and monitoring complex processes, is not easily surmounted. This is probably the reason why many of the more interesting recent instruments, are screen based. While, the mouse is a very limited controller that we should definitely try to avoid.

24. Screen based visual feedback can still be used in combination with physical controllers, but in these cases an undesirable indirection component is added (the performer does not play IN the screen).

25. For including realtime interactive visualizations and, at the same time, overcoming mouse limitations without adding indirections, interfaces should be able to reflect their own states and behaviors. They should integrate, like the abacus, both representation and control.


Being developed by the Interactive Sonic Systems Team at MTG (UPF)

• Sergi Jordà (conception & direction)• Günter Geiger (sound synthesis & control)• Martin Kaltenbrunner (computer vision & net logic)• Marcos Alonso (visual synthesis)• Hugo Solis (computer vision & hardware)

With the collaboration of • Ross Bencina (computer vision)• Ignasi Casasnovas (visual synthesis)

Postlude: The Postlude: The reacTablereacTable**


Thank you !Thank you !

Sergi Jordà

Music Technology GroupInstitut Universitari de l’AudiovisualUniversitat Pompeu Fabra, Barcelona

[email protected]://www.iua.upf.es/~sergi

digital lutherie - tecn.upf.essjorda/gmv2004/tesis2005def.pdf · 11/04/2007 digital lutherie -...

Documents