NEXUS: Collaborative Performance for the Masses,Handling Instrument Interface Distribution through the

Web

Jesse AllisonLouisiana State University

216 Johnston HallBaton Rouge, Louisiana

70803jtallison@lsu.edu

Yemin OhLouisiana State University

216 Johnston HallBaton Rouge, Louisiana

70803yoh1@lsu.edu

Benjamin TaylorLouisiana State University

216 Johnston HallBaton Rouge, Louisiana

70803btayl61@lsu.edu

ABSTRACTDistributed performance systems present many challengesto the artist in managing performance information, distri-bution and coordination of interface to many users, andcross platform support to provide a reasonable level of in-teraction to the widest possible user base.

Now that many features of HTML 5 are implemented,powerful browser based interfaces can be utilized for distri-bution across a variety of static and mobile devices. Theauthor proposes leveraging the power of a web applicationto handle distribution of user interfaces and passing interac-tions via OSC to and from realtime audio/video processingsoftware. Interfaces developed in this fashion can reach po-tential performers by distributing a unique user interface toany device with a browser anywhere in the world.

KeywordsNIME, distributed performance systems, Ruby on Rails,collaborative performance, distributed instruments, distributedinterface, HTML5, browser based interface

1. INTRODUCTIONWith the proliferation of mobile devices, avenues for dis-tributing a live performance system, or instrument, for col-laborative performance are growing. Distributed perfor-mance systems have undergone many leaps forward with theincreased speed of networks, proliferation of smart phones,and the emergence of ensembles such as laptop orchestrasthat are ideal for exploring these kinds of interaction. Asmobile devices continue to become more powerful and ubiq-uitous, integrating them into performance systems is in-creasingly desired[8]. Along with these explorations in col-laborative performance come some significant distributionand data management challenges. A number of approachesfor distributing a performance system, the challenges in im-plementing them, and a testable solution will be examined.Many of the approaches have been put into practice in theauthor’s collection of laptop ensemble and audience piecesentitled Perception[3] and will be noted when appropriate.

2. DISTRIBUTED PERFORMANCE SYSTEMS

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME’13, May 27 – 30, 2013, KAIST, Daejeon, Korea.Copyright remains with the author(s).

Distribution of an instrument for collaborative performancerequires breaking down the system into parts and decidingwhat to distribute and how these separated parts shouldcommunicate[6]. Typical parts of such a composed instru-ment[10] system are the user interface such as buttons,sliders, and accelerometers, the mapping structure defininghow the controls are interpreted by the instrument to createsound, the communication system defining how instrumentparts pass and receive control messages, and the audio pro-duction engine or audio graph actually producing the wave-forms. With these components of performance systems inmind a number of approaches to distributed instrumentsbecome apparent.

Audience

Figure 1: Autonomous Performance System.

2.1 Autonomous Collaborative PerformanceSystem

Perhaps the simplest approach is to have the instrument beself-contained, installed on each performer’s device, and op-erating individually. Each user has an interface, mappingstructure, and audio production engine installed on theirdevice. This approach is taken by many pieces for laptop

�1

orchestra1, traditional mobile music applications such asThumbJam2 and Ocarina3, as well as the approach by suchtoolkits and libraries as libPd[4], MoMU Toolkit[5], and uR-Mus[7]. Although such a system may communicate witheach other or with a central server/coordinator to increasethe collaboration in performance, at its core, each couldoperate and make sounds autonomously. The collaborativenature in this type of system would be analogous to a bandwhere each individual has a sonic contribution to the soundof the ensemble. Admittedly the delineation between en-semble and collaborative meta-instrument is debatable, butthe idea of a group of collaborating performers combining toform a meta-instrument is a useful analogy, and as softwarecan augment communication and influence between indi-vidual performers, the meta-instrument metaphor becomeseven more significant (Figure 1).

2.2 Centralized Audio Production, DistributedInterface

A second approach is through seperation of the interfacefor an instrument and the sound production engine of theinstrument itself. This approach is taken by such softwareas TouchOSC4, Squidy Interaction Library5, and OSCula-tor6, among many others, to pass control parameters fromgraphic user interfaces and sensors to a central audio en-gine for sound production. This separation requires a gooddeal more structure and coordination in the software de-sign which by necessity makes the interface more rigid oncedeployed. Computationally, it is both restrictive and free-ing to have a centralized audio engine. The audio engine isno longer restricted by the computational power of an in-dividual mobile device, but cannot leverage the power andscalability of many individual rendering engines. Anotherdifference is the localization of the final produced sound.The sound can no longer emanate from the point of interac-tion except through the use of streaming audio back to theinterface which has inherent latency and bandwidth issues.Also, instead of relying on the small and typically impotentsound system on mobile devices, the sound can be producedover a more adequate sound system.

A consequent approach that is conceptually distinct, yetstill derives from audio engine and interface separation, isthe division of the interface into parts, with distributionof those smaller parts of the overall instrument to manydevices/performers. In this scenario, performers contributeto the state of a central audio engine and collaborativelydefine what sonic events are produced(Figure 2).

2.3 Responsive Server, Adaptive InterfaceFinally, a responsive system could be created where the cen-tral server handling distribution of the interface would alsoreceive, process, and send messages and data back to theinterfaces. These commands could be displayed directly tothe performer, setting parameters on the interface, or com-pletely updating the interface. In Perception[Divergence] [3],a system was built where images can be drawn on a de-vice and submitted back to the server. Once received, theserver passes them out to another user for further annota-tion, creating a collaborative image interface. All sorts ofcoordinated collaborative experiences could be envisioned.

1examples: Plork http://plork.cs.princeton.edu/and the Laptop Orchestra of Louisianahttp://laptoporchestrala.wordpress.com/2http://www.thumbjam.com/3http://ocarina.smule.com/4http://hexler.net/software/touchosc5http://merkur61.inf.uni-konstanz.de:8080/squidy/6http://www.osculator.net/

Audience

Interface Server

Centralized Audio Production Engine

(MaxMSP)

AggregateControl Changes

Control Changes

Figure 2: Distributed interface with centralized au-dio production.

�2

2.4 Possibility of Other ApproachesNote that there are many degrees of variation between eachof these types of systems. One could envision distributed en-vironments that span the range of cross-performer influence,instrument segmentation and distribution, local versus cen-tralized sound production, and bi-directional influence onthe user interface itself. The main similarity is a need fordistributing an interface, coordinating communications be-tween collaborators, and coordinating control changes foraudio production.

2.5 Challenges to DistributionThere are many varieties of operating systems, developmentkits, frameworks, and proprietary code making the incorpo-ration of more than one type or generation of device difficultto do except under very narrow use cases. Although thereare cross-platform development efforts such as OpenFrame-works7, they are not yet fully realized and are constantlydealing with upgrades and incompatabilities across manydevices. The current state of the art requires a platformdependent application to be built for each device to be in-corporated in the system. This can be very daunting andin many cases down right impossible due to budgetary andtime constraints. The simple idea of distributing an instru-ment to people with mobile phones may require supportingiOS, Android, Blackberry OS, Symbian, PalmOs, WindowsPhone, and any other system who may want to participate.

Once the application is built, managing communicationsbetween more than just a few devices becomes remarkablycomplex. Beyond the significant task of defining the inter-action, a suitable communications protocol must be chosenalong with a framework to support it on each platform.Then, a system must be coded for passing, receiving, andhandling the interactions. Master/slave pairings must benegotiated, not to mention the difficulties of trying to ag-gregate many sources of information to create a final pa-rameter. The complexity of such a system only increases asmore devices are added to the system, or even worse, addedand taken away dynamically.

2.6 Enter the WebIn each of the distributed performance systems describedabove, two significant problems arise: cross-platform appli-cation development and coordinating communication andinteraction.

Pushing the user interface of a distributed performancesystem into the browser helps to alleviate both of theseissues. Due in large part to the ubiquitous usage of theinternet, much effort has been poured into creating stan-dards for display and interaction with web pages8. Al-though browsers are notorious for behaving somewhat dif-ferently, overall they can be made to look and functionsimilarly across a wide variety of platforms both desktopand mobile. Compared to other cross platform initiatives,browser functionality is quite well developed. Once thechoice to adopt a browser based UI is made, the artist candevelop a single interface that can be utilized by most webenabled devices.

An additional concern with browser based performance isthat of perceived latency. Interactivity in the browser has afew bottlenecks - notably, javascript as the coding language,http requests and AJAX as the communications protocol,and the lack of availability of sensor data within browsers.Fortunately, browser implementations of javascript in HTML5

7http://www.openframeworks.cc/8W3C’s latest specification of HTML5:http://dev.w3.org/html5/spec/Overview.html

are making marked speed improvements to the point thatmost types of interactivity can be made to feel responsive.AJAX is a bit slower than the ubiquitous UDP used in mostdistributed user interface software, however with the rise ofweb sockets as well as browsers embedded in applications,communications can be made quite snappy, or simply madethrough UDP instead. Finally, browsers are gaining moreand more functionality including access to accelerometers,multi-touch data, camera, audio input, and location infor-mation; finally approaching the performance of their nativecounterparts. In Divergence (Figure 3)the browser was usedin an iPad to send continuous accelerometer and touch datato a central server to control the sampling and pitch shiftingof a marimba.[2]

Figure 3: Browser based accelerometer and touchperformance interface.

3. BROWSER BASED USER INTERFACEMoving the display and interaction of a user interface intoa browser allows for cross-platform distribution. Dynamicwebpage distribution is typically handled through a serverside web application. There are a variety of approaches andsolutions to web communication, state management, andcontrol logic, which have tradeoffs in complexity, ease of use,ease of support and deployment, and compliance to evolv-ing best practices. Various options include everything fromjavascript, Perl, and php, to full frameworks like Rails and.NET. The authors have created a php based distributedUI, Node.js, and full web app implementations using Rubyon Rails.

Ruby on Rails is quickly adaptable to a wide variety ofscenarios that allow for interesting performance paradigms.The MVC architecture can be easily adapted to cover in-strument interface distribution, parameter management, andcontrol logic. As the Rails community focus on Agile de-velopment supports the ability to quickly try out variousimplementations and Rails tends to be painfully near or atthe cutting edge of web best practices, we chose to do muchof our prototypical development utilizing the framework andwill discuss further implementation details in that light.

3.1 Rails: a useful solutionRuby on Rails is a framework for developing web applica-tions that handles much of the distribution and informa-tion management issues through well-trodden conventionsas opposed to configuration. The end result is a very rapiddevelopment process that is easily adaptable to the devel-oper’s specific goals. It handles many of the normal issuesof scalability, testing, database configuration and manage-ment, session management, and customized user interface

�3

generation. When applied to the distribution of user in-terface objects and coordinating their incoming responses,standard Rails techniques have performed admirably.

3.2 MVC Architecture Adapted for PerformanceInstrument

Rails uses a traditional MVC software architecture, adoptedfor handling web applications. The process for serving tra-ditional web pages goes something like this (Figure 4):

routes.rb

Request

Controller

Active Record

Database

Method Call

CRUD

Active Viewrender .erb.html

files

Data Variables

Figure 4: Overview of the Rails process.

An incoming page request is handled by the route.rb filespecifying what controller method to call. The controllermethod then handles the gathering and processing of anyinformation needed for the requested page. This step typ-ically involves polling the Model (database) to retrieve in-formation like blog posts, calendar events, and user infor-mation for display. Once the controller has prepared theneeded information, it is passed to the appropriate view filesfor display. The view files are usually written with embed-ded ruby (ex. index.erb.html) which handles the iterationthrough arrays and hashes of information, the generationof html tags, image names, user control options, insertingpartials and the like. These ruby commands are processedto produce the final markup which is passed back to therequesting browser.

Of course this can become much more complicated withlogic occuring in the view in the form of helpers, AJAXmethods, end user specific code, etc., but for the most part,this series of events is the standard approach. In usage as aperformance instrument environment, the model/view/controlleractivities are divided in the following ways.

The Model in our application handles any data that wewant to store and recall. In the simplest use cases no in-formation would be stored, instead passing interactions di-rectly along to the audio engine. The model could be usedto store user specific information such as which parametersare actively assigned to which user, routings, musical sec-

tions, etc. In a complex scenario, it could store user definedsequences, images, preferences, snippets of audio, aggregateparameters, just about anything.

The Controller has the assignment of divvying up theuser interface parameters and coordinating the incomingresponses. Upon first request from a potential participantthis would include selecting parameters and controls to beassigned to the user, storing any user specific informationthat may be used later, and calling the appropriate viewfiles to be rendered. Further contact from the page wouldbe from UI changes. The controller would then either putthem into the database or simply pass them along to theaudio production engine as an OSC message (Figure 5). Thecontroller would also handle sending each user any updatesto the UI that may occur: waveform changes, color events,complete UI replacement, etc.

Rails Controller• Keep track of user's session id• Assign UI objects and UI-ID's for each view• Pass UI movements to Audio engine

1. UI Requests

2. UIReturned

3. AJAX UIMovementsAction View

. . . . . . n

Figure 5: Rails UI Distribution.

Finally, the View receives information from the controllerand uses that to create user specific interfaces. These cantake the form of traditional buttons, drop down menues, andtext boxes, but can also incorporate sophisticated graphi-cal UI objects through javascript. The widespread adop-tion of the HTML5 canvas tag allows for dynamic draw-ing and interaction with a graphical element. The NEXUSopen source project [1] contains a set of nexusUI javascriptobjects demonstrating many dynamic functions that canbe done within the canvas object including: timer basedautomation, inter-canvas communications, multi-touch, ac-celerometer input, and AJAX callbacks.

3.3 Web App as Parameter Pass-ThroughThe web application can be used to simply split up theinterface and pass any returning commands directly to theaudio rendering engine. In this simple scenario, after theinterface has been distributed to the user, any returningposts are simply routed on to the audio engine through rosc.Individual users can be identified via session informationwith each request, or by customizing the submit name whenthe original html interface file is generated (Figure 6).

Here you can see the id of the slider inserted into the idattribute of the canvas tag.

�4

<canvas id="slider_<%=@slider_id%>" > [Slider] </can-

vas>

This will generate code like:<canvas id="slider_1" > [Slider] </canvas>

The id can then be used as an osc name when passed onto the audio rendering engine.

routes.rb

Request forInterface

Controller

InterfaceRequest

Active Viewrender individual

interface

Variables

AJAXCommands

MaxMSP, etc.OSC

AJAX

HTML View

Figure 6: Rails distributed interface and interactionpass-through.

3.4 Web App as Instrument Distribution Cen-ter

To divide and distribute an instrument, Rails can handlethis in two ways. For simple matters, it can dynamicallyassign a different parameter value for each slider, button,or UI element that is dispensed, for example: slider_1,

slider_2. This works well if there are a finite number ofUI elements in the instrument and you know how manypeople are playing it at any one time.

A more sophisticated approach would involve the use ofsessions. Every browser request has a session id that can beused to identify who is making the request. With this ap-proach, information can be stored through the model abouteach users interactions, then as an interaction is receivedthe controller can use this collected knowledge to respond.For instance, the model can store the fact that user xyz isassigned sliders 5 and 7. Then when a slider change is re-ceived it can pull up the slider assignment and pass alongthe correct changes.

In Relativity, (Figure 7) a movement of Perception, a vari-able number of audience members is provided a radial se-quencing interface where a pulse emanates from the centerand any points that they touch in its path triggers musicalevents.[2] The web application distributes a unique inter-face to each audience-performer. It then receives the inter-actions via AJAX and sends OSC to the audio productionengine for rendering the collaborative musical experience.The communication is also bi-directional allowing for theserver to send messages to each user to set the pace of thepulse collectively for the group. At the premiere in Marchof 2012, more than 100 unique devices were connected atone point during the performance.

3.5 Database as State EngineAnother approach is to use the Database for state manage-ment for the instrument. Every change in user interfaceobjects can be stored in the database as well as setting theaudio engine. The database could store many users’ inputand aggregate it to come to a collaborative parameter set-

Figure 7: Radial sequencing interface distributed toaudience’ mobile devices.

ting. In these scenarios, the database state could also becopied and stored as a preset and complicated states couldbe quickly retrieved or even interpolated.

4. FUTURE DIRECTIONSThe viability of these methods have been established throughperformances, interactive kiosks, and installations, and manynew avenues of exploration present themselves. On the webapplication front, various use cases could include: smallensemble with direct realtime responses, broad distribu-tion utilizing audio streaming, location awareness, a con-trol parameter database for aggregating interactions, andpicture/drawing based interactions.

A library of javascript user interface objects, nexusUI, hasbeen developed to facilitate performance actions and easeof incorporation into Rails applications, other web technolo-gies like Node.js, other programming platforms implement-ing javascript canvas drawing, embedded browsers, and WebAudio.

Embedding browsers into native applications and usingthem for user interface has proved to be a fruitful field. Itallows us to use the nexusUI javascript library to quicklycreate native user interfaces which can then engage extrasensors, audio/video processing, or UDP communicationsfor added responsivity.

Another interesting development concerns the growth ofthe Jamoma Audio Graph9. In a recent incarnation, crosscompilation was added allowing one to create an audio graphin Max and export it as a pd patch, c++ code, or ruby[9]. Ina series of experiments we were able to get a Ruby JamomaAudio Graph running inside a rails application. This wouldallow for a web service to be deployed which would handleall of the audio production as well as user interaction.

Further testing and benchmarking are being carried outto see what kinds of performance interactions will be viablefor various scales of participants.

9Jamoma Foundation, http://jamoma.org

�5

5. CONCLUSIONSWeb applications such as those built on Ruby on Rails haveproven to be a viable framework for handling the servingof web pages containing user interface objects to many po-tential performers. The cross-platform nature of currentbrowsers solves many issues dealing with dissemination andsupport. The ability to coordinate dynamically assigneduser interface objects is very useful. A built in databasegives flexibility in the types of interactions that can be ac-quired and used. Finally, a well trodden path for scalabilityis available so that the accomodation of many simultaneoususers is possible.

Now that many features of HTML 5 are being incorpo-rated among the leading browsers, powerful browser basedinterfaces can be utilized for distribution across a variety ofstatic and mobile devices - making worldwide collaborativecreative arts a distinct possibility. Interfaces developed inthis fashion can reach potential performers by distributinga unique user interface to any device with a browser any-where in the world. In overcoming platform dependencyissues, the emerging viability of browser based interface hasbecome an exciting frontier.

6. ACKNOWLEDGMENTSMany thanks to the CCT and LSU AVATAR Initiative, theLSU School of Music, as well as Timothy Place and theJamoma Foundation for their work on the Jamoma AudioGraph and Ruby.

7. REFERENCES[1] J. Allison. Nexus,

https://github.com/jesseallison/nexus.

[2] J. Allison. Divergence for marimba and mobileensemble, http://perceptionevent.com, March 2012.

[3] J. Allison. Perception, http://perceptionevent.com,March 2012.

[4] P. Brinkmann, P. Kirn, R. Lawler, C. McCormick,M. Roth, and H.-C. Steiner. Embedding pure datawith libpd. In Proc Pure Data Convention 2011, 2011.

[5] N. J. Bryan, J. Herrera, J. Oh, and G. Wang. Momu:A mobile music toolkit. In Proceedings of theInternational Conference on New Interfaces forMusical Expression (NIME), Sydney, Australia, 2010.

[6] P. L. Burk. Jammin’ on the web - a new client/serverarchitecture for multi-user musical performance. InICMC 2000 Conference Proceedings. InternationalComputer Music Conference, 2000.

[7] G. Essl and A. Muller. Designing mobile musicalinstruments and environments with urmus. In NewInterfaces for Musical Expression, pages 76–81, 2010.

[8] L. Gaye, L. E. Holmquist, F. Behrendt, andA. Tanaka. Mobile music technology: report on anemerging community. In Proceedings of the 2006conference on New interfaces for musical expression,NIME ’06, pages 22–25, Paris, France, 2006. IRCAM -Centre Pompidou.

[9] T. Place, T. Lossius, and N. Peters. The jamomaaudio graph layer. In Proc. of the 13th Int.Conference on Digital Audio Effects (DAFx10), pages78–86, September 2010.

[10] N. Schnell and M. Battier. Introducing composedinstruments, technical and musicological implications.In Proceedings of the 2002 conference on Newinterfaces for musical expression, NIME ’02, pages1–5, Singapore, 2002. National University ofSingapore.

�6