endogenous biologically-inspired visualization immersed ... · and science (kriss), korea (aug...

Endogenous Biologically-Inspired VisualizationImmersed within an Art of Complex Systems

Haru (Hyunkyung) Ji∗Art and Technology,

Sogang University, Republic of Korea

Graham Wakefield†

Arts Media Performance and Design,York University, Canada

Figure 1: Selected installations in the Artificial Nature series. Left: Time of Doubles (curved screen, projectors, loudspeakers, RGBD cameras).Photographed at the type:wall exhibition, Seoul Olympic Museum of Art (SOMA), Korea (March 31–May 29 2011). Center: Endless Current(head-mounted display, loudspeakers, RGBD cameras). Photographed at the Artience Project exhibition, Korea Research Institute of Standardsand Science (KRISS), Korea (Aug 26–Sep 4 2014). Right: Archipelago (kinetic sand, carved styrofoam, projectors, loudspeakers, RGBDcameras). Photographed at the Capitaine Futur exhibition, La Gaıte Lyrique, Paris, France (Oct 18 2014–Feb 8 2015).

ABSTRACT

We document techniques and insights gained through the creationof interactive visualizations of biologically-inspired complex sys-tems that have been exhibited as mixed-reality art installationssince 2007. A binding theme is the importance of endogenousaccounts: that all perceivable forms have dynamic ontological ca-pacities within the world; that the simulated world is able to au-tonomously originate; that as a result interaction can lead to ex-ploratory discovery; and that visitors become part of the ecosys-tem, both through immersive display and through interactions thatinduce presence. Details of how each of these components havebeen applied in the visualization, sonification, and interaction de-sign are given with specific examples of prototypes and exhibitedinstallations.

1 INTRODUCTION

Since 2007 we have been engaged in a research-creation projectdefined by the creation of a series of “artificial natures”: inter-active visualizations of biologically-inspired complex systems thatcan evoke aesthetic experiences we know from nature, but withinmixed-reality art installations [10]. By “mixed-reality” we refer tothe spectrum of augmented virtual reality (VR) to augmented real-ity (AR), that is, blending elements from the real world into a vir-tual space and/or elements from a virtual world into real space [21].Our visualizations are displayed at high-levels of sensory immer-sion, through the use of large-scale displays, wide fields of view,stereoscopic rendering, high frame-rates, and spatialized audio.

Each artificial nature presents a computational world with itsown physics and biology, within which visitors interact to become

∗e-mail: [email protected]†e-mail: [email protected]

essential participants within the system. As artists we are moti-vated to use the open-ended mechanisms and processes of life toevoke extended aesthetic experiences that recapitulate somethingakin to the child-like wonder regarding the complexity, beauty, andsublimity of nature. The resulting installations have been exhib-ited and presented at more than thirty international events and havereceived national and international awards.1

Developing these works has involved many of the techniques andchallenges characteristic of interactive data visualization in general,however some of the specific implications of their goals have ledus to unique solutions and principles of development. In compari-son to the principles outlined in Tufte’s landmark text [34], we findconvergence in the emphasis of showing the data above all else;in maximizing the “data-ink ratio” (avoiding the use of “non-data-ink”); in revealing multiple levels of detail in large or complex data;and in inducing the viewer to think about the substance of the dataand the relationships between its elements. However our workingprinciples diverge by eschewing the integration of statistical or ver-bal descriptions of the data; nor do our works serve clear analyticpurposes of data description, tabulation, or comparison. In this pa-per we show how these differences are derived from developmentprinciples specific to our problem domain, and detail how they areapplied in the visualization, sonification, and interaction design ofprototypes and exhibited installations. We hope that documentingour progress in this paper will positively inform data visualists andcomputational artists in the broader community engaging similarchallenges.

1.1 Form and functionTufte’s first principle of “show the data”, to induce the viewer tothink about its substance [34], in our case becomes “show the sys-tem”, to induce the viewer’s attention toward its substantial quali-ties. And echoing Friedman, both aesthetic form and functionalitymust go hand in hand, providing insights by communicating key

1The project is also documented at http://www.artificialnature.net

30 Proceedings of the IEEE VIS 2015 Arts Program, VISAP'15: Data Improvisations, Chicago, Illinois, October 25th-30th 2015

aspects of the world in a more intuitive way [7]. But here lies animportant subtlety. Since the system realizes a model world, theprinciple must be followed more exclusively–to a principle of en-dogenous visualization. This requires that every perceivable ele-ment must have dynamic ontological2 capacity in the world, play-ing an active role in multiple world-level processes with other suchelements. If one views this as a restricted subset of Tufte’s conceptof “data-ink” (the non-erasable, data-dependent core of a graphic),the principle implies a corresponding requirement to maximize thethe ratio of “endogenous-ink” to total ink.

In terms of visualization this poses some interesting challenges.It implies that “non-diegetic” media3 should be avoided: no sec-ondary or symbolic notations, none of the well-researched statisti-cal graphical devices familiar to data artists, etc. Instead, all pro-cesses of the world must be conveyed through the sensory displaysof components of the world itself. For example, although proto-types may feature textual annotations, such as property labels spa-tially attached to agents, we cannot include such non-diegetic ele-ments in an exhibited work as they have no ontological capacities inits process. Whatever salience such annotations communicate to usmust instead be conveyed through intrinsic perceivable features ofthe agents themselves. So, just as in nature we perceive the wind byhow it moves the leaves, in our virtual world we perceive the fluidsimulation by how it moves suspended particles. Just as the fallenleaf’s color describes its state of decay, a virtual organisms textureimparts information about its internal changes over time.

In practice, as will be detailed below, this also means that thedevelopment of the visualization and the underlying system (or, insoftware-engineering terms, the view and the model), evolve hand-in-hand. There is no separation of concerns here: instead theybecome increasingly interdependent with neither side interchange-able.

1.2 Becoming part of the systemData visualization and interactive art are both deeply concernedwith machine-mediated experience. The meaning of a particularwork depends essentially upon what we can experience through itsresponses, what we can do within it, and how this reflects criticallyupon ourselves and our environment [13]. In the case of artworks,what we can do may be purposefully left quite open-ended; unlikecanonical data visualizations, participants are not executing specificanalytic tasks, and the work is not designed to satisfy user-drivengoals or convey specific ideas effectively; rather the meaning is typ-ically multi-layered and often contextually-derived.

In our case, as artists we are deeply motivated to create compu-tational environments that draw more from nature’s sense of open-ended continuation, than rational senses of utilitarian closure. Ourchallenge is to design interaction such that the visitors may explorean open-ended space, drawing meaningful responses, while indi-rectly influencing its adaptive conditions, and thus partaking in eachothers living time as a fulfilled aesthetic experience [5].

We believe this is made possible by considering the visitorsas fully-integrated components within the simulated world. Inthis regard we concur with interactive art pioneers Sommerer andMignonneau, who stated a goal that “a visitor must become partof the system to realize that there are no pre-defined solutions ofwhat to do and what to see, and that instead the artwork develops

2In this paper we use the term “ontological” with its original philo-sophical meaning–regarding the nature of existence, being, reality, or ulti-mate substance–rather than the usage it has taken on in information science,which specifies categories and relationships of entities within a domain ofdiscourse.

3This terminology is borrowed from film theory: diegetic images andsounds emerge from within the space of the story world, whereas the sourcesof non-diegetic media, such as the sounds of narrator commentary, appearto come from outside the story world [3].

through his or her interaction...the longer one interacts the more onebecomes part of the system” [32] (emphasis added).

In order that visitors feel that they are part of the system, wehave focused upon methods of display and interaction that empha-size immersion, presence, and agency. We prioritize indirect modesof interaction that integrate with the complex network of feedbackrelations in the world, which tends to reduce simplification into atask-oriented or pre-defined roles. Moreover, to support exploratorydiscovery, we choose simulation strategies that can engender open-ended behaviors.

1.3 From immersion to presence, driven by agencyThe success of our works depends in part on the extents visitorsbecome absorbed within the generated world and find meaning inthe actions they perform. We can address this through concepts ofimmersion, presence, and agency, which contribute to the intensityof the experience. Presence is generally understood in terms of asubjective sense of being there, or being engaged such that the arti-ficiality of the situation becomes suppressed. Lombard and Dittondescribe it as the continuous illusion of non-mediation, with hu-mans behaving as if the mediation was not there [17]. Naturally,immersion is conducive to presence, but is not the sole factor. Pres-ence appears to also depend on interactivity, and crucially on howsuccessful actions are supported; experience in general is groundedmore in functionality than appearance [30]. Moreover, presencehas been found to correlate very strongly with agency, a subjectiveproperty defined as the satisfying power to take meaningful actionand see their results [9], that is deeply rooted in bodily experience.

In that regard our interaction inputs have gradually shifted awayfrom hand-held devices oriented to direct active control, toward am-bient sensing with microphones, infra-red and RGBD range cam-eras capable of low latency natural “transparent UI”, and a widerspectrum of embodied interaction. This also signals a shift fromvoluntary control to forms of interaction in which continuous nu-ances have more value than discrete commands. By eschewing pre-defined tasks it also lends weight to the world, which appears lessin need of human action for its reason to exist, and in fact willcontinue its life without human presence. Moreover it is chosen topermit a much more open-ended range of possible interactions tobe discovered, such that visitors can discover behaviors and revealthe emergent relationships between elements by interaction and ob-servation, as a child learns by playing in nature.

1.4 Open-endednessThe richer the resolution, speed and flexibility of interaction, themore integrated its evolution can be, and the more humans and ma-chines can cohere into new awareness. But for this to be possible,the simulated world itself must carry the capacity to continuouslyadapt and generate new patterns of behavior. To that end we turnto biologically-inspired complex systems, as found in the field ofartificial life. In particular, almost all iterations of our work aremodels of ecosystems in which large numbers of mobile agents(evolving populations of organisms) interact with a dynamic en-vironment.4 Evolution in the ecosystemic approach features an en-dogenous selection criterion, such as maintaining sufficient energylevels to survive and reproduce, which differentiates it from evolu-tionary approaches ‘steered’ evolution by fitness functions that areeither defined in advance or actively directed at run-time by a user.

The ecosystemic approach to evolutionary art is well-established. Antunes et al. provide a detailed review comprisingforty artworks (including three artificial natures) using evolutionary

4We note the similar set of concerns and methods (sustaining interestwith endlessly fascinating interaction by means of passive interaction and anagent-based complex adaptive system within a liminal space), but applied ona level of social rather than ecosystemic dynamics, in a recent art installation[19].


Figure 2: Early 2D prototype visualizing agent-environment interac-tion (2007).

computational ecosystems produced since the mid-1990s [1]. Animportant observation made is that, perhaps surprisingly, very fewof these biologically-inspired works attempt realistic representationof life as we know it; instead, abstraction and/or alien surfaces andvolumes are dominant. Similarly, the majority of works eschew pre-recorded naturalistic sounds in favor of algorithmically synthesizedaudio. To an extent this echoes the “life-as-it-could-be” motto of ar-tificial life research [15], apparently liberating artists from realism.Nevertheless, as Whitelaw previously noted ([39]), artificial life artremains representational, and still owes a debt to “organicism” inthe arts. Though is not a mirror to the appearance of nature, itnevertheless represents the way life operates: “an aesthetic that islargely focused around visualizations of processes of life.” 5

2 VISUALIZING DYNAMIC ENVIRONMENTS

The strength of a particular ecosystemic model derives significantlyfrom the richness of the environment that supports it [20]. Theenvironments used in artificial natures are dynamic, incorporatingphysically-inspired processes that conserve matter-energy, save forsome entropic loss, in every transaction. Moreover the environ-ments are dissipative, subject to the kinds of energetic gradientsthat keep them away from equilibrium, and the varying rates of dif-fusion that can lead to naturally-occurring self-organized structures[23].

2.1 2D and 2.5DIn our earliest two-dimensional prototypes, the environment ismodeled as a discrete field of cells spanning the space throughwhich organism-agents may move. Each cell stores a concentrationof each of three pseudo-chemical substances, simply visualized be-neath agents as red, green and blue intensities (see Figure 2).

Each agent continuously extracts from this environmental fieldthe quantities of each chemical it requires; in effect leaving a dark-ened trail behind as it travels. With sufficient nutrition and appro-priate age agents may reproduce, but in regions of scarcity theyperish. In addition to motile behavior, the agents’ genetic structurealso specifies the relative mixture of chemicals it requires to subsist,which we visualize in the color of the agent. Reproduction incor-porates a degree of genetic mutation by which organisms can adaptto the environment, often resulting in clusters of similarly-coloredagents occupying specific regions of the world. The environmentitself is a continuous process incorporating an energetic gradient–each cell gradually accumulates a location-specific nutrient chemi-cals (akin to photosynthesis)–as well as a process of diffusion grad-ually smoothing out the chemical landscape. Population cycles andexplosions, evolved searching behaviors, nomadic migrations, ge-netic distributions, and the dynamics of the environment itself areall readily visible through this direct visualization.

We extended this model to attach to each cell a rate of nutrient re-covery. Following our principle of deriving form from function, weextruded the visualization into 3D by displaying this rate as height.

5In fact, as Shanken noted, it would be more accurate to describe themas visualizations of current theories of the processes of life [31].

Figure 3: Visualizing fertility of the environment as altitude, and map-ping its gradient to tendencies of diffusion, led to more complex pop-ulation dynamics.

In effect, this created an easily understandable “2.5D” topographyin which the higher altitudes are the most fertile. Then, in a re-verse step of creating function from form, we used the gradient ofthe landscape to skew the rate and direction of environmental dif-fusion. These simple modifications led to immediate increases inbehavioral complexity. Rival populations compete for dominanceof the rich mountain peaks, but once there the increased availabil-ity of resources leads to decimation by overpopulation, allowingmore nutrients to drain through valleys to the more sparsely pop-ulated lowlands, inviting other groups toward the peaks again (seeFigure 3).

This topographical approach to the environment was reprisedseveral years later in Archipelago (see Figure 1, right). In thiswork the visualization is projected from above onto a physical land-scape constructed of sand, whose topographic height map is derivedthrough the use of ceiling-mounted RGBD cameras. The simulatedenvironment comprises a larger number of fields, including chem-ical traces left by a variety of different species of organisms (seeFigure 4).

2.2 Immersive 3D

A number of our works have emphasized the immersive qualities of3D virtual world, with a navigable viewpoint embedded inside thesystem (Infinite Game, Fluid Space and Endless Current). In mov-ing to a fully three-dimensional, first-person environment we facedthe challenges of how to both simulate and visualize an dissipativeenvironment that occupies all surrounding space.

The most obvious 3D equivalent of our 2D prototypes wouldbe cloud-like volume rendering. However in this approach localdensities filter or occlude more distant features, and in lacking dis-cernible edges, depth and motion are more difficult to accuratelyperceive (see Figure 5). Data visualizations often refine volumerendering by extracting isosurfaces from level sets or using col-oring algorithms to categorize areas, but these refinements wouldsubvert our principle that all visible elements have ontological re-ality in the world. An isosurface is an abstraction of the underlyingdata that filters out all but an arbitrarily chosen isolevel, whereasthe organisms in the world perceive the full continuum of levels.

We also experimented with lattice-like structures such as grids,trees and networks, to provided sharper features for depth cueswhile being more visually porous, at the cost of requiring an on-tological role within the world. Some artificial natures have re-tained these components as weed-like organisms drifting in space,through which nutrients can be transported (see the purple vines inFigure 8).

Our most widely-used method of visualizing the dynamics of the3D environment has been through particles. Particle movement candisplay the rich complexities of fluid flow and turbulence, particlecolor can display variations in pseudo-chemical constitution, par-ticle quantity conveys concentration without unduly obscuring dis-


Figure 4: Close-up details of Archipelago. Top: A trail of ant-likecreatures (pink and green) is following and replenishing a stigmer-getic path of pheromones (yellow). In the lower-right area of thisimage a lone forager is depositing a new pheromone trail. Bottom:A flocking group organisms (red bodies, blue tails) in the lower al-titudes have over-consumed the white mould-like species coveringmany parts of the landscape and are now dying out, leaving palecarcasses surrounded by a diffusing field of decay (red), which mayattract the scavengers visible in the higher altitudes.

tant regions, and particle size serves as a cue for distance (see Fig-ure 6). As a result particles reveal properties of the world at severallevels of detail, from broad overview to fine structure. Moreover as-signing an ontological role to the particles, as carriers of chemicalnutrients, is trivial.

3 VISUALIZATION WITH AGENTS

Multi-agent systems are well-established model for simulatingcomplex systems. Although they are perhaps less familiar as a toolof visualization, they have been a recurrent visual and sonic fea-ture of works in the AlloSphere immersive instrument [36]. Themodel is closely-related to particle-based visualization: like par-ticles, agents are mobile entities that visualize and respond to lo-cal features. However where particles generally follow physically-inspired, simple, global rules, the rules agents follow are oftenmore biologically-inspired, showing autonomous, stateful, individ-ual, and even intelligent behaviors. Populations of agents can sup-port higher-level macro-behaviors such as flocking [27].

Some software visualization toolkits, such as Behaviorism, in-clude support for attaching behaviours to visual and data elements(and to other behaviours), effectively supporting the constructionof agent-based visualizations [6]. With toolkits such as Repast[24] and MASON [18] the assumption is that agents are an inte-gral part of the simulation model being visualized. In some caseshowever agents are being used as a supplementary method for visu-alization, such as computer-assisted distribution of data exploration

Figure 5: Prototype visualizing the dynamic environment in 3D viaraycast volume-rendering. Note that even with accurate depth occlu-sion with other objects, cloud depth remains difficult to apprehend.

Figure 6: Visualizing the dynamic environment via particles (InfiniteGame, 2008).

roles [33], or the intelligent and autonomous distribution of textualannotations [8]. These examples demonstrate the use of softwareagents to augment human capabilities by focusing users’ attentionto relevant information elements [29].

In our work, each agent is an evolved artificial organism travers-ing the dynamic environment, seeking nutrition to survive and re-produce. Following our principle of endogenous visualization, weattempt to show key aspects of the underlying processes of the or-ganisms and their relations to other elements of the world. Forexample, in our work Fluid Space it is possible to watch organ-isms gradually mature from spherical-like eggs into fully-developedcreatures with undulating petal-like appendages, and up close it isalso possible to observe inside them the food particles that they haveeaten, also gradually changing in color as they are metabolized, un-til they are ultimately ejected back into the environment.

Evolution acts upon not just phenotypic properties but also be-haviors of the agent, through the development of a unique programaccording to a genotypic process of inheritance and mutation. Forexample, in Time of Doubles the agent’s phenotype program is theevolutionary product of an intermediate metaprogram derived fromthe individual’s genome tree, in a variation of genetic programming[11]. This program will be invoked repeatedly through the organismlifespan (see the much-simplified example in Figure 7), using a sub-sumption architecture [4] to blend decision-making with metabolicviability constraints. It takes a number of inputs (external and in-ternal sensors), applies a series of expressions, and according to


applicative predicate conditionals may produce a number of side-effects including motor actions, growth, reproduction, and memorystorage, along with their energetic cost. To side-step the haltingproblem, backward control-flow is not permitted; looping behav-ior must be implemented in terms of memory stores and reads dis-tributed over successive invocations of the program (more details in[35]).

The longer-term organism-environment relationship is one ofknowledge gathering: species must adapt to extract inferences fromthe complex data set in which they are immersed–a fluid simulationin which nutrient-particles are suspended. The evolutionary modelhas no fitness measure, but the viability requirement that organismsmust locate and consume food in order to reproduce imparts anendogenous selection pressure that can only drive the gene pooltoward developing agent-programs that result in superior searchstrategies. In this regard, the agents form a distributed, adaptivesearch algorithm within a data-set of nutrient distribution, and eachindividual organism is an improvised proposal for its solution. Butagain, this is a search without end, circular rather than linear, as theenvironmental data-set is itself modified (depleted) by the agents’activities.

We note here that Ware defined information visualization as theinterface connecting the human perceptual system–understood asa flexible pattern finder coupled with an adaptive decision-makingmechanism–with the calculating power and information resourcesof a computer [38]. Of course the domains of artificial intelli-gence and artificial life explore purely computational mechanismsof pattern-finding and decision-making. Our own integration of vi-sualization with evolving autonomous agents is suggestive of thepotential of a hybrid spatial interaction merging both human anddigital forms.

4 IMPLEMENTATION EFFICIENCY

The combination of high-resolution, high frame-rate display, andlow-latency, high-bandwidth sensing puts tremendous pressure onthe efficient implementation of our works. We generally use stan-dard languages and software techniques to maximize the capabili-ties of hardware, however there is one aspect particular to our workthat warrants further explanation.

The simplicity and homogeneity of particle-based visualizationslend themselves to optimized implementations that can support vastnumbers of particles. Agent-based models as described above how-ever are much more challenging, due to the increased complexityof and diverse heterogeneity between phenotype programs. Therehave been examples of using GPGPU programming to address theissue of program complexity [28], but this is less able to resolve theissue of heterogeneity: the ahead-of-time, massively parallel opti-mizations that GPGPU computing can provide are generally incom-patible with the characteristic tendencies of open-ended phenomenaaway from invariances (the breaking of symmetry).

We therefore utilize dynamic code generation and just-in-time(JIT) compilation to avoid unnecessary compromises, and havebeen able to support thousands of unique yet concurrent agents atimmersive simulation rates. Agent programs are dynamically com-piled to efficient machine code at birth, using the LLVM compilerinfrastructure [16]. In this way we can leverage the open-endedspace of genetic programming with far greater efficiency, creatingnew machine code components at high frequency and in parallel.

For Time of Doubles were able to maintain a frame-rate con-sistently above 50Hz with a population of 2000 concurrent uniqueorganisms of 40 or more intermediate operations each, replacingon average 30 individuals per second (measured on a 2GHz IntelCore i7 processor with 2GB RAM). Genetic processing and dy-namic compilation occur in background threads to avoid renderingstalls during population explosions, and careful garbage collectionpermitted continuous uptime over several months. We hope to ex-

Figure 7: Above: A simplified example of a phenotype program (sim-plified for legibility; actual graphs approximately 10x larger). Out-line arrows show control-flow, solid arrows show data-flow. Bluenodes are control-flow blocks, white nodes are data operations,green nodes are terminal sensor, memory, or constant inputs, orangenodes are side-effect operations. Note that the resulting phenotypeprogram structure is a graph, not a tree. Below: The same programtranslated to equivalent C code, for explanatory purposes (in the run-ning artwork, the genomes generate LLVM bitcode directly, for moreefficient translation to native machine code).

tend this open-endedness to include the processes of genetic muta-tion and development within the phenotype program, and broaderaspects of the system in future installations.

We note that although bytecode generation is well explored inartificial life [25, 26], run-time machine code generation has rarelybeen explored for interactive generative art. We have elsewhereindicated the potential for exploratory data visualization within theconstraints of immersive performance [36].

5 SONIFICATION

Augmenting visual display with sound allows events to be appre-hended even if the corresponding visual objects are occluded, andaffords an unlimited “field of regard”. Furthermore humans candetect much finer temporal structures over a wider range of fre-quencies in sound, and more readily perceive and process audio inparallel [12]. Accordingly, all artificial natures have utilized algo-rithmic approaches to sound generation at microsonic time scalesto fluidly reflect subtle changes in the ongoing simulation.

For example, in Fluid Space, sound is used to reveal lateral genetransfer. When an agent detects another close by, it may emit its ge-


netic information through a characteristic song, synchronized witha visual flash to precisely identify the source agent. The sonicdesign is also algorithmic, a frequency- and amplitude-modulatedgrain parameterized by the genetic data. Nearby agents may absorbthis genetic information, with errors (mutations).6

The agents in Time of Doubles are sonified using a granular pro-cess inspired by cricket chirps, where each sound event consistsof a brief train of narrow-band pulses followed by a longer restinginterval. These chirps are spatialized with direction and distancecues according to the agent location. The agent’s genome is usedto parameterize the various properties of the pulses, including pulsefrequency, burst rate, burst length, envelope shape, etc. We notehere the promising strategy of biological inspiration for data soni-fication. Since many organisms have evolved sounds specificallyadapted to be easily localized and identified, even in a noisy en-vironment. In our case, the cricket-inspired chirps are readily lo-calizable due to the bursty envelope, while the narrow frequencyrange used allows many individual voices to be concurrently iden-tified. Moreover, as populations grow and collapse the soundscapedevelops from isolated pulses to dense clouds of sound, whose tim-bres vary with the evolving gene pool. Evolutionary events (suchas the discovery of a better search program) cause rapid and readilyperceivable changes to the timbre of the clouds. As with visual par-ticles, this approach to sonification reveals properties of the worldat multiple levels of detail.

Sound enhances the sense of presence, most dramatically whenthe sound is properly spatialized as surround audio. As a metaphorfor information display [22] agents are well-suited to spatializedsonification.7 Arbitrary sound sources can be attached to agentlocations and trajectories, using directional cues of amplitude andphase differences over multiple loudspeakers or head-related trans-fer functions (HRTFs) with headphones, and further modulatedwith physically-based cues according to distance and velocity in-cluding amplitude attenuation, high-frequency absorption, Dopplershift, and reverberation mix. We have utilized all of these tech-niques for our exhibits of Fluid Space and Time of Doubles in theAlloSphere immersive instrument [14], and for Endless Current onthe Oculus Rift head-mounted display (see Figure 1, center); andhave adapted them to the degree possible for multi-channel galleryexhibits.

6 INSTALLATION AND INTERACTION DESIGN

Each artificial nature installation invites participants to become partof the ecosystem through a responsive environment utilizing im-mersive or mixed-reality audiovisual display. Our first installationsutilized a typical immersive format, projecting on walls or largescreens within darkened rooms. Navigation devices allow visitorsto explore the worlds with six degrees of freedom. This directionnaturally has found its best expression in deeply immersive facili-ties such as the AlloSphere (see Figure 8).

However we found that the first-person egocentric perspectivehas a tendency to create a distanced objectivity due to disembodi-ment, weakening presence. To bring the body back into the worldwe began exploring mixed realities of augmented virtuality (bring-ing the body into the virtual world) and augmented reality (project-ing the world back into a body-centric spatiality).

In Time of Doubles we removed navigation and began projectingonto curved, architectural/sculptural objects mounted in the mid-dle of gallery spaces (see Figure 1, left), creating a mixed realityemphasizing continuity between the real and virtual spaces. The

6Although we describe this as lateral gene transfer, we could equally re-cast it as social knowledge sharing. The net result is the same: behavioralchanges can propagate through populations faster than population regener-ation.

7Not surprisingly, they have also been utilized in interactive art, see forexample [2].

Figure 8: Wide-angle photograph of Fluid Space taken from thebridge of the AlloSphere, University of California Santa Barbara,2012. This version of the Fluid Space application is distributedover 16 computers and 26 projectors covering an almost completelyspherical screen, and 54 loudspeakers mounted behind it.

mixture of real and virtual is augmented by the projection of visi-tors’ “double” into the ecosystem (see Figure 9). These doubles arenot avatars, but mirror existences that closely reconstruct the shapeand movements of participants as volumes of high-density, high-energy particles, by means of an array of RGBD cameras. Thepresentation of a recognizably human double induces an immediatepsychological link with the otherwise alien virtual world. (This de-sign is addressed in terms of presence and agency in [37]). Howeverthe doubles in the virtual world have quite distinct behaviors, whichextend visitors into alternate roles within the network of relationsas energetic sources and kinetic disturbance. Participants thus see,hear, and feel how they are fed to unknown species, bringing fortha meaningful tension.

In Archipelago the format becomes more sculptural: we projectfrom above onto an archipelago landscape constructed of sand.From above, an array of RGBD cameras is used to determine thetopography of the landscape, shaping the adaptive conditions of thespecies below. The choice of an archipelago reflects the necessity ofniche conditions to support divergent evolution, and also echoes theenvironment in which Darwin’s theory of evolution was inspired.The sensors also track visitors in the space in order to mirror theiractual shadows over the landscape with virtual counterparts in thesimulation. Shadows are re-projected in black, ensuring that vis-itors are never themselves illuminated by the patterns of the land.All life in regions of the world under shadow is destroyed, yet theland is simultaneously refertilized. The interaction becomes moresubtle and sensitive when visitors reach down to touch the land.The landscape itself is malleable, using a special kind of “kinetic”sand that does not dry, allowing visitors to reshape the topographyor even to destroy and create new islands. Moreover, participantsmay also watch organisms creep onto their hands, from where theycan be carefully carried to other islands, as tracked by optical flow(see Figure 10).

Human presence and activity has primarily indirect influenceson the evolving life, which is predominantly realized by diffuse ef-fects on the shared environment: as a source of turbulence (in FluidSpace); as a source of high-energy particles (in Time of Doubles);or the removal primary foodstuffs and gradual reshaping of topog-raphy (in Archipelago). In each case, consequences of actions areeasy to perceive, but ultimately difficult to predict; thus encourag-ing exploratory behavior.


Figure 9: Participants and their virtual “doubles”–emanations of yel-low nutrient particles fed upon by the organisms in Time of Doubles(Flux). Photographed at the Microwave International New Media Fes-tival, City Hall, Hong Kong (Nov 3 to Nov 25, 2012).

Figure 10: Transporting virtual organisms by hand from one island toanother in Archipelago. Note also the shadow underneath the hands,which is replicated by the projection of black: cast shadows annihilatethe white mould-like species below, but also refertilize the land.

7 CONCLUSION AND FUTURE DIRECTIONS

The apparently oxymoronic terminology “artificial nature” reflectsa conjunction of anthropocentric (subjective, authored) and cosmo-centric (objective, universal) values. As a cultural artifact (an art-work) it reveals “nature as it could be”, through endogenous pro-cesses at the level of her operation. Accordingly, our speculativevisualizations of possible natures need not be true to any originaldata set (or simulation). Thus in contrast to the clear directionalityin the development of most data visualizations, from the data itselfto the communication of its most significant aspects, in our caseboth model and view evolve in tandem as the work is developed(and to a lesser extent continue to evolve as they run).

In keeping with the theme of endogeny, an important strategyfor future work is to replace statically aggregated systems with ge-nealogical processes that may generate them: that is, fully-workingaccounts as to how every structure and function emerges from sim-pler initial elements. We hope that, in this regard, our explorationof code generation and rewriting systems opens a path to a broaderdiversity of worlds. But we must also acknowledge that we areattempting to engender nature-like complexity on vastly smallerscales of space, time and complexity. For example, the number of

operations in our agents’ programs are less than 1% of the neuronsof a C. Elegans nematode worm.

Despite the whole-world mechanics, each artificial nature re-mains a work that is oriented to human experience. Nevertheless,for us, deep interaction implies the growth of both the world andthe participants. This means the possibility of providing continualvariation and new potentials for new patterns of behavior in bothvisitors and world. To liberate this experience, an artificial naturemust grant freedom to participants as to what they see, observe,feel, and do. For example, both a three-year old child and an aneighty-year-old scientist can enjoy playing with waves, a swell, awash, and a breaker at the same time, yet each learns and playswith quite different values. The medium with which participantsinteract, and in which they are immersed, must share with nature itsmulti-dimensionality, multi-modality and dynamic complexity.

In our work, all four of the components of endogenousvisualization–form and function, human as part of the system, de-sign for agency, and open-ended generativity–are chosen to inten-sify primary aesthetic and exploratory experience. Reflecting upona cultural context that is increasingly immersed in computation, wehope the continued development of artificial natures will grant moretransformative experiences, and stimulate more creation and open-ended thinking in terms of nature and artifice.

ACKNOWLEDGEMENTS

This research was supported by the Sogang University ResearchGrant of 201210042 and the Canada Research Chairs program, So-cial Sciences and Humanities Research Council, Canada, and theNational Science Foundation grant IIS-0855279 at the AlloSphereResearch Group and the Media Arts and Technology Program at theUniversity of California Santa Barbara, USA.

REFERENCES

[1] Rui Filipe Antunes, Frederic Fol Leymarie, and William Latham. Twodecades of evolutionary art using computational ecosystems and itspotential for virtual worlds. Journal For Virtual Worlds Research,7(3), 2014.

[2] Daniel Bisig, Jan C Schacher, and Martin Neukom. Flowspace–a hy-brid ecosystem. In Proc. of the International Conference on New In-terfaces for Musical Expression, Oslo, Norway, pages 260–263, 2011.

[3] David Bordwell, Kristin Thompson, and Jeremy Ashton. Film art: Anintroduction, volume 7. McGraw-Hill New York, 1997.

[4] Rodney Brooks et al. A robust layered control system for a mobilerobot. Robotics and Automation, IEEE Journal of, 2(1):14–23, 1986.

[5] John Dewey. Art as experience [1934]. ALA Booklist, 30:272, 1980.[6] Angus Graeme Forbes, Tobias Hollerer, and George Legrady. behav-

iorism: A framework for dynamic data visualization. Visualizationand Computer Graphics, IEEE Transactions on, 16(6):1164–1171,2010.

[7] Vitaly Friedman. Data visualization and infographics. Graphics, Mon-day Inspiration, 14:2008, 2008.

[8] Timo Gotzelmann, Knut Hartmann, and Thomas Strothotte. Agent-based annotation of interactive 3d visualizations. In Smart Graphics,pages 24–35. Springer, 2006.

[9] Gerardo Herrera, Rita Jordan, and Lucı Vera. Agency and presence:a common dependence on subjectivity? Presence, 15(5):539–552,2006.

[10] Haru (Hyunkyung) Ji. Artificial Natures: Creating Nature-Like Aes-thetic Experiences through Immersive Artificial Life Worlds. PhD the-sis, University of California, Santa Barbara, California, USA., 2012.

[11] John R Koza. Genetic programming: A paradigm for geneticallybreeding populations of computer programs to solve problems. Stan-ford University, Department of Computer Science, 1990.

[12] Gregory Kramer. An introduction to auditory display. In AuditoryDisplay: Sonification, Audification and Auditory Interfaces, SFI Stud-ies in the Sciences of Complexity, Proceedings, pages 1–77. Addison-Wesley, 1994.


[13] Myron W. Krueger. Responsive environments. In Proceedings of theJune 13-16, 1977, national computer conference, AFIPS ’77, pages423–433, New York, NY, USA, 1977. ACM.

[14] JoAnn Kuchera-Morin, Matthew Wright, Graham Wakefield, CharlesRoberts, Dennis Adderton, Behzad Sajadi, Tobias Hollerer, and AditiMajumder. Immersive full-surround multi-user system design. Com-puters & Graphics, 40:10–21, 2014.

[15] Christopher G Langton. Artificial life: An overview. Mit Press, 1997.[16] Chris Lattner. The llvm compiler system. In Bossa Conference on

Open Source, Mobile Internet and Multimedia, Recife, Brazil, 2007.[17] Matthew Lombard and Theresa Ditton. At the heart of it all: The

concept of presence. Journal of Computer-Mediated Communication,3(2):0–0, 1997.

[18] Sean Luke, Claudio Cioffi-Revilla, Liviu Panait, and Keith Sullivan.Mason: A new multi-agent simulation toolkit. In Proceedings of the2004 swarmfest workshop, volume 8, page 44, 2004.

[19] Lindsay MacDonald, John Brosz, Miguel A Nacenta, and SheelaghCarpendale. Designing the unexpected: Endlessly fascinating interac-tion for interactive installations. In Proceedings of the Ninth Interna-tional Conference on Tangible, Embedded, and Embodied Interaction,pages 41–48. ACM, 2015.

[20] Jon McCormack. Creative ecosystems. In Computers and Creativity,pages 39–60. Springer, 2012.

[21] Paul Milgram and Fumio Kishino. A taxonomy of mixed reality vi-sual displays. IEICE TRANSACTIONS on Information and Systems,77(12):1321–1329, 1994.

[22] Keith V Nesbitt. Modeling the multi-sensory design space. InProceedings of the 2001 Asia-Pacific symposium on Informationvisualisation-Volume 9, pages 27–36. Australian Computer Society,Inc., 2001.

[23] Gregoire Nicolis, Ilya Prigogine, et al. Self-organization in nonequi-librium systems, volume 191977. Wiley, New York, 1977.

[24] M North, Thomas R Howe, Nick T Collier, and J Vos. The repastsimphony runtime system. In Agent 2005 Conference on GenerativeSocial Processes, Models, and Mechanisms. Argonne, Illinois, USA:Argonne National Laboratory. Citeseer, 2005.

[25] Steen Rasmussen, Carsten Knudsen, Rasmus Feldberg, and MortenHindsholm. The coreworld: Emergence and evolution of cooperativestructures in a computational chemistry. Physica D: Nonlinear Phe-nomena, 42(1):111–134, 1990.

[26] Thomas S Ray. Evolving complexity. Artificial Life and Robotics,1(1):21–26, 1997.

[27] Craig W Reynolds. Flocks, herds and schools: A distributed behav-ioral model. In ACM Siggraph Computer Graphics, volume 21, pages25–34. ACM, 1987.

[28] Paul Richmond and Daniela Romano. Agent based gpu, a real-time 3dsimulation and interactive visualisation framework for massive agentbased modelling on the gpu. In Proceedings International Workshopon Supervisualisation, 2008.

[29] Jeffrey C Rimland, Michael McNeese, and David Hall. Conservinganalyst attention units: use of multi-agent software and cep methodsto assist information analysis. In SPIE Defense, Security, and Sensing,pages 87580N–87580N. International Society for Optics and Photon-ics, 2013.

[30] Maria V Sanchez-Vives and Mel Slater. From presence to conscious-ness through virtual reality. Nature Reviews Neuroscience, 6(4):332–339, 2005.

[31] Edward A Shanken. Life as we know it and/or life as it could be:Epistemology and the ontology/ontogeny of artificial life. Leonardo,pages 383–388, 1998.

[32] Christa Sommerer and Laurent Mignonneau. Art as a living system:interactive computer artworks. Leonardo, 32(3):165–173, 1999.

[33] John Thompson, JoAnn Kuchera-Morin, Marcos Novak, Dan Over-holt, Lance Putnam, Graham Wakefield, and Wesley Smith. Theallobrain: An interactive, stereographic, 3d audio, immersive vir-tual world. International Journal of Human-Computer Studies,67(11):934–946, 2009.

[34] Edward R Tufte and PR Graves-Morris. The visual display of quanti-tative information, volume 2. Graphics press Cheshire, CT, 1983.

[35] Graham Wakefield. Real-time meta-programming for interactive com-

putational arts. PhD thesis, Media Arts and Technology, Universityof California at Santa Barbara, California, USA., 2012.

[36] Graham Wakefield, Tobias Hollerer, JoAnn Kuchera-Morin, CharlesRoberts, and Matthew Wright. Spatial interaction in a multiuserimmersive instrument. IEEE computer graphics and applications,(6):14–20, 2013.

[37] Graham Wakefield and Haru Hyunkyung Ji. Becoming-there: Naturalpresence in an art of artificial ecologies. In Proceedings of the 2013International Symposium on Ubiquitous Virtual Reality, pages 11–14.IEEE Computer Society, 2013.

[38] Colin Ware. Information Visualization: Perception for Design. Mor-gan Kaufmann Publishers, second edition, 2004.

[39] Mitchell Whitelaw. Metacreation: art and artificial life. Mit Press,2004.


endogenous biologically-inspired visualization immersed ... · and science (kriss), korea (aug...

Documents