defining digital space through a visual language

1

Defining Digital Space Through a Visual Language

by

Axel Kilian

Dipl. Ing., Department of Architecture (1998)University of the Arts Berlin

Submitted to the Department of Architecturein Partial Fulfillment of the Requirements for the Degree of

Master of Science in Architecture Studiesat the

Massachusetts Institute of Technology

June 2000

© 2000 Massachusetts Institute of Technology. All rights reserved

Signature of Author ___________________________________________________________ Department of Architecture, May 16,1999

Certified by ___________________________________________________________

William J. Mitchell, Thesis SupervisorProfessor of Architecture and Media Arts and SciencesDean, School of Architecture and Planning

Accepted by ___________________________________________________________

Roy Strickland, Chair, Department Committee on Graduate Students

© 2000 Axel Kilian. All rights reserved

The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis

document in whole or in part.

2


The Readers of this Thesis are

William Lyman Porter, FAIALeventhal Professor of Architecture and Plannig


John Maeda Sony Career Development Professor of Media Arts and Sci-ences, Assistant Professor of Design and Computation at the MIT Media Laboratory

3


by

Axel Kilian

Submitted to the Department of Architecturein Partial Fulfillment of the Requirements for the Degree of

Master of Science in Architecture Studiesat the


June 2000

William J. Mitchell, Thesis SupervisorProfessor of Architecture and Media Arts and SciencesDean, School of Architecture and Planning

Abstract:

Current approaches toward digital spaces mainly mimic the physical space that surrounds us. While this approach is valid in a wide range of applications and research, the goal of this thesis is to propose an alternative approach towards digital space based on principles of vision and memory to provide a theoretical framework for the development of a model for accessing and moni-toring information.. The core idea is to incorporate the principle of dynamic reaction of the space model to the point of attention of the user through methodically scaling the level of detail. The intention behind the creation of such a model is to enhance the communication between the information and the user as well as to allow multiple users to collectively influence and expand the space they view. This involves the definition of a visual language, defini-tion of the space model and multi-user relations. Techniques proposed in this thesis are gaze tracking to map the point of attention and the programming of visual interfaces that respond dynamically to user input.

5

AcknowledgementsI am deeply thankful to Prof. John Maeda and his Aesthetics and Computation Group. He invited me into his group for the thesis semester and everyone in the group gave me great sup-port and inspiration both through their skills and personalities. I want to thank everyone in the group, Prof. John Maeda, Tom White, Elise Co, Ben Fry, Golan Levin, Casey Reas, Jared Schiffman and Max van Kleek.

I also want to thank Prof. William Porter who supported the development of this thesis through awarding me a Research Assistantship in a collaboration of the SPORG group with MERL.I enjoyed the work and am very thankful for the support of Paul Keel in numerous discussion around thesis and the RA projects ewall and viper which explore applications of this thesis for collaborative work environments.

I want to thank Prof. Peter Testa and Devyn Weiser for their continuous support from the very beginning of my thesis and awarding me a Teaching Assistantship for the Design studio of the future/emergentDesign studio in Fall 1999. I found the work with them both challenging and rewarding.

I would like to thank Dean William J. Mitchell for being my advisor for the two years of my studies. He has provided me both with inspiration and good advice and supported me in the path of my thesis.

I want to thank everyone in the mezzanine for being of great inspiration and always willing to discuss each others work. Among them are Mine Ozkar, Michael Fox, Paul Keel, Patrick Charles and many others passing by.

I also want to thank the German American Fulbright Program for making my stay at MIT possible through a stipend for the academic year 1998-1999. I want to thank the Germanistic Society of America for supporting my studies with a grant enabling me to finish my degree. I also would like to thank the Department of Architecture for supporting my studies in my second year.

And I want to dedicate this thesis to Susanne who gives me the strength to carry on even if it seems impossible.And last but not least I want to thank my parents for always supporting me in my goals and for their continuous interest in my work.

7

1. Introduction 1.1 Overview 1.2 Problem statement 1.3. Purpose - The Importance of Programming 1.4 Scope 1.5 Necessity for a Model for enhanced Handling of Data

2. Background 2.1. Context 2.1.1 The Use of Image Space 2.1.2 The Interface and its History of Using Metaphors 2.2. Related Work 2.2.1 Precedents in the Field of Information Visualization 2.3. Image Space Precedents in the Arts 2.3.1. Medieval Painting - Erwin Panofsky - Perspective as Symbolic Form 2.3.2. Painting as Formulation of Thought 2.3.3. Link between Space and the World 2.3.4. On Innovation in the Arts 2.3.5 Cubism

3. Theory 3.1. The Visual Interface 3.1.1 John Maeda – “Design by Numbers” 3.1.2 William J. Mitchell – “The Reconfigured Eye” 3.1.3 David Kirsch - “Complementary Strategies: Why We Use Our Hands When We Think” 3.1.4 Marvin Minsky - K-Lines 3.1.5 Alfred Ultsch “Hybrid Systems”. 3.1.6 Richard Held and Alan Hein “Movement-Produces Simulation in the Development of Visually Guided Behaviors.” 3.1.7 Stuart K.Card, Jock D. Mackinlay, Ben Shneiderman - “Visualization Using Vision to Think.” 3.1.8 Joachim Sauter “Zerseher.” 3.2. Definition of Image space 3.2.1 Set of all Images Displayable in the Medium 3.3. Role of Attention 3.3.1 Perception=Interaction 3.3.2 Gaze Tracking 3.3.3 Dynamic Visual Feedback System 3.4. Interface Development

4. Design and Decision Criteria 4.1 Themes 4.1.1 Interaction through Perception 4.1.2 Omnipresence of the Entire Space within the Visual Frame 4.1.3 Attention based Navigation

9

4.1.4 Problems 4.2. Interface definitions 4.2.1. A Dynamic, Reactive User Interface 4.2.2. Distortion Viewing 4.2.3. Dynamic Level of Detail Expansion 4.2.4. Omnipresence of all Elements 4.2.5. History of Interaction Influences Viewed Space 4.2.6. Collective Property of Digital Space 4.2.7. Individual Viewing of global Sets of Data 4.2.8. Influence on the Data through the Act of Viewing

4.3 Principles 4.3.1. Characteristics 4.3.2. Design Criteria 4.3.3. Design Process 4.3.3.1. Fixed Topology 4.3.3.2. Loose Topology 4.3.3.3. Reactive Space 4.3.3.4. Shared Space 5. Application 5.1. Development Process 5.1.1. Image Library 5.1.2. Visual History and User and Context Awareness for Web Browsing 5.1.3. Way Finding Composite Maps for Individualized Way Descriptions 5.1.4. Collaborative Work Environment 5.1.4.1. Collaborative Aspects of Digital Space

6. Discussion 6.1. Architecture and the Digital Medium 6.2. Architecture and Image Space 6.3. On Distance 6.4. Limitations of the Physical Metaphor 6.5. Information Space

7. Conclusion 7.1. Recommendations 7.1. Focus on the Development of Visual Navigation Models 7.2. Future Development 7.2.1. Hardware Enhancements 7.2.2 Visual Communication

10

Defining digital space through a visual language

11

Image composition of the cityscape of Sao Paolo. Through the composition the original image turns into an image space which looses its perspective quality. The result is a texture of elements. [Image by Sampa at http://pasture.ecn.purdue.edu/~agenhtml/agenmc/brazil/images/sampa1.gif]

1. Introduction

1.1 Overview

The fascination of the idea of an abstract space model for interacting with digital sets of informa-tion drove me to write this thesis.

The amount of data in digital form is increasing at a steady pace and so is the usage of this data by an increasing number of people. A space of information has developed which is rapidly unfold-ing. The space is based on digital data that grew out of a text-based format. With the spread of the use of images and temporal representations, information has become increasingly complex.

The properties of this information space have changed little from its paper-based origins. Anima-

A Map of the thesis experiments that identifies the general directions that were taken during the process. The respective experiments are placed along those lines

12

Issue of communicating space with the senses.

Set of digital information.

tion and time-based changes are common but appropriate dynamically computed visual inter-faces are lacking. The use of visual feedback could greatly enhance the perception of the space as a whole and develop metaphors unique to the system.

In order to experience a space one has to be able to communicate it to the senses of a human. Vision is an important part of the communication. The visual representation of physical space has undergone several stages over the history of pictorial representation. The thesis will focus on the visual aspects of space. Our contemporary view of space representation is dominated by the perspectival method, a method that assumes a homogenous continuous space that is being pro-jected onto an image plane.

The subject of the thesis is the definition of digital space through a visual language. I understand digital space as the set of all information in digital form. People can access this information space through digital interfaces. I use the definition of space since the set contains both information and a representation of the people accessing it.

The current notion of digital space is dominated by the approach of mimicry of physical space that surrounds us. The idea of a simulated physical space developed out of simulation projects for the military and closely related to the development of real time computing in the 1950’s. With the advent of the Internet the notion of the physical space metaphor has outlived itself, The Web is an abstract information space with no apriori dimen-

13

Perspective Sketch.

Communication Channel.

sions. The thesis attempts to provide a theoretical framework for the development of a model for accessing and monitoring information by estab-lishing a theoretical background.

The work develops the notion of a genuine space for digital information that does not rely on meta-phors of other spaces. In order to access the contents of this space there is the need for a communication channel. It is developed through by applying properties of attention based visual interaction. The idea of incorporating attention into the visual language is to make visual interaction a part of the perception.

The problem being addresses is the lack of models for dealing with the ever increasing amount of information in digital form and the expansion of individual digital space. This expan-sion brings about a need for a more capable model for handling and creating awareness of the social and knowledge context inherent in digital space. The development of a model is important since we rely increasingly on digital space in our professional, personal and educational environ-ments. The potential of the inter connectivity of such a space is under utilized, if only addressed by current access and monitoring technologies.

It ties together a variety of techniques from the field of data visualization into a larger model capa-ble of serving as a platform for interaction within digital space. The goal is to demonstrate the feasi-bility of attention based visual interaction in the domain of heterogeneous data sets such as the Internet.

14

The development of the concept of focus and con-text and the related idea of distortion viewing in the 1980’s introduced the idea of dynamic visual response into the field of computer graphics. The direct visual interaction using information displays as the interface was developed in the “SAGE” air defense system based on the whirlwind computer developed at MIT in the 1950’s. Today’s visual interfaces are mostly static multitasking universal keyboards where buttons are being created to map to specific functional features. The notion of direct dynamic visual interaction has been largely neglected. With both the advances in technology and the miniaturization of the devices the role of interaction and monitoring of digital data will become a problem if performed solely through static interfaces.

1.2 Problem Statement

The task is to provide a model for visual output of digital data in a collaborative space that allows access and retrieval of data through visual naviga-tion based on attention.

Models of visualization operate by mapping data to visual output on the screen. The model can be a direct mapping of numerical values to color values and positions in an array of pixels for instance. More complex models process the input data through a mathematical model as it is the case in perspective models. The Perspective model is heavily used in the digital domain due to its capa-bility of simulating physical space visually. This thesis is trying to define a model for post-perspec-

15

tive visualization techniques which is paying trib-ute to the growing importance and reliance on digital data in the professional, educational and personal context. In science the use of statistical visualization models for digital data is common for the analyses of large data sets. Few of this models display dynamic properties though as one would need them in the rapidly changing context of heterogeneous digital data for a broader public. The motivation for the definition of an alternate model comes out of the experience of the increas-ing complexity of the entities contained within digital space. This is true both for social context as well as for data context as more communication and data exchange is being processed digitally. To communicate these inherently invisible processes to the human perception it is necessary to develop depiciton models that extract the genuine prop-erties of the digital medium and translate them into visual form. Important is the structure of the underlying model not so much the superficial resemblance of the visual output since a powerful model will allow the development of a visual lan-guage that potentially can function without the use of metaphorical references to physical space.

16

Thoughts about the Physical Realm and Estab-lished Architectural Practice. An obvious question is where the relation is between the development of a model for the visualization of digital data and Architecture. I believe that there is a strong similarity in the prob-lems Architecture is trying to solve in physical space and the problems at hand in digital space. In both cases the task is to find ways to deal with the complexity of program and the presence and interaction of people within the designed structure. In the case of physical space Architecture articu-lates itself in physical form through the tectonics of the materials and the composition of space. An Architecture in digital space could constitute itself similar using the tectonics of the digital systems and information as its materiality. The problems of program and inter human communication are similar. Over time Architecture in digital space may develop cultural value and historical depth as its physical counterpart has over the last thousands of years. So it is daring to speak of digital Archi-tecture as an equally developed discipline as its physical counterpart and wrong to simply carry over visual artifacts from one realm into the other. But I still believe that there is a need to develop the digital side further to develop a canon of work that will help to shape and define the discipline.

The profession of architecture has adapted the digital medium over the last decade on a broader scale. Drafting has largely been replaced by digital based drawing and modeling techniques. The computer is of further great use as a simulation tool for finalizing and updating in the design pro-cess. 3D modeling software enjoys great popular-

Bilbao’s appearance has brought computer-controlled design into the popular spotlight of architecture [Image Frank Gehry]

17

ity as simulation tools to review design decisions in the design process. The geometric capabilities of the software have made it possible to construct more fluid and free form shapes and manufacture them at acceptable cost. In this sense the digital medium has expanded the space of archi-tecture through providing tools that allow realizing schemes too difficult to realize before. It has also profoundly influenced the style of architectural representation. Probably the stylistic influence has been stronger then the technical advances. The availability of the family of “blob” free form-shapes and their coverage in the media and the huge attention architects like Frank Gehry received for their work have caused a paradigmatic shift in schools and in practice. The majority of the software being used is based on direct manual input by the designer in order to shape their designs. Tools augment the intention of the designer at various levels by employing algorithmic based support in shaping complex three and two dimensional shapes and lines. But the true challenge of giving structure to the new entity of digital data has largely not been addressed by architects. The term architecture is frequently used in the engineering domain and in connection with large-scale systems. To elevate these structures beyond their utilitarian levels and make them experienceable visually is a challenge that remains unmet to date. I believe the trained abilities of architects to design with high levels of synchronous abstraction affords them the ability to address this problem.

Example of programming in design. moss - from Peter Testa’s emergentDesignGroup - Markus Kangas, Axel Kilian

18

The Impact on Architects and their Self Definition -Many architects find themselves challenged by the problems arising out of the growing importance of the digital medium. Web design has become a commodity fro Architects. The medium is used for the simulation of physical space rather then explored in its potential as a genuine space of its own. The need is there since enough people make use of this space and spend substantial amounts of time on digital networks. So how would architecture for digital space look like or better be experienced? To develop architecture it is neces-sary to explore the space at hand, to analyses its properties and qualities. But there are also more challenging approaches from architects towards the digital medium. The term “Information Architecture” is used among others by the ETH Zurich in the CAAD Lab formed under Gerhard Schmitt. It addresses the approach towards information from a spatial point of view in using the medium as a testing field for imaginary constructs or for the spatial visualization of data sets. There has been a split between those architects that stayed with the traditional profession and those that departed and joined other designers and programmers in the task of defining digital space

19

1.3. Purpose - The Importance of Programming

The biggest potential of the digital medium, its programmability, stays largely untouched by most users. Programmability is a unique property of the medium. It is a high level property that is not inher-ent in the digital information itself. But through the construction of logical statements in digital form, the possibility of a controlled execution of these statements arises. To program means to specify an exact sequence of commands for execution in a machine. In its exactness it opposes the abstract approach of design at first glance. But there is just as in design the possibility to deal with larger constructs abstractly.. Programming is a way of interfacing the medium in its own lan-guage. The programming languages are two way communication devices that allow the description of processes in human and machine readable terms. Programming is capable of describing pro-cess and structures that are arbitrarily complex and get constructed while the program is exe-cuted - it is a way to describe structure which produces form. This the point when programming can become design.

The architect is not really faced with the task of imagining the digital medium and how it could be adapted to the task of dealing with architec-tural constructs. In fact, digital space has its own architectural structure embedded in the machine descriptions. In software engineering one speaks of the architecture of a project and refers to its structural layout, its process flows and procedure and most of all to the higher level abstraction that governs its composition. This could be viewed

20

as a dividing point between physical and digital understanding of architecture. It is true that a large part of architecture is derived from and embedded in physical space. But that might also have to do with the importance of physical space in our lives and might change with the current shift of many activities into the digital domain. I do not point towards a notion of “cyberspace” since cyber-space as a mere simulation of physicality only doubles the physical reality. I point towards a genuine digital architecture that acts as a bridge between our physical presence and the abstract space of the digital medium. The development of such a cannon of work takes time to develop and will not happen unless there is a greater aware-ness among those who are considering them-selves architects and designers.

How could one address the layout and the experi-ence of digital space? Assuming that the com-plexity will increase with more information and more people involved in the structuring of digital data sets most likely in a form of the World Wide Web. One central question is which path to go in the development of the experiential qualities. The bodily conceived path that is closely related to our experience of physical space or the path of

Axes of the definitions of the thesis experi-ments

The St.Francis image shows the saint in dif-ferent stages of his live simultaneously using scaling to guide the focus of attention[St Francis 1235 Tempera on wood Church of San Francesco, Pescia, web Gallery of art at http://gallery.euroweb.hu/html/b/berlingh/index.html]

21

abstract understanding of space based on visual and or linguistic properties? I suggest to go the path of abstraction.The characteristics of digital space should be pushed forward and developed into genuine quali-ties, which expand our notion of space rather then mimic our physical experience.

1.4 Scope

In my thesis I focus on the visual properties of communication in the digital space because I believe that the visual domain can act as a common denominator in transporting a variety of models and act as the most powerful bridge between our minds, and the abstracted and sym-bolic information. The central idea of the thesis is the underlying principle of interaction through perception. The point of attention can potentially be used as input for navigation or manipulation of the information that is being read.

Evolving image space - The idea of a visual start-ing point that evolves in detail through the reading of the accessed space. The visual and organiza-tional structure evolves dynamically in accordance with the visual tracking of the user. In the process of this thesis I have experimented with a series of visual interfaces dealing with gaze tracking and dynamic visual expansion in response to attention. The results of these inqui-ries are being combined into a demo operating on digital data from the Internet that demonstrates the advantages of attention based navigation models.

The sage air defense system - real time com-puting with visual interfaces and visual feed-back system using a light gun[Image from STS035 http://web.mit.edu/STS.035/www/index.html]

The gun turret of a B-17 bomber exhibits the brutal directness of the use of the point of attention in the interaction with space[Image from http://www.ratol.fi/~tmannine/b-17/index.htm]

Both Cubism and Futurism break out of the confined perspectival image space into frac-tured imagespace

22

1.5 Necessity for a model for the enhanced handling of data

The data in digital format grows and so does the number of people using the medium. So far the complexity of experience in the medium is rather flat and mostly based on the print and animation metaphor. With the help of dynamic interaction and intelligent data browsing the handling of this content will be easier and will scale better.

The data environment is changing rapidly mainly through the expansion of the data stored and net-worked in digital format. The use of hierarchical representations is largely rendered useless in understanding the data because of its dynamic properties and size. Essentially the structures to access the data should display relational proper-ties of the data. This involves coping with the dynamic properties of the evolving data set.

Libraries have tried to cope with this problem in physical space by setting up a format of indexing the data and then adding to the existing set based on the initial index. The library model has its limita-tions both in size and in flexibility since a category has to be found for each new entity and the entity will be retrieved again through that category. In additional to this categorical access, there is also the physical access through browsing the stacks. This technique is interesting in terms of the random finding without search or the additional finding within the context of a category. But overall the library model is too unresponsive and does not scale very well since its physical accessibility is limited.

Search engines operate on symbolic language based data and come up with linear lists of matches based on their search algorithms. The sense of context and overview is very limited[screenshot by author from altavista search result - http://www.altavista.com]

In physical based information systems the dimensions of the objects and the users require the creation of navigation spaces to access the content. Their scale is limited by the human scale

23

The digital format with its ability to be dynamically adapted or updated suggests more powerful and appropriate models of interaction. Currently the main interaction is based on search engines that crawl the web spaces they can access and cata-logue the found data very much like in the library model. The problem is that the searched data is “dead” in the sense that it is not directly connected with is source which means a change or activity there will not be “dynamically” recognized.

More powerful would be a contextual visual repre-sentation of the information that allows for the par-allel perception of many data entities at a higher level without having to go into the detailed descrip-tion for accessing it. The context would allow for meaningful browsing of the data based on the visual clues. Most important is that the data could communicate with the system in dynamically orga-nizing itself in response to the user rather then the user adapting to a predefined structure.

In approaching this goal I have accumulated a series of examples and inspirations in the arts and sciences that will help to define and conceptualize the later experiments.

Final presentation slides

Fixed topologies

Loose topologies

Reactive space

Shared space

Conclusion

24

2. Background

2.1. Context

2.1.1 The Use of Image space

Historically the issue of image space has played an important part in the arts and science. The painter was in a way at the forefront of information visualization and expression of thought. In the middle ages with widespread analphabetism the image was beside speech the main information medium. The carrier of imagespace has spread out over other media. Traditionally image space has been static due to technologic limitations. With the advent of time based imagespace there was a shift towards the simulation of physical reality through photographic reproduction. The concept of using the imagespace as construct depicting a worldview was confined mostly to the arts. Per-spective introduced science into painting by estab-lishing a model for the depiction of a Cartesian space onto a two dimensional plane. The per-spectival model puts its main emphasis on the existing physical world and is therefore limited to visible things. More abstract ideas or meanings have to be encoded into symbols within the image construction.

In the early 20th century the idea of the image space was resurrected with expressionism, cubism and futurism. The image space again captures more than the perspectival construction could produce by including a multi faceted view of the world as an extension of the individual view of the painter. The tendency from early abstract

Xerox Parc’s Document Lens is a nice exam-ple of a simple distortion viewing principle applied to document navigation closely related to microfiche model.[Image George G. Robert-son and Jock D. Mackinlay,Xerox parc, Palo Alto]

Pong - one of the first video games after Star wars is a beautiful and pure example of the essentials of visual computing captured in an application. It set the path for video games but it also has genuine aesthetic derived from its digital context.[http://www.discovery.com/stories/history/toys/PONG/birthday1.html]

25

imagespaces of computing, as for instance in SAGE air-defense system radar monitoring, the shift was towards a simulation of visual aspects of physical reality. The arts were slow to adapt the extremely expensive and expert technology and therefore the examples and the innovations arose mostly out of the military, government, and corporate environments.

2.1.2 The Interface and its History in the Use of Metaphors

The interaction with computer systems was based on a symbolic; mostly text based interaction, which developed out of the earlier hard wired or punch card interfaces. The abstract space of machine language is the closest one can get to the char-acter of the machine. But its closeness to the machine structure also causes it to be alien to the untrained human. This alienation caused the development of interfaces to access the machine in more understandable terms.

The development of the interface is a history of the use of metaphors. The use of metaphors makes sense for the introduction of a new tech-nology since it ensures a certain familiarity in approaching it. It is also technological justified through the use of pre-existing technology and its incorporation into a future system that requires continuity. One major metaphor introduced with the development of the personal computer is the metaphor of the desktop. Second is the usage of the visual output as a reconfigurable keyboard, which in combination with the mouse creates an infinite possibility of button configurations. Both

The keyboard is the oldest component of the human machine interface and has stayed prac-tically unchanged since its creation in the late 19 century. It works through direct mapping of a key to functionality. It enables to construct language constructs for programming.

As machines became larger and larger the inter-face became an issue of architectural scale with similarities to libraries where the machine com-ponents are placed as the books in the shelves. The user is surrounded by the machine and has direct access to each element. [Image IBM]

The adoption of metaphors in the creation of user interfaces with the appearance of desktop computer went form text based interaction to the use of the desktop metaphor. The history of interfaces is a history of metaphors.

The visual interface pioneered by Xerox and later adapted by Apple and by Microsoft brought visual interaction into the mainstream. The interfaces are static though and lack dynamic interaction and follow tree based deci-sion models

26

the button and the desktop metaphors rely on the notion of physical objects that are mapped to certain functionality.

The desktop metaphor has proven itself in making the personal computer a mainstream device that most people can use. The limitations of the meta-phors become visible as the information space becomes more complex. There is only a certain amount of things a person’s attention can handle simultaneously. As more applications and informa-tion inputs come together in a limited visual envi-ronment such as a monitor, the task of observing and putting the events into a context becomes increasingly demanding.

The approach taken by current operating systems is mainly to hierachically organize the data in a tree file system. The shortcoming of a hierachical file system is the lack of overview and the need for categorization. Similar problems arise with graphs as with the WWW. There have been approaches to visualize data spaces using three-dimensional spaces in per-spectival view using avatars to navigate them. In these spaces the benefit of the space metaphor is far outweighed by the inefficiency in usage of the image space and the tendency to get lost within a space that does not provide the sense of orientation in relation to the body.

Example of Carpendale, Coperthwaite and Fracchia - “Extending Distortion Viewing from 2D to 3D”. An analyses of existing distortion viewing techniques in 2D is translated into 3D model. The line of site is being cleared by moving the geometric positions of the ele-ments, while keeping the topology intact

27

2.2. Related work

2.2.1 Precedents in the Field of Information Visu-alization

In the field of attention based viewing the work by Spence and Apperly from 1982 “The Bifocal Lens” and “The Hyperbolic Browser a Focus+Content technique for visualizing large hierachies” by Lamping and Rao and Leung’s “A review and Tax-onomy of Distortion-Oriented Presentation Tech-niques” provide a context for this work. Sheelagh and Carpendale provided an extension of distor-tion viewing techniques with their paper “Extend-ing Distortion viewing from 2D to 3D”. The papers deal with visually accessing data through the expansion of sub-areas of the visual display. The goal is to provide a focused view of a detail of the data set while leaving it within its context in a digital visual environment.

Apple has adopted some principles of attention based visualization in their new operating system MacOSX. The file icons do not have a fixed size anymore but adjust their size on the desktop to the room that is left on along the screen width. The transition between an open page and an icon is done smoothly through an animation in order to let the user see where the file is going. Upon moving once mouse over the icons they are scaled up around the point of attention. This is the first clear incident of a dynamic visual response in a desktop environment and with the increased general avail-able processing power probably only the start of a series of changes in the visual interfaces of the desktop computer.

Mac OS X desktop makes use of simple distor-tion viewing principles for its system of file icons. The icons are dynamically scaled to fit the width of the desktop and respond through scaling to the point of attention.

The hyperbolic tree example developed at Xerox parc and now developed as a commer-cial application by inxight. The hyperbolic tree is an attempt to work with dynamic focus prin-ciples to enhance the viewing of graphs [image inxight]

A visualization of the WWW as a giant graph using dense packing of the nodes and color coding based on countries.The limitation of line based representation become clear in terms of the clarity of the display[Hal Burch during a summer 1999 internship

28

All the examples are based on static topologies of the data set in either a hierarchical tree form in the case of “hyperbolic tree” or a grid based data set in the other examples. While this is enough in the case of existing static data sets the limitations of the static topology becomes less efficient in dynamically changing data sets like the web.

A number of dynamic examples are collected on the very informative (in year 2000) site “atlas of cyberspace” site. Luc Geradin from the Gradu-ate Institute of International Studies, implemented the project “Cyberspace geography visualization - Mapping the World-Wide Web to help people find their way in cyberspace”. which he describes as

“A method has been developed and implemented to create a representation similar to geographical maps. Toward this end, self-organizing maps (Koho-nen neural networks) and a distance matrix method have been used to produce two-dimensional maps with a distance-representing relief structure for regions of the World-Wide Web. The prototype actu-ally permits creation of maps with up to several hun-dred resources depicted on them, but the method is scalable in its spirit. These maps are dynamic and show the location of the current resource in the map. They also allow you to jump to any resource that has been mapped. They therefore provide a new percep-tion of the space we are navigating.” [Luc Giradin, http://www.girardin.org/luc//cgv/]

It is an interesting example because it uses the metaphor of a geographic map but develops it new out of the properties of the space he is trying to define. This produces a readable result that fol-lows its inner logic and at the same time relates to previous reading experiences with geographical maps. But most important it updates and adjusts itself dynamically as the space it depicts does.

A screenshot from the smartmoney website - which depicts the current trading volumes and areas of change through the use of a area subdivision and color coding.

Luc Giradins “Geography for Cyberspace” map example. The map is constructed using a metric that is established out of the analysis of the connectivity of the nodes.

UMAP example - a file structure and website visualization tool that establishes a continues surface figure to depict the relations between the file structures

29

Ben Fry Master’s thesis in the Aesthetic and computation group at the Media Laboratory is an excellent example of an innovative approach towards the handling of very large data sets and the problem of interacting with them. Ben Fry uses both image and 3D representations of data struc-tures. The goal of his work is to construct an organic entity, which exhibits response both to the manipulations of the user and the changes in the data structure it is representing. In one of his examples Ben Fry uses a book text by Mark Twain and parses the words assigning them positions in three dimensions based on the frequency of their usage. Over time a pattern of usage is being established which allows certain judgements on the structure of the text.

The thesis attempts to demonstrate how to com-bine novel and previously established principles of visual information handling into a space model that responds to the changing information context that could, in a later version, replace the desktop metaphor. The thesis is being written to combine aspects of spatial navigation with the principles of attention based visualization in an attempt to define a model for digital space.

Ben Fry’s thesis in the Aesthetics and Com-putation Group deals with the visualization of very large dataspaces as in this example with a novel by Mark Twain. Words are analyzed and placed into three-dimensions and relation-ships that allow the identification of patterns of usage in the overall body of the text are identified through connecting lines.

30

2.3. Image Space Precedents in the Arts

2.3.1. Medieval painting - Erwin Panofsky - Per-spective as symbolic form

The homogenized space of mathematics lead to a departure of the visual space and tactile space of the human perspective. Panofsky defines perspec-tive as

“. The point when turning the actual material the image is displayed on meaningless since it is turned into a window into the depicted space, where all of the space is constructed following the principles of the model. By turning the perception of space into a mathematical formula the general notion of a worldview was cropped to that of window into Euclid-ean space. This world view does not have room for the complexity of thought and expression any more that was incorporated into the non-perspectival paintings.” [Christopher Woods]

An important point in the development of painting. The completeness of the view of the world in a painting is given up in favor of a mathematical cor-rect depiction model. This was a scientific achieve-ment since for the first time it was possible to work with a visual representation that was not tied to an individual but could be correctly repro-duced through a mathematical method by every-one following the rules. One cannot underestimate this achievement for the sciences at a time when painting was in that sense still a part of science. It was however, in terms of the communicating a worldview from an individual point of view still not satisfactory. Mathias Gruenewald, a painter in the late period of Gothic painting had mastered perspective in earlier paintings but in many of his late work he chose the non-perspectival painting

31

technique over perspective. I do agree with Woods about the loss of complex-ity of thought and expression in the perspective paintings. This might be attributed to a transition phase in which the artists had to develop a new language for perspective.

But there is a fundamental difference in the spaces of pre-perspective painting as opposed to the perspective one. One is image space and the other is Cartesian space. Medieval painting funda-mentally treats everything as objects on a plane. Empty space only exists as the in between of objects on the image plane. In perspective space is homogenized and everything in it is treated equally as the voids around it since it is subject to the same rules. This gives a tremendous amount of freedom in the arrangement of the objects in image space to the medieval painter. The visual composition is the driving force behind the spatial arrangements not the absolute position in a homogenized space. This allows for very compact and highly specified arrangements in the visual space spanning multiple times and locations and unifying them in a common plane for the viewer. The scaling of the objects allows the attribution of meaning to the objects based on the judgement of the composer rather then on their position in space. Woods quotes Panofsky on the impact of perspective on the viewing of a painting

“For instance after showing how difficult it has been since the Renaissance to overcome the habit of seeing in linear perspective, Panofsky makes the point that this habit was no more an arbitrary imposi-tion upon the public eye: for the linear perspective employed by the painter is “comprehensible only for a quite specific, indeed specifically modern, sense of space, or if you will, sense of the world. (p.34)

Les très riches heures du Duc de Berr 1412-16 Illumination on parchment Musée Condé, Chantilly by the Dutch Limbourg broth-ers is taken as an example of medieval space depiction.

The painting consist of a series of viewpoints on certain objects, persons or events that are being worked out in detail and could stand alone as a composition.

If one subtracts these spots of higher detail from the image space one is, mainly left with “in between” image space - in some form the background although functions more as a con-nector between the elements.

32

World carries a heavy burden here it is more then the physical universe, it is shorthand for experience in general.” [Christopher Woods]

The world Panofsky is referring to here is a rather interesting term in connection with painting since from a contemporary view most people would refer to it probably by an image of our planet earth or an abstract idea of a global whole. But does the view from space capture the essence of the worldview? There is more then the physical presence to the world if one understands it as a complex set of relationships and dependencies that make up its political and cultural existence. But these abstract clues seldom manifest them-selves in physical form and even less often do they coexist in the same physical location that would make it proper to visualize them using a perspectival model. One way around this problem would be to bring together the different sites and objects together into a coherent singular space. But that does not solve issues of scale and conti-nuity.

2.3.2. Painting as Formulation of Thought

Woods asks the question

“Does this mean that the experience of space is somehow central to, or generative of other experi-ence?” And continues. The association of experi-ence in general with the experience of space is the first of two successive links that together connect world views to paintings (and to other concrete formulations of thought) The second link in the chain is the relationship between the experience of space and the construction of paintings.” [Christo-pher Wood]

Is there an adequate way to express the experi-

The same is true for the physical locations of the depicted events. They were not located in a cone of vision but rather spread out in physical space. The image space brings them unified together.

The composition of the single patches of con-tent into the larger piece was accomplished by the painters in creating visual relationships between the objects. The composition could also be a different one without violating the model.

These elements are the components of the overall composition and are key to the con-struction of the image .

The events depicted do not fall all within the same time frame but rather represent a col-lapsed period of time unified in the simultaneity of the image space.

33

ence of space in an image that will not be con-nected back to the person who perceived the experience? It will certainly differ from the optical image perceived on the retina of the viewer at the time since the experience includes a context of experience which will be lost in the optical image. Perspective gives equal importance to all aspects of space with a homogenous scaling of the parts based on the relative position of the viewer. According to Christopher Woods the ability and willingness to read perspective is routed in modernity when he states:

“Modernity is characterized as an epoch whose per-ception was governed by a conception of space, expressed by strict linear perspective. ...This expres-sion is evidently a simple and derivable relationship; it is a species of equivalency or mimesis. It is the mimesis of the optical impression that strikes me as the most obvious shortcoming of the perspec-tive view in this context. As the human processes the information received and analysis and weighs it according to his or her knowledge and assembles an understanding of the scenery the optical impres-sions gets altered and enriched and filtered based on the individual. The perspective allows through the composition or through its symbolic overtone for some additional information but basically has to fight against its own laws to do so.” [Christopher Wood]

The painting tries to recreate correct reconstruc-tion of the visual space that is set equal to the physical space by means of the laws of construc-tion. One can only see what is present. It seems like the construction of a mental image, a repre-sentation of the imagination of the artists world-view was shifted away from the visual imaginative level into the symbolic narrative level.

Albrecht Duerers set up for the construction of perspectival images from nature. The math-ematical construction governs the position of the lines [Albrecht Dürer, Draughts man Making a Perspective Drawing of a Woman , 1525]

Mathias Gruenewald was capable of perspec-tive image construction and used it but in many of his most important work he chose the freedom of free spatial composition over per-spective to construct his extremely rich image spaces. [ Mathias Gruenewald - Isenheimer Altar, 1515 ]

The image cosmos - the human breaks through the barriers of a previously established model to discover what lies beyond {“Cosmos - medieval painting around 13th cent.]

34

2.3.3. Link between Image Space and the World

“Again there is an initial link between “space” and “world” this time accomplished by a chiasmus that crosses the familiar term Weltanschauung with the new term Raumvorstellung.” [Christopher e. Wood ]

Raumvorstellung or the idea of space or the understanding of space is essentially the concept of space underlying the image. It is a part of the image and necessary for its reading. It also sets the framework for the construction of the image.

“The arbitrariness of direction and distance within the modern pictorial space bespeaks and confirms the indifference to direction and distance of modern intellectual space [Denkraum]’ made it perfectly cor-responds both chronologically and technically, to that stage in the development of theoretical perspective when, in the hands of Desargues, it became a gen-eral projective geometry” [Panofsky (p 70)]

The transformation of perspective into a tool for the visualization of space independent of its con-tent could be viewed as its departure from being an artistic style. Panofsky describes this transition point:

“And precisely here it becomes quite clear that aes-thetic space and theoretical space recast perceptual space in the guise of one and the same sensation: in one case that sensation is visually symbolized, in the other it appears in logical form (p 44-45)

The transition of sensation into logical form in perspective requires a different language and sen-sation that cannot be understood when isolated from the understanding of the viewer’s sensation. The symbolic visualization can be read only with knowledge of the symbols involved since they make up a part of the world at the moment of creation. I also would argue that the perspective in its logical construction had to incorporate symbol-

Color places an important role in the achieve-ment of continuous image space. It draws the composition together and counter plays the sense of depth helping to abstract the image.[Image webmuseum Paris]

The merging of symbols of text with the visuals of the image produces a rich information col-lage, where the borders between the media are fluent. [The Great Chronicle of France c. 1375 Illumination on parchment Biblothèque Nationale, Paris]

35

ism in order to make up for the loss of freedom in the arrangement of the objects.

2.3.4. On Innovation in the Arts

Panofsky general remark about the phenomena of cycles in artistic innovation are most interesting to me in the context of this thesis since I believe that digital art is undergoing a similar process in its rapid development. Panofsky says:

“When work on certain artistic problems has advanced so far that further work in the same direc-tion proceeding from the same premises, appears unlikely to bear fruit, the result is often a great recoil, or perhaps better, a reversal of direction. Such rever-sals, which are often associated with a transfer of artistic leadership to a new country or a new genre create the possibility of erecting a new edifice out of the rubble of the old; they do this precisely by abandoning what has already been achieved, that is by turning back to apparently more primitive modes of representation (P. 47)”

These more “primitive” modes of representation allow for a new approach towards a medium or task without being forced into thinking patterns of earlier creations. The primitive approach also allows the redefinition of starting parameters and thereby avoiding falling back into tracks estab-lished by previous attempts. Panofsky continues introducing Hegel:

“The Heglian notion that the historical process unfolds in a sequence of thesis, antithesis and syn-thesis appears equally valid for the development of art. For all stylistic “progress” that is, each discovery of new artistic values must first be purchased with a partial abandonment of whatever has already been achieved. Further development, then, customarily, aims at taking up anew (and from new points of view) that which was rejected in the initial onslaught, and making it useful to the altered artistic process. (P 28)”

At the transition point from non perspective to perspective space there are many examples in transitory state - like in this example of the bed scene where the foreground figures are scaled as large as the center of attention in the back although the furniture in the foreground which follows perspective is larger then the persons.[Image webmuseum Paris]

An odd example of the use of perspective in connection with a non rational event using the rational construction of perspective to depict it.[Image webmuseum Paris]

36

This is certainly true for the connection of space and perspective in the digital realm where the artistic challenge derived from the digital medium have been tackled largely through the use of physical metaphors derived from a notion of Car-tesian space which has become predominant in advances of real-time visualization.

“The distinction between Panofsky’s antique and renaissance perspective is this: the ancients pro-duced superficially false pictures because they would not abandon what they knew about the truth of perception. (p.43) This assumes that the object of representation was not the thing itself but our mental image of it (Sehbild). Indeed it is far from obvious why anyone would want to reproduce the results of vision (Indeed Wittgenstein wondered how one ever could do so).”

This is a question that has appeared to me repeat-edly while viewing attempts of visualization of architectural form in space as well as in artistic expression in digital space. What value does the use of Cartesian space have that justifies the limitations in expression brought about by its logi-cal constructs? One reason is certainly the ready availability of the Cartesian space model both as a mathematical construct and as a functioning tool implemented in the machines being used. I wonder whether it is even possible at this point to produce recoil from the mathematical space model back to a more “primitive” space out of a belief similar to the ancient belief of the truth of perception.

37

2.3.5. Cubism

Nicolas Pioch defines cubism as follows

“Cubism, highly influential visual arts Style of the 20th century that was created principally by the painters Pablo Picasso and Georges Braque in Paris between 1907 and 1914. The Cubist style empha-sized the flat, two-dimensional surface of the picture plane, rejecting the traditional techniques of per-spective, foreshortening, modeling, and chiaroscuro and refuting time-honoured theories of art as the imitation of nature. Cubist painters were not bound to copying form, texture, color, and space; instead, they presented a new reality in paintings that depicted radically fragmented objects, whose several sides were seen simultaneously.”

Cubism is the continuation of the departure from realism in the arts of the late 19th century with expressionism. It is radical in the sense that it returns to the image plane and rejects the mathe-matical derived model of perspective and replaces it with a highly subjective, individual component in the visualization model. The view of the painter is not reproducible and in that sense non scientific. But the attempt of the cubistic painters was to provide a more holistic approach in their expres-sion of the world supplying a view that is an expe-riential account of the painter externalized in form of a painting. Cubism appears in the context of this thesis since I believe that the digital medium is at a similar point in its artistic and utilitarian devel-opment as the arts were when cubism came to the foreground. Perspectival visualization models are generally available and are dominating visual expression in spatial regards. The popularity of software like flash for instance shows the need for more abstract visualization techniques, which

Le Lavabo 1912; Oil on canvas with paper and mirror collage, 51” X 35”; [Collection Vicomt-esse de Noailles, Paris]

38

allow for a variety of models for the visualization of ideas and information. What motivated cubism in the arts I can see as a pressing problem today in the realm of digital data. Solutions are necessary to deal with parallel process and multiple views of a data entity for instance in collaborative work. It is not “only” an artistic task but also a technical task to develop models to come up with a reproducible depiction method that extends perspective as cubism did it for the arts at the beginning of the 20th century.

39

3. Theory

3.1. The Visual Interface

The visual interfaces created for the thesis was tested with a head mounted gazetracker with head movement compensation in order to get insight in how the scanning process of visual perception influences the visual interaction. The difference between the use of a hand moved mouse and a gaze tracker lies in the coupling of visual reading and interaction of the gaze tracker as opposed to the mouses’ disjunctive movements. To use the eye derived point of attention allows using subconscious reaction to visual stimuli whereas the mouse requires a more conscious decision to operate. This is valuable in the case of dynamically responsive visual interfaces since the interaction time is reduced. Current difficulties include the additional processing time necessary for the eye tracking that causes concurrency problems and the accuracy of the tracking device. The use of the point of attention as the interaction device poses the question how to cope with more com-plex situation where a simple pointing action is not sufficient. I was inspired by a number of books and articles that deal with the theme of memory construction and ideas of augmenting mental pro-cesses through external representations.

3.1.1 John Maeda – “Design by Numbers”

John Maeda’s “design by numbers” language focuses at the core of digital design, the art of creating expression through programming the medium. DBN is a unique interface since it pro-

40

vides a language interface that aims at helping to teach the fundamentals of computational design rather then to provide a predefined mapping of functionality. Very few commands are defined for the user. They are divided into output methods like line draw and operational methods which govern the flow of the command sequence as for instance the repeat structure.

“DBN is both a programming environment and language. The environment provides a unified space for writing and running programs and the language introduces the basic ideas of computer programming within the context of drawing. Visual elements such as dot, line, and field are combined with the computational ideas of variables and condi-tional statements to generate images.” [John Maeda, description of DBN on dbn.media.mit.edu]

Although the approach of a programming lan-guage with graphic emphasis is not unique the design of the language and its coupling to the 100x100 pixel output is. It is an educational inter-face which helps to introduce people of various backgrounds to the world of visual expression through programming. The structure of the lan-guage becomes an integral part of the aesthetics both as inspiration and limitation of the design space. The language interface is an integral part of the design process.

3.1.2 William J. Mitchell – “The Reconfigured Eye”

William Mitchell explored the influence of digital techniques on the domain of images in the art and public perception. He shows various examples of how computational processes are used in recon-figuring the visual expression and content of the

The book “Design by numbers lays out the variety of the graphical solutions possible within the DBN language. [Design by numbers, cover - john maeda]

The DBN programming interface combines the language interface with the visual interface and thereby stresses the relation of visual output and program structure [DBN - interface - john maeda and Aesthetics and Computation Group]

41

modern image. The image is being traced in its role in history and the increasing role of manipula-tion in the production of images as an instrument of control..It becomes clear that the image can serve both a simulated reality or as a claim to truth. The book published in 1992 shows early signs of the dete-riorating of the belief in Cartesian space depiction as the sole truth in image representation. The digi-tal medium opened up new ways of manipulating the image data and became a common process part of image production.

3.1.1 David Kirsch

The work of David Kirsch - “Complementary strat-egies: why we use our hands when we think” lays out the importance of complementary strategies. In his paper he refers to physical representation and the rearrangement of physical objects to aid the thinking process. He states

“Typical organizing activities include pointing, arrang-ing the position and orientation of nearby objects, writing things down, manipulating counters, rulers or other artifacts that can encode the state of a process or simplify perception”. A key idea is in using the hands to think since “ ... Sometimes the best way to solve a cognitive problem is by adapting the world rather then adapting oneself (page 2). He concludes “Intelligent creatures amplify their cognitive abilities by adapting their environment of action to environ-ments where they can get the best results from their limited resources.” [David Kirsch p 2]

The visual environment of the digital space is usu-ally not perceived as an environment that allows for the adoption Kirsch describes for the physical environment. But especially the digital visual envi-ronment with its adaptability is ideal to augment

Cover page “The Reconfigured Eye” - William J. Mitchell - A quadtree subdivision of self repeating images configured to display itself

42

the mental processes of the human with the help of visual clues. These visual clues have to provide some form of memory to allow the human to hand over part of their mental state to the system in order to free up mental resources.

3.1.2 Marvin Minsky’s K-Lines Theory

The theory of memory and memory retrieval for the human mind has been discussed by Marvin Minsky in his book “Society of the mind.”

“Most theories of memory suggest that we learnor memorize something, some representation of that something is constructed stored and later retrieved. This raises questions like

How is information represented?How is it stored?How is it retrieved?Then, how is it used?

This paper tries to deal with all these at once. When you get an idea and want to remember it, you can create a K-Line for it. When later activated the K-Line introduces a partial mental stateresembling the one that created it.....”

How could the principle of K-lines be applied to a dynamically changing visual interface? Currently information is stored in the form of bits and can be retrieved through a hierarchical file tree. Little is in place to construct a context let alone a visual con-text for the moment that information was acquired. The model of K-lines suggested using higher level states of minds connected with a memory agent to be used in retrieval of a set of information. This could also be translated into the visual realm where one visual clue leads to the expansion of a set of further visual information connected to the initial clue. For instance this principle could

A problem P which was solved earlier with the help of a set of agents connected to kp if a similar problem like P for instance Q comes about kp is activated and helps to solve the problem that the Q agents are trying to solve.... long ago you solved a certain problem P... lets suppose a certain “learning process” caused the agens that were active then to become attached to a certain agent kP which we’ll call a K-Line...today your mind is in a different state with agents Q aroused... something in your mind suspects that Q is similar to P and acti-vates kP. Now two sets of agents are active in your mind at once: the Q-agents of your recent thoughts and the P-agents aroused by that old memory. [Marvin Minsky - Society of mind page 83]

43

be applied for the dynamic expansion of the level of detail. An initial visual clue that takes up only a small portion of visual attention could expand to a set of information if visited by the point of attention.

The visual representation of knowledge is a multi-layered problem. There are images that depict their information visually, Although the information they represent maybe conveyed through the use of symbols. This is an important point since sym-bolic visual representation does not obey the same rules as sub symbol based information. For instance symbol based information does only scale to the point the symbol can still be read. This is the case in text where readability is only possible if the text is at the symbol level. This limits the use of symbolic visual information in dynamic visual systems.

3.1.5.Alfred Ultsch “Hybrid Systems.”

Alfred Ultsch follows these problems in his paper “Hybrid Systems”. The article makes the distinc-tion between symbol-based knowledge and sub-symbol based. Ultsch defines “Hybrid systems are identified as systems which are in a transitory stage between symbol based knowledge and sub symbol based knowledge.”. The use of visual infor-mation produces hybrid systems where text and image are mixed and the boundaries between symbol based and sub symbol based information are fluent.

The different levels of the image -symbolic and visual and meaning derived from the combina-tion

44

3.1.5. Richard Held and Alan Hein “Movement-produces simulation in the development of visually guided behaviors”

The thesis claims that it is necessary to search for alternative space model beside the metaphor of physical space and its perspectival depiction. This has to do with the connection of the body with the space and the problem of the disconnection of body and space in the case of visualization of a physical space model visually. An interesting paper by Richard Held and Alan Hein “Movement-produces simulation in the development of visually guided behaviors” sets up a series of experiments to explore the connection between body move-ment and the visual perception of space.

“Full and exact adaptation to sensory rearrangement in adult humans requires movement produces sen-sory feedback. Riesen”s work suggests that this factor also operates in the development of higher mammals but he proposes that sensory associations are the prerequisites. To test these alternatives stim-ulations of the active member (A) of each of ten pairs of neonatial kittens was allowed to vary with its loco motor movements while equivalent stimulation of the second member (P) resulted from passive motion. Subsequent tests of visually guided paw placement, discrimination on a visual cliff and the blink response were normal for A but failing in P.”

This suggests the importance of a previous expe-rience in the ability to use metaphors efficiently and also the close connection between body movements and the visual input in order to pro-duce a notion of space. The use of physical space in the visualization of digital information is there-fore strongly connected to the experiences we have in physical space. A large part of information is abstract and does not have a physical represen-

45

tation and requires the translation into a physical object in order to be visualized. If the qualities of the simulated space leave their physical precedent the lack of experience of physical motion makes its understanding difficult or even useless. I would argue for the development of visual properties of a digital space that rely on abstract principles to avoid the loss of connection to the simulation and improve the efficiency of the space model in the use of abstract information.

3.1.5. Stuart K.Card, Jock D. Mackinlay, Ben Shneiderman -“Visualization Using Vision to Think”

There is a body of research related to visual inter-action and display of information. The most recent publication on the topic - “Information Visualization - Using vision to think” accumulated a series of papers on the topic. A lot of work has been done in the static display of raw data in visual form. There is a range of projects dealing with spatial displays of numerical data. The authors identify a number of topics they view as unsolved issues in the field. Among them they name

“...4. Collaborative visualizationsPeople often work on problems together. While some systems have begun to explore shared visualizations, there is much more that can be learned...

6. The perceptual analysis of dynamic information displays. Much work remains to be done to under-stand dynamic displays and how they work percep-tually.

7. Advances in the science of dynamic spatial cogni-tion There is currently a growing literature in spatial

46

cognition. This literature needs to be connected to information visualization and the spatial cognition of dynamic display explored...”

The work is intersecting in aspects of collabora-tion, dynamic display properties and the attempt to construct space. The aspect of collaboration has also been explored by a previous thesis in the SPORG group. The work “filter mediated design” by John Haymaker addressed collaborative com-munication through a shared digital object that was accessed by various expert with their indi-vidual view achieved by a filtering technique.

The theory of information visualization is strongly influenced by the work of Edward Tufte with his series of books, most important “visual expla-nations” where he defines the aspects of well designed data visualization in a paper based envi-ronment. Tufte values density of data within his examples and clarity of layout. Many of the param-eters are strongly connected to the quality of paper based displays and do not easily translate to the screen resolution. Eventually the problem of limited screen resolution will disappear with advancements of technology. A more fun-damental difference is the time factor that comes into play with time based displays. Tuftes theories are focused on the parallel evenly distributed display of information and leaves it to the person to navigate it visually.

3.1.8. Joachim Sauter “Zerseher”

Joachim Sauter from art+com, based in Berlin Germany created an installation in 1991/1992 that focused on interactivity of the computer as a

Filter Mediated Design Participants: Edith Ackermann, John Hay-maker, Paul Keel, and Kerstin Hoger Sponsor: MERL Mitsubishi Electric Research Laboratory “Design practice has evolved to rely intensively on communication and integration of diverse perspectives. This research project explores the possibilities and implications of shared vir-tual environments in the field of collaborative design.” [FMD project description]

47

medium. They describe the piece

Interactive Installation 1991/92 The observer finds himself in a museum environ-ment, a framed picture hanging on a wall. Upon coming closer, the viewer notices that exactly the spot of the picture he is looking at is changing under his gaze. Our motivation for this project was the fact that, at the end of the 1980’s, people were still look-ing at the computer primarily as a tool and not as a medium. The painter exchanged his brush for the mouse, but he used it to do almost exactly the same things that he once did on an analog basis. For us this was art with computers, not the beginning of computer art. With this installation we have thus tried to promote one of the most important media qualities of the computer, namely interactivity.

I find this piece important because it does away with the notion of goal directed interaction and replaces it with an interaction that is derived from perception. The perception of the piece provides the input for altering the piece. The interaction does not require a conscious decision by the user but only his or her presence. The painting becomes a visual piece of information in transition and is altered through the act of reading it. This thesis is inspired by these aspects of the work and tries to develop them further and develop non destructive methods for modification through perception.

3.2. Definition of Image Space

3.2.1. Set of all images displayable in one medium The digital screen uses a finite number of pixels to display its information. This limited amount of pixels plus the range of colors makes up a combi-natorial set of images that can be derived from

“Der Zerseher” an interactive installation by Joachim Sauter at art+com, Berlin. An instal-lation in which the viewers eyes are tracked to while he or she watches the image. The movement of the eyes gradually disintegrate the image leaving the mark of the visitor on the representation of the painting.

48

a display. The set of all possible images for a display device resembles the universal library that contains all possible works of literature from a given language. The problem with both sets is the overwhelming majority of elements in the set are meaningless. This is more problematic with the combinatorics of the letter symbols into words and sentences since the rules permit only certain com-binations. The image combinations are not limited since they are not judged solely by their symbolic meaning but in visual terms.To watch a TV display the noise of a no-reception signal basically means to experience a part of the set of all displayable images of that particular screen. It is a visual language that gives these signals meaning either based on the precedents of earlier images one recognizes or through the abstract meaning of these patterns within a lan-guage.

3.3. The Role of Attention

3.3.1. Perception=Interaction

Perception is directly connected with the visual point of attention. The visual point of attention is not necessarily directly connected to the mental point of attention. But there is a correlation between the mental activity and the visual point of attention in cases of visual activity like reading or viewing an image. This connection however vaguely could be strengthened through creation of a feedback loop between the viewed entity and the point of attention making the user aware of his or her visual point of attention. The powerful aspect of this principle is that the decision based

Noise image - one image of the set of all pos-sible images of this resolution.Random noise factor will most likely not pro-duce any meaning.

Another image out of the same resolution set - evoking the idea of persons sitting in a car.

An example of a very dense use of the image space for an eight dimensional display of 1000 data items in three different arrangements.[Keim and Kriegel, Using Vision to Think]

49

interaction can be at least partially be integrated into the perception part of the interaction - making the visual exchange more powerful and smooth. Similar like a good salesperson who knows the customer would predict certain moves or wishes by reading the eyes of the customer a visual feed-back system could help to establish a meaningful response of the interface based on the point of attention.

3.3.2 Gaze Tracking

A preverbal partially unconscious form of interac-tion with visual Information. Possible to be used for interest prediction and dynamic feedback The technique of gaze tracking is still a rather bulcky one due to the dependency on large hard-ware that needs to be worn close to the eyes. But the concept is a powerful one and there are pro-toypes using camera tracking and the like, which promise advancements in the usability of the sys-tems. The hope in developing the eye tracking systems in the first place was to defer cognitive processes from the movement of the eyes and thereby to be able to make assumptions about the mental point of attention as well. The use of gaze tracking in this thesis goes beyond the analysis of the eye movement by using the eye position as input in the image generation.

3.3.3 Dynamic Visual Feedback System

The combination of perception with interaction by using gaze tracking produces a dynamic visual feedback system. Important is the continuous

Commercial gaze tracker by ion - developed mainly to allow physically handicapped people to access the computer interface.

An example of a gaze tracker by ion - the image of the eye is mirrored into a camera, which tracks the eye position.

An early model of the image as a attention topology where the depth of the model equals the level of attention to the part of the image.

50

adaptation of the feedback to the users’ actions in order to prevent discontinuity in perception. Visual continuity in the flow of interaction allows for dynamic navigation patterns, which are useful for the viewing of large entities of data, where constant disruption in the perception could lead to disorientation. The change in the visual interface always allows a smooth transition between its visual states. In contrast to button based inter-faces where the pressing of a button causes the replacement of the previous display and a distinct step based navigation model is applied.Visual continuity provides a seamless navigation model where the movement on the interface is mapped to the navigation position and the visual display is both information and navigation. 3.5. Interface Development

The typewriter paradigms. In the development of interfaces the keyboard has probably played the most dominant and least flexible role. The key layout has remained mostly unchanged since the very first widely used commercial typewriter in the 19th century, based on the argument that costs and time involved in retraining the employ-ees already accustomed to the initial layout would be to high to justify the effort for a redesign. In a sense the keyboard is a paradox in the age of the computer, the fully determined universal machine. Each key maps to a specific letter that appears on the screen upon pressing the key. Essentially the process is still the same as 100 years ago with the difference that there is a vast sequence of digital intermediate processes involved compared to the mechanical mapping of the 19th century.

The visual interface in an early sketch from 1998 - the subareas are scaled based on the point of attention

Three different types of interaction - keyboard based, mouse based and eyetracker based

Interface loop between the eye and the visual display

51

The mouse is much more suited to the universal machine since its functionality is only based to a small extent on its physical mechanism. The functionality comes about through the mapping of its input onto the visual display that again provides the user with visual feedback. Ironically large parts of todays screen interfaces are based on button like visual clues very much like the keyboard but this time in digital format. Through the mouse it is therefore possible to have rapidly changing interfaces without dependency on physical distri-bution but the idea of pressing unique buttons for a mapped functionality stayed the same.

A question would be how to derive further devel-oped interfaces in the face of increasing com-plexity of the information involved. One approach would be to simplify the visual output and have a hidden intelligent system in the background make the decisions what to display. This is in a way the state of the current system where a desktop metaphor is chosen to provide initial familiarity with the system and hide internal processes from the inexperienced user. I believe this is a valid approach in the limited range of introducing first time users to an unknown system. But the prob-lem is that the metaphor does not hold up with increasing knowledge of the system but rather stands in the way of a deeper understanding of the processes involved. The model that for every key/button there is a unique or meaningful event is misguiding as well as the use of object icons as representations of collections of data. The uniqueness of events does not correspond with the simple procedure involved in re-mapping an event. A dynamic model of mapping would be

An early desktop “Monroe” calculator with a keyboard based number input which devel-oped out of the typewriter technology

Vannevar Bush and his Differential Analyser - a analog mechanical computer that operated via mechanical line tracing by hand for input and automated graph drawing for output in 1923 [corbis image collection]

Charles Babbage’s Difference Engine - one of the first mechanical computation devices built in 1821 - the mechanism is the interface and representation of the processes

52

much more desirable. The use of objects to stand for groups of bites which themselves can be read in different groupings producing different results oversimplifies the structure. It also suggests an object like simple recombinability, which is not true in most cases. Ted Nelson illustrated that in one of his early speeches about the copy and paste function of the early macintosh. He argued that the whole idea of copy and pasting involved the work-ing with parallel shreds and pieces of information. On a table like surface not loosing any of the pieces involved and going through different itera-tions of collaging. A suggestion for the rethinking of the visual inter-face and therefore also for the representation of the digital space involves the introduction of dynamic world models that are both responsive to the attention of the user and internal changes. It might prove helpful to give up the notion of a digital mapping of the actions of the users and rather find an analog expression for the intentions of a user. In that sense the navigation of three-dimensional space might prove useful.

The Dymaxion Map is the only flat map of the entire surface of the earth that reveals our planet as it really is an island in one ocean without any visible distortion of the relative shapes and sizes of the land areas, and with-out splitting any continents. [Top Image Peter’s Projection of the world, Lower Image Buckmin-ster Fuller]

ijfsbdijdbm dpoujopvt

Continuity of a data entity as opposed to hier-archical tree structure with distinct nodes

Transition of visual information from symbolic to image in the example of “cow”

55

4. Design and Decision Criteria

4.1 Themes

4.1.1. Interaction through Perception

A fundamental theme evolving out of the principles described is interaction through perception. The idea behind this theme is to use the process of perception in the navigation of the data that is being viewed. If perception tells us something about the information we are looking at, why can we then not use the process of acquiring that information visually a navigational process? This requires the analysis of the perception process while it is taking place for instance through the use of eyetracking. This data can then be used to enhance or anticipate the perception process by projecting the patterns that were discovered in the future and thereby helping the person view the data and at the same time navigate it.The connection of two processes can create prob-lems of dependency of the two processes on each other but at the same time it allows for a dynamic real time response of the viewed information in order to support the viewers intention.

4.1.2. Omnipresence of the Entire Space within the Visual Frame

Our bodily presence is set in physical space where the laws of optics restrain our natural vision to the cone of vision that projects from our eyes. This cone limits the amount of space that can be viewed at one time. The majority of the space stays invisible and has to be recreated from

Space contained in the image cone of per-spective as opposed to image space contained within the frame of perception

Density through compression of image space into the frame of perception

Occlusion - in three dimensions and in two dimensions

56

memory whenever it is not directly viewable. This has the side effect that the complexity and number of objects a person has to deal with is limited optically which helps in the processing of that information.In digital space the notion of a viewing cone might be considered as a means of limiting complexity as well. This is being done in perspectival simu-lations of physical spaces. But the more interest-ing task is to cope with the omnipresence of all information within the dimensionless set of digital data. The model follows the theme to conserve the omnipresence of all data and to render them to the screen at all times.This offers the possibility to visually access any content within the frame of perception directly through line of site and it abolishes the notion of movement as a necessity to experience space. Movement is replaced by the shift of focus.

4.1.3. Attention based Navigation

All navigational issues in the model are closely related to the point of attention. The point of attention is either represented by the mouse or preferable through an eye tracker system. As a direct input factor it is to unsteady and direct. It requires mediation to produce useful input since the normal scanning process of the human eye is very unsteady. The experiments explore different methods of mediating the point of attention and memorizing its path in the visual space. Over time the different paths affect the visual space and thereby feed back into the perception of the person viewing the space.

57

4.1.4. Problems in the Application of the three Themes

The problem of the interdependency of perception and interaction can lead to negative feedback loops that prevent successful navigation. Another problem might be the misinterpretation of the users moves in the perception stage preventing him or her to access the desired information

The presence of all information within the visual frame causes the problem of overcrowding on the limited screen area. The presentation of relational data structures poses the problem of having to be precise on the one hand but on the other hand having to be visually informative and to avoid redundancy. This is often a problem in the visual-ization of large sets of relational data. The large amount of linking lines and the rather large spac-ing distances create a level of visual complexity that is hard to read and very wasteful in the use of the visual resource. The thesis attempts to use proximity rather then line links in establishing rela-tions between the visual elements.

The use of the visual point of attention brings about the question of the validity of the use of the visual point of attention to predict the mental point of attention. As we all know the distance between them for instance in the case of staring while drift-ing off in thoughts can be quite substantial. The assumption of how to avoid this problem is to use the visual response of the image to synchronize the mental and the visual point of attention. With the visual response the visual point of attention gets stimulated and reactivated.

A diagram of the projector that connects the global data set with the individual viewer.A sequence of steps is necessary to process the data from the global set to the individual frame of perception.

An early example of distortion viewing as illus-trated by Albrecht Duerer in the proportion study of humans. The topology of the object stays the same as the geometric relations a shifted.

58

4.2. Interface Definitions

4.2.1 A Dynamic User Interface

The visual interface demonstrates dynamic prop-erties in its reaction to user interaction. The mouse movement is used as the point of attention and is constantly being considered in the visualization of the interface. The reaction of the interface is dynamic because it does not require a decisive action by the user like clicking or dragging in order to interact with it but gets continuously updated based on the position of the point of attention.The interaction is not hierachical layered but hori-zontally organized with each state of the interface being adjacent to another connected through con-tinuous transitions.

4.2.2. Distortion Viewing and Focus and Context

A technique that maps the point of attention to the geometry of the viewed space. The mapping causes the distortion of the geometry of the dis-play while leaving the topology of the image intact.

A potential expansion of distortion viewing is to make the view inflicted changes in the geometry permanent. The geometry then acts as a memory device over time and for other users. The accumu-lative effects of the distortion viewing lead to an alteration of the viewed space helping the user to determine their path. The effects of the point of attention can spread out over a wide time spec-trum allowing for a wider range of response of the geometry to the movements of the user. A quick move might only affect the immediate surround-

B!tqbdf!npefm!cftjef!uifnfubqips!pg!qiztjdbm!tqbdf!boejut!qfstqfdujwbm!efqjdujpo/!Uijt!ibtup!ep!xjui!uif!dpoofdujpo!pg!uifcpez!xjui!uif!tqbdf!boe!uifqspcmfn!pg!uif!ejtdpoofdujpo!pgcpez!boe!tqbdf!jo!uif!dbtf!pgwjtvbmj{bujpo!pg!b!qiztjdbm!tqbdfnpefm!wjtvbmmz/!Bo!joufsftujoh!qbqfscz!Sjdibse!Ifme!boe!Bmbo!Ifjo�Npwfnfou.qspevdft!tjnvmbujpojo!uif!efwfmpq

B!tqbdf!npefm!cftjefuif!nfubqips!pgqiztjdbm!tqbdf!boe!jutqfstqfdujwbm!efqjdujpo

B!tqbdf!npefm tqbdf t

wjtvbm!ufyu!dpnqsfttjpo

gpdvt!boe!!dpoufyu

The elements are repositioning themselves within the visual space - avoiding occlusion. Each element is reactive to the point of atten-tion of users viewing the space. The pattern evolving demonstrates usage of the informa-tion

Distortion viewing - focus + context

59

ings, whereas a slow and long move may influ-ence a large portion of the geometry. The same is true for the quickness of the response.

It is a technique that is avoiding the shortcomings of zooming where the context of the enlarged information is lost. It is achieved by scaling the context in order for it to fit within the displayable frame beside the enlarged detail of the of the focus point. Focus and Context is helpful in view-ing situations where it is important to have a very high level of detail and the detail requires the embedding in its surrounding context to be meaningful as in the case of maps for instance. Ramana Rao and Stuart K. Card applied the prin-ciples to large table collections. (Ramana Rao and Stuart K. Card)

4.2.3. Dynamic Expansion of the Level of Detail The expansion of the level of detail is a way to deal with visual scaling of information. With decreasing size the image information gets scaled down and thereby visually compressed. The sim-plest approach is to take visual representation of the information viewed and generate higher reso-lution versions of it. It can also mean to provide additional context for the viewed information node which was not recognizable at the lower level of detail. It has been implemented in the timeline piece, where upon the expansion of the image additional text based information is displayed. This is a very simple version of possible dynamic expansions. Symbol based information is the most difficult case since the symbolic information has a threshold of readability connected to its size.

A diagram showing the connection between the individual views of the users on the global data set and the topology of the global data set. The connections are achieved through the connectors which process the information and cast it to the individual viewer

The line of history of navigation is being dis-torted through the ongoing distortion of the grid it is based on..

60

For text this is being dealt with by representing the body of text successively with smaller portions of the text and ultimately with a keyword or title. Beyond the readability of text it would be helpful to have higher level visual grouping methods like color-coding to bring together elements into a meaningful image map.

4.2.5. Omnipresence of all Elements at all Times

The principle of omnipresence comes from the fact that a networked digital set of information is theoretical equally accessible in all its parts at all times. This means that all information is present everywhere in the space at all times. In visual terms this means that all visual representations of the information nodes are viewable providing direct line of site to the user.Omnipresence of all data within the visual frame of the user is closely related to the focus and context principle. To provide the user with an over-view and context of the entire data set gives a sense of the scope of the experience and helps to position any following navigation moves that are being performed later.

4.2.6. History of Interaction influences viewed Space

Most navigation techniques do not involve the sense of leaving a trail or altering the space that one is visiting. A possibility is to make the user more aware of his or her actions is to use the his-tory of the person’s moves to alter the visualiza-tion of the respective data. This makes it easier to return to previous data and helps to create

, >

gsbnf!pg!qfsdfqujpo

Merging of multiple point of attention to one

Principles of element interaction within the frame of reference

61

context based on the individual actions. The space stores the behavioral pattern of the user and responds based on the stored navigation data. For instance in the grid line experiment each node has a memory of how long it has been visited. This value is used in determining the speed of the expansion of the cell when a user is over it. The more frequently it is viewed the quicker is the scal-ing response. Preferred areas become accessible quicker that way and the space adapts itself to the behavioral pattern of the user.

4.2.7. Collective Property of Digital Space

The advantage of the digital space is that it puts few limitations on the amount of people sharing the same data. The goal is to create a sense of user awareness and the ability to share the experience of the data and collaborate through the digital space. In order for the application to scale well the idea of visualizing the users as nodes in the data is not necessary the best way to go. The examples explore the possibility of visual-izing other users mainly through their effect on the viewed data. For example through the scaling of a particular visual representation in accordance to the frequency of visits to it. This allows one to perceive a far greater number of users. Once one focuses onto a particular node the information of the individual users could become explicitly vis-ible to the extent that direct interaction becomes possible.

cbju

cbju

cbju

Intended rules of a complete set of visual inter-action (not implemented)

The overlapping of multiple “bait” creates inter-section between queries.

62

4.2.8. Individual Viewing of Global Sets of Data

One way of dealing with complex and unknown sets of information is to allow the individual to alter the data arrangement. As David Kirsch puts it

“ ... sometimes the best way to solve a cognitive problem is by adapting the world rather then adapting oneself (page 2). His concludes “Intelligent creatures amplify their cognitive abili ties by adapting their environment of action to environments where they can get the best results from their lim ited resources. “(page 2- Complemen tary strategies: why we use our hands to think - David Kirsch)

The key argument is the adaptation of the envi-ronment in which the individual thinks in. David Kirsch refers to the hands as the manipulator of the physical environment to help our cognition. In a digital environment the manipulation could be achieved through the eyes by giving them the ability to manipulate the visual data that is being viewed. The environment of action is in this case the visual environment. The act of viewing and accessing the data creates an individual arrange-ment of the data and therefore helps in processing it.

Scaling of pixel based information- 1 scaling - two recombination based on matching rules (not implemented)

63

4.2.9. Global Data Set Constellation is being influ-enced by the Individual Act of Viewing

In addition to the constellation achieved through the individual viewing, the personal arrangements influence the global arrangements. In this way the collective actions of the users create a representa-tion of the accumulated actions of all users. 4.3. Principles

4.3.1. Characteristics of the Proposed Model of Digital Space

The information is in digital form and content is identified by grouping sequence of digital data together into a content carrier or elements.These elements are currently fixed entities like images or text blocks but in future implementa-tions should be thought of as re-combinable.A fundamental shortcoming of the element model is the specification of the information in the form of discrete parts. As a static set this might be useful but as the set becomes larger and a lot of the information elements contain redundant informa-tion the element principle might prove to be less successful. If redundancy could be detected and novel parts in respective elements be merged into a new entity. The best model would be. To purge the discrete elements and work with a pool of visually represented clues. Problem of abstraction/representation. Text can not be compressed with-out loss of content. Can be symbolized - necessity of analysis by who by humans or by agent type processing. Symbols with meaning or abbreviation

The following list describes the characteristics of

gsbdubm!sftqpotf!pg!uif!jnbhf!tqbdf

tqbdf!vogpmet!bt!b!sftqpotf!up!uif!vtfst!bdujpot

Quadtree model connected to the point of attention of a user

Navigation generates the space that is being viewed

64

the elements in their behavior within the model and the principles of the visual model in general.

• The visual frame of perception is finite in area but can contain an infinite amount of elements.

• No element leaves the visual frame of perception.

• Elements avoid occlusion.

• Elements are expanded based upon the amount of attention they receive.

• Elements are attracted to the point of attention.

• Each user has one moving point of attention and arbitrary many stationary points of attention, also referred to as “bait”.

• The point of attention increases the visual weight of the element it is related to.

• The space is distorted through the user’s viewing.

• The distortion of the space is stored in the structure of the space and effects later viewing.

Vocabulary of Visual Reactions of the Elements to the Point of Attention.

• Scaling - sub areas are being scaled in relation to the sur rounding elements increasing visual exposure.

• Movement - increase or decrease of the speed of elements in relation to their surroundings.

• Shifts in color and or shade in relation to the surroundings.

• Merging of visual entity based on their neighbors - overcoming the principle of atomization.

The scaling of the elements as a way of response to users point of attention opens up the question of how the scaling can be accomplished Visual compression - visual compression in the form of scaling is possible to the point of the

joufsbdujpo!uispvhi!b!hmpcbm!tqbdf

vtfs!bxbsfoftt!uispvhi!ijtupsz

Sharing of one information space through the mediation of individual views for each user which are being generated based on the point of attention

The visual history of users can intersect

65

resolution available however, visual content is not necessarily obtained from the scaled version. One possible strategy is the grouping of clusters into patterns representing higher-level concepts.

The Fractal properties of visual entities.-The attention to one point of an image is dynamically creating a higher level of detail in the specific area and causing an increase in the percentage taken up by that entity in relation to the total screenspace. The increase in detail of the viewed area is achieved through the fractal subdivision of the space cells in that particular area. In terms of a grid it can be described as a binary or quad tree principle in which an area is consecutively subdivided into smaller sub areas that add up to the whole.

The Relation of the individual view to the global set of data in multi-user models - If the model is expanded to incorporate multiple users additional rules are necessary to determine the relation of the individual constellations of the users to the global data set. If each user views an individual constellation of the data elements, what proper-ties are used to determine the global layout of the elements involved? Is the visual representation of a user necessarily a visual clue or could it also be one step removed the effects of the attention on the space? Through the experiments with the simulation of multiple users in one visual space it seemed preferable to represent the other users beside the viewer not explicitly as a graphical entity but rather only through the visual reactions they trigger in the elements. With this technique it is possible to scale the model more easily

66

since the visual complexity is not increasing with the process of scaling up the number of users involved. The accumulative effects of viewing the set of information is affecting the global set. The individual viewing of each user feeds back their preferences into the global set which makes it vis-ible to all other users in the form of a collective point of attention.

These visual principles are the foundation of the visual language. They are by no means complete but they provide ways of articulating the interac-tion between the point of attention and the visual interface.

67

4.3.2. Design Criteria

The outlined characteristics were then used as design criteria for a simple model of digital space.The criteria of the model are:

• Dynamic visual response to the point of attention of the user which involves no other action then viewing the elements

• Provide a collaborative space for an arbitrary number of people for higher level collaboration and monitoring

• Dynamic expansion of the level of detail when an element is being looked at

• Visual proximity stands for elements being related.

• Data elements are self-organizing within the visual field pro-viding a presorted view

- Stationary points of attention called the baits, act as attrac-tors to the elements based on the content of the element. This allows for visual search using proximity as the matching criteria.

• Have the actions of one user affect the views of the other users

• Create the image space in a way that it has memory of the actions that took place in it - which will influence further actions. The memory acts as a mediator between many users.

• Providing long-term clues of change in addition to the imme-diate response.

These design principles were explored in the fol-lowing set of experiments that focused on parts of the criteria and subsequently builds up more layers of functionality.

68

The initially uniform grid gets distorted through the accumulative effect of attention from the point of attention of the viewer.

Line of navigation history in the cells centers of the last 20 cells visited

Distorted grid cells

4.3.3.1. Fixed Topology

The grid tracks the point of attention and each cell contains an attention counter that is constantly updated. The scaling is row and column based to keep orthogonality. I later found similar work by Y.K.Leung from Swin-burne University of Technology in his paper “A review and Taxonomy of Distortion-Oriented Presentation Techniques” featured in the col-lection Information visualization - using vision to think and by Ramana Rao and Stuart K. Card “The table lens...” from Xerox Palo Alto Research Center.

69

My implementation of the distortion grid differs from previous implementations in that it uses a simple accumulative attention tracking system where each cell has a counter which is used in determining the final proportion as opposed to the formula based approach of the earlier work. The difference is mainly that the cells have a sense of history since their expansion takes some time to restore itself and therefore the users movement pattern is visible over time. In addition I explored the notion of his-tory in the navigation of the grid. The interesting effect is that if the navigation history is drawn as geometrically based on the center of the cells vis-ited in time, the geometry changes constantly while the topology remains the same. If the same tech-nique is applied to drawing on an attention sensi-tive image space the effect is that whatever is being drawn gets constantly changed by the following actions. This effect will be explored in the later experiments. A linear, point of attention sensitive, line of elements. This model allows for the possibility to have all elements constantly present in the frame of perception and glide through them. The focus point increases the scale of the element. A click on the element stores the current attention status in the element and causes it to be more prominent among its neighbors. It can therefore be easier found visually. The topography of the elements rep-resents the viewing history. It gives an overview of the preference for elements among greater num-bers and very fast seamless access to each com-ponent. In addition to the visual scaling of the ele-ments, is a dynamic expansion of the level of text, which is being displayed once a certain threshold of attention is reached. This makes it possible, in spite of limited viewing space, to provide longer verbal description in addition to the images. The linearity of the installation gives a sense of time in the arrangement of the elements and is therefore a possible way to illustrate linear historic image data while allowing the counter posi-tion of certain time periods utilizing the points of attention.

The drawing example where the line changes the geometry of the previous lines drawn. This is an interesting effect since each addition changes the arrangement from a perceived point of view and in this case actually changes the numerical represented geometry as well.

Another example of an attention grid, which works with an algorithm, based distortion of the cells and a superimposed accumulative memory for attention in each cell. This allows quick individual access to all cells but also incorporates the advantages of the memory of attention of the previous examples.

A visual variation using diagonals in the grid cells which makes the changes in proportions more clearly visible through curving patterns of the diagonals of neighboring cells.

70

Expanded subareas of the painting

Additional detail in text form

A color-based attention grid. The technique is the same as in the grid example. The differ-ence is the way it is visualized. In a neutral state the image space is unstructured and uni-colored. Upon acting in it the changing grid cells influence the color of the cells causing the grid structure to emerge out of the homogeneous color space.

An example using “Les tres riches heures”, a French painting from the sixteenth hundred century and applying the fluid transition of the attention grid to it. It is interesting to see the spatial model of the medieval image come together with the dynamic distortion of the grid. The relative scaling of the persons and objects in the original image is heightened by watching it through the grid setup..

71

provides an overview over the set of images with dynamic scaling of the images within the context. This principle of grid distortion was published by Y.K.Leung from Swinburne University of Technol-ogy who applied it to maps among others.I independently developed this method again with the idea of distortion of Cartesian grids to allow a response to the users interaction with an image. The advantage of the grid-based distortion is that the topology of the viewed data set remains stable whereas the geometry the visualization is subject to the viewing based distortion.The piece uses the painting “Les tres heures” in demonstrating the similarity of distortion viewing principles with the medieval image space with its fluid transition between the scales of objects and people in the composition and therefore also in the representation of space. The image reveals additional text based information upon certain expansion of a subarea. The use for Map naviga-tion is a striking one as previously published by Y.K. Leung since it allows the user to browse without loosing context when shifting to larger scales. I applied the principles to the idea of a grid memory where certain cells are given information and can be revisited for retrieval of this information later. The navigation is accomplished through the point of attention. A last example is the use as an image library where the previewing and scanning of large bodies of images can be done fast and efficiently

An example of the application to a set of aerial photography from the greater Boston area shows the focus tile in the center and the context compressed around it.

The use of the grid as a memory field where each field can be marked and linked to a certain information allowing to give the infor-mation spatial connotation for easier later retrieval.

An example showing the use of the grid with an image library of 100 images from a televi-sion series “message from uncle”. Photos are easy identifiable at small sizes and can then be examined at larger scales while leaving them in their topological position.

72

Text blocks appear when the point of focus has reached a threshold level.

Composite curve of attention

Individual elements scaled based on the composite magnitude of attention.

The linear arrangements of image elements is responsive to the point of attention.

73

The diagram shows how the composite curve is mapped to the x dimension of the elements. Memory of attention is expressed in increased size.

Level of attention of one element

Point of focus peak

Gaussian bell curves for each element plus a mobile point of focus one make up the attention memory. The focus curve scales with length of time over one element. Upon clicking on an element the current focus height gets transferred to the element curve - storing the attention. This causes the element to be larger then its neighbors even if the point of focus moves on.

1 2 3 4 5 6 7 8 9 10

The diagram shows how the composite curve is mapped to the x dimension of the elements. Memory of attention is expressed in increased size.

12

3

4

56

7 89 10

An early sketch of the piece showing 100 elements in different colors and their compos-ite attention curve. The dots show previously clicked elements.

A linear, point of attention sensitive, line of ele-ments. The model allows to have all elements constantly present in the frame of perception and glide through them. The focus point increases the scale of the element. A click on the element stores the current attention status in the element and causes it to be more prominent among its neigh-bors. It can therefore be easier found visually. The topography of the elements represents the viewing history. It gives an overview of the preference for elements among greater numbers and very fast seamless access to each of it. In addition to the visual scaling of the elements comes a dynamic expansion of the level of text which is being displayed at a certain degree of attention if an element is reached. This makes it possible ,in spite of limited viewing space, to provide longer verbal description in addition to the image. The linearity of the installation gives a sense of time in the arrangement of the elements and is therefore a possible way to illustrate linear historic image data while allowing the counter position of certain time periods by using the point of attention.

Peak from focus

Stored peak

Stored peak

Peak

74

Point of attention

Point of attention

Swarm implementation - changing topology based on the attracting forces of each element. The space sorts itself visually through the introduction of several points of attention.

75

4.3.3.2. Loose Topology.

After the initial series of experiments it became clear that the limitation of the fixed topology out-weighs its advantages if applied to more general sets of information. For an image library with very similar elements the fixed topology with neighbor relations makes sense, however, for evolving sets of information with diverse properties it seemed more plausible to come up with a method that allowed for reconfiguration of the neighborhood relations.

The following series of applets dealt with the possibility of allowing not only for a dynamic geo-metric response but also for a topological one, meaning the position of the elements relative to each other is not fixed anymore. This allows for a greater variety of responses to the activity of the user and allows for a recombination of the data. The first set used simple autonomous moving ele-ments that are attracted to the point of attention of one or multiple users. The example shown here works on a set of dots distributed over the color spectrum. The combined actions of two points of attention each positioned at the opposite end of the color spectrum sorts the 10000 dots dynami-cally in a visual way.

Distance A Distance B

Braitenberg vehicle

Eye B

Eye A

Point of focus

Direction of movement

Direction of aim

The flock starts out with a random distribution of 10000 elements with a color distribution over the whole spectrum. Each element has a direc-tional tendency based on its color towards one of two points of attention.

As time evolves the flock develop patterns around the points of attention based on their color. This allows making visual assumptions on the position of elements of a certain color range.

Clicks of the mouse allows the inversion of the forces and repel all the elements to the perimeter.

76

An implementation of the binary tree model with 100 hundred point of attention simulations accessing the content carriers. The lines are showing the selected next node

Binary subdivision tree diagram - the initial image space is recursively sub-divided with alternating horizontal and vertical orientation of the cutting line. This produces an adaptive level of detail for the sub cells based on the maximum amount of element in each sub cell. Once the threshold is passed a subdivision takes place and all ele-ments are copied in the respective sub-space. This method is frequently used in surface rendering optimizations for subdividing the areas to be shaded into smaller sub areas adapted to the level of detail of the object area. An imple-mentation of the binary tree model with 100 hundred point of attention simu-lations accessing the content carriers. The lines are showing the selected next node.

Maximum element per cell 10

Before subdivision

17

9

9

17

2

15

11 4

6 5

77

Integration of the Distortion Viewing - The next step was to integrate the distortion view-ing principles back into the model. The fifth exam-ple shows an image space with elements. The ele-ments are responsive to the point of attention by scaling their display size. The dynamic response is at the level of the element. Each element rear-ranges its position in the topology based on attrac-tion to the point of attention and autonomous movement. The model avoids occlusion of ele-ments to allow direct line of site to each element at all times. The model contains a binary tree space subdivision model to speed up collision detection for large number of elements. In addi-tion, the dynamic space partitioning is intended to be used as a space-scaling device where each sub cell is being scaled up by attention taking a greater percentage of the overall screen area. The elements therefore get scaled based on the sub cell they are contained in.

An early sketch of the intended configuration of the image space drawn by hand. Showing the elements in a frame of perception plus representations of users, which are now only represented through their point of attention. The colored areas stand for the idea of “bait” in the image space for the grouping of elements and the retrieval of information through visual “keywords”.

The implementation of the piece required the use of a fast collision detection technique to allow the scalability of the model without having exponentially increasing collision checking time in the computation. This goal is achieved by using a binary tree space subdivi-sion that dynamically adapts to the density of elements in the area by creating subdivisions until the maximum density threshold is passed.

Both trees are displayed with their elements turned of. The first is a 15 level 10 element example. The second one is a 20 level sub-division with 5000 elements in it. The compu-tation time increases linear with increase of elements.

78

The density of the image space at a total of 10000 with the binary subdivision grid turned on. The pattern of the grid reveals the density of the elements.

79

As the number of elements increases, the density increases. Each element competes for maximum size in the given display area and therefore also for exposure to attention. As the elements collide they limit each other’s growth. If the point of atten-tion hits an element its growth rate increases and it will grow faster then its neighbors. This principle was applied to very simple networked versions where 100 simulated users interact with several hundred elements and their distribution of attention. The goal is to see clusters of elements around groups of attention and use the space as a mediator between the points of attention of many users. Instead of displaying each user explicitly the space acts as a mediator through the relative scaling of the elements to each other.

A distributed model where the multiple users are logged in the same shared dataspace. Each user has their own view of the dataspace. The presence of the other users is visible through the mapping of their point of attention into the space. This does affect the scaling relations of the elements to each other.

The interface with about 40 elements intro-duced and as they are distributing themselves across the visual frame.

The same interface with 1000 elements show the scalability of the system also visible is the dynamic adaptation of the subdivision of the space based on the density of the space.

The same interface with 1000 elements show the scalability of the system also visible is the dynamic adaptation of the subdivision of the space based on the density of the space.

The space with 10000 elements and the back-ground texture represents the density mapped to darkness of color.

Spatial division

80

The image space memory implementation after approximately 10 minutes with the point of atten-tion signified by the circle on the top right. The space has deformed itself according to the movements of the user. The form of the line bunch reflects the actions that took place over time in the imagespace and therefore memo-rizes them.

81

4.3.3.3. Reactive Space - Visual Memory

The following variations focus on the aspect of image space long-term memory. So far the elements demonstrate an immediate dynamic response to the point of attention. By giving the image space memory of its state it becomes pos-sible to have long-term reactions. Essentially this means that space starts to move based on atten-tion and its effects influence its state. The closest comparison I could think of is a geologic land-scape that is being shaped by external and inter-nal forces over time. One standing on such a landscape is unlikely to experience any immediate change. The landscape is however, in constant flux at different scales of time and also shaped by the very people that look at it. The implementation is a pointer model where each pointer has a tendency to aim at the point of attention. The quickness of the response depends on the distance from the point of atten-tion. The speed of the change is exponential to its distance to the point of attention.

Pointers

Point of focus

Direction of aim

Relative movement

Movement based on lengthand direction

20 seconds

1 min

2 min

4 min

8 min

82

The image space as an implementation of “space pointers” which react to the position of “bait” and the point of attention of users by ori-enting themselves toward them at a distance dependent speed. This creates a temporary memory effect in the image space since the pointers readjust themselves slowly.

83

A combination of the distortion viewing principle with the space memory principle. The space reacts immediately to the shift of attention but also has long-term memory of the accumulated actions. Through this layering of properties the directness of access to all parts of the space is obtained as well as the shaping of the space through the actions of all users interacting with it.

The purpose of this is to be able to scale up the number of elements and users without reducing the visual clarity of the display since the other user are experienced as mediating the changes in the space. The changes in the space cause a cluster-ing or dispersion of the elements in certain areas creating various foci of attention. These clusters can be viewed as areas of higher activity where more information flows in order to get exposure to attention.

Direction of movement is being read from the closest space pointers

BaitPoint of atten-

Stages in the navigation in the pointer space. The level of interaction is one step removed from the user since it is mediated through the pointer space.

A pattern of movement is viewable in the pointer space and the elements have config-ured themselves around it.

A large scale example of the pointers in a grid fashion which creates interesting more effects between the pointer lines giving patterns of lines.

Flow effects in the field of pointers caused through distance based latency of the reaction to the user’s point of attention.

84

An implementation which overlays the direct view over the distorted view giving detail and overview at the same time.

85

As the last stage of the development I super-imposed distortion-viewing techniques over the space memory principle.

The use of the distortion model is combined with distortion principles by using a hyperbolic scaling technique for the element space. The benefit of combining the distortion viewing with the space pointing memory is useful since it allows rapid access to each part of the space but also creates a history in the image space over time.

The incorporation of the bait principle adds the notion of Stationary points of attention to the navigation. This piece is notable for its depth of image space and intuitive navigation.

A different state in the navigation with the ele-ments spread out more sparsely connected to the bait.

Next stage of the process when the point of attention is shifted from the main area to the right.

An implementation which overlays the direct view over the distorted view giving detail and overview at the same time.

point of attention pointers

pointers

pointers

pointers

pointers

Distance dependent reading into the pointer orientation

elements

elements

elements read their direction of the closest pointer in space

imag

e ou

tput

inuences the point of attention

The layering of the interaction between interface and the user represented as the point of attention.

86

A visual browser displays a fixed topology in a dynamic geometric constellation. All nodes have attention memory and are scaled while being viewed.

The nodes are grouped radially around their parent and can rear-range through rotation. Adjacency signifies the connection rather then line connections that would take up more space.

Sketch of the visual browser developing the idea of adjacency of the linked pages in expressing the sequence of pages

87

Navigation of large databases through visual his-tory. The last piece tries to connect the notion of a visual accessible history to the principles of atten-tion based navigation. The node structure expands similar to infinite streams always one level beyond the users current position and thereby creating on the fly the space that can be navigated. Once accessed all nodes stay visible and their scale represents the amount of attention spent on them over time. Thereby the user gets a visual feedback of both the paths he or she has taken and also a sense of the relative importance of each of the steps taken in time. The application was used with the WWW as an example graph. Each page is being dynamically uploaded to its maximum size and a screenshot is taken in the background and the image is scaled to the node size. The html page is parsed for links and for each link a child is created ahead of time if the node has reached a certain level of attention.

It uses a graph-based navigation where each navi-gation node is attached peripherally to its parents and is able to slide along the edges and rotate its orientation based on the center of its parent.

The development of the navigation starting from one node to a dynamically expanding graph of navigational nodes into a continuous node space where each node is tangential connected to its children and parents. This allows continuous movement of the point of attention along the navigation path while each of the nodes expands while it is being visited and keeps a memory of the time it is accessed in terms of its scale.

The navigational nodes with a direct live connection to the WWW with the MIT page as a starting point.

91

5. Application

5.1. Application Scenarios

5.1.1. Image Library

A very direct implementation of the described prin-ciples is the creation of an interface for the visual browsing of large sets of images in a grid like fashion. The user accesses the entire set of the collection through a visual interface based on two dimensional mapping, and browse through the visual data directly with dynamic scaling of the viewed images.

5.1.2. Visual History and User and Context Aware-ness for Web Browsing

In a web navigation context the application could be the use of the visual layout of a navigation history and the introduction of user awareness a through relative scaling of the visual elements relative to the amount of users looking at the con-tent. The visual structure will help to locate previ-ously accessed data through the visual context and make it possible to read content in parallel rather than strictly sequentially.

5.1.3. Way Finding Composite Maps for Individual-ized Way Descriptions

An expansion of the individualized viewing space would be the compositing of heterogeneous visual data into individual data entities. One example could be the construction of demand based way finding maps that display experience as distorted

A sketch of applying information collaging to the realm of maps - based on the level of detail necessary for navigation the maps of dif-ferent scales and types are collaged together to provide a narrative of information from A to B - with detail based on the perceived difficulty of orientation.

92

“way finding” information. This would mean to col-lage a series of maps based on the scale of travel and destination into a visually continuous mapped representation from a starting point of lets say, Zurich to Boston taking into account the scale of movement involved in the different transportation devices. The 7 hour plane trip might take a lot of time and distance of the total trip but very little attention in terms of navigation and may therefore be only marginally represented whereas the last 200 yards to the unknown house of a friend would have to be very detailed.

5.1.4 Collaborative Work Environment

The advantage of digital space is being applied to the task of collaborative working environment. The visual space is useful in providing a visual context of the data elements shared by a group of people involved in the collaboration. The tool allows users to get an overview of the current topics that are being worked on and associate the people with the projects they are currently working on through visual relations. It is possible to visually monitor the progress of a project through the area and constellation of the elements it contains. The digi-tal space allows for physical mobility while keep-ing a continuous presence within the digital space allowing for the expansion of the personal work-space overlaid on representations of physical its locations. The development of applications that-implement actual working tools for collaborative features lies outside the scope of this thesis and will be explored in a continuation of the project following this thesis.

Sketches - the collaborative space could serves both as monitoring and interaction device. Proximity is expression of relations.

Sketch - with the expansion of a subarea the data and person context becomes visible.

Sketch - in the area of focus the adjacent ele-ments with similar color coding expand.

93

5.1.4.1. Collaborative Aspects of Digital Space

The notion of the digital set as a spatial entity poses the question of the qualities of this spatial entity in regard to interaction between different users within that space.The chat room is a very widespread installation serving the purpose of multi-user exchange based on a language space. The properties of the lan-guage space are that there is a locality to the typed language exchange in the form of a typing window in which a linear sequence of the con-versations is displayed, headed with the identifica-tion synonym of each user contribution. Based on these rudimentary properties it is quite surprising how successful this principle is. Other examples of vrml worlds like for instance alpha world have been developed with a simulation of physical Car-tesian space in mind. Avatars represent the users and their movement in a Cartesian space reflects their position within the world. In the case of alpha world the chat room model is still intact in parallel though and most people end up leaving their ava-tars motionless and use the chat room solely. Which makes sense since the visual world that is presented to them through the simulation repre-sents little more then clues of who is present and the majority of the exchange is being received through language. How can the visual component of multi-user exchange be addressed more suc-cessfully? There is certainly the option of aiming at higher level of simulation to try to achieve a better match with the visual clues of the physical world. An alternative would be to focus on the genuine qualities of the digital media on which the exchange is based on and how content is

94

represented through its visual components. The size of the individual component is scaled to allow the complete content to fit within the visual frame of reference. The users who are accessing the space are not necessarily represented through a visual clue. Their influence on the space through their point of attention is mapped onto the space through the memory of the space. The arrange-ment of the data content on the image space is different for each user based on their point of attention. The distortion viewing principle allows them to almost instantaneously access each part of the information through a direct line of sight.

The content carriers are sensitive to points of attention and in another dimension sensitive to the content of other information carriers.Matching or similar related content will attract each other and result in proximity in the image space. The same is true for the point of attention. All information content has a desire to be seen. This interest is expressed through the tendency of the information content to drift towards the point of attention of the users. This results in a response of the information space even when the point of attention is static since the elements are reorga-nizing themselves based on the attraction forces of the stationary and mobile points of attention.

97

6. Discussion

6.1. Architecture and the Digital Medium

Architects have early on been among those who have most appreciated the digital medium. If I say Architects I do not mean Architects in general but a few visionaries who stepped outside their professional boundaries and took on the medium. This is probably true for other professional fields as well but in few other areas there has been so much opposition to the medium by those who have refused to embrace the medium. Even today there is a lot of suspicion and irony in the way professionals in the field address the medium. This has a lot to do with personal insecurities towards the medium and what it is capable of doing but also a general belief that architecture is tied to the physical space and should not be too far removed from it.

The other aspect is the aspect of design. From a professional viewpoint the medium has been integrated fairly well. The drafting has almost disappeared making way for the digital based drawing. 3D modeling is further dominating the visualization area, and architects are in general making full use of the available techniques. Architecture has a lot to do with higher level abstraction and structuring in order to be able to handle very large amounts of detail at a design level. This quality is in many ways similar to the development of software. The difference lies in the digital space that the structure is being placed into. Of course, architecture is not limited to these aspects, in particular not in the process of building

98

and detailing, which one might argue is the main part of architecture.

6.2. Architecture and Image Space

A similar process to the integration of the digital medium today was taking place in the renaissance with the development of perspective. Perspective was introducing a scientific method into the repre-sentation of space and objects. Perspective was innovative over previous depiction methods in the sense that it did not distinguish between object and background.

The reason this thesis is being written in the Archi-tecture department is that I believe Architecture has developed out of coping with the demands of physical space and the people inhabiting it. It is a multi-layered discipline with one of the longest tra-ditions of mankind. It stems from the fundamental need for shelter and the development of society and culture and its expression through physical form. The physical aspect of Architecture has been the dominant and persistent factor in the evaluation of architecture. The physical form per-sists over time, shapes our everyday lives and is being used and therefore cared for by the people inhabiting it. This is of course not generally true but certainly for the pieces that have outlasted time and shape the history of the discipline. Archi-tecture necessitates individual interpretation since it is necessary to experience it in person through ones own body to access its qualities. Physical space is its medium and movement in space through time delineates its experiential quality. The experience of architecture is through the bodily

Thesis of Axel Kilian 1998 - The cone of vision is applied as a scaling force to the viewed object and extends it. The extension is not vis-ible from the point of view due to the perspec-tive. Space expands without visual alteration

99

senses with a large emphasizes on vision. The body gives scale to the spaces both through the point of vision and through the relative size of things to the body.

Visual media has a long history in the develop-ment of architectural structures. The plan as a medium to represent the building for visualization both for reading the building and constructing it has developed its own language. Perspective of the Renaissance has created a method for objectively rendering visual constructions as a standardized representation for construction and representation. The use of the image for architec-ture has been mostly for reasons of representa-tion. The image is a literal representation for the architecture being depicted. The image being a plane however, cannot substitute for the physical spatial construct of architecture. The perspective comes closes to substituting the physical con-struct since it constructs the optical image pro-jected on the image plane. The image is singular view of an infinite amount of views in the archi-tectural space it is representing. Also the image does not capture the bodily sensorial experience of the physical space. The viewer is capable of reconstructing the space from an image based on prior experiences of similar spaces. The image becomes a substitute reality capable of invoking the idea of the space depicted.

Image Space in the Digital Media.- With the devel-opment of computing and the existence of real time computing in the 1950’s the images space has shifted from being a purely representational medium to one capable of simulation. The image

Markus Nowak - builds pure visual spaces from spatial composition. The images are derived from digital three dimensional models and rendered onto an image plane.

100

can be generated in real time based on the inter-action with a user providing a visual input similar to that being perceived in a physical space. Simu-lation creates parallel space representation of the physical space. Imagespace in the digital medium reaches beyond simulation - the medium has its own Reality. Prof. John Maeda’s work has pioneered the use of the digital medium as its own aesthetic and expressionist entity in exploring visual inter-activity through movement and color. “Timepaint” is a variation of the common drawing interface. By adding the dimension of time to the drawing experience, it differs fundamentally from its static cousins. The act of drawing is reinterpreted from the perspective of time. In a work for a calender series the cursor is transformed into a flame that lights up the days of the week and lets them burst like firework. Play-ful interactivity is the theme of the piece. It is freed from its utilitarian directness and beauty unfolds in the visual reactions.

An important contribution towards the develop-ment of imagespace in the digital media is the work of Casey Reas from the Aesthetics and Computation Group headed by Prof. John Maeda. Casey Reas has constructed a series of visual machines that operate on the basis of visual rela-tions, that establish the base for the movement and changes in the visual display. The visual space is an abstract and genuine one that takes the visual properties directly and gives them func-tionality within its self-defined system.

Golan Levin from the Aesthetics and Computation

Casey Reas studies into “visual machines” is an interesting approach towards the construc-tion of a digital image space since its incorpo-rates visual properties into the workings of a mechanism that is purely visual (it includes also sound) and not the use of metaphors from the physical cousins

Timepaint - a piece by Prof. Maeda that explores the possibility of the integration of time into a painting interface and thereby adds genuinely digital property to the Interface

One of the calender pieces - the cursor is actuator as a flame it lights up the days of the week which burst as number firework into the screenspacep

101

Group developed a series of pieces that process gesture input and produce both sound and visual output in response. The input is not done in sym-bolic form or through a keyboard but through the interpretation of the path of movement of the user’s hand. The interaction is therefore based on its own visual language that is used to interpret the gesture and direct the generation of sound and image.

6.3. On Distance

Physical space is defined through distances between all points contained within it. The space time continuum determines that to cover any distance time is involved in completing the move. There cannot be physical omnipresence. This defines architecture very strongly since the physical construct is experienced in a temporal sequence cannot be grasped at once in its entity. It takes time to cover distance and the space evolves. Scale becomes apparent through the passing of time compared to the experience of the moving body.In digital space the same does not hold true. Of course the physical laws that describe the physi-cal world are not undone in the digital world but they act at a different level. The experiential level is modifiable since it does not rely on the laws of physic but merely implements on top of them. Physical distance is almost irrelevant in digital space aside from running times through. Omni-presence is a possibility if one accepts the repre-sentation in the digital reality as a presence. These are properties highlight a few of the differences of digital and physical space. Omnipresence

The image “score” is recording the gesture input of the user and is read in by the system to generate the sound composition

103

In physical space the presence of many people causes the problem of assembly spaces. How does a grouping of many people look like in digital space? Both the scale and the time of such pres-ences in the digital realm are not limited. There are no physical constraints to a dimensionless space.The constraints come into play if one considers how to make the persons involved aware of their presence and how this can be visualized. The dimensionless space would suggest a collapse of all information into a point, but the point would not be visible. Giving the space dimensions in order to visualize it opens up the question what dimen-sions to use. One choice is the use of the physical space metaphor that is being widely used. It offers the advantage of being able to connect to the pre-sented information by using prior experience. The prior acquired information is useful up to a point where the example does not leave the area of the metaphor. Once the limits of the metaphor are reached the metaphor becomes confusing since it suggests different readings then those used to explain the event.

6.4. Limitations of the Physical Metaphor

Presence - our presence is a continuum in physi-cal space. We perceive the presence of others through their appearance or through the traces they leave in physical space or through the works they leave behind. Digital presence is not depen-dent on continuity. The presence can split and be disrupted without endangering the person respon-sible for it.

On retina display technique by microvision. The image is directly projected onto the retina instead of viewed from a screen device. This allows the seamless integration of digital images into the image derived from physical space.

104

7. Conclusion 7.1. Focus on the Development of Visual Navi-gation Models

I believe in the possibility and validity of an architecture of digital space. One also talks about the architecture of the machine. The term applies to the inherent structure of the machine that is often referred to as architecture. It is to be understood as a hierachy or as a layout of the components of the system. However, architecture in broad terms reaches beyond the scope of engineering systems in the aspect that it incorporates people and a sense of space that is not driven by engineering concerns. The increasing demand projected upon the digital realm through the popularity of the internet has changed the expectations of the medium. It is developing its own qualities out of the dynamics of popularity and its own underlying properties. Currently there is a lack of use of the medium in its genuine form although it has substantially changed over the last years with the appearance programs like flash. Flash as a popular portable application helped to spread interactive graphics to a large audience. The direction it could go to is to have a larger audience exposed to and knowledgable of programming. John Maeda’s DBN language is pointing in the right direction and hopefully will gain enough influence to create awareness among designers towards addressing the need to explore the structure of the medium.The medium has defining properties. These could be listed as being dimensionless, immaterial, electronically based and theoretical timeless. The

105

limitations of these properties have created great deals of frustration when they were encountered with the expectations of finding an unconstraint replication of physical space in the digital medium. I believe it is worth the effort to try to develop the properties of the medium into an framework and platform for possible development of meaningful constructs in digital space. What perspective achieved in the Renaissance was establishing a reproducible method for the depiction of physical space that was not dependent on the individuals’ perception anymore. The depiction was rather based on a mathematical model of projection. To me, what is necessary today is a model that allows for the incorporation of the properties of digital space into a depiction model. An important aspect is the variable number of people that can participate in the space model, I outlined in this thesis. This suggests the inappropriateness of a single point of view perspectival model. It is necessary to find reproducible methods for mediating between the views of many to construct an individual view based on a collective model. Currently most software applications focus on the visual outcome of their operation. The model that can serve digital space has to focus more on the underlying structure and make it part of the experience. The thesis concludes with a suggestion to a further focus point on the development of visual navigation techniques to anticipate increased demand in the information-processing realm. As devices become smaller and more omnipresent the visual accessing and interaction will become more important and the acceptance of keyboard input be less due to space constraints and time constraints.

106

I believe we are entering a post-perspectival era in which other depiction models will take over from the dominating perspectival one. The new models will have to bridge between the individual “painter’s” view of medieval times and the reproducible “scientific” view of modernity in creating an integral model which adapts itself to the users and their intent.

7.2. Future development

7.2.1. Hardware enhancements

Gaze trackers are quite humble for the moment and do not perform in the intuitive and reliable way our vision works. Future gaze tracking hopefully will work without the necessity for special devices [Hermann Helmholtz institute Berlin example]The methods of interfacing the digital realm span between two extremes. One is the complete inte-gration of the interface into the body with on retina displays that augment vision and expand the phys-ical-visual domain with the digital depicition. The other extreme is to reinsert the properties of digital space back into the physical realm and operate the interface at an architectural scale. In this case the hardware would be the building itself the inter-face would be undirected and ubiquitous. A room could be an input device, a person a point of atten-tion. The response could be visual through the surfaces of the room or physical through a kinetic response of the surfaces.I prefer to think of the model as a model that tries to close in on the min, which approaches cognitive structures rather then physical structures.

107

7.2.2. Visual communication

The display has great limitations in being a rather inflexible, stationary object that does not travel well. Other retina display techniques are being developed and might prove useful to enhance and augment the users’ view and lead to a bridge between the abstract digital space and the physi-cal one. One can think of an augmented vision via the eye displaying relevant additional information in spatial or abstract format into the view of a person. This ides is nothing new but the aspect of integrating it seamless into the visual field and utilizing it for interaction through the point of atten-tion opens up a new dimension for interaction and navigation.

7.2.3. Closing remarks

The Thesis is a continuation of my previous thesis in Architecture at the University of the Arts, Berlin in 1998 - which dealt with the simulation of digital space. With the frustration of the available repre-sentations grew the desire to develop a unique and genuine language for the medium. Although the thesis has not achieved that goal, I believe it is valid as an inspiration and motivation to others to push forward the limits of the perception of the digital realm and to search for genuine forms of expressions for the digital medium. I believe in the development of the digital medium as a space of growing importance, not in direct competition with physical space but rather as an augmentation of our physical presence geared towards our cogni-tive capabilities and abstract visual senses.

108

It is important to teach the fundamentals of the medium in order to expand its understanding and I hope that this will be the continuation of this thesis .

Prof. John Maeda and the Aesthetics and Com-putation Group have taken on that challenge for several years now and the examples of the “funda-mentals of computational design” show the prog-ress.

110

- Gary C Borchardt – Causal Reconstruction, AI Memo 1453 February, 1993

- David Bordwell – The Viewers activity, “Nav-igation in the ction Film” 1988 - Rodney A. Brooks

Intelligence without representation, AI 47 (1991) 139-159, Elsevier

- Stuart K. Card, Jock D. Mackinlay, Ben Shneiderman

Information Visualization – Using vision to think – Morgan Kaufmann Publishers, Inc., 1999

- Robert L. Fantz – The origin of form perception,

- C. R. Gallistel and Rochel Gelman – Preverbal and verbal counting and computa-tion, Cognition, 44 (1992) 43-74

- Gad Geiger, Jerome Y. Lettvin and Olga Zegarra-Moran – Task-determined strategies of visual process, Cognitive Brain Research (1992) 39-52

- Luc Giradin – Mapping the virtual geography of the World-Wide-Web

- Richard Held and Alan Hein – Movement-produced stimulation in the devel-opment of visually guided behavior, , Journal of Comparative Psychology (1963) Vol. 56, No. 5, 872-876

- Linda Hermer, Elisabeth Spelke – Modularity and development: the case of spatial reorientation , Cognition 61 (1996) 195-232

REFERENCES:

All Images are by the author unless otherwise marked

111

- Lawrence A. Hirschfeld - Authors Susan Carey and Elisabeth Spelke–

Mapping the Mind – Domain specity in cognition and culture, Cambridge Universtiy Press 1994

- Hansjoerg Klock & Joachim M. Buhmann – Data Visualization by Multidimensional Scaling: A Deterministic Annealing Approach, Paper University of Bonn, Ger-many June 5, 1997

- John Kominek – Advances in Fractal Compression for Mul-timedia Applications, Paper, Department of Computer Science University of Water-loo, Ontario Canada

- J. Y. Lettvin H.R. Maturana, W.S. McCulloch and W. H. Pitts –

What the frogs eye tells the frogs brain, “the Mind Biological Approaches and its Functions”, 1968, pps 233-258

- D. Marr, March 1976 – Articial Intelligence – a personal view,

Artical Intelligence Memo, March 1976

- Marvin Minsky – K-Lines: A theory of memory, Articial

Intelligence Memo No. 516 June 1979

- J. Piaget (1954) – The construction of reality in the child,

Basic Books, Inc, N.Y. (1954)

- Michael I. Posner, Marcus E. Raichle – Images of the mind, New York : Scientic

American Library, c1994.

- Satyajit Rao – Visual Routines and Attention, Ph.D. Thesis February 1998, Department of

Electrical Engineering and Computer Sci-ence

- Elisabeth Spelke, Peter Visthon and Claes von Hofstein –

Object Perception, Object-directed Action and Physical Knowledge in Infancy, Per-ception and Action in Infancy

- Edward Tufte – Display of Visual Information, Graphics Press 1983

- A.M. Turing – Computing machinery and intelligence, Computers and Thought pp 11-35, 1963

- Shimon Ullman – From “High Level Vision” Chapter 9 Visual Cognition and Visual Routines, MIT Press June 1996

- Shimon Ullman – From “High Level Vision” Chapter 10

Sequence Seeking and counter streams: A Model for Information Flow in the Visual Cortex, MIT Press June 1996

- Prof. Dr. Alfred Ultsch, Dipl.-Math. Dieter Korus – Self-Organizing Neural Networks for

Acquisition of fuzzy Knowledge,

Lupton, Ellen and Miller, J. Abbott. Design Writing Research: Writing on Graphic Design. Kiosk, New York. 1996.

defining digital space through a visual language

Documents