LONG PAPER

Steve Goschnick Æ Connor Graham

Augmenting interaction and cognition using agent architecturesand technology inspired by psychology and social worlds

Published online: 20 November 2005� Springer-Verlag 2005

Abstract Intelligent agents can play a pivotal role inproviding both software systems and augmented inter-faces, to individual users from all walks of life, to utilisethe Internet 24 h a day, 7 days a week (24·7), includinginteraction with other users, over both wireless andbroadband infrastructures. However, traditional ap-proaches to user modelling are not adequate for thispurpose, as they mainly account for a generic, approx-imate, idealised user. New user models are thereforerequired to be adaptable for each individual and flexibleenough to represent the diversity of all users usinginformation technology. Such models should be able tocover all aspects of an individual’s life—those aspects ofmost interest to the individual user themselves. Thispaper describes a novel intelligent agent architecture andmethodology both called ShadowBoard, based on acomplex user model drawn from analytical psychology.An equally novel software tool, called the DigitalFriendbased on ShadowBoard, is also introduced. This paperillustrates how aspects of user cognition can be out-sourced, using, for example, an internationalised bookprice quoting agent. The Locales Framework fromComputer Supported Co-operative Work is then used tounderstand the problematic aspects of interaction in-volved in complex social spaces, identifying specificneeds for technology intervention in such social spaces,and to understand how interactions amongst mobileusers with different abilities might be technically assistedin such spaces. In this context, the single user-centredmulti-agent technology demonstrated in the Digital-Friend is adapted to a multi-user system dubbed Shad-owPlaces. The aim of ShadowPlaces is to outsourcesome of the interaction necessary, for a group of mobileindividuals with different abilities to interact coopera-

tively and effectively in a social world, supported bywireless networks and backed by broadband Internetservices. An overview of the user model, the architectureand methodology (ShadowBoard) and the resultingsoftware tool (the DigitalFriend) is presented, and pro-gress on ShadowPlaces—the multi-user version—isoutlined.

Keywords User model Æ User agent ÆUniversal interface

1 Introduction

A compelling vision of the future is one with pervasiveand ubiquitous computing spanning our public andprivate lives, usefully but not intrusively. As broadbandaccess to the Internet becomes widely available, andwireless networks extend access to it, individual usersrequire new levels of interaction with the ever-presentsystem. At the outset of the ShadowBoard project[11]—which is central to this paper—it was envisagedhaving a software system on call, 24 h a day 7 days aweek (24·7), to enable an individual user to have aneffective and productive Internet presence, regardless oftheir current circumstance.

Agent-oriented analysis, design and programming(AOP) has emerged as a paradigm facilitating higherlevel modelling of user needs and the mapping of thoseneeds to flexible and diverse systems [3, 43]. Along withconcepts from Psychology, the project adopted an agent-oriented paradigm to form a novel architecture forbuilding a Digital Self—a continuous representation ofthe user via the Internet, which monitors,summarises,computes,alerts and notifies the user in an assistivemanner, not overtaxing their attention. The architectureis called ShadowBoard [10], and its features include

– a complex but flexible user model that includes thedecomposition of a user’s multiplicity of roles into ahierarchy of sub-agency, based on sub-selves in thepsychology underpinning the model.

S. Goschnick (&) Æ C. GrahamInteraction Design Lab, Department of Information Systems,University of Melbourne, 111 Barry St,Carlton, VIC 3010, AustraliaE-mail: stevenbg@unimelb.edu.auTel.: +61-3-83441522E-mail: cgraham@unimelb.edu.au

Univ Access Inf Soc (2006) 4: 204–222DOI 10.1007/s10209-005-0012-x

– relaxing of the traditional need for autonomy of theagents in a multi-agent system (MAS), by putting sub-agents under the control of a single autonomousagent—the Aware Ego Agent—which is autonomousin the usual agent sense, and fully aware of all itscomponent sub-agents.

– wrapping of external services and agencies, includingweb services, as if they were internal sub-agents.

– application of ontologies at a localised, role-basedlevel.

The architecture has been instantiated in an opensource, end-user software product known as the Digi-talFriend [6]. In this paper, the name ShadowBoard isused interchangeably to refer to the architecture (theblueprint) and to the accompanying methodology (thetechnique for analysis and design in the building of auser’s Digital Friend using the DigitalFriend software).

Features of the methodology, which is well integratedwith the DigitalFriend software, include

– a generic range of roles and sub-roles with enoughbreadth to harness sub-agencies capable of servicingwork, home and recreational interests of an individualuser 24·7, engaged in multiple social worlds [36].

– generic envelopes-of-capability (EoC) within the agenttypes, which help the user to identify, collect andorganise a collection of sub-agents into usefulgroupings for further thought, discussion, design andimplementation of a useful Digital Friend—onemodelled on Self.

The ShadowBoard methodology and DigitalFriendsoftware provide a technique and a development envi-ronment for rapidly building custom multi-agent appli-cations from constituent sub-agents and web services,often accessing Internet resources, for an individual userwhen connected to the Internet via standard http pro-tocol.

In building the DigitalFriend tool and in promotingits use to new users, the design for all philosophy ofUniversal Access has been followed. The web servicesparadigm is a programmatic version of the web, enablingdistributed functionality over Internet protocols, as op-posed to a user manually navigating distributed infor-mation via a web browser. While the DigitalFriend isnot a web browser, at some level it is competitive to abrowser, although largely dependent on sites publishingweb services via the SOAP [32] or WSDL [39] protocols,or real simple syndication via RSS [17], to access dy-namic content. The DigitalFriend also strives to supportaccessibility efforts currently aimed at browser tools, inparticular adapting, where possible, aspects of the W3CUser Agent Guidelines UAAG 1.0 [40]. In this respect,future expansion of UAAG, addressing pervasive com-puting technologies such as personal digital assistants(PDAs) and smart devices, as foreshadowed by Gun-derson [16], are of particular interest.

Furthermore, with the advent of wireless networksand the subsequent expansion of the Internet’s reach

into potentially every corner of an individual’s life, theneed arises to expand the ShadowBoard architecture toaddress the mobile aspects of an individual user’s life,specifically with respect to PDAs and mobile phones.

Mobile systems pose serious challenges to the systemdesigner. Graham and Kjeldskov [14] describe these asthreefold: means of input are limited, screens are small,and use contexts are dynamic. Of these three issues,highly dynamic use contexts are a challenge for the de-signer. The users of mobile systems are particularlychallenged in their interaction with the device when theworld around them is ‘‘fluid’’ [20]. Mol [26] noted that influid interactions ‘‘neither boundaries nor relations markthe difference between one place and another’’. Bound-aries and relations among places are permeable, transi-tory and transmutable: ‘‘... sometimes boundaries comeand go, allow leakage and disappear altogether whilerelations transform themselves without fracture. Some-times, then, social space behaves like a fluid’’ [Ibid].

Fluid interaction can also be understood as the usermoves among multiple social worlds [37], or amongorbits of collective action. In a social world a body ofusers is united in a shared activity and symbolisation.Travelling, wandering and visiting have been identified asthree key activities that are pertinent to current work-related mobility [22]. Travelling is ‘‘the process of goingfrom one place to another in a vehicle’’, visiting is‘‘spending time in a place for a temporal period of timebefore moving on to another place’’, and wandering is‘‘extensive local mobility in a building or local area’’[Ibid]. Much of the shared symbolisation in theseactivities involves a common understanding of signs andlanguage [1].

The challenge posed by fluidity is heightened by avision of pervasive and ubiquitous computing, whereonline and offline activities involving others are layeredon top of the individual’s activities. Being on call andcollaborating with others constantly, is technologicallyfeasible but humanly impossible. This paper presents atechnology that facilitates the user having an effectiveand productive Internet presence continuously, andwithin changing use contexts involving others. Thecentral theme in this approach is the outsourcing ofsome interaction and cognition. This involves handingover responsibility for particular tasks to software sys-tems, namely the agents and web services integrated intoan individual’s Digital Friend. The developed architec-ture and tools are currently being extended to includemobile technology, enabling the mobile user to be en-gaged in fluid, cooperative interaction, while movingbetween different social worlds. In maintaining thenaming theme, this expanded architecture is calledShadowPlaces.

Interestingly, the hierarchy of an individual’s roles asmodelled in ShadowBoard (see Fig. 1) corresponds clo-sely with the network of multiple social worlds, [36], theuser moves between and participates in, through thecourse of their daily life. This conveniently makes theShadowBoard agent architecture and methodology

205

eminently suitable for building systems that cross intothe mobile context, beginning at a theoretical level. Thepremise here is that Strauss’s notion of social worlds isdescriptive of mobile contexts given their dynamism andfluidity [20]: social worlds form, segment, splinter anddissipate and re-form over time [36].

The argument in this paper is structured as follows:Section 2 of the paper outlines the motivation in

developing a complex, psychology-derived user modelupon which the ShadowBoard architecture is based.This architecture is then overviewed as the blueprint ofthe technology that has been developed, along with anintroduction to the psychological theory upon which it isbased.

Section 3 presents the associated ShadowBoardmethodology, which allows developers and end-usersalike to build agent-based applications realising a Digi-tal Friend for a given user. Section 4 describes aspects ofthe DigitalFriend software that has been developed,which includes the ShadowSpaces middleware, theShadowFaces user interface, and a built-in constraintlogic programming language named CoLoG. An exam-ple of how CoLoG can be used to transform colours fora colour-blind user is provided.

Section 5 describes an example of the design andbuilding of an assistive sub-agent within the Digital-Friend—one that returns the internationalised currencyamount for books selected from a user’s book wish-list,automatically converting the price from the US dollaramount that is returned from a book-seller’s web service.Specifically, the provided example illustrates the inter-agent communication mechanism.

In Section 6 Universal Access recommendations toDigitalFriend end-users and developers are provided.Section 7 draws upon concepts from the LocalesFramework [7], based on Social World theory [36], tooutline the architectural requirements for taking Shad-owBoard into mobile contexts that will support userswith different abilities, thus allowing fully-functional24·7 assistive sub-agents. The resulting enhancedarchitecture ShadowPlaces is then depicted.

Section 8 uses the Locales Framework to evaluateShadowPlaces as a viable multi-user environment suit-able for groups of differently enabled mobile users.Thus, we use a framework that has been exploited todesign and evaluate systems supporting collaborativework, to explore the potential effectiveness of Shadow-Places for supporting shared activity among multiple

Fig. 1 The ShadowBoardarchitecture

206

individuals with distinct abilities and disabilities whomove around geographically and interact with a widerange of people [20].

2 Motivation, theory and architecture for a complexindividual agent

Traditionally, the design of an interactive applicationprogram encapsulates a model of the user in support oftheir activities, based on the envisaged user tasks iden-tified in requirements gathering for the application [4].The user model embedded in an interactive system ismost often limited to some very narrow aspects of theuser, relating only to the way they go about performingthose specific tasks for which the application is explicitlydesigned.

An early approach with respect to user modelling wasto model the user’s knowledge of the interface through aTask Action Language [19]. A later advance was theGOMS (goals, operators, methods and selection rules)model, which included the modelling of user intention,and which is predominantly used to predict the timetaken for an expert to complete given tasks, hence pro-viding a metric to compare alternative interface designs.

Interaction user models are not the only type ofinteraction models built, and software designers are notthe only people who build them. Young [44] first dif-ferentiated the types of models being built (both usermodels and models of the computer system) according tothose who were doing the modelling, namely: ‘designers,researchers, users and the computer systems themselves’.

All of these models strive for a generic, approximate,idealised user. Most of them are designed to inform thesoftware designer, and to allow them to communicatewith other designers and/or other stakeholders in thedevelopment process. Some of them simulate the idealuser, though often for a specific cohort of users. Most ofthem are concerned with measuring metrics with regardto performance of the user achieving given tasks thoughinteraction.

What was needed in the context of the present workwas a user model adaptable for each individual, a modelflexible enough to represent the complete diversity ofusers employing information technology. Such a modelshould not be limited to task metrics, but should be ableto cover all aspects of an individual’s life—particularlythose aspects of most interest to the individual userthemselves.

Significant activity has occurred in modelling the usersince early TAL and GOMS. Some efforts extend theearlier concepts of modelling the user’s knowledge of theinteraction facility. Others extend the ability to simulatean idealised user more accurately, and some do lean inthe required direction. For example, in so-called mixed-initiative AI systems, in which the software calls uponthe user for some information or for the user to makesome decisions, the software builds or maintains a modelof the user [9, 45]. Preference modelling also leans a little

in this direction, but not very far. Again, all of these usermodels tend to be very application specific (e.g., mostpreference models are for web browsers), and as with theother user models discussed, they invariably assume thatthe user is currently present to interact with the appli-cation.

Conversely, the primary requirement in the workpresented here was to have a 24·7 user-representationavailable while the user sleeps or is otherwise offline,within a tool capable of autonomous computation, somedecision-making, some information filtering and withthe ability to concentrate the presentation of relevantinformation and to inform the user at the most conve-nient time.

In the context of this work, the need with regard tothe user model is on a much broader scale per individualthan any of the previously existing user models. Thedeveloped interactive system is a generic tool more thanan application program, and enables end-user configu-ration of web services (and various sub-agents) of alldescription. The tasks that a specific end-user will wantassistance with from their DigitalFriend are diverse,unpredictable and onwardly divergent as the variety ofweb services continue to expand. The need arises to fill agap in the user models currently available.

Interestingly, Kay and Thomas [21] identified a sim-ilar gap in usability evaluation. They proposed the notionof personal usability which encapsulates the need forusability for a single user. They point out that the pre-existing definitions of usability relates to a whole popu-lation of users. While conventional usability assessmentis generally concerned with building or selecting a userinterface, the concept of personal usability only beginswhen the development phase is over. Predominantly,they were interested in the user’s long-term learning offeatures of a substantial application, facilitated by anautomated user model, one that the user was able toaccess and contemplate. However, somewhat ironically,the user model they used was that of the user’s under-standing of the interactive features of the specific appli-cation program (a programmer’s editor, in their casestudy), rather than a detailed model of the individual useracross many aspects of their life.

Just as in the work of Kay and Thomas, the need fora personal usability emerged; in the context of the workpresented here, there was a need for a personal, detailed,model of the individual user, for use by that user—thatis, a detailed model of Self, to be built up by the userthemselves (possibly with a close collaborator/s) using ageneric software tool. Rather than designing andimplementing software and then measuring the user’scapacity to accomplish tasks efficiently using the appli-cation, the issue of interest here is the software accom-plishing tasks for the user, where tasks may be spreadacross the user’s interest areas including their work,home, leisure and other activities.

Figure 1 is a graphic overview of the ShadowBoardarchitecture, collectively representing an individualwhole agent modelled upon a complex user model of

207

Self. It is made up of numerous sub-components each insupport of a user sub-self (often outwardly acknowl-edged as a role in their life). The structural and com-putational implications inferred from the theory, basedon the Psychology of Subselves psychology, are out ofthe scope of this paper and are elaborated upon else-where [11]. However, the methodology presented in Sect.3 draws upon that work.

In the centre of Fig. 1 the Aware Ego Agent—the pri-mary sub-agent in the whole cluster of sub-agents—appears. The Aware Ego Agent is surrounded by eightfirst-level sub-agents, diagrammatically drawn as circlesof the same size. Five of these sub-agents in the exampleare not nested any deeper, but could be, as sub-agentscan be clustered recursively. The other three have clus-ters of circles within them, representing a second-level ofsub-agents grouped into numerous envelopes-of-capa-bility (EoC).

Each envelope-of-capability represents a different areaof expertise that a particular EoC agent embodies. Thewhole agent can perform a number of consecutive anddiverse tasks, depending on what goals via what userroles it supports in the user’s life.

Each EoC Agent contains a number of sub-agentswith related capabilities, but each of these differ from itsneighbours’ abilities in some specialised way.

The central Aware Ego Agent, as well as each of thoseother sub-agents that are shaded as spheres in Fig. 1(delegation sub-agents), have knowledge about the capa-bilities of the sub-agents in their respective EoCs. Thisknowledge is embedded in CoLoG—the internal logiclanguage—and it is used by the agent in order to selectthe appropriate sub-agent to achieve the particular goalthat has been sent their way from higher up the hierar-chy. One type of sub-agent can wrap external web ser-vices incorporating their functionality as if they were aninternal capability.

2.1 Sub-agents as subselves

The mentalistic notion of a sub-self at work within thepsyche of an individual is a metaphor for the sub-agentof a ShadowBoard agent. To broadly place this work inthe context of research upon multi-agent systems(MAS)—the closest comparative field of study—mostmulti-agent systems can be described as inter-agentsystems. In contrast, the ShadowBoard architectureoutlines an intra-agent system, enabling the incorpora-tion of many components that together represent onewhole agent, albeit a very sophisticated entity. Such awhole agent, based upon the ShadowBoard architecture(and built using the DigitalFriend software), can becharacterised as an autonomous individual agent, onethat is compatible with contemporary definitions ofagency such as that of [43].

Unlike the autonomous whole agent, the inner sub-agents are only semi-autonomous or sometimes eventotally subservient to the Aware Ego Agent—the execu-

tive controller within a ShadowBoard agent. Sub-agentsmay themselves be sophisticated agents capable of theirown semi-autonomous work, or they may be conven-tional programs, expert systems, or, more commonly,wrapped web services as discussed later on. This is asignificant relaxation of the requirement, assumed inmost MAS systems, for each individual agent to be afully functioning autonomous agent.

This flexibility in capability within a ShadowBoardagent is aimed at building assistive technologies whichaugment individual human skills and attention ranges,rather than at building synthetic entities (the generalgoal of most other MAS). It is attained by making allsub-agents ultimately subordinate to the Aware EgoAgent. Such a relaxation of autonomy is comparable atsome level with the work of Scerri et al. [29] on adjust-able autonomy on the Electric Elves project.

The inclusion of sub-agents and EoCs of sub-agentswithin the ShadowBoard architecture allows a DigitalFriend to be populated with many different types ofdomain-specific sub-agents. This allows for an open-ended expansion of capability and knowledge, depend-ing on an individual human’s needs, including theirspecific needs with regard to the social worlds theyparticipate in, via the groups of people they interact withon a regular or less regular basis.

3 Methodology

The aim of building a Digital Friend includes providing acomprehensive user-proxy—something that not onlyaugments the user’s ability and attention span in theassistive technology sense, but which also represents theuser at times when they are off-line technically orotherwise. In order to do so for a wide range anddiversity of individuals in an anyone, anywhere, anytimeperspective, Goschnick devised an accompanyingmethodology [11], which has since been enhanced in thiswork as follows.

The ShadowBoard methodology contains elements ofboth top–down and bottom–up approaches for analysisand design. In the top–down phase, the individual/clientbeing analysed is asked to conceptualise the roles they fillduring a normal period (initially a week, later iterations,a month and even a year) of their lives. To facilitate thissession, they are presented with a broad range of genericroles which are likely to encompass the sorts of roles thatany given user may have at times in their life.

These roles are a starting point, used for individualsto articulate, specify and accumulate details, but are alsoused at a subsequent stage in order to source and employhelper sub-agents including web services. Additionally,these roles are used later on, within the working Digi-talFriend software in order to filter and identify thestream of information messages displayed to the userfrom the various enacted sub-agents. The main genericroles also each encompass generic sub-roles. Theopening list of ShadowBoard roles together with their

208

sub-roles appears in Table 1. The complete drill-downlist of 62 generic roles and sub-roles is depicted graphi-cally within the user interface of the DigitalFriend (seeFig. 2). Clearly, it is unlikely that a given individual willhave sub-roles of all these types. The methodologypresents the full generic range to help any user identify,collect and organise a combination of sub-agents intocommon groupings, based on all roles and sub-roles intheir life.

For each role (and sub-role) that the client has posi-tively identified with, they are asked about the goalswhich they expect to achieve—both short-term andlong-term—within that role. When they have expressedthe goals to a self-satisfactory level, they are then askedabout the tasks required to achieve each goal.

This first part of the method is predicated on thealignment of roles in an individual’s life with some of thesub-selves in their psyche; i.e., in the psychology, sig-nificant roles in an individual’s life, such as parent, tea-cher, husband, researcher, traveller, are known to alignwith some sub-selves in the person’s psyche. This part ofthe method also picks up those aspects of a life wherethere are clear commitments and obligations, includingwork and other community involvement, and alsofamily and significant relationship roles.

The harder part of the analysis is in uncovering thesub-selves for which the individual does not perceive anexisting role in their life, but which nonetheless have aninfluence on their behaviour and regular activities.

At this point, two distinct paths are taken within themethodology to obtain further information: the analystgoes through those roles in Fig. 2 for which the indi-vidual did not identify a role within their life. They areasked if they know anyone who does fit such a role well,in a sense, like a mentor or at least an exemplarregarding that role. If so, they are then asked if at anytime their own activities or tasks bring that person tomind. Or if they would like to be able to perform such

activities or tasks as competently as that person. Theunderlying psychology for this approach is that peoplewill often model a sub-self on a role-model—somebodythey admire with regard to a particular talent or skill orbehaviour or track-record.

The second part of the second phase is the bottom–upaspect of the methodology, which complements the role-based phase. It involves gathering information aboutactivities and tasks that the individual regularly engagesin during a day (and later, each day of the week) in theirlife. The idea is to catch and articulate those activitieswhich are seemingly unrelated to any specific goals. Forexample, they might just be pleasant or pleasurableexperiences. This also frequently captures some activitiesthat are related to the user’s goals, but which simply didnot occur to them during the role-based phase of theanalysis.

The bottom–up process is structured as follows: aftera regular user activity has been identified, the client isasked: what task does that activity complete, or whatwould you call that regular task? (i.e., performing a givenactivity, is an instance of a regular task). The subsequentquestion is: what goal does that task work towards, and/orwhat role would you use to classify that activity?

Clearly, there are a number of existing complemen-tary methods for capturing this type of user-specificactivity data which can be used in conjunction with theShadowBoard methodology, particularly ethnographi-cally oriented studies, contextual enquiries, user-keptdiaries, and scenario-based requirements analysis [28] toname a few. The emphasis in this phase is on the non-work, non-commitment-oriented aspects of the individ-ual’s life. However, as described above, it may alsoidentify tasks which do relate to a primary role in theuser’s life, but which they simply overlooked in the firstphase of the analysis. The tasks and activities capturedhere are written in a verb-first style as is done in a TaskAnalysis [5], e.g.:

Table 1 ShadowBoard methodology: roles and sub-roles

Manager Protector Personal assistant Initiator

Benevolent manager (arch.) Safety officer (archetype) Selfless pleaser (archetype) Inventor (archetype)Conciliatory manager Defender Service provider Success seekerPlanner Risk analyst Networker Resource masterScheduler Environmentalist Communicator Trouble shooterCoordinator Pacifier Marketer PusherRecycler Doctor Teacher Reminder (reactive)Controller Exit strategist (reactive) AdviserDecisive manager (reactive)

Critic Adventurer Knowledge seeker Intuitive

Perfectionist (archetype) Explorer (archetype) Knowledge worker (arch.) Seer (archetype)Editor Risk taker Concept learner Mood senserQuality controller Traveller Learner Pattern finderDoubter Vacationer Information officer DreamerCynic (reactive) 2D situator Data miner Profiler

3D situator Random generator (react.) Role keeperLazybones (reactive) Core value bearer

Affirmation agent

209

Select a book from my current wish-list of purchases.Price a specific book.Determine if a specific book price is within currentbudget for it.Find all recent released movies.Determine which recent movies are playing at a localcinema.Calculate the nearest cinema where a specific movie isplaying.

These activities can be in a related sequence, or ahierarchy of activities, as observed in Task Analysis.

With the superset of tasks collected from all phasesoutlined thus far, the user/client is then asked, task-by-task:

1. Whether they enjoy (manually) doing the taskthemselves;

2. Whether they would prefer to offload that task en-tirely to an automated assistant capable of compe-tently performing it;

3. Whether they would happily accept some automatedassistance with the task, but would prefer to retainsome manager/overseer/collaborator role, in theevent of the availability of a competent softwareagent.

(a) Under what conditions they would like such assis-tance.

User responses to these question allows preference tobe placed on seeking sub-agents and web services forincorporation into their DigitalFriend, and helps toestablish the rules regarding how and when the sub-agents are invoked. For example, two different individ-

uals may both identify the task of finding a newly re-leased movie at a nearby cinema as something they doregularly. One person may prefer to have that taskautomated, with the presentation of all recently releasedmovies showing at nearby cinemas in a list. A secondperson may enjoy using a web browser to go and find themovies themselves, leisurely perusing the detailsregarding the cast, the storyline, and the views of peoplewho have seen it. On occasions when time is short, thesecond person may also want a quick list of what is onoffer locally, so their preference reverts to one similar tothe first person’s general preference.

At all times during the analysis there is an emphasison the fact that the technological assistance that will bemade available via the DigitalFriend is about enhancingthe individual’s life. Enhancement is provided bystreamlining aspects which are repetitive, by increasingusers’ performance at certain tasks when they desire it,or by strengthening those aspects of their life where theyare either weakest or have least interest, but are boundby duty or circumstance to undertake related activities.At no time is there any sense of replacing the user atsome future time—as there can be in work-centredstudies of human efficiency and performance involvingtasks which utilise computerised technology. It is amethodology that squarely places the interest of theindividual foremost—their working life being of interestto just one part of the analysis.

Proceeding from the analysis to the implementation ofsub-agents using the DigitalFriend software also resultsin two directions paralleling the analysis—top–down andbottom–up. The generic role hierarchy is already builtinto the system (see Fig. 2). From within the software it

Fig. 2 Use of roles and sub-roles to filter messages to theuser within the DigitalFriend

210

is used to differentiate or filter the messages that aredisplayed for the user’s attention (see right-hand back-ground pane in Fig. 2). When creating or editing a givensub-agent, the user categorises it by role and sub-role tofacilitate this filtering (see second and third fields in theproperties dialog in Fig. 5).

Those tasks which have been identified as high pri-ority for automation may be further analysed—againthere are complementary techniques outside the Shad-owBoard methodology, such as entity relationship(ER) modelling and workflow analysis that can helphere—and can be used as aids to source and configurethird-party agents or web services. These sub-agents arealso built into a hierarchy of agents—a separate struc-ture from the generic roles structure (see left-hand sideFig. 5). There is a third hierarchical structure whichrepresents a Knowledge Tree of the user’s data and otherfiles (see Fig. 3). The example envelope-of-capability sub-agent presented later on in Sect. 5 demonstrates howboth local resources in the Knowledge Tree and dis-tributed resources via web services are combined toinstantiate new, more complex sub-agents.

This methodology is generally successful when theuser knows himself or herself well, and has a clear viewof the roles and tasks in their life. However, the bottom–up aspect of the process allows for the capture of lessstructured and less goal-oriented activities in a life,broadening the applicability of the methodology to mostpeople. Furthermore, it should be noted the Shadow-Board methodology for building a Digital Friend is verymuch about utilising third party agents, particularly webservices as sub-agents, currently from within the

DigitalFriend software tool, but potentially also usefulwith other multi-agent systems (MAS).

4 Implementation of ShadowBoard: the DigitalFriend

Enacting the ShadowBoard architecture has been has-tened by the use of several existing technologies, whichhave been modified as necessary, and enhanced with newtechnology, thus resulting in an integrated, cohesivesystem.

There are several levels of technology. There is apersistent object storage system called ShadowSpaceswhich maintains the dynamic model of the hierarchy ofsub-agents (represented by the octagonal structure in themain/left panel of Fig. 2). Together, those sub-agentsare also inter-related with logic programming to enact auser’s DigitalFriend.

There is a graphic user interface (GUI) system namedShadowFaces (Sect. 4.2), which makes extensive use of 8-way octagonal tiles of varying sizes per level. The sameinterface is used as an editor to create and modify sub-agents, as a user-friendly file manager, and as a devicewhich visually alerts the user about certain events andsituations. ShadowFaces remains running at all times asthe face of the DigitalFriend, receiving notifications, fil-tering information and details of decisions that the Digi-talFriend has made, and referring some decisions back tothe user for their executive override when necessary. Itdisplays visual messages and alerts for the user’s attention,together with auditory icons (while other sub-agents maysend text messages via email and SMS sub-agents).

Fig. 3 The Knowledge Treedisplayed via the built-in FUNfile manager

211

There is an underlying logic language parser andcomputation system, which uses constraint logic pro-gramming concepts [25, 34] to bind together the varioussub-agents into a synergy of functionality. This con-straint logic language is called CoLoG. For further de-tails see Goschnick and Sterling [13]. There is also arecent research effort that has combined the Shadow-board architecture with distributed database transactionconcepts, to combine sub-agents and web services [18].

4.1 ShadowSpaces: a dynamic object system

The model behind ShadowSpaces is not unlike theW3C document object model (DOM), and some lim-ited effort could be made to be DOM compliant [42].It is used to dynamically maintain a hierarchy ofobjects, independent of the way they are instantiatedand stored in the language the systems are createdin—Java in this case. As with the W3C DOM model,the ShadowSpace tree is mutable; i.e., it is able todynamically deal with events that modify the structureof the tree itself.

4.2 ShadowFaces

The interface adopts an agent-metaphor (in addition tobeing based on an agent technology) rather than thedesktop metaphor—which, in the context of this work, isconsidered to be a severely challenged metaphor in the24·7 space. It is worth noting that throughout the era ofGUI, document-centric operating systems (MS Win-dows, Mac OS), within common applications such as,for example, word processors, there has always beensome functionality following an agent-metaphor. Inproducts such as Microsoft Word and WordPerfect, thefind/search and replace functions follow an agent-meta-phor style of operation—as opposed to the direct-manipulation operation of say, the marking and boldingof text in a document [24].

ShadowFaces—the ShadowBoard interface—is anenhanced version of an earlier file manager technologyfor Friendly User Navigation, called FUN, developedby one of the authors. It displays three levels (four,counting the leaves) of the hierarchy of components atany one time, and can be envisaged as an informationlens that the user can use to navigate the recursivestructure of a hierarchy. This octagonal tile lens is usedin three different, but interrelated, hierarchies within theDigitalFriend. Firstly, it is used to display the currentstate of the hierarchy of sub-agents in a user’s DigitalFriend, as depicted in the left-hand side of Fig. 2. Sec-ondly, it is used to filter notifications and messages byrole and sub-role, as displayed in the right-hand side ofFig. 2 previously discussed. Thirdly, the FUN FileManager has been retained within the DigitalFriendprogram (see Fig. 3) to access and edit the Digital-Friend’s Knowledge Tree—a local disk/secondary stor-

age directory space, set-out according to a generic butuser-modifiable ontology.

Sub-agents store their information in files within theknowledge tree structure. This information includesdatabase lookup files and web service agents data stores(current-state of variables between web service pollingtimes). The user may also store and activate their usualword processing documents and other files within thesame directory structure. They can activate theirfavourite editors and word processors from within theDigitalFriend.

Note that the root of this file structure is alwayscalled AAAA_Root, but all sub-directories after that aresimply read from the hard disk, as they are found. Thisallows a user to create their own ontology or modify thedefault ontology distributed with the DigitalFriend.

The icon that will be displayed for a given sub-directory within FUN is always named ‘thisDirIcon.gif’so that the user can supply their own preferred icons. Inaddition, in accordance with the UAAG 1.0 standard, itis recommended that users always create a text filenamed ‘thisDirIcon.txt’ and within it describe the con-tent of the icon, so that assistive technology by otherdevelopers can be adapted to the DigitalFriend, whichmay then read textual equivalents of the icons.

4.3 Advantages of the ShadowFaces interface

Advantages of the interface include the following:

– It is based on an agent-metaphor rather than thedocument-centric interface of the desktop metaphorof most current-day operating systems, making itapplicable to various devices in addition to desktopand laptop computers.

– Up to 589 individual nodes in the hierarchy can bedisplayed and are selectable at one time, versus the useof a tree manager approach (such as the MicrosoftWindows Explorer, or the Finder in Mac OSX), whichdisplay no more than about 50 items in a typicaldisplay.

– The layout and its recursive nature echo the graphicdepiction of the ShadowBoard architecture, reinforc-ing the system image in the user’s mind—which isuseful when introducing a new software interface [31].

– The interface can be expanded to full-screen or re-duced down to the size of a hand-held screen (as lowas 160·160 pixels), making it a suitable interface for anumber of different consumer devices, includingPDAs.

In addition, to aid the selection of the interface’soctagonal tiles via a range of interface devices, there areeight so-called live-boundary areas, which the user mayselect to choose an appropriate child octagon tile of thecurrently selected tile. This is depicted in Fig. 4, inwhich the current octagonal tile is the second level oneto the north-west of the centre tile. It has eight childoctagonal tiles, marked in white for explanation pur-

212

poses. Any one of the eight can be selected either di-rectly via mouse, track-ball or pen, or indirectly byclicking within the larger representative live-boundaryarea for that child tile. The appropriate representativearea is directional; i.e., the child tile to the north isrepresented by the north live boundary area, and so onfor South, East, West, NE, NW, SE and SW. The live-boundary areas are particularly useful on small screens.On larger screens they suit devices which favour atarget area on the edge of the screen (e.g., Mouse,trackball and joystick), while the selection of thescreen-centred tiles favours interfaces that make use ofa stylus or a pen (e.g., a PC Tablet).

This input device flexibility makes it ideal for 24·7operation due to its suitability to different types ofconsumer devices, each applicable to different physicallocations. The same interface can be used to zoom,navigate, filter and access properties (See Fig. 5, editingan Envelope-of-Capability agent).

There are a number of issues with regard to theusability of this novel interface, which are currently thesubject of an ongoing HCI research project at the Uni-versity of Melbourne, to make it increasingly usable tothe uninitiated. Some aspects were investigated in a re-cent honours student project [23].

4.4 The computational engine: CoLoG

The computation engine is based on a distributed con-straint logic language called CoLoG, developed as partof the DigitalFriend tool. CoLoG is derived from a Javaimplementation of a single-threaded language parser,with added logic and constraint commands, multi-threaded support, inclusion of system predicates, andinterfaces to Java objects and database systems. It alsohas four inter-agent communicative actions (Give, SendMessage, Tell and Gesture)—which give the user theability to design the flow of messages and data betweensub-agents and to the user.

Each EoC agent within the DigitalFriend is capableof its own thread of execution, and the primary logic forexecution of each EoC agent is held in its own CoLoGprogram. An individual’s DigitalFriend may containdozens of such logic programs, executing or on-call, atany one time. This aspect of the implementation iscovered elsewhere [10]; however, two small demonstra-tive examples follow.

4.5 Minimalist CoLoG code example

The first minimalist example is included to introduce thereader to the CoLoG language itself, but it also illus-

trates how the proposed framework can contribute toUniversal Access, as it addresses dealing with colourblindness using CoLoG. Recall that the web servicesparadigm represents a programmatic interface to theInternet, which is parallel to the human-centric web-browser interface. Therefore, the Web Accessible Ini-tiative guidelines can be seen as a guide to implementagents based on web services, which interact with theuser on the client-side. Consider the ‘Online shopper withcolor blindness—scenario’ in the W3C Working Draft:How People with Disabilities Use the Web [41]).

Consider the following style as if they come from aremote web service, which provides an alternative pro-grammatic source of data, to the data provided in theirusual web pages:

If the colour-blind user prefers grey where there iscurrently green, darkgrey where there is currently blue,and yellow where there is currently red, then within theuser’s Digital Friend the following agent rule can beformulated in CoLoG:

where the predicate theirMeaning is provided by theexternal web service, and colorConversion is a locallookup table as follows:

Fig. 4 Live-boundary areas within the ShadowFaces GUI interface

213

Then the user can simply be told by the sub-agentwhat their preferred colours mean with regard to theinformation generated by the web service in question, asit is displayed within their DigitalFriend. See Fig. 6 forthe agent that generates the values of the meaning vari-able for the user’s preferred colours.

In this sense, the DigitalFriend can be used as analternative interface to the Web Browser, provided thatthe web site publisher outputs parallel information inWeb Service format. As XML is used more to store web-centric data, this is becoming a publishing scenario thatcan be highly automated.

5 Price quote internationalisation (i18n) agent

In the following,more extensive example, an agent is builtusing the DigitalFriend which returns book price quota-tions in the appropriate currency of the user, regardless ofwhere they are in the world. The agent is able to do thisdespite the fact that the web service employed to retrievebook prices always responds inUS dollar amounts. To doso, we logically relate several different sub-agents, usingthe built-in CoLoG language and its communicative ac-tions. The sub-agents involved (see Fig. 7 below) arerepresented by the following five predicates:

– book (Title,Country,Cost,Currency)– bookWishList (ISBN,Title,Year)

– quote (ISBN,CurrencyCode2,Amount,Vendor)– currencyConversion (CurrencyCode1,CurrencyCode2,

Rate)– countryCurrency (CountryCode,Country,Currency-

Code1,Currency)

Fig. 5 The property edit dialogfor the example BookPriceI18nEoC agent

Fig. 6 Minimalist CoLoG example: sub-agent for the colour-blinduser scenario

colorConversion (grey, green);colorConversion (darkgrey, blue);colorConversion (yellow, red);colorConversion (black, black);

214

In the left-hand side of Fig. 7, the CoLoG programfor the booPriceI18n EoC agent is presented, which isdynamically constructed from its subagents using theMindProbe button1.

The CoLoG logic rule needed to generate the book2

predicate is:

book (Title,Country,Cost,Currency) IF bookWish-List(ISBN,Title,Year)quote(ISBN,CurrencyCode2,Amount,Vendor) ANDcurrencyConversion(CurrencyCode1,CurrencyCode2,Rate) ANDcountryCurrency(CountryCode,Country,Currency-Code1,Currency) AND# (Cost, (Rate)*(Amount)); // Assignment.

The predicates in this example are derived as follows:

1. quote—This is from a web service feed from thebooksellers Noble and Barnes; given the ISBNnumber of a specific book, it returns the current pricein US dollars. The web service is wrapped by aShadowBoard sub-agent called BookQuote.

2. countryCurrency—This is from a lookup table ofCountry and Currency codes, originally sourcedmanually from the United Nations web site. It isfairly non-volatile and is stored on the local disk.Some sample records follow:

3. currencyConversion—This is from a database EoCagent called CurrencyConverisonEoC which receivesupdates to its records via a number of web servicewrapping agents—AussieUS2, EuroUS2, PoundUS2and YenUS2. It holds the current exchange rate be-tween each pair of currencies, represented by the twovariables—CurrencyCode1, CurrencyCode2.

4. bookWishList—This represents the user’s list ofbooks he/she may buy if the price is right, held in adatabase lookup file on the local drive in theKnowledge Tree (a sub-directory structure represent-ing an ontology) of the DigitalFriend (see Fig. 3).The sub-agent that wraps it, has the samename—BookWishList.

BookPriceI18n is an EoC agent that supplies the newprice internationalised (I18n) service, delivered via thepredicate called book, enacted by the logic rule reportedabove. CoLoG dynamically combines the terms comingfrom the various sources just described, and finds theappropriate solution/s in the appropriate search-space.

The solution in Fig. 7 is Australian $97.40, calculatedfrom the Noble and Barnes figure of US $75 for thecurrent book of interest. If the goal/query indicated thatthe user is located in Japan, as follows:

book (‘‘A Guide to SQL’’,japan,Cost,Currency),then the following solution would be obtained:

If the goal/query indicated the user is located in theUnited Kingdom, as follows:

book(‘‘A Guide to SQL’’,united_kingdom,Cost,Cur-rency),

then the solution would be the following:

1Usually, this program executes behind the scenes, but theMindProbe button gives the end-user/developer a way of checkingwhat CoLoG program is built by the software and whether it workscorrectly.2In CoLoG, the terms starting with an uppercase character arevariables, while those starting with lowercase are instances. Predi-cates—complex term names which precede an open bracket—alsostart with lowercase characters. The rule would be similar in theProlog language and its derivatives.

215

Various sub-agents generate different parts of theprogram in real time, often triggered by timers in theagents, polling the wrapped web services. Figure 8 rep-resents the flow of agent communicative actions whichtake place in this example. If the sub-agents that wrapweb services feeds, such as EuroUS2, are running atdifferent frequencies, and when any one of them placesnew data in currencyConversionEoC, it needs to Tell theEoC about the change. In turn, currencyConversionEoCGestures the parent agent BookPriceI18n, indicating thatit should dynamically rebuild its CoLoG program andcompute a new book price, which it then sends to theuser as a Message (as seen in Fig. 2). Even though thecommunicative actions flow up the hierarchy of sub-agents in this example, there is no such restriction in theDigitalFriend tool; i.e., a given sub-agent can be con-figured to communicate with any number of other sub-agents, including sub-agents across branches of the tree.

When an agent is present that is capable of auditoryoutput, the solution messages can be passed to that sub-agent, so that it can speak the results to the user. Simi-larly such messages can be passed to sub-agents whichwrap web services capable of sending email or SMSmessages3.

6 Universal access recommendations to DigitalFriendend-users and developers

In the spirit of the UAAG 1.0 standard, while antic-ipating further developments to the standard related to

new types of user agent software (such as the Digital-Friend) as foreshadowed by Gunderson [16], the fol-lowing recommendations are made to end-users anddevelopers using the DigitalFriend V1.0 tool:

– In addition to the ‘thisDirIcon.gif’ file in each sub-directory in the Knowledge Tree, always create a textfile named ‘thisDirIcon.txt’ and describe the symbolicintention of the icon in it and hence the intendedcontent of the sub-directory, so that assistive tech-

Fig. 7 Probing the exampleEoC agent’s dynamic CoLoGprogram, showing book price inlocal currency

Fig. 8 Designated flow of agent communicative actions in exampleagent

3Such SMS web services currently exist for many countries, but theuser typically needs to be a paying subscriber to access them.

216

nology adapted to the DigitalFriend will be able toread textual equivalents of the icons. This is doublyuseful, since all icons have the same name (to facilitateusers building their own Knowledge Trees), this textfile differentiates them.

– Users of pictograms should place appropriate sub-directories in the Knowledge Tree, which representthe categories in which the user saves information,and should place a version of the pictogram file re-named/reformatted as ‘thisDirIcon.gif’ for each suchsub-directory.

– Users may accumulate http-formatted links in a filenamed ‘thisDirLinks.txt’ in each of the sub-directorieswithin the Knowledge Tree. In the next minor versionrelease of the DigitalFriend (V1.01), GUI access willbe enabled to those links within such files, within theKnowledge Tree structure, allowing the user to accessand display the linked page in an HTML viewerwindow from within the DigitalFriend. It will bepossible to save local versions of these linked files.

– The DigitalFriend has keyboard equivalent access toall features of the program accessible through themenus. The developers of assistive technology, whowish to give their user-base access to the DigitalFriendprogram, may use those keyboard shortcuts.

– In accordance with UAAG 1.0, documentation on theuse of the DigitalFriend, conforming to the WebContent Accessibility Guidelines at the Double AAlevel, is being made available on the www.Digital-Friend.org web site.

7 ShadowPlaces: moving among social worlds

As noted in the introduction, interactions among mobileusers tend to be ‘‘fluid’’ [20]. This fluid interaction isrationalised by Kakihara and Sorenson [20] in terms ofboundaries. This paper draws on Strauss’s [35] notion ofsocial worlds in order to extend ShadowBoard to ac-count for interaction among mobile individuals withdifferent abilities. Like boundaries, social worlds haveboundaries that vary in nature and existence. However,the notion of social worlds is richer and therefore moreappropriate to account for the interactions to be facili-tated:

‘‘Some worlds are small, others huge; some areinternational, others are local. Some are inseparablefrom given spaces; others are linked with sites that aremuch less spatially definable. Some are highly public andpublicised; others are barely visible. Some are so emer-gent as to be barely graspable; others are well estab-lished, even well organised. Some have relatively tightboundaries; others possess permeable boundaries. Someare very hierarchical; some are less so or scarcely at all.Some are clearly class-linked, some (like baseball) runacross class’’ [36].

The notion of social worlds has been used as a met-aphor to understand complex group interaction in recentresearch efforts. Fitzpatrick et al. [8], within the Locales

Framework, uses social worlds to define the core notionof locale:

‘‘A locale is constituted by the relationship between aparticular social world and its interactional needs, andthe ‘site and means’ used to meet those needs, i.e., thespace together with the resources available there.’’

Thus a locale is defined as a place which includes boththe space and the resources and affordances of thatspace. These are utilised by a group of people, or a socialworld, united in pursuing a shared activity. A socialworld is defined as having ‘‘at least one primary activity’’[36] and a ‘‘universe of regulated mutual response[whose boundaries are] set neither by territory nor for-mal membership but by the limits of effective commu-nication’’ [30].

Fitzpatrick envisaged Locales as ‘‘a set of conceptsfor describing what is as well as envisioning what couldbe’’ [7] and as ‘‘a shared abstraction in the sense of beinga common background or starting point against whichdomain-specific concepts and abstractions can be drawn,and design solutions can be explored’’ [7]. Thus theargument is that Locales can play a communicative rolein understanding complex problems in the developmentof virtual environments for users engaged in cooperationand collaboration, acting as a means of generatingcommon understanding between the technical and socialscientists engaged in CSCW (Computer-SupportedCooperative Work) research.

Here social worlds are adopted in order to enrich theShadowBoard architecture, in a fashion similar to theexploitation of social worlds within the LocalesFramework, i.e., as an abstraction acting as a supportfor design. Locales are also used as a heuristic scheme toevaluate the effectiveness of this enrichment of the pro-posed extended architecture, ShadowPlaces, in a mannersuggested by Greenberg et al. [15].

ShadowBoard has been used as a rich abstraction forunderstanding individual whole agents in complex situ-ations, although it was always envisaged [12] thatadditional agency built upon it would then plug backinto a multi-user system. Current research efforts focuson exploring how ShadowBoard can be applied to multi-user situations in which the user is mobile. It is believedthat social worlds can help to account for interactionamong mobile users with different abilities.

Given that the ShadowBoard agent architecture re-flects the Psychology of Subselves, which in turn reflectsthe complex roles and multi-faceted identity of a typical,busy, modern individual, it is not surprising that itcomplements Strauss’s social world view of an individ-ual’s actions and interactions. As noted above, socialworlds may contain sub-worlds and those sub-worldsmay contain further sub-worlds. We have seen in Sect. 3that very often a user’s subselves equate to roles andsub-roles that the individual performs within the realmsof work, leisure and family activities, i.e., within differ-ent social worlds and sub-worlds.

In mobile interactions, social groups form andsplinter through the disruption of communications and

217

the situation in which the individual user is engaged is influx as the user is moving through real time and space. Inaddition, the limit of a social world is constrained not byphysical space, but, as noted by Shibutani (1955:524), bythe limits of effective communication. This is particu-larly relevant for interactions among mobile individualswith different abilities: not only do these individuals driftin and out of communicative range, but they also havespecific resources through which they communicate.Social worlds also support shared symbolisation such as,for example, talk. These kinds of interactions can besupported by current technology, e.g., through hand-held PDAs and mobile phones networked using Blue-Tooth or Wi-Fi technologies. However, these devices,with ShadowBoard agency built into them, present theopportunity to outsource and mediate some interaction.

The opportunities presented by an augmented Shad-owBoard architecture are illustrated in Fig. 9. The figuredepicts four individuals, each owning a mobile device,participating in a single social world. This social world issupported by the enhanced ShadowPlaces architecture,which acts as a locale. In the figure, all four actors have amultitude of roles in their lives, but only one of theseroles is currently being exploited within this socialworld. Each actor may be participating in other socialworlds via other roles, either simultaneously or sequen-tially. Thus the emergent interaction for these individ-uals is complex, temporal, and transitory, and involvesparticular resources using particular symbolisations.

The argument put forward here is that the sub-agentsbuilt into the ShadowPlaces systems shown in Fig. 9help the actors participate in multiple social worldssimultaneously by dealing with some of the less urgentrequests and interactions directed at, and expected of,the individual. Therefore, ShadowPlaces helps managecomplexity. In addition, this technology helps the actorsto participate in social worlds continuously(24·7)—monitoring, registering, queuing, notifying, andalerting the individual about appropriate events (actionstaken, required actions) at appropriate times, as man-aged by their individual configurations of the DigitalFriend. Thus, ShadowPlaces helps manage the temporaland transitory nature of interaction among these fouractors: the agent system expands their range of mean-ingful participation in this social world. The architecturealso supports multiple means of communication: theactors themselves can type or talk. Agents can transformthis to appropriate agent predicates, using the users’ontologies (represented in their Knowledge Trees), toenable communication in a standard way among theagents. These interactions can then be transformed byeach actor’s configured Digital Friend to meet individualneeds.

The notion that individuals drift between and amongactivities is, again, not new. The individual who is per-forming Role A in the social world represented in Fig. 9,is doing so at a sub-role level, at a lower priority ofinteraction than the other three actors. The notion of

Fig. 9 ShadowPlacesarchitecture supporting a socialWorld, using networkedDigitalFriend-enabled devices

218

different levels of engagement maps onto Fitzpatrick’snotion of Individual Views:

‘‘People engage in a range of work activities withdifferent social worlds and hence different locale views.... They move seamlessly and often unconsciously be-tween activities, maintaining dynamically varying levelsof focus and participation in different locales, from fullintense focus, as if being in only one locale, to havingbackground awareness, to being ‘out of sight’.’’ [8].

At this point Locales can be taken into account inorder to measure and further enrich ShadowPlaces. Indoing so, more specific aspects and applications ofShadowPlaces will be considered.

8 Measuring ShadowPlaces with the locales framework

In addition to aiding the understanding of complexsystems across the technical and social divide [8], Fitz-patrick, with others, have also used the LocalesFramework as a means of conducting heuristic evalua-tions of groupware systems [15]. In this context, Localesis being used to evaluate systems, the output of theevaluation being a measure of how well they supportcooperative work mediated by a virtual environment. AsShadowPlaces is not yet a fully implemented technologywith an associated environment, Locales is used here toenrich the description of the architecture presented thusfar. This description explores how ShadowPlaces cansupport mobile users with different abilities. This, again,has a precedent: the heuristics generated from Localeshave been used to evaluate a single user mobile route-planning application exploiting wireless technology andrunning on a PDA [38]. Here, Locales are used toevaluate and enrich a multi-user agent architecture.

Greenberg et al. [15] described five heuristics forevaluating groupware environments generated from theLocales Framework. While the current framework ex-tends beyond just work and work-oriented social worlds,locales are nonetheless envisaged as a large generator ofinteractions among the group of users under consider-ation. These heuristics are reproduced below.

Locale foundations Provide centres (locales) that col-lect people, artefacts and resources in relation to thecentral purpose of the social world.

Mutuality Provide awareness (mutuality) within localesthat helps maintain a sense of shared place that keepsthem informed about shared activity.

Individual views Allow individual views so that one canview a locale or aggregate multiple locales as they relateto one’s responsibilities, activities and interests.

Interaction trajectories Allow people to manage and stayaware of their evolving interactions over time.

Civic structures Provide a way to organise and relatelocales to one another (There is no equivalent of Civicstructures in the ShadowPlaces architecture – apart fromthe web-servers of the Internet itself – so it is not used inthe discussion in this paper).

8.1 Locale foundations and ShadowPlaces

ShadowPlaces clearly facilitates centralisation that inturn facilitates the social world activity: PDAs (arte-facts) and agent technology (resources) are networkedvia BlueTooth or Wi-Fi so that individuals possessingthese artefacts can interact. The ‘‘centre’’ formed by aShadowPlace has a limited boundary, as does a socialworld: both BlueTooth and Wi-Fi have a limited range.However, if the artefact is within the proximity of a basestation, it will connect the user to the Internet, therebygiving them global range. WiFi is now seen as the pre-eminent technology to help cover ‘the last mile‘ in con-necting everyone, everywhere to broadband Internet.

Incorporating software agency into the networkedartefact effectively extends the boundary of the socialworld, beyond what would normally be considered alimit of communication. Asynchronous communicationssuch as store-and-forward email similarly extend theboundary of communication, but in a less dynamicmanner. The particular software agency used (Shadow-Board agents) also highlights a difference between thehuman actor’s experience and the agent’s experience. Inthe case of accessing Internet-based sources of infor-mation, the human actor will typically experience a re-source equating to a Web browser in a PDA, while thesub-agents employed within their Digital Friend will useweb services and RSS feeds, a programmatic interface tothe Internet, to source the same or similar information.

The notion of role, or in Locales terms ‘membership’,within centres is also very important. In the case of anetwork with Bluetooth only devices, one of the deviceshas to engage in a master/server role, while the otherBluetooth devices are slaves/clients. This is a function ofspontaneous Bluetooth networking. For example, inFig. 9, the device enabling Role C has taken on thisserver role by effectively facilitating the ShadowPlacewhich supports the social world. The shaded sphere inthe centre of the figure represents the Aware Ego Agentof Role C, enhanced with the extra functionality tosupport BlueTooth connections. If the four people rep-resenting Roles A, B, C, and D remain in proximity,interaction can be facilitated. If any of these roles dropout of range, however, others may continue to interactwith one another. However, if C drops out of range ofthe other roles, they will all lose their connections andanother master/server has to be instantiated from thegroup members remaining in range.

The negotiation of membership, roles and the focusof the work within social spaces are important issueshighlighted by the Locales Framework that will beconsidered in future enhancements to ShadowPlaces.

219

8.2 Mutuality in ShadowPlaces

Mutuality is not currently supported within the Shad-owFaces interface of the DigitalFriend, and so Shad-owPlaces represents an opportunity for an extensionthat will help support a shared sense of place amongusers. Part of the issue here is that the users describedand presented in Sect. 7 tend to fluctuate in and out ofrange of base-stations. If this is the case, it is still nec-essary to consider them (and represent them perhapsvisually) as within the group, even when out of range.This situation, of being out of range but consciouslywilling to be active, is captured by Mutuality.

The continuous (24·7) nature of mobile users’interaction means that often members of a social worldwill be present, though at some reduced level of par-ticipation. For example, members could be representedas asleep. The individuals will have specific capabilities,dependent on their resources and abilities and maychoose to represent themselves to others in particularways. These capability and choice factors are aug-mented and supported by the digital representation ofthemselves, or their Digital Friends. These ‘‘selves’’ canbe configured to support different levels of participa-tion in a social world, via the ability to monitor, alertand notify the individual when certain events andtriggers occur. Here, Locales has highlighted importantissues in member representation and interaction in asocial world.

8.3 Individual views in ShadowPlaces

The DigitalFriend agent system supports multiple viewsof self and self’s engagement in activities, well beyondthe representation of an individual’s involvement in asingle social world. The recursive structure of sub-selvesin the underlying ShadowBoard architecture reflects therecursive structure of social worlds and sub-worlds de-scribed in Sect. 7 . As noted earlier, sub-selves in thepsychology drawn upon often equate to roles in a life,the same roles that social worlds cast upon their mem-bers. Just as the membership of a social world can bedescribed as formal and persistent to informal andtransient, so can the sub-selves supported by Shadow-Board. Thus the sub-selves represented by Shadow-Board, when introduced into a social world supportedby ShadowPlaces, can represent the user’s focus anddegree of participation in that social world.

The DigitalFriend built upon the ShadowBoardarchitecture is designed to enable the individual to par-ticipate in multiple social worlds. This feature of aShadowBoard agent is the one that distinguishes it fromother currently available agent architectures and thatwhich makes it a good candidate for supportingsophisticated agent-enabled interfaces offering groupinteraction. In this case, Locales has highlighted theimportance of enabling views across the social worlds inwhich an actor is involved.

8.4 Interaction trajectories in ShadowPlaces

In ShadowPlaces—with agents-in-the-loop—many ofthe interactants are software agents. Whether actionsand interactions supported by ShadowPlaces are per-formed by human actors or member sub-agents, theunderlying ShadowBoard architecture is capable ofcapturing the temporal nature of actions and interac-tions through recording past actions from the actionstack. Envisaged future courses of action are representedby goals embedded in the underlying agents and plansare, again, represented by queries embodied in EoCagents. These are examples of trajectories that can becaptured by the technology described here. It is impor-tant for group interaction to facilitate support for thetemporal nature of interaction supported by Shadow-Places. Although certain aspects of trajectories are cur-rently captured or held briefly by the DigitalFriend forits internal computation, further opportunities exist toutilise these in ongoing interactions among both agentsand users. Here Locales has shown the importance ofcapturing and representing interactions among agentsand human actors alike, over time.

9 Conclusion

This paper has presented efforts in drawing inspirationfrom Psychology, Sociology and Computer Science (AI)to establish an architecture, a methodology and a tech-nology for building new types of software applications,with new opportunity for universal access, centred onenhancing and empowering the individual. It then pre-sented an enhancement to these existing milestones, toenable the social worlds in which the individual partic-ipate, to be better represented within them. The en-hanced architecture—ShadowPlaces—was thenevaluated for further opportunity for improvement,using Fitzpatrick’s Locales Framework.

The DigitalFriend software, together with the Shad-owBoard methodology, are now available as useful,complementary generic tools for building a diverse rangeof end-user developed applications. These applicationscan potentially be very broad ranging, given the initialemphasis on addressing the diversity aspect of UniversalAccess. However, the web services that are currentlyavailable to the general Internet population as candidatesub-agents are limited in both number, and in imagi-native uses of knowledge in most domain areas. Thisproblem will hopefully subside as web services continueto gain in popularity, as a way to provide distributedfunctionality via programmatic interfaces.

The complex user model gleened from psychology,initially developed for the underlying ShadowBoardarchitecture, has paid significant dividends in both theuseful and timely individual-oriented ShadowBoardmethodology for analysis and design of a Digital Friend.It has also proved to be a successful way to identify and

220

filter the messages and data from sub-agents to the user,within the interface of the DigitalFriend tool itself(Fig. 3). This continual use of the role and sub-rolehierarchy reinforces the user model to the end-user, whowill often also be the developer and ongoing maintainerof the Digital Friend. It provides a comprehensive ap-proach in dealing with end-user applications, tailoredfor an individual user 24·7.

Concepts from Sociology are only now beginning tobe adopted in order to extend the system functionality(ShadowPlaces) into the realm of spontaneous networksin mobile applications, an area where the approachholds significant promise.

Prior to the conducted investigation into the LocalesFramework, conventional multi-agent systems(MAS)—where each agent is autonomous—appeared tobe an appropriate direction to take. Conventional MAShave been successfully applied to robot soccer teams andto numerous eCommerce applications. However, build-ing an individual agent with significant complexity isaimed at augmenting a human user in 24·7, not atbuilding a fully autonomous synthetic agent. Theresultant software agency in the DigitalFriend mapsnicely onto social worlds occupied by humans as de-scribed by Strauss [36], transforming them into socialworlds that are supplemented with assistive technology.As noted in Sect. 8.2, the Psychology of Subselves behindShadowBoard and the sociology behind Social Worlds,are two sides of the same coin—people take on roles,formal or otherwise, as they negotiate the fluidity ofmultiple social worlds. Given the rich user modellingemployed in the ShadowBoard methodology andinstantiated within the DigitalFriend, there is anopportunity for intelligent client-side technology draw-ing upon Web Services, operating on mobile devices toenhance and support interaction among different abledusers. In many cases, all users are partially abled (forexample, a mobile user may be partially sighted for atime), due to the rapidly changing environment, or dueto the multiple social worlds the user is involved in, eachdemanding participation.

The ShadowBoard research and developmentinvestment has provided an appropriate methodologyand computer technology, while the Locales Frameworkhas been a useful analytical tool to extract additionalrequirements for the ShadowPlaces extension to theexisting system, to make it a locale for mobile and In-ternet users, in 24·7 time.

Future research activity will investigate severaldirections for improvements to both the methodologyand the tools. The applicability of Semiology—the studyof signs—to user interface analysis and design [1, 27, 33],is likely to provide dividends. It was mentioned earlierthat as the user moves amongst multiple social worlds,much of the shared symbolisation in collective activitiesinvolves a common understanding of signs and lan-guage. In addition, with the strong emphasis on icons inthe DigtialFriend interface, it is expected that insightsand synergies will be found in the area of Semiology.

Furthermore, as seen in the example represented inFigs. 7 and 8, the flow of communication actions, acrossinter-related sub-agents, suggests that investigation intoTask Analysis [5], data modelling [2] and workflowmodelling are likely to provide further insights in con-structing more complex and useful sub-agents, and inenhancing the ShadowBoard methodology.

Acknowledgements Much of this research has now been appliedin the realisation of the software tool DigitalFriend V1.0. It is aproduct of a development effort titled The Digital Self Project,an open source project funded by a Telstra Broadband FundDevelopment Grant, provided by Telstra Ltd, Australia. TheDigitalFriend product is the first software tool from the effort tobecome available for public use, and can be downloaded fromwww.DigitalFriend.org

References

1. Anderson PB (1997) The theory of computer semiotics. Cam-bridge University Press, Cambridge

2. Benyon D (1997) Information and data modelling, 2nd edn.McGraw-Hill, England

3. Bradshaw J (1997) Software Agents. MIT Press, Cambridge4. Caroll JM (1991) Designing interaction: psychology at the

human-computer interface. Cambridge University Press, NewYork

5. Diaper D, Stanton N (eds) (2003) The handbook of taskanalysis for human-computer interaction. Lawrence ErlbaumAssociates, London, 650p

6. DigitalFriend Web Site (2005) DigitalFriend user manual,Document available at http://www.DigitalFriend.org/ Cited,April 2005

7. Fitzpatrick G (2003) The locales framework: understandingand designing for wicked problems. Kluwer, London

8. Fitzpatrick G (1998) The locales franework: understanding anddesigning for cooperative wor. Doctoral dissertation, Univer-sity of Queensland, Brisbane

9. Fleming M, Cohen R (1997) A user modeling approach todetermining system initiative in mixed initiative AI systems. In:Proceedings of the 6th international conference on user mod-eling, UM’97

10. Goschnick SB (2003) Enacting and an agent-based digital self ina 24·7 web services world. In: Proceeding ISMIS-2003 the 14thinternational symposium on methodologies for intelligent sys-tems, Maebashi, Japan, Springer LNAI, vol 2871, pp 187–196

11. Goschnick SB (2001) ShadowBoard: an agent architecture forenabling a sophisticated digital self. Thesis, Deparment ofComputer Science, University of Melbourne, Australia, 199p

12. Goschnick SB (2000) ShadowBoard: a whole-agent architecturethat draws abstractions from analytical psychology. In: Pro-ceedings of PRIMA 2000, August, Melbourne, Australia

13. Goschnick SB, Sterling L (2003) Enacting and interacting withan agent-based digital self in a 24·7 web services world. In:Proceedings, IEEE joint conference on Web Intelligence andIntelligent Agent Technology (WI/IAT), Halifax, Canada

14. Graham C, Kjeldskov J (2003) Indexical representations forcontext-aware mobile devices. In: Proceedings of the IADIS e-Society conference 200 3, Portugal, IADIS Press, pp 373–380

15. Greenberg S, Fitzpatrick G, Gutwin C, Kaplan S (1999)Adapting the locales framework for heuristic evaluation ofgroupware. In: Proceedings of OzCHI’99, Nov, pp 30–36

16. Gunderson J (2004) W3C user agent accessibility guidelines 1.0for graphical web browsers. Univ Access Inform Soc 3(1):38–47

17. Libby D (1999) RSS 0.91 Specification. http://my.net-scape.com/publish/formats/rss-spec-0.91.html. Cited 2005

18. Jin T, Goschnick SB (2003) An agent-based transaction modelin web services. In: Proceedings Web-services and agent-basedengineering, workshop at AAMAS, Melbourne

221

19. Johnson P (1992) Human computer interaction: psychology,task analysis and software engineering. McGraw-Hill Interna-tional Ltd, UK, pp 114–133

20. Kakihara M, Sørensen C (2002) Mobility: an extended per-spective. In: Proceedings of the 35th hawai international con-ference on system sciences (HICSS-35). Hawaii, USA, IEEE

21. Kay J, Thomas RC (2000) Personal usability based upon ascrutable, dynamic, individual user model. In: Proceedings ofOZCHI-2000, Sydney, Australia, pp 292–298

22. Kristoffersen S, Ljungberg F (1999) Mobile use of IT. In:Proceedings of the 22nd information systems research seminarin Scandinavia (IRIS22), Jyvaskyla, Jyvaskyla UniversityPrinting House, Finland, pp 271–284

23. Lane S (2004) User interfaces for navigating informationhierarchies. Honours Thesis, Department of Information Sys-tems, University of Melbourne, Australia, 77p

24. Lewis M (1998) Designing for human-agent interaction. AIMagazine, Summer 1998, pp 67–78

25. Marriott K, Stuckey PJ (1998) Programming with constraints:an introduction. MIT, Cambridge

26. Mol A, Law J (1994) Regions, networks and fluids: anaemiaand social topology. Soc Stud Sci 24(4):641–671

27. Oliveira OL, Baranauskas MC (1999) Communicating entities:semiotic-based methodology for interface design. In: Proceed-ings of the 8th international conference on human-computerinteraction, Munich

28. Rosson MB, Carroll JM (2001) Usability engineering: scenario-based development of human-computer interaction. MorganKaufmann, San Francisco

29. Scerri P, Pynadath D, Tambe M (2002) Why the Elf actedautonomously: towards a theory of adjustable autonomy. In:International conference on autonomous agents and multi-agent systems

30. Shibutani T (1955) Reference groups as perspectives. Am JSociol 60:522–529

31. Shneiderman B (1997) Designing the user interface, strategiesfor effective human-computer interaction, 3rd edn. Addison-Wesley, USA

32. SOAP Version 1.2 Part 0: Primer (2004) W3C candidate rec-ommendation, http://www.w3.org/TR/soap12-part0/ Cited,2004

33. Souza CS (1993) The semiotic engineering of user interfacelanguages. Int J Man Mach Stud 39:753–773

34. Sterling L, Shapiro E (1994) The art of prolog, 2nd edn. MITPress, Cambridge

35. Strauss A (1993) Continual permutations of action. Aldine DeGruyter, New York

36. Strauss A (1978) A Social World Perspective. Stud SymbInteract 1:119–128

37. Strauss A (1976) Life styles and urban space. In: ProshanskyHM, Ittelson WH, Rivlin LG (eds) Environmental psychology:people and their physical settings. Holt, Rinehart& Winston,New York, USA, pp 528–537

38. Vetere F, Howard S, Pedell S, Balbo S (2003) Walking throughmobile use: novel heuristics and their application. In: Pro-ceedings of OzCHI 2003, Brisbane, Australia

39. Web services description language (WSDL) Version 1.2 (2004)W3C working draft, http://www.w3.org/TR/wsdl12/ Cited 2004

40. World wide web consortium (2004) User agent accessibilityguidelines. http://www.w3.org/TR/UAAG10/. Cited 2004

41. World wide web consortium (2004) Working draft: how peoplewith disabilities use the web, July, 2004. http://www.w3.org/WAI/EO/Drafts/PWD-Use-Web/. Cited 2004

42. World wide web consortium (2001) World wide web consor-tium document object model (DOM). Documentation availableat http://www.w3.org/DOM/.Cited 2004

43. Wooldridge M, Jennings N (1995) Intelligent agents: theoryand practice. Knowl Eng Rev 10(2):115–152

44. Young RM (1983) Surrogates and mapping: two kinds ofconceptual models for interactive devices. In: Gentner D, Ste-vens AL (eds) Mental models. Lawrence Erlbaum Associates,Hillsdale

45. Zukerman I (2001) An integrated approach for generatingarguments and rebuttals and understanding rejoiners. In: Pro-ceedings of the 8th international conference on user modeling,UM 2001

222