ORIGINAL ARTICLE

Web system requirements: an overview

Received: 29 May 2001 / Accepted: 27 September 2002 / Published online: 6 February 2003� Springer-Verlag London Limited 2003

Abstract The development of online and Web-basedsystems is becoming increasingly critical to the businessstrategy of many organisations. Although this develop-ment can be viewed as a form of software development,it has various unique characteristics. One of the mostsignificant is the uncertainty in domain understandingby both clients and developers. Another is the way inwhich Web solutions typically lead to changes in busi-ness models and hence requirements. These are mani-fested as significant volatility in the requirements ofthese systems. In this survey paper we investigate thehandling of requirements for Web systems includinglooking at both commercial practice and current re-search. We report on the outcomes of a comprehensiveset of interviews and follow-up surveys, and argue thatthe Web system requirements process needs to be fun-damentally different from more conventional systems,incorporating the design process into the identificationof requirements resulting in a design-driven require-ments process.

Keywords Domain uncertainty Æ Online systems ÆRequirements Æ Volatility Æ Web systems

1 Introduction

As user demand for the provision of online services grows,the ability to develop Internet-enabled systems isbecoming increasingly crucial [1, 2] to both businesssuccess and to the provision of social and governmentservices. Investment in these systems is growing at aphenomenal rate (see http://cyberatlas.internet.com/ forexample statistics illustrating this growth). The systems

being developed leverage the infrastructure of the Internetand an increasingly complex set of Web standards,protocols and technologies to provide sophisticatedbusiness solutions that merge Web-based front-ends withcomplex back-end software.

The nature of online and Web-based applicationstends to be extremely diverse, covering applications suchas information provision, e-commerce, collaborativeand community building and communications. Many(though certainly not all) Web-based applications sharea common characteristic: they have a significant impacton the nature of the interaction that an organisation haswith its clients and other external stakeholders (such asbusiness partners and suppliers). This impact will typi-cally lead to a change in both the fundamental businessprocesses and in the business model that relies on theseprocesses. While this is true of some non-Web systems(such as customer relationship management systems),most ‘conventional’ software systems can have signifi-cant internal impacts on the organisation, but do notsubstantially change the way in which the organisationinteracts with its clients, and hence is unlikely to affectthe core business model. In effect, we see Web-basedsystems as (typically, though admittedly not always)being an exemplar of those systems where the nature ofthe solution being developed fundamentally changes theproblem being addressed and hence affects the require-ments of the solution! In effect, the problem and thesolution are mutually constituted, as they are bothinterdependent. Later in the paper we revisit this issue,and describe the supporting development process as adesign-driven requirements process.

In this paper, when we discuss Web-based systems weare predominantly referring to that class of system wherethe system changes the nature of the external businessinteractions and hence the business model. This willencompass e-commerce applications, but also applica-tions such as customer management, online inventoryand supply-chain management. Much of what is dis-cussed will also be relevant to non-Web-based applica-tions that share this core characteristic.

Requirements Eng (2003) 8: 102–113DOI 10.1007/s00766-002-0153-x

David Lowe

D. LoweUniversity of Technology, Sydney,PO Box 123, Broadway 2007, NSW, AustraliaE-mail: david.lowe@uts.edu.auTel.: +61-2-9514-2526

Returning to the issue of the growth of these appli-cations, and the obvious commercial impacts of failureto deliver effective solutions, we argue that little researchhas focused on understanding the processes for under-taking development of these Web-based systems. Simi-larly, there has been a lack of attention on improving thecost effectiveness of these development processes. This isdespite the fact that there are (as we shall discuss later)significant differences between Web systems and con-ventional software systems (discussed below).

As we shall consider in some length later in the paper,development practices from related domains do nottypically address these differences particularly well. Ir-respective of this there has been a tendency to adoptspecific development practices from various constituentdomains that contribute to Web projects (the choiceoften being dictated by the disciplinary background ofthe project management). For example, Web develop-ment companies often adapt approaches from softwareengineering (such as Rapid Application Development[3]), graphic design or marketing [4] – with only minorattempts to integrate all of these elements.

One of the key areas for which this often createsproblems is the handling of requirements. Web projectsoften have a considerable degree of domain uncertaintyand requirements volatility. Both the rapid pace oftechnological change and the significant impacts of thesesystems on business models and processes lead to a lackof clarity from both developers and clients. The result isthat clients not only have difficultly articulating theirneeds, but difficulty also in even understanding whattheir needs are, particularly given that the introductionof these systems often leads to changes in businessprocesses or models, and hence leads to subsequentchanges in the requirements.

Conventional development processes do not handlethis issue well. Typical requirements processes arelargely designed to elicit requirements rather than eitherto support the development of domain understanding orto assist in understanding the impacts of a particulardesign. This is not a criticism, as this is not somethingthat is intended to be the focus of requirementselicitation processes (and is often not a necessity inconventional development). It does, however, presentdifficulties when the solution under development affectsthe nature of the problem being addressed. While workon domain understanding (particularly approaches suchas Soft Systems Methodology [5] which attempt tosupport a systems analysis of processes in which tech-nological processes and human activities are interde-pendent) assists in understanding the nature of theproblem, it still does not clarify the impact of a solutionon the problem domain.

Even incremental and iterative development pro-cesses that provide client feedback mechanisms havedifficulty handling a situation where the requirementsare simply not known, or are understood initially butsubject to change as a result of a particular solution.This is largely because the feedback is predominantly

focused on determining whether the emerging systemmeets the known requirements. They also largely fail tomanage the requirements volatility that results from anevolving client understanding of needs.

In this paper we consider these issues, and howrequirements might be managed for Web projects. Webegin by looking at existing research, including castingthe net rather widely into the literature from variousrelated disciplines. We then move on to consider currentcommercial practice, exploring the results of a wide-ranging set of industry interviews and surveys, as wellas looking at commercial development guides anddescriptions of best practice. We discuss the view thatthere is a strong division between current research andcommercial practice.

We conclude with two key points. The first is topropose the hypothesis that the design process can, andshould, play a much greater role in managing thedomain uncertainty and requirements volatility. Thesecond is the claim that in considering requirements weneed to develop a much clearer understanding of therelationship between business architectures, informationarchitectures and technical architectures.

2 Web requirements literature

Although there is a significant volume of research onrequirements management and requirements engineeringapproaches, little of this has filtered through into aspecific consideration of requirements for Web projects.The small amount of research that does exist tends tocome from a very diverse set of disciplines: softwareengineering, publishing, marketing, business informa-tion systems, etc. This tends to complicate the analysis ofexisting work, as these different discipline areas typicallyhave distinct terminology, notations and models. As aresult similar concepts are described quite differently inthe literature belonging to different disciplines, makinganalysis of this area difficult.

One common theme is that the adoption of a struc-tured, well-organised approach is important for ensuringa predictable, repeatable, manageable and cost-effectivedevelopment, though this has emerged more clearly inindustry publications than in the research literature (see,for example, [6]).

A logical place to commence understanding the Webrequirements process is to understand the differencesbetween Web systems and more conventional softwaresystems, particularly with respect to the impacts on thedevelopment process.

2.1 Unique characteristics of Web systems

There is a growing body of literature regarding the dif-ferences between Web systems and conventional soft-ware systems. In general, this literature identifies uniquecharacteristics of Internet-enabled systems that are both

103

technical and organisational. For example, the technicalstructure of Web systems merges a sophisticated busi-ness architecture (that usually implies significantchanges to the business model of the client) with both acomplex information architecture and a highly compo-nent-based technical architecture [2].

The distributed nature of the Internet, the visibility ofWeb systems to external stakeholders, the rapidlychanging nature of the underlying technologies and thelightweight component-based structure of most Websystems often means that the technology is much morevisible to the users of the systems. This also means thatthe linkage between the business architecture and thetechnical design of the system may be much tighter thanfor conventional software systems. Similarly, the infor-mation architecture, which covers aspects such as thecontent viewpoints, interface metaphors and naviga-tional structures, is substantially more sophisticatedthan conventional software systems.

More fundamental, however, are some of the or-ganisational characteristics that are either unique orheightened in Web systems [7]. Two of these are uncer-tainty in the project domain and volatility of the clientneeds and available technology. With Internet-basedsystems, the technology, development skills, businessmodels and competing systems are changing so rapidlythat the domain is often not only poorly understood butalso constantly evolving [8]. This provides a substantialdegree of uncertainty in the project context, and conse-quently makes resolving requirements very problematicand the requirements that do exist tend to be very vol-atile. Specifically, clients’ understanding of not only thetechnology’s capabilities but also their own needs typi-cally changes dramatically during the course of a pro-ject. In particular, many Web projects are vision drivenrather than needs driven, leading to an initial lack ofclarity. This is also coupled with business models thatare evolving rapidly as organisations migrate to an in-creased reliance on Internet technologies [1]. These is-sues are exacerbated by the fact that Web projects tendto have a short time-to-market, and are not currentlywell supported by development tools and techniques.

A number of additional specific differences have alsobeen highlighted in the literature. These are discussed inmore detail elsewhere [4, 7, 9]:

– Short time frames for initial delivery. Web develop-ment projects often have delivery schedules that aremuch shorter than for conventional IT projects. Thisis partly a consequence of the rapid pace of tech-nological development and partly related to therapid uptake of Web systems.

– Highly competitive. Web projects tend to be highlycompetitive. This is, of course, not new, being typicalof the IT industry in general. The nature of thecompetitiveness is, however, somewhat different.There is regularly a perception that with simple Webauthoring tools anyone can create an effective site.This creates inappropriate expectations from clients

coupled with numerous small start-up companiesclaiming to be doing effective Web design but inreality offering little more than a combination ofHTML skills and rudimentary graphic design. Theresult is a highly uninformed competitiveness.

– Fine-grained evolution and maintenance. Web sitestypically evolve in a much finer-grained manner thanconventional IT applications. The ability to makechanges that are immediately accessible to all userswithout their intervention means that the nature ofthe maintenance process changes. Rather than aconventional product maintenance/release cycle, wetypically have an ongoing process of contentupdating, editorial changes, interface tuning, etc.The result is a much more organic evolution.

– Increased emphasis on user interface. With conven-tional software systems users must make an (oftenconsiderable) investment in time and effort to installand learn to use an application. With Web applica-tions, however, users can very quickly switch fromone Web site to another with minimal effort. Assuch, the need to engage users and provide muchmore evident satisfaction of users’ needs andachievement of their objectives becomes critical. Theresult is an increased emphasis on the user interfaceand its associated functionality.

– Increased importance of quality attributes. Web sys-tems represent an increase in mission-critical appli-cations that are often directly accessible to externalusers and customers. Flaws in applications (be theyusability, performance or robustness) are thereforemuch less able to be ‘hidden’ and hence much moreproblematic.

– Open modularised architectures. Although not uniqueto Web applications, it is still worth mentioning theemphasis that is typically placed on open and mod-ularised architectures for Web systems. They areoften constructed from multiple COTS (commercialoff-the-shelf) components that are adapted and in-tegrated together. Indeed, strong integration skillsbecome much more critical in most Web projects.

– Rapidly changing technologies. The technology thatunderpins most Web systems is changing very rap-idly. This has several consequences. The first is thatit increases the importance of creating flexible solu-tions that can be updated and migrated to newtechnologies with minimal effort. For example, theneed for reusable data formats (such as XML) in-creases substantially. A second consequence is thatdevelopers’ understanding of these technologies isoften restricted, increasing project risks.

– Highly variable client understanding. It is extremelycommon for clients’ understanding of the systemcapabilities to evolve and improve during a project.This typically means that the clients’ understandingof their own needs, and their expectations of theproject, grow during the course of the project. Thiscan exacerbate problems with requirements ‘creep’

104

and make developing a product that satisfies userexpectations extremely difficult. A related problem isthe poor understanding that is often typical of clientsearly in a project, making initial system definitiondifficult. This increases the importance of incre-mental and prototyping approaches.

Recognising that Web systems have some uniquecharacteristics, a number of researchers have researchedspecific developmental methodologies. The methodolo-gies have, however, largely focused on specific elementsof the system: information modelling, functionality,business models, etc. We can begin by considering someof these approaches.

2.2 Information modelling

One of the most obvious differences between Web sys-tems and more conventional software systems (i.e. thosewhere the system does not substantially change the wayin which the organisation interacts with its clients) is theway in which information is managed and utilised. It isnot surprising, therefore, that much of the earliest workon Web development techniques focused on informationmodelling and structuring.

Early approaches in this area evolved out of workon Entity-Relationship modelling and applied this tomodelling the information domain associated withapplications. Indeed much of this work predates theWeb and focused on hypermedia design. For example,RMM (Relationship Management Methodology [10])claims to provide a structured design methodology forhypermedia applications. In reality, the focus is verymuch on modelling the underlying content, the userviewpoints onto this content and the navigationalstructures that interlink the content. OOHDM (Ob-ject-Oriented Hypermedia Design Model [11]) is asimilar approach, though somewhat richer in terms ofthe information representations and based on object-oriented software modelling approaches. Other similarexamples include EORM [12] and work by Lee [13].WSDM [14] attempts to take these approaches onestep further, by beginning more explicitly from userrequirements, but these are only addressed in a veryrudimentary fashion. In general, these techniques wereeither developed explicitly for modelling informationin the context of the Web, or have been adapted tothis domain.

More recently, work on WebML (Web ModellingLanguage [15]) has begun to amalgamate these conceptsinto a rich modelling language for describing Web appli-cations. However, despite its aim to support comprehen-sive descriptions, the focus (as with the above techniques)is very much on content modelling rather than describingthe functionality that is a key element of most currentcommercial Web systems. One of the few approachesthat attempts to integrate content representation withfunctionality is that of Takahashi and Liang [16].

Further, these approaches generally provide reason-able modelling notations for representing the informa-tion architectures but rarely succeed in providingeffective support for identifying the requirements thatdrive this architecture. The modelling approaches arelargely design focused. An exception to this is some ofthe work extending OOHDM, to include tools such asuser scenarios and use cases which look at how a systemis likely to be used [17] and develops content models thatsupport this usage. Another exception is work thatfocuses on supporting maintenance processes such asthat of Gellersen et al. [18].

There has been some limited work on specific ap-proaches to formally representing certain aspects of Webapplications, though this has tended to focus again oncontent and navigational issues to the exclusion offunctionality. For example, Hadez [19, 20] looks at theuse of formal methods (using the Z notation) to specifyconceptual, structural and perspective schemas. Otherapproaches have focused on specification of timingconstraints [21, 22] rather than content structure. Again,however, the focus is very narrow and fails to couple thespecifications with broader application requirements.

2.3 Software engineering methods

Whereas information modelling approaches focus oncontent modelling, software engineering approachestend to focus on understanding required functionalityand how systems to support this functionality can bedesigned. As with information management techniques,a number of researchers have looked at the adaptationof conventional software engineering to support Webdevelopment.

Much of this work has looked at the extension ofmodelling languages to support Web concepts, mostnotably work on adapting UML (Unified ModellingLanguage – an emerging industry standard for model-ling object-oriented systems). Often this work simplyuses the notation to effectively model informationstructures (such as [23]) and does not integrate this withthe modelling of Web site functionality. One exception isthe work by Conallen [24], which attempts to look at thelink between content and functionality: in particular,how the software components generate content and aretriggered by navigational events. Again, though, theapproach is restricted. The modelling is not particularlyscalable, nor are effective abstractions provided formodelling the content (the content modelling is essen-tially restricted to representing individual Web pages).Even more problematically, the modelling allows rep-resentation of low-level designs, but does not provideany indication of how this design is developed, nor itslink to client requirements.

Other researchers have used UML in rather differentways. For example, Vilain et al. have adapted UML torepresenting user interactions [25]. Although this work isnot explicitly addressing Web requirements or design,

105

user interaction is a key element of Web systems. Indeedwork such as this is moving closer to providing tools forunderstanding client requirements. The authors of thiswork see it as supporting use cases a UML tool aimed atunderstanding how a system will be utilised. Other ex-amples of work that has adapted UML include Russell[2], Lord [26] and Fournier [27]. In effect, the approachesbased on UML provide a useful notation, but still mustbe integrated into a broader framework.

The above work has emphasised relatively detailedmodelling of designs. An alternative thread of researchhas looked at modelling Web systems at a more abstractlevel. In particular, there is a growing body of worklargely emerging from large technology vendors such asIBM, Sun and Microsoft that considers supportedbusiness functions and the broad technical architecturesrequired to support these. Probably the most mature ofthese approaches is the patterns for e-business workbeing developed by IBM (see http://www.ibm.com/framework/patterns/). This work provides a frameworkfor identifying common patterns of business models. Asstated in Lord [26]:

The paths to creating e-businesses are repeatable.Many companies assume that they are uniqueand that therefore every creation of an e-businesshas to be learned as you go. … in fact, there arelessons and architectural paths or patterns thatcan be discerned from all these engagements.Whether your company is a startup or has ex-tensive legacy applications, these patterns allowyou to reuse existing technologies so that yourprojects can be completed quickly.

For each business pattern, a number of logical ar-chitectures (or topologies) are defined. These topologiesprovide a mechanism for fulfilling a particular businessneed. In terms of requirements, the business patterns canhelp clients understand potential applications and theassociated functionality. The approach does not, how-ever, provide an explicit process for supporting the de-velopment of this understanding. Another problem isthat the architectures tend to emphasise functionality,with little consideration of the information architecture.In particular aspects such as content modelling, infor-mation viewpoints, etc. are not addressed.

All of the above techniques support the design ofWeb systems albeit at significantly varying levels ofabstraction. They do not provide supporting processesfor handling the requirements. Conventional softwareprocesses tend to assume that requirements are knownto clients, and simply need to be elicited and analysed.These processes usually differentiate between userrequirements (often formalised in a URD or userrequirements definition) that capture the user under-standing of their needs, and the system specification(SRS or system requirements specification) that repre-sents the system that will meet these needs [28]. In effectthe two documents are different representations of thesame concepts. A typical process will be to elicit

requirements (which are documented in the URD), andthen analyse these requirements to construct the SRS,iterating to refine the URD as necessary.

One significant difficulty with this paradigm is that itpresumes that clients either understand their require-ments, or at the very least understand the problem thatis being addressed. Even when clients are not able toarticulate their requirements precisely, they are at leastable to understand whether a given design will addresstheir needs. In cases such as these, the design maycommence prior to full resolution of requirements. Thedesign will then be used to ascertain (from client feed-back) whether the proposed solution addresses theidentified need. This is problematic for Web projects,where many clients not only have a poor understandingof their requirements, but they also have a poor under-standing of the problems being addressed by the newsystem. In these circumstances, simply using a design toclarify whether it addresses the problem will be insuffi-cient, as the problem itself is only poorly understood.

As an illustrative example of these issues, consider theincreasing use of lightweight development processes forsoftware projects [27, 29, 30]. One of the approachesreceiving the most attention is the use of XP (eXtremeProgramming) [31]. XP is based on the incremental de-velopment of partial (or ‘spike’) solutions that are usedto subsequently resolve specific requirements. Thesespike solutions are then integrated into the evolvingsystem through refactoring of the current solution toincorporate these components. When used in conven-tional software development XP has proven to be ef-fective for projects that are initially ill defined [31], acharacteristic of many Web projects. As a result, manyof the proponents of XP and similar approaches see it isan ideal approach to be adopted for Web development[32].

There are, however, certain problems that restrictapproaches such as these to Web projects. The first isthat a number of studies (see, for example, [33, 34]) haveshown that approaches such as XP only work effectivelyfor projects that have cohesive development teams. Thisis often not the case with Web projects, which often lackcohesiveness between the technical development and thecreative design as a result of the disparate disciplinarybackgrounds of the development team members. XP canalso result in a brittle architecture and poor documen-tation, which makes ongoing evolution of the systemdifficult – something that is important for Web systems.Finally, and perhaps most fundamentally, XP utilisespartial solutions to resolve uncertainty in requirements,but does not inherently handle subsequent changes inthese requirements (i.e. requirements volatility) as thesystem evolves. This creates problems for Web systems,where the emerging design results in an evolving clientunderstanding of their needs, and hence volatile re-quirements [35].

In effect, conventional software engineering processessee requirements as preceding and driving the designprocess. Even where an iterative or incremental

106

approach (such as XP) or a spiral approach (involvingmultiple feedback loops) is adopted the design is viewedas a way of assisting in the identification and validationof requirements, but rarely does it help the client toactually formulate their needs. In Web development, thesituation is fundamentally different. The design processnot only helps developers and clients articulate theneeds, but also helps clients understand the system do-main and formulate their understanding of their needs.In effect, the design allows developers to manage theinherent uncertainty and volatility.

There has been significant work on considering re-quirements volatility within conventional requirementsengineering (see, for example [36]) and especially ontailorable systems where requirements change leads tothe need to evolve systems (see [3]). This work, whileuseful, does not explicitly consider the ways in whichdesign itself changes the nature of the requirements andhence leads to a closed loop in the development.

Approaches that focus on domain understandinghave certain merit in the context of the changing re-quirements of Web-based systems. For example, SoftSystems Methodology [5, 37] attempts to support asystems analysis in which technological processes andhuman activities are interdependent. This, however, stillemphasises design of systems (albeit composite techno-logical/human systems) but does not explicitly addressthe issue of the way in which the system will ultimatelychange the nature of the business problem.

One attempt to integrate support for Web projectsdirectly into the development process is work on Web-OPEN. OPEN (Object-oriented Process, Environmentand Notation) is a third-generation, public domain ap-proach that was designed for the development of soft-ware-intensive applications, particularly object-orientedand component-based development [38, 39]. The OPENprocess framework (OPF) supports the instantiation ofspecific development processes from a set of work units(activities, tasks and techniques), work products andproducers. The work on WebOPEN extends OPEN toinclude additional tasks [40] and roles and techniques[41] that are applicable to Web projects. Although asignificant step forward, this work is limited insofar as itdoes not inherently provide guidance in structuring theprocess to deal with the requirements volatility. Thisremains an open issue within the software engineeringliterature.

2.4 Further afield

A number of related areas can potentially providesome interesting insights into the specification anddesign of web systems. These include areas as disparateas publishing, marketing, architecture and gardening.

Beginning with publishing, we can investigate aspectssuch as publishing guidelines and what these say aboutthe development (i.e. authoring) focus. Whereas soft-ware processes emphasise an understanding of (and

subsequent addressing of) required functionality, pub-lishing looks more at characterising audiences and theirreaction to the material being authored. For example,the following fragments, describing what should beincluded in book proposal, are extracted from a numberof publishing guidelines made available by numerouspublishers:

– A one-page overview of the book– A chapter-by-chapter outline– Author’s background/credentials– The audience you wish to reach– Market/audience information– A list of competing works– Who is the audience– Competing books and how this differs– How the book helps the reader– Expected delivery date of manuscript

If we replace book by site, reader by user, and authorby developer then most (if not all) of the above would bedirectly relevant to understanding the focus of a Website. It is interesting to note that at the proposal stagethere is almost universal absence of any consideration ofaspects such as detailed content, stylistic considerationsand production processes.

Areas such asmarketing and advertising are also likelyto be able to provide useful insights focusing on design-ing explicitly for user engagement, conveying a desiredmessage, and managing user or consumer reactions.

One unusual area that has been used as an analogyfor Web development is landscape gardening. Web sitedevelopment is often about creating an infrastructure(laying out the garden) and then ‘tending’ the informa-tion that grows and blooms within this garden. Overtime the garden (i.e. Web site) will continue to evolve,change and grow. A good initial architecture shouldallow this growth to occur in a controlled and consistentmanner. This analogy has been discussed in terms ofproviding insights into how a site might be managed[42].

3 Current commercial practice

The above discussions indicate that the research litera-ture is extremely fragmented, with few attempts to drawthe work together into a cohesive picture. Nevertheless,Web development is carried out and carried out suc-cessfully in at least some cases. It is therefore worthwhilelooking at what actually occurs in commercial practice.We begin by looking at the results of a broadly focusedset of industry interviews and follow-up surveys.

3.1 Surveys and interviews

In order to develop a strong understanding of com-mercial practice, and in particular to investigate the

107

ways in which commercial practice in Web-based pro-jects differed from commercial practice in more con-ventional software systems development (particularly asit relates to management of requirements), we undertooka comprehensive set of industry interviews and follow-up surveys. These focused on the issues that are raisedduring the early stages of Web development projects,and the characteristics of systems that are resolvedduring these stages. Detailed results of these surveyshave been published elsewhere [43, 44, 45, 46]. We shallprovide an overview of the results here.

3.1.1 Background

A significant volume of data was collected in the form ofboth interviews and surveys. The interviews were in-tended to identify general perceptions and qualitativetrends and were conducted primarily over a period of 6weeks during April/May 2000. The interviews weretypically 20–40 min and involved a series of questionsfocusing on interactions with clients and the processesfor understanding client needs. The surveys were in-tended to capture more quantitative information. Thesurvey was either completed as a follow-up to the in-terview or as a stand-alone survey. The survey data wascollected during the period April–June 2000. The surveyconsisted of 15 sets of detailed questions concerningcompany profiling, project profiling, client relationshipsand development practices.

The respondent companies that took part in the in-terviews and surveys covered a broad spectrum of de-velopment areas: multimedia development, Internetdevelopment and intranet development. Similarly, theircore business varied considerably, from consulting tocontract development to in-house development. Theapplication domains also covered a broad spectrum:financial institutions, medical practitioners, generalSMEs, top 500 corporations, finance, travel and tour-ism, legal, manufacturing, government and consumeradvisory services.

3.1.2 Cost, complexity and scale of projects

The projects of individual companies varied greatly incomplexity and cost (a reflection of the range of scale ofapplications being developed for the Web). In terms ofcost the range was from Australian $2000 to $50 million,with the average company working on contracts rangingfrom Australian $100,000 to $1 million. In correlatingcost to other measures or project scale, various factorshad poor correlation. In particular, both number ofsource documents and number of resultant site pageswere very poorly correlated to overall cost. Thefrequency of content changes had a surprisingly highcorrelation.

Numerous respondents felt that hidden costs were amajor issue in projects (average of 3.4 on a scale of

0=strongly disagree to 5=strongly agree), and thatclients have difficulty understanding the implications ordetails of project bids (average of 3.4).

The duration of projects also varied considerably: 2–3days on small projects, 2–4 months on medium-sizedprojects and 9–12 months on large-scale projects. Theaverage expected lifetime of systems was marginallyunder 18 months, with few projects expected to have alifetime of over 24 months. Most respondents felt thatprojects were generally completed on time, though anumber of factors impacted on this. In particular, theelements which most affected the ability to adhere to aschedule were: the existence of a detailed risk analysis;good project management; staff stability; clients meetingmilestones; scope creep.

There was a high level of outsourcing amongst most ofthe respondents. This extended to aspects such as graphicdesign, usability evaluation, functionality testing, script-ing or application development and development ofspecific functional areas (database development and ad-ministration, server management, etc.). Outsourcingsupported the gaps in respondent companies’ areas ofexpertise and tended to be the areas where contractualstaff and strategic alliances were called upon.

3.1.3 Client/developer interaction

Many of the respondents’ companies aim towards hav-ing only one point of client contact (56%) and using adesignated project leader rather than having multiplepoints of contact (44%). Almost all of these companiesalso stressed the importance of teamwork. The one clientliaison seemed to be important though the respondentswho opted for several leaders or team focusing hadstrong team processes in place.

There was a very strong feeling (average rating 4.4 ona 0–5 scale) that clients did not understand well the ca-pabilities of the technologies. Similarly it was felt (aver-age rating 4.2) that clients did not understand their ownneeds. Anecdotal comments indicate that clients are alsounaware of aspects such as the workload involved indeveloping Internet sites and the implications on theircompanies. If the clients’ liaison was from their market-ing team then the IT side was often poorly comprehended(and hence articulated and therefore specified). If theclients’ liaison was from the organisations’ IT team thenthe marketing or business side was not well understood.

Perhaps surprisingly, respondents felt that clients hada low understanding of their own organisations andexisting processes (most of the time undocumented) thatneed to be changed to allow for the effective integrationof the new system. Other concerns were the implicationsof an incorrect specification and the consequences of thefact that the client is often responsible for content pro-vision. There was a consensus from the majority (76%)of the respondents that there needed to be a process atthe beginning of the projects focused on educating theirclients.

108

3.1.4 Identifying requirements

Almost all respondents interviewed clients as part of theprocess for identifying requirements. A much smallernumber felt that interviewing potential users was im-portant. The majority of respondents (84%) found thatthe look and feel and content issues were secondary tothe business case. Of the remaining 16% of respondents,11% are primarily involved in the front end of multi-media development and either contract or are contractedto complement backend work. Perhaps surprisingly,critical success factors (i.e. acceptance criteria or essen-tial requirements) were brought up only by 5% of re-spondents as a vehicle for capturing the business case.

The majority of respondents (76%) indicated theimportance of getting the requirements and specificationcorrect, though there was considerable divergence ofopinion as to when this should occur. Some other im-portant points that contribute to the success of a Webproject are senior executives and management commit-ment to the project, early user analysis and usabilityevaluation, and recognising the concomitant changes tothe organisation’s workflows and business practices.

The majority of respondents (84%) felt that there wasa major shift in the level of awareness from the begin-ning of a project to the end. The extent to which this wasconsidered a problem was largely correlated to the scopeof the project. For small projects, the project durationand budget left little room for changes in client expec-tations and hence requirements during the project. It wasviewed by a number of respondents that it was thereforecritical that clients for small projects be educated asearly as possible in the development cycle.

There was reasonable consistency in views about theelements that should be captured in specifications. Keyelements that reoccurred consistently included: currentsystems; vision for the project; business strategies;business drivers; business case; budget; stakeholders.

3.1.5 Development processes

Most of the respondents attempt to capture requirementsbefore they sign final contracts. It was also recognised thatinitial tendering often occurs before this point, leading toa two-step contract negotiation. Some clients were seen tobe happy to commit to a budget (during the tender pro-cess) based on broad business objectives, and then finalisethe contract at a later stage based on specific analysis ofthe detailed requirements. The scope of the contractedrequirements is typically constrained by retaining a focuson the business case and establishing that there is goodbasis for specific detailed requirements.

The majority of respondents (84%) had a standardpro forma for documenting requirements and contracts,and the majority (76%) believed that they adhered to itconsistently. The majority of respondents (68%) carriedout formal evaluation, though this was primarily on thefinal product and typically against the specification.

Several respondents indicated that due to the evolutionof the specifications (which were typically not main-tained in a coherent documented form) direct testingagainst the specification was not feasible.

All the respondents felt that they carried out somelevel of maintenance of the project files though theelements that were considered important to maintainvaried considerably (for example, often the design pro-totypes were considered to be a better level at which tomaintain the documentation of requirements than theoriginal specifications).

All the respondents had an initial sign-off on either aproject brief or proposal. Only one respondent pointedout that an initial sign-off was difficult when they wereworking off a concept. There seemed to be two distinctmethods of specifying. The first was the traditionalsoftware specification method; essentially:

a. requirements analysis;b. functional specification;c. technical specifications;d. testing specification.

Or alternatively:

a. initial brief;b. design documents;c. content information;d. technical specifications (including testing).

It was recognised by essentially all respondents thatclient needs will change and evolve over the life of aproject, despite a well-written requirements specificationdocument. Most respondents had some form of changemanagement process, though few companies had formaltool-level support for change or configuration manage-ment of applications. Most respondents document allchanges whether minor or major in formal proceedingsmainly to justify their reasons for working out of scope.

Of the respondents that did not indicate having inplace formal procedures there seemed to be a feeling thatbusiness cases change during the development phase andthat to support new media development these changesneed to be allowed, rather than a fixed price paradigm beset and adhered to.

3.1.6 Primary divergences of views

There were a number of areas where there was diver-gence of views. In particular, there were significant dif-ferences in the levels of agreement with the followingstatements (from the survey):

– General relationship with client– We interview the clients to determine requirements– We interview intended users to determine require-

ments– It is important to respond to changes in user re-

quirements as they occur– Changes in the user requirements require the site to be

renegotiated

109

– Clients’ needs evolve considerably during the lifetimeof a project

– Development processes– It is important to identify technologies to use as soon

as possible– It is important to be able to modify the system once it

is completed

3.2 Analysis of specification documentation

Based on the primary data collected and analysed duringthe interviews and surveys, we were able to gain a broadpicture of the approaches taken to resolving client needsin Web development projects. In order to focus this ontomore specific detail, particularly with regard to thoseelements that tend to be identified as important tospecifying a system, we followed up the interviews andsurveys by analysing a significant number of commercialWeb specifications.

These specifications covered a broad range: frominitial expressions of interest to responses for requestsfor tender and to full contractual specifications. Theyalso varied from specifications for completely new sys-tems to specifications for redevelopment or modifica-tions to existing systems.

It was interesting to note that there were both sig-nificant areas of overlap in the aspects considered inthese documents, as well as significant areas of diver-gence. The primary areas of divergence were as follows:

3.2.1 Consideration of the underpinning business model

The specifications varied widely in the extent to whichthey actively identified and documented the businessmodel that drove the development of the system. Initialindications are that those that had a clear business modelwere usually much more cohesive, though this requiresfurther investigation for confirmation and quantification.

More interestingly, the results indicate that commer-cial practice has tended to emphasise business modellingas a driver for structuring requirements, coupled with theutilisation of designs in assisting clients to develop anunderstanding of the problem and potential solutions,and especially the way in which proposed solutions maylead to consequent changes in business practices ormodels (and hence requirements). We would argue thatthis stronger emphasis on business modelling is, in effect,a tacit acknowledgement by developers of the impactsthat the solution has on the nature of the problem, andthe consequent need therefore to understand at least theshape of the solution space when clarifying the problem.

3.2.2 Extent of emphasis on site look and feel

The stage at which look and feel issues (both naviga-tional and graphic design) are introduced varied

considerably. Some specifications focused on this veryearly in the process, to the detriment of other aspects.

3.2.3 Consideration of changes to organisationalwork flows

There was significant variation in the way in whichspecifications addressed potential impacts outside theimmediate technical issues related to the site. These weremost noticeable with regard to impacts on organisa-tional processes.

More interesting than the areas of divergence were theareas where the specifications had a degree of consistency,but where this was at odds with practice in other domains,especially conventional software development. The mostnoticeable of these areas is the extent to which strict userand client needs aremixedwithdesign information.This isparticularly noticeable once an initial expression of clientneeds evolves into a full specification. These full specifi-cations almost uniformly mixed design information withclient requirements. Indeed the specifications often useddesigns as awayof representing requirements, bypassing aformal requirements specification.

3.3 Organisation-specific methodologiesand best practice guidelines

Finally, we considered process documentation fromvarious companies. This includes aspects such as de-scriptions of methodologies, best practice guidelines,development standards, etc.

The dominant trend was towards the utilisation ofdocumentation, not to support development processes,but as a marketing tool to illustrate claimed capabilities.In numerous cases this documentation was relativelysuperficial, and in those cases that were investigated inmore detail often did not map to more detailed processdocumentation. This is not to say that these organisa-tions did not adhere to the processes described; ratherthat it was often not documented in any substantialdetail. This is an area that requires further investigation.

4 Analysis

Bringing all the above information together, we candraw some interesting conclusions. The first, and mostobvious is that there is a significant gap between currentresearch and commercial practice. It is unclear whetherthis is a consequence of a lack of awareness of researchoutcomes by practitioners, or research outcomes that arenot yet suited to practical development. Indications are,however, that the core focus of much commercial workdoes not map well to research emphases.

For example, research has tended to emphasise de-tailed design issues both in terms of content and func-tionality, albeit using discrete approaches that are notwell linked and has barely considered approaches to

110

understanding clients’ needs and how they are formu-lated and represented. Conversely, commercial practiceplaces significantly more emphasis on understandingclient needs, and in particular how business models andprocesses are impacted and how they are linked to sys-tem architectures (both informational and functional).

The second major observation is with regard to therole of the design process. Conventional requirementsengineering and management processes see the require-ments as preceding and driving the design process. Evenwhere an iterative or incremental approach (such as XP),or a spiral or prototyping approach (involving multiplefeedback loops) is adopted the design is viewed as a wayof assisting in the elicitation, clarification and validationof requirements. In some cases the client understandsvery well their needs and these approaches are aimed atassisting the developer to identify and formalise theseneeds. In other cases the client has trouble formulatingor understanding their needs and the design process isaimed at facilitating an understanding of the needs.

With Web development, the design process will sup-port the above role, but can also play a very differentrole. The design process not only helps developers andclients articulate the needs, but also helps clients un-derstand the system domain and how it changes with theintroduction of a new system. The result is that thedesign process can assist clients to formulate needs thatare interdependent on a given design [35]. In effect, thedesign allows developers to manage the domain uncer-tainty and volatility that are a consequence of theintroduction of different designs.

A useful comparison at this point is with the ap-proach that is often referred to as Ready–Fire–Aim [47].This essentially is referring to approaches where thedesign is commenced prior to a full understanding of therequirements (or coding commenced prior to a fulldesign, depending on the interpretation) as a way ofinforming clients in the presence of uncertainty. Incontrast, commercial Web development is typicallyabout developing prototype solutions as a way not ofresolving uncertainty, but rather to understand theimpact of a given solution. This is a little bit like saying‘Well, if we fire there, then it will have this impact, but ifwe fire there it will have that impact’; i.e. possible so-lutions are jointly investigated by the developer andclient (typically, through a design prototyping approachbut prior to committing to a specific solution) in termsof their impact on the problem domain and hence therequirements, with the ultimate result that a solution isidentified that matches a problem that has been changedby that solution.

Although this concept is well accepted within indus-try, very little consideration has been given to it withinthe research literature. Research has been limited toempirical work using scenario-based redesign ofpartially developed sites [48], though this work has atleast recognised the importance of designs in assistingclarification of client needs:

We practice a revised method of scenario-baseddesign inferred from a theoretical perspectivewhich treats design as inquiry, inquiry as dia-logue and dialogue as the source of all tools, in-cluding mental constructs. The result is a set oftechniques for using structured dialogue betweenusers and designers to increase designers’ under-standing of specific domains of users’ work.

In commercial Web projects, these concepts (andparticularly the mutual interdependence of requirementsand design) are typically reflected in the absence ofseparate requirements and design documents. Rather,developers tend to create a hybrid specification thatblends design and requirements (something that isusually viewed as anathema in conventional softwareengineering).

In other words, system design allows stakeholders tounderstand technical possibilities and limitations, andhence improve their understanding of the developmentcontext. The result is a vehicle for reducing the under-lying uncertainty. For this to be effective, however, weneed to develop a suitable model of the relationshipbetween system design, client requirements and uncer-tainty within these requirements. This uncertaintymodelcan then be used to adapt the requirements engineer-ing process, resulting in a design-driven requirementsprocess. This is the focus of our ongoing research.

Finally, there are a number of minor issues that arehighlighted by either commercial practice or currentresearch:

– It is important to establish a client liaison with aperson who understands both the IT/technical as-pects and the business domain. Although such aperson may not be in a position within the client’sorganisation that allows them to readily interactwith developers or contractors, there is much to begained from encouraging the client to bring such aperson into the liaison, particularly in terms ofreducing the uncertainty discussed above.

– Establish effective client education mechanisms. Amajor problem in many projects appears to be thelack of understanding by the client. This extendsfrom an understanding of the capabilities of thetechnologies to an understanding of their ownbusiness processes and how these should possibly bemodified with the introduction of the new Websystem(s). It is important that clients accept the needto develop a clearer understanding, and be willing towork with the developers in building up thisknowledge.

– Clarify business cases as early in the process as pos-sible. The business case forms the foundations onwhich the requirements, specifications and subse-quent development are grounded. Conventionaldevelopment is often based on an implicit under-standing of possible business cases – something thatis not true in Web development projects. Without a

111

clear understanding of the clients’ business case aproject can waste considerable effort going around incircles proposing designs that subsequently requireconsiderable change.

– Adopt early evaluation: A primary characteristic ofmany Web projects is the lack of client under-standing with regard to desired attributes of thesystem being developed. A key way to handle suchuncertainty is to adopt early evaluation of potentialdesigns. This can be through usability evaluations,focus groups, etc.

5 Conclusions and further work

In this paper we have analysed the current situation withregard to the management of requirements for Webprojects. In particular, we have contrasted the researchwork documented in the literature with existing com-mercial practices. There is a significant discrepancy be-tween the foci in these areas. Research has tended tofocus on design methods, but largely without consider-ing how these design processes contribute to a domainunderstanding and clarification of the requirements.

A key paragraph from the analysis is worth repeat-ing, given the crucial significance that it has. Commer-cial practice has tended to emphasise business modellingas a driver for structuring requirements, coupled withthe utilisation of designs in assisting clients to develop anunderstanding of the problem and potential solutions,and especially the way in which proposed solutions maylead to consequent changes in business practices ormodels (and hence requirements).

This has many superficial similarities to prototyping,insofar as both approaches use partial solutions to im-prove understanding. Indeed, Web development makesheavy use of prototypes. However, whereas conventionalprototyping is largely focused on assisting developers tounderstand the clients’ requirements, Web developmentuses prototypes to inform clients of the implications ofpotential solutions on their business processes and thesubsequent possible changes to their requirements. Wecan view the problem domain and solution domain asbeingmutually constituted or interdependent, and neithercan be resolved in the absence of the other.

Considerable work still remains to be carried out inthis area. There are two key areas that we are investi-gating. The first is the development of a design-drivenrequirements process that structures the way in whichdesign activities can be linked to the clarification of re-quirements through an appropriate model of domainuncertainty.

The second is an improved linkage between the ar-chitectural elements of a Web system. In particular,given the evolving role of designs, it is important that thebusiness architecture can be linked to both the infor-mation architecture and the technical architecture of asystem. Indeed a fruitful area for further investigation is

to look at how the informational and functional aspectsof the architecture can be coupled appropriately duringthe design. This is particularly important in terms ofunderstanding how changes to the technical architectureaffect the nature of the business architecture. We are justbeginning work that address this by integrating conceptsfrom pattern theory, Jackson’s problem frames andAlexander’s work on architecture.

One element driving both these research areas is ourongoing refinement of a characterisation model thatcaptures the various elements that emerge in the speci-fication and design of Web sites. This model provides abasis for understanding which elements should be clar-ified at which point in the process, and the linkagesbetween these elements.

Acknowledgements The authors wish to acknowledge the collabo-rative funding support from the Australian Research Council,Access Online Pty Ltd and Allette Systems Ltd under grant no.C4991-7612. In particular we wish to thank John Eklund, RossJeffery, Louise Scott, Lucila Carvalho and John D’Ambra for theircontributions to this research project, and Brian Henderson-Sellersand Brendan Haire for the related discussions on WebOPEN andthe differences between Web development and conventional soft-ware development. We also wish to acknowledge the valuablecontributions of the numerous companies and individuals whoparticipated in the industry interviews and surveys. Finally, I wouldlike to acknowledge the valuable feedback, advice and suggestionsof the reviewers of this paper.

References

1. Stein LD (2000) Profit, the prime directive. WebTechniques5:14–17

2. Russell P (2000) Infrastructure – make or break your e-busi-ness. In: TOOLS-Pacific 2000: Technology of object-orientedlanguages and systems, Sydney, 2000

3. Stamoulis D, Kanellis P, Martakos D (2001) Tailorable infor-mation systems: resolving the deadlock of changing userrequirements. J Appl Syst Stud 2

4. England E, Finney A (1999) Managing multimedia: projectmanagement for interactive media, 2nd edn. Addison-Wesley,Reading, MA

5. Checkland P, Scholes J (1990) Soft systems methodology inaction. Wiley, Toronto

6. Gates L (2001) Analysis and design: critical yet complicated.Application Dev Trends February:40–42

7. Burdman J (1999) Collaborative Web Development. Addison-Wesley, Reading, MA

8. Sinha G (1999) Build a component architecture for e-com-merce. e-Business Advisor March. http://www.advisor.com/doc/05328

9. Overmyer S (2000) What’s different about requirements engi-neering for web sites? Requirements Eng 5:62–65

10. Isakowitz T, Stohr E, Balasubramanian P (1995) RMM: amethodology for structured hypermedia design. CommunACM 38:34–44

11. Schwabe D, Rossi G (1995) The object-oriented hypermediadesign model. Communications of the ACM 38:45–46

12. Lange D. An object-oriented design method for hypermediainformation systems. In: HICSS-27: Proceedings of the 27thHawaii international conference on system sciences, Maui,Hawaii, 1994

13. Lee SC (1997) A structured navigation design method for in-tranets. In: Third Americas conference on information systems,Association for Information Systems (AIS), Indianapolis, 1997

112

14. De Troyer O, Leune C (19970 WSDM: a user-centered designmethod for Web sites. In: 7th international World Wide Webconference, Brisbane, 1997

15. Ceri S, Fraternali P, Bongio A (2000) Web Modeling Language(WebML): a modeling language for designing web sites. In:Proceedings of WWW9 conference, Amsterdam, 2000

16. Takahashi K, Liang E (1997) Analysis and design of web-basedinformation systems. In: 7th international World Wide Webconference, Brisbane, 1997

17. Guell N, Schwabe D, Vilain P (2000) Modeling interactionsand navigation in web applications. In World Wild Web andconceptual modeling’00 workshop – ER’00 Conference, SaltLake City, Utah, 2000

18. Gellersen HW, Wicke R, Gaedke M (1997) WebComposition:an object-oriented support system for the web engineering lifecycle. Comput Netw ISDN Syst 29:8–13

19. German DM (2000) Hadez, a framework for the specificationand verification of hypermedia applications. University ofWaterloo

20. German DM, Cowan DD (1999) Formalizing the specificationof web applications. Lecture notes in computer science, vol1727. Springer, Berlin Heidelberg New York, pp 281–292

21. Courtiat JP, Diaz M, Oliveira RCD, Senac P (1996) Formalmethods for the description of timed behaviors of multimediaand hyper-media distributed systems. Comput Commun19:1134–1150

22. Paulo FB, Augusto M, Turine S, Oliveira MCFd, Masiero PC(1998) XHMBS: a formal model to support hypermedia spec-ification. In 9th ACM conference on hypertext, Pittsburgh,Pennsylvania, 1998

23. Baumeister H, Koch N, Mandel L (1999) Towards a UMLextension for hypermedia design. In: <<UML>> 1999: Thesecond international conference on the Unified ModelingLanguage, Fort Collins, Colorado, 1999

24. Conallen J (1999) Building web applications with UML.Addison-Wesley, Reading, MA

25. Vilain P, Schwabe D, Souza CSd (2000) A diagrammatic toolfor representing user interaction in UML. In: <<UML>>2000: the third international conference on the Unified Mod-eling Language, York, 2000

26. Lord J (2000) Patterns for e-business: lessons learned frombuilding successful e-business applications. IBM, White Plains,New York, p 4

27. Fournier R (1999) Methodology for client/server and web ap-plication development. Yourdon Press, Englewood Cliffs, NJ

28. Sommerville I, Kotonya G (1998) Requirements engineering:processes and techniques. Wiley, New York

29. Angelique E (1999) A lightweight development process forimplementing business functions on the web. In: WebNet’99,Honolulu, Hawaii, 1999

30. Haggard M (1998) survival guide to web site development.Microsoft Press

31. Beck K (1999) Extreme programming explained. Addison-Wesley, Reading, MA

32. Thomas D (2000) Managing software development in web timesoftware. In: XP2000, Cagliari, Italy, 2000

33. Martin R (2000) A case study of XP practices at work. In:XP2000, Cagliari, Italy, 2000

34. Siddiqi J (2000) eXtreme programming: pros and cons. IEEEComputer Society, Los Alamitos, CA

35. Lowe D (2000) A framework for defining acceptance criteriafor web development projects. In: 2nd ICSE workshop on webengineering, Limerick, Ireland, 2000

36. Zowghi D, Offen R, Nurmuliani R (2000) The impact ofrequirements volatility on the software development lifecycle.In: International conference on software, theory and practice(ICS2000), China, 2000

37. Checkland P (1981) Systems thinking, systems practice. Wiley,New York

38. Graham I, Henderson-Sellers B, Younessi H (1997) The OPENprocess specification. Addison-Wesley, Harlow, UK

39. Henderson-Sellers B, Simons A, Younessi H (1998) The OPENtoolbox of techniques. Addison-Wesley, Harlow, UK

40. Haire B, Henderson-Sellers B, Lowe D (2001) Supporting webdevelopment in the OPEN process: additional tasks. In:COMPSAC’2001: International computer software and appli-cations conference, Chicago, Illinois, 2001

41. Haire B, Lowe D, Henderson-Sellers B (2002) Supporting webdevelopment in the OPEN process: additional roles and tech-niques. In: OOIS, Montpellier, France, 2002 (accepted)

42. Lowe D (2000) Web engineering or web gardening? WebNet J2:9–10

43. Lowe D, Henderson-Sellers B (2001) Impacts on the develop-ment process of differences between web systems and conven-tional software systems. In: SSGRR 2001: Internationalconference on advances in infrastructure for electronic busi-ness, science, and education on the internet, L’Aquila, Italy,2001

44. Lowe D, Henderson-Sellers B (2001) Web development: ad-dressing process differences. Cutter IT J 14(7):11–17

45. Lowe D, Eklund J (2001) Commercial issues in specification ofWeb systems. In: AWRE 2001: 6th Australian workshop onrequirements engineering, Sydney, 2001

46. Lowe D, Eklund J (2002) Client needs and the design process inweb projects. In: WWW 2002: 11th international World WideWeb conference, Hawaii, 2002

47. Holtzman JK (1993) Ready. Fire!! Aim???. In: Proceedings ofthe 11th annual international conference on systems docu-mentation, Waterloo, Canada, 1993

48. Erskine L, Carter-Tod D, Burton J (1997) Dialogical tech-niques for the design of web sites. Int J Hum–Comput Stud47:169–195

113