a living-systems design model for web-based knowledge management

A Living-Systems Design Model for Web-basedKnowledge Management Systems

Jan L. PlassMark W. Salisbury

Most of the currently available instructionaldesign models were conceptualized to developinstructional solutions to needs andrequirements that remain relatively stable overtime. Faced with the problem of designing aknowledge management (KM) system thatneeded to accommodate continuouslychanging requirements over its fielded lifetime,we developed a new design model that is basedon a living-systems approach. In this article,we briefly review currently availableinstructional systems design models anddescribe this new model and the mechanisms itcontains for accommodating change andgrowth. We illustrate the application of thephases of the model (analyze initialrequirements, design the informationarchitecture, develop the information design,develop the interaction design, implement theWeb-based system, and conduct adevelopmental evaluation of the system) in thedevelopment of a KM system withliving-system features.

The systematic design of instruction has longbeen recognized as a key to the successfuldevelopment of effective and appropriatematerials (Dick & Carey, 1990; Gagné & Briggs,1979; Rothwell & Kazanas, 1997). Earlydevelopers of computer-based instruction (CBI)recognized the difficulty in successfully taking aCBI project from conception to delivery. To helpmanage the complexity of the developmentprocess, they adopted methods and techniquescreated in the computer industry for managingsoftware development projects. Many of theseearly software development methods wereadaptations of the “waterfall approach” (Your-don & Constantine, 1978), a highly structuredmethod for project management that delineatesthe design process into clearly defined phases—analysis, design, implementation, testing, anddelivery. The metaphorical implications of thewaterfall approach are obvious: Each phase hasa clearly defined output, and this outputprovides the input to the next phase. The pur-pose of the analysis phase is to produce thespecification of the new application. It stateswhat the new application should do, but doesnot describe how it will be accomplished. Thedesign phase results in the specification beingtranslated into a paper-based design of the sys-tem. In the implementation phase, this design isfinally realized as a computer program—thentested in the testing phase—before it is deliveredto the customer in the delivery phase. The water-fall approach did improve project managementfor CBI developers, but it was a method adoptedfrom software engineering that did not suffi-ciently focus on the instructional aspects of com-puter applications.

As the CBI field matured, development

ETR&D, Vol. 50, No. 1, 2002, pp. 35–57 ISSN 1042–1629 35

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 03-04-2002 / 17:27

methods appearing in the literature began to in-corporate concepts and terminology taken fromthe instructional systems design (ISD) approach(Gagné & Briggs, 1979). Since ISD was concep-tualized and accepted for instructional projectsbefore the popularity of CBI, it was a naturalprogression for developers to adopt an ISD ap-proach for CBI projects to incorporate the in-structional focus of ISD. Most popularadaptations used ISD front-end methods to es-tablish the purpose of the application and tech-niques adopted from software developmentmethods to implement the application (Alessi &Trollip, 1991; Fenrich, 1997; Roblyer, 1988;Wager & Gagné, 1988). These ISD-based ap-proaches quickly became popular because CBIapplications were resource intensive and CBIproject managers were looking for more effec-tive alternatives to the waterfall approach. Thetypical steps of such an instructional design (ID)approach for the development of CBI projectsare (Alessi & Trollip, 1991):

• Determine needs and goals, including learnercharacteristics and entry knowledge.

• Collect resources, regarding content, designand delivery of instruction.

• Learn the content, that is, become familiar withthe subject matter.

• Generate ideas, that is, brainstorm andgenerate creative ideas.

• Design instruction. Perform concept and taskanalyses, reassess goals, evaluate design.

• Flowchart the lesson. Describe operations thecomputer will perform, interactions, lessonstructure.

• Storyboard displays. Prepare textual and pic-torial displays, content and form of presenta-tion.

• Program the lesson. Select programming orauthoring language, program instruction.

• Produce supporting materials. Develop studentand instructor manuals, a technical manual,and adjunct instruction.

• Evaluate and revise. Assess how well lessonswork and how much students learn.

Using approaches based on the ISD processwas an improvement over the basic waterfall ap-proach because of the added focus on instruc-

tional aspects in the design process, but it wasstill perceived by many as a rather linearmethod of development, with each phase—more or less—completed before the next onewas begun (Boyle, 1997).

Meanwhile in the software-engineeringworld, another revolution in developmentmethods was taking place (de Hoog, 1994).Flexible methods of project management basedon iterative prototyping—sometimes calledevolutionary system development—were be-coming increasingly popular for commercialsoftware projects (e.g., Jones & Richey, 2000;Moody, Hudson, & Salisbury, 1988). Collective-ly, they have become known as rapid applica-tion development (RAD) methods (Boar, 1984).Illustrating the popularity of this approach, in1995, a standard nonproprietary RAD methodwas introduced by a consortium of nearly 100companies. This approach was called thedynamic systems development method(DSDM). The manual that resulted from this ef-fort, Dynamic Systems Development Method(1995), defines DSDM as a method that providesa framework for building and maintaining sys-tems that meet tight time constraints throughthe use of incremental prototyping in a control-led project environment.

Because of the success of iterative prototyp-ing methods such as RAD and DSDM fordeveloping general software applications, com-bining ISD methods with iterative prototypingbecame the next breakthrough in developmentmethods for CBI (Koper, 1995). Figure 1 showsthe iterative prototyping process. Combiningthe iterative approach with an ISD approachresults in an instructional focus of the iterativeapproach to the design of materials. This ap-proach differs from previous methods by ex-plicitly supporting an interleaved form of projectdevelopment on different levels. In the earlystages of prototyping the application, for ex-ample, interleaving is used in order to refine theobjectives of the system (Boyle, 1997). Duringthis process, the prototype becomes the object ofcommunication between the designer and theusers. Requirements are clarified and miscon-ceptions between the designer and the clientscome to light at this stage. For example, Salis-bury (1988) used iterative prototyping to define


36 ETR&D, Vol. 50, No. 1

human-computer interface requirements of anatural language interface. Using a prototypingtool, developers implemented a natural lan-guage interface that was able to process a subsetof the end-user actions for an application. Afteran evaluation of the acceptance of the naturallanguage interface by end users, developersadded more grammar rules and vocabularywords to the interface to enable it to processmore end-user actions and to support requestsfor the same actions with differing sentencestructures and vocabulary. This process con-tinued until end users were able to perform mostof the required actions for the applicationthrough the natural language interface.

DESIGNING SYSTEMS WITH CHANGINGNEEDS AND REQUIREMENTS

Combining ISD methods with iterativeprototyping has addressed many of the limita-tions of using a more traditional approach fordeveloping CBI applications. However, wefound that this hybrid approach, and those thatcame before it, were not well suited for the chal-

lenges that we were facing in a project where wedeveloped an integrated and evolvingknowledge management (KM) system. Whilethese methods were able to address some chan-ges in needs and requirements during thedevelopment of the project, they were notdesigned for building systems that have con-tinuously changing requirements over theirfielded lifetime caused by changing needs of theorganization that is using the system. In otherwords, the approaches discussed so far wereable to accommodate certain changes in the sys-tem requirements that occur during the projectdevelopment cycle. A design method for thedevelopment of a KM system, however, needs toaddress two additional levels of change.

The first level of change concerns the or-ganization that will use the system. Here, duringthe fielded lifetime of the KM system, new re-quirements and needs may arise that result fromchanges in the organization. These changes mayinclude new characteristics of the targetaudience for the system due to retirements andnew hires. New products and markets result inchanging amount, content, type and structure ofthe information included in the system, and, as a

Figure 1 Iterative-prototyping approach to software development


LIVING-SYSTEMS DESIGN MODEL 37

consequence, in changing forms and methods oflearning and instruction that the system has tobe able to facilitate.

The second level of change is a consequenceof the first one and concerns the KM system it-self. Most instructional materials are notdesigned to be modified by the users throughtheir interaction with the system. In the case of aKM system, however, this functionality con-stitutes one of the fundamental system require-ments: Acting as learners, users retrieveinformation from the system to solve a par-ticular problem. Once they have solved thisproblem, however, they act as authors and addtheir newly constructed solution to the system.The resulting growth of the KM system willdepend on the needs of the organization andwill change over time. Therefore, mechanismshave to be designed and implemented that canaccommodate these changes.

These two types of change closely resemblecharacteristics we can observe in living systems(Maturana & Varela, 1980). Using this analogy, aKM system needs to be able to adapt to its en-vironment, that is, the changing organizationalneeds. It also needs to be able to grow and learn,that is, to respond to the individual needs oflearners. Traditional ID models, however, werenot conceptualized to include features that canaccommodate this change and growth. In the IDmethods described so far, once a need was estab-lished and the parameters and requirements forthe instructional solution were defined and im-plemented, no procedures were in place thatwould allow for their revision and the sub-sequent modification of the system after it hadbeen deployed. We found that other more recentdevelopment methods and models do not ex-plicitly address this problem of building systemswith evolving requirements either. Object-oriented design approaches (e.g., Coad & Your-don, 1991), while solving many previouslyunresolved issues such as recycling and reusingof project components, do not offer a built-insolution for accommodating evolving require-ments during the lifetime of the system. Similar-ly, methods such as user-design approaches(e.g., Banathy, 1991; Carr, 1997; Reigeluth, 1993;Schuler & Namioka, 1993), where users are em-powered to make critical design and develop-

ment decisions, include an initial phase to defineparameters and requirements but do not ex-plicitly support building dynamically changingsystems. Even the more recently proposed con-structivist models of ID (Schwier, 1999; Ten-nyson, 1997; Willis, 1995), while allowing morelearner input in the design of the materials andin the definition of the learning goal and objec-tives than previous models, still aim for thedevelopment of a relatively static final productand do not adequately address the need for amechanism or procedure that allows for chang-ing requirements and specifications after thedevelopment of the system has been completed.In fact, Dick (1996) observed, “when construc-tivist models are proceduralized, they look verymuch like traditional design models” (p. 62).Therefore, given the limitations of thesemethods for building systems with continuouslychanging requirements, and the specific situa-tion of our development project that called forsuch changes, we were faced with the need tocreate a new design model. In this article, wedescribe this model and the living-systems ap-proach it is based on, and show how it was ap-plied to build an integrated and evolving KMsystem for a large government organization andits affiliates. First, however, we will providesome information on the project that inspiredthe development of this new design model.

CASE EXAMPLE—THE PRODUCTREALIZATION PROCESS

Over the last several years, the large govern-ment organization for which the KM system wasdeveloped has streamlined its operations tomake production more efficient in a variety ofcoordinated engineering, manufacturing, as-sembly, and management activities. In so doing,eight separate laboratories and plants aroundthe United States have agreed to utilize theProduct Realization Process with a common setof technical business practices (TBPs) thatprescribe and guide operations. The goal of theproject was to get a potential user community of1 to 2 thousand individuals at these eight sites tounderstand and apply the TBPs, related docu-ments, and terminology to their projects. The


38 ETR&D, Vol. 50, No. 1

user community at these sites is an agingpopulation, not unlike the rest of the organiza-tion. While these experienced users are highlyknowledgeable about how business has been orshould be conducted, others are being asked forthe first time to subscribe to the common set ofbusiness practices. Employees who are ex-perienced with the TBPs, approaching theirretirement, have a large amount of tacitknowledge about the TBPs that would be lost totheir organization if it were not captured. In ad-dition, newcomers to these eight facilities needto be provided with an orientation while, at thesame time, getting a more complete picture ofprocesses, procedures, and practices.

Initial efforts to address this problem with in-troductory, classroom-based training, createdwith input from subject-matter experts and dis-tributed to representatives at the eight facilitiesfor them to customize and deliver to their ownstaff had not been sufficient in promoting theuse of TBPs. Therefore, a new way of providingaccess to the TBPs and related documents, tosearch those documents, to provide decisionsupport that would help users through the mazeof TBPs, and to deliver short, focused, just-in-time instruction on a variety of related topicswas approached—the development of a KM sys-tem.

A LIVING-SYSTEMS APPROACH FORINSTRUCTIONAL DESIGN MODELS

The design and development of a system withchanging requirements calls for a design modelthat can reveal the complex and changing re-quirements of the system and provide a meansto control further development and main-tenance costs. One possible approach to thedevelopment of such a design model is to viewthe resulting system as a living and adapting or-ganism. That is, since growing and sharingknowledge is, by definition, an ongoing andself-modifying process, the goal is to design andbuild a system that is adaptable to its environ-ment—a living system. This view of a system asa living entity falls under research that has beenlabeled autopoiesis theory. This concept wasdeveloped more than 30 years ago in biology

through the work of Maturana and Varela(1980), enabling them to make a distinction be-tween living and nonliving systems. Autopoies isGreek for self production, and an autopoieticsystem is one that has within its own boundariesthe mechanisms and processes that enable it toproduce and reproduce itself. Applying thisconcept to human social systems such as thosefacilitated by a KM system, Luhmann (1986)described such a system as one that is both openand closed to the environment. As such, it isclosed to information and knowledge, but opento data. The system constructs its ownknowledge through the process of accommodat-ing data from the environment, shaping andchanging the very structure and nature of thesystem in the process (von Krogh, Roos, &Slocum, 1996).

In the case of the KM example, the require-ments for the system are changing for a numberof reasons. While the target audience currentlyconsists primarily of experienced engineers, awave of upcoming retirements, which will befollowed by new hiring, leads us to expect achange in that population to include an increas-ing number of new, less experienced andyounger employees. At the same time, the infor-mation made available through the system is ina process of continuous updates and revisions.TBPs and their documentation are frequentlymodified according to new technical develop-ments and the changing needs of the organiza-tion.

Aside from these externally generated causesfor change, the design of the system’s owncapabilities and features will result in its changeand growth. The developers of early KM sys-tems and expert systems realized that it was dif-ficult to get enough subject-matter experts tocontribute their knowledge to the system, whichwas referred to as knowledge acquisition bottleneck.A second, related challenge, often referred to asthe knowledge cliff problem, concerns the breadthof knowledge within a system. While theknowledge contributed to the system by expertsmay cover a high level of depth, it has often onlya very narrow breadth, covering only a narrowarea of expertise. Users who attempt to accessknowledge outside of this narrow area fall offthe knowledge cliff (Salisbury & Plass, 2001). A



KM system, therefore, needs to includecapabilities that allow users to peruse a varietyof methods and features of learning and instruc-tion, including means of contributing their ownmaterials to the site: Once the user constructs asolution, it can be added to the system. Thesefeatures collectively provide a rich learning en-vironment for users and allow them to avoid theknowledge-cliff problem. These capabilities alsocontribute to the changing and evolving content,which in turn will result in changing and evolv-ing ways of learning.

As a practical mechanism for the design ofKM systems that can incorporate such changeon different levels, we developed a designmodel with the following phases (see Figure 2):

• Analyze End-User Requirements. Define the in-itial end-user requirements, instructionalneed, target audience, goal, and objectives forthe system.

• Design Instructional Information Architecture.Specify the content and functionality of thesystem as well as its structure, including aliving system functionality to accommodatesystem change and growth; define thenavigation system.

• Develop Instructional Interaction Design.Design the human-computer interaction foreach system feature, including instructionalstrategies, methods of learning and instruc-tion, and navigation.

• Develop Instructional Information Design.Define the appearance of interface elements;specify the forms of representation of infor-mation in the different presentation modesand for different sensory modalities with afocus on the cognitive science foundations oflearning from text, sound, animation, maps,graphs, and other multimedia information;develop methods of “wayfinding” fornavigation.

• Implement System Design. Develop aprototype KM system based on the design ofthe capabilities specified in the informationdesign, interaction design, and informationarchitecture.

• Conduct Developmental Evaluation. Evaluatethe product of each of the phases to deter-mine if it attained the intended objectives for

the current needs and requirements of the or-ganization; evaluate usability and effective-ness of the features for the describedobjectives.

Below, we describe the purpose of eachphase, the input required, steps performed, andoutput generated, and provide a theoreticalbackground that informs the execution of thesteps in the phase. We also discuss how each ofthese phases was realized in the KM case ex-ample by describing the most relevant aspects ofthe output of each phase. Because the focus ofthis article is on the description of the designmodel and not on the detailed description of theKM system we implemented, however, we willnot provide an exhaustive description of the out-come of each phase here. More informationabout the KM system itself can be found in Salis-bury and Plass (in press).

Analyze End-User Requirements

The purpose of this phase is the analysis of theinitial end-users’ needs and requirements andthe identification of the target audience, as wellas the goals and objectives for the system. Theanalysis follows mostly the procedures specifiedin other, more traditional ISD models describedearlier. The input for this phase can be takenfrom interviews with stakeholders and per-formers in the organization the system isdeveloped for, observation of the daily workperformance of potential users, surveys, ordocument reviews. In many cases, a needs as-sessment will have to be conducted to identifywhat the exact requirements are and what ap-proaches to a solution might be feasible. Out-comes of this phase include (a) the decision if aKM system can indeed meet the organization’sneeds and, if this is the case, (b) the initial end-user requirements of the goals and objectivesregarding the function and performance of thesystem, (c) the target audience that will use thesystem and its learner characteristics, and (d) adescription of the minimum configuration of thedelivery platform for the system, that is, theresources and constraints of the available com-puter hard- and software, as well as networkconnectivity for the delivery of the materials.


40 ETR&D, Vol. 50, No. 1

Note that the goals and objectives do not neces-sarily determine specific learning outcomes, asthey usually do in traditional ISD models. Theymay instead describe the system features andfunctionality that will allow learners to constructknowledge according to their needs.

Another issue in which this phase differsfrom other models is in the use of the outcome ofthe analysis. In many models, the results of theneeds analysis are used as input for severalother phases of the design process, but are notreassessed or updated after they are initially ob-tained. In our approach of treating the systemwe develop as constantly changing, we followthe idea of a needs reassessment outlined byTessmer, McCann and Ludvigsen (1999) byrecognizing that key outcomes of this analysishave to be reviewed frequently and updated inorder to provide valid input for the next phases.

In the example of the KM system, the assess-ment of the organization’s needs was performedin a series of meetings with different members ofthe organization, in which we used structuredinterviews with some open-ended questions.We identified three different groups as target

audience: (a) members, (b) managers, and (c)customers. Members are the individualsemployed in the organization who are directlyresponsible for correctly applying the TBPs onan everyday basis. Managers are supervisors ofthe members who apply the TBPs, but they donot use them as part of their own daily work.Managers are, however, responsible for the cor-rect application of the TBPs by the members ofthe organization that they supervise. Customerstake possession of the products that result from theapplication of the TBPs by the members of the or-ganization. Customers have a vested interest in thecorrect application of the TBPs since it affects thequality of the product that they receive.

After the primary goal of the project was es-tablished—developing a system to promote theuse of the TBPs throughout the organization—we identified secondary goals, which were (a)the capturing of knowledge from experiencedmembers of the organization and (b) trainingnew employees who were inexperienced. Con-straints of importance were other systems (e.g.,document management systems) that were cur-rently in use in the organization, as well as

Figure 2 Living-systems approach to the development of knowledge management systems



legacy solutions that had to be integrated withthe new system. The objectives we identified forthis system included the availability of access tothe TBP documents, decision support in apply-ing the TBPs, just-in-time instruction in the useof TBPs, a mechanism to share case examples ofthe actual use of TBPs in the work environment,and forums for the collaboration among usersthat helps build a community of users.

Design Instructional InformationArchitecture

Based on the results of the analysis of the end-user requirements, the purpose of the informa-tion architecture phase is to define the contentand functionality of the system and its generalstructure as well as the navigation system(Rosenfeld & Morville, 1998). We use the terminstructional information architecture for a concep-tual design that emphasizes the instructionalfunction of the system and that focuses on thefacilitation of the users’ learning processes. Thisimplies that the design of the functionality andstructure of the system is informed by ap-propriate learning theories and frameworks,and that the navigation supports the instruction-al function of the system.

Based on the learner characteristics, end-userrequirements, and goals and objectives deter-mined in the previous phase, the first step of thisphase is to specify the functionality that willrealize the objectives of the system. In thisprocess of mapping system functionality ontoobjectives, one or more system features arechosen that can help achieve a particular objec-tive within the resources and constraints of theculture of the organization and the charac-teristics of the learners. Next, the content ismapped onto the features, that is, the contentcovered by or contained within each feature isspecified. In cases where the model is applied tothe design of an environment that relies on con-tent contributed by the users, the type, format,and structure of this content are defined. Thecontent itself can be organized based on themost appropriate organization scheme (e.g., al-phabetical, chronological, geographical, topical,or task-specific) and structure (e.g., hypertext,

hierarchical, or hybrid) (Rosenfeld & Morville,1998; Rothwell & Kazanas, 1997). The output ofthis phase is an information architecture flowchart that shows the major components of thesystem and how they are connected, and thatprovides initial specifications for theirfunctionality.

One way in which this phase differs fromother ISD approaches is that the functionality ofthe system also includes features that allow forthe accommodation of change and growth.These features include administrative tools, KMcapabilities, and living system capabilities (seeTable 1). Administrative tools allow for the easyupdate and modification of user informationand of certain periodically changing content thatwould otherwise require changes to the hyper-text markup language (HTML) documents.They also include project management tools forplanning and communication between the or-ganization and the developers. KM capabilitiesallow users not only to retrieve informationfrom the system, but also to capture and storerelevant information in the system, making itavailable to others. Living-systems capabilities

Table 1 Examples of capabilities thatallow for accommodation ofchange in the KnowledgeManagement system

Capabilities

Administrative ToolsUser-Management ToolsContent-Management ToolsProject-Management Tools

Knowledge Management CapabilitiesKnowledge AcquisitionDecision Support On-line InstructionCommunication Tools Reference MaterialsSearch

Living System CapabilitiesOverall System Status and GrowthCurrent Demand on System ResourcesUsage Statistics and Growth Rate for FeaturesTracking of Usage Patterns (Log File Information)


42 ETR&D, Vol. 50, No. 1

can be described as the digital nervous system ofthe KM system that provides real-time informa-tion, such as overall system status, usage statis-tics for features, demand on resources, andsystem growth. This information can be used forautomatic, semiautomatic or manual adjust-ments to the system to accommodate the chang-ing requirements and system growth.

In our case example, the design of the infor-mation architecture is based on situated learningtheory (Lave, 1988; Lave & Wenger, 1990; Wil-son, 1993), and cognitive flexibility theory (Spiro& Jehng, 1990), which we integrated in a col-laborative cognition model (Plass, Salisbury, &March, 2000) that describes the construction ofknowledge on an individual level, team level,and organizational level. Using this model, weidentified the categories of knowledge that eachof the objectives included—factual, conceptual,procedural, and metacognitive—which arebased on a revision of Bloom’s taxonomy(Bloom, 1956) developed by Anderson et al.(1998).1 Factual knowledge concerns terminology,specific details, and elements of the technicalbusiness processes. Conceptual knowledge relatesto theories, models, principles, and generaliza-tions used in the TBPs. Procedural knowledge in-cludes skills, algorithms, techniques, and othermethods that are specific to a product or processwithin the TBPs. Finally, metacognitive knowledge,an addition to Bloom’s (1956) taxonomy byAnderson et al. (1998), is knowledge aboutknowledge and involves general strategies forlearning, thinking, and problem solving.Metacognitive knowledge also includesknowledge concerning the appropriate contextsand conditions for the use of the strategies them-

selves. We defined the functionality of the sys-tem by choosing system capabilities that wereable to capture these different categories ofknowledge, and defined the structure and or-ganization of the system as task oriented. Wealso included KM features that allow for the ac-commodation of change and growth (Table 2).

A checkmark in the intersection between therow of a capability and the column of an end-user category in Table 2 indicates that thiscapability is designed for that group of endusers. We designed at least one capability foreach of the groups in the target audience: mem-bers, managers, and customers:

• Knowledge acquisition capability includes twotypes of features that were designed tobenefit mainly the members of the organiza-tion while being useful for the decision-making process of managers. One featureallows organizing and storing of casestudies,2 that is, cases that illustrate, in depth,the correct application of a TBP, mainlycreated by subject-matter experts for use asexamples during instruction. The best-prac-tices feature enables the members of the or-ganization to capture and share actualreal-life case examples that illustrate solu-tions to problems of the correct application ofTBPs in the field. With these two features, theknowledge-acquisition capability facilitatesthe capturing and sharing of factual and pro-cedural knowledge as it is created by endusers to solve unique problems—each solu-tion is represented as a one-of-a-kind case.This knowledge contains the facts and stepsthat were used in solving their uniqueproblems. Metacognitive knowledge is usedin this feature to assist end users in finding aprevious solution that will help them withtheir current problem.

• Decision support capability consists of twotypes of features that were designed tobenefit mostly the organizational members.The first decision-support tool providesguidance in finding where a particular sub-

1 One of the major differences in the structure of this revisedtaxonomy by Anderson et al. (1998) to the original taxonomyis that knowledge was moved into a separate dimension thatdistinguishes between factual, conceptual, procedural, andmetacognitive knowledge. The other categories of the originalframework are grouped into a “Process Dimension” thatdescribes the cognitive processes of the learner for retentionand transfer. Another major change is that the revisedtaxonomy renames the categories from Bloom’s original“knowledge, comprehension, application, analysis, synthesis,and evaluation” to “remember, understand, apply, analyze,evaluate, and create.” In the revision, “create” is now thehighest level of cognition that describes individuals puttingelements together to form a novel coherent whole or makingan original product.

2 The term case study is used here for an in-depth example ofa concept or procedure, not as a reference to the qualitativemethod of inquiry.



topic is discussed in the TBPs. The seconddecision-support tool provides guidance forfinding where a specific activity in theProduct Realization Process is discussed inthe TBPs. Both features facilitate knowledgetransfer in the organization: Experiencedmembers provide the expertise for the designof the decision-support capability whilemembers with less experience use the em-bedded expertise to find and complete theTBPs. Managers with less detailedknowledge of the TBPs may also benefit fromuse of the expertise in their supervisory role.Knowledge in the decision-support capa-bility is the step-by-step information neededto complete a procedure from start to finish.The decisions of which step to completeunder what kinds of conditions is a form ofconceptual knowledge. This means that thedecision-support capability also embodies ahigh level of conceptual knowledge concern-ing the general application of the TBPs. Thiscapability also utilizes metacognitive knowl-edge to assist end users in finding and apply-ing this knowledge in a just-in-time manner.

• Instruction includes instructor-led classroom

instruction and just-in-time on-line tutorials.Members of the organization can benefitfrom all the instruction that is available—in-structor-led and on-line. One advantage ofinstructor-led instruction for members is thatit provides an opportunity to ask questionsconcerning specific details about applyingthe TBPs. Managers and customers, ingeneral, can meet their needs with on-line in-struction that provides just-in-time tutorialsfor all levels of TBPs, from a general over-view to the specifics of applying TBPs in aparticular context, such as software applica-tion development. On-line instruction in-cludes some factual, but mostly conceptualknowledge. It is high-level knowledge con-cerning the general principles of a complexsystem. Hence, the on-line instruction doesnot focus on the intimate characteristics of theTBPs, but rather on the general application ofthe policies and procedures.

• Communication consists of three features, (a) a“points of contact” e-mail feature, allowingusers to contact experts within a subject area,(b) a “threaded discussion” list that allows allusers to participate in ongoing discussions of

Table 2 Functionality of the Knowledge Management system

End-UsersFunctionality Members Managers Customers

Knowledge AcquisitionCase Studies √Real Life Examples (Best Practices) √ √Decision SupportDecision Support by Subject √ √ √Decision Support by Activity √InstructionInstructor Led (Synchronous) √Online Tutorials (Asynchronous) √ √ √CommunicationPoints of Contact (e-mail) √ √Threaded Discussion √Frequently Asked Questions √User Feedback √ √ √Reference MaterialsTechnical Business Practices (TBPs) √ √ √Related Materials (Manuals, Documents) √SearchFull Text Search √ √ √Search Index √ √ √


44 ETR&D, Vol. 50, No. 1

issues regarding TBPs that are of concern tothem, and (c) an “FAQ” list, providing meansto collect frequently asked questions andresponses to them. These communication fea-tures were designed to extend the methodsavailable for end users of the KM system tocapture and share knowledge by building acommunity of learners and users.

• On-line reference materials is a capability thatprovides access to on-line documentationdescribing the TBPs for all end users. Thesematerials, containing mostly factual and pro-cedural knowledge, describe the details andstep-by-step knowledge needed to completea procedure from beginning to end. The easyaccess and cross-reference of related on-linematerials mainly will benefit organizationalmembers by helping them to ensure that theapplication of the TBPs is consistent with theother regulations they must follow.

• Search capability provides all users of the KMsystem with quick access to referencematerials that relate to specific topics. It in-cludes a full text search where end users caninput keywords or phrases to the search en-gine and receive customized references to alloccurrences that match the search criteria.The search results can, depending on theuser’s request, be presented in different levelsof “granularity,” such as on a paragraph, sec-tion or subsection, or document level. In ad-dition, a topical search, based on an index ofterms, can be employed by users who are lessfamiliar with the TBP terminology.

Develop Instructional Interaction Design

The purpose of the interaction-design phase is todesign the user interaction with the (visual) in-terface of the system capabilities and features,that is, design the communication between theuser and the system functionality. Based on theinformation architecture flow chart and the ini-tial specifications for the system featuresprovided in the instructional-information-ar-chitecture phase, in this phase the details of thefunctionality are defined.

The steps to be performed in this phase in-clude (a) a task analysis, (b) a cognitive task

analysis, (c) the design of the steps of thehuman-computer interaction for each featureand the selection of interface elements that sup-port them, and (d) some general screen layoutdecisions. They further include (e) the definitionof the logic of the feature, that is, the way itprocesses and manipulates the user input, and(f) the type of system response that will beprovided. This includes questions about theinput required and how it is given, as well asabout the output and its format. For example,should a set of radio buttons or a pull-downmenu be used for making a selection? How willthe system respond, especially in the case of anerror? How can it be assured that meaningfuloutput will be provided?

As in the previous phase, we use the term in-structional interaction design to put special em-phasis on the users’ learning processes and onthe cognitive processes involved in the learningtask. The goal of instructional interaction designis the design of instructional strategies that areappropriate for the target audience and the in-structional content, and that take advantage ofthe unique characteristics of the deliverymedium. This can be achieved, for example, byapplying principles derived from cognitive loadtheory (Sweller, 1994; Sweller, Chandler, Tier-ney, & Cooper, 1990), which distinguishes be-tween the intrinsic cognitive load imposed bythe instructional material itself and the ex-traneous cognitive load imposed by the instruc-tional activities, as well as principles derivedfrom media attribute theory (Salomon, 1979),which provides a framework to describe the uni-que features and capabilities of a particularmedium. This instructional focus also concernsthe design of interactions with the system thatreduce cognitive overhead they themselves im-pose, allowing the users to allocate more of theircognitive resources to the processing of the in-structional material. This may require a cogni-tive task analysis to help identify the cognitivedemands of a task performed in the system (Les-gold, 2000; Schraagen, Chipman, & Shute, 2000).

The output of the interaction-design phase isdetailed specifications of the way the interactionof the user with the system is performed. Inmany cases, these specifications are completedin the form of written descriptions along with



Figure 3a Interactive design of the decision support feature of the sample knowledgemanagement system

Figure 3b Information design of the decision support feature of the sample knowledgemanagement system


46 ETR&D, Vol. 50, No. 1

nonfunctional implementations in HTML thatshow the type and placement of the interfaceelements. As an alternative, storyboards can beused, in which the interaction design is visual-ized.

In the example of the KM system, the interac-tion design was where we defined the details ofthe functionality of the features included in theinformation architecture. For the decision-sup-port feature, for example, we defined the userinteractions and interface elements as shown inFigure 3a. For the topic listed at the top of thescreen (Product Realization Process), subtopicsare presented for the user to choose from. Whenthe user moves the cursor over a subtopic,detailed information designed to assist the userin the decision-making process is presented inthe lower part of the screen. When the user clickson a subtopic it becomes the new topic for whichfurther subtopics are presented.

Develop Instructional Information Design

The purpose of the information-design phase isto specify the appearance of the interface ele-ments defined in the interaction design and todefine the representation mode of the informa-tion presented on the screen, for example, astext, picture, video, animation, or sound. Infor-mation design also includes the color paletteused for the visual interface, color coding, back-ground images, typefaces and their forms (at-tributes) used for on-screen text, and theappearance of system controls such as buttons,scroll bars, and navigational elements.

Appearance goes beyond the “look and feel”of the visual interface, however. Based on theinput from all three previous phases, the infor-mation design also includes design decisionsthat will have an impact on the users’ learningprocesses (Plass, 1998). Such decisions concernthe presentation mode and sensory modality fora particular piece of information and the com-bination of information in different presentationmodes. We use the term presentation mode for theformat used to represent instruction (e.g., text orpictures), and the term sensory modality todescribe the sensory channel used to perceivethe information (e.g., auditory, visual, tactile).

For example, the information designer mighthave to decide if an instructional text should bepresented visually as text printed on the screenor in auditory format as narration. If instruction-al animation is to be included, text labels couldbe added in the animation or separately in alegend. Visual information could be presentedas photographic image, line drawing, or video.The combination of visual and verbal informa-tion raises a number of questions as to their cog-nitive processing as well: Some forms ofcombining visual and verbal information havebeen shown to result in improved learning,while other forms have negative effects causedby the high cognitive load imposed by thematerials (Mayer, 2001; Plass, Chun, Mayer, &Leutner, 1998; Sweller, 1994). We refer to infor-mation design that takes into account such con-siderations of the cognitive impact of the choiceof the presentation mode of information as in-structional information design.

The information-design phase is conductedbased on the needs, requirements, and learnercharacteristics identified in the analysis phase;the specifications of the content and features aswell as the structure of the system defined in theinformation architecture; and the specificationof their functioning described in the interactionphase. Information design is informed by thecognitive science principles of learning from textand pictures (Mandl & Levin, 1989), from maps(Schnotz & Kulhavy, 1994) and from multimedia(Mayer, 2001). To define the visual appearanceof the navigation system, information designersemploy methods of wayfinding, that is, strategiesand design elements that assist users in navigat-ing the system and in orienting themselveswithin it and with relation to the desired loca-tion or function (Passini, 1999). The output ofthis phase is the specification of the appearancesof these features, either in form of nonfunctionalHTML pages or as storyboards.

In the example of the KM system, the infor-mation design had to be performed under theorganization’s strict constraints of acceptable“browser” plug-ins and “streaming” media.Due to the settings of the existing “firewall,” itwas not possible to use any streaming media orplug-in-based components such as MacromediaFlash™ or Director™. The information designer



had to work within these constraints to designthe visual appearance of the Web pages. Figure3b shows the visual design of the decision-sup-port tool. In comparison to the interactiondesign in Figure 3a, the information design inFigure 3b shows how color was applied andhow the information was organized.

Implement System Design

Based on the input from the previous phases, thepurpose of the implementation phase is toproduce a prototype of the KM system. Ideally,the development and programming tools usedfor the implementation of the system are chosenin this phase based on the features that need tobe implemented. In reality, however, these toolsare often specified as part of the initial require-ments of the organization.

Often referred to as the production phase,this phase includes the production of graphicsand other media elements such as sound andvideo, the scripting and programming, and thedesign and implementation of the database back-end, a set of tables in a data base managementsystem (DBMS) that comprise the content of thesystem. This separation of content and codeenables the technical implementation of the fea-tures and capabilities that allow the system togrow and accommodate change: Users can addtheir own examples, case studies, and solutionsto the existing knowledge base. Authors can useforms to add, delete or change information inthe DBMS tables without taking the system off-line, and without making modifications to thesource code. The system administrator can usean analysis of the number of new records inspecific tables to gauge the rate of growth foreach feature. These instruments to measuregrowth in the various components of the system,combined with tools that analyze the user logfiles, constitute the digital nervous system of theKM system.

The output of this phase is a functioningprototype of the KM system that includes all ofthe features specified in the information ar-chitecture. The reason the system is classified asa prototype, however, is that some features maynot be fully implemented, the need for user test-

ing and evaluation remains, and most featuresdo not yet contain extensive amounts of cap-tured knowledge. The evaluation of theprototype will be the content of the next phase;the capturing of knowledge into the system willbe done by the users of the system once it isdeployed.

In our example, we used our conceptual sys-tem design—the information architecture—aswell as the interaction design and informationdesign to develop a prototype KM system. Thesystem was implemented as a Web applicationusing HTML, JavaScript scripting language, andMacromedia ColdFusion Web application ar-chitecture, with a backend implemented usingMicrosoft SQL Server DMBS.

Conduct Developmental Evaluation

The purpose of the evaluation phase is to assessif the features and functionality of the productmeet the stated objectives. One level of evalua-tion is performed on the output of each phasebefore the next phase can begin. After onedesign cycle has been completed and all phaseshave been realized, an evaluation of the result-ing product is conducted before the nextdevelopment cycle can begin. This ongoingevaluation is at the heart of the system’scapability to accommodate changes in userneeds and requirements. Tessmer, McCann andLudvigsen (1999) described how a needs reas-sessment, conducted after training has beendelivered, makes it possible to determine if thetraining need was met or if the instructionresulted in overtraining or undertraining. Thisphase of the design model, following the samespirit as the needs reassessment, is concernedwith the assessment of whether the needs of theorganization and of individual users have beenunderserved or overserved in order to allow thedevelopers to modify the system accordingly.The nature of the resulting evaluation is there-fore both formative and developmental. The on-going process of monitoring the status ofmeeting, overserving or underserving userneeds gives the evaluation a developmentalcharacter since it is seen as an essential part ofthe design process without which a successful


48 ETR&D, Vol. 50, No. 1

product could not be achieved and maintained.Developmental evaluation is used here todescribe the long-term partnering relationshipswith the organization for the purpose of col-laboratively developing and maintaining thesystem (Patton, 1994).

In order to be able to identify changing needsof the organization in a timely manner, a forma-tive evaluation of each phase is necessary beforethe next phase can be approached, which isvisualized using dotted lines in Figure 2. Thisevaluation should always include a reassess-ment of the requirements of the users and the or-ganization. In the first phase of our model,Analyze End-User Requirements, the results ofthe analysis need to be evaluated. Methods forthis type of evaluation include the use of focusgroups or an expert review of the goals, objec-tives, and target audience specifications bystakeholders in the organization. The living sys-tem features of the KM system provide anothermethod for identifying changes in these areas.When users are required to complete a one-timelog-in procedure that asks about demographicinformation, such as their level of expertise withthe subject matter and in their job, changes in theuser demographics can be identified. Beyondthat, the analysis of the functionality used bymembers of a particular group within the targetaudience provides initial information regardingthe kinds of needs that the members of thisgroup have. If combined with surveys and inter-views, this method allows for a timely accom-modation of changes in the target audience or inthe needs of certain groups of users.

The purpose of the evaluation of the outputof the next phase, Design Instructional Informa-tion Architecture, is to validate that the featuresand capabilities included in the information ar-chitecture and their content and structure willindeed be able to meet the requirementsspecified in the analysis phase. As mentionedabove, for this evaluation to be meaningful, areassessment of these requirements has to beperformed first. The evaluation of this high-levelconceptual design is not aimed at achieving cer-tainty that any of the features in the informationarchitecture will indeed be effective in thedesired way. What is more important in thisphase is to verify that the instructional theories

and frameworks as well as epistemic andphilosophical beliefs that informed the design ofthe features and their structure are appropriatefor the organization and its culture and that thefeatures have the potential to sufficiently sup-port the intended type and format of learning. Inaddition, the appropriateness of the navigationsystem is assessed. Therefore, an evaluation ofthe information architecture might include areview by experts of the mapping process of fea-tures onto objectives and content onto features,as well as the navigation, and the approval ofthe features by the client. The evaluation of theeffectiveness of the functionality, structure, andnavigation can only be performed once aprototype has been implemented.

The Develop Instructional Interaction Designphase and the Develop Instructional Informa-tion Design phase can also be evaluated bestonce a prototype is available. However, aftereach phase is completed, the interaction designspecifications and information design specifica-tions can be evaluated in terms of how thedesign was informed by the appropriatetheoretical foundation. In other words, an expertreview can reveal whether the required condi-tions for the application of a particular designprinciple were met and if this design principle ortheory was applied in an appropriate way.Another focus of this evaluation is Do thechosen instructional strategies and the way theyare designed have the potential to supportlearners’ cognitive processes and can theyfacilitate the intended type of learning? Again,this is no assurance that the interactions and ap-pearance of the features designed are effective,but it does allow for the elimination of a firstlevel of potential problems.

A method that has a lot of promise for thisreview of the application of design principles ortheories in the design is the use of a pattern lan-guage that articulates and communicates thedesign of the entire system in a coherent, formalway (Alexander, Ishikawa, & Silverstein, 1977;Tidwell, 1999). The units of such a language aredesign rules, or patterns, that capture the solu-tions to specific issues or problems in the designprocess in a particular context, and are thereforeneither too abstract nor too specific (Tidwell,1999). Applied to the design process of a com-



plex system such as a KM system, the specifica-tion of these patterns and their use in the designof the various features of the system can assist ateam of designers in agreeing on a common andcoherent way of communicating with the userand in the evaluation of the application of theappropriate pattern.

Once one cycle of the design process has beencompleted and a prototype of the system is im-plemented, usability and learning outcome ofthe resulting system are evaluated. Usabilitytesting includes the assessment of the actual easeof use of the system and of learning with it,which is evaluated using criteria such as thetime needed to learn specific functions of thesystem, user retention of these commands overtime, the speed of task performance, the errorrate in such performance, and the number ofclicks or steps for the task performance(Shneiderman, 1992). It also includes user accep-tance testing that assesses the perceived useful-ness of the system and of particular features, theperceived ease of navigation, the appropriate-ness of terminology used, and the consistencyand match of the features with user needs andtasks. Methods of inquiry used for usability as-sessment include surveys and interviews, butalso think-aloud protocols, walk-through tech-niques, field observations, videotaping, andrecording log files of user actions. Applied toour model, usability evaluation means theevaluation of specific design decisions made inthe information architecture, interaction design,and information design.

The evaluation of the learning outcome canbe conducted on any of Kirkpatrick’s (1994) fourlevels of evaluation. On Level I, the reaction ofthe users and how they perceive the usefulnessand value of the system is assessed, which cor-responds to a usability assessment specific tolearning. While it serves only a limited functionin actually assessing learning and in determin-ing the effectiveness of the system, this level isused most often for formative evaluation pur-poses. Level II is concerned with the learningoutcome and instructional effectiveness of thesystem. If the system has well-defined instruc-tional objectives, this effectiveness is measuredby using the criterion-referenced test items thatwere developed in the design phase of more

traditional ISD models. For systems that do nothave such well-defined instructional objectives,alternative methods for the assessment of thelearning outcome have to be considered. Thesecould include interviews or observations of theperformance of users while they are using thesystem, and the analysis of journal entries,projects, and portfolios. In many cases, however,the absence of well-defined learning objectiveswill result in the use of a Level III evaluation,which measures transfer of learning. In otherwords, the application of the knowledge gainedfrom the system in the users’ daily work perfor-mance is measured, for example, using observa-tions, interviews, surveys, or the analysis ofwork results. The most difficult outcome tomeasure is the impact of the system on the or-ganization, which is described in Level IV. Onthis level, the impact of the system as a changeagent is assessed, and the resulting impact onthe organizational level is described (Preskill&Torres, 1999).

In the example of the KM system, we per-formed evaluations of each of the phases of thedesign process. We especially focused on thereassessment of the user needs and require-ments before we began work in a new phase.This reassessment was performed on a formallevel by inviting a group of “power users,” ex-perts in the Product Realization Process and theTPBs, to a focus group meeting in which theresults of our needs assessment and a draft ofthe proposed information architecture were dis-cussed. Later, we conducted the reassessmenton a more informal level by involving subjectmatter experts, potential users, and instructorsfrom the training department into the process ofdesign and assessment. This combination of par-ticipatory design and formative evaluation hasbeen referred to as developmental evaluation (Pat-ton, 1994).

This approach of using a developmentalevaluation to compare the features andfunctionality of the implemented system withthe current end-user requirements by closing thecircle from Implement System Design toAnalyze End-User Requirements (Figure 2)results in an inherent capability of the model toaccommodate change and growth. If the re-quirements of the end users have changed,


50 ETR&D, Vol. 50, No. 1

modifications to the system can be made auto-matically (e.g., scheduled automatic updates tofrequently changing information), semi-automatically (e.g., update of information basedon administrative tools) or manually (e.g., thedevelopment of new features based on newneeds and requirements).

The ongoing evaluation is based on a multi-level approach to data collection: The living-sys-tem capabilities allow us to analyze the statusand use of each feature in the system at any time,for particular (unidentified) users and acrossgroups of users. On-line surveys, conducted pe-riodically, provide us with information aboutspecific emerging areas of interest. Field visitsare conducted by members of the organization,in which they follow up on trends detected inthe surveys and living-system features. Finally,we receive numerous unsolicited responses byusers through the e-mail feedback feature in thecommunication section. Together, the informa-tion collected on these different levels providesus with a powerful tool of continuously reas-sessing user needs.

Evaluation of the Case Example KM System

After the prototype KM system was imple-mented, we conducted a first evaluation of thesystem capabilities and features. The purpose ofthis evaluation was to obtain feedback regardingthe perceived usability of the system and its per-ceived usefulness, that is, to evaluate user accep-tance of the system. Similar to the method usedin iterative prototyping, we implemented someof the features with limited depth (e.g., on-linetutorials) and others with limited breadth (e.g.,decision support features). We were thereforeinterested in receiving feedback from potentialusers as to which features should have priorityin the implementation of the system. Once thefindings of this evaluation are incorporated intothe system design and a threshold of the amountof content included in the system has beenreached that would make learning experienceswith the KM system meaningful, the effective-ness of the features for learning can beevaluated.

For the evaluation of perceived usefulnessand usability, we developed a questionnairewith items soliciting demographic informationand items where users rated the importance ofthe system’s features for their work and the ac-curacy of the information in the system, andchose which features of the site they used mostand which they used least. We posted a link tothe survey on the main page of the KM systemand by e-mail, asked all users of the system toparticipate in the survey. After a three-weekperiod, 49 of the estimated 150 users across theparticipating facilities had participated in thesurvey, a response rate of about 33%. We foundthat, of the participants in the survey, 45% wereengineers, 31% managers, 14% support staff,and 10% were classified as other. Of the respon-dents 31% rated themselves as being highly ex-perienced or experts in TBPs, 51% had mediumexperience, and 18% low or no experience; 33%had been working for the government organiza-tion for more than 25 years, 39% for 16–25 years,16% for 6–15 years, and only 12% for 5 years orless. In other words, the respondents were main-ly seasoned engineers and managers with manyyears of experiences. On a scale from 1 (lowest) to5 (highest), the means of the responses regardingthe ease of use and accuracy of the informationof each feature were consistently above 4.0, withthe exception of the on-line training tutorials,which had a mean of M = 3.97 (SD = .71) (see Ap-pendix A). On the same scale, the responsesregarding the quality of the overall design of theWeb site had a mean of M = 4.35 (SD = 1.01), theease of navigating the site a mean of M = 4.12(SD = 1.01), and the likelihood that users woulduse the site to find needed information a mean ofM = 4.37 (SD = 1.07).

Aside from this positive feedback regardingthe acceptance of the system and its features, themost important result of the evaluation was thatwe were able to determine a rank order for theimportance of the features for the users by as-king which of the features they used most,second, third, and last (see Appendix B). As theresponses show, the access to the TBP docu-ments, Decision Support by Subject and theSearch function, along with the Dictionarieswere ranked highest. Tutorials, Decision Sup-port by Activity and Case Studies were among



the lowest ranks. Considering the high level ofexperience of the current users, it is not surpris-ing that tutorials or case studies would rankvery low, and access to the documents andsearch tools would rank very high. Of particularinterest, however, was the clear preference forone type of decision support, organized bytopic, over another, organized by the task theuser has to complete, even though both featuresreceived very similar responses with regard totheir accuracy, ease of use, and general impor-tance (Appendix A). These results show that afrequent reassessment of the user population isnecessary to make sure that changes in the re-quired functionality, which can be expected, forexample, as a result of new hires who are less ex-perienced in the subject matter, can be detectedand accommodated.

The results of this first evaluation of the useracceptance of the system provided us with ini-tial feedback on accuracy, ease of use and impor-tance of the features for the respondents of thesurvey. Interviews with managers and en-gineers in the organization indicated that thedistribution of the respondents to the surveywas a fairly accurate reflection of the currentuser population of the system, and that thecapabilities that had received the highest ranks(Document Access, Decision Support by Subject,Search Engine, Dictionaries and Real-life Ex-amples) were indeed a priority for the organiza-tion and should be included in the fieldedsystem. Other features, such as Decision Sup-port by Activity and Case Studies, were given alower priority. Based on these priorities, wewere able to allocate more resources to the fea-tures ranking very high, and fewer to those thatwere ranked lowest.

DISCUSSION AND CONCLUSIONS

While all of the existing methods for thedevelopment of CBI described earlier have beensuccessfully used to develop applicationsthrough the phases of analysis, design, develop-ment, implementation, and evaluation, theywere not conceptualized for building systemswith continuously changing requirements, suchas changes in the characteristics of the target

audience; in the amount, content, type, andstructure of the information; and in the formsand methods of learning and instruction used inthe system. In this article, we therefore describea new design model for KM systems that is con-ceptualized to provide explicit mechanisms todesign features for KM systems that can accom-modate growth and change by applying aliving-systems approach.

The ability to grow and change and, there-fore, to adapt to the changing requirements of itsusers, is the most important characteristic of thesystems for which this approach was developed.In our opinion, this capability has fundamentalconsequences for the design and development ofsystems that take advantage of it. While mostother systems may reach a point where they areconsidered to be a finished product, these sys-tems are, by design, never completed. The con-sequence of such a constant state of change isthat the development process is never com-pleted either. The evaluation of such a system istherefore always of a formative nature, and sinceit involves the ongoing collaboration with theorganization that owns the system, it can also beconsidered a developmental evaluation (Patton,1994). A comparison of this cycle of feedbackand accommodation in the KM system wedesigned and those in an autopoietic systemshows that both have within their own boun-daries the mechanisms and processes thatenable them to produce and reproduce themsel-ves, which is characteristic of living systems.

The design process we employed is cyclic innature, reflecting the philosophy of acceptingchange as a factor in the definition of the KMsystem. While the execution of the first cycle ofthe design involves the initial analysis and as-sessment of need, design of the information ar-chitecture, interaction design, informationdesign, and system implementation, the nextdesign cycle can take advantage of the living-systems features implemented in the system. Inother words, much of the information that hadto be collected in the needs assessment using in-terviews, document reviews, and observation ofperformers can now be captured using the digi-tal nervous system of the KM system, which col-lects information on the use of the system and itsfeatures and gauges its growth. The information


52 ETR&D, Vol. 50, No. 1

obtained from the digital nervous system, com-bined with other sources of information (sur-veys, field visits, and e-mail feedback fromusers) results in a multilevel approach to forma-tive evaluation, in which data collection andevaluation are ongoing. Ethical issues involvedin this data collection do not differ from those inthe collection of any research data and thereforerequire measures to keep the information con-fidential.

The living-systems approach of the designmodel is realized in the form of digital nervoussystem functionality that monitors and recordschanges in the system, a developmental evalua-tion that continuously assesses whether theneeds and requirements of the users havechanged, using the data from the digital nervoussystem and from other means of data collection,and a design model that explicitly takes ad-vantage of this information in order to modifythe resulting KM system.

A number of unresolved issues remain forthe KM system that was implemented based onthe design model described in this article. A firstissue is that more research has to be conductedwith regard to the instructional components ofthe system. For example, the living-systems ap-proach is designed to provide more explicit sup-port for facilitating situated learning (Brown,Collins, & Duguid, 1989) than found in tradi-tional ISD approaches. The KM systemdescribed in this article supports learning at theindividual and team levels from constructivist(Duffy & Jonassen, 1991) and cognitiveflexibility (Spiro & Jehng, 1990) perspectives byallowing learners to construct and share theirown meaning in an ill-structured domainthrough the use of minicases (Plass, Salisbury, &March, 2000). It is also designed to supportlearning at the organizational level (Nonaka &Takeuchi, 1995). However, research is needed toevaluate the effectiveness of this design methodin supporting individual, team, and organiza-tional learning. For our example KM system, anevaluation of the effectiveness of the features forlearning would require that the system had beenin use for a sufficient period of time so that thethreshold of captured information is reachedthat allows for meaningful learning to takeplace.

Another issue concerns the cultural setting inwhich the system will be used. The effectivenessof a KM system that is based on the idea of userssharing their knowledge depends on the readi-ness of the organization to change and to be-come a learning organization. A KM system canact as a change agent, supporting the process oftransforming a hierarchically structured or-ganization into a learning organization. If the or-ganization is not ready for this change, however,even the best KM system is bound to fail. Theanalyze-end–user-requirements phase should,therefore, include an assessment as to whetherthe culture of the organization supports the useof a KM system.

The reliance of the system on user contribu-tion of knowledge poses another challenge. Oneissue is the problem of motivating users to sharetheir knowledge and make it available to others.Because of the additional effort involved in con-tributing information to the KM system, mean-ingful incentives are required to get learners toshare their knowledge with others. Some or-ganizations assess how often the knowledgecontributed by a particular employee was ac-cessed by others and determines incentivesbased on this impact factor. A related issue con-cerns the quality control of the information thatis being contributed. Methods for such a qualitycontrol include review by expert panels or ratingby peers, which often cannot easily be scaled tolarger systems and therefore requires newmechanisms.

Several issues also remain to be solved tomake the design model described in this papermore easily applicable to other projects. Themodel was developed for the specific context ofthe design and development of KM systems.Following the conceptual framework for thecomparison of ISD models described by Ed-monds, Branch & Mukherjee (1994) our modelcan be classified as a descriptive and prescrip-tive model (orientation) with procedural anddeclarative knowledge structures that require anexpert level of experience for the designer. Thetheoretical origin of the model is a living-sys-tems approach, and it is conceptualized for thedesign of KM systems for business training andgovernment (instructional context) on an institu-tional and team level (level of implementation).



Although its general phases should be ap-plicable to the development of other instruction-al systems of a similar nature, we have not yetused this model for such a development.

To assure a clear and coherent communica-tion of the system’s features and functionality tousers, future versions of our model wouldbenefit from adding the use of a pattern lan-guage to the design process (Alexander et al.,1977; Tidwell, 1999). As part of the informationarchitecture, interaction design, and informationdesign phases, the patterns of this languagewould be specified as design rules that capturethe solutions to specific issues or problems in thedesign process of the project. Examples for pat-terns could describe the implementation ofspecific instructional strategies, the type of inter-action with the decision support feature, or theuse of the search tool. By specifying patterns thatare neither too specific no too general, it can beassured that they can be applied to similardesign decisions in different parts of the pro-gram.

Hannafin (1992) laments the insulation of ISD

from innovative design methods and modelsused in other fields and calls for the design of in-struction that gives individuals a “greater role inregulating, not merely participating” in thelearning process (p. 61). The living-systems ap-proach we described in this article aims to sup-port the development of environments that notonly allow individuals to regulate their learningprocess, but that indeed grow and change inorder to accommodate learners’ needs.

Jan L. Plass is with the Educational Communicationand Technology Program, New York University. Mark W. Salisbury is with the OrganizationalLearning and Instructional Technologies Program,University of New Mexico. All trademarks used are properties of theirrespective owners. Correspondence concerning this article should beaddressed to Jan L. Plass at [email protected] or bymail to New York University, ECT Program, 239Greene St., New York, NY 10003, or to MarkSalisbury at [email protected] or by mail toEducational Office Building, OLIT, University ofNew Mexico, Albuquerque, NM 87131.


54 ETR&D, Vol. 50, No. 1

REFERENCES

Alessi, S., & Trollip, S. (1991). Computer-based instruc-tion: Methods and development (2nd ed.). EnglewoodCliffs, New Jersey: Prentice-Hall

Alexander, C., Ishikawa, S., & Silverstein, M. (1977). Apattern language. Towns, buildings, construction. NewYork: Oxford University Press.

Anderson, L.W.; Krathwohl, D.R.; Airasian, P.W.;Cruikshank, K.A.; Mayer, R.E.; Pintrich, P.R.; Raths,J.D.; & Wittrock, M.C. (1998). Taxonomy for learning,teaching and assessing: A revision of Bloom’s taxonomyof educational objectives. New York: Longman.

Banathy, B. (1991). Educational systems design: A journeyto create the future. Englewood Cliffs, NJ: EducationalTechnology Publications.

Bloom, B.S. (Ed.) (1956). Taxonomy of behavioral objec-tives: Handbook I: Cognitive domain. New York: DavidMcKay.

Boar, B. (1984). Application prototyping. New York: JohnWiley & Sons.

Boyle, T. (1997). Design for multimedia learning. London:Prentice Hall.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situatedcognition and the culture of learning. Educational Re-searcher, January-February.

Carr, A. (1997). User-design in the creation of humanlearning systems. Educational Technology Researchand Development, 45(3), 5–22.

Coad, P., & Yourdon, E. (1991). Object-oriented design.Englewood Cliffs, NJ: Yourdon Press.

de Hoog, Robert (1994). Constraint-driven softwaredesign: An escape from the waterfall model. Perfor-mance Improvement Quarterly, 7(3), 48–63.

Dick, W. (1996). The Dick and Carey model: Will it sur-vive the decade? Educational Technology Research andDevelopment, 44(3), 55–63.

Dick, W., & Carey, L (1990). The systematic design of in-struction (3rd Ed). Glenville, IL: Harper-Collins.

Dynamic systems development method, Version 2. 1995.Tesseract Publishing.

Duffy, T., & Jonassen, D (1991). Constructivism: Newimplications for educational technology? EducationalTechnology, 31(5), 7–12.

Edmonds, G.S., Branch, R.C., & Mukherjee, P. (1994).A conceptual framework for comparing instruction-al design models. Educational Technology Researchand Development, 42(4), 55–72.

Fenrich, P. (1997) Practical guidelines for creating instruc-tional multimedia applications. Fort Worth, TX: TheDryden Press.

Gagné, R.M., & Briggs, L.J. (1979). Principles of instruc-tional design. New York: Holt, Rineholt, & Winston.

Hannafin, M. (1992). Emerging technologies, ISD, andlearning environments: Critical perspectives. Educa-tional Technology Research and Development, 40(1), 49–63.

Jones, T.S., & Richey, R.C. (2000). Rapid prototypingmethodology in action: A developmental study.Educational Technology Research and Development,

48(2), 63–80.Kirkpatrick, Donald, (1994). Evaluating training

programs. San Francisco, CA: Berrett-Koehler Pub-lishers, Inc.

Koper, R. (1995). PROFIL: A method for the develop-ment of multimedia. British Journal of EducationalTechnology, 26(2), 94–108.

Lave, J. (1988). Cognition in practice: Mind, mathematics,and culture in everyday life. Cambridge, UK:Cambridge University Press.

Lave, J., & Wenger, E. (1990). Situated learning:Legitimate peripheral participation. Cambridge, UK:Cambridge University Press.

Lesgold, A. (2000). On the future of cognitive taskanalysis. In J.M. Schraagen, & S.F. Chipman (Eds.),Cognitive task analysis (pp. 451–465). Mahwah, NJ:Lawrence Erlbaum.

Luhmann, N. (1986). The autopoiesis of social systems.In F. Geyer and J. Van de Zouwen (Eds.) Sociocyber-netic paradoxes: Observation, control, and evolution ofself-steering systems, London: Sage Publications.

Mandl, H., & Levin, J.R. (Eds.). (1989). Knowledge ac-quisition from text and pictures. New York: North-Holland.

Maturana, H., & Varela, F. (1980). Autopoiesis and cogni-tion. London: Reidl.

Mayer, R.E. (2001). Multimedia learning. Cambridge,MA: Oxford.

Moody, S., Hudson, T., & Salisbury, M. (1988). RAPID:A prototyping environment for battle managementinformation systems. Proceedings of the Third AnnualUser-System Interface Conference, Austin, TX.

Nonaka, I., & Takeuchi, H. (1995). The knowledge-creat-ing company. New York: Oxford University Press.

Passini, R. (1999). Information design: An old hag infashionable clothes? In R. Jacobson (Ed.), InformationDesign (pp. 83–98). Cambridge, MA: MIT Press.

Patton, M.Q. (1994). Developmental evaluation. Spe-cial invitational volume: Past, present, future assess-ments of the field of evaluation. Evaluation Practice,15(3), 311–319.

Plass, J.L. (1998). Design and evaluation of the user in-terface of foreign language multimedia software: Acognitive approach. Language Learning and Technol-ogy, 2(1), 35–45.

Plass, J.L., Chun, D.M., Mayer, R.E., & Leutner, D.(1998). Supporting visual and verbal learningpreferences in a second language multimedia learn-ing environment. Journal of Educational Psychology,90, 25–36.

Plass, J.L., Salisbury, M.W., &. March, J. (2000). In-tegrated instruction as a component of a knowledgemanagement system: A case example. Proceedings ofWebNet ‘00, Charlottesville, VA: Association for theAdvancement of Computers in Education (AACE).

Preskill, H., & Torres, R.T. (1999). Evaluative inquiry forlearning in organizations. Thousand Oaks, CA: SagePublications.

Reigeluth, C. (1993). Principles of educational systemsdesign. International Journal of Educational Research,



19(2), 117–131.Roblyer, M. (1988). Fundamental problems and prin-

ciples of designing effective courseware. In D. Jonas-sen (Ed.), Instructional designs for micro-computercourseware (pp. 7–33). Hillsdale, New Jersey:Lawrence Erlbaum Associates.

Rosenfeld, L., & Morville, P. (1998). Information ar-chitecture for the World-Wide-Web. Sebastopol, CA:O’Reilly.

Rothwell, W.J., & Kazanas, H.C. (1997). Mastering theinstructional design process (2nd ed.). San Francisco,CA: Jossey-Bass.

Salisbury, M. (1988). PARGEN: A prototyping tool fornatural language interfaces. Proceedings of the ThirdAnnual User-System Interface Conference, Austin, TX.

Salisbury, M.W., & Plass, J.L. (in press). A conceptualframework for a knowledge management system.Human Resource Development International.

Salisbury, M.W., & Plass, J.L. (2001). Utilizing decisionsupport and case examples for capturing and disseminat-ing knowledge in organizations. Submitted for publica-tion.

Salomon, G. (1979). Interaction of media, cognition, andlearning. Hillsdale, NJ: Lawrence Erlbaum.

Schnotz, W., & Kulhavy, R.W. (Eds.). (1994). Com-prehension of graphics. New York: North-Holland.

Schraagen, J.M, Chipman, S.F., & Shute, V.J. (2000).State-of-the-art review of cognitive task analysistechniques. In J.M. Schraagen, & S.F. Chipman(Eds.), Cognitive task analysis (pp. 467–487). Mahwah,NJ: Lawrence Erlbaum.

Schuler, D., & Namioka, A. (1993). Participatory design:Principles and practices. Hillsdale, NJ: Erlbaum.

Schwier, R.A. (1999). Constructivist approaches to in-structional design. [On-line], Retrieved from theWorld Wide Web on 9/6/2000. Available:http://members.home.net/rschwier/presentations/construct/

Shneiderman, B. (1992). Designing the user interface.Strategies for effective human-computer interaction.Reading, MA: Addison-Wesley.

Spiro, R.J., & Jehng, J.C. (1990). Cognitive flexibilityand hypertext: Theory and technology for the non-linear and multidimensional traversal of complex

subject matter. In D. Nix & R. Spiro (Eds.), Cognition,education and multimedia: Exploring ideas in high tech-nology. Hillsdale, NJ: Lawrence Erlbaum.

Sweller, J. (1994). Cognitive load theory, learning dif-ficulty, and instructional design. Learning and In-struction, 4, 295–312.

Sweller, J., Chandler, P., Tierney, P., & Cooper, M.(1990). Cognitive load as a factor in the structuringof technical material. Journal of Experimental Psychol-ogy: General, 119, 176–192.

Tennyson, R.D. (1997). A system dynamics approachto instructional systems development. In: R.D. Ten-nyson, F. Schott, N. Seel, & S. Dijkstra (Eds.), Instruc-tional design: International perspectives. Volume 1:Theory, research and models, (pp. 413–426). Mahwah,NJ: Lawrence Erlbaum.

Tessmer, M., McCann, D., & Ludvigsen, M. (1999).Reassessing training programs: A model for iden-tifying training excess and discrepancies. Education-al Technology Research and Development, 47(2), 86–99.

Tidwell, J. (1999). Common ground: A pattern languagefor human-computer interface design. [On-line],Retrieved from the World Wide Web on September18, 1999. Available: http://www.mit.edu/jtid-well/common_ground.html

von Krogh, G., Roos, J., & Slocum, K. (1996). An essayon corporate epistemology. In G. von Krogh & J.Roos (Eds.) Managing knowledge: Perspectives oncooperation and competition. London: Sage Publica-tions.

Wager, W., & Gagné, R. (1988). Fundamental problemsand principles of designing effective courseware. InJonassen, D. (Ed.), Instructional designs for micro-com-puter courseware (pp. 7–33). Hillsdale, New Jersey:Lawrence Erlbaum Associates.

Willis, J. (1995). A recursive, reflective instructionaldesign model based on constructivist-interpretivisttheory. Educational Technology, 30, Nov-Dec, 5–23.

Wilson, A.L. (1993). The promise of situated cognition.New Directions for Adult and Continuing Education, 57,71–79.

Yourdon, E., & Constantine, L. (1978). Structured design(2nd Ed). New York: Yourdon Press.


56 ETR&D, Vol. 50, No. 1

Appendix A Means and standard deviations of the users’ rating of the importance, ease ofuse, and accuracy of the features of the KM system (N = 49).

Feature Importance Ease of Use Accuacy

Decision Support by Subject 3.87 (0.87) 4.17 (0.57) 4.10 (0.54)Decision Support by Activity 3.62 (0.89) 4.09 (0.59) 4.10 (0.55)Reference Materials: TBP architecture visualization 3.66 (0.99) 4.16 (0.53) 4.20 (0.61)Reference Materials: TPB numerical list 3.95 (1.01) 4.18 (0.58) 4.26 (0.51)Case Studies for Online Tutorials 3.03 (1.01) 4.13 (0.53) 4.00 (0.53)Examples of Forms 3.33 (1.05) 4.07 (0.47) 4.07 (0.27)Real life Examples 3.63 (0.83) 4.15 (0.42) 3.92 (0.72)Documents (access to TBPs, QC-1, D&P) 4.41 (0.72) 4.26 (0.53) 4.27 (0.63)Online Tutorials 3.54 (0.84) 3.97 (0.71) 3.88 (0.44)Online Dictionaries—Acronyms & Definitions 4.02 (0.81) 4.24 (0.48) 4.10 (0.48)Search 4.20 (0.71) 4.22 (0.48) 4.25 (0.53)Meeting Announcements 3.00 (0.93) 4.22 (0.47) 4.00 (0.86)Communication-Points of Contact 3.23 (0.89) 4.28 (0.60) 4.00 (0.67)FAQs 3.07 (0.71) 4.20 (0.51) 4.09 (0.53)Descriptions of classroom training 2.95 (0.82) 4.13 (0.46) 3.79 (0.72)Announcements of other training 3.05 (0.84) 4.05 (0.50) 3.89 (0.66)

Note. Reponses were given on a 5 point Likert scale, with 1 as the lowest, and 5 as the highest possible response. Standarddeviations are printed in parentheses after the mean.

Appendix B Scores and Ranks for Evaluation Survey Question: Which features of this sitewould you use most, second, third and last? (N = 49)

Rank Score Feature

1 478 TBP and related Documents2 273 Decision Support by Subject3 211 Search—Topical Index4 206 Search—Full Text5 186 Online Dictionaries6 116 Real life Examples: Form Examples7 109 Reference Materials: TPB numerical list8 91 Reference Materials: TBP architecture visualization9 74 Online Tutorials

10 27 Communication–Points of Contact11 25 Decision Support by Activity12 –22 Case Studies for Online Tutorials13 –34 Description of Classroom Training14 –68 FAQ15 –97 Training Announcements16 –128 Communication Mail Room

Note. Ranks was computed based on responses to the question which feature the user expected to use most, second, thirdand last. Ten points were assigned for features selected as most, 9 for second, 8 for third, and –10 for least.

