use of knowledge management systems and the...

77

JOURNAL OF INFORMATION SYSTEMSVol. 22, No. 2Fall 2008pp. 77–101

Use of Knowledge Management Systemsand the Impact on the Acquisition

of Explicit KnowledgeHolli McCall

University of Connecticut

Vicky ArnoldUniversity of Central Florida and

University of Melbourne

Steve G. SuttonUniversity of Central Florida and

University of Melbourne

ABSTRACT: In an era where knowledge is increasingly seen as an organization’s mostvaluable asset, many firms have implemented knowledge-management systems (KMS)in an effort to capture, store, and disseminate knowledge across the firm. Concernshave been raised, however, about the potential dependency of users on KMS and therelated potential for decreases in knowledge acquisition and expertise development(Cole 1998; Alavi and Leidner 2001b; O’Leary 2002a). The purpose of this study, whichis exploratory in nature, is to investigate whether using KMS embedded with explicitknowledge impacts novice decision makers’ judgment performance and knowledgeacquisition differently than using traditional reference materials (e.g., manuals, text-books) to research and solve a problem. An experimental methodology is used to studythe relative performance and explicit knowledge acquisition of 188 participants parti-tioned into two groups using either a KMS or traditional reference materials in problemsolving. The study finds that KMS users outperform users of traditional reference ma-terials when they have access to their respective systems/materials, but the users oftraditional reference materials outperform KMS users when respective systems/ma-terials are removed. While all users improve interpretive problem solving and encodingof definitions and rules, there are significant differences in knowledge acquisition be-tween the two groups.

Keywords: knowledge management systems; knowledge management; declarativeknowledge; explicit knowledge; knowledge acquisition; knowledge trans-fer; ACT-R theory.

We thank Alex Yen for his help in administering the experiments, and Mohamed Hussein, Gim Seow, Ton Cillessen,Cliff Nelson, Jesse Dwyer, Clark Hampton, Angela Han, Sarfraz Khan, Linda Kolbasovksy, Kate Odabashian,Sylvia Santiago, and Stan Veliotis for their assistance during the various stages of this research. We also thankparticipants at a workshop at the University of Connecticut, the 2006 Mid-Year Meeting of the Information Systemssection, and the 2006 American Accounting Association Annual Meeting, and in particular the editor Brad Tuttle,an associate editor, and two reviewers for their feedback on previous versions of the manuscript.

78 McCall, Arnold, and Sutton

Journal of Information Systems, Fall 2008

Data Availability: Data availability is limited by Institutional Review Board guidelinesand restrictions.

I. INTRODUCTION

In recent years, organizations have increasingly realized that one of their most valuableassets is the knowledge that is developed internally and possessed by individuals withinthe organization. For instance, Nagle (1999), a knowledge manager at KPMG, noted

that one of the biggest challenges facing the firm was how to capture, store, retain, andshare the knowledge possessed by the firm’s professionals. Cameron (2000, 3) similarlynoted, ‘‘Knowledge is power, but without the adequate management of that knowledge, theconsequences for [organizations] could be devastating.’’ Not surprisingly, technology isviewed as the key enabler to effective knowledge management. Early technological strat-egies focused on the use of intelligent systems,1 but these strategies have not been terriblyeffective (Duchessi and O’Keefe 1992). More recently, corporate efforts are focusing on aclass of technologies referred to as knowledge-management systems (KMS) (Leech andSutton 2002).

Knowledge management can be defined as the organizational ‘‘efforts designed to (1)capture knowledge; (2) convert personal knowledge to group-available knowledge; (3) con-nect people to people, people to knowledge, knowledge to people, and knowledge to knowl-edge; and (4) measure that knowledge to facilitate management of resources and helpunderstand its evolution’’ (O’Leary 2002a, 273). Knowledge-management systems (KMS)focus on bringing together the explicit knowledge that exists in organizations, the know-what that can be easily documented and shared (Sambamurthy and Subramani 2005), suchas basic definitional information (e.g., technical terminology), procedures for performingtasks (e.g., audit checklists), guidelines for interpretation (e.g., GAAP guidance), and pre-vious problem resolution examples (e.g., client memos outlining solutions to issuesraised)—information often referred to as ‘‘three-ring binder’’ knowledge (Dilnutt 2002). Asnoted by Alavi and Leidner (2001b), KMS initially contain these types of explicit knowl-edge and are later expanded upon with a body of tacit knowledge that continues to growas users add their interpretations of the explicit knowledge to the system’s knowledge base.Tacit knowledge is the know-how that is difficult to document and emerges from experiences(Sambamurthy and Subramani 2005). Alavi and Leidner (2001b) further note that accessand/or assimilation of the explicit knowledge in such systems is a necessary precursor toeffective use of the accumulated tacit knowledge in the system. This is consistent withrecent findings in the knowledge-based system (KBS) literature showing that novice usersgravitate toward explicit knowledge support while experienced decision makers gravitatetoward available tacit knowledge support (Arnold et al. 2006).

The use of KMS to support an organization’s professionals in their decision makingthrough organizational knowledge creation is a double-edged sword. The ready availabilityof explicit knowledge support in a KMS should allow individuals to improve decisionperformance (Gonzalez et al. 2005), but the potential impact on the development of exper-tise by individuals within the organization remains an unknown. Alavi and Liedner (2001b)note that some researchers raise questions as to whether KMS users may not develop theirown knowledge while relying on the expertise of others, which may lead to a lack ofexpertise development in the next generation of organizational ‘‘experts’’ and ultimately a

1 A type of knowledge-based system that is also commonly referred to as expert systems, intelligent decisionsupport systems, and intelligent decision aids.

Use of Knowledge Management Systems 79


dwindling of human expertise within the firm (e.g., Cole 1998; Powell 1998; O’Leary2002a).

The purpose of this study is to empirically investigate whether a KMS providing explicitknowledge impacts the decision-making performance and the acquisition of explicit knowl-edge by novice users. The entire basis for the investment in and development of knowledge-management technologies is premised on the belief that an effective KMS should dissem-inate knowledge throughout the organization and provide the necessary components toimprove decision-making capabilities (Alavi and Leidner 1999). The impact of the use ofKMS on explicit knowledge acquisition is critical given that explicit knowledge providesthe foundation for and is the precursor of tacit knowledge development (Alavi and Leidner2001b; Anderson 1987; Anderson 1990; Anderson 1993; Anderson et al. 1997; Chi et al.1989; Roberts and Ashton 2003). As such, acquisition of explicit knowledge is a criticalcomponent in the development and sustenance of expertise (Herz and Schultz 1999). Yet,prior research provides little insight on the effects of contemporary decision support tech-nologies, such as KMS, where the user initiates search and retrieval of the embeddedknowledge (Alavi and Leidner 2001b).

This study utilizes an experimental methodology to investigate the impact of KMS ondecision-making performance and acquisition of explicit knowledge. A pretest-posttest de-sign is implemented to investigate the acquisition of explicit knowledge focusing on dif-ferences between individuals using a KMS (KMS group) versus individuals using traditionalreference materials such as office manuals and textbooks (traditional group). Results indi-cate that the KMS group outperforms individuals in the traditional group when they haveaccess to a KMS; however, the advantage disappears when the KMS is removed. Addi-tionally, results indicate that both groups acquire various types of explicit knowledge. Thetraditional group tends to encode more rules in memory, while the KMS group tends toacquire higher-level explicit knowledge (i.e., interpretative problem-solving skills) which iskey to the formulation of tacit knowledge.

This paper contributes to the literature by experimentally examining the impact of KMSuse on the acquisition of explicit knowledge and represents an initial step in addressing theassociated knowledge transfer issues. Little empirical evidence is available on the impactof KMS on user performance (e.g., Gonzalez et al. 2005). Rather, the extant KMS literaturegenerally consists of descriptive studies (Davenport et al. 1998; Alavi and Leidner 1999;Dilnutt 2002; O’Leary 2002b), design science studies (Earl 2001), and case studies (Alavi1997; Baird et al. 1997; Bartlett 1996; Henderson et al. 1998; Thomas et al. 2001; andWickramasinghe and Mills 2002). Prior research has not addressed whether knowledgetransfer actually occurs (Grover and Davenport 2001). Alavi and Leidner (2001b) note thatfuture research needs to address if and to what degree knowledge can be transferred withinthe firm. This study focuses on this gap in research and addresses knowledge acquisitionassociated with KMS use.

The remainder of this paper is organized into four sections. The first section presentsthe theory and associated hypotheses and research questions. The second and third sectionsprovide an overview of the research method and the results of the experimental study,respectively. The fourth and final section discusses the implications of the study results andconsiders opportunities for future research that could extend the research reported here.

II. BACKGROUND AND THEORETICAL DISCUSSIONKnowledge is defined as a ‘‘justified true belief’’ (Nonaka 1994, 15) and can be viewed

as a state of mind, an object, a process, a stipulation of having access to information, or a



FIGURE 1Stages of Individual Knowledge Acquisition

Procedural StageDeclarative Stage

Declarativeencoding

Interpretiveproblemsolving

Compilation(of productionrules)

Tuning (ofproductionrules)

Declarativeknowledgeacquisition

Proceduralknowledgeacquisition

Definitions,rules,examples

capability (Nonaka 1994). In an organizational setting, knowledge is an asset that enablesfirms to obtain a competitive advantage (Alavi and Leidner 1999) and is of limited valueif it is not disseminated to others (Grant 1996). Knowledge management enables organi-zations to leverage the collective knowledge among members of the organization and sustaincompetitive advantage.

A key component of knowledge management is to provide access to stored knowledgecomponents to improve decision making and to facilitate knowledge acquisition by the user.ACT-R theory (depicted in Figure 1) provides a conceptualization of the process by whichknowledge is acquired and provides a foundation for understanding how a KMS mightimpact individuals’ knowledge-acquisition processes (Anderson 1990; Anderson 1993;Anderson et al. 1997). While Anderson presents his theory in terms of declarative andprocedural knowledge, knowledge embedded in a KMS is generally referred to as explicitand tacit, respectively (Alavi and Leidner 2001b; Sambamurthy and Subramani 2005). Inboth literatures declarative and explicit are defined as know-what knowledge, and proce-dural and tacit are defined as know-how knowledge (Anderson 1993; Sambamurthy andSubramani 2005).

Act-R theory proposes that an individual encodes or stores definitions, examples, andrules into long-term memory and utilizes this declarative knowledge in a problem-solvingstrategy called interpretive problem solving. Interpretive problem solving is defined as solv-ing problems by analogizing from examples. These examples may come from an externalsource (e.g., KMS) or be retrieved from declarative knowledge that has been encoded intolong-term memory. With practice an individual encodes an increasing number of examples,resulting in declarative knowledge acquisition. After declarative knowledge has been ac-quired, the individual may then move into the procedural stage of knowledge acquisitionby compiling the analogy process into a production rule (i.e., compilation). Both declarativeencoding and interpretive problem solving must occur prior to compilation (rule creation);



however, declarative encoding alone can result in declarative knowledge acquisition. Theproduction rules (or condition-action pairs likened to if-then statements in a programminglanguage) created by the compilation process are then tuned (improved or enhanced), whichexpands procedural knowledge (i.e., procedural knowledge acquisition). Procedural knowl-edge is the ability to apply and extend declarative knowledge and is acquired throughexperience; thus, it is considered a key antecedent to expertise (Anderson 1993).

Production rules and the linked declarative knowledge are stored in long-term memory.Long-term memory stores consist of declarative memory and procedural memory. The basicunit of knowledge in declarative memory is a chunk, while the basic unit of knowledge inprocedural memory is a production rule. Declarative knowledge is factual knowledge thatcan be described (definitions, rules, and examples), while procedural knowledge is mech-anistic and can only be inferred by behavior (Anderson 1993). For example, declarativeknowledge would entail knowing that a bike has wheels, handlebars, pedals, a seat, andthat the pedals are used to turn the wheels while one is seated and holding on to thehandlebars. However, knowing how to ride a bike would represent procedural knowledge.Individuals know how to ride a bike, but cannot actually describe all of the processesrequired.

The primary sub-process in the declarative stage is declarative encoding of explicitknowledge which can be described as storing experiences—instructions, examples, rules,definitions, and successes and failures of our own attempts. Declarative knowledge is com-mitted to long-term memory by encoding external events or the action side of a condition-action production pair. Declarative encoding occurs as newly identified explicit knowledgeis first considered by working memory and then may be encoded to declarative long-termmemory in chunks. Once the chunk is in long-term memory, its retrieval is controlled byits level of spreading activation or the ease with which a chunk of knowledge can be recalledfrom memory. A particular chunk’s level of activation is strengthened as the number ofconnections or related chunks increase—spreading activation represents how easily andoften the knowledge is retrieved. As the level of activation increases, the chunks can beretrieved more easily.

When an individual has no applicable production rule instantiations (i.e., procedural-level knowledge), they look to examples from the past to analogize to solve a problem, aprocess referred to as interpretive problem solving. In this second sub-process of the de-clarative stage, Anderson et al. (1997) find that encoding of declarative knowledge intolong-term memory is not necessary to use interpretive problem solving. The individualemploying interpretive problem solving may simply work from declarative knowledge (def-initions, examples, rules, etc.) active within working memory rather than draw upon en-coded knowledge (Anderson et al. 1997).

At the core of KMS typically used by accounting firms is top-down knowledge in-cluding manuals, directories, and newsletters; work processes knowledge consisting ofworking papers, proposals, client correspondence, and other engagement materials; andcustomer related knowledge including customer continuity and history information(O’Leary 1998). While these various components can appear to be fairly complex, in realitythey can largely be summed up as definitions (e.g., terminology and explanations ofterminology), rules (e.g., regulations, standards, corporate policies, and interpretationsthereof), and examples (e.g., stories of how problems have been overcome, memos describ-ing problem resolution in a given context, preferred business practices under certain con-ditions, and summaries of previously researched issues)—the three building blocks fordeclarative or explicit knowledge.



All of these facilities are designed to ease the mental workload and make it easier forthe user to acquire the knowledge necessary to complete the task at hand (Alavi and Leidner2001b). To accomplish this, the KMS should allow the user to have easy access to explicitknowledge stored in the system (e.g., definitions, rules, and examples) that can be appliedto solve the problem. The KMS should also improve the ease by which the user can finda rule and/or example that applies to the current situation and facilitate interpretive problemsolving (Alavi and Leidner 2001b). To a certain degree, the KMS relieves the user of theneed for encoding of explicit knowledge in long-term memory as applicable knowledgecomponents can be readily accessed by the user’s active working memory. Easy access toexplicit knowledge via the KMS also reduces the likelihood the user will draw upon en-coded explicit knowledge in long-term memory as drawing from long-term memory in-creases the effort required of the user, thus increasing mental workload (Alavi and Leidner2001a).

Cognitive load theory suggests that as mental workload decreases, an individual willhave more working memory available for problem solving and will lead to better perform-ance (Sweller 1988; Sweller 1989; Chandler and Sweller 1991; Chandler and Sweller1996; Sweller and Chandler 1991; Sweller et al. 1998; Paas et al. 2003; van Merrienboerand Sweller 2005). Thus, having explicit knowledge readily accessible via a KMS shouldenhance performance. Accordingly, this leads to the first hypothesis:

H1: A user of a KMS providing access to explicit knowledge required for problemsolving will perform better than an individual using traditional reference materials.

The concern that has been raised is that this easy access to explicit knowledge thatnegates the need to encode (and subsequently activate) explicit knowledge in long-termmemory may result in the individual user failing to develop foundational explicit knowledgethat is in turn needed to develop tacit knowledge and expertise (Cole 1998; Powell 1998;Alavi and Leidner 2001b). On the other hand, the easy availability of knowledge compo-nents gives KMS users improved accessibility to a large volume of explicit knowledge (i.e.,definitions, rules, and examples) and increases the individual’s opportunities to encode thisknowledge into long-term memory. The concern is whether this explicit knowledge will beencoded in long-term memory as effort is considered a key influence on the successfulacquisition of knowledge (Hiltz 1986).

Substantial research with other types of KBS indicate that similar high expectations byearly promoters of intelligent systems for accelerated transfer of knowledge from systemto user (Eining and Dorr 1991) did not come to fruition. Rather, explorations of the impacton acquisition of explicit knowledge through explanation provision in intelligent systemsprovide mixed evidence. Bransford et al. (1982), Franks et al. (1982), Stein et al. (1982a),and Stein et al. (1982b) found that the precision of an explanation embedded within anintelligent system positively affects the development of explicit knowledge. Intelligent sys-tem explanations embedded with explicit knowledge have been shown to successfully in-crease long-term memory storage of explicit knowledge (Smedley and Sutton 2004). Al-ternative research indicates that users of intelligent systems may acquire and encode lessexplicit knowledge than users of other reference materials (Brody et al. 2003; Marchant etal. 1991; Murphy 1990). Odom and Dorr (1995) also find evidence indicating that preciseexplanations embedded within an intelligent system with examples actually decreased ac-quisition of explicit knowledge.

Prior research using intelligent systems may or may not be applicable to the class ofKBS falling under the KMS domain. In an intelligent systems environment, most of the



experimental studies have explanations automatically provided at relevant points in thedecision-making process. This provides the explicit knowledge for potential storage in long-term memory and a context in which to help activate the knowledge, but does not requireany effort to be exerted by the user. In these settings, the user is unlikely to store andactivate the explicit knowledge unless they also exhibit behavior consistent with intentionallearning (Anderson 1993; Smedley and Sutton 2007)—behavior that requires both moti-vation and effort (Hiltz 1986).

In a KMS environment, two factors may counter the likelihood of encoding explicitknowledge. First, the volume of information and multiple ways of accessing informationavailable in a KMS could potentially increase mental workload while retrieving the infor-mation—a factor found to impact knowledge acquisition (Rose and Wolfe 2000; Rose2005). However, the searching capability embedded in KMS arguably makes accessibilityeasier rather than more difficult and should ease mental workload. The second factor is theperceived ease of accessibility. If the system does facilitate easy access, a user focused onproblem solving may very likely choose to retrieve available explicit knowledge from theKMS, move this knowledge into active working memory to support interpretive problemsolving, but not feel any motivation to actually encode the knowledge (Alavi and Leidner2001b).

These competing effects imply that knowledge acquisition will differ between users ofa KMS and those that use traditional reference materials, but it is unclear which group islikely to acquire more knowledge. This knowledge-acquisition effect is investigated throughthree research questions related to the different types of knowledge components that areavailable to the decision maker. These research questions are stated in the alternative formas follows:

RQ2: Will there be a difference in definition recall between individuals using a KMSembedded with definitions and individuals accessing definitions through tradi-tional reference materials?

RQ3: Will there be a difference in rule recall between individuals using a KMS em-bedded with rules and individuals accessing rules through traditional referencematerials?

RQ4: Will there be a difference in example recall between individuals using a KMSembedded with examples and individuals accessing examples through traditionalreference materials?

RQ2–4 relate to encoding of explicit knowledge and levels of increase in encodedknowledge. The other dimension of explicit knowledge acquisition that is of interest isthe interpretive problem solving ability. For the same reasons that there are concerns that theKMS may impede encoding of explicit knowledge, interpretive problem solving shouldstrongly increase for users of the KMS. A KMS is designed to make it relatively efficientand easy for a user to retrieve the explicit knowledge components that are needed to solvea problem. Consistent with cognitive load theory (Sweller 1988; Sweller 1989; Chandlerand Sweller 1991; Chandler and Sweller 1996; Sweller and Chandler 1991; Sweller et al.1998; Paas et al. 2003; van Merrienboer and Sweller 2005), this ease of effort allows theuser to easily view the knowledge components as desired while maintaining low levels ofmental workload, but at the same time the ease of access does not necessarily motivate theuser to encode the data. However, once the requisite explicit knowledge has been accessed,the user can focus efforts on honing interpretive problem-solving skills that allow the user



to match available solution examples with similar decision-making tasks faced by the user(Anderson 2000). The easy access to rules and examples should facilitate the user becomingquite adept at using these rules and/or examples to solve similar problems—i.e., interpretiveproblem solving. The hypothesis is stated as:

H5: A user of a KMS embedded with explicit knowledge (i.e., definitions, rules, andexamples) will acquire more interpretive problem-solving abilities than an individ-ual not using a KMS.

As suggested by Anderson’s (1993) ACT-R theory, acquisition of explicit knowledgeis a necessary precursor to the development of tacit knowledge which is fundamental tothe establishment of expertise. Ultimately, the ability to support knowledge acquisitionby the individual user through the use of a KMS is critical to an organization’s overallknowledge-management efforts. Yet, with the questions raised earlier, one should be con-cerned with the direction of the change in knowledge acquisition as predicted in RQ2–4.Failing to encode explicit knowledge at a level equal to or greater than that achieved throughtraditional approaches should be of serious concern to knowledge managers. While theremay be variances between types of knowledge that are encoded more effectively by KMSusers than users of traditional reference materials, the cumulative effect is perhaps ofgreatest concern. Variances in knowledge acquisition will also lead to variances in perform-ance when a KMS is not available. The research question is stated as:

RQ6: Will an individual who acquires knowledge through solving problems with theassistance of a KMS achieve a different level of unassisted problem-solving abil-ity than will an individual who acquires knowledge through solving problemswith the assistance of traditional reference materials?

An increase in problem-solving abilities by users of a KMS would be consistent withthe belief that the rich repository of explicit knowledge components combined with easyaccess facilitates a user in experiencing a greater breadth and depth of knowledge. On theother hand, a decrease in problem-solving abilities by users of a KMS may be indicativeof technology dominance by the KMS where the user allows the system to dominate thedecision process and takes a passive role in problem resolution (Arnold and Sutton 1998),thereby reducing knowledge acquisition by the user.

III. RESEARCH METHODTo simulate an environment in which novice decision-makers would make a business

decision, we needed (1) participants with little or no prior knowledge to make the decision,(2) a simple decision task that entry-level accountants might make, and (3) appropriatereference materials to support that task. We chose a managerial accounting task whereparticipants were asked to make three different, but related, decisions that an entry-levelaccountant might be asked to complete. In a business environment, entry-level accountantsmight refer to previous examples, company manuals, or textbooks for guidance. Alterna-tively, they might refer to the company’s KMS with these items embedded if one wasavailable. Students enrolled in a managerial accounting class were chosen to insure thatthey had not previously acquired the requisite explicit or tacit knowledge to make thedecisions. Practicing professionals would not be feasible as they might already havethe knowledge to complete the task.



In order to study the effect of KMS use, a three-stage experiment was conducted tocompare the acquisition of explicit knowledge components by KMS users (KMS group)to individuals using traditional, paper-based reference materials (traditional group). Theexperiment covered three consecutive 75-minute class periods with each stage correspond-ing to the three class periods. During the first stage of the experiment, participants weregiven an introductory lecture accompanied by a set of lecture notes. The introductory lectureprovided the participants with a base level of explicit knowledge, including definitions,rules, and examples.

During the second stage of the experiment, participants completed a pretest recall in-strument designed to assess the amount of explicit knowledge the participants had encodedprior to treatment. Next, participants completed a series of treatment cases—some partici-pants had access to a KMS, while others had access to traditional paper-based materialsincluding a textbook, handouts, and lecture notes. The treatment cases required the partic-ipants to access explicit knowledge already encoded in long-term memory or availablethrough either the KMS or traditional reference materials. The knowledge needed to com-plete the treatment cases was beyond the scope of the introductory lectures; thus participantshad to access the materials provided by either the KMS or traditional reference materialsto complete the task. By completing the cases and accessing the KMS or traditional ref-erence materials, participants had an opportunity to encode the explicit knowledge intolong-term memory.

During the third stage of the experiment, which was conducted in the subsequent classsession, the participants completed a posttest recall instrument. The time lag between ses-sions allowed us to assess improvements due to long-term knowledge acquisition. Theposttest recall assessed how much explicit knowledge individuals had encoded as a resultof the treatment. In addition, the participants solved a series of posttest cases. The posttestcases provided a measure of the participants’ interpretive problem-solving abilities. Figure2 provides an overview of the three experimental sessions.

Experimental ProcedureThe experiment was designed to examine participant encoding of definitions, rules, and

examples, and to measure interpretive problem-solving abilities. To enable such measure-ments, a decision-making task requiring the use of elementary knowledge of definitions,rules, and examples was used. The decision-making task consisted of three managerialdecisions: (1) special order, (2) sell at split-off or process further, and (3) product/depart-ment elimination.

The experiment consisted of an explicit knowledge recall instrument, three treatmentcases, and two posttest decision-making cases. The recall instrument for explicit knowledgewas developed based on common rules and definitions found in traditional reference ma-terials. In addition, the recall instrument contained common examples that would be usedto demonstrate the various types of explicit knowledge. The recall instrument consisted of30 multiple-choice questions. The first ten questions examined knowledge of definitions.Each question defined a term and called for the selection of the correct term among fiveanswers. The next ten questions assessed knowledge of rules. The questions called for thecompletion of the rule with the appropriate information from a set of five answers. The lastten questions tested recall of examples and stated a complete example, including the answer.Each participant was asked to recall whether he/she had seen the example, not seen theexample, or did not remember.

The recall instrument was administered twice to achieve a pretest-posttest design asshown in Figure 2. The recall instrument was initially administered during the second



FIGURE 2Overview of Experiment

EXPERIMENTALSESSION 1

Introductory LectureLecture Notes

Complete:Demographic QuestionnaireGoal Orientation SurveyConsent Form

EXPERIMENTAL SESSION 2

KMS GroupComplete:Pretest Recall (definitions, rules,examples)Three Treatment Cases (while usingKMS)

Traditional GroupComplete:Pretest Recall (definitions, rules,examples)Three Treatment Cases (while usinga textbook and lecture notes)

EXPERIMENTAL SESSION 3

Complete:Posttest Recall (definitions, rules,examples)Two Posttest Cases (while referringto a set of examples)

experimental session and was again administered after completing the treatment cases. Theposttest recall score less the pretest recall score provided a measure for the encoding ofexplicit knowledge that occurred as a result of the treatment.

The cases were developed based on a cost/managerial accounting test manual(Schoenebeck 2003). The three treatment cases were modified slightly, by including sup-plemental questions relating to the case, in order to incorporate all of the rules and defi-nitions that were tested in the recall instrument. This was done to ensure that participantsneeded to use either the materials that were provided in the KMS or the textbook andlecture notes to complete the treatment cases. Researching within the materials provided tothe groups to complete the cases should improve recall in posttesting. Posttest Case 1 wasidentical to a portion of Treatment Case 1, which contained supplemental questions notincluded in Posttest Case 1. The purpose of Posttest Case 1 was to measure interpretiveproblem-solving abilities and was embedded in both the KMS and class notes. Thus, theparticipants had seen the case at least twice before completing Posttest Case 1. PosttestCase 2 is similar but not identical to Treatment Case 3. The participants had been exposedto the explicit knowledge required to solve this case, but had not worked this particularcase previously. The purpose of the posttest cases was to provide a measure of improvementof interpretive problem solving resulting from the treatment.

The recall instrument, treatment cases, and posttest cases were pilot tested by 32 un-dergraduate managerial accounting students and five PhD students. As a result of the pilottests, minor grammatical changes were made and one of the supplemental questions wasreplaced.



Participants and IncentivesIn order to examine whether acquisition of explicit knowledge by a KMS user would

be greater than the user of traditional reference materials, novice-level users with minimalknowledge of the tasks were needed as participants. While participants needed sufficientbaseline knowledge to have the ability to understand the lecture and notes in experimentalsession 1, they needed to be novice enough to allow for acquisition of new explicit knowl-edge. As noted earlier, prior research has shown that novice users pursue explicit knowledgeexplanations while solving problems (Arnold et al. 2006). As a result, undergraduate man-agerial accounting students were selected to participate in this experiment as they had abase level of knowledge enabling an understanding of the task materials, but did not havethe necessary level of explicit knowledge necessary to successfully complete the experi-mental materials prior to the treatment. The participant pool consisted of 222 businessstudents enrolled in six sections of a sophomore-level managerial accounting course. The188 participants who attended all three class sessions and completed all materials formedthe final sample, with the other 34 participants being dropped for missing one or moresessions.

The experiment was conducted as part of the regular class material and covered threeconsecutive 75-minute class periods. In order to induce participants to attend all three ses-sions and take the task seriously, performance-based and participation-based class bonuspoints were awarded toward their final grade. In experimental session 1, points wereawarded for attending the lecture, completing the demographic questionnaire, and com-pleting the goal-orientation survey. In experimental session 2, participants could earn pointsfor performance on pretest recall and for completing the treatment cases. In experimentalsession 3, participants could earn points for performance on posttest recall and posttestcases. In addition, the material covered during these three class sessions represented ap-proximately 20 percent of the material covered on the course’s final examination, providingadditional participant motivation to attend to the task.

Demographic information on the participants is provided in Table 1. Participants wererandomly assigned to one of the two treatment groups.2 The KMS group consisted of 87participants, while the traditional group consisted of 101 participants; t-tests were conductedand no significant differences were found for any of the demographic data.

Experimental KMSWebCT, an Internet-based course portal, was used to implement the KMS in the ex-

periment for several reasons. First, the WebCT interface is very similar to Lotus Notesdatabases, which are applied as the software architecture in most KMS (O’Leary 2002b).Second, the participants were proficient with WebCT and had used it in the same courseto access materials and grades. Alavi and Leidner (2001b) note the importance of ease ofuse in getting users to actually use a KMS. Third, WebCT provides a robust facility fortracking student activity, enabling analysis of participant interaction. Finally, WebCT al-lowed the course administrator to allow and deny participant access, facilitating controlover the KMS.

The KMS used in the experimental sessions was restricted to the inclusion of explicitknowledge components as this was the aspect of KMS of most interest in this study. Alaviand Leidner (2001b) note that a KMS includes both explicit and tacit knowledge. However,

2 The experiment was administered as part of regular class time so participants in each class were randomlyassigned to report to either the regular classroom or to a computer laboratory. Due to seating limitations in thecomputer laboratory, more participants were assigned to the traditional group, resulting in unequal sample sizes.



TABLE 1Participant Information

Variable KMS Group Traditional Group Combined

Number of Participants 87 101 188GPA 3.27 3.30 3.29Age 20.06 19.56 19.79Hours a Week Using a Computer 9.56 8.73 9.12MajorsAccounting 27.59% 30.69% 29.20%MIS related 4.60% 6.93% 5.85%ClassSophomore 71.26% 75.76% 73.68%Junior 24.14% 21.22% 23.12%Senior 4.60% 2.02% 3.23%Employed 47.13% 49.50% 45.70%Use a KMS at Work 10.34% 6.93% 8.51%Have used a KMS 14.94% 11.88% 13.30%

the explicit knowledge is first provided in the system as it is the most easily definable,then the tacit knowledge which makes sense of the explicit knowledge is added by usersto create a shared-knowledge environment. Users lacking adequate prior encoded explicitknowledge must access and make sense of explicit knowledge before they can effectivelyuse tacit knowledge (Alavi and Leidner 2001b). Further, prior studies of KMS implemen-tations find that novice users’ application of explicit knowledge has largely been successful(e.g., Alavi et al. 2006; Butler 2003; Gonzalez et al. 2005), but that attempts to captureand effectively use tacit knowledge have often failed to gain traction (Alavi et al. 2006;Butler 2003; Gonzalez et al. 2005). Both Butler (2003) and Alavi et al. (2006) found intheir studies of organizations that without strong interventions, users focused on the explicitknowledge embedded in KMS and relied on routinized solutions to less complex problems.

The KMS in this study focused on provision of explicit knowledge and routinizedsolutions by providing the ability to search through definitions, rules, examples, and tem-plates. The KMS group used these facilities while solving the treatment cases in experi-mental session 2. While solving the cases, the participants used the KMS to search forunknown terms or rules. In addition, the user had the ability to look at examples to facilitateproblem solving. The user also had access to templates that illustrated how to set upthe problem solution. The intent was to create a KMS environment that would contain thetypes of materials that are commonly available to KMS users in a business environment.

The KMS organized definitions, rules, examples, and templates by both the knowledgecategory (definitions, rules, examples, templates) and the decision type (special order, dropa product or department, sell at split-off or process further decision associated with jointproducts). The KMS main menu displayed icons and links to definitions, rules, examples,templates, special order, drop a product, department, or division, and split-off versus processfurther. This allowed the user to navigate to either a specific decision or to a specific typeof knowledge. In addition, the main menu included a search facility. The search facilityprovided the user with all occurrences in the KMS of the search term entered by the user.The user could select a link to any occurrence.

The navigation of the KMS was based on the category selected by the user, and op-erated as follows based on the selection:



● Definition category took the user to a list of terms. These terms, when clicked, linkedto the respective definition.

● Rules category took the user to a list of rules. These rules, when clicked, took theuser to the respective rule. The rules were categorized by decision type.

● Templates category took the user to a list of templates, which were linked to therespective template. The templates were categorized by decision type.

● Examples category took the user to a list of examples categorized by decision type.The examples were linked to the respective example.

Every term in the rules, templates, and examples was hyperlinked to the respectivedefinition in the KMS. Each template contained a link to the respective rule used to generatethe template and vice versa (each rule was linked to the respective template). Each examplecontained links to the respective templates and rules. This interconnectivity allowed theuser to navigate easily from anywhere in the KMS. In addition, from any page withinthe KMS, the user could select any of the main menu options from the navigation bar.

Traditional Reference MaterialsParticipants in the traditional reference materials treatment were allowed access to their

textbook, lecture notes, and handouts. These materials contained all of the same informationas the KMS, but the information was embedded within the materials and required typicalmanual search techniques to find relevant explicit knowledge components. These materialsare representative of the types of materials available in pre-KMS environments where op-erations manuals, workpaper templates, audit guides and checklists, accounting standards(e.g., GAAP) guides, and so forth are all available through printed books, three-ring bindergrouping, and/or .pdf documents with limited search capability. Accordingly, such materialsrequire a more effortful search strategy to identify relevant supporting knowledge compo-nents and increase the mental workload associated with pulling together problem solutions.

Measurement and DesignThe independent variable was treatment group (KMS or traditional) for testing all

hypotheses and research questions. For H1, the dependent variable was the score attainedon Treatment Cases 1 through 3. For the definitions research question (RQ2), the dependentvariable was the posttest with pretest definition recall used as a covariate. For the rulesresearch question (RQ3), the dependent variable was posttest rule recall with pretest rulerecall included as a covariate. For the examples research question (RQ4), the dependentvariable was posttest example recall with pretest example recall included as a covariate.

For H5, which posits that individuals using a KMS will improve interpretive problem-solving skills more than individuals using traditional reference materials, the dependentvariable was the difference in percentage score of Posttest Case 1 and Posttest Case 2.Treatment Case 1, which constituted a measure of initial interpretive problem-solving skills,was used as a covariate. For RQ6, the dependent variables were the score attained onPosttest Cases 1 and 2.

Three covariates were used in the analysis of all six hypotheses/research questions—ability, performance goal orientation, and learning goal orientation. Research indicates abil-ity is an antecedent to knowledge acquisition. Furthermore, ability and knowledge affectperformance (Libby 1995; Libby and Luft 1993). Ability is a critical prerequisite to knowl-edge acquisition and enhances performance; accordingly, it was included as a covariate inall models as participants’ ability may affect recall and performance. Participants’ GPA wasused as a proxy for ability.



TABLE 2Mean Performance Results

KMS GroupMean (Std. Dev.)

Traditional GroupMean (Std. Dev.)

OverallMean (Std. Dev.)

Pretest Scoresa

Definition Recall 7.37 (1.78) 7.68 (1.43) 7.54 (1.61)Rule Recall 5.33 (2.34) 5.97 (2.21) 5.68 (2.29)Example Recall 5.86 (1.99) 6.15 (2.38) 6.02 (2.21)

Posttest ScoresDefinition Recall 8.24 (1.49) 8.40 (1.23) 8.32 (1.35)Rule Recall 6.31 (2.26) 7.45 (1.91) 6.92 (2.15)Example Recall 5.20 (2.19) 5.73 (2.19) 5.48 (2.20)

Treatment Casesb

Case 1 87.4% (33.4%) 72.8% (44.5%) 79.5% (40.3%)Case 2 61.3% (36.0%) 47.5% (36.4%) 53.9% (36.8%)Case 3 78.7% (34.6%) 65.3% (42.9%) 71.5% (37.7%)

Posttest CasesCase 1 89.7% (29.7%) 93.1% (24.5%) 91.5% (27.0%)Case 2 85.1% (27.6%) 93.1% (20.0%) 89.4% (24.1%)

a Maximum score for each pretest and posttest recall is 10.b Treatment cases and posttest cases are measured on a scale of 0 to 100 percent.

Participants’ orientation toward goals could affect the level of recall and level of per-formance. Anderson (2000) refers to this as ‘‘attention to learning.’’ If an individual is notoriented toward learning, they are less likely to encode knowledge even when encounteredduring problem resolution. Smedley and Sutton (2007) differentiate ‘‘intentional learners’’from other participants in their study of procedural knowledge acquisition from use of aKBS. They found that ‘‘intentional learners’’ showed significantly greater knowledge ac-quisition and responded better to system explanations that presented knowledge compo-nents. In this study, we control for such effects by monitoring participants’ goal orientationacross two potentially conflicting goals—performance goal orientation and learning goalorientation. Performance goal orientation leads an individual to strive for high performanceor avoid low performance, while learning goal orientation leads an individual to understandsomething new or increase level of performance in a given activity (Button et al. 1996).Participants exhibiting performance goal orientation are likely to attempt to achieve highperformance and outperform participants not exhibiting performance goal orientation. Par-ticipants exhibiting learning goal orientation strive to acquire knowledge and may acquiremore knowledge than individuals not exhibiting learning goal orientation. Either type ofgoal orientation could affect performance and knowledge acquisition and is accordinglycontrolled in our analyses of participants’ task-related responses. Both types of goal ori-entation were included as covariates in all models and were measured using the workpreference inventory scale developed and validated by Button et al. (1996).

IV. RESULTSData collected during the three experimental sessions provide the basis for testing the

six hypotheses/research questions. Table 2 provides the mean results of the performancefor pretest recall, posttest recall, treatment cases, and posttest cases. The data indicate anotable difference in performance on the treatment cases between the two groups; the KMS



TABLE 3Results of t-Tests of Recall Differences

Group Pretest Score Posttest Score t-StatisticSignificance

Level

OverallDefinition Recall 7.54 8.32 7.106 0.000Rule Recall 5.68 6.92 8.880 0.000Example Recall 6.02 5.48 �3.903 0.000KMSDefinition Recall 7.37 8.24 4.902 0.000Rule Recall 5.33 6.31 4.625 0.000Example Recall 5.86 5.20 �3.140 0.002TraditionalDefinition Recall 7.68 8.40 5.161 0.000Rule Recall 5.97 7.45 7.980 0.000Example Recall 6.15 5.73 �2.364 0.020

group performed consistently better than the traditional group when the KMS was available,but consistently poorer on the posttest cases when it was not available. (The table alsoshows that the KMS group scored a greater difference between the pretest and posttestdefinition recall (increase of .87) than the traditional group (increase of .72), meaning theyacquired more definitional knowledge than the traditional group. On the other hand, thetraditional group had a greater increase from the pretest to the posttest scores for rule recall(increase of 1.48) than the KMS group (increase of .98) indicating that the traditional groupacquired more rule-type knowledge than the KMS group. Interestingly, example recall se-verely degraded for both groups as exhibited by decreases in scores from the pretest to theposttest.

To determine whether participants acquired explicit knowledge as a result of the treat-ment, t-tests were performed on differences between the posttest and pretest recall scoresfor definitions, rules, and examples for the KMS group, traditional group, and overall. Asindicated in Table 3, all t-tests are significant. A t-test of all participants indicates knowledgeacquisition occurred for definitions (p � .001) and rules (p � .001), but example recalldegraded (p � .001). Separate analysis of the KMS group shows significantly improveddefinition recall (p � .001) and rule recall (p � .001), but degraded example recall (p� .002); while separate analysis of the traditional group followed the same pattern withsignificantly improved definition recall (p � .001) and rule recall (p � .001), and worseexample recall (p � .020).

KMS Performance EffectsHypothesis 1 focuses on the performance effects of using the experimental KMS to

solve problems, hypothesizing that users of a KMS will have a higher level of performancethan those who have access to traditional reference materials. Hypothesis 1 is first testedusing a MANCOVA with the scores on Treatment Cases 1, 2, and 3 as the dependentvariables, group as the independent variable, and GPA, performance goal orientation, andlearning goal orientation as the covariates. The MANCOVA results indicate that there wasa significant difference between groups (p � .004). As noted previously, the mean valuesreported in Table 2 show that on average the KMS group outperformed the traditional groupfor all three treatment cases. Separate ANCOVAs are performed for each of the three



TABLE 4Results for Performance during Treatment Cases (H1)

Panel A: MANCOVA Results

Source of Variance F-value (Wilk’s criterion) p-value

Independent Variable:Group 4.674 .004

Covariates:GPA 8.452 .000Performance Goal Orientation 0.309 .819Learning Goal Orientation 1.731 .162

Panel B: ANCOVA Results

Source of Variance Type III SS df F-value p-value

Treatment Case 1Model 2.034 4 3.285 .013Independent Variable:

Group 1.002 1 6.474 .012Covariates:

GPA 0.169 1 1.092 .297Performance Goal Orientation 0.016 1 .101 .751Learning Goal Orientation 0.767 1 4.954 .027Error 28.332 183



GPA 2.659 1 22.612 .000Performance Goal Orientation 0.067 1 0.573 .450Learning Goal Orientation 0.003 1 0.027 .870Error 21.518 183



GPA 1.292 1 8.637 .004Performance Goal Orientation 0.000 1 0.002 .969Learning Goal Orientation 0.008 1 0.055 .815Error 27.374 183

treatment cases in order to isolate the specific sources of effects captured within theMANCOVA analysis. As shown in Table 4, each of the treatment cases results in a signif-icant difference in performance between the KMS and traditional groups (Treatment Case1, p � .012, Treatment Case 2, p � .003, and Treatment Case 3, Panel C, p � .014). Theresults consistently support the expected relationship as tested through H1.

Encoding of Explicit KnowledgeThe purpose of the research questions related to encoding of explicit knowledge is to

assess the participants’ ability to recall explicit knowledge components. The explicit knowl-edge research questions, RQ2, RQ3, and RQ4 measure the participants’ recall of definitions,



rules, and examples, respectively. The test of RQ2 examines whether individuals using aKMS embedded with explicit knowledge components exhibit a difference in definition recallthan individuals accessing traditional reference materials. To test for the significance of thedifference in definition recall between the KMS and traditional groups, an ANCOVA isused with posttest definition recall as the dependent variable, group as the independentvariable, and pretest definition recall, GPA, learning goal orientation, and performance goalorientation as the covariates. Pretest definition recall is used as a covariate to control forthe knowledge that participants had before completing the treatment.3 Even though theaverage improvement in definition recall was better for the KMS group than the traditionalgroup (.87 compared to .73), the results of the ANCOVA shown in Table 5, Panel A, donot show a significant difference in definition recall between the two groups (p � .942).

The test of RQ3 examines whether individuals using a KMS embedded with explicitknowledge components exhibit a difference in rule recall than individuals accessing tradi-tional reference materials. To test for differences in rule recall between the two groups, anANCOVA is used with the posttest rule recall as the dependent variable, group as theindependent variable, and pretest rule recall, GPA, performance goal orientation, and learn-ing goal orientation as covariates. The traditional group had greater improvement in rulerecall than the KMS group (1.475 compared to .98), which is significant (p � .002) sup-porting RQ3 (see Table 5, Panel B).

The test of RQ4 examines whether individuals using a KMS embedded with explicitknowledge components will exhibit a difference in example recall than individuals accessingtraditional reference materials. To test for differences in example recall, an ANCOVA wasused with posttest example recall as the dependent variable, group as the independentvariable, and pretest example recall, GPA, performance goal orientation, and learning goalorientation as covariates. While the ability to recall examples actually declined in bothgroups, the KMS group had a greater mean decrease in example recall than the tradi-tional group (�.67 versus �.42), the differences between the groups are not statisticallysignificant (p � .134), as shown in Table 5, Panel C. Thus, RQ4 is not supported.

Interpretive Problem SolvingHypothesis 5 focuses on the interpretive problem-solving aspect of explicit knowledge

acquisition, hypothesizing that users of a KMS will improve their interpretive problem-solving skills substantially more than individuals who do not have access to a KMS. Toisolate participants utilizing interpretive problem solving skills, only those with a PosttestCase 1 score greater than or equal to Posttest Case 2 score were included in the analysis.Posttest Case 1 was identical to Treatment Case 1, a case embedded in the KMS, and acase in the class notes. According to Anderson et al. (1997), ‘‘If participants were perform-ing better for those original examples, then it would be evidence that they were solvingthese problems by means of retrieving the study example’’ and utilizing interpretive problemsolving (Anderson et al. 1997, 934–935). Better performance on Posttest Case 2 indicatesthat participants had developed tacit knowledge and were not using explicit knowledge. Asa result, 15 participants that performed better on Posttest Case 2 were removed from thesample to examine H5, which is specifically focused on a type of explicit knowledgeacquisition.

3 Using posttest score as the dependent variable and pretest score as the covariate is a better measure of changethan the difference score between posttest and pretest. Using the difference ignores the relative value of thechange that occurred as a result of the treatment.



TABLE 5Results of ANCOVA for Recall (RQ2–4)


Panel A: Dependent Variable: Posttest Definition Recall

Model 92.590 5 13.448 .000Independent Variable:Group .007 1 0.005 .942Covariates:Pretest Definition Recall 71.218 1 51.719 .000GPA 6.752 1 4.903 .028Performance Goal Orientation 4.984 1 3.619 .059Learning Goal Orientation .242 1 .176 .675Error 250.618 182

Panel B: Dependent Variable: Posttest Rule Recall

Model 375.419 5 28.096 .000Independent Variable:Group 25.930 1 9.703 .002Covariates:Pretest Rule Recall 251.989 1 94.292 .000GPA 1.661 1 .621 .432Performance Goal Orientation 5.014 1 1.876 .172Learning Goal Orientation 4.448 1 1.665 .199Error 486.384 182

Panel C: Dependent Variable: Posttest Example Recall

Model 382.368 5 26.532 .000Independent Variable:Group 6.544 1 2.270 .134Covariates:Pretest Example Recall 364.319 1 126.397 .000GPA 3.307 1 1.147 .286Performance Goal Orientation .045 1 .016 .901Learning Goal Orientation .523 1 .181 .671Error 524.584 182

Hypothesis 5 is tested by examining the difference in interpretive problem solvingbetween the KMS and traditional groups when completing Posttest Case 1. The meanperformance increase for the KMS group was .05 and for the traditional group was .00,indicating that on average the KMS group demonstrated greater interpretive problem-solving ability than the traditional group. An ANCOVA was performed with interpretiveproblem solving (Posttest Case 1 score less Posttest Case 2 score) as the dependent variable,group as the independent variable, and GPA, performance goal orientation, learning goalorientation, and Treatment Case 1 as covariates. Treatment Case 1 is a covariate in orderto isolate the improvement in the participant’s interpretive problem-solving skills. SinceTreatment Case 1 was identical to Posttest Case 1, Treatment Case 1 functions as the initiallevel of interpretive problem-solving skills. As shown in Table 6, a significant differencein improvement of interpretive problem-solving skills is found between the groups (p� .048), providing evidence supporting the hypothesized relationship (H5).



TABLE 6Results of ANCOVA for Interpretive Problem Solving (H5)—Dependent Variable:

Interpretive Problem Solving Score


Model .674 5 2.870 .016Independent Variable:Group 0.186 1 3.964 .048Covariates:GPA 0.098 1 2.094 .150Performance Goal Orientation 0.081 1 1.729 .190Learning Goal Orientation 0.014 1 0.289 .591Treatment Case 1 Score 0.311 1 6.615 .011Error 7.846 167

Knowledge Acquisition from a KMS, Performing Without ItResearch Question 6 focuses on concerns over the potential impact on future perform-

ance capabilities of individuals not having access to a KMS after solving problems onlythrough access and use of a KMS. Acquisition of explicit knowledge is a critical step inthe development of expertise and the impact of KMS usage on individuals’ acquisition ofexplicit knowledge and problem-solving skill development within a critical decision domainhas major implications to KMS designers and implementers. As noted earlier, the meanperformance values reported in Table 2 show that on average the traditional group outper-formed the KMS group on both posttest cases.

A MANCOVA was first used to test for the overall differences between the two groups.The MANCOVA included Posttest Case 1 and Posttest Case 2 scores as the dependentvariables, group as the independent variable, and GPA, performance goal orientation, andlearning goal orientation as the two covariates. The MANCOVA results indicates that groupis marginally significant (p � .062) indicating that the traditional group outperformed theKMS group.

The results from the posttest cases were further examined using separate ANCOVAs.As shown in Table 7, the difference between the two groups was not significant for PosttestCase 1 (p � .373), but was significant for Posttest Case 2 (p � .022) thus the overall resultsfrom the MANCOVA was driven by the differences in Posttest Case 2. In aggregate theresults provide partial support for RQ6, indicating that use of the KMS may have a dele-terious effect on the development of problem-solving skills in absentia of a KMS. The lackof significance in the Posttest Case 1 analysis may also be influenced by the dependenceon interpretive problem solving which the KMS group showed a stronger ability to apply.

V. DISCUSSION AND CONCLUSIONAnecdotal information has suggested that KMS can effectively improve decision mak-

ing by relatively novice users. Additionally, KMS may impact the knowledge acquisitionof the user—although there are conflicting theoretical views on whether this is likely to bea positive or a negative impact. Given the widespread adoption of KMS in professionalenvironments, research examining the impacts of KMS adoption on user performance andexpertise development is imperative to fully understand the consequences of KMS use. Thestudy reported in this paper has provided initial evidence on the impact of a KMS on userperformance and acquisition of explicit knowledge.



TABLE 7Analysis of Posttest Cases (RQ6)

Panel A: MANCOVA Results

Source of VarianceF-value (Wilk’s

criterion) p-value

Independent Variable:Group 2.827 .062

Covariates:GPA 5.999 .003Performance Goal Orientation 2.282 .105Learning Goal Orientation 2.583 .078

Panel B: ANCOVA Results


Posttest Case 1Model 1.220 4 4.494 .002Independent Variable:

Group .054 1 .796 .373Covariates:

GPA .763 1 11.250 .001Performance Goal Orientation .271 1 3.997 .047Learning Goal Orientation .251 1 3.697 .056Error 12.418 183

Posttest Case 2Model .543 4 2.406 .051Independent Variable:

Group .303 1 5.364 .022Covariates:

GPA .106 1 1.885 .171Performance Goal Orientation .062 1 1.098 .296Learning Goal Orientation .049 1 .877 .350Error 10.329 183

The results indicate that there are both positive and negative consequences associatedwith the use of KMS by novice decision makers. From a performance standpoint, KMSusers were found to perform better than users of traditional reference materials in solvingstructured problems. However, the users of traditional reference materials were found to bemore proficient at the task when supporting materials were subsequently not available. Froma knowledge-acquisition standpoint, the implications are less clear. As expected, the readyavailability of examples did significantly improve KMS users’ interpretive problem-solvingskills in comparison to the level of skill acquired from the use of traditional referencematerials. However, the impact of KMS use on the other component of explicit knowledgeacquisition, encoding of explicit knowledge, was less clear. Use of traditional referencematerials yielded significantly greater encoding of decision rules, but no significant differ-ences in encoding of knowledge related to definitions. The encoding of knowledge relatedto examples was most concerning in that both users of a KMS and users of traditionalreference materials actually experienced decreases in encoding.

The increases in KMS users’ performance (i.e., when the KMS is available) helpsvalidate the anecdotal data that has been reported in the business press and the speculation



of academics working in the research domain. The poorer performance when the KMS isnot available, however, provides evidence supporting the de-skilling effects that have beentheorized from a technology dominance standpoint (Arnold and Sutton 1998; O’Leary2002a). These combined results present a dilemma to professional organizations usingKMS. In the short term, the use of a KMS appears beneficial, but in the long term, thereare concerns about the development of domain expertise by KMS users.

The results from a knowledge-acquisition perspective raise warnings that will neces-sitate additional research to better understand the overall impact of KMS use. First, theresults indicate that KMS users show a greater improvement in interpretive problem-solvingskills. KMS users should continue to enhance their interpretive problem-solving skills asthey continue to use the KMS—interpretive problem solving is well supported by a KMSand reduces the cognitive effort required to solve problems. However, Anderson (1993)theorizes that when interpretive problem solving requires less cognitive effort, even usersthat have the necessary procedural-level knowledge will regress to using interpretive prob-lem solving for problem resolution. Projecting from this behavior, novices with easy accessto examples to support decision making may continue to use interpretive problem solvingfor problem resolution and not necessarily be motivated to begin developing higher-leveltacit knowledge.

On the other hand, individuals using traditional reference materials exhibited betterencoding of rules. The individuals may be progressing to the procedural (tacit) stage ofknowledge acquisition and developing production rules. Anderson and Fincham (1994)argue and find that procedural knowledge can be developed by solving as little as oneproblem. It is quite plausible that the users of traditional reference materials may haveincurred greater cognitive effort during the treatment stage and began to develop tacitknowledge while solving the treatment cases. As a result, the associated rule recall improvedby using developed production rules (a component of tacit knowledge). This phenom-enon should be investigated further in future research examining the linkages betweenexplicit knowledge and tacit knowledge acquisition in KMS-supported decision-makingenvironments.

The degrading of knowledge encoding of examples also raises concerns. The mostlikely driver of this effect is information overload. When the information load exceeds theinformation-processing capacity, information overload occurs (Schick et al. 1990). Schroderet al. (1967) describe the limitations of human information processing by modeling theassociation between information load and information-processing complexity. The modelsuggests that individuals encountering information overload will respond by increasinginformation-processing complexity from low-level (concrete) to high-level (abstract). Morecues are identified, finer distinctions are drawn from them, and the cues are integrated inan increasing number of methods. This process continues until the cognitive structure is soinundated with integration of cues that the facilities formerly used to identify cues arecompensating for the integration; and hence, the identification of cues declines (Schroderet al. 1967). This phenomenon may have caused the decline in example recall as participantsare provided with a multitude of examples.

As with any study, there are limitations to the research that should be considered whenmaking inferences about the results. There is a risk that some participants may have studiedthe material outside of the experimental sessions. While the lecture notes were collectedat the end of each session and access to the KMS was deactivated, any of the participantsmay have studied the textbook. Additionally, although the participants were instructed notto talk to one another about the experiment, whether they did discuss the experiment withone another outside of the experimental sessions is indeterminable. Hence, complete control



was not available, but there is no reason to believe that this would impact one treatmentgroup more than the other.

An additional limitation is use of student participants completing a fairly simple de-cision task. While use of students was desirable for control purposes in this study with thefocus on acquisition of explicit knowledge, students may not be completely representativeof professional behavior. On the other hand, it would be exceedingly difficult to obtainprofessional participants lacking baseline explicit knowledge needed to perform their jobs.Future research should move beyond explicit knowledge to investigate the impact of KMSon the development of tacit knowledge, as this study provides initial evidence that tacitknowledge acquisition may be hindered when individuals use KMS. In addition, researchshould investigate long-term effects of KMS such as de-skilling and reduction of firmexpertise as suggested by prior research (Arnold and Sutton 1998; O’Leary 2002a; Maschaand Smedley 2007; Dowling and Moroney 2008).

While the research reported in this study examines whether the use of a KMS impactsexplicit knowledge acquisition, future research should examine why that difference occurs.For instance, by utilizing an enhanced design such as eye-tracking software that compareshandouts on screen to the KMS using eye-tracking software, we could gain insight intohow the use of a KMS impacts knowledge acquisition.

REFERENCESAlavi, M. 1997. KPMG Peat Marwick U.S.: One Giant Brain. (Case 9-379-108) Boston, MA: Harvard

Business Publishing.———, and D. L. Leidner. 1999. KMS: Issues, challenges, and benefits. Communications of the

Association for Information Systems (February): 1–37.———, and ———. 2001a. Research commentary: Technology-mediated learning—A call for greater

depth and breadth of research. Information Systems Research 12 (1): 1–10.———, and ———. 2001b. Review: Knowledge management and knowledge management systems:

Conceptual foundations and research issues. MIS Quarterly 25 (1): 107–137.———, T. R. Kayworth, and D. L. Leidner. 2006. An empirical examination of the influence of

organizational culture on knowledge management practices. Journal of Management Informa-tion Systems 22 (3): 191–224.

Anderson, J. R. 1987. Skill acquisition: Compilation of weak-method problem solutions. Psycholog-ical Review 94 (2): 192–210.

———. 1990. The Adaptive Character of Thought. Hillsdale, NJ: Lawrence Erlbaum Associates.———. 1993. Rules of the Mind. Hillsdale, NJ: Erlbaum.———, and J. M. Fincham. 1994. Acquisition of procedural skills from examples. Journal of Ex-

perimental Psychology: Learning, Memory, and Cognition 20 (6): 1322–1340.———, ———, and S. Douglass. 1997. The role of examples and rules in the acquisition of a

cognitive skill. Journal of Experimental Psychology: Learning, Memory, and Cognition 23 (4):932–945.

———. 2000. Learning and Memory. New York, NY: John Wiley and Sons, Inc.Arnold, V., and S. G. Sutton. 1998. The theory of technology dominance: Understanding the impact

of intelligent decision aids on decision makers’ judgments. Advances in Accounting BehavioralResearch 1: 175–194.

———, N. Clark, P. A. Collier, S. A. Leech, and S. G. Sutton. 2006. The differential use and effectof knowledge-based system explanations in novice and expert judgment decisions. MIS Quar-terly 30 (1): 79–97.

Baird, L., J. C. Henderson, and S. Watts. 1997. Learning from action: An analysis of the center forarmy lessons learned. Human Resource Management 36 (4): 385–396.



Bartlett, C. 1996. McKinsey & Company: Managing Knowledge and Learning. (Case 9-396-357)Boston, MA: Harvard Business Publishing.

Bransford, J. D., B. S. Stein, N. J. Vye, J. J. Franks, P. M Auble, K. J. Mezynski, and G. A. Perfetto.1982. Different approaches to learning: An overview. Journal of Experimental Psychology:General (4): 390–398.

Brody, R. G., T. K. Kowalczyk, and J. M. Coulter. 2003. The effect of a computerized decision aidon the development of knowledge. Journal of Business and Psychology (Winter): 157–174.

Butler, T. 2003. From data to knowledge and back again: Understanding the limitations of KMS.Knowledge and Process Management 10 (3): 144–155.

Button, S. B., J. E. Mathieu, and D. M. Zajac. 1996. Learning goal and performance goal orientation.Organizational Behavior and Human Decision Processes (July): 26–48.

Cameron, P. 2000. Managing the wealth. CMA Management 74 (9): 46–49.Chandler, P., and J. Sweller. 1991. Cognitive load theory and the format of instruction. Cognition and

Instruction 8 (4): 293–332.———, and ———. 1996. Cognitive load while learning to use a computer program. Applied Cog-

nitive Psychology 10 (2): 151–170.Chi, M. T. H., M. Bassok, M. W. Lewis, P. Reimann, and R. Glaser. 1989. Self-explanations: How

students study and use examples in learning to solve problems. Cognitive Science 13 (2): 145–182.

Cole, R. E. 1998. Introduction. California Management Review 40 (3): 15–21.Davenport, T. H., D. W. DeLong, and M. C. Beers. 1998. Successful knowledge management projects.

Sloan Management Review 39 (2): 43–57.Dilnutt, R. 2002. Knowledge management in practice: Three contemporary case studies. International

Journal of Accounting Information Systems 3 (2): 75–82.Dowling, C., and R. Moroney. 2008. (Forthcoming.) Auditor performance variation: Impact of sub-

specialty knowledge differences between industry specialists. Advances in Accounting Behav-ioral Research 11.

Duchessi, P., and R. M. O’Keefe. 1992. Contrasting successful and unsuccessful expert systems.European Journal of Operations Research 61: 122–134.

Earl, M. 2001. Knowledge management strategies: Toward a taxonomy. Journal of ManagementInformation Systems 18 (1): 215–234.

Eining, M., and P. Dorr. 1991. The impact of expert system usage on experiential learning in anauditing setting. Journal of Information Systems 5 (1): 1–16.

Franks, J. J., N. J. Vye, P. M. Auble, K. J. Mezynski, G. A. Perfetto, J. D. Bransford, B. S. Stein,and J. Littlefield. 1982. Learning from explicit versus implicit texts. Journal of ExperimentalPsychology: General 4: 414–422.

Gonzalez, L. M., R. E. Giachetti, and G. Ramirez. 2005. Knowledge management-centric help desk:Specification and performance evaluation. Decision Support Systems 40: 389–405.

Grant, R. M. 1996. Prospering in dynamically competitive environments: Organizational capability asknowledge integration. Organization Science 7 (4): 375–387.

Grover, V., and T. H. Davenport. 2001. General perspectives on knowledge management: Fostering aresearch agenda. Journal of Management Information Systems 18 (1): 5–23.

Henderson, J. C., S. W. Sussman, and J. Thomas. 1998. Creating and exploiting knowledge for fast-cycle organizational response. Advances in Applied Business Strategy 5: 103–128.

Herz, P. J., and J. J. Schultz. 1999. The role of procedural and declarative knowledge in performingaccounting tasks. Behavioral Research in Accounting 11: 1–27.

Hiltz, S. R. 1986. The virtual classroom: Using computer-mediated communication for universityteaching. Journal of Communication 36 (2): 95–104.

Leech, S. A., and S. G. Sutton. 2002. Knowledge management issues in practice: Opportunities forresearch. International Journal of Accounting Information Systems 3 (2): 69–74.

Libby, R., and J. Luft. 1993. Determinants of judgment performance in accounting settings: Ability,knowledge, motivation, and environment. Accounting, Organizations and Society 18 (5): 425–450.



———. 1995. The role of knowledge and memory in audit judgment. In Judgment and DecisionMaking Research in Accounting and Auditing, edited by R. H. Ashton, and H. Ashton, 176–206. Cambridge, U.K.: Cambridge University Press.

Marchant, G., J. Robinson, U. Anderson, and M. Schadewald. 1991. Analogical transfer and expertisein legal reasoning. Organizational Behavior and Human Decision Processes 480 (2): 272–290.

Mascha, M. F., and G. Smedley. 2007. Can computerized decision aids do ‘‘damage’’? A case fortailoring feedback and task complexity based on task experience. International Journal of Ac-counting Information Systems 8 (2): 73–91.

Murphy, D. S. 1990. Expert system use and the development of expertise in auditing: A preliminaryinvestigation. Journal of Information Systems 4 (3): 18–35.

Nagle, C. 1999. Research opportunities in knowledge management. Presented at the Mid-Year Meetingof the Auditing Section of the American Accounting Association (January 14).

Nonaka, I. 1994. A dynamic theory of organizational knowledge creation. Organization Science 5(1): 14–37.

Odom, M. D., and P. B. Dorr. 1995. The impact of elaboration-based expert system interfaces on de-skilling: An epistemological issue. Journal of Information Systems 9 (1): 1–18.

———. 1998. Enterprise knowledge management. Computer (March): 54–61.———. 2002a. Knowledge management in accounting and professional Services. In Researching

Accounting as an Information Systems Discipline, edited by V. Arnold, and S. G. Sutton, 273–283. Sarasota, FL: American Accounting Association.

———. 2002b. Knowledge management across the enterprise resource planning systems life cycle.International Journal of Accounting Information Systems 3 (2): 99–110.

Paas, F., A. Renkl, and J. Sweller. 2003. Cognitive load theory and instructional design: Recentdevelopments. Educational Psychologist 38 (1): 1–4.

Powell, W. 1998. Learning from collaboration: Knowledge and networks in the biotechnology andpharmaceutical industries. California Management Review 40 (3): 228–240.

Roberts, M. L., and R. H. Ashton. 2003. Using declarative knowledge to improve information searchperformance. Journal of the American Taxation Association 25 (1): 21–38.

Rose, J. M., and C. J. Wolfe. 2000. The effects of system design alternatives on the acquisition oftax knowledge from a computerized tax decision aid. Accounting, Organizations and Society25: 285–306.

———. 2005. Decision aids and experiential learning. Behavioral Research in Accounting 17: 175–189.

Sambamurthy, V., and M. Subramani. 2005. Special issue on information technologies and knowledgemanagement. MIS Quarterly 29 (1): 1–7.

Schick, A. G., L. A. Gordon, and S. Haka. 1990. Information overload: A temporal approach.Accounting, Organizations and Society 15 (3): 199–220.

Schoenebeck, K. 2003. Test Item File: Cost Accounting: A Managerial Emphasis, 11th Edition. UpperSaddle River, NJ: Pearson Education.

Schroder, H. M., M. J. Driver, and S. Streufert. 1967. Human Information Processing. New York,NY: Holt, Rinehart & Winston.

Smedley, G. A., and S. G. Sutton. 2004. Explanation provision in knowledge-based systems: A theory-driven approach for knowledge transfer designs. Journal of Emerging Technologies in Account-ing 1: 41–61.

———, and ———. 2007. The effect of alternative procedural explanation types on proceduralknowledge acquisition during knowledge-based systems use. Journal of Information Systems 21(1): 27–51.

Stein, B. S., J. D. Bransford, J. J. Franks, R. A. Owings, N. J. Vye, and W. McGraw. 1982a. Differ-ences in the precision of self-generated elaborations. Journal of Experimental Psychology: Gen-eral 4: 399–405.

———, ———, ———, N. J. Vye, and G. A. Perfetto. 1982b. Differences in judgments of learningdifficulty. Journal of Experimental Psychology: General 4: 406–413.



Sweller, J. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science 12:257–285.

———. 1989. Cognitive technology: Some procedures for facilitating learning and problem solvingin mathematics and science. Journal of Educational Psychology 81 (4): 457–466.

———, and P. Chandler. 1991. Evidence for cognitive load theory. Cognition and Instruction 8: 351–362.

———, J. van Merrienboer, and F. Paas. 1998. Cognitive architecture and instructional design. Ed-ucational Psychology Review 10 (3): 251–296.

Thomas, J. B., S. A. Watts, and J. C. Henderson. 2001. Understanding ‘‘strategic learning’’: Linkingorganizational learning, knowledge management, and sensemaking. Organizational Science 12(3): 331–345.

van Merrienboer, J., and J. Sweller. 2005. Cognitive load theory and complex learning: Recent de-velopments and future directions. Educational Psychology Review 17 (2): 147–177.

Wickramasinghe, N., and G. L. Mills. 2002. Integrating e-commerce and knowledge management—What does the Kaiser experience really tell us. International Journal of Accounting InformationSystems 3 (2): 83–98.

use of knowledge management systems and the...

Documents