Instructional Science26: 263–280, 1998. 263c 1998Kluwer Academic Publishers. Printed in the Netherlands.

Authoring instructional expertise in knowledge based tutors�

TOM MURRAYCenter for Knowledge Communication, University of Massachusetts, Amherst, MA, USA,Email: tmurray@cs.umass.edu, www.cs.umass.edu/�tmurray

Abstract. Our contribution to the Special Issue on the GTE system begins with a responseto five major claims made in Van Marcke’s paper on GTE. We follow with a descriptionof a system called Eon, a suite of authoring tools for intelligent tutoring systems (ITS),which we are developing in our lab. Next we discuss several general issues in ITS authoringsystems as they pertain to GTE, Eon, and other systems, including who the intended audienceis, where instructional expertise comes from, tradeoffs between generality and inferencingpower, managing complexity for users, and the use of “knowledge types.” Finally, we suggestseveral areas for synergy and future work for GTE and Eon.

Key words: intelligent tutoring systems, knowledge acquisition, teaching strategies

Introduction

The question of how to represent instructional expertise has been under-explored in the 20-year history of intelligent tutoring systems (ITS) research,as most work has focused on representation and inferencing in the areasof domain models and student models, or on building engaging learningenvironments. Van Marcke’s work on the GTE system represents a majorcontribution to this important area. Few since Collins & Stevens (1982), intheir early work on Socratic and inquiry teaching, have attempted to modelcomplex instructional methods to such depth. Van Marcke describes both aformalism for representing instructional expertise, and a large database ofencoded instructional knowledge.

In this paper I will discuss Van Mark’s paper and its claims, describe howinstructional strategies are represented in the ITS authoring tools I am build-ing (collectively called Eon), compare GTE with Eon and other formalismsthat have been proposed for representing instructional expertise, and finallysuggest future directions for GTE, Eon, and the ITS research community.Specifically, I will address these issues about representing, authoring, and

� This research is based on work supported by the National Science Foundation and theAdvanced Research Projects Agency under Cooperative Agreement No. CDA-940860.

264

using instructional expertise (in relation to the GTE system and the Eonsystem):� The sources of instructional expertise,� Methods for achieving generality,� Identifying the intended author,� Inferencing power and sophistication,� Managing complexity with visualization and debugging tools,� Multiple strategies and meta-strategies, and� Using “knowledge types” to index strategies.

Claims and assumptions in Van Marcke’s paper

In the central paper in this issue Van Marcke argues for the following claimsand assumptions, which I address in turn:1. Instructional strategies and teaching expertise are generic,2. Generic instructional strategies can be gleaned from experienced teachers,3. A particular formalism, embodied in GTE, has been devised which can

successfully represent these generic human instructional strategies in acomputer tutor,

4. A valid knowledge base of generic instructional strategies has been devel-oped, and

5. GTE allows authors to develop courseware without having to encodeinstructional strategies.

1. Instructional strategies and teaching expertise are genericVan Marcke substantiates this claim by demonstrating that a generic instruc-tional knowledge base was built. This argument relies on two assumptions:that the knowledge base indeed reflects good human teaching, and that it wasused in enough contexts to demonstrate true generality, both of which areonly weakly indicated in the article. This first claim is really an underlyingassumption, and, I think, a bit of a straw-man. It is a reasonable and still openassumption which Van Marcke’s work gives additional weight to, and thefield is still waiting for more data points that argue for and against it.

2. Generic instructional strategies can be gleaned from experiencedteachersThe paper says very little about where the instructional knowledge basecomes from or about the knowledge acquisition process used with teachers.Therefore this claim is again more of a plausible working assumption. Gettingteachers to articulate what they do and why they do it is a very difficult butworthwhile task. Van Marcke argues for the representational competence of

265

GTE, but not for the cognitive validity or understandability of the frameworkto non-programmers. Understandability may not be essential however, giventhe goal to allow teachers to plug in instructional content and make use ofpre-defined instructional strategies.

3. A particular formalism, embodied in GTE, has been devised which cansuccessfully represent these generic human instructional strategies in acomputer tutorThe validity and usefulness of the representationalformalismcould be arguedfor in several ways: 1) by demonstrating that it can simulate a varietyof humantutorial interactions, 2) by demonstrating that it can represent a variety ofdocumented instructional strategies that are based on respected theories orempirical data, or 3) by demonstrating that the formalism was successfullyused by a number of instructional designers to design teaching strategies fortutors. It is not apparent how well the instructional knowledge base simulateshuman teaching or how easy the formalism is to use. There are no evaluationscomparing GTE performance with human dialogues. (Van Marcke has someshort dialog comparisons in other papers, but the scale-up is not a given.) Thataside, the formalism described is certainly powerful, general, and seeminglyuseful. However, there have been a number of formalisms proposed for repre-senting planning and discourse behavior in the AI literature, and a few in theITS literature as well, and I would like to see some analysis of how the GTEmethod compares and contrasts with some of these, to allow potential usersto evaluate why or when they should choose GTE over another formalism (asis done in W. Murray (1990)).

4. A valid knowledge base of generic instructional strategies has beendevelopedSimilarly to the previous item, the validity and usefulness of thecontentsofthe instructional knowledge base could be argued for in three ways: 1) theGTE methods were derived from respected or commonly accepted instruc-tional theories or principles; 2) the GTE methods were tested empiricallyor were implementations of empirically tested instructional methods, or 3)the knowledge base was used by a number of instructional designers makingGTE-derivative tutors, and was expressive enough to account for most ofthe strategies they needed. Again, Van Marcke gives only a few examples ofdialogues generated with GTE, and the generality of the knowledge base isan empirical question awaiting more applications in diverse domains. Muchto Van Marcke’s credit, GTE’s instructional knowledge base is extensive andwell organized. This knowledge base, tediously constructed, is a major contri-bution to the field. Even if considered apart from the GTE framework, it canhelp inform the design of other instructional systems.

266

5. GTE allows authors to develop courseware without having to encodeinstructional strategiesThis claim is questionable. Ignoring the already mentioned fact that there areno descriptions of semi-naive users using the system, there are other issues.First, there is no indication that a viable user interface exists for enteringcontent. It is not apparent whether it takes a programmer’s skill to do so.Second, even if content could easily be entered, it is not clear that contentcan be intelligently designed, encoded, tested, and modified, all the whileignoring the control structure that determines how the content is selected andsequenced. Of course, the instructional knowledge base can be tailored andrefined for each application, but this is even more clearly in the realm ofprogramming and knowledge engineering.

Van Marcke’s paper is not an evaluation – it is a detailed description of aparticular system; a particular solution to the difficult problem of representinginstructional expertise. GTE is an ambitious work in progress, and the factthat the above claims have been only weakly substantiated does not detractfrom the contribution Van Marcke has made, but it does indicate the typesof evaluation needed in the future. Later we will look more closely at GTE’srepresentational formalism and knowledge base, the main contributions ofthis work, and compare them with other systems. But first I will describeEon, the authoring tools that our research group has developed.

The Eon ITS authoring tools

Eon is a suite of authoring tools for ITS which are a direct descendant andextension of the KAFITS system referenced in Van Marcke’s article.

Our general approach has these significant differences with Van Marcke’s:1. Our goal is to make ITS design, including the design of teaching strategies,

available to a much wider audience.2. Therefore, our work has focused on building highly usable interactive

tools which allow clear visualization and easy design, testing, and modifi-cation of instructional content and strategies. GTE is more of an ITS shellthan an ITS authoring system.

3. We provide authoring tools for interfaces, student models, content, andinstructional strategies, whereas GTE focuses on instructional strategies,and is working on collaborating with other projects to use their contentrepresentation and interface design tools (I think such collaboration isexemplary, taking advantage of the strengths of several research projects,and hope to do the same with our work in the future.)

4. Though we plan to develop libraries of instructional strategies, content,and user interface components, at present Eon offers a general mechanism

267

for representing instructional strategies, but does not contain specificinstructional strategies, as GTE does. (Tutorials built with Eon containinstructional strategies, but we have not extracted them yet as reusable“plug-in” strategies.)

We will describe all the components of Eon, but focus our discussion andcomparison on the authoring tools and representational formalism that dealswith tutoring strategies. In Eon the strategy representation formalism isdesigned to be used by expert teachers and instructional designers to representtheir own strategies, whereas in GTE the strategies are built by knowledgeengineers with programming skills, and then used by teachers who entercontent material. The overall goals of the two systems are compatible andoverlapping, but the base-lines are different. Eon’s design and usability base-line is highly usable off-the shelf CAI authoring tools such as Authorwareand Icon Author which have demonstrated their usefulness in a wide com-mercial base, and our goal is to evolve this type of authoring environmenttoward authoring “intelligent” tutors. GTE’s base line is an AI expert systemformalism for representing instructional behavior, and the goal is to increasethe depth and flexibility of teaching interactions. The upshot is that Eon ismore usable and accessible, while GTE is more powerful and expressive.

Also, Van Marcke has labored to encode a library of ready-to-use instruc-tional strategies, while we are in the process of designing a conceptual vocab-ulary of primitive tutorial actions and parameters that teachers can combine intheir own way to express a wide variety of strategies. [Footnote: The distinc-tion is not really so clear, since in GTE the given knowledge base can becustomized, and in Eon we expect knowledge engineers to be involved toproduce more complex strategies than teachers could.]

The tutorial design process in Eon involves describing the curriculum orknowledge goals as a network of related topics, creating reusable interactivepresentation screens, filling a data base with the textual and graphical contentwhich will appear in these screens, and authoring the procedures (“strategies”)which tell the tutor how and when to interact with the student. Authoring isopportunistic – it can proceed top down from the topic network, and/or bottomup from the specific content the student will see. Below we describe the majorauthoring tools in Eon, as illustrated in Figure 1. This description contrastsEon with CAI authoring tools. We will compare Eon to GTE later.

Description of the tools

The Interaction Editor (see Figure 1) allows trainers to construct studentscreens, student interactions, simulations, learning environments, and multi-media materials. The “widget palette” shown to the far left allows for thecreation of about 15 widgets (in five categories; the “basic” widget category

268

Figure 1. Interaction Editor (UL), Topic Network Editor (UR), Contents Editor (LL), andStrategy Editor (LR).

is shown in the Figure), including sliders, movies, hot spots, and tables. Whatdistinguishes the screens in Eon from those in CAI authoring tools is that thescreens aretemplates. For instance, the screen shown in the Figure is a generictemplate for multiple choice questions with a picture, a picture description,and a confidence slider. The author creates a number of “Presentation Con-tents” to fill in this template with specific values, i.e. specific text and picturesfor each use of the generic screen.

Additional widgets needed for specific applications can be created byprogrammers and “dropped into” Eon, where they will show up on thewidget pallet, in the “custom widget” category. Custom widgets must follow asimple protocol and specify their widget parameters and the events that theycan recognize. Widgets can be arbitrarily complex, and can be entire learn-ing environments, such as an eco-system simulation widget, or a car enginesimulation widget.

269

As the author builds the interactive screens, a data-base-like template isautomatically created showing the relevant parameters of each widget. ThePresentation Contents Editor(see Figure 1) shows the parameters for eachwidget on the screen, and is used to create and edit Presentation Contents. TheFigure shows the contents for “FlyOnATable,” one of the many Presentationscreated for this template. As can be seen in the Figure, some widgets, suchas graphics and text, have only one parameter (the name of the graphic, orthe text string), while others have several (i.e. the multiple choice widget hasthe answers, the question text, and which answers are correct). The values ofwidget parameters can be bound to scripts or functions, allowing the studentscreens to be generated dynamically as well as containing canned material.

TheTopic Network Editor (see Figure 1) is an interactive graphical toolthat allows authors to design curriculum networks and represent relation-ships between domain concepts. Topics have a Type (such as fact, concept orprocedure), Links to other topics (as shown in the network), Topic Levels (notshown in the network, but described in Murray (1996b)), and Topic Proper-ties. The graphical properties shape, color, pattern, border color, and borderthickness are used to depict various topic properties, such as importance,difficulty, and knowledge type. Links can be of various types, as indicated bythe link color. Objects called Ontologies are used to customize the represen-tational infrastructure of the domain, by specifying, for example, the types oftopics, topic links, and topic properties used in a given tutor.

The Topic Network allows curriculum design at anabstract, pedagogicallyrelevant level, a capability not provided by CAI authoring tools. In contrast,presentation Contents represent theconcretematerial the student will see onthe screen. A tool called the Topic Browser (not shown) is used to link Presen-tation Contents to Topics, e.g. to specify which presentations are needed toteach the topic Gravity.

So far we have described how the knowledge base is constructed fromobjects such as Topics and Presentations, but have not indicated how theauthor specifies how to sequence these objects and react to student behavior.TheStrategy Editor (see Figure 1) utilizes a flow-line paradigm for graph-ically representing procedures, which we call Strategies. Icons representinginteractive screens, branching and looping constructs, variable manipulation,etc., are dragged from the Strategy Icon Pallet (bottom center of the Figure)and dropped onto the Strategy flow line. One thing that distinguishes theseflow lines from similar ones used in CAI authoring tools is that Eon strategiesare fully functional procedures, with input parameters, local variables, andreturned values, and they can be called recursively, resulting in a much morepowerful and expressive the programming paradigm. Another distinguishing

270

factor is that Eon facilitates the representation of alternate teaching strategiesand meta-strategies (described later).

Eon contains several other tools not shown in the Figure, including aTool Launcher which provides easy access and documentation for all of theother tools, and a Student Model Editor, which allows trainers to specifyhow student behavior is used to infer student knowledge states and masterylevels of the Topics. Decisions in the Strategies can thus be predicated on thestudent’s state.

These tools are currently being used to build five tutors. The Statics Tutorteaches introductory Statics concepts, and includes a “crane boom” simula-tion which lets students manipulate configurations of beams and cables andobserve the resulting forces. The Bridging Analogies Tutor incorporates aSocratic teaching method developed and tested by cognitive scientists toremediate common persistent misconceptions in science. The ChemistryWorkbench tutor provides a more open ended learning environment forlearning about solvency and chemical reactions by interactively mixing chem-icals and measuring the results. The Keigo Tutor teaches a part of Japaneselanguage called “honorifics,” dealing with the complicated rules used todetermine verb conjugation appropriately honoring the listener and topic of aconversation. For this tutor Eon is interfaced with a rule based expert system.The Refrigerator Tutor explains the thermodynamic principles underlyingrefrigeration. All of these systems are in prototype stages.

Issues and comparisons

In our discussion of the GTE and Eon research the central issue is: How tomodel instructional expertise at a computational level of precision, and doit with enough depth to do the necessary reasoning, and enough flexibilityto handle a wide range of domains and instructional styles. Implied in thisquestion are a number of tradeoffs and assumptions. In Murray (1996a)we discuss how tradeoffs among scope, depth, learnability, and productivityaffect the design of the four functional components of ITSs: the learningenvironment, the domain model, the teaching model, and the student model.Below we look at a number of design issues for ITS shells, such as the levelsof generality, complexity, and power, and the expected user skill level. Wealso discuss the representation of meta-strategies, the use of knowledge typeformalisms, and merits of the potential sources of instructional expertise.

What are the sources of instructional expertise?

On [pg. 4] Van Marcke says that “existing [instructional] theories are notsufficiently precise and reliable to serve as a basis for rational decision

271

making in [computer based] instruction,” and that the teaching model must“necessarily” reflect the knowledge of experienced teachers. Bloom (1984)demonstrated a two-sigma difference between classroom teaching and one-on-one tutoring, and it would appear that no instructional theory could producethat level of proficiency in a classroom situation. It is also true that instruc-tional theories are too vague to be implemented directly into computer tutors,but this does not mean that they are of no worth to ITS designers.

Over the last several decades there has been a vast expenditure of effortin trying to understand learning and prescribe effective teaching methods,performed by psychologists and instructional theorists. Why should computerscientists think they can do better starting from scratch, or by observingexpert teachers? A fresh approach to the problem, especially one based onempirical information (watching what good teachers do within a knowledgeacquisition framework), is warranted, but instructional theories should notbe causally discarded. There are a number of existing sources for the rulesof instructional expertise, including cognitive theories, instructional designprinciples, empirical educational research, and observing expert teachers.There are reasons for and against using each, based on one’s preference forempirical validation, practicality, or explanatory power, and all sources havesomething to offer.

Our approach attempts to utilize both theory and practical expertise, butdoes not aim for broad instructional competence. The basic Eon shell doesnot contain any specific strategies or theory. We intend to build instructionalmodels that are generic but optimized for particular classed of domains.These would be paired with similarly specialized Ontology objects to producespecial purpose authoring tools, as explained in the next section.

Our goal is not to implement the “best” teaching methods, but to lookfor commonalties and underlying principles among all methods we haveseen. To find a balance between instructional theory and practice, our currentapproach utilizes a broad taxonomy of instructional actions (like GTE tasks)and instructional parameters, and this taxonomy reflects commonalties amonginstructional theories. This taxonomy (Murray, 1996c), like GTE, tries forcompleteness, but it is a paper-based knowledge acquisition tool rather thanpart of an expert system. The taxonomy is a conceptual vocabulary withhundreds of terms which can be used to help organize and focus the designprocess. Practicing teachers, even very good ones, are rarely familiar withthe range of instructional and learning theories, nor can they articulate whatthey do or why they do it. We aim to work with practicing teachers andinstructional theorists to represent their knowledge in terms of the theory-informed representational primitives.

272

How can generality be achieved?

One finds an extremely wide range of instructional methods in the literature.A single formalism can not hope to capture even a moderate coverage ofthese. All representational formalisms are well suited for representing somethings and poor at representing others. GTE deals with this inherent biasby representing instructional tasks independently of the methods used toaccomplish that task, and by allowing designers to add, modify, or refineinstructional methods. Still, the GTE knowledge base seems to aim for widecoverage of instructional competence, but there must be some bias towardcertain underlying assumptions and instructional methods.

Also, even if many of the diverse, often contradictory, instructional methodsavailable could be incorporated into one system, the resulting system wouldbe completely baroque in its entangled complexity.

In contrast to GTE’s goal for broad instructional competence, we take aleast commitment approach with the basic Eon shell. To obtain breadth andgenerality we propose ameta-shellapproach in which the basic authoringtools are used to build special purpose authoring tools. No system can beexpected to work equally well for all domains and teaching styles, and asystem designed for a particular type of domain (such as device diagnosis) orstyle (such as Socratic teaching) has the potential to provide a more power-ful formalism tailored to its needs. For example, Eon could be used to buildspecial purpose authoring tools for ITSs dealing with science concepts, humanservice/customer contact skills, language learning, and equipment mainte-nance. Instructional designers using special purpose authoring tools do nothave to start from scratch, but can immediately start constructing a tutor inan environment that supports and helps clarify and organize the knowledgeacquisition process. They would be provided with default teaching strategies,default student modeling rules, default student interactions, and a topic struc-ture (Ontology) which is tailored to a specific type of domain and/or task, allof which could be used as is, or modified.

The meta-shell approach allows for the proliferation of special-purposeshells with a common underlying structure, so that inter-domain commonal-ties can be exploited in both content creation and in training authors to usethe shells. The architectural mechanisms for creating special purpose shellshave been implemented, but we are still producing our first tutors with Eon,and have not tried to build a meta-shell yet.

Who is the intended author?

Encoding instructional expertise in GTE seems to be a complex task requir-ing a highly skilled programmer/knowledge engineer familiar with the GTE

273

knowledge base. With Eon, we intend to provide authoring tools that willmake this task available to a much wider audience.

To build a tutor of any reasonable level of sophistication will usuallyrequire the efforts of a design team rather than an individual, and will requirea tool with complexity at least on the order of magnitude of Photoshop, orAutoCAD. So we have no allusions of the average classroom teacher buildingan ITS. But whereas now building ITSs is restricted to a few initiates, theright tools could allow every company and every school to have at least oneperson or team capable of ITS authoring. This team could work with contentexperts and visual artists to create highly effective and motivating systems.

As alluded to previously, our goal is to create a three-tiered suite of author-ing tools, at three levels of abstraction, each appropriate for a different cate-gory of users. At the first tier is the bare Eon shell, a general purpose ITSauthoring system that requires moderate knowledge engineering and instruc-tional design expertise to use. At the second tier are ready-to-use specialpurpose ITS authoring systems that require minimal knowledge engineeringand instructional design expertise to use. The third tier involves simplifiedtools for the average teacher using an ITS in her class.

Even though we do not expect the average teacher to be able to build anITS, once a tutor is built, any teacher could use a simplified subset of the Eontools to customize the tutor for a particular class or student; for example bymaking a teaching strategy more verbose, changing a picture to one that ismore relevant to her class, or changing a prerequisite relationship betweentopics. This is important because teachers may be reluctant to incorporateinstructional systems that they can’t understand or customize. Such toolswill also enable teachers to obtain dynamic visual representations of thecurriculum, student model, and teaching strategies to better understand howthe tutor works.

How powerful must the inferencing be?

There have been a number of approaches to representing control and strategicknowledge in ITS shells, and these can be classified as using either reactivemethods or plan-based methods. “Reactive” control methods do not explicitlymanipulate planning objects, but act opportunistically based on the state ofthe tutorial session and the student. Most ITS’s use reactive control, as doesEon’s procedural (flowline) paradigm. Formal planning methods, in contrast,generate and reason about plans. Examples (other than GTE) include goal-based production rules (Russell et al., 1988; Anderson & Pelletier, 1991;Major, 1992), black board architectures (McCalla & Greer, 1988; W. Murray,1990; Macmillan et al., 1988), and agent-based planning (Cheikes, 1995).Planning mechanisms can also be classified according to whether they pre-

274

plan their actions or make decisions at run time. Decisions related to creatingthe default sequence of instructional units can reasonably be pre-plannedbased on the lesson goals and topic prerequisites, but decisions that have todo with giving feedback to the student must clearly be made on the fly.

Choosing a control method depends on several factors, including:1. How complex must the tutorial behavior be?2. How self explanatory must the tutorial behavior be?3. Who needs to be able to understand the reasons why the tutor behaves as

it does?4. Who needs to be able to design, debug, and modify the instructional

strategies?The primary benefit of planning methods over reactive methods is that theycan ‘think about’ what they are going to do before they do it. They can search alarge space of potential actions (or action sequences) to optimize performance.Both methods can make inferences based on available knowledge and reactintelligently to the student’s needs. But plans can also use information aboutwhat the tutor expects to do in the future. For example, a plan-based systemcan alter the concluding summary of a topic based on whether thenexttopicit expects to teach is a concept or a procedure. Both methods support thedecomposition of instructional tasks (or procedures) in to sub-tasks (sub-procedures or methods), and allow recursive invocation of these.

Unlike reactive methods, planning can notice potential conflicts along onepossible path of actions, and modify that path or take an alternative pathto avoid such conflicts. Another potential benefit of plan-based methods,which has not been made use of in ITS systems (to my knowledge), isthe ability to create plans which achieve more than one instructional goalsimultaneously (called conjunctive goals). Plans, by their nature, think ahead,so they must make some assumptions about the future state of the worldwhen the plan is executed. Tutoring situations are of course unpredictable,and therefore most ITS planning systems incorporate dynamic (“on the fly”)re-planning. In GTE the task interruption mechanism continually puts controlin the hands of the highest level tasks, which have a more global and goal-based view of the situation. This models a teacher’s ability to operate in anuncertain or under constrained situation by acting with least commitmentto the implications of actions. Task decomposition also provides a robustmechanism for allowing student interruption and choice as one alternativemethod for many instructional tasks.

Finally, since plan-based methods reason explicitly about the sequence ofactions they will take, they are better at explaining what they are doing andwhy they are doing it to the instructional designer, or as is infrequently done,to the student. But is the extra complexity necessary for plan-based methodswarranted? Below we argue that it may not be.

275

Managing complexity: Visualization and debugging tools

As mentioned above, GTE’s task decomposition can produce more com-plicated tutorial and discourse behavior than is possible with Eon’s proce-dural representation (also called “parameterized action networks” (Murray &Woolf, 1991)). The drawback of plan-based methods is that they are morecomplicated. Procedural flow lines are intuitively clear and it is easy for thedesigner to follow what is happening (the flow line tool can trace the tutorialexecution by highlighting the current action in the flowline). The task decom-position method seems accessible to a much smaller range of designers. Also,we believe that teachers may want to make small changes to the strategies,and at least will want to be able to view them to understand the strategies.

Plan based methods, including the task decomposition method, are highlymodular representations of human strategic knowledge. Each plan operator(or task or method) is a small self-contained unit. Explicitly representingalternative methods for instructional tasks seems elegant and cognitivelyperspicuous. But this seeming simplicity can be misleading, because controlinformation elicited from human experts often has clearly defined structure(Gruber, 1987), and a high degree of modularity hides the structure of strategicknowledge, obfuscates the context of strategy decisions, and makes strategydesign unwieldy (Lesser, 1984). It is difficult to trace and understand howthe components of a large plan library interact in complex ways to producea specific behavior. In contrast, graphically portrayed procedural representa-tions explicitly shows structural and contextual control information.

Multiple strategies and meta-strategies

Several ITS systems, including GTE, Eon, COCA (Major, 1992), andREDEEM (Major, 1993) allow for very flexibly tutorial behavior by incorpo-rating multiple alternative tutoring strategies. For example, several methodsfor giving hints may be represented. When multiple strategies are used, someform of meta-strategic control is needed, which dynamically chooses the bestamong alternative strategies. As mentioned in Van Marcke’s paper, GTE’smeta strategic knowledge is embedded in the methods themselves, whichmakes their context more explicit, but also obscures the relationship betweenthe elements of meta-strategic information.

COCA, REDEEM, and Eon have a meta-strategic knowledge base that isdistinct from their basic strategic knowledge base. COCA’s meta-strategiesare production rules which enable, disable, or prioritize its instructional rules.The REDEEM system’s goal is to allow teachers to specify teaching behaviorwithout assistance of a knowledge engineer, and it trades very high usabilityfor less flexibility. It uses a parameterized approach where teachers simply

276

have to set a number of parameters (which involves choosing values fromlists or adjusting numerical values using slider widgets, including teacheror student choice, starting with general or specific information, amount ofhinting, amount of interruption). These parameters effect how a set of veryflexible yet pre-defined instructional strategies behave. A set of these para-meter settings is, in effect, a meta strategy.

Eon also uses a parameterized approach (Murray & Woolf, 1991), but ismore flexible since the basic strategies can be built from scratch, and the meta-strategy parameters can also be defined by the user. Users define a number of“strategy parameters”, for example, “degree of hinting,” “degree of interrup-tion” and “preference for general vs. specific information.” Then they createmeta strategies, which specify combinations of values for these parame-ters. For example, the author could create a meta-strategy called “AdvancedLearner,” which includes moderate hinting; give general information beforespecific; and skim (don’t give much information). Based on student perfor-mance, a tutor might change the current strategy to “Advanced Learner,”which then sets the values of the strategy parameters, e.g. setting Degree OfHinting to “moderate.” These global parameters are used in the decisions ofteaching strategy flow lines to, for example, take one branch for moderatehinting and another for maximum hinting.

Using knowledge types

I believe that the issue of “knowledge types” has received far too little atten-tion in the ITS field. The basic idea is summarized in the “Gagne Hypothesis”that there are many types of knowledge or learned behavior, and that theserequire different methods for most effective learning/instruction. For exam-ple, teaching procedural knowledge requires different methods than teach-ing conceptual knowledge, which requires different methods than teachingmetacognitive skills. There seem to be quite numerous and contradictoryinstructional methods in instructional systems, and more generally in instruc-tional theory. I propose that a large portion of the seeming disagreement onthe best instructional methods really boils down to disagreement on whattype of knowledge it is most important to teach. Discussions about knowl-edge types would be far more productive than comparisons of instructionalapproaches,or would at least illuminate such comparisons. For example, thereis a long standing discussion about whether constructivist methods are betterthan “instructivist” methods. Much of this debate is really about whetherit is more important to teach self-regulatory and meta-cognitive skills thandomain-specific facts and skills.

Our conceptual vocabulary of tutorial decision factors includes a knowledgetype taxonomy. Eon Ontology objects define the knowledge types used in a

277

particular tutor, and the types of relationships and properties allowed foreach knowledge type. Knowledge type can then be used as a parameter ininstructional strategies, to branch to different methods of teaching, hinting,explaining, etc., based on the knowledge type. In keeping with our meta-shell approach, the entire knowledge type taxonomy is not included in Eon.Instructional designers choose (off line) which knowledge types to use for aparticular application.

For example, Merrill’s performance/content matrix (a knowledge typingscheme from Merrill (1983), and mentioned in Van Marcke’s paper on page32) is represented in our system by creating knowledge types called Fact,Concept, Principle, and Procedure, and by creating topic levels (discussed inMurray, 1996b) for each performance or mastery level.

Future suggestions and synergy

Below are a number of areas for potential synergy between Eon and GTEresearch, and some general suggestions for future directions for ITS shellwork.

Adding visualization and debugging tools

Ultimately, to make more powerful and complex formalisms usable, authoringtools need to be developed which can clearly and visually portray the staticstructure of the task library, and dynamic structure of task execution. Moretools for tracing task execution and debugging task libraries will also beneeded. This is a challenging task, since, though some design tools havebeen developed for expert system shells to allow for visualizations and tracesof rule invocations, the design of large expert systems is still a black artrequiring significant skills. GTE needs more tools for visualizing and editingits knowledge, and both GTE and Eon need more tools that support tracing,analysis, and debugging of knowledge. The difficulty and complexity of aknowledge base increases exponentially with its size, when its componentscan interact in complex ways. So such tools will be even more essential aswe move from building prototype tutorials to large scale tutoring systems.

Defining the meaning of tasks and methods

When generic instructional strategies are made available to multiple authors,the meanings of the primitives used should be fairly unambiguous. However,the meaning of a term (e.g. a task or method name) is highly dependent, andarguably completely dependent, on how the term is used in a given system.

278

“Introduce topic” could mean completely different things in two differentsystems or for two different teachers, e.g. give an overview, give a motivatingexample, describe why the knowledge is needed. It could be more misleadingthan helpful to provide such a primitive if the possible interpretations aretoo various. GTE accounts for this partially in that its task/method formalismacknowledges the different interpretations (methods) for instructional actions(tasks). However, there is still a lot of room for ambiguity. To address thisissue in both GTE and Eon, I recommend associating a (canned text) meaningor purpose to all primitives (GTE tasks and methods, and Eon strategies andprocedural actions), such as “provides material which will inspire the studentto want to learn the topic” for “Introduction.”

An object oriented paradigm for strategy invocation

GTE has inspired me to consider something similar to the task/methodparadigm, but which has an object-oriented flavor, for implementing multipletutoring strategies. Eon could allow alternative strategies to have the samename, but specify different preconditions. For example, in the current system,in order to implement alternate hinting strategies, the designer has to give eachhinting procedure a different name, e.g. Give-hint-1, Give-hint-2, and Give-hint-3, and create a decision branch in a higher level flow line that essentiallysays “if hsome conditioni run Give-hint-1; IFhanother conditioni run Give-hint-2; else run Give-hint-3.” A better method might be to allow all three hintstrategies to be called Give-hint, and in the higher level flow line would simplerun Give-hint. At this point the preconditions or applicability conditions ofeach Give-hint procedure would be evaluated and the most appropriate onewould be run.

Evaluation

Clearly, both GTE and Eon will need more serious evaluation as the systemsmature. For the near future, colleagues will be more interested in reports ofwhatworks and what didnotwork, andwhy, in addition to demonstrations thata systemdoeswork. Iterative design with user feedback is necessary to insureusability, and a useful formative evaluation can be accomplished by takingcareful notes during these design iterations, and reporting what was found.Many aspects need to be looked at, including generality of tools (what rangeof domains and teaching methods can be encoded); usability of tools (whatlevel of skill is needed, and can robust “real sized” tutorials be built); andthe successfulness of specific instructional strategies. Comparing computerbased tutoring to human tutoring is very difficult since it is improbable thatcomputer and human tutorial sessions can be controlled enough to assign

279

credit and blame. A feasible alternative would be to let experienced teachersevaluate the validity or the output of a compute tutor (called a “blue ribbonpanel” evaluation). The field is still young, and quantitative and summa-tive evaluation methods are not always needed. Qualitative and formativemethods, including “lessons learned” reports, are very useful to colleagues(Murray, 1993).

Common conceptual vocabularies

The field as a whole needs to pay more attention to identifying the types ofknowledge, behavioral objectives, and/or tasks that are taught by intelligenttutors. The field also needs to work toward a shared taxonomy of primitivetutorial actions. We should begin to work toward a common vocabulary ofterms for describing these phenomena, and then enter into discussion andmore experimentation about the best methods to teach of the knowledgetypes in computer based environments. [Footnote: An IEEE standards effortto develop guidelines for terminology and architectures for computer basedlearning systems is now underway. See http://ww2.readadp.com/p1484/.] Wehave been compiling one perspective on these conceptual vocabularies, whichis currently a paper-based knowledge acquisition tool, and have incorporatedsome of GTE’s task and method names into it.

Conclusions

Earlier we described how the goals and baselines of Eon and GTE differ.Actually, we are just approaching the difficult and many faceted problemfrom different directions, Ultimately, Van Marcke wants GTE to be highlyusable, without sacrificing power, and we would like Eon to be powerful andexpressive, without sacrificing usability. Each system is exploring a differentlocation in the space of design tradeoffs. Both systems aim for generality,but GTE’s focuses more on expressiveness, while Eon’s focuses more onusability.

Representing instructional expertise computationally is a difficult chal-lenge. We are all still feeling our way in that dark in many respects. Forwork in progress it is much easier to find faults and inadequacies than toappreciate the contributions, so I applaud Van Marcke for offering his systemas a starting point for comparison and critique. The result of such an offering,this journal issue, will certainly help move the field forward.

280

References

Anderson, J.R. & Reiser, B. (1985). The Lisp Tutor.BYTE(April): 159–175.Anderson, J.R. & Pelletier, R. (1991). A development system for model-tracing tutors, in

Proceedings of the International Conference on the Learning Sciences(pp. 1–8). Evanston,IL.

Cheikes, B. (1995). Should ITS designers be looking for a few good agents? in AIED-95workshop papers for Authoring Shells for Intelligent Tutoring Systems.

Clancey, W. & Joerger, K. (1988). A practical authoring shell for apprenticeship learning.Proceedings of ITS-88(pp. 67–74). June, Montreal.

Gruber, T. (1987). A method for acquiring strategic knowledge from experts. University ofMassachusetts CS Dept. Tech. Report.

Lesgold, A., Lajoie, S., Bunzo, M. & Eggan, G., (1990). A coached practice environment foran electronics troubleshooting job, in Larkin, Chabay & Sheftic, eds.,Computer AssistedInstruction and Intelligent Tutoring Systems Establishing Communication and Collabora-tion. Hillsdale, NJ: Erlbaum.

Lesser, V. (1984). Control in complex knowledge-based systems. Tutorial at the IEEE Com-puter Society AI Conference.

Macmillan, S., Emme, D. & Berkowitz, M. (1988). Instructional planners, lessons learned, inPsotka, Massey & Mutter, eds.,Intelligent Tutoring Systems, Lessons Learned,Hillsdale,NJ: Lawrence Erlbaum.

McCalla, G. & Greer, J. (1988). Intelligent advising in problem solving domains: The SCENT-3architecture.Proceedings of ITS-88(pp. 124–131). June, Montreal, Canada.

Major, N.P. & Reichgelt, H (1992). COCA – A shell for intelligent tutoring systems, in C.Frasson, G. Gauthier & G.I. McCalla, eds.,Proc. of Intelligent Tutoring Systems ’92.Berlin: Springer Verlag.

Major, N.P., (1993). Teachers and teaching strategies.Proc. of the Seventh International PEGConference, Heriot-Watt Univ., Edinburgh.

Merrill, M.D. (1983). Component display theory, in C.M. Reigeluth, ed.,Instructional-DesignTheories and Models: An Overview of Their Current Status(pp. 279–333). London:Lawrence Erlbaum Associates.

Merrill, M.D. (1989). An instructional design expert system.Computer-Based Instruction16(3): 95–101.

Munroe, A., Pizzini, Q., Towne, D., Wogulis, J. & Coller, L. (1994). Authoring proceduraltraining by direct manipulation. USC working paper WP94-3.

Murray, T. (1993). Formative qualitative evaluation for “exploratory” ITS research.J. of AIand Education4(2).

Murray, T. (1996a). Having it all, maybe: Design tradeoffs in ITS authoring tools, inProc. ofITS 96, Third Inter. Conf. on Intelligent Tutoring Systems. Berlin: Springer-Verlag.

Murray, T. (1996b). Special purpose ontologies and the representation of pedagogical knowl-edge, inProc. of Second International Conference on the Learning Sciences. Charlottes-ville, VA: AACE.

Murray, T (1996c). Toward a conceptual vocabulary for intelligent tutoring systems. Workingpaper available at http://www.cs.umass.edu/�tmurray/.

Murray, T. & Woolf, B. (1991). A knowledge acquisition framework for intelligent learningenvironments.ACM SIGART Bulletin2(2): 1–13.

Murray, W. (1990). A blackboard-based dynamic instructional planner, inProc. of EightNational Conference on Artificial Intelligence(pp. 434, 441).

Russell, D., Moran, T. & Jordan, D. (1988). The instructional design environment, in Psotka,Massey & Mutter, eds.,Intelligent Tutoring Systems, Lessons Learned. Hillsdale, NJ:Lawrence Erlbaum.

Van Marcke, K. (1997). GTE, an epistemological approach to instructional modeling. In thissame issue ofInstructional Science.