chapter 4 pedagogically- informed feedback - sfu.caheift/trude/chapter4.pdf · chapter 4...

33
Designed Intelligence: A Language Teacher Model 107 CHAPTER 4 Pedagogically- Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires linguistic and pedagogic knowledge. The task of the grammar and the parser is to generate phrase descriptors which provide the linguistic analysis of students’ input. The phrase descriptors, however, need to be processed further to reflect principles of language learning. The Analysis Module, the Student Model, and the Filtering Module, respectively, constitute the pedagogic components carrying out the analysis described. The Licensing Module, discussed in Chapter 3, chooses the desired parse. All phrase descriptors associated with the selected parse are then sent to the Analysis Module, and the information obtained is subsequently passed to the Student Model and Filtering Module. Figure 4.1 illustrates the relation between the Analysis Module, the Student Model, and the Filtering Module. 1

Upload: buixuyen

Post on 01-Mar-2019

223 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 107

CHAPTER 4 Pedagogically-Informed Feedback

4.1 Introduction

Analyzing student input in an ILTS requires linguistic and pedagogic

knowledge. The task of the grammar and the parser is to generate phrase

descriptors which provide the linguistic analysis of students’ input. The

phrase descriptors, however, need to be processed further to reflect principles

of language learning. The Analysis Module, the Student Model, and the

Filtering Module, respectively, constitute the pedagogic components carrying

out the analysis described.

The Licensing Module, discussed in Chapter 3, chooses the desired

parse. All phrase descriptors associated with the selected parse are then sent

to the Analysis Module, and the information obtained is subsequently passed

to the Student Model and Filtering Module. Figure 4.1 illustrates the relation

between the Analysis Module, the Student Model, and the Filtering Module.1

Page 2: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 108

The Analysis Module takes all incoming phrase descriptors and

generates possible instructional feedback at different levels of granularity

correlated with student expertise. In ILTSs, granularity is particularly

important to framing responses to learners’ errors. Inexperienced students

1. Of these three modules, only the Analysis Module is language-dependent but to a trivialextent: the feedback messages in the current implementation happen to be in English.

Figure 4.1: Pedagogic Modules

Analysis Module

and produces instructional feedbackof increasing abstraction

All Phrase Descriptors

Student Model

keeps learner model updatesand decides on the student level

Filtering Module

decides on the order of instructional feedback

Error-contingent Feedback Suited to Learner Expertise

returns student model updates

from the Selected Parse

Page 3: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 109

require detailed instruction while experienced students benefit best from

higher level reminders and explanations [LaReau & Vockell 1989].

For instance, in example (1a) the student made an error with the

determiner einen of the prepositional phrase. Von is a dative preposition and

Urlaub is a masculine noun. The correct article is einem.

(1a) *Sie träumt von einen Urlaub.

(b) Sie träumt von einem Urlaub.

She is dreaming of a vacation.

In a typical student-teacher interaction, feedback depends on the

students’ previous performance history. For instance, for the error in example

(1a), a learner who generally has mastered the concept of dative assigning

prepositions might merely receive a hint indicating that a mistake occurred

within the prepositional phrase. For the learner who generally knows the

grammatical rule but still needs practice in its application, the feedback is

less general: in addition to location, the teacher might point out the type of the

error (case). For the novice learner, the feedback would be as specific as

possible. In addition to the location and type of the error, information as to the

exact source of the error, that is, the precise grammatical rule that has been

violated (dative case), would be provided.

Note, however, that even the most specific report aimed at the novice

learner refrains from revealing the correct answer. For the error in example

(1a), the beginner student is still required to decide on the correct inflection of

a masculine indefinite article in the dative case. The pedagogical principle

underlying this design is guided discovery learning. According to Elsom-Cook

[1988], guided discovery takes the student along a continuum from heavily

structured, tutor-directed learning to where the tutor plays less and less of a

role. Applied to feedback, the pedagogy scales messages on a continuum from

least-to-most specific guiding the student towards the correct answer.

Page 4: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 110

Burton and Brown [1982] state that hinting at the source of an error

supports the development of students’ self-regulation, a prerequisite for

guided discovery learning. A system can lead the learner towards the correct

answer rather than simply supplying it, thus assisting the students in finding

the source of an error themselves. The ability to discover the source of an

error, however, strongly correlates with the expertise of the student: the more

expert the learner the less explicit feedback is necessary [Fischer & Mandl

1988].

Granularity captures this pedagogically intuitive concept: the

Analysis Module generates three categories of instructional feedback

corresponding to three learning levels: expert, intermediate, and novice. The

responses are scaled from least-to-most specific, respectively. Provided with

three categories of feedback by the Analysis Module, the Student Model

selects the error response suited to the student’s expertise according to a

record of the student’s previous performance history on a particular

grammatical phenomenon.

The Student Model tracks 79 grammar constraints, corresponding to

the grammatical concepts being monitored in an introductory course for

German.2 For each of the grammar constraints, the Student Model keeps a

counter which, at any given instance in the evaluation process, falls in the

range of one of the three learner levels. If a grammatical constraint has been

met, the counter is decremented. If the constraint has not been met, the

counter is incremented and a feedback message suited to the performance

level of the learner is selected.

2. The grammar constraints have been chosen according to Wie geht’s, the course bookcommonly used in an introductory course for German [see Sevin, Sevin, & Bean 1991].

Page 5: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 111

An example of a grammar constraint is pp-dat which records the

student’s performance on dative assigning prepositions. Each student is

initially assessed as an intermediate, which has been chosen as a reasonable

default level.3 A learner who violates the constraint on dative prepositions

will, at first, obtain the feedback message for the intermediate. If the student

commits the same error in subsequent exercises, s/he will soon be assessed a

novice. At this point, the Student Model will select the more detailed feedback

message suited to the beginner. However, each time the student applies the

grammatical constraint correctly, the Student Model records the success. After

demonstrating proficiency, the student will again be assessed as intermediate,

or, even expert. Maintaining a large number of grammatical constraints

allows for a very detailed portrait of an individual student’s language

competence over a wide-range of grammatical phenomena.

The Filtering Module decides on the order in which instructional

feedback is displayed. Feedback is provided for one error at a time to avoid

overloading the student with extensive error reports. According to van der

Linden [1993], displaying more than one feedback message at a time makes

the correction process too complex for the student.

The ordering of the instructional feedback is decided by an Error

Priority Queue which ranks the 79 grammar constraints maintained by the

Student Model. The ordering of the grammar constraints and ultimately

feedback is, however, adjustable. Depending on the pedagogy of a particular

language instructor, they can be reordered to reflect the focus of a particular

exercise. In addition, errors which are not relevant to a particular exercise

3. The intention of the system is primarily to augment but not to replace classroominstruction. The system is therefore a practice tool for the student who previously has beenexposed to a particular grammatical concept of German. It is thus safe to assume that thelearner will not be a complete beginner nor an expert on the grammatical phenomenapracticed.

Page 6: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 112

need not be reported at all, although the error itself still is recorded in the

Student Model.

The ordering of the grammar constraints also accommodates

contingent errors, a special class of multiple errors, and thus avoids

misleading or redundant feedback. For instance, in example (2a) the student

made a mistake with the prepositional phrase. The verb denken

subcategorizes for the preposition an which requires the accusative pronoun

dich, while von is a dative preposition requiring the dative pronoun dir. If we

ignore the dependency of the errors with the preposition and the pronoun, the

feedback to the student would be “von” is the wrong preposition and This is the

wrong case of the pronoun “dich”. However, the pronoun dich is not incorrect if

the student changes the preposition von to an because an requires the

accusative pronoun dich. Depending on the order of the feedback, the student

might end up changing von to an and dich to dir and wind up utterly

confused. S/he might ultimately fail to find the correct case of the pronoun

because it had been flagged as an error in the original sentence.

(2a) * Ich denke von dich.

dat prep. acc pronoun

(b) Ich denke an dich.

acc prep. acc pronoun

I am thinking of you.

The error in the pronoun is correctly flagged by the system from a

purely logical point of view. However, from a pedagogical perspective reporting

the error in the pronoun dich is redundant and possibly misleading. In such

instances, only the error in the preposition von is reported although the error

in the pronoun is recorded in the Student Model. Thus, the system, while

silently recording all errors, does not necessarily comment on all of them.

Page 7: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 113

The final result of the three modules is error-contingent feedback

suited to students’ expertise. The following sections will discuss the Analysis

Module, the Student Model, and the Filtering Module in detail.

4.2 The Analysis Module

The function of the Analysis Module is to take all incoming phrase

descriptors as input and return student model updates and potential error

feedback at different levels of granularity for each phrase descriptor.

Granularity has been previously applied to an Intelligent Tutoring System for

LISP programming [McCalla & Greer 1994, Greer & McCalla 1989]. In their

system SCENT, Greer and McCalla implemented granularity to “recognize

the strategies novice students employ when they solve simple recursive LISP

programming problems.”4 In this analysis, however, granularity is used in

framing responses to learners’ errors: inexperienced students obtain detailed

instruction while experienced students receive higher level reminders and

explanations. For example, consider the ungrammatical sentence in (3a).

(3a) *Der Mann dankt dem Frau.

(b) Der Mann dankt der Frau.

The man thanks the woman.

In example (3a), the student has provided the wrong determiner for

the indirect object. For the error dem Frau, the system generates feedback of

increasing abstraction that the instruction system can use when interacting

with the student. The level of the learner, either expert, intermediate, or

novice according to the current state of the Student Model, determines the

4. Greer & McCalla [1989], p. 478.

Page 8: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 114

particular feedback displayed. The responses, given in (4a) - (c) correspond to

the three learner levels for the error in example (3a), respectively:

(4a) There is a mistake with the indirect object.

(b) There is a mistake in gender with the indirect object.

(c) This is not the correct article for the indirect object. The noun isfeminine.

For the expert, the feedback is most general, providing a hint to where

in the sentence the error occurred (indirect object). For the intermediate

learner, the feedback is more detailed, providing additional information on the

type of error (gender). For the beginner, the feedback is the most precise. It not

only pinpoints the location and type of the error but also refers to the exact

source of the error (feminine noun).

Figure 4.2 displays the partial Granularity Hierarchy for constraints

in feature matching.5 The Granularity Hierarchy is a representation of the

instructions given to the student correlated with the grammatical constraints

monitored by a phrase descriptor. Each term in a phrase descriptor

corresponds to a level in the Granularity Hierarchy. For example, for the

indirect object of the sentence *Der Mann dankt dem Frau, given in (3a) on p.

113, the grammar and the parser will generate the phrase descriptor

[main_clause [vp_indirobj [fem error]]]. The top node of the Hierarchy

specifies in which kind of clause the error occurred. The phrase descriptor

indicates that a mistake was made in a main-clause. The next level in the

Granularity Hierarchy lists possible errors in each clause type. As indicated

by the phrase descriptor, the mistake refers to the indirect object. An even

finer-grained constraint specification is found in the next lower level of the

Granularity Hierarchy. For instance, an indirect object can be incorrectly

5. The hierarchy here is simplified for the purpose of illustration. The lowest nodes in thehierarchy split into further nodes as illustrated with gender.

Page 9: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 115

inflected for either case, number, or gender. The phrase descriptor specifies

that an error in gender occurred, specifically with a feminine noun which

corresponds to the lowest level in the Granularity Hierarchy.

Each node in the Granularity Hierarchy corresponds to a level of

specificity of a feedback message. Granularity works along one dimension,

namely, abstraction: travelling downward through the Hierarchy, nodes lower

in the Hierarchy will have associated with them messages of increasing

specificity. Generally, the more experienced the student the coarser-grained

the message and the higher the node.

The Analysis Module is implemented in DATR [Evans and Gazdar

1990], a language designed for pattern-matching and representing multiple

inheritance. Nodes in DATR are represented by the name of the node followed

Figure 4.2: Granularity Hierarchy for Constraints in Feature Matching

Main Clause

Coordinate Clause

Subordinate Clause

Subject-Verb Subject Direct Object Indirect Object Prep. Phrase Verb Phrase

NumberPerson

Semantic Constraints

GenderNumber

Case

Semantic Constraints

Aux/Pastpp.

Modal/InfinitiveVerb Inflection

FeminineMasculine

Neutersein/habenSingular

Plural

Specificity

etc.etc. etc.

Decreasing

Preposition

Page 10: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 116

by paths and their values. For example, the node given in Figure 4.3

corresponds to the phrase descriptor that records the position of a finite verb

in a subordinate clause.

The paths in a node definition represent descriptions of grammatical

constraints monitored by phrase descriptors. The matching algorithm of

DATR selects the longest path that matches left to right. Each path in a node

is associated with atoms on the right of ‘==’. For example, if there has been an

error in word order in a subordinate clause, the parse will contain the phrase

descriptor [sub_clause [position_subclause [finite error]]]. This will match the

path <sub_clause position_subclause finite error> which specifies four atoms

for each of three groups. Each group represents a learning level. The three

learning levels considered are: expert, intermediate, and novice6, given in

Figure 4.3. 7

6. Finer distinctions can be made by looking at the actual error count either for the purposesof evaluation and remediation; however, three levels are sufficient to distinguish among thefeedback messages encoded.

<sub_clause position_subclause finite error> ==

('possubclfin' '3' 'true' 'The verb in the subordinate clause is not in the correct position.'

'possubclfin' '2' 'true' 'The finite verb in the subordinate clause is not in the correct

position.'

'possubclfin' '1' 'true' 'The finite verb in the subordinate clause is not in the correct

position. It has to be the last element of the sentence.')

<sub_clause position_subclause finite correct> == ('possubclfin' '1' 'false' ''

'possubclfin' '1' 'false' ''

'possubclfin' '1' 'false' '')

<sub_clause position_subclause finite absent> == ('').

Figure 4.3: DATR Code Listing for a Finite Verb in a Subordinate Clause

Page 11: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 117

The first atom in each group specifies the grammar constraint as

described by the incoming phrase descriptor. For example, the grammar

constraint possubclfin represents finite verb position in a subordinate clause.

The second atom specifies a value that is decremented or incremented

depending on whether a grammatical constraint has been met or not,

respectively.

The Boolean values, as the third atom, indicate whether it is an

increment or decrement: true for increment, false for decrement.

Finally, the fourth atom specifies a feedback message. For example,

for the path <sub_clause position_subclause finite error> provided earlier, the

specificity of each response corresponds to the grammar constraints in the

Granularity Hierarchy which frames responses to constraints on verb

position, given in Figure 4.4.

The three instructional responses associated with each node

correspond to the three learner levels. The feedback for the beginner student

reflects the lowest node in the Granularity Hierarchy, while for the

intermediate and expert student the message will refer to nodes higher in the

hierarchy. In a typical student-teacher interaction, as the student becomes

more proficient in the use of a grammatical construction, error feedback

becomes less specific.

If there has been no error in the student’s input, the phrase descriptor

is [sub_clause [position_subclause [finite correct]]] and no message is

associated with the name of the grammar constraint. However, the first three

atoms, the grammar constraint, the decrement, and the Boolean value are

7. A phrase descriptor that contains the value absent is ignored by the system. Thus, the listto the right of ‘==’ is empty.

Page 12: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 118

still specified, given in Figure 4.3 on p. 116. They contribute to a Student

Model update to record the success.

Finally, if the phrase descriptor is [sub_clause [position_subclause

[finite absent]]], then the grammatical phenomenon was not present in the

student input. As a result, the phrase descriptor is ignored by the system,

indicated by an empty list in Figure 4.3 on p. 116.

The final output of the Analysis Module is a student model update

and a list of possible instructional responses from a coarse to fine grain size

for each incoming phrase descriptor. The more expert the student with a

particular grammatical construction, the more general the feedback.

Figure 4.4: Granularity Hierarchy for Constraints in Linear Precedence

Main Clause

Coordinate ClauseSubordinate Clause

DecreasingSpecificity

Verb Position

Finite

Past Participle

Infinitive

Second Position Final Position Second-to-last Position

Page 13: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 119

The modular design of the system also allows the encoding of

instructional feedback suited to the pedagogy of a particular teacher. For

instance, an instructor who preferred to avoid the term finite verb in beginner

messages could substitute the simpler verb.8

The list of alternative instructional responses from the Analysis

Module has to be processed further before being displayed to the student. The

Student Model, which picks a response suited to the level of the student, will

be discussed in the following section.

4.3 The Student Model

Individualization of the learning process is one of the features of a

student-teacher interaction that distinguishes it from gross mainstreaming of

students characteristic of workbooks. Students learn at their own pace and

often, work for their own purposes. Learners also vary with respect to prior

language experience, aptitude, and/or learning styles and strategies [Oxford

1995]. According to the Individual Differences Theory as described by Oxford

[1995], if learners learn differently then they likely benefit from

individualized instruction. An ILTS can adapt itself to different learner

needs.9 The requirement, however, is that the system incorporates and

8. There is an argument to be made for softening technical terms in the intermediate andnovice categories. Rather than implementing a pseudo-vocabulary, however, a system canunderline or highlight the error and provide hot links to information and examples on thelinguistic terminology. The advantage of using precise linguistic terminology for all learnerlevels might result in students not only learning the grammatical concepts but the properterminology at the same time.9. In its strict definition, ILTSs which do not implement a student model are ICALL systemsrather than ILTSs. However, in this thesis all systems which make use of Natural LanguageProcessing are referred to as ILTSs.

Page 14: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 120

updates knowledge about the learner. Student modelling provides the key to

individualized knowledge-based instruction [McCalla & Greer 1992].

Student modelling has not been a strong focus of parser-based ILTSs,

likely because the challenging task of representing the domain knowledge in

ILTSs is still largely incomplete [Holland & Kaplan 1995b].10 If the grammar

is not accurate and complete, even a precise student model cannot

compensate. For instance, Holland [1994] states that a system which does not

flag ambiguous and contingent errors accurately will obscure a student

model. A further possible reason is that many significant differences in

language learning styles (situational, aural, visual, etc.) are precluded from

consideration in text-based ILTSs.

ILTSs which do have a student model primarily concentrate on

subject matter performance. Modelling students’ surface errors assists in

individualizing the language learning process and “is sufficient to model the

student to the level of detail necessary for the teaching decisions we are able

to make.”11 The technique aids in teaching the required skills and

remediating the student.12

10. According to Holland and Kaplan [1995b] there are two trends with student modelling insystems for language learning. The North American focus lies on the NLP module, the domainknowledge. Due to the complexity of the NLP component, these systems implement studentmodels only to the extent of analyzing surface errors. In contrast, European systemsconcentrate on more sophisticated student models by hypothesizing causes of errors (seeChanier et al., [1992]). However, they “opt for narrow NLP bandwidth”. Holland & Kaplan[1995b], p. 364. 11. Elsom-Cook [1993], p. 238.12. The modelling techniques employed in language learning are distinct from those appliedto procedural tasks. In procedural tasks, the modelling techniques presume that the studentcan learn the skill in terms of a number of formal steps. Yet although languages are rule-governed systems and we can represent linguistic ability in terms of formal step-by-step rules,we do not produce language by consciously following them [Bailin 1990]. As a result, themodelling techniques in language learning primarily diagnose the sources of errors ratherthan model strategies which the student used in solving a particular problem.

Page 15: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 121

McCalla [1992] makes a distinction between implicit and explicit

student modelling which is particularly useful in classifying the student

models in ILTSs.

An implicit student model is static, in the sense that the Student

Model is reflected in the design decisions inherent to the system and derived

from a designer’s point of view. For instance, in an ILTS the native language

of the learner can be encoded as a bug model and ultimately used to diagnose

errors.

In contrast, an explicit student model is dynamic. It is a

representation of the learner which is used to drive instructional decisions.

For ILTSs, for instance, the student model can assist in guiding the student

through remedial exercises or it can adjust instructional feedback suited to

the level of the learner. In either case, the decisions are based on the previous

performance history of the learner. The following discussion will provide

examples of ILTSs which have implemented implicit and explicit student

models.

4.3.1 Implicit Student Models

Implicit student modelling has been applied to ILTSs to diagnose

errors. For example, in Catt & Hirst’s [1990] system Scripsi the native

language of the student represents the learner model. It is used to model the

learner’s interlanguage. With regard to student modelling, the pitfall of such

an implementation is that it is a static conception. The system’s view of the

learner cannot change across interactions with the system. It has no impact

on instructional decisions and provides only a gross individualization of the

learning process when ideally, a student model is dynamic [Holt et al. 1994].

Page 16: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 122

In a more individualized example, Bull [1994] developed a system

that teaches clitic pronoun placement in European Portuguese. The student

model is based on the system’s and the student’s belief measures, language

learning strategies, and language awareness.

The system’s belief measure is comprised of the proportion of

incorrect/correct uses of the rule; the students provide the data for the

student’s belief measure, being required to state their confidence in their

answer when entering sentences. Learners also identify their preferred

learning strategies when using the program. According to Bull [1994],

language awareness is achieved by allowing the student access to all

information held in the system. None of the information, however, is used to

drive the instructional process. In addition, a number of studies have shown

that students tend to not take advantage of the option to access additional

information. For example, Cobb & Stevens [1996] found that in their reading

program learners' use of self-accessible help was virtually non-existent, in

spite of their previously having tried it in a practice session, and also having

doubled the success rate as compared to either a no help or dictionary help

option in the practice session.

4.3.2 Explicit Student Models

In developing an explicit student model one typically starts by

making some initial assumptions based on pretests or stereotypical

postulations about the learner. For example, initially every student could be

assessed as an intermediate. During the instructional process, the student

model adjusts to student’s behaviour moving to a novice or expert profile, as

appropriate. This technique is used in explicit student models to make

instructional decisions.

Page 17: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 123

Explicit student modelling has been used in a number of ILTSs,

primarily in the form of tracking. Tracking can be as simple as calculating

percentages of correct answers or more sophisticatedly, identifying particular

errors which occurred in the student’s input. The information is then used to

alter the instructional process, either in the form of further language tasks or

feedback.

Explicit student modelling is found in the system The Fawlty Article

Tutor [Kurup, Greer & McCalla 1992] which teaches correct article use in

English. The system presents the student with scenarios whereby the student

must select the correct article form and the appropriate rule. The tutor keeps

an error count and selects the scenarios on the basis of the performance of the

student; thus the path through the program is individualized by altering the

instructional process according to prior performance of the student.

Bailin [1988, 1990] in his system Verbcon/Diagnosis also employs the

tracking method. Diagnosis provides practice in using English verb forms in

written texts. All verbs are presented in their infinitival form challenging the

student to provide the appropriate verb form. The system tracks the most

frequent error occurrence and the context in which the error occurred. The

information is used to provide informative feedback based on contrasting

correct and ungrammatical uses of tenses. In addition, Diagnosis suggests

exercises to help with the remediation process.

The student model under the analysis of this dissertation is based on

students’ prior performance and it ultimately has two main functions. The

first is to select instructional feedback suited to learner expertise and second

to use the student’s performance for assessment and remediation.13 The

following section will discuss the technique employed.

Page 18: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 124

4.3.3 Feedback Suited to Learner Expertise

To select instructional feedback suited to the level of the learner, the

Student Model keeps track of 79 grammar constraints. These grammar

constraints correspond to the grammatical concepts to be monitored in an

introductory course for German. The grammar constraints are split among

the three clause types, main, subordinate, and coordinate, each containing 25

nodes. In addition, there are two grammar constraints monitoring noun

phrases practiced in isolation, one for verb-initial position in main-clauses,

and one for the entire sentence.

The Student Model assumes three types of learners: the novice, the

intermediate, and the expert. Each student level is represented by a range of

values14:

novice: 20 ≤ X ≤ 30

intermediate: 10 ≤ X < 20

expert: 0 ≤ X < 10

Initially, the learner is assessed with the value 15 for each grammar

constraint, representing the mean score of the intermediate learner. The

values are used by the Student Model to decide on the specificity of the

instructional feedback being displayed. The intermediate learner has been

chosen as a reasonable default. While the messages might be initially too

overspecified for the expert and too underspecified for the novice, they will

quickly adjust to the actual learner level. Pre-testing could be used to

individualize the model from the outset.

13. The system does not yet contain any pre-defined exercises, but in an extended versionerror counts held in the Student Model will be available to the learner. They form the basis forbranching decisions and remediation recommendations.14. The ranges chosen for each student level roughly correspond to the average size of agrammar exercise unit. Since the default decrement for the error count is 1, successfulcompletion of 10-15 exercises will record a change in student level.

Page 19: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 125

For each learner level, the Student Model receives four atoms from

the Analysis Module, described previously: the name of a grammar

constraint, an increment/decrement, a Boolean value and in case of the latter

being true, a feedback message.

The counter of the grammar constraint determines which of the three

learner responses is selected. If the number is 20 or greater, the system

displays the message suited for the beginner. If it is less than 10, the system

treats the learner as expert, and any number in between points to an

intermediate. The counter is bounded at 0 and 30.

Once the Student Model has selected the feedback message to be

displayed to the learner, the counter is incremented by the incoming weighted

constant. For the expert, the increment is generally 3, for the intermediate 2,

and for the novice 1. The consequence of this sequence is that in case of errors,

the student will switch quickly from expert to intermediate, and somewhat

less quickly from intermediate to novice. As a result, if a student is not an

expert with a given grammatical construction after all, the feedback will

quickly become more informative from the expert to the intermediate level.

The transition from the intermediate to the beginner level is slower since the

feedback for the former already points at a specific error type. For instance, for

the grammar constraint possubclfin, given on page 116 the feedback for the

intermediate level specifies The finite verb in the subordinate clause is not in

the correct position.15

15. Ultimately, it might be desirable to incorporate knowledge of the student’s pastperformance. For example, at the end of each practice session the current error count for aparticular grammar constraint could be averaged with a historical count representing the lastN sessions. By considering a historical error count in determining student level, momentarylapses in performance would be balanced by previous performance.

Page 20: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 126

In the case of a correct response, as indicated by the Boolean value

false received from the Analysis Module, the constant of the grammar

constraint is subtracted from the counter. The decrement for all grammar

constraints is 1. Thus, the transition from novice to intermediate to expert is

gradual. The result of assigning a small decrement at all learner levels is that

any student has to apply a correct construction many times before the

feedback becomes sparse.

The Student Model presented has a number of advantages. It takes

into account students’ past performance, and by adjusting the value to be

incremented or decremented, it is adaptable to a particular grammatical

constraint in an exercise or the pedagogy of a particular instructor. For

example, a language instructor might rate some errors more salient than

others in a given exercise. In such an instance, the increment/decrement of

some grammar constraints can be tuned to change their sensitivity.

Its main strength, however, lies in the fact that a single errorful

sentence will not drastically change the overall assessment of the student.

The phrase descriptors collect errors that indicate precisely the grammatical

context of the mistake. This enables the Analysis Module to create grammar

constraints which reflect very specific grammatical concepts that are

maintained in the Student Model. The consequence is that a student can be at

a different level for any given grammar constraint reflecting the performance

of each particular grammar skill. This is desirable in a language teaching

environment because as a student progresses through a language course a

single measure is not sufficient to capture the knowledge of the learner and to

distinguish among learners. The Student Model described allows the student

to be geared toward error-contingent and individualized remediation.

Page 21: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 127

The Student Model can also be used for assessment and remediation.

A branching program can be implemented where students’ tasks are

determined by their prior performance. The grammar constraints can be

weighted for a particular set of exercises, so as to be especially sensitive to

salient errors. In an authoring program, these settings could be adjusted by

the language instructor.

Finally, the data obtained can be used for further research in

improving the overall performance of the system, and might prove useful in

providing objective measures of student performance.

The final step in analyzing students’ input is handled by the Filtering

Module. The task here is to accommodate multiple errors. Multiple errors

have been largely overlooked in ILTSs. From a pedagogical perspective,

however, instructional feedback messages need to be prioritized by the system

and displayed one at a time to the student to avoid multiple error reports and

redundant and/or misleading feedback in the case of contingent errors. The

following discussion will focus on this task.

4.4 The Filtering Module

While it is desirable to construct a system capable of detecting and

accurately explaining all errors, it does not follow that the system should

display each and every error detected. In the absence of an error filtering

mechanism, the sheer amount of feedback would overwhelm the student. For

example, in evaluating her own system Schwind [1990a] reports that

“[s]ometimes, however, the explanations were too long, especially when

students accumulated errors.”16 In a language learning exercise, a student

Page 22: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 128

might make more than one error. However, a language instructor typically

skips irrelevant errors, and discusses the remaining ones one at a time.

Example (5a) illustrates an example of multiple errors.

(5a) * Heute meine Kindern haben gespeilt mit der Ball.

(b) Heute haben meine Kinder mit dem Ball gespielt.

Today my children were playing with the ball.

In example (5a) the student made the following five errors:

1. word order: the finite verb haben needs to be in second position

2. word order: the nonfinite verb gespielt needs to be in final position

3. spelling error with the verb spielen

4. wrong plural inflection for the subject Kinder

5. wrong case for the dative determiner dem

From a pedagogical and also motivational point of view, a system

should not overwhelm a student with instructional feedback referring to more

than one error at a time. Schwind’s [1990a] solution to this problem is that

multiple errors should be avoided from the outset. She suggests that sentence

construction exercises should focus on specific grammatical phenomena such

as prepositions or verb cases [see also Kenning & Kenning 1990].

While Schwind’s approach is probably inherent in many ILTSs17,

limiting the teaching domain is only a partial solution. Even a basic sentence

in German, as illustrated in example (5a), requires a number of rules and

knowledge about the case system, prepositions, word order, etc.18

Little research has been done in Computer-Assisted Language

Learning regarding the volume of feedback for different kinds of learners at

16. Schwind [1990a], p. 577.17. ILTSs concentrate on sublanguages to also achieve a higher degree of accuracy. [Levin &Evans 1995].18. Holland [1994], in her system BRIDGE, displays only one error at a time, and permitsinstructors to divide errors into primary, which are automatically displayed, and secondary,which are displayed only at the student’s request.

Page 23: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 129

different stages in their language development. However, van der Linden

[1993] found that “feedback, in order to be consulted, has to be concise and

precise. Long feedback (exceeding three lines) is not read and for that reason

not useful.”19 She further states that displaying more than one feedback

response at a time makes the correction process too complex for the student

[van der Linden 1993].

Van der Linden’s [1993] study makes three final recommendations:

1. feedback needs to be accurate in order to be of any use to the student,

2. displaying more than one error message at a time is notvery useful because at some point they probably will notbe read, and

3. explanations for a particular error should also be keptshort.

With regard to feedback display, van der Linden’s [1993]

recommendations require a system submodule to sift all incoming errors. The

errors have to be reported one at a time and the error explanations should be

brief. This provides the student with enough information to correct the error,

but not an overwhelming amount, and yet records detailed information within

the student model for assessment and remediation.

The analysis described implements an Error Priority Queue which

ranks student errors so as to display a single feedback message in case of

multiple constraint violations. The ranking of student errors in the Error

Priority Queue is, however, flexible: the grammar constraints can be reordered

to reflect the desired emphasis of a particular exercise. In addition, a language

instructor might choose not to report some errors. In such an instance, some

grammar constraints will display no feedback message at all, although the

19. Van der Linden [1993], p. 65.

Page 24: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 130

error will still be recorded in the Student Model. The following section will

discuss the Error Priority Queue.

4.4.1 The Error Priority Queue

The Student Model maintains grammar constraints and selects

instructional feedback suited to learners’ expertise. In case of multiple errors,

the Error Priority Queue determines the order in which instructional

feedback messages are displayed to the learner. It ranks instructional

feedback with respect to

1. the importance of an error within a given sentence, and

2. the dependency of errors of syntactically lower and higherconstituents.

The Error Priority Queue for the grammar constraints of a main

clause is partially given in (6).20 The names of the grammar constraints

generated and maintained by the Analysis Module and Student Model,

respectively are given in parentheses.

(6) Error Priority Queue

I. Word Order in a Main Clause

1. position of a finite verb in a main-clause (posmainclfin)

2. position of a nonfinite verb in a main-clause (posmainclnonfin)

3. position of a finite verb in initial position (posmainclinitial)

II. Indirect Objects in a Main Clause

1. case of the noun of the indirect object (indirobjnounmaincl)

2. case, number, and gender of the determiner of the indirect object(indirobjmaincl)

III. Conjoined Indirect Objects in a Main-Clause

1. case agreement of conjoined nouns of indirect objects (indirobjconjnounmaincl)

20. A complete list of the grammar constraints implemented in the system is provided in theAppendix.

Page 25: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 131

2. case, number, and gender of conjoined determiners of indirectobjects (indirobjconjmaincl)

IV. Prepositional Phrases in a Main Clause

1. choice of preposition (prepmaincl)

2. case of the noun of a prepositional phrase in the accusative(pp_accnounmaincl)

3. case, number, and gender of the determiner of a prepositionalphrase in the accusative (pp_accmaincl)

4. case of the noun of a prepositional phrase in the dative(pp_datnounmaincl)

5. case, number, and gender of the determiner of a prepositionalphrase in the dative (pp_datmaincl)

Each grammar constraint, given in the Error Priority Queue in (6)

correlates to a node in the Granularity Hierarchy, provided earlier. The

grammar constraints are grouped according to grammatical phenomena. For

example, the group prepositional phrases in a main clause contains all

constraints monitored with prepositional phrases and it corresponds to the

node prep. phrase in the Granularity Hierarchy in Figure 4.2 on p. 115. Each

member of a group in the Error Priority Queue refers to a node lower in the

Granularity Hierarchy: the grammar constraint choice of preposition, for

example, correlates with the node preposition in the Granularity Hierarchy.

The groups in the Error Priority Queue are sorted according to the

importance of an error within a sentence and the members within the group

are sorted according to the dependency of errors of syntactically lower and

higher constituents. If the student made multiple errors, the system ranks

instructional feedback messages according to the order specified and displays

them one at a time.

The Error Priority Queue shown in (6) reflects the default setting for

the importance of an error in a given exercise. For example, grammar

constraints 1. to 3. of the group Word Order in a Main Clause refer to errors in

linear precedence. In the default setting, they are reported first since word

Page 26: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 132

order is one of the fundamental concepts of a language and thus likely to have

high priority in most exercises.

The ordering of the groups of grammar constraints can, however, be

altered to reflect the pedagogy of a particular language instructor. For

example, an instructor might want to centre exercises around dative case

assignment. In such an instance, the grammar constraints can be reordered so

that errors of indirect objects are reported first. In addition, a language

instructor might choose to suppress some errors, so as not to distract the

student from the main task. These would be errors which are not relevant to a

particular exercise. Suppressing certain errors, does not affect their

contribution to the Student Model, on the rationale that behind-the-scenes

information should be as detailed as possible.

The Error Priority Queue also takes into account contingent errors, a

special class of multiple errors. Contingent errors are due to a dependency

between syntactically higher and lower constituents and can result in

redundant or even misleading feedback. Contingent errors require that the

syntactically higher constituent be reported first. The following section will

discuss the process in detail.

4.4.2 Contingent Errors

The Error Priority Queue also ranks grammar constraints according

to the position of the error constituents in the syntactic tree. For example,

grammar constraints, 1. - 5. of group IV. on p. 131, refer to prepositions and

their noun complements. Here it is especially important to report the error on

the preposition before an error with its noun complement to avoid redundant

and misleading feedback due to the contingency between these errors. To

Page 27: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 133

illustrate the problem with contingent errors, consider example (7a) as noted

by Schwind [1990a].

(7a) * Ich warte für dir.

acc prep. dat pronoun

(b) Ich warte auf dich.

acc prep. acc pronoun

I am waiting for you.

In example (7a), the student made two errors: Für is an accusative

preposition which requires the accusative pronoun dich. In addition, warten

subcategorizes for auf and not for für. The lurking cause here is likely

language interference. The English verb to wait subcategorizes for for, thus

the misuse of für. For the two errors flagged in example (7a) the student would

receive the feedback This is the wrong preposition and This is the wrong case

of the pronoun. However, the case of the pronoun depends solely on the

preposition. In example (7a) the pronoun is in the incorrect case for either

preposition, für and auf, both of which take the accusative. However, feedback

on contingent errors can even mislead the student. This applies to instances

where two prepositions require different cases, as given in example (8a):

(8a) * Sie denkt von ihn.

dat prep. acc pronoun

(b) Sie denkt an ihn.

acc prep. acc pronoun

She is thinking of him.

Example (8a) illustrates that denken subcategorizes for the accusative

preposition an, while von is a dative preposition requiring the dative pronoun

ihm. As in example (7a), the feedback to the student would be This is the

wrong preposition and This is the wrong case of the pronoun. However, the

pronoun ihn is not incorrect if the student changes the preposition von to an

because an requires the accusative pronoun ihn. Depending on the order of

Page 28: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 134

the feedback, the student might end up changing ihn to ihm and von to an and

wind up utterly confused because the correct case of the pronoun had been

flagged as an error in the original sentence.21

The errors in examples (7a) and (8a) are due to a dependency between

the verb, the preposition, and the noun phrase. Consider the tree structure

given in Figure 4.5 which illustrates the contingent error in example (7a).

In German, certain verbs subcategorize for particular prepositions.

The preposition in turn assigns case to its noun phrase complement. As a

result of this dependent relationship, the error in the pronoun ihn of example

(7a) is correctly flagged by the system from a purely logical point of view.

However, from a pedagogical perspective reporting the error in the pronoun is

redundant and even misleading. Ideally, the system would report only the

error in the preposition, the syntactically higher level error, because the error

in the noun phrase, the lower constituent in the syntactic tree, is dependent

on the choice of the preposition. The error in the noun phrase could still be

recorded in the Student Model. Once recorded in the student model the

information can be used for assessing and remediating the student.

21. Holland [1994] lists further contingent errors with verb phrases. Holland refers to thesekinds of errors as redundant errors. In this thesis, the term contingent has been chosen since,although the feedback might be redundant, the errors themselves are not. They are caused bya dependency between higher and lower constituents in the syntactic tree. Consider example (1a) which illustrates a contingent error with a verb phrase:

(1a) * Wir gehen in die Berges.(b) Wir gehen in die Berge.

We are going to the mountains.Holland’s system flags the following three errors for the sentence given in (1a):

Determiner-Noun Error: die, BergesPreposition-Noun Error: in, die BergesVerb-Preposition Error: gehen, in die Berges

The first error results because the genitive singular Berges does not match the determiner die;the second, because Berges is genitive, but the preposition in requires dative or accusative; thethird because Berges is not accusative, as required by the verb gehen. In Holland’s system, allthree errors are reported while, form a pedagogical point of view, reporting solely the errorwith the noun Berges would be sufficient. See Holland [1994], pp. 246-248.

Page 29: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 135

An example of a system which reports the redundant error of a

syntactically lower constituent is described by Schwind [1990a] who reports

how students evaluated such ILTS:

“..., frequently a wrong preposition for a verb was chosen andthen the wrong case for that preposition. Our system thenexplained both errors but some students felt that they did notneed to know the correct case of a preposition which they werenot going to use anyway. Clearly it would be possible not tocheck the agreement of the case of a noun phrase and thepreposition when the preposition is already wrong, but thismeans that the student is left with a misconception about agrammar rule which has been detected by the system.”22

As Schwind states, the student should not be left with misconception

in the case of contingent errors. However, as human tutors, we do not

inundate students with error explanations, particularly when an error is not

immediately pertinent. A human tutor might make a mental note to monitor

22. Schwind [1990a], p. 577.

Figure 4.5: Contingent Errors

VPNP

*S

V PP

P NP

case

Sie

denkt

von ihn

Page 30: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 136

the lower-level agreement construct, but would focus on the more important

error.

Only a few scholars [Holland 1994, Schwind 1990a, 1995] have

addressed the problems of contingent errors. Holland [1994] suggests several

improvements to her own system:

1. “... omitting error classification entirely and showing studentsonly the locations of errors in a sentence, which are then stored instudents’ performance records. However, this means losing therelevant information about error classification in the clear cases.

2. “... surpressing the outermost flags in a nested series and makingit obligatory to address the innermost flags. The rationale is thathaving students first correct lower level errors; typically the lessserious gender disagreements gives higher level grammaticalproblems a chance to present themselves. This strategy makessense for a flag series in which the outermost flag is perfectlyredundant, however, this strategy runs the risk ofmisrepresenting student error for certain other flagconfigurations.”23

The suggestions made by Holland [1994] are unsatisfactory.

The first solution is a step backward in the development of ILTSs. For

example, Nagata [1991, 1995, 1996] studied the efficacy of simply underlining

errors as opposed to providing the student with detailed feedback about the

nature of the error and she found that the latter is indeed more effective and

more appreciated by students.

The strategy of addressing syntactically lower level (innermost) errors

first, given in 2., is strongly motivated by the need to account for ambiguous

errors, Holland [1994] and Schwind [1990a, 1995], as discussed in Chapter 2.

However, as Holland [1994] states “the strategy runs the risk of

misrepresenting student errors for certain other flag configurations”24 which

23. Holland [1994], p. 249. For the purpose of discussion, the suggestions made by Holland [1994] have been reordered.Also, Holland [1994] list a third suggestion which, however, is very general and provides norecommendation for actual implementation.

Page 31: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 137

are precisely those contingent errors illustrated in examples (7a) - (8a).25 For

these, it is the syntactically higher level (outermost) errors, such as the choice

of a wrong preposition which need to be reported first. The problem with both

Holland’s [1994] and Schwind’s [1990a, 1995] approaches is that they attempt

to handle ambiguous and contingent errors with basically the same technique.

The assumption is that one can process either higher level errors or lower

level errors first. Yet as with ambiguous errors, any system which cannot deal

with contingent errors effectively runs the risk of producing misleading

feedback, and further misrepresents the student’s knowledge of the

grammatical structures involved. As a result, the Student Model will not be

accurate.

The analysis described can successfully address ambiguous and

contingent errors because a separate technique for each class of errors is

employed. Ambiguous errors are handled by feature percolation and finer-

grained feature values specified in phrase descriptors, as discussed in Chapter

2. In contrast, contingent errors are resolved by the Filtering Module. The

Filtering Module sifts incoming errors in such a way that the syntactically

lower level error is not reported to the student. This is achieved by the Error

Priority Queue which specifies that errors of syntactically higher constituents

are reported first. For example, if the student chooses an incorrect preposition,

the error will be reported before an error with its noun complement. The error

with the noun complement will, however, be recorded in the Student Model.

Once the error with the preposition has been corrected successfully, the error,

if still present, with the noun complement will be addressed. The analysis

24. Holland [1994], p. 249. 25. The same shortcoming is found in Schwind’s [1995] system. As discussed in Chapter 2,Schwind [1995] also addresses lower-level errors first to handle ambiguous errors.

Page 32: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 138

ensures that the system does not misrepresent the student’s knowledge, but

at the same time, provides the student with relevant and accurate feedback.

The grammar constraints of the Error Priority Queue, can always be

reordered to reflect the pedagogy of a particular language instructor. However,

the ranking of the grammar constraints referring to contingent errors, as with

grammatical constraints 1. - 5., which correspond to the choice of a particular

preposition and its noun complement, constitute a fixed subgroup of grammar

constraints. While the whole group can be assigned priority over other

grammar constraints, the order of individual grammar constraints within the

group is fixed to ensure that contingent errors are addressed in a

pedagogically sound way.

4.5 Summary

The Analysis Module, the Student Model, and the Filtering Module

are essential elements of the approach presented in this dissertation. They

incorporate pedagogical and psychological aspects of language learning

essential to an ILTS that models a language instructor in evaluating and

responding to student input.

The strength of the Analysis Module lies in its ability to generate

instructional feedback of different granularity. Granularity is implemented

unidimensionally arranging grammar constraints and instructional feedback

from a coarse to fine grain size.

The Student Model classifies students according to three performance

levels: the novice, the intermediate, and the expert. 79 grammar constraints

are maintained by the model, reflecting the grammatical constraints to be

Page 33: CHAPTER 4 Pedagogically- Informed Feedback - SFU.caheift/trude/chapter4.pdf · CHAPTER 4 Pedagogically-Informed Feedback 4.1 Introduction Analyzing student input in an ILTS requires

Designed Intelligence: A Language Teacher Model 139

met in an introductory course for German. A student can be at different levels

for any given grammar constraint reflecting performance on each particular

grammar skill. As a consequence, a single error will not drastically change

the assessment of the student. The information stored can be used for

tailoring instructional feedback suited to the level of the learner and also for

assessment and remediation.

The Filtering Module is responsible for handling multiple errors. The

approach to reporting multiple errors is pedagogically motivated, also taking

into account contingent errors. The system displays one response at a time

and the ranking of feedback responses is established by an Error Priority

Queue. The Error Priority Queue is, however, flexible and can be easily

adjusted to reflect the focus of a particular grammatical constraint in a given

exercise. The final result of the analysis is a single error message tailored to

the level of the learner and pedagogical considerations.