chapter 4 pedagogically- informed feedback - sfu.caheift/trude/chapter4.pdf · chapter 4...
TRANSCRIPT
Designed Intelligence: A Language Teacher Model 107
CHAPTER 4 Pedagogically-Informed Feedback
4.1 Introduction
Analyzing student input in an ILTS requires linguistic and pedagogic
knowledge. The task of the grammar and the parser is to generate phrase
descriptors which provide the linguistic analysis of students’ input. The
phrase descriptors, however, need to be processed further to reflect principles
of language learning. The Analysis Module, the Student Model, and the
Filtering Module, respectively, constitute the pedagogic components carrying
out the analysis described.
The Licensing Module, discussed in Chapter 3, chooses the desired
parse. All phrase descriptors associated with the selected parse are then sent
to the Analysis Module, and the information obtained is subsequently passed
to the Student Model and Filtering Module. Figure 4.1 illustrates the relation
between the Analysis Module, the Student Model, and the Filtering Module.1
Designed Intelligence: A Language Teacher Model 108
The Analysis Module takes all incoming phrase descriptors and
generates possible instructional feedback at different levels of granularity
correlated with student expertise. In ILTSs, granularity is particularly
important to framing responses to learners’ errors. Inexperienced students
1. Of these three modules, only the Analysis Module is language-dependent but to a trivialextent: the feedback messages in the current implementation happen to be in English.
Figure 4.1: Pedagogic Modules
Analysis Module
and produces instructional feedbackof increasing abstraction
All Phrase Descriptors
Student Model
keeps learner model updatesand decides on the student level
Filtering Module
decides on the order of instructional feedback
Error-contingent Feedback Suited to Learner Expertise
returns student model updates
from the Selected Parse
Designed Intelligence: A Language Teacher Model 109
require detailed instruction while experienced students benefit best from
higher level reminders and explanations [LaReau & Vockell 1989].
For instance, in example (1a) the student made an error with the
determiner einen of the prepositional phrase. Von is a dative preposition and
Urlaub is a masculine noun. The correct article is einem.
(1a) *Sie träumt von einen Urlaub.
(b) Sie träumt von einem Urlaub.
She is dreaming of a vacation.
In a typical student-teacher interaction, feedback depends on the
students’ previous performance history. For instance, for the error in example
(1a), a learner who generally has mastered the concept of dative assigning
prepositions might merely receive a hint indicating that a mistake occurred
within the prepositional phrase. For the learner who generally knows the
grammatical rule but still needs practice in its application, the feedback is
less general: in addition to location, the teacher might point out the type of the
error (case). For the novice learner, the feedback would be as specific as
possible. In addition to the location and type of the error, information as to the
exact source of the error, that is, the precise grammatical rule that has been
violated (dative case), would be provided.
Note, however, that even the most specific report aimed at the novice
learner refrains from revealing the correct answer. For the error in example
(1a), the beginner student is still required to decide on the correct inflection of
a masculine indefinite article in the dative case. The pedagogical principle
underlying this design is guided discovery learning. According to Elsom-Cook
[1988], guided discovery takes the student along a continuum from heavily
structured, tutor-directed learning to where the tutor plays less and less of a
role. Applied to feedback, the pedagogy scales messages on a continuum from
least-to-most specific guiding the student towards the correct answer.
Designed Intelligence: A Language Teacher Model 110
Burton and Brown [1982] state that hinting at the source of an error
supports the development of students’ self-regulation, a prerequisite for
guided discovery learning. A system can lead the learner towards the correct
answer rather than simply supplying it, thus assisting the students in finding
the source of an error themselves. The ability to discover the source of an
error, however, strongly correlates with the expertise of the student: the more
expert the learner the less explicit feedback is necessary [Fischer & Mandl
1988].
Granularity captures this pedagogically intuitive concept: the
Analysis Module generates three categories of instructional feedback
corresponding to three learning levels: expert, intermediate, and novice. The
responses are scaled from least-to-most specific, respectively. Provided with
three categories of feedback by the Analysis Module, the Student Model
selects the error response suited to the student’s expertise according to a
record of the student’s previous performance history on a particular
grammatical phenomenon.
The Student Model tracks 79 grammar constraints, corresponding to
the grammatical concepts being monitored in an introductory course for
German.2 For each of the grammar constraints, the Student Model keeps a
counter which, at any given instance in the evaluation process, falls in the
range of one of the three learner levels. If a grammatical constraint has been
met, the counter is decremented. If the constraint has not been met, the
counter is incremented and a feedback message suited to the performance
level of the learner is selected.
2. The grammar constraints have been chosen according to Wie geht’s, the course bookcommonly used in an introductory course for German [see Sevin, Sevin, & Bean 1991].
Designed Intelligence: A Language Teacher Model 111
An example of a grammar constraint is pp-dat which records the
student’s performance on dative assigning prepositions. Each student is
initially assessed as an intermediate, which has been chosen as a reasonable
default level.3 A learner who violates the constraint on dative prepositions
will, at first, obtain the feedback message for the intermediate. If the student
commits the same error in subsequent exercises, s/he will soon be assessed a
novice. At this point, the Student Model will select the more detailed feedback
message suited to the beginner. However, each time the student applies the
grammatical constraint correctly, the Student Model records the success. After
demonstrating proficiency, the student will again be assessed as intermediate,
or, even expert. Maintaining a large number of grammatical constraints
allows for a very detailed portrait of an individual student’s language
competence over a wide-range of grammatical phenomena.
The Filtering Module decides on the order in which instructional
feedback is displayed. Feedback is provided for one error at a time to avoid
overloading the student with extensive error reports. According to van der
Linden [1993], displaying more than one feedback message at a time makes
the correction process too complex for the student.
The ordering of the instructional feedback is decided by an Error
Priority Queue which ranks the 79 grammar constraints maintained by the
Student Model. The ordering of the grammar constraints and ultimately
feedback is, however, adjustable. Depending on the pedagogy of a particular
language instructor, they can be reordered to reflect the focus of a particular
exercise. In addition, errors which are not relevant to a particular exercise
3. The intention of the system is primarily to augment but not to replace classroominstruction. The system is therefore a practice tool for the student who previously has beenexposed to a particular grammatical concept of German. It is thus safe to assume that thelearner will not be a complete beginner nor an expert on the grammatical phenomenapracticed.
Designed Intelligence: A Language Teacher Model 112
need not be reported at all, although the error itself still is recorded in the
Student Model.
The ordering of the grammar constraints also accommodates
contingent errors, a special class of multiple errors, and thus avoids
misleading or redundant feedback. For instance, in example (2a) the student
made a mistake with the prepositional phrase. The verb denken
subcategorizes for the preposition an which requires the accusative pronoun
dich, while von is a dative preposition requiring the dative pronoun dir. If we
ignore the dependency of the errors with the preposition and the pronoun, the
feedback to the student would be “von” is the wrong preposition and This is the
wrong case of the pronoun “dich”. However, the pronoun dich is not incorrect if
the student changes the preposition von to an because an requires the
accusative pronoun dich. Depending on the order of the feedback, the student
might end up changing von to an and dich to dir and wind up utterly
confused. S/he might ultimately fail to find the correct case of the pronoun
because it had been flagged as an error in the original sentence.
(2a) * Ich denke von dich.
dat prep. acc pronoun
(b) Ich denke an dich.
acc prep. acc pronoun
I am thinking of you.
The error in the pronoun is correctly flagged by the system from a
purely logical point of view. However, from a pedagogical perspective reporting
the error in the pronoun dich is redundant and possibly misleading. In such
instances, only the error in the preposition von is reported although the error
in the pronoun is recorded in the Student Model. Thus, the system, while
silently recording all errors, does not necessarily comment on all of them.
Designed Intelligence: A Language Teacher Model 113
The final result of the three modules is error-contingent feedback
suited to students’ expertise. The following sections will discuss the Analysis
Module, the Student Model, and the Filtering Module in detail.
4.2 The Analysis Module
The function of the Analysis Module is to take all incoming phrase
descriptors as input and return student model updates and potential error
feedback at different levels of granularity for each phrase descriptor.
Granularity has been previously applied to an Intelligent Tutoring System for
LISP programming [McCalla & Greer 1994, Greer & McCalla 1989]. In their
system SCENT, Greer and McCalla implemented granularity to “recognize
the strategies novice students employ when they solve simple recursive LISP
programming problems.”4 In this analysis, however, granularity is used in
framing responses to learners’ errors: inexperienced students obtain detailed
instruction while experienced students receive higher level reminders and
explanations. For example, consider the ungrammatical sentence in (3a).
(3a) *Der Mann dankt dem Frau.
(b) Der Mann dankt der Frau.
The man thanks the woman.
In example (3a), the student has provided the wrong determiner for
the indirect object. For the error dem Frau, the system generates feedback of
increasing abstraction that the instruction system can use when interacting
with the student. The level of the learner, either expert, intermediate, or
novice according to the current state of the Student Model, determines the
4. Greer & McCalla [1989], p. 478.
Designed Intelligence: A Language Teacher Model 114
particular feedback displayed. The responses, given in (4a) - (c) correspond to
the three learner levels for the error in example (3a), respectively:
(4a) There is a mistake with the indirect object.
(b) There is a mistake in gender with the indirect object.
(c) This is not the correct article for the indirect object. The noun isfeminine.
For the expert, the feedback is most general, providing a hint to where
in the sentence the error occurred (indirect object). For the intermediate
learner, the feedback is more detailed, providing additional information on the
type of error (gender). For the beginner, the feedback is the most precise. It not
only pinpoints the location and type of the error but also refers to the exact
source of the error (feminine noun).
Figure 4.2 displays the partial Granularity Hierarchy for constraints
in feature matching.5 The Granularity Hierarchy is a representation of the
instructions given to the student correlated with the grammatical constraints
monitored by a phrase descriptor. Each term in a phrase descriptor
corresponds to a level in the Granularity Hierarchy. For example, for the
indirect object of the sentence *Der Mann dankt dem Frau, given in (3a) on p.
113, the grammar and the parser will generate the phrase descriptor
[main_clause [vp_indirobj [fem error]]]. The top node of the Hierarchy
specifies in which kind of clause the error occurred. The phrase descriptor
indicates that a mistake was made in a main-clause. The next level in the
Granularity Hierarchy lists possible errors in each clause type. As indicated
by the phrase descriptor, the mistake refers to the indirect object. An even
finer-grained constraint specification is found in the next lower level of the
Granularity Hierarchy. For instance, an indirect object can be incorrectly
5. The hierarchy here is simplified for the purpose of illustration. The lowest nodes in thehierarchy split into further nodes as illustrated with gender.
Designed Intelligence: A Language Teacher Model 115
inflected for either case, number, or gender. The phrase descriptor specifies
that an error in gender occurred, specifically with a feminine noun which
corresponds to the lowest level in the Granularity Hierarchy.
Each node in the Granularity Hierarchy corresponds to a level of
specificity of a feedback message. Granularity works along one dimension,
namely, abstraction: travelling downward through the Hierarchy, nodes lower
in the Hierarchy will have associated with them messages of increasing
specificity. Generally, the more experienced the student the coarser-grained
the message and the higher the node.
The Analysis Module is implemented in DATR [Evans and Gazdar
1990], a language designed for pattern-matching and representing multiple
inheritance. Nodes in DATR are represented by the name of the node followed
Figure 4.2: Granularity Hierarchy for Constraints in Feature Matching
Main Clause
Coordinate Clause
Subordinate Clause
Subject-Verb Subject Direct Object Indirect Object Prep. Phrase Verb Phrase
NumberPerson
Semantic Constraints
GenderNumber
Case
Semantic Constraints
Aux/Pastpp.
Modal/InfinitiveVerb Inflection
FeminineMasculine
Neutersein/habenSingular
Plural
Specificity
etc.etc. etc.
Decreasing
Preposition
Designed Intelligence: A Language Teacher Model 116
by paths and their values. For example, the node given in Figure 4.3
corresponds to the phrase descriptor that records the position of a finite verb
in a subordinate clause.
The paths in a node definition represent descriptions of grammatical
constraints monitored by phrase descriptors. The matching algorithm of
DATR selects the longest path that matches left to right. Each path in a node
is associated with atoms on the right of ‘==’. For example, if there has been an
error in word order in a subordinate clause, the parse will contain the phrase
descriptor [sub_clause [position_subclause [finite error]]]. This will match the
path <sub_clause position_subclause finite error> which specifies four atoms
for each of three groups. Each group represents a learning level. The three
learning levels considered are: expert, intermediate, and novice6, given in
Figure 4.3. 7
6. Finer distinctions can be made by looking at the actual error count either for the purposesof evaluation and remediation; however, three levels are sufficient to distinguish among thefeedback messages encoded.
<sub_clause position_subclause finite error> ==
('possubclfin' '3' 'true' 'The verb in the subordinate clause is not in the correct position.'
'possubclfin' '2' 'true' 'The finite verb in the subordinate clause is not in the correct
position.'
'possubclfin' '1' 'true' 'The finite verb in the subordinate clause is not in the correct
position. It has to be the last element of the sentence.')
<sub_clause position_subclause finite correct> == ('possubclfin' '1' 'false' ''
'possubclfin' '1' 'false' ''
'possubclfin' '1' 'false' '')
<sub_clause position_subclause finite absent> == ('').
Figure 4.3: DATR Code Listing for a Finite Verb in a Subordinate Clause
Designed Intelligence: A Language Teacher Model 117
The first atom in each group specifies the grammar constraint as
described by the incoming phrase descriptor. For example, the grammar
constraint possubclfin represents finite verb position in a subordinate clause.
The second atom specifies a value that is decremented or incremented
depending on whether a grammatical constraint has been met or not,
respectively.
The Boolean values, as the third atom, indicate whether it is an
increment or decrement: true for increment, false for decrement.
Finally, the fourth atom specifies a feedback message. For example,
for the path <sub_clause position_subclause finite error> provided earlier, the
specificity of each response corresponds to the grammar constraints in the
Granularity Hierarchy which frames responses to constraints on verb
position, given in Figure 4.4.
The three instructional responses associated with each node
correspond to the three learner levels. The feedback for the beginner student
reflects the lowest node in the Granularity Hierarchy, while for the
intermediate and expert student the message will refer to nodes higher in the
hierarchy. In a typical student-teacher interaction, as the student becomes
more proficient in the use of a grammatical construction, error feedback
becomes less specific.
If there has been no error in the student’s input, the phrase descriptor
is [sub_clause [position_subclause [finite correct]]] and no message is
associated with the name of the grammar constraint. However, the first three
atoms, the grammar constraint, the decrement, and the Boolean value are
7. A phrase descriptor that contains the value absent is ignored by the system. Thus, the listto the right of ‘==’ is empty.
Designed Intelligence: A Language Teacher Model 118
still specified, given in Figure 4.3 on p. 116. They contribute to a Student
Model update to record the success.
Finally, if the phrase descriptor is [sub_clause [position_subclause
[finite absent]]], then the grammatical phenomenon was not present in the
student input. As a result, the phrase descriptor is ignored by the system,
indicated by an empty list in Figure 4.3 on p. 116.
The final output of the Analysis Module is a student model update
and a list of possible instructional responses from a coarse to fine grain size
for each incoming phrase descriptor. The more expert the student with a
particular grammatical construction, the more general the feedback.
Figure 4.4: Granularity Hierarchy for Constraints in Linear Precedence
Main Clause
Coordinate ClauseSubordinate Clause
DecreasingSpecificity
Verb Position
Finite
Past Participle
Infinitive
Second Position Final Position Second-to-last Position
Designed Intelligence: A Language Teacher Model 119
The modular design of the system also allows the encoding of
instructional feedback suited to the pedagogy of a particular teacher. For
instance, an instructor who preferred to avoid the term finite verb in beginner
messages could substitute the simpler verb.8
The list of alternative instructional responses from the Analysis
Module has to be processed further before being displayed to the student. The
Student Model, which picks a response suited to the level of the student, will
be discussed in the following section.
4.3 The Student Model
Individualization of the learning process is one of the features of a
student-teacher interaction that distinguishes it from gross mainstreaming of
students characteristic of workbooks. Students learn at their own pace and
often, work for their own purposes. Learners also vary with respect to prior
language experience, aptitude, and/or learning styles and strategies [Oxford
1995]. According to the Individual Differences Theory as described by Oxford
[1995], if learners learn differently then they likely benefit from
individualized instruction. An ILTS can adapt itself to different learner
needs.9 The requirement, however, is that the system incorporates and
8. There is an argument to be made for softening technical terms in the intermediate andnovice categories. Rather than implementing a pseudo-vocabulary, however, a system canunderline or highlight the error and provide hot links to information and examples on thelinguistic terminology. The advantage of using precise linguistic terminology for all learnerlevels might result in students not only learning the grammatical concepts but the properterminology at the same time.9. In its strict definition, ILTSs which do not implement a student model are ICALL systemsrather than ILTSs. However, in this thesis all systems which make use of Natural LanguageProcessing are referred to as ILTSs.
Designed Intelligence: A Language Teacher Model 120
updates knowledge about the learner. Student modelling provides the key to
individualized knowledge-based instruction [McCalla & Greer 1992].
Student modelling has not been a strong focus of parser-based ILTSs,
likely because the challenging task of representing the domain knowledge in
ILTSs is still largely incomplete [Holland & Kaplan 1995b].10 If the grammar
is not accurate and complete, even a precise student model cannot
compensate. For instance, Holland [1994] states that a system which does not
flag ambiguous and contingent errors accurately will obscure a student
model. A further possible reason is that many significant differences in
language learning styles (situational, aural, visual, etc.) are precluded from
consideration in text-based ILTSs.
ILTSs which do have a student model primarily concentrate on
subject matter performance. Modelling students’ surface errors assists in
individualizing the language learning process and “is sufficient to model the
student to the level of detail necessary for the teaching decisions we are able
to make.”11 The technique aids in teaching the required skills and
remediating the student.12
10. According to Holland and Kaplan [1995b] there are two trends with student modelling insystems for language learning. The North American focus lies on the NLP module, the domainknowledge. Due to the complexity of the NLP component, these systems implement studentmodels only to the extent of analyzing surface errors. In contrast, European systemsconcentrate on more sophisticated student models by hypothesizing causes of errors (seeChanier et al., [1992]). However, they “opt for narrow NLP bandwidth”. Holland & Kaplan[1995b], p. 364. 11. Elsom-Cook [1993], p. 238.12. The modelling techniques employed in language learning are distinct from those appliedto procedural tasks. In procedural tasks, the modelling techniques presume that the studentcan learn the skill in terms of a number of formal steps. Yet although languages are rule-governed systems and we can represent linguistic ability in terms of formal step-by-step rules,we do not produce language by consciously following them [Bailin 1990]. As a result, themodelling techniques in language learning primarily diagnose the sources of errors ratherthan model strategies which the student used in solving a particular problem.
Designed Intelligence: A Language Teacher Model 121
McCalla [1992] makes a distinction between implicit and explicit
student modelling which is particularly useful in classifying the student
models in ILTSs.
An implicit student model is static, in the sense that the Student
Model is reflected in the design decisions inherent to the system and derived
from a designer’s point of view. For instance, in an ILTS the native language
of the learner can be encoded as a bug model and ultimately used to diagnose
errors.
In contrast, an explicit student model is dynamic. It is a
representation of the learner which is used to drive instructional decisions.
For ILTSs, for instance, the student model can assist in guiding the student
through remedial exercises or it can adjust instructional feedback suited to
the level of the learner. In either case, the decisions are based on the previous
performance history of the learner. The following discussion will provide
examples of ILTSs which have implemented implicit and explicit student
models.
4.3.1 Implicit Student Models
Implicit student modelling has been applied to ILTSs to diagnose
errors. For example, in Catt & Hirst’s [1990] system Scripsi the native
language of the student represents the learner model. It is used to model the
learner’s interlanguage. With regard to student modelling, the pitfall of such
an implementation is that it is a static conception. The system’s view of the
learner cannot change across interactions with the system. It has no impact
on instructional decisions and provides only a gross individualization of the
learning process when ideally, a student model is dynamic [Holt et al. 1994].
Designed Intelligence: A Language Teacher Model 122
In a more individualized example, Bull [1994] developed a system
that teaches clitic pronoun placement in European Portuguese. The student
model is based on the system’s and the student’s belief measures, language
learning strategies, and language awareness.
The system’s belief measure is comprised of the proportion of
incorrect/correct uses of the rule; the students provide the data for the
student’s belief measure, being required to state their confidence in their
answer when entering sentences. Learners also identify their preferred
learning strategies when using the program. According to Bull [1994],
language awareness is achieved by allowing the student access to all
information held in the system. None of the information, however, is used to
drive the instructional process. In addition, a number of studies have shown
that students tend to not take advantage of the option to access additional
information. For example, Cobb & Stevens [1996] found that in their reading
program learners' use of self-accessible help was virtually non-existent, in
spite of their previously having tried it in a practice session, and also having
doubled the success rate as compared to either a no help or dictionary help
option in the practice session.
4.3.2 Explicit Student Models
In developing an explicit student model one typically starts by
making some initial assumptions based on pretests or stereotypical
postulations about the learner. For example, initially every student could be
assessed as an intermediate. During the instructional process, the student
model adjusts to student’s behaviour moving to a novice or expert profile, as
appropriate. This technique is used in explicit student models to make
instructional decisions.
Designed Intelligence: A Language Teacher Model 123
Explicit student modelling has been used in a number of ILTSs,
primarily in the form of tracking. Tracking can be as simple as calculating
percentages of correct answers or more sophisticatedly, identifying particular
errors which occurred in the student’s input. The information is then used to
alter the instructional process, either in the form of further language tasks or
feedback.
Explicit student modelling is found in the system The Fawlty Article
Tutor [Kurup, Greer & McCalla 1992] which teaches correct article use in
English. The system presents the student with scenarios whereby the student
must select the correct article form and the appropriate rule. The tutor keeps
an error count and selects the scenarios on the basis of the performance of the
student; thus the path through the program is individualized by altering the
instructional process according to prior performance of the student.
Bailin [1988, 1990] in his system Verbcon/Diagnosis also employs the
tracking method. Diagnosis provides practice in using English verb forms in
written texts. All verbs are presented in their infinitival form challenging the
student to provide the appropriate verb form. The system tracks the most
frequent error occurrence and the context in which the error occurred. The
information is used to provide informative feedback based on contrasting
correct and ungrammatical uses of tenses. In addition, Diagnosis suggests
exercises to help with the remediation process.
The student model under the analysis of this dissertation is based on
students’ prior performance and it ultimately has two main functions. The
first is to select instructional feedback suited to learner expertise and second
to use the student’s performance for assessment and remediation.13 The
following section will discuss the technique employed.
Designed Intelligence: A Language Teacher Model 124
4.3.3 Feedback Suited to Learner Expertise
To select instructional feedback suited to the level of the learner, the
Student Model keeps track of 79 grammar constraints. These grammar
constraints correspond to the grammatical concepts to be monitored in an
introductory course for German. The grammar constraints are split among
the three clause types, main, subordinate, and coordinate, each containing 25
nodes. In addition, there are two grammar constraints monitoring noun
phrases practiced in isolation, one for verb-initial position in main-clauses,
and one for the entire sentence.
The Student Model assumes three types of learners: the novice, the
intermediate, and the expert. Each student level is represented by a range of
values14:
novice: 20 ≤ X ≤ 30
intermediate: 10 ≤ X < 20
expert: 0 ≤ X < 10
Initially, the learner is assessed with the value 15 for each grammar
constraint, representing the mean score of the intermediate learner. The
values are used by the Student Model to decide on the specificity of the
instructional feedback being displayed. The intermediate learner has been
chosen as a reasonable default. While the messages might be initially too
overspecified for the expert and too underspecified for the novice, they will
quickly adjust to the actual learner level. Pre-testing could be used to
individualize the model from the outset.
13. The system does not yet contain any pre-defined exercises, but in an extended versionerror counts held in the Student Model will be available to the learner. They form the basis forbranching decisions and remediation recommendations.14. The ranges chosen for each student level roughly correspond to the average size of agrammar exercise unit. Since the default decrement for the error count is 1, successfulcompletion of 10-15 exercises will record a change in student level.
Designed Intelligence: A Language Teacher Model 125
For each learner level, the Student Model receives four atoms from
the Analysis Module, described previously: the name of a grammar
constraint, an increment/decrement, a Boolean value and in case of the latter
being true, a feedback message.
The counter of the grammar constraint determines which of the three
learner responses is selected. If the number is 20 or greater, the system
displays the message suited for the beginner. If it is less than 10, the system
treats the learner as expert, and any number in between points to an
intermediate. The counter is bounded at 0 and 30.
Once the Student Model has selected the feedback message to be
displayed to the learner, the counter is incremented by the incoming weighted
constant. For the expert, the increment is generally 3, for the intermediate 2,
and for the novice 1. The consequence of this sequence is that in case of errors,
the student will switch quickly from expert to intermediate, and somewhat
less quickly from intermediate to novice. As a result, if a student is not an
expert with a given grammatical construction after all, the feedback will
quickly become more informative from the expert to the intermediate level.
The transition from the intermediate to the beginner level is slower since the
feedback for the former already points at a specific error type. For instance, for
the grammar constraint possubclfin, given on page 116 the feedback for the
intermediate level specifies The finite verb in the subordinate clause is not in
the correct position.15
15. Ultimately, it might be desirable to incorporate knowledge of the student’s pastperformance. For example, at the end of each practice session the current error count for aparticular grammar constraint could be averaged with a historical count representing the lastN sessions. By considering a historical error count in determining student level, momentarylapses in performance would be balanced by previous performance.
Designed Intelligence: A Language Teacher Model 126
In the case of a correct response, as indicated by the Boolean value
false received from the Analysis Module, the constant of the grammar
constraint is subtracted from the counter. The decrement for all grammar
constraints is 1. Thus, the transition from novice to intermediate to expert is
gradual. The result of assigning a small decrement at all learner levels is that
any student has to apply a correct construction many times before the
feedback becomes sparse.
The Student Model presented has a number of advantages. It takes
into account students’ past performance, and by adjusting the value to be
incremented or decremented, it is adaptable to a particular grammatical
constraint in an exercise or the pedagogy of a particular instructor. For
example, a language instructor might rate some errors more salient than
others in a given exercise. In such an instance, the increment/decrement of
some grammar constraints can be tuned to change their sensitivity.
Its main strength, however, lies in the fact that a single errorful
sentence will not drastically change the overall assessment of the student.
The phrase descriptors collect errors that indicate precisely the grammatical
context of the mistake. This enables the Analysis Module to create grammar
constraints which reflect very specific grammatical concepts that are
maintained in the Student Model. The consequence is that a student can be at
a different level for any given grammar constraint reflecting the performance
of each particular grammar skill. This is desirable in a language teaching
environment because as a student progresses through a language course a
single measure is not sufficient to capture the knowledge of the learner and to
distinguish among learners. The Student Model described allows the student
to be geared toward error-contingent and individualized remediation.
Designed Intelligence: A Language Teacher Model 127
The Student Model can also be used for assessment and remediation.
A branching program can be implemented where students’ tasks are
determined by their prior performance. The grammar constraints can be
weighted for a particular set of exercises, so as to be especially sensitive to
salient errors. In an authoring program, these settings could be adjusted by
the language instructor.
Finally, the data obtained can be used for further research in
improving the overall performance of the system, and might prove useful in
providing objective measures of student performance.
The final step in analyzing students’ input is handled by the Filtering
Module. The task here is to accommodate multiple errors. Multiple errors
have been largely overlooked in ILTSs. From a pedagogical perspective,
however, instructional feedback messages need to be prioritized by the system
and displayed one at a time to the student to avoid multiple error reports and
redundant and/or misleading feedback in the case of contingent errors. The
following discussion will focus on this task.
4.4 The Filtering Module
While it is desirable to construct a system capable of detecting and
accurately explaining all errors, it does not follow that the system should
display each and every error detected. In the absence of an error filtering
mechanism, the sheer amount of feedback would overwhelm the student. For
example, in evaluating her own system Schwind [1990a] reports that
“[s]ometimes, however, the explanations were too long, especially when
students accumulated errors.”16 In a language learning exercise, a student
Designed Intelligence: A Language Teacher Model 128
might make more than one error. However, a language instructor typically
skips irrelevant errors, and discusses the remaining ones one at a time.
Example (5a) illustrates an example of multiple errors.
(5a) * Heute meine Kindern haben gespeilt mit der Ball.
(b) Heute haben meine Kinder mit dem Ball gespielt.
Today my children were playing with the ball.
In example (5a) the student made the following five errors:
1. word order: the finite verb haben needs to be in second position
2. word order: the nonfinite verb gespielt needs to be in final position
3. spelling error with the verb spielen
4. wrong plural inflection for the subject Kinder
5. wrong case for the dative determiner dem
From a pedagogical and also motivational point of view, a system
should not overwhelm a student with instructional feedback referring to more
than one error at a time. Schwind’s [1990a] solution to this problem is that
multiple errors should be avoided from the outset. She suggests that sentence
construction exercises should focus on specific grammatical phenomena such
as prepositions or verb cases [see also Kenning & Kenning 1990].
While Schwind’s approach is probably inherent in many ILTSs17,
limiting the teaching domain is only a partial solution. Even a basic sentence
in German, as illustrated in example (5a), requires a number of rules and
knowledge about the case system, prepositions, word order, etc.18
Little research has been done in Computer-Assisted Language
Learning regarding the volume of feedback for different kinds of learners at
16. Schwind [1990a], p. 577.17. ILTSs concentrate on sublanguages to also achieve a higher degree of accuracy. [Levin &Evans 1995].18. Holland [1994], in her system BRIDGE, displays only one error at a time, and permitsinstructors to divide errors into primary, which are automatically displayed, and secondary,which are displayed only at the student’s request.
Designed Intelligence: A Language Teacher Model 129
different stages in their language development. However, van der Linden
[1993] found that “feedback, in order to be consulted, has to be concise and
precise. Long feedback (exceeding three lines) is not read and for that reason
not useful.”19 She further states that displaying more than one feedback
response at a time makes the correction process too complex for the student
[van der Linden 1993].
Van der Linden’s [1993] study makes three final recommendations:
1. feedback needs to be accurate in order to be of any use to the student,
2. displaying more than one error message at a time is notvery useful because at some point they probably will notbe read, and
3. explanations for a particular error should also be keptshort.
With regard to feedback display, van der Linden’s [1993]
recommendations require a system submodule to sift all incoming errors. The
errors have to be reported one at a time and the error explanations should be
brief. This provides the student with enough information to correct the error,
but not an overwhelming amount, and yet records detailed information within
the student model for assessment and remediation.
The analysis described implements an Error Priority Queue which
ranks student errors so as to display a single feedback message in case of
multiple constraint violations. The ranking of student errors in the Error
Priority Queue is, however, flexible: the grammar constraints can be reordered
to reflect the desired emphasis of a particular exercise. In addition, a language
instructor might choose not to report some errors. In such an instance, some
grammar constraints will display no feedback message at all, although the
19. Van der Linden [1993], p. 65.
Designed Intelligence: A Language Teacher Model 130
error will still be recorded in the Student Model. The following section will
discuss the Error Priority Queue.
4.4.1 The Error Priority Queue
The Student Model maintains grammar constraints and selects
instructional feedback suited to learners’ expertise. In case of multiple errors,
the Error Priority Queue determines the order in which instructional
feedback messages are displayed to the learner. It ranks instructional
feedback with respect to
1. the importance of an error within a given sentence, and
2. the dependency of errors of syntactically lower and higherconstituents.
The Error Priority Queue for the grammar constraints of a main
clause is partially given in (6).20 The names of the grammar constraints
generated and maintained by the Analysis Module and Student Model,
respectively are given in parentheses.
(6) Error Priority Queue
I. Word Order in a Main Clause
1. position of a finite verb in a main-clause (posmainclfin)
2. position of a nonfinite verb in a main-clause (posmainclnonfin)
3. position of a finite verb in initial position (posmainclinitial)
II. Indirect Objects in a Main Clause
1. case of the noun of the indirect object (indirobjnounmaincl)
2. case, number, and gender of the determiner of the indirect object(indirobjmaincl)
III. Conjoined Indirect Objects in a Main-Clause
1. case agreement of conjoined nouns of indirect objects (indirobjconjnounmaincl)
20. A complete list of the grammar constraints implemented in the system is provided in theAppendix.
Designed Intelligence: A Language Teacher Model 131
2. case, number, and gender of conjoined determiners of indirectobjects (indirobjconjmaincl)
IV. Prepositional Phrases in a Main Clause
1. choice of preposition (prepmaincl)
2. case of the noun of a prepositional phrase in the accusative(pp_accnounmaincl)
3. case, number, and gender of the determiner of a prepositionalphrase in the accusative (pp_accmaincl)
4. case of the noun of a prepositional phrase in the dative(pp_datnounmaincl)
5. case, number, and gender of the determiner of a prepositionalphrase in the dative (pp_datmaincl)
Each grammar constraint, given in the Error Priority Queue in (6)
correlates to a node in the Granularity Hierarchy, provided earlier. The
grammar constraints are grouped according to grammatical phenomena. For
example, the group prepositional phrases in a main clause contains all
constraints monitored with prepositional phrases and it corresponds to the
node prep. phrase in the Granularity Hierarchy in Figure 4.2 on p. 115. Each
member of a group in the Error Priority Queue refers to a node lower in the
Granularity Hierarchy: the grammar constraint choice of preposition, for
example, correlates with the node preposition in the Granularity Hierarchy.
The groups in the Error Priority Queue are sorted according to the
importance of an error within a sentence and the members within the group
are sorted according to the dependency of errors of syntactically lower and
higher constituents. If the student made multiple errors, the system ranks
instructional feedback messages according to the order specified and displays
them one at a time.
The Error Priority Queue shown in (6) reflects the default setting for
the importance of an error in a given exercise. For example, grammar
constraints 1. to 3. of the group Word Order in a Main Clause refer to errors in
linear precedence. In the default setting, they are reported first since word
Designed Intelligence: A Language Teacher Model 132
order is one of the fundamental concepts of a language and thus likely to have
high priority in most exercises.
The ordering of the groups of grammar constraints can, however, be
altered to reflect the pedagogy of a particular language instructor. For
example, an instructor might want to centre exercises around dative case
assignment. In such an instance, the grammar constraints can be reordered so
that errors of indirect objects are reported first. In addition, a language
instructor might choose to suppress some errors, so as not to distract the
student from the main task. These would be errors which are not relevant to a
particular exercise. Suppressing certain errors, does not affect their
contribution to the Student Model, on the rationale that behind-the-scenes
information should be as detailed as possible.
The Error Priority Queue also takes into account contingent errors, a
special class of multiple errors. Contingent errors are due to a dependency
between syntactically higher and lower constituents and can result in
redundant or even misleading feedback. Contingent errors require that the
syntactically higher constituent be reported first. The following section will
discuss the process in detail.
4.4.2 Contingent Errors
The Error Priority Queue also ranks grammar constraints according
to the position of the error constituents in the syntactic tree. For example,
grammar constraints, 1. - 5. of group IV. on p. 131, refer to prepositions and
their noun complements. Here it is especially important to report the error on
the preposition before an error with its noun complement to avoid redundant
and misleading feedback due to the contingency between these errors. To
Designed Intelligence: A Language Teacher Model 133
illustrate the problem with contingent errors, consider example (7a) as noted
by Schwind [1990a].
(7a) * Ich warte für dir.
acc prep. dat pronoun
(b) Ich warte auf dich.
acc prep. acc pronoun
I am waiting for you.
In example (7a), the student made two errors: Für is an accusative
preposition which requires the accusative pronoun dich. In addition, warten
subcategorizes for auf and not for für. The lurking cause here is likely
language interference. The English verb to wait subcategorizes for for, thus
the misuse of für. For the two errors flagged in example (7a) the student would
receive the feedback This is the wrong preposition and This is the wrong case
of the pronoun. However, the case of the pronoun depends solely on the
preposition. In example (7a) the pronoun is in the incorrect case for either
preposition, für and auf, both of which take the accusative. However, feedback
on contingent errors can even mislead the student. This applies to instances
where two prepositions require different cases, as given in example (8a):
(8a) * Sie denkt von ihn.
dat prep. acc pronoun
(b) Sie denkt an ihn.
acc prep. acc pronoun
She is thinking of him.
Example (8a) illustrates that denken subcategorizes for the accusative
preposition an, while von is a dative preposition requiring the dative pronoun
ihm. As in example (7a), the feedback to the student would be This is the
wrong preposition and This is the wrong case of the pronoun. However, the
pronoun ihn is not incorrect if the student changes the preposition von to an
because an requires the accusative pronoun ihn. Depending on the order of
Designed Intelligence: A Language Teacher Model 134
the feedback, the student might end up changing ihn to ihm and von to an and
wind up utterly confused because the correct case of the pronoun had been
flagged as an error in the original sentence.21
The errors in examples (7a) and (8a) are due to a dependency between
the verb, the preposition, and the noun phrase. Consider the tree structure
given in Figure 4.5 which illustrates the contingent error in example (7a).
In German, certain verbs subcategorize for particular prepositions.
The preposition in turn assigns case to its noun phrase complement. As a
result of this dependent relationship, the error in the pronoun ihn of example
(7a) is correctly flagged by the system from a purely logical point of view.
However, from a pedagogical perspective reporting the error in the pronoun is
redundant and even misleading. Ideally, the system would report only the
error in the preposition, the syntactically higher level error, because the error
in the noun phrase, the lower constituent in the syntactic tree, is dependent
on the choice of the preposition. The error in the noun phrase could still be
recorded in the Student Model. Once recorded in the student model the
information can be used for assessing and remediating the student.
21. Holland [1994] lists further contingent errors with verb phrases. Holland refers to thesekinds of errors as redundant errors. In this thesis, the term contingent has been chosen since,although the feedback might be redundant, the errors themselves are not. They are caused bya dependency between higher and lower constituents in the syntactic tree. Consider example (1a) which illustrates a contingent error with a verb phrase:
(1a) * Wir gehen in die Berges.(b) Wir gehen in die Berge.
We are going to the mountains.Holland’s system flags the following three errors for the sentence given in (1a):
Determiner-Noun Error: die, BergesPreposition-Noun Error: in, die BergesVerb-Preposition Error: gehen, in die Berges
The first error results because the genitive singular Berges does not match the determiner die;the second, because Berges is genitive, but the preposition in requires dative or accusative; thethird because Berges is not accusative, as required by the verb gehen. In Holland’s system, allthree errors are reported while, form a pedagogical point of view, reporting solely the errorwith the noun Berges would be sufficient. See Holland [1994], pp. 246-248.
Designed Intelligence: A Language Teacher Model 135
An example of a system which reports the redundant error of a
syntactically lower constituent is described by Schwind [1990a] who reports
how students evaluated such ILTS:
“..., frequently a wrong preposition for a verb was chosen andthen the wrong case for that preposition. Our system thenexplained both errors but some students felt that they did notneed to know the correct case of a preposition which they werenot going to use anyway. Clearly it would be possible not tocheck the agreement of the case of a noun phrase and thepreposition when the preposition is already wrong, but thismeans that the student is left with a misconception about agrammar rule which has been detected by the system.”22
As Schwind states, the student should not be left with misconception
in the case of contingent errors. However, as human tutors, we do not
inundate students with error explanations, particularly when an error is not
immediately pertinent. A human tutor might make a mental note to monitor
22. Schwind [1990a], p. 577.
Figure 4.5: Contingent Errors
VPNP
*S
V PP
P NP
case
Sie
denkt
von ihn
Designed Intelligence: A Language Teacher Model 136
the lower-level agreement construct, but would focus on the more important
error.
Only a few scholars [Holland 1994, Schwind 1990a, 1995] have
addressed the problems of contingent errors. Holland [1994] suggests several
improvements to her own system:
1. “... omitting error classification entirely and showing studentsonly the locations of errors in a sentence, which are then stored instudents’ performance records. However, this means losing therelevant information about error classification in the clear cases.
2. “... surpressing the outermost flags in a nested series and makingit obligatory to address the innermost flags. The rationale is thathaving students first correct lower level errors; typically the lessserious gender disagreements gives higher level grammaticalproblems a chance to present themselves. This strategy makessense for a flag series in which the outermost flag is perfectlyredundant, however, this strategy runs the risk ofmisrepresenting student error for certain other flagconfigurations.”23
The suggestions made by Holland [1994] are unsatisfactory.
The first solution is a step backward in the development of ILTSs. For
example, Nagata [1991, 1995, 1996] studied the efficacy of simply underlining
errors as opposed to providing the student with detailed feedback about the
nature of the error and she found that the latter is indeed more effective and
more appreciated by students.
The strategy of addressing syntactically lower level (innermost) errors
first, given in 2., is strongly motivated by the need to account for ambiguous
errors, Holland [1994] and Schwind [1990a, 1995], as discussed in Chapter 2.
However, as Holland [1994] states “the strategy runs the risk of
misrepresenting student errors for certain other flag configurations”24 which
23. Holland [1994], p. 249. For the purpose of discussion, the suggestions made by Holland [1994] have been reordered.Also, Holland [1994] list a third suggestion which, however, is very general and provides norecommendation for actual implementation.
Designed Intelligence: A Language Teacher Model 137
are precisely those contingent errors illustrated in examples (7a) - (8a).25 For
these, it is the syntactically higher level (outermost) errors, such as the choice
of a wrong preposition which need to be reported first. The problem with both
Holland’s [1994] and Schwind’s [1990a, 1995] approaches is that they attempt
to handle ambiguous and contingent errors with basically the same technique.
The assumption is that one can process either higher level errors or lower
level errors first. Yet as with ambiguous errors, any system which cannot deal
with contingent errors effectively runs the risk of producing misleading
feedback, and further misrepresents the student’s knowledge of the
grammatical structures involved. As a result, the Student Model will not be
accurate.
The analysis described can successfully address ambiguous and
contingent errors because a separate technique for each class of errors is
employed. Ambiguous errors are handled by feature percolation and finer-
grained feature values specified in phrase descriptors, as discussed in Chapter
2. In contrast, contingent errors are resolved by the Filtering Module. The
Filtering Module sifts incoming errors in such a way that the syntactically
lower level error is not reported to the student. This is achieved by the Error
Priority Queue which specifies that errors of syntactically higher constituents
are reported first. For example, if the student chooses an incorrect preposition,
the error will be reported before an error with its noun complement. The error
with the noun complement will, however, be recorded in the Student Model.
Once the error with the preposition has been corrected successfully, the error,
if still present, with the noun complement will be addressed. The analysis
24. Holland [1994], p. 249. 25. The same shortcoming is found in Schwind’s [1995] system. As discussed in Chapter 2,Schwind [1995] also addresses lower-level errors first to handle ambiguous errors.
Designed Intelligence: A Language Teacher Model 138
ensures that the system does not misrepresent the student’s knowledge, but
at the same time, provides the student with relevant and accurate feedback.
The grammar constraints of the Error Priority Queue, can always be
reordered to reflect the pedagogy of a particular language instructor. However,
the ranking of the grammar constraints referring to contingent errors, as with
grammatical constraints 1. - 5., which correspond to the choice of a particular
preposition and its noun complement, constitute a fixed subgroup of grammar
constraints. While the whole group can be assigned priority over other
grammar constraints, the order of individual grammar constraints within the
group is fixed to ensure that contingent errors are addressed in a
pedagogically sound way.
4.5 Summary
The Analysis Module, the Student Model, and the Filtering Module
are essential elements of the approach presented in this dissertation. They
incorporate pedagogical and psychological aspects of language learning
essential to an ILTS that models a language instructor in evaluating and
responding to student input.
The strength of the Analysis Module lies in its ability to generate
instructional feedback of different granularity. Granularity is implemented
unidimensionally arranging grammar constraints and instructional feedback
from a coarse to fine grain size.
The Student Model classifies students according to three performance
levels: the novice, the intermediate, and the expert. 79 grammar constraints
are maintained by the model, reflecting the grammatical constraints to be
Designed Intelligence: A Language Teacher Model 139
met in an introductory course for German. A student can be at different levels
for any given grammar constraint reflecting performance on each particular
grammar skill. As a consequence, a single error will not drastically change
the assessment of the student. The information stored can be used for
tailoring instructional feedback suited to the level of the learner and also for
assessment and remediation.
The Filtering Module is responsible for handling multiple errors. The
approach to reporting multiple errors is pedagogically motivated, also taking
into account contingent errors. The system displays one response at a time
and the ranking of feedback responses is established by an Error Priority
Queue. The Error Priority Queue is, however, flexible and can be easily
adjusted to reflect the focus of a particular grammatical constraint in a given
exercise. The final result of the analysis is a single error message tailored to
the level of the learner and pedagogical considerations.