needs detmar meurers introduction detmar meurers i nlp so far … · 2016-05-02 · input...

NLP addressingLanguage Learning

NeedsDetmar Meurers

IntroductionAnalyzing learner language

Analyzing native language

Multidisciplinarity required

Interactive learningFrom CALL to ICALL

TAGARELA

Activity types

Feedback

System Architecture

Range of activity types

Content Assessment

Current projects

Adaptive materialsReadability ranking the web

FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Language Learning and NLP:Connecting Needs and Opportunities

Detmar MeurersDepartment of Linguistics and LEAD Graduate School

Eberhard-Karls Universitat Tubingen

Kortrijk/Louvain-la-Neuve21./22. April 2015

1 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Introduction

I NLP so far plays no role inI real-life Foreign Language Teaching and Learning (FLTL)I or Second Language Acquisition (SLA) research.

I Are there actual needs for automatic language analysis inreal-life FLTL practice or SLA research?

I students in schools increasingly heterogeneous, requiringmore individually adapted materials and interaction

I life-long learning increasingly importantI distance education requires more support for individual

assessment and feedback, especiallyI when scaling up (e.g., MOOCs)I to support more open, functional tasks

I large scale learner corpora are becoming available for SLA

⇒ NLP can play an important role addressing real needs

2 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Introduction

NLP can address needs by supporting the analysis of

I. learner language toI analyze learner capabilitiesI provide interactive feedbackI support interaction

II. native language toI search, modify, or enhance language input for learnersI create exercise materials

that are adapted to individual learner needs

We here focus on analyzing written language; additional issues areinvolved in spoken language interaction and pronunciation training.

3 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Why analyze learner language?

Intelligent Tutoring Systems (ITS)

I online analysis of learner language aimed atsupporting language acquisition

I provide immediate, individualized scaffolding feedback:I meta-linguistic feedback in form-focused activitiesI incidental Focus-on-Form in meaning-based activitiesI feedback on meaning (essential for functional tasks, TBLT)

I determine progression through pedagogical material andsuport interaction with system

⇒ Example: interactive feedback in TAGARELA

4 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School


Learner Corpora

I offline analysis of learner language

I support effective search and analysis of annotated dataI to provide insights into typical learner needs in FLTI to provide empirical evidence for SLA research, e.g.,

I identify developmental sequences and task effects,I linguistic correlates of CEFR proficiency levels, orI native language transfer

I gold-standard training & testing data for development ofNLP for learner language

5 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School


Language Testing

I support assessment of learner competenceI automate (some) gradingI support more efficient grading by grouping learner answers

I draw valid inferences about a learner’s state of knowledgeI also central (but little discussed) for Tutoring Systems

Writer’s aid tools

I provide feedback aimed at producing textI identify and correct errors in orthography, grammar, usage

6 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Why analyze native language for learners?

Input Tailoring and Input Enrichment

I identify authentic materials appropriate in readability, andI richly representing language forms targeted by curriculumI tailored to the needs in the learner’s developmental path

⇒ Example: linguistically-aware search engine FLAIR

Input Enhancement

I Enhanced presentation of materials, adapted to learnerI visual input enhancement supporting noticingI generation of annotations (e.g., vocabulary)

I Generation of exercises

⇒ Example: Input enhancement system VIEW

7 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Multidisciplinary collaboration is required

I For NLP to address real-life needs, it must connect toI Second Language Acquisition (SLA) research

I tasks, instructional interventions, relevance of input/output,interaction, meaning, focus-on-form, developmental seq.

I Foreign Language Teaching and Learning (FLTL)I address teacher needs, while keeping them in charge

I Cognitive PsychologyI attention, memory, learning, motivation, lab studies

I Empirical Educational ScienceI intervention studies, real-life evaluation, multi-level modeling

8 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Interactive learning environments

I The time a student can spend with an instructor typicallyis very limited, limiting individual interaction and feedback.

I Individual support is increasingly essential givenI more heterogeneous classes due to falling numbers and

more children with migration background and other needsI informal learning environments and lifelong learning

I Intelligent tutoring systems support learners inI incrementally completing tasks with individual feedbackI selecting learning materials driven by individual needs

9 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

An opportunity for CALL

I Good opportunity for developing CALL tools toI support acquisition of formsI practice language production with individual feedbackI practice receptive skills (difficult to do in class)I raise linguistic awareness in general

I But existing systems typicallyI offer limited exercise types such as decontextualized

vocabulary practice, multiple choice, point&click, form fillingI with feedback limited to true/false or letter-by-letter

matching of the learner response with pre-stored answersI Example Site: “Spanish Grammar Exercises”

10 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Making CALL tools aware of language

I String matching as the traditional technique used toautomatically analyze learner answers is effective when

I correct answers and potential errors are predictable andlistable→ little well-formed or ill-formed variation

I listable answers correspond to intended feedback

I Computational linguistic analysis must be added whenI all possible correct and incorrect answers are not

(conveniently) listable for a given activityI individualized feedback is desired which requires more

linguistic characteristics of the learner language

11 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

An example ILTS: TAGARELA

I A concrete example for an Intelligent Tutoring System:

TAGARELA: Teaching Aid for Grammatical Awareness,Recognition and Enhancement of Linguistic Abilities

I intelligent web-based workbook complementing instructionI targeting beginning learners of PortugueseI designed to satisfy real-life FLT needs:

I regular classroom instruction at OSUI individualized instruction at OSUI long-distance courses at UMass

I Focus: learner language interpretation, learner modeling,system interface design, NLP architecture, and how thesystem satisfies real-life needs in current FLT approaches(Amaral & Meurers 2006, 2008, 2009; Amaral, Meurers & Ziai 2011)

12 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

13 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Activity types

I SLA research emphasizes the need for activities to involveboth meaning and form.

I TAGARELA offers six types of activities:I listening comprehensionI reading comprehensionI picture descriptionI fill-in-the-blankI rephrasingI vocabulary

= similar to traditional workbook exercises, plus audio

14 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

15 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

16 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Nature of the feedback

I Which forms of feedback are most successful in fosteringawareness of forms and categories?

I Some results from SLA studies on feedback carry over tohuman-computer interaction and CMC:

I recasts are as effective in a dialogue system for learningEnglish question formation (Petersen 2010)

I recasts in synchronous CMC (Sachs & Suh 2007)I recasts and meta-linguistic feedback in dialogue system

for maptask and appointment scheduling (Wilske 2015)

17 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Nature of the feedback in TAGARELA

I TAGARELA provides on-the-spot feedback onI orthographic errors (non-word errors, spacing,

capitalization, punctuation)I syntactic errors (nominal and verbal agreement)I semantic errors (missing or extra concepts, word choice)

I Nature of feedback realized for university students:I meta-linguistic feedback in form-focused activitiesI incidental focus-on-form in meaning-based activitiesI feedback on meaning prioritized over feedback on form

18 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

19 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Feedback on Agreement

20 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

21 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Feedback on Word Choice

22 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

23 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Feedback on Wrong Lexical Content

24 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

25 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Feedback on Missing Verb

26 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

General Architecture of TAGARELA

Expert Module

Linguistic Analysissub-modules

Strategic Analysissub-modules

• task strategies

• task appropriateness

• transfer

An

alys

is M

anag

er•

Ling

uist

ic A

naly

sis

(for

m a

nd c

onte

nt)

• S

trat

egic

Ana

lysi

s

Activity Model

Error Taxonomy

Instruction Model

Personal information

Interaction Preferences

Student Model

Language Competence

Fee

db

ack

Man

ager

(pe

dago

gica

l mod

ules

)

• E

rror

Filt

erin

g•

Ran

king

• S

tude

nt a

naly

sis

• F

eedb

ack

sele

ctio

n

Feedback

Generation• Form Analysis:

• tokenizer• spell-checker• lexical look-up• disambiguator• parser

• Content Analysis:• difflib• correct answer• token matcher• canonic matcher• pos matcher

Web InterfaceStudent Input Feedback Message

27 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

NLP analysis modules in TAGARELA

I Form Analysis:I tokenizer: takes into account specifics of Portuguese

(cliticization, contractions, abbreviations)I lexical/morphological lookup: returns multiple analyses

based on CURUPIRA lexicon (Martins et al. 2006)I disambiguator: finite state disambiguation rules narrow

down lexical information, in the spirit of ConstraintGrammar (Karlsson et al. 1995; Bick 2000, 2004)

I parser: bottom-up chart parser establishes relations tocheck agreement, case and global well-formedness

I Content Analysis:I shallow semantic matching strategies between learner

answer and target answer (Bailey & Meurers 2006, 2008)I since 2009 topic of CoMiC project (http://purl.org/icall/comic)

28 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Adding learner and activity models

I The TAGARELA architecture includesI model of domain knowledge (linguistic knowledge)I learner modelI instruction/activity model

I What is the point of learner and activity models?

⇒ Providing feedback involvesI identifying properties of the learner production andI interpreting them in terms of likely (mis)conceptions of a

specfific learner trying to complete a particular activityI This interpretation goes beyond linguistic form as such.I It needs to model the learner’s use of language for a

specific task in a specific context (Amaral & Meurers 2008).

29 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

How to plug it all together?

I Allow the analysis manager to flexibly employ NLPmodules relevant to a particular activity.

I Flexible control also relevant from NLP perspective, tosupport interleaving of contributions from modules, e.g.:

I part-of-speech ambiguity in Portuguese: a can be aI preposition (to)I pronoun (her, clitic direct object)I article (the, feminine singular)I abbreviation (association, alcoholic, etc.)

I tokenization can resolve some part-of-speech ambiguities:I da = de + a (article)I ve-la = ver + a (clitic pronoun)I a = a (preposition) + a (article)I A.A.A. = Associacao dos Alcolicos Anonimos

→ TAGARELA tokenizer annotates some part-of-speech

30 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Annotation-based processing

I For a flexible control structure, data structures serving asinput/output of analyses need to be uniform and explicit.

I NLP analysis = enriching learner data with annotationsI parallel to XML-based corpus annotation

I UIMA-based version of TAGARELA (Amaral et al. 2011)I Unstructured Information Management Architecture

I In addition to information obtained by analyzing learnerproduction, integrate information on activity and learner.

31 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Learner language analysis is task dependent

I In TAGARELA, different activity types require differentinformation to interpret learner production:

I FIB: spell-checking, lexical informationI Rephrasing: as above + syntactic processing and basic

token matcher for content assessmentI Reading: as above + all content analysis modules

I Why not always run everything?I “Don’t guess what you know.’

I here: use what we know from task specificationI The more we know the linguistic properties, the types of

variation, and the potential errors the NLP needs to detectI the more specific information we can diagnoseI with higher reliability

32 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Analysis is task dependent in general

I How would you analyze the following sentences from theHiroshima English Learners’ Corpus (Miura 1998)?

(1) I didn’t know(2) I don’t know his lives.(3) I know where he lives.

They are taken from a translation task, for the Japanese of

(4) I don’t know where he lives.

⇒ To reliably interpret learner language in ITS and learnercorpus research, we should more seriously consider

I the particular task andI the learner characteristics.

33 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

NLP performance confirms needs

I The best approach to grammatical error correction onlyreaches 39.7% precision, 30.1% recall (Ng et al. 2014)

I Inter-annotator agreement for error annotation of learnercorpora is only starting to be reported (Rosen et al. 2014).

I By adding explicitI task design (Amaral & Meurers 2011; Quixal & Meurers 2016)I learner modeling (Michaud et al. 2001; Amaral & Meurers 2008)

we can constrain the well-formed and ill-formed variationenough to obtain effective analysis of learner language.

34 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School


Tightly Restricted Responses Loosely Restricted Responses

Decontextualized

grammar fill-in-

the-blanks

Short-answer reading

comprehension

questions

Essays on

individualized

topics

The Middle Ground

Viable Processing Ground

In the middle ground, there is a range of attractive activitytypes (reading/listening comprehension, information gap, . . . ):

I a good fit with current task-based or communicativeinstruction settings

I effective analysis possible given predictable variability oflearner responses (Quixal 2012; Quixal & Meurers 2016)

I but sophisticated meaning assessment required⇒ CoMiC project: http://purl.org/icall/comic

35 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Evaluating meaning for reading comprehensionAn example from CREE (Bailey & Meurers 2008)

Q: What are the methods of propaganda mentioned in the article?

T: The methods include use of labels, visual images, and beautiful orfamous people promoting the idea or product. Also used is linkingthe product to concepts that are admired or desired and to create theimpression that everyone supports the product or idea.

Sample Learner Responses:

(5) A number of methods of propaganda are used in the media.

(6) Bositive or negative labels.

(7) Giving positive or negative labels. Using visual images. Havinga beautiful or famous person to promote. Creating theimpression that everyone supports the product or idea.

36 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

CREG Example

37 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

General CoMiC Approach(Bailey & Meurers 2008; Meurers, Ziai, Ott & Bailey 2011a)

The overall approach has three phases:

1. Annotation uses NLP to enrich the student and targetanswers, as well as the question text, with linguisticinformation on different levels and types of abstraction.

2. Alignment maps elements of the learner answer toelements of the target response using annotation.

I Global alignment solution computed by TraditionalMarriage Algorithm (Gale & Shapley 1962)

3. Classification analyzes the possible alignments andlabels the learner response with a binary contentassessment and a detailed diagnosis code.

38 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Types of alignment

Alignment can involve different types of representation:

Alignment Type Example MatchToken-identical advertising

advertisingLemma-resolved advertisement

advertisingSpelling-resolved campaing

campaignReference-resolved Clinton

heSemantic similarity-resolved initial

beginningSpecialized expressions May 24, 2007

5/24/2007

39 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Levels of alignment

Alignment can take place at different levels of representation:

Level Example AlignmentTokens The explanation is simple. explanation

The reason is simple. reasonChunks A brown dog sat in a nice car. a brown dog

A nice dog sat in a car. a nice dogDepen- He knows the doctor. obj(knows, doctor)dency John knows him. obj(knows, him)triples

40 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

NLP tools used

Annotation Task Language Processing ToolSentence Detection, EN: MontyLingua (Liu 2004)Tokenization, DE: OpenNLPLemmatization EN: PC-KIMMO (Antworth 1993)

DE: TreeTagger (Schmid 1994)Spell Checking Edit distance (Levenshtein 1966),

SCOWL word list (Atkinson 2004)igerman98 word list

Part-of-speech Tagging TreeTagger (Schmid 1994)Noun Phrase Chunking CASS (Abney 1996)Lexical Relations WordNet (Miller 1995)

GermaNet (Hamp & Feldweg 1997)Similarity Scores PMI-IR (Turney 2001; Mihalcea et al. 2006)

Dependency Relations Stanford Parser (Klein & Manning 2003)

MaltParser (Nivre et al. 2007)

41 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

ClassificationFeatures

I Content Assessment is based on 13 features:% of Overlapping Matches:

I keyword (head)I target/learner tokenI target/learner chunkI target/learner triple

Nature of Matches:I % token matchesI % lemma matchesI % synonym matchesI % similarity matchesI % sem. type matchesI match variety

I We combined the evidence with memory-based learning(TiMBL, Daelemans et al. 2007)

I Trained seven classifiers using different distance metrics,overall outcome obtained through majority voting.

42 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Meaning Assessment: Some results

I English CREE corpus: 88% accuracy of CoMiC-EN system(binary assessment, Bailey & Meurers 2008)

I Competitive with ETS automatic scoring of native speakershort answers by C-Rater (Leacock & Chodorow 2003)

I Alternative techniques in essay grading systems (e.g.,E-Rater, Burstein et al. 2003; AutoTutor, Graesser et al. 1999) donot generalize well to short responses of 1–2 sentences.

I For German, we developed two systemsI CoMiC-DE (Meurers, Ziai, Ott & Kopp 2011b)I CoSeC-DE (Hahn & Meurers 2012)

achieving 84.6%–86.3% acccuracy on CREG corpus.

I Integration of more context information (text, question)further improves the analysis (Ziai & Meurers 2014).

43 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Current projects

I Linguistic form and meaning in a CL analysis of learnerlanguage. On the integration of morpho-syntactic andsemantic analysis (DFG project 2014–2019)

I Develop NLP approach capable of interleaving bottom-upinformation from string with top-down information from task

I Extend analysis of form errors (Ng et al. 2013)

I Developing an interactive workbook for English foreignlanguage teaching: Integrating state-of-the-art form andmeaning assessment from CL into a current workbook forthe Gymnasium (DFG project 2016–2019)

I Develop broader range of activity types integratingstate-of-the-art content-assessment

I Support and evaluate real-life use in secondary school

44 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Teaching materialsWhat is offered as input to learners and how is it presented?

I Teaching materials are developed based on the contentsto be communicated.

I The complexity of the language used to express thecontents so far has received only little attention.

I Common Core State Standards in the US raises thequestion of incremental textual sophistication and targets.

I In Germany, related needs are starting to be recognized:I “Hinfuhrung zu Bildungssprache”

[Progression towards academic language]I How can teaching materials be selected or adapted to

learner populations (age, ability, migration backgr., . . . )?

45 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Teaching materials: What can be done?

I Identify texts at level of complexity adapted to individualneeds based on multi-faceted analysis of complexity

I English (Vajjala & Meurers 2012, 2013, 2014a,b,c)I French (Francois & Fairon 2012; Todirascu, Francois, Gala,

Fairon, Ligozat & Bernhard 2013; Francois & Bernhard 2014)I German (Hancke, Meurers & Vajjala 2012)

I Linguistically-aware search engine(Ott & Meurers 2010; Chinkina & Meurers 2016)

I search for authentic texts at the right level of complexityI richly representing language forms targeted by curriculumI tailored to the needs in the learner’s developmental path

I Input Enhancement of texts (Meurers et al. 2010)

46 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Readability-ranking the web(Vajjala & Meurers 2013)

I Are state-of-the-art readability models actually useful forclassifying texts as found on the web?

I Can we re-rank search results based on reading levels?

I Implementation details:I feature set inspired by SLA measuresI WEKA linear regression, since we want output on a scaleI trained model on 5-level WeeBit corpus

I We applied the readability model to search resultsobtained through BING search API.

I took 50 search queries from a public query logI computed reading levels for Top-100 results

47 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Results: Reading levels of top search resultsVajjala & Meurers (2013)

Result Rank: 1 2 3 4 5 6 7 8 9 10 Avg.Top100

Query:copyright copy law 1.8 4.6 1.4 2.7 4.6 6.2 2.7 1.1 3.9 5.6 4.6halley comet 1.7 4.5 4.5 4.2 2.4 4.1 4.9 3.6 4.2 3.6 4.0europe union politics 3.6 4.9 6.3 4.0 2.2 4.5 1.5 1.6 4.9 6.3 4.3shakespeare 2.4 2.9 4.2 4.7 4.7 3.9 1.5 2.1 2.6 4.0 3.6euclidean geometry 3.9 4.7 4.7 4.3 4.5 4.6 4.0 4.1 3.5 2.6 3.2. . .

I Results:

I avg. reading level of search results high (5 = GCSE)I full range of reading levels among most relevant results

returned by search engine

I Re-ranking of search results potentially useful in real life

48 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Supporting material retrieval: FLAIR(Chinkina & Meurers 2016)

I Form-focused Linguistically Aware Information Retrieval

I identifies the 87 grammar topics spelled out in completeofficial English curriculum of schools (Baden-Wurttem.)

I designed to support teachers in identifying texts thatprovide the forms targeted by the curriculum

I reranks search results based on the selected(de)prioritization of grammatical forms

I interactively visualizes results, supporting inspection ofdistribution of targeted forms

I accessible at http://purl.org/icall/flair

49 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

FLAIR Interface

50 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

FLAIR Interactive Visualization of Results

51 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

FLAIRExploring the distribution of constructions in web search results

I distribution of grammaticalconstruction across top 55results for the query “2016US presidential elections”

I variability shown by heatmap confirms: rerankingcan enrich representation ofmany forms in curriculum

I also supports retrievingdocuments showcasingcontrasts: adj vs. adv,present vs. past simple, etc.

52 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Input Enhancement

I Second language learners benefit from or may require aso-called focus on form to overcome incomplete orincorrect knowledge (Long 1991; Lightbown 1998).

I Focus on Form: “an occasional shift of attention tolinguistic code features” (Long & Robinson 1998, p. 23).

I Strategies highlighting the salience of language forms andcategories are referred to as input enhancement(Sharwood Smith 1993).

⇒ Automatic input enhancement for language learnersI WERTi v1 (Amaral/Meurers/Metcalf, CALICO & EUROCALL 06)I WERTi/VIEW: Firefox Add-on + UIMA-based NLP server

(Meurers et al. 2010)

53 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

WERTi systemWorking with English Real Text interactively

I Provide learners of English with input enhancementfor any web pages they are interested in.

→ good for learner motivation:I learners can choose material based on their interestsI includes news, up-to-date information, hip stuffI pages remain fully contextualized (video, audio, links)

→ wide range of potential learning contexts:I can supplement regular classroom instructionI can support voluntary, self-motivated pursuit of

knowledge, i.e., lifelong learning.I can foster implicit learning, e.g., for adult immigrants:

I already functionally living in second language environment,but stagnating in acquisition

I without access/motivation to engage in explicit learning,but browsing the web for information and entertainment

54 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

What language properties should we enhance?

I A wide range of linguistic features can be relevant forawareness, incl. morphological, syntactic, semantic, andpragmatic information (Schmidt 1995).

I We focus on enhancing language patterns which arewell-established difficulties for ESL learners:

I determiner and preposition usageI use of gerunds vs. to-infinitivesI wh-question formationI phrasal verbs

NLP identifying other patterns can easily be integrated aspart of a flexible NLP architecture.

55 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

How should the targeted forms be enhanced?

I WERTi currently offers three types of input enhancement:a) color highlighting of the pattern or selected parts thereofb) pages supporting clicking, with automatic color feedback

I automatic feedback compares automatic annotation ofclicked on form with targeted form

c) pages supporting practice (e.g., fill-in-the-blank), withautomatic color feedback

I automatic feedback compares form entered by learner withform in original text

I This follows standard pedagogical practice (“PPP”):a) receptive presentationb) presentation supporting limited interactionc) controlled practiced) (free production)

56 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Prepositions: Presentation (Color)

Source: http://news.bbc.co.uk/2/hi/5277090.stm

57 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Prepositions: Practice (FIB)

Source: http://news.bbc.co.uk/2/hi/5277090.stm

58 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Prepositions: Presentation + Interaction (Click)

Source: http://www.guardian.co.uk/environment/green-living-blog/2009/oct/29/car-free-cities-neighbourhoods 59 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Prepositions: Presentation + Interaction (Click)

Source: http://www.guardian.co.uk/environment/green-living-blog/2009/oct/29/car-free-cities-neighbourhoods 60 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Phrasal verbs: Presentation (Color)

61 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Phrasal verbs: Practice (Fill-in-the-blank)

62 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Gerunds vs. infinitives: Practice (FIB)

Source: http://www.guardian.co.uk/education/2009/oct/14/30000-miss-university-place63 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Wh-questions: Presentation + Interaction (Click)

Source: http://simple.wikipedia.org/wiki/Illegal drugs

64 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Wh-questions: Presentation + Interaction (Click)

Source: http://simple.wikipedia.org/wiki/Illegal drugs

65 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Relation to Data-Driven Learning

I One can view automatic Input Enhancement as anenrichment of Data-Driven Learning (DDL).

I DDL is an “attempt to cut out the middleman [the teacher]as far as possible and to give the learner direct access tothe data” (Boulton 2009, p. 82, citing Tim Johns)

I VIEW:I learner is in control of the dataI but NLP uses ‘teacher knowledge’ about relevant

properties to make those more prominent to the learner

66 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Some Research Questions & Current Research

I For which language patterns is input enhanced effective?I Which instances or aspects of their context in a given text?I Using which type of input enhancement?

I Which aspects of the interaction should be trackedI for learners? (e.g., Open Learner Models)I for teachers? (e.g., satisfy grading needs)I for researchers? (e.g., observe incremental learning,

complementing pre-/posttest design of study)

I Empirical studies needed to properly explore such issuesI Simon Ruiz PhD project in LEAD investigates the

effectiveness of input enhancement of phrasal verbsI A pilot study on article selection with Nicole Ziegler, Jose

Luis Moreno, Wenjing Li, Simon Ruiz, Maria Chinkina,Sarah Grey, Detmar Meurers, and Patrick Rebuschat

67 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Observe learning as it happens

●

●

●

● ●

●

●

●

●

●

● ●●

●

●

● ●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ●

●

● ●● ●

●

●

● ●

●

● ●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●

●

●

● ●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

● ●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

● ● ●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

● ●

●

●

●●

● ●●● ●● ●● ●●

●● ●●● ● ● ●

●

●

●

● ● ● ● ●

●

●

●

●

● ●

●●

●

●●

●

●●●●● ●

●

●● ●

●

●● ● ● ●●

●

●

●

●

● ●●●

●

●

●

●

●

●

●

●

●

●

●

●

● ● ●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●● ●●●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ●●

●

●

●

●

●●

●●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●●

●

●

●●

●●

● ● ●

● ●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●●

● ●

●

●

●●

●

●

●

●

●

●

●

●

●● ●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●●

●●

●● ● ●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 250 500 750 1000 1250cumulativeAttempted

prec

isio

n

activity

●

●

●

click

fib

mc

68 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

ConclusionI There is a wide range of opportunities for NLP supporting

interactive learning environments and adapted materials.

I Interactive workbooks such as TAGARELA can provideimmediate learner feedback on form and meaning.

I wide range of exercise types possible using sophisticatedshort-answer meaning assessment→ usable in real life

I Input material for the learner can be selected based onreadability, curricular, and learner needs (FLAIR)

I Input enhancement tools such as WERTi/VIEW supportadaptation and presentation of authentic learning material.

I Empirical studies (including tracking, pre-/posttest design)needed to validate approaches and feed back into SLA

I Interface to SLA and FLTL importantI Special Issue of Language Learning to appear in 2017 targets

“Language learning research at the intersection of experimental,corpus-based and computational methods”

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

ReferencesAbney, S. (1996). Partial Parsing via Finite-State Cascades. In The Robust Parsing

Workshop of the European Summer School in Logic, Language and Information(ESSLLI ’96). Prague, Czech Republic, pp. 1–8. URLhttp://www.vinartus.net/spa/97a.pdf.

Amaral, L., V. Metcalf & D. Meurers (2006). Language Awareness through Re-use ofNLP Technology. Presentation at Preconference Workshop on “NLP in CALL –Computational and Linguistic Challenges”. CALICO 2006. May 17, 2006.University of Hawaii. URLhttp://purl.org/net/icall/handouts/calico06-amaral-metcalf-meurers.pdf.

Amaral, L. & D. Meurers (2006). Where does ICALL Fit into Foreign LanguageTeaching? URL http://purl.org/net/icall/handouts/calico06-amaral-meurers.pdf.23rd Annual Conference of the Computer Assisted Language InstructionConsortium (CALICO), May 19, 2006. University of Hawaii.

Amaral, L. & D. Meurers (2008). From Recording Linguistic Competence toSupporting Inferences about Language Acquisition in Context: Extending theConceptualization of Student Models for Intelligent Computer-Assisted LanguageLearning. Computer-Assisted Language Learning 21(4), 323–338. URLhttp://purl.org/dm/papers/amaral-meurers-call08.html.

Amaral, L. & D. Meurers (2009). Little Things With Big Effects: On the Identificationand Interpretation of Tokens for Error Diagnosis in ICALL. CALICO Journal 26(3),580–591. URL http://purl.org/dm/papers/amaral-meurers-09.html.

Amaral, L. & D. Meurers (2011). On Using Intelligent Computer-Assisted LanguageLearning in Real-Life Foreign Language Teaching and Learning. ReCALL 23(1),4–24. URL http://purl.org/dm/papers/amaral-meurers-11.html.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Amaral, L., D. Meurers & R. Ziai (2011). Analyzing Learner Language: Towards AFlexible NLP Architecture for Intelligent Language Tutors. Computer-AssistedLanguage Learning 24(1), 1–16. URLhttp://purl.org/dm/papers/amaral-meurers-ziai-10.html.

Antworth, E. L. (1993). Glossing Text with the PC-KIMMO Morphological Parser.Computers and the Humanities 26, 475–484. URLhttp://www.springerlink.com/content/r20w66k70976ur9l/fulltext.pdf.

Atkinson, K. (2004). Spell Checking Oriented Word Lists (SCOWL). URLhttp://wordlist.sourceforge.net/. Web resource.

Bailey, S. & D. Meurers (2006). Exercise-driven selection of content matchingmethodologies. Peer reviewed conference presentation. EUROCALL’06.September 6, 2006. University of Granada.

Bailey, S. & D. Meurers (2008). Diagnosing meaning errors in short answers toreading comprehension questions. In J. Tetreault, J. Burstein & R. D. Felice (eds.),Proceedings of the 3rd Workshop on Innovative Use of NLP for BuildingEducational Applications (BEA-3) at ACL’08. Columbus, Ohio, pp. 107–115. URLhttp://aclweb.org/anthology/W08-0913.

Bick, E. (2000). The Parsing System “Palavras”: Automatic Grammatical Analysis ofPortuguese in a Constraint Grammar Framework . Aarhus University Press. URLhttp://beta.visl.sdu.dk/∼eckhard/pdf/PLP20-amilo.ps.pdf.

Bick, E. (2004). PaNoLa: Integrating Constraint Grammar and CALL. In H. Holmboe(ed.), Nordic Language Technology, Arbog for Nordisk SprogteknologiskForskningsprogram 2000-2004 (Yearbook 2003), Copenhagen: MuseumTusculanum, pp. 183–190. URLhttp://beta.visl.sdu.dk/∼eckhard/pdf/PaNoLa-CALL-yearbook2003.ps.pdf.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Boulton, A. (2009). Data-driven Learning: Reasonable Fears and RationalReassurance. Indian Journal of Applied Linguistics 35(1), 81–106.

Burstein, J. & M. Chodorow (1999). Automated Essay Scoring for Nonnative EnglishSpeakers. In M. B. Olson (ed.), Proceedings of a Workshop onComputer-Mediated Language Assessment and Evaluation of Natural LanguageProcessing, Joint Symposium of the Association of Computational Linguistics(ACL-99) and the International Association of Language Learning Technologies.pp. 68–75. URL http://www.ets.org/Media/Research/pdf/erater acl99rev.pdf.

Burstein, J., M. Chodorow & C. Leacock (2003). Criterion: Online Essay Evaluation:An Application for Automated Evaluation of Student Essays. In Proceedings of theFifteenth Annual Conference on Innovative Applications of Artificial Intelligence(IAAI-03). Acapulco, Mexico, pp. 3–10. URLhttp://ftp.ets.org/pub/res/erater iaai03 burstein.pdf.

Chinkina, M. & D. Meurers (2016). Linguistically-Aware Information Retrieval:Providing Input Enrichment for Second Language Learners. In Proceedings of the11th Workshop on Innovative Use of NLP for Building Educational Applications.San Diego, CA.

Daelemans, W., J. Zavrel, K. van der Sloot & A. van den Bosch (2007). TiMBL: TilburgMemory-Based Learner Reference Guide, ILK Technical Report ILK 07-03.Induction of Linguistic Knowledge Research Group Department of Communicationand Information Sciences, Tilburg University, Tilburg, The Netherlands. URLhttp://ilk.uvt.nl/downloads/pub/papers/ilk.0703.pdf. Version 6.0.

Doughty, C. & J. Williams (eds.) (1998). Focus on form in classroom second languageacquisition. Cambridge: Cambridge University Press.

Francois, T. & D. Bernhard (eds.) (2014). International Journal of Applied Linguistics,Special Issue on Current Research in Readability and Text Simplification.Benjamins.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Francois, T. & C. Fairon (2012). An “AI readability” formula for French as a foreignlanguage. In Proceedings of the 2012 Joint Conference on Empirical Methods inNatural Language Processing and Computational Natural Language Learning.

Gale, D. & L. S. Shapley (1962). College Admissions and the Stability of Marriage.American Mathematical Monthly 69, 9–15. URLhttp://www.econ.ucsb.edu/∼tedb/Courses/Ec100C/galeshapley.pdf.

Graesser, A. C., K. Wiemer-Hastings, P. Wiemer-Hastings & R. Kreuz (1999).AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research1, 35–51.

Hahn, M. & D. Meurers (2012). Evaluating the Meaning of Answers to ReadingComprehension Questions: A Semantics-Based Approach. In Proceedings of the7th Workshop on Innovative Use of NLP for Building Educational Applications(BEA-7) at NAACL-HLT 2012. Montreal, pp. 94–103. URLhttp://purl.org/dm/papers/hahn-meurers-12.html.

Hamp, B. & H. Feldweg (1997). GermaNet – a Lexical-Semantic Net for German. InProceedings of ACL workshop Automatic Information Extraction and Building ofLexical Semantic Resources for NLP Applications. Madrid. URLhttp://aclweb.org/anthology/W97-0802.

Hancke, J., D. Meurers & S. Vajjala (2012). Readability Classification for Germanusing lexical, syntactic, and morphological features. In Proceedings of the 24thInternational Conference on Computational Linguistics (COLING). Mumbay, India,pp. 1063–1080. URL http://aclweb.org/anthology-new/C/C12/C12-1065.pdf.

Johns, T. (1994). From printout to handout: Grammar and vocabulary teaching in thecontext of data-driven learning. In T. Odlin (ed.), Perspectives on PedagogicalGrammar , Cambridge: Cambridge University Press, pp. 293–313.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Karlsson, F., A. Voutilainen, J. Heikkila & A. Anttila (eds.) (1995). Constraint Grammar:A Language-Independent System for Parsing Unrestricted Text . No. 4 in NaturalLanguage Processing. Berlin and New York: Mouton de Gruyter.

Klein, D. & C. D. Manning (2003). Accurate Unlexicalized Parsing. In Proceedings ofthe 41st Meeting of the Association for Computational Linguistics (ACL 2003).Sapporo, Japan, pp. 423–430. URL http://aclweb.org/anthology/P03-1054.

Leacock, C. & M. Chodorow (2003). C-rater: Automated Scoring of Short-AnswerQuestions. Computers and the Humanities 37, 389–405.

Levenshtein, V. I. (1966). Binary Codes Capable of Correcting Deletions, Insertions,and Reversals. Soviet Physics Doklady 10(8), 707–710. URLhttp://www.mendeley.com/research/binary-codes-capable-of-correcting-insertions-and-reversals/.

Lightbown, P. M. (1998). The importance of timing in focus on form. In Doughty &Williams (1998), pp. 177–196.

Liu, H. (2004). MontyLingua: An End-to-End Natural Language Processor withCommon Sense. Software Website. Media Laboratory, Massachusetts Institute ofTechnology, Cambridge, Massachusetts. URLhttp://web.media.mit.edu/∼hugo/montylingua/.

Long, M. H. (1991). Focus on form: A design feature in language teachingmethodology. In K. De Bot, C. Kramsch & R. Ginsberg (eds.), Foreign languageresearch in cross-cultural perspective, Amsterdam: John Benjamins, pp. 39–52.

Long, M. H. & P. Robinson (1998). Focus on form: Theory, research, and practice. InDoughty & Williams (1998), pp. 15–41.

Martins, R., R. Hasegawa & M. das Gracas Nunes (2006). Curupira: a functionalparser for Brazilian Portuguese. In Computational Processing of the Portuguese

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Language, 6th International Workshop, PROPOR. Lecture Notes in ComputerScience 2721. Faro, Portugal: Springer. URLhttp://www.springerlink.com/content/b48vjft1l88yvrj0/fulltext.pdf.

Metcalf, V. & D. Meurers (2006). Generating Web-based English PrepositionExercises from Real-World Texts. URLhttp://purl.org/net/icall/handouts/eurocall06-metcalf-meurers.pdf. Presentation atEUROCALL 2006. Granada, Spain. September 4–7, 2006.

Meurers, D., R. Ziai, L. Amaral, A. Boyd, A. Dimitrov, V. Metcalf & N. Ott (2010).Enhancing Authentic Web Pages for Language Learners. In Proceedings of the5th Workshop on Innovative Use of NLP for Building Educational Applications(BEA-5) at NAACL-HLT 2010. Los Angeles, pp. 10–18. URLhttp://aclweb.org/anthology/W10-1002.pdf.

Meurers, D., R. Ziai, N. Ott & S. Bailey (2011a). Integrating Parallel Analysis Modulesto Evaluate the Meaning of Answers to Reading Comprehension Questions.IJCEELL. Special Issue on Automatic Free-text Evaluation 21(4), 355–369. URLhttp://purl.org/dm/papers/meurers-ziai-ott-bailey-11.html.

Meurers, D., R. Ziai, N. Ott & J. Kopp (2011b). Evaluating Answers to ReadingComprehension Questions in Context: Results for German and the Role ofInformation Structure. In Proceedings of the TextInfer 2011 Workshop on TextualEntailment . Edinburgh: ACL, pp. 1–9. URLhttp://aclweb.org/anthology/W11-2401.pdf.

Michaud, L. N., K. F. McCoy & L. A. Stark (2001). Modeling the Acquisition of English:An Intelligent CALL Approach. In Proceedings of The 8th InternationalConference on User Modeling. Sonthofen, Germany, pp. 14–25. URLhttp://www.eecis.udel.edu/research/icicle/pubs/MiMcSt01.ps.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Mihalcea, R., C. Corley & C. Strapparava (2006). Corpus-based andKnowledge-based Measures of Text Semantic Similarity. In Proceedings of theNational Conference on Artificial Intelligence. Menlo Park, CA: AmericanAssociation for Artificial Intelligence (AAAI) Press, vol. 21(1), pp. 775–780. URLhttp://www.cse.unt.edu/∼rada/papers/mihalcea.aaai06.pdf.

Miller, G. (1995). WordNet: a lexical database for English. Communications of theACM 38(11), 39–41. URL http://aclweb.org/anthology/H94-1111.

Miura, S. (1998). Hiroshima English Learners’ Corpus: English learner No. 2 (EnglishI & English II). Department of English Language Education, Hiroshima University.http://purl.org/icall/helc.

Ng, H. T., J. Tetreault, S. M. Wu, Y. Wu & C. Hadiwinoto (eds.) (2013). Proceedings ofthe Seventeenth Conference on Computational Natural Language Learning:Shared Task . Sofia, Bulgaria: Association for Computational Linguistics. URLhttp://aclweb.org/anthology/W13-36.

Ng, H. T., S. M. Wu, T. Briscoe, C. Hadiwinoto, R. H. Susanto & C. Bryant (2014). TheCoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of theEighteenth Conference on Computational Natural Language Learning: SharedTask . Baltimore, Maryland: Association for Computational Linguistics, pp. 1–14.

Nivre, J., J. Nilsson, J. Hall, A. Chanev, G. Eryigit, S. Kubler, S. Marinov & E. Marsi(2007). MaltParser: A Language-Independent System for Data-DrivenDependency Parsing. Natural Language Engineering 13(1), 1–41. URLhttp://w3.msi.vxu.se/∼nivre/papers/nle07.pdf.

Ott, N. & D. Meurers (2010). Information Retrieval for Education: Making SearchEngines Language Aware. Themes in Science and Technology Education. Specialissue on computer-aided language analysis, teaching and learning: Approaches,perspectives and applications 3(1–2), 9–30. URLhttp://purl.org/dm/papers/ott-meurers-10.html.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Petersen, K. (2010). Implicit Corrective Feedback in Computer-Guided Interaction:Does Mode Matter? Ph.D. thesis, Georgetown University. URLhttp://purl.org/net/Petersen-10.pdf.

Quixal, M. (2012). Language Learning Tasks and Automatic Analysis of LearnerLanguage. Connecting FLTL and NLP in the design of ICALL materials supportingeffective use in real-life instruction. Ph.D. thesis, Universitat Pompeu Fabra,Barcelona and Eberhard-Karls-Universitat Tubingen.

Quixal, M. & D. Meurers (2016). How can writing tasks be characterized in a wayserving pedagogical goals and automatic analysis needs? CALICO Journal 33.URL http://purl.org/dm/papers/Quixal.Meurers-16.html. To appear.

Rosen, A., J. Hana, B. Stindlova & A. Feldman (2014). Evaluating and automating theannotation of a learner corpus. Language Resources and Evaluation 48(1),65–92.

Sachs, R. & B.-R. Suh (2007). Textually enhanced recasts, learner awareness, and L2outcomes in synchronous computer-mediated interaction. In Oxford appliedlinguistics, Oxford: Oxford University Press.

Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. InProceedings of the International Conference on New Methods in LanguageProcessing. Manchester, UK, pp. 44–49. URLhttp://www.ims.uni-stuttgart.de/ftp/pub/corpora/tree-tagger1.pdf.

Schmidt, R. (1995). Consciousness and foreign language learning: A tutorial on therole of attention and awareness in learning. In R. Schmidt (ed.), Attention andawareness in foreign language learning, Honolulu, HI: University of Hawaii, pp.1–63.

Sharwood Smith, M. (1993). Input enhancement in instructed SLA: Theoretical bases.Studies in Second Language Acquisition 15, 165–179.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Todirascu, A., T. Francois, N. Gala, C. Fairon, A.-L. Ligozat & D. Bernhard (2013).Coherence and Cohesion for the Assessment of Text Readability. In Proceedingsof the 10th International Workshop on Natural Language Processing andCognitive Science. URL http://www.kicorangel.com/wp-content/uploads/2013/10/NLPCS2013-proceedings.pdf.

Turney, P. (2001). Mining the Web for Synonyms: PMI-IR Versus LSA on TOEFL. InProceedings of the Twelfth European Conference on Machine Learning(ECML-2001). Freiburg, Germany, pp. 491–502.

Vajjala, S. & D. Meurers (2012). On Improving the Accuracy of ReadabilityClassification using Insights from Second Language Acquisition. In J. Tetreault,J. Burstein & C. Leacock (eds.), In Proceedings of the 7th Workshop on InnovativeUse of NLP for Building Educational Applications. Montreal, Canada: Associationfor Computational Linguistics, pp. 163–173. URLhttp://aclweb.org/anthology/W12-2019.pdf.

Vajjala, S. & D. Meurers (2013). On The Applicability of Readability Models to WebTexts. In Proceedings of the Second Workshop on Predicting and Improving TextReadability for Target Reader Populations. pp. 59–68.

Vajjala, S. & D. Meurers (2014a). Assessing the relative reading level of sentencepairs for text simplification. In Proceedings of the 14th Conference of theEuropean Chapter of the Association for Computational Linguistics (EACL). ACL,Gothenburg, Sweden: Association for Computational Linguistics, pp. 288–297.

Vajjala, S. & D. Meurers (2014b). Exploring Measures of “Readability” for SpokenLanguage: Analyzing linguistic features of subtitles to identify age-specific TVprograms. In Proceedings of the Third Workshop on Predicting and Improving TextReadability for Target Reader Populations. Gothenburg, Sweden: ACL, pp. 21–29.

69 / 69


NeedsDetmar Meurers





TAGARELA

Activity types

Feedback

System Architecture


Content Assessment

Current projects


FLAIR search engine

Input Enhancement

Example enhancement

Current research

Conclusion

LEADGraduate School

Vajjala, S. & D. Meurers (2014c). Readability Assessment for Text Simplification: FromAnalyzing Documents to Identifying Sentential Simplifications. InternationalJournal of Applied Linguistics, Special Issue on Current Research in Readabilityand Text Simplification 165(2), 142–222.

Wiemer-Hastings, P., K. Wiemer-Hastings & A. Graesser (1999). Improving anIntelligent Tutor’s Comprehension of Students with Latent Semantic Analysis. InS. Lajoie & M. Vivet (eds.), Artificial Intelligence in Education, IOS Press, pp.535–542. URL http://eprints.kfupm.edu.sa/45213/1/45213.pdf.

Wilske, S. (2015). Form and meaning in dialogue-based computer-assisted languagelearning. Ph.D. thesis, Universitat des Saarlandes, Saarbrucken. URLhttp://scidok.sulb.uni-saarland.de/volltexte/2015/6251/.

Ziai, R. (2009). A Flexible Annotation-Based Architecture for Intelligent LanguageTutoring Systems. Master’s thesis, Universitat Tubingen, Seminar furSprachwissenschaft. URLhttp://www.sfs.uni-tuebingen.de/∼rziai/papers/Ziai-09.pdf.

Ziai, R. & D. Meurers (2014). Focus Annotation in Reading Comprehension Data. InProceedings of the 8th Linguistic Annotation Workshop (LAW VIII, 2014).COLING, Dublin, Ireland: Association for Computational Linguistics, pp. 159–168.URL http://www.aclweb.org/anthology/W14-4922.

69 / 69