Proceedings of the Third ACL Workshop on Innovative Use of NLP for Building Educational Applications, pages 80–88,Columbus, Ohio, USA, June 2008. c©2008 Association for Computational Linguistics

Retrieval of Reading Materials for Vocabulary and Reading Practice

Michael Heilman, Le Zhao, Juan Pino and Maxine EskenaziLanguage Technologies Institute

Carnegie Mellon UniversityPittsburgh, PA 15213, USA

{mheilman,lezhao,jmpino,max}@cs.cmu.edu

AbstractFinding appropriate, authentic reading mate-rials is a challenge for language instructors.The Web is a vast resource of texts, but mostpages are not suitable for reading practice, andcommercial search engines are not well suitedto finding texts that satisfy pedagogical con-straints such as reading level, length, text qual-ity, and presence of target vocabulary. Wepresent a system that uses various languagetechnologies to facilitate the retrieval and pre-sentation of authentic reading materials gath-ered from the Web. It is currently deployed intwo English as a Second Language courses atthe University of Pittsburgh.

1 Introduction

Reading practice is an important component of firstand second language learning, especially with re-gards to vocabulary learning (Hafiz and Tudor,1989). Appropriating suitable reading material forthe needs of a particular curriculum or particular stu-dent, however, is a challenging process. Manuallyauthoring or editing readings is time-consuming andraises issues of authenticity, which are particularlysignificant in second language learning (Peacock,1997). On the other hand, the Web is a vast resourceof authentic reading material, but commercial searchengines which are designed for a wide variety of in-formation needs may not effectively facilitate the re-trieval of appropriate readings for language learners.

In order to demonstrate the problem of finding ap-propriate reading materials, here is a typical exam-ple of an information need from a teacher of an En-glish as a Second Language (ESL) course focused

on reading skills. This example was encounteredduring the development of the system. It shouldbe noted that while we describe the system in thecontext of ESL, we claim that the approach is gen-eral enough to be applied to first language readingpractice and to languages other than English. Tofit within his existing curriculum, the ESL teacherwanted to find texts on the specific topic of “interna-tional travel.” He sought texts that contained at leasta few words from the list of target vocabulary thathis student were learning that week. In addition, heneeded the texts to be within a particular range ofreading difficulty, fifth to eighth grade in an Ameri-can school, and shorter than a thousand words.

Sending the query “international travel” to a pop-ular search engine did not produce a useful list of re-sults1. The first result was a travel warning from theDepartment of State2, which was at a high readinglevel (grade 10 according to the approach describedby (Heilman et al., 2008)) and not likely to be ofinterest to ESL students because of legal and techni-cal details. Most of the subsequent results were forcommercial web sites and travel agencies. A queryfor a subset of the target vocabulary words for thecourse also produced poor results. Since the searchengine used strict boolean retrieval methods, the topresults for the query “deduce deviate hierarchy im-plicit undertake” were all long lists of ESL vocabu-lary words3.

We describe a search system, called REAPSearch, that is tailored to the needs of language

1www.google.com, March 5, 20082http://travel.state.gov/travel/cis pa tw/cis pa tw 1168.html3e.g., www.espindle.org/university word list uwl.html

80

teachers and learners. The system facilitates the re-trieval of texts satisfying particular pedagogical con-straints such as reading level and text length, and al-lows the user to constrain results so that they con-tain at least some, but not necessarily all, of thewords from a user-specified target vocabulary list.It also filters out inappropriate material as well aspages that do not contain significant amounts of textin well-formed sentences. The system provides sup-port for learners including an interface for readingtexts, easy access to dictionary definitions, and vo-cabulary exercises for practice and review.

The educational application employs multiplelanguage technologies to achieve its various goals.Information retrieval and web search technologiesprovide the core components. Automated text clas-sifiers organize potential readings by general topicarea and reading difficulty. We are also developingan approach to measuring reading difficulty that usesa parser to extract grammatical structures. Part ofSpeech (POS) tagging is used to filter web pages tomaintain text quality.

2 Path of a Reading

In the REAP Search system, reading materials take apath from the Web to students through various inter-mediate steps as depicted in Figure 1. First, a crawl-ing program issues queries to large-scale commer-cial search engines to retrieve candidate documents.These documents are annotated, filtered, and storedin a digital library, or corpus. This digital library cre-ation process is done offline. A customized searchinterface facilitates the retrieval of useful readingmaterials by teachers, who have particular curricu-lar goals and constraints as part of their informationneeds. The teachers organize their selected readingsthrough a curriculum manager. The reading inter-face for students accesses the curriculum manager’sdatabase and provides the texts along with supportin the form of dictionary definitions and practice ex-ercises.

3 Creating a Digital Library of Readings

The foundation of the system is a digital library ofpotential reading material. The customized searchcomponent does not search the Web directly, butrather accesses this filtered and annotated database

of Web pages. The current library consists of ap-proximately five million documents. Constructionof the digital library begins with a set of target vo-cabulary words that might be covered by a course orset of courses (typically 100-1,500 words), and a setof constraints on text characteristics. The constraintscan be divided into three sets: those that can be ex-pressed in a search engine query (e.g., target words,number of target words per text, date, Web domain),those that can be applied using just information inthe Web search result list (e.g., document size), andthose that require local annotation and filtering (e.g.,reading level, text quality, profanity).

The system obtains candidate documents byquery-based crawling, as opposed to followingchains of links. The query-based document crawl-ing approach is designed to download documentsfor particular target words. Queries are submittedto a commercial Web search engine4, result links aredownloaded, and then the corresponding documentsare downloaded. A commercial web search engineis used to avoid the cost of maintaining a massive,overly general web corpus.

Queries consist of combinations of multiple tar-get words. The system generates 30 queries for eachtarget word (30 is a manageable and sufficient num-ber in practice). These are spread across 2-, 3-,and 4-word combinations with other target words.Queries to search engines can often specify a daterange. We employ ranges to find more recent mate-rial, which students prefer. The tasks of submittingqueries, downloading the result pages, and extract-ing document links are distributed among a dozenor so clients running on desktop machines, to run asbackground tasks. The clients periodically uploadtheir results to a server, and request a new batch ofqueries.

Once the server has a list of candidate pages, itdownloads them and applies various filters. The fi-nal yield of texts is typically approximately one per-cent of the originally downloaded results. Many webpages are too long, contain too little well-formedtext, or are far above the appropriate reading levelfor language learners. After downloading docu-ments, the system annotates them as described inthe next section. It then stores the pages in a full-

4www.altavista.com

81

Figure 1: Path of Reading Materials from the Web to a Student.

text search engine called Indri, which is part ofthe Lemur Toolkit5. This index provides a consis-tent and efficient interface to the documents. UsingLemur and the Indri Query Language allows for theretrieval of annotated documents according to user-specified constraints.

4 Annotations and Filters

Annotators automatically tag the documents in thecorpus to enable the filtering and retrieval of read-ing material that matches user-specified pedagogicalconstraints. Annotations include reading difficulty,general topic area, text quality, and text length. Textlength is simply the number of word tokens appear-ing in the document.

4.1 Reading LevelThe system employs a language modeling ap-proach developed by Collins-Thompson and Callan(Collins-Thompson and Callan, 2005) that creates amodel of the lexicon for each grade level and pre-dicts reading level, or readability, of given docu-ments according to those models. The readabil-ity predictor is a specialized Naive Bayes classi-fier with lexical unigram features. For web docu-ments in particular, Collins-Thompson and Callanreport that this language modeling-based predictionhas a stronger correlation with human-assigned lev-els than other commonly used readability measures.This automatic readability measure allows the sys-tem to satisfy user-specified constraints on readingdifficulty.

We are also experimenting with using syntac-tic features to predict reading difficulty. Heilman,Collins-Thompson, and Eskenazi (Heilman et al.,2008) describe an approach that combines predic-tions based on lexical and grammatical features. The

5www.lemurproject.org

grammatical features are frequencies of occurrenceof grammatical constructions, which are computedfrom automatic parses of input texts. Using multiplemeasures of reading difficulty that focus on differentaspects of language may allow users more freedomto find texts that match their needs. For example,a teacher may want to find grammatically simplertexts for use in a lesson focused on introducing dif-ficult vocabulary.

4.2 General Topic AreaA set of binary topic classifiers automatically clas-sifies each potential reading by its general topic, asdescribed by Heilman, Juffs, and Eskenazi (2007).This component allows users to search for readingson their general interests without specifying a par-ticular query (e.g., “international travel”) that mightunnecessarily constrain the results to a very narrowtopic.

A Linear Support Vector Machine text classifier(Joachims, 1999) was trained on Web pages fromthe Open Directory Project (ODP)6. These pages ef-fectively have human-assigned topic labels becausethey are organized into a multi-level hierarchy oftopics. The following general topics were manuallyselected from categories in the ODP: Movies andTheater; Music; Visual Arts; Computers and Tech-nology; Business; Math, Physics and Chemistry; Bi-ology and Environment; Social Sciences; Health andMedicine; Fitness and Nutrition; Religion; Politics;Law and Crime; History; American Sports; and Out-door Recreation.

Web pages from the ODP were used as gold-standard labels in the training data for the classi-fiers. SVM-Light (Joachims, 1999) was used as animplementation of the Support Vector Machines. Inpreliminary tests, the linear kernel produced slightly

6dmoz.org

82

better performance than a radial basis function ker-nel. The values of the decision functions of the clas-sifiers for each topic are used to annotate readingswith their likely topics.

The binary classifiers for each topic category wereevaluated according to the F1 measure, the harmonicmean of precision and recall, using leave-one-outcross-validation. Values for the F1 statistic rangefrom .68 to .86, with a mean value of .76 acrosstopics. For comparison, random guessing would beexpected to correctly choose the gold-standard labelonly ten percent of the time. During an error analy-sis, we observed that many of the erroneous classifi-cations were, in fact, plausible for a human to makeas well. Many readings span multiple topics. Forexample, a document on a hospital merger might beclassified as “Health and Medicine” when the cor-rect label is “Business.” In the evaluation, the goldstandard included only the single topic specified bythe ODP. The final system, however, assigns multi-ple topic labels when appropriate.

4.3 Text Quality

A major challenge of using Web documents for ed-ucational applications is that many web pages con-tain little or no text in well-formed sentences andparagraphs. We refer to this problem as “Text Qual-ity.” Many pages consist of lists of links, navigationmenus, multimedia, tables of numerical data, etc. Aspecial annotation tool filters out such pages so thatthey do not clutter up search results and make it dif-ficult for users to find suitable reading materials.

The text quality filter estimates the proportion ofthe word tokens in a page that are contained in well-formed sentences. To do this it parses the DocumentObject Model structure of the web page, and orga-nizes it into text units delineated by the markup tagsin the document. Each new paragraph, table ele-ment, span, or divider markup tag corresponds to thebeginning of a new text unit. The system then runsa POS tagger7 over each text unit. We have foundthat a simple check for whether the text unit con-tains both a noun and a verb can effectively distin-guish between content text units and those text unitsthat are just part of links, menus, etc. The proportion

7The OpenNLP toolkit’s tagger was used(opennlp.sourceforge.net).

of the total tokens that are part of content text unitsserves as a useful measure of text quality. We havefound that a threshold of about 85% content text isappropriate, since most web pages contain at leastsome non-content text in links, menus, etc. This ap-proach to content extraction is related to previouswork on increasing the accessibility of web pages(Gupta et al., 2003).

5 Constructing Queries

Users search for readings in the annotated corpusthrough a simple interface that appears similar to,but extends the functionality of, the interfaces forcommercial web search engines. Figure 2 showsa screenshot of the interface. Users have the op-tion to specify ad hoc queries in a text field. Theycan also use drop down menus to specify optionalminimum and/or maximum reading levels and textlengths. Another optional drop-down menu allowsusers to constrain the general topic area of results. Aseparate screen allows users to specify a list of tar-get vocabulary words, some but not all of which arerequired to appear in the search results. For ease ofuse, the target word list is stored for an entire session(i.e., until the web browser application is closed)rather than specified with each query. After the usersubmits a query, the system displays multiple resultsper screen with titles and snippets.

5.1 Ranked versus Boolean Retrieval

In a standard boolean retrieval model, with AND asthe default operator, the results list consists of doc-uments that contain all query terms. In conjunc-tion with relevance ranking techniques, commercialsearch engines typically use this model, a great ad-vantage of which is speed. Boolean retrieval can en-counter problems when queries have many terms be-cause every one of the terms must appear in a doc-ument for it to be selected. In such cases, few orno satisfactory results may be retrieved. This issueis relevant because a teacher might want to searchfor texts that contain some, but not necessarily all,of a list of target vocabulary words. For example,a teacher might have a list of ten words, and anytext with five of those words would be useful to giveas vocabulary and reading practice. In such cases,ranked retrieval models are more appropriate be-

83

Figure 2: Screenshot of Search Interface for Finding Appropriate Readings.

cause they do not require that all of the query termsappear. Instead, these models prefer multiple occur-rences of different word types as opposed to multipleoccurrences of the same word tokens, allowing themto rank documents with more distinct query termshigher than those with distinct query terms. Docu-ments that contain only some of the query terms arethus assigned nonzero weights, allowing the user tofind useful texts that contain only some of the targetvocabulary. The REAP search system uses the IndriQuery Language’s “combine” and “weight” opera-tors to implement a ranked retrieval model for targetvocabulary. For more information on text retrievalmodels, see (Manning et al., 2008).

5.2 Example Query

Figure 3 shows an example of a structured queryproduced by the system from a teacher’s originalquery and constraints. This example was slightlyaltered from its original form for clarity of presen-tation. The first line with the filrej operator filtersand rejects any documents that contain any of a longlist of words considered to be profanity, which areomitted in the illustration for brevity and posterity.The filreq operator in line 2 requires that all of theconstraints on reading level, text length and qualityin lines 2-4 are met. The weight operator at the startof line 5 balances between the ad hoc query terms inline 5 and the user-specific target vocabulary termsin lines 6-8. The uw10 operator on line 5 tells thesystem to prefer texts where the query terms appeartogether in an unordered window of size 10. Suchproximity operators cause search engines to preferdocuments in which query terms appear near each

other. The implicit assumption is that the terms inqueries such as “coal miners safety” are more likelyto appear in the same sentence or paragraph in rele-vant documents than irrelevant ones, even if they donot appear consecutively. Importantly, query termsare separated from target words because there areusually a much greater number of target words, andthus combining the two sets would often result inthe query terms being ignored. The higher weightassigned to the set of target words ensures they arenot ignored.

6 Learner and Teacher Support

In addition to search facilities, the system providesextensive support for students to read and learn fromtexts as well as support for teachers to track stu-dents’ progress. All interfaces are web-based foreasy access and portability. Teachers use the searchsystem to find readings, which are stored in a cur-riculum manager that allows them to organize theirselected texts. The manager interface allows teach-ers to perform tasks such as specifying the orderof presentation of their selected readings, choosingtarget words to be highlighted in the texts to focuslearner attention, and specifying time limits for eachtext.

The list of available readings are shown to stu-dents when they log in during class time or forhomework. Students select a text to read and moveon to the reading interface, which is illustrated inFigure 4. The chosen web page is displayed in itsoriginal format except that the original hyperlinksand pop-ups are disabled. Target words that were

84

Figure 3: Example Structured Query. The line numbers on the left are for reference only.

chosen by the teacher are highlighted and linked todefinitions. Students may also click on any otherunknown words to access definitions. The dictio-nary definitions are provided from the CambridgeAdvanced Learner’s Dictionary8, which is authoredspecifically for ESL learners. All dictionary accessis logged, and teachers can easily see which wordsstudents look up.

The system also provides vocabulary exercises af-ter each reading for additional practice and reviewof target words. Currently, students complete cloze,or fill-in-the-blank, exercises for each target word inthe readings. Other types of exercises are certainlypossible. For extra review, students also completeexercises for target words from previous readings.Students receive immediate feedback on the prac-tice and review exercises. Currently, sets of the ex-ercises are manually authored for each target wordand stored in a database, but we are exploring auto-mated question generation techniques (Brown et al.,2005; Liu et al., 2005). At runtime, the system se-lects practice and review exercises from this reposi-tory.

7 Related Work

A number of recent projects have taken similar ap-proaches to providing authentic texts for languagelearners. WERTi (Amaral et al., 2006) is an in-telligent automatic workbook that uses texts fromthe Web to increase knowledge of English gram-matical forms and functions. READ-X (Miltsakakiand Troutt, 2007) is a tool for finding texts at spec-ified reading levels. SourceFinder (Sheehan et al.,2007) is an authoring tool for finding suitable textsfor standardized test items on verbal reasoning and

8dictionary.cambridge.org

reading comprehension.The REAP Tutor (Brown and Eskenazi, 2004;

Heilman et al., 2006) for ESL vocabulary takes aslightly different approach. Rather than teacherschoosing texts as in the REAP Search system, theREAP Tutor itself selects individualized practicereadings from a digital library. The readings containtarget vocabulary words that a given student needsto learn based on a student model. While the in-dividualized REAP Tutor has the potential to bettermatch the needs of each student since each studentcan work with different texts, a drawback of its ap-proach is that instructors may have difficulty coor-dinating group discussion about readings and inte-grating the Tutor into their curriculum. In the REAPSearch system, however, teachers can find texts thatmatch the needs and interests of the class as a whole.While some degree of individualization is lost, theadvantages of better coordinated support from teach-ers and classroom integration are gained.

8 Pilot Study

8.1 Description

Two teachers and over fifty students in two ESLcourses at the University of Pittsburgh used the sys-tem as part of a pilot study in the Spring of 2008.The courses focus on developing the reading skillsof high-intermediate ESL learners. The target vo-cabulary words covered in the courses come fromthe Academic Word List (Coxhead, 2000), a listof broad-coverage, general purpose English wordsthat frequently appear in academic writing. Studentsused the system once per week in a fifty-minute classfor eight weeks. For approximately half of a ses-sion, students read the teacher-selected readings andworked through individualized practice exercises.

85

Figure 4: Screenshot of Student Interface Displaying a Reading and Dictionary Definition.

For the other half of each session, the teacher pro-vided direct instruction on and facilitated discussionabout the texts and target words, making connec-tions to the rest of the curriculum when possible.For each session, the teachers found three to fivereadings. Students read through at least two of thereadings, which were discussed in class. The extrareadings allowed faster readers to progress at theirown pace if they complete the first two. Teacherslearned to use the system in a training session thatlasted about 30 minutes.

8.2 Usage AnalysisTo better understand the two teachers’ interactionswith the search system, we analyzed query log datafrom a four week period. In total, the teachers usedthe system to select 23 readings for their students.In the process, they issued 47 unique queries to thesystem. Thus, on average they issued 2.04 queriesper chosen text. Ideally, a user would only have toissue a single query to find useful texts, but fromthe teachers’ comments it appears that the system’susability is sufficiently good in general. Most ofthe time, they specified 20 target words, only someof which appeared in their selected readings. Theteachers included ad hoc queries only some of thetime. These were informational in nature and ad-

dressed a variety of topics. Example queries in-clude the following: “surviving winter”, “coal min-ers safety”, “gender roles”, and “unidentified flyingobjects”. The teachers chose these topics becausethey matched up with topics discussed in other partsof their courses’ curricula. In other cases, it wasmore important for them to search for texts with tar-get vocabulary rather than those on specific topics,so they only specified target words and pedagogicalconstraints.

8.3 Post-test and Survey ResultsAt the end of the semester, students took an exit sur-vey followed by a post-test consisting of cloze vo-cabulary questions for the target words they prac-ticed with the system. In previous semesters, theREAP Tutor has been used in one of the two coursesthat were part of the pilot study. For comparisonwith those results, we focus our analysis on the sub-set of data for the 20 students in that course. Theexit survey results, shown in 5, indicate that stu-dents felt it was easy-to-use and should be used infuture classes. These survey results are actually verysimilar to previous results from a Spring 2006 studywith the REAP Tutor (Heilman et al., 2006). How-ever, responses to the prompt “My teacher helpedme to learn by discussing the readings after I read

86

Figure 5: The results from the pilot study exit survey, which used a Likert response format from 1-5 with 1=StronglyDisagree, 3=Neither Agree nor Disagree, and 5=Strongly Agree. Error bars indicate standard deviations.

them” suggest that the tight integration of an edu-cational system with other classroom activities, in-cluding teacher-led discussions, can be beneficial.

Learning of target words was directly measuredby the post-test. On average, students answered89% of cloze exercises correctly, compared to lessthan 50% in previous studies with the REAP Tutor.A direct comparison to those studies is challengingsince the system in this study provided instructionon words that students were also studying as part oftheir regular coursework, whereas systems in previ-ous studies did not.

9 Discussion and Future Work

We have described a system that enables teachersto find appropriate, authentic texts from the Webfor vocabulary and reading practice. A variety oflanguage technologies ranging from text retrieval toPOS tagging perform essential functions in the sys-tem. The system has been used in two courses byover fifty ESL students.

A number of questions remain. Can languagelearners effectively and efficiently use such a systemto search for reading materials directly, rather thanreading what a teacher selects? Students could usethe system, but a more polished user interface andfurther progress on filtering out readings of low textquality is necessary. Is such an approach adaptableto other languages, especially less commonly taughtlanguages for which there are fewer available Webpages? Certainly there are sufficient resources avail-able on the Web in commonly taught languages suchas French or Japanese, but extending to other lan-guages with fewer resources might be significantlymore challenging. How effective would such a toolbe in a first language classroom? Such an approachshould be suitable for use in first language class-

rooms, especially by teachers who need to find sup-plemental materials for struggling readers. Are thereenough high-quality, low-reading level texts for veryyoung readers? From observations made while de-veloping REAP, the proportion of Web pages belowfourth grade reading level is small. Finding appro-priate materials for beginning readers is a challengethat the REAP developers are actively addressing.

Issues of speed and scale are also important toconsider. Complex queries such as the one shownin Figure 3 are not as efficient as boolean queries.The current system takes a few seconds to return re-sults from its database of several million readings.Scaling up to a much larger digital library may re-quire sophisticated distributed processing of queriesacross multiple disks or multiple servers. However,we maintain that this is an effective approach forproviding texts within a particular grade level rangeor known target word list.

Acknowledgments

This research was supported in part by the Insti-tute of Education Sciences, U.S. Department of Ed-ucation, through Grant R305B040063 to CarnegieMellon University; Dept. of Education grantR305G03123; the Pittsburgh Science of LearningCenter which is funded by the National ScienceFoundation, award number SBE-0354420; and a Na-tional Science Foundation Graduate Research Fel-lowship awarded to the first author. Any opin-ions, findings, conclusions, or recommendations ex-pressed in this material are the authors, and do notnecessarily reflect those of the sponsors.

References

Luiz Amaral, Vanessa Metcalf and Detmar Meurers.

87

2006. Language Awareness through Re-use of NLPTechnology. Pre-conference Workshop on NLP inCALL – Computational and Linguistic Challenges.CALICO 2006.

Jon Brown and Maxine Eskenazi. 2004. Retrieval ofauthentic documents for reader-specific lexical prac-tice. Proceedings of InSTIL/ICALL Symposium 2004.Venice, Italy.

Jon Brown, Gwen Frishkoff, and Maxine Eskenazi.2005. Automatic question generation for vocabularyassessment. Proceedings of HLT/EMNLP 2005. Van-couver, B.C.

Kevyn Collins-Thompson and Jamie Callan. 2005.Predicting reading difficulty with statistical languagemodels. Journal of the American Society for Informa-tion Science and Technology, 56(13). pp. 1448-1462.

Averil Coxhead. 2000. A New Academic Word List.TESOL Quarterly, 34(2). pp. 213-238.

S. Gupta, G. Kaiser, D. Neistadt, and P. Grimm. 2003.DOM-based content extraction of HTML documents.ACM Press, New York.

F. M. Hafiz and Ian Tudor. 1989. Extensive readingand the development of language skills. ELT Journal43(1):4-13. Oxford University Press.

Michael Heilman, Kevyn Collins-Thompson, Maxine Es-kenazi. 2008. An Analysis of Statistical Models andFeatures for Reading Difficulty Prediction. The 3rdWorkshop on Innovative Use of NLP for Building Edu-cational Applications. Association for ComputationalLinguistics.

Michael Heilman, Alan Juffs, Maxine Eskenazi. 2007.Choosing Reading Passages for Vocabulary Learningby Topic to Increase Intrinsic Motivation. Proceedingsof the 13th International Conferenced on Artificial In-telligence in Education. Marina del Rey, CA.

Michael Heilman, Kevyn Collins-Thompson, JamieCallan, and Maxine Eskenazi. 2006. Classroom suc-cess of an Intelligent Tutoring System for lexical prac-tice and reading comprehension. Proceedings of theNinth International Conference on Spoken LanguageProcessing. Pittsburgh, PA.

Thorsten Joachims. 1999. Making large-Scale SVMLearning Practical. Advances in Kernel Methods -Support Vector Learning, B. Schlkopf and C. Burgesand A. Smola (ed.) MIT-Press.

Chao-Lin Liu, Chun-Hung Wang, Zhao-Ming Gao, andShang-Ming Huang. 2005. Applications of LexicalInformation for Algorithmically Composing Multiple-Choice Cloze Items Proceedings of the SecondWorkshop on Building Educational Applications Us-ing NLP. Association for Computational Linguistics.

Christopher D. Manning, Prabhakar Raghavan and Hin-rich Schtze. 2008. Introduction to Information Re-trieval. Cambridge University Press. Draft available

at http://www-csli.stanford.edu/∼hinrich/information-retrieval-book.html.

Eleni Miltsakaki and Audrey Troutt. 2007. Read-X: Au-tomatic Evaluation of Reading Difficulty of Web Text.Proceedings of E-Learn 2007, sponsored by the Asso-ciation for the Advancement of Computing in Educa-tion. Quebec, Canada.

Matthew Peacock. 1997. The effect of authentic mate-rials on the motivation of EFL learners. ELT Journal51(2):144-156. Oxford University Press.

Kathleen M. Sheehan, Irene Kostin, Yoko Futagi. 2007.SourceFinder: A Construct-Driven Approach for Lo-cating Appropriately Targeted Reading Comprehen-sion Source Texts. Proceedings of the SLaTE Work-shop on Speech and Language Technology in Educa-tion. Carnegie Mellon University and InternationalSpeech Communication Association (ISCA).

88