summarizing skills
TRANSCRIPT
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 1/36
Summarizing with LSA-Based Feed back 1
Developin g Summ arization Skills through th e Use of LSA-Based Feedback*
Eileen Kin tsch, Dave Steinh art, Gerry Stahl, and
LSA Research Group**
University of Colorado
Cindy Matthews and Ronald Lamb
Platt Midd le School, Bould er Colorado
running head : Summarizing w ith LSA-Based Feedback
send correspondence to: Dr. Eileen Kintsch
Institu te of Cogn itive Science
Campus Box 0344
University of Colorad o
Boulder, CO 80309-0344
(303) 492-0736
email: [email protected] o.edu
fax: (303) 492-7177
(in pr ess). Interactive Learning Environments . [Special Issue, J. Psotka, gu esteditor].
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 2/36
Summarizing with LSA-Based Feed back 2
Abstract
This paper d escribes a series of classroom tr ials du ring w hich we d eveloped
Summary Street , an edu cational software system th at uses Latent Seman tic
Analysis to sup p ort wr iting an d r evision activities. Summary Street provides
various kinds of feedback, primarily about wh ether a student su mm ary
ad equately covers imp ortant sou rce content and fulfills other requ irements, such
as length . The feedback allow s stud ents to engage in extensive, ind epen d ent
pr actice in wr iting an d r evising w ithout p lacing excessive dem and s on teachers
for feedback. We first discu ss the un d erlying edu cational rationale, then p resent
some resu lts of the trials cond ucted w ith the system. We describe the
collaborative process am ong researchers and teachers which enabled the
development of a viable and sup portive edu cational tool and its integration into
classroom instruction.
Keyword s: Latent Semantic Analysis (LSA), edu cational technology
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 3/36
Summarizing with LSA-Based Feed back 3
Summary Street is an edu cational software system tha t uses Latent Seman tic
Analysis (LSA) to sup por t the reading an d w riting activities by wh ich stu d ents
develop and expan d th eir know ledge in new topic areas. Summary Street
determines the degree to wh ich a stud ent summ ary covers imp ortant source
content and conforms to requirem ents, su ch as length . It tells the stud ent w hat
information in the source is missing, provides comments on redu nd ancy,
extraneou s content and certain asp ects of mechanics. Its current op eration is
d escribed in m ore detail later. First, how ever, we d iscuss the u nd erlying
edu cational rationale and review the course of its research and d evelopm ent.
Text-based activities are ind ispu tably a major veh icle for acquiring basic
content know ledge in most school settings, across a ran ge of pedag ogical mod els,
from those that em ph asize trad itionally structured classroom s to those in w hich
stud ents direct their own p aths of inqu iry. One form of comp uter sup port for
comp rehension an d learning that ou r team has d eveloped uses LSA to prov ide
stud ents with imm ediate feedback on how well their su mm aries of informative,
expository texts cover the top ic they are w orking on . We intend for this tool to be
used by stud ents independ ently, though still within a classroom setting, so that
they can assess their own initial attemp ts to com pose and revise their sum mar ies.
We hope thereby to pr ovide students w ith more experience in extended writing
and revising, w hile leaving teachers m ore time for other kind s of edu cational
activities, such as coaching and m od eling w riting and su mm arization
techniqu es, providing ind ividu al help, planning an d delivering instruction,
evaluating final versions of stud ents’ wr iting an d other p rojects. Thu s, in n o
sense is the tool intend ed to r eplace the teachers’ role, for it is they w ho m ust
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 4/36
Summarizing with LSA-Based Feed back 4
teach the skills and, at least in our im plem entation, evaluate the final pr od ucts of
stud ents’ writing. Even thou gh stu dents are able to use the sum marization tool
on their ow n, we w ant to emp hasize that it is a system that seeks to comp lement
classroom instru ction, rather th an existing as a stand -alone system. Its pu rp ose is
to reinforce wh at is being tau ght rath er than just p rovid e an adjunct learning
activity. Thu s, in d esigning our first p rototyp e, called State the Essence , we began
w ith the prem ise that this wou ld take place in collaboration w ith teachers w ho
were the intended users. The sum marization tool in its many transformations
and its integration into th e instructional cur riculu m r epresents a collaborative
effort of researchers and teachers.
The cur rent system evaluates only th e completeness of the conten t, for the
mo st part, leaving other imp ortan t aspects of w riting, such as sentence stru cture,
organ ization an d style, for traditional instru ctional metho d s. N onetheless, we
believe that in ad d ition to imp roving th eir writing skill, stud ents w ill benefit
metacognitively from w orking independ ently, guided by the imm ediate
feed back they receive. With frequen t p ractice in assessing an d revising the
content of their su mm aries, w e believe that stud ents w ill also become m ore
attun ed to their own thinking an d w riting processes; they w ill be mo re likely to
realize wh at they do and do n ot und erstand and better able to express wh at they
mean in w riting.
Imp ortance of Summ arization as a Learnin g Skill
Ou r initial discussions qu ickly conv erged on su mm arization as the kind of
learning activity that LSA technology could effectively su pp ort an d that
conformed well with th e teachers' instru ctional goals. The sixth-grade classroom s
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 5/36
Summarizing with LSA-Based Feed back 5
in wh ich the tool is being tested em ploys a pro blem-based learning ap pr oach for
instructing the d istrict m and ated curriculum. Learning h ow to sum marize text is
emp hasized through out the school year as a crucial stud y skill that helps
stud ents acquire a basic un d erstand ing of d ifficult and n ovel subject matter
wh ich they can then app ly to solving p roblems or developing a p roject.
Sum marizing is more constrained than an open-ended writing task, with w hich
youn g stud ents often flound er, and it has a nu mber of advantages over simp ly
reading text and answering “comp rehension questions”, including the following:
• Sum mar izing n ot only prov ides practice in extended expository wr iting, it
also teaches imp ortan t stud y skills, such as identifying imp ortan t conten t
and separating m ain ideas from d etails. The fact that stu d ents at this age
tend to high light everything in a text – creating a “sea of yellow” – is
symp toma tic of their inability to d o this. This hap pen s especially w hen
stud ents are dealing w ith content that is comp letely new to them.
• Sum mar izing for a given pu rp ose (e.g., to wr ite a repor t on Mayan
religious beliefs) requires even d eeper th inking an d analysis to select the
relevant information.
• Sum marizing is a way to develop solid u nd erstand ing of comp lex
material and also to articulate one’s un d erstand ing so that it can be shar ed
with others. The teachers with w hom we w ork have noted clear
differences in d epth of u nd erstanding of topics that stud ents have
sum marized as opp osed to those they have only read about. Stud ents
app ear to retain app reciably m ore information over longer p eriods of time
if they hav e sum ma rized it, and in classroom d iscussion they d isplay an
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 6/36
Summarizing with LSA-Based Feed back 6
ow nersh ip of those topics, w hich show s up in their ability to contribu te
detailed an d well reasoned ideas.
• H aving to expr ess content adequ ately yet concisely makes stud ents awar e
of the need to learn sum marization strategies that go beyond just ad ding
and d eleting single w ord s, ph rases or sentences. This aw areness becomes
a starting p oint for introd ucing stud ents to higher-level strategies, such as
how to reformu late text content by combining several ideas in a single
sentence and generalizing across details.
• Sum mar izing requ ires active meaning constru ction to a mu ch greater
d egree than choosing a respo nse on a m u ltiple-choice recognition test, or
even than w riting short answ ers to isolated q uestions. Thu s, not only is
sum ma ry wr iting an effective means to constru ct and integrate new
know ledge, it is also a m ore auth entic method for assessing w hat stud ents
do and do n ot und erstand than traditional comp rehension tests.
The Use of LSA to Provide Writers with Content Feedb ack
As the rationale as w ell as techn ical details about LSA can be foun d in variou s
other p ub lications, we w ill not review th em here (please see Land au er & Du ma is
(1997), Land au er (1998), Land au er, Foltz & Laham (1998), as well as the
introd uctory article by Land au er & Psotka in this volum e). Essentially, LSA is an
au tomatic statistical meth od for repr esenting the meaning of word s and text
pa ssages based on the analysis of a large amou nt of textual inpu t. A semantic
space is generated in w hich word s, sentences, and w hole texts can be
rep resented a s vectors. H ow closely related th ese vectors are to each other is
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 7/36
Summarizing with LSA-Based Feed back 7
measu red b y the cosine betw een them . We use this cosine m easure to calculate
wh at feedback to provide w riters.
The most general LSA space available today is based on an inp ut of abou t
11M w ord s from carefully selected texts that form a rep resentative samp le of
wh at a single stud ent finishing h igh school might have read d uring h is or her
school year s. This sp ace is sufficient for ou r an alysis, except for technical topics.
Thus, for stud ents writing on the functioning of the pulmon ary and cardiac
systems, or stud ents w riting o n Meso-American civilizations, the general space
does n ot have enou gh information to m ake the fine d istinctions requ ired. It has
some basic inform ation abou t the Inca and Maya cultures, for instan ce, bu t not
enou gh to tell apart d etails of their religion or ag ricultu ral p ractices. Therefore, a
specialized space mu st be constru cted in or der to u se LSA. For instan ce, the
Heart space discussed below w as constructed from an inpu t of 830 docum ents
comp rising a bou t 17,688 wo rd s describing the fun ction of the h eart. The Meso-
Am erican space was based on 530 d ocum ents, com pr ising 46,951 w ord s dealing
with this topic. At the mom ent we do not yet have a good und erstand ing when
specialized spaces are required and w hen th e general space suffices. Thus, ad h oc
decisions m ust be m ade based on the p erformance of the system.
Becau se misspelled w ord s are not considered w ord s by LSA, we first have to
correct spelling. For this p ur pose, all misspelled wo rd s (or rather, all strings LSA
d oes not recognize) are flagged with asterisks, and the stud ent is asked to m ake
sur e that they are sp elled correctly. In p rinciple, althou gh th is is not don e in th e
present system, a stand ard spell checker can provide th e stud ent w ith
alternatives, and LSA can select the m ost pr om ising alternative(s) by looking at
the cosine betw een each alternative identified b y the sp ell checker and the
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 8/36
Summarizing with LSA-Based Feed back 8
immed iate neighborhood of the word . Most likely, words w ith a higher cosine to
the context are th e right choice.
Content feedback is provid ed in th e following man ner. Sup pose stud ents
are asked to sum mar ize a text T containing the sections {T 1, T 2,…., T k }. The
teacher requires that each of these sections be covered in the stud ent’s summ ary.
What w e do is to comp ute th e cosine, C i, between the sum mary a stud ent wrote
and each of the sections T i. If C i ≤ t i, where t is an emp irically d etermined
thresh old value, the stud ent is told that section T i is not ad equ ately covered in
the sum mary. The student then h as the option to look at the app ropriate section
of the text on the comp uter screen and ad d som e material about this section to
the sum mary. If C i ≥ t for all sections, the stud ent is told that h e or she h as now
covered all par ts of the text.
Since the teachers require sum mar ies to be of a given w ord length to
avoid extensive cop ying (about on e quar ter of the source text), stud ents are told
how man y word s they have written so far and w hether this is within the allowed
limits. If the text is too long, the stu d ent is given tw o kind s of feedb ack to help
shorten it. One the one hand , irrelevant sentences in the su mm ary are identified.
The cosine is com pu ted betw een each sentence in the su mm ary and the text as a
w hole. If it is below som e lower th reshold, th e sentence is iden tified as (possibly)
irrelevan t. This relevancy check tend s to pick up sentences that are tru ly
irrelevan t (such as “I hop e you like the sum mar y I wrote”) or sentences that refer
to obscure d etails in th e text that are not ap prop riate for a sum mary. On the
other hand , redun dan t sentences are identified by comp uting the cosines among
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 9/36
Summarizing with LSA-Based Feed back 9
all sentences in th e sum ma ry. If a cosine is greater than some u pp er limit, the
tw o sentences are highlighted in the text and th e stud ent is told to inspect them
for the pu rpose of combining them or d eleting one. Sixth-grade stud ents tend to
rep eat themselves, so this is a very u seful check. No te, how ever, that both th e
relevance and th e redu nd ancy check occasionally p ick u p false positive:
sentences, for examp le, w ith several overlapp ing wor d s, bu t distinct mean ings.
This has the positive result that stu d ents mu st critically evaluate the compu ter’s
adv ice and decide w hether they agree w ith it or n ot. Upper an d lower limits for
the relevance and red un d ancy checks are, once again, set emp irically. For
examp le, sentences with a cosine to the text that are below .30 migh t be termed
irrelevan t, and sentences with a cosine greater than .80 between them selves
might be termed redun dant.
The system itself is thu s quite simp le. H ow ever, what w as not simple was
to determine the best ways to provide this kind of feedback to stud ents and the
op timal sequencing of this feedback, as described below.
History of Trials Using Sta te t he Essence : Fall 1997 – Fall 1998
Instruction
Two team-tau ght classes participa ted in trials using an early version of the
sum ma rization tool called State the Essence d uring th e 1997-1998 school year, and
a su bsequen t trial took place in th e fall of the next academ ic year. The system
was d esigned to sup port stud ents’ sum mary w riting in three curricular un its,
each lasting abou t th ree-to-four w eeks: Energy Sour ces (Septem ber, 1997 and
Septem ber, 1998), Ancient Civilizations of th e Western H emisph ere (Janu ary,
1997) and The H um an Circulatory System (Ap ril, 1998). Stud ents first comp osed
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 10/36
Summarizing with LSA-Based Feed back 10
their sum maries using a w ord p rocessor or pen and pap er in ad vance. They then
pasted or typed them into State the Essence in ord er to receive feedback on h ow to
revise them. For the trial on the circulatory system, w e collected su mm aries that
stud ents wrote u sing trad itional means as well as those written w ith State the
Essence , wh ich allowed u s to ma ke within-subject comp arisons. Ho w ever, our
ma in goal du ring this initial period w as to test the system rath er than to collect
learning and performance data.
1. Sou rces of Energy . In ad d ition to teaching stud ents abou t the new
content, du ring the first un it the teachers’ instru ction introd u ced stu d ents to the
concept of sum mar ization an d th e app rop riate strategies. The teachers’
instru ction inclu ded d irectly explaining the strategies and their pu rp ose, together
with m odeling the strategies and class discussion of good and poor examp les of
summ ary w riting.
Stud ents read 10 brief texts (two to tw o-and -a half pag es) abou t different
sour ces of energy (non renew able: coal, natu ral gas, nu clear, petroleum , pr opan e;
and renewable: biomass, geothermal, hydrop ower, solar, wind) and wr ote one
sum ma ry (75 - 200 w ord s) of each energ y typ e. Stud ents used this task as the
starting p oint for their projects, which involved becoming an expert in on e
energy sou rce, organ izing a science station and teaching th e subject to other
stud ents in small groups.
2. Ancient Civilizations. For th is unit stud ents w ere requ ired to
sum ma rize three texts (each about tw o-and -one-half to three pages) about the
Maya, Aztec and Inca civilizations, again to d evelop basic know ledge abou t the
cultures. The sum mar ies were to be betw een 200 to 300 wor ds long. Each class
then d ivided into thr ee group s, each focussing on one of the cu ltures, and each
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 11/36
Summarizing with LSA-Based Feed back 11
mem ber of a group researched on e particular aspect of the culture (e.g., history,
religion, artistic or scientific contribu tions, social stru cture). Fina lly, each grou p
ma d e a joint presentation w ith visual prop s to the class as a who le, each memb er
filling in a p iece of the top ic in jigsaw fashion. The su mm arization instru ction
this time focused on h igher-level strategies, such as sentence combining a nd
constructing generalizations to achieve conciseness. Stud ents pr epared two of
their summ aries in the traditional mann er, using a w ord processor or p en and
pap er, and revised a third sum mary gu ided by feedback from the sum marization
software.
3. Circulatory System. Un like the pr eced ing u nits, the instru ctional focus
here was p rimarily on d eveloping a d eep un derstand ing of the content - a
challenging top ic w ith a great d eal of un familiar technical vocabulary an d
d ifficult concepts. Sum ma rization of tw o texts about the lun gs and the heart w as
used to help stud ents integrate this information and to assess their conceptu al
un derstand ing of the d ual-loop circulatory system. The sum maries were to be
150-250 w ord s in length, and stud ents used State the Essence to work on one of
these summ aries. They w rote the other sum mary using trad itional means.
Evolution of State the Essence
Initial trials with State the Essence wer e beset by techn ical prob lems from
overloading th e system with too m any simu ltaneous su bmissions. How ever,
these pr oblems w ere overcome in ou r later trials. In gen eral, the school trials
with th e sum marization software w ere a success in term s of student enthusiasm
and teacher satisfaction, at least to some d egree: the system w orked w ell, was
relatively easy to learn, an d using State the Essence did not interfere with
stud ents’ learning of the content (there wa s no significant d ifference between
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 12/36
Summarizing with LSA-Based Feed back 12
sum ma rizing cond itions in scores received on a short-answ er test on th e unit of
stud y). How ever, as m entioned, the p urp ose of these school trials was not a
formal evaluation of the system bu t rather to further d evelop and refine it.
There are three classes of changes that w e explored:
1. H ow the stu d ent’s wr iting is to be evaluated by LSA: There are several
option s here; for examp le, a given essay can be ma tched aga inst a set of
pre-graded essays, or against an expert sum mary p repared by the teacher
or expert writer. In the end w e adop ted a m ore practical method th at
wou ld only require a teacher to subm it the text to be sum marized,
subd ivided into topic sections, a m ethod that has been incorporated into
the later versions of the system.
2. What feedback to give the stud ent, and in w hat ord er: It is easy to
overwh elm users and confuse them w ith the rich feedback the system is
able to provide. Over the course of the year w e experimented with several
d ifferent feedback formats before arriving at a system that is somew hat
constrained yet still flexible to u se. “Less is more” w as ou r take-hom e
message - less feedback and more su pp ort.
3. H ow to em bed ou r system into classroom instru ction: Use of the
sum ma rization tool as a stand -alone system is rather inefficient for
mid dle-school stud ents. Most stud ents at this level need explicit
instruction on how to sum marize, and how to revise. Furthermore,
available techn ology has m ad e it difficu lt to use th e system in a classroom
w ithout taking too m u ch time aw ay from other instru ctional activities.
Ou r trials therefore took place over on e or two sessions with th e entire
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 13/36
Summarizing with LSA-Based Feed back 13
class – a pr actical necessity, thou gh n ot an op timal wa y to learn r evision
skills.
Evaluating the sum maries
Ou r initial problem in d elivering feedback to the stu d ents was to d ecide
w hat text to use as a basis for comp arison. Several different ap pr oaches to
evaluating college stud ents’ essays are described in Land au er, Foltz, and Laham
(1998), some of which w e also app lied to eva luating th e stud ents’ sum mar ies.
One app roach is to comp are a summ ary to a corpu s of previously graded
sum ma ries. The sum mar y w hich is the closest match in term s of the LSA cosine
becomes the basis for assigning a g rad e of A, B, or C, and so on. Since we had not
yet accumu lated a set of graded su mm aries to draw on, this option was n ot open
to u s. Hence, w e first tried m atching the sixth-graders’ summ aries against a set of
four or five sum mar ies w ritten by expert writers (teachers and r esearchers).
Given that even expert w riters d o not comp letely agree on wh at content to
includ e or exclud e, the stud ent’s overall score w as based on th e best fit (i.e., the
highest LSA cosine) to one of the exp ert texts. Section scores were based on a
comp arison of the sum mary to each section of a “golden” sum mary that
incorporated the m ain content in all the expert sum maries. Although this method
w orked q u ite w ell, pu tting together a set of expert sum mar ies for each novel text
proved too cum bersome in the long run.
An alternative basis of comp arison is to u se the sou rce text itself. A
holistic score can be obtained from th e cosine between th e stud ent’s sum mar y
and the original source text. In ad d ition, section scores ma y be derived by
d ividing th e text into d istinct topic sections, app roximately equal in length, and
comp aring th e entire sum mar y to each of these sections. As described earlier in
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 14/36
Summarizing with LSA-Based Feed back 14
this pap er, a set of emp irically determ ined thr esholds is used as the basis for the
feed back given to the stud ent on h ow ad equ ately each section was covered. The
sum ma ry “p asses” wh en all sections h ave m et the criterion for each section
w ithin the given length constraints. This meth od u nd erlies all the versions of the
sum marizing software described here.
Presenting the Feedback
LSA-based feedback goes far beyond other form s of autom atic feed back, such as
spelling an d g ram mar checks, by evalu ating the seman tic content of a piece of
w riting. For essays and sum mar ies, it can tell the writer w hether or n ot all the
important su btopics have been covered and wh at kind of information is missing;
it can p oint out sentences that app ear to have too m uch overlap in content with
each other or w ith the original text; and it can su ggest sentences that seem to
hav e little relevance to th e top ic of the text.
In ad dition to th is conten t information, in ou r initial trial on Energy
Sources we pr ovided stu dents w ith feedback on the length of their sum maries.
Length constraints across all three trials varied between 100 to 300 word s for
texts that ran ged from abou t 800 to 1450 word s. Stud ents received a n overall
score weighted to reflect appr opr iateness of length, the ad equa cy of section
coverage and overall conten t coverage. In ad d ition, they could requ est checks for
(a) red un d ancy, (b) relevance (both based on a comp arison of sentences in th e
sum ma ry w ith those in the original text), and (c) rep etition (based on a
comp arison of all sentence pa irs in the sum mar y). Ou r sixth-grad e stud ents,
althou gh ap preciative and highly motivated, seemed confused an d floun dered in
their attemp ts to revise their sum ma ries. In ad dition to solving various technical
pr oblems, it was clear that w e needed to pr ovide better editing tools, a clearer
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 15/36
Summarizing with LSA-Based Feed back 15
presentation, and m ore supp ort for sum marizing and revising both within the
system and through classroom instruction. We especially n eeded to p resent the
feedback in a w ay that w as easier to un derstand than th e set of nu merical scores
that w ere initially presented simu ltaneously.
In ou r second tr ial on An cient Civilizations th e feedba ck was given in
three stages, accessed by the u ser’s requ est first for gen eral feed back, then
successively m ore. The gen eral feedback includ ed length ( too long, too short ), an
overall score, and ad equa te/ inad equa te section coverage, as before. Requ ests for
mo re feedba ck first displayed irrelevant an d r elevan t sentences (the latter were
praised); then, at an adv anced level, feedback was pr ovided on redu nd ant
sentences (sum mary sentences with too mu ch overlapping content). In add ition,
we ad ded an overview of sum marization strategies to the Introd uction to State
the Essence and hyp erlinks to further hints and examp les. Links were also
pr ovided to the Maya, Aztec, and Inca sour ce texts and to add itional backgroun d
information.
The results of this classroom trial were both encou raging an d r evealing of
significant w eaknesses in the system. Again, the overall point score was a gr eat
motivator: stud ents were challenged to try to imp rove their scores and remained
focused on th e task. H ow ever, the scores were n ot always reliable, tend ing to be
inflated and too sensitive to sm all local variations. Sentence level feedba ck w as
especially prob lematic, w ith too many inap pr op riate flags (both good and bad ),
and d ifficult to use because pr oblematic sentences were p resented in a list, out of
context and on a sep arate screen from the w riter’s textbox. Presenting misspelled
w ord s as a list posed similar d ifficulties for m aking corrections. Even thou gh
pr esented in stages, or at different levels, stud ents w ere still overw helmed by the
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 16/36
Summarizing with LSA-Based Feed back 16
am oun t of feed back they received and often dismayed at the mu ltiplicity of
problems to deal with. Furth er, many stud ents needed extensive and quite
explicit guidan ce on ho w to m ake mean ingful chan ges in revising their
sum maries; in p articular, they needed to be show n how to generalize across
sentences or how to combine ideas from several places into a single sentence in
the context of their ow n w ork. This need clearly goes beyond w hat LSA-based
feed back prov ides, but high lights an area w here the teacher’s classroom
intervention can be helpful.
Ou r next attemp t to imp rove the system consisted in greatly simplifying
the feedback, both w hat was p rovided an d h ow it was pr esented. Thu s, for the
un it on th e Circulatory System, feedback consisted on ly of a p oint score (0-100
po ints); length ( too short , too long , or about right ); an eva luation of the conten t of
each section ( good , ok , needs improving , or missing ); and listing th e w eakest section
w ith a hyp erlink to th at section of the source text. The same version of the system
w as used again w ith minor chan ges for the fall 1998 un it on Energy. A screenshot
showing the first feedback page for a summ ary on hyd ropow er is shown in
Figure 1.
Insert Figure 1 About H ere
This version w as easier to u se, althou gh th e overall conten t scores w ere
still not sufficiently reliable, and interactions w ere som etimes confusing. For
examp le, stud ents were frequen tly fru strated to see large d ecreases in their point
score w hen content w as cut to stay within th e length constraints. Indeed , the
d ifficulty of balancing the n eed to be comp lete and the need to be concise
revealed again a need for more exp licit instru ction of higher -level summ arization
strategies in th e classroom. It is still imp ossible to p rovid e au tomatically the k ind
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 17/36
Summarizing with LSA-Based Feed back 17
of concrete help often need ed in the context of a stud ent’s own su mm ary
without, so to speak, giving the answ er away. In other word s, our comp uter tool
cannot yet interact w ith a stud ent abou t their writing like a hu man tutor , eliciting
an ap pr op riate respon se throug h carefu lly calibrated q uestions. How ever, it can
make both teachers and students aw are of where the gap s lie in skill and
un derstand ing, which can then be add ressed ind ividu ally or in general class
discussion.
To sum mar ize, the goals for ou r sum mar ization tool consisted of the
following:
• to prov ide sup por t for a challenging activity that fosters both d eep learning of
d ifficult new content and pr omo tes writing skills;
• to give students extended practice in w riting and revising sum maries wh ile
relieving teachers from the bu rd en of reviewing an d gr ad ing successive dr afts;
• to motivate stud ents to work hard and ind epend ently by providing immediate
and individualized feedback on h ow to revise their writing.
H ow w ell d id th e system fulfill these expectations? As stated p reviously,
we h ave a m odest amou nt of formal emp irical results at this point and quite a lot
in the way of informal observations and feedb ack from b oth teachers and
students.
Emp irical results w ith State the Essence
Du ring ou r classroom trials with State the Essence in the sp ring of 1998 we
collected d ata comp aring the sum mary scores award ed by th e system w ith those
of hu man scorers (Ancient Civilizations). In ad d ition, w e examined the effects on
learning of sum marizing with State the Essence vs. using trad itional mean s in a
w ithin-subjects design in th e un it on the Circulatory System. And in the fall of
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 18/36
Summarizing with LSA-Based Feed back 18
1998 w e again comp ared teacher assigned scores w ith LSA scores for su mm aries
of two d ifferent energy top ics.
1. Comp arison of LSA scores with h u man grad ers: Ancient Civilizations.
For this un it stud ents sum mar ized texts on all three cultures, one using State the
Essence for feedback on rev ision. Stud ents w ere allow ed to choose the culture they
w ished to sp ecialize in. Most stud ents chose to sum mar ize the Inca text. For the
first comp arison w e derived the LSA cosine between the stud ent sum mary and the
text the stud ents had read. We then comp ared this cosine w ith the overall content
grad e assigned by an ou tside teacher (LK) to the stu den ts’ Inca and Az tec
sum ma ries. For these 50 sum ma ries, the correlation betw een the teacher grad e and
the LSA cosine was r = 0.64. The correlation betw een a second scorer (EK) and the
teacher w as r = 0.69. Therefore, LSA scores are qu ite comp arable to h ow an
experienced teacher rates these sum mar ies.
The second comp arison was intend ed to assess w hether LSA could m atch
a given senten ce to a p articular section of the source text as well as hu man
grad ers. Thus, the same two expert gr ad ers (LK & EK) were show n 119
ran d omly chosen sentences and asked to choose w hich of the five sections of the
Aztec text the sentence was d escribing. For each senten ce the LSA cosine w as
comp u ted against each of the five sections, and the section w ith the highest
cosine w as considered to be LSA’s “choice.” The tw o expert gr ad ers w ere in
agreement for 109 of the sentences (91.6%). LSA agreed w ith the first gra d er on
101 of the sentences (84.9%) and with the second grad er on 99 of the senten ces
(83.2%). Therefore, LSA d oes almost as w ell as hu man s at d etermining the sou rce
of know ledge for a given sentence, a fact that w e hop e can be u seful in d esigning
futu re versions of the system.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 19/36
Summarizing with LSA-Based Feed back 19
2. Posttest scores from su mm arizing w ith State the Essence versus with
traditional means: Hu man Circulatory System. During th e unit on the H um an
Circu latory System, 39 stud ents from tw o classes sum ma rized tw o texts, one on
how the lungs function and one on the h eart and th e circulation of blood in the
body. Each stud ent wr ote and revised a sum mary on one of the topics by
conventional means (using pen an d p aper or w ord p rocessor) and one using the
State the Essence software. To see if there were d ifferences in how well stud ents
had learned the material about each topic, we compared their scores on an end-
of-un it, short-answ er test on the h um an circulatory system w ith respect to these
two topics. We foun d no d ifference in students’ und erstanding of the tw o topics
related to h ow the sum maries were wr itten, although one class performed
consistently better than th e other on all questions. A com par ison of the average
grad es (0-10 points) given by two outsid e teachers (LK & AW) likewise show ed
no d ifference in qu ality of the stud ents’ sum mar ies related to cond ition. The
average gra d es were 6.80 for traditionally written sum mar ies and 6.74 for the
sum ma ries w ritten with LSA-based feedb ack. The agreement between the tw o
hu man grad ers was r = .59. Thu s, based on evidence from a single trial, the
sum marization software did n ot app ear either to benefit or to harm stud ents’
learning or writing.
3. Comp arison of LSA and hu ma n grad ers: Energy u nit. Fifty-six stud ents
wr ote their two required sum maries on chosen topics as homework an d u sed
State the Essence to revise them. The averag e correlation based on the grad es of
the tw o classroom teachers (CM & RL) with LSA scores for four of the texts
(biom ass, hyd rop ow er, petroleum, & pr opan e) w as quite high: r = 0.88.
H ow ever, the average correlation between teacher and LSA grad es was qu ite low
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 20/36
Summarizing with LSA-Based Feed back 20
for the r emaining six topics: r = 0.32. In pa rt th is low correlation betw een LSA
and teacher grad es is du e to missing data (the summ ary topics were un equally
d istribu ted am ong th e two classes). Add itionally it results from the fact that State
the Essence used a single thresh old for all topics which, how ever, are not equal in
terms of their conceptua l difficulty. Hen ce, for som e topics stud ents’ sum ma ries
received h igher scores from LSA than th e teachers thou ght th ey deserved .
Evaluation of the system based on classroom o bservations
Despite overall encourag ing comm ents on State the Essence by both teachers at
the end of the year, our concern abo ut th e un reliability of the overall score
remained. In order to avoid frustrating the stud ents, we w anted to m ake it
po ssible to obtain 100 points. How ever, this often m ad e it too easy to reach a
high score, wh ich th en d iscouraged stud ents from continu ing to revise their
work. As m entioned earlier, minor changes often r esulted in un reasonable jum ps
in the score. The stud ents tend ed to regard their scores as an overall measu re of
w riting qua lity. Hence, once they had reached 100 points or w ere close enou gh,
they often did n ot review w hat they had written and consequently were up set
w hen th ey received poo r grad es from th e teachers d ue to lack of organ ization
and poor writing style. These observations led u s to qu estion w hether an overall
score was a good kind of feedback to pr ovide. Many stud ents treated th e score as
an en d in itself, trying to increase it by the cheap est means p ossible, rather than
focusing on imp roving their wr iting. These problems su ggested that the feed back
should be displayed in a form that w as more concrete and easier for students to
use tha n th e 100-point score comb ined w ith textu al pointers in State the Essence
(good, ok, needs more work , etc.).
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 21/36
Summarizing with LSA-Based Feed back 21
History of Trials Using Summary Street : Sp rin g 1999
In 1999 we chan ged the n ame an d mod ified the interface to reflect a m ajor
revision in our ap proach to prov iding stud ents feedback on th eir sum maries.
After typing their sum mar ies into the textbox, stud ents now receive the
following k ind s of feedb ack:
1. Misspelled w ord s are highlighted an d can be corrected in th e textbox; the
stud ent’s sum mary is automatically saved by th is operation.
2. The request for feedback retur ns a grap hic d isplay ind icating the length of
the sum ma ry and how w ell the content of each section of the original text
has b een covered (see Fig. 2). The display for content coverage consists of
horizon tal green bars extending ou t to a vertical line symbolizing the
thresh old. The w eakest section is ind icated, and a hyp erlink is prov ided
that th e wr iter can u se to access that top ic section in th e original text.
Instead of an actual w ord coun t, length is shown by a vertical bar on the
left, with bisecting lines ind icating the p rescribed m inimu m a nd
maximum . A green bar is displayed if the sum mary length is within th ese
limits, while a red bar is show n if the summ ary is either too shor t or too
long. Praise is given on ce the su mm ary h as passed the criterion for
content coverage for all sections.
3. Furth er help for revising is available at this point, for examp le, if the
sum mary exceeds the p rescribed length, in the form of a red un dan cy
check an d a relevance check. These tools help stu d ents locate sentences
that have overlapp ing content or seem not very related to th e topic being
sum marized and that wou ld be possible cand idates for d eleting or
collapsing togeth er.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 22/36
Summarizing with LSA-Based Feed back 22
4. Finally, a “Format for Printing” bu tton allows the stu d ent to obtain a
dou ble-spaced version to p rint out, review an d hand in to the teacher.
Emp irical results w ith Summary Street
The new system functioned quite well du ring tw o classroom trials in spring
1999. Stud ents w ere better able to deal with th e feedback on th eir own an d th is
helped th em to stay focussed on their w riting for extend ed p eriods of time.
Although we still see ways to imp rove the system, futu re changes should be
fairly minor on es. With a stable system now in place w e have begu n m ore formal
testing of the system than w as possible un til now .
In trials that took p lace du ring sp ring 1999 two classes of the sixth-grad e
stud ents used Summary Street to comp ose or revise some of their sum mar ies on-
line, guided by the feedback, and other sum maries using a word processor or
pen an d p aper. Fifty-tw o sixth-grad e stud ents participa ted in both the trial on
Ancient American Civilizations and the trial on the Hu man Circulatory System.
Classroom instruction wh ich incorpora ted u se of the Summary Street software
followed the sam e procedure as in th e trials the p revious sp ring w ith State the
Essence . Thus, as part of their learning activities du ring the u nit on Ancient
Civilizations, stu den ts were required to w rite sum ma ries on three cultures -
Maya, Aztec and Inca – one u sing Summary St ree t, the others by hand or on a
word processor. The stud ents prepared rough drafts of their sum maries as
hom ewor k and revised them in class. For the Circu latory System u nit, stud ents
comp osed and revised both of their summ aries of texts about the heart and lungs
on-line, either u sing the sum marization software or a word processor, with the
experimental conditions (Summary Street or w ord processor) coun terbalanced.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 23/36
Summarizing with LSA-Based Feed back 23
1. Ancient Civilizations sum ma ries. The two classroom teachers each
scored half of the trad itionally written sum mar ies and half of the Summary Street
sum ma ries on a 10-point scale, bu t they w ere not blind to experim ental cond ition
since the m ain pu rp ose of this trial w as to try out th e new system . The results,
w hich are presented in Figu re 3, show for the first time a significant adv antage
for the Summary Street condition: Grad es assigned by the teachers are
significantly higher for the Inca sum mar ies w ritten with Summary Street than
those w ritten by h and or a w ord pr ocessor (t(50) = 2.47, p = .02). Interestingly,
both teachers and stud ents considered th e Inca text the m ore d ifficult of the
three, wh ich is confirmed by the lower m ean grad es these sum mar ies received
(8.02 for Inca sum maries vs. 8.94 for Aztec and 8.79 for Maya ).
Insert Fig. 3 about h ere
2. Circulatory System trial. The results obtained from th is trial consist of
stud ents’ Heart and Lungs sum maries and their scores on a posttest at the end of
the u nit. Sum ma ries of the tw o texts w ere scored on a 10-point scale by the tw o
classroom teachers w ho w ere not aw are of students’ identity and experimental
cond ition. Overall, the Lung s sum ma ries received lower gr ad es than those of the
H eart texts (mean grad e = 3.39 vs. 4.00, respectively), confirming ou r im pr ession
that th is text, althou gh sh orter, was the m ore difficult of the tw o. Quite stringen t
length constraints also ad d ed to th e difficulty of sum ma rizing this text. It is
noteworthy, therefore, that the Lungs su mm aries comp osed w ith Summary Street
received significantly higher grad es than th ose comp osed on a w ord p rocessor
(t(50) = 2.32, p = 0.02 ), thu s confirm ing th e “har d text effect” w e had foun d w ith
the Inca text in th e pr evious trial. There was n o d ifference in sum mar ies of the
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 24/36
Summarizing with LSA-Based Feed back 24
H eart text, regardless of how they w ere written. These results are show n in
Figu re 4. The p osttest grad es did not d iffer significantly across text or cond ition.
Insert Figure 4 about h ere
The results of these two p relimin ary trials sug gest that the content
feed back delivered b y Summary Street is especially helpful wh en stud ents are
faced w ith mor e difficult task d eman d s or a hard er text. In a recent trial we hav e
furth er investigated the effectiveness of the Summary Street with hard text in
mo re tightly controlled circum stances. More sp ecifically, we comp ared the
revision process, and its time course, wh en stud ents are gu ided by content
feed back versus w hen th ey only receive feedback on length, such as they wo uld
get from a w ord pr ocessor. Preliminary resu lts are very encoura ging, but since
the analyses are not yet com plete, they m ust be repor ted elsewhere. In fu tur e
pr ojects, we h ope to replicate the resu lts we obtained w ith d ifficult text in an
older p opu lation of college stud ents. Finally, we w ou ld like to investigate the u se
of this software in collaborative wor k sessions as opp osed to ind ividu al work
sessions. Ou r informal observations d ur ing the Spring 1999 trials sug gest that
Summary Street could pr ovide an especially rich context for collaborative
learning.
Inform al observ ations of edu cational benefits: 1998-1999 trials
Based on both our observations and w hat the teachers have reported to us w e
believe that the n ew ver sion of the LSA-based su mm arization tool has the
po tential to effectively enh ance wh at teachers teach in a m eaningful w ay.
Providing students with man y opp ortunities to do extend ed w riting w ith
feed back abou t their w riting is one of the m ost effective tools for h elping th em
imp rove their writing. Such op por tun ities are necessarily red uced if stud ents
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 25/36
Summarizing with LSA-Based Feed back 25
hav e to rely on teachers’ feed back for every p iece of w riting. Furth ermor e, as
some sixth grader s have pointed ou t in their evaluation of the softwa re, getting
feed back from th e teacher just takes too long. Therefore, if we can d evelop a
system that gives students imm ediate feedback on their w riting in a form they
can use, teachers w ill not hesitate to give more w riting assignmen ts.
It may b e d ifficult to attribute d irect benefits to stud ents’ learning from use of
the software by any form al means, given th e difficulty of controlling
instru ctional, teacher, and a nu mb er of other variables in a classroom setting and
w ith a fairly small samp le size. Yet w e can assu me some positive effects on
learning from the kind of behaviors we h ave witnessed in the course of
classroom trials, especially th ose that took p lace in sp ring 1999 with the n ew
system:
• Stud ents are able to assum e more respon sibility for writing and revising on
their own , using th e feedback on how w ell their sum mar y covers the essential
content. Moreover, they seem to enjoy the challenge.
• By focussing their attention on specific content th at is not ad equa tely covered
stud ents find it easier to iden tify imp ortan t information – certainly a good
beginning in learning how to summ arize!
• Stud ents who u sed the sum marization tool in the classroom w orked h ard an d
long to satisfy th e criteria set for them . Being able to tra ck their p rogress
motivated them to w ork throu gh m any cycles of revision gu ided by th e
imm ediate feedb ack: they kep t trying to get it right and in so doing they
interacted w ith the text content for a longer period of time and at a deep an d
analytic level. Withou t feedb ack mark ing their p rogress, stud ents typ ically
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 26/36
Summarizing with LSA-Based Feed back 26
mak e few changes in th eir writing an d are likely to be satisfied w ith a first
draft, nicely formatted w ith a w ord processor.
• The sum ma rization tool helps stu d ents locate the specific problems w ith their
sum maries and makes them aware of the task demand s involved in summ ary
wr iting. It’s hard to balance the conflicting d eman d s of topic coverage and
conciseness. And it is frustrating to see the coverage ind icator go backw ard s
wh en you d elete material to satisfy the length constraint. Thus, in work ing
with th e system stud ents feel a need for the strategies being taug ht. They
eventually d iscover that ad ding or d eleting a word or a sentence is not
enou gh. And besides, w hich sentence? As they try to revise their sum mar ies
stud ents come to realize that they n eed better metho d s. These are valuable
metacognitive insights that m ake stud ents mor e receptive to instru ction.
Integrating Sum marization Software into Classroom Instru ction
It has become clear to us in observing stud ents work on th eir sum maries,
stud ents are not alwa ys able to apply the feedback they receive from either State
the Essence or Summary Street . For it is one thing to kn ow wh at the p roblems are
with you r wr iting and quite another to know how to fix them. Nor is it enough to
be told w hat th e processes are. Most young w riters still need mor e explicit
instruction on h ow to make ap prop riate changes, on h ow to app ly strategies like
combining sentences or find ing a generalizing term in their own su mm aries.
Taking these lessons back to th e classroom, th e teachers (CM & RL) revised th e
w ay they w ere teaching su mm arizing, d evoting mo re time to explicit instru ction
of the skill at a more concrete level than p reviously. Their new app roach involves
discussing all phases of planning and wr iting a sum mary u sing a comm on text:
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 27/36
Summarizing with LSA-Based Feed back 27
• discussing the p urp ose for su mm arizing information: to d evelop the
ability to share kn owledge w ith others an d to come to a deep
un derstand ing of comp lex information;
• mod eling h ow to d ifferentiate main id eas from d etails in highlighting, to
counteract stud ents' tend ency to select all;
• developing together an ou tline of the most imp ortant information;
• developing an und erstanding of wh at makes a good summ ary: a focus on
main ideas, good organization, low redu nd ancy and making it interesting
for other read ers;
• showing m odels of good an d poor su mm aries and jointly identifying the
specific properties of each;
• mod eling strategies for iden tifying ap pr op riate content to includ e, for
collapsing and reorganizing information;
• providing more op portu nities for stud ents to collaborate in p lanning,
writing and revising their summ aries.
The goal of this instruction is to develop a sh ared vision am ong th e
classroom comm un ity of the steps involved in sum marizing an d a common
language to talk about and improve w riting. This shared k now ledge becomes the
basis for rich discussion of matters involving bo th content an d style, both in the
classroom and between individual stud ents. We find that the sum marization tool
pr ovides a natu ral setting for collaborative pr oblem solving by helpin g stud ents
identify specific problems w ith their w riting on th eir own . Teachers and
researchers observed many instances of students spontaneously discussing their
work - about wh ether a given idea really w as important enou gh to includ e, how
information could be combined and collapsed - evidence that stud ents were
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 28/36
Summarizing with LSA-Based Feed back 28
thinking a bou t the content of their sum ma ries at a d eep and critical level. We
believe that such discussions came abou t throu gh the shared learning experience
that includ ed more p ractice in w riting, d iscussions and mod eling of
sum ma rization strategies in the classroom, collaboration, and the u se of the new
sum marization tool.
Factors affecting use of the system
The system m ay have p otential not only for giving stud ents more op portu nities
to w rite, bu t also for low ering class size for limited times. On ce stud ents are
familiar with the system, small grou ps, sup ervised by a parap rofessional
edu cator or adu lt volun teer, could work independ ently on their sum maries in
the comp uter lab, while the others work w ith the teacher in the classroom .
Rotating stu dents in the lab and classroom has the d ual ad vantage of giving
them m ore practice at w riting – w ith m ore feedback than th ey norm ally w ould
receive – and more op portu nity to interact with the teacher on an ind ividu al or
small group basis. H ow ever, this op timistic scenario d epen d s on factors that are
not en tirely u nd er the control of either teachers or th e researchers. Nam ely, a
per son w ith techn ical skills mu st be available at the school to deal with n ot
infrequent interrup tions in n etwork connections, through wh ich Summary Street
op erates. The lack of a techn ology expert in a school places the bu rd en of
maintaining the compu ters on teachers, whose highest pr iority m ust be on
planning an d executing a rich curriculum and mentoring stud ents. In fact this
ma y be one of the ma jor obstacles to ad opting th e sum mar ization tool as broadly
as intend ed. It is essential, how ever, that any comp onen ts of the system th at
ma ke it u nreliable or inconvenient to use (e.g., occasional p roblems w ith log-in
failures or losing stu den t work ) must be corrected.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 29/36
Summarizing with LSA-Based Feed back 29
Sug gestions for using th e system
The amoun t of time requ ired for teachers and students to learn the system is not
un reasonable, but the time comm itment mu st be mad e. Classroom instruction on
sum ma rization mu st precede introd uction of the software. It takes about 20-30
min. to demonstrate Summary Street to first-time u sers, w hich shou ld be follow ed
by ad equate time for students to try it out. Thus, teachers shou ld p lan one-to-two
class period s to familiarize stud ents w ith the software, because they w ill need
guidan ce initially to u nd erstand the software an d how to use the feedback.
How ever, once stud ents have this framework, they can u se the feedback from
Summary Street on their ow n. If the software is introd uced at the beginn ing of the
school year, midd le-school stud ents cou ld continue to u se it with m inimal
assistance from a p ara-edu cator or volunteer in all content areas. Ou r broad ly
stated goal for this summ arization tool was to p rovide stud ents with more
op por tun ities to write. Yet there are a variety of w ays it can be u sed to enh ance
the effectiveness of teachers’ limited resou rces. Writing a su mm ary n eed n ot be
viewed as an end in itself, but rath er as a step toward s another task, such as
pr epar ing a report or p resentation, or taking a test. The goal of a sum mar ization
assignm ent d oes not have to be to p rovide a finished prod uct every time; instead
teachers might u se a printout of Summary Street feed back as a q uick, yet
authentic way to check stud ents’ curren t u nd erstand ing of a topic.
In sum , although we aren’t quite there yet, we app ear to be on the right
track towar d m eeting ou r goals. Teachers cannot be th e sole dispen sers of
know ledge and feed back. Their job is to give stud ents tools that allow th em to
evaluate critically the content th ey learn, and to comm un icate pr oficiently w ith
others. Summary Street is one mor e tool that can help accomplish this goal.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 30/36
Summarizing with LSA-Based Feed back 30
References
Foltz, P. W. (Guest Ed.) (1998). Qu antitative ap pr oaches to sem antic know ledge
representations [Special Issue]. Discourse Processes , 25, 127-363.
Land au er, T. K. (1998). Learning and repr esenting verb al mean ing: The Latent
Seman tic analysis th eory. Current Directions in Psychological Science , 7 , 161-
164.
Land au er, T. K., and Du mais, S. T. (1997). A solu tion to P lato’s pr oblem: The
Latent Seman tic Analysis theory of th e acquisition, ind uction, and
representation of know ledge. Psychological Review , 104 , 211-240.
Land au er, T. K., Foltz, P. W., and Lah am , D. (1998). An In trod uction to Latent
Seman tic Analysis [Special Issue]. Discourse Processes , 25 , 259-284.
Land au er, T. K., and Psotka, J. (in p ress). Simu lating text u nd erstand ing for
edu cational ap plications w ith Latent Seman tic Analysis: Introd uction to
LSA. Interactive Learning Environments .
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 31/36
Summarizing with LSA-Based Feed back 31
Author Note
*The autho rs wou ld like to acknow ledge the sup por t of the CSEP Program of the
McDonnell Foun d ation. We also thank the sixth-grad e stud ents from Platt
Midd le School of Bould er Colorado for th eir participation in this wor k.
We ded icate this pap er to the memory of Ann Brown , in gratitud e and profound
admiration.
** The mem bers of the LSA Research Grou p ar e Walter Kintsch, Thomas
Land au er, Rogerio De Pau la, Eileen Kintsch, Darr ell Laham , Maur een Schreiner,
Gerry Stahl, and Dave Steinhar t.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 32/36
Summarizing with LSA-Based Feed back 32
Figure Caption s
Figu re 1. Screen shot show ing a stud ent sum mar y and first-level feedback from
State the Essence : overall score, word length, sections with a d equate content
coverage, and sections with m issing information.
Figu re 2. Screen shot show ing a stud ent sum mar y and first level of conten t
feed back from Summary Street . Green h orizontal bars indicate to what extent the
content of each section w as adequ ately covered in th e sum mar y, with the
thresh old ind icated by a black v ertical line. The vertical bar o n th e left is a length
ind icator, shown in green if the sum mar y is w ithin the prescribed limits and red
if not .
Figu re 3. Mean gr ad es of Ancient Civilizations su mm aries by condition.
Figure 4. Mean grades of Heart and Lungs sum maries by cond ition.
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 33/36
Summarizing with LSA-Based Feed back 33
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 34/36
Summarizing with LSA-Based Feed back 34
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 35/36
Summarizing with LSA-Based Feed back 35
6.5
7
7.5
8
8.5
9
G r a d e
Maya Aztec Inca
Traditional
Summary Street
8/6/2019 Summarizing Skills
http://slidepdf.com/reader/full/summarizing-skills 36/36
Summarizing with LSA-Based Feed back 36
2
2.5
3
3.5
4
4.5
G r a d e
Lungs Heart
WordProcessor
SummaryStreet