testing reading comprehension of theoretical discourse with cloze

Journal of Research in Reading, ISSN 0141-0423Volume 24, Issue 1, 2001, pp 82±98

Testing reading comprehension oftheoretical discourse with clozeBenjamin B. Greene, Jr

The Franklin P. Perdue School of Business, Salisbury State University, USA

ABSTRACT

The ability of cloze tests containing frequent, every n-th word deletions to measurecomprehension of macropropositions has been challenged on both theoretical andempirical grounds, calling into question the validity of such tests for assessingcomprehension of much of the discourse encountered by university-level students. Toevaluate the comprehension of a writer's reasoning, it is recommended that clozetests position gaps so as to target recognition of cohesive devices and the ability todraw inferences from other sentences. To test the validity of such a design, a largesample of scores on discourse cloze tests administered in introductory collegeeconomics is compared to scores on true±false comprehension tests designed to targetrecognition of connective propositions. The two distributions of scores do not differsignificantly in terms of mean value, dispersion or frequency distribution, suggestingthat appropriately designed cloze tests can provide a valid assessment of the reader'sintegration of theoretical text. In addition, the usefulness of readability formulasbased on surface characteristics of text is challenged when readability is defined interms of the difficulty of constructing a coherent representation of theoretical text.

INTRODUCTION

The validity of the cloze method for assessing reading comprehension, contentknowledge and text readability has been generally accepted for over 30 years on thebasis of correlations between cloze scores and results of other forms of assessmentreported by Taylor (1957), Rankin (1959), Bormuth (1967, 1968a, 1968b) and others.Support for the construct validity of cloze can also be found in wide recognition thatgap-filling exercises directly involve many skills associated with effective readingwhile reducing potential sources of bias (e.g. poor wording of questions, ambiguity inrecalls, unrepresentativeness of coverage, delayed inference) in other forms of assess-ment (Bormuth, 1968a, 1968b, 1975; Rye, 1982; Harrison, 1980; Gilliland, 1972; Klare,1984; Ulijn and Strother, 1995).

However, several researchers have challenged the validity of cloze tests, especiallythose containing a high frequency of random or arbitrary deletions, for assessing the

# United Kingdom Reading Association 2001. Published by Blackwell Publishers, 108 Cowley Road,Oxford OX4 1JF, UK and 350 Main Street, Malden, MA 02148, USA

reader's comprehension of the macrostructure of text. Kintsch and Yarbrough (1982)report the failure of tests deleting every fifth word to capture macroprocesses inreading. Levenston, Nir and Blum-Kulka (1984) clarify the problem by demon-strating that higher reading skills and global coherence can be clearly assessed only ifthe placement of deletions is deliberately designed to do so. Given these arguments, itseems reasonable to ask whether cloze tests provide a valid measure of the compre-hension skills required in theoretical disciplines such as economics and the physicalsciences at the university level. Some researchers (Alderson and Urquhart, 1985;Pouw and van der Tuin 1997; Taillefer and Pugh, 1998) have assumed that they do;but little evidence has been presented for the validity of cloze assessment of readingcomprehension for the material typically encountered in such courses.In this paper I present evidence from a large sample of reading test scores for the

validity of cloze-based assessments of reading comprehension for the discoursetypically encountered in introductory college economics textbooks. Given the simi-larities, to be described, between the discourses of economics and the physicalsciences, I believe that these results provide strong evidence that appropriatelydesigned cloze tests permit valid assessments of reading comprehension in theoreticalknowledge domains at the college level. I will also discuss the implications of theseresults for measurement of readability at the end of this paper.

Comprehension requirements for reading theoretical discourse

As the purpose of text moves from being primarily a description or classification ofthe concrete or a narrative of experience to an examination of the causal factorsbehind observed reality, an increase in the complexity of text seems inevitable. Batenand Cornu (1984) point out that, in non-fiction text, it becomes essential for thereader to `arrive at the same gist as the one meant by the writer' (p. 192). Lucy (1997)describes the expository nature of college texts in terms of claims, warrants, backing,data and rebuttals, and presents evidence that `the conceptual difficulty of a text' isrelated to `the inferential demands relating to the underlying logical structure' (p. 87).When the reader is asked to build a representation of complex interactions amonga large number of variables, the level of attention required to achieve coherencenecessarily increases. The reader must recognise the cohesive devices used by thewriter to express or imply a potentially large and diverse array of connective propo-sitions (Turner and Greene, 1978) or coherence relations (Sanders, Spooren andNoordman, 1993) so that the message obtained is more than the sum of its parts.Without developing the macropropositions (van Dijk and Kintsch, 1983; Kintschand Yarbrough, 1982; Levenston, Nir and Blum-Kulka, 1984) that give a textcoherence, the reader may be able to recall isolated bits of information but lack aclear sense of how those bits are thought to be related.In other words, successful reading requires building a representation that is both

more abstract and more complex. The requisite comprehension skills move beyondinstantiation and formation of isolated schemata, or hierarchies of set-membershipschemata, to development of increasingly complex networks in which schemata forindividual concepts (e.g. interest rate, aggregate output) are linked by their capacityto change, and in changing cause, require, prevent, reinforce, discourage and other-wise impact on change in each other. These complex systems of schemata may appearas scripts (Mandler, 1984; Rumelhart and Ortony, 1977), in which a change in one

# United Kingdom Reading Association 2001

CLOZE AND THEORETICAL DISCLOSURE 83

variable is expected to trigger a potentially complex sequence of reactions involvingan elaborate network of schemata, or as `pragmatic reasoning schema' (Cheng andHolyoak, 1985) consisting of deductive rules for particular types of situations.Developing a coherent representation of text appears to be a challenging task for

many college-age readers. The relevant cognitive processes reflect the full spectrum oflearning in Bloom's taxonomy (Bloom et al, 1956), and the deductive reasoningdescribed in more theoretical texts involves Piaget's highest (`formal') stage ofcognitive operations (Case, 1985). Numerous researchers have reported on the failureof many college-age readers to carry out the necessary processing. Lucy (1997)reports on the relatively poor recall of `warrants' (those clauses that express the logicor legitimacy of a `claim'). Marton (1975), Marton and Saljo (1976), and Marton andDahlgren (1976) report that many college-age readers do not attend to the manner inwhich information is integrated, even when deliberate attempts are made to invokewhat they describe as deep level processing. Grossman and Simon (1997) reportsimilar weakness with respect to summarisation and synthesis in reading. Otero andKintsch (1992) found that high-school seniors and college students frequently fail toreport contradictions contained in short reading passages, and Baker (1985) findsthat many college-age readers are unaware of inconsistencies in text. Simon, Hinsleyand Hayes (1989) report that experienced maths students often base their solutionstrategy to a word problem on initial information and ignore later informationpointing to a different approach. Liau, Bassin, Martin and Coleman (1976) find thatthe frequency of coordinating conjunctions has a significant negative impact on clozescores among undergraduates. Insufficient attention to cohesion markers, particu-larly conjunctions, is also reported for non-native speakers of English readingtechnical text (Cohen et al, 1988). Such findings all suggest that many college-agereaders frequently fail to obtain a completely coherent representation of text.The ubiquity of what Marton and others have labelled surface level processing

raises interesting questions that go beyond the scope of this paper. Kingston (1960)argues that the comprehension problems of undergraduates are more a matter ofmotivation than ability. Reif and Larkin (1991) elaborate on this hypothesis for thescientific disciplines.The importance of `seeing the connections' would seem to make the cloze task

particularly appropriate for evaluating the coherence of a student's comprehension.Multiple-choice or true±false items can address concept definitions, broad con-clusions or particular steps in an argument. The cloze format, on the other hand, cantest a student's recognition of the cohesive devices that make a carefully constructedargument possible, just as writing an essay would. Numerous researchers have notedthe uniqueness of the cloze format in requiring readers to connect each propositionwith what is elsewhere established in the text. Rye (1982) comments that the readermust reason using `evidence derived from context' (p. 3) and `scanning, in search ofunspecified information' (p. 32). Harrison (1980) states that the cloze proceduremeasures a reader's grasp of `between sentence meanings and relationships' (p. 88).Gilliland (1972) points out that cloze measures a reader's ability `to use a variety ofcontextual interrelationships' (p. 103).Whether or not a cloze test provides a valid test of the integration of propositions

into macropropositions depends, however, on the design of the cloze exercise. Intheir critical analysis of cloze design, Levenston, Nir and Blum-Kulka (1984) draw acrucial distinction between cloze gaps that can be filled on the basis of lexical or


84 GREENE

grammatical clues contained in isolated sentences or purely external generalknowledge and gaps that target inter-sentence relationships and thereby revealwhether the reader is obtaining a coherent grasp of the reasoning presented by thewriter. Cloze scores based on every n-th word or randomly placed gaps produce anambiguous picture of the skills being exercised by the reader. To provide an accuratetest of higher-order reading skills, which will necessarily also test lower-order skills,these researchers recommend a discourse cloze design that will reveal the reader'sàbility to reconstruct the macrostructure' (Levenston, Nir and Blum-Kulka, 1984,p. 207) of text. In such a cloze test, a substantial portion of deletions will requireinferences that draw on information provided in sentences other than the onecurrently being processed.

Similarities of discourse in the physical sciences and economics

Because they seek to describe and justify a highly integrated model of the way theworld works, the natural sciences and economics provide a solid testing ground forassessing comprehension of macrostructure for the type of text described by Lucy(1997). In addition to being densely populated with claims and warrants, andemphasising measurement of phenomena, these disciplines are distinctive in theirjustification of hypotheses in terms of universally applicable principles and thefrequent chaining of hypotheses to demonstrate the predictive power of fundamentalclaims (Reif and Larkin, 1991; Siegfried et al, 1991). A good example of suchchaining of claims is provided by the cloze passage in Appendix A. (The reasoningchains in this passage may be briefly summarised as follows: Governmentexpenditure : G Inventories ; G Production : G Income : G Consumptionspending : G Production up : all of which G Change in production4Change ingovernment purchases, but that Production : G Demand for money : G Interestrate : G Investment spending ; G Production ; .)Reading in scientific disciplines has long been recognised as more demanding than

simple narrative or descriptive text. Effective reading in scientific disciplines hasbeen noted to require precision in the assignment of meaning to words, greatattention to detail and strong organisational and inferencing skills (Bloomfield, 1969;Bracken, 1958; Harris and Sipay, 1971; Weidler, 1984; Pugh 1988; Vachon andHaney, 1991). Kintsch et al (1975) found that scientific discourse places greaterdemands on undergraduate readers than history text. One could also infer thatstrong reading skills are essential to success in scientific disciplines from the fact thatcollege-bound students in the United States who plan to major in science consistentlyrecord the highest verbal Scholastic Aptitude Test scores (US Department of Edu-cation, 1996).Notwithstanding its identification as a social science, there are strong parallels

between the discourse of economics and that of the physical sciences. Wheneconomists are asked to define what it means to `think like an economist' they speakof `. .. using chains of deductive reasoning', `tracing the behavioral implications ofsome change while abstracting from other aspects of reality', àmassing data toevaluate and refine our understanding', `testing alternative hypotheses', ìnsights . . .derived logically from a set of premises', àn emphasis on parsimonious models',`formulating hypotheses to explain these relationships', àssembling empiricalobservations bearing on these relationships, and testing the hypotheses using



quantitative techniques' (Siegfried et al, 1991, pp. 199±200), a description almostidentical to that given by Reif and Larkin (1991) for learning in the physical sciences.The similarities between theoretical models in economics and the physical sciencesare noted by Boulding (1958) and the basis for the similarities between the models ofeconomists and physicists is examined in depth by Mirowski (1989).Introductory economics textbooks tend to present students with a view of the

world as rational and predictable, shaped by globally consistent laws of causality. Inthis, economists `have taken economics away from the other social sciences' andmaintain `a positivist self-image' (Backhouse, Dudley-Evans and Henderson, 1993,p. 2). The phenomena studied may be social in origin rather than inorganic, but thehypotheses are the product of scientific investigation and verification (McConnelland Brue, 1996).The first college economics course, whether it be macroeconomics or microeconomics,

presents a multitude of hypotheses, definitions and measurement techniques relatedto (i) individual and group behaviour, (ii) technological relationships and (iii) re-source constraints, and then demonstrates how those ideas can be applied to forecastor alter a host of economic conditions. Reading and other activities frequentlyinvolve piecing such ideas together to form complex arguments in which the studentmust reason forward from a specific event (e.g. tax cut, government regulation) to aset of immediate and eventual consequences, or backward from a desired outcome(e.g. lower unemployment, improved allocation of resources) to a logical course ofaction. Analyses generally involve multiple hypotheses of cause and effect to describethe decisions by economic actors about spending, saving, lending, borrowing,producing, selling, pricing and other activities, decisions that are constrained by avariety of natural and social conditions and motivated by a variety of immediate andfuture goals. These individual hypotheses each invoke an array of variables, that arethen linked through chain reactions and feedback effects to provide the requiredanalysis.The peculiarities of scientific discourse go beyond its emphasis on the justification

and linking of claims, however, and include a number of lexical and syntacticpeculiarities that could, without deliberate selection of words to be deleted, confounda test of cloze' ability to assess comprehension of macropropositions. These disciplineshave been noted for their more frequent use of heavy noun phrases, nominalisations,and particular word forms (participles, infinitives and passives), restrictive use of non-technical words, level of abstraction, and reference to or involvement of mathematicalrelationships. (Prins and Ulijn, 1998; Ulijn and Strother, 1995; Reif and Larkin, 1991;Bloomfield, 1969). The presence of such lexical peculiarities strengthens the case forcareful deliberation in the choice of words to be deleted.These peculiarities also tend to characterise economic discourse, and Ulijn and

Strother (1995) are incorrect in suggesting that economic discourse is more similar tobusiness language, and common language, than it is to the language of science.Greene (1992) has compared the lexicon of introductory economics and accountingtextbooks and found a number of significant differences which suggest that thedegree of similarity between economics and even the more quantitiative dimension ofbusiness is questionable. Based on samples of text totalling 9,000 words each fromgroups of introductory economics texts and introductory accounting texts, he foundthat the economics texts contained significantly (i) broader vocabulary requirementsin terms of proportion of distinct root words, (ii) greater variety of modifiers,


86 GREENE

(iii) greater frequency of words associated with spatial relationships and matter,(iv) larger proportion of conjunctions and conjunctive adverbs, including morefrequent use of the conjunction `but', and (v) a larger proportion of words connotinguncertainty.The distinction between economists' scientific writing and business English is also

noted by Olah (1984). Unfortunately, she goes on to attribute greater subjectivityto the social science of economics than to the natural sciences, a perception thatvirtually all introductory economics texts seek to immediately dispel by emphasisingthe positive rather than normative nature of economic analysis (Case and Fair, 1994;McConnell and Brue, 1996). Economics shares the physical sciences' tendency tostress the nature of action or change in the abstract more than the particularinstantiation of acting or changing. Economists are also equally prone to restrict themeaning applied to words (e.g. `price' is not the same thing as `cost', `earning income'is quite different from `making money' and so on). In sum, there is more evidence forplacing economic discourse at the science end of the spectrum than there is forplacing it at the common language end.

TESTING THE VALIDITY OF CLOZE MEASURES OF READINGCOMPREHENSION OF INTRODUCTORY COLLEGE ECONOMICSTEXTBOOKS

Method

Background

In an effort to enhance crucial language skills and motivate careful reading, I havemade reading comprehension an integral component of assessment in my intro-ductory economics classes since 1991. From 1991 to 1993, this was done using atraditional closed response format. Students were given a carefully selected newsstory from the Wall Street Journal or a passage from an introductory economicstextbook to read. In the succeeding class period (forty-eight hours later), studentstook a comprehension quiz consisting of equal numbers of true and false statements.The quiz questions were accompanied by a clean copy of the reading assignment towhich students could refer during the quiz. Beginning in 1994, the quiz format wasaltered to a cloze design aimed at the ability to follow an explanation, similar to the`discourse cloze' concept described by Levenston, Nir and Blum-Kulka (1984). Thetexts for these exercises were restricted to a set of passages taken from the adoptedtextbook and designated at the beginning of each semester. Since different samples ofstudents took the true±false and cloze format quizzes, correlation of individual scoresis not possible. A comparison of the summary statistics of the two distributions ofscores obtained with the different assessment formats is possible, however, and usefulconclusions can be drawn from that comparison.

The subjects

Students in the classes were predominantly in their second year of college, makingthem about 20 years of age, although some were upperclassmen and therefore a yearor two older, and a very few were older, non-traditional students. In most sections



two-thirds or more of the students had declared a business-related major; theremaining students probably elected the course as a means of satisfying a generalsocial-science requirement. Of 326 different students who took one or more of thetrue-false quizzes, 64% were male. Of 343 students taking one or more of the clozequizzes, 59% were male. Students not having English as their first language wererare, and make up less than 3% of the samples.In many semesters, students were allowed to drop their lowest quiz grade, a

practice which clearly impacted on the scores. `Drop-able' quiz scores are excluded inthe analyses that follow, as are the scores for students who were not present whenreading assignments were distributed for the true±false quizzes. The total numbers ofuseable comprehension quiz scores are 695 in the true-false format and 1,075 in thecloze format.

Materials

True-false comprehension quizzes

The design of the true±false quizzes was heavily influenced by the idea of text as a setof propositions with text cohesion represented by connective propositions (Kintschet al, 1975; Kintsch and Vipond, 1978; van Dijk and Kintsch, 1983; Turner and Greene,1978). The passages for these quizzes were selected on the basis of the importanceand variety of connective propositions expressed or implied as well as estimatedreading level and appropriateness of ideas discussed. In addition to dealing witheconomic concepts already covered in class discussion and regular readingassignments, the passages typically involved a contrast of viewpoints, circumstances,or possibilities. Passage length ranged from 800±1,200 words and reading levels wereusually no higher than grade 12 or 13 according to the Flesch±Kincaid index.The true±false items were deliberately designed to test students' comprehension of

the reasoning presented in the passage. This was done by targeting the connectivepropositions expressed or implied in the passage (e.g. points of causality, contrast,exception or rebuttal, motivation, conditionality, and timing as well as conjunctionand disjunction). Since the passages assigned for these quizzes were not discussedelsewhere in the course, the knowledge students displayed on the quizzes reflectedprimarily their comprehension of what they read. For a more detailed description ofthese passages and quizzes, and an analysis of error patterns, the reader is referred toGreene (1997).

Cloze comprehension quizzes

Between the fall semester of 1994 and the fall of 1997, I taught twelve sections ofintroductory economics, using three different, widely adopted textbooks (Case andFair, 1994; Schiller, 1994; McConnell and Brue, 1996). A Study Guide was preparedfor students in each class which provided key definitions and question formats tobe used on examinations plus a list of two to four key textbook passages for eachchapter to be covered (approximately 50 passages in total per semester). Passageswere two to five paragraphs in length and were selected to focus on the reasoninginvolved in important measurement procedures or applications of economicconcepts.Individual textbook passages were combined to create cloze format quizzes, which

are the exercises reported on in this paper. In all, 52 quizzes were administered, with


88 GREENE

a mean length of 405.3 words (sd=134.95 words). Conventional measures ofreadability were computed for each quiz using the Grammatik software (Wampler,1990). Over 90% of the quizzes scored for readability at a grade level between eightand twelve using the Flesch±Kincaid formula or between eleven and sixteen using theGunning Fog formula. Flesch Reading Ease values (original formula) range from36±69 with a mean of 53.4 (sd=8.7). In addition, mean cloze scores were predictedfor the quizzes using both the Bormuth and the Coleman±Liau formulas reported inKlare (1984). Mean cloze scores predicted by Bormuth's (1975) formula range from17±39% with a grand mean of 27% and a standard deviation of 5.6%. Mean clozescores predicted using the Coleman±Liau formula range from 31.7% to 57.3% with agrand mean of 44.4% and a standard deviation of 5.9% .The percentage of words replaced with blanks on the cloze quizzes ranged from

8% (one word in 12.5) to 17% (one word in 6) with a mean of 11.5% (one word in8.7). A small but significant positive correlation (0.29) between mean cloze scoresand frequency of deletions indicates that more frequent deletions did not interferewith comprehension and reflects efforts to assure adequate redundancy in thequizzes. The deletion rates used are within the upper bound of 20% (every fifthword) reportedly recommended by McGinitie (Bormuth, 1975; Harrison, 1980), andare not out of line with deletion rates used or recommended by other researchers(Taylor, 1957; Rankin, 1959; Gilliland, 1972; Jongsma, 1980; Pugh, 1988). Thenumber and placement of deletions reflected the desire to test integration of sentencesand is therefore less likely to encounter the problems associated with very frequentdeletions noted earlier.The sample passage presented in Appendix A is taken from a quiz given late in the

semester in an introductory macroeconomics section. As the semester progresses,discussions tend to more often involve chaining of concepts and claims introducedearlier in the semester. Passages used earlier in the semester more frequently focus onthe warrant, backing and possible rebuttal of a particular claim as described by Lucy(1997).This particular cloze passage contains 36 gaps, answers for more than half of

which can be inferred from sentences found elsewhere in the passage. To clarify suchconnections, I have inserted a sentence number at the beginning of each sentencein the passage (which was not present in the quiz) and a sentence reference numberaccompanies answers (indicated in brackets) when those answers might be inferredfrom information contained in a different sentence.Deletions did not generally target terms unique to economic discourse. The

primary objective governing selection of words for deletion was to ensure sufficientredundancy to enable students to follow the reasoning presented in the textbook.Approximately 50% of deleted words are nouns, and altogether, nouns, modifiersand verbs constitute 95% or more of deletions. Support for this `hard' form of clozeis provided by Taylor (1957) and Rye (1982).

Administration and scoring of cloze exercises

For all reading comprehension quizzes, students were allowed to take as much timeas they needed, usually 15±20 minutes on the true-false quizzes and about 30 minuteson the cloze quizzes.



Over the course of a semester, three to six cloze format quizzes were administeredto each class, with a student's average quiz grade constituting 20±30% of his/her finalcourse grade. Each quiz was announced at the preceding class meeting but studentswere not told which passages would appear on the quiz, only that passages related tomaterial already covered would be used. Students were advised at the beginning ofeach semester that quizzes would be based on the key passages, and that the passagesdeserved special attention in their reading, both to do well on the quizzes and todevelop their understanding of the vocabulary and reasoning involved in appli-cations. It was hoped that having students focus on key passages would help themprepare for examinations by improving their comprehension of economic ideas, aswell as by contributing to development of general language skills.Since students had prior knowledge of the reading materials on which they were

tested, these cloze quizzes should be in the nature of post-comprehension tests. Assuch, general knowledge obtained through classroom presentation and regularreading assignments could have facilitated some answers. However, as the samplequiz in Appendix A makes clear, unless the student had memorised all designatedpassages, he/she would need to figure out the appropriate responses by following thegist of the argument. Rye (1982) recommends the post-test when the objective is toassess how accurately and completely a student has constructed the schema (systems)corresponding to a body of factual or theoretical knowledge, and Taylor (1956, 1957)and Rankin (1959) have reported post-tests to be as or more valid than pre-tests.Allowing students to `study' for the quizzes does not appear to have impacted

significantly on the cloze scores. To test the effect of announcing the quizzes on thecloze scores, an experiment was performed with a small section of students duringthe spring semester of 1998. Sixteen days (including a week-long spring break) afterthe administration of an announced quiz, an unannounced quiz was administeredthat included identical cloze exercises on two textbook passages from the precedingquiz. Fourteen students were present for both quizzes and scores were recordedfor the separate textbook passages. For one passage, the correlation between`announced' and `unannounced' scores was 0.71, and the difference between meanscores was not significant (t51). For the second passage, the correlation between thetwo sets of scores was 0.89, and the mean scores were again not significantly different(t51)).In scoring these exercises, synonyms or `semantic equivalents' were accepted as

correct. As an assessment of comprehension, the purpose of the quizzes was tomeasure the extent to which a student comprehended the propositions intended by thetextbook authors, not whether he/she shared the writer's general vocabulary orprecise choice of phrasing. Accordingly, except for instances where economic jargonwas clearly called for, either the author's original word or a synonym was accepted ascorrect, as long as spelling errors did not prevent recognition of the intended word.In addition, half credit was sometimes given when the answer conveyed a sense of thedesired meaning but syntactically or semantically preferable choices should havebeen available to the student (e.g. replacing an intransitive verb with a transitiveverb, concocting a verb out of a noun). On extremely rare occasions, credit was givenif answers departed from the meaning intended by the author but conveyed arelevant idea within the context of the quiz passage.The scoring of answers obviously did not follow the rigid guidelines often used in

readability research (Rankin, 1959; Bormuth, 1968a; Harrison, 1980). However,


90 GREENE

acceptance of synonyms has been demonstrated to correlate closely with morerestrictive scoring regimens and with alternative comprehension tests (Taylor, 1957;Alderson, 1979; Harrison, 1980), and has been employed or recommended in theteaching or assessment of content knowledge by Jongsma (1980), Rye (1982),Anderson (1972) and Pugh (1988).

Comparison of true±false and cloze measure of comprehension

As Table 1 shows, the scores generated by the true-false and cloze formats haveremarkably similar distributions. The means, medians and standard deviations of thetwo distributions each differ by less than 1.5 percentage points, and both distri-butions are characterised by significant negative skewness. The difference betweenmean scores on the two types of reading diagnostic produces a Z statistic of 1.87, notquite large enough to be significant at a 5% risk of Type I error. However, even ifone allows that the two population means are not identical, the noteworthy point isby how little they appear to differ. Perhaps most startling is the finding that seven ofnine decile values differ by less than 2.5 points. A contingency table analysis of thetwo distributions, using break points of 50, 60, 70, 80 and 90, generates a Chi-squarestatistic (with five degrees of freedom) of 9.99, again failing to indicate a significantdifference between the two distributions at 5% risk of Type I error.While quizzes in both formats were deliberately designed with the connections

intended by the author of the passage in mind, the likelihood of such similar scoredistributions seems remote a priori. After all, a poor reader in terms of vocabulary,command of syntax, pre-existing schemata or other relevant knowledge is quite likelyto fill cloze gaps with terms that are irrelevant, even nonsensical, but always has a

Table 1. Comparison of distribution of individual cloze scores with true±false scores on

conventional reading comprehension test.

Cloze scores True±false scores

N of scores

Mean

Median

Standard deviation

Skewness

Decile scores

10

20

30

40

50

60

70

80

90

1,075

78.35

80.95

15.28

71.15

57.14

67.81

73.09

77.17

80.95

84.09

88.16

91.89

95.23

695

77.01

80.00

14.30

70.61

57.14

66.67

71.43

75.00

80.00

81.82

83.33

90.91

91.67



50% chance of choosing the correct answer to a true±false statement. On the otherhand, a strong reader can often take advantage of a variety of clues of different typesin the cloze format, clues frequently located in the same or an adjacent sentence,something that he or she cannot do in the true±false format. There is, in fact, someindication that cloze format is advantageous for very strong readers in that thehighest 10% of scores exceeded 95 with the cloze format but only 91.7 with the true±false format and the highest 30% of scores exceeded 88 with cloze but only 83.3 withtrue±false. If a cloze format test is in fact easier than a closed-response format forgood readers and more difficult for poor readers, then score distributions would beexpected to reflect that.The strong similarity between the two score distributions clearly suggests that the

two forms of test measured the same set of skills. Had the two test formats assesseddifferent types or a substantially different mix of comprehension skills, there wouldalmost certainly be a larger difference in the distribution means and probably thestandard deviations of scores as well. Since the true±false quizzes were designed totest grasp of the connections in the text and reflected knowledge that could only havecome from the students' reading, it follows that the cloze format achieved its purposeof testing similar comprehension skills.To further examine the consistency between cloze format and conventional com-

prehension exercises, a series of cloze format quizzes was administered in a smallsection in the spring of 1998. Each quiz also included 4-6 multiple-choice items basedon the key textbook passages. Analysis of the cumulative percentage scores on thecloze and multiple choice components of the quizzes yields a significant correlationcoefficient of 0.69 for fourteen students that took all of the quizzes. (No adjustmentwas made in this case for drop-able quiz scores.)These results suggest that a cloze test can be designed to measure a reader's

macroprocessing of theoretical text. A cloze format test that requires a substantialnumber of inferences from surrounding sentences will yield a very similar pattern ofscores to a closed-response format test that focuses on recognition and compre-hension of the connective propositions that give coherence to the text. Researchersor instructors requiring a valid assessment of reading ability in theoretical collegecourses can tailor the cloze test to target the appropriate reading skills. However,given the issues raised by Kintsch and Yarbrough (1982) and Levenston, Nir andBlum-Kulka (1984), these results should not be taken as a validation of a cloze designbased on frequent, arithmetically determined (every fifth word) deletions. Based onthe mean cloze scores predicted by the Coleman and Liau (44.4%) or Bormuth(27%) formulas, it appears that a more traditional and restrictive cloze test mighthave produced a much different distribution of scores.

CLOZE SCORES AND CONTENT KNOWLEDGE

At the same time, it would not be advisable to make the cloze test the only format forassessing content knowledge in such courses. While Taylor (1957) concluded fromhis original study of Air Force trainees that cloze provides an accurate measure ofcontent knowledge, that conclusion appears less justified in college-level economics.Though highly significant, correlations between individual students' cloze scores andtheir scores on other measures of economic knowledge are low.


92 GREENE

In four sections of microeconomics, the final examination included a section ofmultiple choice questions designed by the Educational Testing Service for potentialincorporation in the Advanced Placement Examination in economics. Eighty-threestudents took these pseudo-AP exams at the end of the semester and the correlationcoefficient between their 259 cloze quiz scores and their percentage scores on thequestions prepared by ETS, though highly significant, is only 0.35.In all twelve sections, students who completed the course took three hourly

examinations which consisted of definition completions and a variety of structuredquestions and problems. Since students had been provided with the definitions andquestion formats in the Study Guide, their percentage scores on these examinationsshould represent an unambiguous index of their success in meeting clearly definedlearning objectives. While highly significant, the correlation between students' meanexamination score and their cloze scores is only 0.58, considerably lower than the0.80 reported by Taylor (1957).The smaller size of these correlations is not surprising since reading comprehension

is not the only ability being tested on general examinations in economics. Most testsof content knowledge in economics focus heavily on the student's ability to expressand apply deductive reasoning, as well as use quantitative skills to construct andinterpret diagrams and mathematical representations. Siegfried and Strand (1977),Lumsden and Scott (1987) and Anderson, Benjamin and Fuss (1994) have all foundthat quantitative reasoning skills have a significant influence on achievement inintroductory economics.

CLOZE SCORES AND READABILITY

Numerous criticisms have been made of readability formulas based on surfacefeatures of text (e.g. word length or familiarity, sentence length) (Gilliland, 1972;Perera, 1980; Davison and Kantor, 1982; McConnell 1983; Klare, 1984; Carrell,1987; Rye, 1982; Taylor, 1957; Miller, 1975), prompting researchers to search forvariables more indicative of semantic complexity (Kintsch, 1979; Kemper, 1983;Harrison and Bakker, 1998). The cloze scores reported on here support suchcriticism.As indicated in Table 2, when mean scores are computed for the 52 different cloze

format quizzes administered, they do not correlate very well with readability formulapredictions. While the correlation coefficients all have the expected sign, the only

Table 2. Correlation of mean cloze scores with readability formula predictions.

Readability Formula Correlation Coefficient Significance

Flesch reading ease (orig.)

Bormuth

Coleman/Liau

Flesch-Kincaid GL

Gunning Fog GL

0.173

0.265

0.263

70.193

70.172

0.220

0.057

0.059

0.170

0.223



correlations to approach statistical significance are those between the mean quizscores and the predictions of the two formulas constructed to predict cloze scores.The limited ability of formulas based on surface-text characteristics to predict

comprehension of economics should not be surprising. Simple word and sentenceattributes will not be able to capture the more restricted definition of common words,the tendency to use particular word forms, the importance of cohesive devices, or theimportance of precision in word choice to accurately express quantifiable phenomenasuch as changes in the direction, magnitude or rate of change of an economicvariable.In addition, the sampling of subjects may contribute to lower correlations for two

reasons. First, Klare (1984) has noted the inflation of many reported correlationsbetween readability and comprehension scores resulting from high levels of variancein both difficulty of materials and reading ability of subjects. Those concerns seemless relevant here since both the materials and the subjects are constrained by thelevel and nature of the course. Second, the fact that the group of subjects is not heldconstant adds a new source of variation to the observed relationship between surfacefeature measures of readability and mean cloze scores.An analysis of variance in individual cloze scores grouped by Flesch±Kincaid

grade levels of the quizzes further illustrates the inability of word and sentence lengthvariables to capture the difficulty of the texts. The confidence intervals reported inTable 3 suggest that significant differences in comprehension occur only with surfacefeatures associated with an eighth grade reading level. If the analysis is repeated usingonly scores for passages with a predicted grade level of 9±12, the F statistic becomesnon-significant (F=1.90). (The validity of the F-test reported in Table 3 is ques-tionable given the differences in the size of the standard deviation across Flesch±Kincaid levels. By limiting the analysis to quizzes with Flesch±Kincaid levels between9 and 12, it is less likely that the homoscedasticity assumption of the analysis ofvariance procedure is violated.)

Table 3. Analysis of variance of individual cloze scores with Flesch±Kincaid grade level of material.

Sum of Squares df Mean Square F Significance

Between groups

Within groups

Total

12,550.9

238,254.6

250,805.5

6

1,068

1,074

2,091.82

223.09

9.377 0.000

Flesch±Kincaid

grade level

n Cloze

scores

Mean cloze Standard

deviation

95% Confidence interval

for Cloze mean

7

8

9

10

11

12

13

32

113

144

294

300

153

39

81.51

87.27

78.17

77.23

75.38

78.60

80.73

12.35

10.45

14.53

17.00

15.56

13.68

12.03

77.06±85.96

85.32±89.22

75.78±80.56

75.28±79.18

73.62±77.15

76.41±80.78

76.83±84.63


94 GREENE

CONCLUSIONS

Previous research has established the validity of cloze tests for assessing compre-hension of many types of text, but has done little to examine its suitability to theprecise and highly connected discourse of theoretical disciplines at the college level.Although the cloze format naturally tests inferencing skills, it does not automaticallytest comprehension of the macrostructure of text. The extent to which cloze assessesglobal coherence depends on the deletion strategy employed. While the validity ofcloze tests based on frequent, random deletions for assessing comprehension oftheoretical discourse remains in question, the evidence presented here suggests thatcloze exercises can provide a reliable tool for assessing coherence of comprehensionof theoretical arguments if they focus on the connections between sentences.

ACKNOWLEDGEMENTS

The research reported in this paper was supported by a Summer Research Grantfrom the Franklin P. Perdue School of Business. I also thank an anonymous reviewerfor several valuable suggestions.

REFERENCES

ALDERSON, J. C. (1979) The Cloze procedure and proficiency in English as a foreign language. Teaching

English to Speakers of Other Languages Quarterly, 13, 219±227.

ALDERSON, J. C. and URQUHART, A. H. (1985) This test is unfair: I'm not an economist. In P. Hauptman,

R. LeBlanc, and M. B. Wesche (eds.), Second language performance testing/L'evaluation de la `perform-

ance' en langue seconde Ottawa: University of Ottawa Press. [Reprinted in P. L. Carrell, J. Devine and

D. E. Eskey (eds.), Interactive Approaches to Second Language Reading (1988) Cambridge: Cambridge

University Press, 168±182.]

ANDERSON, G., BENJAMIN, D. and FUSS, M. A. (1994) The determinants of success in university introductory

economics courses. Journal of Economic Education, 25, 99±119.

ANDERSON, R. C. (1972) How to construct achievement tests to assess comprehension. Review of Educational

Research, 42, 145±170.

BACKHOUSE, R., DUDLEY-EVANS, T. and HENDERSON, W. (1993) Exploring the language and rhetoric of

economics. In W. Henderson, T. Dudley-Evans and R. Backhouse (eds.) Economics and language.

London: Routledge, 1±20.

BAKER, L. (1985) Differences in the standards used by college students to evaluate their comprehension of

expository prose. Reading Research Quarterly, 20, 297±313

BATEN, L. and CORNU, A.-M. (1984) Reading strategies for LSP texts: a theoretical outline on the basis of text

function, with practical application. In A. K. Pugh and J. M. Ulijn (eds.) Reading for professional

purposes. London: Heinemann, 190±201.

BLOOM, B. S., ENGELHART, M. D., FURST, E. J., HILL, W. H. and KRATHWOHL, D. R. (1956) Taxonomy of educational

objectives. New York: David McKay.

BLOOMFIELD, L. (1969) Linguistic aspects of science. In O. Neurath, R. Carnap and C. Morris (eds.)

Foundations of the unity of science Chicago: University of Chicago Press, 216±277.

BORMUTH, J. R. (1967) Comparable cloze and multiple-choice comprehension test scores. Journal of

Reading, 10, 291±299.

BORMUTH, J. R. (1968a) The cloze readability procedure. Elementary English, 45, 429±436.

BORMUTH, J. R. (1968b) Cloze test readability: criterion reference scores. Journal of Educational

Measurement, 5, 189±196.

BORMUTH, J. R. (1975) The Cloze procedure: literacy in the classroom. In W. D. Page (ed.) Help for the

reading teacher: New directions in research Urbana, IL: National Conference on Research in English,

60±90



BOULDING, K. E. (1958) The skills of the economist. Toronto: Clarke Irwin.

BRACKEN, D. K. (1958) Appraising competence in reading in content areas. In Evaluation of Reading,

Supplementary Monograph No. 88. Chicago: University of Chicago. [Reprinted in R. Farr (1970) (ed.)

Measurement and evaluation of reading. New York: Harcourt, Brace, 301±309.]

CASE, R. (1985) Intellectual development ± birth to adulthood. New York: Academic Press.

CASE, K. E. and FAIR, R. C. (1994) Principles of macroeconomics (3rd edn.) Englewood Cliffs, NJ: Prentice

Hall.

CARRELL, P. L. (1987) Readability in ESL. Reading in a Foreign Language, 4, 21±40.

CHENG, P. W. and HOLYOAK, K. J. (1985) Pragmatic reasoning schemas. Cognitive Psychology 17, 391±416.

COHEN, A., GLASMAN, H., ROSENBAUM-COHEN, P. R., FERRARA, J. and FINE, J. (1988) Reading English for

specialized purposes: discourse analysis and the use of student informants. In P. Carrell, J. Devine and

D. E. Eskey (eds.), Interactive approaches to second language reading. Cambridge: Cambridge University

Press, 152±167.

DAVISON, A. and KANTOR, R. N. (1982) On the failure of readability formulas to define readable texts: a case

study from adaptations. Reading Research Quarterly, 26, 187±209.

GILLILAND, J. (1972) Readability. London: University of London (for the United Kingdom Reading

Association)

GREENE, B. B. (1992) An exploratory comparison of the vocabulary characteristics of discourse in

macroeconomics and accounting ± why reading economics may be different from reading accounting.

Unpublished mimeo. 35 pages.

GREENE, B. B. (1997) Verbal abilities, gender and the introductory economics course: a new look at an old

assumption. Journal of Economic Education, 28, 13±30.

GROSSMAN, F. and SIMON, J-P. (1997) Comment les etudiants de premiere annee lisent les textes de specialite.

`Saussure, il a fait ses trucs sans penser a nous. In G. Taillefer and A. K. Pugh (eds.) Reading in the

university ± first, second and foreign languages. Toulouse: Presses de L'Universite des Sciences Sociales,

213±228.

HARRIS, A. J. and SIPAY, E. R. (1971) Effective teaching of reading (2nd edn.). New York: McKay.

HARRISON, C. (1980) Readability in the classroom. Cambridge: Cambridge University Press.

HARRISON, S. and BAKKER, P. (1998) Two new readability predictors for the professional writer: pilot trials

Journal of Research in Reading, 21, 121±138.

JONGSMA, E. A. (1980) Cloze instruction research: A second look Newark, DE: International Reading

Association.

KEMPER, S. (1983) Measuring the inference load of a text. Journal of Educational Psychology, 75, 391±401

KINGSTON, Jr., A. J. (1960) The measurement of reading comprehension. In O. S. Causey and E. P. Bleismer

(eds.) Research of evaluation in college reading Ninth Yearbook of the National Reading Conference..

Fort Worth: Texas Christian University Press, 88±93. [Reprinted in R. Farr (ed.) Measurement and

evaluation of reading (1970). New York: Harcourt, Brace, 230±236.]

KINTSCH, W. (1979) On modeling comprehension. Educational Psychologist, 14, 3±14.

KINTSCH, W., KOZMINSKY, E., STREBY, W., MCKOON, G. and KEENAN, J. (1975) Comprehension and recall of text

as a function of content variables. Journal of Verbal Learning and Verbal Behavior 14, 196±214.

KINTSCH, W. and VIPOND, D. (1978) Reading comprehension and readability in educational practice and

psychological theory. In L. Nilsson (ed.) Memory: processes and problems. Hillsdale, NJ: Erlbaum,

329±365.

KINTSCH, W. and YARBROUGH, J. C. (1982) Role of rhetorical structure in text comprehension. Journal of

Educational Psychology, 74, 828±834.

KLARE, G. R. (1984) Readability. In P. D. Pearson (ed.) Handbook of reading research. New York:

Longman, 681±744.

LIAU, T., BASSIN, E., MARTIN, C., and COLEMAN, E. (1976) Modification of the Coleman readability formulas.

Journal of Reading Behavior, 8, 381±386.

LEVENSTON, E. A., NIR, R. and BLUM-KULKA, S. (1984) Discourse analysis and the testing of reading

comprehension by cloze techniques. In A. K. Pugh and J. M. Ulijn (eds.), Reading for professional

purposes. London: Heinemann, 202±212.

LUCY, G. (1997) Argument structure as context in persuasive-expository texts. In G. Taillefer, and A. K.

Pugh (eds.) Reading in the university ± first, second and foreign languages. Toulouse: Presses de

L'Universite des Sciences Sociales, 87±98.

LUMSDEN, K. G. and SCOTT, A. (1987) The economics student reexamined: male-female differences in

comprehension. Journal of Economic Education. 19, 365±75.


96 GREENE

MANDLER, J. M. (1984) Stories, scripts, and scenes: Aspects of schema theory. Hillsdale, NJ: Erlbaum.

MARTON, F. (1975) On non-verbatim learning: I. Level of processing and level of outcome. Scandinavian

Journal of Psychology, 16, 273±279.

MARTON, F. and DAHLGREN, L. O. (1976) On non-verbatim learning: III. The outcome space of some basic

concepts in economics. Scandinavian Journal of Psychology, 17, 49±55.

MARTON, F. and SALJO, R. (1976) On qualitative differences in learning: I. Outcome and process British

Journal of Educational Psychology, 46, 4±11.

MCCONNELL, C. R. (1983) Readability: blind faith in numbers? Journal of Economic Education, 14, 65±71.

MCCONNELL, C. R. and BRUE, S. L. (1996) Macroeconomics (13th edn.) New York: McGraw-Hill.

MILLER, L. R. (1975) Predictive powers of multiple-choice and cloze-derived readability formulas. Reading

Improvement, 12, 52±58.

MIROWSKI, P. (1989) More heat than light. Cambridge: Cambridge University Press.

OLAH, E. (1984) How special is special English? In A. K. Pugh and J. M. Ulijn (eds.) Reading for professional

purposes ± studies and practices in native and foreign languages London: Heinemann, 223±226.

OTERO, J. and KINTSCH, W. (1992) Failures to detect contradictions in a text: What readers believe versus

what they read. Psychological Science, 3, 229±235.

PERERA, K. (1980) The assessment of linguistic difficulty in reading material. Educational Review, 32,

151±161.

PRINS, E. D. and ULIJN, J. M. (1998) Linguistic and cultural factors in the readability of mathematics texts: the

Whorfian hypothesis revisited with evidence from the South African context. Journal of Research in

Reading, 21, 139±159.

POUW, A. and VAN DER TUIN, P. (1997) Appreciation de contenu d'un texte non-litteraire en langue etrangere

ou en langue premiere. Code linguistique et theme: quelle interaction? In G. Taillefer and A. K. Pugh

(eds.) Reading in the university ± first, second and foreign languages. Toulouse: Presses de L'Universite

des Sciences Sociales, 121±136.

PUGH, J. E. (1988) Using the Cloze test to predicate college biology grades. The American Biology Teacher,

50, 148±149.

RANKIN, E. F. (1959) The Cloze Procedure ± Its Validity and Utility. In O. S. Causey and W. Eller (eds.)

Eighth yearbook of the National Reading Conference Fort Worth: Texas Christian University Press,

131±44. [Reprinted in R. Farr (ed.) Measurement and evaluation of reading. (1970) New York: Harcourt,

Brace & World, Inc., 237±253.]

REIF, F. and LARKIN, J. (1991) Cognition in scientific and everyday domains: comparison and learning

implications. Journal of Research in Science Teaching, 28, 733±760

RUMELHART, D. E. and ORTONY, A. (1977) The representation of knowledge in memory. In R. C. Anderson,

R. J. Spiro and W.E. Montague (eds.) Schooling and the acquisition of knowledge. Hillsdale, NJ:

Erlbaum, 99±135.

RYE, J. (1982) Cloze procedure and the teaching of reading London: Heinemann.

SANDERS, T., SPOOREN, W. and NOORDMAN, L. (1993) Coherence relations in a cognitive theory of discourse

representation. Cognitive Linguistics, 4, 93±133.

SCHILLER, B. R. (1994) The Micro economy today (6th edn.) New York: McGraw-Hill.

SIEGFRIED, J. J., BARTLETT, R. L., HANSEN, W. L., KELLEY, A. C., MCCLOSKEY, D. N. and TIETENBERG, T. H. (1991)

The status and prospects of the economics major. Journal of Economic Education, 22, 197±224.

SIEGFRIED, J. J. and STRAND, S. H. (1977) Sex and the economics student. The Review of Economics and

Statistics, 59, 247±49.

SIMON, H. A., HINSLEY, D. A. and HAYES, J. R. (1989) From words to equations: meaning and representation in

algebra word problems. In H. A. Simon (ed.) Models of thought, volume II New Haven: Yale University

Press, 469±480.

TAILLEFER, G. and PUGH, T. (1998) Strategies for professional reading in L1 and L2. Journal of Research in

Reading, 21, 96±108.

TAYLOR, W. L. (1956) Recent developments in the use of cloze procedure. Journalism Quarterly, 33, 42±48.

TAYLOR, W. L. (1957) `Cloze' readability scores as indices of individual differences in comprehension and

aptitude. Journal of Applied Psychology, 41, 19±26

TURNER A. and GREENE, E. (1978) The construction and use of a propositional text base (JSAS Catalog of

selected documents in psychology. MS 1713)

ULIJN, J. M. and STROTHER, J. B. (1995) Communicating in business and technology. Frankfurt am Main: Peter

Lang.

US DEPARTMENT OF EDUCATION (1996) Digest of education statistics. US Government Printing Office.



VACHON, M. K. and HANEY, R. E. (1991) A procedure for determining the level of abstraction of science

reading material. Journal of Research in Science Teaching, 28, 343±352.

VAN DIJK, T. A. and KINTSCH, W. (1983) Strategies of discourse comprehension. New York: Academic Press.

WAMPLER, B. E. (1990) Grammatik IV (Version 1.0). San Francisco: Reference Software International.

WEIDLER, S. D. (1984) Reading in the content area of science. In M. M. Dupuis (ed.) Reading in the content

areas: research for teachers. Newark: International Reading Association, 54±65.

Address for correspondence: PROFESSOR BENJAMIN B. GREENE, Department ofEconomics & Finance, Perdue School of Business, Salisbury State University, 1101Camden Avenue, Salisbury, MD 21801-6860, USA. E-mail: [email protected]

APPENDIX A: SAMPLE CLOZE QUIZ[Correct answers in square brackets]

1. Consider, for example, an increase in government _______________ [purchases, 2](G) of $10 billion. 2. This increase in expenditure causes firms' _______________[inventories, 3] to be smaller than planned. 3.Unplanned inventory _______________[reductions, 2] stimulate production, and firms increase _______________ [output](Y). 4. But because _______________ [added, 3] output means added income,_______________ [some] of which is subsequently spent, _______________ [con-sumption] spending (C) also increases. 5. The final _______________ [equilibrium]level of output is thus higher by a _______________ [multiple, 6] of the initialincrease in _______________ [government, 1] purchases.

6. This multiplier story is _______________ [incomplete], however. 7. As aggregateoutput (income) increases, an impact is _______________ [felt] in the _______________ [money, 8] market. 8. Specifically, the increase in income (Y) increases the_______________ [demand, 9+10] for money (_______________ [MD]). 9. (For themoment, we assume that the _______________ [FED] holds the quantity of moneysupplied [MS] constant.) 10. The resulting disequilibrium, with the quantity of_______________ [money, 8, 9] demanded _______________ [greater, 8+9] than thequantity of money supplied, causes the _______________ _______________ [interestrate, 12+13] to rise. 11. An _______________ [increase, 1] in G thus increases_______________ [both] Y and i. 12. This increase in `i' has a _______________ [side]effect, however. 13. Remember that a higher interest rate causes planned__________________ [investment, 15] spending to decline. 14. Because planned__________________ [investment, 15] spending is a component of planned__________________ [aggregate, 15] expenditure, the decrease in IP works__________________ [against] the increase in G. 15. An increase in governmentspending increases planned aggregate expenditure and increases aggregate_______________ [output, 3 or 7] , but a decrease in planned investment_______________ [reduces] planned aggregate expenditure and _______________[decreases] aggregate output.

16. This tendency for increases in government spending to _______________[cause] reductions in private investment is called the ______________________________ [crowding out] effect. 17. Without any expansion in the_______________ [money, 9] supply to accomodate the rise in income and increasedmoney demand, planned investment spending is partially _______________[crowded] out by the higher _______________ [interest, 13] rate.


98 GREENE

testing reading comprehension of theoretical discourse with cloze

Documents