Are banked cloze items sensitive to discourse level constraints?

深圳大学 外国语学院 郭 丽

A study into the validity of a new test format in College English Test, Band 4 (CET4), a national English test for university students in the mainland of China.

1. Introduction 1. 1 Types of cloze

* Fixed-ratio vs rational cloze A bird watcher, in the course _ ___ a day’s rambling, almost always f①

inds __ ___ species of woodpeckers. Sometimes they reveal ___ _② ③__ whereabouts by piercing call notes, sometimes __ ___ their stead④y hammering on tree trunks.

(Key: of, several, their, by)

……In __ __, on the other hand, while individuals are important, peo①ple also try to do everything they can for the other group members. They do this because it is thought that if the __ __ succeeds then each ②member will succeed. So a person’s life away from work is still more relevant to his or her job is __ ___. ③

(Key: Japan, group, America) * Constructed-response vs selected-response cloze (open-ended vs multiple-choice or banked cloze)

* Exact word scoring vs acceptable word scoring

1.2 Banked cloze in CET4

A new test format in College English Test Band 4

Old test format (before 2007)

(Vocabulary and structure)

** We don’t know why so many people in that region like to wear dresses of such _____ colors.

A) low B) humble C) mild D) dull

(College English Test Band 4, June 2005)

Example of banked cloze Some years ago I was offered a writing assignment that would

require three months of travel through Europe. I had been abroad a couple of times, but I could hardly __1__ to know my way around the continent. Moreover, my knowledge of foreign languages was __2__ to a little college French.

I hesitated. How could I, unable to speak the language, __3_ unfamiliar with local geography or transportation systems, set up __4__ and do research? It seemed impossible, and with considerable __5__I sat down to write a letter begging off. Halfway through, a thought ran through my mind: you can’t learn if you don’t try. So I accepted the assignment.

There were some bad__6__. But by the time I had finished the trip I was an experienced traveler. And ever since, I have never hesitated to head for even the most remote of places, without guides or even __7___ bookings, confident that somehow I will manage.

The point is that the new, the different, is almost by definition __8__. But each time you try something, you learn, and as the learning piles up, the world opens to you.

I’ve learned to ski at 40, and flown up the Rhine River in a ___9___. And I know I’ll go on doing such things. It’s not because I’m braver or more daring than others. I’m not. But I’ll accept anxiety as another name for challenge and I believe I can ___10___ wonders.

totally scary regret reduced newsmoments manufacture limited interviews declareconstantly claim balloon advanced accomplish

(College English Test 4, June 2008)

ppt22

The purpose of banked cloze component in CET4——

“ to assess students’ ability to understand and employ words at discourse level”

(National College English Testing Committee, China. 2006: Syllabus for College English Test Band 4. )

2. Review of literature and Necessity for the study

Contradictory conclusions concerning the construct validity of cloze

** can measure text-level processing ability

(Ramanauskas, 1972; Chihara et al., 1977; Bachman, 1985; Jonz, 1991; Fotos, 1991; Chavez-Oller et al., 1994; Storey, 1997; Yamashita, 2003)

** cannot assess comprehension beyond sentence level.

(e.g., Alderson, 1979;1980; Kibby, 1980; Markham, 1985)

Relevant research (on rational deletion and banked cloze)

Yamashita, Junko. 2003 (Processes of taking a gap-filling test: comparison of skilled and less

skilled EFL readers. Language Testing 20: 267-293.)

*12 Japanese EFL students * Gap-filling test -- key content words and cohesive devices were deleted * Thinking aloud * Text-level information 52% sentence level 17%; clause level 10%; extra-textual 4% * A gap-filling test can be used as a test to measure higher order processing ability.

Relevant research (continued) Two studies into banked cloze procedures by Chinese researchers.

Gao Xiaoying et al (2008)* Simultaneous introspection and immediate retrospection* 18 Chinese University students* Information sources: clause (47%) ; text level (23%) sentence (22%) ; extra-textual (8%) * The proficient readers -- context-based reading model the less proficient readers -- word-based approaches.

Liu (2009) * Similar to the above in research focus and design* 6 second-year students in a Chinese university* Local syntactic (36%) local semantic (29%); sentence (20%); text-level (16%) information.

Relevant research (continued) The present study adopts an approach which share similarities with Jo

nz (1991) and Chihara et al (1977) —— examining the sensitivity of the cloze items to the disruptions to the normal order of texts.

Scrambled-text research / on standard fixed-ratio cloze

Need for this study

Little research has been conducted into banked cloze procedure and consequently there is little evidence as to whether it is sensitive to inter-sentential constraints as the CET4 designers claim it to be.

The previous research on banked or rational deletion cloze is mostly qualitative, involving only a small number of subjects.

As a high-stake national exam, CET4 exerts a powerful backwash impact upon the English instruction and testing at tertiary level in China.

Research questions

1) Do subjects use text-level information in completing the banked cloze test?

2) What types of information (within-sentence or beyond-sentence) are used to answer each item?

3) Does English proficiency affect the use of text-level inform

ation?

3. Study One Materials Normal Test One

*Reading in Depth Section, College English Test Band 4 Dec, 2007 *222 words long *10 contents words (4 nouns, 2 verbs, 2 adjectives and 2 adverbs) deleted Scrambled Test One *Produced by scrambling the sentences in Normal Test One *Cause maximum interruption of any constraint on comprehension attribut

able to sentence sequence. Response and scoring Students were required to select one word for each blank from the list of c

hoices. Since no change in the form was necessary, the scoring was entirely objective.

Normal Test One As war spreads to many corners of the globe, children sadly have been

drawn into the center of conflicts. In Afghanistan, Bosnia, and Colombia, however, groups of children have been taking part in peace education __1__. The children, after learning to resolve conflicts, took on the __2__ of peacemakers. The Children’s Movement for Peace in Colombia was even nominated (提名 ) for the Nobel Peace Prize in 1998. Groups of children __3___  as peacemakers studied human rights and poverty issues in Colombia, eventually forming a group with five other schools in Bogota known as The Schools of Peace. The classroom __4__ opportunities for children to replace angry, violent behaviors with __5__, peaceful ones. It is in the classroom that caring and respect for each person empowers children to take a step __6__ toward becoming peacemakers. Fortunately, educators have access to many online resources that are __7__ useful when helping children along the path to peace. The Young Peacemakers Club, started in 1992, provides a Website with resources for teachers and ___8__ on starting a Kindness Campaign. The World Centers of Compassion for Children International call attention to children’s rights and how to help the __9__ of war. Starting a Peacemakers’ Club is a praiseworthy venture for a class and one that could spread to other classrooms and ideally affect the culture of the __10__school.

acting assuming comprehensive cooperative entire especially    forward          images              information         offers      projects       respectively   role               technology        victims

Subjects 165 second-year students, 100 male and 65 female, majoring in Busin

ess Administration, Architecture, Software Design, Electronic Engineering and Chemistry at Shenzhen University, P.R. China

Procedure The subjects were randomly assigned to the two versions of cloze ——

the odd rows (Group A1) took Normal Test One and the even rows (Group B1) completed Scrambled Test One.

Proficiency Exam * Administrated a week after the cloze tests * Check the equality of the two groups * Divide the subjects into 3 proficiency groups * Listening, Vocabulary and Structure, Reading, Translation and Essay

writing. * Reliability value (Cronbach’s Alpha) -- 0.78.

N Mean SD t p

Group A1 87 63.03 12.03

.983 .327Group B1 78 61.29 10.54

Table 1 Proficiency Exam Scores for Groups A1 and B1

Table 2 Table 2 Proficiency Exam Scores for Three Proficiency Groups

N Mean SD F p

High Group 1 58 74.17 5.93

267.44 .000Middle Group 1 49 61.88 2.86

Low Group 1 58 50.53 6.64

Total 165 62.21 11.35

pos hoc tests: significant difference in each of the three pairs (.000).

Table 3 Mean Scores in Normal and Scrambled Conditions

N Mean SD t p

Group A1 87 6.30 2.61

1.962 .052Group B1 78 5.53 2.53

Question 1: Do subjects use text-level information in completing the banked cloze test?

Results and discussion

Facility Index

IF

Difference

t pNormal

(N=87)

Scrambled

(N=78)

projects .79 .62 +.17 2.52 .01

role .74 .53 +.21 2.85 .01

acting .66 .56 +.10 1.19 .23

offers .92 .95 -.03 -.75 .46

cooperative .36 .36 .00 -.04 .97

forward .47 .37 +.10 1.22 .22

especially .72 .72 .00 -.03 .98

information .34 .29 +.05 .63 .53

victims .77 .68 +.09 1.35 .18

entire .53 .45 +.08 1.10 .28

Table 4 Facility Index in Normal and Scrambled Conditions

Question 2: What types of information are used to answer each item?

PH=.02 <.05PL=.08>.05

Question 3: Does English proficiency affect their use of text-level information?

PH=.04 <.05

PL=.08 >.05

PM=.08 >.05

4. Study TwoPurpose Expand the generalizability of the research, same issues explored

Materials Normal Test Two *Reading in Depth Section, College English Test Band 4, June, 2008. *242 words *Chronological account / different type of text *10 contents words (4 verbs, 3 nouns, 2 adjectives and 1 adverbs) deleted Scrambled Test Two * Produced by rearranging the sentences in Normal Test Two * Inter-sentential connections were meant to be thoroughly disrupted

Subjects 141 second-year students, 108 male and 31 female, majoring in In

formation Technology, Civil Engineering and Maths at Shenzhen University, P.R. China

Procedure The subjects were randomly assigned to the two versions of cloze

—— the odd rows (Group A2 ) took Normal Test Two and the even rows (Group B2) completed Scrambled Test Two.

Proficiency Exam * Administrated a week after the cloze tests * Applied as a criterion to evaluate their English competence * Listening, Vocabulary and Structure, Reading, Translation and Es

say writing. * Reliability value (Cronbach’s Alpha) -- 0.77

N Mean SD t p

Group A2 69 67.00 11.07

.487 .627Group B2 72 66.14 9.93

Table 5 Proficiency Exam Scores for Groups A2 and B2

Table 6 Proficiency Exam Scores for Three Proficiency Groups

N Mean SD F p

High Group 2 54 77.04 4.89

255.947 .000Middle Group 2 36 66.42 2.66

Low Group 2 51 55.57 5.91

Total 141 66.56 10.47

pos hoc tests: significant difference between each of the three pairs (.000).

Table 7 Mean Scores in Normal and Scrambled Conditions

N Mean SD t p

Group A2 69 5.55 2.10

3.341 .001Group B2 72 4.39 2.03

Question 1Do subjects use text-level information in completing the banked cloze test?

Results and discussion

Text type? Cloze passage in Study One was largely an account of facts and did not exhibit much top-level constraint on the order in which propositions were presented while the passage in the Study Two was a chronological account of a writer’s experience, thus more sensitive to sentence reordering.

What has brought out the different alpha values in the two studies? (Study One: MGroupA1=6.30; MGroupB1= 5.53; t=1.962, p>.05)

Question 2 Does English proficiency affect the use of text level clues?

Diff=1.49

PH=.003 <.05

Diff=0.77PL=.234 >.05

Diff=0.98PM=.140 >.05

As subjects become more proficient in English, they are better able to benefit from discourse constraints ranging across sentence boundaries.

Facility Index

IF

Difference

t pNormal

(N=69)

Scrambled

(N=72)

accomplish .54 .53 +.01 .024 .981

limited .84 .84 .00 .080 .937

scary .25 .10 +.15 2.399 .018

totally .74 .75 - .01 -.194 .846

interviews .64 .44 +.20 2.412 .017

claim .38 .27 +.11 1.304 .194

advanced .58 .56 +.02 .216 .829

balloon .81 .56 +.25 3.320 .001

moments .52 .12 +.40 5.541 .000

regret .26 .23 +.03 .384 .701

Table 8 Facility Index in Normal and Scrambled Conditions

Question 3 What types of information are used to answer each item?

5. Conclusion

There may be considerable variation in the effectiveness of banked cloze in assessing discourse level abilities. Different cloze items carry with them different amount or level of information. More proficient test-takers have better access to discourse level clues than the less proficient. And a well-ordered narrative/chronological text could function better than a loosely connected encyclopedic description which simply lists facts. Therefore it can be concluded that tester/researcher intervention is necessary for developing sound banked cloze tests.

Limitations

Only two samples of banked cloze tests Convenience sampling of subjects Subjects in the two separate studies –

equivalent in their English competence?

Test one Test Two

Group 1 Normal Scrambled

Group 2 Scrambled Normal

Thank you very much!