myth, metaphor and reality of washback in greek elt ...testingforum.hau.gr/docs/c.tsagari-ok.pdf ·...

2nd Language Testing & Evaluation ForumHellenic American Union, Athens, Greece June 16-18,2006

Myth, Metaphor and Reality of Washbackin Greek ELT: Findings from an Empirical

Research Study

Dina [email protected]

• The FCE exam (Cambridge/ESOL)

• Motivation for the study

Background

Brief outline of the FCE exam

Sections Administration Time

Paper 1: Reading Comprehension Section A: 25 multiple-choice items Section B: passage-based, 15 multiple-choice in total for 3 texts

Group 60 min (am)

Reading Comprehension

Brief outline of the FCE exam Sections Administration Time

Paper 2: Composition Two 120-180 word essays Ratings: 6 bands, 0-20

Group 90 min (am)

Composition


Paper 3: Use of English Task 1: 20 gap-filling cloze Task 2: 8-10 transformations Task 3: 10 gap-filling, separate sentences Task 4: 7-8 rewording, separate sentences Task 5: directed writing task, table/graphic/note-based, two to three paragraphs long

Group 120 min (pm)

Use of English


Paper 4: Listening Number of items, score-per-item, and item type vary per test form; usually three or four recorded texts employing multiple-choice, true-false, note-taking etc

Group approx.35 min

Listening


Paper 5: Speaking The interview uses themes based on information packages Stage 1: general conversation (brief) Stage 2: picture(s): describe, discuss Stage 3: theme-related passage(s): read and discuss Stage 4: discussion task, problem-solving task, role-play, etc. Ratings: Fluency, Grammatical accuracy, Pronunciation (sentences), Pronunciation (individual sounds), Interactive communication, Vocabulary resource; all on scales 0-5

Individual, pair or group

15 min

Speaking

• The FCE exam (Cambridge/ESOL)

• Motivation for the study

Background

Articles published by local EFL practitioners(Gabrielatos, 1993; Kenny,1995; Dourtourekas, 1998, etc)

FCE exam has negative washback on◙ Teaching◙ Materials◙ Students◙ Teachers◙ Parents

The Myth

Does the FCE exam influence the teaching and learning that takes place in the Greek EFL context of intermediate level frontistiria classes? If yes, how?

General Research Question

Washback studies (Alderson & Wall, Hughes, Bailey, Cheng, Watanabe, Gosa, etc)

EFL/ESL teaching, learning and motivation (Gardner, Deci,Weiner, Dornyei & Csizer, etc)

Materials construction and evaluation (Allwright, McDonough and Shaw; Ellis; Littlejohn; Roberts; Tomlinson, etc)

Theoretical underpinnings



Materials construction and evaluation (Allright, McDonough and Shaw; Ellis; Littlejohn; Roberts; Tomlison, etc)


Washback studies (Alderson & Wall, Bailey, Hughes, Cheng, Watanabe, Gosa, etc)




Overview of the study

Interviews with 15 FCE teachers



FCE exam



Teaching materials

FCE exam



Teaching materials Methodology

FCE exam

The Metaphor


Teaching materials Methodology Students

FCE exam




Textbooks

FCE exam

How has the FCE examination influenced the nature of the teaching materials teachers use in their classes?

How exactly do the materials reflect the aims and specifications of the FCE exam?

To what extent do material writers understand the exam requirements and realise these in the materials they design?

FCE textbooks -Research Questions

• 5 FCE textbooks (+ TB, audiotapes)

• 2 General English textbooks (+ TB, audiotapes)

FCE textbooks -Main Data

Special instrument (CATM-FCE): • 15 page long• 3 stages (A. General, B. Detailed, C. Overall)• 9 parts • 89 questions: variety of features, e.g. info about

the exam, exam techniques, nature of texts (authenticity, type, source..), tasks (type, no of items ..), instructions, scoring criteria, time pressure, etc

FCE textbooks -Instrumentation

2 Phases of Piloting:• Myself with 3 textbooks + CATM-FCE v1• 12 raters with 1 textbook unit + CATM-FCE v2

1 Phase of Validation:• 5 evaluators with 7 textbooks and part of CATM-FCE

FCE textbooks –Piloting and Validation

• Total number of texts: 283

• Total number of tasks: 1934

FCE textbooks –The textbook data

• Microsoft ACCESS and EXCEL 2003

• SPSS, v. 11.5

FCE textbooks -Analysis of the data

FCE textbook washback varied in intensity and type

Strong(no of tasks,

text and task types)

Positive(read/list texts,gram/phr vs)

Negative(writing tasks,

authentic list/readtexts)

No washback(time pressure,

no passages, nature of ‘com. act.’)

Weak(instructions, word limit, listening/speaking mode)

Textbook washback –Results 1

FCE textbooks• vary in the way they reflect the exam

specifications in their materials

• locally-produced vs internationally-produced textbooks

Textbook Washback –Results 2

Omissions:• Scoring criteria for speaking• Work on pronunciation• Tasks on optional reading/writing tasks• ‘Communicative activities’ and ‘short

texts’


Features not directly related to the exam :• Keep a thematic link across sections • Include pre- and post-tasks• Follow the same approach to writing• Include extra Grammar-Vocabulary-

Phrasal Verb sections


The FCE exam specifications are reflected in the content of the textbooks used to prepare students for it but this varies across exam features and textbooks

General Conclusion

Exam-preptextbooks

writers/publishers

Exam specifications

The role of writers/publishers




Textbooks

FCE exam

Use of exam-prep teaching materials in class

• To what extent do FCE textbooks determine the content and methods of teaching and testing?

• To what extent do such materials shape teachers’& students’ understanding of the exam?

• What is the reaction of students towards the content of their FCE textbooks?

Research Questions

Students’ attitudes and feelings

• Does the FCE exam influence students’ attitudes towards learning English?

• Does the FCE exam induce feelings of stress and anxiety in students?

• Does the FCE exam make students feel bored and demotivated?

Research Questions




Textbooks

FCE exam




Textbooks Ss’ Diaries

FCE exam

• a way of collecting data discreetly and regularly providing also a description of ‘phenomena over time’

• elicit a rich source of introspective information on how learners view their experiences in acquiring an L2

• students feel more comfortable and confident recording their thoughts and feelings by themselves without any intrusion

• give me the opportunity to try a new research method in the study of washback

Why student diaries?

• 29 FCE Student-Diarists (16F/13M: 13-17 years old)

• 1 NNS English Teacher

• 1 frontistirio

Students’ Diaries -Participants

Students’ Diaries –The diary data

Female Male Overall

Total 673 475

Percent 58.6% 41.37%

Average 42 36.5

Total 48,932 31,791

Percent 60.6% 39.4%

Average 3,058 2,445

80,723Words

Entries 1,148

Phase 1: First reading of the diary data

Phase 2: ‘Transcribing’ the diary data

Phase 3: The Analytic Framework

Students’ diaries -Data Analysis

Coding Scheme

Literature on test

washback

Teacher’s claims

(preliminary interviews) FCE

specifications and exam

papers

My own predictions

and teaching experience

Typing, editing and

analyzing the diary dataThe

textbook analysis

Students’ diaries -The Analytic Framework

• Content (language skills, topics, tasks, methods, test-taking, problems, etc)

• Homework• Assessment practices • Ss’ Feelings/Attitudes (above)• Attitudes (English & FCE)

Students’ diaries -Categories of analysis

Validity of the Coding Scheme• 12 raters

2 Phases of Validation of Interpretation• 12 raters (various coded extracts)• 1 rater (two categories of coded data)

Students’ Diaries –Validation procedures

Qualitative analysis software

• Atlas.ti 4.2 (Murh, 1997)

• WordSmith (Scott, 1999)

Students’ Diaries -Analysis of the data

Students’ Diaries – Results 1Textbook Washback

• Reliance on Textbook materials

topics, texts, task types, grammar structures, classroom tests

rate and sequencetextbook terminology impacted on students’

discoursepronunciation, FCE rating scales

Students’ Diaries – Results 2Teacher Washback

• Deviations from Textbook materialsFrequency of skills, methods, feedback, homework

• Teacher was unaware of students’Language problemsAttitudes and feelingsStudents’ extra skills practice (reading, writing, listening)

• Teacher used ‘dictation’ technique

Students’ Diaries – Results 3School Washback

• Background reading text

• ‘Dictation’ technique

• Intensification of exam preparation

• Importance of homework & practice

• Were negative towards reading, listening & grammar

• Suffered under the burden of homework and preparation and performance on classroom tests

• Experienced enormous levels of stress and anxiety• Experienced boredom - uninteresting activities,

materials, frequent testing, homework but• Were willing to expend time and effort

Students’ Diaries – Results 4Students’ Attitudes - Feelings

Students’ DiariesExtracts

‘The lesson was boring because we did reading (I hate reading)’ (D10/M-164)

‘Today we corrected our exercises and then we did some listening. This lesson was too boring because I hate listening exercises’ (D10/M-127-128)

‘We did some boring grammar, which is my worst enemy in the English language’(D16/M-377:379)

Students’ DiariesExtract

‘You know, I’ve just bought the books for Saturday’s lesson. Are they crazy? What have they gave us for homework? I’m wondering if they think that we are machines which can produce and solve problems and do homework at two minutes. That’s too much! And I’m thinking of something else. We didn’t have the books and they gave us so much homework. What are they going to give us during these Saturday’s? A whole book? I feel so pressed! That’s too much! Enough!’


• Suffered under the burden of homework andpreparation and performance on classroom tests





‘Today we wrote a test at English lesson. It was difficult but I believe that I wrote well. I had many stress about the test and when I started writing I tried to focus to find the correct answers. I was studing for the test all day. When I finished the test I felt like the prisoners when they are released’ (D21/F-170:173)


‘It was a dull and boring day. I sat over the exam but I have done great mistakes and feel stupid’(D2/M-75:76)



• Reported increasing levels of stress and anxiety• Experienced boredom - uninteresting activities,



‘However I must admit feeling quite nervous lately, while thinking that there are only three (almost) months left for me to take the FCE exam. In the mean time, I try not to think about it very frequently. And you know what? The bad thing is that the time I have these thoughts the last days is just before going to sleep which makes me stay awake for many hours in the night’’

(D22/F-404:409, Monday 8th March)


‘…. Now that the days are coming for the FCE test I’m nervous and I’m thinking that I don’t have to take this test. But of course this is very stupid’(D26/F-140:141, Saturday 13th April )


‘So. Almost one week has left for the Lower. Every day that passes, I think more and more of it and, of course, feel more stressed. We’ll see’

(D22/F-527:529, Tuesday 21st May)




• Experienced enormous levels of stress and anxiety• Experienced boredom (uninteresting activities,

materials, frequent testing, homework) but• Were willing to expend time and effort


‘Today at the English lesson we done reading and especially the multiple choice. It is boring but it is very good for use because we are testing how good we are for the Lower’ (D21/F-109:111 )


‘Also I heard that we will take a test next Thursday so I was ready to said that I hate to write tests and that I will never take and write a test but I didn’t do it because then I will understand that I said a very stupid thing because if I want to take the FCE I must to try a lot and very hard and also to do this thing I will pass a lot of happy and bad things and I don’t to give up because it has no sense’ (D17/M-196:202)

Students’ Diaries Extract

• Instrumental reasons (job-related) • Testing purposes • Self-confidence & Relatedness • Milieu (parents)• Prevalent beliefs and convictions in

the Greek context

Students’ Diaries – Results 5Students’ motivation towards FCE

• Instrumental reasons (job-related) • Testing purposes• Self-confidence & Relatedness • Milieu (parents)• Prevalent beliefs and convictions in

the Greek context

Students’ Diaries – Results 5Students’ motivation towards FCE

• Minimum cultural interest and need to relate to English speakers/community

• Few references to self-confidence• Instrumentality (job-related)• Milieu (parents)

Students’ Diaries – Results 6Students’ motivation for English


FCE exam

The Myth and Metaphorof FCE washback

TeacherTextbook Learners

The FCE

exam

Greek Society/Milieu

Authors/Publishers

The school

The Reality of FCE washback

• Local educational community• Private language sector• Testing boards• Textbook writers/publishers• Teacher trainers • Teachers• Students & Parents

• Other researchers in test washback

Implications of the study

To give feedback or for further details please contact:

Dina [email protected]

Thank you!

myth, metaphor and reality of washback in greek elt ...testingforum.hau.gr/docs/c.tsagari-ok.pdf ·...

Documents