1 examination impact: monitoring how teachers cope with change dianne wall tania hork lancaster...

27
1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Horák Lancaster University UK

Upload: gertrude-goodman

Post on 18-Jan-2018

216 views

Category:

Documents


0 download

DESCRIPTION

3 Our impact study Where? Central and Eastern Europe Who? TOEFL preparation course providers

TRANSCRIPT

Page 1: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

1

Examination Impact: Monitoring how teachers

cope with change

Dianne WallTania Horák

Lancaster University UK

Page 2: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

2

Background

• Next Generation TOEFL (NGT) to be launched September 2005 (in certain sites only)

• Two impact studies commissioned by Educational Testing Service (producer of TOEFL)

Page 3: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

3

Our impact study

• Where?Central and Eastern Europe

• Who?TOEFL preparation course providers

Page 4: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

4

Phase 1 Phase 2 Phase 3

Baseline Transition Impact

Jan 03 to Oct 04 Oct 04 to Mar 06 After introduction of NGT

-Description of current TOEFL preparation classes-Class observations, interviews with teachers, DoS and students.-10 sites in 6 countries

-Tracking study -’Interviews’ via Internet

-Description of NGT preparation classes -Class observations, interviews with teachers, DoS and students.- same sites

Page 5: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

5

Theoretical underpinnings 1) Henrichsen (1989) – Hybrid Model

of the Diffusion / Implementation Process

2) Messick (1996) - ‘evidential link’

Page 6: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

6

Henrichsen’s model

Antecedents Process Consequences

Page 7: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

7

Process • The Receivers• The Innovation• Characteristics of Communication• The Resource System • The User System • Inter-elemental factors

Page 8: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

8

How our study fits the model:

Antecedents Process Consequences

Baseline Transition Impact

Page 9: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

9

Messick’s ‘evidential link’

Antecedents Process Consequences

‘evidential link’

Page 10: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

10

Phase 2• Who? 6 teacher participants from

Phase 1 • Where? 5 countries: Bulgaria, Croatia,

Lithuania, Poland, Slovakia.• When? data collection period: January

- May 05• How? Computer Mediated

Communication• What? investigation of pps’ awareness

and understanding of and reactions to NGT, to help us understand how they are preparing for the arrival of NGT

Page 11: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

11

What?• Interviews based on monthly

tracking questions • Completion of tasks designed to

familiarise participants with NGT • Interviews discussing the task

responses

Page 12: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

12

Data collection activities Interview 1

Task sentto pps

Pps return taskResearchers

discuss responses

Interview 2

Page 13: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

13

The Tasks• January - Features of TOEFL preparation

classes

• February - Awareness of current and future TOEFL

• March - Focus on the new writing tasks

• April - Focus on the new speaking tasks

• May - Prediction of actual content and methodology of pps’ NGT preparation classes

Page 14: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

14

Findings

• Understanding of NGT

• Materials

• Teacher Support

Page 15: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

15

Understanding of NGT

Teachers seem clear about the format of NGT but less clear about the actual constructs.

Page 16: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

16

Understanding of NGT

‘I don't think the matter of accent was really addressed, though maybe those who grade the tests have an implicit understanding of how accent plays a role and will score accordingly. In preparing students for tests, it would help to have the issue of accents and how they affect scoring addressed directly in the rubric.’

Page 17: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

17

Understanding of NGT

Conclusion

Awareness-raising requires more than providing examples of the test.

Page 18: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

18

Materials

Teachers rely heavily on published materials.

Page 19: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

19

Materials ‘I can’t even dare think of something like that but I suppose I’ll have to improvise. The other possibility is to reschedule the [NGT] classes to the moment when there will be materials.’

Page 20: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

20

Materials

ConclusionsImpact will probably be affected by the textbook writers Impact of the test may well be hard to distinguish from impact of the textbook

Page 21: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

21

Teacher Support

The test provider has supplied various forms of teacher support but they are not necessarily accessible to the teachers who need them.

Page 22: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

22

Teacher Support

‘I just heard it mention by [the Director] … She also said it is very expensive …. I cannot afford, the center cannot afford.’

Page 23: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

23

Teacher SupportConclusion

Careful consideration must be paid to the logistics of teacher support if it is to be effective and reach those who need it, when they need it.

Page 24: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

24

Professionalism• Test Provider - by commissioning

impact studies

• TOEFL teachers – by participating in our study

Page 25: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

25

For further information • About the project:[email protected] [email protected]

• About NGT:http://toeflpractice.ets.org

Page 26: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

26

Page 27: 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Hork Lancaster University UK

27

Tracking questions• Have you learnt anything about New TOEFL

which you didn’t know last month? If so, what?•  Is the new exam being discussed in your

institution (e.g. by other teachers/ management)? If so, what are the main issues being discussed? 

• Have students asked about the new TOEFL? If so, what have they asked about?

• Are you worried/ concerned about anything concerning the new TOEFL? If so, what?

• Is there anything else of interest concerning new TOEFL which has happened this month?