Romanian SpeakingEvaluation

Cpt. Corina Ispas

Romania

Testing Speaking

From practice to

perfection?

Romanian Speaking Evaluation- short history

• 1998 – testing based on scripted scenario

• 1999 – training received from DLIELC, USA

• 1999-2004 – testing speaking in accordance with the training received (OPI)

• 2002 – beginning of the new testing project (UK, USA)

• 2004 – re-shaping of speaking evaluation to accommodate national needs;

• 2008 – re- analyzing of the requirements to accommodate + rating and evaluation

STRUCTURE OF THE TEST

Warm up

Core

Wind down

Phases of the test

• Warm up

- to put the examinee at ease

- to give an initial idea of the examinee’s level

• Wind down

- to have the examinee leave the test with a feeling of accomplishment

Core of the test

• Level checks

- confirmation of the examinee’s ability to perform the requirements of the given level;

• Level probe- the examiner’s attempt to raise the level of the

examinee’s language by increasing the linguistic demand on the examinee.

From Old to New - From good to better ?!

• Evaluation of national needs– large number of population– OPI – time-consuming

(max.40 min)– difficulties in identifying

certified examiners (elicitation/response chain, three way conversation…)

– Face – validity– Reliability dependent on

human element

Test specifications

• Purpose

e.g. types of speaking being targeted

• Tasks

e.g. test method

Special characteristics

• Materials/ stimuli

e.g. input

• Length

• Topics

• Number/length of tasks

• Channel

Candidatesi.e. number, characteristics

Examiners

i.e. role, characteristics, delivery speed

Tasks:

Test method

Materials/stimuli

Input

Length

Topics

Number/length of tasks

Channel

•question/answer, role-play, information gathering

•verbal and/or visual

•30 words

• basic needs such as meals, lodging, transportation, shopping;

• travel, family, job, everyday activities, hobbies and interests (based on types of speaking elicited)

•13,5 minutes, 3 tasks + warm-up and wind down

•face to face

Candidates:

Number

Characteristics

Examiners

Role

Characteristics

Delivery speed

Administration guidelines

Criteria of assessment

•one at a time

•female/male, military/ civilian working in the military, adults, unknown to the examiners

•examiner and assessors

•female/male, non-native speaker, RoSE trained

•occasionally slower (when re-wording)

• see attached document

•in accordance with STANAG 6001 (2nd edition)

•RoSE rating grid

Level 1Purpose

In typical everyday situations:

1. can begin, maintain and close short conversations by asking and answering short simple questions

2. can exchange greetings

3. can meet minimum courtesy, introduction and identification requirements

4. can elicit and provide predictable, skeletal biographical information

5. can communicate about simple routine tasks in work place

6. can ask for goods, services and assistance

7. can request for information and clarification

8. can express satisfaction, dissatisfaction and confirmation

Level 1

• Short conversation (1, 3, 8,+/- 5)

• Examinee asks questions (1, 4, 7)

• Role play/situation (2, 3, 6, +/-7, 8)

Short conversation – example of level 1 task

Do you have a family? Who are the members of your family?Do you help your (member of family)in the house? What are some of the activities you usually do? How often do you...(name the activity)? What is your favorite activity? When do you usually do this?

- begin, maintain, close short conversation (1)- meet minimum courtesy, introduction and identification requirements(3)

- express satisfaction, dissatisfaction and confirmation(8)

- communicate about simple, routine workplace situation (+/- 5)

Short conversation3 min.

IG3 min.

Role Play3 ½ min

Warm up2 min.

Wind down2 min.

Level 2

Purpose

In typical everyday social and routine workplace situations:

1. can describe people, places and things 2. can narrate current, past and future activities 3. can state facts 4. can compare and contrast 5. can give straightforward instructions and directions 6. can ask and answer predictable questions 7. can often elaborate in common daily communicative

situations

Level 2

• Narration and description (1, 2,+/- 3)

• Compare and contrast (3, 4)

• Instructions/Directions (5)

• Role-play/situation (6, 7)

Narration + description – example of level 2 tasks

People take many courses in the military. Think of one of thecourses you have attended. Tell me what you did there on the first day. Describe one of Instructors so that I can recognize him/her.

- can narrate current, past and future events (2)

- can describe people, places and things (1)

- can state facts (+/- 3)

Narration and description3 ½ min

Compare and Contrast3 min

Role Play3 ½ min

Warm up2 min.

Wind down2 min.

Instr./direction1 ½ min

Level 1-2

Warm up2 min.

Short conversation3 min.

Narration in the pastDescription

3 ½ min

IG3 min.

Compare/Contrast3 min.

Role Play3 ½ min

Instr./direction1 ½ min

Role Play3 ½ min

Wind down2 min.

Level 1 Level 2

Basic questions on surrounding objects

3 min.

Level 0

Testing for level 1-2 – example of variant

Warm up

Level 1 Level 2

Short conversation- 2 Narration + description- 7

Probe L2 (Narration/

description) -7

IG- 5 Compare + contrast - 15

Role Play- 11 Instructions/ Directions- 9

Role Play- 4

Wind down

In most formal and informal conversations on practical, social and professional topics:

1. can discuss particular interests and special fields of competence

2. can use the language to perform such common professional tasks as answering objections, clarifying points, justifying decisions

3. can demonstrate language competence when conducting meetings, delivering briefings, or other extended and elaborate monologues, hypothesizing and dealing with unfamiliar subjects and situations

4. can reliably elicit information and informed opinion 5. can convey abstract concepts in discussions on different

topics 6. can produce extended discourse and convey meaning

Level 3

Level 3

• Supported opinion with descriptive prelude (1, 2, 5, 6)

• Regular monologue (2, 3, 6)

• Role play/situations (2, 4)

• Hypothetical monologue (2, 3, 6)

Supported opinion with descriptive prelude – example of level 3 task

Leadership is one of the most attractive and difficult to define of all human qualities. On The basis of your experience, what do you think makes a leader and which are the qualities and characteristics most frequently associated with leadership? Give 2-3 reasons to support your opinion.

- discuss particular interests and special field of competence (1)

- justify decision (2)

- convey abstract concepts in discussions on different topics(5)

- can produce extended discourse and convey meaning (6)

Supported opinion4 ½ min

Regular monologue3 ½ min

Hypothetical monologue3 ½ min

Warm up2 min.

Wind down2 min.

Role Play5 ½ min

Level 2-3

Warm up2 min.

Narration in the pastDescription

3 ½ min

Support opinion4 ½ min.

Compare/Contrast3 min.

Reg. monologue3 min.

Role Play3 ½ min

Role Play5 ½ min

Hypo. monologue.3 ½ min

Wind down2 min.

Instr./direction1 ½ min

IG3 min.

Role Play3 ½ min

Short conversation3 min.

Level 1 Level 2 Level 3

Testing for level 2-3 – example of variant

Level 1 Level 2 Level 3

Short conversation- 2

Narration +Description - 7

Supported Opinion - 4

Probe L2 (Narration + Description) - 7

Probe L3 (Support Op.6) Regular Monologue - 9

Probe L3(Reg. Monologue 9)

IG- 5 Compare + contrast 4 Role Play - 14

Role Play - 11 Instructions/ directions 4 Hypothetical Monologue - 7

Role Play 4

RATING

Scale focusing on 5 areas of “interest”:• Task Fulfilment

• Lexical & Structural Control

• Delivery

• Output

• Communication Strategies

TASK FULFILMENT

LEVEL 1

• Can communicate in simple everyday situations.

• Functions are typically satisfied within task parameters.

• Response to interlocutor/situation is sufficiently appropriate.

LEVEL 2

• Can communicate in everyday social and routine workplace situations.

• Functions are clearly performed within task parameters.

• Response to interlocutor/situation is generally appropriate.

LEVEL 3

• Can communicate in most formal and informal conversations on practical, social and professional topics.

• Functions are effectively performed within task parameters

• Response to audience/situation is on the whole stylistically appropriate.

LEXICAL AND STRUCTURAL CONTROLLEVEL 1

Frequent errors in vocabulary and grammar. Time concepts are vague. May use only one tense or tend to avoid certain structures. Vocabulary is limited to basic needs and daily routine.

LEVEL 2

Basic grammatical structures are typically controlled. Uses concrete vocabulary. Grammatical and/or vocabulary errors may occur. Although they may cause misunderstandings, the message is conveyed.

LEVEL 3

Effectively combines structure and vocabulary to convey meaning accurately. Structural inaccuracy is rarely the cause of misunderstanding. The use of structural devices is flexible and elaborate. Able to use low frequency vocabulary to convey abstract topics. Errors may occur in low

frequency vocabulary and some highly complex structures.

DELIVERY

LEVEL 1

Seldom speaks with natural fluency. Pronunciation, stress and intonation often distort meaning. Speech is often characterized by hesitations, erratic word order, frequent pauses, straining and groping for words.

LEVEL 2

Handles the tasks with confidence but not with facility. E.g.: pauses for lexical and grammatical planning. Pronunciation, stress and intonation may distort meaning.

LEVEL 3

Speaks readily and naturally in a way that is appropriate to the situation. No need to search for words and phrases. Errors in pronunciation do not distort meaning. Pronunciation may be obviously foreign.

OUTPUT

LEVEL 1

Can speak at the sentence level and may produce strings of to or more simple, short sentences joined by common linking words.

LEVEL 2

Organizes speech in complete but simple paragraphs.

LEVEL 3

Produces extended discourse.

ADMINISTRATION AND RATING ISSUES

- administration guide lines

- two examiners - one examinee

- interaction

- rating process

- monitoring

Measures

• Intensive training

- refresher annual training (40 hours; 5-10 practice interviews)

- annual recertification required

- random observations of the evaluation process

• Task development process

- task developer’s network

- “to…from” feedback process

- field-testing of the newly created items

Conclusions

• Questions ???

• Thank you !