By Dongxue Qin and Moriah Kent

SPEAKING ASSESSMENT

Trading Questions!•Dongxue and Moriah model the task•Students choose questions and practice individually•Play the game!

Language Area: Speaking• What kind of skill is speaking?

- Production skill

- Differs from writing in what ways?

Writing Speaking

Full, complex sentences, well-organized

Incomplete, simple and loose organization of sentences and thoughts

Information densely packed Simple discourse with less information

Specific vocabulary General vocabulary

Use of discourse markers Frequent use of fillers to facilitate speech

Text written for an unseen audience

Face-to-face communication

A relatively solitary process Negotiation between 2 or more people

Alterations and crossings out kept to minimum

Alternations, corrections, miscues are very common

Reference can easily be made to what was previously written

Memory limitations are important as speech is transitory

Source: Jones, W. (2005). Assessing students’ oral proficiency. In D. Lloyd, P. Davidson, & C. Coombe (Eds.), The Fundamentals of language assessment: A practical guide for teachers in the Gulf (pp.75-86) Dubai: TESOL Arabia Publications, p. 77.

Language Area

Language Area: Why? •Why do we assess speaking?•How do we assess speaking? •What standards should we hold second-language speakers to?

Language Area: Overarching Constructs

•Fluency- Include smoothness, rate of speech, absences of excessive pausing/hesitation, length of utterance and connection, and repetition-Self-corrections can be evaluated by both machines and humans. •Accuracy-Linguistic correctness, grammatical knowledge and ability

Language area: Speaking Competencies

•Grammatical competence•Discourse competence•Sociolinguistic competence•Strategic competence

Language area: Sub-constructs within Speaking•Pronunciation•Pitch, volume, speed, pausing, stress, intonation, rhythm, etc•Comprehensibility/Communicative competence/Fluency •Spoken grammar (Accuracy)•Formality/informality •Vocabulary

Research- Interlanguage (Selinker, 1972 in Folse, p.32)-Before the 1980s, classroom goals were focused on language knowledge, rather than communicative use of a language. (Folse, p.34)

-Krashen’s i+1 in terms of speaking-Swain’s (1985, 1995, 2005) “pushed output” for speakers, “When they are unable to communicate effectively, they must rethink their utterances and modify the relevant parts” (Folse, p. 42).

-Both comprehensible input and pushed output are important for speaking tasks.

Task Creation: Types• Speaking tests can be live or recorded

- TOEFL iBT = recorded- IELTS = live and recorded

•Open-ended or structured/restricted - Open: Give a lecture, presentation, have reciprocal conversation/debate or discussion - Structure: Answer direct questions, expected right/wrong answer• Individual, pair, group tasks - Individual = interview - Pair = role play - Group = group discussions or debates• Construct-based or task-based assessments - Construct: Language ability - Task: Can you use the language in a given domain? •Majority of tasks for assessment are integrated

Task Creation: ProcedurePre-creation: • What’s the focus? Fluency? Accuracy? Or both? • What are your criteria? • How will you score? • Formal or informal language? • Individual, group or pair work? • How much time? • What channel with the input come in? Listening? Visual?

Written? Or a combination?

During creation: • Choose a topic/task • Outline subtopics that learners can discuss as well as

vocabulary they might need to use• Is it a one-way or two-way task? • Is it timed? • Is it open-ended (divergent) or restricted (convergent)?

Task Creation: Considerations • Things to consider when creating a speaking assessment task

- Practicality

- What is your student’s personality like? • Possible considerations with administering speaking assessments

- One-on-one?

- More than one rate? • Inter-rater reliability

- Decide on the criteria/constructs

- Two raters: one to interact with the S and one to evaluate

- Use a score sheet

- What kind of speaking sample are you trying to elicit? (A speech with a purpose? Chunks of speech? Controlled speech? Specific points, etc?

Scoring: Issues•Issues with developing speaking rubrics:

1. Analytic or holistic or combo?2. What constructs are you assessing in

speaking? And what criteria corresponds to a score?

3. What should you say? Scales need to be concrete and clear, practical and avoid vagueness.

4. Norm-referenced or criterion-referenced?

Scoring: How to develop? •Methods of developing scales/rubrics:

1. Intuitive 2. Qualitative3. Quantitative•Ensuring Reliability and validityReliability-Validity-Scoring example scales…

ACTFL speaking scale

- 10 levels, with four sublevels (superior, advanced, intermediate and novice), the last three are further divided into 3 sublevels (high, mid, low)

- Holistic scale focused on what learners can do, not what they know

- Focuses on beginning and intermediate

- Novice Low: “Speakers at the novice-low level have no real functional ability and because of their pronunciation, they may be unintelligible. Given adequate time and familiar cues, they may be able to exchange greetings, give their identity and name a number of familiar objects from their immediate environment. They are unable to perform functions or handle topics pertaining to the intermediate level and cannot, therefore, participate in a true conversational exchange”. (Luoma, p.67)

TOEFL iBT Integrated Speaking Rubric (part)

TOEFL Activity

• Read the passage from a psychology textbook below and listen to the lecture that follows it. Then answer the question. You will have 45-50 seconds to read the text.

• FLOW• In psychology, the feeling of complete and energized focus in an activity is called flow. People who enter a state of flow lose their sense of time and have a feeling of great satisfaction. They become completely involved in an activity for its own sake rather than for what may result from the activity, such as money or prestige. Contrary to expectation, flow usually happens not during relaxing moments of leisure and entertainment, but when we are actively involved in a difficult enterprise, in a task that stretches our mental or physical abilities.

TOEFL Activity continued…• Now, listen to the following lecture, which is about this topic. You can take notes. You will hear it one time, only.

• Question: Explain flow and how the example used by the professor illustrates the concept.

• You will have 30 seconds to prepare. • STOP! • You will now have 60 seconds to respond. • STOP!

TOEFL iBT Integrated Speaking Rubric (part)

INTO CSU AEIN100 Speaking Rubric 

5 - Advanced 4 - Satisfactory 3 – Needs Improvement2 and Below – Does Not

Meet Expectations

 Content andVocabulary

 (Specific ideas and vocabulary from

the text)

 Specific vocabulary from the

curriculum is used accurately. All ideas are clear and appropriate for

the context.

 General vocabulary is used

correctly. All meaning is clear.

 General vocabulary is used, though

not always correctly. Some meaning is unclear.

 Little vocabulary is used correctly.

Most meaning is unclear.

 Language

 (e.g. verb tense (present and past),

subject-verb agreement, word order)

 All verbs are used correctly. Word

order is consistently correct.

 Most verbs are used correctly.

Word order is generally correct.

 Verbs are not used correctly. Word order is inconsistent and incorrect.

 Verbs are missing or incorrect.

Word order follows no structure.

 Pronunciation

 (e.g. individual sounds, stress,

intonation)

 Pronunciation of individual sounds

is clear and accurate. Very little interference from the first language is present, and the student can be

easily understood.

 Almost all sounds are pronounced

clearly and accurately. Little interference from the first language may be present, and the student can

be understood.

 Some sounds are not pronounced

clearly.The student can generally be

understood, but interference from the first language is present and

causes confusion.

 Many sounds are pronounced

incorrectly. Heavy interference from the first language disrupts

understanding.

 Fluency and Communicative

Ability 

(e.g. pace, flow, on topic, ease of speaking)

 Speaking is conversational, flowing smoothly and logically. Meaning is complete and clear; there is no need

for follow up questions from the instructor.

 Speaking generally flows smoothly, and there is very little hesitation. Meaning is communicated, but the

instructor may ask for clarification.

 Speaking is slow and halting.

Meaning is unclear, so the instructor must ask for clarification.

 Speaking is very choppy. Little or no meaning is communicated. The instructor guides the conversation

with questions.

 

     

Total: /20

INTO CSU Speaking Rubric

- 4 levels- Holistic scale (giving one score)-Used to rate low-intermediate learners:Communicate clearly about concrete topics related to living in an American university community using American English pronunciation

Thank you for your time!

References:

Brown, J.D. (Eds.). (2013) New ways of classroom assessment. Alexandria, Virginia: TESOL International Association.

Coombe, C., Folse, K. & Hubley, N. (2010). A practical guide to assessing English language learners. Ann Arbor, Michigan: The

University of Michigan Press.

Folse, K.S. (2009). The art of teaching speaking: Research and pedagogy for

the ESL/EFL classroom. Ann Arbor, Michigan: The University of

Michigan Press.

Jones, W. (2005). Assessing students’ oral proficiency. In Lloyd, D.,

Davidson, P., & Coombe, C. (Eds.), The fundamentals of language

assessment: a practical guide for teachers in the Gulf (pp. 75-86).

Dubai: TESOL Arabic Publication.

Luoma, S. (2004). Assessing Speaking. Cambridge, UK: Cambridge

University Press.

ETS. (2014). TOEFL iBT speaking rubric.

Retrieved from: www.ets.org/toefl