as slippery as an eel? assessing speaking and writing part one
DESCRIPTION
As slippery as an eel? Assessing speaking and writing Part One. Ülle Türk University of Tartu Estonian Defence Forces 23rd CSW, Tampere, 27-29 March 2009. Testing writing?. Fill in the gaps with suitable words so that the text is true for you. Re-write the text in the future tense. - PowerPoint PPT PresentationTRANSCRIPT
As slippery as an eel?
Assessing speaking and writing Part One
Ülle TürkUniversity of Tartu
Estonian Defence Forces
23rd CSW, Tampere, 27-29 March 2009
2
Testing writing? Fill in the gaps with suitable words so that the text is true
for you. Re-write the text in the future tense. Fill in the form with the information given in the box. Read the letter and write an answer. Write an essay on the topic “Why study English?” Study the pictures, put them in the order you think best and
write the story. Read the text and write a short summary of it. You bought a new dictionary yesterday, but found later that
several pages were missing. Write a letter to the manager of the shop informing him of the problem and telling him what you want him to do about it.
Read the basic facts about Australian history and then write a short report.
3
Questions
What is it exactly that we assess when we say we assess students’ speaking and writing skills?
How do we arrive at a common understanding of what is ‘good’ writing, what is a ‘good’ oral presentation or what constitutes ‘good’ spoken or written communication?
4
Terms
Assessment Formal informal Continuous fixed-point Formative summative
Testing Achievement Proficiency Diagnostic Placement High-stakes low-stakes
5
Assessment/ test quality
Validity Reliability
Authenticity Washback Practicality
6
Validity: definitions
A good test needs to be valid. = It must test what it is meant to test.
an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores.
S. A. Messick (“Validity” in R. L. Linn (ed.) Educational Measurement. 1989, p. 13)
7
Validity
Does the test match the curriculum, or its specifications?
Is the test based adequately on a relevant and acceptable theory?
Does the test yield results similar to those from a test known to be valid for the same audience and purpose?
Does the test predict a learner’s future achievements?
8
Validity Content validity Construct validity Criterion-related validity Predictive validity
Construct validity is indeed the unifying concept that integrates criterion and content considerations into a common framework for testing rational hypotheses about theoretically relevant relationships.
Messick, S. A. “Test validity and the ethics of assessment.” American Psychologist 35, 1980, p. 1015
9
Threats to test validity
construct irrelevant variance construct under-representation
10
Factors affecting validity
Lack of specifications Lack of training of item/ test writers Lack of / unclear criteria for marking Lack of piloting/ pre-testing Lack of detailed analysis of items/ tasks Lack of feedback to candidates and
teachers
11
Communicative competence 1
Canale & Swain (1980), Bachman (1990): language knowledge types Linguistic knowledge Discourse knowledge Sociolinguistic knowledge
12
Grabe & Kaplan (1996): Model ofWriting
Components of language knowledge relevant to writing linguistic knowledge: written code,
morphology, vocabulary, syntax discourse knowledge: cohesion, structure,
genre sociolinguistic knowledge: functional uses of
writing, register, situational parameters Influential in teaching and testing of
writing (e.g., Weigle, 2002)
13
Communicative competence 2
Bachman & Palmer (1996): communicative language ability Language knowledge Strategic competence
14
Douglas (2000): Specific Purpose Language Ability Language knowledge
grammatical knowledge textual knowledge
rhetorical organization cohesion
functional knowledge sociolinguistic knowledge
Strategic competence assessment goal setting planning control of execution
Background knowledge discourse domain
15
Common European Framework of Reference for Languages: Learning, teaching, assessment (2001)
General competences Communicative language
competences
16
General competences
Declarative knowledge
Skills and know-how
‘Existential’ competence
Ability to learn
knowledge of the world
practical skills (social, living, vocational/ professional, leisure skills)
attitudes language and communicative awareness
motivationsociocultural knowledge
valuesbeliefscognitive styles
general phonetic awareness and skills
intercultural awareness
intercultural skills
personality factors
study skillsheuristic skills
17
Communicative language competences Linguistic competences
Sociolinguistic competences
Pragmatic competences
lexical competence
linguistic markers of social relations
discourse competence
grammatical competence
politeness conventions functional competence
semantic competence
expressions of folk wisdom
phonological competence
register differences
orthographic competence
dialect and accent
orthoepic competence
18
Communicative language activities and strategies
productive activities and strategies receptive activities and strategies interactive activities and strategies mediating activities and strategies non-verbal communication
practical actions paralinguistics paratextual features
19
Oral production public address (information, instructions, etc.) addressing audiences (speeches at public
meetings, university lectures, sermons, entertainment, sports commentaries, sales presentations, etc.) reading a written text aloud speaking from notes, or from a written text, or from
visual aids acting out a rehearsed role speaking spontaneously singing
20
Spoken interaction transactions; casual conversation; informal discussion; formal discussion, debate; interview, negotiation; co-planning; practical goal-oriented co-operation.
21
Oral mediation simultaneous interpretation (conferences,
meetings, formal speeches, etc.) consecutive interpretation (speeches of
welcome, guided tours, etc.) informal interpretation:
of foreign visitors in own country of native speakers when abroad in social and transactional situations for friends family, clients, foreign guests, etc. of signs, menus, notices, etc.
22
CEFR levelsThe Common European Framework of Reference
(Council of Europe 2001) defines communicative proficiency
At six levels, arranged in three bandsA1 A2
B1 B2C1 C2
in relation to six skills: listening, reading, spoken interaction, spoken production, written interaction, written production
in the form of “can do” statements
23
Getting to know the levels
The self-assessment grid is not enough More specific scales:
CEFR Ch 4: descriptors of communicative activities CEFR Ch 5: descriptors of linguistic competence
The ELP (European Language Portfolio) Manual: Relating Language Examinations to
the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR)
24
I can deal with most situations likely to arise whilst travelling in an area where the language is spoken. I can enter unprepared into conversation on topics that are familiar, of personal interest or pertinent to everyday life (e.g. family, hobbies, work, travel and current events).
Self-assessment grid (CEFR and standard adult passport)
25
CercleS ELP: goal-setting and self-assessment checklistsLevel B1 My
next goal
* * * * * *
I can readily handle conversations on most topics that are familiar or of personal interest, with generally appropriate use of register I can sustain an extended conversation or discussion but may sometimes need a little help in communicating my thoughts I can take part in routine formal discussion on familiar subjects in my academic or professional field if it is conducted in clearly articulated speech in standard dialect I can exchange, check and confirm factual information on familiar routine and non-routine matters within my field with some confidence I can express and respond to feelings and attitudes (e.g., surprise, happiness, sadness, interest, uncertainty, indifference) I can agree and disagree politely, exchange personal opinions, negotite decisions and ideas I can express my thoughts about abstract or cultural topics such as music or films, and give brief comments on the views of others I can explain why something is a problem, discuss what to do next, compare and contrast alternatives I can obtain detailed information, messages, instructions and explanations, and can ask for and follow detailed directions I can handle most practical tasks in everyday situations (e.g., making telephone enquiries, asking for a refund, negotiating purchase) I can provide concrete information required in an interview/ consultation (e.g., describe symptoms to a doctor), but with limited precision I can take some initiatives in an interview/ consultation (e.g., bring up a new subject) but am very dependent on the interviewer to provide support
26
Questions to ask What competences should my students have in
Spoken interaction Spoken production Written interaction Written production
What tasks should they be able to perform to demonstrate their mastery of the competences?
How well should they be able to perform them?
27
Reliability
A test needs to be reliable. = It must produce consistent results at different times.
NB! A test that is not reliable cannot, by definition, be valid.
28
Reliability If I take the test again tomorrow, will I get
the same result? If I take a different version of the test, will I
get the same result? If the test had had different items, would I
have got the same result? Do all markers agree on the mark I got? If the same marker marks my test paper
again tomorrow, will I get the same result?
29
Factors affecting reliability
Poor administration conditions – noise, lighting, cheating
Lack of information beforehand Lack of specifications Lack of marker training Lack of standardisation Lack of monitoring
30
References Bachman, Lyle F. (1990) Fundamental
Considerations in Language Testing, Oxford University Press, Oxford.
Bachman, Lyle F. and Palmer, Adrian (1996) Language Testing in Practice, Oxford University Press, Oxford.
Cushing Weigle, Sara (2002) Assessing Writing, Cambridge University Press, Cambridge.
Douglas, Dan (2000) Assessing Languages for Specific Purposes, Cambridge University Press, Cambridge.