workshop t7: do indonesians make mobile calls and do russians go on holiday?
DESCRIPTION
Do Indonesians make mobile calls and do Russians go on holiday? Cultural determination in tests. By academic staff of the University of Amsterdam and the Catholic University of LeuvenTRANSCRIPT
DO INDONESIANS MAKE MOBILE CALLS AND DO RUSSIANS
GO ON HOLIDAY?
NHTV Breda – january 2008
Katholieke Universiteit Leuven
Universiteit van Amsterdam
E.Teijsse-Jongkind @uva.nl
The effect of culture on testingThe Certificate Dutch as a Foreign Language
Content of this presentationContent of this presentation- Purpose of this presentation: to be aware
of cultural bias in the classroom
- Background of the CNaVT (Certificaat Nederlands als Vreemde Taal)
- Definition of cultural bias and the effect of culture on different levels
- Task: Does this test contain cultural bias?
- How does the CNaVT manage the effect of culture in the CNaVT-exams?
- Discussion: own experiences with cultural bias
The Certificate of Dutch as a The Certificate of Dutch as a Foreign LanguageForeign Language
- CNaVT (Certificaat Nederlands als Vreemde Taal)
- administration in more than 40 countries around the world: from Madrid to Jakarta and from Warsaw to Rio de Janeiro
- 2500 candidates each year
Assess as direct as possibleAssess as direct as possible
- Based on a needs analysis
- authenticity: 6 exams represent 6 different domains
- task-based: exam tasks represent real life language use situations
The effect of culture: cultural biasThe effect of culture: cultural bias
Definition of cultural bias:Scores of groups are distorted, because the items favour some groups over others. This distortion is not caused by a difference in (language) proficiency.
THE EFFECT OF CULTURE ON THE EFFECT OF CULTURE ON DIFFERENT LEVELSDIFFERENT LEVELS(Van Baalen, Beheydt & van Kalsbeek, 2003)
On the level of:1. Words and expressions2. Images3. Subjects and concepts4. Knowledge of the world5. Relation examiner - candidate6. Classroom / test situation
1. WORDS and EXPRESSIONS1. WORDS and EXPRESSIONS
verpotteren
flitspaalgedrag
speeddaten
(zit)bank vs. zetel
lopen vs. stappen
namiddag
iets afkloppen/ hout vasthouden
2. IMAGES2. IMAGES
3. SUBJECTS3. SUBJECTS
4. KNOWLEDGE OF THE WORLD4. KNOWLEDGE OF THE WORLDexample 1example 1
•You want to organize a party for your birthday.
•Describe what you want to your friend.
4. KNOWLEDGE OF THE WORLD4. KNOWLEDGE OF THE WORLDexample 2example 2
vs.
5. RELATION EXAMINER-STUDENT5. RELATION EXAMINER-STUDENT
6. CLASSROOM PRACTICE/ TEST 6. CLASSROOM PRACTICE/ TEST SITUATIONSITUATION
MINI TESTMINI TEST
• Read the test and answer the following question.
• Does this test contain cultural bias for some groups of students? Which levels of cultural bias can appear? Why?
A TENSIONA TENSION
Validity in the sense that we want our exams to measure language proficiency.
Reliability in the sense that we want to be able to state that our exams are not influenced by other factors like the culture of the candidate.
DIF
Differential Item Functioning
= the statistical view of cultural bias
DIF ANALYSISDIF ANALYSIS
CNaVT item bias researchCNaVT item bias research
Example 1
Country: Indonesia
Item: Question in form
‘What kind of object have you lost?’
CNaVT item bias researchCNaVT item bias research
Example 2
Country: Russia
Item: Question from friend in letter
(…) ‘But I often think back with nostalgia about my holiday in your country. (…) And you? What did you do during your holidays? Did you travel? Where to? Or did you stay at home and what did you do?’ (…)
How does the CNaVT How does the CNaVT manage the effect of culture?manage the effect of culture?
- awareness of what we test: what (concerning culture) is relevant, what is irrelevant
- generate input on possible cultural bias from different sources in different stadia
- In case of suspicion of irrelevance: DIF analysis
BUT… CNaVT is a test of Dutch as a BUT… CNaVT is a test of Dutch as a foreign language! foreign language! Difference between test situation and the classroom
Cultural bias is not something that has to be avoided in the classroom
But: teachers should use cultural miscommunications to learn from it
Intercultural competence!
Teacher and test developer: recognize and be aware of the effect of culture
Tips: How to make students Tips: How to make students interculturally competent? interculturally competent? Three important principles:
Perspective
Context
Dialogue
DISCUSSIE / VRAGENDISCUSSIE / VRAGEN
What is your experience with cultural bias in your own classroom/test practice? Can you give an example? How do you manage?
LITERATUURVERWIJZINGENLITERATUURVERWIJZINGEN
Van Baalen, C., Beheydt, L. & Van Kalsbeek, A. (2003). Cultuur in taal. Interculturele vaardigheden voor docenten Nederlands aan anderstaligen. Utrecht: Nederlands centrum buitenlanders, 167p.
DO INDONESIAN MAKE MOBILE CALLS AND DO RUSSIAN
GO ON HOLIDAY?
NHTV Breda – januar 2008
Katholieke Universiteit Leuven
Universiteit van Amsterdam
E.Teijsse-Jongkind @uva.nl
Language testing and cultureThe certificate of Dutch as a foreign language