a locally appropriate english language test … locally appropriate english language test locality,...
TRANSCRIPT
A Locally Appropriate English Language Test Locality, Globality & Validity
Dr Jessica Wu(吳若蕙)
R&D Program Director
The Language Training & Testing Center (LTTC)
Taipei, Taiwan
Some definitions
• International tests: TOEFL, TOEIC, IELTS, Aptis, Cambridge English, etc.
• Locally-produced tests
• Which is more locally-appropriate? (Implicit assumption: Local tests
are more appropriate in assessing local learners because it is tailored to the specific educational systems and the contexts of test use)
• Test Localization (Adapting tests to the local context – presented by
O’Sullivan at 2014 New Directions)
“Localization is required when we are testing a well defined population within a well defined context in order to make decisions that will apply only to that context.” (O’Sullivan, 2013)
• Both local and international tests could serve as locally-appropriate tests.
2
NEAT/ TEPS
National English Ability Test/ Test of English Proficiency (Korea)
National Foreign Languages 2020 Project (Vietnam)
2020 Project
The Development of EFL tests in Asia
EIKEN - Test in Practical English Proficiency (Japan)
GEPT-General English Proficiency Test (Taiwan)
College English Test (China)
An Emerging Trend
3
English Education in Taiwan
An EFL context
• A compulsory subject starting Grade 1-3 up to University 1st year.
• To enhance the global competitiveness, English ability is demanded by schools and employers.
• Standardized English language tests are popular.
• Over one million test-takers per year take various kinds of standardized English language tests. (Taiwan population: 23 million)
4
Changing Landscape: Standardized English Language Tests
Test Description Test-takers in Taiwan/yr
GEPT A five-level EFL testing system designed to assess the general English proficiency of Taiwanese learners at all levels; developed by LTTC.
550,000
TOEIC Designed to measure the everyday English skills of people working in an international environment; developed by ETS.
300,000
IELTS Designed to assess the English language proficiency of people who want to study or work in an English speaking country; developed by British Council, Cambridge, IDP.
25,000
TOEFL Designed to measure the ability to use and understand English at the university level; developed by ETS.
23,000
5
Two major challenges
The GEPT is a locally-appropriate test? – Wishful thinking unless it can be supported by evidence through validation. Challenges
• Evidence supporting local relevance
• Evidence supporting international recognition
• Tension between locality and globality
6
Challenge 1
Locality: Reflecting macro and micro contextual features in test construction and implementation
• Culture
• Curriculum
7
Contextual mediation in test construction/revision
Educational and social features
Teaching & learning
objectives
(local curriculum)
Test construct,
content and format
WHY Communicative needs in social & educational systems
WHAT Knowledge, skills and abilities to be taught and learnt
HOW Test construction to positively impact teaching and learning
Impact by design (Saville, 2009; 2010)
8
Impact by design: Contextual mediation in the construction of the GEPT
Important features:
• Local curriculum
• Local cultural/social references
• Appropriate for local learners linguistically, visually, and conceptually (O’Sullivan, 2014)
• Increasing learners’ motivation and autonomy
• Helping learners to demonstrate language ability at their best
• Introducing positive washback
9
Measured Construct
“With an increased public expectation of transparent and explicit test, …., test providers need to satisfy the expectations of stakeholders (learners, employers, receiving institutions, professional bodies) concerning the comparability of the constructs measured by each test version in terms of both cognitive and contextual validity, and scoring validity.” (Weir, 2013: 3-4)
10
• Test content • Task requirement Evaluative criteria
11
Washback
The socio-cognitive framework for test development and validation (Weir, 2005; O’Sullivan, 2000; O’Sullivan & Weir, 2011)
Relationships with CEFR levels
12
GEPT Kids
GEPT HI
GEPT I
GEPT E
GEPT A
GEPT A
Grade 1-9 Curriculum Guidelines
National Curriculum Guidelines for Senior High Schools
Based on a survey of 8,000 GEPT test takers who are currently enrolled in colleges or universities in Taiwan
Benefit Learning and Teaching
• When you were in high school, did you take the GEPT for the purpose of applying to universities?
• Also, 60% reported that what the GEPT was testing was very similar to what they had learned in high school.
15
Increase the Globality of the GEPT
• Following international quality standards (e.g., AERA/APA/NCME Standards, ILTA Code of Ethics, ALTE Principles of Good Practice, ISO-9001)
• Aligning the GEPT to internationally-recognized frameworks, e.g., the CEFR
• Carrying out validation studies
• Promoting exchange with the global community through research, academic, and educational networks
16
Recognized by UK General Medical Council
• The UK GMC reviewed a number of English language tests in terms of their suitability for use within the Council’s registration process for internationally trained doctors wishing to practice in the UK.
• IELTS Equivalence Research Project (GMC 133) Final Report: six alternatives can be broadly predictive of overall IELTS scores.
– Cambridge English: Advanced, CAE
– Cambridge English: Proficiency , CPE
– GEPT Advanced
– TOEFL iBT
– Occupational English Test, OET
– Pearson Test of English – Academic , PTE-A
http://www.gmc-uk.org/about/research/28134.asp
17
International Recognition of the GEPT
100+ universities around the world have accepted GEPT scores to admit Taiwanese students to various types of programs.
18
Challenge 2
Globality: responding to the changing context
• Free market society
• New assessment needs
19
All in One: Total Solution
• TOEIC/TOEFL (ibt & ITP) for high school students
• IELTS/Cambridge English exams
Applications for study abroad
programs
University entry &
graduation benchmark
Certification
• Achievement
assessment • Assessing learning
outcome • Monitoring teaching
effectiveness 20
TOEIC growth
208,867 230,173
114,133 122,673
2014 2015
TOEIC Numbers
Students Non-students
21
1. Market needs 2. Commercial promotion and operation 3. Easier to achieve at a certain CEFR level 4. Recognized globally
+15%
Which Test to Use
‘…to understand the working of political, institutional, economic, and practical factors in effecting the development of a test.’ (Measured Words – Spolsky, 1995) The decision of which standardized English test to use is affected by
• personal preference
• stakeholders’ dynamic views towards the exams 22
Attitudes and perceptions of teachers and students towards
the GEPT and international standardized tests
• Perceived values
– Assessing English ability
– Reflecting learning outcome
– Resembling learning/working contexts
– Culturally friendly
• Factors
– Group (students vs teachers)
– Educational level (JH, SH, U/C, AL)
Are the US/UK-made English language tests preferred due to country image? Country image is viewed as a halo that consumers use to evaluate product
attributes (Han, 1989). - e.g. the country image for French wine vs Japanese
wine; German cars vs Chinese cars
23
GEPT
IELTS TOEFL iBT
TOEIC
Resembling Learning or Working Contexts
Junior and senior high school students consider the GEPT to be more relevant to their learning/working contexts.
24
Attitudes towards Exams
Overall, GEPT is well-known and the most cost-friendly exam in Taiwan, while international tests are considered to be of higher quality than the locally-produced one.
27
2
3
4
5
Brand namerecognition
Test qualityTest fee
GEPT IELTS TOEFL iBT TOEIC
Opportunity for international tests: Enhancing the usefulness of international tests in local contexts
How can the usefulness of international tests be enhanced in local contexts?
28
TOEFL/IELTS
(EAP-oriented)
TOEIC
(Workplace -oriented)
Support the internationalization of education in Taiwan; promote for use with
younger learners
Need to demonstrate:
Link to local context ( T/L in classroom;
content relevance to curriculum);
promoting learning (washback/impact)
International Cooperation Cases: Cambridge YLE Impact Study; British Council Aptis-GEPT Comparison Study Collaboration between test provider and local researchers – • Capture of relevant data • Deeper and more accurate understanding of the educational context • Fuller consideration of macro- and micro-level contextual parameters
Localization & Validation (O’Sullivan, 2014)
29
Demonstrating appropriate localization is a critical aspect of validation
Localization can only happen when we are aware of and take account of the test-taker in the context of the social domain
Localization is therefore a recognition of test consequence and social values as a critical aspect of the test development and validation process
Opportunity for locally-produced tests: ongoing validation efforts
• Promoting assessment literacy and communicating with
stakeholders. – Communication is essential for advancement (Fulcher, 2015; O’Sullivan, 2014).
• Increasing transparency; Being more open-minded (welcome and support external research)
• Cooperating with policy makers, teachers, test developers, and researchers – responding to the emerging needs for assessment
30
New assessment needs
• One-size-fits-all
• Understanding macro and micro contexts within the society where tests are being used
• Determining which elements facilitate/hinder the desired outcomes
Dynamical interplay between many sub-systems
(Saville, 2009 & 2012):
– social systems
– educational systems
– school systems
– individuals
31
Assessment in support of Internationalization of Higher Education
English is essential to the Internationalization of Higher Education.
• Graduation requirement
• EAP/ESAP courses
• EMI programs
Assessment should be better integrated with subject-specific teaching and learning – ESAP tests in business, engineering, medicine
32
Assessment in support of better learning – self assessment & autonomous learning
33
Assessments
Instruction Learning
Collecting information
Interpreting information
Making decisions about
Instruction or students
Using assessment as an instructional tool (involving students in assessment processes –
self-assessment)
34
Dr. Writing: Blended Learning
Student
Use the GEPT Online Marking System(OMS)
-Choose a writing task -Complete the task -Submit the essay
Teacher Use the GEPT OMS
-Mark the essay based on the GEPT criteria -Provide individualized feedback
Student Receive feedback via the GEPT OMS
-Rewrite [Submit a revised essay]
Teacher -Diagnose writing problems -Provide consultancy to facilitate learning
Online submission
Online instruction
Face-to-face instruction
Further instruction?
Teachers and learners dialogue around the assessment process and the criteria they apply to evaluate performance
Major types of errors
35
1. The six types of mistakes make up of over 93-96% of the errors.
2. Most of the errors were caused by the difference between Chinese, their first language, and English.
Types of tag
Word choice
Collocation errors
Incorrect use of words
Typos
Verb form Tense
Voice
Gerund/infinitive
Pluralising Incorrect use of plural or singular forms of nouns or verbs
Format errors
Wrong punctuation Lower-case/upper-case errors
Semantically unclear
Wrong prepositions
Final Note
Assessment is evidence-based practice.
International tests and locally-produced tests should consider both locality and globality as necessary components of validation rather than merely marketing claims.
36