Slide 1

1 Vocabulary Assessment Norbert Schmitt University of Nottingham Slide 2 2 Vocabulary Assessment Nearly all teachers do vocabulary assessment of some sort, ranging from informal observation, to short quizzes, to more formal examinations While informal assessment may not be difficult, designing good vocabulary measures for higher stakes purposes requires a considerable amount of expertise Most teachers (and educators and researchers in general!) lack this expertise Slide 3 3 Vocabulary Assessment Ive been thinking about vocabulary measurement since the early 1990s. Here are 4 questions on test development which I came up with in 1994 (Thai TESOL Bulletin). Slide 4 4 Vocabulary Assessment 1. WHY DO YOU WANT TO TEST? 2.WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) 3.WHAT ASPECTS OF THESE WORDS DO YOU WANT TO TEST? 4.HOW WILL YOU ELICIT STUDENTS' KNOWLEDGE OF THESE WORDS? Slide 5 5 Vocabulary Assessment 1.WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) Slide 6 6 Vocabulary Assessment 1.WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) To see if students have vocabulary gaps (diagnostic) Slide 7 7 Vocabulary Assessment 1. WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) To see if students have vocabulary gaps (diagnostic) Placement Slide 8 8 Vocabulary Assessment 1.WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) To see if students have vocabulary gaps (diagnostic) Placement Part of a proficiency test Slide 9 9 Vocabulary Assessment 1.WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) To see if students have vocabulary gaps (diagnostic) Placement Part of a proficiency test Motivation Slide 10 10 Vocabulary Assessment 1.WHY DO YOU WANT TO TEST? To see if students have learned taught words (achievement) To see if students have vocabulary gaps (diagnostic) Placement Part of a proficiency test Motivation Washback (tests reflect educator goals) Slide 11 11 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) It depends on the purpose of the test Slide 12 12 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Achievement = ? Slide 13 13 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Achievement = lexical items that have been taught Slide 14 14 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Diagnostic = ? Slide 15 15 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Diagnostic = The lexical items a student is expected to know, or should know at a certain level Slide 16 16 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Placement = ? Slide 17 17 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Placement = The lexical items that will be taught in a course, or that a student may know at the level being taught in the course. Also the foundation vocabulary expected to be learned before entering the course. Slide 18 18 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Proficiency = ? Slide 19 19 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Proficiency = A range of vocabulary, especially some that will be challenging for the best students Slide 20 20 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Motivation = ? Slide 21 21 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Motivation = Lexical items that were recently taught, or the items that the students see as useful for reaching their goals (e.g. TOEFL, university entrance exam) (or any vocabulary : testing always makes students study?) Slide 22 22 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Washback = ? Slide 23 23 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) Washback = any vocabulary, as the act of putting vocabulary on a test shows that it is important Is a way of highlighting education goals Slide 24 24 Vocabulary Assessment 2. WHAT WORDS DO YOU WANT TO TEST? (AND HOW MANY?) It depends How long should the test be? (low/high stakes) Longer is better, but it must be a practical length What sampling rate will you accept? Slide 25 25 Vocabulary Assessment Sampling Rate You typically cannot test every lexical item So you need to extract a representative sample Depends on item format: checklist format allows more items than multiple-choice 1/5, 1/10, 1/100, 1/1,000? Many vocabulary tests have very low sampling rates (e.g. VLT is only 3/100) Slide 26 26 Vocabulary Assessment How to Sample? Random Systematically: every n th item, every n th page, etc. Equal proportions of different word classes (nouns, verbs, etc.) Only the most difficult (least frequent?) items, on the assumption that these are the items which will not be known) Slide 27 27 Vocabulary Assessment 3. WHAT ASPECTS OF THESE WORDS DO YOU WANT TO TEST? Which word knowledge aspects will you cover? Form-meaning link is the minimum specification It is also the typical specification (Why do you think this is so?) Slide 28 28 Vocabulary Assessment 4.HOW WILL YOU ELICIT STUDENTS KNOWLEDGE OF THESE WORDS? Which item format will you use? Slide 29 29 Item Formats Lets look at a number of item formats What word knowledge aspects do they address? Are they receptive or productive? Are they size or depth tests? What are their advantages and disadvantages? For what testing purposes might they most useful? Least useful? Slide 30 30 Slide 31 31 Slide 32 32 Slide 33 33 Slide 34 34 Slide 35 35 Slide 36 36 Slide 37 37 Slide 38 38 Slide 39 39 Slide 40 40 Slide 41 41 Slide 42 42 Slide 43 43 Slide 44 44 Slide 45 45 Size & Depth Test Formats Next, lets look at a number (semi-) established test formats: Vocabulary Size test formats Multiple-choice formats Vocabulary Levels Test Vocabulary Depth Formats Developmental Scales Vocabulary Knowledge Scale Schmitt and Zimmerman Scale Word Associates Format Slide 46 46 Slide 47 47 Checklist (Yes-No) Tests Checklist tests are straightforward to take Learners just check ( ) which words they think they know Here is a checklist test from one of the best known studies into the vocabulary size of native English speakers (NZ university students) Slide 48 48 Slide 49 49 Checklist (Yes-No) Tests Checklist tests are an efficient way of testing a lot of lexical items This allows to a high sample rate Easy to build and easy to mark But learners sometimes overestimate their knowledge (i.e. they check words they dont actually know) How to control for this? Mearas 1992 Checklist Tests Slide 50 50 Slide 51 51 Checklist (Yes-No) Tests The most common way is to add nonwords to the test, and see if they check them as known If so, then their scores are adjusted down Mearas adjustment table However, the adjustment formulas are all a bit wonky In some research, data is deleted if a certain number of nonwords are checked as known In the end, checklist tests dont work very well if examinees are not honest and careful So the usefulness of the test format depends on the examinees behavior to a large extent Slide 52 52 Slide 53 53 Adjusting Checklist (Yes-No) Tests Reaction Time (speed of response) is a viable way of adjusting accuracy Faster responses are usually more sure Pellicer-Sanchez and Schmitt (2012) Language Testing Best adjustment formula by individual result and False Alarm rate FA rate Best adjustment formula NS NNS 0 RT RT 1H FA > RT RT = m 2 H FA = RT H FA 3 H FA 4 H FA 8 Isdt> H FA Slide 54 54 Vocabulary Knowledge Scale Often used as a depth test Is a developmental type of measurement But there are many problems with this scale: See Researching Vocabulary for a full critique: How many stages should scale have? Not an interval scale Cant use inferential statistics with it Sentences often not informative Not clear what VKS is measuring Slide 55 55 Slide 56 56 Schmitt & Zimmerman Scale Suffers from many of the same problems as VKS Fewer stages make it more transparent? Written in a can-do manner: easier for learners to say what they can do than what they know More closely connected to receptive vs. productive mastery Tests uses non-words (artivious, ploat) to assure honesty of response Which is better? Slide 57 57 Slide 58 58 Word Associates Format (Read, 2000) One of the most used depth test formats Comes in 8-word and 6-word versions, some with boxes and some with words in lists Learners circle all of the words which are associated with the target word Left box is meaning-based Right box has collocations Ratio of answers per box can vary to make guessing more difficult Slide 59 59 Slide 60 60 Word Associates Format (Read, 2000) If learner correctly selects all correct associations and none of the distractors, then this shows good knowledge of the target word If learner selects none of the correct options, and this indicates little or no knowledge of word But what about split answers: some correct options and some incorrect ones? MA research at Nottingham (Schmitt, Ng, & Garras, 2011) shows that this actually corresponds to little real knowledge of the words That is, split scores do not indicate reliable knowledge Slide 61 61 Vocabulary Website Most Schmitt (and colleagues) research is available on Norbert Schmitts personal website: www.norbertschmitt.co.uk There are also vocabulary resources, including vocabulary tests on the site