development and validation of additional language oral ... · cantonese, korean, and vietnamese...

©2013, PSI Services LLC. All rights reserved.

Development and Validation of Oral

Examinations for Medical Interpreter

Certification:

Mandarin, Russian, Cantonese, Korean,

and Vietnamese Forms

Technical Report

November 2013

PSI Services, LLC

www.psionline.com

©2013, PSI Services LLC. All rights reserved. i

Table of Contents

Table of Contents ______________________________________________________________ i

List of Tables ________________________________________________________________ iii

Acknowledgements ____________________________________________________________ 1

Executive Summary ___________________________________________________________ 2

Test Design ______________________________________________________________________ 2

Item Development _________________________________________________________________ 2

Pilot Administration _______________________________________________________________ 2

Test Form Construction ____________________________________________________________ 2

Standard Setting __________________________________________________________________ 3

Introduction _________________________________________________________________ 4

1. Item Development __________________________________________________________ 5

Item Writing Workshops ___________________________________________________________ 5

Initial Exam Development Plan ______________________________________________________ 5

Oral Exam Item Development _______________________________________________________ 5

2. Item Review and Evaluation __________________________________________________ 5

3. Pilot Testing _______________________________________________________________ 8

Experimental Test Forms __________________________________________________________ 8

Rater Selection, Training, and Calibration ____________________________________________ 8

Rating Protocol __________________________________________________________________ 10 Scoring Dictionary ______________________________________________________________________ 11

Pilot Administration ______________________________________________________________ 11

Results _________________________________________________________________________ 12

4. Test Form Assembly ________________________________________________________ 15

5. Recommended Minimum Passing Scores _______________________________________ 17

Russian _________________________________________________________________________ 18

Mandarin _______________________________________________________________________ 18

Cantonese ______________________________________________________________________ 18

Korean _________________________________________________________________________ 18

Vietnamese _____________________________________________________________________ 18

Ongoing Review Process ______________________________________________________ 19

Rater Calibration ________________________________________________________________ 19

Exam Review ____________________________________________________________________ 19

Summary and Conclusion _____________________________________________________ 20

References _________________________________________________________________ 21

©2013, PSI Services LLC. All rights reserved. ii

Appendix A. Oral Examination Expert Advisory Committee _________________________ A-1

Appendix B. Scoring Rubric __________________________________________________ B-1

Appendix C. New Form Item Statistics __________________________________________ C-1

Appendix D. Angoff Ratings __________________________________________________ D-1

©2013, PSI Services LLC. All rights reserved. iii

List of Tables

Table 1: Oral Exam Pilot Form Descriptive Statistics ________________________________________________ 14 Table 2: Statistical Properties of Final Oral Examination Forms for Five Languages _______________________ 16 Table A-1. Oral Examination Expert Advisory Committee ____________________________________________ A-1 Table C-1. Russian New Form A Item Statistics ____________________________________________________ C-1 Table C-2. Russian New Form B Item Statistics ____________________________________________________ C-8 Table C-3. Russian New Form C Item Statistics ___________________________________________________ C-15 Table C-4. Mandarin New Form A Item Statistics _________________________________________________ C-22 Table C-5. Mandarin New Form B Item Statistics _________________________________________________ C-29 Table C-6. Mandarin New Form C Item Statistics _________________________________________________ C-36 Table C-7. Cantonese New Form A Item Statistics _________________________________________________ C-43 Table C-8. Cantonese New Form B Item Statistics _________________________________________________ C-49 Table C-9. Cantonese New Form C Item Statistics ________________________________________________ C-55 Table C-10. Korean New Form A Item Statistics __________________________________________________ C-61 Table C-11. Korean New Form B Item Statistics __________________________________________________ C-67 Table C-12. Korean New Form C Item Statistics __________________________________________________ C-73 Table C-13. Vietnamese New Form A Item Statistics _______________________________________________ C-79 Table C-14. Vietnamese New Form B Item Statistics _______________________________________________ C-85 Table C-15. Vietnamese New Form C Item Statistics _______________________________________________ C-91 Table D-1. Russian New Form Sight Test Angoff Ratings ___________________________________________ D-1 Table D-2. Russian New Form Consecutive Test Angoff Ratings ______________________________________ D-3 Table D-3. Mandarin New Form Sight Test Angoff Ratings _________________________________________ D-13 Table D-4. Mandarin New Form Consecutive Test Angoff Ratings ___________________________________ D-16 Table D-5. Cantonese New Form Sight Test Angoff Ratings ________________________________________ D-26 Table D-6. Cantonese New Form Consecutive Test Angoff Ratings __________________________________ D-29 Table D-7. Korean New Form Sight Test Angoff Ratings ___________________________________________ D-39 Table D-8. Korean New Form Consecutive Test Angoff Ratings _____________________________________ D-41 Table D-9. Vietnamese New Form Sight Test Angoff Ratings _______________________________________ D-51 Table D-10. Vietnamese New Form Consecutive Test Angoff Ratings _________________________________ D-53

©2013, PSI Services LLC. All rights reserved. 1

Acknowledgements

This project was completed with the cooperation and support of numerous medical

interpreters throughout the United States. PSI Services LLC, Language Line Academy, and The

National Board extend our appreciation to the many individuals who completed job analysis surveys

and participated in assessment sessions. In addition, special thanks are extended to the State of

Oregon Health Authority, Office of Equity and Inclusion, for their crucial role in the funding of this

project through a grant received by The National Board.

Language Line Academy staff, Danyune Geertsen, Director, Language Line Academy and Janet

Erickson-Johnson, Language Line Academy Director of Interpreter Certification, were instrumental

in the completion of the development of the oral examination. The successful completion of this

project would not have been possible without their continued effort and support.

PSI project staff included John Weiner, Project Director, and Monica Elcott, Project Consultant.


Executive Summary

This report describes the methodology and results of a project conducted jointly by Language Line

Academy (LLU), an affiliate of Language Line Services, The National Board, and PSI Services LLC

(PSI) to construct a series of oral examinations for medical interpreter certification in the languages

of Mandarin, Russian, Cantonese, Korean, and Vietnamese. The test development methodology was

modeled after a previous research and development project conducted for a Spanish language oral

examination (Development and Validation of an Oral Examination for Medical Interpreter

Certification: Technical Report, PSI, February 2010).

The present report documents the process of exam development, pilot testing and psychometric

analysis and assembly of equivalent alternate forms of each language exam. The report describes

psychometric characteristics of the resulting examinations, along with future analyses planned after

the examinations are implemented to ensure acceptable measurement quality. The test development

and validation methodology was designed to comport with the Standards for Educational and

Psychological Testing (AERA, APA, NCME, 1999), as well as the Standards for the Accreditation of

Certification Programs (National Commission for Certifying Agencies, 2010).

Test Design

Specifications for the examinations were founded upon a national job analysis of the medical

interpreter profession (PSI, Feb. 2010). The exam content outlines specified topics, subtopics and

their relative weights in constructing the examinations.

Item Development

PSI conducted intensive item writing and standard setting training sessions with LLU staff. Each test

design subject matter expert received the training according to the specifications and guidelines

provided by PSI. Test item development for the five language exams occurred between March and

May 2011 and entailed extensive review, discussions, and revisions. In addition, a scoring dictionary

was developed for rating purposes as part of this process.

Pilot Administration

Two experimental versions were assembled for each of the Russian, Mandarin, Cantonese, Korean,

and Vietnamese oral examinations. These forms were pilot tested in a series of experimental test

administrations that took place between June 2011 and April 2013 The pilot samples ranged in size

from 30 to 32 experienced medical interpreters per language exam.

Evaluator Training and Calibration

The oral examinations are reviewed and scored by expert interpreters in each language, who received

extensive training and calibration in using the scoring rubric prior to scoring the pilot examinations.

Test Form Construction

Statistical analyses were conducted on the pilot test data to ensure that the items demonstrated

acceptable psychometric properties and to assemble equivalent alternate exam forms. Final forms of

the new oral exams were assembled based on the examination content specifications outlined in the

aforementioned technical report (PSI, 2010), as well as the statistical analysis from the pilot test data,

resulting in three equivalent alternate forms for each language.


Standard Setting

Recommended minimum passing scores (cut scores) were established through a standard setting

study for each of the oral exams. Subject matter experts rated each item using a modified Angoff

procedure. PSI staff then conducted analyses of the Angoff ratings and pilot data identify a

recommended cut score for each language exams, designed to ensure safe and competent practice as a

medical interpreter.

Conclusion

Overall, this report presents substantive evidence supporting the psychometric quality and validity of

the Mandarin, Russian, Cantonese, Korean, and Vietnamese oral examinations for medical interpreter

certification. The exams are founded upon a nationally representative occupational analysis survey,

subject matter expert-developed test items and linkage to the job analysis, pilot testing and statistical

analysis, and empirically-based test form assembly.


Introduction

This document describes the methodology used to develop and validate a series of oral

examinations for medical interpreter certification in the Russian, Mandarin, Cantonese, Korean,

and Vietnamese languages. The overarching aim was to ensure that the exams measure the most

current knowledge and skills considered essential to a certified medical interpreter’s performance

of services in a manner that ensures patient safety. The test development methodology was

modeled after a previous research and development project conducted for a Spanish language

oral examination (Development and Validation of an Oral Examination for Medical Interpreter

Certification: Technical Report, PSI, February 2010).

The current exam development initiative took place between June 2011 and October 2013, and

consisted of several key steps, including: (1) test item development in accordance with exam

specifications based on prior job analysis; (2) item review and evaluation; (3) pilot testing; (4)

standard setting; and (5) test form assembly. Each step of the test development and validation

process is described in this technical report. These steps combine to provide evidence of

psychometric quality and validity, helping to ensure that those who pass the national certification

oral examination are qualified to practice in a manner that protects patient safety.

Highlights of the present report include the following:

A technical overview of the procedures followed to review the medical interpreter oral

examination item banks and assemble test forms in alignment with the previously-defined

content and statistical specifications.

A summary of the technical properties of the medical interpreter oral examinations, including

the recommended minimum passing scores.


1. Item Development

Item Writing Workshops

PSI conducted comprehensive training sessions on item writing and standard setting procedures

(modified Angoff) with LLU staff Janet Erickson-Johnson and Danyune Geertsen at PSI

headquarters in January 2009. During the training, PSI psychometric staff provided various

supporting materials, such as rating forms, procedural checklists, and PowerPoint presentations,

so that LLU employees could conduct oral exam item development and review sessions with

subject matter experts.

Further details regarding the item writing training session, participants, and procedures are given

in separate technical reports for the written and oral exams (PSI, April 2010, Development and

Validation of Oral and Written Examinations for Medical Interpreter Certification: Technical

Report), and in a supplemental standard setting report (PSI, June 2012, Addendum: Development

and Validation of Oral and Written Examinations for Medical Interpreter Certification:

Standard Setting Technical Report).

Initial Exam Development Plan

In consideration of the volume of examinations administered annually for the Russian, Mandarin,

Cantonese, Korean, and Vietnamese languages (ranging from 50 to 100 oral exams in each

language), it was determined that three (3) alternate test forms would be sufficient to develop for

each language exam to maintain a reasonable degree of item exposure over a 4- or 5-year period.

Oral Exam Item Development

LLU assembled teams of subject matter experts for each language, selecting individuals with

extensive experience in the fields of medical interpreting, medical practice, and test design and

development (see Appendix A). The subject matter experts received intensive training according

to the specifications and guidelines provided by PSI. The training sessions covered PSI’s Item

Reviewing and Writing Principles and modified Angoff rating process.

The oral test design team completed the development of the test items in a sufficient quantity to

be representative of a broad range of medical knowledge and to exceed the number of test items

needed to create two distinct test forms. This process involved intensive work and review.

A standard scoring rubric was used across each of the language exams. The scoring rubric was

developed in the aforementioned previous exam development project (PSI, February 2010).

2. Item Review and Evaluation

LLU staff convened separate panels of subject matter experts to review and formally evaluate the

items for each oral exam language. The expert panels evaluated each item with respect to

relevance to occupational requirements, the need for competence upon certification and

estimated difficulty for initial certification candidates.

The panelists’ qualifications appear in Appendix A. In the table within the appendix, there are

columns indicating in which phase of test development they participated in (i.e., item writing,

item review, standard setting). The participants were selected primarily on the basis of

experience and involvement in the industry, as well as familiarity with test administration.


The item review process included training sessions for the subject matter experts, covering PSI’s

Item Reviewing and Writing Principles and modified Angoff rating process. After training, the

advisory panel, members reviewed and formally rated the test items. The panel members

provided Angoff ratings twice – first without the answer key, and then again after having seen

the answer key.

As part of training, the expert panel was calibrated on how to define a “Minimally Acceptable

Candidate” at the time of certification on which to base their item review ratings, using the

following standard:

An individual who possesses the minimum knowledge, skills and ability to practice

without endangering the public

NOT an individual who is unacceptable

NOT all test-takers, since some are unacceptable (those who fail) and some are

acceptable (those who pass)

NOT an individual you would hire to work for you, since he or she is typically beyond

minimally acceptable

NOT all acceptable candidates, since some are minimally acceptable and some are

beyond minimally acceptable

NOT all graduates of a training program, since some are unacceptable, some are

minimally acceptable, and some are beyond minimally acceptable

The panel was also educated on important item characteristics; i.e., an item should:

Be relevant and important to the job tasks commonly performed and the job knowledge

regularly used

Be related to public protection and designed to eliminate those candidates who would

pose a danger to the public

Target a minimally acceptable candidate at time of certification

Include the most current information available to all potential candidates

Be phrased in the working language of the job

Be realistic and practical

Be at an appropriate reading level

Be supported by a reference

Advisory panel members rated each item as follows:

1. Item Acceptable – Panelists were asked, “Is this item acceptable for use on the

certification exam?” by selecting “Yes” (1), “No” (2), or “OK as edited” (3).

2. Relevance – Panelists were asked, “How relevant is this item to overall competent

performance of a medical interpreter’s job?” Response options were: “Of no importance”

(1), “Of little importance” (2), “Moderately important” (3), “Very important” (4), and

“Critical” (5).

3. Difficulty – Panelists participated in a modified Angoff procedure (described below) to

identify the percentage of candidates, at initial time of certification, who would answer

the question correctly.


Advisory panel members rated both Consecutive and Sight utterances (elements) of the oral

exams. On the Consecutive portion, each utterance contains two scoring units, and on the Sight

Translation portion, each segment includes 3-4 scoring units. The Angoff rating was completed

on the basis of the entire utterance/segment based on the rationale that all the scoring units in

each utterance/segment are critical to rating the acceptability, relevance and difficulty of the

given segment.

In order to ensure SME rating calibration and consistency, workshop facilitators reviewed in

detail each of the above three ratings, providing further clarity and allowing time to answer any

questions from the panel members.

1. Item Acceptable Rating

Consider the following item characteristics when completing the “acceptability” rating:

Correctness of the key

Incorrectness of the distractors

Correctness of the terminology

Applicability to the field

Use the following process to rate the acceptability of each item:

When reviewing an item independently, check either the "Yes" or the "No" box.

When the group discusses an item that one or more reviewers found unacceptable, the item

may be revised to make it acceptable.

If the revision makes the item acceptable in your opinion, check the "OK as Edited" box.

2. Relevance

How relevant is this item to minimally acceptable performance by an entry-level licensee?

0 1 2 3 4

Of no relevance Of little

relevance Moderately relevant

Very relevant Extremely relevant

3. Difficulty/Standard Setting Use the following process to rate the difficulty of each item:

Think of 100 individuals that are EXACTLY minimally acceptable

Remember that these 100 minimally acceptable candidates will have differing strengths

and weaknesses

Rate the percentage of these 100 minimally acceptable candidates that would respond

correctly to the item

Base the rating on the difficulty of the concept tested and the alternatives presented

Use a starting point of 25%, since 25% will respond correctly to the item by chance given

four alternatives

Use increments of 5%

Do NOT write a % sign

Additional materials used to facilitate the item review workshops can be found in a separate

technical report (PSI, June 2012).


Following the workshops, PSI analyzed the SME’s Item acceptability, Angoff, and Relevance

ratings. Item acceptability was calculated by examining the percentage of raters who rated an

item “Acceptable” for those items that did not need to be modified or “OK as Edited” for those

items that were modified in order to be acceptable. An item was considered acceptable if 60% or

more of the raters deemed it so. Item relevance and item competence ratings were evaluated to

determine which items were appropriate for the exam. If an item had a mean relevance 3 or

higher, the item was retained for use in the item pool.

3. Pilot Testing

Experimental forms of each oral language exam were assembled and pilot tested. LLU selected

and trained raters to evaluate examinee performance on the oral exam, and then conducted pilot

testing sessions with the oral exam. PSI analyzed the test item response data to cast equivalent

alternate forms of the oral examination. Further details regarding the methodology are described

below.

Experimental Test Forms

PSI worked with LLU to assemble two experimental forms of each of the five language oral

examinations, including the scenarios and instructions. Each exam form was comprised of two

parts, including sight interpretation (6 segments and 30 scoring units) and consecutive

interpretation (15 scenarios and 180 scoring units), for a total of approximately 21 test items and

210 scoring units per form.

Rater Selection, Training, and Calibration

Selection. LLU conducted a rigorous process for the selection and training of medical

interpretation experts to serve as raters to score the oral exams. The process incorporated input

from higher institutions of learning, such as the Monterey Institute of International Studies, and

healthcare organizations and professional associations, to form a pool of potential raters to be

used in the pilot testing process. The recommendations and resumes of candidates for this role

were meticulously reviewed and only the most qualified candidates were then invited for an

interview. The interview process gleaned additional information regarding each candidate and

their background, education, and experience in the field, as well as other important elements in

rater qualification, such as demeanor, pronunciation and enunciation, and tone of voice. It also

provided the Project Manager with a perspective on other factors related to successful testing and

rating that cannot be quantitatively measured, such as the ability to work as a team, attitude

towards coaching, and the desire to grow professionally. The candidates chosen for this role met

the highest standards in all of the aforementioned areas and also met the following criteria:

Required

At least 5 years of experience working as a medical interpreter

Completion of at least 40 hours of medical interpreter training

An advanced degree in a language, interpreting and translating, linguistics, language

pedagogy, medicine or medical specialties

At least 1 professional reference

Preferred

Prior experience in testing, test development and/or test rating

Excellent on-the-job performance ratings


Certificates of Completion, professional certification documents

Other interpreting experiences

Rater Profile

The following is a sample profile of a typical Rater:

M.A., Magna Cum Laude

University-level Instructor

10 years of experience as a medical interpreter

Free-lance translator

State Certified Court Interpreter

Training and Calibration. Once the raters were selected, intensive training was conducted by

Language Line Academy, based on its 30 years of experience in the development and rating of

interpreter skills assessments. The training included practice with test delivery, as well as

calibration sessions in which the scoring rubric (displayed in Appendix B) was applied to

evaluate sample exams and the results were reviewed and discussed. The scoring rubric is

comprised of a methodology that includes both subjective and objective elements to capture the

entire skill set needed to qualify as a certified medical interpreter. Each of the scenarios on the

examination included units that are scored on the following factors: linguistic equivalence,

conservation of register, grammatical correctness, and pronunciation. There is also a holistic

rating, which applies to the entire segment. Within each of these factors, there are detailed

performance elements that are described in the scoring rubric. Furthermore, instruction was also

provided in the proper management of responses not included in the scoring rubric, which entails

review by the test design team. The purpose of the training was to ensure that the raters shared a

common understanding of each rating element and the standards by which performance would be

evaluated. The pilot testing phase only began once the raters had reached a level of consistency

through calibration exercises, as well as quality in the delivery of the exam.

Materials used in the Rater Training included all test versions, rating forms, test results, report

forms, scoring rubric (dictionary), and a detailed description of the rating protocol.

The Rater Training consisted of:

Detailed review of test rationale and content

Extensive instruction in Rating Protocol and Procedures

Discussion of items in Scoring Dictionary and process for adding new items to dictionary

Use of Rating Form and Test Results Report

The training also involved intensive calibration exercises, as follows:

Reviewing pilot test scores, item-by-item, to gain a greater understanding of rating

process

Listening to pilot tests and rating them, until all Raters have consistent scores and all

discrepancies and doubts are resolved

o As a group first

o Individually

Filling out Rating Forms using the Scoring Dictionary


Comparing Rating Forms as a group

Test role-plays designed to present challenges for rating

o All Raters rate the test role-plays and discuss rating

The Project Manager and Lead Tester continued all of the above exercises until the Rater was

deemed to be adequately prepared to rate a test on their own, based on consistency,

understanding of rating protocol, pace of rating, and correct documentation of scores.

All tests rated by a new Rater were reviewed by the Lead Tester to identify any rating issues and

return to any of the steps above to resolve those issues. A new Rater was not assigned to rate a

test on his/her own until the Project Manager and Lead Tester were satisfied that they had a

complete understanding of the rating protocol and process. For the first two weeks, all tests rated

by a new Rater were reviewed in detail by the Lead Tester.

The same raters who participated in the pilot study also serve as raters for the operational testing

program. Future training of any new Raters who are selected will include 40 hours of training

with the Project Manager and Test Developers, and will encompass all of the same elements

described above.

Rating Protocol

The scoring rubric (see Appendix B) was developed in the 2010 exam development study, cited

earlier. The rubric was designed to be highly detailed and specific with respect to performance

behaviors such that the raters have a common frame of reference in evaluating candidate

performance. To this end, the scoring rubric presents evaluation factors and detailed performance

elements.

The Rating Protocol for the National Board Test was established by Language Line Academy’s

team of Subject Matter Experts in stages, as follows:

Research and review of alternate rating protocols for interpreter tests

Input from psychometrician re. the rationale for a rating protocol

Preparation of initial draft of rating protocol

Further discussion and modification of rating protocol

The Rating Protocol, or scoring rubric, is comprised of a methodology that includes both

objective (scoring unit methodology) and holistic rating elements to capture the entire skill set

needed to qualify as a certified medical interpreter. Each of the scenarios on the examination

includes units that are scored on the following factors: linguistic equivalence, conservation of

register, grammatical correctness, and pronunciation. The rationale for this protocol was based

on the belief that a rating methodology based solely on the objective methodology of scoring

units does not ensure a comprehensive assessment of an interpreter’s skill and knowledge, given

that interpreting involves conveying complete and accurate meaning, and not only accurate

terms. The inclusion of the more comprehensive or holistic rating ensures that other elements,

besides knowledge of terminology and language, are also measured by the test.

To ensure the utmost consistency, the Raters of the National Board oral exam utilize a Scoring

Dictionary developed by the Test Designers, according to the process described below.


Scoring Dictionary

The initial version of the Scoring Dictionary was developed by the Test Developers, who

conducted an extensive review of all scoring units on the test and possible responses, and came

to a consensus about response scoring weights. The Scoring Dictionary serves as a means of

ensuring the objectivity of the rating in that all Raters refer to the Scoring Dictionary as their sole

reference when assigning a point value to each term provided by the test candidate. It also

ensures inter‐rater reliability and consistency, since a term that is not in the Scoring Dictionary

must be researched and agreed upon by a panel, according to the following process for awarding

a point value. Furthermore, rating consistency is further enhanced by the fact that there are so

few raters per language to date.

Whenever a term comes up that is not in the Scoring Dictionary, it is submitted by the Rater to

the Expert Panel, comprised of the Lead Tester and other members of the Test Design Team.

The term is not awarded any point value until the Expert Panel has rendered an informed

decision about the acceptability of the term.

To summarize, the Scoring Dictionary is maintained in an ongoing manner according to the

following process:

A new item given by a test candidate that is not found in the Scoring Dictionary is

submitted by the Rater to the Expert Panel for research and review;

The Expert Panel comes to a consensus regarding the point value of the new test item;

The Rater is informed of the point value and applies it accordingly to the rating of the test

in which it appeared;

The Lead Tester adds the new term and its corresponding point value to the Scoring

Dictionary;

The Lead Tester provides an updated Scoring Dictionary to all Raters.

The use of the Scoring Dictionary ensures utmost consistency, as individual Raters do not rely on

their personal judgment at the moment to award value points, but rather, they go by what has

been provided in the Scoring Dictionary.

Once the Rater completes the scoring of an individual test, it is submitted to the Lead Rater for

review. The Lead Tester reviews the Score Sheet and Test Results, and also listens to the

recording of the test as appropriate. Once this review has been completed, the scored test results

are submitted to the Project Leader, who is also a qualified Rater, for a final review. Only when

both the Lead Tester and Project Leader are satisfied that the test has been rated appropriately,

according to the Rating Protocol, are the results emailed to the National Board.

The National Board subsequently provides the test candidate with the results of their test and

prepares a Certificate for candidates who have obtained a passing score.

Pilot Administration

LLU conducted a targeted recruitment of medical interpreters to complete one of the

experimental oral exams for the pilot study. Pilot participants came from all regions of the

country and included representation from colleges and universities, healthcare institutions,

professional interpreter associations, and included free-lance professional interpreters, staff

interpreters and dual-role interpreters. A proctor was identified at each testing location and

provided with detailed instruction in the confidential test delivery process. This element

represented a means of ensuring test security and, ultimately, valid test results.


In light of the relatively limited number of medical interpreters practicing in the five languages

(Russian, Mandarin, Cantonese, Korean, and Vietnamese), pilot testing samples were comprised

of approximately 30 medical interpreters per experimental Form A and Form B, per language.

Scoring

As noted above (see Training and Calibration), evaluators were selected and trained to apply the

scoring rubric to score the pilot examinees. For a subset of participants in the pilot study, two

raters completed the scoring rubric to enable calculation of reliability statistics for the oral exam.

LLU collected the oral exam ratings and created a data file for analysis by PSI. PSI conducted

preliminary edit checks of the data file and worked with LLU to ensure that the data were

accurate.

It should be noted that in the operational administration and scoring of the oral exams, the same

pool of raters in the pilot study will be used in scoring the oral examinations. Moreover, all test

results reports are reviewed at two different levels before they are provided to the candidate. The

process for rating all tests is for the rater to submit his/her Score Sheet to the Lead Tester, who

then reviews the Score Sheet in its entirety, and also listens to the recording of the test, as

appropriate, if there are any questions about a term and its rating. Once this review has been

completed, the Score Sheet and Test Results Report are submitted to the Project Leader, who is

also a qualified rater, for a final review. Only when both parties are ensured that the test has been

rated appropriately, according to the rating protocol, are the results emailed to the National

Board. This is a two-level review process that ensures the quality and consistency of all test

ratings.

Results

Statistical analyses of the experimental forms were conducted, which are summarized in Table 1,

including the number of examinees completing the form (N), number of item scoring units, mean

rating, standard deviation (SD), Cronbach’s alpha coefficient of internal consistency reliability,

mean relevance rating, the mean Angoff rating, and the inter-rater correlation index of rating

reliability for a subset of examinees (Russian: N=22; Mandarin: N=28; Cantonese: N=31;

Korean: N=30; Vietnamese: N=30); all total score alphas were observed to be .93 or higher

across all forms of all five language exams.

Inter-rater Reliability Estimate. The inter-rater reliability coefficient is the correlation

between pairs of raters who applied the scoring rubric to evaluate a subset of examinees. These

raters are similar in professional experience to the full set of raters and they received the same

training and followed the same standard rating procedures. Thus, it is reasonable to expect that

the sample of raters is representative of the rater population who apply the same scoring rubric

and receive the same training. The high level of agreement can be attributed to rater calibration

training and to sound item characteristics. The scoring rubric was found to be highly reliable, as

evidenced by the inter-rater reliability coefficients ranging from .87 to .99, thus reflecting the

high quality and effectiveness of the rater training. Because the oral exam is manually scored by

raters who apply the scoring rubric, the inter-rater reliability index is useful in describing the

degree of measurement precision (freedom from rater error) that can be expected. The reported

inter-rater reliability is a strong indication of uniformity among raters. This reliability can be

assumed by other exam forms due to the exam form equivalency. The exam form characteristics


(mean, SD, reliability, avg. Angoff, avg. Relevance) are parallel across the three forms and this

supports the generalization of the rater reliability to these parallel forms.


Table 1: Oral Exam Pilot Form Descriptive Statistics

Russian Pilot Examination Forms

N

#

Scenarios /

# Scoring

Units Mean SD

Cronbach's

Alpha

Mean

Relevance

Rating

Mean

Angoff

Rating

Rater

Correlation

(N=22)

Form A

Sight 30 6 / 30 41.5 6.63 0.78 3.79 71.92 0.91

Consecutive 30 15 / 180 213.98 29.65 0.94 3.74 80.5 0.87

Total 30 21 / 210 255.48 33.99 0.94 3.75 79.5 0.88

Form B

Sight 32 6 / 30 41.63 6.49 0.75 3.79 67.54

Consecutive 32 15 / 180 207.75 30.58 0.93 3.88 78.59

Total 32 21 / 210 249.38 33.61 0.93 3.87 77.29

Mandarin Pilot Examination Forms

N

#

Scenarios /

# Scoring

Units Mean SD

Cronbach's

Alpha

Mean

Relevance

Rating

Mean

Angoff

Rating

Rater

Correlation

(N=28)

Form A

Sight 31 6 / 30 46.23 5.96 0.8 3.99 82.89 0.95

Consecutive 31 15 / 180 244.44 27.16 0.94 3.76 82.89 0.92

Total 31 21 / 210 290.66 30.64 0.94 3.78 82.89 0.96

Form B

Sight 32 6 / 30 46 6.34 0.76 3.98 74.35

Consecutive 32 15 / 180 246 24.78 0.92 3.79 81.84

Total 32 21 / 210 292 29.61 0.93 3.81 80.96

Cantonese Pilot Examination Forms

N

#

Scenarios /

# Scoring

Units Mean SD

Cronbach's

Alpha

Mean

Relevance

Rating

Mean

Angoff

Rating

Rater

Correlation

(N=31)

Form A

Sight 32 6 / 30 39.53 7.78 0.83 4 81.87 0.93

Consecutive 32 15 / 180 195.22 39.72 0.96 3.91 84.76 0.96

Total 32 21 / 210 234.75 45.97 0.97 3.92 84.42 0.96

Form B

Sight 30 6 / 30 39.17 9.51 0.87 4 78.77

Consecutive 30 15 / 180 199 28.23 0.92 3.91 84.76

Total 30 21 / 210 238.17 35.71 0.94 3.92 84.06


Korean Pilot Examination Forms

N

#

Scenarios /

# Scoring

Units Mean SD

Cronbach's

Alpha

Mean

Relevance

Rating

Mean

Angoff

Rating

Rater

Correlation

Form A

Sight 30 6 / 30 36.97 9.19 0.84 4.00 77.89 0.93

Consecutive 30 15 / 180 192.68 41.03 0.96 3.77 85.48 0.96

Total 30 21 / 210 229.65 47.39 0.96 3.80 84.59 0.96

Form B

Sight 30 6 / 30 32.43 10.93 0.88 4.00 73.00

Consecutive 30 15 / 180 196.60 31.98 0.93 3.86 86.85

Total 30 21 / 210 229.03 40.62 0.95 3.88 85.22

Vietnamese Pilot Examination Forms

N

#

Scenarios /

# Scoring

Units Mean SD

Cronbach's

Alpha

Mean

Relevance

Rating

Mean

Angoff

Rating

Rater

Correlation

Form A

Sight 31 6 / 30 29.82 8.13 0.82 3.91 79.67 0.99

Consecutive 31 15 / 180 160.29 36.32 0.95 3.75 85.67 0.98

Total 31 21 / 210 188.32 43.10 0.96 3.77 84.96 0.99

Form B

Sight 30 6 / 30 29.43 9.44 0.84 4.00 66.20

Consecutive 30 15 / 180 180.23 28.57 0.92 3.89 77.94

Total 30 21 / 210 209.67 34.04 0.93 3.90 76.56

4. Test Form Assembly

The test item statistics were reviewed and served as a basis for assembling three alternate oral

exam forms (A, B, and C) for each of the five languages that are all representative of the content

outline and statistically equivalent (i.e., comparable in mean, SD, and reliability), meeting

conventional standards of equivalence (e.g. Allen & Yen, 1978). The item-level statistics for the

items comprising each new test form are shown in Appendix C.

Statistical properties of the resulting oral examination Forms A-C are summarized in Table 2.

These new forms were comprised of a combination of items from the experimental Forms A and

B. Item selection for each new exam form was based on content and statistical equivalence, as

well as the mean Angoff rating. Also, the inter-rater reliability ratings for Form A were taken

into consideration, because two raters scored the items on this form. Furthermore, the new test

forms were cast shorter than the experimental versions to reduce testing time and the potential

for test fatigue. The consecutive/oral section was reduced from 15 to 12 scenarios.


Table 2: Statistical Properties of Final Oral Examination Forms for Five Languages

Russian

N

# Scenarios /

# Scoring

Units

Total

Points

Possible Mean % SD

Test

Reliab

ility

Mean

Relevance

Rating

Mean

Angoff

Rating

Form A

Sight 30 6 / 30 54 41.5 76.90% 4.7 0.78 3.79 71.92

Consecutive 30 12 / 144 240 161.57 67.30% 25.45 0.92 3.71 79.14

Total 30 18 / 174 294 203.07 69.10% 30.15 0.93 3.74 78.08

Form B

Sight 32 6 / 30 54 41.63 77.10% 3.52 0.75 3.79 67.54

Consecutive 32 12 / 144 240 161.62 67.30% 26.66 0.92 3.93 79.54

Total 32 18 / 174 294 203.31 69.20% 33.71 0.93 3.91 77.16

Form C

Sight 30-32 6 / 30 54 41.27 76.40% 4.1 0.34 3.78 69.46

Consecutive 30-32 12 / 144 240 162.29 67.60% 26.22 0.92 3.83 78.52

Total 30-32 18 / 174 294 203.56 69.20% 34.43 0.93 3.82 77.22

Mandarin

N

# Scenarios /

# Scoring

Units

Total

Points

Possible Mean % SD

Test

Reliab

ility

Mean

Relevance

Rating

Mean

Angoff

Rating

Form A

Sight 31 6 / 30 54 46.23 85.60% 5.96 0.8 3.99 82.89

Consecutive 31 12 / 144 240 191.35 79.70% 23.98 0.92 3.7 82.03

Total 31 18 / 174 294 237.59 80.80% 27.86 0.93 3.74 82.15

Form B

Sight 32 6 / 30 54 46 85.20% 5.12 0.76 3.98 74.35

Consecutive 32 12 / 144 240 192.13 80.10% 21.79 0.91 3.81 80.62

Total 32 18 / 174 294 238.13 81.00% 26.91 0.93 3.83 79.72

Form C

Sight 31-32 6 / 30 54 44.61 82.60% 4.83 0.56 3.99 76.18

Consecutive 31-33 12 / 144 240 193.83 80.80% 23.06 0.91 3.76 80.89

Total 31-34 18 / 174 294 238.44 81.10% 27.88 0.93 3.8 80.22

Cantonese

N

# Scenarios /

# Scoring

Units

Total

Points

Possible Mean % SD

Test

Reliab

ility

Mean

Relevance

Rating

Mean

Angoff

Rating

Form A

Sight 32 6 / 30 54 39.53 73.20% 6.69 0.83 4 81.87

Consecutive 32 12 / 144 240 155.59 64.80% 32.45 0.95 3.9 84.4

Total 32 18 / 174 294 195.12 66.40% 39.14 0.96 3.92 84.04

Form B

Sight 30 6 / 30 54 39.13 72.50% 7.88 0.87 4 78.77

Consecutive 30 12 / 144 240 156.41 65.20% 28.06 0.93 3.93 85.26

Total 30 18 / 174 294 195.55 66.50% 35.98 0.95 3.94 84.33

Form C

Sight 30-32 6 / 30 54 37.62 69.70% 7.67 0.77 4 78.8

Consecutive 30-32 12 / 144 240 158.11 65.90% 29.85 0.95 3.89 83.79

Total 30-32 18 / 174 294 195.73 66.60% 37.55 0.96 3.9 83.08


Korean

N

# Scenarios /

# Scoring

Units

Total

Points

Possible Mean % SD

Test

Reliab

ility

Mean

Relevance

Rating

Mean

Angoff

Rating

Form A

Sight 30 6 / 30 54 36.97 68.5% 6.82 0.68 4.00 77.89

Consecutive 30 12 / 144 240 150.52 62.7% 33.31 0.95 3.81 86.69

Total 30 18 / 174 294 187.48 63.8% 40.12 0.95 3.83 85.44

Form B

Sight 30 6 / 30 54 32.43 60.1% 9.19 0.82 4.00 73.00

Consecutive 30 12 / 144 240 154.97 64.6% 31.75 0.95 3.74 86.41

Total 30 18 / 174 294 187.40 63.7% 41.01 0.96 3.78 84.49

Form C

Sight 30 6 / 30 54 36.57 67.7% 7.38 0.74 4.00 77.70

Consecutive 30 12 / 144 240 150.92 62.9% 30.42 0.94 3.81 84.74

Total 30 18 / 174 294 187.48 63.8% 37.82 0.95 3.84 83.74

Vietnamese

N

# Scenarios /

# Scoring

Units

Total

Points

Possible Mean % SD

Test

Reliab

ility

Mean

Relevance

Rating

Mean

Angoff

Rating

Form A

Sight 30 6 / 30 54 29.82 55.2% 7.09 0.78 3.91 79.67

Consecutive 30 12 / 144 240 137.35 57.2% 20.68 0.86 3.87 82.80

Total 30 18 / 174 294 167.18 56.9% 14.82 0.95 3.88 82.35

Form B

Sight 30 6 / 30 54 29.43 54.5% 6.34 0.60 4.00 66.20

Consecutive 30 12 / 144 240 137.88 57.5% 19.28 0.84 3.86 80.52

Total 30 18 / 174 294 167.31 56.9% 14.67 0.94 3.88 78.47

Form C

Sight 30 6 / 30 54 28.18 52.2% 7.41 0.74 3.91 71.85

Consecutive 30 12 / 144 240 138.99 57.9% 21.11 0.86 3.87 81.30

Total 30 18 / 174 294 167.17 56.9% 14.74 0.95 3.88 79.95

5. Recommended Minimum Passing Scores

Empirical examinee data were balanced with modified Angoff ratings to establish recommended

passing scores, or cut scores, for the five language oral examinations. In addition, language

experts were consulted to explain similarities and differences in the difficulty of interpreting in

different languages.

The Angoff method is a widely accepted and researched method for standard-setting, and has

accumulated considerable empirical evidence to support its validity (Hurtz & Auerbach, 2003).

In this process, experts were asked to rate the percentage of candidates (at the time of initial

certification) who would successfully answer each item on the examination. After each expert

completed these ratings, mean ratings were determined for each item. These mean ratings were

aggregated to determine the cut score for a specific examination, based upon the items included

on the exam.

PSI psychometric staff conducted statistical analyses of the Angoff ratings for the oral item

ratings provided by the committee members during the SME review meeting described earlier in


this report in section 2.3. Ratings were averaged across all raters who provided data for an item.

Full item Angoff and Relevance ratings for the oral exam are shown in Appendix D, and are

summarized in Table 2 for each language exam by form and section.

Results and recommendations for each language exam are summarized below.

Russian

Pilot exam descriptive statistics (Mean, Standard Deviation, and Cronbach’s Alpha) were

reviewed in conjunction with SME’s Angoff ratings of the items selected to comprise the

Russian oral exam forms. Balancing these two factors, a cut score of 70% was recommended,

which was consistent with the previously developed oral certification exam for Spanish

interpreters (see Development and Validation of Oral and Written Examinations for Medical

Interpreter Certification Technical Report, April 2010. While the average Angoff ratings were

higher than 70 on the Russian exam, the empirical results were comparable to the Spanish forms

and thus given more weight than the SME judgments in identifying a recommended cut score

that would be consistent with established standards.

Mandarin

Pilot study participants performed considerably higher on the sight and consecutive portions of

the Mandarin oral examination form in comparison to the Russian and Spanish forms. This

result was consistent with higher Angoff ratings for the Mandarin items, indicating that SMEs

thought the exam was easier in the Mandarin language. It is hypothesized that the higher pilot

scores and Angoff ratings for the Mandarin oral exam may be due to inherent characteristics of

the language, which may offer fewer “wrong choices.” Because the Mandarin test form statistics

and average Angoff ratings were consistently higher than the Russian and Spanish forms, a

correspondingly higher cut score was recommended for the Mandarin oral exam forms. An

overall cut score of 80% was established for the Mandarin medical interpreter certification oral

examination.

Cantonese

The Cantonese pilot exam participants scored substantially lower than the average Angoff

ratings. In balancing the two sets of information, along with anecdotal information about the

population of Cantonese medical interpreters, more weight was given to the empirical results and

a cut score of 70% was recommended, consistent with the Spanish and Russian exams, which

had similar empirical results.

Korean

The Korean pilot exam participants, similar to the Cantonese pilot study, scored substantially

lower than the average Angoff ratings. In balancing the two sets of information, along with

anecdotal information about the population of Korean medical interpreters, more weight was

given to the empirical results and a cut score of 70% was recommended, consistent with the

Spanish, Russian and Cantonese exam, which had similar empirical results.

Vietnamese

The Vietnamese pilot exam participants performed substantially lower than examinees on the

other language exams, with a total score average of 57% correct. The Angoff ratings were

somewhat higher (ranging from 78 to 82) and thus, in balancing the two sets of data, a cut score

of 65% was recommended.


Ongoing Review Process

This section describes the testing program practices that are planned for the five language

interpreter oral exams after they are implemented.

Rater Calibration

The ongoing quality and consistency of the rating process is of the utmost importance to the

National Board. Any minor rating anomalies noted by the Lead Rater during her regular review

of test results are identified and resolved immediately before the Rater continues to rate any tests.

Additionally, annual calibration exercises are conducted with all Raters who are actively rating

National Board Tests. All calibration exercises conducted with the Raters indicate that they are

highly consistent and well-calibrated. Their accuracy is at the same level as identified during the

initial inter-rater reliability study, as described in the 2010 technical report (Development and

Validation of Oral and Written Examinations for Medical Interpreter Certification Technical

Report, April 2010). These annual calibration sessions will consist of the following:

Review of updated Scoring Rubric (Dictionary)

Calibration sessions with sample of recorded tests

Discussion of any test-related issues that may surface

Exam Review

Exam and item analysis reports (technical reports) will be prepared by PSI annually in the first

quarter of each year to provide statistical and diagnostic summaries for each test item and exam

form, where feasible. Each exam item will be monitored to ensure effectiveness through the life

of the item.

The item analysis will address common parameters, such as difficulty, discrimination (point-

biserial correlation) and distractor effectiveness (percentage of respondents in conjunction with

point-biserial correlation). These results, in conjunction with item exposure rates, will be

examined to develop a plan for new content development. The aim of the plan will be to

introduce at least 1/3 fresh content per year, with item exposure of 100 to 500 examinees, and

acceptable item statistics (significant positive total point-biserials; negative distractor-total

correlations, etc.).

The exam analyses will address common parameters, such as item difficulty, discrimination

(point-biserial correlation) and distractor effectiveness. In addition, it will include a calculation

of the exam mean, SD, internal consistency reliability, and pass rates. For the oral exam forms,

an analysis of inter-rater reliability will be conducted by drawing a 10% sample of candidate

exams, a minimum of 10 per exam form A, B, and C (a total of at least 30 candidates). For this

sample, a second trained and qualified rater will independently evaluate the candidates’ recorded

oral responses to the oral certification exam. These second scores will be correlated with the

candidates’ original scores using the same methods described earlier to estimate the reliability of

a single rater. This sample is expected to be sufficient with respect to standard error of the

correlation.

Exam analysis results, in combination with item exposure rates, will be examined to develop a

plan for new content development. The aim of the plan will be to introduce at least 1/3 fresh


content per year, with item exposure of 100-500 examinees, and acceptable item statistics

(significant positive total point-biserials; negative distractor-total correlations, etc.).

PSI, LLU, and the National Board plan to continually monitor the performance of the written and

oral exams each calendar year, as described above, and will continuously update the exams as

needed to ensure that the items and exam forms are performing satisfactorily, and that new

content is introduced to avoid over-exposure of the item content, all in a manner that is

reasonable and consistent with the size and economics of the examination program.

Summary and Conclusion

The Russian, Mandarin, Cantonese, Korean, and Vietnamese language medical interpreter

certification oral examination development project was founded upon a national job analysis

with representation across the US, including a wide range of medical interpreting settings. The

exam content specifications incorporate knowledge domains that were identified in the national

job analysis and which were reviewed by a panel of subject matter experts. These specifications

served to guide test item development and examination construction. An extensive pilot test

phase and rigorous psychometric analysis provided the basis for assembling statistically

equivalent oral examination forms that meet psychometric criteria for reliability and content

validity.


References

Allen, M.J., & Yen, W.M. (1979). Introduction to measurement theory. Monterey, CA:

Brooks/Cole.

Hurtz, G.M., & Auerbach, M.A. (2003). A Meta-Analysis of the Effects of Modifications to the

Angoff Method on Cutoff Scores and Judgment Consensus. Educational and

Psychological Measurement, 63 (4), 584-601.

PSI Services, LLC (2010). Development and Validation of Oral and Written Examinations for

Medical Interpreter Certification. Burbank, CA.

PSI Services, LLC (June, 2012). Addendum: Development and Validation of Oral and Written

Examinations for Medical Interpreter Certification: Standard Setting Technical Report.

Burbank, CA.

Standards for Educational and Psychological Testing. (1999). Washington, DC: American

Educational Research Association.

©2013, PSI Services LLC. All rights reserved. A - 1

Appendix A. Oral Examination Expert Advisory Committee

Table A-1. Oral Examination Expert Advisory Committee

Expert Company Position

*Affiliation with Industry

Associations/Positions

Yea

rs o

f

Ex

per

ien

ce

Su

rvey

Des

ign

Co

nte

nt

Sp

ecif

ica

tio

n

Item

Rev

iew

an

d

Sta

nd

ard

Set

tin

g

Russian Language Committee

Angelina Levitskaya University of Arkansas

for Medical Sciences

Director, Medical Interpreter

Training Program

IMIA; Russian-American Partnership,

Assistant Director; AR International

Visitors Center, UALR;

39 No Yes Yes

P. Elana Pick Self-employed

Interpreter/Translator; Adjunct

Faculty, NYU School of Continuing

Education and Professional Studies;

Language and Interpreting Skills

Coach; ATA Slavic Division

Assistant Administrator;

ATA (Certified Translator); NCIHC;

People to People International; New York

Circle of Translators

34 No Yes Yes

Natalia Petrova Albert Einstein Medical

Center

Staff Russian Medical Interpreter;

Certified Russian Court Interpreter;

Teacher

Member, ATA; NAJIT; DVTA; 24 No Yes Yes

Irina Milnes Language Line Services Interpreter/Translator; Interpreter

Trainer, Mentor and Examiner n/a 31 No No Yes

Karin Elliot

Russian Language

Services; Southern

Maine Community

College

Independent Interpreter/Translator,

Trainer and Interpreter Evaluator;

Adjunct Professor

Member, NAJIT; IMIA; AMIT; Deputy Director/ Program Co-Director, Global Survival Network (GNA);

20 No Yes Yes

Mandarin Language Committee

Yilu Ma

Brigham and Women’s

Hospital; Cambridge

College and Boston

University

Director, Interpreter Services

Department; Adjunct Professor of

Interpreting

IMIA; China’s Translators Association 21 No Yes Yes

Yoshie Ng

Tufts Medical Center,

Boston; Culture Smart,

Inc.

Medical Interpreter and Trainer;

language coach; teacher

IMIA; BAHEC Board Member; Medical

Reserve Corps 14 No Yes Yes

Ka-Lam Chen Language Line Services Interpreter, Mentor n/a 12 No Yes Yes

Ying Shi Language Line Services Interpreter, Interpreter Trainer and n/a 15 No Yes Yes





Yea

rs o

f

Ex

per

ien

ce

Su

rvey

Des

ign

Co

nte

nt

Sp

ecif

ica

tio

n

Item

Rev

iew

an

d

Sta

nd

ard

Set

tin

g

Examiner

Li Ying Catholic Charities and

Maine Medical Center Interpreter n/a 20 No Yes No

Danyune Geertsen Language Line Services Director, Training and Quality

Assurance ASTM, ATA, CHIA, NCIHC 24 Yes Yes Yes

Meng Zhang Language Line Services Interpreter and Examiner n/a 13 No Yes Yes

Cantonese Language Committee

Garrison Chan Certified Languages

Int’l Interpreter n/a 2 No Yes Yes

Icarus Tsang

California Pacific

Medical

Center, San Francisco

Medical Interpreter; QA Team Lead;

Language Coach; Test Developer n/a 10 No Yes Yes

Johnson Siu Language Line Services Interpreter, Examiner n/a 10 No Yes Yes

Ka-Lam Chen Language Line Services Interpreter, Mentor n/a 12 No Yes Yes

Yoshie Ng

Tufts Medical Center,

Boston; Culture Smart,

Inc.

Medical Interpreter and Trainer;

language coach; teacher

IMIA; BAHEC Board Member; Medical

Reserve Corps 14 No Yes Yes

Rebecca Pfahl Language Line Services Primary Care Physician, Medical

Intepreter, Examiner n/a 6 No Yes Yes

Korean Language Committee

Sung Cho Abington Memorial

Hospital

Medical Interpreter/Trainer,

Translator n/a 14 No Yes Yes

David Lee Grady Memorial

Hospital Medical Interpreter n/a 6 No Yes Yes

Tina Kang Language Line Services Medical Interpreter n/a No Yes Yes

Monika Lee Language Line Services Medical Interpreter, Examiner,

Mentor n/a 17 No Yes Yes

Sally Koo Language Line Services Medical Interpreter, Examiner n/a 15 No Yes Yes

Vietnamese Language Committee

Lien-Hoa Pham Maine Medical Ctr. and

Catholic Charities Maine Medical Interpreter, Translator n/a 12 No Yes Yes

Thuy Van Language Line Services Interpreter, Examiner n/a No Yes Yes





Yea

rs o

f

Ex

per

ien

ce

Su

rvey

Des

ign

Co

nte

nt

Sp

ecif

ica

tio

n

Item

Rev

iew

an

d

Sta

nd

ard

Set

tin

g

Anna Nguyen Language Line Services Interpreter, Examiner n/a 16 No Yes Yes

Tin Quan Language Line Services Interpreter n/a 7 No Yes Yes

Tram Bui Language Line Services Interpreter n/a 9 No No Yes

* Affiliation information is not complete or all-inclusive, since this was not a required criterion for participation in all stages of the test development.

©2013, PSI Services LLC. All rights reserved. B - 1

Appendix B. Scoring Rubric

NMICT Oral Test Scoring Rubric

Consecutive Role-Play

Each of the utterances or segments of the consecutive interpreting portion of the NMICT

Oral Test will be rated according to the following scoring guidelines. NOTE: When the term

“reference materials” is used, that includes the Scoring Dictionary.

Scoring Unit Rating

For each of the two identified scoring units in the utterance/segment, the test candidate will receive 2

points if the term used is

cited in reference materials as a proper linguistic equivalent1 for the term, with no deviations

2

from standard pronunciation3, no grammatical errors

4 and conservation of register.

a synonym cited in reference materials as a proper linguistic equivalent for the term, with no

deviations from standard pronunciation, no grammatical errors, and conservation of register.

a word or phrase that accurately conveys the meaning of the term, as defined in reference

materials, with no deviations from standard pronunciation, no grammatical errors, and

conservation of register, (when a standard and proper linguistic equivalent does not exist in

the target language).


point if the term used is

cited in reference materials as a linguistic equivalent for the term, with no deviations from

standard pronunciation and no grammatical errors but without conserving register.

cited in reference materials as a proper linguistic equivalent for the term, with no deviations

from standard pronunciation, conservation of register, and no more than one grammatical

error.

a definition of the scoring unit’s meaning cited in reference materials, with no deviations from

standard pronunciation, no grammatical errors, and conservation of register when a standard

and proper linguistic equivalent exists in the target language.

cited in reference materials as a proper linguistic equivalent for the term, with no grammatical

errors, conservation of register and more than one deviation from standard pronunciation.


points if the term used is

1 Regional variations that can also be verified in reference materials as being proper linguistic equivalents are also

considered to be two-point answers. 2 Deviations are defined as the following: omitting a phoneme, adding a phoneme or substituting one phoneme for

another. 3 Standard pronunciation is subject to regional variations, all of which will be considered acceptable, provided that the

pronunciation is readily understood by speakers who are not from the same region. 4 Grammatical errors are limited to gender agreement and plural/singular agreement among parts of the scoring unit only.

Only grammatical errors forming part of the scoring unit shall be considered. Grammatical errors in other parts of the

sentence will not be taken into account.


omitted from the interpretation.

cited in reference materials as a proper linguistic equivalent for the term, with two or more

deviations from standard pronunciation.

not cited in reference materials as the proper linguistic equivalent for the term.

left in the source language.

a definition of the scoring unit’s meaning not cited in reference materials.

cited in reference materials as a lower register linguistic equivalent for the term, with one or

more deviations from standard pronunciation.

cited in reference materials a lower register linguistic equivalent for the term, with one or

more grammatical errors.

a definition of the scoring unit’s meaning cited in reference materials, with one or more

deviations from standard pronunciation.

a definition of the scoring unit’s meaning cited in reference materials, with one or more

grammatical errors.

a false cognate.

Holistic Rating

An additional 1 point will be awarded for each utterance/segment as long as none of the following

errors are noted:

Significant meaning change of the utterance

Omission of a significant part of the utterance that causes a change in the outcome of the

segment

Significant additions or embellishments to the utterance

Coining or invention of words that do not exist in reference materials

Noticeably heavy accent/mumbling/incoherent speech that impedes understanding by the

Examiner

A total of 6 repetitions of utterances/segments will be allowed during the entire test. Repetitions

must be requested by using the appropriate protocol and the Examiner will repeat the entire

utterance/segment.

Total possible points: 240


Sight Test

The Sight Test will be scored according to the aforementioned rubric for the consecutive role-play,

with the exception of the Holistic Rating.

Scoring Unit Rating

Each Sight Test is comprised of 2 separate scripts, each of which have 12 scoring units, to be rated

according to the 2-1-0 pt. Scoring Unit rubric described above.

Holistic Rating

Each separate script has 3 paragraphs, each of which will receive a holistic rating, according to the 1-

0 pt. Holistic Scoring rubric described above.

The total possible number of points that a candidate can receive for the Sight Test, therefore, will be

6 points, as follows:

24 scoring units X 2 possible points = 48 points

6 paragraphs X 1 possible point = 6 points

Total Number of Possible Points = 54

©2013, PSI Services LLC. All rights reserved. C - 1

Appendix C. New Form Item Statistics

Table C-1. Russian New Form A Item Statistics

Original Variable N Mean

Points

Possible SD

Item total

Correlation

Form A, Block 1, Paragraph 1, Scoring Unit 1 SU1_Ave 30 0.95 2 0.65 -0.06

Form A, Block 1, Paragraph 1, Scoring Unit 2 SU2_Ave 30 1.75 2 0.5 0.22



Form A, Block 1, Paragraph 1, Holistic Scoring

Unit

HS1_Ave

30 0.88 1 0.28 0.58

Item Total 7.18 0.26






Unit

HS2_Ave

30 0.85 1 0.27 0.23







Unit

HS3_Ave

30 0.88 1 0.25 0.54







Unit

HS4_Ave

30 0.82 1 0.31 0.46







Points

Possible SD

Item total

Correlation




Unit

HS5_Ave

30 0.93 1 0.22 0.39



Form A, Block 2, Paragraph 3, Scoring Unit 2 SU23_Ave 30 1.53 2 0.81 -0.1



Unit

HS6_Ave

30 0.97 1 0.13 0.22

Item Total 4.92 0.2

tot_sight1 41.5 54

A 1, Option A, Scoring Unit 1 @1ASU1_Ave 30 0.82 2 0.5 0.43


A 1, Option A, Holistic Scoring Unit @1AHS_Ave 30 0.83 1 0.36 0.5

A 1, Option B, Scoring Unit 1 @1BSU1_Ave 30 1.1 2 0.67 0.28


A 1, Option B, Holistic Scoring Unit @1BHS_Ave 30 0.87 1 0.26 0.21

A 1, Option C, Scoring Unit 1 @1CSU1_Ave 30 1.1 2 0.59 0.15


A 1, Option C, Holistic Scoring Unit @1CHS_Ave 30 0.73 1 0.34 0.28

A 1, Option D, Scoring Unit 1 @1DSU1_Ave 30 1.28 2 0.69 0.27


A 1, Option D, Holistic Scoring Unit @1DHS_Ave 30 0.78 1 0.39 0.44

Item Total 11.87












Points

Possible SD

Item total

Correlation




Item Total 14.48








A 4, Option C, Scoring Unit 2 @4CSU2_Ave 30 0.97 2 1 0.26





Item Total 13.67













Item Total 15.9






Points

Possible SD

Item total

Correlation










Item Total 15.5

A 7, Option A, Scoring Unit 1 @7asu1_Ave 30 0.75 2 0.85 0.5


A 7, Option A, Holistic Scoring Unit @7ahs_Ave 30 0.72 1 0.41 0.38

A 7, Option B, Scoring Unit 1 @7bsu1_Ave 30 1.73 2 0.41 0.08


A 7, Option B, Holistic Scoring Unit @7bhs_Ave 30 0.82 1 0.31 0.64

A 7, Option C, Scoring Unit 1 @7csu1_Ave 30 1.67 2 0.58 0.42


A 7, Option C, Holistic Scoring Unit @7chs_Ave 30 0.75 1 0.39 0.72

A 7, Option D, Scoring Unit 1 @7dsu1_Ave 30 1.35 2 0.57 0.37


A 7, Option D, Holistic Scoring Unit @7dhs_Ave 30 0.62 1 0.47 0.52

Item Total 12.27









A 8, Option C, Holistic Scoring Unit @8chs_Ave 30 0.73 1 0.41 -0.03




Points

Possible SD

Item total

Correlation



Item Total 14.4













Item Total 15.06













Item Total 12.97







Points

Possible SD

Item total

Correlation









Item Total 12.18













Item Total 13.03

B 1, Option A, Scoring Unit 1 1ASU1 32 0.88 2 0.94 0.18

B 1, Option A, Scoring Unit 2 1ASU2 32 0.91 2 1 0.19

B 1, Option A, Holistic Scoring Unit 1AHS 32 0.38 1 0.49 0.36

B 1, Option B, Scoring Unit 1 1BSU1 32 1.72 2 0.58 0.21


B 1, Option B, Holistic Scoring Unit 1BHS 32 0.94 1 0.25 0.08

B 1, Option C, Scoring Unit 1 1CSU1 32 0.47 2 0.76 0.28


B 1, Option C, Holistic Scoring Unit 1CHS 32 0.59 1 0.5 0.42

B 1, Option D, Scoring Unit 1 1DSU1 32 0.97 2 0.97 0.27




Points

Possible SD

Item total

Correlation

B 1, Option D, Holistic Scoring Unit 1DHS 32 0.75 1 0.44 0.05

Item Total 10.25

tot_oral1 161.57 240


Table C-2. Russian New Form B Item Statistics

Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation

Form B, Block 1, Paragraph 1, Scoring Unit 1 SU1 32 1.56 2 0.67 0.18




Form B, Block 1, Paragraph 1, Holistic Scoring

Unit

HS1

32 0.91 1 0.39 0.25

Item Total 6.88



Form B, Block 1, Paragraph 2, Scoring Unit 3 SU7 32 1.72 2 0.68 -0.19



Unit

HS2

32 0.84 1 0.37 0.37

Item Total 7.16






Unit

HS3

32 0.91 1 0.3 0.23

Item Total 6.63






Unit

HS4

32 0.84 1 0.37 0.32

Item Total 7.22







Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation


Unit

HS5

32 0.84 1 0.37 0.18

Item Total 7.69





Unit

HS6

32 0.91 1 0.3 0.39

Item Total 6.13

tot_sight1 41.69 54


B 1, Option A, Scoring Unit 2 1ASU2 32 0.91 2 1 0.19











Item Total 10.25













Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation


Item Total 14.53










B 3, Option D, Scoring Unit 1 3DSU1 32 2 2 0 .(a)



Item Total 16.72












B 4, Option D, Holistic Scoring Unit 4DHS 32 0.94 1 0.25 -0.09

Item Total 14.84







Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation








Item Total 13.75



B 6, Option A, Holistic Scoring Unit 6AHS 32 1 1 0 .(a)










Item Total 15.75

B 7, Option A, Scoring Unit 1 7asu1 32 1.75 2 0.44 0.27


B 7, Option A, Holistic Scoring Unit 7ahs 32 0.72 1 0.46 0.05

B 7, Option B, Scoring Unit 1 7bsu1 32 1.16 2 0.81 0.55


B 7, Option B, Holistic Scoring Unit 7bhs 32 0.75 1 0.44 0.4

B 7, Option C, Scoring Unit 1 7csu1 32 0.97 2 0.82 0.46


B 7, Option C, Holistic Scoring Unit 7chs 32 0.59 1 0.5 0.1

B 7, Option D, Scoring Unit 1 7dsu1 32 1.81 2 0.4 0.49


B 7, Option D, Holistic Scoring Unit 7dhs 32 0.91 1 0.3 0.3


Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation

Item Total 13.41






B 8, Option B, Holistic Scoring Unit 8bhs 32 0.81 1 0.4 0







Item Total 14.63













Item Total 11.34








Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation







Item Total 12.97





B 14, Option B, Scoring Unit 2 14bsu2 32 1.19 2 0.64 -0.02








Item Total 11.75







B 15, Option C, Scoring Unit 1 15csu1 32 1 2 0.98 0.26






Item Total 11.69


Original

Variable

Name N Mean

Total

Points

Possible SD

Item total

Correlation

tot_oral1 161.62 240


Table C-3. Russian New Form C Item Statistics

Original

Variable

Name N Mean SD

Item total

Correlation

Form B, Block 1, Paragraph 1, Scoring Unit 1 SU1 32 1.56 0.67 0.18





Unit

HS1

32 0.91 0.39 0.25

Item Total 6.88



Form B, Block 1, Paragraph 2, Scoring Unit 3 SU7 32 1.72 0.68 -0.19



Unit

HS2

32 0.84 0.37 0.37

Item Total 7.16






Unit

HS3

32 0.91 0.3 0.23

Item Total 6.63

Form A, Block 2, Paragraph 1, Scoring Unit 1 SU13_Ave 30 1.32 0.74 0.19





Unit

HS4_Ave

30 0.82 0.31 0.46

Item Total 6.97







Original

Variable

Name N Mean SD

Item total

Correlation


Unit

HS5_Ave

30 0.93 0.22 0.39

Item Total 8.73


Form A, Block 2, Paragraph 3, Scoring Unit 2 SU23_Ave 30 1.53 0.81 -0.1



Unit

HS6_Ave

30 0.97 0.13 0.22

Item Total 4.92

tot_sight1 41.27

A 1, Option A, Scoring Unit 1 @1ASU1_Ave 30 0.82 0.5 0.43


A 1, Option A, Holistic Scoring Unit @1AHS_Ave 30 0.83 0.36 0.5

A 1, Option B, Scoring Unit 1 @1BSU1_Ave 30 1.1 0.67 0.28


A 1, Option B, Holistic Scoring Unit @1BHS_Ave 30 0.87 0.26 0.21

A 1, Option C, Scoring Unit 1 @1CSU1_Ave 30 1.1 0.59 0.15


A 1, Option C, Holistic Scoring Unit @1CHS_Ave 30 0.73 0.34 0.28

A 1, Option D, Scoring Unit 1 @1DSU1_Ave 30 1.28 0.69 0.27


A 1, Option D, Holistic Scoring Unit @1DHS_Ave 30 0.78 0.39 0.44

Item Total 11.87

B 2, Option A, Scoring Unit 1 2ASU1 32 1.72 0.63 0.53


B 2, Option A, Holistic Scoring Unit 2AHS 32 0.84 0.37 0.19

B 2, Option B, Scoring Unit 1 2BSU1 32 0.97 0.69 0.45


B 2, Option B, Holistic Scoring Unit 2BHS 32 0.84 0.37 0.03

B 2, Option C, Scoring Unit 1 2CSU1 32 1.53 0.72 0.23


B 2, Option C, Holistic Scoring Unit 2CHS 32 0.94 0.25 0.33

B 2, Option D, Scoring Unit 1 2DSU1 32 1.56 0.72 0.27



Original

Variable

Name N Mean SD

Item total

Correlation

B 2, Option D, Holistic Scoring Unit 2DHS 32 0.91 0.3 0.14

Item Total 14.53








A 4, Option C, Scoring Unit 2 @4CSU2_Ave 30 0.97 1 0.26





Item Total 13.67

B 9, Option A, Scoring Unit 1 9asu1 32 1.5 0.72 0.65


B 9, Option A, Holistic Scoring Unit 9ahs 32 0.44 0.5 0.13

B 9, Option B, Scoring Unit 1 9bsu1 32 1.41 0.87 0.24


B 9, Option B, Holistic Scoring Unit 9bhs 32 0.53 0.51 0.26

B 9, Option C, Scoring Unit 1 9csu1 32 1.59 0.61 0.23


B 9, Option C, Holistic Scoring Unit 9chs 32 0.84 0.37 0.32

B 9, Option D, Scoring Unit 1 9dsu1 32 1.13 0.91 0.24


B 9, Option D, Holistic Scoring Unit 9dhs 32 0.59 0.5 0.1

Item Total 11.34







Original

Variable

Name N Mean SD

Item total

Correlation


B 15, Option C, Scoring Unit 1 15csu1 32 1 0.98 0.26






Item Total 11.69












B 4, Option D, Holistic Scoring Unit 4DHS 32 0.94 0.25 -0.09

Item Total 14.84














Original

Variable

Name N Mean SD

Item total

Correlation

Item Total 15.5













Item Total 13.75

A 7, Option A, Scoring Unit 1 @7asu1_Ave 30 0.75 0.85 0.5


A 7, Option A, Holistic Scoring Unit @7ahs_Ave 30 0.72 0.41 0.38

A 7, Option B, Scoring Unit 1 @7bsu1_Ave 30 1.73 0.41 0.08


A 7, Option B, Holistic Scoring Unit @7bhs_Ave 30 0.82 0.31 0.64

A 7, Option C, Scoring Unit 1 @7csu1_Ave 30 1.67 0.58 0.42


A 7, Option C, Holistic Scoring Unit @7chs_Ave 30 0.75 0.39 0.72

A 7, Option D, Scoring Unit 1 @7dsu1_Ave 30 1.35 0.57 0.37


A 7, Option D, Holistic Scoring Unit @7dhs_Ave 30 0.62 0.47 0.52

Item Total 12.27






B 8, Option B, Holistic Scoring Unit 8bhs 32 0.81 0.4 0


Original

Variable

Name N Mean SD

Item total

Correlation







Item Total 14.63









A 8, Option C, Holistic Scoring Unit @8chs_Ave 30 0.73 0.41 -0.03




Item Total 14.4













Item Total 13.41


Original

Variable

Name N Mean SD

Item total

Correlation

tot_oral1 161.89


Table C-4. Mandarin New Form A Item Statistics

Original Variable N Mean SD

Item total

Correlation






Unit

HS1_Ave

31 0.9 0.27 0.45

Item Total 7.65






Unit

HS2_Ave

31 0.92 0.23 0.29

Item Total 7.06






Unit

HS3_Ave

31 0.97 0.12 0.35

Item Total 7.34






Unit

HS4_Ave

31 0.84 0.3 0.54

Item Total 7.47








Item total

Correlation


Unit

HS5_Ave

31 0.95 0.2 0.53

Item Total 10.18





Unit

HS6_Ave

31 1 0 .(a)

Item Total 6.55

tot_sight1 46.24










A 1, Option D, Scoring Unit 1 @1DSU1_Ave 31 2 0 .(a)



Item Total 16.5











B 13, Option D, Scoring Unit 2 13dsu2 32 1.88 0.49 0



Item total

Correlation


Item Total 16










A 3, Option D, Scoring Unit 1 @3DSU1_Ave 31 1.97 0.18 -0.01



Item Total 17.06







B 8, Option C, Scoring Unit 1 8csu1 32 2 0 .(a)




B 8, Option D, Scoring Unit 2 8dsu2 32 1.88 0.49 -0.05


Item Total 16.06





A 5, Option B, Scoring Unit 2 @5BSU2_Ave 31 1.94 0.36 -0.11



Item total

Correlation








Item Total 16.53













Item Total 16.13















Item total

Correlation

Item Total 15.06













Item Total 14.94



A 12, Option A, Holistic Scoring Unit @12ahs_Ave 31 0.94 0.21 -0.09








A 12, Option D, Scoring Unit 2 @12dsu2_Ave 31 2 0 .(a)


Item Total 17.85









Item total

Correlation

A 13, Option C, Scoring Unit 1 @13csu1_Ave 31 2 0 .(a)





A 13, Option D, Holistic Scoring Unit @13dhs_Ave 31 0.92 0.23 -0.1

Item Total 15.1







B 4, Option C, Scoring Unit 1 4CSU1 32 0.97 1 0.31




B 4, Option D, Scoring Unit 2 4DSU2 32 2 0 .(a)

B 4, Option D, Holistic Scoring Unit 4DHS 32 1 0 .(a)

Item Total 14.16













Item Total 15.95



Item total

Correlation

tot_oral1 191.35


Table C-5. Mandarin New Form B Item Statistics

Original

Variable

Name N Mean SD

Item total

Correlation


Form B, Block 1, Paragraph 1, Scoring Unit 2 SU2 32 2 0 .(a)




Unit

HS1

32 0.94 0.25 0.59

Item Total 8.22






Unit

HS2

32 0.78 0.42 0.62

Item Total 7.72

Form B, Block 1, Paragraph 3, Scoring Unit 1 SU9 32 1 0.95 0.45





Unit

HS3

32 0.94 0.25 0.5

Item Total 7.5



Form B, Block 2, Paragraph 1, Scoring Unit 3 SU15 32 1.09 1 0.17



Unit

HS4

32 0.94 0.25 0.28

Item Total 7







Original

Variable

Name N Mean SD

Item total

Correlation


Unit

HS5

32 0.91 0.3 0.68

Item Total 9.03





Unit

HS6

32 0.94 0.25 0.58

Item Total 6.53

tot_sight1 46













Item Total 14.94







B 2, Option C, Scoring Unit 1 2CSU1 32 1.84 0.51 -0.07






Original

Variable

Name N Mean SD

Item total

Correlation


Item Total 16.56




B 3, Option B, Scoring Unit 1 3BSU1 32 2 0 .(a)









Item Total 16.84







B 4, Option C, Scoring Unit 1 4CSU1 32 0.97 1 0.31





B 4, Option D, Holistic Scoring Unit 4DHS 32 1 0 .(a)

Item Total 14.16


B 5, Option A, Scoring Unit 2 5ASU2 32 1.56 0.84 -0.03





Original

Variable

Name N Mean SD

Item total

Correlation


B 5, Option C, Scoring Unit 1 5CSU1 32 2 0 .(a)






Item Total 17.25













Item Total 16.72


B 7, Option A, Scoring Unit 2 7asu2 32 1.94 0.35 -0.1










B 7, Option D, Holistic Scoring Unit 7dhs 32 1 0 .(a)


Original

Variable

Name N Mean SD

Item total

Correlation

Item Total 16.5













Item Total 15.06













Item Total 15.1








Original

Variable

Name N Mean SD

Item total

Correlation




B 11, Option D, Scoring Unit 1 11dsu1 32 2 0 .(a)



Item Total 16.13












B 14, Option D, Holistic Scoring Unit 14dhs 32 0.97 0.18 -0.11

Item Total 16.75





B 15, Option B, Scoring Unit 2 15bsu2 32 1.69 0.59 -0.02








Item Total 16.12


Original

Variable

Name N Mean SD

Item total

Correlation

tot_oral1 192.13


Table C-6. Mandarin New Form C Item Statistics

Original

Variable

Name N Mean SD

Item total

Correlation






Unit

HS1_Ave

31 0.9 0.27 0.45

Item Total 7.65






Unit

HS2_Ave

31 0.92 0.23 0.29

Item Total 7.06






Unit

HS3_Ave

31 0.97 0.12 0.35

Item Total 7.34



Form B, Block 2, Paragraph 1, Scoring Unit 3 SU15 32 1.09 1 0.17



Unit

HS4

32 0.94 0.25 0.28

Item Total 7







Original

Variable

Name N Mean SD

Item total

Correlation


Unit

HS5

32 0.91 0.3 0.68

Item Total 9.03





Unit

HS6

32 0.94 0.25 0.58

Item Total 6.53

tot_sight1 44.61













Item Total 16.56













Original

Variable

Name N Mean SD

Item total

Correlation


Item Total 16.5




B 3, Option B, Scoring Unit 1 3BSU1 32 2 0 .(a)









Item Total 16.84













Item Total 15.1







Original

Variable

Name N Mean SD

Item total

Correlation






B 8, Option D, Scoring Unit 2 8dsu2 32 1.88 0.49 -0.05


Item Total 16.06





A 5, Option B, Scoring Unit 2 @5BSU2_Ave 31 1.94 0.36 -0.11








Item Total 16.53














Original

Variable

Name N Mean SD

Item total

Correlation

Item Total 16.72













Item Total 16.13


B 7, Option A, Scoring Unit 2 7asu2 32 1.94 0.35 -0.1











Item Total 16.5








Original

Variable

Name N Mean SD

Item total

Correlation







Item Total 14.94











B 13, Option D, Scoring Unit 2 13dsu2 32 1.88 0.49 0


Item Total 16













Item Total 15.95


Original

Variable

Name N Mean SD

Item total

Correlation

tot_oral1 193.83


Table C-7. Cantonese New Form A Item Statistics

Original Variable Name N Mean SD

Item total

Correlation





Form A, Block 1, Paragraph 1, Holistic Scoring Unit HS1_Ave 32 0.797 0.3074 0.650

Item Total 6.91






Item Total 5.73






Item Total 6.02






Item Total 6.34

Form A, Block 2, Paragraph 2, Scoring Unit 1 SU17_Ave 32 2.000 0.0000 .(a)






Item Total 8.42






Item total

Correlation


Item Total 6.11

tot_sight1 39.53 17.38













Item Total 14.20













Item Total 14.63

A 4, Option A, Scoring Unit 1 @4ASU1_Ave 32 1.094 0.9284 -0.012







Item total

Correlation








Item Total 11.63













Item Total 13.70













Item Total 13.66

B 13, Option A, Scoring Unit 1 @13asu1_Ave 30 1.53 0.63 0.34



Item total

Correlation


B 13, Option A, Holistic Scoring Unit @13ahs_Ave 30 0.87 0.35 0.39

B 13, Option B, Scoring Unit 1 @13bsu1_Ave 30 0.80 0.96 0.13


B 13, Option B, Holistic Scoring Unit @13bhs_Ave 30 0.53 0.51 0.11

B 13, Option C, Scoring Unit 1 @13csu1_Ave 30 1.33 0.66 0.38


B 13, Option C, Holistic Scoring Unit @13chs_Ave 30 0.97 0.18 0.22

B 13, Option D, Scoring Unit 1 @13dsu1_Ave 30 0.87 0.90 0.28


B 13, Option D, Holistic Scoring Unit @13dhs_Ave 30 0.53 0.51 0.32

Item Total 11.63













Item Total 12.23












Item total

Correlation




Item Total 15.58











A 12, Option D, Scoring Unit 2 @12dsu2_Ave 32 1.109 0.8399 -0.012


Item Total 13.92













Item Total 10.63








Item total

Correlation








Item Total 11.11













Item Total 12.67

tot_oral1 155.59 144


Table C-8. Cantonese New Form B Item Statistics


Item total

Correlation





Form B, Block 1, Paragraph 1, Holistic Scoring Unit HS1 30 0.87 0.35 0.34

Item Total 7.13






Item Total 5.77






Item Total 7.27






Item Total 5.67







Item Total 7.47





Item total

Correlation



Item Total 5.83

tot_sight1 39.13

B 1, Option A, Scoring Unit 1 @1ASU1_Ave 30 0.80 0.85 0.38


B 1, Option A, Holistic Scoring Unit @1AHS_Ave 30 0.50 0.51 0.20

B 1, Option B, Scoring Unit 1 @1BSU1_Ave 30 1.20 0.71 0.17


B 1, Option B, Holistic Scoring Unit @1BHS_Ave 30 1.00 0.00 .(a)

B 1, Option C, Scoring Unit 1 @1CSU1_Ave 30 0.30 0.65 0.34


B 1, Option C, Holistic Scoring Unit @1CHS_Ave 30 0.57 0.50 0.23

B 1, Option D, Scoring Unit 1 @1DSU1_Ave 30 1.57 0.77 0.46


B 1, Option D, Holistic Scoring Unit @1DHS_Ave 30 0.77 0.43 0.38

Item Total 11.67

B 2, Option A, Scoring Unit 1 @2ASU1_Ave 30 0.00 0.00 .(a)





B 2, Option B, Holistic Scoring Unit @2BHS_Ave 30 0.87 0.35 0.16







Item Total 13.67






Item total

Correlation






B 3, Option C, Holistic Scoring Unit @3CHS_Ave 30 1.00 0.00 .(a)


B 3, Option D, Scoring Unit 2 @3DSU2_Ave 30 1.93 0.37 -0.09


Item Total 14.20











B 6, Option D, Scoring Unit 2 @6DSU2_Ave 30 0.87 0.68 -0.02


Item Total 14.10














Item total

Correlation


Item Total 11.90



B 11, Option A, Holistic Scoring Unit @11ahs_Ave 30 0.80 0.41 -0.12




B 11, Option C, Scoring Unit 1 @11csu1_Ave 30 1.30 0.79 -0.01






Item Total 13.00


B 9, Option A, Scoring Unit 2 @9asu2_Ave 30 0.87 0.43 -0.02




B 9, Option B, Holistic Scoring Unit @9bhs_Ave 30 0.77 0.43 -0.05







Item Total 12.27









Item total

Correlation


A 9, Option C, Scoring Unit 2 @9csu2_Ave 32 1.297 0.4554 -0.116





Item Total 12.53













Item Total 13.43






B 15, Option B, Holistic Scoring Unit @15bhs_Ave 30 0.83 0.38 -0.08







Item Total 15.37




Item total

Correlation












Item Total 13.66













Item Total 10.63

tot_oral1 156.41 141


Table C-9. Cantonese New Form C Item Statistics


Item total

Correlation






Item Total 6.91






Item Total 5.73






Item Total 6.02






Item Total 5.67







Item Total 7.47






Item total

Correlation


Item Total 5.83

tot_sight1 37.62













Item Total 11.11






B 1, Option B, Holistic Scoring Unit @1BHS_Ave 30 1.00 0.00 .(a)







Item Total 11.67








Item total

Correlation








Item Total 14.20

B 2, Option A, Scoring Unit 1 @2ASU1_Ave 30 0.00 0.00 .(a)












Item Total 13.67













Item Total 13.66




Item total

Correlation












Item Total 14.91

A 4, Option A, Scoring Unit 1 @4ASU1_Ave 32 1.094 0.9284 -0.012












Item Total 11.63













Item total

Correlation



Item Total 11.63













Item Total 13.70













Item Total 14.63









Item total

Correlation





A 12, Option D, Scoring Unit 2 @12dsu2_Ave 32 1.109 0.8399 -0.012


Item Total 13.92













Item Total 13.43

tot_oral1 158.15 142


Table C-10. Korean New Form A Item Statistics


Item total

Correlation






Item Total 6.62






Item Total 6.02






Item Total 5.80






Item Total 5.08







Item Total 7.97





Item total

Correlation



Item Total 5.48

tot_sight1 36.97

B 1, Option A, Scoring Unit 1 @1ASU1_Ave 30 .20 .610 0.5


B 1, Option A, Holistic Scoring Unit @1AHS_Ave 30 .10 .305 .133

B 1, Option B, Scoring Unit 1 @1BSU1_Ave 30 1.30 .952 0.448


B 1, Option B, Holistic Scoring Unit @1BHS_Ave 30 .83 .379 .251

B 1, Option C, Scoring Unit 1 @1CSU1_Ave 30 .00 .000 .a

B 1, Option C, Scoring Unit 2 @1CSU2_Ave 30 .80 .997 0.393

B 1, Option C, Holistic Scoring Unit @1CHS_Ave 30 .57 .568 .149

B 1, Option D, Scoring Unit 1 @1DSU1_Ave 30 .80 .847 0.412

B 1, Option D, Scoring Unit 2 @1DSU2_Ave 30 1.67 .758 .126

B 1, Option D, Holistic Scoring Unit @1DHS_Ave 30 .87 .346 .099

Item Total 9.03

A 2, Option A, Scoring Unit 1 @2ASU1_Ave 30 1.567 .8172 0.358

A 2, Option A, Scoring Unit 2 @2ASU2_Ave 30 1.633 .7184 .244

A 2, Option A, Holistic Scoring Unit @2AHS_Ave 30 .767 .4097 .197

A 2, Option B, Scoring Unit 1 @2BSU1_Ave 30 .283 .6254 0.529

A 2, Option B, Scoring Unit 2 @2BSU2_Ave 30 1.233 .8880 0.741

A 2, Option B, Holistic Scoring Unit @2BHS_Ave 30 .533 .4536 .251

A 2, Option C, Scoring Unit 1 @2CSU1_Ave 30 1.200 .9523 0.51

A 2, Option C, Scoring Unit 2 @2CSU2_Ave 30 .800 .7381 0.453

A 2, Option C, Holistic Scoring Unit @2CHS_Ave 30 .867 .3198 .191

A 2, Option D, Scoring Unit 1 @2DSU1_Ave 30 1.183 .9513 0.554

A 2, Option D, Scoring Unit 2 @2DSU2_Ave 30 1.600 .7701 .275

A 2, Option D, Holistic Scoring Unit @2DHS_Ave 30 .617 .4857 0.556

Item Total 12.28

A 3, Option A, Scoring Unit 1 @3ASU1_Ave 30 .933 .9444 0.525


A 3, Option A, Holistic Scoring Unit @3AHS_Ave 30 .733 .4302 0.331



Item total

Correlation


A 3, Option B, Scoring Unit 2 @3BSU2_Ave 30 1.150 .5111 .304







A 3, Option D, Holistic Scoring Unit @3DHS_Ave 30 .783 .3640 .276

Item Total 13.55






A 5, Option B, Holistic Scoring Unit @5BHS_Ave 30 .883 .2842 0.52

A 5, Option C, Scoring Unit 1 @5CSU1_Ave 30 1.850 .5111 .067


A 5, Option C, Holistic Scoring Unit @5CHS_Ave 30 .450 .4614 0.604


A 5, Option D, Scoring Unit 2 @5DSU2_Ave 30 .833 .9855 0.375


Item Total 12.77




A 6, Option B, Scoring Unit 1 @6BSU1_Ave 30 .700 .6644 .122










Item total

Correlation


Item Total 12.32

B 7, Option A, Scoring Unit 1 @7asu1_Ave 30 1.87 .507 .037


B 7, Option A, Holistic Scoring Unit @7ahs_Ave 30 .70 .466 .256

B 7, Option B, Scoring Unit 1 @7bsu1_Ave 30 1.73 .640 .260


B 7, Option B, Holistic Scoring Unit @7bhs_Ave 30 .57 .504 .250

B 7, Option C, Scoring Unit 1 @7csu1_Ave 30 1.70 .651 0.341


B 7, Option C, Holistic Scoring Unit @7chs_Ave 30 .33 .479 -.027

B 7, Option D, Scoring Unit 1 @7dsu1_Ave 30 1.93 .254 .037


B 7, Option D, Holistic Scoring Unit @7dhs_Ave 30 .57 .504 0.546

Item Total 14.33

A 8, Option A, Scoring Unit 1 @8asu1_Ave 30 .883 .6254 .145

A 8, Option A, Scoring Unit 2 @8asu2_Ave 30 1.450 .8939 .007

A 8, Option A, Holistic Scoring Unit @8ahs_Ave 30 .667 .4420 0.441

A 8, Option B, Scoring Unit 1 @8bsu1_Ave 30 1.400 .8030 0.338


A 8, Option B, Holistic Scoring Unit @8bhs_Ave 30 .850 .2980 0.44

A 8, Option C, Scoring Unit 1 @8csu1_Ave 30 1.350 .8625 0.709

A 8, Option C, Scoring Unit 2 @8csu2_Ave 30 .517 .7484 0.473

A 8, Option C, Holistic Scoring Unit @8chs_Ave 30 .583 .4749 0.496

A 8, Option D, Scoring Unit 1 @8dsu1_Ave 30 .900 .7701 0.463

A 8, Option D, Scoring Unit 2 @8dsu2_Ave 30 1.583 .6170 0.382

A 8, Option D, Holistic Scoring Unit @8dhs_Ave 30 .900 .2421 .061

Item Total 12.72


A 9, Option A, Scoring Unit 2 @9asu2_Ave 30 1.300 .7144 0.348

A 9, Option A, Holistic Scoring Unit @9ahs_Ave 30 .817 .3075 .211

A 9, Option B, Scoring Unit 1 @9bsu1_Ave 30 .883 .9067 0.483





Item total

Correlation





A 9, Option D, Scoring Unit 2 @9dsu2_Ave 30 .683 .7008 .167

A 9, Option D, Holistic Scoring Unit @9dhs_Ave 30 .683 .4251 0.314

Item Total 11.17





A 11, Option B, Scoring Unit 2 @11bsu2_Ave 30 1.417 .7081 .092

A 11, Option B, Holistic Scoring Unit @11bhs_Ave 30 .917 .2653 .299


A 11, Option C, Scoring Unit 2 @11csu2_Ave 30 1.500 .8610 .264

A 11, Option C, Holistic Scoring Unit @11chs_Ave 30 .550 .4224 .155


A 11, Option D, Scoring Unit 2 @11dsu2_Ave 30 .867 1.0080 0.495


Item Total 12.55











A 12, Option D, Scoring Unit 2 @12dsu2_Ave 30 1.783 .5522 .243


Item Total 16.33




Item total

Correlation

A 13, Option A, Scoring Unit 2 @13asu2_Ave 30 .900 .8550 0.401



A 13, Option B, Scoring Unit 2 @13bsu2_Ave 30 .567 .8277 .302








Item Total 10.85













Item Total 12.62

tot_oral1 150.52 143


Table C-11. Korean New Form B Item Statistics


Item total

Correlation

Form B, Block 1, Paragraph 1, Scoring Unit 1 SU1 30 1.73 .691 .072


Form B, Block 1, Paragraph 1, Scoring Unit 3 SU3 30 1.30 .837 0.348


Form B, Block 1, Paragraph 1, Holistic Scoring Unit HS1 30 .40 .498 0.451

Item Total 6.87




Form B, Block 1, Paragraph 2, Scoring Unit 4 SU8 30 .80 .805 0.528


Item Total 5.10






Item Total 6.07



Form B, Block 2, Paragraph 1, Scoring Unit 3 SU15 30 1.00 1.017 .288



Item Total 4.63







Item Total 4.93





Item total

Correlation



Item Total 4.83

tot_sight1 32.43







B 1, Option C, Scoring Unit 1 @1CSU1_Ave 30 .00 .000 .a






Item Total 9.03

B 3, Option A, Scoring Unit 1 @3ASU1_Ave 30 .60 .894 .282

B 3, Option A, Scoring Unit 2 @3ASU2_Ave 30 1.87 .507 .010

B 3, Option A, Holistic Scoring Unit @3AHS_Ave 30 .77 .430 0.474

B 3, Option B, Scoring Unit 1 @3BSU1_Ave 30 1.67 .758 .242

B 3, Option B, Scoring Unit 2 @3BSU2_Ave 30 .37 .765 0.485


B 3, Option C, Scoring Unit 1 @3CSU1_Ave 30 1.63 .490 .058

B 3, Option C, Scoring Unit 2 @3CSU2_Ave 30 2.00 .000 .a

B 3, Option C, Holistic Scoring Unit @3CHS_Ave 30 1.00 .000 .a

B 3, Option D, Scoring Unit 1 @3DSU1_Ave 30 1.67 .711 0.332


B 3, Option D, Holistic Scoring Unit @3DHS_Ave 30 .27 .450 0.418

Item Total 13.83

B 5, Option A, Scoring Unit 1 @5ASU1_Ave 30 1.73 .521 0.48





Item total

Correlation

B 5, Option B, Scoring Unit 1 @5BSU1_Ave 30 .77 .774 .243




B 5, Option C, Scoring Unit 2 @5CSU2_Ave 30 1.60 .770 0.359

B 5, Option C, Holistic Scoring Unit @5CHS_Ave 30 .70 .466 0.392




Item Total 11.93

B 15, Option A, Scoring Unit 1 @15asu1_Ave 30 .67 .922 0.485


B 15, Option A, Holistic Scoring Unit @15ahs_Ave 30 .67 .479 0.379


B 15, Option B, Scoring Unit 2 @15bsu2_Ave 30 1.77 .568 0.381

B 15, Option B, Holistic Scoring Unit @15bhs_Ave 30 .33 .479 0.316



B 15, Option C, Holistic Scoring Unit @15chs_Ave 30 .83 .379 .108


B 15, Option D, Scoring Unit 2 @15dsu2_Ave 30 1.07 .785 0.387


Item Total 12.73














Item total

Correlation


Item Total 12.28













Item Total 13.55




B 12, Option B, Scoring Unit 1 @12bsu1_Ave 30 .70 .535 -.052



B 12, Option C, Scoring Unit 1 @12csu1_Ave 30 .80 .997 .250

B 12, Option C, Scoring Unit 2 @12csu2_Ave 30 .50 .820 0.495

B 12, Option C, Holistic Scoring Unit @12chs_Ave 30 .57 .504 0.309

B 12, Option D, Scoring Unit 1 @12dsu1_Ave 30 2.00 .000 .a


B 12, Option D, Holistic Scoring Unit @12dhs_Ave 30 .83 .379 .222

Item Total 13.53









Item total

Correlation







Item Total 13.63













Item Total 12.72











A 11, Option D, Scoring Unit 2 @11dsu2_Ave 30 .867 1.0080 0.495


Item Total 12.55




Item total

Correlation












Item Total 16.33

B 11, Option A, Scoring Unit 1 @11asu1_Ave 30 .70 .535 .048

B 11, Option A, Scoring Unit 2 @11asu2_Ave 30 1.40 .932 0.396








B 11, Option D, Scoring Unit 1 @11dsu1_Ave 30 1.80 .610 -.050



Item Total 12.83

tot_oral1 154.97 140


Table C-12. Korean New Form C Item Statistics


Item total

Correlation






Item Total 6.87






Item Total 5.10






Item Total 6.07






Item Total 5.08







Item Total 7.97





Item total

Correlation



Item Total 5.48

tot_sight1 36.57




B 12, Option B, Scoring Unit 1 @12bsu1_Ave 30 .70 .535 -.052






B 12, Option D, Scoring Unit 1 @12dsu1_Ave 30 2.00 .000 .a



Item Total 13.53

A 1, Option A, Scoring Unit 1 @1ASU1_Ave 30 .150 .3256 .304












Item Total 8.35






Item total

Correlation










Item Total 13.55








B 5, Option C, Scoring Unit 2 @5CSU2_Ave 30 1.60 .770 0.359





Item Total 11.93














Item total

Correlation


Item Total 12.83













Item Total 12.73













Item Total 13.63









Item total

Correlation







Item Total 12.72

B 3, Option A, Scoring Unit 1 @3ASU1_Ave 30 .60 .894 .282







B 3, Option C, Scoring Unit 2 @3CSU2_Ave 30 2.00 .000 .a

B 3, Option C, Holistic Scoring Unit @3CHS_Ave 30 1.00 .000 .a




Item Total 13.83













Item Total 12.62




Item total

Correlation








B 7, Option C, Holistic Scoring Unit @7chs_Ave 30 .33 .479 -.027




Item Total 14.33













Item Total 10.85

tot_oral1 150.92 141


Table C-13. Vietnamese New Form A Item Statistics


Item total

Correlation

Form A, Block 1, Paragraph 1, Scoring Unit 1 SU1_Ave 31 .71 .602 0.486

Form A, Block 1, Paragraph 1, Scoring Unit 2 SU2_Ave 31 1.68 .653 0.595



Form A, Block 1, Paragraph 1, Holistic Scoring Unit HS1_Ave 31 .61 .460 0.309

Item Total 5.11

Form A, Block 1, Paragraph 2, Scoring Unit 1 SU5_Ave 31 .82 .640 .226


Form A, Block 1, Paragraph 2, Scoring Unit 3 SU7_Ave 31 1.24 .575 .268



Item Total 4.06






Item Total 4.40






Item Total 4.50







Item Total 7.47





Item total

Correlation



Item Total 4.27

tot_sight1 29.82

A 2, Option A, Scoring Unit 1 @2ASU1_Ave 31 1.85 .469 -.049












Item Total 10.47













Item Total 9.03






Item total

Correlation





A 5, Option C, Scoring Unit 2 @5CSU2_Ave 31 .82 .737 .224


A 5, Option D, Scoring Unit 1 @5DSU1_Ave 31 1.19 .511 -.012



Item Total 10.26

B 6, Option A, Scoring Unit 1 @6ASU1_Ave 30 1.17 .461 -.062












Item Total 13.53














Item total

Correlation


Item Total 10.55







A 9, Option C, Scoring Unit 1 @9csu1_Ave 31 .29 .588 .199






Item Total 9.66













Item Total 13.58



A 14, Option A, Holistic Scoring Unit @14ahs_Ave 31 .85 .321 -.036






Item total

Correlation







Item Total 12.53

A 10, Option A, Scoring Unit 1 @10asu1_Ave 31 1.52 .851 -.023












Item Total 10.31













Item Total 12.03




Item total

Correlation




B 7, Option B, Scoring Unit 2 @7bsu2_Ave 30 .90 .662 .173


B 7, Option C, Scoring Unit 1 @7csu1_Ave 30 1.43 .568 .195




B 7, Option D, Scoring Unit 2 @7dsu2_Ave 30 .77 .858 .036


Item Total 13.67





B 9, Option B, Scoring Unit 2 @9bsu2_Ave 30 .60 .932 0.361








Item Total 11.73

tot_oral1 137.35


Table C-14. Vietnamese New Form B Item Statistics


Item total

Correlation





Form B, Block 1, Paragraph 1, Holistic Scoring Unit HS1 30 .30 .466 -.013

Item Total 5.13



Form B, Block 1, Paragraph 2, Scoring Unit 3 SU7 30 .93 .868 .259


Form B, Block 1, Paragraph 2, Holistic Scoring Unit HS2 30 .37 .490 .181

Item Total 5.23






Item Total 4.47






Item Total 4.63







Item Total 5.60





Item total

Correlation



Item Total 4.37

tot_sight1 29.43













Item Total 13.77













Item Total 10.55






Item total

Correlation










Item Total 12.03


B 2, Option A, Scoring Unit 2 @2ASU2_Ave 30 .77 .971 -.096






B 2, Option C, Scoring Unit 2 @2CSU2_Ave 30 .50 .777 .269





Item Total 9.30














Item total

Correlation


Item Total 13.67



B 8, Option A, Holistic Scoring Unit @8ahs_Ave 30 .17 .379 -.014










Item Total 10.43

B 6, Option A, Scoring Unit 1 @6ASU1_Ave 30 1.17 .461 -.062












Item Total 13.53









Item total

Correlation







Item Total 9.79













Item Total 9.66













Item Total 13.58




Item total

Correlation












Item Total 12.53













Item Total 9.03

tot_oral1 137.88


Table C-15. Vietnamese New Form C Item Statistics


Item total

Correlation






Item Total 5.11






Item Total 4.06






Item Total 4.40






Item Total 4.63







Item Total 5.60





Item total

Correlation



Item Total 4.37

tot_sight1 28.18








A 5, Option C, Scoring Unit 2 @5CSU2_Ave 31 .82 .737 .224


A 5, Option D, Scoring Unit 1 @5DSU1_Ave 31 1.19 .511 -.012



Item Total 10.26



B 8, Option A, Holistic Scoring Unit @8ahs_Ave 30 .17 .379 -.014










Item Total 10.43






Item total

Correlation










Item Total 10.55













Item Total 12.03

A 2, Option A, Scoring Unit 1 @2ASU1_Ave 31 1.85 .469 -.049













Item total

Correlation


Item Total 10.47













Item Total 10.31













Item Total 9.66









Item total

Correlation







Item Total 13.67













Item Total 13.58













Item Total 11.73




Item total

Correlation












Item Total 12.53













Item Total 13.77

tot_oral1 138.99

©2013, PSI Services LLC. All rights reserved. D - 1

Appendix D. Angoff Ratings

Table D-1. Russian New Form Sight Test Angoff Ratings

Russian Sight Test, Form A

Original

Item

Number

Angoff (without key - source

language)

Acceptable

(i.e., important) Relevance

Angoff

(with key - target language)

Average Angoffs

(of with and without key)

R1 R2 R3 R4 Averages

Yes

(/)

No

(/)

OK as

Edited

(/) R1 R2 R3 R4 Averages R1 R2 R3 R4 Averages R1 R2 R3 R4 Averages

Form A, Block 1

1 95 95 95 100 96.3 //// 4 4 3 4 3.75 90 90 95 90 91.3 45.5 90 95 92.5 80.75

2 90 90 90 90 90 //// 3 4 3 3 3.25 80 85 90 85 85 41 85 90 87.5 75.875

3 80 90 90 90 87.5 //// 4 4 4 4 4 85 85 85 85 85 44 85 82.5 87.5 74.75

4 80 80 80 85 81.3 //// 4 4 4 4 4 80 80 80 80 80 42 80 80 80 70.5

5 80 80 80 90 80 //// 4 4 4 4 4 80 80 85 80 81.3 42.5 80 82.5 80 71.25

6 85 85 95 90 88.8 //// 3 4 4 3 3.5 85 90 95 90 90 45.5 90 90 87.5 78.25

7 85 80 85 80 82.5 //// 3 3 3 3 3 80 75 80 70 76.3 43.5 75 82.5 75 69

8 85 80 85 85 83.8 //// 4 4 3 3 3.5 70 75 80 80 76.3 39 75 82.5 80 69.125

9 80 80 85 90 83.7 //// 4 3 4 4 3.75 80 80 85 80 81.3 44.5 80 82.5 80 71.75

Form A, Block 2

10 85 90 85 80 85 //// 4 4 4 3 3.75 80 85 85 85 83.8 45 85 85 87.5 75.625

11 75 75 80 80 77.5 //// 4 4 4 4 4 75 80 90 80 81.3 43 80 82.5 77.5 70.75

12 85 80 90 95 85 //// 4 4 4 4 4 80 90 90 90 87.5 46 90 87.5 85 77.125

13 70 75 80 80 76.3 //// 4 4 4 4 4 70 80 80 80 77.5 41.5 80 75 77.5 68.5

14 80 85 80 80 81.3 //// 4 4 4 4 4 80 85 80 80 81.3 47 85 80 82.5 73.625

15 80 75 80 80 78.8 //// 3 4 4 4 3.75 70 70 75 70 71.3 42.5 70 77.5 72.5 65.625

16 80 80 80 85 81.3 //// 4 4 4 4 4 75 75 80 70 75 45.5 75 80 75 68.875

17 70 70 75 80 73.8 //// 4 4 4 4 4 70 75 75 80 75 43.5 75 72.5 75 66.5

18 85 80 80 85 82.5 //// 4 4 4 4 4 75 80 80 80 78.8 46.5 80 82.5 80 72.25

19 75 75 70 70 72.5 //// 4 4 4 4 4 70 70 70 70 70 44.5 70 72.5 72.5 64.875

Russian Sight Test, Form B

Original

Item

Number


language)

Acceptable


Angoff


Average Angoffs



Yes

(/)

No

(/)

OK as

Edited


Form B, Block 1

1 95 95 95 100 96.3 //// 3 3 3 3 3 95 90 95 95 93.8 48 90 95 95 82

2 80 85 85 85 83.8 //// 4 4 4 4 4 75 80 80 85 80 38.5 80 80 85 70.875

3 70 75 70 70 71.3 //// 4 4 4 3 3.75 70 75 70 70 71.3 36.5 75 70 72.5 63.5

4 80 80 80 80 80 //// 4 4 4 4 4 75 70 75 70 72.5 39.5 70 77.5 75 65.5

5 75 70 80 80 76.3 //// 4 4 4 4 4 75 75 80 85 78.8 40 75 77.5 77.5 67.5


6 65 70 70 70 68.3 //// 3 3 3 3 3 65 65 65 65 65 35.5 65 65 67.5 58.25

7 75 70 65 75 71.3 //// 3 3 3 3 3 70 75 65 70 70 38.5 75 70 70 63.375

8 90 80 80 90 85 //// 4 4 4 4 4 90 85 90 90 88.8 49 85 90 85 77.25

9 80 85 85 85 83.8 //// 3 4 4 4 3.75 70 75 75 80 75 39.5 75 77.5 82.5 68.625

Form B, Block 2

10 80 80 80 85 81.3 //// 4 4 4 4 4 70 75 75 85 76.3 40 75 77.5 82.5 68.75

11 70 75 70 80 73.8 //// 4 4 4 4 4 70 75 70 75 71.3 40.5 75 70 75 65.125

12 75 75 75 80 76.3 //// 4 4 4 4 4 70 70 75 80 73.8 41 70 75 77.5 65.875

13 75 70 75 80 75 //// 4 4 4 4 4 70 70 75 80 73.8 41.5 70 75 75 65.375

14 75 75 70 75 73.8 //// 3 4 4 4 3.75 65 70 65 70 67.5 39.5 70 70 72.5 63

15 75 75 75 80 76.3 //// 4 4 4 4 4 70 75 75 75 73.8 42.5 75 75 75 66.875

16 75 75 80 85 78.8 //// 4 4 4 4 4 75 80 80 85 80 45.5 80 77.5 80 70.75

Russian Sight Test, Form C

Original

Item

Number


language)

Acceptable


Angoff


Average Angoffs



Yes

(/)

No

(/)

OK as

Edited


Form B, Block 1

1 95 95 95 100 96.3 //// 3 3 3 3 3 95 90 95 95 93.8 48 90 95 95 82

2 80 85 85 85 83.8 //// 4 4 4 4 4 75 80 80 85 80 38.5 80 80 85 70.875

3 70 75 70 70 71.3 //// 4 4 4 3 3.75 70 75 70 70 71.3 36.5 75 70 72.5 63.5

4 80 80 80 80 80 //// 4 4 4 4 4 75 70 75 70 72.5 39.5 70 77.5 75 65.5

5 75 70 80 80 76.3 //// 4 4 4 4 4 75 75 80 85 78.8 40 75 77.5 77.5 67.5

6 65 70 70 70 68.3 //// 3 3 3 3 3 65 65 65 65 65 35.5 65 65 67.5 58.25

7 75 70 65 75 71.3 //// 3 3 3 3 3 70 75 65 70 70 38.5 75 70 70 63.375

8 90 80 80 90 85 //// 4 4 4 4 4 90 85 90 90 88.8 49 85 90 85 77.25

9 80 85 85 85 83.8 //// 3 4 4 4 3.75 70 75 75 80 75 39.5 75 77.5 82.5 68.625

Form A, Block 2

10 85 90 85 80 85 //// 4 4 4 3 3.75 80 85 85 85 83.8 45 85 85 87.5 75.625

11 75 75 80 80 77.5 //// 4 4 4 4 4 75 80 90 80 81.3 43 80 82.5 77.5 70.75

12 85 80 90 95 85 //// 4 4 4 4 4 80 90 90 90 87.5 46 90 87.5 85 77.125

13 70 75 80 80 76.3 //// 4 4 4 4 4 70 80 80 80 77.5 41.5 80 75 77.5 68.5

14 80 85 80 80 81.3 //// 4 4 4 4 4 80 85 80 80 81.3 47 85 80 82.5 73.625

15 80 75 80 80 78.8 //// 3 4 4 4 3.75 70 70 75 70 71.3 42.5 70 77.5 72.5 65.625

16 80 80 80 85 81.3 //// 4 4 4 4 4 75 75 80 70 75 45.5 75 80 75 68.875

17 70 70 75 80 73.8 //// 4 4 4 4 4 70 75 75 80 75 43.5 75 72.5 75 66.5

18 85 80 80 85 82.5 //// 4 4 4 4 4 75 80 80 80 78.8 46.5 80 82.5 80 72.25

19 75 75 70 70 72.5 //// 4 4 4 4 4 70 70 70 70 70 44.5 70 72.5 72.5 64.875


Table D-2. Russian New Form Consecutive Test Angoff Ratings Russian Consecutive Test, Form A

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating

A 1, Option A, Scoring Unit 1 @1ASU1_Ave


A 1, Option A, Holistic Scoring Unit @1AHS_Ave 77.50 3.50

A 1, Option B, Scoring Unit 1 @1BSU1_Ave


A 1, Option B, Holistic Scoring Unit @1BHS_Ave 84.38 4.00

A 1, Option C, Scoring Unit 1 @1CSU1_Ave


A 1, Option C, Holistic Scoring Unit @1CHS_Ave 83.75 3.00

A 1, Option D, Scoring Unit 1 @1DSU1_Ave


A 1, Option D, Holistic Scoring Unit @1DHS_Ave 78.13 4.00























































A 7, Option A, Scoring Unit 1 @7asu1_Ave


A 7, Option A, Holistic Scoring Unit @7ahs_Ave 76.65 4.00

A 7, Option B, Scoring Unit 1 @7bsu1_Ave


A 7, Option B, Holistic Scoring Unit @7bhs_Ave 73.35 4.00

A 7, Option C, Scoring Unit 1 @7csu1_Ave


A 7, Option C, Holistic Scoring Unit @7chs_Ave 76.65 4.00

A 7, Option D, Scoring Unit 1 @7dsu1_Ave


A 7, Option D, Holistic Scoring Unit @7dhs_Ave 75.85 4.00




































































B 1, Option A, Scoring Unit 1 1ASU1



B 1, Option A, Holistic Scoring Unit 1AHS 76.25 4.00

B 1, Option B, Scoring Unit 1 1BSU1


B 1, Option B, Holistic Scoring Unit 1BHS 85.00 4.00

B 1, Option C, Scoring Unit 1 1CSU1


B 1, Option C, Holistic Scoring Unit 1CHS 68.75 3.75

B 1, Option D, Scoring Unit 1 1DSU1


B 1, Option D, Holistic Scoring Unit 1DHS 72.50 4.00


Russian Consecutive Test, Form B

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating



























































B 5, Option B, Holistic Scoring Unit 5BHS 67.50 -





















B 7, Option A, Scoring Unit 1 7asu1


B 7, Option A, Holistic Scoring Unit 7ahs 78.75 4.00

B 7, Option B, Scoring Unit 1 7bsu1


B 7, Option B, Holistic Scoring Unit 7bhs 69.38 4.00


B 7, Option C, Scoring Unit 1 7csu1


B 7, Option C, Holistic Scoring Unit 7chs 72.50 4.00

B 7, Option D, Scoring Unit 1 7dsu1


B 7, Option D, Holistic Scoring Unit 7dhs 80.63 4.00




































































Russian Consecutive Test, Form C

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating
















































B 5, Option B, Holistic Scoring Unit 5BHS 67.50 -




























Table D-3. Mandarin New Form Sight Test Angoff Ratings

Mandarin Sight Test, Form A

Original

Item

Num

ber

Angoff (without key - source language)

Acceptable


Angoff


Average Angoffs


R1 R2 R3 R4 R5

Averag

es

Ye

s

(/)

N

o

(/)

OK

as

Edit

ed

(/)

R

1

R

2

R

3

R

4

R

5

Ave

rage

s R1 R2 R3 R4 R5

Aver

ages R1 R2 R3 R4 R5

Averag

es

Form A, Block 1

1 80 90 85 85 90 84

////

/ 4 4 4 4 4 4 85 85 80 85 85 84

82.

5

87.

5

82.

5 85

87.

5 85

2 80 80 80 80 80 80

////

/ 4 4 4 4 4 4 85 90 85 85 85 86

82.

5 85

82.

5

82.

5

82.

5 83

3 80 80 85 85 75 81

////

/ 4 4 4 4 4 4 80 80 75 80 75 78 80 80 80

82.

5 75 79.5

4 75 75 75 80 75 76

////

/ 4 4 4 4 4 4 75 75 70 80 75 75 75 75

72.

5 80 75 75.5

5 75 80 75 75 75 76 ///// 4 4 4 4 4 4 75 75 75 75 75 75 75

77.5 75 75 75 75.5

6 80 80 85 80 90 83 ///// 4 4 4 4 4 4 85 85 85 85 85 85

82.5

82.5 85

82.5

87.5 84

7 90 90 90 85 85 88

////

/ 3 4 4 4 4 3.8 90 90 85 85 85 88 90 90

87.

5 85 85 87.5

8 75 75 70 80 75 75

////

/ 4 4 4 4 4 4 90 85 80 85 80 84

82.

5 80 75

82.

5

77.

5 79.5

9 80 90 85 90 90 87

////

/ 4 4 4 4 4 4 80 80 85 90 90 85 80 85 85 90 90 86

Form A,

Block 2

10 85 85 90 85 85 86

////

/ 4 4 4 4 4 4 80 70 70 80 80 76

82.

5

77.

5 80

82.

5

82.

5 81

11 75 70 70 70 70 71

////

/ 4 4 4 4 4 4 70 70 70 70 70 70

72.

5 70 70 70 70 70.5

12 80 80 90 85 85 84

////

/ 4 4 4 4 4 4 95 95 95 90 90 93

87.

5

87.

5

92.

5

87.

5

87.

5 88.5

13 85 90 90 90 90 89

////

/ 4 4 4 4 4 4 90 95 95 90 90 92

87.

5

92.

5

92.

5 90 90 90.5

14 80 90 85 90 85 86

////

/ 4 4 4 4 4 4 85 95 85 85 85 87

82.

5

92.

5 85

87.

5 85 86.5

15 85 80 90 85 90 86

////

/ 4 4 4 4 4 4 85 90 90 85 90 88 85 85 90 85 90 87

16 85 85 80 85 85 84 ///// 4 4 4 4 4 4 90 90 85 85 85 88

87.5

87.5

82.5 85 85 85.5

17 85 85 90 85 85 86

////

/ 4 4 4 4 4 4 85 85 85 85 85 85 85 85

87.

5 85 85 85.5

18 90 80 85 85 90 86

////

/ 4 4 4 4 4 4 90 90 85 85 90 88 90 85 85 85 90 87

19 80 80 80 75 75 78

////

/ 4 4 4 4 4 4 80 85 80 75 75 79 80

82.

5 80 75 75 78.5

Mandarin Sight Test, Form B

Original

Item

Num

ber Angoff (without key - source language)

Acceptable


Angoff


Average Angoffs



R1 R2 R3 R4 R5

Averag

es

Ye

s

(/)

N

o

(/)

OK

as

Edit

ed

(/)

R

1

R

2

R

3

R

4

R

5

Ave

rage

s R1 R2 R3 R4 R5

Aver

ages R1 R2 R3 R4 R5

Averag

es

Form B,

Block 1

1 90 85 85 80 85 85

////

/ 4 4 4 4 4 4 90 95 85 90 95 91 90 90 85 85 90 88

2 85 80 85 85 85 84 ///// 4 4 4 4 4 4 85 85 80 85 85 84 85

82.5

82.5 85 85 84

3 80 75 75 80 75 77

////

/ 4 4 4 4 4 4 80 75 75 80 75 77 80 75 75 80 75 77

4 85 85 85 85 75 83

////

/ 4 4 4 4 4 4 80 85 75 85 75 80

82.

5 85 80 85 75 81.5

5 80 80 80 80 75 79

////

/ 4 4 4 4 4 4 75 75 75 80 75 76

77.

5

77.

5

77.

5 80 75 77.5

6 75 75 70 70 70 72

////

/ 4 4 4 4 4 4 75 60 70 75 65 69 75

67.

5 70

72.

5

67.

5 70.5

7 80 70 70 75 70 72

////

/ 4 4 3 3 4 3.6 80 75 70 80 70 75 80

72.

5 70

77.

5 70 74

8 75 75 80 75 80 74

////

/ 4 4 4 4 4 4 80 75 80 80 80 79

77.

5 75 80

77.

5 80 78

9 70 70 70 75 75 72

////

/ 4 4 4 4 4 4 70 70 70 75 75 72 70 70 70 75 75 72

Form B, Block 2

10 80 70 75 75 75 75

////

/ 4 4 4 4 4 4 75 70 75 70 70 72

77.

5 70 75

72.

5

72.

5 73.5

11 70 60 65 65 65 65

////

/ 4 4 4 4 4 4 70 70 65 65 70 68 70 65 65 65

67.

5 66.5

12 65 65 65 65 65 65 ///// 4 4 4 4 4 4 70 70 65 70 70 69

67.5

67.5 65

67.5

67.5 67

13 65 65 70 80 75 71 ///// 4 4 4 4 4 4 70 70 70 80 75 73

67.5

67.5 70 80 75 72

14 70 65 70 70 65 68

////

/ 4 4 4 4 4 4 70 65 65 70 65 67 70 65

67.

5 70 65 67.5

15 60 60 70 65 70 65

////

/ 4 4 4 4 4 4 65 75 70 75 75 72

62.

5

67.

5 70 70

72.

5 68.5

16 80 75 80 80 80 79

////

/ 4 4 4 4 4 4 80 80 75 80 85 80 80

77.

5

77.

5 80

82.

5 79.5

Mandarin Sight Test, Form C

Original

Item

Num

ber


Acceptable


Angoff


Average Angoffs


R1 R2 R3 R4 R5

Averag

es

Ye

s

(/)

N

o

(/)

OK

as

Edit

ed

(/)

R

1

R

2

R

3

R

4

R

5

Ave

rage

s R1 R2 R3 R4 R5

Aver

ages R1 R2 R3 R4 R5

Averag

es

Form A,

Block 1

1 80 90 85 85 90 84

////

/ 4 4 4 4 4 4 85 85 80 85 85 84

82.

5

87.

5

82.

5 85

87.

5 85

2 80 80 80 80 80 80

////

/ 4 4 4 4 4 4 85 90 85 85 85 86

82.

5 85

82.

5

82.

5

82.

5 83

3 80 80 85 85 75 81

////

/ 4 4 4 4 4 4 80 80 75 80 75 78 80 80 80

82.

5 75 79.5

4 75 75 75 80 75 76

////

/ 4 4 4 4 4 4 75 75 70 80 75 75 75 75

72.

5 80 75 75.5


5 75 80 75 75 75 76

////

/ 4 4 4 4 4 4 75 75 75 75 75 75 75

77.

5 75 75 75 75.5

6 80 80 85 80 90 83

////

/ 4 4 4 4 4 4 85 85 85 85 85 85

82.

5

82.

5 85

82.

5

87.

5 84

7 90 90 90 85 85 88

////

/ 3 4 4 4 4 3.8 90 90 85 85 85 88 90 90

87.

5 85 85 87.5

8 75 75 70 80 75 75

////

/ 4 4 4 4 4 4 90 85 80 85 80 84

82.

5 80 75

82.

5

77.

5 79.5

9 80 90 85 90 90 87

////

/ 4 4 4 4 4 4 80 80 85 90 90 85 80 85 85 90 90 86

Form B,

Block 2

10 80 70 75 75 75 75

////

/ 4 4 4 4 4 4 75 70 75 70 70 72

77.

5 70 75

72.

5

72.

5 73.5

11 70 60 65 65 65 65 ///// 4 4 4 4 4 4 70 70 65 65 70 68 70 65 65 65

67.5 66.5

12 65 65 65 65 65 65

////

/ 4 4 4 4 4 4 70 70 65 70 70 69

67.

5

67.

5 65

67.

5

67.

5 67

13 65 65 70 80 75 71

////

/ 4 4 4 4 4 4 70 70 70 80 75 73

67.

5

67.

5 70 80 75 72

14 70 65 70 70 65 68

////

/ 4 4 4 4 4 4 70 65 65 70 65 67 70 65

67.

5 70 65 67.5

15 60 60 70 65 70 65

////

/ 4 4 4 4 4 4 65 75 70 75 75 72

62.

5

67.

5 70 70

72.

5 68.5

16 80 75 80 80 80 79

////

/ 4 4 4 4 4 4 80 80 75 80 85 80 80

77.

5

77.

5 80

82.

5 79.5


Table D-4. Mandarin New Form Consecutive Test Angoff Ratings

Mandarin Consecutive Test, Form A

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating


























































Mandarin Consecutive Test, Form B

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating


























































Mandarin Consecutive Test, Form C

Original Variable

Average

Angoff

Rating

Average

Relevance

Rating

























Table D-5. Cantonese New Form Sight Test Angoff Ratings

Cantonese Sight Test, Form A

Original Item

Number


Acceptable (i.e., important)

Relevance (use scale below)

Angoff (with key - target language)

Average Angoff (with & without key)

R1 R2 R3 R4 Averages Yes (/)

No (/)

OK as Edited

(/) R1/R2/R3/R4 Averages R1 R2 R3 R4 Averages R1 R2 R3 R4 Averages

Form A, Block 1

1 80 95 80 85 85 //// 4/4/4/4 4 90 95 90 90 91.25 85 95 85 88 88.125

2 80 90 80 75 81.25 //// 4/4/4/4 4 75 80 70 75 75 78 85 75 75 78.125

3 75 85 75 80 78.75 //// 4/4/4/4 4 85 85 75 70 78.75 80 85 75 75 78.75

4 75 75 70 75 73.75 //// 4/4/4/4 4 80 85 75 75 78.75 78 80 73 75 76.25

5 70 80 65 65 70 //// 4/4/4/4 4 65 70 65 65 66.25 68 75 65 65 68.125

6 80 90 80 85 83.75 //// 4/4/4/4 4 70 85 75 80 77.5 75 88 78 83 80.625

7 90 85 85 85 86.25 //// 4/4/4/4 4 90 85 80 85 85 90 85 83 85 85.625

8 85 75 80 75 78.75 //// 4/4/4/4 4 85 80 75 70 77.5 85 78 78 73 78.125

9 75 80 85 80 80 //// 4/4/4/4 4 80 75 85 80 80 78 78 85 80 80

Form A, Block 2

10 85 90 80 80 83.75 //// 4/4/4/4 4 85 90 80 80 83.75 85 90 80 80 83.75

11 75 80 75 70 75 //// 4/4/4/4 4 80 90 75 75 80 78 85 75 73 77.5

12 90 80 95 90 88.75 //// 4/4/4/4 4 95 95 95 95 95 93 88 95 93 91.875

13 90 95 90 90 91.25 //// 4/4/4/4 4 95 95 95 90 93.75 93 95 93 90 92.5

14 85 80 90 85 85 //// 4/4/4/4 4 80 95 85 80 85 83 88 88 83 85

15 95 85 80 90 87.5 //// 4/4/4/4 4 90 90 85 85 87.5 93 88 83 88 87.5

16 80 85 75 85 81.25 //// 4/4/4/4 4 85 80 80 85 82.5 83 83 78 85 81.875

17 80 75 85 75 78.75 //// 4/4/4/4 4 85 80 85 85 82.5 83 78 85 80 81.25

18 90 90 90 95 91.25 //// 4/4/4/4 4 95 95 85 90 91.25 93 93 88 93 91.25

19 70 75 75 70 72.5 //// 4/4/4/4 4 75 65 75 70 71.25 73 70 75 70 71.875

Cantonese Sight Test, Form B

Original Item

Number







No (/)

OK as Edited



Form B, Block 1

1 85 95 85 85 87.5 //// 4/4/4/4 4 90 95 85 90 90 88 95 85 88 88.75

2 80 80 85 85 82.5 //// 4/4/4/4 4 85 85 85 85 85 83 83 85 85 83.75

3 70 65 65 75 68.75 //// 4/4/4/4 4 70 65 65 65 66.25 70 65 65 70 67.5

4 80 80 85 85 82.5 //// 4/4/4/4 4 80 80 80 80 80 80 80 83 83 81.25

5 75 80 85 75 78.75 //// 4/4/4/4 4 80 80 75 85 80 78 80 80 80 79.375

6 65 65 65 65 65 //// 4/4/4/4 4 65 65 65 60 63.75 65 65 65 63 64.375

7 70 65 75 70 67.5 //// 4/4/4/4 4 75 75 80 75 76.25 73 70 78 73 73.125

8 95 95 90 95 93.75 //// 4/4/4/4 4 95 90 90 95 92.5 95 93 90 95 93.125

9 80 90 80 80 82.5 //// 4/4/4/4 4 80 80 85 80 81.25 80 85 83 80 81.875

Form B, Block 2

10 85 85 80 90 85 //// 4/4/4/4 4 80 85 80 85 82.5 83 85 80 88 83.75

11 70 70 70 75 71.25 //// 4/4/4/4 4 70 65 70 70 68.75 70 68 70 73 70

12 80 75 80 85 80 //// 4/4/4/4 4 75 65 75 85 75 78 70 78 85 77.5

13 85 85 85 90 86.25 //// 4/4/4/4 4 85 80 85 85 83.75 85 83 85 88 85

14 70 70 70 70 70 //// 4/4/4/4 4 65 65 70 70 67.5 68 68 70 70 68.75

15 70 65 80 80 73.75 //// 4/4/4/4 4 75 75 80 75 76.25 73 70 80 78 75

16 85 85 85 90 86.25 //// 4/4/4/4 4 90 95 85 90 90 88 90 85 90 88.125

Cantonese Sight Test, Form C

Original Item

Number







No (/)

OK as Edited


Form A, Block 1

1 80 95 80 85 85 //// 4/4/4/4 4 90 95 90 90 91.25 85 95 85 88 88.125

2 80 90 80 75 81.25 //// 4/4/4/4 4 75 80 70 75 75 78 85 75 75 78.125

3 75 85 75 80 78.75 //// 4/4/4/4 4 85 85 75 70 78.75 80 85 75 75 78.75

4 75 75 70 75 73.75 //// 4/4/4/4 4 80 85 75 75 78.75 78 80 73 75 76.25

5 70 80 65 65 70 //// 4/4/4/4 4 65 70 65 65 66.25 68 75 65 65 68.125

6 80 90 80 85 83.75 //// 4/4/4/4 4 70 85 75 80 77.5 75 88 78 83 80.625

7 90 85 85 85 86.25 //// 4/4/4/4 4 90 85 80 85 85 90 85 83 85 85.625

8 85 75 80 75 78.75 //// 4/4/4/4 4 85 80 75 70 77.5 85 78 78 73 78.125

9 75 80 85 80 80 //// 4/4/4/4 4 80 75 85 80 80 78 78 85 80 80


Form B, Block 2

10 85 85 80 90 85 //// 4/4/4/4 4 80 85 80 85 82.5 83 85 80 88 83.75

11 70 70 70 75 71.25 //// 4/4/4/4 4 70 65 70 70 68.75 70 68 70 73 70

12 80 75 80 85 80 //// 4/4/4/4 4 75 65 75 85 75 78 70 78 85 77.5

13 85 85 85 90 86.25 //// 4/4/4/4 4 85 80 85 85 83.75 85 83 85 88 85

14 70 70 70 70 70 //// 4/4/4/4 4 65 65 70 70 67.5 68 68 70 70 68.75

15 70 65 80 80 73.75 //// 4/4/4/4 4 75 75 80 75 76.25 73 70 80 78 75

16 85 85 85 90 86.25 //// 4/4/4/4 4 90 95 85 90 90 88 90 85 90 88.125


Table D-6. Cantonese New Form Consecutive Test Angoff Ratings

Cantonese Consecutive Test, Form A

Original

Variable

Name

Average

Angoff

Rating

Average

Relevance

Rating



A 2, Option A, Holistic Scoring Unit @2AHS_Ave 86.25 4



A 2, Option B, Holistic Scoring Unit @2BHS_Ave 83.75 4



A 2, Option C, Holistic Scoring Unit @2CHS_Ave 81.25 4



A 2, Option D, Holistic Scoring Unit @2DHS_Ave 85 4




A 3, Option A, Holistic Scoring Unit @3AHS_Ave 80 4









A 3, Option D, Holistic Scoring Unit @3DHS_Ave 89.375 4













A 4, Option D, Holistic Scoring Unit @4DHS_Ave 75 3.4


















A 6, Option A, Holistic Scoring Unit @6AHS_Ave 85 3.8











B 13, Option A, Scoring Unit 1 @13asu1_Ave


B 13, Option A, Holistic Scoring Unit @13ahs_Ave 80 4

B 13, Option B, Scoring Unit 1 @13bsu1_Ave


B 13, Option B, Holistic Scoring Unit @13bhs_Ave 77 4

B 13, Option C, Scoring Unit 1 @13csu1_Ave


B 13, Option C, Holistic Scoring Unit @13chs_Ave 88.5 4

B 13, Option D, Scoring Unit 1 @13dsu1_Ave


B 13, Option D, Holistic Scoring Unit @13dhs_Ave 79 4







A 11, Option B, Holistic Scoring Unit @11bhs_Ave 85 4






A 11, Option D, Holistic Scoring Unit @11dhs_Ave 86.25 4




A 14, Option A, Holistic Scoring Unit @14ahs_Ave 88 4







A 14, Option C, Holistic Scoring Unit @14chs_Ave 86 4



A 14, Option D, Holistic Scoring Unit @14dhs_Ave 86 3.8







A 12, Option B, Holistic Scoring Unit @12bhs_Ave 80 3.6



A 12, Option C, Holistic Scoring Unit @12chs_Ave 85 3.4







A 8, Option A, Holistic Scoring Unit @8ahs_Ave 83.333333 4



A 8, Option B, Holistic Scoring Unit @8bhs_Ave 82.5 4






























A 15, Option C, Holistic Scoring Unit @15chs_Ave 82.5 4





Cantonese Consecutive Test, Form B

B 1, Option A, Scoring Unit 1 @1ASU1_Ave


B 1, Option A, Holistic Scoring Unit @1AHS_Ave 79.375 3.6

B 1, Option B, Scoring Unit 1 @1BSU1_Ave


B 1, Option B, Holistic Scoring Unit @1BHS_Ave 61.875 3.6

B 1, Option C, Scoring Unit 1 @1CSU1_Ave


B 1, Option C, Holistic Scoring Unit @1CHS_Ave 88.125 3.8

B 1, Option D, Scoring Unit 1 @1DSU1_Ave


B 1, Option D, Holistic Scoring Unit @1DHS_Ave 93.75 4




B 2, Option A, Holistic Scoring Unit @2AHS_Ave 86.25 4



B 2, Option B, Holistic Scoring Unit @2BHS_Ave 83.75 4



B 2, Option C, Holistic Scoring Unit @2CHS_Ave 81.25 4



B 2, Option D, Holistic Scoring Unit @2DHS_Ave 85 4




B 3, Option A, Holistic Scoring Unit @3AHS_Ave 80 4














B 6, Option A, Holistic Scoring Unit @6AHS_Ave 85 3.8









B 6, Option D, Holistic Scoring Unit @6DHS_Ave 92.5 3.8




B 7, Option A, Holistic Scoring Unit @7ahs_Ave 85.833333 4



B 7, Option B, Holistic Scoring Unit @7bhs_Ave 90.833333 4






B 7, Option D, Holistic Scoring Unit @7dhs_Ave 75.833333 4




B 11, Option A, Holistic Scoring Unit @11ahs_Ave 88.125 3.75






B 11, Option C, Holistic Scoring Unit @11chs_Ave 86.875 3.75








































B 14, Option C, Holistic Scoring Unit @14chs_Ave 86 4



B 14, Option D, Holistic Scoring Unit @14dhs_Ave 86 3.8










































Cantonese Consecutive Test, Form C







































A 2, Option D, Holistic Scoring Unit @2DHS_Ave 85 4













B 2, Option D, Holistic Scoring Unit @2DHS_Ave 85 4








































A 4, Option D, Holistic Scoring Unit @4DHS_Ave 75 3.4













B 13, Option D, Holistic Scoring Unit @13dhs_Ave 79 4

















A 3, Option A, Holistic Scoring Unit @3AHS_Ave 80 4

















A 12, Option B, Holistic Scoring Unit @12bhs_Ave 80 3.6



A 12, Option C, Holistic Scoring Unit @12chs_Ave 85 3.4
















B 14, Option D, Holistic Scoring Unit @14dhs_Ave 86 3.8



Table D-7. Korean New Form Sight Test Angoff Ratings

Korean Sight Test, Form A

Original Item

Number




Angoff (with key - target

language)


R1 R2 R3 Averages Yes (/)

No (/)

OK as Edited

(/) R1 R2 R3 Averages R1 R2 R3 Averages R1 R2 R3 Averages

Form A, Block 1

1 90 85 90 88.3 /// 4 4 4 4 90 85 85 86.7 90 85 88 87.50

2 80 85 80 81.7 /// 4 4 4 4 80 85 85 83.3 80 85 83 82.50

3 75 85 70 76.6 /// 4 4 4 4 75 85 80 80 75 85 75 78.33

4 70 75 60 68.3 /// 4 4 4 4 70 75 70 71.6 70 75 65 70.00

5 70 75 60 68.3 /// 4 4 4 4 70 70 70 70 70 73 65 69.17

6 75 75 80 76.6 /// 4 4 4 4 80 75 80 78.3 78 75 80 77.50

7 70 70 75 71.6 /// 4 4 4 4 75 75 75 75 73 73 75 73.33

8 65 65 70 66.7 /// 4 4 4 4 65 65 70 66.7 65 65 70 66.67

9 70 75 75 73.3 /// 4 4 4 4 70 70 70 70 70 73 73 71.67

Form A, Block 2

10 85 80 80 81.7 /// 4 4 4 4 85 80 80 81.7 85 80 80 81.67

11 65 70 60 65 /// 4 4 4 4 65 70 60 65 65 70 60 65.00

12 90 90 95 91.7 /// 4 4 4 4 90 90 95 91.7 90 90 95 91.67

13 85 85 90 86.7 /// 4 4 4 4 85 85 90 86.7 85 85 90 86.67

14 75 75 80 76.7 /// 4 4 4 4 75 75 75 75 75 75 78 75.83

15 75 80 80 78.3 /// 4 4 4 4 75 80 75 76.7 75 80 78 77.50

16 85 80 90 85 /// 4 4 4 4 85 80 85 83.3 85 80 88 84.17

17 85 80 85 83.3 /// 4 4 4 4 85 80 85 83.3 85 80 85 83.33

18 90 90 90 90 /// 4 4 4 4 90 90 90 90 90 90 90 90.00

19 75 70 70 71.6 /// 4 4 4 4 70 65 70 68.3 73 68 70 70.00

Korean Sight Test, Form B

Original Item

Number





language)



No (/)

OK as Edited


Form B, Block 1

1 95 90 90 91.7 /// 4 4 4 4 90 90 90 90 93 90 90 90.83

2 85 80 85 83.3 /// 4 4 4 4 85 85 85 85 85 83 85 84.17

3 65 60 60 61.7 /// 4 4 4 4 65 60 60 61.7 65 60 60 61.67

4 90 90 95 91.7 /// 4 4 4 4 90 90 90 90 90 90 93 90.83

5 85 80 80 81.7 /// 4 4 4 4 80 80 80 80 83 80 80 80.83

6 60 60 60 60 /// 4 4 4 4 60 55 50 55 60 58 55 57.50

7 60 65 70 65 /// 4 4 4 4 65 70 60 60 63 68 65 65.00

8 80 80 75 78.3 /// 4 4 4 4 75 75 70 73.3 78 78 73 75.83

9 65 70 70 68.3 /// 4 4 4 4 65 65 65 65 65 68 68 66.67

Form B, Block 2

10 75 85 70 75 /// 4 4 4 4 75 80 75 76.7 75 83 73 76.67

11 65 65 50 60 /// 4 4 4 4 55 60 50 55 60 63 50 57.50

12 80 75 75 76.7 /// 4 4 4 4 75 70 70 71.7 78 73 73 74.17


13 70 80 70 73.3 /// 4 4 4 4 70 75 70 71.7 70 78 70 72.50

14 70 65 70 68.3 /// 4 4 4 4 65 65 65 65 68 65 68 66.67

15 75 70 70 71.7 /// 4 4 4 4 70 70 70 70 73 70 70 70.83

16 80 70 85 78.3 /// 4 4 4 4 85 75 85 81.7 83 73 85 80.00

Korean Sight Test, Form C

Original Item

Number





language)



No (/)

OK as Edited

(/) R1/R2/R3/R4 Averages R1 R2 R3 Averages R1 R2 R3 Averages

Form B, Block 1

1 95 90 90 91.7 /// 4 4 4 4 90 90 90 90 93 90 90 90.83

2 85 80 85 83.3 /// 4 4 4 4 85 85 85 85 85 83 85 84.17

3 65 60 60 61.7 /// 4 4 4 4 65 60 60 61.7 65 60 60 61.67

4 90 90 95 91.7 /// 4 4 4 4 90 90 90 90 90 90 93 90.83

5 85 80 80 81.7 /// 4 4 4 4 80 80 80 80 83 80 80 80.83

6 60 60 60 60 /// 4 4 4 4 60 55 50 55 60 58 55 57.50

7 60 65 70 65 /// 4 4 4 4 65 70 60 60 63 68 65 65.00

8 80 80 75 78.3 /// 4 4 4 4 75 75 70 73.3 78 78 73 75.83

9 65 70 70 68.3 /// 4 4 4 4 65 65 65 65 65 68 68 66.67

Form A, Block 2

10 85 80 80 81.7 /// 4 4 4 4 85 80 80 81.7 85 80 80 81.67

11 65 70 60 65 /// 4 4 4 4 65 70 60 65 65 70 60 65.00

12 90 90 95 91.7 /// 4 4 4 4 90 90 95 91.7 90 90 95 91.67

13 85 85 90 86.7 /// 4 4 4 4 85 85 90 86.7 85 85 90 86.67

14 75 75 80 76.7 /// 4 4 4 4 75 75 75 75 75 75 78 75.83

15 75 80 80 78.3 /// 4 4 4 4 75 80 75 76.7 75 80 78 77.50

16 85 80 90 85 /// 4 4 4 4 85 80 85 83.3 85 80 88 84.17

17 85 80 85 83.3 /// 4 4 4 4 85 80 85 83.3 85 80 85 83.33

18 90 90 90 90 /// 4 4 4 4 90 90 90 90 90 90 90 90.00

19 75 70 70 71.6 /// 4 4 4 4 70 65 70 68.3 73 68 70 70.00


Table D-8. Korean New Form Consecutive Test Angoff Ratings

Korean Consecutive Test, Form A

Original Variable Name

Average

Angoff

Rating

Average

Relevance

Rating


























































Korean Consecutive Test, Form B
































B 5, Option B, Holistic Scoring Unit @5BHS_Ave 70 3.25





















































B 12, Option B, Holistic Scoring Unit @12bhs_Ave 85.625 2.75
























Korean Consecutive Test, Form C






B 12, Option B, Holistic Scoring Unit @12bhs_Ave 85.625 2.75








































B 5, Option B, Holistic Scoring Unit @5BHS_Ave 70 3.25






































Table D-9. Vietnamese New Form Sight Test Angoff Ratings

Vietnamese Sight Test, Form A

Original Item

Number





language)



No (/)

OK as Edited


Form A, Block 1

1 85 85 80 83 /// 4 4 4 4 80 85 80 81.7 85.0 85.0 80.0 83.3

2 90 90 80 87 /// 3 4 4 3.7 90 90 85 88.3 90.0 90.0 80.0 86.7

3 80 85 80 81.7 /// 4 4 3 3.7 80 85 80 81.7 80.0 85.0 80.0 81.7

4 85 80 80 81.7 /// 4 4 4 4 85 80 80 81.7 85.0 80.0 80.0 81.7

5 80 80 75 78.3 /// 4 4 4 4 80 70 75 75 80.0 80.0 75.0 78.3

6 85 85 80 83 /// 4 4 4 4 85 80 80 81.7 85.0 85.0 80.0 83.3

7 85 85 80 83 /// 3 4 4 3.7 90 85 80 85 85.0 85.0 80.0 83.3

8 75 75 70 73.3 /// 4 4 4 4 75 70 65 70 75.0 75.0 70.0 73.3

9 90 80 80 83.3 /// 3 4 3 3.3 80 70 70 73.3 90.0 80.0 80.0 83.3

Form A, Block 2

10 85 80 75 80 /// 4 4 4 4 85 75 75 78.3 85.0 80.0 75.0 80.0

11 75 70 70 71.7 /// 4 4 4 4 75 70 70 71.7 75.0 70.0 70.0 71.7

12 90 90 85 88.3 /// 4 4 4 4 90 90 90 90 90.0 90.0 85.0 88.3

13 85 80 80 81.7 /// 4 4 4 4 85 80 75 80 85.0 80.0 80.0 81.7

14 75 70 70 71.7 /// 4 4 4 4 75 75 75 75 75.0 70.0 70.0 71.7

15 90 80 75 81.7 /// 4 4 4 4 85 75 70 76.6 90.0 80.0 75.0 81.7

16 90 80 70 80 /// 4 4 4 4 90 80 75 81.7 90.0 80.0 70.0 80.0

17 80 80 75 78.3 /// 4 4 4 4 75 75 70 73.3 80.0 80.0 75.0 78.3

18 80 75 70 75 /// 4 4 4 4 80 70 70 73.3 80.0 75.0 70.0 75.0

19 70 70 65 68.3 /// 4 4 4 4 70 70 65 68.3 70.0 70.0 65.0 68.3

Vietnamese Sight Test, Form B

Original Item

Number





language)



No (/)

OK as Edited


Form B, Block 1

1 90 80 75 81.7 /// 4 4 4 4 85 75 75 78.3 87.5 77.5 75.0 80.0

2 80 75 75 76.7 /// 4 4 4 4 75 70 70 71.7 77.5 72.5 72.5 74.2

3 70 70 65 68.3 /// 4 4 4 4 75 70 65 70 72.5 70.0 65.0 69.2

4 90 70 70 76.7 /// 4 4 4 4 80 70 70 73.3 85.0 70.0 70.0 75.0

5 70 70 70 70 /// 4 4 4 4 70 65 65 66.7 70.0 67.5 67.5 68.3

6 65 60 60 61.7 /// 4 4 4 4 65 60 60 61.7 65.0 60.0 60.0 61.7

7 70 60 55 61.7 /// 4 4 4 4 65 60 55 60 67.5 60.0 55.0 60.8

8 80 70 70 73.3 /// 4 4 4 4 80 70 70 73.3 80.0 70.0 70.0 73.3


9 80 70 70 73.3 /// 4 4 4 4 75 65 65 68.3 77.5 67.5 67.5 70.8

Form B, Block 2

10 70 70 70 70 /// 4 4 4 4 65 65 60 63.3 67.5 67.5 65.0 66.7

11 60 60 55 58.3 /// 4 4 4 4 60 55 50 55 60.0 57.5 52.5 56.7

12 65 60 60 61.7 /// 4 4 4 4 60 60 55 58.3 62.5 60.0 57.5 60.0

13 75 70 65 70 /// 4 4 4 4 65 65 60 63.3 70.0 67.5 62.5 66.7

14 60 60 60 60 /// 4 4 4 4 60 60 60 60 60.0 60.0 60.0 60.0

15 60 60 60 60 /// 4 4 4 4 60 50 50 53.3 60.0 55.0 55.0 56.7

16 75 60 60 65 /// 4 4 4 4 70 70 70 70 72.5 65.0 65.0 67.5

Vietnamese Sight Test, Form C

Original Item

Number





language)



No (/)

OK as Edited

(/) R1/R2/R3/R4 Averages R1 R2 R3 Averages R1 R2 R3 Averages

Form B, Block 1

1 90 80 75 81.7 /// 4 4 4 4 85 75 75 78.3 87.5 77.5 75.0 80.0

2 80 75 75 76.7 /// 4 4 4 4 75 70 70 71.7 77.5 72.5 72.5 74.2

3 70 70 65 68.3 /// 4 4 4 4 75 70 65 70 72.5 70.0 65.0 69.2

4 90 70 70 76.7 /// 4 4 4 4 80 70 70 73.3 85.0 70.0 70.0 75.0

5 70 70 70 70 /// 4 4 4 4 70 65 65 66.7 70.0 67.5 67.5 68.3

6 65 60 60 61.7 /// 4 4 4 4 65 60 60 61.7 65.0 60.0 60.0 61.7

7 70 60 55 61.7 /// 4 4 4 4 65 60 55 60 67.5 60.0 55.0 60.8

8 80 70 70 73.3 /// 4 4 4 4 80 70 70 73.3 80.0 70.0 70.0 73.3

9 80 70 70 73.3 /// 4 4 4 4 75 65 65 68.3 77.5 67.5 67.5 70.8

Form A, Block 2

10 85 80 75 80 /// 4 4 4 4 85 75 75 78.3 85.0 80.0 75.0 80.0

11 75 70 70 71.7 /// 4 4 4 4 75 70 70 71.7 75.0 70.0 70.0 71.7

12 90 90 85 88.3 /// 4 4 4 4 90 90 90 90 90.0 90.0 85.0 88.3

13 85 80 80 81.7 /// 4 4 4 4 85 80 75 80 85.0 80.0 80.0 81.7

14 75 70 70 71.7 /// 4 4 4 4 75 75 75 75 75.0 70.0 70.0 71.7

15 90 80 75 81.7 /// 4 4 4 4 85 75 70 76.6 90.0 80.0 75.0 81.7

16 90 80 70 80 /// 4 4 4 4 90 80 75 81.7 90.0 80.0 70.0 80.0

17 80 80 75 78.3 /// 4 4 4 4 75 75 70 73.3 80.0 80.0 75.0 78.3

18 80 75 70 75 /// 4 4 4 4 80 70 70 73.3 80.0 75.0 70.0 75.0

19 70 70 65 68.3 /// 4 4 4 4 70 70 65 68.3 70.0 70.0 65.0 68.3


Table D-10. Vietnamese New Form Consecutive Test Angoff Ratings

Vietnamese Consecutive Test, Form A

Original Variable Name

Average

Angoff

Rating

Average

Relevance

Rating









A 2, Option C, Holistic Scoring Unit @2CHS_Ave 80 4

































































B 5, Option B, Holistic Scoring Unit @5BHS_Ave 80 3



B 5, Option C, Holistic Scoring Unit @5CHS_Ave 80 3
































Vietnamese Consecutive Test, Form B




















































B 2, Option D, Holistic Scoring Unit @2DHS_Ave 80 3.3

























































Vietnamese Consecutive Test, Form C






























































A 2, Option C, Holistic Scoring Unit @2CHS_Ave 80 4





















development and validation of additional language oral ... · cantonese, korean, and vietnamese...

Documents