C

Q

Implementation* and Initial Validation of theCombined English Language Skills Assessment

(CELSA) at Golden West College

GOLDEN WEST COLLEGEHuntington Beach, California

January, 1993

U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

KThis document has been reproduced asreceived from the person or organizationoriginating itMinor changes have been made to improvefeprOductiOn Quality

Points of view or opinions stated in this docu-ment do not necessarily represent officialOERI position or policy

'PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

S. Isonio

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER tEPIC).

Steven Isonio, PhDAdvisor, matriculation Research & Assessment

BEST COPY PP5

Abstract

The Combined English Language Skills Assessment (CELSA) was pilottested in a sample of ESL classes at Golden West College duringthe Spring 1992 semester. Results indicated that CELSA scoresare significantly related to referrals that students in all ESLcourse levels received from their instructors at the end of theterm. Further, scores adequately differentiate among studentswho received different referral levels. Finally, the multipleclassification scheme (MCS) was used to generate cut scores thatcomprise a placement rule recommended for implementation at GWC.This placement rule incorporates information from a writingsample to support a definitive placement for students scoring inthe upper ranges of the distribution. The need to closelymonitor the placement rule after implementation is stressed.Also, studies of possible disproportionate impact of the CELSAmust be conducted.

Implementation and Initial Validation of the Combined EnglishLanguage Skills Assessment (CELSA) at Golden West College

Steven Isonio, Ph.D.

Introduction.

Background. Initial placement into basic skills and ESL

courses is typically accomplished with the use of standardized

placement tests. At Golden West College, commercially available

instruments in various approval categories have previously been

identified and implemented for both English and mathematics. As

of Fall 1992, only a single ESL placement test, the Combined

English Language Skills Assessment (CELSA) has met the standards

for approval put forth by the California Community Colleges

Chancellor's Office (California Community Colleges, 1992). In

anticipation of the full approval of this instrument and with the

expectation that it would eventually be implemented at Golden

West College, the CELSA was piloted in Spring 1992. This report

describes the pilot study, presents evidence supportive of its

validity, and offers recommendations foT. the implementation of

the CELSA at GWC.

Description of the CELSA. The CELSA uses a cloze format,

including parts of conversations and short dialogues. In

developing items, common student errors offered by experienced

language teachers were selected to be distractors in the

multiple-choice format. This experience was combined with

extensive item analyses to yield items that have high

discriminating power and appropriate difficulty levels.

1

Additionally, the CELSA User Manual (Ilyin, 1992) presents some

normative data from studies at a number of community colleges and

adult learning programs. Results of these studies establish the

internal consistency of the test and offer predictive validity

evidence for its use as a placement instrument. Additionally,

relationships between the CELSA and other measures of English

language skills are represe 4-.ed, indicating a degree of

convergent validity.

The CELSA includes items from the beginning, intermediate,

and advanced levels of the English Language Skills Assessment

(ELSA). As such, it can be used for making placement

recommendations for the students along the full range of English

language skill levels. Normative information and conversion

tables with descriptive labels used by San Francisco Community

College ESL and Adult Education Programs are provided (see

Appendix A). This descriptive information can serve as an

"anchor" for colleges seeking to implement the CELSA and to

coordinate score ranges with their own ESL course levels.

Method.

Design. The basic design of the study entailed

administering the CELSA to a sample of ESL students and then

relating CELSA test scores to instructor evaluations of the

students. The evaluations were made in the form of

referrals/recommendations for an ESL or English course the

following semester.

2

(-I

Participants. The sample consisted of a total of 377

students enrolled in 18 sections of ESL courses during the Spring

1992 semester. All course levels were represented (see Table 1;

all Tables and Figures appear in the Appendix). Analyses were

conducted on data from students who both tested with the CELSA

and received an instructor's recommendation (n = 349).

Procedure. A study designed to determine the

appropriateness of the CELSA as a placement instrument at GWC was

planned in early Spring of 1992. At that time, no ESL instrument

had earned any level of approval from the Chancellor's review

process. Further, GWC had been notified that the currently used

instrument, the ELSA was no longer published and that its

publisher (Newbury House) had indicated that it would not provide

the necessary supportive information for it to be eventually

approved. Discussions of the merits of the CELSA had taken place

at statewide assessment workshops, and within the Coast Community

College District at a meeting of ESL chairpersons form Golden

West and Orange Coast Colleges, interested faculty, and

assessment specialists.

A number of ESL classes during the Spring 1992 semester were

identified for inclusion in the sample. With the cooperation of

the then Chairperson of the GWC ESL Department, instructors were

contacted and asked for permission to use approximately one hour

of class time during the last two weeks of the semester in order

to administer the CELSA to students. The tests were administered

to both day and evening classes representing the full range of

3

ESL levels. All participating instructors were further asked to

provide copies of their rosters showing the referrals/

recommendations they had made for the students in their classes.

Test data and referral information were then matched for

analysis.

Results.

Overview of the Analyses. There were two major components,

to the analysis. The first was designed to assess the

relationship between CELSA test scores and instructor referrals.

Specifically, correlations were computed within ESL levels and

the median correlation was calculated across levels. These

analyses served to establish whether the CELSA had predictive

validity. The second set of analyses was intended to develop

recommended cut scores (an overall placement rule) for actually

implementing the CELSA at GWC. In this case, a variety of

approaches was undertaken.

Predictive Validity. The primary index of predictive

validity of a placement test is its relationship, as evidenced by

a correlation coefficient, to a reasonable criterion of

performance in class; typically this criterion is the course

grade. Since ESL courses at GWC are ungraded (students receive

"credit/no credit" instead of the letter grade scale), the

instructors referral/recommendation was used as the criterion

against which to gauge the predictive ability of the CELSA. The

referrals have an added strength in this regard; they constitute

4

an instructor's judgment of the class which is most appropriate

for the student the next semester. Like grades, they reflect an

evaluation of work performed over the course of the term, but

unlike grades they are more clearly a statement of the student's

readiness for a future class. This, of course, is precisely the

intended function of placement tests.

Table 2 presents correlations between CELSA test scores and

ESL instructor referral levels (considered an interval scale of

measurement), by ESL course level. These values have been

corrected for restriction of range due to the use of intact, pre-

selected groups (Matriculation Local Research Options Committee,

1991). The r values range from .41 (for ESL 004) to .79 (for

both ESL 001 and the combined ESL 900/001), with a median value

of r = .59. Collapsing across levels, the overall relationship

between CELSA scores and instructor referrals is somewhat high

(r = .72). In all cases, the correlation coefficients are

statistically significant. Thus, there is clear evidence for the

predictive validity of the CELSA as a placement tool for ESL

courses at GWC.

Development of a Placement Rule. Since the instructor

referrals indicate which course is considered most appropriate

for a student the following semester, they were used to create

sub-distributions for comparison purposes in support of the

development of cut scores. Table 3 shows selected descriptive

statistics for the sub-distributions of students receiving each

of the seven referral levels. As can be seen, the CELSA is able

5

to adequately differentiate among referral levels, as evidenced

by the spread of mean and median score values across levels

(Figure 1 depicts mean CELSA scores, by referral level). An

analysis of variance performed on CELSA scores by referral levels

indicated highly significant differences among the levels [F (6,

342) = 91.21, p < .001.]. The Tukey-Kramer test was used for

multiple comparisons because of its appropriateness in cases

where there is at least a moderate imbalance in sample sizes

(Dunnett, 1980). This analysis revealed that all pairwise

difference comparisons are statistically significant, except for

the difference between the values for (ESL 001) and (ESL 002);

these results are summarized in Table 4. Finally, the

coefficients of skewness indicate that", with the exception of

referral level 2, the sub-distributions are relatively

symmetrical.

The CELSA User Manual (Ilyin, 1992) describes and recommends

the Multiple Classification Scheme (MCS; developed by Cooley and

Lohnes, 1971). It incorporates mean and standard deviation

values for adjacent groups in identifying a score level (a cut

score) that best differentiates the two groups. The formula for

using the MCS is presented in Appendix A of this report. The

result of the MCS computations and resulting recommended cut

scores are presented in Table 5. The second column of that Table

contains the MCS-calculated cut scores which should serve as the

upper limits of the score ranges. These values are rounded

upward and used to define placement score ranges depicted in the

6

ti

last column of Table 5. As can be seen, the score ranges for the

lowest group is "0 to 23". Ten s-ore values (range 24 to 33) are

associated with ESL 001; the next 8 and then 9 score values are

associated with ESL 002 (34 to 41) and ESL 003 (42 to 50),

respectively. The next five score values (51 to 55) would

recommend placement into ESL 004 and the following five (56 to

60) to ESL 005. Finally, scores of 61 or greater would result in

placement recommendations to English 10.

As shown in Table 5, the score ranges differentiating the

highest placements are not as wide as those associated with the

lower levels. An ESL placement writing sample (PWS) has been

used at Golden West College to assist in making definitive

placements for students scoring in the upper ranges of score

valuest $olistic evaluations of the ESL PWS have previously been

shown to be reliable (Isonio, 1992). The ESL PWS can be

incorporated into the model developed for the CELSA in a similar

way so as to make definitive placements. A reasonable scheme for

doing so would be to evaluate writing samples for all students

scoring in the ranges associated with ESL 005 and English 10. In

this way, those students scoring 56 to 60 would either retain the

ESL 005 recommendation or receive a placement recommendation to

ESL 004 or to English 10. Similarly, students scoring 61 or

above would be referred to ESL 005, or to English 10, depending

upon the evaluation of their writing sample. (The placement

scheme that incorporates the ESL placement writing sample is

depicted in Table 6.) In short, both the multiple

7

classification scheme and the examination of CELE% score sub-

distributions, in 'a variation of the contrasting groups approach,

support the appropriateness of the recommended initial placement

rule.

Summary and Discussion.

Evidence has been presented that supports the value of the

CELSA as a placement instrument at Golden West College. Using

instructor referrals/recommendations as the primary criterion

against which to gauge CELSA scores, impressive predictive

validity coefficients were obtained across the range of ESL

course levels thereby establishing that there is a significant

relationship between CELSA scores and this criterion. Further,

both analyses using the MCS approach and the contrasting groups

method to develop a rule for generating course placement

recommendations were performed. These approaches converge in

their support of a placement rule for ESL levels 900 through ESL

5, and above. Further, to clarify the placement associated with

high score ranges, the ESL Placement Writing Sample should be

used to make a definitive placement for students scoring in the

ranges associated with ESL 005 and English 10/100.

Recommendations.

1. The Combined English Language Skills Assessment (CELSA)should be adopted and implemented as the primary ESLplacement measure at Golden West College. Thisinstrument has received provisional Epproval by theChancellor's Office and the evidence presented in thisreport supports its appropriateness as a placement toolat GWC.

2. The placement rule depicted in Table 6 should be usedinitially. It was derived from the MultipleClassification Scheme recommended by the author of theCELSA and is consistent with an examination of thegraphical representation of CELSA score sub-distributionsfor contrasting groups.

3. This placement rule should be monitored for itseffectiveness. Instructors and students in a sample ofESL courses should provide midterm ratings of theappropriateness of the placement. These ratings wouldtherefore serve to corroborate the initial placementscheme or provide a basis for adjustments deemednecessary.

4. Information from other measures should be collected andanalyzed to support the eventual development of amultiple measures placement model for ESL courses.

5. Studies of possible disproportionate impact anddifferential validity of the CELSA as a placementinstrument at Golden West College should be conducted.

6. Item analyses by referral level subgroups should beperformed to support the ongoing examination of the ESLcurriculum.

References

California Community Colleges. (1992). Assessment InstrumentsApproval Status Summary.

Cooley, W. W. & Lohnes, P. R. (1971). Multivariate DataAnalysis. New York: Wiley and Sons.

Dunnett, C. W. (1980). Pairwise multiple comparisons in thehomogeneous variance, unequal sample size case. Journal ofthe American Statistical Association, 75, 789-795.

Ilyin, D. (1992). Users Guide for Combined English LanguageSkills Assessment (CELSA). San Francisco: Association ofClassroom Teacher Testers.

Isonio, S. (1992). Judgments of ESL placement writing samplesat Golden West College: An evaluation of inter-raterreliability. Golden West College, Huntington Beach, CA.

Matriculation Local Research Options Committee. (1991).Assessment Validation Project Local Research Options.February, 1991.

10

APPENDICES

Page

A Adult Education Proficiency Levels --San Francisco Community College InstructionalLevels 12

Multiple Classification Scheme--CalculationFormula 13

B Table 1: Number of Students (and Sections)Sampled in the CELSA Pilot Study, By ESLCourse Level 14

Table 2: Predictive Validity Coefficients,By Course Level 15

Table 3: Selected CELSA Score DescriptiveStatistics For Referral Sub-Distributions ' 16

Table 4: Results of the Multiple Comparison Testfor Differences Between Pairs of Means 17

Table 5: Multiple Classification Scheme (MCS)Placement Rule 18

Table 6: Recommended Initial Placement Rule forCELSA at Golden West College 19

C Figure 1: Mean CELSA Score, By Referral Level 20

13.

APPENDIX 8-1

ADULT EDUCATION ESL PROFICIENCY LEVELS

The San Francisco Community College Centers' revised ESL MasterPlan describes eight levels of instruction beginning with level 50,which is a basic literacy course as well as an introductory course forstudents with zero proficiency in speaking, reading, and comprehendingEnglish. For obvious reasons, formal testing is not done at thislevel. Levels 100-700 are described below. These levels lendthemselves well to the two CELSA test forms. Conversion tables forplacement are presented in Tables 1 and 2 and on the keys.

LOWER BEGINNING TESL 100): Designed for the student with littleknowledge of English, the basic goal is for students to begin tocommunicate and conduct the affairs of daily life in spoken and writtenEnglish. High frequency vocabulary is emphasized and regular forms ofthe present, past, and future are covered in the standardtransformations (affirmative, negative yes/no and WH-questions).

UPPER DEGINNING (ESL 200): Students continue to gain simplecommunicative ability to function in practical areas necessary forsurvival in the United States. Although students read short authenticpassages and simple conversations, emphasis continues to be on oralproduction and listening comprehension with an expansion of thestructures of level 100 to include irregular past forms, presentperfect, modals, and comparison using adjectives and adverbs.

LOWER INTERMEDIATE (ESL 300): A period of transition when studentsassimilate previously learned material and begin to develop fluency aswell as accuracy both in spoken and written English. Students read bothshort authentic materials as well as adapted reading selections.Complex sentences using noun, adjective and adverb clauses areintroduced; past forms in the continuous and perfect tenses andconditionals in the present and future are stressed.

UPPER INTERMEDIATE (ESL 400): Advanced grammatical structures such ascomplex sentences using embedding and relative clauses, passive forms,future perfect and continuous, unreal conditionals, and complex wordorder are introduced. Students continue to read authentic materials andselections appearing in ESL texts. Students often are in vocationalcourses for training or upgrading.

LOWER ADVANCED (ESL 500): Students learnto use more advancedstructures including modal auxiliaries in past forms, future perfect,perfect continuous forms, and perfect conditional forms. They alsocontinue to assimilate proper stress, intonation, and rhythm of spokenAmerican English. Written compositions and oral presentations are abasic part of the course along with the use of mass media (radio, TV)to improve listening skills. Students read authentic materials as wellas text materials in ESL texts.

UPPER ADVANCED (ESL 600): Students review materials from previouscourses and prepare for high school diploma courses, college entrance,or more advanced vocational training. A thorough review of Englishgrammar, advanced reading skills, composition, and student researchprojects are basic components of the course. A division certificate isawarded upon successful completion of this level.

ADVANCED PLUS (ESL 700): Students work on the skills necessary tofunction in academic situations in non-ESL classes on high school,college, and university levels. Emphasis is on the expansion ofvocabulary, grammatical analysis and comprehension and other readingskills. Writing skills, including composition and research are alsoemphasized.

12

. ti

LEVEL 1LEVEL 2

CUTOFF =

Multiple Classification Schemesto Establish Cut Scores Between Adjacent Levels

);1(S2) + i2(51)Si + S2

-where S = Standard Deviation

X = Mean

Subscripts 1 and 2 indicate level 1 and 2, respectively

1 (SEE; Cooley & Lohnes, 1971)

13

Table 1

pumber of - Pilot

Study. By ESL Course Level

Course Level Number of Sections Number of Students

ESL 900 1 24

ESL 900/001 2 45

ESL 001 1 27

ESL 002 3 67

ESL 003 2 57

ESL 004 3 87

ESL 005 3 70

Total 15 377

14

.L

Table 2

Predictive Validity Coefficients, By Course Level

Course Level Validity Coefficient

ESL 900

ESL 900/001

ESL 001

ESL 002

ESL 003

ESL 004

ESL 005

.52

.79

.79

.70

.59

.41

.47

Median r = .59

Note: All validity coefficients for individual course levelshave been corrected for restriction of range.

Table 3

elected CELSA Score Descriptive Statistics of Referral Level

sub- Distributions

Referral Level

CELSA Score

Mean Median St.Deviation

ESL 900 16.50 15.5 12.19

ESL 001 30.72 30 11.95

ESL 002 33.72 37 9.90

ESL 003 46.74 47 8.93

ESL 004 52.97 54 9.67

ESL 005 56.58 57 8.86

ENG 010 62.23 63 9.40

Total 48.32 14.38

16

Table 4

Results of Multiple Comparison Tests For Differences

Between :airs of Means

Levels / Meansthe Comparison

In CriticalDifference

Obtained(p<.05) Difference

ESL 900 (16.50)ESL 001 (30.72) 8.60 14.22*

ESL 001 (30.72)ESL 002 (33.70) 4.57 2.98

ESL 002 (33.70)ESL 003 (46.74) 3.31 13.04*

ESL 003 (46.74)ESL 004 (52.97) 3.20 6.23*

ESL 004 (52.97)ESL 005 (56.58) 3.23 3.61*

ESL 005 (56.58)ENG 010 (62.23) 3.59 5.65*

* p < .05

(Note: Comparisons were based on the Tukey-Kramer procedure.)

Table 5

Mltiple Classification Scheme (MCS) Placement Rule

Referral Level MCS Cut (Upper)Placement

Score Range

ESL 900 22.76 <= 23

ESL 001 32.42 24-33

ESL 002 40.25 34-41

ESL 003. 49.73 42-50

ESL 004 54.85 51-55

ESL 005 59.32 56-60

ENG 010 >=61

Table 6

Recommended Initial Placement Rule for CELSA at

Golden West College

Placement RecommendationCELSA ScoreRange

ESL 900 <= 23

ESL 001 24 - 33

ESL 002 34 - 41

ESL 003 42 - 50

ESL 004 51 - 55

.ESL 005 or higher* 56 - 60

English 010 or higher* >= 61

*ESL Placement Writing Samples are evaluated for students scoringin these ranges.

Fig

ure

1.M

ean

CE

LSA

Sco

re, B

y R

efer

ral L

evel

75

Ref

erra

l Lev

el /

Mea

n

Acknowledgments

I wish to thank Donna Ilyin for granting permission to include anexcerpt from the CELSA User Manual in the Appendix of this reportand Carol Cooperman for her help in preparing the final draft ofthis report.

21