examinees’ attitudes about the toefl-cbt, possible ...examinees’ attitudes about the toefl-cbt,...

RESEARCHREPORT

March 2001RR-01-01

Statistics & Research DivisionPrinceton, NJ 08541

Examinees’ Attitudes Aboutthe TOEFL-CBT, PossibleDeterminants, and RelationshipsWith Test Performance

Lawrence J. StrickerGita Z. Wilder

Examinees’ Attitudes About the TOEFL-CBT,Possible Determinants, and Relationships

with Test Performance

Lawrence J. Stricker and Gita Z. Wilder

March 2001

Research Reports provide preliminary and limiteddissemination of ETS research prior to publication. They areavailable without charge from the

Research Publications OfficeMail Stop 07-REducational Testing ServicePrinceton, NJ 08541

Examinees� Attitudes About the TOEFL-CBT,Possible Determinants, and Relationships

with Test Performance

Lawrence J. Stricker and Gita Z. Wilder

March 2001

Abstract

The principal aim of this study was to assess examinees� acceptance of the TOEFL-CBT,

and its associations with possible determinants of this acceptance and with test

performance. A secondary goal was to evaluate differences in the pattern of results for

examinees from different countries. A questionnaire concerning attitudes about the test,

familiarity with computers, and other relevant variables was administered to TOEFL-

CBT examinees at large testing centers in Buenos Aires, Cairo, and Frankfurt. Attitudes

about the TOEFL-CBT were moderately positive and had similar patterns of relationships

in the three countries: slight or moderate with test performance, moderate with attitudes

about admissions tests in general, slight with computer anxiety and test anxiety, and

minimal with other variables. (Key words: TOEFL, computer-adaptive testing, attitudes,

test anxiety, computer anxiety, computer familiarity, test preparation)

The introduction of the TOEFL-CBT inevitably raises questions about examinees�

reactions to it. This issue is important not only because of customer service concerns but

also because the test�s validity may be affected insofar as these reactions represent an

irrelevant source of variance in test performance.

Kirsch, Jamieson, Taylor, and Eignor (1998) have already demonstrated that

computer familiarity is related to performance on the paper-and-pencil TOEFL (high

scorers are more familiar with computers) and to examinees� gender (men are more

familiar) and their native language (e.g., Spanish speakers are more familiar, Japanese

speakers are less familiar).

Other research with college students and adults has established, in turn, that

computer familiarity is related to acceptance and other attitudes about computers (Burke,

Normand, & Raju, 1987; Powers & O�Neill, 1993; Wilder, Mackie & Cooper, 1985) and

to anxiety about computers (Kernan & Howard, 1990; Powers & O�Neill, 1993). Less is

known about the relationship of familiarity and computer anxiety with performance on

computerized tests. Familiarity was related to performance in one study (Lee, 1986) but

not in three others (Powers & O�Neill, 1993; Taylor, Jamieson, Eignor, & Kirsch, 1998;

Wise, Barnes, Harvey, & Plake, 1989), and anxiety and performance were unrelated in

three studies (Mazzeo, Druesne, Raffeld, Checketts, & Muhlstein, 1992; Powers &

O�Neill, 1993; Wise et al., 1989). The Taylor et al. (1998) study is especially relevant,

for it concerns examinees� performance on a prototype TOEFL-CBT. Finally, several

studies have established high acceptance of computerized tests (Powers & O�Neill, 1993;

Schmitt, Gilliand, Landis, & Devine, 1993; Schmidt, Urry, & Gugel, 1978). And

Jamieson, Taylor, Kirsch, and Eignor (1999) found that providing TOEFL examinees

with a computer-administered tutorial on taking a prototype TOEFL-CBT increased their

acceptance of that test, particularly among examinees who were less familiar with

computers.

A number of key issues remain unresolved. The bulk of this research, with the

notable exceptions of the Kirsch et al. (1998), Taylor et al. (1998), and Jamieson et al.

work (1999), has been done with Americans and may not be applicable to TOEFL

examinees. Nothing is known about the acceptance of the operational version of the

2

TOEFL-CBT. And nothing is known about the implications of acceptance for

computerized test performance in general or on TOEFL-CBT performance in particular.

Moreover, both the determinants and generality of CBT acceptance remain unclear. Is

CBT acceptance distinct from acceptance of tests in general, paper-and-pencil or

otherwise? Is CBT acceptance linked to anxiety about computers or anxiety about tests?

Does CBT acceptance and its determinants vary from culture to culture? Answers to

these questions are needed for a clear understanding of the nature of acceptance of the

TOEFL-CBT.

Accordingly, the main purpose of this study was to assess examinees� acceptance

of the TOEFL-CBT, and its associations with possible determinants and with test

performance. A secondary goal was to evaluate differences in the pattern of results for

examinees from different language groups and different countries.Method

Sample

The sample was drawn from examinees taking the TOEFL-CBT in the Spring and

Summer of 1999 at a testing center operated by or affiliated with Sylvan Learning

Systems in each of three cities: Buenos Aires, Cairo, and Frankfurt. These three centers

were chosen because of their large testing volumes; diversity, spanning the world and

major language groups; and relative within-center homogeneity in native language and

country of origin of the examinees.

The sample consisted of everyone who agreed to participate: 144 examinees in

Buenos Aires tested between June 12 and August 11, 142 examinees in Cairo tested

between June 14 and June 28, and 403 examinees in Frankfurt tested between June 7 and

August 12, (one examinee with unusable data was excluded), a total of 689 examinees1.

Examinees tested during these periods but not participating in the study numbered 152 in

Buenos Aires, 75 in Cairo, and 182 in Frankfurt, a total of 409 examinees. The

participation rates were 48.6% in Buenos Aires, 65.4% in Cairo, and 68.9% in Frankfurt.

Measures

Questionnaire. A questionnaire (in English) made up of a set of scales and two

single-item measures was employed. (The questionnaire appears in the Appendix.) Most

3

of the scales were derived from existing measures to maximize the scales� validity and to

ensure comparability with previous studies.

1. TOEFL-CBT Acceptance. This is a 12-item scale (e.g., �The TOEFL tells

how well people can use English in school�) with Agree, Do Not Agree, and Do Not

Know options. (To obviate language problems, the instructions were to use the Do Not

Know option if �you do not know whether you agree with the statement or do not

understand the statement�.) Most of the items were adapted from measures of attitudes

towards admissions tests and tests used in industrial selection (Arvey, Strickland,

Drauden, & Martin, 1990; Baird, 1977; Rowan & Blewitt Incorporated, 1998; Schmitt et

al., 1993; Smither, Reilly, Millsap, Pearlman, & Stoffey, 1993).

2. Acceptance of Admissions Tests. This is a five-item scale (e.g., �People who

receive high scores on university admissions tests will be successful in school�) with

Agree, Do Not Agree, and Do Not Know options. All of the items were adapted from

measures of attitudes towards admissions tests, standardized tests, or tests used in

industrial selection (Fiske, 1967; Gilliand, 1994; Schmitt et al., 1993; Smither et al.,

1993).

3. Computer Liking. This is a five-item scale adapted from the Computer Liking

factor subscale (Bandalos & Benson, 1990) on the Computer Attitude Scale (Loyd &

Gressard, 1984 (e.g., �I want to use computers�), a measure of computer anxiety,

with Agree, Do Not Agree, and Do Not Know options. Note that a high score represents

low anxiety.

4. Computer Confidence. This is a seven-item scale adapted from the Computer

Confidence factor subscale (Bandalos & Benson, 1990) on the Computer Attitude Scale

(e.g., �I feel I know what I am doing when I use computers�) with Agree, Do Not Agree,

and Do Not Know options. Again note that a high score represents low anxiety.

5. Total Computer Attitude. This is the total score for the Computer Liking and

Computer Confidence scales. Note that a high score represents low anxiety.

6. Worry. This is a five-item scale adapted from the Worry subscale on the

Revised Test Anxiety Scale (Benson & El-Zahhar, 1994) (e.g., �When I am taking a test,

4

I often think how difficult the test is�) with Agree, Do Not Agree, and Do Not Know

options.

7. Tension. This is a five-item scale adapted from the Tension subscale of the

Revised Test Anxiety Scale and, to secure sufficient items, from the Test Anxiety

Inventory (Spielberger, 1980) and Reactions to Tests (Sarason, 1984) (e.g.., �I am

nervous about tests�) with Agree, Do Not Agree, and Do Not Know options.

8. Test-Irrelevant Thinking. This is a four-item scale adapted from the Test-

Irrelevant Thinking subscale on the Revised Text Anxiety Scale (e.g., �When I am taking

a test, I think about things I will do after the test�) with Agree, Do Not Agree, and Do

Not Know options.

9. Bodily Symptoms. This is a five-item scale adapted from the Bodily

Symptoms subscale on the Revised Test Anxiety Scale (e.g., �My mouth becomes dry

during a test�) with Agree, Disagree, and Do Not Know options.

10. Total Test Anxiety Scale. This is the total score for the Worry, Tension,

Test-Irrelevant Thinking, and Bodily Symptoms subscales.

11. Computer Familiarity. This is a six-item scale adapted from the Computer

Familiarity Questionnaire (Eignor, Taylor, Kirsch, & Jamieson, 1998) (e.g., �How often

do you use a computer?�) with Never, Once a Week or Less, More than Once a Week,

and Do Not Know options.

12. Preparation for TOEFL. This is a single-item measure, �About how many

hours did you prepare for the computer-based TOEFL?,� with a seven-point scale (0

Hours to More than 40 Hours) plus a Do Not Know option.

13. Admissions Tests Taken. This is a single-item measure, �About how many

different university admissions tests have you taken?� with a five-point scale (1 test to 5

or More Tests) plus a Do Not Know option.

The items on the questionnaire were written or revised (1) to be clear to ESL

students and easy for them to answer and (2) to balance the direction of keying of items

on the parent scales to minimize acquiescence response style. Versions of the

questionnaire were successively revised on the basis of reviews by an ESL expert and

5

pilot tests with three groups of Arabic-, German-, and Spanish-speaking ESL students

from the same geographical regions or even the same countries where the testing centers

were located: 13 students from Hunter College, City University of New York; 12 students

from Rutgers University--Newark, and 6 students from Middlesex Community College

(New Jersey). The students were at the intermediate level in ESL programs at these

universities, and were not matriculated at the institutions.

Each scale was item analyzed for the total sample, using two indexes: (1) the

item-total score correlation (corrected for item overlap in the total score) and (2) the item-

subgroup (i.e., testing center) partial correlation, controlling for the total score. The latter

is an index of differential item functioning (Stricker, 1982).

In scoring items and obtaining total scores for all scales except Computer

Familiarity, items were scored 1 for the keyed response (Agree or Do Not Agree), -1 for

the unkeyed response (Agree or Do Not Agree), and 0 for the Do Not Know response.

For Computer Familiarity, items were scored 1 for the Never response to 4 for the More

Than Once a Week response, and Do Not Know responses were not scored.

A total score was not obtained on a scale if an examinee had an excessive number

of omitted or not reached responses on the scale: more than two on TOEFL-CBT

Acceptance, and more than one on Acceptance of Admissions Tests, Total Test Anxiety

Scale (Worry, Tension, Test-Irrelevant Thinking, and Bodily Symptoms combined),

Total Computer Attitudes (Computer Liking and Computer Confidence combined), and

Computer Familiarity. A total score for Computer Familiarity was also not obtained if an

examinee had any Do Not Know responses on this scale. If an examinee had fewer than

the critical number of omitted or not reached responses on a scale, the center�s modal

responses to the items were substituted for the missing responses in obtaining the total

score and in item and reliability analyses.

Items with item-total score correlations below .15 or item-subgroup partial

correlations at or above ± .15 were eliminated: three items on Acceptance of TOEFL-

CBT (leaving a nine-item scale), two items on Acceptance of Admissions Tests (leaving

a three-item scale), and one item on Worry (leaving a four-item scale).

6

In scoring the single-item measures, responses on Preparation for TOEFL were

dichotomized: 0 hours and 1 to 5 Hours = 0, 6 to 10 Hours to More than 40 Hours = 1,

and Do Not Know responses were not scored. And Admissions Tests Taken was scored:

1 Test = 1, 2 Tests = 2, 3 Tests = 3, 4 Tests = 4, and 5 or More Tests = 5; Do Not Know

responses were not scored.

Other Variables. Additional variables were obtained from other information

provided by the examinees during the TOEFL-CBT administration.

1. Age (age in years at the time of testing).

2. Sex.

3. Native language.

4. Native country.

5. Reason for taking TOEFL.

6. Planning study in United States or Canada.

7. TOEFL Listening score. This is a scaled score, ranging from 0 to 30.

8. TOEFL Structure/Writing score. This is a scaled score, ranging from 0 to 30.

9. TOEFL Reading score. This is a scaled score, ranging from 0 to 30.

10. TOEFL Total score. This is a scaled score, ranging from 0 to 300.

Procedure

The test center staff was instructed to recruit everyone who took the TOEFL

during the data collection period. After an examinee completed the test and logged off, a

staff member explained the purpose of the study and asked him or her to stay and

complete the questionnaire. Participants were given a Parker ballpoint pen with a retail

value of $19. The questionnaire took approximately 15 minutes to complete.

Analysis

Chi Square tests of the categorical background variables for participants and

nonparticipants were carried out for each center. Parallel one-way analyses of variance

7

of the means for age and the TOEFL scores for participants and nonparticipants were also

carried out.

Chi Square tests for individual items on TOEFL-CBT Acceptance and

Acceptance of Admissions Tests in the three centers were carried out.

One-way analyses of variance of the means for the questionnaire scales and

Admissions Tests Taken, age, and the TOEFL scores in the three centers were carried

out. Parallel Chi Square tests of Preparation for TOEFL and the categorical background

variables were also carried out.

The product-moment intercorrelations of the questionnaire measures, TOEFL

scores, and background variables were computed separately for each center, using a pair-

wise missing data program. In these analyses, dummy codes were used for Sex (male =

0, female = 1), Native Language (same as the country where the center is located = 1, all

others = 0), Native Country (same as the country where the center is located = 1, all

others = 0), Reason for Taking TOEFL,2 and Planning to Study in United States or

Canada (yes = 1, no = 0).

The difference between corresponding correlations for pairs of centers was

evaluated by a Z test of the transformed correlations.

The internal-consistency reliability of the questionnaire scales for each center was

computed by Coefficient Alpha.

Both statistical and practical significance were considered in evaluating the

results. For statistical significance, an .05 alpha level was used in all analyses. For

practical significance, indexes that reflect a �small� effect size, accounting for 1% of the

variance, were used: an η of .10 in the analyses of variance, an r of .10 and a difference

in r of .10 in the correlation analyses, and a W of .10 in the Chi Square analyses

(Cohen, 1988). Because of the large number of variables, the focus was on consistent

results within and between centers.

8

Results

Comparisons of Background Variables and TOEFL Scores for Participants andNonparticipants

The frequency distributions of the background variables for the participants and

nonparticipants in each center and Chi Square tests are summarized in Table 1. The

means for age and TOEFL scores, and analysis of variance statistics are summarized in

Table 2.

In Buenos Aires, the differences in background characteristics and TOEFL scores

for participants and nonparticipants were not statistically or practically significant. In

Cairo, the differences in background characteristics were not significant, but the

differences in TOEFL Reading and TOEFL Total were significant (participants had

higher scores than nonparticipants on both variables). In Frankfurt, the differences in two

background characteristics were significant: Native Country and Native Language (more

participants were German-born and native speakers of German); differences in all

TOEFL scores were also significant (participants had higher subtest and total scores).

Summary. In brief, study participants and nonparticipants did not differ in

Buenos Aires, but did differ in Cairo and Frankfurt, with the participants performing

better on the TOEFL in both cities, and being more likely to be German-born and native

speakers of German in Frankfurt.

______________________________

Insert Tables 1 and 2 about here.______________________________

Individual Items on Acceptance of TOEFL-CBT and Acceptance of Admissions Tests

The responses to the individual items on TOEFL-CBT Acceptance in the three

centers and Chi Square analyses are summarized in Table 3; the corresponding data for

the items on Acceptance of Admissions Tests are summarized in Table 4.

TOEFL-CBT Acceptance. In general, most examinees gave favorable responses

to the items on TOEFL-CBT Acceptance. In only four out of 27 instances (nine items by

three centers) did most examinees choose an unfavorable response, and three of the

instances involved the same item. Examinees in every center predominantly choose the

9

unfavorable response (Agree) to this item, �My English is better than my TOEFL scores

demonstrate.�

The differences among the centers were statistically and practically significant for

six items. For these items, the most favorable (Agree) responses were made by Cairo to

�A school or company can learn a lot about a person�s English ability from their TOEFL

scores� and to �The TOEFL tells how well people can use English in school�, and the

most favorable (Do Not Agree) responses were made by Frankfurt to �The TOEFL

examines English skills I will never need�, by Cairo to �My English skills are better than

my TOEFL scores demonstrate�, by Frankfurt to �The TOEFL did not give me a good

opportunity to demonstrate my English ability�, and by Frankfurt to �The English skills

tested by the TOEFL are different from the English needed in school�.

Acceptance of Admissions Tests. The results were more mixed for examinees�

responses to the items on Acceptance of Admission Tests. In five out of nine instances

(three items by three centers), most examinees chose an unfavorable response. Four of

these instances involved the same two items. Examinees in Buenos Aires and Frankfurt

predominantly chose the unfavorable response (Do Not Agree) to these items, �People

who receive high scores on university admissions tests will be successful in school� and

�My scores on university admissions tests accurately demonstrate my skills and abilities.�

The differences among the centers were statistically and practically significant for

all items. The most favorable (Agree) responses were made by Cairo to �People who

receive high scores on university admissions tests will be successful in school�, �My

scores on university admissions tests accurately demonstrate my skills and abilities�, and

�University admissions tests are a good way to learn about a person�s skills and abilities�.

Summary. In short, examinees� attitudes were generally positive towards theTOEFL-CBT and more mixed towards admissions tests in general. The centers differedinconsistently on individual attitude items concerning the TOEFL-CBT, but Cairo haduniformity more favorable responses than other centers on items concerning admissionstests in general.

______________________________


10

Comparisons of Questionnaire and Background Variables, and TOEFL Scores forCenters

The mean scores on all of the questionnaire variables (except Preparation for

TOEFL) for the three centers and analysis of variance statistics are summarized in Table

5; the corresponding data for age and TOEFL scores are summarized in Table 2. The

frequency distributions for the remaining variables and the Chi Square tests are

summarized in Tables 1 and 6.

Questionnaire Variables. The differences among the three centers were not

statistically and practically significant for TOEFL-CBT Acceptance, but they were

significant for Acceptance of Admissions Tests (Cairo was highest) and most of the other

questionnaire variables: Computer Liking (Cairo was highest), Total Computer Attitude

(Cairo was highest), Worry (Cairo was highest), Test Irrelevant Thinking (Cairo was

highest), Bodily Symptoms (Frankfurt was lowest), Total Test Anxiety Scale (Cairo was

highest), Computer Familiarity (Buenos Aires was highest), and Preparation for TOEFL

(Buenos Aires was highest).

It is noteworthy that the mean scores for TOEFL-CBT Acceptance and

Acceptance of Admissions Tests were consistent with the item-level analyses. The

scores were moderately positive for TOEFL-CBT Acceptance (item means of .25 to .30;

the theoretical score range is 1 to �1) and moderately positive to moderately negative for

Acceptance of Admissions Tests (item means of -.20 to .21; the theoretical score range is

1 to �1).

Background Variables. The differences among the centers were significant for

several background variables: Age (Buenos Aires was highest), Sex (Cairo had the most

males), and Reason for Taking TOEFL--Undergraduate (Buenos Aires had the fewest

undergraduates).

TOEFL Scores. The differences among the centers were significant for all of the

TOEFL scores: Listening, Structure/Writing, Reading, and Total (Frankfurt was highest

on every score).

11

Summary. In sum, the three centers did not differ in attitudes about the TOEFL-

CBT but did differ in attitudes about admissions tests, TOEFL performance, and most

other questionnaire and background variables.

______________________________


Correlations of TOEFL-CBT, Acceptance of Admissions Tests, and TOEFL Scores WithQuestionnaire and Background Variables.

The correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests,

and TOEFL with the other variables in the three centers are reported in Tables 8 to 10.

The internal-consistency reliability of the questionnaire scales are reported in Table 7.

Reliability of Questionnaire Scales. The reliability of TOEFL-CBT Acceptance

was modest, ranging from .60 to .68 over the three cities; the reliability of Acceptance of

Admissions Tests was similar, ranging from .50 to .55. The reliability of the other

questionnaire scales was also generally moderate.

TOEFL-CBT Acceptance Correlates. TOEFL-CBT Acceptance had statistically

and practically significant correlations in every center with several variables: Acceptance

of Admissions Tests, Computer Liking, Total Computer Attitude, TOEFL

Structure/Writing, and TOEFL Total. All of these correlations were positive. In

addition, in Cairo, TOEFL-CBT Acceptance correlated significantly and negatively with

Test-Irrelevant Thinking, Total Test Anxiety, and Preparation for TOEFL. And in

Frankfurt, TOEFL-CBT Acceptance also correlated significantly and positively with

Computer Confidence, Age, and Reason for Taking TOEFL�Graduate student, and

negatively with Worry, Test-Irrelevant Thinking, Total Test Anxiety, Admissions Tests

Taken, and Reason for Taking TOEFL�Undergraduate.

Acceptance of Admissions Tests Correlates. Acceptance of Admissions Tests,

besides its consistent significant correlations with TOEFL-CBT Acceptance across

centers, had sparse and inconsistent correlations with other variables. In Buenos Aires,

Acceptance of Admissions Tests correlated negatively with all TOEFL scores, except

TOEFL Structure/Writing. In Cairo, this variable correlated negatively with TOEFL

12

Reading. And in Frankfurt, it correlated positively with Computer Liking and negatively

with Sex.

TOEFL Total Correlates. TOEFL Total correlated significantly but inconsistently

with a variety of other variables, besides its consistent positive correlations with TOEFL-

CBT Acceptance in the three centers, which was noted previously. (Only the correlations

for TOEFL Total will be discussed for simplicity). TOEFL Total also correlated

consistently and negatively with Bodily Symptoms. In addition, in Buenos Aires TOEFL

Total correlated significantly and positively with Computer Liking, Computer

Confidence, Total Computer Attitude, Reason for Taking TOEFL�Graduate student,

and negatively with Total Test Anxiety and Reason for Taking TOEFL�Other. In

Cairo, TOEFL Total correlated negatively with Preparation for TOEFL. And in

Frankfurt, TOEFL Total correlated positively with Computer Liking, Computer

Confidence, Total Computer Attitude, Native Country, Native Language, and Reason for

Taking TOEFL�Graduate student, and negatively with Worry, Preparation for the

TOEFL, and Reason for Taking TOEFL�Undergraduate.

Center Differences in Correlates. It should be pointed out that many of the

differences in the patterns of correlations for the three centers may be attributable to

sampling error, and the smaller level of correlation required for significance in Frankfurt

(r = .10) relative to Buenos Aires and Cairo (r = .16), given differences in sample sizes

for these centers (403 vs. 142 and 144). However, some of the differences in correlations

were statistically and practically significant: TOEFL-CBT Acceptance�s negative

correlation with Preparation for TOEFL in Cairo and its positive correlations with Reason

for Taking TOEFL�Graduate student and Age in Frankfurt, and TOEFL Total�s

minimal correlations with Preparation for TOEFL in Buenos Aires and Reason for

Taking TOEFL�Graduate student in Cairo, and its positive correlations with Native

Country and Native Language in Frankfurt.

Summary. In brief, favorable attitudes about the TOEFL-CBT were generally

related to favorable attitudes about admissions tests, high performance on the TOEFL,

and low computer anxiety and test anxiety. These favorable attitudes were related to

minimal preparation in Cairo and to age and better education in Frankfurt. Favorable

13

attitudes about admissions tests in general were essentially unrelated to any variable

besides favorable attitudes about the TOEFL-CBT. High TOEFL performance, besides

related to favorable attitudes about the TOEFL-CBT, was also generally related to low

computer anxiety and test anxiety, minimal preparation for the TOEFL, and, in Frankfurt,

better education and native origin.

______________________________

Insert Tables 7 to 10 about here.______________________________

Discussion

Level and Correlates of TOEFL-CBT Acceptance

A central finding was that attitudes about the TOEFL-CBT appeared to be

moderately positive in the countries studied, though attitudes towards admission tests in

general seemed to be mixed in two of the countries (the relatively positive attitudes in

Egypt was the exception). This similarity in attitudes about the TOEFL-CBT in the three

countries is especially remarkable, given the differences among the countries on a wide

array of relevant variables, including attitudes about admissions tests, personality traits,

background characteristics, and TOEFL-CBT performance. Although attitudes towards

the TOEFL-CBT could be more favorable, they are still more favorable than attitudes

towards admissions tests in general. These positive attitudes about the TOEFL-CBT are

consistent with previous studies that found positive attitudes about other computer-

administered tests (Powers & O�Neill, 1993; Schmitt et al., 1993; Schmidt et al., 1978).

In all of these studies, including the present one, the positive attitudes were found after

the examinees had taken the tests. One of these studies (Powers & O�Neill, 1993) also

observed that the attitudes were less favorable before examinees took the test, suggesting

that the actual test-taking experience may have been better than the examinees

anticipated.

An equally important result was that attitudes about the TOEFL-CBT were no

more than moderately related to performance on the test in the three countries. This

outcome suggests that these attitudes do not represent an important source of irrelevant

variance on the test. These modest correlations are especially striking because the

14

TOEFL-CBT routinely reported the test scores to the examinees before they described

their attitudes about the test on the questionnaire, making it possible for the score

feedback to influence the reported attitudes.

Apart from the TOEFL scores, the associations were sparse between attitudes

about the TOEFL-CBT and their potential determinants across the three countries, and

the variables found to be related were not surprising: general attitudes towards

admissions tests, computer anxiety, and test anxiety. One of the few differences among

countries in the relationships between these attitudes and other variables stands out: the

inverse relationship with the amount of test preparation in Egypt. This result is intriguing

and deserves to be followed up. At first glance, this outcome suggests that preparing for

the TOEFL leads to unfavorable attitudes about the test. However, it is equally plausible

that examinees who prepare most do so because their ESL skills are most deficient (e.g.,

Powers & Hecht, 1982), and these poor skills or the low scores the examinees earn on the

TOEFL, despite the preparation, colors their attitudes about the test.

The moderate relationship between attitudes about the TOEFL and attitudes about

admissions tests in general�even corrected for attenuation in both variables, the

correlations ranged only from .44 to .64�along with the different patterns of

relationships between these two kinds of attitudes and the other variables in this study

make it clear that the two, though related, are distinguishable.

Level and Correlates of TOEFL Performance

Although not the focus of the study, TOEFL performance and its correlates are of

interest. The level of performance varied appreciably from country to country, consistent

with other findings (e.g., Computer-Based TOEFL Score User Guide, 1998), and within

countries was weakly related to a few variables, besides attitudes about the TOEFL-CBT.

The inverse relationship between test preparation and test performance in two

countries is noteworthy. Again, the explanation may simply be that examinees with poor

ESL skills prepare the most but this skill deficiency is still reflected in poor scores on the

test. This counterintuitive relationship between test preparation and test performance has

been found in studies of test preparation for admissions tests, such as the

15

Graduate Record Examination (GRE) General Test (Powers, 1985) and the Graduate

Management Admission Test (Powers & Hecht, 1982).

The negative association between test anxiety and test performance is consistent

with other studies of admissions tests, such as the Scholastic Assessment Test (e.g.,

Alpert & Haber, 1960; Sarason, 1957; Sarason & Mandler, 1952) and the GRE General

Test (Powers, 1988).

The negative relationship between computer anxiety and test performance is not

congruent with previous studies of computer-administered tests that found no association

(Mazzeo et al., 1992; Powers & O�Neill , 1993; Wise et al., 1989). One possible

explanation for the divergent results is that these investigations assessed the relationship

between anxiety and performance on the computer-administered test after controlling for

performance on a paper-and-pencil version of the test, but the present study did not use

this control.

The absence of an association between computer familiarity and test performance

is consistent with three of four previous studies of computer-administered tests (Powers

& O�Neill, 1993; Taylor et al., 1998; Wise et al., 1989; the exception was Lee, 1986).

One potentially important difference between all of these studies and the present one is

that, like the computer anxiety studies already discussed, they controlled for performance

on a paper-and-pencil version of the test.

Methodological Issues

In view of the frequent absence of significant results in this study, it is worth

pointing out that the statistical power was adequate (Cohen, 1988). For example, the

power to detect a medium effect size in the correlational analysis, an r of .30, using an

alpha level of .05 and a two-tail test, was .95 for both the Buenos Aires and Cairo

analyses, and .99 for the Frankfurt analysis. And the power to detect a medium effect

size in the analyses of variance that compare the three cities, an η of .24, using an alpha

level of .05, was .95.

The questionnaire used in this study also deserves comment. Because most

TOEFL test takers have sufficient grasp of English that they plan to study in North

16

America, a single English-language questionnaire was feasible. Using an English-

language questionnaire avoided the need to develop separate versions in Arabic, German,

and Spanish and then establish that they were linguistically and psychometrically

comparable, a complex, lengthy, and expensive endeavor (e.g., Hambleton, 1994).

In devising the questionnaire for this study, special attention was given to

ensuring that it would be comprehensible to examinees with limited English and function

similarly for examinees from the three countries. In addition, to deal with the possibility

that examinees might still be unable to understand particular items, Do Not Know

alternatives were provided for use in this case as well as when examinees did not have

opinions or could not report on their characteristics or behavior.3

The success of the entire questionnaire development process�writing, revising,

and pilot testing items; item analyses; and the use of Do Not Know alternatives�is

demonstrated by the relatively small proportion of Do Not Know responses (an average

of 6.9% to 10.4% in the three centers), the small proportion of scores for scales and

single-item measure that were unavailable because of missing data (a maximum of 1.3%

to 3.5% in the centers for all but three variables),4 the adequate reliability of the scales,

and the coherent pattern of correlations among the scales and single-item measures.

Conclusion

The main lesson to be drawn from this study is that the TOEFL-CBT possesses a

reasonable degree of acceptance by examinees, but there is clearly room for

improvement. However, it is probably unrealistic to expect extremely favorable attitudes

towards any high-stakes test. Steps could be taken to increase acceptance of the test,

such as improving test preparation materials. These efforts might be worthwhile from a

customer service standpoint, but there seems to be no compelling reason to do them in an

attempt to enhance the test�s construct validity, given the modest relationship between

acceptance and test performance.

17

References

Alpert, R., & Haber, R.N. (1960). Anxiety in academic achievement situations. Journalof Abnormal and Social Psychology, 61, 207-215.

Arvey, R.D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivationalcomponents of test taking. Personnel Psychology, 43, 695-716.

Baird, L.L. (1977). What graduate and professional school students think aboutadmissions tests. NCME Measurement in Education, 7 (3), 1-7.

Bandalos, D., & Benson, J. (1990). Testing the factor structure invariance of a computerattitude scale over two grouping conditions. Educational and PsychologicalMeasurement, 50, 49-60.

Benson, J., & El-Zahhar, N. (1994). Further refinement and validation of the RevisedTest Anxiety Scale. Structural Equation Modeling, 1, 203-221.

Burke, M.J., Normand, J., & Raju, N. S. (1987). Examinee attitudes toward computer-administered ability tests. Computers in Human Behavior, 3, 95-107.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).Hillsdale, NJ: Erlbaum.

Computer-based TOEFL score user guide, 1998-1999 (1998). Princeton, NJ:Educational Testing Service.

Eignor, D., Taylor, C., Kirsch, I., & Jamieson, J. (1998). Development of a scale forassessing the level of computer familiarity of TOEFL examinees (TOEFLResearch Report 60; ETS Research Report 98-7). Princeton, NJ: EducationalTesting Service.

Fiske, D.W. (1967). The subject reacts to tests. American Psychologist, 22, 287-296.

Gilliand, S.W. (1994). Effects of procedural and distributive justice on reactions to aselection system. Journal of Applied Psychology, 79, 691-701.

Hambleton, R.K. (1994). Guidelines for adapting educational and psychological tests: Aprogress report. European Journal of Psychological Assessment, 10, 229-244.

Jamieson, J., Taylor, C., Kirsch, I., & Eignor, D. (1999). Design and evaluation of acomputer-based TOEFL tutorial (TOEFL Research Report 62; ETS ResearchReport 99-01). Princeton, NJ: Educational Testing Service.

18

Kernan, M.C., & Howard, G. S. (1990). Computer anxiety and computer attitudes: Aninvestigation of construct and predictive validity issues. Educational andPsychological Measurement, 50, 681-690.

Kirsch, I., Jamieson, J., Taylor, C., & Eignor, D. (1998). Computer familiarity amongTOEFL examinees (TOEFL Research Report 59; ETS Research Report 98-6).Princeton, NJ: Educational Testing Service.

Lee, J.A. (1986). The effects of past computer experience on computerized aptitude testperformance. Educational and Psychological Measurement, 46, 727-733.

Loyd, B.H., & Gressard, C. (1984). Reliability and factorial validity of computer attitudescales. Educational and Psychological Measurement, 44, 501-505.

Mazzeo, J., Druesne, B., Raffeld, P., Checketts, K., & Muhlstein, A. (1992).Comparability of computer and paper-and-pencil scores for two CLEP generalexaminations (College Board Report 91-5; ETS Research Report 92-14). NewYork: College Entrance Examination Board.

Powers, D.E. (1985). Effects of coaching on GRE Aptitude Test scores. Journal ofEducational Measurement, 22, 121-136.

Powers, D.E. (1988). Incidence, correlates, and possible causes of test anxiety ingraduate admissions testing. Advances in Personality Assessment, 7, 49-75.

Powers, D.E., & Hecht, L.W. (1982). Preparing for the Graduate ManagementAdmission Test: A descriptive study of candidates� use of alternative methods oftest preparation (GMAC Research Report 82-2; ETS Research Report 82-36).Princeton, NJ: Educational Testing Service.

Powers, D.E., & O�Neill, K. (1993). Inexperienced and anxious computer users: Copingwith a computer-administered test of academic skills. Educational Assessment, 1,153-173.

Robinson, J.P., Shaver, P.R., & Wrightsman, L.S., (Eds.). (1991). Measures ofpersonality and social psychological attitudes. San Diego: Academic Press.

Rowan & Blewitt Incorporated. (1998). Educational Testing Service nationwide opinionsurvey final report. Washington, DC: Author.

Sarason, I.G. (1957). Test anxiety, general anxiety, and intellectual performance.Journal of Consulting Psychology, 21, 485-490.

Sarason, I.G. (1984). Stress, anxiety, and cognitive interference: Reactions to tests.Journal of Personality and Social Psychology, 46, 929-938.

19

Sarason, S.B., & Mandler, G. (1952). Some correlates of test anxiety. Journal ofAbnormal and Social Psychology, 47, 810-817.

Schmidt, F.L., Urry, V. W., & Gugel, J. F. (1978). Computer assisted tailored testing:Examinee reactions and evaluations. Educational and PsychologicalMeasurement, 38, 265-273.

Schmitt, N., Gilliand, S. W., Landis, R. S., & Devine, D. (1993). Computer-based testingapplied to selection of secretarial applicants. Personnel Psychology, 46, 149-165.

Smither, J.W., Reilly, R. R., Millsap, R. E., Pearlman, K., & Stoffey, R. W. (1993).Applicant reactions to selection procedures. Personnel Psychology, 46, 49-76.

Spielberger, C.D. (1980). Preliminary professional manual for the Test AnxietyInventory. Palo Alto, CA: Consulting Psychologists Press.

Stricker, L.J. (1982). Identifying test items that perform differentially in populationsubgroups: A partial correlation index. Applied Psychological Measurement, 6,261-273.

Taylor, C., Jamieson, J., Eignor, D., & Kirsch, I. (1998). The relationship betweencomputer familiarity and performance on computer-based TOEFL test tasks(TOEFL Research Report 61; ETS Research Report 98-8). Princeton, NJ:Educational Testing Service.

Wilder, G., Mackie, D., & Cooper, J. (1985). Gender and computers: Two surveys ofcomputer-related attitudes. Sex Roles, 13, 215-228.

Wise, S.L., Barnes, L. B., Harvey, A. L., & Plake, B. S. (1989). Effects of computeranxiety and computer experience on the computer-based achievement testperformance of college students. Applied Measurement in Education, 2, 235-241.

20

Footnotes

1Data collection in Buenos Aires and Cairo extended beyond these dates but was

unsystematic and unusable.

2The dummy codes for Reasons for Taking TOEFL were: to be an undergraduate

= 1, all others = 0; to be a graduate student = 1, all others = 0; some other reason = 1, all

others = 0.

3Not comprehending an item because of its language and not having an opinion or

being able to report on a personal characteristic or behavior obviously have different

psychological meaning, but providing separate alternatives would have unduly

complicated the examinee�s task in completing the questionnaire.

In order to preserve as much usable data as possible, Do Not Know responses

were retained in scoring items on scales and assigned a neutral score (0, with keyed

alternatives scored +1 and �1), a common practice in scoring don�t know, uncertain, and

neutral responses on personality and attitude scales (e.g., Robinson, Shaver, &

Wrightsman, 1991).

Retaining these responses but assigning them the mean score for the sample, as

was done with the omitted items, would be inappropriate when they signify that

examinees simply do not have opinions or cannot report on their characteristics or

behavior. Indeed, this seems to be the predominant meaning of these responses, not lack

of comprehension because of language difficulties, judging from the weak or minimal

correlations of Do Not Know responses with the TOEFL scores. For example, the total

number of Do Not Know responses correlated �.08, -.15, and -.18 with TOEFL Total in

the three centers (only the -.18 correlation, in Buenos Aires, was statistically and

practically significant).

Excluding Do Not Know responses and not obtaining a score for a scale when an

examinee had such responses would result in an excessive amount of missing data. For

instance, the percentage of examinees with one or more Do Not Know responses on the

original 12-item TOEFL-CBT Acceptance scale ranged from 60.6% to 71.4% in the

centers.

21

4For the three exceptions, Computer Familiarity, Preparation for TOEFL, and

Admissions Tests Taken, the maximum percentage of missing data was 4.5% to 24.6% in

the three centers.

22

Author Note

Thanks are due to Kurt F. Geisinger, Theresa Hoff Macan, Neal W. Schmitt, and

Thomas E. Scruggs for providing measures of attitudes about tests; Pamela McPartland-

Fairman for reviewing the questionnaire; Eileen Hansen, Mary Mora, and Allison Rice

for assisting in the pilot testing of the questionnaire; Baker Evans, M. Hashem, Roger

Hermodsson, Edwin Kolen, Selma Laporte, Bea Lichtensteiger, Mark Porzuc, and Linda

A. Silverstein for supervising the data collection in the test centers; Edward J. Kelly,

Sandra Lee, and Edward S. Paul for providing logistical advice and support; Debra E.

Friedman for coordinating the data collection; Patricia E. Carey, Karen A. Copper, Hai

Jiang, and Cindy Nguyen for providing TOEFL program data; Min Hwei Wang for doing

the data analysis; and Brent Bridgeman and Howard Wainer for reviewing a draft of this

report.

Table 1

Percentage Distributions of Background Variables for Participants and Nonparticipants in Each Testing Center

_______________________________________________________________________________________________________________________________________________________ Participants in

Buenos Aires Cairo Frankfurt Testing Centers

Variable Participants Nonparticipants χ2 W Participants Nonparticipants χ 2 W Participants Nonparticipants χ 2 W χ 2 W_______________________________________________________________________________________________________________________________________________________

Sex (N = 143) (N = 152) 3.12 .10 (N = 138) (N = 75) .06 .02 (N = 402) (N = 182) .00 .00 17.29** .16

Male 49.0 59.2 73.2 74.7 60.0 59.9

Female 51.0 40.8 26.8 25.3 40.0 40.1

Native Country (N = 138) (N = 150) .05 .01 (N = 131) (N = 67) .61 .06 (N = 389) (N = 176) 19.84** .19 2.71 .06

Same 86.2 85.3 90.8 94.0 91.0 77.3

Different 13.8 14.7 9.2 6.0 9.0 22.7

Native Language (N = 134) (N = 148) .88 .06 (N = 127) (N = 65) .26 .04 (N = 389) (N = 170) 19.54** .19 4.40 .08

Same 91.8 94.6 96.9 95.4 91.3 77.6

Different 8.2 5.4 3.1 4.6 8.7 22.4

Planning Study inUnited States or Canada (N = 143) (N = 151) .29 .03 (N = 139) (N = 74) .01 .01 (N = 402) (N = 182) .06 .01 5.74 .09

Yes 68.5 65.6 65.5 66.2 58.2 57.1

No 31.5 34.4 34.5 33.8 41.8 42.9

Table 1 (continued.)

_______________________________________________________________________________________________________________________________________________________ Participants in

Buenos Aires Cairo Frankfurt Testing Centers

Variable Participants Nonparticipants χ2 W Participants Nonparticipants χ 2 W Participants Nonparticipants χ 2 W χ 2 W_______________________________________________________________________________________________________________________________________________________

Reason for takingTOEFL (N = 144) (N = 152) .93 .06 (N = 138) (N = 74) 4.00 .14 (N = 402) (N = 182) .04 .01 23.82** .19

To be undergraduate 18.1 20.4 28.3 31.1 28.9 28.0

To be graduate Student 49.3 52.0 44.9 54.1 55.5 56.0

Other reasona 32.6 27.6 26.8 14.9 15.7 15.9_______________________________________________________________________________________________________________________________________________________

aTo enter a school other than a college or university, to become licensed to practice my profession in the USA or Canada, to demonstrate my proficiency in English to the

company for which I work or plan to work, other than the above.

*p < .05; ** p < .01.

Table 2

Means of Age and TOEFL Scores for Participants and Nonparticipants in Each Testing Center

_______________________________________________________________________________________________________________________________________________________Participants in

Buenos Aires Cairo Frankfurt Testing CentersVariable Participants Nonparticipants F η Participants Nonparticipants F η Participants Nonparticipants F η F η

Mean S. D. Mean S. D. Mean S. D. Mean S. D. Mean S. D. Mean S. D._______________________________________________________________________________________________________________________________________________________

Age 25.95 5.66 26.86 6.08 1.75 .08 24.83 6.32 24.27 6.18 .40 .04 23.96 4.33 24.14 4.80 .20 .02 8.37** .15

TOEFL ListeningScore 22.03 4.72 22.26 5.14 .16 .02 20.25 5.46 18.09 6.43 6.77**.17 25.28 3.04 23.41 4.11 37.62** .25 94.82** .47

TOEFL Structure/ Writing Score 23.96 3.47 23.37 4.15 1.14 .06 20.65 4.37 21.03 4.86 1.74 .09 25.41 3.42 23.58 4.19 26.04** .21 45.62** .34

TOEFL ReadingScore 22.83 4.48 24.46 3.57 1.51 .07 21.93 4.75 19.11 5.08 5.44* .16 25.27 3.47 23.89 4.27 21.06** .19 89.79** .46

TOEFL TotalScore 229.44 37.81 233.64 38.58 .90 .06 209.44 42.43 194.12 48.42 5.80* .16 253.19 29.00 236.24 37.14 35.74** .24 94.02** .46

_______________________________________________________________________________________________________________________________________________________

Note. The Ns for the participants and nonparticipants are 144 and 152 in Buenos Aires, 142 and 75 in Cairo, and 403 and 182 in Frankfurt.

* p < .05; ** p < .01.

Table 3

Percentage Distributions of Responses to Items on TOEFL-CBT Acceptance in Each Testing Center

___________________________________________________________________________________________________________________________________________________ Participants in

Buenos Aires Cairo Frankfurt Testing CentersDo Not Do Not Do Not Do Not Do Not Do Not

Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W______________________________________________________________________________________________________________________________________

2. The different types of questions in theTOEFL are good for testing the Englishskills needed in school. 144 80.6 8.3 11.1 140 78.6 14.3 7.1 400 76.8 14.0 9.3 4.34 .08

3. The TOEFL examines English skills Iwill never need. 144 13.9 72.2 13.9 141 19.9 69.5 10.6 399 8.5 80.5 11.0 14.66** .15

4. A school or company can learn a lotabout a person�s English ability fromthe TOEFL scores. 142 47.9 35.9 16.2 141 61.7 26.2 12.1 401 42.6 33.2 24.2 19.20** .17

5. A low score on the TOEFL showsthat your English ability is limited. 144 47.2 40.3 12.5 140 50.0 41.4 8.6 400 49.5 36.0 14.5 4.04 .08

6. The TOEFL was difficult becauseI had to use a computer. 143 11.2 87.4 1.4 142 12.7 85.9 1.4 402 9.0 87.8 3.2 3.90 .08

7. My English skills are better than myTOEFL scores demonstrate. 142 49.3 18.3 32.4 140 55.7 21.4 22.9 402 31.3 16.4 52.2 46.00** .26

Table 3 (continued)

___________________________________________________________________________________________________________________________________________________ Participants in

Buenos Aires Cairo Frankfurt Testing CentersDo Not Do Not Do Not Do Not Do Not Do Not

Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W______________________________________________________________________________________________________________________________________

8. The TOEFL did not give me a goodopportunity to demonstrate my Englishability. 144 35.4 38.9 25.7 142 50.0 38.7 11.3 400 32.0 44.3 23.8 19.55** .17

9. The TOEFL tells how well people canuse English in school. 144 36.8 28.5 34.7 141 61.7 27.0 11.3 402 40.0 37.6 22.4 35.27** .23

11. The English skills tested by the TOEFLare different from the English neededin school. 142 16.9 46.5 36.6 142 42.3 48.6 9.2 402 33.1 49.3 17.7 44.55** .25

___________________________________________________________________________________________________________________________________

Note. The favorable response is underlined.

* p < .05; ** p < .01.

Table 4

Percentage Distributions of Responses to Items on Acceptance of Admissions Tests in Each Testing Center

_______________________________________________________________________________________________________________________________________________________ Participants in

Buenos Aires Cairo Frankfurt Testing CenterDo Not Do Not Do Not Do Not Do Not Do Not

Item N Agree Agree Know N Agree Agree Know N Agree Agree Know χ2 W_________________________________________________________________________________________________________________________________________

3. People who receive high scores onuniversity admissions tests will besuccessful in school. 138 17.4 51.4 31.2 139 40.3 30.2 29.5 400 15.5 53.5 31.0 43.30** .25

4. My scores on university admissionstests accurately demonstrate my skillsand abilities 141 24.8 37.6 37.6 142 40.1 32.4 27.5 399 17.8 37.8 44.4 30.57** .21

5. University admissions tests are a goodway to learn about a person�s skills andabilities. 140 45.0 26.4 28.6 142 62.0 18.3 19.7 399 34.3 34.8 30.8 34.11** .22

________________________________________________________________________________________________________________________________________

Note. The favorable response is underlined.

* p < .05; ** p < .01.

Table 5

Means of Questionnaire Variables in Each Testing Center

_______________________________________________________________________________________

Buenos Aires Cairo Frankfurt Variable N Mean S. D. N Mean S. D. N Mean S. D. F η

_______________________________________________________________________________________

TOEFL-CBT Acceptance 144 2.38 3.45 142 2.29 3.92 403 2.55 3.77 .31 .03

Acceptance of Admissions Tests 140 -.29 1.75 142 .63 1.75 401 -.59 1.66 27.26** .27

Computer Liking 144 2.84 2.54 140 3.34 2.06 401 2.36 2.63 8.50** .16

Computer Confidence 144 4.56 2.86 140 4.81 2.75 399 4.45 3.08 .73 .05

Total Computer Attitude 144 7.40 4.87 140 8.15 4.20 399 6.81 5.20 3.92* .11

Worry 142 -.31 2.29 142 -.04 2.21 400 -.60 2.27 3.41* .10

Tension 142 1.48 2.89 140 1.06 2.72 398 .89 3.21 1.96 .08

Test-Irrelevant Thinking 141 -2.66 2.07 140 -1.56 2.45 398 -2.26 2.41 7.94** .15

Bodily Symptoms 141 -2.57 2.36 141 -2.05 2.53 400 -3.36 2.07 20.11** .24

Total Test Anxiety Scale 139 -4.10 6.14 140 -2.60 6.64 398 -5.33 6.12 9.38** .16

Computer Familiarity 125 15.60 2.13 107 14.34 3.28 357 14.68 2.27 8.99** .17

Admissions Tests Taken 121 1.35 .68 115 1.49 .88 385 1.32 .70 2.17 .08_______________________________________________________________________________________

* p < .05; ** p < .01.

Table 6

Percentage Distributions of Preparation for TOEFL in Each Testing Center

______________________________________________________________________________

Variable Buenos Aires Cairo Frankfurt χ2 W(N = 121) (N = 119) (N = 388)

______________________________________________________________________________

0 to 5 hours 25.6 48.7 53.1 28.13** .21

6 hours or more 74.4 51.3 46.9______________________________________________________________________________

*p < .05; ** p < .01

Table 7

Reliability of Questionnaire Scales in Each Testing Center

_____________________________________________________________________________________________

Scale Buenos Aires Cairo Frankfurt_____________________________________________________________________________

TOEFL-CBT Acceptance .60 .65 .68

Acceptance of Admissions Tests .58 .50 .55

Computer Liking .69 .50 .66

Computer Confidence .68 .64 .74

Worry .50 .36 .46

Tension .64 .49 .70

Test-Irrelevant Thinking .71 .65 .75

Bodily Symptoms .50 .50 .54

Computer Familiarity .36 .69 .60______________________________________________________________________________

Note. The Ns appear in Table 5.

Table 8

Correlations of TOEFL-CBT Acceptance, Acceptance of Admissions Tests, and TOEFL Scores with Other Questionnaire and Background Variables

in Buenos Aires

_______________________________________________________________________________________________________________________

TOEFL TOEFL-CBT Acceptance of Listening Structure/Writing Reading Total

Variable Acceptance Admissions Tests Score Score Score Score________________________________________________________________________________________________________________________________________________

TOEFL-CBT Acceptance - .35** .14 .19* .11 .16*

Acceptance of Admissions Tests .35** - -.18* -.12 -.17* -.18*

Computer Liking .16* -.08 .22** .18* .11 .19*

Computer Confidence .13 -.14 .30** .30** .23** .31**

Total Computer Attitude .16* -.12 .29** .27** .20* .28**

Worry .02 -.05 -.07 -.09 -.03 -.07

Tension -.03 -.03 -.08 -.07 -.03 -.07

Test-Irrelevant Thinking -.04 -.01 -.11 -.02 -.05 -.07

Bodily Symptoms -.03 -.02 -.18* -.26** -.22** -.24**

Total Test Anxiety Scale -.03 -.04 -.18* -.18* -.13 -.18*

Computer Familiarity .06 -.09 .16 .22** .10 .18*

Table 8 (continued)

________________________________________________________________________________________________________________________



Preparation for TOEFL -.14 .03 -.09 -.04 -.12 -.10

Admissions Tests Taken -.08 .07 -.04 -.09 .05 -.02

Age -.02 -.05 -.13 .01 -.18* -.12

Sex -.07 -.06 -.13 -.20* -.06 -.14

Native Country .11 .07 -.07 -.13 -.10 -.11

Native Language .06 .00 -.02 .07 -.07 -.01

Planning to Study in United States or Canada -.09 .12 -.07 -.10 -.03 -.07

Reason for Taking TOEFL-Undergraduate -.09 -.03 -.08 -.25** -.07 -.14

Reason for Taking TOEFL-Graduate Student .04 -.11 .16 .37** .19* .26**

Reason for Taking TOEFL-Other .03 -.14 -.10 -.20* -.15 -.16*

Table 8 (continued)

________________________________________________________________________________________________________________________



TOEFL Listening Score .14 -.18* - .68** .70** .90**

TOEFL Structure/Writing Score .19* -.12 .68** - .72** .88**

TOEFL Reading Score .11 -.17* .70** .72** - .91**

TOEFL Total Score .16* -.18* .90** .88** .91** -________________________________________________________________________________________________________________________

Note. Ns range from 118 to 144.

* p < .05 (two-tail); ** p < .01 (two-tail).

Table 9


in Cairo

_______________________________________________________________________________________________________________________



TOEFL-CBT Acceptance - .25** .40** .32** .17* .34**

Acceptance of Admissions Tests .25** - -.02 -.11 -.20* -.12

Computer Liking .21** .10 .13 .13 .00 .10

Computer Confidence .11 .15 .12 .17* -.04 .09

Total Computer Attitude .18* .15 .14 .18* -.03 .11

Worry -.05 -.05 .10 -.07 -.01 .02

Tension -.09 -.16 -.01 .00 .05 .01

Test-Irrelevant Thinking -.20* .05 -.06 -.21** -.08 -.13

Bodily Symptoms -.13 .09 -.19* -.20* -.11 -.19**

Total Test Anxiety Scale -.18* -.03 -.07 -.18* -.05 -.11

Computer Familiarity .16 .12 .14 .05 -.09 .05

Table 9 (continued)

________________________________________________________________________________________________________________________



Preparation for TOEFL -.21* .05 -.36** -.17 -.30** -.33**

Admissions Tests Taken -.13 -.04 .01 -.14 .04 -.03

Age -.08 .00 -.22** .13 -.08 -.08

Sex -.02 .03 .01 -.10 .15 .02

Native Country -.10 -.13 -.19* -.02 -.09 -.13

Native Language -.02 -.14 .02 .15 .03 .07

Planning to Study in United States or Canada .09 -.08 .09 .00 .10 .08

Reason for Taking TOEFL--Undergraduate .14 .08 .09 -.15 .08 .05

Reason for Taking TOEFL--Graduate Student -.10 -.14 .16 .01 -.16 -.12

Reason for Taking TOEFL--Other .03 .08 -.16 .13 .10 .09

Table 9 (continued)

________________________________________________________________________________________________________________________



TOEFL Listening Score .40** -.01 - .67** .59** .88**

TOEFL Structure/Writing Score .32** -.11 .67** - .67** .88**

TOEFL Reading Score .17* -.20* .59** .67** - .86**

TOEFL Total Score .34** -.12 .88** .88** .86** -________________________________________________________________________________________________________________________



Table 10


in Frankfurt

_______________________________________________________________________________________________________________________



TOEFL-CBT Acceptance - .39** .21** .25** .15** .23**

Acceptance of Admissions Tests .39** - -.07 -.09 -.09 -.09

Computer Liking .14** .11* .14** .13** .07 .13**

Computer Confidence .18** .06 .23** .19** .15** .21**

Total Computer Attitude .17** .09 .21** .18** .13** .19**

Worry -.13** .05 -.11* -.17** -.10* -.14*

Tension -.01 -.02 .08 .02 .11* .08

Test-Irrelevant Thinking -.14** -.01 -.09 -.09 -.07 -.10

Bodily Symptoms -.10 .08 -.12* -.16** -.04 -.12*

Total Test Anxiety Scale -.13** .03 -.07 -.13** -.02 -.08

Computer Familiarity .08 .07 .06 .07 .03 .06

Table 10 (continued)

________________________________________________________________________________________________________________________



Preparation for TOEFL .01 .09 -.19** -.09 -.13** -.15**

Admissions Tests Taken -.14** -.04 .06 -.03 .03 -.02

Age .15** .04 -.03 .10* -.09 -.01

Sex -.14** -.10* -.07 -.08 .04 -.04

Native Country -.08 -.09 .25** .17** .19** .23**

Native Language -.07 -.07 .26** .16** .18** .23**

Planning to Study in United States or Canada .00 .05 -.02 .02 -.06 -.02

Reason for Taking TOEFL--Undergraduate -.15** -.04 -.13** -.20** -.10* -.16**

Reason for Taking TOEFL--Graduate Student .13** -.02 .17** .21** -.14** .20**

Reason for Taking TOEFL--Other .01 .08 -.08 -.03 -.07 -.07

Table 10 (continued)

________________________________________________________________________________________________________________________



TOEFL Listening Score .21** -.07 - .66** .64** .87**

TOEFL Structure/Writing Score .25** -.09 .64** - .64** .88**

TOEFL Reading Score .15** -.09 .66** .64** - .88**

TOEFL Total Score .23** -.09 .87** .88** .88** -________________________________________________________________________________________________________________________



Appendix

examinees’ attitudes about the toefl-cbt, possible ...examinees’ attitudes about the toefl-cbt,...

Documents