DOCUMENT RESUME

ED 241 581 TN 840 128

AUTHOR McKinley, Robert L.; Reckase, Mark D.TITLE An Extension of the Two-Parameter Logistic Model to

the Multidimensional Latent Space.INSTITUTION American Coll. Testing Program, Ioisa City, Iowa.SPONS AGENCY Office of Naval Research, Arlington, Va. Personnel

and Training Research Programs Office.REPORT NO ONR83-2PUB DATE Aug 83CONTRACT N00014-81-K0817NOTE 33p.PUB TYPE Reports - Research/Technical (143)

EDRS PRICE MFO1 /PCO2 Plus Postage.DESCRIPTORS *Estimation (Mathematics); *Latent Trait Theory;

*Mathematical Models; Maximum Likelihood Statistics;.:14.4 Measurement Techniques; *Statistical AnalysisIDENTIFIERS Information Function (Tests); *Multidimensional

Approach; *Two Parameter Model

ABSTRACTItem response theory (IRT) has proven to be a very

powerful asu useful measurement tool. However, most of the IRT modelsthat have been proposed, and all of the models commonly used, requirethe assumption of unidimensionality, which prevents their applicationto a wide range of tests. The few models that have been proposed foruse with multidimensional data have not been developed to the pointthat they can be applied in actual testing situations. The purpose ofthis report is to present a model for use with multidimensional dataand to discuss some of its characteristics. This discussion willinclude information on the interpretation of the model parameters,the sufficient statistics for the model parameters, and theinformation function for the model. In addition, the estimation ofthe parameters of the model using the maximum likelihood estimationtechnique is also discussed. (Author/FN)

************************************************************************ Reproductions supplied by EDRS are the best that C84 be made *

* from the original document. ************************************************************************

IICC)

An Extension of the Two-Parameter Logistic Model4-.

-4-N0Lei Robert L McKinley

andMark D. Reckase

to the Multidimensional Latent Space

Research Report ONR83-2August 1983

U.S. DEPARTMENT OF EDUCATIONNATIONAL INSTITUTE OF EDUCATION

EDUCATIONAL RESOURCES INFORMATION

CENTER IERICItX Mrs document has been reproduced as

Mewed horn the OertOn or Orgenitaloonongebelinit It.

U Rohr Changes have been made to improveteotoduceon nuAtY,

The American College Testing ProgramResident Programs Department

Iowa City, Iowa 52243

Prepared under Contract No. N00014-81-K0817with the Personnel and Training Research Programs

Psychological Sciences DivisionOffice of Naval Research

Approved for public release, distribution unlimited.Reproduction in whole or in part is permitted forany purpose of the United States Government

2

Points of view or op roonSeratechn this OW,.men; do not necessarily teolosent MEOW MEpositron or poky

"PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

alt. ,.. )c

/144.A.oi t42.41..ea-t

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER {ERIC."

SECURITY cLAsstric ATION Or THIS PAGE (WMon Data ani*md)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONSBEFORE COMPLETING FORM

REPORT NUMBER

ONR83-2

2. GOVT ACCESSION NO, 3. RECIPIENT'S CATALOG NUMBER

TITLE (imi sublid.)

An Extension of the Two-ParameterLogistic Model to the Multidimensional.

Latent Space.

5. TYPE or RE*ORT 0 PER100 COVERED

Technical Report

6 PERroR6..NG ORG. REPORT NUMBER

7 AUTh R(.)

Robert L. McKinley andMark D. Reckase

6 CONTRACT OR GRANT NUMBER(a)

N00014-81-K0817

9 pER,091.4.NG ORGANIZATION MANE ANO AOOREss

American College Testing ProgramP.O. Box 168Iowa City, IA 52243

10 PROGRAM ELENENT.RoJECT. TASKAREA 910191( UNIT NUMBERS

P.E.: 61153N Proj.: RRO4

T.A.: 042-04-01W,V.! NR1S0-474

II CONTROLLING or rice Name AND woonessPersonnel and Training Research ProgramsOffice of N,:val ResearchArlington, VA 22217

12. 'Kew,. DmAugust 1983

3 mumeenoreAces18

1 .40...TOPHNG AGENCY NAME 6 ADDRESS(// doiftme hem Cerbirelline owe.) Is SECURITY CLASS. (cI HO* tepees)

unclassified

1st DECLASS.C.CATIoN/GOTINGRAMNGSCHEDULE

14 DISTRIBUTION STATEMENT (or Nirr Raper!)

Approved for public release, distribution unlimited. Reproductionin whole or part is permitted for any purpose of the United Statesgovernment.

17. 037 Oti SIMON STATEMENT Col lb. ob.boei entwod in eloeb 20. el di(104rne beet Rboti)

IS SUPPLEMENTARY NOTES

IS. KEY 190190S ICentInu on ftworo aid. IS A art bid IdblUY 46'001 mlibba)Latent trait modelsMultidimensional latent trait modelsMultidimensional two-parameter logistic modelEstimation

20 ABSTRACT (CoAHHH on 'ifo.. olds ll nocooNmy sit Adontny by Woes minium)

A multidimensional extension of the two-parameter logistic latenttrait model is presented and some of its characteristics are discussed.In addition, sufficient statistics for the parameters of the model arederived, as is the information function. Finally,the estimation of theparameters of the model using the maximum likelihood estimation techniqueis also discussed.

DD 1 rJAN13 1473 COITION Of I NOV SS IS OBSOLETE

S/N 0102 0-014- 6601 n on *Vino, n in IsOn "SW..

-04

CONTENTSPage

Introduction

The Model and Its Characteristics 1

The Model . 1

Interpretation of the Model Parameters 2

Sufficient Statistics . ... 5

Definition. 5

Sufficient Statistic for the Ability Parameter 6

Sufficient Statistics for the Item Parameters 6

Information Function 6

Definition 6

Information Functions for the NM, Model 8

Maximum Likelihood Estimation 9

Discussion 13

Directional Derivatives 13

Interpretation of the Model Parameters 14

Sufficient Statistics 16

Information Function 17

Maximum Likelihood Estimation 17

Summary 17

References ... 19

4

An Extension of the Two-Parameter Logistic Modelto the Multidimensional Latent Space

Item response theory (IRT) has proven to be a very powerful and usefulmeasurement tool. However, most of the 1RT models that have been proposed,and all of the models commonly used, require the assumption of unidimensionality,which prevents their application to a wide range of tests. The few modelsthat have been proposed for use with multidimensional data have not beendeveloped to the point that they can be applied in actual testing situations.The purpose of this report is to present a model for use with multidimensionaldata and to discuss some of its characteristics. This discussion willinclude information on the interpretation of the model parameters, thesufficient statistics for the model parameters, and the information functionfor the model. In addition, a procedure for estimating the parameters ofthe model will be discussed.

The Model and Its Characteristics

The Model

The model proposed in this report is a multidimensional extension ofthe two-parameter logistic model. The two-parameter logistic (2PL) model,proposed by Birnbaum (1968), is given by

exp(Dai(Oi - bi))P (e ) =

1 exp(Dai(Oi - bi)) '(1)

where ai is the discrimination parameter for item i, b. is the difficultyparameter for item i, 0. is the ability parameter for examinee j, 27.(6.) is

the probability of a correct response to item i by examinee j, and D = 1.7.The multidimensional extension of the 2PL model (M2PL), as presented byMcKinley and Reckase (1982), is given by

Pile 0 )j 1 exp(d + a o )

exp(di a o )

(2)

where a.1is a row vector of discrimination parameters for item i, 0. is a

column vector of ability parameters for examinee j, P.(83) is the probability1 -

of a correct response to item i by examinee j, and di is given by

d = -E1

aik

bik '

(3)

where aik is the discrimination parameter for item i on dimension k, bik is

the difficulty parameter for item i on dimension k, and m is the number ofdimensions being modeled. The di term, then, is related to item difficulty,

1

5

2

but is not e difficulty parartter in the same sense as the bi parameter isin the unidimensional model.

Interpretation of the Model Parameters

The interpretation of the parameters of unidimensional IRT models isclosely tied to the item characteristic curve (the regression of item scoreon ability). The item difficulty parameter is defined as the point on theability scale where the point of inflection of the item characteristiccurve (ICCroccurs. This is equivalent to saying the item difficulty valueis the point on the ability scale where the second derivative of the ICCfunction is equal to zero. For the 2PL model, the second derivative isgiven by

OPD2a 2PQ(1 - 2P) ,

de 32

where P is the probability of a correct responseQ = I-P, and a. and D are as previously defined.

side of (4)P = 1.0 andcases whereoccurs at P

(4)

to item i given ability j,Setting the right hand

equal to zero yields a solution of P = Q = 0.5. Of course,P = 0.0 are also solutions, but these represent degenerate0 = + and 0 = -m, respectively. Thus, the point of inflection=0.5,whichoccurswhereb.1 = 0.. The difficulty of an item

for the 2PL model, then, is the point on the ability scale which yields aprobability of a correct response equal to 0.5. Figure 1 shows a typicalICC for the 2PL model. The dotted line shows the relationship among theitem difficulty value, ability, and the probability of a correct response.

Figure 1

A Typical ICC for the 2P1. Model

-3.0 -2.0 -1.0 0.0THETA

1.0 2.0 3.0

3

The item discrimination parameter is related to the slope of the ICCat the point of inflection. The slope of the ICC at the point of inflectionis found by taking the first derivative of the ICC and evaluating it at thepoint of inflection. For the 2PI. model the first derivative is given by

6P

60m.Da

iPQ , (5)

where all the terms are as previously defined. It was previously foundthat the pdint of inflection for the 2PI. model occurs where P = 0.5.Substituting 0.5 into (5) yields a slope at the point of inflection ofDai/4.

Difficulty and discrimination are defined somewhat differently formultidimensional models. To begin with, the response function (the model)defines a multidimensional item response surface (IRS) rather than a curve.This surface may have many points of inflection, and the points of inflectionmay vary depending on the direction relative to the 0-axes. Because of this,the item parameters for the M2P1, model arc defined in terms of directionalderivatives (Kaplan, 1952).

For multidimensional models, difficulty is defined as the locus ofpoints of inflection of the IRS for a particular direction. This is foundby taking the second directional derivative of the response function,setting it .equal to zero, and solving for the 0-vector. The second directionalderivative for the M2PI. model is given by

601602

62p 62p 62PV 2P = "1" cos +i COSO2 . . cos$1 COSOm

4)

6012oviovm

62P62p

- cee.2 conch . , cos42 ces+m"2481 uovvin

62p

602m

cos24m

(6)

where f represents the vector of angles with respect to each of the m axes.Solving the derivatives in (6) and simplifying yields

22P = PQ(1 - 2P) (alcos#1 + .92cos02 + . . . + a cos. )Of

m m

7

(7)

4

'Setting (7) equal to zero and solving yields three solutions--P = 0.0,P = 0.5, and P = 1.0. The solutions 0.0 and 1.0 represent degenerate caseswhere 0 = I'm. P = 0.5 occurs when the exponent of the M2PL model is zero.That is P = 0.5 when

d + 3101 -I- 3202 + . . . -I- am0m = 0. (8)

In the two-dimensional case this is the equation for a line.

For the M2PL model, as can be seen from the above derivations, thedirection, I, falls out of the equations. Item difficulty for the M2PL modelis the same for all directions of travel. This is not necessarily the casefor all multidimensional models.

Figure 2 shows a typical response surface for the M2PL model in thetwo-dimensional case. The dotted line indicates the line of difficulty.In the m-dimensional case (8) is the equation for a hyperplane.

Figure 2

A Typical Response Surface for the M2PL Model

5

For multidimensional models, item discrimination is a function of theslope of the IRS at the locus of points of inflection in a particulardirection. This is obtained by taking the first directional derivative ofthe response function and evaluating it at the locus of points of inflection.For the H2PL model the first directional derivative is given by

6P 6P 6PGAP = cosi1

602cos02 . . . 4- w -cosOm

m601(9)

where 4 represents the vector of angles of the direction in the 0-spacewith respect to each of the m axes. For the two-dimensional case (9) isgiven by

V0P = aIPQ cos 0 a2PQ sin 0 . (10)

where Q is the angle with the 01 axis. When 0 = 0° (direction parallel to81 axis) the slope is a1PQ, and when $ = 90° (parallel to 02 axis) the slopeis a2PQ. In general, when the direction is parallel to the Om axis, the slope

is amPQ. Since P = Q = 0.5 at the line of inflection, the slope parallel to

them

axis at those points is am/4. In the unidimensional case $ = 0°, and

the slope of the ICC at the point of inflection is Da/4.

Sufficient Statistics

Definition Assume that there exists some distribution that is ofknown form except for some unknown parameter 0, and that x represents a setof observations from that distribution. Also assume that S(x) is somestatistic which is a function of x. If S(x) is a sufficient statistic for6, then it must be possible to factor the probability function of x, P(x10)into the form:

P(xIO) = f[S(x)181g(x) (11)

In this form it is easy to see that g(x) is independent of 6, and so providesno information about O. Selection of 0 to maximize the probability of x istantamount to selecting 8 to maximize the probabilty of S(x).

In item response theory x is typically a resnonse string, either byone examinee to a set of items or by a set of examinees to a single item.In this case, P(x18) is the likelihood of the response string. for theM2PL model, the likelihood of an examinee's response string is given by

P(x.10.) = R P(x..10.)i -I

where x.. is the response to

abilities for examinee j, xj

is the number of items. Theis given by:

(12)

item i by examinee j, O.1

is the vector of

is the response string for examinee j, and n

likelihood of the set of responses Lc, an item

9

N

P(x Idi,ai ) = P(xij

Id ta ) , (13)

where P(x. d a ) is the probability of response xis for item i, d. and

ai are the item parameters for item i, xi is the vector of responses to

item i, and N is the number of examinees. In order for any statistic to bea sufficient statistic for a parameter of the N2PL model, it must be possibleto factor the appropriate likelihood function into the form given by (11).

Sufficient Statistic for the Ability Parameter For the N2PL model(12) can be factored into the form:

P(x 10 ) = Q (0 )exp(8 E a x )exp(ij

).

1i ij i=1

From (14) it can be seen that

n

s(x ) = E a x1=1

(14)

(15)

isavectorofsufficientstatisticsfor0(For a discussion of the

derivation of the sufficient statistic for ability in the unidimensionalcase, see Lord and Novick, 1968, chapter 18).

Sufficient Statistics for the Item Parameters For the item parametersof the N2PL model, (13) can be factored into the form:

N N NPOI Id a ) = 0 Q. (da )exp(ai E lixij)exp(di E xi4)

.i i'i (16)j=1 3 1 i j=1 J j=1 J

From (16) it can be seen that

Ns (x ) = E xd ij

j=1

is a sufficient statistic for the d-parameter, and

sa(x ) = r e

jxij

is a vector of sufficient statistics for the a-parameter.

(17)

(18)

Information Function

Definition In item response theory the precision of estimates basedon a given scoring formula are generally described in terms of the informationfunction of the scoring formula. The information function of a particular

10

7

scoring formula, as given by Lord and Novick (1968), is given by

I [o,s(x)] E

02 i2(x)''

2(19)

where s(x) represents a given scoring formula for the model of interest,efs(X)61 is the variance of the scoring formula, and the derivativeafs(X)161/66 specifies how the mean of the scoring formula changes as 6changes,

If s(x) takes the form

s(x) = wixi ,

i=1(20)

where wi is a positive number, then the expected value Els(X)161 is givenby

E [e(X) le) = r wiPi (e)

1.1

and the variance of the scoring formula is given by

o2 [s(x),e] = r ypi(e)cli(e).

(For a discussion of these derivations, see Lord and Novick, 1968).Substituting (21) and (22) into (19) yields

/ [O,s(x)] =

1E 1wi pi

2 (e)Qi

(0) -1

=E w

iPi-(0)

i=1

2

(21)

(22)

(23)

where P.'(6) =6Pi(6)/66. For a single item (23) takes the form

/ 10,s(x)] = Pi'(e)2/Pi(e)41.(e) (24)

which is the item information function. If (24) is written in terms of theresponse x., rather than the scoring formula s(x), the same result isobtained. 'That is, l(6,xi) = I[O,s(xi)J. Lord and Novick (1968) haveshown that, unless s(x) represents the locally best weights at 6,I(6,s(x)J < E l(6,x.). That is, the sum of the item information functions,which is independent of the the scoring formula, represents an upper boundon each and all information functions obtained using different scoringformulas. The sum of the item information functions is called the testinformation function, and is given 11

11

8

n nI(0) = E I(O,x ) = E P'(0)2/P (8)Q

i(8).

i=1 i=1

Information Functions for the M2PI, Model For the unidimensionalmodel, given by (1), the item information function is given by

(25)

I(0,xi) = D2ai2Pi(0)Qi(0) (26)

Test information for the unidimensional 2PI, model is given by

n

I(e) E D2ai2Pi(0)Qi(e).1=1

(27)

As was the case for discrimination, information for the M2PI, modelvaries depending on the direction relative to the 0-axes. Therefore,item and test information for the M2PI, model are defined using the firstdirectional derivative of the response function, which is given by (9).Item information for the M2PI, model is given by

I(8,xi) = a12 PQ cos2+1 + a22 PQ cos202 + . . . + am2 PQ cos2+m +

2a1a2 PQ cos+1 cosh + . . . + 2a1am PQ cos+1 cos+ra +

(28)

2a(ra 1)

am

PQ1)

cos, cosfm

.

011

For the two dimensional case, this simplifies to

2I(0,xi) = FQ(alcos0 + a2sin+) . (29)

12

9

Note that when the direction of travel is parallel to the 0-axis (f = 00),item information is given by a12PQ. When only 02 is of interest (f = 900),item information is given by a22PQ. If the two dimensions are weighted equally($ = 45°), item information is given by 0.5(a12PQ + 2a1 a2PQ + a22PQ). Figures3, 4, and 5 show typical item information surfaces for $ = 0°, 45°, and 90°respectively. Note that these are not the same surface seen from differentangles. They are different surfaces, all for the same item, obtained byvarying the direction with respect to the 0 -axes. As can be seen, they arequite different. Test information for the M2PL model is simply the sum of(29) over all of the items. Figures 6, 7, and 8 show typical test informationsurfaces for f = 0°, 45°, and 90°, respectively. Again, the three surfacesare quite different, indicating that the test gives different amounts ofinformation that are concentrated at different places in the 0-space whendifferent weighted composites of ability are of interest.

Maximum Likelihood Estimati:qi

Maximum likelihood estimation of the parameters of the M2PL model isrelatively straightforwaA. The likelihood of a response matrix for theM2P1. model (or for any latent trait model) is given by

n NL = R 11 P(x

ij) .

i=1 j=1(30)

where all the terms have been previously defined. For an examinee's responsestring, the likelihood is given by (12), and the likelihood of a responsestring for an item is given by (13). The first derivative of tie log ofthe likelihood given in (12) is given by:

(Slog I,. n n----L1-=Eax -EaP

60 i=1 i ii 1.80. i if--i

and the first derivative of the loge of the likelihood given in (13) isgiven by

(31)

6logeLi N=

xis6di

for the difficulty parameter, and

N=E0x E$Pikj=1i j=1i

(32)

(33)

for the discrimination parameter.

The estimation of ability using maximum likelihood techniques simplyinvolves setting (31) equal to zero and solving for O.3 . Of course, since

this involves solving simultaneous nonlinear equations, some type ofiterative procedure is generally required. The estimation of item parametersinvolves setting (32) and (33) equal to zero and solving for di and ai,

13

10

Figure 3

An Item Information Surface

Figure 4

An Item Information Surface

3.0

14

ti

two e 1

13

respectively. Again, the solution of simultaneous nonlinear equationsrequires an iterative procedure. McKinley and Reckase (1983) describe aprocedure for the simultaneous estimation of the item and person parametersof the M2PI. model using a Newton-Raphson procedure for solving thesimultaneous nonlinear equations. A computer program is available.

Discussion

Although IRT has gained popularity over the last few years, applicationsof IRT models have been limited to tests for which the assumption ofunidimensionality is at least defensible. There have been a few IRT modelsproposed for use with multidimensional data (see McKinley and Reckase, 1982,for a summary), but there have been few successful attempts at theirapplication. Use of these models has been limited due to the absence ofpractical algorithms for parameter estimation, and, at least in part, becausethe models are not well understood.

McKinley and Reckase (1982) have proposed a model, the M2PL model, foruse with multidimensional data, and they have developed a program for theestimation of the parameters of the model (McKinley and Reckase, 1983).The purpose of this report is to provide information necessary for theunderstanding and use of the M2PI. model.

Many of the characteristics of the M2PL model are not straightforwardextensions from the tit-4dimensional case. Rather, the unidimensional caseis a special case of the multidimensional model in which much of the richnessand complexity of the model is not evident. Because of this, some of thecharacteristics of the model described in this report may be somewhatdifficult to grasp. In order to aid in the understanding of these character-istics, they will now be discussed in some depth. An attempt will be madein each case to describe how the information provided relates to real-worldapplications. The ascussion will begin with the interpretation of themodel parameters, and will include the sufficient statistics, informationfunctions, and parameter estimation. Before beginning the discussion ofthe characteristics of the NHL model, however, a brief discussion ofdirectional derivatives will be presented, since directional derivativesare so important to the understanding of multidimensional IRT models.

Directional Derivatives

One of the most interesting and complex aspects of the multidimensionalIRT models which is lost when the unidimensional case is discussed is thenotion of directional derivatives. In the unidimensional case the onlydirection ever discussed is parallel to the 8 -axis ($ = 0°), in which caseall the trigonometric terms so evident in the material presented above areabsent--they always equal 1.0 or 0.0 and therefore drop out of the equations.

Directional derivatives are necessary in the multidimensional casebecause the derivatives of the response function vary depending on thedirection taken relative to the 8 -axes. The first derivative of a functiongives the slope of the function at any given point. The second derivativegives the rate at which the slope is changing at a particular point. Thepoint of maximum slope is where the slope stops increasing and startsdecreasing. At the point where that change occurs, the change in slope

14

crosses from being positive (increasing) to negative (decreasing). Thus,

at that point the change in slope is neither positive nor negative, butrather is zero. Since the second derivative gives the rale of change ofslope, the point of maximum slope is where the second derivative is zero.In the unidimensional case, this has a straightforward application indetermining item difficulty and discrimination, as illustrated in Figure 1.The point where the dotted line crosses the ICC is the point of maximum slopeand minimum (zero) change in slope.

In Figure 2, it can easily be seen that there is no one point on thesurface where the slope is at a maximum. Moreover, for any one point., theslope varies depending on the direction. Consider the point on the surfacewhere 01 = 0.0 and 02 = -2.5. This point is indicated on the surface by anx. Moving along the 02 = -2.5 line parallel to the 01 axis, the surface isrising fairly rapidly at the point indicated. However, moving along the01 = 0 line parallel to the 02-axis, the surface is still relatively flatand is rising slowly. Clearly the slope of the surface is differentdepending on the direction of travel. The same is true of the change inslope. Because of this, when taking derivatives of a multidimensionalresponse function, it is necessary to consider the direction. Directionalderivatives are a way of doing this. The actual interpretation of thederivatives in different directions will be addressed in the next section,since it is closely related to the interpretation of the model parameters.

Interpretation of the Model

A straightforward extension of item difficulty from the unidimensionalto the multidimensional case would seem to lead to the conclusion thatdifficulty in the multidimensional case ought to be a vector of b-parameters,with one b for each dimension. In Figure 1 the b-parameter is the point onthe 0-scale below the point of inflection. It represents the point on the0-scale where the item best discriminates between high and low ability. At

the point represented by the b-parameter, a very small change in abilitycorresponds to a large change in the probability of a correct response.Nowhere on the 0-scale does an equal change in ability result in as large achange in the probability of a correct response. Thus, in the unidimensionalcase, the item difficulty parameter indicates the point on the ability scaleat which the item does the best job of discriminating between differentlevels of ability.

On the surface, it seems logical to conclude that if there are twodimensions, there should be two b-parameters. One b-parameter shouldindicate the point of maximum discrimination on one dimension, while theother b-parameters indicate the point of maximum discrimination on theother dimension. Figure 2, however, clearly illustrates that this isinadequate.

As can be seen in Figure 2, the two ability dimensions do not actindependently. It is the combination of ability on the two dimensionswhich determines the probability of a correct response. An examinee with02 = 2.0 clearly has a higher ability on that dimension than an examineewith 02 = -2.0. However, if the second examinee has 01 = 3.0, while thefirst examinee has 01 = -3.0, the second examinee has a much higherprobability of a correct response to the item described by the IRS. Clearly,

then, considering the two dimensions separately does not contribute greatly

18

15

to discriminating between examinees who have different probabilities of acorrect response. This is reflected in the fact that the item difficultyfor Figure 2 is a line which is not parallel to either axis.

This has important implications for test construction and analysisusing the 1.12P1. model. It is common practice, for instance, to order itemson a test by difficulty, or to construct a test having a specifieddistribution of item difficulty. In the unidimensional case this is doneusing the b-parameter. Clearly in the multidimensional case it is morecomplicated. An item having a smaller d-parameter than a second item is onlyuniformly more difficult than the second item if their difficulty functionsare parallel. If the difficulty functions intersect, then item I is moredifficult than item 2 in some regions of the 6- plane, while item 2 is moredifficult in other regions.

This would seem to indicate that it is only reasonable, in the multi-dimensional case, to talk about ordering items on difficulty or obtaining aspecified distribution of difficulty if all the items to be considered haveparallel lines of difficulty, Of course, in the m-dimensional case theitems would all have to have parallel (m-1)-dimensional hyperplanes.

In order to determine whether two items have parallel lines of difficultyin the two-dimensional case, first determine the form of the difficulty line.The equation for the line of difficulty is given by (8). The two lines areparallel only if the slopes of the lines are equal. Putting (8) into aslope-intercept form yields

6j2

ae +

ai2 j1 ai2(34)

where a.1 is the item discrimination parameter for item i for dimension I,

ai2 is for dimension 2, oil is the ability parameter for examinee j for

dimension 1, and 032 is the ability parameter for dimension 2. If item 2 is

denoted by a prime ("), then the lines of difficulty for items I and 2 areparallel only if

ail

ail

=ai2

a12

(35)

If all items meet the condition set out in (35), then they can be orderedby difficulty, by simply ordering them by their d-parameters.

The ordering of items on difficulty implies that there is some underlyingvariable being measured that has some correspondence to the criterion usedfor the ordering. In this case there is some composite of the es whichcorresponds to the difficulty continuum formed by the items having parallellines of difficulty, The composite is determined by the orientation of thelines of difficulty.

The extension of item discrimination to the multidimensional case iseven more complex than the extension of item difficulty. Unlike difficulty,the concept of item discrimination in the multidimensional case includes aconsideration of direction--the angles indicating the direction do not fall

19

16

out of tile equations. Although the slope of the 1RS shown in Figure 2 isconstant all along the the line of difficulty for a given direction, itvaries with different uirections.

The need to consider direction has important implications for bothtest construction and test analysis. lt is not enough in test construction,for instance, to merely select the item with the highest discriminationfrom among the available items. One item is uniformly more discriminatingthan a second item only if the slope of its IRS at the points of inflectionis greater than the slope of the IRS for the second item for all directions.If item 1 has the higher a-value on one dimension, but a lower a-value onanother dimension, there may be directions for which the slope of the 1RS atthe points of inflection will be greater for item 2. For example, considerthe case where item 1 has discrimination parameters a = (1.0, 0.5) and item2 has discrimination parameters a = (0.5, 1.0). When 4) in (10) is 30 degrees,the slope of the 1RS at the points of inflection is 0.279 for item 1 and 0.233for item 2. When 40 is 60 degrees, the slope for item 1 is 0.233, while theslope for item 2 is 0.279. At 4, = 45 degrees, both items have a slope of0.265 at the points of inflection.

lt seems to follow from the above discussion and example that, ininterpreting item discrimination in the multidimensional case, the particularcomposite of abilities of interest must be considered. The composite mightbe specified a priori, as in test construction, or discovered by postadministration analyses.

Sufficient Statistics

The notion of a sufficient statistic is not a simple one to grasp.Essentially, a statistic t is a sufficient statistic, for the parameter 0 ifit contains all the information in the sample data about 0. For example,the number-correct score for an item provides all the information in theresponse data about the d-parameter. For the a-parameter for a particulardimension, a sufficient statistic is provided by a weighted sum of the itemresponses. The response of each examinee to the item of interest is weightedby the examinees ability on the dimension of interest. Thus, a correctresponse to an item by an examinee of high ability (0 > 0.0) adds to thevalue of the statistic, while a correct response by an examinee of lowability (0 < 0.0) decreases the value.

For ability, a sufficient statistic is provided by a weighted sum ofan examinee's responses to all the items. The weighting factor is thediscrimination parameter for the dimension of interest. Thus, a correctresponse to a highly discriminating item adds more to the statistic than acorrect response to an item of low discriminating power.

Although the availability of sufficient statistics for the parametersof the M2P1, model is important from an estimation standpoint, it should bepointed out that, with the exception of the d-parameter, the sufficientstatistics described above are not observable. While the number-correctscore of an item can be observed, the a-parameter of an item, and thus thesum of item responses weighted by discrimination parameters, is not observable.This complicates estimation somewhat by requiring that provisional estimatesof some parameters be provided during the estimation of the remainingparameters. Solutions, then, are obtained by a series of approximations by

20-iil:;1W1011=111M=i1M61111W

17

varying from one step to the next which parameters are estimated. In eachstep the provisional estimates for the parameters not being estimated arethe most recent estimates of those parameters.

Information Function

Item information in the multidimensional case is like item discrimina-tion in that the information yielded by an item for a particular 0 varieswith the direction of travel. This has important implications for suchapplications as adaptive testing, in which items may be selected foradministration on the basis of item information. As was the case with itemdiscrimination, the interpretation and use of item information requires theconsideration of the particular composite of abilities which is of interest.

Maximum Likelihood Estimation

Estimation of the parameters of the M2PL model is surprisingly straight-forward. implementation of the procedure described earlier in this reportis not particularly difficult. However, it is rather expensive.

One serious limitation of the procedure described is that there is noway to determine in advance how many dimensions should be included. Theprocedure is too expensive to allow successive runs for an increasingnumber of dimensions until a satisfactory solution is obtained. It is

clear that, if the M2PL model is to be used, more work is needed in thisarea

More work also needs to be done to determine sample size requirementsfor estimation. Some guidelines are needed for determining the maximumnumber of items and subjects required for good estimation.

Summary

Item response theory has become an increasingly popular area forresearch and application in recent years. Areas where item response theoryhas been applied include test scoring (Woodcock, 1974), criterion-referencedmeasurement (Hambleton, Swaminathan, Cook, Eignor, and Gifford, 1978), testequating (Marco, 1977; Rentz and Bashaw, 1977), adaptive testing (McKinleyand Reckase, 1980), and mastery testing (Patience and Reckase, 1978).While many of these applications have been successful, one unsolved problemis repeatedly encountered--most IRT models assume unidimensionality. As a

result., applications of these models have been limited to areas in whichthe tests used measure predominantly one factor (or can be sorted intosubtests which measure predominantly one factor). When the assumption ofunidimensionality is not met, most IRT models are inappropriate.

The purpose of this report is to present an IRT model that does notrequire unidimensional tests. With such a model the great power of itemresponse theory as a measurement tool can be applied for many of the pur-poses for which unidimensional models are employed, without the limitationon what kinds of tests are involved (i.e., the dimensionality of the tests).Of course, much more work is needed before the model can be employed inreal testing situations. Procedures for the use of the model for different

21.

18

applications must be worked out in greater detail, and limitations on thepracticality of the estimation procedures must be overcome. The informationprovided in this report provides a firm foundation for future work in theseareas.

22

19

REFERENCES

Birnbaum, A. Some latent, trait models and their use in inferring an examinee'sability. In F.M. Lord and M.R. Novick, Statistical theories of mentaltest scores. Reading, MA: Addison-Wesley, 1968.

Mambleton, R.K., Swaminathan, H., Cook, L.L., Eignor, D.R., and Gifford, J.A.Developments i latent trail theory, models, technical issues, andapplications. Review cf Educational Research, 1978, 48, 467-510.

Kaplan, W. Advanced.calculus. Reading, MA: Addison-Wesley, 1952.

Lord, .M. and Novick, H.R. Statistical theories of mental test scores.Reading, MA: Addison-Wesley, 1968.

NAtp, C.L. item thanItterisLic curve solutions to three intractable testingproblems. Journal of Educational Measurement, 1977, 14, 139-160.

R.L. and Reekase, M.D. MAXIA1; A computer program for the(%timation of the paeameters of a multidimensional logistic model.kehavior Researih Methods and instrumentation 1983 15. 389-390.

4(Kinley, R.I. and Recluse, :1. D. The_oseof_the_general_Rasch model withrilti I t 01 littett5 qrt1.1 itemresponsedata (Research Report ONR82-1). Iowa City,IA: The American College Testing Program, August 1982.

MtKinlev, R.L. Ind Reckase, M.D. A successful application of latent trailtheory L,, tailored_achievement testing (Research Report 80 -1). Colambia,M6: University of Missouri, Department of Educational Psycholny,February 1980.

Patii,ntes W.H. and Reckase, M.D. Se11-paced_versuspaced evaluation utilizingcomputerized ta_ilored testing. Paper presented al. the annual meeting ofthe National Council on Measurement in Education, Toronto, 1978.

Rentz, R.R. ani Bashaw, W.L. The National Reference Scale for Reading: Auapplication of the Rasch model. Joornal_of_Educational Measurement,1977, 14, 161-180.

Woodcock, R.W. Woodcock Reading Mastery...Test. Circle Pines, MN: AmericanGuidance Service, 1974.

23

Idtvy .l ivy

Or. E4 likenNavy Porsonnet R&D CenterSin Diego. 0 92(52

I Or. krthue SachrochEnvironimmtat Qr.:0gs Pro.grum Conte('Nvo.la 4441,clt ft:search Institute

Sethi:via. 40 2n1t4

I Or. Mlryt S. BikerNtey Porsoonet RSO :enterSin Diego. CA 92152

I Ltoison Sctontiat)f face of Nivit Rose arch'cinch lfftce. London

10 *t 11

FP1 4ov York. Y 1111)

1 Dr. Tait Oaffy

Nivy Pareoonet RAO CenterSin °too. CA 92152

I like OurmeyerIngtructIonot Program OtvetopmlntMalang 90ser-pncoGrqlt Lows ITC. IL 0005,

I Or. RichtrA RtstorOoportient of Administrative ScioncegNaval, Poititro4uott School.

lonterAy. CA 919SO

I OR. PAT FROERIC1':ot10 PII

1)+10e.m C1 111"0

I IA. 11.(titIor d'ey%opt t it Forth, o tv 1 . P.r,tinis

iotgoro%ent Oivtiton Ntvy Porgonnot RAO lonterN11RL Son Otolgo. C1 92152

NkS Panst:ota. FL 12593

1 Or. Robort ilreoukIWIR1ROUIPCF.4Colo 4-3/5R

into. FL 1241 ;

I Or. Robort CirrottMOP 115ighIngton 211110

1 Or. Jilt goltanCo le 1'i

*levy ?..rgonan1 R n 'Ionr.oc

in °tog°. Ck 91151

I Or. R %Wan,tvy Porgnnntt RSOItn 171 12151

I Or. slorsin I. K..cr

$1111 'if 'ltv.tt 104111t inn tril trdnIn4 d7h1A of 4.1vol r1Ch,t(,: it friinin4

1.1 oton Off Itvil 11r it itlon AtnnhIg (FS)

11 r Fn ree In in R .4norra L tbor ttorY 1 1 lolton ISIS%

1p.britton4 rr 'franc; 1,0.0.91tio

"MUM 1FR, 15224 l Or. L,.nntr4 (ro.o.or'l ivy Pir000n..1 Rcn 1,nPqr

I Or. Sninloy ;loilyer i in nt,40, (: ilts2

f ico of I tv tt T ochoo1m.ty

190 4. 9ssiney Str 1 Or. '41111 14 I,. Itloy ('i1)

riiingtna, V% 112L/ ';'11..f of `)iv t1 KIti^ttlon rrtininI

j1 it 'It r qt it Inn

';in ltto Corr to P..ng wall, Ft. 11514

qfflt if 4 vo 11 R trOl

1111. Onin:y St. l Or. I %v., 4..111.14o

Co 21') 'l ivy P..raonnot itco ':nntor

krl1n.tton. V% 22217 Sin nt.,110, CA ')2152

Or. Oong Olvtg l Or tlitm lontaAtiot;s1gr `)PRO'. Clan 11

Pecrt teal 1. Ft. Sin Otqo. Ck 92152

24

Navy Navy

1 Bill Norilbrock1032 FaIrlAwn Ave.

Libartyville, IL 61148

1 Library, Cole P2OILsitvy Personnel Rcn 1n ter

Sin !Negri. CA 92152

1 Teehnictl DirectorVivy Persnnnel R11 -.enterSan Diegn. CA 92151

1 Nr. 1)r o1 Symnsnn

Navy Personnel R&D .enterin nt.,go. co 9102

1 9r. Fronk Vicino

Navy Personnel Rcn lenterStn Diego, Cn 92152

1 Dr. ENAr4 14qmtnOffie, nf Naval lt.stAr-h (Cni. 411S4P)9) 1 North Dainey Street%rlington. V 27117

6 rsannet 4 Trainioq Resetreh (roonCn 4427r 1 Dr. 10,1314 4.itzmon

4ffiee nf lovtl R.s.oreh ti tv 11 Pastgra /nate Seta-tat

Arllovjon, V 22217 D.pirtm.nt nf 4lmlnisrr4tivSelenes

1 SowIt1 Ast. for e4nettion and 4antry. cn 4194orrtinini (9P-91F.)

Rn. 270S ArlIne.ton Annex 1 n.. nqn0.13c 4tz.1

Ultshinlinn. n1 20170 Clie 17.N ivy Prwann.1 R44 Center

1 LT Frtnk C. Petho. NSC. USN (P'i.n) Sin Oi.e.n. C4 92151

CM!' (N-412)44S 1 nR. 1Rri4 F. WISXIFFP..nstnolo. FL 11509 41VY PERSTML R4 1) '7.4TER

qnN nmcn. co R2 i52

1 Dr. B.rntrd Rimlanl (01C)Ntvy Persannel R4D CenterSan Diegn. C4 92152

1 Dr. Carl RossCNET-PrOSoildIng 90Oren Lakes NrC. IL 41914

1 Dr. Robert C. SmithOffice of Chief of NtvAt 0norationsOP-91/NWtshington, 2015')

1 Dr. Aifrul F. Slade. DirectorTraining Analysts & Evaloatinn TroupDept. of the NavyDrIanlo. FL 32811

Riehmi SnrensenNtvy Prsoonel R4D CenterSin Diego. CA 91152

1 Dr. Froierick Sceinh.iserC91 - OPI15Navy AnnexArlington. VA 20170

1 4r ,inhn II. Wolfe

N tvy Persnnl R41 CenterSan nwgo, C4 92151

1

25

Dr. titiloee Wolfeck, 111Navy Rrstannet R&D CenterSin DIsto.n. C% 92152

!Juin.: Corps Army

1 U. William (re:nonElul:Ulan Advisor (E011)Eelueltion Center, 4":0:C

()wooden. VN 2211'4

1 Otr-etor. 0ffie, of 4npna.r Otlizttin01, 4trine Cqrnq (4P0)ar:11. 811. 2011

14Intiro. V% 1211N

1 '1.tlgoirt-rs. U. S. ltrin Gq-pC.1 4P1-291.1WtinItnn. 01 21141

1 Spleill Aggistnt fnr orinCgrog 'litter.;

Cob. 1014Offtee f NivAl R.setrehW)) N. ()oinv St.Arlinvon. V% 2121 7

1 01. A. t,. SLAMSKYslIVArtric kOvIsrt 01131 R0-1)11 '). U.S. mqtyg '7E041.1%%0IN3t14. Ir. 211411

1 41inr Fr ink YethInnin. UST":

11-aignirters. Mirin enrol(Cole 4P1-20)Wtgoinston, Of: 2:1141

1 Techniett DirectorU. S. Army 11;,seAreh in4titote for the

B,h,vinrtl ani Sorill Seteor^q5n11 Zisenh74.r Av..nuAlexawlril. VA 22111

1 4r. .1iius SOtc,r.

Arms, R.s..oreh inctitute

5)/1 eicenh,4-r Avenw.Alexinirlt. VA 27111

1 Or. 1(.nt vttni

%ray 11....tetr.a in.:titute

5011 Elqnhnvor Riv.Alen tnirl, . VA 'lilt

1 Ur. 4.itrteft J. Forr

U. S. Army it..s..trch ingritov511 igenhlwr AvenuoAlftxtniria. VA nits

1 Dr. 'lyrnn Flxrhi

U.S. Army itex-Irch tostitote for theWial An1 gAlvinral Sriolees5911 Fix,..nsinw.r Avenu.

Alexanirit. VA 2211

1 Dr. Miltnn S. Kit,.Training Teen-lir:Ai Are4U.S. Army Rex.trch ingtitute5411 F.14enhow.r Av..no-

AloxindrIt. VA 11111

I Dr. Ilirold F. 0114:41. Jr.

Oir.:rtnr. 71lining ites-lrrh 1,11.1

Armv R-soirrh in.:dome5191 Ets-nhnw,r Avenu,Al.,xtniril. V% 22111

1 illmmlnier. U.S. Army Rstrrh in.:titianfnr th alhtvinral 4 Snriat Scienceq

ATTN. PERt-SS (Dr. Jwlith ')rasing)

5111 Eigenh.wr AvenueAlex:twirl,. VA rill

1 Joseph Psot1/41. :4.0.ATTN: PERI-1(1

Army Rosoarch instittst5991 Etc-oh,wr Ave.Ai:in-10A. V% 22111

26

Army

1 4r. Rnrt RossU.S. Army Rusearch Institut.: fnr ch.sletii nui $4111vinr41 Srienef

5111 Els..nhlw..r Avnu.Alexanirii. VA 22111

1 Or. Roburt SissorII, S. Army Resetreb 104tittit. fr Ur.H.litvioral lni %nett! Slenc.s51q1 Eis.ahiv..r Aveno11xt1irli. VA 22113

I Or. loye'. shi..tis

Ariy R..sotrch Institut. for 04.1tivinrl tol :n.'1 1I Sien:s5131 Eisenhnwr A.):flu.Ai octndril, V% 22111

1 Dr. H1141 WingArmy R..s.trrh insiitot..

5111 Ris-ohlw.r A.Aloximiria. V% 22111

1 Or. Rob.rt Wish.rArmy Rs.crch Inst1tutS1.1I Eis.niciut.r Avonu.

Ilex.cniril, V% 22111

27

Air Fnrce

1 Air Fnrce. Hulan Resnure.es LibAFIIRL /MPt)

,rani, APR. TX 74215

I Techiicil n1comzots CenterIir Fnrce Burin it^,nurr4s LibnratnorWPAra, 03 45411

I U.S. Air F.Ire. Wrier. of ScintifirR..soirch

Life ,- lent,, 01r.,etnrito. NL4111inq Air Fnrc,. Rise

ticsbinvon. m 2'1112

1 Air University Lihrlo,AUL/LSE 76/44141*.a.01 APR. AL 16112

1 Dr. FAO A. 4110141ID, APIIRL OPS::1BroAs APR, TX 78215

1 sir. itlym11.1 E. CristlAFIIRLONEfirnolts AF4, TX 7q115

1 Ur. Alfre4 R. FrectiyAPOSRMLRliling APR, n. 11112

1 Or. C..nivieve 11,4411Prngr41 giolgor

Life Sciences nirertnrateAP1SRRotting %FR, Dr: 20112

1 Or. T. H. LongricictoAPIIRL/OTE

Williams APR. 47. RS224

1 Mr. Rininiph ParkAPHRL/4n14

!Woks APR. TX 74215

I Or. Rnger P.nnellAir Fnrco 4u' R,sources tibnr.itory

Lawry APB, CO 111219

1 Or. Malcolm R..41

APHRL/4PBrnnks APR. TX 78215

Ntr Force

1 37q3 TCHIWTTG4R21.t TattarigoShnivirl 08, TX 76111

1 Lt. Cra Jim 2S E. WagonIll OSIF/MPXON

Ili- Pontasgon

WnshIngtnn, Or: 20110

1 iltor inhn 4.1444Flin:

Rinolph WU. TX

! Or. .Ins1111 Ylsiuk4v11111, /LUf

L.,.wry Ars. 0 Ao7.10

Owtrtmnnt of [Miens!!

12 0fens^ Inchnient Inform-4ton ClittnrCtriv.ron Station. Olaq S

Atexanirin. VN %.2114Nttn: 1T

I Or. Cralq I. Fllf1s4ivIcte04 ksotrch Prn10,:s Ngenrs140 1 41:.:nn Riel.

4.1 indston, VA 7210to

1 JerrV trsttnigqlilt iiiirltriAt tn: 'vwr -i'Fort Slir1,1 Int 11, 6 ni/

1 1111ttry Assinttnt for Tv11111m. tn1PorsInnl T%-ltnnlo:tvIfilet. of th. Oilier Sers.trs of 0.fonnfnr Rstrch 4 Enlinarin:Kon 1O129, Th. 11n! !Inn4,11shinclton. 01 2111

t ')r. Illy:w 511stst

Offic.. of th Assisrtnt S..eratIrtoof O.fons- (TIN 4 LI24149 Th. P.nttcon

WinhInnton. Of: 201n1

1 ilitnr itch Mown%DOW%140 Ailsnn Mel.Arttnntna, VA 22209

28

CIVIII44 Agneies

I Or. Susie ChltninLarninl ni Davolnpa-ntMittnntl tnctitotn of Elaeittnn120 19th Stre:t 4WIshingtnn. nr: 20203

1 0r. Warn W. UrryPqrconn0 RSD 1-ntaroffic.. of P..rsrptin:1 4nnsgelt.nt

1901 E Stre.t 414tchIngton. 01 214IS

I Mr. Ittm to A. WarmU. S. clIst Cntr4 logtitutoP. '1. subitttion 18OtlihTlis City. OX 71164

1 Dr. Joszn4 L. Yourt, DlooetnrM:in:y 4 Cntiniti.t Prn:ovissNalnoll Selen-: Fetal-WooWhittington, DI 20551

29

Privnut Snetnr

1 Dr. kilos AlttiniUniversity of Florid 1Glinnewillo, FL 124

1 Dr. Erling R. Aqii-rsanloartmnt of StitisticsStollectrte4" 514SS 1np,WInga0E4141X

1 1 fcvrh,lngirl Rsatreh tinit49-144 1ttn7 LihrlrlinNqrtitharn lita4.Turtvr 4,1r 2491

AUlfRALIA

I Or. lstir RytirKiarginntl Tating 1rvie.,PrIneton, Ni nviso

1 Dr. 4,0arlit Rironharn;zitinl of eld-31100

r.1 Aviv UlivoraltvTot Aviv, Rillit Aviv 49974ir tal

1 Or. R. Dirroll dqvk9.ptrtmant of Margin()University of Cleinnillti-agn, 1L 41417

1 Dr. Robert RrnninAgar-le-1n Cnllego T4tingP. 0. ttnt 148

twat City. to 51241

1 Or. Frost R. Ci4ntte117 Stnkolythtivarcity if TennasserKnntvitle, TN 17914

1 Or. John A. CArrnil409 Elliott R.C4ipol MM. Mr. 27S14

1 Dr, hinratin Cliff

Dpt. of PsyrotqlngyUniv. of 9o. CiltforniAUniversity Pirk(As Angalec, CA 93117

Progrtits

Private Sector

I Dr. fins CrombsqE4ucatlnn Rqse,srch CcntorUnlvorsity of tlydenEls.rhiavolAln 22114 EN tqyi.nnh. NE111E414,49;

1 Or. Dittprad ii OlvalSyr4cuse UniversityD.ssrtiont of PsychilnlySyriens., Niv. 11210

Dr. Fritz Or4sgo4DvIrtmont nf Psy:lisingyUniversity nf Illinois611 E. DintAl St.Chempliqn. IL 618/1

Or. Slssn Emb.rtsqnPSYCH1L/CY DZPNRT4R4fUNIVERSITY OF KVISASLiwrftneq, KS 66145

1 ERIC Ficilltv-Acquisitions4411 Ru4hy AvenueRothesit, 40 2n1t4

I Dr. R.niamln A. Falrhink, jr,41Fsnn-GrAy S 4ssoci.stes, Inc.5425 2allaghinSit° 22SSin Nntnnlo, TX 74228

I Dr. L'onord Fnl(itLindquist ,nter fnr M.surm.sntUniversity if 1041Iowa City, 14 S2242

I Dr. Richir4 L. FergusonThe Amt,rican Colleg Tstinl Progr.mP.O. flax 168

tows City, 14 52143

1 Univ, Prof. Or. Gerhird Fisch2rLieblgq.ssse 5/1

4 1010 VionniAUSTR14

I Professor Olnild FitzqrillUniversity of N.'w Enginnikrmlialn, Nlw South Wiles 2151

AUSTRALIA

Privnts Sctor

I Or. DIxter Flestch,*r1.110T Rscarch Institute1475 S. State St.lre, or 22111

1 Dr. iintal GiffordOnivnrslty nf 4IssichhsottsSchonl nf EineationImhnrsr, IN 11112

I Ur, Rnbrt 011sserLsrning Rsoirrls S 0.velnpsoni.

nf PIttsburqh1919 Str-o,t

PIUS404C4. P% 1110

1 Dr. Rrt Ore)Johns Winking Unlv.rsityDeportment of PsychqlnavC1004 S 140 qtreot911timnre, 44 11218

1 Dr. Rnn IlimhlernnSchnol of E4u-stinnUniversity of l.sssichns.ttsNmh.?rst, 4 01012

1 Or. Oolwyn 41rnisehUniversity nf Illinnis242h Eoetionstrbtnt, IL 61811

I Or. Plul677 0 Strnt, 0144Chula Vista, C4 90010

t Dr. Lloy3 HumphreysColoirtm,nt of PsyehlInsy

University of IllinoisEra ()Into St rent

chisplign, IL 61821

I Or. Jack Minter2122 Cnolidge St.'Arising, MI 44916

1 pr. Unynh guynh1:olleg4 of EiucationUniversity nf Snuth CirnliniColumbia, S' 29104

30

Private Sector

1 Or. n3U1(39 H. JonesAdvinced Statistical T.JchnologiesCornoration10 Trif4litar Court

Livrenteville. NI 04144

1 Professor John N. Kqttsm.ptrt.m.of of Psy41111nly

Th Onivrsity of Nw-isti1.4.W. 211114411141,11

1 Dr. %Mills Kn4hUniversity of T.s.ts-44slinP.I.turennt an Ey-lto -Woo CenterAqstio. TX mini

I Or. Alin 1.,.sgold

1.: truing R49 ::ntr

Univ.rgity of Pittshrgh1914 3'111ri Street

Pittsburgh, P% 15241

1 Or. MIchsel L-uin0prtment of FAucationti Pshingy210 Eiucition Blast.

UolvorsitY .if 111111ICntsptign, IL 41111

1 'Jr. Chtrls Lewis '

Facoltlt Whir' 1.1.teosgh+PnRliksonivrsitelt CronlaxoOlio Boterlogestrait 219112111 CroningonN.thrlans

1 Or. Rob..rt Linn

Col l of ElocationUniversItV of Illinoislrbott, IL 61311

1 Mr. Phillip LivingstonSystoms and Apoll(' Snloor.s Corporitio6'111 Konilworth AvemRivoraale, 40 201140

I Dr. Robert LockmanCenter for Nmv.ml A011ygi,200 North 11,soregari St.Alexanaria, VA 22311

31

Private Sector

1 Dr. Frederic 4. tor'Ewational Testing SrvIcePrinelton. N1 nifAl

1 Dr. inms Lsionhpartent of PsveholoeVOniv-rsity of ii.stern Nostrilia4.(ilin.le W.A. 4119

AUSTRIL14

1 Oe. Cary Marco:top 11-RFAucitionil Tsting SorvlePrinreton, NJ 11551

1 Dr. S.ott 11xw.11Dlltrtm.ovt. of PnyrholnIVUniversity of 'intro DoseNotre olm. 14 44554

1 Or. S11, T. MayoLoyn14 University of (7:11rpqn420 4orth Mir.higan Avenn.Cnirago. 1L 61SII

I Ir. Robrt M(11(inley

alric4n Initew. Testing ProgramsP.O. Box 168lowq City. to 522.4.1

7. ilsrbor4 44ns'onus Rsoorcon Research Organization113 North Vishingtonklexaoiria. Vs 22314

1 Or. Rotrt MislevV711 Illinois Street

Conva, IL (0114

1 Dr. Alien MunroBehavioral Technology Laboratories1845 Mena Ave., Fourth FloorS-lonao Beach, C% 90277

1 Dr. V. Alan NicewinderUniversity of Oklahom,0,partment of PsychologyOklahoma City. OK 73069

I Dr. 41vin R. Novick356 LinMoist Cotor forUniversity of to"Lows City. to 51242

MoasUrniont

Pi-Iva:ft Sector

1 Dr. J1m1s OlsonWICAI Inc.1875 South Stte StreetDrem, Ut 84151

1 Wsyne H. Patience4lecicin Council on Edolotio1GEO Instinct Service. Sit,. 21011 DApont Clrle, N4WtshInIton. 01 201)

1 Dr. ism., 4. P104,10PortlInl Stste UllversityP.O. Box 751Portllni, DR 47117

..

1 Or. Mork D. R.ckssl4111"

P. 1. 41x 161Iowa City, IA 5L1'.1

1 Dr. Thnlls R-vnoldsUniversity of Toxas-lillas41riv.ting D.p-irtm-nt

P. O. Rnx 611Richardson, TX 75111

1 Or. Logrenco Rgin.n.411 Elm 1venu.Ilkom1 Park, 40 21112

1 Or. J. RylnWpartment of Edu:attoaUniversity of South Cirolin,Columbia, SC 29201

1 PROF. MI K:, S1MEJIM401.1q. OF PIYCII1L03Y

UNIVERSITY OF TENNESSEEWinxviLiE. TN 1:916

I Frank L. Schmidt0.psrtm.nt of PsychologyBldg. GC;

George Wishington UniversityWishinuon, DC 20152

1 Dr. Witter 5:h-widrPsychology Dzpirtm...nt

613 E. Dinic1Chli%psign. IL 611i2n

Private Sector

1 LowIll Scho'r1Np-111100c:11 4 lolntititive

FoolltIonsCollege of EiucationUniversity of 1043lows City, IA 52242

1 DR, RTIERr .1, SEMIINST40111DA1L TE07441W1Y CROW1Pri1R1

3)1 N. 1411111411'01 Sr.

51.6A111111%. VA 72)14

1 Dr. KtZU3 ShIg.mtsoUniversity of Tolsnn .p ortm g r. , of E hic it in, it INvrnrit 01V

1C-wan:hi, Seniii 411J1PAN

I 4r. Edwin Shirk.v0.p1rtmni of Psw.hologvUniversity of Centr11 Florll1rIan1n, FL 12816

1 Dr. Willi-tie Sims

Center for NV4i 1411V41,211 North 1elnregird Street

Alexaniril. V% 12111

I Or. W. Willie.. SinsiknProgr4m Olro?torNanoow,r Rstreh ni Advisry ServtresSmithsonian instftur100$01 North Pitt Stre-rAlexaniria, 1/1 22114

I Dr. Robert Sternh.rg0pt. of PsychologyYale OniversitY*lox 114. Isle UnionNew 11-wen, CT 06520

1

,

Dr. P.ter StaloffCenter for Noval Anilysis211 North ilmoregard Stre-tAlesanirla, VA 22311

1 Dr. William StoutUniversity of IllinoisDa1rtmmit of 4sthomiticsOrhon. IL 611141

32

Private' Soctor

1 Dr. lisrilitrin Swritioithin

Liboratory of PsYchlm.tric antgvluttion RaseirchS.thool of gincationUniversity of fitssachosittsilit.rst. 4% 01011

1 Dr. Kikusti Tntsuoki

Cospater Rsd Elticailoa it.s,areh Lib252 nqin,artne, Resoireh LiblritoryOrhtut, IL 61811

1 Dr. itorire Titsuokt220 E4u-ttion Ritig1110 S. Sixth St.Chilooign. IL 41820

1 Dr. Divt ThissnOTtrtmAnt of PsyehololyUniversity of Kitts is

Liwrene, KS.64114

I Dr. RobJrt Tsukoltswt0partmnt of StativAcsUniversity of MissouriColassbit, MI 45201

I Dr. .1. UhlovrUhl finer COOStiltWitS

4258 ioatvito DriveEncino. C% 91414

1 Dr. V. R. R. UpouluriUnion f:irbi4n CorparationNttlear DivisionP. O. Box Y01k Ridge, TV 17810

1 Dr. David VtloAssossxnt Systems Corprition2211 University Avenu.Solt° 110St. Paul, MN S5114

1 Dr. Iiiwirsi Wtin,r

Division of Psych,lqqictl StuliesElucationil Testing ServicePrinceton. NI 0'1550

1 Dr. Michil T. WallerD.:wow...at of Eiulitiontl PsychlloKYUniversity of Wisconsin--MilwiukqeSilwiuXel. WI 51201

33

Privity. Sector

1 Or. 9riin WitersIhrtiRRI

Ina 41rth UtshingtomNlexinirii, VN 22114

1 Dr. Diviti 1. 4,iss4661 Elliott DalUniversity of iinwxoti/5 g. River Rotlinoenpnlis, 44 55455

1 Dr. Rini R. WileosUnivorsity of Sloth,rn .:111fo..nit

0spirwnt of Psy-hnlo-0to Nueeles, ,14 9111/

1 Wolfeing 1411111ml"

StreltkrnefteistBox 20 59 31n-slul Roan 2WEST nER144Y

1 Dr. !truce 011111ms

0-pirtm-nt of Elucntionil PnY^h1101YUniversity of IllinoisUrbint, IL 61811

1 Dr. 14nly YenCris /4 -Craw 4111

01 'bnte Research Prklqnteroy, CN 0194n