estimation of ability using globally optimal scoring weights
DESCRIPTION
Estimation of Ability Using Globally Optimal Scoring Weights. Shin-ichi Mayekawa Graduate School of Decision Science and Technology Tokyo Institute of Technology. Outline. Review of existing methods Globally Optimal Weight: a set of weights that maximizes the Expected Test Information - PowerPoint PPT PresentationTRANSCRIPT
Estimation of Ability Using
Globally Optimal Scoring Weights
Shin-ichi Mayekawa
Graduate School of Decision Science and Technology
Tokyo Institute of Technology
2OutlineReview of existing methodsGlobally Optimal Weight: a set of
weights that maximizes the Expected Test Information
Intrinsic Category WeightsExamplesConclusions
3BackgroundEstimation of IRT ability on the basis of
simple and weighted summed score X.Conditional distribution of X given
as the distribution of the weighted sum of the Scored Multinomial Distribution.
Posterior Distribution of given X.
h(x) f(x|) h()Posterior Mean(EAP) of given X.Posterior Standard Deiation(PSD)
4Item Score
We must choose w to calculate X.
IRF
5Item Score
We must choose w and v to calculate X.
ICRF
6Conditional distribution of X given Binary items
Conditional distribution of summed score X.Simple sum: Walsh(1955), Lord(1969)Weighted sum: Mayekawa(2003)
Polytomous itemsConditional distribution of summed score X.
Simple sum: Hanson(1994), Thissen et.al.(1995)
With Item weight and Category weight: Mayekawa & Arai(2007)
7ExampleEight Graded Response Model items
3 categories for each item.
8Example (choosing weight)Example: Mayekawa and Arai (2008)
small posterior variance good weight. Large Test Information (TI) good weight
9Test Information Function Test Information Function is proportional
to the slope of the conditional expectation of X given (TCC), and inversely proportional the squared width of the confidence interval (CI) of given X.
Width of CIInversely proportional to the conditional
standard deviation of X given .
10Confidence interval (CI) of given X
11Test Information Functionfor Polytomous Items
ICRF
12Maximization of the Test Informationwhen the category weights are known.Category weighted Item Score
and the Item Response Function
13Maximization of the Test Informationwhen the category weights are known.
14Maximization of the Test Informationwhen the category weights are known.Test Information
15Maximization of the Test Informationwhen the category weights are known.First Derivative
16Maximization of the Test Informationwhen the category weights are known.
17Globally Optimal WeightA set of weights that maximize
the Expected Test Informationwith some reference distribution of .
It does NOT depend on .
18Example
NABCT A B1 B2 GO GOINT A AINT Q1 1.0 -2.0 -1.0 7.144 7 8.333 8Q2 1.0 -1.0 0.0 7.102 7 8.333 8Q3 1.0 0.0 1.0 7.166 7 8.333 8Q4 1.0 1.0 2.0 7.316 7 8.333 8Q5 2.0 -2.0 -1.0 17.720 18 16.667 17Q6 2.0 -1.0 0.0 17.619 18 16.667 17Q7 2.0 0.0 1.0 17.773 18 16.667 17Q8 2.0 1.0 2.0 18.160 18 16.667 17
LOx LO GO GOINT A AINT CONST 7.4743 7.2993 7.2928 7.2905 7.2210 7.2564 5.9795
19Maximization of the Test Informationwith respect to the category weights.Absorb the item weight in category
weights.
20Maximization of the Test Informationwith respect to the category weights.Test Information
Linear transformation of the categoryweights does NOT affect the information.
21Maximization of the Test Informationwith respect to the category weights.First Derivative
22Maximization of the Test Informationwith respect to the category weights.Locally Optimal Weight
23Globally Optimal WeightWeights that maximize
the Expected Test Informationwith some reference distribution of .
24Intrinsic category weightA set of weights which maximizes:
Since the category weights can belinearly transformed, we set v0=0, ….. vmax=maximum item score.
25Example of Intrinsic Weights
26Example of Intrinsic Weightsh()=N(-0.5, 1): v0=0, v1=*, v2=2
27Example of Intrinsic Weightsh()=N(0.5, 1): v0=0, v1=*, v2=2
28Example of Intrinsic Weightsh()=N(1, 1 ): v0=0, v1=*, v2=2
29Summary of Intrinsic WeightIt does NOT depend on , but
depends on the reference distributionof : h() as follows.
For the 3 category GRM, we found thatFor those items with high discrimination
parameter, the intrinsic weights tendto become equally spaced: v0=0, v1=1, v2=2
The Globally Optimal Weight isnot identical to the Intrinsic Weights.
30Summary of Intrinsic WeightFor the 3 category GRM, we found that
The mid-category weight v1 increases according to the location of the peak ofICRF. That is:
The more easy the category is,
the higher the weight .
v1 is affected by the relative location ofother two category ICRFs.
31Summary of Intrinsic WeightFor the 3 category GRM, we found that
The mid-category weight v1 decreases according to the location of the reference distribution of h()
If the location of h() is high, the mostdifficult category gets relatively high weight,and vice versa.
When the peak of the 2nd categorymatches the mean of h(), we haveeqaully spaced category weights:
v0=0, v1=1, v2=2
32Globally Optimal w given v
33Test Information
LOx LO GO GOINT CONST 30.5320 30.1109 30.0948 29.5385 24.8868
34Test Information
35Bayesian Estimation of from X
36Bayesian Estimation of from X
37Bayesian Estimation of from X
(1/0.18)^2 = 30.864
38ConclusionsPolytomous item has the Intrinsic
Weight.By maximizing the Expected Test
Information with respect to either Item or Category weights, we can calculate the Globally Optimal Weights which do not depend on .
Use of the Globally Optimal Weights when evaluating the EAP of given X reduces the posterior variance.
39References
40
ご静聴ありがとうございました。
Thank you.
41
42
43