a comparison of exposure control procedures in cats using the 3pl model
TRANSCRIPT
![Page 1: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/1.jpg)
A comparison of exposure control procedures in CATs using the 3PL model
![Page 2: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/2.jpg)
Andrey Leroux Myriam LopezIan HembryBarbara Dodd
![Page 3: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/3.jpg)
• The purpose of the study• Compare the progressive-restricted standard error, the randomesque, the Sympson-Hetter and no exposure control methods in computerized adaptive testing• Manipulated conditions:
• Item pool size• Stopping rules
• Criteria• Bias• RMSE• Item utility• Item overlap
![Page 4: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/4.jpg)
• The application of computerized adaptive tests and computer-based tests has increased.• Reduce item exposure rates and increase item pool usage.
• Refers to constraining the administration of more popular items that would otherwise become compromised due to repeated administrations (Georgiadou, Triantafillow, & Economides, 2007)• Guarantees more variety in the items the examinees receive.
![Page 5: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/5.jpg)
Variables related to the control of item exposure• Precision of measurement
• The degree that the CAT system with exposure controls estimates examinees’ abilities when compared to the examinees’ known abilities.• Exposure rate
• The number of times an item is administered to the total number of CATs administered.• Pool utilization
• The percentage of items not administered throughout any of the CAT administrations.• Test overlap
• The number of common items amongst the examinees.
![Page 6: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/6.jpg)
![Page 7: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/7.jpg)
Randomization strategies
• Randomly select an item for administration from a group of several items near the optimal level of maximum information.• 5-4-3-2-1 strategy (McBride & Martin, 1983)• Randomesque strategy (Kinsbury & Zara, 1989)
• Repeated selects the same number of the most informative items from which one is randomly selected for administration through testing.• Decrease the overlap in items seen by examinees of similar abilities.
![Page 8: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/8.jpg)
Conditional strategies
• Specify a desired maximum value and use the exposure control parameters to control whether or not the item can be administered. • The values of these parameters have to be set through an iterative adjustment process in which at each step the effects of the previous adjustments are estimated using computer simulations of adaptive test administrations.
![Page 9: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/9.jpg)
Sympson-Hetter (SH)
• Let t denote the iteration steps; P (t)(Ai|Si), the value of the control parameter for item i at step t; and P (t)(Si) and P (t)(Ai), the probabilities of selecting and administering item i at step t. If the simulation at step t is completed, P (t)(Si) and P (t)(Ai), are estimated, and for items for which the estimates of P(t)(Ai)do not meet the requirement, the values of the control parameters are adjusted.
![Page 10: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/10.jpg)
Stratified strategies• When using maximum information, items with large a are more likely to be selected than those with small a values.• Stratify the item pool and are constrained to be administered from a given strata.• a-stratified procedure (Chang & Ying, 1999)
• Group items with similar a values and select within a group at each stage.• a-stratified with b-blocking (Chang, Qian, & Ying, 2001)
• Group items into M blocks in ascending order of b-parameter values. Then, each of the M blocks is stratified into K strata according to their a parameters.
![Page 11: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/11.jpg)
Combined strategies
• Two or more than two procedures are combined.• Progressive-restricted procedure (PR) (Revuelta & Ponsoda, 1998)
• Progressive--Decide the maximum exposure rate (100k)%• Restricted—
s=h/m• Progressive-restricted standard error procedure (PR-SE)
• s: the ratio of stopping rule SE over the current SE.• Dichotomous 3PL model: Administer fewer items than PR, but yield similar correlations between estimated and known theta and low item overlap as PR.• Polytomous partial credit model: use most of the item pool.
![Page 12: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/12.jpg)
Method• Study design
• Four exposure methods• Progressive-restricted standard error procedure (exposure rate =.3)• Sympton-Hetter (exposure rate =.3)• Randomesque (five items)• No exposure control
• Dichotomous 3PL model• 2 item pool sizes (300, 540 items)• Two stopping rules
• fixed-length with 50 items• Variable-length with .3 of the SE or a maximum of 50 items
![Page 13: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/13.jpg)
• Item pools• Nine different test forms
• Each contained 60 items, so the large item pool has 540 items• Randomly selected 5 of the 9 forms for the small item pool
• Each test form contained 6 content areas• 24%, 16%, 15%, 15%, 23%, 7%
• Data generation• SAS macro IRTGEN• 200 datasets
![Page 14: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/14.jpg)
• CAT simulations• Content balancing
• Kingsbury and Zara (1989) : the every next item is chosen from the item content group with the largest difference in percentage between desired and current.• EAP for ability estimation.
• Criteria• Bias, RMSE, item exposure rates, pool utilization and item overlap across test administrations.
![Page 15: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/15.jpg)
Results
![Page 16: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/16.jpg)
Since EAP was estimated, theta are shrink-aged, and the negative theta will be positively biased.
![Page 17: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/17.jpg)
![Page 18: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/18.jpg)
![Page 19: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/19.jpg)
![Page 20: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/20.jpg)
Discussion
• The PR-SE method yielded good precision, low non-administration rates, and reduction in item overlap.• Suggestions for use:
• Randomesque: minimal exposure control with high precision and test efficiency• SH: for more balance between precision and item exposure control • PR-SE: higher item exposure control
![Page 21: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/21.jpg)
Comments
• Stratified strategies have been used in CAT. Why the authors stated these method are seldom used?• By definition, bias is the difference between the expected estimate and the parameter, and RMSE is the root of averaged squared difference between the estimate and the parameter. Therefore, Equation 2 should be called averaged residual, and Equation 3 called the root of squared averaged residual.• Barrada, Olea, Ponsoda, & Abad (2008) incorporated a quadratic form to have a nonlinear function. The purpose of nonlinear s is to control the contribution from a random component and to make information component more flexible.
![Page 22: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/22.jpg)
Future studies
• Investigate the impact of different distributions of ability• Compare to other exposure control procedures• The PR combined with the different stopping rule• The PR-SE method should be compared with contemporary ones like on-the-fly SH method, omega method proposed by Chen (2011).• It is possible to incorporate a nonlinear s to the Progressive-restricted standard error procedure (PR-SE). The s parameter in PR-SE method is defined as the ratio of stopping rule SE over the current SE when administrating an item.
![Page 23: A comparison of exposure control procedures in CATs using the 3PL model](https://reader035.vdocuments.site/reader035/viewer/2022062407/56649db15503460f94aa0489/html5/thumbnails/23.jpg)
• Another possible idea is to apply PR method and PR-SE method to CDM-CAT.