at-sat: a test battery for screening air traffic controller applicants gordon waugh laurie wise...
TRANSCRIPT
![Page 1: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/1.jpg)
AT-SAT:A Test Battery for Screening
Air Traffic Controller Applicants
Gordon WaughLaurie Wise
Human Resources Research Organization (HumRRO)
![Page 2: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/2.jpg)
Topics Today
Background of AT-SAT Overview of AT-SAT tests Air Traffic Scenarios Test Score weighting methods Setting the cut score
![Page 3: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/3.jpg)
Background of AT-SAT
![Page 4: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/4.jpg)
ATCS Selection backgroundATCS Selection background
Previous Selection System Written aptitude test battery Academy screening (9 weeks) On-the-job training
Issues Test utility after 12-15 years use Academy screening cost & fairness
![Page 5: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/5.jpg)
Air Traffic Selection and Training Air Traffic Selection and Training (AT-SAT) Validation Research(AT-SAT) Validation Research
Integrated validation study Job-performance criterion measures 8 new computerized predictor tests (6 hrs) Concurrent validation study on en route controllers
![Page 6: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/6.jpg)
Overview of AT-SAT Tests
![Page 7: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/7.jpg)
Dial ReadingLetter FactoryAT ScenarioAnglesEQApplied MathAnalogiesScan
![Page 8: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/8.jpg)
Dial Reading Test
Temp.
1) 22 2) 24 3) 28 4) 30 5) 45
![Page 9: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/9.jpg)
Order Boxes
Belt A Belt B Belt C Belt D
Quality Control
Conveyor Belt
Order Boxes AreaLoading AreaQuality Control Area
Box Storage Area
Available Zone
Letter Factory Test
![Page 10: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/10.jpg)
Order Boxes
Belt A Belt B Belt C Belt D
Quality Control
A
CD
P
B
![Page 11: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/11.jpg)
Which boxes should be in the loading area in order to correctly place all the letters on the belts?
1. One purple box and one orange box2. One green box, one purple box, and one orange box3. Two purple boxes, one green box, and one orange box4. Two green boxes, one orange box, and one purple box
Letter Factory Test - Situational Awareness Questions
![Page 12: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/12.jpg)
Air Traffic Scenarios Test
![Page 13: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/13.jpg)
Angles Test
This test includes two different types of questions:
The first presents a picture of an angle and asks you the measure of that angle in degrees (From 1 to 360)
What is the measure of this angle?
2) 10°1) 90° 3) 125° 4) 190°
![Page 14: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/14.jpg)
This test includes two different types of questions:
The second provides you with the measure of an angle (from 1 to 360 degrees). It then asks you to select an angle that represents that measure.
Which of the following represents a 10 degree angle?
1)
2)
3)
4)
![Page 15: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/15.jpg)
Experience Questionnaire
1- Definitely True2- Somewhat True3- Neither True nor False4- Somewhat False5- Definitely False
Your emotions have sometimes prevented you from solving a difficult problem.
![Page 16: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/16.jpg)
Applied MathApplied Math
The distance from point A to point B is 560 miles. If the aircraft left point A at 8:00 “Zulu”, and flew at 400 kts, what time would the aircraft cross point B?
A. 8:56 B. 9:02
C. 9:24 D. 10:02
![Page 17: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/17.jpg)
Analogies Test
Water: Liquid Ice: ?
Gas Cube Solid Oxygen Freeze (1) (2) (3) (4) (5)
![Page 18: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/18.jpg)
Visual Analogies:
: :
(1) (2) (3) (4) (5)
![Page 19: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/19.jpg)
V41610
Type the identification numbers contained in the data block with lower line numbers falling beyond the range (360-710):
B12810
P45580
T65120
Y85710
K23250
F75560
C20390
Scan Test
![Page 20: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/20.jpg)
Air Traffic Scenarios Test
![Page 21: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/21.jpg)
Simulation Overview
Description of simulation Design considerations Scoring Instructions and practice
![Page 22: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/22.jpg)
ATST: Description of Simulation
![Page 23: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/23.jpg)
• Display of airspace sector
• Control several planes
• Land at two airports
• Exit at four locations
AT Scenarios in a Nutshell
![Page 24: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/24.jpg)
F2eF2e
F4DF4D
S1S1AA
F3fF3f
F4B
M4C
![Page 25: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/25.jpg)
Plane Icon
M2BExit/Airport B
Altitude Level 2
Speed Medium
Heading
![Page 26: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/26.jpg)
Design Elements Environment: airspace sector with exits, airports,
planes, and controls Events: planes appear, move, and disappear Actions: user clicks controls to control planes. Rules: eight rules related to speed, altitude,
separation, etc.
![Page 27: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/27.jpg)
• Display of air sector:
• four exits
• two airports
• planes
• Plane controls
• Display of time remaining
Environment
![Page 28: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/28.jpg)
• Direction – 8 headings
• Speed - 3 levels
• Altitude - 4 levels
• Accept handoff
Actions: Plane Controls
![Page 29: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/29.jpg)
• New planes appear
• Planes move
• Planes disappear:
• land, exit, or crash
Events
![Page 30: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/30.jpg)
• Land at slowest speed• Land at lowest altitude• Exit at highest altitude• Land/exit at right place• Land in correct direction• Don’t fly over airports• Keep planes separated• Don’t crash!
Eight Simple Rules
![Page 31: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/31.jpg)
ATST: Design
Considerations
![Page 32: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/32.jpg)
• Measure KSAs
• Simulate wide range of job tasks
• Simulate a key job activity
Possible Approaches
![Page 33: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/33.jpg)
• Maximize score variance
• No disadvantage for computer novices
• Short testing time
Design Goals
![Page 34: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/34.jpg)
• Number of planes on the screen at one time
• Initial speed, level, heading, & location
• N of actions per plane
Difficulty Factors: Planes
![Page 35: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/35.jpg)
• Speed of plane movement• N and complexity of rules• N of exits/airports• N of controls• N of control levels• N of things to remember
Difficulty Factors: Other
![Page 36: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/36.jpg)
ATST: Scoring
![Page 37: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/37.jpg)
• Rule violations
• Separation errors
• Crashes
• Elapsed Time
Record the Outcomes
![Page 38: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/38.jpg)
• N of crashes & separation errors
• N of procedural errors
• Percent of successful flights
• Total delay time (handoff & en route)
Á Priori Rational Scales
![Page 39: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/39.jpg)
• Moved “flyovers” based on data• Standardize scores before
combining and weighting• Rescale scores if useful:
• reverse• fix skewness• make scores more sensitive to
differences at higher levels
Adjust Scales and Scores
![Page 40: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/40.jpg)
ATST:Instructions
&Practice
![Page 41: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/41.jpg)
• Will computer experience affect scores?
• If so, can you minimize its effect?
• If computer skill is job relevant, it’s not as serious.
Computer Skill Effects
![Page 42: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/42.jpg)
• Reduces effects of:• learning ability• computer game skills
Simulation Practice
Instructions with examples
Practice a few things in mini-trial
Give feedback and tips
Several mini-trials
![Page 43: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/43.jpg)
• Add practice trials until more trials add little to:
• reliability
• validity
Post-Instruction Practice
![Page 44: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/44.jpg)
• Pilot test with lots of trials
• Compute trial-total r
• Use last one or more trials with high rs
• Trial-criterion rs are even better if available
Which Trials to Score
![Page 45: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/45.jpg)
Score Weighting Methods
![Page 46: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/46.jpg)
Final AT-SAT Validity
Validity, corrected for shrinkage, range restriction, and unreliability in criterion:
.76 with composite criterion
.78 with CBPM
.38 with Ratings
N
1029
1032
1053
![Page 47: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/47.jpg)
Criteria
Reliability .80 - CBPM (test-retest) .71 - Ratings (interrater reliability) .76 - Composite (mean of CBPM & Ratings)
Composite = .6 * CBPM + .4 * Ratings r CBPM J Ratings = .24
![Page 48: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/48.jpg)
Uncorrected Test Validities
.41 Applied Math .38 Analogies .33 Angles .33 Letter Factory .32 Air Traffic Scenarios .27 Time Wall .25 Planes
![Page 49: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/49.jpg)
Uncorrected Test Validities (cont’d)
.24 Dials .23 Memory Retest .19 Memory .19 Scan .17 Experiences Questionnaire .14 Sound
![Page 50: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/50.jpg)
Goals
Find best set of tests for the final battery
Determine scale weights
Measure validity: How well test battery predicts job performance
![Page 51: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/51.jpg)
Decisions to be Made
Which predictors to keep
How to weight predictors
![Page 52: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/52.jpg)
Decision 1: Choosing Predictors
Phase I- Evaluate predictors according to: simple validity incremental validity fairness group differences test administration time
Predictors chosen by group consensus
![Page 53: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/53.jpg)
Decision 1: Choosing Predictors
Phase 2: Optimal weighting algorithm iterated regression negative weights set to zero maximizes R2 while minimizing differences in group means, slopes, and
intercepts
Many runs done while varying the relative importance of R2 and group differences (10 parameters)
![Page 54: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/54.jpg)
Decision 2: How to Weight Predictors
Alternatives Considered regression weights validity weights optimal weights unit weights
Chosen Weighting Method mean of validity and optimal weights
![Page 55: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/55.jpg)
Conclusions: Optimal Weights
An optimal weighting algorithm can balance the many considerations in selecting and weighting predictors
Optimal weighting tends to exclude too many predictors
Optimal weighting used in conjunction with another weighting method can perform very well
![Page 56: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/56.jpg)
Conclusions: Validity Weights
In the current study, validity weights performed very well, especially when combined with optimal weights
Research is needed to compare the shrinkage (due to overfitting and predictor selection) and R2 of validity weights to other weighting methods
![Page 57: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/57.jpg)
Estimating Shrinkage in R
Shrinkage formula has shortcomings Shrinkage formula corrects for overfitting but not predictor
selection Validity weighting - only selection shrinkage Optimal weighting - shrinkage is severe because of extra
parameters
![Page 58: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/58.jpg)
Conclusions: Shrinkage
Research is needed to help estimate shrinkage of R (due to overfitting and predictor selection) using validity weights and optimal weights.
Research needed to compare validity weights, optimal weights, and regression weights in terms of shrunken R2 (due to overfitting and predictor selection)
![Page 59: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/59.jpg)
Setting the Cut Score
![Page 60: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/60.jpg)
Regression Method
Set cut score on criterion Compute corresponding score on predictor based
on regression line Works well when R-squared is high and good
criterion is available
![Page 61: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/61.jpg)
Criterion Cut Point Options
Cut score set at 5% (for example) of current incumbents’ distribution
Set cut score such that mean expected performance of candidates passing is at 60th percentile (for example) of current incumbents
Set score at anchor on ratings scale (e.g., “acceptable.”
![Page 62: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/62.jpg)
Cut Point Considerations
% of applicants passing % of incumbents passing % of each minority passing adverse impact ratio If low pass rate:
need many applicants, or recruiting must target high-quality applicants
![Page 63: AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO)](https://reader035.vdocuments.site/reader035/viewer/2022062312/551a2cab550346545e8b4898/html5/thumbnails/63.jpg)
Questions?