WEB-BASED SCREENING OF APPLICANT PERSONAL CHARACTERISTICS

CAN NOW BE DONE RELIABLY AND VALIDLYSHOULD YOU USE IT?

THE DEVELOPMENT AND REAL-WORLD IMPLEMENTATION OF

COMPUTER-BASED ASSESSMENT FOR SAMPLING PERSONAL CHARACTERISTICS

- Harold Reiter- Kelly Dore

Stemmler Fund

WEB-BASED SCREENING OF APPLICANT PERSONAL

CHARACTERISTICS

WHY I’M WORRIED Length of talk It’s the economy, stupid

COMBINING TWO CONCEPTS

“If you nail two things together

that have never been nailed together

before… some s****k will buy

it”

THING 1 & THING 2THING 1 – situational judgment testing

THING 2 – online testing

SITUATIONAL JUDGMENT TESTING

Scores responses to hypothetical but realistic scenarios

1950’s, 1960’s used for managerial assessments1990’s – for workforce, medical and dental schools

PLAY FIRST CASPER VIDEO HERE

Question 1 – What is your decision and why?

Question 2 – Would your response be any different if this family has four younger children expected to attend the same school in future?

Question 3 – Would your response be any different if the school was in a district with a wide range of income – some very privileged families, some underprivileged?

PERSONAL/PROFESSIONAL CHARACTERISTICS – FINANCIAL

COST

$50,000,000,000

Profes-sionalism;

92%

Knowledge; 8%

STANDARDIZED TESTS IN 2011

Series1

0 500,000 1,000,000 1,500,000

SAT ACT GRE GMAT LSAT MCAT

ESSAYLIB.com

High-quality custom papers starting from $10/page

All papers written from scratchWe hire professional writers onlyOriginal and creative work

Custom Personal Statements

STANDARD MEASURES OF PERSONAL/PROFESSIONAL CHARACTERISTICS

PERSONALITY TESTING – BIG 5 FACTOR, CPI…

Cost, 2011

0 1,000,000,0002,000,000,000 3,000,000,000

United States United Kingdom

GPA MMIUSMLE Step 3.6

.4

.2

0

Clinical Clerk

MCCQE I cogn

MCCQE I C2LEO

MCCQE II pass

COGNITIVEPERSONAL/PROFESSIONAL CHARACTERISTICS

3rd yr GPA

Predictive Validity

CHARACTERISTICS OF THE DESIRED INSTRUMENT Absolute Requirements

Multiple samples Withhold control from examinees

Relative Requirements Highly context dependent Score hard with hard, soft with soft

Societal Requirements Cheap Fair to the less privileged

MEDICAL COLLEGE ADMISSIONS TEST – 94,000

$

020

040

060

080

010

0012

0014

0016

0018

0020

00

Prep CourseTravel/HotelChange TestInternationalRegister

HOW SMALL PROBLEMS LOOM LARGE WHEN CHANGING MEDIUM OPTION # 1

Moving from MMI to online video-based SJT

BELGIAN MEDICAL & DENTAL STUDIES ADMISSIONS EXAMINATION

1999 – 20025,444 test-takersVideo stems30 MCQ responses723 admitted

Internship performanceR = 0.22, p < 0.05

Stopped after 2002 - financial and technical constraints

CASPer

Computer-based Assessment for Sampling Personal characteristics

2003 – 2008: Psychometric research

2008 – 2010: Technological development

2010 – 2012: Implementation

8 Video Vignettes & 4 Self-Descriptive Non-clinical 3 follow-up questions 5 min short answer Scored by two raters

CASPerComputer-based Assessment for Sampling

Personal characteristics

PLAY SECOND CASPER VIDEO HERE

STUDENTS CHEAT OPTION # 2

How can a non-proctored online test be secure?

0%10%20%30%40%50%60%70%80%90%

Written Tests

Written Work

Total Admitted

Plagiarism isn't real cheating

How Common is Cheating?

CASPer as a pre-interview Screen

2006 – MCC – 110 applicants

2007 - NBME – 167 applicants

Reliable Inter-Rater Correlation with MMI.9

.6

.3

0

GPA MMIUSMLE Step 3.6

.4

.2

0

Clinical Clerk

MCCQE I cogn

MCCQE I C2LEO

MCCQE II pass

COGNITIVEPERSONAL/PROFESSIONAL CHARACTERISTICS

3rd yr GPA

Predictive Validity

2006 MCC & 2007 NBME STUDIES:

MCC 110 – 63 MCCQE Part I, 53 MCCQE Part II

NBME 167 – 46 MCCQE Part I, 12 MCCQE Part II

Stemmler Fund

CASPer.6

.4

.2

0

COGNITIVEPERSONAL/PROFESSIONAL CHARACTERISTICS

MCCQE I C2LEOMCCQE I PHELOMCCQE II C2LEO

MCCQE Part I med expert

Predictive Validity

p < 0.05

MICHAEL G. DEGROOTE SCHOOL OF MEDICINE

# Applicants

0 1000 2000 3000 4000 5000

201220112010

TARGET < 5% Incomplete

Helpde

sk Res

olved

Backu

p Date

Resolv

ed

Declin

ed to

Writ

e

Incom

plete

0

2

4

6

8

10

12

14

16

201020112012

TARGET < 5% Incomplete

ONGOING RESEARCH OPTION # 3

What further research can be done/is being done?

WHAT’S IMPORTANT TO ME OPTION # 4

Under what circumstances, if any, does CASPer become useful to me?

THANKS FOR PARTICIPATING!

Harold Reiter MD MEd FRCPC DABR reiterh@mcmaster.ca

Kelly Dore PhD dore@mcmaster.ca

HOW SMALL PROBLEMS LOOM LARGE WHEN CHANGING MEDIUM OPTION # 1

Moving from MMI to online video-based SJT

STUDENTS CHEAT OPTION # 2

How can a non-proctored online test be secure?

0%10%20%30%40%50%60%70%80%90%

Written Tests

Written Work

Total Admitted

Plagiarism isn't real cheating

How Common is Cheating?

ONGOING RESEARCH OPTION # 3

What further research can be done/is being done?

WHAT’S IMPORTANT TO ME OPTION # 4

Under what circumstances, if any, does CASPer become useful to me?