Online Innovations to the Online Innovations to the Assessment Interview Assessment Interview

Impacts on Accuracy, Scoring, Impacts on Accuracy, Scoring, and Participant Reactions and Participant Reactions

Tom Janz, Ph DTom Janz, Ph D

Chief Science OfficerChief Science Officer

Behavior Description TechnologiesBehavior Description Technologies

.00 - .05 Graphology, Numerology, Astrology.00 - .05 Graphology, Numerology, Astrology

.05 - .15 Resume Ratings.05 - .15 Resume Ratings

.15 - .20 Unstructured 1-on-1 interviews.15 - .20 Unstructured 1-on-1 interviews

.20 - .24 Traditional reference checks.20 - .24 Traditional reference checks

.30 - .35 Panel Interviews.30 - .35 Panel Interviews

.35 - .40 Validated Personality Tests.35 - .40 Validated Personality Tests

.50 - .55 Cognitive Ability Tests.50 - .55 Cognitive Ability Tests

.50 - .55 Structured Behavior-based Interviews.50 - .55 Structured Behavior-based Interviews

.60 - .65 Ability Test plus Structured Interview.60 - .65 Ability Test plus Structured Interview

.65 - .70 Ability Test plus Work Simulation.65 - .70 Ability Test plus Work Simulation

Why the Assessment Interview went Why the Assessment Interview went BehavioralBehavioral

Why the Behavioral Interview is Why the Behavioral Interview is going Onlinegoing Online

Increase in Candidate Comfort:Increase in Candidate Comfort: Online Interviews rated as less stressfulOnline Interviews rated as less stressful Online Interviews leave a better first impressionOnline Interviews leave a better first impression Interviewers report feeling better prepareInterviewers report feeling better prepare

Increase in Decision Power:Increase in Decision Power: Study finds Online Interview .69 vs .48 Study finds Online Interview .69 vs .48 Campion & MuellerCampion & Mueller

Reduction in Staffing Spend:Reduction in Staffing Spend: Fewer finalists needed for on-site interviewsFewer finalists needed for on-site interviews Interviews started online take 30-50% less timeInterviews started online take 30-50% less time

The Road to Online InterviewingThe Road to Online Interviewing

No one thought candidates would type out No one thought candidates would type out answers to behavioral questions– Not even answers to behavioral questions– Not even Lowell Hellervik, Bill Byham, or Paul Green. Lowell Hellervik, Bill Byham, or Paul Green.

Following the lead of computerized testing, Following the lead of computerized testing, early versions put the questions on screen.early versions put the questions on screen.

Candidates typed whole answers into a text Candidates typed whole answers into a text box, guided by answer hints or probes. box, guided by answer hints or probes.

Innovations in the Assessment Interview Innovations in the Assessment Interview supported Online:supported Online:

Answer CoachingAnswer Coaching

Performance Auto-ConfirmPerformance Auto-Confirm

Best-Answer AnchoringBest-Answer Anchoring

Behavioral Judgment TestingBehavioral Judgment Testing

Innovation 1: Answer CoachingInnovation 1: Answer Coaching

Candidate Reactions to the Online Interview:Candidate Reactions to the Online Interview:Here are the questions we asked--Here are the questions we asked--

We graph the answer ratings from over 10,000 We graph the answer ratings from over 10,000 Restaurant Manager, 900 Pharmaceutical Sales, and Restaurant Manager, 900 Pharmaceutical Sales, and 70 Mine Manager candidates who completed a profile.70 Mine Manager candidates who completed a profile.

2 34 64

9 58 33

9 27 34

0% 20% 40% 60% 80% 100%

Restaurant Mgr

Pharma Sales

Mine Managers

On the degree to which E.ssessor is difficult and stressful:

MORE About the SAME LESS

37 55 8

52 44 4

74 24 1

0% 20% 40% 60% 80% 100%

Restaurant Mgr

Pharma Sales

Mine Managers

On the degree to which E.ssessor is complete and thorough:

MORE About the SAME LESS

27 62 11

32 61 7

60 34 6

0% 20% 40% 60% 80% 100%

Restaurant Mgr

Pharma Sales

Mine Managers

On the degree to which E.ssessor is a fair assessment:

MORE About the SAME LESS

61 38 1

57 42 1

71 29 0

0% 20% 40% 60% 80% 100%

Restaurant Mgr

Pharma Sales

Mine Managers

E.ssessor left me feeling positive about working for you:

MORE About the SAME LESS

39 50 11

37 52 11

64 33 3

0% 20% 40% 60% 80% 100%

Restaurant Mgr

Pharma Sales

Mine Managers

Compared to past interviews, I like E.ssessor:

MORE About the SAME LESS

N=924

N=10151

N=70

Innovation 2: Performance Auto-Confirm.Innovation 2: Performance Auto-Confirm.

Merely collecting a contact name and a Merely collecting a contact name and a means to connect compels more accurate means to connect compels more accurate self evaluations (bogus pipeline research).self evaluations (bogus pipeline research).

Users who provide email addresses for their Users who provide email addresses for their contacts enable a labor-less confirmation.contacts enable a labor-less confirmation.

Users who give permission to confirm their Users who give permission to confirm their performance at any time make auto-performance at any time make auto-confirmation possible. confirmation possible.

Here is how it works. Here is how it works.

Innovation 3: Best Answer Anchoring.Innovation 3: Best Answer Anchoring.

Most quality approaches to rating behavioral Most quality approaches to rating behavioral interview answers feature rating anchors.interview answers feature rating anchors.

Research finds behavioral anchors add about Research finds behavioral anchors add about 10 validity points over Lickert answers.10 validity points over Lickert answers.

However, behavioral anchors often take on However, behavioral anchors often take on the flavor of apple pie and motherhood. the flavor of apple pie and motherhood.

Performance component levels that are Performance component levels that are differentiated primarily by frequency differentiated primarily by frequency adjectives (still better than no guideline).adjectives (still better than no guideline).

How about providing BOTH performance How about providing BOTH performance components and REAL “Best Answers”? components and REAL “Best Answers”?

Innovation 4: Behavioral Judgment Innovation 4: Behavioral Judgment TestingTesting Situational Judgment Tests probe how people Situational Judgment Tests probe how people

WOULD ACT in challenging, job-related, situations. WOULD ACT in challenging, job-related, situations.

How people WOULD ACT cannot be confirmed. How people WOULD ACT cannot be confirmed.

How people WOULD ACT overlaps with their mental How people WOULD ACT overlaps with their mental ability for picking the most effective answers. ability for picking the most effective answers.

Behavioral Judgment Tests focus on what Behavioral Judgment Tests focus on what candidates DID DO, in challenging, job-related, candidates DID DO, in challenging, job-related, situations, seeking confirmation from third parties. situations, seeking confirmation from third parties.

Introductory Screen and Sample Item. Introductory Screen and Sample Item. Candidates begin by rating HOW OFTEN they Candidates begin by rating HOW OFTEN they faced this scenario.faced this scenario.

Since this scenario was faced more than 2 times, the Since this scenario was faced more than 2 times, the next rating anchors WHEN the Best Example took next rating anchors WHEN the Best Example took place. Then we show the first Action Option, and collect place. Then we show the first Action Option, and collect its similarity to what the person DID.its similarity to what the person DID.

Now we independently collect the candidate’s similarity Now we independently collect the candidate’s similarity rating to the second action option. (Note: The candidate rating to the second action option. (Note: The candidate does not know how many action options there are for does not know how many action options there are for each scenario.)each scenario.)

The Ratings Review Page displays ALL the action The Ratings Review Page displays ALL the action options (up to 5) and asks candidates whether they options (up to 5) and asks candidates whether they would like to adjust their ratings.would like to adjust their ratings.

Finally, candidate are asked to provide the name, Finally, candidate are asked to provide the name, position, and contact info for someone who can confirm position, and contact info for someone who can confirm their actions in the situation they selected for this their actions in the situation they selected for this scenario. scenario.