global selection systems

8/4/2019 Global Selection Systems

http://slidepdf.com/reader/full/global-selection-systems 1/48

Global

SelectionSystemsCase Studies & Lessons LearnedCase Studies & Lessons Learned

©2008 Hogan Assessment Systems



Overview of Session

• Survey of Issues in Global Selection – Jarrett Shalhoop, Hogan Assessment Systems

• Validation of the Shell Recruitment Process – Thi Bui, Royal Dutch Shell

• Global Assessment Development &Implementation

– Kelly Kaminski, Starwood Hotels & Resorts

• Applying Assessments Across Cultures: ACase Study

– Amie Lawrence, Select International

• Q&A Session ©2008 Hogan Assessment Systems



Global Selection Issues

• Perspective – Transporting an assessment outside of the US

and using for selection/assessment

• General Issues

– Language/Translation

– Assessment Norms

– Validation

• Logistical• Psychometric

– Technology




Language/Translation

• Three approaches to translation (van deVijver & Leung, 1997)

– Application – literal translation

– Adaptation – altering as necessary

– Assembly – developing a parallel assessment

• Back Translation

• Forward Translation




Assessment Norms

3-Component Model of Error (Meyer & Foster, 2008)

Sample Differences

Translation Differences

Cultural Differences

Absolute Sample Size

Relative Sample Size

Sample Composition

Translation Quality

Lack of Congruous Words

Cultural Relevance

Strength of Item Wording

Response Styles

Reference Group Effects

True Cultural Differences

Session 143 ©2008 Hogan Assessment Systems



Validation

• Logistical – Coordinating research globally a challenge

• Psychometric – Culture as a moderator of relationship to

performance

– Manifestation of characteristics

• Cultural response biases

– Willingness to provide accurate information




Technology

• Compatibility – Front end

• Web-based assessment

• Light assessment portal

– Back End

• HRIS Integrations• HR-XML Compliance

• Intellectual Property

– Protection of IP




Human Resources

LEARNING

4/28/2008

Copyright: Shell International Ltd 2005

Global Selection Validation Study Results



About Shell

• Active in over 130 countries

• Employs 108,000 employees world wide

• What we do:

– Exploration & Production

– Downstream

– Gas & Power

– Trading

– Renewables & Hydrogen

– Global Solutions



Global Recruitment Process

• Online application form via www.shell.com/careers

• People who pass screening are invited for interview

• Candidates who are successful at the first interview will be invitedto take part in a formal assessment – a chance for us to assess

their suitability and for the candidates to get a real insight into whatworking life at Shell would be like.

• Graduates can choose one of the three routes to Shell for their

assessment – Shell Recruitment Day, Gourami or AssessedInternship – while experienced professionals take part in astructured interview.



Background and Objectives

• Validation study of the Graduate and Experienced Hire (XP) selection

processes using data from a convenient sample from 2005 and 2006 hireswith readily available performance scores and selection data.

• This study represents the first in a series of ongoing annual validationwork. It is the first global validation study examining the relationship

between selection processes and work performance,1st year IndividualPerformance Factor (IPF) across the Shell Group.

• Systematic efforts to better understand the effectiveness of the selectionprocesses, including; the competence framework, the assessments, the

assessment exercises, and how each of these function in the regions.

• A major objective of this initial study was to better understand how data iscurrently being collected and stored.



Graduate Assessments

• Campus Interviews are functioning differently across the regions

• Evidence supports the effectiveness of SRD

• Assessed Internships are not functioning optimally

• Evidence supports the effectiveness of Gourami assessments



Graduate Competence Framework

• Capacity competence dimension is effective in predicting workperformance however regional differences are present.

• Achievement competence dimension is effective in predicting workperformance.

• Relationship dimension scores are not currently predictive of work

performance.



Graduate Assessment Exercises

• Campus Interview shows a positive but not statistically significant

relationship with work performance.• Campus Interview is more positively correlated with work performance in

the Americas.

• SRD exercises show positive but not significant relationships with workperformance.

• Gourami exercises show positive correlations with work performance.

• Assessed Internship show negative correlations with work performance.



Graduate Assessment Recommendations

• Expand the range of outcome measures

– First year IPF was used as a proxy measure for work performance in this initial study. – This metric does not cover the full range of success outcomes

– Recommend using additional criteria, such as manager ratings

• Improve data collection and archiving

– Automate and integrate data collection as much as possible

– Collect data on applications that do not progress fully through the selection system – totrack flow percentages and pass rates at successive assessment steps

• Improve measures of Relationship competence on Campus Interview and SRD

– The Relationship dimension is a valuable one (as evidenced by the Gourami dimensioncorrelations). Shell Learning will recommend improvements to the Campus Interview andSRD measures.

• Improve predictive validity of Assessed Internship

– Shell Learning will conduct an examination of the training and process factors that may beinhibiting these measures.



XPA Assessments

• Experienced Hire assessment exercises show weak (or slightly negative)

correlations with IPF 1st

year after hire.• Functional scores vary in relation to work performance by exercise.

– Project Discussion functional scores are more indicative to future workperformance than functional scores from the Professional Interview.

– Possible that the Project Discussion is used as the final assessment, with theStructured Interview and Professional Interview acting as precursor orpreliminary hurdles.



XPA Competence Framework

• The XP Competence framework dimensions show weak (and often weak

negative) correlations with work performance.



XPA Recommendations

• 2005 and 2006 XPA was not globally mandated, there may have been a

diversity of practice in implementation. – More rigorous and standardized training and implementation are essential to

improving the performance of these assessments.



Tips & Hints

• Setting client expectations. Timing, Data Collection, Results

• Use first study to understand data collection practices and issues

• Establish a partnership with Recruitment Department to identify dataneeded

• Utilize manager rating forms in addition of other performance indicators

• Have as few focal contact points as possible to collect data (1 per region)



©2008 Starwood Hotels & Resorts Worldwide, Inc.

Assessment Around the World:A Case Study from Starwood Hotels

Kelly A. Kaminski & Monica A. HemingwayStarwood Hotels & Resorts Worldwide, Inc.

23rd Annual SIOP Conference

April 2008, San Francisco



Starwood: A Global Company…

World class hotel and spa brands

890 managed & franchised properties in over 100 countries

Approximately 400 hotels managed by Starwood; we oversee theHR systems

145,000 associates

Conduct business in as many as 40 languages



…With a Global Assessment Program

CallCenter

US, Canada,and Ireland

2 versionslocally norms

Englishonly

Assessment Usage Versions Languages

Sales andMarketing

400 hotelsglobally

Varies by leveland division

9

GuestService

US, UK,China

Local norms 14



Our Approach to Job Analysis

Think globally, act globally: involve experts around the world

Sift through thousands of job titles

Job observations, and SME focus groups

One branded core competency model+ One guest service training program

+ One performance management system

= A common understanding of the jobs



Jobs Vary Across Divisions:

An Example for Sales Managers

Standard Title ComparisonTask

OverlapTask

rComp

OverlapComp

r

Sales Manager NA AP 76.9% .75 95.2% .92

Sales Manager NA EAME 68.8% .86 90.9% .84

Sales Manager NA LA 68.8% .82 95.2% .85

Sales Manager AP EAME 75.0% .84 95.5% .82

Sales Manager AP LA 75.0% .75 90.9% .81

Sales Manager EAME LA 100.0% .82 95.5% .88



Rank Ordering Guest Service Competencies

in North America and Latin America

Importance Ranking (1-25)

North America Latin America

Problem Solving 16 1

Attendance 4 15

Coping with Stress 3 13

Persistence 23 8

Multitasking 8 18



Translation

Translate and back-translate

Involve I/Os who are native speakers of the language

Use in-country reviewers to capture local dialects

Translate before validation Have your validation sample provide feedback about the quality

of the translations



How Many Languages Do You Need?

Our Sales Leaders Said “Just a Few”

English

57%

Bahasa2%

Japanese

3%

Italian

0%German

13%

French5%

Spanish (EU)

1%

Spanish (LA)

9% Chinese10%

N = 281



Getting Technology To Speak Your Language

Starwoodhotels.com English French Spanish

German Chinese JapaneseItalian

ATS Recruiter Site Thai Polish Portuguese

Job Postings Malay Dutch Korean

Assessment Authoring English Spanish (LA) French (CA)



Call Center Candidates in North America

and Ireland Respond Differently

32 of 53 items had statistically significant differences inresponse patterns between Ireland and the US applicants.

How would you describe yourself?Ireland USA

25% Dependable 78%

44% Clever 9%

33% Happy 11%



Implementation and Delivery

Legal requirements and data collection/retention policies vary

Difficulty integrating assessments with e-recruit platforms

Must allow candidates to choose a language

Your norms or mine? Does a candidate test in their homecountry or the country where the job resides?



In Conclusion

Think globally, act globally to avoid problems and get buy-in

Same job title does not mean same job

Same language does not mean same responses

Translate early but scope the need firstGood luck with the technology…

23rd Annual SIOP Conference



Amie Lawrence, Ph.D.

Lance Andrews & Matthew O’Connell, Ph.D.

Applying Assessments Across Cultures: A Case Study

April 10-12, 2008

San Francisco

Our Case Study



y

GLOBAL AUTOMOTIVE MANUFACTURER

• Needed standardized entry-level assessment process in 4 countries

– US, Canada, Mexico, UK

– All locations’ applicants were primarily local

• Web-based assessment

– Personality, SJT, & Cognitive Ability – Administered in group proctored settings

– Part of a multiple-hurdle selection process

Issue #1: Candidate Pr ivacy



y

CANDIDATE PRIVACY

• Other countries have laws limiting the collection and/ortransfer of candidate personal information

W HAT DID W E DO?

• Did not collect demographic data from any candidatesoutside the US

• Obtained approval from all UK candidates regardingtheir personal data

Issue #1: Candidate Pr ivacy



y

LESSONS LEARNED

• Know where candidate data are stored and where they will be transferred

• Involve the client’s legal department in the planning

stages of the project to ensure that these issues areaddressed

Issue #2: Applicant Tracking



pp g

APPLICANT TRACK ING

• Unique Identifier• Database Configuration

W HAT DID W E DO?• Created a unique identifier from telephone number and

month and day of birth

• Reviewed all fields in the system to align with thatcountry

Issue #2: Applicant Tracking



LESSONS LEARNED

• Discuss these issues early in project life-cycle• Verify every field that a candidate completes throughout

the selection process

Issue #3: Assessm ent Content



ASSESSMENT CONTENT

• The American version of the assessment was notappropriate for use in any of the foreign countries

W HAT DID W E DO ?• Translated into Spanish

• Reviewed assessment content for phrasing and spelling

changes• Programmed changes

Issue #3: Assessm ent Content



LESSONS LEARNED

• Changes may be needed even in other English-speakingcountries

• Account for the time and resources needed for this step

Issue #4: Assessment Norm s



ASSESSMENT NORMS

• No local norms available• No concurrent validation study planned

W HAT DID W E DO?

• Delayed report availability

• Conducted Norm Analysis




LESSONS LEARNED

• Local Norms were necessary • Cultural & Regional differences were identified




EVIDENCE FOR LOCAL NORMS

• Web-based Assessment Measurements

– 6 personality scales

– 6 cognitive measures (SJT, IP, Cognitive Ability)

• Assessment Differences by Country

• Assessment Differences by Item Type

Personality Difference Scores



Canada Mexico UK

Teamwork -.13 +.58 -.05

Conscientiousness -.07 +.02 -.44

Locus of Control -.08 +.87 -.65

Positive Attitude -.03 +.05 -.49

Attn to Detail -.11 +1.11 -.46

Safety -.02 +1.31 -.15

Mean Difference Scores as Compar ed to US Norm s

Cognitive Difference Scor es



Canada Mexico UK

IP: Gauge +.43 -.76 -.05

IP: Count +.32 -.57 +.06

SJT: Teamwork +.04 -1.56 -.47

SJT: Safety +.09 -1.25 -.03

Qualitative PS +.15 -1.45 -.44

Quantitative PS +.48 -1.33 +.03

Mean Difference Scores as Compar ed to US Norm s

Sum m ary of Differences



• Applicants in Mexico scored higher than applicants in

the US on all 6 personality measures

• Applicants in Mexico scored lower than applicants in theUS on all cognitive measures with the largest differences

on SJT and traditional cognitive ability measurements

• Applicants in the UK scored lower than applicants in theUS on all 6 personality measures

• Applicants in Canada were most similar to applicants inthe US

Futur e Research Questions



• What is the impact on validity with local v. global norms?

• What is causing the score differences? Are they truescore differences or is it “cultural” variance?

• Are all item types developed in and for US companiestransferable across cultures?

• What are the best processes and procedures for ensuringthat assessments can be confidently applied across

cultures?

About Select Intern ation al



• Founded in 1993, Select International is a leading

provider of selection and development solutions forGlobal 2000 companies

• From entry-level to executive, organizations identify,

hire and retain top talent using Select International'scutting-edge products, systems and recruitment processoutsourcing services

• Headquartered in Pittsburgh, PA, Select maintainsoffices in Dallas, San Diego, Toronto, London and South Africa

global selection systems

Documents