patterson
DESCRIPTION
Fiona Patterson's presentationTRANSCRIPT
Evaluating new methods & approaches for high stakes selection:
Implications for policy & practice
Professor Fiona Patterson Work Psychology Group & University of Cambridge
University of Athens, 2011
Overview
• The selection process & best practice• Case studies of new selection methods & approaches
in high stakes selectionCase Study 1: Selecting DoctorsCase Study 2: Selecting Private Bankers
• Implications for policy & practice
Create competency
model
Identify selection criteria
Make selection decisions
Create person specification
Knowledge, skills, abilities, characteristics
JobAnalysis
Tasks, roles, responsibilities
Required level of competence
Choose selection methods
Interviews, Simulations,
Psychometric tests
Reliability, validity, utility, fairness
Validate selection decisions
Attract pool of applicants
Selection
Select out unsuitable applicants
Select in (rank) suitable
applicants
The Selection Process
Evaluate candidate reactions
Case Study 1.Selecting doctors in the UK National Health Service (NHS)
Organisational context• High profile, strong public interest• Large applicant numbers - security risks• Legal scrutiny of selection processes• Applicants above average intelligence• Strong professional ‘Trade Union’ • Independent coaching firms - sole purpose is to help
applicants be successful in selection
“Work for me, son – I knew your father.”
1970
“ Fill out the application form for HR
and the job is yours, mate.”
1980 1990
“It isn’t an interview – just
an informal chat, sweetie.
Just a formality.”
Selection in medicine through the ages…
Help!
• 23,000 applicants for 8,000 medical school places
• 8,000 medical students apply for their first post
• 10,000 speciality applicants
• 24,000 + interviews
• Weeks of offering, rejecting, cascading 1000s Consultant hours
What attributes are important to be an effective clinician?
What selection methods are available to test these attributes?
Given the costs, beyond some basic assessment, is a lottery the best option?
Key questions
Why not use a lottery system?
Selecting doctors in the UK
• 8,000 applicants per year for 3,250 training posts in a centrally coordinated recruitment office– job analysis – design of situational judgement tests (SJTs)– validation studies
Job analysis Three independent studies (Patterson et al, 2000):
• interviews with doctors• observations studies• interviews with patients
1) Empathy & sensitivity (sensitive to patient's emotions & feelings)2) Communication skills (active listening, clarity of explanation)3) Problem-solving (identifies root cause & decision-making)4) Professional integrity (respect, vocational enthusiasm)5) Coping with pressure (calm under pressure)6) Clinical expertise (clinical process awareness, identifying options)
Job Role
“…establish immediate empathic rapport…define nature, history of problems, the aetiology, patient ideas, concerns, expectations, effects of problems….consider other continuing problems/at-risk factors…reach shared understanding of problems with patient….choose appropriate action for each problem…use time/resources appropriately” (Pendleton et al, 1984)
• Role requires rapid decision-making in an interpersonal environment, involving complex & significant cognitive/ non-cognitive demands of the individual
Using Situational Judgement Tests (SJTs) to select doctors
What’s a Situational Judgement Test?• Situational Judgement Tests (SJTs) are a measurement
method designed to assess judgement in work-relevant situations:– Present challenging situations likely to be encountered at work– Make judgements about possible responses– Scored against expert responses
• Validity evidence is well-established
Example SJT itemYou are reviewing a routine drug chart for a patient with rheumatoid arthritis during an overnight shift. You notice that your consultant has inappropriately prescribed methotrexate7.5mg daily instead of weekly.Rank in order the following actions in response to this situation (1= Most appropriate; 5= Least appropriate)A Ask the nurses if the consultant has made any other drug errors
recentlyB Correct the prescription to 7.5mg weeklyC Leave the prescription unchanged until the consultant ward round the
following morning D Phone the consultant at home to ask about changing the prescription E Inform the patient of the error
Validation studies
• Cost-effective to develop, administer & score• Validation studies show good predictive validity• Favourable applicant reactions• Number failures reduced from 8% to less than 1%• Significant savings in human cost (to doctors & patients)• Significant savings in financial cost
Case Study 2.Selecting Private Bankers
Resolving Tensions between Growing Profit & being a Trusted Advisor for Private Bankers at RBS
Private Bankers
What attributes are important to be an effective private banker?
What are the job role requirements?
What selection methods are available to test these attributes?
Key questions
Job analysis of Private Bankers
• Provide investment services to customers across a number of services including investment & portfolio management; tax, estate planning & stock broking
• Capabilities include:– Build excellent client relationships & looks after the
client’s long-term interests – Generating income – Act as a part of a team– Follow regulations & manage risk
Private Banker Capability Dynamics
Tension / Potential Conflict
Positive interaction
BUILDING CLIENT RELATIONSHIPS
BEING A TRUSTED ADVISOR
ORGANISATION, COMPLIANCE & MANAGING RISK
TEAM INVOLVEMENT
GROWING PROFIT
Using Situational Judgement Tests (SJTs) to select bankers
Example Situational Judgement Test item You have a meeting with a potential new client scheduled for later today. His name is Ioannis, he is 49 years old and a Director of an company called Computers-R-Us. Computers-R-Us banks with your company and Ioannisholds personal banking accounts with Barclays (a competitor). Ioannis is planning to sell his business and is likely to have substantial money to invest as a result.
Rank the importance of information you wish to elicit during the meeting (1= Most appropriate and 5 = Least appropriate)
A An understanding of how much income Ioannis has – now and in the future both in terms of what he earns and spends
B Details of the intended sale of Computers-R-US – when this will take place, how much it is likely to be sold for, who are the key decision makers etc
C What types of personal banking services Ioannis already uses and any areas where your bank could enhance this service
D Details about Ioannis’s personal/family situationE Any interests (business or otherwise) Ioannis has outside of Computers-R-Us.
Example Situational Judgement Test item You have a meeting with a potential new client scheduled for later today. His name is Ioannis, he is 49 years old and a Director of an company called Computers-R-Us. Computers-R-Us banks with your company and Ioannisholds personal banking accounts with Barclays (a competitor). Ioannis is planning to sell his business and is likely to have substantial money to invest as a result.
Rank the importance of information you wish to elicit during the meeting (1= Most appropriate and 5 = Least appropriate)
A An understanding of how much income Ioannis has – now and in the future both in terms of what he earns and spends
B Details of the intended sale of Computers-R-US – when this will take place, how much it is likely to be sold for, who are the key decision makers etc
C What types of personal banking services Ioannis already uses and any areas where your bank could enhance this service
D Details about Ioannis’s personal/family situationE Any interests (business or otherwise) Ioannis has outside of Computers-R-Us.
Implications• Job analysis results used to explore the dynamics between
capabilities - which may help make better selection decisions• Understanding how someone manages the tensions in their
role• Gives applicants a realistic job preview• Evaluation shows early signs of improved predictive validity &
added value • Challenge – there is reluctance from some stakeholders to
expose the tensions in the role requirements
Implications for policy & practice• Importance of bespoke job analysis in high stakes selection • Selection methods in high stakes settings must reflect the
capability dynamics & tensions in the job role• SJTs useful & valid method for assessing important
professional attributes • Strong business case for the added value of bespoke
evidence-based measures, although persuading client groups is often challenging
• Evidence-based approach is the key to stakeholder buy-in where results are tailored to the organisational context
Generic competency frameworksBenefits?• Defines what is required of all employees and/or leaders• Seen to act as ‘glue’ that binds people together• Provides sense of what is important (values, culture)• Drives consistency in performance management, development, induction,
appraisal, succession planning, etcLimitations?• Fails to identify role specific requirements that are different eg. HR
consulting model, motivational qualities associated with call centre tenure• Comprehensiveness is overwhelming & difficult to use by line managers• Inhibits use of business specific priorities• Difficult to refresh – scale of change
Designing a new selection system for Private Bankers
• Move to develop role specific capability frameworks:– Off-the-shelf competency frameworks may prove invalid and unreliable
without situational customisation (Raelin & Cooledge,1995)• Focus on core attributes that underpin success• Describe in term of
– Motivation, ability, interpersonal characteristics• Simplify to enable change to take place ‘just in time’ • Build for the line, not HR / psychologists• Provide complexity where needed and to those who need it• Understand the interplay between capabilities (dynamic model)