dr. christof obermann jarosław chybicki assessment center around the world: where do we stand? what...
TRANSCRIPT
Dr. Christof Obermann
Jarosław Chybicki
Assessment Center around the world: Where do we stand?
What is new?
1. Assessment Center – how common is this method?Look on the usage around the worldSpecial: Empirical survey of AC application in
Germany2. How valid are Assessment Center?
Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
Importance of personal networks
+ France, ItalyLegal regulations, labor
market Workers Council
+ Germany, Italy, France- Switzerland
Tradition of psychometric
testing- Germany
- + Great Britain
Openness of the culture / exports
+ Netherlands, Northern Europe,Switzerland
AC-Use
Different weight of criteriae.g. Eastern Europe /
Southern Europe
AC use in European countries
Abb. 1.4-8
German-speaking countries USA
Group discussion
In-Basket Interview Role play Presentation Fact-finding. case study
´Testing0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Sources: Arbeitskreis AC, 2001 Spychalski et al. , 1997
Type of exercises: Europe vs. USA
4
3,7
3,7
3,5
3,4
3,4
3,3
3,1
3,1
3
3
2,9
2,9
2,8
2,7
2,6
2,6
2,5
2,3
1,7
1,6
0 0,5 1 1,5 2 2,5 3 3,5 4 4,5
Malaysia
Schw eden
Belgien
Neuseeland
Südafrika
Großbritannien
Portugal
Spanien
alle Länder
Niederlande
Irland
Hong Kong
Kanada
Australien
Griechenland
Singapur
Frankreich
USA
Japan
Deutschland
Italien
Einsatzhäufigkeit
Usage of psychometric testing
1. Assessment Center – how common is this method?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
73,4%
26,6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Yes No
companies and AC
Most of listed companies uses AC
Be
gin
nin
g o
f A
C
Number
Use of AC-method since …
Frequency of the AC
study 2007
study 2001
No information
Increasing Constant Declining
Is AC rising or declining?
AC runs in one Year
study 2007
study 2001
How many runs by company?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Personalauswahl Entwicklungsmaßnahme Potenzialanalyse
AC Studie 2008 AC Studie 2001
Objectives of AC?
Yes No No Information
Accomplishment necessary for next steps
study 2007
study 2001
Other
Biographical Questionnaire
Group Discussion with role players
Computer – based model
Fact Finding
Role play
Personality test
Cognitive testing
In Basket
Group Discussion with candidates
Interview
Case study
One-on-one Interview
Presentation
What exercises / elements used?
study 2001
study 2007
Internal psychologist
Coach
Board member
Experts
Other:
Agent
Workers council
Manager
Consultants
HR
Who are the assessors?
Average number of participants in a AC
study 2007
study 2001
131 2-4 5-7 8-10 11-13
How many participants attending?
study 2007
study 2001
On objectives
On dimensions
None Behavior expected
Information for candidates provided?
Feedback
Verbal and in written
form
Verbal In written form
Neither
study 2007
study 2001
How does the participant receive feedback?
Just after Assessment
Up to one week later
More than one week
later
No information
study 2007
study 2001
When is the feedback given?
Up to 15 15-30 30-45 45-60 60-90 more than 90 minutes
Feedback in minutes
study 2007
study 2001
How long does the feedback take?
Up to 1 year 1 – 2 3 – 4 more than 4
study 2007
study 2001
Second chance?
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
• Knows (and develops) own strengths and weaknesses• Ability to learn
Motivation and ability to learn
• Develops new ideas / innovations / inventions which could be named as “out of the box”
• Open for new ideas
Openness
• Physical fitness• Copes with pressure, high expectations and complexity while
maintaining performance level
Toughness and stress management
• Demands from others´ only what he / she is willing to do him- / herself
• Acts as an ambassador for important values and aspirations• Is open about personal beliefs and feelings
Extraversion / social competency
• Shows commitment, determination and enthusiasm in delivering required business results
• Developing of (own) ambitious goals (driving for excellence)
Drive and Commitment
• Expands the big picture by detecting long-term and side-effects or new interdependencies
• Explains complex subjects in simple words and abstracts the relevant messages
• Above average grasp of new subjects
Handle complexity
Observable behaviorPotential criteria
Future leaders - what is expected in general?
M a n a g e m e n t - P r o g r e s s S t u d i e v o n A T & T
A b b . 3 . 2 - 5
P r ä d i k t o r
K a n d i d a t w i r d i n a c h tJ a h r e n i m m i t t l e r e nM a n a g e m e n t s e i n
K r i t e r i u m
N a c h a c h t J a h r e nt a t s ä c h l i c h i m m i t t l e r e nM a n a g e m e n t
N M i t C o l l e g e O h n e C o l l e g eJ a 2 0 3 6 4 % 4 0 %N e i n / f r a g l i c h 2 1 9 3 2 % 9 %
V a l i d i t ä t s k o e f f i z i e n t . 4 6 . 4 6
N a c h T h o r n t o n & B y h a m , 1 9 8 2
1948 – Historical management progress study
Validity measures of diagnostic tools
1998 – Metaanalysis
Validity Additive validity
Additive percentage on validity
Standardized regression
Intelligence test .51 Work sample .54 .12 24% .41 Integrity test .41 .14 27% .41 Interview - structured .51 .14 27% .41 Interview – not structured .31 .07 14% .27 Test on technical knowleged .48 .07 14% .31 First weeks in the job .44 .07 14% .31 Peer rating .49 .07 14´% .31 Job reference letters .26 .06 12% .26 Assessment center .36 .01 2% .14 Graphology .02 .00 0% .02
2003 - Validity of Assessment Center by dimension
Meta-analysis: Regression of the AC-dimensions
15 Added dimension R R² R²
1 Problem solving .39 .15
2 Influencing others .43 .18 .03
3 Organizing and planning .44 .19 .01
4 Communication .45 .20 .01
5 Drive .45 .20 .00
6 Consideration/Awarness of others .45 .20 .00
Gradual regression, starting with highest validity
Source: Arthur, Day & McNelly (2003)
p SDpUntere Grenze
Obere Grenze
Gesamt 11.136 106 40 .22 .14 .26 .13 -.01 .52
Leistung 4.198 49 29 .20 .08 .28 .00 .28 .28
Training 3.503 15 10 .31 .10 .35 .09 .16 .53
Absatz 267 15 4 .11 .05 .15 .00 .15 .15
Beförderung 1.738 13 10 .27 .15 _d _d -.02 .56
Fluktuation 1.430 9 6 .07 .15 _d _d -.22 .36
Anmerkungen: a Anzahl der unterschiedlichen Validitäten aus unterschiedlichen Stichproben. Dies ist die Anzahl der Validitäten, auf der die Berechnungen basieren. b Für Beförderung und Fluktuation waren keine Korrekturen bezüglich der Kriteriumunreliabilität möglich
Anzahl der
Studien
(k)a
Anzahlderr's
Stich-proben-größe
Korrigiertbzgl.
Kriteriums-unreliabiltät
90%-Vertrauens-intervalle
SDr
mittleresr
Hardison, C.M. & Sacket, P.R (2007)
1. Enrollment and clarification of roles
2. Job analysis
3. Development of exercises
4. Observation and rating
5. Selection of observers and training
6. Pre-selection and preparation
7. Organization and execution
8. Feedback and follow-up
9. Evaluation
http://www.arbeitskreis-ac.de/
German Quality Standards on AC
Pitfalls:Simple sampling of criteriaJob analysis only looking to the pastBiased methods which lead only to simulation or testing
Job analysis
Pitfalls:Use of ethically difficult exercises like NASA exerciseFocus on group discussions for economic reasons“If you were ...“ simulationsInstructions with assignments for behavior
Exercises
Pitfalls:No operationalizing of criteriaFinal assessment without pre-defined rules
Ratings
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
The more content the more valid …
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
Protocol vs. guided assessment„Protocol assessment“ „Guided assessment“
Kundenorientierung
Customer orientation
Conflict ability
Argumentation
notes notes
Different types of observation tools
Observation sheet 1: Type: polarity profile - unspecific appellation of poles
Dimension: Conflict ability
Definition: Does not avoid conflicts, tries to find solutions, includes oppinion of others
Polarity profile
1 2 3 4 5
Has to concentrate on one single project
Is able to handle more than one project at the same time
Wird mit vagen Projektlinien fertig, sofern sie in konkreten Strukturen oder Methoden verankert sind
Kann mit nichtspezifizierten Zielvorhaben und unter Unsicherheit arbeiten, sofern gelegentlich Rückgriff auf einen konkreten Orientierungspunkt möglich ist
Tolerates ambiguity in projects and is able to handle long-term projects
Observation sheet 2: Type: polarity profile - unspecific appellation of poles
Did
not
sho
w t
he o
bser
ved
beha
vior
Did
not
mat
ch t
he n
eede
d cr
iter
ia
Unt
erdu
rchs
chni
ttli
ch:
erfü
llt
im A
llge
mei
nen
nich
t di
e er
ford
erli
chen
V
erha
lten
skri
teri
en -
qu
anti
tati
v un
d qu
alit
ativ
Dur
chsc
hnit
tlic
h:
ents
pric
ht d
en
Ver
halt
ensk
rite
rien
-
quan
tita
tiv
und
qual
itat
iv
Übe
rdur
chsc
hnit
tlic
h:
über
stei
gt g
ener
ell d
ie
erfo
rder
lich
en
Ver
halt
ensk
rite
rien
-
quan
tita
tiv
und
qual
itat
iv
Ver
y m
uch
abov
e-av
erag
e:
show
muc
h m
ore
than
ne
eded
0 1 2 3 4 5
Comments
Different types of observation tools
Different types of observation tools
Observation sheet 3:Type: behavioral checklist
Dimension: Decision orientation
Definition: Tries to find solutions, arranges effective checkup actions
Quotes 1 2 3 4 5 - + Structures the conversation (asks questions...)
Analyses, asks questions to find reasons for the problems
Sets clear directives
Structures partial steps, that could solve the problem
Arranges checkup
Overall score
Observation sheet 4:Type: Behavioral coding
Different types of observation tools
Different types of observation tools
Exercise oriented
Observation sheet 5: Type: Exercise oriented
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
Festsetzung eines einheitlichen Standards
Visualisierung der Beurteilungstendenzen/Wahrnehmungsfehler
Höhere Akzeptanz Gesteigerte Effizienz
Schritt 1
Professionelles Aufzeichnen der wichtigsten Übungen auf Video
Schritt 2
Erstellung eines differenzierten Benchmarks
Schritt 3
Beobachtertraining: Darbietung des Videos / Individuelle Beobachtung
Schritt 4
Abgleich Expertenurteil / Beobachterurteil, Reflexion der persönlichen Beurteilungstendenzen
Oberserver training according frame of reference
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
W i e d e r h o l u n g s s i m u l a t i o n i m A C
A b b . 4 . 3 - 1
V o r m i t t a g
M i t a r b e i t e r -R o l l e n s p i e l
B e u r t e i l u n g A
M i t t a g
I n t e r v e n t i o n
N a c h m i t t a g
M i t a r b e i t e r -R o l l e n s p i e l
B e u r t e i l u n g B
P r o b l e m l ö s e k o m p e t e n z
E i n s t e l l u n g e n / W e r t e
K o n z e p t i o n e l l eF ä h i g k e i t e n
A n a l y s e v e r m ö g e n / S t r a t e g i s c h e sD e n k e n
E n t s c h e i d u n g s v e r h a l t e n /H a n d l u n g s o r i e n t i e r u n g
A r b e i t s o r g a n i s a t i o n
F a c h w i s s e n
E i n s t e l l u n g e n u n dW e r t e
L e i s t u n g s m o t i v a t i o n
U n t e r n e h m e r i s c h e s D e n k e n
L e r n b e r e i t s c h a f t
K o m m u n i k a t i v e K o m p e t e n zS o z i a l e K o m p e t e n z K o n t a k t v e r h a l t e n
Ü b e r z e u g u n g s k r a f t
K o o p e r a t i o n s v e r h a l t e n
D u r c h s e t z u n g s v e r m ö g e n
P r o b l e m l ö s e k o m p e t e n z
E i n s t e l l u n g e n / W e r t e
K o n z e p t i o n e l l eF ä h i g k e i t e n
A n a l y s e v e r m ö g e n / S t r a t e g i s c h e sD e n k e n
E n t s c h e i d u n g s v e r h a l t e n /H a n d l u n g s o r i e n t i e r u n g
A r b e i t s o r g a n i s a t i o n
F a c h w i s s e n
E i n s t e l l u n g e n u n dW e r t e
L e i s t u n g s m o t i v a t i o n
U n t e r n e h m e r i s c h e s D e n k e n
L e r n b e r e i t s c h a f t
K o m m u n i k a t i v e K o m p e t e n zK o n t a k t v e r h a l t e n
Ü b e r z e u g u n g s k r a f t
K o o p e r a t i o n s v e r h a l t e n
D u r c h s e t z u n g s v e r m ö g e n
D i r e k t e R ü c k m e l d u n g d u r c h B e o b a c h t e r
S e l b s t - E i n s c h ä t z u n g
V i d e o a n a l y s e d e s T e i l n e h m e r s
G r u p p e n t r a i n i n g
Basic idea of learning Assessment
Abb. 4.3-3
Lerngewinne nach Rückmeldung
0
2
4
6
8
10
12
-0,67 -0,33 0,00 0,33 0,67 1,00 1,33 1,67 2,00 2,33 2,67 3,00 3,33 3,67
Anzahl
Quelle: Obermann, 1994
Typcial result of a learning center
1. Assessment Center – how common is the procedure?Look on the usage around the worldSpecial: Empirical survey of AC application in Germany
2. How valid are Assessment Center? Classic AT&T StudyNew Meta-Analysis on validity Comparison to other methodsSpecial: Quality standards
3. What´s new?Diverse AC vs. classical AC Standardization Frame of reference training Learning Center
4. International AC – look on our projects
Agenda
• 1st leadership level: Teamleader/ Bank Manager
• 3 different parts of the bank: Back-Office, Headquarters, Sales
• Back-Office in Duisburg: e.g. Call Center, Operations, Central Control...
• German Headquarters in Düsseldorf: e.g. HR, Marketing, Project Office...
• Branch Office, Mobile Sales Force, Investment Center
IPAD case study DC for Citibank
1. Day Observer Activity08.00 - 08.30 Introduction for the role players & observer08.30 - 09.00 Welcome!09.30 - 11.00 2x Self-Presentation (leadership understanding) 11.15 - 12.45 2x Case Study (either Back-Office or Sales or Project Management
(Headquarters)13.30 - 15.00 2x Role Play/ Employee Discussion (either Back-Office or Sales or
Headquarters)15.15 - 18.30 4x Role Play/ Team Discussion (either Back-Office or Sales or Headquarters)19.15 - 21.00 Observer Conference
2. Day Observer Activity08.30 - 11.00 2x Interview11.15 - 11.35 Evaluation of the participant‘s self-assessment11.35 - 12.00 Writing the Management Summary12.00 - 14.00 Conference14.00 - 14.30 Debriefing14.30 - 15.00 Individual Feedback for each participant
* 1 HR-Manager + 1 Executive in the observer group, 4 groups of 8 participants
IPAD case study DC for Citibank
Example # 1 - DC Citibank• specific cases & role plays for several units which represent the business
• high commitment to the DC within top management
• high social validity
• high business need to fill the jobs
• high quality of the job-specific criteria
• no psychological tests to measure cognitive skills
• high pressure to be in business, clear processes
• high frustration when the prognosis on position is longer than 6 months
IPAD case study DC for Citibank
Dr. Christof Obermann
Jarosław Chybicki