psychometrics slides

Emma Rush Consulting

Psychometrics 101

Emma Rush

Session for the Legal Education and Training Group

February 2011


Agenda

Gain a better understanding of the ‘added value’ a psychometric can bring within a legal environmentGet a clear overview of the six main instruments on the market and when they can be usedUnderstand the resource issues of psychometrics: the cost, the time required, the trained personnel neededDevelop your own case, if required, for introducing psychometrics to all or any of the ‘people processes’

Starting with meAs a consultant:

Hogan™, MBTI Step II® and OPQ32r™ for coaching and career consulting

Previous law firm:16PF® for Assessment for Development CentreMBTI Steps I & II ® for team workshopsFIRO ® for Leadership programme

FSI:16PF ® mapped on to a competency framework for partner recruitment and partnership trackFIRO® for AssertivenessDestiny for secretarial recruitment



Brief history of psychometrics

Began in late C19th with “intelligence” testingFrancis Galton, UK: Anthropometrics – taking a mathematical approach to the measure of individual differences – eugenicsCharles Spearman, USA 1910 – contributed ‘g’ or the measurement of general intelligence factorAnd “factor analysis” – the means by which psychometric tests are found to be valid


1917 Stanford-Binet test: first standardisedtest of intelligence1949: First publication of the 16PF by Raymond Cattell (Spearman’s student)1962: MBTI first published – type-based instrument; four preference scales



1963 First appearance of the “Big Five” of personality factors1970: Cronbach (Cronbach’s Alpha): emphasises the importance of validity – being able to predict accurately what a person will do


Brief history of psychometrics1984: OPQ32 introduced, published by SHL R&D by Peter Saville and Roger Holdsworth1986: Hogan’s Personality Inventory first published2004 Saville and Holdsworth ousted from SHL’s board2008 Saville’s Personality Questionnaire / Wave introduced



Key conceptsReliability: The degree to which an instrument measures the same way each time it is used under the same conditions with the same subjects – its “consistency”Validity: The extent to which you actually are measuring what you are professing to measureType vs. Trait: Type gives you a preference, trait measures how you compare to a wider population

Where to use psychometrics

Used in thousands of organisations at all stages of the employee “life cycle”

We’re looking at three: Selection; Development ; Promotion


USING PSYCHOMETRICS IN SELECTION


Why select with psychometrics?Type of intervention Validity ( r ) where 1.0 is

perfect prediction General Mental Ability Tests .51Structured Employment Interviews .51Used together .63Job Knowledge Tests .48Unstructured Employment Interviews .38Assessment Centres .37Reference checks .26Job experience (years) .18Education (years) / Interests .10Graphology .02Chance 0


Why select with psychometrics?


Why select (or train) with psychometrics?

To make the business case you first need to know:

What it currently costs (directly and indirectly) to recruit, train and retain someone so you can measure/prove an improvementExactly what you are looking for – job criteria, competency frameworkWhat outcomes you require from training interventions


In addition testing…Allows for better and clearer discrimination between candidates – “beyond the good egg factor”Supports your diversity initiatives: more transparent, fairer and no adverse impactAllows a better fit with your job criteria or competency frameworkAdministrative convenience (depending on your resources)Costs and development time are reasonable


What do you test?

Five key areas to measureKnowledgeExperience

BehavioursPersonalityCapabilityMotivation

How do they get measured?Areas on which traditional or unstructured approaches concentrate – psychometrics can still assess and assist

Areas where online tests and structured approaches can assess these areas and increase validity of recruitmentIn descending order of ease to change – motivation hardest


Intellectual Ability

Job-based or reasoning tests – used frequently at trainee level, but not at higher and more expensive levelStandard intellectual ability tests:

Verbal reasoningNumerical reasoningThinking style


PERSONALITY INSTRUMENTS IN USE


A word of warning to begin!

“..it seems that users of some questionnaires become attached to the tests that “look right” or appear to be appropriate, which is known as faith validity..” (Saville, 1975) For every argument there is a counter-argument


The Big , in in test terms

OPQ32r®

The 16PF5 ®

NEO PI-R™

Hogan Personality Inventory and Development Survey™

Saville Wave Professional Styles®


How valid are they?

Figures taken from Saville’s report How valid is your questionnaire?


High validity

Moderate validity

Chance

Instrument No of Questions Typical Completion timeOPQ32i 416 60 minutesNEO-PI-R 240 40 minutesWave Professional Styles

216 40 minutes

Hogan 206 30 minutes16PF5 185 30 minutesSaville’s Personality Questionnaire

72 13 minutes


A time comparison


Tests to avoid

Ipsative TestsTests without appropriate normsTests with low reliability estimatesOld tests that have not been re-evaluated in the last 10 years.Glossy, packaged tests with no psychometric details

And not to be used for selection

DISC: “The test suffers from questionable reliability and unknown validity”

MBTI: a “type” based instrument so without norm group validity


Costs

Using psychometrics may have a long term benefit but also has a short term cost

Per instrument – from £12 to £298 per reportTime: an hour’s minimum feedbackOnline – no additional cost to set upReputational risk


Which tests to use?OPQ32r™Pros: an “Occupational” questionnaire so about the “you at work”“r” is the latest version, and has three rather than four item choicesPublished by SHL and designed by Professor Peter SavilleCan be mapped onto a competency framework Multiple report options Relatively quick to complete


OPQ32r™

Relationships with People

InfluenceSociabilityEmpathy

Thinking StyleAnalysisCreativity and ChangeStructure

Feelings and Emotion

EmotionDynamism


OPQ32r™

Cons:32 factors – do they add anything extra? They take longer to give feedback onOverlap between factors e.g. relationships with people / feelings and emotionNow quite old (1984), doesn’t cover to same degree motivations, values, influencing style, leadershipIs it an ideal fit with professional services – norm groups used?Hard to get an overview of scores in the reports, more interpretative than detailed


16PF®

Pros: A combination of five global factors and sixteen primary factors – gives some granularity but not overwhelmingUsed world-wide for recruitment and developmentCan be mapped onto a competency frameworkFactors relate to universal traits rather than forced into three pronged modelReports give practitioner option as well as interpretative reportsRelatively quick to complete


16PF®

Cons: Meta-perceptions: candidates frequently report “I do this differently in social situations”Measures drive (obliquely) but not specifically motivationsReasoning questions can’t be relied on as measure of intellectual ability, particularly for lawyersIs it an ideal fit for professional services – norm groups used?


Hogan Personality Inventory/Development Survey/Motivations, Preferences and Values Inventory (all™)

Pros: Uniquely covers both “bright” and “dark” side of personalityCovers what drives individuals, what derails them and what they value in an organisationParticularly suited for senior recruits e.g.. lateral hires, as reports based on strengths and competencies for leaders


Hogan Personality Inventory/Development Survey/Motivations, Preferences and Values Inventory (all™)

Cons: quite American – language in reports, language in questionsThree separate questionnaires to completeRelevance to professional services?Relevance to more junior roles in professional services?Supplied in UK by smaller distributors e.g. Mentis; Psychological Consultancy


NEOThe NEO PI-R was developed by Costa and McCraeFive domains measured: Neuroticism, Extraversion, Openness to Experience, Agreeableness and Conscientiousness.


NEO

Each five domains is divided into six facets: thirty areas of personality which predict employees’ work behaviour including leadership style, team style, decision-making and stress management.240-item questionnaire either online or by paper and pencilThe current UK Edition was published in 2004.


Saville Consulting’s Wave ®Professional Styles edition

To quote Peter Saville: “Saville Consulting Wave® Professional Styles measures motives, talents, preferred culture and competency potential in one dynamic online questionnaire”A wealth of detail: 4 Clusters; 12 Sections; 36 Dimensions; 108 FacetsMore reliableQuick to completeNormed for professionals


USING PSYCHOMETRICS IN DEVELOPMENT AND PERFORMANCE



Ongoing development: workshops and coaching

Specific learning e.g. leadership, assertivenessCoachingTeam workshops

Why train with psychometrics?


The sheer cost of trainingHave to do something 200 times to be completely proficient in a new skill -accelerates the process of self-awarenessAssessing motivation means you can see in advance:

How this person likes to workWhat motivates this person to workWhat types of goals this person likes

What type of intervention?

How you like to learn e.g. Honey and Mumford’s Learning Styles QuestionnaireWhat motivates you – how do we reinforce learning e.g. Hogan MPVIWhy you need to learn e.g. FIRO-B with leadershipHow you operate in a team e.g. MBTI Step IIWhat the real issues are in coaching e.g. Saville Consulting Wave® Professional Styles


Team Interventions

Three instances you may instigate a team workshops:

Forming a teamDealing with team performanceTeam building


FIRO-BA unique way of looking at how a team operatesThree “scales”, no typology, scored from 1 to 9Inclusion; Control; Affection (or Openness)An “Expressed” score: what you show to othersA “Wanted” score: what you want from others but may not show


Team Workshops

MBTI Steps I and IIUniquely lends itself to group feedback with a reported type followed by self-assessed type and ending with best fitStep II adds factors to the four dimensions giving more granularity and better comparison between team membersCan be a powerful aide to raising self-awareness


However…

Can feel over-simple even at Step IIMBTI Step I gives only four dimensions for comparison – does this adequately explain how a team interacts or personality works?Slip between “reported type” and “best fit” can be as much as 25% e.g.. only 75% of people confirm their MBTI resultsLacks real scientific validity


BelbinDr Meredith Belbin; dates from the 1970s


Belbin’s Nine Team Roles

Plant Co-ordinator

Monitor-Evaluator Resource-Investigator

Implementer Completer-finisher

Team worker Shaper

Specialist

USING PSYCHOMETRICS IN PROMOTION


A form of selection

The same instrument should be used for promotion purposes as for selection for reasons of:

ConsistencyFairnessTransparencyProper metrics of progress (from selection to now)


How do you make the bestdecision on promotion?


The more structured methods you use the better validity you gainAssessment centres alone are less valid than when combined with e.g.

Structured interviewsPsychometrics

Questions?


SuppliersOPP:http://www.opp.eu.com/Pages/home.aspxSHL:http://www.shl.com/default.aspxHogrefe: http://www.hogrefe.co.uk/Get Feedback: http://www.getfeedback.net/Saville Consulting:http://www.savilleconsulting.com/contact/contact.aspx


http://www.opp.eu.com/Pages/home.aspx

http://www.opp.eu.com/Pages/home.aspx

http://www.shl.com/default.aspx

http://www.hogrefe.co.uk/

http://www.getfeedback.net/

http://www.savilleconsulting.com/contact/contact.aspx

http://www.savilleconsulting.com/contact/contact.aspx

Organisations

The Psychometrics Forumhttp://www.psychometricsforum.org/


http://www.psychometricsforum.org/


Training

I recommend working out which test(s) you want to use, and then qualifying with that test publisher For example OPP’s Level B course qualifying you in both MBTI® and 16PF®Or Saville Consulting qualifying you in Wave™

psychometrics slides

Technology

valid emma rush consulting

02chance0 emma rush

emma rushsession

job knowledge tests

online tests

reasoning tests

psychometric tests

development time