people assessments in industry (pai)

70
People Assessments in Industry (PAI) Technological Advances in Testing & Associated Practitioner Implications 2020 Webinar

Upload: others

Post on 13-Jun-2022

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: People Assessments in Industry (PAI)

People Assessments in Industry (PAI)

Technological Advances in Testing & Associated Practitioner Implications

2020 Webinar

Page 2: People Assessments in Industry (PAI)

Discussion Points

• About People Assessments in Industry (PAI)

• Current SA legislation and its implications

• ASSA

• Evolution of the World of Work

• Implications on Assessments

• Big Data – LinkedIn and Facebook

• Virtual Reality and Assessments

• The use of Artificial Intelligence in Assessments

• Gamification

• Robotics

• Implication for Assessment Practitioners

• Questions

Page 3: People Assessments in Industry (PAI)

About People Assessments in Industry (PAI)

Page 4: People Assessments in Industry (PAI)

PAI background

• Established in 1998 to proactively engage various stakeholders interested

in psychological assessment in South African industry.

• Functions as an interest group of the Society for Industrial and

Organisational Psychology in South Africa (SIOPSA) since 2005.

• Purpose of PAI is to ensure testing exists in the South African context as a

value adding, ethical and fair practice.

Page 5: People Assessments in Industry (PAI)

PAI acts as an independent body but seeks to build relationships and tocollaborate with such bodies as the:

• Health Professions Council of South Africa (HPCSA),• Association of Test Publishers (ATP),• Psychological Society of South Africa (PsySSA)• Assessment Centre Study Group of South Africa (ACSG),• The International Test Commission (ITC),• Business SA, test users and test takers who have similar interests with

regards to assessments in the workplace.

PAI Background

Page 6: People Assessments in Industry (PAI)

• David Bischof – (Organisational Management Technology - Evalex)• Jaco de Jager (Top Talent Solutions)• Nadene Venter (Independent Consultant)• Dr Paul Vorster (Researcher)• Kevin Distiller (Organisational Management Technology - Odyssey)• Ntswaki Raboroko (Head Talent Acquisition - Nedbank)

PAI Committee

Page 7: People Assessments in Industry (PAI)

Current SA Legislation and its implications

Page 8: People Assessments in Industry (PAI)

According to the Health Professions Act (no. 56 of 1974),instruments that measure psychological constructs must be used, interpreted and controlled by psychologists. Furthermore, only individuals registered with the Professional Board of Psychology (HPCSA) may use psychological tests.

Psychological Tests - Chapter 5, point 55 in particular states that -A psychologist shall ensure that tests used have been classified by the board and that the provisions of any applicable legislation, such as the Employment Equity Act, 1998 (Act No. 55 of 1998), have been complied with.

What does the Legislation say

Page 9: People Assessments in Industry (PAI)

Employers and/or psychologists and psychometrists need to comply with the provisions of the EEA act section 8 (a), (b) and (c) which state that the use of Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used-

✓ Has been scientifically shown to be valid and reliable;✓ Can be applied fairly to all employees✓ Is not biased against any employee or group;

The above prohibition holds true for all testing of employees irrespective if it is a Psychological test, or not, being used. All tests measuring a Psychological construct must be registered with the HPCSA and can only be used by a person registered with the HPCSA.

What does the Legislation say

Page 10: People Assessments in Industry (PAI)

• To classify any device, instrument, questionnaire, apparatus, method, technique or test aimed at theevaluation of emotional, behavioural and cognitive processes or adjustment of personality of individuals orgroups of persons, or for the determination of intellectual abilities, psychopathology, personality make-up,personality functioning, aptitude or interests by the usage and interpretation of questionnaires, testsprojections or other techniques or any apparatus, whether of SA origin or imported, and to report thereonto the Professional Board.

• The Annual publication of a list of psychological tests/psychometric instruments classified by theProfessional Board

• Develop training guidelines/standards related to psychometrics and psychological assessment that caninform and be used in the accreditation of qualifications, universities and internship programmes, whensetting the national Board examinations, and for continuing professional development purposes

• Develop guidelines for ethical practice related to test use and psychological assessment and how to assesswhether a psychological test meets the required standards

• Develop minimum requirements/standards for psychological tests• Classification will entail verifying whether a test was psychological or not. To this end, practitioners and

publishers should submit the full test manual that states the construct(s) tapped by the test, evidence ofpsychometric properties, an indication as to whether the item content was culturally appropriate. No costswould be attached to test classification by the Professional Board.

HPCSA – Revised Mandate of the Psychometrics Committee of the Professional Board for Psychology and its Implications

Page 11: People Assessments in Industry (PAI)

• Message was that the Psychometrics committee mandate was to classify tests that then goonto list;

• There is no legislative framework for HPCSA to issue certification; no mandate to certify• The reason for training guidelines is to quality assure so that practitioners are trained well

enough to evaluate if can or cannot use a test• The registration of a Psychological test with the HPCSA as published and Gazetted by the

HPCSA is not a certification of the compliance of the EEA or any other evaluative“certification”. The HPCSA is indicating that the test measures a psychological construct andas such is classified as a Psychological test that must be controlled and used by a HPCSAregistered person

• Online and Computer - Based testing - SA guidelines on computerised testing would no longerbe reviewed; HPCSA to develop a position statement instead of guidelines; positionstatement provide practitioners with board stance concerning online assessments;collaboration with all stakeholders on this position statement; position statement to guideprofession; maybe adopt international guidelines

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

Page 12: People Assessments in Industry (PAI)

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

Test classification process entails:✓ test developer submits test to the board✓ You have to be registered Psychologist/Psychometrist to submit test✓ Test manual examined to determine psych constructs✓ classified as psychological or not psychological✓ test publishers no longer obliged to inform board of tests under development or

being adapted✓Once test classified, test material shipped back to publishers due to no space

available to store tests✓ no cost for classification✓ Classification indicates test is limited to use by psychologists - guideline for the

public and practitioners✓ Test classification list to be published twice year on HPCSA website

Page 13: People Assessments in Industry (PAI)

• An independent non-regulatory external assessments evaluation ‘body’working collaboratively with the support of the Department of Labour and theHPCSA to assist in implementing a robust, best practice and technologyenabled process that could be used to review people assessment instrumentsand tests.

• It is envisaged that this body will look at the broad spectrum of instrumentsthat are used in South Africa and will not be limited to psychological tests only.In cases where an instrument meets the criteria for test classification as apsychological test, the applicants will be informed accordingly aboutsubmitting the test for classification to the HPCSA as per the statutoryrequirements.

ASSA Formation

Page 14: People Assessments in Industry (PAI)

The core purpose of Assessment Standards South Africa (ASSA) would be to review the quality of tests that are available for use in South Africa. Based on

the experience in other countries the voluntary submission of assessment instruments for objective evaluation and reviews will raise the general

standard and awareness of using quality tests.

Vision and Mission of ASSA

Page 15: People Assessments in Industry (PAI)

• The ASSA would provide a minimum standards focus in South Africa for all activities inrelation to tests and testing.

• Maintain a website and online platform to provide information for best practice, results oftest reviews and access to information about tests and testing

• Management and dissemination of information on standards relating to tests and testing.This includes information on qualifications in test use, tests reviews, test registration andthe publishing and dissemination of advisory statements, in addition to guidelines andother literature on standards for the construction, use and availability of tests.

• Quick evolution of testing – (AI, Gamification, Big Data) – field is evolving faster thanlegislation can keep pace with; ASSA can assist in staying on top of this andcollaborating locally and internationally to look at ways to maintain quality andadherence to EEA.

ASSA Activities

Page 16: People Assessments in Industry (PAI)

• This proposed procedure could greatly assist the HPCSA in that it it would ensure thattests received by the HPCSA for final classification, have already been reviewed based onobjective criteria by competent and qualified reviewers. This could ultimately provide for amore streamlined and timeous review by the HPCSA.

• Body of qualified and experienced professionals willing to assist and help from guidelinesrevision, to evaluation and management of the process

• Consideration of two possible collaborative models:• 1 – HPCSA focus on classification; ASSA focus on evaluation and certification• 2 – CPD model of outsourcing function but retaining oversight• Quick evolution of testing – use of AI, Gamification, Big Data – field is evolving faster than

legislation can keep pace with; ASSA can assist in staying on top of this and collaboratinglocally and internationally to look at ways to maintain quality and adherence to EEA.

ASSA benefits to a Collaborative Relationship with the HPCSA

Page 17: People Assessments in Industry (PAI)

ASSA activities to date

• Task force established under leadership of Deon Meiring developed test content andevaluation criteria, process and documents based on EFPA guidelines and otherinternational models

• Task force established under leadership of Hennie Kriek (including Nicola Taylor andMarie de Beer) established ASSA test review process based on successful internationalmodels. To include test submission and review process

• Sharing of ASSA test review process and criteria with HPCSA, SIOPSA, ATP a PsySSAstakeholders for comment and input

• ASSA currently designing online system to house test evaluation process.• Aim to launch November 2020 with a pilot offered to all test providers

Page 18: People Assessments in Industry (PAI)

Overview of Test Review Process

Senior editor checks facts

Consultant editor compiles reviews

Submit reviews and trainee reviewer feedback to Consultant editor

2 reviewers review & trainee reviewer allocated

Consultant editor appointed

Senior Editor content audit

Administrative check

Submit application

Page 19: People Assessments in Industry (PAI)

Overview of Test Review Process

Page 20: People Assessments in Industry (PAI)

Scope of Practice

• A high court case application launched in 2013 by Recognition of Lifelong Learning inpsychology Action Group and Justice Alliance of South Africa

• The application was an application for the review and setting aside of the Minister’s decisionto amend the scope of practice regulations

• Reasons for the court challenge included amongst others, that there was no consultationfrom the Minister’s side with the HPCSA in making such amendments. Typical requirementfrom Health Professions Act section 61(1) is that the HPCSA must be consulted beforeamendments are made

• Challenge was more on the process and procedure followed

Page 21: People Assessments in Industry (PAI)

• The amendment regulations as published on 2 September 2011 be declared invalid• The Minister’s decision to promulgate the amendment regulations be remitted for

reconsideration• The invalidity order of the relations amendment suspended for a period of 24 months• Post the court case ruling, a working group was formed which was mandated to:

✓ Review the judgement of the court case✓ Review the relations defining the scope of the Professions & Annexures relating to the

scope✓ Consult stakeholders✓ Develop a revised scope of practice of the profession for psychology for promulgation

Scope of Practice

Page 22: People Assessments in Industry (PAI)

• The working group concluded their assessment and submitted a recommendation withre-drafted regulations on the scope of practice

• The revised regulations were submitted to the Board on 23 February 2018 and approved• The council approved these amendments in March 2018• Revised regulations were out for public comment

September 2019 – HPCSA decision not to proceed with proposed regulations

Where are we now: Scope of Practice?

Page 23: People Assessments in Industry (PAI)

Where are we now: Scope of Practice?

Page 24: People Assessments in Industry (PAI)

Evolution of the World of Work

Page 25: People Assessments in Industry (PAI)

Michael Smith (Testing People at Work, 2005)

Page 26: People Assessments in Industry (PAI)

1970s 1980s 2000s1990s Today2010s

Technology

Individuals

Businesses

Public policy

Rate

of

change

The gap between Technology and Public Policy - 2017 Deloitte Global Human Capital Trends

Page 27: People Assessments in Industry (PAI)

The Gap is widening

Page 28: People Assessments in Industry (PAI)

The top 10 skills that will be in demand by all employers by 2020 - World Economic Forum - 2016 report

Position Competence Questionnaires Ability / Cognitive Simulations

10 Cognitive flexibility No Yes Somewhat

9 Negotiation skills Propensity to No Yes

8 Service orientation Yes No Yes

7 Judgment No Yes Yes

6 Emotional intelligence Yes No Somewhat

5 Coordinating Propensity to No Yes

4 People management Propensity to No Yes

3 Creativity Propensity to Somewhat Yes

2 Critical thinking No Yes Yes

1 Complex problem solving No Somewhat Yes

Page 29: People Assessments in Industry (PAI)

World Economic Forum – Future of Jobs Report 2018

Page 30: People Assessments in Industry (PAI)

World Economic Forum – Jobs of Tomorrow Mapping Opportunity in the New Economy –January 2020

• Successful professions in future will increasingly require both “digital” and “human” competence and emergingprofessions reflect the continuing importance of human interaction in the new economy.

• There are seven emerging professional clusters and 96 jobs of tomorrow within them that vary in theirindividual rate of growth and in the scale of job opportunities they offer in the aggregate. These clusters are:

✓ Data and AI✓ Engineering and Cloud Computing

✓ People and Culture✓ Product Development

✓ Sales, Marketing and Content✓ Care Economy✓ Green Economy

• The highest-demand skills required in these emerging professional clusters span both technical and cross-functional skills. These in-demand skills can be divided into five distinct skills clusters:

✓ Business Skills✓ Specialized Industry Skills✓ General and Soft Skills✓ Tech Baseline Skills✓ Tech Disruptive Skills

Page 31: People Assessments in Industry (PAI)

Implications on Assessments

Page 32: People Assessments in Industry (PAI)

Seismic shifts in the Human Capital Talent Landscape

Page 33: People Assessments in Industry (PAI)

Everyone it telling me to do a coding course!!!

• Better question is around how to apply relevant IT skills - which could bearound developing new apps or technology in the human capital space. Orto better analyse data and design easy to navigate dashboards etc to makedata-driven HC decisions.

• Bigger need is to be able to support the psychology side of newtechnologies being implemented/designed in organisations. Important skillshere are Data analytics / Agile methodology/ Design thinking/ Leanprototyping etc

• Coding courses around Python or Java (LinkedIn learning and Udemy)

Where to focus as a practitioner?

Page 34: People Assessments in Industry (PAI)

2017 - Robert McHenry - Psychometric Testing: Critical Perspectives – ‘The Future of Psychometric Testing’

• Prediction #1: Smartphones will replace computers for employee assessment• Prediction #2: High-quality psychometric testing services will be sold direct to

consumers• Prediction #3: Advances in neuroscience of personality will reveal which are the

most valid individual differences to measure and how best to measure them• Prediction #4: The digital movement, coupled with the use of big data and new

forms of digital CV will render many of the current applications for high-stakespsychometric testing redundant

• Prediction #5: The basis for employee development will be derived from thedata yielded by wearable devices and not from psychometric tests

Page 35: People Assessments in Industry (PAI)

• The testing industry - in ‘ideational stagnation’, change is slow and with minor cosmeticchanges, only the numbers in the names of tests substantially change

• Face to face, paper and pencil a thing of the past - Covid-19 has pushed everything online• Next Generation Assessments - different in design, look, feel, scoring and administration• Innovation in technology - online delivery through mobile phone and tablets• Big Data; Artificial intelligence; machine learning; behavioural and predictive modelling

(Computational Modelling and Psychology); gamification; bio screening; virtual reality;robotics; auto transcribing; language modelling and natural language processing;personality assessment via behaviour-linguistic analysis and social network data(Facebook; LinkedIn and Twitter).

• Completely autonomous scoring and reporting generation is the new kid on the block• Video interviews where AI software is used to analyse the candidate’s facial expression,

tone and language to ascertain job fit• Resume filters scan for keywords and phrases on resumes• Chatbots schedule interviews and algorithms help predict a job candidate’s future success

The next generation of Assessments

Page 36: People Assessments in Industry (PAI)

The rise of Digital Psychometrics

Page 37: People Assessments in Industry (PAI)
Page 38: People Assessments in Industry (PAI)

Big Data - Facebook

Page 39: People Assessments in Industry (PAI)

Big Data - Facebook

Page 40: People Assessments in Industry (PAI)

• LinkedIn used as ‘screening tool’ by hiring managers• Provides for Skills Assessments: short, multiple-choice tests that users can take to verify

their knowledge in areas like computer languages, software packages and other work-related skills.

• Research conducted by Julia Levashina, Ph.D., associate professor of management at KentState University, along with her colleague Nicolas Roulin, associate professor ofindustrial/organizational psychology at Saint Mary's University in Canada

• LinkedIn profiles could be used to make inferences about job applicants’ visible skills (suchas leadership, communication and planning), personality traits (such as extraversion) andcognitive ability. However, hiring managers are not recommended to use LinkedIn profiles toassess less visible skills (such as conflict management or adaptability) and personality traits(such as conscientiousness and emotional stability).

• The researchers also demonstrate that hiring recommendations based on LinkedIn profilesare influenced by the job applicant’s LinkedIn profile length, photo and number ofconnections. It was found that other features of LinkedIn profiles (such as listing more skills,collecting endorsements, joining groups or describing volunteering involvements) had anegligible impact on hiring recommendations.

Big Data - LinkedIn

Page 41: People Assessments in Industry (PAI)

Virtual Reality and Assessments

Page 42: People Assessments in Industry (PAI)

• Virtual and augmented reality (VR and AR) use biometric tracking data - micro-movements of head, torso, hands, and eyes - on the job training

• Can diagnose or predict anxiety, depression, schizophrenia, addiction, ADHD, autism spectrumdisorder and more about a person’s cognitive and physical function.

• What if this medically-relevant data is fed into users’ psychometric profiles?• Profiles may start out as relatively harmless, merely predicting when someone might be getting

ready to buy a new car.• However, sprawling psychographic profiles with medical inputs could leave people vulnerable• Anonymizing VR and AR tracking data is nearly impossible because individuals have unique

patterns of movement.• Using gaze, head direction, hand position, height, and other behavioural and biological

characteristics collected in VR headsets, researchers have personally identified users with 8 to12 times better accuracy than chance.

• Dirk Geldenhuys ACSG 2020 – Use VR to develop resilience in the VUCA world?• Need for stronger regulation

Virtual Reality and Assessments

Page 43: People Assessments in Industry (PAI)

• Intelligent software has predominantly been used to automate screening • Chatbots used as a part of the screening process• Robotics - Call center in the UK needed to hire representatives to respond to

customers who speak seven different languages; AI-enabled test where candidates were “interviewed” by a robot. The software evaluated their speech for syntax and grammar, and the company made offers based on the results

• Vera - a lifelike Russian robot with impressive conversational skills used to phone and video interview candidates for positions

• Socially capable robots can provide many advantages to diagnostic practice: engage people, guarantee standardized administration and assessor neutrality, perform automatic recording of subject behaviours for further analysis by practitioners.

Robotics and Assessments

Page 44: People Assessments in Industry (PAI)

Robotics and Assessments

Page 45: People Assessments in Industry (PAI)

Robotics and Assessments

Page 46: People Assessments in Industry (PAI)

Artificial Intelligence involves the

building of artificial neurological

networks and is underpinned by a

multitude of statistical algorithms to

emulate the human mind

Artificial Intelligence

Page 47: People Assessments in Industry (PAI)

https://digitalwellbeing.org/artificial-intelligence-timeline-infographic-from-eliza-to-tay-and-beyond/

Page 48: People Assessments in Industry (PAI)

• Robotic process automation: This is achieved by gathering and transferring expert knowledge andthen programming the system with an 'if/then' rule-based approach. Chatbots are a great example.However, this rule-based system is not capable of learning and improving without being givenexplicit instructions. In talent assessment, computer-generated interpretative reports make use ofthis technology.

• Machine learning: Even though a computer cannot think for itself (not yet anyway), statistical toolscan enable a system to model predictions from any given data – and to add to the model to improveits predictions over time. This is used in data analysis to create predictive people analytics as ameans to help employers make better talent decisions.

Pattern matching: This AI technique uses a computer to check the sequence of responses todetermine if there is a pattern. It can be used to carry out some 'human' tasks, such as recognizingfaces or identifying emotions.

• Natural language processing: Makes use of text and speech analytics to extract the underlyingmeaning. This can be applied to analyse speech in interview question responses

Artificial Intelligence and Assessments

Page 49: People Assessments in Industry (PAI)

Artificial Intelligence and Assessments

• Filip Lievens, (Professor from the Department of Personnel Management, Work and Organizational Psychology at Ghent University) - EASE tool (Electronic Assessment Simulation Exercise) -all participants start with one e-mail. They can then obtain further information by organizing meetings, requesting archive documents or sending e-mails to colleagues. The participants need to ask the right people the right questions in the right way. The computer then generates the requested information, and it also functions as an assessor that evaluates and records all actions objectively.

• http://www.fmaconferences.org/Vietnam/Papers/Hrazdil%20et%20al.%20FMA%202019.pdf (2018) study on utilising IBM Watson Personality Insights service to measure executive personalities (big 5) based on CEO’s responses to questions raised by analysts during conference calls.

• Video interviewing – Audio transcribed and analysed for clarity of speech and language proficiency. AI used to analyse the visual elements through emotion tracking software and facial recognition

• Assessment systems designed to integrate constructs from across psychometric instruments to assist with the development of psychometric competency frameworks, the identification of inhibitors and job position fit

Page 50: People Assessments in Industry (PAI)

Benefits: • Improvement of assessment process flows • Capturing of real-time data as it happens • Equality - a system cannot identify race, gender or age.• Engagement: the candidate finds these types of assessments modern, engaging and novel.• Depth of Information: a tapestry of information collection; not possible by even an

experienced assessor.• If systems are stable then large volumes can be handled• Software algorithms produce accurate and error-free calculations• An AI neurological network can produce a result without any human intervention• AI can analyse vast amounts of data accurately and error-free. This assists with faster decision

making• Elimination of human conscious and unconscious bias BUT an algorithm is only as good as the

data that’s fed into it.

Artificial Intelligence and Assessments

Page 51: People Assessments in Industry (PAI)

Its all in the Algorithm!!

• In 2018, Amazon had to shut down an AI recruiting tool designed to screen applicantresumes because it was discriminating against women.

• Another 2018 study on gender and race discrimination by machine-learning algorithmsconducted by MIT and Microsoft researchers found that prominent facial recognitionsoftware had higher error rates when presented with pictures of women. Error rates wereeven higher when the subject was a woman with darker skin. Similarly, there are numeroususe cases — such as voice and speech recognition — where AI applications have had worseperformance for women.

• AI software to eliminate the prejudice of human hiring managers has producedencouraging early results at corporations.

• But tech executives with experience at Google, Microsoft and Facebook say the algorithmicrevolution in hiring is moving too fast.

• Algorithm auditing firms want to see the code; public policy experts want to pressgovernments to force the algorithms into the open before it’s too late.

Artificial Intelligence and Assessments

Page 52: People Assessments in Industry (PAI)

Concerns:• Companies exploit public confusion slap the “AI” label on whatever they are selling ‘ (How to

recognise AI snake oil - Arvind Narayanan https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf)

• Ethical concerns regarding accuracy of areas such as content identification, facial recognition, medical diagnosis from scans, speech to text and ‘deepfakes’.

• Defensibility. How do you defend the outcomes of standardised 'plug-and-play' AI systems which are not aligned with specific job requirements and lead to custom decisions?

• Ethics. There is an ethical question around how much support to take from an AI system for decision making. i.e. allow AI to reject outright or flag to check?

• Do and can organisations provide proof that AI predicts performance? • Data handling. AI excels at analysing massive amounts of data, but the results can be

misinterpreted or even deliberately abused. • Video interviewing – applicability – can you really identify emotions from facial recognition?

Generalisability of facial expressions across racial and cultural groups; If you look anxious are you really anxious?

Artificial Intelligence and Assessments

Page 53: People Assessments in Industry (PAI)

Gamification

• Instead of developing a full game, gaming elements are layered over an existing program or context in order to amplify users’ motivation to engage within that context (Kapp, 2014)

• Designed to motivate users by providing feedback, recognition, status and the potential for competition among users (Muntean, 2011). -

https://www.youtube.com/watch?v=VwGKdKTNYxM

Page 54: People Assessments in Industry (PAI)

Gamification

Page 55: People Assessments in Industry (PAI)

Gamification

Page 56: People Assessments in Industry (PAI)

Advantages of Gamification

Page 57: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 58: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 59: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 60: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 61: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 62: People Assessments in Industry (PAI)

Inappropriate Eurocentric Gamification

Page 63: People Assessments in Industry (PAI)

Challenges of Gamification

• The time involved in truly customising and validating a game,

• Ongoing investment as the life cycle of the games may be short,

• Some people just don’t enjoy games,

• Games are only fun in low-stake environments,

• Candidates may feel belittled / not taken seriously,

• If poorly designed and irrelevant to the construct being assessed, mechanics introduce error and noise into assessments,

• The game might lack face validity and proving psychometric properties difficult.

• If a game is developed in a Eurocentric environment, gaming mechanics, skin, storyline and challenges may not be culturally relevant to SA candidates.

• Beware high graphics and data requirements in developing environments.

Page 64: People Assessments in Industry (PAI)

Challenging our Testing Paradigms

Page 65: People Assessments in Industry (PAI)

• EFPA Model - Main goal to provide a description and a detailed and rigorous assessment of psychological assessment, tests, scales and questionnaires used in the fields of Work, Education, Health, and other contexts – 37 European countries and other countries who have adopted the model

• Barrett - Examines the applicability and future relevance of the EFPA test review guidelines• Barrett - ‘EFBA relies upon outdated thinking and assumption laden test theory and

statistics that are neither wise, nor justifiable any longer’.• Barrett - The EFPA guidelines on reliability and validity in sections 10 are not current best

practice at all, but merely repeat prescriptions from a time when psychologists knew no better

• Barrett – proposed a Next Generation Pragmatic Review Framework with different measurement assumptions and ideas on validity and reliability

Challenging our Testing Paradigms

Page 66: People Assessments in Industry (PAI)

Implications for Practitioners

Page 67: People Assessments in Industry (PAI)

Implications for Practitioners

• In South Africa psychologists and psychometrists still need to comply with the provisions of the EEA act section 8 (a), (b) and (c)

• The above prohibition holds true for all testing of employees irrespective if it is a Psychological test or not being used.

• All tests measuring a Psychological construct must be ‘classified’ with the HPCSA and can only be used by a registered person with the HPCSA

• ASSA are looking at a Test Criteria Evaluation process and partnership with the HPCSA in this regard

• Local (and global) Legislation and regulations, research and classification/certification is lagging in terms of the next generation of assessments which is already here

• Innovations in technology are allowing for easier delivery of assessments as well autonomous scoring and reporting to improve assessment efficiency and accuracy

• The use of Big Data, Machine Learning, Artificial Intelligence, VR, Robotics and Gamification and are leading the way in the assessment and talent space

Page 68: People Assessments in Industry (PAI)

• Although proving to be more effective, efficient and accurate in nature, we need to be careful of how these assessments are constructed, where they can be effectively used and what the implications are for the South African environment.

• Research into validity and reliability on digital assessments should be a combination of theory- and data-driven models and more research is required.

• AI’s role should be restricted to providing additional information and enhancing efficiency. AI can deliver useful information, at various stages of the selection process, but should be used to support, and not make the final decision.

• We need to be careful as Algorithms don’t have an ethical compass. They may be based on undesired societal biases. Rubbish in, rubbish out!

• Data-driven models and final decision-making based on assessment results should remain under control of humans - leverage technology and assessments learnings to enable better decisions

• Role of practitioners will change – more about job analysis and algorithm design and ensuring ethical and scientific use of technological enabled assessments.

Implications for Practitioners

Page 69: People Assessments in Industry (PAI)

https://www.psychologytoday.com/za/blog/the-situation-lab/201910/beware-these-marketing-trends-in-psychological-assessment

Ryne A. Sherman, Ph.D., - Chief Science Officer at Hogan Assessment Systems –

‘Many of the new assessment firms use flashy technology and claim new insights into workplace performance. Hiring managers and HR professionals

need to be wary of companies using these common marketing trends.

Only two things matter in psychological assessment: fairness and predicting performance. Companies that emphasize neuroscience, big data, and

gamification may be trying to distract you from the fact that their assessments don’t predict workplace performance’.

Understand what is inside the black box!

Implications for Practitioners

Page 70: People Assessments in Industry (PAI)

David BischofChairperson: People Assessments

in Industry (PAI)

Director – Research Psychologist &

Psychometrist

OMT | Evalex

Email: [email protected]: 073 254 5119

Thank You!