psychometric testing in credit risk assessment · psychometric testing in credit risk assessment...

Psychometric testing in credit risk assessment

A digital readiness primer and case study

@eflglobal // [email protected]

Agenda

EFL Intro Where EFL works best Digital readiness primer HCD: Designing for the feature phone Equity Bank case study Q&A

Traditional credit scoring is blind to billions of people 3 billion are unscorable, another 3 billion will enter the middle class in the next 20 years

Covered by credit

bureau, 30%

Not covered,

70%

Not covered •  Billions of good applicants rejected •  Enormous missed opportunity

70%

Source: http://www.ey.com/gl/en/issues/driving-growth/middle-class-growth-in-emerging-markets; http://data.worldbank.org/indicator/IC.CRD.PRVT.ZS

Copyright © 2017. All Rights Reserved. 3

We are entering a new era of lending

1990’s

Loan officers with deep

relationships

Loan officers +

Traditional credit scoring

2000’s Today

Digital approvals without loan officers

+ Traditional and

alternative scoring

2010’s

Loan officers +

Traditional and alternative scoring

And credit scoring needs to adapt to serve more people with end to end digital processes

Predictive power and availability vary by data source Psychometric is only data source with 100% availability – everyone has a personality

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Predictive Power (Out of sample gini coefficient with comparable model & sample)

Less

Ava

ilabl

e

Mor

e Av

aila

ble

PSYCHOMETRIC DATA

MOBILE DATA

BUREAU DATA ONLINE DATA

Less Predictive More Predictive

Avai

labi

lity

(Hit

rate

on

sam

ple

of lo

w-in

com

e ap

plic

ants

in e

mer

ging

mar

kets

)


EFL is the proven universal score for today’s lenders

•  4 years R&D at Harvard on psychometric, demographic, social & CDR scoring

•  6 years implementing scores with lenders across Africa, Latin America & Asia

Alternative scoring leader •  Score anyone, anywhere

with our psychometric and behavioral science based assessment

•  No client history or 3rd party database needed

Universal coverage •  Published 3rd party

validations by World Bank and others

•  Won multiple awards including G-20 SME Finance Challenge, African Business Award for Innovation, and Mahindra Finance Challenge

Third party vetted

Link

Empowering digital and in-person approvals across $1.5B+ and counting in disbursed loans


EFL assesses risk with a short online loan application

Unscorable Applicant Takes EFL assessment on the web, mobile or SMS

EFL Algorithms Analyze the answers and

determine risk level Score Generation Create 3-digit EFL Score and

Score Confidence rating

Continuous Learning EFL receives monthly loan

repayment used to constantly customize and improve

models

1 3

2

4

1

Attitude

Control

Entrepreneurial potential

Interaction metadata

Social Behavior

How we score anyone The character traits our assessment measures


Agenda



EFL works best when there’s a clear need for an alternative credit score, and the FI is challenged by:

Low Acceptance Rates (due to lack of data)

High rejection rates due to lack of data – too

many grey zone or thin file clients

High Levels of Default

Default rates over target; hesitant to dramatically decrease acceptance

rate

Costly & Lengthy Loan Application Processes

Loan application process takes time and is resource-

intensive for lender

1. 2. 3.


Use the EFL score to… Our partners typically use our behavioral psychometric score in one of three ways:

Increase Your Acceptance Rate

Approve more people, without taking on more

risk

Decrease Your Default Rate

Decrease and control your default rate, with limited volume impact

Increase Your Operational Efficiency; Lower Application Cost

Fast track: Cut pre-filters

for high-scoring applications

1. 2. 3.

‘Institutional Readiness’ plays a large role in success

Ability to receive a 3-digit, numerical, credit score and use it in their existing credit decision making

process.

Receiving thousands of credit applications each month.

Extra Plus? Ability to combine EFL

score with other credit scores via scorecard or ensemble model

Strong desire across management & teams to

implement and scale solution quickly.

Motivated team, with

willingness and motivation to change existing process

Willingness to try new solutions and consider a departure from

traditional processes.

Willingness to take calculated risks and move in digital direction.

High Volume, Credit Score Ready Driven to Push Innovation Mindset


We’ve noticed certain traits within organizations that allow the EFL digital credit assessment to thrive

Agenda



Financial institutions are going digital globally Digital technologies can cut the cost of providing financial services by 80 to 90% and improve scalability

Annual cost to serve one customer in emerging economies, 2014 ($)

Traditional bank branch

Digital

Cost savings due to digitization

Accounts Cash-in, cash-out Transactions Total

Source: McKinsey Global Payments Map; “Fighting poverty, profitably: Transforming the economics of payments to build sustainable, inclusive financial systems,” Bill and Melinda Gates Foundation, September 2013; McKinsey Global Institute analysis

Copyright © 2017. All Rights Reserved.

Going digital? It’s more than building an app What does it mean for a bank to be “digital”?

15

Affordability Risk KYC / ID Servicing Acquisition Online acquisition is

possible, though usually pricey.

Web scrappers are emerging to support digital affordability, however, limited coverage and

inconsistent data remain an issue

Multiple companies, including EFL are proving that risk

profiles can be built via digital

interactions.

Proving ID online is an emerging field,

however, in EM limited databases

coverage is a factor.

Few companies are focused on are looking to outsource collections.

Being digital means being scalable. If you have to add staff to increase lending, you aren’t digital.

But there are challenges to going digital Moving the credit application process from in person with a loan officer to online comes with its own risks

Increased Risk Digital applications can come with increased risk and fraud. Effective credit scoring is key to keeping a healthy portfolio.

Lose Personal Touch No human interaction can dampen the bank-to-customer relationship. Tools are needed to increase human designed interaction.

EFL is your digital loan officer. Efficiently assess risk on anyone, anywhere, online. Build trust with applicants.



Going digital is a process

No existing digital processes

Hybrid model – some parts digitized

Fully digital process; pushing for scale

Add EFL to partially digital process flow, see results in: •  Operational efficiency

Use EFL as a stand alone tool, focused on: •  Customer Acquisition •  Reject Recovery

Integrate EFL into a digital application: •  Control risk while

maintaining efficiency.

•  Increase acceptance rates. %

DIG

ITA

L

100%

0%

We work with partners at each stage

Agenda



Product Design: Understand Your “Bones”

19

What framework are you working within?

•  SMS = a necessarily text-heavy product •  Question length limited to 160 chars –containing both question and answer key •  Need to battle potential mistrust of the product – concerns of SMS scams •  Open answer responses – need to validate and send appropriate responses

Understand your customers: •  Design for the mass-market, and try to incorporate the low-common denominator as

much as possible. People will struggle with:

Technology Literacy Understanding Connectivity

Know What You NEED

Honesty

Intelligence

Attitudes

Stability

General Behavior

Fraud

Hypothesize: Why does someone

default?

•  Relatively easy to understand (both the question and desired answer format)

•  Predictive even within question

number and character limits •  Forgiving if users make mistakes

and need to correct answers •  Trust building – Reliable product

that is connected to bank and reward believed

Product design PROCESS

1. Field observation & interviews: on

both question framing and content

2. Experimental design: validate/quantify hypotheses from field via more real-life

channel using flow split-testing and within-flow experiments

3. Data analysis of flows and field

interviews: ensure which key metrics are improving and

which aren’t

Watch! How and when are people interacting with their phones?

Get to know your competition – what mobile services are people

using? What do they like / dislike about them?

Build: Fast and easy prototypes Paper print outs of phones with text questions

Talk with people- get them to use your product, explain as little as possible.


Get OUT! 1

“Hey! Can I ask your opinion?”

“What do you think it means?”

START


2 Create the RIGHT data Test many versions of your product, over wide audiences, through various channels

Output: Well-structured quantitative and qualitative data to analyze

CHANNEL EXPERIMENTAL DESIGN

Parallel Flows

Distance Testing

In Person

Single Flows

Test two similar products at the same time, with different versions of the same content you need to test – isolates the “treatment effects”

Intersperse new iterations of product parts within your current product iteration to see if behavior changes – controls for individual differences


3 Explore Your DATA Define metrics for what makes the product “good” – explore the data to see what's working…

•  Answer distribution (extreme values vs. well-distributed •  Answer timing (confusing questions take longer) •  Drop off rates (do certain questions cause people to disengage?) •  % of unusable answers

0%

5%

10%

15%

20%

25%

30%

35%

10-point scale Percent scale

Percent of responses with extreme answers or unusable answers 10 Point Scale

Reply with a NUMBER ONLY on a scale from 0 - 10 (0=never 5=sometimes 10=always). Do you show up late or miss appointments?

Percentages What PERCENT (%) of the time do you show up late or miss appointments?

BEF

OR

E A

FTER

…Do we see an improvement in performance metrics as a result of a new iteration?


Final SMS Content: Three Targeted Core Traits

Locus of Control

Opportunism/ Progress mindset

Integrity

Understanding the user’s concept of

honesty and fairness in the context of her

financial life

Understanding the user’s sense of

personal responsibility for and control over

her life

Understanding the user’s perspective on

opportunities and motivations to take risks

to improve her life

Objective: Identify a few core traits that are most relevant to the population. Focus a majority of psychometric content around these to max predictive power.

10%

15%

20%

25%

Locus of Control, Integrity,

Opportunism

Confidence, Optimism,

GSE

% of responses with no valid answer


Maximizing predictive power 1)  3 core traits focused on 2)  Granular answer potential (%s)

… through ensuring understandability 1)  Answer in percentages 2)  Non-hypotheticals – concrete questions / “mini stories” 3)  Data quality check – smartphone? Help on survey?

… and building trust 1)  Human approach – make people feel comfortable 2)  Start with easy, but non-invasive questions (back load more intrusive

demographic questions) 3)  Trust this product – start with the person’s name

Reply with a NUMBER in KSH. If you found 100,000 KSH today, how much would you spend on leisure?

Final SMS Product

Agenda


Copyright © 2017 EFL. All Rights Reserved. 28

Overview - Scoring on the feature phone Equity extends credit through chat, proves ability to boost acceptance rates

The Challenge High volume, feature phone mobile lending comes with significant risk. Equity needed an innovative solution to accept more of their customers without taking on additional risk. The Engagement EFL designed a new product – taking the learnings of our traditional psychometric content, and transforming them into a 22 SMS question assessment. Instead of being flat-out rejected, low data customers could take the assessment remotely on their feature phones, and gain an opportunity to be approved.

The Results Collecting 24k completed SMS tests, EFL built a custom SMS model, and saw significant predictive power in the data (psychometric & meta). The strong model results indicate potential SMS uses across markets / channels.

Tine, an Equity customer, applies for a loan via her Equitel SIM card.

She’s new to the bank, so she receives the first EFL SMS asking her to start the test.

Tine’s survey is analyzed, and her credit score sent back to Equity in real time. Tine is disbursed!

How does it work?


Designing the SMS Pilot – Process & Criteria

29

…TO VALIDATE TWO MAIN HYPOTHESIS

Build & Integrate SMS Platform

Design SMS Test 1

2 Run Pilot;

Gather Results 3 Build Custom

Model 4

HYPOTHESIS MEASUREMENT DESIGN

1. The SMS channel will facilitate a high

conversion rate. UX of SMS

Test

Test Completion Rate (%s in

funnel)

2. SMS psychometric testing will enable Equity

to better segment and predict risk

Psychometric Assessment

Predictive Power of SMS

Model

Out-of-Sample Model

Validation 5

FIVE STEPS UNDERTAKEN TO COMPLETE THE SMS PILOT… Equity & EFL partnered in 2013 to develop a custom

feature phone credit scoring tool

SMS – USSD ~20 psychometric questions

4 test sections Any phone, any location Only need to score once

Whatpercent(%)ofbusinessescheattheir

clients?

Derives predictive power from: 1.  Psychometric answers 2.  Timing data 3.  Free-text input data 4.  Self-reported demographic data

Assess willingness to repay through a conversation Through EFL’s chatbot platform, we converse with your clients, one SMS at a time. Through an automated conversation, we ask our traditional psychometric questions – straight to their phone. Predictive, real-time, credit scores available in minutes.


© 2016, EFL Global. All rights reserved. 31

Benson,anewcustomer,appliesforloan.Normally,hewouldberejected.

Instead,heimmidietelyreceivesaninia9onSMSmaskedfromthe

lender

Bensonengages,conversa9onstyle,inthe

SMStest.

CREDITSCORE337

CreditScorereturnedtolender

PsychometricsBehavioral – Text Analysis; Timer Analysis

What time of day are they responding?What are response answer patterns?

Do they understand the question?How is their spelling? Grammar?

What words or phrases do they use?

Seamlessly assess an applicant’s character PR

ODU

CT F

LOW


EFL SMS test seamlessly fits in with existing UX mobile lending product flow Before rejecting low-data and new applicants, Equity can digitally score and asses risk on borrowers.

32

Benson,acustomerofEquityBank,

appliesforloanviaEquitelSIMSDK/

USSDmenu-

Immidietelyreceivesinia9on

SMSfromdedicatedEquity

shortcode-

BensonrepliestotheSMS(zero-

rated)andbeginsthetest

Bensonengages,conversa9onstyle,intheSMStest.

-Answerstestin4secPons.SMSreminderssenttokeepBenson

focused.

Totaltest)me<15minutes,takenatonceoroverhours/

days

CREDITSCORE337

CreditScore-

Returnedinreal-PmethroughAPIto

Equity’sCU


Results: Predictive Model; Strong Usability

33

1 Worst 2 3 4 5 Best

Def

ault

Rat

e

EFL Score Quartiles

bad45_m3 bad30_m3 bad60_m3

Customized Model With 24k tests completed, EFL built a customized model for SMS – achieving a predictive power of 63.7% AUC

Usability validated through low drop off rates.

Average completion rate of 78%, with an average test time of 33 minutes.

80% of drop-off occurred within first 4 questions. 56% of tests were completed within 15 minutes.

Out of Sample AUC: 63.7

Low Default

High Default

1 Worst Scoring

4 Best Scoring

3

2

Def

ault

Rat

e


Ability to increase acceptance rates by 20% when combined with internal behavioral score

EFL & Equity’s Internal Behavioral Score

34

Combining the EFL score with an internal behavioral (transactional) score: boost acceptance rate by 20% without taking on additional risk

Ensemble Experian EFL

CumulatedDefaultRates

20%

Def

ault

Rat

e

Acceptance Rate

1x 1.2x

Low Default

High Default

High Low

Behavioral


EFL finds different bads, and different goods, than traditional credit bureau scores

35

EFL & Traditional Bureau Score Combining the EFL score with a traditional bureau score: boost acceptance rate by 53% without taking on additional risk

Ensemble Bureu EFL

Cumulated Default Rates

Def

ault

Rat

e

Acceptance Rate

Low Default

High Default

High Acceptance

Low Acceptance

Bureau

53%

Agenda


psychometric testing in credit risk assessment · psychometric testing in credit risk assessment...

Documents