validation analysis results

Fair Isaac Proprietary Information Page i

Validation Analysis Results

Small Business Administration

Small Business Scoring Service (SBSS)

Version 6.0 - Pooled

January 2015

Validation Analysis

Fair Isaac Proprietary Information Page ii

This is the confidential, unpublished property of Fair Isaac Corporation. Receipt or possession

of it does not convey rights to divulge, reproduce, use, or allow others to use it without the

specific written authorization of Fair Isaac and use must conform strictly to the license

agreement between user and Fair Isaac.

© Fair Isaac Corporation - All Rights Reserved.

Fair Isaac Proprietary Information Page iii

TABLE OF CONTENTS

EXECUTIVE SUMMARY ......................................................................................................................................... 1

VALIDATION DESIGN ............................................................................................................................................. 2

PORTFOLIOS............................................................................................................................................................... 2

OBSERVATION AND PERFORMANCE DATES ............................................................................................................... 2

Observation Dates: July 2012 to November 2014 .............................................................................................. 2

Performance Period: 18 Month Window ............................................................................................................ 2

Performance Date: December 2014 .................................................................................................................... 2

DATA EXCLUSIONS .................................................................................................................................................... 3

PERFORMANCE DEFINITIONS ..................................................................................................................................... 3

POPULATION OVERVIEW ..................................................................................................................................... 4

POPULATION STATISTICS ........................................................................................................................................... 4

SUMMARY COUNTS ................................................................................................................................................... 4

DETAILED COUNTS .................................................................................................................................................... 5

SCOREABLE COUNTS ................................................................................................................................................. 7

DESCRIPTIVE STATISTICS .......................................................................................................................................... 9

Portfolio Group 1 – Performance Category by Score Distributions .................................................................... 9

New versus Existing Business ............................................................................................................................ 11

Loan Size ............................................................................................................................................................ 12

Delivery Method ................................................................................................................................................. 13

SCORE DISTRIBUTIONS AND GRAPHS ............................................................................................................ 14

SCORE DISTRIBUTION .............................................................................................................................................. 14

Fair Isaac Proprietary Information Page 1

Executive Summary

This validation analysis has been completed for SBA to provide information on the strength of

the SBSS 6.0 score - Term Loan/Line of Credit models. Minimal data availability has limited

the extent of the analysis. Based on the score reports, FICO recommends that the SBA continue

to use the SBSS 6.0 score for their portfolio.

This document contains the final project deliverables, including:

a. Validation Design

a. Portfolios

b. Observation and Performance Dates

c. Exclusions

d. Performance Definitions

b. Population Overview

a. Population Statistics

b. Summary Counts

c. Detail Counts

c. Score Reports

a. Summary Score Statistics

b. Trade-Off Curves

c. Score Distributions

d. Cumulative Score Distributions


Validation Design

The SBA portfolio has been validated based on SBSS 6.0 Term Loan/Line of Credit models.

The following validation design decisions have been made at the on-set of the project.

Portfolios

Observation and Performance Dates

Data Exclusions

Performance Definitions

Portfolios FICO evaluated the following portfolio groups and segments on the scores noted in the table.

Recommendations have been made at the portfolio level.

Portfolio Scores Evaluated

Portfolio Group 1 Overall Portfolio SBSS 6.0 Term Loan/Line of Credit

Observation and Performance Dates For validations of scores, FICO uses two snapshots of data: one from the observation date and

one from the performance date.

Observation Dates: July 2012 to November 2014

A sample of applications was taken from the time period July 2012 to November 2014. The data

as of the observation date captures the SBSS 6 score as of the application date, and other key

metrics about the small business. Typically in a validation of origination score, both approved

and declined application data is available. For this effort, only approved application data was

provided.

Performance Period: 18 Month Window

This is the forecast period over which the scores predict account risk. The performance period

begins the month after the observation date. To measure non-event/event behavior, the statuses

over the entire window has been reviewed.

Performance Date: December 2014

This date marks the last month of the performance period. Using account status information over

the performance period, an account is assigned to a performance category for modeling purposes.


Data Exclusions

The following tables show the accounts that have been excluded from the analysis during the

data collection phase.

Category Description

Cancelled Loan approved but cancelled before disbursement

Committed Loan approved but not yet disbursed

Performance Definitions

All accounts active as of the observation dates were used in the development of the models.

Accounts were segmented into three performance categories based on their behavior during the

performance period.

Performance Definition – Early Default Definition

Event Account transfers to liquidation or purchase within the first 18 months on book

Non-Event Account is past due, pre-paid in full, or current

Indeterminate

Account becomes delinquent or in deferment within the first 18 months on book

Months on book < 18

Account has no reported statuses in the performance file

Performance Definition – Early Payment Problem Definition

Event Account becomes delinquent, in deferment, transfers to liquidation, or purchase

within the first 18 months on book

Non-Event Account is past due, paid in full, or current

Indeterminate Months on book < 18

Account has no reported statuses in the performance file


Population Overview

This section provides an overview of the portfolios being analyzed, including population

statistics, summary counts, and detailed counts. Population statistics show the overall risk

associated with each portfolio through non-event-to-event odds and event rates. These figures

are ratios of the non-event and event populations. The summary counts indicate the distribution

of non-events, events and indeterminates over the portfolio. The detailed counts indicate the

distribution applicants which the larger assigned categories of non-event, event and

indeterminate.

Population Statistics

The following table shows the statistics for Portfolio Group 1.

Early Default

Early Payment

Problem

Population Odds 25.20 8.33

Event Rate 0.28% 0.84%

Summary Counts

The following table shows the frequency of accounts for Portfolio Group 1. These counts

represent the scoreable population only.

Non-Event Event Indeterminate

# % # % # %

Early Default 1,890 6.98% 75 0.28% 25,119 92.70%

Early Payment Problem 1,890 6.98% 227 0.84% 24,967 92.15%


Detailed Counts

Early Default

Category Order Assign

Exclusions Total

(Row)

Total

# (Category)

Exclusions –

Booked

Accounts

10 20 Cancelled 6 11

11 22 Commit 5

Event 1 30 Purchase 26

75 2 31 Liquidation 49

Indeterminate –

Approved

3 60 Delinquent 127

25,119

4 61 Deferred 25

5 62 Month on Books < 1 7,400



8 65 Months on Books < 18 1,791

9 66 No MISStatus History Available 1

Non-Event

12 70 Past Due 42

1,890 13 80 Paid in Full 121

14 81 Current 1,727

Total Accounts 27,095 27,095


Early Payment Problem

Category Order

Assign

# Exclusions

Total

(Row)

Total

(Category)

Exclusions –

Booked

Accounts

10 20 Cancelled 6 11

11 22 Commit 5

Event

1 30 Purchase 26

227 2 31 Liquidation 49

3 40 Delinquent 127

4 41 Deferred 25

Indeterminate

- Approved


24,967



8 65 Months on Books < 18 1,791

9 66 No MISStatus History Available 1

Non-Event

12 70 Past Due 42

1,890 13 80 Paid in Full 121

14 81 Current 1,727

Total Accounts 27,095 27,095


Scoreable Counts

Data Source Scorecard Description

Event -

Early

Default (#)

Non-

Event (#)

Indeterminate

(#)

Total

(#)

Term Loan &

LOC -

Consumer

126

Non-derogatory, oldest trade line is less than 20

years old, and total number of trade lines is less

than 20.

10 147 1,909 2,066

127


years old, and total number of trade lines is greater

than 20.

15 171 2,184 2,370

128 Non-derogatory and oldest trade line is greater

than or equal to 20 years old. 6 158 2,720 2,884

129 Derogatory, one trade line is 60 or more days

delinquent. 6 112 1,335 1,453

130 Severe derogatory, two or more trade lines are 60

or more days delinquent. 4 94 1,486 1,584

Term Loan &

LOC -

Consumer

and D&B

131


years old, and total number of trade lines is less

than 20.

5 126 1,719 1,850

132


years old, and total number of trade lines is greater

than 20.

7 226 2,332 2,565

133 Non-derogatory and oldest trade line is greater

than or equal to 20 years old. 6 322 4,313 4,641

134 Derogatory, one trade line is 60 or more days

delinquent. 7 128 1,814 1,949

135 Severe derogatory, two or more trade lines are 60

or more days delinquent. 4 118 2,056 2,178


Data Source Scorecard Description

Event -

Early

Default (#)

Non-

Event

(#)

Indeterminate

(#)

Total

(#)

Other Data

Combinations

145 &146 Term Loan & LOC – No Consumer report, D&B - 18 217 235

149 Term Loan & LOC – Application, Consumer,

Financial 3 110 415 528

150 Term Loan & LOC – Application, Consumer,

Financial, and D&B 2 152 494 648

152 Term Loan & LOC – Application and Financial - 1 8 9

153 Term Loan & LOC – Application, Financial, and

D&B - 4 13 17

199 Unscoreable - 3 73 76

75 1,890 25,119 27,084


Descriptive Statistics

Portfolio Group 1 – Performance Category by Score Distributions


New versus Existing Business

Values

Early Default Definition

Event

Non-

Event

Indeterminates

Delq or Defer

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

Existing Business 32 1,112 79 4,441 10,947 34.75

New Business 43 778 73 2,940 6,575 18.09

Missing -- -- -- 19 45 --

Total 75 1,890 152 7,400 17,567 25.20

Values

Early Payment Problem Definition

Event

Non-

Event

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

Existing Business 111 1,112 4,441 10,947 10.01

New Business 116 778 2,940 6,575 6.71

Missing -- -- 19 45 --

Total 227 1,890 7,400 17,567 8.33


Loan Size

Values


Event

Non-

Event

Indeterminates

Delq or Defer

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

$0-$350,000 74 1,889 129 4,238 10,707 25.53

$350,001+ 1 1 23 3,162 6,860 1.00

Total 75 1,890 152 7,400 17,567 25.20

Values


Event

Non-

Event

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

$0-$350,000 203 1,889 4,238 10,707 9.31

$350,001+ 24 1 3,162 6,860 0.04

Total 227 1,890 7,400 17,567 8.33


Delivery Method

Values


Event

Non-

Event

Indeterminates

Delq or Defer

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

CLP - - - 53 164 -

CMLAND - - 1 95 101 -

OTH 7A - - 10 1,246 2,198 -

PLP 3 - 25 3,103 6,818 -

RUREXP - - - 8 23 -

SBAEXP - - - 3 5 -

SMLNAD 72 1,890 116 2,892 8,258 26.25

Total 75 1,890 152 7,400 17,567 25.20

Values


Event

Non-

Event

Indeterminates

MOB = 0

Indeterminates

Insufficient

History Odds

CLP - - 53 164 -

CMLAND 1 - 95 101 -

OTH 7A 10 - 1,246 2,198 -

PLP 28 - 3,103 6,818 -

RUREXP - - 8 23 -

SBAEXP - - 3 5 -

SMLNAD 188 1,890 2,892 8,258 10.05

Total 227 1,890 7,400 17,567 8.33


Score Distributions and Graphs

Score Distribution

The table below tallies the total accounts by score range, as well as the cumulative % of the column population. Odds are presented for

both the odds associated with the row’s score range as well as the cumulative ascending odds throughout the total range of scores.

The cumulative percentages are based on accounts at or below the score range.


Portfolio Group 1, SBSS 6.0, Early Default:

Non-event Event Indeterminates: Total Odds

Score

Range # Cum % # Cum % # Cum % # Cum % Actual

Cum

Ascending

80-<90 - 0.00% 0 0.00% 2 0.01% 2 0.01% -- --

90-<100 - 0.00% 0 0.00% 16 0.06% 16 0.06% -- --

100-<110 - 0.00% 0 0.00% 33 0.13% 33 0.12% -- --

110-<120 - 0.00% 0 0.00% 103 0.41% 103 0.38% -- --

120-<130 - 0.00% 0 0.00% 250 1.00% 250 0.92% -- --

130-<140 - 0.00% 0 0.00% 509 2.03% 509 1.88% -- --

140-<150 108 5.71% 8 10.67% 1,766 7.03% 1,882 6.95% 13.50 13.50

150-<160 280 14.81% 18 24.00% 3,801 15.13% 4,099 15.13% 17.20 15.56

160-<170 527 27.88% 32 42.67% 6,814 27.13% 7,373 27.22% 17.64 16.47

170-<180 832 44.02% 44 58.67% 10,757 42.82% 11,633 42.95% 25.42 18.91

180-<190 1,172 62.01% 60 80.00% 14,804 58.94% 16,036 59.21% 21.25 19.53

190-<200 1,479 78.25% 68 90.67% 18,869 75.12% 20,416 75.38% 38.38 21.75

200-<210 1,690 89.42% 74 98.67% 22,401 89.18% 24,165 89.22% 35.17 22.84

210-<220 1,835 97.09% 75 100.00% 24,315 96.80% 26,225 96.83% 145.00 24.47

220-<230 1,876 99.26% 75 100.00% 24,907 99.16% 26,858 99.17% -- 25.01

230-<240 1,886 99.79% 75 100.00% 25,033 99.66% 26,994 99.67% -- 25.15

240-<250 1,887 99.84% 75 100.00% 25,042 99.69% 27,004 99.70% -- 25.16

250-<998 1,887 99.84% 75 100.00% 25,043 99.70% 27,005 99.71% -- 25.16

998 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% -- 25.20

Total 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% -- 25.20



Risk Level Score Range # Cum % # Cum % # Cum % # Cum % Actual Cum

Ascending

High Less than 160 280 14.81% 18 24.00% 3,801 15.13% 4,099 15.13% 15.56 15.56

Medium 160 -< 200 1,199 63.44% 50 66.67% 15,068 59.99% 16,317 60.25% 23.98 21.75

Low 200 or more 408 21.59% 7 9.33% 6,174 24.58% 6,589 24.33% 58.29 25.16

Unscoreable 998 3 0.16% -- 0.00% 76 0.30% 79 0.29% -- --

Total 1,890 100.00% 75 100.00% 25,119 100.00% 27,084 100.00% 25.20 25.20


Portfolio Group 1, SBSS 6.0, Early Payment Problem:


Score

Range # Cum % # Cum % # Cum % # Cum % Actual

Cum

Ascending

80-<90 0 0.00% 0 0.00% 1 0.01% 2 0.00% -- --

90-<100 0 0.00% 0 0.00% 15 0.06% 16 0.06% -- --

100-<110 0 0.00% 0 0.00% 32 0.13% 33 0.13% -- --

110-<120 0 0.00% 0 0.00% 102 0.41% 103 0.41% -- --

120-<130 0 0.00% 0 0.00% 249 1.00% 250 1.00% -- --

130-<140 0 0.00% 1 0.44% 507 2.03% 509 2.03% 0.00 0

140-<150 108 5.71% 19 8.37% 1,754 7.03% 1,882 7.02% 6.00 5.68

150-<160 280 14.81% 55 24.23% 3,763 15.13% 4,099 15.07% 4.78 5.09

160-<170 527 27.88% 87 38.33% 6,758 27.13% 7,373 27.07% 7.72 6.06

170-<180 832 44.02% 129 56.83% 10,671 42.82% 11,633 42.74% 7.26 6.45

180-<190 1,172 62.01% 164 72.25% 14,699 58.94% 16,036 58.87% 9.71 7.15

190-<200 1,479 78.25% 192 84.58% 18,744 75.12% 20,416 75.07% 10.96 7.70

200-<210 1,690 89.42% 215 94.71% 22,259 89.18% 24,165 89.15% 9.17 7.86

210-<220 1,835 97.09% 223 98.24% 24,166 96.80% 26,225 96.79% 18.13 8.23

220-<230 1,876 99.26% 227 100.00% 24,754 99.16% 26,858 99.15% 10.25 8.26

230-<240 1,886 99.79% 227 100.00% 24,880 99.66% 26,994 99.65% -- 8.31

240-<250 1,887 99.84% 227 100.00% 24,889 99.69% 27,004 99.69% -- 8.31

250-<998 1,887 99.84% 227 100.00% 24,890 99.70% 27,005 99.69% -- 8.31

998 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% -- 8.33

Total 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% -- 8.33



Risk Level Score Range # Cum % # Cum % # Cum % # Cum % Actual Cum

Ascending

High Less than 160 280 14.81% 55 24.23% 3,763 15.13% 4,099 15.07% 5.09 5.09

Medium 160 -< 200 1,199 63.44% 137 60.35% 14,981 59.99% 16,317 60.00% 8.75 7.70

Low 200 or more 408 21.59% 35 15.42% 6,146 24.58% 6,589 24.62% 11.66 8.31

Unscoreable 998 3 0.16% -- 0.00% 76 0.30% 79 0.31% -- --

Total 1,890 100.00% 227 100.00% 24,966 100.00% 27,084 100.00% 8.33 8.33

validation analysis results

Documents