model risk management in the age of digital analytics and ... · (anomaly detection) ml technique...

Model risk management in the age of digital analytics and machine learningSwiss Risk Association

18 Sept 2017

Dr. Karl Ruloff

MRM & ML – Swiss Risk 2017-09-18

1. What is Machine Learning?

2. Where does Machine Learning impact Model Risk?

3. Machine Learning challenges and remediation in MRM

4. Where can Machine Learning support in MRM?

Agenda


What is Machine Learning?


What is Machine Learning?

The “field of study that gives computers the ability to learn without

being explicitly programmed”

(Arthur Samuel, 1959)

Making data-driven predictions or decisions, through building a

model from sample inputs

Closely related to (and often overlapping with) computational

statistics


Deep Learning (DL)

Machine Learning (ML)

Rule-Based System

AI Spectrum

► Explicit programming. If “x”, then do “y”

► Key-word filtration

► Application of domain knowledge to create rules

► Predictive analytics (e.g., logistic regression)

► Neural Network with few hidden layers

► Natural Language Processing (NLP)

► Bayesian probability frameworks

► Support Vector Machines (SVM)

► Multi-layered high level abstraction artificial “neuron replication”

► Deep Neural Networks (CNNs, RNNs)

► Deep Belief Networks

Rule-Based System automation based on domain knowledge

Machine Learning ability to self-learn with the use of statistical and optimization techniques

Deep Learning complex branch of ML built on neural networks tomimic the functioning of the human brain to providehigher accuracy on more complex tasks.

Where is Machine Learning in the Artificial Intelligence spectrum?


Some Machine Learning model terminology

Random Forests /Decision Trees

Neural Nets

k-Means

Supervised Unsupervised

n derivatives …learning by error minimization

approx. mapping of x to y

weight adjustment

labelled data

competitive / reward learning

find unknown patterns

no adjustment

no labels

Semi-Supervised

for regression &

classification problems

for clustering &

association problems

Support VectorMachines

n derivatives …

…

Hierarchical Clustering

Isolation Forest

…


Broad application space achievable with a compact set of Machine Learning methods

Segmentation for Revenue Optimization (Clients, Channels, Products)

Customer Preference

Market Characteristics

Asset Management

Transaction / Trade / Market Surveillance

Fraud Detection, AML

Compliance Concerns

High Risk Behavior Analysis

Default Risk

Forecasting (Cash Flows, Reserves, Write-Offs)

Profit Optimization

Discovering Structure

(Clustering)

Value Estimation

(Regression)

Irregularity Identification

(Anomaly Detection)

APPROACH USE CASESML TECHNIQUE

Categorical Prediction

(Classification)

► Neural Networks

► Linear / Polynomial Regression

► XGBoost

► Isolation Forest

► Bayesian Analysis

► RBMs

► GBMs

► Decision tree / Random forests

► Support Vector Machines (SVM)

► Naive Bayes

► DBSCAN

► K-means clustering

► SOM Neural Network


Feature Engineering

Development of Machine Learning models: Process view

Define analytical Problem

Develop & optimize model

Deploy & monitor

1 3 4Data Pre-Processing

2

Analytics lifecycle

Business Domain expert

Owns business case

Decision maker

Monitoring

Functional data expert

Report developer

Data analysis & transformation

Methods expert

Predictive modeller

Architect

Developer

Data preparation

Deployment & monitoring

Project Manager Business Analyst Quant Modeller IT Expert

5


ApplyPredictive Algorithms

Scoring Model

Output

Fine-tuning

Calibration

Feedback loop

Dimensionality Reduction using Generalized Low Ranking Model (GLRM), t-SNE

Applicable to all data types

Projection into a lower dimensional space

Variable ranking

Imputation of missing values

Removal of noise

Step 1- Unsupervised machine learning for clustering and to identify local / global outliers

Restr. Boltzmanmachines

KSOM, k-Means DBSCAN

Step 2- Supervised machine learning (Random Forests, GBMs, SVMs) to approximate target function(s)

Display output using dashboards, reports or bespoke end-user apps

Set scoring thresholds & create alerts for further processing and investigation

Feature Extraction

Assess Results & Create Alerts

Listed algorithms represent just a subset of applied methods

Development of Machine Learning models: Technical view


What is the difference between Statistics and Machine Learning?

Shared goal: Learning from data

Difference: ML emphasizes optimization and performance over

inference

Implication for Model Development and Model Risk: Machine Learning

Utilizes a richer set of modeling approaches with increased predictive

power

Leads to increased complexity, reduced exploratory insight and

transparency


Where does Machine Learning impact Model Risk?


How does Machine Learning impact Model Risk

Model risk is the potential for adverse consequences of decisions based on incorrect or misused model outputs and reports. (SR 11/7)

Bu

sin

ess P

rocess

Business DecisionsInputs Outputs

Internal Data

External Data

Assumptions

Other Model Inputs

e.g., Ratings,

Exposures

Reporting

Analytics

Input to Other

Models

Model

Model Lifecycle

Business Purpose

Development

Implementation

Validation

► More data (observations, variables)

► Unstructured data (social networks,…)

► Generally less (distributional) assumptions

► «Let the data speak for itself»

► Highly explorative

► Strong emphasize on feature engineering, pattern recognition techniques and sampling

► Model tuning, not parameter estimation

► Many new different model types

► Highly non-linear

► Meta parameters

► Partially non-transparent

► Overfitting

► Non-transparency might reducebusiness acceptance


Machine Learning challenges and remediations in MRM


MRM challenges of Machine Learning models – and some remediations (1/2)

MR Area

Input

Model

Additional data sources Deep business understanding

Large & diverse data sets Visualization techniques, Anomaly detection

More data Computing power, dimensionality reduction techniques

Sampling Impact assessment of different sampling methods

Diverse model types, choice of methodology

Understand assumptions and limitations of different model types

Overfitting Cross validation

Increased complexity & effort

Templates/ frameworks for ML model type evaluation steps

Reduced transparency Outcome analysis, Feature importance charts

Assessment of conceptual soundness

Understand applied models and develop deep business understanding

Model performance Specific tools & metrics (Confusion matrix, Recall/Precision), but also well known ones (AUROC,..)

Challenge Remediation


MRM challenges of Machine Learning models – and some remediations (2/2)

MR Area

Use

Diverse, new data sources Monitoring of data quality

Complex models Monitoring of calculation results

Increased complexity, reduced transparency

Strong governance

Educating users & Senior Management on pros &

cons, development process, and limitations of ML

models

Challenge Remediation


Where can Machine Learning support in MRM?


How can Machine Learning support MRM?

MR Area Application of Machine Learning

Anomaly detection

Name mapping, removal of duplicates, e.g. for consolidation of exposures

Data imputation proposal => What if analyses

Clustering and segmentation of input data

Development of challenger models

Monitoring of model input: Automatic identification of anomalous data input

& proposal of remediation (replacement of outliers; data imputation for

missing data)

Monitoring of calculation results: Automatic identification of anomalous

calculation results

Input

Model

Use


Summary


Conclusion

Machine Learning is not pure science, it is also part art

Machine Learning adds additional layers of complexity to the model

lifecycle

More mathematics is not sufficient to handle this complexity

It takes in-depth business understanding from developers, model risk

managers, users and governance stakeholders

=> Critical validation & strong governance is key for Machine

Learning MRM

Contacts

Dr. Karl Ruloff

Senior Manager

Financial Services Risk EY Switzerland

[email protected]

Dr. Oliver Kaufmann

Executive Director, Head Quants & Analytics

Financial Services Risk EY Switzerland

[email protected]

mailto:[email protected]

mailto:[email protected]

model risk management in the age of digital analytics and ... · (anomaly detection) ml technique...

Documents