Smooth Isotonic Regression: A New Method to Calibrate Predictive ModelsAMIA Summits Transl Sci Proc. 2011;2011:16-20. Epub 2011 Mar 7

20121210

Statistical method paper

Topic

Smooth calibration

2 4 6 8 10

-20

02

04

06

08

01

00

12

0

concentration

via

bility

wild type

mutant

従来法Hill equation (HE)

新手法Isotonic regression (IR)

最小二乗法によりロジスティック曲線に回帰毒性評価値はEC50

細胞間順序を加えて二次計画法により推定毒性評価値はAUC

)50log(log

infinf

1)(

ECxn

zero

e

EEExE

2

1

)}()({min ii

n

i

i cfcgw

Hill, A.V. J. Physiol. 1910Best, M.J. & Chakravarti N. Mathematical Programming. 1990

ΔEC50

ΔAUC

Reliability diagram

• Out of thirty 20% probability forecasts, the predicted event should verify six times, i.e. in 20% of the time, not more, not less.

Smooth non-parametric estimators

(a) (b)

Figure 1: Calibration plot with fitted probabilities by

(a) sigmoid fitting and (b) isotonic regression.

methods proposed by Platt [13] and Zadrozny and

Elkan [15].

Theparametric approach of Platt applies to model out-

put rank p consists of finding the parameters A and B

for a sigmoid function f s(p) = 11+ eA ·p + B , such that

the negative log-likelihood is minimized. However,

thismethod may lead to poor calibration when theout-

putsdo not fit thesigmoid function. Theapplication of

this method, referred to as sigmoid fitting, is shown in

Figure 1 (a) by the dotted red line.

The non-parametric approach of Zadrozny and Elkan

applies a pair-adjacent violators algorithm [2] to the

previously sorted output probabil ities p of the model

in order to find a stepwise-constant isotonic function

that best fits according to a mean-squared error crite-

rion. However, the outputs of this calibration method

tend to overfit the data if no smoothing regularization

is applied. The use of this method, referred to as iso-

tonic regression, isshown in Figure1 (b) by thedotted

red line.

Smooth non-parametric estimators are expected to al-

leviate overfitting and underfitting problems, and thus

havereceived moreattention recently. Themethodsby

Wang et al. [14] and Meyer [10] find anon-decreasing

mapping function t() that minimizes:

i

(ci − t(pi ))2 + λ

b

a

[t (m ) (γ)]2 dγ, (1)

where m corresponds to a smoothness parameter, a

and b represent the range of input predictions, and λ

balances thegoodness-of-fit (first component) and the

smoothness (second component) of thetransformation

function t(). When m = 1, Equation 1 corresponds

to a piece-wise linear estimator. When m = 2, Equa-

tion 1 represents asmooth monotoneestimator. In the-

ory, these models are smoother than isotonic regres-

sion and more flexible than sigmoid regression, but

their inferences requiremuch heavier computation and

tedious parameter tuning. Moreover, approximation

algorithms show large empirical losses, e.g. 30% us-

ing second-order cone programming [14].

Method

We intended to develop a smoother yet computation-

ally affordable method to further improve the cali-

bration of predictive models. We observed that iso-

tonic regression is a non-parametric method that joins

predictions into larger bins, as indicated by the flat

regions in Figure 1(b). By interpolating between a

few representative values, we can obtain a smoother

function. However, we must ensure that such inter-

polation function g() is monotonically increasing to

maintain the discriminative ability of the predictive

model. Let P = { pi } the set of all predictions pi and

C = { ci } their corresponding class labels, then the

function t∗ () = g(f (P ), C ) is also monotonically

increasing, where f () is the isotonic regression func-

tion, P and C are subsets of predictions and their

corresponding class labels, repectively.

Based on these considerations, we propose a novel

approximation to the optimal smooth function t∗ ()

that minimizes Equation 1 in three steps. First,

we apply isotonic regression to obtain a mono-

tone non-parametric function f () that minimizes

i (ci − f (pi ))2. Second, we select s representative

points from theisotonic mapping function. Finally, we

construct a monotonic spline, called Piecewise Cubic

Hermite Interpolating Polynomial (PCHIP) [7], that

interpolates between the sampled points from the iso-

tonic regression function to obtain asmoothed approx-

imation to Equation 1. Note that the interpolation by

PCHIPismonotonic to keep thepartial ordering of the

predictive model probabilities, and that it introduces

thesmoothness. Algorithm 1 gives thedetails on these

steps in acompact form.

Inputs: Prediction probabilities P = p1, . . . , pn , class

labels C = c1, . . . , cn .

Output: Smoothed isotonic regression function h.

1. Obtain f ∗ = argminf i (ci − f (pi ))2, subject

to f (pi ) ≤ f (pi + 1),∀i (Isotonic Regression).

2. Samples pointsfrom f ∗ (γ), γ ∈ (0, 1), onepoint

per flat region of f ∗ (·). Denote samples as P ,

and their corresponding class labels asC .

3. Construct a Piecewise Cubic Hermite Interpolat-

ing Polynomial function t∗ (f ∗ (P ), C )) to ob-

tain a monotone smoothing spline as the final

transformation function for calibration.

Algorithm 1: Smooth Isotonic Regression

17

ma ~ b

l

smoothness parameter

the range of input predictions

balances the goodness-of-fit and the smoothness

Their inferences require much heavier computation and tedious parameter tuning

Spline curve

多項式近似

Piecewise Cubic Hermite Interpolating Polynomial(PCHIP)

区分的 3 次元エルミート内挿多項式

順序も加えたものらしい

Hosmer-Lemeshow test

A statistical test for goodness of fit for logistic regression model

: Model is well fit

: Model is NOT well fit

library(eRm)library(ResourceSelection)library(MKmisc)

0H

H1Oc

i

E i

G

ni

the No.group (= 10)

the sum of cases (c = 0 or 1)

the estimated probabilities

the No.cases in group i

Simulation data analysisLR

sigm

oid

IRsI

R

Real biological data analysis

LRsi

gmo

idIR

sIR

http://en.wikipedia.org/wiki/Calibration_curve

http://www.ecmwf.int/products/forecasts/guide/The_reliability_diagram.html

http://en.wikipedia.org/wiki/Sigmoid_function

http://en.wikipedia.org/wiki/Hill_equation_(biochemistry)

http://en.wikipedia.org/wiki/Isotonic_regression

http://en.wikipedia.org/wiki/Hosmer%E2%80%93Lemeshow_test

http://en.wikipedia.org/wiki/Spline_(mathematics)

区分的 3 次元エルミート内挿多項式

http://www.mathworks.co.jp/jp/help/matlab/ref/pchip.html