ETC5410: Nonparametric smoothing methods 1

ETC5410: Nonparametricsmoothing methods

July 2008

Rob J Hyndmanhttp://www.robjhyndman.com/

ETC5410: Nonparametric smoothing methods 2

Outline

1 Density estimation

2 Kernel regression

3 Splines

4 Additive models

5 Functional data analysis

ETC5410: Nonparametric smoothing methods 3

ETC5410: Nonparametricsmoothing methods

3. Splines

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Interpolating splines 4

Outline

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Interpolating splines 5

Interpolating splines

ETC5410: Nonparametric smoothing methods Interpolating splines 6

Interpolating splines

Observations: (xi , yi) for i = 1, . . . , n.

A spline is a continuous function f (x) interpolatingall points and consisting of polynomials betweeneach consecutive pair of ‘knots’ xi and xi+1.

Parameters constrained so that f (x) iscontinuous.

Further constraints imposed to give continuousderivatives.

ETC5410: Nonparametric smoothing methods Interpolating splines 6

Interpolating splines

Observations: (xi , yi) for i = 1, . . . , n.

A spline is a continuous function f (x) interpolatingall points and consisting of polynomials betweeneach consecutive pair of ‘knots’ xi and xi+1.

Parameters constrained so that f (x) iscontinuous.

Further constraints imposed to give continuousderivatives.

ETC5410: Nonparametric smoothing methods Interpolating splines 6

Interpolating splines

Observations: (xi , yi) for i = 1, . . . , n.

A spline is a continuous function f (x) interpolatingall points and consisting of polynomials betweeneach consecutive pair of ‘knots’ xi and xi+1.

Parameters constrained so that f (x) iscontinuous.

Further constraints imposed to give continuousderivatives.

ETC5410: Nonparametric smoothing methods Interpolating splines 6

Interpolating splines

Observations: (xi , yi) for i = 1, . . . , n.

A spline is a continuous function f (x) interpolatingall points and consisting of polynomials betweeneach consecutive pair of ‘knots’ xi and xi+1.

Parameters constrained so that f (x) iscontinuous.

Further constraints imposed to give continuousderivatives.

ETC5410: Nonparametric smoothing methods Interpolating splines 7

Interpolating splines

0 20 40 60 80 100

05

1015

Linear interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 7

Interpolating splines

0 20 40 60 80 100

05

1015

Linear interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 8

Interpolating splines

0 20 40 60 80 100

05

1015

Quadratic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 8

Interpolating splines

0 20 40 60 80 100

05

1015

Quadratic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 8

Interpolating splines

0 20 40 60 80 100

05

1015

Quadratic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 8

Interpolating splines

0 20 40 60 80 100

05

1015

Quadratic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 8

Interpolating splines

0 20 40 60 80 100

05

1015

Quadratic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 9

Interpolating splines

0 20 40 60 80 100

05

1015

Cubic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Interpolating splines 10

Interpolating splines

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

ETC5410: Nonparametric smoothing methods Interpolating splines 10

Interpolating splines

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

ETC5410: Nonparametric smoothing methods Interpolating splines 10

Interpolating splines

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

ETC5410: Nonparametric smoothing methods Interpolating splines 10

Interpolating splines

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

ETC5410: Nonparametric smoothing methods Interpolating splines 10

Interpolating splines

To define a cubic spline uniquely, it is necessaryto specify two additional constraints.

Usually, this is done by specifying the form ofthe function at the two extremes, x1 and xn.

For example, a “natural” spline is obtained byrequiring f ′′(x1) = f ′′(xn) = 0, thus making thecurve linear at the extremes.

Implementation in R

plot(x,y)

lines(spline(x,y))

ETC5410: Nonparametric smoothing methods Interpolating splines 11

Monotonic interpolation

‘Hyman filter’ ensures spline is monotonic byconstraining derivatives while (possibly) sacrificingsmoothness.

Second derivative may no longer be continuous atall knots.

Implementation in R

require(demography)

plot(x,y)

lines(cm.spline(x,y), col=4)

Reference: Smith, Hyndman and Wood (JPR, 2001)

ETC5410: Nonparametric smoothing methods Interpolating splines 11

Monotonic interpolation

‘Hyman filter’ ensures spline is monotonic byconstraining derivatives while (possibly) sacrificingsmoothness.

Second derivative may no longer be continuous atall knots.

Implementation in R

require(demography)

plot(x,y)

lines(cm.spline(x,y), col=4)

Reference: Smith, Hyndman and Wood (JPR, 2001)

ETC5410: Nonparametric smoothing methods Interpolating splines 11

Monotonic interpolation

‘Hyman filter’ ensures spline is monotonic byconstraining derivatives while (possibly) sacrificingsmoothness.

Second derivative may no longer be continuous atall knots.

Implementation in R

require(demography)

plot(x,y)

lines(cm.spline(x,y), col=4)

Reference: Smith, Hyndman and Wood (JPR, 2001)

ETC5410: Nonparametric smoothing methods Interpolating splines 12

Monotonic interpolation

0 20 40 60 80 100

05

1015

Cubic monotonic interpolating spline

Age

Cum

ulat

ive

deat

hs (

'000

)

ETC5410: Nonparametric smoothing methods Smoothing splines 13

Outline

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Smoothing splines 14

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.

Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

ETC5410: Nonparametric smoothing methods Smoothing splines 14

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

ETC5410: Nonparametric smoothing methods Smoothing splines 14

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

ETC5410: Nonparametric smoothing methods Smoothing splines 14

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

ETC5410: Nonparametric smoothing methods Smoothing splines 14

Spline smoothing

The average squared prediction error is∑[yj − r(xj)]2.

Reduces to zero if r interpolates the data.Penalized sum of squares

Sλ(r) =∑n

j=1[yj − r(xj)]2 + λ∫

(r ′′(x))2dx

λ denotes a smoothing parameter.

First term measures closeness to the data.

Second term penalizes curvature in thefunction.

ETC5410: Nonparametric smoothing methods Smoothing splines 15

Cubic smoothing splines

A cubic smoothing spline is the function rλ(x)which minimizes Sλ(r) over the class of all twicedifferentiable functions on the range of xj.

It consists of piecewise cubic polynomials, withthe pieces separated by the xj values.

At the design points, xj , rλ(x) and its first twoderivatives are continuous. The third derivativemay be discontinuous.

At the minimum and maximum xj values, thesecond derivative of rλ(x) is zero. Hence, thesmoothing spline is linear beyond the extremedata points.

ETC5410: Nonparametric smoothing methods Smoothing splines 15

Cubic smoothing splines

A cubic smoothing spline is the function rλ(x)which minimizes Sλ(r) over the class of all twicedifferentiable functions on the range of xj.

It consists of piecewise cubic polynomials, withthe pieces separated by the xj values.

At the design points, xj , rλ(x) and its first twoderivatives are continuous. The third derivativemay be discontinuous.

At the minimum and maximum xj values, thesecond derivative of rλ(x) is zero. Hence, thesmoothing spline is linear beyond the extremedata points.

ETC5410: Nonparametric smoothing methods Smoothing splines 15

Cubic smoothing splines

A cubic smoothing spline is the function rλ(x)which minimizes Sλ(r) over the class of all twicedifferentiable functions on the range of xj.

It consists of piecewise cubic polynomials, withthe pieces separated by the xj values.

At the design points, xj , rλ(x) and its first twoderivatives are continuous. The third derivativemay be discontinuous.

At the minimum and maximum xj values, thesecond derivative of rλ(x) is zero. Hence, thesmoothing spline is linear beyond the extremedata points.

ETC5410: Nonparametric smoothing methods Smoothing splines 15

Cubic smoothing splines

A cubic smoothing spline is the function rλ(x)which minimizes Sλ(r) over the class of all twicedifferentiable functions on the range of xj.

It consists of piecewise cubic polynomials, withthe pieces separated by the xj values.

At the design points, xj , rλ(x) and its first twoderivatives are continuous. The third derivativemay be discontinuous.

At the minimum and maximum xj values, thesecond derivative of rλ(x) is zero. Hence, thesmoothing spline is linear beyond the extremedata points.

ETC5410: Nonparametric smoothing methods Smoothing splines 16

Cubic smoothing splines

Large values of λ produce smoother curveswhile smaller values produce rougher curves.

At λ→∞, the penalty term dominates Sλ(r),forcing r ′′(x) = 0 for all x . So the solution isthe least squares straight line.

As λ→ 0, the penalty term becomes negligibleand the solution tends to an interpolatingfunction which is twice differentiable.

ETC5410: Nonparametric smoothing methods Smoothing splines 16

Cubic smoothing splines

Large values of λ produce smoother curveswhile smaller values produce rougher curves.

At λ→∞, the penalty term dominates Sλ(r),forcing r ′′(x) = 0 for all x . So the solution isthe least squares straight line.

As λ→ 0, the penalty term becomes negligibleand the solution tends to an interpolatingfunction which is twice differentiable.

ETC5410: Nonparametric smoothing methods Smoothing splines 16

Cubic smoothing splines

Large values of λ produce smoother curveswhile smaller values produce rougher curves.

At λ→∞, the penalty term dominates Sλ(r),forcing r ′′(x) = 0 for all x . So the solution isthe least squares straight line.

As λ→ 0, the penalty term becomes negligibleand the solution tends to an interpolatingfunction which is twice differentiable.

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 2

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 3

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 4

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 5

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 6

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 7

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 8

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 9

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 10

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 11

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 12

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 13

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 14

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 15

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 16

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 17

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 18

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 19

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 20

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 21

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 22

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 23

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 24

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 17

Cubic smoothing splines

df= 25

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

ETC5410: Nonparametric smoothing methods Smoothing splines 18

Cubic smoothing splines

Implementation in R

plot(price, shipments)

lines(smooth.spline(price, shipments))

lines(smooth.spline(price, shipments,

df=5), col=2)

Fits a cubic smoothing spline to the data.

The value of λ is selected automatically usingcross-validation.

The argument df can be supplied and then λ ischosen to give approximately df degrees offreedom.

ETC5410: Nonparametric smoothing methods Smoothing splines 18

Cubic smoothing splines

Implementation in R

plot(price, shipments)

lines(smooth.spline(price, shipments))

lines(smooth.spline(price, shipments,

df=5), col=2)

Fits a cubic smoothing spline to the data.

The value of λ is selected automatically usingcross-validation.

The argument df can be supplied and then λ ischosen to give approximately df degrees offreedom.

ETC5410: Nonparametric smoothing methods Smoothing splines 18

Cubic smoothing splines

Implementation in R

plot(price, shipments)

lines(smooth.spline(price, shipments))

lines(smooth.spline(price, shipments,

df=5), col=2)

Fits a cubic smoothing spline to the data.

The value of λ is selected automatically usingcross-validation.

The argument df can be supplied and then λ ischosen to give approximately df degrees offreedom.

ETC5410: Nonparametric smoothing methods Smoothing splines 18

Cubic smoothing splines

Implementation in R

plot(price, shipments)

lines(smooth.spline(price, shipments))

lines(smooth.spline(price, shipments,

df=5), col=2)

Fits a cubic smoothing spline to the data.

The value of λ is selected automatically usingcross-validation.

The argument df can be supplied and then λ ischosen to give approximately df degrees offreedom.

ETC5410: Nonparametric smoothing methods Smoothing splines 19

Hodrick-Prescott filter

The Hodrick-Prescott filter is a special

case of a cubic smoothing spline.

HP recommended λ = 1600 for quarterly

data. There is no theoretical justification

for this.

Better to use proper bandwidth selection

tools.

ETC5410: Nonparametric smoothing methods Smoothing splines 19

Hodrick-Prescott filter

The Hodrick-Prescott filter is a special

case of a cubic smoothing spline.

HP recommended λ = 1600 for quarterly

data. There is no theoretical justification

for this.

Better to use proper bandwidth selection

tools.

ETC5410: Nonparametric smoothing methods Smoothing splines 19

Hodrick-Prescott filter

The Hodrick-Prescott filter is a special

case of a cubic smoothing spline.

HP recommended λ = 1600 for quarterly

data. There is no theoretical justification

for this.

Better to use proper bandwidth selection

tools.

ETC5410: Nonparametric smoothing methods Smoothing splines 20

Cross-validation again

Recall: Find smoothing parameter which minimises

CV(h) =1

n

n∑j=1

[rj(xj)− yj ]2

where rj(xj) uses all data except (xj , yj).

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

df chosen by CV

ETC5410: Nonparametric smoothing methods Smoothing splines 20

Cross-validation again

Recall: Find smoothing parameter which minimises

CV(h) =1

n

n∑j=1

[rj(xj)− yj ]2

where rj(xj) uses all data except (xj , yj).

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

df chosen by CV

ETC5410: Nonparametric smoothing methods Smoothing splines 21

Cross-validation again

5 10 15 20 25

1015

20

Degrees of freedom

CV

ETC5410: Nonparametric smoothing methods Smoothing splines 22

Cross-validation again

500 550 600 650 700 750 800 850

1015

2025

3035

World pulp price

Pul

p sh

ipm

ents

df = 5.253

ETC5410: Nonparametric smoothing methods Smoothing splines 23

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

ETC5410: Nonparametric smoothing methods Smoothing splines 23

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

ETC5410: Nonparametric smoothing methods Smoothing splines 23

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

ETC5410: Nonparametric smoothing methods Smoothing splines 23

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

ETC5410: Nonparametric smoothing methods Smoothing splines 23

General cubic splines

Let κ1 < κ2 < · · · < κk be knots in interval (a, b).

Let h1(x) = 1, h2(x) = x , h3(x) = x2, h4(x) = x3,hj(x) = (x − κj−4)3

+ for j = 5, . . . , k + 4.

h1, . . . , hk+4 form a basis for the set of cubic splines atthese knots, called the truncated power basis.

Any cubic spline r(x) with these knots can be written as

r(x) =k+4∑j=1

βjhj(x).

A cubic smoothing spline is obtained by setting κj = xj ,j = 1, . . . , n.

ETC5410: Nonparametric smoothing methods Smoothing splines 24

Matrix form

Let Bij = hj(xi) be the basis matrix. Then forsmoothing splines we need to minimize

(Y − Bβ)′(Y − Bβ) + λβ′Ωβ

where

Ωjk =

∫h′′j (x)h′′k(x)dx .

This gives β = (B ′B + λΩ)−1B ′Y.

ETC5410: Nonparametric smoothing methods Smoothing splines 24

Matrix form

Let Bij = hj(xi) be the basis matrix. Then forsmoothing splines we need to minimize

(Y − Bβ)′(Y − Bβ) + λβ′Ωβ

where

Ωjk =

∫h′′j (x)h′′k(x)dx .

This gives β = (B ′B + λΩ)−1B ′Y.

ETC5410: Nonparametric smoothing methods Smoothing splines 25

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.

r = SY where r = [r(x1), . . . , r(xn)]¿

The effect of λΩ is to shrink the regressioncoefficients to give a smoother fit.

Because this is in the form of a linearsmoother, the inference derived for localpolynomial smoothing can be applied here too.

df = trace(S)

ETC5410: Nonparametric smoothing methods Smoothing splines 25

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.

r = SY where r = [r(x1), . . . , r(xn)]¿

The effect of λΩ is to shrink the regressioncoefficients to give a smoother fit.

Because this is in the form of a linearsmoother, the inference derived for localpolynomial smoothing can be applied here too.

df = trace(S)

ETC5410: Nonparametric smoothing methods Smoothing splines 25

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.

r = SY where r = [r(x1), . . . , r(xn)]¿

The effect of λΩ is to shrink the regressioncoefficients to give a smoother fit.

Because this is in the form of a linearsmoother, the inference derived for localpolynomial smoothing can be applied here too.

df = trace(S)

ETC5410: Nonparametric smoothing methods Smoothing splines 25

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.

r = SY where r = [r(x1), . . . , r(xn)]¿

The effect of λΩ is to shrink the regressioncoefficients to give a smoother fit.

Because this is in the form of a linearsmoother, the inference derived for localpolynomial smoothing can be applied here too.

df = trace(S)

ETC5410: Nonparametric smoothing methods Smoothing splines 25

Inference

Any cubic smoothing spline can be written as alinear smoother with smoothing matrixS = B(B ′B + λΩ)−1B ′.

r = SY where r = [r(x1), . . . , r(xn)]¿

The effect of λΩ is to shrink the regressioncoefficients to give a smoother fit.

Because this is in the form of a linearsmoother, the inference derived for localpolynomial smoothing can be applied here too.

df = trace(S)

ETC5410: Nonparametric smoothing methods Regression splines 26

Outline

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Regression splines 27

Regression splines

Rather than have knots at each data point(κi = xi), use fewer knots.

Then do ordinary linear regression on the basisfunction.

Choice of knots can be difficult and arbitrary.

Automatic knot selection algorithms very slow.

β = (B ′B)−1B ′Y.

ETC5410: Nonparametric smoothing methods Penalized regression splines 28

Outline

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Penalized regression splines 29

Penalized spline regression

r(x) =K∑

j=1

βjhj(x).

Idea: retain all the knots but constrain theirinfluence by ∑

β2j < C .

Let

D =

[02×2 02×K

0K×2 IK×K

].

Then we want tominimize ‖y − Bβ‖2 subject to β′Dβ ≤ C .

ETC5410: Nonparametric smoothing methods Penalized regression splines 29

Penalized spline regression

r(x) =K∑

j=1

βjhj(x).

Idea: retain all the knots but constrain theirinfluence by ∑

β2j < C .

Let

D =

[02×2 02×K

0K×2 IK×K

].

Then we want tominimize ‖y − Bβ‖2 subject to β′Dβ ≤ C .

ETC5410: Nonparametric smoothing methods Penalized regression splines 30

Penalized regression splines

A Lagrange multiplier argument shows that this isequivalent to minimizing

‖y − Bβ‖2 + λ2β′Dβ

for some number λ ≥ 0.

Solution: βλ = (X′X + λ2D)−1X′y.

Fitted values: y = X(X′X + λ2D)−1X′y.

A type of ridge regression.

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

ship

men

ts

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 10

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 40

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 70

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 100

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 130

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 160

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 190

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 220

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 250

ETC5410: Nonparametric smoothing methods Penalized regression splines 31

Penalized regression splines

500 550 600 650 700 750 800 850

010

2030

40

price

lambda = 280

ETC5410: Nonparametric smoothing methods Penalized regression splines 32

Penalized regression splines

Implementation in R

require(SemiPar)

fit <- spm(shipments ∼ f(price))

plot(fit)

points(price,shipments)

For lots of examples and an introduction to thetheory:http://www.uow.edu.au/∼mwand/SPmanu.pdf

ETC5410: Nonparametric smoothing methods Penalized regression splines 32

Penalized regression splines

Implementation in R

require(SemiPar)

fit <- spm(shipments ∼ f(price))

plot(fit)

points(price,shipments)

For lots of examples and an introduction to thetheory:http://www.uow.edu.au/∼mwand/SPmanu.pdf

ETC5410: Nonparametric smoothing methods Other bases 33

Outline

1 Interpolating splines

2 Smoothing splines

3 Regression splines

4 Penalized regression splines

5 Other bases

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivatives

In penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

500 600 700 800 900

0.0

0.2

0.4

0.6

0.8

1.0

Truncated linear spline basis

x

h(x)

ETC5410: Nonparametric smoothing methods Other bases 34

Spline bases

Truncated power basis of degree p1, x , . . . , xp, (x − κ1)p

+, . . . , (x − κK )p+

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

500 600 700 800 900

0.0

0.2

0.4

0.6

0.8

1.0

Truncated cubic spline basis

x

h(x)

ETC5410: Nonparametric smoothing methods Other bases 35

Spline bases

B-splines

Equivalent to truncated power bases but withmore stable properties.

ETC5410: Nonparametric smoothing methods Other bases 35

Spline bases

B-splines

Equivalent to truncated power bases but withmore stable properties.

500 600 700 800 900

0.0

0.2

0.4

0.6

0.8

1.0

Cubic B−spline basis

x

h(x)

ETC5410: Nonparametric smoothing methods Other bases 36

Spline bases

Radial basis functions of degree p1, x , . . . , xp, |x − κ1|p, . . . , |x − κK |p

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 36

Spline bases

Radial basis functions of degree p1, x , . . . , xp, |x − κ1|p, . . . , |x − κK |p

p − 1 continuous derivatives

In penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 36

Spline bases

Radial basis functions of degree p1, x , . . . , xp, |x − κ1|p, . . . , |x − κK |p

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 36

Spline bases

Radial basis functions of degree p1, x , . . . , xp, |x − κ1|p, . . . , |x − κK |p

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

ETC5410: Nonparametric smoothing methods Other bases 36

Spline bases

Radial basis functions of degree p1, x , . . . , xp, |x − κ1|p, . . . , |x − κK |p

p − 1 continuous derivativesIn penalized regression splines, none of thepolynomial coefficients is penalized.

500 600 700 800 900

0.0

0.2

0.4

0.6

0.8

1.0

x

h(x)