Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Models for RegressionGreg Mori - CMPT 419/726
Bishop PRML Ch. 3
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Outline
Regression
Linear Basis Function Models
Loss Functions for Regression
Finding Optimal Weights
Regularization
Bayesian Linear Regression
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Regression
• Given training set {(x1, t1), . . . , (xN , tN)}• ti is continuous: regression• For now, assume ti ∈ R, xi ∈ RD
• E.g. ti is stock price, xi contains company profit, debt, cashflow, gross sales, number of spam emails sent, . . .
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Functions
• A function f (·) is linear if
f (αu + βv) = αf (u) + βf (v)
• Linear functions will lead to simple algorithms, so let’s seewhat we can do with them
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Regression
• Simplest linear model for regression
y(x,w) = w0 + w1x1 + w2x2 + . . .+ wDxD
• Remember, we’re learning w• Set w so that y(x,w) aligns with target value in training data
• This is a very simple model, limited in what it can do
�
�
�����
0 1
−1
0
1
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Basis Function Models
• Simplest linear model
y(x,w) = w0 + w1x1 + w2x2 + . . .+ wDxD
was linear in x (∗) and w• Linear in w is what will be important for simple algorithms• Extend to include fixed non-linear functions of data
y(x,w) = w0 + w1φ1(x) + w2φ2(x) + . . .+ wM−1φM−1(x)
• Linear combinations of these basis functions also linear inparameters
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Basis Function Models
• Bias parameter allows fixed offset in data
y(x,w) = w0︸︷︷︸bias
+w1φ1(x) + w2φ2(x) + . . .+ wM−1φM−1(x)
• Think of simple 1-D x:
y(x,w) = w0︸︷︷︸intercept
+ w1︸︷︷︸slope
x
• For notational convenience, define φ0(x) = 1:
y(x,w) =M−1∑j=0
wjφj(x) = wTφ(x)
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Linear Basis Function Models
• Function for regression y(x,w) is non-linear function of x,but linear in w:
y(x,w) =M−1∑j=0
wjφj(x) = wTφ(x)
• Polynomial regression is an example of this• Order M polynomial regression, φj(x) =?
• φj(x) = xj:
y(x,w) = w0x0 + w1x1 + . . .+ wMxM
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Basis Functions: Feature Functions
• Often we extract features from x• An intuitve way to think of φj(x) is as feature functions
• E.g. Automatic CMPT726 project report grading system• x is text of report: In this project we apply thealgorithm of Mori [2] to recognizing blueobjects. We test this algorithm onpictures of you and I from my holiday photocollection. ...
• φ1(x) is count of occurrences of Mori [
• φ2(x) is count of occurrences of of you and I
• Regression grade y(x,w) = 20φ1(x)− 10φ2(x)
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Other Non-linear Basis Functions
−1 0 1−1
−0.5
0
0.5
1
−1 0 10
0.25
0.5
0.75
1
−1 0 10
0.25
0.5
0.75
1
• Polynomial φj(x) = xj
• Gaussians φj(x) = exp{− (x−µj)2
2s2 }• Sigmoidal φj(x) = 1
1+exp((µj−x)/s)
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Example - Gaussian Basis Functions: Temperature
• Use Gaussian basis functions, regression on temperature
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Example - Gaussian Basis Functions: Temperature
• µ1 = Vancouver, µ2 = San Francisco, µ3 = Oakland
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Example - Gaussian Basis Functions: Temperature
• µ1 = Vancouver, µ2 = San Francisco, µ3 = Oakland• Temperature in x = Seattle? y(x,w) =
w1 exp{− (x−µ1)2
2s2 }+ w2 exp{− (x−µ2)2
2s2 }+ w3 exp{− (x−µ3)2
2s2 }
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Example - Gaussian Basis Functions: Temperature
• µ1 = Vancouver, µ2 = San Francisco, µ3 = Oakland• Temperature in x = Seattle? y(x,w) =
w1 exp{− (x−µ1)2
2s2 }+ w2 exp{− (x−µ2)2
2s2 }+ w3 exp{− (x−µ3)2
2s2 }• Compute distances to all µ, y(x,w) ≈ w1
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Example - Gaussian Basis Functions: 726 ReportGrading
• Define:• µ1 = Crime and Punishment• µ2 = Animal Farm• µ3 = Some paper by Mori
• Learn weights:• w1 = ?• w2 = ?• w3 = ?
• Grade a project report x:• Measure similarity of x to each µ, Gaussian, with weights:
y(x,w) =w1 exp{− (x−µ1)
2
2s2 }+ w2 exp{− (x−µ2)2
2s2 }+ w3 exp{− (x−µ3)2
2s2 }• The Gaussian basis function models end up similar to
template matching
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Loss Functions for Regression
• We want to find the “best” set of coefficients w• Recall, one way to define “best” was minimizing squared
error:
E(w) =12
N∑n=1
{y(xn,w)− tn}2
• We will now look at another way, based on maximumlikelihood
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Gaussian Noise Model for Regression
• We are provided with a training set {(xi, ti)}• Let’s assume t arises from a deterministic function plus
Gassian distributed (with precision β) noise:
t = y(x,w) + ε
• The probability of observing a target value t is then:
p(t|x,w, β) = N (t|y(x,w), β−1)
• Notation: N (x|µ, σ2); x drawn from Gaussian with mean µ,variance σ2
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Maximum Likelihood for Regression• The likelihood of data t = {ti} using this Gaussian noise
model is:
p(t|w, β) =N∏
n=1
N (tn|wTφ(xn), β−1)
• The log-likelihood is:
ln p(t|w, β) = lnN∏
n=1
√β√2π
exp(−β2(tn − wTφ(xn))
2)
=N2
lnβ − N2
ln(2π)︸ ︷︷ ︸const. wrt w
−β 12
N∑n=1
(tn − wTφ(xn))2
︸ ︷︷ ︸squared error
• Sum of squared errors is maximum likelihood under aGaussian noise model
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Finding Optimal Weights
• How do we maximize likelihood wrt w (or minimize squarederror)?
• Take gradient of log-likelihood wrt w:
∂
∂wiln p(t|w, β) = β
N∑n=1
(tn − wTφ(xn))φi(xn)
• In vector form:
∇ ln p(t|w, β) = βN∑
n=1
(tn − wTφ(xn))φ(xn)T
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Finding Optimal Weights• Set gradient to 0:
0T =N∑
n=1
tnφ(xn)T − wT
(N∑
n=1
φ(xn)φ(xn)T
)
• Maximum likelihood estimate for w:
wML =(ΦTΦ
)−1ΦT t
Φ =
φ0(x1) φ1(x1) . . . φM−1(x1)φ0(x2) φ1(x2) . . . φM−1(x2)
......
. . ....
φ0(xN) φ1(xN) . . . φM−1(xN)
• Φ† =
(ΦTΦ
)−1ΦT known as the pseudo-inverse
(numpy.linalg.pinv in python)
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Geometry of Least Squares
St
yϕ1
ϕ2
• t = (t1, . . . , tN) is the target value vector• S is space spanned by ϕj = (φj(x1), . . . , φj(xN))
• Solution y lies in S• Least squares solution is orthogonal projection of t onto S• Can verify this by looking at y = ΦwML = ΦΦ†t = Pt
• P2 = P, P = PT
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Sequential Learning
• In practice N might be huge, or data might arrive online• Can use a gradient descent method:
• Start with initial guess for w• Update by taking a step in gradient direction ∇E of error
function• Modify to use stochastic / sequential gradient descent:
• If error function E =∑
n En (e.g. least squares)• Update by taking a step in gradient direction ∇En for one
example• Details about step size are important – decrease step size
at the end
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Regularization
• Last week we discussed regularization as a technique toavoid over-fitting:
E(w) =12
N∑n=1
{y(xn,w)− tn}2 +λ
2||w||2︸ ︷︷ ︸
regularizer
• Next on the menu:• Other regularlizers• Bayesian learning and quadratic regularizer
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Other Regularizers
q = 0.5 q = 1 q = 2 q = 4
• Can use different norms for regularizer:
E(w) =12
N∑n=1
{y(xn,w)− tn}2 +λ
2
M∑j=1
|wj|q
• e.g. q = 2 – ridge regression• e.g. q = 1 – lasso• math is easiest with ridge regression
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Optimization with a Quadratic Regularizer
• With q = 2, total error still a nice quadratic:
E(w) =12
N∑n=1
{y(xn,w)− tn}2 +λ
2wTw
• Calculus ...w = (λI +ΦTΦ︸ ︷︷ ︸
regularlized
)−1ΦT t
• Similar to unregularlized least squares• Advantage (λI +ΦTΦ) is well conditioned so inversion is
stable
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Ridge Regression vs. Lasso
w1
w2
w?
w1
w2
w?
• Ridge regression aka parameter shrinkage• Weights w shrink back towards origin
• Lasso leads to sparse models• Components of w tend to 0 with large λ (strong
regularization)• Intuitively, once minimum achieved at large radius,
minimum is on w1 = 0
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Linear Regression
• Last week we saw an example of a Bayesian approach• Coin tossing - prior on parameters
• We will now do the same for linear regression• Prior on parameter w
• There will turn out to be a connection to regularlization
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Linear Regression
• Start with a prior over parameters w• Conjugate prior is a Gaussian:
p(w) = N (w|0, α−1I)
• This simple form will make math easier; can be done forarbitrary Gaussian too
• Data likelihood, Gaussian model as before:
p(t|x,w, β) = N (t|y(x,w), β−1)
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Linear Regression• Posterior distribution on w:
p(w|t) ∝
(N∏
n=1
p(tn|xn,w, β)
)p(w)
=
[N∏
n=1
√β√2π
exp(−β
2(tn − wTφ(xn))
2)]( α
2π
)M2 exp(−α
2wTw)
• Take the log:
− ln p(w|t) = β
2
N∑n=1
(tn − wTφ(xn))2 +
α
2wTw + const
• L2 regularization is maximum a posteriori (MAP) with aGaussian prior.
• λ = α/β
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Linear Regression - Intuition
• Simple example x, t ∈ R,y(x,w) = w0 + w1x
• Start with Gaussian prior inparameter space
• Samples shown in data space• Receive data points (blue
circles in data space)• Compute likelihood• Posterior is prior (or
prev. posterior) timeslikelihood
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Predictive Distribution
• Single estimate of w (ML or MAP) doesn’t tell whole story• We have a distribution over w, and can use it to make
predictions• Given a new value for x, we can compute a distribution
over t:p(t|t, α, β) =
∫p(t,w|t, α, β)dw
p(t|t, α, β) =∫
p(t|w, β)︸ ︷︷ ︸predict
p(w|t, α, β)︸ ︷︷ ︸probability
dw︸︷︷︸sum
• i.e. For each value of w, let it make a prediction, multiply byits probability, sum over all w
• For arbitrary models as the distributions, this integral maynot be computationally tractable
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Predictive Distribution
�
�
0 1
−1
0
1
�
�
0 1
−1
0
1
�
�
0 1
−1
0
1
• With the Gaussians we’ve used for these distributions, thepredicitve distribution will also be Gaussian
• (math on convolutions of Gaussians)• Green line is true (unobserved) curve, blue data points, red
line is mean, pink one standard deviation• Uncertainty small around data points• Pink region shrinks with more data
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Model Selection
• So what do the Bayesians say about model selection?• Model selection is choosing modelMi e.g. degree of
polynomial, type of basis function φ
• Don’t select, just integrate
p(t|x,D) =L∑
i=1
p(t|x,Mi,D)︸ ︷︷ ︸predictive dist.
p(Mi|D)
• Average together the results of all models• Could choose most likely model a posteriori p(Mi|D)
• More efficient, approximation
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Bayesian Model Selection
• How do we compute the posterior over models?
p(Mi|D) ∝ p(D|Mi)p(Mi)
• Another likelihood + prior combination• Likelihood:
p(D|Mi) =
∫p(D|w,Mi)p(w|Mi)dw
Regression Linear Basis Function Models Loss Functions for Regression Finding Optimal Weights Regularization Bayesian Linear Regression
Conclusion
• Readings: Ch. 3.1, 3.1.1-3.1.4, 3.3.1-3.3.2, 3.4• Linear Models for Regression
• Linear combination of (non-linear) basis functions• Fitting parameters of regression model
• Least squares• Maximum likelihood (can be = least squares)
• Controlling over-fitting• Regularization• Bayesian, use prior (can be = regularization)
• Model selection• Cross-validation (use held-out data)• Bayesian (use model evidence, likelihood)