Prediction of House Prices- Multiple Linear Regression -

P5

Group G4-101a

Aalborg UniversityMathematics-Economics

Copyright c© Aalborg University 2016

Mathematics-EconomicsAalborg University

http://www.aau.dk

Title:Prediction of House Prices

Theme:Statistical Modelling and Analysis

Project Period:Fall Semester 2016

Project Group:G4-101a

Participants:Ann-Katrine Kjærsgaard NielsenPernille Lisbeth HaugstrupPernille Krog PoulsenSebastian StrandbergSilke Cecilie Fich Fisker

Supervisor:Matt Dziubinski

Copies: 8

Page Count: 87

Date of Completion:April 26, 2017

Abstract:

In this project we build a model to pre-dict house prices using multiple linearregression. We realise that there al-ways is a trade-off between bias andvariance in a model. We find thatthe best method to build a multiplelinear regression model is elastic netwhich improves upon the ordinaryleast squares, ridge regression, andLasso. The model is trained and testedon data from the real estate agencyhome. The model we get using elas-tic net has a root mean squared errorof 524, 813 DKK (41.89%) when testedon the validation set. When examin-ing learning curves, we see that thepredictive model has high bias. There-fore, we create two new variables us-ing feature engineering and find thatthe model predicts better with the twonew variables, since this model hasa root mean squared error of 521, 278DKK (41.61%).

The content of this report is freely available, but publication (with reference) may only be pursued due to

agreement with the author.

Contents

Preface vii

1 Introduction 1

2 Exploratory Data Analysis 32.1 Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Multiple Linear Regression 113.1 Ordinary Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Mean Squared Error 234.1 The Bias-Variance Trade-Off . . . . . . . . . . . . . . . . . . . . . . . . 24

5 Resampling Methods 315.1 The Validation Set Approach . . . . . . . . . . . . . . . . . . . . . . . 315.2 Leave-One-Out Cross-Validation . . . . . . . . . . . . . . . . . . . . . 325.3 k-Fold Cross-Validation . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 Model Selection 356.1 Choosing the Optimal Model . . . . . . . . . . . . . . . . . . . . . . . 356.2 Subset Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.3 Shrinkage Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

7 Maximum Likelihood Estimation 437.1 Log Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

8 Hypothesis Testing 478.1 Confidence Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498.2 Prediction Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

9 Application 519.1 Initial Data Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . 51

v

vi Contents

9.2 Ridge Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529.3 Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559.4 Comparing Ridge Regression and Lasso . . . . . . . . . . . . . . . . . 579.5 Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589.6 Model Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629.7 Improving the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

10 Summary 69

11 Critique 71

Bibliography 73

A Implementation in R 75A.1 Cleaning the Data For Modelling . . . . . . . . . . . . . . . . . . . . . 75A.2 Ridge Regression Implementation . . . . . . . . . . . . . . . . . . . . 78A.3 Lasso Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 80A.4 Elastic Net Implementation . . . . . . . . . . . . . . . . . . . . . . . . 81A.5 Validation Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83A.6 Learning Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84A.7 Feature Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

B From Log of Cash Price To Cash Price 87

Preface

This project has been compiled in the fifth semester by group G4-101a in the timefrom September 1, 2016 to April 26, 2017. The Chicago method is used for refer-ences. Proofs are ended with �. We want to thank our adviser Matt Dziubinski forthe dedicated help.

Aalborg University, April 26, 2017

Ann-Katrine Kjærsgaard Nielsen<annkni14@student.aau.dk>

Pernille Lisbeth Haugstrup<phaugs14@student.aau.dk>

Pernille Krog Poulsen<ppouls14@student.aau.dk>

Sebastian Strandberg<sstran14@student.aau.dk>

Silke Cecilie Fich Fisker<scff14@student.aau.dk>

vii

Chapter 1

Introduction

This chapter is based on [James et al., 2015, pp. 15-19, 24-16].

In this project we create a model to predict output given a set of input variables.The output variable is denoted Y and is referred to as the dependent variable. Theinput variable is denoted X and is referred to as the independent variable. Wewant to examine the relationship between Y and X, which can be written as thegeneral model

Y = f (X) + ε,

where f is an unknown Data Generating Process (DGP), that can contain more thanone independent variable, and ε is a random error term with mean zero. Since fdescribes the relation between Y and X we want to estimate this function. Themain objective is to use the estimated function to accurately predict Y. The outputvariable Y can be predicted by

Y = f (X),

where Y and f respectively is the prediction of Y and the estimated function. Theaccuracy of this prediction depends on a reducible error and an irreducible error.The reducible error originates from f since it is unlikely to perfectly estimate theDGP. This error can be reduced by choosing the appropriate statistical model. Theirreducible error comes from Y also being a function of ε and this error can not bereduced since ε is independent of X.

We are interested in finding the independent variables that are associated with thedependent variable. Furthermore we want to understand how Y is influenced byX as the input variables change. This is called inference. The influence can beeither negative or positive. It could also be useful to know the relation between theindependent variables. The last thing to explore is whether the relation between Yand the independent variables can be described as a linear model or whether the

1

2 Chapter 1. Introduction

relation is more complicated. These things are examined throughout the project,including a prediction of Y.

When building a model there is a trade-off between prediction accuracy and modelinterpretability. Flexibility is a measurement of the prediction accuracy. A highlyflexible approach creates a wide range of shapes to approximate f , whereas a lessflexible approach produce a small range of shapes. High flexibility can provide abetter prediction accuracy, but can also result in estimates of f where it is difficultto understand how the independent variables are associated with the dependentvariable. A model with a high flexibility can also lead to overfitting the data. Thismeans that the model captures patterns in the data that are only present due to therandom error.

If we are interested in inference, a parsimonious model is preferred, since a restric-tive model is more interpretable.

In this project we use linear regression to predict Y. Linear regression is a relativelyinflexible approach since it only generates linear functions. However in linearregression the relation between Y and each of the independent variables is moreinterpretable due to linearity in the parameters.

We want to predict the cash price of a house based upon the features of the house.The variable Y is the cash price and the independent variables are the features ofthe house. To do this we use a data set from the real estate company home with8,250 observations and 31 variables.

To predict Y, we use a model to select the independent variables that have aninfluence on the cash price. Thereby we obtain a model we can use to predict thecash price.

The methods we can use to find this model are best subset selection, stepwiseselection and shrinkage methods (namely ridge regression, Lasso, and elastic net).To determine which model gives the best prediction we use root mean squarederror, which is a measurement of how accurately the models estimate y.

We implement the methods in the programming language R, but we do not de-scribe programming theory.

The limitations stated above lead us to the main problem in the project.

Problem Statement

Can we predict house prices using a multiple linear regression model? If so, whichmethod gives the model that predicts the best, and how accurately does it predict?

Chapter 2

Exploratory Data Analysis

The given data from home consists of 31 variables:

Age The age of the residence, in years, reset when a resi-dence is renovated.

Balcony Yes (1) or no (0).Basement Yes (1) or no (0).Basement size Basement size in square metres.Cash price Price in danish kroner (DKK).Close to water Yes (1) or no (0).Date of sale The date the residence was sold.Full address The full address of the residence which includes street

name and number.Garage/Carport Yes (1) or no (0).High-rise (1) if more than 14 storeys, (0) otherwise.Large Lot (1) if lot size is larger than 1,500 square metres, (0) oth-

erwise.Living space Living space in square metres.Lot size Lot size in square metres.Many toilets (1) if number of toilets is larger than two, (0) otherwise.New bathroom (1) if new bathroom after 2005, (0) otherwise.New kitchen (1) if new kitchen after 2005, (0) otherwise.New windows (1) if new windows after 2005, (0) otherwise.Number of bedrooms Number of rooms besides a living room.Number of storeys The number of storeys in the residence.Number of toilets The number of toilets in the residence.Quarter 1 (1) if sold in first quarter, (0) otherwise.Quarter 2 (1) if sold in second quarter, (0) otherwise.

3

4 Chapter 2. Exploratory Data Analysis

Quarter 3 (1) if sold in third quarter, (0) otherwise.Quarter 4 (1) if sold in fourth quarter, (0) otherwise.Renovation (1) if renovated, (0) otherwise.State of residence "Good", "Medium" or "Bad".Turnover time Number of days the house has been on the market.Type of residence "Condo", "Terrace house", "Villa, 1 family" or "Villa, 2

families".Year of construction The year the residence was constructed.Year of sale The year of sale spans from 2004 to 2016.Zip code Residences with zip code 9200 to 9990 (Northern Jut-

land).

The original data set consists of 8,250 observations and 31 variables. We choose tofocus on residences of type "Villa, 1 family" since this type of residence contains86.55% of the observations in the data. Thus all other types are discarded, whichgives us 7, 140 observations.

We filter the data by deleting "Many toilets" and "Number of toilets" because18.14% of the villas have zero toilets, which seems to be a PASA error, since everyvilla should have a minimum of one toilet.

In addition, we delete "New bathroom", "New kitchen", and "New windows" dueto many numbers that are not available (NAs) in the data, respectively 93.61%,93.38%, and 95.71%.

There are 23 observations with lot size zero, which are all discarded.

We change 15 and 25 storeys to, respectively, 1.5 and 2.5 storeys, since it is unlikelyto have a villa with 15 or 25 storeys. Since the villa with the most storeys has 5and a high-rise per definition consist of 14 storeys or more, we delete the variable"High-rise". This leaves us with 7, 117 observations and 25 variables.

2.1. Plots 5

2.1 Plots

0e+00

2e+06

4e+06

6e+06

8e+06

0 50 100 150 200Age

Cas

h pr

ice

50100150200

count

(a)

12

14

16

0 2 4Log of age

Log

of c

ash

pric

e

4080120160

count

(b)Figure 2.1: (a): Age against cash price. (b): Log of age against log of cash price.

In Figure 2.1 we see a dependency between age and cash price. The older villashave lower cash prices and the younger villas have higher cash prices. We can seethat the majority of the villas have been constructed or renovated within the last 50years. We have plotted log of age against log of cash price to better see the negativelinear dependency between these two variables. The Spearman correlation (takingnonlinearities into account) between age and cash price is 0.35, as seen in Table 2.2.This confirms that there is some dependency between the variables.

0e+00

2e+06

4e+06

6e+06

8e+06

0 200 400Living space

Cas

h pr

ice

100

200

count

(a)

12

14

16

3 4 5 6Log of living space

Log

of c

ash

pric

e

50100150200

count

(b)Figure 2.2: (a): Living space against cash price. (b): Log of living space against log of cash price.

Figure 2.2 shows a linear dependency between living space and cash price. Thisis confirmed by a Spearman correlation in Table 2.2 of 0.48. We see this lineardependency more clearly when we plot the log of living space against log of cashprice. The positive linear dependency means that a large living space equals a highcash price.

6 Chapter 2. Exploratory Data Analysis

0e+00

2e+06

4e+06

6e+06

8e+06

5 10Number of bedrooms

Cas

h pr

ice

100200300400

count

(a)

11

12

13

14

15

16

0 1 2Log of number of bedrooms

Log

of c

ash

pric

e

100

200

300count

(b)Figure 2.3: (a): Number of bedrooms against cash price. (b): Log of number of bedrooms against logof cash price.

Figure 2.3 shows the relation between number of bedrooms and the correspondingcash price of the villa. We see that as the amount of bedrooms increases so doesthe cash price. This positive linear dependency is seen more clearly when the logof number of bedrooms is plotted against log of cash price. The dependency is alsoseen in the Spearman correlation, which is 0.46, see Table 2.2.

0e+00

2e+06

4e+06

6e+06

8e+06

0 500 1000 1500Turnover time

Cas

h pr

ice

100200300400500

count

Figure 2.4: Turnover time against cash price.

In Figure 2.4 we see that the longer a villa has been on the market the lower thecash price. However, in Table 2.2 the Spearman correlation of 0.02 suggests thatthere is no dependency between these two variables. Most villas are sold within500 days.

2.1. Plots 7

0e+00

2e+06

4e+06

6e+06

8e+06

1800 1850 1900 1950 2000Year of construction

Cas

h pr

ice

100

200

count

Figure 2.5: Year of construction against cash price.

In Figure 2.5 we see that the cash price has a positive linear dependency on the ageof the villa. The Spearman correlation is 0.40 in Table 2.2, which also indicates adependency. We also see that the majority of villas are constructed after the 20thcentury.

0e+00

2e+06

4e+06

6e+06

8e+06

9200 9400 9600 9800 10000Zip code

Cas

h pr

ice

50

100

150

count

Figure 2.6: Zip code against cash price.

In Figure 2.6 we see that the price at which the greatest amount of villas is sold isapproximately the same in every zip code. Hereby there seems to be some kindof positive linear dependency, which is consistent with a Spearman correlation of0.31 in Table 2.2.

8 Chapter 2. Exploratory Data Analysis

Cash price vs. independent variable Spearman correlation

Age 0.35Living space 0.48Number of bedrooms 0.46Turnover time 0.02Year of construction 0.40Zip code 0.31

Table 2.2: The Spearman correlation between cash price and the six chosen independent variables.

Living space

0.0 0.5 1.0 1.5 2.0 2.5

0.65 −0.26

1800 1900 2000

0.14

34

56

−0.092

0.0

1.0

2.0

●●●●

●●●

●●●

●

●●

●

●

●●

●

● ●

●

●

●●●●

●

●

●●

●● ●●●

●

●●● ●

●

●●● ●●

●

●

● ●

●

● ●●●●● ● ●

●

●

● ●●●

●● ●●●

●●

●

●

●

●●

●● ●

●

●●

●●●●

●

●●●●

●● ●

●●

●

●

●●●

● ●●●●

●

●●●

●●

●●●●● ●

●●

●●

●

●●●●

●

●

●

●●●

●●

●●●

●

●

●●●● ●

●●●

●●●●● ●●

●

● ●●●

●

●

●●

●●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●

●●

●

●

●●●●

●●

●

●

●

●

●

●

●

●● ●

●

●

●●

●

●

●●●

●

●●●

●

●

●

●

●●●

●

●●

●

●●●

●

●

●

●●●

●

●

●

●

●

●●●●

●

●

●●●●●

●

●

●●

●

●

●

●●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●● ●

●

●

●

●

●●

●●

●●

●●●●

●

●●

●

● ●

●●

●●

●●

●

●●●●● ●

●●●●●●●

●

●●

●

●

●

●●●

●●

● ●●

●●

●

●●

●●●●

●●●

●

●

●●

●

●

●

●●

●

●●

●

●●

●

●

●●

●●

●●

●

● ●

●

● ●●

●

●●

●

●

●●

●●●

●

●

●●●●

●

●

●●● ●●

●

●●

●

●●

●●●

●●● ●

●

●●

●●●

●

●●

●

●

●

●●

●

●●

●

●●

●●●●

●●●●

●

●

● ●

●●

● ●●●●

●

●

●●

●

●

●

●

● ●

●●●

●

●●●●●●

●

●

●●

●

●

●

●

●

●

●

●

● ●●●●●

●●

●

●

●●

●●●● ●●●

●

●

●●●●●●

●

●●●● ●●

●

●

●●●

●

●

●●●●

●

● ●●

●

●

●● ●●

●

●

●

●●●

●

●

●● ●●●● ●●●

●●

●● ●

●●●

●●

●

●

●●

●●

●

●

● ●●

●

●

●

●

●●

●●● ●●●●

●

●●

●● ● ●●

●●

●●

●

●●●●

●

●●●

●●

●●●

● ●●

●●

●●●●

●●●

●

●●●

●●

●

●

●● ●

●

●

●

●●●●

●●

●●●

●●●

●●●

●

●●●

● ●●●●

●

●●●

●●

●●●

●● ●

●●●●●●

●

●

●

●

●●●●●●●

●

●●●●

●

●

●

●

●●●●

●●●

●

●●

●

●●

●

●

●

● ●

●

●

●

●

●●

●●●

●●

●●● ●

●

●●

●● ●

● ●●

●●●●●

●

●

●●

●

●

●

●● ●●

●

●●

●●●

●

●●●

●

● ●●●

●

●●●

●

●●

●●● ●●

●

●

●●

●

●●

●

●● ●●●●

●

●

●●●●

●●●●

●●●●●● ●

●

●

●

●●●

●

●

●

●●●●● ●

●

●●●● ●

●

●

●●● ●●●●●

●

●●

●●

●

● ●●●●

●

●

●●● ●● ●

● ●●●●

●●●●

●●

●●●

●●

●

●

●●

●

●

●●

●●

●●●●●

●●

●

●

●● ●

●

●

●●

●

● ●

●

●

●

●●

●●

●

●

●

● ●

●

●●

●

●

●

●

●●●

●●●

●● ●

●

●

●

●

●●

●●●

●●

●

●●●●

●

●●

●

●●●

● ● ●

●

●

●

●

●●●

●●

●

●●●●

●

●

●

●●●●

●●● ●●●

●

●

●

●●

●

●●

● ●●●● ●

●●●●●● ●

●●

●

●●

●●●

● ●

●

●

●

●

●

●●●●

●

●●

●

●●●●●● ●

●

●●●●●

●●●●●●●●

●

●●

●

●

●

●●

●●

●

●

●●

●●

● ●●●●

● ●●●● ●

●●

●

●●

●●

●

●●● ●

●

●●

●●

● ●●

●

●●●

●●

●●●●

●

●

●●

●

●●

●●

●

●

●●

●●●●●

●●

●

●

●

●●●

●●

●● ●●

●●

● ●●

●●●

●●

●

●

●

●●●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●

●

●

●●

●●

●

●●●

●●

●

●

●

●●

●●●●

●

●

●

●●

●

●

●●

●●●

●

● ●

●

●

●

●

●●●●

●

●

●

●

●

● ●●●

●

● ●●

●●●

● ●

●●

●●●●

●

●

●

●● ●●

●

●

●●●

●●

● ●●

●●●

●●

● ●●

●●●

●●

●●

●●●

●

●

●● ●●

●

●●●●

●●●

●

●●

●●

●

●●

●

●●

●

●●

●

●●● ●●

● ●●●

●

●●

●●

●

●●

●●

●●

●●● ●●

●

●

●●

●

●●

●

●

●

●

●●●●●

●●

● ●

●●●●●●

●●

●

●

●

●●

●●

● ●●●●

●

●

●

●●

●

●

●

●●

●●●

●

●

●●

●

● ●●●

●

●

●● ●●

●

●●●●

●

●●

●●● ● ●

●

●● ●●●●

●●

●●●

●

●● ●

●

●

●● ●

●

●●

●●

●

●

●●● ●●●

●

●●

●

●

●

●●●

● ●

●

●

●

●● ●●●●

● ●

●

●●●●●

●●

●●

●

●

●●● ●

●

●●

●

●

●

●

●

●

●

●●● ●●

●●●

●●●●●●

●

●

●●

●

●●

●

● ●●

●

●●

●

●● ●

●

●

●

●●

●●

●

●●●

●

●

●

●

●

●● ●● ●

● ●●

●

●

●

●●●

●●●

●●●

●●●

●

●

●●

●

●●●●

●●●

●

● ●

●

●●●

● ●●

● ●

●

●

●

●

●

●

●

●●

●●●●●

●●

●●●

●

●●

●

●●●

●

●

●

●●●●●● ●● ●●● ●

●●●

●●

●

●

●●

●●●● ●●

●●

●● ●

●

●●

●

●

●

●●●

●

●

● ●● ●●● ●●●

●

●● ●●

●●

●

● ●

●

●●

● ●●

●● ●●●

●●

●●

●

●●●

●●●●●

●●

●

●●●

●● ●

●

● ●

●

●●● ●

●●●

●

●●

●

●

●● ●●●

● ●

●●●●

●●●

●

●●● ●●● ●●●

●

●

●

●●●

●●

●

●●●

●●

●

●●

●

●

●●

●

●●

●

●●

●●

●

●

●

●● ●

●●

●●●

●

●

●

●

●

●

●

●

●●

●●

●● ●

● ●●

●●●●

●

●●

●

●●

●

●

●●●

●

●●●●●●●

●●●

●●●

●●●●

●

●●

●

● ●●

●●

●

●

●●

●●●

●

●

● ●

●

●●●●

●

●●

●

●

●

●●

●

●

●●●●●

●

●●

●●●

●

●●

●

●

●●●●

●● ●● ●●●

●●●●●

● ●●

●

●●●●

● ●●●●

●

●

●●

●●

●●

●

●

●●●●●

●

●

●

●

●

●●●●●

●

●●●●

●●●

●●

●●●●●

●●

●●

●●

●

● ●●

●●

●●●●●

●●●

●●

●

●

●●●

●●

●

●

●

●

●

●

●●● ●

●

●

●●●

●

●

●

●

●

●

●

● ●

●

●●

●●●●

●●

●●●

●●●●

●

●●

●

●●

●

●●●●●

●●

●

●

● ●

●●●

●

●

●

●●●

●

●●

●

●

●●●● ●●●

●

●

●●

●

● ●●●●

●●

●

●●●

●

●

●

●

●●

●●

●

● ●●

●

●●●●

● ●

●● ●

● ●

●

●●

●

●

●

●

●

●●

●

● ●

●

●●●

●●

●

●●●

●●●●

●●●

●

●

●●●●

●

●●●

●●

●

●●●

●● ●

●●

●

●

●●

●

●

●

●● ●●

●

●

●

● ●

●●

●

●●●

●●●●

●●

● ●●●

● ●●

●●

●●●

●

●

●

●●●

● ●

●●● ●● ●●

●

●●●

● ● ●●

●●

●●●●●

●

●

●

●

●

●●

●●●

●●

●

●●●

●●

●●●

●

●

●●●●

● ●●

●

●

●●

●●

●

●

●

●●●●

●●

●●

●

●

●

●●●

●●

● ●

●●●

●●

●●

●

●

● ●●

●●

●

●

●

●●

●●

●

●●●

●

●

●●

●●●●

●

●

●

●●●

●

● ●

●

●

●

●

●● ●●

●

●●●

●

●●●

●●

●

●●

●●●●

●

● ●●

●

●

●●●●●

●

●

●

●●

●

●

●

●

●●

●●●●

●

●

● ●●

●

● ●●

●●●

●

●

● ●●● ●

●

●

● ●●

●

●●

●●

●

●● ●● ●●●

●

● ●●●

●●

● ●

●

●

●

●

●●

●●

●●

●●

●●●

●● ●

●

●●

●

●

●

● ● ●

●

●

●●

●

●

●

●

● ●

●

●

●●

●

●●●●

●

●●●●●

●

●

●

●

●●

●●

●

●●

●

●

●

●●●● ●●

●● ●●● ●●●

● ●●

●

●

●●● ●

●

●●

●

●

●●● ●

●

●

●

●●

●

●●●

●●●

●

● ●● ●

● ●●

●

●

●

●

●

●●●●

●●●

●●

●

●

●●●● ●

● ●●

●

●

●●●

●

●●●●●●●● ●●●●

●●

● ●

●● ●

●

●

●●

●●●●

●

●

●●

● ●

●

●●

●●●

●

●●

●

● ●●●

●

●●

●● ●● ●●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

● ●●●

●

●

●● ●

●

●

●

●

●●

●

●●

●

●

●

●●●

●

●●

●

●

●●

●●●

●

● ●

●

●

●● ●

●

●●●

●●

●

●

●●

●● ●●●●●

●●●

●

●●●

●●

●●

●●

●

●

●

●●●●

●

●

●

●●●

●

●

●●

●

●

●● ●

●

●

●

●●●

●

●

●●●

●

●

●

●

●●

●

●●●

●

●

●●

●●●

●●

●●

●

●●●

●

●

●

●●● ●●●

● ●

● ●● ●●●●

●

●

●

●●

● ●●●

●●●

●

●

●

●●●●●

●●

●

●

●● ●

●●●

●●

●●●

●

●

●

●

●●

●●● ●●●

●

●

●●

●

●●

●●

●●

●

●●

●

●

●

●

●●● ●● ●

●●

●

●●●●● ●

●

●

●

●

●

●

●

●

●

●●●●

● ●

●●

●

●●●●

●●

●●●●

●

●●

●● ●

●

●

●

●●● ●

●

●●

●●

●

●●●

●

●

●

●

●●

● ●●

●

● ●●●

●●●●

●

●●

●●

●●●●●

●●●●

●

●

●●●●●

●

●●

●

●

●●● ●

●

●

●●

●

●● ●

●

●

●

●●

●●●●● ●●●●●● ●

●

●●

●

●

● ●● ●●

●

●●

●

●

● ●●

●

●●

●

●●

●

●

●

●●●●

●

●●

●

●

●

●●

●

●

●

●

●

●●●●●●

●●

●

●●● ●

●●

●

●

● ●

●●

●

●

●

●●●

●

●●●●

●

●●●

● ●●

●

●●●

●

●

●

●● ●●

●●

●

●●●

●●

●

●

●●

●●●●●

●

●●

●●● ●

●●

●●● ●

●

●●

●●

●

● ●●

● ●

●

●

●●

●

●

●●

●

●

●

●

●●

● ●● ●

●●

●● ●

●

●

●

●

●●

●

● ●

●

●

●

●●

●●●

●

● ●●

●

●

●●

●

●

●●●

●

●

●●●

●●●

●●●

●

●

●

●

●

●●

●●

●

●

●●

●

● ●

●

●

●

●●

●●

●

●

●

●

● ●

●●

●

●●

●

●●

●●

●●●● ●

●●●

●●

●

●●

●

●

●

●

●●

●●

●●

●

●

●● ●● ●●●● ●

●

●● ●●●●

●●

●

●●●●●

●

● ●●

●●●●●

●

●●

●●

●●●● ●

●

●●●●

●

●

●

●

●

● ●●

●

●● ●●

●

●

●●●

●

●

●● ● ●●

●●

●●●

●●

●

●

●●●

●●

●●

●

●●

●●

●●

●

●

●

●

●●

● ●

●●

●●●●

●

●●

●

●●●

●

●●●●

●● ●●

●

●●● ●

●

●

●

● ●

●●

● ●●●●

●

●

●

●●●●● ●

●

●

●

●● ●

●●

●

●

●

● ●●

●

●

●●●

●●

●●

●

●●

●

●●

●●

●●●

●

●●●

●

●●●

●●

●

●●

●

●●●● ●

● ●

● ●

● ●

●

●

●

●

●●●

●●

●●●

●

●

●

●●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●●●●

●

●

●

●

●●

● ●

●

●●● ●

●

●●●

●●●

●

●●● ●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●●●●

●●●

●

●

●

●●●

● ●●

●

●●●●●

●●●●

●●

●

●●

●

●

●●

●●

●

●

●

●

●●●●●

●●●

●

●●●●

● ●●●

●

●

●●

●

●

●●● ●

●

●

●

●●

●

●●●●

●

●

●

●●●

●

● ●●●

●

●

●

● ●●●● ●●●

●

●

●●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●● ●

●

●

●●●

●●

●●●●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●● ●●

●●

●

●●●

●

●

●●

●●

●● ●

●

●

●

●

●●

●●●

●

●

●

●●

●●●

●●

●

●●●

●●

●

●●

●

●●●●●●

●

●●

●

●●

●●●●

●

●

●

●●●

●

●

●●

●●●●

●

●●

●

●●● ●

●●

●

●●

●

●●

●●●

●

●●

●●

●

●

●●

● ●

●

●

●

●

●●●

●●●

●

●

●

●●

●●●

● ●

●

●●

●

●

●●

●

●

●●

●

●

●●●

●

● ●●

●

● ●

●

●●●

●

●

●

●●●

●●●

●

●●

●

●●

●●

●

●●

●● ●

● ●●

●

●● ●

●● ●●

●●

●●●

●

●

●●

●●

●

●●

●

●

●●

●

●●

●

●● ●

●

●●

●

●

●

●

●

●●

●

●

●●

●●●●●●

●●●

●●

●●●

●●●

●●

●●

●

●●●

●

●

●●

●●●●

●

●

● ●●

●●

●

● ●

●

●●● ●

●

●●

●●

●●

●●

●●

●

●●●● ●●

●●

●

●● ●

●

●●

●●

●●

●

●●●

● ●●●●

●●●●

●

● ●●

●●●

●

●●

●●●

●●

●

●

●

●●●●

●

●●

●

●●

●

●

●

●●

● ●

●●

●●

●●

●●

●

●

●

●●

●

●

●●

●

●

●

●●●

●

●

●●●

●

●●

●

●●

●●

●●●●

●

●●● ●●

●●●

●

●

●

●●●

●

●

●● ●●

●●

●

●

●

●●

●●

●●●●

●

●

●

●●

●●●

●●

●

●●

●

●●

●

●

●

●●

● ●●●●●

●

●●

●●

●

●●

●

●●

●

●

●

●

●●

●

●

●●●

●

●●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●●

●

●●

●

● ●●

●

●●

●

●

●

●

●

●

●

●●●

●●

●●

●●●●●

●

●●●●●●

●

●

●

●

●

● ●

●

●

●●

●●

●

●●●

●

●● ●

●

●● ●

●

●●●●●●

●●

●

●●●●

●

●

●

●

●●●●

●

●

●●

●●

● ●●●

●

●●●

●●●●●●

●

●

●●●●

●

●

●

●

●

●

●

● ●

●●

●

●

●

●

●

●

●● ●

●

●

●

●

●●

●●●●

●●

●●●

●

●●●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●

●●

●

●

●

●●

●

●●●

●

●

●●

●

●

●●●

●

●●● ●●

●

●●

●●

●

●●

●

●

●●

●

●●●

●

●●

●●● ●

●

●

●●

●

●

●●

●

●

●

●●

●●

● ●

●●● ●●

●●

●

●

●● ● ●

●

●

●●●

●

●●

●

●

●●●

●●●

●

●

●●

●

● ●●●

●

●

●●

●●

●

●

●●● ●●●

●●●

●

●

●

● ●●

●●

●

●

●●

●●

●●

●

● ●

●

●

●

●●●

●●●●

● ●

●●●

●●

●

●●

●●

●●

●

●●●●

●

●● ●

●

● ●●

●

●

●

●

● ●

●

●

●

●

●

●

●●

●●●●

●

●

●●

●●●

●

●●

●●

●●●

●●

●

●●●

●

●

●

●●●

●●

●●●

●●

●● ●●

●

●● ●

●●●

●

●

●

●●●

●

● ●●

●

● ●

●●

●●

●●

●

●

●

●

●

●●

●●

●

●●●●●

●●●

●

●

●●● ●

●●

●

●●●

●

●

●●●

●

●● ●

●●

●

●●●

●●●

●

●●

● ●

●

●●

●●

●●●

●

●●●●

●

●●

●

●

●●

●

●● ●

●●

● ●●

●● ●●

●

●●

● ●

●●

●

●

●●

●

●

●●

●

●● ●

●

●

●

●

●

●● ●

● ●

●

●

●

●

●●

●

●●

●● ●

●

●

●

●

●●●

●●

●

●

●

●●●

●●

●

●●●

●

● ●

●●

●●

●

●

●●

●

●

● ●●

●

●

●

●●●

●

Number of bedrooms

−0.26 0.25 −0.072

●

●

●

●

●●●●

●●●●●● ●

●●

●

●

●●

●●●● ●

●

●●●●

●● ●

●

●●●

●

●●

●

●

●

●●●

●●

●●

●●

● ● ●●

●●

●

●

●

●●

●●

●

●●

●

●

●● ●● ●

●●

●●

●

●●

●●

●

●●

●

●●●●● ●●●●

●

●●●

●●

●

●●

●

●

●

●

●●●● ●●

●● ●

●●

●●● ●●

●●

●●

●

●

●

●

●

●

●●

●

●● ●●●

●●

●●●

●●●●

●● ●●

●

●

●●

●

●●●

●●

●

●●●

●

●

●●

●

●●●●

●●●

●●

●

● ●● ●

●

●●●

●

●●●●●

●●●

●●● ●

●

●●

●●●

●

● ●●●

●

●●●

●

●● ●●

● ●●

●●

●

●● ●●

●●

●

●●

●

●●

● ●

●●

●

●

●● ● ●●

●●●●●●

●

●

●●

●

●

● ●

●

●●

●●●

●●

●

●

●

● ●● ●

●

●● ●●

●

●

●●●●

●

●

●●●

●

●●●

●

●● ●●●

●

●●●

●

●

●

●●

●

●

●

●

●

●●

●

● ●●●●

●●● ●

●

●●●

●

●

●●● ●●●

●

●●●

●●

●

●●

●

●

●

●●

●

●

●●●

●●

●●

●

●●

●●

●

●● ●●

●●

●

● ●●●●

●●●

● ●●●

●

●

●●

●●

●

●

●●

●

●●

● ●●

● ●●●

●

●●

●

● ●●●

● ●

●●●

● ●

●

●●

●●

●●

●

●●

●●

●●

●●●

●

● ●● ●

●

● ●●●

●

●

●

●

●

●

●

●

●● ●

●●●●●●

●●

●●

●●

●

●● ●

●

●

●●

●●

●

● ●●● ●●

●●

●●●●●●●●

●

●●●

●

●

●

●

●●

●

●

●

● ●

●

●

●

●

●

●●

●

●

●

●●

● ●

●

●●

●

●●●●

●

●●●

●

●●●

● ●●●●

●

●●

●

● ●

●

●●●●●

●●

●●●

●

●●

●●

●

●●●

●

●

●●●

● ●●

●

●●●

●●

●● ●

●●

●

●

●●●●

●●●

●●

●

●

●●

● ●●

●●●●●

●● ●●

●

● ●●

●●

●●●●●●

●

●

●● ●

●

●●●●●

● ●

●●

●●

●●

●

●

●●

●

●● ●

●

●●

●●

●

●●

●

●

●

●

● ●●

● ●●●●●

●●●

●●

●

●●

●●

●

●

●●● ●

● ●

●●

●●

●

●●●

●●

●●

●●

● ●●

●

●

●●

●

●●●●●

●●

●●●

●

●●●●

●

●

●●

●●

●●●●

●●

●●

●

●●

●● ●

●●

●●

●●●

●●

●●●

● ●●

●

●●

●

● ●

●●●

●

●●●●

●

●●●

●●

●●

●●●

●

●

●

●●●●

● ●● ● ●

●

●

●●

●●

●

●●

●

●●●

●●

●●●

●

●●●

●●

● ●●

●

●● ● ●

●●●●

●

●●●

●● ●

●

●

● ●

●●●

●●

●

●

●●●●

●● ●

● ●●

●●●

●

●

●

●●

●

●●●● ●

●●

●●●●

●

●

●

●

●● ●●

●

●● ●●●● ●

●●

●●●●

● ●

●●

●●

●● ●●

● ●

●●

●●●

●

●

●●

●●

●● ●● ●●● ●

●

●

●●●

●

●●

●

●

●

●●

●

●●●●

●

●●

●●

●

●

●

●

●

●● ●●●

●●

●●●

● ●●

●●

●●●●

●

●●

●●●

●●

●● ●●●

●

●● ●

●

●

●●

● ●●

●

●

●●

● ●● ●●●

●●●● ●●

●● ●

●●

●●● ●

●●

● ●●

●

●● ●

●●●

●●

●

●● ●

●●●●

●

●●

●●

●● ●●● ●

●●●

●●

●●

●●

●

●●●

● ●

●●

●●

●●

●

●

●●

●

●● ●

●● ●●

●●

●●

●●●

●●

●● ●

●

●

●●

●●

●

●

●● ●●●

●●

●

●●●●●

●

●

●●

●●●

●

●●●●●

●●

●●

●●

● ●●●

●●

●

●●

●

●●●

●

●

●●

●

● ●●

●

●● ●

●●

●●●

●● ●

●●●

●●

●

●

● ●●

●●

●● ●

●

● ●

●●●●

●●●

● ●●

●●

●

●

●●

●

●●

●

●

●●●

●

●●● ● ●●●

●●●

●

●●

●●

●

●

●●●●

●

●●

●

●●

●

●●

●●

●

● ●● ●

●●●

●

●

●

●●

● ●●● ● ●●●

●●

●

●●

●●

●

●●

●

●

●

●

●

●

●●

●●●●● ●

●

●

●●

● ●●●

●●

●

●

●

●●

●

●●

●

●

●●●

●●

●

●● ●

●

●●

●

●

●

●●

●●●

●●

●

●●● ●●●

●

●● ●

●●

●

●

●

●●●

● ●●

●●●●

●●

●●●● ●

●

●●

●

●●●●

●●●

●

●●●

●

●

●

●

●●●

●

●●●●

●●● ●●

●●

●●

●

●●

●●

●

●●

●

●

●

●

●

●●●●

●●●

●

●●●

●●●

●

●

●●

●

●

●●

●

●

●

● ●●

● ●

●●

●

●

●

●●

●

●

●

●

●●

● ●●● ●●

●

●

●

●

●●●

●●●

●●

●● ●

●

●●

●

●

●●

● ●●●

●●

●

●●

●● ●

●●

●●●

● ●●

●●

●

● ●●

●

●●

●●

●

●●●

●

●●●

●●●●

●

●● ● ●●

●

●●●●

●● ●●

●

●●●

●

●●

●

●●

●

●●●

●● ●

●●

●

●

●● ●●

●

●●

●●●

●● ●●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●●●●●

● ●●

●

●

●

●●

●

●

●

●●●● ●

●

● ●●●●

●●

●

●●

●●

●●

●

●●●

●●●

●

●

●

●

●●●

●

●●●●

●

●

●● ●

●

●

●●●

● ●●●

●

●●

●●

●●

●●●

●

●●● ●

●●●

●

●●●

●

●●

●●

●●

●● ●●●●

●●

●●

●●

●

●

●●●

●●●

●●●

●

●

●

●● ●●●●

●● ●

●

●

●

●

● ●●●

●

●● ●●●

● ●

●

●

●

●●●●

●●

●●●

●● ●

●

●

●

●

●●●

●

● ●●

●●

● ● ●●

●

●

● ●●●

●

●● ●●●●

●

●●●

●●

● ●

●●

●●

●

● ●

●

●●

●●

●

●

●●

●●

●

●

●●

●

●

● ●●●

●

●

●

●

●●

●●●●● ●●

●●● ●

●●

●

●

●

●

●●●

● ●●●

●●

● ●●

●●

●●

●

●●●

●

●●

●

●●

●

●●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●●●

●●

●

●● ●●●

●

●

●

●

●●

●●

●

●

●●●

●

●

●● ●●● ●●●●

●●

●●

●●

●

●●●●

●

●

●●●

●

●

●●●

●●

●

●

● ●●

●●

●●

●

●

●

●

●●

●

●

●● ●●●

●● ●

●

● ●●

●●

●

●●● ●●

●

●

●

●

●

●

●●● ●●

●●

●●●

●●

●

●●

●

●● ●

●●

●

●●

● ●

●

●● ●●●

● ●●

● ●

●

●●

●●●

●●

● ●

●

●●

●

●

●

●

●

●●●●

● ●●

●●

●●

●●

●●

●

●●

●●

●●●

●

●

●●

●●●

●●

●

●●● ●●● ●●●

●

●

●

●

● ●●

●●

●●●●

●

● ●

●●

●

●●●

●

●

●●

●● ● ●

● ●●●

●

●●●●

●●●

●● ●

●

●

● ●

●

●● ●●●

●●●

●●

●

●●

●●

●

●●●

●

●

●

●●

●

●

●

●●●●

●

●

●

●

●

●●●● ●●

●

●●

●

● ●●●●●

●

●

●●

● ●●●

●● ●

●●

●

●

●●●

●●●

●●

●

●●

●

●

●

●

●●●●● ●●●●

●

●● ●●●●

●

●●●

●●

● ●●●

●

●

●●●

●●

●

●●●●●

●

●

●

●

● ●●●

● ●

●

● ●●●

●● ● ●●

●●●●●●

●

●

●●●

●

●

●●

●●●●●

●●

●

●

●●

●● ●●

●

●

●

● ●●●

●

●●

●

●

● ●

●

● ●●

●

●

●

●

●

●

●● ●●

●●●●

●

●

●●●

●●●

● ●

●

●●

●●●●● ●

●●

●

●

●●

●●

●

●

●

●●

●●

●

●

●●

●●

●

●●●●●

●●●

●●

●●

●●●

●

●

●

●

●●

●

●● ●

●●●●

●

●●

●●● ●●●●● ●

●●● ●

●

●● ●●

●●●

●

●● ●●

●●● ●●●

●●

●●

●●●

●●● ●

●●●

●● ●●●●●

●

●

●●

●●

●

●

●

●●●

●

● ●●

●●

●●●●●

●

● ● ●

● ●

●●

●●

●●

●●

● ●

●

●●●

● ●●●●

●●●

●● ●●

●

●●●●●

●

● ●●●

●

●

●

●●

● ●● ● ●●●

●●

●

● ●●●●

●

●

●

●

●●

●●

●

●

●●●●●●●

●

●

●●●

●

●●

● ●

●

●

●

● ●●●

●●●

●●

●

●

●

●

●●●● ● ●

●●● ● ●●●

●

●

●●

●●●●●●●

●●●●

●●●

●●●

●

●

●

●●

●●● ●●

●

●

●

●

●● ●●

●●

●●●●

●

●

●

●

●

●

● ●●

●

●● ●

●

●● ●●

●●

●●

●●●●●

●●●

●●

●●

●

●

●

●●

●●●

●●

● ●●

●

●●

●●●●

● ●●●

●●●●

●● ●● ●●

●

●●●

●

●●

● ●●

●●

●

●●

●●●●

●● ●

●● ●●

●●● ●

●●●●

●

●

●● ● ●

●●

●●●

●●●

●●●

●

●

●●

●●●

●●●

●●●● ●●

●

●●●●●

●

●

●

●●●●

●

●

●

●

●●●●●●●

● ●●●

●●●● ●

●

●

●●●

●●●●●

●● ●

●

●

●

●

●●●

●

●

● ●●● ●● ●●●●

●●

●●●

●● ●●●

●● ●

●

●●

●●

●

●● ●

●

●●●

●

●●●

●

●

●

●

●

● ●

●

● ●

●

●●●

●

●●

●

●

●

● ●

●●●

●●

●●

●●● ●● ●●

●

●

●●●

●●●

●

●

●● ●

●

●●

●●●

●

●

●

●●

●

●

●

●

●

●

●●

●

● ●

●●

●

●

● ●●●●

●

●●●

●

●

●●

●

● ●

●

●●

●●●●

●●●

●●●●

●

●

● ●

●

●

●

●

● ●

●

●

●●

●●

●

●

●

● ●●

●

●

●

●

●● ●●●●●●

●

●●

●● ●●●

●

●●●●●

●

●●●

●

●●●

●

●●

●

●

●

●●●●

●●●●

● ●●

●●

●●

●●

●●●

●●●

●● ●

●●

● ●●●●●●

●

●

● ●●

●

●●

●

●●● ●

●

●●

●●

● ●

●●

●

●●

●●

●

● ●

●

●

●●

●●●

●

●●

●

●

●

●●●●

●

●

●●

●●

●●

●●●

●

●● ●

●

●●

●●●

●●●

●

●● ●

●

●●

●●●

●● ●

●

●

●

●

●

● ●●●●●●

●

●●

●●●

●●● ●●

●●●

●

● ●

●●●

●●

●●● ●●

●

●●●●●●●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●

●●

●

● ●●●

●●● ●● ●

●●●

●●

●

●

●

●● ●●●

●

●

●

●●

●

●●

●

●

●● ●●●

●●●

●

●

● ●●●

●

● ●

●

●

●● ●

●

●

● ●●●

●●●

●

●

●●●●

●

●

●●

●●

●

●

●

●●

●●● ●

●●

● ●●

●

●

●●

●●

●

●●

●●●● ●

●●● ●

●

●

●

●

●●

●●

●●●●

●●

●● ●●●

●●

●

●●● ●●

●●

●●

●●

● ●●●●

●● ●

● ●● ●

●●

●

●

●

●

●●

● ● ●

●

●●● ●

●

●●

●●●

●●

●

●

● ●●●●

●

●●

●

●

●

●

●●

●

●

●●

●●●●

● ●

●

●

●●

●

●●

●

●

●●

●

●

●

● ●●● ●

●

●●●

●●●●

●

● ●

●

●● ●

●●

●

● ●●●

●● ●

●●

●●●

●

●

●

●●

● ●●● ●

●

● ●●

●

●

●

●●●

●

●●●●

●

●

●

●

●

●●

●●

●●●●

●

●

●

●●●

●

●

●

●

●●

● ●

●

●●● ●●●

●

●

●

●●

●

●●●

● ●●●

●●●

●●

●

●●●●

●

●

● ● ●●

● ●

●

●●●

●

●●

●

●● ●

●

●

●

●

●●●●

●

●●

●●● ●● ●

●

●●

●

●●●

●

●● ●●

●

●●● ●

●

●

●

●●●

● ● ●●

●●●

●●●

●● ●●●●

●●●

●● ●● ●●

●●

●●

●●

●●

●●●

●

●

●

●

●

● ●●

●●●●

●●

●●●●

●● ●

●

●

●

●●

●●

●●

●

● ●●

●

●●●

●●

●●

●

●

●●

●●● ●

●●●

●

●

●

●

●●●

●● ●

●

●

●

●

● ●

●

●●

●

●

●

●●

●●●

●●

●

●●

●

●

●●

● ●●●

●●●●

●● ●

●

● ●

●●

●● ●

●

●

●●

●

●

●

●● ● ●●●●●

●

●

●●●

● ●●●

●

●●●

●●

●

●●

●

●

●

●●●

●

●

●●

●●●

●

●●●

●●

●

●●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●●●●

●

●●

● ●●●● ●●●

●●●

●● ●

●●

●●

●

●

●●●

●

●

●●●

●●

●

●

●

●●●

●●

●

● ●

●●

●

●●

● ●

●

●

●

●●

●

●●

●●

●●●●●●

●

● ●

●

●●

●●

●●

● ●

●●

●●●●

●

●

●

●●

●●

●●

●●

●●

●

●●

●●

●

●

●●

● ●

●●●

●

●

●

●

●●

● ●●

●●● ●

●

●●

● ●●

●●

●●

●●

●●● ●●

●● ●●

●

●

●

●

●●

●

●

●

●

●● ●

●●●●

●

●●

●

●●

●

●●●

●

●●●● ●

●●

●●

●

●●●

●●

●

●

●

●

●

●● ●

●

●

●●●●

●●

●●

●

●●●

●●●

●

●

●

●

● ●

●

●

●

● ●

●●●

●●

●●

●

●

●●

●

●●●

●●●

●

●●

●●

●●●●

●

●

●

●

●●

●

●

●●●

●●

●

●● ● ●●

●●

●●

●

●●

●

●

●

●●

●●

●●●

●

●●

●

●

●●

● ●●●●

●

●● ●

●●●●

●●

●

●

●●● ● ●●●

●●●

●●

●●

●●

●

●

●●

●

●● ● ●●● ●●

●

●

●●

●

●● ●

●● ●●

●

●●

●●

●●

●● ●●●●●

●

● ●●

●●●

●

●

●

●●

●

●

●

●

●

●●●●● ●

●

●

●

●●

●●●

●

●

●●●

●●● ●

●

●●

●

●

●●

●

● ●●

●●

●●

● ●●

●●

●●

●

●●●

●●●

●●

● ●●●

●

●

●●

●●

●

●

●●●

● ●●

●

●●

●●●●● ●

●

●●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●

● ●● ●

●●●

●

●

●●

●●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●●●●

●

●

●●●●● ●

●●

●●

●●●

●●●

●● ●

● ●●●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●●

●● ●

●

●●●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●●●

● ●●

●

●

●●

●

●

●

● ●

●●

●

●

●●

● ●●●

●●

●●

●●

●●

●

●●

● ●●

●

●

●

●●

●

●

●

●

●●●

●

●

●●

●●

●

●●

●●

●●●●

●

●●

●

●●

●

●

●

●

●●●

●●●

●

●●

●

●

●

●

●●●

● ●

●●

●

●

●

●

●●

●●

●

●●●●

● ●

●

●

●

●

●●●

●

●●●

●

●

●

●

●●●

●

● ●●

●

●●●

●●

●

●

●●

●●

●

●

●●●●●●

●

●●●

●

●

●

●● ●

●●

●

●●

●

●●●

●

●●

●

●●

●

●●●

●

●

●● ●

●

●●

●●●

●

●● ●●

●●●

●●

●

●

●

●●

●

●● ●

●● ●

●

●

●

●

●

●

●

● ●●

●●

●●●●

●●

● ●

●

●

●

●

●

●

●

● ●

●

● ●●

●

●

●

●● ● ●

●

●

●●

●

●

●

●

● ●●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●

● ●

●

●●●

●

●

●●●

●

●● ●●

●

●

●

●

● ●●

●●●

●

●

●●

●● ●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

● ●●

●

●●

●●●●●

●●

●● ●●

●●

●●

●●●●

● ●

●● ●

●

●●

●

●

●

●

●● ● ●● ●

●●●

●

●●

●●● ●

●

●● ●●

●● ●

●●

●●

●●

●●

●●● ●●

●

●

●

●

●● ● ●●

●

●

●

●●●

●●●

●

●●

● ●● ●

●

●

●●●●●

●

● ●●

●

●● ●●

●

●●●●●

●●● ●●

●●

●

●●●

●

●

●

●●●

●●

●●

●●●

●

●

●●

●● ●●

●

●

●

●●● ●● ●●●

●

●

●●●●

●

●

●●

●

●●

●

●●

●

●●

●

●●

●●●

● ●

●●

●●

●

●

●●

●● ●

●●

●●

●

●

●

●

●

●

●● ●

●

●

●

●

●

●●

●

●

●●

●●

● ●●

●

●

● ●●●●

●

●

●

●●

●●● ●●

●

●●

●

●

●●

● ●●

●●

●●

●●●

●

●

●●●

●●

●

●

●

●●●

●

● ●●

●●●●

● ●

●●

●●

●

●●

●

●●●●●● ● ●

●● ●●

●●●

●

●●

●● ●

●

●

●

● ●●

●●

●●●

●

●●●

●●

●

●

●●● ●●

●

●

●

●●

●●●●●

●

●●●

●

●

●●●

●

●

●

●●●

●●●●

●● ●●●

●●

●

●●●

●●●

●●

●

●●

●

●

●●

●●●

●●●●

●●

●●●● ●

●

●●

●

●●

●

●

●

●●

●●

●

●●●

●

●●

●

●●

●●

●

●

●

●●●

●

● ●● ●●●

●●●●

●

●●●

●●● ●●●

●

●

●●

●●● ●●

●●

●

●

●●●●

●

●

● ●●●●●

● ●●

●●●●

●● ●●

●

●

●

●

●● ●●●●

●●●

●●

●●

●●

●

●

● ●●●

●●●

●●●●

●

●

●

●● ●●

● ●●

●●

●●● ●

●●●●

●●

●●

●●

●

●

●●

●

●●

●●

● ●●●

● ●

●

●●

●●●

●●●●●

●● ●●

●●●●

●

●●

●●

●

●●

● ●● ●

●

●

●

●

●●

●●

●

●

●

●

●●●

●

●●●

●●

●

●●● ●

●●●●

● ●●

●

●●

●●

●

●●●●

●

●

●●

●

●●

●

●

●●●

●

●● ●

●

●

● ●●●

●

●

●

●

●●●

●●

●

●

●

●

●●●●

●

●

●●

●

●●

●

●●

●

●●●

●●●●

●●

●

●●

●●

●

●●●●

●

●●

●●

●●●●●

●●

●●

● ●

●

●●

●●

●

●●●

●●

●

●

●●

●●

●●●●

●●

●●● ●

●●

●

●

●●● ●

●

●

●

●

●●

●●●●

●

●●

●

● ●●

●

●●

●● ●

●●

● ●●

●

●●●●

●

●● ●●●

●●

●

●●

●●

●●●

●

●●

●●● ●●

●●●

●●●

●

● ●

●

●●●

●

●●●

●

●

●●

●●●●

●●

●

●●

●●

●

●●

●

●

●

●

●

● ●

●●

●

●●● ●

●●●●

● ●

●●

●

●●●●

●●● ●

●

●●

●

●

●●● ●●●

●●

● ●

●

● ●

●●

●●●● ●

●●

●●

●● ●

●●

●●

●●●

●●●●

●●

●●●

●●

●

●

●●●●● ●●

●

●

●

●●●

●

●● ●

●

●

●●●

●

●● ●●●

●●

●

●

●●● ●

●●

●● ●●●

●

●

●

●

●●●●

● ●

●

●

●●

●

●

●

●

●● ●

●

●

●●

●

●

● ●

●●

●

●

● ●●●●

●●

●● ●

● ●●

●

●

●●

●

●

●

●

●●

●

●

●●●

●●

●●●● ●●●●

●●

●●●

●●● ●●● ●●●

●●

●

●

●

●

●

●●

●●●

●● ●

●

●●

●●●

●● ●

●

●●●

●●●● ●

●

●

●●

●

●●●●

●●●●

●●

●●●

●

●

● ●●

●

●

●●

●● ●

●●

●●

●●●

●

●●

●

●

●

●

● ●

●●

●●●

●●

● ●

●

●●●

●● ●

●

●●●

●● ●

●●

●●●

●●●●

●●●●

●

●●●

● ●

●

●●●

●●

●

●

●

●●

●

●● ●

●●

●●●

●●

●

●●

●

●●●●

●●

●

●●● ●●●

●●

●

● ●●

●●●●

●●●

●● ●

●

●●

●

●●

●●

●

●●●●

●

●

●●●

●●●

● ●●

● ●●●

●

●

●

●●●●●

●

●

●● ●

●

● ●

●●

●

●●

●

●

●

●●

●

●

●● ●

●

●● ● ●

● ●

●

●● ● ●●

●●●●●●

●

●●

●●●● ●

●

●

●●

●

●

●●

●

●●

●

● ●● ●●

●

●

●

●

●●●

●

●

●●

●●●● ●

●●●

●●

●●●

●●

●● ●

●●●●

●

●●

●●●

●●

●

●

●

●

●●●

●●●

●

●

●

●●●

●

●●●

●

●● ●●●●●

●

●●

●

●● ●

●

●

●●

●●

●●

●●

●

●

●●

●●

●

●

●

● ●●●

●●●● ●

●

●●

●●

●

●●● ●●● ●●

●

●●●

●●●●

●●

●●

●●● ●

●

●

●●●●● ●

●

●

●●

●●

●●

● ●●

●

●●

●

●

●●

●●●

●●

●

●●

●●●

●

●

●

●

●●

● ●

●●

●

●

●●

●●●●

● ●●

●

●●● ●

●●

●

●

●

● ●●●●

●

●●

●

●●

●

●

●●●

● ●●

●

●●

●

●●● ●

●●●

●

●●●● ●

●●

●

●

● ●●

● ●● ●

● ●●● ●● ●

●

●●

●●●● ●

●

●●

●

●●●

● ●●

●

●●

●

●

●

●●●●●

●●

●●● ●

● ●

●

●●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●●

●

●

●●

●

●

●

●●●

●●

●●●●

●● ●

●

● ●●

●

●

●

●●●●

●●● ●

●

●●

●

●●

●● ●●

●● ●●●

●

●●

●●●●●

●● ●

●

●

● ●

●

●●●

●●

●

● ●●

●●●

●

●

●

●●

●

●

●●● ●●

●

●

●●

●

●●

●●● ●

●

●●●●● ●

●●

●

●

●

●● ●●

●

●

●●●

●●●● ●●

● ●●●●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●● ●

●●●●

●

●

●●

●● ●

●●

●

●●

●●●

●●

●●●

●●

●●

●●●

●●

●

●

●

●●

●●

●

●●

●

●

●●

●

●

●

●●

●

● ●

●

●

●●

● ●

●●

●●

●●

●

●

●●

●●

●

●

●●

●

● ●●

●

●

●●

●

●●●●●

●●●

●●●

●

●

●●

●

●●

●

●

●● ●●

●●

●

●

●

●● ●

●

●

● ●

●● ●

●● ●

●

●●●●●●

●

●

●

●

●●●●●

●

●

●●

●

●●

●●

●

●●●●

● ●●

●● ●

●●

●

●

●●●

●

●●●

●

●

●

●●●

●

●●●●●●●●●●●●

●●

●●●● ●●●●

●●

●●

●

●●●

●●

●●

●

●

●●●● ●●●●●

●

●

●

●

●●

● ●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●●●

● ●●

●

●●

●●●

●●●

●●●●

●●

● ●

●

●

●

● ●

●

●

●●●●

●

●

●

●●●●

●

●●●

●●

●

●

●● ●●

●●

●●

● ●●

●

●

●

●●

●

●

●

●

● ●

●

●●

●

●

●●

●

●

●

●●● ●

●●

●●●●

●

●●●●

●●

●

●●

●● ●

● ●●

●●●●

●

●●

●

●

●

●●

●

●

●

●●●

● ●●

●

●●●

●

●

●●●

●

●● ●●●

●

●

●●

●

●●●●

●●

●

●

●●

●●●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●●● ●

● ●

●●●

●

●

●

●

●●● ●

●

●

●

●

●

●

●

●

●●

●

●

●●●●●

●●

●●●

●●●● ●

●

●●●

●● ●●

●●●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●● ●

●● ●●

● ●

●●

●

●

●●●

●

●●● ●●●●

●

●●

●●

●

●

●

●

●●●●● ●

●●

●●● ●●●●

●

●

●

●●

●●

●

●

●●

●

●

●●●

●

●

●●●

● ●●●●●●

●

●●●

●

●● ●

●●● ●●

●

●●

●●●●

●

●

●●

● ●●●● ●●●

●●●●●

●●● ●

●●●

● ● ●●

● ●●

●●

●●●

●●

●● ●

●

● ●●●

●

●●●

●

●

●

●

●

●

●

●

●●●●

●● ●●

●●●●

●

●

●●

●●

●

●

●

●●

●

● ●●

●●

●

●●

●

●

●● ●●

●●

●

● ●●●●

●●●

●●●

●

●

●●

●●

●

●

●●

●

●●

● ●●

● ●●●

●

●

●

● ●●●

●●

●●●

●● ●●

●

● ●●●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●●● ●●● ●●

●●

●

●●

●

●

●

●●

●●

● ●

●●●

●

●●

●●●●●

●●

●●●

●

●●●●

●●

●

● ●●

●

●● ● ●

● ●

●●●●

●

●●●●●

●●

●●

● ●●

●

●●

●

●●●●●

●●

●●●●

●●●

●●

●● ●●●

●●●●●

●

●

●●

●●

●●●

● ●●

●

●●

● ●

●●

●

●● ●

● ●●●

●

●●● ●●

●

●

●●●

●● ●●

●

●

●

●●●

●

●●

●

●●●●●● ●

●

●●●●

● ●●

●

●● ●●

●●●

●●●

●●●●

●●

●

●●

●●

●

●●

●●●●●

●● ●●●●● ●

●●

●

●●●

●●

●●

●●●

●●

●●●

●

●●

●●

●● ●● ●●●●

●

●

●●

●●

●●

●

●

●

●●●●

●●

●

●

●

●● ●●●●

●●●

●● ●

●●●

●●●●

●

●●

●●●●●● ●●●

●● ●●

●●

●

●●●●

●●●● ●●

●

●● ●

●●● ●

●● ●●

●

●●

●● ●

●

●

●● ●

●●●●●

●●

● ●●● ●

●●●●

●●

●●

●

●●●

● ●

● ●

●

●

●●●

●●●●

●●●●

●●

●

●

●●

●●

●

●

● ●●●

●●

●

●●

●●

●

●

●●●

●

●

●●

●

●●

●●

●●

● ●●

●

●●

●

●● ●

●●

●●●

●●●

●●

●

●●●

●●● ●●●

●●

● ●●

●●

●●

●

●

● ●

●

●● ● ●●

●●●

●

●

●●●●●

●●

●

●●

●●

●

●●●●

●●●

●

●

● ●●●● ●

●●●

●

●●●

●

●

●

●● ●●●

●

●●

● ●●

●●●

●

● ●

●

●

●●●●●●

●

●●

●●

●

●

●●●●●●

●●

● ●

●

●

●

●●●

● ●●●

●●● ●●●●●

●● ●

●

●●●●

●

●

●●●

●●

●●

●●●●●

●●● ●

●●●●

●

●● ●

●●

●

●

●

● ● ●

●●

●●

●●

●

●

●

●●

●●● ● ●

●

●

●

●

●●●

●●●

●●●●

●

●●

●

●●

●●●

●●

●●

●●

●●

●●

● ●●

●●

●●●

●

●●

●●

●● ●

●

●●●

●●

●●

●

●●● ●●

●

●●● ●

●● ● ●

●

● ●●●

●

●

●

● ●●

●

●

●●● ●

●● ●

●

●●

●

●

●

●●

●●

●●●●●

●

●

●

●

●

●

●●●

●

●●

●●●

●● ●●

●

● ●●●●

●

●●

●

●

●

●

●● ●

●

●

●●●

●●

●

● ●●

●

●●●

●

●● ●

● ●

● ●● ●●●

●●

●●

●●

●●●●● ●

●

●

●

●●●●

●

●

●● ● ●●● ●

●●

● ●

●

●

●●●

●● ●

●

●● ●●

●

● ●●●●

●

●

●

●● ●

●

● ●●

●

●

●●●

●●

●●

●●

●

●

●

●●

●

●●

●●

●

●

●

●● ● ●●

●● ●●●

●

●

●

● ●●●

●

●●

●●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●●● ● ●●

●●

●●

●●

●

●

●●

●

●

●●●

●

●●

●●

●

●

●●●

●●●

●●●

●

● ●

● ●● ●

●

●

●

●

●●

●

●●●●

●

●●

●● ●●

● ●●

●

●

●●

●●●● ●

●

●●

●

●●

●

●●

●●●●

●●

●●

●

●

● ●

●●

● ●

●

●

●

●

●●

●

●● ●● ●● ●● ●

●

●●●●

●●

●●

●●● ●●

●

●

●●

●● ●

● ●●●●● ●●●

●

● ●

●

●

● ●

●

●● ●●●

●●

●●

●● ●●

●

●

●

●●

●

●

●

● ●●●

●●

●● ●●●

●●●● ●

●●●●

●● ● ●●

●●●

●●

●● ●●

●

●

●●

●

●●●●●●

●

●● ●●●

●

●●

●

●●

● ●●●

●

● ●●●●

●

●●

●

●

● ● ●●

●

●

●●●●● ●

● ●● ●●

●

●

●

●

●●●

●

●

●●●●

●

● ●●●

●

● ●

●

●

● ●

●

●●●

●

●

●

●

●●●●

●●●

●●

●●

●●●

● ●

●

●●

●●● ●●

●

●●

●●

●

●●

●

●

● ●●●●

●●●

●●

●●

●●●

●

● ●

●●

●

●● ●●● ●●●

●●● ●

●●● ●

●● ● ●●● ●●●

●●●

● ●●●

●● ●

●● ●

●●●

●●

●

●●

●●●

●

●●●

●

● ●●

●●

●●

●●●

●● ●

●

●●

●●

●●

● ●

●

●●● ●●●●

●●●

●●●

●

●●●●●

●●

●●●

●●

● ●●● ●●●●● ●● ●

●●

●●

●●

●●

●●●● ●●

●● ●●

●

●●

●●

●

●

●

●● ●●●●

●

●

●

●

● ●●●●●● ●

●●●

●●●●

●●

●

●●●●●

●

●

●

●

●●

●●●

●●●

●●●●

●

●

●

●

●

●

● ●●

●

●● ●

●

●● ● ●

●●

●

●●●●

●●●

●●

●●

●

●

●●

●●

●●

●●●

●

●●●●

●●●

● ●●●

●● ●● ●●

●

●● ●

●

●●● ●

●● ●

●●●●

●●●

●●● ●●

●●●●

●●●

●

●

●●●

●● ●●●

●● ●●

●●

●●●

●●●

●●

●

●

●

●

●●● ●●

●

●

●

●

● ●●●

●●● ●

●●●●● ●

●●●

●

●

●●

●

●●●● ●● ●●●

●●● ●

●● ●●

●

●●

●

●● ●

●

●●●

●

● ●

●

●

●

●●● ●●●

●

●

● ●●●●● ●

●●

●

●●●●

●

●●

●●

●

●

●●

●

●

●

●

●●

●●

●●●

●● ●●

●

●

●●

●●

●●●

● ●

● ●

●

●

●●

●

●

● ●●● ●●●

●●

●●

●●●●

●

●●●

●

●● ●●

●●

●●

●

●●

●●

●●

●●●

●

●● ●

●

●●

● ●●●

●●

●

●●

●●

●

●●

●

●

●

●●

●●

●

●

●

●

●●

● ●●

● ●

●

●

●●

●

●

●

●●

●●

●●●

●

●●

●

●●●

●

●● ●

●●

●

●●

●

● ●● ●●

●

●●●

●

●●

●●●●

●

●●●

●

●●● ●●

●

●●●●●●●

●

● ●●

●

●

●

● ● ● ●●● ●●

●

●

●

●●

●

● ●

●

●

● ●● ●

●

●●

●●

●●●●

●

●●●

●

●●

●● ●●●

●

●●●

●

●

●

●

●●

●

●

●

●●

●● ●●

●

● ●●

●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●●

●●

●

●●

●

●●●●

●

●

●

● ●●●

●●

●●●●● ●

●

●●

●

● ●●●

●●

●

●

● ●●● ●

●●

●

●

●

●●●●

●●

●●●

●●

●

●

●●

●●

● ●

●●●●

●

●

●

●●

● ●

●

●● ●● ●●

●

●●

●

●●●

● ●●●●●

●

●

●●●●

●

●

●● ●●

● ●

●

●●●

●●

●

●●

●

●

●

●

●●●

●

●●

●●

●●● ●●

●

●

●●●

●

●●●●

●

● ●

●

●● ●

● ●●

●●

●●●●●●●●●

●●●●

●●

●●●●

●

●

●

● ●●

●●●●

●●

●●●●

●

●

●

●●

●

●●

●

● ●●

●

●●●

●●●

●

●

●

●●

●●●

●

●

●●●●●

● ●

●

●

●●

●

●●

●

●

●

●●

●●●

●

● ●

●●●

●●●

●●●● ●

●

● ●

●●

●● ●

●

●

●

●

●

●● ●●

●●●

●

●●● ●

●●

●

●● ●●

●●

●●

●● ●

●

●●● ●●

●

●● ●

●●

●

●●

●●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●●●●● ●

●●

●●●●

●●

●●

●

●

●

● ●●

●●

●

●

●

● ●●

●●

●

●●

●

●●●

●

●

●

●●

●

●●●●●● ●

●

●

● ●

●

●●

●

●●

●

●●

●●●

●

●

●

●●

●●●

●●

●●

●

●●

●●

●

●

●●● ●

● ●●

●

●

●

●

●●

● ●●

● ●● ●

●

●●

● ●●

●●

●●

●●

●●●●

●

●

●

●

●●●

●

●● ●

●●●●

●

●●

●

●● ●●

●●● ●●●

●

●

●

●

●

●

●

●

●

●

●● ●

●

●

●● ● ●

●●

●●

●

●●●

●●

●

●

●

●

● ●●

●

● ●

●●

●

●

●

●

●●

●

●●●●●●

●

●●●●

●●●●

●

●●

●

●

●●

●● ●●

●●

●●

●

●●

●

●

●

●

●

●●●

●

●●●● ●

●

●●

●●● ●

●●

●●●●●

●●

●●

●

●

●● ●●● ●

●

●

●●

●

●●

●●

●●

●

●●● ●●●

●

●●●

●●

●

●●

●

●

●

●

●●●●●●

●

●

●

●●

●●●● ●

●●●

●● ●●●

●

●

●

●

● ●●

●●

●●

●●

●●

●●

●

●●

●●●

●●

●●●

●

●

●●

●●

●

●

●●●

● ●●

●

●

● ●●●

● ●

●

●●●●

●

●●

●

●

●●

●●●

●●

●

●

●●

●●

● ●

●

●

●

●

●

●●●

●●●●●●

●●

●●

●● ●●

●●●● ●

●●

●●

●●

●

●

●

●

●

●●

●

●

●●

●●

●

●● ●

●●

●●

●

●

●

●

●●

●

●

●● ●●

●

●

●●●

●

●●

●

●

●● ●●

●●

●●

●●

●●

●

●●

● ●●

●

●●

●

●●●

●

●

●

●

●

●●

●●

●●

●

●

●

●●

●

●

●●

●

●●

● ●

● ●●

●

●

●

●

●●● ●●●

●

●

●

●●●

●●

●

●

●●

● ●●

●

●

●●

●

●●● ●●

●●●●

●●

●●●

●●

●

●

● ● ●

●

● ●

●●

●● ●●●

●●●● ●

●● ●

●

●

●

●●

●

● ●●

●●● ●● ●

●

● ●

●

● ●

●

● ●●

●

●●● ●

●

●

●

●

●

● ●●

●

●●

●

●●

●●

●

●

●

●● ●

●

●● ●●●

●

●

●

●

● ●●

●●●

●

●

●●

●● ●

●

●

●

●

●

●

●

●

●●●●●

●

●●●●●

●●

●

●

●

●●

●

●

●●

●● ●● ●

●●●

●●

●●● ●●

●●●

●●●

●●

● ●● ●●

●

●

●

●

●●●

●

●

●

●●●

●

●

●●●●●

●

●

●●●● ●

●

● ●●

●

● ●●

●

●● ●● ●●●

●

●

●

●

●

●●●

●●

●●

●

●

●

●

●●

●● ●●

●

●

●

●●●● ●● ●

●

●●

●

●

●●●

●

●●● ●

●●●●

● ●

●●

●

●●●

●● ●

●●

●●

●

●

●

●

●

●

●● ●

●

●

●

●

●●

●●

●●

●●

●● ●●●

●

●●

●● ● ●●●●

●●●

●●

●●

●

●●●

●●●

● ●

●

●

●

●● ●

● ●●

● ●●●

● ●

●●●

●●●● ● ●

●● ●●

●●●

●

●●●

● ●●

● ●●● ●

●

●●

●●

●

●

●● ●● ●

●

●●

●●●

●

●●●●

●

●● ●●

●●●

●●●

●●●

●

●

●●

●

●

●●● ●

●●●

●●

●●●

●

●●

●

●

●

●

●

●●

●●

●●●●

●

●●

●●

●

●

●

●●●

●

● ●● ●●●●● ●

●

●●●●● ● ●●

●

●

●●

●●● ●●

● ●

●

●

●●●

●

●

●●●●●

● ●

●●●●●● ●●●

●

●

●

●● ●●● ●

●●●●●

●●

●●

●

●

●●●●●●

●●●●

●

●

●

●● ●●

● ●●

●●

●●● ●

●●

●●●

●●●

●●

●

●

● ●

●

●●

●●● ●

●●●●●●

●●●

● ●●●●

●● ●●

●●●●

●

●

●●

●

●●

● ●●

● ●

●

●●

●●

●

●

●

●

●●●

●

●● ●

●

●●● ●

●●

●●● ●

●●

●●

●●

●

●●●

●

●

●

●●

●

●

●

●

●●●

●

●● ●●

●

● ●●●

●

●

●

●●●

●●●

●

●

●●●●●

●

●●●

●●

●

●●

●

● ●●

●●●

●●

●

●

●●

●●

●

●●● ●

●

●●

●●

●●●●●

●●

●●

●

●

●●

●●

●

●●●

●●●

●

●●

●●

●●●●

●●

●●● ●

●●●

●

●●● ●

●

●

●

●

●●

●● ●●

●

●●

●●

●

●●

● ● ●

●●

● ●●

●

●●●●

●

●● ●●●●

●

●

●●

●●

●●●

●

●●

●●● ●●

●●●

●●●

●

● ●

●

●●●

●

●●●

●

●

●●

●●●

●

●●

●

●●

●●●

●●

●

●

●

●

●

●●

●●●

●● ● ●●

● ●●

●●

●●

●●●● ●

●●●●●●●

●

●

●●

●● ●

●●

●● ●●●

●●

● ●●

●●

●●

●●●

●

●

●

● ●●●● ● ●

●

●

●

●●●

●

●●●

●

●

●●●

●

● ●●●

●●

●

●

●●●●

●●

● ●●

●

●

●

●

●

●●●

● ●●

●●

●

●

●

●

●● ●

●

●

●●

●

●

● ●

●●

●

● ●● ●●●

●●● ● ●

●

● ●●

●●● ●●

●● ●●

●

● ●●●

● ●

●

● ●●●

●

●

●●

●● ●●

●

●

●● ●●

●●

●●

●●

● ●●●

●

●●●●

●●

●

●●●

●●

●●

● ●●

●●●● ●

●

●

●

●●

●

●

●

● ●

●

●

●

●

●●

●

●

●●

●●

●● ● ●●

●

● ● ●●●●

●●●

●●●●●●

●

● ● ●●●●

●

●●●

●●●

●

●● ●

●

●

●

●●●

●●●

●●

●

●●

●● ●

●

●

●

●

●

●

●●●● ●

●

●●

●

● ●●●●●●●

●● ●●

●●

●● ●

●●● ●

●

●

●●●●● ●

●●●

●●

●●

● ●●

●●

●

●

●●

●●

●

●

●●●●●

●●●

● ●● ●

●

●

● ●●●

●

●

●

●● ●

●

●

●●● ●

●

●●

●

●●

●

●●

●●● ●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●● ●●●●

●

● ●●

●

●

●

●●●●

●●● ●

●

●●

●

●●

●●●●

●● ●●●

●

●●

● ●●●●●

●●

●

●

●●

●

●●

●●

●

●

●● ●●

●●

●

●

●

●

●

●

●●● ●●

●

●

●●●

●●

●●●

●

●●● ●●●

●

●

●●●

●

●●●●

●●●

●

●

●

●●●●

●

●

●●

●●●

●●

●●

●

●●●●●

●●●

●

●

●

●

●

● ●

●

●

● ●

● ●

●

●●●

●

●

●●

●

●

●

●●●●

●●

●

●

●●

●

●●

●

●● ●

●●●

●●

●

●●● ●

● ●●

●● ●

●●

●

●

●●●

●

● ●●

●●● ●

● ●

● ●

●

●● ●

●●

●●

●

● ●●● ●●●●●●

●

●

●

●●

●●●

●●●●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●● ●●

●

●●●●●

●●●●●●

●

●●

●●

●

●● ●

●

● ●●

●●●

●

● ● ●●

●●

●●●

●

●

●

●

●●

● ●

●

● ●

●

●●

●

●

●

●

●

●

●●●●

●

●●●●● ●●

●

● ●

●

●●

●● ●

● ●

●●

●

●

●

●

●

●●

●

●

●

●●● ●●

●

●●

●●●●

●

●

●●

●

●● ●●●

●

●

●

●●●

●

●●

●

●

●

●

●●

●●● ●

● ●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

Age

−0.50

010

020

0

0.077

1800

1950 ●

●

●●

●

●●●

●●●

●●●

●●

●

●

●

●●

●● ●●

●

●

●●●

●

●

●

●

● ●●●

●

●●

●

●

●

●●

● ●●

●●

●●

● ● ●●●●

●

●

●

●●

●●

●

●

●

●

●

●● ●● ●

●●

●

●

●

●● ●●●

●●

●

●

●

●●● ●●●●

●●●

●●●

●●

● ●

●

●

● ●●● ● ●●

●● ●

●●

●●

●●●●●

●●

●

●

●

●

●

●

●

●

●

●● ●●

●

●●●

●●●● ●

●

● ●●

●

●

●●

●

●●●

●

●●

●

●●

●

●

●●●

●●● ●●

●

●

●●

●

● ●● ●

●

●●●

●

●●●●

●

●●

●

●●● ●

●

●

●●

●●

●●

●●

●

● ●

●●

●●

● ●●

● ●●

●

●●

●●

●

● ●●

●●

●●●

●●

●

●

●●

●

●● ● ●

●●

● ●●●●●

●

●●

●

●● ●

●

●●●●●

●

●

●

●●

● ●●

●●

●

● ●●

●

●

●

●●●

●

●

●●●

●

●

●

●●

●● ●

●

●●●●

●●

●

●

●

●●

●

●

●● ●●

●

● ●●

●●●●● ●

●

●

●

●●

●

●●● ●●●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●● ●

●

●

●

●

●

●

●

● ●

●

●●

●

● ●●

●

●●●

●●

●●●

●

●

●●

●●

●

●

●●●

●●

●

●

●

●

●

●●

●

●●

●

●●●

● ● ●

●

●

●●

●

●●●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●● ●●●

●

● ●

●

●

●

●

●

●●

●●●●●●● ●●

●●

●●

●●

● ●●●

●

●●

●

●

● ●●●

●

●

●

●

●●●●

●

●

●●●

●●●

●●

● ●

●●

●

●●

●●

●

●

●

●

●

●●

●

●

●

● ●

●●

●

●●

●

●●●

●

●

●●●

●●●

●●

●

●●●

●

●

●

●● ●

●

●●●●

●●●

●

●

●● ●

●

●●

●

●●●

●

●

●

●

●

● ●● ●

●●●

●

●

●

●

●●●●

●

●●●●

●●●

● ●●

●

●

●● ●

●●●

●●●●●

●●

●

●

●

● ●●

●●

●

●

●

●

●

●

●●●●

●● ●●●●●

●●

●● ●● ●

●

●●

●

●● ●

●

●

●●

●

●

●●

●

●●

●

●●●

● ●●●●● ●

●

●

●●

●

●●● ●

●

●

●●

● ●● ●

●●

●

●

●●

●

● ●●●

●

●

●

● ●●

●

●●●

●●

●●●

●●

●●●

●

●●●●●

● ●●●●●

●●●●

● ●● ●

●

●●● ●

●

●● ●

●

● ●●

●● ●●

●

● ●●

●

●●

●●

●

●

●●●●

●●●●

●●●

●

●

● ●

●●

●

● ●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

● ●

●●●●

●● ●●●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●

● ●●

●

●●

●●

●

●●

●●●● ●

●

●

●

●●●

●

●

●

● ●

●

●●

●

●

●

●

●

●●●●

●

●

●

●

●● ●●

●

●●●●●●

●●●

●

●●

●

● ●●● ●●●● ●●● ●

●●

●●

●

●●●

●●●

●● ●

●●●●

●●

●

● ●●

●● ●

●● ●

●

●

●

●

●●● ●●●

●

● ●

●

●

●

●●●

●●

●

●●

●● ●●

●

●●● ●

●●●●

●●

●●●●

● ●● ●●●●●● ●

●

●

●●● ●●●

●

● ●● ●● ●●●

●●●● ●●

●

● ●● ● ●●●

●

●●● ●●

●●● ●●

●●

●

●●

●● ●

●●●●

●● ●

●

●●●●●

●●

●

●●●●

●●●●

●

●●●

● ●

●●

●●●

● ●

●●

●

●●● ●●● ●●

●● ●●

●●

● ● ●●●

●

●

●

●

● ●●

●

●

●● ●●● ●●●

●

●

●●●

●

●

●

●●

●● ●●

●●●

●●●

●●

●● ●

●●

●

●

●

●

●●

●

●●

●

●●

●

●●

● ●●

●

●

● ●

●●

●

●●●

●

●●

●●

●●

●

● ● ●

●

●●

●● ●

●

●

●

●●

●

●●

●●● ● ●●●

●

●●●

●

●●

●

●

●

●

●●

●●

●

● ●

●

●

●

●●

●

●●●

●

●

●

●●●●

●

●●

●

●●

●

●

● ●●

●

● ●● ●●

●

●

●

●●

●●

● ●

●

● ● ●●●●●●

●●●●

●

● ●●

●

●●●

●

●●

●●●●●

●●

●●

●● ●●●

●●

● ●

●●●

●

●

●

●

●●●

●

●●

●

●● ●

●

●●

●●

●

●●● ●

●●●

●

●●● ●●

●

●

●● ●●●

●●

●●● ●● ●●

●

●

●

●●●

●●

●●

●

●

●

●

●

●

●●●

●

●●● ●●●●●●

●

●●●●●●

●●

●

●● ●

●

●● ●●●●●●

●●●

●

●

●

●

●

●●

●●●

●●●

●

●●●

●●●●

●

●●

●

●

●●

●●

●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●

● ●● ●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●● ●

●

●●

●

●

●●

● ●●

●

●●●

●●●

● ●

●

●●●●● ●●

●● ●●

●

●●

●●●

●

●

●●●

●

●●

●

●●●●

●●

● ●●

●

●

●●●●

●

●●●

●

●●●●

●

●

●

●●

●

●

●●

●●

●●

●●●●● ●●

●

●

● ●●

●●● ●

●

●

●●● ● ●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●●

●●●

●

●

●

●●

●

●

●●●●●

●

●

●

●

●●●

●

●●

●

●

●

●●

●

●●● ●●

●

● ●●●

●●● ●●●●

●

●● ●

●●

●

●●●

●● ●

●●

●

●

●

●●

●

●

●

●

●

●

●●● ●

●

●●● ●●●●●●●●●●●

●

●

●●●

●●

●

●●

●●●

●

●●●

●●

●●●

●

●

●

●● ●

●●

● ●●

●●

●

●

●

● ●●●●

●●●

●●●

●

●

●

●●●● ●● ●

●

●●

●

●●

●

●

●

●

●●

●

●●

●●

●

● ● ●●

●

●● ●●●●●● ●●●●●

●●●

● ● ● ●

●● ●●

●

● ●● ●●

●●

●

●

●●

●

●

●

●

●●

●●

● ●●

●

●

●

●

●

●●●

●●●● ●●

●●

●●

●●

●●

●

●

●●●

● ●●●

●

●

● ●●●●●

●

●

●

●●

●

●●

●

●●●●

●● ●

●

●

●●● ●

●

●●

●●●●●

●

●●

●●

●

●

●● ●●●

●

●

●

●

●

● ●●●●

●

●●●

●

●● ●●● ●●●●●

●

●●

●●

●

●●

●●●

●

●●●

●

●

●●●

●

●

●

●

●

●

●●

●

●●●

●

●

●●

●●

●●● ●

●

●●● ●

●● ●● ●

●

●

●●● ●●

●

●●

● ●●●●● ●●

●

● ●●●●

● ●●

●●

●

●

●●●●

●●

● ●

●

●● ●●●

● ●●

● ●●

●●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●●●●● ● ●●●

●

●

● ●● ●

●

●●

●●

●●●

●●

●●●

●●

●●

●●●

●

●

●● ●

●

●

●

●

●

●● ●

●

●● ●●●●

●

● ●

●●

●

●●

●

●

●

●●

●

●

●

●

● ●●●

●

●●●

●

●●

●●● ●

●

●

● ●●

●● ●●● ●●

●●

●●

●

●●

●

●

●●

●●

● ●●

●

●

●

●

●●●●

●

●

●

●●

●●●● ●●●

●●●

●●

●●●●●

●

●●

● ●

●●●● ●●●

●●

●●●

●●

●●●

●●

●●

●

●

●

●●

●

●● ●●●●

●●

● ●●●●

●

●●●

●●

●●●

●

●

●

●●●●

● ●●

●●

●●

●●

●

●

● ●

●

●

● ●

●

●

●●●

●

● ● ●●● ●●

●

●

●

●

●

●●●●

●

●●

●●●●●●●

●

●

●●

●

● ●

●

●●

●

● ●●●

●

●●

●

●

● ●

●

●

●●●

●

●

●

●

●

●● ●●

●

●

●●

●

●

●●

●

●

●

●

● ●

●

●●

●●● ●● ●

●

● ●

●

●● ●●●

●

●

●●

●●

●

●

●

● ●●

●

●●●●●●●

●

● ●●

●

●●●

●●●●

●●●

●

●

●

●●●● ●●●

●●● ●●●● ●

●●●

● ●●●● ●●

●●●

●

●● ●●

●●● ●●●●

●

●●

●●●●

●

● ●●●●

●● ●●●●● ●●

●●

●

●●●

●

●●

●

●

●

●●

●●

●●●

●●●

●●

●●

●

●●●

●

●●

●●

●

●

●

●●

●● ●●●

●●

●

●

●● ●●

●●

●●●●

●● ●●● ●●

●●

●● ●● ● ●

●● ●●

●

● ●●●● ●

●●

●

●●

●●●

●

●●●●●

●●●

●

●●

●

●

●

●

●

●●

●

●

● ●●●●

●

●

●●

●

●

●●

●

●●● ● ●●●● ●

●

●●●●

●● ●●●

●

●●● ●●●●●●● ●●

●

●

●

●

●

●

●●● ●●●

●

●

●

●●●

●●●

●●●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●●●

●●

●

●●● ●

●

●●

●

●

●

●

● ●

●●●

● ●●

●●

●

●

●

●●●●

● ●●●

●●●●●

● ●● ●

●

●●●

●

●

●●● ●

●●

●

●●●●● ●●●●

●●● ● ●●●● ●

●

●●

● ●

●●

● ●

●

●

●●● ●

●●●

●●

●

●

●●●●●

●●●●

●●●●

●

●●

●●●●●

●

● ●●

●●●●

●

●

●●

●●●●●●

● ●●●

●●●●

●

●

●

●●●

●●●●●●● ●

●●●●●

●●

●●

●●●● ●● ●●●

●

●●

●●● ●● ●●●●

●

●

●●●

●

●● ●●

●

●●●

●

●●

●

●●

●●

●

●● ●

●

●●

●●●●

● ●●●

●

●●

●

●●

●●

●●● ●●

● ●● ●

●●

●

● ●●

● ●

●

●

●●

● ●

●

●

●

●● ●●

●

●

●

●

●

●●

●

●

●

●● ●

●

●

●●

●● ●

●●●●

●●

●●

●

●

●●●● ●

●

●●

●

●

●

●● ●●●

●●●●●● ●

●

●

●

●●●

●

●●

●

●

●●

●●

● ●●●

● ●

●

●

●

●●●●●●●

●

●

●●

●

●

●

●●●

●●

●

●●●●●●●

●

●

●●●

●

●

●

●

●●●

●●●

●

●

●

●●

●

●

●

●●

●

●●●●

●● ●

●●

●●●●

●

●●●●

● ●●

●

●●

●

●●● ●

●

●●

●●

●●

●●

●

●

●

●●

●● ●●●

●●

●● ●●

●

●

●

●

●

●●●●●

●

●

●

●●

●

● ●●●

●

●● ●

●●

●

●●

●

●●●

●

●● ●●●

●

●●● ●● ●

●●

●●●

●●●

●●

●

●●

●

●

●

●●●

●

● ●

●

●●●●

● ●●

●

●●

●●

●● ●●●

●●●●●

●

●

●

●●●

●

●

●●

●●●●

●

●

●

●●

●

●

●

●

●

● ●●

●

●

●● ●●

●

●●

●

●●

● ●

● ●●

●

● ●●

●●

●

●●●

●

●● ●●●●●●

●

●

● ●●

●

●

● ●

●

●●

●

●●

●

● ●●●●

●● ●

●

●●●●

●

●●●

●●● ●

●

●

●

●● ● ●●

●

●

● ●

●

●●

●

●

●

●●

●●

●● ●

●

●●●

●

●

●

●

●●

●● ●●

●●●

●● ● ●

●

●

●

●●

●

●

●●●●●

●

●●●

●●●●●●●●

● ●●●●●

●●●

●●●

●

●

●●

●●●

●

●

●●●●●

●●

●●

● ●●●●

●

●●

●●

●●● ●

●●

●●●●● ●

● ●

●

●

●●

●

●

●●

●

●● ●

●

●

●●

●

●

●● ● ●●

●●

●●

●●

●●

●●

●● ●●

●●

●●

●

●

●●●

●●

●

●

●●●

●

● ●●●

●

●

●●●

●

●●

●

●●●●

●●●

●●

●

●

●

●●●● ●●●

●●●

●

●●●

●●

●

●

●●● ●

●●

●

●

●●

●●

●

● ●●

●

●

●

●● ●●●

● ●●

●

●● ●●●●

●●

●

●

●

●●●●

●●

●

●

●

●●

●

●

●●

●

●

●

●●●

●●

●●●

●

●

●●

●

●

●

● ●

●

●●

●●● ●●

●●

●●

●

●

●

●

●●●

● ● ●●● ●

●

●●● ●● ●●●●

●

●

●

●

●●

●

●● ●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●● ●●

●●

●

●

●

●

●●

●●

●

●

●●

●●

●●

●●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●●

●

●

●●

●

●●●

●

●

●● ●●

●

●

●●

● ●●● ●●●●●●

●●● ●●●

●

● ●

●

●

●●

●●

●

●●●

●

●●●● ●

●

●●●● ●●●

●

●●

●●●

●●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●●●

●●

●●

●● ●● ●

●

●

●

●●

●●

●●

●

●

●●●●

●

●●●

●●●●

●●●

●●●

●●

●

●●

●

●●●

●

●●●

●

●●●

● ●

●

●●

●●

●

●●

●

●

●

●

●●

●●●

●

●● ●

●

●● ●

●

●●

●●●●●●

●

● ●

●

●●

●

●

●●

● ●

●

●●●●

●

●

●

●

●●

●

●

●●

●●

●●

●

●●

●●●

●●

●●

●

●●●

●●

●

●

●●●

●●

●

●

●

●

● ●●

●

●●●

●

●

●●

●● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

● ●●●●

●

●

●●

●

●●

●

●●●

●●●●

●●

●●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●●●●●

●●

●

●

●●●

●●●

●

●

●

●

●

●

●

●●

●

●

●●●

●●

●

● ●

●

●●

●

●●●●●

●

● ●

● ●●

●

●

●●

●

●

●

●

●●

●

●●

●●●●

●

●

●

●

●

●●● ●

●

●

●●

●

● ●● ●●

●● ●●

●

●●

●

●

●●● ●●●●●●

●

●

●

●

●

●●●●

●

●●

●

●

●

●●

●●●

●●

●●

●●

●

●

●●

●

●● ● ●●●

●

●

●

●

● ●●●●

●

●

●

●

●

●● ●

●●●

●

●● ●●●●

●

●

● ●●

●

●●●●

●

●

●

●●

●

●

●

●●●

●

●●

●

●

●

● ●

●

●●

● ● ●

●●●

●● ●

●

●● ●

●

●●

●

● ●●

●●

●

●●

●

●●●

●●●

●●

●● ●●

● ●

●

●●

●

●●●

●●●●

●

●

●

●

●

●●●

●● ●

●

●●● ●

●

●●●

●●

●●●

●●●

●●●

●●●

●●

● ●● ●

●●

●

●

●

●

●●

●●

●

●

●

●●●

●●

●

●

●

●●●●●●● ●●

●

●●●●

●●

● ●●

●●

●

● ●●

●● ●●

●

●●

●

●●●●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●●●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●●●●

● ●●

●

●

●●

●

●●

● ●●● ●

●

●

●● ●

● ●●

●

●●

●●

●●

●

●●

●●

●●

●

●

●●

●●

●

●

●●●●

●

●●

●

●●●

●●

●●●●●

●

●●●

●●● ●

●

●

●●●

●

●● ●

●

●

● ●

●

●

●●●

● ●

●●

●

●

●

●

●●

●●

●

●●●●

● ●

●●

●

●●●●

●

● ●

●●

●

●●

● ●

●●

●

●●

●

●●●

●

●

●

●

●

● ●●

●

●

● ●

●●●●

●

●●●

●

●

●

●

● ●

●●

●

●●

●

●●●

●●

●

●

●●

●

●●

●●●

● ●

●

●●

●●

●

●

●● ●

●

●●

●●

●●

●

●●

●●

●●

●●

● ●

●

●●

●

●

●

●

●

●

●●

●

●●

●●●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●●

●

●●

●

●

●●

● ●●

●

●

●

●

●

●

●●

● ●

●●

●

●

●●●

●

●● ●●

●●●

●●

●

●●

●●

●

●●●

●

●

●

●●●● ●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●●●●

●●

●

●● ●●

●●

● ●

●●●●

● ●

●● ●

●●●

●

●

●

●●● ●

●

● ●

●

●●

●

●●

●

●

●

●● ●●

●

●

●●

●

●●

●●

●●

●●●●

●

●

●

●

●

●

●

● ● ● ●●

●

●●

●●

●

●

●

●

●

●●

●●●

●●

●

●●● ●●●

● ●●

●●● ●

●

●

●● ●●●

●●● ●

●

●

●

●

●● ●●

●

●

●

●

●

●● ●●

●●●

●

●

●

●

●● ●●

●

●

●●●

●●● ●●●

●

●

●●● ●

●

●

●

●● ●●

●

●

●●

●

●

●

●●

●

●●

● ●

●

● ●

●

●

●●

●

●● ●●

●●●

●

●

●

●●

●

●●

●●

●●

●

●

●

●●

●

●●

●

● ● ●●●

●

●

●●

●

●

● ●

●

●●●

●● ●●●●●

●

●

●●●

●

●

●●

●●

●●

● ●

●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●●●

● ●●●

●

●

●● ●

●●●

●●

●

●● ●●● ●●

●●

●●

●

●

●

● ●●

●

●

●●

●

●●

●●●

●●

●●

● ● ●

●●●●

●●

●

●

●

●●

●●●● ●●

●

●

●●

●

●●

●

●

● ●

●● ●●●●

●● ●●●

●●

●

●●

●

●●●

●

●

●

●●●

●

●●

●●●

●●

●●●●

●●●● ●

●

●

●●

●●●

●

●

●●● ●●

●

●

●●

●

●

●

●●

●●

●

●

●

●●●●

●●

●●●

●●●● ●

●●●●

●●●

●

●● ●

●

●●

●●● ●●●

●

●

●

●

●

●●●

●● ●●●●●

●

●

●●

●●●● ●

●●●

●

●

●

●● ●●

●●●●●

●●

●●

●● ● ●●

●●●

●

●●

●●●

●

●

●

●●● ●

●

● ●

●●●

●●

●

●

●●●●

●●

●●●●●

●

●●

●

●●●●

● ● ●●● ●

●●

●

●●

●●●

●●

●

●● ●●

●●●●●

●●

●●●

●

●

● ●●

●●

●

●

●

●●

●●●

●

●

●

●●

●

●

●

●

● ●●

●

●●● ●

● ●

●

●

●

●

●● ●

●

●●

●

●● ●●

●

●

●● ●

●●

●●

●●●

●●

● ●●

●

●●●●

●

●

●

●●

●

●

●●●

●

●

●

●●●●

●

●

●●

●

●●

●

●●●●

●●●

●● ●●●

●

●●

●●●

●

●

●●

●

●●●●

●

●

●

●

●

●●

●●

●

●●

●●

●

●

●

●●●

●● ●

●

●

●

●

●

●

●

●●

●●

●

●● ●● ●

●

●

●●● ●

●

●

●

●

●●

●

●●

●

●

●

●

●

● ●●●

●

●●

●

●●●

● ●●●

●●●

●●

●

●

●●●

● ●

●

●

●● ●●●

●

●●●

●

●

●

●

●●

●●

●●●

●

●

●

●

●●

●●

●●●

●

●

●●

● ●●

●

●

●

●

●

●

●

●●●

●

●

●

●

● ●

●●

●●

●●

●● ●● ●

●●

●

●

●●

●

●●

●

●●●

● ●

●

●●

●

●

● ●● ●●● ●●●●

●

● ●

●

●●●

●

●

●●●

●●●

●

●●●

●●

●●●●●●● ●

●

●● ●●● ●

●

●●●●●●

●●

●● ●●●

●

●

●

●●

●

● ●●●

●

● ●●●

●

●●

●

●●

●

●

●

●●●●

●●

●

●●

●

●●●●

● ●

●

●

●

●

●

●

●

●

●● ●

●

●

●●

●

●

● ●

●

●

●

●

● ●●●

●

●●

●●●● ●

●

●

●

●●

●

●

●●

●●

●

●

●●

●

●● ●●

●● ●

●●

●

●● ●●

● ●

●

●

●

●●

●●● ●●

●

●

●

●

●

●● ●●●●● ●●

●● ●●

●●●●

●

●●

●

●

●

●●

●●

●●

●

●

●●●

●

●

●

●●●●●●

●

●

●● ●

● ●●

●●

● ● ●●

●

●●●

●●●

●●

●

●●

●● ●●

●

●●●

●

●

● ●

●

●●●●

●

●●

●●●

● ● ●●

● ●●●

● ●●●● ●●

●

●

● ●●●●

●

●●●

●

●

●●

●●

●

●

●●

●●

●●

●● ●●

●

●●●

●●●●

●

●●

●

●● ●●●

●●

●●

●●

●●●●

●●

●●● ●

● ●

●

●

●●

● ●

●

●●●●

●

●

●●●

●●

●

● ●

●● ●●●

●

●

●

●●●●

●●

●

●● ●●

● ●●

●

●

●●

●●

●

●●

●

●

●● ●

●

●●

● ●● ●

●

●● ●

●

●●●●

●●●●●

●●●

●

● ●●

●●

●

●●

●●

●

●●

●

● ●●●

●●

●

●●

●●

●●

●

●●

●●●

●

● ●

●

●

●●

●

●

●●

●●

●

●

●●●●● ●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●●

●

●●●●

●●

●●

●

●●●●

●

●●

●

●●

●

●● ●

●●

●●

●

●●

●●

●

●

●

●

●●●

●

●

●

●●

●

●

●●●● ●

●

●●

●●

●

●●

●

●

●

●

●●

●

●●

● ●●●●

●●

●●● ●● ●●

●

●●●●●

●

●

●

●●

●

●●

●

●

●●

●

●● ●

●●

●

●

●

●

●

●

●

●● ●●●●

●●

●

●

●● ●

●

●

●

●●●●●●

●

● ●

●

●● ●

●

●

●●

●●

●

● ●

●

●●●

●●

●

●● ●

●

●●●

●

●●

●

●

●●

●●

● ●●

●●

●

●●

● ● ●●●

●●● ●●

●

●

● ●●

●

●

●

●

●

●

●●

●●

●●● ●

●

●●

●

●●●

● ●●●

●●

●

●

●

●●●

●

●●●

●●● ●●●● ●●●●●

●●

●

● ●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●●

●●

●●● ●

●

●● ●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●●

●●●

●

●

●

●

●●

●●

●●●

●●

●

●

●

●

●

● ●

●

●●●

●●

●

● ●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●●●

●

●●●● ●

●

●

●●

●●●

●●

●

●●

●

●

●●●

●

●●

●

●●●

●

●

●

● ●● ●●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●●

●●

●●●

●

●

●

●

●

●

● ●

●

● ●●

●

●●

●

●

●

●

●●

●

●●●●

●●

●

●

●

●●

●

●●

●

●

●●

●

●●

●

●

●●

●

● ●

●

●

●●

●

●

●●

●●

●●

●

●

●● ●●●

●

●●●● ●●

●

●

●●●

●

●

●

●

●●●

●

●●

●●

●●

●●

●

●

●●

● ●●● ●

●●

●●●

●

●

●

● ●

●

●●

●●

●

●

●●

●●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●●●

●

●

●

●●

●

●

●

●●● ●

●●

●

●

●

●

●

●●●

●

●●

●●●● ●

●●●●●●●●●●●●

● ●

●●●● ●●●●

●

●

●●

●

●●

● ●●

●

●●

●

●

●●● ●●●●●

●

●

●

●

●

●

● ●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●●

●●

●

● ●

●

●●●

●●●●

●●

● ●

●

●

●

● ●

●

●

●●

●●

●

●

●

●●●●

●

●●●●

●

●

●

●●

●

●

●

●

●●● ●●●

●

●

●

●

●

● ●

●

● ●

●

●●

●

●

●●

●

●

●

●●

●

●

● ●●● ●

●●

●●●

● ●● ●●●

●● ●

●

●

●

●●●

●

●

●

● ●●

●

● ●

●●

●

●●●

●

●

●

●

●● ●●

●

●●●

●

●

●

●

●●

●

●

●

●● ●

●

●●●● ●●

●

●

●

●

●

●● ●●

●●

●

●●●

●

●

●

●●

●

●●● ● ● ●

●●●

●

●

●● ●●

●

●

● ●

●

●

●

●

●

●

●●●

●

●●●●

●●

●

●●

●●● ●

●●

●

●●●

●

●

●

● ●●●

●

●

●

●

●●

●●

●

●

●●●

●

●●

●

●

●

●● ●●

● ●

●●●

●

●●●

●●●●

●

●●●

●●●

●●

●●●

●

●●●●

● ●●●

●●●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●●●

●

●

●●●●●

●

●●●

●

●● ●

●

●● ●●

●

●

●

●●

●

●

●

● ●

●●

●●●● ●

●

●●●

●

●

●●

●●

●

●

● ●● ● ●

●●

● ●● ●●

●●

●●

●

●

●●

● ●

●

●● ●●

●●

●

●

●

●

●

●

●●●●●

●●

●●

● ●●●●

●

●●

●●

●

●

●

●

●

● ●

●●● ●

●

●

●

●

●

●

● ●

●

●●

●

● ●●

●

●●●

● ●

●●

●

●

●●

●●

●

●

●●●

●●

●

●

●

●

●

●●

●●

●

●

●●●

●

●●

●●●●

●

●

●

● ●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●●

●●● ●

●

● ●●

●●

●

●

●

●

●

●●

●

●●● ●

●●●●

●●

●

●

●●●●

● ●

●

●

●

●●

●

●

●●

●

● ●

●

●

●●

● ●● ●

●●●

●

●●

●

●●●●

●

●

●

● ●●

●

●●● ●

●●

●●

●●

●●●

●●●●

●●● ● ●●●●●

●●●●● ●●●

●● ●●

● ●●

●

●● ●

●

● ●●●

●●

●●●

●

●

●●

●●

●

●●●

●

●

●

●● ●

●

●

●

●

●

●

●●

● ●

●●●●● ●●

●●

●

●●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●●

●●●

●●

●● ●●●●●

●● ●

●

●

●● ●●●●

● ●●●●

●●●

●●

●●●

●● ●

●●●●

●●●

●●●●● ●

●●●

●●●● ●

●

●

●

●●

●●

●

●

●●

●● ●●

●

●●● ●

●●●●

●●

●

●●●●● ●●●●

●

●●●●

●

● ●●●

●●●● ●●●

●

● ●●●●

●

●● ●●●

●●●

● ●

●

●●

● ●●

●●●

●

●

●● ●●

●●●

●●● ●●●●

●

●●●

● ●

● ●

●

●

●

●● ●●

●●●●●

●●●●

●

●

● ●●

●

●

● ●●● ●●●

●

●

●●

●

●

●

●●

●●●● ●●●

●●

●

●● ●

●

●

●●

●

●

● ●

●●

●

●●●

●

●●

●

● ●

●

●

●●

●

●

●

●●

●● ● ●

●

●●

●●

●●

●●

●●

● ●

●

● ●

●

●

●

●●●●

●

●●

●

●

● ●●

●

●●●●●

●

●

●●

● ●

●

●● ●●●● ●

●●●

●● ●

●● ●●●

●

●

●● ●

●

●

●●

●

●

●

●

●●●

●●●●

●●

●●●●

●

●●●●●

●

●●

● ●

●●

●●●●● ●●

●

●●●

●●●

●

●●●

●● ●●●● ●● ●●

●● ●

●●

●●

●●●

●●●●

●●●●

●

●● ●

●●

●●

●

● ●

●●●

● ●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●●

●

●●

●

●●

●●

●

●●● ●●●

●

●●●● ●●●●

●

●

●●

●●●

●

●●

●

●

●●

●

●●

●●●●●●●●

●

●●● ●

●

●● ●

●

● ●●●

●

●

●

●

●●

●●●●●

●

●

●●● ●●

●

●●

●

●●●

●

●●●●

● ●●●

●

●

●●●

●●

●

●

●

●●● ●●

●

●●●●●

● ●

●●●

●● ●●

●

●

●●●

●

●●

●

●

●●

●

●

●

●

●

●● ●

●

● ● ●● ●●●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●●●

●●

●● ● ●●●

●●●● ●

●

●●● ●

●

●●

●

●

●●

●

●●

●●

●

●

●

●●● ●● ● ●●

●

● ● ●●

●●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●● ● ●●

●●

●

●

●●

●

●

● ●●●

●

● ●●

●

●

●

●●

●●

●● ●

●

●

●●

●

●●●●

●●

●

●

●●●

●

●

●

● ●● ●

●●●●

●●●● ●●●

●

●●

●●●

●●

●

●

●

●

●

●

●

●●●

●●

●●●● ●

●

●

●●●●

● ●●

●● ●

●

●●●●

●●

●

●

●●●

●●

●● ●●● ●

●●

●

●●

●

●

●● ●

●

●●

●

●●

●

● ●●● ●●

●

●

●● ●

●

● ●

●●

● ●

●●●●

●●

●●

●

●

● ●

●

●● ●

●

● ●●●●●

●●

●●● ●

●

●

●

●●

●

●

● ● ●●●●● ●●●

●● ●

●

●

● ●●

●● ●●

●● ●

●

●●● ●●

●

● ●●

●

●

●

●

● ●●●

●●

●● ●●●

●●●●

●●●●●

●● ● ●● ●●●

●●

●

●●

●●

●●●

●

●

●

●●●●

●●

● ●●●

●

●●●

●●

●●●●

●

● ●●●● ●

●●

●●

● ● ●

●●

●

●

●

●

●● ●● ●●

●●

●

●

●

●

●●●

●

●

●●

●

●

●

● ●●●

●

● ●

●

●

● ●

●

●

●●●

●

●

●

●●●●

●

●

●●

●●

●

●

●

●

● ●

●

●●

●●● ●●

●

●● ●●

●

●●

●

●

● ●●●● ●●

●

● ●●●

●●●

●●

●

●●●

●

●

●●● ●●●

●●●●●

●

● ●●● ● ●

●● ●●●

●●●

● ●

●●

●● ●●● ●●●

●●●●●●●

●●

●

●●

●

●

●

●●

●●

● ●●●●

●●

●●●

●●●

●

●

●

●

●

●●● ●●●

●●

●

●

●●●

●●

●●●●● ●● ●

●●

●● ●●● ●●●●● ●● ●

●

●

●●

●●●

●●●●●● ●●●●

●

●

●

●

●

●●

●

●

●● ●●●

●●

●

●●

● ●●●●●●

●

●●● ●●●

●

●● ●●●●●

●

●

●

●

●● ●●

●●●

●● ●●●

● ●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

● ●

●

● ●●

●

●

●●● ●

●

●●

●

●

●

● ●

●

●●●

●

●●

●

●

●

●●●●

●● ●●

●●● ●● ●

●

●●●

●

●

●●● ●

●● ●●●●● ●

●● ●●

● ●● ●●●●

●

●●●

●

●

●

● ●●

●●●

●●

●

●●● ●●

●●●●

●

●●●

● ●●●

● ●●

●

●

●●● ●●●●●●

●●●●●● ●●●●●●●

●

●●●●● ●● ●●

●

●●● ●●

●

●●●

●

●● ●●

●

●●●

●

●● ●●

●

●●●●

●● ●●●●

●

●

●●● ●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●● ●● ●●

●●●●

● ● ●●●

●● ●●

●●

●

●●

●

●●

●

●

●

● ●

●

● ●●●●

●●

●●●

●●

●●●

●●

●

●

●●●

●●

●

●

●●

●

●

●

●●

●

●●

●● ●

●

● ●●

●●●

●●

●

●● ●●

●

●●

●●

●

●

●

●●

●●●●

●●

● ●●

●

●

●

●

●●●

●

●

●●

●

●●●●

●

●●

●

●●●

●

●● ●●

●●

●●● ●●●

●●

●●

●●● ●

●

●●●●

●●

●

●●

●●● ●●

●

●●●● ●

●

●●

● ●●

●

●

●

●

●

● ●

●● ●

●

●●

● ●

●● ●

●

●

●

● ●● ●

●

●●● ●●●●●

●

●●●●

● ●● ●●

●

●●

●●

●

●

●

●

●●

●●

●●

●

●

● ●●

●

● ●●●●●●

● ●

●

●

●

●●●

●●

●●

●

●● ●●

●

●●

●

●

●●● ●

●

●

●●

●

●

●●

●●

●●

●●

●

●

● ●

●●●

●●●

●

●

● ●●●●

●●

●

●●●

●●●●

●●●●●●

●

●●●●

●●

●

●●

●

●●

●

●

●●● ●

●●

●

●

● ●

●●

●

●●

●

●

●● ●●● ●●

●

● ●●●●

●●

●

●

●

●●●●

●●●●

●●

●

●

●

●

●

●

●●●

●

●●●

●

●

●●

●

●

●

●

●

●●

●●●●●

●

● ●

●

●● ●

● ●●● ● ●●

●●●●●●●

●

● ●

●

●

●●

● ●

●

●

●

●

●●

●

●● ●●

●●

●

●

●

●●

●

●

●●

●

●●

●●

●

●

●

●●

●

●●●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●●●

●

●

●●

●

●●●●●● ●

●

●●

●●

● ●●●●

●●● ●●●

●

● ●

●

●

●

●

●

●●

●

● ●●●

●

●●● ●●●

●

●●

●

●●●

●●

●

●

●

●

●●● ●

●

●

●● ●

●●

●●

●●●

●●

●

●

●●

●●

●●

●

●

●●●

●

●

●●●●

●●●●

●●●

●

●●

●

●●

●

● ●●

● ●

●

●●

● ●

●

●●

●

●●

●

●●●

●● ●

●

●●●

●

●●●● ●

●

●

● ●

●

●●

●

●●

●

●

● ●●●

●

●

●

●●

●

●

●

●●

●●

●

●●

●●●

●●

●●

●

● ●●

●●

●

●

●●●

●●

●

●

●

●

● ●●

●

●●

●●●

● ●

●

●

●

●

●

●

●

●

●●●

●

●● ●

● ●●●

●

●●

●

●●●

● ●●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●● ●●

●●

●●

●

●●●

●●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●●

●

●●●●●●

●●

●●●

●

●

● ●

●

●●

●

●●

●●●

●

●●● ●

●

●

●●

●

● ●●

●

●●●

●

● ●●

●

●

●

●

●

●●

●

●

●●

●●●●●

●●

●● ●

●

●● ●●

●

●

●

●

● ●● ●●

●

●

●

●●

●●●

●●●

●

●

●●● ●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

● ●

●

●●● ●

●●●

●● ●●●●

●

●●

● ●●

●●

●

●●

●●●

●●●

● ●● ●●

●●

●●

●

●●●

●●●●

●

●

●

●

●

●●●

●●

●

●●

● ●●

●

●●●●● ●

●●

●●

●●

●

●

●

●

●

●

●●●●

●●

●

●

●●●●●

●

●●●

●

●●●

●

●

●● ●●●

●

●●

●●●●●

●

●

●

●

●●

●

●●●

●

●

●●●

●

●

●

●●

●●

●

●

●

●

●●

●●

●

●

● ●●

●

●

●●●●

●●

●

●

●●● ●

●●

●●

●●

●●

●

●●

●●

●●

●

●

●

●●●●

● ●

●

●●

●●

● ●●

●

●●

●●● ●

●● ●

●●

● ●

● ●●

●

●

●

●

●●● ●

●●

●

●

●●●●

●

●

●●● ●

●

●●

●

●

●●

●

● ●

● ●●

●●

●●

●●

●●●

●●

●

●

● ●

●

●

● ●

●●

●● ●●●

●

●●● ●

●● ●

●

●●

●

●

●

●

●

●●

●

●●

● ●

●

● ●

●

● ●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●●

●●

●

●

●

●●

●

●

●● ●●●

●● ●

●●

●

●●

●●

●

●●●

●

●

●

●●●

●

●

●

●

●●

●●●●●

●●●●●● ●

●

●

●●●

●

●

● ●●●

●

● ●

●

●●

●●

●●●

●

●

●●

●

●

●●

●●● ●

●

●

●

●

●

●

●

● ●●●

●

●●

●●

●

●

●●●●●●

●

●●● ● ●●

● ●●

●● ●

●

●

●●●

● ●

●

●

●

●●

●

●

●

●

●

●● ●●

●

●

●

●

●

●

●● ●●

●

●

●●●

●● ●● ●

●

●●

●

●

●

●● ●

●

●

●●

●●

●●

● ●

●

●

●

●

●

●

●● ●●●

●●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●●

●

● ● ●●

●

●●

●

● ●

●●● ● ●●●●

●●●

●

●

●●

●●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●●●

● ●●●

●

●●●

●

● ●●● ●●●

●●

●●

●

●

● ●

●●

●

●

●●

●●

●●●●

●●● ●

● ●

●

●●

●●●●

●

●

●●

●

●● ●●

●●●

●●

●

●●●

●

●

●●●

●

●●● ●

●●●● ●

●●●

●

●

●●

●●

●

●

●● ●●

●

●

●

●

●

●●

●●

●

●

●

●●●●

●●●

●●●●● ●

●●

●●●●

●

●●●

●

●●

●●● ●●●

●

●

●

●

●●●

● ●●●●

●

●

●●●●●●●

●●●

●

●

●

●● ●●

● ●●

●●

●●

●●

●● ● ●●

●●

●

●●

●●●

●

●

●

●●●

●●

● ●

●●●

●●

●

●

●●●●●

●

●●●● ●

●

● ●

●

●●●●

● ● ●●●●

●

●

●●

●●●

●●

●

●● ●●

●●●●●

●

●● ●

●

●

● ●●

● ●

●

●●

●●●

●

●

●

●●

●

●

●

●

●

●

●●● ●

● ●

●

●

●

●

●● ●

●

●●

●

●●●

●

●

●

●● ●

●

●●

●●●

●●

● ●●

●

● ●●●

●

●

●●

●

●

●●

●

●

●

●●●●●

●

●●

●

●●

●

●●●●

●●●

●● ●● ●

●

●●●●

●

●

●

● ●

●

●●●●

●

●

●

●

●

●●

●●

●●

●●

●

●

●

●●●

● ●●

●

●

●

●

●

●

●

●●

●●

●

●● ●● ●

●

●

●●● ●

●

●

●

●

●●

●

● ●

●

●

●●

●●●

●

●●

●

●●

●

● ●●●

●●●

● ●

●

●

●●●● ●

●

●

●● ●●●

●

●●●

●

●

●

●

●●

●●

● ●●

●

●

●

●

●●

●●

●●●

●

●

●●

● ●●●

●

●

●

●

●

●

●●●

●

●

●

●

● ●

●●

●●●

●● ● ●● ●

●●

●

●

●● ●

●

●● ●

● ●●● ●●●

●

●

●

●●●

●●●

●

●

●●●

●●

● ●● ●● ●

●

●● ●

●●

●

● ●●●●●

●●

●● ●●●

●

●

●

●●

●

● ●●●

● ●●●

●

●●

●

●●

●

●

●

●● ●●

●

●

●●

●

●●●

● ●●

●

●

●

●

●

●

●● ●

●

●

●●

●

●

● ●

●●

●

● ●●

●

●●

●●● ● ●

●

● ●

●

●●

●

●

● ●

●

●

●

●

● ●●●

● ●

●

● ●

●

●●

●

●●

● ● ●●

●

●● ● ●●

●●

●●

●●

● ●●

●●

●●● ●●

●●

●●●

●●

●●

●●

●

●●●

●

●

●

●●

●● ●

●

●

● ●●

●

●

●

●●●

●

●

●

●●

●●

● ●●

●

● ●

●

●●●

●●●

●●●

●

●●

●●

● ●●

●●

●●

●

●

●●

●

●●

●

●●

●

●

●●●

●

●

●

●●

●

●

●

●● ●

●

●

●●

●

●●

●●● ●

●

●

●

●

●●

●

●

●

●

●●

●

●●●●●

●●

●● ●● ●●

●

●●●

●●●

●

●●

●

●●

●

●

●●

●●●

●●

●●

●●

●

●●●●●

●●●

●

●

●

●

●

●●

●

●●

●●

●

●●

●●

●

●●

● ●●

●●●●

●

●

●●●●● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ●

●

●

●

●● ●●●●

●

●

●●

●

●

●

●

●●●

●

●

●

●

● ●

●

●

●●

●●●

●

●

●

●

●●

●●

● ●●●

●●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●

●●●

●

●●●●

●

●

●●

●●●

●

●

●

●●

●

●

●

●●●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

● ●

●

● ●●●●

●●

●

●

●

●

●

●●

●●

●

● ●

● ●

●

●●●

●

●

●●

●

● ●●

●

●● ●●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●●

●

●

●

●●● ●

●●

●

●

●

●

●

●

●

●

●

●●● ●

●

●

● ●

●

●●

●●●

●

●●

●

●●● ●●●●●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●●

●●

●

●●

●

●

●●

●

●●

●

●●●●●●

●●●

●●

●

●● ●

●● ●

●● ●●

●

● ● ●●

●

●●●●● ●

●

●

●

●

●

●

●

● ●

●

●●

●

●

●

●

●

●

●●

●

●

●●●●

●●

●●

●●● ●●●

●● ●

●

●●●

●

●● ●●● ●

●●●

●●

●

●● ●●

●

●

●

●●

●

●

●

●● ●● ●●

●

●●●

●●

●

●●

●

●

●●

●

●●

●● ● ●

● ●●

●

●

●● ●

●

●

● ●●

●

●

●

●

●

●●

●●

●

●●●

●● ●●●●

●●

●

●

●

●●

●●●●

●

●

●●●

●

●

●

●

●● ●●

●

●●

●

●

●

●●

● ●●

●●

●●

●●●● ●

●

●

●

●

●●

●●

●

●

●

●

●

●●●●●

●●

●

●

●

●●●● ●

● ●

●

●

●

●●●●●●●

●●●

●●

●● ●

● ●

●

●

●● ●●●●●

●●●

●●

●●

●●●

●●

●●

●

●

●

●

●

●

●

●

●

●●●●

●

●●●

●●

●●●

●

●●●

●

●

● ●

●

●●●

●

●●

●

●●

●

●

● ●●

●●● ●●

●

●

●●

●

●●●●

●

●●●

●

●●●●

●

●●

●

●●●●

●

●

● ●

●●

●●

●●

●

●●

●●

● ●

●●●

●● ●

●

● ●

●●

●

● ●●

●●

●● ●

●●

●

●

● ●

●

●●●●

● ●

● ●●●●●

●

● ●

●

● ●●

●

●●●

●●

●

●

●

●●

●● ●

●●

●

●●●

●

●

●

●●

●●

●

●● ●

●

●

●

●●

●●●

●

● ●● ●

●●

●

●

●

● ●

●

●

●●●●

●

●●●

●●

●● ●●

●

●

●

●●

●

●● ●●●●

●

●●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●● ●●

●

●

●

●

●

●

●

●●

●

●●

●

●●●

●

●●●

● ●

●●●

●

●

●●

●●

●

●

●● ●

●●

●

●

●

●

●

●●

●

●●

●

●●●

● ●●

●

●

●●●

●● ●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●●●●●

●

●●

●

●

●

●

●

●●

●● ●●●●● ●●

●●

●●

●●

●●● ●

●

●●

●

●

●●●●

●

●

●

●

●● ●●

●

●

●●●

●● ●

● ●

●●

●●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●●

●

●●●

●

●

●●●

●●●●

●●

●●●

●

●

●

●●●

●

●●●●

● ●●

●

●

●●●

●

●●

●

●●

●

●

●

●

●

●

●●●●

●●

●

●

●

●

●

●●●

●

●

●●●●

●●●

●●●

●

●

●●●

●●●

●●●●●

●●

●

●

●

● ●●

●●

●

●

●

●

●

●

●●● ●●●●●● ●

●

●●

●●● ● ●

●

●●

●

●●●

●

●

●●●

●

●●

●

●●

●

●● ●

●●●

●●●●

●

●

●●

●

●● ● ●

●

●

● ●

●●●●

●●

●

●

●●●

● ●●●

●

●

●

●●●

●

●●●

●●

●●●

●●

●●●

●

● ●●●●

●●● ●● ●

● ●●●

●●●●

●

●●●●

●

●●●

●

●●●

●●●

●●

●●●

●

●●

●●

●

●

●● ●●

●●● ●●

● ●●

●

●●

●●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●● ●●

●●● ●●●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●●●

●

●●

●●

●

●●

●● ●●●

●

●

●

●●●

●

●

●

● ●

●

●●

●

●

●

●

●

●●●●

●

●

●

●

●●●●

●

●●●●●●

●●●

●

●●

●

●●● ● ●●● ●●●●●

● ●

●●

●

● ●●

●● ●

●●●

●●●●

●●

●

● ●●

●● ●

●● ●

●

●

●

●

●●●● ●●

●

● ●

●

●

●

●●●●

●

●

●●

●●●●

●

● ●●●

●●●●

●●

●●● ●

● ●●●●●● ●●●

●

●

●● ●●●●

●

● ● ●●●●●

●●●●●●●

●

●● ●●●● ●

●

● ●●●●

●●

●●●●●

●

●●

●●●

●●●●

●●●

●

● ●●●●

●●

●

●● ●●

● ●●●

●

●●●

●●

●●

● ●●

●●

●●

●

●●●● ●●● ●

●●● ●

●●

● ●●● ●

●

●

●

●

● ●●

●

●

●●●●●●● ●

●

●

● ●●

●

●

●

●●

●●●●

●●●● ●

●

●●

●●●

●●

●

●

●

●

●●

●

●●

●

●●

●

● ●

●●●

●

●

●●

●●

●

●●

●

●

●●

●●

●●

●

●●●

●

●●

●●●

●

●

●

●●

●

●●

●●●●● ●●

●

●●

●

●

●●

●

●

●

●

●●

●●

●

●●

●

●

●

●●

●

● ●●

●

●

●

●●

●●

●

●●

●

● ●

●

●

● ●●

●

●●●● ●

●

●

●

●●

●●

●●

●

●●● ●●●● ●

● ● ●●

●

●● ●

●

● ●●

●

●●

●●●●●

●●

●●

●●●● ●

●●

●●

●●●

●

●

●

●

●●●

●

●●

●

●●●

●

●●

●●

●

●●● ●

●●●

●

●●●●●

●

●

●●●

●●

●●

●●●● ●●●

●

●

●

●●●

●●

●●

●

●

●

●

●

●

●●●

●

●●● ●●●● ●●

●

●●

●● ●

●●●

●

●●●

●

●●● ●● ●●●

● ●●

●

●

●

●

●

●●

●●●

●●●

●

●●●

●●●●

●

●●

●

●

●●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

● ●●●

●

●●

●

●

●

●

●

●

●

●●

●●●

●

●●●

●

●●

●

●

●●

●●●

●

●● ●●

●●●●

●

●● ●● ●●●

●● ●●

●

●●

●● ●

●

●

●●●

●

●●

●

●●

●●●

●●●

●●

●

●●●●

●

●●●

●

●● ●●

●

●

●

●●

●

●

●●

●●

●●

● ●● ●●●●●

●

●●●

●●●●

●

●

●● ●● ●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●●

●●●

●

●

●

● ●

●

●

●● ●●●

●

●

●

●

●●●

●

●●

●

●

●

●●

●

●●● ●●●

●● ● ●

●●●● ●●●

●

●● ●

●●

●

●●●

●●●

●●

●

●

●

●●

●

●

●

●

●

●

●●●●

●

● ●● ●●●● ●●●●

●●●

●

●

●●●

●●

●

●●

●● ●

●

●●

●●●

●●●

●

●

●

●●●

●●● ●

●● ●

●

●

●

●●● ●●

●●●

●●●

●

●

●

●●●● ●●●

●

●●●

●●

●

●

●

●

●●

●

●●

●●

●

●●●●

●

● ●●●●● ●●●●●

●●

●●●

● ● ●●

●●● ●

●

●●● ●●

●●

●

●

●●

●

●

●

●

●●

●●

●●●

●

●

●

●

●

●●●

●●●●●●

●●

●●

●●

● ●

●

●

●●●

●●●●

●

●

●●●●● ●

●

●

●

●●

●

●●

●

●●● ●

●● ●

●

●

●●● ●

●

●●

● ●●●

●●

●●●

●●

●

●●●●●

●

●

●

●

●

● ●●● ●

●

●●●

●

●●●●●●●

● ●●

●

●●

●●

●

●●

●●●

●

●●●

●

●

●●●

●

●

●

●

●

●

● ●

●

●●●

●

●

●●

●●

●●●●

●

● ●●●

● ●●●●

●

●

●●●●●

●

●●

● ●●●

●●●●

●

● ●●

●●● ●●

● ●

●

●

●●●

●

●●

●●

●

●●●●●

●●●

●●●

●●

●

●

●

●

●

●●

●

● ●

●

●

●

●

●

● ●●●●● ●●

●

●

●

●●● ●

●

●●

●●

●●●

●●

●●●

●●

●●

● ●●

●

●

●●●

●

●

●

●

●

●●●

●

●● ●●●●

●

●●

●●

●

●●

●

●

●

●●

●

●

●

●

●●●●

●

●●●

●

●●

●●●●

●

●

●●●

●●●●● ●●

●●

●●

●

●●

●

●

●●

●●

●● ●

●

●

●

●

●●●●

●

●

●

●●

●●●●● ●●

●●

●●

●● ●●●

●

●

●●

●●

●●●●● ●●

●●

●●●

●●

●●●

●●

●●

●

●

●

●●

●

●●●●●●

●●

●●●●●

●

●● ●

●●

●●●

●

●

●

●●●●

●●●

●●

●●

● ●

●

●

●●

●

●

●●

●

●

● ●●

●

●●●● ●●●

●

●

●

●

●

●●●●

●

●●

●●●●●●●

●

●

●●

●

●●

●

● ●

●

●● ●●

●

●●

●

●

●●

●

●

●● ●

●

●

●

●

●

●●●●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●●

●●●●●●

●

● ●

●

●●●

●●

●

●

●●

●●

●

●

●

● ●●

●

●●●●● ●●

●

● ●●

●

●●●

●● ●●

●●●

●

●

●

●●

● ●● ●●

●●●●●●●●

● ●●

●●● ●●●●

●●●

●

●●●●

●●●●● ●●

●

●●

●●●●

●

●● ●●●

●●●●●● ●

● ●

●●

●

●●●

●

●●

●

●

●

● ●

●●

●●●

●● ●

●●●

●

●

●●●

●

●●

●●

●

●

●

●●

● ●●●●

●●

●

●

●●● ●●

●

●●●●● ●●●

●● ●

●●

● ●●●●●●

●●●

●

●●● ●●●

●●

●

●●

●● ●

●

●●●●●

● ●●

●

●●

●

●

●

●

●

●●

●

●

●●●●●

●

●

● ●

●

●

●●

●

●●●●●●●●●

●

●● ●●

●● ●●●

●

●●● ●●●●●

●● ●●

●

●

●

●

●

●

● ● ●●● ●

●

●

●

●●●

●●●

●●●

● ●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●●● ●●●

●

●●● ●

●

●●

●

●

●

●

●●

●●●

●●●

●●

●

●

●

●● ●●

●●●●

●●●●

●●●●●

●

●●●

●

●

●●●●

●●

●

●● ● ●●●●

●●●●●●● ●●●●

●

● ●

● ●

●●

●●

●

●

●●● ●

●●●

●●

●

●

●● ● ●●

●●● ●

●●●●

●

● ●

●●●●●

●

●●●

●●●●

●

●

●●

●●●●●●

●●●●

●●●●

●

●

●

●●●

●●●●●● ●●

● ● ●● ●

●●

●●

●●● ●●●●●●

●

●●

●●● ●●●●●●

●

●

●● ●

●

● ●●●

●

●●●

●

●●

●

●●

●●

●

● ●●

●

●●

●●●●

● ●●●

●

●●

●

●●

●●

●●

● ●●●●

●●●●

●

●●●

●●

●

●

●●

●●

●

●

●

●●●●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●●

●●●

●●●●

●●●●

●

●

●●● ●●

●

●●

●

●

●

●●●● ●

●●●●●●●

●

●

●

●●●

●

●●

●

●

●●

● ●

●●●●

●●

●

●

●

●●●●●●●

●

●

●●

●

●

●

●●●

●●

●

●●

●●

●●●

●

●

●●●

●

●

●

●

●● ●

●●●

●

●

●

●●

●

●

●

●●

●

● ●●●

● ●●

●●

●●●●

●

●● ●●

●●●

●

●●

●

●●●●

●

●●

●●

●●

●●

●

●

●

●●

●●●

● ●

●●

●● ●●

●

●

●

●

●

●●● ●●

●

●

●

●●

●

● ●●● ●

●●●

●●

●

●●

●

●● ●

●

●●●●

●●

●●● ●●●● ●

●● ●

●●●

●●

●

●●

●

●

●

●●●

●

●●

●

●●●●

●●●

●

●●

●●

●●●●●

●●●●●

●

●

●

●●●

●

●

●●

●●●●

●

●

●

●●

●

●

●

●

●

● ●●

●

●

●●●●

●

●●

●

●●

●●

●●●

●

●●●

●●

●

●● ●

●

●●●●● ●●●

●

●

●●●

●

●

●●

●

●●

●

● ●

●

●●●●●

●●●

●

●●●●

●

●●●

● ●● ●

●

●

●

●●● ●●

●

●

● ●

●

●●

●

●

●

●●

● ●

●●●

●

●●●

●

●

●

●

●●

●● ●●

● ●●

●● ●●

●

●

●

● ●

●

●

●●

● ●●

●

● ●●

●● ●●● ●●

●●●●●● ●●

● ●

●●●

●

●

● ●

●●●

●

●

●● ●● ●

●●

●●

●●●●●

●

●●

●●

●● ●●

●●

●●●●●●

●●

●

●

●●

●

●

●●

●

●● ●

●

●

●●

●

●

●●● ●●

●●

●●

●●

●●

●●●

● ● ●

●●

●●

●

●

●● ●

●●

●

●

●● ●

●

●● ●●●

●

●● ●

●

●●

●

●●● ●

●●●

●●

●

●

●

●●●● ● ●●

●● ●

●

●●

●

● ●

●

●

●● ●●

●●●

●

●●

●●

●

●●●

●

●

●

●●●●

●●●●

●

● ●●●●●

●●

●

●

●

●●● ●

●●●

●

●

●●

●

●

●●

●

●

●

●●●●

●

● ● ●

●

●

●●

●

●

●

● ●

●

●●

●● ●●●

●●

●●

●

●

●

●

●●●

●●●● ●●

●

●●●●

●●●●●●

●

●

●

●●

●

● ●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●● ●●

●●

●

●

●

●

●●

●●

●

●

●●

● ●

●●

●●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●●

●●●

●●

●

●

●●

●

●●

●

●

●

●● ●●

●

●

●●

●●●● ● ●●●

●●

●● ●●●

●

●

●●

●

●

●●

●●

●

●●●

●

● ●●● ●

●

●●● ●●●

●

●

●●

●● ●

●●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●●●

●●

●●

●●●● ●

●

●

●

●●

●●

●●

●

●

●●●●

●

●●●

●●

● ●● ●●

●●●

●●

●

●●

●

●● ●

●

●● ●

●

●●●

●●

●

●●

●●

●

●●

●

●

●

●

●●

●●●

●

●●●

●

●●●

●

●●

●●●●●●

●

●●

●

●●

●

●

●●

●●

●

● ●●●

●

●

●

●

●●

●

●

●●

●●

●●

●

●●

●●●

●●

●●

●

●●●

●●

●

●

● ●●

●●

●

●

●

●

● ●●

●

●● ●

●

●

●●

●● ●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●●●●

●

●

●●

●

●●

●

●●

●●● ●●

●●

●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

● ●●●

●●

●●

●

●●●

●● ●

●

●

●

●

●

●

●

●●

●

●

●●●

●●

●

● ●

●

●●

●

●●●●●

●

●●

● ●●

●

●

● ●

●

●

●

●

●●

●

●●

●●● ●

●

●

●

●

●

●●●●

●

●

●●

●

●●● ●●

●●●●

●

●●

●

●

● ●●● ●●●● ●

●

●

●

●

●

● ●●●

●

●●

●

●

●

●●

●●●

●●

●●

●●

●

●

●●

●

●●●●●●

●

●

●

●

●●● ●●

●

●

●

●

●

● ● ●

●●●

●

●●●●●●

●

●

●●●

●

● ● ●●

●

●

●

●●

●

●

●

●●●

●

●●

●

●

●

● ●

●

●●

● ●●

●●●

●●●

●

●●●

●

● ●

●

●●●

●●

●

●●

●

● ●●

●●●

●●

●●●●

●●

●

●●

●

●●●

●●● ●

●

●

●

●

●

●● ●

●●●

●

●●

●●●

● ●●

●●

● ●●

● ●●

●●

●●

● ●

● ●

●●●●

●●

●

●

●

●

●●

●●

●

●

●

●●●

●●

●

●

●

●●

●●●●

●● ●

●

●●

●●●●

● ●●

●●

●

●●●

●●●●

●

● ●

●

●● ●●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●● ●●

●●●

●

●

●●

●

●●

●●●●●

●

●

●●●●●●

●

●●

●●

●●

●

●●

●●

●●

●

●

●●

●●

●

●

●●● ●

●

●●

●

● ●●

●●

●●●●●

●

● ●●

● ●● ●

●

●

●●●

●

● ● ●

●

●

●●

●

●

●●●

●●

●●

●

●

●

●

●●

●●

●

● ●●●

●●

●●

●

● ●●●

●

● ●

●●

●

●●

● ●

●●

●

●●

●

●●●

●

●

●

●

●

● ●●

●

●

●●

●●●●

●

●●

●

●

●

●

●

●●

● ●

●

●●

●

●●●

● ●

●

●

●●

●

●●

●●

●●●

●

●●

●●

●

●

●●●

●

●●

●●

●●

●

●●

●●

●●●

●●●

●

●●

●

●

●

●

●

●

●●

●

●●

●●●

●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●●●

●

●●

●

●

●●

●●●

●

●

●

●

●

●

●●

●●

●●

●

●

●● ●

●

●●●●

●●●

●●

●

●●

●●

●

● ●●

●

●

●

●●

●● ●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●●●●

●●

●

●●●●

●●

● ●

●●●●

●●

●●●

●●●

●

●

●

●● ●●

●

●●

●

● ●

●

●●

●

●

●

●● ●●

●

●

●●

●

●●

●●

●●

● ●●●

●

●

●

●

●

●

●

●●●●●

●

●●

●●

●

●

●

●

●

●●

●● ●●

●

●

●●●●● ●

●●●

●●●

●

●

●

●●●● ●

●●●●

●

●

●

●

●● ●●

●

●

●

●

●

● ●●●

●●

●

●

●

●

●

●●●

●

●

●

●●●

●●●●●●

●

●

●●●●

●

●

●

●● ●●

●

●

●●

●

●

●

●●

●

●●

●●

●

● ●

●

●

●●

●

● ●●●

●●

●

●

●

●

●●

●

●●

●●

●●

●

●

●

●●

●

●●

●

● ●●●●

●

●

●●●

●

●●

●

●● ●

●●●●

● ●●

●

●

●● ●

●

●

●●

●●

●●

●●

●

●

●● ●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●●●●

●

●

●●●●●

●●

●

●

●●● ●●● ●

●●●

●●

●

●

●●●

●

●

●●

●

●●

●●

●

●●

●●

●●●

●●●●

●●

●

●

●

●●

●●●● ●●

●

●

● ●

●

●●

●

●

●●

● ●●●●●

●●●●●

●●

●

●●

●

●●●

●

●

●

● ● ●

●

●●

●●●

●●

●●● ●

●●●●●

●

●

●●

● ● ●

●

●

●●● ● ●

●

●

●●

●

●

●

●●

●●

●

●

●

●●●●

●●

●●●

●●●●●

●● ●●

●●●

●

●●●

●

●●

● ●●●●

●

●

●

●

●

●

●●

●● ●●●

●●●

●

●

●●

●●● ●●

●●●

●

●

●

●●●●

●● ●●●

●●

●●

●●● ●●

● ●●

●

●●

●●●

●

●

●

●●●●

●

●●

● ●●

●●

●

●

●●●●

●●

●●●

●●●

●●

●

●●● ●

●●●●●●

●●

●

●●

●●●

●●

●

●●●●

●●● ●●

●●

●●●

●

●

●●●

●●

●

●

●

●●

●●●

●

●

●

●●

●

●

●

●

●●●

●

●●●●

●●

●

●

●

●

●●●

●

●●

●

●●●

●

●

●

● ●●

●●

●●

● ●●

●●

●●●

●

●●

●●●

●

●

●●

●

●

●● ●

●

●

●

●●●●

●

●

●●

●

●●

●

●●● ●

●●●

●●●● ●

●

●●

●●●

●

●

●●

●

●● ●●

●

●

●

●

●

●●

●●

●

●●

●●

●

●

●

●●●

●● ●

●

●

●

●

●

●

●

●●

●●

●

●●●●●

●

●

●●●●

●

●

●

●

●●

●

●●

●

●

●

●

●

●● ●●

●

●●

●

●●

●

●●● ●

●●●

● ●

●

●

●●●

● ●

●

●

●●● ●●

●

● ●●

●

●

●

●

●●

●●

●●●

●

●

●

●

●●

● ●

●●●

●

●

●●

●●●

●

●

●

●

●

●

●

●●●

●

●

●

●

● ●

●●

●●

●●

●●●●●

●●

●

●

● ●

●

●●

●

●●●

●●

●

●●

●

●

●●●●●●●● ●●

●

●●

●

●●●

●

●

●●●

● ●●

●

●●

●

●●

●●●●● ●●●

●

●●●●● ●

●

●●●●●●

●●

●● ●●●

●

●

●

●●

●

●●●●

●

●●●●

●

●●

●

●●

●

●

●

● ●●●

●●

●

●●

●

●●●

●●●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●●

●

●

●

●

●●●●

●

●●

●●● ●●

●

●

●

●●

●

●

●●

●●

●

●

●●

●

●●●●

●●●●●

●

●●●●

●●

●

●

●

●●

●● ●●●

●

●

●

●

●

● ●●● ●● ●●●

● ●●●

●●●●

●

●●

●

●

●

●●

●●

●●

●

●

●●●

●

●

●

●●● ●●●

●

●

●●●

●●●

●●

●●●●

●

●●●

● ●●

●●

●

●●

●●●●

●

●●

●

●

●

●●

●

● ●●●●

● ●

●● ●● ●●

●● ●●

●●●

●● ● ●●

●

●

●●●●●

●

●●●

●

●

●●

●●

●

●

●●●

●●

●●●●

●●

●●●

● ●●●

●

● ●

●

●●●●●

●●

●●

●●

● ●●●

●●

●●●●

●●

●

●

●●

●●

●

●●●●

●

●

●●

●●

●

●

●●

●●●● ●

●

●

●

●●●●

● ●

●

●●●●

●●●

●

●

●●

●●

●

●●

●

●

●●●

●

●●●● ●●

●

●●●

●

●●●●

●●● ●●

●●●

●

●●●

●●

●

●●

● ●

●

●●

●

●●●●

●●

●

●●

●●

●●

●

●●

●●●

●

●●

●

●

●●

●

●

● ●

●●

●

●

●●●●●●

●

●

●

● ●

●

●

●

●

●●

●

●

●●

●●

●

●●●

●

● ●

● ●

●

●●●●

●

●●

●

●●

●

● ●●

●●

●●

●

●●

●●

●

●

●

●

● ●●

●

●

●

●●

●

●

● ●●●●

●

●●

●●

●

●●●

●

●

●

●●

●

●●

●●● ●●

●●

●● ●●●● ●

●

●●●

●●●

●

●

●●

●

●●

●

●

●●

●

●●●

●●

●

●

●

●

●

●

●

●●●●

●●

●●

●

●

●●●

●

●

●

●●● ●

●●

●

●●

●

●●●

●

●

● ●

● ●

●

●●

●

●●●

●●

●

●●●

●

●●●

●

●●

●

●

●●

●●

●●●

●●

●

●●

●●●●●

●● ●●●

●

●

●●●

●

●

●

●

●

●

●●

●●

●●

●●

●

●●

●

●●●

●●●●

●●

●

●

●

●●●

●

●●

●●

●●●●● ●●●

●●●●

●●

●●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●●

●●

●●●●

●

●●●

●

●

●●

●

●

●

●

●● ●

●

●

●

●

● ●

●

●

●●

●●●

●

●

●

●

●●

●●

●●●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●● ●

●

●●

●●●

●

●

●●

●●●

● ●

●

●●

●

●

●● ●

●

●●

●

●●●

●

●

●

●●● ●●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●●

●●

●●●

●

●

●

●

●

●

●●

●

● ●●

●

●●

●

●

●

●

●●

●

●●●●

● ●

●

●

●

●●

●

● ●

●

●

●●

●

● ●

●

●

●●

●

●●

●

●

●●

●

●

●●

●●

●●

●

●

●●●

●●

●

●●● ●●●

●

●

●●●

●

●

●

●

●●●

●

●●

●●

●●

●●

●

●

●●

●●●● ●

●●

●●

●

●

●

●

●●

●

●●

●●●

●

●●

●●

●

●

●

●

●

●

●●

●●

●●

●●

●

●

●●●

●

●

●

●●

●

●

●

●●●●

●●

●

●

●

●

●

●●●

●

● ●

● ●●●●

●●●●●●●●●●●●

●●

●●●●●●●●

●

●

●●

●

●●

● ●●

●

●●

●

●

●●●●●●●●

●

●

●

●

●

●

●●

●

●

●

● ●●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●●

●

●

●●

●

●●

●●

●

●●

●

●●●

●●●●

●●

●●

●

●

●

●●

●

●

●●

●●

●

●

●

●●● ●

●

● ●●●

●

●

●

●●

●

●

●

●

●● ●●● ●

●

●

●

●

●

● ●

●

●●

●

●●

●

●

●●

●

●

●

●●

●

●

● ●●●●

● ●

●●●

●●●● ● ●

●●●

●

●

●

●●●

●

●

●

● ●●

●

●●

●●

●

●●●

●

●

●

●

●●●●

●

●●●

●

●

●

●

●●

●

●

●

● ●●

●

●●●● ●

●●

●

●

●

●

●●●●

●●

●

●●●

●

●

●

●●

●

●●

●●●●

●●●

●

●

●● ●●

●

●

●●

●

●

●

●

●

●

●●●

●

Year of construction

−0.087

3 4 5 6

● ●●● ●●●●●●●●●● ●●● ●●●● ●● ●● ●●●●●● ●● ●● ●●●●●●● ●●●● ● ●●● ●●●● ● ●●●● ●● ●● ●●●●● ●●●●●● ●● ● ●●●● ●● ● ●●● ●●● ●●●●● ●●● ●●●●●●● ●● ● ●●●● ●●● ● ●●● ● ●●●●●● ●● ●●●●●● ●●●● ●●● ●● ● ●● ●●●●● ●●● ● ●● ●●● ●●● ●●●●●●● ●●●● ● ●● ●●●● ●● ●●●●●● ●● ●●● ●●● ●●●●●●● ●●●● ●●●●●● ●●● ●● ●● ●●●●●● ●●● ●●●●●●● ●● ●● ●●● ●●● ● ●●●●●● ●● ● ●●●● ●●●●●● ●● ●●● ●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●●● ●●●●●● ●●●● ●● ●● ●●●●●●●● ●●● ● ●●● ●● ●● ●●●●●●● ●●●●● ●●●●● ●●● ● ●●●● ●● ●●●●● ●●● ●●●●●●● ●● ●● ●●● ●● ●● ●●●● ●●● ●●● ●● ●●●●● ● ●●●●● ●●●●●● ●●● ●●● ●●●●● ●●● ● ●● ●●● ●●●● ●●● ●●●●● ●● ●●●●●● ● ●● ●●● ●●●●● ●● ●●●●●● ●●●●●●● ●●●● ●●● ●● ●●● ●●●● ●● ●●● ●●●●●●●●●●●●● ●●● ●●● ●●●●● ●● ●●● ●● ● ●●

●●●● ●● ●●●● ●●●●●●●●● ●● ●●● ●●●●●●● ●● ● ● ●●●● ●●● ●●●● ● ●● ●●●●●●● ●● ●● ●● ●●●●● ● ●● ●●●● ●●●●●● ●●● ● ●●●●● ●

●●● ●●●●● ●

●●● ●● ●● ●●●● ●●

● ●●● ●●●● ●●

●● ● ●●● ● ●● ● ●●●● ●● ●●●

● ●●●●●

●●●●● ●●● ●●

●●● ●●

● ● ●● ●

●● ●●

●●●● ●●

●●●●

● ● ●● ●

●●

● ●●●● ●●●

● ●●●●●●●● ●

● ●●● ●●●

●●● ●●●●● ●●●●● ●● ●●●●● ● ●●● ●●● ●●● ● ●●●● ●

● ●●●●● ●●●●●● ●●

●●● ●●

●● ●● ●●

●

●● ●●●●●● ●● ● ●● ●●●● ●●●● ● ●●

●●●●● ●● ●●●

●● ● ●●● ●● ● ●●●●●●● ●●●● ●●●● ●●●●●●● ●●●● ●●● ● ● ●●● ●●● ● ●● ●● ●●●● ●●●

●●●●● ● ●●●● ●● ●●● ●● ●● ●●● ●●● ●● ●●● ●●●● ●●● ●●●● ●●

●●●● ●●

●● ●● ●●● ●●●●

●●●● ●●● ●●●●●

●●● ●● ●

●● ●● ●●

●●● ●●● ●● ●● ●● ●●●● ● ●●●●● ● ●●●● ● ●● ●●●●● ● ●●● ●●● ●●●● ● ●● ●● ●●● ●●●● ●●●● ●● ● ●●● ●●●● ●●● ●● ●●●● ●●● ●● ●●●●●●● ●●●●● ●●● ●● ●●●● ●●●●● ●●● ● ●●● ●● ●● ●●●●●●● ●●● ●●●● ●● ●●● ● ●●● ●●● ●● ●●●● ●● ●●● ●●● ●●●●● ●● ●●●●● ●●●●● ●●● ● ●●● ● ●●

●● ●● ●● ●● ●●●●●●●● ●●●●● ●●● ●●●● ● ●● ●● ● ●●● ● ●●●●●● ●●● ●● ●●●●●●● ● ●●● ●● ●●●● ●●●● ●●● ●●● ● ●● ●●●● ●●●● ●●●● ●●● ●●●● ●● ●● ● ●●●● ●● ●● ●●●● ●●● ● ●●● ● ●●●●●● ●●●● ●● ●● ●●●●●●●●●●●● ●●●●● ● ●●

● ● ●● ●●

●●● ●●●● ●

●●●● ●

●● ●● ●●●

●

● ●

●● ●●●●● ●

●● ●●●

●●● ●●

● ●●● ●● ●

● ●

● ●●●●●

● ●●●● ●● ●

● ● ●●● ●● ●●● ●●●●● ●●

●●●●●●●●

●●● ●● ●

● ●●●●●●

●● ●● ● ●●●● ●

●●●●●

●●●●

● ●●●

● ●●● ●●●● ●●● ●

●●● ●●

●●● ●●● ●●

● ●● ●●

● ●● ●●●● ●●

● ●●●●●

● ●● ● ●●●● ● ●●●

● ●●● ●●●

● ●

●● ●●●●●●

● ●●●●●● ●●●

●●●●●●●●

●●●●●●●●

●●● ● ●●

● ●●●●●

● ●●● ●●● ●●●●●●

● ●●● ●●

●●●●●●

● ●●● ●● ● ●●●●

●●● ●●

● ●●●

●●

● ●● ●● ●

●●● ●●●●●● ●●

●●●●

●●●●●●

●● ●●●

●●●● ●●●●●

●●

●●● ●●● ●●

●● ●●● ●

●● ●●●●●● ●

●● ●● ●●● ●● ●●●

●●●●●● ● ●●

●● ●●●

●● ●●

● ●●●●●●

●●●●

●● ●●●● ●

●●●●●

●● ●●●●

●●● ●● ●●

●●●●●●●

●● ●● ●●

●● ●●●

●●

● ●●●● ●

● ●●

●●●● ●● ●●●●

● ●●●●●●

●● ●● ●●

● ●●

● ●● ●●● ●●●

●

●● ●●●●

●

● ●●●

● ● ●●● ●● ●● ●● ●●

●●●●

●● ●●●

● ●●● ●

● ●●● ●

●●●

●●●● ●●●

●●● ●● ●●●●● ●

● ●● ●● ●

●● ●

●● ●●●●●

●●●●●●●

●●

●● ●● ●● ●●●

●●● ●

●●● ●●●●

●● ●●●

● ●

● ●● ●●

●

●● ●●

●● ●●

●● ●

●●●● ●

● ●●

● ●●●●

●

●●● ●

● ●●● ●●

●● ●●

●● ●● ●●

●

●

● ●● ●●

● ●●●●

●

●● ●● ●● ●● ●●●●● ●

●● ●● ●●●

●●● ●●

●

● ●●

●●●●● ●● ●● ●●●●●

●● ●●●●

●●●

●●

●● ●●●● ●●●● ●●

● ●

●●●

●●●●●● ●

●●● ●

●● ●

● ●

●●●● ● ●

●● ●

●● ●● ●● ●

●●●

●●● ●●

●● ●●● ●●

●●●● ●●

● ●●

● ●●● ●

● ●●

●● ●●● ●

●

● ● ●

● ●●●●

●

●●● ●

● ● ●● ●●

●

● ●●●● ●

●●●● ●● ●● ●

● ●● ●●● ●

●

●● ●●●

●● ●● ●●

●●

● ●● ●●●

●●●●●●

●

●●

● ●●●● ●●● ●●●

● ●

●●●● ●●●●

● ●

●●●● ●●

●

● ● ●●●

●●●●

●●●

●●● ●● ●●

●●

● ●●●●

● ●●

●●●●● ●●●

●● ● ●

● ●●

●●●●●

● ●

● ●●

●

●●

●● ●● ●●

●● ●●

●

●●●●● ● ●●● ●●

●

●●

● ● ●●●

●● ●●

●●●●●●

● ●● ●●●

● ●● ●●●

● ●●●● ●●●● ● ●● ● ●

●●●

● ●●● ●● ●

● ●

●●●●● ●●●●●●●

●●●● ●●● ●

● ●●● ●

● ●● ●●●● ●

●● ●●●● ●● ●●●●●

● ●●●●●● ●●●● ●●●●●●●●● ● ● ●● ●●

● ●●● ●●

● ● ●●● ● ●●● ● ●●●● ●● ●●●

●●● ●●●●● ●●●● ●

● ● ●●● ●● ● ●●● ●●● ●●●●●

●●●● ●● ●●●

● ●●●●●●●● ●●●●●●

●

● ●● ●●●● ●●●●●● ●● ●

● ●● ●●

●● ●

● ●●●

●● ●●● ●●● ●● ●● ● ●●● ● ●● ●● ● ●●● ●●● ● ●●●● ●●●●● ●●

●●●

● ●●●●●●

●● ●●●● ●●● ●● ●●● ●●●●●● ●●● ●●● ●

●●● ● ●●●

● ● ●●

●●● ●● ●●

●●

●●● ●●

●● ●●● ●●

●●● ● ● ●●

●● ●●●● ● ●●● ●● ●● ●●●●●●

● ●● ●● ●●●●● ●● ●●

●●● ●● ●●●●●●●● ●

● ●●

●●● ●● ● ●●●● ●●

● ●●●●

●●●●● ●●● ●●●●●● ●● ●●● ●●●●

●●● ●●●● ●●●●● ●●

●● ●●● ● ●●●● ●●●●

● ● ●●

● ●

●●●●● ●●●

●●●● ●●

●●●●● ●●●●●●●●●●

●●●●●

●● ●●●●●

●● ●●●●●●●●●

● ●●●

●●● ● ●

● ●●●●

●●●●●●

● ●●●●

●●

●●●●● ●●● ●● ●●●●

●● ●●● ●

● ●●●●

● ●● ●●●●● ●● ●●

●● ●●●●●

● ●● ●●● ●●● ●● ●●●● ●

●●● ●● ●●●●●●●●

●●● ●● ●●

●●● ●

●● ●●●●

●● ●● ●

●●● ●●

● ●●●●

● ●●●● ●●

●● ●● ●●● ● ●●●●● ●●● ●●

●● ●● ●

●●●● ●●●● ●●● ●●●●●● ●●●

● ●● ●● ●● ●

●●●●● ●●●●● ●

● ●● ●●●●●●●● ●●● ●●●

● ●●●●● ●

●●●●●● ●●●●●●●●●●●

●●●●● ●●●

●● ●●●

●● ●●●●

●● ●●

●● ●●●●

●●● ●

● ●● ●●

● ●●●● ●●●● ●●● ●●● ●●●●

● ● ● ●●●● ●●●

● ●●●●

● ●●●●●●● ●●● ●●

● ● ●● ● ●●

●●●● ●●●

●●●● ●● ●

●●●●●● ●

● ● ●●● ●●

● ●●●

●●●●● ●

●●●●

●

● ●●●●●●

● ● ● ●●●

● ●● ●● ●●●● ●●●●

●●

●●●●●●● ●●

●●●●●

●●●●●●● ●● ● ● ●● ●● ●●●●● ●●

●● ●● ●●●

● ●●

●● ●●●●

●●● ●●●●

●● ●●● ●●

●● ●

●●● ●● ●● ●● ●●●

● ●● ●●●

●●●●

● ● ●●●●

● ●●● ●●

● ● ●●●

●●

●● ●● ● ●●●●●●●● ●●●●

●● ●●●

● ●●● ●●●●●●

● ● ●●●● ●● ●●●

●●●● ●●● ●● ●●●●●

● ●●

●● ●●●

● ●● ●

●●● ● ●

●

●●● ●

●●●●●

● ● ●●●● ●●

●● ●●●●

●●●● ●● ●●● ●●● ●● ●

●●

●●●●●●●●●

●●

●● ●●● ●● ● ●● ●

●●●●

● ●●●● ●

● ●●● ●● ●

●● ●●●●●●● ●●●

●● ●●● ●● ● ●●● ●● ●●● ●●●●●● ●●●●●●●● ●●●

● ●

● ●●●●●●

●● ●

●● ●● ●●● ●● ●●

●● ● ●● ●●

●●

●●●● ●●●

●● ●●●●● ●● ● ●

●● ●●● ●●●●

●● ●● ●● ●●● ●●●●● ●●●●● ●● ●● ●● ●●●●

● ●● ●●

●●●

● ●● ●●● ●

●● ● ●

●●

●● ●●● ●● ●●●● ●●●●●

●● ●

● ● ●● ●●

● ●● ●

●●●● ●●

●● ● ●●●

●● ●● ●●

●● ●●

●● ● ●●●

●●

● ● ●

● ● ●●

●● ●

●● ●● ●●

●●

●

●● ●●●●

●● ●●

●●●● ● ●

● ●●

●● ●●● ●

●●●● ● ●

● ●●●● ●●●● ● ●● ●

●● ●●●● ●● ●●● ●

●● ●● ●●●

●●

● ●

●●● ● ●●

●

●● ●● ●

●●● ●●●●

● ●●●● ●●

●

●●●●●●

●●●●

●●●●●

●●●●

●●●● ●

● ●

●●●● ●

●●●● ●●●

●●●●●●

● ●●●● ●●

●

●●● ● ●

●

●●● ●

●● ●●●●

●●●

● ● ●● ● ●●●● ●

●●● ●

●● ●● ●● ●

●● ●

●●● ●● ●●

●●●●●

● ●● ● ●●

●●●●

● ● ●●

●●● ●

●

●●●●● ●●

●● ●●●●

● ●●●●●● ●

●● ●●● ●

● ●●

● ●●● ●

● ●

●● ●● ● ●●

●●●●●●

●●●

● ● ●●●● ●●●

●● ●●● ●

●●

●● ● ●●

●

●●●● ●●

●●

●● ●●●

●●●● ●●●

●●●

●●● ●

●● ● ●

●● ●●

●●●●●●

●●●● ●● ●●● ●●● ●

● ●

●● ●●●●

● ●●●●

●● ●

● ●● ●●●

●●● ●

●●● ●● ●●

●●

●● ●●● ●●

● ●●● ●

●●●●●● ●

●●●● ●

●●

●●●● ●

● ●●●● ●●

● ●●●

●

●●● ●●●

●●●● ●●

●●●

●●● ●●●

● ● ●●

● ●●● ●

● ●●●●

●● ●●● ●● ● ●●●

●●●●● ●

●●

● ● ●● ●● ●● ● ●●●● ●● ●●●●● ●● ●● ●●●● ●

● ● ●● ●

●●● ●●

●●● ●

●●●● ●

●● ●● ●●

●●● ●● ●

●●●

● ●●●●● ●● ●● ●●● ●●

●● ●●●

●

●●● ●

●● ●● ●●●●●●●

●●●●

●●●● ●●● ●●

●●●●● ●●

●●

●●●●●

●●●● ●●

● ●●●

●●●

●● ●●

●● ●●

●●● ●●●● ● ●●●

●● ●● ●

●●●●●●● ●●●●●

●● ●●●

●● ●

●● ● ●●●● ●● ●●●

●●●●

● ●

●●● ● ●●

●●● ● ●● ●

● ●●●●●● ● ● ●●

● ●●●

● ●●●●●

● ●●● ●

●●● ●●●

● ●●●● ●●

●● ●● ●●● ●● ●

●● ●● ●●●

● ●●●●

●●

●●● ●

●●●●● ●● ●●

●● ●●

● ● ●●●●

●● ●●●●●

●● ●●●

●●●●

●● ●●●● ●●● ●●●● ●● ● ●●●●

●●●●●●

● ●●●● ● ●●●●● ●●

● ● ●●●

●● ● ●●●●●

● ● ●●● ●

●● ●●●●●●●●●●

●●● ● ●

●●●●●● ●●●●● ●●●●●●●●

● ● ●●

● ●● ●●

●●● ● ●

●●● ●● ●● ●

●●●

● ●●● ●

●●●●

●● ●● ●

●●● ● ●

●● ● ●

●●

● ●● ●●●

●●● ●●●

●●●●● ●●● ●●●●

●● ●●● ●● ●● ● ●●

● ●●● ●

● ●● ●● ● ●●● ●● ●●●● ● ● ●●●

● ●●●● ●

● ●●● ●

●●●● ● ●

● ●●●

●●●

●●●●●● ●

●

●● ●● ●●●

●● ●

●●●●● ●●●●● ●

● ●●

●●●●●●●●

● ●

●● ●●●●● ●●●●●● ●●● ●●●

●●●●

●

●● ● ●● ●●●●

●● ●●●● ●●

●● ●●●● ●●●

●● ●●

●●●●● ●●

● ●●●●

● ● ●●●●●●● ●●

●

●● ●● ●

●●

●●● ●●● ●●●● ●● ●●● ●

●●●

● ●● ●●●● ●●●● ●

●● ● ●●

●●● ●●

● ●●●●●● ●● ●

●● ●●

● ●

●●●● ●●●●●●

●●●● ●

● ●●● ●

●●●●●●

●

● ●●●●●

●

● ● ●● ●●●● ● ●

●● ●●

●●●● ●● ● ●● ●●

●●● ●●● ● ●●

●●● ●● ● ●

●●

●● ● ●●●

● ●

● ●●●●● ●●●●●● ● ●●●

●●●●

● ● ●●● ●● ●

●● ●●●

● ●●●●●

● ●

●●

●●● ● ●●●

●● ●

● ● ●●●●

●●●

● ●●● ●●● ●● ●●●●

● ●● ●●● ●

●●● ●●●

●● ●●

● ● ●● ●●

● ●●● ●●●

●●●●● ●●

● ●●

●● ●

● ●●●● ●●● ●●

●●●●

●●●

●●●

● ●●

●●● ●●●

●●

● ●● ● ●●●●

●●●●●

●●●● ●● ●●●

●●

●●● ●●

● ●●●

● ●● ●●●●●● ●● ●● ●●● ●● ●● ●●●

●●●

●● ●

●

●●●● ●●● ●●

●

● ●●● ●

●

●●●● ●

●●

●●●● ●● ●●●● ●

●● ●

●●● ● ●●

● ●● ●

●● ●●●●●

●● ●

● ●● ●● ● ●● ●●● ●●● ●●

● ●●

●● ●● ●

●● ●●

● ● ● ●

● ●●●

●● ●●●●

●●●

●●●●●

● ●●● ●

●●● ●●●

●●●●●●●● ●●

● ●● ●●●●●●●●●●●

●●

●● ●●

●● ●●●

●●● ●

●● ●●

● ●●●● ●●●● ●● ●● ●●● ●

●● ● ●●

●● ●●

● ●●●●

●● ●● ●●

●●●● ●●● ●●●

● ●●● ● ●

● ●● ●●●●● ●●●●●●● ●● ●● ●● ●●● ●●●●● ●●● ●●●● ●● ●●● ● ●●●●● ●●●●●●●●●● ●●● ●●● ●●●●●●● ●● ●● ●●● ●●● ●●●●●●● ●● ●●●●● ● ●●● ●●●● ●●●● ●●● ●● ●● ●● ●●● ●●●● ●●●●● ●●●●● ●●●● ●● ●●● ●●●● ●●●● ●● ●●● ●●●●● ●● ●●●● ●●●●● ●● ●●● ●●●●● ●● ●● ●●● ●●●●● ●● ●● ●● ●●● ●● ●●●● ●●●

● ●●●●●●● ●●●●● ●● ●● ●●● ●●● ●●● ●●●●●●● ●●● ●● ●● ●●●●● ●●●●●●● ●● ●● ●●● ●●●●●●● ●●●●● ●●● ●●● ● ●● ● ●●● ●● ●●●● ●●●●● ● ●●●●● ●●●●●● ●●● ●●● ●●● ●●● ● ●●● ●● ●●●● ● ●●● ●●●● ●● ●●●●●●

●● ●● ● ●● ●●●●● ●●●●●● ●●● ●● ●●●● ●●●● ●● ●●● ●●● ●●● ●●● ●●●●●● ●●● ●●●●● ●● ●● ●● ●●● ●●●●● ●●●● ●● ●●● ●●●●●●●●●●● ●●●●● ● ●● ●●● ●● ● ●●● ●●●●●● ● ●● ●● ●● ●● ● ●●●●● ● ●● ●●● ●● ●● ●●● ● ● ●●● ●● ●● ●●●● ●●● ●● ●●● ●●●●●●● ●● ●● ●●●● ●●●● ●● ●●● ●●● ●●●●● ●● ●●● ●●● ●●●● ●●●●●●● ●●●● ●● ●●● ●● ● ●● ●●●● ●●●●●● ●●● ●●● ●●● ●● ●●●●● ●●● ●● ●●● ●●●●●● ●●● ●● ●● ●● ●● ●● ● ●● ●●●● ● ●●●●●●●●●●●●●●● ●●● ● ●●● ●●●● ●●● ●● ●●●● ●●●●● ●●●●● ●●● ● ●●● ●●●●●●● ●● ●●●●●● ●●●● ●●●●● ●●●●●●● ●●● ●●●● ●●● ●●● ●● ●●● ●●●●●●●● ● ●●● ●●●● ●● ●●● ● ●●●● ●●● ● ●● ●●●● ●●●● ●●● ●●● ●●● ● ●●●●● ●●●●●●●●● ●● ●●●●● ●●● ●●● ● ●● ●● ●● ● ●●●● ●●● ●●●●● ●● ●● ●●●● ●● ●● ●● ● ●●●● ●●● ●●●● ●● ●● ●● ●●● ●●●● ● ●●●● ●●●●● ●● ●●● ● ●●●●●● ●●● ●● ●●● ●●● ● ●●●● ●● ●●● ●● ●● ●●● ●● ●●●● ●●●● ●● ●●●●● ●● ●●●●● ●●● ●●●● ●●● ●●● ●● ●● ●● ●●●●● ●● ●

●●● ●●●●● ●●●● ●● ●●● ●●● ●●●●●●● ●●●●● ●●● ●●● ●● ●●● ●●●● ●●●● ● ●● ●●●●● ●●●● ●●● ●●● ●● ●●●● ●● ●● ●● ●●● ●●●●●●●● ●●●● ●●●●●● ●● ● ●●● ●●●● ●●● ●● ●●● ●●● ●● ●●● ● ●●● ●● ●●●●● ●●● ●● ●●● ●●● ●●●●●●●● ●●● ●● ●● ●● ●● ●●● ●● ●●● ●●● ● ●●●● ●● ●● ● ●●●●●● ●●●● ● ●● ●● ● ●●●● ●●● ●●●●●●●●●●●● ●● ●●●●● ●●●● ● ●●●● ●● ●● ● ●● ●●● ●●● ●●●●● ● ●● ●●●● ●●●● ●●●● ●●● ●●●●●●●●●● ●● ●●●●●● ●● ●● ●●● ●● ●● ●● ●● ●● ● ●● ● ●● ●●● ●●●●● ●●●● ●●●

● ●●● ● ●● ●●● ●●● ●●●● ●●●● ●●●●● ●●●●●●● ●●●●●● ●●●●●●●●●●● ●●● ●●●●● ●●●● ●●●●● ●●● ●●●●●● ●●●● ●●● ●●●● ● ●● ●● ● ●●●●● ●●●● ●● ●● ●●● ●●● ●●●● ●● ●●● ●● ●● ●● ●● ●● ●●●●●●●● ●●● ●● ●● ●●●●● ●●●●● ●●● ●●●● ●● ● ●● ●● ●● ●●● ●●●●●●●●● ●●● ●● ●● ●● ●● ●● ●● ●● ●● ●●● ●●●●●●● ●● ●●● ●● ●● ●●●●●●● ●● ●● ●●

●●●● ●●●● ● ●●●● ●● ●●●●● ●● ●●● ●● ●

●

● ●●●●●● ●●● ●●●●●● ●●●● ●●●●●

●● ●●●●●● ● ● ●●●●● ●● ● ●● ● ●● ●●●● ●●● ●●●●

●●●● ●●● ●●● ●● ●● ● ●●●●● ● ●● ●●●●● ● ●●●●●●● ●● ●●● ●●●● ●● ● ●●● ●●●● ●●●●● ●●● ●●●●● ●●●●● ●● ● ●●●●● ● ●●●●● ●● ●●●●●● ●●● ●●●● ●●●●●● ● ●●● ● ●●●●●●●● ●●● ●● ●●●● ●●●●●● ●●● ●● ●●● ●●●● ●●● ●●●●●● ●● ●●●● ●● ● ●●● ● ●● ●● ●●●● ●● ●● ● ●● ●● ●●● ●●● ●● ● ●● ●●●● ●● ● ●● ●●●● ● ●● ●● ●● ●●●● ● ●●● ●● ●● ●● ●● ●● ●● ●● ●●● ●●● ● ●●●●● ● ●●● ●● ●●● ●● ● ●●● ●●● ●● ●● ●●●● ●● ●● ●● ●●●● ●●●●● ● ●●●● ●●●● ● ●●

●●●●●● ●●● ●●●●● ●●● ●● ●●●●● ●

●●●

● ●● ●● ●

●●●● ●

●

● ● ●●●

● ● ●● ●

● ●

●●● ● ● ●

●●●●

● ●●● ●

●●

● ●●● ● ● ●●

●● ● ●●●●● ●

● ●●●●●●

●● ●● ● ●●● ●●●●●●● ●● ●● ● ●●● ●

● ●● ● ● ●● ●●● ● ●

● ●●

●● ●●● ●

●

●●●●● ●● ●● ●● ●● ●●●●● ● ●●

●●●● ●● ●●

● ●● ●●● ●● ●●●●●● ●● ●●●● ●●●●● ●●●●●

●●●●● ●● ●●●●● ●● ● ●●●● ●●●●● ●●●●● ●●

●●● ●●

●● ●● ●●●●●●●

● ●●● ●●●● ●

●●● ● ●

●●●

●● ●● ●● ● ●● ●● ●●●● ●●●● ●● ● ●●●●●● ●●●●● ● ●●●● ● ●●●● ●●● ●●● ●● ●●●● ●●● ●●●● ●●● ● ●●●● ●●●●●● ●●● ●●● ●●●●● ●● ●●●●●● ● ●● ●●●● ●● ●● ●●● ●●●●● ●● ●● ●●●●● ●●● ●●● ●● ●● ●● ●●●●●●● ●●● ●● ●

●● ●● ●● ●● ●● ●●● ●●●● ●●● ●● ● ●●●●● ●●●●●● ● ● ●●● ●●●● ●● ●● ● ●●● ●●●● ●●●●● ●●●● ● ●●● ●●●●● ●●● ●● ●●●● ●●●● ● ●●● ●● ●●● ●●●●●●● ●

● ●

●● ●● ●● ● ●

●

●●●● ●

●

●●●●● ●

●●●●●

●●●

●● ●● ●●●

● ●

●

● ●●● ●●●

●●

●● ●● ●●●●

●

● ●●●● ●

●● ●

●●● ● ●

●●

● ●●●

● ● ●● ●●●● ●● ●

● ●●●

●● ●●●

● ●●● ●

●● ● ●●● ●●

● ●●●●●

●●●●●●●● ●●

●●● ●●●

●

●●●● ●●

● ●●●●●● ●●

● ●● ●●● ●

● ● ●●●● ●●

●●●● ●●

●●●● ●●

● ● ●● ● ●●● ●●

●● ●●

●●●

●●

●●● ●

● ●●●

●●●

●● ●● ●●●●●

●

●● ●●●●

●●

●● ●●●●●

●●●● ●●●

●●●●

●●● ●

●● ●● ●●● ● ●●

●● ●●●●●

●●●●

●●

● ● ●● ●●

● ●● ●●

● ●●●

●●●

●●●

●●●●●

●● ●

● ● ●

●

● ●●●● ●

●●

●●● ●● ●

● ●●

●● ●● ●●

●●

● ●●●● ●

●

● ●●

●

● ●

●●● ●● ●●●

● ●●

●●

● ●●

●●

● ●●

●● ●

●●● ● ●●●● ●

●● ●

●● ●

● ●●

●●●●● ●●●● ●● ●

●●

●●●●

● ●

● ●

● ●●●

●● ●●

●

● ●

● ●● ●

●●●

● ●●

●

●●● ●

●

●●

●

●

● ●●● ●●

●● ● ●●

●●●● ●●●

●●

●●●

●● ●● ● ●●●

●● ●● ●●

●●

● ●

●●● ●●● ●●

●

●●●

●● ●● ●

● ●●●

●●

●●

●●● ●

●●

●●● ●● ●

●●●

● ●●●

●● ●●

●●● ●

● ●●

●

● ● ●

● ●●●●

●

●● ●

●● ●●

●

●●● ●

● ● ● ●●

●

●● ●●

●●● ● ●●● ●

● ●● ●●

●● ●●

●●● ●●

●

● ●●● ●●

●● ●●●●

●

●● ●●● ●●

●●

●●●● ●●● ● ●● ●●

●

● ●●

● ●●●

●

●●● ●

●

●●●●

● ●●

●●● ● ●●●

●● ● ●

●●

●● ●●●

● ●●

●●

●● ● ●●

●●

●

●●●● ●● ●●

●

●

●●

●● ●●●●● ●●●

●●

● ●●●●● ●●● ● ●● ●●

●● ●

●● ●●●

● ●

● ● ●● ●● ●●●●

●● ●● ●●●

●●

●●● ●●●

●●● ●● ●

● ●● ●●●● ●●●● ●●●● ●●●● ●● ●●●

●

●● ●●● ●●● ● ●●● ● ●●●

●●● ● ●●● ●● ● ●● ●●● ●●

● ●●●

●●●●●●●

●●●●●●●●● ● ●●●●

●

● ●●● ●●●● ●● ●● ●

●● ●●

● ●●

●●●●

●● ●● ●●●● ●● ●●● ●● ●●● ●●●●● ●● ●●●● ●●

●●

● ●●●● ● ●

●● ●●●● ●●●● ●●●● ●●●● ● ●●●

● ●●●●

●● ●

●●● ●●

●●

●●

●●●●●

●●● ● ●●

●● ●●● ●● ●●●● ●●

● ●● ●● ●●●●● ●● ●●

● ●●● ●● ●●●●● ●

● ●●

●● ●● ●● ●●● ●

● ●●●●

●● ●●● ● ●●● ●● ●● ●●

●●● ●●●●● ●●●

●●●●● ●●● ●●● ●● ●●●● ●●●

●●

●

●●●● ●●

● ●● ●●

●● ●●● ●●●

●●●

●● ●●●● ●

●● ●●●

● ●●●

●●● ●

●

●●●● ●

●●●●

●●

●● ●●●● ●● ●● ●

●●

● ●●

● ●●●●●● ●● ● ●

●● ●●●

●● ●●●●● ●●●●●● ●●●

●● ●●

●● ●●●●● ●

●●●

● ●● ●

●●●● ●●

●● ●●● ●●●

● ● ●●●

●●●●●●

●● ●● ●● ●

●● ●●●

● ●●●● ●●

● ●●● ● ●

●● ●●●

● ●●

● ●● ●●●

●●● ●●

●● ●●

●● ●●

● ●●

●●●

●●● ●

● ●● ●●● ●● ●●●●

●● ●●●● ●● ●

●● ●●

● ●●●● ●● ●●

●● ●●● ●

●●

●●●● ●● ●

● ●●

●●● ● ●

● ●●

●

●●●● ●●●●●●

● ● ●●

● ●● ●● ●● ●● ●●● ●

●

● ●

● ●● ●●

●● ● ● ●●● ●● ●●●

● ●

● ●

●●

●●

●● ●

● ●

●● ●●

● ●● ●●

●●●

●●

●

● ●●

●

●●●

● ●●●●

●● ●

●●● ●●●

●●●● ●

● ●● ●●

●●

●

●

●●●●

● ●

●● ●●

● ●

●●

●●●●●●●

●●

●● ●●●●● ●

● ●●

● ●● ●

● ●● ●

● ●●●●

● ● ●●● ●● ●● ●●● ●● ●●● ●●● ●●● ●● ●●

● ●

●● ●● ●●

●● ●

●● ●●●● ●● ● ●●

●●● ●●

●●

●●● ●● ●

● ●●●● ● ●●● ●

●● ●●● ●●●

●● ●●● ● ●● ● ●●● ●●●● ●●● ●●● ●●●●

● ●●●●

●

● ●● ●●

●● ●

●●

● ●●●●●●●●● ●

● ●

●● ●

●●

●●● ●

●● ● ●●●

●● ●● ●●

●

●● ● ●●●

●

●●●

● ●●●

●● ●

● ●●● ●●

●● ●●●

●● ●

●●●● ● ●

● ●●

● ●● ●

● ●● ●● ●

●●●● ●●●●●

●● ●●●● ●●● ●

●●●● ●●●

●

●

●●● ●●

●

●● ●

●●● ●●●●

●● ●●●

●

●● ●● ●

●●●

● ●● ●●

● ●●●

● ●●●●

●●

● ●● ●

●●● ●●●

●●●●

●●●●● ●●

●

● ●●

●● ●

●●●●

●● ●

● ● ●●●● ● ●● ●

●●●

● ●●●●

●● ●

●●●●●●●

●● ●

● ●● ●● ●

●● ●

● ● ●

● ●● ●

●●● ●● ●●

●●● ●●

● ●●●●● ●●

●● ●● ● ●

● ●●

● ●●● ●

●●

●● ●●● ●●

●●● ●

●● ● ●●●●●●

●● ●●●

● ●

●● ●●

●

●● ●● ●●

●

● ●

● ●●● ●●●

●●

●● ●

● ● ●

●● ●●

● ●● ●● ●

● ●●● ●● ●●● ●● ●

●●

●● ●●●

● ●●●

● ●

●● ●●●

●●●●

●●●● ●●●

●●

● ● ●●

● ●● ●●

●●

●●●● ●

● ●

●●●● ●

●

● ●●●●

●

●●● ●●

●●

● ●

● ●●

● ●●●●

● ●●● ●

●● ●● ●●

● ●● ●● ●

● ●

● ●●●●●● ●● ●●●●●●●● ●

●● ● ●

● ●● ●●

●●●●

●●● ● ●

●●●●●

●●● ●● ●

●●● ● ●● ● ●

●●● ●●● ●

● ● ●●●

●

●● ●

●●●●●●

●●● ●

●●●●

●●●● ●●● ●

● ●●● ● ●● ●●●●●

●● ●●

●● ●●

●

●● ●●

●●●●●

●● ●● ●

●●●●●●

●●●●

●●

●● ●● ●●●●●●

●●●

● ●

●●●

●●● ●●● ●

● ●●●●● ●● ●

● ●● ●

●●

● ●● ● ●

●●

● ●●●● ●●

●● ●●● ●

●● ● ●● ●

● ●●●●

● ●

●●● ●

●● ●●

●● ●●

●●●●

●●● ●● ●●

● ●

● ●●●

●●● ●●●● ●●

●●●●●

● ●●● ●●●

● ● ●●●

● ●● ●

●● ●●

●●●●●

●● ●

●● ●●●● ●●●●●

●● ●

● ●●

● ●●●●

● ●●

●● ●

●●

● ●●●

●● ● ●

●●

● ●●●●

●●●●● ●

● ●●● ●●● ●●● ●●● ●●

● ●●●

● ●● ● ●● ●●● ●●●● ●●

● ●●● ● ●●● ●

●●●

● ●●

●●●● ●● ●

●

●● ●● ●

●●●

● ● ●●●

●●●

●●●

●

●●●●● ●●● ●●●●●●

●●● ●● ●●●●● ●●●● ●●●●●●

●●

●●● ●● ●●

●●●● ●

●●●●● ● ●

●

●● ●●●

●●

●● ●●● ● ●●● ●● ●● ●

●● ●

● ●●●●● ●

● ●●

● ●●●●

● ●●●●● ● ● ●

●● ●●● ●

●●●●● ●● ●

●●●

● ●●●●

●●●

●

●●●●

●

● ●●●●●● ●

●● ●●

● ●●● ●● ●●● ●●

●●●●● ●

●● ● ●● ●

●●● ●●●

● ●

● ●● ● ●●●●● ●●● ●●●

●●●● ●

●●● ● ●

●● ●● ●

● ●

●●

●●● ● ●●●

● ●

●

●●

● ●●● ●●● ●● ●●●●

● ●●● ●

●● ●

●● ●●

●●● ●●

●● ●●●

●●●● ●

● ●●●

●●●

●●

●

●●● ●●

●●● ●●

●●

● ●● ●● ●●

●●● ●

●●●● ●●●

●

● ●● ●

● ●●●

● ●●● ●●● ●● ●● ●●● ●● ●● ●●●

●●

●● ●● ●●● ● ●●●●

●

● ●●● ●

●

● ●●● ●

●

●●● ●● ●●●●

●●●

●●●● ●●

● ●● ●

●● ●●● ●●

●● ●

● ●● ●● ● ●●●●● ●● ●●

● ●●

●●●● ●

●● ●●

● ● ● ●

● ●●●

●● ●●●●

●● ●

●● ●●●

● ●●● ●

●● ●●●

●● ●●● ●●●● ●

● ● ● ●●●●● ●●● ●●

●●

●●●

●●●●

●●● ●

●● ●●

●●●● ●●● ●● ●● ●●● ●

●● ● ●●

●● ●●

●● ●●●

●● ●●●

●●●● ● ●● ●●●

● ●●● ●

● ●● ●●● ●●● ●● ●●● ●● ●●●● ●●● ●●● ●● ● ●● ●● ●●● ●●●● ●●● ●●● ●●● ●●● ●●●● ●● ●●● ● ●●●● ●●● ●●● ●● ●●●●● ● ●●●●● ●● ●● ●●● ● ●● ●● ●● ● ●● ●●●● ●●● ●● ●● ● ● ●● ● ●●●●● ●● ●●● ●● ●● ●● ●●● ●●●● ●●●● ●● ●●● ●●● ●●●● ●● ●●●● ●●●●● ●● ● ●●●●● ●●● ● ●●●● ●● ●● ●●●● ●●●● ● ●● ● ●●●●●●

●●●● ●● ●●● ●●●● ●● ●●● ●●● ●● ●●●●● ● ●● ●● ●● ●● ●●●●● ●●●● ● ●● ●● ●●● ●●●● ●● ● ●●● ● ●●●● ●●●●● ●●●●● ●● ●●●● ●●● ●● ●● ●●● ●● ●●●● ●●● ● ● ●●● ●●● ●● ●● ●●●

●●● ● ●● ● ●● ●●● ●● ●● ●●●● ●●● ● ●●● ●●●● ● ●●● ● ●● ●●● ● ●●● ● ●● ●● ● ●●●● ●●● ●● ●● ●●●●● ●● ● ●● ● ●●● ●● ● ●●● ● ●● ● ●● ●● ●● ●●● ●● ●● ●● ● ●●●● ● ●●●● ●●● ●●●●●●● ● ● ●●●● ● ●●● ●●● ●●●● ●●●● ●●●● ●●● ●●● ●●● ●●● ●● ●●● ●●● ● ●● ●●● ●●●●●● ●●● ●●●●●● ●● ●● ●●● ●● ●●● ●●● ●● ●● ●● ● ●●●● ●● ●● ● ●● ●●●●● ●●●● ●● ●● ●● ●●● ●●●● ● ●●● ●●● ● ●● ●●●● ● ●●●●● ●

●●● ●● ●●● ● ●● ●● ●● ●● ●●●● ●●●● ●● ●●● ●● ●●●● ●●● ●● ● ●● ●● ●●●● ●● ●● ●●● ●● ●●● ●●●● ●●●●●●● ●● ●●●● ●● ●● ●● ●● ●● ●●● ●●● ●● ●●● ●●●●● ●●● ●●● ●● ●●●●●●●● ●●● ●●● ●●●● ● ●●● ●●● ●●● ●● ● ●● ●●●●●● ● ●●●● ●●● ● ●● ●● ●●●● ●● ●●● ●● ●●● ● ●●● ●●● ●● ●●●

●● ●●● ●●● ●● ●● ●●● ● ● ●●● ● ●●●●● ●●●●● ●● ●● ●● ●● ●● ●● ●●● ●●● ●●● ●●●● ●● ● ●●● ●●●● ●●● ●● ●● ●● ●●● ●●●● ●●● ●●● ● ●●●● ●● ●●●●● ●●●● ●●● ●●● ●●● ●●●● ● ●●●● ●●●● ●●● ●●● ●●● ●● ●● ●● ●● ●●● ●●● ●●●●●● ●●●● ● ●●● ●● ●● ●●● ● ●● ●●●

●● ●●●●● ●● ●● ● ●●●● ●●●

●

● ●● ●●●●● ●●● ●●●

●● ●●● ●● ●● ● ●● ●●● ●● ● ●●●● ●● ● ●●●●

0 50 100 150 200

● ●● ●● ●●●●● ●●●●●●● ●● ●● ●●●●●● ●●● ●● ●● ●● ●●● ●● ●● ●●●● ● ●●● ●● ●●●● ● ● ●● ●● ● ●●● ●●● ●●●● ●●● ●●● ●●●● ● ●● ● ●●●●●●●●●● ●●●● ● ●● ●● ●●● ●● ●●●●● ●●● ●●● ●●●●●● ●● ●●● ●● ●●● ●●●● ●●● ● ●● ●● ●●● ●●●● ●● ● ● ●●●●●● ●● ● ● ●●● ●●●●● ●● ●●●● ●●●●● ●●●● ●● ●●●● ●●● ●●●● ●●● ●●● ●●●●● ●● ●●● ●●●● ●● ●● ● ●●●● ● ●● ●● ●● ●●●● ●● ● ●●●●● ● ●● ●●●● ●● ● ●●● ●●● ● ●● ●●●● ●● ●●● ●● ●● ●●●● ●● ● ●● ● ●●● ● ●● ●● ●●●●●● ●● ●● ●● ●●● ●● ●● ●●● ●●●●●●●● ●●● ●●● ●● ●● ●●● ●● ● ●●●● ●●● ● ●●●● ●● ●●● ●● ●● ●● ●●● ●●●● ●● ● ●●●●● ●●●● ●●●●● ●●● ● ●● ●●● ● ●●●●● ●●● ●● ●●● ●●●● ●●● ●● ●● ●● ●●● ● ●● ●●● ●● ●●● ● ●●●●● ●●●●●● ●● ● ●● ●●● ●●● ●●●●● ●●● ● ●●● ● ●●● ●● ●● ● ●●●●●●● ●● ●● ●●●●●● ●●● ●● ● ●● ● ●●● ●●●● ●● ●● ●●

●● ● ●●●● ●●●● ●●●●● ●●● ●●●● ●●● ●● ●● ●● ●●● ●●●●● ● ●●● ● ●● ● ●● ●●● ●● ●●●● ●●●● ●●●● ●●●●●● ● ●●●●● ●●● ●● ●● ●● ●●

●●● ●●●●●●

● ●●●● ●●● ●●● ●●

●● ●●● ● ●●●●

● ●●●● ●●● ● ●● ●● ●●●● ●●

● ●●● ●●

● ●● ●●● ●●● ●

●●●●●

●● ● ●●

● ● ●●

●● ●●●●

● ● ●●

● ●●● ●

●●

●● ●● ●● ● ●

●●● ●●●●● ● ●

●●●●● ●●

●● ●● ● ●● ●● ●●● ●● ●●● ●● ●●● ●●●● ●●● ●●●● ● ●●

●● ● ●●●● ●●● ●● ●●

● ● ●●●

●●● ●●●

●

●●● ●●●● ●●●●● ●● ●● ●●● ●● ●●●

● ●● ●●● ●●● ●

● ●●●●● ●●●● ●● ● ●● ●●●●●● ●● ●●●● ● ● ● ● ●●●● ● ●●●●●● ●●● ●● ●● ● ●● ●●●● ● ●

●●●● ●● ●● ●●●●● ●●●●●●● ●● ●●●● ●●● ● ●●● ●● ●●●● ●● ●●

● ●● ●● ●

●●● ●●●●●● ●●

● ●●● ● ●● ●●● ●●

●●●● ●●

●● ●● ● ●

●● ●●●● ● ●●●●●● ● ●●● ●●●● ●●● ●● ●● ●●●●●● ●●●● ●● ● ●●●● ●● ● ●●●●● ●●●●●●●● ●● ●●●● ●● ● ●●●● ●● ●●● ●●●● ●●●● ● ●●●● ●●● ● ●●●●●●● ●● ● ●● ●●● ●●●● ●●●● ● ●● ●●● ●●● ●●● ●●● ●● ●● ●●●● ●●● ●●● ●●● ● ● ●● ●●●● ●●● ●● ●● ●● ●● ●● ● ●●● ●●●●● ● ●●● ● ●●●●

●●● ●● ● ●● ●●● ●● ● ●●●● ●●●● ●●●● ●● ●●● ●● ●●● ●●●●●● ●●● ●●● ●●●● ●●● ●●● ●●● ●●● ●●● ● ●●●● ●●●●●●● ●● ●● ●● ●● ● ●●● ● ●●● ●● ●● ● ●● ● ●● ●●●●● ●●● ●●● ● ●●●●●●● ●●●● ●● ● ●● ●●● ●● ● ●●● ●● ●●●●●● ●●●● ●●●

● ● ●●● ●

●● ●●●● ●●

●● ●● ●

● ●● ●●● ●

●

●●

●● ●● ●● ●●

●●●●●

● ●●●●

● ●●● ●●●

●●

●●●● ●●

●●● ●●● ●●

● ●● ●●● ●●●● ●●●● ●● ●

●●●● ●●●●

●●●●● ●

●● ●● ●●●

● ●● ●● ●● ● ●●

●●● ●●

● ●●●

● ●●●

●● ● ●● ●●● ●● ●●

●● ●●●

● ●● ●● ● ●●

● ●● ●●

●●●●●● ●● ●

●●● ●●●

●● ● ●● ●●● ●● ●●

●●● ●●● ●

●●

●●●● ●● ●●

●●● ●●●●● ● ●

●● ● ●● ●●●

●●●● ●●●●

● ●●●●●

● ●●●●●

●●● ●●● ●●●● ●●●

● ●●● ●●

● ● ● ●● ●

●●●● ●●●●● ●●

●●● ●●

●●●●

● ●

● ● ●● ●●

● ●●●● ●● ●●●●

● ●●●

● ● ●● ●●

●●● ●●

●● ●●● ●● ●●

●●

●● ●●●● ●●

● ●●● ●●●●● ●●●

● ●●●●●● ●

●● ●●● ●●●●● ●●

● ●●●

● ●●●●

●● ● ●

● ●●●● ●●

●● ● ●

●●●●●● ●

●●● ● ●

● ●●●● ●

●● ●●●● ●

● ●●●●●●

●●●●● ●

● ●●● ●

● ●

●● ●●●●

●●●

●●●● ●●● ●● ●

●● ●● ●● ●

●● ●●● ●

●● ●

●●● ●● ●●●●

●

●●●●●●

●

●●●●

● ●● ●●● ●● ●●● ●●

●● ● ●

● ●●● ●

●●● ●●

●●●●●

●● ●

● ●●● ●●●

●● ●●●● ●●● ●●

●●● ●●●

● ●●

● ●●●● ● ●

● ●●● ● ●●

●●

●●● ● ●● ● ●●

●●● ●

●● ●● ● ●●

● ●●●●

● ●

●●●●●

●

●● ●●

● ● ●●

● ●●

● ●● ●●

●●●

●●●● ●

●

● ●●●

● ●●● ●●

● ●●●

● ●● ● ●●

●

●

● ●●●●

●● ●●●

●

● ●●● ●● ●●●● ● ●●●

● ●●●● ●●

●●●●●

●

● ●●

●● ●●●●●● ● ●● ●●●

●●● ●●●

● ●●

●●

● ●●● ●●●●● ●● ●

●●

● ●●

●●● ●● ●●

●● ● ●

● ●●

●●

●●●●● ●

● ●●

● ●●● ● ●●

● ●●

●●●● ●

●● ●● ●● ●

● ●●●●●

●●●

● ●● ●●

●● ●

●● ●●●●

●

●●●

● ●●●●

●

●● ●●

●●● ●●●

●

●●●●● ●

●● ●●● ●●●●

●●●●●● ●

●

● ●● ●●

● ●● ●●●

●●

●● ●●● ●

● ●●●● ●

●

●●

●●●●●● ●●● ● ●

●●

● ●●● ●● ●●

●●

●●●●● ●

●

● ●●●●

●●● ●

●● ●

●● ●● ●●●

●●

●●●●●

●●●

●●●● ●●● ●

●●●●

●● ●

●●●●●

●●

●●●

●

●●

●● ●●●●

● ●●●

●

● ●● ●●●●● ●●●

●

●●

● ●●●●

● ●●●

●●●●●●

● ● ●●● ●

●●●● ● ●

●● ●● ●●● ●●●● ●●●

● ●●

●● ● ●●●●

●●

●●● ● ●●●● ●●●●

● ●● ●● ●●●

●● ●● ●

●●● ●●● ●●

●●●● ● ●● ● ●● ●●●

●●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●●● ● ●

●● ●● ●●

●●●●●●● ●●●●● ●●●●● ●●

●●●●● ●●●●

● ●●●

● ●●●●●● ●● ●●● ●●●●●● ●

● ●● ●● ● ●● ●

●●● ●●● ● ●●●● ●●● ●

●

●● ●● ●●●● ●●●●●● ●●

●● ●●●

●● ●

●● ●●

● ● ●●● ●●●● ●●●●● ●● ●● ●●●●●● ●●● ●●●● ●●●● ●● ● ●●

● ● ●

●●●●●● ●

● ●●●●● ●●●● ●●● ●●●● ●●●● ● ●● ● ●●

●●●●●●●

●●●●

● ●● ●● ●●

●●

●●● ●●

●●●●● ●●

● ●●● ●●●

● ●●● ●● ●● ●●●● ●● ●●●● ●●

● ● ●● ●●● ●● ●● ●●●

●● ●●● ● ●●●●●●● ●

● ●●

●●● ●●●● ●● ●●●

● ● ●●●

● ● ●●●●● ●●●●● ●●●●●●●●●● ●● ●●●● ●

●●● ● ●●●●●●● ●●●● ●●●●●● ●

● ●● ●

●●

●● ●●● ●● ●

●● ●●● ●

● ● ●● ●●● ●●●●●

●● ●●●●●●

●●● ●●●●

● ●●●●●●●●●●

●●● ●

●●●●●

● ●●●●

●●●●● ●

●●● ● ●

● ●

●● ●● ●●● ●●●●●●●

●● ●●● ●

●●●●●

●● ●● ●● ●●● ●●●

●●●● ● ●●

●●● ●●●●● ●● ●●●●● ●

●● ●●●● ●● ●●● ●●

●●●●●● ●

● ●●●

● ●●●● ●

●●● ●●

● ●●● ●

●● ●● ●

●●● ● ●● ●

●●● ● ●● ●●● ●● ●●●●● ●●

●●● ●●

● ●●● ●●● ●●● ●●●●● ●●● ● ●

● ●●●● ●● ●

● ●● ● ●●●● ● ●●

●●●●●●●●● ●● ●●●●●●

● ●●●●● ●

●● ● ●●●● ●●●● ●●

●●●●

●● ●●●● ● ●

●● ● ●●

●● ● ●● ●

● ●● ●

●●●●●●

●● ●●

●●●● ●

●● ●●●● ●●●● ● ●●●● ●● ● ●

●● ●●● ●● ●●●

●●●● ●

●● ●●● ●● ●●● ● ●●

● ●●● ●●●

● ●● ● ●●●

●● ● ●●●●

● ●● ●●● ●

●●● ● ●● ●

●●● ●

●●● ●● ●

● ●●●

●

●●● ●●●●

●●●●●

●● ●●●●●● ●●●●●

●●

● ●● ●●● ● ●●

●● ● ●●

● ●● ●● ●●●● ●● ●●●●●●● ●● ● ●

● ●● ●●●●

●● ●

●●● ●● ●

●●●●●● ●

●●● ●●●●

● ●●

● ●●● ●● ● ●●●●●

●●●●● ●

●●● ●

●● ●●● ●

● ●● ● ●●

●● ●●●

● ●

●● ● ●●● ● ●●● ●●●●● ●●

● ●● ●●

● ●● ● ●●● ●●●

● ●● ●●● ● ●● ●●

●● ●●● ● ● ●●● ●●● ●

●● ●

●●●● ●

●● ●●

●●●●●

●

●●●●

● ●● ●●

● ●●● ● ●●●

●● ●●● ●

● ●● ●● ●●●● ●●●● ●●

● ●

●● ●●●● ●●●

●●

● ●●●●● ●● ●●●

● ●● ●

●● ●● ●●

● ● ●●●●●

● ●●● ●●● ● ●● ● ●

●●● ●●● ●●● ● ●● ●● ●●● ● ●●●●● ●● ●●●● ● ● ●●

●●

●● ●●●● ●

● ●●

● ●● ●●●●●●● ●

● ●● ● ●●●

● ●

●● ●●●●●

● ●●● ●●● ●●●●

●●● ●● ●●● ●

● ●●●● ●● ● ●● ●●● ●● ● ●● ●●●● ●● ● ●●●●

●● ●●●

●●●

●● ●● ●● ●

● ●●●

● ●

●● ●● ●●●●●●●● ●● ●●

●●●

●●● ●● ●

● ●● ●

●● ● ● ●●

● ●●● ●●

●●● ●● ●

●● ●●

● ● ●● ● ●

● ●

●● ●

●●● ●

●●●

●● ● ●● ●

● ●

●

●●● ●●●

●● ●●

●●● ●●●

●●●

●●● ●●●

●● ●● ●●

● ●● ●● ●●● ●● ●●●

●● ● ●●●●●●● ●●

●● ●●● ●●

●●

●●

●●●●● ●

●

●● ● ●●

●● ●●● ● ●

●●● ● ●● ●

●

● ●● ●●●

● ●●●

●●●● ●

●● ●●

●●● ●●

● ●

●● ●● ●

●● ●● ●● ●

●●● ●●●

●●●● ●● ●

●

●●●● ●

●

● ●● ●

● ●●● ● ●

●●●

●●● ●●● ●●●●

● ●●●

●● ●● ●●●

● ●●

● ● ●● ●● ●

●●●●●

● ●●● ●●

●●● ●

●●●●

● ●●●

●

●● ● ●●● ●

●● ●● ●●

●●● ●●● ●●

●●● ●●●

● ●●

●● ● ●●

●●

●●● ●● ●●

●● ● ●●●

●●●● ●●●●●●● ●

● ● ●●● ●

● ●

● ●●● ●

●

●● ●●● ●

● ●

● ●●● ●

● ●●●● ● ●

●● ●

● ●●●

● ● ●●

● ●● ●

● ●●● ●●

●● ●● ●● ●● ●●● ●●

●●

●●● ●● ●

●● ●●●

●●●

● ● ●● ● ●

●●●●

●● ●● ● ●●

●●

● ● ●● ●● ●

●● ●●●

● ● ●● ●●●

●● ●●●

● ●

●●● ●●

● ●●● ●●●

● ●● ●

●

● ●●● ●●

●● ●●● ●

●●●

●● ●● ●●

●●● ●

●●● ●●

● ● ●●●

● ●●● ●●●●●●●

●● ●● ●●

● ●

●●●●● ●● ● ●● ●●● ●●●●●●●● ●●●●● ● ●●

●● ● ●●

●● ●●●

●●●●

●● ●● ●

●● ●● ●●

●● ●●●●

●● ●

● ●● ●● ●●

● ●●●● ●●●

●●●● ●

●

●● ●●

● ●●●●●●

●● ●●

●● ● ●

● ●●●● ● ●●

●

●●●● ●● ●

● ●

●● ●● ●

●●●● ● ●

● ●● ●

● ●●

● ●●●

●● ●●

● ●● ●●● ●● ●●●

● ●● ●●

● ●● ●●●●● ●●● ●

●●●●●

●●●

●● ●●● ●●●●●● ●

● ● ● ●

● ●

● ●● ● ●●

●● ●● ●● ●

●● ●● ● ●●●●● ●

● ● ●●

●● ●● ●●

●● ●● ●

● ●● ● ●●

●●● ● ●●●

● ● ●●● ●●● ●●

● ●●●●●●

● ●●● ●

● ●

●●●●

● ●●●●● ● ●●

●●●●

●● ●●● ●

●●● ● ●●●

●● ●● ●

● ● ●●

●● ●●●●●● ● ●● ● ●● ●● ●●● ●

●● ●● ●●

●●● ● ●●● ●●●●● ●

● ●● ●●

● ●● ●● ●● ●

●● ●● ●●

● ● ●●● ●●● ● ●●●

●●●● ●

●●●●● ● ●●● ●● ●●●● ● ● ● ●

● ●● ●

●● ●● ●

●● ●● ●

●● ●● ●● ●●

●●●

●● ●●●

● ●●●

● ●● ●●

●● ●●●

● ●●●

● ●

●● ●● ●●

● ●● ●●●

●●●● ●●● ●●● ●●

●●● ●●●● ● ●● ●●

●● ●● ●

●● ●● ●●●● ●● ●●● ● ●●● ●● ●

●● ●● ●●

●●●● ●

● ●● ● ●●

●●●●

●● ●

●●● ●●● ●

●

● ●●●●●●

●● ●

● ● ●● ●●●●● ● ●

●● ●

●● ●●●●●●

●●

●●●●●● ● ●●●●● ●●●●●● ●

●● ●●

●

● ●●●●●●● ●

●●●●● ●●●

●●●●● ●● ●●

●●● ●

● ●●●●● ●

●●●● ●

●●● ●●●● ●●●●

●

● ●● ●●

● ●

● ● ●●●●●●● ●●●● ●●●

●● ●

●●● ●● ●●●●● ●●

●● ●● ●

● ●●●●

●●● ●●● ●●●●

●●●●

● ●

● ●●●●●●●● ●

● ●●●●

● ●●●●

●● ●● ● ●

●

●● ●● ●●

●

●●●● ●●● ●● ●

● ●● ●

●● ● ●●●● ●● ●●

● ●● ● ●●● ● ●

● ●●●● ●●

● ●

●●● ●●●

●●

●● ● ●●●●●● ●●● ●●● ●

● ●●●

●●●●●● ●●

●● ●● ●

● ●●●●●

●●

● ●

●● ●●●● ●

●●●

●● ●●● ●

●●●

●● ●●● ● ●●●● ●●●

●● ●● ●●●

● ● ●●●●

●● ●●

●●● ●●●

●●● ●●● ●

● ●●●● ●●

● ●●

●● ●

●● ●● ●●● ●●●

●● ●●

●● ●

● ●●

● ●●

●● ●● ● ●

● ●

●● ● ●●●● ●

●●●● ●

●●●●● ●●● ●

● ●

●● ●●●

● ● ●●

● ● ●●● ●● ● ●● ●● ●●●● ●●●●●● ●

●●●

● ●●

●

● ●●●●●●● ●

●

●● ●●●

●

●● ● ●●

●●

● ●● ●●●●●●●●

● ●●

●● ●●●●

● ●● ●

●●●●●● ●

●● ●

● ●●●●● ●●● ●●● ●●●●

● ●●

●● ●●●

● ●●●

● ●●●

●● ●●

●●●● ●●

●●●

●●● ●●

● ● ● ●●

●●●● ●●

● ●●●●● ●●● ●

●● ●●●● ●● ●●●● ●●

● ●

●●● ●

● ● ●●●

●● ●●

● ●●●

●● ●● ●●● ●●●●● ●● ●●●

● ●●● ●

●●● ●

●●● ● ●

● ●● ● ●●

●● ●●● ●●●● ●

●● ●●●●

● ●●●● ●● ●●● ●●●●● ●● ●● ● ●● ●●●● ●●●● ●●● ● ● ●●●●● ●● ●●●● ●● ●●●●●●● ●●● ●●● ●● ● ●● ●● ●●●● ● ●● ● ●●●●● ● ●●● ●●● ●●● ●●● ●● ●●● ●● ●●●●● ●●● ●●● ●● ● ●●●● ● ●●●● ●●●●● ●●●●●● ● ● ●●● ●● ●●●● ●●● ●●●●● ●● ●●● ●●● ●●●● ●●● ●● ●●● ●●● ● ●●● ●● ●●● ●●● ● ●● ●●● ●● ●● ●●● ●●●●●● ● ●

● ●●● ●●● ●●● ● ●● ●●●●●●●●● ●● ●●●● ● ●●●●●● ● ● ●● ●●● ● ●●●● ●●● ●●● ● ●●● ●● ●● ●●●●●●●● ●● ●● ● ●●●●● ●●●● ●● ●●●●● ● ● ●●●●●●● ●●● ●● ●●●● ●●● ● ●● ●●● ●● ●● ●●●● ● ●●● ●●●● ●● ●● ●●●●● ●●

●●● ●● ●● ●●●● ●●● ●● ● ●●● ●● ●●●●●●●●● ●●●● ●●●●●●●●● ●●● ●●● ●● ● ●●● ●● ●●● ● ●●● ●●●● ●●● ●● ●●●● ●● ● ●● ●●● ●●● ●●● ●●●●● ●●● ●● ●●● ●●● ●●● ●● ● ● ●● ● ●● ●● ●●●● ●● ●● ●●● ●● ●●●●● ●●● ●● ●●●● ●●● ●●●● ● ●●●● ●●●●●● ●●● ●●● ●● ●● ●●●●● ● ●●●●● ●●● ●● ●●● ●● ●●● ●●●●●● ●●● ●● ●●● ●●● ●●● ●●● ●● ●●●● ●●● ● ●● ●●● ●●● ●●● ●●● ●● ● ●●●●●● ● ●●●● ● ●●● ●● ●●●● ●● ● ●●● ●● ●● ●●● ●● ●●●●● ●●●● ●●● ●● ●● ●●●●● ●●● ● ●● ●● ●● ●●●●●● ● ●●● ●●● ● ●●● ●●●●●●●●● ● ●● ● ●● ●●● ●●●●●● ●●● ● ● ●●● ●● ●● ●● ●● ●● ● ●● ●● ●● ● ●●●●●●●●● ●● ●● ●●● ●●●● ● ●● ●●● ●● ● ●● ●● ●●●●●● ●●●● ●● ●● ● ●●●●●●●● ●● ●● ●●● ●●● ● ●● ●●●● ● ●●●●●●● ●● ●●● ● ● ●●●● ●●●● ●●● ●● ●●● ●●●● ● ●●● ● ●● ● ●●●● ● ●●● ●● ●● ●● ●● ●●●● ● ●● ● ●●●●●● ●●● ●● ●● ●● ●●●●●● ●●● ●●●●● ●●● ● ●●●●●● ●● ●●● ●● ●●●● ●●●●● ●● ● ●●●●● ●●● ●●●● ●●●●●●● ●● ●●●●● ● ●● ●●●●● ●●● ●●● ● ●● ●●

● ● ●●● ●● ● ●●● ●● ●● ●●●● ●● ●● ● ●● ●●● ● ●● ●● ●●● ●● ●●● ●●● ●●●● ●●●● ●●● ●● ●●●● ●●●●●● ●● ●● ● ●●● ●●●●●● ●● ● ●●●●● ●●● ● ●●●● ●●●● ● ●●●●● ●●● ●● ●●● ●●●●●● ●●●● ●●● ●●●● ●●● ●●●●● ● ●●● ●●●● ●●●● ● ●●●●●● ●●● ●●● ●● ● ●●● ● ●● ●● ●● ●●● ●●● ●● ●●●● ●●●● ●●● ●●●● ●● ●● ●● ● ●●● ●●●● ●● ● ●●●● ●● ●●●● ●● ●●● ●●● ● ●● ●● ●● ●●● ●● ●●● ●●●● ●● ●● ●●●● ● ●●●● ●●●● ●●●● ●● ●● ●●● ●●●● ●● ●● ●●●● ●● ●●●● ●● ●● ●●● ●● ●●● ●● ●●● ●●●●●●●● ● ●●●●●

●● ●● ●● ●●●● ● ●●●●● ●● ●● ●● ●● ●●●● ● ●●● ●● ●● ●●●●●●●●●●●●●● ●● ●●●●●●●● ●●●● ●● ●● ●● ●●● ● ●● ●●●●●●●●● ● ●●●● ● ●●● ●●● ●● ●● ●● ●● ●● ● ●● ● ●●● ● ●●● ●●●● ● ●●● ●●● ●●● ●●● ●●● ●●● ●● ●● ●● ●●●● ● ●● ●●● ● ●● ●●●● ● ●●●●●● ●● ● ●●● ●● ●●● ●● ●● ●● ●●●● ●●●● ●● ●●●● ● ●●●●● ● ●●● ●●●● ● ●● ●●●●● ●●●● ●●● ●● ●●● ●● ●●● ●●●● ●●●● ●● ●●● ●●●● ●●● ●●

●

●●●●● ●● ●● ●●● ●● ●●● ●●●● ●● ● ●

● ●●● ●● ● ●●●● ●● ● ●● ●● ●●●● ●● ● ●● ●●● ●●● ●

●● ●●● ● ●● ●●●●●●●● ● ● ●● ●●●●● ● ●●●●●● ●● ●●● ● ●●●● ●● ●●●●● ● ●● ●●●● ●●●● ●● ● ●● ● ●● ● ●● ●●●●● ●●● ●● ● ●●●● ●●● ●● ●●●●●● ●● ●● ●●● ●●●● ● ●●●●●●● ●● ●●●● ●●● ●● ●●● ●● ● ●● ●●● ●● ●●●●● ●●●●● ●●●● ●●●● ●●●● ●●● ●●●● ●●● ● ●● ● ●●●●●●● ●●● ●●●●● ● ●●● ●●●● ●●●● ●●●●● ●●● ● ●●● ●●● ●● ●●●● ●● ●●●● ●●● ●● ●●●●●● ● ●● ● ●●● ●●● ● ●●●●● ●●●●●●● ●●● ●●● ●● ●● ●●● ●●● ● ●●●●●● ●●● ●●● ●● ●●● ●●●●● ● ●● ●●●● ●●●●● ● ●● ●●● ●● ●●●●● ●● ●●● ●●●●● ●● ●● ●●●●●●●● ●●●● ●● ●●●● ●● ●● ●●● ●●●●●● ●●● ●● ● ●●● ● ●● ●●●● ●●●●● ●● ●● ● ●●● ● ●● ●●●●●● ● ● ●● ●● ●● ●● ● ●●●●●● ● ●● ●● ●●● ●● ●●●●● ●●● ●● ●● ●● ● ●● ●●●● ●●●● ● ●● ●●● ●●●●●●●●●● ●●● ● ●●●●●● ●● ●● ●●●●● ●●● ●●●● ●● ●●● ●●●●●●● ●● ●●● ● ● ●● ●● ●●●

● ●●●● ● ●● ● ●● ●●●● ●●●●●● ● ● ● ●● ●● ● ●● ●●●● ●●●● ●●● ●● ●●● ●● ●● ●●●● ● ●●● ●●●●● ●● ●● ●● ●●● ●● ●●●●● ●●●●● ●● ●●●

● ●●●●●●● ●

●● ● ●●●●●● ● ● ●●

● ●●●●●●●●●

●●● ● ●●●●●●● ●●● ● ●●●●

● ●● ●●●

● ●●● ●●● ●●●

●●●●●

● ● ●● ●

●●● ●

● ●● ●● ●

●●● ●

● ● ●● ●

●●

● ●● ● ●● ●●

● ●●●● ●● ●●●

● ●● ●●●●

● ●●● ●●●● ●●● ●●●●● ●●●●●● ●●● ●●● ●● ●● ●●●●

●● ●● ●● ●● ●●●● ●

●●● ●●

● ● ●●● ●

●

● ●● ● ●● ●● ●●● ●●● ●●● ●● ●●●● ●

●● ●●●●● ●●●

● ●● ● ●● ●● ●● ●●● ●● ● ●● ●● ●●● ●●●● ●●●●● ●●● ●●●● ●● ●● ●●● ●●●● ●● ●●● ● ●●

●●●● ● ●● ●●●●● ●●●●● ●● ●●●● ●● ●●●●●●●●●●●●● ●●● ● ●

●● ● ●●●

●●●● ●●● ● ●● ●

●● ●●●●●● ●● ●●

●●●●● ●

●●● ●●●

●● ●● ●● ●● ●●●● ●●● ●●●●● ●● ● ●●●● ●●●●●●●● ●●● ● ●●●●●●● ●●●● ●●●● ●●●●●● ●●●●●●●● ●●● ●●●● ●●●●●●● ●● ● ●●●●●●● ●● ●●●● ●●● ● ●●●●● ●● ●●● ●●●●● ● ●●● ●●● ●●● ●●●●●●●● ●●●● ● ●●●●●●● ●● ●●● ●●● ●●●●●●●● ●● ●●●● ●● ● ●●●● ●●●●●●● ● ●●●● ●●

● ● ●● ●●● ●●●●● ●●● ●● ●● ●●● ●●● ● ●● ●●● ●●● ● ●●●● ●●● ●●●● ●● ● ●● ●● ●●●●●●● ●● ● ●● ●● ●●● ● ●● ● ●● ●● ●●● ●●● ●●●●● ● ●●●●● ● ●● ●●●● ●●●● ●●●●●● ●● ● ●●● ●●● ●●●●●●●● ●●●●● ●●●● ●●● ●●● ●●●●●●● ●●● ●●●

●●● ●● ●

●●●● ● ●● ●

●● ●● ●

●●●● ●●●

●

● ●

●●●● ●●● ●

●●●● ●

●● ●●●

●● ● ●●●●

●●

●● ●● ●●

● ● ●●● ●● ●

●● ●●●● ● ●●●●●●●●●●

● ●● ●●● ●●

●●●●● ●

●●●●●●●

●●● ●● ●● ●● ●

●●●● ●

●● ●●

●●●●

● ●●● ●●●●● ● ●

●●●● ●

● ●● ●● ●●●

●● ●● ●

●● ●● ●●● ●●

●● ●● ●●

●● ●●●●●●● ●● ●

●●●● ●●●

● ●

● ●● ●● ●●●

●●●●●● ● ●●●

● ●●● ●● ●●

● ●●●● ●●●

●● ●●● ●

● ●●●●●

● ●●● ●●● ● ●●● ●●

● ●● ● ●●

●●●● ●●

●●●●● ●● ● ●●●

● ●● ●●

●●● ●

●●

●●● ●● ●

●● ● ●● ● ●●●●●

●● ● ●

●●●●●●

●●●● ●

●● ●● ●● ●● ●

● ●

●●● ● ●●●●

●●●●● ●

● ●●●●●●● ●

● ● ●● ●●●●●● ● ●

● ● ●● ●●● ●●

●● ●● ●

● ●●●

●●●●●●●

●● ●●

●●● ● ●●●

●● ●●●

●● ●●● ●

●●● ●●● ●

● ●●●● ● ●

●● ●● ●●

●●●●●

●●

● ●●●●

● ●●

●●●●● ●● ●● ●

●●● ●● ●●

●●● ●●●

●● ●

●●●● ●●●●●

●

●●●●●●

●

● ●●●

●●●● ●●● ●●●●●●

●● ●●

●●● ●●

● ●●●●

●●● ● ●

● ●●

●●● ●●●●

●●● ●● ● ● ●●● ●

● ●●● ●●

●●●

●●●●● ●●

●● ●● ●● ●

● ●

●● ●●●●●● ●

● ●●●

● ●● ●●●●

●● ●● ●

●●

● ●●●●

●

● ●● ●

●●●●

●●●

●●●● ●

●●●

●●●●●

●

● ● ● ●

●● ● ● ●●

●● ●●

●● ●●●●

●

●

●● ●● ●

●● ●●●

●

●● ● ●●● ●●● ●●● ●●

●●●●● ●●

●●●● ●

●

●● ●

●● ●●●●● ●●● ●●●●

● ●●● ●●

●●●

● ●

●●● ●●●●●● ●●●

●●

●● ●

●● ●● ●●●

● ●●●

●● ●

● ●

●●● ●●●

●●●

●●●●●● ●

●●●

●●●● ●

●● ●● ● ●●

●●●● ● ●

●●●

●● ●● ●

●●●

●●●●● ●

●

●● ●

●● ●●●

●

● ●● ●

● ● ●●●●

●

● ●●●● ●

●● ●●●● ●●●

● ●●●●●●

●

●● ●●●

●● ●● ●●

● ●

●●●● ●●

●●●● ●●

●

● ●

● ●●●●●●● ●●●

●●

●●●●● ●● ●

●●

●●●●●●

●

●● ●● ●

●● ● ●

● ●●

●●● ●● ●●

● ●

●●●●●

● ● ●

●●●●● ●●●

●● ●●

●● ●

● ●●● ●

●●

● ●●

●

●●

● ●● ●●●

●●● ●

●

●● ●● ●●●●●●●

●

● ●

●● ●●●

●● ● ●

●●●●● ●

●●● ●●●

●●● ●●●

●●●●● ●●● ● ●●● ●●

●● ●

● ●●● ●●●

● ●

● ●● ●● ●●● ●● ● ●

●● ●● ● ●●●

●●● ●●

● ●●● ●●● ●

● ● ●●●● ●●●●● ●●

● ●●● ●● ●● ●● ●●●● ●●● ●●● ●● ● ●●●

●●●● ●●

●● ●●●●●●● ●●●● ●●● ●●●

● ●●● ●●● ●●

●●● ●

● ● ●●● ● ●●●●● ●●●●●●●●

●●● ● ●● ●●●

●●● ●●●●● ● ●● ●●●●

●

●●● ●● ●● ●●● ●●● ●● ●

● ●●●●

●● ●

● ●●●

●●● ● ●●●●●●●● ●●●● ● ●●● ●●● ●●●●● ●●●●●●●● ●●● ●

●●●

● ●●●● ●●

●● ●●● ● ●● ●●● ●● ●●●●● ●● ●●● ●●● ●

●●●●●●●

●●● ●

●●● ● ●●●

●●

●●●●●

●● ●●●●●

●● ●● ●● ●

●●●●● ●● ●●●● ●●●● ●●●● ●

●●● ●●● ●● ● ● ●●● ●

●●● ●●●● ●●● ●●●●

●● ●

● ●● ● ●● ●●●●●●

●●● ●●

●●●● ●●●● ●●●● ●●●●● ●

●●● ●●●●●● ●●

● ●●●●●● ●●●●●●●●●●●● ●●●

●● ●●

●●

● ●●● ●● ●●

● ●●● ●●

●●●●● ●●● ●●●●

● ●●●●●●●

● ●●●● ●●

●● ● ●●●●●●●●

●●●●

●●● ●●

●● ●●●

●●●●● ●

●●●●●

●●

●●● ● ●●●●●●●●●●

●●●●●●

●●●●●

● ●● ●● ●●●● ●● ●

●●● ● ●●●

● ●●● ●●● ●●●● ●●● ●●

●●● ●●● ●● ●● ●●●

●●●● ●●●

●● ●●

●●●● ●●

● ●●● ●

●●●● ●

● ●● ●●

● ● ●●● ●●

● ●● ●● ●●● ●●●●● ● ●●● ●

●●●●●

●●●● ●● ●●●●● ●●●●●●●● ●

●●● ●● ●● ●

●● ●●● ●●●●●●

● ●● ●●●●●●● ●●●●● ●●

●●●● ● ●●

● ●●●● ●●● ●●●●●

●● ●●

● ●●●●● ●●

● ●●● ●

●● ●●●●

●●●●

● ●●●● ●

●●● ●

●●● ●●

● ●●●●●● ●● ●●●●● ● ●● ●●

● ●●●●● ●● ●●

● ●●● ●

● ●●●●●●● ●● ●● ●

●● ●●● ●●

●●● ●●●●

●●●● ●●●

●● ●●●●●

●●●●● ●●

● ●●●

●● ●●● ●

● ● ●●

●

●● ●●●●●

●● ●● ●

● ●● ●●●● ●● ●●●●

●●

●●●● ● ●● ● ●

●●●● ●

●● ●● ●● ●● ●● ●● ● ●●●●● ● ●●●

●●● ●●●●

●●●

●● ●●●●

● ●●●●●●

●● ●● ●●●

●● ●

●● ● ● ●●●● ●●●●

● ●●● ●●

●●●

● ●●●●●

●● ●●● ●

●●●●●

● ●

● ●●● ●● ●●●●● ●●●● ●●

●● ●● ●

● ●● ●●● ●●● ●

●●●● ●● ●● ●● ●

● ●●● ●●●● ●●●●●●

● ●●

●●●●●

●●● ●

●●●●●

●

●●●●

●●●●●

●● ● ●●●●●

●●● ● ●●

● ●●●●● ● ●● ●●●●●●

●●

●●● ●●●● ●●

● ●

●●● ●● ●●●●●●

● ● ●●

● ●● ●●●

●●● ●● ● ●

●● ●●● ●●●● ● ●●

● ●●●● ●● ●●●● ● ●● ●●●●● ●●● ●● ●● ●●● ●●●●

● ●

●● ● ●●●●

●● ●

●●●● ● ●● ●●●●

●●● ●● ●●

●●

●●● ●●● ●

●●●● ●●●●● ●●

●●●● ●●● ●●

●● ●● ●● ●●● ●●●●● ●●●● ● ●●● ● ●●●● ●●

● ●●●●

●● ●

● ●● ●● ●●

●●●●

●●

● ●●●●● ●●●●●●● ● ●●

●● ●

●●● ● ●●

●● ●●

●● ●●● ●

●● ● ●●●

●● ●● ●●

●● ●●

● ●● ●●●

●●

● ●●

● ● ●●

●●●

● ● ●● ●●

● ●

●

●● ●●●●

● ● ●●

●● ●● ●●

● ●●

● ● ●● ● ●

● ●●● ● ●

●● ●●●●●● ● ●● ●●

●●●●●● ●●●●●●

● ●● ●●● ●

● ●

● ●

● ●● ●● ●

●

●●●● ●

●●●●●●●

●● ●●●● ●

●

●● ●●● ●

●●● ●

●●● ●●

●●●●

● ●●●●

●●

● ●● ●●

●● ● ●●● ●

●●●● ●●

● ● ●●●●●

●

● ●● ●●

●

●●● ●

●● ●●●●

● ●●

● ●●● ●● ●●● ●

●● ●●

● ●● ●●●●

●●●

●●●● ● ●●

●●●●●

●● ●●● ●

●●●●

● ●● ●

●●● ●

●

● ●●● ● ●●

●●●●●●

● ●●● ●●● ●

●● ●●● ●

●● ●

● ●●●●

●

●● ●● ●●●

● ●●●● ●

●●●●●● ●●

●● ●●

●●● ●●●

●●

●● ●●●

●

●●● ● ●●

●●

●● ● ●●

●●●● ● ●●

●●●

● ●●●

● ●● ●

●● ●●

●●● ●●●

● ●● ●● ●●●● ●●● ●

● ●

●● ●● ● ●

●● ●●●

●● ●

●●● ●●●

●●● ●

●●●● ●●●

● ●

●●● ●● ●●

● ●● ● ●

●●● ●●● ●

● ●●●●

●●

● ● ●●●

●●● ●●●●

● ●●●

●

●● ●●●●

●●●● ●●

●● ●

●●●● ●●

● ●● ●

● ●●● ●

●●● ● ●

●● ●●● ●●●● ●●

● ●● ●●●

●●

● ●● ●● ●●●● ●●●● ●●●●●●● ●●● ● ●●●● ●

● ● ●● ●

● ●● ● ●

● ●●●

● ●● ●●

●●● ●●●

●●● ●●●

●●●

●● ●●● ●●

●● ●●●● ●●

● ● ●●●

●

● ● ●●

●●●●●● ●

●● ● ●

●●●●

●● ●● ●●● ●

●

●● ● ●●●●

●●

● ●●●●

● ●●● ●●

● ● ●●

●●●

●●● ●

●●● ●

● ●● ●●●● ●●● ●

●● ●● ●

●● ●●●●●●● ●● ●

●● ●●●

● ● ●

● ●●● ●●●●● ●●●

●●●●

● ●

●● ●●● ●

● ●●● ●●●

●● ● ●●●● ●● ●●

● ●● ●

● ●● ●●●

● ●● ●●

●● ●●●●

●●●●● ●●

●●● ●●●●●● ●

●●● ●●● ●

●● ● ●●

●●

●● ● ●

● ●●● ● ●● ● ●

● ● ●●

●● ●●● ●

● ● ●●●●●

●●● ●●

● ●●●

● ●● ●●● ●●●● ●● ●●● ●● ●●●

● ●● ●● ●

● ●●●● ● ●●●● ●●●

●● ●●●

●● ●●●●●●

● ●●● ●●

●●● ●●●●● ●●● ●

●● ●● ●

● ●●●●●● ● ●● ●● ● ●●●● ●●

●● ●●

●●● ●●

●●● ●●

● ●●●●● ●

●●●

●● ●●●

● ●●●

● ● ●● ●

●● ● ● ●

●● ●●

●●

●● ●● ●●

●●● ●● ●

●●●● ●●●●● ●● ●

● ●●●● ●●●● ● ●●

●● ●●●

● ●● ●● ● ●●● ●● ●●●● ●● ●● ●

●● ● ●● ●

●● ● ●●

●●●●● ●

●●●●

●● ●

●●●● ● ●●

●

●●● ●● ●●

●●●

● ●● ●● ●● ● ●●●

● ●●

● ●● ●●●●●

● ●

●●● ●●●●● ●●●●●● ●●●● ●

●● ● ●

●

●●●● ●● ●●●

●● ●● ●● ●●

●●● ●● ●● ●●

●● ●●

●●●●● ●●

● ●● ●●

●●●●● ●● ●●● ●

●

●● ●● ●

●●

●●● ●●●●●●●●● ●●●●

●●●

●●●● ●●●●●● ●●

●●● ●●

● ●●●●

●● ●● ●●● ● ●●

●●● ●

●●

●●●● ●●●●●●

●●●●●

●● ●● ●

●●●● ●●

●

●● ● ●●●

●

●● ●●● ● ●● ●●

●●●●

● ●●● ●● ●● ● ● ●

● ● ●●●●● ●●

●● ● ●●●●

●●

●●●● ● ●

●●

● ●●●●●●●●●● ●● ●●●

● ● ●●

● ● ●●●● ● ●

●●● ●●

●● ●●●●

●●

●●

●●● ●●● ●

●● ●

●●●● ●●

●● ●

●●●●●●● ●● ●●● ●

●● ● ●● ●●

● ●● ● ●●

●● ●●

●●●● ●●

●●●● ●●●

●●●●●● ●

●● ●

●●●

●● ●●●●●●●

● ●●●●●●

●●●

●●●

●●● ●●●

●●

● ●●● ● ●●●

●●●●●

● ●●●●● ●●●

●●

● ●● ●●

●●● ●

● ●● ● ●● ●●● ●● ●● ●●●● ● ●● ●●●

●●●

●● ●

●

●●●●● ●●●●

●

● ●●●●

●

● ●●● ●

● ●

●● ●●●● ●●●●●

●● ●

●●●● ●●

●● ●●

● ●●●●●●

●●●

●● ● ●●●●● ●●●● ●● ●●

●● ●

● ●●●●

●●●●

●● ● ●

● ●●●

●● ● ●●●

● ●●

● ●●● ●

● ●●●●

●● ●●●●

●● ● ●● ●●●● ●

●●● ●●●●●● ● ●●● ●

● ●

● ●● ●

●●● ●●

●●● ●

●● ● ●

● ●●● ●●●● ● ●●●●●● ● ●

● ● ●● ●

●● ●●

●● ●●●

●●●●● ●

● ●●● ●● ●●●●

● ●●● ●●

●● ● ● ●●●● ● ●● ● ●●●● ●●●● ●●● ● ●●● ●●● ●●●●●● ● ● ●●● ●● ●● ●●●●● ●● ●● ●●● ●● ●● ● ●●● ●● ●●● ●●●● ● ●● ●●● ● ●●●● ●●●●● ●● ●●●●● ● ●● ●●● ●●● ● ●● ● ●●●●● ● ● ●● ●● ●●●● ●● ●●● ●● ●● ●●●● ●● ●●●●● ●●●● ● ●● ●● ●●●● ●● ● ●● ●●●● ●●● ●● ● ●●● ●●● ● ●● ●● ●●● ●● ●● ●● ●●● ●● ●●●●●● ●● ●● ●●

● ●●● ●●● ●● ●●● ●● ●●●●●●●●● ●● ●●● ●●● ●● ●● ●●● ●● ● ●●● ● ●●●●●●●●● ●●●●●● ●●●●●● ●●● ●●●●●● ●● ● ●● ●●●●● ●●●● ●●●● ●●● ● ●●●● ●● ●●●● ●●● ●●●●● ● ●● ●● ●● ●●●●● ● ●● ● ●●● ●● ●●●● ●● ● ●

●●●●●● ●● ● ● ●●● ● ● ●●● ●●● ●●●● ●●● ●●● ● ●●●●● ●● ●●● ●●●●●● ● ●● ●●●●●●●●●● ●●● ●●●● ●● ●●● ●● ●●● ● ●●● ●●●●● ●● ●●●● ●● ●● ● ●●●●● ● ●●●●●● ●●● ●●● ●●● ●● ●●●● ●● ●●● ● ●●● ●●● ●●●● ● ●● ●●● ●●●● ●●● ●●●●●● ●●●●● ● ●●●●● ● ●●● ● ●● ● ●● ●●● ●●●●● ●●● ●● ●●● ●●● ●●●●● ● ●●● ●● ●●●●● ● ●●●●●● ●● ●● ●●● ●●●●● ●● ● ●● ●● ●● ●●●●●● ●● ●●●● ●●● ●●●●●●● ●● ● ● ●●● ● ●● ● ●●●● ●● ●●●●● ●●●● ●● ●● ●●●●● ●●●● ●●●● ● ●● ● ●●● ● ●● ●●● ●● ●● ●● ●●● ●● ● ●● ●●● ●●● ●● ● ● ●●●● ● ●● ●● ●●●●●● ● ●●●● ●●● ●● ●● ● ●●● ●●●● ●● ●●● ●●●●● ●●● ● ●● ●●●●● ● ●● ●● ●● ● ●●● ●●●● ●● ● ● ●● ●● ●●●● ●● ●●● ● ●●● ●● ●●● ● ●●● ●●●●● ●●●●●●●● ●● ●●● ●●● ● ●● ●●●● ● ●● ●●●●● ● ●● ●● ●● ●●● ●●● ●●● ●● ●● ●● ●● ●●● ●● ●●●● ●●●● ●●● ●●● ●●● ●●● ●●● ●●●● ●●●● ●●● ●●● ●● ●● ●●●● ●● ●●● ●●● ●●●●●● ●●● ●●● ●● ●●● ●● ●● ●●● ●●● ●●● ●●●●● ● ●●● ●●●● ● ●● ●●●●● ●●●● ●●● ●● ●●●

●●● ● ●● ●●● ● ●● ●●● ● ● ●● ● ●●● ●● ●● ●●● ●● ●●● ● ●● ●● ●●● ● ●●●● ●● ●●●● ●●●● ●● ● ●● ●● ●● ●●● ●● ● ●●● ●● ●● ●● ●●●●● ● ●●● ●●● ●● ●● ● ● ●●● ●●●● ● ● ●● ●●● ●● ● ●● ●●● ● ● ●●● ●● ●●●● ●● ● ●● ●●●● ●●● ●●●● ● ●●● ●●●● ● ●●●●●● ●● ● ●● ●● ●● ●●● ● ●● ● ●● ●●● ●● ● ●●● ●● ●● ● ●● ● ●●●● ●●●● ● ● ●●● ● ●●● ●●●● ●●●● ● ●●●●●● ●● ● ●●● ●● ●● ●●●● ● ●● ●● ● ● ●●●●● ●●●● ●●● ●●●●●● ●● ●●● ●●● ● ●●●● ● ●● ●● ●● ● ●● ● ●●●● ●●● ●● ●● ●●● ●● ●●● ●● ●● ● ●●●● ● ●●● ● ●●●

●● ● ● ● ●● ●●●● ● ●● ●● ●● ●● ●●● ●● ●● ● ●●● ● ●●●●●●●● ●●●●●●●●●●●●●● ●●●●●●●●● ●●●● ●●●● ●● ●● ●● ●●●●●● ●●●● ● ● ●● ●●● ●● ●●●● ●● ●● ●● ●●● ●●● ●●●● ●● ● ● ●●● ● ● ●●●● ●● ●●●●● ●● ●● ●● ●●●● ●●● ● ●●● ●●● ●●● ●●●● ●●● ●●● ● ●● ●●●●●● ●● ● ●● ●● ●●●● ●● ●● ●● ●● ●● ●● ●●●●●●●● ●● ●●●●●●● ●●●● ● ●●●●● ● ●●●●● ●

●● ●● ●●● ●● ●● ●●●● ●● ●●● ●●● ●●● ●●

●

●● ●●●●●● ● ●● ●● ●●●●● ●●● ●●● ●

●● ● ●● ●●●●●●●●●● ● ● ●●●● ●●● ●● ●● ● ●● ●●●

9200 9600 10000

9200

9600

Zip code

Figure 2.7: Lower triangle: Living space, number of bedrooms, age, year of construction and zipcode against each other. Diagonal: Histograms of the variables. Upper triangle: Pearson correlationsbetween the variables.

In Figure 2.7, turnover time is excluded due to no correlation with the cash price.

We see that there seems to be some negative linear dependency between age andyear of construction. When the year of construction decreases the age increases,which intuitively makes sense. This is also seen in the Pearson correlation, whereage and year of construction have the most negative correlation of −0.50

The group of villas that lie beneath the diagonal line consists of villas where theage has been reset due to renovation.

2.1. Plots 9

Furthermore there seems to be no correlation between the living space and the age.We see the same tendency considering year of construction due to the dependencybetween age and year of construction.

We see that the highest Pearson correlation of 0.65 is between living space andnumber of bedrooms. However, to discard one of the variables to avoid multi-collinearity in the designmatrix, the correlation should be closer to 1.

There is no large correlation between the zip code and the remaining variables,when the Pearson correlation is less than 0.1.

Chapter 3

Multiple Linear Regression

This chapter is based on [James et al., 2015, pp. 71-72], [Davidson and MacKinnon,2003, pp. 55-60] and [Hayashi, 2000, pp. 6-8, 10-12, 15-17, 19, 27, 29-31].

Multiple linear regression is a model that expresses the value of the dependentvariable Y by conditioning on several independent variables x1, . . . , xp. This can bewritten mathematically as

y = β0 + β1x1 + · · ·+ βpxp + ε, (3.1)

where β0, . . . , βp are unknown model parameters and ε is a random error term.

We can interpret β j as the effect on Y as xj changes one unit and all the otherindependent variables are fixed. β0 is the intercept while β1, . . . , βp are the slopeparameters.

We are interested in predicting the value of Y, which can be done as follows

y = β0 + β1x1 + · · ·+ βpxp,

where y is the predicted value of Y and β0, . . . , βp are the estimated model param-eters. To find the β estimates we use the Ordinary Least Squares method (OLS).

3.1 Ordinary Least Squares

Let n be the number of observations. The matrix form of Equation (3.1) is

y = Xβ + ε, (3.2)

where y and ε are n× 1 vectors, β is a (p + 1)× 1 vector (since we have p slopeparameters and one intercept), and X is a n× (p + 1) matrix.

11

12 Chapter 3. Multiple Linear Regression

Assumption 3.1 The error term ε has conditional mean zero,

E [ε i | X] = 0, i = 1, . . . , n.

An implication of this assumption is that the unconditional expectation of the errorterm ε is equal to zero. By using the Law of Total Expectation we get

E [E [ε i | X]] = E [ε i] = 0, i = 1, . . . , n.

Assumption 3.2 The matrix X has full rank.

This implies that the columns of X are linearly independent and that the n × nmatrix XTX is invertible.

OLS-Derivation

Let β be a hypothetical value of β and define e as the vector of residuals. Then

e(

β)= y− Xβ.

The Sum of Squared Residuals (SSR) is

SSR(

β)=

n

∑i=1

e2i

= e(

β)T e

(β)

=(y− Xβ

)T (y− Xβ)

= yTy− yTXβ− βTXTy + β

TXTXβ.

(3.3)

The vector that minimizes SSR is called the OLS estimator written as β. The OLSestimator is found by minimizing SSR(β), using the first order condition.

∂SSR(

β)

∂β= −2XTy + 2XTXβ = 0.

This can be rewritten asXTXβ = XTy, (3.4)

where the p + 1 equations are called normal equations. We elaborate on this inSection 3.1.1.

3.1. Ordinary Least Squares 13

Solving for β and using Assumption 3.2 we get

β =(

XTX)−1

XTy. (3.5)

Now we need to show that SSR(

β)

is a minimum and not a maximum. This isdone by showing that the Hessian matrix is positive definite, which is the secondorder condition. We get

∂2SSR(

β)

∂βT

∂β=

∂(−2XTy + 2XTXβ

)∂β

T = 2XTX.

We see that XTX is a n × n Gram matrix. We know from linear algebra that allGram matrices are positive semidefinite and that the Gram matrix XTX is positivedefinite if and only if null (X) = {0}.

Given Assumption 3.2, which implies null (X) = {0}, it follows that XTX is positivedefinite and thus SSR

(β)

is a minimum.

3.1.1 The Geometry of Ordinary Least Squares

Let S (X) be the span of X. Since Xβ is a linear combination of the columns in X,every point in S (X) can be written as Xβ for some β.

Since we used Equation (3.4) to derive β we have that

0 = XTy− XTXβ = XT(

y− Xβ)

. (3.6)

Let y− Xβ = e(

β)= e be the vector of the least squares residuals. If we consider

the ith element of the matrix product in Equation (3.6) as an inner product, we get

0 = 〈xi, e〉,

which means that the vector of the least squares residuals is normal to all theindependent variables. This is the reason why Equation (3.6) is called the normalequation. This means that the vector of the least squares residuals is normal to Xβ

since this was just a linear combination of the columns in X and we have

〈Xβ, e〉 = (Xβ)T e = βTXTe = 0.

14 Chapter 3. Multiple Linear Regression

y

Xβ

e

x1

x2

θ ⌃

S(X)

⌃

Figure 3.1: Multiple linear regression for two independent variables.

In Figure 3.1 we see the geometrical interpretation of linear regression. The hori-zontal plane is S (X) and since Xβ is just a linear combination of the columns in X,it is a vector that lies in the plane. Since e is normal to all the independent variablesit must be a vertical line and therefore the angle between the horizontal plane andthis vector is right. Hereby we have that the vectors Xβ, e, and y form a triangle,where y is the hypotenuse and the remaining two vectors form the other sides.

The Pythagorean theorem gives us

‖y‖2 = ‖Xβ‖2 + ‖e‖2,

where ‖y‖2 can be interpreted as the Total Sum of Squares (TSS), ‖Xβ‖2 can beinterpreted as the Explained Sum of Squares (ESS), and ‖e‖2 is the SSR as definedearlier which can be interpreted as the unexplained sum of squares.

If we want to know how much of the variation in y is explained by Xβ we cancalculate R2:

R2 =ESSTSS

= 1− SSRTSS

. (3.7)

Since SSR at most can be equal to TSS, the equation implies that R2 is a numberbetween 0 and 1.

When R2 = 0 we have that SSR = TSS. This means that none of the variability iny is explained by the model and since e is orthogonal to the plane this must meanthat y is orthogonal to the plane. This corresponds to the angle θ in Figure 3.1being equal to 90 degrees.

When R2 = 1 we have that TSS = ESS. This means that all of the variability in y isexplained by the model and since Xβ lies in the plane this must mean that y alsolies in the plane. This corresponds to the angle θ being equal to 0 degrees.

The two extremes imply that θ → 0◦ for R2 → 1 and θ → 90◦ for R2 → 0.

3.1. Ordinary Least Squares 15

3.1.2 Orthogonal Projections

A projection is an idempotent linear mapping. We saw in OLS when estimating ywe implicitly used orthogonal projections to map y into Xβ and ε. An orthogonalprojection maps a point into the point in the subspace closest to it. When mappingorthogonal projections we use projection matrices by premultiplying the vector thatneeds to be projected by the suitable projection matrix. Consider the OLS methodwhere the two following projection matrices are given

PX = X(

XTX)−1

XT (3.8)

MX = In − PX = In − X(

XTX)−1

XT. (3.9)

We call PX a projection matrix since PXX = X and we call MX an annihilator sinceMXX = 0. We have

PXy = X(

XTX)−1

XTy = Xβ = y,

MXy =

(In − X

(XTX

)−1XT)

y = y− PXy = y− Xβ = e.

When PX is applied to the vector y, the output is the vector y that contains thefitted values. This means that applying PX "puts the hat on y" and therefore wecall this matrix the hat matrix. The output when MX is applied to the vector y isthe vector of residuals, e.

PX projects onto the subspace S (X) since Xβ lies in S (X). Since MXy yields thevector of residuals which is orthogonal to the elements in the subspace S (X), wemust have that MX projects onto S⊥ (X), which is the orthogonal complement toS (X).

From Equation (3.8) and (3.9) we have PX + MX = I. Since the sum of the twomatrices return the identity matrix, we say that PX and MX are complementaryprojections and we have a decomposition. To lighten the notation we will from thismoment denote the projection matrix PX by P and the annihilator MX by M.

3.1.3 Properties of OLS

We want to show that β is unbiased, which means that E[

β | X]= β.

Proposition 3.3 The estimator β is unbiased.

16 Chapter 3. Multiple Linear Regression

Proof 3.4 We have

E[

β | X]= E

[(XTX

)−1XTy | X

]=(

XTX)−1

XTE [Xβ + ε | X]

=(

XTX)−1

XT (E [Xβ | X] + E [ε | X])

Ass. 3.1 =(

XTX)−1

XTXβ

= β.

�

This means that on average we have β = β.

Assumption 3.5

E[ε iε j | X

]=

{σ2 i = j0 i 6= j

i, j = 1, . . . , n, (3.10)

where σ2 > 0 is a constant.

We have two implications from Assumption 3.1. These two implications come fromthe definitions of conditional variance and covariance. We get

Var (ε i | X) = E[ε2

i | X]− E [ε i | X]2 = E

[ε2

i | X]

, (3.11)

Cov(ε i, ε j | X

)= E

[ε iε j | X

]− E [ε i | X]E

[ε j | X

]= E

[ε iε j | X

].

These two equations relate to Assumption 3.5.

We get Var (ε i | X) = σ2, i = 1, . . . , n, and Cov(ε i, ε j | X

)= 0, i, j = 1, . . . , n, i 6= j.

We can write Assumption 3.5 more compactly by introducing the matrix form

E[εεT | X

]= σ2In,

which means that σ2In is the covariance matrix of the error term ε.

Since we do not know σ2 we have to estimate it.

Definition 3.6 Assume that n > p + 1. Let the OLS estimate of σ2 be given by

σ2 =SSR

(β)

n− (p + 1)=

eT en− (p + 1)

.

3.1. Ordinary Least Squares 17

We want to know whether the estimate σ2 is unbiased as stated in following propo-sition.

Proposition 3.7 The estimate σ2 is unbiased.

Proof 3.8 Since σ2 =eT e

n− (p + 1)we would like to show that E

[eT e | X

]= σ2 (n− (p + 1)).

To show this, it is convenient to relate the vector of squared residuals e to the error term ε.

This can be done using the annihilator matrix in Equation (3.9).

eT e =(

y−Xβ)T (

y−Xβ)

= (y− Py)T (y− Py)

=(

yT − yTP)(y− Py)

= yT (I− P) (I− P) y

=(

εT + βTXT)

M2 (Xβ + ε)

= εTMε.

(3.12)

This means that we now want to show that E[εTMε | X

]= σ2 (n− (p + 1)). We have

εTMε =n

∑i=1

n

∑j=1

mijε iε j.

The expected value of this is

E[εTMε | X

]=

n

∑i=1

n

∑j=1

mijE[ε iε j | X

] Ass. 3.5=

n

∑i=1

miiσ2 = σ2

n

∑i=1

mii = σ2trace(M).

Now we need to show that trace (M) = n− (p + 1).

trace (M) = trace (In − P)

= trace (In)− trace (P)

= n− trace (P)

= n− trace(

X(

XTX)−1

XT)

= n− trace((

XTX)−1

XTX)

= n− trace(Ip+1

)= n− (p + 1) .

18 Chapter 3. Multiple Linear Regression

Hereby we have

E[σ2 | X

]=

E[eT e | X

]n− (p + 1)

=σ2 (n− (p + 1))

n− (p + 1)= σ2,

and σ2 is on average equal to σ2.

�

Proposition 3.9 We have

Var(

β | X)= σ2

(XTX

)−1.

Proof 3.10 We have the following

Var(

β | X)= E

[β

2 | X]− E

[β | X

]2= E

[β

2 | X]− ββT. (3.13)

Let us look at the first term:

E[

β2 | X

]= E

[((XTX

)−1XTy

)2

| X

]

= E

[(XTX

)−1XTy

((XTX

)−1XTy

)T

| X

]

= E[(

XTX)−1

XTyyTX(

XTX)−1| X]

=(

XTX)−1

XTE[yyT | X

]X(

XTX)−1

=(

XTX)−1

XTE[(Xβ + ε) (Xβ + ε)T | X

]X(

XTX)−1

=(

XTX)−1

XTE[(Xβ + ε)

(εT + βTXT

)| X]

X(

XTX)−1

Ass. 3.1 + 3.5 = ββT + σ2(

XTX)−1

XTX(

XTX)−1

= ββT + σ2(

XTX)−1

.

(3.14)

When substituting Equation (3.14) into Equation (3.13), we get

Var(

β | X)= ββT + σ2

(XTX

)−1− ββT = σ2

(XTX

)−1.

�

3.1. Ordinary Least Squares 19

Define the sampling error as β− β. The relation between the sampling error andthe error term is given by

β− β =(

XTX)−1

XTy− β

=(

XTX)−1

XT (Xβ + ε)− β

= β +(

XTX)−1

XTε− β

=(

XTX)−1

XTε.

(3.15)

We use this in Proof 3.12 and 3.14.

Proposition 3.11 Given Assumptions 3.1, 3.2 and 3.5 we have

Cov(

β, e | X)= 0.

Proof 3.12 We have

Cov(

β, e | X)= E

[(β− E

[β])

(e− E [e])T | X]= E

[(β− β

)eT | X

]. (3.16)

If we substitute β− β with Equation (3.15) we get

E[(

β− β)

eT | X]= E

[(XTX

)−1XTεeT | X

]= E

[(XTX

)−1XTε

(y−Xβ

)T| X]

= E[(

XTX)−1

XTε (Xβ− ε)T −(

XTX)−1

XTεβT

XT | X]

Ass. 3.1 = −σ2(

XTX)−1

XT − E

[(XTX

)−1XTε

((XTX

)−1XTy

)T

XT | X

]

Ass. 3.5 = −σ2(

XTX)−1

XT − E[(

XTX)−1

XTεyTP | X]

= −σ2(

XTX)−1

XT − E[(

XTX)−1

XTε (Xβ− ε)T P | X]

Ass. 3.1 + 3.5 = −σ2(

XTX)−1

XT + σ2(

XTX)−1

XTP

= −σ2(

XTX)−1

XT + σ2(

XTX)−1

XT

= 0

�

20 Chapter 3. Multiple Linear Regression

Proposition 3.11 is useful in Chapter 8.

Theorem 3.13 Gauss-Markov Theorem.For any unbiased estimator b that is linear in y we have that Var (b) ≥ Var

(β)

.

Proof 3.14 We can write b as b = Ay for some matrix A. Let A = C +(XTX

)−1 XT.Then we have

b =

(C +

(XTX

)−1XT)

y

= Cy +(

XTX)−1

XTy

= C (Xβ + ε) + β

= CXβ + Cε + β.

The expected value of each side of the equation yields

E [b | X] = CXβ + E [Cε | X] + E[

β | X]

.

Since b and β are unbiased we have E [b | X] = E[

β | X]= β. We also have that

E [Cε | X] = CE [ε | X] = 0 and from this it follows that CXβ = 0. This has to hold forevery β and hereby we must have that CX = 0.

This means that we have b = Cε + β and

b− β = Cε +(

β− β)=

(C +

(XTX

)−1XT)

ε.

3.1. Ordinary Least Squares 21

Now lets look at the variance of b.

Var (b | X) = Var (b− β | X)

= Var((

C +(

XTX)−1

XT)

ε | X)

=

(C +

(XTX

)−1XT)

Var (ε | X)

(C +

(XTX

)−1XT)T

Ass. 3.5 + Eq. (3.11) = σ2(

C +(

XTX)−1

XT)(

CT + X(

XTX)−1

)= σ2

(CCT +

(XTX

)−1XTCT + CX

(XTX

)−1

=, +(

XTX)−1

XTX(

XTX)−1

)= σ2

(CCT +

(XTX

)−1)

≥ σ2(

XTX)−1

= Var(

β | X)

,

where the inequality holds because CCT is positive semidefinite, which proves the Gauss-Markov Theorem.

�

The Gauss-Markov Theorem tells us that, among linear unbiased estimators, theestimator β has the smallest conditional variance, which means that β is the BestLinear Unbiased Estimator (BLUE).

Chapter 4

Mean Squared Error

This chapter is based on [James et al., 2015, pp. 29-31] and [Hastie et al., 2013, p.38].

Mean Squared Error (MSE) is a measurement of how well a model predicts or fitsthe data. The MSE decreases as the difference between predicted or fitted outputand observed output gets smaller.

We use two types of MSE: training MSE and test MSE. The MSE computed from atraining set is referred to as the training MSE:

Training MSE =1m

m

∑i=1

(yi − yi)2 ,

where yi is the predicted value of the ith observation and m is number of observa-tions in the training set.

Test MSE is an MSE calculated on a data set consisting of pairs of observations,(x0, y0), that are not in the training data. This set of data is called test data. Thetest MSE is given by

Test MSE =1k

k

∑i=1

(y0,i − y0,i)2 ,

where k is the number of pairs of observations in the test data, and y0,i and y0,i,respectively, is the observed output and the predicted value for the ith pair ofobservations.

The test MSE is obtained by estimating y0,i repeatedly using a large number oftraining sets, and testing each at x0.

Given that we want to predict the data, we want to obtain a small test MSE, ratherthan a small training MSE. The smallest training MSE does not necessarily give

23

24 Chapter 4. Mean Squared Error

Model Flexibility

Pred

ictio

n E

rror

Low High

Test Set

Training Set

Figure 4.1: Model flexibility against prediction error.

the smallest test MSE. This typical behavior is shown in Figure 4.1, where the MSEis plotted against model flexibility. We see that the training MSE decreases as themodel flexibility increases. This is because the training MSE overfits the trainingdata, and captures patterns in the training data that do not exist in the test data.This yields a large test MSE. We also see that the test MSE decreases as the modelflexibility increases, however, at some point the test MSE will increase and resultin a U-shape. This yields the fundamental property of the MSE: As the modelflexibility increases, training MSE will decrease, but test MSE may not.

Furthermore, we define training Root Mean Squared Error (RMSE) as the squareroot of the training MSE and test RMSE as the square root of the test MSE. Whencalculating MSE, we square the distance between yi and yi and thereby obtainDKK2. When calculating the RMSE, we take the square root of DKK2 and therebyobtain DKK, which is a more interpretable unit than DKK2.

4.1 The Bias-Variance Trade-Off

This section is based on [James et al., 2015, pp. 33-34], [Vijayakumar, 2016], [Hastieet al., 2013, pp 223-224] and [Davidson and MacKinnon, 2003, pp. 112-115].

The geometric interpretation of the test MSE depends on the variance and the bias.Variance is a measure of how much y will change if estimating using a differenttraining set. Bias refers to how well y will fit a data set.

Flexible models are more sensitive to changes in the data, since this may change yconsiderably. This means that flexible models have a higher variance than simplemodels.

4.1. The Bias-Variance Trade-Off 25

Bias on the other hand is large for simple models, since it is unlikely that a simplemodel fits the observations perfectly. This yields that, for more flexible methods,the variance will increase and the bias decrease. This trade-off is referred to asthe bias-variance trade-off. To show this trade-off mathematically we need thefollowing assumption.

Assumption 4.1 Let f be a single valued deterministic function and y = f (x) + ε, wherethe error term ε has mean zero and variance σ2.

We want to find a function y that approximates the true function y as well aspossible. This is done by measuring the test MSE between y0,i and y0,i. We wantthe term (y0,i − y0,1)

2 to be as small as possible. We take the expected value of thetest MSE and get

E [test MSE] = E

[1k

k

∑i=1

(y0,i − y0,i)2

]=

1k

k

∑i=1

E[(y0,i − y0,i)

2]

.

The expected value of the term inside the sum can be written as

E[(y0,i − y0,i)

2]= E

[(y0,i − fi + fi − y0,i)

2]

= E[(y0,i − fi)

2]+ E

[( fi − y0,i)

2]+ 2E [( fi − y0,i) (y0,i − fi)]

= E[ε2

i]+ E

[( fi − y0,i)

2]+ 2

(E [ fiy0,i]− E

[f 2i]− E [y0,iy0,i] + E [y0,i fi]

)Ass. 4.1 + Eq. (3.11) = Var (ε i) + E

[( fi − y0,i)

2]+ 2

(f 2i − f 2

i − E [y0,i fi] + E [y0,i fi])

= Var (ε i) + E[( fi − y0,i)

2]

.

The last term can be written as

E[( fi − y0,i)

2]= E

[( fi − E [y0,i] + E [y0,i]− y0,i)

2]

= E[( fi − E [y0,i])

2]+ E

[(E [y0,i]− y0,i)

2]+ 2E [(E [y0,i]− y0,i) ( fi − E [y0,i])]

= bias2 + Var (y0,i) + 2(

E [ fiE [y0,i]]− E[E [y0,i]

2]− E [y0,i fi] + E [y0,iE [y0,i]]

)Ass. 4.1 = bias2 + Var (y0,i) + 2

(fiE [y0,i]− E [y0,i]

2 − fiE [y0,i] + E [y0,i]2)

= bias2 + Var (y0,i) .

where bias2 = E[( fi − E [y0,i])

2]

and Var (y0,i) = E[(E [y0,i]− y0,i)

2]. This yields

E[(y0,i − y0,i)

2]= Var (ε i) + bias2 + Var (y0,i) . (4.1)

26 Chapter 4. Mean Squared Error

A special case of Equation 4.1, when contemplating OLS1 is

σ2 + 0 +σ2 (p + 1)

n,

where we can see that an OLS model has bias equal to zero and that the variancewill decrease as the number of observations increases.

The term Var(ε i) in Equation (4.1) is referred to as the irreducible error, since theerror term can not be reduced. Furthermore we see that we need both a lowvariance and a low bias to minimize the equation. This is rather difficult since thetwo parameters are influenced differently by the same changes in the model.

The bias-variance trade-off can also be seen in overspecification and underspecifi-cation.

Overspecification

Overspecification is when irrelevant independent variables are included in themodel, which makes the model larger than it needs to.

When overspecifying, the model takes the form

y = Xβ + Zγ + ε, ε ∼ IID(0, σ2I

), (4.2)

when the true DGP is

y = Xβ0 + ε, ε ∼ IID(0, σ2

0 I)

. (4.3)

The model in Equation (4.2) is not misspecified, as Equation (4.3) is a special caseof Equation (4.2) where β = β0, γ = 0, and σ2 = σ2

0 .

Let us look at the linear regression in Equation (4.2). By the Frisch-Waugh-Lovell2

Theorem we haveMZy = MZXβ + MZε,

where MZ = I− Z(ZTZ

)−1 ZT. We get

β =(

XTMZX)−1

XTMZy.

If we substitute y with Equation (4.3), we get

β = β0 +(

XTMZX)−1

XTMZε. (4.4)

1[Tibshirani, 2013, p. 14]2[Davidson and MacKinnon, 2003, p. 69]

4.1. The Bias-Variance Trade-Off 27

The expectation of the last term in Equation (4.4) conditional on Z and X is 0.

The OLS estimate of Equation (4.3) is β. Note that both β and β are unbiased OLSestimators and linear in y.

We now look at the variance of β.

Var(

β)= E

[(β− β0

) (β− β0

)T]

=(

XTMZX)−1

XTMZE[εεT]

MZX(

XTMZX)−1

= σ20

(XTMZX

)−1XTM2

ZX(

XTMZX)−1

= σ20

(XTMZX

)−1

(4.5)

Since we know that for a projection matrix, PX, that ‖PXy‖ ≤ ‖y‖3, we must have

σ20

(xTMZx

)−1≥ σ2

0

(xTx)−1

The only time σ20(xTMZx

)−1= σ2

0(xTx)−1 is when x ∈ S⊥ (Z), which means that

x does not explain anything about Z.

To find out what happens with the variance when irrelevant independent variablesare included in a model, we look at Var

(β)− Var(β). If the matrix produced by

this is positive semidefinite, ε has greater variance than ε. This is equivalent toshowing that Var(β)−1 −Var

(β)−1 is positive semidefinite. We get

XTX− XTMZX = XT (I−MZ)X = XTPZX = (PZX)T PZX,

which is clearly positive semidefinite. The only thing left is to divide by σ20 .

This means that when we add irrelevant variables to the model, the variance willincrease.

Underspecification

We can also have underspecification, where relevant variables are omitted. Thisgives the model

y = Xβ + ε, ε ∼ IID(0, σ2I

).

whereas the actual DGP is given by

y = Xβ0 + Zγ0 + ε, ε ∼ IID(0, σ2

0 I)

.

3[Hastie et al., 2013, p. 61]

28 Chapter 4. Mean Squared Error

This is the opposite of overspecification; however, when we underspecify we alsosay that we misspecify the model, since not all relevant independent variables areincluded in the model.

Now the β regressed by Equation (4.2) is the correct estimator, and β is regressedby the underspecified model. β is then biased, since

E[

β]= E

[(XTX

)−1XT (Xβ0 + Zγ0 + ε)

]= β0 +

(XTX

)−1XTZγ0.

(4.6)

β is unbiased when the last term is equal to zero, which is when XTZ = 0 orγ0 = 0. This only happens when two sets of independent variables are orthogonalor when the model is not underspecified.

When β is biased, we can not evaluate the accuracy of the estimate using thecovariance matrix. Instead we use the MSE of β.

MSE(

β)= E

[(β− β0

) (β− β0

)T]

.

We have from Equation (4.6)

β− β0 =(

XTX)−1

XTZγ0 +(

XTX)−1

XTε,

which means that(β− β0

) (β− β0

)T=(

XTX)−1

XTZγ0γT0 ZTX

(XTX

)−1

=, +(

XTX)−1

XTεεTX(

XTX)−1

=, +(

XTX)−1

XTZγ0εTX(

XTX)−1

=, +(

XTX)−1

XTεγT0 ZTX

(XTX

)−1.

We get

MSE(

β)= σ2

0

(XTX

)−1+(

XTX)−1

XTZγ0γT0 ZTX

(XTX

)−1, (4.7)

where the last term occurs because of the bias of β.

Now we want to compare MSE (β) to MSE(

β)= Var

(β). Since Var

(β)

in Equa-tion (4.5) can not be smaller than the first term in (4.7) the difference betweenMSE (β) and MSE

(β)

depends on the last term in Equation (4.7). If bias is small,

4.1. The Bias-Variance Trade-Off 29

the second term in Equation (4.7) is small, and it may be small enough for β to bebetter than β. However if bias is large, the second term in Equation (4.7) is large,and it may be large enough for β to be better than β.

Additionally we have that the OLS estimate σ2 typically will be larger than σ2 ifsome important independent variables are omitted, which means that the estimateof the first term in Equation (4.7) also will be biased.

This means that when omitting relevant variables, bias will increase.

Underspecification leads to higher bias whereas overspecification leads to highervariance and both lead to higher MSE. We want to have low bias and low variance.Variance tends to zero as the number of observations increases. The bias due tounderspecification does not disappear as the number of observations increases.To determine whether underspecification or overspecification is worse, we have tocontemplate the bias-variance trade-off as well as the sample size.

Chapter 5

Resampling Methods

This chapter is based on [James et al., 2015, pp. 175-184].

When resampling data we divide the training set into different subsets on whichwe fit the model. This method can be used to evaluate the performance of themodel, which is called model assessment, and to choose the best level of flexibility,which is called model selection.

To evaluate the performance of the model we observe the test error. The test erroris estimated by holding out a subset of the training data during fitting and thenapplying the fitted model to the held out subset.

We discuss three resampling methods: The validation set approach, Leave-One-Out Cross-Validation, and k-Fold Cross-Validation.

5.1 The Validation Set Approach

The Validation Set Approach (VSA) randomly divides the training data into twosets: a training set and a hold-out set. When we calculate the test MSE of the datain the hold-out set we get an estimate of the test error rate. The VSA is simplebecause we fit the model on a single set of data.

If we repeat the approach we will get a different estimate, and thus the test errorrate depends on the data in each set. Another disadvantage of the VSA is that amodel often performs better when trained on more observations and the approachtherefore tends to overestimate the error rate. This is what the Leave-One-OutCross-Validation (LOOCV) takes into account.

31

32 Chapter 5. Resampling Methods

5.2 Leave-One-Out Cross-Validation

LOOCV also divides the training data into two sets. The difference between VSAand LOOCV is that the hold-out set consists of a single observation (x1, y1). Theremaining observations form the training set.

To acquire an estimate for the test error, we use

MSE1 = (y1 − y1)2 ,

where y1 is the predicted value for the held out observation, x1. Since (x1, y1) wasnot used in the fitting process this is also an unbiased estimate. Even though thisestimate is unbiased, it is a poor one given that it is based on one observation,which makes it highly variable.

To make the model less variable the approach is repeated by selecting a new hold-out set consisting of the observation (x2, y2), and repeating this n times to get nsquared errors. We then define the LOOCV estimate, CV(n), to be the average ofthe n test error estimates:

CV(n) =1n

n

∑i=1

MSEi.

This gives an estimate which has less bias and does not overestimate as much asthe VSA.

Repeating the LOOCV multiple times will always give the same result, since thetraining set always is divided into n held out observations. For a large n theLOOCV can be expensive to implement, because the model has to be fitted n times.

This is what the k-fold Cross-Validation (k-fold CV) takes into account.

5.3 k-Fold Cross-Validation

The k-fold CV strategy randomly divides the training data into k folds of approxi-mately the same size.

We let the first fold be the hold-out set and fit the model on the remaining k − 1folds. The MSE1 is calculated from the observations in the hold-out set. This isrepeated for the k different hold-out sets to get k estimates of the test error, givenby MSE1, . . . , MSEk. The estimate for the k-fold CV, CVk, is found by taking theaverage of these k estimates:

CVk =1k

k

∑i=1

MSEi.

5.3. k-Fold Cross-Validation 33

The k-fold CV is equal to the LOOCV when k = n.

The advantage of this strategy is computational, since dividing the data into k foldsmakes it less expensive to calculate than the LOOCV. Thus this approach is morefeasible.

There will be some variability in the k-fold CV estimate as a result of the variabilityin how the observations are divided into the k folds. However this variability istypically much lower than the one in the test error estimate from the VSA.

An important aspect of k-fold CV is the one-standard-error rule, which roughlystates that if two models are not different in a statistically significant way, weshould choose the model with the least independent variables.

5.3.1 Bias-Variance Trade-Off for Leave-One-Out Cross-Validation andk-Fold Cross-Validation

Comparing bias reduction for LOOCV and k-fold CV, the LOOCV is preferredsince the estimate of the test error of this approach is approximately unbiased. Thereason for this is that the LOOCV training set includes n− 1 observations. Throughthe k-fold CV we will obtain a higher level of bias since the training set for thisstrategy contains (k−1)n

k observations, which is fewer than in the LOOCV strategy.

When comparing the variance and the estimating procedure for LOOCV and k-foldCV, the LOOCV has higher variance than k-fold CV when k < n. This is due to theoverlap between the training sets. For LOOCV only one observation is selected asthe hold-out set and therefore the model is fitted on almost identical sets, whichgives highly correlated outputs. In comparison the k-fold CVs outputs of k fittedmodels are less correlated, since the training sets are less identical.

Theory states that the mean of many highly correlated quantities has higher vari-ance1. The LOOCV has higher correlated outputs than the k-fold CV, and thushigher variance.

Therefore we see that there is a bias-variance trade-off associated with the choiceof k in k-fold CV.

1[James et al., 2015, pp. 183-184]

Chapter 6

Model Selection

This chapter is based on [James et al., 2015, pp. 203-215, 219, 227-228], [Hastieet al., 2013, pp. 64-68], [van Wieringen, 2015, pp. 7, 14], and [Breheny, 2011, p. 7].

In this chapter we introduce alternatives to fitting a model using OLS that takeirrelevant independent variables into account. This can improve the model gen-eralizability of the simple linear model. Alternative methods to linear regressionfitted by OLS are the regularized regression models fitted by either ridge regres-sion, Lasso, or elastic net.

6.1 Choosing the Optimal Model

When performing subset selection we attain a set of models, each containing adifferent amount of the p independent variables. There are different approachesto determine the best model of this set. Intuitively, one would consider the modelwith the largest number of independent variables as the best model since thismodel has the largest R2 and smallest SSR. However when choosing the bestmodel out of the attained set we want a model with a low test error, thus R2 andSSR are discarded as suitable solutions to our problem, since these measurementsare linked to the training error as seen in Chapter 4.

Instead we can indirectly estimate the test error by adjusting the training error toaccount for the bias that comes from overfitting.

When indirectly estimating the test error we consider four methods: Adjusted R2,Cp, AIC, and BIC.

35

36 Chapter 6. Model Selection

Adjusted R2

We have from Equation (3.7) that R2 = 1− SSRTSS , where TSS = ∑n

i=1 (yi − y)2. As weadd more variables to the model, R2 increases. The adjusted R2 for an OLS modelwith d independent variables is defined as

Adjusted R2 = 1− SSR/ (n− (d + 1))TSS/ (n− 1)

.

A large value of the adjusted R2 indicates a model with a small test error. Max-imizing the adjusted R2 is equivalent to minimizing SSR

n−(d+1) . We know that theSSR decreases as the number of variables increases. This is not always the case for

SSRn−(d+1) .

When all the optimal independent variables are included in the model, addingmore independent variables will lead to an increase in SSR

n−(d+1) which equivalently

decreases the adjusted R2. Theoretically the optimal model will only have theoptimal variables and no added noise.

Cp

The Process Capability (Cp) estimate for an OLS model is given by

Cp =1n(SSR + 2dσ2) ,

where σ2 is the estimated variance of the error term ε. This means that the Cp addsa penalty of 2dσ2 to adjust for the underestimation of the test error (optimism).

AIC

The Akaike Information Criterion (AIC) for an OLS model is given by

AIC =1

nσ2

(SSR + 2dσ2) ,

The Cp and AIC are proportional for OLS models.

BIC

The Bayesian Information Criterion (BIC) for an OLS model is defined as

BIC =1n(SSR + log (n) dσ2) .

6.2. Subset Selection 37

The penalty term from Cp is replaced with log (n) dσ2, where n is the number ofobservations. Since log (n) > 2 for any n > 7, the BIC adds a greater penalty onmodels with many variables, which results in a selected set of models with fewervariables than Cp.

As the sample size increases, the probability that BIC will choose the correct modeltends to 1, whereas AIC chooses models that have more independent variablesthan needed, making the model too complex. For finite samples we have that BICwill choose models that are too simple, because of the greater penalty on flexiblemodels, so the AIC will be better in this instance.

An alternative to estimating the test error indirectly is to do it directly by usingVSA or k-fold CV. Since VSA and k-fold CV make fewer assumptions about themodel, they can be used even if we do not know the number of independentvariables or the estimate of σ2. Therefore, the direct methods are preferred to theindirect methods.

6.2 Subset Selection

The two main methods used for selecting subsets of the independent variables areBest Subset Selection (BSS) and Stepwise Selection.

6.2.1 Best Subset Selection

When performing BSS we fit a separate OLS model for each possible combinationof the p independent variables. This means that we fit all (p

k) = p(p−1)k models

that contain k independent variables for k = 1, . . . , p. We then choose the modelwith the smallest SSR or largest R2 for each k and denote these M1, . . . , Mp. Thissimplifies the problem from 2p models to p models. The null model which doesnot contain any independent variables is denoted by M0 and this model capturesthe sample mean for each observation.

The next step is to identify the best model among the p + 1 models M0, . . . , Mp. Todo this we use k-fold CV.

BSS is a simple approach but it has computational limitations since the number ofpossible models grows exponentially as p increases. Another problem is that whenp is large, our search space k will expand and therefore the BSS is more likelyto find models that overfit the data. This means that the model might fail whenpredicting on test data. Therefore we introduce the stepwise selection methods.

38 Chapter 6. Model Selection

6.2.2 Stepwise Selection

Stepwise Selection can be divided into Forward Stepwise Selection (FS) and Back-ward Stepwise Selection (BS).

Forward Stepwise Selection

In the FS method we first consider a null model M0. We then add independentvariables to this model until all variables are in the model.

For k = 0, . . . , p− 1 we consider all of the k+ 1 models that improve Mk and choosethe model among these that has the smallest SSR or largest R2 , which we denoteMk+1. This means that we are fitting the original model as well as p− k models inthe kth iteration, which simplifies to 1 + ∑

p−1k=0 (p− k) = 1 + p(p+1)

2 , which is lessthan 2p for p > 2. This is the first step in the FS method.

The second step consists of choosing the best model among the p + 1 modelsM0, . . . , Mp. To do this we use k-fold CV.

Backward Stepwise Selection

Instead of an initial model with zero independent variables as was the case withthe FS method, the BS method has an initial model containing all the independentvariables. The first step in this approach is to iteratively single out the variable thatcontributes the least to the model and remove it. The contribution is measured onhaving the smallest SSR or highest R2. The second step is to find the best modelusing k-fold CV.

As was the case with the FS method, the BS method searches for the best modelfrom the set of models with different amount of variables.

The difference between the two methods is that the FS method can be used whenn < p, but the BS method can only be used when n > p since we fit the full modelfor BS from the beginning.

The FS and BS methods are not guaranteed to yield the best model containing asubset of the p predictors.

Another aspect of subset selection is instability1: If one observation is changed itcan cause a large change in the prediction equation.

1[Breiman, 1994, p. 373]

6.3. Shrinkage Methods 39

6.3 Shrinkage Methods

Shrinkage methods take a model containing all of the p independent variables andshrink the β estimates towards zero. Two shrinkage methods are ridge regressionand the Least Absolute Shrinkage and Selection Operator (Lasso).

6.3.1 Ridge Regression

From the least squares method we know that β0, . . . , βp minimizes Equation (3.3).This equation can be written as

SSR = ‖y− β0 − Xβ‖22,

where β is a vector containing the parameters β1, . . . , βp.

The ridge regression is similar to least squares, but instead of β j for j = 0, . . . , p

minimizing the SSR, the estimate βRλ minimizes

‖y− β0 − Xβ‖22 + λ‖β‖2

2 = SSR + λ‖β‖22, (6.1)

where λ ≥ 0 is a tuning parameter. Equation (6.1) is a trade-off between the SSR

and the shrinkage penalty l2, which is defined as ‖β‖2 =√

∑pj=1 β2

j . The l2 normof a vector measures the distance of β from zero. The shrinkage penalty is smallwhen the β estimates are close to zero, while the tuning parameter λ controls theamount of shrinkage. As λ→ ∞ the l2 norm of β

Rλ will decrease.

Note that in Equation (6.1), the shrinkage penalty is not applied to the interceptβ0 since multiplying the intercept with λ would make the model depend on theorigin of y. To modify only the intercept and not the slope parameters, we shiftxi to have mean zero and thereby translate all points to the origin. Using centeredinputs gives that xij is replaced with xij − xj, where xj denotes the mean of the jthindependent variable and β0 = y = 1

n ∑ni=1 yi. By using this centered xij, the rest

of the β parameters are estimated by ridge regression. This gives a design matrixwith p columns instead of p + 1 columns.

Assuming this centering and given SSR in Equation (3.3), the ridge regression canbe written in matrix form as

SSR (λ) = (y− Xβ)T (y− Xβ) + λβT β.

An analogous derivation to the one of Equation (3.5) yields

βRλ =

(XTX + λI

)−1XTy.

40 Chapter 6. Model Selection

The solution adds λ > 0 on the diagonal of XTX such that the problem becomesnonsingular, even if XTX does not have full rank.

The following three theorems (stated without proof) illustrate the bias-variancetrade-off when using ridge regression:

Theorem 6.1 The variance of the ridge regression estimate is

Var(

βRλ

)= σ2WXTXW,

where W = (XTX + λI)−1.

Theorem 6.2 The bias of the ridge regression estimate is

Bias(

βRλ

)= −λWβ.

Theorem 6.3 There always exists a λ such that the MSE of βRλ is less than the MSE of β.

The last theorem indicates that in theory ridge regression should always be pre-ferred to OLS. In practice we need to find the optimal value of the hyperparameterλ.

To analyze the impact of λ we use Singular Value Decomposition (SVD). The SVDof X is given as

X = UDVT.

After performing SVD we obtain the orthogonal matrices U and V, where U hasdimension n× p and V has dimension p× p. The columns v1, . . . , vp of the matrixV are the eigenvectors of XTX and the columns u1, . . . , up of the matrix U are theeigenvectors of XXT. The diagonal matrix D has dimension p × p with diagonalentries d1 ≥ d2 ≥ . . . ≥ dp ≥ 0.

Using SVD on βRλ and multiplying with X we obtain the fitted vector. This gives us

XβRλ = X

(XTX + λI

)−1XTy

= UDVT(

VD2VT + λI)−1

VDTUTy

= UDVT(

V(D2 + λI

)VT)−1

VDTUTy

= UDVT(

VT)−1 (

D2 + λI)−1

V−1VDTUTy

= UD(D2 + λI

)−1DUTy

=p

∑j=1

ujd2

j

d2j + λ

uTj y,

6.3. Shrinkage Methods 41

where uj are the columns of U. We have λ > 0, which means thatd2

j

d2j +λ

< 1. Ridge

regression first calculates the values of y with respect to the orthonormal basis U

and then shrinks the value by the fractiond2

j

d2j +λ

. Smaller values of d2j gives a higher

amount of shrinkage.

df (λ) = trace(

X(

XTX + λI)−1

XT)=

p

∑j=1

d2j

d2j + λ

(6.2)

where df (λ) denotes the effective degrees of freedom of the ridge regression fit.Given that smaller values of d2

j equals a higher amount of shrinkage, Equation (6.2)is a decreasing function of λ. For a linear regression the degrees of freedom areequal to the amount of estimated independent variables included in the model:When λ = 0, we have df(λ) = p, and we have df (λ) → 0 for λ → ∞. Note thatthe number of independent variables becomes p + 1 instead of p since we includethe intercept.

Ridge regression has the disadvantage that all p variables are included in the modeland does not set any of the estimates equal to zero, which makes the model lessgeneral. This is what Lasso takes into account.

6.3.2 Lasso

The Lasso estimate βLλ minimizes the following

‖y− β0 − Xβ‖22 + λ‖β‖1 = SSR + λ‖β‖1. (6.3)

The difference between ridge regression and Lasso is that the penalty ‖β‖22 in ridge

regression given in Equation (6.1) has now been replaced by the l1-norm, which isdefined as ‖β‖1 = ∑

pj=1 |β j|.

We want to minimize the penalized SSR in Equation (6.3). This means that, for asufficiently large λ, some of the estimates are forced equal to zero.

Therefore Lasso performs shrinkage and variable selection simultaneously. Themodels produced by Lasso are more parsimonious than those produced by ridgeregression since they contain fewer variables.

Lasso has the disadvantage that when there is a group of highly correlated vari-ables, Lasso will only arbitrarily select one of them. The coefficients of the dis-carded variables are set equal to zero. A shrinkage method that improves uponthis weakness is the elastic net.

42 Chapter 6. Model Selection

6.3.3 Elastic Net

Elastic net performs shrinkage and variable selection simultaneously, but unlikeLasso, the elastic net method selects or discards the correlated variables as a group.Elastic net is obtained by combining the shrinkage penalties, l1 and l2, from Lassoand ridge regression and is given by

‖y− β0−Xβ‖22 ++λ

(α‖β‖2

2 + (1− α) ‖β‖1)= SSR+ λ

(α‖β‖2

2 + (1− α) ‖β‖1)

,

where α ∈ [0, 1]. The parameter α determines the weight of the penalties. Whenα = 0 the ridge regression penalty l2 is omitted and when α = 1 the Lasso penaltyl1 is omitted. To determine the best α we use k-fold CV. When a group of highlycorrelated variables is selected, the value of the beta estimates corresponding tothese variables will approximately be equal to one another.

Choosing lambda

Ridge regression, Lasso, and elastic net determine a βλ estimate for each value ofλ. Therefore, the choice of λ is important and is determined by k-fold CV. Wechoose a grid of λ values and then compute the k-fold CV error for each value ofλ. The λ with the smallest k-fold CV error is selected. We then refit the modelusing all of the available observations and the selected tuning parameter λ.

Chapter 7

Maximum Likelihood Estimation

This chapter is based on [Hayashi, 2000, pp. 47-51], [Davidson and MacKinnon,2003, p. 394] and [Madsen and Thyregod, 2011, pp. 12, 17-18].

The basic idea of the Maximum Likelihood principle is to maximize the probabilityof observing the vector of parameters θ given y, which is denoted L (θ; y).

The likelihood function of a model in a given data set is defined as evaluatingf (y; θ) at the vector y, and is denoted L (θ; y). We obtain the maximum likelihoodby maximizing the likelihood function with respect to the parameters. The parame-ter vector θ that contains the maximum values of the likelihood function is denotedθ and is called the Maximum Likelihood Estimate (MLE) of the parameters.

7.1 Log Likelihood

We start by stating the following assumption about the distribution of the errorterm.

Assumption 7.1 The distribution of ε conditional on X is jointly normal.

From Equation (3.2) we know that y = Xβ + ε and due to Assumptions 3.1, 3.5,and 7.1 we have

y | X ∼ N(Xβ, σ2In

).

We know from probability theory that the density function for a n-variate normal

distribution is (2π)−n/2 |Σ|−1/2 exp(−1

2(y− µT)Σ−1 (y− µ)

). Thus

f (y | X) =(2πσ2)−n/2

exp(− 1

2σ2 (y− Xβ)T (y− Xβ)

),

43

44 Chapter 7. Maximum Likelihood Estimation

where µ = Xβ and Σ = σ2In. By taking the log of this, we obtain the normal loglikelihood function

l(

β, σ2) = log L(

β, σ2) = −n2

log (2π)− n2

log(σ2)− 1

2σ2

(y− Xβ

)T (y− Xβ)

,

where β and σ2, respectively, is a hypothetical value of β and σ2.

Given linearity and the Assumptions 3.1, 3.2, 3.5, and 7.1 we have that the β thatmaximizes the likelihood function does not depend on σ2. Given this fact, weknow that to maximize the log likelihood function we have to minimize the sumof squared residuals. We know from OLS that the minimum of SSR is β and that(

y− Xβ)T (

y− Xβ)= eTe is the minimized sum of squares. Hereby we obtain a

log likelihood function dependent on σ2, and we denote this the concentrated loglikelihood

concentrated log likelihood = −n2

log (2π)− n2

log(σ2)− 1

2σ2 eTe.

The value that maximizes the concentrated log likelihood is the MLE of σ2 and isfound by taking the derivative with respect to σ2 and setting it equal to zero. Thisyields

σ2MLE =

1n

eTe =SSR

n=

n− (p + 1)n

σ2.

We know from the definition of σ2 that this estimate is unbiased. Since σ2 ismultiplied by a factor other than 1, the MLE of σ2 must be biased.

This section has shown that β, which is the OLS estimate of β, is also the MLE.The OLS estimator of σ2 given in Definition 3.6 only differs slightly from the MLEof σ2.

7.1.1 Cramer-Rao Lower Bound

Definition 7.2 Let θ =(

β, σ2). The information matrix I(θ) is given as

I (θ) = E[s (θ) s (θ)T

],

where s (θ) is the score vector given as

s (θ) =∂l (θ)

∂θ

The information matrix can also be written as

I (θ) = −E[

∂2l (θ)

∂θ∂θT

].

This is called the information matrix equality.

7.1. Log Likelihood 45

Theorem 7.3 Let f (Y; θ) be the parametric density for the observations Y, where θ isan m-dimensional vector in some parameter space Θ. Given that the information matrixI(θ) is invertible for any θ ∈ Θ and assuming sufficient regularity to allow differentiationunder integration, the covariance matrix for any unbiased estimator θ (Y) of θ satisfies

Var(θ (Y)

)≥ I (θ)−1 ≡ Cramer-Rao Lower Bound

where I (θ) is the information matrix.

Proof 7.4 To ease the notation we denote σ2 = γ. To find the information matrix we needthe partial derivatives of the log likelihood function. The first order derivatives are

∂l (θ)∂γ

= − n2γ

+1

2γ2 (y−Xβ)T (y−Xβ)

∂l (θ)∂β

=∂

∂β

(− 1

2γ(y−Xβ)T (y−Xβ)

)= − 1

2γ

(−XT (y−Xβ) + (y−Xβ)T (−X)

)= −−2

2γXT (y−Xβ)

=1γ

XT (y−Xβ) .

The second order derivatives are

∂2l (θ)

∂β∂βT = − 1

γXT (X)

∂2l (θ)∂2γ

=n

2γ2 −1

γ3 (y−Xβ)T (y−Xβ)

∂2l (θ)∂β∂γ

= − 1γ2 XT (y−Xβ)

∂2l (θ)

∂γ∂βT = − 1

γ2 XT (y−Xβ) .

Note that the mixed second order derivatives are the same. If we evaluate the second orderderivatives at the true parameter ε = y−Xβ we get

I (θ) = −E

−1γ

(XTX

)− 1

γ2 XTε

− 1γ2 XTε

n2γ2 −

1γ3 εTε

.

The likelihood function is conditional on X and therefore when we use the assumptions thatare conditional on X, E [ε | X] = 0 and E

[εTε | X

]= nσ2, and substituting γ = σ2 we

46 Chapter 7. Maximum Likelihood Estimation

get

I (θ) =

1σ2

(XTX

)0

0 − n

2 (σ2)2 +nσ2

(σ2)3

=

1σ2

(XTX

)0

0n

2 (σ2)2

.

We invert this diagonal matrix to obtain the Cramer-Rao lower bound

I (θ)−1 =

σ2 (XTX)−1 0

02(σ2)2

n

.

Therefore the unbiased estimator β that has variance σ2 (XTX)−1 attains the Cramer-Rao

lower bound.

�

The OLS estimate β is BLUE due to linearity and Assumptions 3.1, 3.2, and 3.5.When we add Assumption 7.1 the OLS estimate β becomes the Best Unbiased Es-timator (BUE), since any other unbiased estimator has larger conditional variancein matrix sense.

BUE is a stronger result than BLUE, since β in BLUE has the minimum variancefor unbiased linear estimators in y. BUE includes non-linear unbiased estimators,and those thereby include a wider range of estimators.

Chapter 8

Hypothesis Testing

This chapter is based on [Hayashi, 2000, pp. 35-37] and [Faraway, 2002, pp. 36-37,39].

Since β is a random vector, we have to take this randomness into account, if we areto make inference about the true β. To do this we consider sampling variability.Let βk = βk, where βk is a hypothetical value of βk. A typical hypothesis whendealing with linear regression is βk = βk, which is called the null hypothesis and isdenoted H0. The counterpart to H0 is the alternative hypothesis, H1, which statesthat βk 6= βk. We say that we fail to reject the hypothesis if the null hypothesisholds, otherwise we reject it. The idea is to calculate some test statistic that has aknown distribution when either H0 or H1 is true. To determine whether or not H0

holds, we can use the z-ratio given by

zk ≡βk − βk√Var

(βk) ,

where βk and βk are the kth element of β and β.

We know from Equation (3.15), Assumption 3.1, and Proposition 3.9 that β − β

has mean zero and variance σ2(

XTX)−1

. It follows from Assumption 7.1 that β isnormally distributed and we have

β− β ∼ N(

0, σ2(

XTX)−1

)which means

βk − βk ∼ N(

0, σ2{(

XTX)−1

}kk

).

47

48 Chapter 8. Hypothesis Testing

Since Var(

βk)= σ2

{(XTX

)−1}

kk, we can conclude the following

zk ∼ N (0, 1) .

In this case zk measures whether there is a significant difference between the trueβ and the estimated β. The closer zk is to zero, the more likely it is that there is nosignificant difference between β and β and vice versa. To determine whether zk istoo small to reject the null hypothesis we need to know σ2.

For a known σ2 the statistic zk can be calculated from the sample. The distributionconditional on X does not depend on X and we know the distribution of zk. If wedo not know σ2 we can replace this with the OLS estimate σ2 given in Definition3.6.

Thereby we obtain the t-statistic

tk ≡βk − βk√

σ2

{(XTX

)−1}

kk

,

where the substitution of σ2 does not change the properties of a known σ2, but onlychanges the distribution. Under H0 the t-statistic is distributed as t (n− (p + 1)).

Proposition 8.1tk ∼ t (n− (p + 1)) .

Proof 8.2 We have

tk =βk − βk√

σ2

{(XTX

)−1}

kk

√σ2

σ2 =zk√σ2

σ2

=zk√

eT e/ (n− (p + 1))σ2

=zk√ q

n− (p + 1)

,

where q =eT eσ2 .

Since eT e = εMε, we have

q =eT eσ2 =

ε

σM

ε

σ.

Since ε | X ∼ N(0, σ2In

)we have that

ε

σ∼ N (0, In). This means that

ε

σM

ε

σis χ2

distributed with degrees of freedom equal to the rank of M. Since M is idempotent we havethat rank (M) = trace (M). In Proof 3.8 we found trace (M) = n− (p + 1), and herebywe have

q | X ∼ χ2 (n− (p + 1)) .

8.1. Confidence Interval 49

Since both β and e are linear functions of ε they are jointly normal conditionally on X. Wehave from Proposition 3.11 that they are uncorrelated conditionally on X, and hereby theyare independently distributed conditionally on X. Since zk is a function of β and q is afunction of e, zk and q are independently distributed conditionally on X.

By definition of the t-distribution we have that tk is t-distributed with degrees of freedomn− (p + 1).

�

A p-value is a probability measure given through the t-test. The p- and t-valueare linked in such a manner that the greater the magnitude of the t-value, the p-value tends to zero. It is customary within the field of statistics to contemplate asignificance level of 5%, meaning that when the p-value is less than this thresholdwe reject the null hypothesis. Note that when performing a t-test one shouldalways consider whether a significance level of 5% is fitting. When dealing withsignificance levels and p-values, a natural extension would be confidence intervals.

8.1 Confidence Interval

An alternative way of making inference about the true parameter β is with confi-dence intervals. For a (1− α) confidence interval, every point within this intervalwill support the null hypothesis while every point outside this interval will sup-port the rejection under hypothesis H1.

A (1− α) confidence interval for βk is given as

βk ± tα/2n−(p+1)σ

√{(XTX

)−1}

kk.

This means that we reject the null hypothesis H0 if tk lies outside the interval[βk − tα/2

n−(p+1)σ

√{(XTX

)−1}

kk, βk + tα/2

n−(p+1)σ

√{(XTX

)−1}

kk

].

8.2 Prediction Interval

To predict a future value y0, where y0 is a future observation given a known newset of independent variables denoted x0, we set y0 = xT

0 β.

In this project we want to know the cash price of a house when we have the datax1, . . . , xp. Thus the house price is given by xT

0 β + ε. Due to E [ε] = 0 we have that

50 Chapter 8. Hypothesis Testing

the predicted price will be xT0 β, however, the variance of ε influences the variance

of the prediction. The variance is Var(

xT0 β)= xT

0

(XTX

)−1x0σ2. This yields the

(1− α) prediction interval

y0 ± tα/2n−(p+1)σ

√1 + xT

0

(XTX

)−1x0.

This means that the real value y0 lies in the interval[y0 − tα/2

n−(p+1)σ

√1 + xT

0

(XTX

)−1x0, y0 + tα/2

n−(p+1)σ

√1 + xT

0

(XTX

)−1x0

].

Chapter 9

Application

9.1 Initial Data Preparation

First, we sort the data by date and change the independent variables that are char-acter vectors to factors. We clean the data by removing some of the independentvariables and the observations that are not available (NAs). The independent vari-ables we remove are "Date of sale", "Type of residence", "Full address", "Quarter1", "Quarter 2", "Quarter 3", "Quarter 4", "Turnover time", and "Year of sale". "Dateof sale" and "Full address" are removed because they are both character vectors,which can not easily be converted into a factor. We remove "Type of residence"since "Villa, 1 fam." is already chosen. We remove "Quarter 1", "Quarter 2", "Quar-ter 3", "Quarter 4", "Year of sale" and "Turnover time" since we want to predicthouse prices and therefore do not know beforehand when the villas will be sold.The reduced data set now consists of 16 variables and 3,625 observations.

We hold out the 5% most recently sold villas from the data to test for predictionaccuracy. The remaining data set is divided into two sets: training set and test set.This set is referred to as the entire data set.

Motivating Log Transformation

First we fit a model on the training set using ridge regression, which is then usedto predict cash prices using the test set.

When we plot the predicted cash prices against the actual cash prices as seen inFigure 9.1a, we see that the relation seems to be log-linear.

51

52 Chapter 9. Application

●

●

●

●

●

●

● ●●

●

●●

●●●●●

●●

●

●

●

●

●● ●

● ●●●

●

●

●

●

●

●●●

●

●

●

●●●

●

●● ●●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

● ●

●

●

●

●

●

●

●●

●●●●

●

●

●●

●

●●●

●

●

●●

●●

●

●

●●●

●●●

●

● ●●●

●●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●●

● ●

●●

●

●●

●●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●●

●

●

●

●

●●

●

●●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●●●

●

●

●

●●●● ●

●

●●

●

●●●

● ●

●

●●

●●

●●

● ●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●●

●●

●●●

●

●

●● ●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●● ●

●●

●●

●

●●

●

●●

●

●

●●

●

●● ●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●●

●

●

●

●●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●●

●

● ●

●

●

●●

●

●

●

●●

●

● ●●● ●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

● ●

●

● ●

●

●●

●

●

● ●

●●

●●

●

●● ●

●

● ●●

●●

●

●●

●

●●

●●

●

●

●●

●●

●

●●

●

●●

●●

●

●●●

●

●

●

●●●

●

●

●●

●

●

●●

●●

●

●●●●

●

●●●●

●

●

●●

●

●

●●

● ●

●

●

●●

●●

●

●

●

●

●

●

●

● ●●

●

●

●

●●

● ●

●

●●

●

●●

●

●

●●

●

● ●●

●

●

●●

●● ●

●

●●

●

●

● ●

●

● ●●●

●●

●

● ●

●

●●●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●●

●

● ●

●● ●● ●

●

●

●

●

●● ●●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●●

●

●

●●

●●

●●

●●

●

●●

●●

●

●

●

●●

● ●●●

●●●

●

●

●

●●

●

●

●

●●

●●

●●

●

●●

●●

● ●

●

●●

●

●●●

●●

●

●●

●●

●●

●

●

●●●●

●

●

●

●

●●

●●

●●

●

●●

●

●●●

●

●

●

●

●●●

●●

●●●

●

●

●

● ●

●

●

●

●● ●

●

●

●

●●

●

●●

●

●

●● ●●

●

●

●

●

●●

●

●●

●

●●● ●

●

●

●

●

●● ●●

● ●

●

●

●

●

●●

●

●●●

●

●●

●●

●

● ●

●

●

●

●●

●

●●

●●● ●

●

●

●●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

● ●●

●

●

●●

●

●●

●

● ●●

●●

●●

●

●

●●

●

●

●●

●●

●

●●

● ●●●

●

●

●●

●●

●●●

●

●

●

●●

●

●● ●●●

●

●●

●

●

●

●●

●

● ●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●●●

●

●

● ● ●●● ●

●

●

●●

●●● ●

●

●●

●●●

●

●●

●●

●

● ●

●

●

●

●

● ●

●

●

●●

● ●●

●

●

●

●

●

●●●

●●● ●

●

●

●

●●

●

●

●

●●

● ●●●

●

●

●●

●

●●

●●

●

●

●●●

●●

● ●●

●●

●● ●

●

●● ●

●

●● ●

●

●

●●

●

●

●●

●

●●

●

● ●●

●●

●●

●

●

● ●

● ●

●

●●

●

●

●

●

●

●

●

●●

●●

●●

●

●●●

●

●

●●

●●●

●●

●

●

●

●

●

●●

●●

●

●

●

● ●

●●

●

● ●

●

●

●

●●

●

● ●

●

●

●

●●

●

●

●

●● ●

●

● ●

●

●

● ●

●

●●

●

●

●●●

●

●●

●

● ●

●

●

●●

●

● ●●

●

●

●● ●●

●

●●

●

●

●● ●

●●

●

●●

●

●●

●

●

●

● ●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●

●● ●●

●

●

●●

●

●

●●●●

●●

●● ●

●

●●

●

●

●

●●●

●●●●

●●

●

●

●

●

●

●

●●

●

●● ●

●

●

●

●

●

●

●●

●●●

●●

●

●●

●●●

●

●●

●

●

●

●●●

●●●

●●

●● ●●●

●●

●

●●

●

●

●●

●

●●●

●●

●●

●

●●

●

●

●●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

● ● ●●●●

●●

●●●●

●●

●

●●●

●

●●

●●●

●

●

●●

●

● ●●●

●

●

●

● ●●●

●

●●●

●

●●

●

●●

●

●

●

●● ●

●

●

●● ●●

●

●●

●

●

●●

●● ●

●●

●●

●

●

●

●

●

●

●

●●●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●●

●●

●

●●●

●

●● ●●

●●

●

●

●

● ●● ●

●

●

●●

● ●●

●

●

●

●

●

●

●

●

●

●

●●●●

●●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ● ●

●

●●

●

●● ●

●

●

●

●●

●

●●

●

●● ●

●

●●

●●

●●●

●

●

●●

● ●

●

●

●

●

●

●

●●

●

●

●● ●

●●

●●

●

●

●●

●●●●

●

●

●

●

●

●

●●

●●●

● ●

●

●

●

●●

●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●●●

●

●●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●● ●

●

●

●●

●

●

●

●●

●●

● ●●

●●

●●

●

●

●●

●

●

●

●●●●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●● ●

0e+00 2e+06 4e+06−50

0000

1500

000

y.test

ridge

.pre

d

(a)

●●

●

●

●

●

● ●●

●●

●

●●●●

●

●●●

●

●

●●● ●● ●●●

●●

●

●

●

● ●●

●

●

●

●●●

●

●

● ●●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●●

●

●●●

●

●

●●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●●●●

●

●

●

●●

●

●●●

●

●

●

●●

●

●

●

●●

●

●●●

●

● ●●●

●●●

●

●

●●

●

●

●●

●

●●●

●

●

●●

● ●

●●

●

●●

●●

●●

●

●

●

●

●●

●●

●

●

●

●

●

● ●●

●

●

●

●

●●

●

● ●

●

●

●

●

●●

●●

●●

●

●

●●

●

●●

●●●●

●

●

●

●●●

● ●

●

●●

●

●●●

● ●

●

●●

●●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●●●

●

●

●●

●●

●●●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●

●●

●●

●

●●●

●●

●

●●

●

●

●●

●

●● ●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

● ●

●

●

●

●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●● ●

●

● ●

●

●

●●

●

●

●

●●

●

●●●

● ●

●

●

●

●●●

●

●●

●

●

● ●

●

●

●

●

●

●●

●

● ●

●

● ●

●●

●●

●

● ●

●●

●●

●

●● ●

●

●●●

● ●

●

●●

●

●●

●●

●

●

●●

●●●

●●

●

●●

●●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●

● ●●

●

●

●● ●●

●

●● ●●

●

●

●●

●

●

●●

●●

●●

●●●

●

●

●

●

●

●

●

●

● ●

●

●

●●

●●

● ●

●

●●

●

●●

●●

●●

●

● ●●

●

●

●● ●● ●

●

●●●

●

● ●

●

● ●●●

●

●

●

● ●

●

●● ●

●● ●●

●

●

●

●

●

●●

● ●●

●

● ●●

● ●

●● ●● ●

●

●

●

●

●

● ●●

●

●

●

●

●●

●

●

●

●

●

●●●

●

● ●●

●

●●

●●

●●

●●

●

●●

●●

●

●

●

●●

● ●●●

●●

●

●

●

●

●●

●

●

●

●●

●●

●●

●

● ●

●●

● ●

●

●●

●

● ●●

●

●●

●●

●●

●●

●

●

●●●●

●

●

●

●

●

●●

●●

●

●

●●●

●

●●

●

●

●

●

●●

●

●●●

●●

●

●

●

● ●

●

●

●

●● ●

●

●

●

●●

●

●●

●

●

●●

●

●

●●

●●

●●

●

●●

●

●●● ●

●

●

●

●

●● ●●

● ●

●

●

●

●

●●

●

●●●

●

●

●

●●

●

●●

●

●

●

●●

●

●●

●●● ●

●

●

●●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

● ●●

●

●● ●

●

●●

●

● ●●

●●

●●

●

●

●●

●

●

●●

●●

●

●●

● ●●●

●

●

●●

●●

●●●

●

●

●

●●

●

●● ●●●

●

●●

●

●

●

●●

●

● ●

●●

●

● ●

●●●

●

●

●

●

●

●

●

●● ●●

●

● ●●●

●●

●●

●●

●●●

●

●

●●

●●●

●

●●

●●

●

●●

●

●

●

●

● ●

●

●

●●

● ●●

●

●

●●

●

●●●

●●● ●

●

●

●

● ●

●

●

●

● ●● ●

●●

●

●

●●

●

●●

●

●

●●

●●●

● ●

● ●●

●●

●● ●

●

●● ●

●●

●●

●

●

●●

●

●

●●

●

●●

●

● ●●

●●

●●

●

●

● ●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●●

●

●

●●

●●●

●●

●

●

●

●

●

●

● ●●

●

●

●

● ●

●

●

●

● ●

●

●

●

●●

●

● ●

●

●

●

●●

●

●

●

●●●

●

● ●

●

●

● ●

●

●●

●

●

●●

●

●

●●

●

● ●

●

●

●●

●

● ●●

●●

●● ●●

●

●●

●

●

●● ●

●●●

●●

●

●●

●

●

●

●●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●● ●●

●

●

●●

●

●

●● ●●●●

●● ●

●

●●

●

●

●

●●● ●

●●●

●●

●

●

●

●

●

●

●●

●

●● ●

●

●

●

●

●

●

●●

●●

●● ●

●

●●

●●●

●

●●

●

●●

●● ●

●

● ●

●●

● ● ●●●

●●

●

●●

●

●

●●

●

● ●●

●● ●●

●

●●

●

●

●●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

● ● ●●●●

●●

●●●●

●●

●

●

●●●

●

●

●●●

●

●

● ●

●

● ●●●

●

●

●

● ●●●

●

●●●

●

●●

●

●●

●

●

●

●●●

●

●

●● ●●

●

●●

●

●

●●

●● ●

●●

●

●

●

●

●

●

●

●

●● ●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

● ●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●●

●●

●

●●●

●●● ●●

●●

●

●

●

● ●●●

●

●

● ●

● ●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●●

●●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●●

●

●

● ●

●

●

●

●●

●

●

●

●

●● ●

●

●●

●●

●●●

●

●

●●

● ●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●●

●

●

●●

●●●●

●●

●

●

●

●

●

●

●●●

● ●

●

●

●

● ●

●

●●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●●●

●

●●●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●● ●

●

●

●●

●

●

●

●●

●●

● ●●

● ●

●●

●

●●

●

●

●

●●●

●●●

●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●

●●

●

12 13 14 15

12.5

13.5

14.5

y.test

ridge

.pre

d

(b)Figure 9.1: (a): Predicted cash prices using ridge regression against actual cash prices. (b): Predictedcash prices using ridge regression and log of cash price against log of actual cash prices.

This is also seen in Section 2.1, where we plot cash price and log of cash priceagainst some of the independent variables. The relation between log of cash priceand these independent variables seems to be linear, which is also an indicator thatwe should use log of cash price in the regression.

For this reason we use log of cash price instead of cash price in the regression.

When fitting the same model as before using log of cash price instead of cash priceand plotting the predicted cash prices against the actual cash prices, the relation isnow linear. This is seen in Figure 9.1b.

We create the model matrix x with the remaining independent variables and avector y containing the log of cash price as the dependent variable. The R imple-mentation is seen in Appendix A.1.

The next step is to perform ridge regression, Lasso, and elastic net on the data.

9.2 Ridge Regression

We perform ridge regression on the training data and select variables using 10-foldCV. The R implementation of ridge regression is seen in Appendix A.2.

In Figure 9.2 we see how the βs change as the value of λ changes. As λ increases,the βs decrease towards zero. The numbers above the plot indicate how manyindependent variables are included in the model at a given value of λ. We see thatnone of the beta coefficients are equal to zero at any point.

9.2. Ridge Regression 53

−4 −2 0 2 4

0.0

0.1

0.2

0.3

0.4

Log Lambda

Coe

ffici

ents

16 16 16 16 16

Figure 9.2: Ridge regression: βs against log (λ).

−4 −2 0 2 4

0.14

0.18

0.22

0.26

log(Lambda)

Mea

n−S

quar

ed E

rror

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

16 16 16 16 16 16 16 16 16 16 16 16 16

Figure 9.3: Ridge regression: Training MSE against log(λ).

In Figure 9.3 we see the training MSE as the value of λ changes. The MSE increasesas the value of λ increases. When λ increases, the βs shrink and the SSR increases.This means that the training MSE increases, since the training MSE is the aver-age error. The numbers above the plot indicate how many independent variables

54 Chapter 9. Application

are included in the model at a given value of λ. Since ridge regression does notperform variable selection, all independent variables are included in the model atevery value of λ.

The blue vertical line indicates the value of λ which gives the smallest trainingMSE. We denote this value λ∗. Here we have λ∗ = 0.0261.

The gray vertical line indicates the value of λ that gives the most regularized modelsuch that the training MSE is within one standard error of the minimum.

We use λ∗ to predict the cash price using the test set and get a test RMSE of 515, 562DKK. In contrast the average cash price per villa is 1, 252, 698 DKK. This means thatthe prediction on average is 41.16% off.

We then fit the model on the entire data set using λ∗ to obtain the βs seen in Table9.1.

Independent variables β

Intercept 9.191e+00Age -3.213e-03Balcony 1.724e-01Basement 3.505e-02Basement size 5.808e-04Close to water 1.216e-01GarageCarport 5.906e-02Large lot -5.657e-02Living space 3.114e-03Lot size 6.997e-06Number of bedrooms 6.995e-02Number of storeys 2.148e-02Renovation -4.322e-02State of residence Good 4.551e-01State of residence Medium 2.091e-01Year of construction 2.589e-03Zipcode -1.430e-04

Table 9.1: Ridge regression: The βs for the independent variables and the intercept.

9.3. Lasso 55

9.3 Lasso

As in ridge regression we perform Lasso shrinkage and selection on the trainingset and select independent variables using 10-fold CV. The R implementation ofLasso is seen in Appendix A.3.

−8 −7 −6 −5 −4 −3 −2

0.0

0.2

0.4

0.6

Log Lambda

Coe

ffici

ents

16 15 14 14 11 7 5

Figure 9.4: Lasso: β values against l1-norm.

In Figure 9.4 we see how the βs change as the value of λ changes. We see thatall the βs are equal to zero when λ is large. As λ decreases, more independentvariables are included in the model.

In Figure 9.5 we see the relation between the training MSE and the value of λ. Wesee that the MSE increases as the value of λ increases. We also see that the numberof independent values that are included in the model decreases as the value of λ

increases. We have λ∗ = 0.0017.

We predict the cash price using λ∗ and the test set and get a test RMSE of 516, 411DKK. Thus the prediction is on average 41.22% off.

We then use Lasso shrinkage and selection on the entire data set and find the βsusing λ∗. These βs are seen in Table 9.2. We see that all 16 independent variablesare included in the model and none of the independent variables are set equal tozero.

56 Chapter 9. Application

−8 −7 −6 −5 −4 −3 −2

0.14

0.18

0.22

0.26

log(Lambda)

Mea

n−S

quar

ed E

rror

●

●

●

●

●

●

●

●

●

●●

●●

●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

16 16 15 14 14 14 14 11 9 7 5 5 5 1

Figure 9.5: Lasso: Training MSE against log(λ).

Independent variables β

Intercept 9.018e+00Age -3.132e-03Balcony 1.629e-01Basement 3.529e-02Basement size 5.352e-04Close to water 1.276e-01GarageCarport 4.675e-02Large lot -5.086e-02Living space 3.263e-03Lot size 6.865e-06Number of bedrooms 6.547e-02Number of storeys 1.863e-02Renovation -4.326e-02State of residence Good 6.738e-01State of residence Medium 4.256e-01Year of construction 2.571e-03Zipcode -1.428e-04

Table 9.2: Lasso: The βs for the independent variables and the intercept.

9.4. Comparing Ridge Regression and Lasso 57

9.4 Comparing Ridge Regression and Lasso

Both the ridge regression method and Lasso regularisation includes 16 indepen-dent variables in the models. We see that the test RMSE is different in the twomodels. In the Lasso model we have a test RMSE of 516, 411 DKK while in ridgeregression we have a test RMSE of 515, 562 DKK. This means that ridge regressionregularisation results in a not worse model since the model obtained by the Lassomethod gives less accurate predictions.

In Table 9.1 and 9.2 we see the β for "Age" is negative. This indicates a negativeinfluence meaning that as the age increases the price decreases, which intuitivelymakes sense. The β for "State of residence Good" is larger than the β for "State ofresidence Medium". This indicates that a villa in a good state has a higher impacton the cash price, which also intuitively makes sense.

We can state that a model obtained using the ridge regression method will be abetter model for the data than the model obtained using Lasso.

We will test the elastic net method as well to see whether a model obtained usingthis method can give a more accurate prediction than a model obtained using ridgeregression.

58 Chapter 9. Application

9.5 Elastic Net

First we fit a model using the elastic net method with different values of α and λ.The R implementation is given in Appendix A.4

Mixing Percentage

RM

SE

(R

epea

ted

Cro

ss−

Val

idat

ion)

0.40

0.45

0.50

0.0 0.2 0.4 0.6 0.8 1.0

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●

● ● ● ● ● ● ● ● ●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●

● ● ● ● ●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●

●

●

●

●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●●

● ● ●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●

●

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

Regularization Parameter0.010.01373823795883260.0188739182213510.02592943797404670.03562247890262440.04893900918477490.06723357536499330.0923670857187387

0.1268961003167920.1743328822199990.2395026619987490.3290344562312670.4520353656360250.6210169418915620.8531678524172811.17210229753348

1.610262027560942.212216291070453.03919538231324.17531893656045.736152510448687.8804628156699110.826367338740514.8735210729351

20.433597178569428.072162039411838.566204211634752.983169062837172.7895384398315100

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Figure 9.6: Elastic net: Training RMSE against values of α and λ.

In Figure 9.6 we see the RMSE from different combinations of α and λ. On thex-axis we have α which we let vary from zero to one. The colored graphs representdifferent values of λ which we let vary from 0.01 to 100. The values of λ are seenabove the plot. The training RMSE tends to decrease as α approaches zero andsmaller values of λ results in a smaller training RMSE.

Therefore, we fit a model using elastic net where we let α vary from zero to 0.1 andλ from 0.01 to one.

In Figure 9.7 we see the RMSE for different combinations of these values. We seethe same tendency as in Figure 9.6 which indicates that the value of α that gives thesmallest training RMSE is closer to zero and the value of λ that gives the smallesttraining RMSE is closer to 0.01. The value of α that gives the smallest trainingRMSE is α∗ = 0.006.

In Figure 9.8 we see how the βs change as the value of λ changes. We see that

9.5. Elastic Net 59

Mixing Percentage

RM

SE

(R

epea

ted

Cro

ss−

Val

idat

ion)

0.40

0.42

0.44

0.46

0.00 0.02 0.04 0.06 0.08 0.10

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●

● ●●

● ● ●●

● ● ●●

● ● ●●

● ●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●

Regularization Parameter0.010.01172102297533480.01373823795883260.01610262027560940.0188739182213510.02212216291070450.02592943797404670.030391953823132

0.03562247890262440.0417531893656040.04893900918477490.05736152510448680.06723357536499340.07880462815669910.09236708571873860.108263673387405

0.1268961003167920.1487352107293510.1743328822199990.2043359717856940.2395026619987490.2807216203941180.3290344562312670.385662042116347

0.4520353656360240.5298316906283710.6210169418915620.7278953843983150.8531678524172811

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Figure 9.7: Elastic net: Training RMSE against values of α and λ.

−4 −2 0 2 4

0.0

0.2

0.4

0.6

Log Lambda

Coe

ffici

ents

16 15 15 12 2

Figure 9.8: Elastic net: βs against log (λ).

60 Chapter 9. Application

all the βs are equal to zero when λ is large. As λ decreases, more independentvariables are included in the model.

−4 −2 0 2 4

0.14

0.18

0.22

0.26

log(Lambda)

Mea

n−S

quar

ed E

rror

●●●●●●●●●●●●●●●●●●●●●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

16 16 15 16 15 16 16 14 13 12 10 8 5

Figure 9.9: Elastic net: Training MSE against log(λ)

.

We fit a model using elastic net and α∗ and plot the training MSE against differentvalues of λ. This is seen in Figure 9.9. Once again we see that the training MSEtends to increase as the value of λ increases. The blue vertical line indicates λ∗ =

0.0043.

We predict the cash price using α∗ and λ∗ and the test set, and we get a test RMSEof 515, 372 DKK. On average our prediction is 41.14% off.

We then fit a model using elastic net, α∗, and λ∗ on the entire data set to obtain theβs seen in Table 9.3. We see the same tendency regarding the βs for "Age", "Stateof residence Good" and "State of residence Medium" as in Table 9.1 and 9.2.

9.5. Elastic Net 61

Independent variables β

Intercept 9.126e+00Age -3.247e-03Balcony 1.732e-01Basement 3.645e-02Basement size 5.512e-04Close to water 1.303e-01GarageCarport 5.083e-02Large lot -5.951e-02Living space 3.257e-03Lot size 7.529e-06Number of bedrooms 6.650e-02Number of storeys 2.119e-02Renovation -4.911e-02State of residence Good 6.561e-01State of residence Medium 4.102e-01Year of construction 2.544e-03Zipcode -1.475e-04

Table 9.3: Elastic net: The βs for the independent variables and the intercept.

We conclude that the model fitted using elastic net is better than both the ridgeregression model and the Lasso model since the test RMSE of the model fittedusing elastic net is not worse than that of ridge regression and Lasso. Since themodel fitted using elastic net has the smallest test RMSE, we choose to use thismodel for prediction of the cash price.

62 Chapter 9. Application

9.6 Model Validation

To validate the model we start by fitting a model using elastic net on the entiredata set consisting of the 95% of our data, where we before tested elastic net onhalf of the data. The R implementation is given in Appendix A.5.

We then use 10-fold CV to find λ∗. This is seen in Figure 9.10, where MSE is plottedagainst log of λ.

−4 −2 0 2 4

0.14

0.18

0.22

0.26

log(Lambda)

Mea

n−S

quar

ed E

rror

●●●●●●●●●●●●●●●●●●●●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

16 16 16 16 16 16 15 15 13 13 9 8 6 2

Figure 9.10: Testing elastic net: Training MSE against log(λ).

We see that the MSE is smallest for small values of λ. The blue vertical line illus-trates a λ∗ of 0.0042.

We use this λ∗ to predict the cash price using the held out set consisting of the last5% of the data and get a RMSE of 524, 813 DKK. We predict on average 41.89% offwhen testing on the last 5% of the data. The βs we get using the λ∗ are given inTable 9.4.

9.7. Improving the Model 63

Independent variables β

Intercept 9.138e+00Age -3.243e-03Balcony 1.730e-01Basement 3.614e-02Basement size 5.610e-04Close to water 1.275e-01GarageCarport 5.366e-02Large lot -5.883e-02Living space 3.218e-03Lot size 7.354e-06Number of bedrooms 6.766e-02Number of storeys 2.139e-02Renovation -4.733e-02State of residence Good 5.826e-01State of residence Medium 3.363e-01Year of construction 2.566e-03Zipcode -1.478e-04

Table 9.4: Testing elastic net: The βs for the independent variables and the intercept.

We can conclude that a test RMSE of 524, 813 DKK is quite high and that theprediction of the model differs 41.89% from the average cash price.

9.7 Improving the Model

This section is based on [Ng, 2014a], [Ng, 2014c], and [Ng, 2014b].

Some ways of improving predictive models are to

• Get more training observations.

• Try smaller sets of independent variables.

• Try getting additional independent variables.

• Try adding polynomial independent variables.

• Try decreasing λ.

• Try increasing λ.

64 Chapter 9. Application

• Perform feature engineering.

The two options – get more training examples and try getting additional indepen-dent variables – can take months. We can use machine learning diagnostics tochoose which way we should improve our model.

In a predictive model it is the bias-variance trade-off we want to improve. Highvariance can be reduced by using the following options: get more training exam-ples, try smaller sets of features, and increasing λ. If we want to reduce a highbias, then we can use: getting additional independent variables, adding polyno-mial independent variables, decreasing λ, and feature engineering.

From the models learning curve, we can see whether the model has high varianceor high bias.

9.7.1 Learning Curves

To see whether there is high bias or high variance we plot error as a function of thenumber of observations in the training set, m. We work with two errors: TrainError

is the mean squared error of the training set, and CVError is the mean squared errorof the unknown CV set.

The TrainError is small for a small m, because we can fit the training set close toperfectly. For a larger m it can be difficult to find a function that fits the training setperfectly, which makes the error increase. This means that the TrainError increasesas m → ∞. The CVError decreases as m → ∞, since the more data you have, thebetter you can fit the model.

m

Err

or

High Bias

CVError

TrainError

(a)m

Err

or

High Variance

CVError

TrainError

Gap

(b)Figure 9.11: (a): Error plotted against m in the case of high bias. (b): Error plotted against m in thecase of high variance.

An example of what would cause high bias, is if we try to fit a straight line on adata set, where the line does not fit the data well. Here collecting more data is not

9.7. Improving the Model 65

going to change the straight line much. This means that the CVError decreases abit, but when you reach a certain m the CVerror curve will flatten out. The TrainError

will be small for a small m, and increase until it gets close to the CVError. This isillustrated in Figure 9.11a. We see that high bias will give a high error, which cannot be compensated by a high m.

The case of high variance is illustrated in Figure 9.11b. En example of high vari-ance, would be a higher order polynomial. The polynomials tend to fit the datawell for a small m, but give high variance since they tend to overfit the data. Asm increases, the TrainError increases because it gets more difficult to fit the dataperfectly. The CVError is high for a low m, and as m increases, the CVError de-creases. These two curves converge towards each other and the gap between themdecreases. Therefore as m increases, the variance will decrease.

Implementation

We start by creating a sample set of the data of size m · 2. This set is divided intotwo sets: training set and CV set, both of size m. We fit a model using elastic netand the values α∗ = 0.006 and λ∗ = 0.0042 on the training set. We predict the cashprice using the training set to get the in-sample error and calculate the MSE. Weplot the MSE for each size of the training set.

●0 500 1000 1500

4000

0010

0000

0

m

erro

r

●

●

●

●

●

●

●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●●●

●

●●●●●

●●

●

●

●

●

●●

●●●●●

●●●

●

●●●●●●

●●●●●

●●●●

●

●

●

●●●●●●●

●

●●●●●●●●●●

●●●

●

●

●●●●

●

●

●

●

●●●●

●

●●●

●

●●●●●●●●●

●●●●

●

●●●●●●●

●

●●●●●

●

●●●●●●●●●●●

●●●●●

●

●●●●●●

●●●●●

●

●●●●●

●

●●●

●

●

●●●●●●●●●●●

●●●●●●●

Figure 9.12: Elastic net: RMSE against number of observations in training set.

This is illustrated by the blue points in Figure 9.12. We then predict the cash priceusing the CV set to get the out-of-sample error and calculate the MSE. This error

66 Chapter 9. Application

is plotted for each size of the training set, which is illustrated by the red points inFigure 9.12.

In Figure 9.12 we see that the model suffers from high bias, which gives us theknowledge to improve the model.

Since the model has high bias we can choose to get additional independent vari-ables, add polynomial independent variables, decrease λ, or perform feature engi-neering. We choose to perform feature engineering.

9.7.2 Feature Engineering

Feature engineering is the process of constructing independent variables from (theknowledge of) the data. This is done to improve the performance of the model.An independent variable gives information that may be useful for the prediction.Independent variables are important to predictive models since they influence theresults we achieve. The process of feature engineering is to brainstorm and decidewhich new variables to construct. The next step is to create the variables and checkhow they work in the model. If the new variables are not satisfactory, we eitherimprove them or brainstorm again.

By dividing living space with number of bedrooms we get the average room size,which could have an impact on the cash price. Likewise by dividing lot size withliving space we can see how large a percentage of the lot is living space, whichcould also impact the cash price. We therefor create the two new variables

Living spaceNumber of bedrooms

,Living space

Lot size.

Implementation

We create the two new variables: the average size of the rooms in the villa (AverageRoom Size) and the fraction of living space to lot size (Living Space To Lot Size).

We then make a model using elastic net with the values of α∗ and λ∗ in Section 9.6.

We have α∗ = 0.006 and λ∗ = 0.0042. We use these values to predict the cash priceusing the last 5% of the data and get a test RMSE of 521, 278 DKK which is a bitsmaller than the RMSE of the model in Section 9.6. We predict 41.61% off fromthe average cash price when we improve the model with two new independentvariables. The βs we get are given in Table 9.5.

In Table 9.5 we see the two new variables: average room size and living space tolot size. Since the β for average room size is larger than the β for living space to

9.7. Improving the Model 67

lot size, we can intuitively conclude that average room size has a larger impact onthe cash price.

Independent variables β

Intercept 9.055e+00Age -3.256e-03Average room size 4.484e-04Balcony 1.685e-01Basement 4.390e-02Basement size 5.757e-04Close to water 1.280e-01GarageCarport 5.862e-02Large lot -1.533e-02Living space 3.053e-03Lot size 8.516e-06Living space to lot size 3.187e-01Number of bedrooms 6.711e-02Number of storeys 9.872e-03Renovation -5.029e-02State of residence Good 6.533e-01State of residence Medium 3.090e-01Year of construction 2.589e-03Zipcode -1.525e-04

Table 9.5: Elastic net with feature engineering: The βs for the independent variables and the inter-cept.

Chapter 10

Summary

Initially we clean up the given data and plot some of the independent variablesagainst each other and against the cash price. Some of these plots show a lineardependency which is enhanced when we plot the log of cash price against the logsof some of the independent variables.

Using multiple linear regression we can model house prices using a linear regres-sion model containing the unknown parameter β. We introduce the ordinary leastsquares method to estimate this parameter as well as orthogonal projections to bet-ter understand the geometry of the ordinary least squares. Furthermore we seethat β and σ2 are both unbiased, which leads us to the Gauss-Markov theorem.The theorem concludes that β is a best linear unbiased estimator.

To give a measure of how well a model predicts or fits the given data, we explorethe theory of mean squared error. We contemplate training and test mean squarederror as well as the bias-variance trade-off to minimize the test mean squared errorand thereby optimize our model. We conclude the section by discussing two typicalmistakes when fitting a model.

To further assess a model we want to be able to tell how it performs and thuswe introduce three different methods for resampling our data set. The validationset approach is quickly discarded as it overestimates the error term of the model.The leave-one-out cross-validation and k-fold cross-validation turn out to be equalwhen n = k. We contemplate the bias-variance trade-off for leave-one-out cross-validation and k-fold cross-validation.

To account for alternatives to fitting a model we describe subset selection andshrinkage methods. For the shrinkage methods we use ridge regression, Lasso,and elastic net.

69

70 Chapter 10. Summary

Since we previously suggested a best linear unbiased estimator, we discuss thistopic further in the maximum likelihood section. Here we see that given the unbi-asedness and variance of the estimate β, the β attains the Cramer-Rao lower bound.This means that β is the best unbiased estimator, which is a stronger statement thanthe best linear unbiased estimator.

We discuss how to make inferences about the true β via hypothesis testing. Wecursorily describe z-ratio and t-values relation to confidence intervals. This leadsto a definition of the prediction interval, which can tell us how much uncertaintyis linked to the prediction.

In the application section we see that the model obtained by elastic net is favoredto the models obtained by ridge regression and Lasso, since the root mean squarederrors of these two models are larger than that of the model obtained by elasticnet. The root mean squared error for elastic net is 515, 872 DKK and when wetest it on the last 5% of data, we get a root mean squared error for elastic net of524, 813 DKK. Thus the models prediction is 41.89% off the average cash price of avilla. Therefore we try to improve the model by examining the learning curves andhereby see whether it suffers from high bias or variance. Since the model suffersfrom high bias, we try to improve it by performing feature engineering and createtwo new independent variables. This improves the root mean squared error to521, 278 DKK, but we still predict 41.61% off.

Chapter 11

Critique

When using OLS to estimate the model parameters in multiple linear regression,we assume that the independent variables are observed correctly. This means, thatthe only errors that we take into account are the errors in the output observations.However this is not always the case. There may be some measurement errorsin the observations of the independent variables. If we also take these errors intoaccount we can no longer use OLS. Instead we could use Total Least Squares (TLS).In TLS we seek to minimize the orthogonal error from the observed output tothe predicted error. This means that we now get a prediction where the errorsin the independent variables are considered, and we therefore get different betaparameters.

To check whether there are errors in the observations of the independent variablesin the home data, we can fit a linear regression model using TLS. If the predictedoutput produced by TLS is the same as the predicted output produced by OLS,there are no errors in the independent variables. If the sets of predicted outputare not equal to each other, there are errors in the observations of the independentvariables, and we should use TLS to predict housing prices. This is not accountedfor in the project.

If we had had more time we could have explored feature engineering further bytesting quadratic independent variables. We could also contemplate what wouldhappen if we multiplied the independent variables with each other.

We get a high RMSE of the model of 524, 813 DKK which means that the predictionis 41.89% off. If a linear model does not give an acceptable prediction, we couldtry and fit a nonlinear model.

71

Bibliography

bquanttrading (2016). Fitting elastic net model in R. URL: https://quantmacro.wordpress.com/2016/04/26/fitting-elastic-net-model-in-r/.

Breheny, P. (2011). Ridge regression. URL: http://web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf.

Breiman, L. (1994). Heuristics of instability in model selection. URL: http://projecteuclid.org/euclid.aos/1032181158.

Davidson, R. and MacKinnon, J. G. (2003). Econometric Theory and Methods. OxfordUniversity Press.

Faraway, J. J. (2002). Practical regression and ANOVA using R. URL: https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf.

Giles, D. (2013). Forecasting from log-linear regres-sions. URL: https://davegiles.blogspot.dk/2013/08/forecasting-from-log-linear-regressions.html.

Hastie, T., Tibshirani, R., and Friedman, J. (2013). The Elements of StatisticalLearning. Springer, 2 edition. URL: http://statweb.stanford.edu/~tibs/ElemStatLearn/.

Hayashi, F. (2000). Econometrics. Princeton University Press.

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2015). An Introduction to Statis-tical Learning. Springer. URL: http://www-bcf.usc.edu/~gareth/ISL/.

Madsen, H. and Thyregod, P. (2011). Introduction to General and Generalized LinearModels. CRC Press.

Ng, A. (2014a). Machine learning w6 1 deciding what to trynext. URL: https://www.youtube.com/watch?v=c03aZtGvKVw&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=59.

73

74 Bibliography

Ng, A. (2014b). Machine learning w6 6 learning curves.URL: https://www.youtube.com/watch?v=rczX4iAHhS0&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=64.

Ng, A. (2014c). Machine learning w6 7 deciding what to do next re-visited. URL: https://www.youtube.com/watch?v=Kum1UFqKEp4&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=65.

"nograpes" (2014). How to specify log link in glm-net? URL: http://stackoverflow.com/questions/25206844/how-to-specify-log-link-in-glmnet.

Tibshirani, R. (2013). Regression 2: More perspectives, shortcomings. URL: http://www.stat.cmu.edu/~ryantibs/datamining/lectures/14-reg2-marked.pdf.

van Wieringen, W. N. (2015). Lecture notes on ridge regression. URL: https://arxiv.org/abs/1509.09169.

Vijayakumar, S. (2016). The bias-variance tradeoff. URL: http://www.inf.ed.ac.uk/teaching/courses/mlsc/Notes/Lecture4/BiasVariance.pdf.

Appendix A

Implementation in R

The implementation is based on ["nograpes", 2014].

A.1 Cleaning the Data For Modelling

The implementation in this section is based on [James et al., 2015, pp. 244, 251,253].

names(HOME1)[names(HOME1) == "Postnr"] = "ZipCode"names(HOME1)[names(HOME1) == "Salgsdato"] = "DateOfSale"names(HOME1)[names(HOME1) == "EjdType"]= "TypeOfResidence"names(HOME1)[names(HOME1) == "Boligtilstand"]= "StateOfResidence"names(HOME1)[names(HOME1) == "Kontantpris"] = "CashPrice"names(HOME1)[names(HOME1) == "Boligareal"]= "LivingSpace"names(HOME1)[names(HOME1) == "Kaelderareal"]= "BasementSize"names(HOME1)[names(HOME1) == "Grundareal"] = "LotSize"names(HOME1)[names(HOME1) == "Opfoerelsesaar"]= "YearOfConstruction"names(HOME1)[names(HOME1) == "AntalPlan"]= "NumberOfStoreys"names(HOME1)[names(HOME1) == "NytKokken"] = "NewKitchen"names(HOME1)[names(HOME1) == "AntalToiletter"]

75

76 Appendix A. Implementation in R

= "NumberOfToilets"names(HOME1)[names(HOME1) == "NyeVinduer"] = "NewWindows"names(HOME1)[names(HOME1) == "NytBadevaerelse"]= "NewBathroom"names(HOME1)[names(HOME1) == "AntalSovevaerelser"]= "NumberOfBedrooms"names(HOME1)[names(HOME1) == "Altan"] = "Balcony"names(HOME1)[names(HOME1) == "TaetVedVand"]= "CloseToWater"names(HOME1)[names(HOME1) == "FuldAdresse"]= "FullAddress"names(HOME1)[names(HOME1) == "Liggetid"] = "TurnoverTime"names(HOME1)[names(HOME1) == "Salgsaar"] = "YearOfSale"names(HOME1)[names(HOME1) == "MangeToil"] = "ManyToilets"names(HOME1)[names(HOME1) == "Kvartal1"] = "Quarter1"names(HOME1)[names(HOME1) == "Kvartal2"] = "Quarter2"names(HOME1)[names(HOME1) == "Kvartal3"] = "Quarter3"names(HOME1)[names(HOME1) == "Kvartal4"] = "Quarter4"names(HOME1)[names(HOME1) == "Kaelder"] = "Basement"names(HOME1)[names(HOME1) == "Storgrund"] = "LargeLot"names(HOME1)[names(HOME1) == "Alder"] = "Age"names(HOME1)[names(HOME1) == "Hoejhus"] = "HighRise"names(HOME1)[names(HOME1) == "OmbygningSket"]= "Renovation"

library("dplyr")

HOME1 = subset(HOME1 , TypeOfResidence %in% "Villa ,1 fam.")HOME1 = subset(HOME1 , LotSize > 0)remove = c("NumberOfToilets", "ManyToilets","NewKitchen", "NewWindows", "NewBathroom", "HighRise")HOME1 = HOME1[,!(names(HOME1) %in% remove )]HOME1$NumberOfStoreys[HOME1$NumberOfStoreys == 15] = 1.5HOME1$NumberOfStoreys[HOME1$NumberOfStoreys == 25] = 2.5

# Sort the data by date.HOME1 = HOME1[order(as.Date(HOME1$DateOfSale , format= "%d-%m-%Y")),]

A.1. Cleaning the Data For Modelling 77

hometry = HOME1

remove = c("Quarter1", "Quarter2", "Quarter3", "Quarter4","TurnoverTime")hometry = hometry[,!(names(hometry) %in% remove )]

# Change some of the independent variables to factors.hometry$Basement = factor(hometry$Basement , labels= c("No","Yes"))hometry$LargeLot = factor(hometry$LargeLot , labels= c("No","Yes"))hometry$Renovation = factor(hometry$Renovation , labels= c("No","Yes"))hometry$StateOfResidence= factor(hometry$StateOfResidence , labels= c("Bad","Good","Medium"))hometry$Balcony = factor(hometry$Balcony , labels= c("No", "Yes"))hometry$CloseToWater = factor(hometry$CloseToWater ,labels = c("No", "Yes"))hometry$GarageCarport = factor(hometry$GarageCarport ,labels = c("No", "Yes"))

# Remove the independent variables DateOfSale ,# TypeOfResidence , FullAdress og YearOfSale.remove = c("DateOfSale", "TypeOfResidence","FullAddress", "YearOfSale")hometry = hometry[,!(names(hometry) %in% remove )]summary(hometry)

# Remove NAs.sum(is.na(hometry ))hometry = na.omit(hometry)dim(hometry)sum(is.na(hometry ))

# Take out the 5 percent latest sold villas (outhometry ).rows = dim(hometry )[1]fivepercent = floor(rows * 0.05)inhometry = hometry [1:(rows -fivepercent ),]

78 Appendix A. Implementation in R

outhometry = hometry [(rows -fivepercent +1):rows ,]

# Create the model matrix with all the independent# variables.x = model.matrix(CashPrice ~ ., inhometry )[,-1]xout = model.matrix(CashPrice ~ ., outhometry )[,-1]

# Create a vector containing the dependent variable.y = inhometry$CashPriceyout = outhometry$CashPrice

# Split the data into two sets: training set and test set.set.seed (1)train = sample (1: nrow(x), nrow(x)/2)test = (-train)

# Create the vector containing the dependent variables# for the test set.y.test = y[test]

A.2 Ridge Regression Implementation

The implementation in this section is based on [James et al., 2015, pp. 251, 253-254].

# Create 30 lambda values from 1e-2 to 1lambdagrid = 10^seq(0,-2, length = 30)

## Ridge Regression

library(glmnet)

# Fit ridge regression (alpha = 0) on the training data.ridge.mod = glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = 0, lambda = lambdagrid)

# Fit rdige regression (alpha = 0) on the training set# using 10-fold CV (by default ).

A.2. Ridge Regression Implementation 79

set.seed (1)cv.out = cv.glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = 0)

# Plot the beta values vs log of lambda.plot(cv.out $glmnet.fit , "lambda")

# Plot MSE vs. log of lambda.plot(cv.out)

# Find the lambda with the smallest MSE.bestlam = cv.out$lambda.minbestlamabline(v = log(bestlam), col = "blue")

# Predict the dependent variable where lambda is the best# lambda value using the independent variables in the# test set.ridge.pred = predict(ridge.mod , s = bestlam , newx= x[test ,])

# Calculate the RMSE.RMSE = sqrt(mean((exp(ridge.pred) - y.test )^2))RMSE

# How large is the RMSE compared to the mean cash price# of a villa?mean = mean(y)RMSE/mean*100

# Fit the ridge regression model on the entire data set.out = glmnet(x, log(y), family = "gaussian", alpha = 0,lambda = bestlam)

# Get the beta values for this model when lambda is the# best lambda.ridge.coef = predict(out , type = "coefficients")[1:dim(coef(ridge.mod ))[1] ,]ridge.coef

80 Appendix A. Implementation in R

A.3 Lasso Implementation

The implementation in this section is based on [James et al., 2015, p. 255].

# Fit a lasso model (alpha = 1) on the training data.lasso.mod = glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = 1)

# Fit lasso (alpha = 1) on the training set using 10-fold# CV (by default ).set.seed (1)cv.out = cv.glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = 1)

# Plot the beta values vs log of lambda.plot(cv.out $glmnet.fit , "lambda")

# Plot the MSE vs lambda values.plot(cv.out)

# Find the best lambda.bestlam = cv.out$lambda.minbestlamabline(v = log(bestlam), col = "blue")

# Predict the dependent variable using the best lambda# and the independent variables in the test set.lasso.pred = predict(lasso.mod , s = bestlam , newx= x[test ,])

# Calculate the RMSE.RMSE = sqrt(mean((exp(lasso.pred) - y.test )^2))RMSE

RMSE/mean*100

# Fitting the lasse on all the data.out = glmnet(x, log(y), family = "gaussian", alpha = 1,lambda = bestlam)

A.4. Elastic Net Implementation 81

# Find the beta estimates using the best lambda.lasso.coef = predict(out , type = "coefficients")[1:dim(coef(lasso.mod ))[1] ,]lasso.coef

A.4 Elastic Net Implementation

The implementation in this section is based on [bquanttrading, 2016].

# Create grid for alpha.alphagrid = seq(0, 0.1, length = 51)

# Fit elastic nets using 10-fold CV on the training set.library(caret)trnCtrl = trainControl(method = "repeatedcv", number = 10,repeats = 5)searchgrid = expand.grid(.alpha = alphagrid , .lambda= lambdagrid)set.seed (2016)my.train = train(CashPrice ~ ., data = inhometry ,

method = "glmnet",tuneGrid = searchgrid ,trControl = trnCtrl ,standardize = TRUE)

plot(my.train)

# Find the best alphabestalpha = drop(as.matrix(my.train$bestTune [1]))bestalpha

# Fit a model on the training set.elanet.mod = glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = bestalpha)

# Fit Elastic Net for best alpha = 0.004 on the training# set using 10-fold CV (by default ).set.seed (1)

82 Appendix A. Implementation in R

cv.out = cv.glmnet(x[train ,], log(y[train]), family= "gaussian", alpha = bestalpha)

# Plot the beta values vs log of lambda.plot(cv.out $glmnet.fit , "lambda")

# Plot the MSE vs lambda values.plot(cv.out)

# Find the best lambda.bestlam = cv.out$lambda.minbestlamabline(v = log(bestlam), col = "blue")

# Predict the dependent variable using the best lambda# and the independent variables in the test set.elanet.pred = predict(elanet.mod , s = bestlam , newx= x[test ,])

# Calculate the RMSE.RMSE = sqrt(mean((exp(elanet.pred) - y.test )^2))RMSE

RMSE/mean*100

# Fitting the Elastic Net on all the data.out = glmnet(x, log(y), family = "gaussian", alpha= bestalpha , lambda = bestlam)

# Find the beta estimates using the best lambda.elanet.coef = predict(out , type = "coefficients")[1:dim(coef(elanet.mod ))[1] ,]elanet.coef

A.5. Validation Elastic Net 83

A.5 Validation Elastic Net

# Fit on last 5 % of data.

# Fit elastic net on the entire data.FINAL.mod = glmnet(x, log(y), family = "gaussian", alpha= bestalpha , lambda = lambdagrid)

# Fit elastic net on entire data using 10-fold CV# (by default ).set.seed (1)cv.out = cv.glmnet(x, log(y), family = "gaussian", alpha= bestalpha)

# Plot the beta values vs log of lambda.plot(cv.out $glmnet.fit , "lambda")

# Plot MSE vs. lambda value.plot(cv.out)

# Find the lambda with the smallest MSE.bestlam = cv.out$lambda.minbestlamabline(v = log(bestlam), col = "blue")

# Predict the dependent variable where lambda is the best# lambda value using the independent variables in the# test set.FINAL.pred = predict(FINAL.mod , s = bestlam , newx = xout)

# Calculate the RMSE.RMSE = sqrt(mean((exp(FINAL.pred) - yout )^2))RMSE

RMSE/mean*100

# Fit elastic net on the entire data set.out = glmnet(x, log(y), family = "gaussian", alpha= bestalpha , lambda = bestlam)

84 Appendix A. Implementation in R

# Get the beta values for this model when lambda is the# best lambda.FINAL.coef = predict(out , type = "coefficients")[1:dim(coef(FINAL.mod ))[1] ,]FINAL.coef

A.6 Learning Curves

m = 0error = 0plot(m, error , col = "white", xlim = c(0, 1600), ylim= c(300000 , 1500000))for(m in seq(from = 16, to = 1600, by = 16)){

# Split the data into two sets: training set and test# set.set.seed (1)splitting = sample(nrow(x), size = m*2)x_m = x[splitting ,]y_m = y[splitting]train = sample (1: nrow(x_m), m)test = (-train)x.train = x_m[train ,]x.test = x_m[test ,]

# Create the vector containing the dependent variables# for the test set.y.train = y_m[train]y.test = y_m[test]

# Fit a model on the training set.elanet.mod = glmnet(x.train , log(y.train), family= "gaussian", alpha = 0.006)

# Predict the dependent variable using the best lambda# and the independent variables in the test set.elanet.pred = predict(elanet.mod , s = 0.0042 , newx

A.7. Feature Engineering 85

= x.train)

# Calculate the RMSE.error = sqrt(mean((exp(elanet.pred) - y.train )^2))points(m, error , type = "p", col = "blue")

# Predict the dependent variable using the best lambda# and the independent variables in the test set.elanet.pred.cv = predict(elanet.mod , s = 0.0042 , newx= x.test)

# Calculate the RMSE.error = sqrt(mean((exp(elanet.pred.cv) - y.test )^2))points(m, error , type = "p", col = "red")

}

A.7 Feature Engineering

# Create the two new independent variablesHOME1 = mutate(HOME1 , AvgSizeRoom = LivingSpace/ (NumberOfBedrooms + 1))HOME1 = mutate(HOME1 , LivingSpaceToLotSize = LivingSpace/ LotSize )

# Fit on last 5 % of data.

# Fit elastic net on the entire data.EN.mod = glmnet(x, log(y), family = "gaussian", alpha= bestalpha , lambda = bestlam)

# Predict the dependent variable where lambda is the best# lambda value using the independent variables in the# test set.EN.pred = predict(EN.mod , s = bestlam , newx = xout)

# Calculate the RMSE.RMSE = sqrt(mean((exp(EN.pred) - yout )^2))

86 Appendix A. Implementation in R

RMSE

RMSE/mean*100

# Fit elastic net on the entire data set.out = glmnet(x, log(y), family = "gaussian", alpha= bestalpha , lambda = bestlam)

# Get the beta values for this model when lambda is the# best lambda.EN.coef = predict(out , type = "coefficients")[1:dim(coef(EN.mod ))[1] ,]EN.coef

Appendix B

From Log of Cash Price To Cash Price

This Appendix is based on [Giles, 2013].

When using log of cash price to find the βs, we predict log of cash price. To get thepredicted cash price instead of log of cash price, we usually use the exponentialfunction. However this is not correct when doing linear regression. In Assumption7.1 we assume that the error term follows a normal distribution and therefore thedependent variable, log of cash price, also follows a normal distribution. Thismeans that cash price follows a log-normal distribution. We have

X ∼ N(µ, σ2) ,

Y = exp (X) ∼ Log-N (m, v) ,

where m and v are the mean and the variance of the log-normal distribution. Wehave from probability theory

m = exp(

µ +σ2

2

)v =

(exp

(σ2)− 1

)exp

(2µ + σ2) .

Due to Assumption 3.1 we have

E [yt] = exp(

β0 + β1x1t + · · ·+ βpxpt +σ2

2

).

This means that if we want to convert the predicted log of cash price to predictedcash price, we use

yt = exp(

log (yt) +σ2

2

),

where log (yt) = β0 + β1x1t + · · · + βpxpt and σ2 is the unbiased estimate of σ2

given in Definition 3.6.

87