Statistics and Linear AlgebraStatistics and Linear Algebra

(the real thing)(the real thing)

VectorVector

A vector is a rectangular arrangement of number in several rows and one column. A vector is denoted by a lowercase bold font weight: x

A vector element is given by the value (scalar) of a given row. x1=6 and x2=1

The number of elements gives its dimensionality Since there are two elements, the dimensionality of x is 2. (Each elements can be considered as a subject score)

The transpose of vector, denoted T (or `), is a rotation of the column and rows.

6

1

x

Definitions

TT 6 1x

VectorVector If the dimensionality of a vector is equal or less than 3, it can be represented graphically A vector is represented as an arrow (orientation and length)

6

1

x

VectorVector

1.5

6 91.5

1 1.5

s

s

x

Operations: scalar multiplication When a vector is multiply by a scalar, each element of the vector is multiply by the scalar. When a vector is multiply by a scalar (number), then its length is increased by the factor of the scalar.

VectorVector

1

6 61

1 1

s

s

x

When a vector is multiply by -1, then it will reverse is direction.

Operations: -1 multiplication

VectorVector

6 2 and

1 5

6 2 8 +

1 5 6

x y

x y

When two vectors are added together, we sum their corresponding elements. Graphically, we put the beginning of the second vector at the end of the first.

Operations: Addition of two vectors

VectorVector

6 2 and

1 5

6 2 6 2 4 - +

1 5 1 5 4

x y

x y

When two vectors are added together, we sum their corresponding elements. Graphically, we put the beginning of the second vector at the end of the first one, once the second vector has been multiply by -1.

Operations: Subtraction of two vectors

What 2x-3y will gives ?

MatricesMatrices

A matrix can be view as a collection of vectors. A matrix is denoted by an uppercase bold font weight: M The matrix dimension is given by its number of rows and columns

Example: 2 rows and 4 columns: 24 A vector can be view as a matrix with many rows and one column. A matrix is called “square”, if n×n. A matrix element is given by the junction of the given row and column, denoted at Mij

Definitions

1 2 5 6

2 4 5 6

M 13 5M

MatricesMatrices

When a matrix is multiply by a scalar, each element is multiply by the scalar.

Operation: multiplication of matrix by a scalar

1.5

1 2 5 6 1.5 3 7.5 91.5

2 4 5 6 3 6 7.5 9

s

s

M

If two vectors have the same dimension, then they can be multiply together There will be two possible results: a) A scalar or b) A matrix

Scalar (inner product, dot product) Two vectors will output a scalar, if the first vector is transposed before being multiplied with the second vector (of equal dimension). The row of the first vector is multiplied by the corresponding element of the second vector, and the resulting products are sum up.

MatricesMatricesOperation: Product of two vectors

1 1

2 2

3 3

1TT

1 2 3 2 1 1 2 2 3 3

3

and ;

x y

x y

x y

y

x x x y x y x y x y

y

x y

x y

T

4 3 and

6 5

?

x y

x y

If we divided xTy by their corresponding degrees of freedom (n-1) we obtain the covariance between the two variables (if the mean is zero).

MatricesMatrices

Matrix (outer product) Two vectors will output a matrix, if the second vector is transposed before being multiplied. The column of first vector is multiplied by the corresponding element of the second

vector row.

Operation: Product of two vectors

1 1

2 2

3 3

1 1 1 1 2 1 3TT

2 1 2 3 2 1 2 2 2 3

3 3 1 3 2 3 3

and ;

x y

x y

x y

x x y x y x y

x y y y x y x y x y

x x y x y x y

x y

xy

T

4 3 and

6 5

?

x y

xy

MatricesMatrices

Two matrices can be multiplied together, if the number of columns of the first matrix is equal to the number of rows of the second matrix. Ex: If A is a m3 matrix, then B must be a 3n matrix. The resulting matrix C will be a mn matrix

The matrix product is not commutative: ABBA

Operation: Product of two matrices

3 12 3 1

and 4 2 ;1 4 0

5 3

3 12 3 1 (2 3) ( 3 4) (1 5) (2 1) ( 3 2) (1 3) 1 7

4 21 4 0 ( 1 3) (4 4) (0 5) ( 1 1) (4 2) (0 3) 13 7

5 3

A B

C AB

9 8 7

1 2 3 4 6 5 4 and ; ?

5 6 7 8 3 2 1

0 9 8

X Y XY

MatricesMatrices

There is a special kind of matrix that is similar to the arithmetic multiplication by one. 51=5 This matrix is called: Identity, denoted by I, where all its diagonal elements are set to one and the remaining elements to 0. Since this matrix has the same number of columns and rows: AI=A or IA=A

Identity matrix

1 0 0 0 1 2 3 4

0 1 0 0 5 6 7 8 and = ;

0 0 1 0 9 0 1 2

0 0 0 1 3 4 5 6

1 0 0 0 1 2 3 4 1 2 3 4

0 1 0 0 5 6 7 8 5 6 7 8

0 0 1 0 9 0 1 2 9 0 1 2

0 0 0 1 3 4 5 6 3 4 5 6

A

IA

MatricesMatrices

A vector whose all elements are equal to 1. It is denoted by 1

Addition-Operator Vector

30 15

25 101

and 28 121

32 14

22 13

30 15 45

25 10 351

28 12 401

32 14 46

22 13 35

X 1

X1

1

1

1

1

1

MatricesMatrices

If the norm is divided by the degrees of freedom (n-1), then the standard deviation (if the mean is zero) is obtained.

The Norm-Operation of a vector

2 2By Pythagoras, 6 1 37 6.08276 x

6

1

T 6In vector notation, 6 1 37

1

x x x

6

1

x

MatricesMatrices

Is a function that associates a scalar, det(A), to every n×n square matrix A. This can be interpreted as the volume of the matrix. In 2D, the area of the parallelogram

The Determinant of Matrix

4 3

2 5

A

MatricesMatrices

Is a function depending on n that associates a scalar, det(A), to every n×n square matrix A. S4 = the area of the rectangle

The Determinant of Matrix

4 3

2 5

A

T1

T1

S3

T2

S3S1 T1 T1

S2 T2 T2

Area S4 2T1 2T2 2S3

Area S4 S1 S2 2S3

Area (4 2)(5 3) 4 3

2 5 2 (2 3) 14

T2

MatricesMatrices

Is a function depending on n that associates a scalar, det(A), to every n×n square matrix A. S4 = the area of the rectangle

The Determinant of Matrix

a b

c d

A

T1

T1

S3

T2

S3S1 T1 T1

S2 T2 T2

Area S4 2T1 2T2 2S3

Area S4 S1 S2 2S3

Area (a+c)(b+d)-ab-cd-2bc

Area ab+ad+bc+cd-ab-cd-2bc

Area ad-bc

det( ) 4 5 2 3 14 A

4 3

2 5

AT2

MatricesMatrices

In analogy with the exponential notation for reciprocal of a number (1/a=a -1), the inverse of matrix is denoted A-1. If a matrix is square, then A-1A=AA-1=I. For a 22 matrix, the operation is

The Inverse of Matrix

1

;

1 1

det( )

a b

c d

d b d b

c a c aad bc

A

AA

1

1

1 4;

3 2

2 41 1

3 12 12 10

0.2 0.4

0.3 0.1

d b

c a

A

A

A1

1

1 4 0.2 0.4 1 0

3 2 0.3 0.1 0 1

0.2 0.4 1 4 1 0

0.3 0.1 3 2 0 1

AA

A A

Linear Algebra and StatisticsLinear Algebra and Statistics

A vector is normalized if its length is equal to one. Normalizing a vector = data standardization.

The Normalization of Vector

3; 25 5

4

3

0.64

0.85

x x

z

T 1

xz

x

z z

Linear Algebra and StatisticsLinear Algebra and Statistics

If two variables (u and v) have the same score, then the two vectors are superposed on each other. However, as the two variables differs from one another, the angle between them will increase.

Relation between two vectors

Linear Algebra and StatisticsLinear Algebra and Statistics

The greater the angle between the two vectors, the lesser they share in common. If the angle reach 90° then there are no common part.

Relation between two vectors

Linear Algebra and StatisticsLinear Algebra and Statistics

The cosine of that angle is the correlation coefficient. If the angle is null (or 180°) then the cosine is 1 (or -1); indicating a perfect

relation. If the relation is 90° (or 270 °) then the cosine is 0; indication an absence of relation.

Relation between two vectors

T1 cov

cos

n

i ii r

s s

uv

uvu v

u vu v

u v u v

Linear Algebra and StatisticsLinear Algebra and StatisticsRelation between two vectors

T cosu v u v

6 2 and

1 5

u v

T 17u v

37 and 29 u v

T 17cos 0.52

37 29r uv

u v

u v

T1 cov

cos

n

i ii r

s s

uv

uvu v

u vu v

u v u v

Linear Algebra and StatisticsLinear Algebra and StatisticsThe Mean

T / nx 1 X30

25

, ?28

32

22

X x

1

1

30 25 28 32 22 / 51

1

1

27.4

x

x

From the previously defined addition-operation, the mean is straightforward. Let us say that we have one variable with 5 participants.

Linear Algebra and StatisticsLinear Algebra and StatisticsThe Mean (several variables)

30 15

25 10

, [27.4 12.8]28 12

32 14

22 13

M x

Let us say that we have two variables with 5 participants.

Let us define a 52 mean-score matrix as

27.4 12.8

27.4 12.8

27.4 12.8

27.4 12.8

27.4 12.8

X

Linear Algebra and StatisticsLinear Algebra and Statistics

Example of statistical Import

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

X M X

Deviation Score Matrix

30 15 27.4 12.8

25 10 27.4 12.8

, 28 12 27.4 12.8

32 14 27.4 12.8

22 13 27.4 12.8

30 15 27.4 12.8 2

25 10 27.4 12.8

28 12 27.4 12.8

32 14 27.4 12.8

22 13 27.4 12.8

M X

X M X

.6 2.2

2.4 2.8

0.6 0.8

4.6 1.2

5.4 0.2

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

T , where is a deviation score matrixSSCP X X X

Sums of Square and Cross Product (SSCP)

T

2.6 2.2

2.4 2.8

0.6 0.8

4.6 1.2

5.4 0.2

2.6 2.2

2.4 2.82.6 2.4 0.6 4.6 5.4 63.2 16.4

0.6 0.82.2 2.8 0.8 1.2 0.2 16.4 14.8

4.6 1.2

5.4 0.2

X

SSCP X X

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Sums of Square and Cross Product (SSCP) Note: the SSCP is square and symmetric If the SSCP is divided by the number of degrees of freedom, then we get the information about variance and covariance for all the data.

63.2 16.4

16.4 14.8

SSCP

63.2 16.4 15.8 4.1/(5 1)

16.4 14.8 4.1 3.7

VARCOV

1s

2s

1s 2s

Variance for the first variable

Variance for the second variable

Covariance

21 1 1( )s s s

22 2 2( )s s s

1 2 2 1 12( )s s s s Cov

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Simple regression How can we find the weights that describe (optimally) the following function ?

0 1v̂ b b u

The solution is to find the shadow of v on u that has the shortest distance

The shortest distance is the one that crosses at 90° the vector u

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Simple regression Therefore, the error e can be defined as:

1b e v u

Where b1 is the value that multiply u that makes the shadow of v the shortest (90°)

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Simple regression As a consequence, the angle between u and e will also be 90° Therefore, the correlation (and covariance) between u and e will be zero.

T 0u e

By substitution, we can isolate the b1 coefficient.

T

T1

T T1

T T1

T 1 T T 1 T1

T 1 T1 1

0

( ) 0

0

( ) ( ) ( ) ( )

( ) ( ) 1

b

b

b

b

b b

u e

u v u

u v u u

u v u u

u u u v u u u u

u u u v

This is the least mean squared method

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Simple regression With 2 variables this is identical to the solution given in textbooks.

-1 -1T T 21 2

covcovb s

s uv

u uvu

u u u v 2.6 2.2

2.4 2.8

0.6 0.8

4.6 1.2

5.4 0.2

X

u v

Deviation score matrix

-1

11

2.6 2.2

2.4 2.8

2.6 2.4 0.6 4.6 5.4 2.6 2.4 0.6 4.6 5.4 63.2 16.4 0.260.6 0.8

4.6 1.2

5.4 0.2

b

Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import

Simple regression The constant b0 is obtained the usual way

0 1

0 1

v b b u

b v b u

0

27.4 and 12.8

12.8 0.26 27.4 5.69

u v

b

Therefore, the final regression equation is

ˆ 5.69 0.26v u