Assignment 6

Question 1.1Odds ratio

Linear model

Objective function

Question 1.1Odds ratio

Linear model

Objective function

First step

replace withusing the fact that,

Second step

Apply “exp” on both sides to get rid of the log

Third stepSolve for and plug it into the objective function

Objective function

+ = 1

Starting from equation 1

Question 1.2 One-vs-all Logistic Regression

Question 1.2 One-vs-all Logistic Regression

use findMin with LogisticLoss instead(see assignment 4 for the LogisticLoss function)

Matrix WMatrix X Matrix y

=*

n samples, k classes, p features

dimensions = ? dimensions = ? dimensions = ?

Question 1.2 One-vs-all Logistic Regression

use findMin with LogisticLoss instead(see assignment 4 for the LogisticLoss function)

Matrix WMatrix X Matrix y

=*

n samples, k classes, p features

dimensions = n x p dimensions = p x k dimensions = n x k

Question 1.3 Softmax Loss and derivative● The softmax probability function is given as,

● Assume

● Convert y to binary form; i.e.

Question 1.3 Softmax Loss and derivative● The softmax probability function is given as,

● Assume

● Convert y to binary form; i.e.

● Therefore,

● C is the number of classes ● W j refers to column j of W● is the binary predicted target value for class ‘c’ of

sample ‘i’

Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,

Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,

Only one of these terms is non-zero for any training example ‘i’

Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,

(apply log)

(multiply by -1)

● The negative logarithm of the probability:

= ?

= ?

● The derivative of the negative log probability with respect to can be broken into two cases :

Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,

(apply log)

(multiply by -1)

● The negative logarithm of the probability:

= ?

= ?

● The derivative of the negative log probability with respect to can be broken into two cases :

is column c of row j of Wwhich corresponds to the coefficient offeature j of class c

Question 1.3 Softmax Loss and derivative● The softmax probability function is now formulated as,

(apply log)

(multiply by -1)

● The negative logarithm of the probability:

= ?

= ?

● The derivative of the negative log probability with respect to :

Hint: Use the indicator function to distinguishbetween the two cases

Question 1.4 - Softmax Classifier

Question 1.4 - Softmax Classifier

Use findMin instead with the softmax loss grad function

Question 1.4 - Softmax Classifier

Change the contents of the green box on the left using that of the green boxes on the right

Question 1.4 - Softmax Classifier

Change the contents of the green box on the left using that of the green boxes on the right

Question 1.5 - Cost of Multinomial Logistic Regression

Time complexity for processing one example = ?

Time complexity for predicting one example = ?

Question 1.5 - Cost of Multinomial Logistic Regression

Question 2 random walk

un

un

-1

un

1

1

1

2 3

4

56

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un

-1

un

1

1

1

2 3

4

56

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

v=

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un

-1

un

1

1

1

2 3

4

56

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

prob = 1/2

prob = 1/2v=

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un

-1

un

1

1

1

2 3

4

56

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

v=

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un -1

un

1

1

1

2 3

4

56

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

v=prob = 1/2

prob = 1/2

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un

-1

un

1

1

1

2 3

456

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

v=

yhat = -1 withprob = 1/(1+2) = 1/3

prob = 1/3

prob = 1/3

3 1

4 1

5 -1

matrix labelList

Question 2 random walk

un

un

-1

-1

1

1

i=1

2 3

456

1

2

3

4

5

6

1 2 3 4 5 6

0 1 0 0 0 1

1 0 1 0 0 0

0 1 0 0 0 0

0 0 0 0 1 0

0 0 0 1 0 1

1 0 0 0 1 0

matrix AAdjacency matrix

v=

yhat = -1 withprob = 1/(1+2) = 1/3

prob = 1/3

prob = 1/3

3 1

4 1

5 -1

matrix labelList