10 back propagation - virginia techcourses.cs.vt.edu/cs4824/spring19/slide_pdfs/10 back... ·...
TRANSCRIPT
Back Propagation
Machine Learning CS4824/ECE4424
Bert Huang Virginia Tech
Outline
• Logistic regression and perceptron as neural networks
• Likelihood gradient for 2-layered neural network
• General recipe for back propagation
Back Propagation• Back propagation:
• Compute hidden unit activations: forward propagation
• Compute gradient at output layer: error
• Propagate error back one layer at a time
• Chain rule via dynamic programming
Plan
• Start by computing gradients by hand (we’ll get messy!)
• See common patterns for organizing back-propagation algorithm
• Show general matrix form for feed-forward neural networks
Logistic Squashing Function�(x) =
1
1 + exp(�x)
Logistic Squashing Function�(x) =
1
1 + exp(�x)
d �(x)
d x= �(x)(1� �(x))
Multi-Layered Perceptron
x1 x2 x3 x4 x5
h1 h2
yh = [h1, h2]>
h1 = �(w>11x) h2 = �(w>
12x)
p(y |x) = �(w>21h)
p(y |x) = �⇣w>21
⇥�(w>
11x),�(w>12x)
⇤>⌘
Gradients
p(y |x) = �(w>21h)
p(y |x) = �⇣w>21
⇥�(w>
11x),�(w>12x)
⇤>⌘
ll(W ) =nX
i=1
log p(yi |xi )
rw21 ll =nX
i=1
1
p(yi |xi )⇥rw21p(yi |xi )
rw21 ll =nX
i=1
1
p(yi |xi )⇥rw21�(yiw
>21h)
<latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="d382k1x0XKL9NDe1EPY+v+UjH+0=">AAACyXicfZLPahsxEMblbdImTtom516WhELIwez2khwLvfSYQpwYbBNmtbNeYf1ZpNmkRmxvpZBrnqiP0bepvNmCHScdEPr4fhokzUxWSeEoSf70oldb26/f7Oz29/b7b9+9P9i/cqa2HIfcSGNHGTiUQuOQBEkcVRZBZRKvs/mXJb++ReuE0Ze0qHCqYKZFIThQsC5uDo6TQdJGvCnSThyzLm4Oe78nueG1Qk1cgnPjNKlo6sGS4BKb/qR2WAGfwwzHQWpQ6Ka+fWcTfwxOHhfGhqUpbt3VDA/KuYXKwkkFVLqnbGm+xKhUz6FxTcX51Atd1YSaP76hqGVMJl7WI86FRU5yEQRwK8I3Yl6CBU6hamuXuKK9f83LjJkTZC64Gu+oRGNR+W5v/GUnVunpP3y6wtsD3CgFOvcTsDMldOMnpkILZOyyjHeCSimUIOc73mxmwff/ZwXehJ6nTzu8Ka4+DdJkkH5L2A77wI7YCUvZGfvMvrILNmSc5eyePUSj6Ef083E2ol43JIdsLaJffwEsRusy</latexit><latexit sha1_base64="GqY3BA4xSL/yK/jVspsvJ7/Pteo=">AAADQnicfZLPb9MwFMed8GNbGdBx5WIxIW07VMkucEFC2oXjkNZtUtNFjus0Vv0jsl9oK+M/jr+AM/wf3BDCTYPUrsCTIn/1/bwX2++5qAW3kCTfovjBw0eP9/YPek8Onz573j86vLa6MZQNqRba3BbEMsEVGwIHwW5rw4gsBLspZhcrfvOJGcu1uoJlzcaSTBUvOSUQrLwPmSKFILmb5+489R5nwBZgpBPC43c4s43MHQ8q9XcKZ6Uh1KXe1SfLnOPPeJHz01UNl8zinV9ZPpWkzVxbdxnoGlenef84GSRt4F2RduIYdXGZH0VfsommjWQKqCDWjtKkhrEjBjgVzPeyxrKa0BmZslGQioTzjF3bHo9fB2eCS23CpwC37maFI9LapSxCpiRQ2ftsZf6LQSX/hkYNlG/Hjqu6Aabo+gxlIzBovBoDnnDDKIhlEIQaHq6BaUVCdyEMa2sTW7b7b3mF1jMghQ2uYnOomDZMum717qoTm/TsDz7b4G0C1VISNXEZMVPJlXeZrpkhoM2qjXMOleCSg3Ud97tVZPH/qsB9GHp6f8S74vp8kCaD9GOC9tFL9AqdoBS9Qe/RB3SJhoii7xGKDqJe/DX+Ef9cP4846t7JC7QV8a/fbqMXjQ==</latexit><latexit sha1_base64="GqY3BA4xSL/yK/jVspsvJ7/Pteo=">AAADQnicfZLPb9MwFMed8GNbGdBx5WIxIW07VMkucEFC2oXjkNZtUtNFjus0Vv0jsl9oK+M/jr+AM/wf3BDCTYPUrsCTIn/1/bwX2++5qAW3kCTfovjBw0eP9/YPek8Onz573j86vLa6MZQNqRba3BbEMsEVGwIHwW5rw4gsBLspZhcrfvOJGcu1uoJlzcaSTBUvOSUQrLwPmSKFILmb5+489R5nwBZgpBPC43c4s43MHQ8q9XcKZ6Uh1KXe1SfLnOPPeJHz01UNl8zinV9ZPpWkzVxbdxnoGlenef84GSRt4F2RduIYdXGZH0VfsommjWQKqCDWjtKkhrEjBjgVzPeyxrKa0BmZslGQioTzjF3bHo9fB2eCS23CpwC37maFI9LapSxCpiRQ2ftsZf6LQSX/hkYNlG/Hjqu6Aabo+gxlIzBovBoDnnDDKIhlEIQaHq6BaUVCdyEMa2sTW7b7b3mF1jMghQ2uYnOomDZMum717qoTm/TsDz7b4G0C1VISNXEZMVPJlXeZrpkhoM2qjXMOleCSg3Ud97tVZPH/qsB9GHp6f8S74vp8kCaD9GOC9tFL9AqdoBS9Qe/RB3SJhoii7xGKDqJe/DX+Ef9cP4846t7JC7QV8a/fbqMXjQ==</latexit><latexit sha1_base64="+Uhh8qtOs/p7rMc5hu9KqTxSPVs=">AAADTXicfVHLbtNAFB27QNvwSmHJZkSE1HYR2d3ABqkSG8SqSE1bKU6t8WQcjzIPa+aaJBr8cXwBa/gPdggxdoyUNMCVRnN0zj3zuCcrBbcQRd+CcO/e/Qf7B4e9h48eP3naP3p2ZXVlKBtRLbS5yYhlgis2Ag6C3ZSGEZkJdp3N3zX69SdmLNfqElYlm0gyUzznlICn0j4kimSCpG6RurO4rnECbAlGOiFq/BYntpKp4x7F9a3CSW4IdXHtyuNVyvFnvEz5SePhklm8c5TlM0nazjV1m4AucXGS9gfRMGoL74K4AwPU1UV6FHxJpppWkimgglg7jqMSJo4Y4FSwupdUlpWEzsmMjT1UxL9n4trx1PiVZ6Y418YvBbhlNx2OSGtXMvOdkkBh72oN+S8NCvk3aVxB/mbiuCorYIqu35BXAoPGTQx4yg2jIFYeEGq4/wamBfHTBR/W1iU2b+/f4jKt50Ay61nFFlAwbZh03V67yw5sqqd/5NMNvW2gWkqipi4hZia5ql2iS2YIaNOMccGhEFxysK7T610XWf7f5fXahx7fjXgXXJ0N42gYf4wG5x+6+A/QC/QSHaMYvUbn6D26QCNE0fcABYdBL/wa/gh/hr/WrWHQeZ6jrdrb/w2WxhhL</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit><latexit sha1_base64="zJMIAcw5yWP1qWLYqeGgmGgG678=">AAADTXicfVHLahsxFNVM2iZxX0677EbUFJIszEwItJtAoJvSVQpxEvA4g0bWeIT1GKQ7tY0yH9cv6Lr9j+5KqTyegh23vSB0OOcePe7JSsEtRNG3INx58PDR7t5+5/GTp8+edw9eXFldGcoGVAttbjJimeCKDYCDYDelYURmgl1n0/dL/fozM5ZrdQmLko0kmSiec0rAU2kXEkUyQVI3S91JXNc4ATYHI50QNT7Dia1k6rhHcX2rcJIbQl1cu/JwkXJ8h+cpP1p6uGQWbx1l+USSpnNF3SagS1wcpd1e1I+awtsgbkEPtXWRHgRfkrGmlWQKqCDWDuOohJEjBjgVrO4klWUloVMyYUMPFfHvGblmPDV+45kxzrXxSwFu2HWHI9Lahcx8pyRQ2PvakvyXBoX8mzSsIH83clyVFTBFV2/IK4FB42UMeMwNoyAWHhBquP8GpgXx0wUf1sYlNm/u3+AyradAMutZxWZQMG2YdO1eu8sWrKvHf+TjNb1poFpKosYuIWYiuapdoktmCGizHOOMQyG45GBdq9fbLjL/v8vrtQ89vh/xNrg66cdRP/502jv/2Ma/h16h1+gQxegtOkcf0AUaIIq+ByjYDzrh1/BH+DP8tWoNg9bzEm3Uzu5vmAYYTw==</latexit>
rw21 ll =nX
i=1
�(yiw>21h)(1� �(yiw>
21h))
�(yiw>21h)
⇥rw21yiw>21h
<latexit sha1_base64="8xh92dQirtp6Yzf3CViXl2ixHOc=">AAADinicfVFdaxNBFJ3t+lFr1VQffPBlMAhpwbAbBBUpFBQVnyo0bSGbLrOT2eyQ+Vhm7pqGYX+UP8Gf4b9xkqyQD+OFYQ7n3MO9MycrBbcQRb+DvfDO3Xv39x8cPDx89PhJ6+jppdWVoaxPtdDmOiOWCa5YHzgIdl0aRmQm2FU2+TjXr34wY7lWFzAr2VCSseI5pwQ8lbZ+JopkgqRumrpeXNc4AXYLRjohanyKE1vJ1HGP4vpG4SQ3hLrE8rEknVnK8dJ1k4AucXGMOzF+jXfKx/Vu63wwl8zizX22WtNWO+pGi8LbIG5AGzV1nh4Fv5KRppVkCqgg1g7iqIShIwY4Faw+SCrLSkInZMwGHiri9xi6xd/W+JVnRjjXxh8FeMGuOhyR1s5k5jslgcJuanNylwaF/Jc0qCB/N3RclRUwRZc75JXAoPE8QzzihlEQMw8INdw/A9OC+GjAJ702xOaL+WtcpvUESGY9q9gUCqYNk665a3fRgFX15K98sqIvGqiWkqiRS4gZS658wrpkhoA282+ccigElxysa/R620Vu/+/yeu1Djzcj3gaXvW4cdePvb9pn35r499EL9BJ1UIzeojP0FZ2jPqLB8+A0+Bx8CQ/DXvg+/LBs3QsazzO0VuGnP2KxKoc=</latexit><latexit sha1_base64="8xh92dQirtp6Yzf3CViXl2ixHOc=">AAADinicfVFdaxNBFJ3t+lFr1VQffPBlMAhpwbAbBBUpFBQVnyo0bSGbLrOT2eyQ+Vhm7pqGYX+UP8Gf4b9xkqyQD+OFYQ7n3MO9MycrBbcQRb+DvfDO3Xv39x8cPDx89PhJ6+jppdWVoaxPtdDmOiOWCa5YHzgIdl0aRmQm2FU2+TjXr34wY7lWFzAr2VCSseI5pwQ8lbZ+JopkgqRumrpeXNc4AXYLRjohanyKE1vJ1HGP4vpG4SQ3hLrE8rEknVnK8dJ1k4AucXGMOzF+jXfKx/Vu63wwl8zizX22WtNWO+pGi8LbIG5AGzV1nh4Fv5KRppVkCqgg1g7iqIShIwY4Faw+SCrLSkInZMwGHiri9xi6xd/W+JVnRjjXxh8FeMGuOhyR1s5k5jslgcJuanNylwaF/Jc0qCB/N3RclRUwRZc75JXAoPE8QzzihlEQMw8INdw/A9OC+GjAJ702xOaL+WtcpvUESGY9q9gUCqYNk665a3fRgFX15K98sqIvGqiWkqiRS4gZS658wrpkhoA282+ccigElxysa/R620Vu/+/yeu1Djzcj3gaXvW4cdePvb9pn35r499EL9BJ1UIzeojP0FZ2jPqLB8+A0+Bx8CQ/DXvg+/LBs3QsazzO0VuGnP2KxKoc=</latexit><latexit sha1_base64="8xh92dQirtp6Yzf3CViXl2ixHOc=">AAADinicfVFdaxNBFJ3t+lFr1VQffPBlMAhpwbAbBBUpFBQVnyo0bSGbLrOT2eyQ+Vhm7pqGYX+UP8Gf4b9xkqyQD+OFYQ7n3MO9MycrBbcQRb+DvfDO3Xv39x8cPDx89PhJ6+jppdWVoaxPtdDmOiOWCa5YHzgIdl0aRmQm2FU2+TjXr34wY7lWFzAr2VCSseI5pwQ8lbZ+JopkgqRumrpeXNc4AXYLRjohanyKE1vJ1HGP4vpG4SQ3hLrE8rEknVnK8dJ1k4AucXGMOzF+jXfKx/Vu63wwl8zizX22WtNWO+pGi8LbIG5AGzV1nh4Fv5KRppVkCqgg1g7iqIShIwY4Faw+SCrLSkInZMwGHiri9xi6xd/W+JVnRjjXxh8FeMGuOhyR1s5k5jslgcJuanNylwaF/Jc0qCB/N3RclRUwRZc75JXAoPE8QzzihlEQMw8INdw/A9OC+GjAJ702xOaL+WtcpvUESGY9q9gUCqYNk665a3fRgFX15K98sqIvGqiWkqiRS4gZS658wrpkhoA282+ccigElxysa/R620Vu/+/yeu1Djzcj3gaXvW4cdePvb9pn35r499EL9BJ1UIzeojP0FZ2jPqLB8+A0+Bx8CQ/DXvg+/LBs3QsazzO0VuGnP2KxKoc=</latexit><latexit sha1_base64="8xh92dQirtp6Yzf3CViXl2ixHOc=">AAADinicfVFdaxNBFJ3t+lFr1VQffPBlMAhpwbAbBBUpFBQVnyo0bSGbLrOT2eyQ+Vhm7pqGYX+UP8Gf4b9xkqyQD+OFYQ7n3MO9MycrBbcQRb+DvfDO3Xv39x8cPDx89PhJ6+jppdWVoaxPtdDmOiOWCa5YHzgIdl0aRmQm2FU2+TjXr34wY7lWFzAr2VCSseI5pwQ8lbZ+JopkgqRumrpeXNc4AXYLRjohanyKE1vJ1HGP4vpG4SQ3hLrE8rEknVnK8dJ1k4AucXGMOzF+jXfKx/Vu63wwl8zizX22WtNWO+pGi8LbIG5AGzV1nh4Fv5KRppVkCqgg1g7iqIShIwY4Faw+SCrLSkInZMwGHiri9xi6xd/W+JVnRjjXxh8FeMGuOhyR1s5k5jslgcJuanNylwaF/Jc0qCB/N3RclRUwRZc75JXAoPE8QzzihlEQMw8INdw/A9OC+GjAJ702xOaL+WtcpvUESGY9q9gUCqYNk665a3fRgFX15K98sqIvGqiWkqiRS4gZS658wrpkhoA282+ccigElxysa/R620Vu/+/yeu1Djzcj3gaXvW4cdePvb9pn35r499EL9BJ1UIzeojP0FZ2jPqLB8+A0+Bx8CQ/DXvg+/LBs3QsazzO0VuGnP2KxKoc=</latexit>
rw21 ll =nX
i=1
(1� �(yiw>21h))⇥rw21yiw
>21h
<latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit>
Gradients
p(y |x) = �(w>21h)
p(y |x) = �⇣w>21
⇥�(w>
11x),�(w>12x)
⇤>⌘
ll(W ) =nX
i=1
log p(yi |xi )
rw21 ll =nX
i=1
(1� �(yiw>21h))⇥rw21yiw
>21h
<latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit><latexit sha1_base64="UJwfPVkCKQ075oFUZTeEU6mNG/c=">AAADTnicfVFbixMxGM3MeqldL1199CVYhO6CZWYR9EVY8EV8WmG7u9DpDpk00wnNZUi+sVvC/Dj/ga/i//BNNJ2O0Iv6QcjhnO/kS3KyUnALUfQtCA/u3L13v/Oge/jw0eMnvaOnl1ZXhrIR1UKb64xYJrhiI+Ag2HVpGJGZYFfZ/P1Kv/rMjOVaXcCyZBNJZornnBLwVNqrEkUyQVK3SN1pXNc4AXYLRjohavwOJ7aSqeMexfWNwoMYv/Icn0kyWKYcr003CegSF8fH3swls3j3zL3WtNePhlFTeB/ELeijts7To+BLMtW0kkwBFcTacRyVMHHEAKeC1d2ksqwkdE5mbOyhIv4eE9f8T41femaKc238UoAbdtPhiLR2KTPfKQkUdldbkf/SoJB/k8YV5G8njquyAqbo+g55JTBovMoBT7lhFMTSA0IN98/AtCCGUPBpbQ2xeTN/i8u0ngPJrGcVW0DBtGHStXvtLlqwqZ78kU829KaBaimJmrqEmJnkqnaJLpkhoM3qGxccCsElB+tavd53kdv/u7xe+9Dj3Yj3weXpMI6G8afX/bOPbfwd9By9QAMUozfoDH1A52iEKPoeBEE3OAy/hj/Cn+GvdWsYtJ5naKsOOr8BEekXWA==</latexit>
rw21 ll =nX
i=1
(1� �(yiw>21h))yih
<latexit sha1_base64="AJJBNyAVpS31tUKexW27KiWuER0=">AAADK3icfZFdaxNBFIYn61etH0310pvBIKQFy24R9EYoeCNeVWnaQjZdzk4m2SHzscycNQ3D/iTvvfU3eKWId6I/w0myQtKoB4Z5eZ/zMh8nL6VwGMdfWtG16zdu3tq6vX3n7r37O+3dB6fOVJbxHjPS2PMcHJdC8x4KlPy8tBxULvlZPnk152fvuXXC6BOclXygYKzFSDDAYGXtd6mGXELmp5k/TOqapsgv0SovZU1f0tRVKvMiqKS+0LSb0KfBE2MF3Vkm6DJ0kaIpabG3R+dekbU78UG8KLopkkZ0SFPH2W7rUzo0rFJcI5PgXD+JSxx4sCiY5PV2WjleApvAmPeD1KC4G/jF42v6JDhDOjI2LI104a4mPCjnZioPnQqwcFfZ3PwXw0L9DfUrHL0YeKHLCrlmyzuMKknR0Pkn06GwnKGcBQHMivAMygqwwDCMYu0QN1qcv+blxkwQchdczadYcGO58s1e+5NGrNL9P3h/hS8amFEK9NCnYMdK6NqnpuQW0Nj5N04FFlIogc43vN5MweX/U4HXYejJ1RFvitPDgyQ+SN4+6xy9aca/RR6Rx6RLEvKcHJHX5Jj0CCMfyXfyk/yKPkSfo6/Rt2Vr1GoyD8laRT9+A6v/DPg=</latexit><latexit sha1_base64="AJJBNyAVpS31tUKexW27KiWuER0=">AAADK3icfZFdaxNBFIYn61etH0310pvBIKQFy24R9EYoeCNeVWnaQjZdzk4m2SHzscycNQ3D/iTvvfU3eKWId6I/w0myQtKoB4Z5eZ/zMh8nL6VwGMdfWtG16zdu3tq6vX3n7r37O+3dB6fOVJbxHjPS2PMcHJdC8x4KlPy8tBxULvlZPnk152fvuXXC6BOclXygYKzFSDDAYGXtd6mGXELmp5k/TOqapsgv0SovZU1f0tRVKvMiqKS+0LSb0KfBE2MF3Vkm6DJ0kaIpabG3R+dekbU78UG8KLopkkZ0SFPH2W7rUzo0rFJcI5PgXD+JSxx4sCiY5PV2WjleApvAmPeD1KC4G/jF42v6JDhDOjI2LI104a4mPCjnZioPnQqwcFfZ3PwXw0L9DfUrHL0YeKHLCrlmyzuMKknR0Pkn06GwnKGcBQHMivAMygqwwDCMYu0QN1qcv+blxkwQchdczadYcGO58s1e+5NGrNL9P3h/hS8amFEK9NCnYMdK6NqnpuQW0Nj5N04FFlIogc43vN5MweX/U4HXYejJ1RFvitPDgyQ+SN4+6xy9aca/RR6Rx6RLEvKcHJHX5Jj0CCMfyXfyk/yKPkSfo6/Rt2Vr1GoyD8laRT9+A6v/DPg=</latexit><latexit sha1_base64="AJJBNyAVpS31tUKexW27KiWuER0=">AAADK3icfZFdaxNBFIYn61etH0310pvBIKQFy24R9EYoeCNeVWnaQjZdzk4m2SHzscycNQ3D/iTvvfU3eKWId6I/w0myQtKoB4Z5eZ/zMh8nL6VwGMdfWtG16zdu3tq6vX3n7r37O+3dB6fOVJbxHjPS2PMcHJdC8x4KlPy8tBxULvlZPnk152fvuXXC6BOclXygYKzFSDDAYGXtd6mGXELmp5k/TOqapsgv0SovZU1f0tRVKvMiqKS+0LSb0KfBE2MF3Vkm6DJ0kaIpabG3R+dekbU78UG8KLopkkZ0SFPH2W7rUzo0rFJcI5PgXD+JSxx4sCiY5PV2WjleApvAmPeD1KC4G/jF42v6JDhDOjI2LI104a4mPCjnZioPnQqwcFfZ3PwXw0L9DfUrHL0YeKHLCrlmyzuMKknR0Pkn06GwnKGcBQHMivAMygqwwDCMYu0QN1qcv+blxkwQchdczadYcGO58s1e+5NGrNL9P3h/hS8amFEK9NCnYMdK6NqnpuQW0Nj5N04FFlIogc43vN5MweX/U4HXYejJ1RFvitPDgyQ+SN4+6xy9aca/RR6Rx6RLEvKcHJHX5Jj0CCMfyXfyk/yKPkSfo6/Rt2Vr1GoyD8laRT9+A6v/DPg=</latexit><latexit sha1_base64="AJJBNyAVpS31tUKexW27KiWuER0=">AAADK3icfZFdaxNBFIYn61etH0310pvBIKQFy24R9EYoeCNeVWnaQjZdzk4m2SHzscycNQ3D/iTvvfU3eKWId6I/w0myQtKoB4Z5eZ/zMh8nL6VwGMdfWtG16zdu3tq6vX3n7r37O+3dB6fOVJbxHjPS2PMcHJdC8x4KlPy8tBxULvlZPnk152fvuXXC6BOclXygYKzFSDDAYGXtd6mGXELmp5k/TOqapsgv0SovZU1f0tRVKvMiqKS+0LSb0KfBE2MF3Vkm6DJ0kaIpabG3R+dekbU78UG8KLopkkZ0SFPH2W7rUzo0rFJcI5PgXD+JSxx4sCiY5PV2WjleApvAmPeD1KC4G/jF42v6JDhDOjI2LI104a4mPCjnZioPnQqwcFfZ3PwXw0L9DfUrHL0YeKHLCrlmyzuMKknR0Pkn06GwnKGcBQHMivAMygqwwDCMYu0QN1qcv+blxkwQchdczadYcGO58s1e+5NGrNL9P3h/hS8amFEK9NCnYMdK6NqnpuQW0Nj5N04FFlIogc43vN5MweX/U4HXYejJ1RFvitPDgyQ+SN4+6xy9aca/RR6Rx6RLEvKcHJHX5Jj0CCMfyXfyk/yKPkSfo6/Rt2Vr1GoyD8laRT9+A6v/DPg=</latexit>
rw21 ll =nX
i=1
((1� �(w>
21h))h if yi = +1
(�1 + �(�w>21h))h if yi = �1
<latexit sha1_base64="0ToUJ8ZIsppPHC5k2yVTTCG3HAU=">AAADlHicjVFdaxNBFN3t+lHjV6rgiy+jQUlbEnaLoA8KhSKKD5JC0xYy6TI7mWSHzMcyM2sahvllPvkz/DfOJhtJGhUvLHM45565O/dkBaPaxPHPcCe6dfvO3d17jfsPHj563Nx7cq5lqTDpY8mkusyQJowK0jfUMHJZKIJ4xshFNj2p9ItvRGkqxZmZF2TI0UTQMcXIeCptfocCZQyldpbao8Q5AA25Nopbxhz4AKAueWqpR4m7EgBmZEKFxX6gdo12Ajq+g044ai/tV9DIAuT7+yAHr39fBegYODBPq2sOEwgb7U4CDlfOzv9aO5UVEjFazU+brbgbLwpsg6QGraCuXroX/oAjiUtOhMEMaT1I4sIMLVKGYkZcA5aaFAhP0YQMPBSIEz20iy078MozIzCWyn/CgAW77rCIaz3nme/kyOT6plaRf9NMzv8kDUozfje0VBSlIQIv/2FcMmAkqNIEI6oINmzuAcKK+mcAnCOFsPGZbwzR48X8DS6TcmpQ5lcJBZmZnEhFuK1PZ89qsK4erOSDNX3RgCXnyIcDkZpwKpyFsiAKGamqNc6oyRnl1Ghb627bha7/7fK686EnNyPeBudH3STuJqdvWsdf6vh3g+fBy6AdJMHb4Dj4HPSCfoDDF+GnsBeeRs+i99FJ9HHZuhPWnqfBRkVffwELQymW</latexit><latexit sha1_base64="0ToUJ8ZIsppPHC5k2yVTTCG3HAU=">AAADlHicjVFdaxNBFN3t+lHjV6rgiy+jQUlbEnaLoA8KhSKKD5JC0xYy6TI7mWSHzMcyM2sahvllPvkz/DfOJhtJGhUvLHM45565O/dkBaPaxPHPcCe6dfvO3d17jfsPHj563Nx7cq5lqTDpY8mkusyQJowK0jfUMHJZKIJ4xshFNj2p9ItvRGkqxZmZF2TI0UTQMcXIeCptfocCZQyldpbao8Q5AA25Nopbxhz4AKAueWqpR4m7EgBmZEKFxX6gdo12Ajq+g044ai/tV9DIAuT7+yAHr39fBegYODBPq2sOEwgb7U4CDlfOzv9aO5UVEjFazU+brbgbLwpsg6QGraCuXroX/oAjiUtOhMEMaT1I4sIMLVKGYkZcA5aaFAhP0YQMPBSIEz20iy078MozIzCWyn/CgAW77rCIaz3nme/kyOT6plaRf9NMzv8kDUozfje0VBSlIQIv/2FcMmAkqNIEI6oINmzuAcKK+mcAnCOFsPGZbwzR48X8DS6TcmpQ5lcJBZmZnEhFuK1PZ89qsK4erOSDNX3RgCXnyIcDkZpwKpyFsiAKGamqNc6oyRnl1Ghb627bha7/7fK686EnNyPeBudH3STuJqdvWsdf6vh3g+fBy6AdJMHb4Dj4HPSCfoDDF+GnsBeeRs+i99FJ9HHZuhPWnqfBRkVffwELQymW</latexit><latexit sha1_base64="0ToUJ8ZIsppPHC5k2yVTTCG3HAU=">AAADlHicjVFdaxNBFN3t+lHjV6rgiy+jQUlbEnaLoA8KhSKKD5JC0xYy6TI7mWSHzMcyM2sahvllPvkz/DfOJhtJGhUvLHM45565O/dkBaPaxPHPcCe6dfvO3d17jfsPHj563Nx7cq5lqTDpY8mkusyQJowK0jfUMHJZKIJ4xshFNj2p9ItvRGkqxZmZF2TI0UTQMcXIeCptfocCZQyldpbao8Q5AA25Nopbxhz4AKAueWqpR4m7EgBmZEKFxX6gdo12Ajq+g044ai/tV9DIAuT7+yAHr39fBegYODBPq2sOEwgb7U4CDlfOzv9aO5UVEjFazU+brbgbLwpsg6QGraCuXroX/oAjiUtOhMEMaT1I4sIMLVKGYkZcA5aaFAhP0YQMPBSIEz20iy078MozIzCWyn/CgAW77rCIaz3nme/kyOT6plaRf9NMzv8kDUozfje0VBSlIQIv/2FcMmAkqNIEI6oINmzuAcKK+mcAnCOFsPGZbwzR48X8DS6TcmpQ5lcJBZmZnEhFuK1PZ89qsK4erOSDNX3RgCXnyIcDkZpwKpyFsiAKGamqNc6oyRnl1Ghb627bha7/7fK686EnNyPeBudH3STuJqdvWsdf6vh3g+fBy6AdJMHb4Dj4HPSCfoDDF+GnsBeeRs+i99FJ9HHZuhPWnqfBRkVffwELQymW</latexit><latexit sha1_base64="0ToUJ8ZIsppPHC5k2yVTTCG3HAU=">AAADlHicjVFdaxNBFN3t+lHjV6rgiy+jQUlbEnaLoA8KhSKKD5JC0xYy6TI7mWSHzMcyM2sahvllPvkz/DfOJhtJGhUvLHM45565O/dkBaPaxPHPcCe6dfvO3d17jfsPHj563Nx7cq5lqTDpY8mkusyQJowK0jfUMHJZKIJ4xshFNj2p9ItvRGkqxZmZF2TI0UTQMcXIeCptfocCZQyldpbao8Q5AA25Nopbxhz4AKAueWqpR4m7EgBmZEKFxX6gdo12Ajq+g044ai/tV9DIAuT7+yAHr39fBegYODBPq2sOEwgb7U4CDlfOzv9aO5UVEjFazU+brbgbLwpsg6QGraCuXroX/oAjiUtOhMEMaT1I4sIMLVKGYkZcA5aaFAhP0YQMPBSIEz20iy078MozIzCWyn/CgAW77rCIaz3nme/kyOT6plaRf9NMzv8kDUozfje0VBSlIQIv/2FcMmAkqNIEI6oINmzuAcKK+mcAnCOFsPGZbwzR48X8DS6TcmpQ5lcJBZmZnEhFuK1PZ89qsK4erOSDNX3RgCXnyIcDkZpwKpyFsiAKGamqNc6oyRnl1Ghb627bha7/7fK686EnNyPeBudH3STuJqdvWsdf6vh3g+fBy6AdJMHb4Dj4HPSCfoDDF+GnsBeeRs+i99FJ9HHZuhPWnqfBRkVffwELQymW</latexit>
rw21 ll =nX
i=1
I (yi = 1)� �(w>21h))h
<latexit sha1_base64="NKQkoniprFREJa8afESpDRZps1Y=">AAADLHicfZFNixMxGMfT8W2tb109egkWoV2wzCyCexEWvKinFbe7C53ukEkzndC8DMkz2y1hvpJ3j34HLyLiTfRbmE5HaLfqAyH/+f+eP5nkSQvBLYThl1Zw7fqNm7d2brfv3L13/0Fn9+GJ1aWhbEi10OYsJZYJrtgQOAh2VhhGZCrYaTp7teSnF8xYrtUxLAo2lmSqeMYpAW8lnfexIqkgiZsnbj+qKhwDuwQjnRAVfoljW8rEca+i6lzhN71FUn/08TPP+FSS3ip4HoMucN7v47yddLrhIKwLb4uoEV3U1FGy2/oUTzQtJVNABbF2FIUFjB0xwKlgVTsuLSsInZEpG3mpiGR27OrbV/ipdyY408YvBbh21xOOSGsXMvWdkkBur7Kl+S8GufwbGpWQHYwdV0UJTNHVP2SlwKDx8pXxhBtGQSy8INRwfw1Mc2IIBT+LjUNsVp+/4aVaz4Ck1ruKzSFn2jDpmr1yx41Yp3t/8N4arxuolpKoiYuJmUquKhfrghkC2iyfcc4hF1xysK7h1XaKXP4/5Xnlhx5dHfG2ONkfROEgeve8e/i2Gf8OeoyeoB6K0At0iF6jIzREFH1E39FP9Cv4EHwOvgbfVq1Bq8k8QhsV/PgNlS8Meg==</latexit><latexit sha1_base64="NKQkoniprFREJa8afESpDRZps1Y=">AAADLHicfZFNixMxGMfT8W2tb109egkWoV2wzCyCexEWvKinFbe7C53ukEkzndC8DMkz2y1hvpJ3j34HLyLiTfRbmE5HaLfqAyH/+f+eP5nkSQvBLYThl1Zw7fqNm7d2brfv3L13/0Fn9+GJ1aWhbEi10OYsJZYJrtgQOAh2VhhGZCrYaTp7teSnF8xYrtUxLAo2lmSqeMYpAW8lnfexIqkgiZsnbj+qKhwDuwQjnRAVfoljW8rEca+i6lzhN71FUn/08TPP+FSS3ip4HoMucN7v47yddLrhIKwLb4uoEV3U1FGy2/oUTzQtJVNABbF2FIUFjB0xwKlgVTsuLSsInZEpG3mpiGR27OrbV/ipdyY408YvBbh21xOOSGsXMvWdkkBur7Kl+S8GufwbGpWQHYwdV0UJTNHVP2SlwKDx8pXxhBtGQSy8INRwfw1Mc2IIBT+LjUNsVp+/4aVaz4Ck1ruKzSFn2jDpmr1yx41Yp3t/8N4arxuolpKoiYuJmUquKhfrghkC2iyfcc4hF1xysK7h1XaKXP4/5Xnlhx5dHfG2ONkfROEgeve8e/i2Gf8OeoyeoB6K0At0iF6jIzREFH1E39FP9Cv4EHwOvgbfVq1Bq8k8QhsV/PgNlS8Meg==</latexit><latexit sha1_base64="NKQkoniprFREJa8afESpDRZps1Y=">AAADLHicfZFNixMxGMfT8W2tb109egkWoV2wzCyCexEWvKinFbe7C53ukEkzndC8DMkz2y1hvpJ3j34HLyLiTfRbmE5HaLfqAyH/+f+eP5nkSQvBLYThl1Zw7fqNm7d2brfv3L13/0Fn9+GJ1aWhbEi10OYsJZYJrtgQOAh2VhhGZCrYaTp7teSnF8xYrtUxLAo2lmSqeMYpAW8lnfexIqkgiZsnbj+qKhwDuwQjnRAVfoljW8rEca+i6lzhN71FUn/08TPP+FSS3ip4HoMucN7v47yddLrhIKwLb4uoEV3U1FGy2/oUTzQtJVNABbF2FIUFjB0xwKlgVTsuLSsInZEpG3mpiGR27OrbV/ipdyY408YvBbh21xOOSGsXMvWdkkBur7Kl+S8GufwbGpWQHYwdV0UJTNHVP2SlwKDx8pXxhBtGQSy8INRwfw1Mc2IIBT+LjUNsVp+/4aVaz4Ck1ruKzSFn2jDpmr1yx41Yp3t/8N4arxuolpKoiYuJmUquKhfrghkC2iyfcc4hF1xysK7h1XaKXP4/5Xnlhx5dHfG2ONkfROEgeve8e/i2Gf8OeoyeoB6K0At0iF6jIzREFH1E39FP9Cv4EHwOvgbfVq1Bq8k8QhsV/PgNlS8Meg==</latexit><latexit sha1_base64="NKQkoniprFREJa8afESpDRZps1Y=">AAADLHicfZFNixMxGMfT8W2tb109egkWoV2wzCyCexEWvKinFbe7C53ukEkzndC8DMkz2y1hvpJ3j34HLyLiTfRbmE5HaLfqAyH/+f+eP5nkSQvBLYThl1Zw7fqNm7d2brfv3L13/0Fn9+GJ1aWhbEi10OYsJZYJrtgQOAh2VhhGZCrYaTp7teSnF8xYrtUxLAo2lmSqeMYpAW8lnfexIqkgiZsnbj+qKhwDuwQjnRAVfoljW8rEca+i6lzhN71FUn/08TPP+FSS3ip4HoMucN7v47yddLrhIKwLb4uoEV3U1FGy2/oUTzQtJVNABbF2FIUFjB0xwKlgVTsuLSsInZEpG3mpiGR27OrbV/ipdyY408YvBbh21xOOSGsXMvWdkkBur7Kl+S8GufwbGpWQHYwdV0UJTNHVP2SlwKDx8pXxhBtGQSy8INRwfw1Mc2IIBT+LjUNsVp+/4aVaz4Ck1ruKzSFn2jDpmr1yx41Yp3t/8N4arxuolpKoiYuJmUquKhfrghkC2iyfcc4hF1xysK7h1XaKXP4/5Xnlhx5dHfG2ONkfROEgeve8e/i2Gf8OeoyeoB6K0At0iF6jIzREFH1E39FP9Cv4EHwOvgbfVq1Bq8k8QhsV/PgNlS8Meg==</latexit>
Gradients
p(y |x) = �(w>21h)
p(y |x) = �⇣w>21
⇥�(w>
11x),�(w>12x)
⇤>⌘
ll(W ) =nX
i=1
log p(yi |xi )
rw11 ll =nX
i=1
1
p(yi |xi )⇥rw11p(yi |xi )
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))rw11w
>21h
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w
>21(rw11h)
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w21[1]rw11�(w
>11xi )
Gradients
p(y |x) = �(w>21h)
p(y |x) = �⇣w>21
⇥�(w>
11x),�(w>12x)
⇤>⌘
ll(W ) =nX
i=1
log p(yi |xi )
rw11 ll =nX
i=1
1
p(yi |xi )⇥rw11p(yi |xi )
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))rw11w
>21h
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w
>21(rw11h)
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w21[1]rw11�(w
>11xi )
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w21[1]�(w
>11xi )(1� �(w>
11xi ))xi
ll(W ) =nX
i=1
log p(yi |xi )=nX
i=1
log �⇣w>21
⇥�(w>
11xi ),�(w>12xi )
⇤>⌘
w>21h h1 = �(w>
11x)
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w21[1]�(w
>11xi )(1� �(w>
11xi ))xi
log �(yiw>21h)
<latexit sha1_base64="QpvEonLPYBUEr9XH5G59yQiGLg4=">AAAC93icfZHNitRAEMd74te6fuysHgVpHIR1D0OyCHpc8CKeVtjZXZiModLTSZrpj9BdcQwhN9/Dm4g30RfwMXwbe7IRZnbUgqb//H9VVHdVWkrhMAx/DYJr12/cvLVze/fO3Xv394b7D86cqSzjE2aksRcpOC6F5hMUKPlFaTmoVPLzdPFqxc/fc+uE0adYl3ymINciEwzQW8nwcSxNTmMncgUHdSLoMmmOovZdjKakxbNkOArHYRd0W0S9GJE+TpL9wc94bliluEYmwblpFJY4a8CiYJK3u3HleAlsATmfeqlBcTdruo+09Kl35jQz1h+NtHPXKxpQztUq9ZkKsHBX2cr8F8NC/Q1NK8xezhqhywq5ZpdvyCpJ0dDVwOhcWM5Q1l4As8J/g7ICLDD0Y91o4rKu/4aXGrNASJ13NV9iwY3lqunvtjntxTo9/IMP13iXwIxSoOdNDDZXQrdNbEpuAY1djXEpsJBCCXRNz9vtKvjw/yrPW7/06OqKt8XZ0TgKx9Hb56PjN/36d8gj8oQckIi8IMfkNTkhE8LIR/KVfCc/gjr4FHwOvlymBoO+5iHZiODbbwcR+W0=</latexit><latexit sha1_base64="QpvEonLPYBUEr9XH5G59yQiGLg4=">AAAC93icfZHNitRAEMd74te6fuysHgVpHIR1D0OyCHpc8CKeVtjZXZiModLTSZrpj9BdcQwhN9/Dm4g30RfwMXwbe7IRZnbUgqb//H9VVHdVWkrhMAx/DYJr12/cvLVze/fO3Xv394b7D86cqSzjE2aksRcpOC6F5hMUKPlFaTmoVPLzdPFqxc/fc+uE0adYl3ymINciEwzQW8nwcSxNTmMncgUHdSLoMmmOovZdjKakxbNkOArHYRd0W0S9GJE+TpL9wc94bliluEYmwblpFJY4a8CiYJK3u3HleAlsATmfeqlBcTdruo+09Kl35jQz1h+NtHPXKxpQztUq9ZkKsHBX2cr8F8NC/Q1NK8xezhqhywq5ZpdvyCpJ0dDVwOhcWM5Q1l4As8J/g7ICLDD0Y91o4rKu/4aXGrNASJ13NV9iwY3lqunvtjntxTo9/IMP13iXwIxSoOdNDDZXQrdNbEpuAY1djXEpsJBCCXRNz9vtKvjw/yrPW7/06OqKt8XZ0TgKx9Hb56PjN/36d8gj8oQckIi8IMfkNTkhE8LIR/KVfCc/gjr4FHwOvlymBoO+5iHZiODbbwcR+W0=</latexit><latexit sha1_base64="QpvEonLPYBUEr9XH5G59yQiGLg4=">AAAC93icfZHNitRAEMd74te6fuysHgVpHIR1D0OyCHpc8CKeVtjZXZiModLTSZrpj9BdcQwhN9/Dm4g30RfwMXwbe7IRZnbUgqb//H9VVHdVWkrhMAx/DYJr12/cvLVze/fO3Xv394b7D86cqSzjE2aksRcpOC6F5hMUKPlFaTmoVPLzdPFqxc/fc+uE0adYl3ymINciEwzQW8nwcSxNTmMncgUHdSLoMmmOovZdjKakxbNkOArHYRd0W0S9GJE+TpL9wc94bliluEYmwblpFJY4a8CiYJK3u3HleAlsATmfeqlBcTdruo+09Kl35jQz1h+NtHPXKxpQztUq9ZkKsHBX2cr8F8NC/Q1NK8xezhqhywq5ZpdvyCpJ0dDVwOhcWM5Q1l4As8J/g7ICLDD0Y91o4rKu/4aXGrNASJ13NV9iwY3lqunvtjntxTo9/IMP13iXwIxSoOdNDDZXQrdNbEpuAY1djXEpsJBCCXRNz9vtKvjw/yrPW7/06OqKt8XZ0TgKx9Hb56PjN/36d8gj8oQckIi8IMfkNTkhE8LIR/KVfCc/gjr4FHwOvlymBoO+5iHZiODbbwcR+W0=</latexit><latexit sha1_base64="QpvEonLPYBUEr9XH5G59yQiGLg4=">AAAC93icfZHNitRAEMd74te6fuysHgVpHIR1D0OyCHpc8CKeVtjZXZiModLTSZrpj9BdcQwhN9/Dm4g30RfwMXwbe7IRZnbUgqb//H9VVHdVWkrhMAx/DYJr12/cvLVze/fO3Xv394b7D86cqSzjE2aksRcpOC6F5hMUKPlFaTmoVPLzdPFqxc/fc+uE0adYl3ymINciEwzQW8nwcSxNTmMncgUHdSLoMmmOovZdjKakxbNkOArHYRd0W0S9GJE+TpL9wc94bliluEYmwblpFJY4a8CiYJK3u3HleAlsATmfeqlBcTdruo+09Kl35jQz1h+NtHPXKxpQztUq9ZkKsHBX2cr8F8NC/Q1NK8xezhqhywq5ZpdvyCpJ0dDVwOhcWM5Q1l4As8J/g7ICLDD0Y91o4rKu/4aXGrNASJ13NV9iwY3lqunvtjntxTo9/IMP13iXwIxSoOdNDDZXQrdNbEpuAY1djXEpsJBCCXRNz9vtKvjw/yrPW7/06OqKt8XZ0TgKx9Hb56PjN/36d8gj8oQckIi8IMfkNTkhE8LIR/KVfCc/gjr4FHwOvlymBoO+5iHZiODbbwcR+W0=</latexit>
w>21h h1 = �(w>
11x)log �(w>21h)
rw11 ll =nX
i=1
(I (yi = 1)� �(w>21h))w21[1]�(w
>11xi )(1� �(w>
11xi ))xi
x1 x2 x3 x4 x5
h1 h2
y
raw errorblame for error
gradient of blamed error
Matrix Formx
h1 = s(W1x)
…
Matrix Formx
h1 = s(W1x)
w11
h1[1]
w12
w13 w14 w15
w11 w12 w13 w14 w15
W1 =
s(v) = [s(v1), s(v2), s(v3), ...]>
Matrix Formx
h1 = s(W1x)
w11 w12 w13 w14 w15
w21 w22 w23 w24 w25
w31 w32 w33 w34 w35
W1 =
s(v) = [s(v1), s(v2), s(v3), ...]>
# of output units
# of input units
Matrix Formx
h1 = s(W1x)
…
hm-1 = s(Wm-1 hm-2)
f(x, W) = s(Wm hm-1)
…
h2 = s(W2 h1)
J(W ) = `(f(x,W ))
Matrix Gradient Recipeh1 = s(W1x)
…
hm-1 = s(Wm-1 hm-2)
f(x, W) = s(Wm hm-1)
h2 = s(W2 h1)
J(W ) = `(f(x,W ))
�m = `0(f(x,W ))rWmJ = �mh>
m�1
rWm�1J = �m�1h>m�2
rWiJ = �ih>i�1
rW1J = �1x>
�m�1 = (W>m�m)� s0(Wm�1hm�2)
�i = (W>i+1�i+1)� s0(Wihi�1)
Matrix Gradient Recipe
h1 = s(W1x)
f(x, W) = s(Wm hm-1)
hi = s(Wi hi-1)
J(W ) = `(f(x,W ))
�m = `0(f(x,W )) rWiJ = �ih>i�1
rW1J = �1x>
Feed Forward Propagation Back Propagation
�i = (W>i+1�i+1)� s0(Wihi�1)
Challenges
• Local minima (non-convex)
• Overfitting
Remedies
• Regularization
• Parameter sharing: convolution
• Pre-training: initializing weights smartly
• Training data manipulation, e.g., dropout, noise, transformations
• Huge data sets