back-propagation primer
TRANSCRIPT
APrimeronBack-PropagationofErrors (appliedtoneuralnetworks)
AuroTripathy
Outline
• SummaryofForward-Propagation• TheCalculusofBack-propagation• Summary
2
AuroTripathy
AFeed-ForwardNetworkisaBrain-InspiredMetaphor
3
AuroTripathy
Feed-forwardtocalculatetheerrorrelativetothedesiredoutput
Error-Function(akaLoss-,Cost-,orObjective-Function)
• Inthefeed-forwardpath,calculatetheerrorrelativetothedesiredoutput• Wedefineaerror-functionE(X3,Y)asthe“penalty”ofpredictingX3whenthetrueoutputisY.• Theobjectiveistominimizetheerroracrossallthetrainingsamples.• Theerror/lossE(X3,Y)assignsanumericalscore(ascalar)forthenetwork’soutputX3given
theexpectedoutputY.• Thelossiszeroonlyforcaseswheretheneuralnetwork’soutputiscorrect.
4
AuroTripathy
SigmoidActivationFunction
Thesigmoidactivationfunction
σ(x) = 1/(1 + e−x)
isanS-shapedactivationfunctiontransformingallvaluesofxintherange,[0,1]
5https://en.wikipedia.org/wiki/File:Logistic-curve.svg
AuroTripathy
GradientDescent
6
Note,inpractice,wedon’texpectaglobalminima,asshownhere
ab
AuroTripathy
“Unshackledbythechain-rule” -PatrickWinston,MIT
7
AuroTripathy
DerivativeoftheErrorEwith-respect-totheOutput,X3
8
AuroTripathy
DerivativeoftheSigmoidActivationFunction
9
P3 X3
FortheSigmoidfunction,thecoolthingis,thederivativeoftheoutput,X3(withrespecttotheinput,P3)isexpressedintermsoftheoutput,i.e.,
X3.(1-X3)
http://kawahara.ca/wp-content/uploads/derivative_of_sigmoid.jpg
AuroTripathy
DerivativeofP3with-respect-toW3
10
AuroTripathy
Propagatetheerrorsbackwardandadjusttheweights,w,sotheactualoutputmimicsthedesiredoutput
11
AuroTripathy
ComputationsareLocalized&PartiallyPre-computedinthePreviousLayer
12
AuroTripathy
Summary
☑Ifthere’sarepresentativesetofinputsandoutputs,thenback-propagationcanlearnthetheweights.
☑Back-propagationhaslinearperformancerelativetothenumberoflayers.
☑Simpletoimplement(andtest)
13
AuroTripathy
Credits
14
ConceptscrystalizedfromMITProfessorPatrickWinston’slecture,https://www.youtube.com/watch?v=q0pm3BrIUFo