Upload File

reinforcement learning with dnns: alphago to alphazerocraven/cs760/lectures/alphazero.pdf · higher...

  • Home
  • Documents
  • Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible
28
Reinforcement Learning with DNNs: AlphaGo to AlphaZero CS 760: Machine Learning Spring 2018 Mark Craven and David Page www.biostat.wisc.edu/~craven/cs760 1

Upload: domien

Post on 25-Aug-2019

231 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

ReinforcementLearningwithDNNs:AlphaGo toAlphaZero

CS760:MachineLearningSpring2018

MarkCravenandDavidPage

www.biostat.wisc.edu/~craven/cs760

1

Page 2: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

GoalsfortheLecture

• Youshouldunderstandthefollowingconcepts:

• MonteCarlotreesearch(MCTS)• Self-play• Residualneuralnetworks• AlphaZero algorithm

2

Page 3: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

ABriefHistoryofGame-PlayingasaCS/AITestofProgress• 1944:AlanTuringandDonaldMichie simulatebyhandtheirchessalgorithmsduringlunchesatBletchleyPark

• 1959:ArthurSamuel’scheckersalgorithm(machinelearning)

• 1961:Michie’s MatchboxEducableNoughts AndCrossesEngine(MENACE)

• 1991:Computersolveschessendgamethoughtdraw:KRBbeatsKNN(223moves)

• 1992:TDGammon trainsforBackgammonbyself-playreinforcementlearning

• 1997:ComputersbestinworldatChess(DeepBluebeatsKasparov)

• 2007:Checkers“solved”bycomputer(guaranteedoptimalplay)

• 2016:ComputersbestatGo(AlphaGo beatsLeeSodol)

• 2017(4monthsago):AlphaZero extendsAlphaGo tobestatchess,shogi

Page 4: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

OnlySome oftheseinvolvedLearning• 1944:AlanTuringandDonaldMichie simulatebyhandtheirchessalgorithmsduringlunchesatBletchleyPark

• 1959:ArthurSamuel’scheckersalgorithm(machinelearning)

• 1961:Michie’s MatchboxEducableNoughts AndCrossesEngine(MENACE)

• 1991:Computersolveschessendgamethoughtdraw:KRBbeatsKNN(223moves)

• 1992:TDGammon trainsforBackgammonbyself-playreinforcementlearning

• 1997:ComputersbestinworldatChess(DeepBluebeatsKasparov)

• 2007:Checkers“solved”bycomputer(guaranteedoptimalplay)

• 2016:ComputersbestatGo(AlphaGo beatsLeeSodol)

• 2017(4monthsago):AlphaZero extendsAlphaGo tobestatchess,shogi

Page 5: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

OnlySome oftheseinvolvedLearning• 1944:AlanTuringandDonaldMichie simulatebyhandtheirchessalgorithmsduringlunchesatBletchleyPark

• 1959:ArthurSamuel’scheckersalgorithm(machinelearning)

• 1961:Michie’s MatchboxEducableNoughts AndCrossesEngine(MENACE)

• 1991:Computersolveschessendgamethoughtdraw:KRBbeatsKNN(223moves)

• 1992:TDGammon trainsforBackgammonbyself-playreinforcementlearning

• 1997:ComputersbestinworldatChess(DeepBluebeatsKasparov)

• 2007:Checkers“solved”bycomputer(guaranteedoptimalplay)

• 2016:ComputersbestatGo(AlphaGo beatsLeeSodol)

• 2017(4monthsago):AlphaZero extendsAlphaGo tobestatchess,shogi

Page 6: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

Background:GamePlaying

• Untillastyear,state-of-the-artformanygamesincludingchesswasminimax searchwithalpha-betapruning(recallIntrotoAI)

• Mosttop-performinggame-playingprogramsdidn’tdolearning

• GameofGowasoneofthefewgameswherehumansstilloutperformedcomputers

Page 7: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

MinimaxinaPicture(thanksWikipedia)

Page 8: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

MonteCarloTreeSearch(MCTS)inaPicture(thanksWikipedia)

Rollout(RandomSearch)

Page 9: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

ReinforcementLearningbyAlphaGo,AlphaGo Zero,andAlphaZero:KeyInsights

• MCTSwithSelf-Play• Don’thavetoguesswhatopponentmightdo,so…• Ifnoexploration,abig-branchinggametreebecomesonepath• Yougetanautomaticallyimproving,evenly-matchedopponentwhoisaccuratelylearningyourstrategy

• “Wehavemettheenemy,andheisus”(famousvariantofPogo,1954)• Noneedforhumanexpertscoringrulesforboardsfromunfinishedgames

• Treatboardasanimage:useresidualconvolutionalneuralnetwork

• AlphaGo Zero:Onedeepneuralnetworklearnsboththevaluefunctionandpolicyinparallel

• AlphaZero:RemovedrolloutaltogetherfromMCTSandjustusedcurrentneuralnetestimatesinstead

Page 10: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero (Dec2017):MinimizedRequiredGameKnowledge,ExtendedfromGotoChessandShogi

Page 11: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero’s versionofQ-Learning

• Nodiscountonfuturerewards

• Rewardsof0untilendofgame;thenrewardof-1or+1

• ThereforeQ-valueforanactiona orpolicyp fromastateS isexactlyvaluefunction:Q(S,p) =V(S,p)

• AlphaZero usesoneDNN(detailsinabit)tomodelbothp andV

• UpdatestoDNNaremade(trainingexamplesprovided)aftergame

• Duringgame,needtobalanceexploitationandexploration

Page 12: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero Algorithm

InitializeDNN!"RepeatForever

PlayGame Update"

PlayGame:

RepeatUntilWinorLose: FromcurrentstateS,performMCTSEstimatemoveprobabilities#byMCTSRecord(S,#)asanexampleRandomlydrawnextmovefrom#

Update ": Letzbepreviousgameoutcome(+1or-1)Samplefromlastgame’sexamples(S,#, &)TrainDNN!"onsampletogetnew"

Page 13: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero Play-Game

Page 14: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero TrainDNN

Page 15: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZeroMonteCarloTreeSearch(MTCS)

Page 16: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

WhyNeedMCTSAtAll?

• CouldalwaysmakemoveDNNsayshashighestQ:noexploration• CouldjustdrawmovefromDNN’spolicyoutput• PaperssayMCTSoutputprobabilityvectorp selectsstrongermovesthatjustdirectlyusingtheneuralnetwork’spolicyoutputitself(isthereapossiblelessonhereforself-drivingcarstoo??)• StillneedtodecidehowmanytimestorepeatMCTSsearch(game-specific)andhowtotradeoffexplorationandexploitationinMCTS…AlphaZero paperjustsayschoosemovewith“lowcount,highmoveprobability,andhighvalue”—AlphaGo papermorespecific:maximizeupperconfidencebound• Where𝝉 istemperature[1,2],andN𝝉(s,b)iscountoftimeactionbhasbeentakenfromstates,raisedtothepower1/𝝉,choose:

Page 17: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:InputNodesRepresentCurrentGameState,IncludinganyneededHistory

Page 18: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:OutputNodesRepresentPolicyandValueFunction

• Apolicyisaprobabilitydistributionoverallpossiblemovesfromastate,soneedunitstorepresentallpossiblemoves

• Chessismostcomplicatedtodescribemoves(thoughGoandShogihavehighernumbersofmovestoconsider),sohereisforChessmoves:• 8x8=64possiblestartingpositionsforamove• 56possibledestinationsforqueenmoves:8compassdirections{N,NE,E,SE,S,SW,W,NW}times7possiblemove-lengths

• Another17possibledestinationsforirregularmovessuchasknight• Somemovesimpossible,dependingontheparticularpieceataposition(e.g.,pawncan’tmakeallqueenmoves)andlocationofotherpieces(queencan’tmovethrough2otherpiecestoattackathird)

• Weightsforimpossiblemovesaresetto0andnotallowedtochange• Anotherlayertonormalizeresultsintoprobabilitydistribution

• Onedeepneuralnetworklearnsboththevaluefunctionandpolicyinparallel:oneadditionaloutputnodeforthevalue function,whichestimatestheexpectedoutcomeintherange[-1,1]forfollowingthecurrentpolicyfrompresent(input)state

Page 19: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

DeepNeuralNetworksTrick#9:ResNets(ResidualNetworks)

• Whatifyourneuralnetworkistoodeep?

• Intheory,that’snoproblem,givensufficientnodesandconnectivity:early(orlate)layerscanjustlearnidentityfunction(autoencoder)

• Inpracticedeepneuralnetworksfailtolearnidentitywhenneeded

• Asolution:makeidentityeasyoreventhedefault;havetoworkhardtoactuallylearnanon-zeroresidual(andhenceanon-identity)

Page 20: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

ResidualNetworkinaPicture(He,Zhang,Ren,Sun,2015):IdentitySkipConnection

Note:outputandinputdimensionalityneedtobethesame.

Whycalled“residual”?

Page 21: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

DeepResidualNetworks(ResNets):Startofa35-layerResNet (He,Zhang,Ren,Sun,2015)

DottedlinedenotesincreaseinDimension(2moresuchincreases)

Page 22: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

ABriefAside:LeakyReLUs

• RectifiersusedcouldbeReLU or”LeakyReLU”

• LeakyReLU addresses“dyingReLU”problem-–wheninputsumisbelowsomevalue,outputis0,sonogradientfortraining

• ReLU:f(x)=max(0,x)

• LeakyReLU:

• ReLU LeakyReLU

Page 23: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:HiddenUnitsArrangedinaResidualNetwork(aCNNwithResidualLayers)

PolicyHead

ConvBlock3x3,256,/1

ResBlock3x3,256,/1

ResBlock3x3,256,/1

... Repeatfor39ResBlocks

ValueHead

Page 24: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:ConvolutionBlock

Page 25: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:ResidualBlocks

Page 26: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:PolicyHead(forGo)

Page 27: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero DNNArchitecture:ValueHeadAlphaZero DNNArchitecture:ValueHead

Page 28: Reinforcement Learning with DNNs: AlphaGo to AlphaZerocraven/cs760/lectures/AlphaZero.pdf · higher numbers of moves to consider), so here is for Chess moves: •8 x 8 = 64 possible

AlphaZero ComparedtoRecentWorldChampions


MOVES I vs MOVES II PREGUNTA MOVES I MOVES II€¦ · PREGUNTA MOVES I MOVES II del 10 % a vehículos pesados propulsados por Autogás y Gas Natural. esta asignación un máximo del

Use of Laser Triangulation and Deep Neural Networks (DNNs) for … · 2020. 7. 7. · Use of Laser Triangulation and Deep Neural Networks (DNNs) for Railway Track Safety Inspections

[moves] - SátiradeCamilo · Desperation Moves BariBari Vulcan Punch Umanori Vulcan Punch Super Desperation Moves BariBari Vulcan Punch Hidden Super Desperation Moves Umanori Ultimate

Teaching DNNs to design fast fashion - arXivTeaching DNNs to design fast fashion KDD 2019 Workshop, August 2019, Anchorage, Alaska - USA Our system also aids design by allowing the

1 DORY: Automatic End-to-End Deployment of Real-World DNNs

Directing DNNs Attention for Facial Attribution Classification … · 2021. 4. 3. · Directing DNNs Attention for Facial Attribution Classification using Gradient-weighted Class

Improving the Affordability of Robustness Training for DNNs · 2020. 5. 30. · Improving the affordability of robustness training for DNNs Sidharth Gupta University of Illinois at

Machine Learning - University of Wisconsin–Madisonpages.cs.wisc.edu/~dpage/cs760/course-overview.pdf · 2017-01-19 · unsupervised learning, reinforcement learning, active learning,

[moves] - Neo-Arcadianeo-arcadia.com/neoencyclopedia/3_count_bout_moves.pdf · 3 Count Bout [moves] ... Standing Moves Hell Stabs Low Hell Stab tap Knife Attack ... Grapple Moves

Small is Big: Making DNNs faster & energy-efficient for

Smart Moves

"Making Moves"

PARALLEL TRAINING OF DNNS WITH NATURAL GRA P …3.2.1 LEARNING RATE SCHEDULE It was found in (Senior et al., 2013) that when training DNNs for speech recognition, an exponen-tially

Comparing brains and DNNs – methods and findings

NORMALIZATION TECHNIQUES IN TRAINING DNNS: …

Forced Moves #1 Forced Moves #4 · Level Up Your Sudoku: 15 Training Techniques Printable Puzzles by Tommy Howell pg. 1. Forced Moves #1. Forced Moves #2. Forced Moves #3. Forced

Regression - University of Wisconsin–Madisonpages.cs.wisc.edu/~dpage/cs760/Regression.pdf · • linear regression • RMSE, MAE, and R-square • logistic regression • convex

Neural Networks and Deep Learningpages.cs.wisc.edu/~dpage/cs760/ANNs.pdf · Neural Networks and Deep Learning dpage/cs760 1 . Goals for the lecture you should understand the following

DNNS Spring Meeting, May 2013

Lecture 7: Hardware Acceleration of DNNs

Seminario MoveS MoveS Seminar

Placing DNNs on Wafer-Scale AI Accelerator with Optimal ... · Placing DNNs on Wafer-Scale AI Accelerator with Optimal Kernel Sizing Bentian Jiang, Jingsong Chen, Jinwei Liu, Lixin

Data classes for which DNNs can overcome the curse of

EVO Custom Moves Sheet - EVO Moves

Bboy moves

Integration of Unsupervised and Supervised Criteria for DNNs Training

WHAT MOVES US FORWARD MOVES US ALL - HSS

14 graphical model - University of Wisconsin–Madisonpages.cs.wisc.edu/~jerryzhu/cs760/14_graphical_model.pdfBayesian network example Burglary Earthquake Alarm JohnCalls MaryCalls

MOVES Moves Me

Title DNNBrain for mapping DNNs and brains · 7/5/2020  · Title: DNNBrain: a unifying toolbox for mapping deep neural networks and brains Running head: DNNBrain for mapping DNNs

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs · Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs Timur Garipov 1;2 Pavel Izmailov 3 Dmitrii Podoprikhin

Towards Efficient On-Board Deployment of DNNs on Intelligent …cas.ee.ic.ac.uk/people/alk15/files/ISVLSI_2019_dnn... · 2019-09-23 · DNNs on Intelligent Autonomous Systems

Reinforcement Learning Approaches for Locomotion …pages.cs.wisc.edu/~tpejsa/files/CS760-PejsaAndrist.pdf · CS 760, Fall 2011 Reinforcement Learning Approaches for Locomotion Planning

PROGRAMA MOVES II COMUNITAT VALENCIANA 2020moves.ivace.es/images/Moves/Información Plan MOVES 2020.pdf · 2020. 6. 22. · PROGRAMA MOVES COMUNITAT VALENCIANA ... comunidad propietarios,

Winning Moves