Upload File

asynchronous methods for deep reinforcement learning · asynchronous methods for deep reinforcement...

  • Home
  • Documents
  • Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
25
Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu Presented by: Pihel Saatmann

Upload: others

Post on 21-May-2020

31 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous Methods for Deep Reinforcement Learning PaperbyVolodymyrMnih,AdriàPuigdomènechBadia,MehdiMirza,

AlexGraves,TimothyP.Lillicrap,TimHarley,DavidSilver,KorayKavukcuoglu

Presentedby:PihelSaatmann

Page 2: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Reinforcement learning

•  State–„snapshot“oftheenvironment

• Ac'on–leadstonewstate,someNmesreward

• Reward–Nmedelayed,sparse

• Policy–rulesforchoosingacNon

Page 3: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

So far

•  ThoughtthatonlineRLalgorithmswithdeepNN-sareunstable.• Problems-correlatedandnon-staNonaryinputdata.

•  Tocountertheseproblemsdatacanbestoredinexperiencereplaymemory.•  Thisusesmorememory/computaNonalpower.

• DeepRLmethodsrequirespecializedhardware(GPUs)ormassivedistributedarchitectures.

Page 4: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Q-learning

• AteachNmestept,theagentreceivesastatestandselectsanacNonaaccordingtoitspolicyπ.Thentheagentgetsthenextstatest+1andascalarrewardrt.•  Thegoalistomaximizetheexpectedreturnfromeachstatest.• QfuncNonesimatestheacNon’svalue.•  EachNmetheagentdoesanacNontheQvalueisupdated.• Off-policymethod–updaNngQfndoesnotdependonpolicy.

Page 5: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous RL framework

•  InsteadofexperiencereplaytheyasynchronouslyexecutemulNpleagentsinparallelonmulNpleinstancesoftheenvironment.

• Parallelactor-learnershaveastabilizingeffectontraining.• RunsonasinglemachinewithastandardmulN-coreCPU.

Page 6: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous RL framework II

• AsyncvariantsoffourstandardRLalgorithms:•  1-stepQ-learning•  N-stepQ-learning•  1-stepSarsa•  Advantageactor-criNc(A3C)

Page 7: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

1-step Q-learning

• NNisusedtoapproximatetheQ(s,a;Θ)funcNon.•  Theparameters(weights)ΘarelearnedbyiteraNvelyminimizingasequenceoflossfuncNons,wherethei-thlossfuncNonisdefinedas:

Page 8: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Async 1-step Q-leaning

•  Eachthreadhasowncopyofenvironment.• AteachstepcomputesagradientoftheQ-learningloss.• AccumulategradientsovermulNpleNmestepsbeforeapplying.•  Sharedandslowlychangingtargetnetwork.

Page 9: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous 1-step Sarsa

•  Sameas1-stepQ-learning,butusesadifferenttargetvalue:

Page 10: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous n-step Q-learning

• PotenNallyfasterwaytopropagaterewards.• Uses‘forward-view’-selectsacNonsusingitspolicyforuptonstepsinthefuture.• Receivesuptotmaxrewardssincelastupdate.•  Totalaccumulatedreturn:• ValuefnisupdatedadereverytmaxacNonsoraderterminalstate.•  Foreachupdateusesthelongestpossiblen-stepreturn.

Page 11: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
Page 12: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Asynchronous advantage actor-criBc

• On-policymethod-hasapolicyandesNmatedvaluefuncNon.• Uses‘forward-view’.• Receivesuptotmaxrewardssincelastupdate.• Policyandvaluefn-sareupdatedadereverytmaxacNonsoraderterminalstate.•  Foreachupdateusesthelongestpossiblen-stepreturn.

Page 13: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
Page 14: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Performance evaluaBon

•  Fourdifferentplaforms:•  Atari2600-differentgames•  TORCS3D-carracingsimulator• MuJoCo-physicssimulatorforconNnuousmotorcontrol(A3Conly)•  Labyrinth-findingrewardsinrandomlygenerated3Dmazes(A3Conly)

Page 15: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Atari 2600 games

• AllfourmethodscansuccessfullytrainNNcontrollers.• AsyncmethodsmostlyfasterthanDQN(DeepQ-Network).• Advantageactor-criNcwasthebest.

Page 16: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Async A3C on 57 atari games

Page 17: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

TORCS Car Racing Simulator

•  EvaluatedonlytheA3Calgorithm.• Agenthadtodrivearacecarusingonlyrawpixelsasinput.• Duringtraining,theagentwasrewardedformaintaininghighvelocityalongthecenteroftheracetrack.

hgps://youtu.be/0xo1Ldx3L5Q

Page 18: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
Page 19: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

MuJoCo Physics Simulator

•  EvaluatedonlytheA3Calgorithm.• Rigidbodyphysicswithcontactdynamics.• ConNnuousacNons.•  InallproblemsA3CfoundgoodsoluNonsinlessthan24hoursoftraining(typicallyafewhours).

hgps://youtu.be/0xo1Ldx3L5Q

Page 20: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Labyrinth

•  Theagentwasplacedinrandommazeandhad60stocollectpoints.• Apples–1point• Portals–10points,respawnedapplesandagentinrandomlocaNons• Visualinputonly.

•  Theagentlearnedaresonablygoodgeneralstrategyforexploringrandommazes.

hgps://youtu.be/nMR5mjCFZCw

Page 21: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Scalability

•  Theframeworkscaleswellwiththenumberofparallelworkers.•  Evenshowssuperlinearspeedupsforsomemethods.

Page 22: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
Page 23: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,
Page 24: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

Robustness and stability

•  Trainedmodelsonfivegamesusing50differentlearningratesandrandominiNalizaNon.•  EachgameandalgorithmcombinaNonhadarangeoflearningratesforwhichallrandominiNalizaNonsachievedgoodscores.•  Stabilityindicatedbyvirtuallyno0scoresinregionswithgoodlearningrates.

Page 25: Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning Paper by Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza,

To summarize

• Asynchronousmethodsforfourstandardreinforcementlearningalgorithms(1-stepQ,n-stepQ,1-stepSARSA,A3C).• Abletotrainneuralnetworkcontrollersonavarietyofdomainsinstablemanner.• Usingparallelactorlearnerstoupdateasharedmodelstabilizedthelearningprocess(alternaNivetoexperiencereplay).•  InAtarigamestheadvantageactor-criNc(A3C)surpassedthecurrentstate-of-the-artinhalfthetrainingNme.•  Superlinearspeedupwhenincreasingthreadcountfor1-stepmethods.


Asynchronous Counters

ASYNCHRONOUS COUNTER

Asynchronous Assertions

Safe and efficient off-policy reinforcement learning · [Mnih et al., 2016] Asynchronous Methods for Deep Reinforcement Learning [Munos, Stepleton, Harutyunyan, Bellemare, 2016] Safe

LEARNET: Reinforcement Learning Based Flow Scheduling ......LEARNET: Reinforcement Learning Based Flow Scheduling for Asynchronous Deterministic Networks Jonathan Prados-Garzon∗,

AT07896: Universal Asynchronous Receiver Transceiver (UART ...ww1.microchip.com/...Universal-Asynchronous-Receiver-Transceiver-UART... · AT07896: Universal Asynchronous Receiver

Asynchronous Methods for Deep Reinforcement …proceedings.mlr.press/v48/mniha16.pdf?spm=a2c4e.11153940...Asynchronous Methods for Deep Reinforcement Learning One way of propagating

Asynchronous LBS

A Level-Crossing Flash Asynchronous Analog-to-Digital ... Level... · Asynchronous ADC In CAD Mentor Graphics Asynchronous Data Processing System ASYNCHRONOUS PARALLEL RESISTORLESS

Jordi Puigdomènech - Genealogía y esperanza en la Filosofía de la existencia de Ingmar Bergman

Inverse Reinforcement Learning CS885 Reinforcement

Ajax · Asynchronous JavaScript + XML” Asynchronous JavaScript And XML HTML JSON. Asynchronous JavaScript And XML DHTML DOM Scripting. Asynchronous “Not at the same time” JavaScript

Asynchronous Methods for Deep Reinforcement Learning · forcement learning tasks. In one example, (Koutník et al., 2014) evolved convolutional neural network controllers for the

Asynchronous Consensus

Asynchronous communication

Asynchronous Circuits

Asynchronous Design - University of · PDF fileJ. Frenzel Asynchronous Design 3 What is “Asynchronous? ... J. Frenzel Asynchronous Design 48 Sun FIFO • Focused on building FIFO

Reinforcement Learning - Multi-Agent Reinforcement

Asynchronous Methods for Deep Reinforcement Learning · Distributed version of DQN (Then) State-of-the-art results on the Atari Training takes days on 100+ machines Lavrenti Frobeen

Asynchronous Deep Q Learning for Breakoutcs229.stanford.edu/proj2016/poster/BonillaZengZhen...Asynchronous methods for deep reinforcement learning.arXiv preprint arXiv:1602.01783

Asynchronous Machine

Asynchronous interface

Asynchronous FIFO

Asynchronous Components

Asynchronous Methods for Deep Reinforcement Learning · Asynchronous Methods for Deep Reinforcement Learning One way of propagating rewards faster is by using n-step returns (Watkins,1989;Peng

Asynchronous Chip

Asynchronous Learning

Asynchronous Wrapper for Globally Asynchronous Locally Synchronous

Deep Reinforcement Learning for Robotic Manipulation … · Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates Shixiang Gu;1;2;3 and Ethan Holly;1

Asynchronous USB

Asynchronous Methods for Deep Reinforcement Learningproceedings.mlr.press/v48/mniha16.pdf · Asynchronous Methods for Deep Reinforcement Learning ... Q-learning, which aims to

for high REINFORCEMENT SYSTEMS - nevoga.com€¦ · 2 REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEM PLEXUS®, PYRAPLEX®, FTW The reinforcement system PLEXUS®,

Asynchronous Presentation

Asynchronous Pipelines

Asynchronous Services