building a brain with raspberry pi and zulu embedded jvm

© Copyright Azul Systems 2017


@speakjava

Building A Brain WithRaspberry Pi And

Zulu Embedded JVM

Simon Ritter

Deputy CTO, Azul Systems

1

mailto:@azulsystems

mailto:@azulsystems

mailto:@azulsystems

http://www.azulsystems.com


Introduction


Inspiration

3


Human Brain Neocortex


Hierarchical Temporal Memory

Theoretical framework for how the neocortex works

– Models neurons and synapses and how they interact

– Also includes dendrites and precise synapses

This is different to basic neural network topologies

HTM simulated neuron far more complex

– More adaptable than AI neuron

– links decay over time if not used (temporal)

Potentially much better at learning

5


Introduction To Machine Learning

6


Learning For Computers

Machine Learning

– Giving “computers the ability to learn without being

explicitly programmed” - Arthur Samuel

– Requires training data to predict future values

7


Learning For Computers

Deep Learning

– “Neural networks with more than two layers”

– Not necessary to have training data

Deep Reinforcement Learning

– No training data

– Learn by experience

– Much like a human: trial and error

8


Markov Decision Process

Mathematical model of decision making

– Outcome partly random, partly controlled by decision

5-tuple

– S: State

– A: Action

– P: Probablility that action a leads to state s

– R: Reward for transitioning

– γ: Discount factor

Difference in importance between future and present rewards

9


Markov Decision Process

10

By waldoalvarez - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=59364518


Deep Q Network

Q: Quantity of a state-action combination

– Learning rate

– Discount factor

– Initial conditions

Used by Deep Mind

to learn to play Atari

games with no previous

experience


Project Hardware


Raspberry Pi

Fantastically successful development board

– Cheap, powerful, great connectivity

– Original boards $25, Pi Zero only $5!

Officially launched on Febuary 29th 2012

– First production run was 10,000 boards

– RS reported over 100,000 pre-orders in one day

Crashed servers

– Over 10 million boards have been sold (Sept, 2016)

13


Raspberry Pi Variations

14

Model A Model B Pi Zero

ARM v6

700 MHz

ARMv6/7/8

700/900/1200 MHz

ARM v6

1 GHz

1 core 1 or 4 core 1 core

256 Mb

512 Mb

512 Mb

1 Gb

512 Mb

1 x USB

No Network

2/4 USB

100 Mb Ethernet

WiFi/Bluetooth (v3)

1 micro USB

No ethernet

WiFi/Bluetooth (Zero W)


Pi Brain Architecture

Network

Client

Laptop

Raspberry Pi

Cluster(Emulating HTM)


Raspberry Pi Cluster (Mk 1)

16


Raspberry Pi Cluster (Mk 2)

17


Cluster Hat

18

USB GPIO

Raspberry Pi 3

Master

PORT 1

PORT 2

PORT 3

PORT 4

Pi Zero W

Slaves

GPIO 22

GPIO 23

GPIO 24

GPIO 25

Alert LED

GPIO 21


Java For Embedded

Applications


Why Java?

“Write once, run anywhere”

– Easy to move code between different platforms

Wide range of libraries and frameworks

– Takes a lot of hard work out of app development

Simple interfacing

– Pi4J and DIO libraries for Raspberry Pi

Simple concurrency

– Comprehensive multi-threading support

Native code integration for complex situations

– JNI20


Azul Zulu Java For Embedded

Built from OpenJDK code base

Passes all TCK tests

Ports for Intel, ARM, PowerPC

ARM v6, v7, v8

– Soft and hard float

– 64 bit close to release

– C2 and C1+C2 compiler enhancements

Drop in replacement for other JVMs

No licensing restrictions (FoU)

21


Machine Learning In Java


Deep Learning Basics

Neural network nodes

– Neuron takes weighted inputs that can change

Organised as layers

– Non-linear processing

– Different levels of problem abstraction

– Good analogy to HTM model

Unsupervised learning

– No initial data set

– Learn by experience

23


Neural Network Elements: Node

24

Σ λ

1

X1

X2

X3

W0

W1

W2

W3


Neural Network Elements: Layers

25

Output Layer

Hidden Layer

Input Layer


DeepLearning4J (DL4J)

Toolkit for building, training and deploying Neural Networks

– configure neural networks and build computation graphs

ND4J: N-dimensional arrays for Java

– Core technology for deep learning

DataVec

– Transform data into feature vectors

Arbiter

– Used to evaluate and tune ML models

– Hyperparameter optimisation functionality

26


RL4J Basics

Low-dimensional (state) and high-dimensional (pixels)

processing

– MDP class (Markov Decision Process)

– DQN class (Deep Q Network)

– Policy abstract class

Choose the next action given a state

ACpolicy (Actor Critic)

BoltzmannQ (Probablilty distribution)

DQNPolicy (Action with maximum Q value)

EpsGreedy (Epsilon Greedy, partially random)

28


Minecraft And Project Malmo


Minecraft

Immensly popular ‘construction’ game

– Especially amongst children

– Different modes: creative, survival, hardcore

PC (first) version written in Java

– 121 million copies sold worldwide (all versions)

– Probably most successful Java desktop app ever

– Mojang aquired by Microsoft in 2014

30


Minecraft


Project Malmo

Microsoft Labs project

AI experimentation platform built on top of Minecraft

– Mod for Java version of Minecraft

– Libraries to write AI agents to act as player in game

Including Java

32

Image courtesy of Microsoft


Malmo Example Code (1)

33

AgentHost host = new AgentHost();ClientPool clientPool = new ClientPool();clientPool.add(new ClientInfo("10.0.0.2"));

MissionSpec mission = new MissionSpec();mission.timeLimitInSeconds(120);mission.requestVideo(640, 480);


Malmo Example Code (2)

34

MissionRecordSpec missionRecord = new MissionRecordSpec("./saved_data.tgz");

missionRecord.recordCommands();missionRecord.recordMP4(20, 400000);missionRecord.recordRewards();missionRecord.recordObservations();

host.startMission(mission, clientPool,missionRecord, 0, "TEST");


Minecraft Interaction

MissionSpec

– setModeToCreative

– startAt/endAt

– drawBlock/drawLine/drawSphere

AgentHost

– startMission

– peekWorldState

– sendCommand

35


Putting It All Together


Demo Software Architecture

37

LaptopMinecraft with

Malmo mod

Pi Zero 1

(Input)SS RL4J

Pi Zero 2

(Fliter 1)RL4JSS

Pi Zero 3

(Fliter 2)RL4JSS

Pi Zero 4

(Output)RL4JSS

Raspberry Pi 3

Demo Controller

Malmo Library

TCP/I

P


Demo Scenario

Original idea:

– Have agent build a house

– Turns out that’s really hard

Demo revision 1

– Have the agent build a wall

– Simpler, but not much, than building a house

Demo revision 2

– Have agent walk around without getting killed

– Easier reinforcement learning, reward is to stay alive


Applying ML To Minecraft

DL4J/RL4J uses vectors to represent data

– Initially use Minecraft world state

– Long term goal is to use video frames

Have neural network extract state from this

Start with random moves

– Record success/failure (live/die)

Use reinforcement to (hopefully) learn how to navigate

minecraft world safely

39


Conclusions & Future

Directions


Conclusions

Embedded hardware can be used for AI/ML/DL

– Model on the structure of the human brain

– Cheap to build multi-core, multi-CPU cluster

Java is an ideal language for this

– Easy to learn

– Simple threading model

– Many, many useful libraries

Minecraft is a fun tool for AI testing and development

41


Future Directions

More work on ML/DL for Minecraft

– Make a better player!

Investigation of jCUDA

– Use NVidia Jetson TK1 board

Use of machine vision

– OpenCV Java bindings

42


Further Information

www.raspberrypi.org

clusterhat.com

deeplearning4j.org

minecraft.net

www.microsoft.com/en-us/research/project/project-malmo

zulu.org

Code to be published on Github after some cleaning up

– Will post a blog about this

43


Demo