“The Rise ofMachine Intelligence”

Big Thought Leaders Colloquium Series – Spring 2017Jackson State University

Jackson, MSApril 11, 2017

Dr. Larry SmarrDirector, California Institute for Telecommunications and Information Technology

Harry E. Gruber Professor, Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSDhttp://lsmarr.calit2.net

1

Abstract

Over the next decade, we can expect autonomous machine intelligence to become pervasive in our society. To help spur that future, Calit2 has set up a Pattern Recognition Laboratory with a variety of novel low-energy processors that can execute real-time trained neural networks in the exploding mobile environment of drones, robots, and self-driving cars. However, the training of these neural networks requires massive amounts of Big Data and computing time. To support this need the NSF-funded Pacific Research Platform (PRP), which connects two dozen research universities at 100-1000 times the speed of the commodity Internet, is creating a new community of computer science machine learning researchers and proposing using the optical fiber backbone of the PRP to create a distributed Graphics Processing Unit computing “cloud.” Finally, I will speculate on the exponentially growing machine intelligence and how it will increasingly inter-operate with human intelligence.

Things Are About to Get Very Interesting…

Source: Hans Moravecwww.transhumanist.com/volume1/power_075.jpg

Smarr Slide from 2001

The Defining Issue in IT for the Coming Decades

May 5, 2015August 25, 2015

Traffic Control for Drone Air Delivery is Under Development by NASA, Amazon, & Google

Self-Driving Cars Have Appeared on the Market

I Am Living in the Self-Driving Future

Autopilot at 71 MPH

Streaming Data From the Tesla Fleet Trains Self-Driving Algorithms: The “Hive-Mind” Advantage

Note: Google Self-Driving Cars Have Only Driven 1.5 Million Miles

One Major Source of JobsMay Be the First Victim of Driverless Vehicles

DOD: “Perdix Drones Share One Distributed Brain for Decision-Making, Adapting to Each Other Like Swarms in Nature.”

The Planetary-Scale Computer Fed by a Trillion SensorsWill Drive a Global Industrial Internet

www-bsac.eecs.berkeley.edu/frontpagefiles/BSACGrowingMEMS_Markets_%20SEMI.ORG.html

Next Decade

One Trillion “Within the next 20 years the Industrial Internet

will have added to the global economy

an additional $15 trillion.”--General Electric

www.ge.com/docs/chapters/Industrial_Internet.pdf

What is the Cyberinfrastructure NeededFor The World of Big Data Autonomous Machines?

• Massive Multi-Architecture Cloud Computing

• Trained Neural Nets Downloaded onto Robots with NvN ML Accelerators

• Robots Use Neural Nets to Navigate with Real-Time Data Streams

• Swarm Input to Update Training on Neural Nets

For ¾ of a Century, Computing Has Reliedon von Neumann’s Architecture

Is It Time to Radically Expand Our Computer Architectures?

NCSA 1988

Supercomputer Architectures Remain von NeumannShared Memory CPU Plus SIMD Co-Processor

NCSA 2016

The DOE and NSF Petascale SupercomputersAll Are Built with CPU/GPU Nodes

Like Supercomputers, Commercial Cloud Providers Are Adding GPU Accelerators

All Use Double Precision Nvidia Tesla GPUs

But, Commercial Cloud Providers Are Also Introducing NvN Accelerators:Microsoft is Using Field Programmable Gate Arrays (FPGAs)

• Microsoft Installs FPGAs into Bing Servers– FPGAs are a Non von Neumann (NvN) Architecture– Improved the Ops/Sec of a Critical Component of Bing’s Search Engine by Nearly 2x– Many Other Applications and Services Can be Accelerated as Well

www.microsoft.com/en-us/research/project/project-catapult/

The Democratization of Deep Learning:Google’s TensorFlow

https://exponential.singularityu.org/medicine/big-data-machine-learning-with-jeremy-howard/

From Programming Computers Step by Step To Achieve a Goal

To Showing the Computer Some Examples of

What You Want It to Achieve and Then Letting the Computer

Figure It Out On Its Own--Jeremy Howard, Singularity Univ.

2015

Google Designed a NvN Machine Learning Accelerator

AI is Advancing at an Unprecedented Pace:Deep Learning Algorithms Working on Massive Datasets

1.5 Years!

Training on 30M Moves, Then Playing Against Itself

Google Used TPUs to Achieve the Go Victory

Exascale (1000 PetaFLOPs) Will Blend Traditional HPC and Data Analytics:U.S. Committed to Building by 2025

“High Performance Computing Will Evolve Towards a Hybrid Model,

Integrating Emerging Non-von Neumann Architectures, with Huge Potential in Pattern Recognition,

Streaming Data Analysis, and Unpredictable New Applications.”

Horst Simon, Deputy Director, U.S. Department of Energy’s

Lawrence Berkeley National Laboratory

Realtime Simulation of Human Brain Possible Within the Next Ten Years With Exascale Supercomputer

Horst Simon, Deputy Director, Lawrence Berkeley National Laboratory’s

National Energy Research Scientific Computing Center

Fastest Supercomputer

Trend LineTianhe-2

Using Nanotechnology to Read Out the Living BrainIs Accelerating Under the Federal Brain Initiative

www.whitehouse.gov/infographics/brain-initiative

Reverse Engineering of the Brain:Large Scale Microscopy of Mammal Brains Reveals Complex Connectivity

Source: Rat Cerebellum Image, Mark Ellisman, UCSD

NeuronCell Bodies

Neuronal DendriticOverlap Region

The Rise of Brain-Inspired Computers:Left & Right Brain Computing: Arithmetic vs. Pattern Recognition

Adapted from D-Wave

Brain-Inspired ProcessorsAre Accelerating the non-von Neumann Architecture Era

“On the drawing board are collections of 64, 256, 1024, and 4096 chips.

‘It’s only limited by money, not imagination,’ Modha says.”Source: Dr. Dharmendra ModhaFounding Director, IBM Cognitive Computing Group

August 8, 2014

Calit2’s Qualcomm Institute Has Established a Pattern Recognition Lab For Machine Learning on non-von Neumann Processors

“On the drawing board are collections of 64, 256, 1024, and 4096 chips.

‘It’s only limited by money, not imagination,’ Modha says.”Source: Dr. Dharmendra Modha

Founding Director, IBM Cognitive Computing Group

August 8, 2014

UCSD ECE Professor Ken Kreutz-Delgado Brings the IBM TrueNorth Chip

to Start Calit2’s Qualcomm Institute Pattern Recognition Laboratory

September 16, 2015

Our Pattern Recognition Lab is Exploring Mapping Machine Learning Algorithm Families Onto Novel Architectures

• Deep & Recurrent Neural Networks (DNN, RNN)• Graph Theoretic• Reinforcement Learning (RL)• Clustering and Other Neighborhood-Based• Support Vector Machine (SVM)• Sparse Signal Processing and Source Localization• Dimensionality Reduction & Manifold Learning• Latent Variable Analysis (PCA, ICA)• Stochastic Sampling, Variational Approximation• Decision Tree Learning

Source: Prof. Ken Kreutz-Delgado, Director PRL, UCSD

New Brain-Inspired Non-von Neumann Processors Are Emerging:KnuEdge is Essentially a Cloud-on-a-Chip That Scales to 512K Chips

www.tomshardware.com/news/knuedge-announces-knuverse-and-knupath,31981.html

www.calit2.net/newsroom/release.php?id=2704

“KnuEdge and Calit2 have worked together since the early days of

the KnuEdge LambdaFabric processor, when key

personnel and technology from UC San Diego

provided the genesis for the first processor design.”

www.calit2.net/newsroom/release.php?id=2726

June 6, 2016

KnuEdge Has Provided Processor to Calit2’s PRL

Neurobiological Systems are Flexible and Scalable

30

Animal # NeuronsFlatworm 302Medicinal leech 10,000Pond snail 11,000Sea slug 18,000Fruit fly 100,000Lobster 100,000Ant 250,000Honey bee 960,000Cockroach 1,000,000Frog 16,000,000Mouse 75,000,000Bat 110,000,000Octopus 300,000,000Human 100,000,000,000

Elephant 200,000,000,000

Current generation machine learning capabilities

Where KnuEdge wants to be in 2021:MindScale.

Source: Doug Palmer, CTO KnuEdge

Building a Big Data Cyberinfrastructure for Machine Learning

Based on Community Input and on ESnet’s Science DMZ Concept,NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways

2012-2015 CC-NIE / CC*IIE / CC*DNI PROGRAMS

Red 2012 CC-NIE AwardeesYellow 2013 CC-NIE AwardeesGreen 2014 CC*IIE AwardeesBlue 2015 CC*DNI AwardeesPurple Multiple Time Awardees

Source: NSF

Next Step: The Pacific Research Platform Creates a Regional End-to-End Science-Driven “Big Data Superhighway” System

NSF CC*DNI Grant$5M 10/2015-10/2020

PI: Larry Smarr, UC San Diego Calit2Co-Pis:• Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UCSD SDSC, • Frank Wuerthwein, UCSD Physics and SDSC

Letters of Commitment from:• 50 Researchers from 15 Campuses• 32 IT/Network Organization Leaders

FIONAs and FIONettes – Flash I/O Network Appliances:Linux PCs Optimized for Big Data Over Distance

FIONAs Are Science DMZ Data Transfer Nodes (DTNs) &

Also Compute/Visualization/ML Nodes

Phil Papadopoulos & Tom DeFantiJoe Keefe & John Graham

FIONAS—40G, $8,000FIONette—1G, $1,000

PRP Continues to Expand Rapidly While Increasing Connectivity:1 1/2 Years of Progress – 12 Sites to 23 Sites

January 29, 2016 March 29, 2016

Connected 23 DMZs at 10G and 40G, demonstrating disk-to-disk GridFTP

at ~7.5G and 12.5G respectively

PRP’s First 1.5 Years: Connecting Campus Application Teams and Devices

Adding a Cognitive Hardware and Software EcosystemTo the Pacific Research Platform

• Working With 30 CSE Machine Learning Researchers– Goal is 320 Game GPUs in 32-40 FIONAs at 10 PRP Campuses– PRP Couples FIONAs with GPUs into a Condor-Managed Cloud

• PRP Access to Emerging Processors– IBM TrueNorth, KnuEdge, FPGA, and Qualcomm Snapdragon

• Software Including a Wide Range of Open ML Algorithms • Metrics for Performance of Processors and Algorithms

Source: Tom DeFanti, Calit2

Multiple Proposals Under Review

FIONA with 8-Game GPUs

Single vs. Double Precision GPUs:Gaming vs. Supercomputing

8 x 1080 Ti: 1 million GPU core-hours every two days, 500 million for $15K in 3yrs

Contextual Robots Need Low Energy Neuromorphic Processors That Can See and Learn Wirelessly Tied Into the Planetary Cloud Computer

Professor Tajana Rosing

Kurzweil’s Theory of Mind: The Human Neocortex is a Self-Organizing Hierarchical System of Pattern Recognizers

“There are ~300M Pattern Recognizers

in the Human Neocortex.”

In the Emerging Synthetic Neocortex, “Why Not a Billion?

Or a Trillion?”

November 13, 2012

Should We Give Robots Autonomy?

This Next Decade’s Computing TransitionWill Not Be Just About Technology

"Those disposed to dismiss an 'AI takeover' as science fiction may think again after reading this original and well-argued book." —Martin Rees, Past President, Royal Society

If our own extinction is a likely, or even

possible, outcome of our technological

development, shouldn't we proceed with great

caution? – Bill Joy

Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks. – Steven Hawking

For Further Information:

http://lsmarr.calit2.net/