HPC Opportunities In Deep LearningGreg Diamos, SC16

AN OVERVIEW …

1. Why is Deep Learning Important Now in HPC?2. Published Results with ImageNet, Google DeepMind,

Baidu AI Lab.3. Getting Started with Deep Learning in HPC.4. Future Direction and Opportunities for Growth with

HPC and Deep Learning.

Source: Greg Diamos SC16 Talk

Before, we had no idea how to train neural networks. The prevailing opinion, at the time, was that they were impossible to train.

Image Source: NVIDIA

Why is Deep Learning Important Now in HPC?

But now, we have powerful tools that can start applying to problem after problem and making progress on those that are really incredible inherently difficult.Content Source: Greg Diamos SC16 Talk

THE PUBLISHED EVIDENCE SPEAKS FOR ITSELF…

The ImageNet Challenge We first found success in the ImageNet challenge, in which ImageNet was given images and had to produce a corresponding label. The challenge encompassed over a very large dataset of images and then classified into a thousand different categories. We’ve approached human-level accuracy with deep learning algorithms for these systems. Content Source: Greg Diamos SC16 Talk Image Source: NVIDIA, Greg Diamos SC16

Talk

THIS PROGRESS ONLY CONTINUED GROWING EXPONENTIALLY…

DeepMind at Google Just last year, a deep neural network defeated one of the best players, the best human players in a game of ‘Go.’ This is a game with an absolutely enormous optimization space. There’s no way to search over all possible combinations. Content Source: Greg Diamos SC16 Talk

Image Source: Greg Diamos SC16 TalkImage Source

Baidu’s AI LabAt our lab, we can approach human-level accuracy on many test sets.For example, when you build a speech recognition system, you would hand-design all of these components. You would not have one neural network. You would have five or six components all hand-designed by linguists, speech, signal processing, and mathematicians.

Content Source: Greg Diamos SC16 Talk

Image Source

Image Source

We cut all that out.

Image Source

Content Source: Greg Diamos SC16 Talk

Baidu’s AI Lab Cont. We can now take a team of five people who don’t speak any Mandarin, and produce a speech recognition system that beats all of the existing systems that we have and actually does better than a human grader.

But these things are incredibly computationally intensive to train.

AND WE’RE NOW GETTING INTO THE RELATIONSHIP BETWEEN DEEP LEARNING AND HIGH PERFORMANCE COMPUTING SYSTEMS.

GETTING STARTED WITH DEEP LEARNING IN HPC

1.Big Model2.Big Data3.Big Computer

Source: Greg Diamos SC16 Talk

What do you need in order to get started solving a new problem that you want to apply deep learning to? There are three simple, but high-level factors:

1. Big Model

Image Source

First, you need a big model. Your model has to be able to approximate the function that you’re trying to represent. For example, the function that maps images to text is complicated. Many parameters are needed to actually represent it. The model must be big in order to capture a extremely intricate function. Content Source: Greg Diamos SC16 Talk

2. Big Data

Image Source

Deep Learning doesn’t perform very well with small datasets. This was another reason as to why people might not have thought Deep Learning was important before. Deep Learning on smaller datasets would easily be beaten out by simpler, more explicit methods. But, as the datasets get larger, they start to surpass other methods.”

Content Source: Greg Diamos SC16 Talk

3. Big Computer

Image Source

And when you have a big network and big data, you need a powerful supercomputer to run it. If you don’t have a fast enough computer, one can be stuck waiting years or decades for a result. So we come to this need for speed. And this is really the most important point in the talk.

Content Source: Greg Diamos SC16 Talk

WHAT ARE THE OPPORTUNITIES OF GROWTH FOR HPC IN DEEP LEARNING?

Opportunities For Growth

Image Source

First, we need to figure out a way of scaling up models. Currently, the biggest model that runs at a high percent of efficiency is about 100 processors, which is large from a machine learning perspective but it is small from an HPC perspective. Second, we are far away from the power limit in CMOS. Right now, we’re around ten teraflops per processor. I think we can get to 20 petaflops before we hit the power limit. You can make progress on speech, vision, and language problems by making faster computers. Content Source: Greg Diamos SC16 Talk

Image Source

Future Directions of HPC and Deep LearningThe two big directions that we see are around speech powered interfaces and self-driving cars. Speech powered interfaces are really three different components: recognition, human-level accuracy, and computer generation. Self-driving cars are also highly valuable as they leverage a lot of vision technology that’s already been developed.Both areas are significant directions going forward, but there are definitely even more application beyond these that are close to becoming possible using deep learning.Content Source: Greg Diamos SC16 Talk

Image Source

About the Speaker: Greg DiamosGreg Diamos is a senior researcher at Baidu’s Silicon Valley AI Lab (SVAIL). Previously, he was on the research team at NVIDIA. Greg holds a PhD from the Georgia Institute of Technology, where he contributed to the development of the GPU-Ocelot dynamic compiler, which targeted CPUs and GPUs from the same program representation.

FOR THE FULL RECORDING: WATCH HERE

LEARN MORE ABOUT THE INTERSECTION OF AI AND HPC

INSIDEBIGDATA GUIDE