the cloud & its impact on it

© 2013 IBM Corporation

December 16th 2014

Anand Haridass

Senior Technical Staff Member

STG Strategy

IBM India

The Cloud & Its Impact On High Performance Computing

© 20142

Overview

� What is Cloud Computing ?

� Scale-up vs. Scale-out

� Open Hardware

� Cloud Data Centers

� Power

� Bandwidth

� Disaggregation

© 2014

What is Cloud Computing ?

CharacteristicsOn-demand self-serviceBroad network accessResource poolingRapid elasticityMeasured service

4 Source: NIST Special Publication 800-146 Cloud Computing Synopsis and Recommendations

"Cloud computing is a model for enabling convenient, on-demand network accessto a shared pool of configurable computing resources (e.g., networks, servers,storage, applications, & services) that can be rapidly provisioned and releasedwith minimal management effort or service provider interaction.”

ServicesInfrastructure as a Service (IaaS)

Platform as a Service (PaaS)Software as a Service (SaaS)

Development Models Private CloudPublic CloudHybrid Cloud

© 20145

Standardization; OPEX savings; faster time to value

Networking

Storage

Servers

Virtualization

O/S

Middleware

Runtime

Data

Applications

Traditional On-Premises

Networking

Storage

Servers

Virtualization

O/S

Middleware

Runtime

Data

Applications

Platformas a Service

Networking

Storage

Servers

Virtualization

O/S

Middleware

Runtime

Data

Applications

Softwareas a Service

Networking

Storage

Servers

Virtualization

Middleware

Runtime

Data

Applications

Infrastructureas a Service

O/S

Vendor Manages in CloudClient Manages

Customization; higher cost; slower time to value

On Premise vs. IaaS vs. PaaS vs. SaaS

© 2014

“Cloud” Services are here to stay

6Source: *GMV 2H13

Cloud Revenue opportunity 2017 ~$57 Billion (~$30Billion ’14) @ 24% CAGR*

© 2014

Traditional vs. “Cloudified” Hardware

� “Scale-Up” � Symmetrical Multiprocessing Systems� Large shared memory machines� Expensive to scale beyond a certain

size� 4 / 8 / 16 / 32 sockets� 4U/10U/Rack Sized Systems

� “Scale-Out”� Loosely coupled systems� “Infinite” Scale� Mostly 1 & 2 sockets� 1U / 2U Form Factor (0.33/0.5/1 wide)

7

Scale Out

Sca

le U

p

Mainframes

Towers Servers

Easy to Program Hard to Scale beyond

Standard RackServers

Easy to Scale Hard to Program

SMP’s

Optimized RackServers

Blade Servers

Loosely

Coupled

Systems

� Significant changes in programming & application paradigms � Hadoop/HDFS / NoSQL DB’s….

� Open source software community driven � Linux / OpenStack …

© 2014

What this has translated to ..

� Lots of ODM/OEM Vendors�Build with ‘cheap’ commodity hardware over ‘exotic’ hardware

� ‘Inexpensive’ commodity components failure is a way of life�Redundancy (N+1) moved out of hardware ($) �Software stack needs to plan for failure �Significant work on systems resilient to storage failures

� Example *�Disk drives – 4 to 6% annual failure rate (AFR) � 5% AFT MTB of �Servers – 2 to 4% � 3% AFR translates to MTBF of 292K hrs (33yrs)� In a Datacenter with 64000 servers w/ 2 disks

� 5 servers & 17 disks fail daily !

8

Stringent SLA requirements � Applications should continue to function even if the underlying

physical hardware fails / is removed / replaced Envision a fail (Everything WILL Fail At Scale)

� work backwards Source: *James Hamilton, Amazon

© 2014

What this has translated to … Evolving

9

Accelerating examples:•Google Spanner, Omega, …•Microsoft Bing, Azure Storage•Various Amazon S3, Glacier, …•Alibaba ecommerce services•IBM Watson

Accelerating examples:•Google Spanner, Omega, …•Microsoft Bing, Azure Storage•Various Amazon S3, Glacier, …•Alibaba ecommerce services•IBM Watson

� The pendulum swinging the other way……

� Cloud vendors drive huge volumes (100K+ to a few million servers)

� Now seeing ‘customization’ for different workloads

� Accelerators / Flash / SSD’s

� More aggressive hardware – software co-optimization

� Open ‘Hardware’

Google SpannerMicrosoft BingCatapult

© 2014

Open Compute Project

10

http://opencompute.org/

© 201411

The goal of the OpenPOWER Foundation is to create an open ecosystem, using the POWER Architecture to share expertise, investment, and

server-class intellectual property to serve the evolving needs of customers.

OpenPOWER

Opening the architecture to give the industry the ability to innovate across the full Hardware and Software stack

• Simplify system design with alternative architecture

• Includes SOC design, Bus Specifications, Reference Designs, FW OS and Open Source Hypervisor

• Little Endian Linux to ease the migration of software to POWER

Driving an expansion of enterprise class Hardware and Software stack for the data center

Building a complete ecosystem to provide customers with the flexibility to build servers best suited to the Power architecture

© 201412

Growing Community

Boards / Systems

I/O / Storage / Acceleration

Chip / SOC

System / Software / Services

Implementation / HPC / Research

Complete member list at www.openpowerfoundation.org

Oct/2014

© 201413

Ecosystem Enablement

XCATXCAT

System Operating Environment Software Stack

A modern development environment is emerging based on tools and services

CloudSoftware

OperatingSystem / KVM

Standard OperatingEnvironment

(System Mgmt)

So

ftwa

re

Power Open Source Software Stack Components

ExistingOpen

Source Software

Communities

Firmware

Hardware

New OSS Community

OpenPOWERTechnology

OpenPOWERFirmware

CAPP

PC

Ie

POWER8

CAPI over PCIe

“Standard POWER Products” – 2014

Ha

rdw

are

“Custom POWER SoC” – Future

Customizable

Framework to Integrate System IP on Chip

Industry IP License Model

Multiple Options to Design with POWER Technology Within OpenPOWER

44,000 packages now available

© 2014

OpenPOWER – Find out more !

www.tyan.com/campaign/openpower/openpowerfoundation.org/

The Google reference board� two single-chip module (SCM)� four modified SATA ports� Google use only

http://www.enterprisetech.com/2014/04/28/inside-google-tyan-power8-server-boards/

© 2014

Cloud Scale Data Center

Component Sub-Components

Servers* CPU, memory, disk

Infrastructure* UPS, Cooling, Power Distribution

Power (W) Electric Utility Costs

Networking Switches/Links/Transit

15

* 3 yr amortization on Servers & 15yrs on infrastructure

Investing in a Datacenter (250million - 1Billion+) High scale� 10’s to 100’s of thousands of

serversGeo-distribution� 10s to 100s of DCsStringent availability & performance requirements� 99.9th percentile SLAs� Cost per transaction / cost per

data “unit”Complexity� Lots of components: Load

Balancers, operating system, middleware, virtualization/containers, switches, servers, racks …

© 201416

Let’s drill down on Power (Watts)� Datacenter power usage

� Total global data center power use ~320 TWHr (Data Center Dynamics Focus, Nov 2012)

� Total data center global electricity use 1.8%

� PUE is defined as the ratio of total facilities energy to IT equipment energy(Not a perfect metric – but that’s a discussion for another day *)

� Datacenter efficiency – Average DC efficiency with PUE over 2.0 (Source: EPA) � Lots of High-end cloud services in 1.2 to 1.5 range� Lowers computing cost & better for environment � Corporate Responsibility � Best of Breed < 1.1 (1.05 …1.08)

� Outside air cooled (no chillers) � seeing DC’s in low temp belts� Sea / River water cooled

* The Green Grid White paper #49 – “PUE : A comprehensive examination of the metric”

� Baseline Server Failure rate = 2%� 40C inlet (cold aisle temperature)

degrades by 1.65x � Enhanced failure rate due to

temperature = 3.3 � More economical to run hotter ??

Source: Published by ASHRAE TC 9.9 ; Roger Schmidt, IBM

© 201418

Rack Level PowerData Center IT Power Trends: Maximum kW/Rack(survey)

� Water Advantages� Order of magnitude lower unit

thermal resistance� 3500X heat carrying capacity� Lower temperature

� Lower power (less leakage)� Better reliability

� Water Disadvantages� Added complexity� Added cost (but not necessarily

cost/performance)� The perception of water cooling

Water flow onWater flow off

Rear Door Heat Exchanger

� Eliminate rack heat exhaust

� Same dimensions as standard rear door 4” deep

� Liquid cooling at the rack is 75%-95% more efficient than air cooling by a CRAC

� No electrical or moving parts

� No condensation

Source: Roger Schmidt, IBM

© 201419

Server Level Power

� Maximize Utilization� Typical datacenter utilization ~10-15%� Virtualization / Containers � get it to 80-

90% (depending on performance SLA)

� Fans/Blowers (~20-30% of Server Power)

� Understand Regulator efficiency (op. zone)

� Energy Proportional Computing� Idle Power % of Max Power ?

� Performance/Watt metrics

Fan-based Power Optimization

Increasing Microprocessor Temperature (Decreasing fan speed)

Incr

ea

sin

g P

ow

er

Co

nsu

mp

tio

n

Microprocessor leakage power

Fan power

Total power

Source: Ryan Waite, Microsoft

© 201420

Processor Level Power

� Very Aggressive DVFS � Transients / switching currents (typically over predicted) – inductive noise� Model / validate accurately � ‘Guard-band’ voltages margins – power wastage� Emergency brakes (CPM)

� Rapid Power / Thermal Cycling� CTE mismatch Silicon Organic packages� Electron migration effects / Reliability

� Circuit Techniques - Resonant Clocking � Leverages inductive clock grid

� Leverage technology (deep trench capacitance)� Enables significant on-chip decap (~20uF) � Mid/high-freq noise significantly reduced

Source: Dale Becker, IBM

© 201421

Networking

� Networking is THE biggest issue that Cloud DC are grappling with

� Cost take down not following ‘Moore’s Law’ � Big push for SDN

� Intra-datacenter (East-West) traffic increasing� 44% CAGR in DC traffic, 76% within DC

(Cisco 2012*)� 80%+ of Google traffic now internal facing

(B. Koley, Google OI Conf. 2012)� Every 1kb of external traffic entering the

datacenter generates 930kb of internal traffic (N.Farrington, Facebook OI Conf. 2013)

Source: Marc Taubenblatt, IBM

© 2014

The Case for Optics

CopperCopper OpticsOptics

Up t

o 8

0K

m f

or

Eth

ern

et,

100G

bps

Bandw

idth

at

low

pow

er

• Increasing benefits with optical, but products generally cost more than copper • Optics less expensive when integrated with silicon - Silicon Photonics

• Photonics integrated into silicon base• Reduces cost and provides higher bandwidth

© 201423

Cloud Network

Speed 10�40�100�400Gbps/drawer

Distance m’s to km’s

Protocol Commodity*/ Standards

Packaging Mid Card / Card edge

The Case for Optics

Source: Marc Taubenblatt, IBM

Electrical getting harder with increased data rate as loss increases

© 201424

Why Disaggregation

Storage

Memory

Network and I/O

GPU/ Accelerators

CPU’s

Today� Balanced Compute/Network/Storage� Refreshed to optimize sub-

component (cost / performance / bottlenecks)

Compute Node(s)

Mem CPU NIC

Patch Panel / Switch and/or Shared “NIC”

Storage Node(s)

GPU Node(s)

Memory Node(s)

Network Connectivity

Fabric

Disaggregation� Flexible, Composable (workload optimized)� Enables higher density compute architecture � Software defined deployment� Improved utilization, TCO & TCA� Independent technology refresh cycles

© 201425

Disaggregation

Electrical Switch

Tight Integration< few U

Rack Level Integration

POD/DC Level Integration

Electrical?

Optics ?

Optical Switch

the cloud & its impact on it

Devices & Hardware