System Development & Technology EnablementIBM Printed Circuit Board Symposium – Nov 2011

Michael G. NealonDirector – IBM Enterprise Systems Hardware Development

2

New ERA of Computing – Client Driven “Smart” Computing

� New Compute Environment Paradigm

� Performance Alone will not drive buying decision � BUT is a defacto requirement

� Heterogeneous Environment

� Utility Compute Delivery Models � CLOUD Computing � Grid Compute Networks

� Virtualization

� Workload Consolidation

� Data Center Energy Efficiency

� RASU – Reliability, Availability, Serviceability & USEABILITY

� System Management

� Security

� Workload Accelerators / Specialty Engines

� Integrated Networking & Storage

TCOTotal

Cost ofOwnership

& BUSINESSValue

3

Data Center Cost Factors

SW Licenses6%SW Maintenance

5%DB+OS Licenses

3%

Hardware Costs8%

Implementation Costs29%

Internal Implementation Costs

8%

Application Dev & Support Costs

25%

Infrastructure Support Costs

10%

Administrative Costs6%

Rising Costs and Complexity Threaten Profits and Competitiveness

Spending(US$B)

Source: IDC, Virtualization 2.0: The Next Phase in Customer Adoption, Doc #204904, Dec 2006

$0

$50

$100

$150

$200

$250

$300

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Installed Base(M Units)

Spending(US$B)

New server spendingServer mgmt and admin costs x4Power and cooling costs x8

05101520253035404550

Total Cost of Ownership Factors - Example

4

Data Center Cooling Cost and Directions

5

Server Market Trend and Directions

Internet Data CentersUltra Low CostHuge Scale Out

Server Redundancy

Micro-ServersUltra Low Power

Workload Optimized Integrated SolutionsBundled Hardware &

Software w/ Application

Management

Cloud ComputingIT Utility Service

Model

6

System Hardware Initiatives for “Smarter” Computing

� So How Does New Compute Environment Drive System Hardware?

• System Integration Density (Volumetric Area)� Compute Density driving bandwidth, memory & IO density� Integration of Storage & Networks� Flash Memory

• System Design for RAS� Complexity, Density & Power drive reliability and serviceability� New RAS models??? Fail in Place???

• Energy Efficiency at Data Center Level� Power Management� High Voltage DC Power Distribution� Water Cooling� Heat exchangers

• COST� Cost of hardware� Cost of RAS� Cost Enablement of TCO Value

77

� All data center power & cooling infrastructure included in compute/storage/network rack� No need for external power distribution or computer room air handling equipment.� All components correctly sized for max efficiency – very good 1.18 Power Utilization Efficiency� Integrated management for all compute, storage, network, power, & thermal resources.� Scales to 512K P7 cores (192 racks) – without any other hardware except optical fiber cables

P7 775 System “Data-Center-In-A-Rack” System Architecture

Integrated Storage – 384 2.5” HDD or SSD drives /drawer230 TBytes\drawer (w/600 GB 10K SAS disks), 154 GB/s BW/drawer, software-controlled RAID, up to 6/rack (replacing server drawers) (up to 1.38 PBytes / rack)

Integrated Cooling – Water pumps and heat exchangersAll heat transferred directly to building chilled water – no thermal load on room

Integrated Power Regulation, Control, & DistributionRuns off any building voltage supply world-wide (200-480 VAC or 370-575VDC). Full in-rack redundancy and automatic fail-over.

Up to 252 kW/rack max / 163 kW Typ.

Servers – 256 Power7 cores / drawer, 1-12 drawers / rackCompute: 8-core Power7 CPU chip, 3.7 GHz, 45nm technology, 32 MB L3 eDRAM/chip, 4-way SMT, 4 FPUs/core, Quad-Chip Module; >90 TF / rack

No accelerators: normal CPU instruction set, robust cache/memory hierarchyEasy programmability, predictable performance, mature compilers & libraries

Memory: 512 Mbytes/sec per QCM (0.5 Byte/FLOP), 12 Terabytes / rackExternal IO: 16 PCIe Gen2 x16 slots / drawer; SAS or external connectionsNetwork: Integrated Hub (HCA/NIC & Switch) per each QCM (8 / drawer), with 54-port switch, including total of 12 Tbits/s (1.1 TByte/s net BW) per Hub:

Host connection: 4 links, (96+96) GB/s aggregate (0.2 Byte/FLOP) On-card electrical links: 7 links to other hubs, (168+168) GB/s aggregateLocal-remote optical links: 24 links to near hubs, (120+120) GB/s aggregateDistant optical links: 16 links to far hubs (to 100M), (160+160) GB/s aggregatePCI-Express: 2-3 per hub, (16+16) to (20+20) GB/s aggregate

FrontFront

RearRear

88

P7 775 System Node Hardware – Node Front View

P7 QCM (8x)

Hub Module (8x)

D-Link Optical InterfaceConnects to other Super Nodes

360VDC Input Power Supplies

Water Connection

L-Link Optical InterfaceConnects 4 Nodes to form Super Node

MemoryDIMM’s (64x)

PCIe Interconnect

1m W x 1.8m D x 10cm H

MLC Module

Hub Assembly

PCIe Interconnect

D-Link Optical InterfaceConnects to other Super Nodes

Avago microPODTM All off-node communication optical

9

Hub Module – MCM with Optical I/Os

�This shows the Hub module with full complement of Optical I/Os. �Module in photo is partially assembled, to show construction – full module HW is symmetric

Heat Spreader for Optical DevicesCooling / Load Saddle for Optical Devices

Optical Transmitter/Receiver Devices 12 channel x 10 Gb/s 28 pairs per Hub - (2,800+2,800) Gb/s of optical I/O BW

Heat Spreader over HUB ASIC

Strain Relief for Optical RibbonsTotal of 672 Fiber I/Os per Hub, 10 Gb/s each

Hub ASIC (Under Heat Spreader)

10

What’s Next?

0.1

1

10

100

1000

1996 1998 2000 2002 2004 2006 2008 2010 2012

Date of publish

Re

lati

ve

Pe

rfo

rma

nc

eSocket-Level Perfomanceis growing at 50% CGR

•Socket Compute density will continue to grow•Si Device Perf. & Operating Freq Flatten

•Performance Obtained Through:• Increased Cores per Die• Increased Cache per Core• Higher Levels of Integrated Function• Need Innovation to “Feed the Beast”

•Differentiation will come through Integration•At Processor & System Level•Data Center Integration in a Chassis•BW is Critical to Success

1

10

100

2005 2006 2007 2008 2009 2010 2011 2012

Date of publish

Re

lati

ve

Ca

pa

cit

y

Cores per socket

0.1

1

10

100

2006 2007 2008 2009 2010 2011 2012

Date of publish

Re

lati

ve

Ca

pa

cit

y

Memory Capacity is growing at about 60% per year.

4-Socket GB/skt

2-Socket GB/skt

4-Socket GB/core

2-Socket GB/core

11

Innovation that Matters – Future Technology Directions

• Improved Materials & Process Controls• Low K Materials• Moisture Resilient • Built in Quality

• Stacked Memory (TSV Enabled)• 4-8X Density Improvement• RAS Enhancements to Enable

• Solid State Memory Exploitation• Reduced Cost• PCI Attached• Disk Drives• DIMMS

• Optics / Silicon Photonics• Reduced Cost Tx/Rx • Integrated Silicon Photonics• Advanced Packaging

• Integrated Modules• PCB Waveguides

TSV Si carrier Optochips assembled

6.4mm x 10.4mm

2x12 PD

array

2x12 RX IC

2x12 LDD IC

2x12 VCSEL array

RXTXLDD RX

Si CarrierVCSEL

Lens Arrays

PD

Organic Carrier

PCB

Polymer Waveguides

or Flex

To optical connector

TSV Si carrier Optochips assembled

6.4mm x 10.4mm

2x12 PD

array

2x12 RX IC

2x12 LDD IC

2x12 VCSEL array

RXTXLDD RX

Si CarrierVCSEL

Lens Arrays

PD

Organic Carrier

PCB

Polymer Waveguides

or Flex

To optical connector

Optical PCB Polymer waveguides

12

System Hardware Initiatives for “Smarter” Computing

Summary

� IT Compute Environment is Changing� New Deployment Models� Holistic Client Perspective – E2E Cost � Opens New Ways to Integrate Solutions

� Integration Wins!!!� Integration at all levels drives IT density / value� HW/SW Integration provides Workload & Management Optimized Solutions

� Technology Development which provides VALUE is required� Need Integrated Solution Based Technology

� Drives Need for Collaboration across Technology Development� Cost & Reliability are critical

Thank You