dtu – distributed systems · spi tpm1.2 system bios 16mb spi parallel exp. nand flash 2gb pbdsa...

© 2012 IBM Corporation

DTU – Distributed SystemsIBM PureFlex SmartCloud Entry Solution Technical WalkthroughLyngbyApril 18th, 2013

Ib F. Kristensen, [email protected] ArchitectNordic PureFlex Product Manager

Agenda IBM PureFlex SmartCloud Entry Solution

� IBM

� Concept overview of the system

� Design issues/requirement for the Flex Chassis

� The physics of the Flex Chassis

� The networking in the Flex Chassis

� The Compute power in the Flex Chassis

� Management of the hardware

� The Cloud software stack

� Wrap up

IBM Corporation

* source: ibm.com/investor/4q12/presentation/4q12.pdf

Key figures

� Headquarters

Armonk, New York, USA

� CEOVirginia M. Rometty

� Employees 2012434,246

� Revenue 2012104.5 billion US-Dollar

� Homepageibm.com

With revenues of 104.5 billion US-Dollar in 2012, IBM is

one of the world’s largest provider of information technology (hardware, software and services) and B2B

solutions. The company has more than 430,000 employees worldwide and operates in more than 170

countries.

For more financial information please visit

ibm.com/investor/financials/index.phtml

IBM revenue by industry* (in billion USD)

IBM revenue

Revenue in million USD 2012 2011 2010

Global Services

Global Technology ServicesGlobal Business Services

58,802

40,23618,566

60,163

40,87919,284

56,424

38,20118,223

Hardware 17,667 18,985 17,973

Software 25,448 24,944 22,485

Global Financing 2,013 2,102 2,238

Other 577 722 750

Total revenue 104,507 106,916 99,870

Other: 5.9 Small and mediumbusiness: 24.2

Communications: 9.5

Distribution: 9.2

Industrial: 9.7

Financial services: 30.7

Public: 15.3

Total revenue104.5 billion

USD

5

Introduction to the “New” IBM

Concept overview of the system

What is Cloud Computing?

A user experience and a business modelRapidly provisioned standardized offeringsUsage based pricing

An infrastructure management and services delivery methodology

Virtualized resources with elastic scaling

Monitor & ManageServices & Resources

CloudAdministrator

DatacenterInfrastructure

Service Catalog,ComponentLibrary

Service Consumers

Component Vendors/Software Publishers

Publish & UpdateComponents,Service Templates

IT Cloud

AccessServices

The IBM CCRA (Cloud Computing Reference Architecture) defines 3 different roles: the Cloud Service Consumer, the CloudService Provider and the Cloud Service Creator.

Governance

Security, Resiliency, Performance & Consumability

Cloud ServiceCreator

Cloud ServiceConsumer

Cloud Service Provider

Common Cloud

Management Platform (CCMP)

Operational

Support

Services

(OSS)

Cloud Services

Infrastructure-as-a-Service

Platform-as-a-Service

Sof tware-as-a-Service

Business-Process-

as-a-Service

Business

Support

Services

(BSS)

Cloud Service

IntegrationTools

ConsumerIn-house IT

Service Creation

Tools

Inf rastructure

Existing & 3rd party

services, Partner

Ecosystems



Application-as-a-Service

Servers Networking Storage

Middleware

Collaboration

Financials

CRM/ERP/HR

Industry Applications

Data Center Fabric

Shared virtualized, dynamic provisioning

Database

Web 2.0 ApplicationRuntime

JavaRuntime

DevelopmentTooling

Four major categories of Cloud Computing services are emerging Examples

Business Process-as-a-Service

Employee Benefits Mgmt.

Industry-specific Processes

Procurement

Business Travel

The IBM CCRA (Cloud Computing Reference Architecture) defines 3 different roles: the Cloud Service Consumer, the CloudService Provider and the Cloud Service Creator.

Governance

Security, Resiliency, Performance & Consumability

Cloud ServiceCreator

Cloud ServiceConsumer

Cloud Service Provider

Common Cloud

Management Platform (CCMP)

Operational

Support

Services

(OSS)

Cloud Services



Sof tware-as-a-Service

Business-Process-

as-a-Service

Business

Support

Services

(BSS)

Cloud Service

IntegrationTools

ConsumerIn-house IT

Service Creation

Tools

Inf rastructure

Existing & 3rd party

services, Partner

Ecosystems

High-level view of IBM SmartCloud Entry for software cloud solution

IBM Flex System Enterprise Chassis

�Support technology advancements through the next several years

– Future processors, memory, I/O, storage, specialty hardware

– Faster internal bus speeds

– Faster I/O speeds

�Energy efficient cooling and power system

– High efficiency components

• Sophisticated control systems

• Compatible with future, high efficiency data center infrastructure

– New Compute and Storage nodes (Leadership Intel Performance)

– New scalable switch elements and IO adapters

– New integrated management node and chassis management elements

• Ability to manage multiple chassis

Design issues/requierment for the Flex Chassis

Some of the design issues/requirement for the Flex System

� Based on standard architecture and components (x86, Power, FC, ethernet...)

� Lifetime of appx. 10 years

� Low cost

� Simplicity

� Flexibility

� Integration

� Reliability, redundancy

� upgradeability

Infrastructure Application Platform Data Platform

Delivering Cloud Infrastructure Services

Delivering Cloud Application Platform

Services

Delivering Big Data Platform Services

New Analyt

ics Model

The physics of the Flex Chassis

2 nodes each with 2 CPU sockets, 24 DIMMs and 2 Hot-Swap Hard Drives

2 node Fillers

2 node Fillers

2 nodes without Hot-swap HDDs

Full Width node

Full Width node with up to 4 processor sockets, 48 DIMMs and Hot-Swap HDDs

2 CPU node with PCIe adaptor expansion node

Bay 1

Bay 14

Chassis Front View and Compute Node examples

Power Supplies (6X) IO Modules (4X)

40 mm Fans (2X)

CMM (2X)

80 mm Fan Packs (8X)

Fan Mux Card (2X)

Chassis Rear View

4231

1

6

1

10

Node Bay Shelf

� Remove for Full Wide node

� Remove Two Shelves for Full Wide, Double High node

Chassis

Shuttle

Chassis Bay Shelf Removal

Flex System Power – Overview

� Flex chassis is optimized for 3-phase, 60A, 200-208VAC (US/Japan)

– Efficient use of 3-phase, 32A 220-240VAC / phase (International)

� Chassis contains up to six 2505W power supplies

– Supports both N+N and N+1 power redundancy

– Efficient balancing of N+N power across all 3 phases

– Power supply unit (PSU) is sized to efficiently use PDU power

� High Efficiency

– PSU certified to 80 PLUS Platinum

– Low power loss mid-plane and connectors

Cooling Paths – Zoned Cooling Between Subsystems

Switch Cooling

Node Cooling Power Supply Cooling

Zoned Cooling:• Nodes

• Left / Right

•

•Subsystems

• Node, Switch, and PSU Modules

The networking in the Chassis

I/OAdapter

1I/O

Adapter2

Node I/O: LAN-on-Motherboard (LoM) and I/O Adapter slotsx86 nodes have models with LoM and models without LoM, Power nodes only have models without LoM

CMM

L2 Switch

Flex System ManagerBay 1

IMM2 LOMEth

EthL2 Switch

Compute Node x86Bay 2

IMM2

CMM

L2 Switch

Adapter 1 Adapter 2

Compute Node POWERBay 14

FSP Adapter 1 Adapter 2

Switch Bay 1Switch Bay 2

Switch Bay 3

Switch Bay 4

Management Network

Data Network

I/O Bandwidth: “Future-proofed” mid-plane capability

CMM

L2 Switch

Flex System ManagerBay 1

IMM2 LOMEth

EthL2 Switch

Compute Node x86Bay 2

IMM2

Management Network

Data Network

CMM

L2 Switch

Adapter 1 Adapter 2

Compute Node POWERBay 14

FSP Adapter1 Adapter 2

Switch Bay 1Switch Bay 2

Switch Bay 3

Switch Bay 4

In summary … a very capable multi-layered mid-plane design

Syste

m i

nfr

as

tru

ctu

reFlexible networking solution, allowing for best price/performance IBM 10Gb Switch: Wired for up to 16 10Gb ports per node and twenty two external ports

Networking

NIC

Mezz card-1

Mezz card-2

MidplaneCPU-Node

Up to 4 KR PortsPer Switch Bay

NIC

Scalable Switches 5,6




4-port Mezz

4-port Mezz

Uplink Ports

ScS – Bay 1

ScS – Bay 2

(A) (B)

(A) (B)

(A) (B)

(A) (B)

2 going

to 4 per

physical

bay

8 going to 16

ScS – Bay 3

ScS – Bay 4

NIC

Ethernet & FCoE Fibre Channel InfiniBand

– 52 port 1Gb Switch • Base: 14/10

(internal/external)• Upgrade: 14/10• Upgrade: four 10Gb

uplinks– 64 port 10Gb Ethernet Switch

• Base: 14/10• Upgrade: 14/8 (two

40Gb uplink)• Upgrade: 14/4

– 1/10Gb Pass-Thru

– 20 port 8Gb (Qlogic)– 20 port 8Gb Pass-Thru

(Qlogic)– 12 port 16Gb (Brocade)– 24 port 16Gb w/ ESB

(Brocade)

� QDR Switch (Mellanox)– Upgrade: FDR

– 4 port 1Gb (Broadcom)– 4 port 10Gb (Emulex)– 2 port 10Gb (Mellanox)

– 2 port 8Gb (Qlogic)– 2 port 8Gb (Emulex)– 2 port 16Gb (Brocade)

� QDR & FDR Adapter (Mellanox)

Sw

itch

Ad

ap

ter

FlexChassis I/O Portfolio view

� IBM 10Gb Scalable Switch for IBM Flex System Chassis

� One, two, or three 10G ports per server – selectable by software

license

� Base Switch: 14 x 10Gb server port and 10 x 10Gb uplinks

� Switch upgrade 1: 28 x 10Gb server ports and 16 x 10Gb

uplinks. 40Gb uplinks enabled.

� Switch upgrade 2: 42 x 10Gb

server ports and 22 x 10Gb uplinks. 40Gb uplinks

enabled.

� 1.28 Tbps – first 1Tbps+

blade switch

� 5+ Tbps per chassis

IBM Flex System 10Gb Virtual Fabric Scalable Switch

Two 40 G uplink ports. Each port can also be converted to 4*10G

using QSFP to SFP+ cable

Full featured, Scalable bandwidth bringing convergence and simplicity to Datacenter

applications

The Compute power in the Chassis

Syste

m i

nfr

as

tru

ctu

re

Compute

Diverse offerings to match the diverse workloads

System Portfolio tuned to

workloads

◊

Reduce acquisition costs

through virtualization

consolidation

◊

Maximum platform capability

provides deployment

flexibility

IBM Flex System x220

IBM Flex System p260

IBM Flex System p24L

IBM Flex System x240

IBM Flex System p460

IBM Flex System x440 Compute Node

IBM Flex System PEN (PCIe Expansion Node)

• Standard Compute Node Form Factor

• 2-Socket Sandy Bridge-EP (Xeon E5-2600 Series Processor) 135W, 130W, 115W, 95W, 80W, 70W, 60W (4 – 8 core)

• 24 LP DDR3 DIMMs / Up to 1600MHz• Up to 384GB Memory capacity with 16GB RDIMMs• Up to 768GB Memory capacity with 32GB LR-DIMMs

• Chipset – Patsburg B (Intel C600 Series)

• 2x2.5” Hotswap SAS/SATA/HDD/SSD

• Onboard ServeRAID H1135 SAS Controller (RAID 0, 1) 6Gbs + Optional ServeRAID M5115 (RAID 0,1,5,6,10,50) with Flash-to-cache, supports eight 1.8inch SSDs

• SAS or SATA drives• LSI 2004 connects to Patsburg with x4 PCIe Gen2

• Emulex BE3 10Gb Dual Ethernet LOM using Periscope connector

• LOM supports iSCSI and FCoE with FoD (feature On Demand) option

• Models with LOM and LOM-less

• Up to 2 Fabric Mezzanine Cards (PCIe Gen3 – 8GTs capable)• Each Mezz connector: 1x16 PCIe port and 1x8 PCIe port• Installing Mezz 1 card requires removal of LOM Periscope

connector

• uEFI / IMMv2 / TPM 1.2 Rev 1.03

• Power Management• xSmartEnergy Control

• Capping w/Pstate and SCI, Sys Power Maximizer

• Embedded Hypervisor• ESXi on Flash key option (2 USB Keys for redundant

boot option supported on unique DIMM air baffle)

• Front panel – one USB connector and one dongle• Dongle includes two USB, video, and one serial port• Front panel connector does not have enough power to

run external USB drives

• Management• iMM V2 Management Controller• RTMM for Power Exec and Power Sequencing

x240 Compute Node Overview

PC

Ie

x1

LP

C

US

B

iMM v2

QPI Links

DDR3 DIMMS

IntelSandyBridge

EP Processor 2

DDR3 Bus G

DDR3 Bus H

DDR3 Bus E

DDR3 Bus FDDR3 DIMMS

DDR3 DIMMS

DDR3 DIMMS

DDR3 DIMMS

DDR3 Bus C

DDR3 Bus D

DDR3 Bus A

DDR3 Bus BDDR3 DIMMS

DDR3 DIMMS

DDR3 DIMMS

NC (8 PCIe lanes +

unused x4 ESI)

x4 ESI

Cn

tl, M

isc

HS HDD/SSDs

PatsburgSSB

(C600 Series)

LSI 2004SAS

Management to

Midplane

x16 PCIe Gen3

x16 PCIe Gen3

IntelSandyBridge

EP Processor 1

FRM or

PME/SME Interface

ETH Out

ETH Out

PCIe x2

Internal

USBUSB

USB(2)

Dual KR to Midplane

x4 PCIe

Gen2

x8 PCIe Gen3

Mezz

0

10Gb

LOM

ETH In

ETH In

Video (RGB)

Front

USB

Front KVM

Dongle

US

B(2

)

Mezz1 Mezz2

PHY

PHY

Serial Port

USB

QPI 1.1 – 8GT/sDDR3 Ch – up to 1600MT/s

PCIe Gen3 – 8GT/s

SPI

TPM1.2

SystemBIOS16MB

SPI ParallelExp.

NAND Flash

2GBpbDSA

iBMC Flash

8MbitBoot

RTMM

Power Seq.Power Exec

DDR3Video

256MB

I2C

I2C

LPC

LPC

TPMIMM

x8 PCIe Gen2

QPI – 8.0GT/s

16GB/s/dir

16GB/s/dir

8GB/s/dir

4GB/s/dir

DDR3 – 1600MT/s/ChDDR3 – 1600MT/s/Ch

Pop/No-Pop

for FRM

X240 Compute Node Block Diagram

Flex x86 Node Relative Performance

Syste

m i

nfr

as

tru

ctu

reStorage Node Design Overview

Storage

� In Chassis Enterprise Storage

�Double high / double wide node

�Dual hot swappable controllers for HA

�Shared storage for multiple CPU nodes

�High performance/ high function storage

�Midrange class performance

�FCoE & iSCSI block storage

�FC block storage as well

�NAS file storage (future)

�24 HDDs/SSDs – 2.5” hot swap disk trays

�Via dual SAS paths to both controllers

�Add’l storage via Storage Disk Expansion

�Automated FW for hot spot data migration

�Migrate content between SSDs and HDDs

�Chassis and Management Appliance Integration

�Tight integration with chassis mgmt

�Tight integration for storage and fabric mgmt

�VM provisioning, copy svcs, mirroring, etc

34

Enclosure Internal layout

Full Wide

24, 2.5” HDD Bays

ControllerFull Wide

Double High

24, 2.5” HDD Bays

Controller

Node Canister

Enclosure

Hard Disk Bay

HIC card

Management of the hardware

Syste

m i

nfr

as

tru

ctu

re

Management

Simplified Virtualization Management experience

Ph

ysic

al

Vir

tua

lW

ork

load

PhysicalConsolidation

& Setup

ManagementIntegration

ResourceUtilization

ResourcePooling

IntelligentAutomation

Choice of Hypervisors

� Automated virtual machine placement

� Dynamic allocation of virtual server,

storage and network resources

� Network QoS

� Virtual workload definition

� Placement services and advisors

� Pooling of all network switch resources to

enable consistent network policy application

� Monitors VM network utilization within the fabric � Virtual Machine relocation for compute, storage

and networking� Monitors the Network Fabric for congestion

FSM Flex System Explorer

�Dashbar �Explorer�Hardware View

Chassis only, otherwise Explorer shows a table

�Rich Menus

�Finder

�Status Pods�Framework Page

The Cloud software stack

IBM SmartCloud Entry solution architecture, concept

Power Systems Reference Configurations

IBM Systems

Director 6.2

PowerVM ESXi

VMControl 2.3.1 VMware

vCenter

VMControl API vCenter API

Cloud administration and management

TivoliProvisioning

Manager for

Images 7.1.1

REST API

Self-service UI

IBM SmartCloud Entry management stack

Customer

integration• Approval policy,

project management,

users and roles

• Events and auditing

• Metering

• Image library

Common functionality Power Systems only System x only

IBM Systems

Director 6.2

System x Reference Configurations

Network sample



IBM Systems

Director 6.2

PowerVM ESXi


vCenter



TivoliProvisioning

Manager for

Images 7.1.1

REST API

Self-service UI


Customer


project management,

users and roles


• Metering

• Image library


IBM Systems

Director 6.2


vCenter API

The Vix API is object-based. Most API functions either create objects or operate on the

properties of existing objects.

Client applications reference Vix objects with handles. Handles are opaque identifiers (actually

integers) that can be passed as parameters to functions. Handles are run-time only and are

unique only within a client's address space.

Most functions in the C-language API take a handle as a parameter. Because a handle value

represents an object to the API, this document uses the terms "handle" and "object"

interchangeably.

There are several handle types, but a few of the key types are:

– Virtual Machine -- A single virtual machine, which might or might not be powered on.

– Host -- A single host computer, either the local host or a remote host.

– Job -- An object used in managing asynchronous operations.

– Snapshot -- A snapshot of a virtual machine.

IBM SmartCloud Entry solution architecture,concept


IBM Systems

Director 6.2

PowerVM ESXi


vCenter



TivoliProvisioning

Manager for

Images 7.1.1

REST API

Self-service UI


Customer


project management,

users and roles


• Metering

• Image library


IBM Systems

Director 6.2


REST API reference

IBM SmartCloud Entry provides the following services

for appliance libraries.� GET /appliances: Retrieve appliances from the cloud.

� GET /appliances/{id}: Retrieve specific appliance from the cloud.

� PUT /appliances/{id}: Update specific appliance in the cloud.

� DELETE /appliances/{id}: Delete a captured appliance that has failed.

� GET /appliances/{id}/targets: Retrieve the targets that can handle a workload of this appliance.

� GET /appliances/{id}/customization: Retrieve default customization available when deploying this

� appliance.

� PUT /appliances/{id}/customization: Update the default customization for this appliance (admin

� service).

� DELETE /appliances/{id}/customization: Resets the customization for this appliance.

� GET /appliances/{id}/log: Retrieve a appliance's capture progress logs.

� POST /appliances: Create a new appliance by taking a workload capture.

The representational state transfer (REST) application programming interface (API) is

provided by SmartCloud Entry.



IBM Systems

Director 6.2

PowerVM ESXi


vCenter



TivoliProvisioning

Manager for

Images 7.1.1

REST API

Self-service UI


Customer


project management,

users and roles


• Metering

• Image library


IBM Systems

Director 6.2


49

IBM SmartCloud Entry: Comprehensive cloud capabilitiesAdministrator Functions

• Configure workload to host

• Configure workload targets to system pool

• Configure Virtual Appliance Parameters

• Register Virtual Appliance for User Selection

• Approve/Reject New Workload Requests

• Approve/Reject Workload Resize Requests

• Configure Billing

• Charging accounts, account assignment

• Configure approvals

• Configure to generate metering records

• Create & Manage Projects

• Add users to projects as admin, deployer, viewer

• Create Users

• Configure Network Pools

• Configure LDAP environment

• Cloud Configuration to VMControl or VMware

• Manual Intervention

• Review event logs & failures

• Initiate workload movement (between projects or

via virtualization manager)

End User Functions

• Request access to a project

• Request CFS access from login

panel

• Request workload creation –

appliance deploy

• Set user instance parameters (CPU,

memory)

• Resize Running Workloads

• Delete a Workload

• Clone a workload

• Start/Stop a workload

• Review workload properties

• Add additional disk (VMControl)

• Set target disk size at deploy

(VMware)

IBM SmartCloud Entry delivers cloud experience to users

� Easy to access, easy to use Service Request Catalog

� Hides underlying infrastructure from user and shifts focus to services delivered

� Enables the ability to provide standardized and lower cost services

� Facilitates a granular level of services metering and billing

� Workload standardization eases complexity

The IBM SmartCloud Entry end-user self-service scenario

End Users Service Portal

Service Request Catalog

Provisioning Engine� Workflows� Expert Systems� Scripts

Service Modules � Metering/Usage� Billing� Approvals

Virtualized Cloud Infrastructure

� Easy to access, easy to use Service Request Catalog

� Hides underlying complex infrastructure from user and shifts focus to services provided

� Enables the ability to provide standardized and lower

cost services

� Facilitates granular level of services metering and billing

� Workload standardization eases complexity

� Self-service Web UI

� Basic usage metering

IBM SmartCloud Entry delivers cloud benefits to managers

Wrap up



IBM Systems

Director 6.2

PowerVM ESXi


vCenter



TivoliProvisioning

Manager for

Images 7.1.1

REST API

Self-service UI


Customer


project management,

users and roles


• Metering

• Image library


IBM Systems

Director 6.2


ISV and IBM Patterns

SmartCloud Provisioning

Compute

Network

Storage


SmartCloud Entry

Compute

Network

Storage


SmartCloud Entry

Pre-NGP orNon IBM HW

Flex System PureFlex PureApplication

Compute

Network

Storage

Compute

Network

Storage

Pre-integrated entry point Upgrade opportunity

Platform and Infrastructure Management

SmartCloud Orchestration

SmartCloud Orchestration




ISDM or SmartCloud

Orchestration

Orchestrates

Foundation Patterns

Links

� IBM SmartCloud Entry Documentation

� Representational state transfer - REST

� Redbook - Implementing SmartCloud Entry on IBM PureFlex System

� Redbook - ReadyPack for Cloud with Hyper-V on IBM Flex System

QUESTIONS ?

Ib F. KristensenIT Architect

Nordic PureFlex Product Manager

Systems & Technology Group

Bytoften 1

8240 Risskov

+45 2880 6217

[email protected]

Thank You

dtu – distributed systems · spi tpm1.2 system bios 16mb spi parallel exp. nand flash 2gb pbdsa...

Documents