interop: the 10gbe top 10

11

Shaun WalshEmulex

The 10GbE Top 10

2

Discrete Data Center

10G Ethernet and FCoE are enabling technologies to support the Virtual Data Center and Cloud Computing.

The Data Center of the Future

Virtual Data Center

Cloud Data Center

• Cloud Computing (Private & Public)• On-Demand Provisioning and Scale• Modular Building Blocks “Legos”• Avoid CAPEX & OPEX

Private cloud

• 3 Discrete Networks• Equipment Proliferation• Management Complexities• Expanding OPEX & CAPEX

• Converged Networks• Virtualized• Simplifies I/O Management• Reduces CAPEX & OPEX

3

Drivers of the Move to 10/40GbE

Storage

Universe

• 7ZB by 2013

• Mobile and VDI

• Device-Centric

o

Virtual

Networking

• VM I/O Density

• Scalable vDevices

• End-to-End vI/O

Cloud

Connectivity

• New I/O Models

• I/O Isolation

• New Server Models

Network

Convergence

• Multi-Fabric I/O

• Evolutionary Steps

• RoCEE Low Latency

4

VM Density Drives More I/O Throughput

© 2011 Enterprise Strategy Group

0%

5%

10%

15%

20%

25%

30%

35%

40%

26%

36%

23%

12%

3%

8%

24%

30% 31%

7%

What would you estimate is the average number of virtual machines per phys-ical x86 server in your environment today? How do you expect this to change

over the next 24 months? (Percent of respondents, N=463)

Today 24 months from now

<5 5-10 11-25 >25

Server Virtualization Impact on the

Network

It has created more network traffic in the data center

(30%)

10GbE & 16 Gb FC

40 Gb

Don’t Know

5

ESG’s Evolution of Network Convergence

© 2011 Enterprise Strategy Group

Curr

ent L

evel Dedicated

NetworksOrganizations keep LAN, SAN and HPC clusters on their own separate networksSeparate management tools and teamsUnique skills and training required

Prog

ress

ing

Leve

l Consolidated NetworksStarts with Ethernet – run LAN, IP storage and HPC together Maintain existing investment in FCConsolidate connectivity at the server and storage level“Convergence Ready”

Adva

nced

Lev

el Fully Converged NetworksMerge on to a single fully converged networkRun LAN and all storage over Ethernet -FCoEConverged adaptors, cabling and switches

6

Evolving Network Models

Target ConnectHost Connect

DiscreteNetworks

ConvergedFabric

Networks

ConvergedNetworks

7

Emulex Connect I/O Roadmap

Ethernet HighPerformanceComputing

Unified Storage

2010 2011 2012 2013

Multi-Fabric Technology

Low Latency RoCEERDMA

Value AddedI/O Services

I/O Management

Networked Server/ Power Management

16Gb

Converged Networking

UniversalLOMs

10Gb

10GBaseT10Gb 40Gb 100Gb

3rd Gen BMC

32Gb

PCIe Gen3

SR-IOV, Multichannel

8Gb Fibre Channel

40Gb10GBaseT

4th Gen BMC

8

Waves of 10GbE Adoption

10Gb LOM and Adapters for Blade Servers50% of IT Managers Cite Virtualization as the Driver of 10GbE in Blades

10G

bE A

dopt

ion

Wav

es

2010 2011 2013 Beyond2012

1-2 Socket Serverwith NICs and Modular LOMs

Cloud Container Servers for Web Giants and Telcos

FCoE Storage 25% of I/O10GbE LOM Everywhere

Sources:

Dell’Oro 2010 Adapter Forecast

ESG Deployment Paper 2010

IDC WW External Storage Report 2010

IT Brand Pulse 10GbE Survey April 2010

10Gb NAS and iSCSI Drive IP SAN Storage for Unstructured Data

2-4 Socket X86 and Unix Rack Servers Drive 10Gb with Modular LOMs and UCNAs

Wav

e 1

Wav

e 2

Wav

e 3

9

Wave 2 of 10GbE Market Adoption

65% of Server Shipments

More that 2X the Revenue Opportunity

Romley from Q4:11 to Q2:11

Modular LOM on Rack

More Cards vs. LOMs

x86 & Unix Rack Servers

Wave 2 – More Than Doubles 10GbE Revenue Opportunity CY12/13

Cisco, Juniper, Brocade announced Multi-Hop FCoE in last 90 days

10GbE NAS, ISCSI & FCoE Convergence

10GbE IP Storage

Next Gen Internet Scale Applications (Search, Games & Social Media)

Container & High Density servers migrating to 10GbE

Growing to 15% of Servers

Web Giants

10

The 10Gb Transition – Cross Over in 2012

2009 2010 2011 2012 2013 20140%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 Gbps 10 Gbps

Serv

er C

onne

ction

Sha

re

Source Dell Oro Group

11

#1) Switch Architecture Top of Rack

Modular unit design – managed by rack

Servers connect to 1-2 Ethernet switches inside rack

Rack connected to data center aggregation layer with SR optic or twisted pair

Blades effectively incorporate TOR design with integrated switch

FIBER FIBER FIBER

SANEth Aggregation

SERVER CABINET ROW

12

Switch Architecture End of Row

Server cabinets lined up side-by-side

Rack or cabinet at end (or middle) of row with network switches

Fewer switches in the topology

Managed by row – fewer switches to manage

SANEth Aggregation

SERVER CABINET ROW

FIBER FIBER FIBER FIBER

COPPER

13

#2) 10Gb LOM on Rack Servers

Generally more I/O & memory expansion

More expensive 10G physical interconnect

Later 10G LOM due to 10GBASE-T

Typically limited I/O expansion (1-2 slots)

Much cheaper 10G physical interconnect – backplane Ethernet & integrated switch – leading 10G transition

Earlier 10G LOM

VS.

14

#3) Cables

Today’s Data Center– Typically Twisted Pair for

Gigabit Ethernet

• Some Cat 6, more likely Cat 5

– Optical cable for Fibre Channel

• Mix of older OM1 and OM2 (orange)and newer OM3 (green)

• Fibre Channel may notbe wired to every rack

15

100Mb 1Gb 10Gb

UTP Cat 5UTP Cat 5SFP Fiber

10Mb

UTP Cat 3

Mid 1980’s

Mid 1990’s

Early 2000’s

Cable TransceiverLatency (link)

Power(each side)

DistanceTechnology

Copper

Twinax~0.1ms~0.1W10mSFP+ Direct Attach

Optic

Single-ModeVar1W

10GBase LR(long range)

Optic

Multi-Mode

~01W

10 km

10GBase SR

(short range)

10GbE Cable Options

SFP+ Cu, FiberX2, Cat 6/7

Late 2000’s

10GBASE-TCopper

Twisted Pair

Cat6 - 55mCat 6a 100m

Cat7 -100m

~2.5W (30m)~3.5W (100m)

~1W (EEE idle)

~2.5ms

62.5mm - 82m50mm - 300m

16

3X Cost and 10X Performance

1Gb Copper

10GbDAC

10GbOptical

10GbBase-T

LOM $ 5 $50

NIC/UCNA $118 $337 $690 $250

Switch Ports $87 $531 $531 $350

CAT Cables $15 $132 $25

Switch SFP $924

Optic Cable $100

Total $225 $ 1,000 $ 2,245 $675

17

10G BASE-T Will Dominate IT Connectivity

Crehan Research 2011

18

New 10GBASE-T Options

SFP+ is lowest cost option today– Doesn’t support

existing gigabit Ethernet LOM ports

10GBASE-T support emerging– Lower power at 10-30m

reach (2.5w)– Energy Efficient

Ethernet reduces to ~1w on idle

Optical is only option today

10GBASE-T support emerging– Lower power at 10-30m

reach (2.5w this year)– Can do 100m reach

at 3.5w– Energy Efficient

Ethernet reduces to ~1w on idle

Top of RackTop of Rack End of RowEnd of Row

19

#4) Data Center Bridging

Ethernet is a “best-effort” network– Packets may be dropped– Packets may be delivered out of order

• Transmission Control Protocol (TCP) is used to reassemble packets in correct order

Ethernet is a “best-effort” network– Packets may be dropped– Packets may be delivered out of order

• Transmission Control Protocol (TCP) is used to reassemble packets in correct order

Data Center Bridging(aka “lossless Ethernet”)

– The “bridge to somewhere” – pause-based link between nodes

– Provides low latency required for FCoE support– Expected to benefit iSCSI, enable

iSCSI convergence

Data Center Bridging(aka “lossless Ethernet”)

– The “bridge to somewhere” – pause-based link between nodes

– Provides low latency required for FCoE support– Expected to benefit iSCSI, enable

iSCSI convergence

20

IEEE Data Center Bridging Standards

Priority-based Flow Control (PFC)

Enables lossless Ethernet – manages I/O between initiator and target on a multi-

protocol Ethernet linkIEEE 802.1Qbb

Quality of Service (QoS) Supports 8 priorities for network traffic IEEE 802.1p

Enhanced Transmission Selection (ETS)

Allocate bandwidth to IP, iSCSI and FCoE traffic – managed with OneCommand 5.0 IEEE 802.1Qaz

Data Center Bridging Capability Exchange (DCBX)

Extends DCB network by exchanging Ethernet parameters between DCB switches IEEE 802.1ab

FeatureFeature BenefitBenefit StandardsActivity

StandardsActivity

21

Savings with Convergence

Before Convergence 140 Cables

Before Convergence 140 Cables

IP IP IP IP IP IP IP IP IP IP




FC FC FC FC FC FC FC FC FC FC




After Convergence

60 Cables

After Convergence

60 CablesLOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM

LOM













CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

CNA

(Based on 2 LOM, 8 IP and 4 FC Ports on 10 Servers)

(Based on 2 LOM, 8 IP and 4 FC Ports on 10 Servers)

Savings Up To:• 28% on switches, adapters• 80% on cabling• 42% on power and cooling

Savings Up To:• 28% on switches, adapters• 80% on cabling• 42% on power and cooling

22

Volume x86 systemsVolume x86 systems High End Unix SystemsHigh End Unix Systems

To Converge or Not To Converge?

Best business case for convergence

Systems actually benefit by reducing adapters, switch ports and cabling

Limited convergence benefit

Typically large number of SAN and LAN ports

System may go from 24 Ethernet and 24 Fibre Channel ports to 48 Converged Ethernet ports

Benefits in later stages of data center convergence when Fibre Channel SAN fully running over Ethernet physical layer (FCoE)

23

#7) Select 10Gb Adapter

10Gb LOM becoming standard on high-end servers

Second adapter for high availability

Should support FCoE and iSCSI offload for network convergence

Compare performance for all protocols

24

Convergence Means Performance

40 Gb/Sec

Faster Engine: More Transactions

More VMs/CPU

QOS you demand

Performance for Storage

Wider Track: More Data Lanes

No Virtualization Conflicts

Capacity for Provisioning

24

900K IOPS/Sec

25

#8) Bandwidth and Redundancy

NIC Teaming – Link Aggregation

– Multiple physical links (NIC ports) combined into one logical link

– Bandwidth aggregation

– Failover redundancy

FC Multipathing

– Load balancing over multiple paths

– Failover redundancy

Core Core

Core Core

Redundancy

Redundancy

Poor Den

sity

a b

c d

26

NETWORKFRAMEWORK

LAN SANNETWORKCONVERGENCE

#9 Management ConvergenceF

utu

re P

roof

Inve

stm

ent

Pro

tect

ion

Protects YourConfigurationInvestment

Protects YourManagementInvestment

Protects YourLAN & SAN Investment

SANLAN

27

Convergence and IT Management

Traditional IT management

– Server

– Storage

– Networking

Convergence is latest technology to perturb IT management

– Prior: Blades, Virtualization, PODs

Drives innovation Server/Virtualization

StorageNetworking

28

#10) Deployment Plan

Upgrade switches as needed to support 10Gb links to servers

Plan for network convergence with switches that support DCB

Unified 10GbE platform for LOM, blade and stand-up adapters

Focus on new servers

2929

Shaun WalshEmulex

The 10GbE Top 10

30

Flexible Converged Fabric Adapter Technology

Dual Port 8Gb FC

2x8

Quad Port 10Gb CNA

Dual Port 16Gb FC

Single Port 40Gb CNA

Dual Port 8Gb FCDual Port10Gb CNA

Dual Port 10Gb CNA

Single Port 16Gb FCDual Port 10Gb CNA

Quad Port 8Gb FC

4x82x16

2x10

4x102x8 2x10

1x16 2x10

1x40

31

Emulex at Interop Booth # 743

InteropMay 9, 2011

EMC WorldMay 9, 2011

InteropMay 9, 2011

First 16Gb HBAFibre Channel Demonstration

First UCNA10GbE Base-T Demonstration

First UCNA40GbE

Demonstration

32

Wrap Up

10GBASE-T is Like Hanging Out with an Old Friend

Many Phase to Market Transitions and Implementation

Network Convergence is Coming; Best to be Ready

Management of Domains Will be the Most Challenging

10GBASE-T will drive 10Gb Cost in Racks

33

THANK YOU

For more information, please contact Shaun Walsh949.922.7472 [email protected]

interop: the 10gbe top 10

Technology

opex capex

capex opex

cloud computing private

discrete networks

converged networks

management complexities

io management

future10g ethernet