ethernet fabric technology for hyperscale data centre · in the physical network • stp disables...

Ethernet Fabric Technology for Hyperscale Data Centre

Jason Lin / 林奇志 Senior Technical Consultant

What’s Changing in Data Center Networks?

1/15/2013 © 2012 Brocade Communications Systems, Inc. CONFIDENTIAL—For Internal Use Only 2

DATA CENTER NETWORK

VM Density and Mobility

East-West Traffic Patterns Flash Storage

Software-defined Networking

Active/Active Data Centers

Data Center Leadership Strategy


FIBRE CHANNEL FABRICS

ETHERNET FABRICS

CLOUD EXTENSION

SOFTWARE-DEFINED NETWORKING

1/15/2013 © 2012 Brocade Communications Systems, Inc. CONFIDENTIAL. FOR INTERNAL USE ONLY 4

Data Center Challenges Today AND REQUIREMENTS FOR TOMORROW

Underutilized Network Resources

Explosive Growth on Virtual Machines

Rip and Replace to Support New Tech Iss

ues

More Efficient and Automated Networking

Increased Scalability and Performance

Flexibility to Support New Technologies Re

quire

men

ts

2009

Newly deployed Virtual Machines (VMs) Newly deployed physical hosts

The Tipping Point

Source: IDC

Building Next Generation Data Center Networks Some Key Design Considerations

© 2012 Brocade Communications Systems, Inc. CONFIDENTIAL—For Internal Use Only 5 1/15/2013

Performance & Scalability

•Linear Scale to meet growing elasticity

demands of today’s virtualized Data Center

environments

•Multiple Layer 3 Gateways for optimal Layer 3

forwarding with for HA, Resiliency and Network

Scaling

•Multiple Layer 2 Paths with Active-Active

capability to support growing east-west

bandwidth

•Efficient Load Balancing on LAG channel

member links for perfect load distribution

•Low Latency (Fewer hops) for predictable and

better Application Performance

Building Next Generation Data Center Networks Contd.

Network Provisioning & Management

•Simplified provisioning with reduced complexity

•Efficient management

Some Key Design Considerations

© 2012 Brocade Communications Systems, Inc. CONFIDENTIAL—For Internal Use Only 6 1/15/2013

Storage and Other Traffic Classes

•Lossless traffic classes (DCB) for Bulk TCP Traffic

•Integrated Network & Storage over 10GbE links

(iSCSI or FCoE)

Advanced Capabilities

•Tight Integration with Server Virtualization

•Support Emerging SDN Architecture

1/15/2013 7 © 2011 Brocade Communications Systems, Inc. Proprietary Information

Cloud

Ethernet Fabrics

Server Virtualization Pools of Compute and Storage

Resources Dedicated to Applications

A Network That Dynamically Meets the Needs of Applications

User Benefits Quicker response to: • Needs • Requests • Concerns

ETHERNET FABRICS Foundation for the Cloud

Shared pool of resources that can be

dynamically allocated to users

Business Benefits Increased: • Business agility • Fiscal responsibility

Core

Ag

greg

atio

n

Servers with 1 or 10 Gbps Connections

Edge

Scalability

Acce

ss

Ethernet Fabric Architecture

1/15/2013 © 2011 Brocade Communications Systems, Inc. Proprietary Information 8

Classic Hierarchical IP/Ethernet Architecture

Client/Server Flows (North-South)

80%

Server/Server Flows (East-West)

Server/Server Flows (East-West)

20%

–Gartner

“By 2014, 80% of networking traffic will be between servers.”


Scaling Virtual Server Environments Technical Challenges Today

© 2011 Brocade Communications - Company Proprietary Information 10

Layer 2: only 1 active path STP disables other paths

40% of Links Not Used Increase utilization using MSTP (spanning tree per VLAN)

Increases complexity Creates multiple single-path networks; limits sphere of mobility

Poor application response during link failure STP reconvergence – network is down Broadcast storms stress network

Layer 3 as an alternative Greater complexity; higher cost VM mobility limited to rack

TRILL-Based Networking

Layer 2: multiple active paths • TRILL calculates multiple routes

100% of links used

ECMP handles even distribution • ASIC-level frame-based load balancing

Link failure • Within a trunk triggers no reconvergence • Total path failure has very quick recovery

TOR Layer 3 complexity NOT needed • Simple flat Layer 2 networks • Fast low-latency switching in hardware

11

Transparent Interconnection of Lots of Links

© 2011 Brocade Communications - Company Proprietary Information

Brocade Virtual Cluster Switching

© 2010 Brocade Communications Systems, Inc. Company Proprietary Information 12 1/15/2013

Virtual Cluster Switching

© 2011 Brocade Communications Systems, Inc. 13

Logically flattens and collapses network layers Scale edge and manage as if single switch Auto-configuration Centralized or distributed mgmt; end-to-end

Self-forming Arbitrary topology Fabric is aware of all members, devices, VMs Masterless control, no reconfiguration VAL interaction

No Spanning Tree Protocol Multi-path, deterministic Auto-healing, non-disruptive Lossless, low latency Convergence-ready

Ethernet Fabric

Distributed Intelligence

Logical Chassis

Connectivity over Distance, Native Fibre Channel, Security Services, Layer 4-7, etc. Dynamic Services

CORE TECHNOLOGY

Brocade Virtual Cluster Switching (VCS)

• First data center Ethernet fabric

• No Spanning Tree Protocol

• Multi-path, deterministic

• Auto-healing, non-disruptive

• Lossless, low latency

• Built for convergence

© 2010 Brocade Communications Systems, Inc. Company Proprietary Information 14

NAS iSCSI FCoE

ETHERNET FABRIC

DISTRIBUTED INTELLIGENCE

LOGICAL CHASSIS

1/15/2013

Ethernet Fabrics A New Network Architecture

© 2010 Brocade Communications Systems, Inc. 15

• Classic architectures often require three tiers in the physical network

• STP disables links in the fabric to prevent loops, limiting network utilization

• Each switch has to be managed individually

• Fabric architectures flatten and seamlessly scale out the Layer 2 network at the edge

• All links in the VCS fabric are active and it is managed as one

• Switches in the VCS fabric are managed at one

Adding a new switch to an Ethernet Fabric Automatic Fabric Creation and Expansion & Automatic Trunk Creation


30GbE DCB Trunk (3x10GbE)

10GbE DCB Link

RB1 RB2

RB3

RB1 RB2

RB3

Ethernet Fabrics Equal Cost Multi-Pathing

17

ESX2 ESX1

30Gb Trunk = 3 x 10Gb Links

100%

75% 25% 25% 25% 25%

33% 33% 33%

50% 50%

100%

Traditional Link Aggregation Switch-to-Switch LAG – Hashing algorithms cause imbalances


70% 20% 30%

Brocade Advantage Virtual Link Aggregation (ISL Trunking) Balanced Connections with no hashing algorithms


40% 40% 40%

VCS Unique Advantage: Frame-Based Trunking

© 2012 Brocade Communications Systems, Inc. Company Proprietary Information

• High link utilization and ease-of-use • All 10GE ports are not alike

• Frame-level, hardware-based trunking at Layer 1 • Near 100% link utilization versus 802.3ad

LAG groups ~50-60% link utilization

• Single flows can be split across all links

• Frames are evenly distributed across links in the trunk

• Built into Brocade fabric switching ASIC

• Self Forming ISL Trunks

80 Gbps

1/15/2013 20

~50 Gbps

Brocade ISL Trunking (8 links active)

802.3ad Link Aggregation (8 links active)

= 10GE link, width represents utilization

Brocade Trunking is highly differentiated

Frame-based trunking at Layer 1

Flow-based trunking at Layer 2

RB1 RB2

RB3

Ethernet Fabrics Host Based vLAG – vLAG to different physical VDX Switches

21

ESX2 ESX1

Link Aggregation

Link Aggregation

vLAG Proviosing Similar to LAG

1/15/2013 © 2010 Brocade Communications Systems, Inc. Company Proprietary Information 22

VIRTUAL MOBILITY Automatic Migration Port Profiles

RB1 RB2

RB3

24

MAC ID 2 MAC ID 2

MAC ID 2

MAC ID 1

ESX2 ESX1

MAC ID 2

MAC ID 1

MAC ID 1

MAC ID 1

Intelligence Is Always Shared Throughout Fabric Fabrics are self-healing—re-routing happens automatically Distributed MAC address tables—384k+ per fabric

CORE TECHNOLOGY

Brocade Virtual Cluster Switching (VCS)

• Fully distributed control plane

• Arbitrary topology, self-forming

• Network-wide knowledge of all members, devices, VMs

• Automatic Migration of Port Profiles (AMPP)


ETHERNET FABRIC

DISTRIBUTED INTELLIGENCE

LOGICAL CHASSIS

NAS iSCSI FCoE 1/15/2013

Brocade VM-Aware Network Automation Migration Dynamic configuration and secure communication


No need for manual configuration of MAC addresses and port profiles; less error-prone

Minimizes procedural delays between server and network IT teams

Eases configuration of multiple VCS fabrics

Protection against VM/MAC spoofing via secure vCenter communication

Brocade Network Advisor

NAS iSCSI FCoE FC iSCSI InfiniBand

vCenter

Sharing Port Profiles Automatic Sharing to simplify management


Port Profile WebServer: Enable QoS Enable VLAN Enable Security Enable FCOE

Simplified Virtual Machine Migration Automatic Migration of Port Profiles

28

ESX 2

ESX 1

MAC ID MAC ID

MAC ID

© 2011 Brocade Communications - Company Proprietary Information

MAC ID MAC ID MAC ID MAC ID MAC ID

MAC ID

Hypervisor Agnostic

Tied only to VM Mac Addresses

29

SOLUTIONS PORTFOLIO

The Power of Open Solutions


NETWORK

SERVER

HYPERVISOR

STORAGE

BROCADE VCS ARCHITECTURE

Hyper-V

iSCSI NAS FC FCoE

1/15/2013

Scaling Data Center Networks With Brocade VCS Fabrics Architecture


Scaling Data Center with VCS Fabrics Linear Scalability with VDX Switches


VCS Fabric with VDX Switches – Flexible Design Choices

Small Scale: 24 – 500 Ports Use Fixed Port Configuration (VDX-6710, VDX-6720 and VDX-6730) TOR Switches with desirable N/W oversubscription Medium Scale: 500 – 2000 Ports Use Mixed configuration (Fixed Port Configuration + Modular Platform) with desirable N/W oversubscription Large Scale: 2000+ Ports Use Modular Platforms (VDX-8770-4 and VDX-8770-8) to build scale out Fabrics with desirable N/W oversubscription

Net

wor

k Sc

ale

100 Ports 4,000 VM

2,000 Ports 80,000 VM

500 Ports 20,000 VM

8,000+ Ports 320,000+ VM

Brocade VDX6K Series

Brocade VDX6K Series

Brocade VDX8K Series Brocade VDX8K Series

Small Scale– ToR (Fixed)

Medium Scale - ToR + Modular (Mixed)

Large Scale - Modular Platform

Design Benefits of VDX Switches • Linear Scalability for Growth • Configurable Oversubscription • Optimized for East-West traffic

Brocade VDX6K and VDX8K Series Brocade VDX6K and

VDX8K Series

Assumptions: 2 ports per server, 80 VMs per server

Compute Scale

Brocade VDX Product Family The Flexible Choice for the Evolving Data Center

Ideal for every stage of network evolution

Ultra low latency for unmatched performance

Superior size and power efficiency critical for today’s data center

Flexible storage connectivity for iSCSI, NAS and FCoE

Brocade VDX 6710 Switch




10 GbE Fixed Port Data Center Portfolio

Brocade VDX 6710 Brocade VDX 6720 Brocade VDX 6730

Physical Description 1 SKU • 48 Gig copper + 6 10G SFP+ ports

2 SKUs • 24 port 10G SFP+ ports • 60 port 10G SFP+ ports

2 SKUs • 24 port 10G SFP+ and 8 8G FC • 60 port 10G SFP+ and 16 8G FC

Licenses • VCS License • POD License • VCS License • FCoE License

• POD License • VCS License • FCoE License

Positioning

High Performance, fully nonblocking, VCS enabled 1 GbE switch providing cost-effective connectivity to 1G servers

High performance, low latency, VCS-enabled 10 GbE switch targeted at virtual data centers. Enables end-to-end LAN/SAN convergence.

High Performance, low latency VCS enabled convergence switch with FC connectivity. Enables FC, FCoE and iSCSI solutions with VCS fabric

Use Cases • 1 GbE ToR server access • Standalone and as part of VCS

fabric

• 10GbE ToR, L2 agg. • Large-scale virtualization • LAN/SAN Convergence

• First hop TOR connectivity to FC • Multihop FCoE with FC

connectivity

VDX 6710 VDX 6720 VDX 6730


Introducing Brocade VDX 8770 Modular Switch for Next Generation DataCenters

• Simplicity and Automation • VCS Fabric Technology! • Supports 1G/10G/40G

• Scalability and Performance • Backplane scales to 4Tbps per slot • Best in class 3.5 microseconds any-to-any latency • Multilayer multipathing for reliability and efficiency • Best-in-class power efficiency

• Built to Last • 100G-ready • Scales from 12 ports to 8,000+ ports per fabric • Hardware-enhanced network virtualization*

* h/w ready, some features to be enabled post-GA 37 © 2012 Brocade Communications Systems, Inc. CONFIDENTIAL. FOR INTERNAL USE ONLY

Data Center Access/Aggregation

VDX 8770-8 Chassis Design

39

Switch Fabric 6 half slot SFM

Interface Module 1 GbE, 10 GbE and 40 GbE Linecards

Modular Cooling System Front to Back Airflow

Management Modules 1:1 Redundancy

8 GB of DDR III SDRAM N+1 / N+N

Power supply redundancy (AC & DC)

Interface Module 1 GbE, 10 GbE and 40 GbE Linecards

© 2012 Brocade Communications Systems, Inc. CONFIDENTIAL—For Internal Use Only 1/15/2013

Improve Network Utilization MULTIPATHING IN MULTIPLE LAYERS

© 2012 Brocade Communications Systems, Inc. CONFIDENTIAL. FOR INTERNAL USE ONLY 42

HOW IT WORKS

1. Layer 3 GW multipathing (in red) is achieved via multiple active L3 gateways

2. Layer 2 multipathing (in blue) is performed with industry standard TRILL.

3. Layer 1 multipathing (in green) is achieved via Brocade Trunking, providing industry’s best load balancing across a trunk group

L3 gateway resiliency

Maximum Network Utilization

Automated Load Balancing

User Benefits

40% 40%

40%

Industry First

1/15/2013

Configuration simplicity – what do we

mean?

Traditional

Active L3 router Standby L3 router

•Step 1: Ensure L2 Connectivity (STP) •Step 2: Create Vlans. •Step 3: LACP/Trunk Configuration •Step 4: Even-odd vlan load balancing Repeat N times for N switches!

Reduces network config from 4xN steps to only 1 step!

……

VCS Fabric

•Step 1: Ensure L2 Connectivity(no STP, auto L2MP) •Step 2: Create Vlans •Step 3: LACP/Trunk Configuration(auto ISL/BTRNK) •Step 4: Even-odd vlan load balancing (auto) Repeat N times for N switches!

Tunneled Frame Format

Flexibility to support new technologies Regain visibility into tunneled traffic

1/15/2013 44 Brocade Confidential.

What is Network Virtualization? A framework for overlaying virtualized Layer 2 networks over Layer 3 networks

Why Do Network Virtualization?

• Network isolation for security, multi-tenancy • Deliver workload-specific network services • Better VM mobility, virtual asset utilization

Challenges in Tunneled Environments: • VxLAN/NVGRE/other protocol tunnels traffic

from one server to another • Traditional networking devices lose visibility

into tunneled flows • Existing networking policies start breaking

Brocade VDX 8770 Benefits: Regain network visibility, control of tunneled traffic • Protocol-agnostic • Future-proof implementation based on flexible,

programmable hardware • Line-rate services consistently applied

Customer Ethernet Header

Ether Payload

VXLAN/ Tenant

Network ID (24 bit)

Outer Ethernet Header

VDX8770 is hw ready. Software support planned post-GA

VXLAN example

1st VxLAN aware

Switch

52 byte ~ 56 byte

Brocade VDX Product Family NETWORK SWITCHES FOR BROCADE VCS FABRICS

1/10/40 GbE server connectivity Native Fibre Channel for SAN integration Unmatched simplicity with automated

configuration Ultra low latency and power consumption Industry’s lowest entry-level cost for

Ethernet Fabrics

1/15/2013 45 © 2012 Brocade Communications Systems, Inc. Proprietary Information

VDX 6710

VDX 6730-60 VDX 6720-60

VDX 6730-24 VDX 6720-24

VDX 8770-8

VDX 8770-4

Brocade Solution for Virtualized Data Center Today and Future


Long Distance vMotion Architecture Today

© 2011 Brocade Communications Systems, Inc.

47

Brocade FC SAN

WAN

Stretched L2 Network with MPLS/VPLS

Brocade Extension (FCIP)

Brocade Extension (FCIP)

Headquarters Remote DC

Latency <5 ms OC-12 Connection

Brocade FC SAN

vSphere Servers

vSphere Servers

Brocade Edge Brocade

Core

Brocade Core

Brocade Edge

VM is still talking to its storage at Headquarters

Can then trigger a Storage VMotion to Remote Datacenter

© 2012 Brocade Communications Systems, Inc. Proprietary Information

Extending Clouds Across Data Centers GLOBAL SERVER LOAD BALANCING AND SEAMLESS VM MIGRATION

Internet

Data Center A

MLX

Servers

Fibre Channel SAN

Storage

Data Center B

SAN Extension

VMware vCenter

Brocade ADX

MLX

Servers

Fibre Channel SAN

Storage

SAN Extension

Brocade ADX

Brocade ADX

WAN

GSLB Controller

ES ES

APP

OS

STORAGE

Virtualized Servers

Dedicated Fibre Channel SAN for Tier 1 applications

Core Routers

FCoE/iSCSI/NAS Storage

VM

SAN

VM VM

VM

VM

VM VM

VM

VM

Public Network

True Active/Active Datacenters

VCS Fabric Extension

Layer 4-7 Application Delivery

Security Services (firewall, encryption)

Native Fibre Channel

VCS Fabric Extension

PRIMARY DATACENTER

EXTENDED DATACENTER

VM

VM

© 2012 Brocade Communications Systems, Inc. Proprietary Information

ethernet fabric technology for hyperscale data centre · in the physical network • stp disables...

Documents