virtualization in data center - unified fabric with nexus

Upload: marius-diaconu

Post on 02-Jun-2018

224 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    1/97

    2006 Cisco. Al l r ights reserved. Cisco Conf ident ialPresentation_ID 1

    Virtualization inData CenterUnified Fabric with Nexus

    Maciej Bocian

    [email protected]

    Architecture Sales Manager

    Data Center and Virtualization, Central Europe

    CCIE#7785

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    2/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 2

    Traditional Data Center Network Topology

    L2

    VLAN A

    Module 1

    VLAN B

    L3

    VLAN D VLAN EVLAN C

    L3

    L2

    Core

    Aggregation

    Access

    Hierarchical Design

    Triangle and Square Topologies

    Multiple Access Models: Modular, Blade Switches and ToR

    Multiple Oversubscription Targets (Per Application Characteristics)

    2000 10000 Servers

    10,000 to 50,000 ports

    Module 2

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    3/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 3

    New Data Center Architecture

    Topology Layers:Core Layer: Support high density L3 10GE aggregation

    Aggregation Layer: Support high density L2/L3 10GE aggregation

    Access Layer: Support EoR/MoR, ToR, & Blade for 1GE, 10GE, DCE & FCoE attached servers

    Topology Service:Services through service switches attached at L2/L3 boundary

    Topology Flexibility:Pod-wide VLANs, Aggregation-wide VLANs or DC-wide VLANs

    Trade off between flexibility and fault domain

    Agg-wide VLANs

    DC-wide VLANs

    Pod-wide VLANs

    L2

    L3

    L2L3

    SAN Fabric

    Fabric AFabric B

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    4/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 4

    Physical facilities,

    cabling andstandards

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    5/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 5

    Data Center Strategy in ActionPhysical Facilities

    Servers

    Network

    Pod

    4 - 6 Zones Per DC & 6 15 MW per DC

    60,000 80,000 SQF per zone 1-3 MW per zone200 400 racks/cabinets per zone

    Cooling and power per pod (per pair of rack rows)

    8 48 servers per rack/cabinet 1-1.5 KW percabinet

    2 11 interfaces per server

    2500 30000 server per DC

    4000 120,000 ports per DC

    Zone

    DC

    Pod

    Storage

    It all depends on server typesand network access layer model

    COLD AISLE

    HOT AISLE

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    6/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 6

    Reference Physical TopologyNetwork Equipment and Zones

    Server Rack

    Network Rack

    Zone

    DC

    Pod

    Storage Rack

    COLD AISLE

    HOT AISLE

    Pod

    Pod

    Module 1 Module N

    Pod

    Pod

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    7/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 7

    Pod ConceptNetwork Zones and Pods

    Pod

    Pod/Module SizingTypically mapped to access topology Size: determined by distance and density Cabling distance from server racks to network racks

    100m Copper 200-500m Fiber

    Cable density: # of servers by I/Os per serverRacks Server: 6-30 Servers per rack Network (based on access model) Storage: special cabinets

    DC SizingDC: a group of zones (or clusters, or areas)Zone: Typically mapped to aggregation pairNot all use hot-cold aisle designPredetermined cable/power/cooling capacity

    DC

    COL D AISLE

    HOT AISLE

    Pod

    Pod

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    8/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 8

    Network Equipment DistributionEnd of Row and Middle of Row

    Patch panel Patch panel

    NetworkAccess Point

    A - B

    End of Row

    server

    server

    server

    server

    Patch panelX-connect

    NetworkAccess Point

    C - D

    Patch panelX-connect

    Patch panel

    NetworkAccess Point

    A - B

    Middle of Row

    server

    server

    serverPatch panelX-connect

    NetworkAccess Point

    C - D

    Patch panelX-connect

    Patch panel

    End of Row

    Traditionally usedCopper from server to access switchesPoses challenges on highly dense server farms

    Distance from farthest rack to access point Row length may not lend itself well toswitch port density

    Middle of RowUse is starting to increase given EoR challengesCopper from servers to access switchesIt addresses aggregation requirements for ToRaccess environmentsFiber may be used to aggregate ToR

    Common CharacteristicsTypically used for modular accessCabling is done at DC build-outModel evolving from EoR to MoR

    Lower cabling distances (lower cost)Allows denser access (better flexibility)

    6-12 multi-RU servers per Rack4-6 Kw per server rack, 10Kw-20Kw per networkrackSubnets and VLANs: one or many per switch.Subnets tend to be medium and large: /24, /23

    server

    Fiber

    Copper

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    9/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 9

    Network Equipment DistributionTop of Rack

    Top of Rack

    server

    server

    Top of Rack

    server

    ToRUsed in conjunction with dense accessracks(1U servers)Typically one access switch per rack

    Some customers are considering two +cluster

    Use of either side of rack is gaining traction Cabling:

    Within rack: Copper from server toaccess switchOutside rack (uplink):

    Copper (GE): needs a MoR model for fiberaggregationFiber (GE or 10GE):is more flexible and alsorequires aggregation model (MoR)

    Subnets and VLANS: one or many subnets per access switch Subnets tent to be small: /24, /25, /26

    Patch panel

    Network

    AggregationPointA - B

    server

    server

    serverPatch panelX-connect

    Network

    AggregationPointA - B

    Patch panelX-connect

    Patch panel

    server

    Top of Rack Top of Rack

    NetworkAggregation

    PointA - B

    Patch panelX-connect

    NetworkAggregation

    PointC - D

    Patch panelX-connect

    Top of Rack

    server

    Top of Rack

    Patch panel Patch panel

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    10/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 10

    Network Equipment DistributionBlade Chassis

    Switch to SwitchPotentially higher oversubscriptionScales well for blade server racks(~3 blade chassis per rack)Most current uplinks are copper butthe newer switches offer fiberMigration from GE to 10GE uplinksis taking place

    Pass-throughScales well for pass-through bladeracksCopper from servers to accessswitches

    ToRHave not seen it used inconjunction with blade switchesMay be a viable option on pass-through environments is theaccess port count is rightEfficient when used with BladeVirtual Switch environments

    Blade Chassis

    sw1 sw2

    Blade Chassis

    sw1 sw2

    Blade Chassis

    sw1 sw2

    Blade Chassis

    Pass-through

    Blade Chassis

    Pass-through

    Blade Chassis

    Pass-through

    NetworkAggregation

    PointA B C - D

    Patch panelX-connect

    NetworkAggregation

    PointA B - C - D

    Patch panelX-connect

    Top of Rack

    Blade Chassis

    Pass-through

    Blade Chassis

    Pass-through

    Blade Chassis

    Pass-through

    Blade Chassis

    sw1 sw2

    Blade Chassis

    sw1 sw2

    Blade Chassis

    sw1 sw2

    Network

    AggregationPoint

    A B C - D

    Patch panelX-connect

    Network

    AggregationPoint

    A B - C - D

    Patch panelX-connect

    Patch panel

    Patch panel

    Patch panel

    Patch panel

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    11/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 11

    10 Gigabit Ethernet Server Connectivity

    CableTransceiver

    Latency (link)Power

    (each side)DistanceConnector

    (Media)

    Twinax ~ 0.1s~ 0.1W

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    12/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 12

    Cost Effective 10GE ConnectivityToday

    SFP+ USR Ultra Short Reach

    100M on OM3 fiber, 30M on

    OM2 fiber

    Support on all Cisco Catalystand Nexus switches

    Low Cost: $995 NTE

    SFP+ Direct Attach

    1, 3, 5 and 7M on Twinax

    0.1W Power

    Support across all NexusSwitches

    Low Cost: $150 - $260

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    13/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 13

    Nexus 7000

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    14/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 14

    Nexus 7010 10-Slot Chassis

    First chassis in Nexus 7000 product

    family Optimized for data center environments

    High density256 10G interfaces per system

    384 1G interfaces per systems

    High performance64 non-blocking 10G ports1.2Tbps system bandwidth at initial

    release

    80Gbps per slot

    60Mpps per slot

    Future proofInitial fabric provides up to 4.1TbpsProduct family scaleable to 15+Tbps

    40/100G and Unified Fabric ready

    33.1-38(84-96.5cm)

    17.3 (43.9cm)

    21 RU36.5

    (92.7cm)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    15/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 15

    Nexus 7010 Chassis

    Optional frontdoors

    Front Rear

    System statusLEDs

    Integrated cablemanagement

    with cover

    Supervisorslots (5-6)

    Payload slots(1-4, 7-10)

    Air intake withoptional filter

    Air exhaust

    Crossbar fabric

    modules

    System fan trays

    Power supplies

    Fabric fan trays

    21RU

    ID LEDs onall FRUs

    Front-to-back airflow

    Lockingejectorlevers

    Common equipmentremoves from rear

    Two chassisper 7 rack

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    16/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 16

    Nexus 7018 18-Slot Chassis

    Second chassis in Nexus 7000 product family

    Ultra-high density512 10G interfaces per system

    768 1G interfaces per system

    High performance

    128 non-blocking 10G ports

    2.5Tbps system bandwidth at initial release

    80Gbps per slot

    60Mpps per slot

    Future proof

    Initial fabric provides up to 7.8Tbps

    Chassis scaleable to 17.6Tbps

    40/100G and Unified Fabric ready

    Shared equipment

    Supervisors, I/O modules, power suppliescommon between chassis

    Fabrics and fan trays chassis-specific33.1-38

    (84-96.5cm)

    17.3 (43.9cm)

    25 RU43.5

    (110.5cm)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    17/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 17

    Nexus 7018 Chassis

    Optional frontdoors

    Front Rear

    System statusLEDs

    Integrated cable

    management

    Supervisorslots (9-10)

    Power supplyair intake

    Crossbarfabric

    modules

    Power supplies

    25RU

    ID LEDs onall FRUs

    Side-to-sideairflowLocking

    ejector

    levers

    Common equipmentremoves from rear

    Systemfan trays

    Payload slots(1-8, 11-18)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    18/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 18

    Nexus 7000SupervisorEngine

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    19/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID19

    Compact Flash cover

    Supervisor Engine

    Dual-core 1.66GHz Intel Xeon processor with 4GB DRAM Connectivity Management Processor (CMP) for lights-out

    management

    2MB NVRAM, 2GB internal bootdisk, 2 external compact flash slots

    10/100/1000 management port with 802.1AE LinkSec

    Console & Auxiliary serial ports USB ports for file transfer

    Blue beacon LED for easy identification

    BeaconLED

    Console Port

    AUX PortManagement

    Ethernet

    USB Ports CMP Ethernet

    Reset ButtonStatusLEDs

    Compact Flash

    Slots

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    20/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 20

    Out-of-bandmanagement

    network

    CMP

    CMP

    CMP

    CMP

    DataNetwork

    CMP

    CMP

    CMP

    CMP

    Connectivity Management Processor (CMP) Standalone, always-on microprocessor on

    supervisor engine

    Provides lights out remote management anddisaster recovery via 10/100/1000 interface

    Removes need for terminal servers

    Monitor supervisor and modules, access log

    files, power cycle supervisor, etc.Runs lightweight Linux kernel and network stack

    Completely independent of DC-OS on main CPU

    DataNetwork

    console cab les

    Terminal Servers(out-of-band

    console connectivity)

    Out-of-band

    managementnetwork

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    21/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 21

    Nexus 7000I/O Modules

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    22/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 22

    32-Port 10GE I/O Module

    32 10GE ports with SFP+transceivers

    80G full duplex fabric connectivity

    Integrated 60Mpps forwardingengine for fully distributedforwarding

    4:1 oversubscription at front panel

    Virtual output queueing (VOQ)ensuring fair access to fabricbandwidth

    802.1AE LinkSec on every port

    Buffering:

    Dedicated mode: 100MB ingress,80MB egress

    Shared mode: 1MB + 100MBingress, 80MB egress

    Queues: 8q2t ingress, 1p7q4tegress

    Blue beacon LED for easyidentification

    SFP+

    SR at initial release 300m over MMFLR post-release 10km over SMF

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    23/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 23

    32-Port 10GE I/O Module Architecture

    2,4 6,8 10,12 14,16 18,20 22,24 26,28 30,32

    Fabric Interface

    and VOQLayer 2Engine

    Fabric Interface

    and VOQ

    Fabric ASIC

    To Fabr ics

    Port ASIC Port ASIC Port ASIC Port ASIC

    CTS and4:1 Mux

    CTS and4:1 Mux

    CTS and4:1 Mux

    CTS and4:1 Mux

    MAC/PHY

    CTS and4:1 Mux

    CTS and4:1 Mux

    CTS and4:1 Mux

    CTS and4:1 Mux

    Port ASIC Port ASIC Port ASIC Port ASIC

    Mezzanine Card

    1,3 5,7 9,11 13,15 17,19 21,23 25,27 29,31

    Layer 3Engine

    FE Daugh te r

    Card

    LCCPU

    To Central ArbiterEOBC

    (to Por t ASIC)(to L C CPU)

    Inband

    ReplicationEngine

    MET

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    MAC/PHY

    ReplicationEngine

    MET

    ReplicationEngine

    METReplication

    EngineMET

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    24/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 24

    48-Port 1GE I/O Module

    48 1GE 10/100/1000 RJ-45 ports 40G full duplex fabric connectivity

    Integrated 60Mpps forwardingengine for fully distributedforwarding

    Virtual output queueing (VOQ)ensuring fair access to fabricbandwidth

    802.1AE LinkSec on every port Buffer: 7.5MB ingress, 6.2MB

    egress

    Queues: 2q4t ingress, 1p3q4tegress

    Blue beacon LED for easyidentification

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    25/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 25

    48-Port 1GE I/O Module Architecture

    ReplicationEngine

    MET

    Port ASIC

    17-24

    Octal PHY

    Layer 2Engine

    Fabric ASIC

    FE Daugh te r

    Card

    To Fabr ics

    Layer 3

    Engine

    Fabric Interfaceand VOQ

    CTS CTS CTS

    Port ASIC

    CTS CTS CTS

    Port ASIC

    CTS CTS CTS

    Port ASIC

    CTS CTS CTS

    9-16

    Octal PHY

    1-8

    Octal PHY

    41-48

    Octal PHY

    33-40

    Octal PHY

    25-32

    Octal PHY

    To Central Arbiter

    LCCPU

    EOBC

    (to Por t ASIC)

    (to L C CPU)

    Inband

    ReplicationEngine

    MET

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    26/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 26

    Nexus 7000ForwardingEngine

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    27/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 27

    Forwarding Engine HardwareAdvanced hardware forwarding engine integrated on every I/O module

    60Mpps Layer 2 bridging with hardware MAC learning

    60Mpps IPv4 and 30Mpps IPv6 unicast

    IPv4 and IPv6 multicast (SM, SSM, bidir)

    IPv4 and IPv6 security ACLs

    Cisco TrustSec security group tag support

    Unicast RPF check and IP source guard QoS remarking and policing policies

    Ingress and egress NetFlow (full and sampled)

    GRE tunnels

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    28/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 28

    Forwarding Engine Details

    Forwarding engine chipset consists of two ASICs:

    Layer 2 EnginePerforms ingress and egress SMAC/DMAC lookups

    Hardware MAC learning

    True IP-based Layer 2 multicast constraint

    Performs lookups on ingress I/O module, and egress I/O module for bridgedpackets

    Layer 3 Engine60Mpps IPv4 and 30Mpps IPv6 Layer 3/Layer 4 lookups

    Performs all FIB, ACL, QoS, NetFlow processing

    Linear, pipelined architecture every packet processed in ingress and egress

    pipePerforms lookups on ingress I/O module, and egress I/O module for multicastreplicated packets

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    29/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 29

    Nexus 7000Fabric andBandwidth

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    30/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 30

    Fabric Module

    Provides 46Gbps per I/Omodule slot

    Also provides 23G persupervisor slot

    Up to 230Gbps per slot with 5

    fabric modulesInitially shipping I/O modules

    do not leverage full fabricbandwidth

    Load-sharing across all fabricmodules in chassis

    Multilevel redundancy withgraceful performance degradation

    Non-disruptive OIR

    Blue beacon LED for easyidentification

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    31/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 31

    46Gbps92Gbps138Gbps184Gbps230Gbps

    Fabric Capacity and Redundancy Per-slot bandwidth capacity increases with each fabric module

    1G module requires 2 fabrics for N+1 redundancy

    10G module requires 3 fabrics for N+1 redundancy

    4th and 5th fabric modules provide additional level of redundancy

    Future modules will leverage additional fabric bandwidth

    Fabric failure results in reduction of overall system bandwidth

    Fabrics

    ModuleSlots

    40G

    1G Module

    80G

    10G Module

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    32/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 32

    Access to Fabric Bandwidth

    Supervisor engine controls access to fabric bandwidthusing central arbitration

    Fabric bandwidth represented byVirtual Output Queues

    (VOQs)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    33/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 33

    Virtual Output Queues (VOQs) on ingress modules representbandwidth capacity on egress modules

    Guaranteed delivery to egress module for arbitrated packetsentering fabric

    If VOQ available on ingress, capacity exists on egress

    VOQ is NOT equivalent to ingress or egress port buffer or queuesRelates ONLY to ASICs at ingress and egress to fabric

    VOQ is virtual because it represents EGRESS capacity but resideson INGRESS module

    It is PHYSICAL buffer where packets are stored

    What Are VOQs?

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    34/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 34

    What Is VOQ?Ingress module

    Module 1 Module 2(1G module)

    Module 3(10G module)

    Module 4(10G module)

    VOQs forModule 2

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    VOQs forModule 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    VOQs forModule 4

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    Egress modules

    Fabricmodule

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    Destination 1

    Destination 2

    Destination 3

    Destination 4

    Destination 5

    Destination 6

    Destination 7

    Destination 8

    Destination 1

    Destination 2

    Destination 3

    Destination 4

    Destination 5Destination 6

    Destination 7

    Destination 8

    0 1 2 30 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    Destination 1

    Destination 2

    Destination 3

    Destination 40 1 2 3

    0 1 2 3

    0 1 2 3

    0 1 2 3

    EgressCapacity

    (ability to receive trafficfrom fabric)

    VOQ Buffers correspondto Egress Capacity

    (send traffic into fabric basedon destination)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    35/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 35

    Centralized Fabric Arbitration

    Access to fabric bandwidth on ingress module controlledby central arbiter on supervisor

    In other words, access to the VOQ for the destination across the fabric

    Arbitration works on credit request/grant basis

    Modules communicate egress fabric buffer availability to central arbiterModules request credits from supervisor to place packets in VOQ fortransmission to destination over fabric

    Supervisor grants credits based on egress fabric buffer availability forthat destination

    Arbiter discriminates among four classes of servicePriority traffic takes precedence over best-effort traffic across fabric

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    36/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 36

    CentralArbiter

    Module 2

    Fabrics

    VOQ Operation

    Supervisor

    Buffer

    Credits

    VOQ for

    e2/1,3,5,7

    VOQ for

    e1/1,3,5,7

    0 1 2 3VOQ for

    e3/1,3,5,7

    0 1 2 3 0 1 2 3

    Capacity

    available!

    Capacity

    available! Capacity

    available!

    Module 1 Module 3

    0 1 2 3Egress

    Destination

    Capacity

    Egress

    Destination

    Capacity

    0 1 2 3Egress

    Destination

    Capacity

    0 1 2 3

    Egress modules havecapacity to receive t raffic

    from fabric

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    37/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 37

    Fabrics

    VOQ Operation

    Supervisor

    Module 1 Module 2 Module 3

    0 1 2 3VOQ for

    e3/1

    0 1 2 3Egress

    Destination

    Capacity

    0 1 2 3Egress

    Destination

    Capacity

    0 1 2 3VOQ for

    e2/1

    INGRESS MODULE EGRESS MODULES

    VOQs on ingress modulecorrespond to capacity

    on egress modules

    CentralArbiter

    Buffer

    Credits

    VOQ for

    e2/1,3,5,7

    VOQ for

    e1/1,3,5,7

    0 1 2 3VOQ for

    e3/1,3,5,7

    0 1 2 3 0 1 2 3

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    38/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 38

    Fabrics

    VOQ Operation

    Supervisor

    Module 1 Module 2 Module 3

    0 1 2 3VOQ for

    e3/1 Destined toe3/1, priorit y

    level 1

    Request to

    transmit to

    e3/1, prior ity 1!

    Request

    granted!

    0 1 2 3Egress

    Destination

    Capacity

    Buffer for VOQ

    prior ity 1 now

    available!

    0 1 2 3Egress

    Destination

    Capacity

    0 1 2 3VOQ for

    e2/1

    INGRESS MODULE EGRESS MODULES

    CentralArbiter

    Buffer

    Credits

    VOQ for

    e2/1,3,5,7

    VOQ for

    e1/1,3,5,7

    0 1 2 3VOQ for

    e3/1,3,5,7

    0 1 2 3 0 1 2 3

    Deduct cred it

    fr om VOQ

    priority 1

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    39/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 39

    Benefits of Central Arbitration andVOQ

    Ensures fair access to bandwidth for multiple ingressports transmitting to one egress port

    Prevents congested egress ports from blocking ingresstraffic destined to other ports

    Priority traffic takes precedence over best-effort trafficacross fabric

    Engineered to support Unified I/O

    Can provide no-drop service across fabric for future FCoEinterfaces

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    40/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 40

    Layer 2Engine

    Layer 3Engine

    Forwarding Engine

    Nexus 7000 Packet Flow

    Fabric Module 1

    Fabric ASIC

    Fabric Interfaceand VOQ

    Port ASIC

    CTS and4:1 Mux

    MAC/PHY

    MAC/PHY

    ReplicationEngine

    Fabric ASIC

    Module 1e1/1

    Layer 2Engine

    Layer 3Engine

    Forwarding Engine

    Fabric Interfaceand VOQ

    Port ASIC

    CTS and4:1 Mux

    MAC/PHY

    MAC/PHY

    ReplicationEngine

    Fabric ASIC

    Module 2

    Supervisor Engine

    Central Arbiter

    Fabric Module 2

    Fabric ASIC

    Fabric Module 3

    Fabric ASIC

    e2/7

    Ingressqueueing andscheduling

    CTS LinkSec decryption andverification

    Ingress queueing and schedulingin shared mode

    Submit packetfor lookup

    Ingressmulticast

    replication Layer 2 andIGMP snooping

    lookups

    Layer 3

    and Layer4 lookups

    Queueing andVOQ arbitrationrequest

    Transmit tofabric

    Credit grant forfabric access

    Packet transmission

    Packet transmission

    Receive fromfabric

    Queue andschedule

    toward egress Return buffer

    credits

    Submit packetfor lookup

    Egressmulticast

    replication

    Layer 2 andIGMP snooping

    lookups

    Layer 3and Layer4 lookups

    Egressqueueing andscheduling

    CTSLinkSec

    encryption

    Receivepacket

    fromwire

    Transmitpacket on

    wire

    Packet transmission

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    41/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 41

    Virtual DeviceContexts

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    42/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 42

    Virtual Device Contexts (VDCs)

    VDC Virtual DeviceContext

    Flexible separation/distribution ofSoftware Components

    Flexible separation/distribution ofHardware Resources

    Securely delineatedAdministrative Contexts

    Infrastructure

    Layer-2 Protocols Layer-3 Protocols

    VLAN mgr

    STP

    OSPF

    BGP

    EIGRP

    GLBP

    HSRP

    VRRP

    UDLD

    CDP

    802.1XIGMP sn.

    LACP PIMCTS SNMP

    RIBRIB

    Protocol Stack (IPv4 / IPv6 / L2)

    Layer-2 Protocols Layer-3 Protocols

    VLAN mgr

    STP

    OSPF

    BGP

    EIGRP

    GLBP

    HSRP

    VRRP

    UDLD

    CDP

    802.1XIGMP sn.

    LACP PIMCTS SNMP

    RIBRIB

    Protocol Stack (IPv4 / IPv6 / L2)

    Kernel

    VDC

    VDC B

    VDC A VDC B

    VDC n

    VDCs are notThe ability to run different OS levelson the same box at the same time

    based on a hypervisor model; thereis a single infrastructure layer that

    handles h/w programming

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    43/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 43

    Virtual Device ContextsAn Introduction to the VDC Architecture

    Virtual Device Contexts provides virtualization at the device level allowing multiple instances of the

    device to operate on the same physical switch at the same time

    Linux 2.6 Kernel

    Infrastructure

    Protocol Stack (IPv4/IPv6/L2)

    L2 Protocols

    VDC1

    VLAN Mgr

    Physical Switch

    VDCnProtocol Stack (IPv4/IPv6/L2)

    L3 Protocols

    UDLD

    VLAN Mgr UDLD

    LACP CTS

    IGMP 802.1x

    RIB

    OSPF GLBP

    BGP HSRP

    EIGRP VRRP

    PIM SNMP

    RIB

    L2 Protocols

    VLAN Mgr

    L3 Protocols

    UDLD

    VLAN Mgr UDLD

    LACP CTS

    IGMP 802.1x

    RIB

    OSPF GLBP

    BGP HSRP

    EIGRP VRRP

    PIM SNMP

    RIB

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    44/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 44

    Virtual Device ContextsThe Default VDC

    When the system is activated for the first time, it will have a default VDC enabled - this VDC is present

    at all times during the operation of the switch

    Linux 2.6 Kernel

    Infrastructure

    Protocol Stack (IPv4/IPv6/L2)

    L2 Protocols

    VDC1

    VLAN Mgr

    Physical Switch

    L3 Protocols

    UDLD

    VLAN Mgr UDLD

    LACP CTS

    IGMP 802.1x

    RIB

    OSPF GLBP

    BGP HSRP

    EIGRP VRRP

    PIM SNMP

    RIB

    Default VDC VDC #1 (DEFALT_VDC) is the default VDC - bydefault, all ports in the physical chassis are

    assigned to the default VDC while not assigned to

    any other VDC

    Users cannot create or delete the default VDC

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    45/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 45

    Virtual Device ContextsVDC Fault Domain

    A VDC builds a fault domain around all running processes within that VDC - should a fault occur in a

    running process, it is truly isolated from other running processes and they will not be impacted

    Linux 2.6 Kernel

    Infrastructure

    Protocol StackVDCA

    Physical Switch

    VDC A

    ProcessAB

    C

    ProcessDE

    F

    ProcessXY

    Z

    Protocol StackVDCB

    VDC B

    ProcessAB

    C

    ProcessDE

    F

    ProcessXY

    Z

    Fault Domain

    Process DEF in VDC Bcrashes

    Processes in VDC A arenot affected and will

    continue to run unimpeded

    This is a function of theprocess modularity of theOS and a VDC specific

    IPC context

    Nexus 7000 Roadmap

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    46/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 46

    Nexus 7000 RoadmapLow er ing P ow er Cos t w h i le I n c r eas ing S ca le

    15+ TerabitInfrastructure

    M1 Series I/O Modules32 por t 10G (80G/slot)48 por t 1G (46G/slot)

    M1 Series I/O Module48 port 1G (46G/slot)

    M1 Series I/O Modu les8 por t 10G-XL (80G/slot )48 port 1G-XL (46G/slot)

    M1 Series I/O Modules16 port 10G (160G/slot )

    D1 Series I/O Modules32 port 10G DCB SFP+ (230G/slot )

    32 por t 10G DCB 10GBASE-T (230G/slot)D2 Series I/O Modules

    48 port 10G DCB SFP+ w/L3 (480G/slot)48 port 10G DCB 10GBASE-T w/L3 (480G/slot)

    40G/100G mod

    vPC1GbE

    18 Slot Chassis

    OTV, MPLSN2K Suppor t

    DCB , L2MP, FCoE

    LISPCCN

    Service Modules

    ISSUCTSVDC

    2008 2010 20112009

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    47/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 47

    Nexus 5000

    Nexus 2000

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    48/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 48

    OS

    Cisco Nexus 5000 Series

    56-Port L2 Switch

    40 Ports 10GE/FCoE/DCE, fixed

    2 Expansion module slots

    Cisco Fabric Manager and Cisco Data Center Manager

    Cisco DC-OS

    FC + Ethernet

    4 Ports 10GE/FCoE/DCE

    4 Ports 1/2/4G FC

    Fibre Channel

    8 Ports 1/2/4G FC

    Mgmt

    Cisco DC-OS

    Ethernet

    6 Ports 10GE/FCoE/DCE

    DC-NM and Fabric Manager

    NX-OS

    28-Port L2 Switch

    20 Ports 10GE/FCoE/DCE, fixed

    1 Expansion module slot

    Nexus 5010Nexus 5020

    All 10GE switch/module ports are FCoE/Data Center Ethernet capable

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    49/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 49

    Nexus 2000 Fabric Extender1GE Connectivity

    48 x 1 GE interfaces 4 x 10 GE interfaces

    Beacon and status LEDs

    Redundant, hot-swappable

    power supplies

    Hot-swappable fan tray

    N 2000 F b i E t d

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    50/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 50

    Nexus 2000 Fabric ExtenderVirtual Chassis

    The Nexus 2000 Fabric Extender (FEX) acts as a remote linecardfor the Nexus 5000, retaining all centralized management and configurationon the Nexus 5000, transforming it into a Virtualized Chassis

    Nexus 5000Virtualized chassis

    +

    Nexus 5000

    Nexus 2000 Fabric Extender=

    D t C t A A hit t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    51/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 51

    Data Center Access ArchitectureVirtualized Access Switch

    Nexus 5010/5020

    Nexus 5000/2148T Virtualized

    Access Switch provides a number ofdesign options to address evolvingData Center requirements

    Fabric Extender provides for flexibilityin the design of the physicaltopologies

    Aids in building larger layer 2 designssafely

    Support of latest spanning treeenhancements

    Single virtual access switch(Simplifies the layer 2 design)

    Support of 16-way 10GEEtherchannel combined with vPCprovides increased network capacity

    Nexus 2148T FabricExtender 48 GE Ports

    4 x 10GE FabricLinks per FabricExtender (CX-1

    Cu)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    52/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 52

    Data Center ArchitectureN5K/N2K - Logical Topology

    Nexus 5000/2000Vir tualized Access

    Switch Pods . . .

    Cisco Nexus 2148T FabricExtender (N2K) and Nexus5000 (N5K) Pod

    N2K + N5K Podrepresents networking

    Access layer Nexus 7000 at Distribution

    Layer

    Each VirtualizedAccess Switch Podconfigured to supportup to 576 1GE server

    ports at FCS

    D t C t A A hit t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    53/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 53

    Data Center Access ArchitectureOptimizing Layer 1 and Layer 2 Designs

    1GE Attached Servers - Maintain Existing Cat5e Server

    Wiring Infrastructure with EoR topology

    Nexus 5000/2000EoR

    . . .

    Cisco Nexus 2148T Fabric Extender and Nexus

    5000 provide a Flexible Access Solution De-Coupling of the Layer 1 and Layer 2 Topologies

    Optimization of both Layer 1 (Cabling) and Layer 2(Spanning Tree) Designs

    Provides for simultaneous support of EoR, MoR and

    ToR

    D t C t A A hit t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    54/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 54

    Data Center Access ArchitectureN5K/N2K Advantages Flexible Cabling

    Combination of EoR and ToR cabling

    Nexus 5000/2000Mixed ToR & EoR

    . . .

    Cisco Nexus Fabric Extender (FEX) and Nexus

    5000 provide a Flexible Access Solution

    Migration to ToR for 10GE servers or selective1GE server racks if required (mix of ToR and EoR)

    Mixed cabling environment (optimized as required)

    Flexible support for Future Requirements

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    55/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 55

    Fabric ExtenderFabric Modes

    Fabric Extender associates (pins)a server side (1GE) port with anuplink (10GE) port

    Server ports are either individuallypinned to specific uplinks (staticpinning) or all interfaces pinned to

    a single logical port channel Behavior on FEX uplink failure

    depends on the configuration

    Static Pinning Server portspinned to the specific uplink are

    brought down with the failure ofthe pinned uplink

    Port Channel Server traffic isshifted to remaining uplinks basedon port channel hash

    Static Pinning

    Port Channel

    Server Interfacegoes down

    Server Interfacestays active

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    56/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 56

    Nexus 2148 Fabric ExtenderConfiguring the Fabric Extender

    Two step process

    Define the Fabric Extender (100-199) and the number of fabricuplinks to be used by that FEX (valid range: 1-4)

    Nexus- 5000# swi t ch# conf i gur e t ermi nalswi t ch( conf i g) # f ex 100swi t ch( conf i g- f ex) # pi nni ng max- l i nks 4

    Nexus- 5000# swi t ch# swi t ch# conf i gur e t er mi nalswi t ch( conf i g) # i nt er f ace et her net 1/ 1swi t ch( conf i g- i f ) # swi t chpor t mode f ex- f abr i cswi t ch( conf i g- i f ) # f ex associ at e 100. . .

    Configure Nexus 5000 ports as fabric ports and associate thedesired FEX

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    57/97

    D t C t A A hit t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    58/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 58

    Data Center Access ArchitecturevPC Redundancy Models Dual Chassis

    MCEC fr om server to theaccess switch

    vPC provides two redundancy designs for the virtualized access switch

    Option 1 - MCEC connectivity from the server

    Two virtualized access switches bundled into a vPC pair

    Full redundancy for supervisor, line card, cable or NIC failure

    Logically a similar HA model to that currently provided by VSS

    vPC peers

    Two Virtualized access switchesEach with a Sing le Supervisor

    D t C t A A hit t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    59/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 59

    Data Center Access ArchitecturevPC Redundancy Models Dual Supervisor

    Act ive/Standby NIC teaming

    vPC Option 2 Fabric Extender connected to two Nexus 5000

    From the server perspective a single access switch with each line cardsupported by redundant supervisors

    Full redundancy for supervisor, fabric via vPC and cable or NIC failurevia active/standby NIC redundancy

    Logically a similar HA model to that currently provided by dual

    supervisor based modular switch

    Fabric Extender dual homed toredundant Nexus 5000

    N 5000 & 2000 R d

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    60/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 60

    Nexus 501028-Port 1RU Switch

    Nexus 502056-Port 2RU Switch

    Q2CY08

    Q4CY08

    Nexus 5000 & 2000 Roadmap

    FEX-1GE

    N2148T-1GE

    48x1GE + 4x10GE

    FEX-100M/1GT

    N2248T-1GE48 port 100/1GT

    downlinks, 4x10GEuplinks

    Q1/Q2CY10

    Q1CY09

    Next-Generation Nexus 500048-ports & 96-port s Switch

    FEX-10GE

    N2232-10GE32 ports 10GE

    SFP+ downlinks,8x10GE SFP+

    uplinks

    2HCY10

    N2K

    N5K

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    61/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 61

    Nexus 1000V

    The Story TodayNetworking with

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    62/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 62

    The Story TodayNetworking withVI3.5

    Separation of Network andServer provisioning andmanagement systems

    Virtual Center managing& provisioning ESX hostsand vSwitches

    Physical network

    managed andprovisioning separately

    Network visibility ends atphysical switch port

    Different interfaces and tools

    IOS or IOS-like cli forphysical network

    VC GUI and esxcfg cli forvSwitches

    vSwitch vSwitch vSwitch

    NetworkManagement

    Virtual Center

    vNetwork Distributed Switch & Cisco Nexus

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    63/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 63

    vNetwork Distributed Switch & Cisco Nexus1000V

    vSwitchCURRENT

    vSwitch vSwitch

    VDS

    vNetwork Distributed Switch Cisco Nexus 1000V

    Enterprise networking vendors canprovide proprietary networkinginterfaces to monitor, control andmanage virtual networks

    First offering: Cisco Nexus 1000V

    Virtual machines retain policies,QoS as they move around thedatacenter

    Ci N 1000V C t

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    64/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 64

    Cisco Nexus 1000V Components

    Cisco VEM

    VM1 VM2 VM3 VM4

    Cisco VEM

    VM5 VM6 VM7 VM7

    Cisco VEM

    VM9 VM10 VM11 VM12

    Virtual Ethernet Module(VEM)

    Replaces Vmwares virtual switch

    Enables advanced switching capabilityon the hypervisor

    Provides each VM with dedicatedswitch ports

    vCenter Server

    Virtual Supervisor Module(VSM)

    CLI interface into the Nexus 1000V

    Leverages NX-OS 4.04a

    Controls multiple VEMs as a singlenetwork device

    Cisco VSMs

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    65/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 65

    Cisco Nexus 1000V Virtual Chassis

    Cisco VEM

    VM1 VM2 VM3 VM4

    Cisco VEM

    VM5 VM6 VM7 VM8

    pod5- vsm# show modul eMod Por t s Modul e- Type Model St atus

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 0 Vi r t ual Supervi sor Modul e Nexus1000V act i ve *2 0 Vi r t ual Supervi sor Modul e Nexus1000V ha-s t andby

    3 248 Vi r t ual Et hernet Modul e NA ok

    Cisco VSMs

    Nexus 1000V Faster VM Deployment

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    66/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 66

    Nexus 1000V Faster VM Deployment

    Nexus 1000V VSMvCenter

    vSphere

    Nexus1000VVEM

    vSphere

    Nexus1000VVEM

    Defined Policies

    WEB AppsHR

    DB

    DMZ

    VM Connection Policy

    Defined in the network

    Applied in Virtual Center

    Linked to VM UUID

    Policy-BasedVM Connectiv ity

    Mobil ity of Network &Securi ty Propert ies

    Non-DisruptiveOperational Model

    Cisco VN-Link: Virtual Network Link

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    Nexus 1000V Richer Network Services

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    67/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 67

    Nexus 1000V Richer Network Services

    Nexus 1000V VSM

    vSphere

    Nexus1000VVEM

    vSphere

    Nexus1000VVEM

    VN-Link Property Mobility

    VMotion for the network

    Ensures VM security

    Maintains connection state

    VMs Need to Move

    VMotion

    DRS

    SW Upgrade/Patch

    Hardware Failure

    vCenter

    Policy-BasedVM Connectivity

    Mobil ity of Network &Securi ty Propert ies

    Non-DisruptiveOperational Model

    Cisco VN-Link: Virtual Network Link

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    Nexus 1000V Increased Operational Efficiency

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    68/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 68

    Nexus 1000V - Increased Operational Efficiency

    Nexus 1000V VSM

    vSphere

    Nexus1000VVEM

    vSphere

    Nexus1000VVEM

    vCenter

    Network Admin Benefits Unifies network mgmt and ops

    Improves operational security

    Enhances VM networkfeatures

    Ensures policy persistence

    Enables VM-level v isibility

    VI Admin Benefits Maintains existing VM mgmt

    Reduces deployment time

    Improves scalability

    Reduces operational workload

    Enables VM-level v isibility

    Policy-BasedVM Connectiv ity

    Mobil ity of Network &Secur ity Propert ies

    Non-DisruptiveOperational Model

    Cisco VN-Link: Virtual Network Link

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    VM

    Key Features of the Nexus 1000V

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    69/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 69

    Key Features of the Nexus 1000V

    Switching L2 Switching, 802.1Q Tagging, VLAN Segmentation, Rate Limiting (TX)

    IGMP Snooping, QoS Marking (COS & DSCP)

    Security Policy Mobility, Private VLANs w/ local PVLAN Enforcement

    Access Control Lists (L24 w/ Redirect), Port Security

    Provisioning Automated vSwitch Config, Port Profiles, Virtual Center Integration

    Optimized NIC Teaming with Virtual Port Channel Host Mode

    Visibility VMotion Tracking, ERSPAN, NetFlow v.9 w/ NDE, CDP v.2

    VM-Level Interface Statistics

    Management Virtual Center VM Provisioning, Cisco Network Provisioning, CiscoWorks

    Cisco CLI, Radius, TACACs, Syslog, SNMP (v.1, 2, 3)

    Cisco Nexus 1000V

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    70/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 70

    Cisco Nexus 1000V3 new fea tu res t ha t m ake a d i ffe r ence

    Great for mixed useESX clusters

    Segment VMs w/oburning IP addresses

    Supports isolated,community and

    promiscuous trunkports

    Follows your VM w/VMotion or DRS

    Private VLANs(PVLANs)

    View flow based statsfor individual VMs

    Captures multi-tieredapp traffic inside a single

    ESX host

    Export aggregate statsto dedicated collector for

    DC-wide VM view

    Follows your VM w/VMotion or DRS

    Netflow v.9 withData Export

    Mirror VM interfacetraffic to a remote sniffer

    Identify root cause forconnectivity issues

    No host based sniffervirtual appliance to

    maintain Follows your VM w/

    VMotion or DRS

    EncapsulatedRemote SPAN

    (ERSPAN)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    71/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 71

    Differnet aspects of

    Virtualizationin Data Center

    Virtualization in the Data Center

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    72/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 72

    Virtualization in the Data Center

    Common Benefits Increase resource usability Lower CAPEX Loosely couple re-usable functions - Flexibility Dynamic allocation of virtual instances - Automation Centralized policy Management Lower TCO

    Distributed Capabilities Broad use of functions

    Servers: the capability of decoupling CPU, Mem and I/O functions fromphysical devices that provide them to increase their effective utilization,

    to enhance the flexibility in how they are utilized and to allow thedynamic management of logical instances

    Storage: the capability of abstracting the physical location of datastorage by presenting logical storage to the user for thus achievinglocation independence

    Network-based Services: the capability of manipulating serviceinstances independent from the service device thus providing flexibilityin their usage

    Network Infrastructure: the capability of partitioning groups ofnetworked resources providing logical isolation and common policy

    Virtual Ethernet Switching

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    73/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 73

    Virtual Switches: Logical instances of physical switches

    - Many to one: grouping of multiple physical switches

    - Reduce management overhead (single switch) and simplify configuration (single sw config)

    - One to Many: partitioning of physical switches

    - Isolate control plane and control plane protocols

    Virtual PortChannels: Etherchannel across multiple chassis

    - Simplify L2 pathing by supporting non-blocking cross-chassis concurrent L2 paths

    - Lessen reliance on STP (loopfree L2 paths are not established by STP)

    Virtual Switching Implementations

    - Virtual Switching System VSS: Catalyst 6500

    - Virtual Blade Switches VBS: 10GE-based Blade Switches

    - Virtual Device Context VDC: Nexus 7000

    - Virtual Port-Channel vPC: Nexus Family

    Virtual Ethernet SwitchingImproving Management and Pathing

    Virtual Switching

    Virtual Switching:

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    74/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 74

    Many to OneMany switches look like one

    Two switches are physicalOne switch is virtual

    Virtual Switch:i. All ports appear to be on the same physical switch

    ii. Single point of managementiii. Single configurationiv. Single IP/macv. Single control plane protocol instance

    Benefitsi. Simplify infrastructure managementii. 1 switch to manage

    A1AA2

    A1AA2

    STP HSRP

    OSPF SNMP

    STP HSRP

    OSPF SNMPSTP HSRP

    OSPF IGMP

    Virtual Switching:Many to One VSS

    Virtual Blade Switching

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    75/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 75

    Many to OneMany switches look like one

    Up to Eight switches are physicalOne switch is virtual

    Virtual Switch:

    i. All ports appear to be on the same physical switchii. Single point of managementiii. Single configurationiv. Single IP/mac

    Benefitsi. Simplify infrastructure managementii. 1 switch to manage

    AA

    A2A1

    A8A7

    A4A3

    A6A5

    A2A1

    A8A7

    A4A3

    A6A52

    2

    Virtual Blade SwitchingMany to One VBS

    Virtual Switching

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    76/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 76

    One to ManyOne switch looks like one

    1 switch is physicalMany switches are logical

    Virtual Switch:i. Switch ports only exist on a single logical instance

    ii. Per virtual switch point of managementiii. Per virtual switch configurationiv. Per virtual switch IP/macv. Per virtual switch control plane protocol instance

    Benefitsi. Control plane isolationii. Control protocol isolation

    Virtual SwitchingOne to Many VDC

    A

    A1 A2

    A3 A4

    A

    A1 A2

    A3 A4

    STP HSRP

    OSPF IGMP

    STP HSRP

    OSPF IGMP

    STP HSRP

    OSPF IGMP

    STP HSRP

    OSPF IGMP

    STP HSRP

    OSPF IGMP

    Virtual Switching

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    77/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 77

    Virtual SwitchingOne to Many VDC Topology

    AG1 AG2

    AC

    VLAN X

    AC1 AC2

    VLAN X

    AG11

    AG21

    AG12

    AG22AG1 AG2

    AC1 AC2

    VLAN Y

    AC11 AC12

    VLAN XVLAN Y

    AC21 AC22

    VLAN X

    AG12

    AG22

    AG11

    AG21

    One to ManyOne switch looks like many

    Devices connect to a single virtual switchVirtual Switching Topologies are isolated from one anotherVirtual Topology:

    i. Distinct physical ports form virtual topologiesii. Each topology has independent and isolated control protocols

    Benefitsi.Support Isolated but parallel topologiesii.Supports smaller logical environments

    Virtual Portchannels

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    78/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 78

    Two to oneTwo Physical to a single logical

    Devices connect to a single logical switchConnections are treated as portchannelVirtual PortChannel:

    i. Ports to virtual switch could form a cross-chassis portchannelii. virtual Portchannel behaves like a regular Etherchannel

    Benefitsi.Provide non-blocking L2 pathsii.Lessen Reliance on STP

    A

    AG1 AG2

    2 2

    4

    2 2

    AG1 AG2

    2 2

    H

    AG

    8

    ACAC1 AC2

    1 1

    2

    Virtual PortchannelsTopology VSS and vPC (through MCEC)

    H

    New Topology Using Virtual Switching

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    79/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 79

    New Topology Using Virtual Switching

    BA

    4 4

    4

    2 4 2 2

    4

    Physical View Logical View

    Key characteristicsOptimized L2 Pathing through Virtual PortChannels

    i. Increase bandwidth usage on: server to switch and switch to switchii. Loop-free forwarding paths

    Less Reliance on STPi. Loopfree topology established through Virtual Switch mechanismii. STP is strictly used as a fail-safe mechanism

    Isolation of L2 Domainsi. Separate Logical Topologiesii. Distinct STP Topologies

    core1 core2

    agg2agg1

    acc2acc1

    agg4agg3

    accYaccN

    Virtualswitch

    Virtualswitch

    Virtualswitch

    Virtualswitch

    Virtualswitch

    C D BAC D

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    80/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 80

    New standards:

    Fibre Channel overEthernet

    What Is FCoE?

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    81/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 81

    What Is FCoE?Fibre Channel over Ethernet

    From a Fibre Channel standpoint its

    FC connectivity over a new type of cable called an Ethernetcloud

    From an Ethernet standpoints itsYet another ULP (Upper Layer Protocol) to be transported, but

    a challenging one!

    And technically

    FCoE is an extension of Fibre Channel

    onto a Lossless Ethernet fabri c

    FCoE

    FC over Ethernet (FCoE)

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    82/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 82

    Encapsulate Fibre Channel framesonto Lossless Ethernet

    FCoE

    FC over Ethernet (FCoE)

    FibreChannelTraffic

    Ethernet

    Destination MAC Address

    Source MACAddress

    IEEE 802.1Q Tag

    ET = FCoE Ver Reserved

    Reserved

    Reserved SOF

    Encapsulated FC Frame(Including FC-CRC)

    EOF Reserved

    FCS

    Reserved

    FCoE Frame Format

    Bit 0 Bit 31

    Ethernet

    Header

    FCoE

    Header

    FC

    Header

    FC Payload CRC

    EOF

    FCS

    Byte 0 Byte 2179

    Fibre Channel over Ethernet

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    83/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 83

    Fibre Channel over EthernetBrief look at the Technology

    A method for a direct mapping of FC framesover Ethernet

    Seamlessly connects to FC networks

    Extends FC in the datacenter over the Ethernet

    FCoE appears as FC to the host and the SAN

    Preserves current FC infrastructure

    and managementFC frame is unchanged

    Can operate over standard switches(with jumbo frames)

    Priority Flow Control guarantees no-drops

    Mimics FC credit-buffer system, avoidsTCP

    Does not require expensive off-loads

    FibreChannelTraffic

    Ethernet

    FCoE

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    84/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 84

    FC HBA

    FC HBA

    NIC

    NIC

    FC Traffic

    FC Traffic

    Enet Traffic

    Enet Traffic

    FC HBA

    FC HBA

    NIC

    NIC

    FC HBA

    FC HBA

    NIC

    NIC

    Today: Parallel LAN/SAN Infrastructure

    Inefficient use of NetworkInfrastructure

    5+ connections per server higheradapter and cabling costs

    Adds downstream port costs;cap-ex and op-ex

    Each connection adds additionalpoints of failure in the fabric

    Power and cooling

    Longer lead time for serverprovisioning

    Multiple fault domains complexdiagnostics

    Management complexity firmware,driver-patching, versioning

    Unified I/O Use Case

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    85/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 85

    Management

    SANB

    SANA

    LAN

    Today:

    Ethernet

    FC

    Aggregation/Coreswitches

    Access Top of the

    Rack switches Servers

    Unified I/O Use Case

    FC HBA

    FC HBA

    NIC

    NIC

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    86/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 86

    Management

    SANB

    SANA

    LAN

    FCoE

    Ethernet

    FC

    Today

    Unified I/O Use Case

    Unif ied I/O Phase 1

    FCoESwitch

    Unified I/O Reduction of server adapters

    FewerCables

    Simplificationof access layer & cabling

    Gateway free implementation - fits ininstalled base of existing LAN and SAN

    L2 Multipathing Access Distribution

    Lower TCO

    Investment Protection (LANs and SANs)

    Consistent Operational Model

    One set of ToR Switches

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    87/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 87

    How the design

    has changed

    Discrete Network FabricsT i l Eth t d St T l FC E

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    88/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 88

    Typical Ethernet and Storage Topology

    Fabric A Fabric B

    SAN Fabric

    Enet

    FC

    Single Ethernet Network Fabric

    Typically 3 tiers

    Access Switches are dual-homed

    Servers are single or multi-homed

    VSAN 2 VSAN 3

    A B E FC D

    L2

    VLAN A VLAN B

    L3

    VLAN C

    L3

    L2

    Core

    Aggregation

    Access

    Dual Storage Fabrics

    Typically 2 tiers

    Edge sw itches are dual-homed

    Servers are dual-homed to d ifferent fabrics

    FCoE

    Unified Fabric: DCEFCoE Se e A e FCoE

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    89/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 89

    FCoE Server Access

    Fabric A Fabric B

    SAN Fabric

    L2

    VLAN A VLAN B

    L3

    L3

    L2

    Core

    Aggregation

    Access

    Enet

    FC

    DCE

    A B EDVLAN C VLAN D

    CNA

    CNA Converged Network Adaptor

    FCoE

    New TopologyE h d L2 D i

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    90/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 90

    Enhanced L2 Design

    VLAN A

    Module 1

    VLAN B

    L2

    Module 2

    L3

    VLAN D VLAN EVLAN C

    aggx aggx+1

    accX accX+1 accY accY+1

    Enhanced L2 Topology

    3-tier L2 Topology

    Nexus at Core and Aggregation Layers

    6500 at Aggregation and Services Layers

    Topology Highlights

    DC-Wide VLANs

    Higher Stability of STP environment New STP Features

    Lower Oversubscription - if NeededHigher Density 10 GE at Core and Agg Layers

    acc1 acc2 accN accN+1

    core1 core2

    agg2agg1

    Enhance L2 TopologyE d t d Vi t l S it hi

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    91/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 91

    End to end Virtual Switching

    VLAN A

    Module 1

    VLAN B

    L2

    Module 2

    L3

    VLAN D VLAN EVLAN C

    accX accX+1

    Enhanced L2 Topology

    3-tier L2 Topology

    Nexus at Core and Aggregation Layers

    6500 at Aggregation and Services Layers

    Topology Highlights

    DC-Wide VLANs

    Higher Stability of STP environment New STP Features

    Lower Oversubscription - if NeededHigher Density 10 GE at Core and Agg Layers

    acc1 acc2

    accY

    agg1 aggx

    core1

    1 X

    ServerN

    accN

    New Topology Isolating Collapsed L2 DomainsVi t l D i C t t @ A L

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    92/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 92

    Virtual Device Contexts @ Agg Layer

    Pods are isolated at aggregation layer

    Each Pod runs its own STP instance (instanceper VDC)

    Multiple pods could exist in a single VDC

    VLANs contained within Agg Module per VDC

    L2

    L3

    VLAN C VDC2

    VLAN C VDC1

    1 2 agg2agg1

    Pods are logically isolated two topologiesEach Pod belong to multiple VDCs

    Each VDC topology requires dedicated Ports

    VLANs contained within Agg Module per VDC

    Higher 10GE Port Density Allows multiple Agg Pairs to be collapsed

    Collapsed Agg Pair could still be L2 isolated (different STP instances)

    VLAN IDs could be replicated on different VDC shared infrastucture

    Module 1

    L2

    Module 1

    L3

    VLAN C VLAN C

    acc1 acc2 accN accN+1

    agg1 agg2 agg3 agg4

    acc1 acc2 accN accN+1

    L2

    Module 1

    L3

    VLAN C VDC1 VLAN C VDC2

    acc1 acc2 accN accN+1

    agg1 agg2

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    93/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 93

    Brief summary

    Q&A

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    94/97

    IaaS Service= Compute + Storage + Network

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    95/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 95

    = Compute + Storage + NetworkBronze Gold

    Bronze CPU RAM SAN

    Web1 .5 4 500G

    DB1 .5 8 500G

    App1 .5 16 500G

    Silver

    Silver CPU RAM SAN

    Web2 1 4 Nx500G

    DB2 1 8 Nx500G

    App2 1 16 Nx500G

    Gold CPU RAM SAN

    Web3 2 4 Nx500G

    DB3 2 8 Nx500G

    App3 2 16 Nx500G

    Cisco Unified Data Center of 2010

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    96/97

    2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 96

    Cisco Unified Data Center of 2010

    Unified Compute Nexus 70001/10GE / 10GBaseT

    FCoE

    N2K / N5K1 GE

    Nexus 500010 GEFCoE

    Top of Rack Desig nsEnd o f Row

    N2K / N7K1 GE

    VM

    VM

    VM

    VM

    VMVM

    VM

    VM

    VM

    VM

    VMVM

    VM

    VM

    VM

    VM

    VMVM

    VM

    VM

    VM

    VM

    VMVM

    CoreL3 boundary to the DC network. Functional point forroute summarization, the injection of default routes

    and termination of segmented virtual transportnetworks

    AggregationTypical L3/L2 boundary. DC aggregation point for

    uplink , storage and DC services offering keyfeatures: VPC, VDC, 10GE density and 1st point of

    migration to 40GE and 100GE

    1Gb to 10Gb to Unified FabricPrice performance

    Next gen N5K (48-96 Universal Port)FCoE on N7K

    100/1Gb FEX 10G FCoE + 10GBase-T

    VM Visibil ityVM Securit y

    Optimized VMotionFast VM Provisioning

    High Density VM Deploym ent

    Access

    Virtual AccessNEXUS 1000v

    Virtua l Adapter

    Expanded Memory

    NEXUS 5000NEXUS 2000

    NEXUS 7000vPC

    Catalyst6500

    ServiceModules

    NEXUS 7000

    NEXUS1000v

    MDS-StorageNEXUS 7000vPC, L2MP

    FCoE

  • 8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus

    97/97