designing mpls in next generation data center: a case...

Designing MPLS in Next Generation Data Center: A Case Study BRKMPL-2108

Khalil Jabr – Distinguished Engineer [email protected]

© 2013 Cisco and/or its affiliates. All rights reserved. BRKMPL-2108 Cisco Public

Session Goals

At the end of the session, the participants should:

Understand the design requirements

Understand the technical building blocks

Understand the selected design and reasoning behind it

Understand the lessons learned

3


Agenda

Design Requirements

Technology Involved

– The Fabric

– The Service Layer

– The WAN Connectivity

Design Options

Selected Design

Lessons Learned

4


Customer Requirements – NG Data Center

Multi-tenancy – Public SaaS

Highly Scalable DC Architecture

L2 Connectivity Between Racks

Optimized for East/West as well as North/South

Minimize Oversubscription

Scalable L4-7 Service Layer

Highly available WAN

Scalable WAN Architecture

Some DCs connect via Internet

Simplicity!

5


Strategy

Virtual FW/LB per tenant

Flexible placement

Incremental capacity

Multi-tenancy

Security and

Separation

Traffic Eng

Scalable

Network

Topology

(DC & WAN)

Virtualized

L4-7

Services

Network Virtualization

Flexible topology

Minimize oversubscription

Scale out and scale up

No spanning tree

Incremental scale

6

Background Info: The Building Blocks












Simplicity!

8


Virtual Network

Cust2

Network Virtualization

Giving one physical network the ability to support multiple virtual networks

Customer requirement: 60-80 virtual networks

Separation between:

Line of business

Customers

App layers

9

Actual Physical Infrastructure

Alpha Network

Virtual Network Virtual Network

Cust2


Provider Edge VRFs (PE VRFs)

802.1q

VRF

VRF

VRF

VPN LSP/Tunnel Logical or Physical

Int (Layer 3)

MPLS/Tunnel Labels

and Route Targets

PE Router

IP Switching Label Switching (MPLS) or

Tunneling (L2TPv3)

10


MPLS in Data Center

11

Business drivers Business Goal Solution

L3

VPN

MVPN TE 6PE /

6VPE

Multi-tenant hosting

Mergers / acquisitions

Network consolidation

Shared services

Compliance

Segmentation

Bandwidth provisioning Capacity

Planning

High Availability

Path Diversity

Ensure SLAs

IPv4 address depletion

IPv6 readiness / migration

Expansion


MPLS and VDCs

Secure and flexible way of software process partitioning

All MPLS features are VDC aware

Each VDC operates as separate MPLS router (LER / LSR):

No internal communication between VDCs

Multiple logical P / PE routers can be configured

Each VDC has independent label space for prefix labels: LDP, VPN, TE

Note: per-VRF / aggregate VPN labels - globally significant for whole chassis, all others are locally significant to VDC

12

VDC 1

VDC 2

VDC 3

VDC 4

Kernel

Infrastructure

VDC 8

…


MPLS and VDCs

Vertical consolidation – collapse layers of P/PE routers

Horizontal consolidation – collapse PE’s from several PODs

13

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

P1

(VDC 3)

PE7

(VDC 2)

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

PE5

(VDC 4)

P2

(VDC 3)

PE8

(VDC 2)

PE6

(VDC 4)

PE1

(VDC 2)

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

Serv

er

PE3

(VDC 3)

PE2

(VDC 2)

PE4

(VDC 3)

MPLS

Core

POD 1 POD 2












Simplicity!

14


Number of Spine Switches

Need f

or

HA

Spine-Leaf DesignChanges to the Approach to Structured High Availability

Spine

Leaf

15


DC Fabric w/FabricPath

Externally the Fabric looks like a single switch

Internally, ISIS adds Fabric-wide intelligence and ties the elements together.

Provides in a plug-and-play fashion: Optimal, low latency connectivity any to any

High bandwidth, high resiliency

Open management and troubleshooting

ISIS for multipathing and reachability

FabricPath FabricPath

16


Layer 2 with FabricPath

Allows extending vlans with no limitation (no risks of loop)

Devices can be attached active/active to the fabric using IEEE standard port channels using LACP and without resorting to STP

17

FabricPath

A

s3 s8 s7

B

s5

VLAN X VLAN Y VLAN Z


Edge Device Integration

Hosts see a single default gateway

The fabric provide them transparently with multiple simultaneously active default gateways

Allows extending the multipathing from the inside to the fabric to the L3 domain outside the fabric

Equal Cost Multipathing (ECMP) from the upstream network to the servers

18

Hosts can leverage multiple L3 default gateways

FabricPath

A

s3

dg dg L3


Split VLANs

Some polarization

Inter-VLAN traffic can

be suboptimal

VLAN

100-200 VLAN 300-

400

FabricPath

GLBP

Host is pinned to a

single gateway

Less granular load

balancing

VLAN 100-400

FabricPath

Anycast HSRP

All active

Available in NX-OS 6.2

release

VLAN 100-400

FabricPath

spine spine

Default Gateway Options

19


Cisco Data Center Fabric

FabricPath

Spine Layer

Default Gateway

Elastic East/West

Performance

Low Latency Security

Multipathing

Any VLAN Anywhere

Efficient Performance

Open

20


Hardware Support

All MPLS functionality supported on all M-series I/O modules

F1 and F2e* modules support FabricPath and

MPLS via proxy-mode

21

* - Software support in NX-OS 6.2

N7K-M108X2-

12L

N7K-M148GS-11

N7K-M148GS-11L N7K-M148GT-11

N7K-M148GT-11L

N7K-M132XP-12

N7K-M132XP-12L

N7K-M224XP-23L

N7K-M202CF-22L

N7K-M206FQ-23L


MPLS Layer 3 VPNs - I/O modules

F1 / F2e* I/O modules – mixed chassis design with M I/O modules

F2 / F2e I/O modules – separate VDC design with VRF-lite

22

L3

L2

MPLS

VR

F-L

ite

M M M M

M M

F2/F2e F2/F2e

F2/F2e

F2/F2e F2/F2e F2/F2e F2/F2e

* - NX-OS 6.2

Required

L3

L2 F1/F2e/M F1/

F2e

M

MPLS

M M

F1/

F2e


F2e-M for FabricPath + MPLS Designs

F2e + M2 (or M1-XL) at the aggregation in the same VDC *

Layer 3 routing performed by the M-Series. F2e in Layer 2 mode

FabricPath towards the Access

MPLS towards the Core

MPLS

FabricPath

* Requires NX-OS 6.2

23












Simplicity!

24


Firewall Options

Three options to consider

– Virtual Firewall (ASA 1000v) and VSG Virtualized services

High scale

Leverages vPath technology

– IOS Zone Based Firewall Router based

Native routing

– ASA Purpose built hardware

Advanced firewall and security features

Next slides explore the ASA and ZBFW options

25


ASA FW + Fusion Router

Fusion router:

– Inter-VPN connectivity

– Shared resource connectivity

Internet, servers, etc.

ASA contexts:

– VPN isolation / protection

– Per VPN policies: ACL, NAT …

– 256 contexts per FW

– Map to VLANs

I-Net

FW Contexts

Shared Services

VPN A

VPN B

VPN C

VPN D

Fusion VDC

Context functionality available on the ASA

26


Virtual Firewall per VRF

VDC or VRF Sandwich Design

Virtual firewalls assigned to VRF by VLAN association

One pair of physical or virtual firewall per VRF

Each firewall requires two VLANs; inside and outside

Firewall in transparent or routed mode

Can be made simpler by delegating default gateway functionality to the firewall

VDC-Agg

Active/Standby

VDC-Sub-Agg

VDC-Agg

VDC-Sub-Agg

VRF A

VRF B VRF C

27


Sample VRF w/ASA

Default Gateway

28


Traffic Flow w/Server Load Balancer

VDC-Agg

VDC-Sub-Agg

VLAN 5

VLAN 12

VLAN 10

Transparent

802.1q trunk

VLAN 4

VLAN 13

Transparent

VLAN 11

S-NAT

VDC-Agg

VDC-Sub-Agg

VLAN 5

VLAN 12

VLAN 10

Transparent

802.1q trunk

VLAN 4

VLAN 13

Transparent

VLAN 11

Source NAT

from LB to

guarantee

symmetry

Bypass LB

when

needed

29


ASA Option: ASA Cluster Solution

Cluster up to 8 ASA appliances

• One unit is designated as master and rest are slaves

• All of them have a dedicated interface for Cluster Control Link (CCL)

• Keepalive/CP/DP messages, forwarded traffic and RPC are sent over CCL

Load balancing approach

• Stateless load balancing by external switch(EtherChannel) or router(Policy Based Routing, Equal-Cost Multi-Path Routing)

• Conn-rebalance between cluster units over CCL

Clu

ste

r C

ontr

ol Lin

k

30


ASA Option: Solution Overview (cont)

Fully distributed data-path so there is no single point of failure

• All units are active

Coordinated data-path to locate the owner and forward packets through the cluster to accomplish stateful firewall inspection

• Multiple roles for a connection in data-path among units (owner, director, forwarder etc.)

• Consistent hashing algorithm to redirect packets within cluster

In-Cluster High Availability

• Connection state sharing/backup between units

• N + 1 redundancy

• Hitless upgrade

Centralized management and monitoring

31


Zone Based Firewall w/ASR1000

Hardware Based Performance

IOS Based

Zone-pair

VRF-aware

Fusion VRF (Gray VRF in later slides)

Native MPLS Connectivity

Per Zone Firewall Policy

I-Net

FW Zone-Pairs

Shared Services

VPN A

VPN B

VPN C

VPN D

Fusion VRF

VRF-Aware ZBFW

Internal Link

32


Cisco ASR1000: VASI Feature

A point to point virtual link

Internal to the router

Connects two VRFs together

Allows for direct peering (IGP/BGP)

Allows for ACLs, NAT, WCCP etc

VRF aware firewall applied prior to traversing the virtual link

33












Simplicity!

34


Extend VPN services over multiple independently managed MPLS domains

Fast geographic service coverage expansion

Two MPLS VPN Providers peering to cover for a common customer base

Build MPLS VPN networks on original multi-domain network

IGP isolation with service continuity

Interconnect BGP confederations with different IGPs in the same AS

Two available as described in RFC 4364 : 1.Carrier Supporting Carrier (CSC)

2.Inter-Autonomous Systems (I-AS)

MPLS Inter-AS Use Cases

AS3 DC2

WAN Core (AS2)

AS1 DC1

Cust1 Cust1

Cust2 Cust2

35


Extending MPLS with Inter-AS

VPN-R1 VPN-R2

PE22

CE2 CE1

AS #1

MPLS AS #2

MPLS

PE11

MP-eBGP for VPNv4

(Option B)

Multihop MP-eBGP

between RRs

(Option C)

MP-eBGP+Labels

Back-to-Back VRFs

(Option A) ASBR1 ASBR2

Option C: Most interesting since we offload the VPN routes from ASBRs

36












Simplicity!

37


Deployment & Implementation Scenarios

P-to-P tunneling (MPLS MPLS)

IP WAN Transport

IPSEC Option for security

P to P Tunnel

Looks like an MPLS Link

Drawbacks:

– Cumbersome with multiple sites (MPLSoMGRE is an alternate solution)

– MTU

IGP Label

VPN Label

IP Payload

GRE Header

IGP Label’

VPN Label

IP Payload

MPLSoGRE

IP NetworkMPLSDC1

MPLSDC2

PE1 PE2P1 P2

IGP Label

VPN Label

IP Payload

38


Pulling the Building Blocks Together

39

MPLS

FabricPath

Firewalls

WAN

Design Options


MPLS Layer 3 VPN – Multi-POD

Requirement:

Secure Segmentation for Hosted / Enterprise Data Centers or Campus networks via MPLS VPNs

Solution:

One MPLS network infrastructure for all services

MPLS PE boundary in POD EoR/ToR access/ aggregation layer

Below MPLS boundary: L2 or L3 (VRF-lite with PE-CE)

Direct PE-PE or PE-P-PE networks

Scaling POD architecture without operational overhead using Fabric Extenders

POD POD POD

L3 L2

Internet

Global Interconnect Campus

/WAN Edge

MP

LS

Laye

r-2

41


Starting Point

42

Default Gateway

FabricPath

Spine Layer (N7k)F2e


Design Option Leveraging FabricPath Zone Based Design

43

• Segmentation by separating default gateway

• Each segment considered a Zone

• Each Zone has unique FWs and LBs

• Can leverage VDCs

• Simple

Default GatewayZone1

FabricPath

Spine Layer (N7k)FabricPath Only

CORE

Default GatewayZone2

LB

LB

vPC or FP

LB

LB

vPC or FP


Design: Firewall Placement w/Virtualization

44

Default Gateway

FabricPath

Spine Layer (N7k)

LB

LB

COREOption1 Option2

Default Gateway

FabricPath


MPLS


Option1: Traffic Flow

45

Default Gateway

FabricPath

Spine Layer (N7k)

LB

LB

CORE

Default Gateway

FabricPath

Spine Layer (N7k)

LB

LB

CORE

Intra VRF

Inter VRF


Option1: Solution w/ASA Cluster

46

Default Gateway

FabricPath

Spine Layer (N7k)

LB

LB

CORE

Inside Outside

• Use ASA cluster for firewalling

• One ASA context per virtual segment

• Scale up by growing ASA cluster and add additional clusters

• VRF or VDC sandwich design

• Core layer is simple. No VRFs.

• Traffic symmetry is automatically handled by ASA cluster


Option2: Traffic Flow

47

Default Gateway

FabricPath


MPLS

Default Gateway

FabricPath


MPLS

Intra VRF

Inter VRF


Comparing Options

Option1: ASA Firewall

– Scales up by way of distributing customers to firewalls and leveraging clusters

– Stateful HA

– Purpose built hardware

– Management tools

– Inter-VRF traffic flow leverages spine layer

Option2: ASR1k ZBFW:

– MPLS attached

– Additional services like NAT and WCCP

– Hardware forwarding

– No concerns about trunking VLANs

There is absolutely nothing wrong with going with either option. The choice is dependent on many factors such as requirements, comfort level with product, management and operations etc.

48


Option2: Zone Based Firewall (ZBFW)

49

QFP

MPLS

QFPZBFW ZBFW• ASR1k Hardware Performance

• Native MPLS Attachment

• VRF-Aware

• Attach Anywhere with MPLS Reachability


Option2: ZBFW w/VASI Details

50

GrayVRF

VRFs 100-199

vasi

MPLS

LDP

vasi

BGP or OSPFOver VASI

ASR1k

Per-VRF Security Policy Applied Before

Traversing VASI

• Native MPLS termination

• Gray VRF interconnects tenant VRFs

• Leverage VASI

• Each ‘tenant’ gets a security policy zone-pair

• NAT possible and WCCP Possible on VASI


Option2: Firewall Design w/Zone Based Firewall

51

• Redundancy by way of routing

• Active/Standby

• Leverage metrics

• Limiting factors:

• Throughput

• Number of connections

• Number of conn/sec

GrayVRF

GrayVRF

VRFs 100-199 VRFs 100-1

99

vasi

MPLS

vasi

Services VRF



52

• Per-VRF loadbalancing

• N+1 redundancy

• Very scalable design

• Grow as you go

• Scalability is additive



53

• Second Gray VRF for further segmentation

• Same logic as before

• Per-vrf loadbalancing

• Grow as you go


Inter-DC Flow Connectivity

54

Inter-DC: Same VRF

WAN Core

DC1 DC2

Cust1_VRF1 Cust1_VRF1

• Symmetric traffic flow is critical

Inter-DC: Different VRFs

WAN CoreFW-DC1

FW-DC2

DC1 DC2

Gray VRF Gray VRF

Cust1_VRF1 Cust1_VRF2

Supernet or Defaultr oute

Supernet or Defaultr oute


Customer Selected

55

• Spine/leaf architecture

• FabricPath for L2 multi-pathing

• No spanning tree

• Default gateway at spine layer

• ASR1ks w/ZBFW for firewall layer

• Nexus 5k/2k at the access Default Gateway

FabricPath


QFP QFP

MPLS

Inter-VRF Firewalls

ALG ALG

ASR9kASR9k

ASR9k ASR9k


How do I Scale Up?

56

QFP QFPQFP QFP QFP

FabricPath FabricPath FabricPath

MPLS MPLSMPLS

MPLS

P-Layer

QFP QFPRoute Reflector Route Reflector

Firewall Layer

PE LayerPE Layer

WAN Design


WAN Requirements

Highly available

IGP reconvergence or instability should not affect other DCs

Minimize state in the WAN

Add/remove data centers without network outage

Connect DCs with fiber, leased lines and encrypted tunnels

Traffic engineering

58


A WAN Core Layer – Dual Plane

59

• IGP Isolation between each plane

• Isolate topology changes

• Flexible topology

• Highly redundant

• Similar to two provider environments

• Traffic Engineering


Back to Design

60

• WAN Core routers are co-located in major DCs

• DC Core routers connect directly to WAN core routers

• No connection between WAN core routers

Default Gateway

FabricPath


QFP QFP

MPLS

Inter-VRF Firewalls

ALG ALG

ASR9kCore1

ASR9kCore2

ASR9kWAN1

ASR9kWAN2


A WAN Core Layer – With Inter-AS

DCs connect using dark fiber, GRE, or leased lines

The IGP used in the WAN core is separate

DCs peer to the WAN core using eBGP

Inter-AS option C

– Only feed infra routes to WAN Core

– VPN exchanged between RRs at each DC

Advantages:

– Scale & Flexibility

– IGP Isolation

– Adding/removing DCs is seamless

– High level of HA

61


Summary

CLOS Architecture for Scale and Flexibility

FabricPath for any VLAN Anywhere in the DC

Spine layer with Integrated MPLS PE

Firewalls Native Attached to MPLS

Scalable Architecture

Grow as you Go

Highly Flexible WAN that Scales and Highly Redundant

Flexible Growth with Multiple DCs

62


Lessons Learned - 1

FabricPath Scale

– MAC, SVI and VLAN limits

– Topology size (number of switch-IDs) and links

Active/Active HSRP

– Requires either vPC or GLBP today

– Anycast HSRP in the 6.2 release. Requires a new release on the N5k (roadmap)

F2e/M2 mix mode with proxy routing mode

– Requires 6.2 release of NX-OS

– All routing including SVI routing done on the M2

– Relationship to oversubscription ratio

– Increases MAC address scale to 128k

Firewall design

– Asymmetric routing challenges with ASR1k. Requires BGP metric

– DC to DC flows with symmetry. Requires supernet routes 63


Lessons Learned - 2

Inter-AS

– Option C not supported on N7k yet (roadmap)

GRE: MTU requirement

Routing over VASI:

– OSPF and iBGP were possible options over VASI initially

– eBGP support with local-AS/Remote-AS support in 3.7.2 release on the ASR1k

– Deciding on which routes to advertise from Gray VRF requires BGP filters

MPLS PE placement

– VRF-lite harder to manage and operate

– Using the N7k F2e/M2 mix makes the design way simpler

Virtual firewalls, like the ASA1000v, would make an interesting solution

64


R25

SP

AS100

172.16.81.0/24

E0/3

E0/1

172.16.80.0/24

LTRMPL-3102 (Advanced MPLS Lab)

65

R1

R5

R6

R2

R4

R3

10.1

.14.

0/24

10.1.15.0/24

10.1.16.0/24

10.1

.17.

0/24

10.1.18.0/24

10.1.19.0/24

E0/3E0/3

E1/0

E0/3

E0/3

E1/0

E1/0

E1/0

R7

R8R13

R15

R16

R14

FW1

Host1

Host2

AS65001DC1

10.1

.1.0

/24

10.1.2.0/24

10.1.3.0/24

192.168.1.0/24

192.168.2.0/24

10.1.10.0/24

10.1.11.0/24

10.1

.12.

0/24

10.1.13.0/24

E0/0

E0/0

E0/0

E0/0

E0/0

E0/3

E0/0

E0/2

E0/0

E1/0

E0/0

E0/2

E1/0

E0/1

E0/2

E0/3

E1/0

E0/3

E0/2

PE1

R9 R10

R17

R19

R20

R18

FW2

Host3

Host4

AS65002DC2

10.1

.4.0

/24

10.1.5.0/24

10.1.6.0/24

192.168.3.0/24

192.168.4.0/24

10.1.20.0/2410.1.21.0/24

10.1.22.0/24

E0/0

E0/0

E0/0

E0/3 E0/3

E0/0 E0/2

E0/0

E0/1

E1/0 E1/0

E0/2

E0/2 E0/0

E0/0

E0/2

E0/0 E0/2

E0/3

E1/0

PE2

R12

R11R21

R23

R24

R22

FW3

Host5

Host6

AS65003DC3

10.1.7.0/24

10.1.8.0/24

10.1

.9.0

/24

192.

168.

5.0/

24

192.168.6.0/24

172.16.82.0/24

172.16.83.0/24

E0/0

E0/0

E0/0

E0/0E0/3

E0/3E0/0

E0/0

E0/0

E1/0

E1/0

E0/1

E0/2

E0/2

E0/3

E1/0

PE3


R25

SP

AS100

172.16.81.0/24

E0/3

E0/1

172.16.80.0/24

R1

R5

R6

R2

R4

R3

10.1

.14.

0/24

10.1.15.0/24

10.1.16.0/24

10.1

.17.

0/24

10.1.18.0/24

10.1.19.0/24

E0/3E0/3

E1/0

E0/3

E0/3

E1/0

E1/0

E1/0

R7

R8R13

R15

R16

R14

FW1

Host1

Host2

AS65001DC1

10.1

.1.0

/24

10.1.2.0/24

10.1.3.0/24

192.168.1.0/24

192.168.2.0/24

10.1.10.0/24

10.1.11.0/24

10.1

.12.

0/24

10.1.13.0/24

E0/0

E0/0

E0/0

E0/0

E0/0

E0/3

E0/0

E0/2

E0/0

E1/0

E0/0

E0/2

E1/0

E0/1

E0/2

E0/3

E1/0

E0/3

E0/2

PE1

R9 R10

R17

R19

R20

R18

FW2

Host3

Host4

AS65002DC2

10.1

.4.0

/24

10.1.5.0/24

10.1.6.0/24

192.168.3.0/24

192.168.4.0/24

10.1.20.0/2410.1.21.0/24

10.1.22.0/24

E0/0

E0/0

E0/0

E0/3 E0/3

E0/0 E0/2

E0/0

E0/1

E1/0 E1/0

E0/2

E0/2 E0/0

E0/0

E0/2

E0/0 E0/2

E0/3

E1/0

PE2

R12

R11R21

R23

R24

R22

FW3

Host5

Host6

AS65003DC3

10.1.7.0/24

10.1.8.0/24

10.1

.9.0

/24

192.

168.

5.0/

24

192.168.6.0/24

172.16.82.0/24

172.16.83.0/24

E0/0

E0/0

E0/0

E0/0E0/3

E0/3E0/0

E0/0

E0/0

E1/0

E1/0

E0/1

E0/2

E0/2

E0/3

E1/0

PE3

66


Related Sessions

67

Session-ID Session Name

BRKMPL-1100 Introduction to MPLS

BRKMPL-2100 Deploying MPLS Traffic Engineering

BRKMPL-2102 Deploying MPLS-based IP VPNs

BRKMPL-2109 MPLS in Multi-Tenant Data Centers

LTRMPL-2102

LTRMPL-3102

Enterprise Network Virtualization using IP and MPLS

Technologies (intro and adv)

BRKDCT-2081 Cisco FabricPath Technology and Design

BRKDCT-2121 Virtual Device Context (VDC) Design and Impl.

BRkSEC-2021 Firewall Architectures


Maximize your Cisco Live experience with your

free Cisco Live 365 account. Download session

PDFs, view sessions on-demand and participate in

live activities throughout the year. Click the Enter

Cisco Live 365 button in your Cisco Live portal to

log in.

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Daily Challenge points for each session evaluation you complete.

Complete your session evaluation online now through either the mobile app or internet kiosk stations.

68

designing mpls in next generation data center: a case...

Documents