over the top routing protocols joe harris consulting systems engineer

Over the Top Routing Protocols

Joe HarrisConsulting Systems Engineer

© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

Agenda

• Overview of Current Solutions• How OTP works• Peering over the WAN• Considerations• Case Study

© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

Overview of Current WAN Solution

• Allow customers to segment their network using an MPLS VPN backbone

• Impose little requirements or no restrictions on customer networks– Customer sites may be same or different Autonomous Systems– Customer sites may consist of multiple connections to the MPLS VPN backbone– Customer sites may consist of one or more connections not part of the MPLS VPN

backbone (“backdoor” links)

PE-CE Overview

3

PE1 PE2

CE1 CE2

MPLS VPNCloud

Backdoor LinkSite 1 Site 2



• Service Provider must redistribute and carry Enterprise routes via MP-iBGP;– Route flaps within sites results in BGP convergence events– Route metric changes results in new extended communities flooded into the core

• Either EIGRP or eBGP must be run between the PE/CE– Provider had to have trained staff on hand to manage

PE/CE Link– Provider’s often prefer vender flexibility

• Provider must be be involved with deploymentchanges in enterprise customer’s network

PE-CE Issues for the Service Provider

PE1 PE2

CE1 CE2

MPLS VPNCore

Site 2Site 1



• Managed services is required, even if not needed– Provider often limits number of routes being redistributed

• Enterprise and Service Provider must co-support deployment– Control of traffic flow using multiple providers is problematic– Changing providers results in migration issues

• Service Provider route propagation impactssite to site convergence

• Redistribution at the edge leads to additional complexity and operational costs for an Enterprise.

• Carrier involvement restricts network design change and evolution

PE-CE Issues for the Enterprise

PE1 PE2

CE1 CE2

MPLS VPNCore

Site 2Site 1



• Route redistribution adds deployment complications– Without PE/CE support, back-door must be redistributed into a second instance of EIGRP– With PE/CE support, use of SoO (route) tagging must be used to prevent count-to-infinity issues due

to BGP’s slower convergence and all routers between CE an Backdoor must have support for SoO

PE-CE Issues with Backdoor Links

CE1

CE2

Backdoor Link

C3

PE1 PE2

CE1 CE2

MPLS VPNCloud

Site 2Site 1

C4

CE2


Problem Solution

• EIGRP OTP provides a single end-to-end EIGRP routing domain transparent to the underlying Public or Private WAN transport .

• EIGRP OTP can hide the complexity of BGP-4 or other routing protocol used as their interface to the network transport provider.

• Reduces L3-VPN costs by reducing the required customer routes in the VPNv4/v6 address family.

Service Provider

MPLS VPN

= Data Plane= Control Plane

Customer Site 1 Customer Site 2


WAN Virtualization using OTP

• EIGRP “end-to-end” solution with:NO special requirement on Service Provider

NO special requirement on Enterprise

NO routing protocol on CE/PE link

NO need for route redistribution

NO no need for default or static routes

EIGRP OTP supports transparent CE to CE Routing

8

PE / CE

BGPComplexity

Carrier Involvement

Multiple Redistributio

n

Public & Unsecure

EIGRP

OTP

EIGRP Simplicity

Carrier Independence

Zero Redistribution

Private & Secure

Availability- ISR 4451-X, ASR 1K Series – IOS-XE 3.10- ISR, ISR G2, 7200 Series – IOS 15.4(1)T



• No additional routing protocol to administer– No routing protocol is needed on CE to PE link– All user traffic appears and unicast IP data packets

• Limit impact on Service Providers Network– Customer routes are NOT carried in MPLS VPN backbone– Customer route flaps do not generate BGP convergence events– Smaller BGP routing tables, smaller memory foot print, lower CPU usage

• Works with existing PE equipment– Multivendor PE support– No upgrade requirements for PE or any MPLS VPN backbone router

Service Provider Benefits



• Single routing protocol solution– Simple configuration and deployment for both IPv4 and IPv6– Convergence is not depending on Service Provider– Only the CE needs to be upgraded

• Routes are carried over the Service Provider’s network, not though it– No artificial limitation on number of routes being exchanged between sites– Convergence speed not impacted by BGP timers

• Works with both traditional managed and non-managed internet connections– Compliments an L3 Any-to-Any architecture (optional hair pinning of traffic)– Support for multiple MPLS VPN connections– Support for connections not part of the MPLS VPN (“backdoor” links)

Enterprise Benefits


WAN Solution OverviewEIGRP OTP DMVPN / Internet MPLS VPN MPLS+DMVPN

Provider Dependence No No Yes Yes/No

Control Plane EIGRP IGP/BGP + NHRP;LAN IGP

eBGP/iBGP;LAN IGP

IGP/BGP + NHRP;eBGP; LAN IGP

Data Plane LISP mGRE IP IP + mGRE

Privacy GETVPN IPSec over mGRE GETVPN GETVPN + DMVPN

Routing Policies EIGRP, EIGRP Stub EIGRP Stub Redistribution and route filtering

EIGRP Stub, Redistribution, filtering, Multiple AS

Network Virtualization VRF/EVN to LISP multi-tenancy

DMVPN VRF-Lite; MPLS o DMVPN

Multi-VRF CEs and multiple IP VPNs

Multi-VRF Ces and DMVPN VRF-Lite

ConvergenceBranch/Hub

Branch Fast;Hub – Fast

Branch Fast;Hub - Fast

Branch / Hub carrier dependent

Carrier and DMVPN hub dependent

Multicast Support Planned XE3.14 PIM Hub-n-Spoke PIM MVPN MVPN + DMVPN Hub-n-Spoke

VRF Support Planned XE3.15 Yes No No


OTP – How it Works

• CE Routers have ‘private’ and ‘public’ interfaces & routers exchange information using unicast packets– Private interfaces use addresses that are part of the Enterprise network– Public interfaces use addresses that are part of the Service Providers network– For OTP neighbors to form, the Public interface must also be included in the EIGRP topology

database (covered by the “network” command in IPv4)

• Packets are sourced from/to the public interface address eliminates the need for static routes– EIGRP packets which are normally sent via multicast (Hello, Update, etc..) are sent unicast via the

public interface– Site-to-site traffic is encapsulated using LISP and sent unicast from/to the public interface address



• EIGRP creates the LISP0 interface, and starts sending Hello packets to remote site via the Public interface

• Once neighborship is formed, EIGRP sends and receives routes from the peer, installing the routes into the RIB with the nexthop interface LISP0

• Traffic that arrives on the router destined for the remote side, is first sent to LISP0

• LISP encaps the traffic and then sends it to the Public interface

EIGRP, LISP, and RIB – Oh My!

EIGRP

PublicInterface

InsideInterface

DefaultTraffic

Site to Site

Traffic

RIB

RouteUpdates

LISP0


OTP – Data Plane

• Why use LISP to encapsulate the data as it traverses the WAN?

• Its “stateless” tunneling, so it;– Requires NO tunnels to configure or manage– Is transparent to the endpoints and to the IP core– Supports both hair-pin and site-to-site traffic– Supports both IPv4 and IPv6 traffic

• Provides an overlay solution that enables transparent extension of network across WAN– IP-based for excellent transport independence– Service provider picks optimal traffic path for site to site data– Supports multicast and VLANs to allow for future enhancements

LISP Data Encapsulation


OTP – Data Plane

• Path MTU needs to be considered when deploying OTP– LISP encapsulation adds 36 bytes (20 IP + 8 UDP + 8 LISP) for IPv4

(56 bytes for IPv6)– This could be significant for small packets (e.g., a VoIP packet)

• LISP handles packet fragmentation– If the DF bit is set, it will generate an ICMP Destination Unreachable message

• LISP does not handle packet reassembly– As a consequence, it is required to adjust the MTU to ensure the control plan does not

fragment– Best practice - set the MTU is set to to 1444 (or lower) bytes.

LISP Data Encapsulation Properties



• EIGRP OTP is deployed in one of two ways

• Remote Routers– Used for configuring a router to peer with one specific neighbor– Forms a full mesh topology– Configured with the command

• Route Reflectors– Used to configure a router as a ‘hub’ – Forms a Hub and Spoke topology– Configured with the command

Modes of Deployment

17

remote-neighbors source [interface] unicast-listen lisp-encap

neighbor [ipv4/v6 address] [interface] remote [max-hops] lisp-encap [lisp-id]


Peering over the WAN

• Remote Routers

• Route Reflectors

• Redundant Remote Routers

• Redundant Route Reflectors


• Control Plane peering is accomplished with EIGRP “neighbor” statement– CE-1 sends unicast packets to CE-2’s public address (192.168.2.2)– CE-2 sends unicast packets to CE-1’s public address (192.168.1.1)

• Data Plane packet delivery is accomplished with LISP encapsulation

DATALISP

Remote RoutersPoint to Point Peers

Service Provider

MPLS VPNEIGRP

AS 4453

EIGRPAS 4453

Hello Hello

router eigrp ROCKS address-family ipv4 unicast auto 4453 neighbor 192.168.2.2 Serial1/0 remote 100 lisp-encap ...

router eigrp ROCKS address-family ipv4 unicast auto 4453 neighbor 192.168.1.1 Serial1/0 remote 100 lisp-encap ...

DATADATA CE-1 CE-2


Remote Routers

CE2# 00:01:57: %DUAL-5-NBRCHANGE: EIGRP-IPv4 4453: Neighbor 192.168.2.2 (Serial1/0) is up: new adjacencyCE2# CE2#show eigrp address-family ipv4 neighbors detailEIGRP-IPv4 VR(ROCKS) Address-Family Neighbors for AS(4453)H Address Interface Hold Uptime SRTT RTO Q Seq (sec) (ms) Cnt Num0 192.168.2.2 Se1/0 13 00:01:15 171 1026 0 21 Remote Static neighbor (static multihop) (LISP Encap) Version 16.0/2.0, Retrans: 0, Retries: 0, Prefixes: 5 Topology-ids from peer - 0 Max Nbrs: 0, Current Nbrs: 0CE2#

Remote Peers

20


Remote Routers

• In order to form peers, the Public interface must be enabled for EIGRP

• For IPv4, you must include a ‘network’ statement to cover the public interface

• This does not mean the ip address of the remote peer has to match the network/mask of the public interface

• The interface is used to send packets,so the IP address of the remote peerjust has to be reachable via the WAN

Remote Peers address properties

interface Serial1/0 description Service Provider ip address 172.16.0.1 255.255.255.0!router eigrp ROCKS ! address-family ipv4 unicast auto 4453 ! topology base exit-af-topology neighbor 192.168.2.2 Serial1/0 remote 100 lisp-encap network 172.16.0.0 0.0.0.255 network 10.1.0.0 0.0.255.255exit-address-family



• Remote Routers





Route Reflectors

• EIGRP Route-Reflectors simplifies setting up multiple branches

• Chose one of the CE routers to function asRoute Reflector (RR)– Purpose of the Route Reflector is to

‘reflect’, or advertise routes received toother CE routers

• Control plane is deployed in a“Hub-and-spoke” topology

Point to Multi-Point – Multiple Branch Sites

router eigrp ROCKS address-family ipv4 unicast auto 4453 remote-neighbors source Serial 0/0 unicast-listen lisp-encap network 10.0.0.0

RR = DP

= CP

Site 3 Site 2

Site 1

CSCuj68811:

15.4(1.16)S0.2, 15.4(1.16)S0.315.4(1.16)S0.4, 15.4(2.1)S15.4(2.2)S


Route Reflectors

• Question:In the example, if CE in Site 1 advertises aroute to the Route Reflector, will the routepropagate to other CE routers?

• Answer: No!– The split horizon rule prohibits a router

from advertising a route through aninterface that it uses to reach thedestination.

• Solution:– In order for the route to be ‘reflected’

to the other sites, use theno split-horizon command on thepublic interface

Point to Multi-Point – Multiple Branch Sites

router eigrp ROCKS address-family ipv4 unicast auto 4453 remote-neighbors source Serial 0/0 unicast-listen lisp-encap network 10.0.0.0 af-interface serial 0/0 no split-horizon exit-af-interface

Site 3 Site 2

Site 1

CSCuj68811:

15.4(1.16)S0.2, 15.4(1.16)S0.315.4(1.16)S0.4, 15.4(2.1)S15.4(2.2)S


Site 3

Route Reflectors

• EIGRP Route Reflector simplifies adding additional branches

Point to Multi-Point – Adding Branch Sites

25

• Configure the new CE to point to the RR• New CE and RR exchange routes, and

RR sends new routes to other CEs• Adding additional CE routers does not

require changes to configurationof the Route Reflector

25

Site 2

address-family ipv4 unicast auto 4453 neighbor 192.168.1.1 Serial 0/2 remote 100 lisp-encap network 10.0.0.0 network 192.168.0.0 0.0.255.255 ...

Site 4

RR = DP

= CP

Site 1


Route Reflectors

• Each CE normally shows the Route Reflector (RR) as the next hop– Data will ‘hairpin‘ though the RR to get to other sites– Useful for applying Policy and filtering traffic– Will increase bandwidth requirements for the Route Reflector

• What if I want to send traffic directlyfrom site to site?

• The solution: 3rd Party Nexthops

Point to Multi-Point – Any-to-Any Data

26

Site 3

CSCuj68811:

15.4(1.16)S0.2, 15.4(1.16)S0.315.4(1.16)S0.4, 15.4(2.1)S15.4(2.2)S

Site 2

RR = DP

= CP

Site 1


Route Reflectors

• Normally the Route Reflector would send the nexthopas 0.0.0.0 which tells CE1 and CE2 to use it to reachthe destination

• When “no next-hop-self” configured, the RR preservesthe next hop of the peer that sent it the route

• When CE1 and CE2 receives an update from theRR, they install the route in the RIB with thesupplied nexthop

• Traffic leaving CE1 goes directly to router CE2

3rd Party Nexthops

RR

CE1 CE2

10.1.1.0/24

.3.2

.1

EIGRP-IPv4 VR(ROCKS) Topology Table for AS(4453)/ID(10.0.0.1)....P 10.1.1.0/24, 1 successors via 192.168.1.3

router eigrp ROCKS address-family ipv4 auto 4453 af-interface Serial0/0 no next-hop-self



• Remote Routers





Redundant Remote Routers

• In an OTP setup, an RR can learn two or more equal-cost paths to a site.

• However, the RR router will only advertise one of the paths to other spokes in the OTP network.

• Implication:– Site to Site traffic will only be sent to one router– Sites are not able to leverage multi-router

setups

Multiple Next Hops10.2.0.0 [90/18600] via 192.168.1.5, LISP0 via 192.168.1.6, LISP0

10.2.0.0 [90/32600] via 192.168.1.5

Site 2

10.2.0.0/24

.5 .6

Site 3

Site 1RR



• While this isn't a route propagation problem, per se, it's still a situation that may take you by surprise and therefore may be useful to understand

• One of the designs being implemented with OTP uses multiple paths from the hub to reach spoke subnets. This could be two paths to the same spoke or through two spokes (as shown on the previous slide)

• The problem is that EIGRP still uses normal distance vector rules and sends updates based on the top topology table entry.

• Even if there are two equal cost paths, EIGRP sends updates based on the top entry, even though there are two paths available.

Multiple Next Hops



• To avoid this situation and enable Remotes to use all paths, configure the “add-path” option on the RR (hub)

• Add Path Support enables the Route Reflector to advertise multiple best paths

• Up to 4 additional Nexthops addresses(5 total)

Solution: Add-Path

Site 2

10.2.0.0/24

.5 .6

Site 3

Site 1RR

router eigrp ROCKS address-family [ipv4 or ipv6] unicast auto 4453 af-interface serial 0/0 no split-horizon no next-hop-self add-path 1 exit-af-interface remote-neighbors source Serial 0/0 unicast-listen lisp-encap

10.2.0.0 [90/32600] via 192.168.1.5 via 192.168.1.6



• Remote Routers





Redundant Route Reflectors

• Adding a second Route Reflector does not change the original Route Reflector’s, configuration

• On the Remote Routers, add the new remoteneighbor configuration for the new Route Reflector

• Remotes do not have to be configure to connectto all Route Reflectors

Adding second RR

router eigrp ROCKS address-family ipv4 unicast auto 4453 neighbor 192.168.10.1 Serial0/1 remote 100 lisp-encap neighbor 192.168.20.2 Serial0/2 remote 100 lisp-encap ...

RR-2RR-1

Site 1



• If the Route Reflectors are in different sites, you may want to exchange routing information between the Route Reflectors

• You might be tempted to setup a remote neighbor;

• Don’t! This is not supported.

• Instead, consider adding a GRE tunnel betweenthe Route Reflectors, and share routing information

Exchanging routes between RR’s

router eigrp ROCKS address-family ipv4 unicast auto 4453 remote-neighbors source Serial 0/0 unicast-listen lisp-encap neighbor 192.168.2.2 Seral0/2 remote 100 lisp-encap ...

Site 1Site 2

RR-2RR-1



• Support for additional Service Providers is also possible

• Choose a Route Reflector per Service Provider to ensure each CE hasreachability to other sites

• Normal EIGRP metric selection and costing will influence path selection

Support for Multiple Providers

ISP1

RR-1

Site 1 Site 2 Site 3

ISP2

RR-2


Deployment Considerations

• Route Filtering

• Backdoor Links


Route Filtering

• When you setup an OTP peer, you must add a network statement covering the public interface

• This means the public network will show up in the EIGRP topology database;– EIGRP will split-horizon the local public address out the

public interface– EIGRP will advertise to EIGRP neighbors on the LAN

interface– EIGRP will advertise any public address it receives via the

LAN from another neighbor over the WAN

• Generally this is not an issue… however…

Limiting leaking of public routes into the LAN

10.2.0.0/24

.20.13

Site 2

.31.14

address-family ipv4 unicast auto 4453 neighbor 192.168.1.1 Serial 0/2 remote 100 lisp-encap network 192.168.0.0 0.0.255.255 network 10.2.0.0 0.0.255.255 ...


Route Filtering

• Looking on the Route Reflector we see the new peer come up..

• But we also see an traffic is being drop due to the LISP encapsulation failure

Limiting leaking of public routes into the LAN

CE1#02:24:05: %DUAL-5-NBRCHANGE: EIGRP-IPv4 4453: Neighbor 192.168.31.14 (Serial1/0) is up: new adjacency

02:24:07: %CFC_LISP-5-ADJ_STACK: Stacking adjacency IP adj out of LISP0, addr 192.168.31.14 (incomplete) onto other LISP adjacency IP midchain out of LISP0, addr 192.168.20.13 F0732BB8 forcing drop

CE3#ping 192.168.31.14 Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.168.31.14, timeout is 2 seconds:.....Success rate is 0 percent (0/5)


Route Filtering

• From “show ip route” We can see the public address is recursive though another public address– To get to 192.168.20.0/24, the packet needs to be sent to 192.168.32.14 though the

LISP interface– To get to 192.168.31.14, the route lookup for 192.168.0.0/24 also goes though LISP

interface

• Peers are not effected by the LISP encap failure as EIGRP sends packets directly to the public interface

Public routes subnets in the LAN can result in recursion issues

CE1#show ip route …D 192.168.20.0/24 [90/114980571] via 192.168.31.14, 00:00:29, LISP0D 192.168.31.0/24 [90/114980571] via 192.168.20.13, 00:23:10, LISP0


Route Filtering

• Best practice is to prevent the public networks from entering the LAN by filtering it at each CE

• Use “distribute-list out” to prevent public addressfrom leaking into customer site

Solution – filter public routes from being reached via the LAN

CE2b#sh run...router eigrp ROCKS ! address-family ipv4 unicast autonomous-system 4453 ! topology base distribute-list 10 out exit-af-topology neighbor 192.168.10.12 Serial1/0 remote 100 lisp-encap network 10.0.0.0 network 192.168.0.0 0.0.255.255 exit-address-family!access-list 10 deny 192.168.0.0 0.0.255.255access-list 10 permit any

10.2.0.0/24

.20.13 .31.14

Site 3



• Route Filtering

• Backdoor Links



• The use of “back-door” links for OTP does not require special handling– Path selection determined by setting ‘delay’ on backdoor links

Site to Site - Backdoor Links

• Use “distribute-list out” on CE’s to prevent address from leaking between sites

RemoteOffice

HeadquartersISP

Backdoor Link

CE

CE

C2C1

interface Serial0/0 delay 40000 . . .


Case Study


The Acme Corporation

Requirements:– Fast convergence (<1s if possible)– Direct Spoke-to-spoke traffic– 1600+ sites across four countries– Active/active load balancing– Encryption across WAN

Nice to have:– Easy provisioning

• No config changes on hubs as new sites are added• Zero touch deployment of branch wan router (CE)

– Provider flexibility• Multiple providers in each country• Easy migration between providers• No routing exchange of internal addresses



45

Corporate Backbone

France

… …

MPLS VPN

MPLS VPN

Sweden

… …

MPLS VPN

MPLS VPN

England

… …

MPLS VPN

MPLS VPN

USA

… …

MPLS VPN

MPLS VPN


The Acme CorporationRoute Exchange

46

Spokes

WAN Hubs

2 x ASR1000

… …

MPLS VPN for Branches and ATMs

B


A

RRRR


The Acme CorporationWAN Security with GET VPN

47

KEY SERVER

MEMBER MEMBER

WAN Services

2 x 3945E

WAN Hubs

2 x ASR1000

MEMBERS

… …

RR RR


B


A



– IGP speeds via end-to-end EIGRP solution– Use of no nexthop-self on RR– Up to 500 EIGRP spokes per RR– Ability to add 4 additional ECMP via addpath – GET VPN

– Route Reflectors – Route Reflectors

– Multiple neighbor configs supported – Built into OTP – Built into OTP

Requirements:– Fast convergence (<1s if possible)– Direct Spoke-to-spoke traffic– 1600+ sites across four countries– Active/active load balancing– Encryption across WAN

Nice to have:– Easy provisioning

No config changes on hubs as new sites are added Zero touch deployment of branch wan router (CE)

– Provider flexibility Multiple providers in each country Easy migration between providers No routing exchange of internal addresses


Summary: What Have We Learned?

• WAN deployments are greatly simplified with OTP

• Both the Enterprise and Service Provide benefits from OTP

• EIGRP OTP supports both IPv4 and IPv6 deployments

• EIGRP’s scalability is an important factor in OTP deployment

• OTP can work over traditional WAN and LAN networks


For more Information on OTP

• EIGRP OTP:– http://www.cisco.com/en/US/docs/ios-xml/ios/iproute_eigrp/configuration/xe-3s/ire-eigrp-

over-the-top.html

• Open EIGRP (IETF Draft):– ftp://ftp.ietf.org/internet-drafts/draft-savage-eigrp-02.txt

• OTP OSPF (IETF Draft):– http://www.ietf.org/staging/draft-white-ospf-otc-01.txt

http://www.cisco.com/en/US/docs/ios-xml/ios/iproute_eigrp/configuration/xe-3s/ire-eigrp-over-the-top.html

http://www.cisco.com/en/US/docs/ios-xml/ios/iproute_eigrp/configuration/xe-3s/ire-eigrp-over-the-top.html

ftp://ftp.ietf.org/internet-drafts/draft-savage-eigrp-02.txt

ftp://ftp.ietf.org/internet-drafts/draft-savage-eigrp-02.txt

http://www.ietf.org/staging/draft-white-ospf-otc-01.txt

http://www.ietf.org/staging/draft-white-ospf-otc-01.txt

over the top routing protocols joe harris consulting systems engineer

Documents

cisco andor

service provider mpls

pece provider

control plane customer

network transport provider

soo pece issues

l3vpn costs

enterprise routes