sp edge aggr ha bcp.2013 eur pdf brkspg-2402

© 2013 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public

Best Practices to Deploy

High-Availability in Service Provider

Edge and Aggregation Architectures

BRKSPG-2402


Abstract

As Service Providers are deploying value-added triple-play or quadruple play services to maintain or generate a higher average revenue per

user, overall Service Availability becomes increasingly important. High Availability techniques such as Fast Convergence or MPLS TE FRR

have focused on raising the availability of the network core in the past. Recently, these techniques are being increasingly deployed in

Ethernet Aggregation networks, for example by introducing MPLS TE FRR in the aggregation. Also, additional high-availability mechanism are

being developed to enhance the resilience of the IP Edge against failures. Examples of new developments include IP Fast-Reroute, BGP

Prefix Independent Convergence for both the Core and Edge, or even stateful application inter-chassis redundancy mechanisms to overcome

single-system outages. This Session aims to provide the audience with best current practices to increase service availability by deploying

Cisco High-Availability mechanisms in both the Aggregation and the IP Edge. Traditional HA techniques such as NSF/SSO, BFD, Fast

convergence or NSR are reviewed. The details of new technologies such as IP FRR, BGP PIC are discussed in depth. Furthermore,

advanced topics such as achieving HA for Layer 4-7 services or stateful inter-chassis redundancy solutions are introduced. The Session also

provides the best current practices of deploying the tools offered by the Cisco High-availability toolset, in particular the deployment of MPLS

TE FRR in the aggregation. Furthermore, possible stateful and stateless clustering approaches are introduced, which SPs may use to

increase the availability of their IP Edge architecture.

For Your Reference

3


Glossary NHAT next hop address tracking EOBC Ethernet out of band management ACL Access Control List ESP Embedded Services Processor ACT Active EVC Ethernet Virtual Circuit APS Automatic Protection Switching EVDO Evolution Data Only ARP Address Resolution Protocol FECP Forwarding Engine Control Processor AS autonomous System FIB Forwarding Information Base ATM Asynchronous Transfer Mode FM Forwarding Manager BFD Bi Directional Forwarding Detection FR Frame Relay BNG Broadband Network Gateway FRR Fast Re Route

BW Bandwidth FSOL First Sign of Life CC Continuity Check FWLB Firewall Loadbalancing CC control connection GEC Gigabit Ether Channel CDR call detail record GLBP Global Load Balancing Protocol CE Customer Edge GR Graceful Restart CE Customer Edge GRE Generic Route Encapsulation CF checkpoint facility GW Gateway CFM Configuration and Fault Management HA High Availability CLI Command Line Interface HSRP Hot Standby Routing Protocol CM Chassis Manager HW Hardware CP Control Plane IETF Internet Engineering Task Force CPLD Complex Programmable Logic Device ? IF Interface CSC Carrier's Carrier IGP Internal Gateway Protocol DHCP Dynamic Host Configuration Protocol IOCP Input Output control Processor DP Data Plane IOS Internet Operating System DPM Defects per Million IP Internet Protocol DSLAM DSL Access Multiplexer IPC Inter process communication E2E End to end ISG Intelligent Services Gateway ECMP equal cost multipath iSPF incremental Shortest Path First EEM Embedded Event Manager ISSU in service software upgrade EOAM Ethernet OAM IWF Interworking function

For Your Reference

4


Glossary (Cont.) L2TP Layer 2 transport protocol NIC network interface card LAC L2TP access concentrator Nr receive sequence number LACP Link aggregation control Protocol Ns send sequence number LAN Local Area Network NSF non stop forwarding LC Linecard NSR non stop routing LDP label Distribution Protocol NVRAM non volatile random access memory LFA loop free alternate OAM operations, administration and maintenance LI Lawful Intercept OCE Object Chain Element LMI Local management interface OIR online insertion and removal LNS L2TP network Server OS operating system LOS Loss of signal PADR PPP active discovery LSDB link state database PE provider edge LSP label switched path PIC prefix independent convergence LTE long term evolution PIM protocol independent multicast MC LAG multi chassis link aggregation PPP Point to point protocol mcast multicast PS power supply MD5 message Digest algorithm 5 PSN Packet Switched Network MFIB multicast forwarding information base PTA PPP termination and aggregation MLD multicast listener discovery PVRSTP Per VLAN rapid spanning tree MME mobile management entity PW pseudowire MoFRR Multicast Only fast reroute QFP Quantum flow Processor MPLS Multiprotocol label switching RADIUS remote authentication dial in user service MRIB multicast routing information base RF redundancy facility MSC mobile switching center RMA Return material authorization MSPP Multi-service provisioning platform RNC radio network controller MST Minimum spanning tree RP route processor MTBF mean time between failures RPR route processor redundancy MTSO mobile telephone switching office RSP route switch processor MTTR mean time to repair RSVP resource reservation protocol NAT network address translation SAA service assurance agent

For Your Reference

5


Glossary (Cont.) SBC session border controller SBY standby SGW SAE gateway SIP Session initiation protocol SLA service level assurance SLB server loadbalancing SP service Provider SPA Shared port adapter SPF shortest path first SRLG shared risk link group SSH secure shell SSO stateful switchover STP spanning tree protocol SW software T&C terms & conditions TCAM ternary content addressable memory TE traffic engineering TR Traceroute UC unified communications uRPF unicast reverse path forwarding VAI virtual access interface VC Virtual Circuit VCCV VC connection verification VIP virtual IP VLAN virtual LAN VMAC virtual MAC VPN virtual private network VRF virtual routing and forwarding table VRRP virtual router redundancy protocol WAN wide area network

For Your Reference

6


Agenda

Motivation for High Availability in SP Aggregation Networks

Network Level High Availability

System High Availability

Service High Availability

Stateful Inter-chassis Redundancy

Case Studies

Summary and Conclusions

7

Motivation for High-Availability in

the IP Edge and Aggregation


High Availability and Service Level Agreements

Many SPs specify their SLAs in the T&Cs

Important characteristic of both business and residential services

Historically given for Core network, but expanding to end-end SLAs

Metrics

Service Availability (averaged over time)

Mean time to repair (MTTR)

Packet Loss / Delay / Jitter

Examples

AT&T

Sprint

Verizon Business

BT

Level 3

9

http://ipnetwork.bgtmo.ip.att.net/pws/index.html

http://www.sprint.com/business/resources/mpls_vpn.pdf

http://www.verizonbusiness.com/terms/us/products/vpn/remotecpe/

http://business.bt.com/assets/pdf/BTnet Service Level Agreement.pdf

http://www.level3.com/userimages/DotCom/pdf/3pkt_useng_global_letter_forscreen.pdf


What Is High Availability?

Availability DPM Downtime per Year (24x365)

99.000% 10000 3 Days 15 Hours 36 Minutes

99.500% 5000 1 Day 19 Hours 48 Minutes

99.900% 1000 8 Hours 46 Minutes

99.950% 500 4 Hours 23 Minutes

99.990% 100

53 Minutes

99.999% 10 5 Minutes

99.9999% 1 30 Seconds

Reactive

Proactive

“High Availability”

Predictive

Two ways to state availability of a network:

Percentage Method

DPM Method = Defects per Million (Hours of Running Time)

For Your Reference

10


Availability Definitions

MTBF is Mean Time Between Failure

When does it fail?

MTTR is Mean Time To Repair

How long does it take to fix?

‘Uptime divided by the total time’ to create the

percentage time your network is operational

Availability = MTBF

MTBF + MTTR

For Your Reference

11


Calculated vs. Measured Availability

Calculated Availability based on:

Network design

Component MTBF and MTTR

different underlying models, simulations

Cisco uses Industry standards to compute Hardware MTBF

Basic Availability Calculation Formulae:

Measured Availability based on:

ICMP Reachability (E2E, Device)

Cisco Service Assurance Agent (SAA)

Trouble Ticket / Outage Log Analysis

Observed Method: Shipping/RMA Method

= A1 ´A2 ´....´AN

=1- (1-A1)´....´(1-AN )

=

= N

k K

A 1

Series A

=

- - = N

K k

A 1

Parallel ) 1 ( 1 A

For Your Reference

12


Reduction of MTTR

Stateful inter-chassis redundancy allows for additional

resilience against

System Failures

Interface Failures

Issue is not really MTBF of hardware modules, but

rather

Line failures / optical path failures

Interface failures

Power outages

Goal of stateful inter-chassis redundancy is sub-

second failover with state preservation for applications

Product ID MTBF (hrs)

ASR1000-RP2 380532

ASR1000-ESP20 335317

ASR1000-SIP10 287549

ASR1006 1986649

ASR1006-PWR-AC 570776

ASR1006-PWR-DC 357781

ASR1000-SIP40 283225

ASR1000-ESP40 118790

SPA-8X1GE-V2 482023

SPA-1X10GE-L-V2 411892

For Your Reference

13


Device Availability Calculation

IOS Chassis and Backplane

P/S

P/S

IOS = = 0.999997 30.000 30.000 + 0.1

P/S = = 0.999995 750.000 750.000 + 4

System Availability = 0.999997 * ....*0.999983*(1-(1-0.999995)2) = 0.999961 = 99.9961%

IF1

CISCO7206-VXR

PWR-7200-AC

PA-E3 PA-POS-OC3

IF2 CPU

NPE-400

BB = = 0.999983 460.000 460.000 + 8

CPU = = 0.999992

490.000 490.000 + 4

12.0 GD Cisco 7206

IF1 = = 0.999996 1.120.000

1.120.000 + 4

IF2 = = 0.999993 600.000

600.000 + 4

Calculated MTBF Values from Cisco Database

For Your Reference

14


Availability of R1 and R2 in Series

= (0.999961*0.999961) = 0.99992175

1

Network Availability Calculation

Router R1,R2,R3, R4: 0.999961

Availability of Parallel Network Path (R1-R4)

= 1 - ((1-0.999921)(1 - 0.999921)) = 0.999999994

2

Network Availability = 99.9999%

Only Based on Device Availability Values

3

R1

R4 R3

R2

but not considered: -Links (WAN, LAN) -Computer NICs -Computer OS -Computer Applications

For Your Reference

15


Cisco High-Availability Focus

• increase MTBF using resilient HW/SW

• minimize MTTR for system failures (RP, LCs and SW)

• Mitigate planned outages by providing hitless software upgrades

System Level Resilience

• in the core and where redundant paths exist

• Deliver features for fast network convergence, protection & restoration

Network Level Resilience

• Embed intelligent event management for proactive maintenance

• Automation and configuration management to reduce human errors

Embedded Management

and Automation

16


Cisco HA Feature Toolbox: System Level

RPR, SSO

NSF, NSR

SSO Multirouter APS

Stateful NAT/IPSec/Firewall/SLB stateful failover within single chassis

MPLS HA (L3VPN, L2VPN, InterAS, CSC, TE, FRR)

IOS / IOS XR / IOS XE ISSU, dual IOS XE

Service Provider Core

Service Provider Edge

Service Provider Aggregation

Access Layer

Data Center Building Block

Internet

17


Network-Level Resiliency

Network Design Resiliency Dual-homing

APS, GEC, MC-LAG

Event Dampening

Fast Convergence iSPF Optimization (OSPF, IS-IS)

BGP Optimization

Fast BGP Convergence

Graceful Restart (MBGP, OSPF, RSVP, LDP)

EMCP, Anycast, dual RR

VRRP/HSRP/GLBP/SLB/FWLB

MPLS High Availability

LDP Graceful Restart

MPLS/VPN NSF

BFD

MPLS FRR Path Protection

MoFRR

IP FRR

Pseudowire Redundancy

Spanning Tree (MST, PVRSTP...)

‒..................

Internet

DC

Access

1st Level Aggregation

2nd Level Aggregation

IP Edge

SP Core

18


Cost of High Availability

Designing a network for higher Service Availability

comes at a cost

Redundant Network Elements

Redundant Links

Redundant System Components (route processors,

forwarding processors, power supplies, etc.)

Operational costs

Lower steady-state Utilization levels

Increased configuration and management

Tighter maintenance windows

Availability

Cost ($)

0% 100%

19


Cost of High-Availability: Example

Large SP Network

Residential Services (3-Play)

10M Subscribers

1.25 Mbps / subscriber

Up to 96% increased CAPEX

for full redundancy!

Opex increased due to higher number of

network elements

Edge MPLS Aggregation Access Residential Residential

Core

Subscribers AN Agg1 Agg2 Agg3 BNG P

Locations 200,000 4000 500 74 74 74

System Type Generic ASR 9000 ASR 9000 ASR 9000 ASR 1013 CRS-3

Redundancy Scheme Total Cost $M

Chassis Costs $M

Interface Costs (SPA, SFPs), $M

Number of nodes

No redundancy $1,232 $529 $704 4658

Access NW uplink redundancy (Agg1, Agg2, Agg3)

$1,250 $529 $721 4658

AN uplink redundancy $1,563 $531 $1,032 4680

Access Network node redundancy (Agg1, Agg2, Agg3)

$2,423 $1,044 $1,379 9222

Edge link redundancy $2,425 $1,044 $1,381 9222

Edge Node redundancy $2,437 $1,056 $1,381 9296 Values for AN, Agg1, Agg2, Agg3 and Edge nodes only (No Pp-routers). Cumulative redundancy Schemes, GPL

20

Network High Availability


End-to-End Service HA Solution Set

0

0 – System-level HA (Baseline) RSP failover: 0 packet loss All L3 protocols are NSF capable NSR: OSPF, ISIS, BGP Routing timers and protocol configs are optimized by default

1

1 – Distributed BFD Rapid failure detection 15ms timer High scale

2 – BGP PIC Prefix Independent Convergence Fast Core/edge failure convergence

2

3

3 – TE/FRR Link/Node/Path Protection sub-50msec network convergence

4

4 – ECMP 32-way IGP/LDP, 8-way BGP Dynamic ECMP

5 – LACP Ethernet bundle With Full L3 support

6

7

7 – HSRP/VRRP Excellent Scale 100ms timer BFD Integration

5

6 – MPLS IP/TE FRR IP FRR w/ OSPF/ISIS sub-50msec network

convergence

SP Core Aggregation

For Your Reference

22


HA Network Map

AN <-> AGG <-> Edge <->

L4-7

App

Failure Detection Keepalives

Recovery Stateful App Redundancy

L3 Failure Detection BFD Keepalives BFD, Keepalives

Recovery MPLS TE FRR, IP Event Dampening, Fast Convergence, IP FRR, NSR / NSF

HSRP /

VRRP/GLBP/SLB/FWLB

Multicast HA

ECMP, iSPF, BGP PIC Core /

Edge, IP / MPLS FRR, LNS

Load sharing / Anycast / Dual

RR, Fast Hello

L2 Failure Detection

EOAM, (VCCV) Keepalives VCCV, EOAM,

MPLS Ping / TR

Keepalives EOAM, MPLS Ping / TR

Recovery GEC / APS / MC-LAG PW redundancy

Bridge Domains

GEC / APS / MC-

LAG

PPP / FR / ATM / HDLC /

GE SSO

GEC, APS, MC-LAG

L0/1 Failure Detection Interrupts Loss of Signal Interrupts Loss of Signal Interrupts

Recovery Module

Redundancy

Path diversity / dual

homing

Module Redundancy Path diversity / dual

homing

Module Redundancy

Aggregation Core Edge Access

Access Domain

23


HA Network Map

AN <-> AGG <-> Edge <->

L4-7

App

Failure Detection Keepalives

Recovery Stateful App Redundancy

L3 Failure Detection BFD Keepalives BFD, Keepalives

Recovery MPLS TE FRR, IP Event Dampening, Fast Convergence, IP FRR, NSR / NSF

HSRP /

VRRP/GLBP/SLB/FWLB

Multicast HA

ECMP, iSPF, BGP PIC Core /

Edge, IP / MPLS FRR, LNS

Load sharing / Anycast / Dual

RR, Fast Hello

L2 Failure Detection

EOAM, (VCCV) Keepalives VCCV, EOAM,

MPLS Ping / TR

Keepalives EOAM, MPLS Ping / TR

Recovery GEC / APS / MC-LAG PW redundancy

Bridge Domains

GEC / APS / MC-

LAG

PPP / FR / ATM / HDLC /

GE SSO

GEC, APS, MC-LAG

L0/1 Failure Detection Interrupts Loss of Signal Interrupts Loss of Signal Interrupts

Recovery Module

Redundancy

Path diversity / dual

homing

Module Redundancy Path diversity / dual

homing

Module Redundancy

Aggregation Core Edge Access

Access Domain

For Your Reference

24


Link Failure Detection Mechanisms

Loss of Signal (LOS) key to link failure detection

O(ms) with interrupt driven LoS detection on ASR 9000

Carrier Delay may be used to become resilient to link flaps

BFD

Ethernet OAM

MPLS OAM

Media type CC CP CC DP Loopback Performance Traceroute

Ethernet Last Mile IEEE 802.1ah - - -

Ethernet Provider Bridge

IEEE 802.1ag (MAC: Broadcast Domain)

MPLS LDP LDP Hello BFD, Y.1713, Y.1711

LSP Ping -

LSP TR MPLS TE RSVP Hello -

MPLS PW LDP Hello BFD, Y.1711 VCCV Ping - -

IPv4 IGP/BGP

Hello BFD IP Ping - IP TR

For Your Reference

25


E-LMI - Provides protocol and mechanisms used for:

Notification of EVC addition, deletion or status to CE

Communication of UNI and EVC attributes to CE

CE auto-configuration

Notification of Remote UNI name and status to CE

IEEE 802.3ah

OAM Discovery

Link Monitoring

Fault Signaling

Remote MIB Variable Retrieval

Remote Loopback

IEEE 801.3ag (CFM)

Family of protocols that provides capabilities to detect, verify, isolate and report end-to-end ethernet connectivity faults

Protocols (Continuity Check, Loopback and Linktrace) used for Fault Management activities

Ethernet OAM Overview BRKSPG-2202

For Your Reference

26


Ethernet OAM Overview

802.1ag

E-LMI

802.3ah 802.3ah 802.3ah 802.3ah 802.3ah 802.3ah

Core

E-LMI

Ethernet LMI: Automated configuration of CE based on EVCs and bandwidth profiles

L2 connectivity management

IEEE 802.3ah: When applicable, physical connectivity management between devices.

IEEE 802.1ag: Connectivity Fault Management (CFM)

Uses Domains to contain OAM flows and bound OAM responsibilities

Provides per EVC connectivity management and fault isolation

Three types of packets: Continuity Check, L2 Ping, L2 Traceroute

BRKSPG-2202

27


IEEE 802.1ag CFM Concepts

Nested Maintenance Domains (MDs)

break up the responsibilities for network administration of a given end-to-end service

Defined by operational boundaries

Nest & touch, but do not intersect

Maintenance Associations (MAs)

monitor service instances under a given MD

Defined by set of MEPs at the edge of a domain

Identified by {MA Name + MD ID}

Maintenance Association End Points (MEPs)

generate and respond to CFM PDUs

Define boundaries of MD

Initiate & respond to CFM PDUs

Per-Maintenance Association multicast “heart-beat” messages

Carries status of port on which MEP is configured

Uni-directional (no response required)

Transmitted at a configurable periodic interval by MEPs

Catalogued by MIPs at the same MD-Level and service, Terminated by remote MEPs in the same MA

Maintenance Domain

Maintenance Association

Maintenance Association End Points

28


ITU-T Y.1731 Overview

OAM Functions for Fault Management

Ethernet Continuity Check (ETH-CC) (Y.1731 adds unicast CCM)

Ethernet Loopback (ETH-LB) (Y.1731 adds multicast LBM)

Ethernet Linktrace (ETH-LT)

Ethernet Remote Defect Indication (ETH-RDI)

Ethernet Alarm Indication Signal (ETH-AIS)

Ethernet Locked Signal (ETH-LCK)

In addition: ETH-TEST, ETH-APS, ETH-MCC, ETH-EXP, ETH-VSP

OAM Functions for Performance Management

Frame Loss Measurement (ETH-LM)

Frame Delay Measurement (ETH-DM)

Covered by IEEE 802.1ag

29


Virtual Circuit Connection Verification (VCCV)

Overview checks connectivity between egress and ingress PEs

VCCV allows sending control packets in band of pseudowires (PW)

Signaling component: communicate VCCV capabilities as part of VC label

Switching component: cause the PW payload to be treated as a control packet

VCCV capability is negotiated when the AToM tunnel is brought up

depends on the LDP peer and the VC type

Both endpoints must have the same capabilities

marks the payload as control packet for switching purpose; packet follows the PW data path

Control packets sent over the AToM tunnels are intercepted by the egress PE

Type 1

(in-band vccv)

To signal in-band VCCV [RFC4385] using PW ID from PW Control Word

Type 2

(out-of-band VCCV)

Signal out-of-band VCCV inserting MPLS router alert label between tunnel and PW Labels

Type 3 (TTL expiry) Manipulate and Signal TTL exhaust (TTL == 1) for multiple switching point PEs

30


AC AC PE2 PE1 LDP Notification Message PW Status TLV

PW Status Code

MPLS Pseudowire Status Signaling Procedure

PW Status Signaling method selected if supported by both peers.

PEs exchange label mapping messages upon PW configuration.

Simple Label Withdraw status method will be used if one of the peers doesn’t support PW Status Signaling.

PW label won’t be withdrawn unless AC is administratively down or the PW configuration is deleted.

PW state set to “down” if the Label mapping is not available.

Capability is on by default.

31


Virtual Circuit Connection Verification (VCCV)

Multiple Packet Switched Network (PSN) Tunnel Types MPLS, IPSEC, L2TP, GRE,…

Motivation One tunnel can serve many pseudo-wires.

MPLS LSP ping is sufficient to monitor the PSN tunnel (PE-PE connectivity), but not

Virtual Circuits (VCs) inside of tunnel.

CE1 CE2 PE1 PE2

PSN Tunnel PW1

PW2

Emulated Service

Pseudo Wire

Native Service Native Service

For Your Reference

32


BFD Protocol Overview

Accelerates convergence by running fast keepalives in a consistent, standardized mechanism across routing protocols

Lightweight hello protocol

Neighbors exchange hello packets at negotiated regular intervals

Configurable transmit and receive time intervals

Unicast packets, even on shared media

No discovery mechanism

BFD sessions are established by the clients e.g. OSPF, IS-IS, EIGRP, BGP, …

Client hello packets transmitted independently

BFD Control Packets

OSPF

BGP

EIGRP

IS-IS

BFD

OSPF

BGP

EIGRP

IS-IS BFD

33


BFD Details

Session established between two peers

Timers are negotiated

Hello packets similar to IGP control packets

Does NOT react to failures itself -> notifies clients

Async mode (no echo): periodic control packets sent

Neighbour declared dead if no pkt is received for <interval *

multiplier> period

Session established using async control session

Echo mode: echo packets sent at negotiated rate,

used for failure detection

Control packets sent at low rate

Scalability: between 500 and 4000 sessions

Scalability depends on timer settings (50ms – 1sec)

green is alive

orange is alive

orange is alive

green is alive

Async Mode

Echo Mode

For Your Reference

34


Network Convergence — Why It Takes So Long

Detection of Link layer failure ms

ms

10’s of ms

10’s of ms

100’s of ms

10’s of ms

Report failure to Route Controller

Generate and flood an LSP

Trigger and Compute an SPF

Communicate new FIB entries to linecards

Install new FIB entries into linecard HW path

Bottleneck

35


Network Convergence — Why It Takes So Long

Detection of Link layer failure ms

ms

10’s of ms

10’s of ms

100’s of ms

10’s of ms

Report failure to Route Controller

Generate and flood an LSP

Trigger and Compute an SPF

Communicate new FIB entries to linecards

Install new FIB entries into linecard HW path

Optimize IGP

Convergence

BGP PIC

Optimize LDP & BGP

Convergence

36


Hierarchical CEF

Optimizes the data plane for sub-second convergence

CEF Data Structure Enhancements

Solves the FIB Download Convergence Bottleneck LSP and Prefix Independent

Optimizes FIB

Hierarchical CEF Technologies

MPLS-FRR

IP-FRR

BGP PIC Core

BGP PIC Edge

Non-Hierarchical CEF Technologies

MPLS Path Protection

Default – IP to MPLSDefault – IP to MPLS

One CEF Update Message per

prefix

BGP Prefix FIB Entry






Adjacency OCE - Interface



X

Failure

Repair

MPLS Label OCE

MPLS Label OCE

MPLS Label OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE

PIC Core – IP to MPLSPIC Core – IP to MPLS

One CEF Update Message for

Multiple Prefixes




X

Failure

Repair

Load Balance OCE

Load Balance OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE

37


IP/MPLS Aggregation

MPLS FRR 50 ms Convergence

Key Features

Fast Convergence for Link and Node Failures

Supported Across all Network Topologies

MPLS-TE Traffic Management

SRLG

BW Reservation

Per Tunnel Traffic Statistics

Caveats

Requires MPLS and MPLS-TE

No Protection for Ingress or Egress Tunnel Failures

Requires Pre-Computed Backup Paths

Requires “(n-1)!” Tunnels for Full Protection

Applicability Protecting Links in the aggregation network

Tunnel LSP

VC LSPs

Link Failure

FRR LSP

BRKRST-3363

38


MPLS-FRR — CEF

Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second

MPLS-FRR – IP and MPLS

One CEF Update Message

Adjacency OCE - Interface X

Failure

Repair

Midchain OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE Adjacency OCE - Interface

FRR OCE Label OCE

Label OCE

Midchain OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE Adjacency OCE - Interface

FRR OCE Label OCE

Label OCE

IP & MPLS CEF

IP & MPLS CEF

Pre-computed MPLS-FRR Backup Path

For Your Reference

39


IP/MPLS Aggregation

Tunnel LSP

VC LSPs

Protect Tunnel LSP

Link Failure


Key Features

Optimized for Ring Topologies

Utilizes Pre-Signaled Backup Tunnel

MPLS-TE Traffic Management

SRLG

BW Reservation

Per Tunnel Traffic Statistics

Caveats

Requires MPLS and MPLS-TE

No Protection for Ingress or Egress Tunnel Failures

Convergence Dependant on IGP Prefixes and L2VPN LSPs Under

Protection

Applicability

Protecting Ring Topologies

40


MPLS Path Protection — CEF


MPLS-FRR – IP and MPLS


Failure

Repair

Midchain OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE

MPLS-TE Path Protect Tunnel

Label OCE IP & MPLS CEF

Adjacency OCE - Interface Midchain OCE Label OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE

IP & MPLS CEF

Adjacency OCE - Interface Midchain OCE Label OCE

One CEF Update Message per IGP Prefix and L2VPN LSP!

For Your Reference

41


IP FRR-LFA: 50 ms Convergence

Key Features

50 msec Convergence for Link and Node Failures

Works for MPLS and IP Only Environments

Simple

Automatic configuration of “Loop Free Alternate Paths” via OSPF or ISIS

No Tunnels

Caveats

Requires a “Loop Free Path” for Protection

No Bandwidth Reservation

No Support for SRLG

New Feature

Applicability

Strong Solution for Deployments with Cost Effective Bandwidth

Loop Free Path

R1 R2

R3 R4

R5

Link Failure

No Convergence Required on

Routers R2, R3, R4 and R5 to

Maintain Green Traffic Flow!

BRKRST-3363

42


Tight SLA Protection with IPFRR

IPFRR Loop Free Alternate (LFA) - Principle

of Operation

If A finds an alternate path to B

Then this alternate path is valid for any destinations that A normally routes via B

IPFRR LFA – Properties and Benefits

Automated – No additional setup

No IETF protocol change - all the needed info is already in classical LSDB

Incremental deployment

No inter-operability testing

<50msec, prefix-independent

Applicable to MPLS LDP networks

IPFRR LFA – Deployment Dependency

IPFRR LFA coverage depends on the network topology

Two-plane network topologies are most “friendly” for IPFRR LFA deployments

Topology analysis required to assess IPFRR LFA efficiency

IPFRR LFA – Principle of Operation

Two-plane Network Topology

LSDB = Link State DataBase LDP = Label Discovery Protocol

For Your Reference

43


IP FRR-LFA — CEF Enhancement


IP FRR-LFA – IP to MPLS

One CEF Update Message for Multiple Prefixes


Failure

Repair

Load Balance OCE IP Prefix FIB Entry

IP Prefix FIB Entry

IP Prefix FIB Entry

MPLS Label OCE


IP-FRR OCE Load Balance OCE MPLS Label OCE

Load Balance OCE MPLS Label OCE


IP Prefix FIB Entry

IP Prefix FIB Entry

MPLS Label OCE


IP-FRR OCE Load Balance OCE MPLS Label OCE


Cleanup After Repair – Assuming No Available Loop Free Path


IP Prefix FIB Entry

IP Prefix FIB Entry

MPLS Label OCE






Pre-computed backup Path

For Your Reference

44


Non Stop Forwarding (NSF)

Routers to maintain forwarding state when communication between them is lost

Routing sessions are established with NSF aware peers. Upon HA event, neighboring peers maintain forwarding until routing sessions are reestablished.

Copy of FIB maintained on secondary and used on failure for continuously traffic flow.

Requires neighboring routers to be NSF aware.

Traffic is forwarded continuously

NSF Aware eBGP

CE1 PE1 P1

NSF Aware IGP and LDP

45


BGP PIC Edge

PE1

VPN1 VPN1

PE2

CE1 CE2

PE3

Link Failure

Optimizes BGP Convergence for BGP Next-Hop Change

PE to CE Link Failures

PE Node Failures

CE Node Failures

Applicability

PE Routers

Requires “bgp advertise-best-external” to enable

BRKRST-3363

46


BGP PIC Edge PE-CE Link Protection

PE2

PE1

CE1 RR1

RR2

PE3

CE2

Traffic flow due to BGP best path

47



PE2

PE1

CE1 RR1

RR2

PE3

CE2


The

BG

P

pre

-cal

cula

ted

B

acku

p P

ath

48



PE2

PE1

CE1 RR1

RR2

PE3

CE2


The

BG

P

pre

-cal

cula

ted

B

acku

p P

ath

The best BGP path to CE1 is now through

PE2

PE-CE link Failure

Detects that link is down and CEF layer

will switch to pre-computed backup path

49



PE2

PE1

CE1 RR1

RR2

PE3

CE2

The best BGP path to CE1 is now through

PE2

PE-CE link Failure

50



PE1 and PE2 precomputes bgp backup paths using bgp best-external approach

When primary link PE1 - CE1 fails:

PE1 holds on to the bgp local labels and re-routes CE1’s traffic to PE2 using labels advertised by PE2

PE1 uses fixed timer to clean up stale local labels

PE3 is expected to converge to start using PE2 as the BGP nexthop and IGP label for PE2 to send traffic from CE2 to CE1

CE2

PE1

PE2

CE1 PE3

MPLS-VPN

On Backup PE: router bgp 100 address-family ipv4 vrf red bgp advertise-best-external bgp additional-paths install

On Primary PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install bgp advertise-best-external

For Your Reference

51


BGP PIC Edge PE-Node Protection

PE2

PE1

CE1 RR1

RR2

PE3

CE2


Relies on BGP

Add-path

The best BGP path

to CE1 is through

PE1

52



PE2

PE1

CE1 RR1

RR2

PE3

CE2


The best BGP path

to CE1 is through

PE1

53



PE2

PE1

CE1 RR1

RR2

PE3

CE2

Detects that PE1 is

down and CEF layer

will switch to pre-

computed backup path

IGP signals router

is dead

PE-CE node

Failure

54



PE2

PE1

CE1 RR1

RR2

PE3

CE2

next BGP next-hop

scan the path through

PE2 will become the

best Path PE-CE node

Failure

55


BGP PIC Edge PE Node Protection

PE1, PE2 and PE3 precomputes bgp backup

When node PE1 fails:

IGP notification on PE3 invalidates active path

PE3 switches to backup path

PE3 is expected to converge to start using PE2 as the BGP nexthop and IGP label for PE2 to send traffic from CE2 to CE1

CE2

PE1

PE2

CE1 PE3 MPLS-VPN

On Primary PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install bgp advertise-best-external

On Backup PE: router bgp 100 address-family ipv4 vrf red bgp advertise-best-external bgp additional-paths install

On Ingress PE: router bgp 100 address-family ipv4 vrf red bgp additional-paths install

For Your Reference

56


BGP PIC Edge Notes

Supported for IPv4/v6 and VPNv4/v6

Not supported for L2VPN and mVPN address families

Failures detected using BFD or IGP

57

For Your Reference


BGP PIC Edge — CEF


PIC Edge – IP to MPLS

One CEF Update Message for Multiple Prefixes


Failure

Repair

Load Balance OCE




MPLS Label OCE X Adjacency OCE - Interface MPLS Label OCE

Pre-Computed Backup Path

Load Balance OCE



BGP Prefix FIB Entry Adjacency OCE - Interface MPLS Label OCE


Cleanup After Repair

Load Balance OCE




MPLS Label OCE

Adjacency OCE - Interface MPLS Label OCE Load Balance OCE

For Your Reference

58


VPN1 site1

VPN1 Site2

1

3

BGP PIC core – when IGP path to BGP Next-Hop changes 1. Examples: PE-P or P-P link failure, P node failure Sub-second convergence (prefix independent) vs. multiple seconds convergence (prefix and hardware dependent) Enabled by default since IOS XE 2.5.0 (cef table output-chain build favor convergence-speed)

BGP PIC edge – When BGP Next-hop changes 2. when remote PE node fails or no longer reachable. 3. when PE-CE link fails. Immediate to sub-second convergence (prefix independent) vs. multiple seconds convergence (prefix and hardware dependent)

BGP PIC Edge vs. BGP PIC Core

PE3

PE1

PE2 2

CE1

CE2

For Your Reference

59


IP/MPLS High-Availabilty Options: Scorecard

Network Infrastructure- all transit links and nodes

IGP Fast Convergence (IGP FC)

Broke the barrier of <200msec restoration time

Covers all faults, including multiple failures

IP/MPLS Fast ReRoute Loop Free Alternate (IPFRR LFA)

Provides local protection (link, node) with <50msec recovery

Tool to improve on IGP FC for most topologies (triangle, square, mesh)

MPLS TE Fast ReRoute (TE FRR)

Provides local protection (link, node, path) with <50msec recovery

Service Edge- edge node and access links

BGP Prefix Independent Convergence (BGP PIC)

IP/IPVPN scale independent recovery in line with IGP FC and FRR

Applicable to all BGP based services (IPv4, IPv6, VPNv4, VPNv6)

Fault Coverage

Operational Simplicity

Recovery Time

O(x00ms)

<50ms

<50ms

O(x00ms)

Feasible to deliver very tight E2E Service Availability SLAs without increasing operational complexity

For Your Reference

60


System High-Availability with Hardware

Redundancy Redundant hardware components

Power Supplies

Route Processors

Forwarding Processors

Switching Matrix

SPA Interface Cards

Interface Redundancy typically achieved using IEEE 802.3ad / LACP or APS

Hardware Redundancy needs to be complemented by Software redundancy Features

Cisco Platforms supporting hardware redundancy

CRS-3 ASR 9000 ASR 5000 ASR 1000 Cisco 12000 Cisco 7600 62


Distributed Forwarding Plane for Performance

Up to Eight Linecards

(Autonomous Forwarding)

Distributed IOS®XR based Control Plane for Scale

Dual Route Switch Processors (RSPs)

Dual-Core CPU on Each Linecard

Active/Active Switch Fabric for HA

Non-blocking Memory-less Fabric

Service Intelligence with Hi / Lo Priorities,

Unicast & Multicast Recognition, and VoQ’s

Redundant EOBC, Fan Trays, Power supplies

ASR 9000 System Architecture

MAC

FIC

NPU

NPU

NPU

NPU MAC

MAC

FIC

NPU

NPU

NPU

NPU MAC

FIC

CPU BITS/DTI

RSP(s) Line Card (40G)

FIC

CPU BITS/DTI

MAC

FIC

NPU

NPU

NPU

NPU MAC

CPU

MAC

FIC

NPU

NPU

NPU

NPU MAC

CPU

63


Control Plane Data Plane

Example: ASR 1000 System Redundancy

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

Route Processor (Standby)

Route Processor

(active)

Embedded Services Processor (Standby)

FECP

Interconn.

QFP subsys-tem Crypto

assist

Embedded Services Processor

(active)

FECP

Interconn.


assist

Passive Midplane

RP RP

GE, 1Gbps I2C SPA Control SPA Bus

Route Processor (standby)

RP

Interconn.

Embedded Services Processor

(active)

FECP

Interconn.


assist

Embedded Services Processor (standby)

FECP

Interconn.

QFP subsystem Crypto

assist

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

Passive Midplane

Route Processor

(active)

RP

Interconn.

SPA-SPI, 11.2Gbps Hypertransport, 10Gbps

ESI, (Enhanced Serdes) 11.5Gbps

64


ASR 1000 Software Architecture

ESP FECP

Interconn.

Crypto assist

RP CPU Chassis Mgr.

Forwarding Mgr.

Chassis Mgr.

Forwarding Mgr.

QFP Client / Driver

Interconn.

Interconn.

SIP

SPA SPA

IOCP

SPA Agg.

…

Interconn.

Kernel (incl. utilities)

Chassis Mgr. SPA driver

SPA driver

SPA driver

SPA driver

IOS





QFP subsys-tem

QFP code

Runs Control Plane Generates configurations Populates and maintains routing tables (RIB, FIB…)

Provides abstraction layer between hardware and IOS (manages ESP

redundancy) Maintains copy of FIB and interface list Communicates FIB status to active & standby ESP (or bulk-download

state info in case of restart)

Maintains copy of FIBs Programs QFP forwarding plane and QFP DRAM Statistics collection and communication to RP

Communicates with Forwarding manager on RP Provides interface to QFP Client / Driver

Implements forwarding plane Programs PPEs with forwarding information

For Your Reference

65


RPact

FMRP

QFP Client

RPsby

SPAs

ASR 1006 High Availability Infrastructure

HA operates in a similar manner to other protocols on the ASR 1000

Reliable IPC transport used for synchronization

RF RF

IPC Message Qs

IDB State Update Msg IDB State Update Msg

IOSact IOSsby

I P C

I P C

CF CF Interconnect Used for IPC and Check-pointing

Non-HA-Aware Application


Driver/Media Layer

Mcast

CEF

Config

…

Driver/Media Layer

Mcast

CEF

Config

… MLD MLD

IPC Message Qs

ESPsby ESPact

QFP Client

FMRP

FIB MFIB FIB MFIB

FMESP FMESP

IDB RIB RT MRIB IDB RIB RT MRIB

FIB MFIB FIB MFIB

66


Which Events Trigger Failovers?

The following events may trigger failovers on the RP/ESP

1. Hardware component failures

2. Software component failures

3. Online Insertion and Removal (OIR)

4. CLI-initiated failover (e.g. reload command, force-switchover command)

67


1. Failover Triggers: Hardware Failures

What hardware failures?

a. CPUs: RP-CPU, QFP, FECP, IOCP, interconnect CPU, I2C Mux, ESP Crypto Chip,

b. Memory: NVRAM, TCAM, Bootflash, RP SDRAM, FECP SDRAM, resource DRAM, Packet

buffer DRAM, particle length DRAM, IOCP SDRAM, …

c. Interconnects: ESI Links, I2C links, EOBC Links, SPA-SPI bus, local RP bus, local FP bus …

d. Other: heat-sinks, …

Detected using

CPLD interrupts / register bits within O(ms) controlled by CMRP

Watchdog timers: low level watchdogs running in O(min) that can initiate a reset (e.g. RP)

JTAG: RP can program CPLD on other modules. Test interconnects and other boards (primarily

for RMAd hardware)

Interrupts generated by hardware failures initiate fail-over events

Hardware failures are typically fatal such that modules need to be

replaced!

SIP

SPA SPA

IOCP

SPA

Agg.

…

Interconn.

ESP FECP

Interconn. QFP subsys-tem

Crypto assist

RP CPU

Interconn.

Interconn.

68


2. Failover Triggers: Software Failures

SIP

SPA SPA

IOCP

SPA

Agg.

…

Interconn.

ESP FECP

Interconn. QFP subsys-tem

Crypto assist

RP CPU

IOS

Chassis Mgr.

Forwarding Mgr.

Chassis Mgr.

Forwarding Mgr. QFP Client / Driver

Interconn.

Chassis Mgr. SPA driver

SPA driver

SPA driver

SPA driver

Interconn.

QFP code

IOS




What Software Failures?

a. Kernel: Linux on RP / ESP / SIP

b. Middleware: Chassis Manager (CM), Forwarding Manager (FM)

c. IOS

d. SPA drivers

Detected using

Kernel: the kernel supervises middleware or SPA driver processes (kmonitor()). It always knows if a process is healthy

IPC: between 2 IOS (and only for IOS)

Kernel will take the module down in a controlled manner

IOS, CMESP, CMSIP, FMESP, QFP Driver/Client, IMSIP are not re-startable!

Also setting register bits to initiate fail-over for ESP or RP

Note: some other processes are re-startable (CMRP, FMRP, SSH, Telnet…)

Kernel will try to re-start the processes in this case

69


SIP

CMSIP Kernel

RP (slot 1)

CMRP Kernel IOS

RPact Failover Procedure

ESP (slot 0) CMESP Kernel FMESP FMRP

RP (slot 0)

CMRP Kernel IOS FMRP

ESP (slot 1) CMESP Kernel FMESP

SIP

CMSIP Kernel

SIP

CMSIP Kernel

ACT ACT SBY SBY

Failure

Detect RPact

failure

ACT Restart New RPact information

Close ESI links (ESP)

Establish ESI links

State information

If not received in time, send restart message.

Update H/W component file system

ESI link status

For Your Reference

70


Failover

Take-over control using checkpointed state Forwarding State information

Check updated state and discard old state

Check updated state and discard old state

Service recovered

Start kernel start

Start CM Start IOS

H/W initialization

Initialize EOBC

SIP

CMSIP Kernel

RP (slot 1)

CMRP Kernel IOS

RPact Failover Procedure (cont.)


RP (slot 0)



Start FM

SIP

CMSIP Kernel

SIP

CMSIP Kernel

Detect RPsby

SBY Forwarding State information ESI link status

Other RP information

Run Mastership

For Your Reference

71


SIP

CMSIP Kernel

RP (slot 1)

CMRP Kernel IOS

ESPact Failover Procedure


RP (slot 0)



SIP

CMSIP Kernel

SIP

CMSIP Kernel

ACT ACT SBY SBY

Failure Interrupt

ESI link status

Reconfigure ESI link w/ RPs

Disable ESI link w/ failed ESP

Change state of ESI link w/ new ESPact

ESI link status

Detect ESPact failure

State information of failed ESP

ACT Failed

Service Recovered with momentary packet loss

Resend state information

ACT ACT SBY

For Your Reference

72


SIP

CMSIP Kernel

RP (slot 1)

CMRP Kernel IOS

ESPact Failover Procedure (cont.)


RP (slot 0)



SIP

CMSIP Kernel

SIP

CMSIP Kernel

H/W initialization

Initialize EOBC

Restart

Wait for RPact

RPact information

Detect RPact

Activate ESI-link Download software packages

Start kernel

Start CM Register with CMRP

Other-ESP information (e.g. mastership)

SBY

Start FM

For Your Reference

73


IOS High Availability

During the initialization process IOS is loaded on both the RPact and Rpsby

IOS learns about any events Uses Redundancy and checkpointing facility

Redundancy facility (“when to synchronize”) A process to help in synchronization and coordination of switchovers (e.g. switchover events, switchover control and monitoring)

Clients of the RF maintain the databases that are synchronized using the RF

Examples: reliable internal data transfer service, Event notification mechanisms, logging, etc.

Checkpointing facility: (“how to synchronize”) Defines a set of APIs and transport for different SSO-aware features to copy state

Helps to synchronize state data in a consistent, repeatable and well-ordered manner

Keeps checkpointing state

Interfaces to RF

Line Card

RF RF

IPC Message Queues

IPC Message Queues

IDB State Update Msg IDB State Update Msg

Active RP Standby RP

I P C

I P C

CF CF

Interconnect Used for IPC and Check-pointing



Driver/Media Layer

PPP

CEF

L2TP

Config

…

Driver/Media Layer

PPP

CEF

L2TP

Config

…

IDB FIB IDB FIB

For Your Reference

74


Cisco Software High-Availability Support

Stateful Switchover (SSO) support for features provides the synchronization of dynamic feature state between hardware modules

Configuration synchronization ensures that the running config is synchronized on the route processors

Dynamic State Preservation ASR 1000 ASR 9000

Connectivity Protocols FR, PPP, MLPPP, HDLC, 802.1Q, BFD (BGP, IS-IS, OSPF) BFD (OSPF, BGP, IS-IS, Static)

Routing & IP Services

RP, HSRP, IPv6 NDP, uRPF, SNMP, GLBP, VRRP, NSR (MP-iBGP,

eBGP), ISSU, GRE,

NSF (ISIS, OSPF, BGP), NSR (ISIS, OSPFv2,

OSPFv3, BGP)

Multicast IPv4 Multicast (IGMP), IPv6 Multicast (MLD, PIM-SSM, MLD Access

group), MoFRR

NSF Multicast, BFD for PIM, MoFRR

MPLS Protocols MPLS L3VPN, MPLS LDP , VRF-aware BFD,

Roadmap: NSR LDP, T-LDP

NSF (LDP, T-LDP, RSVP-TE)

NSR (LDP), BFD for MPLS FRR, VRF-aware BFD

Broadband PPPoE, L2TP (LAC, LNS), DHCPv4/v6, AAA, session state (virtual

templates), ISG, ANCP, LI

PPPoE (including nV)

Security SSO, Stateful Inter-chassis redundancy for FW / NAT Roadmap

SBC SSO Roadmap

For Your Reference

75


Non Stop Routing (NSR)

Routers to maintain routing state and forwarding state when communication between them

is lost

Routing sessions are maintained between processors on a failure, allowing routing sessions

to stay up with Peer

Copy of FIB maintained on secondary and used on failure for continuously traffic flow

No need for neighboring routers to be NSF aware or capable. Can give high

reliability without upgrading CE.

Traffic is forwarded continuously

eBGP

CE1 PE1 P1

IGP and LDP

76


BGP NSR

Implemented by hardening code for

BGP RIB checkpointing

BGP TCP interaction

Only supported for IPv4 unicast, VPNv4 unicast Address families in Cisco IOS

Configuration router bgp <asn>

address-family ipv4 vrf RED

neighbor x.x.x.x ha-mode sso

No peer session flaps on VPNv4 CE when RP switches over

Route updates during RP switchover are announced to VPNv4 CE peers

NO delays which prevents data black-holes during RP switchover as in the case of graceful restart peers

77


OSPFv2 NSR

Provides the ability to perform hitless RP switchovers when OSPF is used as the routing protocol (Expect zero-traffic loss across such HA events)

To enable OSPFv2 Non-Stop Routing (NSR) router ospf <process id> [vrf <vrf name>]

nsr

Activated on a per-process basis (for both ipv4 or ipv4 VRF for PE-CE sessions)

Depends on the forwarding plane’s ability to retain state across control plane restarts and RP switchovers

Alleviates dependency on OSPFv2 protocol extensions (NSF) Neighboring routers are unaware that a router is NSR-capable

Neighboring routers are unaware that a router has gone through an RP switchover

Provides near-transparent RP switchover capability OSPF adjacencies remain up

Minimal state refreshed by the restarting router post switchover

Scalable to larger link state database sizes and number of neighbors

78


High Availability for Advanced Service Models

Many SP Services already go beyond standard L3VPN / L2VPN / transport services

Increasing subscriber management capabilities and L4-L7 services

Examples:

Subscriber Management

Multicast

Session Border Controller

Firewall

IPSec

LI

Some Services can be made highly-available using Intra-chassis redundancy (e.g. IPSec, Firewall, NAT, PPPoX, L2TP)

Stateless inter-chassis redundancy available for BNG

Stateful Inter-chassis redundancy available for NAT, Firewall and SBC on the Cisco ASR 1000

80


L3VPN Key HA Technologies

Physical Circuit Diversity

Multihoming

Link Detection IP Event Dampening

BFD

Chassis Redundancy NSF/NSR

Routing Protocols and Convergence BGP PIC Core

BGP PIC Edge

IP-FRR


Edge Access

Mobile

CPE

Business

VRF Green

Core

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Access

Mobile

CPE

Business VRF Green

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Edge

Site A

Site B

Site C

Site D

For Your Reference

81


L3VPN Key HA Technologies

CPE

BFD for PE-CE Link

Detection

NSF/NSR for Chassis HA

PE Multihoming Intra-Site PE for PE Diversity

Inter-Site for SP Facility Diversity

Access

Circuit Diversity - Physical

Diversity for Multihomed CPE

Physical Circuit Diversity is Not the

Default

Must be Requested from the SP

Edge

BFD for PE-CPE / PE-P Link

Detection

NSF/NSR for Chassis HA

IP Event Dampening for PE-CPE

IP-FRR for PE-P For Cost Effective PE-P Bandwidth

BGP PIC Core

BGP PIC Edge for Multi-Homed CPE

Edge Access

Mobile

CPE

Business

VRF Green

Core

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Access

Mobile

CPE

Business VRF Green

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Edge

Site A

Site B

Site C

Site D

82


L2VPN — Pseudowire Redundancy

Active-Standby PW Access Circuit Redundancy

L2TPv3 and MPLS Support

Detection Mechanisms

IGP Convergence for Remote PE Failure

LDP Signaling for PE-CE Failure

LDP Timeout for Remote PE Software Failure

CE

P

P

P

CE

PE1 PE2

PE23

P

Standby PW

Active PW

BRKSPG-2207

83


Multicast High-Availability Behavior

Before failure

Multicast state is synchronized from RPact to RPsby

Configuration

MLDv1/v2 state information

PIM or MRIB state are NOT synchronized

MFIB also synched to ESPact and ESPsby

After failure

RPsby sends out PIM hellos to all neighbors

PIM neighbors re-send PIM state

Newly active RP re-builds the PIM state

IGP reconverges to assure uRPF check

MFIB and ESP updates proceed to incorporate refreshed PIM state

ESPact continues to forward multicast traffic based on its version of the MFIB

Forwarding of multicast packets is NOT disrupted

RPact

FMRP

QFP Client

RPsby

SPAs

RF IDB State Update Msg

IOSact IOSsby

I P C

I P C

CF

Driver/Media Layer

Mcast

CEF

Config

…

MLD

ESPsby ESPact

QFP Client

FMRP

FMESP FMESP

MFIB

RF

CF

IDB State Update Msg

Driver/Media Layer

Mcast

CEF

Config

…

MLD

MFIB MRIB MFIB

MFIB

MRIB

84


Receiver

Multicast join on primary path

Multicast join on backup path

Data packets are received from the primary and secondary paths

The redundant packets are discarded at topology merge points due to RPF checks

Failure:

Interface chance on where packets are accepted

Backup path interfaces become ‘active’

Multicast only Fast Re-Route (MoFRR)

POP1

POP2 POPN

IPTV source

Normal path

Alternate PIM

Configuration and Restrictions Dependency on ECMP and will not work without it Disabled by default and enabled through a cli Applicable to IPv4 multicast only and not IPv6 multicast Works only for SM S,G and SSM routes Works where the rpf lookups are done in a single vrf Extranet routes are not supported Both primary and secondary paths should exist in the same multicast topology.

85


Stateful Application Switchover: PPP

Copies state information for PPP, PPPoE, and PPPoEoVLAN Sessions

Switch-over is transparent to peers Sessions are not torn-down / re-established

PPP, PPPoE, and PPPoEoVLAN Session States: Configuration (through config synch), including QoS configuration, ACLs

Session identifiers

PADR frame (cached)

RADIUS session attributes

Physical interface

VAI identifier

MD5 signature

Statistics are synchronized on ASR 1000!

86


Stateful Application Switchover: L2TP

RPact synchronizes state with RPsby

State includes configuration, PPP session IDs, L2TP CC sequence numbers

etc.

Sequence numbers (Ns, Nr) for L2TP Control Connections (CC) are only

synched once for a packet window of X (i.e. once every X L2TP control packets)

ESP

L2TP Control Connections

RPsby

L2TP Tunnel

LNS

RPact

87


Residential

STB

BNG Cluster

BNG Service Edge High Availability

PPP Smart Server Selection allows user to configure specific PADO delay for a received PADI packet

Can be configured per bba-group or based on circuit-id/remote-id

In case of an outage of a BNG in the cluster, other BNG stand ready to accept subscriber sessions

Detection of failure possible at both ends of PPPoE session because of missing keepalives

Subscriber sessions have to be re-established

Allows BNG redundancy with predictable behavior

E-DSLAM

Ethernet Aggregation

PADI 1

PADI

PADO

PADI PADI

PADO

PADO

2 PADO

3 PADR

4 PADS

PADR

PADS

Delay

88

Stateful Inter-Chassis

Redundancy


Motivation for Stateful Application Inter-Chassis

Redundancy

Current Intra-chassis HA typically protects against

Control Plane (RP) Failures

Forwarding Plane (ESP) failures

Interface failures can be mitigated using link bundling (e.g. GEC)

Any other failures may result in recovery times O(hours)

Inter-chassis redundancy provides additional resilience

against

Interface Failures

System failures

Site failures (allowing for geographic redundancy)

RP

RP FP

FP

SIP

SIP

SIP

SIP

90


Stateful System Redundancy Models

Different deployment models

1+1 – one system is actively processing and passing traffic, the other in standby mode.

1:1 – two systems are actively processing and passing traffic, and backing each other up

N+1 – N systems are actively processing and passing traffic, and share a single standby

System vs. Application

Is the inter-chassis resilience applicable to ALL of the features / functions configured on the system, or only for a particular application?

System-level: provide resilience for ALL applications and traffic configured on a system

Application-Level: provide resilience for a particular application and its traffic

Hot-standby vs. Cold-standby

Cold-standby: FIB / adjacency updates are NOT synchronized between active and standby system

Hot-standby: forwarding/state information is synchronized between active and standby system

Different Approaches can also be categorized into

Control plane active-standby / active-active

Forwarding plane active-standby / active-active

For Your Reference

91


System Level

Redundancy

Application Level

Redundancy Example: VSS

Failover Granularity at the System Level

Control-plane active-standby

Active RP considers ‘remote’ linecards under its

control

Forwarding-plane active-active

No application granularity for failover

Need to ensure all features are SSO capable

Example: RG Infra

Failover Granularity at the Application Level (NAT,

Firewall, SBC etc)

Control plane active-active

Each RP only considers its own linecards, but synchronizes

application state

Forwarding-plane active-active

E.g. can have one set of firewall services resilient,

and other set of firewall services non-resilient

RPact

Fabric

LC

Fabric

LC LC LC

RPsby RPact RPsby

Failover

RPact

ESP

SIP

ESP

SIP SIP SIP

RPact

Failover

FW FW

92


RP

Stateful INTRA-Chassis Redundancy Revisited

Building-blocks required to achieve stateful interchassis redundancy are for ASR 1006 / 1013:

1. Redundant Hardware components

RP / ESP / ESI links

SIPs/SPAs are NOT redundant

2. Forwarding / Application State Tables

3. Control mechanism to synchronize between active-standby components

Who is active / who is standby?

Initiate failover in case of failure

4. State transfer mechanism

Copy forwarding / application state tables to standby and keep synchronized

Currently provided by IOS RF/CF infrastructure over IPC

5. Failure detection mechanism

Interrupts

RP

ESP ESP

SIP SIP SIP

IPC

Active Enhanced Serdes Link ESI (internal dataplane) Standby Enhanced Serdes Link ESI (internal dataplane)

Note: EOBC (internal control plane) infrastructure NOT shown.

RT RIB NAT RT RIB NAT

93


nV Edge Overview Control Plane EOBC Extension (L1 or L2 connection) One or two 10G/1G from each RSP

Control plane EOBC extension is through special 1G or 10G EOBC ports on the RSP. External EOBC could be over dedicated L1 link, or over port-mode L2 connection

Data plane extension is through regular LC ports (it can even mix regular data ports and inter-chassis data plane ports on the same LC)

Doesn’t require dedicated fabric chassis flexible co-located or different location deployment, lower cost

Special external EOBC 1G/10G ports on RSP

Active

RSP

Secondary

RSP

LC LC LC LC

0 Standby

RSP

Secondary

RSP

LC LC LC LC

1

Inter-chassis data link (L1 connection) 10G or 100 G bundle (up to 32 ports)

Regular 10G or 100G data ports

Internal EOBC

For redundancy purpose, minimal two control plane and two data plane links are required

External EOBC link fail won’t cause RP failover as long as it has alternative EOBC link

94


Inter-Chassis Control Plane and Data Plane

Packet Format Inter-chassis control plane link

Ethernet snap with special ethertype and internal mac addresses

Work over L2 circuit as well assuming it’s port mode: transparently forward every packet

Recommend L1 link, with up to 10msec latency

Inter-chassis data plane link

Regular Ethernet frame, with 802.1q tag (VLAN=1)

In theory, it can work over L2 circuit, but it’s never tested and won’t be supported officially

95


Multi-chassis LAG

mLACP uses ICCP to synchronize LACP configuration & operational state between PoAs, to provide DHD the perception of being connected to a single switch

All PoAs use the same System MAC Address & System Priority when communicating with DHD

Configurable or automatically synchronized via ICCP

Every PoA in the RG is configured with a unique Node ID (value 0 to 7). Node ID + 8 forms the most significant nibble of the Port Number

For a given bundle, all links on the same PoA must have the same Port Priority

ICCP

DHD

PoA1

LACP PoA2

Node ID: 1

Node ID: 2

Port #: 0x9001, Port Priority 1

Port #:0xA001, Port Priority 2

System MAC: aaaa.bbbb.cccc

System Priority: 1

96


Inter-chassis Communication Protocol

ICCP allows two or more devices to form a ‘Redundancy Group’

ICCP provides a control channel for synchronizing state between devices

ICCP uses TCP/IP as the underlying transport ICCP rides on targeted LDP session, but MPLS need not be enabled

Various redundancy applications can use ICCP: mLACP

Pseudowire redundancy

RG

ICCP over Dedicated Link

RG

ICCP over Shared Network

97


Control Plane HA Model

Only one Active RSP, Only one standby RSP at a given time, which are located on two different chassis

SSO/NSF/NSR works exactly the same way as two RSPs on the same chassis

Reliable out of band control channel between two chassis

IOS-XR control plan can tolerant hundreds of msec latency*, although the latency can impact overall service convergence time

Virtual Chassis is always on as long as there is one chassis and one RSP alive

Active

RSP

Secondary

RSP

LC LC LC LC

0 Standby

RSP

Secondary

RSP

LC LC LC LC

1

DSC Chassis Non DSC Chassis

Active control plane

Standby control plane

Standby

RSP

Active

RSP

Standby

RSP

* Practically, recommend maximum 10msec latency between two chassis

For Your Reference

98


Data Plane Forwarding Model

Inter-chassis data links simulate the switch fabric , which provide the data connection between two chassis. It has similar features as switch fabric, for example, fabric qos. Packet load balancing over inter-chassis links is same as regular link bundle: per-flow based

Keep the existing IOS-XR two-stage forwarding model no forwarding architecture change for single chassis vs. nV Edge system

In case of ECMP or link bundle paths cross two chassis, it prefer local port instead of load balancing packet to the other chassis. This is to reduce the inter-chassis link usage as much as possible. However, this feature (local rack preference) could be turn off by user CLI

Only single Multicast copy is sent over inter-chassis link. Multicast replication is done on egress line cards and fabric on the local chassis

Active

RSP

Secondary

RSP

LC LC LC LC

0 Standby

RSP

Secondary

RSP

LC LC LC LC

1

Simulated switch fabric

For Your Reference

99


Data Forwarding

Ingress LC

P2

P1

Data Plane

Lookup

1

Lo

ad

B

ala

nce

Inter-Chassis LC

P2

P1

Data Plane

Encapsula

tion

Egress LC

Data Plane

LO

OK

UP

Inter-Chassis LC

P2

P1

Data Plane E

ncapsula

tion

Inter-Chassis LC

Data Plane

De

ca

psu

lat

ion

P2

P1

Inter-Chassis LC

Data Plane

De

ca

psu

lat

ion

P2

P1

P2

P1

2

3

3

4

4

5

Inte

r-C

has

sis

Lin

k b

un

dle

Chassis 0 Chassis 1

Ingress Forwarding Lookup L2/L3/Mcast regular lookup

Inter-Chassis Load Balance Load balance across multiple inter-chassis links

Inter-Chassis Encapsulation Egress Forwarding Lookup L2/L3/Mcast regular lookup

1

2

3

4

5

Inter-Chassis Decapsulation

For Your Reference

100


Introduction to RG-Infra

RG Infra is the IOS Redundancy Group Infrastructure to enable the synchronization of application state data between different physical systems

Does the job of RF/CF between chassis

Infrastructure provides the functions to

Pair two instances of RG configured on different chassis for application redundancy purposes

Determine active/standby state of each RG instance

Exchange application state data (e.g. for NAT/Firewall)

Detect failures in the local system

Initiate & manage failover (based on RG priorities, allows for pre-emption)

Assumptions

Application state has to be supported by RG infra (ASR 1000 currently supports NAT, Firewall, SBC)

Connectivity redundancy solved at the architectural level (need to ‘externalize’ the redundant ESI links of

the intra-chassis redundancy solution)

101


Redundancy Groups Functions

Registers applications as clients

Registers (sub)interfaces / {SA/DA}-tuplets in case of firewall

Determines if traffic needs to be processed or not E.g. for Firewall: if a subset of sessions are associated with a RG in active state, then the Firewall application will perform normal

processing for those sessions and actively sync the session state to another device that has the same RG in STANDBY state.

For Firewall sessions that are associated with a RG in STANDBY state, the session information will be synchronized from a device

that has the RG in ACTIVE state.

Communicates control information between RGs using a redundancy group protocol Advertisement of RGs and RG state

Determination of peer IP address

Determination of presence of active RG

Synchronizes application state data using a transport protocol

Manages Failovers! RPact

ESP

SIP

FW RG

SPA SPA

SIP SPA SPA

RPact

ESP

SIP

RG

SPA SPA

SIP SPA SPA

FW

RG state data

RG control data

Active

102


Redundancy Groups Functions — Details

Configuration of stateful system redundancy Priority (similar to HSRP priority for RG state determination)

Preemption, Name

RG State control Init, Active, Standby, disabled

Communicating state changes to other software entities in the system (e.g. QFP software)

Synchronization management Synchronization state tracking (standby has to request bulk-updates from active)

Determines when synchronization is started (e.g. ensures transport is available)

Peer Management Maintain information about peers

Fault Handling Changing priorities of RG (may affect RG state)

Fault event dampening

Logging

Integration with Enhanced Object tracking / BFD

Transport Connectivity Knows via which interface application state is synchronized

Can be different for application state data and RG control messages

For Your Reference

103


Intra-Chassis vs. Inter-Chassis Redundancy

Function / Method Stateful Intra-chassis Stateful Inter-chassis

Hardware redundancy ESP, RP ESP, RP, Interfaces

Redundant connectivity Internal ESI links Redundant links to neighbor nodes

Redundancy control RF/CF RG

State synchronization IPC over EOBC External GEC

Failure detection mechanism Interrupts BFD, Hellos

Failover mechanism Chassis Manager RG Protocol (HSRP like)

For Your Reference

104


Possible RG-Infra Redundancy Models

Active-Standby

All application traffic associated with a SINGLE RG instance

Failures would switch all traffic over to the standby chassis

Active-Active

Multiple RG instances configured per system

Subset of traffic associated with a particular RG instance

Single failure only affects subset of overall application traffic

2+1 Active-Standby

2 or more chassis loadshare application traffic, backed up by a single standby system

Subset of traffic associated with a particular RG instance on different chassis

Single failure only affects subset of overall application traffic

RGact RGsby

RG1act RG1

sby

RG2act RG2

sby

RG3act

RG3sby

RG1act

RG1sby

RG2act RG2

sby

105

Case Studies


Case Study: Highly Available IP Architecture for

Mobile

Cellsite

Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MTSO

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Service Termination

GE 10 GE QFP

Internet Core

L2 Domain L3 Domain L3 Domain

Local VLANs or T1s

Transport VLANs / Static Routes / BGP PIC Edge / BFD protection OSPF/RIP/VRRP

FE T1

EoMPLS Backhaul

One Second Convergence Requirement

CSR EvDO/LTE VRF

1xRTT VRF

QFP

107


Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet Core

Static Routes establish connectivity between loopback addresses

VRF VRF

VRF

VRF

Static routes for cellsite reachability BGP PIC Edge for Layer-3 convergence VRRP for MTSO

VRF Tables Updated

BFD sessions established

CSR EvDO/LTE VRF

1xRTT VRF

QFP


Mobile — Transport

108


Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

Steady state: CSR distributes flows across both Agg’s using ECMP. Traffic could flow across Agg inter switch links. Each Agg handles traffic related to all services from the cell-site.

1xRTT

1xRTT

EVDO

EVDO

1xRTT


Mobile — Steady-State Traffic Flows

109



Mobile — Link Failure Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

Steady state: Traffic flows distributed across both Agg. Failure: GE link from MSPP to Agg1 fails. Action:

BFD session to Agg1 times out at CSR. Agg1 next hop removed from forwarding table. Traffic flows resume across existing path to Agg2.

Results: Traffic flows to Agg1 via Agg2.

No changes to LAN side connectivity

x

110



Mobile — Aggregation Switch Failure Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

x

Steady state: Traffic flows distributed across Agg. Failure: Agg1 power outage. Action:

BFD and VRRP sessions time out BGP and OSPF neighbors drop due to BFD BGP PIC Edge ensures sub-second convergence Traffic flows resume across existing path thru Agg2.

Results: Traffic flows via Agg2 to end hosts. 111



Mobile — CSR Failure Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

x

Steady state: Traffic flows distributed across CSR. Failure: CSR power outage. Action:

BFD sessions time out BGP neighbors drop due to BFD Mobile handsets resync to neighboring cell site

Results: Mobile handset voice connectivity is maintained.

112

Summary


Summary

Motivation for High Availability in SP Aggregation Networks

Network Level High Availability



Stateful Inter-chassis Redundancy

Case Studies

Summary and Conclusions

114


Key Takeaways

High-Availability becoming increasingly deployed in Aggregation Networks

Motivated by experiences with MPLS Core Networks

Many high-availability techniques deployed in the core are now applied in MPLS aggregation networks

MPLS TE FRR, BFD, EOAM, Pseudowire Redundancy …

Service High Availability requires comprehensive approach including the deployment of

Network level resiliency

System Level resiliency

L4-7 service resiliency

Stateful Inter-chassis redundancy increasingly being considered to provide geographic redundancy for applications

High Availability comes at a cost (CAPEX & OPEX)!

115


Call to Action

• Visit the Cisco Campus at the World of Solutions to experience Cisco innovations in action

• Get hands-on experience attending one of the Walk-in Labs

• Schedule face to face meeting with one of Cisco’s engineers

at the Meet the Engineer center

• Discuss your project’s challenges at the Technical Solutions Clinics

116


Recommended Reading

N. Stringfield et. Al, “Cisco Express Forwarding”, ISBN-13: 978-1-58705-236-1

D. C. Lee, “Enhanced IP Services for Cisco Networks”, ISBN-13: 978-1-57870-106-3

A. Sayeed, M. Morrow, “MPLS and Next-Generation Networks”, ISBN-13: 978-1-58720-120-2

J. Davidson et. al, “Voice over IP Fundamentals”, 2nd Edition, ISBN-13: 978-1-58705-257-6

V. Bollapragada et. Al, “Inside Cisco IOS Software Architecture “, ISBN-13: 978-1-57870-181-0.

R. Wood, “Next-generation Network Services”, ISBN-13: 978-1-58705-159-3.

K. Lee, F. Lim, B. Ong, “Building Resilient IP Networks”, ISBN-13: 978-1-58705-215-6

T. Szigeti, C. Hattingh, “End-to-End QoS Network Design: Quality of Service in LANs, WANs, and VPNs:, ISBN-13: 978-1-58705-176-0

B. J. Carroll, “Cisco Access Control Security”, ISBN-13: 978-1-58705-124-1.

A. Khan, “Building Service-Aware Networks: The Next-Generation WAN/MAN”, ISBN-13: 978-1-58705-788-5

118


Whitepapers on CCO

Cisco IOS High Availability

http://www.cisco.com/en/US/tech/tk869/tk769/tech_white_papers_list.html

http://www.cisco.com/en/US/products/ps6550/prod_white_papers_list.html

Campus Network for High Availability Design Guide

http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html

Cisco Validated Designs

http://www.cisco.com/en/US/netsol/ns742/networking_solutions_program_category_home.html

ASR 9000

Cisco ASR 9000 Series High Availability: Continuous Network Operations

Introduction to Cisco ASR 9000 Series Network Virtualization Technology

Distributed Virtual Data Center for Enterprise and Service Provider Cloud

ASR 1000

Cisco ASR 1000 Series Aggregation Services Routers

Cisco ASR 1000 Series: ISSU Deployment Guide and Case Study

Cisco Unified Border Element (SP Edition) on Cisco ASR 1000 Series

Cisco Unified WAN Services: Services, Security, Resiliency, and Intelligence

119









http://www.cisco.com/en/US/prod/collateral/routers/ps9853/white_paper_c11-501797.html

http://www.cisco.com/en/US/solutions/collateral/ns341/ns524/ns562/ns592/asr_nv_100611.pdf

http://www.cisco.com/en/US/solutions/collateral/ns341/ns524/ns562/ns592/asr_nv_100611.pdf



http://www.cisco.com/en/US/prod/collateral/routers/ps9343/white_paper_c11_517184.html


http://www.cisco.com/en/US/prod/collateral/routers/ps9343/solution_overview_c22-450358_ps9343_Products_White_Paper.html

sp edge aggr ha bcp.2013 eur pdf brkspg-2402

Documents

high availability techniques

cisco andor

overall service availability

ip frr

ip fast

ip edge architecture

cisco public best practices

service provider edge