best practices to deploy high-availability...

Best Practices to Deploy High-Availability in Service Provider Edge and Aggregation Architectures BRKSPG-2402

Matthias Falkner, Distinguished Engineer, Technical Marketing

© 2014 Cisco and/or its affiliates. All rights reserved. BRKSPG-2402 Cisco Public

Agenda

Motivation for High Availability in SP Aggregation Networks

Network Level High Availability

System High Availability

Service High Availability

Stateful Inter-chassis Redundancy

Case Studies

Summary and Conclusions

7

Motivation for High-Availability in the IP Edge and Aggregation


High Availability and Service Level Agreements

Many SPs specify their SLAs in the T&Cs

Important characteristic of both business and residential services

Historically given for Core network, but expanding to end-end SLAs

Metrics

Service Availability (averaged over time)

Mean time to repair (MTTR)

Packet Loss / Delay / Jitter

Examples

AT&T

Sprint

Verizon Business

9

http://ipnetwork.bgtmo.ip.att.net/pws/index.html

http://www.sprint.com/business/resources/mpls_vpn.pdf

http://www.verizonbusiness.com/terms/us/products/vpn/remotecpe/


Cisco High-Availability Focus

• increase MTBF using resilient HW/SW

• minimize MTTR for system failures (RP, LCs and SW)

• Mitigate planned outages by providing hitless software upgrades

System Level Resilience

• in the core and where redundant paths exist

• Deliver features for fast network convergence, protection & restoration

Network Level Resilience

• Embed intelligent event management for proactive maintenance

• Automation and configuration management to reduce human errors

Embedded Management

and Automation

15


Cisco HA Feature Toolbox: System Level

RPR, SSO

NSF, NSR

SSO Multirouter APS

Stateful NAT/IPSec/Firewall/SLB stateful failover within single chassis

MPLS HA (L3VPN, L2VPN, InterAS, CSC, TE, FRR)

IOS / IOS XR / IOS XE ISSU, dual IOS XE

Service Provider

Core

Service

Provider

Edge

Service Provider

Aggregation

Access Layer

Data Center

Building Block

Internet

16


Network-Level Resiliency

Network Design Resiliency

Dual-homing

APS, GEC, MC-LAG

Event Dampening

Fast Convergence iSPF Optimization (OSPF, IS-IS)

BGP Optimization

Fast BGP Convergence

Graceful Restart (MBGP, OSPF, RSVP, LDP)

EMCP, Anycast, dual RR

VRRP/HSRP/GLBP/SLB/FWLB

MPLS High Availability

LDP Graceful Restart

MPLS/VPN NSF

BFD

MPLS FRR Path Protection

MoFRR

IP FRR

Pseudowire Redundancy

Spanning Tree (MST, PVRSTP...)

..................

Internet

DC

Access

1st Level

Aggregation

2nd Level

Aggregation

IP Edge

SP Core

17


Cost of High Availability

Designing a network for higher Service Availability comes at a cost

Redundant Network Elements

Redundant Links

Redundant System Components (route processors, forwarding processors, power supplies, etc.)

Operational costs

Lower steady-state Utilization levels

Increased configuration and management

Tighter maintenance windows

Availability

Cost ($)

0% 100%

18


Cost of High-Availability: Example

Large SP Network Residential Services (3-Play)

10M Subscribers

1.25 Mbps / subscriber

Up to 96% increased CAPEX for full redundancy! Opex increased due to higher number of network elements

Edge MPLS Aggregation Access Residential Residential

Core

Subscribers AN Agg1 Agg2 Agg3 BNG P

Locations 200,000 4000 500 74 74 74

System Type Generic ASR 9000 ASR 9000 ASR 9000 ASR 1013 CRS-3

Redundancy Scheme Total Cost $M

Chassis Costs $M

Interface Costs (SPA, SFPs), $M

Number of nodes

No redundancy $1,232 $529 $704 4658

Access NW uplink redundancy (Agg1, Agg2, Agg3)

$1,250 $529 $721 4658

AN uplink redundancy $1,563 $531 $1,032 4680

Access Network node redundancy (Agg1, Agg2, Agg3)

$2,423 $1,044 $1,379 9222

Edge link redundancy $2,425 $1,044 $1,381 9222

Edge Node redundancy $2,437 $1,056 $1,381 9296 Values for AN, Agg1, Agg2, Agg3 and Edge nodes only (No Pp-routers). Cumulative redundancy Schemes, GPL

19

Network High Availability


HA Network Map

AN <-> AGG <-> Edge <->

L4-7

App

Failure Detection Keepalives

Recovery Stateful App Redundancy

L3 Failure Detection BFD Keepalives BFD, Keepalives

Recovery MPLS TE FRR, IP Event Dampening, Fast Convergence, IP FRR, NSR / NSF

HSRP /

VRRP/GLBP/SLB/FWLB

Multicast HA

ECMP, iSPF, BGP PIC Core /

Edge, IP / MPLS FRR, LNS

Load sharing / Anycast / Dual

RR, Fast Hello

L2 Failure Detection

EOAM, (VCCV) Keepalives VCCV, EOAM,

MPLS Ping / TR

Keepalives EOAM, MPLS Ping / TR

Recovery GEC / APS / MC-LAG PW redundancy

Bridge Domains

GEC / APS / MC-

LAG

PPP / FR / ATM / HDLC /

GE SSO

GEC, APS, MC-LAG

L0/1 Failure Detection Interrupts Loss of Signal Interrupts Loss of Signal Interrupts

Recovery Module

Redundancy

Path diversity / dual

homing

Module Redundancy Path diversity / dual

homing

Module Redundancy

Aggregation Core Edge Access

Access Domain

22


Link Failure Detection Mechanisms

Loss of Signal (LOS) key to link failure detection

O(ms) with interrupt driven LoS detection on ASR 9000

Carrier Delay may be used to become resilient to link flaps

BFD

Ethernet OAM

MPLS OAM

Media type CC CP CC DP Loopback Performance Traceroute

Ethernet Last Mile IEEE 802.1ah - - -

Ethernet Provider Bridge

IEEE 802.1ag (MAC: Broadcast Domain)

MPLS LDP LDP Hello BFD, Y.1713, Y.1711

LSP Ping -

LSP TR MPLS TE RSVP Hello -

MPLS PW LDP Hello BFD, Y.1711 VCCV Ping - -

IPv4 IGP/BGP Hello BFD IP Ping - IP TR

24


MST Access Gateway - Motivation

Access Gateway aggregates many L2 Access networks into L2/L3-based aggregation networks

Problem: consistent L2 Topology view?

Could run 1 STP per access network

Scalability on access Gateway

Could maintain single access topology across all access networks

Maintenance difficult

Could tunnel BPDUs between legs of access network

Convergence issues upon topology changes O(sec)

25

Aggregation (VPLS/L2VPN …


MST Access Gateway

MST access Gateway sends pre-canned BPDUs into the access network at the hello timer

BPDUs have zero cost to the best STP root bridge (the latter is statically configured)

Forces access network to block one of the access links.

Also snoops the MSTP Topology Change Guard from the access network, flushes the local MAC and triggers VPLS MAC withdrawal (in case of VPLS)

26


Access Link Redundancy using Multi-chassis LAG

mLACP uses ICCP to synchronize LACP configuration & operational state between PoAs, to provide DHD the perception of being connected to a single switch

All PoAs use the same System MAC Address & System Priority when communicating with DHD

Configurable or automatically synchronized via ICCP

Every PoA in the RG is configured with a unique Node ID (value 0 to 7). Node ID + 8 forms the most significant nibble of the Port Number

For a given bundle, all links on the same PoA must have the same Port Priority

ICCP

DHD

PoA

1

LACP PoA2

Node ID: 1

Node ID: 2

Port #: 0x9001, Port Priority 1

Port #:0xA001, Port Priority 2

System MAC:

aaaa.bbbb.cccc

System Priority: 1

27


Ethernet OAM Overview

Ethernet LMI: Automated configuration of CE based on EVCs and bandwidth profiles

L2 connectivity management

IEEE 802.3ah: When applicable, physical connectivity management between devices.

IEEE 802.1ag: Connectivity Fault Management (CFM)

Uses Domains to contain OAM flows and bound OAM responsibilities

Provides per EVC connectivity management and fault isolation

Three types of packets: Continuity Check, L2 Ping, L2 Traceroute

802.1ag

E-LMI

802.3ah 802.3ah 802.3ah 802.3ah 802.3ah 802.3ah

Core

E-LMI

BRKSPG-2202

29


IEEE 802.1ag CFM Concepts

Nested Maintenance Domains (MDs) break up the responsibilities for network administration of a given end-to-end service

Defined by operational boundaries

Nest & touch, but do not intersect

Maintenance Associations (MAs) monitor service instances under a given MD

Defined by set of MEPs at the edge of a domain

Identified by {MA Name + MD ID}

Maintenance Association End Points (MEPs) generate and respond to CFM PDUs

Define boundaries of MD

Initiate & respond to CFM PDUs

Per-Maintenance Association multicast “heart-beat” messages

Carries status of port on which MEP is configured

Uni-directional (no response required)

Transmitted at a configurable periodic interval by MEPs

Catalogued by MIPs at the same MD-Level and service, Terminated by remote MEPs in the same MA

Maintenance Domain

Maintenance Association

Maintenance Association End Points

31


Virtual Circuit Connection Verification (VCCV) Overview

checks connectivity between egress and ingress PEs

VCCV allows sending control packets in band of pseudowires (PW) Signaling component: communicate VCCV capabilities as part of VC label

Switching component: cause the PW payload to be treated as a control packet

VCCV capability is negotiated when the AToM tunnel is brought up depends on the LDP peer and the VC type

Both endpoints must have the same capabilities

marks the payload as control packet for switching purpose; packet follows the PW data path

Control packets sent over the AToM tunnels are intercepted by the egress PE

Type 1

(in-band vccv)

To signal in-band VCCV [RFC4385] using PW ID from PW Control Word

Type 2

(out-of-band VCCV)

Signal out-of-band VCCV inserting MPLS router alert label between tunnel and PW Labels

Type 3 (TTL expiry) Manipulate and Signal TTL exhaust (TTL == 1) for multiple switching point PEs

33


AC AC PE2 PE1 LDP Notification Message PW Status TLV

PW Status Code

MPLS Pseudowire Status Signaling Procedure

PW Status Signaling method selected if supported by both peers.

PEs exchange label mapping messages upon PW configuration.

Simple Label Withdraw status method will be used if one of the peers does not support PW Status Signaling.

PW label will not be withdrawn unless AC is administratively down or the PW configuration is deleted.

PW state set to “down” if the Label mapping is not available.

Capability is on by default.

35


BFD Protocol Overview

Accelerates convergence by running fast keepalives in a consistent, standardized mechanism across routing protocols

Lightweight hello protocol

Neighbors exchange hello packets at negotiated regular intervals

Configurable transmit and receive time intervals

Unicast packets, even on shared media

No discovery mechanism

BFD sessions are established by the clients e.g. OSPF, IS-IS, EIGRP, BGP, …

Client hello packets transmitted independently

BFD Control Packets

OSPF

BGP

EIGRP

IS-IS

BFD

OSPF

BGP

EIGRP

IS-IS BFD

36


Network Convergence — Why It Takes So Long

Detection of Link layer failure ms

ms

10’s of ms

10’s of ms

100’s of ms

10’s of ms

Report failure to Route Controller

Generate and flood an LSP

Trigger and Compute an SPF

Communicate new FIB entries to linecards

Install new FIB entries into linecard HW path

Bottlenec

k

38


Network Convergence — Why It Takes So Long

Detection of Link layer failure ms

ms

10’s of ms

10’s of ms

100’s of ms

10’s of ms

Report failure to Route Controller

Generate and flood an LSP

Trigger and Compute an SPF

Communicate new FIB entries to linecards

Install new FIB entries into linecard HW path

Optimize IGP

Convergence

BGP PIC

Optimize LDP & BGP

Convergence

39


Hierarchical CEF

Optimizes the data plane for sub-second convergence

CEF Data Structure Enhancements

Solves the FIB Download Convergence Bottleneck

LSP and Prefix Independent

Optimizes FIB

Hierarchical CEF Technologies MPLS-FRR

IP-FRR

BGP PIC Core

BGP PIC Edge

Non-Hierarchical CEF Technologies

MPLS Path Protection

Default – IP to MPLSDefault – IP to MPLS

One CEF Update Message per

prefix

BGP Prefix FIB Entry






Adjacency OCE - Interface



X

Failure

Repair

MPLS Label OCE

MPLS Label OCE

MPLS Label OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE

PIC Core – IP to MPLSPIC Core – IP to MPLS

One CEF Update Message for

Multiple Prefixes




X

Failure

Repair

Load Balance OCE

Load Balance OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE







MPLS Label OCE

MPLS Label OCE

MPLS Label OCE

40


IP/MPLS Aggregation

MPLS FRR 50 ms Convergence

Key Features Fast Convergence for Link and Node Failures

Supported Across all Network Topologies

MPLS-TE Traffic Management

SRLG

BW Reservation

Per Tunnel Traffic Statistics

Caveats Requires MPLS and MPLS-TE

No Protection for Ingress or Egress Tunnel Failures

Requires Pre-Computed Backup Paths

Requires “(n-1)!” Tunnels for Full Protection

Applicability Protecting Links in the aggregation network

41

Tunnel LSP

VC LSPs

Link Failure

FRR LSP


MPLS Path Protection

IP/MPLS Aggregation

Tunnel LSP

VC LSPs

Protect Tunnel LSP

Link Failure

Key Features

Optimized for Ring Topologies

Utilizes Pre-Signaled Backup Tunnel

MPLS-TE Traffic Management

SRLG

BW Reservation

Per Tunnel Traffic Statistics

Caveats

Requires MPLS and MPLS-TE

No Protection for Ingress or Egress Tunnel Failures

Convergence Dependant on IGP Prefixes and L2VPN LSPs Under Protection

Applicability

Protecting Ring Topologies

42


MPLS-FRR — CEF For Your Reference

Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second

MPLS-FRR – IP and MPLS

Adjacency OCE - Interface X

Failure

Repair

Midchain OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE

MPLS-TE Path

Protect Tunnel

Label OCE IP &

MPLS

CEF

Adjacency OCE - Interface Midchain OCE Label OCE

Loadbalance OCE

Loadbalance OCE

Loadbalance OCE

IP &

MPLS

CEF

Adjacency OCE - Interface Midchain OCE Label OCE

One CEF Update Message per IGP Prefix and L2VPN LSP!

43


IP FRR-LFA: 50 ms Convergence

Key Features

50 msec Convergence for Link and Node Failures

Works for MPLS and IP Only Environments

Simple

Automatic configuration of “Loop Free Alternate Paths” via OSPF or ISIS

No Tunnels

Caveats

Requires a “Loop Free Path” for Protection

No Bandwidth Reservation

No Support for SRLG

New Feature

Applicability

Strong Solution for Deployments with Cost Effective Bandwidth

Loop Free

Path

R1 R2

R3 R4

R5

Link Failure

No Convergence Required on Routers R2, R3, R4

and R5 to Maintain Green Traffic Flow!

44


Non Stop Forwarding (NSF)

Routers to maintain forwarding state when communication between them is lost

Routing sessions are established with NSF aware peers. Upon HA event, neighboring peers maintain forwarding until routing sessions are reestablished.

Copy of FIB maintained on secondary and used on failure for continuously traffic flow.

Requires neighboring routers to be NSF aware.

Traffic is forwarded

continuously

NSF Aware

eBGP

CE1 PE1 P1

NSF Aware

IGP and LDP

47


BGP PIC Edge

Optimizes BGP Convergence for BGP Next-Hop Change

PE to CE Link Failures

PE Node Failures

CE Node Failures

Applicability

PE Routers

Requires “bgp advertise-best-external” to enable

PE1

VPN1 VPN1

PE2

CE1 CE2

PE3

Link

Failure

BRKRST-3363

48


BGP PIC Edge PE-CE Link Protection

PE2

PE1

CE1 RR1

RR2

PE3

CE2

Traffic flow due to BGP best path

49



PE2

PE1

CE1 RR1

RR2

PE3

CE2


Th

e B

GP

p

re-c

alc

ula

ted

Ba

cku

p P

ath

50



PE2

PE1

CE1 RR1

RR2

PE3

CE2


Th

e B

GP

p

re-c

alc

ula

ted

Ba

cku

p P

ath

PE-CE link Failure

Detects that link is down and CEF layer will

switch to pre-computed backup path

51



PE2

PE1

CE1 RR1

RR2

PE3

CE2


PE-CE link Failure

The best BGP path to CE1 is now through

PE2

52


BGP PIC Edge PE-Node Protection

PE2

PE1

CE1 RR1

RR2

PE3

CE2


Relies on BGP

Add-path

The best BGP path

to CE1 is through

PE1

54



PE2

PE1

CE1 RR1

RR2

PE3

CE2

Traffic flow due to BGP best path The best BGP path

to CE1 is through

PE1

55



PE2

PE1

CE1 RR1

RR2

PE3

CE2

The best BGP path

to CE1 is through

PE1

IGP signals router

is dead

PE-CE node

Failure


56


BGP PIC Edge — CEF

For Your Reference

Typical FIB Programming Rate - ~5000 – 10,000 CEF Updates per second

PIC Edge – IP to MPLS

One CEF Update Message for Multiple Prefixes

Adjacency OCE - Interface X

Failure

Repair

Load Balance OCE




MPLS Label OCE X Adjacency OCE - Interface MPLS Label OCE

Pre-Computed

Backup Path

Load Balance OCE



BGP Prefix FIB Entry Adjacency OCE - Interface MPLS Label OCE


Cleanup After Repair

Load Balance OCE




MPLS Label OCE

Adjacency OCE - Interface MPLS Label OCE Load Balance OCE

59


System High-Availability with Hardware Redundancy

Redundant hardware components Power Supplies

Route Processors

Forwarding Processors

Switching Matrix

SPA Interface Cards

Interface Redundancy typically achieved using IEEE 802.3ad / LACP or APS

Hardware Redundancy needs to be complemented by Software redundancy Features

Cisco Platforms supporting hardware redundancy

CRS-3 ASR 9000 ASR 5000 ASR 1000 Cisco 12000 Cisco 7600 62


CPU BITS/DTI

Interconnect

Con / Aux

Ether

ASR 9000 System Architecture

MAC

NPU

NPU

NPU

NPU

MAC

CPU

Interconnect

MAC

NPU

NPU

NPU

NPU

MAC

CPU

Interconnect

…

Distributed Forwarding Plane for Performance

Up to Eight Linecards

(Autonomous Forwarding)

Distributed IOS®XR based Control Plane for Scale

Dual Route Switch Processors (RSPs)

Dual-Core CPU on Each Linecard

Active/Active Switch Fabric for HA Non-blocking Memory-less Fabric

Service Intelligence with Hi / Lo Priorities,

Unicast & Multicast Recognition, and VoQ’s

Redundant EOBC, Fan Trays, Power supplies (not shown)

CPU BITS/DTI

Interconnect

Con / Aux

Ether

a

rb i t

a

rb i t

63


ASR 9000 Software Architecture

IOS XR offers distributed software architecture

Distributed control plane (e.g. routing)

Linecards have independent FIB/IDB for only local interfaces

Two stage forwarding

Control plane information pushed from RSP to linecards using reliable multicast channel (IPC)

Improves scale and performance

RSPs maintain single consolidated system view

64

Control Plane Data Plane Management Plane



BG

B

RIP

ISIS

OS

PF

Rou

tin

g

PIM

IGM

P

RIB

L2

Dri

ve

rs

AC

L

FIB

Qo

S

LP

TS

Host

Se

rvic

e

PF

I

Inte

rfa

ce

CLI

SN

MP

XM

L

Netf

low

Ala

rm

Pe

r.fM

gm

t

SS

H

SS

H

SS

H

Checkpoint DB Multicast IPC System DB

Distributed Infrastructure

Memory Mgmt IPC Mech Synch. Services Scheduler

Kernel System Services


ASR 9000 High availability Infrastructure

65

Distribution improves fault tolerance and recovery time by localizing the database and system management functionality to each node

Granular process restart allows for fast recovery from failures

Hitless process re-starts!

IOS XR is designed to optimize the switch over between redundant hardware elements IOS XR is designed to route around fabric failure

Line cards are protected by link bundling, APS, IPS, ECMP etc.

RSP failover has no impact on the control protocols or packet forwarding running on the linecards

Contained Contained

Contained

BGP IS-IS RIB QoS FIB IP

Stack CLI XML Alarm File System

Inter Process

Communication SSH SNMP Netflow L2

Drivers PFI ACL IGMP PIM OSPF

OS Distributed Middleware

Hot

Warm

Cold

ACTIVE RSP STANDBY RSP

Process A

Process B

Process C

Process A

Process B

Process C

Check Point Server

Check Point Server


Control Plane Data Plane

Example: ASR 1000 System Redundancy

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

Route Processor (Standby)

Route Processor

(active)

Embedded Services Processor (Standby)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Embedded Services Processor

(active)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Passive Midplane

RP RP

GE, 1Gbps I2C SPA Control SPA Bus

Route Processor (standby)

RP

Interconn.

Embedded Services Processor

(active)

FECP

Interconn.

QFP subsys-

tem Crypto assist

Embedded Services Processor (standby)

FECP

Interconn.

QFP subsystem Crypto

assist

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

SPA SPA

IOCP SPA

Agg.

…

Interconn.

Passive Midplane

Route Processor

(active)

RP

Interconn.

SPA-SPI, 11.2Gbps Hypertransport, 10Gbps

ESI, (Enhanced Serdes) 11.5Gbps

66


RPact

FMRP

QFP Client

RPsby

SPAs

ASR 1006 High Availability Infrastructure

HA operates in a similar manner to other protocols on the ASR 1000

Reliable IPC transport used for synchronization

RF RF

IPC Message

Qs

IDB State Update Msg IDB State Update Msg

IOSact IOSsby

I

P

C

I

P

C

CF CF Interconnect

Used for

IPC and

Check-

pointing

Non-HA-Aware

Application

Non-HA-Aware

Application

Driver/Media

Layer

Mcast

CEF

Config

…

Driver/Media

Layer

Mcast

CEF

Config

…

MLD MLD

IPC Message

Qs

ESPsby ESPact

QFP Client

FMRP

FIB MFIB FIB MFIB

FMESP FMESP

IDB RIB RT MRIB IDB RIB RT MRIB

FIB MFIB FIB MFIB

68


1. Failover Triggers: Hardware Failures

What hardware failures?

a. CPUs: RP-CPU, QFP, FECP, IOCP, interconnect CPU, I2C Mux, ESP Crypto Chip,

b. Memory: NVRAM, TCAM, Bootflash, RP SDRAM, FECP SDRAM, resource DRAM, Packet buffer DRAM, particle length DRAM, IOCP SDRAM, …

c. Interconnects: ESI Links, I2C links, EOBC Links, SPA-SPI bus, local RP bus, local FP bus …

d. Other: heat-sinks, …

Detected using

Software running on the failed hardware will crash -> see software crashes

Watchdog timers: low level watchdogs running in O(min) that can time out

Notification of other field-replaceable units (FRUs)

CPLD interrupts / register bits within O(ms) controlled by CMRP

Initiate fail-over events

Hardware failures are typically fatal such that modules need to be replaced!

JTAG: RP can program CPLD on other modules. Test interconnects and other boards (primarily for RMAd hardware)

SIP

SPA SPA

IOCP

SPA

Agg.

…

Interconn.

ESP FECP

Interconn. QFP subsys-

tem

Crypto assist

RP CPU

Interconn.

Interconn.

70


2. Failover Triggers: Software Failures

What Software Failures? a.Kernel: Linux on RP / ESP / SIP

b.Middleware: CM, FM, IM

c.IOS

d.SPA drivers

Detected using the process manager (PMAN) PMAN: every software process has a corresponding PMAN process to check its health

if software process crashes, PMAN will detect via a signal from the kernel in O(ms) and react

IPC: between 2 IOS (and only for IOS)

Hardware watchdog timers that supervises Linux and software stack (O(min))

Kernel will take the module down in a controlled manner IOS, CMESP, CMSIP, FMESP, QFP Driver/Client, are not re-startable!

PMAN-initiated failover using CPLD register bits for ESP or RP (Failover in O(3ms))

Note: some other processes are re-startable (CMRP, FMRP, SSH, Telnet…)

Kernel will try to re-start the processes in this case

If unsuccessful, then the process will be held down and console message logged

SIP

SPA SPA

IOCP

SPA

Agg.

…

Interconn.

ESP FECP

Interconn. QFP subsys-tem

Crypto assist

RP CPU

IOS

Chassis Mgr.

Forwarding Mgr.

Chassis Mgr.

Forwarding Mgr.

QFP Client / Driver

Interconn.

Chassis Mgr. SPA driver

SPA driver

SPA driver

SPA driver

Interconn.

QFP code

IOS

Kernel (incl. utilities)



71


SIP CMSIP Kernel

RP (slot 1) CMRP Kernel IOS

RPact Failover Procedure

ESP (slot 0) CMESP Kernel FMESP FMRP

RP (slot 0) CMRP Kernel IOS FMRP

ESP (slot 1) CMESP Kernel FMESP

SIP CMSIP Kernel

SIP CMSIP Kernel

ACT ACT SBY SBY

Failure

ACT Restart New RPact information

Close ESI links (ESP)

Establish ESI links

State information

If not received in time,

send restart message.

Update H/W

component

file system ESI link status

For Your Reference

Detect RPact

failure

72


Failover

Take-over control

using checkpointed

state Forwarding State information

Check updated

state and discard

old state

Check updated

state and discard

old state

Service recovered

Start kernel start

Start CM Start IOS

H/W initialization

Initialize EOBC

SIP CMSIP Kernel

RP (slot 1) CMRP Kernel IOS

RPact Failover Procedure (Cont.)

ESP (slot 0) CMESP Kernel FMESP FMRP

RP (slot 0) CMRP Kernel IOS FMRP

ESP (slot 1) CMESP Kernel FMESP

Start FM

SIP CMSIP Kernel

SIP CMSIP Kernel

Detect RPsby

SBY Forwarding State information ESI link status

Other RP information

Run Mastership

For Your Reference

73


IOS High Availability

During the initialization process IOS is loaded on both the RPact and RPsby

IOS learns about any events

Uses Redundancy and checkpointing facility

Redundancy facility (“when to synchronize”)

A process to help in synchronization and coordination of switchovers (e.g. switchover events, switchover control and monitoring)

Clients of the RF maintain the databases that are synchronized using the RF

Examples: reliable internal data transfer service, Event notification mechanisms, logging, etc.

Checkpointing facility: (“how to synchronize”)

Defines a set of APIs and transport for different SSO-aware features to copy state

Helps to synchronize state data in a consistent, repeatable and well-ordered manner

Keeps checkpointing state

Interfaces to RF

Line Card

RF RF

IPC Message

Queues

IPC Message

Queues

IDB State Update Msg IDB State Update Msg

Active RP Standby RP

I

P

C

I

P

C

CF CF

Interconnect

Used for

IPC and

Check-pointing

Non-HA-Aware

Application

Non-HA-Aware

Application

Driver/Media

Layer

PPP

CEF

L2TP

Config

…

Driver/Media

Layer

PPP

CEF

L2TP

Config

…

IDB FIB IDB FIB

For Your Reference

76


IOS High Availability – Steady State

In steady state:

Routing state is synchronized between IOSact and IOSsby

Routing processes have been created on IOSsby

If NSR, checkpointed current neighbor state and routing DB

If GR, protocols have the config from the config sync, but no neighbor DB and no routing DB (needs

to be re-acquired from neighbors)

RIB is empty on IOSSBY

CEF IS Stateful

Interfaces are logically in DOWN state on IOSsby

QoS synchronized via Config-sync

Statistic counters are 0 in general

Exception e.g. for Broadband, where statistics counters are synchronized

77


Cisco Software High-Availability Support

78

Stateful Switchover (SSO) support for features provides the synchronization of dynamic feature state between hardware modules

Configuration synchronization ensures that the running config is synchronized on the route processors

Dynamic State Preservation ASR 1000 ASR 9000

Connectivity Protocols &

Infrastructure

FR, ATM, PPP, MLPPP, HDLC, 802.1Q, BFD (BGP, IS-IS, OSPF),

CEF, SNMP,

BFD (OSPF, BGP, IS-IS, Static)

Routing & IP Services

IS-IS, ARP, HSRP/GLBP, QoS, OSPF, DHCP, VRRP

NSR: BGP, OSPF, MP-iBGP (rodmap for IS-ISv6)

NSF/GR: IPv6, OSPFv3, IS-IS, RIPng, PIMv6, MLD, (roadmap for IS-

ISv6

NSF (ISIS, OSPF, BGP),

NSR (ISIS, OSPFv2, OSPFv3, BGP)

Multicast MFIB, IGMP, PIM-SM, PIM-SSM, PIM-BiDir, MLD, MSDP, Auto RP,

BSR, mVPN, mLDP GR, NSR for mLDP

NSF Multicast, BFD for PIM, MoFRR

MPLS Protocols L3VPN, LDP , VRF-aware BFD, QoS, TE, EoMPLS, L2VPN, L2TPv3,

LDP, t-MDP, NSR MPLS TE (with OSPF, IS-IS, RSVP)

NSF (LDP, T-LDP, RSVP-TE)

NSR (LDP, T-LDP), BFD for MPLS FRR, VRF-aware

BFD

Broadband PPPoE, L2TP (LAC, LNS), DHCPv4/v6, AAA, session state (virtual

templates), ISG/IP Sessions, ANCP, LI

PPPoE (including nV)

Security SSO, Stateful Inter-chassis redundancy for FW / NAT Roadmap

SBC SSO n.a.

For Your Reference


Non Stop Routing (NSR)

Routers to maintain routing state and forwarding state when communication between them is lost

Routing sessions are maintained between processors on a failure, allowing routing sessions to stay up with Peer

Copy of FIB maintained on secondary and used on failure for continuously traffic flow

No need for neighboring routers to be NSF aware or capable. Can give high reliability without upgrading CE.

Traffic is forwarded continuously

eBGP

CE1 PE1 P1

IGP and LDP

79


BGP NSR

Implemented by hardening code for

BGP RIB checkpointing

BGP TCP interaction

Configuration IOS/IOS XE router bgp <asn>

address-family ipv4 vrf vrf-name

address-family ipv6 [unicast | multicast | vpnv6] [vrf vrf-name]

address-family vpnv6 [unicast | multicast]

neighbor x.x.x.x ha-mode sso

No peer session flaps on VPNv4/v6 CE when RP switches over

Route updates during RP switchover are announced to VPNv4/v6 CE peers

NO delays which prevents data black-holes during RP switchover as in the case of graceful restart peers

80


OSPFv2/v3 NSR

Provides the ability to perform hitless RP switchovers when OSPF is used as the routing protocol (Expect zero-traffic loss across such HA events)

To enable OSPFv2 Non-Stop Routing (NSR) router ospf <process id> [vrf <vrf name>]

nsr

To enable OSPFv3 Non-Stop Routing (NSR) router ospfv3 process-id

nsr

Activated on a per-process basis (for both ipv4 or ipv4 VRF for PE-CE sessions)

Depends on the forwarding plane’s ability to retain state across control plane restarts and RP switchovers

Alleviates dependency on OSPFv2 protocol extensions (NSF) Neighboring routers are unaware that a router is NSR-capable Neighboring routers are unaware that a router has gone through an RP switchover

Provides near-transparent RP switchover capability OSPF adjacencies remain up Minimal state refreshed by the restarting router post switchover Scalable to larger link state database sizes and number of neighbors

81


High Availability for Advanced Service Models

Many SP Services already go beyond standard L3VPN / L2VPN / transport services

Increasing subscriber management capabilities and L4-L7 services

Examples:

Subscriber Management

Multicast

Session Border Controller

Firewall

IPSec

LI

Some Services can be made highly-available using Intra-chassis redundancy (e.g. IPSec, Firewall, NAT, PPPoX, L2TP)

Stateless inter-chassis redundancy available for BNG

Stateful Inter-chassis redundancy available for NAT, Firewall and SBC on the Cisco ASR 1000

83


L3VPN Key HA Technologies

CPE

BFD for PE-CE Link

Detection

NSF/NSR for Chassis HA

PE Multihoming Intra-Site PE for PE Diversity

Inter-Site for SP Facility Diversity

Access

Circuit Diversity - Physical

Diversity for Multihomed

CPE

Physical Circuit Diversity is Not

the Default

Must be Requested from the SP

Edge

BFD for PE-CPE / PE-P Link Detection

NSF/NSR for Chassis HA

IP Event Dampening for PE-CPE

IP-FRR for PE-P For Cost Effective PE-P Bandwidth

BGP PIC Core

BGP PIC Edge for Multi-Homed CPE

Edge Access

Mobile

CPE

Business

VRF Green

Core

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Access

Mobile

CPE

Business

VRF Green

Business

VRF Blue

Business

VRF Red

Business

VRF Orange

Edge

Site A

Site B

Site C

Site D

84


L2VPN — Pseudowire Redundancy

Active-Standby PW Access Circuit Redundancy

L2TPv3 and MPLS Support

Detection Mechanisms

IGP Convergence for Remote PE Failure

LDP Signaling for PE-CE Failure

LDP Timeout for Remote PE Software Failure

CE

P

P

P

CE

PE1 PE2

PE23

P

Standby PW

Active PW

BRKSPG-2207

85


MPLS-TP Infrastructure Resiliency

Statically setup Working and Protection LSP, 1:1 protection

Protection switching can be triggered by 1. LOS, Physical failure

2. BFD timeout

3. Detected defect condition (LDI/AIS, LKR)

4. Administrative action (lockout), Far end request (lockout)

Revertive mode, after wait to restore the traffic is restored over working path

3800 3600

ASR903 ASR9000

Core MPLS-TP

CPE

Working Path

Protection Path

BFD CC

AIS/LDI AIS/LDI

1

2

3

4

86


Multicast High-Availability Behavior

Before failure

Multicast state is synchronized from Rpact to RPsby

Configuration

MLDv1/v2 state information

PIM or MRIB state are NOT synchronized

MFIB also synched to ESPact and ESPsby

After failure RPsby sends out PIM hellos to all neighbors

PIM neighbors re-send PIM state

Newly active RP re-builds the PIM state

IGP reconverges to assure uRPF check

MFIB and ESP updates proceed to incorporate refreshed PIM state

ESPact continues to forward multicast traffic based on its version of the MFIB Forwarding of multicast packets is NOT disrupted

RPact

FMRP

QFP

Client

RPsby

SPAs

RF IDB State

Update Msg

IOSact IOSsby

I

P

C

I

P

C

CF

Driver/Media

Layer

Mcast

CEF

Config

…

MLD

ESPsby ESPact

QFP

Client

FMRP

FMESP FMESP

MFIB

RF

CF

IDB State

Update Msg

Driver/Media

Layer

Mcast

CEF

Config

…

MLD

MFIB MRIB MFIB

MFIB

MRIB

88


Receiver

Multicast join on primary path

Multicast join on backup path

Data packets are received from the primary and secondary paths

The redundant packets are discarded at topology merge points due to RPF checks

Failure:

Interface chance on where packets are accepted

Backup path interfaces become ‘active’

Multicast only Fast Re-Route (MoFRR)

POP1

POP2 POPN

IPTV source

Normal path

Alternate PIM

Configuration and Restrictions Dependency on ECMP and will not work without it

Disabled by default and enabled through a cli

Applicable to IPv4 multicast only and not IPv6 multicast

Works only for SM S,G and SSM routes

Works where the rpf lookups are done in a single vrf

Extranet routes are not supported

Both primary and secondary paths should exist in the same multicast

topology.

89


Stateful Application Switchover: PPP

Copies state information for PPP, PPPoE, and PPPoEoVLAN Sessions

Switch-over is transparent to peers Sessions are not torn-down / re-established

PPP, PPPoE, and PPPoEoVLAN Session States: Configuration (through config synch), including QoS configuration, ACLs

Session identifiers

PADR frame (cached)

RADIUS session attributes

Physical interface

VAI identifier

MD5 signature

Statistics are synchronized on ASR 1000!

90


Stateful Application Switchover: L2TP

RPact synchronizes state with RPsby

State includes configuration, PPP session IDs, L2TP CC sequence numbers etc.

Sequence numbers (Ns, Nr) for L2TP Control Connections (CC) are only synched once for a packet window of X (i.e. once every X L2TP control packets)

ESP

L2TP Control Connections

RPsby

L2TP Tunnel

LNS

RPact

91


Motivation for Stateful Application Inter-Chassis Redundancy

Current Intra-chassis HA typically protects against

Control Plane (RP) Failures

Forwarding Plane (ESP) failures

Interface failures can be mitigated using link bundling (e.g. GEC)

Any other failures may result in recovery times O(hours)

Inter-chassis redundancy provides additional resilience against

Interface Failures

System failures

Site failures (allowing for geographic redundancy)

RP

RP

FP

FP

SIP

SIP

SIP

SIP

94


System Level Redundancy

Example: VSS, nV

Failover Granularity at the System Level

Control-plane active-standby

Active RP considers ‘remote’ linecards under its control

Forwarding-plane active-active

No application granularity for failover

Need to ensure all features are SSO capable

Example: RG Infra

Failover Granularity at the Application Level (NAT, Firewall, SBC etc)

Control plane active-active

Each RP only considers its own linecards, but synchronizes application state

Forwarding-plane active-active

E.g. can have one set of firewall services resilient, and other set of firewall services non-resilient

RPact

Fabric

LC

Fabric

LC LC LC

RPsby RPact RPsby

Failover

RPact

ESP

SIP

ESP

SIP SIP SIP

RPact

Failover

FW FW

Application Level Redundancy

95


Achieving Redundancy via Clustering

Active/active

PW

Backup PE

Primary PE

Backup PE

PW 1

PW 2

PW 4

PW 3 Standby

Active

Standby

Standby Standby

Active

Redundancies required in current architecture: New protocols, extra routing peers

Clustering simplifies UNI & NNI-facing links, end-to-end network connectivity

Primary PE

Active

Active

ICCP

Active / Standby

LAG

98


ASR 9000 nV Edge Overview

Chassis Members are linked via L1/L2 control plane links (10ms latency assumed) Dedicated ports on RSP

Extend internal EOBC between the chassis members

1GE or 10GE using Ethernet SNAP encapsulation on internal MAC addresses

Data plane fabric is extended between the chassis via regular Ethernet VLAN line-card ports Can mix ports on a linecard for data plane extension and downstream connectivity

No requirement for external fabric chassis

10GE or 100GE up to 32 ports

Both contol plane and Data plane extensions are assumed to be redundant

99

Chassis 1

RSP1

(act) RSP2

LC LC LC LC

Chassis 2

RSP1

(sby) RSP2

LC LC LC LC


ASR 9000 nV Edge Forwarding

Keep the existing IOS-XR two-stage forwarding model

Inter-chassis data links simulate the switch fabric has similar features as switch fabric, for example, fabric qos. Packet load balancing over inter-chassis links is same as regular link bundle: per-flow based

In case of ECMP or link bundle paths cross two chassis, it prefer local port instead of load balancing packet to the other chassis. This is to reduce the inter-chassis link usage as much as possible. Can be turn off by user CLI

Only single Multicast copy is sent over inter-chassis link. Multicast replication is done on egress line cards and fabric on the local chassis

100

Chassis 1

RSP1

(act) RSP2

LC LC LC LC

Chassis 2

RSP1

(sby) RSP2

LC LC LC LC


ASR 9000 nV Edge Failures

Only one Active RSP, Only one standby RSP at a given time, which are located on two different chassis

SSO/NSF/NSR works exactly the same way as two RSPs on the same chassis

Reliable out of band control channel between two chassis

IOS-XR control plan can tolerant hundreds of msec latency*, although the latency can impact overall service convergence time

Virtual Chassis is always on as long as there is one chassis and one RSP alive 102

Chassis 1

RSP1

(act) RSP2

LC LC LC LC

Chassis 2

RSP1

(sby) RSP2

LC LC LC LC

RSP2

(sby)

RSP1

(act)

RSP2

(sby)


Introduction to RG-Infra

RG Infra is the IOS Redundancy Group Infrastructure to enable the synchronization of application state data between different physical systems

Does the job of RF/CF between chassis

Infrastructure provides the functions to Pair two instances of RG configured on different chassis for application redundancy purposes

Determine active/standby state of each RG instance

Exchange application state data (e.g. for NAT/Firewall)

Detect failures in the local system

Initiate & manage failover (based on RG priorities, allows for pre-emption)

Assumptions Application state has to be supported by RG infra (ASR 1000 currently supports NAT, Firewall, SBC)

Connectivity redundancy solved at the architectural level (need to ‘externalize’ the redundant ESI links of the intra-chassis redundancy solution)

103


Redundancy Groups Functions

Registers applications as clients

Registers (sub)interfaces / {SA/DA}-tuplets in case of firewall

Determines if traffic needs to be processed or not

E.g. for Firewall: if a subset of sessions are associated with a RG in active state, then the Firewall application will perform normal

processing for those sessions and actively sync the session state to another device that has the same RG in STANDBY state.

For Firewall sessions that are associated with a RG in STANDBY state, the session information will be synchronized from a

device that has the RG in ACTIVE state.

Communicates control information between RGs using a redundancy group protocol Advertisement of RGs and RG state

Determination of peer IP address

Determination of presence of active RG

Synchronizes application state data using a transport protocol

Manages Failovers! RPact

ESP

SIP

FW RG

SPA SPA

SIP SPA SPA

RPact

ESP

SIP

RG

SPA SPA

SIP SPA SPA

FW

RG state data

RG control data

Active

104

Case Studies


Case Study: Highly Available IP Architecture for Mobile

Cellsite

Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MTSO

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Service Termination

GE 10 GE QFP

Internet

Core

L2 Domain L3 Domain L3 Domain

Local VLANs or T1s

Transport VLANs / Static Routes / BGP PIC Edge / BFD protection OSPF/RIP/VRRP

FE T1

EoMPLS Backhaul

One Second Convergence Requirement

CSR EvDO/LTE VRF

1xRTT VRF

QFP

108


Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet

Core

Static Routes establish connectivity between loopback addresses

VRF VRF

VRF

VRF

Static routes for cellsite reachability BGP PIC Edge for Layer-3 convergence VRRP for MTSO

VRF Tables Updated

BFD sessions

established

CSR EvDO/LTE VRF

1xRTT VRF

QFP

Case Study: Highly Available IP Architecture for Mobile — Transport

109


Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet

Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

Steady state: CSR distributes flows across both Agg’s using ECMP. Traffic could flow across Agg inter switch links. Each Agg handles traffic related to all services from the cell-site.

1xRTT

1xRTT

EVDO

EVDO

1xRTT

Case Study: Highly Available IP Architecture for Mobile — Steady-State Traffic Flows

110


Case Study: Highly Available IP Architecture for Mobile — Link Failure

Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet

Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

Steady state: Traffic flows distributed across both Agg. Failure: GE link from MSPP to Agg1 fails. Action:

BFD session to Agg1 times out at CSR. Agg1 next hop removed from forwarding table. Traffic flows resume across existing path to Agg2.

Results: Traffic flows to Agg1 via Agg2.

No changes to LAN side connectivity

x

111


Case Study: Highly Available IP Architecture for Mobile — Aggregation Switch Failure

Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet

Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

x

Steady state: Traffic flows distributed across Agg. Failure: Agg1 power outage. Action:

BFD and VRRP sessions time out BGP and OSPF neighbors drop due to BFD BGP PIC Edge ensures sub-second convergence Traffic flows resume across existing path thru Agg2.

Results: Traffic flows via Agg2 to end hosts.

112


Case Study: Highly Available IP Architecture for Mobile — CSR Failure

Agg1 CSR

MSPP MSPP

MME

SGW

MSC

RNC

LTE Core

CDMA Core

PE

PE

EvDO/LTE VRF

1xRTT VRF

QFP

MPLS VPN

VRF 1xRTT

VRF EvDO/LTE

Agg2

Internet

Core

CSR EvDO/LTE VRF

1xRTT VRF

QFP

x

Steady state: Traffic flows distributed across CSR. Failure: CSR power outage. Action:

BFD sessions time out BGP neighbors drop due to BFD Mobile handsets resync to neighboring cell site

Results: Mobile handset voice connectivity is maintained.

113

Summary


Summary

Motivation for High Availability in SP Aggregation Networks

Network Level High Availability




Case Studies

Summary and Conclusions

115


Key Takeaways

High-Availability becoming increasingly deployed in Aggregation Networks

Motivated by experiences with MPLS Core Networks

Many high-availability techniques deployed in the core are now applied in

MPLS aggregation networks

MPLS TE FRR, BFD, EOAM, Pseudowire Redundancy …

Service High Availability requires comprehensive approach including the

deployment of

Network level resiliency

System Level resiliency

L4-7 service resiliency

Stateful Inter-chassis redundancy increasingly being considered to provide

geographic redundancy for applications

High Availability comes at a cost (CAPEX & OPEX)!

116


Recommended Reading

D. C. Lee, “Enhanced IP Services for Cisco Networks”,

ISBN-13: 978-1-57870-106-3

A. Sayeed, M. Morrow, “MPLS and Next-Generation Networks”,

ISBN-13: 978-1-58720-120-2

V. Bollapragada et. Al, “Inside Cisco IOS Software Architecture “,

ISBN-13: 978-1-57870-181-0.

R. Wood, “Next-generation Network Services”,

ISBN-13: 978-1-58705-159-3.

K. Lee, F. Lim, B. Ong, “Building Resilient IP Networks”,

ISBN-13: 978-1-58705-215-6

T. Szigeti, C. Hattingh, “End-to-End QoS Network Design: Quality of Service in LANs, WANs, and VPNs:,

ISBN-13: 978-1-58705-176-0

117


Whitepapers on CCO

Cisco IOS High Availability

http://www.cisco.com/en/US/tech/tk869/tk769/tech_white_papers_list.html

http://www.cisco.com/en/US/products/ps6550/prod_white_papers_list.html

Campus Network for High Availability Design Guide

http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html

Cisco Validated Designs

http://www.cisco.com/en/US/netsol/ns742/networking_solutions_program_category_home.html

ASR 9000 Cisco ASR 9000 Series High Availability: Continuous Network Operations

Introduction to Cisco ASR 9000 Series Network Virtualization Technology

Distributed Virtual Data Center for Enterprise and Service Provider Cloud

ASR 1000 Cisco ASR 1000 Series Aggregation Services Routers

Cisco ASR 1000 Series: ISSU Deployment Guide and Case Study

Cisco Unified Border Element (SP Edition) on Cisco ASR 1000 Series

Cisco Unified WAN Services: Services, Security, Resiliency, and Intelligence

118









http://www.cisco.com/en/US/prod/collateral/routers/ps9853/white_paper_c11-501797.html

http://www.cisco.com/en/US/solutions/collateral/ns341/ns524/ns562/ns592/asr_nv_100611.pdf



http://www.cisco.com/en/US/prod/collateral/routers/ps9343/white_paper_c11_517184.html


http://www.cisco.com/en/US/prod/collateral/routers/ps9343/solution_overview_c22-450358_ps9343_Products_White_Paper.html


Call to Action…

Visit the World of Solutions:-

Cisco Campus

Walk-in Labs

Technical Solutions Clinics

Meet the Engineer

Lunch Time Table Topics, held in the main Catering Hall

Recommended Reading: For reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2014

119

http://www.pearson-books.com/CLMilan2014




Complete your online session evaluation

Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt

Complete Your Online Session Evaluation

120

best practices to deploy high-availability...

Documents