iwan implementing performanced2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkrst-2362.pdf · •iwan...

IWAN – Implementing Performance Routing (PfRv3)

Jean-Marc Barozet – Principal Engineer, IWAN Technical Marketing

BRKRST-2362

• IWAN Introduction

• IWAN Domain

• Performance Routing Principles

• Next Hop Selection Logic

• Deploying IWAN Intelligent Path Control

• Conclusion

Agenda

IWAN Introduction

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Intelligent WANSolution Components

MPLS

Unified

Branch

3G/4G-LTE

Internet

PrivateCloud

VirtualPrivateCloud

PublicCloud

Application Optimization

Enhanced Application

Visibility and Performance

Secure Connectivity

Comprehensive

Threat Defense

Intelligent Path Control

Application

Aware Routing

TransportIndependent

Simplified

Hybrid WAN

Enterprise IWAN - IWAN-App/APIC-EM

SP-IWAN - vMS/NSOORCHESTRATION

BRKRST-2362 5


IWAN: Intelligent Path ControlPerformance Routing

Branch

MPLS

Internet

Virtual PrivateCloud

Private Cloud

• PfR monitors network performance and routes applications

based on application performance policies

• PfR load balances traffic based upon link utilization levels

to efficiently utilize all available WAN bandwidth

Other traffic is load

balanced to maximize

bandwidth

Voice/Video/Critical will be

rerouted if the current path

degrades below policy thresholds

Voice/Video/Critical take

the best delay, jitter, and/or

loss path

BRKRST-2362 6


Master Controller

commands path changes

based on traffic class

policy definitions

Best

Path

MC+BR MC+BR BR MC+BR

Path Enforcement

BR BR

MC

Measure the traffic flow

and network performance

and report metrics to the

Master Controller

Performance

Measurements

MC+BR MC+BR MC+BR MC+BR

MC

Measurement

MC

BR BR

Border Routers learn

current traffic classes

going to the WAN based

on classifier definitions

Learning

Active TCs

MC+BR MC+BR MC+BR MC+BR

Traffic

Classes

Learn the Traffic

BR BR

MC

How PfR Works – Key Operations

Define Traffic Classes

and service level

Policies based on

Applications or DSCP

Define Your Traffic Policy

BRKRST-2362 7


Overlay routing over tunnels

Overlay tunnels (DMVPN)

Internet Routing

Transport routing

Perimeter

Security

Perimeter

Security

MPLS-VPN Routing

PfR path selection policies

PfR intelligent routing

• CPE-to-CPE overlay

enables separation of

transport (underlay) and

VPN service (overlay)

• Point to multipoint WAN

connections with secure

tunnel overlay

architecture

• Intelligent policy routing

to provide cost

optimization and dynamic

load balancing

AVC/QoSAVC/QoS

IWAN Layered Solution

BRKRST-2362 8

IWAN Domain


PfR Components

• The Decision Maker: Master Controller (MC)

• Apply policy, verification, reporting

• No packet forwarding/ inspection required

• Standalone of combined with a BR

• VRF Aware

• IPv4 only (IPv6 Future)

• The Forwarding Path: Border Router (BR)

• Gain network visibility in forwarding path (Learn, measure)

• Enforce MC’s decision (path enforcement)

• VRF aware

• IPv4 only (IPv6 Future)

• The BRs automatically build a tunnel (known as an auto-tunnel) between other BRs at a site. If the MC instructs a BR to redirect traffic to a different BR, traffic is forwarded across the auto-tunnel to reach the other BR

MC1

BR1 BR2

MC/BR

MC/BR BR

Site1

Site3

Site5

BRKRST-2362 10


IWAN Domain

Site ID 10.3.0.31

MC/BR MC/BR

BR1 BR2 BR3 BR4

MC1 MC2

DMVPNPATH1

DMVPNPATH2

DCIWAN Core

DC1 DCn

• Collection of sites that share the same set of policies

• An IWAN domain includes:

– A mandatory Hub site, optional Transit sites,

– As well as Branch sites.

– Each site has a unique identifier (Site-Id, derived from the loopback address of the local MC)

• Central and headquarter sites play a significant role in PfR and are called an IWAN Point of Presence (POP).

– Can act as a transit site to access servers in the datacenters or for spoke-to-spoke traffic

– Each of these sites will have a unique identifier called a POP-ID

• Branch Sites

– These will always be a DMVPN spoke, and are a stub sites where traffic transit is not allowed

– BRs must be directly connected to the branch MC. This can be a direct link, a transit VLAN through a switch, or a GRE tunnel

POP1 - HUBSite ID = 10.1.0.10

POP2 - TRANSITSite ID = 10.2.0.20

Site ID 10.4.0.41

Site ID 10.5.0.51

MC/BR BR

BRKRST-2362 11


Site ID 10.3.0.31

BR1 BR2 BR3 BR4

DMVPNPATH1

DMVPNPATH2

DCIWAN Core

DC1 DCn

POP1 - HUBSite ID = 10.1.0.10

POP2 - TRANSITSite ID = 10.2.0.20

Site ID 10.4.0.41

Site ID 10.5.0.51

BR

IWAN Domain

• Each site runs PfR

• The local MC peers with the logical domain controller (aka Hub MC) to get its policies, and monitoring guidelines.

• Local MC gets its path control configuration and policies from the logical IWAN domain controller through the IWAN Peering Service

• IWAN Peering based on Service Announcement Framework (SAF)

IWAN Peering

MC1

MC/BR MC/BR MC/BR

MC2

BRKRST-2362 12


Site ID 10.3.0.31

R31 R41

R11 R12 R21 R22

R10 R20

DMVPNPATH1

DMVPNPATH2

POP1 - HUBPOP-ID 0

Site ID = 10.1.0.10

POP2 - TRANSITPOP-ID 1

Site ID = 10.2.0.20

Site ID 10.4.0.41

Site ID 10.5.0.51

R51 R52

Path Discovery

• Assign Path Name and Path Id to every tunnel on Hub/Transit BRs

• Hub and Transit BRs send Discovery Packet with path names from to all discovered sites

Path MPLSPath-id 1

Path INETPath-id 2

Path INETPath-id 2

Path MPLSPath-id 1

1. Discovery

Probes

2. WAN Path

Discovered

3. Three Performance

Monitors dynamically

Assigned

Site ID 10.1.0.10

Site ID 10.2.0.20

BRKRST-2362 13


WAN Interface – Performance Monitors• PfR automatically configures 3 Performance

Monitors instances (PMI) over external interfaces

• Monitor1 – Site Prefix Learning (egress direction)

• Monitor2 – Aggregate Bandwidth per Traffic Class (egress direction)

• Monitor3 – Performance measurements (ingress direction)

R31

2 31 2 31

BRKRST-2362 14


Site Prefix

SITE1

PfR Site-Prefix

Configured

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

Configured

10.2.0.0/16

10.10.0.0/16

Hub/Transit sites: static definition of site prefixes is mandatory

BR2

MC/BR MC/BR MC/BR BR

Hub MC

BR1 BR3 BR4

MC1Transit MC

MC2

Path MPLSPath-id 1

Path INETPath-id 2

Path INETPath-id 2

Path MPLSPath-id 1

DMVPNMPLS

DMVPNINET

POP-ID 0 POP-ID 1

HUB SITESite ID = 10.1.0.10

TRANSIT SITESite ID = 10.2.0.20

Branch Branch Branch

SITE3

PfR Site-Prefix

Automatic

10.3.0.0/16

Branch sites: static definition of site prefixes is optional but recommended

10.10.0.0/16

BRKRST-2362 15


PfR Multiple Next Hop Support

BranchSITE-ID 10.3.0.31

R31 R41

R13 R14

R10 R20

MPLS1

R11 R12 R23 R24R21 R22

INET1

Hub SiteSITE-ID 10.1.0.10POP-ID 0

Hub MC Transit MCTransit SiteSITE-ID 10.2.0.20POP-ID 1



Branch sites: Only one BR per transport – Multiple Next Hop not supported

R51 R52

Hub/Transit: Multiple BRs per transport supported

BRKRST-2362 16


PfR Channels


R31

R10 R20

MPLS1

R11 R12 R21 R22



SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

Channels

• Logical entities used to measure path performance per DSCP between two sites

- Per destination site, Path Id and DSCP

- Created based on real traffic observed on border routers


R41

INET1

R13 R14 R23 R24

User traffic DSCP AF21to 10.10.0.0/16

BRKRST-2362 17


PfR Channels


R31

R10 R20

MPLS1

R11 R12 R21 R22



Channels

• Logical entities used to measure path performance per DSCP between two sites

- Per destination prefix, DSCP and Path Id

- Created based on real traffic observed on border routers


R41

INET1

R13 R14 R23 R24

SITE4

PfR Site-Prefix

10.4.4.0/24

User traffic DSCP AF21to 10.4.4.0/24

BRKRST-2362 18


PfR Channels – Hub to Spoke


R10 R20

MPLS1

R11 R12 R21 R22



Channels

0 0 0 1

R13

INET1

0 0 0 3

R31SITE3

PfR Site-Prefix

10.3.3.0/24

BRKRST-2362 19


PfR Channels – Spoke to Hub


R31

R10 R20

R11 R12 R21 R22



SITE1

PfR Site-Prefix

10.10.0.0/16

SITE2

PfR Site-Prefix

10.10.0.0/16

Channels

Path MPLSPath-id 1

Path MPLSPath-id 2

Path MPLSPath-id 1 Path MPLS

Path-id 2

0 1 0 0

1 2 0 0

MPLS1

0 0 POP-ID PATH-ID

POP-ID PATH-ID 0 0

Hub to Spoke

Spoke to Hub

A PfR-label uniquely identify a path between sites across clouds (embedded in GRE encapsulation)

BRKRST-2362 20


PfR Channels – Spoke to Spoke


R10 R20

MPLS1

R11 R12 R21 R22



R13

INET1

0 0 0 0


R41R31SITE3

PfR Site-Prefix

10.3.3.0/24

BRKRST-2362 21

Performance Routing Principles


Define PfR Traffic Policies

Define your Traffic Policy

Identify Traffic Classes based on Application or DSCP

Performance thresholds (loss, delay and Jitter), Preferred Path

Centralized on a Domain Controller

CLASS MATCH ADMIN PERFORMANCE

VoiceDSCP

Application

Preferred: MPLS

Fallback: INET

Next Fallback: 4G

Delay threshold

Loss threshold

Jitter threshold

Interactive VideoDSCP

Application

Preferred: MPLS

Fallback: INET

Delay threshold

Loss threshold

Jitter threshold

Critical DataDSCP

Application

Preferred: MPLS

Fallback: INET

Delay threshold

Loss threshold

Jitter threshold

Best EffortDSCP

Application

- Delay threshold

Loss threshold

Jitter threshold

Hub MC

Principle only – Check CVD or IWAN-App for recommended policies

BRKRST-2362 23


Traffic Class – DSCP Based

Traffic with EF, AF41, AF31 and 0

DSCP Based Policies

Prefix DSCP AppID Dest SiteNext-Hop

10.3.3.0/24 EF N/A Site 3 ?

10.3.3.0/24 AF41 N/A Site 3 ?

10.3.3.0/24 AF31 N/A Site 3 ?

10.3.3.0/24 0 N/A Site 3 ?

10.4.4.0/24 EF N/A Site 4 ?

10.4.4.0/24 AF41 N/A Site 4 ?

10.4.4.0/24 AF31 N/A Site 4 ?

10.4.4.0/24 0 N/A Site 4 ?

10.5.5.0/24 EF N/A Site 5 ?

10.5.5.0/24 AF41 N/A Site 5 ?

10.5.5.0/24 AF31 N/A Site 5 ?

10.5.5.0/24 0 N/A Site 5 ?

Traffic Class

Destination Prefix

DSCP Value

Application (N/A when DSCP policies used)

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R11 R12 R21 R22

R10



DMVPNMPLS

DMVPNINET

R20

R31 R41

10.1.0.0/16 10.2.0.0/16

R51 R52

BRKRST-2362 24


Traffic Class– Application Based

Traffic with EF, AF41, AF31 and 0App1, App2, etc

Application based Policies

Prefix DSCP AppID Dest Site Next-Hop

10.3.3.0/24 EF N/A Site 3 ?

10.3.3.0/24 AF41 App1 Site 3 ?

10.3.3.0/24 AF41 App2 Site 3 ?

10.3.3.0/24 AF41 N/A Site 3 ?

10.3.3.0/24 AF31 N/A Site 3 ?

10.3.3.0/24 0 N/A Site 3 ?

10.4.4.0/24 EF N/A Site 4 ?

10.4.4.0/24 AF41 App1 Site 4 ?

10.4.4.0/24 AF31 N/A Site 4 ?

10.4.4.0/24 0 N/A Site 4 ?

10.5.5.0/24 EF N/A Site 5 ?

10.5.5.0/24 AF41 App2 Site 5 ?

10.5.5.0/24 AF31 N/A Site 5 ?

10.5.5.0/24 0 N/A Site 5 ?

Traffic Class

Destination Prefix

DSCP Value

Application (N/A when DSCP policies used)

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R11 R12 R21 R22

R10



DMVPNMPLS

DMVPNINET

R20

R31 R41

10.1.0.0/16 10.2.0.0/16

R51 R52

BRKRST-2362 25


SITE1

Performance Monitoring

MPLS

INET

Performance Monitor Egress

• collects Bandwidth• Per Traffic Class

(dest-prefix, DSCP, AppName)

2

3

2BR BR

MC/BR

MC/BR

BR

Performance Monitor Ingress• Collect Performance Metrics

• Per Channel- Per DSCP

- Per Source and Destination Site

- Per Interface

3

SITE3Single CPE

SITE2Dual CPE

Auto-Tunnel between BRs

MC

BRKRST-2362 26


Smart Probes Details

• Without actual traffic

• 20 pps for channel without traffic

• IOS-XE: BR sends 10 probes spaced 20ms apart in the first 500ms and another similar 10 probes in the next 500ms

• IOS: BR sends one packet every 50ms

• With actual traffic

• Lower frequency when real traffic is observed over the channel

• Probes sent every 1/3 of [Monitor Interval], ie every 10 sec by default

• Measured by Performance Monitor just like other data traffic

For Your Reference

BRKRST-2362 27


Performance Violation

MPLS

INET

2

3

2BR BR

MC/BR

MC/BR

BR

3

SITE3Single CPE

SITE2Dual CPE

ALERT – Threshold Crossing Alert (TCA)• From destination site

• Sent to source site

• Loss, delay, jitter, unreachable

MC

SITE1

!

BRKRST-2362 28


Policy Decision

MPLS

INET

2

3

2BR BR

MC/BR

MC/BR

BR

3

SITE3Single CPE

SITE2Dual CPE

MC Instructs BRs• Reroute Traffic to a Secondary Path across the

auto-tunnel

• PfR Dataplane Path Control

MC

SITE1

!

BRKRST-2362 29

Next Hop Selection Logic


Path SelectionBetween POPs

• No PfR control between Transit Sites (Future)

• No Channels created

• Normal Routing

R11 R12 R21 R22

R10 R20

DMVPNMPLS

DMVPNINET

R31

10.3.3.0/24



Hub MCPOP-ID 0

Transit MCPOP-ID 1

NO SUPPORT

BRKRST-2362 31


Path SelectionFrom POPs to Spokes

• Each POP is a unique site by itself and so it will only control traffic towards the spoke on the WAN’s that belong to that POP.

• PfRv3 will NOT be redirecting traffic between POP across the DCI or WAN Core. If it is required that all the links are considered from POP to spoke, then the customer will need to use a single MC.

• Only one next hop (on branch) per DMVPN network

R11 R12 R21 R22

R10 R20

DMVPNMPLS

DMVPNINET

R31

10.3.3.0/24



Hub MCPOP-ID 0

Transit MCPOP-ID 1

BRKRST-2362 32


Path SelectionFrom Spokes to POPs

• The spoke considers all the paths (multiple NH’s) towards the POPs

• The concept of "active" and "standby" next hops based on best metrics in routing is used to gather information about the preferred POP for a given prefix.

• We moved away from tagging a next hop individually as active/standby and moved towards tagging a whole DC as active/standby. Path-preference is used to choose one path over other.

• If the best metric for a given prefix is on DC1 then all the next hops on that DC for all the ISPs are tagged as active (only for that prefix).

• Best Metrics: • Advertised mask length

• BGP Weight and Local Preference

• EIGRP FD and Successor FD

MPLSLP 100000

MPLSLP 3000

MPLSLP 20000

INETLP 50

R11 R12 R21 R22

R10 R20

R31

10.3.3.0/24

R13

10.10.0.0/16DC1 DC2

INETLP 400

Note Next Hop Status - active/standby tagging happens irrespective of transit site affinity enabled/disabled

BRKRST-2362 33


Next hop status for prefix – Details

• Active next hop: A next hop is considered active if it is located at the POP site which has the next hop with the best routing metric for a given prefix

• Standby next hop: A next hop is considered standby if it is located at the POP site which advertises a route for prefix but does not have any next hop with best metric.

• Routable* next hop: A next hop is considered routable for a given prefix if it advertises one or more routes for the prefix and it was not a candidate channel for any traffic class

• Unreachable next hop: A next hop is considered unreachable for a given prefix if it is down or does not advertise any route for the prefix

• The sorting for active/standby considers all the channels/next hops on all WAN interfaces which are “Routable”.

Note: Routable is a new status visible starting from XE

3.16.1/15.5(3)M. On the border prior to XE 3.16.1/15.5(3)M

active, standby and unreachable were supported.

For Your Reference

BRKRST-2362 34


PfRv3 and Routing Best Metrics

• A next hop in a given list is considered to have a best metric based on following metrics/criteria:

• Advertised mask length ()

• BGP: Weight() , Preference length ()

• EIGRP : FD () , Successor FD ()

• Mask length takes precedence. Only if advertised mask lengths are equal, the protocol specific metrics are used.

For Your Reference

BRKRST-2362 35


Transit Site Affinity

• Transit Site Affinity (also called POP Preference) is used in the context of a Multiple Transit Site deployment with the same set of prefixes advertised from all central sites.

• A specific Transit site is preferred for a specific prefix, as long as there are available ‘in policy’ channels for this site.

• Based on routing metrics and advertised mask length in routing

• Transit Site preference is a higher priority filter and takes precedence over path-preference.

domain IWAN

vrf default

master hub

advanced

no transit-site-affinity

Transit Site Affinity is enabled by default.

To disable use:

Transit Site Affinity introduced in 15.5(3)M1 and XE 3.16.1 and is the default

BRKRST-2362 36


Path Preference

• With Path Preference configured, PfR will then first consider all the links belonging to the preferred path preference (i.e it will include the active and the standby links belonging to the preferred path) and will then use the fallback provider links.

• Without Path Preference configured PfR will give preference to the active channels and then the standby channels (active/standby will be per prefix) with respect to the performance and policy decisions• Note that the Active and Standby channels per prefix will span across the POP’s.

• Spoke will randomly (hash) choose the active channel

BRKRST-2362 37


Transit Site Affinity and Path Preference Usage

R11 R12 R21 R22

R10 R20

DC1 DC2

LP 100000 LP 3000 LP 20000 LP 400

Path MPLSPath Id 1

Path MPLSPath Id 1

Path INETPath Id 2

Path INETPath Id 2

• Next Hop Status - active/standby tagging happens irrespective of transit site affinity enabled/disabled.

POP PATHNEXT

HOPPREFIX STATUS

1 MPLS R11 10.10.0.0/16 Active

1 INET R12 10.10.0.0/16 Active

2 MPLS R21 10.10.0.0/16 Standby

2 INET R22 10.10.0.0/16 Standby

PfR Site-Prefix

10.10.0.0/16

PfR Site-Prefix

10.10.0.0/16

R31

TRANSIT SITE AFFINITY AND PATH PREFERENCE ORDER

With POP Preference (DC1) and Path Preference (MPLS) R11, R12, R21, R22

Without POP Preference (DC1) and with Path Preference (MPLS) R11, R21, R12, R22

With POP preference(DC1) and without path preference R11/R12, R12/R22

Without POP preference and without path preference (assuming all are active) R11/R12/R21/R22

BRKRST-2362 38


Load Balancing• Load balancing (LB) works on physical interface

• Looks at the local interface bandwidth utilization and selects Path/local interface

• Tu100 vs Tu200

• Non Performance TC

• Load balancing at any time (not only at creation time).

• TC will be moved to ensure bandwidth on all links is within the defined range

• Performance TC

• Load balances only at creation time

• TC will NOT be moved to ensure bandwidth on all links is within the defined range

• PfR does not account for the Performance TCs getting fatter

R11 R12 R21 R22

R31

10.3.3.0/24

DC1Site ID = 10.1.0.10

DC2Site ID = 10.2.0.20

R14

DMVPNMPLS

DMVPNINET

10.10.0.0/16

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

R20R1010.1.0.0/16

10.2.0.0/16

domain IWAN

vrf default

master hub

load-balance advanced

path-preference MPLS1 MPLS2

fallback INET1 INET2

next-fallback blackhole

Option to prevent placing

non-performance based

traffic classes on certain path

Tu100 Tu200

BRKRST-2362 39


Load Sharing Next Hops• Load Sharing (LS) works on next hops

(NHs) on the same DMVPN network

• Looks at remote next hops of same Path at the hub sites: R14/R11/R21 and R12/R22

• Load-share among the equals (iematching datacenter preference, path-preference and path)

• Statistically distribute the load among NHs on the same Path (hashing algorithm)

• Applicable only for branch-to-hub traffic

R11 R12 R21 R22

R31

10.3.3.0/24

DC1Site ID = 10.1.0.10

DC2Site ID = 10.2.0.20

R14

DMVPNMPLS

DMVPNINET

10.10.0.0/16

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

R20R1010.1.0.0/16

10.2.0.0/16

(XE 3.16.1 and IOS 15.5(3)M1)

Tu100 Tu200

BRKRST-2362 40

Deploying IWAN Intelligent Path Control


Enterprise IWAN Deployment Models

Dual MPLS

Internet

Highest SLA guarantees

– Centralized Internet Access

– Expensive

Public

MPLS

Branch

MPLS

More BW for key applications

Balanced SLA guarantees

– Moderately priced

PublicEnterprise

Branch

MPLS+

Internet

Consistent VPN Overlay Enables Security Across Transition

Best price/performance

Most flexibility

– Enterprise responsible for SLAs

Internet

Branch

Enterprise Public

Hybrid Dual Internet

Internet

BRKRST-2362 42


Service Provider IWAN (SP-IWAN)

vMS/NSO Orchestration

Hybrid WAN Hybrid WAN

ServiceProvider 1

ServiceProvider 2

Branch Office

Private

CloudPublic

Cloud

InternetHQ

MPLS internet

ServiceProvider 1

ServiceProvider 2

Branch Office

Private

Cloud

`Public

Cloud

HQ

Dedicatedinternet

Dedicatedinternet

INET INET

Internet

PE INET

Application Aware Cloud Services Optimization Pervasive Security WAN Optimization Usage Based Pricing

Hybrid WAN

ServiceProvider 1

ServiceProvider 1

Branch Office

Private

CloudPublic

Cloud

InternetHQ

MPLSDedicatedinternet

PE INET

LivingObjects EyeLo Reporting

BRKRST-2362 43


Performance Routing – Platform Support

Cisco ISR G2 family

3900-AX2900-AX1900-AX

890

Cisco ISR 4000

44004300

Cisco ASR-1000

Cisco CSR-1000

MCBR

MCBR

MCBR

MCBR*

* BR support only on Branch

BRKRST-2362 44


IWAN 2.1 – HUB-MC Scaling

ISR 443150 sites

ASR 1001-X1000 sites

ISR 4451200 sites

ASR 1002-X2000 sites

CSR1000v1 vCPU

200 sites

CSR1000v 2 vCPU

500 sites

CSR1000v 4 vCPU

2000 sites

XE 3.16.2

BRKRST-2362 45


Bootstrapping CPE – Auto Configuration (PnP)

1. CPE boots and registers with auto configuration server (IP + serial + model + capabilities)

2. CPE is authenticated and authorised

3. Assuming it has been claimed - CPE is served with appropriate configuration

• Including DMVPN

• Including BGP/EIGRP routing

• Including Per tunnel QoS

• Including Performance Routing (PfRv3)

• Including Application Visibility (Performance Monitors)

Controller

vMS/NSOAPIC-EM

INET

PnP Server

R31R12

IWAN-App

BRKRST-2362 46


Bootstrapping the router – 3 options

https connection to

download config

<config-data>

Blah

Blah

Blah

…

</config-data>

Bootstrap Config

Console

USB Ethernet

Plug and Play

Gateway

DHCP

Server

DHCP discover

Read DHCP option 43/60

OR local DNS

pnpserver.localhost = APIC-EM IP Address

DHCP option 43/60 =

IP address of APIC-EM

nnn.nnn.nnn.nnn

Internet

MPLS

3G/4G/LTE1

2

3

SDN

Controller

For Your Reference

BRKRST-2362 47


Site3 Site4 Site5

R31 R41 R51 R52

R13 R14

R10 R20

MPLS1

R11 R12 R23 R24R21 R22

MPLS2 INET1 INET2

R15 R25

4G

Branch 3 and Branch 4 do not share MPLS and INET paths

DC1 DC2

• 5 paths

• Up to 5 BR’s per DC is supported. Multi-DC up to two DC’s is supported

• Single VPN Termination per Hub BR.

• Up to three VPN cloud support on single Branch Router and up to 5 on dual router branch.

• DCI connectivity between Multi-DC. NON-DCI WAN connectivity is not supported.

• Up to 20 VRFs

IWAN 2.1.1 – Up to 5 WAN Connections

BRKRST-2362 48


Hub Sites &

Settings

BRKRST-2362 50


Manage Branch

Sites

BRKRST-2362 51


Application

Policy

BRKRST-2362 54


IWAN-App Deployed Services

• Greenfield (Brownfield Future)

• Overlay (DMVPN) with EIGRP (BGP coming soon)

• Path Control – PfRv3 with Application Based Policies

• H-QoS with Cisco Recommended Design configuration (CVD)

• With automatic application classification (NBAR2) and DSCP marking

• Per tunnel QoS

• Enterprise to Service Provider DSCP mapping

• Application Visibility

• Ok – but can you show me what has been deployed

BRKRST-2362 56


IWAN Deployment – DMVPN

• IWAN Prescriptive Design – Transport Independent Design based on DMVPN

• Branch spoke sites establish an IPsec tunnel to and register with the hub site

• Data traffic flows over the DMVPN tunnels

• WAN interface IP address used for the tunnel source address (in a Front-door VRF)

• One tunnel per user inside VRF

• Per-tunnel QOS is applied to prevent hub site oversubscription to spoke sites

R31 R41 R51 R52

R11 R12 R21 R22

R10

Site1 Site2

R20

MPLS INET

DCIWAN Core

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

10.1.0.0/16 10.2.0.0/16

BRKRST-2362 57


IWAN Routing - Which protocol should I use?

• IWAN Profiles are based upon BGP and EIGRP for scalability and optimal Intelligent Path Control

• Scalability:

• BGP (Path Vector) and EIGRP (Advanced Distance Vector) provide best scale over large hub-and-spoke topologies like DMVPN

• OSPF (Link State) maintains a lot of network state which cannot scale or be subdivided easily in large DMVPN networks

• Intelligent Path Control:

• PfR can be used with any routing protocols by relying on the routing table (RIB). • Requires all valid WAN paths be ECMP so that each valid path is in the RIB.

• For iBGP and EIGRP, PfR can look into protocol’s topology information to determine best paths and secondary paths thus, ECMP is not required.• EIGRP => uses feasible successor and sorts by metric

• iBGP => uses and sorts by Local-Pref then Weight

BRKRST-2362 58


IWAN Deployment – EIGRP• Single EIGRP process for Branch, WAN and

POP/hub sites

• Extend Hello/Hold timers for WAN

• Adjust tunnel interface “delay” to ensure WAN path preference (MPLS primary, INET secondary)

• Hubs

• Disable Split-Horizon

• Advertise Site summary, enterprise summary, default route to spokes

• Summary metrics: A summary-metric is used to reduce computational load on the DMVPN hubs.

• Ingress filter on tunnels.

• Spokes

• EIGRP Stub-Site functionality builds on stub functionality that allows a router to advertise itself as a stub to peers on specified WAN interfaces, but allows for it to exchange routes learned on LAN interface

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R31 R41

R10

Site1 Site2

R20

MPLS INET

DCIWAN Core

Delay 1000

Set TunnelDelay to

influence best path

EIGRPStub Site

Delay 2000

R11 R12 R21 R22

Delay 24000 Delay 24000

Delay 20000Delay 1000Delay 1000

Delay 20000

R51 R52Delay 20000

Delay 25000 Delay 25000

Delay 25000 Delay 25000 Delay 25000 Delay 25000

BRKRST-2362 59


IWAN Deployment – BGP• A single iBGP routing domain is used

• Appropriate Hello/Hold timers for WAN

• Hub

• DMVPN hub routers function as BGP route-reflectors for the spokes.

• No BGP peering between RR.

• BGP dynamic peer feature configured on the route-reflectors

• Site specific prefixes, Enterprise summary prefix and default route advertised to spokes

• Set local preference for all prefixes

• Redistribute BGP into local IGP with a defined metric cost to attract traffic from the central sites to the spokes across MPLS.

• Spokes

• Peer to Hub/Transit BRs in each DMVPN cloud

• Mutual redistribution OSPF/BGP

• Set a route tag to identify routes redistributed from BGP

• Preferred path is MPLS due to highest Local Preference10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R31 R41 R51 R52

R11 R12 R21 R22

R10

Site1 Site2

R20

MPLS INET

DCIWAN Core

LP 100000 LP 3000LP 20000 LP 400

OSPF

Metric: 1000 Metric: 2000

OSPF

OSPF

Metric: 1000 Metric: 2000

BRKRST-2362 60


R11 R12

R10


10.1.0.0/16

PfR Deployment – Hubdomain IWAN

vrf default

master hub

source-interface Loopback0

enterprise-prefix prefix-list ENTERPRISE_PREFIX

site-prefixes prefix-list SITE1_PREFIX

domain IWAN

vrf default

border

master 10.1.0.10


!

interface Tunnel100

description -- Primary Path --

domain IWAN path MPLS path-id 1

R10

R11

Path MPLSId 1

Path INETId 2

domain IWAN

vrf default

border

master 10.1.0.10


!

interface Tunnel200

description – Secondary Path --

domain IWAN path INET path-id 2

R12

Site Prefix: static definition of prefixes for a site

MANDATORY

POP-ID 0

Policies

Monitors

BRKRST-2362 61


R10

R11 R12

R31 R41 R51 R52

Redundant MC – Anycast IP

Backup Hub MC10.1.0.10/31R100

Hub MC10.1.0.10/32

HUB SITE

• What happens when a MC fails?

• Traffic forwarded based on routing information –ie no drop

• What happens when the Hub MC fails?

• Branch MCs keep their configuration and policies

• Continue to optimize traffic

• A backup MC can be defined on the hub.

• Using the same IP address as the primary

• Routing Protocol is used to make sure BRs and branch MC connect to the primary

• Stateless redundancy

• Backup MC will re-learn the traffic

DMVPNMPLS

DMVPNINET

BRKRST-2362 62


R21 R22


R20

10.2.0.0/16

POP-ID 1PfR Deployment – Transit Sitedomain IWAN

vrf default

master transit 1


site-prefixes prefix-list SITE2_PREFIX

hub 10.1.0.10

domain IWAN

vrf default

border

master 10.2.0.20


!

interface Tunnel100

description -- Primary Path --

domain IWAN path MPLS path-id 1

domain IWAN

vrf default

border

master 10.2.0.20


!

interface Tunnel200

description – Secondary Path --

domain IWAN path INET path-id 2

R20

R21

Path MPLSId 1

Path INETId 2

R22

Site Prefix: static definition of prefixes for a site

MANDATORY

BRKRST-2362 63


PfR Enterprise & Site Prefix

Site

PrefixesEnterprise Prefix

Legacy

Site

Prefixes Site

Prefixes

Site

Prefixes

Without enterprise-prefix: all the

traffic to Non-PfR enabled will be

learnt as internet traffic class and

therefore subjected to load

balancing.

Internet

Site prefixes for particular sites

with PFRv3 enabled

Hubs act as transit sites –site-

prefix statically defined

Non PfR

enabled

Site (*)

Branches learn Site Prefixes

Dynamically

(*) Only routing is used between

non-PfR enabled site in

Enterprise Prefix

Site

Prefixes

BRKRST-2362 64


Enterprise Prefix List

• The main use of the enterprise prefix list is to determine the enterprise boundary.

• The enterprise prefix prefix-list defines the boundary for all the internal enterprise prefixes.

• A prefix that is not from the prefix-list is considered as internet prefix and is load balanced over the DMVPN tunnels.

• The enterprise prefix prefix-list is defined only on the Hub MC under the master controller configuration with the command enterprise-prefix prefix-list prefix-list-name.

domain IWAN

vrf default

master hub


enterprise-prefix prefix-list ENTERPRISE_PREFIX

!

ip prefix-list ENTERPRISE_PREFIX seq 10 permit 10.0.0.0/8

BRKRST-2362 65


Site Prefix List

• The site-prefix prefix-list defines static site-prefix for the local site and disables automatic site-prefix learning on the border router.

• The static-site prefix list is only required for Hub and Transit MCs.

• A site-prefix prefix-list is optional on Branch MCs.

• The site prefix is defined under the master controller configuration with the command site-prefixes prefix-list prefix-list-name

domain IWAN

vrf default

master hub


site-prefixes prefix-list SITE_PREFIX

!

ip prefix-list SITE_PREFIX seq 10 permit 10.1.0.0/16

ip prefix-list SITE_PREFIX seq 20 permit 10.2.0.0/16

!

BRKRST-2362 66


PfR Deployment – Single CPE Branch

• Site Prefix: static definition of prefixes for a site is optional but recommended

domain IWAN

vrf default

master branch


hub 10.8.3.3

border

master local


R31

R41

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R11 R12 R21 R22

R10



DMVPNMPLS

DMVPNINET

R20

R31 R41

10.1.0.0/16 10.2.0.0/16

R51 R52

BRKRST-2362 67


PfR Deployment – Dual CPE Branch

domain IWAN

vrf default

border

master 10.2.12.12


R52

domain IWAN

vrf default

master branch


hub 10.8.3.3

border

master local


R51

10.3.3.0/24 10.4.4.0/24 10.5.5.0/24

R11 R12 R21 R22

R10



DMVPNMPLS

DMVPNINET

R20

R31 R41

10.1.0.0/16 10.2.0.0/16

R51 R52

• Site Prefix: static definition of prefixes for a site is optional but recommended

• BR must be directly connected –physical interface, vlan or GRE tunnel

BRKRST-2362 68


PfR Policies – DSCP or App Baseddomain IWAN

vrf default

master hub

load-balance

class MEDIA sequence 10

match application <APP-NAME1> policy real-time-video

match application <APP-NAME2> policy custom

priority 1 one-way-delay threshold 200

priority 2 loss threshold 1

path-preference MPLS fallback INET

class VOICE sequence 20

match dscp <DSCP-VALUE> policy voice

path-preference MPLS fallback INET

class CRITICAL sequence 30

match dscp af31 policy low-latency-data

R10

• Pre-defined thresholds

• Custom thresholds

• When load balancing is enabled, PfRv3 adds a

“default class for match all DSCP (lowest priority

compared to all the other classes)” and PfRv3

controls this traffic.

• When load balancing is disabled, PfRv3 deletes this

“default class” and as a part of that frees up the TCs

that was learnt as a part of LB – they follow the

routing table

Check the CVD for the recommended config

BRKRST-2362 69


PfR Policies – Built-in Policy Templates

Pre-defined

Template

Threshold Definition

Voice priority 1 one-way-delay threshold 150 threshold 150 (msec)

priority 2 packet-loss-rate threshold 1 (%)

priority 2 byte-loss-rate threshold 1 (%)

priority 3 jitter 30 (msec)

Real-time-video priority 1 packet-loss-rate threshold 1 (%)


priority 2 one-way-delay threshold 150 (msec)

priority 3 jitter 20 (msec)

Low-latency-data priority 1 one-way-delay threshold 100 (msec)



Pre-

defined

Template

Threshold Definition

Bulk-data priority 1 one-way-delay threshold 300 (msec)



Best-effort priority 1 one-way-delay threshold 500 (msec)



scavenger priority 1 one-way-delay threshold 500 (msec)



For Your Reference

BRKRST-2362 70


Traffic Classes Timers

• Traffic Classes time out after 5 min with no activity

• Advanced options – with 3.16 15.5(3)S / 15.5(3)M

• traffic-class-ageout-timer 24 hours

• default 5 minutes

For Your Reference

BRKRST-2362 71


Unreachable Timer

10.3.3.0/24

Hub MC

R31

R11 R12 R21 R22

R10

Path MPLSId 1

Path MPLSId 1

Transit MCR20

DMVPNMPLS

DMVPNINET

POP ID 0 POP ID 1



Path INETId 2

Path INETId 2

• Channel Unreachable

• PfRv3 considers a channel reachable as long as the site receives a PACKET on that channel

• A channel is declared unreachable in both direction if

• There is NO traffic on the Channel, probes are the only way of detecting unreachability. So if no probe is received within 1 sec, PfR detects unreachability.

• When there IS traffic on the channel, if PfR does not see any packet for more than a second on a channel PfR detects unreachability.

Default: 1 Sec

Recommended: 4 sec

Advanced options – with 3.16 15.5(3)S / 15.5(3)Mchannel-unreachable-timer 4

BRKRST-2362 72


Failover TimeHub MC

R11 R12 R21 R22

R10

Path MPLSId 1

Path MPLSId 1

Transit MCR20

DMVPNMPLS

DMVPNINET

POP ID 0 POP ID 1

Path INETId 2

Path INETId 2

• Ingress Performance Violation detected

• Delay, loss or jitter thresholds

• Based on Monitor-interval (30 sec default)

• Quick Monitor for fast failover

domain IWAN

vrf default

master hub

monitor-interval <sec> dscp <value>

R31

10.3.3.0/24



BRKRST-2362 73


Zero SLA Support

• Zero-sla added on the WAN interface path configuration

• PfR will only probe the default channel (DSCP 0).

• It will mute all other smart-probes besides the default channel

DMVPNMPLS

DMVPNINET

R31

R12 R21 R22

R10 R20

R11

10.3.3.0/24

SITE1Site ID = 10.1.0.10


DMVPNLTE

R13 R23

BRKRST-2362 74


Path of Last Resort

DMVPNMPLS

DMVPNINET

R31

R12 R21 R22

R10 R20

R11

10.3.3.0/24



DMVPNLTE

R13 R23

• Path of last resort (PLR) option for metered links

• PLR Channels muted when in standby mode

• Once it is active, smart probes will only be sent on dscp 0 (zero sla) to conserve bandwidth

• Smart probe frequency will be reduced to 1 packet every 10 secsfrom 20 packets per secs.

• Unreachable detection will be extended to 60 secs

R13 – R23

interface Tunnel300

description – LTE Path --

domain IWAN path LTE path-id 3 path-last-resort

BRKRST-2362 75


Load Balancing – Non Performance TC

• PfRv3 load balances non-performance based traffic-classes on ALL available links

• Load balancing at any time (not only at creation time).

• TC will be moved to ensure bandwidth on all links is within the defined range

• Advanced Mode

- Option to prevent placing non-performance based traffic classes on certain path

domain IWAN

vrf default

master hub

load-balance advanced

path-preference MPLS1 MPLS2 fallback INET1 INET2 next-fallback blackhole

BRKRST-2362 76


Dual Datacenters

R11 R12 R21 R22

R31

10.3.3.0/24

DC1Site ID = 10.1.0.10

DC2Site ID = 10.2.0.20

R13

DMVPNMPLS

DMVPNINET

10.10.0.0/16

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

R10

10.1.0.0/16

10.1.0.0/16

10.0.0.0/8

BGP

10.2.0.0/16

10.0.0.0/8

BGP

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

10.2.0.0/16R20

BRKRST-2362 77


Dual Datacenters #1 – Separate PrefixPrefix 10.1.0.0/16

R31

10.3.3.0/24

10.1.0.0/16

10.0.0.0/8

BGP

10.2.0.0/16

10.0.0.0/8

BGP

R11 R12 R21 R22

R10 R20

R14

DC1Site ID = 10.1.0.10

DC2Site ID = 10.2.0.20

SITE1

PfR Site-Prefix

10.1.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

LP 100000 LP 20000 LP 400 LP 3000 LP 50

PREFIXPATH

PREFERENCE

NEXT-HOPS

ORDER

10.1.0.0/16YES R11/R14, R12

NO R11/R14/R12

• With load sharing enabled, the TCs might be distributed across “equal” channels – same tunnel, same active/standby status and same policy status (in policy/out of policy).

• Hence, load sharing can randomly choose between channel #1 and channel #2

Path-preference MPLS fallback INET

BRKRST-2362 78


Dual Datacenters #2 – Shared PrefixPOP Preference with Different Metrics

R31 PfR View with Transit Site Affinity enabled

R31

10.3.3.0/24

BGP

R11 R12 R21 R22

R10 R20

R14

10.10.0.0/16DC1

Site ID = 10.1.0.10DC2

Site ID = 10.2.0.20

10.1.0.0/16

10.10.0.0/16

10.0.0.0/8

10.2.0.0/16

10.10.0.0/16

10.0.0.0/8

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

10.1.0.0/1610.2.0.0/16

LP 100000 LP 20000 LP 3000LP 400 LP 50

• Prefix 10.10.0.0/16

• Advertised by Site1 and Site2 MCs

• Channels per DSCP for this destination prefix

• R14 best metric => Site1 preferred, all BRs active on Site1

• Transit Site Affinity enabled by default

PREFIXPATH

PREFERENCENEXT-HOPS ORDER

10.1.0.0/16YES R14/R11, R12, R21, R22

NO R14/R11/R12, R21/R22


BRKRST-2362 79


Dual Datacenters #3 – Shared PrefixNo POP Preference with Different Metrics

• Transit Site Affinity disabled

• Different Local Preference values used per BRs on Site1 and Site2

R31

10.3.3.0/24

BGP

R11 R12 R21 R22

R10 R20

R14

10.10.0.0/16DC1

Site ID = 10.1.0.10DC2

Site ID = 10.2.0.20

10.1.0.0/16

10.10.0.0/16

10.0.0.0/8

10.2.0.0/16

10.10.0.0/16

10.0.0.0/8

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

10.1.0.0/1610.2.0.0/16

LP 100000 LP 20000 LP 3000LP 400 LP 50

PREFIXPATH


10.1.0.0/16YES R14/R11, R21, R12, R22

NO R14/R11/R12, R21/R22

R31 PfR View – No Transit Site Affinity


BRKRST-2362 80


Dual Datacenters #4 – Shared PrefixNo POP Preference with Same Metrics

domain IWAN

vrf default

master hub

advanced

no transit-site-affinity

• To disable and come back to previous default:

R31

10.3.3.0/24

BGP

R11 R12 R21 R22

R10 R20

R14

10.10.0.0/16DC1

Site ID = 10.1.0.10DC2

Site ID = 10.2.0.20

10.1.0.0/16

10.10.0.0/16

10.0.0.0/8

10.2.0.0/16

10.10.0.0/16

10.0.0.0/8

SITE1

PfR Site-Prefix

10.1.0.0/16

10.10.0.0/16

SITE2

PfR Site-Prefix

10.2.0.0/16

10.10.0.0/16

10.1.0.0/1610.2.0.0/16

LP 100000 LP 100000 LP 100000LP 100000 LP 100000

PREFIXPATH


10.1.0.0/16YES R14/R11/R21, R12/R22

NO R14/R11/R12/R21/R22

R31 PfR View


BRKRST-2362 81

Key Takeaways


Performance Routing Phases – Summary PfR version 3

IOS 15.4(3)M

IOS-XE 3.13

PfR version 3

IOS 15.5(1)T

IOS-XE 3.14

PfR version 3

IOS 15.5(2)T

IOS-XE 3.15

PfR version 3

IOS 15.5(3)M

IOS-XE 3.16

PfR version 3

IOS 15.5(3)M1

IOS-XE 3.16.1

• PfR Domain

• One touch provisioning

• Auto Discovery of sites

• NBAR2 support

• Passive Monitoring

(performance monitor)

• Smart Probing

• VRF Awareness

• IPv4/IPv6 (Future)

• <10 lines of configuration

and centralized

• Zero SLA

• WCCP Support

• Transit Sites

• Multiple Next Hop per

DMVPN

• Multiple POPs

• Syslog (TCA)

• Show last 5 TCA

• Path of Last Resort

• EIGRP IWAN

Simplification (Stub

site)

• POP Affinity

• Blackout ~ sub second

• Brownout ~ 2 sec

• Scale 2000 sites

BRKRST-2362 83


Key Takeaways

• IWAN Intelligent Path Control pillar is based upon Performance Routing (PfR)

• Maximizes WAN bandwidth utilization

• Protects applications from performance degradation

• Enables the Internet as a viable WAN transport

• Provides multisite coordination to simplify network wide provisioning.

• Application-based policy driven framework and is tightly integrated with existing AVC components.

• Smart and Scalable multi-sites solution to enforce application SLAs while optimizing network resources utilization.

• PfRv3 is the 3rd generation Multi-Site aware Bandwidth and Path Control/Optimization solution for WAN/Cloud based applications.

• Available this summer on ASR1k, 4451-X, ISR-G2

BRKRST-2362 84


More Information• Cisco.com IWAN , AVC and PfRv3 Page:

• http://www.cisco.com/go/iwan

• http://www.cisco.com/go/avcportal

• http://www.cisco.com/go/pfr

• DocWiki

• http://docwiki.cisco.com/wiki/PfRv3:Home

• dCloud

• http://dcloud.cisco.com

• dCloud IWAN 4D Lab: https://dcloud-cms.cisco.com/demo/16360

• IWAN 2.1 CVD

• IWAN Technical Design Guide (IWAN 2.1, Feb 2016)http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANDesignGuide-FEB16.pdf

• Intelligent WAN Config Files (IWAN 2.1, Feb 2016:http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANConfigurationFilesGuide-FEB16.pdf

• IWAN Security for Remote Site DIA and Guest Wirelesshttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-DIADesignGuide-Mar15.pdf

• IWAN Application Optimization using Cisco WAAS and Akamai Connecthttp://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-WAASDesignGuide-Mar15.pdf

BRKRST-2362 85

http://www.cisco.com/go/iwan

http://www.cisco.com/go/avcportal

http://www.cisco.com/go/pfr

http://docwiki.cisco.com/wiki/PfRv3:Home

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANDesignGuide-FEB16.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANConfigurationFilesGuide-FEB16.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-DIADesignGuide-Mar15.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Mar2015/CVD-IWAN-WAASDesignGuide-Mar15.pdf


IWAN Book

• Pre-order available

• https://t.co/CsSeG1GkFK

• VIRL lab available

Coming

Soon

BRKRST-2362 86

https://t.co/CsSeG1GkFK


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKRST-2362 87

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKRST-2362 88

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you

iwan implementing performanced2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkrst-2362.pdf · •iwan...

Documents