cisco asr1000 and microsoft azure expressroute design …on the azure side. to connect to microsoft...
TRANSCRIPT
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 35
Cisco ASR1000 and Microsoft Azure ExpressRoute
Design and Deployment Guide
Extend your enterprise network into Azure with
Cisco ASR®1000
Written by Jason Yang and Kevin Echols II
July 2018
Design and Deployment Guide
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 35
Contents
Executive summary ................................................................................................................................................. 3 Cisco Multicloud Portfolio: Overview ..................................................................................................................... 3 Cloud Connect: Overview ..................................................................................................................................... 4 Cloud Connect: Use cases.................................................................................................................................... 4 Cloud Connect: Benefits ....................................................................................................................................... 4
Introduction .............................................................................................................................................................. 5 Target Audience .................................................................................................................................................... 5 Purpose of This Document .................................................................................................................................... 5 Solution Overview ................................................................................................................................................. 5
Product Overview .................................................................................................................................................... 6
Preparation ............................................................................................................................................................... 7 Getting Started ...................................................................................................................................................... 8
Configuration: ExpressRoute Peering on Azure ................................................................................................... 9
Configuration: Cisco ASR1000 ............................................................................................................................... 9 Two Router Deployment vs. One Router Deployment .......................................................................................... 9 Interface Configurations ...................................................................................................................................... 10
802.1Q-in-Q VLAN ID Sample Interface Configuration .................................................................................. 10 802.1Q VLAN ID Sample Interface Configuration .......................................................................................... 11
BGP Configurations ............................................................................................................................................ 12 Setup eBGP Sessions .................................................................................................................................... 12 Advertise Prefixes Over the BGP Session to Azure ....................................................................................... 13 Filter Prefixes Received from Azure (Optional) .............................................................................................. 13 High Availability and Optimize Routing Configuration .................................................................................... 14 AS Path Prepending to Influence Routing ...................................................................................................... 15 Avoid Asymmetric Routing ............................................................................................................................. 16
NAT Configuration .............................................................................................................................................. 17 NAT Common Best Practices ......................................................................................................................... 18
Route Redistribution into EIGRP ......................................................................................................................... 18
Value-Added Feature Configurations ................................................................................................................... 19 Configure Flexible Netflow .................................................................................................................................. 19 Configure Quality of Service ............................................................................................................................... 20
Advanced Services Configurations ...................................................................................................................... 21 Configure Application Visibility and Control (AVC) .............................................................................................. 21 Configure IPsec VPN .......................................................................................................................................... 22
Test Connectivity ................................................................................................................................................... 24 Verify the BGP Neighbors ................................................................................................................................... 24 Verify ExpressRoute Connectivity ....................................................................................................................... 30 Verify NAT Translation Entries and Pool ............................................................................................................. 32 Verify Netflow Entries .......................................................................................................................................... 33
ASR1000 Proactive System Monitoring ............................................................................................................... 34
References ............................................................................................................................................................. 35
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 35
Executive summary
This guide focuses on how to extend your on premises network into the Microsoft Azure Virtual Private Cloud
(VPC) with ExpressRoute using the Cisco ASR1000 Series Routers with VPN connectivity back to a private data
center at the corporate site. The design uses an ExpressRoute managed VPN connection through a virtual private
gateway (VGW) attached to the VPC. Advanced features such as Application Visibility & Control (AVC)/Next-
Generation Network-Based Application Recognition (NBAR2), and Flexible NetFlow data export are also discussed
for traffic and application-level visibility at the ASR 1000 Series routers within the private data center.
Cisco Multicloud Portfolio: Overview
In a multicloud world, growing complexity is driving a cloud gap between what your customers require and what
your people, processes, and tools can support. With the Cisco Multicloud Portfolio, we make it simple: simple to
connect, simple to protect, and simple to consume.
The Cisco Multicloud Portfolio is a set of essential products, software, and services supported with simplified
ordering and design deployment guides to help you when it comes to multicloud adoption. The Cisco Multicloud
Portfolio consists of four component portfolios (Figure 1):
● Cloud Advisory: Helps you design, plan, accelerate, and remove risk from your multicloud migration.
● Cloud Connect: Securely extends your private networks into public clouds and helps ensure the
appropriate application experience.
● Cloud Protect: Protects your multicloud identities, direct-to-cloud connectivity, data, and applications,
including Software as a Service (SaaS).
● Cloud Consume: Helps you deploy, monitor, and optimize applications in multicloud environments.
Figure 1. Multicloud Portfolio: Cloud Advisory, Cloud Connect, Cloud Protect, and Cloud Consume
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 35
Cloud Connect: Overview
Cloud Connect consists of essential products that help securely extend your private networks – including data
center, branches, and campuses – to public clouds and to help ensure that the application experience is optimal:
● Cisco Cloud Services Router (CSR) 1000V Series
● Viptela® vEdge with Cisco Umbrella™
For detailed use cases, see the section about Cloud Connect on the portfolio’s solution page at
https://www.cisco.com/go/multicloud.
Cloud Connect: Use cases
Cloud Connect delivers value in the following use cases:
● Securely extending a private network to single or multiple public cloud environments. Includes multiple
clouds (for example, multiple AWS and Azure), multiple regions in a cloud, or multiple VPCs in a cloud;
VPN; multicloud and multi-VPC connectivity; scaling; and performance optimization-transit VPC. Also
supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a
branch has a Cisco 4000 Series Integrated Services Router [ISR] and wants to connect the clouds or a
branch has vEdge and requires a software-defined WAN [SD-WAN] extension to the cloud).
● Optimizing data center and branch connectivity performance to cloud Infrastructure as a Service (IaaS) and
SaaS. Includes best path to destination (SD-WAN), cloud segmentation, monitoring to assure best
performance, visibility into traffic going to applications, and traffic shaping/Quality of Service (QoS). Also
supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a
branch has a 4000 Series ISR and wants to connect the clouds or a branch has vEdge and requires an SD-
WAN extension to the cloud).
● Securing access to the Internet and SaaS from the branch. Includes connecting and protecting branch office
users directly to the multicloud environment using Direct Internet Access (DIA), SD-WAN (vEdge), and
secure Internet gateways (Cisco Umbrella).
Cloud Connect: Benefits
Cloud Connect benefits include the ability to:
● Extend a private network to a multicloud environment while leveraging existing investments
● Apply consistent security policies across a private and public cloud footprint
● Enhance and secure the app experience on a cloud network by enabling visibility and path selection and
optimization
● Centralize management in a manner that is intuitive, fast, and easy to design, provision, and apply policies
across the entire network
● Achieve faster and more simple adoption of cloud
● Improve TCO
● Access a richer networking security feature set and higher performance
● Improve ease of use through consistency of management tools for both on-premises and cloud
● Simplify implementation through increased visibility into the public cloud network
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 35
Introduction
The majority of enterprises have started deploying business-critical applications that span on premise equipment
and cloud infrastructure in what is known as a hybrid cloud deployment. These enterprises seek to benefit from
reduced Total Cost of Ownership (TCO), the ability to scale applications to meet growing demands, and an always
on guarantee via distributed workloads across multiple availability zones and geographic regions.
Establishing a reliable connection from on premise to the cloud has proven difficult for many of these enterprises
as the Internet does not guarantee the metrics required for crucial business applications. Cisco and Microsoft have
partnered to make the transition to a hybrid cloud deployment easier for our mutual customers by creating a
jointly-validated designs between Microsoft Azure ExpressRoute and Cisco CSR100v and ASR1000.
Microsoft’s ExpressRoute lets you extend your on premise networks into Microsoft Azure over a private connection
facilitated by a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud
services, such as Microsoft Azure, Office 365, and Dynamics 365.
Target Audience
The intended audience for this document includes sales engineers, field consultants, professional services staff, IT
managers, partner engineering staff, and customers deploying the Microsoft Azure ExpressRoute with Cisco
ASR1000 routers. External references are provided wherever applicable, but readers are expected to be familiar
with the technology, infrastructure, and enterprise security policies of the customer installation.
Purpose of This Document
Cisco-Microsoft Joint Validated Designs provide guidelines for creating an end-to-end solution that enable you to
make informed decisions with the goal of successfully creating a hybrid cloud deployment.
This document describes the steps required to extend your on premises network into the Microsoft Azure with
ExpressRoute using the Cisco ASR1000 Series Routers on premise in your data center and the Cisco CSR 1000v
on the Azure side. To connect to Microsoft Azure services using ExpressRoute, Microsoft provides best practices
for network security, optimize routing, asymmetric routing, and NAT. This guide will focus on how to implement
these best practices with ASR1000 configurations, recommend advanced features and services on the ASR1000.
Please note that this guide is not meant to be a comprehensive overview of the ASR1000 platform and routing
technologies, see References section for platform and feature configuration guides.
Cisco validation provides further confirmation of solution compatibility, connectivity, and correct operation for the on
premise deployment. Although readers of this document are expected to have sufficient knowledge to install and
configure the products used, the Cisco-Microsoft Design and Deployment Guide provides configuration details that
are important to the deployment of this solution.
Solution Overview
ExpressRoute supports layer 3 connectivity between your on premise network and Microsoft Azure through a
connectivity provider in 3 connectivity models: CloudExchange Co-location, Point-to-point Ethernet Connection,
and IP VPN Connection. ExpressRoute connections do not go over the public Internet, which allows ExpressRoute
connections to offer more reliability, faster speeds, lower latencies, and higher security than typical connections
over the Internet. As shown in Figure 1, ExpressRoute circuits have multiple routing domains associated with them:
Azure private peering, and Microsoft peering. Each of the routing domains are configured in separate Virtual
Routing and Forwarding (VRF) domains on a pair of ASR1000 routers for high availability. In Figure 1, these
routers are shown located in the partner edge block.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 35
Figure 2. Common ExpressRoute Deployment
ExpressRoute capabilities and features are identical across all of the connectivity models. The ASR1000 physical
connectivity configuration to each of the service providers may vary, but the configuration to ExpressRoute will be
identical.
Product Overview
Cisco ASR1000 Series Aggregation Services Routers aggregate multiple WAN connections and network services,
including encryption and traffic management, and forward them across WAN connections at line speeds from 2.5 to
200Gbps. ASR1000 Series routers offer elastic service delivery; programmability and automation; up to five-nines
availability; comprehensive and flexible QoS; and advanced services, such as IPsec VPN and Application Visibility
and Control (AVC) for enterprise networks.
The Cisco ASR1000 Series platforms vary in I/O connectivity speed, density, system performance, and
redundancy options. All models use the Cisco Quantum Flow Processor and support the same feature set
available on the Cisco IOS XE Operating System. All this commonality simplifies management and operations.
ExpressRoute circuits are purchased based on a number of bandwidth options. Table 1 outlines ASR1000 platform
recommendations for each of the ExpressRoute bandwidth options.
Table 1. ExpressRoute Circuit Bandwidth to ASR1000 Platform Recommendations
ExpressRoute Circuit Bandwidths ASR1000 Platform Interface Type
50 Mbps ASR 1001-X GigabitEthernet
100 Mbps ASR 1001-X GigabitEthernet
200 Mbps ASR 1001-X GigabitEthernet
500 Mbps ASR 1001-X GigabitEthernet
1 Gbps ASR 1001-X or
ASR1001-HX
GigabitEthernet or
TenGigabitEthernet
2 Gbps ASR 1001-HX TenGigabitEthernet
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 35
ExpressRoute Circuit Bandwidths ASR1000 Platform Interface Type
5 Gbps ASR 1001-HX TenGigabitEthernet
10 Gbps ASR 1001-HX TenGigabitEthernet
The ASR1001X, pictured in Figure 2, is a 1 RU form factor, supports redundant power supplies, and the Embedded
Services Processor (ESP) has default throughput of 2.5Gbps that is upgradable to 5-, 10-, or 20Gbps via software
activation. The platform consumes 250W at max with front-to-back airflow. Onboard, the ASR1001X has 6 Gigabit
Ethernet SFP ports, 2 TenGigabit Ethernet SFP+ ports, and has a single half-height Shared Port Adapter (SPA)
that can be configured with a range of interfaces from a 2-port Gigabit Ethernet SPA to a T1/E1 NIM. See the
ASR1001X Datasheet for more details on the platform, and the ASR1001X Hardware Installation Guide from a
complete list of supported hardware.
Figure 3. ASR1001X
The ASR1001HX, pictured in Figure 3, is a 1 RU form factor, supports redundant power supplies, and the
Embedded Services Processor (ESP) has throughput up to 60Gbps. The platform consumes 360W at max with
front-to-back airflow. Onboard, the ASR1001HX has 8 Gigabit Ethernet SFP ports and 8 TenGigabit Ethernet SFP+
ports, where 4 of the TenGigabit Ethernet ports (Te4-7) are compatible with SFPs. See the ASR1001HX Datasheet
for more details on the platform, and the ASR1001HX Hardware Installation Guide from a complete list of
supported hardware.
Figure 4. ASR1001HX
Preparation
The configuration guide will include numerous value substitutions provided for the purpose of example only. Any
references to IP addresses, device IDs, shared secrets or keys account information or project names should be
replaced with the appropriate values for your environment when following this guide. Values unique to your
environment will be highlighted in bold.
This guide is not meant to be a comprehensive setup for entire device configuration for all network connectivity, for
example, the same device may also have connectivity to the enterprise data center, campus, or branches, the
configuration of which is outside the scope of this guide. This configuration guide will focus on the connectivity to
the ExpressRoute. List 1 provides a high-level overview of the configuration process that will be covered.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 35
List 1: High-Level Overview of ASR1000 Configuration Process
1. Interface Configurations
a. 802.1Q-in-Q VLAN ID Sample Interface Configuration
b. 802.1Q VLAN ID Sample Interface Configuration
2. BGP Configurations
a. Setup eBGP Sessions
b. Advertise Prefixes Over the BGP Session to Azure
c. Filter Prefixes Received from Azure (Optional)
d. High Availability and Optimize Routing Configuration
e. AS Path Prepending to Influence Routing
f. Avoid Asymmetric Routing
g. NAT Configuration
h. NAT Common Best Practices
3. Route Redistribution into EIGRP
4. Advanced Feature Configurations
a. Flexible Netflow Configuration
b. Quality of Service Configuration
5. Advanced Services Configurations
a. Application Visibility and Control (AVC) Configuration
b. IPsec VPN Configuration
Getting Started
It is assumed that you met all the requirements in ExpressRoute prerequisites & checklist, the ExpressRoute
circuits have been created, and the ExpressRoute circuit provisioned by the service provider.
The first step in configuring your Cisco ASR1000 for use with the ExpressRoute connectivity is to ensure that the
following prerequisite conditions have been met:
The essential feature set (BGP, NAT, VRF-Lite, IPv4/IPv6 dual-stack) required for setting up the ExpressRoute
connectivity and the advanced features are supported by the ASR1000 universal image, that is, no additional
license is required.
The advanced services require AES license in addition to base licenses:
1. NBAR/AVC requires AVC feature license
2. The IPsec application requires:
a. Advanced Enterprise Services(SLASR1-AES) or Advanced IP Services Technology Package License
(SLASR1-AIS)
b. IPsec RTU license (FLASR1-IPSEC-RTU)
c. Encryption HW module (ASR1001HX-IPSECW) and Tiered Crypto throughput license which applies to
ASR1001-HX chassis
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 35
Refer to the ASR1000 Routers Ordering Guide for more details on ASR1000 Series Router license information.
The recommended software image is 16.6.2 and onward. Suggested images are recommended on the on
cisco.com download software page, where the suggested image is labeled by icon.
Configuration: ExpressRoute Peering on Azure
Follow the ExpressRoute peering steps in Azure portal.
Configuration: Cisco ASR1000
Two Router Deployment vs. One Router Deployment
We recommend the deployment of two ASR1000s in a redundant pair to connect to the ExpressRoute service.
Each router will need two QinQ subinterfaces on the physical interface. At the Microsoft Edge (see Figure 1) an
ExpressRoute service is terminated on a pair of Microsoft ExpressRoute Edge (MSEE) routers. The MSEE routers
hand off to a pair of Connectivity Provider routers, and then down to the customer’s ASR1000 routers. Microsoft
will always have two BGP sessions for each of the peering types.
As an example, assume Connectivity Provider defines an outer dot1Q tag of 10 for ER circuit, and the customer
requests an inner tag of 310 for the Microsoft peering, and 3101 for the private peering. Table 2 outlines the
example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in the customer edge
dual router design.
Table 2. Router, Interface, Subinterfaces, VRFs and Peering for Customer Edge Dual Router Design
Routers R1 R2
Interfaces TE0/1/0 TE0/1/1 TE0/1/0 TE0/1/1
Interface description
Connection to ER Primary Connection to customer corp network
Connection to ER Secondary
Connection to customer corp network
Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101
Subinterface description
Primary Microsoft Peering
Primary Private Peering
DMZ VLAN Corp VLAN Secondary Microsoft Peering
Secondary Private Peering
DMZ VLAN Corp VLAN
Encapsulation dot1Q 10 second-dot1q 310
or
dot1Q 310
dot1Q 10 second-dot1q 3101
or
dot1Q 3101
dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310
or
dot1Q 310
dot1Q 10 second-dot1q 3101
or
dot1Q 3101
dot1Q 10 dot1Q 101
VRFs* C10 C101 C10 C101 C10 C101 C10 C101
IP Addresses 216.221.237.33/30
172.16.0.1/30 192.168.0.1/30
192.168.0.5/30
216.221.237.37/30
172.16.0.5/30 192.168.0.1/30
192.168.0.5/30
Note: It is best practice to separate private peering and Microsoft peering with two separate VRFs. The
private peering is considered trusted, whereas the Microsoft peering is a public network. The customer can
send each VRFs/VLANs to the appropriate security zone before entering/exiting their corporate VLANs.
Unless otherwise stated, this configuration guide provides configuration example on Router R1. Router R2 should
have the same configuration as R1, with the exception of IP addresses/subnets. The subinterface, IP address, and
VRF will use the example provided in Table 2.
Optionally, if the customer chooses to deploy one router for connection to ER circuit, Table 3 outlines an
example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in single customer
edge router design.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 35
Table 3. Interface, Subinterfaces, VRFs and Peering for Customer Edge Single Router Design
Interfaces TE0/1/0 TE0/1/1 TE0/1/2
Interface description Connection to ER Primary Connection to customer corp network
Connection to ER Secondary
Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101
Subinterface description Primary Microsoft Peering
Primary Private Peering
DMZ VLAN Corp VLAN Secondary Microsoft Peering
Secondary Private Peering
Encapsulation dot1Q 10 second-dot1q 310
or
dot1Q 310
dot1Q 10 second-dot1q 3101
or
dot1Q 3101
dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310
or
dot1Q 310
dot1Q 10 second-dot1q 3101
or
dot1Q 3101
VRFs C10 C101 C10 C101 C10 C101
IP Addresses 216.221.237.33/30
172.16.0.1/30 192.168.0.1/30 192.168.0.5/30 216.221.237.37/30
172.16.0.5/30
Interface Configurations
This section provides the interface configuration of Cisco ASR1000 to connect to ER. At least one internal facing
interface is required to connect to your own network, and one external facing interface is required to connect to
ExpressRoute.
You will require a subinterface per peering in every router you connect to ER. A subinterface can be identified with
an 802.1Q-in-Q VLAN ID or 802.1Q VLAN ID based on the connectivity providers’ requirement and an IP address.
Follow ER IP address requirements for the BGP peering.
802.1Q-in-Q VLAN ID Sample Interface Configuration
ip vrf C10
rd 65021:10
!
ip vrf C101
rd 65021:101
!
interface TenGigabitEthernet0/1/0
description connection to ER Primary
no ip address
dot1q tunneling ethertype 0x9100
! The default ethertype is 0x8100, can be changed to 0x88A8|0x9100|0x9200 to meet
the connectivity provider’s requirement
!
interface TenGigabitEthernet0/1/0.310
description Customer 10 Primary Microsoft peering to Azure
encapsulation dot1Q 10 second-dot1q 310
ip vrf forwarding C10
ip address 216.221.237.33255.255.255.252
!
interface TenGigabitEthernet0/1/0.3101
description Customer 10 Primary private peering to Azure
encapsulation dot1Q 10 second-dot1q 3101
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 35
ip vrf forwarding C101
ip address 172.16.0.1 255.255.255.252
!
interface TenGigabitEthernet0/1/1
description Customer 10 Corporate facing interface
no ip address
!
interface TenGigabitEthernet0/1/1.10
description Customer 10 DMZ VLAN
encapsulation dot1Q 10
ip vrf forwarding C10
ip address 192.168.0.1 255.255.255.252
!
interface TenGigabitEthernet0/1/1.101
description Customer 10 Corp VLAN
encapsulation dot1Q 101
ip vrf forwarding C101
ip address 192.168.0.5 255.255.255.252
802.1Q VLAN ID Sample Interface Configuration
ip vrf C10
rd 65021:10
!
ip vrf C101
rd 65021:101
!
interface TenGigabitEthernet0/1/0
description connection to ER
no ip address
!
interface TenGigabitEthernet0/1/0.310
description Customer 10 Primary Microsoft peering to Azure
encapsulation dot1Q 310
ip vrf forwarding C10
ip address 216.221.237.33 255.255.255.252
!
interface TenGigabitEthernet0/1/0.3101
description Customer 10 Primary private peering to Azure
encapsulation dot1Q 3101
ip vrf forwarding C101
ip address 172.16.0.1 255.255.255.252
!
interface TenGigabitEthernet0/1/1
description Customer 10 Corporate facing interface
no ip address
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 35
!
interface TenGigabitEthernet0/1/1.10
description Customer 10 DMZ VLAN
encapsulation dot1Q 10
ip vrf forwarding C10
ip address 192.168.0.1 255.255.255.252
!
interface TenGigabitEthernet0/1/1.101
description Customer 10 Corp VLAN
encapsulation dot1Q 101
ip vrf forwarding C101
ip address 192.168.0.5 255.255.255.252
Note: The MTU for the ExpressRoute interface is 1500 Bytes, which is the default MTU on ASR1000
Ethernet interface.
BGP Configurations
Setup eBGP Sessions
You must setup a BGP session with Microsoft for every peering. The sample below enables you to setup a BGP
session with Microsoft. If the IPv4 address you used for your subinterface was a.b.c.d, the IP address of the BGP
neighbor (Microsoft) will be a.b.c.d+1. The last octet of the BGP neighbor's IPv4 address will always be an even
number.
Follow ER ASN requirements for the peering.
router bgp 65021
bgp router-id 10.6.32.241
bgp log-neighbor-changes
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 remote-as 12076
neighbor 216.221.237.34 description Microsoft peering to Azure
neighbor 216.221.237.34 local-as 394749
neighbor 216.221.237.34 activate
neighbor 216.221.237.34 password A1B2C3D4
neighbor 216.221.237.34 soft-reconfiguration inbound
redistribute connected
exit-address-family
!
address-family ipv4 vrf C101
neighbor 172.16.0.2 remote-as 12076
neighbor 172.16.0.2 description private peering to Azure
neighbor 172.16.0.2 local-as 64512
neighbor 172.16.0.2 activate
neighbor 172.16.0.2 password A1B2C3D4
neighbor 172.16.0.2 soft-reconfiguration inbound
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 35
redistribute connected
exit-address-family
Note: Password configuration is an optional feature for the ER BGP peering and not enabled by default.
See BGP Command Reference for more information to set up a password on the BGP peering.
Advertise Prefixes Over the BGP Session to Azure
Use network statement or redistribution from IGP to advertise your internal network prefixes to Azure.
router bgp 65021
!
address-family ipv4 vrf C101
network 192.168.0.4 mask 255.255.255.252
redistribute connected
redistribute static
Microsoft peering does not accept default route or private IP addresses (RFC 1918), the sample below use prefix-
list to filter them out.
router bgp 65021
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 prefix-list rfc1918 out
!
ip prefix-list rfc1918 deny 0.0.0.0/8 le 32
ip prefix-list rfc1918 deny 10.0.0.0/8 le 32
ip prefix-list rfc1918 deny 127.0.0.0/8 le 32
ip prefix-list rfc1918 deny 169.254.0.0/16 le 32
ip prefix-list rfc1918 deny 172.16.0.0/12 le 32
ip prefix-list rfc1918 deny 192.0.2.0/24 le 32
ip prefix-list rfc1918 deny 192.168.0.0/16 le 32
ip prefix-list rfc1918 deny 224.0.0.0/3 le 32
ip prefix-list rfc1918 deny 0.0.0.0/0
ip prefix-list rfc1918 permit 0.0.0.0/0 le 32
Microsoft Azure has policy of accepting up to 4000 (10,000 for Premium ExpressRoute) route prefixes for private
peering and 200 route prefixes for Microsoft peering. It is your responsibility to manage and aggregate network
prefix while advertising your internal network, otherwise Microsoft will drop the BGP session once prefix count goes
above the limit.
Filter Prefixes Received from Azure (Optional)
You can use route-maps and prefix lists to filter prefixes propagated into your network. You can use the sample
below to accomplish the task. Ensure that you have appropriate prefix lists setup.
router bgp 65021
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 route-map <MS_Prefixes_Inbound> in
address-family ipv4 vrf C101
neighbor 172.16.0.2 route-map <PP_Prefixes_Inbound> in
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 35
!
route-map <PP_Prefixes_Inbound> permit 10
match ip address prefix-list <PP_Prefixes>
!
route-map <MS_Prefixes_Inbound> permit 10
match ip address prefix-list <MS_Prefixes>
High Availability and Optimize Routing Configuration
We recommend that both ASR1000 routers have L3 peering to south bound corporate network router so that
customers can leverage High Availability or Equal Cost Multi-Path to load share traffic to ExpressRoute
Follow ER Optimize Routing from customer to Microsoft, BGP local preference is used to influence the routing.
Make sure you have the correct BGP community for region, e.g. USW is 12076:51006 and USW2 is12076:51026.
A detailed list of region to ER BGP communities can be found here under “Support for BGP Communities” section.
The sample below use BGP community “12076:51004” for the prefixes received from US East, and BGP
community “12076:51006” for the prefixes received from US West. We will assign US West region, e.g.
13.100.0.0/16, to higher local preference in the US West, and assign US East region, e.g. 23.100.0.0/16, to higher
local preference in the US East.
#US West ASR1000
!
router bgp 65021
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 route-map Peer-USW in
!
ip bgp-community new-format
!
ip community-list 1 permit 12076:51006
!
route-map Peer-USW permit 10
match community 1
set local-preference 400
#US East ASR1000
!
router bgp 65021
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 route-map Peer-USE in
!
ip bgp-community new-format
!
ip community-list 1 permit 12076:51004
!
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 35
route-map Peer-USE permit 10
match community 1
set local-preference 400
AS Path Prepending to Influence Routing
In order to optimize routing from Microsoft to your network, AS Path prepending is used to influence routing.
Microsoft removes private AS numbers in the AS PATH for the prefixes received on Microsoft Peering, so it is
important to append public AS numbers in the AS PATH to influence routing for Microsoft Peering. The sample
below did not follow the AS and IP scheme in Table 2, but based on the Microsoft ER example as shown in
Figure 4.
Figure 5. AS Path Prepending Sample
You can lengthen the AS PATH for 177.2.0.0/31 in US East so that Microsoft will prefer the ExpressRoute circuit in
US West for traffic destined for this prefix (as Microsoft network will think the path to this prefix is shorter in the
west). Similarly, by lengthening the AS PATH for 177.2.0.2/31 in US West so that Microsoft will prefer the
ExpressRoute circuit in US East.
#US West ASR1000
!
router bgp 345
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 route-map Prepend-USW out
network 177.2.0.0 mask 255.255.255.254
network 177.2.0.2 mask 255.255.255.254
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 35
!
ip prefix-list prefix_USW seq 10 permit 177.2.0.2/31
!
route-map Prepend-USW permit 10
match ip address prefix prefix_USW
set as-path prepend 345
!
route-map Prepend-USW permit 20
#US East ASR1000
!
router bgp 345
!
address-family ipv4 vrf C10
neighbor 216.221.237.134 route-map Prepend-USE out
network 177.2.0.0 mask 255.255.255.254
network 177.2.0.2 mask 255.255.255.254
!
ip prefix-list prefix_USE seq 10 permit 177.2.0.0/31
!
route-map Prepend-USE permit 10
match ip address prefix prefix_USE
set as-path prepend 345
!
route-map Prepend-USE permit 20
Avoid Asymmetric Routing
Follow ER asymmetric routing solutions, in the example, if you want to use the Internet for authentication traffic and
ExpressRoute for your mail traffic or other public services, you should not advertise your Active Directory
Federation Services (AD FS) public IP addresses over ExpressRoute. This best practice can be enforced with an
outbound route-map configuration:
router bgp 65021
!
address-family ipv4 vrf C10
neighbor 216.221.237.34 route-map AD_FS_Prefixes out
!
ip prefix-list AD_FS permit 121.10.0.1/32
!
route-map AD_FS_Prefixes deny 10
match ip address prefix-list AD_FS
route-map AD_FS_Prefixes permit 20
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 35
NAT Configuration
As per Microsoft NAT for ExpressRoute, Microsoft expects to support bi-directional connectivity on the Microsoft
peering. Traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they
enter the Microsoft network. You can use the sample configuration below to accomplish the task, it is using the MS
peering subinterface ip address as the NAT pool (216.221.237.33), so the returning traffic will be sent back to this
router, un-NATed before forwarded out of the DMZ VLAN.
interface TenGigabitEthernet0/1/0.310
description Customer 10 Primary Microsoft peering to Azure
encapsulation dot1Q 10 second-dot1q 310
ip vrf forwarding C10
ip address 216.221.237.33 255.255.255.252
ip nat outside
!
interface TenGigabitEthernet0/1/1.10
description Customer 10 DMZ VLAN
encapsulation dot1Q 10
ip vrf forwarding C10
ip address 192.168.0.1 255.255.255.252
ip nat inside
!
ip route vrf C10 216.221.236.33 255.255.255.255 null0
!
ip nat pool Cust10_MSFT_Pool 216.221.236.33 216.221.236.33 netmask
255.255.255.252
!
ip nat inside source route-map Cust10_MSFT_sNAT pool Cust10_MSFT_Pool vrf C10
overload
!
ip access-list extended Local_BGP_C10
remark deny BGP session from being NATed
permit tcp host 216.221.237.33 host 216.221.237.34 eq bgp
permit tcp host 216.221.237.34 host 216.221.237.33 eq bgp
!
access-list 10 permit 216.221.237.34
!
route-map Cust10_MSFT_sNAT deny 5
match ip address Local_BGP_C10
!
route-map Cust10_MSFT_sNAT permit 10
description NAT any traffic in VRF C10 with NH 216.221.237.34 toward Microsoft
Peering
match ip next-hop 10
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 35
It is your responsibility to ensure that the NAT IP pool advertised to Microsoft is NOT advertised to the Internet
(even as a subnet of the Internet advertisement, they must be completely non-overlapping). Failure to meet this
requirement may break connectivity to other Microsoft services.
NAT Common Best Practices
1. Set the NAT max-entries per system scale, which is 2M on ASR1001-X and ASR1001-HX. Other ASR1000
systems may have different NAT scale, please follow the relevant product datasheet.
ip nat translation max-entries 2000000
2. It is recommended to keep the default NAT timeout. If the user has specific needs to reduce the timer, for
example the pools are being exhausted, then the user can refer to the sample commands below to make
configuration changes:
The default NAT timeout values can be seen in show command
ASR1000#show platform hardware qfp active feature nat data time
Timeouts: default 86400; TCP 86400; TCP PPTP 86400; UDP 300; FINRST 60;
SYN 60; DNS 60; ICMP 60; Skinny 60; ICMP error 60; ESP 300
To change the timeout values for example:
ip nat translation tcp-timeout 10800
3. If there is the requirement that both NAT and non-NATted traffic must co-exist in the NAT outside interface,
then use Gatekeeper to optimize system performance:
ip nat settings gatekeeper-size 65535
Route Redistribution into EIGRP
In order to redistribute routes from the Private and Microsoft BGP Peerings to EIGRP, add the following
configuration
router eigrp 1
!
address-family ipv4 vrf C10
redistribute static route-map BGP_Private_to_App_EIGRP
redistribute bgp 65021 metric 1000000 100 255 1 1500
network 10.0.0.0 0.0.0.255
no auto-summary
autonomous-system 2
exit-address-family
!
address-family ipv4 vrf C101
redistribute bgp 65021 metric 1000000 100 255 1 1500
network 10.1.0.0 0.0.0.255
no auto-summary
autonomous-system 3
!
router bgp 65021
!
address-family ipv4 vrf C10
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 35
redistribute eigrp 2 route-map EIGRP_App_to_BGP
!
ip prefix-list BGP_Private_to_App_EIGRP seq 5 permit 10.3.0.0/23
!
ip access-list extended EIGRP_App_to_BGP
permit ip 10.0.0.0 0.0.0.255 any
!
route-map EIGRP_App_to_BGP permit 10
match ip address EIGRP_App_to_BGP
!
route-map BGP_Private_to_App_EIGRP permit 10
match ip address prefix-list BGP_Private_to_App_EIGRP
!
In order to NAT traffic from your corporate network, adjust the NAT configuration as follows
access-list 11 permit 10.1.0.0 0.0.0.255
route-map Cust10_MSFT_sNAT permit 10
description NAT any traffic in Corp_NET toward public peering
match ip address 11
Value-Added Feature Configurations
Configure Flexible Netflow
Flexible Netflow (FNF) is an embedded instrumentation capability within the ASR1000 to characterize network
operation, to characterize IP traffic, and understand how and where it flows is critical for network availability,
performance, and troubleshooting. The sample below shows how simple it can be to turn on FNF for ASR1000.
flow exporter C10_expo
destination 10.10.10.9 vrf C101
transport udp 9996
!
flow monitor C10_mon
exporter C10_expo
record netflow-original
!
interface TenGigabitEthernet0/1/0.310
description Customer 10 Primary Microsoft peering to Azure
ip flow monitor C10_mon input
ip flow monitor C10_mon output
!
interface TenGigabitEthernet0/1/0.3101
description Customer 10 Primary private peering to Azure
ip flow monitor C10_mon input
ip flow monitor C10_mon output
To be able to see bi-directional traffic in the ASR1000 system, the user can turn on ingress NetFlow on all
interfaces, or if the user is only interested in the bi-directional traffic from and to ER, turn on ingress and egress
NetFlow on ER. We recommend the use of full NetFlow instead of sampled NetFlow.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 35
Configure Quality of Service
Follow ER QoS requirements, a 6-class QoS model, as shown in Table 4, can be implemented to fulfill the
requirements while protecting the mission critical applications and network control traffic in the events of ER circuits
congestion. Use the sample QoS configuration below to accomplish the task.
Table 4. 6-Class QoS Model
Traffic Class DSCP Values Business workload Bandwidth % Congestion avoidance
Voice EF Skype / Lync voice 10 (PQ) -
Video AF41 Interactive Video, VBSS 30 remaining WRED
Network Control CS6 NET-CTRL* 5 remaining -
Transactional Data AF21 App Sharing 25 remaining WRED
Bulk Data AF11 File Transfer 25 remaining WRED
Class-default Catch-all Catch-all 15 remaining WRED
Note: BGP is always marked as CS6 by ASR1000 so it is protected in the NET-CTRL class.
class-map match-any VOICE
match dscp ef
class-map match-any VIDEO
match dscp af41
class-map match-any NETWORK-CONTROL
match dscp cs6
class-map match-any TRANSACTIONAL-DATA
match dscp af21
class-map match-any BULK-DATA
match dscp af11
!
! example of 500Mbps of ER circuit, adapt it to your circuit BW accordingly.
policy-map ER-500MBPS-POLICY
class class-default
shape average 500000000
service-policy ER
!
policy-map ER
class VOICE
priority level 1
police cir percent 10
class VIDEO
bandwidth remaining percent 30
random-detect
class NETWORK-CONTROL
bandwidth remaining percent 5
class TRANSACTIONAL-DATA
bandwidth remaining percent 25
random-detect
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 35
class BULK-DATA
bandwidth remaining percent 25
random-detect
class class-default
bandwidth remaining percent 15
random-detect
set dscp 0
! Microsoft require user to rewrite all other DSCP to 0 before sending the
packets to ER
!
interface TenGigabitEthernet0/1/0.310
description Customer 10 Primary Microsoft peering to Azure
service-policy output ER-500MBPS-POLICY
Advanced Services Configurations
Configure Application Visibility and Control (AVC)
If the DSCP values for applications above have not been marked properly or not preserved in your network before
reaching the ASR1000, use the Solution Reference Network Designs (SRND) policy model to simply application
classification in NBAR, and mark the application to the DSCP specified by Microsoft.
class-map match-all VOICE
match protocol attribute traffic-class voip-telephony
match protocol attribute business-relevance business-relevant
class-map match-all BROADCAST-VIDEO
match protocol attribute traffic-class broadcast-video
match protocol attribute business-relevance business-relevant
class-map match-all INTERACTIVE-VIDEO
match protocol attribute traffic-class real-time-interactive
match protocol attribute business-relevance business-relevant
class-map match-all MULTIMEDIA-CONFERENCING
match protocol attribute traffic-class multimedia-conferencing
match protocol attribute business-relevance business-relevant
class-map match-all MULTIMEDIA-STREAMING
match protocol attribute traffic-class multimedia-streaming
match protocol attribute business-relevance business-relevant
class-map match-all SIGNALING
match protocol attribute traffic-class signaling
match protocol attribute business-relevance business-relevant
class-map match-all NETWORK-CONTROL
match protocol attribute traffic-class network-control
match protocol attribute business-relevance business-relevant
class-map match-all NETWORK-MANAGEMENT
match protocol attribute traffic-class ops-admin-mgmt
match protocol attribute business-relevance business-relevant
class-map match-all TRANSACTIONAL-DATA
match protocol attribute traffic-class transactional-data
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 35
match protocol attribute business-relevance business-relevant
class-map match-all BULK-DATA
match protocol attribute traffic-class bulk-data
match protocol attribute business-relevance business-relevant
class-map match-all SCAVENGER
match protocol attribute business-relevance business-irrelevant
!
policy-map MARKING
class VOICE
set dscp ef
class BROADCAST-VIDEO
set dscp af41
class INTERACTIVE-VIDEO
set dscp af41
class MULTIMEDIA-CONFERENCING
set dscp af41
class MULTIMEDIA-STREAMING
set dscp af41
class SIGNALING
set dscp af41
class NETWORK-CONTROL
set dscp cs6
class NETWORK-MANAGEMENT
set dscp default
class TRANSACTIONAL-DATA
set dscp af21
class BULK-DATA
set dscp af11
class SCAVENGER
set dscp default
class class-default
set dscp default
!
interface TenGigabitEthernet0/1/1.10
description Customer 10 DMZ VLAN
service-policy input MARKING
!
Configure IPsec VPN
A common use utilizes the Cisco Cloud Services Router, CSR1000v, deployed as an application VNet gateway in
Azure to provide IPsec gateway for entire VNet. See Extending Enterprise Network into Public Cloud with Cisco
CSR1000v. The ASR1000 connecting to ER is the ideal on-premises gateway for the IPsec tunnel termination in
Enterprise network as the platform delivers embedded hardware acceleration for IPsec VPN. For details on
ASR1000 system IPsec throughput, refer to the relevant product datasheet.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 35
The Cisco CSR1000v is ASR1000 in virtual form factor. They run the same IOS XE software release, inherit the
same IOS XE software architecture, support the same CLIs and feature sets of IPsec VPN.
Once you have deployed CSR1000v on Microsoft Azure, you would configure the IPsec VPN on the CSR1000v by
using the step-by-step procedure outlined in this video demo or as per the sample:
crypto isakmp policy 200
encryption aes
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp key cisco123 address 0.0.0.0
crypto isakmp keepalive 10 10
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set csr esp-aes esp-sha-hmac
mode tunnel
crypto ipsec df-bit clear
!
crypto ipsec profile csr
set transform-set csr
!
interface Tunnel1
ip address 192.168.100.2 255.255.255.252
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 172.16.0.1
tunnel protection ipsec profile csr
You should have the IPsec tunnel peer configuration on the ASR1000 enabled as per the sample:
crypto isakmp policy 200
encryption aes
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp key cisco123 address 0.0.0.0
crypto isakmp keepalive 10 10
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set csr esp-aes esp-sha-hmac
mode tunnel
crypto ipsec df-bit clear
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 35
!
crypto ipsec profile csr1
set transform-set csr
!
interface Tunnel1
ip vrf forwarding C101
ip address 192.168.100.1 255.255.255.252
tunnel source TenGigabitEthernet0/1/0.3101
tunnel mode ipsec ipv4
tunnel destination 10.0.0.4
tunnel protection ipsec profile csr1
Test Connectivity
While there are steps to verify ExpressRoute connectivity with Microsoft, there are also verification steps can be
performed on ASR1000 and in the customer on-premises network.
Verify the BGP Neighbors
Use the following commands to verify the Microsoft peering and Private BGP peering are established and Up
ASR1000#show ip bgp vpnv4 vrf C10 neighbor 216.221.237.34
BGP neighbor is 216.221.237.34, vrf C10, remote AS 12076, local AS 394749,
external link
Description: Microsoft peering to Azure
BGP version 4, remote router ID 207.46.160.94
BGP state = Established, up for 00:39:52
Last read 00:00:16, last write 00:00:39, hold time is 180, keepalive interval
is 60 seconds
Neighbor sessions:
1 active, is not multisession capable (disabled)
Neighbor capabilities:
Route refresh: advertised and received(new)
Four-octets ASN Capability: advertised and received
Address family IPv4 Unicast: advertised and received
Enhanced Refresh Capability: advertised and received
Multisession Capability:
Stateful switchover support enabled: NO for session 1
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 2 3
Keepalives: 45 45
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 35
Route Refresh: 0 0
Total: 48 49
Do log neighbor state changes (via global configuration)
Default minimum time between advertisement runs is 0 seconds
For address family: VPNv4 Unicast
Translates address family IPv4 Unicast for VRF C10
Session: 216.221.237.34
BGP table version 1326, neighbor version 1326/0
Output queue size : 0
Index 17, Advertise bit 1
17 update-group member
Inbound soft reconfiguration allowed
Outbound path policy configured
Outgoing update prefix filter list is rfc1918
Route map for outgoing advertisements is AD_FS_Prefixes
Slow-peer detection is disabled
Slow-peer split-update-group dynamic is disabled
Sent Rcvd
Prefix activity: ---- ----
Prefixes Current: 1 144 (Consumes 19584 bytes)
Prefixes Total: 1 144
Implicit Withdraw: 0 0
Explicit Withdraw: 0 0
Used as bestpath: n/a 144
Used as multipath: n/a 0
Used as secondary: n/a 0
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
prefix-list 3 0
Bestpath from this peer: 144 n/a
Total: 147 0
Number of NLRIs in the update sent: max 73, min 0
Last detected as dynamic slow peer: never
Dynamic slow peer recovered: never
Refresh Epoch: 1
Last Sent Refresh Start-of-rib: never
Last Sent Refresh End-of-rib: never
Last Received Refresh Start-of-rib: never
Last Received Refresh End-of-rib: never
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 35
Sent Rcvd
Refresh activity: ---- ----
Refresh Start-of-RIB 0 0
Refresh End-of-RIB 0 0
Address tracking is enabled, the RIB does have a route to 216.221.237.34
Route to peer address reachability Up: 4; Down: 1
Last notification 03:14:13
Connections established 5; dropped 4
Last reset 00:42:26, due to BGP Notification received, Connection Collision
Resolution
External BGP neighbor configured for connected checks (single-hop no-disable-
connected-check)
Interface associated: TenGigabitEthernet0/1/0.3101 (peering address in same
link)
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
SSO is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 216.221.237.33, Local port: 48945
Foreign host: 216.221.237.34, Foreign port: 179
Connection tableid (VRF): 2
Maximum output segment queue size: 50
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x153EE19F):
Timer Starts Wakeups Next
Retrans 47 0 0x0
TimeWait 0 0 0x0
AckHold 46 42 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 1521 520 0x153EE253
DeadWait 0 0 0x0
Linger 0 0 0x0
ProcessQ 0 0 0x0
iss: 2713507505 snduna: 2713508500 sndnxt: 2713508500
irs: 120760723 rcvnxt: 120762358
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 35
sndwnd: 15390 scale: 0 maxrcvwnd: 16384
rcvwnd: 16213 scale: 0 delrcvwnd: 171
SRTT: 998 ms, RTTO: 1014 ms, RTV: 16 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms
uptime: 2392902 ms, Sent idletime: 16537 ms, Receive idletime: 16737 ms
Status Flags: active open
Option Flags: VRF id set, nagle, path mtu capable
IP Precedence value : 6
Datagrams (max data segment is 1460 bytes):
Rcvd: 93 (out of order: 0), with data: 47, total data bytes: 1634
Sent: 94 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0),
with data: 47, total data bytes: 994
Packets received in fast path: 0, fast processed: 0, slow path: 0
fast lock acquisition failures: 0, slow path: 0
TCP Semaphore 0x7FAA555FB670 FREE
ASR1000#show ip bgp vpnv4 vrf C10 neighbor 172.16.0.2
BGP neighbor is 172.16.0.2, vrf C10, remote AS 12076, local AS 64512, external
link
Description: private peering to Azure
BGP version 4, remote router ID 207.46.160.94
BGP state = Established, up for 4d01h
Last read 00:00:19, last write 00:00:03, hold time is 180, keepalive interval
is 60 seconds
Neighbor sessions:
1 active, is not multisession capable (disabled)
Neighbor capabilities:
Route refresh: advertised and received(new)
Four-octets ASN Capability: advertised and received
Address family IPv4 Unicast: advertised and received
Enhanced Refresh Capability: advertised and received
Multisession Capability:
Stateful switchover support enabled: NO for session 1
Message statistics:
InQ depth is 0
OutQ depth is 0
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 35
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 43 2
Keepalives: 6410 6400
Route Refresh: 0 0
Total: 6456 6403
Do log neighbor state changes (via global configuration)
Default minimum time between advertisement runs is 0 seconds
For address family: VPNv4 Unicast
Translates address family IPv4 Unicast for VRF C10
Session: 172.16.0.2
BGP table version 1326, neighbor version 1326/0
Output queue size : 0
Index 16, Advertise bit 0
16 update-group member
Inbound soft reconfiguration allowed
Outbound path policy configured
Route map for outgoing advertisements is Prepend-USW
Slow-peer detection is disabled
Slow-peer split-update-group dynamic is disabled
Sent Rcvd
Prefix activity: ---- ----
Prefixes Current: 1 1 (Consumes 136 bytes)
Prefixes Total: 1 1
Implicit Withdraw: 0 0
Explicit Withdraw: 147 0
Used as bestpath: n/a 1
Used as multipath: n/a 0
Used as secondary: n/a 0
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
route-map: 0 7
Other Policies: 291 n/a
Total: 291 7
Number of NLRIs in the update sent: max 143, min 0
Last detected as dynamic slow peer: never
Dynamic slow peer recovered: never
Refresh Epoch: 1
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 35
Last Sent Refresh Start-of-rib: 04:36:39
Last Sent Refresh End-of-rib: 04:36:39
Refresh-Out took 0 seconds
Last Received Refresh Start-of-rib: never
Last Received Refresh End-of-rib: never
Sent Rcvd
Refresh activity: ---- ----
Refresh Start-of-RIB 1 0
Refresh End-of-RIB 1 0
Address tracking is enabled, the RIB does have a route to 172.16.0.2
Route to peer address reachability Up: 1; Down: 0
Last notification 4d01h
Connections established 1; dropped 0
Last reset never
External BGP neighbor configured for connected checks (single-hop no-disable-
connected-check)
Interface associated: TenGigabitEthernet0/1/0.3103 (peering address in same
link)
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
SSO is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 172.16.0.1, Local port: 179
Foreign host: 172.16.0.2, Foreign port: 28211
Connection tableid (VRF): 2
Maximum output segment queue size: 50
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x1542D0A0):
Timer Starts Wakeups Next
Retrans 6431 0 0x0
TimeWait 0 0 0x0
AckHold 6401 6287 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0
Linger 0 0 0x0
ProcessQ 0 0 0x0
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 35
iss: 763195641 snduna: 763324044 sndnxt: 763324044
irs: 1048104958 rcvnxt: 1048226686
sndwnd: 15054 scale: 0 maxrcvwnd: 16384
rcvwnd: 16099 scale: 0 delrcvwnd: 285
SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms
uptime: 349616252 ms, Sent idletime: 3404 ms, Receive idletime: 3203 ms
Status Flags: passive open, gen tcbs
Option Flags: VRF id set, nagle, path mtu capable
IP Precedence value : 6
Datagrams (max data segment is 1460 bytes):
Rcvd: 12802 (out of order: 0), with data: 6402, total data bytes: 121727
Sent: 12808 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion:
0), with data: 6435, total data bytes: 128402
Packets received in fast path: 0, fast processed: 0, slow path: 0
fast lock acquisition failures: 0, slow path: 0
TCP Semaphore 0x7FAA5454F230 FREE
BGP session is essential to maintain ER connectivity. To protect BGP packets in the ASR1000 punt path and
mitigate potential DDoS attacks, it is recommended you implement Control Plane Policing as per the Control Plane
Policing template on page 50 -53.
Verify ExpressRoute Connectivity
Follow the procedure here to verify ExpressRoute connectivity. The ExpressRoute circuit can be validated by using
the Azure portal “Home > ExpressRoute circuit”, and looking at the “Essentials” field. If you see “Circuit status” is
Enabled, then the ExpressRoute Circuit is up on the Microsoft side, and the “Provider status” as Provisioned, then
the circuit is up on the service provider side, as shown in Figure 5.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 35
Figure 6. Verify ExpressRoute Circuit Status in Azure Portal Snapshot
To further validate that the circuit is up from the customer side, click “Home > ExpressRoute circuit > Azure
Private/Microsoft Private > Get route table summary” to see if your subinterface networks are reachable, as shown
in Figure 6 and 7 respectively.
Figure 7. Verify Private Peering Customer Networks are Reachable in Azure Portal Snapshot
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 35
Figure 8. Verify Microsoft Peering Customer Networks are Reachable in Azure Portal Snapshot
Verify NAT Translation Entries and Pool
Follow NAT monitoring and Maintaining guide to verify NAT translation entries are set up properly.
ASR1000#show ip nat translation
Pro Inside global Inside local Outside local Outside global
icmp 216.221.236.33:98 192.168.0.1:98 216.221.237.34:98
216.221.237.34:98
Total number of translations: 1
To monitor the pool stats:
ASR1000#show platform software nat fp active pool
Dump NAT pool config
ID: 1, Name: Cust10_MSFT_Pool, Type: Generic, Mask: 255.255.255.252
Flags: Unknown, Acct name:
Address range blocks: 1
Start: 216.221.236.33, End: 216.221.236.33
Last stats update: 02/13 17:35:39.556
Last refcount value: 1
ASR1000#show platform software nat fp active pool-stats id <id>
NAT Pool Statistics
Pool name Cust10_MSFT_Pool, id 1
Assigned Available
Addresses 0 1
UDP Low Ports 0 512
TCP Low Ports 0 512
UDP High Ports 0 64512
TCP High Ports 0 64512
(Low ports are less than 1024. High ports are greater than or equal to 1024.)
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 35
Verify Netflow Entries
The ASR1000 exports the NetFlow cache entries directly from the Quantum Flow Processor to the external
collector via in-band interface. Do NOT connect the collector on the management interface (GigabitEthernet0). Use
the following command to verify the flow monitor is exporting data to the exporters.
ASR1000#show flow monitor C10_mon
Flow Monitor C10_mon:
Description: User defined
Flow Record: netflow-original
Flow Exporter: C10_expo
Cache:
Type: normal (Platform cache)
Status: allocated
Size: 200000 entries
Inactive Timeout: 15 secs
Active Timeout: 1800 secs
Trans end aging: off
Use the Top N talkers capability, which facilitates real-time traffic analysis of the most traffic volume consumers.
ASR1000#show flow monitor C10_mon cache sort counter packets top 3 format table
Processed 2 flows
Aggregated to 2 flows
Showing the top 2 flows
IPV4 SRC ADDR IPV4 DST ADDR TRNS SRC PORT TRNS DST PORT INTF
INPUT FLOW SAMPLER ID IP TOS IP PROT ip src as ip dst as
ipv4 next hop addr ipv4 src mask ipv4 dst mask tcp flags intf
output bytes pkts time first time last
=============== =============== ============= ============= ====================
=============== ====== ======= ========= ========= ==================
============= ============= ========= ==================== ========== ==========
============ ============
10.3.0.5 192.168.0.1 0 2048
Te0/1/0.3103 0 0x00 1 0 0
0.0.0.0 /0 /0 0x00 Null
91860 1531 17:16:36.049 17:42:19.371
192.168.0.1 10.3.0.5 0 0 Null
0 0x00 1 0 12076 172.16.0.2
/32 /23 0x00 Te0/1/0.3103 91620
1527 17:16:40.065 17:42:19.371
Note: ASR1000 does not support aggregate flows in Top N talkers.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 35
ASR1000 Proactive System Monitoring
Proactive monitoring system resources allows you to detect potential problems before they happen, thus avoiding
outages. Figure 8 highlights key system resources to monitor on ASR1000.
Figure 9. Key System Resources to Monitor - Summary
The system resources to be consumed by each of the features discussed in the configuration guide are listed in
Table 5.
Table 5. Feature to System Resources Consumption
Features System Resources Consumed
BGP IOS memory/CPU, RP memory/CPU
FIB IOS memory/CPU, RP memory/CPU
NAT QFP, resource DRAM, TCAM
Netflow QFP, resource DRAM
QoS QFP, TCAM
AVC QFP, resource DRAM, TCAM
IPsec IOS memory/CPU, RP memory/CPU, QFP, Crypto Assist, TCAM
The best practice is that during steady state the system should have minimum 25% of IOS memory, RP memory,
and resource DRAM available to accommodate potential network churning and reconvergence events; otherwise,
you should plan to upgrade system memory or upgrade to a higher performance ASR1000 variant such as the
ASR1002-HX.
For exact CLIs and MIBs to monitor each system source, follow the Operating an ASR1000 guide page 24 - 37.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 35 of 35
References
Please refer to the following documentation for ASR1000 platform architecture, packet flow, feature configuration
guide and datasheet:
● ASR1000 System Architecture Overview
● BGP Configuration Guide
● NAT Configuration Guide
● QoS Configuration Guide
● Flexible Netflow Configuration Guide
● NBAR Configuration Guide
● AVC Configuration Guide
● Security for VPNs with IPsec
● IPsec Virtual Tunnel Interface
● ASR1000 Routers Datasheet
● ASR1000-X Router Hardware Installation Guide
● ASR1000-HX Router Hardware Installation Guide
● ASR1000 ESP Datasheet
● ASR1000 Ordering Guide
● IOS-XE NGE Support Product Tech Note
Refer to the following documentation for common error messages and troubleshooting notes:
● Troubleshooting of ASR1k Made Easy
● ASR1000 Troubleshooting TechNotes
● ASR1000 Error and System Messages
● Embedded Packet Capture for IOS-XE
Printed in USA C07-740698-00 07/18