cisco asr1000 and microsoft azure expressroute … · cisco asr1000 and microsoft azure...

Cisco ASR1000 and Microsoft Azure ExpressRoute Joint Validated Design Extend your enterprise network into Azure with Cisco ASR®1000

Written by Jason Yang and Kevin Echols II

March 13, 2018

Cisco ASR1000 and Microsoft Azure ExpressRoute Joint Validated Design

2

Table of Contents

Introduction .......................................................................................................................... 3 Target Audience .............................................................................................................................3 Purpose of This Document .............................................................................................................3 Solution Overview .........................................................................................................................4

Product Overview .................................................................................................................. 5

Preparation ........................................................................................................................... 6 Getting Started ..............................................................................................................................7

Configuration: ExpressRoute Peering on Azure ...................................................................... 7

Configuration: Cisco ASR1000 ................................................................................................ 8 Two Router Deployment vs. One Router Deployment .....................................................................8 Interface Configurations .................................................................................................................9

802.1Q-in-Q VLAN ID Sample Interface Configuration ........................................................................ 9 802.1Q VLAN ID Sample Interface Configuration............................................................................... 10

BGP Configurations ...................................................................................................................... 11 Setup eBGP Sessions .......................................................................................................................... 11 Advertise Prefixes Over the BGP Session to Azure ............................................................................ 12 Filter Prefixes Received from Azure (Optional) .................................................................................. 12 High Availability and Optimize Routing Configuration ....................................................................... 13 AS Path Prepending to Influence Routing .......................................................................................... 14 Avoid Asymmetric Routing ................................................................................................................. 15

NAT Configuration ....................................................................................................................... 15 NAT Common Best Practices .............................................................................................................. 16

Route Redistribution into EIGRP ................................................................................................... 17

Value-Added Feature Configurations ................................................................................... 18 Configure Flexible Netflow ........................................................................................................... 18 Configure Quality of Service ......................................................................................................... 19

Advanced Services Configurations ....................................................................................... 20 Configure Application Visibility and Control (AVC) ........................................................................ 20 Configure IPsec VPN ..................................................................................................................... 21

Test Connectivity ................................................................................................................. 23 Verify the BGP Neighbors ............................................................................................................. 23 Verify ExpressRoute Connectivity ................................................................................................. 32 Verify NAT Translation Entries and Pool........................................................................................ 34 Verify Netflow Entries .................................................................................................................. 35

ASR1000 Proactive System Monitoring ................................................................................ 36

References .......................................................................................................................... 37


3

Introduction The majority of enterprises have started deploying business-critical applications that span on premise equipment and cloud infrastructure in what is known as a hybrid cloud deployment. These enterprises seek to benefit from reduced total cost of ownership (TCO), the ability to scale applications to meet growing demands, and an always on guarantee via distributed workloads across multiple availability zones and geographic regions. Establishing a reliable connection from on premise to the cloud has proven difficult for many of these enterprises as the Internet does not guarantee the metrics required for crucial business applications. Cisco and Microsoft have partnered to make the transition to a hybrid cloud deployment easier for our mutual customers by publishing a jointly-validated Microsoft Azure ExpressRoute and Cisco ASR1000 Design Guide. Microsoft’s ExpressRoute lets you extend your on premise networks into Microsoft Azure over a private connection facilitated by a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud services, such as Microsoft Azure, Office 365, and Dynamics 365.

Target Audience The intended audience for this document includes sales engineers, field consultants, professional services staff, IT managers, partner engineering staff, and customers deploying the Microsoft Azure ExpressRoute with Cisco ASR1000 routers. External references are provided wherever applicable, but readers are expected to be familiar with the technology, infrastructure, and enterprise security policies of the customer installation.

Purpose of This Document Cisco-Microsoft Joint Validated Designs provide guidelines for creating an end-to-end solution that enable you to make informed decisions with the goal of successfully creating a hybrid cloud deployment. This document describes the steps required to extend your on premises network into the Microsoft Azure with ExpressRoute using the Cisco ASR1000 Series Routers. To connect to Microsoft Azure services using ExpressRoute, Microsoft provides best practices for network security, optimize routing, asymmetric routing, and NAT. This guide will focus on how to implement these best practices with ASR1000 configurations, recommend advanced features and services on the ASR1000. Please note that this guide is not meant to be a comprehensive overview of the ASR1000 platform and routing technologies, see References section for platform and feature configuration guides.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-introduction

https://www.cisco.com/c/en/us/products/routers/asr-1000-series-aggregation-services-routers/index.html

https://docs.microsoft.com/en-us/azure/best-practices-network-security

https://docs.microsoft.com/en-us/azure/best-practices-network-security

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-optimize-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-asymmetric-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-nat


4

Cisco validation provides further confirmation of solution compatibility, connectivity, and correct operation for the on premise deployment. Although readers of this document are expected to have sufficient knowledge to install and configure the products used, the Cisco-Microsoft Joint Validated Design provides configuration details that are important to the deployment of this solution.

Solution Overview ExpressRoute supports layer 3 connectivity between your on premise network and Microsoft Azure through a connectivity provider in 3 connectivity models: CloudExchange Co-location, Point-to-point Ethernet Connection, and IP VPN Connection. ExpressRoute connections do not go over the public Internet, which allows ExpressRoute connections to offer more reliability, faster speeds, lower latencies, and higher security than typical connections over the Internet. As shown in Figure 1, ExpressRoute circuits have multiple routing domains associated with them: Azure private peering, and Microsoft peering. Each of the routing domains are configured in separate virtual routing and forwarding (VRF) domains on a pair of ASR1000 routers for high availability. In Figure 1, these routers are shown located in the partner edge block.

Figure 1: Common ExpressRoute Deployment

ExpressRoute capabilities and features are identical across all of the connectivity models. The ASR1000 physical connectivity configuration to each of the service providers may vary, but the configuration to ExpressRoute will be identical.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-connectivity-models


5

Product Overview Cisco ASR1000 Series Aggregation Services Routers aggregate multiple WAN connections and network services, including encryption and traffic management, and forward them across WAN connections at line speeds from 2.5 to 200Gbps. ASR1000 Series routers offer elastic service delivery; programmability and automation; up to five-nines availability; comprehensive and flexible QoS; and advanced services, such as IPsec VPN and Application Visibility and Control (AVC) for enterprise networks. The Cisco ASR1000 Series platforms vary in I/O connectivity speed, density, system performance, and redundancy options. All models use the Cisco Quantum Flow Processor and support the same feature set available on the Cisco IOS XE Operating System. All this commonality simplifies management and operations. ExpressRoute circuits are purchased based on a number of bandwidth options. Table 1 outlines ASR1000 platform recommendations for each of the ExpressRoute bandwidth options.

Table 1: ExpressRoute Circuit Bandwidth to ASR1000 Platform Recommendations

ExpressRoute Circuit Bandwidths

ASR1000 Platform Interface Type

50 Mbps ASR 1001-X GigabitEthernet




1 Gbps ASR 1001-X or ASR1001-HX

GigabitEthernet or TenGigabitEthernet

2 Gbps ASR 1001-HX TenGigabitEthernet



The ASR1001X, pictured in Figure 2, is a 1 RU form factor, supports redundant power supplies, and the Embedded Services Processor (ESP) has default throughput of 2.5Gbps that is upgradable to 5-, 10-, or 20Gbps via software activation. The platform consumes 250W at max with front-to-back airflow. Onboard, the ASR1001X has 6 Gigabit Ethernet SFP ports, 2 TenGigabit Ethernet SFP+ ports, and has a single half-height Shared Port Adapter (SPA) that can be configured with a range of interfaces from a 2-port Gigabit Ethernet SPA to a T1/E1 NIM. See the ASR1001X Datasheet for more details on the platform, and the ASR1001X Hardware Installation Guide from a complete list of supported hardware.

https://developer.cisco.com/site/ios-xe/

https://www.cisco.com/c/en/us/products/collateral/routers/asr-1002-router/solution_overview_c22-449961.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-cfg-vpn-ipsec.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide/avc_config.html


6

Figure 2: ASR1001X

The ASR1001HX, pictured in Figure 3, is a 1 RU form factor, supports redundant power supplies, and the Embedded Services Processor (ESP) has throughput up to 60Gbps. The platform consumes 360W at max with front-to-back airflow. Onboard, the ASR1001HX has 8 Gigabit Ethernet SFP ports and 8 TenGigabit Ethernet SFP+ ports, where 4 of the TenGigabit Ethernet ports (Te4-7) are compatible with SFPs. See the ASR1001HX Datasheet for more details on the platform, and the ASR1001HX Hardware Installation Guide from a complete list of supported hardware.

Figure 3: ASR1001HX

Preparation The configuration guide will include numerous value substitutions provided for the purpose of example only. Any references to IP addresses, device IDs, shared secrets or keys account information or project names should be replaced with the appropriate values for your environment when following this guide. Values unique to your environment will be highlighted in bold. This guide is not meant to be a comprehensive setup for entire device configuration for all network connectivity, for example, the same device may also have connectivity to the enterprise data center, campus, or branches, the configuration of which is outside the scope of this guide. This configuration guide will focus on the connectivity to the ExpressRoute. List 1 provides a high-level overview of the configuration process that will be covered.

List 1: High-Level Overview of ASR1000 Configuration Process 1. Interface Configurations

a. 802.1Q-in-Q VLAN ID Sample Interface Configuration b. 802.1Q VLAN ID Sample Interface Configuration

2. BGP Configurations a. Setup eBGP Sessions b. Advertise Prefixes Over the BGP Session to Azure


7

c. Filter Prefixes Received from Azure (Optional) d. High Availability and Optimize Routing Configuration e. AS Path Prepending to Influence Routing f. Avoid Asymmetric Routing g. NAT Configuration h. NAT Common Best Practices

3. Route Redistribution into EIGRP 4. Advanced Feature Configurations

a. Flexible Netflow Configuration b. Quality of Service Configuration

5. Advanced Services Configurations a. Application Visibility and Control (AVC) Configuration b. IPsec VPN Configuration

Getting Started It is assumed that you met all the requirements in ExpressRoute prerequisites & checklist, the ExpressRoute circuits have been created, and the ExpressROute circuit porivisoned by the service provider. The first step in configuring your Cisco ASR1000 for use with the ExpressRoute connectivity is to ensure that the following prerequisite conditions have been met: The essential feature set (BGP, NAT, VRF-Lite, IPv4/IPv6 dual-stack) required for setting up the ExpressRoute connectivity and the advanced features are supported by the ASR1000 universal image, that is, no additional license is required. The advanced services require AES license in addition to base licenses:

1. NBAR/AVC requires AVC feature license 2. The IPsec application requires:

a. Advanced Enterprise Services(SLASR1-AES) or Advanced IP Services Technology Package License (SLASR1-AIS)

b. IPsec RTU license (FLASR1-IPSEC-RTU) c. Encryption HW module (ASR1001HX-IPSECW) and Tiered Crypto throughput

license which applies to ASR1001-HX chassis Refer to the ASR1000 Routers Ordering Guide for more details on ASR1000 Series Router license information. The recommended software image is 16.6.2 and onward. Suggested images are recommended

on the on cisco.com download software page, where the suggested image is labeled by icon.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-prerequisites

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-portal-resource-manager

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/guide-c07-731639.html


8

Configuration: ExpressRoute Peering on Azure Follow the ExpressRoute peering steps in Azure portal.

Configuration: Cisco ASR1000

Two Router Deployment vs. One Router Deployment We recommend the deployment of two ASR1000s in a redundant pair to connect to the ExpressRoute service. Each router will need two QinQ subinterfaces on the physical interface. At the Microsoft Edge (see Figure 1) an ExpressRoute service is terminated on a pair of Microsoft ExpressRoute Edge (MSEE) routers. The MSEE routers hand off to a pair of Connectivity Provider routers, and then down to the customer’s ASR1000 routers. Microsoft will always have two BGP sessions for each of the peering types. As an example, assume Connectivity Provider defines an outer dot1Q tag of 10 for ER circuit, and the customer requests an inner tag of 310 for the Microsoft peering, and 3101 for the private peering. Table 2 outlines the example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in the customer edge dual router design.

Table 1: Router, Interface, Subinterfaces, VRFs and Peering for Customer Edge Dual Router Design

Routers R1 R2

Interfaces TE0/1/0 TE0/1/1 TE0/1/0 TE0/1/1

Interface description

Connection to ER Primary

Connection to customer corp network

Connection to ER Secondary

Connection to customer corp network

Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101

Subinterface description

Primary Microsoft Peering

Primary Private Peering

DMZ VLAN Corp VLAN Secondary Microsoft Peering

Secondary Private Peering

DMZ VLAN Corp VLAN

Encapsulation dot1Q 10 second-dot1q 310 or dot1Q 310

dot1Q 10 second-dot1q 3101 or dot1Q 3101

dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310 or dot1Q 310


dot1Q 10 dot1Q 101

VRFs* C10 C101 C10 C101 C10 C101 C10 C101

IP Addresses 216.221.237.33/30

172.16.0.1/30

192.168.0.1/30

192.168.0.5/30

216.221.237.37/30

172.16.0.5/30

192.168.0.1/30

192.168.0.5/30

Note: it is best practice to separate private peering and Microsoft peering with two separate VRFs. The private peering is considered trusted, whereas the Microsoft peering is a public network. The customer can send each VRFs/VLANs to the appropriate security zone before entering/exiting their corporate VLANs.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-routing-portal-resource-manager


9

Unless otherwise stated, this configuration guide provides configuration example on Router R1. Router R2 should have the same configuration as R1, with the exception of IP addresses/subnets. The subinterface, IP address, and VRF will use the example provided in Table 2. Optionally, if the customer chooses to deploy one router for connection to ER circuit, Table 3 outlines an example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in single customer edge router design.

Table 2: Interface, Subinterfaces, VRFs and Peering for Customer Edge Single Router Design Interfaces TE0/1/0 TE0/1/1 TE0/1/2

Interface description

Connection to ER Primary Connection to customer corp network

Connection to ER Secondary

Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101

Subinterface description

Primary Microsoft Peering

Primary Private Peering

DMZ VLAN Corp VLAN Secondary Microsoft Peering

Secondary Private Peering

Encapsulation dot1Q 10 second-dot1q 310 or dot1Q 310


dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310 or dot1Q 310


VRFs C10 C101 C10 C101 C10 C101

IP Addresses 216.221.237.33/30

172.16.0.1/30 192.168.0.1/30 192.168.0.5/30 216.221.237.37/30

172.16.0.5/30

Interface Configurations This section provides the interface configuration of Cisco ASR1000 to connect to ER. At least one internal facing interface is required to connect to your own network, and one external facing interface is required to connect to ExpressRoute. You will require a subinterface per peering in every router you connect to ER. A subinterface can be identified with an 802.1Q-in-Q VLAN ID or 802.1Q VLAN ID based on the connectivity providers’ requirement and an IP address. Follow ER IP address requirements for the BGP peering.

802.1Q-in-Q VLAN ID Sample Interface Configuration

ip vrf C10

rd 65021:10

!

ip vrf C101

rd 65021:101

!

interface TenGigabitEthernet0/1/0

description connection to ER Primary

no ip address

dot1q tunneling ethertype 0x9100

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-routing


10

! The default ethertype is 0x8100, can be changed to

0x88A8|0x9100|0x9200 to meet the connectivity provider’s requirement

!

interface TenGigabitEthernet0/1/0.310

description Customer 10 Primary Microsoft peering to Azure

encapsulation dot1Q 10 second-dot1q 310

ip vrf forwarding C10

ip address 216.221.237.33255.255.255.252

!


description Customer 10 Primary private peering to Azure



ip address 172.16.0.1 255.255.255.252

!


description Customer 10 Corporate facing interface

no ip address

!


description Customer 10 DMZ VLAN

encapsulation dot1Q 10


ip address 192.168.0.1 255.255.255.252

!


description Customer 10 Corp VLAN



ip address 192.168.0.5 255.255.255.252

802.1Q VLAN ID Sample Interface Configuration

ip vrf C10

rd 65021:10

!

ip vrf C101

rd 65021:101

!


description connection to ER

no ip address

!





ip address 216.221.237.33 255.255.255.252

!





ip address 172.16.0.1 255.255.255.252

!



11

description Customer 10 Corporate facing interface

no ip address

!





ip address 192.168.0.1 255.255.255.252

!


description Customer 10 Corp VLAN



ip address 192.168.0.5 255.255.255.252

Note: The MTU for the ExpressRoute interface is 1500 Bytes, which is the default MTU on ASR1000 Ethernet interface.

BGP Configurations

Setup eBGP Sessions

You must setup a BGP session with Microsoft for every peering. The sample below enables you to setup a BGP session with Microsoft. If the IPv4 address you used for your subinterface was a.b.c.d, the IP address of the BGP neighbor (Microsoft) will be a.b.c.d+1. The last octet of the BGP neighbor's IPv4 address will always be an even number. Follow ER ASN requirements for the peering.

router bgp 65021

bgp router-id 10.6.32.241

bgp log-neighbor-changes

!

address-family ipv4 vrf C10

neighbor 216.221.237.34 remote-as 12076

neighbor 216.221.237.34 description Microsoft peering to Azure

neighbor 216.221.237.34 local-as 394749

neighbor 216.221.237.34 activate

neighbor 216.221.237.34 password A1B2C3D4

neighbor 216.221.237.34 soft-reconfiguration inbound

redistribute connected

exit-address-family

!


neighbor 172.16.0.2 remote-as 12076

neighbor 172.16.0.2 description private peering to Azure

neighbor 172.16.0.2 local-as 64512

neighbor 172.16.0.2 activate

neighbor 172.16.0.2 password A1B2C3D4

neighbor 172.16.0.2 soft-reconfiguration inbound




12

exit-address-family

Note: password configuration is an optional feature for the ER BGP peering and not enabled by default. See BGP Command Reference for more information to set up a password on the BGP peering.

Advertise Prefixes Over the BGP Session to Azure

Use network statement or redistribution from IGP to advertise your internal network prefixes to Azure.

router bgp 65021

!


network 192.168.0.4 mask 255.255.255.252


redistribute static

Microsoft peering does not accept default route or private IP addresses (RFC 1918), the sample below use prefix-list to filter them out.

router bgp 65021

!


neighbor 216.221.237.34 prefix-list rfc1918 out

!

ip prefix-list rfc1918 deny 0.0.0.0/8 le 32








ip prefix-list rfc1918 deny 0.0.0.0/0

ip prefix-list rfc1918 permit 0.0.0.0/0 le 32

Microsoft Azure has policy of accepting up to 4000 (10,000 for Premium ExpressRoute) route prefixes for private peering and 200 route prefixes for Microsoft peering. It is your responsibility to manage and aggregate network prefix while advertising your internal network, otherwise Microsoft will drop the BGP session once prefix count goes above the limit.

Filter Prefixes Received from Azure (Optional) You can use route-maps and prefix lists to filter prefixes propagated into your network. You can use the sample below to accomplish the task. Ensure that you have appropriate prefix lists setup.

router bgp 65021

!


https://www.cisco.com/c/en/us/td/docs/ios/iproute_bgp/command/reference/irg_book/irg_bgp3.html#wp1108507


13

neighbor 216.221.237.34 route-map <MS_Prefixes_Inbound> in


neighbor 172.16.0.2 route-map <PP_Prefixes_Inbound> in

!

route-map <PP_Prefixes_Inbound> permit 10

match ip address prefix-list <PP_Prefixes>

!

route-map <MS_Prefixes_Inbound> permit 10

match ip address prefix-list <MS_Prefixes>

High Availability and Optimize Routing Configuration We recommend that both ASR1000 routers have L3 peering to south bound corporate network router so that customers can leverage High Availability or Equal Cost Multi-Path to load share traffic to ExpressRoute Follow ER Optimize Routing from customer to Microsoft, BGP local preference is used to influence the routing. Make sure you have the correct BGP community for region, e.g. USW is 12076:51006 and USW2 is12076:51026. A detailed list of region to ER BGP communities can be found here under “Support for BGP Communities” section. The sample below use BGP community “12076:51004” for the prefixes received from US East, and BGP community “12076:51006” for the prefixes received from US West. We will assign US West region, e.g. 13.100.0.0/16, to higher local preference in the US West, and assign US East region, e.g. 23.100.0.0/16, to higher local preference in the US East.

#US West ASR1000

!

router bgp 65021

!


neighbor 216.221.237.34 route-map Peer-USW in

!

ip bgp-community new-format

!

ip community-list 1 permit 12076:51006

!

route-map Peer-USW permit 10

match community 1

set local-preference 400

#US East ASR1000

!

router bgp 65021

!


neighbor 216.221.237.34 route-map Peer-USE in

!

ip bgp-community new-format

!

ip community-list 1 permit 12076:51004

!

route-map Peer-USE permit 10

match community 1




14

set local-preference 400

AS Path Prepending to Influence Routing In order to optimize routing from Microsoft to your network, AS Path prepending is used to influence routing. Microsoft removes private AS numbers in the AS PATH for the prefixes received on Microsoft Peering, so it is important to append public AS numbers in the AS PATH to influence routing for Microsoft Peering. The sample below did not follow the AS and IP scheme in Table 2, but based on the Microsoft ER example as shown in Figure 4.

Figure 4: AS Path Prepending Sample

You can lengthen the AS PATH for 177.2.0.0/31 in US East so that Microsoft will prefer the ExpressRoute circuit in US West for traffic destined for this prefix (as Microsoft network will think the path to this prefix is shorter in the west). Similarly, by lengthening the AS PATH for 177.2.0.2/31 in US West so that Microsoft will prefer the ExpressRoute circuit in US East.

#US West ASR1000

!

router bgp 345

!


neighbor 216.221.237.34 route-map Prepend-USW out

network 177.2.0.0 mask 255.255.255.254

network 177.2.0.2 mask 255.255.255.254

!

ip prefix-list prefix_USW seq 10 permit 177.2.0.2/31



15

!

route-map Prepend-USW permit 10

match ip address prefix prefix_USW

set as-path prepend 345

!

route-map Prepend-USW permit 20

#US East ASR1000

!

router bgp 345

!


neighbor 216.221.237.134 route-map Prepend-USE out

network 177.2.0.0 mask 255.255.255.254

network 177.2.0.2 mask 255.255.255.254

!

ip prefix-list prefix_USE seq 10 permit 177.2.0.0/31

!

route-map Prepend-USE permit 10

match ip address prefix prefix_USE

set as-path prepend 345

!

route-map Prepend-USE permit 20

Avoid Asymmetric Routing

Follow ER asymmetric routing solutions, in the example, if you want to use the Internet for authentication traffic and ExpressRoute for your mail traffic or other public services, you should not advertise your Active Directory Federation Services (AD FS) public IP addresses over ExpressRoute. This best practice can be enforced with an outbound route-map configuration:

router bgp 65021

!


neighbor 216.221.237.34 route-map AD_FS_Prefixes out

!

ip prefix-list AD_FS permit 121.10.0.1/32

!

route-map AD_FS_Prefixes deny 10

match ip address prefix-list AD_FS

route-map AD_FS_Prefixes permit 20

NAT Configuration

As per Microsoft NAT for ExpressRoute, Microsoft expects to support bi-directional connectivity on the Microsoft peering. Traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they enter the Microsoft network. You can use the sample configuration below to accomplish the task, it is using the MS peering subinterface ip address as

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-asymmetric-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-nat


16

the NAT pool (216.221.237.33), so the returning traffic will be sent back to this router, un-NATed before forwarded out of the DMZ VLAN.





ip address 216.221.237.33 255.255.255.252

ip nat outside

!





ip address 192.168.0.1 255.255.255.252

ip nat inside

!

ip route vrf C10 216.221.236.33 255.255.255.255 null0

!

ip nat pool Cust10_MSFT_Pool 216.221.236.33 216.221.236.33 netmask

255.255.255.252

!

ip nat inside source route-map Cust10_MSFT_sNAT pool Cust10_MSFT_Pool

vrf C10 overload

!

ip access-list extended Local_BGP_C10

remark deny BGP session from being NATed

permit tcp host 216.221.237.33 host 216.221.237.34 eq bgp

permit tcp host 216.221.237.34 host 216.221.237.33 eq bgp

!

access-list 10 permit 216.221.237.34

!

route-map Cust10_MSFT_sNAT deny 5

match ip address Local_BGP_C10

!

route-map Cust10_MSFT_sNAT permit 10

description NAT any traffic in VRF C10 with NH 216.221.237.34 toward

Microsoft Peering

match ip next-hop 10

It is your responsibility to ensure that the NAT IP pool advertised to Microsoft is NOT advertised to the Internet (even as a subnet of the Internet advertisement, they must be completely non-overlapping). Failure to meet this requirement may break connectivity to other Microsoft services.

NAT Common Best Practices

1. Set the NAT max-entries per system scale, which is 2M on ASR1001-X and ASR1001-HX. Other ASR1000 systems may have different NAT scale, please follow the relevant product datasheet.

ip nat translation max-entries 2000000

http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731640.html


17

2. It is recommended to keep the default NAT timeout. If the user has specific needs to reduce the timer, for example the pools are being exhausted, then the user can refer to the sample commands below to make configuration changes:

The default NAT timeout values can be seen in show command

ASR1000#show platform hardware qfp active feature nat data time

Timeouts: default 86400; TCP 86400; TCP PPTP 86400; UDP 300; FINRST 60;

SYN 60; DNS 60; ICMP 60; Skinny 60; ICMP error 60; ESP 300

To change the timeout values for example:

ip nat translation tcp-timeout 10800

3. If there is the requirement that both NAT and non-NATted traffic must co-exist in the

NAT outside interface, then use Gatekeeper to optimize system performance:

ip nat settings gatekeeper-size 65535

Route Redistribution into EIGRP In order to redistribute routes from the Private and Microsoft BGP Peerings to EIGRP, add the following configuration

router eigrp 1

!


redistribute static route-map BGP_Private_to_App_EIGRP

redistribute bgp 65021 metric 1000000 100 255 1 1500

network 10.0.0.0 0.0.0.255

no auto-summary

autonomous-system 2

exit-address-family

!


redistribute bgp 65021 metric 1000000 100 255 1 1500

network 10.1.0.0 0.0.0.255

no auto-summary

autonomous-system 3

!

router bgp 65021

!


redistribute eigrp 2 route-map EIGRP_App_to_BGP

!

ip prefix-list BGP_Private_to_App_EIGRP seq 5 permit 10.3.0.0/23

!

https://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/210869-ASR1k-NAT-intermittently-fails-to-transl.html


18

ip access-list extended EIGRP_App_to_BGP

permit ip 10.0.0.0 0.0.0.255 any

!

route-map EIGRP_App_to_BGP permit 10

match ip address EIGRP_App_to_BGP

!

route-map BGP_Private_to_App_EIGRP permit 10

match ip address prefix-list BGP_Private_to_App_EIGRP

!

In order to NAT traffic from your corporate network, adjust the NAT configuration as follows

access-list 11 permit 10.1.0.0 0.0.0.255

route-map Cust10_MSFT_sNAT permit 10

description NAT any traffic in Corp_NET toward public peering

match ip address 11

Value-Added Feature Configurations

Configure Flexible Netflow

Flexible Netflow (FNF) is an embedded instrumentation capability within the ASR1000 to characterize network operation, to characterize IP traffic, and understand how and where it flows is critical for network availability, performance, and troubleshooting. The sample below shows how simple it can be to turn on FNF for ASR1000.

flow exporter C10_expo

destination 10.10.10.9 vrf C101

transport udp 9996

!

flow monitor C10_mon

exporter C10_expo

record netflow-original

!



ip flow monitor C10_mon input

ip flow monitor C10_mon output

!



ip flow monitor C10_mon input

ip flow monitor C10_mon output

To be able to see bi-directional traffic in the ASR1000 system, the user can turn on ingress NetFlow on all interfaces, or if the user is only interested in the bi-directional traffic from and to


19

ER, turn on ingress and egress NetFlow on ER. We recommend the use of full NetFlow instead of sampled NetFlow.

Configure Quality of Service

Follow ER QoS requirements, a 6-class QoS model, as shown in Table 4, can be implemented to fulfill the requirements while protecting the mission critical applications and network control traffic in the events of ER circuits congestion. Use the sample QoS configuration below to accomplish the task.

Table 3: 6-Class QoS Model

Traffic Class DSCP Values

Business workload Bandwidth % Congestion avoidance

Voice EF Skype / Lync voice 10 (PQ) -

Video AF41 Interactive Video, VBSS

30 remaining WRED

Network Control

CS6 NET-CTRL* 5 remaining -

Transactional Data

AF21 App Sharing 25 remaining WRED

Bulk Data AF11 File Transfer 25 remaining WRED

Class-default Catch-all Catch-all 15 remaining WRED

Note: BGP is always marked as CS6 by ASR1000 so it is protected in the NET-CTRL class.

class-map match-any VOICE

match dscp ef

class-map match-any VIDEO

match dscp af41

class-map match-any NETWORK-CONTROL

match dscp cs6

class-map match-any TRANSACTIONAL-DATA

match dscp af21

class-map match-any BULK-DATA

match dscp af11

!

! example of 500Mbps of ER circuit, adapt it to your circuit BW

accordingly.

policy-map ER-500MBPS-POLICY

class class-default

shape average 500000000

service-policy ER

!

policy-map ER

class VOICE

priority level 1

police cir percent 10

class VIDEO

bandwidth remaining percent 30

random-detect

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-qos


20

class NETWORK-CONTROL


class TRANSACTIONAL-DATA


random-detect

class BULK-DATA


random-detect

class class-default


random-detect

set dscp 0

! Microsoft require user to rewrite all other DSCP to 0 before sending

the packets to ER

!



service-policy output ER-500MBPS-POLICY

Advanced Services Configurations

Configure Application Visibility and Control (AVC) If the DSCP values for applications above have not been marked properly or not preserved in your network before reaching the ASR1000, use the Solution Reference Network Designs (SRND) policy model to simply application classification in NBAR, and mark the application to the DSCP specified by Microsoft.

class-map match-all VOICE

match protocol attribute traffic-class voip-telephony

match protocol attribute business-relevance business-relevant

class-map match-all BROADCAST-VIDEO

match protocol attribute traffic-class broadcast-video


class-map match-all INTERACTIVE-VIDEO

match protocol attribute traffic-class real-time-interactive


class-map match-all MULTIMEDIA-CONFERENCING

match protocol attribute traffic-class multimedia-conferencing


class-map match-all MULTIMEDIA-STREAMING

match protocol attribute traffic-class multimedia-streaming


class-map match-all SIGNALING

match protocol attribute traffic-class signaling


class-map match-all NETWORK-CONTROL

match protocol attribute traffic-class network-control


class-map match-all NETWORK-MANAGEMENT

match protocol attribute traffic-class ops-admin-mgmt


https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-3s/qos-nbar-xe-3s-book/clsfy-traffic-nbar.html#GUID-D4B22457-1195-4433-B68C-4CC9C96278E7

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-3s/qos-nbar-xe-3s-book.html


21

class-map match-all TRANSACTIONAL-DATA

match protocol attribute traffic-class transactional-data


class-map match-all BULK-DATA

match protocol attribute traffic-class bulk-data


class-map match-all SCAVENGER

match protocol attribute business-relevance business-irrelevant

!

policy-map MARKING

class VOICE

set dscp ef

class BROADCAST-VIDEO

set dscp af41

class INTERACTIVE-VIDEO

set dscp af41

class MULTIMEDIA-CONFERENCING

set dscp af41

class MULTIMEDIA-STREAMING

set dscp af41

class SIGNALING

set dscp af41

class NETWORK-CONTROL

set dscp cs6

class NETWORK-MANAGEMENT

set dscp default

class TRANSACTIONAL-DATA

set dscp af21

class BULK-DATA

set dscp af11

class SCAVENGER

set dscp default

class class-default

set dscp default

!



service-policy input MARKING

!

Configure IPsec VPN

A common use utilizes the Cisco Cloud Services Router, CSR1000v, deployed as an application VNet gateway in Azure to provide IPsec gateway for entire VNet. See Extending Enterprise Network into Public Cloud with Cisco CSR1000v. The ASR1000 connecting to ER is the ideal on-premises gateway for the IPsec tunnel termination in Enterprise network as the platform delivers embedded hardware acceleration for IPsec VPN. For details on ASR1000 system IPsec throughput, refer to the relevant product datasheet.

https://clnv.s3.amazonaws.com/2017/usa/pdf/BRKARC-2749.pdf

https://clnv.s3.amazonaws.com/2017/usa/pdf/BRKARC-2749.pdf



22

The Cisco CSR1000v is ASR1000 in virtual form factor. They run the same IOS XE software release, inherit the same IOS XE software architecture, support the same CLIs and feature sets of IPsec VPN. Once you have deployed CSR1000v on Microsoft Azure, you would configure the IPsec VPN on the CSR1000v by using the step-by-step procedure outlined in this video demo or as per the sample:

crypto isakmp policy 200

encryption aes

authentication pre-share

group 2

lifetime 28800

!

crypto isakmp key cisco123 address 0.0.0.0

crypto isakmp keepalive 10 10

!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set csr esp-aes esp-sha-hmac

mode tunnel

crypto ipsec df-bit clear

!

crypto ipsec profile csr

set transform-set csr

!

interface Tunnel1

ip address 192.168.100.2 255.255.255.252

tunnel source GigabitEthernet1

tunnel mode ipsec ipv4

tunnel destination 172.16.0.1

tunnel protection ipsec profile csr

You should have the IPsec tunnel peer configuration on the ASR1000 enabled as per the sample:

crypto isakmp policy 200

encryption aes

authentication pre-share

group 2

lifetime 28800

!

crypto isakmp key cisco123 address 0.0.0.0

crypto isakmp keepalive 10 10

!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set csr esp-aes esp-sha-hmac

mode tunnel

crypto ipsec df-bit clear

!

crypto ipsec profile csr1

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/b_CSR1000v_Configuration_Guide/b_CSR1000v_Configuration_Guide_chapter_00.html#con_1092033

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure.html

https://www.youtube.com/watch?v=U2-lc8oewhA


23

set transform-set csr

!

interface Tunnel1


ip address 192.168.100.1 255.255.255.252

tunnel source TenGigabitEthernet0/1/0.3101

tunnel mode ipsec ipv4

tunnel destination 10.0.0.4

tunnel protection ipsec profile csr1

Test Connectivity

While there are steps to verify ExpressRoute connectivity with Microsoft, there are also verification steps can be performed on ASR1000 and in the customer on-premises network.

Verify the BGP Neighbors Use the following commands to verify the Microsoft peering and Private BGP peering are established and Up

ASR1000#show ip bgp vpnv4 vrf C10 neighbor 216.221.237.34

BGP neighbor is 216.221.237.34, vrf C10, remote AS 12076, local AS

394749, external link

Description: Microsoft peering to Azure

BGP version 4, remote router ID 207.46.160.94

BGP state = Established, up for 00:39:52

Last read 00:00:16, last write 00:00:39, hold time is 180, keepalive

interval is 60 seconds

Neighbor sessions:

1 active, is not multisession capable (disabled)

Neighbor capabilities:

Route refresh: advertised and received(new)

Four-octets ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Enhanced Refresh Capability: advertised and received

Multisession Capability:

Stateful switchover support enabled: NO for session 1

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-troubleshooting-expressroute-overview


24

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 1 1

Notifications: 0 0

Updates: 2 3

Keepalives: 45 45

Route Refresh: 0 0

Total: 48 49

Do log neighbor state changes (via global configuration)

Default minimum time between advertisement runs is 0 seconds

For address family: VPNv4 Unicast

Translates address family IPv4 Unicast for VRF C10

Session: 216.221.237.34

BGP table version 1326, neighbor version 1326/0

Output queue size : 0

Index 17, Advertise bit 1

17 update-group member

Inbound soft reconfiguration allowed

Outbound path policy configured

Outgoing update prefix filter list is rfc1918

Route map for outgoing advertisements is AD_FS_Prefixes

Slow-peer detection is disabled

Slow-peer split-update-group dynamic is disabled

Sent Rcvd


25

Prefix activity: ---- ----

Prefixes Current: 1 144 (Consumes 19584 bytes)

Prefixes Total: 1 144

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 144

Used as multipath: n/a 0

Used as secondary: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

prefix-list 3 0

Bestpath from this peer: 144 n/a

Total: 147 0

Number of NLRIs in the update sent: max 73, min 0

Last detected as dynamic slow peer: never

Dynamic slow peer recovered: never

Refresh Epoch: 1

Last Sent Refresh Start-of-rib: never

Last Sent Refresh End-of-rib: never

Last Received Refresh Start-of-rib: never

Last Received Refresh End-of-rib: never

Sent Rcvd

Refresh activity: ---- ----

Refresh Start-of-RIB 0 0

Refresh End-of-RIB 0 0


26

Address tracking is enabled, the RIB does have a route to

216.221.237.34

Route to peer address reachability Up: 4; Down: 1

Last notification 03:14:13

Connections established 5; dropped 4

Last reset 00:42:26, due to BGP Notification received, Connection

Collision Resolution

External BGP neighbor configured for connected checks (single-hop no-

disable-connected-check)

Interface associated: TenGigabitEthernet0/1/0.3101 (peering address

in same link)

Transport(tcp) path-mtu-discovery is enabled

Graceful-Restart is disabled

SSO is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 216.221.237.33, Local port: 48945

Foreign host: 216.221.237.34, Foreign port: 179

Connection tableid (VRF): 2

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x153EE19F):

Timer Starts Wakeups Next

Retrans 47 0 0x0

TimeWait 0 0 0x0

AckHold 46 42 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0


27

GiveUp 0 0 0x0

PmtuAger 1521 1520 0x153EE253

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0

iss: 2713507505 snduna: 2713508500 sndnxt: 2713508500

irs: 120760723 rcvnxt: 120762358

sndwnd: 15390 scale: 0 maxrcvwnd: 16384

rcvwnd: 16213 scale: 0 delrcvwnd: 171

SRTT: 998 ms, RTTO: 1014 ms, RTV: 16 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms

uptime: 2392902 ms, Sent idletime: 16537 ms, Receive idletime: 16737 ms

Status Flags: active open

Option Flags: VRF id set, nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 93 (out of order: 0), with data: 47, total data bytes: 1634

Sent: 94 (retransmit: 0, fastretransmit: 0, partialack: 0, Second

Congestion: 0), with data: 47, total data bytes: 994

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore 0x7FAA555FB670 FREE


28

ASR1000#show ip bgp vpnv4 vrf C10 neighbor 172.16.0.2

BGP neighbor is 172.16.0.2, vrf C10, remote AS 12076, local AS

64512, external link

Description: private peering to Azure

BGP version 4, remote router ID 207.46.160.94

BGP state = Established, up for 4d01h

Last read 00:00:19, last write 00:00:03, hold time is 180, keepalive

interval is 60 seconds

Neighbor sessions:

1 active, is not multisession capable (disabled)

Neighbor capabilities:

Route refresh: advertised and received(new)

Four-octets ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Enhanced Refresh Capability: advertised and received

Multisession Capability:

Stateful switchover support enabled: NO for session 1

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 1 1

Notifications: 0 0

Updates: 43 2

Keepalives: 6410 6400

Route Refresh: 0 0

Total: 6456 6403

Do log neighbor state changes (via global configuration)


29

Default minimum time between advertisement runs is 0 seconds

For address family: VPNv4 Unicast

Translates address family IPv4 Unicast for VRF C10

Session: 172.16.0.2

BGP table version 1326, neighbor version 1326/0

Output queue size : 0

Index 16, Advertise bit 0

16 update-group member

Inbound soft reconfiguration allowed

Outbound path policy configured

Route map for outgoing advertisements is Prepend-USW

Slow-peer detection is disabled

Slow-peer split-update-group dynamic is disabled

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 1 1 (Consumes 136 bytes)

Prefixes Total: 1 1

Implicit Withdraw: 0 0

Explicit Withdraw: 147 0

Used as bestpath: n/a 1

Used as multipath: n/a 0

Used as secondary: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

route-map: 0 7

Other Policies: 291 n/a


30

Total: 291 7

Number of NLRIs in the update sent: max 143, min 0

Last detected as dynamic slow peer: never

Dynamic slow peer recovered: never

Refresh Epoch: 1

Last Sent Refresh Start-of-rib: 04:36:39

Last Sent Refresh End-of-rib: 04:36:39

Refresh-Out took 0 seconds

Last Received Refresh Start-of-rib: never

Last Received Refresh End-of-rib: never

Sent Rcvd

Refresh activity: ---- ----

Refresh Start-of-RIB 1 0

Refresh End-of-RIB 1 0

Address tracking is enabled, the RIB does have a route to 172.16.0.2

Route to peer address reachability Up: 1; Down: 0

Last notification 4d01h

Connections established 1; dropped 0

Last reset never

External BGP neighbor configured for connected checks (single-hop no-

disable-connected-check)

Interface associated: TenGigabitEthernet0/1/0.3103 (peering address

in same link)

Transport(tcp) path-mtu-discovery is enabled

Graceful-Restart is disabled

SSO is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1


31

Local host: 172.16.0.1, Local port: 179

Foreign host: 172.16.0.2, Foreign port: 28211

Connection tableid (VRF): 2

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x1542D0A0):

Timer Starts Wakeups Next

Retrans 6431 0 0x0

TimeWait 0 0 0x0

AckHold 6401 6287 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 0 0 0x0

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0

iss: 763195641 snduna: 763324044 sndnxt: 763324044

irs: 1048104958 rcvnxt: 1048226686

sndwnd: 15054 scale: 0 maxrcvwnd: 16384

rcvwnd: 16099 scale: 0 delrcvwnd: 285

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms


32

uptime: 349616252 ms, Sent idletime: 3404 ms, Receive idletime: 3203 ms

Status Flags: passive open, gen tcbs

Option Flags: VRF id set, nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 12802 (out of order: 0), with data: 6402, total data bytes:

121727

Sent: 12808 (retransmit: 0, fastretransmit: 0, partialack: 0, Second

Congestion: 0), with data: 6435, total data bytes: 128402

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore 0x7FAA5454F230 FREE

BGP session is essential to maintain ER connectivity. To protect BGP packets in the ASR1000 punt path and mitigate potential DDoS attacks, it is recommended you implement Contol Plane Policing as per the Control Plane Policing template on page 50 -53.

Verify ExpressRoute Connectivity Follow the procedure here to verify ExpressRoute connectivity. The ExpressRoute circuit can be validated by using the Azure portal “Home > ExpressRoute circuit”, and looking at the “Essentials” field. If you see “Circuit status” is Enabled, then the ExpressRoute Circuit is up on the Microsoft side, and the “Provider status” as Provisioned, then the circuit is up on the service provider side, as shown in Figure 5.

http://d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKARC-2019.pdf

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-troubleshooting-expressroute-overview#validate-circuit-provisioning-and-state


33

Figure 5: Verify ExpressRoute Circuit Status in Azure Portal Snapshot

To further validate that the circuit is up from the customer side, click “Home > ExpressRoute circuit > Azure Private/Microsoft Private > Get route table summary” to see if your subinterface networks are reachable, as shown in Figure 6 and 7 respectively.

Figure 6: Verify Private Peering Customer Networks are Reachable in Azure Portal Snapshot


34

Figure 7: Verify Microsoft Peering Customer Networks are Reachable in Azure Portal Snapshot

Verify NAT Translation Entries and Pool Follow NAT monitoring and Maintaining guide to verify NAT translation entries are set up properly.

ASR1000#show ip nat translation

Pro Inside global Inside local Outside local

Outside global

icmp 216.221.236.33:98 192.168.0.1:98 216.221.237.34:98

216.221.237.34:98

Total number of translations: 1

To monitor the pool stats:

ASR1000#show platform software nat fp active pool

Dump NAT pool config

ID: 1, Name: Cust10_MSFT_Pool, Type: Generic, Mask: 255.255.255.252

Flags: Unknown, Acct name:

Address range blocks: 1

Start: 216.221.236.33, End: 216.221.236.33

Last stats update: 02/13 17:35:39.556

Last refcount value: 1

ASR1000#show platform software nat fp active pool-stats id <id>

NAT Pool Statistics

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-16/nat-xe-16-book/iadnat-monmain.html


35

Pool name Cust10_MSFT_Pool, id 1

Assigned Available

Addresses 0 1

UDP Low Ports 0 512

TCP Low Ports 0 512

UDP High Ports 0 64512

TCP High Ports 0 64512

(Low ports are less than 1024. High ports are greater than or equal to

1024.)

Verify Netflow Entries The ASR1000 exports the NetFlow cache entries directly from the Quantum Flow Processor to the external collector via in-band interface. Do NOT connect the collector on the management interface (GigabitEthernet0). Use the following command to verify the flow monitor is exporting data to the exporters.

ASR1000#show flow monitor C10_mon

Flow Monitor C10_mon:

Description: User defined

Flow Record: netflow-original

Flow Exporter: C10_expo

Cache:

Type: normal (Platform cache)

Status: allocated

Size: 200000 entries

Inactive Timeout: 15 secs

Active Timeout: 1800 secs

Trans end aging: off


36

Use the Top N talkers capability, which facilitates real-time traffic analysis of the most traffic volume consumers.

ASR1000#show flow monitor C10_mon cache sort counter packets top 3

format table

Processed 2 flows

Aggregated to 2 flows

Showing the top 2 flows

IPV4 SRC ADDR IPV4 DST ADDR TRNS SRC PORT TRNS DST PORT INTF

INPUT FLOW SAMPLER ID IP TOS IP PROT ip src as ip dst as

ipv4 next hop addr ipv4 src mask ipv4 dst mask tcp flags intf

output bytes pkts time first time last

=============== =============== ============= =============

==================== =============== ====== ======= =========

========= ================== ============= ============= =========

==================== ========== ========== ============

============

10.3.0.5 192.168.0.1 0 2048

Te0/1/0.3103 0 0x00 1 0

0 0.0.0.0 /0 /0 0x00 Null

91860 1531 17:16:36.049 17:42:19.371

192.168.0.1 10.3.0.5 0 0 Null

0 0x00 1 0 12076 172.16.0.2

/32 /23 0x00 Te0/1/0.3103 91620

1527 17:16:40.065 17:42:19.371

Note: ASR1000 does not support aggregate flows in Top N talkers.

ASR1000 Proactive System Monitoring Proactive monitoring system resources allows you to detect potential problems before they happen, thus avoiding outages. Figure 8 highlights key system resources to monitor on ASR1000.

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/configuration/15-mt/fnf-15-mt-book/cgf-topn.html#GUID-0340CBF8-EBAE-448D-B119-B345243C94C5


37

Figure 8: Key System Resources to Monitor - Summary

The system resources to be consumed by each of the features discussed in the configuration guide are listed in Table 5.

Table 5: Feature to System Resources Consumption

Features System Resources Consumed

BGP IOS memory/CPU, RP memory/CPU

FIB IOS memory/CPU, RP memory/CPU

NAT QFP, resource DRAM, TCAM

Netflow QFP, resource DRAM

QoS QFP, TCAM

AVC QFP, resource DRAM, TCAM

IPsec IOS memory/CPU, RP memory/CPU, QFP, Crypto Assist, TCAM

The best practice is that during steady state the system should have minimum 25% of IOS memory, RP memory, and resource DRAM available to accommodate potential network churning and reconvergence events; otherwise, you should plan to upgrade system memory or upgrade to a higher performance ASR1000 variant such as the ASR1002-HX. For exact CLIs and MIBs to monitor each system source, follow the Operating an ASR1000 guide page 24 - 37.

References Please refer to the following documentation for ASR1000 platform architecture, packet flow, feature configuration guide and datasheet:

http://d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKARC-2019.pdf


38

ASR1000 System Architecture Overview

BGP Configuration Guide

NAT Configuration Guide

QoS Configuration Guide

Flexible Netflow Configuration Guide

NBAR Configuration Guide

AVC Configuration Guide

Security for VPNs with IPsec

IPsec Virtual Tunnel Interface

ASR1000 Routers Datasheet

ASR1000-X Router Hardware Installation Guide

ASR1000-HX Router Hardware Installation Guide

ASR1000 ESP Datasheet

ASR1000 Ordering Guide

IOS-XE NGE Support Product Tech Note Refer to the following documentation for common error messages and troubleshooting notes:

Troubleshooting of ASR1k Made Easy

ASR1000 Troubleshooting TechNotes

ASR1000 Error and System Messages

Embedded Packet Capture for IOS-XE

https://clnv.s3.amazonaws.com/2017/eur/pdf/BRKARC-2001.pdf

https://clnv.s3.amazonaws.com/2017/eur/pdf/BRKARC-2001.pdf

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-16/irg-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-16/irg-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-16/nat-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-16/nat-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/xe-16/qos-plcshp-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/xe-16/qos-plcshp-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/configuration/xe-16/fnf-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/configuration/xe-16/fnf-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-16/qos-nbar-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-16/qos-nbar-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-cfg-vpn-ipsec.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-cfg-vpn-ipsec.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-ipsec-virt-tunnl.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-ipsec-virt-tunnl.html

https://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731632.html

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/install/guide/1001-x/asr1hig-book.html

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/install/guide/1001HX_1002HX/b_ASR1001HX-1002HX_HIG.html





http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/116055-technote-ios-crypto.html

http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/116055-technote-ios-crypto.html

http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKCRS-3147.pdf

http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKCRS-3147.pdf

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-tech-notes-list.html

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-tech-notes-list.html

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-system-message-guides-list.html

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-system-message-guides-list.html

http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-embedded-packet-capture/116045-productconfig-epc-00.html

http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-embedded-packet-capture/116045-productconfig-epc-00.html

cisco asr1000 and microsoft azure expressroute … · cisco asr1000 and microsoft azure...

Documents