expressroute overview: extend your on-premises network to azure

Table of ContentsTable of Contents

ExpressRoute Documentation Overview

What is ExpressRoute? ExpressRoute FAQ Connectivity models Circuits and routing domains Locations and partners

Providers by location Locations by provider

Virtual network gateways for ExpressRoute Get Started

Prerequisites Workflows Routing requirements QoS requirements About moving circuits from classic to Resource Manager

How To Create and modify a circuit

Azure portal Azure PowerShell Azure CLI

Create and modify peering configuration Azure portal Azure PowerShell Azure CLI

Link a virtual network to an ExpressRoute circuit Azure portal Azure PowerShell Azure CLI

Configure a site-to-site VPN over Microsoft peering Configure a virtual network gateway for ExpressRoute

Azure portal Azure PowerShell

Configure ExpressRoute and site-to-site coexisting connections Configure route filters for Microsoft peering

Azure portal Azure PowerShell Azure CLI

Move from public peering to Microsoft peering Move a circuit from classic to Resource Manager Migrate associated virtual networks from classic to Resource Manager Configure a router for ExpressRoute

Configure a router Router configuration samples for NAT

Configure Network Performance Monitor for ExpressRoute Classic deployment model articles

Modify a circuit Create and modify peering configuration Link a virtual network to an ExpressRoute circuit Configure ExpressRoute and S2S coexisting connections Add a gateway to a VNet

Best Practices Best practices for network security and cloud services Optimize routing Asymmetric routing NAT for ExpressRoute

Troubleshoot Verifying ExpressRoute connectivity Resolving network performance issues Reset a failed circuit Getting ARP tables

Getting ARP tables (Classic) Reference

Azure PowerShell Azure CLI REST REST (classic)

Related Virtual Network VPN Gateway Virtual Machines Load Balancer Traffic Manager

Resources Azure Roadmap Case Studies Networking Blog Pricing Pricing calculator Service updates SLA Subscription and Service Limits ExpressRoute for Cloud Solution Providers (CSP) Videos

Connect a virtual network gateway to a circuit Create a virtual network for ExpressRoute Create a virtual network gateway for ExpressRoute Create an ExpressRoute circuit Evolve your network infrastructure for connectivity How to set up private peering for your circuit Hybrid partnerships: Enabling on-premises scenarios Set up Microsoft peering for your circuit Set up public peering for your circuit

https://docs.microsoft.com/powershell/module/azurerm.network/

https://docs.microsoft.com/cli/azure/network/express-route

https://msdn.microsoft.com/library/azure/mt586720

https://msdn.microsoft.com/library/azure/dn606310

https://docs.microsoft.com/azure/virtual-network/

https://docs.microsoft.com/azure/vpn-gateway/

https://docs.microsoft.com/azure/virtual-machines/

https://docs.microsoft.com/azure/load-balancer/

https://docs.microsoft.com/azure/traffic-manager/

https://azure.microsoft.com/roadmap/

https://customers.microsoft.com/Pages/advancedsearch.aspx

https://azure.microsoft.com/blog/topics/networking/

https://azure.microsoft.com/pricing/details/expressroute/

https://azure.microsoft.com/pricing/calculator/

https://azure.microsoft.com/updates/

https://azure.microsoft.com/support/legal/sla/

https://azure.microsoft.com/documentation/videos/index/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-a-connection-between-your-vpn-gateway-and-expressroute-circuit/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-a-virtual-network/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-a-vpn-gateway-for-your-virtual-network/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-an-expressroute-circuit/

https://go.microsoft.com/fwlink/p/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-set-up-azure-private-peering-for-your-expressroute-circuit/

https://go.microsoft.com/fwlink/p/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-set-up-microsoft-peering-for-your-expressroute-circuit/

https://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-set-up-azure-public-peering-for-your-expressroute-circuit/

Reference

Learn how to use ExpressRoute to set up a fast, private connection to Microsoft cloud services from your on-premisesinfrastructure or colocation facility. Tutorials, REST APIs, and other documentation help you configure and manage anExpressRoute circuit.

Learn About Express RouteLearn About Express Route

Azure Express Route Video LibraryAzure Express Route Video Library

Get started with ExpressRouteGet started with ExpressRoute

Command-LineCommand-Line

RESTREST

PowerShell

Azure CLI

REST API Reference

Classic REST API Reference

https://docs.microsoft.com/azure/expressroute/expressroute-introduction

https://azure.microsoft.com/documentation/videos/index/?services=expressroute

https://docs.microsoft.com/azure/expressroute/expressroute-prerequisites

https://docs.microsoft.com/powershell/module/azurerm.network/#expressroute

https://docs.microsoft.com/cli/azure/network/express-route

https://msdn.microsoft.com/library/azure/mt586720

https://msdn.microsoft.com/library/azure/dn606310

ExpressRoute overview3/12/2018 • 5 min to read • Edit Online

Key benefits

FeaturesLayer 3 connectivityLayer 3 connectivity

Microsoft Azure ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a privateconnection facilitated by a connectivity provider. With ExpressRoute, you can establish connections to Microsoftcloud services, such as Microsoft Azure, Office 365, and Dynamics 365.

Connectivity can be from an any-to-any (IP VPN) network, a point-to-point Ethernet network, or a virtual cross-connection through a connectivity provider at a co-location facility. ExpressRoute connections do not go over thepublic Internet. This lets ExpressRoute connections offer more reliability, faster speeds, lower latencies, and highersecurity than typical connections over the Internet. For information on how to connect your network to Microsoftusing ExpressRoute, see ExpressRoute connectivity models.

Layer 3 connectivity between your on-premises network and the Microsoft Cloud through a connectivityprovider. Connectivity can be from an any-to-any (IPVPN) network, a point-to-point Ethernet connection, orthrough a virtual cross-connection via an Ethernet exchange.Connectivity to Microsoft cloud services across all regions in the geopolitical region.Global connectivity to Microsoft services across all regions with ExpressRoute premium add-on.Dynamic routing between your network and Microsoft over industry standard protocols (BGP).Built-in redundancy in every peering location for higher reliability.Connection uptime SL A.QoS support for Skype for Business.

For more information, see the ExpressRoute FAQ.

Microsoft uses industry standard dynamic routing protocol (BGP) to exchange routes between your on-premisesnetwork, your instances in Azure, and Microsoft public addresses. We establish multiple BGP sessions with yournetwork for different traffic profiles. More details can be found in the ExpressRoute circuit and routing domains

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-introduction.md

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-connectivity-models


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-faqs

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-circuit-peerings

RedundancyRedundancy

Connectivity to Microsoft cloud servicesConnectivity to Microsoft cloud services

NOTENOTE

Connectivity to all regions within a geopolitical regionConnectivity to all regions within a geopolitical region

Global connectivity with ExpressRoute premium add-onGlobal connectivity with ExpressRoute premium add-on

Rich connectivity partner ecosystemRich connectivity partner ecosystem

Connectivity to national cloudsConnectivity to national clouds

Bandwidth optionsBandwidth options

article.

Each ExpressRoute circuit consists of two connections to two Microsoft Enterprise edge routers (MSEEs) from theconnectivity provider/your network edge. Microsoft requires dual BGP connection from the connectivityprovider/your side â€“ one to each MSEE. You may choose not to deploy redundant devices/Ethernet circuits atyour end. However, connectivity providers use redundant devices to ensure that your connections are handed offto Microsoft in a redundant manner. A redundant Layer 3 connectivity configuration is a requirement for our SL Ato be valid.

ExpressRoute connections enable access to the following services:

Microsoft Azure servicesMicrosoft Office 365 servicesMicrosoft Dynamics 365

Software as a Service offerings, like Office 365 and Dynamics 365, were created to be accessed securely and reliably via theInternet. Because of this, we recommend ExpressRoute for these applications only for specific scenarios. For informationabout using ExpressRoute to access Office 365, visit Azure ExpressRoute for Office 365.

For a detailed list of services supported over ExpressRoute, visit the ExpressRoute FAQ page

You can connect to Microsoft in one of our peering locations and have access to all regions within the geopoliticalregion.

For example, if you connected to Microsoft in Amsterdam through ExpressRoute, you have access to all Microsoftcloud services hosted in Northern Europe and Western Europe. For an overview of the geopolitical regions, theassociated Microsoft cloud regions, and corresponding ExpressRoute peering locations, see the ExpressRoutepartners and peering locations article.

You can enable the ExpressRoute premium add-on feature to extend connectivity across geopolitical boundaries.For example, if you are connected to Microsoft in Amsterdam through ExpressRoute, you will have access to allMicrosoft cloud services hosted in all regions across the world (national clouds are excluded). You can accessservices deployed in South America or Australia the same way you access the North and West Europe regions.

ExpressRoute has a constantly growing ecosystem of connectivity providers and SI partners. For the latestinformation, refer to the ExpressRoute providers and locations article.

Microsoft operates isolated cloud environments for special geopolitical regions and customer segments. Refer tothe ExpressRoute providers and locations page for a list of national clouds and providers.

You can purchase ExpressRoute circuits for a wide range of bandwidths. Supported bandwidths are listed below.Be sure to check with your connectivity provider to determine the list of supported bandwidths they provide.

50 Mbps100 Mbps


http://aka.ms/ExpressRouteOffice365


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-locations




Dynamic scaling of bandwidthDynamic scaling of bandwidth

Flexible billing modelsFlexible billing models

FAQ

Next steps

200 Mbps500 Mbps1 Gbps2 Gbps5 Gbps10 Gbps

You can increase the ExpressRoute circuit bandwidth (on a best effort basis) without having to tear down yourconnections.

You can pick a billing model that works best for you. Choose between the billing models listed below. For moreinformation, see the ExpressRoute FAQ.

Unlimited data. The ExpressRoute circuit is charged based on a monthly fee, and all inbound and outbounddata transfer is included free of charge.Metered data. The ExpressRoute circuit is charged based on a monthly fee. All inbound data transfer is free ofcharge. Outbound data transfer is charged per GB of data transfer. Data transfer rates vary by region.ExpressRoute premium add-on. The ExpressRoute premium is an add-on over the ExpressRoute circuit. TheExpressRoute premium add-on provides the following capabilities:

Increased route limits for Azure public and Azure private peering from 4,000 routes to 10,000 routes.Global connectivity for services. An ExpressRoute circuit created in any region (excluding nationalclouds) will have access to resources across any other region in the world. For example, a virtual networkcreated in West Europe can be accessed through an ExpressRoute circuit provisioned in Silicon Valley.Increased number of VNet links per ExpressRoute circuit from 10 to a larger limit, depending on thebandwidth of the circuit.

For frequently asked questions about ExpressRoute, see the ExpressRoute FAQ.

Learn about ExpressRoute connectivity models.Learn about ExpressRoute connections and routing domains. See ExpressRoute circuits and routing domains.Find a service provider. See ExpressRoute partners and peering locations.Ensure that all prerequisites are met. See ExpressRoute prerequisites.Refer to the requirements for Routing, NAT, and QoS.Configure your ExpressRoute connection.

Learn about some of the other key networking capabilities of Azure.

Create an ExpressRoute circuitConfigure peering for an ExpressRoute circuitConnect a virtual network to an ExpressRoute circuit



https://docs.microsoft.com/en-us/azure/expressroute/expressroute-connectivity-models

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-circuit-peerings


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-prerequisites

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-nat

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-qos

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-routing-portal-resource-manager

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-linkvnet-portal-resource-manager

https://docs.microsoft.com/en-us/azure/networking/networking-overview

ExpressRoute FAQ4/9/2018 • 18 min to read • Edit Online

What is ExpressRoute?

What are the benefits of using ExpressRoute and private network connections?What are the benefits of using ExpressRoute and private network connections?

Where is the service available?Where is the service available?

How can I use ExpressRoute to connect to Microsoft if I donâ€™t have partnerships with one of theHow can I use ExpressRoute to connect to Microsoft if I donâ€™t have partnerships with one of theExpressRoute-carrier partners?ExpressRoute-carrier partners?

How much does ExpressRoute cost?How much does ExpressRoute cost?

If I pay for an ExpressRoute circuit of a given bandwidth, does the VPN connection I purchase from myIf I pay for an ExpressRoute circuit of a given bandwidth, does the VPN connection I purchase from mynetwork service provider have to be the same speed?network service provider have to be the same speed?

If I pay for an ExpressRoute circuit of a given bandwidth, do I have the ability to burst up to higher speeds ifIf I pay for an ExpressRoute circuit of a given bandwidth, do I have the ability to burst up to higher speeds ifnecessary?necessary?

Can I use the same private network connection with virtual network and other Azure servicesCan I use the same private network connection with virtual network and other Azure servicessimultaneously?simultaneously?

Does ExpressRoute offer a Service Level Agreement (SLA)?Does ExpressRoute offer a Service Level Agreement (SLA)?

Supported services

ExpressRoute is an Azure service that lets you create private connections between Microsoft datacenters andinfrastructure thatâ€™s on your premises or in a colocation facility. ExpressRoute connections do not go overthe public Internet, and offer higher security, reliability, and speeds with lower latencies than typicalconnections over the Internet.

ExpressRoute connections do not go over the public Internet. They offer higher security, reliability, and speeds,with lower and consistent latencies than typical connections over the Internet. In some cases, usingExpressRoute connections to transfer data between on-premises devices and Azure can yield significant costbenefits.

See this page for service location and availability: ExpressRoute partners and locations.

You can select a regional carrier and land Ethernet connections to one of the supported exchange providerlocations. You can then peer with Microsoft at the provider location. Check the last section of ExpressRoutepartners and locations to see if your service provider is present in any of the exchange locations. You can thenorder an ExpressRoute circuit through the service provider to connect to Azure.

Check pricing details for pricing information.

No. You can purchase a VPN connection of any speed from your service provider. However, your connection toAzure is limited to the ExpressRoute circuit bandwidth that you purchase.

Yes. ExpressRoute circuits are configured to allow you to burst up to two times the bandwidth limit youprocured for no additional cost. Check with your service provider to see if they support this capability.

Yes. An ExpressRoute circuit, once set up, allows you to access services within a virtual network and otherAzure services simultaneously. You connect to virtual networks over the private peering path, and to otherservices over the public peering path.

For information, see the ExpressRoute SL A page.

ExpressRoute supports three routing domains for various types of services.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-faqs.md



Private peeringPrivate peering

Public peeringPublic peering

NOTENOTE

Microsoft peeringMicrosoft peering

Data and connectionsAre there limits on the amount of data that I can transfer using ExpressRoute?Are there limits on the amount of data that I can transfer using ExpressRoute?

What connection speeds are supported by ExpressRoute?What connection speeds are supported by ExpressRoute?

Which service providers are available?Which service providers are available?

Virtual networks, including all virtual machines and cloud services

Microsoft peering is the preferred way to access all services hosted on Azure.

Power BIDynamics 365 for Finance and Operations (formerly known as Dynamics AX Online)Most of the Azure services are supported. Please check directly with the service that you want to use toverify support.The following services are NOT supported:

CDNVisual Studio Team Services Load TestingMulti-factor AuthenticationTraffic Manager

Office 365Dynamics 365 Customer Engagement applications (formerly known as CRM Online)

Using route filters, you get access to the same public services with the Microsoft peering :

Dynamics 365 for SalesDynamics 365 for Customer ServiceDynamics 365 for Field ServiceDynamics 365 for Project Service

Power BIDynamics 365 for Finance and OperationsMost of the Azure services are supported. Please check directly with the service that you want to useto verify support.The following services are NOT supported:

CDNVisual Studio Team Services Load TestingMulti-factor AuthenticationTraffic Manager

We do not set a limit on the amount of data transfer. Refer to pricing details for information on bandwidthrates.

Supported bandwidth offers:

50 Mbps, 100 Mbps, 200 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, 10 Gbps

See ExpressRoute partners and locations for the list of service providers and locations.



Technical detailsWhat are the technical requirements for connecting my on-premises location to Azure?What are the technical requirements for connecting my on-premises location to Azure?

Are connections to ExpressRoute redundant?Are connections to ExpressRoute redundant?

Will I lose connectivity if one of my ExpressRoute links fail?Will I lose connectivity if one of my ExpressRoute links fail?

How do I ensure high availability on a virtual network connected to ExpressRoute?How do I ensure high availability on a virtual network connected to ExpressRoute?

If I'm not co-located at a cloud exchange and my service provider offers point-to-point connection, do IIf I'm not co-located at a cloud exchange and my service provider offers point-to-point connection, do Ineed to order two physical connections between my on-premises network and Microsoft?need to order two physical connections between my on-premises network and Microsoft?

Can I extend one of my VLANs to Azure using ExpressRoute?Can I extend one of my VLANs to Azure using ExpressRoute?

Can I have more than one ExpressRoute circuit in my subscription?Can I have more than one ExpressRoute circuit in my subscription?

Can I have ExpressRoute circuits from different service providers?Can I have ExpressRoute circuits from different service providers?

I see two ExpressRoute peering locations in the same metro, e.g. Singapore and Singapore2. Which peeringI see two ExpressRoute peering locations in the same metro, e.g. Singapore and Singapore2. Which peeringlocation should I choose to create my ExpressRoute circuit?location should I choose to create my ExpressRoute circuit?

Can I have multiple ExpressRoute circuits in the same metro? Can I link them to the same virtual network?Can I have multiple ExpressRoute circuits in the same metro? Can I link them to the same virtual network?

See ExpressRoute prerequisites page for requirements.

Yes. Each ExpressRoute circuit has a redundant pair of cross connections configured to provide high availability.

You will not lose connectivity if one of the cross connections fails. A redundant connection is available tosupport the load of your network and provide high availability of your ExpressRoute circuit. You canadditionally create a circuit in a different peering location to achieve circuit-level resilience.

You can achieve high availability by connecting ExpressRoute circuits in different peering locations (e.g.Singapore, Singapore2) to your virtual network. If one ExpressRoute circuit goes down, connectivity will failover to another ExpressRoute circuit. By default, traffic leaving your virtual network is routed based on EqualCost Multi-path Routing (ECMP). You can use Connection Weight to prefer one circuit to another. SeeOptimizing ExpressRoute Routing for additional details on Connection Weight.

If your service provider can establish two Ethernet virtual circuits over the physical connection, you only needone physical connection. The physical connection (for example, an optical fiber) is terminated on a layer 1 (L1)device (see the image). The two Ethernet virtual circuits are tagged with different VL AN IDs, one for theprimary circuit, and one for the secondary. Those VL AN IDs are in the outer 802.1Q Ethernet header. The inner802.1Q Ethernet header (not shown) is mapped to a specific ExpressRoute routing domain.

No. We do not support layer 2 connectivity extensions into Azure.

Yes. You can have more than one ExpressRoute circuit in your subscription. The default limit is set to 10. Youcan contact Microsoft Support to increase the limit, if needed.

Yes. You can have ExpressRoute circuits with many service providers. Each ExpressRoute circuit is associatedwith one service provider only.

If your service provider offers ExpressRoute at both sites, you can work with your provider and pick either siteto set up ExpressRoute.

How do I connect my virtual networks to an ExpressRoute circuitHow do I connect my virtual networks to an ExpressRoute circuit

Are there connectivity boundaries for my ExpressRoute circuit?Are there connectivity boundaries for my ExpressRoute circuit?

Can I link to more than one virtual network to an ExpressRoute circuit?Can I link to more than one virtual network to an ExpressRoute circuit?

I have multiple Azure subscriptions that contain virtual networks. Can I connect virtual networks that are inI have multiple Azure subscriptions that contain virtual networks. Can I connect virtual networks that are inseparate subscriptions to a single ExpressRoute circuit?separate subscriptions to a single ExpressRoute circuit?

I have multiple Azure subscriptions associated to different Azure Active Directory tenants or EnterpriseI have multiple Azure subscriptions associated to different Azure Active Directory tenants or EnterpriseAgreement enrollments. Can I connect virtual networks that are in separate tenants and enrollments to aAgreement enrollments. Can I connect virtual networks that are in separate tenants and enrollments to asingle ExpressRoute circuit not in the same tenant or enrollment?single ExpressRoute circuit not in the same tenant or enrollment?

Are virtual networks connected to the same circuit isolated from each other?Are virtual networks connected to the same circuit isolated from each other?

Can I have one virtual network connected to more than one ExpressRoute circuit?Can I have one virtual network connected to more than one ExpressRoute circuit?

Can I access the Internet from my virtual networks connected to ExpressRoute circuits?Can I access the Internet from my virtual networks connected to ExpressRoute circuits?

Can I block Internet connectivity to virtual networks connected to ExpressRoute circuits?Can I block Internet connectivity to virtual networks connected to ExpressRoute circuits?

Yes. You can have multiple ExpressRoute circuits with the same or different service providers. If the metro hasmultiple ExpressRoute peering locations and the circuits are created at various peering locations, you can linkthem to the same virtual network. If the circuits are created at the same peering location, you canâ€™t linkthem to the same virtual network.

The basic steps are:

Establish an ExpressRoute circuit and have the service provider enable it.You, or the provider, must configure the BGP peering(s).Link the virtual network to the ExpressRoute circuit.

For more information, see ExpressRoute workflows for circuit provisioning and circuit states.

Yes. The ExpressRoute partners and locations article provides an overview of the connectivity boundaries foran ExpressRoute circuit. Connectivity for an ExpressRoute circuit is limited to a single geopolitical region.Connectivity can be expanded to cross geopolitical regions by enabling the ExpressRoute premium feature.

Yes. You can have up to 10 virtual networks connections on a standard ExpressRoute circuit, and up to 100 on apremium ExpressRoute circuit.

Yes. You can authorize up to 10 other Azure subscriptions to use a single ExpressRoute circuit. This limit can beincreased by enabling the ExpressRoute premium feature.

For more information, see Sharing an ExpressRoute circuit across multiple subscriptions.

Yes. ExpressRoute authorizations can span subscription, tenant, and enrollment boundaries with no additionalconfiguration required.

For more information, see Sharing an ExpressRoute circuit across multiple subscriptions.

No. From a routing perspective, all virtual networks linked to the same ExpressRoute circuit are part of thesame routing domain and are not isolated from each other. If you need route isolation, you need to create aseparate ExpressRoute circuit.

Yes. You can link a single virtual network with up to four ExpressRoute circuits. They must be ordered throughfour different ExpressRoute locations.

Yes. If you have not advertised default routes (0.0.0.0/0) or Internet route prefixes through the BGP session,you can connect to the Internet from a virtual network linked to an ExpressRoute circuit.

Yes. You can advertise default routes (0.0.0.0/0) to block all Internet connectivity to virtual machines deployedwithin a virtual network and route all traffic out through the ExpressRoute circuit.

If you advertise default routes, we force traffic to services offered over public peering (such as Azure storageand SQL DB) back to your premises. You will have to configure your routers to return traffic to Azure through

Can virtual networks linked to the same ExpressRoute circuit talk to each other?Can virtual networks linked to the same ExpressRoute circuit talk to each other?

Can I use site-to-site connectivity for virtual networks in conjunction with ExpressRoute?Can I use site-to-site connectivity for virtual networks in conjunction with ExpressRoute?

Can I move a virtual network from site-to-site / point-to-site configuration to use ExpressRoute?Can I move a virtual network from site-to-site / point-to-site configuration to use ExpressRoute?

Why is there a public IP address associated with the ExpressRoute gateway on a virtual network?Why is there a public IP address associated with the ExpressRoute gateway on a virtual network?

What do I need to connect to Azure storage over ExpressRoute?What do I need to connect to Azure storage over ExpressRoute?

Are there limits on the number of routes I can advertise?Are there limits on the number of routes I can advertise?

Are there restrictions on IP ranges I can advertise over the BGP session?Are there restrictions on IP ranges I can advertise over the BGP session?

What happens if I exceed the BGP limits?What happens if I exceed the BGP limits?

What is the ExpressRoute BGP hold time? Can it be adjusted?What is the ExpressRoute BGP hold time? Can it be adjusted?

After I advertise the default route (0.0.0.0/0) to my virtual networks, I can't activate Windows running on myAfter I advertise the default route (0.0.0.0/0) to my virtual networks, I can't activate Windows running on myAzure VMs. How to I fix this?Azure VMs. How to I fix this?

Can I change the bandwidth of an ExpressRoute circuit?Can I change the bandwidth of an ExpressRoute circuit?

the public peering path or over the Internet. If you've enabled a service endpoint (preview) for the service, thetraffic to the service is not forced to your premises. The traffic remains within the Azure backbone network. Tolearn more about service endpoints, see Virtual network service endpoints

Yes. Virtual machines deployed in virtual networks connected to the same ExpressRoute circuit cancommunicate with each other.

Yes. ExpressRoute can coexist with site-to-site VPNs.

Yes. You will have to create an ExpressRoute gateway within your virtual network. There is a small downtimeassociated with the process.

The public IP address is used for internal management only, and does not constitute a security exposure ofyour virtual network.

You must establish an ExpressRoute circuit and configure routes for public peering.

Yes. We accept up to 4000 route prefixes for private peering and 200 each for public peering and Microsoftpeering. You can increase this to 10,000 routes for private peering if you enable the ExpressRoute premiumfeature.

We do not accept private prefixes (RFC1918) in the public and Microsoft peering BGP session.

BGP sessions will be dropped. They will be reset once the prefix count goes below the limit.

The hold time is 180. The keep-alive messages are sent every 60 seconds. These are fixed settings on theMicrosoft side that cannot be changed. It is possible for you to configure different timers, and the BGP sessionparameters will be negotiated accordingly.

The following steps help Azure recognize the activation request:

1. Establish the public peering for your ExpressRoute circuit.2. Perform a DNS lookup and find the IP address of kms.core.windows.net3. The Key Management Service must recognize that the activation request comes from Azure and honor

the request. Perform one of the following three tasks:

On your on-premises network, route the traffic destined for the IP address that you obtained in step2 back to Azure via the public peering.Have your NSP provider hair-pin the traffic back to Azure via the public peering.Create a user-defined route that points the IP that has Internet as a next hop, and apply it to thesubnet(s) where these virtual machines are.

Yes, you can attempt to increase the bandwidth of your ExpressRoute circuit in the Azure portal, or by using

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview

How do I change the bandwidth of an ExpressRoute circuit?How do I change the bandwidth of an ExpressRoute circuit?

ExpressRoute premiumWhat is ExpressRoute premium?What is ExpressRoute premium?

How many VNets can I link to an ExpressRoute circuit if I enabled ExpressRoute premium?How many VNets can I link to an ExpressRoute circuit if I enabled ExpressRoute premium?

ExpressRoute LimitsExpressRoute Limits

RESOURCE DEFAULT LIMIT

ExpressRoute circuits per subscription 10

ExpressRoute circuits per region per subscription for ARM 10

Maximum number of routes for Azure private peering withExpressRoute standard

4,000

Maximum number of routes for Azure private peering withExpressRoute premium add-on

10,000

Maximum number of routes for Azure public peering withExpressRoute standard

200

Maximum number of routes for Azure public peering withExpressRoute premium add-on

200

PowerShell. If there is capacity available on the physical port on which your circuit was created, your changesucceeds.

If your change fails, it means either there isnâ€™t enough capacity left on the current port and you need tocreate a new ExpressRoute circuit with the higher bandwidth, or that there is no additional capacity at thatlocation, in which case you won't be able to increase the bandwidth.

You will also have to follow up with your connectivity provider to ensure that they update the throttles withintheir networks to support the bandwidth increase. You cannot, however, reduce the bandwidth of yourExpressRoute circuit. You have to create a new ExpressRoute circuit with lower bandwidth and delete the oldcircuit.

You can update the bandwidth of the ExpressRoute circuit using the REST API or PowerShell cmdlet.

ExpressRoute premium is a collection of the following features:

Increased routing table limit from 4000 routes to 10,000 routes for private peering.Increased number of VNets that can be connected to the ExpressRoute circuit (default is 10). For moreinformation, see the ExpressRoute Limits table.Connectivity to Office 365 and Dynamics 365.Global connectivity over the Microsoft core network. You can now link a VNet in one geopolitical regionwith an ExpressRoute circuit in another region.Examples:

You can link a VNet created in Europe West to an ExpressRoute circuit created in Silicon Valley.On the public peering, prefixes from other geopolitical regions are advertised such that you canconnect to, for example, SQL Azure in Europe West from a circuit in Silicon Valley.

The following tables show the ExpressRoute limits and the number of VNets per ExpressRoute circuit:

The following limits apply to ExpressRoute resources per subscription.

Maximum number of routes for Azure Microsoft peeringwith ExpressRoute standard

200

Maximum number of routes for Azure Microsoft peeringwith ExpressRoute premium add-on

200

Number of virtual network links allowed per ExpressRoutecircuit

see table below


Number of Virtual Networks per ExpressRoute circuitNumber of Virtual Networks per ExpressRoute circuit

CIRCUIT SIZE NUMBER OF VNET LINKS FOR STANDARDNUMBER OF VNET LINKS WITH PREMIUMADD-ON

50 Mbps 10 20

100 Mbps 10 25

200 Mbps 10 25

500 Mbps 10 40

1 Gbps 10 50

2 Gbps 10 60

5 Gbps 10 75

10 Gbps 10 100

How do I enable ExpressRoute premium?How do I enable ExpressRoute premium?

How do I disable ExpressRoute premium?How do I disable ExpressRoute premium?

Can I pick and choose the features I want from the premium feature set?Can I pick and choose the features I want from the premium feature set?

How much does ExpressRoute premium cost?How much does ExpressRoute premium cost?

Do I pay for ExpressRoute premium in addition to standard ExpressRoute charges?Do I pay for ExpressRoute premium in addition to standard ExpressRoute charges?

ExpressRoute for Office 365 and Dynamics 365

ExpressRoute premium features can be enabled when the feature is enabled, and can be shut down byupdating the circuit state. You can enable ExpressRoute premium at circuit creation time, or can call the RESTAPI / PowerShell cmdlet.

You can disable ExpressRoute premium by calling the REST API or PowerShell cmdlet. You must make surethat you have scaled your connectivity needs to meet the default limits before you disable ExpressRoutepremium. If your utilization scales beyond the default limits, the request to disable ExpressRoute premium fails.

No. You can't pick the features. We enable all features when you turn on ExpressRoute premium.

Refer to pricing details for cost.

Yes. ExpressRoute premium charges apply on top of ExpressRoute circuit charges and charges required by theconnectivity provider.

Software as a Service offerings, like Office 365 and Dynamics 365, were created to be accessed securely and


How do I create an ExpressRoute circuit to connect to Office 365 services and Dynamics 365?How do I create an ExpressRoute circuit to connect to Office 365 services and Dynamics 365?

IMPORTANTIMPORTANT

Do I need to enable Azure public peering to connect to Office 365 services and Dynamics 365?Do I need to enable Azure public peering to connect to Office 365 services and Dynamics 365?

Can my existing ExpressRoute circuits support connectivity to Office 365 services and Dynamics 365?Can my existing ExpressRoute circuits support connectivity to Office 365 services and Dynamics 365?

What Office 365 services can be accessed over an ExpressRoute connection?What Office 365 services can be accessed over an ExpressRoute connection?

How much does ExpressRoute for Office 365 services and Dynamics 365 cost?How much does ExpressRoute for Office 365 services and Dynamics 365 cost?

What regions is ExpressRoute for Office 365 supported in?What regions is ExpressRoute for Office 365 supported in?

Can I access Office 365 over the Internet, even if ExpressRoute was configured for my organization?Can I access Office 365 over the Internet, even if ExpressRoute was configured for my organization?

How can I plan for high availability for Office 365 network traffic on Azure ExpressRoute?How can I plan for high availability for Office 365 network traffic on Azure ExpressRoute?

Can I access Office 365 US Government Community (GCC) services over an Azure US GovernmentCan I access Office 365 US Government Community (GCC) services over an Azure US GovernmentExpressRoute circuit?ExpressRoute circuit?

Can Dynamics 365 for Operations (formerly known as Dynamics AX Online) be accessed over anCan Dynamics 365 for Operations (formerly known as Dynamics AX Online) be accessed over an

reliably via the Internet. Because of this, we recommend ExpressRoute for these applications only for specificscenarios. For information about using ExpressRoute to access Office 365, visit Azure ExpressRoute for Office365.

1. Review the ExpressRoute prerequisites page to make sure you meet the requirements.2. To ensure that your connectivity needs are met, review the list of service providers and locations in the

ExpressRoute partners and locations article.3. Plan your capacity requirements by reviewing Network planning and performance tuning for Office 365.4. Follow the steps listed in the workflows to set up connectivity ExpressRoute workflows for circuit

provisioning and circuit states.

Make sure that you have enabled ExpressRoute premium add-on when configuring connectivity to Office 365 servicesand Dynamics 365.

No, you only need to enable Microsoft Peering. Authentication traffic to Azure AD is sent through MicrosoftPeering.

Yes. Your existing ExpressRoute circuit can be configured to support connectivity to Office 365 services. Makesure that you have sufficient capacity to connect to Office 365 services and that you have enabled premiumadd-on. Network planning and performance tuning for Office 365 helps you plan your connectivity needs.Also, see Create and modify an ExpressRoute circuit.

Refer to Office 365 URLs and IP address ranges page for an up-to-date list of services supported overExpressRoute.

Office 365 services and Dynamics 365 require premium add-on to be enabled. See the pricing details page forcosts.

See ExpressRoute partners and locations for information.

Yes. Office 365 service endpoints are reachable through the Internet, even though ExpressRoute has beenconfigured for your network. Please check with your organization's networking team if the network at yourlocation is configured to connect to Office 365 services through ExpressRoute.

See the recommendation for High availability and failover with Azure ExpressRoute

Yes. Office 365 GCC service endpoints are reachable through the Azure US Government ExpressRoute.However, you first need to open a support ticket on the Azure portal to provide the prefixes you intend toadvertise to Microsoft. Your connectivity to Office 365 GCC services will be established after the support ticketis resolved.


http://aka.ms/tune/

http://aka.ms/tune/

http://aka.ms/o365endpoints


https://aka.ms/erhighavailability

ExpressRoute connection?ExpressRoute connection?

Route filters for Microsoft peeringI am turning on Microsoft peering for the first time, what routes will I see?I am turning on Microsoft peering for the first time, what routes will I see?

I turned on Microsoft peering and now I am trying to select Exchange Online, but it is giving me an errorI turned on Microsoft peering and now I am trying to select Exchange Online, but it is giving me an errorthat I am not authorized to do it.that I am not authorized to do it.

Do I need to get authorization for turning on Dynamics 365 over Microsoft peering?Do I need to get authorization for turning on Dynamics 365 over Microsoft peering?

I enabled Microsoft peering prior to August 1st, 2017, how can I take advantage of route filters?I enabled Microsoft peering prior to August 1st, 2017, how can I take advantage of route filters?

I have Microsoft peering at one location, now I am trying to enable it at another location and I am notI have Microsoft peering at one location, now I am trying to enable it at another location and I am notseeing any prefixes.seeing any prefixes.

Yes. Dynamics 365 for Operations is hosted on Azure. You can enable Azure public peering on yourExpressRoute circuit to connect to it.

You will not see any routes. You have to attach a route filter to your circuit to start prefix advertisements. Forinstructions, see Configure route filters for Microsoft peering.

When using route filters, any customer can turn on Microsoft peering. However, for consuming Office 365services, you still need to get authorized by Office 365.

No, you do not need authorization for Dynamics 365. You can create a rule and select Dynamics 365community without authorization.

Your existing circuit will continue advertising the prefixes for Office 365 and Dynamics 365. If you want to addAzure public prefixes advertisements over the same Microsoft peering, you can create a route filter, select theservices you need advertised (including the Office 365 service(s) you need and Dynamics 365), and attach thefilter to your Microsoft peering. For instructions, see Configure route filters for Microsoft peering.

Microsoft peering of ExpressRoute circuits that were configured prior to August 1, 2017 will have allservice prefixes advertised through Microsoft peering, even if route filters are not defined.

Microsoft peering of ExpressRoute circuits that are configured on or after August 1, 2017 will not haveany prefixes advertised until a route filter is attached to the circuit. You will see no prefixes by default.

https://www.microsoft.com/dynamics365/operations

ExpressRoute connectivity models6/27/2017 • 1 min to read • Edit Online

Co-located at a cloud exchange

Point-to-point Ethernet connections

Any-to-any (IPVPN) networks

Next steps

You can create a connection between your on-premises network and the Microsoft cloud in three different ways,CloudExchange Co-location, Point-to-point Ethernet Connection, and Any-to-any (IPVPN) Connection.Connectivity providers can offer one or more connectivity models. You can work with your connectivity provider topick the model that works best for you.

If you are co-located in a facility with a cloud exchange, you can order virtual cross-connections to the Microsoftcloud through the co-location provider ’s Ethernet exchange. Co-location providers can offer either Layer 2 cross-connections, or managed Layer 3 cross-connections between your infrastructure in the co-location facility and theMicrosoft cloud.

You can connect your on-premises datacenters/offices to the Microsoft cloud through point-to-point Ethernet links.Point-to-point Ethernet providers can offer Layer 2 connections, or managed Layer 3 connections between yoursite and the Microsoft cloud.

You can integrate your WAN with the Microsoft cloud. IPVPN providers (typically MPLS VPN) offer any-to-anyconnectivity between your branch offices and datacenters. The Microsoft cloud can be interconnected to your WANto make it look just like any other branch office. WAN providers typically offer managed Layer 3 connectivity.ExpressRoute capabilities and features are all identical across all of the above connectivity models.

Learn about ExpressRoute connections and routing domains. See ExpressRoute circuits and routing domains.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-connectivity-models.md

Learn about ExpressRoute features. See the ExpressRoute Technical OverviewFind a service provider. See ExpressRoute partners and peering locations.Ensure that all prerequisites are met. See ExpressRoute prerequisites.Refer to the requirements for Routing, NAT, and QoS.Configure your ExpressRoute connection.

Create an ExpressRoute circuitConfigure routingLink a VNet to an ExpressRoute circuit

ExpressRoute circuits and routing domains4/9/2018 • 5 min to read • Edit Online

ExpressRoute circuits

Quotas, limits, and limitationsQuotas, limits, and limitations

ExpressRoute routing domains

You must order an ExpressRoute circuit to connect your on-premises infrastructure to Microsoft through aconnectivity provider. The following figure shows a logical representation of connectivity between your WANand Microsoft.

An ExpressRoute circuit represents a logical connection between your on-premises infrastructure and Microsoftcloud services through a connectivity provider. You can order multiple ExpressRoute circuits. Each circuit can bein the same or different regions, and can be connected to your premises through different connectivityproviders.

ExpressRoute circuits do not map to any physical entities. A circuit is uniquely identified by a standard GUIDcalled as a service key (s-key). The service key is the only piece of information exchanged between Microsoft, theconnectivity provider, and you. The s-key is not a secret for security purposes. There is a 1:1 mapping betweenan ExpressRoute circuit and the s-key.

An ExpressRoute circuit can have up to three independent peerings: Azure public, Azure private, and Microsoft.Each peering is a pair of independent BGP sessions each of them configured redundantly for high availability.There is a 1:N (1 <= N <= 3) mapping between an ExpressRoute circuit and routing domains. An ExpressRoutecircuit can have any one, two, or all three peerings enabled per ExpressRoute circuit.

Each circuit has a fixed bandwidth (50 Mbps, 100 Mbps, 200 Mbps, 500 Mbps, 1 Gbps, 10 Gbps) and is mappedto a connectivity provider and a peering location. The bandwidth you select is shared across all the peerings forthe circuit.

Default quotas and limits apply for every ExpressRoute circuit. Refer to the Azure Subscription and ServiceLimits, Quotas, and Constraints page for up-to-date information on quotas.

An ExpressRoute circuit has multiple routing domains associated with it: Azure public, Azure private, and

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-circuit-peerings.md

Azure Private peeringAzure Private peering

Azure Public peeringAzure Public peering

IMPORTANTIMPORTANT

Microsoft. Each of the routing domains is configured identically on a pair of routers (in active-active or loadsharing configuration) for high availability. Azure services are categorized as Azure public and Azure private torepresent the IP addressing schemes.

Azure compute services, namely virtual machines (IaaS) and cloud services (PaaS), that are deployed within avirtual network can be connected through the private peering domain. The private peering domain is consideredto be a trusted extension of your core network into Microsoft Azure. You can set up bi-directional connectivitybetween your core network and Azure virtual networks (VNets). This peering lets you connect to virtualmachines and cloud services directly on their private IP addresses.

You can connect more than one virtual network to the private peering domain. Review the FAQ page forinformation on limits and limitations. You can visit the Azure Subscription and Service Limits, Quotas, andConstraints page for up-to-date information on limits. Refer to the Routing page for detailed information onrouting configuration.

All Azure PaaS services are also accessible through Microsoft peering. We recommend you to create Microsoft peering andconnect to Azure PaaS services over Microsoft peering.

Services such as Azure Storage, SQL databases, and Websites are offered on public IP addresses. You canprivately connect to services hosted on public IP addresses, including VIPs of your cloud services, through thepublic peering routing domain. You can connect the public peering domain to your DMZ and connect to allAzure services on their public IP addresses from your WAN without having to connect through the internet.

Connectivity is always initiated from your WAN to Microsoft Azure services. Microsoft Azure services will notbe able to initiate connections into your network through this routing domain. Once public peering is enabled,you can connect to all Azure services. We do not allow you to selectively pick services for which we advertiseroutes to.

You can define custom route filters within your network to consume only the routes you need. Refer to theRouting page for detailed information on routing configuration.

For more information about services supported through the public peering routing domain, see the FAQ.


Routing domain comparison

PRIVATE PEERINGPUBLIC PEERING (DEPRECATEDFOR NEW CREATIONS) MICROSOFT PEERING

Max. # prefixessupported per peering

4000 by default, 10,000with ExpressRoute Premium

200 200

IP address rangessupported

Any valid IP address withinyour WAN.

Public IP addresses ownedby you or your connectivityprovider.

Public IP addresses ownedby you or your connectivityprovider.

AS Number requirements Private and public ASnumbers. You must own thepublic AS number if youchoose to use one.

Private and public ASnumbers. However, youmust prove ownership ofpublic IP addresses.

Private and public ASnumbers. However, youmust prove ownership ofpublic IP addresses.

IP protocols supported IPv4 IPv4 IPv4, IPv6

Routing Interface IPaddresses

RFC1918 and public IPaddresses

Public IP addressesregistered to you in routingregistries.

Public IP addressesregistered to you in routingregistries.

MD5 Hash support Yes Yes Yes

Next steps

Software as a Service offerings, like Office 365 and Dynamics 365, were created to be accessed securely andreliably via the Internet. Because of this, we recommend ExpressRoute for these applications only for specificscenarios. For information about using ExpressRoute to access Office 365, visit Azure ExpressRoute for Office365.

Connectivity to Microsoft online services (Office 365, Dynamics 365, and Azure PaaS services) is through theMicrosoft peering. We enable bi-directional connectivity between your WAN and Microsoft cloud servicesthrough the Microsoft peering routing domain. You must connect to Microsoft cloud services only over public IPaddresses that are owned by you or your connectivity provider and you must adhere to all the defined rules. Formore information, see the ExpressRoute prerequisites page.

See the FAQ page for more information on services supported, costs, and configuration details. See theExpressRoute Locations page for information on the list of connectivity providers offering Microsoft peeringsupport.

The following table compares the three routing domains:

You can choose to enable one or more of the routing domains as part of your ExpressRoute circuit. You canchoose to have all the routing domains put on the same VPN if you want to combine them into a single routingdomain. You can also put them on different routing domains, similar to the diagram. The recommendedconfiguration is that private peering is connected directly to the core network, and the public and Microsoftpeering links are connected to your DMZ.

If you choose to have all three peering sessions, you must have three pairs of BGP sessions (one pair for eachpeering type). The BGP session pairs provide a highly available link. If you are connecting through layer 2connectivity providers, you are responsible for configuring and managing routing. You can learn more byreviewing the workflows for setting up ExpressRoute.


Find a service provider. See ExpressRoute service providers and locations.Ensure that all prerequisites are met. See ExpressRoute prerequisites.Configure your ExpressRoute connection.

Create and manage ExpressRoute circuitsConfigure routing (peering) for ExpressRoute circuits

ExpressRoute partners and peering locations4/11/2018 • 7 min to read • Edit Online

ExpressRoute connectivity providers

Azure regions to ExpressRoute locations within a geopolitical region.Azure regions to ExpressRoute locations within a geopolitical region.

GEOPOLITICAL REGION AZURE REGIONS EXPRESSROUTE LOCATIONS

North America East US, West US, East US 2, West US 2,Central US, South Central US, NorthCentral US, West Central US, CanadaCentral, Canada East

Atlanta, Chicago, Dallas, Denver, LasVegas, Los Angeles, Miami, New York,San Antonio, Seattle, Silicon Valley,Washington DC, Montreal, Quebec City,Toronto

South America Brazil South Sao Paulo

Europe France Central, France South, NorthEurope, West Europe, UK West, UKSouth

Amsterdam, Dublin, London,Newport(Wales), Paris

Asia East Asia, Southeast Asia Hong Kong, Singapore, Singapore2

The tables in this article provide information on ExpressRoute connectivity providers, ExpressRoute geographicalcoverage, Microsoft cloud services supported over ExpressRoute, and ExpressRoute System Integrators (S Is).

ExpressRoute is supported across all Azure regions and locations. The following map provides a list of Azureregions and ExpressRoute locations. ExpressRoute locations refer to those where Microsoft peers with severalservice providers.

You will have access to Azure services across all regions within a geopolitical region if you connected to at least oneExpressRoute location within the geopolitical region.

The following table provides a map of Azure regions to ExpressRoute locations within a geopolitical region.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-locations.md

Japan Japan West, Japan East Osaka, Tokyo

Australia Australia Southeast, Australia East,Australia Central, Australia Central 2

Melbourne, Sydney

India India West, India Central, India South Chennai, Mumbai

South Korea Korea Central, Korea South Busan, Seoul


Regions and geopolitical boundaries for national cloudsRegions and geopolitical boundaries for national clouds


US Government cloud US Gov Arizona, US Gov Iowa, US GovTexas, US Gov Virginia, US DoD Central,US DoD East

Chicago, Dallas, New York, Seattle,Silicon Valley, Washington DC

China China North, China East Beijing, Shanghai

Germany Germany Central, Germany East Berlin, Frankfurt

Connectivity provider locations

Production AzureProduction Azure

SERVICE PROVIDER MICROSOFT AZUREOFFICE 365 AND DYNAMICS365 LOCATIONS

AARNet Supported Supported Melbourne, Sydney

Airtel Supported Supported Chennai, Mumbai

Aryaka Networks Supported Supported Amsterdam, Dallas, HongKong, Silicon Valley,Singapore, Tokyo,Washington DC

Ascenty Data Centers Coming soon Coming soon Sao Paulo

AT&T NetBond Supported Supported Amsterdam, Chicago, Dallas,London, Silicon Valley,Singapore, Sydney, Tokyo,Toronto, Washington DC

The table below provides information on regions and geopolitical boundaries for national clouds.

Connectivity across geopolitical regions is not supported on the standard ExpressRoute SKU. You will need toenable the ExpressRoute premium add-on to support global connectivity. Connectivity to national cloudenvironments is not supported. You can work with your connectivity provider if such a need arises.

The following table shows locations by service provider. If you want to view available providers by location, seeService providers by location.

https://www.aarnet.edu.au/network-and-services/cloud-services-applications/azure-expressroute/

http://www.airtel.in/business/connexion

http://www.aryaka.com/

https://ascenty.com/solucoes/conectividade-e-interconexoes/Microsoft-express-route/

https://www.synaptic.att.com/clouduser/html/productdetail/ATT_NetBond.htm

Bell Canada Supported Supported Montreal, Toronto, QuebecCity

British Telecom Supported Supported Amsterdam, Hong Kong,London, Silicon Valley,Singapore, Sydney, Tokyo,Washington DC

C3ntro Coming soon Coming soon Miami

CenturyLink CloudConnect

Supported Supported New York, San Antonio,Tokyo, Toronto

China Telecom Global Supported Not Supported Hong Kong

Cologix Supported Supported Dallas, Montreal, Toronto

Colt Supported Supported Amsterdam, Dublin, London,Paris, Tokyo

Comcast Supported Supported Chicago, Silicon Valley,Washington DC

CoreSite Supported Supported Chicago, Denver, LosAngeles, New York, SiliconValley, Washinton DC

eir Supported Supported Dublin

Epsilon GlobalCommunications

Supported Supported Singapore

Equinix Supported Supported Amsterdam, Atlanta,Chicago, Dallas, Hong Kong,London, Los Angeles,Melbourne, Miami, NewYork, Osaka, Paris, Sao Paulo,Seattle, Silicon Valley,Singapore, Sydney, Tokyo,Toronto, Washington DC

euNetworks Supported Supported Amsterdam, London

GÉANT Supported Supported Amsterdam

Global Cloud Xchange(GCX)

Supported Supported Chennai, Mumbai

InterCloud Supported Supported Amsterdam, London, Paris,Singapore, Washington DC

Internet2 Supported Supported Washington DC


https://business.bell.ca/shop/enterprise/cloud-connect-access-to-cloud-partner-services

http://www.globalservices.bt.com/uk/en/news/bt_to_provide_connectivity_to_microsoft_azure

https://c3ntro.com/

http://www.centurylink.com/cloudconnect

http://www.cologix.com/solutions/cloud-connect/public-clouds/microsoft-cloud/

http://www.colt.net/uk/en/news/colt-announces-dedicated-cloud-access-for-microsoft-azure-services-en.htm

https://business.comcast.com/landingpage/microsoft-azure

http://www.coresite.com/solutions/cloud-services/public-cloud-providers/microsoft-azure-expressroute

http://www.equinix.com/partners/microsoft-azure/

http://globalcloudxchange.com/cloud-platform/cloud-x-fusion/cloud-x-fusion-for-azure/

https://www.intercloud.com/

Internet Initiative JapanInc. - IIJ

Supported Supported Osaka, Tokyo

Internet Solutions - CloudConnect

Supported Supported Amsterdam, London

Interxion Supported Supported Amsterdam, Dublin, London,Paris

IX Reach Supported Supported Toronto

Jisc Supported Supported London

KINX Supported Supported Seoul

KPN Supported Supported Amsterdam

Level 3 Communications Supported Supported Amsterdam, Chicago, Dallas,Las Vegas, London, Newport,Sao Paulo, Seattle, SiliconValley, Singapore,Washington DC

LG CNS Supported Supported Busan, Seoul

Megaport Supported Supported Amsterdam, Chicago, Dallas,Denver, Dublin, Hong Kong,Las Vegas, London, LosAngeles, Melbourne, Miami,New York, Quebec City, SanAntonio, Seattle, SiliconValley, Singapore,Singapore2, Sydney, Toronto,Washington DC

MTN Supported Supported London

Neutrona Networks Supported Supported Miami, Sao Paulo

Next Generation Data Supported Supported Newport(Wales)

NEXTDC Supported Supported Melbourne, Sydney

NTT Communications Supported Supported Amsterdam, Hong Kong,London, Los Angeles, Osaka,Singapore, Sydney, Tokyo,Washington DC

NTT SmartConnect Supported Supported Osaka

Optus Supported Supported Melbourne+ , Sydney


http://www.iij.ad.jp/en/news/pressrelease/2015/1216-2.html

http://www.interxion.com/why-interxion/colocate-with-the-clouds/Microsoft-Azure/

https://www.ixreach.com/services/cloud-connectivity/microsoft-azure/

http://www.kpn.com/cloudconnect

http://your.level3.com/LP=882?WT.tsrc=02192014LP882AzureVanityAzureText

https://www.megaport.com/services/microsoft-expressroute/

http://www.neutrona.com/index.php/azure-expressroute/

http://www.nextgenerationdata.co.uk/ngd-cloud-gateway/

http://www.ntt.com/en/services/network/virtual-private-network.html

http://cloud.nttsmc.com/cxc/azure.html

https://www.optus.com.au/enterprise/networking/network-connectivity/express-link

Orange Supported Supported Amsterdam, Hong Kong,London, Paris, Silicon Valley,Singapore, Sydney,Washington DC

PacketFabric Supported Supported Chicago, Silicon Valley

PCCW Global Limited Supported Supported Hong Kong

Sejong Telecom Supported Supported Seoul

SIFY Supported Supported Chennai, Mumbai

SingTel Supported Supported Singapore, Singapore2

Softbank Supported Supported Osaka, Tokyo

Sprint Supported Supported Washington DC

Tata Communications Supported Supported Amsterdam, Chennai, HongKong, London, Mumbai,Silicon Valley, Singapore,Washington DC

TeleCity Group Supported Supported Amsterdam, Dublin, London

Telefonica Supported Supported Amsterdam, Sao Paulo

Telehouse - KDDI Supported Supported London

Telmex Uninet Coming soon Coming soon Dallas+

Telenor Supported Supported Amsterdam, London

Telstra Corporation Supported Supported Melbourne, Sydney

Telus Supported Supported Montreal, Toronto

UOLDIVEO Supported Supported Sao Paulo

Verizon Supported Supported Amsterdam, Chicago, Dallas,Hong Kong, London, SiliconValley, Singapore, Sydney,Tokyo, Washington DC

Vodafone Supported Not Supported London

Zayo Supported Supported Amsterdam, Chicago, Dallas,London, Los Angeles,Montreal, New York, SiliconValley, Toronto, WashingtonDC


http://www.orange-business.com/en/products/business-vpn-galerie

https://www.packetfabric.com/packetcor/microsoft-azure/

http://telecom.sify.com/azure-expressroute.html

http://info.singtel.com/about-us/news-releases/singtel-provide-secure-private-access-microsoft-azure-public-cloud

http://www.softbank.jp/biz/cloud/cloud_access/direct_access_for_az/

https://business.sprint.com/solutions/cloud-networking/

http://www.tatacommunications.com/lp/izo/azure/azure_index.html

http://www.telecitygroup.com/investor-centre/news_details.htm?locid=03100500400b00d&xml

https://www.business-solutions.telefonica.com/es/enterprise/solutions/efficient-infrastructure/managed-voice-data-connectivity/

http://www.telehouse.net/solutions/cloud-services/cloud-link

http://www.telstra.com.au/business-enterprise/network-services/networks/cloud-direct-connect/

http://www.telus.com

http://www.uoldiveo.com.br/solucoes/cloud.html#rmcl

http://www.verizonenterprise.com/products/networking/secure-cloud-interconnect/

http://www.vodafone.com/business/global-enterprise/global-connectivity/vodafone-ip-vpn-cloud-connect

http://www.zayo.com/solutions/industries/cloud-connectivity/microsoft-expressroute

National cloud environmentNational cloud environment

US Government cloudUS Government cloud

SERVICE PROVIDER MICROSOFT AZURE OFFICE 365 LOCATIONS

AT&T NetBond Supported Supported Chicago, Washington DC

Equinix Supported Supported Chicago, Dallas, New York,Seattle, Silicon Valley,Washington DC

Level 3 Communications Supported Supported Chicago, New York+, SiliconValley, Washington DC

Megaport Supported Supported Chicago, Dallas

Verizon Supported Supported Chicago, Dallas, New York,Washington DC

ChinaChina


China Telecom Supported Not Supported Beijing, Shanghai

GermanyGermany


Colt Supported Not Supported Berlin+, Frankfurt

Equinix Supported Not Supported Frankfurt

e-shelter Supported Not Supported Berlin

Interxion Supported Not Supported Frankfurt

Megaport Supported Not Supported Berlin

T-Systems Supported Not Supported Berlin

Connectivity Through Exchange Providers

+ denotes coming soon

To learn more, see ExpressRoute in China.

If your connectivity provider is not listed in previous sections, you can still create a connection.

Check with your connectivity provider to see if they are connected to any of the exchanges in the table above.You can check the following links to gather more information about services offered by exchange providers.Several connectivity providers are already connected to Ethernet exchanges.

CologixCoreSite





http://news.verizonenterprise.com/2014/04/secure-cloud-interconnect-solutions-enterprise/

http://www.windowsazure.cn/home/features/expressroute/



https://www.e-shelter.de/en/microsoft-expressroutetm


http://www.cologix.com/

http://www.coresite.com/

Connectivity Through Additional Service ProvidersCONNECTIVITY PROVIDER EXCHANGE LOCATIONS

1CLOUDSTAR Equinix Singapore

Airgate Technologies, Inc. Equinix, Cologix Toronto, Montreal

Alaska Communications Equinix Seattle

Altice Business Equinix New York, Washington DC

Arteria Networks Corporation Equinix Tokyo

Bezeq International Ltd. euNetworks London

BICS Equinix Amsterdam, Franfurt, London,Singapore, Washington DC

BroadBand Tower, Inc. Equinix Tokyo

C3ntro Telecom Equinix, Megaport Dallas

Cinia Equinix, Megaport Frankfurt, Hamburg

Cogeco Peer 1 Equinix Montreal, Toronto

Cox Business Equinix Dallas, Silicon Valley, Washington DC

Data Foundry Megaport Dallas

Epsilon Telecommunications Limited Equinix London, Singapore, Washington DC

Eurofiber Equinix Amsterdam

Exponential E Equinix London

Fastweb S.p.A Equinix Amsterdam

Have your connectivity provider extend your network to the peering location of choice.

Order an ExpressRoute circuit with the exchange as your connectivity provider to connect to Microsoft.

Equinix Cloud ExchangeInterxionIX ReachMegaportNextDCTeleCity CloudIX

Ensure that your connectivity provider extends your connectivity in a highly available manner so thatthere are no single points of failure.

Follow steps in Create an ExpressRoute circuit to set up connectivity.

http://www.equinix.com/services/interconnection-connectivity/cloud-exchange/

http://www.interxion.com/products/interconnection/cloud-connect/



http://www.nextdc.com/

http://www.telecitygroup.com/colocation-services/cloud-ix.htm

http://www.1cloudstar.com/service/cloudconnect-azure-expressroute/

http://airgate.ca/cloud-express/

http://www.alaskacommunications.com/For-Your-Business/Direct-Cloud-Service

https://golightpath.com/transport

https://arteria-net.com/business/service/cloud_access/sca/

https://www.bezeqint.net/english

https://bics.com/bics-solutions-suite/cloud-connect/

http://www.bbtower.co.jp/product-service/data-center/network/dcconnect-for-azure/

http://www.c3ntro.com/data/cloud-conectivity/

https://www.cinia.fi/en/services/connectivity-services/direct-public-cloud-connection.html

https://www.cogecopeer1.com/en/

https://www.cox.com/business/networking/cloud-connectivity.html

https://www.datafoundry.com/services/cloud-connect

http://www.epsilontel.com/data-connectivity/cloud-access/

https://eurofiber.nl/microsoft-azure/

http://www.exponential-e.com/services/connectivity-services/cloud-connect-exchange

http://www.fastweb.it/grandi-aziende/connessione-voce-e-wifi/scheda-prodotto/rete-privata-virtuale/

Gtt Communications Inc Equinix Washington DC

HSO Equinix London, Slough

LGA Telecom Equinix Singapore

Lightower Equinix Chicago, New York, Washington DC

Macroview Telecom Equinix Hong Kong

Macquarie Telecom Group Megaport Sydney

MainOne Equinix Amsterdam

Masergy Equinix Washington DC

NexGen Networks Interxion London

Nianet Telecity Amsterdam, Frankfurt

QSC AG Interxion Frankfurt

Rogers Cologix, Equinix Montreal, Toronto

Tamares Telecom Telecity London

Telia Equinix Amsterdam

ThinkTel Equinix Toronto

Transtelco Equinix Dallas, Los Angeles

United Information Highway (UIH) Equinix Singapore

Webair Megaport New York

Windstream Equinix Chicago, Silicon Valley, Washington DC

Zain Equinix London

Zertia Level 3 Madrid

Zirro Equinix Montreal, Toronto

CONNECTIVITY PROVIDER EXCHANGE LOCATIONS

Connectivity Through Datacenter ProvidersPROVIDER EXCHANGE

Cyrus One Megaport

https://www.gtt.net/wp-content/uploads/2017/04/EtherCloud-Data-Sheet.pdf

http://www.hso.co.uk/products/cloud-direct

http://www.lightower.com/network-solutions/cloud-connect/#microsoft-azure

http://www.macroview.com/en/scripts/catitem.php?catid=solution&sectionid=expressroute

https://macquariegovernment.com/secure-cloud/secure-cloud-exchange/

https://www.mainone.net/services/connectivity/cloud-connect/

https://www.masergy.com/solutions/hybrid-networking/cloud-marketplace/microsoft-azure

http://www.nexgen-net.com/nexgen-networks-direct-connect-microsoft-azure-expressroute.html

https://nianet.dk/produkter/internet/microsoft-expressroute

https://www.qsc.de/de/produkte-loesungen/cloud-services-und-it-outsourcing/pure-enterprise-cloud/multi-cloud-management/azure-expressroute/

http://www.tamarestelecom.com/our-services/#Connectivity

https://www.telia.se/foretag/losningar/produkter-tjanster/datanet

http://www.thinktel.ca/services/agile-ix-data/expressroute/

http://www.transtelco.net/tcloud/microsoft

https://www.uih.co.th/en/internet-solution/cloud-direct/uih-cloud-direct-for-microsoft-azure-expressroute

https://www.webair.com/microsoft-express-route-partnership/

http://www.windstreambusiness.com/solutions/cloud-services/cloud-and-managed-hosting-services

http://www.zertia.es/index.php/novedades

https://zirro.com/services/

https://cyrusone.com/enterprise-data-center-services/connectivity-and-interconnection/cloud-connectivity-reaching-amazon-microsoft-google-and-more/microsoft-azure-expressroute/?doing_wp_cron=1498512235.6733090877532958984375

Digital Realty Megaport

EdgeConnex Megaport

RagingWire Data Centers IX Reach

T5 Datacenters IX Reach

PROVIDER EXCHANGE

Connectivity Through National Research and Education Networks(NREN)

PROVIDER

AARNET

DeIC, through GÉANT

GARR, through GÉANT

GÉANT

HEAnet, through GÉANT

Internet2

JISC

RedIRIS, through GÉANT

SINET

Surfnet, through GÉANT

ExpressRoute system integrators

SYSTEM INTEGRATOR CONTINENT

Altogee Europe

Avanade Inc. Asia, Europe, North America, South America

Bright Skies GmbH Europe

If your connectivity provider is not listed here, please check to see if they are connected to any of theExpressRoute Exchange Partners listed above.

Enabling private connectivity to fit your needs can be challenging, based on the scale of your network. You canwork with any of the system integrators listed in the following table to assist you with onboarding to ExpressRoute.

https://www.digitalrealty.com/services/interconnection/service-exchange/

http://www.edgeconnex.com/services/edge-data-centers-proximity-matters/

http://www.ragingwire.com/wholesale/wholesale-data-centers-worldwide-nexcenters

http://t5datacenters.com/network-cloud-connect/

http://www.altogee.be/expressroute

http://www.avanade.com/

http://bskies.io/expressroute

Ensyst Asia

Equinix Professional Services North America

FlexManage North America

Inframon Europe

Lightstream North America

The IT Consultancy Group Australia

MOQdigital Australia

MSG Services Europe (Germany)

Nelite Europe

New Signature Europe

OneAs1a Asia

Orange Networks Europe

Perficient North America

Presidio North America

sol-tec Europe

Vigilant.IT Australia


Next stepsFor more information about ExpressRoute, see the ExpressRoute FAQ.Ensure that all prerequisites are met. See ExpressRoute prerequisites.

http://www.ensyst.com.au

http://www.equinix.com/services/consulting/

http://www.flexmanage.com/cloud

http://www.inframon.com/partner/microsoft/

https://www.ltstream.com/expressroute

http://itconsult.com.au/microsoft-expressroute

http://www.moqdigital.com.au/insights/technical/network-connectivity-options-for-azure

https://www.msg-services.de/it-services/managed-services/cloud-outsourcing/

http://nelite.com/

https://www.newsignature.co.uk/

http://www.oneas1a.com/express-connect-any-cloud-ecac

https://orange-networks.com/blog/88-azureexpressroute

http://www.perficient.com/Partners/Microsoft/Cloud/Azure-ExpressRoute

http://info.presidio.com/microsoft-azure-expressroute

http://www.sol-tec.com/Technologies

https://vigilant.it/expressroute



Azure regions to ExpressRoute locations within a geopolitical regionAzure regions to ExpressRoute locations within a geopolitical region


North America East US, West US, East US 2, West US 2,Central US, South Central US, NorthCentral US, West Central US, CanadaCentral, Canada East

Atlanta, Chicago, Dallas, Denver, LasVegas, Los Angeles, Miami, New York,San Antonio, Seattle, Silicon Valley,Washington DC, Montreal, Quebec City,Toronto





The tables in this article provide information on ExpressRoute connectivity providers, ExpressRoute geographicalcoverage, Microsoft cloud services supported over ExpressRoute, and ExpressRoute System Integrators (S Is).


You will have access to Azure services across all regions within a geopolitical region if you connected to at leastone ExpressRoute location within the geopolitical region.


https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-locations-providers.md



Melbourne, Sydney






US Government cloud US Gov Iowa, US Gov Virginia, US DoDCentral, US DoD East






LOCATION SERVICE PROVIDERS

Amsterdam Aryaka Networks, AT&T NetBond, British Telecom, Colt,Equinix, euNetworks, GÉANT, InterCloud, Internet Solutions -Cloud Connect, Interxion, KPN, Level 3 Communications,Megaport, NTT Communications, Orange, TataCommunications, TeleCity Group, Telefonica, Telenor, Verizon,Zayo

Atlanta Equinix

Busan LG CNS

Chennai Airtel+, Global CloudXchange (GCX), SIFY, TataCommunications

Chicago AT&T NetBond, Comcast, Coresite, Equinix, PacketFabric, Level3 Communications, Megaport, Verizon, Zayo

Dallas Aryaka Networks, AT&T NetBond, Cologix, Equinix, Level 3Communications, Megaport, Telmex Uninet+, Verizon, Zayo



The following table shows connectivity locations and the service providers for each location. If you want to viewservice providers and the locations for which they can provide service, see Locations by service provider.

Denver CoreSite, Megaport

Dublin Colt, eir, Interxion, Megaport, Telecity Group

Hong Kong Aryaka Networks, British Telecom, China Telecom Global,Equinix, Megaport, NTT Communications, Orange, PCCWGlobal Limited, Tata Communications, Verizon

Las Vegas Level 3 Communications, Megaport

London AT&T NetBond, British Telecom, Colt, Equinix, InterCloud,Internet Solutions - Cloud Connect, Interxion, Jisc, Level 3Communications, Megaport, MTN, NTT Communications,Orange, Tata Communications, Telecity Group, Telehouse -KDDI, Telenor, Verizon, Vodafone, Zayo

Los Angeles CoreSite, Equinix, Megaport, NTT, Zayo

Melbourne AARNet, Equinix, Megaport, NEXTDC, Optus+, TelstraCorporation

Miami C3ntro+, Equinix, Megaport, Neutrona Networks

Montreal Bell Canada, Cologix, Telus, Zayo

Mumbai Airtel+, Global CloudXchange (GCX), Sify, TataCommunications

New York CenturyLink Cloud Connect, Coresite, Equinix, Megaport,Zayo

Newport(Wales) Level 3 Communications, Next Generation Data

Osaka Equinix, Internet Initiative Japan Inc. - IIJ, NTTCommunications, NTT SmartConnect, Softbank

Paris Colt, Intercloud, Interxion, Equinix, Orange

Quebec City Bell Canada, Megaport

San Antonio CenturyLink Cloud Connect, Megaport

Sao Paulo Ascenty Data Centers+, Equinix, Level 3 Communications,Neutrona Networks, Telefonica, UOLDIVEO

Seattle Equinix, Level 3 Communications, Megaport

Seoul KINX, LG CNS, Sejong Telecom

Silicon Valley Aryaka Networks, AT&T NetBond, British Telecom, Comcast,Coresite, Equinix, PacketFabric, Level 3 Communications,Megaport, Orange, Tata Communications, Verizon, ZayoGroup


Singapore Aryaka Networks, AT&T NetBond, British Telecom, EpsilonGlobal Communications, Equinix, InterCloud, Level 3Communications, Megaport, NTT Communications, Orange,SingTel, Tata Communications, Verizon

Singapore2 Megaport, SingTel

Sydney AARNet, AT&T NetBond, British Telecom, Equinix, Megaport,NEXTDC, NTT Communications, Optus, Orange, TelstraCorporation, Verizon

Tokyo Aryaka Networks, AT&T NetBond, British Telecom,CenturyLink Cloud Connect, Colt, Equinix, Internet InitiativeJapan Inc. - IIJ, NTT Communications, Softbank, Verizon

Toronto AT&T NetBond, Bell Canada, CenturyLink Cloud Connect,Cologix, Console, Equinix, Megaport, Telus, Zayo

Washington DC Aryaka Networks, AT&T NetBond, British Telecom, Comcast,Coresite, Equinix, Internet2, InterCloud, Level 3Communications, Megaport, NTT Communications, Orange,Sprint, Tata Communications, Verizon, Zayo


National cloud environmentsNational cloud environments



Chicago AT&T NetBond, Equinix, Level 3 Communications, Verizon

Dallas Equinix, Megaport, Verizon

New York Equinix, Level 3 Communications+, Verizon

Silicon Valley Equinix, Level 3 Communications

Seattle Equinix

Washington DC AT&T NetBond, Equinix, Level 3 Communications, Verizon

ChinaChina


Beijing China Telecom

Shanghai China Telecom

GermanyGermany


To learn more, see ExpressRoute in China



Berlin Colt+, e-shelter, Megaport+, T-Systems

Frankfurt Colt, Equinix, Interxion


Connectivity Through Additional Service ProvidersLOCATION EXCHANGE CONNECTIVITY PROVIDERS

Amsterdam Equinix, Telecity BICS, Eurofiber, Fastweb S.p.A,MainOne, Nianet, Telia

Chicago Equinix Lightower, Windstream

Dallas Equinix, Megaport C3ntro Telecom, Cox Business, DataFoundry, Transtelco

Frankfurt Telecity BICS, Nianet, QSC AG

Hong Kong Equinix Macroview Telecom

London Equinix, euNetworks, Telecity Bezeq International Ltd., Epsilon,Exponential E, HSO, NexGen Networks,Tamares Telecom, Zain

Los Angeles Equinix Transtelco

Madrid Level3 Zertia


Check with your connectivity provider to see if they are connected to any of the exchanges in the table above.You can check the following links to gather more information about services offered by exchange providers.Several connectivity providers are already connected to Ethernet exchanges.



CologixCoreSiteEquinix Cloud ExchangeInterXionIX ReachNextDCMegaportTeleCity CloudIX






http://www.interxion.com/





Montreal Cologix, Equinix Airgate Technologies. Inc, Cogeco Peer1, Rogers, Zirro

New York Equinix, Megaport Altice Business, Lightower, Webair

Seattle Equinix Alaska Communications

Silicon Valley Equinix Cox Business, Windstream

Singapore Equinix 1CLOUDSTAR, BICS, EpsilonTelecommunications Limited, LGATelecom, United Information Highway(UIH)

Slough Equinix HSO

Sydney Megaport Macquarie Telecom Group

Tokyo Equinix ARTERIA Networks Corporation,BroadBand Tower, Inc.

Toronto Equinix Airgate Technologies. Inc, Cogeco Peer1, Rogers, Thinktel, Zirro

Washington DC Equinix Altice Business, BICS, GttCommunications Inc, Epsilon, Lightower,Masergy, Windstream

LOCATION EXCHANGE CONNECTIVITY PROVIDERS


CONTINENT SYSTEM INTEGRATORS

Asia Avanade Inc., OneAs1a

Australia Ensyst, IT Consultancy, MOQdigital, Vigilant.IT

Europe Avanade Inc., Altogee, Bright Skies GmbH, Inframon, MSGServices, New Signature, Nelite, Orange Networks, sol-tec

North America Avanade Inc., Equinix Professional Services, FlexManage,Lightstream, Perficient, Presidio

South America Avanade Inc.

Next steps

Enabling private connectivity to fit your needs can be challenging, based on the scale of your network. You canwork with any of the system integrators listed in the following table to assist you with onboarding toExpressRoute.

For more information about ExpressRoute, see the ExpressRoute FAQ.Ensure that all prerequisites are met. See ExpressRoute prerequisites.



Azure regions to ExpressRoute locations within a geopolitical region.Azure regions to ExpressRoute locations within a geopolitical region.


North America East US, West US, East US 2, West US2, Central US, South Central US, NorthCentral US, West Central US, CanadaCentral, Canada East

Atlanta, Chicago, Dallas, Denver, LasVegas, Los Angeles, Miami, New York,San Antonio, Seattle, Silicon Valley,Washington DC, Montreal, QuebecCity, Toronto





The tables in this article provide information on ExpressRoute connectivity providers, ExpressRoutegeographical coverage, Microsoft cloud services supported over ExpressRoute, and ExpressRoute SystemIntegrators (S Is).


You will have access to Azure services across all regions within a geopolitical region if you connected to at leastone ExpressRoute location within the geopolitical region.


https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-locations.md



Melbourne, Sydney






US Government cloud US Gov Arizona, US Gov Iowa, US GovTexas, US Gov Virginia, US DoDCentral, US DoD East







AARNet Supported Supported Melbourne, Sydney

Airtel Supported Supported Chennai, Mumbai

Aryaka Networks Supported Supported Amsterdam, Dallas, HongKong, Silicon Valley,Singapore, Tokyo,Washington DC

Ascenty Data Centers Coming soon Coming soon Sao Paulo

AT&T NetBond Supported Supported Amsterdam, Chicago,Dallas, London, SiliconValley, Singapore, Sydney,Tokyo, Toronto, WashingtonDC



The following table shows locations by service provider. If you want to view available providers by location, seeService providers by location.

https://www.aarnet.edu.au/network-and-services/cloud-services-applications/azure-expressroute/

http://www.airtel.in/business/connexion

http://www.aryaka.com/

https://ascenty.com/solucoes/conectividade-e-interconexoes/Microsoft-express-route/


Bell Canada Supported Supported Montreal, Toronto, QuebecCity

British Telecom Supported Supported Amsterdam, Hong Kong,London, Silicon Valley,Singapore, Sydney, Tokyo,Washington DC

C3ntro Coming soon Coming soon Miami

CenturyLink CloudConnect

Supported Supported New York, San Antonio,Tokyo, Toronto

China Telecom Global Supported Not Supported Hong Kong

Cologix Supported Supported Dallas, Montreal, Toronto

Colt Supported Supported Amsterdam, Dublin,London, Paris, Tokyo

Comcast Supported Supported Chicago, Silicon Valley,Washington DC

CoreSite Supported Supported Chicago, Denver, LosAngeles, New York, SiliconValley, Washinton DC

eir Supported Supported Dublin

Epsilon GlobalCommunications

Supported Supported Singapore

Equinix Supported Supported Amsterdam, Atlanta,Chicago, Dallas, HongKong, London, Los Angeles,Melbourne, Miami, NewYork, Osaka, Paris, SaoPaulo, Seattle, Silicon Valley,Singapore, Sydney, Tokyo,Toronto, Washington DC

euNetworks Supported Supported Amsterdam, London

GÉANT Supported Supported Amsterdam

Global Cloud Xchange(GCX)

Supported Supported Chennai, Mumbai

InterCloud Supported Supported Amsterdam, London, Paris,Singapore, Washington DC

Internet2 Supported Supported Washington DC


https://business.bell.ca/shop/enterprise/cloud-connect-access-to-cloud-partner-services

http://www.globalservices.bt.com/uk/en/news/bt_to_provide_connectivity_to_microsoft_azure

https://c3ntro.com/

http://www.centurylink.com/cloudconnect

http://www.cologix.com/solutions/cloud-connect/public-clouds/microsoft-cloud/


https://business.comcast.com/landingpage/microsoft-azure

http://www.coresite.com/solutions/cloud-services/public-cloud-providers/microsoft-azure-expressroute


http://globalcloudxchange.com/cloud-platform/cloud-x-fusion/cloud-x-fusion-for-azure/

https://www.intercloud.com/

Internet Initiative JapanInc. - IIJ

Supported Supported Osaka, Tokyo

Internet Solutions - CloudConnect

Supported Supported Amsterdam, London

Interxion Supported Supported Amsterdam, Dublin,London, Paris

IX Reach Supported Supported Toronto

Jisc Supported Supported London

KINX Supported Supported Seoul

KPN Supported Supported Amsterdam

Level 3 Communications Supported Supported Amsterdam, Chicago,Dallas, Las Vegas, London,Newport, Sao Paulo, Seattle,Silicon Valley, Singapore,Washington DC

LG CNS Supported Supported Busan, Seoul

Megaport Supported Supported Amsterdam, Chicago,Dallas, Denver, Dublin,Hong Kong, Las Vegas,London, Los Angeles,Melbourne, Miami, NewYork, Quebec City, SanAntonio, Seattle, SiliconValley, Singapore,Singapore2, Sydney,Toronto, Washington DC

MTN Supported Supported London

Neutrona Networks Supported Supported Miami, Sao Paulo

Next Generation Data Supported Supported Newport(Wales)

NEXTDC Supported Supported Melbourne, Sydney

NTT Communications Supported Supported Amsterdam, Hong Kong,London, Los Angeles,Osaka, Singapore, Sydney,Tokyo, Washington DC

NTT SmartConnect Supported Supported Osaka

Optus Supported Supported Melbourne+ , Sydney


http://www.iij.ad.jp/en/news/pressrelease/2015/1216-2.html

http://www.interxion.com/why-interxion/colocate-with-the-clouds/Microsoft-Azure/


http://www.kpn.com/cloudconnect



http://www.neutrona.com/index.php/azure-expressroute/

http://www.nextgenerationdata.co.uk/ngd-cloud-gateway/

http://www.ntt.com/en/services/network/virtual-private-network.html

http://cloud.nttsmc.com/cxc/azure.html

https://www.optus.com.au/enterprise/networking/network-connectivity/express-link

Orange Supported Supported Amsterdam, Hong Kong,London, Paris, Silicon Valley,Singapore, Sydney,Washington DC

PacketFabric Supported Supported Chicago, Silicon Valley

PCCW Global Limited Supported Supported Hong Kong

Sejong Telecom Supported Supported Seoul

SIFY Supported Supported Chennai, Mumbai

SingTel Supported Supported Singapore, Singapore2

Softbank Supported Supported Osaka, Tokyo

Sprint Supported Supported Washington DC

Tata Communications Supported Supported Amsterdam, Chennai, HongKong, London, Mumbai,Silicon Valley, Singapore,Washington DC

TeleCity Group Supported Supported Amsterdam, Dublin,London

Telefonica Supported Supported Amsterdam, Sao Paulo

Telehouse - KDDI Supported Supported London

Telmex Uninet Coming soon Coming soon Dallas+

Telenor Supported Supported Amsterdam, London

Telstra Corporation Supported Supported Melbourne, Sydney

Telus Supported Supported Montreal, Toronto

UOLDIVEO Supported Supported Sao Paulo

Verizon Supported Supported Amsterdam, Chicago,Dallas, Hong Kong, London,Silicon Valley, Singapore,Sydney, Tokyo, WashingtonDC

Vodafone Supported Not Supported London


http://www.orange-business.com/en/products/business-vpn-galerie

https://www.packetfabric.com/packetcor/microsoft-azure/

http://telecom.sify.com/azure-expressroute.html

http://info.singtel.com/about-us/news-releases/singtel-provide-secure-private-access-microsoft-azure-public-cloud

http://www.softbank.jp/biz/cloud/cloud_access/direct_access_for_az/

https://business.sprint.com/solutions/cloud-networking/

http://www.tatacommunications.com/lp/izo/azure/azure_index.html

http://www.telecitygroup.com/investor-centre/news_details.htm?locid=03100500400b00d&xml

https://www.business-solutions.telefonica.com/es/enterprise/solutions/efficient-infrastructure/managed-voice-data-connectivity/

http://www.telehouse.net/solutions/cloud-services/cloud-link

http://www.telstra.com.au/business-enterprise/network-services/networks/cloud-direct-connect/

http://www.telus.com

http://www.uoldiveo.com.br/solucoes/cloud.html#rmcl

http://www.verizonenterprise.com/products/networking/secure-cloud-interconnect/

http://www.vodafone.com/business/global-enterprise/global-connectivity/vodafone-ip-vpn-cloud-connect

Zayo Supported Supported Amsterdam, Chicago,Dallas, London, LosAngeles, Montreal, NewYork, Silicon Valley, Toronto,Washington DC


National cloud environmentNational cloud environment



AT&T NetBond Supported Supported Chicago, Washington DC

Equinix Supported Supported Chicago, Dallas, New York,Seattle, Silicon Valley,Washington DC

Level 3 Communications Supported Supported Chicago, New York+, SiliconValley, Washington DC

Megaport Supported Supported Chicago, Dallas

Verizon Supported Supported Chicago, Dallas, New York,Washington DC

ChinaChina


China Telecom Supported Not Supported Beijing, Shanghai

GermanyGermany


Colt Supported Not Supported Berlin+, Frankfurt

Equinix Supported Not Supported Frankfurt

e-shelter Supported Not Supported Berlin

Interxion Supported Not Supported Frankfurt

Megaport Supported Not Supported Berlin

T-Systems Supported Not Supported Berlin



To learn more, see ExpressRoute in China.

http://www.zayo.com/solutions/industries/cloud-connectivity/microsoft-expressroute





http://news.verizonenterprise.com/2014/04/secure-cloud-interconnect-solutions-enterprise/




https://www.e-shelter.de/en/microsoft-expressroutetm


Connectivity Through Additional Service ProvidersCONNECTIVITY PROVIDER EXCHANGE LOCATIONS

1CLOUDSTAR Equinix Singapore

Airgate Technologies, Inc. Equinix, Cologix Toronto, Montreal

Alaska Communications Equinix Seattle

Altice Business Equinix New York, Washington DC

Arteria Networks Corporation Equinix Tokyo

Bezeq International Ltd. euNetworks London

BICS Equinix Amsterdam, Franfurt, London,Singapore, Washington DC

BroadBand Tower, Inc. Equinix Tokyo

C3ntro Telecom Equinix, Megaport Dallas

Cinia Equinix, Megaport Frankfurt, Hamburg

Cogeco Peer 1 Equinix Montreal, Toronto

Cox Business Equinix Dallas, Silicon Valley, Washington DC

Data Foundry Megaport Dallas


Check with your connectivity provider to see if they are connected to any of the exchanges in the tableabove. You can check the following links to gather more information about services offered by exchangeproviders. Several connectivity providers are already connected to Ethernet exchanges.



CologixCoreSiteEquinix Cloud ExchangeInterxionIX ReachMegaportNextDCTeleCity CloudIX






http://www.interxion.com/products/interconnection/cloud-connect/





http://www.1cloudstar.com/service/cloudconnect-azure-expressroute/

http://airgate.ca/cloud-express/

http://www.alaskacommunications.com/For-Your-Business/Direct-Cloud-Service

https://golightpath.com/transport

https://arteria-net.com/business/service/cloud_access/sca/

https://www.bezeqint.net/english

https://bics.com/bics-solutions-suite/cloud-connect/

http://www.bbtower.co.jp/product-service/data-center/network/dcconnect-for-azure/

http://www.c3ntro.com/data/cloud-conectivity/

https://www.cinia.fi/en/services/connectivity-services/direct-public-cloud-connection.html

https://www.cogecopeer1.com/en/

https://www.cox.com/business/networking/cloud-connectivity.html

https://www.datafoundry.com/services/cloud-connect

Epsilon TelecommunicationsLimited

Equinix London, Singapore, Washington DC

Eurofiber Equinix Amsterdam

Exponential E Equinix London

Fastweb S.p.A Equinix Amsterdam

Gtt Communications Inc Equinix Washington DC

HSO Equinix London, Slough

LGA Telecom Equinix Singapore

Lightower Equinix Chicago, New York, Washington DC

Macroview Telecom Equinix Hong Kong

Macquarie Telecom Group Megaport Sydney

MainOne Equinix Amsterdam

Masergy Equinix Washington DC

NexGen Networks Interxion London

Nianet Telecity Amsterdam, Frankfurt

QSC AG Interxion Frankfurt

Rogers Cologix, Equinix Montreal, Toronto

Tamares Telecom Telecity London

Telia Equinix Amsterdam

ThinkTel Equinix Toronto

Transtelco Equinix Dallas, Los Angeles

United Information Highway (UIH) Equinix Singapore

Webair Megaport New York

Windstream Equinix Chicago, Silicon Valley, Washington DC

Zain Equinix London

Zertia Level 3 Madrid


http://www.epsilontel.com/data-connectivity/cloud-access/

https://eurofiber.nl/microsoft-azure/

http://www.exponential-e.com/services/connectivity-services/cloud-connect-exchange

http://www.fastweb.it/grandi-aziende/connessione-voce-e-wifi/scheda-prodotto/rete-privata-virtuale/

https://www.gtt.net/wp-content/uploads/2017/04/EtherCloud-Data-Sheet.pdf

http://www.hso.co.uk/products/cloud-direct

http://www.lightower.com/network-solutions/cloud-connect/#microsoft-azure

http://www.macroview.com/en/scripts/catitem.php?catid=solution&sectionid=expressroute

https://macquariegovernment.com/secure-cloud/secure-cloud-exchange/

https://www.mainone.net/services/connectivity/cloud-connect/

https://www.masergy.com/solutions/hybrid-networking/cloud-marketplace/microsoft-azure

http://www.nexgen-net.com/nexgen-networks-direct-connect-microsoft-azure-expressroute.html

https://nianet.dk/produkter/internet/microsoft-expressroute

https://www.qsc.de/de/produkte-loesungen/cloud-services-und-it-outsourcing/pure-enterprise-cloud/multi-cloud-management/azure-expressroute/

http://www.tamarestelecom.com/our-services/#Connectivity

https://www.telia.se/foretag/losningar/produkter-tjanster/datanet

http://www.thinktel.ca/services/agile-ix-data/expressroute/

http://www.transtelco.net/tcloud/microsoft

https://www.uih.co.th/en/internet-solution/cloud-direct/uih-cloud-direct-for-microsoft-azure-expressroute

https://www.webair.com/microsoft-express-route-partnership/

http://www.windstreambusiness.com/solutions/cloud-services/cloud-and-managed-hosting-services

http://www.zertia.es/index.php/novedades

Zirro Equinix Montreal, Toronto


Connectivity Through Datacenter ProvidersPROVIDER EXCHANGE

Cyrus One Megaport

Digital Realty Megaport

EdgeConnex Megaport

RagingWire Data Centers IX Reach

T5 Datacenters IX Reach

Connectivity Through National Research and Education Networks(NREN)

PROVIDER

AARNET

DeIC, through GÉANT

GARR, through GÉANT

GÉANT

HEAnet, through GÉANT

Internet2

JISC

RedIRIS, through GÉANT

SINET

Surfnet, through GÉANT


If your connectivity provider is not listed here, please check to see if they are connected to any of theExpressRoute Exchange Partners listed above.

Enabling private connectivity to fit your needs can be challenging, based on the scale of your network. You canwork with any of the system integrators listed in the following table to assist you with onboarding toExpressRoute.

https://zirro.com/services/

https://cyrusone.com/enterprise-data-center-services/connectivity-and-interconnection/cloud-connectivity-reaching-amazon-microsoft-google-and-more/microsoft-azure-expressroute/?doing_wp_cron=1498512235.6733090877532958984375

https://www.digitalrealty.com/services/interconnection/service-exchange/

http://www.edgeconnex.com/services/edge-data-centers-proximity-matters/

http://www.ragingwire.com/wholesale/wholesale-data-centers-worldwide-nexcenters

http://t5datacenters.com/network-cloud-connect/


Altogee Europe

Avanade Inc. Asia, Europe, North America, South America

Bright Skies GmbH Europe

Ensyst Asia

Equinix Professional Services North America

FlexManage North America

Inframon Europe

Lightstream North America

The IT Consultancy Group Australia

MOQdigital Australia

MSG Services Europe (Germany)

Nelite Europe

New Signature Europe

OneAs1a Asia

Orange Networks Europe

Perficient North America

Presidio North America

sol-tec Europe

Vigilant.IT Australia

Next stepsFor more information about ExpressRoute, see the ExpressRoute FAQ.Ensure that all prerequisites are met. See ExpressRoute prerequisites.

http://www.altogee.be/expressroute

http://www.avanade.com/

http://bskies.io/expressroute

http://www.ensyst.com.au

http://www.equinix.com/services/consulting/

http://www.flexmanage.com/cloud

http://www.inframon.com/partner/microsoft/

https://www.ltstream.com/expressroute

http://itconsult.com.au/microsoft-expressroute

http://www.moqdigital.com.au/insights/technical/network-connectivity-options-for-azure

https://www.msg-services.de/it-services/managed-services/cloud-outsourcing/

http://nelite.com/

https://www.newsignature.co.uk/

http://www.oneas1a.com/express-connect-any-cloud-ecac

https://orange-networks.com/blog/88-azureexpressroute

http://www.perficient.com/Partners/Microsoft/Cloud/Azure-ExpressRoute

http://info.presidio.com/microsoft-azure-expressroute

http://www.sol-tec.com/Technologies

https://vigilant.it/expressroute

About virtual network gateways for ExpressRoute3/23/2018 • 2 min to read • Edit Online

Gateway SKUs

Estimated performances by gateway SKUEstimated performances by gateway SKU

MEGABITS PER SECOND PACKETS PER SECONDCONNECTIONS PERSECOND

VPN GATEWAY ANDEXPRESSROUTECOEXIST

Basic SKU(deprecated)

500 Unknown Unknown No

Standard SKU 1,000 100,000 7,000 Yes

High PerformanceSKU

2,000 250,000 14,000 Yes

A virtual network gateway is used to send network traffic between Azure virtual networks and on-premiseslocations. When you configure an ExpressRoute connection, you must create and configure a virtual networkgateway and a virtual network gateway connection.

When you create a virtual network gateway, you specify several settings. One of the required settings specifieswhether the gateway will be used for ExpressRoute or Site-to-Site VPN traffic. In the Resource Managerdeployment model, the setting is '-GatewayType'.

When network traffic is sent on a private connection, you use the gateway type 'ExpressRoute'. This is alsoreferred to as an ExpressRoute gateway. When network traffic is sent encrypted across the public Internet, you usethe gateway type 'Vpn'. This is referred to as a VPN gateway. Site-to-Site, Point-to-Site, and VNet-to-VNetconnections all use a VPN gateway.

Each virtual network can have only one virtual network gateway per gateway type. For example, you can have onevirtual network gateway that uses -GatewayType Vpn, and one that uses -GatewayType ExpressRoute. This articlefocuses on the ExpressRoute virtual network gateway.

When you create a virtual network gateway, you need to specify the gateway SKU that you want to use. When youselect a higher gateway SKU, more CPUs and network bandwidth are allocated to the gateway, and as a result, thegateway can support higher network throughput to the virtual network.

ExpressRoute virtual network gateways can use the following SKUs:

StandardHighPerformanceUltraPerformance

If you want to upgrade your gateway to a more powerful gateway SKU, in most cases you can use the 'Resize-AzureRmVirtualNetworkGateway' PowerShell cmdlet. This will work for upgrades to Standard andHighPerformance SKUs. However, to upgrade to the UltraPerformance SKU, you will need to recreate thegateway.

The following table shows the gateway types and the estimated performances. This table applies to both theResource Manager and classic deployment models.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-about-virtual-network-gateways.md

Ultra PerformanceSKU

9,000 1,000,000 28,000 Yes

MEGABITS PER SECOND PACKETS PER SECONDCONNECTIONS PERSECOND

VPN GATEWAY ANDEXPRESSROUTECOEXIST

IMPORTANTIMPORTANT

REST APIs and PowerShell cmdlets

CLASSIC RESOURCE MANAGER

PowerShell PowerShell

REST API REST API

Next steps

Application performance depends on multiple factors, such as the end-to-end latency, and the number of traffic flows theapplication opens. The numbers in the table represent the upper limit that the application can theoretically achieve in anideal environment.

For additional technical resources and specific syntax requirements when using REST APIs and PowerShellcmdlets for virtual network gateway configurations, see the following pages:

See ExpressRoute Overview for more information about available connection configurations.

See Create a virtual network gateway for ExpressRoute for more information about creating ExpressRoutegateways.

https://msdn.microsoft.com/library/mt270335.aspx


https://msdn.microsoft.com/library/jj154113.aspx


ExpressRoute prerequisites & checklist4/9/2018 • 2 min to read • Edit Online

Azure account

Connectivity provider

Network requirements

Office 365

To connect to Microsoft cloud services using ExpressRoute, you need to verify that the following requirementslisted in the following sections have been met.

Software as a Service offerings, like Office 365 and Dynamics 365, were created to be accessed securely andreliably via the Internet. Because of this, we recommend ExpressRoute for these applications only for specificscenarios. For information about using ExpressRoute to access Office 365, visit Azure ExpressRoute for Office365.

A valid and active Microsoft Azure account. This account is required to set up the ExpressRoute circuit.ExpressRoute circuits are resources within Azure subscriptions. An Azure subscription is a requirementeven if connectivity is limited to non-Azure Microsoft cloud services, such as Office 365 services andDynamics 365.An active Office 365 subscription (if using Office 365 services). For more information, see the Office 365specific requirements section of this article.

You can work with an ExpressRoute connectivity partner to connect to the Microsoft cloud. You can set up aconnection between your on-premises network and Microsoft in three ways.If your provider is not an ExpressRoute connectivity partner, you can still connect to the Microsoft cloudthrough a cloud exchange provider.

Redundant connectivity: there is no redundancy requirement on physical connectivity between you andyour provider. Microsoft does require redundant BGP sessions to be set up between Microsoft’s routersand the peering routers, even when you have just one physical connection to a cloud exchange.Routing: depending on how you connect to the Microsoft Cloud, you or your provider need to set up andmanage the BGP sessions for routing domains. Some Ethernet connectivity provider or cloud exchangeprovider may offer BGP management as a value-add service.NAT: Microsoft only accepts public IP addresses through Microsoft peering. If you are using private IPaddresses in your on-premises network, you or your provider need to translate the private IP addresses tothe public IP addresses using the NAT.QoS: Skype for Business has various services (for example; voice, video, text) that require differentiatedQoS treatment. You and your provider should follow the QoS requirements.Network Security: consider network security when connecting to the Microsoft Cloud via ExpressRoute.

If you plan to enable Office 365 on ExpressRoute, review the following documents for more information aboutOffice 365 requirements.

Overview of ExpressRoute for Office 365Routing with ExpressRoute for Office 365

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-prerequisites.md


https://support.office.com/en-us/article/Azure-ExpressRoute-for-Office-365-6d2534a2-c19c-4a99-be5e-33a0cee5d3bd

https://support.office.com/en-us/article/Routing-with-ExpressRoute-for-Office-365-e1da26c6-2d39-4379-af6f-4da213218408

Dynamics 365

Next steps

High availability and failover with ExpressRouteOffice 365 URLs and IP address rangesNetwork planning and performance tuning for Office 365Network bandwidth calculators and toolsOffice 365 integration with on-premises environmentsExpressRoute on Office 365 advanced training videos

If you plan to enable Dynamics 365 on ExpressRoute, review the following documents for more informationabout Dynamics 365

Dynamics 365 and ExpressRoute whitepaperDynamics 365 URLs and IP address ranges

For more information about ExpressRoute, see the ExpressRoute FAQ.Find an ExpressRoute connectivity provider. See ExpressRoute partners and peering locations.Refer to requirements for Routing, NAT, and QoS.Configure your ExpressRoute connection.


https://aka.ms/erhighavailability

https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2

https://support.office.com/en-us/article/Network-planning-and-performance-tuning-for-Office-365-e5f1228c-da3c-4654-bf16-d163daee8848

https://support.office.com/en-us/article/Network-and-migration-planning-for-Office-365-f5ee6c33-bcd7-4b0b-b0f8-dc1d9fb8d132

https://support.office.com/en-us/article/Office-365-integration-with-on-premises-environments-263faf8d-aa21-428b-aed3-2021837a4b65

https://channel9.msdn.com/series/aer/

http://download.microsoft.com/download/B/2/8/B2896B38-9832-417B-9836-9EF240C0A212/Microsoft Dynamics 365 and ExpressRoute.pdf

https://support.microsoft.com/kb/2655102

https://support.microsoft.com/kb/2728473

ExpressRoute workflows for circuit provisioning andcircuit states6/27/2017 • 4 min to read • Edit Online

This page walks you through the service provisioning and routing configuration workflows at a high level.

The following figure and corresponding steps show the tasks you must follow in order to have anExpressRoute circuit provisioned end-to-end.

1. Use PowerShell to configure an ExpressRoute circuit. Follow the instructions in the Create ExpressRoutecircuits article for more details.

2. Order connectivity from the service provider. This process varies. Contact your connectivity provider formore details about how to order connectivity.

3. Ensure that the circuit has been provisioned successfully by verifying the ExpressRoute circuit provisioningstate through PowerShell.

4. Configure routing domains. If your connectivity provider manages Layer 3 for you, they will configurerouting for your circuit. If your connectivity provider only offers Layer 2 services, you must configurerouting per guidelines described in the routing requirements and routing configuration pages.

Enable Azure private peering - You must enable this peering to connect to VMs / cloud servicesdeployed within virtual networks.Enable Azure public peering - You must enable Azure public peering if you wish to connect to Azureservices hosted on public IP addresses. This is a requirement to access Azure resources if you havechosen to enable default routing for Azure private peering.Enable Microsoft peering - You must enable this to access Office 365 and Dynamics 365.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-workflows.md

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-classic

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-routing-classic

ExpressRoute circuit provisioning states

Possible states of an ExpressRoute circuitPossible states of an ExpressRoute circuit

5. Linking virtual networks to ExpressRoute circuits - You can link virtual networks to your ExpressRoutecircuit. Follow instructions to link VNets to your circuit. These VNets can either be in the same Azuresubscription as the ExpressRoute circuit, or can be in a different subscription.

IMPORTANTIMPORTANTYou must ensure that you use a separate proxy / edge to connect to Microsoft than the one you use forthe Internet. Using the same edge for both ExpressRoute and the Internet will cause asymmetric routingand cause connectivity outages for your network.

Each ExpressRoute circuit has two states:

Service provider provisioning stateStatus

Status represents Microsoft's provisioning state. This property is set to Enabled when you create anExpressroute circuit

The connectivity provider provisioning state represents the state on the connectivity provider's side. It caneither be NotProvisioned, Provisioning, or Provisioned. The ExpressRoute circuit must be in Provisioned statefor you to be able to use it.

This section lists out the possible states for an ExpressRoute circuit.

At creation time

You will see the ExpressRoute circuit in the following state as soon as you run the PowerShell cmdlet to createthe ExpressRoute circuit.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-linkvnet-arm

ServiceProviderProvisioningState : NotProvisionedStatus : Enabled

ServiceProviderProvisioningState : ProvisioningStatus : Enabled

ServiceProviderProvisioningState : ProvisionedStatus : Enabled

ServiceProviderProvisioningState : NotProvisionedStatus : Enabled

IMPORTANTIMPORTANT

Routing session configuration state

When connectivity provider is in the process of provisioning the circuit

You will see the ExpressRoute circuit in the following state as soon as you pass the service key to theconnectivity provider and they have started the provisioning process.

When connectivity provider has completed the provisioning process

You will see the ExpressRoute circuit in the following state as soon as the connectivity provider has completedthe provisioning process.

Provisioned and Enabled is the only state the circuit can be in for you to be able to use it. If you are using aLayer 2 provider, you can configure routing for your circuit only when it is in this state.

When connectivity provider is deprovisioning the circuit

If you requested the service provider to deprovision the ExpressRoute circuit, you will see the circuit set to thefollowing state after the service provider has completed the deprovisioning process.

You can choose to re-enable it if needed, or run PowerShell cmdlets to delete the circuit.

If you run the PowerShell cmdlet to delete the circuit when the ServiceProviderProvisioningState is Provisioning orProvisioned the operation will fail. Please work with your connectivity provider to deprovision the ExpressRoute circuitfirst and then delete the circuit. Microsoft will continue to bill the circuit until you run the PowerShell cmdlet to delete thecircuit.

The BGP provisioning state lets you know if the BGP session has been enabled on the Microsoft edge. Thestate must be enabled for you to be able to use the peering.

It is important to check the BGP session state especially for Microsoft peering. In addition to the BGPprovisioning state, there is another state called advertised public prefixes state. The advertised public prefixesstate must be in configured state, both for the BGP session to be up and for your routing to work end-to-end.

If the advertised public prefix state is set to a validation needed state, the BGP session is not enabled, as theadvertised prefixes did not match the AS number in any of the routing registries.

IMPORTANTIMPORTANT

Next steps

If the advertised public prefixes state is in manual validation state, you must open a support ticket with Microsoftsupport and provide evidence that you own the IP addresses advertised along with the associated Autonomous Systemnumber.

Configure your ExpressRoute connection.


https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-arm

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-routing-arm


ExpressRoute routing requirements4/11/2018 • 10 min to read • Edit Online

NOTENOTE

IP addresses used for peerings

IP addresses used for Azure private peeringIP addresses used for Azure private peering

Example for private peeringExample for private peering

To connect to Microsoft cloud services using ExpressRoute, you’ll need to set up and manage routing. Someconnectivity providers offer setting up and managing routing as a managed service. Check with yourconnectivity provider to see if they offer this service. If they don't, you must adhere to the followingrequirements:

Refer to the Circuits and routing domains article for a description of the routing sessions that need to be set upin to facilitate connectivity.

Microsoft does not support any router redundancy protocols (for example, HSRP, VRRP) for high availabilityconfigurations. We rely on a redundant pair of BGP sessions per peering for high availability.

You need to reserve a few blocks of IP addresses to configure routing between your network and Microsoft'sEnterprise edge (MSEEs) routers. This section provides a list of requirements and describes the rules regardinghow these IP addresses must be acquired and used.

You can use either private IP addresses or public IP addresses to configure the peerings. The address rangeused for configuring routes must not overlap with address ranges used to create virtual networks in Azure.

You must reserve a /29 subnet or two /30 subnets for routing interfaces.The subnets used for routing can be either private IP addresses or public IP addresses.The subnets must not conflict with the range reserved by the customer for use in the Microsoft cloud.If a /29 subnet is used, it is split into two /30 subnets.

The first /30 subnet is used for the primary link and the second /30 subnet is used for the secondarylink.For each of the /30 subnets, you must use the first IP address of the /30 subnet on your router.Microsoft uses the second IP address of the /30 subnet to set up a BGP session.You must set up both BGP sessions for our availability SL A to be valid.

If you choose to use a.b.c.d/29 to set up the peering, it is split into two /30 subnets. In the example below, welook at how the a.b.c.d/29 subnet is used.

a.b.c.d/29 is split to a.b.c.d/30 and a.b.c.d+4/30 and passed down to Microsoft through the provisioning APIs.You use a.b.c.d+1 as the VRF IP for the Primary PE and Microsoft will consume a.b.c.d+2 as the VRF IP for theprimary MSEE. You use a.b.c.d+5 as the VRF IP for the secondary PE and Microsoft will use a.b.c.d+6 as theVRF IP for the secondary MSEE.

Consider a case where you select 192.168.100.128/29 to set up private peering. 192.168.100.128/29 includesaddresses from 192.168.100.128 to 192.168.100.135, among which:

192.168.100.128/30 will be assigned to link1, with provider using 192.168.100.129 and Microsoft using192.168.100.130.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-routing.md


IP addresses used for Azure public peeringIP addresses used for Azure public peering

IP addresses used for Microsoft peeringIP addresses used for Microsoft peering

Public IP address requirementPrivate peeringPrivate peering

Public peeringPublic peering

IMPORTANTIMPORTANT

192.168.100.132/30 will be assigned to link2, with provider using 192.168.100.133 and Microsoft using192.168.100.134.

You must use public IP addresses that you own for setting up the BGP sessions. Microsoft must be able toverify the ownership of the IP addresses through Routing Internet Registries and Internet Routing Registries.

You must use a unique /29 subnet or two /30 subnets to set up the BGP peering for each peering perExpressRoute circuit (if you have more than one).If a /29 subnet is used, it is split into two /30 subnets.

The first /30 subnet is used for the primary link and the second /30 subnet is used for the secondarylink.For each of the /30 subnets, you must use the first IP address of the /30 subnet on your router.Microsoft uses the second IP address of the /30 subnet to set up a BGP session.You must set up both BGP sessions for our availability SL A to be valid.

You must use public IP addresses that you own for setting up the BGP sessions. Microsoft must be able toverify the ownership of the IP addresses through Routing Internet Registries and Internet Routing Registries.

You must use a unique /29 (IPv4) or /125 (IPv6) subnet or two /30 (IPv4) or /126 (IPv6) subnets to set upthe BGP peering for each peering per ExpressRoute circuit (if you have more than one).If a /29 subnet is used, it is split into two /30 subnets.The first /30 subnet is used for the primary link and the second /30 subnet will be used for the secondarylink.For each of the /30 subnets, you must use the first IP address of the /30 subnet on your router. Microsoftuses the second IP address of the /30 subnet to set up a BGP session.If a /125 subnet is used, it is split into two /126 subnets.The first /126 subnet is used for the primary link and the second /126 subnet will be used for the secondarylink.For each of the /126 subnets, you must use the first IP address of the /126 subnet on your router. Microsoftuses the second IP address of the /126 subnet to set up a BGP session.You must set up both BGP sessions for our availability SL A to be valid.

You can choose to use public or private IPv4 addresses for private peering. We provide end-to-end isolation ofyour traffic, so overlapping of addresses with other customers is not possible in case of private peering. Theseaddresses are not advertised to Internet.

The Azure public peering path enables you to connect to all services hosted in Azure over their public IPaddresses. These include services listed in the ExpessRoute FAQ and any services hosted by ISVs on MicrosoftAzure. Connectivity to Microsoft Azure services on public peering is always initiated from your network into theMicrosoft network. You must use Public IP addresses for the traffic destined to Microsoft network.

All Azure PaaS services are also accessible through Microsoft peering. We recommend you to create Microsoft peeringand connect to Azure PaaS services over Microsoft peering.




IMPORTANTIMPORTANT

Dynamic route exchange

Autonomous System numbers

Route aggregation and prefix limits

A Private AS Number is allowed with Public Peering.

The Microsoft peering path lets you connect to Microsoft cloud services that are not supported through theAzure public peering path. The list of services includes Office 365 services, such as Exchange Online,SharePoint Online, Skype for Business, and Dynamics 365. Microsoft supports bi-directional connectivity onthe Microsoft peering. Traffic destined to Microsoft cloud services must use valid public IPv4 addresses beforethey enter the Microsoft network.

Make sure that your IP address and AS number are registered to you in one of the following registries:

ARINAPNICAFRINICL ACNICRIPENCCRADBALTDB

If your prefixes and AS number are not assigned to you in the preceding registries, you need to open a supportcase for manual validation of your prefixes and ASN. Support requires documentation, such as a Letter ofAuthorization, that proves you are allowed to use the resources.

A Private AS Number is allowed with Microsoft Peering, but will also require manual validation. In addition, weremove private AS numbers in the AS PATH for the received prefixes. As a result, you can't append private ASnumbers in the AS PATH to influence routing for Microsoft Peering.

Public IP addresses advertised to Microsoft over ExpressRoute must not be advertised to the Internet. This may breakconnectivity to other Microsoft services. However, Public IP addresses used by servers in your network that communicatewith O365 endpoints within Microsoft may be advertised over ExpressRoute.

Routing exchange will be over eBGP protocol. EBGP sessions are established between the MSEEs and yourrouters. Authentication of BGP sessions is not a requirement. If required, an MD5 hash can be configured. Seethe Configure routing and Circuit provisioning workflows and circuit states for information about configuringBGP sessions.

Microsoft uses AS 12076 for Azure public, Azure private and Microsoft peering. We have reserved ASNs from65515 to 65520 for internal use. Both 16 and 32 bit AS numbers are supported.

There are no requirements around data transfer symmetry. The forward and return paths may traversedifferent router pairs. Identical routes must be advertised from either sides across multiple circuit pairsbelonging you. Route metrics are not required to be identical.

We support up to 4000 prefixes advertised to us through the Azure private peering. This can be increased up to10,000 prefixes if the ExpressRoute premium add-on is enabled. We accept up to 200 prefixes per BGP sessionfor Azure public and Microsoft peering.

https://www.arin.net/

https://www.apnic.net/

https://www.afrinic.net/

http://www.lacnic.net/

https://www.ripe.net/

http://www.radb.net/

http://altdb.net/

Transit routing and cross-region routing

Advertising default routes

NOTENOTE

Support for BGP communities

MICROSOFT AZURE REGION BGP COMMUNITY VALUE

North America

The BGP session is dropped if the number of prefixes exceeds the limit. We will accept default routes on theprivate peering link only. Provider must filter out default route and private IP addresses (RFC 1918) from theAzure public and Microsoft peering paths.

ExpressRoute cannot be configured as transit routers. You will have to rely on your connectivity provider fortransit routing services.

Default routes are permitted only on Azure private peering sessions. In such a case, we will route all traffic fromthe associated virtual networks to your network. Advertising default routes into private peering will result inthe internet path from Azure being blocked. You must rely on your corporate edge to route traffic from and tothe internet for services hosted in Azure.

To enable connectivity to other Azure services and infrastructure services, you must make sure one of thefollowing items is in place:

Azure public peering is enabled to route traffic to public endpointsYou use user-defined routing to allow internet connectivity for every subnet requiring Internet connectivity.

Advertising default routes will break Windows and other VM license activation. Follow instructions here to work aroundthis.

This section provides an overview of how BGP communities will be used with ExpressRoute. Microsoft willadvertise routes in the public and Microsoft peering paths with routes tagged with appropriate communityvalues. The rationale for doing so and the details on community values are described below. Microsoft, however,will not honor any community values tagged to routes advertised to Microsoft.

If you are connecting to Microsoft through ExpressRoute at any one peering location within a geopoliticalregion, you will have access to all Microsoft cloud services across all regions within the geopolitical boundary.

For example, if you connected to Microsoft in Amsterdam through ExpressRoute, you will have access to allMicrosoft cloud services hosted in North Europe and West Europe.

Refer to the ExpressRoute partners and peering locations page for a detailed list of geopolitical regions,associated Azure regions, and corresponding ExpressRoute peering locations.

You can purchase more than one ExpressRoute circuit per geopolitical region. Having multiple connectionsoffers you significant benefits on high availability due to geo-redundancy. In cases where you have multipleExpressRoute circuits, you will receive the same set of prefixes advertised from Microsoft on the public peeringand Microsoft peering paths. This means you will have multiple paths from your network into Microsoft. Thiscan potentially cause suboptimal routing decisions to be made within your network. As a result, you mayexperience suboptimal connectivity experiences to different services. You can rely on the community values tomake appropriate routing decisions to offer optimal routing to users.

http://blogs.msdn.com/b/mast/archive/2015/05/20/use-azure-custom-routes-to-enable-kms-activation-with-forced-tunneling.aspx

East US 12076:51004

East US 2 12076:51005

West US 12076:51006

West US 2 12076:51026

West Central US 12076:51027

North Central US 12076:51007

South Central US 12076:51008

Central US 12076:51009

Canada Central 12076:51020

Canada East 12076:51021

South America

Brazil South 12076:51014

Europe

North Europe 12076:51003

West Europe 12076:51002

UK South 12076:51024

UK West 12076:51025

France Central 12076:51030

France South 12076:51031

Asia Pacific

East Asia 12076:51010

Southeast Asia 12076:51011

Japan

Japan East 12076:51012

Japan West 12076:51013


Australia

Australia Central 12076:51032

Australia Central 2 12076:51033

Australia East 12076:51015

Australia Southeast 12076:51016

India

India South 12076:51019

India West 12076:51018

India Central 12076:51017

Korea

Korea South 12076:51028

Korea Central 12076:51029


IMPORTANTIMPORTANT

SERVICE BGP COMMUNITY VALUE

Exchange Online 12076:5010

SharePoint Online 12076:5020

Skype For Business Online 12076:5030

Dynamics 365 12076:5040

Other Office 365 Online services 12076:5100

NOTENOTE

BGP Community support in National Clouds (Preview)BGP Community support in National Clouds (Preview)

All routes advertised from Microsoft will be tagged with the appropriate community value.

Global prefixes are tagged with an appropriate community value.

In addition to the above, Microsoft will also tag prefixes based on the service they belong to. This applies onlyto the Microsoft peering. The table below provides a mapping of service to BGP community value.

Microsoft does not honor any BGP community values that you set on the routes advertised to Microsoft.

NATIONAL CLOUDS AZURE REGION BGP COMMUNITY VALUE

US Government

US Gov Arizona 12076:51106

US Gov Iowa 12076:51109

US Gov Virginia 12076:51105

US Gov Texas 12076:51108

US DoD Central 12076:51209

US DoD East 12076:51205

SERVICE IN NATIONAL CLOUDS BGP COMMUNITY VALUE

US Government

Exchange Online 12076:5110

SharePoint Online 12076:5120

Skype For Business Online 12076:5130

Dynamics 365 12076:5140

Other Office 365 Online services 12076:5200

Next stepsConfigure your ExpressRoute connection.

Create and modify a circuitCreate and modify peering configurationLink a VNet to an ExpressRoute circuit

ExpressRoute QoS requirements7/26/2017 • 1 min to read • Edit Online

NOTENOTE

TRAFFIC CLASS TREATMENT (DSCP MARKING) SKYPE FOR BUSINESS WORKLOADS

Voice EF (46) Skype / Lync voice

Interactive AF41 (34) Video, VBSS

AF21 (18) App sharing

Default AF11 (10) File transfer

CS0 (0) Anything else

Skype for Business has various workloads that require differentiated QoS treatment. If you plan to consume voiceservices through ExpressRoute, you should adhere to the requirements described below.

QoS requirements apply to the Microsoft peering only. The DSCP values in your network traffic received on Azure publicpeering and Azure private peering will be reset to 0.

The following table provides a list of DSCP markings used by Skype for Business. Refer to Managing QoS forSkype for Business for more information.

You should classify the workloads and mark the right DSCP values. Follow the guidance provided here on howto set DSCP markings in your network.You should configure and support multiple QoS queues within your network. Voice must be a standalone classand receive the EF treatment specified in RFC 3246.You can decide the queuing mechanism, congestion detection policy, and bandwidth allocation per traffic class.But, the DSCP marking for Skype for Business workloads must be preserved. If you are using DSCP markingsnot listed above, e.g. AF31 (26), you must rewrite this DSCP value to 0 before sending the packet to Microsoft.Microsoft only sends packets marked with the DSCP value shown in the above table.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-qos.md

https://technet.microsoft.com/library/gg405409.aspx

https://technet.microsoft.com/library/gg405409.aspx

Next stepsRefer to the requirements for Routing and NAT.See the following links to configure your ExpressRoute connection.


Moving ExpressRoute circuits from the classic to theResource Manager deployment model6/27/2017 • 6 min to read • Edit Online

ExpressRoute circuits that are created in the classic deployment model

ExpressRoute circuits that are created in the Resource Managerdeployment model

This article provides an overview of what it means to move an Azure ExpressRoute circuit from the classic to theAzure Resource Manager deployment model.

You can use a single ExpressRoute circuit to connect to virtual networks that are deployed both in the classic andthe Resource Manager deployment models. An ExpressRoute circuit, regardless of how it is created, can now linkto virtual networks across both deployment models.

ExpressRoute circuits that are created in the classic deployment model need to be moved to the ResourceManager deployment model first to enable connectivity to both the classic and the Resource Managerdeployment models. There isn't connectivity loss or disruption when a connection is being moved. All circuit-to-virtual network links in the classic deployment model (within the same subscription and cross-subscription) arepreserved.

After the move is completed successfully, the ExpressRoute circuit looks, performs, and feels exactly like anExpressRoute circuit that was created in the Resource Manager deployment model. You can now createconnections to virtual networks in the Resource Manager deployment model.

After an ExpressRoute circuit has been moved to the Resource Manager deployment model, you can manage thelife cycle of the ExpressRoute circuit only by using the Resource Manager deployment model. This means that youcan perform operations like adding/updating/deleting peerings, updating circuit properties (such as bandwidth,SKU, and billing type), and deleting circuits only in the Resource Manager deployment model. Refer to the sectionbelow on circuits that are created in the Resource Manager deployment model for further details on how you canmanage access to both deployment models.

You do not have to involve your connectivity provider to perform the move.

You can enable ExpressRoute circuits that are created in the Resource Manager deployment model to be

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-move.md

IMPORTANTIMPORTANT

Controlling access to the classic deployment model

Supported operations in the classic deployment model

Communication between the classic and the Resource Managerdeployment models

accessible from both deployment models. Any ExpressRoute circuit in your subscription can be enabled to beaccessed from both deployment models.

ExpressRoute circuits that were created in the Resource Manager deployment model do not have access to theclassic deployment model by default.ExpressRoute circuits that have been moved from the classic deployment model to the Resource managerdeployment model are accessible from both deployment models by default.An ExpressRoute circuit always has access to the Resource Manager deployment model, regardless of whetherit was created in the Resource Manager or classic deployment model. This means that you can createconnections to virtual networks created in the Resource Manager deployment model by following instructionson how to link virtual networks.Access to the classic deployment model is controlled by the allowClassicOperations parameter in theExpressRoute circuit.

All quotas that are documented on the service limits page apply. As an example, a standard circuit can have at most 10virtual network links/connections across both the classic and the Resource Manager deployment models.

You can enable a single ExpressRoute circuit to link to virtual networks in both deployment models by setting theallowClassicOperations parameter of the ExpressRoute circuit.

Setting allowClassicOperations to TRUE enables you to link virtual networks from both deployment models tothe ExpressRoute circuit. You can link to virtual networks in the classic deployment model by following guidanceon how to link virtual networks in the classic deployment model. You can link to virtual networks in the ResourceManager deployment model by following guidance on how to link virtual networks in the Resource Managerdeployment model.

Setting allowClassicOperations to FALSE blocks access to the circuit from the classic deployment model.However, all virtual network links in the classic deployment model are preserved. In this case, the ExpressRoutecircuit is not visible in the classic deployment model.

The following classic operations are supported on an ExpressRoute circuit when allowClassicOperations is setto TRUE:

Get ExpressRoute circuit informationCreate/update/get/delete virtual network links to classic virtual networksCreate/update/get/delete virtual network link authorizations for cross-subscription connectivity

You cannot perform the following classic operations when allowClassicOperations is set to TRUE:

Create/update/get/delete Border Gateway Protocol (BGP) peerings for Azure private, Azure public, andMicrosoft peeringsDelete ExpressRoute circuits

The ExpressRoute circuit acts like a bridge between the classic and the Resource Manager deployment models.

Access to Azure public and Microsoft peering resources

What's supported

What's not supported

Configuration

Next steps

Traffic between virtual machines in virtual networks in the classic deployment model and those in virtualnetworks in the Resource Manager deployment model flows through ExpressRoute if both virtual networks arelinked to the same ExpressRoute circuit.

Aggregate throughput is limited by the throughput capacity of the virtual network gateway. Traffic does not enterthe connectivity provider's networks or your networks in such cases. Traffic flow between the virtual networks isfully contained within the Microsoft network.

You can continue to access resources that are typically accessible through Azure public peering and Microsoftpeering without any disruption.

This section describes what's supported for ExpressRoute circuits:

You can use a single ExpressRoute circuit to access virtual networks that are deployed in the classic and theResource Manager deployment models.You can move an ExpressRoute circuit from the classic to the Resource Manager deployment model. After it ismoved, the ExpressRoute circuit looks, feels, and performs like any other ExpressRoute circuit that is created inthe Resource Manager deployment model.You can move only the ExpressRoute circuit. Circuit links, virtual networks, and VPN gateways cannot bemoved through this operation.After an ExpressRoute circuit has been moved to the Resource Manager deployment model, you can managethe life cycle of the ExpressRoute circuit only by using the Resource Manager deployment model. This meansthat you can perform operations like adding/updating/deleting peerings, updating circuit properties (such asbandwidth, SKU, and billing type), and deleting circuits only in the Resource Manager deployment model.The ExpressRoute circuit acts like a bridge between the classic and the Resource Manager deployment models.Traffic between virtual machines in virtual networks in the classic deployment model and those in virtualnetworks in the Resource Manager deployment model flows through ExpressRoute if both virtual networksare linked to the same ExpressRoute circuit.Cross-subscription connectivity is supported in both the classic and the Resource Manager deploymentmodels.After you move an ExpressRoute circuit from the classic model to the Azure Resource Manager model, you canmigrate the virtual networks linked to the ExpressRoute circuit.

This section describes what's not supported for ExpressRoute circuits:

Managing the life cycle of an ExpressRoute circuit from the classic deployment model.Role-Based Access Control (RBAC) support for the classic deployment model. You cannot perform RBACcontrols to a circuit in the classic deployment model. Any administrator/coadministrator of the subscriptioncan link or unlink virtual networks to the circuit.

Follow the instructions that are described in Move an ExpressRoute circuit from the classic to the ResourceManager deployment model.

Migrate the virtual networks linked to the ExpressRoute circuit from the classic model to the Azure ResourceManager modelFor workflow information, see ExpressRoute circuit provisioning workflows and circuit states.To configure your ExpressRoute connection:

Create an ExpressRoute circuitConfigure routingLink a virtual network to an ExpressRoute circuit

Create and modify an ExpressRoute circuit2/16/2018 • 5 min to read • Edit Online

Before you begin

Create and provision an ExpressRoute circuit1. Sign in to the Azure portal1. Sign in to the Azure portal

2. Create a new ExpressRoute circuit2. Create a new ExpressRoute circuit

IMPORTANTIMPORTANT

This article describes how to create an Azure ExpressRoute circuit by using the Azure portal and the AzureResource Manager deployment model. The following steps also show you how to check the status of the circuit,update it, or delete and deprovision it.

Review the prerequisites and workflows before you begin configuration.Ensure that you have access to the Azure portal.Ensure that you have permissions to create new networking resources. Contact your account administrator ifyou do not have the right permissions.You can view a video before beginning in order to better understand the steps.

From a browser, navigate to the Azure portal and sign in with your Azure account.

Your ExpressRoute circuit is billed from the moment a service key is issued. Ensure that you perform this operation whenthe connectivity provider is ready to provision the circuit.

1. You can create an ExpressRoute circuit by selecting the option to create a new resource. Click Create aresource > Networking > ExpressRoute, as shown in the following image:

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-circuit-portal-resource-manager.md

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-prerequisites

https://portal.azure.com

http://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-an-expressroute-circuit

http://portal.azure.com

2. After you click ExpressRoute, you'll see the Create ExpressRoute circuit page. When you're filling inthe values on this page, make sure that you specify the correct SKU tier (Standard, or Premium) and datametering billing model (Unlimited or Metered).

3. View the circuits and properties3. View the circuits and properties

Tier determines whether an ExpressRoute standard or an ExpressRoute premium add-on is enabled.You can specify Standard to get the standard SKU or Premium for the premium add-on.Data metering determines the billing type. You can specify Metered for a metered data plan andUnlimited for an unlimited data plan. Note that you can change the billing type from Metered toUnlimited, but you can't change the type from Unlimited to Metered.

IMPORTANTIMPORTANT

Peering Location is the physical location where you are peering with Microsoft.

The Peering Location indicates the physical location where you are peering with Microsoft. This is notlinked to "Location" property, which refers to the geography where the Azure Network Resource Provider islocated. While they are not related, it is a good practice to choose a Network Resource Providergeographically close to the Peering Location of the circuit.

View all the circuits

You can view all the circuits that you created by selecting All resources on the left-side menu.


4. Send the service key to your connectivity provider for provisioning4. Send the service key to your connectivity provider for provisioning

View the properties

You can view the properties of the circuit by selecting it. On the Overview page for your circuit, the service keyappears in the service key field. You must copy the service key for your circuit and pass it down to the serviceprovider to complete the provisioning process. The circuit service key is specific to your circuit.

On this page, Provider status provides information on the current state of provisioning on the service-providerside. Circuit status provides the state on the Microsoft side. For more information about circuit provisioningstates, see the Workflows article.

When you create a new ExpressRoute circuit, the circuit is in the following state:

Provider status: Not provisionedCircuit status: Enabled

The circuit changes to the following state when the connectivity provider is in the process of enabling it for you:

Provider status: ProvisioningCircuit status: Enabled

For you to be able to use an ExpressRoute circuit, it must be in the following state:

Provider status: ProvisionedCircuit status: Enabled

5. Periodically check the status and the state of the circuit key5. Periodically check the status and the state of the circuit key

6. Create your routing configuration6. Create your routing configuration

IMPORTANTIMPORTANT

7. Link a virtual network to an ExpressRoute circuit7. Link a virtual network to an ExpressRoute circuit

Getting the status of an ExpressRoute circuit

Modifying an ExpressRoute circuit

IMPORTANTIMPORTANT

You can view the properties of the circuit that you're interested in by selecting it. Check the Provider status andensure that it has moved to Provisioned before you continue.

For step-by-step instructions, refer to the ExpressRoute circuit routing configuration article to create and modifycircuit peerings.

These instructions only apply to circuits that are created with service providers that offer layer 2 connectivity services. Ifyou're using a service provider that offers managed layer 3 services (typically an IP VPN, like MPLS), your connectivityprovider configures and manages routing for you.

Next, link a virtual network to your ExpressRoute circuit. Use the Linking virtual networks to ExpressRoutecircuits article when you work with the Resource Manager deployment model.

You can view the status of a circuit by selecting it and viewing the Overview page.

You can modify certain properties of an ExpressRoute circuit without impacting connectivity. You can modify thebandwidth, SKU, billing model and allow classic operations on the Configuration page. For information onlimits and limitations, see the ExpressRoute FAQ.

You can perform the following tasks with no downtime:

Enable or disable an ExpressRoute Premium add-on for your ExpressRoute circuit.Increase the bandwidth of your ExpressRoute circuit, provided there is capacity available on the port.Downgrading the bandwidth of a circuit is not supported.Change the metering plan from Metered Data to Unlimited Data. Changing the metering plan fromUnlimited Data to Metered Data is not supported.You can enable and disable Allow Classic Operations.

You may have to recreate the ExpressRoute circuit if there is inadequate capacity on the existing port. You cannot upgradethe circuit if there is no additional capacity available at that location.

Although you can seamlessly upgrade the bandwidth, you cannot reduce the bandwidth of an ExpressRoute circuitwithout disruption. Downgrading bandwidth requires you to deprovision the ExpressRoute circuit and then reprovision anew ExpressRoute circuit.

Disabling the Premium add-on operation can fail if you're using resources that are greater than what is permitted for thestandard circuit.

To modify an ExpressRoute circuit, click Configuration.




Deprovisioning and deleting an ExpressRoute circuit

Next steps

You can delete your ExpressRoute circuit by selecting the delete icon. Note the following information:

You must unlink all virtual networks from the ExpressRoute circuit. If this operation fails, check whether anyvirtual networks are linked to the circuit.If the ExpressRoute circuit service provider provisioning state is Provisioning or Provisioned you mustwork with your service provider to deprovision the circuit on their side. We continue to reserve resourcesand bill you until the service provider completes deprovisioning the circuit and notifies us.If the service provider has deprovisioned the circuit (the service provider provisioning state is set to Notprovisioned), you can delete the circuit. This stops billing for the circuit.

After you create your circuit, continue with the following next steps:

Create and modify routing for your ExpressRoute circuitLink your virtual network to your ExpressRoute circuit



Create and modify an ExpressRoute circuit usingPowerShell10/20/2017 • 9 min to read • Edit Online

Before you begin

Create and provision an ExpressRoute circuit1. Sign in to your Azure account and select your subscription1. Sign in to your Azure account and select your subscription

Login-AzureRmAccount

Get-AzureRmSubscription

Select-AzureRmSubscription -SubscriptionId "<subscription ID>"

2. Get the list of supported providers, locations, and bandwidths2. Get the list of supported providers, locations, and bandwidths

Get-AzureRmExpressRouteServiceProvider

This article describes how to create an Azure ExpressRoute circuit by using PowerShell cmdlets and the AzureResource Manager deployment model. This article also shows you how to check the status of the circuit, updateit, or delete and deprovision it.

Install the latest version of the Azure Resource Manager PowerShell cmdlets. For more information, seeOverview of Azure PowerShell.Review the prerequisites and workflows before you begin configuration.

To begin your configuration, sign in to your Azure account. Use the following examples to help you connect:

Check the subscriptions for the account:

Select the subscription that you want to create an ExpressRoute circuit for :

Before you create an ExpressRoute circuit, you need the list of supported connectivity providers, locations, andbandwidth options.

The PowerShell cmdlet Get-AzureRmExpressRouteServiceProvider returns this information, which youâ€™lluse in later steps:

Check to see if your connectivity provider is listed there. Make a note of the following information, which youneed later when you create a circuit:

NamePeeringLocationsBandwidthsOffered

You're now ready to create an ExpressRoute circuit.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-circuit-arm.md

https://docs.microsoft.com/powershell/azure/overview

3. Create an ExpressRoute circuit3. Create an ExpressRoute circuit

New-AzureRmResourceGroup -Name "ExpressRouteResourceGroup" -Location "West US"

New-AzureRmExpressRouteCircuit -Name "ExpressRouteARMCircuit" -ResourceGroupName "ExpressRouteResourceGroup" -Location "West US" -SkuTier Standard -SkuFamily MeteredData -ServiceProviderName "Equinix" -PeeringLocation "Silicon Valley" -BandwidthInMbps 200

IMPORTANTIMPORTANT

get-help New-AzureRmExpressRouteCircuit -detailed

4. List all ExpressRoute circuits4. List all ExpressRoute circuits

Get-AzureRmExpressRouteCircuit -Name "ExpressRouteARMCircuit" -ResourceGroupName "ExpressRouteResourceGroup"

If you don't already have a resource group, you must create one before you create your ExpressRoute circuit. Youcan do so by running the following command:

The following example shows how to create a 200-Mbps ExpressRoute circuit through Equinix in Silicon Valley. Ifyou're using a different provider and different settings, substitute that information when you make your request.Use the following example to request a new service key:

Make sure that you specify the correct SKU tier and SKU family:

SKU tier determines whether an ExpressRoute standard or an ExpressRoute premium add-on is enabled. Youcan specify Standard to get the standard SKU or Premium for the premium add-on.SKU family determines the billing type. You can specify Metereddata for a metered data plan andUnlimiteddata for an unlimited data plan. You can change the billing type from Metereddata toUnlimiteddata, but you can't change the type from Unlimiteddata to Metereddata.

Your ExpressRoute circuit is billed from the moment a service key is issued. Ensure that you perform this operation whenthe connectivity provider is ready to provision the circuit.

The response contains the service key. You can get detailed descriptions of all the parameters by running thefollowing command:

To get a list of all the ExpressRoute circuits that you created, run the Get-AzureRmExpressRouteCircuitcommand:

The response looks similar to the following example:

Name : ExpressRouteARMCircuitResourceGroupName : ExpressRouteResourceGroupLocation : westusId : /subscriptions/***************************/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/ExpressRouteARMCircuitEtag : W/"################################"ProvisioningState : SucceededSku : { "Name": "Standard_MeteredData", "Tier": "Standard", "Family": "MeteredData" }CircuitProvisioningState : EnabledServiceProviderProvisioningState : NotProvisionedServiceProviderNotes :ServiceProviderProperties : { "ServiceProviderName": "Equinix", "PeeringLocation": "Silicon Valley", "BandwidthInMbps": 200 }ServiceKey : **************************************Peerings : []

Get-AzureRmExpressRouteCircuit

Name : ExpressRouteARMCircuitResourceGroupName : ExpressRouteResourceGroupLocation : westusId : /subscriptions/***************************/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/ExpressRouteARMCircuitEtag : W/"################################"ProvisioningState : SucceededSku : { "Name": "Standard_MeteredData", "Tier": "Standard", "Family": "MeteredData" }CircuitProvisioningState : EnabledServiceProviderProvisioningState : NotProvisionedServiceProviderNotes :ServiceProviderProperties : { "ServiceProviderName": "Equinix", "PeeringLocation": "Silicon Valley", "BandwidthInMbps": 200 }ServiceKey : **************************************Peerings : []

get-help Get-AzureRmExpressRouteCircuit -detailed


You can retrieve this information at any time by using the Get-AzureRmExpressRouteCircuit cmdlet. Making thecall with no parameters lists all the circuits. Your service key is listed in the ServiceKey field:


You can get detailed descriptions of all the parameters by running the following command:

ServiceProviderProvisioningState : NotProvisionedCircuitProvisioningState : Enabled

ServiceProviderProvisioningState : ProvisioningStatus : Enabled

ServiceProviderProvisioningState : ProvisionedCircuitProvisioningState : Enabled



Name : ExpressRouteARMCircuitResourceGroupName : ExpressRouteResourceGroupLocation : westusId : /subscriptions/***************************/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/ExpressRouteARMCircuitEtag : W/"################################"ProvisioningState : SucceededSku : { "Name": "Standard_MeteredData", "Tier": "Standard", "Family": "MeteredData" }CircuitProvisioningState : EnabledServiceProviderProvisioningState : ProvisionedServiceProviderNotes :ServiceProviderProperties : { "ServiceProviderName": "Equinix", "PeeringLocation": "Silicon Valley", "BandwidthInMbps": 200 }ServiceKey : **************************************Peerings : []


ServiceProviderProvisioningState provides information about the current state of provisioning on the service-provider side. Status provides the state on the Microsoft side. For more information about circuit provisioningstates, see Workflows.




Checking the status and the state of the circuit key lets you know when your provider has enabled your circuit.After the circuit has been configured, ServiceProviderProvisioningState appears as Provisioned, as shown in thefollowing example:


For step-by-step instructions, see the ExpressRoute circuit routing configuration article to create and modifycircuit peerings.

IMPORTANTIMPORTANT


Getting the status of an ExpressRoute circuit

Get-AzureRmExpressRouteCircuit




Next, link a virtual network to your ExpressRoute circuit. Use the Linking virtual networks to ExpressRoutecircuits article when you work with the Resource Manager deployment model.

You can retrieve this information at any time by using the Get-AzureRmExpressRouteCircuit cmdlet. Makingthe call with no parameters lists all the circuits.

The response is similar to the following example:

You can get information on a specific ExpressRoute circuit by passing the resource group name and circuit nameas a parameter to the call:



get-help get-azurededicatedcircuit -detailed


To enable the ExpressRoute premium add-onTo enable the ExpressRoute premium add-on

$ckt = Get-AzureRmExpressRouteCircuit -Name "ExpressRouteARMCircuit" -ResourceGroupName "ExpressRouteResourceGroup"

$ckt.Sku.Tier = "Premium"$ckt.sku.Name = "Premium_MeteredData"

Set-AzureRmExpressRouteCircuit -ExpressRouteCircuit $ckt

You can get detailed descriptions of all the parameters by running the following command:

You can modify certain properties of an ExpressRoute circuit without impacting connectivity.

You can do the following tasks with no downtime:

Enable or disable an ExpressRoute premium add-on for your ExpressRoute circuit.Increase the bandwidth of your ExpressRoute circuit provided there is capacity available on the port.Downgrading the bandwidth of a circuit is not supported.Change the metering plan from Metered Data to Unlimited Data. Changing the metering plan fromUnlimited Data to Metered Data is not supported.You can enable and disable Allow Classic Operations.

For more information on limits and limitations, see the ExpressRoute FAQ.

You can enable the ExpressRoute premium add-on for your existing circuit by using the following PowerShellsnippet:

The circuit now has the ExpressRoute premium add-on features enabled. We begin billing you for the premiumadd-on capability as soon as the command has successfully run.

To disable the ExpressRoute premium add-onTo disable the ExpressRoute premium add-on

IMPORTANTIMPORTANT


$ckt.Sku.Tier = "Standard"$ckt.sku.Name = "Standard_MeteredData"


To update the ExpressRoute circuit bandwidthTo update the ExpressRoute circuit bandwidth

IMPORTANTIMPORTANT


$ckt.ServiceProviderProperties.BandwidthInMbps = 1000


To move the SKU from metered to unlimitedTo move the SKU from metered to unlimited

If you're using resources that are greater than what is permitted for the standard circuit, this operation can fail.

Note the following information:

Before you downgrade from premium to standard, you must ensure that the number of virtual networks thatare linked to the circuit is less than 10. If you don't, your update request fails, and we bill you at premiumrates.You must unlink all virtual networks in other geopolitical regions. If you don't do this, your update requestfails, and we bill you at premium rates.Your route table must be less than 4,000 routes for private peering. If your route table size is greater than4,000 routes, the BGP session drops and won't be reenabled until the number of advertised prefixes goesbelow 4,000.

You can disable the ExpressRoute premium add-on for the existing circuit by using the following PowerShellcmdlet:

For supported bandwidth options for your provider, check the ExpressRoute FAQ. You can pick any size greaterthan the size of your existing circuit.


You cannot reduce the bandwidth of an ExpressRoute circuit without disruption. Downgrading bandwidth requires you todeprovision the ExpressRoute circuit and then reprovision a new ExpressRoute circuit.

After you decide what size you need, use the following command to resize your circuit:

Your circuit will be sized up on the Microsoft side. Then you must contact your connectivity provider to updateconfigurations on their side to match this change. After you make this notification, we will begin billing you forthe updated bandwidth option.

You can change the SKU of an ExpressRoute circuit by using the following PowerShell snippet:


$ckt.Sku.Family = "UnlimitedData"$ckt.sku.Name = "Premium_UnlimitedData"


To control access to the classic and Resource Manager environmentsTo control access to the classic and Resource Manager environments


Remove-AzureRmExpressRouteCircuit -ResourceGroupName "ExpressRouteResourceGroup" -Name "ExpressRouteARMCircuit"

Next steps

Review the instructions in Move ExpressRoute circuits from the classic to the Resource Manager deploymentmodel.

Note the following information:

You must unlink all virtual networks from the ExpressRoute circuit. If this operation fails, check to see if anyvirtual networks are linked to the circuit.If the ExpressRoute circuit service provider provisioning state is Provisioning or Provisioned you mustwork with your service provider to deprovision the circuit on their side. We continue to reserve resources andbill you until the service provider completes deprovisioning the circuit and notifies us.If the service provider has deprovisioned the circuit (the service provider provisioning state is set to Notprovisioned), you can delete the circuit. This stops billing for the circuit.

You can delete your ExpressRoute circuit by running the following command:

After you create your circuit, make sure that you do the following next steps:


Create and modify an ExpressRoute circuit using CLI10/20/2017 • 8 min to read • Edit Online

Before you begin

Create and provision an ExpressRoute circuit1. Sign in to your Azure account and select your subscription1. Sign in to your Azure account and select your subscription

az login

az account list

az account set --subscription "<subscription ID>"

2. Get the list of supported providers, locations, and bandwidths2. Get the list of supported providers, locations, and bandwidths

az network express-route list-service-providers

This article describes how to create an Azure ExpressRoute circuit by using the Command Line Interface (CLI).This article also shows you how to check the status, update, or delete and deprovision a circuit. If you want to usea different method to work with ExpressRoute circuits, you can select the article from the following list:

Before beginning, install the latest version of the CLI commands (2.0 or later). For information about installingthe CLI commands, see Install Azure CLI 2.0 and Get Started with Azure CLI 2.0.Review the prerequisites and workflows before you begin configuration.


Check the subscriptions for the account.

Select the subscription for which you want to create an ExpressRoute circuit.

Before you create an ExpressRoute circuit, you need the list of supported connectivity providers, locations, andbandwidth options. The CLI command 'az network express-route list-service-providers' returns this information,which youâ€™ll use in later steps:


https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/howto-circuit-cli.md

https://docs.microsoft.com/cli/azure/install-azure-cli

https://docs.microsoft.com/cli/azure/get-started-with-azure-cli

[ { "bandwidthsOffered": [ { "offerName": "50Mbps", "valueInMbps": 50 }, { "offerName": "100Mbps", "valueInMbps": 100 }, { "offerName": "200Mbps", "valueInMbps": 200 }, { "offerName": "500Mbps", "valueInMbps": 500 }, { "offerName": "1Gbps", "valueInMbps": 1000 }, { "offerName": "2Gbps", "valueInMbps": 2000 }, { "offerName": "5Gbps", "valueInMbps": 5000 }, { "offerName": "10Gbps", "valueInMbps": 10000 } ], "id": "/subscriptions//resourceGroups//providers/Microsoft.Network/expressRouteServiceProviders/", "location": null, "name": "AARNet", "peeringLocations": [ "Melbourne", "Sydney" ], "provisioningState": "Succeeded", "resourceGroup": "", "tags": null, "type": "Microsoft.Network/expressRouteServiceProviders" },

3. Create an ExpressRoute circuit3. Create an ExpressRoute circuit

Check the response to see if your connectivity provider is listed. Make a note of the following information, whichyou will need when you create a circuit:

NamePeeringLocationsBandwidthsOffered

You're now ready to create an ExpressRoute circuit.

IMPORTANTIMPORTANT

az group create -n ExpressRouteResourceGroup -l "West US"

az network express-route create --bandwidth 200 -n MyCircuit --peering-location "Silicon Valley" -g ExpressRouteResourceGroup --provider "Equinix" -l "West US" --sku-family MeteredData --sku-tier Standard

4. List all ExpressRoute circuits4. List all ExpressRoute circuits

az network express-route list

Your ExpressRoute circuit is billed from the moment a service key is issued. Perform this operation when the connectivityprovider is ready to provision the circuit.

If you don't already have a resource group, you must create one before you create your ExpressRoute circuit. Youcan create a resource group by running the following command:

The following example shows how to create a 200-Mbps ExpressRoute circuit through Equinix in Silicon Valley. Ifyou're using a different provider and different settings, substitute that information when you make your request.

Make sure that you specify the correct SKU tier and SKU family:

SKU tier determines whether an ExpressRoute standard or an ExpressRoute premium add-on is enabled. Youcan specify 'Standard' to get the standard SKU or 'Premium' for the premium add-on.SKU family determines the billing type. You can specify 'Metereddata' for a metered data plan and'Unlimiteddata' for an unlimited data plan. You can change the billing type from 'Metereddata' to'Unlimiteddata', but you can't change the type from 'Unlimiteddata' to 'Metereddata'.

Your ExpressRoute circuit is billed from the moment a service key is issued. The following example is a request fora new service key:

The response contains the service key.

To get a list of all the ExpressRoute circuits that you created, run the 'az network express-route list' command. Youcan retrieve this information at any time by using this command. To list all circuits, make the call with noparameters.

Your service key is listed in the ServiceKey field of the response.

"allowClassicOperations": false,"authorizations": [],"circuitProvisioningState": "Enabled","etag": "W/\"1262c492-ffef-4a63-95a8-a6002736b8c4\"","gatewayManagerEtag": null,"id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit","location": "westus","name": "MyCircuit","peerings": [],"provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup","serviceKey": "1d05cf70-1db5-419f-ad86-1ca62c3c125b","serviceProviderNotes": null,"serviceProviderProperties": { "bandwidthInMbps": 200, "peeringLocation": "Silicon Valley", "serviceProviderName": "Equinix"},"serviceProviderProvisioningState": "NotProvisioned","sku": { "family": "UnlimitedData", "name": "Standard_MeteredData", "tier": "Standard"},"tags": null,"type": "Microsoft.Network/expressRouteCircuits]

az network express-route list -h


"serviceProviderProvisioningState": "NotProvisioned""circuitProvisioningState": "Enabled"

"serviceProviderProvisioningState": "Provisioning""circuitProvisioningState": "Enabled"

"serviceProviderProvisioningState": "Provisioned""circuitProvisioningState": "Enabled


You can get detailed descriptions of all the parameters by running the command using the '-h' parameter.

'ServiceProviderProvisioningState' provides information about the current state of provisioning on the service-provider side. The status provides the state on the Microsoft side. For more information, see the Workflowsarticle.




Checking the status and the state of the circuit key lets you know when your provider has enabled your circuit.After the circuit has been configured, 'ServiceProviderProvisioningState' appears as 'Provisioned', as shown inthe following example:

az network express-route show --resource-group ExpressRouteResourceGroup --name MyCircuit

"allowClassicOperations": false,"authorizations": [],"circuitProvisioningState": "Enabled","etag": "W/\"1262c492-ffef-4a63-95a8-a6002736b8c4\"","gatewayManagerEtag": null,"id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit","location": "westus","name": "MyCircuit","peerings": [],"provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup","serviceKey": "1d05cf70-1db5-419f-ad86-1ca62c3c125b","serviceProviderNotes": null,"serviceProviderProperties": { "bandwidthInMbps": 200, "peeringLocation": "Silicon Valley", "serviceProviderName": "Equinix"},"serviceProviderProvisioningState": "NotProvisioned","sku": { "family": "UnlimitedData", "name": "Standard_MeteredData", "tier": "Standard"},"tags": null,"type": "Microsoft.Network/expressRouteCircuits]


IMPORTANTIMPORTANT




For step-by-step instructions, see the ExpressRoute circuit routing configuration article to create and modifycircuit peerings.


Next, link a virtual network to your ExpressRoute circuit. Use the Linking virtual networks to ExpressRoute circuitsarticle.

You can modify certain properties of an ExpressRoute circuit without impacting connectivity. You can make thefollowing changes with no downtime:

You can enable or disable an ExpressRoute premium add-on for your ExpressRoute circuit.You can increase the bandwidth of your ExpressRoute circuit provided there is capacity available on the port.However, downgrading the bandwidth of a circuit is not supported.You can change the metering plan from Metered Data to Unlimited Data. However, changing the meteringplan from Unlimited Data to Metered Data is not supported.

To enable the ExpressRoute premium add-onTo enable the ExpressRoute premium add-on

az network express-route update -n MyCircuit -g ExpressRouteResourceGroup --sku-tier Premium

To disable the ExpressRoute premium add-onTo disable the ExpressRoute premium add-on

IMPORTANTIMPORTANT

az network express-route update -n MyCircuit -g ExpressRouteResourceGroup --sku-tier Standard

To update the ExpressRoute circuit bandwidthTo update the ExpressRoute circuit bandwidth

IMPORTANTIMPORTANT

az network express-route update -n MyCircuit -g ExpressRouteResourceGroup --bandwidth 1000

You can enable and disable Allow Classic Operations.

For more information on limits and limitations, see the ExpressRoute FAQ.

You can enable the ExpressRoute premium add-on for your existing circuit by using the following command:

The circuit now has the ExpressRoute premium add-on features enabled. We begin billing you for the premiumadd-on capability as soon as the command has successfully run.

This operation can fail if you're using resources that are greater than what is permitted for the standard circuit.

Before disabling the ExpressRoute premium add-on, understand the following criteria:

Before you downgrade from premium to standard, you must make sure that you have fewer than 10 virtualnetworks linked to the circuit. If you have more than 10, your update request fails, and we bill you at premiumrates.You must unlink all virtual networks in other geopolitical regions. If you don't unlink all your virtual networks,your update request fails and we bill you at premium rates.Your route table must be less than 4,000 routes for private peering. If your route table size is greater than4,000 routes, the BGP session drops. The session won't be reenabled until the number of advertised prefixes isbelow 4,000.

You can disable the ExpressRoute premium add-on for the existing circuit by using the following example:

For the supported bandwidth options for your provider, check the ExpressRoute FAQ. You can pick any sizegreater than the size of your existing circuit.

If there is inadequate capacity on the existing port, you may have to recreate the ExpressRoute circuit. You cannot upgradethe circuit if there is no additional capacity available at that location.

You cannot reduce the bandwidth of an ExpressRoute circuit without disruption. Downgrading bandwidth requires you todeprovision the ExpressRoute circuit, and then reprovision a new ExpressRoute circuit.

After you decide the size you need, use the following command to resize your circuit:

Your circuit is sized up on the Microsoft side. Next, you must contact your connectivity provider to updateconfigurations on their side to match this change. After you make this notification, we begin billing you for theupdated bandwidth option.

To move the SKU from metered to unlimitedTo move the SKU from metered to unlimited

az network express-route update -n MyCircuit -g ExpressRouteResourceGroup --sku-family UnlimitedData

To control access to the classic and Resource Manager environmentsTo control access to the classic and Resource Manager environments


az network express-route delete -n MyCircuit -g ExpressRouteResourceGroup

Next steps

You can change the SKU of an ExpressRoute circuit by using the following example:

Review the instructions in Move ExpressRoute circuits from the classic to the Resource Manager deploymentmodel.

To deprovision and delete an ExpressRoute circuit, make sure you understand the following criteria:

You must unlink all virtual networks from the ExpressRoute circuit. If this operation fails, check to see if anyvirtual networks are linked to the circuit.If the ExpressRoute circuit service provider provisioning state is Provisioning or Provisioned, you mustwork with your service provider to deprovision the circuit on their side. We continue to reserve resources andbill you until the service provider completes deprovisioning the circuit and notifies us.You can delete the circuit if the service provider has deprovisioned the circuit. When a circuit is deprovisioned,the service provider provisioning state is set to Not provisioned. This stops billing for the circuit.


After you create your circuit, make sure that you do the following tasks:


Create and modify peering for an ExpressRoutecircuit4/9/2018 • 7 min to read • Edit Online

Configuration prerequisites

IMPORTANTIMPORTANT

Microsoft peering

IMPORTANTIMPORTANT

To create Microsoft peeringTo create Microsoft peering

This article helps you create and manage routing configuration for an ExpressRoute circuit in the ResourceManager deployment model using the Azure portal. You can also check the status, update, or delete anddeprovision peerings for an ExpressRoute circuit. If you want to use a different method to work with your circuit,select an article from the following list:

Make sure that you have reviewed the prerequisites page, the routing requirements page, and the workflowspage before you begin configuration.You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit must bein a provisioned and enabled state for you to be able to run the cmdlets in the next sections.If you plan to use a shared key/MD5 hash, be sure to use this on both sides of the tunnel and limit thenumber of characters to a maximum of 25.

These instructions only apply to circuits created with service providers offering Layer 2 connectivity services. Ifyou are using a service provider that offers managed Layer 3 services (typically an IPVPN, like MPLS), yourconnectivity provider configures and manages routing for you.

We currently do not advertise peerings configured by service providers through the service management portal. We areworking on enabling this capability soon. Check with your service provider before configuring BGP peerings.

You can configure one, two, or all three peerings (Azure private, Azure public and Microsoft) for an ExpressRoutecircuit. You can configure peerings in any order you choose. However, you must make sure that you complete theconfiguration of each peering one at a time. For more information about routing domains and peerings, seeExpressRoute routing domains.

This section helps you create, get, update, and delete the Microsoft peering configuration for an ExpressRoutecircuit.

Microsoft peering of ExpressRoute circuits that were configured prior to August 1, 2017 will have all service prefixesadvertised through the Microsoft peering, even if route filters are not defined. Microsoft peering of ExpressRoute circuitsthat are configured on or after August 1, 2017 will not have any prefixes advertised until a route filter is attached to thecircuit. For more information, see Configure a route filter for Microsoft peering.

1. Configure ExpressRoute circuit. Ensure that the circuit is fully provisioned by the connectivity providerbefore continuing further. If your connectivity provider offers managed Layer 3 services, you can ask your

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-routing-portal-resource-manager.md

connectivity provider to enable Microsoft peering for you. In that case, you won't need to followinstructions listed in the next sections. However, if your connectivity provider does not manage routing foryou, after creating your circuit, continue your configuration using the next steps.

2. Configure Microsoft peering for the circuit. Make sure that you have the following information before youproceed.

A /30 subnet for the primary link. This must be a valid public IPv4 prefix owned by you and registeredin an RIR / IRR.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix owned by you andregistered in an RIR / IRR.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Advertised prefixes: You must provide a list of all prefixes you plan to advertise over the BGP session.Only public IP address prefixes are accepted. If you plan to send a set of prefixes, you can send acomma-separated list. These prefixes must be registered to you in an RIR / IRR.Optional - Customer ASN: If you are advertising prefixes that are not registered to the peering ASnumber, you can specify the AS number to which they are registered.Routing Registry Name: You can specify the RIR / IRR against which the AS number and prefixes areregistered.Optional - An MD5 hash if you choose to use one.

3. You can select the peering you wish to configure, as shown in the following example. Select the Microsoftpeering row.

4. Configure Microsoft peering. The following image shows a configuration example:

5. Save the configuration once you have specified all parameters.

If your circuit gets to a 'Validation needed' state (as shown in the image), you must open a support ticketto show proof of ownership of the prefixes to our support team.

You can open a support ticket directly from the portal, as shown in the following example:

6. After the configuration has been accepted successfully, you see something similar to the following image:

To view Microsoft peering detailsTo view Microsoft peering details

To update Microsoft peering configurationTo update Microsoft peering configuration

You can view the properties of Azure public peering by selecting the peering.

You can select the row for peering and modify the peering properties.

To delete Microsoft peeringTo delete Microsoft peering

Azure private peering

To create Azure private peeringTo create Azure private peering

You can remove your peering configuration by selecting the delete icon, as shown in the following image:

This section helps you create, get, update, and delete the Azure private peering configuration for anExpressRoute circuit.

1. Configure the ExpressRoute circuit. Ensure that the circuit is fully provisioned by the connectivity providerbefore continuing. If your connectivity provider offers managed Layer 3 services, you can ask yourconnectivity provider to enable Azure private peering for you. In that case, you won't need to follow

instructions listed in the next sections. However, if your connectivity provider does not manage routing foryou, after creating your circuit, continue your configuration using the next steps.

2. Configure Azure private peering for the circuit. Make sure that you have the following items before youproceed with the next steps:

A /30 subnet for the primary link. The subnet must not be part of any address space reserved forvirtual networks.A /30 subnet for the secondary link. The subnet must not be part of any address space reserved forvirtual networks.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers. You can use a private ASnumber for this peering. Ensure that you are not using 65515.Optional - An MD5 hash if you choose to use one.

3. Select the Azure Private peering row, as shown in the following example:

To view Azure private peering detailsTo view Azure private peering details

To update Azure private peering configurationTo update Azure private peering configuration

4. Configure private peering. The following image shows a configuration example:

5. Save the configuration once you have specified all parameters. After the configuration has been acceptedsuccessfully, you see something similar to the following example:

You can view the properties of Azure private peering by selecting the peering.


To delete Azure private peeringTo delete Azure private peering

Azure public peering

To create Azure public peeringTo create Azure public peering

You can remove your peering configuration by selecting the delete icon, as shown in the following image:

This section helps you create, get, update, and delete the Azure public peering configuration for an ExpressRoutecircuit.

1. Configure ExpressRoute circuit. Ensure that the circuit is fully provisioned by the connectivity providerbefore continuing further. If your connectivity provider offers managed Layer 3 services, you can ask yourconnectivity provider to enable Azure public peering for you. In that case, you won't need to followinstructions listed in the next sections. However, if your connectivity provider does not manage routing foryou, after creating your circuit, continue your configuration using the next steps.

2. Configure Azure public peering for the circuit. Make sure that you have the following items before youproceed with the next steps:

A /30 subnet for the primary link. This must be a valid public IPv4 prefix.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Optional - An MD5 hash if you choose to use one.

3. Select the Azure public peering row, as shown in the following image:

4. Configure public peering. The following image shows a configuration example:

To view Azure public peering detailsTo view Azure public peering details

To update Azure public peering configurationTo update Azure public peering configuration

5. Save the configuration once you have specified all parameters. After the configuration has been acceptedsuccessfully, you see something similar to the following example:

You can view the properties of Azure public peering by selecting the peering.


To delete Azure public peeringTo delete Azure public peering

Next steps

You can remove your peering configuration by selecting the delete icon, as shown in the following example:

Next step, Link a VNet to an ExpressRoute circuit

For more information about ExpressRoute workflows, see ExpressRoute workflows.For more information about circuit peering, see ExpressRoute circuits and routing domains.For more information about working with virtual networks, see Virtual network overview.

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-overview

Create and modify peering for an ExpressRoutecircuit using PowerShell4/9/2018 • 11 min to read • Edit Online


IMPORTANTIMPORTANT

Microsoft peering

IMPORTANTIMPORTANT


This article helps you create and manage routing configuration for an ExpressRoute circuit in the ResourceManager deployment model using PowerShell. You can also check the status, update, or delete and deprovisionpeerings for an ExpressRoute circuit. If you want to use a different method to work with your circuit, select anarticle from the following list:

You will need the latest version of the Azure Resource Manager PowerShell cmdlets. For more information,see How to install and configure Azure PowerShell.Make sure that you have reviewed the prerequisites page, the routing requirements page, and the workflowspage before you begin configuration.You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit must bein a provisioned and enabled state for you to be able to run the cmdlets in this article.

These instructions only apply to circuits created with service providers offering Layer 2 connectivity services. Ifyou are using a service provider that offers managed Layer 3 services (typically an IPVPN, like MPLS), yourconnectivity provider will configure and manage routing for you.

We currently do not advertise peerings configured by service providers through the service management portal. We areworking on enabling this capability soon. Check with your service provider before configuring BGP peerings.

You can configure one, two, or all three peerings (Azure private, Azure public and Microsoft) for an ExpressRoutecircuit. You can configure peerings in any order you choose. However, you must make sure that you complete theconfiguration of each peering one at a time. For more information about routing domains and peerings, seeExpressRoute routing domains.



1. Import the PowerShell module for ExpressRoute.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-routing-arm.md


Install-Module AzureRM

Install-AzureRM

Import-AzureRM

Import-Module AzureRM.Network




You must install the latest PowerShell installer from PowerShell Gallery and import the Azure ResourceManager modules into the PowerShell session in order to start using the ExpressRoute cmdlets. You willneed to run PowerShell as an Administrator.

Import all of the AzureRM.* modules within the known semantic version range.

You can also just import a select module within the known semantic version range.

Sign in to your account.

Select the subscription you want to create ExpressRoute circuit.

2. Create an ExpressRoute circuit.

Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. f your connectivity provider offers managed Layer 3 services, you can ask your connectivityprovider to enable Microsoft peering for you. In that case, you won't need to follow instructions listed inthe next sections. However, if your connectivity provider does not manage routing for you, after creatingyour circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to make sure it is provisioned and also enabled. Use the followingexample:


http://www.powershellgallery.com/

Name : ExpressRouteARMCircuitResourceGroupName : ExpressRouteResourceGroupLocation : westusId : /subscriptions/***************************/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/ExpressRouteARMCircuitEtag : W/"################################"ProvisioningState : SucceededSku : { "Name": "Standard_MeteredData", "Tier": "Standard", "Family": "MeteredData" }CircuitProvisioningState : EnabledServiceProviderProvisioningState : ProvisionedServiceProviderNotes : ServiceProviderProperties : { "ServiceProviderName": "Equinix", "PeeringLocation": "Silicon Valley", "BandwidthInMbps": 200 }ServiceKey : **************************************Peerings : []


A /30 or /126 subnet for the primary link. This must be a valid public IPv4 or IPv6 prefix owned byyou and registered in an RIR / IRR.A /30 or /126 subnet for the secondary link. This must be a valid public IPv4 or IPv6 prefix owned byyou and registered in an RIR / IRR.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Advertised prefixes: You must provide a list of all prefixes you plan to advertise over the BGP session.Only public IP address prefixes are accepted. If you plan to send a set of prefixes, you can send acomma-separated list. These prefixes must be registered to you in an RIR / IRR. IPv4 BGP sessionsrequire IPv4 advertised prefixes and IPv6 BGP sessions require IPv6 advertised prefixes.Routing Registry Name: You can specify the RIR / IRR against which the AS number and prefixes areregistered.Optional:

Customer ASN: If you are advertising prefixes that are not registered to the peering AS number,you can specify the AS number to which they are registered.An MD5 hash if you choose to use one.

Use the following example to configure Microsoft peering for your circuit:

To get Microsoft peering detailsTo get Microsoft peering details


Get-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt


Set-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt -PeeringType MicrosoftPeering -PeerASN 100 -PeerAddressType IPv4 -PrimaryPeerAddressPrefix "123.0.0.0/30" -SecondaryPeerAddressPrefix "123.0.0.4/30" -VlanId 300 -MicrosoftConfigAdvertisedPublicPrefixes "124.1.0.0/24" -MicrosoftConfigCustomerAsn 23 -MicrosoftConfigRoutingRegistryName "ARIN"

Set-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt -PeeringType MicrosoftPeering -PeerASN 100 -PeerAddressType IPv6 -PrimaryPeerAddressPrefix "3FFE:FFFF:0:CD30::/126" -SecondaryPeerAddressPrefix "3FFE:FFFF:0:CD30::4/126" -VlanId 300 -MicrosoftConfigAdvertisedPublicPrefixes "3FFE:FFFF:0:CD31::/120" -MicrosoftConfigCustomerAsn 23 -MicrosoftConfigRoutingRegistryName "ARIN"



Remove-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt




Add-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt -PeeringType MicrosoftPeering -PeerASN 100 -PeerAddressType IPv4 -PrimaryPeerAddressPrefix "123.0.0.0/30" -SecondaryPeerAddressPrefix "123.0.0.4/30" -VlanId 300 -MicrosoftConfigAdvertisedPublicPrefixes "123.1.0.0/24" -MicrosoftConfigCustomerAsn 23 -MicrosoftConfigRoutingRegistryName "ARIN"

Add-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt -PeeringType MicrosoftPeering -PeerASN 100 -PeerAddressType IPv6 -PrimaryPeerAddressPrefix "3FFE:FFFF:0:CD30::/126" -SecondaryPeerAddressPrefix "3FFE:FFFF:0:CD30::4/126" -VlanId 300 -MicrosoftConfigAdvertisedPublicPrefixes "3FFE:FFFF:0:CD31::/120" -MicrosoftConfigCustomerAsn 23 -MicrosoftConfigRoutingRegistryName "ARIN"


You can get configuration details using the following example:

You can update any part of the configuration using the following example:

You can remove your peering configuration by running the following cmdlet:

This section helps you create, get, update, and delete the Azure private peering configuration for anExpressRoute circuit.




Install-Module AzureRMInstall-AzureRM

Import-AzureRM










Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. If your connectivity provider offers managed Layer 3 services, you can ask your connectivityprovider to enable Azure private peering for you. In that case, you won't need to follow instructions listedin the next sections. However, if your connectivity provider does not manage routing for you, aftercreating your circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to make sure it is provisioned and also enabled. Use the followingexample:


To get Azure private peering detailsTo get Azure private peering details


Add-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt -PeeringType AzurePrivatePeering -PeerASN 100 -PrimaryPeerAddressPrefix "10.0.0.0/30" -SecondaryPeerAddressPrefix "10.0.0.4/30" -VlanId 200


Add-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt -PeeringType AzurePrivatePeering -PeerASN 100 -PrimaryPeerAddressPrefix "10.0.0.0/30" -SecondaryPeerAddressPrefix "10.0.0.4/30" -VlanId 200 -SharedKey "A1B2C3D4"

IMPORTANTIMPORTANT


A /30 subnet for the primary link. The subnet must not be part of any address space reserved forvirtual networks.A /30 subnet for the secondary link. The subnet must not be part of any address space reserved forvirtual networks.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers. You can use a private ASnumber for this peering. Ensure that you are not using 65515.Optional:

An MD5 hash if you choose to use one.Use the following example to configure Azure private peering for your circuit:

If you choose to use an MD5 hash, use the following example:

Ensure that you specify your AS number as peering ASN, not customer ASN.


Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt


Set-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt -PeeringType AzurePrivatePeering -PeerASN 100 -PrimaryPeerAddressPrefix "10.0.0.0/30" -SecondaryPeerAddressPrefix "10.0.0.4/30" -VlanId 200



WARNINGWARNING

Remove-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt




You can get configuration details by using the following example:

You can update any part of the configuration using the following example. In this example, the VL AN ID of thecircuit is being updated from 100 to 500.

You can remove your peering configuration by running the following example:

You must ensure that all virtual networks are unlinked from the ExpressRoute circuit before running this example.


Install-Module AzureRM

Install-AzureRM

Import-AzureRM














Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. If your connectivity provider offers managed Layer 3 services, you can ask your connectivityprovider to enable Azure public peering for you. In that case, you won't need to follow instructions listedin the next sections. However, if your connectivity provider does not manage routing for you, aftercreating your circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to ensure it is provisioned and also enabled. Use the following example:


4. Configure Azure public peering for the circuit. Make sure that you have the following information beforeyou proceed further.

A /30 subnet for the primary link. This must be a valid public IPv4 prefix.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Optional:

An MD5 hash if you choose to use one.

To get Azure public peering detailsTo get Azure public peering details


Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -Circuit $ckt


Set-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -ExpressRouteCircuit $ckt -PeeringType AzurePublicPeering -PeerASN 100 -PrimaryPeerAddressPrefix "123.0.0.0/30" -SecondaryPeerAddressPrefix "123.0.0.4/30" -VlanId 600



Remove-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -ExpressRouteCircuit $cktSet-AzureRmExpressRouteCircuit -ExpressRouteCircuit $ckt

Next steps

Add-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -ExpressRouteCircuit $ckt -PeeringType AzurePublicPeering -PeerASN 100 -PrimaryPeerAddressPrefix "12.0.0.0/30" -SecondaryPeerAddressPrefix "12.0.0.4/30" -VlanId 100


Add-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -ExpressRouteCircuit $ckt -PeeringType AzurePublicPeering -PeerASN 100 -PrimaryPeerAddressPrefix "12.0.0.0/30" -SecondaryPeerAddressPrefix "12.0.0.4/30" -VlanId 100 -SharedKey "A1B2C3D4"


IMPORTANTIMPORTANT

Run the following example to configure Azure public peering for your circuit



You can get configuration details using the following cmdlet:



Next step, Link a VNet to an ExpressRoute circuit.



Create and modify routing for an ExpressRoutecircuit using CLI4/9/2018 • 10 min to read • Edit Online


Microsoft peering

IMPORTANTIMPORTANT


This article helps you create and manage routing configuration for an ExpressRoute circuit in the ResourceManager deployment model using CLI. You can also check the status, update, or delete and deprovision peeringsfor an ExpressRoute circuit. If you want to use a different method to work with your circuit, select an article fromthe following list:

Before beginning, install the latest version of the CLI commands (2.0 or later). For information about installingthe CLI commands, see Install Azure CLI 2.0.Make sure that you have reviewed the prerequisites, routing requirements, and workflow pages before youbegin configuration.You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit must bein a provisioned and enabled state for you to be able to run the commands in this article.

These instructions only apply to circuits created with service providers offering Layer 2 connectivity services. Ifyou are using a service provider that offers managed Layer 3 services (typically an IPVPN, like MPLS), yourconnectivity provider will configure and manage routing for you.

You can configure one, two, or all three peerings (Azure private, Azure public, and Microsoft) for an ExpressRoutecircuit. You can configure peerings in any order you choose. However, you must make sure that you complete theconfiguration of each peering one at a time. For more information about routing domains and peerings, seeExpressRoute routing domains.



az login

1. Install the latest version of Azure CLI. Use the latest version of the Azure Command-line Interface (CLI).*Review the prerequisites and workflows before you begin configuration.

Select the subscription for which you want to create ExpressRoute circuit.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/howto-routing-cli.md




"allowClassicOperations": false,"authorizations": [],"circuitProvisioningState": "Enabled","etag": "W/\"1262c492-ffef-4a63-95a8-a6002736b8c4\"","gatewayManagerEtag": null,"id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit","location": "westus","name": "MyCircuit","peerings": [],"provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup","serviceKey": "1d05cf70-1db5-419f-ad86-1ca62c3c125b","serviceProviderNotes": null,"serviceProviderProperties": { "bandwidthInMbps": 200, "peeringLocation": "Silicon Valley", "serviceProviderName": "Equinix"},"serviceProviderProvisioningState": "Provisioned","sku": { "family": "UnlimitedData", "name": "Standard_MeteredData", "tier": "Standard"},"tags": null,"type": "Microsoft.Network/expressRouteCircuits]

2. Create an ExpressRoute circuit. Follow the instructions to create an ExpressRoute circuit and have itprovisioned by the connectivity provider. If your connectivity provider offers managed Layer 3 services,you can ask your connectivity provider to enable Microsoft peering for you. In that case, you won't need tofollow instructions listed in the next sections. However, if your connectivity provider does not managerouting for you, after creating your circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to make sure it is provisioned and also enabled. Use the following example:



A /30 subnet for the primary link. This must be a valid public IPv4 prefix owned by you and registeredin an RIR / IRR.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix owned by you andregistered in an RIR / IRR.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Advertised prefixes: You must provide a list of all prefixes you plan to advertise over the BGP session.Only public IP address prefixes are accepted. If you plan to send a set of prefixes, you can send acomma-separated list. These prefixes must be registered to you in an RIR / IRR.Optional - Customer ASN: If you are advertising prefixes that are not registered to the peering AS


az network express-route peering show -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzureMicrosoftPeering

{ "azureAsn": 12076, "etag": "W/\"2e97be83-a684-4f29-bf3c-96191e270666\"", "gatewayManagerEtag": "18", "id": "/subscriptions/9a0c2943-e0c2-4608-876c-e0ddffd1211b/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit/peerings/AzureMicrosoftPeering", "lastModifiedBy": "Customer", "microsoftPeeringConfig": { "advertisedPublicPrefixes": [ "" ], "advertisedPublicPrefixesState": "", "customerASN": , "routingRegistryName": "" } "name": "AzureMicrosoftPeering", "peerAsn": , "peeringType": "AzureMicrosoftPeering", "primaryAzurePort": "", "primaryPeerAddressPrefix": "", "provisioningState": "Succeeded", "resourceGroup": "ExpressRouteResourceGroup", "routeFilter": null, "secondaryAzurePort": "", "secondaryPeerAddressPrefix": "", "sharedKey": null, "state": "Enabled", "stats": null, "vlanId": 100}


az network express-route peering update --circuit-name MyCircuit -g ExpressRouteResourceGroup --peering-type MicrosoftPeering --advertised-public-prefixes 124.1.0.0/24

To add IPv6 Microsoft peering settings to an existing IPv4 configurationTo add IPv6 Microsoft peering settings to an existing IPv4 configuration

az network express-route peering create --circuit-name MyCircuit --peer-asn 100 --primary-peer-subnet 123.0.0.0/30 -g ExpressRouteResourceGroup --secondary-peer-subnet 123.0.0.4/30 --vlan-id 300 --peering-type MicrosoftPeering --advertised-public-prefixes 123.1.0.0/24

number, you can specify the AS number to which they are registered.Routing Registry Name: You can specify the RIR / IRR against which the AS number and prefixes areregistered.Optional - An MD5 hash if you choose to use one.

Run the following example to configure Microsoft peering for your circuit:


The output is similar to the following example:

You can update any part of the configuration. The advertised prefixes of the circuit are being updated from123.1.0.0/24 to 124.1.0.0/24 in the following example:

az network express-route peering update -g ExpressRouteResourceGroup --circuit-name MyCircuit --peering-type MicrosoftPeering --ip-version ipv6 --primary-peer-subnet 2002:db00::/126 --secondary-peer-subnet 2003:db00::/126 --advertised-public-prefixes 2002:db00::/126


az network express-route peering delete -g ExpressRouteResourceGroup --circuit-name MyCircuit --name MicrosoftPeering




This section helps you create, get, update, and delete the Azure private peering configuration for an ExpressRoutecircuit.

az login


az network express-route show --resource-group ExpressRouteResourceGroup --name MyCircuit

1. Install the latest version of Azure CLI. You must use the latest version of the Azure Command-lineInterface (CLI).* Review the prerequisites and workflows before you begin configuration.

Select the subscription you want to create ExpressRoute circuit

2. Create an ExpressRoute circuit. Follow the instructions to create an ExpressRoute circuit and have itprovisioned by the connectivity provider. If your connectivity provider offers managed Layer 3 services,you can ask your connectivity provider to enable Azure private peering for you. In that case, you won'tneed to follow instructions listed in the next sections. However, if your connectivity provider does notmanage routing for you, after creating your circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to make sure it is provisioned and also enabled. Use the following example:


"allowClassicOperations": false,"authorizations": [],"circuitProvisioningState": "Enabled","etag": "W/\"1262c492-ffef-4a63-95a8-a6002736b8c4\"","gatewayManagerEtag": null,"id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit","location": "westus","name": "MyCircuit","peerings": [],"provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup","serviceKey": "1d05cf70-1db5-419f-ad86-1ca62c3c125b","serviceProviderNotes": null,"serviceProviderProperties": {"bandwidthInMbps": 200,"peeringLocation": "Silicon Valley","serviceProviderName": "Equinix"},"serviceProviderProvisioningState": "Provisioned","sku": { "family": "UnlimitedData", "name": "Standard_MeteredData", "tier": "Standard"},"tags": null,"type": "Microsoft.Network/expressRouteCircuits]

az network express-route peering create --circuit-name MyCircuit --peer-asn 100 --primary-peer-subnet 10.0.0.0/30 -g ExpressRouteResourceGroup --secondary-peer-subnet 10.0.0.4/30 --vlan-id 200 --peering-type AzurePrivatePeering

az network express-route peering create --circuit-name MyCircuit --peer-asn 100 --primary-peer-subnet 10.0.0.0/30 -g ExpressRouteResourceGroup --secondary-peer-subnet 10.0.0.4/30 --vlan-id 200 --peering-type AzurePrivatePeering --SharedKey "A1B2C3D4"

IMPORTANTIMPORTANT


A /30 subnet for the primary link. The subnet must not be part of any address space reserved forvirtual networks.A /30 subnet for the secondary link. The subnet must not be part of any address space reserved forvirtual networks.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers. You can use a private ASnumber for this peering. Ensure that you are not using 65515.Optional - An MD5 hash if you choose to use one.

Use the following example to configure Azure private peering for your circuit:




az network express-route peering show -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePrivatePeering

{ "azureAsn": 12076, "etag": "W/\"2e97be83-a684-4f29-bf3c-96191e270666\"", "gatewayManagerEtag": "18", "id": "/subscriptions/9a0c2943-e0c2-4608-876c-e0ddffd1211b/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit/peerings/AzurePrivatePeering", "ipv6PeeringConfig": null, "lastModifiedBy": "Customer", "microsoftPeeringConfig": null, "name": "AzurePrivatePeering", "peerAsn": 7671, "peeringType": "AzurePrivatePeering", "primaryAzurePort": "", "primaryPeerAddressPrefix": "", "provisioningState": "Succeeded", "resourceGroup": "ExpressRouteResourceGroup", "routeFilter": null, "secondaryAzurePort": "", "secondaryPeerAddressPrefix": "", "sharedKey": null, "state": "Enabled", "stats": null, "vlanId": 100}


az network express-route peering update --vlan-id 500 -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePrivatePeering


WARNINGWARNING

az network express-route peering delete -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePrivatePeering






You must ensure that all virtual networks are unlinked from the ExpressRoute circuit before running this example.



az login



"allowClassicOperations": false,"authorizations": [],"circuitProvisioningState": "Enabled","etag": "W/\"1262c492-ffef-4a63-95a8-a6002736b8c4\"","gatewayManagerEtag": null,"id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit","location": "westus","name": "MyCircuit","peerings": [],"provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup","serviceKey": "1d05cf70-1db5-419f-ad86-1ca62c3c125b","serviceProviderNotes": null,"serviceProviderProperties": { "bandwidthInMbps": 200, "peeringLocation": "Silicon Valley", "serviceProviderName": "Equinix"},"serviceProviderProvisioningState": "Provisioned","sku": { "family": "UnlimitedData", "name": "Standard_MeteredData", "tier": "Standard"},"tags": null,"type": "Microsoft.Network/expressRouteCircuits]

1. Install the latest version of Azure CLI. You must use the latest version of the Azure Command-lineInterface (CLI).* Review the prerequisites and workflows before you begin configuration.

Select the subscription for which you want to create ExpressRoute circuit.

2. Create an ExpressRoute circuit. Follow the instructions to create an ExpressRoute circuit and have itprovisioned by the connectivity provider. If your connectivity provider offers managed Layer 3 services,you can ask your connectivity provider to enable Azure public peering for you. In that case, you won't needto follow instructions listed in the next sections. However, if your connectivity provider does not managerouting for you, after creating your circuit, continue your configuration using the next steps.

3. Check the ExpressRoute circuit to ensure it is provisioned and also enabled. Use the following example:


4. Configure Azure public peering for the circuit. Make sure that you have the following information beforeyou proceed further.

A /30 subnet for the primary link. This must be a valid public IPv4 prefix.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the same


az network express-route peering show -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePublicPeering

{ "azureAsn": 12076, "etag": "W/\"2e97be83-a684-4f29-bf3c-96191e270666\"", "gatewayManagerEtag": "18", "id": "/subscriptions/9a0c2943-e0c2-4608-876c-e0ddffd1211b/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit/peerings/AzurePublicPeering", "lastModifiedBy": "Customer", "microsoftPeeringConfig": null, "name": "AzurePublicPeering", "peerAsn": 7671, "peeringType": "AzurePublicPeering", "primaryAzurePort": "", "primaryPeerAddressPrefix": "", "provisioningState": "Succeeded", "resourceGroup": "ExpressRouteResourceGroup", "routeFilter": null, "secondaryAzurePort": "", "secondaryPeerAddressPrefix": "", "sharedKey": null, "state": "Enabled", "stats": null, "vlanId": 100}


az network express-route peering create --circuit-name MyCircuit --peer-asn 100 --primary-peer-subnet 12.0.0.0/30 -g ExpressRouteResourceGroup --secondary-peer-subnet 12.0.0.4/30 --vlan-id 200 --peering-type AzurePublicPeering

az network express-route peering create --circuit-name MyCircuit --peer-asn 100 --primary-peer-subnet 12.0.0.0/30 -g ExpressRouteResourceGroup --secondary-peer-subnet 12.0.0.4/30 --vlan-id 200 --peering-type AzurePublicPeering --SharedKey "A1B2C3D4"

IMPORTANTIMPORTANT

VL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Optional - An MD5 hash if you choose to use one.

Run the following example to configure Azure public peering for your circuit:



You can get configuration details using the following example:



az network express-route peering update --vlan-id 600 -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePublicPeering


az network express-route peering delete -g ExpressRouteResourceGroup --circuit-name MyCircuit --name AzurePublicPeering

Next steps


Next step, Link a VNet to an ExpressRoute circuit.



Connect a virtual network to an ExpressRoute circuitusing the portal3/9/2018 • 4 min to read • Edit Online

Before you begin

Connect a VNet to a circuit - same subscription

NOTENOTE

To create a connectionTo create a connection

This article helps you create a connection to link a virtual network to an Azure ExpressRoute circuit using theAzure portal. The virtual networks that you connect to your Azure ExpressRoute circuit can either be in the samesubscription, or they can be part of another subscription.

Review the prerequisites, routing requirements, and workflows before you begin configuration.

You must have an active ExpressRoute circuit.

Follow the instructions to create an ExpressRoute circuit and have the circuit enabled by yourconnectivity provider.Ensure that you have Azure private peering configured for your circuit. See the Configure routingarticle for routing instructions.Ensure that Azure private peering is configured and the BGP peering between your network andMicrosoft is up so that you can enable end-to-end connectivity.Ensure that you have a virtual network and a virtual network gateway created and fully provisioned.Follow the instructions to create a virtual network gateway for ExpressRoute. A virtual network gatewayfor ExpressRoute uses the GatewayType 'ExpressRoute', not VPN.

You can link up to 10 virtual networks to a standard ExpressRoute circuit. All virtual networks must be inthe same geopolitical region when using a standard ExpressRoute circuit.

A single VNet can be linked to up to four ExpressRoute circuits. Use the process below to create a newconnection object for each ExpressRoute circuit you are connecting to. The ExpressRoute circuits can be inthe same subscription, different subscriptions, or a mix of both.

You can link a virtual network outside of the geopolitical region of the ExpressRoute circuit, or connect alarger number of virtual networks to your ExpressRoute circuit if you enabled the ExpressRoute premiumadd-on. Check the FAQ for more details on the premium add-on.

You can view a video before beginning to better understand the steps.

BGP configuration information will not show up if the layer 3 provider configured your peerings. If your circuit is in aprovisioned state, you should be able to create connections.

1. Ensure that your ExpressRoute circuit and Azure private peering have been configured successfully. Followthe instructions in Create an ExpressRoute circuit and Configure routing. Your ExpressRoute circuit shouldlook like the following image:

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-linkvnet-portal-resource-manager.md

http://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-a-connection-between-your-vpn-gateway-and-expressroute-circuit

Connect a VNet to a circuit - different subscription

2. You can now start provisioning a connection to link your virtual network gateway to your ExpressRoutecircuit. Click Connection > Add to open the Add connection page, and then configure the values.

3. After your connection has been successfully configured, your connection object will show the informationfor the connection.

You can share an ExpressRoute circuit across multiple subscriptions. The figure below shows a simple schematicof how sharing works for ExpressRoute circuits across multiple subscriptions.

Administration - About circuit owners and circuit usersAdministration - About circuit owners and circuit users

Circuit owner operationsCircuit owner operations

Each of the smaller clouds within the large cloud is used to represent subscriptions that belong to differentdepartments within an organization.Each of the departments within the organization can use their own subscription for deploying their services,but they can share a single ExpressRoute circuit to connect back to your on-premises network.

NOTENOTE

A single department (in this example: IT) can own the ExpressRoute circuit. Other subscriptions within theorganization can use the ExpressRoute circuit and authorizations associated to the circuit, includingsubscriptions linked to other Azure Active Directory tenants and Enterprise Agreement enrollments.

Connectivity and bandwidth charges for the dedicated circuit will be applied to the ExpressRoute circuit owner. Allvirtual networks share the same bandwidth.

The 'circuit owner' is an authorized Power User of the ExpressRoute circuit resource. The circuit owner can createauthorizations that can be redeemed by 'circuit users'. Circuit users are owners of virtual network gateways thatare not within the same subscription as the ExpressRoute circuit. Circuit users can redeem authorizations (oneauthorization per virtual network).

The circuit owner has the power to modify and revoke authorizations at any time. Revoking an authorizationresults in all link connections being deleted from the subscription whose access was revoked.

To create a connection authorization

The circuit owner creates an authorization. This results in the creation of an authorization key that can be used bya circuit user to connect their virtual network gateways to the ExpressRoute circuit. An authorization is valid foronly one connection.

1. In the ExpressRoute page, Click Authorizations and then type a name for the authorization and clickSave.

Circuit user operationsCircuit user operations

2. Once the configuration is saved, copy the Resource ID and the Authorization Key.

To delete a connection authorization

You can delete a connection by selecting the Delete icon on the page for your connection.

The circuit user needs the resource ID and an authorization key from the circuit owner.

To redeem a connection authorization

1. Click the +New button.

2. Search for "Connection" in the Marketplace, select it, and click Create.

3. Make sure the Connection type is set to "ExpressRoute".

5. In the Settings page, Select the Virtual network gateway and check the Redeem authorization check box.

7. Review the information in the Summary page and click OK.

4. Fill in the details, then click OK in the Basics page.

6. Enter the Authorization key and the Peer circuit URI and give the connection a name. Click OK.

Delete a connection to unlink a VNet

Next steps

To release a connection authorization

You can release an authorization by deleting the connection that links the ExpressRoute circuit to the virtualnetwork.

You can delete a connection and unlink your VNet to an ExpressRoute circuit by selecting the Delete icon on thepage for your connection.

For more information about ExpressRoute, see the ExpressRoute FAQ.

Connect a virtual network to an ExpressRoutecircuit3/9/2018 • 5 min to read • Edit Online

Before you begin

Connect a virtual network in the same subscription to a circuit

$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"$gw = Get-AzureRmVirtualNetworkGateway -Name "ExpressRouteGw" -ResourceGroupName "MyRG"$connection = New-AzureRmVirtualNetworkGatewayConnection -Name "ERConnection" -ResourceGroupName "MyRG" -Location "East US" -VirtualNetworkGateway1 $gw -PeerId $circuit.Id -ConnectionType ExpressRoute

Connect a virtual network in a different subscription to a circuit

This article helps you link virtual networks (VNets) to Azure ExpressRoute circuits by using the ResourceManager deployment model and PowerShell. Virtual networks can either be in the same subscription or part ofanother subscription. This article also shows you how to update a virtual network link.

Install the latest version of the Azure PowerShell modules. For more information, see How to install andconfigure Azure PowerShell.

Review the prerequisites, routing requirements, and workflows before you begin configuration.


Follow the instructions to create an ExpressRoute circuit and have the circuit enabled by yourconnectivity provider.Ensure that you have Azure private peering configured for your circuit. See the configure routingarticle for routing instructions.Ensure that Azure private peering is configured and the BGP peering between your network andMicrosoft is up so that you can enable end-to-end connectivity.Ensure that you have a virtual network and a virtual network gateway created and fully provisioned.Follow the instructions to create a virtual network gateway for ExpressRoute. A virtual networkgateway for ExpressRoute uses the GatewayType 'ExpressRoute', not VPN.


A single VNet can be linked to up to four ExpressRoute circuits. Use the process below to create a newconnection object for each ExpressRoute circuit you are connecting to. The ExpressRoute circuits can bein the same subscription, different subscriptions, or a mix of both.

You can link a virtual networks outside of the geopolitical region of the ExpressRoute circuit, or connect alarger number of virtual networks to your ExpressRoute circuit if you enabled the ExpressRoutepremium add-on. Check the FAQ for more details on the premium add-on.

You can connect a virtual network gateway to an ExpressRoute circuit by using the following cmdlet. Make surethat the virtual network gateway is created and is ready for linking before you run the cmdlet:

You can share an ExpressRoute circuit across multiple subscriptions. The following figure shows a simple

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-linkvnet-arm.md


NOTENOTE

Administration - circuit owners and circuit usersAdministration - circuit owners and circuit users


$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"Add-AzureRmExpressRouteCircuitAuthorization -ExpressRouteCircuit $circuit -Name "MyAuthorization1"Set-AzureRmExpressRouteCircuit -ExpressRouteCircuit $circuit

$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"$auth1 = Get-AzureRmExpressRouteCircuitAuthorization -ExpressRouteCircuit $circuit -Name "MyAuthorization1"

schematic of how sharing works for ExpressRoute circuits across multiple subscriptions.

Each of the smaller clouds within the large cloud is used to represent subscriptions that belong to differentdepartments within an organization. Each of the departments within the organization can use their ownsubscription for deploying their services--but they can share a single ExpressRoute circuit to connect back toyour on-premises network. A single department (in this example: IT) can own the ExpressRoute circuit. Othersubscriptions within the organization can use the ExpressRoute circuit.

Connectivity and bandwidth charges for the ExpressRoute circuit will be applied to the subscription owner. All virtualnetworks share the same bandwidth.

The 'circuit owner' is an authorized Power User of the ExpressRoute circuit resource. The circuit owner cancreate authorizations that can be redeemed by 'circuit users'. Circuit users are owners of virtual networkgateways that are not within the same subscription as the ExpressRoute circuit. Circuit users can redeemauthorizations (one authorization per virtual network).

The circuit owner has the power to modify and revoke authorizations at any time. Revoking an authorizationresults in all link connections being deleted from the subscription whose access was revoked.

To create an authorization

The circuit owner creates an authorization. This results in the creation of an authorization key that can be usedby a circuit user to connect their virtual network gateways to the ExpressRoute circuit. An authorization is validfor only one connection.

The following cmdlet snippet shows how to create an authorization:

The response to this will contain the authorization key and status:

Name : MyAuthorization1Id : /subscriptions/&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&/resourceGroups/ERCrossSubTestRG/providers/Microsoft.Network/expressRouteCircuits/CrossSubTest/authorizations/MyAuthorization1Etag : &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&& AuthorizationKey : ####################################AuthorizationUseStatus : AvailableProvisioningState : Succeeded

$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"$authorizations = Get-AzureRmExpressRouteCircuitAuthorization -ExpressRouteCircuit $circuit

$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"Add-AzureRmExpressRouteCircuitAuthorization -ExpressRouteCircuit $circuit -Name "MyAuthorization2"Set-AzureRmExpressRouteCircuit -ExpressRouteCircuit $circuit

$circuit = Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"$authorizations = Get-AzureRmExpressRouteCircuitAuthorization -ExpressRouteCircuit $circuit

Remove-AzureRmExpressRouteCircuitAuthorization -Name "MyAuthorization2" -ExpressRouteCircuit $circuitSet-AzureRmExpressRouteCircuit -ExpressRouteCircuit $circuit


Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"

$id = "/subscriptions/********************************/resourceGroups/ERCrossSubTestRG/providers/Microsoft.Network/expressRouteCircuits/MyCircuit" $gw = Get-AzureRmVirtualNetworkGateway -Name "ExpressRouteGw" -ResourceGroupName "MyRG"$connection = New-AzureRmVirtualNetworkGatewayConnection -Name "ERConnection" -ResourceGroupName "RemoteResourceGroup" -Location "East US" -VirtualNetworkGateway1 $gw -PeerId $id -ConnectionType ExpressRoute -AuthorizationKey "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

To review authorizations

The circuit owner can review all authorizations that are issued on a particular circuit by running the followingcmdlet:

To add authorizations

The circuit owner can add authorizations by using the following cmdlet:

To delete authorizations

The circuit owner can revoke/delete authorizations to the user by running the following cmdlet:

The circuit user needs the peer ID and an authorization key from the circuit owner. The authorization key is aGUID.

Peer ID can be checked from the following command:


The circuit user can run the following cmdlet to redeem a link authorization:


Modify a virtual network connection

$connection = Get-AzureRmVirtualNetworkGatewayConnection -Name "MyVirtualNetworkConnection" -ResourceGroupName "MyRG"$connection.RoutingWeight = 100Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $connection

Next steps


You can update certain properties of a virtual network connection.

To update the connection weight

Your virtual network can be connected to multiple ExpressRoute circuits. You may receive the same prefix frommore than one ExpressRoute circuit. To choose which connection to send traffic destined for this prefix, you canchange RoutingWeight of a connection. Traffic will be sent on the connection with the highest RoutingWeight.

The range of RoutingWeight is 0 to 32000. The default value is 0.


Connect a virtual network to an ExpressRoute circuitusing CLI3/9/2018 • 4 min to read • Edit Online



az network vpn-connection create --name ERConnection --resource-group ExpressRouteResourceGroup --vnet-gateway1 VNet1GW --express-route-circuit2 MyCircuit


This article helps you link virtual networks (VNets) to Azure ExpressRoute circuits using CLI. To link using AzureCLI, the virtual networks must be created using the Resource Manager deployment model. They can either be inthe same subscription, or part of another subscription. If you want to use a different method to connect yourVNet to an ExpressRoute circuit, you can select an article from the following list:

You need the latest version of the command-line interface (CLI). For more information, see Install AzureCLI 2.0.

You need to review the prerequisites, routing requirements, and workflows before you begin configuration.


Follow the instructions to create an ExpressRoute circuit and have the circuit enabled by yourconnectivity provider.Ensure that you have Azure private peering configured for your circuit. See the configure routing articlefor routing instructions.Ensure that Azure private peering is configured. The BGP peering between your network and Microsoftmust be up so that you can enable end-to-end connectivity.Ensure that you have a virtual network and a virtual network gateway created and fully provisioned.Follow the instructions to Configure a virtual network gateway for ExpressRoute. Be sure to use --gateway-type ExpressRoute .


A single VNet can be linked to up to four ExpressRoute circuits. Use the process below to create a newconnection object for each ExpressRoute circuit you are connecting to. The ExpressRoute circuits can be inthe same subscription, different subscriptions, or a mix of both.

If you enable the ExpressRoute premium add-on, you can link a virtual network outside of the geopoliticalregion of the ExpressRoute circuit, or connect a larger number of virtual networks to your ExpressRoutecircuit. For more information about the premium add-on, see the FAQ.

You can connect a virtual network gateway to an ExpressRoute circuit by using the example. Make sure that thevirtual network gateway is created and is ready for linking before you run the command.

You can share an ExpressRoute circuit across multiple subscriptions. The figure below shows a simple schematic

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/howto-linkvnet-cli.md


https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-howto-site-to-site-resource-manager-cli

NOTENOTE

Administration - Circuit Owners and Circuit UsersAdministration - Circuit Owners and Circuit Users

Circuit Owner operationsCircuit Owner operations

az network express-route auth create --circuit-name MyCircuit -g ExpressRouteResourceGroup -n MyAuthorization

of how sharing works for ExpressRoute circuits across multiple subscriptions.

Each of the smaller clouds within the large cloud is used to represent subscriptions that belong to differentdepartments within an organization. Each of the departments within the organization can use their ownsubscription for deploying their services--but they can share a single ExpressRoute circuit to connect back to youron-premises network. A single department (in this example: IT) can own the ExpressRoute circuit. Othersubscriptions within the organization can use the ExpressRoute circuit.

Connectivity and bandwidth charges for the dedicated circuit will be applied to the ExpressRoute Circuit Owner. All virtualnetworks share the same bandwidth.

The 'Circuit Owner' is an authorized Power User of the ExpressRoute circuit resource. The Circuit Owner cancreate authorizations that can be redeemed by 'Circuit Users'. Circuit Users are owners of virtual networkgateways that are not within the same subscription as the ExpressRoute circuit. Circuit Users can redeemauthorizations (one authorization per virtual network).

The Circuit Owner has the power to modify and revoke authorizations at any time. When an authorization isrevoked, all link connections are deleted from the subscription whose access was revoked.

To create an authorization

The Circuit Owner creates an authorization, which creates an authorization key that can be used by a Circuit Userto connect their virtual network gateways to the ExpressRoute circuit. An authorization is valid for only oneconnection.

The following example shows how to create an authorization:

The response contains the authorization key and status:

"authorizationKey": "0a7f3020-541f-4b4b-844a-5fb43472e3d7","authorizationUseStatus": "Available","etag": "W/\"010353d4-8955-4984-807a-585c21a22ae0\"","id": "/subscriptions/81ab786c-56eb-4a4d-bb5f-f60329772466/resourceGroups/ExpressRouteResourceGroup/providers/Microsoft.Network/expressRouteCircuits/MyCircuit/authorizations/MyAuthorization1","name": "MyAuthorization1","provisioningState": "Succeeded","resourceGroup": "ExpressRouteResourceGroup"

az network express-route auth list --circuit-name MyCircuit -g ExpressRouteResourceGroup

az network express-route auth create --circuit-name MyCircuit -g ExpressRouteResourceGroup -n MyAuthorization1

az network express-route auth delete --circuit-name MyCircuit -g ExpressRouteResourceGroup -n MyAuthorization1

Circuit User operationsCircuit User operations

Get-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "MyRG"

az network vpn-connection create --name ERConnection --resource-group ExpressRouteResourceGroup --vnet-gateway1 VNet1GW --express-route-circuit2 MyCircuit --authorization-key "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

Next steps

To review authorizations

The Circuit Owner can review all authorizations that are issued on a particular circuit by running the followingexample:

To add authorizations

The Circuit Owner can add authorizations by using the following example:

To delete authorizations

The Circuit Owner can revoke/delete authorizations to the user by running the following example:

The Circuit User needs the peer ID and an authorization key from the Circuit Owner. The authorization key is aGUID.


The Circuit User can run the following example to redeem a link authorization:




Configure a site-to-site VPN over ExpressRouteMicrosoft peering12/7/2017 • 16 min to read • Edit Online

Architecture

NOTENOTE

This article helps you configure secure encrypted connectivity between your on-premises network and your Azurevirtual networks (VNets) over an ExpressRoute private connection. Configuring a secure tunnel over ExpressRouteallows for data exchange with confidentiality, anti-replay, authenticity, and integrity.

You can leverage Microsoft peering to establish a site-to-site IPsec/IKE VPN tunnel between your selected on-premises networks and Azure VNets.

When you set up site-to-site VPN over Microsoft peering, you are charged for the VPN gateway and VPN egress. For moreinformation, see VPN Gateway pricing.

For high availability and redundancy, you can configure multiple tunnels over the two MSEE-PE pairs of aExpressRoute circuit and enable load balancing between the tunnels.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/site-to-site-vpn-over-microsoft-peering.md

https://azure.microsoft.com/pricing/details/vpn-gateway

IMPORTANTIMPORTANT

Workflow

1. Configure Microsoft peering

VPN tunnels over Microsoft peering can be terminated either using VPN gateway, or using an appropriateNetwork Virtual Appliance (NVA) available through Azure Marketplace. You can exchange routes statically ordynamically over the encrypted tunnels without exposing the route exchange to the underlying Microsoft peering.In the examples in this article, BGP (different from the BGP session used to create the Microsoft peering) is used todynamically exchange prefixes over the encrypted tunnels.

For the on-premises side, typically Microsoft peering is terminated on the DMZ and private peering is terminated on the corenetwork zone. The two zones would be segregated using firewalls. If you are configuring Microsoft peering exclusively forenabling secure tunneling over ExpressRoute, remember to filter through only the public IPs of interest that are gettingadvertised via Microsoft peering.

1. Configure Microsoft peering for your ExpressRoute circuit.2. Advertise selected Azure regional public prefixes to your on-premises network via Microsoft peering.3. Configure a VPN gateway and establish IPsec tunnels4. Configure the on-premises VPN device.5. Create the site-to-site IPsec/IKE connection.6. (Optional) Configure firewalls/filtering on the on-premises VPN device.7. Test and validate the IPsec communication over the ExpressRoute circuit.

To configure a site-to-site VPN connection over ExpressRoute, you must leverage ExpressRoute Microsoft peering.

To configure a new ExpressRoute circuit, start with the ExpressRoute prerequisites article, and then Createand modify an ExpressRoute circuit.

If you already have an ExpressRoute circuit, but do not have Microsoft peering configured, configureMicrosoft peering using the Create and modify peering for an ExpressRoute circuit article.

Once you have configured your circuit and Microsoft peering, you can easily view it using the Overview page inthe Azure portal.

2. Configure route filters

2.1 Configure the route filter2.1 Configure the route filter

2.2 Verify BGP routes2.2 Verify BGP routes

Cisco examplesCisco examples

show ip bgp vpnv4 vrf 10 summary

...

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcdX.243.229.34 4 12076 17671 17650 25228 0 0 1w4d 68

A route filter lets you identify services you want to consume through your ExpressRoute circuit's Microsoft peering.It is essentially a whitelist of all the BGP community values.

In this example, the deployment is only in the Azure West US 2 region. A route filter rule is added to allow only theadvertisement of Azure West US 2 regional prefixes, which has the BGP community value 12076:51026. Youspecify the regional prefixes that you want to allow by selecting Manage rule.

Within the route filter, you also need to choose the ExpressRoute circuits for which the route filter applies. You canchoose the ExpressRoute circuits by selecting Add circuit. In the previous figure, the route filter is associated to theexample ExpressRoute circuit.

Configure a route filter. For steps, see Configure route filters for Microsoft peering.

Once you have successfully created Microsoft peering over your ExpressRoute circuit and associated a route filterwith the circuit, you can verify the BGP routes received from MSEEs on the PE devices that are peering with theMSEEs. The verification command varies, depending on the operating system of your PE devices.

This example uses a Cisco IOS-XE command. In the example, a virtual routing and forwarding (VRF) instance isused to isolate the peering traffic.

The following partial output shows that 68 prefixes were received from the neighbor *.243.229.34 with the ASN12076 (MSEE):

sh ip bgp vpnv4 vrf 10 neighbors X.243.229.34 received-routes

Get-AzureRmBgpServiceCommunity

3. Configure the VPN gateway and IPsec tunnels

To see the list of prefixes received from the neighbor, use the following example:

To confirm that you are receiving the correct set of prefixes, you can cross-verify. The following Azure PowerShellcommand output lists the prefixes advertised via Microsoft peering for each of the services and for each of theAzure region:

In this section, IPsec VPN tunnels are created between the Azure VPN gateway and the on-premises VPN device.The examples use Cisco Cloud Service Router (CSR1000) VPN devices.

The following diagram shows the IPsec VPN tunnels established between on-premises VPN device 1, and theAzure VPN gateway instance pair. The two IPsec VPN tunnels established between the on-premises VPN device 2and the Azure VPN gateway instance pair isn't illustrated in the diagram, and the configuration details are notlisted. However, having additional VPN tunnels improves high availability.

Over the IPsec tunnel pair, an eBGP session is established to exchange private network routes. The followingdiagram shows the eBGP session established over the IPsec tunnel pair :

The following diagram shows the abstracted overview of the example network:

About the Azure Resource Manager template examplesAbout the Azure Resource Manager template examples

NOTENOTE

3.1 Declare the variables3.1 Declare the variables

In the examples, the VPN gateway and the IPsec tunnel terminations are configured using an Azure ResourceManager template. If you are new to using Resource Manager templates, or to understand the Resource Managertemplate basics, see Understand the structure and syntax of Azure Resource Manager templates. The template inthis section creates a greenfield Azure environment (VNet). However, if you have an existing VNet, you canreference it in the template. If you are not familiar with VPN gateway IPsec/IKE site-to-site configurations, seeCreate a site-to-site connection.

You do not need to use Azure Resource Manager templates in order to create this configuration. You can create thisconfiguration using the Azure portal, or PowerShell.

In this example, the variable declarations correspond to the example network. When declaring variables, modifythis section to reflect your environment.

The variable localAddressPrefix is an array of on-premises IP addresses to terminate the IPsec tunnels.The gatewaySku determines the VPN throughput. For more information about gatewaySku and vpnType, seeVPN Gateway configuration settings. For pricing, see VPN Gateway pricing.Set the vpnType to RouteBased.

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-create-site-to-site-rm-powershell

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings

https://azure.microsoft.com/pricing/details/vpn-gateway

"variables": { "virtualNetworkName": "SecureVNet", // Name of the Azure VNet "azureVNetAddressPrefix": "10.2.0.0/24", // Address space assigned to the VNet "subnetName": "Tenant", // subnet name in which tenants exists "subnetPrefix": "10.2.0.0/25", // address space of the tenant subnet "gatewaySubnetPrefix": "10.2.0.224/27", // address space of the gateway subnet "localGatewayName": "localGW1", // name of remote gateway (on-premises) "localGatewayIpAddress": "X.243.229.110", // public IP address of the on-premises VPN device "localAddressPrefix": [ "172.16.0.1/32", // termination of IPsec tunnel-1 on-premises "172.16.0.2/32" // termination of IPsec tunnel-2 on-premises ], "gatewayPublicIPName1": "vpnGwVIP1", // Public address name of the first VPN gateway instance "gatewayPublicIPName2": "vpnGwVIP2", // Public address name of the second VPN gateway instance "gatewayName": "vpnGw", // Name of the Azure VPN gateway "gatewaySku": "VpnGw1", // Azure VPN gateway SKU "vpnType": "RouteBased", // type of VPN gateway "sharedKey": "string", // shared secret needs to match with on-premise configuration "asnVpnGateway": 65000, // BGP Autonomous System number assigned to the VPN Gateway "asnRemote": 65010, // BGP Autonmous Syste number assigned to the on-premises device "bgpPeeringAddress": "172.16.0.3", // IP address of the remote BGP peer on-premises "connectionName": "vpn2local1", "vnetID": "[resourceId('Microsoft.Network/virtualNetworks', variables('virtualNetworkName'))]", "gatewaySubnetRef": "[concat(variables('vnetID'),'/subnets/','GatewaySubnet')]", "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]", "api-version": "2017-06-01"},

3.2 Create virtual network (VNet)3.2 Create virtual network (VNet)

{ "apiVersion": "[variables('api-version')]", "type": "Microsoft.Network/virtualNetworks", "name": "[variables('virtualNetworkName')]", "location": "[resourceGroup().location]", "properties": { "addressSpace": { "addressPrefixes": [ "[variables('azureVNetAddressPrefix')]" ] }, "subnets": [ { "name": "[variables('subnetName')]", "properties": { "addressPrefix": "[variables('subnetPrefix')]" } }, { "name": "GatewaySubnet", "properties": { "addressPrefix": "[variables('gatewaySubnetPrefix')]" } } ] }, "comments": "Create a Virtual Network with Subnet1 and Gatewaysubnet"},

3.3 Assign public IP addresses to VPN gateway instances3.3 Assign public IP addresses to VPN gateway instances

If you are associating an existing VNet with the VPN tunnels, you can skip this step.

Assign a public IP address for each instance of a VPN gateway.

{ "apiVersion": "[variables('api-version')]", "type": "Microsoft.Network/publicIPAddresses", "name": "[variables('gatewayPublicIPName1')]", "location": "[resourceGroup().location]", "properties": { "publicIPAllocationMethod": "Dynamic" }, "comments": "Public IP for the first instance of the VPN gateway" }, { "apiVersion": "[variables('api-version')]", "type": "Microsoft.Network/publicIPAddresses", "name": "[variables('gatewayPublicIPName2')]", "location": "[resourceGroup().location]", "properties": { "publicIPAllocationMethod": "Dynamic" }, "comments": "Public IP for the second instance of the VPN gateway" },

3.4 Specify the on-premises VPN tunnel termination (local network gateway)3.4 Specify the on-premises VPN tunnel termination (local network gateway)

{ "apiVersion": "[variables('api-version')]", "type": "Microsoft.Network/localNetworkGateways", "name": "[variables('localGatewayName')]", "location": "[resourceGroup().location]", "properties": { "localNetworkAddressSpace": { "addressPrefixes": "[variables('localAddressPrefix')]" }, "gatewayIpAddress": "[variables('localGatewayIpAddress')]", "bgpSettings": { "asn": "[variables('asnRemote')]", "bgpPeeringAddress": "[variables('bgpPeeringAddress')]", "peerWeight": 0 } }, "comments": "Local Network Gateway (referred to your on-premises location) with IP address of remote tunnel peering and IP address of remote BGP peer"},

3.5 Create the VPN gateway3.5 Create the VPN gateway

The on-premises VPN devices are referred to as the local network gateway. The following json snippet alsospecifies remote BGP peer details:

This section of the template configures the VPN gateway with the required settings for an active-activeconfiguration. Keep in mind the following requirements:

Create the VPN gateway with a "RouteBased" VpnType. This setting is mandatory if you want to enable theBGP routing between the VPN gateway, and the VPN on-premises.To establish VPN tunnels between the two instances of the VPN gateway and a given on-premises device inactive-active mode, the "activeActive" parameter is set to true in the Resource Manager template. Tounderstand more about highly available VPN gateways, see Highly available VPN gateway connectivity.To configure eBGP sessions between the VPN tunnels, you must specify two different ASNs on either side. It ispreferable to specify private ASN numbers. For more information, see Overview of BGP and Azure VPNgateways.

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-highlyavailable

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-bgp-overview

{"apiVersion": "[variables('api-version')]","type": "Microsoft.Network/virtualNetworkGateways","name": "[variables('gatewayName')]","location": "[resourceGroup().location]","dependsOn": [ "[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName1'))]", "[concat('Microsoft.Network/publicIPAddresses/', variables('gatewayPublicIPName2'))]", "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"],"properties": { "ipConfigurations": [ { "properties": { "privateIPAllocationMethod": "Dynamic", "subnet": { "id": "[variables('gatewaySubnetRef')]" }, "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('gatewayPublicIPName1'))]" } }, "name": "vnetGtwConfig1" }, { "properties": { "privateIPAllocationMethod": "Dynamic", "subnet": { "id": "[variables('gatewaySubnetRef')]" }, "publicIPAddress": { "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('gatewayPublicIPName2'))]" } }, "name": "vnetGtwConfig2" } ], "sku": { "name": "[variables('gatewaySku')]", "tier": "[variables('gatewaySku')]" }, "gatewayType": "Vpn", "vpnType": "[variables('vpnType')]", "enableBgp": true, "activeActive": true, "bgpSettings": { "asn": "[variables('asnVpnGateway')]" } }, "comments": "VPN Gateway in active-active configuration with BGP support" },

3.6 Establish the IPsec tunnels3.6 Establish the IPsec tunnelsThe final action of the script creates IPsec tunnels between the Azure VPN gateway and the on-premises VPNdevice.

{ "apiVersion": "[variables('api-version')]", "name": "[variables('connectionName')]", "type": "Microsoft.Network/connections", "location": "[resourceGroup().location]", "dependsOn": [ "[concat('Microsoft.Network/virtualNetworkGateways/', variables('gatewayName'))]", "[concat('Microsoft.Network/localNetworkGateways/', variables('localGatewayName'))]" ], "properties": { "virtualNetworkGateway1": { "id": "[resourceId('Microsoft.Network/virtualNetworkGateways', variables('gatewayName'))]" }, "localNetworkGateway2": { "id": "[resourceId('Microsoft.Network/localNetworkGateways', variables('localGatewayName'))]" }, "connectionType": "IPsec", "routingWeight": 0, "sharedKey": "[variables('sharedKey')]", "enableBGP": "true" }, "comments": "Create a Connection type site-to-site (IPsec) between the Azure VPN Gateway and the VPN device on-premises" }

4. Configure the on-premises VPN device

Cisco CSR1000 exampleCisco CSR1000 example

!crypto ikev2 proposal az-PROPOSAL encryption aes-cbc-256 aes-cbc-128 3des integrity sha1 group 2!crypto ikev2 policy az-POLICY proposal az-PROPOSAL!crypto ikev2 keyring key-peer1 peer azvpn1 address 52.175.253.112 pre-shared-key secret*1234

The Azure VPN gateway is compatible with many VPN devices from different vendors. For configurationinformation and devices that have been validated to work with VPN gateway, see About VPN devices.

When configuring your VPN device, you need the following items:

A shared key. This is the same shared key that you specify when creating your site-to-site VPN connection. Theexamples use a basic shared key. We recommend that you generate a more complex key to use.The Public IP address of your VPN gateway. You can view the public IP address by using the Azure portal,PowerShell, or CLI. To find the Public IP address of your VPN gateway using the Azure portal, navigate toVirtual network gateways, then click the name of your gateway.

Typically eBGP peers are directly connected (often over a WAN connection). However, when you are configuringeBGP over IPsec VPN tunnels via ExpressRoute Microsoft peering, there are multiple routing domains between theeBGP peers. Use the ebgp-multihop command to establish the eBGP neighbor relationship between the two not-directly connected peers. The integer that follows ebgp-multihop command specifies the TTL value in the BGPpackets. The command maximum-paths eibgp 2 enables load balancing of traffic between the two BGP paths.

The following example shows the configuration for Cisco CSR1000 in a Hyper-V virtual machine as the on-premises VPN device:

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-devices

pre-shared-key secret*1234 !!crypto ikev2 keyring key-peer2 peer azvpn2 address 52.175.250.191 pre-shared-key secret*1234 !!!crypto ikev2 profile az-PROFILE1 match address local interface GigabitEthernet1 match identity remote address 52.175.253.112 255.255.255.255 authentication remote pre-share authentication local pre-share keyring local key-peer1!crypto ikev2 profile az-PROFILE2 match address local interface GigabitEthernet1 match identity remote address 52.175.250.191 255.255.255.255 authentication remote pre-share authentication local pre-share keyring local key-peer2!crypto ikev2 dpd 10 2 on-demand!!crypto ipsec transform-set az-IPSEC-PROPOSAL-SET esp-aes 256 esp-sha-hmac mode tunnel!crypto ipsec profile az-VTI1 set transform-set az-IPSEC-PROPOSAL-SET set ikev2-profile az-PROFILE1!crypto ipsec profile az-VTI2 set transform-set az-IPSEC-PROPOSAL-SET set ikev2-profile az-PROFILE2!!interface Loopback0 ip address 172.16.0.3 255.255.255.255!interface Tunnel0 ip address 172.16.0.1 255.255.255.255 ip tcp adjust-mss 1350 tunnel source GigabitEthernet1 tunnel mode ipsec ipv4 tunnel destination 52.175.253.112 tunnel protection ipsec profile az-VTI1!interface Tunnel1 ip address 172.16.0.2 255.255.255.255 ip tcp adjust-mss 1350 tunnel source GigabitEthernet1 tunnel mode ipsec ipv4 tunnel destination 52.175.250.191 tunnel protection ipsec profile az-VTI2!interface GigabitEthernet1 description External interface ip address x.243.229.110 255.255.255.252 negotiation auto no mop enabled no mop sysid!interface GigabitEthernet2 ip address 10.0.0.1 255.255.255.0 negotiation auto no mop enabled no mop sysid

no mop sysid!router bgp 65010 bgp router-id interface Loopback0 bgp log-neighbor-changes network 10.0.0.0 mask 255.255.255.0 network 10.1.10.0 mask 255.255.255.128 neighbor 10.2.0.228 remote-as 65000 neighbor 10.2.0.228 ebgp-multihop 5 neighbor 10.2.0.228 update-source Loopback0 neighbor 10.2.0.228 soft-reconfiguration inbound neighbor 10.2.0.228 filter-list 10 out neighbor 10.2.0.229 remote-as 65000 neighbor 10.2.0.229 ebgp-multihop 5 neighbor 10.2.0.229 update-source Loopback0 neighbor 10.2.0.229 soft-reconfiguration inbound maximum-paths eibgp 2!ip route 0.0.0.0 0.0.0.0 10.1.10.1ip route 10.2.0.228 255.255.255.255 Tunnel0ip route 10.2.0.229 255.255.255.255 Tunnel1!

5. Configure VPN device filtering and firewalls (optional)

6. Test and validate the IPsec tunnel

Get-AzureRmVirtualNetworkGatewayConnection -Name vpn2local1 -ResourceGroupName myRG | Select-Object ConnectionStatus,EgressBytesTransferred,IngressBytesTransferred | fl

ConnectionStatus : ConnectedEgressBytesTransferred : 17734660IngressBytesTransferred : 10538211

Get-AzureRmVirtualNetworkGatewayConnection -Name vpn2local1 -ResourceGroupName myRG | Select-Object -ExpandProperty TunnelConnectionStatus

Tunnel : vpn2local1_52.175.250.191ConnectionStatus : ConnectedIngressBytesTransferred : 4877438EgressBytesTransferred : 8754071LastConnectionEstablishedUtcTime : 11/04/2017 17:03:30

Tunnel : vpn2local1_52.175.253.112ConnectionStatus : ConnectedIngressBytesTransferred : 5660773EgressBytesTransferred : 8980589LastConnectionEstablishedUtcTime : 11/04/2017 17:03:13

Configure your firewall and filtering according to your requirements.

The status of IPsec tunnels can be verified on the Azure VPN gateway by Powershell commands:

Example output:

To check the status of the tunnels on the Azure VPN gateway instances independently, use the following example:

Example output:

show crypto session detailshow crypto ikev2 sashow crypto ikev2 session detailshow crypto ipsec sa

csr1#show crypto session detail

Crypto session current status

Code: C - IKE Configuration mode, D - Dead Peer DetectionK - Keepalives, N - NAT-traversal, T - cTCP encapsulationX - IKE Extended Authentication, F - IKE FragmentationR - IKE Auto Reconnect

Interface: Tunnel1Profile: az-PROFILE2Uptime: 00:52:46Session status: UP-ACTIVEPeer: 52.175.250.191 port 4500 fvrf: (none) ivrf: (none) Phase1_id: 52.175.250.191 Desc: (none) Session ID: 3 IKEv2 SA: local 10.1.10.50/4500 remote 52.175.250.191/4500 Active Capabilities:DN connid:3 lifetime:23:07:14 IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 279 drop 0 life (KB/Sec) 4607976/433 Outbound: #pkts enc'ed 164 drop 0 life (KB/Sec) 4607992/433

Interface: Tunnel0Profile: az-PROFILE1Uptime: 00:52:43Session status: UP-ACTIVEPeer: 52.175.253.112 port 4500 fvrf: (none) ivrf: (none) Phase1_id: 52.175.253.112 Desc: (none) Session ID: 2 IKEv2 SA: local 10.1.10.50/4500 remote 52.175.253.112/4500 Active Capabilities:DN connid:2 lifetime:23:07:17 IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0 Active SAs: 2, origin: crypto map Inbound: #pkts dec'ed 668 drop 0 life (KB/Sec) 4607926/437 Outbound: #pkts enc'ed 477 drop 0 life (KB/Sec) 4607953/437

You can also check the tunnel status on your on-premises VPN device.

Cisco CSR1000 example:

Example output:

The line protocol on the Virtual Tunnel Interface (VTI) does not change to "up" until IKE phase 2 has completed.The following command verifies the security association:

csr1#show crypto ikev2 sa

IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status2 10.1.10.50/4500 52.175.253.112/4500 none/none READY Encr: AES-CBC, keysize: 256, PRF: SHA1, Hash: SHA96, DH Grp:2, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/3277 sec

Tunnel-id Local Remote fvrf/ivrf Status3 10.1.10.50/4500 52.175.250.191/4500 none/none READY Encr: AES-CBC, keysize: 256, PRF: SHA1, Hash: SHA96, DH Grp:2, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/3280 sec

IPv6 Crypto IKEv2 SA

csr1#show crypto ipsec sa | inc encaps|decaps #pkts encaps: 177, #pkts encrypt: 177, #pkts digest: 177 #pkts decaps: 296, #pkts decrypt: 296, #pkts verify: 296 #pkts encaps: 554, #pkts encrypt: 554, #pkts digest: 554 #pkts decaps: 746, #pkts decrypt: 746, #pkts verify: 746

Verify end-to-end connectivity between the inside network on-premises and the Azure VNetVerify end-to-end connectivity between the inside network on-premises and the Azure VNet

csr1#ping 10.2.0.228Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 10.2.0.228, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 5/5/5 ms

#ping 10.2.0.229Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 10.2.0.229, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/6 ms

Verify the BGP sessions over IPsecVerify the BGP sessions over IPsec

Get-AzureRmVirtualNetworkGatewayBGPPeerStatus -VirtualNetworkGatewayName vpnGtw -ResourceGroupName SEA-C1-VPN-ER | ft

Asn ConnectedDuration LocalAddress MessagesReceived MessagesSent Neighbor RoutesReceived State --- ----------------- ------------ ---------------- ------------ -------- -------------- ----- 65010 00:57:19.9003584 10.2.0.228 68 72 172.16.0.10 2 Connected65000 10.2.0.228 0 0 10.2.0.228 0 Unknown 65000 07:13:51.0109601 10.2.0.228 507 500 10.2.0.229 6 Connected

Get-AzureRmVirtualNetworkGatewayLearnedRoute -VirtualNetworkGatewayName vpnGtw -ResourceGroupName myRG | Where-Object Origin -eq "EBgp" |ft

If the IPsec tunnels are up and the static routes are correctly set, you should be able to ping the IP address of theremote BGP peer :

On the Azure VPN gateway, verify the status of BGP peer :

Example output:

To verify the list of network prefixes received via eBGP from the VPN concentrator on-premises, you can filter byattribute "Origin":

AsPath LocalAddress Network NextHop Origin SourcePeer Weight------ ------------ ------- ------- ------ ---------- ------65010 10.2.0.228 10.1.10.0/25 172.16.0.10 EBgp 172.16.0.10 3276865010 10.2.0.228 10.0.0.0/24 172.16.0.10 EBgp 172.16.0.10 32768

Get-AzureRmVirtualNetworkGatewayAdvertisedRoute -VirtualNetworkGatewayName vpnGtw -ResourceGroupName myRG -Peer 10.2.0.228 | ft

AsPath LocalAddress Network NextHop Origin SourcePeer Weight------ ------------ ------- ------- ------ ---------- ------ 10.2.0.229 10.2.0.0/24 10.2.0.229 Igp 0 10.2.0.229 172.16.0.10/32 10.2.0.229 Igp 0 10.2.0.229 172.16.0.5/32 10.2.0.229 Igp 0 10.2.0.229 172.16.0.1/32 10.2.0.229 Igp 065010 10.2.0.229 10.1.10.0/25 10.2.0.229 Igp 065010 10.2.0.229 10.0.0.0/24 10.2.0.229 Igp 0

csr1#show ip bgp neighbors 10.2.0.228 routesBGP table version is 7, local router ID is 172.16.0.10Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, t secondary path,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path *> 10.2.0.0/24 10.2.0.228 0 65000 i r> 172.16.0.1/32 10.2.0.228 0 65000 i r> 172.16.0.2/32 10.2.0.228 0 65000 i r> 172.16.0.3/32 10.2.0.228 0 65000 i

Total number of prefixes 4

csr1#show ip bgp neighbors 10.2.0.228 advertised-routesBGP table version is 7, local router ID is 172.16.0.10Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, t secondary path,Origin codes: i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path *> 10.0.0.0/24 0.0.0.0 0 32768 i *> 10.1.10.0/25 0.0.0.0 0 32768 i

Total number of prefixes 2

In the example output, the ASN 65010 is the BGP autonomous system number in the VPN on-premises.

To see the list of advertised routes:

Example output:

Example for the on-premises Cisco CSR1000:

The list of networks advertised from the on-premises Cisco CSR1000 to the Azure VPN gateway can be listedusing the following command:

Next stepsConfigure Network Performance Monitor for ExpressRoute

Add a site-to-site connection to a VNet with an existing VPN gateway connection

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-multi-site-to-site-resource-manager-portal

Configure a virtual network gateway for ExpressRouteusing the Azure portal6/27/2017 • 3 min to read • Edit Online

Before beginning

Create the gateway subnet

This article walks you through the steps to add a virtual network gateway for a pre-existing VNet. This articlewalks you through the steps to add, resize, and remove a virtual network (VNet) gateway for a pre-existing VNet.The steps for this configuration are specifically for VNets that were created using the Resource Managerdeployment model that will be used in an ExpressRoute configuration. For more information about virtual networkgateways and gateway configuration settings for ExpressRoute, see About virtual network gateways forExpressRoute.

The steps for this task use a VNet based on the values in the following configuration reference list. We use this listin our example steps. You can copy the list to use as a reference, replacing the values with your own.

Configuration reference list

Virtual Network Name = "TestVNet"Virtual Network address space = 192.168.0.0/16Subnet Name = "FrontEnd"

Resource Group = "TestRG"Location = "East US"Gateway Subnet name: "GatewaySubnet" You must always name a gateway subnet GatewaySubnet.

Gateway Name = "ERGW"Gateway IP Name = "MyERGWVIP"Gateway type = "ExpressRoute" This type is required for an ExpressRoute configuration.Gateway Public IP Name = "MyERGWVIP"

Subnet address space = "192.168.1.0/24"

Gateway Subnet address space = "192.168.200.0/26"

You can view a Video of these steps before beginning your configuration.

1. In the portal, navigate to the Resource Manager virtual network for which you want to create a virtual networkgateway.

2. In the Settings section of your VNet blade, click Subnets to expand the Subnets blade.3. On the Subnets blade, click +Gateway subnet to open the Add subnet blade.

4. The Name for your subnet is automatically filled in with the value 'GatewaySubnet'. This value is requiredin order for Azure to recognize the subnet as the gateway subnet. Adjust the auto-filled Address rangevalues to match your configuration requirements. We recommend creating a gateway subnet with a /27 orlarger (/26, /25, etc.). Then, click OK to save the values and create the gateway subnet.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-add-gateway-portal-resource-manager.md

http://azure.microsoft.com/documentation/videos/azure-expressroute-how-to-create-a-vpn-gateway-for-your-virtual-network

http://portal.azure.com

Create the virtual network gateway1. In the portal, on the left side, click + and type 'Virtual Network Gateway' in search. Locate Virtual network

gateway in the search return and click the entry. On the Virtual network gateway blade, click Create at thebottom of the blade. This opens the Create virtual network gateway blade.

3. Name: Name your gateway. This is not the same as naming a gateway subnet. It's the name of the gatewayobject you are creating.

4. Gateway type: Select ExpressRoute.5. SKU : Select the gateway SKU from the dropdown.

2. On the Create virtual network gateway blade, fill in the values for your virtual network gateway.

Next steps

6. Location: Adjust the Location field to point to the location where your virtual network is located. If thelocation is not pointing to the region where your virtual network resides, the virtual network doesn't appear inthe 'Choose a virtual network' dropdown.

7. Choose the virtual network to which you want to add this gateway. Click Virtual network to open the Choosea virtual network blade. Select the VNet. If you don't see your VNet, make sure the Location field is pointingto the region in which your virtual network is located.

8. Choose a public IP address. Click Public IP address to open the Choose public IP address blade. Click+Create New to open the Create public IP address blade. Input a name for your public IP address. Thisblade creates a public IP address object to which a public IP address will be dynamically assigned. Click OK tosave your changes to this blade.

9. Subscription: Verify that the correct subscription is selected.10. Resource group: This setting is determined by the Virtual Network that you select.11. Don't adjust the Location after you've specified the previous settings.12. Verify the settings. If you want your gateway to appear on the dashboard, you can select Pin to dashboard at

the bottom of the blade.13. Click Create to begin creating the gateway. The settings are validated and the gateway deploys. Creating virtual

network gateway can take up to 45 minutes to complete.

After you have created the VNet gateway, you can link your VNet to an ExpressRoute circuit. See Link a VirtualNetwork to an ExpressRoute circuit.

Configure a virtual network gateway forExpressRoute using PowerShell6/27/2017 • 3 min to read • Edit Online

Before beginning

Add a gateway

This article walks you through the steps to add, resize, and remove a virtual network (VNet) gateway for a pre-existing VNet. The steps for this configuration are specifically for VNets that were created using the ResourceManager deployment model that will be used in an ExpressRoute configuration. For more information aboutvirtual network gateways and gateway configuration settings for ExpressRoute, see About virtual networkgateways for ExpressRoute.

Verify that you have installed the latest Azure PowerShell cmdlets. If you haven't installed the latest cmdlets, youneed to do so before beginning the configuration steps. For more information, see Install and configure AzurePowerShell.

The steps for this task use a VNet based on the values in the following configuration reference list. Additionalsettings and names are also outlined in this list. We don't use this list directly in any of the steps, although we doadd variables based on the values in this list. You can copy the list to use as a reference, replacing the values withyour own.

Configuration reference list

Virtual Network Name = "TestVNet"Virtual Network address space = 192.168.0.0/16Resource Group = "TestRG"Subnet1 Name = "FrontEnd"Subnet1 address space = "192.168.1.0/24"Gateway Subnet name: "GatewaySubnet" You must always name a gateway subnet GatewaySubnet.Gateway Subnet address space = "192.168.200.0/26"Region = "East US"Gateway Name = "GW"Gateway IP Name = "GWIP"Gateway IP configuration Name = "gwipconf"Type = "ExpressRoute" This type is required for an ExpressRoute configuration.Gateway Public IP Name = "gwpip"

Login-AzureRmAccountGet-AzureRmSubscription Select-AzureRmSubscription -SubscriptionName "Name of subscription"

1. Connect to your Azure Subscription.

2. Declare your variables for this exercise. Be sure to edit the sample to reflect the settings that you want touse.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-add-gateway-resource-manager.md


Verify the gateway was created

$RG = "TestRG"$Location = "East US"$GWName = "GW"$GWIPName = "GWIP"$GWIPconfName = "gwipconf"$VNetName = "TestVNet"

$vnet = Get-AzureRmVirtualNetwork -Name $VNetName -ResourceGroupName $RG

Add-AzureRmVirtualNetworkSubnetConfig -Name GatewaySubnet -VirtualNetwork $vnet -AddressPrefix 192.168.200.0/26

$vnet = Set-AzureRmVirtualNetwork -VirtualNetwork $vnet

$subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name 'GatewaySubnet' -VirtualNetwork $vnet

$pip = New-AzureRmPublicIpAddress -Name $GWIPName -ResourceGroupName $RG -Location $Location -AllocationMethod Dynamic

$ipconf = New-AzureRmVirtualNetworkGatewayIpConfig -Name $GWIPconfName -Subnet $subnet -PublicIpAddress $pip

New-AzureRmVirtualNetworkGateway -Name $GWName -ResourceGroupName $RG -Location $Location -IpConfigurations $ipconf -GatewayType Expressroute -GatewaySku Standard

3. Store the virtual network object as a variable.

4. Add a gateway subnet to your Virtual Network. The gateway subnet must be named "GatewaySubnet".You should create a gateway subnet that is /27 or larger (/26, /25, etc.).

5. Set the configuration.

6. Store the gateway subnet as a variable.

7. Request a public IP address. The IP address is requested before creating the gateway. You cannot specifythe IP address that you want to use; itâ€™s dynamically allocated. You'll use this IP address in the nextconfiguration section. The AllocationMethod must be Dynamic.

8. Create the configuration for your gateway. The gateway configuration defines the subnet and the public IPaddress to use. In this step, you are specifying the configuration that will be used when you create thegateway. This step does not actually create the gateway object. Use the sample below to create yourgateway configuration.

9. Create the gateway. In this step, the -GatewayType is especially important. You must use the valueExpressRoute. After running these cmdlets, the gateway can take 45 minutes or more to create.

Use the following commands to verify that the gateway has been created:

Get-AzureRmVirtualNetworkGateway -ResourceGroupName $RG

Resize a gateway

IMPORTANTIMPORTANT

$gw = Get-AzureRmVirtualNetworkGateway -Name $GWName -ResourceGroupName $RGResize-AzureRmVirtualNetworkGateway -VirtualNetworkGateway $gw -GatewaySku HighPerformance

Remove a gateway

Remove-AzureRmVirtualNetworkGateway -Name $GWName -ResourceGroupName $RG

Next steps

There are a number of Gateway SKUs. You can use the following command to change the Gateway SKU at anytime.

This command doesn't work for UltraPerformance gateway. To change your gateway to an UltraPerformance gateway, firstremove the existing ExpressRoute gateway, and then create a new UltraPerformance gateway. To downgrade your gatewayfrom an UltraPerformance gateway, first remove the UltraPerformance gateway, and then create a new gateway.

Use the following command to remove a gateway:


Configure ExpressRoute and Site-to-Site coexistingconnections3/8/2018 • 8 min to read • Edit Online

IMPORTANTIMPORTANT

Limits and limitations

Configuration designsConfigure a Site-to-Site VPN as a failover path for ExpressRouteConfigure a Site-to-Site VPN as a failover path for ExpressRoute

NOTENOTE

Configuring Site-to-Site VPN and ExpressRoute coexisting connections has several advantages. You can configurea Site-to-Site VPN as a secure failover path for ExpressRoute, or use Site-to-Site VPNs to connect to sites that arenot connected through ExpressRoute. We cover the steps to configure both scenarios in this article. This articleapplies to the Resource Manager deployment model and uses PowerShell. This configuration is not available in theAzure portal.

ExpressRoute circuits must be pre-configured before you follow the instructions below. Make sure that you have followed theguides to create an ExpressRoute circuit and configure routing before you proceed.

Transit routing is not supported. You cannot route (via Azure) between your local network connected viaSite-to-Site VPN and your local network connected via ExpressRoute.Basic SKU gateway is not supported. You must use a non-Basic SKU gateway for both the ExpressRoutegateway and the VPN gateway.Only route-based VPN gateway is supported. You must use a route-based VPN Gateway.Static route should be configured for your VPN gateway. If your local network is connected to bothExpressRoute and a Site-to-Site VPN, you must have a static route configured in your local network to routethe Site-to-Site VPN connection to the public Internet.ExpressRoute gateway must be configured first and linked to a circuit. You must create the ExpressRoutegateway first and link it to a circuit before you add the Site-to-Site VPN gateway.

You can configure a Site-to-Site VPN connection as a backup for ExpressRoute. This applies only to virtualnetworks linked to the Azure private peering path. There is no VPN-based failover solution for services accessiblethrough Azure public and Microsoft peerings. The ExpressRoute circuit is always the primary link. Data flowsthrough the Site-to-Site VPN path only if the ExpressRoute circuit fails.

While ExpressRoute circuit is preferred over Site-to-Site VPN when both routes are the same, Azure will use the longest prefixmatch to choose the route towards the packet's destination.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-coexist-resource-manager.md

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways


Configure a Site-to-Site VPN to connect to sites not connected through ExpressRouteConfigure a Site-to-Site VPN to connect to sites not connected through ExpressRoute

NOTENOTE

Selecting the steps to use

You can configure your network where some sites connect directly to Azure over Site-to-Site VPN, and some sitesconnect through ExpressRoute.

You cannot configure a virtual network as a transit router.

There are two different sets of procedures to choose from. The configuration procedure that you select depends onwhether you have an existing virtual network that you want to connect to, or you want to create a new virtualnetwork.

I don't have a VNet and need to create one.

If you donâ€™t already have a virtual network, this procedure walks you through creating a new virtualnetwork using Resource Manager deployment model and creating new ExpressRoute and Site-to-Site VPNconnections. To configure a virtual network, follow the steps in To create a new virtual network andcoexisting connections.

I already have a Resource Manager deployment model VNet.

You may already have a virtual network in place with an existing Site-to-Site VPN connection orExpressRoute connection. The To configure coexisting connections for an already existing VNet section

To create a new virtual network and coexisting connections

walks you through deleting the gateway, and then creating new ExpressRoute and Site-to-Site VPNconnections. When creating the new connections, the steps must be completed in a specific order. Don't usethe instructions in other articles to create your gateways and connections.

In this procedure, creating connections that can coexist requires you to delete your gateway, and thenconfigure new gateways. You will have downtime for your cross-premises connections while you delete andrecreate your gateway and connections, but you will not need to migrate any of your VMs or services to anew virtual network. Your VMs and services will still be able to communicate out through the load balancerwhile you configure your gateway if they are configured to do so.

This procedure walks you through creating a VNet and Site-to-Site and ExpressRoute connections that will coexist.

1. Install the latest version of the Azure PowerShell cmdlets. For information about installing the cmdlets, see Howto install and configure Azure PowerShell. The cmdlets that you use for this configuration may be slightlydifferent than what you might be familiar with. Be sure to use the cmdlets specified in these instructions.

login-AzureRmAccountSelect-AzureRmSubscription -SubscriptionName 'yoursubscription'$location = "Central US"$resgrp = New-AzureRmResourceGroup -Name "ErVpnCoex" -Location $location$VNetASN = 65010

IMPORTANTIMPORTANT

$vnet = New-AzureRmVirtualNetwork -Name "CoexVnet" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -AddressPrefix "10.200.0.0/16"

Add-AzureRmVirtualNetworkSubnetConfig -Name "App" -VirtualNetwork $vnet -AddressPrefix "10.200.1.0/24"Add-AzureRmVirtualNetworkSubnetConfig -Name "GatewaySubnet" -VirtualNetwork $vnet -AddressPrefix "10.200.255.0/24"


2. Log in to your account and set up the environment.

3. Create a virtual network including Gateway Subnet. For more information about creating a virtual network,see Create a virtual network. For more information about creating subnets, see Create a subnet

The Gateway Subnet must be /27 or a shorter prefix (such as /26 or /25).

Create a new VNet.

Add subnets.

Save the VNet configuration.

4. Create an ExpressRoute gateway. For more information about the ExpressRoute gateway configuration, seeExpressRoute gateway configuration. The GatewaySKU must be Standard, HighPerformance, orUltraPerformance.


https://docs.microsoft.com/en-us/azure/virtual-network/manage-virtual-network

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-subnet

$gwSubnet = Get-AzureRmVirtualNetworkSubnetConfig -Name "GatewaySubnet" -VirtualNetwork $vnet$gwIP = New-AzureRmPublicIpAddress -Name "ERGatewayIP" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -AllocationMethod Dynamic$gwConfig = New-AzureRmVirtualNetworkGatewayIpConfig -Name "ERGatewayIpConfig" -SubnetId $gwSubnet.Id -PublicIpAddressId $gwIP.Id$gw = New-AzureRmVirtualNetworkGateway -Name "ERGateway" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -IpConfigurations $gwConfig -GatewayType "ExpressRoute" -GatewaySku Standard

$ckt = Get-AzureRmExpressRouteCircuit -Name "YourCircuit" -ResourceGroupName "YourCircuitResourceGroup"New-AzureRmVirtualNetworkGatewayConnection -Name "ERConnection" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -VirtualNetworkGateway1 $gw -PeerId $ckt.Id -ConnectionType ExpressRoute

$gwSubnet = Get-AzureRmVirtualNetworkSubnetConfig -Name "GatewaySubnet" -VirtualNetwork $vnet$gwIP = New-AzureRmPublicIpAddress -Name "VPNGatewayIP" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -AllocationMethod Dynamic$gwConfig = New-AzureRmVirtualNetworkGatewayIpConfig -Name "VPNGatewayIpConfig" -SubnetId $gwSubnet.Id -PublicIpAddressId $gwIP.IdNew-AzureRmVirtualNetworkGateway -Name "VPNGateway" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -IpConfigurations $gwConfig -GatewayType "Vpn" -VpnType "RouteBased" -GatewaySku "Standard"

$azureVpn = New-AzureRmVirtualNetworkGateway -Name "VPNGateway" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -IpConfigurations $gwConfig -GatewayType "Vpn" -VpnType "RouteBased" -GatewaySku "Standard" -Asn $VNetASN

$MyLocalNetworkAddress = @("10.100.0.0/16","10.101.0.0/16","10.102.0.0/16")$localVpn = New-AzureRmLocalNetworkGateway -Name "LocalVPNGateway" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -GatewayIpAddress *<Public IP>* -AddressPrefix $MyLocalNetworkAddress

5. Link the ExpressRoute gateway to the ExpressRoute circuit. After this step has been completed, theconnection between your on-premises network and Azure, through ExpressRoute, is established. For moreinformation about the link operation, see Link VNets to ExpressRoute.

6. Next, create your Site-to-Site VPN gateway. For more information about the VPN gateway configuration,see Configure a VNet with a Site-to-Site connection. The GatewaySKU must be Standard,HighPerformance, or UltraPerformance. The VpnType must RouteBased.

Azure VPN gateway supports BGP routing protocol. You can specify ASN (AS Number) for that VirtualNetwork by adding the -Asn switch in the following command. Not specifying that parameter will default toAS number 65515.

You can find the BGP peering IP and the AS number that Azure uses for the VPN gateway in$azureVpn.BgpSettings.BgpPeeringAddress and $azureVpn.BgpSettings.Asn. For more information, seeConfigure BGP for Azure VPN gateway.

7. Create a local site VPN gateway entity. This command doesnâ€™t configure your on-premises VPNgateway. Rather, it allows you to provide the local gateway settings, such as the public IP and the on-premises address space, so that the Azure VPN gateway can connect to it.

If your local VPN device only supports static routing, you can configure the static routes in the followingway:

If your local VPN device supports the BGP and you want to enable dynamic routing, you need to know the


https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-bgp-resource-manager-ps

To configure coexisting connections for an already existing VNet

NOTENOTE

$localVPNPublicIP = "<Public IP>"$localBGPPeeringIP = "<Private IP for the BGP session>"$localBGPASN = "<ASN>"$localAddressPrefix = $localBGPPeeringIP + "/32"$localVpn = New-AzureRmLocalNetworkGateway -Name "LocalVPNGateway" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -GatewayIpAddress $localVPNPublicIP -AddressPrefix $localAddressPrefix -BgpPeeringAddress $localBGPPeeringIP -Asn $localBGPASN

8. Configure your local VPN device to connect to the new Azure VPN gateway. For more information about VPNdevice configuration, see VPN Device Configuration.

$azureVpn = Get-AzureRmVirtualNetworkGateway -Name "VPNGateway" -ResourceGroupName $resgrp.ResourceGroupNameNew-AzureRmVirtualNetworkGatewayConnection -Name "VPNConnection" -ResourceGroupName $resgrp.ResourceGroupName -Location $location -VirtualNetworkGateway1 $azureVpn -LocalNetworkGateway2 $localVpn -ConnectionType IPsec -SharedKey <yourkey>

BGP peering IP and the AS number that your local VPN device uses.

9. Link the Site-to-Site VPN gateway on Azure to the local gateway.

If you have an existing virtual network, check the gateway subnet size. If the gateway subnet is /28 or /29, youmust first delete the virtual network gateway and increase the gateway subnet size. The steps in this section showyou how to do that.

If the gateway subnet is /27 or larger and the virtual network is connected via ExpressRoute, you can skip the stepsbelow and proceed to "Step 6 - Create a Site-to-Site VPN gateway" in the previous section.

When you delete the existing gateway, your local premises will lose the connection to your virtual network while you areworking on this configuration.

1. You'll need to install the latest version of the Azure PowerShell cmdlets. For more information about installingcmdlets, see How to install and configure Azure PowerShell. The cmdlets that you use for this configurationmay be slightly different than what you might be familiar with. Be sure to use the cmdlets specified in theseinstructions.

Remove-AzureRmVirtualNetworkGateway -Name <yourgatewayname> -ResourceGroupName <yourresourcegroup>

$vnet = Get-AzureRmVirtualNetwork -Name <yourvnetname> -ResourceGroupName <yourresourcegroup> Remove-AzureRmVirtualNetworkSubnetConfig -Name GatewaySubnet -VirtualNetwork $vnet

2. Delete the existing ExpressRoute or Site-to-Site VPN gateway.

3. Delete Gateway Subnet.

4. Add a Gateway Subnet that is /27 or larger.



To add point-to-site configuration to the VPN gateway

Next steps

NOTENOTE

$vnet = Get-AzureRmVirtualNetwork -Name <yourvnetname> -ResourceGroupName <yourresourcegroup>Add-AzureRmVirtualNetworkSubnetConfig -Name "GatewaySubnet" -VirtualNetwork $vnet -AddressPrefix "10.200.255.0/24"


5. At this point, you have a VNet with no gateways. To create new gateways and complete your connections, youcan proceed with Step 4 - Create an ExpressRoute gateway, found in the preceding set of steps.

If you don't have enough IP addresses left in your virtual network to increase the gateway subnet size, you need toadd more IP address space.

Save the VNet configuration.

You can follow the steps below to add Point-to-Site configuration to your VPN gateway in a co-existence setup.

$azureVpn = Get-AzureRmVirtualNetworkGateway -Name "VPNGateway" -ResourceGroupName $resgrp.ResourceGroupNameSet-AzureRmVirtualNetworkGatewayVpnClientConfig -VirtualNetworkGateway $azureVpn -VpnClientAddressPool "10.251.251.0/24"

$p2sCertFullName = "RootErVpnCoexP2S.cer" $p2sCertMatchName = "RootErVpnCoexP2S" $p2sCertToUpload=get-childitem Cert:\CurrentUser\My | Where-Object {$_.Subject -match $p2sCertMatchName} if ($p2sCertToUpload.count -eq 1){write-host "cert found"} else {write-host "cert not found" exit} $p2sCertData = [System.Convert]::ToBase64String($p2sCertToUpload.RawData) Add-AzureRmVpnClientRootCertificate -VpnClientRootCertificateName $p2sCertFullName -VirtualNetworkGatewayname $azureVpn.Name -ResourceGroupName $resgrp.ResourceGroupName -PublicCertData $p2sCertData

1. Add VPN Client address pool.

2. Upload the VPN root certificate to Azure for your VPN gateway. In this example, it's assumed that the rootcertificate is stored in the local machine where the following PowerShell cmdlets are run.

For more information on Point-to-Site VPN, see Configure a Point-to-Site connection.


https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-point-to-site-rm-ps

Configure route filters for Microsoft peering: Azureportal2/16/2018 • 5 min to read • Edit Online

About route filtersAbout route filters

IMPORTANTIMPORTANT

WorkflowWorkflow

Route filters are a way to consume a subset of supported services through Microsoft peering. The steps in thisarticle help you configure and manage route filters for ExpressRoute circuits.

Dynamics 365 services, and Office 365 services such as Exchange Online, SharePoint Online, and Skype forBusiness, and Azure services such as storage and SQL DB are accessible through Microsoft peering. WhenMicrosoft peering is configured in an ExpressRoute circuit, all prefixes related to these services are advertisedthrough the BGP sessions that are established. A BGP community value is attached to every prefix to identify theservice that is offered through the prefix. For a list of the BGP community values and the services they map to, seeBGP communities.

If you require connectivity to all services, a large number of prefixes are advertised through BGP. This significantlyincreases the size of the route tables maintained by routers within your network. If you plan to consume only asubset of services offered through Microsoft peering, you can reduce the size of your route tables in two ways.You can:

Filter out unwanted prefixes by applying route filters on BGP communities. This is a standard networkingpractice and is used commonly within many networks.

Define route filters and apply them to your ExpressRoute circuit. A route filter is a new resource that letsyou select the list of services you plan to consume through Microsoft peering. ExpressRoute routers onlysend the list of prefixes that belong to the services identified in the route filter.

When Microsoft peering is configured on your ExpressRoute circuit, the Microsoft edge routers establish a pair ofBGP sessions with the edge routers (yours or your connectivity provider's). No routes are advertised to yournetwork. To enable route advertisements to your network, you must associate a route filter.

A route filter lets you identify services you want to consume through your ExpressRoute circuit's Microsoftpeering. It is essentially a white list of all the BGP community values. Once a route filter resource is defined andattached to an ExpressRoute circuit, all prefixes that map to the BGP community values are advertised to yournetwork.

To be able to attach route filters with Office 365 services on them, you must have authorization to consume Office365 services through ExpressRoute. If you are not authorized to consume Office 365 services throughExpressRoute, the operation to attach route filters fails. For more information about the authorization process, seeAzure ExpressRoute for Office 365. Connectivity to Dynamics 365 services does NOT require any priorauthorization.

Microsoft peering of ExpressRoute circuits that were configured prior to August 1, 2017 will have all service prefixesadvertised through Microsoft peering, even if route filters are not defined. Microsoft peering of ExpressRoute circuits that areconfigured on or after August 1, 2017 will not have any prefixes advertised until a route filter is attached to the circuit.

To be able to successfully connect to services through Microsoft peering, you must complete the following

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/how-to-routefilter-portal.md

https://support.office.com/article/Azure-ExpressRoute-for-Office-365-6d2534a2-c19c-4a99-be5e-33a0cee5d3bd

Before you begin

Step 1: Get a list of prefixes and BGP community values1. Get a list of BGP community values1. Get a list of BGP community values

2. Make a list of the values that you want to use2. Make a list of the values that you want to use

Step 2: Create a route filter and a filter rule

1. Create a route filter1. Create a route filter

configuration steps:

You must have an active ExpressRoute circuit that has Microsoft peering provisioned. You can use thefollowing instructions to accomplish these tasks:

Create an ExpressRoute circuit and have the circuit enabled by your connectivity provider before youproceed. The ExpressRoute circuit must be in a provisioned and enabled state.Create Microsoft peering if you manage the BGP session directly. Or, have your connectivity providerprovision Microsoft peering for your circuit.

You must create and configure a route filter.

Identify the services you with to consume through Microsoft peeringIdentify the list of BGP community values associated with the servicesCreate a rule to allow the prefix list matching the BGP community values

You must attach the route filter to the ExpressRoute circuit.

Before you begin configuration, make sure you meet the following criteria:

Review the prerequisites and workflows before you begin configuration.

You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit mustbe in a provisioned and enabled state.

You must have an active Microsoft peering. Follow instructions at Create and modifying peeringconfiguration

BGP community values associated with services accessible through Microsoft peering is available in theExpressRoute routing requirements page.

Make a list of BGP community values you want to use in the route filter. As an example, the BGP community valuefor Dynamics 365 services is 12076:5040.

A route filter can have only one rule, and the rule must be of type 'Allow'. This rule can have a list of BGPcommunity values associated with it.

You can create a route filter by selecting the option to create a new resource. Click Create a resource >Networking > RouteFilter, as shown in the following image:

2. Create a filter rule2. Create a filter rule

You must place the route filter in a resource group.

You can add and update rules by selecting the manage rule tab for your route filter.

You can select the services you want to connect to from the drop down list and save the rule when done.

Step 3: Attach the route filter to an ExpressRoute circuitYou can attach the route filter to a circuit by selecting the "add Circuit" button and selecting the ExpressRoutecircuit from the drop down list.

If the connectivity provider configures peering for your ExpressRoute circuit refresh the circuit from the

Common tasksTo get the properties of a route filterTo get the properties of a route filter

To update the properties of a route filterTo update the properties of a route filter

ExpressRoute circuit blade before you select the "add Circuit" button.

You can view properties of a route filter when you open the resource in the portal.

You can update the list of BGP community values attached to a circuit by selecting the "Manage rule" button.

To detach a route filter from an ExpressRoute circuitTo detach a route filter from an ExpressRoute circuit

To delete a route filterTo delete a route filter

To detach a circuit from the route filter, right click on the circuit and click on "disassociate".

Next Steps

You can delete a route filter by selecting the delete button.


Configure route filters for Microsoft peering:PowerShell1/23/2018 • 7 min to read • Edit Online


IMPORTANTIMPORTANT

WorkflowWorkflow


Dynamics 365 services, and Office 365 services such as Exchange Online, SharePoint Online, and Skype forBusiness, and Azure public services, such as storage and SQL DB are accessible through Microsoft peering.Azure public services are selectable on a per region basis and cannot be defined per public service.

When Microsoft peering is configured on an ExpressRoute circuit and a route filter is attached, all prefixes thatare selected for these services are advertised through the BGP sessions that are established. A BGP communityvalue is attached to every prefix to identify the service that is offered through the prefix. For a list of the BGPcommunity values and the services they map to, see BGP communities.

If you require connectivity to all services, a large number of prefixes are advertised through BGP. Thissignificantly increases the size of the route tables maintained by routers within your network. If you plan toconsume only a subset of services offered through Microsoft peering, you can reduce the size of your routetables in two ways. You can:



When Microsoft peering is configured on your ExpressRoute circuit, the Microsoft edge routers establish a pairof BGP sessions with the edge routers (yours or your connectivity provider's). No routes are advertised to yournetwork. To enable route advertisements to your network, you must associate a route filter.


To be able to attach route filters with Office 365 services on them, you must have authorization to consumeOffice 365 services through ExpressRoute. If you are not authorized to consume Office 365 services throughExpressRoute, the operation to attach route filters fails. For more information about the authorization process,see Azure ExpressRoute for Office 365. Connectivity to Dynamics 365 services does NOT require any priorauthorization.

Microsoft peering of ExpressRoute circuits that were configured prior to August 1, 2017 will have all service prefixesadvertised through Microsoft peering, even if route filters are not defined. Microsoft peering of ExpressRoute circuits thatare configured on or after August 1, 2017 will not have any prefixes advertised until a route filter is attached to the circuit.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/how-to-routefilter-powershell.md


Before you begin

Log in to your Azure accountLog in to your Azure account



To be able to successfully connect to services through Microsoft peering, you must complete the followingconfiguration steps:






Before you begin configuration, make sure you meet the following criteria:

NOTENOTE

Install the latest version of the Azure Resource Manager PowerShell cmdlets. For more information, seeInstall and configure Azure PowerShell.

Download the latest version from the PowerShell Gallery, rather than using the Installer. The Installer currently doesnot support the required cmdlets.


You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuitand have the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuitmust be in a provisioned and enabled state.


Before beginning this configuration, you must log in to your Azure account. The cmdlet prompts you for thelogin credentials for your Azure account. After logging in, it downloads your account settings so they areavailable to Azure PowerShell.

Open your PowerShell console with elevated privileges, and connect to your account. Use the following exampleto help you connect:

If you have multiple Azure subscriptions, check the subscriptions for the account.

Specify the subscription that you want to use.

https://docs.microsoft.com/powershell/azure/install-azurerm-ps

Select-AzureRmSubscription -SubscriptionName "Replace_with_your_subscription_name"


Get-AzureRmBgpServiceCommunity




New-AzureRmRouteFilter -Name "MyRouteFilter" -ResourceGroupName "MyResourceGroup" -Location "West US"


$rule = New-AzureRmRouteFilterRuleConfig -Name "Allow-EXO-D365" -Access Allow -RouteFilterRuleType Community -CommunityList "12076:5010,12076:5040"

3. Add the rule to the route filter3. Add the rule to the route filter

$routefilter = Get-AzureRmRouteFilter -Name "RouteFilterName" -ResourceGroupName "ExpressRouteResourceGroupName"$routefilter.Rules.Add($rule)Set-AzureRmRouteFilter -RouteFilter $routefilter

Step 3: Attach the route filter to an ExpressRoute circuit

Use the following cmdlet to get the list of BGP community values associated with services accessible throughMicrosoft peering, and the list of prefixes associated with them:

Make a list of BGP community values you want to use in the route filter. As an example, the BGP communityvalue for Dynamics 365 services is 12076:5040.


First, create the route filter. The command 'New-AzureRmRouteFilter' only creates a route filter resource. Afteryou create the resource, you must then create a rule and attach it to the route filter object. Run the followingcommand to create a route filter resource:

You can specify a set of BGP communities as a comma-separated list, as shown in the example. Run the followingcommand to create a new rule:

Run the following command to add the filter rule to the route filter :

Run the following command to attach the route filter to the ExpressRoute circuit, assuming you have onlyMicrosoft peering:

$ckt = Get-AzureRmExpressRouteCircuit -Name "ExpressRouteARMCircuit" -ResourceGroupName "ExpressRouteResourceGroup"$ckt.Peerings[0].RouteFilter = $routefilter Set-AzureRmExpressRouteCircuit -ExpressRouteCircuit $ckt



$routefilter = Get-AzureRmRouteFilter -Name "RouteFilterName" -ResourceGroupName "ExpressRouteResourceGroupName"$routefilter.rules[0].Communities = "12076:5030", "12076:5040"Set-AzureRmRouteFilter -RouteFilter $routefilter


$ckt.Peerings[0].RouteFilter = $nullSet-AzureRmExpressRouteCircuit -ExpressRouteCircuit $ckt


Remove-AzureRmRouteFilter -Name "MyRouteFilter" -ResourceGroupName "MyResourceGroup"

Next Steps

To get the properties of a route filter, use the following steps:

$routefilter = Get-AzureRmRouteFilter -Name "RouteFilterName" -ResourceGroupName "ExpressRouteResourceGroupName"

$routefilter = Get-AzureRmRouteFilter -Name "RouteFilterName" -ResourceGroupName "ExpressRouteResourceGroupName"$rule = $routefilter.Rules[0]

1. Run the following command to get the route filter resource:

2. Get the route filter rules for the route-filter resource by running the following command:

If the route filter is already attached to a circuit, updates to the BGP community list automatically propagateappropriate prefix advertisement changes through the established BGP sessions. You can update the BGPcommunity list of your route filter using the following command:

Once a route filter is detached from the ExpressRoute circuit, no prefixes are advertised through the BGP session.You can detach a route filter from an ExpressRoute circuit using the following command:

You can only delete a route filter if it is not attached to any circuit. Ensure that the route filter is not attached toany circuit before attempting to delete it. You can delete a route filter using the following command:


Configure route filters for Microsoft peering: AzureCLI9/27/2017 • 6 min to read • Edit Online


IMPORTANTIMPORTANT

WorkflowWorkflow


Dynamics 365 services, and Office 365 services such as Exchange Online, SharePoint Online, and Skype forBusiness, are accessible through the Microsoft peering. When Microsoft peering is configured in an ExpressRoutecircuit, all prefixes related to these services are advertised through the BGP sessions that are established. A BGPcommunity value is attached to every prefix to identify the service that is offered through the prefix. For a list ofthe BGP community values and the services they map to, see BGP communities.

If you require connectivity to all services, a large number of prefixes are advertised through BGP. This significantlyincreases the size of the route tables maintained by routers within your network. If you plan to consume only asubset of services offered through Microsoft peering, you can reduce the size of your route tables in two ways.You can:



When Microsoft peering is configured on your ExpressRoute circuit, the Microsoft edge routers establish a pair ofBGP sessions with the edge routers (yours or your connectivity provider's). No routes are advertised to yournetwork. To enable route advertisements to your network, you must associate a route filter.


To be able to attach route filters with Office 365 services on them, you must have authorization to consume Office365 services through ExpressRoute. If you are not authorized to consume Office 365 services throughExpressRoute, the operation to attach route filters fails. For more information about the authorization process, seeAzure ExpressRoute for Office 365. Connectivity to Dynamics 365 services does NOT require any priorauthorization.

Microsoft peering of ExpressRoute circuits that were configured prior to August 1, 2017 will have all service prefixesadvertised through Microsoft peering, even if route filters are not defined. Microsoft peering of ExpressRoute circuits that areconfigured on or after August 1, 2017 will not have any prefixes advertised until a route filter is attached to the circuit.

To be able to successfully connect to services through Microsoft peering, you must complete the followingconfiguration steps:

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/how-to-routefilter-cli.md


Before you begin

Sign in to your Azure account and select your subscriptionSign in to your Azure account and select your subscription

az login

az account list



az network route-filter rule list-service-communities







Before beginning, install the latest version of the CLI commands (2.0 or later). For information about installing theCLI commands, see Install Azure CLI 2.0 and Get Started with Azure CLI 2.0.


You must have an active ExpressRoute circuit. Follow the instructions to Create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit mustbe in a provisioned and enabled state.



Check the subscriptions for the account.

Select the subscription for which you want to create an ExpressRoute circuit.

Use the following cmdlet to get the list of BGP community values associated with services accessible throughMicrosoft peering, and the list of prefixes associated with them:

Make a list of BGP community values you want to use in the route filter. As an example, the BGP community value


https://docs.microsoft.com/cli/azure/get-started-with-azure-cli



az network route-filter create -n MyRouteFilter -g MyResourceGroup


az network route-filter rule create --filter-name MyRouteFilter -n CRM --communities 12076:5040 --access Allow -g MyResourceGroup

Step 3: Attach the route filter to an ExpressRoute circuit

az network express-route peering update --circuit-name MyCircuit -g ExpressRouteResourceGroupName --name MicrosoftPeering --route-filter MyRouteFilter


az network route-filter show -g ExpressRouteResourceGroupName --name MyRouteFilter


az network route-filter rule update --filter-name MyRouteFilter -n CRM -g ExpressRouteResourceGroupName --add communities '12076:5040' --add communities '12076:5010'


az network express-route peering update --circuit-name MyCircuit -g ExpressRouteResourceGroupName --name MicrosoftPeering --remove routeFilter

for Dynamics 365 services is 12076:5040.


First, create the route filter. The command 'az network route-filter create' only creates a route filter resource. Afteryou create the resource, you must then create a rule and attach it to the route filter object. Run the followingcommand to create a route filter resource:

Run the following command to create a new rule:

Run the following command to attach the route filter to the ExpressRoute circuit:

To get the properties of a route filter, use the following command:

If the route filter is already attached to a circuit, updates to the BGP community list automatically propagateappropriate prefix advertisement changes through the established BGP sessions. You can update the BGPcommunity list of your route filter using the following command:

Once a route filter is detached from the ExpressRoute circuit, no prefixes are advertised through the BGP session.You can detach a route filter from an ExpressRoute circuit using the following command:


az network route-filter delete -n MyRouteFilter -g MyResourceGroup

Next Steps

You can only delete a route filter if it is not attached to any circuit. Ensure that the route filter is not attached to anycircuit before attempting to delete it. You can delete a route filter using the following command:


Move a public peering to Microsoft peering4/9/2018 • 2 min to read • Edit Online

Before you begin

1. Create Microsoft peering

2. Validate Microsoft peering is enabled

3. Configure and attach a route filter to the circuit

ExpressRoute supports using Microsoft peering with route filters for Azure PaaS services, such as Azure storageand Azure SQL Database. You now need only one routing domain to access Microsoft PaaS and SaaS services. Youcan use route filters to selectively advertise the PaaS service prefixes for Azure regions you want to consume.

This article helps you move a public peering configuration to Microsoft peering with no downtime. For moreinformation about routing domains and peerings, see ExpressRoute circuits and routing domains.

To connect to Microsoft peering, you need to set up and manage NAT. Your connectivity provider may set upand manage the NAT as a managed service. If you are planning to access the Azure PaaS and Azure SaaSservices on Microsoft peering, it's important to size the NAT IP pool correctly. For more information aboutNAT for ExpressRoute, see the NAT requirements for Microsoft peering.

If you are using public peering and currently have IP Network rules for public IP addresses that are used toaccess Azure Storage or Azure SQL Database, you need to make sure that the NAT IP pool configured withMicrosoft peering is included in the list of public IP addresses for the Azure storage account or Azure SQLaccount.

In order to move to Microsoft peering with no downtime, use the steps in this article in the order that theyare presented.

If Microsoft peering has not been created, use any of the following articles to create Microsoft peering. If yourconnectivity provider offers managed layer 3 services, you can ask the connectivity provider to enable Microsoftpeering for your circuit.

Create Microsoft peering using Azure portalCreate Microsoft peering using Azure PowershellCreate Microsoft peering using Azure CLI

Verify that the Microsoft peering is enabled and the advertised public prefixes are in the configured state.

Azure portalAzure PowerShellAzure CLI

By default, new Microsoft peerings do not advertise any prefixes until a route filter is attached to the circuit. Whenyou create a route filter rule, you can specify the list of service communities for Azure regions that you want toconsume for Azure PaaS services, as shown in the following screenshot:

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/how-to-move-peering.md

https://docs.microsoft.com/en-us/azure/storage/common/storage-network-security

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-vnet-service-endpoint-rule-overview

4. Delete the public peering

5. View peerings

Configure route filters using any of the following articles:

Configure route filters for Microsoft peering using Azure portalConfigure route filters for Microsoft peering using Azure PowerShellConfigure route filters for Microsoft peering using Azure CLI

After verifying that the Microsoft peering is configured and the prefixes you wish to consume are correctlyadvertised on Microsoft peering, you can then delete the public peering. To delete the public peering, use any of thefollowing articles:

Delete Azure public peering using Azure portalDelete Azure public peering using Azure PowerShellDelete Azure public peering using CLI

Next steps

You can see a list of all ExpressRoute circuits and peerings in the Azure portal. For more information, see ViewMicrosoft peering details.


Move ExpressRoute circuits from the classic to theResource Manager deployment model usingPowerShell6/27/2017 • 3 min to read • Edit Online

Before you begin

Move an ExpressRoute circuitStep 1: Gather circuit details from the classic deployment modelStep 1: Gather circuit details from the classic deployment model

Step 2: Sign in and create a resource groupStep 2: Sign in and create a resource group

To use an ExpressRoute circuit for both the classic and Resource Manager deployment models, you must movethe circuit to the Resource Manager deployment model. The following sections help you move your circuit byusing PowerShell.

Verify that you have the latest version of the Azure PowerShell modules (at least version 1.0). For moreinformation, see How to install and configure Azure PowerShell.Make sure that you have reviewed the prerequisites, routing requirements, and workflows before you beginconfiguration.Review the information that is provided under Moving an ExpressRoute circuit from classic to ResourceManager. Make sure that you fully understand the limits and limitations.Verify that the circuit is fully operational in the classic deployment model.Ensure that you have a resource group that was created in the Resource Manager deployment model.

Sign in to the Azure classic environment and gather the service key.

Add-AzureAccount

Select-AzureSubscription "<Enter Subscription Name here>"

Import-Module 'C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1'Import-Module 'C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\ExpressRoute\ExpressRoute.psd1'

Get-AzureDedicatedCircuit

1. Sign in to your Azure account.

2. Select the appropriate Azure subscription.

3. Import the PowerShell modules for Azure and ExpressRoute.

4. Use the cmdlet below to get the service keys for all of your ExpressRoute circuits. After retrieving the keys,copy the service key of the circuit that you want to move to the Resource Manager deployment model.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-move-arm.md


Step 3: Move the ExpressRoute circuit to the Resource Manager deployment modelStep 3: Move the ExpressRoute circuit to the Resource Manager deployment model

Move-AzureRmExpressRouteCircuit -Name "MyCircuit" -ResourceGroupName "DemoRG" -Location "West US" -ServiceKey "<Service-key>"

NOTENOTE

Modify circuit accessTo enable ExpressRoute circuit access for both deployment modelsTo enable ExpressRoute circuit access for both deployment models

Sign in to the Resource Manager environment and create a new resource group.


Get-AzureRmSubscription -SubscriptionName "<Enter Subscription Name here>" | Select-AzureRmSubscription

New-AzureRmResourceGroup -Name "DemoRG" -Location "West US"

1. Sign in to your Azure Resource Manager environment.

2. Select the appropriate Azure subscription.

3. Modify the snippet below to create a new resource group if you don't already have a resource group.

You are now ready to move your ExpressRoute circuit from the classic deployment model to the ResourceManager deployment model. Before proceeding, review the information provided in Moving an ExpressRoutecircuit from the classic to the Resource Manager deployment model.

To move your circuit, modify and run the following snippet:

After the move has finished, the new name that is listed in the previous cmdlet will be used to address the resource. Thecircuit will essentially be renamed.

After moving your classic ExpressRoute circuit to the Resource Manager deployment model, you can enableaccess to both deployment models. Run the following cmdlets to enable access to both deployment models:

$ckt = Get-AzureRmExpressRouteCircuit -Name "DemoCkt" -ResourceGroupName "DemoRG"

$ckt.AllowClassicOperations = $true


1. Get the circuit details.

2. Set "Allow Classic Operations" to TRUE.

3. Update the circuit. After this operation has finished successfully, you will be able to view the circuit in theclassic deployment model.

4. Run the following cmdlet to get the details of the ExpressRoute circuit. You must be able to see the service

To disable ExpressRoute circuit access to the classic deployment modelTo disable ExpressRoute circuit access to the classic deployment model

Next steps

get-azurededicatedcircuit

key listed.

5. You can now manage links to the ExpressRoute circuit using the classic deployment model commands forclassic VNets, and the Resource Manager commands for Resource Manager VNets. The following articleshelp you manage links to the ExpressRoute circuit:

Link your virtual network to your ExpressRoute circuit in the Resource Manager deployment modelLink your virtual network to your ExpressRoute circuit in the classic deployment model

Run the following cmdlets to disable access to the classic deployment model.

$ckt = Get-AzureRmExpressRouteCircuit -Name "DemoCkt" -ResourceGroupName "DemoRG"

$ckt.AllowClassicOperations = $false


1. Get details of the ExpressRoute circuit.

2. Set "Allow Classic Operations" to FALSE.

3. Update the circuit. After this operation has finished successfully, you will not be able to view the circuit inthe classic deployment model.


Migrate ExpressRoute associated virtual networksfrom classic to Resource Manager10/10/2017 • 3 min to read • Edit Online

Before you begin

Supported and unsupported scenarios

Move an ExpressRoute circuit from classic to Resource Manager

This article explains how to migrate Azure ExpressRoute associated virtual networks from the classic deploymentmodel to the Azure Resource Manager deployment model after moving your ExpressRoute circuit.

Verify that you have the latest version of the Azure PowerShell modules. For more information, see How toinstall and configure Azure PowerShell.Make sure that you have reviewed the prerequisites, routing requirements, and workflows before you beginconfiguration.Review the information that is provided under Moving an ExpressRoute circuit from classic to ResourceManager. Make sure that you fully understand the limits and limitations.Verify that the circuit is fully operational in the classic deployment model.Ensure that you have a resource group that was created in the Resource Manager deployment model.Review the following resource-migration documentation:

Platform-supported migration of IaaS resources from classic to Azure Resource ManagerTechnical deep dive on platform-supported migration from classic to Azure Resource ManagerFAQs: Platform-supported migration of IaaS resources from classic to Azure Resource ManagerReview most common migration errors and mitigations

An ExpressRoute circuit can be moved from the classic to the Resource Manager environment without anydowntime. You can move any ExpressRoute circuit from the classic to the Resource Manager environment withno downtime. Follow the instructions in moving ExpressRoute circuits from the classic to the ResourceManager deployment model using PowerShell. This is a prerequisite to move resources connected to thevirtual network.Virtual networks, gateways, and associated deployments within the virtual network that are attached to anExpressRoute circuit in the same subscription can be migrated to the Resource Manager environment withoutany downtime. You can follow the steps described later to migrate resources such as virtual networks,gateways, and virtual machines deployed within the virtual network. You must ensure that the virtual networksare configured correctly before they are migrated.Virtual networks, gateways, and associated deployments within the virtual network that are not in the samesubscription as the ExpressRoute circuit require some downtime to complete the migration. The last section ofthe document describes the steps to be followed to migrate resources.A virtual network with both ExpressRoute Gateway and VPN Gateway can't be migrated.

You must move an ExpressRoute circuit from the classic to the Resource Manager environment before you try tomigrate resources that are attached to the ExpressRoute circuit. To accomplish this task, see the following articles:

Review the information that is provided under Moving an ExpressRoute circuit from classic to ResourceManager.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-migration-classic-resource-manager.md


https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-migration-classic-resource-manager

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-migration-classic-resource-manager-deep-dive


https://docs.microsoft.com/en-us/azure/virtual-machines/windows/migration-classic-resource-manager-errors

Migrate virtual networks, gateways, and associated deployments

Migrate virtual networks, gateways, and associated deployments in the same subscription as the ExpressRouteMigrate virtual networks, gateways, and associated deployments in the same subscription as the ExpressRoutecircuitcircuit

Next steps

Move a circuit from classic to Resource Manager using Azure PowerShell.Use the Azure Service Management portal. You can follow the workflow to create a new ExpressRoute circuitand select the import option.

This operation does not involve downtime. You can continue to transfer data between your premises andMicrosoft while the migration is in progress.

The steps you follow to migrate depend on whether your resources are in the same subscription, differentsubscriptions, or both.

This section describes the steps to be followed to migrate a virtual network, gateway, and associated deploymentsin the same subscription as the ExpressRoute circuit. No downtime is associated with this migration. You cancontinue to use all resources through the migration process. The management plane is locked while the migrationis in progress.

1. Ensure that the ExpressRoute circuit has been moved from the classic to the Resource Manager environment.2. Ensure that the virtual network has been prepared appropriately for the migration.

Select-AzureRmSubscription -SubscriptionName <Your Subscription Name>Register-AzureRmResourceProvider -ProviderNamespace Microsoft.ClassicInfrastructureMigrateGet-AzureRmResourceProvider -ProviderNamespace Microsoft.ClassicInfrastructureMigrate

Move-AzureVirtualNetwork -Validate -VirtualNetworkName $vnetNameMove-AzureVirtualNetwork -Prepare -VirtualNetworkName $vnetNameMove-AzureVirtualNetwork -Commit -VirtualNetworkName $vnetName

Move-AzureVirtualNetwork -Abort $vnetName

3. Register your subscription for resource migration. To register your subscription for resource migration, usethe following PowerShell snippet:

4. Validate, prepare, and migrate. To move the virtual network, use the following PowerShell snippet:

You can also abort migration by running the following PowerShell cmdlet:

Platform-supported migration of IaaS resources from classic to Azure Resource ManagerTechnical deep dive on platform-supported migration from classic to Azure Resource ManagerFAQs: Platform-supported migration of IaaS resources from classic to Azure Resource ManagerReview most common migration errors and mitigations


https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-migration-classic-resource-manager-deep-dive


https://docs.microsoft.com/en-us/azure/virtual-machines/windows/migration-classic-resource-manager-errors

Router configuration samples to set up and managerouting6/27/2017 • 4 min to read • Edit Online

IMPORTANTIMPORTANT

MTU and TCP MSS settings on router interfaces

Cisco IOS-XE based routers

1. Configuring interfaces and sub-interfaces1. Configuring interfaces and sub-interfaces

interface GigabitEthernet<Interface_Number>.<Number> encapsulation dot1Q <VLAN_ID> ip address <IPv4_Address><Subnet_Mask>

This page provides interface and routing configuration samples for Cisco IOS-XE and Juniper MX series routers.These are intended to be samples for guidance only and must not be used as is. You can work with your vendor tocome up with appropriate configurations for your network.

Samples in this page are intended to be purely for guidance. You must work with your vendor's sales / technical team andyour networking team to come up with appropriate configurations to meet your needs. Microsoft will not support issuesrelated to configurations listed in this page. You must contact your device vendor for support issues.

The MTU for the ExpressRoute interface is 1500, which is the typical default MTU for an Ethernet interface on arouter. Unless your router has a different MTU by default, there is no need to specify a value on the routerinterface.Unlike an Azure VPN Gateway, the TCP MSS for an ExpressRoute circuit does not need to be specified.

Router configuration samples below apply to all peerings. Review ExpressRoute peerings and ExpressRouterouting requirements for more details on routing.

The samples in this section apply for any router running the IOS-XE OS family.

You will require a sub interface per peering in every router you connect to Microsoft. A sub interface can beidentified with a VL AN ID or a stacked pair of VL AN IDs and an IP address.

Dot1Q interface definition

This sample provides the sub-interface definition for a sub-interface with a single VL AN ID. The VL AN ID isunique per peering. The last octet of your IPv4 address will always be an odd number.

QinQ interface definition

This sample provides the sub-interface definition for a sub-interface with a two VL AN IDs. The outer VL AN ID (s-tag), if used remains the same across all the peerings. The inner VL AN ID (c-tag) is unique per peering. The lastoctet of your IPv4 address will always be an odd number.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-config-samples-routing.md

interface GigabitEthernet<Interface_Number>.<Number> encapsulation dot1Q <s-tag> seconddot1Q <c-tag> ip address <IPv4_Address><Subnet_Mask>

2. Setting up eBGP sessions2. Setting up eBGP sessions

router bgp <Customer_ASN> bgp log-neighbor-changes neighbor <IP#2_used_by_Azure> remote-as 12076 ! address-family ipv4 neighbor <IP#2_used_by_Azure> activate exit-address-family!

3. Setting up prefixes to be advertised over the BGP session3. Setting up prefixes to be advertised over the BGP session

router bgp <Customer_ASN> bgp log-neighbor-changes neighbor <IP#2_used_by_Azure> remote-as 12076 ! address-family ipv4 network <Prefix_to_be_advertised> mask <Subnet_mask> neighbor <IP#2_used_by_Azure> activate exit-address-family!

4. Route maps4. Route maps

router bgp <Customer_ASN> bgp log-neighbor-changes neighbor <IP#2_used_by_Azure> remote-as 12076 ! address-family ipv4 network <Prefix_to_be_advertised> mask <Subnet_mask> neighbor <IP#2_used_by_Azure> activate neighbor <IP#2_used_by_Azure> route-map <MS_Prefixes_Inbound> in exit-address-family!route-map <MS_Prefixes_Inbound> permit 10 match ip address prefix-list <MS_Prefixes>!

Juniper MX series routers

1. Configuring interfaces and sub-interfaces1. Configuring interfaces and sub-interfaces

You must setup a BGP session with Microsoft for every peering. The sample below enables you to setup a BGPsession with Microsoft. If the IPv4 address you used for your sub interface was a.b.c.d, the IP address of the BGPneighbor (Microsoft) will be a.b.c.d+1. The last octet of the BGP neighbor's IPv4 address will always be an evennumber.

You can configure your router to advertise select prefixes to Microsoft. You can do so using the sample below.

You can use route-maps and prefix lists to filter prefixes propagated into your network. You can use the samplebelow to accomplish the task. Ensure that you have appropriate prefix lists setup.

The samples in this section apply for any Juniper MX series routers.

Dot1Q interface definition

interfaces { vlan-tagging; <Interface_Number> { unit <Number> { vlan-id <VLAN_ID>; family inet { address <IPv4_Address/Subnet_Mask>; } } }}

interfaces { <Interface_Number> { flexible-vlan-tagging; unit <Number> { vlan-tags outer <S-tag> inner <C-tag>; family inet { address <IPv4_Address/Subnet_Mask>; } } } }

2. Setting up eBGP sessions2. Setting up eBGP sessions

routing-options { autonomous-system <Customer_ASN>;}}protocols { bgp { group <Group_Name> { peer-as 12076; neighbor <IP#2_used_by_Azure>; } } }

3. Setting up prefixes to be advertised over the BGP session3. Setting up prefixes to be advertised over the BGP session

This sample provides the sub-interface definition for a sub-interface with a single VL AN ID. The VL AN ID isunique per peering. The last octet of your IPv4 address will always be an odd number.

QinQ interface definition

This sample provides the sub-interface definition for a sub-interface with a two VL AN IDs. The outer VL AN ID (s-tag), if used remains the same across all the peerings. The inner VL AN ID (c-tag) is unique per peering. The lastoctet of your IPv4 address will always be an odd number.

You must setup a BGP session with Microsoft for every peering. The sample below enables you to setup a BGPsession with Microsoft. If the IPv4 address you used for your sub interface was a.b.c.d, the IP address of the BGPneighbor (Microsoft) will be a.b.c.d+1. The last octet of the BGP neighbor's IPv4 address will always be an evennumber.

You can configure your router to advertise select prefixes to Microsoft. You can do so using the sample below.

policy-options { policy-statement <Policy_Name> { term 1 { from protocol OSPF; route-filter <Prefix_to_be_advertised/Subnet_Mask> exact; then { accept; } } }}protocols { bgp { group <Group_Name> { export <Policy_Name> peer-as 12076; neighbor <IP#2_used_by_Azure>; } } }

4. Route maps4. Route maps

policy-options { prefix-list MS_Prefixes { <IP_Prefix_1/Subnet_Mask>; <IP_Prefix_2/Subnet_Mask>; } policy-statement <MS_Prefixes_Inbound> { term 1 { from { prefix-list MS_Prefixes; } then { accept; } } }}protocols { bgp { group <Group_Name> { export <Policy_Name> import <MS_Prefixes_Inbound> peer-as 12076; neighbor <IP#2_used_by_Azure>; } } }

Next Steps

You can use route-maps and prefix lists to filter prefixes propagated into your network. You can use the samplebelow to accomplish the task. Ensure that you have appropriate prefix lists setup.

See the ExpressRoute FAQ for more details.

Router configuration samples to set up and manageNAT6/27/2017 • 4 min to read • Edit Online

IMPORTANTIMPORTANT

Cisco ASA firewallsPAT configuration for traffic from customer network to MicrosoftPAT configuration for traffic from customer network to Microsoft

object network MSFT-PAT range <SNAT-START-IP> <SNAT-END-IP>

object-group network MSFT-Range network-object <IP> <Subnet_Mask>

object-group network on-prem-range-1 network-object <IP> <Subnet-Mask>

object-group network on-prem-range-2 network-object <IP> <Subnet-Mask>

object-group network on-prem network-object object on-prem-range-1 network-object object on-prem-range-2

nat (outside,inside) source dynamic on-prem pat-pool MSFT-PAT destination static MSFT-Range MSFT-Range

PAT configuration for traffic from Microsoft to customer networkPAT configuration for traffic from Microsoft to customer network

Source Interface (where the traffic enters the ASA): insideDestination Interface (where the traffic exits the ASA): outside

This page provides NAT configuration samples for Cisco ASA and Juniper SRX series routers. These are intendedto be samples for guidance only and must not be used as is. You can work with your vendor to come up withappropriate configurations for your network.

Samples in this page are intended to be purely for guidance. You must work with your vendor's sales / technical team andyour networking team to come up with appropriate configurations to meet your needs. Microsoft will not support issuesrelated to configurations listed in this page. You must contact your device vendor for support issues.

Router configuration samples below apply to Azure Public and Microsoft peerings. You must not configureNAT for Azure private peering. Review ExpressRoute peerings and ExpressRoute NAT requirements formore details.

You MUST use separate NAT IP pools for connectivity to the internet and ExpressRoute. Using the sameNAT IP pool across the internet and ExpressRoute will result in asymmetric routing and loss of connectivity.

Interfaces and Direction:

Configuration:

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-config-samples-nat.md

object network outbound-PAT host <NAT-IP>

object network Customer-Network network-object <IP> <Subnet-Mask>

object-group network MSFT-Network-1 network-object <MSFT-IP> <Subnet-Mask>

object-group network MSFT-PAT-Networks network-object object MSFT-Network-1

nat (inside,outside) source dynamic MSFT-PAT-Networks pat-pool outbound-PAT destination static Customer-Network Customer-Network

Juniper SRX series routers1. Create redundant Ethernet interfaces for the cluster1. Create redundant Ethernet interfaces for the cluster

interfaces { reth0 { description "To Internal Network"; vlan-tagging; redundant-ether-options { redundancy-group 1; } unit 100 { vlan-id 100; family inet { address <IP-Address/Subnet-mask>; } } } reth1 { description "To Microsoft via Edge Router"; vlan-tagging; redundant-ether-options { redundancy-group 2; } unit 100 { description "To Microsoft via Edge Router"; vlan-id 100; family inet { address <IP-Address/Subnet-mask>; } } }}

2. Create two security zones2. Create two security zones

NAT Pool:

Target Server :

Object Group for Customer IP Addresses

NAT Commands:

3. Create security policies between zones3. Create security policies between zones

security { policies { from-zone Trust to-zone Untrust { policy allow-any { match { source-address any; destination-address any; application any; } then { permit; } } } from-zone Untrust to-zone Trust { policy allow-any { match { source-address any; destination-address any; application any; } then { permit; } } } }}

4. Configure NAT policies4. Configure NAT policies

Trust Zone for internal network and Untrust Zone for external network facing Edge RoutersAssign appropriate interfaces to the zonesAllow services on the interfaces

security { zones { security-zone Trust { host-inbound-traffic { system-services { ping; } protocols { bgp; } }interfaces { reth0.100; } } security-zone Untrust { host-inbound-traffic { system-services { ping; } protocols {bgp; } } interfaces { reth1.100; } } } }

Create two NAT pools. One will be used to NAT traffic outbound to Microsoft and other from Microsoft to thecustomer.Create rules to NAT the respective traffic

5. Configure BGP to advertise selective prefixes in each direction5. Configure BGP to advertise selective prefixes in each direction

6. Create policies6. Create policies

routing-options { autonomous-system <Customer-ASN>;}policy-options { prefix-list Microsoft-Prefixes { <IP-Address/Subnet-Mask;

security { nat { source { pool SNAT-To-ExpressRoute { routing-instance { External-ExpressRoute; } address { <NAT-IP-address/Subnet-mask>; } } pool SNAT-From-ExpressRoute { routing-instance { Internal; } address { <NAT-IP-address/Subnet-mask>; } } rule-set Outbound_NAT { from routing-instance Internal; to routing-instance External-ExpressRoute; rule SNAT-Out { match { source-address 0.0.0.0/0; } then { source-nat { pool { SNAT-To-ExpressRoute; } } } } } rule-set Inbound-NAT { from routing-instance External-ExpressRoute; to routing-instance Internal; rule SNAT-In { match { source-address 0.0.0.0/0; } then { source-nat { pool { SNAT-From-ExpressRoute; } } } } } } } }

Refer to samples in Routing configuration samples page.

<IP-Address/Subnet-Mask; <IP-Address/Subnet-Mask; } prefix-list private-ranges { 10.0.0.0/8; 172.16.0.0/12; 192.168.0.0/16; 100.64.0.0/10; } policy-statement Advertise-NAT-Pools { from { protocol static; route-filter <NAT-Pool-Address/Subnet-mask> prefix-length-range /32-/32; } then accept; } policy-statement Accept-from-Microsoft { term 1 { from { instance External-ExpressRoute; prefix-list-filter Microsoft-Prefixes orlonger; } then accept; } term deny { then reject; } } policy-statement Accept-from-Internal { term no-private { from { instance Internal; prefix-list-filter private-ranges orlonger; } then reject; } term bgp { from { instance Internal; protocol bgp; } then accept; } term deny { then reject; } }}routing-instances { Internal { instance-type virtual-router; interface reth0.100; routing-options { static { route <NAT-Pool-IP-Address/Subnet-mask> discard; } instance-import Accept-from-Microsoft; } protocols { bgp { group customer { export <Advertise-NAT-Pools>; peer-as <Customer-ASN-1>; neighbor <BGP-Neighbor-IP-Address>; } } } } External-ExpressRoute { instance-type virtual-router;

instance-type virtual-router; interface reth1.100; routing-options { static { route <NAT-Pool-IP-Address/Subnet-mask> discard; } instance-import Accept-from-Internal; } protocols { bgp { group edge-router { export <Advertise-NAT-Pools>; peer-as <Customer-Public-ASN>; neighbor <BGP-Neighbor-IP-Address>; } } } }}

Next stepsSee the ExpressRoute FAQ for more details.

Configure Network Performance Monitor forExpressRoute4/9/2018 • 10 min to read • Edit Online

Supported regions

NOTENOTE

Workflow

Network Performance Monitor (NPM) is a cloud-based network monitoring solution that monitors connectivitybetween Azure cloud deployments and on-premises locations (Branch offices, etc.). NPM is part of Log Analytics.NPM now offers an extension for ExpressRoute that lets you monitor network performance over ExpressRoutecircuits that are configured to use Private Peering. When you configure NPM for ExpressRoute, you can detectnetwork issues to identify and eliminate.

You can:

Monitor loss and latency across various VNets and set alerts

Monitor all paths (including redundant paths) on the network

Troubleshoot transient and point-in-time network issues that are difficult to replicate

Help determine a specific segment on the network that is responsible for degraded performance

Get throughput per virtual network (If you have agents installed in each VNet)

See the ExpressRoute system state from a previous point in time

You can monitor ExpressRoute circuits in any part of the world by using a workspace that is hosted in one of thefollowing regions:

West EuropeWest Central USEast USSouth East AsiaSouth East Australia

Support for monitoring of ExpressRoute circuits connected to VNETs in Azure Government cloud is planned for Q2 2018.

Monitoring agents are installed on multiple servers, both on-premises and in Azure. The agents communicate witheach other, but do not send data, they send TCP handshake packets. The communication between the agents allowsAzure to map the network topology and path the traffic could take.

1. Create an NPM Workspace in the one of the supported regions.2. Install and configure software agents:

Install monitoring agents on the on-premises servers and the Azure VMs.Configure settings on the monitoring agent servers to allow the monitoring agents to communicate.(Open firewall ports, etc.)

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/how-to-npm.md

Step 1: Create a Workspace

3. Configure network security group (NSG) rules to allow the monitoring agent installed on Azure VMs tocommunicate with on-premises monitoring agents.

4. Set up monitoring: Auto-Discover and manage which networks are visible in NPM.

If you are already using Network Performance Monitor to monitor other objects or services, and you already haveWorkspace in one of the supported regions, you can skip Step 1 and Step 2, and begin your configuration withStep 3.

Create a workspace in the subscription that has the VNets link to the ExpressRoute circuit(s).

NOTENOTE

2. At the bottom of the main Network Performance Monitor page, click Create to open NetworkPerformance Monitor - Create new solution page. Click OMS Workspace - select a workspace to openthe Workspaces page. Click + Create New Workspace to open the Workspace page.

1. In the Azure portal, select the Subscription that has the VNETs peered to your ExpressRoute circuit. Thensearch the list of services in the Marketplace for 'Network Performance Monitor'. In the return, click toopen the Network Performance Monitor page.

You may create a new workspace or use an existing workspace. If you wish to use an existing workspace, you mustensure that the workspace has been migrated to the new query language. More information...

3. On the OMS Workspace page, select Create New and configure the following settings:

OMS Workspace - Type a name for your Workspace.Subscription - If you have multiple subscriptions, choose the one you want to associate with the newWorkspace.


https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-log-search-upgrade

Step 2: Install and configure agents2.1: Download the agent setup file2.1: Download the agent setup file

4. Click OK to save and deploy the settings template. Once the template validates, click Create to deploy theWorkspace.

Resource group - Create a resource group, or use an existing one.Location - You must select a supported region.

NOTENOTE

Pricing tier - Select 'Free'

The ExpressRoute circuit could be anywhere in the world and does not have to be in the same region as theWorkspace.

5. After the Workspace has been deployed, navigate to the NetworkMonitoring(name) resource that youcreated. Validate the settings, then click Solution requires additional configuration.

2.2: Install a monitoring agent on each monitoring server (on each VNET that you want to monitor)2.2: Install a monitoring agent on each monitoring server (on each VNET that you want to monitor)

NOTENOTE

NOTENOTE

1. Go to the Common Settings tab of the Network Performance Monitor Configuration page for yourresource. Click the agent that corresponds to your server's processor from the Install OMS Agents sectionand download the setup file.

2. Next, copy the Workspace ID and Primary Key to Notepad.

3. From the Configure OMS Agents for monitoring using TCP protocol section, download the PowershellScript. The PowerShell script helps you open the relevant firewall port for the TCP transactions.

We recommend that you install at least two agents on each side of the ExpressRoute connection (i.e., on-premises,Azure VNETs) for redundancy. Use the following steps to install agents:

The agent must be installed on a Windows Server (2008 SP1 or later). Monitoring of ExpressRoute circuits using WindowsDesktop OS and Linux OS is not supported.

Agents pushed by SCOM (includes MMA) may not be able to consistently detect their location, if they are hosted in Azure.We recommend that you do not use these agents in Azure VNETs to monitor ExpressRoute.

1. Run Setup to install the agent on each server that you want to use for monitoring ExpressRoute. The serveryou use for monitoring can either be a VM, or on-premises and must have Internet access. You need to install atleast one agent on-premises, and one agent on each network segment that you want to monitor in Azure.

2. On the Welcome page, click Next.3. On the License Terms page, read the license, and then click I Agree.4. On the Destination Folder page, change or keep the default installation folder, and then click Next.5. On the Agent Setup Options page, you can choose to connect the agent to Azure Log Analytics or

https://technet.microsoft.com/en-us/library/dn465154(v=sc.12).aspx

6. On the Ready to Install page, review your choices, and then click Install.7. On the Configuration completed successfully page, click Finish.

Operations Manager. Or, you can leave the choices blank if you want to configure the agent later. Aftermaking your selection(s), click Next.

If you chose to connect to Azure Log Analytics, paste the Workspace ID and Workspace Key(Primary Key) that you copied into Notepad in the previous section. Then, click Next.

If you chose to connect to Operations Manager, on the Management Group Configurationpage, type the Management Group Name, Management Server, and the Management ServerPort. Then, click Next.

On the Agent Action Account page, choose either the Local System account, or Domain or LocalComputer Account. Then, click Next.

8. When complete, the Microsoft Monitoring Agent appears in the Control Panel. You can review yourconfiguration there, and verify that the agent is connected to Azure Log Analytics (OMS). When connected,

2.3: Configure proxy settings (optional)2.3: Configure proxy settings (optional)

2.4: Verify agent connectivity2.4: Verify agent connectivity

the agent displays a message stating: The Microsoft Monitoring Agent has successfully connected tothe Microsoft Operations Management Suite service.

9. Please repeat this for each VNET that you need to be monitored.

If you are using a web proxy to access the Internet, use the following steps to configure proxy settings for theMicrosoft Monitoring Agent. Perform these steps for each server. If you have many servers that you need toconfigure, you might find it easier to use a script to automate this process. If so, see To configure proxy settings forthe Microsoft Monitoring Agent using a script.

To configure proxy settings for the Microsoft Monitoring Agent using the Control Panel:

1. Open the Control Panel.2. Open Microsoft Monitoring Agent.3. Click the Proxy Settings tab.4. Select Use a proxy server and type the URL and port number, if one is needed. If your proxy server

requires authentication, type the username and password to access the proxy server.

You can easily verify whether your agents are communicating.

1. On a server with the monitoring agent, open the Control Panel.2. Open the Microsoft Monitoring Agent.3. Click the Azure Log Analytics tab.4. In the Status column, you should see that the agent connected successfully to Log Analytics.

https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-windows-agent

2.5: Open the firewall ports on the monitoring agent servers2.5: Open the firewall ports on the monitoring agent servers

NOTENOTE

Step 3: Configure network security group rules

To use the TCP protocol, you must open firewall ports to ensure that the monitoring agents can communicate.

You can run a PowerShell script that creates the registry keys required by the Network Performance Monitor, aswell as creating the Windows Firewall rules to allow monitoring agents to create TCP connections with each other.The registry keys created by the script also specify whether to log the debug logs, and the path for the logs file. Italso defines the agent TCP port used for communication. The values for these keys are automatically set by thescript, so you should not manually change these keys.

Port 8084 is opened by default. You can use a custom port by providing the parameter 'portNumber' to the script.However, if you do so, you must specify the same port for all the servers on which you run the script.

The 'EnableRules' PowerShell script configures Windows Firewall rules only on the server where the script is run. If you have anetwork firewall, you should make sure that it allows traffic destined for the TCP port being used by Network PerformanceMonitor.

On the agent servers, open a PowerShell window with administrative privileges. Run the EnableRules PowerShellscript (which you downloaded earlier). Don't use any parameters.

For monitoring agent servers that are in Azure, you must configure network security group (NSG) rules to allowTCP traffic on a port used by NPM for synthetic transactions. The default port is 8084. This allows a monitoringagent installed on Azure VM to communicate with an on-premises monitoring agent.

For more information about NSG, see Network Security Groups.

https://aka.ms/npmpowershellscript

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-create-nsg-arm-portal

NOTENOTE

Step 4: Configure NPM for ExpressRoute monitoring

Make sure that you have installed the agents (both the on-premises server agent and the Azure server agent) and have runthe PowerShell script before proceeding with this step.

After you complete the previous sections, you can set up monitoring.

1. Navigate to the Network Performance Monitor overview tile by going to the All Resources page, andclicking on the whitelisted NPM Workspace.

2. Click the Network Performance Monitor overview tile to bring up the dashboard. The dashboardcontains an ExpressRoute page, which shows that ExpressRoute is in an 'unconfigured state'. Click FeatureSetup to open the Network Performance Monitor configuration page.

3. On the configuration page, navigate to the 'ExpressRoute Peerings' tab, located on the left side panel. ClickDiscover Now.

Step 5: View monitoring tilesNetwork Performance Monitor pageNetwork Performance Monitor page

4. When discovery completes, you see rules for unique Circuit name and VNet name. Initially, these rules aredisabled. Enable the rules, then select the monitoring agents and threshold values.

5. After enabling the rules and selecting the values and agents you want to monitor, there is a wait ofapproximately 30-60 minutes for the values to begin populating and the ExpressRoute Monitoring tiles tobecome available. Once you see the monitoring tiles, your ExpressRoute circuits and connection resourcesare being monitored by NPM.

The NPM page contains a page for ExpressRoute that shows an overview of the health of ExpressRoute circuitsand peerings.

List of circuitsList of circuits

Trend of Loss, Latency and ThroughputTrend of Loss, Latency and Throughput

To see a list of all monitored ExpressRoute circuits, click on the ExpressRoute circuits tile. You can select a circuitand view its health state, trend charts for packet loss, bandwidth utilization, and latency. The charts are interactive.You can select a custom time window for plotting the charts. You can drag the mouse over an area on the chart tozoom in and see fine-grained data points.

The bandwidth, latency, and loss charts are interactive. You can zoom into any section of these charts, using mousecontrols. You can also see the bandwidth, latency, and loss data for other intervals by clicking Date/Time, locatedbelow the Actions button on the upper left.

Peerings listPeerings list

Circuit topologyCircuit topology

Clicking on the Private Peerings tile on the dashboard brings up a list of all connections to virtual networks overprivate peering. Here, you can select a virtual network connection and view its health state, trend charts for packetloss, bandwidth utilization, and latency.

To view circuit topology, click on the Topology tile. This takes you to the topology view of the selected circuit orpeering. The topology diagram provides the latency for each segment on the network and each layer 3 hop isrepresented by a node of the diagram. Clicking on a hop reveals more details about the hop. You can increase thelevel of visibility to include on-premises hops by moving the slider bar below Filters. Moving the slider bar to theleft or right, increases/decreases the number of hops in the topology graph. The latency across each segment isvisible, which allows for faster isolation of high latency segments on your network.

Detailed Topology view of a circuitDetailed Topology view of a circuit

This view shows VNet connections.

Modify an ExpressRoute circuit using PowerShell(classic)11/9/2017 • 5 min to read • Edit Online

IMPORTANTIMPORTANT

Before you begin

Get the status of a circuit

This article also shows you how to check the status, update, or delete and deprovision an ExpressRoute circuit.

As of March 1, 2017, you can't create new ExpressRoute circuits in the classic deployment model.

You can move an existing ExpressRoute circuit from the classic deployment model to the Resource Managerdeployment model without experiencing any connectivity down time. For more information, see Move an existingcircuit.You can connect to virtual networks in the classic deployment model by setting allowClassicOperations to TRUE.

Use the following links to create and manage ExpressRoute circuits in the Resource Manager deployment model:


About Azure deployment models

Azure currently works with two deployment models: Resource Manager and classic. The two models are notcompletely compatible with each other. Before you begin, you need to know which model that you want to workin. For information about the deployment models, see Understanding deployment models. If you are new toAzure, we recommend that you use the Resource Manager deployment model.

Install the latest versions of the Azure Service Management (SM) PowerShell modules Follow the instructionsin Getting started with Azure PowerShell cmdlets for step-by-step guidance on how to configure your computerto use the Azure PowerShell modules.

You can retrieve this information at any time by using the Get-AzureCircuit cmdlet. Making the call without anyparameters lists all the circuits.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-circuit-classic.md

https://docs.microsoft.com/en-us/azure/resource-manager-deployment-model



Bandwidth : 200CircuitName : MyTestCircuitLocation : Silicon ValleyServiceKey : *********************************ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

Bandwidth : 1000CircuitName : MyAsiaCircuitLocation : SingaporeServiceKey : #################################ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

Get-AzureDedicatedCircuit -ServiceKey "*********************************"

Bandwidth : 200CircuitName : MyTestCircuitLocation : Silicon ValleyServiceKey : *********************************ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

get-help get-azurededicatedcircuit -detailed

Modify a circuit

Enable the ExpressRoute premium add-onEnable the ExpressRoute premium add-on

You can get information on a specific ExpressRoute circuit by passing the service key as a parameter to the call.

You can get detailed descriptions of all the parameters by running the following example:

You can modify certain properties of an ExpressRoute circuit without impacting connectivity.

You can do the following tasks with no downtime:

Enable or disable an ExpressRoute premium add-on for your ExpressRoute circuit.Increase the bandwidth of your ExpressRoute circuit provided there is capacity available on the port.Downgrading the bandwidth of a circuit is not supported.Change the metering plan from Metered Data to Unlimited Data. Changing the metering plan fromUnlimited Data to Metered Data is not supported.You can enable and disable Allow Classic Operations.

Refer to the ExpressRoute FAQ for more information on limits and limitations.

You can enable the ExpressRoute premium add-on for your existing circuit by using the following PowerShellcmdlet:

Set-AzureDedicatedCircuitProperties -ServiceKey "*********************************" -Sku Premium

Bandwidth : 1000CircuitName : TestCircuitLocation : Silicon ValleyServiceKey : *********************************ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : PremiumStatus : Enabled

Disable the ExpressRoute premium add-onDisable the ExpressRoute premium add-on

IMPORTANTIMPORTANT

ConsiderationsConsiderations

To disable the premium add-onTo disable the premium add-on

Set-AzureDedicatedCircuitProperties -ServiceKey "*********************************" -Sku Standard

Bandwidth : 1000CircuitName : TestCircuitLocation : Silicon ValleyServiceKey : *********************************ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

Update the ExpressRoute circuit bandwidthUpdate the ExpressRoute circuit bandwidth

Your circuit will now have the ExpressRoute premium add-on features enabled. As soon as the command hasbeen successfully run, billing for the premium add-on capability begins.

This operation can fail if you're using resources that are greater than what is permitted for the standard circuit.

Make sure that the number of virtual networks linked to the circuit is less than 10 before you downgradefrom premium to standard. If you don't do this, your update request fails, and you are billed the premiumrates.You must unlink all virtual networks in other geopolitical regions. If you don't, your update request fails, andyou are billed the premium rates.Your route table must be less than 4,000 routes for private peering. If your route table size is greater than4,000 routes, the BGP session drops and won't be reenabled until the number of advertised prefixes goesbelow 4,000.

You can disable the ExpressRoute premium add-on for your existing circuit by using the following PowerShellcmdlet:

Check the ExpressRoute FAQ for supported bandwidth options for your provider. You can pick any size that isgreater than the size of your existing circuit as long as the physical port (on which your circuit is created) allows.

IMPORTANTIMPORTANT

Resize a circuitResize a circuit

Set-AzureDedicatedCircuitProperties -ServiceKey ********************************* -Bandwidth 1000

Bandwidth : 1000CircuitName : TestCircuitLocation : Silicon ValleyServiceKey : *********************************ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

Set-AzureDedicatedCircuitProperties : InvalidOperation : Insufficient bandwidth available to perform this circuitupdate operationAt line:1 char:1+ Set-AzureDedicatedCircuitProperties -ServiceKey ********************* ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Set-AzureDedicatedCircuitProperties], CloudException + FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.ExpressRoute.SetAzureDedicatedCircuitPropertiesCommand

Deprovision and delete a circuitConsiderationsConsiderations

Delete a circuitDelete a circuit

Remove-AzureDedicatedCircuit -ServiceKey "*********************************"


You cannot reduce the bandwidth of an ExpressRoute circuit without disruption. Downgrading bandwidth requires you todeprovision the ExpressRoute circuit and then reprovision a new ExpressRoute circuit.

After you decide what size you need, you can use the following command to resize your circuit:

Once your circuit has been sized up on the Microsoft side, you must contact your connectivity provider toupdate configurations on their side to match this change. Billing begins for the updated bandwidth option fromthis point on.

If you see the following error when increasing the circuit bandwidth, it means there is no sufficient bandwidthleft on the physical port where your existing circuit is created. You must delete this circuit and create a newcircuit of the size you need.

You must unlink all virtual networks from the ExpressRoute circuit for this operation to succeed. Check to seeif you have any virtual networks that are linked to the circuit if this operation fails.If the ExpressRoute circuit service provider provisioning state is Provisioning or Provisioned you mustwork with your service provider to deprovision the circuit on their side. We continue to reserve resourcesand bill you until the service provider completes deprovisioning the circuit and notifies us.If the service provider has deprovisioned the circuit (the service provider provisioning state is set to Notprovisioned), you can then delete the circuit. This stops billing for the circuit.


Create and modify peering for an ExpressRoutecircuit (classic)2/27/2018 • 10 min to read • Edit Online

IMPORTANTIMPORTANT


IMPORTANTIMPORTANT

Log in to your Azure account and select a subscriptionLog in to your Azure account and select a subscription

This article walks you through the steps to create and manage routing configuration for an ExpressRoute circuitusing PowerShell and the classic deployment model. The steps below will also show you how to check the status,update, or delete and deprovision peerings for an ExpressRoute circuit.


You can move an existing ExpressRoute circuit from the classic deployment model to the Resource Manager deploymentmodel without experiencing any connectivity down time. For more information, see Move an existing circuit.You can connect to virtual networks in the classic deployment model by setting allowClassicOperations to TRUE.





You will need the latest version of the Azure Service Management (SM) PowerShell cmdlets. For moreinformation, see Getting started with Azure PowerShell cmdlets.Make sure that you have reviewed the prerequisites page, the routing requirements page, and the workflowspage before you begin configuration.You must have an active ExpressRoute circuit. Follow the instructions to create an ExpressRoute circuit andhave the circuit enabled by your connectivity provider before you proceed. The ExpressRoute circuit must bein a provisioned and enabled state for you to be able to run the cmdlets described below.

These instructions only apply to circuits created with service providers offering Layer 2 connectivity services. If you areusing a service provider offering managed Layer 3 services (typically an IPVPN, like MPLS), your connectivity provider willconfigure and manage routing for you.

You can configure one, two, or all three peerings (Azure private, Azure public and Microsoft) for an ExpressRoutecircuit. You can configure peerings in any order you choose. However, you must make sure that you complete theconfiguration of each peering one at a time.

1. Open your PowerShell console with elevated rights and connect to your account. Use the following

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-routing-classic.md








Add-AzureAccount

example to help you connect:

2. Check the subscriptions for the account.

3. If you have more than one subscription, select the subscription that you want to use.

4. Next, use the following cmdlet to add your Azure subscription to PowerShell for the classic deploymentmodel.

This section provides instructions on how to create, get, update, and delete the Azure private peeringconfiguration for an ExpressRoute circuit.

Import-Module 'C:\Program Files\WindowsPowerShell\Modules\Azure\5.1.1\Azure\Azure.psd1' Import-Module 'C:\Program Files\WindowsPowerShell\Modules\Azure\5.1.1\ExpressRoute\ExpressRoute.psd1'


You must import the Azure and ExpressRoute modules into the PowerShell session in order to start usingthe ExpressRoute cmdlets. Run the following commands to import the Azure and ExpressRoute modulesinto the PowerShell session. The version may vary.


Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. If your connectivity provider offers managed Layer 3 services, you can request your connectivityprovider to enable Azure private peering for you. In that case, you won't need to follow instructions listedin the next sections. However, if your connectivity provider does not manage routing for you, after creatingyour circuit, follow the instructions below.

3. Check the ExpressRoute circuit to ensure it is provisioned.

You must first check to see if the ExpressRoute circuit is Provisioned and also Enabled. See the examplebelow.


PS C:\> Get-AzureDedicatedCircuit -ServiceKey "*********************************"

Bandwidth : 200 CircuitName : MyTestCircuit Location : Silicon Valley ServiceKey : ********************************* ServiceProviderName : equinix ServiceProviderProvisioningState : Provisioned Sku : Standard Status : Enabled

ServiceProviderProvisioningState : Provisioned Status : Enabled

Make sure that the circuit shows as Provisioned and Enabled. If it doesn't, work with your connectivityprovider to get your circuit to the required state and status.

4. Configure Azure private peering for the circuit.

Make sure that you have the following items before you proceed with the next steps:

A /30 subnet for the primary link. This must not be part of any address space reserved for virtualnetworks.A /30 subnet for the secondary link. This must not be part of any address space reserved for virtualnetworks.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers. You can use a private ASnumber for this peering. Ensure that you are not using 65515.

IMPORTANTIMPORTANT

An MD5 hash if you choose to use one. This is optional.

You can run the following cmdlet to configure Azure private peering for your circuit.

New-AzureBGPPeering -AccessType Private -ServiceKey "*********************************" -PrimaryPeerSubnet "10.0.0.0/30" -SecondaryPeerSubnet "10.0.0.4/30" -PeerAsn 1234 -VlanId 100

You can use the cmdlet below if you choose to use an MD5 hash.

New-AzureBGPPeering -AccessType Private -ServiceKey "*********************************" -PrimaryPeerSubnet "10.0.0.0/30" -SecondaryPeerSubnet "10.0.0.4/30" -PeerAsn 1234 -VlanId 100-SharedKey "A1B2C3D4"


You can get configuration details using the following cmdlet

Get-AzureBGPPeering -AccessType Private -ServiceKey "*********************************"

AdvertisedPublicPrefixes : AdvertisedPublicPrefixesState : ConfiguredAzureAsn : 12076CustomerAutonomousSystemNumber : PeerAsn : 1234PrimaryAzurePort : PrimaryPeerSubnet : 10.0.0.0/30RoutingRegistryName : SecondaryAzurePort : SecondaryPeerSubnet : 10.0.0.4/30State : EnabledVlanId : 100


Set-AzureBGPPeering -AccessType Private -ServiceKey "*********************************" -PrimaryPeerSubnet "10.0.0.0/30" -SecondaryPeerSubnet "10.0.0.4/30" -PeerAsn 1234 -VlanId 500 -SharedKey "A1B2C3D4"


WARNINGWARNING

Remove-AzureBGPPeering -AccessType Private -ServiceKey "*********************************"



You can update any part of the configuration using the following cmdlet. In the example below, the VL AN ID ofthe circuit is being updated from 100 to 500.

You can remove your peering configuration by running the following cmdlet.

You must ensure that all virtual networks are unlinked from the ExpressRoute circuit before running this cmdlet.

This section provides instructions on how to create, get, update and delete the Azure public peeringconfiguration for an ExpressRoute circuit.




2. Create an ExpressRoute circuit

Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. If your connectivity provider offers managed Layer 3 services, you can request your connectivityprovider to enable Azure public peering for you. In that case, you won't need to follow instructions listedin the next sections. However, if your connectivity provider does not manage routing for you, after creatingyour circuit, follow the instructions below.





3. Check ExpressRoute circuit to ensure it is provisioned

You must first check to see if the ExpressRoute circuit is Provisioned and also Enabled. See the examplebelow.


4. Configure Azure public peering for the circuit

Ensure that you have the following information before you proceed further.

A /30 subnet for the primary link. This must be a valid public IPv4 prefix.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.

IMPORTANTIMPORTANT

An MD5 hash if you choose to use one. This is optional.

You can run the following cmdlet to configure Azure public peering for your circuit

New-AzureBGPPeering -AccessType Public -ServiceKey "*********************************" -PrimaryPeerSubnet "131.107.0.0/30" -SecondaryPeerSubnet "131.107.0.4/30" -PeerAsn 1234 -VlanId 200

You can use the cmdlet below if you choose to use an MD5 hash

New-AzureBGPPeering -AccessType Public -ServiceKey "*********************************" -PrimaryPeerSubnet "131.107.0.0/30" -SecondaryPeerSubnet "131.107.0.4/30" -PeerAsn 1234 -VlanId 200 -SharedKey "A1B2C3D4"

Ensure that you specify your AS number as peering ASN and not customer ASN.

You can get configuration details using the following cmdlet

Get-AzureBGPPeering -AccessType Public -ServiceKey "*********************************"

AdvertisedPublicPrefixes : AdvertisedPublicPrefixesState : ConfiguredAzureAsn : 12076CustomerAutonomousSystemNumber : PeerAsn : 1234PrimaryAzurePort : PrimaryPeerSubnet : 131.107.0.0/30RoutingRegistryName : SecondaryAzurePort : SecondaryPeerSubnet : 131.107.0.4/30State : EnabledVlanId : 200


Set-AzureBGPPeering -AccessType Public -ServiceKey "*********************************" -PrimaryPeerSubnet "131.107.0.0/30" -SecondaryPeerSubnet "131.107.0.4/30" -PeerAsn 1234 -VlanId 600 -SharedKey "A1B2C3D4"


Remove-AzureBGPPeering -AccessType Public -ServiceKey "*********************************"

Microsoft peering


You can update any part of the configuration using the following cmdlet

The VL AN ID of the circuit is being updated from 200 to 600 in the above example.

You can remove your peering configuration by running the following cmdlet

This section provides instructions on how to create, get, update and delete the Microsoft peering configurationfor an ExpressRoute circuit.




2. Create an ExpressRoute circuit

Follow the instructions to create an ExpressRoute circuit and have it provisioned by the connectivityprovider. If your connectivity provider offers managed Layer 3 services, you can request your connectivityprovider to enable Azure private peering for you. In that case, you won't need to follow instructions listedin the next sections. However, if your connectivity provider does not manage routing for you, after creatingyour circuit, follow the instructions below.

3. Check ExpressRoute circuit to ensure it is provisioned

You must first check to see if the ExpressRoute circuit is in Provisioned and Enabled state.






4. Configure Microsoft peering for the circuit

Make sure that you have the following information before you proceed.

A /30 subnet for the primary link. This must be a valid public IPv4 prefix owned by you and registeredin an RIR / IRR.A /30 subnet for the secondary link. This must be a valid public IPv4 prefix owned by you andregistered in an RIR / IRR.A valid VL AN ID to establish this peering on. Ensure that no other peering in the circuit uses the sameVL AN ID.AS number for peering. You can use both 2-byte and 4-byte AS numbers.Advertised prefixes: You must provide a list of all prefixes you plan to advertise over the BGP session.Only public IP address prefixes are accepted. You can send a comma separated list if you plan to send aset of prefixes. These prefixes must be registered to you in an RIR / IRR.Customer ASN: If you are advertising prefixes that are not registered to the peering AS number, youcan specify the AS number to which they are registered. This is optional.Routing Registry Name: You can specify the RIR / IRR against which the AS number and prefixes areregistered.An MD5 hash, if you choose to use one. This is optional.

You can run the following cmdlet to configure Microsoft pering for your circuit

New-AzureBGPPeering -AccessType Microsoft -ServiceKey "*********************************" -PrimaryPeerSubnet "131.107.0.0/30" -SecondaryPeerSubnet "131.107.0.4/30" -VlanId 300 -PeerAsn 1234 -CustomerAsn 2245 -AdvertisedPublicPrefixes "123.0.0.0/30" -RoutingRegistryName "ARIN" -SharedKey "A1B2C3D4"

You can get configuration details using the following cmdlet.

Get-AzureBGPPeering -AccessType Microsoft -ServiceKey "*********************************"

AdvertisedPublicPrefixes : 123.0.0.0/30AdvertisedPublicPrefixesState : ConfiguredAzureAsn : 12076CustomerAutonomousSystemNumber : 2245PeerAsn : 1234PrimaryAzurePort : PrimaryPeerSubnet : 10.0.0.0/30RoutingRegistryName : ARINSecondaryAzurePort : SecondaryPeerSubnet : 10.0.0.4/30State : EnabledVlanId : 300


Set-AzureBGPPeering -AccessType Microsoft -ServiceKey "*********************************" -PrimaryPeerSubnet "131.107.0.0/30" -SecondaryPeerSubnet "131.107.0.4/30" -VlanId 300 -PeerAsn 1234 -CustomerAsn 2245 -AdvertisedPublicPrefixes "123.0.0.0/30" -RoutingRegistryName "ARIN" -SharedKey "A1B2C3D4"


Remove-AzureBGPPeering -AccessType Microsoft -ServiceKey "*********************************"

Next steps

You can update any part of the configuration using the following cmdlet.

You can remove your peering configuration by running the following cmdlet.

Next, Link a VNet to an ExpressRoute circuit.

For more information about workflows, see ExpressRoute workflows.For more information about circuit peering, see ExpressRoute circuits and routing domains.

Connect a virtual network to an ExpressRoute circuitusing PowerShell (classic)3/9/2018 • 5 min to read • Edit Online

IMPORTANTIMPORTANT


This article will help you link virtual networks (VNets) to Azure ExpressRoute circuits by using the classicdeployment model and PowerShell. Virtual networks can either be in the same subscription or can be part ofanother subscription.







1. You need the latest version of the Azure PowerShell modules. You can download the latest PowerShellmodules from the PowerShell section of the Azure Downloads page. Follow the instructions in How to installand configure Azure PowerShell for step-by-step guidance on how to configure your computer to use theAzure PowerShell modules.

2. You need to review the prerequisites, routing requirements, and workflows before you begin configuration.3. You must have an active ExpressRoute circuit.

Follow the instructions to create an ExpressRoute circuit and have your connectivity provider enablethe circuit.Ensure that you have Azure private peering configured for your circuit. See the Configure routingarticle for routing instructions.Ensure that Azure private peering is configured and the BGP peering between your network andMicrosoft is up so that you can enable end-to-end connectivity.You must have a virtual network and a virtual network gateway created and fully provisioned. Followthe instructions to configure a virtual network for ExpressRoute.

You can link up to 10 virtual networks to an ExpressRoute circuit. All virtual networks must be in the samegeopolitical region. You can link a larger number of virtual networks to your ExpressRoute circuit, or link virtualnetworks that are in other geopolitical regions if you enabled the ExpressRoute premium add-on. Check the FAQfor more details on the premium add-on.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-linkvnet-classic.md


https://azure.microsoft.com/downloads/


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-vnet-portal-classic


New-AzureDedicatedCircuitLink -ServiceKey "*****************************" -VNetName "MyVNet"Provisioned


NOTENOTE

AdministrationAdministration


A single VNet can be linked to up to four ExpressRoute circuits. Use the process below to create a new link toeach ExpressRoute circuit you are connecting to. The ExpressRoute circuits can be in the same subscription,different subscriptions, or a mix of both.

You can link a virtual network to an ExpressRoute circuit by using the following cmdlet. Make sure that thevirtual network gateway is created and is ready for linking before you run the cmdlet.

You can share an ExpressRoute circuit across multiple subscriptions. The following figure shows a simpleschematic of how sharing works for ExpressRoute circuits across multiple subscriptions.

Each of the smaller clouds within the large cloud is used to represent subscriptions that belong to differentdepartments within an organization. Each of the departments within the organization can use their ownsubscription for deploying their services--but the departments can share a single ExpressRoute circuit to connectback to your on-premises network. A single department (in this example: IT) can own the ExpressRoute circuit.Other subscriptions within the organization can use the ExpressRoute circuit.

Connectivity and bandwidth charges for the dedicated circuit will be applied to the ExpressRoute circuit owner. All virtualnetworks share the same bandwidth.

The circuit owner is the administrator/coadministrator of the subscription in which the ExpressRoute circuit iscreated. The circuit owner can authorize administrators/coadministrators of other subscriptions, referred to ascircuit users, to use the dedicated circuit that they own. Circuit users who are authorized to use the organization'sExpressRoute circuit can link the virtual network in their subscription to the ExpressRoute circuit after they areauthorized.

The circuit owner has the power to modify and revoke authorizations at any time. Revoking an authorization willresult in all links being deleted from the subscription whose access was revoked.

New-AzureDedicatedCircuitLinkAuthorization -ServiceKey "**************************" -Description "Dev-Test Links" -Limit 2 -MicrosoftIds '[email protected]'

Description : Dev-Test LinksLimit : 2LinkAuthorizationId : **********************************MicrosoftIds : [email protected] : 0

Get-AzureDedicatedCircuitLinkAuthorization -ServiceKey: "**************************"

Description : EngineeringTeamLimit : 3LinkAuthorizationId : ####################################MicrosoftIds : [email protected] : 1

Description : MarketingTeamLimit : 1LinkAuthorizationId : @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@MicrosoftIds : [email protected] : 0

Description : Dev-Test LinksLimit : 2LinkAuthorizationId : &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&MicrosoftIds : [email protected] : 2

Set-AzureDedicatedCircuitLinkAuthorization -ServiceKey "**************************" -AuthorizationId "&&&&&&&&&&&&&&&&&&&&&&&&&&&&"-Limit 5

Description : Dev-Test LinksLimit : 5LinkAuthorizationId : &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&MicrosoftIds : [email protected] : 0

Creating an authorization

The circuit owner authorizes the administrators of other subscriptions to use the specified circuit. In thefollowing example, the administrator of the circuit (Contoso IT) enables the administrator of anothersubscription (Dev-Test) to link up to two virtual networks to the circuit. The Contoso IT administrator enablesthis by specifying the Dev-Test Microsoft ID. The cmdlet doesn't send email to the specified Microsoft ID. Thecircuit owner needs to explicitly notify the other subscription owner that the authorization is complete.

Reviewing authorizations

The circuit owner can review all authorizations that are issued on a particular circuit by running the followingcmdlet:

Updating authorizations

The circuit owner can modify authorizations by using the following cmdlet:

Deleting authorizations

The circuit owner can revoke/delete authorizations to the user by running the following cmdlet:

Remove-AzureDedicatedCircuitLinkAuthorization -ServiceKey "*****************************" -AuthorizationId "###############################"


Get-AzureAuthorizedDedicatedCircuit

Bandwidth : 200CircuitName : ContosoITLocation : Washington DCMaximumAllowedLinks : 2ServiceKey : &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&ServiceProviderName : equinixServiceProviderProvisioningState : ProvisionedStatus : EnabledUsedLinks : 0

New-AzureDedicatedCircuitLink â€“servicekey "&&&&&&&&&&&&&&&&&&&&&&&&&&" â€“VnetName 'SalesVNET1'

State VnetName----- --------Provisioned SalesVNET1

New-AzureDedicatedCircuitLink -ServiceKey "*****************************" -VNetName "MyVNet"

Next steps

Reviewing authorizations

The circuit user can review authorizations by using the following cmdlet:

Redeeming link authorizations

The circuit user can run the following cmdlet to redeem a link authorization:

Run this command in the newly linked subscription for the virtual network:


Configure ExpressRoute and Site-to-Site coexistingconnections (classic)8/22/2017 • 9 min to read • Edit Online

IMPORTANTIMPORTANT

IMPORTANTIMPORTANT

Limits and limitations

Having the ability to configure Site-to-Site VPN and ExpressRoute has several advantages. You can configure Site-to-Site VPN as a secure failover path for ExressRoute, or use Site-to-Site VPNs to connect to sites that are notconnected through ExpressRoute. We will cover the steps to configure both scenarios in this article. This articleapplies to the classic deployment model. This configuration is not available in the portal.






Azure currently works with two deployment models: Resource Manager and classic. The two models are notcompletely compatible with each other. Before you begin, you need to know which model that you want to work in.For information about the deployment models, see Understanding deployment models. If you are new to Azure,we recommend that you use the Resource Manager deployment model.

ExpressRoute circuits must be pre-configured before you follow the instructions below. Make sure that you have followed theguides to create an ExpressRoute circuit and configure routing before you follow the steps below.

Transit routing is not supported. You cannot route (via Azure) between your local network connected viaSite-to-Site VPN and your local network connected via ExpressRoute.Point-to-site is not supported. You can't enable point-to-site VPN connections to the same VNet that isconnected to ExpressRoute. Point-to-site VPN and ExpressRoute cannot coexist for the same VNet.Forced tunneling cannot be enabled on the Site-to-Site VPN gateway. You can only "force" all Internet-bound traffic back to your on-premises network via ExpressRoute.Basic SKU gateway is not supported. You must use a non-Basic SKU gateway for both the ExpressRoutegateway and the VPN gateway.Only route-based VPN gateway is supported. You must use a route-based VPN Gateway.Static route should be configured for your VPN gateway. If your local network is connected to bothExpressRoute and a Site-to-Site VPN, you must have a static route configured in your local network to routethe Site-to-Site VPN connection to the public Internet.ExpressRoute gateway must be configured first. You must create the ExpressRoute gateway first before

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-coexist-classic.md




Configuration designsConfigure a Site-to-Site VPN as a failover path for ExpressRouteConfigure a Site-to-Site VPN as a failover path for ExpressRoute

NOTENOTE

Configure a Site-to-Site VPN to connect to sites not connected through ExpressRouteConfigure a Site-to-Site VPN to connect to sites not connected through ExpressRoute

NOTENOTE

you add the Site-to-Site VPN gateway.

You can configure a Site-to-Site VPN connection as a backup for ExpressRoute. This applies only to virtualnetworks linked to the Azure private peering path. There is no VPN-based failover solution for services accessiblethrough Azure public and Microsoft peerings. The ExpressRoute circuit is always the primary link. Data will flowthrough the Site-to-Site VPN path only if the ExpressRoute circuit fails.

While ExpressRoute circuit is preferred over Site-to-Site VPN when both routes are the same, Azure will use the longest prefixmatch to choose the route towards the packet's destination.

You can configure your network where some sites connect directly to Azure over Site-to-Site VPN, and some sitesconnect through ExpressRoute.

You cannot a configure a virtual network as a transit router.

Selecting the steps to use

To create a new virtual network and coexisting connections

There are two different sets of procedures to choose from in order to configure connections that can coexist. Theconfiguration procedure that you select will depend on whether you have an existing virtual network that you wantto connect to, or you want to create a new virtual network.

I don't have a VNet and need to create one.

If you donâ€™t already have a virtual network, this procedure will walk you through creating a new virtualnetwork using the classic deployment model and creating new ExpressRoute and Site-to-Site VPNconnections. To configure, follow the steps in the article section To create a new virtual network andcoexisting connections.

I already have a classic deployment model VNet.

You may already have a virtual network in place with an existing Site-to-Site VPN connection orExpressRoute connection. The article section To configure coexsiting connections for an already existingVNet will walk you through deleting the gateway, and then creating new ExpressRoute and Site-to-SiteVPN connections. Note that when creating the new connections, the steps must be completed in a veryspecific order. Don't use the instructions in other articles to create your gateways and connections.

In this procedure, creating connections that can coexist will require you to delete your gateway, and thenconfigure new gateways. This means you will have downtime for your cross-premises connections whileyou delete and recreate your gateway and connections, but you will not need to migrate any of your VMs orservices to a new virtual network. Your VMs and services will still be able to communicate out through theload balancer while you configure your gateway if they are configured to do so.

This procedure will walk you through creating a VNet and create Site-to-Site and ExpressRoute connections thatwill coexist.

1. You'll need to install the latest version of the Azure PowerShell cmdlets. See How to install and configure AzurePowerShell for more information about installing the PowerShell cmdlets. Note that the cmdlets that you'll usefor this configuration may be slightly different than what you might be familiar with. Be sure to use the cmdletsspecified in these instructions.

2. Create a schema for your virtual network. For more information about the configuration schema, see AzureVirtual Network configuration schema.

When you create your schema, make sure you use the following values:

The gateway subnet for the virtual network must be /27 or a shorter prefix (such as /26 or /25).The gateway connection type is "Dedicated".


https://msdn.microsoft.com/library/azure/jj157100.aspx

Set-AzureVNetConfig -ConfigurationPath 'C:\NetworkConfig.xml'

New-AzureVNetGateway -VNetName MyAzureVNET -GatewayType DynamicRouting -GatewaySKU HighPerformance

New-AzureDedicatedCircuitLink -ServiceKey <service-key> -VNetName MyAzureVNET

New-AzureVirtualNetworkGateway -VNetName MyAzureVNET -GatewayName S2SVPN -GatewayType DynamicRouting -GatewaySKU HighPerformance

<VirtualNetworkSite name="MyAzureVNET" Location="Central US"> <AddressSpace> <AddressPrefix>10.17.159.192/26</AddressPrefix> </AddressSpace> <Subnets> <Subnet name="Subnet-1"> <AddressPrefix>10.17.159.192/27</AddressPrefix> </Subnet> <Subnet name="GatewaySubnet"> <AddressPrefix>10.17.159.224/27</AddressPrefix> </Subnet> </Subnets> <Gateway> <ConnectionsToLocalNetwork> <LocalNetworkSiteRef name="MyLocalNetwork"> <Connection type="Dedicated" /> </LocalNetworkSiteRef> </ConnectionsToLocalNetwork> </Gateway> </VirtualNetworkSite>

3. After creating and configuring your xml schema file, upload the file. This will create your virtual network.

Use the following cmdlet to upload your file, replacing the value with your own.

4. Create an ExpressRoute gateway. Be sure to specify the GatewaySKU as Standard, HighPerformance, orUltraPerformance and the GatewayType as DynamicRouting.

Use the following sample, substituting the values for your own.

5. Link the ExpressRoute gateway to the ExpressRoute circuit. After this step has been completed, theconnection between your on-premises network and Azure, through ExpressRoute, is established.

6. Next, create your Site-to-Site VPN gateway. The GatewaySKU must be Standard, HighPerformance, orUltraPerformance and the GatewayType must be DynamicRouting.

To retrieve the virtual network gateway settings, including the gateway ID and the public IP, use the Get-AzureVirtualNetworkGateway cmdlet.

Get-AzureVirtualNetworkGateway

GatewayId : 348ae011-ffa9-4add-b530-7cb30010565e GatewayName : S2SVPN LastEventData : GatewayType : DynamicRouting LastEventTimeStamp : 5/29/2015 4:41:41 PM LastEventMessage : Successfully created a gateway for the following virtual network: GNSDesMoines LastEventID : 23002 State : Provisioned VIPAddress : 104.43.x.y DefaultSite : GatewaySKU : HighPerformance Location : VnetId : 979aabcf-e47f-4136-ab9b-b4780c1e1bd5 SubnetId : EnableBgp : False OperationDescription : Get-AzureVirtualNetworkGateway OperationId : 42773656-85e1-a6b6-8705-35473f1e6f6a OperationStatus : Succeeded

IMPORTANTIMPORTANT

New-AzureLocalNetworkGateway -GatewayName MyLocalNetwork -IpAddress <MyLocalGatewayIp> -AddressSpace <MyLocalNetworkAddress>

NOTENOTE

Get-AzureLocalNetworkGateway

GatewayId : 532cb428-8c8c-4596-9a4f-7ae3a9fcd01b GatewayName : MyLocalNetwork IpAddress : 23.39.x.y AddressSpace : {10.1.2.0/24} OperationDescription : Get-AzureLocalNetworkGateway OperationId : ddc4bfae-502c-adc7-bd7d-1efbc00b3fe5 OperationStatus : Succeeded

7. Create a local site VPN gateway entity. This command doesnâ€™t configure your on-premises VPNgateway. Rather, it allows you to provide the local gateway settings, such as the public IP and the on-premises address space, so that the Azure VPN gateway can connect to it.

The local site for the Site-to-Site VPN is not defined in the netcfg. Instead, you must use this cmdlet to specify thelocal site parameters. You cannot define it using either portal, or the netcfg file.

Use the following sample, replacing the values with your own.

If your local network has multiple routes, you can pass them all in as an array. $MyLocalNetworkAddress =@("10.1.2.0/24","10.1.3.0/24","10.2.1.0/24")

To retrieve the virtual network gateway settings, including the gateway ID and the public IP, use the Get-AzureVirtualNetworkGateway cmdlet. See the following example.

8. Configure your local VPN device to connect to the new gateway. Use the information that you retrieved instep 6 when configuring your VPN device. For more information about VPN device configuration, see VPNDevice Configuration.


To configure coexsiting connections for an already existing VNet

NOTENOTE

New-AzureVirtualNetworkGatewayConnection -connectedEntityId <local-network-gateway-id> -gatewayConnectionName Azure2Local -gatewayConnectionType IPsec -sharedKey abc123 -virtualNetworkGatewayId <azure-s2s-vpn-gateway-id>

9. Link the Site-to-Site VPN gateway on Azure to the local gateway.

In this example, connectedEntityId is the local gateway ID, which you can find by running Get-AzureLocalNetworkGateway . You can find virtualNetworkGatewayId by using the Get-AzureVirtualNetworkGateway cmdlet. After this step, the connection between your local network and

Azure via the Site-to-Site VPN connection is established.

If you have an existing virtual network, check the gateway subnet size. If the gateway subnet is /28 or /29, youmust first delete the virtual network gateway and increase the gateway subnet size. The steps in this section willshow you how to do that.

If the gateway subnet is /27 or larger and the virtual network is connected via ExpressRoute, you can skip the stepsbelow and proceed to "Step 6 - Create a Site-to-Site VPN gateway" in the previous section.

When you delete the existing gateway, your local premises will lose the connection to your virtual network while you areworking on this configuration.

1. You'll need to install the latest version of the Azure Resource Manager PowerShell cmdlets. See How to installand configure Azure PowerShell for more information about installing the PowerShell cmdlets. Note that thecmdlets that you'll use for this configuration may be slightly different than what you might be familiar with. Besure to use the cmdlets specified in these instructions.

Remove-AzureVNetGateway â€“VnetName MyAzureVNET

Get-AzureVNetConfig â€“ExportToFile â€œC:\NetworkConfig.xmlâ€

NOTENOTE

2. Delete the existing ExpressRoute or Site-to-Site VPN gateway. Use the following cmdlet, replacing thevalues with your own.

3. Export the virtual network schema. Use the following PowerShell cmdlet, replacing the values with yourown.

4. Edit the network configuration file schema so that the gateway subnet is /27 or a shorter prefix (such as /26or /25). See the following example.

If you don't have enough IP addresses left in your virtual network to increase the gateway subnet size, you need toadd more IP address space. For more information about the configuration schema, see Azure Virtual Networkconfiguration schema.


https://msdn.microsoft.com/library/azure/jj157100.aspx

Next steps

<Subnet name="GatewaySubnet"> <AddressPrefix>10.17.159.224/27</AddressPrefix> </Subnet>

<Gateway> <ConnectionsToLocalNetwork> <LocalNetworkSiteRef name="MyLocalNetwork"> <Connection type="Dedicated" /> </LocalNetworkSiteRef> </ConnectionsToLocalNetwork> </Gateway>

6. At this point, you'll have a VNet with no gateways. To create new gateways and complete your connections, youcan proceed with Step 4 - Create an ExpressRoute gateway, found in the preceding set of steps.

5. If your previous gateway was a Site-to-Site VPN, you must also change the connection type to Dedicated.

For more information about ExpressRoute, see the ExpressRoute FAQ

Configure a virtual network gateway forExpressRoute using PowerShell (classic)6/27/2017 • 2 min to read • Edit Online

IMPORTANTIMPORTANT

Before beginning

NOTENOTE

Add a gateway

New-AzureVNetGateway -VNetName "MyAzureVNET" -GatewayName "ERGateway" -GatewayType DynamicRouting -GatewaySKU Standard

Verify the gateway was created

This article will walk you through the steps to add, resize, and remove a virtual network (VNet) gateway for a pre-existing VNet. The steps for this configuration are specifically for VNets that were created using the classicdeployment model and that will be be used in an ExpressRoute configuration.






Azure currently works with two deployment models: Resource Manager and classic. The two models are notcompletely compatible with each other. Before you begin, you need to know which model that you want to work in.For information about the deployment models, see Understanding deployment models. If you are new to Azure,we recommend that you use the Resource Manager deployment model.

Verify that you have installed the Azure PowerShell cmdlets needed for this configuration (1.0.2 or later). If youhaven't installed the cmdlets, you'll need to do so before beginning the configuration steps. For more informationabout installing Azure PowerShell, see How to install and configure Azure PowerShell.

You must create a VNet and a gateway subnet first, before working on the following tasks.

These examples do not apply to S2S/ExpressRoute coexist configurations. For more information about working withgateways in a coexist configuration, see Configure coexisting connections

Use the command below to create a gateway. Be sure to substitute any values for your own.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-howto-add-gateway-classic.md



Get-AzureVNetGateway

Resize a gateway

IMPORTANTIMPORTANT

Resize-AzureVNetGateway -GatewayId <Gateway ID> -GatewaySKU HighPerformance

Remove a gateway

Remove-AzureVnetGateway -GatewayId <Gateway ID>

Next steps

Use the command below to verify that the gateway has been created. This command also retrieves the gateway ID,which you need for other operations.

There are a number of Gateway SKUs. You can use the following command to change the Gateway SKU at anytime.

This command doesn't work for UltraPerformance gateway. To change your gateway to an UltraPerformance gateway, firstremove the existing ExpressRoute gateway, and then create a new UltraPerformance gateway. To downgrade your gatewayfrom an UltraPerformance gateway, first remove the UltraPerformance gateway, and then create a new gateway.

Use the command below to remove a gateway


Microsoft cloud services and network security6/27/2017 • 37 min to read • Edit Online

Fast start

Microsoft compliance and infrastructure protection

Microsoft cloud services deliver hyper-scale services and infrastructure, enterprise-grade capabilities, and manychoices for hybrid connectivity. Customers can choose to access these services either via the Internet or with AzureExpressRoute, which provides private network connectivity. The Microsoft Azure platform allows customers toseamlessly extend their infrastructure into the cloud and build multi-tier architectures. Additionally, third partiescan enable enhanced capabilities by offering security services and virtual appliances. This white paper provides anoverview of security and architectural issues that customers should consider when using Microsoft cloud servicesaccessed via ExpressRoute. It also covers creating more secure services in Azure virtual networks.

The following logic chart can direct you to a specific example of the many security techniques available with theAzure platform. For quick reference, find the example that best fits your case. For expanded explanations, continuereading through the paper.

Example 1: Build a perimeter network (also known as DMZ, demilitarized zone, or screened subnet) to help protectapplications with network security groups (NSGs).Example 2: Build a perimeter network to help protect applications with a firewall and NSGs.Example 3: Build a perimeter network to help protect networks with a firewall, user-defined route (UDR), and NSG.Example 4: Add a hybrid connection with a site-to-site, virtual appliance virtual private network (VPN).Example 5: Add a hybrid connection with a site-to-site, Azure VPN gateway.Example 6: Add a hybrid connection with ExpressRoute.Examples for adding connections between virtual networks, high availability, and service chaining will be added tothis document over the next few months.

To help organizations comply with national, regional, and industry-specific requirements governing the collectionand use of individualsâ€™ data, Microsoft offers over 40 certifications and attestations. The most comprehensiveset of any cloud service provider.

https://github.com/Microsoft/azure-docs/blob/master/articles/best-practices-network-security.md

Traditional security architectures and perimeter networks

For more information, see the compliance information on the Microsoft Trust Center.

Microsoft has a comprehensive approach to protect cloud infrastructure needed to run hyper-scale global services.Microsoft cloud infrastructure includes hardware, software, networks, and administrative and operations staff, inaddition to the physical data centers.

This approach provides a more secure foundation for customers to deploy their services in the Microsoft cloud.The next step is for customers to design and create a security architecture to protect these services.

Although Microsoft invests heavily in protecting the cloud infrastructure, customers must also protect their cloudservices and resource groups. A multilayered approach to security provides the best defense. A perimeter networksecurity zone protects internal network resources from an untrusted network. A perimeter network refers to theedges or parts of the network that sit between the Internet and the protected enterprise IT infrastructure.

In typical enterprise networks, the core infrastructure is heavily fortified at the perimeters, with multiple layers ofsecurity devices. The boundary of each layer consists of devices and policy enforcement points. Each layer caninclude a combination of the following network security devices: firewalls, Denial of Service (DoS) prevention,Intrusion Detection or Protection Systems (IDS/IPS), and VPN devices. Policy enforcement can take the form offirewall policies, access control lists (ACLs), or specific routing. The first line of defense in the network, directlyaccepting incoming traffic from the Internet, is a combination of these mechanisms to block attacks and harmfultraffic while allowing legitimate requests further into the network. This traffic routes directly to resources in theperimeter network. That resource may then â€œtalkâ€ to resources deeper in the network, transiting the nextboundary for validation first. The outermost layer is called the perimeter network because this part of the networkis exposed to the Internet, usually with some form of protection on both sides. The following figure shows anexample of a single subnet perimeter network in a corporate network, with two security boundaries.

https://azure.microsoft.com/support/trust-center/compliance/

There are many architectures used to implement a perimeter network. These architectures can range from a simpleload balancer to a multiple-subnet perimeter network with varied mechanisms at each boundary to block trafficand protect the deeper layers of the corporate network. How the perimeter network is built depends on the specificneeds of the organization and its overall risk tolerance.

As customers move their workloads to public clouds, it is critical to support similar capabilities for perimeternetwork architecture in Azure to meet compliance and security requirements. This document provides guidelineson how customers can build a secure network environment in Azure. It focuses on the perimeter network, but alsoincludes a comprehensive discussion of many aspects of network security. The following questions inform thisdiscussion:

How can a perimeter network in Azure be built?What are some of the Azure features available to build the perimeter network?How can back-end workloads be protected?How are Internet communications controlled to the workloads in Azure?How can the on-premises networks be protected from deployments in Azure?When should native Azure security features be used versus third-party appliances or services?

The following diagram shows various layers of security Azure provides to customers. These layers are both nativein the Azure platform itself and customer-defined features:

Inbound from the Internet, Azure DDoS helps protect against large-scale attacks against Azure. The next layer is

Overview of Azure virtual networks

NOTENOTE

customer-defined public IP addresses (endpoints), which are used to determine which traffic can pass through thecloud service to the virtual network. Native Azure virtual network isolation ensures complete isolation from allother networks and that traffic only flows through user configured paths and methods. These paths and methodsare the next layer, where NSGs, UDR, and network virtual appliances can be used to create security boundaries toprotect the application deployments in the protected network.

The next section provides an overview of Azure virtual networks. These virtual networks are created by customers,and are what their deployed workloads are connected to. Virtual networks are the basis of all the network securityfeatures required to establish a perimeter network to protect customer deployments in Azure.

Before Internet traffic can get to the Azure virtual networks, there are two layers of security inherent to the Azureplatform:

1. DDoS protection: DDoS protection is a layer of the Azure physical network that protects the Azureplatform itself from large-scale Internet-based attacks. These attacks use multiple â€œbotâ€ nodes in anattempt to overwhelm an Internet service. Azure has a robust DDoS protection mesh on all inbound,outbound, and cross-Azure region connectivity. This DDoS protection layer has no user configurableattributes and is not accessible to the customer. The DDoS protection layer protects Azure as a platformfrom large-scale attacks, it also monitors out-bound traffic and cross-Azure region traffic. Using networkvirtual appliances on the VNet, additional layers of resilience can be configured by the customer against asmaller scale attack that doesn't trip the platform level protection. An example of DDoS in action; if aninternet facing IP address was attacked by a large-scale DDoS attack, Azure would detect the sources of theattacks and scrub the offending traffic before it reached its intended destination. In almost all cases, theattacked endpoint isn't affected by the attack. In the rare cases that an endpoint is affected, no traffic isaffected to other endpoints, only the attacked endpoint. Thus other customers and services would see noimpact from that attack. It's critical to note that Azure DDoS is only looking for large-scale attacks. It ispossible that your specific service could be overwhelmed before the platform level protection thresholds areexceeded. For example, a web site on a single A0 IIS server, could be taken offline by a DDoS attack beforeAzure platform level DDoS protection registered a threat.

2. Public IP Addresses: Public IP addresses (enabled via service endpoints, Public IP addresses, ApplicationGateway, and other Azure features that present a public IP address to the internet routed to your resource)allow cloud services or resource groups to have public Internet IP addresses and ports exposed. Theendpoint uses Network Address Translation (NAT) to route traffic to the internal address and port on theAzure virtual network. This path is the primary way for external traffic to pass into the virtual network. ThePublic IP addresses are configurable to determine which traffic is passed in, and how and where it'stranslated on to the virtual network.

Once traffic reaches the virtual network, there are many features that come into play. Azure virtual networks arethe foundation for customers to attach their workloads and where basic network-level security applies. It is aprivate network (a virtual network overlay) in Azure for customers with the following features and characteristics:

Traffic isolation: A virtual network is the traffic isolation boundary on the Azure platform. Virtual machines(VMs) in one virtual network cannot communicate directly to VMs in a different virtual network, even if bothvirtual networks are created by the same customer. Isolation is a critical property that ensures customer VMsand communication remains private within a virtual network.

Traffic isolation refers only to traffic inbound to the virtual network. By default outbound traffic from the VNet to the internetis allowed, but can be prevented if desired by NSGs.

Perimeter network characteristics and requirements

Perimeter network characteristicsPerimeter network characteristics

Multi-tier topology: Virtual networks allow customers to define multi-tier topology by allocating subnets anddesignating separate address spaces for different elements or â€œtiersâ€ of their workloads. These logicalgroupings and topologies enable customers to define different access policy based on the workload types, andalso control traffic flows between the tiers.Cross-premises connectivity: Customers can establish cross-premises connectivity between a virtual networkand multiple on-premises sites or other virtual networks in Azure. To construct a connection, customers can useVNet Peering, Azure VPN Gateways, third-party network virtual appliances, or ExpressRoute. Azure supportssite-to-site (S2S) VPNs using standard IPsec/IKE protocols and ExpressRoute private connectivity.NSG allows customers to create rules (ACLs) at the desired level of granularity: network interfaces, individualVMs, or virtual subnets. Customers can control access by permitting or denying communication between theworkloads within a virtual network, from systems on customerâ€™s networks via cross-premises connectivity,or direct Internet communication.UDR and IP Forwarding allow customers to define the communication paths between different tiers within avirtual network. Customers can deploy a firewall, IDS/IPS, and other virtual appliances, and route networktraffic through these security appliances for security boundary policy enforcement, auditing, and inspection.Network virtual appliances in the Azure Marketplace: Security appliances such as firewalls, load balancers,and IDS/IPS are available in the Azure Marketplace and the VM Image Gallery. Customers can deploy theseappliances into their virtual networks, and specifically, at their security boundaries (including the perimeternetwork subnets) to complete a multi-tiered secure network environment.

With these features and capabilities, one example of how a perimeter network architecture could be constructed inAzure is the following diagram:

The perimeter network is the front end of the network, directly interfacing communication from the Internet. Theincoming packets should flow through the security appliances, such as the firewall, IDS, and IPS, before reachingthe back-end servers. Internet-bound packets from the workloads can also flow through the security appliances inthe perimeter network for policy enforcement, inspection, and auditing purposes, before leaving the network.Additionally, the perimeter network can host cross-premises VPN gateways between customer virtual networksand on-premises networks.

Referencing the previous figure, some of the characteristics of a good perimeter network are as follows:

Internet-facing:The perimeter network subnet itself is Internet-facing, directly communicating with the Internet.

Perimeter network requirementsPerimeter network requirements

Protected network:

Other common practices and constraints:

Public IP addresses, VIPs, and/or service endpoints pass Internet traffic to the front-end network anddevices.Inbound traffic from the Internet passes through security devices before other resources on the front-end network.If outbound security is enabled, traffic passes through security devices, as the final step, before passing tothe Internet.

There is no direct path from the Internet to the core infrastructure.Channels to the core infrastructure must traverse through security devices such as NSGs, firewalls, orVPN devices.Other devices must not bridge Internet and the core infrastructure.Security devices on both the Internet-facing and the protected network facing boundaries of theperimeter network (for example, the two firewall icons shown in the previous figure) may actually be asingle virtual appliance with differentiated rules or interfaces for each boundary. For example, onephysical device, logically separated, handling the load for both boundaries of the perimeter network.

Workloads must not store business critical information.Access and updates to perimeter network configurations and deployments are limited to only authorizedadministrators.

To enable these characteristics, follow these guidelines on virtual network requirements to implement a successfulperimeter network:

Subnet architecture: Specify the virtual network such that an entire subnet is dedicated as the perimeternetwork, separated from other subnets in the same virtual network. This separation ensures the traffic betweenthe perimeter network and other internal or private subnet tiers flows through a firewall or IDS/IPS virtualappliance. User-defined routes on the boundary subnets are required to forward this traffic to the virtualappliance.NSG: The perimeter network subnet itself should be open to allow communication with the Internet, but thatdoes not mean customers should be bypassing NSGs. Follow common security practices to minimize thenetwork surfaces exposed to the Internet. Lock down the remote address ranges allowed to access thedeployments or the specific application protocols and ports that are open. There may be circumstances,however, in which a complete lock-down is not possible. For example, if customers have an external website inAzure, the perimeter network should allow the incoming web requests from any public IP addresses, but shouldonly open the web application ports: TCP on port 80 and/or TCP on port 443.Routing table: The perimeter network subnet itself should be able to communicate to the Internet directly, butshould not allow direct communication to and from the back end or on-premises networks without goingthrough a firewall or security appliance.Security appliance configuration: To route and inspect packets between the perimeter network and the restof the protected networks, the security appliances such as firewall, IDS, and IPS devices may be multi-homed.They may have separate NICs for the perimeter network and the back-end subnets. The NICs in the perimeternetwork communicate directly to and from the Internet, with the corresponding NSGs and the perimeternetwork routing table. The NICs connecting to the back-end subnets have more restricted NSGs and routingtables of the corresponding back-end subnets.Security appliance functionality: The security appliances deployed in the perimeter network typicallyperform the following functionality:

Firewall: Enforcing firewall rules or access control policies for the incoming requests.Threat detection and prevention: Detecting and mitigating malicious attacks from the Internet.Auditing and logging: Maintaining detailed logs for auditing and analysis.

TIPTIP

Questions to be asked when building network boundariesQuestions to be asked when building network boundaries

1) How many boundaries are needed?1) How many boundaries are needed?

TIPTIP

Reverse proxy: Redirecting the incoming requests to the corresponding back-end servers. Thisredirection involves mapping and translating the destination addresses on the front-end devices, typicallyfirewalls, to the back-end server addresses.Forward proxy: Providing NAT and performing auditing for communication initiated from within thevirtual network to the Internet.Router : Forwarding incoming and cross-subnet traffic inside the virtual network.VPN device: Acting as the cross-premises VPN gateways for cross-premises VPN connectivity betweencustomer on-premises networks and Azure virtual networks.VPN server : Accepting VPN clients connecting to Azure virtual networks.

Keep the following two groups separate: the individuals authorized to access the perimeter network security gear and theindividuals authorized as application development, deployment, or operations administrators. Keeping these groups separateallows for a segregation of duties and prevents a single person from bypassing both applications security and networksecurity controls.

In this section, unless specifically mentioned, the term "networks" refers to private Azure virtual networks createdby a subscription administrator. The term doesn't refer to the underlying physical networks within Azure.

Also, Azure virtual networks are often used to extend traditional on-premises networks. It is possible toincorporate either site-to-site or ExpressRoute hybrid networking solutions with perimeter network architectures.This hybrid link is an important consideration in building network security boundaries.

The following three questions are critical to answer when you're building a network with a perimeter network andmultiple security boundaries.

The first decision point is to decide how many security boundaries are needed in a given scenario:

A single boundary: One on the front-end perimeter network, between the virtual network and the Internet.Two boundaries: One on the Internet side of the perimeter network, and another between the perimeternetwork subnet and the back-end subnets in the Azure virtual networks.Three boundaries: One on the Internet side of the perimeter network, one between the perimeter network andback-end subnets, and one between the back-end subnets and the on-premises network.N boundaries: A variable number. Depending on security requirements, there is no limit to the number ofsecurity boundaries that can be applied in a given network.

The number and type of boundaries needed vary based on a companyâ€™s risk tolerance and the specific scenariobeing implemented. This decision is often made together with multiple groups within an organization, oftenincluding a risk and compliance team, a network and platform team, and an application development team. Peoplewith knowledge of security, the data involved, and the technologies being used should have a say in this decision toensure the appropriate security stance for each implementation.

Use the smallest number of boundaries that satisfy the security requirements for a given situation. With more boundaries,operations and troubleshooting can be more difficult, as well as the management overhead involved with managing themultiple boundary policies over time. However, insufficient boundaries increase risk. Finding the balance is critical.

2) Where are the boundaries located?2) Where are the boundaries located?

3) How are the boundaries implemented?3) How are the boundaries implemented?

The preceding figure shows a high-level view of a three security boundary network. The boundaries are betweenthe perimeter network and the Internet, the Azure front-end and back-end private subnets, and the Azure back-endsubnet and the on-premises corporate network.

Once the number of boundaries are decided, where to implement them is the next decision point. There aregenerally three choices:

Using an Internet-based intermediary service (for example, a cloud-based Web application firewall, which is notdiscussed in this document)Using native features and/or network virtual appliances in AzureUsing physical devices on the on-premises network

On purely Azure networks, the options are native Azure features (for example, Azure Load Balancers) or networkvirtual appliances from the rich partner ecosystem of Azure (for example, Check Point firewalls).

If a boundary is needed between Azure and an on-premises network, the security devices can reside on either sideof the connection (or both sides). Thus a decision must be made on the location to place security gear.

In the previous figure, the Internet-to-perimeter network and the front-to-back-end boundaries are entirelycontained within Azure, and must be either native Azure features or network virtual appliances. Security devices onthe boundary between Azure (back-end subnet) and the corporate network could be either on the Azure side or theon-premises side, or even a combination of devices on both sides. There can be significant advantages anddisadvantages to either option that must be seriously considered.

For example, using existing physical security gear on the on-premises network side has the advantage that no newgear is needed. It just needs reconfiguration. The disadvantage, however, is that all traffic must come back fromAzure to the on-premises network to be seen by the security gear. Thus Azure-to-Azure traffic could incursignificant latency, and affect application performance and user experience, if it was forced back to the on-premisesnetwork for security policy enforcement.

Each security boundary will likely have different capability requirements (for example, IDS and firewall rules on theInternet side of the perimeter network, but only ACLs between the perimeter network and back-end subnet).Deciding on which device (or how many devices) to use depends on the scenario and security requirements. In thefollowing section, examples 1, 2, and 3 discuss some options that could be used. Reviewing the Azure nativenetwork features and the devices available in Azure from the partner ecosystem shows the myriad optionsavailable to solve virtually any scenario.

Another key implementation decision point is how to connect the on-premises network with Azure. Should youuse the Azure virtual gateway or a network virtual appliance? These options are discussed in greater detail in the

Examples: Building security boundaries with Azure virtual networksExample 1 Build a perimeter network to help protect applications with NSGsExample 1 Build a perimeter network to help protect applications with NSGs

Environment descriptionEnvironment description

following section (examples 4, 5, and 6).

Additionally, traffic between virtual networks within Azure may be needed. These scenarios will be added in thefuture.

Once you know the answers to the previous questions, the Fast Start section can help identify which examples aremost appropriate for a given scenario.

Back to Fast start | Detailed build instructions for this example

In this example, there is a subscription that contains the following resources:

A single resource groupA virtual network with two subnets: â€œFrontEndâ€ and â€œBackEndâ€A Network Security Group that is applied to both subnetsA Windows server that represents an application web server (â€œIIS01â€)Two Windows servers that represent application back-end servers (â€œAppVM01â€, â€œAppVM02â€)A Windows server that represents a DNS server (â€œDNS01â€)A public IP associated with the application web server

For scripts and an Azure Resource Manager template, see the detailed build instructions.

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-dmz-nsg


NSG descriptionNSG description

TIPTIP

ConclusionConclusion

Example 2 Build a perimeter network to help protect applications with a firewall and NSGsExample 2 Build a perimeter network to help protect applications with a firewall and NSGs

In this example, an NSG group is built and then loaded with six rules.

Generally speaking, you should create your specific â€œAllowâ€ rules first, followed by the more generic â€œDenyâ€ rules.The given priority dictates which rules are evaluated first. Once traffic is found to apply to a specific rule, no further rules areevaluated. NSG rules can apply in either the inbound or outbound direction (from the perspective of the subnet).

Declaratively, the following rules are being built for inbound traffic:

1. Internal DNS traffic (port 53) is allowed.2. RDP traffic (port 3389) from the Internet to any Virtual Machine is allowed.3. HTTP traffic (port 80) from the Internet to web server (IIS01) is allowed.4. Any traffic (all ports) from IIS01 to AppVM1 is allowed.5. Any traffic (all ports) from the Internet to the entire virtual network (both subnets) is denied.6. Any traffic (all ports) from the front-end subnet to the back-end subnet is denied.

With these rules bound to each subnet, if an HTTP request was inbound from the Internet to the web server, bothrules 3 (allow) and 5 (deny) would apply. But because rule 3 has a higher priority, only it would apply, and rule 5would not come into play. Thus the HTTP request would be allowed to the web server. If that same traffic wastrying to reach the DNS01 server, rule 5 (deny) would be the first to apply, and the traffic would not be allowed topass to the server. Rule 6 (deny) blocks the front-end subnet from talking to the back-end subnet (except forallowed traffic in rules 1 and 4). This rule-set protects the back-end network in case an attacker compromises theweb application on the front end. The attacker would have limited access to the back-end â€œprotectedâ€ network(only to resources exposed on the AppVM01 server).

There is a default outbound rule that allows traffic out to the Internet. For this example, weâ€™re allowingoutbound traffic and not modifying any outbound rules. To lock down traffic in both directions, user-definedrouting is required (see example 3).

This example is a relatively simple and straightforward way of isolating the back-end subnet from inbound traffic.For more information, see the detailed build instructions. These instructions include:

How to build this perimeter network with classic PowerShell scripts.How to build this perimeter network with an Azure Resource Manager template.Detailed descriptions of each NSG command.Detailed traffic flow scenarios, showing how traffic is allowed or denied in each layer.



https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-dmz-nsg-fw-asm



TIPTIP


A single resource groupA virtual network with two subnets: â€œFrontEndâ€ and â€œBackEndâ€A Network Security Group that is applied to both subnetsA network virtual appliance, in this case a firewall, connected to the front-end subnetA Windows server that represents an application web server (â€œIIS01â€)Two Windows servers that represent application back-end servers (â€œAppVM01â€, â€œAppVM02â€)A Windows server that represents a DNS server (â€œDNS01â€)


In this example, an NSG group is built and then loaded with six rules.

Generally speaking, you should create your specific â€œAllowâ€ rules first, followed by the more generic â€œDenyâ€ rules.The given priority dictates which rules are evaluated first. Once traffic is found to apply to a specific rule, no further rules areevaluated. NSG rules can apply in either the inbound or outbound direction (from the perspective of the subnet).

Declaratively, the following rules are being built for inbound traffic:

1. Internal DNS traffic (port 53) is allowed.


Firewall rule descriptionFirewall rule description


Example 3 Build a perimeter network to help protect networks with a firewall and UDR and NSGExample 3 Build a perimeter network to help protect networks with a firewall and UDR and NSG

2. RDP traffic (port 3389) from the Internet to any Virtual Machine is allowed.3. Any Internet traffic (all ports) to the network virtual appliance (firewall) is allowed.4. Any traffic (all ports) from IIS01 to AppVM1 is allowed.5. Any traffic (all ports) from the Internet to the entire virtual network (both subnets) is denied.6. Any traffic (all ports) from the front-end subnet to the back-end subnet is denied.

With these rules bound to each subnet, if an HTTP request was inbound from the Internet to the firewall, bothrules 3 (allow) and 5 (deny) would apply. But because rule 3 has a higher priority, only it would apply, and rule 5would not come into play. Thus the HTTP request would be allowed to the firewall. If that same traffic was trying toreach the IIS01 server, even though itâ€™s on the front-end subnet, rule 5 (deny) would apply, and the trafficwould not be allowed to pass to the server. Rule 6 (deny) blocks the front-end subnet from talking to the back-endsubnet (except for allowed traffic in rules 1 and 4). This rule-set protects the back-end network in case an attackercompromises the web application on the front end. The attacker would have limited access to the back-endâ€œprotectedâ€ network (only to resources exposed on the AppVM01 server).

There is a default outbound rule that allows traffic out to the Internet. For this example, weâ€™re allowingoutbound traffic and not modifying any outbound rules. To lock down traffic in both directions, user-definedrouting is required (see example 3).

On the firewall, forwarding rules should be created. Since this example only routes Internet traffic in-bound to thefirewall and then to the web server, only one forwarding network address translation (NAT) rule is needed.

The forwarding rule accepts any inbound source address that hits the firewall trying to reach HTTP (port 80 or 443for HTTPS). It's sent out of the firewallâ€™s local interface and redirected to the web server with the IP Address of10.0.1.5.

This example is a relatively straightforward way of protecting your application with a firewall and isolating theback-end subnet from inbound traffic. For more information, see the detailed build instructions. These instructionsinclude:

How to build this perimeter network with classic PowerShell scripts.How to build this example with an Azure Resource Manager template.Detailed descriptions of each NSG command and firewall rule.Detailed traffic flow scenarios, showing how traffic is allowed or denied in each layer.



https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-dmz-nsg-fw-udr-asm


UDR descriptionUDR description

Effective routes : Address Prefix Next hop type Next hop IP address Status Source -------------- ------------- ------------------- ------ ------ {10.0.0.0/16} VNETLocal Active Default {0.0.0.0/0} Internet Active Default {10.0.0.0/8} Null Active Default {100.64.0.0/10} Null Active Default {172.16.0.0/12} Null Active Default {192.168.0.0/16} Null Active Default


A single resource groupA virtual network with three subnets: â€œSecNetâ€, â€œFrontEndâ€, and â€œBackEndâ€A network virtual appliance, in this case a firewall, connected to the SecNet subnetA Windows server that represents an application web server (â€œIIS01â€)Two Windows servers that represent application back-end servers (â€œAppVM01â€, â€œAppVM02â€)A Windows server that represents a DNS server (â€œDNS01â€)


By default, the following system routes are defined as:

The VNETLocal is always one or more defined address prefixes that make up the virtual network for that specificnetwork (that is, it changes from virtual network to virtual network, depending on how each specific virtualnetwork is defined). The remaining system routes are static and default as indicated in the table.

In this example, two routing tables are created, one each for the front-end and back-end subnets. Each table isloaded with static routes appropriate for the given subnet. In this example, each table has three routes that directall traffic (0.0.0.0/0) through the firewall (Next hop = Virtual Appliance IP address):


TIPTIP

Effective routes : Address Prefix Next hop type Next hop IP address Status Source -------------- ------------- ------------------- ------ ------ {10.0.1.0/24} VNETLocal Active {10.0.0.0/16} VirtualAppliance 10.0.0.4 Active {0.0.0.0/0} VirtualAppliance 10.0.0.4 Active

NOTENOTE

IP Forwarding descriptionIP Forwarding description


1. Local subnet traffic with no Next Hop defined to allow local subnet traffic to bypass the firewall.2. Virtual network traffic with a Next Hop defined as firewall. This next hop overrides the default rule that allows

local virtual network traffic to route directly.3. All remaining traffic (0/0) with a Next Hop defined as the firewall.

Not having the local subnet entry in the UDR breaks local subnet communications.

In our example, 10.0.1.0/24 pointing to VNETLocal is critical! Without it, packet leaving the Web Server (10.0.1.4) destinedto another local server (for example) 10.0.1.25 will fail as they will be sent to the NVA. The NVA will send it to the subnet,and the subnet will resend it to the NVA in an infinite loop.The chances of a routing loop are typically higher on appliances with multiple NICs that are connected to separatesubnets, which is often of traditional, on-premises appliances.

Once the routing tables are created, they must be bound to their subnets. The front-end subnet routing table, oncecreated and bound to the subnet, would look like this output:

UDR can now be applied to the gateway subnet on which the ExpressRoute circuit is connected.

Examples of how to enable your perimeter network with ExpressRoute or site-to-site networking are shown in examples 3and 4.

IP Forwarding is a companion feature to UDR. IP Forwarding is a setting on a virtual appliance that allows it toreceive traffic not specifically addressed to the appliance, and then forward that traffic to its ultimate destination.

For example, if AppVM01 makes a request to the DNS01 server, UDR would route this traffic to the firewall. WithIP Forwarding enabled, the traffic for the DNS01 destination (10.0.2.4) is accepted by the appliance (10.0.0.4) andthen forwarded to its ultimate destination (10.0.2.4). Without IP Forwarding enabled on the firewall, traffic wouldnot be accepted by the appliance even though the route table has the firewall as the next hop. To use a virtualappliance, itâ€™s critical to remember to enable IP Forwarding along with UDR.

In this example, an NSG group is built and then loaded with a single rule. This group is then bound only to thefront-end and back-end subnets (not the SecNet). Declaratively the following rule is being built:

Any traffic (all ports) from the Internet to the entire virtual network (all subnets) is denied.

Although NSGs are used in this example, its main purpose is as a secondary layer of defense against manualmisconfiguration. The goal is to block all inbound traffic from the Internet to either the front-end or back-endsubnets. Traffic should only flow through the SecNet subnet to the firewall (and then, if appropriate, on to thefront-end or back-end subnets). Plus, with the UDR rules in place, any traffic that did make it into the front-end orback-end subnets would be directed out to the firewall (thanks to UDR). The firewall would see this traffic as anasymmetric flow and would drop the outbound traffic. Thus there are three layers of security protecting thesubnets:

Firewall rulesFirewall rules

NOTENOTE

Firewall rules descriptionFirewall rules description

No Public IP addresses on any FrontEnd or BackEnd NICs.NSGs denying traffic from the Internet.The firewall dropping asymmetric traffic.

One interesting point regarding the NSG in this example is that it contains only one rule, which is to deny Internettraffic to the entire virtual network, including the Security subnet. However, since the NSG is only bound to thefront-end and back-end subnets, the rule isnâ€™t processed on traffic inbound to the Security subnet. As a result,traffic flows to the Security subnet.

On the firewall, forwarding rules should be created. Since the firewall is blocking or forwarding all inbound,outbound, and intra-virtual network traffic, many firewall rules are needed. Also, all inbound traffic hits theSecurity Service public IP address (on different ports), to be processed by the firewall. A best practice is to diagramthe logical flows before setting up the subnets and firewall rules, to avoid rework later. The following figure is alogical view of the firewall rules for this example:

Based on the Network Virtual Appliance used, the management ports vary. In this example, a Barracuda NextGen Firewall isreferenced, which uses ports 22, 801, and 807. Consult the appliance vendor documentation to find the exact ports used formanagement of the device being used.

In the preceding logical diagram, the security subnet is not shown because the firewall is the only resource on thatsubnet. The diagram is showing the firewall rules and how they logically allow or deny traffic flows, not the actualrouted path. Also, the external ports selected for the RDP traffic are higher ranged ports (8014 â€“ 8026) and wereselected to loosely align with the last two octets of the local IP address for easier readability (for example, localserver address 10.0.1.4 is associated with external port 8014). Any higher non-conflicting ports, however, could beused.

For this example, we need seven types of rules:

External rules (for inbound traffic):

TIPTIP


Example 4 Add a hybrid connection with a site-to-site, virtual appliance VPNExample 4 Add a hybrid connection with a site-to-site, virtual appliance VPN

Internal rules (for intra-virtual network traffic)

Fail-safe rule (for traffic that doesnâ€™t meet any of the previous):

1. Firewall management rule: This App Redirect rule allows traffic to pass to the management ports of thenetwork virtual appliance.

2. RDP rules (for each Windows server): These four rules (one for each server) allow management of theindividual servers via RDP. The four RDP rules could also be collapsed into one rule, depending on thecapabilities of the network virtual appliance being used.

3. Application traffic rules: There are two of these rules, the first for the front-end web traffic, and thesecond for the back-end traffic (for example, web server to data tier). The configuration of these rulesdepends on the network architecture (where your servers are placed) and traffic flows (which directionthe traffic flows, and which ports are used).

The first rule allows the actual application traffic to reach the application server. While the otherrules allow for security and management, application traffic rules are what allow external users orservices to access the applications. For this example, there is a single web server on port 80. Thusa single firewall application rule redirects inbound traffic to the external IP, to the web serversinternal IP address. The redirected traffic session would be translated via NAT to the internalserver.The second rule is the back-end rule to allow the web server to talk to the AppVM01 server (butnot AppVM02) via any port.

1. Outbound to Internet rule: This rule allows traffic from any network to pass to the selected networks.This rule is usually a default rule already on the firewall, but in a disabled state. This rule should beenabled for this example.

2. DNS rule: This rule allows only DNS (port 53) traffic to pass to the DNS server. For this environment,most traffic from the front end to the back end is blocked. This rule specifically allows DNS from anylocal subnet.

3. Subnet to subnet rule: This rule is to allow any server on the back-end subnet to connect to any serveron the front-end subnet (but not the reverse).

1. Deny all traffic rule: This deny rule should always be the final rule (in terms of priority), and as such if atraffic flow fails to match any of the preceding rules it is dropped by this rule. This rule is a default ruleand usually in-place and active. No modifications are usually needed to this rule.

On the second application traffic rule, to simplify this example, any port is allowed. In a real scenario, the most specific portand address ranges should be used to reduce the attack surface of this rule.

Once the previous rules are created, itâ€™s important to review the priority of each rule to ensure traffic is allowedor denied as desired. For this example, the rules are in priority order.

This example is a more complex but complete way of protecting and isolating the network than the previousexamples. (Example 2 protects just the application, and Example 1 just isolates subnets). This design allows formonitoring traffic in both directions, and protects not just the inbound application server but enforces networksecurity policy for all servers on this network. Also, depending on the appliance used, full traffic auditing andawareness can be achieved. For more information, see the detailed build instructions. These instructions include:

How to build this example perimeter network with classic PowerShell scripts.How to build this example with an Azure Resource Manager template.Detailed descriptions of each UDR, NSG command, and firewall rule.Detailed traffic flow scenarios, showing how traffic is allowed or denied in each layer.



NOTENOTE

Back to Fast start | Detailed build instructions available soon

Hybrid networking using a network virtual appliance (NVA) can be added to any of the perimeter network typesdescribed in examples 1, 2, or 3.

As shown in the previous figure, a VPN connection over the Internet (site-to-site) is used to connect an on-premises network to an Azure virtual network via an NVA.

If you use ExpressRoute with the Azure Public Peering option enabled, a static route should be created. This static routeshould route to the NVA VPN IP address out your corporate Internet and not via the ExpressRoute connection. The NATrequired on the ExpressRoute Azure Public Peering option can break the VPN session.

Once the VPN is in place, the NVA becomes the central hub for all networks and subnets. The firewall forwardingrules determine which traffic flows are allowed, are translated via NAT, are redirected, or are dropped (even fortraffic flows between the on-premises network and Azure).

Traffic flows should be considered carefully, as they can be optimized or degraded by this design pattern,depending on the specific use case.

Using the environment built in example 3, and then adding a site-to-site VPN hybrid network connection,produces the following design:


Example 5 Add a hybrid connection with a site-to-site, Azure VPN gatewayExample 5 Add a hybrid connection with a site-to-site, Azure VPN gateway

The on-premises router, or any other network device that is compatible with your NVA for VPN, would be the VPNclient. This physical device would be responsible for initiating and maintaining the VPN connection with your NVA.

Logically to the NVA, the network looks like four separate â€œsecurity zonesâ€ with the rules on the NVA beingthe primary director of traffic between these zones:

The addition of a site-to-site VPN hybrid network connection to an Azure virtual network can extend the on-premises network into Azure in a secure manner. In using a VPN connection, your traffic is encrypted and routesvia the Internet. The NVA in this example provides a central location to enforce and manage the security policy. Formore information, see the detailed build instructions (forthcoming). These instructions include:

How to build this example perimeter network with PowerShell scripts.How to build this example with an Azure Resource Manager template.Detailed traffic flow scenarios, showing how traffic flows through this design.



NOTENOTE

Hybrid networking using an Azure VPN gateway can be added to either perimeter network type described inexamples 1 or 2.

As shown in the preceding figure, a VPN connection over the Internet (site-to-site) is used to connect an on-premises network to an Azure virtual network via an Azure VPN gateway.

If you use ExpressRoute with the Azure Public Peering option enabled, a static route should be created. This static routeshould route to the NVA VPN IP address out your corporate Internet and not via the ExpressRoute WAN. The NAT requiredon the ExpressRoute Azure Public Peering option can break the VPN session.

The following figure shows the two network edges in this example. On the first edge, the NVA and NSGs controltraffic flows for intra-Azure networks and between Azure and the Internet. The second edge is the Azure VPNgateway, which is a separate and isolated network edge between on-premises and Azure.


Using the environment built in example 1, and then adding a site-to-site VPN hybrid network connection,produces the following design:


Example 6 Add a hybrid connection with ExpressRouteExample 6 Add a hybrid connection with ExpressRoute

The addition of a site-to-site VPN hybrid network connection to an Azure virtual network can extend the on-premises network into Azure in a secure manner. Using the native Azure VPN gateway, your traffic is IPSecencrypted and routes via the Internet. Also, using the Azure VPN gateway can provide a lower-cost option (noadditional licensing cost as with third-party NVAs). This option is most economical in example 1, where no NVA isused. For more information, see the detailed build instructions (forthcoming). These instructions include:




TIPTIP

Hybrid networking using an ExpressRoute private peering connection can be added to either perimeter networktype described in examples 1 or 2.

As shown in the preceding figure, ExpressRoute private peering provides a direct connection between your on-premises network and the Azure virtual network. Traffic transits only the service provider network and theMicrosoft Azure network, never touching the Internet.

Using ExpressRoute keeps corporate network traffic off the Internet. It also allows for service level agreements from yourExpressRoute provider. The Azure Gateway can pass up to 10 Gbps with ExpressRoute, whereas with site-to-site VPNs, theAzure Gateway maximum throughput is 200 Mbps.

As seen in the following diagram, with this option the environment now has two network edges. The NVA andNSG control traffic flows for intra-Azure networks and between Azure and the Internet, while the gateway is aseparate and isolated network edge between on-premises and Azure.


Using the environment built in example 1, and then adding an ExpressRoute hybrid network connection, producesthe following design:


ReferencesHelpful websites and documentationHelpful websites and documentation

The addition of an ExpressRoute Private Peering network connection can extend the on-premises network intoAzure in a secure, lower latency, higher performing manner. Also, using the native Azure Gateway, as in thisexample, provides a lower-cost option (no additional licensing as with third-party NVAs). For more information,see the detailed build instructions (forthcoming). These instructions include:


Access Azure with Azure Resource Manager :Accessing Azure with PowerShell: https://docs.microsoft.com/powershell/azureps-cmdlets-docs/Virtual networking documentation: https://docs.microsoft.com/azure/virtual-network/Network security group documentation: https://docs.microsoft.com/azure/virtual-network/virtual-networks-nsgUser-defined routing documentation: https://docs.microsoft.com/azure/virtual-network/virtual-networks-udr-overviewAzure virtual gateways: https://docs.microsoft.com/azure/vpn-gateway/Site-to-Site VPNs: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-create-site-to-site-rm-powershellExpressRoute documentation (be sure to check out the â€œGetting Startedâ€ and â€œHow Toâ€ sections):https://docs.microsoft.com/azure/expressroute/


https://docs.microsoft.com/azure/virtual-network/

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-nsg

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview

https://docs.microsoft.com/azure/vpn-gateway/


https://docs.microsoft.com/azure/expressroute/

Optimize ExpressRoute Routing6/27/2017 • 6 min to read • Edit Online

Suboptimal routing from customer to Microsoft

Solution: use BGP CommunitiesSolution: use BGP Communities

When you have multiple ExpressRoute circuits, you have more than one path to connect to Microsoft. As a result,suboptimal routing may happen - that is, your traffic may take a longer path to reach Microsoft, and Microsoft toyour network. The longer the network path, the higher the latency. Latency has direct impact on applicationperformance and user experience. This article will illustrate this problem and explain how to optimize routingusing the standard routing technologies.

Let's take a close look at the routing problem by an example. Imagine you have two offices in the US, one in LosAngeles and one in New York. Your offices are connected on a Wide Area Network (WAN), which can be eitheryour own backbone network or your service provider's IP VPN. You have two ExpressRoute circuits, one in USWest and one in US East, that are also connected on the WAN. Obviously, you have two paths to connect to theMicrosoft network. Now imagine you have Azure deployment (for example, Azure App Service) in both US Westand US East. Your intention is to connect your users in Los Angeles to Azure US West and your users in New Yorkto Azure US East because your service admin advertises that users in each office access the nearby Azure servicesfor optimal experiences. Unfortunately, the plan works out well for the east coast users but not for the west coastusers. The cause of the problem is the following. On each ExpressRoute circuit, we advertise to you both the prefixin Azure US East (23.100.0.0/16) and the prefix in Azure US West (13.100.0.0/16). If you don't know which prefixis from which region, you are not able to treat it differently. Your WAN network may think both of the prefixes arecloser to US East than US West and therefore route both office users to the ExpressRoute circuit in US East. In theend, you will have many unhappy users in the Los Angeles office.

To optimize routing for both office users, you need to know which prefix is from Azure US West and which fromAzure US East. We encode this information by using BGP Community values. We've assigned a unique BGPCommunity value to each Azure region, e.g. "12076:51004" for US East, "12076:51006" for US West. Now that

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-optimize-routing.md

NOTENOTE

Suboptimal routing from Microsoft to customer

you know which prefix is from which Azure region, you can configure which ExpressRoute circuit should bepreferred. Because we use the BGP to exchange routing info, you can use BGP's Local Preference to influencerouting. In our example, you can assign a higher local preference value to 13.100.0.0/16 in US West than in USEast, and similarly, a higher local preference value to 23.100.0.0/16 in US East than in US West. This configurationwill make sure that, when both paths to Microsoft are available, your users in Los Angeles will take theExpressRoute circuit in US West to connect to Azure US West whereas your users in New York take theExpressRoute in US East to Azure US East. Routing is optimized on both sides.

The same technique, using Local Preference, can be applied to routing from customer to Azure Virtual Network. We don'ttag BGP Community value to the prefixes advertised from Azure to your network. However, since you know which of yourVirtual Network deployment is close to which of your office, you can configure your routers accordingly to prefer oneExpressRoute circuit to another.

Here is another example where connections from Microsoft take a longer path to reach your network. In this case,you use on-premises Exchange servers and Exchange Online in a hybrid environment. Your offices are connectedto a WAN. You advertise the prefixes of your on-premises servers in both of your offices to Microsoft through thetwo ExpressRoute circuits. Exchange Online will initiate connections to the on-premises servers in cases such asmailbox migration. Unfortunately, the connection to your Los Angeles office is routed to the ExpressRoute circuitin US East before traversing the entire continent back to the west coast. The cause of the problem is similar to thefirst one. Without any hint, the Microsoft network can't tell which customer prefix is close to US East and whichone is close to US West. It happens to pick the wrong path to your office in Los Angeles.

https://technet.microsoft.com/library/jj200581%28v=exchg.150%29.aspx

Solution: use AS PATH prependingSolution: use AS PATH prepending

IMPORTANTIMPORTANT

There are two solutions to the problem. The first one is that you simply advertise your on-premises prefix for yourLos Angeles office, 177.2.0.0/31, on the ExpressRoute circuit in US West and your on-premises prefix for yourNew York office, 177.2.0.2/31, on the ExpressRoute circuit in US East. As a result, there is only one path forMicrosoft to connect to each of your offices. There is no ambiguity and routing is optimized. With this design, youneed to think about your failover strategy. In the event that the path to Microsoft via ExpressRoute is broken, youneed to make sure that Exchange Online can still connect to your on-premises servers.

The second solution is that you continue to advertise both of the prefixes on both ExpressRoute circuits, and inaddition you give us a hint of which prefix is close to which one of your offices. Because we support BGP AS Pathprepending, you can configure the AS Path for your prefix to influence routing. In this example, you can lengthenthe AS PATH for 172.2.0.0/31 in US East so that we will prefer the ExpressRoute circuit in US West for trafficdestined for this prefix (as our network will think the path to this prefix is shorter in the west). Similarly you canlengthen the AS PATH for 172.2.0.2/31 in US West so that we'll prefer the ExpressRoute circuit in US East.Routing is optimized for both offices. With this design, if one ExpressRoute circuit is broken, Exchange Online canstill reach you via another ExpressRoute circuit and your WAN.

We remove private AS numbers in the AS PATH for the prefixes received on Microsoft Peering. You need to append public ASnumbers in the AS PATH to influence routing for Microsoft Peering.

NOTENOTE

Suboptimal routing between virtual networks

While the examples given here are for Microsoft and Public peerings, we do support the same capabilities for the Privatepeering. Also, the AS Path prepending works within one single ExpressRoute circuit, to influence the selection of the primaryand secondary paths.

With ExpressRoute, you can enable Virtual Network to Virtual Network (which is also known as "VNet")communication by linking them to an ExpressRoute circuit. When you link them to multiple ExpressRoute circuits,suboptimal routing can happen between the VNets. Let's consider an example. You have two ExpressRoutecircuits, one in US West and one in US East. In each region, you have two VNets. Your web servers are deployedin one VNet and application servers in the other. For redundancy, you link the two VNets in each region to boththe local ExpressRoute circuit and the remote ExpressRoute circuit. As can be seen below, from each VNet thereare two paths to the other VNet. The VNets don't know which ExpressRoute circuit is local and which one isremote. Consequently as they do Equal-Cost-Multi-Path (ECMP) routing to load-balance inter-VNet traffic, sometraffic flows will take the longer path and get routed at the remote ExpressRoute circuit.

Solution: assign a high weight to local connectionSolution: assign a high weight to local connection

NOTENOTE

The solution is simple. Since you know where the VNets and the circuits are, you can tell us which path each VNetshould prefer. Specifically for this example, you assign a higher weight to the local connection than to the remoteconnection (see the configuration example here). When a VNet receives the prefix of the other VNet on multipleconnections it will prefer the connection with the highest weight to send traffic destined for that prefix.

You can also influence routing from VNet to your on-premises network, if you have multiple ExpressRoute circuits, byconfiguring the weight of a connection instead of applying AS PATH prepending, a technique described in the secondscenario above. For each prefix, we will always look at the connection weight before the AS Path length when deciding howto send traffic.

Asymmetric routing with multiple network paths6/27/2017 • 6 min to read • Edit Online

Multiple network paths

This article explains how forward and return network traffic might take different routes when multiple paths areavailable between network source and destination.

It's important to understand two concepts to understand asymmetric routing. One is the effect of multiple networkpaths. The other is how devices, like a firewall, keep state. These types of devices are called stateful devices. Acombination of these two factors creates scenarios in which network traffic is dropped by a stateful device becausethe stateful device didn't detect that traffic originated with the device itself.

When an enterprise network has only one link to the Internet through their Internet service provider, all traffic toand from the Internet travels the same path. Often, companies purchase multiple circuits, as redundant paths, toimprove network uptime. When this happens, it's possible that traffic that goes outside of the network, to theInternet, goes through one link, and the return traffic goes through a different link. This is commonly known asasymmetric routing. In asymmetric routing, reverse network traffic takes a different path from the original flow.

Although it primarily occurs on the Internet, asymmetric routing also applies to other combinations of multiplepaths. It applies, for example, both to an Internet path and a private path that go to the same destination, and tomultiple private paths that go to the same destination.

Each router along the way, from source to destination, computes the best path to reach a destination. The router'sdetermination of best possible path is based on two main factors:

Routing between external networks is based on a routing protocol, Border Gateway Protocol (BGP). BGP takesadvertisements from neighbors and runs them through a series of steps to determine the best path to theintended destination. It stores the best path in its routing table.The length of a subnet mask associated with a route influences routing paths. If a router receives multipleadvertisements for the same IP address but with different subnet masks, the router prefers the advertisementwith a longer subnet mask because it's considered a more specific route.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-asymmetric-routing.md

Stateful devices

Asymmetric routing with ExpressRoute

Routers look at the IP header of a packet for routing purposes. Some devices look even deeper inside the packet.Typically, these devices look at Layer4 (Transmission Control Protocol, or TCP; or User Datagram Protocol, orUDP), or even Layer7 (Application Layer) headers. These kinds of devices are either security devices or bandwidth-optimization devices.

A firewall is a common example of a stateful device. A firewall allows or denies a packet to pass through itsinterfaces based on various fields such as protocol, TCP/UDP port, and URL headers. This level of packetinspection puts a heavy processing load on the device. To improve performance, the firewall inspects the firstpacket of a flow. If it allows the packet to proceed, it keeps the flow information in its state table. All subsequentpackets related to this flow are allowed based on the initial determination. A packet that is part of an existing flowmight arrive at the firewall. If the firewall has no prior state information about it, the firewall drops the packet.

When you connect to Microsoft through Azure ExpressRoute, your network changes like this:

You have multiple links to Microsoft. One link is your existing Internet connection, and the other is viaExpressRoute. Some traffic to Microsoft might go through the Internet but come back via ExpressRoute, or viceversa.You receive more specific IP addresses via ExpressRoute. So, for traffic from your network to Microsoft forservices offered via ExpressRoute, routers always prefer ExpressRoute.

To understand the effect these two changes have on a network, let’s consider some scenarios. As an example, youhave only one circuit to the Internet and you consume all Microsoft services via the Internet. The traffic from yournetwork to Microsoft and back traverses the same Internet link and passes through the firewall. The firewallrecords the flow as it sees the first packet and return packets are allowed because the flow exists in the state table.

Then, you turn on ExpressRoute and consume services offered by Microsoft over ExpressRoute. All other servicesfrom Microsoft are consumed over the Internet. You deploy a separate firewall at your edge that is connected toExpressRoute. Microsoft advertises more specific prefixes to your network over ExpressRoute for specific services.Your routing infrastructure chooses ExpressRoute as the preferred path for those prefixes. If you are notadvertising your public IP addresses to Microsoft over ExpressRoute, Microsoft communicates with your public IP

Asymmetric routing solutions

RoutingRouting

Source-based NATSource-based NAT

addresses via the Internet. Forward traffic from your network to Microsoft uses ExpressRoute, and reverse trafficfrom Microsoft uses the Internet. When the firewall at the edge sees a response packet for a flow that it does notfind in the state table, it drops the return traffic.

If you choose to use the same network address translation (NAT) pool for ExpressRoute and for the Internet, you'llsee similar issues with the clients in your network on private IP addresses. Requests for services like WindowsUpdate go via the Internet because IP addresses for these services are not advertised via ExpressRoute. However,the return traffic comes back via ExpressRoute. If Microsoft receives an IP address with the same subnet maskfrom the Internet and ExpressRoute, it prefers ExpressRoute over the Internet. If a firewall or another statefuldevice that is on your network edge and facing ExpressRoute has no prior information about the flow, it drops thepackets that belong to that flow.

You have two main options to solve the problem of asymmetric routing. One is through routing, and the other isby using source-based NAT (SNAT).

Ensure that your public IP addresses are advertised to appropriate wide area network (WAN) links. For example, ifyou want to use the Internet for authentication traffic and ExpressRoute for your mail traffic, you should notadvertise your Active Directory Federation Services (AD FS) public IP addresses over ExpressRoute. Similarly, besure not to expose an on-premises AD FS server to IP addresses that the router receives over ExpressRoute.Routes received over ExpressRoute are more specific so they make ExpressRoute the preferred path forauthentication traffic to Microsoft. This causes asymmetric routing.

If you want to use ExpressRoute for authentication, make sure that you are advertising AD FS public IP addressesover ExpressRoute without NAT. This way, traffic that originates from Microsoft and goes to an on-premises AD FSserver goes over ExpressRoute. Return traffic from customer to Microsoft uses ExpressRoute because it's thepreferred route over the Internet.

Another way of solving asymmetric routing issues is by using SNAT. For example, you have not advertised thepublic IP address of an on-premises Simple Mail Transfer Protocol (SMTP) server over ExpressRoute because youintend to use the Internet for this type of communication. A request that originates with Microsoft and then goes toyour on-premises SMTP server traverses the Internet. You SNAT the incoming request to an internal IP address.Reverse traffic from the SMTP server goes to the edge firewall (which you use for NAT) instead of throughExpressRoute. The return traffic goes back via the Internet.

Asymmetric routing detectionTraceroute is the best way to make sure that your network traffic is traversing the expected path. If you expecttraffic from your on-premises SMTP server to Microsoft to take the Internet path, the expected traceroute is fromthe SMTP server to Office 365. The result validates that traffic is indeed leaving your network toward the Internetand not toward ExpressRoute.

ExpressRoute NAT requirements10/13/2017 • 4 min to read • Edit Online

NAT requirements for Microsoft peering

Traffic originating from your network destined to MicrosoftTraffic originating from your network destined to Microsoft

Traffic originating from Microsoft destined to your networkTraffic originating from Microsoft destined to your network

To connect to Microsoft cloud services using ExpressRoute, you’ll need to set up and manage NATs. Someconnectivity providers offer setting up and managing NAT as a managed service. Check with your connectivityprovider to see if they offer such a service. If not, you must adhere to the requirements described below.

Review the ExpressRoute circuits and routing domains page to get an overview of the various routing domains. Tomeet the public IP address requirements for Azure public and Microsoft peering, we recommend that you set upNAT between your network and Microsoft. This section provides a detailed description of the NAT infrastructureyou need to set up.

The Microsoft peering path lets you connect to Microsoft cloud services that are not supported through the Azurepublic peering path. The list of services includes Office 365 services, such as Exchange Online, SharePoint Online,Skype for Business, and Dynamics 365. Microsoft expects to support bi-directional connectivity on the Microsoftpeering. Traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before theyenter the Microsoft network. Traffic destined to your network from Microsoft cloud services must be SNATed atyour Internet edge to prevent asymmetric routing. The figure below provides a high-level picture of how the NATshould be setup for Microsoft peering.

You must ensure that traffic is entering the Microsoft peering path with a valid public IPv4 address. Microsoftmust be able to validate the owner of the IPv4 NAT address pool against the regional routing internet registry(RIR) or an internet routing registry (IRR). A check will be performed based on the AS number being peeredwith and the IP addresses used for the NAT. Refer to the ExpressRoute routing requirements page forinformation on routing registries.

IMPORTANTIMPORTANT

IP addresses used for the Azure public peering setup and other ExpressRoute circuits must not beadvertised to Microsoft through the BGP session. There is no restriction on the length of the NAT IP prefixadvertised through this peering.

The NAT IP pool advertised to Microsoft must not be advertised to the Internet. This will break connectivity to otherMicrosoft services.

Certain scenarios require Microsoft to initiate connectivity to service endpoints hosted within your network. Atypical example of the scenario would be connectivity to ADFS servers hosted in your network from Office

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-nat.md

NAT requirements for Azure public peering

IMPORTANTIMPORTANT

NAT IP pool and route advertisementsNAT IP pool and route advertisements

365. In such cases, you must leak appropriate prefixes from your network into the Microsoft peering.You must SNAT Microsoft traffic at the Internet edge for service endpoints within your network to preventasymmetric routing. Requests and replies with a destination IP that match a route received via ExpressRoutewill always be sent via ExpressRoute. Asymmetric routing exists if the request is received via the Internet withthe reply sent via ExpressRoute. SNATing the incoming Microsoft traffic at the Internet edge forces reply trafficback to the Internet edge, resolving the problem.

The Azure public peering path enables you to connect to all services hosted in Azure over their public IPaddresses. These include services listed in the ExpessRoute FAQ and any services hosted by ISVs on MicrosoftAzure.

Connectivity to Microsoft Azure services on public peering is always initiated from your network into the Microsoft network.Therefore, sessions cannot be initiated from Microsoft Azure services to your network over ExpressRoute. If attempted,packets sent to these advertised IPs will use the internet instead of ExpressRoute.

Traffic destined to Microsoft Azure on public peering must be SNATed to valid public IPv4 addresses before theyenter the Microsoft network. The figure below provides a high-level picture of how the NAT could be set up tomeet the above requirement.

You must ensure that traffic is entering the Azure public peering path with valid public IPv4 address. Microsoft

IMPORTANTIMPORTANT

Next steps

must be able to validate the ownership of the IPv4 NAT address pool against a regional routing Internet registry(RIR) or an Internet routing registry (IRR). A check will be performed based on the AS number being peered withand the IP addresses used for the NAT. Refer to the ExpressRoute routing requirements page for information onrouting registries.

There are no restrictions on the length of the NAT IP prefix advertised through this peering. You must monitor theNAT pool and ensure that you are not starved of NAT sessions.

The NAT IP pool advertised to Microsoft must not be advertised to the Internet. This will break connectivity to otherMicrosoft services.

Refer to the requirements for Routing and QoS.For workflow information, see ExpressRoute circuit provisioning workflows and circuit states.Configure your ExpressRoute connection.


Verifying ExpressRoute connectivity3/20/2018 • 12 min to read • Edit Online

IMPORTANTIMPORTANT

Overview

ExpressRoute, which extends an on-premises network into the Microsoft cloud over a private connection that isfacilitated by a connectivity provider, involves the following three distinct network zones:

Customer NetworkProvider NetworkMicrosoft Datacenter

The purpose of this document is to help user to identify where (or even if) a connectivity issue exists and withinwhich zone, thereby to seek help from appropriate team to resolve the issue. If Microsoft support is needed toresolve an issue, open a support ticket with Microsoft Support.

This document is intended to help diagnosing and fixing simple issues. It is not intended to be a replacement for Microsoftsupport. Open a support ticket with Microsoft Support if you are unable to solve the problem using the guidance provided.

The following diagram shows the logical connectivity of a customer network to Microsoft network usingExpressRoute.

In the preceding diagram, the numbers indicate key network points. The network points are referenced oftenthrough this article by their associated number.

Depending on the ExpressRoute connectivity model (Cloud Exchange Co-location, Point-to-Point EthernetConnection, or Any-to-any (IPVPN)) the network points 3 and 4 may be switches (Layer 2 devices). The keynetwork points illustrated are as follows:

1. Customer compute device (for example, a server or PC)2. CEs: Customer edge routers3. PEs (CE facing): Provider edge routers/switches that are facing customer edge routers. Referred to as PE-CEs in

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-troubleshooting-expressroute-overview.md



NOTENOTE

Validate circuit provisioning and state

TIPTIP

Verification via the Azure portalVerification via the Azure portal

this document.4. PEs (MSEE facing): Provider edge routers/switches that are facing MSEEs. Referred to as PE-MSEEs in this

document.5. MSEEs: Microsoft Enterprise Edge (MSEE) ExpressRoute routers6. Virtual Network (VNet) Gateway7. Compute device on the Azure VNet

If the Cloud Exchange Co-location or Point-to-Point Ethernet Connection connectivity models are used, thecustomer edge router (2) would establish BGP peering with MSEEs (5). Network points 3 and 4 would still exist butbe somewhat transparent as Layer 2 devices.

If the Any-to-any (IPVPN) connectivity model is used, the PEs (MSEE facing) (4) would establish BGP peering withMSEEs (5). Routes would then propagate back to the customer network via the IPVPN service provider network.

For ExpressRoute high availability, Microsoft requires a redundant pair of BGP sessions between MSEEs (5) and PE-MSEEs (4).A redundant pair of network paths is also encouraged between customer network and PE-CEs. However, in Any-to-any(IPVPN) connection model, a single CE device (2) may be connected to one or more PEs (3).

To validate an ExpressRoute circuit, the following steps are covered (with the network point indicated by theassociated number):

1. Validate circuit provisioning and state (5)2. Validate at least one ExpressRoute peering is configured (5)3. Validate ARP between Microsoft and the service provider (link between 4 and 5)4. Validate BGP and routes on the MSEE (BGP between 4 to 5, and 5 to 6 if a VNet is connected)5. Check the Traffic Statistics (Traffic passing through 5)

More validations and checks will be added in the future, check back monthly!

Regardless of the connectivity model, an ExpressRoute circuit has to be created and thus a service key generatedfor circuit provisioning. Provisioning an ExpressRoute circuit establishes a redundant Layer 2 connections betweenPE-MSEEs (4) and MSEEs (5). For more information on how to create, modify, provision, and verify anExpressRoute circuit, see the article Create and modify an ExpressRoute circuit.

A service key uniquely identifies an ExpressRoute circuit. This key is required for most of the powershell commands mentionedin this document. Also, should you need assistance from Microsoft or from an ExpressRoute partner to troubleshoot anExpressRoute issue, provide the service key to readily identify the circuit.

In the Azure portal, the status of an ExpressRoute circuit can be checked by selecting on the left-side-bar menu and then selecting the ExpressRoute circuit. Selecting an ExpressRoute circuit listed under "Allresources" opens the ExpressRoute circuit blade. In the section of the blade, the ExpressRoute essentialsare listed as shown in the following screen shot:

https://docs.microsoft.com/azure/expressroute/expressroute-howto-circuit-portal-resource-manager

NOTENOTE

Verification via PowerShellVerification via PowerShell

Get-AzureRmExpressRouteCircuit -ResourceGroupName "Test-ER-RG"

TIPTIP

Get-AzureRmExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"

In the ExpressRoute Essentials, Circuit status indicates the status of the circuit on the Microsoft side. Provider statusindicates if the circuit has been Provisioned/Not provisioned on the service-provider side.

For an ExpressRoute circuit to be operational, the Circuit status must be Enabled and the Provider status must beProvisioned.

If the Circuit status is not enabled, contact Microsoft Support. If the Provider status is not provisioned, contact your serviceprovider.

To list all the ExpressRoute circuits in a Resource Group, use the following command:

You can get your resource group name through the Azure . See the previous subsection of this document and note that theresource group name is listed in the example screen shot.

To select a particular ExpressRoute circuit in a Resource Group, use the following command:

A sample response is:


Name : Test-ER-CktResourceGroupName : Test-ER-RGLocation : westus2Id : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-CktEtag : W/"################################"ProvisioningState : SucceededSku : { "Name": "Standard_UnlimitedData", "Tier": "Standard", "Family": "UnlimitedData" }CircuitProvisioningState : EnabledServiceProviderProvisioningState : ProvisionedServiceProviderNotes : ServiceProviderProperties : { "ServiceProviderName": "****", "PeeringLocation": "******", "BandwidthInMbps": 100 }ServiceKey : **************************************Peerings : []Authorizations : []

CircuitProvisioningState : EnabledServiceProviderProvisioningState : Provisioned

NOTENOTE

Verification via PowerShell (Classic)Verification via PowerShell (Classic)


Get-AzureDedicatedCircuit -ServiceKey **************************************

andwidth : 100BillingType : UnlimitedDataCircuitName : Test-ER-CktLocation : westus2ServiceKey : **************************************ServiceProviderName : ****ServiceProviderProvisioningState : ProvisionedSku : StandardStatus : Enabled

To confirm if an ExpressRoute circuit is operational, pay particular attention to the following fields:

If the CircuitProvisioningState is not enabled, contact Microsoft Support. If the ServiceProviderProvisioningState is notprovisioned, contact your service provider.

To list all the ExpressRoute circuits under a subscription, use the following command:

To select a particular ExpressRoute circuit, use the following command:

A sample response is:

To confirm if an ExpressRoute circuit is operational, pay particular attention to the following fields:


NOTENOTE

Validate Peering Configuration

Verification via the Azure portalVerification via the Azure portal

NOTENOTE

NOTENOTE

Verification via PowerShellVerification via PowerShell


If the Status is not enabled, contact Microsoft Support. If the ServiceProviderProvisioningState is not provisioned, contactyour service provider.

After the service provider has completed the provisioning the ExpressRoute circuit, a routing configuration can becreated over the ExpressRoute circuit between MSEE-PRs (4) and MSEEs (5). Each ExpressRoute circuit can haveone, two, or three routing contexts enabled: Azure private peering (traffic to private virtual networks in Azure),Azure public peering (traffic to public IP addresses in Azure), and Microsoft peering (traffic to Office 365 andDynamics 365). For more information on how to create and modify routing configuration, see the article Createand modify routing for an ExpressRoute circuit.

If layer 3 is provided by the service provider and the peerings are blank in the portal, refresh the Circuit configuration usingthe refresh button on the protal. This operation will apply the right routing configuration on your circuit.

In the Azure portal, status of an ExpressRoute circuit can be checked by selecting on the left-side-barmenu and then selecting the ExpressRoute circuit. Selecting an ExpressRoute circuit listed under "All resources"would open the ExpressRoute circuit blade. In the section of the blade, the ExpressRoute essentials wouldbe listed as shown in the following screen shot:

In the preceding example, as noted Azure private peering routing context is enabled, whereas Azure public andMicrosoft peering routing contexts are not enabled. A successfully enabled peering context would also have theprimary and secondary point-to-point (required for BGP) subnets listed. The /30 subnets are used for the interfaceIP address of the MSEEs and PE-MSEEs.

If a peering is not enabled, check if the primary and secondary subnets assigned match the configuration on PE-MSEEs. Ifnot, to change the configuration on MSEE routers, refer to Create and modify routing for an ExpressRoute circuit


https://docs.microsoft.com/azure/expressroute/expressroute-howto-routing-portal-resource-manager


$ckt = Get-AzureRmExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt

Name : AzurePrivatePeeringId : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-Ckt/peerings/AzurePrivatePeeringEtag : W/"################################"PeeringType : AzurePrivatePeeringAzureASN : 12076PeerASN : ####PrimaryPeerAddressPrefix : 172.16.0.0/30SecondaryPeerAddressPrefix : 172.16.0.4/30PrimaryAzurePort : SecondaryAzurePort : SharedKey : VlanId : 300MicrosoftPeeringConfig : nullProvisioningState : Succeeded

$ckt = Get-AzureRmExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering" -ExpressRouteCircuit $ckt

$ckt = Get-AzureRmExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt" Get-AzureRmExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt

Get-AzureRmExpressRouteCircuitPeeringConfig : Sequence contains no matching elementAt line:1 char:1 + Get-AzureRmExpressRouteCircuitPeeringConfig -Name "AzurePublicPeering ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Get-AzureRmExpr...itPeeringConfig], InvalidOperationException + FullyQualifiedErrorId : Microsoft.Azure.Commands.Network.GetAzureExpressRouteCircuitPeeringConfigCommand

NOTENOTE

Verification via PowerShell (Classic)Verification via PowerShell (Classic)

To get the Azure private peering configuration details, use the following commands:

A sample response, for a successfully configured private peering, is:

A successfully enabled peering context would have the primary and secondary address prefixes listed. The /30subnets are used for the interface IP address of the MSEEs and PE-MSEEs.

To get the Azure public peering configuration details, use the following commands:

To get the Microsoft peering configuration details, use the following commands:

If a peering is not configured, there would be an error message. A sample response, when the stated peering(Azure Public peering in this example) is not configured within the circuit:

If a peering is not enabled, check if the primary and secondary subnets assigned match the configuration on the linked PE-MSEE. Also check if the correct VlanId, AzureASN, and PeerASN are used on MSEEs and if these values maps to the onesused on the linked PE-MSEE. If MD5 hashing is chosen, the shared key should be same on MSEE and PE-MSEE pair. Tochange the configuration on the MSEE routers, refer to Create and modify routing for an ExpressRoute circuit.


Get-AzureBGPPeering -AccessType Private -ServiceKey "*********************************"

AdvertisedPublicPrefixes : AdvertisedPublicPrefixesState : ConfiguredAzureAsn : 12076CustomerAutonomousSystemNumber : PeerAsn : ####PrimaryAzurePort : PrimaryPeerSubnet : 10.0.0.0/30RoutingRegistryName : SecondaryAzurePort : SecondaryPeerSubnet : 10.0.0.4/30State : EnabledVlanId : 100

Get-AzureBGPPeering -AccessType Public -ServiceKey "*********************************"

Get-AzureBGPPeering -AccessType Microsoft -ServiceKey "*********************************"

IMPORTANTIMPORTANT

NOTENOTE

Validate ARP between Microsoft and the service provider

To get the Azure private peering configuration details, use the following command:

A sample response, for a successfully configured private peering is:

A successfully, enabled peering context would have the primary and secondary peer subnets listed. The /30subnets are used for the interface IP address of the MSEEs and PE-MSEEs.

To get the Azure public peering configuration details, use the following commands:

To get the Microsoft peering configuration details, use the following commands:

If layer 3 peerings were set by the service provider, setting the ExpressRoute peerings via the portal or PowerShell overwritesthe service provider settings. Resetting the provider side peering settings requires the support of the service provider. Onlymodify the ExpressRoute peerings if it is certain that the service provider is providing layer 2 services only!

If a peering is not enabled, check if the primary and secondary peer subnets assigned match the configuration on the linkedPE-MSEE. Also check if the correct VlanId, AzureAsn, and PeerAsn are used on MSEEs and if these values maps to the onesused on the linked PE-MSEE. To change the configuration on the MSEE routers, refer to Create and modify routing for anExpressRoute circuit.

This section uses PowerShell (Classic) commands. If you have been using PowerShell Azure Resource Managercommands, ensure that you have admin/co-admin access to the subscription. For troubleshooting using AzureResource Manager commands please refer to the Getting ARP tables in the Resource Manager deployment modeldocument.


https://docs.microsoft.com/azure/expressroute/expressroute-troubleshooting-arp-resource-manager

NOTENOTE

Get-AzureDedicatedCircuitPeeringArpInfo -AccessType Private -Path Primary -ServiceKey "*********************************"

ARP Info:

Age Interface IpAddress MacAddress 113 On-Prem 10.0.0.1 e8ed.f335.4ca9 0 Microsoft 10.0.0.2 7c0e.ce85.4fc9

ARP Info:

NOTENOTE

Validate BGP and routes on the MSEE

NOTENOTE

Get-AzureDedicatedCircuitPeeringRouteTableSummary -AccessType Private -Path Primary -ServiceKey "*********************************"

To get ARP, both the Azure portal and Azure Resource Manager PowerShell commands can be used. If errors are encounteredwith the Azure Resource Manager PowerShell commands, classic PowerShell commands should work as Classic PowerShellcommands also work with Azure Resource Manager ExpressRoute circuits.

To get the ARP table from the primary MSEE router for the private peering, use the following command:

An example response for the command, in the successful scenario:

Similarly, you can check the ARP table from the MSEE in the Primary/Secondary path, for Private/Public/Microsoftpeerings.

The following example shows the response of the command for a peering does not exist.

If the ARP table does not have IP addresses of the interfaces mapped to MAC addresses, review the following information:

1. If the first IP address of the /30 subnet assigned for the link between the MSEE-PR and MSEE is used on the interface ofMSEE-PR. Azure always uses the second IP address for MSEEs.

2. Verify if the customer (C-Tag) and service (S-Tag) VLAN tags match both on MSEE-PR and MSEE pair.

This section uses PowerShell (Classic) commands. If you have been using PowerShell Azure Resource Managercommands, ensure that you have admin/co-admin access to the subscription.

To get BGP information, both the Azure portal and Azure Resource Manager PowerShell commands can be used. If errors areencountered with the Azure Resource Manager PowerShell commands, classic PowerShell commands should work as classicPowerShell commands also work with Azure Resource Manager ExpressRoute circuits.

To get the routing table (BGP neighbor) summary for a particular routing context, use the following command:

An example response is:

Route Table Summary:

Neighbor V AS UpDown StatePfxRcd 10.0.0.1 4 #### 8w4d 50

NOTENOTE

NOTENOTE

Get-AzureDedicatedCircuitPeeringRouteTableInfo -AccessType Private -Path Primary -ServiceKey "*********************************"

Route Table Info:

Network NextHop LocPrf Weight Path 10.1.0.0/16 10.0.0.1 0 #### ##### ##### 10.2.0.0/16 10.0.0.1 0 #### ##### #####...

Route Table Info:

Check the Traffic Statistics

Get-AzureDedicatedCircuitStats -ServiceKey 97f85950-01dd-4d30-a73c-bf683b3a6e5c -AccessType Private

As shown in the preceding example, the command is useful to determine for how long the routing context has beenestablished. It also indicates number of route prefixes advertised by the peering router.

If the state is in Active or Idle, check if the primary and secondary peer subnets assigned match the configuration on thelinked PE-MSEE. Also check if the correct VlanId, AzureAsn, and PeerAsn are used on MSEEs and if these values maps to theones used on the linked PE-MSEE. If MD5 hashing is chosen, the shared key should be same on MSEE and PE-MSEE pair. Tochange the configuration on the MSEE routers, refer to Create and modify routing for an ExpressRoute circuit.

If certain destinations are not reachable over a particular peering, check the route table of the MSEEs belonging to theparticular peering context. If a matching prefix (could be NATed IP) is present in the routing table, then check if there arefirewalls/NSG/ACLs on the path and if they permit the traffic.

To get the full routing table from MSEE on the Primary path for the particular Private routing context, use thefollowing command:

An example successful outcome for the command is:

Similarly, you can check the routing table from the MSEE in the Primary/Secondary path, forPrivate/Public/Microsoft a peering context.

The following example shows the response of the command for a peering does not exist:

To get the combined primary and secondary path traffic statistics--bytes in and out--of a peering context, use thefollowing command:

A sample output of the command is:


PrimaryBytesIn PrimaryBytesOut SecondaryBytesIn SecondaryBytesOut-------------- --------------- ---------------- ----------------- 240780020 239863857 240565035 239628474

Get-AzureDedicatedCircuitStats : ResourceNotFound: Can not find any subinterface for peering type 'Public' for circuit '97f85950-01dd-4d30-a73c-bf683b3a6e5c' .At line:1 char:1+ Get-AzureDedicatedCircuitStats -ServiceKey 97f85950-01dd-4d30-a73c-bf ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Get-AzureDedicatedCircuitStats], CloudException + FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.ExpressRoute.GetAzureDedicatedCircuitPeeringStatsCommand

Next Steps

A sample output of the command for a non-existent peering is:

For more information or help, check out the following links:

Microsoft SupportCreate and modify an ExpressRoute circuitCreate and modify routing for an ExpressRoute circuit


https://docs.microsoft.com/azure/expressroute/expressroute-howto-circuit-portal-resource-manager


Troubleshooting network performance12/22/2017 • 15 min to read • Edit Online

Overview

NOTENOTE

Network components

Azure provides stable and fast ways to connect from your on-premises network to Azure. Methods like Site-to-SiteVPN and ExpressRoute are successfully used by customers large and small to run their businesses in Azure. Butwhat happens when performance doesn't meet your expectation or previous experience? This document can helpstandardize the way you test and baseline your specific environment.

This document shows how you can easily and consistently test network latency and bandwidth between two hosts.This document also provides some advice on ways to look at the Azure network and help to isolate problem points.The PowerShell script and tools discussed require two hosts on the network (at either end of the link being tested).One host must be a Windows Server or Desktop, the other can be either Windows or Linux.

The approach to troubleshooting, the tools, and methods used are personal preferences. This document describes theapproach and tools I often take. Your approach will probably differ, there's nothing wrong with different approaches toproblem solving. However, if you don't have an established approach, this document can get you started on the path tobuilding your own methods, tools, and preferences to troubleshooting network issues.

Before digging into troubleshooting, let's discuss some common terms and components. This discussion ensureswe're thinking about each component in the end-to-end chain that enables connectivity in Azure.

At the highest level, I describe three major network routing domains;

the Azure network (blue cloud on the right)the Internet or WAN (green cloud in the center)the Corporate Network (peach cloud on the left)

Looking at the diagram from right to left, let's discuss briefly each component:

Virtual Machine - The server may have multiple NICs, ensure any static routes, default routes, and OperatingSystem settings are sending and receiving traffic the way you think it is. Also, each VM SKU has a bandwidthrestriction. If you're using a smaller VM SKU, your traffic is limited by the bandwidth available to the NIC. Iusually use a DS5v2 for testing (and then delete once done with testing to save money) to ensure adequatebandwidth at the VM.NIC - Ensure you know the private IP that is assigned to the NIC in question.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-troubleshooting-network-performance.md

Tools

AzureCT - the Azure Connectivity ToolkitAzureCT - the Azure Connectivity Toolkit

NIC NSG - There may be specific NSGs applied at the NIC level, ensure the NSG rule-set is appropriate for thetraffic you're trying to pass. For example, ensure ports 5201 for iPerf, 3389 for RDP, or 22 for SSH are open toallow test traffic to pass.VNet Subnet - The NIC is assigned to a specific subnet, ensure you know which one and the rules associatedwith that subnet.Subnet NSG - Just like the NIC, NSGs can be applied at the subnet as well. Ensure the NSG rule-set isappropriate for the traffic you're trying to pass. (for traffic inbound to the NIC the subnet NSG applies first, thenthe NIC NSG, conversely for traffic outbound from the VM the NIC NSG applies first then the Subnet NSGcomes into play).Subnet UDR - User Defined Routes can direct traffic to an intermediate hop (like a firewall or load-balancer).Ensure you know if there is a UDR in place for your traffic and if so where it goes and what that next hop will doto your traffic. (for example, a firewall could pass some traffic and deny other traffic between the same twohosts).Gateway subnet / NSG / UDR - Just like the VM subnet, the gateway subnet can have NSGs and UDRs.Make sure you know if they are there and what effects they have on your traffic.VNet Gateway (ExpressRoute) - Once peering (ExpressRoute) or VPN is enabled, there aren't many settingsthat can affect how or if traffic routes. If you have multiple ExpressRoute circuits or VPN tunnels connected tothe same VNet Gateway, you should be aware of the connection weight settings as this setting affectsconnection preference and affects the path your traffic takes.Route Filter (Not shown) - A route filter only applies to Microsoft Peering on ExpressRoute, but is critical tocheck if you're not seeing the routes you expect on Microsoft Peering.

At this point, you're on the WAN portion of the link. This routing domain can be your service provider, yourcorporate WAN, or the Internet. Many hops, technologies, and companies involved with these links can make itsomewhat difficult to troubleshoot. Often, you work to rule out both Azure and your Corporate Networks firstbefore jumping into this collection of companies and hops.

In the preceding diagram, on the far left is your corporate network. Depending on the size of your company, thisrouting domain can be a few network devices between you and the WAN or multiple layers of devices in acampus/enterprise network.

Given the complexities of these three different high-level network environments, it's often optimal to start at theedges and try to show where performance is good, and where it degrades. This approach can help identify theproblem routing domain of the three and then focus your troubleshooting on that specific environment.

Most network issues can be analyzed and isolated using basic tools like ping and traceroute. It's rare that you needto go as deep as a packet analysis like Wireshark. To help with troubleshooting, the Azure Connectivity Toolkit(AzureCT) was developed to put some of these tools in an easy package. For performance testing, I like to use iPerfand PSPing. iPerf is a commonly used tool and runs on most operating systems. iPerf is good for basicperformances tests and is fairly easy to use. PSPing is a ping tool developed by SysInternals. PSPing is an easyway to perform ICMP and TCP pings in one also easy to use command. Both of these tools are lightweight and are"installed" simply by coping the files to a directory on the host.

I've wrapped all of these tools and methods into a PowerShell module (AzureCT) that you can install and use.

The AzureCT PowerShell module has two components Availability Testing and Performance Testing. Thisdocument is only concerned with Performance testing, so lets focus on the two Link Performance commands in thisPowerShell module.

There are three basic steps to use this toolkit for Performance testing. 1) Install the PowerShell module, 2) Installthe supporting applications iPerf and PSPing 3) Run the performance test.

https://github.com/Azure/NetworkMonitoring/blob/master/AzureCT/AvailabilityTesting.md

https://github.com/Azure/NetworkMonitoring/blob/master/AzureCT/PerformanceTesting.md

Troubleshooting

(new-object Net.WebClient).DownloadString("https://aka.ms/AzureCT") | Invoke-Expression

Install-LinkPerformance

Get-LinkPerformance -RemoteHost 10.0.0.1 -TestSeconds 10

1. Installing the PowerShell Module

This command downloads the PowerShell module and installs it locally.

2. Install the supporting applications

This AzureCT command installs iPerf and PSPing in a new directory "C:\ACTTools", it also opens theWindows Firewall ports to allow ICMP and port 5201 (iPerf) traffic.

3. Run the performance test

First, on the remote host you must install and run iPerf in server mode. Also ensure the remote host islistening on either 3389 (RDP for Windows) or 22 (SSH for Linux) and allowing traffic on port 5201 foriPerf. If the remote host is windows, install the AzureCT and run the Install-LinkPerformance command toset up iPerf and the firewall rules needed to start iPerf in server mode successfully.

Once the remote machine is ready, open PowerShell on the local machine and start the test:

This command runs a series of concurrent load and latency tests to help estimate the bandwidth capacityand latency of your network link.

4. Review the output of the tests

The PowerShell output format looks similar to:

The detailed results of all the iPerf and PSPing tests are in individual text files in the AzureCT tools directoryat "C:\ACTTools."

If the performance test is not giving you expected results, figuring out why should be a progressive step-by-stepprocess. Given the number of components in the path, a systematic approach generally provides a faster path toresolution than jumping around and potentially needlessly doing the same testing multiple times.

NOTENOTE

Advanced ExpressRoute troubleshooting

The scenario here is a performance issue, not a connectivity issue. The steps would be different if traffic wasn't passing at all.

First, challenge your assumptions. Is your expectation reasonable? For instance, if you have a 1-Gbps ExpressRoutecircuit and 100 ms of latency it's unreasonable to expect the full 1 Gbps of traffic given the performancecharacteristics of TCP over high latency links. See the References section for more on performance assumptions.

Next, I recommend starting at the edges between routing domains and try to isolate the problem to a single majorrouting domain; the Corporate Network, the WAN, or the Azure Network. People often blame the "black box" inthe path, while blaming the black box is easy to do, it may significantly delay resolution especially if the problem isactually in an area that you have the ability to make changes. Make sure you do your due diligence before handingoff to your service provider or ISP.

Once you've identified the major routing domain that appears to contain the problem, you should create a diagramof the area in question. Either on a whiteboard, notepad, or Visio as a diagram provides a concrete "battle map" toallow a methodical approach to further isolate the problem. You can plan testing points, and update the map as youclear areas or dig deeper as the testing progresses.

Now that you have a diagram, start to divide the network into segments and narrow the problem down. Find outwhere it works and where it doesn't. Keep moving your testing points to isolate down to the offending component.

Also, don't forget to look at other layers of the OSI model. It's easy to focus on the network and layers 1 - 3(Physical, Data, and Network layers) but the problems can also be up at Layer 7 in the application layer. Keep anopen mind and verify assumptions.

If you're not sure where the edge of the cloud actually is, isolating the Azure components can be a challenge. WhenExpressRoute is used, the edge is a network component called the Microsoft Enterprise Edge (MSEE). When usingExpressRoute, the MSEE is the first point of contact into Microsoft's network, and the last hop leaving theMicrosoft network. When you create a connection object between your VNet gateway and the ExpressRoute circuit,you're actually making a connection to the MSEE. Recognizing the MSEE as the first or last hop (depending onwhich direction you're going) is crucial to isolating Azure Network problems to either prove the issue is in Azure orfurther downstream in the WAN or the Corporate Network.

NOTENOTE

Test planTest plan

IMPORTANTIMPORTANT

The problem is isolated, now what?

ReferencesLatency/bandwidth expectationsLatency/bandwidth expectations

Notice that the MSEE isn't in the Azure cloud. ExpressRoute is actually at the edge of the Microsoft network not actually inAzure. Once you're connected with ExpressRoute to an MSEE, you're connected to Microsoft's network, from there you canthen go to any of the cloud services, like Office 365 (with Microsoft Peering) or Azure (with Private and/or Microsoft Peering).

If two VNets (VNets A and B in the diagram) are connected to the same ExpressRoute circuit, you can perform aseries of tests to isolate the problem in Azure (or prove it's not in Azure)

1. Run the Get-LinkPerformance test between VM1 and VM2. This test provides insight to if the problem is localor not. If this test produces acceptable latency and bandwidth results, you can mark the local VNet network asgood.

2. Assuming the local VNet traffic is good, run the Get-LinkPerformance test between VM1 and VM3. This testexercises the connection through the Microsoft network down to the MSEE and back into Azure. If this testproduces acceptable latency and bandwidth results, you can mark the Azure network as good.

3. If Azure is ruled out, you can perform a similar sequence of tests on your Corporate Network. If that also testswell, it's time to work with your service provider or ISP to diagnose your WAN connection. Example: Run thistest between two branch offices, or between your desk and a data center server. Depending on what you'retesting, find endpoints (servers, PCs, etc.) that can exercise that path.

It's critical that for each test you mark the time of day you run the test and record the results in a common location (I likeOneNote or Excel). Each test run should have identical output so you can compare the resultant data across test runs and nothave "holes" in the data. Consistency across multiple tests is the primary reason I use the AzureCT for troubleshooting. Themagic isn't in the exact load scenarios I run, but instead the magic is the fact that I get a consistent test and data outputfrom each and every test. Recording the time and having consistent data every single time is especially helpful if you later findthat the issue is sporadic. Be diligent with your data collection up front and you'll avoid hours of retesting the same scenarios(I learned this hard way many years ago).

The more you can isolate the problem the easier it is to fix, however often you reach the point where you can't godeeper or further with your troubleshooting. Example: you see the link across your service provider taking hopsthrough Europe, but your expected path is all in Asia. This point is when you should reach out for help. Who youask is dependent on the routing domain you isolated the issue to, or even better if you are able to narrow it downto a specific component.

For corporate network issues, your internal IT department or service provider supporting your network (whichmay be the hardware manufacturer) may be able to help with device configuration or hardware repair.

For the WAN, sharing your testing results with your Service Provider or ISP may help get them started and avoidcovering some of the same ground you've tested already. However, don't be offended if they want to verify yourresults themselves. "Trust but verify" is a good motto when troubleshooting based on other people's reportedresults.

With Azure, once you isolate the issue in as much detail as you're able, it's time to review the Azure NetworkDocumentation and then if still needed open a support ticket.

https://docs.microsoft.com/azure/index#pivot=services&panel=network

https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/overview

TIPTIP

Latency/bandwidth resultsLatency/bandwidth results

IMPORTANTIMPORTANT

ExpressRouteLocation

AzureRegion

EstimatedDistance (km)

Latency 1 SessionBandwidth

MaximumBandwidth

Geographic latency (miles or kilometers) between the end points you're testing is by far the largest component of latency.While there is equipment latency (physical and virtual components, number of hops, etc.) involved, geography has proven tobe the largest component of overall latency when dealing with WAN connections. It's also important to note that the distanceis the distance of the fiber run not the straight-line or road map distance. This distance is incredibly hard to get with anyaccuracy. As a result, I generally use a city distance calculator on the internet and know that this method is a grosslyinaccurate measure, but is enough to set a general expectation.

I've got an ExpressRoute setup in Seattle, Washington in the USA. The following table shows the latency andbandwidth I saw testing to various Azure locations. I've estimated the geographic distance between each end of thetest.

Test setup:

A physical server running Windows Server 2016 with a 10 Gbps NIC, connected to an ExpressRoute circuit.A 10Gbps Premium ExpressRoute circuit in the location identified with Private Peering enabled.An Azure VNet with an UltraPerformance gateway in the specified region.A DS5v2 VM running Windows Server 2016 on the VNet. The VM was non-domain joined, built from thedefault Azure image (no optimization or customization) with AzureCT installed.

Get-LinkPerformance -RemoteHost 10.0.0.1 -TestSeconds 300

The data flow for each test had the load flowing from the on-premises physical server (iPerf client in Seattle) upto the Azure VM (iPerf server in the listed Azure region).The "Latency" column data is from the No Load test (a TCP latency test without iPerf running).The "Max Bandwidth" column data is from the 16 TCP flow load test with a 1-Mb window size.

All testing was using the AzureCT Get-LinkPerformance command with a 5-minute load test for each of thesix test runs. For example:

These numbers are for general reference only. Many factors affect latency, and while these values are generally consistentover time, conditions within Azure or the Service Providers network can send traffic via different paths at any time, thuslatency and bandwidth can be affected. Generally, the effects of these changes don't result in significant differences.

Seattle West US 2 191 km 5 ms 262.0 Mbits/sec 3.74 Gbits/sec

Seattle West US 1,094 km 18 ms 82.3 Mbits/sec 3.70 Gbits/sec

Seattle Central US 2,357 km 40 ms 38.8 Mbits/sec 2.55 Gbits/sec

Seattle South Central US 2,877 km 51 ms 30.6 Mbits/sec 2.49 Gbits/sec

Seattle North Central US 2,792 km 55 ms 27.7 Mbits/sec 2.19 Gbits/sec

Seattle East US 2 3,769 km 73 ms 21.3 Mbits/sec 1.79 Gbits/sec

Seattle East US 3,699 km 74 ms 21.1 Mbits/sec 1.78 Gbits/sec

Seattle Japan East 7,705 km 106 ms 14.6 Mbits/sec 1.22 Gbits/sec

Seattle UK South 7,708 km 146 ms 10.6 Mbits/sec 896 Mbits/sec

Seattle West Europe 7,834 km 153 ms 10.2 Mbits/sec 761 Mbits/sec

Seattle Australia East 12,484 km 165 ms 9.4 Mbits/sec 794 Mbits/sec

Seattle Southeast Asia 12,989 km 170 ms 9.2 Mbits/sec 756 Mbits/sec

Seattle Brazil South * 10,930 km 189 ms 8.2 Mbits/sec 699 Mbits/sec

Seattle South India 12,918 km 202 ms 7.7 Mbits/sec 634 Mbits/sec

Next steps

* The latency to Brazil is a good example where the straight-line distance significantly differs from the fiber rundistance. I would expect that the latency would be in the neighborhood of 160 ms, but is actually 189 ms. Thisdifference against my expectation could indicate a network issue somewhere, but most likely that the fiber run doesnot go to Brazil in a straight line and has an extra 1,000 km or so of travel to get to Brazil from Seattle.

1. Download the Azure Connectivity Toolkit from GitHub at http://aka.ms/AzCT2. Follow the instructions for link performance testing

http://aka.ms/AzCT

https://github.com/Azure/NetworkMonitoring/blob/master/AzureCT/PerformanceTesting.md

Reset a failed ExpressRoute circuit11/29/2017 • 1 min to read • Edit Online

Reset a circuit

Next steps

When an operation on an ExpressRoute circuit does not complete successfully, the circuit may go into a 'failed'state. This article helps you reset a failed Azure ExpressRoute circuit.






1. Install the latest version of the Azure Resource Manager PowerShell cmdlets. For more information, seeInstall and configure Azure PowerShell.

2. Open your PowerShell console with elevated privileges, and connect to your account. Use the followingexample to help you connect:

3. If you have multiple Azure subscriptions, check the subscriptions for the account.

4. Specify the subscription that you want to use.

5. Run the following commands to reset a circuit that is in a failed state:

The circuit should now be healthy. Open a support ticket with Microsoft support if the circuit is still in a failed state.

Open a support ticket with Microsoft support if you are still experiencing issues.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/reset-circuit.md

https://docs.microsoft.com/powershell/azure/install-azurerm-ps



Getting ARP tables in the Resource Managerdeployment model6/27/2017 • 5 min to read • Edit Online

IMPORTANTIMPORTANT

Address Resolution Protocol (ARP) and ARP tables

Age InterfaceProperty IpAddress MacAddress --- ----------------- --------- ---------- 10 On-Prem 10.0.0.1 ffff.eeee.dddd 0 Microsoft 10.0.0.2 aaaa.bbbb.cccc

Prerequisites for learning ARP tables

This article walks you through the steps to learn the ARP tables for your ExpressRoute circuit.

This document is intended to help you diagnose and fix simple issues. It is not intended to be a replacement for Microsoftsupport. You must open a support ticket with Microsoft support if you are unable to solve the problem using the guidancedescribed below.

Address Resolution Protocol (ARP) is a layer 2 protocol defined in RFC 826. ARP is used to map the Ethernetaddress (MAC address) with an ip address.

The ARP table provides a mapping of the ipv4 address and MAC address for a particular peering. The ARP tablefor an ExpressRoute circuit peering provides the following information for each interface (primary and secondary)

1. Mapping of on-premises router interface ip address to the MAC address2. Mapping of ExpressRoute router interface ip address to the MAC address3. Age of the mapping

ARP tables can help validate layer 2 configuration and troubleshooting basic layer 2 connectivity issues.

Example ARP table:

The following section provides information on how you can view the ARP tables seen by the ExpressRoute edgerouters.

Ensure that you have the following before you progress further

A Valid ExpressRoute circuit configured with at least one peering. The circuit must be fully configured by theconnectivity provider. You (or your connectivity provider) must have configured at least one of the peerings(Azure private, Azure public and Microsoft) on this circuit.IP address ranges used for configuring the peerings (Azure private, Azure public and Microsoft). Review the ipaddress assignment examples in the ExpressRoute routing requirements page to get an understanding of howip addresses are mapped to interfaces on your side and on the ExpressRoute side. You can get information onthe peering configuration by reviewing the ExpressRoute peering configuration page.Information from your networking team / connectivity provider on the MAC addresses of interfaces used withthese IP addresses.You must have the latest PowerShell module for Azure (version 1.50 or newer).

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-troubleshooting-arp-resource-manager.md


https://tools.ietf.org/html/rfc826

Getting the ARP tables for your ExpressRoute circuit

ARP tables for Azure private peeringARP tables for Azure private peering

# Required Variables $RG = "<Your Resource Group Name Here>" $Name = "<Your ExpressRoute Circuit Name Here>"

# ARP table for Azure private peering - Primary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType AzurePrivatePeering -DevicePath Primary

# ARP table for Azure private peering - Secodary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType AzurePrivatePeering -DevicePath Secondary


ARP tables for Azure public peeringARP tables for Azure public peering


# ARP table for Azure public peering - Primary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType AzurePublicPeering -DevicePath Primary

# ARP table for Azure public peering - Secodary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType AzurePublicPeering -DevicePath Secondary


ARP tables for Microsoft peeringARP tables for Microsoft peering

This section provides instructions on how you can view the ARP tables per peering using PowerShell. You or yourconnectivity provider must have configured the peering before progressing further. Each circuit has two paths(primary and secondary). You can check the ARP table for each path independently.

The following cmdlet provides the ARP tables for Azure private peering

Sample output is shown below for one of the paths

The following cmdlet provides the ARP tables for Azure public peering


The following cmdlet provides the ARP tables for Microsoft peering


# ARP table for Microsoft peering - Primary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType MicrosoftPeering -DevicePath Primary

# ARP table for Microsoft peering - Secodary path Get-AzureRmExpressRouteCircuitARPTable -ResourceGroupName $RG -ExpressRouteCircuitName $Name -PeeringType MicrosoftPeering -DevicePath Secondary


How to use this information

ARP table when a circuit is in operational state (expected state)ARP table when a circuit is in operational state (expected state)

ARP table when on-premises / connectivity provider side has problemsARP table when on-premises / connectivity provider side has problems

Age InterfaceProperty IpAddress MacAddress --- ----------------- --------- ---------- 0 Microsoft 65.0.0.2 aaaa.bbbb.cccc

Age InterfaceProperty IpAddress MacAddress --- ----------------- --------- ---------- 0 On-Prem 65.0.0.1 Incomplete 0 Microsoft 65.0.0.2 aaaa.bbbb.cccc


The ARP table of a peering can be used to determine validate layer 2 configuration and connectivity. This sectionprovides an overview of how ARP tables will look under different scenarios.

The ARP table will have an entry for the on-premises side with a valid IP address and MAC address and asimilar entry for the Microsoft side.The last octet of the on-premises ip address will always be an odd number.The last octet of the Microsoft ip address will always be an even number.


The same MAC address will appear on the Microsoft side for all 3 peerings (primary / secondary).

If there are issues with the on-premises or connectivity provider you may see that either only one entry will appearin the ARP table or the on-prem MAC address will show incomplete. This will show the mapping between theMAC address and IP address used in the Microsoft side.

or

NOTENOTE

ARP table when Microsoft side has problemsARP table when Microsoft side has problems

Next Steps

Open a support request with your connectivity provider to debug such issues. If the ARP table does not have IP addresses ofthe interfaces mapped to MAC addresses, review the following information:

1. If the first IP address of the /30 subnet assigned for the link between the MSEE-PR and MSEE is used on the interface ofMSEE-PR. Azure always uses the second IP address for MSEEs.

2. Verify if the customer (C-Tag) and service (S-Tag) VLAN tags match both on MSEE-PR and MSEE pair.

You will not see an ARP table shown for a peering if there are issues on the Microsoft side.Open a support ticket with Microsoft support. Specify that you have an issue with layer 2 connectivity.

Validate Layer 3 configurations for your ExpressRoute circuit

Validate data transfer by reviewing bytes in / outOpen a support ticket with Microsoft support if you are still experiencing issues.

Get route summary to determine the state of BGP sessionsGet route table to determine which prefixes are advertised across ExpressRoute



Getting ARP tables in the classic deployment model6/27/2017 • 4 min to read • Edit Online

IMPORTANTIMPORTANT

Address Resolution Protocol (ARP) and ARP tables


Prerequisites for using ARP tables

This article walks you through the steps for getting the Address Resolution Protocol (ARP) tables for your AzureExpressRoute circuit.

This document is intended to help you diagnose and fix simple issues. It is not intended to be a replacement for Microsoftsupport. If you can't solve the problem by using the following guidance, open a support request with Microsoft AzureHelp+support.

ARP is a Layer 2 protocol that's defined in RFC 826. ARP is used to map an Ethernet address (MAC address) to anIP address.

An ARP table provides a mapping of the IPv4 address and MAC address for a particular peering. The ARP tablefor an ExpressRoute circuit peering provides the following information for each interface (primary and secondary):

1. Mapping of an on-premises router interface IP address to a MAC address2. Mapping of an ExpressRoute router interface IP address to a MAC address3. The age of the mapping

ARP tables can help with validating Layer 2 configuration and with troubleshooting basic Layer 2 connectivityissues.

Following is an example of an ARP table:

The following section provides information about how to view the ARP tables that are seen by the ExpressRouteedge routers.

Ensure that you have the following before you continue:

A valid ExpressRoute circuit that's configured with at least one peering. The circuit must be fully configured bythe connectivity provider. You (or your connectivity provider) must configure at least one of the peerings (Azureprivate, Azure public, or Microsoft) on this circuit.IP address ranges that are used for configuring the peerings (Azure private, Azure public, and Microsoft).Review the IP address assignment examples in the ExpressRoute routing requirements page to get anunderstanding of how IP addresses are mapped to interfaces on your aise and on the ExpressRoute side. Youcan get information about the peering configuration by reviewing the ExpressRoute peering configuration page.Information from your networking team or connectivity provider about the MAC addresses of the interfacesthat are used with these IP addresses.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-troubleshooting-arp-classic.md


https://tools.ietf.org/html/rfc826

ARP tables for your ExpressRoute circuit

ARP tables for Azure private peeringARP tables for Azure private peering

# Required variables $ckt = "<your Service Key here>

# ARP table for Azure private peering--primary path Get-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Private -Path Primary

# ARP table for Azure private peering--secondary path Get-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Private -Path Secondary


ARP tables for Azure public peering:ARP tables for Azure public peering:

# Required variables $ckt = "<your Service Key here>

# ARP table for Azure public peering--primary path Get-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Public -Path Primary

# ARP table for Azure public peering--secondary path Get-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Public -Path Secondary



ARP tables for Microsoft peeringARP tables for Microsoft peering

The latest Windows PowerShell module for Azure (version 1.50 or later).

This section provides instructions about how to view the ARP tables for each type of peering by using PowerShell.Before you continue, either you or your connectivity provider needs to configure the peering. Each circuit has twopaths (primary and secondary). You can check the ARP table for each path independently.

The following cmdlet provides the ARP tables for Azure private peering:

Following is sample output for one of the paths:

The following cmdlet provides the ARP tables for Azure public peering:



The following cmdlet provides the ARP tables for Microsoft peering:

# ARP table for Microsoft peering--primary pathGet-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Microsoft -Path Primary

# ARP table for Microsoft peering--secondary pathGet-AzureDedicatedCircuitPeeringArpInfo -ServiceKey $ckt -AccessType Microsoft -Path Secondary


How to use this information

ARP table when a circuit is in an operational (expected) stateARP table when a circuit is in an operational (expected) state

ARP table when it's on-premises or when the connectivity-provider side has problemsARP table when it's on-premises or when the connectivity-provider side has problems

Age InterfaceProperty IpAddress MacAddress --- ----------------- --------- ---------- 0 Microsoft 65.0.0.2 aaaa.bbbb.cccc

NOTENOTE

ARP table when the Microsoft side has problemsARP table when the Microsoft side has problems

Next steps

Sample output is shown below for one of the paths:

The ARP table of a peering can be used to validate Layer 2 configuration and connectivity. This section provides anoverview of how ARP tables look in different scenarios.

The ARP table has an entry for the on-premises side with a valid IP and MAC address, and a similar entry forthe Microsoft side.The last octet of the on-premises IP address is always an odd number.The last octet of the Microsoft IP address is always an even number.


The same MAC address appears on the Microsoft side for all three peerings (primary/secondary).

Only one entry appears in the ARP table. It shows the mapping between the MAC address and the IP addressthat's used on the Microsoft side.

If you experience an issue like this, open a support request with your connectivity provider to resolve it.

You will not see an ARP table shown for a peering if there are issues on the Microsoft side.Open a support request with Microsoft Azure Help+support. Specify that you have an issue with Layer 2connectivity.

Validate Layer 3 configurations for your ExpressRoute circuit:Get a route summary to determine the state of BGP sessions.


Validate data transfer by reviewing bytes in and out.Open a support request with Microsoft Azure Help+support if you are still experiencing issues.

Get a route table to determine which prefixes are advertised across ExpressRoute.


Azure subscription and service limits, quotas, andconstraints4/9/2018 • 59 min to read • Edit Online

NOTENOTE

Limits and the Azure Resource Manager

NOTENOTE

Service-specific limits

This document lists some of the most common Microsoft Azure limits, which are also sometimes called quotas.This document doesn't currently cover all Azure services. Over time, the list will be expanded and updated to covermore of the platform.

Please visit Azure Pricing Overview to learn more about Azure pricing. There, you can estimate your costs usingthe Pricing Calculator or by visiting the pricing details page for a service (for example, Windows VMs). For tips tohelp manage your costs, see Prevent unexpected costs with Azure billing and cost management.

If you want to raise the limit or quota above the Default Limit, open an online customer support request at no charge. Thelimits can't be raised above the Maximum Limit value shown in the following tables. If there is no Maximum Limit column,then the resource doesn't have adjustable limits.

Free Trial subscriptions are not eligible for limit or quota increases. If you have a Free Trial subscription, you can upgrade to aPay-As-You-Go subscription. For more information, see Upgrade Azure Free Trial to Pay-As-You-Go and Free Trialsubscription FAQ.

It is now possible to combine multiple Azure resources in to a single Azure Resource Group. When usingResource Groups, limits that once were global become managed at a regional level with the Azure ResourceManager. For more information about Azure Resource Groups, see Azure Resource Manager overview.

In the limits below, a new table has been added to reflect any differences in limits when using the Azure ResourceManager. For example, there is a Subscription Limits table and a Subscription Limits - Azure ResourceManager table. When a limit applies to both scenarios, it is only shown in the first table. Unless otherwiseindicated, limits are global across all regions.

It is important to emphasize that quotas for resources in Azure Resource Groups are per-region accessible by yoursubscription, and are not per-subscription, as the service management quotas are. Let's use vCPU quotas as an example. Ifyou need to request a quota increase with support for vCPUs, you need to decide how many vCPUs you want to use inwhich regions, and then make a specific request for Azure Resource Group vCPU quotas for the amounts and regions thatyou want. Therefore, if you need to use 30 vCPUs in West Europe to run your application there, you should specificallyrequest 30 vCPUs in West Europe. But you will not have a vCPU quota increase in any other region -- only West Europe willhave the 30-vCPU quota.

As a result, you may find it useful to consider deciding what your Azure Resource Group quotas need to be for yourworkload in any one region, and request that amount in each region into which you are considering deployment. Seetroubleshooting deployment issues for more help discovering your current quotas for specific regions.

Active Directory

https://github.com/Microsoft/azure-docs/blob/master/articles/azure-subscription-service-limits.md

https://azure.microsoft.com/pricing/

https://azure.microsoft.com/pricing/calculator/

https://azure.microsoft.com/pricing/details/virtual-machines/#Windows

https://docs.microsoft.com/en-us/azure/billing/billing-getting-started

https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request

https://azure.microsoft.com/offers/ms-azr-0044p

https://azure.microsoft.com/offers/ms-azr-0044p

https://azure.microsoft.com/offers/ms-azr-0003p/

https://docs.microsoft.com/en-us/azure/billing/billing-upgrade-azure-subscription

https://azure.microsoft.com/free/free-account-faq

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview

https://docs.microsoft.com/en-us/azure/resource-manager-common-deployment-errors

API ManagementApp ServiceApplication GatewayApplication InsightsAutomationAzure Cosmos DBAzure Event GridAzure Redis CacheBackupBatchBizTalk ServicesCDNCloud ServicesContainer InstancesContainer RegistryContainer Service (AKS)Data FactoryData Lake AnalyticsData Lake StoreDatabase Migration ServiceDNSEvent HubsIoT HubIoT Hub Device Provisioning ServiceKey VaultLog AnalyticsMedia ServicesMobile EngagementMobile ServicesMonitorMulti-Factor AuthenticationNetworkingNetwork WatcherNotification Hub ServiceResource GroupSchedulerSearchService BusSite RecoverySQL DatabaseSQL Data WarehouseStorageStorSimple SystemStream AnalyticsSubscriptionTraffic Manager

Subscription limitsSubscription limitsSubscription limitsSubscription limits

RESOURCE DEFAULT LIMIT MAXIMUM LIMIT

Cores per subscription 20 10,000

Co-administrators per subscription 200 200

Storage accounts per region persubscription

200 250

Cloud services per subscription 20 200

Local networks per subscription 10 500

SQL Database servers per subscription 6 150

DNS servers per subscription 9 100

Reserved IPs per subscription 20 100

Hosted service certificates persubscription

199 199

Affinity groups per subscription 256 256

Subscription limits - Azure Resource ManagerSubscription limits - Azure Resource Manager


VMs per subscription 10,000 per Region 10,000 per Region

VM total cores per subscription 20 per Region Contact support

VM per series (Dv2, F, etc.) cores persubscription

20 per Region Contact support

Co-administrators per subscription Unlimited Unlimited

Storage accounts per subscription 200 200

Virtual MachinesVirtual Machine Scale Sets

1

2

Extra Small instances count as one core towards the core limit despite using a partial core.1

The storage account limit includes both Standard and Premium storage accounts. If you require more than 200storage accounts in a single region, make a request through Azure Support. The Azure Storage team will reviewyour business case and may approve up to 250 storage accounts.

2

The following limits apply when using the Azure Resource Manager and Azure Resource Groups. Limits that havenot changed with the Azure Resource Manager are not listed below. Please refer to the previous table for thoselimits.

For information about handling limits on Resource Manager requests, see Throttling Resource Manager requests.

1

1

1

2

https://docs.microsoft.com/en-us/azure/billing-buy-sign-up-azure-subscription

https://docs.microsoft.com/en-us/azure/billing-add-change-azure-subscription-administrator

https://docs.microsoft.com/en-us/azure/storage/common/storage-create-storage-account

https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-choose-me

http://msdn.microsoft.com/library/jj157100.aspx

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-migrate-to-regional-vnet

https://azure.microsoft.com/support/faq/

https://docs.microsoft.com/en-us/azure/resource-manager-request-limits




https://docs.microsoft.com/en-us/azure/billing-add-change-azure-subscription-administrator

https://docs.microsoft.com/en-us/azure/storage/common/storage-create-storage-account

Resource Groups per subscription 980 980

Availability Sets per subscription 2,000 per Region 2,000 per Region

Resource Manager API Reads 15,000 per hour 15,000 per hour

Resource Manager API Writes 1,200 per hour 1,200 per hour

Resource Manager API request size 4,194,304 bytes 4,194,304 bytes

Tags per subscription unlimited unlimited

Unique tag calculations persubscription

10,000 10,000

Cloud services per subscription Not Applicable Not Applicable

Affinity groups per subscription Not Applicable Not Applicable


NOTENOTE

Resource Group limitsResource Group limits


Resources per resource group (perresource type)

800 Varies per resource type

Deployments per resource group in thedeployment history

800 800

Resources per deployment 800 800

Management Locks (per unique scope) 20 20

3

3

4 4

4 4

Default limits vary by offer Category Type, such as Free Trial, Pay-As-You-Go, and series, such as Dv2, F, G, etc.1

This includes both Standard and Premium storage accounts. If you require more than 200 storage accounts,make a request through Azure Support. The Azure Storage team will review your business case and may approveup to 250 storage accounts.

2

You can apply an unlimited number of tags per subscription. The number of tags per resource or resource groupis limited to 15. Resource Manager only returns a list of unique tag name and values in the subscription when thenumber of tags is 10,000 or less. However, you can still find a resource by tag when the number exceeds 10,000.

3

These features are no longer required with Azure Resource Groups and the Azure Resource Manager.4

It is important to emphasize that virtual machine cores have a regional total limit as well as a regional per size series (Dv2, F,etc.) limit that are separately enforced. For example, consider a subscription with a US East total VM core limit of 30, an Aseries core limit of 30, and a D series core limit of 30. This subscription would be allowed to deploy 30 A1 VMs, or 30 D1VMs, or a combination of the two not to exceed a total of 30 cores (for example, 10 A1 VMs and 20 D1 VMs).


https://docs.microsoft.com/en-us/azure/virtual-machines/windows/manage-availability


https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-migrate-to-regional-vnet


https://docs.microsoft.com/rest/api/resources/tags#Tags_List


Number of Tags (per resource orresource group)

15 15

Tag key length 512 512

Tag value length 256 256


Template limitsTemplate limits

VALUE DEFAULT LIMIT MAXIMUM LIMIT

Parameters 256 256

Variables 256 256

Resources (including copy count) 800 800

Outputs 64 64

Template expression 24,576 chars 24,576 chars

Resources in exported templates 200 200

Template size 1 MB 1 MB

Parameter file size 64 KB 64 KB

Virtual Machines limitsVirtual Machines limitsVirtual Machine limitsVirtual Machine limits


Virtual machines per cloud service 50 50

Input endpoints per cloud service 150 150

Virtual Machines limits - Azure Resource ManagerVirtual Machines limits - Azure Resource Manager

You can exceed some template limits by using a nested template. For more information, see Using linkedtemplates when deploying Azure resources. To reduce the number of parameters, variables, or outputs, you cancombine several values into an object. For more information, see Objects as parameters.

If you reach the limit of 800 deployments per resource group, delete deployments from the history that are nolonger needed. You can delete entries from the history with az group deployment delete for Azure CLI, orRemove-AzureRmResourceGroupDeployment in PowerShell. Deleting an entry from the deployment historydoes not affect the deploy resources.

1

2

Virtual machines created in Service Management (instead of Resource Manager) are automatically stored in acloud service. You can add more virtual machines to that cloud service for load balancing and availability.

1

Input endpoints allow communications to a virtual machine from outside the virtual machine's cloud service.Virtual machines in the same cloud service or virtual network can automatically communicate with each other. SeeHow to Set Up Endpoints to a Virtual Machine.

2

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-linked-templates

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-objects-as-parameters

https://docs.microsoft.com/cli/azure/group/deployment#az_group_deployment_delete

https://docs.microsoft.com/powershell/module/azurerm.resources/remove-azurermresourcegroupdeployment

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-about

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/classic/setup-endpoints


Virtual machines per availability set 200

Certificates per subscription Unlimited

Virtual Machine Scale Sets limitsVirtual Machine Scale Sets limits


Maximum number of VMs in a scale set 1000 1000

Maximum number of VMs based on acustom VM image in a scale set

300 300

Maximum number of scale sets in aregion

2000 2000

Container Instances limitsContainer Instances limits


Container groups per subscription 20

Number of containers per container group 60

Number of volumes per container group 20

Ports per IP 5

Container creates per hour 60

Container creates per 5 minutes 20

Container deletes per hour 150

Container deletes per 5 minutes 50

Multiple containers per container group Linux only

Azure Files volumes Linux only

GitRepo volumes Linux only

Secret volumes Linux only

The following limits apply when using the Azure Resource Manager and Azure Resource Groups. Limits that havenot changed with the Azure Resource Manager are not listed below. Please refer to the previous table for thoselimits.

1

With Azure Resource Manager, certificates are stored in the Azure Key Vault. Although the number of certificatesis unlimited for a subscription, there is still a 1 MB limit of certificates per deployment (which consists of either asingle VM or an availability set).

1

1

1

1

1

1

2

2

2

2

1


Container Registry limitsContainer Registry limits

RESOURCE BASIC STANDARD PREMIUM

Storage 10 GiB 100 GiB 500 GiB

ReadOps per minute 1000 3000 10000

WriteOps per minute 100 500 2000

Download bandwidth MBps 30 60 100

Upload bandwidth MBps 10 20 50

Webhooks 2 10 100

Geo-replication N/A N/A Supported (preview)

Container Service (AKS) limitsContainer Service (AKS) limits


Max nodes per cluster 100

Max pods per node 110

Max cluster per subscription 20

Networking limitsNetworking limitsExpressRoute LimitsExpressRoute Limits


ExpressRoute circuits per subscription 10

ExpressRoute circuits per region per subscription for ARM 10

Maximum number of routes for Azure private peering withExpressRoute standard

4,000

Create an Azure support request to request a limit increase. Windows support for this feature is planned.

1

2

The following table details the features and limits of the Basic, Standard, and Premium service tiers.

1, 2

1, 3

1

1

ReadOps, WriteOps, and Bandwidth are minimum estimates. ACR strives to improve performance as usagerequires.

1

docker pull translates to multiple read operations based on the number of layers in the image, plus the manifestretrieval.

2

docker push translates to multiple write operations, based on the number of layers that must be pushed. A docker push includes ReadOps to retrieve a manifest for an existing image.

3

1

Create an Azure support request to request a limit increase.1

The following limits apply to ExpressRoute resources per subscription.

https://ms.portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-skus

https://docs.microsoft.com/azure/container-registry/container-registry-geo-replication

https://docs.docker.com/registry/spec/api/#pulling-an-image

https://docs.docker.com/registry/spec/api/#pushing-an-image

https://ms.portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest

Maximum number of routes for Azure private peering withExpressRoute premium add-on

10,000

Maximum number of routes for Azure public peering withExpressRoute standard

200

Maximum number of routes for Azure public peering withExpressRoute premium add-on

200

Maximum number of routes for Azure Microsoft peering withExpressRoute standard

200

Maximum number of routes for Azure Microsoft peering withExpressRoute premium add-on

200

Number of virtual network links allowed per ExpressRoutecircuit

see table below


Number of Virtual Networks per ExpressRoute circuitNumber of Virtual Networks per ExpressRoute circuit

CIRCUIT SIZE NUMBER OF VNET LINKS FOR STANDARDNUMBER OF VNET LINKS WITH PREMIUMADD-ON

50 Mbps 10 20

100 Mbps 10 25

200 Mbps 10 25

500 Mbps 10 40

1 Gbps 10 50

2 Gbps 10 60

5 Gbps 10 75

10 Gbps 10 100

Networking limitsNetworking limits


Virtual networks 50 100

Local network sites 20 contact support

DNS Servers per virtual network 20 100

Private IP Addresses per virtualnetwork

4096 4096

The following limits apply only for networking resources managed through the classic deployment model persubscription.

Concurrent TCP or UDP flows per NICof a virtual machine or role instance

500K 500K

Network Security Groups (NSG) 100 200

NSG rules per NSG 200 400

User defined route tables 100 200

User defined routes per route table 100 400

Public IP addresses (dynamic) 5 contact support

Reserved public IP addresses 20 contact support

Public VIP per deployment 5 contact support

Private VIP (ILB) per deployment 1 1

Endpoint Access Control Lists (ACLs) 50 50


Networking Limits - Azure Resource ManagerNetworking Limits - Azure Resource Manager


Virtual networks 50 1000

Subnets per virtual network 1000 10000

Virtual network peerings per VirtualNetwork

10 50

DNS Servers per virtual network 9 25

Private IP Addresses per virtualnetwork

4096 8192

Private IP Addresses per networkinterface

256 1024

Concurrent TCP or UDP flows per NICof a virtual machine or role instance

500K 500K

Network Interfaces (NIC) 350 20000

Network Security Groups (NSG) 100 5000

The following limits apply only for networking resources managed through Azure Resource Manager per regionper subscription.

NSG rules per NSG 200 1000

IP addresses and ranges specified forsource or destination in a securitygroup

2000 4000

Application security groups 200 500

Application security groups per IPconfiguration, per NIC

10 20

IP configurations per applicationsecurity group

1000 4000

Application security groups that can bespecified within all security rules of anetwork security group

50 100

User defined route tables 100 200

User defined routes per route table 100 400

Public IP addresses - dynamic (Basic) 60 contact support

Public IP addresses - static (Basic) 20 contact support

Public IP addresses - static (Standard) 20 contact support

Point-to-Site Root Certificates per VPNGateway

20 20


Load Balancer limitsLoad Balancer limits


Load Balancers 100 1000

Rules per resource, Basic 150 250

Rules per resource, Standard 1250 1500

Rules per IP configuration 299 299

Frontend IP configurations, Basic 10 200

Frontend IP configurations, Standard 10 600

Backend pool, Basic 100, single Availability Set 100, single Availability Set

Backend pool, Standard 1000, single VNet 1000, single VNet

HA Ports, Standard 1 per internal frontend 1 per internal frontend

Application Gateway limitsApplication Gateway limits

RESOURCE DEFAULT LIMIT NOTE

Application Gateway 50 per subscription Maximum 100

Frontend IP Configurations 2 1 public and 1 private

Frontend Ports 20

Backend Address Pools 20

Backend Servers per pool 100

HTTP Listeners 20

HTTP load balancing rules 200 # of HTTP Listeners * n, n=10 Default

Backend HTTP settings 20 1 per Backend Address Pool

Instances per gateway 10 For more instances, open support ticket

SSL certificates 20 1 per HTTP Listeners

Authentication certificates 5 Maximum 10

Request time out min 1 second

Request time out max 24 hrs

Number of sites 20 1 per HTTP Listeners

URL Maps per listener 1

Maximum file upload size Standard 2 GB

Maximum file upload size WAF 100 MB

Network Watcher limitsNetwork Watcher limits


Network Watcher 1 per region

Packet Capture sessions 10 per region # of sessions only, not saved captures

Traffic Manager limitsTraffic Manager limits


Profiles per subscription 200

Endpoints per profile 200

Contact support in case you need to increase limits from default.

1

https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request


DNS limitsDNS limits


Zones per subscription 100

Record sets per zone 5000

Records per record set 20

Storage limitsStorage limits


Number of storage accounts per region 200

Max storage account capacity 500 TiB

Max number of blob containers, blobs, file shares, tables,queues, entities, or messages per storage account

No limit

Maximum request rate per storage account 20,000 requests per second

Max ingress per storage account (US Regions) 10 Gbps if RA-GRS/GRS enabled, 20 Gbps for LRS/ZRS

Max egress per storage account (US Regions) 20 Gbps if RA-GRS/GRS enabled, 30 Gbps for LRS/ZRS

Max ingress per storage account (Non-US regions) 5 Gbps if RA-GRS/GRS enabled, 10 Gbps for LRS/ZRS

Max egress per storage account (Non-US regions) 10 Gbps if RA-GRS/GRS enabled, 15 Gbps for LRS/ZRS

Contact support in case you need to increase these limits.1

1

1

Contact Azure Support in case you need to increase these limits.1

For additional details on storage account limits, see Azure Storage Scalability and Performance Targets.

1

2

2

3 4

3 4

3 4

3 4

Includes both Standard and Premium storage accounts. If you require more than 200 storage accounts, make arequest through Azure Support. The Azure Storage team will review your business case and may approve up to250 storage accounts.

1

If you need expanded limits for your storage account, please contact Azure Support. The Azure Storage team willreview the request and may approve higher limits on a case by case basis. Both general-purpose and Blob storageaccounts support increased capacity, ingress/egress, and request rate by request. For the new maximums for Blobstorage accounts, see Announcing larger, higher scale storage accounts.

2

Capped only by the account's ingress/egress limits. Ingress refers to all data (requests) being sent to a storageaccount. Egress refers to all data (responses) being received from a storage account.

3

Azure Storage redundancy options include:4

RA-GRS: Read-access geo-redundant storage. If RA-GRS is enabled, egress targets for the secondary locationare identical to those for the primary location.

https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets



https://azure.microsoft.com/blog/announcing-larger-higher-scale-storage-accounts/


Storage account management operations (read) 800 per 5 minutes

Storage account management operations (write) 200 per hour

Storage account management operations (list) 100 per 5 minutes

Azure Blob storage limitsAzure Blob storage limits

RESOURCE TARGET

Max size of single blob container 500 TiB

Max number of blocks in a block blob or append blob 50,000 blocks

Max size of a block in a block blob 100 MiB

Max size of a block blob 50,000 X 100 MiB (approx. 4.75 TiB)

Max size of a block in an append blob 4 MiB

Max size of an append blob 50,000 x 4 MiB (approx. 195 GiB)

Max size of a page blob 8 TiB

Max number of stored access policies per blob container 5

Target throughput for single blob Up to 60 MiB per second, or up to 500 requests per second

Azure Files limitsAzure Files limits

RESOURCE TARGET

Max size of a file share 5 TiB

Max size of a file in a file share 1 TiB

Max number of files in a file share No limit

Max IOPS per share 1000 IOPS

Max number of stored access policies per file share 5

Maximum request rate per storage account 20,000 requests per second for files of any valid size

GRS: Geo-redundant storage.ZRS: Zone-redundant storage.LRS: Locally redundant storage.

The following limits apply when performing management operations using the Azure Resource Manager only.

For additional details on Azure Files limits, see Azure Files scalability and performance targets.

3

https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets

Target throughput for single file share Up to 60 MiB per second

Maximum open handles for per file 2000 open handles

Maximum number of share snapshots 200 share snapshots

RESOURCE TARGET

Azure File Sync limitsAzure File Sync limits

RESOURCE TARGET HARD LIMIT

Storage Sync Services per subscription 15 Storage Sync Services No

Sync groups per Storage Sync Service 30 sync groups Yes

Registered servers per Storage SyncService

99 servers Yes

Cloud endpoints per Sync Group 1 cloud endpoint Yes

Server endpoints per Sync Group 50 server endpoints No

Server endpoints per server 33-99 server endpoints Yes, but varies based on configuration

Endpoint size 4 TiB No

File system objects (directories and files)per sync group

6 million objects No

File size 100 GiB No

Minimum file size for a file to be tiered 64 KiB Yes

Azure Queue storage limitsAzure Queue storage limits

RESOURCE TARGET

Max size of single queue 500 TiB

Max size of a message in a queue 64 KiB

Max number of stored access policies per queue 5

Maximum request rate per storage account 20,000 messages per second assuming 1 KiB message size

Target throughput for single queue (1 KiB messages) Up to 2000 messages per second

Azure Table storage limitsAzure Table storage limits

RESOURCE TARGET

Max size of single table 500 TiB

Max size of a table entity 1 MiB

Max number of properties in a table entity 252

Max number of stored access policies per table 5

Maximum request rate per storage account 20,000 transactions per second (assuming 1 KiB entity size)

Target throughput for single table partition (1 KiB entities) Up to 2000 entities per second

RESOURCE TARGET

Virtual machine disk limitsVirtual machine disk limits

IMPORTANTIMPORTANT

Managed virtual machine disksManaged virtual machine disks

STANDARDDISK TYPE S4 S6 S10 S20 S30 S40 S50

Disk size 32 GB 64 GB 128 GB 512 GB 1024 GB (1TB)

2048 GB(2TB)

4095 GB (4TB)

IOPS perdisk

500 500 500 500 500 500 500

An Azure virtual machine supports attaching a number of data disks. This article describes scalability andperformance targets for a VM's data disks. Use these targets to help decide the number and type of disk that youneed to meet your performance and capacity requirements.

For optimal performance, limit the number of highly utilized disks attached to the virtual machine to avoid possiblethrottling. If all attached disks are not highly utilized at the same time, then the virtual machine can support a larger numberof disks.

For Azure Managed Disks: The disk limit for managed disks is per region and per disk type. Themaximum limit, and also the default limit, is 10,000 managed disks per region and per disk type for asubscription. For example, you can create up to 10,000 standard managed disks and also 10,000 premiummanaged disks in a region, per subscription.

Managed snapshots and images count against the managed disks limit.

For standard storage accounts: A standard storage account has a maximum total request rate of 20,000IOPS. The total IOPS across all of your virtual machine disks in a standard storage account should notexceed this limit.

You can roughly calculate the number of highly utilized disks supported by a single standard storageaccount based on the request rate limit. For example, for a Basic Tier VM, the maximum number of highlyutilized disks is about 66 (20,000/300 IOPS per disk), and for a Standard Tier VM, it is about 40(20,000/500 IOPS per disk).

For premium storage accounts: A premium storage account has a maximum total throughput rate of 50Gbps. The total throughput across all of your VM disks should not exceed this limit.

See Virtual machine sizes for additional details.

Standard managed virtual machine disks

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes

Throughputper disk

60 MB/sec 60 MB/sec 60 MB/sec 60 MB/sec 60 MB/sec 60 MB/sec 60 MB/sec

STANDARDDISK TYPE S4 S6 S10 S20 S30 S40 S50

PREMIUMDISKS TYPE P4 P6 P10 P20 P30 P40 P50

Disk size 32 GB 64 GB 128 GB 512 GB 1024 GB (1TB)

2048 GB (2TB)

4095 GB (4TB)

IOPS perdisk

120 240 500 2300 5000 7500 7500

Throughputper disk

25 MB/sec 50 MB/sec 100 MB/sec 150 MB/sec 200 MB/sec 250 MB/sec 250 MB/sec


Max IOPS Per VM 80,000 IOPS with GS5 VM

Max throughput per VM 2,000 MB/s with GS5 VM

Unmanaged virtual machine disksUnmanaged virtual machine disks

VM TIER BASIC TIER VM STANDARD TIER VM

Disk size 4095 GB 4095 GB

Max 8 KB IOPS per persistent disk 300 500

Max number of disks performing maxIOPS

66 40


Total disk capacity per account 35 TB

Total snapshot capacity per account 10 TB

Max bandwidth per account (ingress + egress ) <=50 Gbps

Premium managed virtual machine disks: per disk limits

Premium managed virtual machine disks: per VM limits

Standard unmanaged virtual machine disks: per disk limits

Premium unmanaged virtual machine disks: per account limits

1

Ingress refers to all data (requests) being sent to a storage account. Egress refers to all data (responses) beingreceived from a storage account.

1

Premium unmanaged virtual machine disks: per disk limits

PREMIUMSTORAGE DISKTYPE P10 P20 P30 P40 P50

Disk size 128 GiB 512 GiB 1024 GiB (1 TB) 2048 GiB (2 TB) 4095 GiB (4 TB)

Max IOPS perdisk

500 2300 5000 7500 7500

Max throughputper disk

100 MB/s 150 MB/s 200 MB/s 250 MB/s 250 MB/s

Max number ofdisks per storageaccount

280 70 35 17 8


Max IOPS Per VM 80,000 IOPS with GS5 VM

Max throughput per VM 2,000 MB/s with GS5 VM

Cloud Services limitsCloud Services limits


Web/worker roles per deployment 25 25

Instance Input Endpoints perdeployment

25 25

Input Endpoints per deployment 25 25

Internal Endpoints per deployment 25 25

App Service limitsApp Service limits

RESOURCE FREESHARED(PREVIEW) BASIC STANDARD PREMIUM

Web, mobile, orAPI apps per AppService plan

10 100 Unlimited Unlimited Unlimited

Logic apps perApp Service plan

10 10 10 20 per core 20 per core

Premium unmanaged virtual machine disks: per VM limits

1

Each Cloud Service with Web/Worker roles can have two deployments, one for production and one for staging.Also note that this limit refers to the number of distinct roles (configuration) and not the number of instances perrole (scaling).

1

The following App Service limits include limits for Web Apps, Mobile Apps, API Apps, and Logic Apps.

1

2 2 2

1


http://msdn.microsoft.com/library/gg557552.aspx#InstanceInputEndpoint

http://msdn.microsoft.com/library/gg557552.aspx#InputEndpoint

http://msdn.microsoft.com/library/gg557552.aspx#InternalEndpoint

https://azure.microsoft.com/services/app-service/

https://docs.microsoft.com/en-us/azure/app-service/azure-web-sites-web-hosting-plans-in-depth-overview

https://azure.microsoft.com/services/app-service/logic/


App Service plan 1 per region 10 per resourcegroup

100 per resourcegroup



Computeinstance type

Shared Shared Dedicated Dedicated Dedicated

Scale-Out (maxinstances)

1 shared 1 shared 3 dedicated 10 dedicated 20 dedicated (50in ASE)

Storage 1 GB 1 GB 10 GB 50 GB 500 GB

CPU time (5min)

3 minutes 3 minutes Unlimited, pay atstandard rates

Unlimited, pay atstandard rates


CPU time (day) 60 minutes 240 minutes Unlimited, pay atstandard rates



Memory (1 hour) 1024 MB perApp Service plan

1024 MB per app N/A N/A N/A

Bandwidth 165 MB Unlimited, datatransfer ratesapply

Unlimited, datatransfer ratesapply



Applicationarchitecture

32-bit 32-bit 32-bit/64-bit 32-bit/64-bit 32-bit/64-bit

Web Sockets perinstance

5 35 350 Unlimited Unlimited

Concurrentdebuggerconnections perapplication

1 1 1 5 5

azurewebsites.netsubdomain withFTP/S and SSL

X X X X X

Custom domainsupport

X X X X

Custom domainSSL support

Unlimited SNISSL connections

Unlimited SNISSL and 1 IP SSLconnectionsincluded

Unlimited SNISSL and 1 IP SSLconnectionsincluded

Integrated LoadBalancer

X X X X

Always On X X X


3 3 3

3 3

3,4

5 5 5 5 5 4,5

6

6

7


https://docs.microsoft.com/en-us/azure/app-service/web-sites-scale

https://azure.microsoft.com/pricing/details/app-service/


https://azure.microsoft.com/pricing/details/data-transfers/

https://docs.microsoft.com/en-us/azure/app-service/web-sites-dotnet-troubleshoot-visual-studio

https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-custom-ssl

https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-custom-domain

https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-custom-ssl

https://docs.microsoft.com/en-us/azure/app-service/web-sites-configure

ScheduledBackups

Scheduledbackups every 2hours, a max of12 backups perday (manual +scheduled)

Scheduledbackups everyhour, a max of 50backups per day(manual +scheduled)

Auto Scale X X

WebJobs X X X X X

Azure Schedulersupport

X X X X

Endpointmonitoring

X X X

Staging Slots 5 20

Custom domainsper app

500 500 500 500

SLA 99.9% 99.95% 99.95%


Scheduler limitsScheduler limits

RESOURCE LIMIT DESCRIPTION

Job size Maximum job size is 16K. If a PUT or a PATCH results in a joblarger than these limits, a 400 Bad Request status code isreturned.

8

10 9

Apps and storage quotas are per App Service plan unless noted otherwise.The actual number of apps that you can host on these machines depends on the activity of the apps, the size of

the machine instances, and the corresponding resource utilization.Dedicated instances can be of different sizes. See App Service Pricing for more details.Premium tier allows up to 50 computes instances (subject to availability) and 500 GB of disk space when using

App Service Environments, and 20 compute instances and 250 GB storage otherwise.The storage limit is the total content size across all apps in the same App Service plan. More storage options are

available in App Service EnvironmentThese resources are constrained by physical resources on the dedicated instances (the instance size and the

number of instances).If you scale an app in the Basic tier to two instances, you have 350 concurrent connections for each of the two

instances.Run custom executables and/or scripts on demand, on a schedule, or continuously as a background task within

your App Service instance. Always On is required for continuous WebJobs execution. Azure Scheduler Free orStandard is required for scheduled WebJobs. There is no predefined limit on the number of WebJobs that can runin an App Service instance, but there are practical limits that depend on what the application code is trying to do.SL A of 99.95% provided for deployments that use multiple instances with Azure Traffic Manager configured for

failover.

1

2

3

4

5

6

7

8

9

The following table describes each of the major quotas, limits, defaults, and throttles in Azure Scheduler.

https://docs.microsoft.com/en-us/azure/app-service/web-sites-backup

https://docs.microsoft.com/en-us/azure/app-service/web-sites-scale

https://docs.microsoft.com/en-us/azure/app-service/web-sites-create-web-jobs

https://azure.microsoft.com/services/scheduler/

https://docs.microsoft.com/en-us/azure/app-service/web-sites-monitor

https://docs.microsoft.com/en-us/azure/app-service/web-sites-staged-publishing


https://docs.microsoft.com/en-us/azure/app-service/environment/app-service-web-configure-an-app-service-environment

Request URL size Maximum size of the request URL is 2048 chars.

Aggregate header size Maximum aggregate header size is 4096 chars.

Header count Maximum header count is 50 headers.

Body size Maximum body size is 8192 chars.

Recurrence span Maximum recurrence span is 18 months.

Time to start time Maximum â€œtime to start timeâ€ is 18 months.

Job history Maximum response body stored in job history is 2048 bytes.

Frequency The default max frequency quota is 1 hour in a free jobcollection and 1 minute in a standard job collection. The maxfrequency is configurable on a job collection to be lower thanthe maximum. All jobs in the job collection are limited thevalue set on the job collection. If you attempt to create a jobwith a higher frequency than the maximum frequency on thejob collection then request will fail with a 409 Conflict statuscode.

Jobs The default max jobs quota is 5 jobs in a free job collectionand 50 jobs in a standard job collection. The maximumnumber of jobs is configurable on a job collection. All jobs inthe job collection are limited the value set on the jobcollection. If you attempt to create more jobs than themaximum jobs quota, then the request fails with a 409Conflict status code.

Job collections Maximum number of job collection per subscription is200,000.

Job history retention Job history is retained for up to 2 months or up to the last1000 executions.

Completed and faulted job retention Completed and faulted jobs are retained for 60 days.

Timeout Thereâ€™s a static (not configurable) request timeout of 60seconds for HTTP actions. For longer running operations,follow HTTP asynchronous protocols; for example, return a202 immediately but continue working in the background.

RESOURCE LIMIT DESCRIPTION

Batch limitsBatch limits


Batch accounts per region persubscription

1 - 3 50

Dedicated cores per Batch account 10 - 100 N/A

Low-priority cores per Batch account 10 - 100 N/A

1

2

Active jobs and job schedules perBatch account

20 - 100 5000

Pools per Batch account 20 - 100 2500


NOTENOTE

BizTalk Services limitsBizTalk Services limits

RESOURCE FREE (PREVIEW) DEVELOPER BASIC STANDARD PREMIUM

Scale out N/A N/A Yes, in incrementsof 1 Basic Unit

Yes, in incrementsof 1 StandardUnit

Yes, in incrementsof 1 PremiumUnit

Scale Limit N/A N/A Up to 8 units Up to 8 units Up to 8 units

EAI Bridges perUnit

N/A 25 25 125 500

EDI Agreementsper Unit

N/A 10 50 250 1000

HybridConnections perUnit

5 5 10 50 100

HybridConnection DataTransfer (GBs) perUnit

5 5 50 250 500

Number ofconnectionsusing BizTalkAdapter Serviceper Unit

N/A 1 2 5 25

Archiving N/A Available N/A N/A Available

3 4

Default limits vary depending on the type of subscription you use to create a Batch account. Cores quotas shown are forBatch accounts in Batch service mode. View the quotas in your Batch account.

The number of dedicated cores per Batch account can be increased, but the maximum number is unspecified.Contact Azure support to discuss increase options.

1

The number of low-priority cores per Batch account can be increased, but the maximum number is unspecified.Contact Azure support to discuss increase options.

2

Completed jobs and job schedules are not limited.3

Contact Azure support if you want to request an increase beyond this limit.4

The following table shows the limits for Azure Biztalk Services.

https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit

High Availability N/A N/A Available Available Available

RESOURCE FREE (PREVIEW) DEVELOPER BASIC STANDARD PREMIUM

Azure Cosmos DB limitsAzure Cosmos DB limits

Mobile Engagement limitsMobile Engagement limits

RESOURCE MAXIMUM LIMIT

App Collection Users 5 per App Collection

Average Data points 200 per Active User/Day

Average App-Info set 50 per Active User/Day

Average Messages pushed 20 per Active User/Day

Segments 100 per app

Criteria per segment 10

Active Push Campaigns 50 per app

Total Push Campaigns (includes Active & Completed) 1000 per app

Search limitsSearch limits

RESOURCE FREE BASIC S1 S2 S3 S3 HD

Maximumservices

1 12 12 6 6 6

Azure Cosmos DB is a global scale database in which throughput and storage can be scaled to handle whateveryour application requires. If you have any questions about the scale Azure Cosmos DB provides, please send emailto [email protected].

Pricing tiers determine the capacity and limits of your search service. Tiers include:

Free multi-tenant service, shared with other Azure subscribers, intended for evaluation and small developmentprojects.Basic provides dedicated computing resources for production workloads at a smaller scale, with up to threereplicas for highly available query workloads.Standard (S1, S2, S3, S3 High Density) is for larger production workloads. Multiple levels exist within thestandard tier so that you can choose a resource configuration that best matches your workload profile.

Limits per subscription

You can create multiple services within a subscription, each one provisioned at a specific tier, limited only by thenumber of services allowed at each tier. For example, you could create up to 12 services at the Basic tier andanother 12 services at the S1 tier within the same subscription. For more information about tiers, see Choose aSKU or tier for Azure Search.

Maximum service limits can be raised upon request. Contact Azure Support if you need more services within thesame subscription.

1

https://docs.microsoft.com/en-us/azure/search/search-sku-tier

Maximumscale in SU

N/A 3 SU 36 SU 36 SU 36 SU 36 SU



Service LevelAgreement(SLA)

No Yes Yes Yes Yes Yes

Storage perpartition

50 MB 2 GB 25 GB 100 GB 200 GB 200 GB

Partitions perservice

N/A 1 12 12 12 3

Partition size N/A 2 GB 25 GB 100 GB 200 GB 200 GB

Replicas N/A 3 12 12 12 12

Media Services limitsMedia Services limits

NOTENOTE

2

Free is based on shared, not dedicated, resources. Scale-up is not supported on shared resources.1

Search units (SU) are billing units, allocated as either a replica or a partition. You need both resources forstorage, indexing, and query operations. To learn more about SU computations, see Scale resource levels forquery and index workloads.

2

Limits per search service

Storage is constrained by disk space or by a hard limit on the maximum number of indexes, document, or otherhigh-level resources, whichever comes first. The following table documents storage limits. For maximum limits onindexes, documents, and other objects, see limits by resource.

1 2

3

Basic has one fixed partition. At this tier, additional SUs are used for allocating more replicas for increased queryworkloads.

1

S3 HD has a hard limit of 3 partitions, which is lower than the partition limit for S3. The lower partition limit isimposed because the index count for S3 HD is substantially higher. Given that service limits exist for bothcomputing resources (storage and processing) and content (indexes and documents), the content limit is reachedfirst.

2

Service level agreements (SL As) are offered for billable services on dedicated resources. Free services andpreview features have no SL A. For billable services, SL As take effect when you provision sufficient redundancyfor your service. Two or more replicas are required for query (read) SL A. Three or more replicas are required forquery and indexing (read-write) SL A. The number of partitions is not an SL A consideration.

3

To learn more about limits on a more granular level, such as document size, queries per second, keys, requests,and responses, see Service limits in Azure Search.

For resources that are not fixed, you may ask for the quotas to be raised, by opening a support ticket. Do not createadditional Azure Media Services accounts in an attempt to obtain higher limits.

https://docs.microsoft.com/en-us/azure/search/search-capacity-planning

https://docs.microsoft.com/en-us/azure/search/search-limits-quotas-capacity

https://docs.microsoft.com/en-us/azure/search/search-limits-quotas-capacity


Azure Media Services (AMS) accounts in a single subscription 25 (fixed)

Media Reserved Units (RUs) per AMS account 25 (S1)10 (S2, S3)

Jobs per AMS account 50,000

Chained tasks per job 30 (fixed)

Assets per AMS account 1,000,000

Assets per task 50

Assets per job 100

Unique locators associated with an asset at one time 5

Live channels per AMS account 5

Programs in stopped state per channel 50

Programs in running state per channel 3

Streaming endpoints in running state per AMS account 2

Streaming units per streaming endpoint 10

Storage accounts 1,000 (fixed)

Policies 1,000,000

File size In some scenarios, there is a limit on the maximum file sizesupported for processing in Media Services.

(1)

(2)

(4)

(5)

(6)

7

If you change the type (for example, from S2 to S1,) the max RU limits are reset.1

This number includes queued, finished, active, and canceled jobs. It does not include deleted jobs. You can deletethe old jobs using IJob.Delete or the DELETE HTTP request.

2

As of April 1, 2017, any Job record in your account older than 90 days will be automatically deleted, along with itsassociated Task records, even if the total number of records is below the maximum quota. If you need to archivethe job/task information, you can use the code described here.

When making a request to list Job entities, a maximum of 1,000 jobs is returned per request. If you need to keeptrack of all submitted Jobs, you can use top/skip as described in OData system query options.

3

Locators are not designed for managing per-user access control. To give different access rights to individualusers, use Digital Rights Management (DRM) solutions. For more information, see this section.

4

The storage accounts must be from the same Azure subscription.5

There is a limit of 1,000,000 policies for different AMS policies (for example, for Locator policy orContentKeyAuthorizationPolicy).

6

https://docs.microsoft.com/en-us/azure/media-services/media-services-dotnet-manage-entities

http://msdn.microsoft.com/library/gg309461.aspx

https://docs.microsoft.com/en-us/azure/media-services/media-services-content-protection-overview

NOTENOTE

MEDIA RESERVED UNIT TYPE MAXIMUM INPUT SIZE (GB)

S1 325

S2 640

S3 260

CDN limitsCDN limits


CDN profiles 25 25

CDN endpoints per profile 10 25

Custom domains per endpoint 10 25

Mobile Services limitsMobile Services limits

TIER: FREE BASIC STANDARD

API Calls 500 K 1.5 M / unit 15 M / unit

Active Devices 500 Unlimited Unlimited

Scale N/A Up to 6 units Unlimited units

Push Notifications Notification Hubs Free Tierincluded, up to 1 M pushes

Notification Hubs Basic Tierincluded, up to 10 M pushes

Notification Hubs StandardTier included, up to 10 Mpushes

You should use the same policy ID if you are always using the same days / access permissions / etc. For information and anexample, see this section.

If you are uploading content to an Asset in Azure Media Services to process it with one of the media processorsin the service (that is, encoders like Media Encoder Standard and Media Encoder Premium Workflow, or analysisengines like Face Detector), then you should be aware of the constraints on the maximum file sizes supported.

7

The maximum size supported for a single blob is currently up to 5 TB in Azure Blob Storage. However, additionallimits apply in Azure Media Services based on the VM sizes that are used by the service. The following tableshows the limits on each of the Media Reserved Units (S1, S2, S3.) If your source file is larger than the limitsdefined in the table, your encoding job will fail. If you are encoding 4K resolution sources of long duration, you arerequired to use S3 Media Reserved Units to achieve the performance needed. If you have 4K content that is largerthan 260 GB limit on the S3 Media Reserved Units, contact us at [email protected] for potentialmitigations to support your scenario.

A CDN subscription can contain one or more CDN profiles and a CDN profile can contain one or more CDNendpoints. You may wish to use multiple profiles to organize your CDN endpoints by internet domain, webapplication, or some other criteria.

To request an update to your subscription's default limits, open a support ticket.

https://docs.microsoft.com/en-us/azure/media-services/media-services-dotnet-manage-entities

Real time messaging/Web Sockets

Limited 350 / mobile service Unlimited

Offline synchronizations Limited Included Included

Scheduled jobs Limited Included Included

SQL Database (required) Standard rates apply foradditional capacity

20 MB included 20 MB included 20 MB included

CPU capacity 60 minutes / day Unlimited Unlimited

Outbound data transfer 165 MB per day (dailyRollover)

Included Included


Monitor limitsMonitor limits


Autoscale Settings 100 per region per subscription same as default

Metric Alerts 100 active alert rules per subscription call support

Near-Real Time Alerts (Preview) 20 active alert rules per subscription same as default during preview

Notification Hub Service limitsNotification Hub Service limits


Included Pushes 1 Million 10 Million 10 Million

Active Devices 500 200,000 10 million

Tag quota perinstallation/registration

60 60 60

Event Hubs limitsEvent Hubs limits

LIMIT SCOPE NOTES VALUE

Number of event hubs pernamespace

Namespace Subsequent requests forcreation of a new event hubwill be rejected.

10

Number of partitions perevent hub

Entity - 32

For additional details on these limits and for information on pricing, see Mobile Services Pricing.

For additional details on these limits and for information on pricing, see Notification Hubs Pricing.

The following table lists quotas and limits specific to Azure Event Hubs. For information about Event Hubs pricing,see Event Hubs pricing.

https://azure.microsoft.com/pricing/details/mobile-services/

https://azure.microsoft.com/pricing/details/notification-hubs/

https://azure.microsoft.com/services/event-hubs/

https://azure.microsoft.com/pricing/details/event-hubs/

Number of consumergroups per event hub

Entity - 20

Number of AMQPconnections per namespace

Namespace Subsequent requests foradditional connections willbe rejected and an exceptionis received by the callingcode.

5,000

Maximum size of EventHubs event

Entity - 256 KB

Maximum size of an eventhub name

Entity - 50 characters

Number of non-epochreceivers per consumergroup

Entity - 5

Maximum retention periodof event data

Entity - 1-7 days

Maximum throughput units Namespace Exceeding the throughputunit limit causes your datato be throttled andgenerates aServerBusyException. Youcan request a larger numberof throughput units for aStandard tier by filing asupport request. Additionalthroughput units areavailable in blocks of 20 on acommitted purchase basis.

20

Number of authorizationrules per namespace

Namespace Subsequent requests forauthorization rule creationwill be rejected.

12

LIMIT SCOPE NOTES VALUE

Service Bus limitsService Bus limits

QUOTA NAME SCOPE NOTES VALUE

Maximum number of basic /standard namespaces perAzure subscription

Namespace Subsequent requests foradditional basic / standardnamespaces are rejected bythe portal.

100

Maximum number ofpremium namespaces perAzure subscription

Namespace Subsequent requests foradditional premiumnamespaces are rejected bythe portal.

10

The following table lists quota information specific to Service Bus messaging. For information about pricing andother quotas for Service Bus, see the Service Bus Pricing overview.

https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.serverbusyexception

https://docs.microsoft.com/azure/azure-supportability/how-to-create-azure-support-request

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-auto-inflate

https://azure.microsoft.com/pricing/details/service-bus/

Queue/topic size Entity Defined upon creation of thequeue/topic.

Subsequent incomingmessages are rejected andan exception is received bythe calling code.

1, 2, 3, 4 or 5 GB.

If partitioning is enabled, themaximum queue/topic size is80 GB.

Number of concurrentconnections on a namespace

Namespace Subsequent requests foradditional connections arerejected and an exception isreceived by the calling code.REST operations do notcount towards concurrentTCP connections.

NetMessaging: 1,000

AMQP: 5,000

Number of concurrentreceive requests on aqueue/topic/subscriptionentity

Entity Subsequent receive requestsare rejected and anexception is received by thecalling code. This quotaapplies to the combinednumber of concurrentreceive operations across allsubscriptions on a topic.

5,000

Number of topics/queuesper service namespace

Namespace Subsequent requests forcreation of a new topic orqueue on the servicenamespace are rejected. As aresult, if configured throughthe Azure portal, an errormessage is generated. Ifcalled from the managementAPI, an exception is receivedby the calling code.

10,000

The total number of topicsplus queues in a servicenamespace must be lessthan or equal to 10,000.This is not applicable toPremium as all entities arepartitioned.

Number of partitionedtopics/queues per servicenamespace

Namespace Subsequent requests forcreation of a newpartitioned topic or queueon the service namespaceare rejected. As a result, ifconfigured through theAzure portal, an errormessage is generated. Ifcalled from the managementAPI, aQuotaExceededExceptionexception is received by thecalling code.

Basic and Standard Tiers -100Premium - 1,000 (permessaging unit)

Each partitioned queue ortopic counts towards thequota of 10,000 entities pernamespace.

Maximum size of anymessaging entity path:queue or topic


Maximum size of anymessaging entity name:namespace, subscription, orsubscription rule



https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-partitioning



https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-premium-messaging

Message size for aqueue/topic/subscriptionentity

Entity Incoming messages thatexceed these quotas arerejected and an exception isreceived by the calling code.

Maximum message size: 256KB (Standard tier) / 1 MB(Premium tier).

Note Due to systemoverhead, this limit is usuallyslightly less.

Maximum header size: 64 KB

Maximum number of headerproperties in property bag:byte/int.MaxValue

Maximum size of property inproperty bag: No explicitlimit. Limited by maximumheader size.

Message property size for aqueue/topic/subscriptionentity

Entity A SerializationExceptionexception is generated.

Maximum message propertysize for each property is 32K. Cumulative size of allproperties cannot exceed 64K. This applies to the entireheader of theBrokeredMessage, which hasboth user properties as wellas system properties (suchas SequenceNumber, Label,MessageId, and so on).

Number of subscriptions pertopic

Entity Subsequent requests forcreating additionalsubscriptions for the topicare rejected. As a result, ifconfigured through theportal, an error message isshown. If called from themanagement API anexception is received by thecalling code.

2,000

Number of SQL filters pertopic

Entity Subsequent requests forcreation of additional filterson the topic are rejected andan exception is received bythe calling code.

2,000

Number of correlation filtersper topic

Entity Subsequent requests forcreation of additional filterson the topic are rejected andan exception is received bythe calling code.

100,000




https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.brokeredmessage

https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.brokeredmessage.sequencenumber

https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.brokeredmessage.label

https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.brokeredmessage.messageid

Size of SQL filters/actions Namespace Subsequent requests forcreation of additional filtersare rejected and anexception is received by thecalling code.

Maximum length of filtercondition string: 1024 (1K).

Maximum length of ruleaction string: 1024 (1K).

Maximum number ofexpressions per rule action:32.

Number ofSharedAccessAuthorizationRule rules per namespace,queue, or topic

Entity, namespace Subsequent requests forcreation of additional rulesare rejected and anexception is received by thecalling code.

Maximum number of rules:12.

Rules that are configured ona Service Bus namespaceapply to all queues andtopics in that namespace.

Number of messages pertransaction

Transaction Additional incomingmessages are rejected andan exception stating"Cannot send more than100 messages in a singletransaction" is received bythe calling code.

100

For both Send() andSendAsync() operations.


IoT Hub limitsIoT Hub limits

RESOURCE S1 STANDARD S2 STANDARD S3 STANDARD F1 FREE

Messages/day 400,000 6,000,000 300,000,000 8,000

Maximum units 200 200 10 1

NOTENOTE

RESOURCE LIMIT

Maximum paid IoT hubs per Azure subscription 10

Maximum free IoT hubs per Azure subscription 1

Maximum number of characters in a Device Id 128

Maximum number of device identitiesreturned in a single call

1000

The following table lists the limits associated with the different service tiers (S1, S2, S3, F1). For information aboutthe cost of each unit in each tier, see IoT Hub Pricing.

If you anticipate using more than 200 units with an S1 or S2 or 10 units with an S3 tier hub, contact Microsoft support.

The following table lists the limits that apply to IoT Hub resources:

https://docs.microsoft.com/dotnet/api/microsoft.servicebus.messaging.sharedaccessauthorizationrule

https://azure.microsoft.com/pricing/details/iot-hub/

IoT Hub message maximum retention for device-to-cloudmessages

7 days

Maximum size of device-to-cloud message 256 KB

Maximum size of device-to-cloud batch 256 KB

Maximum messages in device-to-cloud batch 500

Maximum size of cloud-to-device message 64 KB

Maximum TTL for cloud-to-device messages 2 days

Maximum delivery count for cloud-to-device messages

100

Maximum delivery count for feedback messages in response to a cloud-to-device message

100

Maximum TTL for feedback messages in response to a cloud-to-device message

2 days

Maximum size of device twin (tags, reported properties, and desired properties)

8 KB

Maximum size of device twin string value 4 KB

Maximum depth of object in device twin 5

Maximum size of direct method payload 128 KB

Job history maximum retention 30 days

Maximum concurrent jobs 10 (for S3), 5 for (S2), 1 (for S1)

Maximum additional endpoints 10 (for S1, S2, S3)

Maximum message routing rules 100 (for S1, S2, S3)

RESOURCE LIMIT

NOTENOTE

NOTENOTE

If you need more than 10 paid IoT hubs in an Azure subscription, contact Microsoft support.

Currently, the maximum number of devices you can connect to a single IoT hub is 500,000. If you want to increase this limit,contact Microsoft Support.

The IoT Hub service throttles requests when the following quotas are exceeded:

https://azure.microsoft.com/en-us/support/options/

THROTTLE PER-HUB VALUE

Identity registry operations (create, retrieve, list, update, delete), individual or bulk import/export

83.33/sec/unit (5000/min/unit) (for S3) 1.67/sec/unit (100/min/unit) (for S1 and S2).

Device connections 6000/sec/unit (for S3), 120/sec/unit (for S2), 12/sec/unit (forS1). Minimum of 100/sec.

Device-to-cloud sends 6000/sec/unit (for S3), 120/sec/unit (for S2), 12/sec/unit (forS1). Minimum of 100/sec.

Cloud-to-device sends 83.33/sec/unit (5000/min/unit) (for S3), 1.67/sec/unit(100/min/unit) (for S1 and S2).

Cloud-to-device receives 833.33/sec/unit (50000/min/unit) (for S3), 16.67/sec/unit(1000/min/unit) (for S1 and S2).

File upload operations 83.33 file upload notifications/sec/unit (5000/min/unit) (forS3), 1.67 file upload notifications/sec/unit (100/min/unit) (forS1 and S2). 10000 SAS URIs can be out for an Azure Storage account atone time.10 SAS URIs/device can be out at one time.

Direct methods 24MB/sec/unit (for S3), 480KB/sec/unit (for S2),160KB/sec/unit (for S1)Based on 8KB throttling meter size.

Device twin reads 50/sec/unit (for S3), Maximum of 10/sec or 1/sec/unit (for S2),10/sec (for S1)

Device twin updates 50/sec/unit (for S3), Maximum of 10/sec or 1/sec/unit (for S2),10/sec (for S1)

Jobs operations (create, update, list, delete)

83.33/sec/unit (5000/min/unit) (for S3), 1.67/sec/unit(100/min/unit) (for S2), 1.67/sec/unit (100/min/unit) (for S1)

Jobs per-device operation throughput 50/sec/unit (for S3), Maximum of 10/sec or 1/sec/unit (for S2),10/sec (for S1)

IoT Hub Device Provisioning Service limitsIoT Hub Device Provisioning Service limits

RESOURCE LIMIT

Maximum Device Provisioning Services per Azure subscription 10

Maximum number of enrollments 10,000

Maximum number of registrations 10,000

Maximum number of enrollment groups 100

The following table lists the limits that apply to IoT Hub Device Provisioning Service resources:

Maximum number of CAs 10

RESOURCE LIMIT

NOTENOTE

THROTTLE PER-SERVICE VALUE

Operations 100/min

Device registrations 100/min

Data Factory limitsData Factory limits

Version 2Version 2


Data factories in an Azure subscription 50 Contact support

Pipelines within a data factory 2500 Contact support

Datasets within a data factory 2500 Contact support

Triggers within a data factory 2500 Contact support

Linked services within a data factory 2500 Contact support

Integration runtimes within a datafactory

2500 Contact support

Concurrent pipeline runs per pipeline 20 Contact support

Max activities per pipeline 20 30

Max parameters per pipeline 20 30

Bytes per object for pipeline objects 200 KB 200 KB

Bytes per object for dataset and linkedservice objects

100 KB 2000 KB

Cloud data movement units 256 Contact support

Retry count for pipeline activity runs 1 day(timeout) 1 day (timeout)

You can contact Microsoft Support to increase the number of instances in your subscription.

The Device Provisioning Service throttles requests when the following quotas are exceeded:

Data factory is a multi-tenant service that has the following default limits in place to make sure customersubscriptions are protected from each other's workloads. Many of the limits can be easily raised for yoursubscription up to the maximum limit by contacting support.

4

1

1

3

https://azure.microsoft.com/support/options/

https://azure.microsoft.com/blog/2014/06/04/azure-limits-quotas-increase-requests/








Write API calls 2500/hr

This limit is imposed by Azure ResourceManager, not Azure Data Factory.

Contact support.

Read API calls 12,500/hr

This limit is imposed by Azure ResourceManager, not Azure Data Factory.

Contact support


Version 1Version 1


Data factories in an Azure subscription 50 Contact support

Pipelines within a data factory 2500 Contact support

Datasets within a data factory 5000 Contact support

Concurrent slices per dataset 10 10

Bytes per object for pipeline objects 200 KB 200 KB

Bytes per object for dataset and linkedservice objects

100 KB 2000 KB

HDInsight on-demand cluster coreswithin a subscription

60 Contact support

Cloud data movement units 32 Contact support

Retry count for pipeline activity runs 1000 MaxInt (32 bit)

1

1

2

3

Pipeline, dataset, and linked service objects represent a logical grouping of your workload. Limits for theseobjects do not relate to amount of data you can move and process with the Azure Data Factory service. Datafactory is designed to scale to handle petabytes of data.

1

On-demand HDInsight cores are allocated out of the subscription that contains the data factory. As a result, theabove limit is the Data Factory enforced core limit for on-demand HDInsight cores and is different from the corelimit associated with your Azure subscription.

2

Cloud data movement unit (DMU) is being used in a cloud-to-cloud copy operation. It is a measure thatrepresents the power (a combination of CPU, memory, and network resource allocation) of a single unit in DataFactory. You can achieve higher copy throughput by using more DMUs for some scenarios. Refer to Cloud datamovement units (V2) and Cloud data movement units (V1) section on details.

3

The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the followingdata integration capabilities across different network environments: data movement, dispatching activities tocompute services, execution of SSIS packages. For more information, see Integration Runtime overview.

4








https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance

https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-copy-activity-performance

https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime

RESOURCE DEFAULT LOWER LIMIT MINIMUM LIMIT

Scheduling interval 15 minutes 15 minutes

Interval between retry attempts 1 second 1 second

Retry timeout value 1 second 1 second

Web service call limitsWeb service call limits

Data Lake Analytics limitsData Lake Analytics limits

RESOURCE DEFAULT LIMIT COMMENTS

Maximum number of concurrent jobs 20

Maximum number of Analytics Units(AUs) per account

250 Use any combination of up to amaximum of 250 AUs across 20 jobs.Contact Microsoft support to increasethis limit.

Maximum script size for job submission 3 MB

Maximum number of ADLA accountsper region per subscription

5 Contact Microsoft support to increasethis limit.

Data Lake Store limitsData Lake Store limits


Max number of Data Lake Storeaccounts, per subscription, per region

10 Contact Support to request an increasefor this limit

Max number of access ACLs, per file orfolder

32 This is a hard limit. Use groups tomanage access with fewer entries

Max number of default ACLs, per file orfolder

32 This is a hard limit. Use groups tomanage access with fewer entries

Database Migration Service LimitsDatabase Migration Service Limits

Azure Resource Manager has limits for API calls. You can make API calls at a rate within the Azure ResourceManager API limits.

Data Lake Analytics makes the complex task of managing distributed infrastructure and complex code easy. Itdynamically provisions resources and lets you do analytics on exabytes of data. When the job completes, it windsdown resources automatically, and you pay only for the processing power used. As you increase or decrease thesize of data stored or the amount of compute used, you donâ€™t have to rewrite code. Many of the default limitscan be easily raised for your subscription by contacting support.

Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads. Data LakeStore enables you to capture data of any size, type, and ingestion speed in one single place for operational andexploratory analytics. There is no limit to the amount of data you can store in a Data Lake Store account.

The Azure Database Migration Service is a fully managed service designed to enable seamless migrations frommultiple database sources to Azure Data platforms with minimal downtime.


Maximum number of services persubscription, per region

2 Contact Support to request an increasefor this limit

Stream Analytics limitsStream Analytics limits

LIMIT IDENTIFIER LIMIT COMMENTS

Maximum number of Streaming Unitsper subscription per region

200 A request to increase streaming unitsfor your subscription beyond 200 canbe made by contacting MicrosoftSupport.

Maximum number of inputs per job 60 There is a hard limit of 60 inputs perStream Analytics job.

Maximum number of outputs per job 60 There is a hard limit of 60 outputs perStream Analytics job.

Maximum number of functions per job 60 There is a hard limit of 60 functions perStream Analytics job.

Maximum number of Streaming Unitsper job

120 There is a hard limit of 120 StreamingUnits per Stream Analytics job.

Maximum number of jobs per region 1500 Each subscription may have up to 1500jobs per geographical region.

Reference data blob MB 100 Reference data blobs cannot be largerthan 100 MB each.

Active Directory limitsActive Directory limits

CATEGORY LIMITS

Directories A single user can only be associated with a maximum of 20Azure Active Directory directories.Examples of possible combinations:

Objects

Here are the usage constraints and other service limits for the Azure Active Directory service.

A single user creates 20 directories.A single user is added to 20 directories as a member.A single user creates 10 directories and later is addedby others to 10 different directories.

A maximum of 500,000 objects can be created in asingle directory by users of the Free edition of AzureActive Directory.A non-admin user can create no more than 250objects.

https://support.microsoft.com/en-us

Schema extensions

Applications A maximum of 100 users can be owners of a singleapplication.

Groups

Access Panel

Reports A maximum of 1,000 rows can be viewed or downloaded inany report. Any additional data is truncated.

Administrative units An object can be a member of no more than 30administrative units.

CATEGORY LIMITS

Azure Event Grid limitsAzure Event Grid limits

RESOURCE LIMIT

Event Subscriptions per region 1000

Custom Topics per region 20

String type extensions can have maximum of 256characters.Binary type extensions are limited to 256 bytes.100 extension values (across ALL types and ALLapplications) can be written to any single object.Only â€œUserâ€, â€œGroupâ€, â€œTenantDetailâ€,â€œDeviceâ€, â€œApplicationâ€ andâ€œServicePrincipalâ€ entities can be extended withâ€œStringâ€ type or â€œBinaryâ€ type single-valuedattributes.Schema extensions are available only in Graph API-version 1.21-preview. The application must be grantedwrite access to register an extension.

A maximum of 100 users can be owners of a singlegroup.Any number of objects can be members of a singlegroup in Azure Active Directory.The number of members in a group you cansynchronize from your on-premises Active Directoryto Azure Active Directory is limited to 15K members,using Azure Active Directory DirectorySynchronization (DirSync).The number of members in a group you cansynchronize from your on-premises Active Directoryto Azure Active Directory using Azure AD Connect islimited to 50K members.

There is no limit to the number of applications thatcan be seen in the Access Panel per end user, for usersassigned licenses for Azure AD Premium or theEnterprise Mobility Suite.A maximum of 10 app tiles (examples: Box, Salesforce,or Dropbox) can be seen in the Access Panel for eachend user for users assigned licenses for Free or AzureAD Basic editions of Azure Active Directory. This limitdoes not apply to Administrator accounts.

StorSimple System limitsStorSimple System limits


Maximum number of storage accountcredentials

64

Maximum number of volumecontainers

64

Maximum number of volumes 255

Maximum number of schedules perbandwidth template

168 A schedule for every hour, every day ofthe week (24*7).

Maximum size of a tiered volume onphysical devices

64 TB for 8100 and 8600 8100 and 8600 are physical devices.

Maximum size of a tiered volume onvirtual devices in Azure

30 TB for 8010 64 TB for 8020

8010 and 8020 are virtual devices inAzure that use Standard Storage andPremium Storage respectively.

Maximum size of a locally pinnedvolume on physical devices

9 TB for 8100 24 TB for 8600

8100 and 8600 are physical devices.

Maximum number of iSCSI connections 512

Maximum number of iSCSI connectionsfrom initiators

512

Maximum number of access controlrecords per device

64

Maximum number of volumes perbackup policy

24

Maximum number of backups retainedper backup policy

64

Maximum number of schedules perbackup policy

10

Maximum number of snapshots of anytype that can be retained per volume

256 This includes local snapshots and cloudsnapshots.

Maximum number of snapshots thatcan be present in any device

10,000

Maximum number of volumes that canbe processed in parallel for backup,restore, or clone

16

Restore and clone recover time fortiered volumes

< 2 minutes


If there are more than 16volumes, they will be processedsequentially as processing slotsbecome available.New backups of a cloned or arestored tiered volume cannotoccur until the operation isfinished. However, for a localvolume, backups are allowedafter the volume is online.

The volume is made availablewithin 2 minutes of restore orclone operation, regardless ofthe volume size.The volume performance mayinitially be slower than normalas most of the data andmetadata still resides in thecloud. Performance may increaseas data flows from the cloud tothe StorSimple device.The total time to downloadmetadata depends on theallocated volume size. Metadatais automatically brought intothe device in the background atthe rate of 5 minutes per TB ofallocated volume data. This ratemay be affected by Internetbandwidth to the cloud.The restore or clone operation iscomplete when all the metadatais on the device.Backup operations cannot beperformed until the restore orclone operation is fullycomplete.

Restore recover time for locally pinnedvolumes

< 2 minutes

Thin-restore availability Last failover

Maximum client read/write throughput(when served from the SSD tier)*

920/720 MB/s with a single 10GbEnetwork interface

Up to 2x with MPIO and two networkinterfaces.

Maximum client read/write throughput(when served from the HDD tier)*

120/250 MB/s

Maximum client read/write throughput(when served from the cloud tier)*

11/41 MB/s Read throughput depends on clientsgenerating and maintaining sufficientI/O queue depth.


Log Analytics limitsLog Analytics limits

The volume is made availablewithin 2 minutes of the restoreoperation, regardless of thevolume size.The volume performance mayinitially be slower than normalas most of the data andmetadata still resides in thecloud. Performance may increaseas data flows from the cloud tothe StorSimple device.The total time to downloadmetadata depends on theallocated volume size. Metadatais automatically brought intothe device in the background atthe rate of 5 minutes per TB ofallocated volume data. This ratemay be affected by Internetbandwidth to the cloud.Unlike tiered volumes, in thecase of locally pinned volumes,the volume data is alsodownloaded locally on thedevice. The restore operation iscomplete when all the volumedata has been brought to thedevice.The restore operations may belong and the total time tocomplete the restore willdepend on the size of theprovisioned local volume, yourInternet bandwidth and theexisting data on the device.Backup operations on the locallypinned volume are allowed whilethe restore operation is inprogress.

* Maximum throughput per I/O type was measured with 100 percent read and 100 percent write scenarios. Actualthroughput may be lower and depends on I/O mix and network conditions.

The following limits apply to Log Analytics resources per subscription:


Number of free workspaces persubscription

10 This limit cannot be increased.

Number of paid workspaces persubscription

N/A You are limited by the number ofresources within a resource group andnumber of resource groups persubscription

NOTENOTE

FREE STANDARD PREMIUM STANDALONE OMS PER GB

Data volumecollected perday

500 MB None None None None None

Dataretentionperiod

7 days 1 month 12 months 1 month 1 month 1 month

CATEGORY LIMITS COMMENTS

Data Collector API Maximum size for a single post is 30MBMaximum size for field values is 32 KB

Split larger volumes into multiple postsFields longer than 32 KB are truncated.

Search API 5000 records returned for non-aggregated data500000 records for aggregated data

Aggregated data is a search thatincludes the measure command

Backup limitsBackup limits

LIMIT IDENTIFIER DEFAULT LIMIT

Number of servers/machines that can be registered againsteach vault

50 for Windows Server/Client/SCDPM 200 for IaaS VMs

Size of a data source for data stored in Azure vault storage 54400 GB max

Number of backup vaults that can be created in each Azuresubscription

25 Recovery Services vaults per region

As of April 2, 2018, new workspaces in a new subscription will automatically use the Per GB pricing plan. For existingsubscriptions created before April 2, or a subscription that was tied to an existing EA enrollment, you can continue choosingbetween the three pricing tiers for new workspaces.

The following limits apply to each Log Analytics workspace:

1

2 2 2

When customers reach their 500 MB daily data transfer limit, data analysis stops and resumes at the start of thenext day. A day is based on UTC.

1

The data retention period for the Standalone, OMS, and Per GB pricing plans can be increased to 730 days.2

The following limits apply to Azure Backup.

1

Number of times backup can be scheduled per day 3 per day for Windows Server/Client 2 per day for SCDPM Once a day for IaaS VMs

Data disks attached to an Azure virtual machine for backup 16

Size of individual data disk attached to an Azure virtualmachine for backup

4095 GB


Site Recovery limitsSite Recovery limits


Number of vaults per subscription 25

Number of servers per Azure vault 250

Number of protection groups per Azure vault No limit

Number of recovery plans per Azure vault No limit

Number of servers per protection group No limit

Number of servers per recovery plan 50

Application Insights limitsApplication Insights limits


Total data per day 100 GB You can reduce data by setting a cap. Ifyou need more, you can increase thelimit up to 1,000 GB from the portal.For capacities greater than 1,000 GB,send mail [email protected].

Free data per month(Basic price plan)

1 GB Additional data is charged per gigabyte.

Throttling 32 k events/second The limit is measured over a minute.

Data retention 90 days This resource is for Search, Analytics,and Metrics Explorer.

Availability multi-step test detailedresults retention

90 days This resource provides detailed resultsof each step.

2

The 54400 GB limit does not apply to IaaS VM backup.1

The following limits apply to Azure Site Recovery:

There are some limits on the number of metrics and events per application (that is, per instrumentation key).Limits depend on the pricing plan that you choose.

https://azure.microsoft.com/pricing/details/application-insights/

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-diagnostic-search

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-metrics-explorer

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-web-app-availability

Maximum event size 64 K

Property and metric name length 150 See type schemas

Property value string length 8,192 See type schemas

Trace and exception message length 10 k See type schemas

Availability tests count per app 100

Profiler data retention 5 days

Profiler data sent per day 10GB


API Management limitsAPI Management limits

RESOURCE LIMIT

Units of scale 10 per region

Cache 5 GB per unit

Concurrent backend connections per HTTP authority 2048 per unit

Maximum cached response size 10MB

Maximum custom gateway domains 20 per service instance

Azure Redis Cache limitsAzure Redis Cache limits

RESOURCE LIMIT

Cache size 530 GB

Databases 64

Max connected clients 40,000

Redis Cache replicas (for high availability) 1

Shards in a premium cache with clustering 10

For more information, see About pricing and quotas in Application Insights.

1

1

2 3

4

API Management limits are different for each pricing tier. To see the pricing tiers and their scaling limits go to APIManagement Pricing. Connections are pooled and re-used, unless explicitly closed by the backend. Per unit ofBasic, Standard and Premium tiers. Developer tier is limited to 1024. Available in Premium tier only.

1

2 3

4

Azure Redis Cache limits and sizes are different for each pricing tier. To see the pricing tiers and their associatedsizes, see Azure Redis Cache Pricing.

For more information on Azure Redis Cache configuration limits, see Default Redis server configuration.

https://github.com/Microsoft/ApplicationInsights-Home/blob/master/EndpointSpecs/Schemas/Docs/



https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-web-app-availability

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-profiler

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-profiler

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-pricing

https://azure.microsoft.com/pricing/details/api-management/

https://azure.microsoft.com/pricing/details/cache/

https://docs.microsoft.com/en-us/azure/redis-cache/cache-configure

Key Vault limitsKey Vault limits

KEY TYPEHSM-KEYCREATE KEY

HSM-KEYALL OTHERTRANSACTIONS

SOFTWARE-KEYCREATE KEY

SOFTWARE-KEYALL OTHERTRANSACTIONS

RSA 2048-bit 5 1000 10 2000

RSA 3072-bit 5 250 10 500

RSA 4096-bit 5 125 10 250

ECC P-256 5 1000 10 2000

ECC P-384 5 1000 10 2000

ECC P-521 5 1000 10 2000

ECC SECP256K1 5 1000 10 2000

TRANSACTIONS TYPEMAX TRANSACTIONS ALLOWED IN 10 SECONDS, PER VAULT PERREGION

All transactions 2000

Multi-Factor AuthenticationMulti-Factor Authentication


Max number of Trusted IPaddresses/ranges per subscription

0 50

Remember my devices - number ofdays

14 60

Max number of app passwords? 0 No Limit

Allow X attempts during MFA call 1 99

Because configuration and management of Azure Redis Cache instances is done by Microsoft, not all Rediscommands are supported in Azure Redis Cache. For more information, see Redis commands not supported inAzure Redis Cache.

Key transactions (Max transactions allowed in 10 seconds, per vault per region ):1

Secrets, Managed Storage Account Keys, and vault transactions:

1

See Azure Key Vault throttling guidance for information on how to handle throttling when these limits areexceeded.

There is a subscription-wide limit for all transaction types, that is 5x per key vault limit. For example, HSM- othertransactions per subscription are limited to 5000 transactions in 10 seconds per subscription.

1

https://docs.microsoft.com/en-us/azure/redis-cache/cache-configure

https://docs.microsoft.com/en-us/azure/key-vault/key-vault-ovw-throttling

Two-way Text message TimeoutSeconds

60 600

Default one-time bypass seconds 300 1800

Lock user account after X consecutiveMFA denials

Not Set 99

Reset account lockout counter after Xminutes

Not Set 9999

Unlock account after X minutes Not Set 9999


Automation limitsAutomation limits

RESOURCE MAXIMUM LIMIT

Max number of new jobs that can be submitted every 30seconds per Automation Account (non Scheduled jobs)

100

Max number of concurrent running jobs at the same instanceof time per Automation Account (non Scheduled jobs)

200

Max number of modules that can be imported every 30seconds per Automation Account

5

Max size of a Module 100 MB

Job Run Time - Free tier 500 minutes per subscription per calendar month

Max amount of memory given to a job 400 MB

Max number of network sockets allowed per job 1000

Max number of Automation Accounts in a subscription No Limit

SQL Database limitsSQL Database limits

SQL Data Warehouse limitsSQL Data Warehouse limits

See also

1

1

These limits apply only to the sandboxes in Azure, for Hybrid workers these are only limited by the capabilities ofthe machine where the hybrid worker is located.

1

For SQL Database limits, see SQL Database Resource Limits.

For SQL Data Warehouse limits, see SQL Data Warehouse Resource Limits.

Understanding Azure Limits and Increases

Virtual Machine and Cloud Service Sizes for Azure

Sizes for Cloud Services

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-resource-limits

https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-service-capacity-limits


https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes

https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-sizes-specs

ExpressRoute for Cloud Solution Providers (CSP)4/11/2018 • 9 min to read • Edit Online

NOTENOTE

Microsoft Azure management

Microsoft Azure resource management

Connect-through modelConnect-through model

Microsoft provides hyper-scale services for traditional resellers and distributors (CSP) to be able to rapidlyprovision new services and solutions for your customers without the need to invest in developing these newservices. To allow the Cloud Solution Provider (CSP) the ability to directly manage these new services, Microsoftprovides programs and APIs that allow the CSP to manage Microsoft Azure resources on behalf of yourcustomers. One of those resources is ExpressRoute. ExpressRoute allows the CSP to connect existing customerresources to Azure services. ExpressRoute is a high speed private communications link to services in Azure.

ExpressRoute is comprised of a pair of circuits for high availability that are attached to a single customer'ssubscription(s) and cannot be shared by multiple customers. Each circuit should be terminated in a different routerto maintain the high availability.

There are bandwidth and connection caps on ExpressRoute which means that large/complex implementations will requiremultiple ExpressRoute circuits for a single customer.

Microsoft Azure provides a growing number of services that you can offer to your customers. ExpressRoute helpsyou and your customers take advantage of these services by providing high speed low latency access to theMicrosoft Azure environment.

Microsoft provides CSPs with APIs to manage the Azure customer subscriptions by allowing programmaticintegration with your own service management systems. Supported management capabilities can be found here.

The contract you have with your customer will determine how the subscription will be managed. The CSP candirectly manage the creation and maintenance of resources or the customer can maintain control of the MicrosoftAzure subscription and create the Azure resources as they need. If your customer manages the creation ofresources in their Microsoft Azure subscription they will use one of two models: â€œConnect-Throughâ€ model,or â€œDirect-Toâ€ model. These models are described in detail in the following sections.

https://github.com/Microsoft/azure-docs/blob/master/articles/expressroute/expressroute-for-cloud-solution-providers.md

https://msdn.microsoft.com/library/partnercenter/dn974944.aspx

Connect-to modelConnect-to model

In the connect-through model, the CSP creates a direct connection between your datacenter and yourcustomerâ€™s Azure subscription. The direct connection is made using ExpressRoute, connecting your networkwith Azure. Then your customer connects to your network. This scenario requires that the customer passesthrough the CSP network to access Azure services.

If your customer has other Azure subscriptions not managed by the you, they would use the public Internet or theirown private connection to connect to those services provisioned under the non CSP subscription.

For CSP managing Azure services, it is assumed that the CSP has a previously established customer identity storewhich would then be replicated into Azure Active Directory for management of their CSP subscription throughAdministrate-On-Behalf-Of (AOBO). Key drivers for this scenario include where a given partner or serviceprovider has an established relationship with the customer, the customer is consuming provider services currentlyor the partner has a desire to provide a combination of provider-hosted and Azure-hosted solutions to provideflexibility and address customer challenges which cannot be satisfied by CSP alone. This model is illustrated inFigure, below.

NOTENOTE

In the Connect-To model, the service provider creates a direct connection between their customerâ€™s datacenterand the CSP provisioned Azure subscription using ExpressRoute over the customerâ€™s (customer) network.

For ExpressRoute the customer would need to create and maintain the ExpressRoute circuit.

This connectivity scenario requires that the customer connects directly through a customer network to access CSP-managed Azure subscription, using a direct network connection that is created, owned and managed either whollyor in part by the customer. For these customers it is assumed that the provider does not currently have a customeridentity store established, and the provider would assist the customer in replicating their current identify store intoAzure Active Directory for management of their subscription through AOBO. Key drivers for this scenario includewhere a given partner or service provider has an established relationship with the customer, the customer isconsuming provider services currently, or the partner has a desire to provide services that are based solely onAzure-hosted solutions without the need for an existing provider datacenter or infrastructure.

The choice between these two option are based on your customerâ€™s needs and your current need to provideAzure services. The details of these models and the associated role-based access control, networking, and identitydesign patterns are covered in details in the following links:

Role Based Access Control (RBAC) â€“ RBAC is based on Azure Active Directory. For more information on

Network speeds

NOTENOTE

Configuring ExpressRoute

Connect-through modelConnect-through model

Connect-to modelConnect-to model

ExpressRoute routing domains

Azure RBAC see here.Networking â€“ Covers the various topics of networking in Microsoft Azure.Azure Active Directory (Azure AD) â€“ Azure AD provides the identity management for Microsoft Azureand 3rd party SaaS applications. For more information about Azure AD see here.

ExpressRoute supports network speeds from 50 Mb/s to 10Gb/s. This allows customers to purchase the amount ofnetwork bandwidth needed for their unique environment.

Network bandwidth can be increased as needed without disrupting communications, but to reduce the network speedrequires tearing down the circuit and recreating it at the lower network speed.

ExpressRoute supports the connection of multiple vNets to a single ExpressRoute circuit for better utilization of thehigher-speed connections. A single ExpressRoute circuit can be shared among multiple Azure subscriptions ownedby the same customer.

ExpressRoute can be configured to support three types of traffic (routing domains) over a single ExpressRoutecircuit. This traffic is segregated into Microsoft peering, Azure public peering and private peering. You can chooseone or all types of traffic to be sent over a single ExpressRoute circuit or use multiple ExpressRoute circuitsdepending on the size of the ExpressRoute circuit and isolation required by your customer. The security posture ofyour customer may not allow public traffic and private traffic to traverse over the same circuit.

In a connect-through configuration the you will be responsible for all of the networking underpinnings to connectyour customers datacenter resources to the subscriptions hosted in Azure. Each of your customer's that want to useAzure capabilities will need their own ExpressRoute connection, which will be managed by the You. The you willuse the same methods the customer would use to procure the ExpressRoute circuit. The you will follow the samesteps outlined in the article ExpressRoute workflows for circuit provisioning and circuit states. The you will thenconfigure the Border Gateway Protocol (BGP) routes to control the traffic flowing between the on-premisesnetwork and Azure vNet.

In a connect-to configuration, your customer already has an existing connection to Azure or will initiate aconnection to the internet service provider linking ExpressRoute from your customerâ€™s own datacenter directlyto Azure, instead of your datacenter. To begin the provisioning process, your customer will follow the steps asdescribed in the Connect-Through model, above. Once the circuit has been established your customer will need toconfigure the on-premises routers to be able to access both your network and Azure vNets.

You can assist with setting up the connection and configuring the routes to allow the resources in yourdatacenter(s) to communicate with the client resources in your datacenter, or with the resources hosted in Azure.

ExpressRoute offers three routing domains: public, private, and Microsoft peering. Each of the routing domains areconfigured with identical routers in active-active configuration for high availability. For more details onExpressRoute routing domains look here.

You can define custom routes filters to allow only the route(s) you want to allow or need. For more information orto see how to make these changes see article: Create and modify routing for an ExpressRoute circuit using

https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal

https://azure.microsoft.com/documentation/services/active-directory/

NOTENOTE

Routing

Default routingDefault routing

User-defined routing (UDR)User-defined routing (UDR)

PowerShell for more details about routing filters.

For Microsoft and Public Peering connectivity must be though a public IP address owned by the customer or CSP and mustadhere to all defined rules. For more information, see the ExpressRoute Prerequisites page.

ExpressRoute connects to the Azure networks through the Azure Virtual Network Gateway. Network gatewaysprovide routing for Azure virtual networks.

Creating Azure Virtual Networks also creates a default routing table for the vNet to direct traffic to/from thesubnets of the vNet. If the default route table is insufficient for the solution custom routes can be created to routeoutgoing traffic to custom appliances or to block routes to specific subnets or external networks.

The default route table includes the following routes:

Routing within a subnetSubnet-to-subnet within the virtual networkTo the InternetVirtual network-to-virtual network using VPN gatewayVirtual network-to-on-premises network using a VPN or ExpressRoute gateway

User-defined routes allow the control of traffic outbound from the assigned subnet to other subnets in the virtualnetwork or over one of the other predefined gateways (ExpressRoute; internet or VPN). The default system routingtable can be replaced with a user-defined routing table that replaces the default routing table with custom routes.With user-defined routing, customers can create specific routes to appliances such as firewalls or intrusiondetection appliances, or block access to specific subnets from the subnet hosting the user-defined route. For anoverview of User Defined Routes look here.

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview

Security

Next steps

Depending on which model is in use, Connect-To or Connect-Through, your customer defines the security policiesin their vNet or provides the security policy requirements to the CSP to define to their vNets. The followingsecurity criteria can be defined:

1. Customer Isolation â€” The Azure platform provides customer isolation by storing Customer ID and vNet infoin a secure database, which is used to encapsulate each customerâ€™s traffic in a GRE tunnel.

2. Network Security Group (NSG) rules are for defining allowed traffic into and out of the subnets within vNetsin Azure. By default, the NSG contain Block rules to block traffic from the Internet to the vNet and Allow rulesfor traffic within a vNet. For more information about Network Security Groups look here.

3. Force tunneling â€”This is an option to redirect internet bound traffic originating in Azure to be redirectedover the ExpressRoute connection to the on premises datacenter. For more information about Forced tunnelinglook here.

4. Encryption â€” Even though the ExpressRoute circuits are dedicated to a specific customer, there is thepossibility that the network provider could be breached, allowing an intruder to examine packet traffic. Toaddress this potential, a customer or CSP can encrypt traffic over the connection by defining IPSec tunnel-mode policies for all traffic flowing between the on premises resources and Azure resources (refer to theoptional Tunnel mode IPSec for Customer 1 in Figure 5: ExpressRoute Security, above). The second optionwould be to use a firewall appliance at each the end point of the ExpressRoute circuit. This will require additional3rd party firewall VMs/Appliances to be installed on both ends to encrypt the traffic over the ExpressRoutecircuit.

The Cloud Solution Provider service provides you a way to increase your value to your customers without the needfor expensive infrastructure and capability purchases, while maintaining your position as the primary outsourcingprovider. Seamless integration with Microsoft Azure can be accomplished through the CSP API, allowing you tointegrate management of Microsoft Azure within your existing management frameworks.

Additional Information can be found at the following links:

Azure in Cloud Solution Provider program.Get ready to transact as a Cloud Solution Provider.Microsoft Cloud Solution Provider resources.

https://azure.microsoft.com/blog/network-security-groups/

https://docs.microsoft.com/azure/cloud-solution-provider

https://partner.microsoft.com/en-us/solutions/cloud-reseller-pre-launch

https://partner.microsoft.com/en-us/solutions/cloud-reseller-resources

expressroute overview: extend your on-premises network to azure

Documents