very pleasant/painful networking: the highs and lows of building and maintaining ipsec based...

Very Pleasant/Painful Networking:Very Pleasant/Painful Networking:The Highs and Lows of Building and Maintaining The Highs and Lows of Building and Maintaining

IPsec Based Customer Access VPNsIPsec Based Customer Access VPNs

Matthew W. BakerMatthew W. Baker

Intel Online Services, Inc.Intel Online Services, Inc.

NANOG22NANOG22

Intel Online Services Confidential - 2

VPN - Behind the ScenesVPN - Behind the Scenes

IPsec based IPsec based VPNs are highly effective, and VPNs are highly effective, and can prove to be very valuable for a myriad of can prove to be very valuable for a myriad of applications. However, building and applications. However, building and maintaining Virtual Private Networks can be maintaining Virtual Private Networks can be difficult and frustrating. This presentation difficult and frustrating. This presentation will highlight some issues that frequently will highlight some issues that frequently pop up, and some strategies for dealing with pop up, and some strategies for dealing with them. them.


AgendaAgenda

Introduction/BackgroundIntroduction/BackgroundThe Distilled Taxonomy of IPSec VPNsThe Distilled Taxonomy of IPSec VPNs

Technology ChallengesTechnology ChallengesBad NATBad NAT

Troubleshooting ComplexityTroubleshooting Complexity

The Many Levels of InteroperabilityThe Many Levels of Interoperability

Environmental ChallengesEnvironmental ChallengesQuality, Reliability, and PerformanceQuality, Reliability, and Performance

External Security Stumbling BlocksExternal Security Stumbling Blocks


More AgendaMore Agenda

Environmental ChallengesEnvironmental ChallengesSecurity ConcernsSecurity Concerns

Routing and AddressingRouting and Addressing

Good NATGood NAT

Human FactorsHuman Factors

ConclusionsConclusions


Distilled VPN TaxonomyDistilled VPN Taxonomy

Site-to-Site/LAN-to-LAN/Branch Office Site-to-Site/LAN-to-LAN/Branch Office VPN connecting two networks, or groups of VPN connecting two networks, or groups of networks. networks. Typically employ main mode IKE with pre-Typically employ main mode IKE with pre-shared keys or certificates for authentication.shared keys or certificates for authentication.CPE based device-to-device...normally either an CPE based device-to-device...normally either an edge or edge-1 gateway device.edge or edge-1 gateway device.Routing and addressing management is a Routing and addressing management is a factorfactorSecurity of tunnel relies on integrity of the Security of tunnel relies on integrity of the participating networks.participating networks.


LAN-to-LAN VPNsLAN-to-LAN VPNs

IPSec VPN Tunnel

The Internet

10.1.1.0 to 10.2.2.010.1.1.0 to 10.3.3.010.1.1.0 to 10.4.4.010.1.1.0 to 10.5.5.0

etc

10.2.2.0 to 10.1.1.010.3.3.0 to 10.1.1.010.4.4.0 to 10.1.1.010.5.5.0 to 10.1.1.0

etc

The Internet

10.1.1.0 to 10.2.2.010.1.1.0 to 10.3.3.010.1.1.0 to 10.4.4.010.1.1.0 to 10.5.5.0

etc

10.2.2.0 to 10.1.1.010.3.3.0 to 10.1.1.010.4.4.0 to 10.1.1.010.5.5.0 to 10.1.1.0

etc

IPSec VPN Tunnel


Distilled VPN TaxonomyDistilled VPN Taxonomy

Client-to-LAN, Remote AccessClient-to-LAN, Remote AccessA VPN tunnel connecting a single node to a A VPN tunnel connecting a single node to a remote network.remote network.

Typically employs aggressive mode IKE with Typically employs aggressive mode IKE with pre-shared keys/passwords, certificates, pre-shared keys/passwords, certificates, tokens, etc. for authentication.tokens, etc. for authentication.

Client software drivenClient software driven

Security appears more tightly controlledSecurity appears more tightly controlled

Access policies can be centrally managedAccess policies can be centrally managed


Client-to-LAN VPNsClient-to-LAN VPNs

The Internet

Tunnel Mode ESPNetworks:

10.2.2.0/2410.3.3.0/2410.4.4.0/2410.5.5.0/24

etc.

No TrafficBifurcation

Client IP pool10.1.1.0/24

IPSec VPN Tunnel

Client IP configured fromremote gateway IP pool

Routing tables modifiedand monitored to ensureno traffic bifurcation

Technology ChallengesTechnology Challenges


Bad NATBad NAT

Network Address Translation presents many Network Address Translation presents many difficult challenges.difficult challenges.

IPsec has inherent issues with NAT and vice versa, IPsec has inherent issues with NAT and vice versa, “many-to-one” NAT is particularly problematic.“many-to-one” NAT is particularly problematic.

Knowing how any single NAT Knowing how any single NAT implementation will effect IPsec is implementation will effect IPsec is impossible...Assume the worst!impossible...Assume the worst!

Some NAT implementations completely kill IPsec. Some NAT implementations completely kill IPsec.

Others will allow a single tunnel to be created which will Others will allow a single tunnel to be created which will be killed by subsequent attempts to create additional be killed by subsequent attempts to create additional tunnelstunnels


Bad NATBad NAT

The use of NAT is pervasive in the The use of NAT is pervasive in the broadband and low cost access markets.broadband and low cost access markets.

Large LECs utilize broadband CPE based NAT to Large LECs utilize broadband CPE based NAT to ease implementation complexity and conserve IP ease implementation complexity and conserve IP space.space.

Many end users are usually unaware of the Many end users are usually unaware of the nature of their Internet connectivity.nature of their Internet connectivity.

Realm Specific IP (RSIP) solutions slow on Realm Specific IP (RSIP) solutions slow on the uptake.the uptake.


Strategies: Dealing with Bad NATStrategies: Dealing with Bad NAT

Be prepared with customer documentation Be prepared with customer documentation describing how to create 1:1 NAT between nodes describing how to create 1:1 NAT between nodes requiring VPN access and the NAT devices.requiring VPN access and the NAT devices.

Be prepared to assist customers by facilitating Be prepared to assist customers by facilitating communication between customer and their service communication between customer and their service provider. provider.

If customers will require access from many nodes If customers will require access from many nodes behind a NAT gateway, consider LAN2LAN access.behind a NAT gateway, consider LAN2LAN access.

Consider another VPN implementation that will allow Consider another VPN implementation that will allow NAT traversal? Is a UDP wrapped client available? NAT traversal? Is a UDP wrapped client available? Leverage your vendor!Leverage your vendor!


Troubleshooting ComplexityTroubleshooting Complexity

Troubleshooting toolsets remain fairly Troubleshooting toolsets remain fairly immature.immature.

Varying “Standard” implementations make Varying “Standard” implementations make root cause fingerprinting difficult.root cause fingerprinting difficult.

The very nature of Virtual Private Networking The very nature of Virtual Private Networking makes troubleshooting extremely difficult.makes troubleshooting extremely difficult.

Sniffing of packets is essentially useless, packets Sniffing of packets is essentially useless, packets are homogenous and encrypted.are homogenous and encrypted.


Strategies: TroubleshootingStrategies: Troubleshooting

Build an IPsec debug target with strong Build an IPsec debug target with strong logging capabilities.logging capabilities.

Reduce complexity by enforcing product Reduce complexity by enforcing product standards, and ensuring number of standards, and ensuring number of device/vendor combinations is minimal.device/vendor combinations is minimal.

Structure access network with a device that Structure access network with a device that accommodates simple packet inspection.accommodates simple packet inspection.


The Many Levels of The Many Levels of InteroperabilityInteroperability

Cross Vendor InteroperabilityCross Vendor InteroperabilityVaried IPsec implementations make cross vendor Varied IPsec implementations make cross vendor interoperability troublesome at bestinteroperability troublesome at bestMany critical features are often vendor specific. Many critical features are often vendor specific. Products are often streamlined for usability, thus Products are often streamlined for usability, thus protocol extensions and other proprietary features protocol extensions and other proprietary features cannot be disabled.cannot be disabled.C2L interop is particularly troublesome. VPN clients C2L interop is particularly troublesome. VPN clients have become commodities, thus the goal of have become commodities, thus the goal of interoperability rare.interoperability rare.Vendor support is problematic Vendor support is problematic Potential cost savings are often consumed by Potential cost savings are often consumed by implementation minutia and unpredictable implementation minutia and unpredictable stability/performance.stability/performance.


The Many Levels of The Many Levels of InteroperabilityInteroperability

Intra Vendor InteroperabilityIntra Vendor InteroperabilityTypically vendor sanctioned and supportedTypically vendor sanctioned and supportedOperationally difficult given disparate command Operationally difficult given disparate command interfaces, orders of operation, etc...interfaces, orders of operation, etc...

Client-to-LAN – OS/Application InteropClient-to-LAN – OS/Application InteropWill a particular client run on all operating Will a particular client run on all operating systems???systems???Unix flavors are conspicuously absent from most Unix flavors are conspicuously absent from most vendor’s OS support lists!vendor’s OS support lists!Multiple VPN client installed simultaneously Multiple VPN client installed simultaneously causes issuescauses issuesExpect application/client interoperability issuesExpect application/client interoperability issues


Strategies: InteropStrategies: Interop

Enforce standard device/client combinations.Enforce standard device/client combinations.

Strongly set expectations and share your Strongly set expectations and share your “supported standards” upfront with “supported standards” upfront with customers.customers.

Proactively publish known compatibility Proactively publish known compatibility issues at all levels.issues at all levels.

Test, test, test!!!Test, test, test!!!

Environmental ChallengesEnvironmental Challenges


Quality, Reliability, and Quality, Reliability, and PerformancePerformance

Customers are many times unaware of the Customers are many times unaware of the costs associated with using the Internet for costs associated with using the Internet for mission critical data transport.mission critical data transport.

Application issues may arise as increased Application issues may arise as increased latency and bandwidth variance are inevitably latency and bandwidth variance are inevitably introduced.introduced.

Varying performance introduced by the Varying performance introduced by the vagaries of the Internet are difficult to isolate vagaries of the Internet are difficult to isolate and difficult to explain.and difficult to explain.

The cost of QR&P investigations is high.The cost of QR&P investigations is high.


Quality, Reliability, and Quality, Reliability, and PerformancePerformance

Customer ISP selection is a factor!Customer ISP selection is a factor!The quality of customer Internet connectivity will The quality of customer Internet connectivity will have an impact on the level of service you are have an impact on the level of service you are able to deliver... able to deliver...

Expectations must be set solidly, and service Expectations must be set solidly, and service level agreements carefully crafted.level agreements carefully crafted.

Beware of custom applications, DB activity, Beware of custom applications, DB activity, etc...etc...


External Security Stumbling External Security Stumbling BlocksBlocks

Many IT organizations are unfamiliar with Many IT organizations are unfamiliar with IPsec!IPsec!

Customer’s hesitate to open holes for IPsec.Customer’s hesitate to open holes for IPsec.UDP is evil incarnate to most firewall admins, UDP is evil incarnate to most firewall admins, IKE openings are therefore most problematic!IKE openings are therefore most problematic!

ESP/AH are little understood – making them ESP/AH are little understood – making them suspect.suspect.

Some ISP have been known to arbitrarily Some ISP have been known to arbitrarily block tunneling protocols...isolation can be block tunneling protocols...isolation can be difficult.difficult.


Strategies: QR&P and Strategies: QR&P and Security Security

Draft SLAs that comprehend the likelihood of Draft SLAs that comprehend the likelihood of performance fluctuations. Protect both yourself and performance fluctuations. Protect both yourself and the customer by setting concrete expectations.the customer by setting concrete expectations.

Educate your customers with regard to the Educate your customers with regard to the implication of Internet based networking, e.g. implication of Internet based networking, e.g. compare and contrast with leased line performance.compare and contrast with leased line performance.

Gather metrics on device utilization, available Gather metrics on device utilization, available bandwidth, etc... This allows for data driven bandwidth, etc... This allows for data driven discussions of performance issues.discussions of performance issues.

Gather Internet performance metrics when possible, Gather Internet performance metrics when possible, and leverage internet communities such as NANOG and leverage internet communities such as NANOG to keep abreast of Internet “events”.to keep abreast of Internet “events”.


VPN Security ConcernsVPN Security Concerns

Client-to-LANClient-to-LANConsidered by many to be more secure. Considered by many to be more secure.

Scope of tunnel is well defined.Scope of tunnel is well defined.

Security policies can be centrally controlled and Security policies can be centrally controlled and pushed out to clients.pushed out to clients.

Strong user level access control features are Strong user level access control features are availableavailable

Traffic bifurcation presents considerable riskTraffic bifurcation presents considerable riskRoute through/Trojan attacks are easily mounted Route through/Trojan attacks are easily mounted when customers are allowed to fork traffic.when customers are allowed to fork traffic.


VPN Security ConcernsVPN Security Concerns

LAN-to-LANLAN-to-LANScope of tunnel is amorphousScope of tunnel is amorphous

Security policies and protective measures cannot Security policies and protective measures cannot be centrally controlledbe centrally controlled

Rogue access is simple and straight forward if Rogue access is simple and straight forward if security of remote network is compromisedsecurity of remote network is compromised

A level of user access control is lostA level of user access control is lost


Strategies: Security Strategies: Security ConcernsConcerns

Enforce “no bifurcation” policy when using Enforce “no bifurcation” policy when using Client-to-LAN...the risks are simply too great Client-to-LAN...the risks are simply too great otherwise.otherwise.Favor Client-to-LAN technology as first Favor Client-to-LAN technology as first option.option.Strictly scope L2L tunnels!Strictly scope L2L tunnels!Educate customers regarding security issues Educate customers regarding security issues with L2L tunnels, e.g. publish white papers.with L2L tunnels, e.g. publish white papers.Expect customer security lapses, and Expect customer security lapses, and vigilantly protect your business and network vigilantly protect your business and network from these lapses.from these lapses.


Routing and Addressing Routing and Addressing ChallengesChallenges

Client-to-LANClient-to-LANC2L makes routing simple, and addressing issues C2L makes routing simple, and addressing issues moot. However, the limitations of C2L VPNs moot. However, the limitations of C2L VPNs quickly introduce the need for a more flexible L2L quickly introduce the need for a more flexible L2L approach.approach.

LAN-to-LANLAN-to-LANRouting and addressing issues aboundRouting and addressing issues abound


Routing/Addressing – An Routing/Addressing – An ExampleExample

Customer A

Customer B

The Internet

Customer B Networks:

10.1.1.0/2410.2.2.0/24

10.3.3.0/24

Customer A Networks:

10.3.3.0/2410.4.4.0/2410.5.5.0/24

Dest 10.3.3.34

Routing can quicklybecome problematicwhen customer endnetworks collide!


LAN2LAN Routing/AddressingLAN2LAN Routing/Addressing

Use of private addressing is pervasive, Use of private addressing is pervasive, making collision inevitable.making collision inevitable.

Advanced routing techniques, like Policy Advanced routing techniques, like Policy Based Routing etc., commonly are not Based Routing etc., commonly are not feasible/available. feasible/available.

Route sharing is typically undesirable and Route sharing is typically undesirable and complicated. complicated.


LAN2LAN LAN2LAN Routing/AddressingRouting/Addressing

If route sharing is not used, customers must If route sharing is not used, customers must figure out how to plumb traffic to their VPN figure out how to plumb traffic to their VPN gateway. gateway.

Route leaking/external redistribution must be Route leaking/external redistribution must be considered, and could have disastrous considered, and could have disastrous consequences! Particularly in a hosting consequences! Particularly in a hosting environment.environment.


Good NATGood NAT

NAT can prove beneficial when coping with NAT can prove beneficial when coping with routing and addressing issues.routing and addressing issues.

Creates layers of abstraction, a buffer.Creates layers of abstraction, a buffer.Can simplify internal routingCan simplify internal routingCan mitigate customer/internal IP address Can mitigate customer/internal IP address collision.collision.Returns control to service providerReturns control to service provider

DownsidesDownsidesComplexityComplexityBi-directional traffic initiation is problematicBi-directional traffic initiation is problematicApplication incompatibilityApplication incompatibility


Good NAT – Datacenter Access Good NAT – Datacenter Access ExampleExample

Customer A

Customer B

Customer ANetworks:

10.3.3.0/2410.4.4.0/2410.5.5.0/24

Customer ATranslate Network:

192.168.1.0/24

Customer BNetworks:

10.1.1.0/2410.2.2.0/24

10.3.3.0/24

Customer BTranslate Network:

192.168.2.0/24

DatacenterNetworkThe Internet


Strategies: Routing and Strategies: Routing and AddressingAddressing

Provide documentation regarding routing Provide documentation regarding routing strategies...cover the customer spectrum from small strategies...cover the customer spectrum from small to large.to large.Consider an architecture that protects your critical Consider an architecture that protects your critical services from misconfigured or leaking routes – services from misconfigured or leaking routes – critical in a datacenter environment. critical in a datacenter environment. Build abstraction layer into your access network Build abstraction layer into your access network where addressing/routing can be “normalized”.where addressing/routing can be “normalized”.Utilize NAT to retain control of addressing.Utilize NAT to retain control of addressing.Have policy in place regarding the use of private Have policy in place regarding the use of private addressing...keep these addressing issues in mind addressing...keep these addressing issues in mind as you design and redesign your network.as you design and redesign your network.


Strategies: Strategies: Routing and Routing and AddressingAddressing

Take a building block approach with your Take a building block approach with your design...design...

Layering typically increases the flexibility of your Layering typically increases the flexibility of your designdesignLayering compartmentalizes elements of VPN Layering compartmentalizes elements of VPN infrastructure (connectivity, access control, etc.)infrastructure (connectivity, access control, etc.)Layering also promotes defense in depth which Layering also promotes defense in depth which can mitigate concern with shared environmentscan mitigate concern with shared environments

Be vigilant when selecting a VPN product, Be vigilant when selecting a VPN product, many are still built with the enterprise in many are still built with the enterprise in mind.mind.

Human FactorsHuman Factors


Perceptions and Perceptions and ExpectationsExpectations

Underlying connectivity architecture is Underlying connectivity architecture is abstracted.abstracted.

Traditional troubleshooting techniques can lead Traditional troubleshooting techniques can lead customers astray – underlying transport customers astray – underlying transport network is concealed.network is concealed.

Education is extremely importantEducation is extremely important

Issue ownership conflicts are commonIssue ownership conflicts are commonThe customer relationship must be well The customer relationship must be well defined. A stable and consistently performing defined. A stable and consistently performing VPN environment is a highly cooperative effort.VPN environment is a highly cooperative effort.


Perceptions and Perceptions and ExpectationsExpectations

Performance ExpectationsPerformance ExpectationsService level agreements are tricky to craft, and Service level agreements are tricky to craft, and difficult to enforce for both parties.difficult to enforce for both parties.Blame games can be common, and are costly to Blame games can be common, and are costly to the service provider.the service provider.

Internal ExpectationsInternal ExpectationsEnsure that your sales and marketing folks are Ensure that your sales and marketing folks are up to speed with the technology (within reason) up to speed with the technology (within reason) and the capabilities of your design. We all know and the capabilities of your design. We all know the consequences if you do not!the consequences if you do not!

Educate yourself...know your customers.Educate yourself...know your customers.


Education is ParamountEducation is Paramount

Setting Expectations is Setting Expectations is EssentialEssential



The large majority of operational issues are The large majority of operational issues are not VPN related per se – they are far more not VPN related per se – they are far more mundane.mundane.

VPN does not mitigate traditional WAN VPN does not mitigate traditional WAN connectivity issues.connectivity issues.

Addressing/Routing is still a major Addressing/Routing is still a major concernconcern

VPN combines all of these traditional WAN VPN combines all of these traditional WAN connectivity issues with the vagaries of the connectivity issues with the vagaries of the Internet – quickly magnifying complexity!Internet – quickly magnifying complexity!



External/environmental issues effecting External/environmental issues effecting performance are extremely common, thus performance are extremely common, thus expectations must be clearly communicated expectations must be clearly communicated and agreed to.and agreed to.Issues over ownership of configuration, Issues over ownership of configuration, connectivity, even hardware are commonconnectivity, even hardware are commonKnow your target customers well...ensure that Know your target customers well...ensure that your design will be flexible enough to your design will be flexible enough to accommodate the vast majority with without accommodate the vast majority with without breeding exceptions. breeding exceptions. Education should serve as the foundation of Education should serve as the foundation of the technical relationship with your customer.the technical relationship with your customer.

very pleasant/painful networking: the highs and lows of building and maintaining ipsec based...

Documents

baker intel online services

use of nat

nat devices

nat implementations

nanog22 slide

managed slide

vpn access

single nat implementation