team 11 - jagadeesh dhandapaani - apr 25 2014 542 pm - group11 viability of deploying ebgp as an igp
TRANSCRIPT
VIABILITY OF DEPLOYING eBGP AS IGP IN DATACENTER NETWORKS
Chavan, Prathamesh
Dhandapaani, Jagadeesh
Kavuri, Mahesh Babu
Mohankumar, Aravind
Faculty Advisors: Jose Santos & Mark Dehus
“A capstone paper submitted as partial fulfilment for the degree of Masters in Telecommunications at the University of Colorado at Boulder “
Viability of implementing eBGP as IGP in datacenters 2
ABSTRACT:
Advent of cloud based infrastructure and continuous growth of web based services has motivated
service providers to set up large scale Datacenters (DC). Traditionally, DCs utilize Border
Gateway Protocol (BGP) to route the customer traffic to and from the Internet. Link state protocols
such as Open Shortest Path First (OSPF) and Intermediate System-Intermediate System (IS-IS)
are used for routing the traffic within DC. The disadvantages of OSPF and IS-IS are: they require
large number of CPU cycles to process the link state updates and maintain an identical link state
database within the area; these protocols also impose a challenge due to the necessity of re-
designing DC routing architecture when the number of network devices increase. Few large scale
DC operators have taken a novel approach in the routing design by adopting exterior Border
Gateway Protocol (eBGP) as an Interior Gateway Protocol (IGP). The significant advantages of
using eBGP as an IGP are:
i) It is a stable and robust routing protocol consuming less number of CPU cycles
ii) It can handle large number of routes without having to re-design the network
iii) It is simple to operate and troubleshoot
Our research will focus on the feasibility, architecture and the impacts of implementing eBGP as
an IGP in large scale DC environments. We will simulate this environment with eBGP as the only
routing protocol and compare the network performance against traditional IGP’s. We will also
analyze the impact of the fundamental datacenter services such as seamless mobility for virtual
machines, extension of Layer 2 adjacency (VLANs) within the datacenter and load balancing
applications critical for the operation of high density server environments. The goal of our
research is to evaluate the incentives and the viability for large scale datacenters to migrate to an
e-BGP only design.
Viability of implementing eBGP as IGP in datacenters 3
Table of Contents
I. Introduction .......................................................................................................................... 4
Statement of the Problem ........................................................................................................ 5 Research Problem .................................................................................................................... 6
Research Sub-problems ........................................................................................................... 6 II. Literature Review................................................................................................................. 7
Traditional Datacenter topology .............................................................................................. 7 Advantages of using CLOS design .......................................................................................... 8
Choosing the appropriate routing protocol to be used in the Datacenter ................................ 9
Impact on legacy datacenter applications .............................................................................. 10
Current state of the art technology ........................................................................................ 10
III. Research Methodology ...................................................................................................... 11
Layer 2 Performance Analysis .............................................................................................. 12
Layer 3 performance analysis OSPF vs BGP ........................................................................ 12 IV. Research Results ................................................................................................................ 17
Layer 2 Performance evaluation ............................................................................................ 17
Layer 3 performance results under normal operations .......................................................... 18
Layer 3 performance analysis during convergence ............................................................... 19
V. Discussion of Research results: ......................................................................................... 19
VI. Conclusion and Future work: ............................................................................................. 20
VII. Appendix ............................................................................................................................ 22
VIII. References ........................................................................................................................ 23
Viability of implementing eBGP as IGP in datacenters 4
I. Introduction
The growing demand for virtualization and cloud based services along with traditional network
practices has created challenges in designing large scale datacenters (DC), which typically includes
more than 20,000 Virtual Machines and nearly 2000 networking devices [1]. Following are some
of the significant requirements of a large scale DC [2]:
i) Selecting an appropriate topology that supports the increasing demand for server to
server communication (east-west traffic) within the Datacenter, by adding more
number of links and cheap commodity devices with identical port density.
ii) Choosing a single control plane protocol (if possible) that could handle large number
of routes, eliminates the need for inter-operability with other control plane protocols
and are supported by most of the network equipment vendors.
iii) Choosing a routing protocol that has a simple implementation in terms of the protocol
design and ease of operational support.
iv) Reduce the scope of failure caused by equipment or protocol issues.
Designing the control plane architecture is the most challenging part. Control plane is a critical
component of the router or a switch architecture that has sufficient intelligence to build the network
map (dynamically or statically) and determines how an individual network device interacts with
its neighbors. It is the signaling component of the network. Few of the control plane functions
include management/configuration of the network device and exchange of routing information.
Control plane protocols are essential for efficient forwarding of the data traffic between any two
devices. Multiple Spanning Tree (MST) may be considered as the control plane protocol on a pure
Layer 2 environment, whereas OSPF and BGP may be considered as the control plane protocol
used in a Layer 3 environment.
Viability of implementing eBGP as IGP in datacenters 5
CLOS topology is preferred in large scale DCs as this topology can be horizontally scaled by
increasing the port density in the devices or by adding similar devices to the topology. This saves
the cost involved in buying high performance devices and avoiding the re-design of the network
[3]. In a large scale DC, the cost of the networking infrastructure amounts to 10-15% of the entire
Capital Expenditure (CAPEX) [4]. There is a constant drive to cut down costs and therefore DCs
prefer networking elements of the same hardware type with minimal features [2]. Traditionally
datacenters utilize Border Gateway Protocol (eBGP), a distance vector protocol, on Wide Area
Networks(WAN) to route the traffic to and from the Internet and link state protocols such as OSPF
and IS-IS in the Local Area networks (LAN) [5] to efficiently route the traffic within the
datacenter. Having two control plane protocols within a DC increases the probability of network
misconfiguration and poses challenges in making both the protocols interact with each other,
thereby demanding a design with simpler control plane protocol that reduces the chance of network
failure [2]. A single control plane protocol makes the network design simple and easy to
troubleshoot; which is one of the fundamental requirements of a large scale data center.
Statement of the Problem
BGP is mostly viewed as the exterior routing protocol used for routing traffic between networks
managed by different administrative bodies. The significant reasons for using BGP are; it provides
a better control over the traffic flow because of its built-in attribute set, it can handle a large number
of routes because of its limited flooding scope and simpler protocol design [6]. Link state Protocols
such as OSPF and IS-IS are designed for routing traffic within the same administrative domain; as
they have a complete view of the network topology within the area and can converge faster [5].
However, as the number of network devices and routes continues to increase, it creates the
following scalability challenge in the network design:
Viability of implementing eBGP as IGP in datacenters 6
a. The scope of Link State Advertisements (LSA) flooding is the entire area which causes
all the network devices within the area to process all the LSAs generated and compute
SPF algorithms. The above processes may consume large number of CPU cycles on a
network device as the instability in the network and number of network devices
increase.
The most widely implemented solution to address the above challenges is to re-design the network
by segmenting a single area into multiple areas, so as to reduce the processing load on the network
device [2]. Re-designing DC network requires careful planning, scheduled network outages,
possibility of misconfiguration and administrative overhead. Currently, the IETF group is working
on standardizing an alternate and innovative solution to the above challenges by deploying BGP
as an IGP, instead of traditional OSPF and IS-IS [2]. The above routing architecture is the basis of
our research in analyzing the practicality and viability of deploying eBGP as an IGP in large scale
Datacenters.
Research Problem
What are the incentives for firms operating large scale datacenters to migrate to an eBGP only
design? Although the design permits, what is the viability of implementing eBGP as an IGP? Does
the new design of deploying BGP as the only control plane protocol affect any existing services
and applications offered by datacenters?
Research Sub-problems
i. Simulating an optimally designed large scale DC network to analyze the viability of
deploying eBGP as an IGP
ii. Network Performance evaluation of Layer 2 and Layer 3 control protocols
a. Layer 2 performance analysis
Viability of implementing eBGP as IGP in datacenters 7
b. Layer 3 performance analysis (OSPF vs. BGP)
iii. Analyzing the possible impacts of deploying eBGP as an IGP
II. Literature Review
Traditional Datacenter topology
Traditional DC environment utilizes a mix of Ethernet based Layer 2 technologies with three levels
of hierarchy namely access, aggregation and core layers [7]. A typical DC network consist of a
hierarchical tree like topology with progressively sophisticated high performance network devices
at the core layer of the network and commodity network devices at the access layer of the network.
These traditional datacenters utilize Spanning Tree Protocols between the access and distribution
layer and link state routing protocols between the distribution and core layer. BGP is used for
providing external connectivity for the DC. The use of a Layer 2 solution at the access layer causes
redundant uplinks on the access layer switches to be blocked by the Spanning Tree Protocol (STP)
to avoid Layer 2 loops; this effectively reduces the net utilization of the up-links [3] because of an
active-passive approach. The advent of Multi-Chassis Link Aggregation protocol (M-LAGs)
provides an active-active approach and offers increased throughput. However, most of the M-LAG
implementations are proprietary and have a little multi-vendor support [2]. The need of STP in a
flat Layer 2 network can be avoided with the introduction of TRILL that addresses most of the
challenges faced by STP. However, TRILL requires new network devices and has limited
implementation experiences. Finally, both TRILL and M-LAG does not eliminate the primary
concerns of a shared broadcast domain such as: large amounts of broadcast and unknown-unicast
storms [2].
Viability of implementing eBGP as IGP in datacenters 8
Advantages of using CLOS design
Until recently, the traffic flows were commonly North-South traffic (enter and exit the datacenter)
and tree-topology served the requirements of these type of traffic with high oversubscription ratios
and high density line cards on the core network devices. But, recently many large scale DC host
applications that require distributed computing across the servers, causing large amount of East-
West traffic flow. The traditional tree topologies do not scale horizontally to match these
bandwidth demands. The fat-tree topology (also known as CLOS or leaf-spine topology), on the
other hand, uses similar type of commodity Ethernet switches to provide cheaper scalable
interconnection bandwidth between the servers. The fat tree topology can be easily scaled by
adding more number of stages or increasing the port density of the network element as shown in
the figure below:
Figure 1.a: Two stage CLOS topology to support 4 server subnets
Figure 1.b: Scaling CLOS topology
Viability of implementing eBGP as IGP in datacenters 9
The above discussed CLOS topology presents a new interconnect architecture that leverages
commodity network elements and meets the scalability, bandwidth and cost requirements of a large
scale datacenter. Figure 1.a shows a two stage fully non-blocking CLOS topology to support four
servers. The CLOS topology can be scaled to support eight servers by either increasing the port
density on the two stage CLOS topology or by moving to a three stage CLOS topology as shown
in figure 1.b.By appropriate choice of routing protocol, that inherently supports load balancing,
the hosts in the network can communicate at the full bandwidth of its local interface.
Choosing the appropriate routing protocol to be used in the Datacenter
IP routing is generally used at the core layer and above. However, Next-generation datacenters
incorporate IP routing intelligence to the access – distribution layers to have efficient utilization
of the uplinks, improved network stability and scalability [7]. It also confines Layer 2 broadcast
domains and thereby significantly reducing the OPEX. Usually the layer 3 routing protocol is
either OSPF or IS-IS, as it is assumed to provide faster convergence and better loop avoidance
features in comparison to their distance vector counterparts [5]. However, as the DC grows in size
and scale, the existing IGPs had to be constantly revisited and re-designed to reduce the amount
of processing required on the routers within an area.
Lapukhov from Microsoft and Premji from Arista Networks have recently identified several
problems with the existing IGPs [8]. They are actively working with the IETF group in proposing
a novel design to utilize BGP as the only IGP in large-scale datacenter environments because of
the following advantages [2]:
i. A network device running BGP advertises only the best path available to its directly
connected neighbors and is less prone to flooding events, which makes it more scalable
without the need to re-design [9].
Viability of implementing eBGP as IGP in datacenters 10
ii. BGP is the most widely implemented WAN protocol in a datacenter environment and
extending its implementation in the LAN would simplify the operation overhead.
iii. BGP has several attributes built-in to control the flow of routing information. It also
supports third party resolved next hops that allow unequal cost load balancing and
application defined forwarding paths [10].
iv. BGP is simple to troubleshoot when compared to OSPF because of the availability of
adjacent Routing Information Base – In (RIB-In) and Routing Information Base – out
(RIB-out) structures with the corresponding Network Layer reachability Information that
can be easily correlated [10].
Impact on legacy datacenter applications
Incorporating Layer 3 intelligence close to the access layer will break the layer 2 adjacency
requirements of large scale DCs. There are several critical Datacenter applications such as
seamless mobility for virtual machines, application load balancing using Layer 2 DSR architecture
that require layer 2 adjacency for their operation.. However with the advent of several Layer 2
tunneling protocols such as GRE tunnel, L2TPv3, OTV and VXLAN, these services can still be
achieved by extending the Layer 2 domain across the Layer 3 backbone. The pros and cons of
these tunneling protocols will be discussed in the appendix section.
Current state of the art technology
The current research by IETF does not compare the performance of OSPF and BGP in a large scale
DC environment and evaluate the incentives for firms to migrate to an e-BGP only design. Our
research will attempt to extend the current IETF’s work and analyze the viability of deploying
BGP as an IGP in datacenter networks and the incentives to migrate to the “eBGP-only” design
by
Viability of implementing eBGP as IGP in datacenters 11
i) Comparing the CPU utilization of OSPF and BGP during normal operations and re-
convergence
ii) Comparing the convergence time of OSPF and BGP
iii) Exploring the most scalable Layer 2 tunneling solutions to address the Layer 2
adjacency requirement of critical datacenter applications.
III. Research Methodology
Our research evaluates the feasibility of deploying eBGP as an IGP in large scale Datacenter
networks. The first sub-problem of our research was to simulate this environment in a lab; while
real world datacenter networks consist of large number of network devices, we replicated a small
subset of DC network to conduct our measurements.
Fig 2.a Data plane Fig 2.b Management plane
We implemented the above specialized and scalable network topology (Fig2.a) that satisfies the
first requirement of the large scale datacenters as mentioned in Section I [3]. This topology is
broken down into three layers: core (Tier 1), distribution/aggregation (Tier 2) and access (Tier 3)
layers. The above network topology was built using 12 Cisco 3550 multi-layer switches, and 1000
Viability of implementing eBGP as IGP in datacenters 12
loopbacks were added on each Tier-3 device to simulate large number of servers. Next, to prevent
the production traffic being affected by the management traffic, we implemented a separate data
plane and management plane as shown in Figure 2.a and 2.b. The management plane was used to
monitor and collect the network performance statistics through automated scripts.
Layer 2 Performance Analysis
We conducted a layer 2 performance analysis by considering three factors: CPU utilization, delay
and bandwidth. A layer 2 network was simulated by configuring Multiple Spanning Tree (MST)
in the test bed. We simulated instability in the network to test the MST convergence and measure
the CPU utilization while the network converged. Next, we simulated traffic flows from one cluster
to another to measure the delay and bandwidth in the network under various levels of broadcast.
Layer 3 performance analysis OSPF vs BGP
The following section describes the design and performance analysis of the layer 3 network with
OSPF and BGP as the only control plane protocol.
OSPF Design and optimization:
Figure 3: IGP design using OSPF
Viability of implementing eBGP as IGP in datacenters 13
The network consisted of 12 devices in a single OSPF area (Area 0). We optimized OSPF to
achieve faster convergence and utilize less CPU cycles by incorporating the following measures:
a. Point-to-Point networks – To avoid Designated Router (DR) elections
b. Reduced Hello timers- To achieve faster re-convergence
c. Incremental SPF (ISPF)- To reduce the CPU utilization for SPF computations
d. Throttle timers- To achieve Faster convergence by reducing the time between SPF
computations and LSA flooding
All the above features are supported by most of the commodity switches, and hence satisfies
requirement II of the large scale datacenters as mentioned in Section 1.
Performance analysis of OSPF:
We tested the CPU utilization under two states: normal operations (stable) and during re-
convergence (unstable). For both the states, we simulated two scenarios: one with 8 device CLOS
topology and other with 12 device CLOS topology to check whether the performance degrades
with more number of devices. The following table illustrates the methodology used to conduct
performance analysis for both states:
State Reason for CPU utilization Methodology Normal operations
Link state database (LSDB) refreshed every 30 minutes (LS Refresh time).
CPU utilization from all network devices was recorded every 30 minutes
Re-convergence
Flapping the server subnets (Loopbacks in Tier 3 devices)
CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes).
Flapping the transit links
Viability of implementing eBGP as IGP in datacenters 14
IGP Network design using eBGP:
The following diagram illustrates the BGP design scheme used in our network topology:
Figure 4: IGP design using eBGP
Internal Routing:
This section describes the eBGP design scheme used for internal routing. External BGP (eBGP)
peering sessions were established over point-to-point links between all network devices. All the
devices in Tier-1 were assigned a unique Autonomous System Number (ASN), each group of Tier
2 devices in a cluster was assigned a unique ASN, and each Tier 3 device within a cluster was
assigned a unique ASN as shown in figure 4. The 16bit ASNs allows the use of only 1023 unique
private ASNs within a datacenter. Since, large scale datacenters may have more than 2000 network
devices, the ASNs will have to be re-used at the Tier 3 level. Considering that ASN’s are used in
BGP as a loop avoidance feature [11]; the reuse of ASN’s at the Tier 3 level will prevent routes
advertised from a tier 3 device in one cluster being accepted by a tier 3 device in another cluster.
In order to overcome this challenge, we utilized “Allow AS In” feature of BGP which allows a
BGP enabled device to accept routes that contain its own ASN in the AS-PATH attribute. This
feature is mostly supported on all commodity network devices [12]. It is assumed that the
Viability of implementing eBGP as IGP in datacenters 15
Datacenter network topology is symmetrical and does not cause routing loops when implementing
this feature.
External routing:
The above BGP design routes the traffic within the datacenter, however doesn’t provide Internet
access. To do so, we have used another border cluster as shown in Figure 4. The Edge devices of
the border cluster peer with the ISP and the core routers of the Border cluster peer with the Tier-1
devices of the datacenter. As private ASNs were used inside the datacenter; we use the “Remove
private-as” feature in the border cluster to strip off the private AS numbers when advertising the
Datacenter routes to the Internet.
BGP optimization:
As BGP is a stable routing protocol consuming less CPU cycles, we modified BGP to achieve faster
convergence by incorporating the following measures:
a. Reduced the default BGP advertisement interval value to zero so that a BGP peer exchanges
the routing updates immediately. Even though the signaling overhead increases, it does not
cause a considerable effect on the performance
b. BGP, by default does not install multiple paths in the routing table. In order to utilize
redundant paths, we enabled “maximum paths” feature available in BGP.
BGP Performance evaluation:
Performance evaluation for BGP was carried out in a similar manner as OSPF performance
evaluation in order to achieve comparable results. The following table illustrates the methodology
used to conduct performance analysis for both states:
Viability of implementing eBGP as IGP in datacenters 16
State Reason for CPU utilization Methodology Normal operations
BGP scanner process that runs every 60seconds by default
CPU utilization from all network devices was recorded after every minute.
Re-convergence
Flapping the server subnets (Loopbacks on Tier 3 devices)
CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes).
Flapping the transit links
Convergence testing for OSPF and BGP:
In order to test the convergence time of BGP and OSPF, continuous traffic was sent from a server
connected to a Tier-3 device ( S9) in cluster 1 to a server connected to another Tier -3 device (S12)
in cluster 2. The worst case scenario (in terms of processing overhead and convergence) for both
the protocols happens when (Refer figure 5):
a) We disable a redundant link L1, and force all the traffic through L2
b) We again enable link L1 and disable link L2 immediately.
The above scenario was simulated, because the convergence depends on the neighbor
establishment, advertisement of routes or link state packets and calculation of the best path.
Figure 5: Network performance evaluation
Viability of implementing eBGP as IGP in datacenters 17
IV. Research Results
In this section we present the performance evaluation of MST, OSPF and BGP, and briefly
describe our results with graphs wherever necessary.
Layer 2 Performance evaluation
We observed that the CPU utilization of all the network devices was under 2% (for both stable and
unstable cases) with MST and we were able to achieve convergence under one second in worst
case scenarios The following figure illustrates the relationship between the amount of broadcasts
and delay in a flat Layer 2 network.
Figure 6: Layer 2 Performance evaluation
0
5
10
15
20
25
0 5 10 15 20 25 30 35 40 45
Traf
fic D
elay
(Sec
onds
)
Percentage of Broadcasts
Performance of a Layer 2 network
Viability of implementing eBGP as IGP in datacenters 18
Layer 3 performance results under normal operations
Figure 7a: CPU Utilization of OSPF under normal operations
Figure 7b: CPU Utilization of BGP under normal operations
Figure 7.a and 7.b show the CPU utilization of OSPF and BGP under stable conditions. From the
above graphs, it can be observed that the CPU utilization of OSPF increases as the number of
network devices increase. However, it can be observed from the graph that the CPU utilization of
BGP stays constant with the addition of more number of network devices.
010203040506070
0 20 40 60 80 100 120
Perc
enta
ge C
PU U
tiliza
tion
Number of Nodes
OSPF CPU utilization in stable conditions
Tier - 1 Tier - 2
Number of Routes = 4000
00.5
11.5
22.5
3
0 20 40 60 80 100 120
Perc
enta
ge C
PU U
tiliza
tion
Number of nodes
BGP CPU utilization in stable conditions
Tier - 1 Devices Tier - 2 Devices
Number of Routes = 4000
Viability of implementing eBGP as IGP in datacenters 19
Layer 3 performance analysis during convergence
Figure 8: Performance metrics of OSPF and BGP during convergence.
From the figure 8, it can be observed that the CPU utilization of OSPF increases to a maximum
value of 56% (approx.) during convergence. Similarly it can be also observed that the CPU
utilization of BGP increases to a maximum value of 37% (approx.) during convergence.
V. Discussion of Research results:
The CPU utilization and convergence of a Layer 2 network with MST was comparable with the
Layer 3 network performance. However, broadcasts are inherently present throughout the Layer 2
network and this effectively reduces the network capacity and increases the delay in the network.
It also consumes the CPU cycles of all the hosts connected in the DC, as all the hosts present in
the same broadcast domain will have to process the broadcast traffic. Therefore, as the number of
hosts in the Datacenter network increase, the flat Layer 2 design does not scale well.
From the Layer 3 performance analysis tests, we observed that the CPU utilization of BGP was
better than the CPU utilization of OSPF under stable conditions and also during network
0
10
20
30
40
50
60
OSPF - Tier 1 OSPF - Tier 2 OSPF - Tier 3 BGP - Tier 1 BGP - Tier 2 BGP - Tier 3
9.2815.45
12.24
1.48
9.134.71
27.1
56.5
29.74
11.9
37.8
19.6
Perc
enta
ge o
f CPU
util
izatio
nCPU utilization of OSPF and BGP during convergence
Average utilization over 60 sec Average utilization over 5 sec
Viability of implementing eBGP as IGP in datacenters 20
convergence. Inherent properties of link state protocols such as SPF computation complexity and
periodic link state updates cause the CPU utilization to increase as the number of network devices
increase. BGP being a distance vector protocol is not impacted by the above factors.
Normally, the convergence of link state protocols is better than distance vector protocols [5].
However, based on our performance test results, we were able to achieve equivalent convergence
time for BGP by optimizing it as discussed in section IV.
VI. Conclusion and Future work:
We first determined that a flat Layer 2 network performance in terms of CPU and convergence
was comparable with the Layer 3 network performance. However, we also identified that as the
size of the network increases, the Layer 2 design will have high amount of broadcasts which would
cause delays and affect the overall throughput of the network.
As we moved towards a Layer 3 approach, we found that the performance of OSPF degrades as
the number of network devices increases; this performance degradation can be addressed by re-
designing OSPF in to multiple areas. However, the redesigning approach requires planning and
scheduled maintenance often, as the network scales. The use of eBGP as the IGP overcomes the
above design challenges and does not require datacenter operators to re-design the complete
routing architecture. Further, our results also showed that the CPU utilization of BGP is better than
OSPF, and BGP can be optimized to achieve comparable convergence time as OSPF.
We also found that a fully routed design with BGP/OSPF breaks the layer 2 adjacency, which is
required for high availability services in a datacenter. However, we found out that several overlay
technologies like GRE tunneling, L2TPv3, OTV and VXLAN can be used to achieve layer 2
adjacency over a fully routed design.
Viability of implementing eBGP as IGP in datacenters 21
In summary, our research shows that using eBGP as an IGP provides several operational
advantages such as: single control plane protocol within the network which is easy to operate and
troubleshoot, avoids network misconfigurations due to inter-operability of different control plane
protocols, and capability to handle large number of routes without having to re-design the routing
architecture. The above advantages provide large scale datacenter operators an incentive to migrate
to an eBGP only design. Further, our approach of using a single control plane protocol within the
network simplifies the migration steps towards Software Defined Networking.
Viability of implementing eBGP as IGP in datacenters 22
VII. Appendix - I
Overlay Technologies
The eBGP as an IGP design would have a complete Layer 3 design in the datacenters. There are
critical applications such as Virtual machine migration, Layer 2 direct server return, etc. which
require Layer 2 adjacency. This requirement can be satisfied by use of overlay technologies such
as Layer 2 Generic routing encapsulation [GRE], Layer 2 transport protocol [L2TP], Overlay
transport virtualization [OTV] and Virtual extensible local area network [VXLAN].
The above mentioned overlay technologies were analyzed to ensure ease of deployment and
scalability. The advantage of GRE and L2TP is that they are easy to deploy and have less
configuration overhead, but each of them create point to point tunnels and therefore become
complex to deploy as the network scales.
The recently developed overlay technologies such as OTV and VXLAN are scalable as they create
dynamic tunnels to traverse through the routed access layer. The performance and configuration
overhead between OTV and VXLAN is comparable. OTV primarily addresses overlaying layer 2
over layer 3 whereas VXLAN also addresses the shortage of vlan ids (only 4094) which is a
realistic issue with increasing scale of datacenters. Also, OTV is a Cisco proprietary protocol
which would necessitate all the devices in the Tier 3 to be cisco devices whereas VXLAN is
currently in IETF draft stage with wide vendor support.
Appendix – II
Further information relating to the implementation and configurations used for various tests conducted can be found at:
https://docs.google.com/document/d/1gl5cW9zeh3HDbwLrbWNs2Ejq8iKARRpTRPwcfOAFTSs/edit?usp=sharing
Viability of implementing eBGP as IGP in datacenters 23
VIII. References
[1] Sengupta, S. (2013, August 16). Data center networking. Retrieved from
http://conferences.sigcomm.org/sigcomm/2013/ttldc.php
[2] Lapukhov, P. L., & Premji, A. P. (2013). Using BGP for routing in large-scale data
centers. IETF. Retrieved from
http://tools.ietf.org/id/draft-lapukhov-bgp-routing-large-dc-06.txt
[3] Al-Fares, M., Loukissas,A., & Vahdat,A. (2008). A scalable, commodity data center network
architecture. ACM SIGCOMM Computer Communication Review, 38(4), 63.
[4] Greenberg, A., Hamilton, J., Maltz, D. A., & Patel, P. (2008). The cost of a cloud: research
problems in data center networks. ACM SIGCOMM Computer Communication Review, 39(1), 68–
73.
[5] Carroll, J. D., & Doyle, J. (2005). Routing TCP/IP: Vol. 1. Indianapolis, Ind: Cisco Press.
[6] Carroll, J. D., & Doyle, J. (2012). Routing TCP/IP: Vol. 2. Indianopolis, Ind: Cisco Press.
[7] Cisco Data Center Infrastructure 2.5 Design Guide. (2007, December 6).
Retrieved August 12, 2013, from
https://www.cisco.com/application/pdf/en/us/guest/netsol/ns107/c649/ccmigration_09186a00807
3377d.pdf
[8] Lapukhov, P. (2012, June 4). Nanog 55. Retrieved September 20, 2013, from
http://www.nanog.org/meetings/nanog55/presentations/Monday/Lapukhov.pdf
Viability of implementing eBGP as IGP in datacenters 24
[9] Halabi, S., & McPherson, D. (2001). Internet routing architectures. Indianapolis, IN: Cisco
Press.
[10] Hankins, G. (2013, October 10). BGP as a Data Center IGP - Brocade Community.
Retrieved October 20, 2013, from
[11] Fernando, R., Kumaki, K., McPherson, D., Patel, K., & Raszuk, R. E. (2012). Distribution
of Diverse BGP Paths. IETF. Retrieved from http://tools.ietf.org/html/rfc6774
http://community.brocade.com/t5/Service-Providers/BGP-as-a-Data-Center-IGP/ba-p/649
[12] Walton, D., Retina, A., Chen, E., & Scudder, J. (2009). IETF. Advertisement of Multiple Paths
in BGP, 06. Retrieved from
http://tools.ietf.org/html/draft-walton-bgp-add-paths-06