data center interconnects: an overview
DESCRIPTION
It's common business policy for organizations of a certain size to have two data centers as part of a disaster recovery or business continuity plan. However, most enterprise - applications are not designed for or intended to use systems in two different locations. Enter the notion of a data center interconnect, which extends an Ethernet network between two physically separate data centers. While the idea is simple, Ethernet wasn't designed to run across a wide area network. Thus, a DCI implementation requires a variety of technological fixes to work around Ethernet's limitations. This report outlines the issues that complicate DCIs, such as loops that can bring down networks and traffic trombones that eat up bandwidth. It also examines the variety of options companies have to connect two or more data centers, including dark fiber, MPLS services and MLAG, as well as vendor specific options such as Cisco OTV and HP EVI. The report looks at the pros and cons of each option.TRANSCRIPT
Report ID S6970513
Next
reports
Data Center Interconnects An OverviewA DCI lets companies link two or more data centers
together for disaster recovery or business continuity but itrsquos
not easy This report provides an overview of the major DCI
technologies and describes their pros and cons
By Greg Ferro
R e p o r t s I n f o r m a t i o nWe e k c om M a y 2 0 1 3 $ 9 9
Previous Next
reports
reportsinformationweekcom May 2013 2
CONT
ENTS
TABLE OF
3 Authorrsquos Bio
4 Executive Summary
5 The DCI Problem
5 Figure 1 Ingress Routing Problems
6 Loop Prevention
7 Three Options
8 Software-Defined Networking
8 Vendor and Standards-Based Technologies
8 Figure 2 Leaf and Spine
9 Figure 3 Partial Mesh
10 Custom Options
10 Figure 4 MLAG
11 DCI Weighing the Choices
12 Related Reports
D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
ABOUT US
InformationWeek Reportsrsquo analysts arm business technology decision-makers with real-world perspective based on qualitativeand quantitative research business and technology assessment andplanning tools and adoption best practices gleaned from experience
OUR STAFFLorna Garey content director lornagareyubmcomHeather Vallis managing editor research heathervallisubmcomElizabeth Chodak copy chief elizabethchodakubmcomTara DeFilippo associate art director taradefilippoubmcom
Find all of our reports at reportsinformationweekcom
May 2013 3
Previous Next
copy 2013 InformationWeek Reproduction Prohibited
reports
reportsinformationweekcom
D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
Greg Ferro has spent more than 20 years in IT sales technical and managementroles but his main work is in network engineering and design Today he worksas a freelance consultant for Fortune 100 companies in the UK focusing ondata centers security and operational automation Greg believes that cloudcomputing is just a new focus on technical operations that software quality isvital and that networking is the unrealized future of everything because itrsquos allabout bandwidth
At night he finds it cathartic to write about networking at Network Computinghis blog at Ethereal Mind and at Packet Pushers He is known for practical opin-ions technical viewpoints and being graceful when getting it wrong Mostly
Greg FerroInformationWeek Reports
Table of Contents
FollowFollowFollowFollow
Want More
Never Miss a Report
May 2013 4
Previous Next
Itrsquos common business policy for organizations of a certain size to have two data centersas part of a disaster recovery or business continuity plan However most enterprise applications are not designed for or intended to use systems in two different locationsFor example a MySQL database is designed to exist on a single server with a single stor-age location Building a resilient MySQL server requires an advanced infrastructure orcomplex software
Enter the notion of a data center interconnect which extends an Ethernet network be-tween two physically separate data centers While the idea is simple Ethernet wasnrsquot de-signed to run across a wide area network Thus a DCI implementation requires a variety oftechnological fixes to work around Ethernetrsquos limitations
This report outlines the issues that complicate DCIs such as loops that can bring downnetworks and traffic trombones that eat up bandwidth It also examines the variety of op-tions companies have to connect two or more data centers including dark fiber MPLSservices and MLAG as well as vendor-specific options such as Cisco OTV and HP EVI Thereport looks at the pros and cons of each option
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
EXECUTIVE
SUM
MAR
Y
Table of Contents
May 2013 5
The most reliable method to connect twodata centers together for high availabilityand disaster recovery is to route IP traffic be-tween the data centers However itrsquos becomemore common to extend the Ethernet net-work over the WAN called a data center in-terconnect or DCI This allows for the use offeatures such as virtual machine migrationFor instance by connecting two data centersvia Ethernet administrators can move a SQLServer instance via VM migration withoutchanging the IP address of the operating sys-tem This is attractive to the server teams be-cause the IP address is a key part of the di-rectory service or configuration databaseMaintaining the same IP address means thatapplication settings remain the same and re-duces the chance of errors when migrationoccurs Service continuity is simpler if the IPaddress is unchanged
A VMware ESXi server can perform vMotionfor up to eight virtual machines at once (pro-vided that you have a 10 Gbps network
adapter or four at 1 Gbps) Given that itrsquos com-mon to have 20 to 40 VMs per physical serveryou can see that evacuating a server will take
some time VMotion performance requiresvery low latency typically less than 50 millisec-onds to achieve control transfer (although op-
Previous Next
Ingress Routing Problems
Source Greg Ferro S69705131
S
Without careful planning after a server migration traffic may unnecessarily traverse one data center and the data center interconnect to connect to servers in another data center
Data Centre Interconnect
Server Migration
reportsinformationweekcom
The DCI Problem
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 1
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
Previous Next
reports
reportsinformationweekcom May 2013 2
CONT
ENTS
TABLE OF
3 Authorrsquos Bio
4 Executive Summary
5 The DCI Problem
5 Figure 1 Ingress Routing Problems
6 Loop Prevention
7 Three Options
8 Software-Defined Networking
8 Vendor and Standards-Based Technologies
8 Figure 2 Leaf and Spine
9 Figure 3 Partial Mesh
10 Custom Options
10 Figure 4 MLAG
11 DCI Weighing the Choices
12 Related Reports
D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
ABOUT US
InformationWeek Reportsrsquo analysts arm business technology decision-makers with real-world perspective based on qualitativeand quantitative research business and technology assessment andplanning tools and adoption best practices gleaned from experience
OUR STAFFLorna Garey content director lornagareyubmcomHeather Vallis managing editor research heathervallisubmcomElizabeth Chodak copy chief elizabethchodakubmcomTara DeFilippo associate art director taradefilippoubmcom
Find all of our reports at reportsinformationweekcom
May 2013 3
Previous Next
copy 2013 InformationWeek Reproduction Prohibited
reports
reportsinformationweekcom
D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
Greg Ferro has spent more than 20 years in IT sales technical and managementroles but his main work is in network engineering and design Today he worksas a freelance consultant for Fortune 100 companies in the UK focusing ondata centers security and operational automation Greg believes that cloudcomputing is just a new focus on technical operations that software quality isvital and that networking is the unrealized future of everything because itrsquos allabout bandwidth
At night he finds it cathartic to write about networking at Network Computinghis blog at Ethereal Mind and at Packet Pushers He is known for practical opin-ions technical viewpoints and being graceful when getting it wrong Mostly
Greg FerroInformationWeek Reports
Table of Contents
FollowFollowFollowFollow
Want More
Never Miss a Report
May 2013 4
Previous Next
Itrsquos common business policy for organizations of a certain size to have two data centersas part of a disaster recovery or business continuity plan However most enterprise applications are not designed for or intended to use systems in two different locationsFor example a MySQL database is designed to exist on a single server with a single stor-age location Building a resilient MySQL server requires an advanced infrastructure orcomplex software
Enter the notion of a data center interconnect which extends an Ethernet network be-tween two physically separate data centers While the idea is simple Ethernet wasnrsquot de-signed to run across a wide area network Thus a DCI implementation requires a variety oftechnological fixes to work around Ethernetrsquos limitations
This report outlines the issues that complicate DCIs such as loops that can bring downnetworks and traffic trombones that eat up bandwidth It also examines the variety of op-tions companies have to connect two or more data centers including dark fiber MPLSservices and MLAG as well as vendor-specific options such as Cisco OTV and HP EVI Thereport looks at the pros and cons of each option
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
EXECUTIVE
SUM
MAR
Y
Table of Contents
May 2013 5
The most reliable method to connect twodata centers together for high availabilityand disaster recovery is to route IP traffic be-tween the data centers However itrsquos becomemore common to extend the Ethernet net-work over the WAN called a data center in-terconnect or DCI This allows for the use offeatures such as virtual machine migrationFor instance by connecting two data centersvia Ethernet administrators can move a SQLServer instance via VM migration withoutchanging the IP address of the operating sys-tem This is attractive to the server teams be-cause the IP address is a key part of the di-rectory service or configuration databaseMaintaining the same IP address means thatapplication settings remain the same and re-duces the chance of errors when migrationoccurs Service continuity is simpler if the IPaddress is unchanged
A VMware ESXi server can perform vMotionfor up to eight virtual machines at once (pro-vided that you have a 10 Gbps network
adapter or four at 1 Gbps) Given that itrsquos com-mon to have 20 to 40 VMs per physical serveryou can see that evacuating a server will take
some time VMotion performance requiresvery low latency typically less than 50 millisec-onds to achieve control transfer (although op-
Previous Next
Ingress Routing Problems
Source Greg Ferro S69705131
S
Without careful planning after a server migration traffic may unnecessarily traverse one data center and the data center interconnect to connect to servers in another data center
Data Centre Interconnect
Server Migration
reportsinformationweekcom
The DCI Problem
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 1
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 3
Previous Next
copy 2013 InformationWeek Reproduction Prohibited
reports
reportsinformationweekcom
D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
Greg Ferro has spent more than 20 years in IT sales technical and managementroles but his main work is in network engineering and design Today he worksas a freelance consultant for Fortune 100 companies in the UK focusing ondata centers security and operational automation Greg believes that cloudcomputing is just a new focus on technical operations that software quality isvital and that networking is the unrealized future of everything because itrsquos allabout bandwidth
At night he finds it cathartic to write about networking at Network Computinghis blog at Ethereal Mind and at Packet Pushers He is known for practical opin-ions technical viewpoints and being graceful when getting it wrong Mostly
Greg FerroInformationWeek Reports
Table of Contents
FollowFollowFollowFollow
Want More
Never Miss a Report
May 2013 4
Previous Next
Itrsquos common business policy for organizations of a certain size to have two data centersas part of a disaster recovery or business continuity plan However most enterprise applications are not designed for or intended to use systems in two different locationsFor example a MySQL database is designed to exist on a single server with a single stor-age location Building a resilient MySQL server requires an advanced infrastructure orcomplex software
Enter the notion of a data center interconnect which extends an Ethernet network be-tween two physically separate data centers While the idea is simple Ethernet wasnrsquot de-signed to run across a wide area network Thus a DCI implementation requires a variety oftechnological fixes to work around Ethernetrsquos limitations
This report outlines the issues that complicate DCIs such as loops that can bring downnetworks and traffic trombones that eat up bandwidth It also examines the variety of op-tions companies have to connect two or more data centers including dark fiber MPLSservices and MLAG as well as vendor-specific options such as Cisco OTV and HP EVI Thereport looks at the pros and cons of each option
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
EXECUTIVE
SUM
MAR
Y
Table of Contents
May 2013 5
The most reliable method to connect twodata centers together for high availabilityand disaster recovery is to route IP traffic be-tween the data centers However itrsquos becomemore common to extend the Ethernet net-work over the WAN called a data center in-terconnect or DCI This allows for the use offeatures such as virtual machine migrationFor instance by connecting two data centersvia Ethernet administrators can move a SQLServer instance via VM migration withoutchanging the IP address of the operating sys-tem This is attractive to the server teams be-cause the IP address is a key part of the di-rectory service or configuration databaseMaintaining the same IP address means thatapplication settings remain the same and re-duces the chance of errors when migrationoccurs Service continuity is simpler if the IPaddress is unchanged
A VMware ESXi server can perform vMotionfor up to eight virtual machines at once (pro-vided that you have a 10 Gbps network
adapter or four at 1 Gbps) Given that itrsquos com-mon to have 20 to 40 VMs per physical serveryou can see that evacuating a server will take
some time VMotion performance requiresvery low latency typically less than 50 millisec-onds to achieve control transfer (although op-
Previous Next
Ingress Routing Problems
Source Greg Ferro S69705131
S
Without careful planning after a server migration traffic may unnecessarily traverse one data center and the data center interconnect to connect to servers in another data center
Data Centre Interconnect
Server Migration
reportsinformationweekcom
The DCI Problem
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 1
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 4
Previous Next
Itrsquos common business policy for organizations of a certain size to have two data centersas part of a disaster recovery or business continuity plan However most enterprise applications are not designed for or intended to use systems in two different locationsFor example a MySQL database is designed to exist on a single server with a single stor-age location Building a resilient MySQL server requires an advanced infrastructure orcomplex software
Enter the notion of a data center interconnect which extends an Ethernet network be-tween two physically separate data centers While the idea is simple Ethernet wasnrsquot de-signed to run across a wide area network Thus a DCI implementation requires a variety oftechnological fixes to work around Ethernetrsquos limitations
This report outlines the issues that complicate DCIs such as loops that can bring downnetworks and traffic trombones that eat up bandwidth It also examines the variety of op-tions companies have to connect two or more data centers including dark fiber MPLSservices and MLAG as well as vendor-specific options such as Cisco OTV and HP EVI Thereport looks at the pros and cons of each option
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
EXECUTIVE
SUM
MAR
Y
Table of Contents
May 2013 5
The most reliable method to connect twodata centers together for high availabilityand disaster recovery is to route IP traffic be-tween the data centers However itrsquos becomemore common to extend the Ethernet net-work over the WAN called a data center in-terconnect or DCI This allows for the use offeatures such as virtual machine migrationFor instance by connecting two data centersvia Ethernet administrators can move a SQLServer instance via VM migration withoutchanging the IP address of the operating sys-tem This is attractive to the server teams be-cause the IP address is a key part of the di-rectory service or configuration databaseMaintaining the same IP address means thatapplication settings remain the same and re-duces the chance of errors when migrationoccurs Service continuity is simpler if the IPaddress is unchanged
A VMware ESXi server can perform vMotionfor up to eight virtual machines at once (pro-vided that you have a 10 Gbps network
adapter or four at 1 Gbps) Given that itrsquos com-mon to have 20 to 40 VMs per physical serveryou can see that evacuating a server will take
some time VMotion performance requiresvery low latency typically less than 50 millisec-onds to achieve control transfer (although op-
Previous Next
Ingress Routing Problems
Source Greg Ferro S69705131
S
Without careful planning after a server migration traffic may unnecessarily traverse one data center and the data center interconnect to connect to servers in another data center
Data Centre Interconnect
Server Migration
reportsinformationweekcom
The DCI Problem
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 1
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 5
The most reliable method to connect twodata centers together for high availabilityand disaster recovery is to route IP traffic be-tween the data centers However itrsquos becomemore common to extend the Ethernet net-work over the WAN called a data center in-terconnect or DCI This allows for the use offeatures such as virtual machine migrationFor instance by connecting two data centersvia Ethernet administrators can move a SQLServer instance via VM migration withoutchanging the IP address of the operating sys-tem This is attractive to the server teams be-cause the IP address is a key part of the di-rectory service or configuration databaseMaintaining the same IP address means thatapplication settings remain the same and re-duces the chance of errors when migrationoccurs Service continuity is simpler if the IPaddress is unchanged
A VMware ESXi server can perform vMotionfor up to eight virtual machines at once (pro-vided that you have a 10 Gbps network
adapter or four at 1 Gbps) Given that itrsquos com-mon to have 20 to 40 VMs per physical serveryou can see that evacuating a server will take
some time VMotion performance requiresvery low latency typically less than 50 millisec-onds to achieve control transfer (although op-
Previous Next
Ingress Routing Problems
Source Greg Ferro S69705131
S
Without careful planning after a server migration traffic may unnecessarily traverse one data center and the data center interconnect to connect to servers in another data center
Data Centre Interconnect
Server Migration
reportsinformationweekcom
The DCI Problem
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 1
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 6
tions exist) A larger DCI bandwidth will resultin faster vMotion and reduce the risk of traffictrombone
Note that you can create a cascading failureas more servers move so you need increasingamounts of bandwidth for interserver appli-cation traffic which makes the remainingvMotion tasks progressively slower Band-width will eventually reach a peak and canprevent vMotion from occurring at the peak
point of transitionOne problem with server
migration is that storagemust be synchronized be-tween the data centers us-ing storage replicationtechnology Replication isusually performed by the
storage array but itrsquos an expensive option andwill consume additional bandwidth betweenthe sites
Provided that the storage is replicated between the sites extending the Ethernetnetwork between data centers results in thesimplest possible server migration between
each data center though it incurs significanttechnical debt It is networking best practiceto use Layer 3 routing between geographi-cally diverse locations and to limit Layer 2connectivity wherever possible thus improv-ing network stability and limiting risk do-mains to a single data center Wersquoll look atsome of the technological challenges of DCIand discuss the pros and cons of various DCIoptions
Loop PreventionEthernet introduces several technical hur-
dles in building a DCI Ethernet was createdsome 30 years ago as a local area networkprotocol with no practical concept of scalingpast a few machines By design Ethernet is amultiaccess technology that allows all Ether-net broadcast frames to be received by allendpoints on the network Thus an Ethernetbroadcast frame must be forwarded acrossall Ethernet networks including the DCI If abroadcast frame is looped back into an Eth-ernet network it will be forwarded by allswitches even though it was already broad-
cast This creates a race condition that rapidlyconsumes all network bandwidth and usu-ally results in catastrophic network failure asthe volumes of broadcasts expand to con-sume all resources
The Spanning Tree Protocol was designedto address the loop problem with Ethernetand has generally served its purpose on theLAN However itrsquos not suitable for control offlooding of packets between data centersbecause Spanning Tree is not easily scalableand risk domains grow as network diametergrows STP has no domain isolation so aproblem in a single data center can propa-gate between data centers In addition first-hop resolution and inbound routing selec-tion can cause verbose inter-data centertraffic over the DCI
When a server is migrated between data cen-ters traffic to and from the server must be in-tentionally designed Outbound flows from theserver will default to a router that may or maynot be in the same data center In this instancetraffic from a server in DC B may traverse theDCI link to reach the router in DC A and then re-
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
When a server is migrated
between data centers traffic to
and from the server must be
intentionally designed
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 7
Previous Next
turn back over the DCI link to another resourcein the DC B This is not optimal because the net-work link between data centers is bearing alltraffic from external users and traffic from therelocated server to other servers (see Figure 1)The resulting traffic pattern is sometimes calleda traffic trombone Consider a Web server withJava Runtime and an MySQL database in DC AAfter migration of the Web server to DC B thetraffic flows over the DCI are
gtgt External flows from the WAN andor Internet
gtgt Flows from the Web server to the databasegtgt Administrative traffic flows such as back-
ups monitoring and patchingConsider performing a backup of the mi-
grated server in DC B to a backup system inDC A How much bandwidth do you need sothat the backup will complete within thebackup window And will the backup impactthe critical traffic like the database queries orcustomer Web traffic
Three OptionsToday there are three common methods for
modifying traffic flow between data centersfirst-hop bypass LISP and load balancing Wersquolllook at each in turn
First-hop bypass relates to the many optionsfor establishing a local default gateway orrouter hop for the server The server will re-quire the same default gateway address ineach data center but sending the traffic fromDC B to DC A leads to failure Therefore meth-ods based around MAC address filtering forHSRP IP gateways are common There are sev-eral ways to handle this specific to each routervendorrsquos software implementation
Location Independent Separation Protocolis an IETF standard proposed by Cisco thatmodifies the concept of routing location In-stead of routing to a subnet in the networktraffic is forwarded to a specific router usinga tunnel The router will then forward the traf-fic to an identifier which is the IP address ofthe server You can find more about LISP hereor at the IETF LISP Working Group LISP worksfor inbound and outbound traffic
Load balancing involves using the sourceNAT features on a load balancing VIP so that
traffic will be sourced from a device within thedata center However this only works for in-bound flows and must be combined withother traffic controls such as first-hop bypassfor a complete solution
A fourth option is route injection which in-volves triggering a dynamic route injectioninto the network routing based on certaintrigger options This method has proved lessreliable in wider use because routing proto-cols have limited trigger capabilities Thisworks for inbound flows and partially for out-bound flows
These technologies address the traffictrombone problem using legacy or existingnetwork tools but you may also wish to con-sider vendor-specific technologies such asCiscorsquos OTV or Hewlett-Packard EVC whichwersquoll discuss later
Software-Defined NetworkingIn the future SDN and controller-based
networking will likely provide new capabili-ties that do not rely on the configuration ofindividual devices or require manual over-
Research The Next-Gen WAN
Respondents to our Next Gen -eration WAN Survey are a highlyconnected bunch 44 have 16or more branch or remote officeslinked to their primary data cen-ters And Ethernet-based serviceslike MPLS outstripped ISDNamong current users 73 to56 Whatrsquos next Demand fordark fiber and private cloudsamong other things
DownloadDownload
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 8
rides of routing configuration If yoursquore plan-ning a DCI deployment you should considerevaluating the new SDN technologies in-cluding Juniper Contrail VMware NSX andAlcatel- Lucentrsquos Nuage Networks
Vendor and Standards-Based TechnologiesThere has been significant demand for DCI
products and this has led to a number oftechnological developments by vendors andby standards bodies Irsquoll look at five optionsdark fiber MPLS pseudowires MLAG TRILLECMP and custom vendor protocols Dark fiber Dark fiber is a broad term used
to describe dedicated fiber-optic cables orser vices that closely emulate dedicated ca-bles For some geographies itrsquos possible tolay your own fiber between data centers andown the right of way or to purchase a dedi-cated fiber from a provider Physical cablesare usually capped at around 50 to 75 kilo-meters (the distance of a long-haul single-mode laser transmitter)
More commonly your local carrier providesa dense wavelength division multiplexer ser -
vice that presents fiber at each site and ap-pears as a dedicated fiber cable to the sitesThe customer can use any signal on that fiberbecause the laser is physically multiplexedand the DWDM drop has limited awareness ofyour protocols The DWDM service providesadditional circuit redundancy through the useof ring technologies and the carrier can pro-
vision multiple services over a single fiber pairMPLS pseudowires When it comes to
MPLS most organizations will choose to pur-chase Layer 2 virtual private network servicefrom a service provider Service providers usethe MPLS protocol internal to their networksto provide a wide range of IP routed servicessuch as WAN and Internet circuits
Previous Next
Leaf and Spine
Source Greg Ferro S69705133
S
A leaf and spine architecture can be used in a DCI
40 Gbps 40 Gbps
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 2
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 9
Typically your service provider will deliveran Ethernet port to your premises and thiswill connect to the providerrsquos MPLS back-bone MPLS standards have organicallygrown a messy bunch of protocols that canprovide a Layer 2 Ethernet emulation over anMPLS network Technologies such as VPLSEoMPLS GREoMPLS and L2TPv3 all provideways for emulating Ethernet networks Yourproviderrsquos MPLS network should be config-ured to support one or more of these tech-nologies These technologies are incorrectlybut widely referred to as ldquopseudowiresrdquo be-cause their original purpose was to emulateATM and frame relay circuits in the early2000s before being modified for Ethernet
Large enterprises may build their own MPLSbackbones to have greater control over theservices and security of the WAN but for mostcompanies this wonrsquot be a viable optionMPLS is a relatively complex group of proto-cols that requires a significant amount of timeto learn and comprehend Building mission-critical business with MPLS is hard and shouldgenerally be avoided
MLAG Multichassis link aggregation de-scribes the logical bonding of two physicalswitches into a single unit as shown in Figure2 The logical switch control plane is a singlesoftware entity This prevents loop conditionsfrom occurring and reduces operational risk
Itrsquos simple to use configure and maintain com-pared with other approaches and is less ex-pensive Your service provider can supply Layer2 services (probably using dark fiber or MPLSpseudowires as discussed previously)
Note that MLAG is not a standard Each ven-
Previous Next
Partial Mesh
Source Greg Ferro S69705134
S
TRILL can be used to create a layer-2 partial mesh DCI topology
Partial Mesh Topology
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
Figure 3
Like This Report
Rate ItSomething we could dobetter Let us know
RateRate
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 10
dor has its own name for the technology suchas Ciscorsquos vPC HPrsquos IRF Juniperrsquos MC-LAG andBrocadersquos Multi-Chassis Trunk
To use MLAG for DCI connect each port onthe MLAG switches to the Layer 2 service toprevent loops Itrsquos recommended not to useMLAG features on core switches in each datacenter instead use fixed switches in a modu-lar design for control and better supportMLAG could handle up to eight point-to-point circuits A service provider failure wouldreduce the bandwidth and will require carefuldesign if yoursquore using quality of service to pro-tect key applications ECMPTRILL Equal Cost Multipath or ECMP
is a more recent addition to the options forDCI The IETF TRILL standard provides a multi-path protocol that ldquoroutesrdquo Ethernet framesacross up to 16 paths that have the samebandwidth or cost
Although intended for data center back-bones to implement a Clos Tree switch fabric(sometimes known as leafspine) TRILL can beused in a DCI technology It provides highavailability because dual switches are used at
all sites and also provides native STP isolation A unique feature of TRILL as a DCI technol-
ogy is that it supports partial meshed topol-ogy for multiple data centers because theLayer 2 traffic is routed over the TRILL core
Although core features arecomplete the TRILL protocolcontinues to be developedMany mainstream vendors havenot released a fully standards-compliant implementation sowhile you can build a TRILL fabricfrom a single vendorrsquos gear youmay run into interoperabilityproblems in a heterogeneous environment Some vendors arealso extending the standard toadd proprietary features Bro-cade VCS and Cisco FabricPathare two of the available optionstoday
Custom OptionsAs you can see there are com-
plex technical challenges to ex-tending Ethernet networks between datacenters The effort often brings more risk thancustomers are willing to accept However ven-dors are developing proprietary protocols toaddress these risks Case in point are Ciscorsquos
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
MLAG
Source Greg Ferro S69705132
S
MLAG logically bonds two physical switches to make them appear as a single unit
Physical MLAG
Logical MLAG
Figure 4
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
May 2013 11
Overlay Transport Virtualization (OTV) andHPrsquos Ethernet Virtual Interconnect (EVI)
The protocols encapsulate Ethernet in IPfor transport over WAN services Softwareagents for these protocols in the edge net-work devices provide features such as Span-ning Tree isolation in each data center re-duced configuration effort and multisitesetups Compared to MPLS OTV and EVI arevery simple to configure and maintainthough you will incur a substantial licensingfee on specific hardware platforms The sim-plicity of these approaches makes them at-tractive options for most enterprises
You can find a more detailed comparison ofOTV and EVI here
DCI Weighing the ChoicesBefore embarking on a DCI project consider
your disaster recovery plan carefully Can youmeet your disaster recovery requirements bycold start from a storage array replication oreven by restoring from a backup If so youmay not need to make the investment in aDCI On the other hand if you are looking for
disaster avoidance where server instancescan be evacuated between data centers whena specific event such as a major storm or po-litical intervention occurs then a DCI may bethe way to go
Perhaps the best advice is to consider care-fully your actual business requirements Mi-grating virtual workloads between data cen-ters creates unique technical problems due tothe complexity of traffic flows The followingtechnical concerns are just a few of the less-obvious problems created by DCI
gtgt Tracing application problems can be dif-ficult when servers might be in two locations
gtgt Applications incur latency over the DCIfor just one or two servers resulting in unpre-dictable performance
gtgt Loop topology failure leads to outagesin both data centers
gtgt Bandwidth exhaustion results in serviceloss and cannot be easily controlled
Layer 2 DCI is a last-resort technology thatallows legacy applications to behave as if theywere in the same Ethernet domain The cor-rect solution is to deploy applications that are
designed to run activeactive in two or moredata centers and avoid deploying DCI If youchoose to implement DCI you should strictlylimit its use to critical applications
Previous Next
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w Table of Contents
LikeLike TweetTweetTweet
ShareShare
Like This Report
Share it
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents
SubscribeSubscribe
Newsletter
Want to stay current on all newInformationWeek Reports Subscribe to our weeklynewsletter and never miss a beat
May 2013 12
Previous
reportsinformationweekcom
reports D a t a C e n t e r I n t e r c o n n e c t s A n O v e r v i e w
MOR
ELIKE THIS
Want More Like ThisInformationWeek creates more than 150 reports like this each year and theyrsquore all free to registeredusers Wersquoll help you sort through vendor claims justify IT projects and implement new systems by pro-viding analysis and advice from IT professionals Right now on our site yoursquoll find
Strategy OpenFlow vs Traditional NetworksOpenFlow and SDN have the potential to simplify network operations and management while driving down hardware costs But they would also requireIT to rethink traditional network architectures At the same time other protocols are available or emerg-ing that can provide many of the same benefits without requiring major changes Wersquoll look at the prosand cons of OpenFlow and SDN and how they stack up with existing options to simplify networking
SDN Buyerrsquos Guide SDN products are finally hitting the enterprise market Do you have a strategyThis report the companion to our online comparison explains key factors to consider in four areas soft-ware-defined networking controllers applications physical or virtual switches and other compatiblehardware
Research IT Pro Ranking Data Center Networking Cisco has an iron grip on the data center network One reason is its reputation for quality The company scores a 43 out of 5 for reliability a ratingno other vendor matched That said technology and market changes are loosening Ciscorsquos hold Will theshift to virtual infrastructure next-gen Ethernet and commodity switching components change thevendor pecking order More than 500 IT pros weighed in to evaluate seven vendors
PLUS Find signature reports such as the InformationWeek Salary Survey InformationWeek 500 and theannual State of Security report full issues and much more
Table of Contents