ITDepartment

1

Using VPLS for VM mobility

Carles.Kishimoto @ cern.chDavid.Gutierrez @ cern.ch

HEPIX Fall 2015

ITDepartment

2

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

ITDepartment

3

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

ITDepartment

4

• Switzerland• 1000 racks

• Hungary• 300 racks• Wigner Research Centre for Physics• Since January 2013• 2x 100GbE links

CERN Data Center (I)

ITDepartment

5

• Network equipment• Brocade routers• HP Procurve / HP H3C / Brocade switches

CERN Data Center (II)

ITDepartment

6

CERN network (I)

x100GbE links

Distribution CoreAccess

x10GbE

ToR x10GbE

ITDepartment

7

CERN network (II)

• Routed network (OSPF)• No vlans, no spanning-tree• ECMP and LACP• IPv4 / IPv6 Dual stack since 2013• x10GbE switch uplinks (LCG)• 100GbE p2p router interconnects (LCG)• MPLS deployed in the backbone

ITDepartment

8

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

8

ITDepartment

9

The objective (I)• Migrate virtual machines transparently to the new hardware

VirtualMachines

ITDepartment

The objective (II)

• Migrate running virtual machines transparently to the new hardware

• Conditions :• Keep the same ip address on all virtual machines• Live migration

• Restrictions:• CERN is a routed network• We do not extend VLANs across racks

10

ITDepartment

11

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

ITDepartment

The idea

MPLS / IP network

Switch B

Router A Router B

Switch A

12

Row A Row B

VM

ITDepartment

The idea

MPLS / IP network

Switch B

Router A Router B

Switch A

Multimode fiber

13

Row A Row B

VM

ITDepartment

The idea

MPLS / IP network

Switch B

Router A Router B

Switch A

VMVM

VM

14

Row A Row B

ITDepartment

The idea

MPLS / IP network

Switch B

Router A Router B

Switch A

• Connect the switches at router level

15

21 1

2

VM

ITDepartment

The idea

MPLS / IP network

Switch B

Router A Router B

Switch A

• Connect the switches logically using VPLS

16

21 1

2

VM

ITDepartment

What is VPLS ?

• Virtual Private LAN Service• RFC4761 / RFC4762

• Signaling based on BGP or LDP

• From Wikipedia: “Virtual Private LAN Service (VPLS) is a way to provide Ethernet-based multipoint to multipoint communication over IP or MPLS networks”

• You need an MPLS enabled backbone

17

ITDepartment

What is VPLS ?

MPLS / IP network

Switch B

Router A Router B

Switch A

• VPLS emulates an Ethernet switch

18

21 1

2

VM

ITDepartment

19

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

ITDepartment

The solution

vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection!

interface ethernet 5/1 no flow-control load-interval 30 enable!

interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24!

Layer 1

Layer 2

Layer 3 MPLS

20

Switch A Switch B

ITDepartment

The solution

vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection!

interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling!

interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24!

Layer 1

Layer 2

Layer 3 MPLS

21

vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51

Switch A Switch B

ITDepartment

The solution

vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection!

interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling!

interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24!

Layer 1

Layer 2

Layer 3 MPLS

22

vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51

Switch A Switch B

(config)# vlan 51(config-vlan-51)# no router-interface ve 51(config-vlan-51)# no untagged eth 5/1(config-vlan-51)# router mpls (config-mpls)# vpls 1 1 (config-mpls-vpls-1)# router-interface ve 51

ITDepartment

The loop cable (I)

MPLS / IP network

Switch B

Router A Router B

Switch A

• Loop cable installed in distribution routers

23

1 12 2

VM

ITDepartment

The loop cable (II)

MPLS / IP network

Switch B

Router A Router B

Switch A

• Routing still provided by Router A

24

VM

Internet

ITDepartment

The loop cable (III)

MPLS / IP network

Switch B

Router A Router B

Switch A

• Increase VRRP priority in router B

25

Internet

VM

ITDepartment

The loop cable (IV)

• Implemented with 10GigabitEthernet ports• Advantage

• It allows us to create the circuit transparently• We can deploy multiple links in parallel if needed• It can be used for multiple migrations

26

ITDepartment

Operations

• Workflow

1. Service manager will create a ticket

2. Network team will enable the circuit

3. Migration of virtual machines

4. The circuit will be removed

• Open questions:

• Duration of the circuit ?

• Number of circuits simultaneously ?

27

ITDepartment

28

Agenda

• CERN data center

• The objective

• The idea

• The solution

• Conclusions

28

ITDepartment

Conclusions

• It allows us to connect the broadcast domain of different IP services transparently

• It supports multiple circuits at the same time• The solution is based on standard protocols (RFC) • It works… although not tested in production yet• Our tools are ready to support this configuration• It scales as we could connect two IP services from

different data centers (Geneva vs Budapest)

29

ITDepartment

30

Thank you