mlag invisibile layer 2 redundancy
TRANSCRIPT
®
®
MLAG: Invisible Layer 2 Redundancy
Scott Emery
Cumulus Networks
May 20, 2015
®Agenda
u What is MLAG?
u How does MLAG work?
u How to set up an MLAG
u Tools for MLAG analysis and debugging
®MLAG Introduction
You need to set up a rack of servers for a new application
u Add some extra servers for redundancy u Uplink to redundant core switches u Redundant Internet connections u Backup power with batteries and generators u Over-provisioned cooling
You receive a midnight call that everything is down
®MLAG Introduction
MLAG – A LAG across more than one node
u Multi-homing for redundancy
u Active-active to utilize all links which otherwise may get blocked by Spanning Tree
u No modification of LAG partner
®MLAG Terminology
S1 S2
H1 H2 H3 H4 H5
Secondary Role
ISL – Inter-Switch Link
Dually Connected
Primary Role
Singly Connected
®MLAG Partner View
S1 S2
H1 H2 H3 H4 H5
Switch
®The Fundamental Job of MLAG
S1 S2
S1 S2
Make this:
Look like this:
Switch
®MLAG and LACP
u Both ends must run LACP
u Normally, when connected to two different systems, only one link is used • Common system ID is used on
each switch
u Identification of which ports on each system are dual-connected pairs
S1 S2
H1 H2 H3 H4 H5
®Eliminating Duplicate Packets
u BUM1 packets are flooded and result in: § Duplicate packets at dual-connected hosts
§ A dual-connected host receives packets which it transmitted
1 BUM packets are: Broadcast, Unknown unicast, and Multicast
®Eliminating Duplicate Packets
S1 S2
H1 H2 H3 H4 H5
H2 sends a BUM packet which goes up the link to S1
®Eliminating Duplicate Packets
S1 sends the packet out all interfaces in the bridge, except the interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5
®Eliminating Duplicate Packets
S2 sends the packet out all interfaces in the bridge, except the interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5
®Eliminating Duplicate Packets
u Dual-connected hosts receive duplicate copies of the packet
u Dual-connected hosts which send BUM packets receive the packet they sent
u To fix this: Packets received on the ISL are not forwarded to dual-connected ports
®Eliminating Duplicate Packets
S2 only sends packet out singly-connected interfaces
S1 S2
H1 H2 H3 H4 H5
®MAC Address Learning
u To act as a single logical switch, both switches must synchronize their MAC address tables
§ Addresses learned on dual-connected ports are added to the corresponding port on the other switch
§ Addresses learned on singly-connected ports are added to the ISL on the other switch
§ Address learning is disabled on the ISL
®MAC Address Learning
H2 sends a BUM packet, S1 learns the port to H2
S1 S2
H1 H2 H3 H4 H5
H2
®MAC Address Learning
S1 sends the packet out all interfaces in the bridge, except the interface on which the packet arrived
S1 S2
H1 H2 H3 H4 H5
H2
®MAC Address Learning
S2 would ordinarily learn H2 on the ISL and forward the packet out all singly-connected ports
S1 S2
H1 H2 H3 H4 H5
H2 H2
®MAC Address Learning
But, learning is disabled on the ISL. Instead, S1 sends a MAC sync message to S2 which adds H2 to the dual connected port
S1 S2
H1 H2 H3 H4 H5
H2 H2
®MAC Address Learning
For singly-connected hosts, the MAC sync message causes the address to be added to the ISL
S1 S2
H1 H2 H3 H4 H5
H1
H1
®MAC Address Learning
Final MAC address tables may look like this. Red: Address originally learned on switch. Blue: Address added by MAC sync
S1 S2
H1 H2 H3 H4 H5
H2 H5 H1 H4 H3 H5
H2 H1
H4 H3
®Switch-Switch MLAG
u Just like a host can be connected to two switches, a pair of MLAG'd switches can be connected to another pair of MLAG'd switches
§ Used to create larger redundant L2 networks
§ Each pair of MLAG'd switches views the other switches as a single logical switch
®Switch-Switch MLAG
S3 S4
S1 S2
®Switch-Switch MLAG
S3 S4 Switch
S1 S2 Switch
®Spanning Tree
u One switch is set as the primary, the other is secondary
u Both switches use the same bridge ID, dual connected ports have the same port ID
u Only primary sends BPDUs on dual-connected ports
u BPDUs received on dual connected ports are sent to the peer unmodified
u BPDUs received on the root port are sent to the peer unmodified
u Source MACs of BPDUs received on peer link are checked
u Peer link never blocks
S1 S2
M1
R1
®Split Brain
u If one switch sees that the ISL is down it cannot distinguish between the link going down (split brain) and the peer switch going down (solo)
u A backup link is used to make this distinction
S1 S2
H1 H2 H3 H4 H5
S1 S2
H1 H2 H3 H4 H5
??? Which One ???
®Split Brain
u When the ISL goes down, the backup link can determine if the peer switch is still alive
S1 S2
H1 H2 H3 H4 H5
®Configuring MLAG
In /etc/network/interfaces put all dual-connected ports in an 802.3ad bond and assign them a clag-id
auto bond1
iface bond1 inet static
bond-slaves swp48
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
auto bond11
iface bond11 inet static
bond-slaves swp4
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-lacp-rate 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
Switch S1 Switch S2
®Configuring MLAG
In /etc/network/interfaces assign clagd parameters on a VLAN sub-interface of the ISL link
auto peer6.4000 iface peer6.4000 inet static address 169.254.0.1 netmask 255.255.255.0 clagd-peer-ip 169.254.0.2 clagd-sys-mac 44:38:39:ff:bb:01 clagd-backup-ip 192.168.1.101
auto peer16.4000 iface peer16.4000 inet static address 169.254.0.2 netmask 255.255.255.0 clagd-peer-ip 169.254.0.1 clagd-sys-mac 44:38:39:ff:bb:01 clagd-backup-ip 192.168.1.100
Switch S1 Switch S2
®MLAG Tools
clagctl can be used to get the current state of the MLAG
# clagctl The peer is alive Peer Priority, ID, and Role: 32768 00:02:00:00:00:17 primary Our Priority, ID, and Role: 32768 70:72:cf:e9:f0:76 secondary Peer Interface and IP: peer6.4000 169.254.0.2 Backup IP: 192.168.1.101 (active) System MAC: 44:38:39:ff:bb:01 Dual Attached Ports Our Interface Peer Interface CLAG Id ---------------- ---------------- ------- bond4 bond14 4 bond5 bond15 5 bond1 bond11 1 bond2 bond12 2 bond3 bond13 3
$ clagctl The peer is alive Our Priority, ID, and Role: 32768 00:02:00:00:00:17 primary Peer Priority, ID, and Role: 32768 70:72:cf:e9:f0:76 secondary Peer Interface and IP: peer16.4000 169.254.0.1 Backup IP: 192.168.1.100 (active) System MAC: 44:38:39:ff:bb:01 Dual Attached Ports Our Interface Peer Interface CLAG Id ---------------- ---------------- ------- bond14 bond4 4 bond15 bond5 5 bond12 bond2 2 bond13 bond3 3 bond11 bond1 1
Switch S1 Switch S2
®MLAG Tools
/var/log/syslog contains MLAG status changes
# grep clagd /var/log/syslog May 19 16:25:31 act-5712-08 clagd[7253]: Beginning execution of clagd version 1.1.0 May 19 16:25:31 act-5712-08 clagd[7253]: Invoked with: /usr/sbin/clagd --daemon 169.254.0.2 peer6.4000 44:38:39:ff:bb:01 May 19 16:25:31 act-5712-08 clagd[7258]: Role is now secondary May 19 16:25:32 act-5712-08 clagd[7258]: Initial config loaded May 19 16:25:33 act-5712-08 clagd[7258]: The peer switch is active. May 19 16:25:33 act-5712-08 clagd[7258]: Initial data sync from peer done. May 19 16:25:33 act-5712-08 clagd[7258]: Initial handshake done. May 19 16:25:33 act-5712-08 clagd[7258]: Initial data sync to peer done. May 19 16:25:37 act-5712-08 clagd[7258]: bond2 is now dual connected. May 19 16:25:37 act-5712-08 clagd[7258]: bond3 is now dual connected. May 19 16:25:37 act-5712-08 clagd[7258]: bond1 is now dual connected. May 19 16:25:37 act-5712-08 clagd[7258]: bond5 is now dual connected. May 19 16:25:37 act-5712-08 clagd[7258]: bond4 is now dual connected.
®
© 2014 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
§ Thank You!
®
cumulusnetworks.com 32