openstack networking juno l3 h-a, dvr

OpenStack Networking - Juno- - DVR & L3 High Availability

Paul SimTechnical Account [email protected]

● Distributed Virtual Router

○ Packet flow

○ Architecture

■ SNAT,

■ DNAT(Floating IP)

■ East<->West

● L3 High Availability

Index

DVR (Distributed Virtual Router)- Installation

Network node

Neutron server

eth1 eth2

eth0

Neutron ML2 plugin

Neutron metadata-agent

Neutron L3/dhcp-agent

External network

Compute node - 1

Nova compute

eth1 eth2

eth0

Neutron ML2 plugin


Neutron L3-agent

Management

Data

Compute node - 2

Nova compute

eth1 eth2

eth0

Neutron ML2 plugin


Neutron L3-agent

DVR (Distributed Virtual Router)- Packet flow

Compute node - 1

br-

ex

VM VMGRE Tunnel

VM

br-int

Network node

br-

ex

br-tun

br-int

br-tun

Compute node - 2

VM VM VM

br-int

br-

tun

1.SNAT

External network

3. East-West traffic

2. Floating IP

OVS bridge

DVR (Distributed Virtual Router)- SNAT : Network node

Namespace

OVS bridgeNetwork node

qdhcp-b

r-tun

eth0

br-intpatch-tun

patch-int

gre~

snat- qrouter-

tap taptap

sg~50.50.6.2ns~ qr~

qg~192.168.10.109

SNAT br-ex

tap

DVR (Distributed Virtual Router)- SNAT : Compute node

Compute node

Namespace

OVS bridge

VM

br-int

br-

tun

gre

~

qvo~

qbr~

tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~

Linux bridge

sg~

on

netw

ork

no

de

packet flow

traffic flow

DVR (Distributed Virtual Router)- SNAT : Compute node

Namespace

OVS bridge

Linux bridge

Compute node

VM

br-int

br-

tun

gre

~

qvo~

qbr~

tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~sg~

(50.

50.6

.2)

on

netw

ork

no

de

ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls

0: from all lookup local 32766: from all lookup main 32767: from all lookup default 842139137: from 50.50.6.1/24 lookup

842139137ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 842139137

default via 50.50.6.2 dev qr-9722faba-b7

DVR (Distributed Virtual Router)- Floating IP/DNAT : Compute node

Compute node

Namespace

OVS bridge

VM

br-int

br-

tun

gre

~

qvo~

qbr~

tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~

Linux bridge

packet flow

br-extap

eth0

fip-

fpr~ rfp~

fg~

RouteRoute

NAT

veth pair


Compute node

Namespace

OVS bridge

VM

br-int

br-

tun

gre

~

qvo~

qbr~

tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~

Linux bridge

packet flow

br-extap

eth0

fip-

fpr~ rfp~

fg~

RouteRoute

NAT

veth pair

ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls0: from all lookup local 32766: from all lookup main 32767: from all lookup default 32770: from 50.50.5.5 lookup 16 842138881: from 50.50.5.1/24 lookup 842138881 842138881: from 50.50.5.1/24 lookup 842138881 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 16

default via 169.254.31.29 dev rfp-20838b7d-a


Compute node

Namespace

OVS bridge

VM

br-int

br-

tun

gre

~

qvo~

qbr~

tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~

Linux bridge

packet flow

br-extap

eth0

fip-

fpr~ rfp~

fg~

RouteRoute

NAT

veth pair

ubuntu@ubuntu-6:~$ sudo ip netns exec fip-02f9d340-2caa-4c05-86fb-460c9580f9df ip route show

default via 192.168.10.1 dev fg-f3887d61-2d 192.168.10.114 via 169.254.31.28 dev fpr-20838b7d-a

DVR (Distributed Virtual Router)- East-West traffic flow : Compute node

Compute node-2

Namespace OVS bridge

VM50.50.6.3

br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~

Linux bridge packet flow

Compute node-1

tap~

qr~50.50.5.1

VM50.50.5.3

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~

ICMP Request

ICMP Replyi.e., ping 50.50.5.3 -> 50.50.6.3

DVR (Distributed Virtual Router)- East-West traffic flow : network topology


Compute node-2


br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~


Compute node-1

tap~

qr~50.50.5.1

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~

VM50.50.6.3

ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip link2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff

ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9-92aa-adec797d600e ip link2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff

VM50.50.5.3


Compute node-2


VM50.50.6.3

br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~


Compute node-1

tap~

qr~50.50.5.1

VM50.50.5.3

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~

ICMP Request 50.50.5.3 -> 50.50.6.3Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3

MAC 50.50.6.3 : fa:16:3e:ff:85:9b50.50.6.1 : fa:16:3e:71:3d:5a50.50.5.1 : fa:16:3e:15:1e:e050.50.5.3 : fa:16:3e:ce:8c:35

DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cfCompute Node -2 : fa:16:3f:72:60:33

SRC MAC : fa:16:3e:71:3d:5aSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3

SRC MAC : fa:16:3e:71:3d:5aSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3

SRC MAC : fa:16:3e:ce:8c:35SRC IP : 50.50.5.3 DST MAC : fa:16:3e:15:1e:e0DST IP : 50.50.6.3

GRE tunnel 0x3

SRC MAC : fa:16:3f:5e:a0:cfSRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.6.3


Compute node-2


VM50.50.6.3

br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~


Compute node-1

tap~

qr~50.50.5.1

VM50.50.5.3

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~

ICMP Reply 50.50.6.3 -> 50.50.5.3

SRC MAC : fa:16:3e:15:1e:e0SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ff:85:9bDST IP : 50.50.5.3

SRC MAC : fa:16:3e:15:1e:e0SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35DST IP : 50.50.5.3

SRC MAC : fa:16:3e:ff:85:9bSRC IP : 50.50.6.3 DST MAC : fa:16:3e:71:3d:5aDST IP : 50.50.5.3

Segmentation ID : 50.50.5.0/24 : 0x150.50.6.0/24 : 0x3



GRE tunnel 0x1

SRC MAC : fa:16:3f:72:60:33SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35DST IP : 50.50.5.3


Compute node-2


VM50.50.6.3

br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~


Compute node-1

tap~

qr~50.50.5.1

VM50.50.5.3

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~




table=0, n_packets=9178, n_bytes=1009035, idle_age=17470, hard_age=65534, priority=1 actions=NORMAL

table=0, n_packets=2066, n_bytes=214544, idle_age=5, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)table=1, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=1,dl_vlan=2,dl_src=fa:16:3e:71:3d:5a actions=mod_dl_src:fa:16:3f:5e:a0:cf,resubmit(,2)table=2, n_packets=1849, n_bytes=183458, idle_age=5, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)table=20, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=2,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,set_tunnel:0x3,output:3


Compute node-2


VM50.50.6.3

br-int

br-

tun

gre

~

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~


Compute node-1

tap~

qr~50.50.5.1

VM50.50.5.3

br-int

br-

tun

qvo~

qbr~tap~

qvb~

patch-int

qrouter-

qr~50.50.6.1

patch-tun

tap~tap~

qr~50.50.5.1

gre~




table=0, n_packets=1789, n_bytes=175146, idle_age=17, hard_age=65534, priority=2,in_port=3,dl_src=fa:16:3f:5e:a0:cf actions=resubmit(,1)table=1, n_packets=1765, n_bytes=172970, idle_age=17, hard_age=65534, priority=4,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,mod_dl_src:fa:16:3e:71:3d:5a,output:8

table=0, n_packets=1857, n_bytes=184993, idle_age=18, hard_age=65534, priority=1,in_port=2 actions=resubmit(,3)table=3, n_packets=1993, n_bytes=195880, idle_age=18, hard_age=65534, priority=1,tun_id=0x3 actions=mod_vlan_vid:2,resubmit(,9)table=9, n_packets=1789, n_bytes=175146, idle_age=18, hard_age=65534, priority=1,dl_src=fa:16:3f:5e:a0:cf actions=output:1

L3 High Availability- Installation

Network node-1

Neutron server

eth1 eth2

eth0

Neutron ML2 plugin



External network

Management

Data

KeepAlived

Network node-2

Neutron server

eth1 eth2

eth0

Neutron ML2 plugin



KeepAlived Compute node - 1

Nova compute

eth1 eth2

eth0

Neutron ML2 plugin

Compute node - 2

eth1 eth2

eth0

Neutron ML2 plugin

Nova compute

Network node-2

Compute node - 3

Compute node - 2

Network node-1

vRouter A - Master

L3 High Availability

Compute node - 1

Subnet 1

Subnet 3

Subnet 2

Subnet 5

vRouter B - Backup

vRouter C - Backup

vRouter D - Master

vRouter C - Master

vRouter D - Backup

vRouter A - Backup

Subnet 3

Subnet 4

vRouter B - Master

Tenant X

Tenant Y

Tenant Z

VRRP



Network node-1

qdhcp-

br-tun

br-int

qrouter-

ha~ns~ qr~qg~

br-ex

Network node-2

qdhcp-

br-

tun

br-int

qrouter-

qr~qg~

br-ex

ns~

KeepAlived KeepAlived

ha~

ubuntu@ubuntu-5:~$ sudo ip netns exec qrouter-d8625260-88a1-4312-b788-c04fc9094356 tcpdump -n -i ha-27fe59da-a8tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on ha-27fe59da-a8, link-type EN10MB (Ethernet), capture size 65535 bytes16:16:25.213440 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:27.214607 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:29.215796 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 2016:16:31.216986 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20


Network node-1

qdhcp-

br-tun

eth0

br-int patch-tun

patch-int

qrouter-

tap taptap

ha~ns~ qr~

qg~

br-extap

Network node-2

qdhcp-

br-

tun

eth0

br-intpatch-tunpat

ch-i

nt

gre

~

qrouter-

tap taptap

ha~ns~ qr~

qg~

br-extap


gre~



Network node-1

qdhcp-

br-tun

br-int

qrouter-

ha~ns~ qr~qg~

br-ex

KeepAlived

ubuntu@ubuntu-5:~$ cat /var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/keepalived.confvrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788-c04fc9094356/notify_fault.sh"}vrrp_instance VR_1 { state BACKUP interface ha-27fe59da-a8 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-27fe59da-a8 } virtual_ipaddress { 192.168.10.118/24 dev qg-8fffbd7e-8a } virtual_ipaddress_excluded { 50.50.1.1/24 dev qr-dee474e1-1e } virtual_routes { 0.0.0.0/0 via 192.168.10.51 dev qg-8fffbd7e-8a }}

Network nodeTenant A



br-tun

br-int

qrouter-

ha~

br-ex

KeepAlived

qrouter-

ha~

KeepAlived

HA network : 169.254.192.0/18 Segmentation id : 0x6

Tenant B

qrouter-

ha~

KeepAlived

qrouter-

ha~

KeepAlived

HA network : 169.254.192.0/18 Segmentation id : 0x7

● One KeepAlived instance per vRouter● One HA network per tenant

○ Each HA network has separate segmentation id

○ allow_overlapping_ips = True

● Maximum 255 HA routers per tenant.

openstack networking juno l3 h-a, dvr

Technology

virtual router snat

lookup default

lookup local

lookup main

snat qrouter

aaadec797d600e ip route

dev qr

aaadec797d600e ip rule