make your dvr playground using devstack
TRANSCRIPT
![Page 1: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/1.jpg)
Make your DVR playground using Devstack
江骏 @ohmystack (Twitter, Instagram)
@任我骏驰 (Weibo)
Distributed Virtual Router
![Page 2: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/2.jpg)
不是每个人天生就是做网络的
![Page 3: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/3.jpg)
但是,离开了网络 & 存储,IaaS 又是没有意义的
![Page 4: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/4.jpg)
Icehouse Juno
Kilo(Count by reviews)
http://stackalytics.com/?release=kilo&metric=marks
![Page 5: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/5.jpg)
应用层
表示层
会话层
传输层 (端到端的数据交换机制)
网络层 (Packet,路由和寻址,使端到端找到最佳路径)
数据链路层(Frame,连接的创建、维持、释放,保证端到端)
物理层
✔
✔
![Page 6: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/6.jpg)
Basic networking
• Ethernet
• VLANs
• ARP
• IP
• ICMP/TCP/UDP
Network components
• Switches
• Routers
• Firewalls
• Load balancers
![Page 7: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/7.jpg)
Tunnel technologies
• Generic routing encapsulation (GRE)
• Virtual extensible local area network (VXLAN)
Network namespaces
• Virtual routing and forwarding (VRF)
• Linux network namespaces
# ip netns# ip netns exec <namespace> <command>
![Page 8: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/8.jpg)
进入虚拟的世界 Linux 抽象网络设备简介
http://www.ibm.com/developerworks/cn/linux/1310_xiawc_networkdevice/index.html
• bridge
• VLAN device
• TAP
• VETH (pair)
![Page 9: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/9.jpg)
Scenario1: VLAN Networking
![Page 10: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/10.jpg)
OpenStack Networking - Neutron
• API server
• Plug-in and Agents
• Messaging queue
![Page 11: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/11.jpg)
OpenStack networking concepts
• Tenant networksFlat, VLAN, GRE and VXLAN
• Provider networks
• Subnets
• Ports
• Routers
• Security groups
• Extensions
![Page 12: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/12.jpg)
ML2 (Modular Layer 2)Tenant network types
• Local
• VLAN
• GRE
• VXLAN
Mechanism
• Linux bridge
• Open vSwitch
• L2 population
• Specialized • Open source
• Proprietary(vendor)
Agents
• L3
• DHCP
• Metadata
![Page 13: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/13.jpg)
ML2 (Modular Layer 2)
·ML2 让大家不局限于每次只能用一种 plugin ( Linux Bridge / OVS )
·底下的实现可以不同,不管是 KVM 还是 hyper-V,只要它们共用一个 VLAN,它们就能通讯。
·让 L3 变成了 L3 as a Service plugin
![Page 14: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/14.jpg)
L2 population为什么需要?
1. Unicast(MAC 未被习得时), multicast, broadcast traffic is flooded out tunnels to all other compute nodes
2. 有非常多的 ARP,有没有好一点的办法处理它们 3. 如果一个 node 上根本没有 VM,那么那个 node 还需不需要收到那些 broadcast。
http://assafmuller.com/2014/02/23/ml2-address-population/
![Page 15: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/15.jpg)
L2 populationL2 population 在宿主机通过 Proxy ARP,
提前拦下 ARP 请求,主动根据数据库回答目标主机在哪。
Without L2 pop With L2 pophttp://docs.openstack.org/admin-guide-cloud/content/ml2_l2pop_scenarios.html
![Page 16: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/16.jpg)
L3 Agent
· L3 Agent 用 namespaces 提供多个独立的 virtual routers(Overlapping IPs. 用户更加自由创建 subnet)
· L3 的功能基本上就是 SNAT、floating-ip
Namespaces: qdhcp-xxx - 第⼀一次创建 VM 时添加 qrouter-xxx - 设置 gateway 时添加
* router/network 删除后,ns 需要⼿手动删除
为什么需要? 1. Flat 需要搭 bridge,VLANs 需要人工手动去配交换机、网关。 2. 所有路由都有上层的设备控制,firewall, router, NAT... 3. 这些都影响灵活和拓展性。
![Page 17: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/17.jpg)
Scenario2: with Open vSwitch
RouterNetwork Node
External
Network1 Network2
E-W Tra
fficE-W TrafficN-S Traffic
N-S Traffic
![Page 18: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/18.jpg)
L3 with HA有三种方法:
1. “迁移法” 但耗时长(故障恢复时间长)。 https://review.openstack.org/#/c/110893/
2. “VRRP active/passive” 让 router 分布在多个 network node 上。 http://assafmuller.com/2014/08/16/layer-3-high-availability/
3. “VRRP + DVR” 至少让 floating-ip 部分能分布式,剩下的 SNAT 仍用 VRRP。
仅采用 VRRP 方式的缺陷:
·TCP Connection Tracking
·⋯
![Page 19: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/19.jpg)
Scenario3: with DVR (Distributed Virtual Router)
![Page 20: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/20.jpg)
Network node cpu-node2cpu-node1
Scenario3: with DVR (Distributed Virtual Router)
SNATRouter
External
VM with Network1
VM with Network2
E-W Traffic
N-S Traffic (with floating IPs)
DistributedRouter
DistributedRouter
N-S Traffic (with floating IPs)
N-S Traffic (without floating IPs)
·East/west traffic: 同个 tenant 的不同 networks。compute-node 之间直接发送,不再需要走 network-node。
·North/south traffic with floating IPs: 直接由 compute-node 进行路由,不再需要走 network-node。
·North/south traffic without floating IPs: 仍旧去 network-node,走 SNAT 出去。
![Page 21: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/21.jpg)
Deploy OpenStack Env• Devstack
• Salt - cloudbase/salt-openstack
• Docker - stackforge/kolla
• Puppet
• Chef
• Juju
• TripleO - OpenStack on OpenStack
• …
![Page 22: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/22.jpg)
Devstack Gate
• http://ci.openstack.org/devstack-gate.html
• An integration test
• All changes to core OpenStack projects are “gated” on a set of tests so that it will not be merged into the main repository unless it passes all of the configured tests.
![Page 23: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/23.jpg)
![Page 24: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/24.jpg)
✔
![Page 25: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/25.jpg)
Devstack 的一些注意点• Proxy 问题
local.conf 中虽然可以配 http_proxy,但不建议使⽤用,它容易导致 pip 安装时出现 md5 check 失败。
• ⻓长时间⽤用同⼀一份 /opt/stack 下⾯面的源码,会引发⼀一些 requirements 的问题。
• 让 Dashboard 拥有 VNC Console 从某⼀一版本对 Devstack 后,cauth 就从默认安装⾥里去除了。所以,我们得往配置上⾯面⼿手动加上。
• 强制从 Git 安装某些 client 包Devstack 安装那些主要服务时,默认会从 Git 拉下来安装;装其它那些 client 包时,就直接从 pypi 装了。如果希望强制某些 client 包也从 Git 安装,可以这样:
• 如何备份 /opt/stack
export http_proxy=‘http://xxx.xxx.xxx.xxx:xxxx’
LIBS_FROM_GIT=python-novaclient,python-neutronclient
enable_service n-cauth
# cd /opt# tar cf ~/stack.tar stack \--exclude='stack/data' \--exclude='stack/logs' \--exclude='stack/status' \--exclude='stack/.wheelhouse' \--exclude='*.pyc' \--exclude='.venv'
![Page 26: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/26.jpg)
https://gist.github.com/ohmystack/93aa8215456124fb147c3 个 node 的 local.conf ⽂文件
net-node cpu-nodeall-in-one-node
SNATRouter
External
VM with Network1
VM with Network
E-W Traffic
N-S Traffic (with floating IPs)
DistributedRouter
DistributedRouter
N-S Traffic (with floating IPs)
N-S Traffic (without
10.0.2.0/24����������� ������������������ GateWay:����������� ������������������ 10.0.2.2
10.0.0.0/24 10.0.1.0/24
10.0.2.2210.0.2.2110.0.2.23
参考:http://assafmuller.com/2015/04/06/multinode-dvr-devstack/
![Page 27: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/27.jpg)
ip link ip link show upip addr # shows status of all physical and virtual devices ip addr show to/dev/label ... # ip addr show label 'eth*' ip route show to 10.18.5.0/24 ip route show table main ip route show to/from/table/type/dev/via/src ... ip route get to/from/... ... # Set a Route to the Locally Connected Network eth0 ip route add 192.168.1.0/24 dev eth0 # Set a default route ip route add default via 192.168.1.254
ip netnsip netns exec <namespasce> <command>
brctl show
ovs-vsctl show # shows interfaces and bridges in the virtual switchovs-vsctl add-br <br-name>ovs-vsctl add-port <br-name> <interface>
ovs-dpctl show # shows data path on the switch ovs-ofctl dump-flows br-tun
iptables -L -n -v # check iptables rules iptables-save tcpdump -n -i <interface> -w <filename>
![Page 28: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/28.jpg)
DEMO
![Page 29: Make your DVR playground using DevStack](https://reader034.vdocuments.site/reader034/viewer/2022042619/58f0a65b1a28ab9e0f8b4665/html5/thumbnails/29.jpg)
How to config br-exhttp://docs.openstack.org/havana/install-guide/install/apt/content/install-neutron.install-plug-in.ovs.html
http://assafmuller.com/2015/04/06/multinode-dvr-devstack/
But,����������� ������������������ how?ifconfig br-ex promisc upifconfig eth0 0.0.0.0ifconfig eth0 promisc ifconfig br-ex 10.0.2.21 netmask 255.255.255.0ovs-vsctl add-port br-ex eth0# Then, re-add the default route if neededip route add default via 10.0.2.2