오픈스택 멀티노드 설치 후기
TRANSCRIPT
System Engineering Laboratory
Kim Young Woo
OpenStack Networking and Installation with Linux Bridge
2016-05-231
2016-05-232
부정확합니다. 전부믿지마세요.
Contents
• Openstack Network
• ML2
• Provider Networks and Self-Service Networks(Tenant Networks?, Project Network?)
• Network Type : Local, Flat, VLAN, GRE and VxLAN
2016-05-233
• 설치환경및구성
• Trouble Shooting
• 결론
2016-05-234
OpenStack Network - ML2
• 기존에사용되던 L2 Agent(Linux Bridge, OVS)
• 새로운 Plug-in을작성할때마다해줘야할게너무많다.
• 쓸때없는코드들줄이자
• 개발및유지비용을줄이자
출처 : https://www.openstack.org/assets/presentation-media/ML2-Past-Present-and-Future.pptx
2016-05-235
OpenStack Network - ML2
• ML2 : Layer 2의다양하고복잡한기술들을동시에이용가능하게 하는 Plugin
• Linux Bridge, OVS agent들을포함하는상위개념• Driver를통해 Type과 Mechanism을 Pluggable 하게만든다.
• 같은네트워크내에서도다양한 Mechanism Driver를이용이가능하다.
출처 : https://www.openstack.org/assets/presentation-media/ML2-Past-Present-and-Future.pptx
2016-05-236
OpenStack Network - ML2
• ML2 L2 Population 적용전
출처 : https://www.openstack.org/assets/presentation-media/ML2-Past-Present-and-Future.pptx
2016-05-237
OpenStack Network - ML2
• ML2 L2 Population 적용전
출처 : https://www.openstack.org/assets/presentation-media/ML2-Past-Present-and-Future.pptx
2016-05-238
OpenStack Network – Provider Networks and Self-Service Networks
• Provider Network와 Self-Service Network(Tenant Networks?, Project Network?)
• Provider Network : L2 서비스와 VLAN 세그멘테이션을가장간단하게구축, L3 서비스에대해서는물리네트워크인프라에의존함. VM 간의 Private network, L3 서비스(라우팅), LBaas, FWaas 지원안함.
• Self-Service Network : VxLAN과같은 Overlay 세그멘테이션방식을사용하여 self-service를가능하게할수있음. LBaas, FWaas 지원
• 내가이해한두네트워크의차이 : 인프라관리자가아닌사람도가상네트워크인프라를만드는것이가능한네트워크 Self-Service Network. 관리자만관리가가능하고외부네트워크를가능하게해주는것이 Provider Network.
출처 : http://docs.openstack.org/liberty/install-guide-rdo/overview.html
2016-05-239
OpenStack Network – Network Type : Local, Flat, VLAN, GRE and VxLAN
• Local : 동일한컴퓨트노드와동일한네트워크에있는인스턴스끼리통신가능
• Flat(nova-network??) : 모두가같은네트워크대역에놓여있는상태. 다른테넌트에있는인스턴스끼리트래픽확인이가능(보안문제)
• VLAN : VLAN ID(802.1Q tagging)를이용하여네트워크를분리. 같은 VLAN에있는인스턴스는같은네트워크에있다고취급. 다른테넌트에있는인스턴스에같은 IP할당가능. 관리자입장에서는각테넌트가같은 Subnet, IP를사용하더라도상관없으므로편리해졌음. VLAN 수 4096개제한은 VLAN ID 비트수가 12개때문.
• VxLAN and GRE : 오버레이네트워크를형성하여모든호스트간터널을뚫어 P2P 통신을가능하게함(Mesh Network 구성). 24비트 VxLAN ID로대략 1600만개의 LAN 구성가능
출처 : http://docs.openstack.org/liberty/networking-guide/intro-os-networking-overview.html
• Liberty 멀티노드설치문서를참고하여설치• Host : Windows 10 Virtual Box 이용• Controller node(Centos 7) 1대 (192.168.122.10, 10.0.3.4) (3core, 6GB)
• Compute node(Centos 7) 1대 (192.168.122.20, 10.0.3.5) (3core, 6GB)
• Keystone, Glance, Nova, Neutron, Horizon 설치• Provider Network : Flat, Project Network : VxLAN
2016-05-2310
설치환경및구성
Windows 10
Controller Compute
enp0s3 enp0s8 enp0s3 enp0s8Management Network(Host-only Network)
192.168.122.x/24
External Network(Nat Network)
10.0.3.x/24
2016-05-2311
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
Internet
VxLAN
TunnelsPhysical NetworkManagement Network(Host-only Network)
192.168.122.x/24
External Network(Nat Network)
10.0.3.x/24
Public Provider Network
10.0.3.x/24
Private Project Network
10.0.4.x/24
2016-05-2312
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2313
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2314
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2315
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2316
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private Bridge
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2317
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2318
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2319
설치환경및구성
Controller
Compute
tap
Port
tap
DHCP Namespace
qdhcp
Public Bridge
brq
enp0s3
10.0.3.4
Port
tap
Router Namespace
qrouter
qg qrIptablestap
DHCP Namespace
qdhcp
Port
tap
Private Bridge
brq
VxLAN
Interface
Port
tap
enp0s8
192.168.122.10
Public Bridge
brq
IptablesPort
tap
enp0s3
10.0.3.5
Private
brq
IptablesPort
tap
VxLAN
Interface
enp0s8
192.168.122.20eth0
Public
Instance
eth0
Private
Instance
VxLAN
TunnelsPhysical Network
2016-05-2320
설치환경및구성
• Nova-vnc로접속
2016-05-2321
설치환경및구성
• 포트포워딩으로외부에서접속
2016-05-2322
설치환경및구성
• 질문!!
• Private Network의인스턴스에게 Floating IP를할당했는데왜 NIC는늘어나지않는지… 상태는왜비활성화되어있는지
2016-05-2323
Trouble Shooting
•NTP 설정문제
• 매뉴얼대로하면 Controller 노드와 Compute 노드가동기화가안됨
• 방화벽설정필요
# firewall-cmd --permanent --zone=public --add-port=123/udp
# firewall-cmd --reload
2016-05-2324
Trouble Shooting
•DB Access deny 에러
• 예) 1045, "Access denied for user 'neutron'@'controller'
• 계정설정문제, DB 비밀번호및설정파일확인
2016-05-2325
Trouble Shooting
•Rabbitmq 통신에러
• ERROR oslo.messaging._drivers.impl_rabbit [req-0387616a-9695-471a-8520-21bd503285ce - - - - -] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.]
• 방화벽문제
# firewall-cmd --permanent --zone=public --add-port=5672/tcp
# firewall-cmd --reload
2016-05-2326
Trouble Shooting
•Rabbitmq 통신에러
• [compute node, nova log] AMQP server controller:5672 closed the connection. Check login credentials: Socket close
• [controller rabbitmq log]
=ERROR REPORT==== 15-May-2016::10:27:23 ===
closing AMQP connection <0.7769.0> (192.168.122.10:34042 -> 192.168.122.10:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'openstack' - invalid credentials",
'connection.start_ok'}}
• Rabbitmq 계정문제
# rabbitmqctl add_user openstack "PASSWORD“
# rabbitmqctl set_permissions openstack ".*" ".*" ".*"
2016-05-2327
Trouble Shooting
•Neutron linuxbridge 에러
• [controller neutron linuxbridge log] AMQP server controller:5672 closed the connection. Check login credentials: Socket close
• [compute neutron linuxbridge log] AttributeError: 'LinuxBridgeNeutronAgentRPC' object has no attribute 'plugin_rpc'
• 원인모름재부팅후해결
2016-05-2328
Trouble Shooting
•문제의시작
• 각머신에서호스트네임을안바꿔서그런지 nova-manage service list나 neutron agent-list와같은명령어를입력했을때 host가 “localhost.localdomain”으로출력
• 마음에들지않음각머신호스트네임을바꾸고재시작
• 명령어입력시이전의호스트네임이남아있음… 더마음에안듬…
2016-05-2329
Trouble Shooting
•문제의시작• Nova와 Neutron DB에접근해서직접삭제
2016-05-2330
Trouble Shooting
•인스턴스구동시문제
• nova hypervisor list 명령어시에러가남. 하이퍼바이저를못찾아서구동이안되는건가?
• LinuxBridge 로그에계속해서에러
• Error in agent loop. Devices info: {'current': set(['tap4e0eaaa4-d9', 'tapc5683608-db', 'tap164d30dd-3d', 'tapbd79f0fd-b1']), 'removed': set([]), 'added': set(['tap4e0eaaa4-d9', 'tapc5683608-db', 'tap164d30dd-3d', 'tapbd79f0fd-b1']), 'updated': set([])}Error in agent loop. Devices info: {'current': set(['tap4e0eaaa4-d9', 'tapc5683608-db', 'tap164d30dd-3d', 'tapbd79f0fd-b1']), 'removed': set([]), 'added': set(['tap4e0eaaa4-d9', 'tapc5683608-db', 'tap164d30dd-3d', 'tapbd79f0fd-b1']), 'updated': set([])}
• Nova api 로그에도에러, 여기저기에러~
• Exception.ComputeHostNotFound(host=host)\ 2016-05-17 06:42:37.514 3155 ERROR nova.api.openstack.extensions ComputeHostNotFound: Compute host localhost.localdomain could not be found.
2016-05-2331
Trouble Shooting
•인스턴스구동시문제
•DB에서호스트네임을가져오는듯함
2016-05-2332
Trouble Shooting
•인스턴스구동시문제
•DB에서호스트네임을가져오는듯함
2016-05-2333
Trouble Shooting
•인스턴스구동시문제
•Nova DB의 compute_nodes 테이블의해당튜플삭제•Nova hypervisor-list 정상적으로동작
•그래도인스턴스구동안됨ㅋㅋㅋㅋ
•한참을헤매다가컴퓨트노드의 local_ip 설정이잘못되어있는것발견. 드디어해결.
2016-05-2334
결론
•네트워크는어렵다.
•설치진행하기전에호스트네임을꼭바꾸자. 설치중간에절대바꾸지말자.
•정신똑바로차리고오타내지말자.