openstackユーザ会資料 - masakari
TRANSCRIPT
![Page 1: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/1.jpg)
© 2016 NTT Software Innovation Center
Masakari と HA コミュニティの状況
2016/03/24室井 雅仁
![Page 2: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/2.jpg)
2Copyright©2016 NTT corp. All Rights Reserved.
自己紹介
• 室井 雅仁 ( むろい まさひと )• 所属 : NTT• OpenStack を利用した OSS クラウドのアーキテクトを担当• 社内向け OpenStack 環境の運用、コミュニティへのアップス
トリーム• OpenStack Congress Core Reviewer
• https://wiki.openstack.org/wiki/Congress• https://thinkit.co.jp/article/8381
• IRC: masahito• エディタ : Emacs 派
![Page 3: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/3.jpg)
3Copyright©2016 NTT corp. All Rights Reserved.
http://www.slideshare.net/masahito12/openstack-masakari
![Page 4: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/4.jpg)
4Copyright©2016 NTT corp. All Rights Reserved.
本日のお品書き
• Masakari• OpenStack HA コミュニティの現状• Masakari デモ ( 時間があれば )
![Page 5: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/5.jpg)
5Copyright©2016 NTT corp. All Rights Reserved.
Masakari
![Page 6: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/6.jpg)
6Copyright©2016 NTT corp. All Rights Reserved.
What’s Masakari
Copyright © いらすとや. All Rights Reserved.
まさかり
![Page 7: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/7.jpg)
7Copyright©2016 NTT corp. All Rights Reserved.
What’s Masakari in OpenStack
• Virtual Machine High Availability (VM-HA) service• Rescue Virtual Machine when any errors occur• Published as OSS at github https://github.com/ntt-sic/masakari
![Page 8: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/8.jpg)
8Copyright©2016 NTT corp. All Rights Reserved.
Motivations
• Pets vs Cattle
• Unable to change all Apps to Cloud Native at once
• Open Source
![Page 9: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/9.jpg)
9Copyright©2016 NTT corp. All Rights Reserved.
Requirements for Pets Model
• Detect 3 types of VM down• VM instance down• VM manager down• Host down
• Recover VM within 5 mins
• Work automatically
![Page 10: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/10.jpg)
10Copyright©2016 NTT corp. All Rights Reserved.
Architecture OverviewC
ompu
te N
odes
Con
trolle
r Nod
es
& B
acke
nd N
odes
![Page 11: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/11.jpg)
11Copyright©2016 NTT corp. All Rights Reserved.
How to detect the 3 down
• VM down• Monitoring libvert’s events
• Manager Process down • Monitoring manager process
• Host down• Monitoring Pacemaker status
![Page 12: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/12.jpg)
12Copyright©2016 NTT corp. All Rights Reserved.
Detect VM Down
Libvirt
Masakari
1. Notify down VM’s Info(VM-ID, Host Name, etc.)
Libvirt Monitor
Detect VM downVM1 VM2 VM3
Libvirt
Libvirt Monitor
VM5 VM6
HostHost
Nova
2. Call Rebuild API for the down VM
3. Rebuild the VM
Down
0. Set callbacks for libvert’s events hook
![Page 13: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/13.jpg)
13Copyright©2016 NTT corp. All Rights Reserved.
Manager Process Down
1. Restart manager process when it’s down
Process Monitor
Masakari
2. Notify manager process down if fail to restart few times
Libvirt Nova-compute
Host A
Libvirt Nova-compute
Host B
Nova
3. Notify Nova to disable schedule for Host A
Process Monitor
Down
![Page 14: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/14.jpg)
14Copyright©2016 NTT corp. All Rights Reserved.
Host Down
RA
CIB
RA
RA
Node’s Status
pacemaker
Heartbeat communications or pacemaker-remote
Masakari
Check its Host’s status
1. Notify another host down
StartStopMonitor
WatchD
og & S
hutdowner
Host Fail Monitor
Polling RA
CIB
RA
RA
Node’s Status
pacemaker
StartStopMonitor
WatchD
og & S
hutdowner
Host Fail Monitor
Polling
Down
Host A Host B
Nova
2. Call Evacuate API for all VM on Host B
![Page 15: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/15.jpg)
15Copyright©2016 NTT corp. All Rights Reserved.
How to use Masakari
1. Prerequisites• Set up Nova and Compute Nodes with KVM• Set up a shared storage per cluster for ephemeral disks (e.g. NFS)
2. Install and Configure Masakari• Download source from github
• https://github.com/ntt-sic/masakari• Install Masakari’s package• Initialize Masakari’s DB• Configure 4 Masakari’s config files
3. Start Masakari• Start all process• Add a reserved host prepared for host down
4. Wait any error• Masakari only works when any error occurs
![Page 16: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/16.jpg)
16Copyright©2016 NTT corp. All Rights Reserved.
Run Masakari in local
• Run masakari-deploy!!http://github.com/ntt-sic/masakari-deploy • For quick QA or demo
![Page 17: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/17.jpg)
17Copyright©2016 NTT corp. All Rights Reserved.
Challenges
• No branch from OpenStack master
![Page 18: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/18.jpg)
18Copyright©2016 NTT corp. All Rights Reserved.
Masakari Next Release
• 1.1.0 is supposed be released until Austin Summit• Supports Pacemaker-Remote for hostmonitor• Supports PostgreSQL and other DB
• By using SQLAlchemy• Supports CentOS
• With python installer and rpm packaging• Uses python-client to call OpenStack API
![Page 19: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/19.jpg)
19Copyright©2016 NTT corp. All Rights Reserved.
OpenStack HA コミュニティの現状
![Page 20: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/20.jpg)
20Copyright©2016 NTT corp. All Rights Reserved.
OpenStack HA チーム
• openstack-ha チーム• VM HA のユースケース議論や今後の実装方針考えるチーム• IRC チャンネル : #openstack-ha• IRC ミーティング : 月曜 18:00 #openstack-meeting• Etherpad:
https://etherpad.openstack.org/p/automatic-evacuation • 参加メンバ ( 企業 )
• NTT, RedHat, SUSE, Intel, ...
ググっても多分みつかりません。チーム名が一般的過ぎて別ページがヒットします。
![Page 21: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/21.jpg)
21Copyright©2016 NTT corp. All Rights Reserved.
チームで議論中の VM HA 方式
1. Masakari• https://github.com/ntt-sic/masakari
2. Resource-agent• https://
github.com/openstack/openstack-resource-agents
3. Mistral workflow• https://github.com/gryf/mistral-evacuate• Mistral プロジェクトを利用した VM-HA フロー
• Mistral: ワークフローエンジン as a Service• https://wiki.openstack.org/wiki/Mistral
![Page 22: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/22.jpg)
22Copyright©2016 NTT corp. All Rights Reserved.
コミュニティ内関係図
VM HA 機能
Masakari Resource-agent
Mistral workflow
HA VMuser story 大
Product WG
執筆
……
VM HA 機能の既存実装
大HA チーム
HA チームとProduct WGで連携しよう
今後の方向性を議論中
あるべき姿の議論中
![Page 23: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/23.jpg)
23Copyright©2016 NTT corp. All Rights Reserved.
コミュニティの今後の展望
• コミュニティとしての VM HA の仕様・実装• Product WG と協力
• 各種 VM HA の実装の方向性議論• Masakari と Mistral のインテグレーションなど
• Austin Summit でチームのサマリーに関するセッションも有ります• High Availability for pets and hypervisors - State of the Nation (
https://www.openstack.org/summit/austin-2016/summit-schedule/events/7327)
• コミュニティでの議論状況の共有が主な話題
• OpenStack HA チームの次回作に乞うご期待 !!
![Page 24: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/24.jpg)
24Copyright©2016 NTT corp. All Rights Reserved.
Masakari デモ
![Page 25: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/25.jpg)
25Copyright©2016 NTT corp. All Rights Reserved.
デモシナリオ
1. VM Instance down
Host1 Host2
VMdown 1.vm down
2.restartController
Nova Masakari
Host1 Host2
VM
down
1.host down
2.evacuateController
Nova Masakari
2. Host down
![Page 26: OpenStackユーザ会資料 - Masakari](https://reader035.vdocuments.site/reader035/viewer/2022070509/58a095a01a28aba73f8b710b/html5/thumbnails/26.jpg)
26Copyright©2016 NTT corp. All Rights Reserved.
Github: https://github.com/ntt-sic/masakariMail: [email protected]
Pull Request 気軽にどうぞ 日本語でも OK です。