batou - multi-(host|component|environment|version|platform) deployment
DESCRIPTION
Batou ist ein Open-Source Werkzeug um mit Python das Deployment von komplexen Anwendungen zu beschreiben und zu automatisieren. Das Konzept besteht darin eine passende allgemeine Modellsprache in Form einer API zu definieren. Dabei stehen Wiederverwendung, Toleranz gegenüber unbekannten Zuständen und Ausdrucksstärke im Vordergrund. Der Vortrag zeigt den Werdegang und unsere Motivation zur Entwicklung von batou auf und zeigt die konzeptionellen Stärken anhand praktischer Beispiele. Wir haben in den letzten Jahren eine Reihe großer und kleiner Projekte beim Deployment betreut. Im Rahmen unserer Plattform "gocept.net" bereiten wir unsere Erfahrungen in Dokumentation und offenen Werkzeugen auf und haben daher batou entwickelt. batou ist sowohl in Python geschrieben und nutzt Python um Deployment-Strukturen zu beschreiben. Wir möchten Batou gerne vorstellen um unsere Erfahrungen weiterzugeben und (auch auf den Sprints) anzuregen sich mit dem Werkzeug auseinanderzusetzen, Feedback zu bekommen, und daran weiterzuarbeiten. Siehe auch: http://bitbucket.org/gocept/batouTRANSCRIPT
![Page 1: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/1.jpg)
BATOUMulti-(Component|Environment|Host|Platform) deployment
Christian Theune <[email protected]>
Wednesday, October 31, 12
![Page 2: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/2.jpg)
Wednesday, October 31, 12
![Page 3: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/3.jpg)
Wednesday, October 31, 12
![Page 4: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/4.jpg)
INHALT
•Motivation
• Hello World
• Tiefer in den Kaninchenbau
• Ausblick
Wednesday, October 31, 12
![Page 5: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/5.jpg)
WARUM?
Wednesday, October 31, 12
![Page 6: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/6.jpg)
Wednesday, October 31, 12
![Page 7: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/7.jpg)
Wednesday, October 31, 12
![Page 8: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/8.jpg)
Wednesday, October 31, 12
![Page 9: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/9.jpg)
Wednesday, October 31, 12
![Page 10: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/10.jpg)
Wednesday, October 31, 12
![Page 11: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/11.jpg)
PARTY LIKE ITS ...
• 1999: PHP FTP-Uploads
• 2001: Shell-Skripte um Zope-Instanzen zu verwalten
• 2003: Zope 2.4 “mkinstance”
• 2006: zc.buildout
• 2009: Fabric
Wednesday, October 31, 12
![Page 12: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/12.jpg)
BATOUVS
Puppet, Chef Fabric, Salt JuJu
Systemkonfiguration versus Service deployment
Modellierung Laufzeitumgebung
Wednesday, October 31, 12
![Page 13: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/13.jpg)
HELLO WORLD
http://www.flickr.com/photos/oskay/472097903
Wednesday, October 31, 12
![Page 14: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/14.jpg)
./.hg
./buildout.cfg
./bootstrap.py
./components
./components/hello
./components/hello/component.py
./environments
./environments/dev.cfg
./environments/prod.cfg
$ find .Wednesday, October 31, 12
![Page 15: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/15.jpg)
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', content='Hello world')
components/hello/component.pyWednesday, October 31, 12
![Page 16: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/16.jpg)
[hosts]localhost = hello
environments/dev.cfgWednesday, October 31, 12
![Page 17: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/17.jpg)
$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12
![Page 18: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/18.jpg)
./
./components
./environments
...
./work
./work/hello
./work/hello/hello
$ find .Wednesday, October 31, 12
![Page 19: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/19.jpg)
$ bin/batou-local dev localhost
Wednesday, October 31, 12
![Page 20: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/20.jpg)
$ cat "foo" > work/hello/hello$ bin/batou-local dev localhostUpdating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12
![Page 21: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/21.jpg)
[environment]service_user = testhost_domain = gocept.net
[hosts]test01 = hellotest02 = hello
environments/prod.cfgWednesday, October 31, 12
![Page 22: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/22.jpg)
$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OKDeploying test02.gocept.net/helloUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)OK
Wednesday, October 31, 12
![Page 23: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/23.jpg)
$ bin/batou-remote prodtest02.gocept.net: connectingtest01.gocept.net: connectingtest01.gocept.net: bootstrappingtest02.gocept.net: bootstrappingOKOKDeploying test01.gocept.net/helloOKDeploying test02.gocept.net/helloOK
Wednesday, October 31, 12
![Page 24: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/24.jpg)
HELLO WORLD
• Komponenten und Environments
• batou-local im Konfiguration eines Hosts lokal zu realisieren
• batou-remote um eine ganze Umgebung zu deployen
• Konvergenz und Idempotenz
Wednesday, October 31, 12
![Page 25: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/25.jpg)
IN DEN KANINCHENBAU
• ein Programm
• verwaltet mit Supervisor
Wednesday, October 31, 12
![Page 26: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/26.jpg)
DAS PROGRAMM
#!/usr/bin/env pythonimport timewhile True: print 'Hello, world!' time.sleep(1)
Wednesday, October 31, 12
![Page 27: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/27.jpg)
KOMPONENTE
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755)
components/hello/component.pyWednesday, October 31, 12
![Page 28: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/28.jpg)
[hosts]localhost = hello
environments/dev.cfg
ENVIRONMENT
Wednesday, October 31, 12
![Page 29: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/29.jpg)
INSTALLATION
$ bin/batou-local dev localhostUpdating Hello > File(hello) > Presence(hello)Updating Hello > File(hello) > Content(hello)Updating Hello > File(hello) > Mode(hello)
Wednesday, October 31, 12
![Page 30: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/30.jpg)
LÄUFT?
$ ./work/hello/helloHello, world!Hello, world!^C
Wednesday, October 31, 12
![Page 31: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/31.jpg)
Wednesday, October 31, 12
![Page 32: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/32.jpg)
BUILDOUT
[buildout]parts = supervisor
[supervisor]recipe = collective.recipe.supervisor
components/supervisor/buildout.cfgWednesday, October 31, 12
![Page 33: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/33.jpg)
BUILDOUT
from batou.component import Componentfrom batou.lib.buildout import Buildout
class Supervisor(Component):
def configure(self): self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
![Page 34: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/34.jpg)
[buildout]parts = supervisor
[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = http://{{component.address.listen}}
SERVER-ADRESSE
components/supervisor/buildout.cfgWednesday, October 31, 12
![Page 35: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/35.jpg)
SERVER-ADRESSE
components/supervisor/component.py
from batou.component import Componentfrom batou.lib.buildout import Buildoutfrom batou.utils import Address
class Supervisor(Component): port = 9000 def configure(self): self.address = Address( self.host.fqdn, self.port) self += Buildout('supervisor', python='2.7')
Wednesday, October 31, 12
![Page 36: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/36.jpg)
NETZWERKADRESSEN
a = Address('localhost:631')a = Address('localhost', '631')
str(a.connect) == 'localhost:631'a.connect.host == 'localhost'a.connect.port == '631'
str(a.listen) == '127.0.0.1:631'a.listen.host == '127.0.0.1'a.listen.port == '631'
Wednesday, October 31, 12
![Page 37: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/37.jpg)
SERVER-ADRESSEfrom batou.component import Componentfrom batou.lib.buildout import Buildoutfrom batou.utils import Address
class Supervisor(Component): port = 9000 def configure(self): self.address = Address( self.host.fqdn, self.port) self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
![Page 38: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/38.jpg)
[hosts]localhost = program, supervisor
[component:supervisor]port = 6000
environments/dev.cfg
SERVER-ADRESSE
Wednesday, October 31, 12
![Page 39: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/39.jpg)
WELCHE PROGRAMME STARTEN?
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755) self.provide('program', '{}/hello'.format(self.workdir))
components/hello/component.pyWednesday, October 31, 12
![Page 40: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/40.jpg)
w
WELCHE PROGRAMME STARTEN?
class Supervisor(Component):
def configure(self): self.programs = self.require( 'program', host=self.host) self += Buildout('supervisor', python='2.7')
components/supervisor/component.pyWednesday, October 31, 12
![Page 41: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/41.jpg)
[buildout]parts = supervisor[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = ...programs ={%- for program in component.programs %} {{loop.index0}} prog{{loop.index}} {{program}} true{% endfor -%}
WELCHE PROGRAMME STARTEN?
components/supervisor/buildout.cfgWednesday, October 31, 12
![Page 42: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/42.jpg)
RESSOURCEN UND ABHÄNGIGKEITEN
•provide(key, value)
•require(key, host=None) => [x, y, z]
•require_one(key, host=None) => x
Wednesday, October 31, 12
![Page 43: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/43.jpg)
RESSOURCEN UND ABHÄNGIGKEITEN
• provide ohne require
• require ohne provide
• require_one mit != 1 Element
• Flapping durch Sortieren vermeiden.
•Dependencies werden automatisch gefunden
Wednesday, October 31, 12
![Page 44: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/44.jpg)
from batou.component import Componentfrom batou.lib.file import File
class Hello(Component):
def configure(self): self += File('hello', source='hello.py', mode=0o755) self += Program('hello', command='hello')
HOOK KOMPONENTEN
components/hello/component.pyWednesday, October 31, 12
![Page 45: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/45.jpg)
class Program(HookComponent): namevar = 'name' key = 'supervisor:Program'
command = None program_template = '{name} {command} true'
def configure(self): super(Program, self).configure() self.command = os.path.normpath( os.path.join(self.workdir, self.command))
def format(self, supervisor): return self.program_template.format( name=self.name, command=self.command)
Wednesday, October 31, 12
![Page 46: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/46.jpg)
class Program(HookComponent): key = 'supervisor:Program' ...
class Supervisor(Component):
def configure(self): self.programs = self.require( Program.key, host=self.host) self += Buildout('supervisor', python='2.7')
HOOK COMPONENTS
components/supervisor/component.py
Kann eigentlich nicht hier leben - muss importierbar sein!
Wednesday, October 31, 12
![Page 47: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/47.jpg)
[buildout]parts = supervisor[supervisor]recipe = collective.recipe.supervisorport = {{component.address.listen}}serverurl = ...programs ={%- for program in component.programs %} {{loop.index0}} {{program.format}}{% endfor -%}
WELCHE PROGRAMME STARTEN
components/supervisor/buildout.cfgWednesday, October 31, 12
![Page 48: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/48.jpg)
class Supervisor(Component): def configure(self): ... def verify(self): self.assert_file_is_current( 'var/supervisord.pid', ['.batou.buildout.success']) self.assert_no_subcomponent_changes()
def update(self): out, err = self.cmd('bin/supervisorctl pid') try: int(out) except ValueError: self.cmd('bin/supervisord') else: self.cmd('bin/supervisorctl reload')
SUPERVISOR STARTEN
Wednesday, October 31, 12
![Page 49: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/49.jpg)
@REBOOT
Nur ein "bag".class Service(Component): namevar = 'executable' pidfile = None
class Supervisor(Component): def configure(self): ... self += Service('bin/supervisord', pidfile='var/supervisord.pid')
Wednesday, October 31, 12
![Page 50: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/50.jpg)
PLATTFORMEN@platform('flyingcircus.io', Service)class UserInit(Component):
def configure(self): target = '/var/spool/init.d/{}/{}'.format( self.environment.service_user, basename(self.parent.executable)) self += File(target, source='init.sh', is_template=True, mode=0o755, leading=True)
Wednesday, October 31, 12
![Page 51: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/51.jpg)
[environment]host_domain = flyingcircus.ioplatform = flyingcircus.io
$ bin/batou-local --platform= \ production test01 # keine Plattform!
PLATTFORMEN
Wednesday, October 31, 12
![Page 52: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/52.jpg)
REVIEW• File taucht häufig auf
• Address
• Jinja2-Templates
• Environment overrides
• Ressourcen + HookComponents
• verify/update-Cycle
• Plattform-Spezifika
Wednesday, October 31, 12
![Page 53: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/53.jpg)
TIPPS
• Komponenten unabhängig von Host
• Gemeinsamkeiten in Komponenten bündeln
• verify/update so kurz wie nur möglich
• je abstrakter eine Komponente desto unwarscheinlicher trifft man verify/update
Wednesday, October 31, 12
![Page 54: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/54.jpg)
VORDEFINIERTE KOMPONENTEN
File VirtualEnv Buildout zip
Secrets Supervisor Download CMMI
git svn tar cron
Wednesday, October 31, 12
![Page 55: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/55.jpg)
FAZIT
•modell-getrieben aber pragmatisch
• Konvergenz und Idempotenz
•Der Teufel liegt im Detail. Immer wieder.
Wednesday, October 31, 12
![Page 56: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/56.jpg)
AUSSICHT
•Dokumentation
• Bessere Fehlermeldungen
• Besserer Status-Output
• Alternativen zu SSH/HG
Wednesday, October 31, 12
![Page 58: Batou - multi-(host|component|environment|version|platform) deployment](https://reader038.vdocuments.site/reader038/viewer/2022103113/553ab9825503463a458b45c3/html5/thumbnails/58.jpg)
FRAGEN?
• http://bitbucket.org/gocept/batou
• http://packages.python.org/batou
• Bitte gebt mir Bierdeckelfeedback!
• http://mobro.co/theuni
Wednesday, October 31, 12