deploying foreman in enterprise environments
DESCRIPTION
Nils Domrose, inovex GmbH Config Management Camp cfgmgmtcamp.eu 3 and 4 February 2014 Gent, BelgiumTRANSCRIPT
Deploying Foreman in Enterprise Environments
best practices
and lessons learned…
Nils Domrose
Gent, February, 3 2014
▸ senior linux systems engineer at
inovex GmbH
▸ worked as a network engineer,
software developer and systems
engineer
▸ using foreman for about 1 year
▸ using bare-metal deployment for
ages
▸ life is short – let’s focus on
interesting stuff!
2
About me
__endy__
@endyman
https://plus.google.com/+NilsDomrose
#irc
▸ http://www.inovex.de
▸ offices in [‘Pforzheim’, ‘Karlsruhe’, ‘Cologne’, ‘Munich’]
▸ we have open positions...
3
About inovex
We use technology to make our customers happy. And ourselves.
1
SHAPE
Consulting
2
BUILD
Application
Development
3
RUN
IT Engineering &
Operations
5
TEACH
Academy
4
TRACK
Business
Intelligence
4
▸ the chicken & egg problem
▸ foreman components
▸ we are the borg
▸ fail! foreman HA
▸ best practices
Agenda
▸ some shell scripts to push
installer and answers.yaml to
new system (installer is subject
to rpm/deb packaging)
▸ run a custom kafo installer
▸ setup repos
▸ configure nics (bonding)
▸ setup pacemaker &
corosync
▸ setup drbd
▸ setup dhcp server
▸ setup postgreSQL
▸ setup puppetDB
▸ run foreman kafo installer
The Chicken & Egg Problem
...deploying your deployment infrastructure
▸ kafo?
▸ kafo! - imagine master-less
puppet plus config data in yaml
(like hiera)
▸ can’t tell wether it’s the chicken
or the egg or both but it’s cool
▸ stores parameters for your
modules in answers.yaml
▸ simply copy your modules into
the installer’s modules directory
▸ or even use puppet-librarian to
manage your modules (poor
man’s git-submodules)
The Chicken & Egg Problem
build your first kafo installer
https://github.com/rodjek/librarian-puppet
https://github.com/theforeman/kafo
The Chicken & Egg Problem
kafo - getting started
i=my-installer; mkdir $i && cd $i && kafofy $i && mkdir /etc/$i && cat
> /etc/$i/$i.yaml << EOF
> ---
> :log_dir: /var/log/$i
> :log_name: configuration.log
> :log_level: info
> :no_prefix: false
> :mapping: {}
> :answer_file: /etc/$i/answers.yaml
> :installer_dir: $(pwd)
> :modules_dir: $(pwd)/modules
> :default_values_dir: /tmp
> :colors: true
> :kafo_modules_dir: /usr/lib/ruby/vendor_ruby/kafo/modules
> EOF
root@foreman:~# cd /var/tmp/
root@foreman:/var/tmp# i=my-installer; mkdir $i; cd $i; kafofy $i
▸ Never give up, never surrender!:
depending on your kafo gem it
might not (yet) be as easy as:
▸ It’s more like:
▸ delete the modules directory
The Chicken & Egg Problem
get the modules
▸ next: package it, install it
root@foreman:/var/tmp/my-installer# librarian-puppet install --clean
root@foreman:/var/tmp/my-installer# ll modules
total 16
drwxr-xr-x 4 root root 4096 Jan 29 12:25 ./
drwxr-xr-x 7 root root 4096 Jan 29 12:17 ../
drwxr-xr-x 7 root root 4096 Jan 29 12:25 ntp/
drwxr-xr-x 6 root root 4096 Jan 29 12:24 stdlib/
root@foreman:/var/tmp/my-installer# cat Puppetfile
forge "http://forge.puppetlabs.com"
mod 'puppetlabs/stdlib'
mod 'ntp',
:git => 'git://github.com/puppetlabs/puppetlabs-ntp.git'
root@foreman:/var/tmp/my-installer# rm -Rf modules/
▸ run the librarian
▸ create a Puppetfile
9
Foreman Components
foreman
foreman
frontendREST API
unattended
resources
apache mod_passenger
Port 80 HTTP Port 443 HTTPS
▸ rails application
▸ apache mod_passenger
▸ Webinterface
▸ REST API
▸ unattended resources –
rendered templates
database
▸ WEBrick based REST
server to manage
supporting infrastructure
10
Foreman Components
smart-proxy
DNSDHCP TFTP
apache
mod_passenger
Port 8140 HTTPS
Port 8443 HTTPS
WEBrick server
BMCpuppetpuppet
CA
Port 69
UDP
tftpdISC
dhcpd
Port 67
UDP
nsupdateipmitool
▸ flows at each side to local puppet
and smart proxy
▸ all VLANs directly access puppet ca
▸ all VLANs directly access foreman
unattended resources
▸ we need DNS only once (so let’s
limit TSIG key distribution)
11
Foreman Components
placement of foreman components
Foreman
Smartproxy, puppet, CA,
DNS, DHCP
Smartproxy, puppet,
DHCP
VLAN A Site1
VLAN B Site1
VLAN C Site2
VLAN D Site2
fw
fwmgmt VLAN2
mgmt VLAN1
12
Foreman
smartproxy, puppet, CA,
DNS, DHCP
smartproxy, puppet,
DHCP, Proxy
VLAN A Site1
VLAN B Site1
VLAN C Site2
VLAN D Site2
fw
fwmgmt VLAN2
mgmt VLAN1
▸ limit access to the smartproxy and
foreman to local clients only
▸ open dedicate flow from proxy to
foreman and puppet
Foreman Components
optimizing comms flows
▸ install apache mod_rewrite on
remote smartproxy
▸ rewrite/proxy unattended resources
▸ rewrite/proxy puppet CA resources
13
Foreman Components
optimizing comms flows
...
rewrite_rule => [
‘^/unattended/(.*)$ http://foreman.mysite.com/unattended/$1 [P]’,
]
rewrite_rule => [
‘^/([^/]+/certificate.*)$ https://puppetca.mysite.com:8140/$1’,
]
...
▸ we must identify proxied servers
▸ foreman url must be customized
▸ templates must be adapted
14
Foreman Components
foreman customization
▸ create parameter in hostgroups for each
segment (unattended_url in 1.4 does not
seam to help here…)
▸ adapt templates
▸ enable tokens (default in 1.4)
Integrating foreman into corporate
infrastructure is:
▸ not a technical issue
▸ mostly even not a security issue
▸ It’s about convincing people to do
things differently
▸ It’s about responsibilities
▸ It’s about fear
▸ It’s about laziness
15
Resistance is futile
Integrating foreman into corporate infrastructure
16
Resistance is futile
The one with the wrong <enter vendor name> DHCP product
The corporate DHCP does not provide a proper API
and changes are distributed across the DHCP
infrastructure in a slow, asynchronous manner…
Deploy your own DHCP server:
- You must be able to create and delete leases
instantly
- You only need it to bootstrap your servers. Later a
static NIC configuration can be applied using your
favorite cfgmgmt. No, it’s not that static – we can
use ENC variables to update IP addresses.
- Repeat: I won’t create my leases manually and I
won’t wait for changes to be replicated until I
power-on my server
17
Resistance is futile
The one with the DHCP caused Zombiecalypse
You are about to deploy your own DHCP but people
complain that your DHCP will nuke the corporate
datacenter and will cause an epic loss of revenue…
Ask for dedicated VLANs and for ip-helper configs
on routers and firewalls
- Your DHCP is limited to your VLANs only
- You only need one DHCP server
- It’s not an evil plan to make the guy running the
corporate DHCP redundant
18
Resistance is futile
The one with static DNS
You kindly ask for TSIG keys to create your A and
PTR records automatically but the DNS is managed
manually and there is no TSIG key although dynamic
updates are enabled…
Make a deal: you won’t talk about the missing TSIG
keys and get your key or keytab and permissions
right after the guy fixed the security issue.
Ask for dedicated zones and for credentials which
allow you to manage your zones using nsupdate
- Alternatively ask for delegation of your zones and
setup a dedicated DNS server (protected by keys)
19
FAIL!
Foreman HA
19
pacemaker pacemaker
DRBD
postgreSQL
Apache
mod_passenger
corosync
puppetDB
HA IP(s)
DRBD
postgreSQL
Apache
mod_passenger
puppetDB
HA IP(s)
node1 node2
▸ use pacemaker/corosync to provide
basic resilience for postgreSQL and
puppetDB, puppetCA
▸ If you scale out, you might setup a
nice pg-pool2 cluster
▸ puppetmasters could be ran active
active as well as the foreman rails
application (keep memcached in
mind)
▸ on pure smartproxies no pacemaker
cluster is required. A simple failover
using VRRP (via keepalived) should
be sufficient – if you replicate the
tftp and dhcp data (for puppet you’ll
have to do the exercise anyhow.)
▸ use separate IP addresses for each service so you
can split up stuff later on
▸ If you can’t use DHCP the bootdisk plugin might be
an option (currently .deb package 1.2.3 is broken
for 1.4 but version plugin source is ready for 1.4)
▸ LDAP or kerberos5 (1.4) for user authentication
▸ use Locations for filtering
▸ classes are not tight to locations so you better use
the same set of classes and import only once from
you master puppet master
▸ use the column plugin for better overview
▸ ENC and smart variables are nice – ever configured
dhcp pool as yaml hash? Use smart variables as
switches, and for simple datatypes only - do the
complex stuff in service modules - ideally by using
data in modules
20
Best practices
▸ integrate the reverse proxy feature into
smart-proxy (there is a ticket for that
somewhere)
▸ implement iso image provisioning based
vm installation in conjunction with the
bootdisk plugin
▸ support more configuration management
stacks
▸ better support for discovery, ipmi and
hardware provisioning (like HW-Raid)
21
Wishlist
...secret guide to world domination
22
Thank You!
Contact
Nils Domrose
Senior Systems Engineer
inovex GmbH
Office cologne
Schanzenstr. 6-20
51063 Köln