puppetdb: new adventures in higher-order automation - puppetconf 2013

The life and times of

PuppetDB

Friday, August 23, 13

DEEPAK [email protected]@grim_radical


mailto:[email protected]


We need to talk!


Puppet agent

Puppet master


Puppet agent

Puppet master

facts


Puppet agent

Puppet master

facts

netmask_lo: 255.0.0.0 augeasversion: 0.10.0 fqdn: pe-debian6.localdomain manufacturer: "VMware, Inc." processorcount: "1" productname: VMware Virtual Platform physicalprocessorcount: 1 facterversion: 1.6.7 boardproductname: 440BX Desktop Reference Platform kernelmajversion: "2.6" hardwareisa: unknown timezone: PDT puppetversion: 2.7.12 (Puppet Enterprise 2.5.1) lsbdistcodename: squeeze is_virtual: "true" operatingsystemrelease: 6.0.2 virtual: vmware type: Other domain: localdomain hostname: pe-debian6 selinux: "false" kernel: Linux

kernelrelease: 2.6.32-5-686 ipaddress: 172.16.245.128 processor0: Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz lsbdistrelease: 6.0.2 uniqueid: 007f0101 hardwaremodel: i686 kernelversion: 2.6.32 operatingsystem: Debian architecture: i386 lsbdistdescription: Debian GNU/Linux 6.0.2 (squeeze) lsbmajdistrelease: "6" interfaces: "eth0,lo" ipaddress_lo: 127.0.0.1 uptime_days: 0 lsbdistid: Debian rubysitedir: /opt/puppet/lib/site_ruby/1.8 rubyversion: 1.8.7 osfamily: Debian memorytotal: &id001 502.57 MB memorysize: *id001 boardmanufacturer: Intel Corporation path: /usr/local/sbin:/usr/local/bin:/


Puppet agent

Puppet master

facts


Puppet agent

Puppet master

catalog


file {“/tmp/foo”: content => “This is a test”}


target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: File

file {“/tmp/foo”: content => “This is a test”}


Relationships

Exec[broker_cert_bundle]

File[/etc/puppetlabs/activemq/broker.pem]

Exec[broker_cert_pkcs12]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]

Service[mcollective]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]

File[/opt/puppet/share/puppet-dashboard/.bashrc]

Service[pe-activemq]

File[/etc/puppetlabs/mcollective/ssl]

File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]

File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]

File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]

Class[Pe_accounts::Data]

Anchor[pe_compliance::end]

File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]

File[/etc/puppetlabs/activemq/broker.ts]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]

Class[Settings] Class[Main]

Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.ssh]

Schedule[daily]

File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]

File[/var/lib/peadmin/.vim]

File[/etc/puppetlabs/mcollective/server.cfg]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]

Anchor[pe_accounts::begin]

Class[Pe_accounts::Groups]

Anchor[pe_accounts::end]

Filebucket[main]

File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]

File[/etc/puppetlabs/activemq/broker.ks]

Cron[pe-mcollective-metadata]

Class[Pe_mcollective]

Class[Pe_mcollective::Plugins]

Anchor[pe_mcollective::end]

File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]

File[/etc/puppetlabs/activemq/activemq.xml]

File[/etc/puppetlabs/mcollective/client.cfg]

Exec[mcollective-client-cert]

File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]

File[/opt/puppet/libexec/mcollective/mcollective/agent]

File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]

File[/etc/puppetlabs/activemq/activemq-wrapper.conf]

Schedule[never] Stage[main]Anchor[pe_mcollective::begin]

Class[Pe_mcollective::Posix]

Class[Pe_mcollective::Metadata]

File[/opt/puppet/libexec/mcollective/mcollective/util]

File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]

Pe_accounts::Home_dir[/var/lib/peadmin]

Exec[broker_cert_keystore]

Group[puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]

User[puppet-dashboard]

Schedule[weekly]

Exec[mcollective-server-cert] File[/var/lib/peadmin]

File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]

File[/var/lib/peadmin/.ssh/authorized_keys]

Class[Pe_accounts]

Exec[broker_cert_truststore]

Schedule[hourly]

Class[Pe_compliance::Agent]

Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]

Schedule[monthly] Filebucket[puppet]

Pe_accounts::User[peadmin]

File[/etc/puppetlabs/activemq/broker.p12]

Node[default]

Pe_accounts::User[puppet-dashboard]

Class[Pe_compliance]

File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]

File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]

File[/opt/puppet/libexec/mcollective/mcollective/security]

Group[peadmin]

User[peadmin]

File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]

File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]


Puppet agent

Puppet master

catalog


Puppet agent

Puppet master

report


Puppet agent

Puppet master

report

"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/foo


Puppet agent

Puppet master

report


Puppet agent

Puppet master


Puppet agent

Puppet master PuppetDB


Puppet agent


facts


Puppet agent


catalog

facts

catalog


Puppet agent


catalog

catalog

facts


Puppet agent


catalog facts


Puppet agent


report

catalog facts


ActiveRecord

Puppet master

catalog


ActiveRecord

Puppet master

catalogcatalogcatalogcatalogcatalogcatalog


ActiveRecord

Puppet master

catalogcatalogcatalogcatalogcatalog catalog


ActiveRecord

Puppet master

catalogcatalogcatalogcatalog catalogcatalog


ActiveRecord

Puppet master

catalogcatalogcatalog catalogcatalogcatalog


ActiveRecord

Puppet master

catalogcatalog catalogcatalogcatalogcatalog


ActiveRecord

Puppet master

catalog catalogcatalogcatalogcatalogcatalog


ActiveRecord

Puppet master

catalog catalog


Puppet master

catalog


Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent

Puppet agent


ActiveRecord


ActiveRecord

Which boxes arerunning nginx?


ActiveRecord


ActiveRecord

How many serversare running a

vulnerable versionof rails?


ActiveRecord


ActiveRecord

What are the IPaddresses of my

webservers?


ActiveRecord


ActiveRecord

Which users have sudo access?


ActiveRecord


ActiveRecord

LOLWUT


ActiveRecord

LOLWUT

ಠ ಠ_


And now for something completely

different


PuppetDB


/resources/Service/nginx

PuppetDB


resources

/resources/Service/nginx

PuppetDB

)

O O


PuppetDB


/resources/Package/rails

PuppetDB


resources

/resources/Package/rails

PuppetDB

)

O O


PuppetDB


/nodes/foo.com/resources/User/

deepak

PuppetDB


resources

/nodes/foo.com/resources/User/

deepak

PuppetDB

)

O O


(demo)


We built something quite different


1. Asynchrony


Storage &Querying


CQRS


CommandQueryResponsibilitySeparation

use a different model to update information than the model you

use to read information


CQRSwrite pipeline

async, parallel, MQ-based, with automatic retry


{ :command "replace catalog" :version 2 :payload {...}}


/commands MQ Parse

Delayed

Dead Letter Office

Process

UUID


Command processors must be retry-aware

expect failure, because it *will* happen.


Failures like, oh I don't know,

a database crash?


2. New runtime


Fast,Free,Portable,Multi-core,Popular,

The JVM is all these thingsFriday, August 23, 13

Haters gonna hate!


Tons and tons of high quality libraries

Web servers, concurrency frameworks, databases, fast

parsing/lexing, clustering, debugging, profiling, etc.


Can ship an uberjar, makes deployment straightforward with few moving pieces


And it's fast.


Nobody cares what runtime we use. Users just want stuff to work.


3. AST querying


Queriesare expressed in their own “language”

domain specific, AST-based query language


["and", ["=", "type", "User"], ["=", "title", "deepak"]]


["and", ["=", ["fact", "operatingsystem"], "Debian"], ["<", ["fact", "uptime_seconds"], 10000]]


["and", ["=", "name", "ipaddress"], ["in", "certname", ["extract", "certname", ["select-resources", ["and", ["=", "type", "Class"], ["=", "title", "Apache"]]]]


["or", ["=", "certname", "foo.com"], ["=", "certname", "bar.com"], ["=", "certname", "baz.com"]]


We walk the tree, compiling it to efficient SQL


Haters gonna hate!


AST-based API lets users write their own languages

ah, you’ve got to love open source!


(Package[httpd] and country=fr)or country=us

Package["mysql-server"]and architecture=amd64

Erik Dalén, Spotifyhttps://github.com/dalen/puppet-puppetdbquery


https://github.com/dalen/puppet-puppetdbquery


AST-based API lets us more safely manipulate queries


daenny, Puppetboardhttps://github.com/nedap/puppetboard


https://github.com/nedap/puppetboard


Puppet Enterprise, Event Inspectorhttps://puppetlabs.com




Foreman Integration (CERN)https://github.com/cernops/puppetdb_foreman

Web UIhttps://github.com/dima-exe/puppetdb-db

Web UIhttps://github.com/gbougeard/puppetdb-frontend


https://github.com/cernops/puppetdb_foreman

https://github.com/cernops/puppetdb_foreman

https://github.com/dima-exe/puppetdb-db




Rubyhttps://github.com/dalen/puppet-puppetdbquery

Ruby (DataMapper)https://github.com/dalen/dm-puppetdb-adapter

Rubyhttps://github.com/ripienaar/ruby-puppetdb






https://github.com/dalen/dm-puppetdb-adapter

https://github.com/dalen/dm-puppetdb-adapter

https://github.com/ripienaar/ruby-puppetdb

https://github.com/ripienaar/ruby-puppetdb

Pythonhttps://github.com/nedap/pypuppetdb

Pythonhttps://github.com/arcus-io/puppetdb-python

Pythonhttps://github.com/JHaals/puppetdb-grep


https://github.com/nedap/pypuppetdb

https://github.com/nedap/pypuppetdb

https://github.com/arcus-io/puppetdb-python

https://github.com/arcus-io/puppetdb-python

https://github.com/JHaals/puppetdb-grep

https://github.com/JHaals/puppetdb-grep

Javahttps://github.com/thallgren/puppetdb-javaclient

Gohttps://github.com/nightlyone/puppetquery

Scalahttps://github.com/gbougeard/puppetdb-frontend

CoffeeScripthttps://gist.github.com/pmuellr/5591686

Node.jshttps://github.com/nightfly19/minidb


https://github.com/thallgren/puppetdb-javaclient

https://github.com/thallgren/puppetdb-javaclient

https://github.com/nightlyone/puppetquery

https://github.com/nightlyone/puppetquery

https://github.com/gbougeard/puppetdb-frontend

https://github.com/gbougeard/puppetdb-frontend

https://gist.github.com/pmuellr/5591686

https://gist.github.com/pmuellr/5591686

https://github.com/nightfly19/minidb

https://github.com/nightfly19/minidb

MCollectivehttps://github.com/ploubser/mcollective-puppetdb-discovery

Rundeckhttps://github.com/sirhopcount/puppetdb-rundeck

Rundeckhttps://github.com/martin2110/puppetdb-rundeck


https://github.com/ploubser/mcollective-puppetdb-discovery




https://github.com/sirhopcount/puppetdb-rundeck

https://github.com/sirhopcount/puppetdb-rundeck

https://github.com/martin2110/puppetdb-rundeck

https://github.com/martin2110/puppetdb-rundeck

OpenStackhttps://github.com/bodepd/puppet-openstack_puppetdb

Vagranthttps://github.com/grahamgilbert/vagrant-puppetmaster

PowerDNShttps://github.com/evenup/evenup-pdns


https://github.com/bodepd/puppet-openstack_puppetdb




https://github.com/grahamgilbert/vagrant-puppetmaster




https://github.com/evenup/evenup-pdns

https://github.com/evenup/evenup-pdns

4. Boring technology


Relational Database, embedded or PostgreSQL

because they’re actually pretty fantastic at ad-hoc queries,

aggregation, windowing, etc. while maintaining safety


Relational Database, embedded or PostgreSQL

we use arrays, recursive queries, indexing inside complex

structures


5. Weird alien technology


-- Jeff Gagliardi


Thousands of deployments,Hundreds of threads per install,Zero deadlocks,Zero bugs involving mutable state

companion Ruby code has ~10x the defect rate


All with a pretty tiny codebase


6. Conjecturesabout performance


Posit:A resource often

exists across multiple hosts


Feature:Single-instance resource storage


Posit:We’ll often receive the

same catalog for a host


Feature:Single-instance catalog storage


In the field, we almost always see Resource and catalog duplication rates of over 85%.


Monitoring and instrumentation is a big deal. Users want easy ways to consume metrics and analyze performance.


Nagioshttps://github.com/jasonhancock/nagios-puppetdb

Nagioshttps://github.com/favoretti/puppetdb-external-naginator

Muninhttps://github.com/vpetersson/munin_puppetdb

Muninhttps://github.com/dalen/puppetdb-muninplugins

Collectdhttps://gist.github.com/mfournier/5615125


https://github.com/jasonhancock/nagios-puppetdb

https://github.com/jasonhancock/nagios-puppetdb

https://github.com/favoretti/puppetdb-external-naginator

https://github.com/favoretti/puppetdb-external-naginator

https://github.com/vpetersson/munin_puppetdb

https://github.com/vpetersson/munin_puppetdb

https://github.com/dalen/puppetdb-muninplugins




https://gist.github.com/mfournier/5615125

https://gist.github.com/mfournier/5615125

Turns out, people appreciate these

efforts


(how many?)


Thousands of production deployments

Small shops with a dozen hosts,large shops with thousands of hosts, standalone, clustered...


There is a new deployment of PuppetDB every15 minutes.


So...long time since we last spoke


Availability


Available in PE3

On by default, fully supported, and the basis for upcoming reporting and analytics features.


Performance


20% faster storage

Improvements to memoization and caching, eliminate double-serialization, nuked superfluous indexes


Much faster terminus

Better caching and data structures. For a catalog with 10k resources, drops serialization time from ~80s to ~6s.


Resiliance


Death to keystores

Can now use PEM certificates directly, eliminating one of the largest sources of configuration problems.


Configurable HTTPS

Can customize the set of cipher suites and SSL protocols you'd like to use, to match your security needs.


Automatic:

- Recovery from MQ corruption- Compression of the DLO- Purging of inactive node data- DB connection recycling


Backup and restore

Now integrated into the daemon, can restore while PuppetDB is running.


Query changes


V2 API

- No need to ask for only active nodes- Full fact queries (instead of just a list of facts for a node)- Node metadata


Wildcard Accept Headers

curl localhost:8080/v2/nodes


Subqueries

You can now correlate data from resource queries with fact queries with node queries.

"Give me the IP address of all machines with the Nginx service configured"


Report storage

- Comes with a report processing plugin- Store report-level metadata- Can do queries on events that span reports- Basis for PE's Event Inspector


Streamingqueries!


Streaming queries

Stream results to clients on-the-fly, as they come in from the database.

Massively lower latency for first response!


resourceresourceresourceresourceresourceresourceresource

PuppetDB



/v2/resources

PuppetDB



/v2/resources

PuppetDB

)

O O


(demo)


Coming up!


We will be developing tools to replicate data from one PuppetDB daemon to another. This will help with HA and DR.

PuppetDBDiff &

Mirror PuppetDB


By initially developing an out-of-band mirroring tool, we can create more interesting replication topologies:

PuppetDBDiff &

Mirror PuppetDB

Diff & Mirror


We can also later optimize the process to lower latency, but preserve eventual consistency:

PuppetDB

Diff & Mirror

PuppetDBDirect MQ connection


More flexible routing is coming, allowing for soft failures and read/write splits:

PuppetDB

Puppetmaster

PuppetDB

Replication

Catalogs, Facts,Reports

Collectionqueries

Log error andcontinue


So anyways,


Documented athttp://docs.puppetlabs.com/puppetdb

install, config, upkeep, specs,the works!


http://github.com/puppetlabs/puppetdb






Packagedas deb and rpm for open source, part of Puppet Enterprise

available in the Puppet Labspackage repositories


Puppetizedusing the puppetlabs/puppetdb module

available now, on theModule Forge!


Open source


same license as Puppet itself!






[email protected]@grim_radical [github twitter freenode]




puppetdb: new adventures in higher-order automation - puppetconf 2013

Technology

puppet enterprise

custom filevarlibpeadmin

mcollective classpe

false file

keys pe

pedebian6 selinux

file node

debian gnulinux