puppet camp amsterdam
TRANSCRIPT
Story of Puppet @eBay Classifieds Group
or how we came to the setup we use
Some bio or “Who is Vladimir”
Sysadmin for eighteen years
Currently in Solutions team of eCG’s marktplaats.nl
All-round nerd
Hobbyist programmer
Puppet at eBay Classifieds:
9 countries
13 platforms
1 codebase
1 change management process
Puppet at eBay Classifieds:
We started adopting puppet in 2008
Since then we had about 4 different iterations of “this is bloated and
doesn’t work, let’s think more and re-do”
Currently we believe we achieved a workable scenario where we:
Enable teams to share and re-use puppet code
Enforce the same code style and standards
Allow for peer-reviews of the code, also enforce them
“If it’s not in Puppet – it does not exist.”
What we are going to talk about:
• Different iterations of puppet implementation at eCG
• Go in-depth on the structure of the current one
• Process around the codebase
• Tools
“Bucket of recipes”
Puppet at eBay Classifieds (1st implementation):
A bunch of modules without visible structure (server-role based)
Relations between modules on any levels (includes, instantiations)
One Subversion repository, where everyone works on trunk
What were the lessons learned?
As puppet adoption grew, modules for different platforms started
clashing (same, yet a bit different functionality)
As number of modules and their dependencies grew people were
afraid of making changes - dependency tree was very hard to follow
Resulted in tons of duplicate code
Change management was very risky – you commit something and
it’s live
“Buckets of recipes”
Puppet at eBay Classifieds (2nd implementation):
What did we improve?
Release management with Subversion branching
Better module grouping
What were the lessons learned?
Waiting on releases is killing. If you need a change - you need it
now, not in 2 or 3 days
Provokes people to lock machines to their own puppet environment
Surprises at times of the releases. For some people it was daytime,
for others it was middle of the night
Puppet at eBay Classifieds (3rd implementation):
What did we improve?
Switched to Git
Completely redesigned modules structure (more on this later)
Introduced extlookup for data separation vs configuration logic
Release more often
What were the lessons learned?
When release process is human-dependent it will be a bottleneck
Extlookup is nice, but in many places unreadable
Puppet at eBay Classifieds (4th implementation):
What did we improve?
Introduced Gerrit for code reviews
Implemented automatic puppet-lint, template syntax and PEP8 checkers
Introduced automated quorum votes for peer reviews
Release as soon as your code passed checks and YOU need it live
Further improved module tree structure
Replaced extlookup with hiera
Introduced puppetdb
So how does it look now?
Shared
eCG
Platform
Config
Four module levels
One include direction
No includes on the same level
Shared modules
Shared
eCG
Platform
Config
Shared modules are very basic, deploy one
piece of software in its standard configuration (no
customization)
Standalone. Includes no other modules
Installs needed packages, basically
Parameterized to allow configuration override
Examples: apache, ntpd, activemq, mysql, postgres
eCG modules
Shared
eCG
Platform
Config
eCG modules add our specific eCG-wide
approved customizations. Rule: it must play well
with majority of teams to make it
Mostly include shared modules
Never include each other
Override default shared module configuration
with eCG specific one via parameters
May be parameterized to allow for configuration
tweaks or overrides
Examples: puppetmaster, nagios checks, collectd
plugins
Platform modules
Shared
eCG
Platform
Config
Platform level introduces company-specific
modules and configuration customizations
May include eCG modules
May include shared modules
Never include each other
Config modules
Shared
eCG
Platform
Config
Config level introduces server-role specific
modules
Are the starting point for our servers
configuration
May include platform, eCG or shared modules
Never include each other
Node definition must include config module
Why did we do it like this?
Shared
eCG
Platform
Config
Easy starting point. Look up a node, see its
class, open the class – you see on one page
how your server is configured on a global level
If you need to know more, dive deeper
We enable code sharing, we do not enforce it
If you see a module in shared you can use – use
it. If you can’t use it, extend it in a backwards-
compatible way or write other extension on an
appropriate level
Hiera
Great way to separate code from data
Company-based structure
Lookup path based on facts to allow company data separation
while keeping the same source tree
:datadir: /var/lib/hiera/%{environment}/%{companyprefix}
Inside company dirs a variety of fact-based lookups to allow for
fine-grained data structures
PuppetDB
Much faster catalog compilations for huge catalogs (nagios
server with 200k resources)
REST API to talk to for inventory and searches
Many brilliant ways to (ab)use it
Stores all facts (can be used to feed CMDBs)
Stores most recent catalog (can be used for inspections)
POC we are running - internal Puppet naginator module is very slow,
collection of exported resources takes about 15 minutes. We use
PuppetDB to generate the same nagios configs in just under 8
seconds
“If there is data, it can be used”. PuppetDB is a central data store.
Love.
So all good, but how does this work in practice?
Per-user environments on puppetmasters
Make the change in your own env
Test it from your own env
Push to Gerrit
Ask team members to review
Merge
Code reviewPush to GerritTest the change
with --environmentLocal
environmentMerge
Gerrit
Great code review tool (https://code.google.com/p/gerrit/)
Acts as a Git server with fine-grained access control
Each commit is a separate change, that requires votes for
approvals
Votes can be +1 for Code Review, +2 for change approval
No +2 – no merge
We use voting system with auto +2
Depending on platform size or number of platforms affected by the
change more votes are needed to get a change through
Each change can be “pushed through” in case of an emergency
More Gerrit goodies
Allows for post-push hooks
Puppet-lint
Ruby templates parsing
PEP8 (we love Python)
Each change is separately checked out by hooks and tested
for syntax or other obvious errors
Concept of “one commit – one change” forces people to create
atomic changes that are easy to revert in case of errors
Keeping it all in the same system, even if you decide to split Git
repositories still makes work done by colleagues VISIBLE
Gerrit and puppetmasters
Master is automatically pulled by crons on puppetmasters
Gerrit disallows pushing directly to master
Even if you force-merge a change, it makes your change
VISIBLE and easily REVERTIBLE
Takes about half a minute extra time
Ensures you didn’t slack out on syntax checkers
To sum it all up
There is no single structure that fits every company
Puppet code is like any other code
Write
Use
Refactor if needed
Layered modules structure can work well in many cases
Complementary tooling around Puppet is a life- (and time-)
saver
Lint(s), Gerrit, PuppetDB
Work to be done in our house
Approval process improvements
Better quorum schemas
Easier ways of code style and syntax checking
See that people don’t leave changes “hanging”
Integrate Gerrit with corporate GitHub
Work more with community
Open source some generic shared modules
Offer changes to PuppetForge modules
POC with Vagrant for puppet code testing
Questions?
Thank you!
You can reach me by:
• Mailing to [email protected]
• Following me on GitHub
(https://github.com/favoretti)
Special thanks to:
• PuppetLabs and eCG for the opportunity to speak here
• Teams of eCG for bearing with me all these years
• You all for attending and listening
We are hiring!
Visit http://ebayclassifiedsgroup.com/careers
for more info.