Keeping AWS in check with Puppetor

“Managing cloud networking with Puppet with a healthy dose of trolling yourself on the side”

Matt Carroll

HELLO.

What’s all this about then?

● We use AWS extensively, especially EC2● We needed a way to organise our AWS networking (VPCs)● It needs to be:

○ Centrally managed○ Reproducible○ Declarative (idempotent)○ Ideally not another solution on top of a stack of solutions

We (Tom) decided to use Puppet

We wrote a set of types and providers to use with the AWS API

Spoiler

What I’m going to cover

1. Intro2. AWS networking basics3. Why we chose Puppet4. How it actually works5. Some of the “interesting” things we learned6. Summary7. Questions

YOU ARE HERE

What I hope you’ll get from this

● A sense of Puppet as an extensible framework that can manage dependencies in external APIs

● Nodes are just units of computing capacity - an operation does not have to be a subset of a node

● Some insight into the types and providers system● Some of the strange things we learned along the way

… and as always...

To serve as a lesson to others.

SOME AWS BASICS.

Some of this

isn’t exactly

retina resolution

The (simplified) Hierarchy

Account

Region

VPC

Routetable

Subnet

The (needlessly complicated simplified) Hierarchy

Account:• Region

• dopt: DHCP Options• vpc: Virtual Private Cloud

• igw: Internet Gateway• cgw: Customer Gateway• vgw: Virtual Gateway

• vpn: VPN• routetable

• subnet

How do we make all these objects?

Just make them in the console!

How do we make all these objects?

Just make them in the console!

IF YOU HATE YOURSELF

How do we actually make all of these objects?

• AWS::SDK!

• We’ll have to ensure that the resources get created (and

purged)...

• Idempotently…

• With all their dependencies…

• Remind you of something?

USE PUPPET!

Create Puppet types for VPC objects!● Resources for each of the objects

○ All API calls made on this level○ Will contain all code for reading and creating individual

resources○ No dependencies other than autorequires in the types -

those in the hierarchy earlier● Business logic in manifests

○ For your site-specific dependencies and network structure○ e.g. we have a separate VPC for each environment

● Data in hiera

(Actually though) Why Puppet?

• With the ability to query and create and modify objects through the API we can state them declaratively

• We can thus create resources which can be included or purged idempotently

• Rather than specify order, we can state dependencies and allow Puppet to figure order out

Why Puppet?Using this model you can even collect dependencies and resources:• Make all resources of a type with a parameter evaluate after

another

• Control purge and no-op of all aws resources

Why Puppet?Admittedly it seems a little asymmetrical:• Puppet runs on nodes which creates

resources in AWS• AWS networking is not a subset of a node• Nodes just act as executors for creating AWS

resource

But actually this plays to our advantage (more on this later)

HOW IT WORKS.

• The bit that you specify in the manifest

• Really just a DDL for the resource’s metadata

• The “front end” for the pluggable “back end” (the provider)

• Interface to specify all properties and parameters

Puppet Types

• The business end• This is where the API

code lives• Quite a lot longer• Handles all application

and querying of the resource including prefetch

Puppet Providers

• Some resources are expensive to read, so you read them all once when you first come across one

• This is done before the catalog is completely compiled

What’s a “prefetch”?

Manifest examples

Associated Infrastructure

• How do you handle multiple accounts?• Dedicated AWS admin box within each

account to apply the resources on• IAM roles to handle credentials• Logging resource changes separately• Why not do it in the code...?

LESSONS LEARNED.(the hard way)

Don’tdo

whatDonny

Don’tdoes

Why not put credentials in the code?

Why not put credentials in the code?• We (I) tried to make an

aws_credentials type• This requires access to the

catalog in the prefetch phase so other resources can query it

• You ALSO need to guarantee type evaluation order and access to credentials in the catalog so that you can use prefetch

Why not put credentials in the code?

• The instances method suddenly requires an argument

• In fact, so does anything that isn’t an instance method

• Actually ended up copying and pasting the resources resource to aws_resources and adding in a “credentials” parameter

On second thoughts...

Prefetch isn’t… Exceptional

• https://tickets.puppetlabs.com/browse/PUP-3656• This means that if anything goes awry in prefetch, puppet will

swallow it• In our case, we hit the API limit occasionally, meaning we got

duplicate resources

So we did a bad thing...

Asynchronous APIs tell lies• When you create an object you get

a 200 OK• This doesn’t mean “I’ve done it”,

this means “I’ll do it”• In a dependency chain, this can

mean that a resource is about to be created, but when it’s checked by another resource it’s not there yet

• In the Puppet paradigm it’s best just to run until convergence.

SO IN SUMMARY...

What I learned (the hard way)

• The “resources” resource• Is hard to understand unless

you read the code (type only) because Googling it is impossible.

• Could do with being able to apply other arbitrary properties and parameters?*

*this may be an awful idea

What I learned (the hard way)

• Instances vs. class methods paradigm gets very confusing because it is tied to applying catalogue vs. prefetch

• Prefetch is generally pretty confusing when you add in dependencies TO the prefetch

What really cool things I learned• You can reproduce AWS networking (and other objects

thanks to other contributors!)• A really cool insight into the types and providers system and

how it could grow in the future• Learning to treat servers not like pets OR cattle but as a

medium by which you do useful stuff

… and you could do this with a whole bunch of APIs

github.com/bobtfish/puppet-aws_apiMatt CarrollSRE at Yelp

@oholiabmattc@yelp.com

yelp.com/careers (we’re hiring)

THANKS FOR LISTENING!