puppet and aws: getting the best of both worlds

Puppet and AWSGetting The Best of Both Worlds

Mike Ryan - Epitech.nl - [email protected]

Sunday, April 14, 13

mailto:[email protected]


• What is AWS?

• How does Puppet work in AWS?

• CloudFormation

• PaaS replacement?

• Vagrant -> EC2

• Using Puppet to build AMIs

• Some workarounds to common problems

• Questions


Hello, I’m Mike

• Londoner in Amsterdam

• Sysadmin with a passion for automation

• Epitech.nl - sysadmin as a service


Amazon Web Services

• EC2

• Amazon Machine Images

• User Data


EC2 - Auto Scaling


{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "MyEC2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "ImageId" : "ami-79fd7eee", "KeyName" : "my-ssh-key", } } }}

CloudFormation


{ "AWSTemplateFormatVersion": "2010-09-09", "Resources": { "elbMyLB": { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "AvailabilityZones": [ "us-east-1b", "us-east-1c" ], "HealthCheck": { "HealthyThreshold": "2", "Interval": "30", "Target": "HTTP:80/", "Timeout": "5", "UnhealthyThreshold": "2" }, "Listeners": [ { "InstancePort": "80", "LoadBalancerPort": "80", "Protocol": "HTTP", "PolicyNames": [

] } ] } }, "distd18k4jybr69gw2cloudfrontnet": { "Type": "AWS::CloudFront::Distribution", "Properties" : { "DistributionConfig" : { "S3Origin" : { "DNSName": "webapplication.s3.amazonaws.com" }, "Enabled" : "true", "Logging" : { "Bucket" : "webapplication.s3.amazonaws.com", "Prefix" : "webapp-logging/" } } } },

"asgMyAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": [ "us-east-1b", "us-east-1c" ], "Cooldown": "300", "DesiredCapacity": "1", "MaxSize": "1", "MinSize": "1", "LaunchConfigurationName": { "Ref": "lcMyLC" }, "LoadBalancerNames": [ { "Ref": "elbMyLB" } ] } }, "s3webapplication": { "Type": "AWS::S3::Bucket" }, "sgwebappsecuritygroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "GroupDescription": "for web app", "SecurityGroupIngress": [ { "IpProtocol": "tcp", "FromPort": "80", "ToPort": "80", "SourceSecurityGroupName": "amazon-elb-sg", "SourceSecurityGroupOwnerId": "amazon-elb" }, { "IpProtocol": "tcp", "FromPort": "3389", "ToPort": "3389", "CidrIp": "0.0.0.0/0" } ] } }, "dbsgmydbsecuritygroup": { "Type": "AWS::RDS::DBSecurityGroup", "Properties": { "GroupDescription": "security group for my web app", "DBSecurityGroupIngress": [ { "EC2SecurityGroupName": { "Ref": "sgwebappsecuritygroup" }, "EC2SecurityGroupOwnerId": "123456789012" } ] } } }, "Description": ""}


Bootstrapping an Infrastructure

www.infrastructures.org


http://www.infrastructures.org/papers/bootstrap/bootstrap.html

http://www.infrastructures.org/papers/bootstrap/bootstrap.html

CloudFormation or Puppet


"Metadata": { "AWS::CloudFormation::Init": { "config": { "sources" : { "/etc/puppet" : "http://example.com/puppet.tar.gz" }, "packages": { "yum": { "puppet-server": [], }, }, "services": { "sysvinit": { "puppetmaster": { "ensureRunning": "true", "enabled": "true" } }

Files, Services and Packages with cfn-init


https://github.com/mikery/cfn-demo/tarball/master

https://github.com/mikery/cfn-demo/tarball/master

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 instance", "Resources" : { "PuppetMasterInstance": { "Type": "AWS::EC2::Instance", "Metadata": { }, "Properties": { "UserData": { "Fn::Base64": { "Fn::Join": [ "", [ "#!/bin/bash\n", "/opt/aws/bin/cfn-init --region ", "\n", "/usr/bin/puppet apply site.pp, "\n" ] ] } } }

Run a script at launch time with User Data


$ cfn-create-stack puppetmaster \--template-file=puppetmaster.json

cfn-create-stack

http://aws.amazon.com/developertools/2555753788650372

http://aws.amazon.com/cli/








CloudFormation +Puppet +

Deploy scripts =PaaS


Autoscale = autosign


The Hostname Issuemike@ip-10-32-34-116:~$ hostname -fip-10-32-34-116.eu-west-1.compute.internal


User Data - Web Console


},"UserData": { "Fn::Base64": { "Fn::Join": [ "", [ “{\”role\”: \”web\”, ”, “\”env\”: \”staging\”} ” ] ] }}

User Data in CloudFormation

{“role”: “web”, “env”: “staging”}Sunday, April 14, 13

include stdlibnode default { $userdata = parsejson($ec2_userdata) $role = userdata[‘role’] $environment = userdata[‘env’]

case $role { ‘web’: { include nginx } ‘db’: { include postgresql } }}


Vagrant -> EC2


Vagrant -> EC2 Workflow


Vagrant::Config.run do |config| config.vm.provision :puppet do |puppet| puppet.manifests_path = "../puppet/manifests" puppet.module_path = "../puppet/modules" puppet.manifest_file = "site.pp" puppet.options = "--verbose --debug" puppet.facter = { :ec2_userdata => { :role => "database", :env => "local_dev", }.to_json, :vagrant => "true" } endend

User Data and Vagrant


Decoupling


Broken Puppet master = no autoscaling


#!/bin/bash

/usr/local/bin/update.sh/usr/bin/puppet apply site.pp

/etc/rc.local


But you lose...• Puppet Dashboard

• Exported resources

• Stored configs

• Shared modules


Role-based Puppet runs


EC2 Tags


env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'db': ['db1', 'db2']}

@roles('db')def restart_db(): sudo('/etc/init.d/postgresql restart')

@roles(‘web’)def restart_web(): sudo('/etc/init.d/nginx restart')

$fab restart_web restart_db


env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'db': ['db1', 'db1']}

@roles('db', ‘web’)def run_puppet(): sudo('puppet apply site.pp')

$fab run_puppet #run puppet everywhere$fab run_puppet --roles db # role-specific


env.roledefs = configure_roles()

@roles('db', ‘web’)def run_puppet(): sudo('puppet apply site.pp')

$fab run_puppet #run puppet everywhere$fab run_puppet --roles db # role-specific


def configure_roles():

tags = EC2TagManager(AWS_KEY, AWS_SECRET, regions=['eu-west-1'])

roles = {}

for role in [‘db’, ‘web’]: roles[role] = tags.get_instances(role=role) return roles

https://github.com/mikery/fabric-ec2




Questions?Mike Ryan - Epitech.nl - [email protected]




puppet and aws: getting the best of both worlds

Technology