chef meap ch01

8/11/2019 Chef Meap Ch01

http://slidepdf.com/reader/full/chef-meap-ch01 1/26



MEAP Edition

Manning Early Access Program

Chef in Action

Version 3

Copyright 2014 Manning Publications

For more information on this and other Manning titles go to

www.manning.com

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and

other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.http://www.manning-sandbox.com/forum.jspa?forumID=912

http://www.manning.com/


http://www.manning-sandbox.com/forum.jspa?forumID=912




WelcomeThank you for purchasing the MEAP for Chef in Action. I'm really excited to

see the book reach this point and look forward to it becoming a resource for

anyone wanting to use Chef. This is a beginner to intermediate level book

designed to bring the reader from just hearing about what Chef can do to the

point where entire infrastructures can be managed with confidence and ease.

Each chapter has been carefully written to make the content asapproachable as possible to readers of all levels. A lot of thought has been put

into making the details of working with Chef as clear as possible, introducing

concepts as it makes sense to do so.

The initial release includes Chapters 1 and 2. Chapter 1 provides a good

introduction to what is possible with Chef. A typical use-case is used to drive

the discussion and introduce the beginning terminology one needs when

working with the tool. By the end of this chapter, you should be comfortable

with the basic concepts behind Chef and understand the various parts that

interact in a Chef environment.

Chapter 2 introduces you to writing a cookbook. Since cookbooks are the

foundation of working with Chef, time is spent laying the groundwork for

writing and using cookbooks. A concrete example is used to explain the

concepts involved. By the end of this chapter, you should feel comfortable

using the terms involved and walk away with a working example of how to

manage a part of your infrastructure with Chef.

Looking ahead, part 2 of the book will describe different methodologies of

working with Chef, and show which is best to use depending on the situation.

Part 3 will get you into the internals of Chef, extending functionality to help

Chef do more than what it can out of the box. Part 4 will round the book off

with details about the Chef server and describe situations in which you may not

even want to use a Chef server.

As you're reading, I hope you'll make use of the Author Online forum. I

look forward to your feedback, as it's extremely helpful in the development

process.

— Dimitri Aivaliotis






brief contents

PART 1: INTRODUCTION TO CHEF

1 Chef fundamenta ls

2 Writing your first cookbook

PART 2: USING CHEF

3 The application cookbook

4 Role-based configuration

5 Vagrant-based testing

6 Integrating continuous integration

PART 3: PROGRAMMING CHEF

7 Writing LWRP-based cookbooks

8 Writing HWRP-based cookbooks

9 Writing Chef handlers

10 Writing Ohai plugins

11 Writing knife plugins

PART 4: THE CHEF SERVER

12 Chef server components

13 Chef-solo: Chef without a server

14 Chef-zero: an in-memory Chef server

APPENDIXES:

A Installing Chef

B The Chef community

C Chef command-line tool reference

D Essential resources and providers reference







1This chapter covers

You are about to embark on a journey of learning how to manage

infrastructure with code. Chef is a tool that embodies the principles of

operating system configuration management. By doing so, Chef enables

you to automate your infrastructure by codifying best practices. Chef is oneof the main tools used by DevOps engineers the world over. These

engineers have learned to merge the software engineering and systems

administration disciplines. The basis for developing a tool like Chef lies in

configuration management, so it's important to understand the underlying

principles. To get you familiarized with where Chef fits into this picture, we'll

cover a brief history of the tool. We'll see a few examples of where Chef is

currently being used and how. Then, we'll choose a real-life example and

see how it can be implemented in Chef.

Chef fundamentals

Getting to know Chef, learning about what's possible

Describing a typical infrastructure with Chef

Running Chef, going from code to configured servers



1




Chef is a configuration management tool for automatically provisioning servers

and managing them throughout their lifecycle. Like other configuration

management tools, Chef eliminates the need for installation CDs, instead using

scripts that let Chef download, install, and configure software with no humanintervention needed. It is OS-agnostic, supporting various Linux flavors, Mac OS

X, Windows, Solaris, AIX, and FreeBSD.

Part of what makes Chef so useful is its flexibility; instead of using flat file

syntax, Chef's "recipes" and "cookbooks" are written in Ruby. These function as

libraries of scripts, which allow Chef to make complex decisions during

provisioning such as pulling properties from the operating system during

installation, installing specific applications based on a machine's IP range, and

discover other services running on a cloud-based platform. In addition, Chef allows

you to safely increment changes to a machine, and take legacy systems under

configuration management control without destroying their profile.

A wide range of community-developed sources are available for those who

don't want to reinvent the wheel. Plus, with a small amount of scripting, you can

write Chef recipes to do most anything.

Chef consists of three main parts: the client, the server, and the infrastructure

code.

Figure 1.1 The three main parts of Chef

1.1 What is Chef?



2




The client is the part that runs on the system that you want to manage. The

server is either a Chef server or any webserver that is accessible to that managed

system. It is the part that serves the code to the system. The infrastructure code is

the part that you write to configure the system to be the way you want it to be.

Chef sits squarely in the realm of DevOps - it works best when you have

cooperation from developers and operations. Chef combines programming

concepts with systems engineering, and requires a different way of writing

configurations. As you learn how to use Chef, you'll also discover new ways to

think about describing your infrastructure, encapsulating it into discrete modules

that can be combined in different ways, and generally taking advantage of the fact

that Chef doesn't enforce dogma while still providing a margin of safety.

Chef was born in the mind of Adam Jacob, as he sought to solve systems

integration problems. Early in 2009, he released the first version publicly, and

momentum began to grow. A community built up around the tool, thanks to the

company behind Chef, Opscode, firmly believing that Chef should be Open Source

and everyone should be free to use it as they need to to get their work done.

One of the major motivating factors for using Chef is the fact that it is so

flexible. Chef as a tool was developed to do its tasks well and get out of your wayas quickly as possible. It has predefined ways of installing software, managing

configuration files, starting/stopping services, and other tasks. If these predefined

ways contrast with the way you do something, you can change them. Chef will

enforce the decisions you make, not force you to make them. Due to this

flexibility, adoption over the past few years has risen dramatically.

Chef development has progressed so rapidly, that since version 10.12.0, Chef

has followed the semver versioning scheme. The developers of Chef changed the

versioning scheme during the 10 series in order to enable better use of the minorversion number to provide bugfix releases more quickly.

1.2 Chef and DevOps

1.3 A brief history of Chef



3




1.

2.

3.

NOTE semver

Semantic versioning, or "semver", as it's frequently called, can be

summarized as follows:

The major version number (to the left of the first period)

changes when there are incompatible changes to the API.

The minor version number (in between the two periods)

changes when there are new features, or other changes that

are backwards-compatible.

The patch version number (to the right of the last period)

changes for backwards-compatible bugfixes.

You can learn more about it at .http://semver.org

Initially, both the Chef client and Chef server were written in Ruby, which is an

easy-to-learn programming language. Starting with version 11, the server has been

rewritten in Erlang, a high-performance networking language. This was done to

reduce memory usage on the server to enable it to handle more clients more

efficiently. We'll take a more in-depth look at the different components of Chef

later on in this chapter. More of the details of how Ruby comes into the picture

when using Chef will be covered in the next chapter.

Chef is being used today in companies of all sizes all over the world, from small

web agencies up to multi-national enterprises. It has proven to be a flexible tool,

adaptable to anyone's specific usage. Whether you need to migrate legacy code to a

new infrastructure, or manage a fleet of newly-minted servers, Chef finds its niche.

Some examples of companies that are using Chef:

Nordstrom: the big retailer has integrated Chef into their backend infrastructure.

Facebook: probably one of the largest Chef server installations on the planet isresponsible for configuring new systems.

Disney: Chef is used in multiple departments to manage the configuration anddeployment of a number of online presences.

One of the notable features of each of these companies is that their

contributions make it in some way back into the Chef community.

1.4 Chef in the real world



4

http://semver.org/


http://semver.org/



Configuration management was made to solve a problem faced by sysadmins: that

of creating reproducible infrastructure. To ease the burden of manually configuring

servers, and ensuring that these configurations remain in-sync, you need to employ

a configuration management tool. And that's how we'll be using Chef.Learning Chef is somewhat akin to learning a new spoken language. You'll see

new words, as well as old words that you never used in this way before. You'll

build mental bridges from old concepts to new, in order to create meaning, to grasp

that language. Mostly, though, you work with that language; you spend time with it

to learn it thoroughly. The same holds true for learning Chef. To understand it, you

really have to spend time with it, to become comfortable with the concepts and

know how to use the various tools.

We will look at an example of managing an infrastructure by hand, by looking

at a simple blog and how the infrastructure behind it may need to grow.

You've worked with servers before, either to manage them yourself, or maybe to

install the software necessary to run a website you're working on. Even if not, the

following 3-tier architecture should look familiar:

1.5 Describing your infrastructure with Chef

1.5.1 The infrastructure behind your blog



5




Figure 1.2 A typical 3-tier architecture

The first tier, or presentation tier, is used to deliver content to the client. This

task is usually accomplished with a web server. The middle tier, known as the

application or logic tier, uses business logic combined with the client's request anddata from the data tier to prepare that content for delivery. This is usually done in

an application server of some sort. The final tier, the data tier, is represented by a

database of some kind.

We will use this architecture to describe the infrastructure you'll be working

with in this chapter. Your blog is set up to run on this infrastructure. There's a web

server serving static resources directly to the client. The application server runs the



6




code that saves your posts and turns them into pages that your subscribers read.

And in the back there's a database where your posts are saved. You'll be learning

how to describe this kind of infrastructure in Chef.

Initially, you just want to run your blog. You know you can start off on one

server:

Figure 1.3 All tiers on one server

That's the cheapest solution at the beginning. Over time, though, your blog

grows in popularity. You get a mention on Hacker News, and all of a sudden your

traffic goes through the roof. You need to do something fast, so you call up your

provider and ask for options.

The first option is to scale up, so you order more RAM and your blog is running



7




smoothly again. Soon, though, you realize that you won't be able to handle another

spike in traffic. That's when you decide to go with the second option: scaling out.

To scale out, you separate each tier onto its own machine:



8




Figure 1.4 Expanding onto multiple servers



9




This way, you can scale each individual server up to its full capacity, as traffic

demands. If you need even more resources, you can add additional servers in the

application tier:

Figure 1.5 Adding more application servers



10




Now, when you reach the point that your database can no longer keep up with

the queries your application servers are sending, the data tier can also be expanded

by adding a database slave into the mix:

Figure 1.6 Adding a database slave

You feel comfortable you can now handle any traffic that your eager readers

throw at you. This comfort comes at a cost, though, and not only a monetary cost



11




for the servers themselves. You now have to manage the configuration of different

applications running on multiple servers.

You may be asking yourself, how can Chef help in a situation like this? By

placing the configuration of each tier into Chef, you gain the flexibility of moving

the pieces around without having to reconfigure anything by hand.

Adding application servers or a database slave is then as easy as provisioning a

new server and assigning it the proper role.You no longer have to remember that you added an application server, so that

your web server configuration is up-to-date. You can move from concept, to

single-server, to multi-server implementations with ease:

Figure 1.7 The Chef overlay

"node": {

"name": "appserver1",

"run_list": [

"role": "appserver"

]

}



12




The rest of this section will help you understand the building blocks of Chef so

that you can start working with the tool. You'll be able to build Chef-managed

infrastructure to host your blog and grow easily with increased traffic. Who knows,

maybe if you blog about how you did it, your traffic will increase as a result?

Describing an infrastructure with Chef involves naming the pieces of that

infrastructure and placing the configuration for it within the framework that Chef

provides. Chef is flexible enough that you can place these configuration blocks in

various places, so it may seem daunting at first to know what to put where. The

next few subsections will describe each of these blocks so you can make decisions

on how to describe each piece of your infrastructure.

The basic building block used to describe a single machine within Chef is called a. A houses information as to what is running on a machine and how tonode node

reach it. You configure a by editing its run list.node

A run list consists of a series of recipes and/or roles. It is this list that specifies

what should be running on a particular machine. For our blog example, a possible

run list for the multi-server variant may look like the following.

Collapsing these down to one server would then show the single-server variant

we started with.

S i n g l e - s e r v e r :

1.5.2 Chef terminology

NODES

Table 1.1 Run-lists for blog infrastructure (multi-server variant)m

Type of server Run list

Front end machine role[webserver]

Application server role[appserver]

MySQL database master recipe[mysql::server],role[master]

MySQL database slave recipe[mysql::server],role[slave]



13




1.

2.

3.

recipe[mysql::server],role[appserver],role[webserver]

The run list is what you as the sysadmin configure on the . Chef actuallynode

adds most of the information found on a itself. These auto-attributes are thennode

available for queries and use in your cookbooks. They are present on thenode

object in Chef, which means you can use them by referring tonode.attribute

.

SIDEBAR Attribute styles

Attributes are pieces of information found in multiple places in Chef. We

will discuss how to find them and where to best place them later on in

this book. To reference them and use them, you can follow one of three

conventions:

node.attribute (as seen above) - we will use this convention

to refer to attributes in prose, as it is easier to read.node[:attribute] - we will use this convention in code

samples, as many text editors can use syntax-highlighting to mark

this representation specially

node['attribute'] or - you will seenode["attribute"]

this convention very often in community cookbooks, so it's good to

know that this is also a valid way of referring to attributes

Where the run list on a node is used to describe a machine should do, awhat

describes it should do it. Chef uses the term tocookbook how cookbook

represent a collection of recipes which contain code that is executed to configure a

machine. It is the workhorse of a Chef deployment.

A is versioned, so that you may specify a particular version of thiscookbook

code collection that should be executed on a machine. You can thereby maintain

releases and ensure that only the desired code is run on the correct machines.

We will be developing our first in the next chapter, so we won't gocookbook

into more detail here.

COOKBOOKS



14




SIDEBAR Idempotence

One of the most important principles of configuration management is

the concept of idempotence. This means basically that any change

made has no side effects. You can repeatedly replace one line in a

configuration file, and only that line will be affected. Adding one file to a

directory will not cause another file to be deleted. Without this principle,

operating system configuration management wouldn't be possible.

The purpose behind programming a configuration management tool for

idempotence is to be assured that any convergence run that starts from

the same state will always achieve the same result. If this were not so,

you could not be sure of the state of the system, and would never know

if you were in that desired state. So, when describing your infrastructure

in Chef, be sure to make the actions performed in your recipes

idempotent.

Along with cookbooks, may be specified in a node's run list to describeroles

what a node should do. A is simply a named set of attributes and/or recipesrole

that should be run on a node. Each is named, but is not otherwise versioned.role

TIP Versioning roles

Versioning roles is a topic that comes up from time to time in the

Chef community. Many would like to have roles versioned just as

cookbooks are, so that a specific version may be specified for

particular nodes. If versioning roles is important to you, one way to

implement them is to use a naming convention that includes the

version number in the name of the role.

A is representative of what you would like a node to do. A web serverrole

would have the role "webserver"; a load balancer the role "load-balancer". Naming

roles after what they should do and including only distinguishing attributes helps to

make them modular and re-usable. An example of a "webserver" role is thefollowing:

ROLES

name "webserver"

description "A web server runs nginx and responds to requests

for blog.example.com."

default_attributes(

"server_name": "blog.example.com"

)



15




This role looks like it configures a server block in nginx to respond to requests

for http://blog.example.com, and that is precisely what it does. We've used adefault attribute to specify our server_name (the configuration piece that is

matched to the Host header in a request), and named the "nginx" cookbook in our

run list. So, nginx will be installed and configured to fulfill the role of "webserver"

on whichever node we assign this to.role

A node may have zero or more , depending on how you encapsulate theroles

various pieces of your infrastructure. For example, you may have a base thatrole

configures some system-wide settings that are valid on all servers that you manage:

This base would then be added to every node. The "webserver"role role

above would come after it in the run list, to configure that node for use as a web

server.

A is a collection of JSON-formatted items. Each item has one or moredata bag

key/value pairs containing data that you'd like to use in a cookbook. The structure

of this data is completely up to you - the only requirement is that it is valid JSON.

An example "databases" could contain the following item todata bag

represent the database that runs behind your blog.

run_list(

"recipe[nginx]"

)

name "base"

description "The basic settings applied to every node."

default_attributes(

:postfix: {

"aliases": { "root": "[email protected]" },

"relayhost": "smtp.example.com"

},

:rsyslog: {

"loghost": "logger.example.com"

}

)

run_list( "recipe[postfix::client]",

"recipe[ntp::client]",

"recipe[rsyslog::client]"

)

DATA BAGS



16

http://blog.example.com/

mailto:[email protected]


mailto:[email protected]

http://blog.example.com/



Each item has an id, which may only contain letters, numbers, anddata bag

the underscore character (_). This is the only way that Chef has to distinguish

items since they are global objects. So, ensure that each item within adata bag

has a unique id.data bag

The "name" is simply what we'd like to call our database. The "type" is here in

case we have multiple database drivers and want to distinguish which one to use to

create and access this database.

The "users" structure bears a bit of explanation here. Multiple users may be

granted different access rights to our database, so we want a structure that can

encompass this. We chose an array for this purpose because it can easily represent

a list of objects. The objects in this case are individual users, each with a

"username" and "password". The "owner" flag is set to indicate that this particular

user should be allowed administrative rights (SUPER) on the database itself.

But wait, you say, we don't want to see our super secret databaseeverybody

password. We can't just store it in plain text.

You're right. We don't want just anybody to be able to access the database

credentials. So, support the concept of encryption. You can encryptdata bags

each item with a secret key that only the appropriate nodes have. Since encryption

is such a detailed topic, we can only brush upon it here. More details about

encrypting items and how to manage distribution of keys will bedata bag

covered later in the book.

{

"id": "blog_example_com",

"name": "blogdb",

"type": "mysql",

"users": [

{

"username": "blogdbuser",

"password": "bigblogs3cret", "owner": true

}

]

}



17




An in Chef is a logical group under which nodes are placed.environment

Depending on your infrastructure, they can represent various stages of code

(production, staging, test) or can be labelled after application (portal1, portal2,

portal3) or after some other criteria that makes sense for your organization. They

exist to help you organize your nodes and set certain parameters that are valid for a

particular set of nodes.

Attributes may be specified in an . They are global for theenvironment

whole environment and will be available to any node in that environment. You

would typically want to place here any non-default value that should only be

present on a certain set of nodes.

An is defined in a Ruby-formatted key/value file, very similarenvironment

to that used for a role.

The cookbook version specification restricts a cookbook to be a certain version

in that . This is useful for rolling-out changes to cookbooks in aenvironment

controlled fashion. Any node that requires the "php" cookbook in the "blogger"

will download version "1.4.6" and configure the node accordingenvironment

to the code available in that version.

Now that the terms should be familiar, we will use them to describe the example

infrastructure from the beginning of the chapter. We'll take it step-by-step,

concentrating on Chef itself, and how to use the tools it provides to accomplish this

task. We will be glossing-over a number of details here, so don't try the commands

as given and expect everything to work. The details as to where to find a cookbook

and how to manage configuration exactly will begin in the next chapter.

ENVIRONMENTS

name "blogger"

description "This environment is responsible for running the

blogger application."

override_attributes(

:haproxy => {

"member_port" => "80"

}

)

cookbook_versions(

"php" => "1.4.6"

)

1.6 How to run Chef



18




SIDEBAR knife

is the administrator's command-line tool. It is used toknife

communicate with the Chef server, as well as to perform administrative

tasks on your workstation. You can use it to communicate directly with

managed nodes. It is versatile enough to query the server, upload new

code to the server, bootstrap a new node, create a new client, and be

extensible through plugins. is the command you'll be using mostknife

often when wanting to perform your day-to-day work with Chef.

Using consists of calling a subcommand and perform an actionknife

with that subcommand. Very often the same actions can be used

amongst multiple subcommands. For example, , , ,create show list

and are used with the , , ,delete environment role cookbook

and subcommands. will give you moreclient node knife help

information about the supported subcommands and the various actions

and options they support.We'll not be covering all the details of the various knife

subcommands, but show examples of using some of them to get you

familiar with the basic usage.

The first thing we need to do is define the environment. This will be where we set

any global attributes we need, as well as where we can fix cookbooks to a specific

version. Since we're building a blog, let's call this the "blogger" environment. We'llgo out on a limb here, and say that our blogging software is written in PHP and is

backed by a MySQL database. The frontend will be taken care of by nginx. So,

we'll specify the versions of these cookbooks we want to use, as well as override

some defaults right here in the environment.

1.6.1 Setting-up the infrastructure: building the foundation

name "blogger"

description "This environment is responsible for running the

blogger application."

override_attributes( :haproxy => {

"member_port" => "8181"

},

:nginx => {

"listen" => ":8181"

}

)

cookbook_versions(

"database" => "2.0.0",

"haproxy" => "1.6.2",



19


chef meap ch01

Documents